The game is actually a combination of an action-shooter game and a puzzle game, where each enemy is a puzzle itself. The more effective you can kill an enemy, the more points you will get.

Each level is rated with five stars, where a one star is bad, and five stars is perfect.

As the game is based on a superhero, I decided to make the intro with a comic-style to it.

Just wanted to quickly mention that my article “Modern art using the GPU” article for SDJ was published earlier this year. The article is based on my previous tutorials for XNA 3.1.

As the article is based on XNA 3.1, it’s a bit out of date. My current tutorial series covering Shader programming for XNA 4.0 will cover most of the differences, but all examples, code, shaders and text is rewritten for the new version.

If you are interested to read this it’s available on http://en.sdjournal.org/modern-art-using-the-gpu-learn-about-xna-sdj-extra-022011/, but I think you need to be a registered member.

Still, nice to see, and I like the cover

Looking forward to work more with my new series. You can find the first three parts in the tutorial section.

Cheers!

Hi, and welcome to Tutorial 3 of my XNA 4.0 Shader Programming tutorial. Today we are going to implement an other lighting algorithm called Specular Lighting. This algorithm builds on my Ambient and Diffuse lighting tutorials, so if you haven’t been trough them, now is the time.

Technique: Specular light

So far, we got a nice lighting model for making a good looking lighting on objects. But, what if we got a blank, polished or shiny object we want to render? Say a metal surface, plastic, glass, bottle and so on? Diffuse light does not include any of the tiny reflections that make a smooth surface shine.

fig 3.1 – specular light in EVE Online (GREAT game btw.)

To simulate this shininess, we can use a lighting model named Specular highlights.
Specular highlights calculate another vector that simulates a reflection of a light source, which hits the camera, or “the eye”.

What’s “the eye” vector, you might think? Well, it’s a pretty easy answer to this. It’s the vector that points from our camera position to the camera target.

We already got this vector in our application code:

viewMatrix   = Matrix.CreateLookAt( new Vector3(x, y, z), Vector3.Zero, Vector3.Up );

The position of “The eye” is the first parameter in CreateLookAt:

new Vector3(x, y, z)

So let’s take this vector, and store it in a variable:

Vector4 vecEye = new Vector4(x, y, z,0);

Note: x,y,z represents a point in 3d space.

Let’s look more closely about how to use the shader after we have created it.

The formula for Specular light is

I=Ai*Ac+Di*Dc*N.L+Si*Sc*(R.V)ⁿ (3.1)

Where

R=2*(N.L)*N-L

fig 3.2 – Vector diagram for specular light

As we can see, we got the new Eye vector V, and also we got a reflection vector R.

To compute the specular light, we need to take the dot product of R and V and use this in the power of n, where n is controlling how “shiny” the object is.

Implementing the shader

We start with the VertexShader. There is only one modification, and that is to add a View vector to the VertexShaderOutput structure, so we will need to calculate this in the VertexShaderFunction.

struct VertexShaderOutput
{
    float4 Position : POSITION0;
	float3 Normal : TEXCOORD0;
	float3 View : TEXCOORD1;
};

To calculate the View vector V (check figure 3.2), we need the position of the “eye”, in other words – the position of the camera, we will need to add a new global variable to the shader.

float3 EyePosition;

Now, in the vertex shader, we need to set the View-vector in the output structure:

output.View = normalize(float4(EyePosition,1.0) - worldPosition);

The whole VertexShaderFunction can be seen below:

// The output from the vertex shader, used for later processing
struct VertexShaderOutput
{
    float4 Position : POSITION0;
	float3 Normal : TEXCOORD0;
	float3 View : TEXCOORD1;
};

// The VertexShader.
VertexShaderOutput VertexShaderFunction(VertexShaderInput input,float3 Normal : NORMAL)
{
    VertexShaderOutput output;

    float4 worldPosition = mul(input.Position, World);
    float4 viewPosition = mul(worldPosition, View);
    output.Position = mul(viewPosition, Projection);
	float3 normal = normalize(mul(Normal, World));
	output.Normal = normal;
	output.View = normalize(float4(EyePosition,1.0) - worldPosition);

    return output;
}

Now we got what we need to calculate the specular light in the pixel shader!

The pixel shader will return a float4, which represents the finished color, I, of the current pixel, based on the formula for specular lighting described earlier.

The new thing in the Pixel Shader for Specular Lighting is to calculate and use a reflection vector for L by N, and using this vector to compute the specular light.

So, we start with computing the reflection vector of L by N:
R = 2 * (N.L) * N – L

As we can see, we have already computed the Dot product N.L when computing the diffuse light.

float4 diffuse = saturate(dot(-LightDirection,normal));

Let’s use this and write the following code:

float4 reflect = normalize(2*diffuse*normal-float4(LightDirection,1.0));

Note: We could also use the reflect function that is built in to HLSL instead, taking an incident vector and a normal vector as parameters, returning a reflection vector:
float3 ref =  reflect( L, N );

Now, all there is left is to compute the specular light. We know that this is computed by taking the power of the dot product of the reflection vector and the view vector, by n: (R.V)^n
You can think of n as a factor for how shiny the object will be. The more n is, the less shiny it is, so play with n to get the result you like.

As you might have noticed, we are using a new HLSL function pow(a,b). What this does is quite simple, it returns a^b.

float4 specular = pow(saturate(dot(reflect,input.View)),15);

Now we are finally ready to put all this together and compute the final pixel color:

return AmbientColor*AmbientIntensity+DiffuseIntensity*DiffuseColor*diffuse+SpecularColor*specular;

This formula should no longer be a surprise for anyone, right?

We start by calculating the Ambient and Diffuse light, and add these together. Then we take the specular light color and multiply it with the specular component we just calculated, and add it with the Ambient and Diffuse color we created in the previous techniques.

The whole PixelShaderFunction code can be seen below.

// The Pixel Shader
float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
	float4 normal = float4(input.Normal, 1.0);
	float4 diffuse = saturate(dot(-LightDirection,normal));
	float4 reflect = normalize(2*diffuse*normal-float4(LightDirection,1.0));
	float4 specular = pow(saturate(dot(reflect,input.View)),15);

    return AmbientColor*AmbientIntensity+DiffuseIntensity*DiffuseColor*diffuse+SpecularColor*specular;
}

And the whole shader effect code:

// XNA 4.0 Shader Programming #2 - Diffuse light

// Matrix
float4x4 World;
float4x4 View;
float4x4 Projection;

// Light related
float4 AmbientColor;
float AmbientIntensity;

float3 LightDirection;
float4 DiffuseColor;
float DiffuseIntensity;

float4 SpecularColor;
float3 EyePosition;


// The input for the VertexShader
struct VertexShaderInput
{
    float4 Position : POSITION0;
};

// The output from the vertex shader, used for later processing
struct VertexShaderOutput
{
    float4 Position : POSITION0;
	float3 Normal : TEXCOORD0;
	float3 View : TEXCOORD1;
};

// The VertexShader.
VertexShaderOutput VertexShaderFunction(VertexShaderInput input,float3 Normal : NORMAL)
{
    VertexShaderOutput output;

    float4 worldPosition = mul(input.Position, World);
    float4 viewPosition = mul(worldPosition, View);
    output.Position = mul(viewPosition, Projection);
	float3 normal = normalize(mul(Normal, World));
	output.Normal = normal;
	output.View = normalize(float4(EyePosition,1.0) - worldPosition);

    return output;
}

// The Pixel Shader
float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
	float4 normal = float4(input.Normal, 1.0);
	float4 diffuse = saturate(dot(-LightDirection,normal));
	float4 reflect = normalize(2*diffuse*normal-float4(LightDirection,1.0));
	float4 specular = pow(saturate(dot(reflect,input.View)),15);

    return AmbientColor*AmbientIntensity+DiffuseIntensity*DiffuseColor*diffuse+SpecularColor*specular;
}

// Our Techinique
technique Technique1
{
    pass Pass1
    {
        VertexShader = compile vs_2_0 VertexShaderFunction();
        PixelShader = compile ps_2_0 PixelShaderFunction();
    }
}

In the example, I made the zombie spin instead of the camera so you can see how the specular light is working.

To implement the shader, all we need to do is to add the two new parameters and set them, check the shader source to see how it all fit together

Download Source (XNA 4.0)

Hi, and welcome to Tutorial 2 of my XNA 4.0 Shader Programming tutorial. Today we are going to work on Tutorial 1 in order to make the lighting equation a bit more interesting, by implementing Diffuse lighting.

Diffuse light isn’t very different from ambient light implementation wise, but it got one very important property, a direction to the light. As we saw, using only ambient light can make a 3D scene look 2D, but adding a diffuse will increase the realism of the scene and add a nice 3D look to it. Figure 1 shows the same zombie we rendered in the first example, but with diffuse white light, and a dark gray ambient light

Figure 1 – Diffuse light

As mentioned above, the ambient light got the following equation:

I = Aintensity * Acolor (1.1)

Diffuse light builds on this equation, adding a directional light to the equation:

I = Aintensity x Acolor + Dintensity x Dcolor x N.L (1.2)

From this equation, you can see that we still use the Ambient light, with an addition of two more variables for describing the color and intensity of the Diffuse light, and two vectors N and L for describing the light direction L and the surface normal N.

We can think of diffuse lighting as a value that indicates how much a surface reflects light. The light that is reflected will be stronger and more visible when the angle between the Normal N and the light direction L gets smaller.

If L is parallel with N, the light will be most reflected, and if L is parallel with the surface, the light will be reflected with the minimal amount.

To compute the angle between L and N, we can use the Dot-product, or the scalar product. This rule is used to find the angle between two given vectors and can be defined as the following:
N.L = |N| x |L| x cos(a)     

where |N| is the length of vector N, |L| is the length of vector L and cos(a) is the angle between the two vectors.

Let’s try to convert this in to HSLS! In order to do this, we start by defining three new global variables:

float3 DiffuseDirection;
float4 DiffuseColor;
float DiffuseIntensity;

These will contain the direction of the light  ( L ), the color of the diffuse light and the intensity of the light.

The Vertex Shader input is the same as before, containing only the position.

struct VertexShaderInput
{
    float4 Position : POSITION0;
};

The VertexShaderOutput will contain one additional member, the Normal. The normal will be calculated in the Vertex Shader function, based on the objects Normal at the given vertex.

struct VertexShaderOutput
{
    float4 Position : POSITION0;
	float3 Normal : TEXCOORD0;
};

But where do we get this Normal from? We need to pass in the Normal as input to the Vertex Shader! So.. Why didn’t we add it to the VertexShaderInput structure? Because the NORMAL semantic isn’t supported when defining structures in XNA. So how do we get the normals? We pass in the Normal as a parameter to the VertexShaderFunction in addition to the VertexShaderInput parameter. Sounds complicated? Some code might clear it up for you

VertexShaderOutput VertexShaderFunction(VertexShaderInput input,float3 Normal : NORMAL)

Here you can see that we pass in the VertexShaderInput structure, as well as the Normal. When passing in the parameter directly, the NORMAL semantic will work

The Vertex Shader function is very similar to the one in the ambient light tutorial, but we also transform the Normal into worldspace, as this is the space we would like to calculate the normal in, just like the position:

VertexShaderOutput VertexShaderFunction(VertexShaderInput input,float3 Normal : NORMAL)
{
    VertexShaderOutput output;

    float4 worldPosition = mul(input.Position, World);
    float4 viewPosition = mul(worldPosition, View);
    output.Position = mul(viewPosition, Projection);
	float3 normal = normalize(mul(Normal, World));
	output.Normal = normal;

    return output;
}

So what’s really happening here? We take the input Normal and multiply it with the World matrix, and then we normalize it and store it in a variable normal. Next, we must remember to set the Normal variable in output, to make sure it’s passed to the Pixel Shader by setting output.Normal = normal.

And that’s really it for the Vertex Shader. But it’s still in the Pixel Shader most of the “magic” happens.

In order to pass the Normal to the pixel shader, we added a float3 Normal : TEXCOORD0 variable to the VertexShaderOutput structure. We needed to store the Normal in a register on the GPU, but since NORMAL doesn’t exist, we just used another one that is not in use yet; the TEXCOORD0. In other words, TEXCOORDn (n = any number up to the supported amount of registers on your GPU, anything between 0 and 7 is usually safe to use) can be used for any values, and as we don’t yet use any texture coordinates, we can easily just use these registers as a storage for our Normal-vector.

Moving on. The pixel shaders job is to calculate the final light equation. It will need to implement 1.2:

I = Aintensity x Acolor + Dintensity x Dcolor x N.L

We already got Aintensity and Acolor. Also, the Dintensity and Dcolor are just variables passed in to the shader, just as the ambient light.

But what about the last part? N.L? This means that we take the dot-product between N and L (remember, the angel of the incoming light, “compared” with the Normal).

Let’s try! First, we store our Normal vector, and convert it to a float4, as this is the variable-type we use in the other calculations. When we use the build in HLSL function dot(x,y) to calculate the dot-product between L and N. Also, we negate L as the value L is containing the direction the light comes FROM, and not where the light points TO.

Now, all that is left is to put this into eq. 1.2 like the code shows below:

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
	float4 norm = float4(input.Normal, 1.0);
	float4 diffuse = saturate(dot(-DiffuseDirection,norm));

    return AmbientColor*AmbientIntensity+DiffuseIntensity*DiffuseColor*diffuse;
}

Not very hard right?

Now if you apply this shader to a scene, the output will be something like this:

There is nothing new when it comes to how this shader is used in XNA. Take a look at the source so see how it all fits together and play with the values.

The complete shader listing can be seen below:

// XNA 4.0 Shader Programming #2 - Diffuse light

// Matrix
float4x4 World;
float4x4 View;
float4x4 Projection;

// Light related
float4 AmbientColor;
float AmbientIntensity;

float3 DiffuseDirection;
float4 DiffuseColor;
float DiffuseIntensity;


// The input for the VertexShader
struct VertexShaderInput
{
    float4 Position : POSITION0;
};

// The output from the vertex shader, used for later processing
struct VertexShaderOutput
{
    float4 Position : POSITION0;
	float3 Normal : TEXCOORD0;
};

// The VertexShader.
VertexShaderOutput VertexShaderFunction(VertexShaderInput input,float3 Normal : NORMAL)
{
    VertexShaderOutput output;

    float4 worldPosition = mul(input.Position, World);
    float4 viewPosition = mul(worldPosition, View);
    output.Position = mul(viewPosition, Projection);
	float3 normal = normalize(mul(Normal, World));
	output.Normal = normal;

    return output;
}

// The Pixel Shader
float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
	float4 norm = float4(input.Normal, 1.0);
	float4 diffuse = saturate(dot(-DiffuseDirection,norm));

    return AmbientColor*AmbientIntensity+DiffuseIntensity*DiffuseColor*diffuse;
}

// Our Techinique
technique Technique1
{
    pass Pass1
    {
        VertexShader = compile vs_2_0 VertexShaderFunction();
        PixelShader = compile ps_2_0 PixelShaderFunction();
    }
}

Download Source (XNA 4.0)

Hi and welcome to tutorial #4 in my Kinect Fundamentals tutorial. In this tutorial, I will show you how you can implement Skeletal Tracking using the Kinect SDK for Windows API, and how you can move a cursor by using your hand.

It’s very simple, and it is quite similar to the approaches we have used in the previous tutorials.

Let’s get started!

First of all, you must initialize your NUI Runtime instance using the RuntimeOptions.UseSkeletalTracking parameter, like this:

kinectSensor.Initialize(RuntimeOptions.UseSkeletalTracking);

Remember, if you want to initialize using more RuntimeOptions you can & them together:

kinectSensor.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseSkeletalTracking);

After the kinect knows that is should start tracking the skeleton, you must tell it how it should be done. You got a lot of different parameters to do this, and the key is to play around with these until you are satisfied.

kinectSensor.SkeletonEngine.TransformSmooth = true;
TransformSmoothParameters p = new TransformSmoothParameters
{
    Smoothing = 0.75f,
    Correction = 0.0f,
    Prediction = 0.0f,
    JitterRadius = 0.05f,
    MaxDeviationRadius = 0.04f
};


kinectSensor.SkeletonEngine.SmoothParameters = p;

Play around with the parameters to see what they do. Basically, it’s all about if you want smooth transforming or not, and how the kinect should anticipate your movement and handle it.

Then you will need to add a new event to your Kinect NUI Runtime object. It’s the same as before but this time you will be using the SkeletonFrameReady event:

kinectSensor.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(kinectSensor_SkeletonFrameReady);

Only one more thing left, and that is the implementation of the event itself. Let me show you the code first as it’s pretty much self explaining:

void kinectSensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    SkeletonFrame allSkeletons = e.SkeletonFrame;
    SkeletonData playerSkeleton = (from s in allSkeletons.Skeletons where s.TrackingState == SkeletonTrackingState.Tracked select s).FirstOrDefault();
    Joint rightHandJoint = playerSkeleton.Joints[JointID.HandRight];

    try
    {
        position = new Vector2((((0.5f * rightHandJoint.Position.X) + 0.5f) * (resolution.X)), (((-0.5f * rightHandJoint.Position.Y) + 0.5f) * (resolution.Y)));
    }
    catch
    {
        // handle error
    }

}

First you get all the skeletons in the returned collection. Then we find all the skeletons that belong to the player currently being tracked (this is automatic), and find the joint of the Right Hand. This joint includes a position that you simply can use to render your object at.

All that is left is to render a sprit or 3d model at the returned position.

Download Source (XNA 4.0)

So, you want to learn the magic that puts the gold into modern games?

Note: This series is an update to the previous XNA Shader Programming series I have written for XNA 3.0. If you know both XNA 3.0 and XNA 4.0, it should be no problem for you to understand the shaders from the other series if you want to move on faster than I can update the old ones.

My name is Petri Wilhelmsen and is a member of Dark Codex Studios. We usually participate in various competitions regarding graphics/game development, at The Gathering, Assembly, Solskogen, Dream-Build-Play, NGA and so on.

The XNA Shader Programming series will cover many different aspects of XNA, and how to write HLSL shaders using XNA and your GPU. I will start with some basic theory, and then move over to a more practical approach to shader programming.

The theory part will not be very detailed, but should be enough for you to get started with Shaders and be able to experiment for yourself. It will cover the basics around HLSL, how the HLSL language works and some keywords that is worth knowing about.

Today I will cover XNA and HLSL, as well as a simple ambient lighting algorithm.

Prerequisites
Some programming in XNA (this tutorial is about shader programming and not XNA in it self), as I wont go much into details about loading textures, 3d models, matrices and some math.

2001: A shader odyssey – A brief history of shaders
Before DirectX8, GPU’s had a fixed way to transform pixels and vertices, called “The fixed pipeline”. This made it impossible to developers to change how pixels and vertices was transformed and processed after passing them to the GPU, and made games looked quite similar graphics wise.

In 2001, DirectX8 introduced the vertex and pixel shaders, as a utility that developers could use to decide how the vertices and pixels should be processed when going through the pipeline, giving them a lot of flexibility.
An assembly language was used to program the shaders, something that made it pretty hard to be a shader developers, and shader model 1.0 was the only supported version. But this changed once DirectX9 was released, giving developers the opportunity to develop shaders in a high level language, called High Level Shading Language( HLSL ), replacing the assembly shading language with something that looked more like the C-language. This made shaders much easier to write, read and understand.

DirectX10.0 introduced a new shader, the Geometry Shader, and was a part of Shader Model 4.0. But this required a new state-of-the-art graphics card, and Windows Vista.

The latest addition to the DirectX series is the DirectX 11 including a tesselator, DirectCompute for paralell programming and much more.

XNA supports Shader Model 1.0 to 3.0, but works on XP, Vista and XBox360!

Taking the red pill
So, the question is.. What is a shader? Well, a shader is simply a set of instructions that will be run on you graphics processing unit (GPU), performing specific tasks of you need. This makes it possible to developer small/tiny applications that make you in control of three stages in the graphics pipeline: The vertex shader stage, the geometry shader stage, and the pixel shader stage.

Fig 1 –High level Programmable pipeline

As you can see in fig 1, you are able to program all the green squares, and the rest is fixed, meaning that you cannot control them. The geometry shader is not supported by Xbox360 and XNA, so it will not be covered in this article. Rather, let’s take a quick tour through the vertex and pixel shaders (don’t get frustrated if you don’t understand the code yet, it will be covered in a later section).

Vertex shader
Vertex shaders are used to manipulate vertex-data, per vertex. This can for example be a shader that makes a model “fatter” during rendering by moving vertexes along their normals to a new position for every vertex in the model (deform shaders).
Vertex shaders get input from a vertex structure defined in the application code, and load this from the vertex buffer, passed into the shader. This describes what properties each vertex will have during shading: Position, Color, Normal, Tangent and so on.

The vertex shader sends its output for later use to the pixel shader. To define what data the vertex shader will pass to the next stage can be done by defining a structure in the shader, containing the data you want to store, and make the vertex shader return this instance, or by defining parameters in the shader, using the out keyword. Output can be Position, Fog, Color, Texture coordinates, Tangets, Light position and so on.

An example of a simple Vertex Shader that transforms an object to a position on the screen can be seen below.

struct VertexShaderInput
{
    float4 Position : POSITION0;
};

struct VertexShaderOutput
{
    float4 Position : POSITION0;
};

VertexShaderOutput VertexShaderFunction(VertexShaderInput input)
{
    VertexShaderOutput output;

    float4 worldPosition = mul(input.Position, World);
    float4 viewPosition = mul(worldPosition, View);
    output.Position = mul(viewPosition, Projection);

    return output;
}

Pixel shader
The Pixel shader manipulates all pixels (per pixel) on a given model/object/collection of vertices. This can be a metal box, where we want to customize the lighting algorithm on, colors and so on. The pixel shader gets data from the vertex shader’s output values, like position, normals and texture coordinates, and interpolates these values to the different pixels. A very simple and small pixel shader can look like the snippet below.

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    return float4(1,0,0,1);
}

The code colors everything that flows through the shader RED.

The pixel shader can have two output values, Color and Depth.

All of the stages displayed in Fig 1 are working together in order to synthesize images and display them on the monitor.

So, are you ready to take control of the GPU using shaders? In that case, sit back, you are about to join a long ride and be reborn in the world of shaders.

HLSL
High Level Shading Language(HLSL) is used to develop shaders using a language similar to C. Just as in C, HLSL gives you tools like declaring variables, functions, data types, testing( if/else/for/do/while and so on) and much more, in order to create a logic for processing vertices and pixels. Below is a table of some keywords that exists in HLSL. This is not all of them, but some of the most important ones.

HSLS offers a huge set of functions that can be used to solve complex equations. As we go through this article, we will cover many, but for now, here is a list with just a handful of them. It’s important to learn all of them in order to create high-performance shaders without re-implementing the wheel.

For a complete list: http://msdn2.microsoft.com/en-us/library/bb509611.aspx

Effect files
Effect files (.fx) makes shader developing in HSLS easier. You can think of them as containers where you can store shader functionality, including vertex-, geometry- and pixel shaders. This includes global variables, functions, structures, vertex shader functions, pixel shader functions, different techniques/passes, textures and so on.

We have already seen how to declare variables and structures in a shader, but what is this technique/passes thing?  It’s pretty simple. One Shader can have one or more techniques. One Technique is a piece of functionality that represents one functionality of a given .fx file. Each technique can have a unique name, and from the game/application, we can select what technique in the shader we want to used when rendering a given geometry, by setting the CurrentTechnique property of the Effect class like this:

effect.CurrentTechnique = effect.Techniques["AmbientLight"];

One .fx file represents one effect. On the line above, we tell the effect to use the technique “AmbientLight”. One technique can have one or more passes, and we must remember to process all passes in order to archive the result we want.

This is an example of a shader containing one technique named “AmbientLight” and one pass named “P0”:

technique AmbientLight
{
    pass P0
    {
        VertexShader = compile vs_1_1 VS();
        PixelShader = compile ps_1_1 PS();
    }
}

This is an example of a shader containing one technique and two passes:
technique Shader
{
pass P0
    {
        VertexShader = compile vs_1_1 VS();
        PixelShader = compile ps_1_1 PS();
    }
pass P1
    {
        VertexShader = compile vs_1_1 VS_Other();
        PixelShader = compile ps_1_1 PS_Other();
    }
}

This is an example of a shader containing two techniques and one pass:
technique Shader_11
{
    pass P0
    {
        VertexShader = compile vs_1_1 VS();
        PixelShader = compile ps_1_1 PS();
    }
}

technique Shader_2a
{
    pass P0
    {
        VertexShader = compile vs_1_1 VS2();
        PixelShader = compile ps_2_a PS2();
    }
}

We can see that a technique got two functions, one for the pixel shader and one for the vertex shader:
VertexShader = compile vs_1_1 VS2();
PixelShader = compile ps_1_1 PS2();

This tells us that the technique will use VS2() function as the vertex shader, PS2() function as the pixel shader, and the shader requires shader model 1.1 or higher. This makes it possible to have a different and more complex shader for GPUs supporting higher shader model versions, and simpler shaders for older hardware that only support the earlier shader models.

XNA and shaders
It’s really easy to implement shaders in XNA. In fact, only a few lines of code are needed to load and use a shader. Here is a list of steps that can be followed when making a shader, each of them covered in detail below:
1. Make the shader
2. Put the shade file (.fx) in “Contents”
3. Make an instance of the Effect class
4. Initiate the instance of the Effect class.
5. Select what technique you want to use
6. Pass different parameters to the shader
7. Draw the scene/object

The steps in a bit more detail:

1.When making a shader, several programs like notepad, the visual studio editor and so on can be used. There are also some shader IDEs available, and personally I like to use nVidias FX Composer: http://developer.nvidia.com/object/fx_composer_home.html

2. When the shader is created, drag it into the ”Content” folder, so it gets an asset name. The asset name will be the same as the filename of the fx file, but you can edit this asset name so it better suits your needs.

3. XNA Framework includes an Effect class that is used to load and compile the shaders. To make an instance of this class, write the following line of code:

Effect effect;
Effect is a part of the “Microsoft.Xna.Framework.Graphics” library, so remember to add this line of code to the using statement block:
using Microsoft.Xna.Framework.Graphics;

4. To initiate the shader, we can use the Content property to either load if from the project or from a file:

effect = Content.Load<Effect>(“Shader”);
In the line above, “Shader” is the asset name of the shader you added to the Contents folder.

5. Select what technique you want to use:
effect.CurrentTechnique = effect.Techniques["AmbientLight"];

6. Pass the parameters you want to set in the shader.

First you need to create a EffectParamter object:
EffectParameter projectionParameter;

Then in the LoadContent() function you bind the parameter object with a variable in the shader:
projectionParameter = effect.Parameters["Projection"];

Now, you can set the parameter by using the SetValue function like this:
projectionParameter.SetValue(projection);

where projection is a matrix (here: representing the projection matrix).

7. All that is left is to go through all the different passes in the shader and render your object. This is done by using a loop:

for (int i = 0; i < effect.CurrentTechnique.Passes.Count; i++)
{
    //EffectPass.Apply will update the device to
    //begin using the state information defined in the current pass
    effect.CurrentTechnique.Passes[i].Apply();

    //sampleMesh contains all of the information required to draw
    //the current mesh
    graphics.GraphicsDevice.DrawIndexedPrimitives(
        PrimitiveType.TriangleList, 0, 0,
        meshPart.NumVertices, meshPart.StartIndex, meshPart.PrimitiveCount);
}

Technique: Ambient light
So, you now got the answer to what a shader is! Let’s take that in to use and create your first real shader! The first shader you will write is a really simple one that just transforms the vertexes and calculates the ambient light on the model.
But wait… What is the “Ambient light” thing we are talking about?

Well, ambient light is the basic light in a scene that’s “just there”. If you go into a complete dark room, the ambient light is typically zero, but when walking outside there is almost always some light that makes it possible to see. This light got no direction and is there to make sure objects that are facing a light source, will have a basic color.

A scene with only a yellow ambient light can be seen in figure 2. The scene consists of a black background color and an un-textured zombie model. We will make this zombie scene look a lot better as we are going through this series.

Fig 2. Scene lit with ambient light.

Before we can implement the ambient light technique, we need to understand it. The formula for Ambient light can be seen in 1.1 below.
I = Aintensity x Acolor ( 1.1)

I is the final light color on a given pixel, Aintensity is the intensity of the light (usually between 0.0 (0%) and 1.0 (100%)), and Acolor is the color of the ambient light. This color can be a hardcoded value, a parameter or a texture.

Ok, let’s start implementing the shader. First of all, we need a matrix that represents the world matrix, the view matrix and the projection matrix:

float4x4 World;
float4x4 View;
float4x4 Projection;

float4 AmbientColor;
float AmbientIntensity;

You might also notice that we added two variables in the end, the AmbientColor and AmbientIntensity. I don’t think I need to explain those now?

Next, we create a structure that will contain the shader inputs, and outputs.
These are used in the Vertex Shader itself.

The structures contains a variable of the type float4 with the name Position. The : POSITION in the end tells the GPU what register to put this value in. So, what is a register? Well, a register is simply just a container in the GPU that contains data. The GPU got different registers to put position data, normal, texture coordinates and so on, and when defining a variable that the shader will pass to the pixel shader, we must also decide where in the GPU this value is stored.

When declaring the Vertex Shader function “VertexShaderFunction” (works like the main() function for Vertex Shaders), we specify that the function should return a VertexShaderOutput object (you can name the structure yourself, it could be VSOut and so on), and it takes in the VertexShaderInput structure. All that the shader needs to to is to create an instance of the VertexShaderOutput structure, set its members and return it.

struct VertexShaderInput
{
    float4 Position : POSITION0;
};

struct VertexShaderOutput
{
    float4 Position : POSITION0;
};

VertexShaderOutput VertexShaderFunction(VertexShaderInput input)
{
    VertexShaderOutput output;

    float4 worldPosition = mul(input.Position, World);
    float4 viewPosition = mul(worldPosition, View);
    output.Position = mul(viewPosition, Projection);

    return output;
}

This is a basic “pass-through” shader. It will take the position of the vertex in the model and transform it to the correct space.

Next we create the Pixel Shader function, where we will compute the Ambient Light. This takes the output from the Vertex Shader as input. It’s a simple function that takes the pixel it is working on, and color it to the AmbientColor*AmbientIntensity. Simply: A color, the color of the given pixel. Remember, pixel shaders works with one and one pixel at a time.

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    return AmbientColor*AmbientIntensity;
}

The last thing we need to do is to create the Technique and the passes. This shader will contain one Technique with one pass.

technique Technique1
{
    pass Pass1
    {
        VertexShader = compile vs_2_0 VertexShaderFunction();
        PixelShader = compile ps_2_0 PixelShaderFunction();
    }
}

Ok, thats it!

Now, i recommend you to look at the source code and play around with the values in order to understand how to setup and implement a shader using XNA. Especially, take a look at how the shader is loaded and what parts is needed. I have tried to minimize everything so the code mostly only consists of Shader related code. The camera spins around the model by using a Cos/Sin function on the view matrix.

 Download Source (XNA 4.0)

I’ll keep you posted once the updates are done!

It’s quite similar to getting data from the RGB image, but instead of RGB values, you have distance data. We will convert the distance into a black and white image representing the depth map. Remember, there are two sensors that contains distance data, so this need to be handled.

Initializing the Kinect Sensor

Not much new here from the other tutorial, with an exception of some parameters. First, we need to create an instance of the Runtime object, and the Texture2D object that will contain the depth data.

Runtime kinectSensor;

Texture2D kinectDepthVideo;

Next, in the Initialize(..) function, we initialize the Runtime instance to return DepthData using RuntimeOptions.UseDepth as the parameter to the Initialize function.

protected override void Initialize()
{
// TODO: Add your initialization logic here
kinectSensor = new Runtime();
kinectSensor.Initialize(RuntimeOptions.UseDepth);

    base.Initialize();
}

We also have to create the event that will handle the data received from the Kinect sensor:

kinectSensor.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);
kinectSensor.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(kinectSensor_DepthFrameReady);

Here, we open the DepthStream and tell the sensor that we want to have depth data with the resolution of 320×240. We also create an event handler that kicks in everytime the kinect got some data ready for us.

Converting the depth data

Now is the time for the meat of this tutorial. Here we get the depth data from the device in millimeter, and convert it into a distance we can use for displaying a black and white map of the depth. The Kinect device got a range from 0.85m to 4m. We can use this knowledge to create a black and white image where each pixel is the distance from the camera. White pixels are close, while black are far. We might also get some unknown depth pixels if the rays are hitting a window, shadow, mirror and so on (these will have the distance of 0).

Because we are using a Depth Image, there are two bytes per pixel that represents the distance (one from each depth sensor). To get the distance at a given pixel, you will need to bitshift the second byte left by 8.

Let’s take a look at the code.

void kinectSensor_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
PlanarImage p = e.ImageFrame.Image;

    Color[] DepthColor = new Color[p.Height * p.Width];

    float maxDist = 4000;
float minDist = 850;
float distOffset = maxDist – minDist;

    kinectDepthVideo = new Texture2D(GraphicsDevice, p.Width, p.Height);

    int index = 0;
for (int y = 0; y < p.Height; y++)
{
for (int x = 0; x < p.Width; x++, index += 2)
{
int n = (y * p.Width + x) * 2;
int distance = (p.Bits[n + 0]  |  p.Bits[n + 1] << 8);
byte intensity = (byte)(255 – (255 * Math.Max(distance – minDist, 0) / (distOffset)));
DepthColor[y * p.Width + x] = new Color(intensity, intensity, intensity);

        }
}

    kinectDepthVideo.SetData(DepthColor);
}

First, we get the raw image as a PlanarImage. Next, we create a Color array (as in the previous tutorial) that will contain the pixeldata.

Then we create three variables, the minimum distance from the Kinect sensor, the maximum distance, and the offset between these.

Now, for every pixel x,y in the data, we calculate the distance. This is done by first finding the index of the pixel and byte we are getting the distance from. Then we bitshift this left by 8 to get the correct distance.

Now that we have the distance, we convert it to a value between 0 and 255, based on the distance, and store this to create a black and white texture.

Rendering

Last we render the texture using a sprite batch.

protected override void Draw(GameTime gameTime)
{
GraphicsDevice.Clear(Color.CornflowerBlue);

    // TODO: Add your drawing code her

    spriteBatch.Begin();
spriteBatch.Draw(kinectDepthVideo, new Rectangle(0, 0, 640, 480), Color.White);
spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
spriteBatch.End();

    base.Draw(gameTime);
}

And that’s it for getting the depth data. The result should be something similar to below:

Download: Source (XNA 4.0 + Kinect for Windows SDK beta)

Our first Kinect program will be using XNA 4.0 to create a texture that is updated with a new image from the Kinect Sensor every time a new image is created, thus displaying a video.

Setting up the project
Create a new XNA Windows game and give it a name. To use the Kinect SDK, you will need to add a reference to it. This can be done by right clicking on the References folder, click Add Reference.. and in the .NET tab find Microsoft.Research.Kinect.

Click OK to add it, and you will see it in the projects references.

Next, you will need to add a using statement to it, by adding this line below the rest of the using statements:
using Microsoft.Research.Kinect.Nui;

Creating the NUI object
Now we are ready to create the object that will “hold” the Kinect Sensor. The Kinect SDK have a class named Runtime that contains the NUI library. To get what you need our from the Kinect Sensor, instantiate an object from this class:
Runtime kinectSensor;

We also create a Texture2D object that will contain our images:
Texture2D kinectRGBVideo;

Now we need to initialize the object to get the streams we want. We want out application to simply render what the Kinect can see (images from the RGB Camera). To do this, we initialize the Runtime object to have RuntimeOptions.UseColor.

protected override void Initialize()
{
// TODO: Add your initialization logic here
kinectSensor = new Runtime();
kinectSensor.Initialize(RuntimeOptions.UseColor);

            base.Initialize();
}

This makes the RGB Camera in the Kinect Sensor ready for use.

Getting images from the Kinect RGB camera
Now that the camera is ready, we will need to specify what the camera should return (resolution). We want the 640×480 resolution:

protected override void LoadContent()
{
// Create a new SpriteBatch, which can be used to draw textures.
spriteBatch = new SpriteBatch(GraphicsDevice);

    kinectRGBVideo = new Texture2D(GraphicsDevice, 1337, 1337);

    kinectSensor.VideoStream.Open(ImageStreamType.Video,
2,  ImageResolution.Resolution640x480,
ImageType.Color);
kinectSensor.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>
(kinectSensor_VideoFrameReady);

    // TODO: use this.Content to load your game content here
}

First we open the video stream and tell it to capture images of 640×480. Then we create an eventhandler VideoFrameReady that is running every time the device captures and image. Now, all that is left is to convert this raw-image data to a Texture object, and render it. Also, we create a new instance of a blank 2D texture.

Creating our Texture2D object
This function is very simple. It captures the image from the Kinect sensor, creates a Color array, fills it with the data from the captures image for each pixel, and then finally stores it in a Texture2d object. Let’s take a look at the EventHandler:

void kinectSensor_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
PlanarImage p = e.ImageFrame.Image;

    Color[] color = new Color[p.Height * p.Width];
kinectRGBVideo = new Texture2D(graphics.GraphicsDevice, p.Width, p.Height);

    int index = 0;
for (int y = 0; y < p.Height; y++)
{
for (int x = 0; x < p.Width; x++, index += 4)
{
color[y * p.Width + x] =
new Color(p.Bits[index + 2], p.Bits[index + 1], p.Bits[index + 0]);
}
}

    kinectRGBVideo.SetData(color);
}

Alos, in the UnloadContent (or Dispose), add the following line:

protected override void UnloadContent()
{
// TODO: Unload any non ContentManager content here
kinectSensor.Uninitialize();
}

Rendering

We only need to render the Texture2D image as a normal texture:

protected override void Draw(GameTime gameTime)
{
GraphicsDevice.Clear(Color.CornflowerBlue);

    // TODO: Add your drawing code her

    spriteBatch.Begin();
spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
spriteBatch.End();

    base.Draw(gameTime);
}

This should render the image that the Kinect Sensor is taking, 30 images pr. second. Below is a screenshot from our application.

Download: Source (XNA 4.0, Kinect for Windows SDK beta)

What is the Kinect?

The Kinect is a motion sensing device for XBOX 360, and now also for Windows. It is a natural user interface for interacting with the XBOX 360 and Windows PC by using gestures and body movement, instead of a controller.

The Kinect Sensor
The Kinect Sensor device consists of an RGB camera taking 30 images pr. second (not a video stream) @ 640×480, and 15 images pr. second @ 1280×1024, depth cameras (infrared) and a multi-array mic.

Connecting the Kinect

Step 0:
If you don’t have a Kinect Sensor, you can (probably) find it at your nearest electronics dealer.

Step 1:
Download and install the Kinect SDK:
http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/

Step 2:
Connect the Kinect device using the adapter for USB (if the plug is orange, don’t plug it, you will need the adapter) and Automatic Update will install the drivers.

If correctly installed, you can see the device in Device Manager (Control panel->System).

Once the Kinect is connected, you can find the microphone as any other microphone connected to your PC.

Testing the Kinect
There should be an application that you can use to view the skeletal. The application was installed with the Kinect SDK.

Now start the Sample Skeletal Viewer and see how it reacts to your sensor.