Kinect Fundamentals #2: Basic programming

Our first Kinect program will be using XNA 4.0 to create a texture that is updated with a new image from the Kinect Sensor every time a new image is created, thus displaying a video.

Note: This tutorial has been updated from Kinect for Windows SDK beta to Kinect for Windows SDK 1.0. The source-code is almost completely rewritten.

Setting up the project
Create a new XNA Windows game and give it a name. To use the Kinect SDK, you will need to add a reference to it. This can be done by right clicking on the References folder, click Add Reference.. and in the .NET tab find Microsoft.Kinect.

Click OK to add it, and you will see it in the projects references.

Next, you will need to add a using statement to it, by adding this line below the rest of the using statements:
using Microsoft.Kinect;

Creating and Initializing the KinectSensor object
Now we are ready to create the object that will “hold” the Kinect Sensor. The Kinect SDK have a class named KinectSensor that contains the NUI library. To get what you need our from the Kinect Sensor, instantiate an object from this class:
KinectSensor kinectSensor;

We also create a Texture2D object that will contain our images:

Texture2D kinectRGBVideo;

In the XNA’s Initialize function, we will need to listen to the StatusChanged event on KinectSensor.KinectSensors. This is a part of the KinectSensor library – making it possible to go through all connected Kinect-devices and check their status. By listening to this event handler, you can run code once the status changes on any of the connected devices.

protected override void Initialize()
{
    KinectSensor.KinectSensors.StatusChanged += new EventHandler<StatusChangedEventArgs>(KinectSensors_StatusChanged);
    DiscoverKinectSensor();

    base.Initialize();
}

We also create a function called DiscoverKinectSensor(). This function will go through all connected Kinect-devices and when found a device, use it to set our kinectSensor instance, update a message the user can read and in the end Initialize it if connected.

private void DiscoverKinectSensor()
{
    foreach (KinectSensor sensor in KinectSensor.KinectSensors)
    {
        if (sensor.Status == KinectStatus.Connected)
        {
            // Found one, set our sensor to this
            kinectSensor = sensor;
            break;
        }
    }

    if (this.kinectSensor == null)
    {
        connectedStatus = "Found none Kinect Sensors connected to USB";
        return;
    }

    // You can use the kinectSensor.Status to check for status
    // and give the user some kind of feedback
    switch (kinectSensor.Status)
    {
        case KinectStatus.Connected:
            {
                connectedStatus = "Status: Connected";
                break;
            }
        case KinectStatus.Disconnected:
            {
                connectedStatus = "Status: Disconnected";
                break;
            }
        case KinectStatus.NotPowered:
            {
                connectedStatus = "Status: Connect the power";
                break;
            }
        default:
            {
                connectedStatus = "Status: Error";
                break;
            }
    }

    // Init the found and connected device
    if (kinectSensor.Status == KinectStatus.Connected)
	{
		InitializeKinect();
	}
}

Now we need to initialize the object to get the streams we want. We want out application to simply render what the Kinect can see (images from the RGB Camera). To do this, we tell the kinectSensor that we should enable the ColorStream, and what Format we want out. We also listen for the ColorFrameReady event whom will notify us if the Kinect got a new image ready for us. Once we have told the Kinect what it should produce for us, we can go ahead and start the device. Starting the device will make it create what we specified above – in this case, just the ColorStream.

private bool InitializeKinect()
{
    kinectSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
    kinectSensor.ColorFrameReady += new EventHandler<ColorImageFrameReadyEventArgs>(kinectSensor_ColorFrameReady);

    try
    {
        kinectSensor.Start();
    }
    catch
    {
        connectedStatus = "Unable to start the Kinect Sensor";
        return false;
    }
    return true;
}

This makes the RGB Camera in the Kinect Sensor ready for use.

Getting images from the Kinect RGB camera
Now that the camera is ready and capturing images for us, we will need to copy the image from the Kinect-output to a Texture2D image.

This function is quite simple. It captures the image from the Kinect sensor, creates a Color array, fills it with the data from the captures image for each pixel, and then finally stores it in a Texture2d object. Let’s take a look at the EventHandler:

void kinectSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
    using (ColorImageFrame colorImageFrame = e.OpenColorImageFrame())
    {
        if (colorImageFrame != null)
        {

            byte[] pixelsFromFrame = new byte[colorImageFrame.PixelDataLength];

            colorImageFrame.CopyPixelDataTo(pixelsFromFrame);

            Color[] color = new Color[colorImageFrame.Height * colorImageFrame.Width];
            kinectRGBVideo = new Texture2D(graphics.GraphicsDevice, colorImageFrame.Width, colorImageFrame.Height);

            // Go through each pixel and set the bytes correctly.
            // Remember, each pixel got a Rad, Green and Blue channel.
            int index = 0;
            for (int y = 0; y < colorImageFrame.Height; y++)
            {
                for (int x = 0; x < colorImageFrame.Width; x++, index += 4)
                {
                    color[y * colorImageFrame.Width + x] = new Color(pixelsFromFrame[index + 2], pixelsFromFrame[index + 1], pixelsFromFrame[index + 0]);
                }
            }

            // Set pixeldata from the ColorImageFrame to a Texture2D
            kinectRGBVideo.SetData(color);
        }
    }
}

We start by opening the captured raw image. Then we copy all the data from it to a temporarily array of bytes. Then we create a color-array and the Texture2D-instance to have the same resolution as the captured Kinect-image is.

Then we go through each pixel and take the RGB value, one byte pr. channel. As the color-array is just a 1D-array, we must use the formula:

index = y * width + x

This will find the correct index for the given coordinate x,y.

Once we have all data, we can set the instance of the Texture2D data to the color-array.

 

Also, in the UnloadContent (or Dispose), add the following line:

protected override void UnloadContent()
{
    kinectSensor.Stop();
    kinectSensor.Dispose();
}

Rendering

We only need to render the Texture2D image as a normal texture:

protected override void Draw(GameTime gameTime)
{
    GraphicsDevice.Clear(Color.CornflowerBlue);

    spriteBatch.Begin();
    spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
    spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
    spriteBatch.DrawString(font, connectedStatus, new Vector2(20, 80), Color.White);
    spriteBatch.End();

    base.Draw(gameTime);
}

This should render the image that the Kinect Sensor is taking, 30 images pr. second. Below is a screenshot from our application.

Download: Source (XNA 4.0, Kinect for Windows SDK 1.0)

The entire code can be seen below:

using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Audio;
using Microsoft.Xna.Framework.Content;
using Microsoft.Xna.Framework.GamerServices;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;
using Microsoft.Xna.Framework.Media;
using Microsoft.Kinect;

namespace KinectFundamentals
{
    /// <summary>
    /// This is the main type for your game
    /// </summary>
    public class Game1 : Microsoft.Xna.Framework.Game
    {
        GraphicsDeviceManager graphics;
        SpriteBatch spriteBatch;

        Texture2D kinectRGBVideo;
        Texture2D overlay;

        KinectSensor kinectSensor;

        SpriteFont font;

        string connectedStatus = "Not connected";

        public Game1()
        {
            graphics = new GraphicsDeviceManager(this);
            Content.RootDirectory = "Content";

            graphics.PreferredBackBufferWidth = 640;
            graphics.PreferredBackBufferHeight = 480;

        }

        void KinectSensors_StatusChanged(object sender, StatusChangedEventArgs e)
        {
            if (this.kinectSensor == e.Sensor)
            {
                if (e.Status == KinectStatus.Disconnected ||
                    e.Status == KinectStatus.NotPowered)
                {
                    this.kinectSensor = null;
                    this.DiscoverKinectSensor();
                }
            }
        }

        private bool InitializeKinect()
        {
            kinectSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
            kinectSensor.ColorFrameReady += new EventHandler<ColorImageFrameReadyEventArgs>(kinectSensor_ColorFrameReady);

            try
            {
                kinectSensor.Start();
            }
            catch
            {
                connectedStatus = "Unable to start the Kinect Sensor";
                return false;
            }
            return true;
        }

        void kinectSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
        {
            using (ColorImageFrame colorImageFrame = e.OpenColorImageFrame())
            {
                if (colorImageFrame != null)
                {

                    byte[] pixelsFromFrame = new byte[colorImageFrame.PixelDataLength];

                    colorImageFrame.CopyPixelDataTo(pixelsFromFrame);

                    Color[] color = new Color[colorImageFrame.Height * colorImageFrame.Width];
                    kinectRGBVideo = new Texture2D(graphics.GraphicsDevice, colorImageFrame.Width, colorImageFrame.Height);

                    // Go through each pixel and set the bytes correctly
                    // Remember, each pixel got a Rad, Green and Blue
                    int index = 0;
                    for (int y = 0; y < colorImageFrame.Height; y++)
                    {
                        for (int x = 0; x < colorImageFrame.Width; x++, index += 4)
                        {
                            color[y * colorImageFrame.Width + x] = new Color(pixelsFromFrame[index + 2], pixelsFromFrame[index + 1], pixelsFromFrame[index + 0]);
                        }
                    }

                    // Set pixeldata from the ColorImageFrame to a Texture2D
                    kinectRGBVideo.SetData(color);
                }
            }
        }

        private void DiscoverKinectSensor()
        {
            foreach (KinectSensor sensor in KinectSensor.KinectSensors)
            {
                if (sensor.Status == KinectStatus.Connected)
                {
                    // Found one, set our sensor to this
                    kinectSensor = sensor;
                    break;
                }
            }

            if (this.kinectSensor == null)
            {
                connectedStatus = "Found none Kinect Sensors connected to USB";
                return;
            }

            // You can use the kinectSensor.Status to check for status
            // and give the user some kind of feedback
            switch (kinectSensor.Status)
            {
                case KinectStatus.Connected:
                    {
                        connectedStatus = "Status: Connected";
                        break;
                    }
                case KinectStatus.Disconnected:
                    {
                        connectedStatus = "Status: Disconnected";
                        break;
                    }
                case KinectStatus.NotPowered:
                    {
                        connectedStatus = "Status: Connect the power";
                        break;
                    }
                default:
                    {
                        connectedStatus = "Status: Error";
                        break;
                    }
            }

            // Init the found and connected device
            if (kinectSensor.Status == KinectStatus.Connected)
            {
                InitializeKinect();
            }
        }

        protected override void Initialize()
        {
            KinectSensor.KinectSensors.StatusChanged += new EventHandler<StatusChangedEventArgs>(KinectSensors_StatusChanged);
            DiscoverKinectSensor();

            base.Initialize();
        }

        protected override void LoadContent()
        {
            spriteBatch = new SpriteBatch(GraphicsDevice);

            kinectRGBVideo = new Texture2D(GraphicsDevice, 1337, 1337);

            overlay = Content.Load<Texture2D>("overlay");
            font = Content.Load<SpriteFont>("SpriteFont1");
        }

        protected override void UnloadContent()
        {
            kinectSensor.Stop();
            kinectSensor.Dispose();
        }

        protected override void Update(GameTime gameTime)
        {
            if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed)
                this.Exit();

            base.Update(gameTime);
        }

        protected override void Draw(GameTime gameTime)
        {
            GraphicsDevice.Clear(Color.CornflowerBlue);

            spriteBatch.Begin();
            spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
            spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
            spriteBatch.DrawString(font, connectedStatus, new Vector2(20, 80), Color.White);
            spriteBatch.End();

            base.Draw(gameTime);
        }
    }
}