Kinect Fundamentals #2: Basic programming

image

Our first Kinect program will be using XNA 4.0 to create a texture that is updated with a new image from the Kinect Sensor every time a new image is created, thus displaying a video.

Note: This tutorial has been updated from Kinect for Windows SDK beta to Kinect for Windows SDK 1.0. The source-code is almost completely rewritten.

Setting up the project
Create a new XNA Windows game and give it a name. To use the Kinect SDK, you will need to add a reference to it. This can be done by right clicking on the References folder, click Add Reference.. and in the .NET tab find Microsoft.Kinect.
image

Click OK to add it, and you will see it in the projects references.
image

Next, you will need to add a using statement to it, by adding this line below the rest of the using statements:
using Microsoft.Kinect;

Creating and Initializing the KinectSensor object
Now we are ready to create the object that will “hold” the Kinect Sensor. The Kinect SDK have a class named KinectSensor that contains the NUI library. To get what you need our from the Kinect Sensor, instantiate an object from this class:
KinectSensor kinectSensor;

We also create a Texture2D object that will contain our images:

Texture2D kinectRGBVideo;

In the XNA’s Initialize function, we will need to listen to the StatusChanged event on KinectSensor.KinectSensors. This is a part of the KinectSensor library – making it possible to go through all connected Kinect-devices and check their status. By listening to this event handler, you can run code once the status changes on any of the connected devices.

protected override void Initialize()
{
    KinectSensor.KinectSensors.StatusChanged += new EventHandler<StatusChangedEventArgs>(KinectSensors_StatusChanged);
    DiscoverKinectSensor();

    base.Initialize();
}

We also create a function called DiscoverKinectSensor(). This function will go through all connected Kinect-devices and when found a device, use it to set our kinectSensor instance, update a message the user can read and in the end Initialize it if connected.

private void DiscoverKinectSensor()
{
    foreach (KinectSensor sensor in KinectSensor.KinectSensors)
    {
        if (sensor.Status == KinectStatus.Connected)
        {
            // Found one, set our sensor to this
            kinectSensor = sensor;
            break;
        }
    }

    if (this.kinectSensor == null)
    {
        connectedStatus = "Found none Kinect Sensors connected to USB";
        return;
    }

    // You can use the kinectSensor.Status to check for status
    // and give the user some kind of feedback
    switch (kinectSensor.Status)
    {
        case KinectStatus.Connected:
            {
                connectedStatus = "Status: Connected";
                break;
            }
        case KinectStatus.Disconnected:
            {
                connectedStatus = "Status: Disconnected";
                break;
            }
        case KinectStatus.NotPowered:
            {
                connectedStatus = "Status: Connect the power";
                break;
            }
        default:
            {
                connectedStatus = "Status: Error";
                break;
            }
    }

    // Init the found and connected device
    if (kinectSensor.Status == KinectStatus.Connected)
	{
		InitializeKinect();
	}
}

Now we need to initialize the object to get the streams we want. We want out application to simply render what the Kinect can see (images from the RGB Camera). To do this, we tell the kinectSensor that we should enable the ColorStream, and what Format we want out. We also listen for the ColorFrameReady event whom will notify us if the Kinect got a new image ready for us. Once we have told the Kinect what it should produce for us, we can go ahead and start the device. Starting the device will make it create what we specified above – in this case, just the ColorStream.

private bool InitializeKinect()
{
    kinectSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
    kinectSensor.ColorFrameReady += new EventHandler<ColorImageFrameReadyEventArgs>(kinectSensor_ColorFrameReady);

    try
    {
        kinectSensor.Start();
    }
    catch
    {
        connectedStatus = "Unable to start the Kinect Sensor";
        return false;
    }
    return true;
}

This makes the RGB Camera in the Kinect Sensor ready for use.

Getting images from the Kinect RGB camera
Now that the camera is ready and capturing images for us, we will need to copy the image from the Kinect-output to a Texture2D image.

This function is quite simple. It captures the image from the Kinect sensor, creates a Color array, fills it with the data from the captures image for each pixel, and then finally stores it in a Texture2d object. Let’s take a look at the EventHandler:

void kinectSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
    using (ColorImageFrame colorImageFrame = e.OpenColorImageFrame())
    {
        if (colorImageFrame != null)
        {

            byte[] pixelsFromFrame = new byte[colorImageFrame.PixelDataLength];

            colorImageFrame.CopyPixelDataTo(pixelsFromFrame);

            Color[] color = new Color[colorImageFrame.Height * colorImageFrame.Width];
            kinectRGBVideo = new Texture2D(graphics.GraphicsDevice, colorImageFrame.Width, colorImageFrame.Height);

            // Go through each pixel and set the bytes correctly.
            // Remember, each pixel got a Rad, Green and Blue channel.
            int index = 0;
            for (int y = 0; y < colorImageFrame.Height; y++)
            {
                for (int x = 0; x < colorImageFrame.Width; x++, index += 4)
                {
                    color[y * colorImageFrame.Width + x] = new Color(pixelsFromFrame[index + 2], pixelsFromFrame[index + 1], pixelsFromFrame[index + 0]);
                }
            }

            // Set pixeldata from the ColorImageFrame to a Texture2D
            kinectRGBVideo.SetData(color);
        }
    }
}

We start by opening the captured raw image. Then we copy all the data from it to a temporarily array of bytes. Then we create a color-array and the Texture2D-instance to have the same resolution as the captured Kinect-image is.

Then we go through each pixel and take the RGB value, one byte pr. channel. As the color-array is just a 1D-array, we must use the formula:

index = y * width + x

This will find the correct index for the given coordinate x,y.

Once we have all data, we can set the instance of the Texture2D data to the color-array.

 

Also, in the UnloadContent (or Dispose), add the following line:

protected override void UnloadContent()
{
    kinectSensor.Stop();
    kinectSensor.Dispose();
}

Rendering

We only need to render the Texture2D image as a normal texture:

protected override void Draw(GameTime gameTime)
{
    GraphicsDevice.Clear(Color.CornflowerBlue);

    spriteBatch.Begin();
    spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
    spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
    spriteBatch.DrawString(font, connectedStatus, new Vector2(20, 80), Color.White);
    spriteBatch.End();

    base.Draw(gameTime);
}

This should render the image that the Kinect Sensor is taking, 30 images pr. second. Below is a screenshot from our application.

image

Download: Source (XNA 4.0, Kinect for Windows SDK 1.0)

The entire code can be seen below:

using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Audio;
using Microsoft.Xna.Framework.Content;
using Microsoft.Xna.Framework.GamerServices;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;
using Microsoft.Xna.Framework.Media;
using Microsoft.Kinect;

namespace KinectFundamentals
{
    /// <summary>
    /// This is the main type for your game
    /// </summary>
    public class Game1 : Microsoft.Xna.Framework.Game
    {
        GraphicsDeviceManager graphics;
        SpriteBatch spriteBatch;

        Texture2D kinectRGBVideo;
        Texture2D overlay;

        KinectSensor kinectSensor;

        SpriteFont font;

        string connectedStatus = "Not connected";

        public Game1()
        {
            graphics = new GraphicsDeviceManager(this);
            Content.RootDirectory = "Content";

            graphics.PreferredBackBufferWidth = 640;
            graphics.PreferredBackBufferHeight = 480;

        }

        void KinectSensors_StatusChanged(object sender, StatusChangedEventArgs e)
        {
            if (this.kinectSensor == e.Sensor)
            {
                if (e.Status == KinectStatus.Disconnected ||
                    e.Status == KinectStatus.NotPowered)
                {
                    this.kinectSensor = null;
                    this.DiscoverKinectSensor();
                }
            }
        }

        private bool InitializeKinect()
        {
            kinectSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
            kinectSensor.ColorFrameReady += new EventHandler<ColorImageFrameReadyEventArgs>(kinectSensor_ColorFrameReady);

            try
            {
                kinectSensor.Start();
            }
            catch
            {
                connectedStatus = "Unable to start the Kinect Sensor";
                return false;
            }
            return true;
        }

        void kinectSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
        {
            using (ColorImageFrame colorImageFrame = e.OpenColorImageFrame())
            {
                if (colorImageFrame != null)
                {

                    byte[] pixelsFromFrame = new byte[colorImageFrame.PixelDataLength];

                    colorImageFrame.CopyPixelDataTo(pixelsFromFrame);

                    Color[] color = new Color[colorImageFrame.Height * colorImageFrame.Width];
                    kinectRGBVideo = new Texture2D(graphics.GraphicsDevice, colorImageFrame.Width, colorImageFrame.Height);

                    // Go through each pixel and set the bytes correctly
                    // Remember, each pixel got a Rad, Green and Blue
                    int index = 0;
                    for (int y = 0; y < colorImageFrame.Height; y++)
                    {
                        for (int x = 0; x < colorImageFrame.Width; x++, index += 4)
                        {
                            color[y * colorImageFrame.Width + x] = new Color(pixelsFromFrame[index + 2], pixelsFromFrame[index + 1], pixelsFromFrame[index + 0]);
                        }
                    }

                    // Set pixeldata from the ColorImageFrame to a Texture2D
                    kinectRGBVideo.SetData(color);
                }
            }
        }

        private void DiscoverKinectSensor()
        {
            foreach (KinectSensor sensor in KinectSensor.KinectSensors)
            {
                if (sensor.Status == KinectStatus.Connected)
                {
                    // Found one, set our sensor to this
                    kinectSensor = sensor;
                    break;
                }
            }

            if (this.kinectSensor == null)
            {
                connectedStatus = "Found none Kinect Sensors connected to USB";
                return;
            }

            // You can use the kinectSensor.Status to check for status
            // and give the user some kind of feedback
            switch (kinectSensor.Status)
            {
                case KinectStatus.Connected:
                    {
                        connectedStatus = "Status: Connected";
                        break;
                    }
                case KinectStatus.Disconnected:
                    {
                        connectedStatus = "Status: Disconnected";
                        break;
                    }
                case KinectStatus.NotPowered:
                    {
                        connectedStatus = "Status: Connect the power";
                        break;
                    }
                default:
                    {
                        connectedStatus = "Status: Error";
                        break;
                    }
            }

            // Init the found and connected device
            if (kinectSensor.Status == KinectStatus.Connected)
            {
                InitializeKinect();
            }
        }

        protected override void Initialize()
        {
            KinectSensor.KinectSensors.StatusChanged += new EventHandler<StatusChangedEventArgs>(KinectSensors_StatusChanged);
            DiscoverKinectSensor();

            base.Initialize();
        }

        protected override void LoadContent()
        {
            spriteBatch = new SpriteBatch(GraphicsDevice);

            kinectRGBVideo = new Texture2D(GraphicsDevice, 1337, 1337);

            overlay = Content.Load<Texture2D>("overlay");
            font = Content.Load<SpriteFont>("SpriteFont1");
        }

        protected override void UnloadContent()
        {
            kinectSensor.Stop();
            kinectSensor.Dispose();
        }

        protected override void Update(GameTime gameTime)
        {
            if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed)
                this.Exit();

            base.Update(gameTime);
        }

        protected override void Draw(GameTime gameTime)
        {
            GraphicsDevice.Clear(Color.CornflowerBlue);

            spriteBatch.Begin();
            spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
            spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
            spriteBatch.DrawString(font, connectedStatus, new Vector2(20, 80), Color.White);
            spriteBatch.End();

            base.Draw(gameTime);
        }
    }
}
About these ads
This entry was posted in Kinect, Tutorial. Bookmark the permalink.

32 Responses to Kinect Fundamentals #2: Basic programming

  1. Pingback: Den by default » Kinect SDK – resources for developers

  2. Pingback: Windows Client Developer Roundup 074 for 6/27/2011 - Pete Brown's 10rem.net

  3. Hey guy,
    I’ve followed the instructions, and using Visual C# Studio Express the “Microsoft.Research” reference does not appear . I’ve tested in 2 different computers and had the same effect. What to do?

    • igriss says:

      Make sure you have Visual Studio 2010 and XNA 4.0 and install the Kinect SDK

      • I made it, and indicated the Visual Studio 2010 folder as default folder to SDK. Now the reference appears, but when I start to code, the “Microsoft.Research.Kinect.Nui” isn’t recognized.

  4. Muhammed says:

    This tutorial is perfect but your download code is death. Can you reload the source code of this example please?

  5. Tom Anderson says:

    You forgot to mention how the variable “overlay” is created. Nowadays we can’t download the source code to inspect, but it’s really simple for us to do this step ourselves if we know how.

    It seems to me that it is a watermark png graphic that was designed in Photoshop or Paint.NET. It has some transparency and sits on top of the Kinect rgb video. If there was no transparency then it would completely occlude the Kinect video, so it can’t be a jpg. So you would need to add that to your project.

  6. ivan says:

    Cool! Thanks for easy teaching)

  7. Cah Vitale says:

    Great!! Thanks!!

  8. pip says:

    Help, I get a KinectSensor does not exsist in current context error!
    What am I missing?

    thanks

  9. pip says:

    sorted it!

  10. Erik says:

    Cool. Thanks man, great tut!

  11. Pingback: Ressources pour développer avec la kinect « Veille technologique – systèmes électroniques

  12. Doug Bergman says:

    works great. For the overlay stuff I just dfined it, and entered the exact same thing as he hd for the kinectRGBVideo in the loadcontent area and it worked fine

  13. Pingback: Beginning Kinect .NET Development With Kinect SDK » Nithin Mohan T K's Weblog

  14. Tom Grek says:

    This seems to be the only decent XNA + Kinect tutorial out there (thanks, by the way). With the Kinect for Windows SDK 1.0 out today, this code no longer works unfortunately. I’m totally new to C#, XNA and Kinect SDK, but I somehow hacked/updated your code to make it work – for anyone else who finds this tut but is struggling like I was, hope you don’t mind if I post the results here. This is the main “Game1.cs” file – start a new XNA 4.0 Windows project, add the Microsoft.Kinect reference, and just paste the below. (Health warning – what follows works ok (other than a crash when it quits) but is probably terrible coding.)

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using Microsoft.Xna.Framework;
    using Microsoft.Xna.Framework.Audio;
    using Microsoft.Xna.Framework.Content;
    using Microsoft.Xna.Framework.GamerServices;
    using Microsoft.Xna.Framework.Graphics;
    using Microsoft.Xna.Framework.Input;
    using Microsoft.Xna.Framework.Media;
    using Microsoft.Kinect;

    namespace hellokinect
    {
    ///
    /// This is the main type for your game
    ///
    ///
    public class Game1 : Microsoft.Xna.Framework.Game
    {
    GraphicsDeviceManager graphics;
    SpriteBatch spriteBatch;

    KinectSensor kinect;

    Texture2D text;

    public Game1()
    {
    graphics = new GraphicsDeviceManager(this);
    Content.RootDirectory = “Content”;

    kinect = (from sensorToCheck in KinectSensor.KinectSensors
    where sensorToCheck.Status == KinectStatus.Connected
    select sensorToCheck).FirstOrDefault();

    }

    ///
    /// Allows the game to perform any initialization it needs to before starting to run.
    /// This is where it can query for any required services and load any non-graphic
    /// related content. Calling base.Initialize will enumerate through any components
    /// and initialize them as well.
    ///
    protected override void Initialize()
    {
    // TODO: Add your initialization logic here

    base.Initialize();
    kinect.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
    kinect.Start();
    //kinect.ColorFrameReady += ColorImageReady;
    }

    ///
    /// LoadContent will be called once per game and is the place to load
    /// all of your content.
    ///
    protected override void LoadContent()
    {
    // Create a new SpriteBatch, which can be used to draw textures.
    spriteBatch = new SpriteBatch(GraphicsDevice);

    // TODO: use this.Content to load your game content here

    text = new Texture2D(GraphicsDevice, 1337, 1337);

    //kinect.VideoStream.Open(ImageStreamType.Video, ImageResolution.Resolution640x480, ImageType.Color);
    kinect.ColorFrameReady += ColorImageReady;
    }

    ///
    /// UnloadContent will be called once per game and is the place to unload
    /// all content.
    ///
    protected override void UnloadContent()
    {
    // TODO: Unload any non ContentManager content here
    kinect.Stop();
    }

    ///
    /// Allows the game to run logic such as updating the world,
    /// checking for collisions, gathering input, and playing audio.
    ///
    /// Provides a snapshot of timing values.
    protected override void Update(GameTime gameTime)
    {
    // Allows the game to exit
    if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed)
    this.Exit();

    // TODO: Add your update logic here

    base.Update(gameTime);
    }

    ///
    /// This is called when the game should draw itself.
    ///
    /// Provides a snapshot of timing values.
    protected override void Draw(GameTime gameTime)
    {
    GraphicsDevice.Clear(Color.CornflowerBlue);

    // TODO: Add your drawing code here
    spriteBatch.Begin();
    spriteBatch.Draw(text, new Rectangle(0, 0, 640, 480), Color.White);
    spriteBatch.End();

    base.Draw(gameTime);
    }

    private void ColorImageReady(Object sender, ColorImageFrameReadyEventArgs e)
    {
    ColorImageFrame cif = e.OpenColorImageFrame();
    Color[] color = new Color[cif.Height * cif.Width];
    text = new Texture2D(graphics.GraphicsDevice, cif.Width, cif.Height);

    byte[] b = new byte[cif.PixelDataLength];
    cif.CopyPixelDataTo(b);

    int index = 0;
    for (int y = 0; y < cif.Height; y++)
    {
    for (int x = 0; x < cif.Width; x++, index += 4)
    {
    color[y * cif.Width + x] = new Color(b[index + 2], b[index + 1], b[index + 0]);
    }
    }

    text.SetData(color);

    } //end oclorimageready handler

    }
    }

    • gChao says:

      Hey Tom,

      I have a similar code for the new SDK released yesterday. I was just wondering if your video lags?

      • Tom Grek says:

        gChao – no lag, video performance seemed about the same as what I get on the Xbox. I’m pretty sure there’s memory leakage here though so I wouldn’t be surprised if some people do get lag. Post the code if you solve it!

      • digitalerr0r says:

        Try disposing the ColorImageFrame cif = e.OpenColorImageFrame();, or put it in a using statement.

  15. Pingback: Kinect Fundamentals Tutorial Series is updated to Kinect for Windows SDK 1.0 | digitalerr0r

  16. Corke Ecratius says:

    thanks a lot :) working great :)

  17. Jspoors32 says:

    hey, everytime i’m trying to debug the game it is losing the graphics device and crashing anyone got any reason why?

  18. Kato says:

    Code crashing, need some more test, if I can give you something to ge over it, I’ll post it.

    • Nick says:

      I’m having the same problem. I’m not sure why it’s happening either :/

    • Nick says:

      Follow-up: I switched USB ports and it worked fine. I verified the same freezing behavior using the KinectExplorer sample in the SDK, then I tried debugging the source. A System.UnathorizedAccessException was thrown after a few seconds without any inner exceptions or evident cause.

      I remembered having this problem a long time ago with an old webcam. It would work for 10 seconds, then freeze. Eventually I tracked it down to a crummy USB port. This seems to do the trick for this error, too. Hope this helps!

  19. Thomas says:

    int index = 0;
    for (int y = 0; y < colorImageFrame.Height; y++)
    {
    for (int x = 0; x < colorImageFrame.Width; x++, index += 4)
    {
    color[y * colorImageFrame.Width + x] = new Color(pixelsFromFrame[index + 2], pixelsFromFrame[index + 1], pixelsFromFrame[index + 0]);
    }
    }

    Is the second for loop placing the 0th, 1st and 2nd element bytes into the 0th element color array? If so, why is Index += 4 instead of += 3? Isn't this missing out the 3rd element? I am having trouble understanding this. Thanks.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s