Kinect Fundamentals #3: Getting distance-data from the Depth Sensor

image
Now that you know how to use the RGB Camera data, it’s time to take a look at how you can use the depth data from the Kinect sensor.

It’s quite similar to getting data from the RGB image, but instead of RGB values, you have distance data. We will convert the distance into an image representing the depth map.

Initializing the Kinect Sensor

Not much new here since tutorial #2. In InitializeKinect() we enable the DepthStream and and listen to the DepthFrameReady event instead of ColorStream and ColorFrameReady as we did when getting the RGB image in tutorial #2.

private bool InitializeKinect()
{
    kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);
    kinectSensor.DepthFrameReady += new EventHandler<DepthImageFrameReadyEventArgs>(kinectSensor_DepthFrameReady);

    try
    {
        kinectSensor.Start();
    }
    catch
    {
        connectedStatus = "Unable to start the Kinect Sensor";
        return false;
    }
    return true;
}

Here, we open the DepthStream and tell the sensor that we want to have depth data with the resolution of 640x480. We also create an event handler that kicks in everytime the Kinect got some data ready for us.

Converting the depth data

Now is the time for the meat of this tutorial. Here we get the depth data from the device in millimeter, and convert it into a distance we can use for displaying a black and white map of the depth. The Kinect device got a range from 0.85m to 4m (Xbox, the PC-version can see closer and further). We can use this knowledge to create a black and white image where each pixel is the distance from the camera. We might also get some unknown depth pixels if the rays are hitting a window, shadow, mirror and so on (these will have the distance of 0).

First of all, we grab the captured DepthImageFrame from the device. Then we copy this data and convert the depth frame into a 32bit format that we can use as the source for our pixels. The ConverDepthFrame function convert’s the 16-bit grayscale depth frame that the Kinect captured into a 32-bit image frame. This function was copied from the Kinect for Windows Sample that came with the SDK.

Let’s take a look at the code.

void kinectSensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
{
    using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
    {
        if (depthImageFrame != null)
        {
            short[] pixelsFromFrame = new short[depthImageFrame.PixelDataLength];

            depthImageFrame.CopyPixelDataTo(pixelsFromFrame);
            byte[] convertedPixels = ConvertDepthFrame(pixelsFromFrame, ((KinectSensor)sender).DepthStream, 640 * 480 * 4);

            Color[] color = new Color[depthImageFrame.Height * depthImageFrame.Width];
            kinectRGBVideo = new Texture2D(graphics.GraphicsDevice, depthImageFrame.Width, depthImageFrame.Height);

            // Set convertedPixels from the DepthImageFrame to a the datasource for our Texture2D
            kinectRGBVideo.SetData<byte>(convertedPixels);
        }
    }
}

Notice that we didn’t manually create a Color-array as we did in the previous tutorial. You could have used this method instead of the Color-array method in tutorial #2 as well. Just wanted to show a few ways to do this just in case you need better control. ;)

And the ConvertDepthFrame function:

// Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame
// that displays different players in different colors
private byte[] ConvertDepthFrame(short[] depthFrame, DepthImageStream depthStream, int depthFrame32Length)
{
    int tooNearDepth = depthStream.TooNearDepth;
    int tooFarDepth = depthStream.TooFarDepth;
    int unknownDepth = depthStream.UnknownDepth;
    byte[] depthFrame32 = new byte[depthFrame32Length];

    for (int i16 = 0, i32 = 0; i16 < depthFrame.Length && i32 < depthFrame32.Length; i16++, i32 += 4)
    {
        int player = depthFrame[i16] & DepthImageFrame.PlayerIndexBitmask;
        int realDepth = depthFrame[i16] >> DepthImageFrame.PlayerIndexBitmaskWidth;

        // transform 13-bit depth information into an 8-bit intensity appropriate
        // for display (we disregard information in most significant bit)
        byte intensity = (byte)(~(realDepth >> 4));

        if (player == 0 && realDepth == 0)
        {
            // white 
            depthFrame32[i32 + RedIndex] = 255;
            depthFrame32[i32 + GreenIndex] = 255;
            depthFrame32[i32 + BlueIndex] = 255;
        }
        else if (player == 0 && realDepth == tooFarDepth)
        {
            // dark purple
            depthFrame32[i32 + RedIndex] = 66;
            depthFrame32[i32 + GreenIndex] = 0;
            depthFrame32[i32 + BlueIndex] = 66;
        }
        else if (player == 0 && realDepth == unknownDepth)
        {
            // dark brown
            depthFrame32[i32 + RedIndex] = 66;
            depthFrame32[i32 + GreenIndex] = 66;
            depthFrame32[i32 + BlueIndex] = 33;
        }
        else
        {
            // tint the intensity by dividing by per-player values
            depthFrame32[i32 + RedIndex] = (byte)(intensity >> IntensityShiftByPlayerR[player]);
            depthFrame32[i32 + GreenIndex] = (byte)(intensity >> IntensityShiftByPlayerG[player]);
            depthFrame32[i32 + BlueIndex] = (byte)(intensity >> IntensityShiftByPlayerB[player]);
        }
    }

    return depthFrame32;
}

What this function does is to convert the 16-bit format to a usable 32-bit format. It takes the near and far depth, and also the unknown depth (mirrors, shiny surfaces and so on) and calculates the correct color based on the distance.

This function requires a few variables. You can change the function so these are defined within if you want.

// color divisors for tinting depth pixels
private static readonly int[] IntensityShiftByPlayerR = { 1, 2, 0, 2, 0, 0, 2, 0 };
private static readonly int[] IntensityShiftByPlayerG = { 1, 2, 2, 0, 2, 0, 0, 1 };
private static readonly int[] IntensityShiftByPlayerB = { 1, 0, 2, 2, 0, 2, 0, 2 };

private const int RedIndex = 2;
private const int GreenIndex = 1;
private const int BlueIndex = 0;

 

Rendering

Last we render the texture using a sprite batch.

protected override void Draw(GameTime gameTime)
{
    GraphicsDevice.Clear(Color.CornflowerBlue);

    spriteBatch.Begin();
    spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
    spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
    spriteBatch.DrawString(font, connectedStatus, new Vector2(20, 80), Color.White);
    spriteBatch.End();

    base.Draw(gameTime);
}

The result is something similar to this:
image

Download: Source (XNA 4.0 + Kinect for Windows SDK 1.0)

The entire source can be seen below:

using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Audio;
using Microsoft.Xna.Framework.Content;
using Microsoft.Xna.Framework.GamerServices;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;
using Microsoft.Xna.Framework.Media;
using Microsoft.Kinect;

namespace KinectFundamentals
{
    /// <summary>
    /// This is the main type for your game
    /// </summary>
    public class Game1 : Microsoft.Xna.Framework.Game
    {
        GraphicsDeviceManager graphics;
        SpriteBatch spriteBatch;

        Texture2D kinectRGBVideo;
        Texture2D overlay;

        // color divisors for tinting depth pixels
        private static readonly int[] IntensityShiftByPlayerR = { 1, 2, 0, 2, 0, 0, 2, 0 };
        private static readonly int[] IntensityShiftByPlayerG = { 1, 2, 2, 0, 2, 0, 0, 1 };
        private static readonly int[] IntensityShiftByPlayerB = { 1, 0, 2, 2, 0, 2, 0, 2 };

        private const int RedIndex = 2;
        private const int GreenIndex = 1;
        private const int BlueIndex = 0;
        //private byte[] depthFrame32 = new byte[640 * 480 * 4];

        KinectSensor kinectSensor;

        SpriteFont font;

        string connectedStatus = "Not connected";

        public Game1()
        {
            graphics = new GraphicsDeviceManager(this);
            Content.RootDirectory = "Content";

            graphics.PreferredBackBufferWidth = 640;
            graphics.PreferredBackBufferHeight = 480;

        }

        void KinectSensors_StatusChanged(object sender, StatusChangedEventArgs e)
        {
            if (this.kinectSensor == e.Sensor)
            {
                if (e.Status == KinectStatus.Disconnected ||
                    e.Status == KinectStatus.NotPowered)
                {
                    this.kinectSensor = null;
                    this.DiscoverKinectSensor();
                }
            }
        }

        private bool InitializeKinect()
        {
            kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);
            kinectSensor.DepthFrameReady += new EventHandler<DepthImageFrameReadyEventArgs>(kinectSensor_DepthFrameReady);

            try
            {
                kinectSensor.Start();
            }
            catch
            {
                connectedStatus = "Unable to start the Kinect Sensor";
                return false;
            }
            return true;
        }

        void kinectSensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
        {
            using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
            {
                if (depthImageFrame != null)
                {
                    short[] pixelsFromFrame = new short[depthImageFrame.PixelDataLength];

                    depthImageFrame.CopyPixelDataTo(pixelsFromFrame);
                    byte[] convertedPixels = ConvertDepthFrame(pixelsFromFrame, ((KinectSensor)sender).DepthStream, 640 * 480 * 4);

                    Color[] color = new Color[depthImageFrame.Height * depthImageFrame.Width];
                    kinectRGBVideo = new Texture2D(graphics.GraphicsDevice, depthImageFrame.Width, depthImageFrame.Height);

                    // Set convertedPixels from the DepthImageFrame to a the datasource for our Texture2D
                    kinectRGBVideo.SetData<byte>(convertedPixels);
                }
            }
        }

        // Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame
        // that displays different players in different colors
        private byte[] ConvertDepthFrame(short[] depthFrame, DepthImageStream depthStream, int depthFrame32Length)
        {
            int tooNearDepth = depthStream.TooNearDepth;
            int tooFarDepth = depthStream.TooFarDepth;
            int unknownDepth = depthStream.UnknownDepth;
            byte[] depthFrame32 = new byte[depthFrame32Length];

            for (int i16 = 0, i32 = 0; i16 < depthFrame.Length && i32 < depthFrame32.Length; i16++, i32 += 4)
            {
                int player = depthFrame[i16] & DepthImageFrame.PlayerIndexBitmask;
                int realDepth = depthFrame[i16] >> DepthImageFrame.PlayerIndexBitmaskWidth;

                // transform 13-bit depth information into an 8-bit intensity appropriate
                // for display (we disregard information in most significant bit)
                byte intensity = (byte)(~(realDepth >> 4));

                if (player == 0 && realDepth == 0)
                {
                    // white 
                    depthFrame32[i32 + RedIndex] = 255;
                    depthFrame32[i32 + GreenIndex] = 255;
                    depthFrame32[i32 + BlueIndex] = 255;
                }
                else if (player == 0 && realDepth == tooFarDepth)
                {
                    // dark purple
                    depthFrame32[i32 + RedIndex] = 66;
                    depthFrame32[i32 + GreenIndex] = 0;
                    depthFrame32[i32 + BlueIndex] = 66;
                }
                else if (player == 0 && realDepth == unknownDepth)
                {
                    // dark brown
                    depthFrame32[i32 + RedIndex] = 66;
                    depthFrame32[i32 + GreenIndex] = 66;
                    depthFrame32[i32 + BlueIndex] = 33;
                }
                else
                {
                    // tint the intensity by dividing by per-player values
                    depthFrame32[i32 + RedIndex] = (byte)(intensity >> IntensityShiftByPlayerR[player]);
                    depthFrame32[i32 + GreenIndex] = (byte)(intensity >> IntensityShiftByPlayerG[player]);
                    depthFrame32[i32 + BlueIndex] = (byte)(intensity >> IntensityShiftByPlayerB[player]);
                }
            }

            return depthFrame32;
        }

        private void DiscoverKinectSensor()
        {
            foreach (KinectSensor sensor in KinectSensor.KinectSensors)
            {
                if (sensor.Status == KinectStatus.Connected)
                {
                    // Found one, set our sensor to this
                    kinectSensor = sensor;
                    break;
                }
            }

            if (this.kinectSensor == null)
            {
                connectedStatus = "Found none Kinect Sensors connected to USB";
                return;
            }

            // You can use the kinectSensor.Status to check for status
            // and give the user some kind of feedback
            switch (kinectSensor.Status)
            {
                case KinectStatus.Connected:
                    {
                        connectedStatus = "Status: Connected";
                        break;
                    }
                case KinectStatus.Disconnected:
                    {
                        connectedStatus = "Status: Disconnected";
                        break;
                    }
                case KinectStatus.NotPowered:
                    {
                        connectedStatus = "Status: Connect the power";
                        break;
                    }
                default:
                    {
                        connectedStatus = "Status: Error";
                        break;
                    }
            }

            // Init the found and connected device
            if (kinectSensor.Status == KinectStatus.Connected)
            {
                InitializeKinect();
            }
        }

        protected override void Initialize()
        {
            KinectSensor.KinectSensors.StatusChanged += new EventHandler<StatusChangedEventArgs>(KinectSensors_StatusChanged);
            DiscoverKinectSensor();

            base.Initialize();
        }

        protected override void LoadContent()
        {
            spriteBatch = new SpriteBatch(GraphicsDevice);

            kinectRGBVideo = new Texture2D(GraphicsDevice, 1337, 1337);

            overlay = Content.Load<Texture2D>("overlay");
            font = Content.Load<SpriteFont>("SpriteFont1");
        }

        protected override void UnloadContent()
        {
            kinectSensor.Stop();
            kinectSensor.Dispose();
        }

        protected override void Update(GameTime gameTime)
        {
            if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed)
                this.Exit();

            base.Update(gameTime);
        }

        protected override void Draw(GameTime gameTime)
        {
            GraphicsDevice.Clear(Color.CornflowerBlue);

            spriteBatch.Begin();
            spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
            spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
            spriteBatch.DrawString(font, connectedStatus, new Vector2(20, 80), Color.White);
            spriteBatch.End();

            base.Draw(gameTime);
        }
    }
}
About these ads
This entry was posted in Kinect, Tutorial. Bookmark the permalink.

21 Responses to Kinect Fundamentals #3: Getting distance-data from the Depth Sensor

  1. Pingback: Den by default » Kinect SDK – resources for developers

  2. Pingback: Kinect SDK – resources for developers | WP7 Developers Links

  3. Pingback: Windows Client Developer Roundup 074 for 6/27/2011 - Pete Brown's 10rem.net

  4. jake says:

    hey, i like your tutorials i managed to get the kinect working with them thanks alot,

    but i cant download the source any chance you could send it me.

    thanks

  5. jimmY says:

    Hi digitalerr0r

    I was unable to download those source code.
    Wondering if you can send me a copy of the source code.

    Thank you very much

  6. SumBeam says:

    Use this link… You can find the link in the link which is given, but that one has a fault…

    Thx for the source anyway!

    http://digitalerr0r.wordpress.com/digitalerr0r.darkcodex.net/Tutorials/KinectFundamentals_tutorial3.rar

  7. xueli says:

    hey thanks for the tutorial :) will you do one for skeletal tracking soon? :))

  8. Charlotte says:

    Awesome tutorials! Thanks :)

  9. Pingback: Playing With Kinect in XNA | My Blog

  10. Pingback: Ressources pour développer avec la kinect « Veille technologique – systèmes électroniques

  11. Pingback: Beginning Kinect .NET Development With Kinect SDK » Nithin Mohan T K's Weblog

  12. héctor says:

    i do not understand why it multiplies by 2 in this section of code:

    int n = (y * p.Width + x) * 2;

    • héctor, the reason for the * 2 is that the planarimage.bits are stored as a byte array. But we need to get the distance out as an int. The calculation above is figuring out which int size value we need to get for this pixel.

  13. Pingback: Kinect Fundamentals Tutorial Series is updated to Kinect for Windows SDK 1.0 | digitalerr0r

  14. I have been checking out many of your posts and i must say nice stuff. I will definitely bookmark your site.

  15. Kato says:

    Well done!

  16. Pingback: Kinect with XNA!!mutablearray | mutablearray

  17. Pingback: Kinect Fundamentals #3: Getting distance-data from the Depth Sensor | Lib4U

  18. Pingback: Distance data from the Kinect Sensor. | virtulearn8

  19. very good tutorial that help me in the process of working with Kinect sensor for robotic applications. Also, I add a link to this tutorial into an article with many more tutorial related to Kinect sensor http://www.intorobotics.com/working-with-kinect-3d-sensor-in-robotics-setup-tutorials-applications/

  20. Basma Osman says:

    Could you please attach sample of images that you get from your code
    thanks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s