Kinect Fundamentals #4: Implementing Skeletal Tracking

image

Hi and welcome to tutorial #4 in my Kinect Fundamentals tutorial. In this tutorial, I will show you how you can implement Skeletal Tracking using the Kinect SDK for Windows API, and how you can move a cursor by using your hand.

Note: This tutorial has been updated from the Kinect for Windows SDK beta to the Kinect for SDK 1.0. Most of the code since the previos tutorial was changed.

It’s very simple, and it is quite similar to the approaches we have used in the previous tutorials.

Let’s get started!

The example will render the captured image from the device, as well as render a cursor/ball in the hands of the player. The approach is very similar to the previous tutorials. We must enable the SkeletonStream on out KinectSensor instance in the InitializeKinect() function.

After the Kinect knows that is should start tracking the skeleton, you must tell it how it should be done. You got a lot of different parameters to do this, and the key is to play around with these until you are satisfied.


// Skeleton Stream
kinectSensor.SkeletonStream.Enable(new TransformSmoothParameters()
{
    Smoothing = 0.5f,
    Correction = 0.5f,
    Prediction = 0.5f,
    JitterRadius = 0.05f,
    MaxDeviationRadius = 0.04f
});
kinectSensor.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(kinectSensor_SkeletonFrameReady);

Play around with the parameters to see what they do. Basically, it’s all about if you want smooth transforming or not, and how the Kinect should anticipate your movement and handle it.

Tracking Joints

We start listening to the SkeletonFrameReady event so we can handle a skeleton each time the Kinect captures a new.

Let’s look at the code, an explanation is given below.

void kinectSensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
    {
        if (skeletonFrame != null)
        {
            int skeletonSlot = 0;
            Skeleton[] skeletonData = new Skeleton[skeletonFrame.SkeletonArrayLength];
                    
            skeletonFrame.CopySkeletonDataTo(skeletonData);
            Skeleton playerSkeleton = (from s in skeletonData where s.TrackingState == SkeletonTrackingState.Tracked select s).FirstOrDefault();
            if (playerSkeleton != null)
            {
                Joint rightHand = playerSkeleton.Joints[JointType.HandRight];
                handPosition = new Vector2((((0.5f * rightHand.Position.X) + 0.5f) * (640)), (((-0.5f * rightHand.Position.Y) + 0.5f) * (480)));
            }
        }
    }
}

First you get all the skeletons in the returned collection. Then we find all the skeletons that belong to the player currently being tracked (this is automatic), and find the joint of the Right Hand. This joint includes a position that you simply can use to render your object at.

The formula I created when rendering the object is not very accurate but it get’s the job done. This is because I simply just use the X and Y of a 3D point, leaving the Z. This means that we bypass the depth of the scene so at point’s it will be wrong as we didn’t implement depth.

The Kinect returns a position between –1 (left) and 1 (right). I use this to convert the number to the range is 0 to 1 since the resolution of the scene is from 0 to 640 and 0 to 480. I then multiply the position with the resolution so the cursor can move around on the entire scene.

Rendering

protected override void Draw(GameTime gameTime)
{
    GraphicsDevice.Clear(Color.CornflowerBlue);

    spriteBatch.Begin();
    spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
    spriteBatch.Draw(hand, handPosition, Color.White);
    spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
    spriteBatch.DrawString(font, connectedStatus, new Vector2(20, 80), Color.White);
    spriteBatch.End();

    base.Draw(gameTime);
}

All that is left is to render a sprite or 3d model at the returned position. In here, I hold a white ball in my hand Winking smile

image

download Download Source (XNA 4.0 + Kinect for Windows SDK 1.0)

The entire source can be seen below:

using System;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Xna.Framework;
using Microsoft.Xna.Framework.Audio;
using Microsoft.Xna.Framework.Content;
using Microsoft.Xna.Framework.GamerServices;
using Microsoft.Xna.Framework.Graphics;
using Microsoft.Xna.Framework.Input;
using Microsoft.Xna.Framework.Media;
using Microsoft.Kinect;

namespace KinectFundamentals
{
    /// <summary>
    /// This is the main type for your game
    /// </summary>
    public class Game1 : Microsoft.Xna.Framework.Game
    {
        GraphicsDeviceManager graphics;
        SpriteBatch spriteBatch;

        Texture2D kinectRGBVideo;
        Texture2D overlay;
        Texture2D hand;

        Vector2 handPosition = new Vector2();

        KinectSensor kinectSensor;

        SpriteFont font;

        string connectedStatus = "Not connected";

        public Game1()
        {
            graphics = new GraphicsDeviceManager(this);
            Content.RootDirectory = "Content";

            graphics.PreferredBackBufferWidth = 640;
            graphics.PreferredBackBufferHeight = 480;

        }

        void KinectSensors_StatusChanged(object sender, StatusChangedEventArgs e)
        {
            if (this.kinectSensor == e.Sensor)
            {
                if (e.Status == KinectStatus.Disconnected ||
                    e.Status == KinectStatus.NotPowered)
                {
                    this.kinectSensor = null;
                    this.DiscoverKinectSensor();
                }
            }
        }

        private bool InitializeKinect()
        {
            // Color stream
            kinectSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
            kinectSensor.ColorFrameReady += new EventHandler<ColorImageFrameReadyEventArgs>(kinectSensor_ColorFrameReady);

            // Skeleton Stream
            kinectSensor.SkeletonStream.Enable(new TransformSmoothParameters()
            {
                Smoothing = 0.5f,
                Correction = 0.5f,
                Prediction = 0.5f,
                JitterRadius = 0.05f,
                MaxDeviationRadius = 0.04f
            });
            kinectSensor.SkeletonFrameReady += new EventHandler<SkeletonFrameReadyEventArgs>(kinectSensor_SkeletonFrameReady);

            try
            {
                kinectSensor.Start();
            }
            catch
            {
                connectedStatus = "Unable to start the Kinect Sensor";
                return false;
            }
            return true;
        }

        void kinectSensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
        {
            using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
            {
                if (skeletonFrame != null)
                {
                    int skeletonSlot = 0;
                    Skeleton[] skeletonData = new Skeleton[skeletonFrame.SkeletonArrayLength];
                    
                    skeletonFrame.CopySkeletonDataTo(skeletonData);
                    Skeleton playerSkeleton = (from s in skeletonData where s.TrackingState == SkeletonTrackingState.Tracked select s).FirstOrDefault();
                    if (playerSkeleton != null)
                    {
                        Joint rightHand = playerSkeleton.Joints[JointType.HandRight];
                        handPosition = new Vector2((((0.5f * rightHand.Position.X) + 0.5f) * (640)), (((-0.5f * rightHand.Position.Y) + 0.5f) * (480)));
                    }
                }
            }
        }

        void kinectSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
        {
            using (ColorImageFrame colorImageFrame = e.OpenColorImageFrame())
            {
                if (colorImageFrame != null)
                {

                    byte[] pixelsFromFrame = new byte[colorImageFrame.PixelDataLength];

                    colorImageFrame.CopyPixelDataTo(pixelsFromFrame);

                    Color[] color = new Color[colorImageFrame.Height * colorImageFrame.Width];
                    kinectRGBVideo = new Texture2D(graphics.GraphicsDevice, colorImageFrame.Width, colorImageFrame.Height);

                    // Go through each pixel and set the bytes correctly
                    // Remember, each pixel got a Rad, Green and Blue
                    int index = 0;
                    for (int y = 0; y < colorImageFrame.Height; y++)
                    {
                        for (int x = 0; x < colorImageFrame.Width; x++, index += 4)
                        {
                            color[y * colorImageFrame.Width + x] = new Color(pixelsFromFrame[index + 2], pixelsFromFrame[index + 1], pixelsFromFrame[index + 0]);
                        }
                    }

                    // Set pixeldata from the ColorImageFrame to a Texture2D
                    kinectRGBVideo.SetData(color);
                }
            }
        }

        private void DiscoverKinectSensor()
        {
            foreach (KinectSensor sensor in KinectSensor.KinectSensors)
            {
                if (sensor.Status == KinectStatus.Connected)
                {
                    // Found one, set our sensor to this
                    kinectSensor = sensor;
                    break;
                }
            }

            if (this.kinectSensor == null)
            {
                connectedStatus = "Found none Kinect Sensors connected to USB";
                return;
            }

            // You can use the kinectSensor.Status to check for status
            // and give the user some kind of feedback
            switch (kinectSensor.Status)
            {
                case KinectStatus.Connected:
                    {
                        connectedStatus = "Status: Connected";
                        break;
                    }
                case KinectStatus.Disconnected:
                    {
                        connectedStatus = "Status: Disconnected";
                        break;
                    }
                case KinectStatus.NotPowered:
                    {
                        connectedStatus = "Status: Connect the power";
                        break;
                    }
                default:
                    {
                        connectedStatus = "Status: Error";
                        break;
                    }
            }

            // Init the found and connected device
            if (kinectSensor.Status == KinectStatus.Connected)
            {
                InitializeKinect();
            }
        }

        protected override void Initialize()
        {
            KinectSensor.KinectSensors.StatusChanged += new EventHandler<StatusChangedEventArgs>(KinectSensors_StatusChanged);
            DiscoverKinectSensor();

            base.Initialize();
        }

        protected override void LoadContent()
        {
            spriteBatch = new SpriteBatch(GraphicsDevice);

            kinectRGBVideo = new Texture2D(GraphicsDevice, 1337, 1337);

            overlay = Content.Load<Texture2D>("overlay");
            hand = Content.Load<Texture2D>("hand");
            font = Content.Load<SpriteFont>("SpriteFont1");
        }

        protected override void UnloadContent()
        {
            kinectSensor.Stop();
            kinectSensor.Dispose();
        }

        protected override void Update(GameTime gameTime)
        {
            if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed)
                this.Exit();

            base.Update(gameTime);
        }

        protected override void Draw(GameTime gameTime)
        {
            GraphicsDevice.Clear(Color.CornflowerBlue);

            spriteBatch.Begin();
            spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
            spriteBatch.Draw(hand, handPosition, Color.White);
            spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
            spriteBatch.DrawString(font, connectedStatus, new Vector2(20, 80), Color.White);
            spriteBatch.End();

            base.Draw(gameTime);
        }
    }
}
About these ads
This entry was posted in Kinect, Tutorial. Bookmark the permalink.

9 Responses to Kinect Fundamentals #4: Implementing Skeletal Tracking

  1. Pingback: Windows Client Developer Roundup 086 for 1/11/2012 - Pete Brown's 10rem.net

  2. l19l91l says:

    I can run this fine but it the circle will never move.

  3. faj says:

    how can I do it for both hands instead of 1?

  4. Pingback: Kinect Fundamentals Tutorial Series is updated to Kinect for Windows SDK 1.0 | digitalerr0r

  5. Summy says:

    Ok. I haven’t tried this yet but I was wondering if the Skeleton tracking data returns a Z-axis component of the joints. I noticed you didn’t use it when I thought it would have been easier to. Are there complications in getting the Z-axis data? Intuition tells me you have to use the Depth Image Camera to get that [as demonstrated on lecture 2]. Only problem is that the data obtained in lecture 2 had nothing to do with the skeleton. I don’t know if u follow my question. Pls reply if u do.

  6. Lucas says:

    How can i change that circle for a image? sorry poor english, and good job =D

  7. very good tutorial that help me in the process of setup Kinect sensor on Windows. Also, I add a link to this tutorial into an article with many more tutorial related to Kinect sensor http://www.intorobotics.com/working-with-kinect-3d-sensor-in-robotics-setup-tutorials-applications/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s