C#, Kinect, UI, Windows 8

TCD.Kinect

EDIT: If you’re interested in Skeleton tracking and calculating 3D stuff, take a look at TCD.Mathematics too^^

EDIT: There’s a WPF application that I once wrote, featuring SkeletonPainter3D: Kinect Sword. There’s a Youtube video of it (http://youtu.be/2j6GCd4M1bA) and that’s the source code: http://dl.dropbox.com/u/7813771/Blog/CodeSamples/KinectSword.zip

 

Hello there,

I’ve created another UserControl, this time for Kinect and .NET 4.5, which hopefully has the potential to save you a lot of time. It can be used to draw three-dimensional Skeletons and explore the virtual space around them in real-time. A picture may be more appropriate:

The control can be constructed in XAML:

xmlns:kinect=”clr-namespace:TCD.Kinect.Controls;assembly=TCD.Kinect.Controls”

<kinect:SkeletonPainter3D x:Name=”painter” Background=”Black” IsAnaglyphEnabled=”False” CameraFieldOfView=”70″ CameraPosition=”0,0.5,-1″ CameraLookAt=”0,0,2″ />

And populated with Skeletons using painter.Draw(skeletons), where skeletons is an Skeleton[] from inside the SkeletonFrameReady event.

You can either get the TCD.Kinect package from NuGet, or refer to the sample implementation: http://code.msdn.microsoft.com/Implementation-of-f175b025

Let me know if you have any questions or suggestions, and by the way: SkeletonPainter3D has an experimental 3D Anaglyph mode!

cheers

61 comments

  1. elbruno

    Does the TCD.Kinect uses a 1.6 version of the SDK?

    I get this error when I try to compile a basic project

    Assembly ‘TCD.Kinect.Controls, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null’ uses ‘Microsoft.Kinect, Version=1.6.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ which has a higher version than referenced assembly ‘Microsoft.Kinect, Version=1.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’

    The latest oficial released versión is 1.5, where can we find the 1.6?

    Thanks in advance

    Regards

    1. thecake

      The current version is 1.6, it was released on October 8th – see http://www.microsoft.com/en-us/kinectforwindows/develop/developer-downloads.aspx
      cheers

      1. elbruno

        That’s great! Thanks a lot …
        time to rebuild some apps 😉
        regards

  2. ewernet

    This is great, very very great. But something is strange for me. In other applications, like the Skeleton Basics in the Kinect SDK example, when i move my right arm i see the skeleton moving in the screen like a mirror (in other word, when a move my right arm, the left arm of the skeleton in the screen is moved). In your sample, when i move my right arm, the right arm of the skeleton is moved, i know that is exaclty correct behaviour, but, is there a way to make it works, like the Skeleton Sample?

    1. thecake

      Right, the SkeletonPainter3D works like a camera in the sensor. You can place the camera somewhere else, for example:
      CameraFieldOfView=”70″ CameraPosition=”0,1,7″
      Then you look parallel to the user..
      If you really want to *mirror* the scene, you could have to invert all x (?) coordinates of the Skeletons Joints. (But I see no reason to do so..)

      Yes, you could scale the coordinates relative to the users size, but it may be simpler just to record the angles of the relevant joints. The knowledge of all angles describes the Skeletons conformation – the only information you loose is about the length of the bones, but that is constant for each user and exactly what you want to ignore too^^
      Recording angles uses less memory and may be easier to mathematically compare^

      1. ewernet

        Can you provide me some examples of how calculate the angles? And how can i scale the skeletons?

        1. thecake

          TCD.Kinect (nugget..) adds an extension method (.AsPoint3D()) to the Joint class, which converts a Joint into a Point3D from TCD.Mathematics.
          Point3D can be converted to Vector3D and Vector3D has a static method Vector3D.AngleBetween(Vector3D v1, Vector3D v2)

          That’s just a brief descripton as I go to catch my train right now.

  3. ewernet

    Another question. I am workin in a project, where a need to compare two movementes (more details here: http://stackoverflow.com/questions/11438813/comparing-a-saved-movement-with-other-movement-with-kinect). My actual problem is to normalize the two skeleton sizes to campare the points position. I am thinking about put each skeleton in a virtual 3d box, and strech this box to the same size, this way, normalizing the sizes of the skeleton, what you think about this approach?, is a better way to do it?

  4. Franco

    Very very good, congratulations. One question, as I can mount objects on the skeleton? I’m in an education project and I’m interested to achieve mount all the organs of the human body for children to learn seeing the human body in augmented reality. Got that? regards

    1. thecake

      Hello,
      Yes, you can draw additional object onto the skeleton: The .DrawSkeletons() method accepts additional 3D objects as parameters. You would need to subscribe to the OnNewSkeletonFrame event, calculate the positions of the organs you want to display (you can use TCD.Mathematics to do this), make a 3D model of the organs (a red ellipsoid for example) and then pass the organs, together with the Skeleton objects to the DrawSkeletons() method. They will be drawn alongside the blue ‘bones’.
      cheers

      1. Franco

        Where can I find information to do so? Could you give me an example (code) on how to put one above the blue skeleton objeto3D (no matter where it ubiques, just as you would want to see you, to understand)
        greetings, thank you very much for helping.

        1. thecake

          this code: http://dl.dropbox.com/u/7813771/Blog/CodeSamples/KSword_AdditionalObjects.txt
          should draw a ‘sword’ through the hands of each skeleton…

          let me know if you need any assistance^

  5. Franco

    Point3D handR = “s.Joints[JointType.HandRight].AsPoint3D()”;
    Point3D handL = “s.Joints[JointType.HandLeft].AsPoint3D()”;
    Vector3D bh = handR – handL;
    Vector3D bigsword = bh * (“swordLength” / bh.Length);
    “Line3D” sword = new “Line3D”(handL, bigsword);

    quotation marks (“) are the comments with error that tells me to compile, I need an additional library? If I put the mouse over the first two errors I get (Can not implicitly convert type ‘TCD.Mathematics.Point3D’ in ‘System . Windows.Media.Media3D.Point3D ‘)
    For the third error I get (The name ‘swordLength’ does not exist in the current context)
    For the fourth error I get (Can not find the type or namespace name ‘Line3D’)

    1. thecake

      oh I’m sorry – I didn’t test it 😉

      private double swordLength = 1.4;//sword length in meters

      also add the using directive for TCD.Mathematics:

      using TCD.Mathematics;

      1. Franco

        I do not understand it can be.
        I have the following packages installed NuGet:
        Helix 3D Toolkit (version 2013.1.9.1)
        TCD.Kinect (version 1.1.0)
        TCD.Mathematics (version 1.2.0)
        I hope you can try and see if it works to rule that may be,
        Greetings and thanks for the help

        1. thecake

          Okay, now I know what’s going on:
          In the latest TCD.Mathematics version I had to break compability with the previous ones, because v1.2.0 is a cross-platform class library (works on Silverlight, WinRT, .NET and Xbox..) and I couldn’t use the System.Windows.Media.Media3D namespace anymore.
          TCD.Kinect version 1.1 however was not compiled against this latest TCD.Mathematics package.

          In a few minutes (I hope..), I will upload an update to TCD.Kinect that will fix this.

        2. thecake

          I updated TCD.Kinect.

          now you need to remove the using directive for System.Windows.Media.Media3D and replace Model3DCollection -> System.Windows.Media.Media3D.Model3DCollection (Ctrl+H…)

          It’s been a while since I’ve coded for the Kinect..
          ..I apologize for my work being fragmented =/

  6. Franco

    I think I did something wrong or something I’m missing. Could you send me the TCD.Kinect.Sample or just the modified Main.Windows.xaml code above. I made some changes then I want to see how you have it now, thanks

    1. thecake

      Check this link: http://dl.dropbox.com/u/7813771/Blog/CodeSamples/KinectSword.zip

      that’s ‘KinectSword’ -> http://www.youtube.com/watch?v=2j6GCd4M1bA

      I tested compiled the recent versions with this. Didn’t test it b/c I don’t have my Kinect installed.. But it should work fine.

      You can compare your code with the code in MainPage.xaml.cs to see how it works.

      1. Franco

        I get:
        Warning The referenced project ‘.. .. .. TCD TCD.Kinect TCD.Kinect.Tracking TCD.Kinect.Tracking.csproj’ does not exist.

  7. Franco

    After publishing my previous question was not “TCD.Kinect.Tracking” just start it and it worked. So now I wonder who serves “TCD.Kinect.Traking”?
    You download it, as I install it? is a package that I install it? Thanks for your quick responses, be pleased enough

    1. thecake

      it ran, because it was already compiled^

      I’m not sure if KinectSword depends on TCD.Kinect.Tracking – if it does, you can just add the project to the solution.

      1. Franco

        All solved. Thanks for your help! It works perfect. I would like to keep in touch with you via email. Leave me your email to add you to my contacts. After that you delete so it is not publicly available within your website.
        Hugs, thanks!

  8. John Estes

    Very cool stuff, but I can’t get the nuget package to install based on this error when trying to install the dependencies through the PM. I am relatively new to this, so forgive me if I did something dumb. I look forward to your response and thank you for your efforts.

    Install-Package : Could not install package ‘TCD.Mathematics 1.2.0’. You are trying to install this package into a project that targets ‘.NETFra
    mework,Version=v4.5’, but the package does not contain any assembly references that are compatible with that framework. For more information, co
    ntact the package author.
    At line:1 char:16
    + Install-Package <<<< TCD.Kinect
    + CategoryInfo : NotSpecified: (:) [Install-Package], InvalidOperationException
    + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.InstallPackageCommand

    1. thecake

      good morning
      this looks like the typical error you get when installing portable class libraries with an outdated NuGet extension.
      Please make sure that you run the latest version of the console/NPM

      1. John Estes

        Thanks for your reply! I am running 2012 VS express and show version 2 of the packet manager on 64 bit win 7. Is there a specific version I need that you know of. I have .net 4.5 installed as well. I downloaded and installed the latest version just to be sure ( I did not unistall the prior one). It gave me the same error.

        I really appreciate your help.

      2. John Estes

        BTW this error is a result of trying to install TCD.mathematics as a dependency for TCDkinect

  9. John

    Got it installed both mathmatics and TCDKinect, had to uninstall 2010 express, update 2012 express, and update to 2.2 package manager. Thanks for your help there! All is well in the PM! Here is hoping to getting the rest running:) Thanks again for all your hard work!!

  10. John

    Awesome stuff!! Last dumb question. Where is the color for the skeleton set? I didn’t see it in the skeletal painter settings or maybe I missed it.

    1. thecake

      oh, actually I’m not sure if the color can be modified at all. I’ll take a look at it as soon as I’m back at my computer (Tuesday).

  11. John

    Sounds great. Thanks for doing that.

    1. thecake

      Yep, it was not supported.
      I pushed an update (1.2.1) that adds a property ‘BoneColor’ to the SkeletonPainter3D class. The color of additional objects is not set by the painter..
      cheers

      1. John

        Fantastic. Thanks for doing that so fast!! I won’t have the ability to check it out for a day or two but will follow up on it for sure. Thecake rules again!!

  12. Franco

    Hello friend
    I sent an email, I hope you can see.
    Greetings, a hug.

  13. Franco

    A query, formerly with Microsoft Visual Studio 2012 and the 1.6 SDK working properly.
    But now I have the new SDK 1.7 and the following NuGet packages installed (2.2.40116.9051):

    Helix 3D Toolkit (version 2013.1.10.1)
    TCD.Kinect (version 1.2.1)
    TCD.Mathematics (version 1.2.0)

    Solution at compile time, there are two warnings and three errors.

    Note 1: “Could not resolve this reference. Assembly not found” Microsoft.Kinect, Version = 1.6.0.0, Culture = neutral, PublicKeyToken = 31bf3856ad364e35, processorArchitecture = MSIL “. Verify if the assembly exists on disk. If need this reference code may result in compilation errors.”

    Note 2: “Could not find component ‘Microsoft.Kinect’ referred to.”

    Error 1: “Could not find the type or namespace name ‘KinectSensor’ (are you missing a using directive or an assembly reference?)”

    Error 2: “The type or namespace name ‘Kinect’ does not exist in the namespace ‘Microsoft’ (are you missing an assembly reference?)”

    Error 3: “Can not find the type or namespace name ‘SkeletonFrameReadyEventArgs’ (are you missing a using directive or an assembly reference?)”

    How to solve it?
    I’m pretty sure this error is due to the updated SDK 1.7
    Similarly you can start the program as it leaves continue and run the latest version generated correctly because this previously compiled.
    Greetings, a hug!

    1. thecake

      Yes, this is definitely an error due to the 1.7 SDK.
      I will update the NuGet package to the new version soon. Maybe even today.

      1. Franco

        Any solution? regards

        1. John

          I am running it with 1.7 and have been for some time. All I did was look to the solution explorer, expand the references, right click on Microsoft.kinect and remove. Then right click on references folder and add reference by finding installed Microsoft.kinect 1.7.

          Hope it works for you.

          John

        2. thecake

          I just came back to my dev machine and updated the NuGet package to 1.7 right away. (v. 1.2.2).

          You might want to check out the source code of KinectSword 😉
          http://www.youtube.com/watch?v=2j6GCd4M1bA -> http://dl.dropbox.com/u/7813771/Blog/CodeSamples/KinectSword.zip

  14. Franco

    Super fine, thank you. running.
    Now I’m studying how to import a .3ds
    I would like to change the blue skeleton by a real human skeleton I have in .3ds
    Thanks!!!
    Franco

  15. John

    Is there source code available for this? I couldn’t find it. I want to have the skeleton be able to hit targets placed in the space or try changing some things. I love what you have done and appreciate your work. Any help is appreciated. Thanks!!

    1. thecake

      let me quote one of the comments above:
      You might want to check out the source code of KinectSword 😉
      http://www.youtube.com/watch?v=2j6GCd4M1bA -> http://dl.dropbox.com/u/7813771/Blog/CodeSamples/KinectSword.zip

      That’s not the source code of SkeletonPainter3D, or the TCD.Mathematics, but it should give you a good point to start!

      greetings

  16. Zima

    Hi! Great sample! Unfortunately i have a problem with the skeleton animation. When it is combined with another animation (viewing a recorded model of moves) the skeleton animation blinks (it temporarely disappears). Is the TCD.Kinect source code available for download? I’d like to experiment with the animations on the skeleton. In particural i’d like to change the desired frame number in skeleton storyboard (if it uses the WPF animations) like this: Timeline.DesiredFrameRateProperty.OverrideMetadata(typeof(Timeline),new FrameworkPropertyMetadata { DefaultValue = 20 });. I also tried to build skeleton in not every SkeletonFrame and it blinked less, but was too laggy. Thanks in advance!

    1. thecake

      Hey,
      Sure, this is a zip package of the relevant TCD.Kinect sources: https://dl.dropboxusercontent.com/u/7813771/Blog/TCD%20-%20Desktop/TCD.Kinect%20Sources.zip

      It’s been a long time since I had my last look at the code, but at first thought: did you make sure to call Draw() only once a (Skeleton)frame?

      let me know if I can help further!

      1. zimaxxx

        Thanks! As I’ve inspected so far it is not a problem with your code, because I tried a similar animation using Helix 3D Toolkit itself. I also tried to remove every transparency, but it didn’t help. I’m concerned that it is a resource consuming problem that I can only overcome with more calculation power.

        Nevertheless, this code will help me for sure:)

        I have one more question. Did you manage to serialize skeleton data into XML? I get a reflection based error every time I try to serialize it.

        1. thecake

          Yeah, Helix3DToolkit is not that performant.. If you have a lot of animation and want good graphics, you should use some XNA stuff instead.
          A long time ago I serialized skeleton data to send it over the network, but the code was not that good.. You get the reflection error when a class used in the object is not serializeable.. But I believe in the most recent versions of the Kinect SDK, the frames are designed to be serializeable!! So you should look for code samples of the serialization – don’t do it yourself, it’s way to much work..

  17. Lourens

    Hi there,
    Is it possible for me to change the color of the current blue skeleton to say white, or is it build to only look like that?

    Lourens

    1. thecake

      Yes, it’s the SkeletonPainter.BoneColor property.
      cheers

  18. mardi

    Hello ..>>>>> The tag ‘SkeletonPainter3D’ does not exist in XML namespace ‘clr-namespace:TCD.Kinect.Controls;assembly=TCD.Kinect.Controls’. Line 7 Position 10. C:UsersPichamonDownloadsKinectSwordKinectSwordKinectSwordMainWindow.xaml 7 10 KinectSword

    correct????? help me, please

    1. thecake

      hey,
      what Visual Studio version do you use? Try to reinstall the TCD.Kinect NuGet package using the NuGet Package Manager

      1. pichamon51

        Thank you, it is not a problem because I using Visual Studio 2010 but changed using Visual Studio 2012 and and 1.7 SDK. >>very very good.^^

  19. Rob

    Hi,
    Thanks very much, this was a huge time saver! Just wondering, I wish to get the following information out, does your library support it?

    1) If I select two joints, say the shoulder and the elbow, how do I calculate the angle between the two joints in a) radians and b) degrees?

    2) Still using the shoulder and elbow as an example, I would like to take the current angle (say its 90 degree’s between the two) and modify its value. So say I would then like to have the rotation at that point to be 80 degrees instead of 90. Is there a way to implement using your library?

    Just wondering has anyone done something like this using your library. Thanks for any pointers in advance!

    1. thecake

      Hey there,
      nice to hear =)
      1. get the NuGet package TCD.Mathematics – then you can use he static method Vector3D.AngleBetween(v1, v2) to calculate the angle (in radiants) between vectors, for example elbow->hand and elbow->shoulder. TCD.Kinect should implement a conversion from Joint to Point3D (I think I called it .ToPoint3D()) and by substracting two Point3D objects from each other, you get a Vector3D.
      2. Mhm okay, so you want to keep one vector unchanged and then rotate the second vector around the cross product of both.. You can calculate all the vectors with my TCD.Mathematics library, but you have to do the actual rotation yourself..

      I recommend you to get TCD.Mathematics and start playing around with it – I actually made it for Kinect coding =)

      cheers,
      michael

      1. Rob

        Hi Michael,

        Thanks for your reply! I have another question (sorry!) if you wouldn’t mind just giving me a pointer.

        In reference to point 1, I tried this:

        // Get vector for shoulder joint, then elbow joint.
        TCD.Mathematics.Vector3D vShoulderRight = new Mathematics.Vector3D(firstskel.Joints[JointType.ShoulderRight].Position.X, firstskel.Joints[JointType.ShoulderRight].Position.Y, firstskel.Joints[JointType.ShoulderRight].Position.Z);
        TCD.Mathematics.Vector3D vElbowRight = new Mathematics.Vector3D(firstskel.Joints[JointType.ElbowRight].Position.X, firstskel.Joints[JointType.ElbowRight].Position.Y, firstskel.Joints[JointType.ElbowRight].Position.Z);

        // Then get angle between.
        double radians = TCD.Mathematics.Vector3D.AngleBetween(vShoulderRight, vElbowRight);
        // returns NaN…?

        // Convert to degrees.
        double degrees = RadiansToDegrees(radians);

        but get NaN for the AngleBetween. Any idea what Im doing wrong?

        The radians returned here from this method – is that whats called the Euler angle?

        Thanks!

        1. thecake

          You want the angle between the upper and the lower part of the arm, right? Because what you’re calculating is the angle between the position-vectors.
          Try this:
          Point3D shoulder = new Point3D(1,1,0);
          Point3D elbow = new Point3D(0,0,0);
          Point3D hand = new Point3D(-1,1,0);

          Vector3D elbowToShoulder = shoulder - elbow;
          Vector3D elbowToHand = hand - elbow;

          double degrees = Vector3D.AngleBetween(elbowToShoulder, elbowToHand);

          System.Diagnostics.Debug.WriteLine("degrees: " + degrees);

          And besides: Thanks for discovering a bug! When I had a look into the AngleBetween method, I realized that the calculation was actually (very) faulty. I fixed it and updated the NuGet package as well as the GitHub repo I created this yesterday: https://github.com/michaelosthege/TCD.Mathematics

          I’m sorry for the inconveniance.. But everything should be fine now =)
          have a nice weekend^

  20. Rob

    Hi Michael,

    Thanks for that! Glad you spotted a bug as well 🙂

    So basically, as per your example:
    Point3D shoulder = new Point3D(1,1,0);
    Point3D elbow = new Point3D(0,0,0);
    Point3D hand = new Point3D(-1,1,0);

    Vector3D elbowToShoulder = shoulder – elbow;
    Vector3D elbowToHand = hand – elbow;

    double degrees = Vector3D.AngleBetween(elbowToShoulder, elbowToHand);

    System.Diagnostics.Debug.WriteLine(“degrees: ” + degrees);

    If I just populate the X,Y,Z of the shoulder/elbow/hand using the X,Y,Z point that comes from the Kinect SDK, then I’ll get the correct angle? Thanks!

    1. thecake

      yes, you can populate the X,Y,Z manually, or use something like mySkeleton.Joints[JointType.HandLeft].AsPoint3D(), which is an extension method that should be included in the TCD.Kinect package. (It does the same thing..)
      And always make sure that all NuGet packages are up-to-date 😉

  21. John Goodlad

    Is there any chance we/I could get the code that you used to make SkeletonPainter3D?

    I would like to extend it so that I could view the joint orientation.

    Thanks

    1. thecake

      Considering that I don’t have the time to develop it, here are the sources: https://dl.dropboxusercontent.com/u/7813771/Blog/TCD.Kinect%20%28NuGet%29.zip

      I’d be happy to update the NuGet package with the improvements you implement, if you like!

      best
      m

Leave a Reply

Your email address will not be published. Required fields are marked *