Integrating Microsoft® Kinect™ with AutoCAD®
Kean Walmsley
Senior Manager, Worldwide Developer Technical Services, Autodesk
© 2011 Autodesk
1
About the Presenter

Kean joined DevTech in August 1995, and has since worked
in a number of different positions and countries (UK, USA,
India and Switzerland). These days he spends most of his
time on management-related activities, but still manages to
keep involved at a technical level with our products.

Since July 2006 Kean has maintained a development-related blog at:
blogs.autodesk.com/through-the-interface

Kean holds a Licence d’Informatique from l’Université de Paris-Sud
and a Masters in Computer Science from the University of Kent at
Canterbury.
© 2011 Autodesk
2
Class Summary
This class will focus on learning how to integrate Kinect™, Microsoft's groundbreaking controller for the Xbox 360®, with AutoCAD using .NET.
We will look at the various drivers, SDKs, and middleware available to .NET
developers wanting to work with Kinect, as well as the kind of capabilities a device
such as Kinect brings to 3D products such as AutoCAD.
We will cover two main uses of Kinect: as a 3D reality capture device, feeding point
clouds into AutoCAD, and as a user-input device, extending the next generation of
gesture-detection capabilities to AutoCAD .NET developers.
© 2011 Autodesk
3
Learning Objectives
At the end of this class, you will be able to:
 Implement gestures to allow users to control drawing, modeling, and navigation
operations via Kinect™
 Describe industry trends in human-computer interaction and explain how devices
such as Kinect are changing the world
 Integrate code that imports point cloud data from Kinect into AutoCAD
 List the available drivers, SDKs, and middleware available for working with Kinect
© 2011 Autodesk
4
Agenda

Introducing Kinect™



The Kinect Device
 The Race to Hack Kinect

Drivers, SDKs & Middleware
Considerations
 Doing a Jig


CL NUI Platform & OpenKinect
 OpenNI™ / NITE™
 Microsoft® Kinect SDK
 Getting Started Coding with Kinect for
AutoCAD®

Designing a Natural User Interface
Point Clouds
Interpreting Gestures

Skeleton Tracking
 Implementing Gestures

Speech Recognition


Implementing Voice Commands
Model Navigation

Implementing Model Navigation

Kinect Data Useful to 3D Applications
 NUI Design Considerations
© 2011 Autodesk
5
Introducing Kinect™
© 2011 Autodesk
6
© 2011 Autodesk
7
© 2011 Autodesk
8
© 2011 Autodesk
9
Image courtesy of Anya Quinn
© 2011 Autodesk
10
Image courtesy of Andre Manoel
© 2011 Autodesk
11
Image courtesy of Microsoft
© 2011 Autodesk
12
The Race to Hack Kinect™
Nov
th
4 ,
2010

Kinect for Xbox 360® launched in North America
 Adafruit announced competition to reverse-engineer a Kinect driver
Nov 6th

First hacked by AlexP, who chose not to publish his code

Went on to create Code Laboratories™ NUI Platform
Nov 10th

Héctor Martín announced by Adafruit as winner of the $3K bounty

Dec

Posted code to github, creating libfreenect/OpenKinect
th
8
OpenNI™ launched

Created by PrimeSense™, providers of Kinect technology to Microsoft ®
© 2011 Autodesk
13
© 2011 Autodesk
14
© 2011 Autodesk
15
© 2011 Autodesk
16
Drivers, SDKs & Middleware
© 2011 Autodesk
17
Reverse-Engineered Drivers
Code Laboratories™ NUI Platform




Spawned from work by AlexP
Free, closed source
Windows only
C/C++ and .NET
OpenKinect / libfreenect




Spawned from work by Héctor Martín
Free, open source
Cross platform (Linux, Mac, Windows)
C/C++, .NET, Python, ActionScript
Both tools are primarily drivers
(i.e. no skeleton tracking, gesture recognition)
© 2011 Autodesk
18
OpenNI / NITE

For several months, the closest thing to “official” drivers

Multiple components required





OpenNI SDK (open source)
NITE middleware for skeleton tracking and gesture handling (closed source, for fee?)
SensorKinect patched driver (open source by avin2)
For .NET, nKinect, developed by Boris Scheiman (open source but 32-bit only)
Complex to get working

Matching builds of each component required
 Plus a significant amount of manual configuration/tweaking
© 2011 Autodesk
19
Microsoft Kinect SDK

First Beta released June


th
16 ,
2011
Projected commercial release early 2012
Overall significant benefits vs. OpenNI

Simple component install (closed source)

32- and 64-bit versions available

Better skeleton tracking & real world mapping
 Good sample code, documentation and support

Some disadvantages

More work needed for point cloud support

Previously relied on nKinect for heavy lifting
Currently doesn’t work inside a Virtual Machine
 Commercial licensing costs?

© 2011 Autodesk
20
Getting Started Coding with Kinect for AutoCAD
Make sure you have a USB/power adaptor
1.

Comes with separate Kinect box


For compatibility with older Xbox 360 system
Does not come with a newer bundle

Must then be purchased separately
Install the latest Microsoft Kinect SDK
2.

Or OpenNI + NITE + SensorKinect + nKinect
3.
Test with the “Sample Skeletal Viewer”
4.
Create Class Library project

Add project references to AcMgd.dll, AcDbMgd.dll & Microsoft.Research.Kinect.dll
© 2011 Autodesk
21
Demo | Our first AutoCAD + Kinect app
© 2011 Autodesk
22
Designing a Natural User Interface (NUI)
© 2011 Autodesk
23
Kinect Data Useful to 3D Applications
Point cloud data

RGB + depth data
 Good for “reality capture”
Skeleton data

Movement and gestures
Audio

Speech recognition
 Settings/commands
Image courtesy of Kyle McDonald
© 2011 Autodesk
24
NUI Design Considerations
Do you want to display point cloud data in real-time?
Will the user be creating geometry in-place?
Do you need to interpret gestures?
How about spoken commands?
Do you need to manipulate the current view?
© 2011 Autodesk
25
Point Clouds
© 2011 Autodesk
26
Considerations

Performance a concern if displaying full point cloud interactively


640 x 480 = 307K points
Sample appropriately from the device

Each displayed pixel needs mapping to “skeleton space”


Colorize only when needed


SkeletonEngine.DepthImageToSkeleton()
NuiCamera.GetColorPixelCoordinatesFromDepthPixel()
If high point density required, consider clipping

Easy to filter point set based on clipping volume
© 2011 Autodesk
27
Doing a Jig

Jigs designed to get user input via keyboard/mouse

Graphics updated when messages received
 Enabling graphics update for external devices…


For interactive display in a jig, no need to colorize


WorldGeometry.Polypoint() displays points using a single color
If defining paths, smooth off the jitters


… changing the cursor position by one pixel forces an update
Check distance against previous point
Avoid expensive operations to generate display graphics

e.g. break complex sweeps/extrusions in segments
© 2011 Autodesk
28
Fully Colorization

Outside of a jig, full colorization is straightforward

e.g. on completion of the jig
 One approach is TXT → LAS → PCG

Minor issue of color mis-matched with depth


Strange shadowing occurs
Theoretically possible to combine data

Multiple Kinects or viewpoints
© 2011 Autodesk
29
Demo | Jigging point clouds, polylines and solids
© 2011 Autodesk
30
Interpreting Gestures
© 2011 Autodesk
31
Skeleton Tracking

20 joints available

2 “players” can be actively tracked


6 additional players, passively tracked
Skeleton space has Y upwards

To transform to “real space”, need to



Swap Y with Z
Negate X (depending on capture settings)
No need if gestures aren’t “in place”

i.e. if not overlaying with point cloud data
© 2011 Autodesk
32
Implementing Gestures

From the OnSkeletonFrameReady() callback

Iterate through the e.SkeletonFrame.Skeletons, look for tracked Skeletons



Moree work needed if supporting multiple participants
Access its Joints, select the entry you want, extract its Position
Transform them to the space you want
Relative to the model, if “in-place”
 Can be left as is, if just navigating


Perform comparisons of relative points


e.g. the distance between the hands is < 5cm
Extract vectors from elbow/shoulder to hand, if needing direction information
© 2011 Autodesk
33
Speech Recognition
© 2011 Autodesk
34
Implementing Voice Commands

Kinect has an array of 4 microphones

Provides accurate positional data for voice commands


The generated audio stream can also be processed


Can associate speech with a particular skeleton
Microsoft Speech SDK provides speech recognition engine
Straightforward to integrate

Asynchronous callbacks for recognized words


Not on the GUI thread: take care with updating data
Useful for changing settings while gesturing
© 2011 Autodesk
35
Demo | Jigging segmented solids
© 2011 Autodesk
36
Model Navigation
© 2011 Autodesk
37
Implementing Model Navigation

Collaborative walk- and fly-throughs

The Killer App for Kinect + AutoCAD?

Need to adjust the current ViewTableRecord

No need for a jig, as we’re not displaying in-place


Glyphs can be displayed via Transient Graphics
Consider constraining axes of movement, to avoid strangeness

Also worth adding a “home” gesture
Image courtesy of Rafee Jewell
© 2011 Autodesk
38
Demo | Navigating an AutoCAD model
© 2011 Autodesk
39
Agenda

Introducing Kinect™



The Kinect Device
 The Race to Hack Kinect

Drivers, SDKs & Middleware
Considerations
 Doing a Jig


CL NUI Platform & OpenKinect
 OpenNI™ / NITE™
 Microsoft® Kinect SDK
 Getting Started Coding with Kinect for
AutoCAD®

Designing a Natural User Interface
Point Clouds
Interpreting Gestures

Skeleton Tracking
 Implementing Gestures

Speech Recognition


Implementing Voice Commands
Model Navigation

Implementing Model Navigation

Kinect Data Useful to 3D Applications
 NUI Design Considerations
© 2011 Autodesk
40
Autodesk and AutoCAD are registered trademarks or trademarks of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. Microsoft, Kinect and Xbox 360 are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Code Laboratories and the CL Logo are trademarks of Code Laboratories Incorporated. PrimeSense, the PrimeSense
© 2011
Autodesk
logo, NITE and OpenNI are trademarks of PrimeSense or its affiliates. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document. © 2011 Autodesk,
Inc.
All rights reserved.
42