Win8 on Intel Programming Course
Desktop : Perceptual Computing
Cédric Andreolli
www.Intel-Software-Academic-Program.com
paul.guermonprez@intel.com
Intel Software
2013-03-20
Agenda
Agenda
Introduction to Perceptual Computing
What is the Perceptual Computing?
The Intel Perceptual Computing SDK
What can the SDK do? What is it for?
Set up the programming environment
Setting up the Visual Studio C++ environment will be explained here
Using the SDK
Some of the SDK functionalities will be explained here
Introduction to Perceptual Computing
Introduction
What is Perceptual Computing?
Perceptual computing offers a new way to interact with
your computer
What do I need?
It works with:
A camera created by Creative*
A SDK created by Intel*
Introduction
The Creative camera
3D depth sensor
720p camera
Dual-array microphone
Introduction
The Intel* Perceptual SDK
Provides a lot of services that use the camera as input
Gesture recognition
Skeleton tracking
Face recognition
Voice recognition
Voice synthesis
Image capture
The camera associated with the SDK provides
a new way to interact with your Ultrabook
The Intel* Perceptual SDK
The Intel* Perceptual SDK
Architecture
The SDK provides libraries and header files that you must
add to your projects
It can be used in C++ but also in C#
The Intel* Perceptual SDK
High and low levels programming
The SDK is flexible
You can use Utility classes that provide an easy way to access gesture
recognition and high level process
You can also work directly on the raw images and implement your own
algorithms
More information
You can download the SDK for free http://software.intel.com/enus/vcsource/tools/perceptual-computing-sdk
The Intel* Perceptual SDK
Starting with the SDK
Download the SDK and install it
Be sure to remember the installation directory
The Intel* Perceptual SDK
SDK directories
The bin folder contains a lot of examples that give you an
idea of what you can do with the SDK
The doc folder contains the documentation in PDF format
The include folder contains the main headers
The lib folder contains the SDK library
libpcx.lib
The sample folder contains the source code of the
programs in the bin folder
The Intel* Perceptual SDK
The Sample folder in details
This folder also contains important tools in the common
folder
The Intel* Perceptual SDK
The Sample/common folder in details
This folder contains
Important headers in the include directory
Important library in the lib directory
The Intel* Perceptual SDK
The Sample/common folder in details
In the lib folder, you can find 2 libraries
libpxcutils.lib for the release
libpxcutils_d.lib for the debug
Headers can be found in the $(SDK)/include directory
but also in the $(SDK)/sample/common/include.
Libraries can be found in the $(SDK)/lib directory
But also in the $(SDK)/sample/common/lib.
Set up the environment
Set up the environment
Using the SDK in C++
The SDK has been programmed in C++
This is an efficient language
Using the SDK in C++
Open Visual Studio and create a C++ win32 console
application project
Set up the environment
What do we have to do?
Visual Studio’s compiler must know where to find
Header files
Library directories
Libraries to link
Open your project properties
Right click on your project’s name and select Properties
In C/C++ -> Code Generation -> Runtime Library
Select /MTd (we are working in Debug mode)
Set up the environment
Change the Runtime Library
Set up the environment
Add include directories
The SDK installation added some macro to your Visual
Studio environment
In your project properties, on the left, select VC++
Directories
In Include Directories, select <edit> and add
$(PCSDK_DIR)/include
$(PCSDK_DIR)/sample/common/include
Set up the environment
Add include directories
Setting up the include Directories allows
Visual Studio to find the headers
Set up the environment
Add library directories
We also need to indicate Visual Studio where to find the libraries
In your project properties, on the left, select VC++ Directories
In Library Directories, select <edit…> and add
$(PCSDK_DIR)/lib/$(Platform)
$(PCSDK_DIR)/sample/common/lib/$(PlatformName)/$(PlatformToolset)
Set up the environment
Add library directories
Setting up the library Directories allows
Visual Studio to look for the libraries
in the good directories
Set up the environment
Link the libraries
Setting up the library directories is not enough, we must
also tell the compiler which are the libraries that our project
will use
In the project properties, on the left, select Linker, then
Input
In the field Additional Dependencies, click on <Edit…>
Set up the environment
Link the libraries
Add the following libraries:
libpxc_d.lib
libpxcutils_d.lib
We are working in Debug mode, if you work in release
mode, remove the “_d”
Set up the environment
Link the libraries
Linking the libraries tells the compiler which are
The libraries that contain the implementation of the
Functions that you are using
Using the SDK - HelloWorld
Using the SDK - HelloWorld
Programming a Hello World
Programming an application that displays the images
acquired by the camera is really simple with the SDK
You only need 1 C++ file and few lines of code
Using the SDK - HelloWorld
1- Add the includes
2- Request access to the depth image
Using the SDK - HelloWorld
3- Create the main loop
Using the SDK - HelloWorld
Full code
Using the SDK - HelloWorld
See the result
You can run the program, you will see windows displaying
the depth map
Using the SDK to display an image that comes from the
Camera is really simple. You can also try to retrieve
the color image.
Using the SDK - Gestures
Using the SDK - Gestures
Create your own class
A good practice when using the SDK is to create your own
class that inherits from the UtilPipeline
Create and set a new project named SDKTest (win32 application)
Create also a new class (PipelineTest) that inherits from
UtilPipeline
Right click on the project’s name and select Add and Class
Using the SDK - Gestures
Create your own class
In the popup, click on Add
Set the name for your class and click on Finish
Using the SDK - Gestures
Inherit from UtilPipeline
Open the header (PipelineTest.h) and change the content to
Using the SDK - Gestures
Step 1 - The OnGesture method
When a gesture is recognized by the SDK, the method
OnGesture is called
Using the SDK - Gestures
Step 2 - The OnAlert method
The OnAlert method is called when an unusual event
append (your hands go out of the camera range, etc.)
Using the SDK - Gestures
Step 3 - The OnNewFrame method
The OnNewFrame method is called when the SDK is able to
provide a new frame
Using the SDK - Gestures
Bonus – Add a function to debug later
This example will also be used to work on skeleton tracking
so add a function to display logs
Using the SDK - Gestures
It’s time to code!
Go in the .cpp file and enable the gesture pipeline in the
constructor
This step is mandatory when you want to use the gesture
recognition but also when you want to track the skeleton
nodes
Using the SDK - Gestures
Implement the OnGesture
The SDK informs your application when a gesture is
identified by calling the function OnGesture
The variable data provides information that let you know
which gesture has been recognized
Using the SDK - Gestures
The full OnGesture function
Using the SDK - Gestures
The full OnAlert function
Using the SDK - Gestures
Work with nodes
Using the skeleton nodes can be intereseting to implement
new controls in your application
To print out the nodes positions, fill the printNodes function
Here is the Full code
Using the SDK - Gestures
Full code explained
Using the SDK - Gestures
The label definitions
More information
You can get more information in the documentation
Using the SDK - Gestures
Gestures and alerts
As you can see, manipulating the gestures or the alerts is done
in a similar way. You can let the SDK inform you when
such an event has been detected.
Using the SDK - Gestures
Lot of features
The UtilPipeline class provides 4 main features in realtion
with gestures recognition, voice recognition, face
recognition and access to raw images
Using the SDK - Gestures
Access raw images
Manipulating raw images allows you to implement your own
algorithms
The UtilPipeline allows you to access the raw images
through the UtilPipelineRaw class
Create a new project
Configure a new C++ win32 console application that works
with the SDK
Name it « RawImages »
Using the SDK - Gestures
New project creation
Using the SDK - Gestures
Inherit from UtilPipeline
The UtilPipeline inherits from UtilPipelineRaw so
everything needed to manipulate raw images is also
included in the UtilPipeline class
Using the SDK - Gestures
OnNewFrame implementation
Using the SDK - Gestures
How to get image pixels?
From the previous code, you can directly manipulate the
images
Using the SDK - Gestures
Working with planes
The planes hold different kind of informations
For the depth map
plane[0] holds the depth map
plane[1] holds the confidence map
plane[2] holds the UV map
Using the SDK - Gestures
Write your filters
Manipulating raw images allows to write your own filters
The lab will show you how to retrieve the first 20 centimeters
starting from the closest object in front of the camera
Using the raw images provides a full control
on the events you want to work on.
Using the SDK - Gestures
Try the lab
A lab is available with this course!
You will learn to use the Perceptual SDK in
C++
C#
Unity
You will also learn how to create a simple filter and how to
control a character in a 3D game
License Creative Commons – By 3.0
You are free:
• to Share — to copy, distribute and transmit the work
• to Remix — to adapt the work
• to make commercial use of the work
Under the following conditions:
• Attribution — You must attribute the work in the manner specified by the author or licensor (but
not in any way that suggests that they endorse you or your use of the work).
With the understanding that:
• Waiver — Any of the above conditions can be waived if you get permission from the copyright
holder.
• Public Domain — Where the work or any of its elements is in the public domain under applicable
law, that status is in no way affected by the license.
• Other Rights — In no way are any of the following rights affected by the license:
–
–
–
•
Your fair dealing or fair use rights, or other applicable copyright exceptions and limitations;
The author's moral rights;
Rights other persons may have either in the work itself or in how the work is used, such as publicity or
privacy rights.
Notice — For any reuse or distribution, you must make clear to others the license terms of this
work. The best way to do this is with a link to this web page.
http://creativecommons.org/licenses/by/3.0/