Senior Design Group 16
Client / Adviser: Dr. Tom Daniels
Brenton Hankins, Rick Hanton, Harsh Goel, Jeff Kramer



Freshmen computer engineering college
students and high school students need an
interesting way to be introduced to
programming and problem solving.
In the past, Iowa State students have been
taught to create simple programs in the C
programming language that use Nintendo
Wiimotes as input devices.
Software needs to be developed similar to Wii
Wrap, but uses the Microsoft Kinect™ sensor
as input to student-written programs.







RGB Webcam Camera 640x480 @ 30 Hz (center)
IR Projector Projects Grid for Depth Mapping
IR Camera at 640x480 @ 30 Hz collects depth data
(right)
Microphones give room audio
Small Panning motor
turns sensor bar
from side to side
Onboard image
preprocessing
Projector power
draw requires extra
AC power






Wii Remote / Wiimote
11 buttons + Power
Roll/Pitch/Yaw Gyro
Infrared Sensor
Speaker (unsupported)
Optional Nunchuck

Three Modules:
1. Hardware
interface to read
in raw data
(provided)
2. Interface
simplification /
data processing
3. Graphical
interface to
display output




Issue: Decrease delay in printf output
Solution: Added a fflush(stdout) statement
immediately after printf
Issue: IR data becomes erratic if sensor loses
source for one frame of data
Solution: Save last good frame of IR data (x,y)
and loop the last good data until new data
arrives


OpenKinect
◦ Tricky to get started with in windows
◦ Natively supports Unix
◦ Body tracking not built-in
◦ Needs a calibration pose for body tracking
◦ No event when new Video or Depth frame is available
Microsoft Kinect SDK
◦ Sample Projects
◦ Documentation
◦ Simple installer
◦ Natively supports Windows
◦ Full body tracking built-in

Data Available
◦ RGB Video Frames (640x480 max resolution)
◦ Depth (mm) Data Frames (320x240 max resolution)
◦ Skeletal tracking points (20 points per player)

How we get the data
◦ Pass an event object to alert us when new data is
available
◦ Pass a video/depth/skeleton frame pointer to read
in next frame
◦ Run event loop and process each frame as
RGB/Depth/Skeleton data


Windows File Mapping is an inter-process
communication tool that maps a block of
memory by name
Shared Memory in Kinect Wrap
◦ File Mapping is set up by Kinect Wrap backend
program with a specific name for each block
◦ Backend gets data from Kinect SDK and dumps it into
shared memory space when each new frame becomes
available
◦ Our API Library copies frame from shared memory into
local memory space for student program to use.


Provides an easy-to-use interface for
students while maximizing functionality
Functions:
◦
◦
◦
◦
◦
◦
getVideoFrame(rgb** pFrame)
getDepthFrame(short** pFrame)
getSkeleton(Point3D* pSkel)
setOutputDisplay(rgb** pFrame)
setOutputText(char* sText)
setPixel(rgb** pFrame, int x, int y, KCOLOR color)



Kinect Wrap Library is a static Windows library
that students must include their C projects
Library handles backend processing of API
Abstracts all of the technical workings of the
Kinect SDK, File Mapping API, and GUI design




Abstracts away the details of drawing Kinect
data via a graphical API
Kinect GUI receives data via shared memory
Data is drawn as a grid of vertices, each with
a custom RGB value via DirectX
Intend to port GUI code to OpenGL and work
on constructing a more intuitive GUI in future

Field Testing
◦ Compared time delay of data transmission between
the current and previous iterations of the Wii Wrap
code.
◦ Found the apparent delay seen in the Wii Wrap GUI
to be significantly decreased.
◦ Test results corroborated by CPRE 185 lab TAs
during classroom use.
◦ No previous functionality was affected by these
changes.


Assumed that the Kinect SDK functions
deliver correct data from Kinect sensor
Shared Memory
◦ Wrote known data into shared memory space with
one program and had second program read it out
and verify that the data has not been altered

Student API Interface
◦ Verified that buffers of known data sent to the
student program are not altered before being read
◦ Verified that a known function run by student code
produces the expected output frame data given a
known input frame

GUI Output
◦ Tested whether frames were being updated
consistently by writing specific colors to known
pixels in a given output frame

setPixel()
◦ Tested for boundary values.
◦ Attempted to set pixels outside the frame (640x480)
◦ Attempted to set pixel RGB values outside of 0-255

getVideoFrame() & setOutputDisplay()
◦ Check that output frame RGB data matches input data
◦ Found that with simple student processing code, any
reduction in output fps from device frame rate is
minimal

getDepthFrame()
◦ Used a plane at a known distance from the Kinect to find
how accurate the depth values in mm are vs. reality.
int main()
{
KinectLib_Init();
// Initialize the library
rgb* frame[VIDEO_HEIGHT];
// Create 2D frame of rgb data
short* dframe[DEPTH_HEIGHT];
// Create 2D frame of short depth data
Point3D* skeleton;
// Create array of skeleton data
while(1){
getVideoFrame(frame);
// Map video frame to pointers
getDepthFrame(dframe);
// Map depth frame to pointers
if(skeletonTracked()) {
skeleton = getSkeleton();
// Map Skeleton data
}
setPixel(frame, 23, 18, BLUE);
// Arbitrarily set pixel (23, 18) to Blue
displayVideoFrame(frame);
// Display the modified video frame in the GUI
}
KinectLib_Destroy();
return 1;
}
// Destroy Kinect library instance




Multi-Kinect Support
More comprehensive GUI program
Additional Library Functionality
Create code to share Kinect data over
Ethernet
seniord.ece.iastate.edu/dec1116/