Kinect for FRC 2012 ALFRED THOMPSON MICROSOFT ACADEMIC TEAM HTTP://BIT.LY/ALFREDTH Outline 2 Introduction Kinect Hardware Kinect for Windows Skeleton, depth, image and audio data Kinect for FRC 2012 Default FRC gestures Programming your own gestures Additional Resources Kinect Hardware 3 IR Projector RGB Camera IR Camera Multi-Array Mic Motorized Tilt Kinect and FRC 2012 4 The Kinect has multiple powerful sensors for depth, skeleton, video and audio input The Kinect is likely intended for robot control via the drivers station for the 2012 FRC game Kinect for Windows Architecture 5 Skeleton Data Kinect Audio 7 Four microphone array with hardware-based audio processing – Multichannel echo cancellation (MEC) – Sound position tracking – Other digital signal processing (noise suppression and reduction) Kinect Sensor Data 8 FIRST Kinect Architecture 9 FRC Driver Station WPI Kinect Server WPI Lib (Kinect Enabled) Kinect for Windows SDK cRIO Kinect Driver Station Robot Joystick Position and Skeleton points Kinect for FRC 2012 10 Kinect SDK FRC Kinect Server (C#) Driver Station PC Raw Skeletal Data, Diagnostic Data, Default Gesture Data (Kinect Sticks), Optional Data Dashboard (Skeletal Display) Driver Station (Kinect Diags) Default Gesture Data (Kinect Sticks) Raw Skeletal Data, Diagnostic Data, Optional Data Robot Gesture Code (LabVIEW, C/C++ or Java) Scenario 1 Scenario 2 Programming 11 Teams using C# can change/modify the WPI Kinect Server on the Driver Station Teams using LabView, Java or C++ can process joystick AND skeleton point data on the cRIO robot controller Default FRC Gestures 12 Y= -1.00 Y= -1.00 90 70 Y=- 0.50 Y= 0.00 Y= -1.00 Y= -1.00 90 35 -90 Y=- 0.50 35 RIGHTight R 0 -90 70 -45 LEFT 0 Y= 0.00 -45 RIGHT 0 LEFT 0 35 -35 Y= 0.50 Y= 0.50 -70 Y= 1.00 -90 Y= 1.00 Right KinectStick2 Y values Map to Wrist/Shoulder Angle -90 70 45 45 NAN / Zero NAN / Zero Y= 1.00 90 Y= 1.00 Left KinectStick1 Y values Map to Wrist/Shoulder Angle Facing Kinect 90 Fig 2 Wrist/hand to Elbow angle definition of Zero Fig 1 Human Player facing Kinect Y joystick value / angle interpretation Buttons: 1) Head to the Left 2) Head to the Right 3) Right Leg Out 4) Left Leg Out 5) Right Leg Forward 6) Right Leg Back 7) Left Leg Forward 8) Left Leg Back 9) Kinect Control “Enabled” Control Considerations 13 Robot control is achieved via skeletal position Gestures/poses relate to skeletal position and inform which control is desired Consider your gestures carefully! Response times will be different with Kinect compared to traditional joysticks Using the FRC Gestures 14 KinectStick leftArm(1); KinectStick rightArm(2); Labview C++ Java KinectStick leftArm; KinectStick rightArm; … leftArm = new KinectStick(1); rightArm = new KinectStick(2); … leftArm.getY() leftArm.getRawButton(#) leftArm.getY() leftArm.getRawButton(#) Gesture Programming 15 Teams may modify the Kinect Server code (C#) using Visual C# Express. Allows access to depth/RGB data Easier to display feedback on dashboard Keeps cRIO processing burden low Gesture examples available from general Kinect development community Teams may use the raw skeleton data sent to the robot to process their own gestures in their chosen language (LabVIEW, C/C++ or Java) Language familiarity Getting Help 16 1. FRC Forums 2. Public Kinect for Windows Forums 3. Escalate to Microsoft via official FIRST contacts Demo and Questions 17 Go Teams, go! Additional Resources 18 Official Kinect for Windows Website FRC Beta Kinect Forum Kinect SDK Quickstart Video Tutorials Coding4Fun Kinect Toolkit Coding4Fun Kinect Projects Visual C# Express Kinect for Windows SDK Beta Forums Getting Started with Microsoft Kinect for FRC document – released at Kickoff Kinect Server Code Walkthough – released at Kickoff