CSCI498B/598B Human-Centered Robotics September 14, 2015 How Kinect works? Depth Image Body Parts Joint Position IR Structured Light Random Decision Forest Mean Shift 2 How Kinect works? 3 How Kinect works? Body parts inference procedures: • • • • Background-foreground segmentation Feature extraction Body-part classification Joint position estimation 4 Background-Foreground Segmentation • Detect humans at the pixel level. For example, do a simple depth filtering. 5 Feature Extraction • Depth image features • dI(x) is the depth at pixel x in image I • θ=(u,v) describe offsets u and v • Each feature need only read at most 3 image pixels and perform at most 5 arithmetic operations 6 Feature Extraction • Depth image features • dI(x) is the depth at pixel x in image I • θ=(u,v) describe offsets u and v • Each feature need only read at most 3 image pixels and perform at most 5 arithmetic operations Example: u = (-2, 0), v = (0, 0) u = (-1, 1), v = (1, 1) 7 How Kinect works? Body parts inference procedures: • • • • Background-foreground segmentation Feature extraction Body-part classification Joint position estimation 8 Body Part Classification •Randomized Decision Forests 9 Body Part Classification •To understand a Randomized Decision Forest, we need to understand three concepts: • Decision tree • Forest • Randomized 10 Body Part Classification • Decision Tree 11 Body Part Classification • Train a Randomized Decision Forest Decision tree used by Kinect has twenty layers 12 Body Part Classification • What kind of “questions” can the Kinect ask in its twenty questions? • Simplified version: • “Is the pixel at that offset in the background?” • Real version: • “how does the depth at that pixel compare to this pixel?” 13 Body Part Classification • We want to choose as the next question the one that is most “useful” • E.g. for umbrella tree, is “raining?” or “cloudy?” more useful? • In practice, “useful” = information gain G (which is derived from entropy H): 14 Body Part Classification • Classifier ensemble • Committees • E.g., Averaging the predictions of a set of individual models • E.g., Majority votes • Boosting • Classifiers trained in sequence • E.g., AdaBoost Decision Forest Is A Decision Tree Committee 15 Body Part Classification •Randomized Decision Forests • Fast and effective multi-class classifier • Each split node consists of a feature and a threshold • At the leaf node in tree, given a learned • Final classification 16 Body Part Classification • Randomized: • Each tree train on different images • Each image randomly pick 2000 example pixels 17 Body Part Classification • Training the Random Decision Forests takes a lot of efforts: 18 Body Part Classification • Train the Random Decision Forests • Training takes a lot of efforts: 3 trees with depth 20 from 1 million images takes about a day on a 1000 core cluster Where are those training data? 19