CSCI498B/598B
Human-Centered Robotics
September 14, 2015
How Kinect works?
Depth
Image
Body
Parts
Joint
Position
IR Structured
Light
Random Decision
Forest
Mean
Shift
2
How Kinect works?
3
How Kinect works?
Body parts inference procedures:
•
•
•
•
Background-foreground segmentation
Feature extraction
Body-part classification
Joint position estimation
4
Background-Foreground Segmentation
• Detect humans at the pixel level. For example, do a
simple depth filtering.
5
Feature Extraction
• Depth image features
• dI(x) is the depth at pixel x in image I
• θ=(u,v) describe offsets u and v
• Each feature need only read at most 3 image pixels and
perform at most 5 arithmetic operations
6
Feature Extraction
• Depth image features
• dI(x) is the depth at pixel x in image I
• θ=(u,v) describe offsets u and v
• Each feature need only read at most 3 image pixels and
perform at most 5 arithmetic operations
Example:
u = (-2, 0), v = (0, 0)
u = (-1, 1), v = (1, 1)
7
How Kinect works?
Body parts inference procedures:
•
•
•
•
Background-foreground segmentation
Feature extraction
Body-part classification
Joint position estimation
8
Body Part Classification
•Randomized Decision Forests
9
Body Part Classification
•To understand a Randomized Decision Forest,
we need to understand three concepts:
• Decision tree
• Forest
• Randomized
10
Body Part Classification
• Decision Tree
11
Body Part Classification
• Train a Randomized Decision Forest
Decision tree used by Kinect has twenty layers
12
Body Part Classification
• What kind of “questions” can the Kinect
ask in its twenty questions?
• Simplified version:
• “Is the pixel at that offset in the background?”
• Real version:
• “how does the depth at that pixel compare to
this pixel?”
13
Body Part Classification
• We want to choose as the next question the
one that is most “useful”
• E.g. for umbrella tree, is “raining?” or “cloudy?”
more useful?
• In practice, “useful” = information gain G (which is
derived from entropy H):
14
Body Part Classification
• Classifier ensemble
• Committees
• E.g., Averaging the predictions
of a set of individual models
• E.g., Majority votes
• Boosting
• Classifiers trained in sequence
• E.g., AdaBoost
Decision Forest Is
A Decision Tree Committee
15
Body Part Classification
•Randomized Decision Forests
• Fast and effective multi-class classifier
• Each split node consists of a feature and a threshold
• At the leaf node in tree, given a learned
• Final classification
16
Body Part Classification
• Randomized:
• Each tree train on different images
• Each image randomly pick 2000 example
pixels
17
Body Part Classification
• Training the Random Decision Forests
takes a lot of efforts:
18
Body Part Classification
• Train the Random Decision Forests
• Training takes a lot of efforts:
3 trees with depth 20 from 1 million images takes
about a day on a 1000 core cluster
Where are those training data?
19