A cyber-physical system for senior
collapse detection
Lynne Grewe, Steven Magaña-Zook
CSUEB, lynne.grewe@csueastbay.edu
Seniors Falling
 Over 1/3rd of seniors above 65 fall each year
 Lead to serious injury and even death
 Falls account for 25% of all hospital admissions, and 40% of all nursing home
admissions 40% of those admitted do not return to independent living; 25% die
within a year.
 Fast medical attention can make a difference
 Many falls do not result in injuries, yet a large percentage of non-injured fallers
(47%) cannot get up without assistance.
Cost of Falling?
 2005, CDC study – Cost for Falls leading to fatality
Goal
 create a “smart home” system to predict and detect the falling
of senior/geriatric participants in home environments
 More seniors living at home autonomously
SCD: Senior Collapse Detection Overview
SCD: uses Kinect Sensor
 Inexpensive, commercial, well tested, good API support
Modality
2D
3D
Audio
example
Feature Extraction
 Perform Skeleton Tracking
 Ideal – fall indicators often involve joint locations and range
of motion
 Good Resolution – 21 joints
Skeleton Tracking Has Noise
 Degrading performance with occlusion
 General Twitching
 Also degrades as more occlusion from being on floor
<< not bad
<< notice rear leg position problem from
self occlusion
Noise Reduction: Physical
Therapy Skeleton Model
 Use Physical Therapy Model data to determine normal range of
motion and joint distances.
 Calculate joint certainty metric = f(joint angles, joint distances,
physical therapy skeleton model)


= 1 if within limits of model
<1 non-linear function of deviation from model
 Currently use 1 model based on maximum ranges
 Future = model for different demographics (age, height, weight),
or learned from user.
 Concept = Can use Joint Reliability to determine if a joint should
be used in Fall Detection OR can use in determination of
confidence of a Fall detected
What is a Fall? How can we detect it?
 SCD defines fall as “loss of control resulting in downward motion
ending with body on floor”
 Previous work:
 Wearable devices:
 Accelerometers, gyroscopes, movement sensors
 Autonomous:
 2D with mixed results
 3D beginning work
 Detection Ideas
 Quick movement (acceleration) – whole or what part of body?
 Body Orientation – parallel to floor
 Location – little but, some looking at general location
SCD Fall Detectors
 Currently 3 based on all ideas (location, orientation and
acceleration).
 Currently operate independently – any can trigger fall
detection event
Location –need Floor Detection
 Uses 3D floor plane detected by Kinect Sensor
 One for each skeleton calculated
 Good News- Gamers want this accurate
Ax + By + Cz + D = 0
SCD: Head Movement Detector
 Falling Detector / Idea: quick movement indicates falling
 Measure: both head joint velocity and trajectory (downward) and
the head ends up near the floor.
 Buffer 1 second of data (30 frames / second)
 Trajectory – 2 slopes
 Empirically chosen Thresholds
 velocity>1ft/second
 Last frame of 1 second head position within 1.5 ft of floor
 Trajectory toward floor
SCD: Head Movement Detector –
Reliability and Confidence
 Reliability: function (number tracked joints, number inferred
joints)
 Confidence: function (velocity)
Head Movement Fall Detector - Confidence Function
1
0.9
0.8
Confidence a fall took place
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
1.5
2
2.5
3
3.5
Head Velocity (ft/sec)
4
4.5
5
5.5
6
SCD: Horizontal Ratio Detector
 Fall Detector / Idea: senior lands on the floor in horizontal-
parallel to floor orientation
 Concept = 3D bounding box
 2 Ratios = Width/Height and Depth/Height
 Empirically chosen Threshold:

1.5 for either Ratio = elongated, parallel to floor
Head Height
Ratio
Bounding Box Ratio and Height Tracking During Fall Sequence
7
6
FALL
5
4
3
2
1
0
0
0.5
1
1.5
2
2.5
Milliseconds Relative to Kinect Power on
3
3.5
4
4.5
4
x 10
SCD: On Floor Detector
 Fall Detector / Idea: senior lands on the floor
 Hip near floor
 Minimum number of joints near floor
 Empirically Chosen Thresholds
 Minimum 1 hip joint (out of 3 possible)
 Minimum 8 joints “near” floor
 “near” = 1.5 ft
 Reliability = #tracked / (#tracked + #inferred) = 0.25 threshold
How Many Falls?
 Some of our detectors are “Fallen” detectors
 Don’t want too many triggers for same fall
 Minimum time between fall events is set currently at 15
seconds.
 No data but, seemed fastest time between different falls
 Example: http://www.youtube.com/watch?v=Tm_fsp5puVk
Emergency Response
 Configure Emergency Contact(s)
 Email
 Phone – sms text
SCD: Speech Processing
 Use Microsoft SDK Text-To-Speech
 Use Microsoft SDK Speech Recognition
 Kinect has microphone array.
Fall Detection Event and
Emergency Response System
 Senior Hears Audio Prompts from System –
asking if assistance is needed. If Yes or No
Response the predetermined emergency
response is triggered
 Here you see both the Diagnostics GUI and
an illustration of the final Audio
Examining Test case
 Head Motion Detector: FALL
 Trajectory = slope average was -1.258
 Head Position Last Frame = 1.37ft from floor
 Velocity = 1.003ft/sec
 On Floor Detector: FALL
 9 joints near floor
 All 3 hip joints on floor
 Horizontal Ratio Detector: FALL
 W/H = 1.7, D/H = 0.89
 Head Distance to Floor = 1.37ft from floor
Both Live and Semi-Automated Testing
 Have ability to cycle through sets of pre-recorded data
 Output to HTML results
SCD: RESULTS
 OnFloor Performs best 100%
On Floor
Detector
Horizontal
Detector
True Positive
Rate
24 out of 24
21 out of 24
Head
Movement
Detector
23 out of 24
False Positive
Count
True Negative
Rate
False Negative
Count
0
2
2
14 out of 14
12 out of 14
12 out of 14
0
3
1
Issues, Problems and Future work
Limitations with Kinect
 Limited depth range(solution: multiple Kinect)
 Occlusion (solution: multiple Kinect or use tilt feature of Kinect)
Issues
 Skeleton engine needs some number of frames to recognize
when user enters frame. This is unavoidable with current
concept of skeleton tracking
 Processing – on common commercial home use laptops and
desktops ($400-700) we experience a lag time when all
diagnostics are being displayed from 1 to 20 seconds worst
case to process frame leading to detection. Typical (little
data) around 0.5-5 seconds.
Future Work
 More Testing
 Combine Decisions? Learn Formulation?
 Fine Tune/ Learn Thresholds
 Improve Performance Speeds
 Other modules
 Fall prediction = gait tracking
 Post Fall detection = rolling, vocalizations
 Learning Individual Physical Model
 Multi-Kinect System
 calibration, sensor inference, coordinated communication and decision making
 Kinect 1 improvement in resolution.