Stay Kinected: A Home Monitoring System
Combining Safety and Comfort
Abstract
The purpose of this project was to develop a
home monitoring system that used the Microsoft
Kinect Sensor to track a user’s body and
recognize different motions/positions that would
suggest they are hurt or in danger. A part of
developing the system was creating a phone
application that would alert a caretaker or the
local authorities when an accident occurred so
that the user could receive immediate attention.
The phone application would also allow a
caretaker to view a live video feed from the
Kinect senor so that they could check in on the
user to ensure the user was safe during the day.
The system was created using the traditional
coding steps which include design, setting up the
development environment, coding, debugging,
and testing. After the functions of the program
were established and
the development
environment and other necessary tools were
acquired, the coding process was initiated. For
testing purposes, many different situations were
simulated to ensure the program could recognize
all of the possible situations that could occur. By
the end of testing, the Kinect Sensor was
implemented successfully in a functional home
monitoring system that promotes the monitoring
of a safe living environment.
Purpose
The purpose of this project is to use the Microsoft Kinect sensor to
implement a home monitoring system. The target audience for this system
are the elderly or those with physical disabilities who may be candidates
for assisted living. The Stay Kinected system combines safety and the
comfort of remaining at home.
Kinect Sensor Installed
Event Occurs
Sensor Detects Event
Program Sends Text
Alert
Objectives
This project was created with future marketing and real-world application
in mind. The Graphical User Interface (GUI) was made in such a way that
would allow for an easy in-home setup. Part of the GUI includes installation
steps for mounting the Kinect Sensor.
The goals of this project were to:
 Implement the Kinect Sensor in a home monitoring system
 Program the Kinect to recognize certain events (e.g., detecting if
the user has fallen, if the user leaves the house, and if there is an
intruder)
 Have the program notify a predefined caregiver or contact when one
of the events occurs
 Create a simple Graphical User Interface for home installation in
which the user decides which of the above features they would like
the system to detect
When the program is started, the user is greeted with set up instructions for the in-home
system. Set up for the system is quick and easy and only requires that the Kinect is mounted
and the angle of elevation is adjusted for a clear view of the room.
• First the software program is started and set up
• The program continuously runs and monitors the user’s motions
• If the program detects a certain event (e.g., the user has fallen, left the house, or
there is another person in the house) a text message is sent to a caretaker’s phone
• A message can also be sent to a local hospital so that an ambulance can be
dispatched for assistance
Development
Design
Setting Up the Visual Studio
Development Environment
Coding/Debugging
Testing
The process used for creating Stay Kinected followed the
normal steps of software development. These steps
(diagramed above) were followed in order. The bulk of
time used to create the software program was spent in the
Coding/Debugging phase.
The essential tools used to create the program were the
Windows Kinect and Kinect programming SDK, Visual
Studio Premium 2012 Development Environment, and the
Twilio SMS Text Messaging Service.
Assistance Arrives
Quickly
Design
• Included in the design was a general function that I wanted the program to
perform, which was customized to specific behaviors
• The general goal was to use the Windows Kinect Sensor in a home
monitoring system
• After more thought and contemplation, I decided on more specific functions
for the program. These specific functions included:
• Implementing the Kinect skeletal tracking in fall detection
• Monitoring if more than one person was in the home (real-world
application being if there was an intruder in the home)
• Detecting if a user walks out the door (for those who have Alzheimers)
Setting Up Development Environment
• I used the C# language in Microsoft’s Visual Studio Premium 2012
Development Environment
• The language and development environment were chosen because their
capabilities best fit what was needed for creating the software
• One setback of using Visual Studio 2012 was that I had no prior experience
using this development environment. Because of this, a large portion of time
was spent teaching myself how to use Visual Studio 2012
Coding/Debugging
• Most of the time was used for programming and debugging
• Some of the features of the Kinect that were pre-programmed
• My program implements some of these capabilities, such as Skeletal
Tracking, which were accessible though reference to other coding projects
Testing
• Trials of 50 scenarios were ran in order to determine the success rate of each
function of the program (50 trials for each function = 150 total)
• The fall detection, intruder detection, and detection of leaving the home
were tested separately
Program Methods
The method used to track if the
user has fallen down. To do
this, the program compares the
position of the Shoulder Center
joint and the Hip Center joint. If
these joints have the same Y
value, the program recognizes the
position and returns true. This
Boolean method is used to
determine if a text message
should be sent.
The method used to detect if the
user has left the house. In this
method, the program compares
the pre-defined location of the
door (measured during set up)
and
the
user’s
position
to
recognize if they have left the
house.
The method is used to detect if
there is another person in the
house. In this method, the
program takes a count of the
number of tracked people in the
home and if the number of people
is larger than some predefined
number (2 is used here), the
program will indicate that an alert
should be sent out.
Kinect Skeleton Joint Mapping And Detection
The Mathematics Behind Fall Detection
Above
The image above shows the different
mechanics of the Kinect Sensor
Left
When the Kinect Sensor tracks a user, it
maps a skeleton onto them by identifying
the 20 different joint positions
Below
The Kinect Sensor can detect up to 6
users and can map skeletons onto 2 of
them.
• When the Kinect Sensor is tilted, a
new set of X, Y, and Z planes are
established. Because of this, the
program can detect the difference
in Y values of the Shoulder Center
Joint and the Hip Center Joint.
• As seen by the sensor, a person
laying on the floor would still have
some distance between these
joints.
• The Elevation Angle of the Kinect
sensor has a range of -27° to 27°.
• The blue dots indicate the Hip_Center joint and the Shoulder_Center joint
• The red lines show the new X and Y axes created when the sensor is tilted
• To find the distance between the
Shoulder_Center joint and the
Hip_Center joint (as seen by the
Kinect) we define the difference in
the Y values as “x”.
• We also use the distance between
the joints (in the regular X and Y
planes), which is measured during
system set up. We label this value
“d” in the diagram.
• Lastly, we use the angle between
the regular X plane and the
Kinect’s X plane. This angle is
equal to the negative value of the
Kinect’s tilt angle.
• From these variables, we derived
an equation for x.
• The green line is the desired length which is labeled “x”
• The red line is the distance between the joints labeled “d”
Future Work
Results
The final version of the program achieved the goals set for the project. The Microsoft Kinect sensor was implemented successfully in a
home monitoring system that provides protection and comfort to any person, especially the elderly or those with physical disabilities who
may not have access to a nursing home or other assisted living facility. The program uses the skeletal joint mapping capabilities of the
Kinect Sensor to analyze different body positions that the user makes. The program recognizes certain positions as indicators that an
accident has occurred or the user may be in danger.
The program successfully implemented the desired functions set for the project. However,
there are a vast number of possibilities for future expansion of this program’s capabilities.
With more time and a better understanding of Visual Studio Premium 2012 and what all the
Kinect Sensor is capable of doing, this home monitoring system could be enhanced in many
ways to further provide more safety detection capabilities.
Some of the future work ideas that will be considered are:
The system is capable of detecting a fall, an intruder, and if the user leaves the house. The alerting ability was also successfully
incorporated into the software program using the Twilio web messaging service. With the functional alerting system, a family member or
friend can receive an SMS text message from the system whenever the user has fallen, left the house, or if there is an intruder.
Another goal set for the program was to create a simple Graphical User Interface, with commercial capability in mind, that would allow
simple home installation in the future. This goal was achieved and customization was incorporated into the setup interface. In the setup,
the system takes measurements of the user, of the front door’s location, and allows the user to check-box the functions (fall detection,
intruder detection, leaving the house detection) that they would like the system to use. Images of the User Interface are below.
-
-
Screenshot of the initial greeting screen that appears
when the program is executed.
The mounting tab provides pictures
instructions for easy installation.
and
simple
The final setup tab allows the user to choose which
functions they would like to use. This adds the desired
customization to fit every user’s distinct needs.
Implementing detection to monitor if the user has taken their medication
Implementing detection to monitor if the user has left the oven or stove on
Implementing detection to monitor if the user has slept too long
Programming a specific (in-wall mounted) sensor to recognize certain actions that the user
can do to indicate that they think there is a problem
• As an example, if the user stands in front of the sensor and makes an “X” signal with
their arms, the sensor would read this as “I don’t feel well and need assistance” and
then the program would send out an alert.
• Also, the program could make use of the Kinect Sensor’s microphone; thus, if the
user says something like “Help”, the program will then send an alert.
Further enhancing the Graphical User Interface for easy in-home setup
Adding customization capabilities in the User Interface so that the system becomes
specialized to cater to the needs of each individual user
Adding a recognition system so Stay Kinected recognizes what the user’s skeleton looks
like. This will enhance the intruder detection.
One large expansion to Stay Kinected would be connecting different Kinect sensors
throughout the house to “communicate” with each other in order to keep track of where
the user is at all times. With this addition to the system, there are even more
enhancements that could be added such as:
• Programming the system to keep a log of the users activities as they move
throughout the house
• Providing an accessible location (i.e., phone application, website, etc.) in which a
member of the users family can check this log.
• Adding a live stream view on the phone application or website that a family member
can login and see, in real-time, what the user is doing.