Microsoft Kinect – Visualising a 3D World
Ryan Durrant - 23945583
University of Southampton
red1g10@soton.ac.uk
Abstract
In this paper I review the technology behind the Microsoft Kinect
and look at current and possible future applications in order to
evaluate the impact of the Kinect on the world of multimedia. I
will compare the advantages and disadvantages the Kinect which
allow it to be useful in these areas. I also propose an experiment to
understand further the effectiveness of the Kinect in education, an
area I believe it could be important and effective in improving the
learning experience for pupils.
Keywords: Kinect, Education, 3D, Computer Vision, Infra Red
Camera, Microsoft, Robot Vision, Depth Imaging
1. Introduction
When first released in November 2010, the Microsoft Kinect was a
whirlwind success. Some ranked it as a technological revolution
rivalling that of the personal computer and the internet [1].
Depth imaging was initially developed as a tool for gait analysis,
facial recognition and skeletonisation in order to combat terrorism.
Depth imaging was an expensive tool, the result of billions of
pounds and many years of development [1], out of the reach of the
common person. The Kinect package itself was rumoured to take
between 5 and 10 years to develop [2]. However, when completed,
it put top of the range technology in the hands of creative and
intuitive civilians.
There are already many applications being developed, beyond the
initial release purpose of a peripheral for the XBOX 360. For
instance, it has been researched in medicine for many possible
uses. Doctors could, for example, control a computer without
having to come into contact with un-sanitary keyboards [3].
In this paper, I will look at how the Kinect operates, the hardware
required and briefly at the software. I will then provide a broad
view of current applications and look at possible future
applications, especially in education, in order to analyse how the
Kinect is successfully affecting multimedia applications.
2. Related Work
Zhang, Z. (2012) wrote about the impact of the Kinect in
multimedia, focussing mainly on the different software attributes,
such as skeletonisation and facial tracking, and suggests that the
Kinect offers a huge range of applications throughout multimedia,
in particular for more immersive video conferencing [30].
good indication of what is in front of the camera. This works as we
know the speed of light to be, roughly, 300,000,000 metres per
second, so we can calculate the distance by multiplying through by
a half the time delay in seconds to get the distance in metres. The
Kinect, however, uses a dense field of infra red (IR) points, which
create a 3D map. The structured light technique can then turn this
into depth data as it is calibrated to know where each point is on a
flat surface a known distance away, it then calculates the
displacement of each point from this and thus its distance. This is
then stored in a 2D greyscale image with the colour value for each
pixel relating to its 3rd dimensional value, the distance, as seen in
figure 1. A pixel’s x and y co-ordinates within the image denote its
position on the xy-plane.
3.2 Why use a depth camera?
Depth cameras in general are very useful. While computers have
been able to analyse still images and even video from standard
cameras before, it is very difficult to get useful, real time data. This
is because, with a picture as an array of coloured pixels, colour
images do not give a great indication of what is going on in a 3D
space as it is very difficult for a computer to establish where one
item ends and another begins. Other limitations arise from the
nature of colour (RBG) images, the camera records an array of
pixels based on the exposure, the amount of light which passes
through the lens. This can be drastically altered by the light level
and source [1]. A picture of a human under sunlight, for example,
would look very different to an identical pose under fluorescent
tubing. Other factors, such as the exposure rate of the camera, can
also result in completely different images. This makes it very
difficult to generalise any software to get consistent results in
different environments and situations.
The depth camera, however, creates a depth image as an array of
pixels relating exactly to the distance from the camera. This allows
a computer to emulate one of the most important aspects of human
sight: the ability to recognise individual objects in 3D space. This
is because it is clear that if the distance on a pixel is suddenly much
further away to those before it, we know we have reached the end
of an object. As can be seen in figure 1, the RGB image on the
right is difficult for a computer to understand and includes
reflections and shadows from a light source which would make the
image very hard to understand in its current form. This is ignored
in the depth image, centre, though this shows up a shadow where
the IR field, left, returns no data.
3. History
Initially, this technology was produced by research established in
particular to observe crowds with more success than normal Closed
Circuit Television. CCTV relies heavily on humans to spot
something really small and easily missed, making it unreliable.
Depth cameras were developed by many pioneers in the fields of
Computer Science and Mathematics before Microsoft did a lot of
research, along with PrimeSense, to produce a civilian version for
the XBOX console. It was hugely successful and became the
fastest selling peripheral in history [1].
3.1 Developing a new sensor
Although the idea of a depth camera is nothing new in itself, the
Kinect marks the first time this technology has been within the
reach of civilians. The system used to power the Kinect is very
cheap, but it is not by any means poor.
Many of the previous depth cameras used Time-of-Flight (ToF)
methods, which use short bursts of light and read the time it takes
for the light to bounce back as well as the wavelength, giving a
Figure 1 – IR image, Depth image and RGB image captured by Kinect
There are two principle methods for creating a depth image:
Passive depth imaging takes multiple 2D images and tries to
compute the distance of pixels and objects, often through the use of
computationally exhaustive correspondence algorithms or optical
flow. Active depth imaging employs additional physical sensors,
such as an IR field as seen in figure 1. The problems with
correspondence can be avoided through active methods, such as
structured-light and ToF, which do not require these complex
algorithms to triangulate the depth of each part of an image. [13]
3.3 Design Motivation
Initially developed to combat terrorism, how did this technology
end up a peripheral in living rooms across the world?
Hardly a new route, technology has evolved from military and
government use to personal and business use for years. The
internet was developed for the military as a way of communicating
in the event of war. This path is also true for depth imaging and the
Kinect.
Microsoft initially had the vision of a controller-less gaming
console, which allowed people to use natural gestures. They took
this brief to PrimeSense, and the Kinect was developed. It initially
allowed Microsoft to combat systems developed by Nintendo and
Sony, which used different methods to capture motion but both still
required controllers. The aim was to enable people who were
perhaps daunted by learning to use the controller in order to play
complex computer games, expanding their target market. [4]
More recently, development has continued to utilise the Kinect in
other fields. Medicine and robotics are two areas being researched
currently [3,7,14,15]. Microsoft released an SDK, and OpenNI
released a framework, meaning that developers could not only use
the Kinect, but actually improve it for their needs. This third party
development is pushing advancements which could apply the
Kinect to aid many aspects of our lives.
4. Hardware
The hardware offers a step forward from previous depth cameras
which have been expensive and highly specialised. The Kinect
costs very little to produce and can provide better results than
much more expensive systems, and it does all of this in the living
room of civilians. [3]
An infrared projector projects the field of IR points as in figure 3.
These are read by an IR camera and fed to the processor by a
Complementary metal oxide semiconductor (CMOS) Active Pixel
Sensor (APS). This type of APS contains an integrated circuit with
an array of photo-detectors and is often used in camera phones and
even some digital SLR cameras. It has the main advantage of being
comparatively cheap and not requiring any special manufacturing
techniques. These sensors are also immune to blooming, in which a
strong light source results in nearby pixels having extra light. One
disadvantage is the rolling shutter effect as it records one row at a
time. These traits are perfect for the Kinect, as it allows the sensor
to be robust in most lighting conditions as well as remaining cheap
and efficient. The RGB camera is also controlled by this sensor
which translates the light into electrical signals which are
understood by the PS1080 chip. The cameras and the projector can
be seen in figure 2 with the RGB camera and IR sensor close
together in the middle, allowing the two images to be overlaid, and
the projector off to the side. There are also 4 microphones in the
body, which allow the Kinect to recognise different sound inputs
and ignore background noise such as the television when
connected to an Xbox.
The upgrade from ToF to structured-light 3D mapping requires a
boost in processing power, made possible by the PS1080 chip
which converts the raw data from the IR camera, the RGB camera
and the microphones into usable data. The Kinect can also rotate
through 27˚ up or down thanks to a motor attached to the base.
5. Software
Shane Kim, an executive at Microsoft, was quoted in 2009, after
the initial announcement of the Kinect, as saying “To me, the
magic is more software.” [4] With each pixel in the image received
from the Kinect’s sensors possibly encoded with RBG, Depth and
even sound information, the software has to manipulate a lot of
information; 9.2 million 3D points per second compared to 6800
2D pts/sec for a laser range finder [14]. PrimeSense developed the
PS1080 chip specifically to process this influx of data [3].
The information from the IR sensor provides red points from the
field which will have slightly different positions and sizes
depending on the distance of the point, as can been seen in figure
3. It then triangulates this with a virtual image, hardwired into the
chip [5] which contains the expected positions at a known distance
to create the depth image. It is worth noting that the shadow caused
by each item, visible in figure 3, does not return any data to the
Kinect, so the chip assigns the maximum depth value, of 2047, to
each point within this space. This allows them to be ignored as
insignificant background space. It then analyses the curves in real
space to produce useable information and a 3D model. It can also
use this depth image to create “visible audio” from the microphone
array, enabling the computer (or Xbox) to understand where in
space each sound is coming from. It can overlay the colour image
to the depth image, as the two cameras are a known distance apart,
to allow the user to see their actions affecting the outcome. This is
desirable as it enables the Kinect to distinguish between players
and sounds, enabling a more engaging interface for multiple users.
However, it results in the vast amount of data mentioned above.
In order to deal with this, the PS1080 chip in the Kinect analyses
depth values from points, multiple points are often found within a
single pixel [10]. Attempts are then made to interpret the data
before sending it on. It can do things such as locate a human body,
map its joints and body parts and estimate where, in all probability,
your body parts will be in a few nano-seconds time [6]. This is
done based on criteria for the human body, for instance it is
unlikely your elbow will bend backward. It then gives already
interpreted data to the connected device which enables it to use the
data without having to process all of the raw data. One of the more
impressive figures is the frame rate of 30Fps, allowing a real-time
experience.
Figure 3 –Magnified IR image captured with Kinect showing field of IR dots as seen by Kinect. The
dots in the foreground are much bigger and closer together than those in the background. This allows
the Kinect to calculate the distance at each point. The shadow caused by the object is also visible.
Other methods exist to speed up this data analysis. A fast sampling
plane filtering algorithm is proposed by Biswas and Veloso [14] to
reduce the volume of the 3D pixel cloud by sampling points to
produce an overview of the 3D map. This can be vital in systems
with less computational power, such as small robotics.
6. Advantages and Limitations
In this section I will analyse the advantages of the Kinect over
other similar depth imaging technologies, as well as looking at its
limitations.
Figure 2 - Shows the RGB camera in the centre, the IR projector on the right and the IR sensor on the
left in the Kinect [25]
However, despite it being hailed as a revolution, the hardware is
nothing massively new. It is, in fact, the software which is most
innovative.
6.1 limitations
There are a few limitations of the Kinect. It lacks the accuracy of
some of its more expensive peers, though it is still robust and
highly accurate, and even considered more accurate than many
other 3D sensors by Forsythe and Green [3]. It also has a limited
resolution, 640 x 480 pixels, though the RGB camera can run at the
higher resolution of 1280 x 1024 pixels but this will result in a
lower frame-rate. It is also not hugely important for a large
resolution depth camera as it will normally be used to follow large
objects which can be defined at lower resolution with enough
clarity to be understood. In this instance a higher resolution could
be a step backward as it would slow down processing times. One
of the main limitations is due to the structured light approach; it is
difficult, though not impossible, to include more than one Kinect in
a system as the array of IR dots will become entwined and will
confuse the results. The use of IR may also be a problem in direct
sunlight, which could limit its applications.
6.2 Comparison to other cameras
There are other cameras available, and other methods for them to
retrieve depth data. One such camera, the Point Grey Bumblebee
XB3, uses passive methods through stereo imaging and
triangulation. It has three cameras which each capture an image.
These are plugged into triangulation software to provide a 3D
image. It has a high resolution, 1280 x 960, and good depth range,
0.2 to 4.5 meters. However, the software is entirely hosted on the
connected computer, which requires a powerful processor, and the
increased frame size, along with computationally complex
algorithms, reduces the frame rate to a maximum of 15 frames per
second. It also requires parameter configuration in each
environment, making it limited in its applications.
Another camera, the Camcube, utilises an active time of flight
(ToF) method. It is very expensive, a common trait with the
Kinect’s competitors, lacks colour, and has a very limited
resolution, just 200 x 200 pixels. However, perhaps the main
limitation could be that depth data is noisy, and affected by lighting
conditions to the degree it is only useable indoors under controlled
lighting conditions.
Comparatively, the Kinect has a reasonable frame size, 640 x 480
pixels. It also maintains a good depth range, 1.2 to 3.5 meters, as
well as an incredible frame-rate at 30fps thanks to the complex
software on the chip. It has a low impact on the computer, as it
conveys depth data directly from the hardware. Also, vitally, it can
be successful in a wide range of situations, increasing its usability.
Pece et al. compared these three cameras [27] and concluded that
the Camcube provided the most accurate depth data. They also
claimed that ToF cameras will soon improve to include colour as
well as quickly come down in price. However, due to the
inflexibility of this camera, as well as the price differential, I would
argue that the Kinect is still a better and more useable option.
Figure 5 – The Kinect attached to wheelchair allowing the instructor to control the chair without
confusing the student [17]
7. Applications
Many applications of the Kinect have been researched and
developed across many disciplines since its release as a gaming
console peripheral.
7.1 Medicine
Medicine is one area in which there are many possible applications
of the Kinect. For example, the ability to operate a computer
without coming in contact with any un-sterile interfaces, such as a
mouse or keyboard, has obvious benefits to doctors who always
need to reduce risk of infection. [3]
The ability to recognise natural gestures allows for the use of a
Kinect to monitor vulnerable patients, ensuring their comfort and
safety in their own home as well as reducing the strain on health
care professionals required in hospitals or old people’s homes.
Current systems require a carer, or the use of a lifeline panic button
worn around the neck. However, the second of these is not robust,
as if the patient finds themselves in trouble; they may not be able
to activate an alarm. The first is also not robust as contact is only
temporary, it is also expensive and time consuming. One method
was proposed, using the natural gestures understood by the Kinect,
in collaboration with a data set containing gestures to watch out
for. The Kinect could then notice if the patient is showing signs of
distress, and communicate information on what the problem could
be and how severe it is with a medical professional. They could
then use the built in microphones to talk to the patient, and be able
to monitor them. This method is as yet only concept, though its
implications are vast and could be hugely beneficial. [3, 16]
Figure 4 – The results collected for one subject by Chang et al. [17]
The Kinect has also been explored as a tool to aid the rehabilitation
of people with motor disabilities requiring regular exercise in order
to improve their mobility. This was explored by Chang et al. [16]
where two patients were asked to do their exercises with and
without the aid of the Kinect. Through a simple animated game,
which saw a whale reacting directly to the motions the patients
made, their motivation and results were massively improved, as
seen in figure 4.
A similar application, though different approach, has been
developed in which a Kinect is used to speed up teaching children
to used motorised wheelchairs, as in figure 5, by minimising the
distractions posed by the carer. If child makes a mistake they risk
hurting themselves or others, which requires the carer to move his
hand into the chair to take control, which can distract and confuse
the child, this can be avoided using a Kinect to override the
wheelchair based on the instructors commands. [17]
7.1.1 Advantages and Disadvantages of the Kinect
The Kinect could be particularly useful in the medicinal world
primarily because of what it enables people to. The price is
particularly important when trying to rehabilitate children or young
adults with motor difficulties as it could be purchased for the home
to encourage continued exercise and increase motivation as well as
keep medical costs for insurers, health services and hospitals down.
The IR camera enables the Kinect to monitor patients in low light,
or even no light, which could be hugely beneficial as night time is
often when patients are most vulnerable. Other types of camera
which rely on visible light such as CCTV and passive depth
cameras could not do this. The use of microphones also enables
another sense to be included which can provide vital information as
well as a more personal and beneficial communicative connection
between patients and medical professionals.
One disadvantage of the Kinect is its static nature, which would
make it difficult to monitor a patient across more than one room.
This could be overcome with the use of personal robotics, as we
will discuss in the next section, or multiple sensors. The Kinect
also raises ethical issues as it captures personal information.
7.2 Robotics
Another area that the Kinect is currently being developed is as a
navigational aid for robotics. Nick Hawes (et al.) have developed a
robot, named Dora the Explorer, which aims to improve its own
knowledge about its surroundings [11,12]. This could lead to
robots helping around the house, and could even be implemented
into other systems such as looking after vulnerable patients as
mentioned above. The need for a 3D sensor exists even for the
simplest of tasks; before a robot can make a cup of tea it needs to
calculate the position of the mug in front of it. Initially Dora was
designed with 2 point grey lasers to produce an idea of its
surroundings, but has since been updated with a Kinect, as in
figure 6, showing how the Kinect can hold its own against more
established technologies. Hawes claims that the Kinect is a “huge
new technology” [28] for robotics due to its incredible cheapness
and its effective data capturing. With the original system, items
were only sensed at the level of the sensors. It was noted that tables
were difficult, as they were not substantial enough at the height of
the sensor to be recognised. The Kinect has improved this
massively, allowing the robot to build a much “richer map” of the
local environment [28]. However, it still has difficulties sensing
things like towels, which are irregular and un-textured, and in
particular glass which doesn’t register with IR. Applications such
as this heavily rely on the ability to recognise objects around it.
Figure 6 – Dora’s Sensor layout including two point grey sensors and a Kinect (left) [29] and more
recently with just the one point grey sensor and a Kinect attached to a laptop.
Methods enabling the Kinect to recognise objects in images are
also being developed. There are many data-sets about, and one
proposed by Janoch et al. utilises both the RGB and depth cameras
to build up a database of images for various objects from different
angles in attempt to enable category level object recognition [18].
This research can add further to the usefulness of the Kinect in
robotics by giving a better understanding of the environment.
It is also shown by Stowers et al. (2011) [13] that the depth camera
and its feedback are robust enough under experimentation to be a
valid controller for the flying altitudes if quadrotors, as in figure 7.
The authors concluded that the Kinect was able to work effectively
under changing conditions and that its “low cost, high frame rate
and absolute depth accuracy over a useful range make it suitable
for use on robotic platforms”.
Figure 7 – The Kinect on a Quadrotor [14]
7.2.1 Advantages and Disadvantages of Kinect
The main advantage of the Kinect to robotics is the very low cost.
Robotics is expensive, however to be plausible in everyday life, it
is required to be affordable and robust. The Kinect excels here as it
provides robust data at a low cost, as well as being small and
sturdy meaning it can be integrated easily. It can also be used in a
wide range of situations as it does not need to be calibrated in
every new environment and can be used in almost any lighting.
The Kinect, however, might be rendered useless by direct sunlight,
as it uses IR. This could be a big dent to its effectiveness in
robotics, though more information is needed on the subject. It also
requires a minimum distance, 0.8 metres, to be able to calculate the
depth, which maybe an issue for a robot who needs to get close to
things. This can often be solved by placing the sensor at the back
of the robot body, though this can result in blind spots.
7.3 Education
One area in which I believe the Kinect could be influential, and
deserves more research, is education. Considering Chang et al.
found that the Kinect can increase motivation [16], I feel that it
could revolutionise primary and secondary education, akin to the
impact of interactive white boards (IWB). In particular, methods
could be developed to enable teachers to teach while moving
around the class, enabling children to get more involved through
interactive games and even alert teachers when a student raises
their hand.
Development in this area has been pushed by Microsoft, who have
many lesson plans available utilising the Xbox and the Kinect,
though many of these are not specific, including games such as
Kinect Sports and relying on teachers to make it educational and
relevant. For example, the distance which a student throws a
javelin can be used to teach decimals. Bowling could be used to
help understand the base 10 system. Darts can be used to learn
addition and subtraction. These can also enable competition in the
classroom which can also improve student motivation.
It is known, and has been shown many times including by Brekke
and Hogstad [26] (2010), that the use of computers and technology
can improve students’ motivation, satisfaction and results. We also
know that students learn in different ways, and that the more senses
used, the better they will retain information, and the stronger the
neurological connections in the brain will be. These are both
evident with the Kinect, which enables children to be physically
active while interacting with technology. When tested in Houston
Texas, as discussed by D. Enriquez-Vontoure in [21], it was found
that the children were much more motivated and engaged in the
lessons. Also, it was evident that students were better behaved,
and developed better team-working skills. It was also found to be
an effective reward and motivational tool to increase homework
completion rates and in class activities. 100% of teachers in the
sample saw these improvements, though only 66% would
recommend it as a useful and relevant instructional tool. 66% is by
no means poor, but why is this so low? Especially if the system
enabled better engagement while combating bullying and
improving behaviour [21]. One suggestion is that teachers were not
sure how to use the resource, or how to relate it to lessons. Another
is that new technology is not yet developed enough in this field. So
why is this technology worth developing? Games have existed in
education for years, so what makes the Kinect such a revolution?
7.3.1 Advantages and Disadvantages of Kinect
The Kinect could be very beneficial in this area for many reasons.
It is cheap, compared to an IWB which will set you back between
£600 and £1600 a Kinect can be bought and plugged into a
computer for less than £100. It is versatile, useful not just for
student interaction, but also for teachers and even parents [21].
Teachers can use the natural gestures interface to interact with the
computer without having to divert their attention from the class.
However, arguably the main benefit of this technology is its
accessibility.
The accessibility of the Kinect means it can be used by anyone.
Students who have disabilities, such as partial or even full
blindness, can use the Kinect easily. Feedback based entirely on
sound to headphones can enable the student to learn and interact
where many other input methods, such as a keyboard or controller,
would fall down [21].
Another aspect of the Kinect which lends itself to education is its
ability to recognise faces. The use of Kinect and Avatar Kinect has
enabled educators in the USA to help autistic children to learn how
to interact socially, by allowing them to interact with themselves,
as well as other people’s avatars in a safe and comforting
environment. [21, 23]
There are other aspects of the Kinect which one can see being
beneficial: The multiple microphone array, for instance, could
allow for a single player to be heard in a room full of children. The
RGB camera has been used as a webcam to provide the
opportunity for international conferences, allowing children to gain
a global awareness [21]. The speed with which the Kinect can track
people would be useful, and the ability to recognise people will
ensure that scores are kept safe even if two players change quickly.
The motorised base will also allow for players of different heights,
which sound trivial, but would save a lot of effort and time.
One disadvantage, however, is that students’ engagement will
decrease over time when playing the same games, so the Kinect
will require constant updating to stay interesting, though this is
nothing new and is true for all tools in education. Teachers will
also be required to plan lessons with it, and to be creative with it,
and this can be daunting as well as incredibly time consuming.
However, despite all these advantages, we still only have 66% of
teachers in Houston schools who were chosen as a sample to test
the Kinect recommending it as a useful tool. To understand why
this is, I would propose an experiment into the effectiveness of the
Kinect in education as follows:
7.3.2 Experiment
I think the future of the Kinect as a tool in education lies, not in the
ingenuity of teachers when applying games to learning, but with
more specific and specialised applications. Games programmed
with the sole purpose of teaching using the Kinect allow for much
more interactive lessons with much more specific lesson
objectives. For this reason, any experiment into the effectiveness of
the Kinect must reflect different methods of teaching. So, I would
propose that a pre-existing game or method is tested, along side
two or three bespoke games programmed from scratch with
varying levels of interactivity and utilising different aspects of the
Kinect. It would then be important to test these fairly, alongside a
control group, to see which, if any, has the best impact on learning.
It would also be important for the testing to adhere to common
practices, such as the experimental design model, so the results can
be reliable. For this reason, a test schedule which includes two
periods of contact as well as two periods without contact should be
applied. This can provide information on how the tool actually
impacts on the class as well as hopefully showing the impact over
time. Theoretically with scores higher during the second noncontact period than the first if the tool is effective.
To get fair results, I would suggest 4 games in the same subject
area, in this case maths. The first should already be implemented, a
good example being the use of Kinect Sports darts game, as seen in
figure 8, to improve basic arithmetic skills.
Figure 8 – Kinect Darts is being used to aid with teaching maths
Secondly, a bespoke game which enables more teaching to be done
directly through the game could be used. Again, I would suggest a
game based on darts, but which utilised maths questions to tell
students what to aim for, and awards a high score for getting the
right answer. This would provide a sense of competition between
students to get the answer right, and could be supplemented with
time constraints or bonuses. This will allow a direct comparison
between bespoke software and use of a pre-existing game.
Thirdly, I would propose a less interactive game which will focus
more on purely getting the right answer. A game which utilises the
Kinect’s ability to follow two players at once could have two
students side by side competing to answer the question on screen
first from a selection of answers simply by moving their hands
over it. The class could help by shouting out the answer and
competition between two teams, say boys and girls, should provide
ample motivation. In addition to this, players could have to swap
after each question to improve team-working skills.
Finally, I would use a more active game. A game in which students
attempt to pop a virtual balloon containing the answer to a question
asked on screen by jumping on the correct one, would allow
students to be much more active. Penalties could be used for
popping the wrong balloon, and scores can be kept to encourage
competition.
The testing method will also need to define what it is to be
successful as well as what it is to be helpful. For the Kinect to be a
helpful teaching tool, it would require the teacher to notice
improvements in the class when teaching. This would be quantified
through a questionnaire which collects information on the
motivation, behaviour and teamwork of the children as well as
other important aspects. In order to call the system successful,
there would need to be some evidence of an improvement in the
amount students are learning. This would require some form of
assessment to be performed before, during and after the
experiment. However, it was suggested by Cameron Evans, a
Microsoft employee, that conventional examination methods such
as multiple choice, might not directly show how successful the
Kinect can be [21]. He even suggests a more interactive exam to
complement the more interactive learning style. However, a short
multiple choice test which must be short enough for the student to
complete more than once could be used.
These games use a wide range of techniques which the Kinect
makes available and the relative successes of each can show us
both if the Kinect can be a useful tool and hopefully which aspects
are most beneficial.
A student undergoing any form of teaching should improve over
time, so there must be a randomly assigned control group which
never gets contact with the system in order to provide a
comparison. A further issue comes from the difference between
understanding and knowledge, for example: is it important for a
student to understand why 1 + 1 = 2 or just that it is? Clearly,
understanding is vital in many situations but in others, such as how
to drive where it is not required for the driver to understand why
turning the wheel turns the car, just knowing that it does is enough.
The tests should include questions which are easy for most students
even before contact, plenty of examples of work which is covered
during the contact as well as some which are not covered, but use
skills which are taught. This will show both how well they retain
knowledge, such as 1 + 1 = 2, but also how well they understand;
for instance if they then know 102 + 1 = 103 even if it never comes
up. The scores for each pupil can then be compared at each stage to
see any improvements, and the overall scores can also be
considered. These can be compared to the control group to see how
much improvement is down to the Kinect. An extension could look
into another subject area, and swap control groups such that the
control group in one subject is the contact group in another, which
will avoid them becoming demoralised which could skew the
results.
The results of the questionnaire and the results of the tests can
provide qualitative and quantitative data with which to conclude
the effectiveness of the Kinect in the setting.
7.3.2.1 The Games
For this experiment, I would test in a primary school, looking at
young children, though the methods could equally be applied,
through different games, to any level. Even in higher education as
is mentioned by Barilovits and DePriest (2011) [24].
7.3.2.2 Collecting Results
There should be a questionnaire for the teacher and tests for the
pupils at each of the stages of the experiment. Initially, looking at
the ability the pupils show and what the teacher’s initial opinions
are, then at each stage we want to see how the scores and the
teachers’ perceptions change until we get the final results.
The questionnaire should focus mainly on how the pupils are
motivated, how they behave, how they perform and how they
interact with their peers. This information is important, as a method
which is effective at teaching, but results in students misbehaving
is not beneficial.
7.3.2.3 Predictions
I would expect the results to reinforce the success of the Kinect in
other applications. Student results should be higher after the
experiment and in particular during contact periods. Also, the
teachers should find that children are more motivated and more [12] Nick Hawes, Matthew Klenk, Kate Lockwood, Graham S.
interested in class as well as other areas of school including
Horn, John D. Kelleher, Towards a Cognitive System that can
homework. I would suggest that the more active methods would be
Recognize Spatial Regions Based on Context,
more fun to start and therefore could have the biggest impact as
http://www.cs.bham.ac.uk/~nah/bibtex/papers/hawesetal12cdsr.
children like to be active. It is also known that physical exercise
pdf, accessed 11/11/2012
strengthens neurological pathways when learning.
[13] Stowers, J., Hayes, M., & Bainbridge-Smith, A. (2011, April).
Altitude control of a quadrotor helicopter using depth map from
7.4 Other Applications
Microsoft Kinect sensor. In Mechatronics (ICM), 2011 IEEE
Other applications range from simple interactive experiences in
International Conference on (pp. 358-362). IEEE, Istanbul,
public buildings, such as banks or museums, to choosing what to
Turkey, accessed 11/11/2012
wear in the morning [20].
[14] Biswas, J., & Veloso, M. (2012, May). Depth camera based
indoor mobile robot localization and navigation. In Robotics
8. Future work
and Automation (ICRA), 2012 IEEE International Conference
on (pp. 1697-1702). IEEE.
I would like to run the experiment proposed in this paper in order
to assess the impact which the Kinect could have on education. I [15] R. Rexit. (2011, December 15). Visualiztaion of Posture (for
Kinenct Pain Recognizer), accessed 12/11/2012
also feel that the Kinect could continually be developed in new
https://docs.google.com/viewer?url=http%3A%2F%2Fwww.cs.
areas as the interface it offers is so easy to use that the possibilities
pitt.edu%2F~chang%2F231%2Fy11%2Fproj11%2Ffinalruh.pdf
really are huge
[16] Chang, Y. J., Chen, S. F., & Huang, J. D. (2011). A Kinect9. Conclusion
based system for physical rehabilitation: A pilot study for young
adults with motor disabilities. Research in developmental
The Microsoft Kinect is a very impressive piece of technology
disabilities, 32(6), 2566-2570.
which enables a plethora of new uses. Interfaces controlled by
natural gestures are a phenomenal advancement as it will allow [17] D.K. Zondervan and D.J. Reinkensmeyer, Kinect-Wheelchair
Interface Controlled (KWIC) Robotic Trainer for Powered
learning curves to be dramatically reduced. The technology is also
Mobility
cheap and durable, while still producing quality results. There are
many areas in which it can be applied, and we have discussed but a [18] A. Janoch, S. Karayev, Y. Jia, J.T. Barron, M. Fritz, K. Saenko,
and T. Darrell, A category-level 3-D object dataset: Putting the
few.
Kinect to work. ;In Proceedings of Computational Methods for
The technology in the Kinect is revolutionary as it enables people
the Innovative Design of Electrical Devices. 2011, 1168-1174.
to do extraordinary things for a very low price. It is also found to [19] Smart Boards have revolutionised the way that we teach today,
be highly accurate with the data it provides, and though not the
http://www.stmarys.pta.school.za/pebble.asp?relid=56, accessed
most accurate camera, it is arguably preferable due to the
12/11/2012
durability, flexibility and ease of use. We also noticed that the use [20] http://www.youtube.com/watch?feature=player_embedded&v=
of infra-red enables the camera to be used in almost all light
1jbvnk1T4vQ#!, accessed 13/11/2012
conditions, an improvement over other stereo methods. It is also [21] A.Popa, D.Enriquez-Vontoure, C.Evans, Bring Learning to Life
less computationally intensive than passive methods as it avoids
with Kinect, accessed 27/11/2012
correspondence algorithms.
http://msevents.microsoft.com/CUI/VideoDisplay.aspx?Registr
ationId=1312602906&culture=en-US
It has been shown how the Kinect could be applied successfully in
the fields of medicine, robotics and education, and proposed a [22] IWB prices, http://www.projected.co.uk/smartboard.htm
[23] http://usatoday30.usatoday.com/news/health/story/2012-05method to evaluate it effectiveness in education. Indeed, the
31/video-games-autism-students/55319452/1, accessed
Kinect offers a huge range of options for many multimedia
27/11/2012
applications across the board, and has successfully been integrated
[24]
DePriest, D., & Barilovits, K. (2011). LIVE: Xbox Kinect© s
into some of these.
Virtual Realities to Learning Games. In 16th ANNUAL TCC
Worldwide Online Conference, Hawa,
10. Bibliography
http://etec.hawaii.edu/proceedings/2011/DePriest.pdf, accessed
[1] Borenstein, G. (2012). Making Things See: 3D vision with
27/11/2012
Kinect, Processing, Arduino, and MakerBot. Make Books, 978- [25] Kinect Hardware Image, http://www.socialphy.com/posts/do-it1-449-30707-3
yourself/7717/Do-it-yourself_-Kinect-teardown.html, accessed
[2] http://gilotopia.blogspot.co.uk/2010/11/how-does-kinect-really28/11/2012
work.html, accessed 11/11/2012
[26] M.Brekke, P.H.Hogstad, New teaching methods - Using
[3] Forsythe, T., & Green, M. ADVANCEMENTS IN 3-D
computer technology in physics, mathematics and computer
SENSING TECHNOLOGY IMPLIMENTED BY THE
science, International Journal of Digital Society (IJDS), Volume
KINECT, accessed 11/11/2012
1, Issue 1, March 2010, accessed 01/12/2012
[4] http://venturebeat.com/2009/06/02/microsoft-games-executive- [27] F.Pece, J.Kautz, T.Weyrich, Three Depth-Camera Technologies
describes-origins-of-project-natal-game-controls/, accessed
Compared , accessed 05/12/2012,
12/11/2012
http://www0.cs.ucl.ac.uk/staff/F.Pece/page9/files/abstract.pdf
[5] http://mirror2image.wordpress.com/2010/11/30/how-kinect[28] N.Hawes, L.Vernall,
works-stereo-triangulation/, accessed 12/11/2012
http://www.birmingham.ac.uk/accessibility/transcripts/Nick[6] http://www.joystiq.com/2010/06/19/kinect-how-it-works-fromHawes-Robots-the-reality.aspx, accessed 05/12/2012
the-company-behind-the-tech/, accessed 12/11/2012
[29] Picture of Dora the Explorer Robot by N.Hawes,
[7] http://www.roborealm.com/help/Microsoft_Kinect.php,
http://www.flickr.com/photos/61648734@N06/6021726504/siz
accessed 11/11/2012
es/m/in/set-72157627384115286/, accessed 05/12/2012
[8] CogX Project website, http://cogx.eu/results/dora/, accessed
[30] MicZhang, Z. (2012). Microsoft Kinect Sensor and Its Effect.
11/11/2012
Multimedia, IEEE, 19(2), 4-10.rosoft Kinect Sensor and Its
[9] http://www.i-programmer.info/news/105-artificialEffect
intelligence/2176-kinects-ai-breakthrough-explained.html,
accessed 11/11/2012
[10] http://electronicdesign.com/content/topic/how-microsoft-sprimesense-based-kinect-really-works/catpath/embedded,
accessed 11/11/2012
[11] http://www.cs.bham.ac.uk/~nah/bibtex/papers/gobelbeckeretal2
012.pdf, accessed 11/11/2012