7. Testing - University of Ulster
An automated system for reading hand measurements in patients with rheumatoid arthritis
Aaron Bond
B00545783
Bond-a@email.ulster.ac.uk
Computer Science BSc Hons
Supervisor: Dr. Kevin Curran
December 2011
1
Contents
2
3
Abstract
Rheumatoid arthritis affects around 1% of the world’s population. Detection of the disease relies heavily on observation by physicians. The effectiveness of these kinds of tests is dependent on ability and experience and can vary depending on the observer. This research will aim to investigate the use of Xbox Kinect for monitoring in rheumatoid arthritis patients as a cost effective and precise method of assessment.
A system has been developed which implements the Kinect sensor for usage in a hand recognition and digit measurement capacity. This system performs the tasks usually completed by a physician such as digit dimension monitoring and exercise observations.
Measurements taken include digit width and height; measurements which can be accomplished different distances from the Kinect and in varied environmental conditions.
The tests completed are stored in a database for retrieval and analysis at a later date. This allows the physician to monitor a patient over a period of time without requiring multiple appointments where the measurements are taken manually. Ultimately, the system is proof that a Kinect-based solution is not only plausible but highly reliable and functional in many scenarios which call for regular observation of a patient. With the system being designed to be portable and easy-to-use, it is an ideal solution for both the physician monitoring patients in a clinic as well as posing a possible solution for patients wishing to monitor their own condition in their homes.
4
Acknowledgements
I would like to thank Dr Kevin Curran who has inspired and supported me throughout this project.
His continued direction and assistance has been invaluable in the development of this solution.
I would also thank my family and friends for their irreplaceable support and encouragement.
5
1. Introduction
Rheumatoid arthritis (RA) is a chronic disease that mainly affects the synovial joints of the human skeleton. It is an inflammatory disorder that causes joints to produce more fluid and increases the mass of the tissue in the joint resulting in a loss of function and inhibiting movement in the muscles. This can lead to patients having difficulties performing activities of daily living (ADLs). Treatment of RA is determined by physicirans through x-rays, questionnaires and other invasive techniques. An example of this would be angle measurements taken using instruments such as tape measures or a dynamometer to measure grip strength. There is no cure for RA but clinicians aim to diagnose it quickly and offer therapies which alleviate symptoms or modify the disease process. These treatment options include injection therapy, physiotherapy, manual therapy (i.e. massage therapy or joint manipulation) and drugs which can reduce the rate of damage to cartilage and bone.
These treatments are assisted by patient education. Patients are shown methods of
Joint Protection, educated in the use of assistive tools to aid in ADLs and shown altered working methods. This document proposes to research the viability of using the Xbox Kinect camera and sensor to accurately record a patient’s hand measurements. Its proposed functionality would allow detection of joint stiffness over a period of time. If shown to be a viable option it would aid in the diagnosis of RA and the discovery of appropriate treatments for the patient.
1.1 Project Aims & Objectives
This main purpose of this document is to analyse the relevant issues faced when implementing a system designed to assist in the assessment of rheumatoid arthritis patients. The primary aim of this project is to assess the viability of a Kinect-based software system for the real-time and historical measurement of hand movement and deformation in
RA patients. Its development proposes a viable gain over current goniometric measurement methods.
6
1.2 Existing approaches
Existing methods and approaches to digitally measuring hand dimensions and movement have failed to address the key issues surrounding RA treatment. While these solutions seek to allow automatic and accurate measurement, many use non-commercial hardware and rely on proprietary software which can be very expensive. Similarly, these devices tend to be highly technical and require the supervision of a trained technician.
1.3 Project Approach
The hand recognition and measurement system designed and implemented in this project will aim to present a functional and user-friendly alternative to current goniometric measurement methods. It will attempt to overcome challenges and limitations of other physical systems and establish the best solution to the common issues.
1.4 Chapter Overview
Chapter 2 provides a background and review in the areas of defining rheumatoid arthritis, current goniometric assessment methods, state-of-the-art computer vision and contemporary glove-based hand monitoring technology. It also outlines important considerations for working with RA patients.
Chapter 3 is a requirements analysis, specifying increased detail on the proposed system and the constraints which apply to it.
Chapter 4 is a development plan. Herein lies the time-management and project planning documentation.
7
2. Background & Related Work
2.1 Methods of detecting and measuring hand movement
Measuring hand movement in this context refers to the ability of a given system to determine finger-digit movement in relation to the rest of the hand. Also, some methods may allow for automatic detection and measurement of swelling and deformities of the hand. These characteristics are essential when tackling the development of a system aimed at assessing the symptoms and progression of an RA patient.
2.1.1 Current physical goniometric methods
Current goniometric methods for monitoring and assessing joint mobility and deformity are mostly analogue. Among the measures and practices used to establish the patient’s disease activity are several self and physical assessments. These are essential for the continued treatment of RA in a patient, allowing the physician to determine jointprotection exercises as well as potential medicinal treatment in the form of antiinflammatory and auto-immune medications.
Measurements of a patients hand is recommended to be taken at regular visits to their doctor (Handout on Health: Rheumatoid Arthritis, 2009). These assessments can include hand measurements, blood tests (among other lab tests), and X-ray imaging of the affected hand.
Sphygmomanometer (Grip Pressure)
A sphygmomanometer is used to assess a patients grip strength in their affected hand. This is achieved by inflating the cuff of the device to a standard pressure in a rolled manner, then having the patient grip the cuff in the palm of their hand. After the patient has squeezed the cuff, the physician can take a reading of pressure which can be used to indicate patient’s grip strength (Eberhardt, Malcus-Johnson, & Rydgren, 1991). However, using the modified sphygmomanometer can proved misleading results. This instruments pressure gauge is activated when the patient squeezes the air filled compartment (Ashton &
Myers, 2004). The limitation of this being that patients will larger hands will have artificially lower pressure readings than patients with smaller hands. This is due to the variance in pressure applied over the surface area (Fess, 1995).
8
Jamar Dynamometer
The Jamar dynamometer is seen as a reliable alternative to the modified sphygmomanometer. It is a hydraulic instrument, functioning within a sealed system. It measures grip strength in kilograms or pounds of force (Ashton & Myers, 2004). Its versatility, simplistic functionality and cost effective features makes this method easily accessible (Fees, 1987). It has been found to provide accurate readings and the results are reproducible (Hamilton, Balnave, & Adams, 1994). This is an additional benefit the Jamar dynamometer has over its mechanical counterpart the Stoelting dynamometer. This mechanical method measures tension when force is applied to a steel spring and is not viewed as a reliable measurement (Richards & Palmiter-Thomas, 1996).
Questionnaires
In assessing the patient’s discomfort, the physician must rely on several questionnaires in order to gain an understanding of disease progression.
The Stanford health assessment questionnaire, for example, is designed to assess the average morning stiffness the patient feels in the affected joints of their hands (Eberhardt,
Malcus-Johnson, & Rydgren, 1991). This is measured and recorded in minutes and is used to gain an understanding of how long it takes for the patients joints to loosen and become supple again.
Similarly, the patient is assessed on their experience of pain levels since their previous examination. This is done through a questionnaire in which they must evaluate their pain levels and discomfort over the preceding period.
Another assessment of this form is comprised of several questions regarding ability and pain when performing ADLs. Commonly, the ADLs which are assessed include “dressing and grooming”, eating, cutting and preparing food and general hand control over cups and jars (Eberhardt, Malcus-Johnson, & Rydgren, 1991).
Patients are also assessed using the Visual Analogue Scale to measure their level of pain and discomfort. This consists of a line, marked on one side as “No pain” and on the other as “Worst pain imaginable” and patients are asked to mark a spot along the line which reflects the current feeling (Schofield, Aveyard, & Black, 2007).
9
Similarly, a Health Assessment Questionnaire is designed to establish the patient’s ability to perform daily tasks, with each question grading their capability using a “four-point grading system”. This measures their “daily functionality level” (Fries, Spitz, Kraines, &
Holman, 1980).
Radiographic assessment
As a result of RA, joints in a patients hand and fingers can suffer bone erosion to varying degrees. In order to measure and document this, the patient will undergo radiographic tests in the form of X-ray imaging and MRI scans of the affected areas. This shows how the bones in the patients hand are affected and can be measured over a period of time to show disease progression and activity.
Another method which has the potential to highlight key areas of bone-degradation and joint swelling is an ultrasound imaging of the affected hand. This offers a less invasive method of assessing bone density and level of swelling in the patient. However, Chen,
Cheng & Hsu (2009) have shown that “prognostic value of MRI is not directly transferable to
Ultrasound” and therefore it is, not yet, an adequate option for assessment.
Clinical tests
Typically, several clinical tests are performed to establish the disease activity level; including urine, blood and other tests. From these tests, the patients Erythrocyte
Sedimentation Rate and C - reactive protein results are established (DAS Booklet - Quick reference guide for Healthcare Professionals, 2010). In patients with rheumatoid arthritis, the C – reactive protein and ESR levels are used as a measurement and indication of inflammation in the patient’s joints (Black, Kushner, & Samols, 2004).
General techniques
When visiting their doctor, the patient will have their movements assessed in the areas where their RA is affecting them. The physician will check for the presence of fingerthumb drift, swan neck/boutonniere deformity, as well as Bouchard and Heberden nodes
(Rheumatoid: Hand Exam, 2011). The examination consists of a patient placing their hand
10
flat on a table (where possible – depending on patient discomfort) with their elbow and wrist resting flat.
Using a goniometer, the physician examines (in degrees) extension, flexion, adduction and abduction of the proximal interphalangeal (PIP), metacarpopalangeal
(MCP) and distal interphalangeal (DIP) joints of the fingers (Arthritis: Rheumatoid Arthritis,
2008).
This determines thumb-index finger drift (position of index finger away from thumb) and palmar abduction (de Kraker, et al., 2009). The measurements are all documented in handwritten forms and are recorded to aid future assessments. These readings are all influenced by physician training and observations therefore they can vary between examiners.
2.1.2 Camera-based movement detection
There are many options when attempting to determine a movement of a subject via camera-based methods. Providing a system with “computer vision” and allowing it to assess variables such as movement, size and depth of an object, is the goal in camera-based solutions. Some camera based-solutions require proprietary hardware, while others are able to utilise common devices and already existing technologies.
Open Source Computer Vision (OpenCV)
OpenCV is a cross-platform function library focusing on real-time image processing.
The aim of this library is to supply an application with “Computer Vision”, the ability to take data from a still or video camera and transform it into new representation or decision
(Bradski & Kaehler, 2008). By taking pixel location and colour information, the library builds an image matrix which it uses to “see”.
OpenCV was originally developed and released by Intel. Since its release in 1999, the library has allowed a method of tracking motion within captured video and given developers the ability to discern movement angles and gestures. Also, in terms of utilising images and making a decision, here it refers to the ability for any given system to then automatically determine people or objects within a scene. Functions like this are possible with statistical
11
pattern recognition, located within a general purpose Machine Learning Library (MLL) included in the library.
This allows for implementation of many features including “Object
Identification, Segmentation and Recognition, Face Recognition, Gesture Recognition,
Camera and Motion Tracking” (Chaczko & Yeoh, 2007).
The library is now supported by Willow Garage, meaning it has a consistent release schedule – therefore the project is fully supported and reliable for future development. Use of the library allowed the Stanford Racing Team from Stanford University to complete and win the DARPA Grand Challenge, an autonomous vehicle race in which OpenCV was used to provide the vehicle with “Computer Vision”.
OpenCV is optimised to run on Intel-based systems where it finds the Intel
Performance Primitives. Bradski & Kaehler (2008) note that while the library consistently outperforms other vision libraries (LTI and VXL), its own processing is optimised by about
20% with the presence of IPP.
The OpenCV library works well with installed camera drivers to ensure that it functions with most commercially available devices. This allows developers to create applications and rely on non-proprietary, widely available camera equipment. Therefore cost and development become a lot more practical for potential developers. Furthermore, in relation to potential environments and scenarios in which applications may be deployed, utilising existing cameras and commonly available devices means that applications can be implemented in a wide array of locations.
Prosilica GC1290 System
The Prosilica GC1290 system is a product designed to facilitate the measurement of a hand for patients with RA (GigE Vision for 3D Medical Research, 2010). Designed by threeRivers 3D, the device is intended to monitor the joint swelling in a hand by recording changes in the volume of the patients joints. A metal frame (80cm high, 60cm wide and
40cm deep) houses a total of four cameras and scanners. Two 3d laser scanners project patterns and grids onto the patient’s hand which is then returned in order to create a 3d representation. The laser scanners are equipped with a monochrome camera in order to record this image and identify the laser grid. A colour camera picks up a standard image and
12
is used to monitor joint deformation, while a thermal imaging camera detects joint inflammation. There is also a device intended to measure thermal information located near the hand rest; this is used to provide reference information: ambient room temperature, patients general thermal information.
All data taken from the device is recorded and displayed in real time in order to minimise problems such as motion blurring because of hand movement. This data is then processed by proprietary software packaged with the device to display this information (at
32 frames per second) to the patient and the physician. The software system used is also deployable to all major operating systems (GigE Vision for 3D Medical Research, 2010). With the range of information gathered by this device, it would allow physicians to gather very specific and relevant information on a patient; and process it in a relatively short period of time. Similarly, using a device which outputs measurements on a patients hand standardises the procedure and readings; making them more assessable. This is because the information gathered by the device is statistical and provides a quantitative assessment of disease progression. Furthermore, this limits human error in measurements taken and does not rely on the physicians judgement. The Prosilica system does have some drawbacks, however.
Since the device is bespoke it is not commercially available but is designed for medical use.
This results in the device requiring direct contact with the manufacturer. This also has an adverse effect on the affordability of the device. The device itself is relatively large, consisting of the aforementioned cameras and frame. While the device could be suited for use in a doctor or physician’s office or surgery it would not accommodate home visits and physician mobility. In cases where a physician is required to perform a home visit to the patient, it is not feasible that the device could accompany them due to size and associate cost.
Microsoft Kinect
The Kinect is a device which facilitates the translation of real-world objects and motion into 3d representations. The basics of the device were initially developed by
PrimeSense, who later sold the technology to Microsoft. The device utilises a number of sensors in order to accumulate input which can be compiled into a digital representation. It
13
has one camera which allows for input in the infra-red (IR) spectrum which returns a depth map. This map is transmitted from an IR transmitter located next to the IR receiver and consists of a projection of dots onto the target area 1 . Also, the sensor contains a third camera which receives standard RGB (human spectrum) input in order to gain a colour image of the target area. The colour input camera receives information at a resolution of
640x480 pixels while the IR receiver gathers input at 320x240 pixels. Both cameras run at 30 frames per second. The field of view on the depth image is 57.8 degrees (Limitations of the
Kinect, 2010).
The device also contains a microphone array for receiving sound input (which can allow voice recognition and commands). This consists of 4 microphones placed along the bottom of the Kinect. Lastly, the Kinect features a motorised base. This base allows for targeting of the sensor bar; adjusting its position to acquire the best perspective of the target space. This base allows for manoeuvring allows for a total alteration of 27 degrees vertically in either direction. All of these features of the Kinect make it capable of processing an area to determine distance to an object as well as colour and audio ambience.
While a standard camera with computer vision software may be able to determine objects in a space, it can become difficult if there is a lack of colour differentiation between the object and the surrounding space. Tölgyessy & Hubinský (2011) assert that with the extra cameras and sensors, performing tasks such as image segmentation becomes a lot easier, especially with the distance threshold which can be assigned to the input. This allows unwanted background data to be filtered out and reduces the noise in the input. Microsoft has also released an SDK which contains drivers and other files associated with producing an application utilising the Kinect. The SDK allows for the device to be used with a Windows 7 operating system and supports C++, C# and Visual Basic programming languages. Along with access to the raw sensor information the Kinect is gathering, the SDK also allows for skeletal tracking (identifying humans and human gestures) via bundled libraries (Ackerman, 2011).
One of the main advantages of the Kinect is the accessibility of its hardware. The
Kinect is a relatively advanced device allowing for computer vision. By combining advanced hardware with a commercial price-point and making an SDK available, Microsoft have
1 Kinect Dots - Night Vision with Kinect Nightvision Infrared IR http://www.youtube.com/watch?v=gbzXjdHfJA&feature=related
14
allowed developers to capitalise on the capabilities of the device at relatively low cost. This promotes its use in varied environments since maintenance and cost are comparatively small when regarding other advanced computer vision utilities. The device is mobile too.
The Kinect sensor was designed and built for home use, making it reliable in many conditions. For optimal functionality, the device requires standard room lighting. It requires that the room be lit well enough that the standard RGB camera can pick up information but also not so bright that the IR patterns become indistinguishable (Carmody, 2010). A downside of the system is that for accurate readings, the subject must be at least the minimum distance from the device. This minimum distance for the Kinect sensor is 0.6m and the maximum range is variable between 5m – 6m (Limitations of the Kinect, 2010). -
However there is an inexpensive add-on for the Kinect which acts as a zoom lens, reducing the minimum distance required.
2.1.3 Glove-based systems
As an alternative to current goniometric methods, there have been many investigations into glove-based technologies. These aim to assess a patient’s finger and joint movement in order to aid in diagnosis and treatment of RA. Existing glove-based solutions, use varied methods of reading joint mobility and tension. Among the technology used are sensors using magnetic technology, electrical resistors and contacts or LEDs with flexible tubes (Dipietro, Sabatini, & Dario, 2008).
5DT Data Glove
Previous research into the use of glove-based technologies has shown the 5DT Data
Glove to be among the most accurate versatile gloves available (Condell, et al., 2010). It utilises fourteen fiber-optic sensors; with two sensors per digit and one sensor for each knuckle on the patients hand. It also has a tilt sensor mounted on the back of the hand to measure the orientation of the patient’s hand. The sensors on the glove work by measuring light travelling through the sensors. As the patient moves their hand, the stress on the sensor changes, altering the amount of light passing through the receiver.
15
The glove is produced by Fifth Dimension Technologies and allows for accurate measurement of hand/finger movements; passing the information via USB to either the bundled software or software developed to utilise specific aspects of the glove. To accomplish the creation of custom software to utilise the glove, it comes with a crossplatform SDK in order for developers to make better use of the data they are able to collect.
However, this glove is only beneficial if the hand to be tested is always going to be either the left or right hand of a patient. Since the glove is designed to fit only one side of the patient, a new glove must be used should the measurements being taken be desired from the other hand. Furthermore, if the measurements are to be taken from a patient with a different sized hand than the glove which is available, a more suitable one must be found.
Dipietro et al. (2008) also found that the most accurate results were read from the device when the cloth of the glove fit the patients hand well. Were the cloth too tight, the glove would restrict movement in the patient and give readings which were more extreme than the actual movements. However, if the glove material was loose on the patient, readings were not representative and were less than the actual movements. While the glove allows for highly accurate information readings from the patient’s hand; it has some problems which are intrinsic to its design. Gloves like this one are designed to measure hand movements and gestures while the software has been designed to incorporate that use into hand assessment tools for RA patients. One of the main symptoms of RA is hand and finger deformation along with “periarticular osteopenia in hand and/or wrist joints” (Arnett, et al.,
1988). Combined, this results in limitations to hand movements and articulation. Thus, the finger and wrist articulation which is needed in order to manoeuvre the hand into a glove can become painful and difficult.
2.2 Considerations for Patients Suffering Rheumatoid Arthritis
Solutions designed to facilitate and aid diagnosis of vulnerable patients – ones which are in chronic or debilitating pain, for example – face an array of unique requirements.
Rheumatoid Arthritis affects around 1% of the population (Worden, 2011) and causes synovial joints in affected areas to become inflamed due to extra synovial fluid being produced. This can lead to a breakdown of the cartilage in the joints and can cause the
16
bones in the joint to corrode. As a result, patients commonly exhibit deformation in their fingers and joints; as well as note regular and occasionally disabling pain (Majithia & Geraci,
2007).
2.2.1 Patient Mobility
Typically, assessing patient mobility is a case of factoring in the patient attending their local medical practitioner for tests or treatment. This can become difficult however if a patient has limited mobility. For a patient who is suffering RA, it is possible that their disease is afflicting more than one set of joints in their body. Also, having the disease increases the risk of osteoporosis in the patient due to the nature of the disease and the medication they are required to take (Handout on Health: Rheumatoid Arthritis, 2009).
In effect, this can mean that the patient would require home visits more commonly than a patient who is not suffering joint pain. Physicians required to visit the home of their patients in order to assess the current disease progression and possible treatments must have access to portable equipment. Therefore the equipment used must be mobile, easily set-up and be an inexpensive product. Portable low cost equipment does exist that aids treatment at home, however these methods have their own limitations that must be considered. The Jamar dynamometer has proven to be an inexpensive and reliable gauge of grip strength, providing data used in assessment. However, in patients with decreased mobility, a grip strength test would prove to aggravate there symptoms and increase levels of pain. This option is also open to false reading from patients not willing to exert their maximum grip strength due to the uncomfortable nature of the test (Richards & Palmiter-
Thomas, 1996). There appears to be a lack of a measurement device that can record patients’ treatment progression that is portable, cost effective and which has maximum consideration for patient discomfort level.
2.2.2 Patient Comfort
It is important to understand the difficultly some RA patients have in completing simple movements. In order to gain some insight it is essential to comprehend how their joint function compares with average joint function (Panayi, 2003). Healthy joints require little energy to move and the movement is usually painless. However for RA patients their joints have a thickened lining, crowed with white blood cells and blood vessels. Movement
17
of affected joints not only causes bone erosion but also triggers the release of a chemical within the white blood cell causing a general ill feeling (Panayi, 2003). This secreted substances cause the joint to swell, become hot and tender to the touch while also inducing varying levels of pain. Increased swelling, triggered by the white blood cells response causes joint deformation.
Severe joint deformity can render traditional methods, such as the manual devices mentioned earlier, ineffective. However it also presents limitations for the proposed advance methods currently being developed. The glove method requires the patient to fit their hand into a standard size glove. This method fails to address the fact that RA patients do not have standard joint movement; therefore manoeuvring their hand into the glove could cause unnecessary pain and discomfort. Additionally a standard glove does not accommodate for joint deformity, especially not the extreme deformities that are symptomatic of RA. The difference in finger and joint size is also not considered. RA patients usually have symmetrical joint deformity, i.e. if there third knuckle on their right hand is affected then it is likely that the same joint on the left hand will be affected (Panayi,
2003). Expanding this example, if the same joint on both hands is swollen then the glove would either fit appropriately to the swollen joints or the surrounding joints. This increases result variability as hand movement cannot be standardised. In order for the glove method to accurately measure joint movement and limit discomfort a custom version would be needed for each patient. This would not be a viable option since the progression of RA would require patients to have multiple gloves fitted.
2.3 Research Conclusions
Current goniometric tests are not repeatable and are subject to human error. This can lead to adverse effects on the patient treatment. However, proposed solutions in the areas of glove-based measurements fail to address the fundamental issues like patient comfort and differing hand sizes. Moreover, the cost incurred with these solutions renders the systems impractical. In order to maximise patients comfort during testing, an external non-contact device is needed for RA patients. This is one of the proposed benefits of a potential Kinect method. The patient would perform movement tasks but they would not be
18
restricted by any outside materials. Movement would be recorded digitally aiding treatment analysis.
The Kinect’s versatility and cost effectiveness address accessibility issues. It would be a beneficial, portable piece of equipment that could be purchased by physicians and also patients. Therefore patients could carry out movement tasks daily; the results would be recorded by the Kinect and a computer. The data could then be assessed by the physician at a later date. A continual data supply would aid treatment planning and could also indicate differences in movement throughout the day. Providing a fuller grasp of movement functionally that is not currently assessed, due to the time restrictions of appointment allocations for patients. Also, since the Kinect is an external sensor and is only a means of providing raw data to a software system, a computer-vision library such as OpenCV can potentially be implemented to handle the standard image recognition tasks. This maximises the effectiveness of the Kinect since it would be combining the libraries available with the
Kinect SDK and also the open source libraries which are contributed to by a large community of developers.
19
3. Requirements Analysis
Detailed in this section is an analysis of the problems surrounding the development of software for a camera-based solution to current rheumatoid arthritis patient assessment.
This section outlines the functional and non-functional requirements of the proposed solution and details several development methodologies which will be considered. The selected methodology will be chosen based on its merits in meeting the requirements of the solution for design and implementation.
3.1 Problem Statement
Already in this document several key areas which prove problematic have been identified in relation to a RA assessment solution. Existing methods in practice by doctors and physicians involve physical measurements and personal judgement to assess the patient’s disease progression. This results in some measurements being inaccurate due to human aspects like perspective and personal opinion which can have adverse effects on patient treatment. These methods can prove inconsistent and fail to provide an accurate representation of the patient’s current disease level. Furthermore, many attempts to automate this process of assessment via glove-based and camera based systems have proven ineffective, not taking into account aspects of patient comfort and mobility as outlined in section 2.2.
The aim of this project is to implement a software based solution which will incorporate the use of the Microsoft Kinect movement sensor to monitor hand movements in patients with RA. Of the current solutions available, most utilise proprietary hardware which tends to be expensive. With the use of advanced features of the Kinect – a commercially available product – this project aims to make the solution affordable and effective. The solution will provide digital feedback on the measurements of a patient’s hand (size, joint angles) over time in order to assess disease progression. Further to this, it will allow physicians to have the patient perform exercises and the system will determine maximum flexion and extension for the manipulated joints among other necessary calculations. These calculations and readings will be collected and stored in a database so that the historical data can be viewed by the physician, expediting treatment selection and disease analysis.
20
3.2 Functional Requirements
Wiegers (2003) describes the functional requirements of a system as the expected or intended behaviour, documented “as fully as necessary”. While this is a difficult part of the development process, it gives the developer a proper definition of exactly what the proposed system is intended to do. The following is a succinct list of the functional requirements of the system which integrates Kinect functionality with a software based solution. These have been established based on research in the area of RA and from communications with RA patients.
The proposed system will be able to:
determine base hand measurements of the patient
determine initial joint angles at resting position
monitor maximum flexion of a specified joint during predefined exercise
monitor maximum extension of a specified joint during predefined exercise
assess time taken to perform predefined exercise
establish a connection with a database in order to record measurements and assessments
give real-time feedback on measurements
run on Windows based computers
3.3 Non-Functional Requirements
Chung, Cesar & Sampaio (2009) state that the requirements analysis of a project are essential as it establishes what the “real-world problem is to which a software system might
be proposed as a solution.” In addition, Wiegers (2003) also defines non-functional requirements as the “performance goals” of the system; including aspects of design such as
“usability, portability, integrity, efficiency, and robustness.”. Below is a list of the nonfunctional requirements of the proposed system. It is categorised based on the recommendations of Roman (1985); Interface, Performance, Operating, Software and
Hardware Requirements.
21
Interface Requirements
This section details how the system will interface with its environment, users and other systems.
The system will conform with HCI best practices in sections which exhibit user interfaces and utilise display which presents an easy-to-understand depiction of the measurements
Performance Requirements
This section details how the system will behave in order to meet its functional requirements in an optimal manner. This includes addressing unexpected situations and also methods which will be employed in order to allow the continued operation of the system.
The system will be able to:
cope with or present notification of adverse lighting conditions for image recognition
handle erroneous measurements taken from the system, disregarding readings which are outside of logical bounds
connect to the database for historical data with little or no wait before the information is retrieved
automatically determine if the patients hand exhibits deformity in order to construct the activities or exercise which will be performed by the patient during examination
determine if the subject is in the correct operating space for optimal reception of information in the sensors, and adjust or notify accordingly
deal with unexpected closure of the software application or disconnection of the
Kinect sensor
run on laptops or desktop computers running Windows , allowing for connectivity of the Kinect sensor via standard USB 2.0 connections
display sensitive patient information to the appropriate users (i.e. if the system is used by multiple physicians, a physician will only see their patients)
encrypt database information so that it does not allow for sensitive data to be accessed on the host machine
22
Operating Requirements
This section details aspects of the design which account for possible user limitations and system accessibility in the case of maintenance. This also includes elements such as portability and repair.
The system will be:
accessible to physicians who have appropriate login information and patient data
user-friendly in that it allows for users with little or no training in the use of the system perform an assessment of a patient by following on screen prompts
easily maintained or replaced as it consists of a commercially available (and relatively inexpensive) device
robust enough to withstand a lifecycle in a physician’s office which is usually quite busy
portable in cases where it is required on home-visits (system would consist of laptop and Kinect sensor)
23
Hardware Requirements
This section details the hardware which is required to develop the system and which is required in its implementation. The hardware required for the development and implementation does not differ.
The required hardware is:
a Microsoft Kinect Sensor (no modification necessary)
a sufficiently powered computer or laptop, capable of running the software outlined in the requirements below
Software Requirements
This section details the software required to design and implement the application.
The required software is:
Microsoft Visual Studio 2010
Kinect SDK (contains drivers required to access the sensor)
24
3.4 Detailed Functional Requirements
Through the use of the Unified Modelling Language (UML), this section will detail the requirements specified in section 3.2.
3.4.1 Use-Case Diagrams
What follows is a specification of the potential use-case scenarios of the proposed system. These use-cases define the actors and their interactions with the system and one another. The primary actors in this system are the physician (or potentially, a nurse) and the patient being assessed. Interaction between the system itself, the physician and the patient
is intrinsic to its design and usage. Figure 1 shows the possible high-level use-case of the
system.
Figure 1: Use Case scenario
T he use-cases reflect only the high level actions of performed within the system. These actions are described below.
Configure Environment – The physician configures the Kinect sensor by placing it in position and connecting it with the computer. This also includes the application portion of the system initialising. Once it has been started, the physician will configure the application for
25
use with the individual patient, setting up intended exercises, personal information and other data which may be recorded.
Begin assessment – The physician begins the assessment in the application. The system will notify the physician if there are any issues with the current set up configuration (including positioning of the sensor). The system will first make note of the patients hand dimensions.
Next it prompts the patient with exercises to perform in order to perform the included actions; monitoring joint angulation and the extremes of the movements performed.
Make assessment – The physician analyses the historical data for the current patient; this includes previous hand measurements and evaluations performed in the system. From this data, the physician can determine a course of action for the treatment of the patient. This is then recorded along with the digital readings taken by the system and saved to a database for future reference.
26
3.5 Software Development Methodologies
Software development methodologies consist of a framework designed to structure the design and implementation process. The main elements of the software development lifecycle are outlined within a methodology in order to establish the plan which will result in the best possible software being developed. Structurally, most methodologies refer to these elements as Analysis, Design, Testing, Implementation and maintenance (Conger, 2011). The main aim of this section is to establish the most prevalent software development methodologies and determine the most appropriate one for this project.
The Waterfall Model
Due to its straightforward nature, the waterfall model has survived since the early days of software design. In this structure, elements of the development cycle flow down through the model from one section to the next. Conger (2011) states that the waterfall model is easily described, where “output of each phase is input to the next phase”. Similarly,
Conger (2011) asserts that the traditional outcome of the waterfall model is an entire application. This means that at each stage of the cycle the overall product is assessed and the model is examined in order to best design the entire system and consider it as part of the development before implementation. However, one of the main ideals of this methodology is a strong reliance on documentation at each stage of the development.
Boehm (1988) states that this can become the “primary source of difficulty with the
waterfall model”. In projects which are producing interaction-intensive systems, the end result may not be fully realised until the system requirements are established. This can result in documentation being forced at a stage when it is not required or needed.
Rapid Application Development
The RAD model allows for faster software development and implementation. In this structure, requirements and designs are changed and updated as the product itself is being produced. This results in the system and the documentation being produced at the same time, allowing for late changes and update to be done. This model is very adaptable and will allow for unforeseen software issues or new requirements being introduced to a project. These are introduced to the specification and are implemented as part of the
27
overall design. Often, the main stakeholders in the system have an active role to play throughout the process. However, this methodology suffers some criticisms. While it offers the stakeholders a strong input into the project at all levels, this can become detrimental to the project design and implementation as it is usually responsible for an increase in scopecreep. In this scenario, the system being developed has a specification which is constantly shifting to match what the stakeholders are seeing of the in-process design.
Incremental and Iterative Development
Incremental development implements a structure whereby the development of an application is broken down into key segments which are reassessed (through each of the different sections of a traditional methodology: analysis, design and implementation). At the initial stages of this process, a basic implementation of the designed system is produced.
This allows the stakeholders to get an idea of overall functionality and then through added increments, additional functionality is included in the specification. This methodology allows for issues in design and implementation to be established and addressed early in the software development lifecycle. Each iteration of the software can be considered a functional implementation of the design.
Deciding the Most Appropriate Methodology
The most appropriate methodology for each project can be different. With unique constraints and requirements, the structure to the development process must also be tailored. Deciding on which of the methodologies listed above is the most appropriate is determined by understanding these key requirements. It is important to realise that the key functionality of this system will be recognising and measuring a patient’s hand. Therefore, this functionality is a priority for the system and is essential for it to prove effective in use.
However, further functionality is required for a better user experience (visual feedback in
UI, historical data). Furthermore, the development process itself may introduce issues of software limitations that are unforeseen until the implementation of the system is performed. It makes most sense, therefore, to implement an incremental and iterative strategy to the development process. This methodology requires that a functional version of the system be created from the outset, with extra functionality being layered on top of this initial design. In practice, this would mean that the proposed system would have the most
28
important features designed and implemented first to ensure functionality. Later, extra layers of functionality can be added which improve user experience but the system as a whole is never rendered useless by an incomplete layer. This provides an overall modular design, ensuring that testing and bug-tracking is also easier due to the potential removability of layers.
29
4. Project Planning
This project aims to develop a camera based solution for assessment of RA patients.
The following chapter will address the proposed timeframe and structure of the development of a Kinect camera based solution. Further, it will outline the details of mitigating possible issues faced in creation of the system, as well as detailing implementing the development with proposed IDEs and other tools.
4.1 Plan of Work
Adhering to the incremental and iterative approach to software development, this project has been separated into desired functionality areas. These areas are:
Recognising the patients hand
Establishing hand dimensions
Monitoring pre-defined exercises
Integrating the system with a database
These general areas allow the development to be separated into individual iterations. These iterations represent products with key functionality.
Iteration 1
Following this iteration the system shall:
Feature a basic user interface which allows the application to be started and the information being received from the Kinect be shown
Display the raw sensor data from the Kinect. This will allow the information being received to be seen and for comparisons to interpreted information
Determine whether a hand is presented in front of the sensor when required. This is only required under ideal conditions at this stage
This stage does not require information essential to assessment be shown. The primary goal is to get the system functional.
30
Iteration 2
As well as performing the functions implemented in the first iteration, at the end of this iteration the system shall:
Recognise the presence of a hand in less-than-ideal conditions, allowing for the device to have a more versatile usage environment
Be capable of displaying basic hand dimensions such as width and height in profile
Implement a more functional UI in order to display the measurements of the hand it is monitoring
Iteration 3
As well as performing the functions implemented in the second iteration, at the end of this iteration the system shall:
Implement a more functional UI which displays the designated exercises for the patient to perform; allowing the system to make more accurate readings
Take readings from the patients hand movements regarding extremes of motion
(flexion and extension) in the joints
Differentiate between resting and in-motion states
Iteration 4
As well as performing the functions implemented in the second iteration, at the end of this iteration the system shall:
Be able to connect with a database in order to record the measurements of the patients hand and exercises
Perform some encryption on the information stored
Allow the physician to analyse historical measurements of the patients hand via an adequate UI
31
4.2 Time Allocation and Milestones
Below, the intended time allocation for each section of the development process, based on the iterative steps outlined in section 4.3.
Development Stage
System design
Iteration 1 in development iteration 2 in development iteration 3 in development iteration 4 in development
Testing and Evaluation
Finalising system and project
2
2
4
1
1
2
3
Time (weeks) Date
02/01/2012
16/01/2012
06/02/2012
20/02/2012
05/03/2012
02/04/2012
09/04/2012
The milestones below will allow progress in the development process of the system to be judged.
6
7
4
5
#
1
2
3
Milestone
Complete system design
Complete iteration 1 in development
Complete iteration 2 in development
Complete iteration 3 in development
Complete iteration 4 in development
Complete testing
Complete project
4.3 Gantt Chart
Date of completion
16/01/2012
06/02/2012
20/02/2012
05/03/2012
02/04/2012
09/04/2012
16/04/2012
32
4.8 Conclusion
Ultimately, this report is intended to produce the fundamental starting blocks which will form the foundation of a solid project. This document achieves this by detailing the core issues surrounding the contemporary and state of the art solutions, presenting these findings in a manner which will aid in the development of the proposed system. By outlining key problems such as patient comfort during assessment, this document has categorically proven that current approaches such as glove-based methods have fundamental weaknesses in design. It is here that the solution this document proposes will prove effective.
Further, the development plan outlined in section 4 of this document will allow for frequent assessment of project progression, ensuring adherence to schedule and efficient resolution to potential issues. Similarly, by conforming to the development methodology outlined in section 3.5, the system will be sure to have a fundamental basic iteration which allows for increased functionality to be layered on top of an already working system. These aspects all point elements outlined within this document being the optimal method of analysing a solution when proposing a software system. This document will ensure that the system created will have the most potential possible to become a fully realised and functional utility to aid physicians treating patients with RA.
33
5. Design
This section documents the planning and design of the project. Included in this section are the technical details and descriptions of the hardware used (Kinect Sensor) and of the software which is the basis of the system. Furthermore, this section describes the process whereby the system will determine finger dimensions from Kinect image data.
5.1 Kinect Sensor
A Kinect Sensor is the medium chosen to receive the images of the subject’s hand. The Kinect is currently available in two models; the “Xbox 360 Kinect” and the “Kinect for Windows”. Both models are functional with a Windows-based PC and can utilise the Kinect SDK released by
Microsoft. The Kinect for Windows has been modified to allow for readings to be taken much closer to the device than allowed by the Xbox 360 version. This ensures a greater accuracy of data taken from the subject. For the purpose of this design the software will be designed to work with both the
Kinect for Windows and the Xbox 360 Kinect; allowing users to utilise whichever is more accessible with the knowledge that Kinect for Windows readings will be more accurate at closer ranges.
Figure 2: Microsoft Kinect for Windows
For the design of this project, the Xbox Kinect will be used due to affordability and accessibility of the device. However, all code produced can run on both platforms as much of the business logic which handles transferring information from the device to the development computer is achieved through drivers and image-processing libraries like the Kinect SDK. This abstraction allows for maximum versatility in the system.
34
5.2 Software
To achieve the level of abstraction from hardware necessary to facilitate both versions of Kinect, several libraries are used to pass the raw information to the program. Also, the drivers which are initially installed when using the Xbox 360 Kinect work extremely well when utilising the Kinect SDK.
However, due to the main focus of that SDK being “skeletal tracking” (meaning that the Kinect SDK is designed to pick up and monitor full body movements) it falls quite short when attempting to use it for the purpose of hand recognition.
PrimeSense hardware drivers/OpenNI middleware
The PrimeSense hardware drivers work with the OpenNI framework to provide an alternative to the
Microsoft-issued drivers and SDK. This open-source combination provides a level of detail unachievable in the standard SDK when using the Xbox 360 Kinect. By default, the standard
Microsoft SDK declares all data within a range of less than ~80cm of the device unusable. This makes it extremely difficult to register information on a hand since the level of detail needed is much higher than that which is afforded by the SDK. However, the PrimeSense /OpenNI framework allows for information up to just over ~50cm.
CandescentNUI
The CandescentNUI project is an open–source Kinect library which works with both versions of the
Kinect in order to very accurately track hand information 2 . This library works with either C# or C++ and can be used along with the Kinect SDK or OpenNI. For the purposes of this project OpenNI is the framework of choice since it would work to a higher level of accuracy in the Xbox 360 Kinect than is achievable through the SDK. Furthermore, C# is chosen as it would allow the use of the Windows
Presentation Foundation for interface design.
Security
Primarily, security of the user’s data is ensured by having all readings and information stored in a secure web-based server. This can be encrypted to safeguard against hacking breaches and can be accessed remotely requiring secure log in information.
The log in information the user will need to provide on each start-up of the system will be a unique username (pre-set before use) and password combination. The user’s password will also be hashed using an MD5 function which is a one-way hashing algorithm. This will ensure the storage of the password is secure on the server and will be unlikely to be compromised.
2 Candescent NUI project page at CodePlex http://candescentnui.codeplex.com/
35
5.3 System design
The user will log in to the system via a login window and will proceed to be presented with a window showing their readings for the past month. From here they can choose to take new readings or perform some hand exercises.
System starts
User attempts login
Login details correct
Yes
User selects an option
No
View history
Selects history
Selects measure
Selects exercise
Do exercises
Take new measurements
Is user finished?
No
Yes
Stop system
Figure 3: System navigation
To ensure that the measurement and exercise sections work properly the system needs to determine that the Kinect sensor has been connected and is receiving data. If this is not the case the system will prompt the user with an error until it has been connected. However, the history section will work without the need for the Kinect to be connected to the system; allowing access to the data even without the hardware needed for full usage of the system.
36
5.4 Database
Using Connector/NET it is possible to integrate a C# system with web-based MySql database implementation. This is the preferred option of database technology for a number of reasons:
Existing familiarity with MySql formats and development
Web-based data access allowing remote tests feeding back to a centralised database
Security of information – hand readings are never stored locally but instead on a web server
For design purposes a localhost server set up will be implemented in order to test functionality. This is directly scalable to a production server environment at a later date.
The system utilises the user’s ID property as a key to link records across the tables. This allows the records to be quickly gathered and sorted for the user upon request. The relationship between the
data taken from the Kinect and the web-based server is shown in Figure 4.
Web-based database
Kinect Sensor
Laptop running system
Figure 4: Kinect system server relationship
Database design tblUsers
Field Name
(KEY) user_id username password
Type
Int (10)
String(16)
Properties
Key – simple int to create user relations
String to hold the users preferred username
String(40) String to hold hashed (md5) password
Figure 5: Users table design
37
tblReadings
Field Name
(KEY) reading_id timestamp thumb_width index_width middle_width ring_width pinky_width thumb_height index_height middle_height ring_height pinky_height
(FK) user_id
Type
Int(10) timestamp double double double
Properties
Key – to arrange readings
Simple int to create user relations
Contains user finger width
Contains user finger width
Contains user finger width double double double double double
Contains user finger width
Contains user finger width
Contains user finger height
Contains user finger height
Contains user finger height double double
Contains user finger height
Contains user finger height
Int(10) Foreign key from user table to gather results
Figure 6: Readings table design
38
5.5 Form design
Hand Recognition Interface - Login
Please enter login information:
Username
Password
Login
Hand Recognition Interface - Measurements
Measure right hand Measure left hand
39
RESULTS
Thumb 20mm
Index 18mm
Middle 19mm
Ring 18mm
Pinky 17mm
X
X
Hand Recognition Interface - History
07/09/2012 – 14/09/2012
Hand Recognition Interface - Exercise
Begin exercise
Show data
Date | Thumb | Index | Middle | Ring | Pinky | Hand
08/09/2012 - 09:30am | 20 | 21 | 20 | 20 | 20 | Right
08/09/2012 - 10:30am | 21 | 21 | 20 | 19.5 | 20 | Left
08/09/2012 - 11:30am | 20 | 18 | 20 | 20 | 20 | Right
08/09/2012 - 12:30pm | 19 | 21 | 20 | 19 | 19 | Left
08/09/2012 - 01:30pm | 20 | 22 | 20 | 21 | 20 | Right
Please stretch hand to display all fingers.
When all fingers are identified, close hand into a fist.
The time from start to finish will be recorded
X
X
40
5.6 Measurements
The data gathered from the CandescentNUI provides many valuable pieces of information which can be analysed and used to produce valid results.
Firstly, the system will only proceed to measure hand data when all 5 fingers are present and readable by the program. This will ensure accuracy since the hand has to be properly oriented in
employed:
Base left Base right
Figure 7: Finger base value location
No
Begin analysing finger
Has width been found
Yes Get (width / 2) Get fingertip Y Get base left Y
Use
Pythagoras’ theorem to find height
Save finger height
Figure 8: Finger height analysis
41
Begin reading hand data
No
Are all fingers present?
Yes
Assign each finger to appropriate variable
Get next finger
Get base-left X,Y
Get base-right X,Y
No
Is base-left Y = base-right Y?
Yes
Finger width is base-right X minus base-left X
No
Use
Pythagoras’
Theorem to determine width
Have all fingers been analysed?
Yes
Completed
Figure 9: Finger width measurement process
This process ensures that each finger examined by the system is assigned to a relevant local variable and thus can be analysed and stored based on which part of the hand object it belongs to.
Employing this method will also mean the readings from the fingers are as accurate as possible no matter the orientation of the hand.
42
6. Implementation
This section details the methods through which the raw image and depth information received from the Kinect sensor is used to create useful hand and finger data. Also detailed is the implementation of the form designs from chapter 5 into a fully realised C# system. Finally, the methods used to access and manipulate data from the web-based database server are described along with explanations of code design.
6.1 Technical structural overview of system
As previously mentioned in chapter 5, the system uses several supplementary software frameworks in order to receive and analyse information from the Kinect sensor. The Kinect sensor reads the scene and this image and depth information is passed from the Kinect to the OpenNI framework via a set of 64bit PrimeSense drivers. The CandescentNUI implemented in the main C# program then accesses the OpenNI data stream and constructs it into usable objects for use by the hand recognition system.
The graphical user interface (GUI) is constructed using Visual Studio 2010’s designer and
XAML code. The type of interface created is a Windows Presentation Foundation (WPF) project which allows for efficient form navigation in the form of XAML “pages” which can be linked to and navigated away from. This layout allows the system to retain a sense of being light-weight and efficient since the pages are only loaded as-and-when they are needed and are not causing too much background processing. Furthermore, the GUI is developed in such a way as to be approachable and easy to navigate for any user since a main objective of this project is to test viability of patient homeuse.
The Connector/NET addition to the C# project which allows integration with MySQL also facilitates quite functional MySQL statement writing. The MySQL statements which can be used can be input as shown in {FIGURE}:
43
6.2 Determining finger information
To begin, we create an instance of the data source in order to handle the information coming from the Kinect into the OpenNI framework:
This code also sets the maximum depth range for the Kinect sensor to receive data as
900mm; this is chosen because at ranges close to and above 900mm, with an image resolution of
640x480 pixels, hand and contour information becomes very difficult to establish and the integrity of the data is questionable.
The CandescentNUI code which is used for the majority of the hand recognition returns detailed information back to the recognition interface. In the system, a listener is set up to handle new frames of information being returned from the Kinect sensor.
This code creates a new “HandDataSource” object which we can then use to establish the listener for the event when a new frame becomes available for analysis. Furthermore, this also allows us to start and stop the hand recognition functions; all computation begins when the
hand.Start() method is initiated.
In order to give feedback to the user of the computation that is being performed on the image data being received, we add a raw data window which contains much of the pertinent information involved with the hand recognition. This window may be presented in either RGB (full color) image format or a black and white depth interpretation of the data. For the purposes of this system, it is more effective to show the depth data since it gives a better idea to the user of the
location they need their hand to be in in order to achieve optimal readings -
44
Now, when the depth data is being analysed by the system we can automatically pass it forward to the interface of the system in order for the user to observe the changing depth and finger
recognition information as seen in Figure 10.
Figure 10: Sensor depth data and RGB with finger recognition
The location of the cluster information and finger information overlays align much better with the depth data than with the RGB. This is due to the location of the two cameras which receive this information being located slightly separately apart.
In the context of the code above, videoControl is the name assigned to the raw data video window placed on the interface. Upon each updated frame, the new information is passed to this control in the form of manipulating its source property.
Lastly, we add a few layers of information over the top of the raw depth data in order to make it more presentable to the user.
These two layers contain the outline of the hand when it has been recognised along with finger point and base pin-points as well as the cluster information which is used to determine the
45
whole hand. This looks like a matrix of dots which appear over the user’s hand in the image to show
that it has been recognised as seen in Figure 11.
Figure 11: Hand cluster data
The data generated by Candescent allows for the hands on screen to be enumerated and for each hand to have a number of “finger” elements. Within these objects numerous pieces of position and depth data are accessible.
In order to establish which finger the data is associated with, we must first determine which finger is which in the hand object. To achieve this, a method utilising the logic of finger position on the X axis is used.
This code is run for each finger. This code is utilised in the case of the left hand being presented as the thumb in that context would be the digit located furthest along the X axis. This code is run for each digit and as it is assigned, it is removed from the hand object; allowing the system to re-enter the hand object and assign any unassigned fingers.
Each finger has a “BaseLeft” and “BaseRight” property as well as a “FingerPoint”. Each of these objects has an X, Y and Z (depth) value. It is from these that we are able to determine dimension data and whether the users hand is positioned correctly for optimal readings to be taken.
46
Optimal position for readings is determined by taking the Z value for the left-most digit and Z value for the right-most digit and comparing them. Using this method we can establish horizontal tilt of the hand and can formulate an algorithm which will alert the user if their hand is positioned too far skewed on the X axis.
This code compares the left-most and right-most digits and uses a threshold of 4% +/- each other to allow for some tilt of the hand. This has been assessed to be the most effective range since it allows enough range of movement that the hand does not feel stiffened in order to be read but maximises the integrity of the hand data. This is especially important in this project since it is quite possible that a sufferer of rheumatoid arthritis will have difficulty orienting their hand to an exact location in order for the system to assess it. Using this method, the user is afforded quite a lot of freedom of movement.
The base left and base right values of an individual finger may have different Y axis values since the hand can be tilted to many different orientations and the system will still detect it and analyse the data. To overcome this, the finger width must be determined by using Pythagoras’ theorem to determine the distance between the two points.
47
Y Y
[A]
X
Figure 12: Finger width Pythagoras method
X
[B]
As seen in Figure 12 the distance on the X axis between the two points can be determined as
“B”. When the distance on the Y axis is also determined we can use it as “A” and find “C” through
Pythagoras’ theorem 3 : 𝑐
2
= 𝑎
2
+ 𝑏
2
.
A similar method is implemented in order to determine the height of a given finger. Since
finger point is not an adequate reference to the height of the finger from its base. For this reason,
the Pythagorean theorem is again utilised in the following fashion detailed in Figure 13.
[A]
[C]
[B]
Figure 13: Finger height measurements
Since we have already established finger width, we can half it and use it to determine a base
value for a triangle as shown in Figure 13.
3 Pythagorean theorem - http://en.wikipedia.org/wiki/Pythagorean_theorem
48
To realise the intent of determining the finger widths as demonstrated in Figure 12, the code
below is implemented after establishing each finger.
This will return the width of the current finger so that it can be recorded in the database. By taking getting the difference between the “BaseRight” and “BaseLeft” values we can construct a base line for the triangle to be used for the Pythagorean theorem. The “B” line referred to in this code snippet is the straight line distance between the lower part of the triangle and the top; effectively giving the second side of the triangle. The “Math.Abs()” function here is the method for returning an absolute value which is part of the Math library of C#. The purpose of this is to always return the difference between the two values, instead of a negative value which may happen if the hand is presented in an orientation other than that which the system was intended to handle. code:
In order to find the height of the finger as noted in Figure 13 we implement the following
As part of a function to return the height of the finger, the purpose of this code is to do so no matter the orientation of the hand. Were the hand in an ideal orientation, the distance from base to tip on the Y axis would be suitable for determining the finger’s height. However, since the fingers can be oriented at many angles diverging from the palm of the hand, this function utilises the midpoint of the finger to construct a triangle to use the proven method noted above. The midpoint is constructed by adding the two co-ordinate values of the “BaseLeft” and “BaseRight” and dividing it by 2. From here the triangle is created in a similar fashion to that utilised above.
49
6.3 Monitoring exercises
The user can utilise the system to perform and monitor exercises in their hands. The data recorded from these exercises is potentially useful to a physician and is therefore recorded to be viewed at a later date. The exercises performed can be on either hand, (which the user must specify before proceeding) and includes determining maximum flexion on the finger muscles. This is assessed by the user stretching their hand to full extension, the system beginning a timer and then prompting the user to flex their fingers inward again.
Figure 14: Hand at full flexion
Figure 14 shows the hand recognition working even when no fingers have been found. This
allows for the exercise code to check for the presence of the fingers at extension and when they have not been found it but the hand is still visible it establishes that the full hand has been clenched into a fist.
Figure 15: Exercise time taken message box
The time taken to perform this exercise as well as the maximum range of motion is potentially useful in establishing a treatment plan for the patient and is stored with the rest of the user’s information in the database.
50
To implement the functionality required for this process, we use the “newDataAvailable” listener which is called each time a frame is passed from the Kinect sensor. In order to ensure the functions such as starting a timer and stopping it are only performed when the hand is visible
(irrespective of how many digits are visible), the following code is executed first.
For the code to work properly, it must assess how many fingers are still being picked up by the sensor. Since the user can take any length of time to begin movement, it makes little sense to begin a timer on key press. Instead, the total finger count is checked upon each received frame. If the count is still 5, the user has yet to begin movement and thus, the timer is restarted.
However, should the hand be visible and the number of digits showing be zero, we can interpret that the user has fully clenched their hand.
This code shows the method whereby the timer is allowed it to terminate, thus recording the duration of the flexion and extension in the user’s hand muscles. The time taken is then displayed to the user and subsequently recorded in the database.
51
Displaying data on screen
One additional aspect of displaying the information constructed by the system involves accessing UI elements while the finger recognition code is running. The code which accepts the new frames from the sensor runs in a separate system thread, parallel to the main interface code. As a result, accessing UI elements from within the finger recognition functions is not as simple as referring to the objects and changing their properties. In order to access the UI thread we must invoke the thread’s dispatcher which allows the code to move to that thread to perform an operation before moving back to continue the finger functions. This is demonstrated in the code below.
This code allows the UI updating operations to be performed within the finger recognition thread. Since accessing any UI element and invoking the dispatcher leaves the current thread, it is unnecessary leave the thread for each piece of UI to be updated independently. Once the dispatcher has accessed the UI, all operations to be performed on it can be done using the “resultsGrid” control’s dispatcher.
52
6.4 Sampling information
The information received from the Kinect can be variable at different distances over a period of time; i.e. at 50cm over a period of 30 frames there may be a variance of around 10% in the information received due to a combination of the hand detection algorithm and the method wherein the Kinect senses depth data. To overcome this, a sampling method for the data has been implemented.
By sampling data retrieved from the fingers over 10 frames it was found that the variance decreased in readings. The level of accuracy this provided was determined to be adequate on the testing device (Xbox Kinect sensor) and would be more than enough on the more powerful Kinect for
Windows device.
This code increments over the empty array which will hold the values for sampling. When a set has been filled, it moves on and will only fill the next when a new frame has been passed for analysis. Once all the samples have been filled, we get the totals and then find the mean values for each; as seen in the code below.
As seen above, the array “Sum()” function is used to total all the values in the sampling array. With this found, the total is divided by 10 (the size of the sample) and then rounded to 4 decimal points in order to maintain readability without sacrificing accuracy.
53
6.5 Graphical user interface (GUI)
The interface created to guide the user through the measurement and exercising process was constructed using the form designs shown in section 5. The main intent for the design of the interface was to create a functional yet simple design which would be as easy to utilise as possible for the user.
Navigation
The system is built on the Windows Presentation Foundation which allows a set of specialised form tools for creating an interface. One such tool is the “Navigation Window”. Through the use of this form control we can implement one singular “Main Window” and from there direct the user to any of the numerous pages. The navigation window is constructed using XAML and is shown in the following code segment.
This window control contains the background image information utilised in all of the other pages of the system. It also allows typical form aspects such as title, backwards and forwards navigation and starting location to be set throughout the entire system. As seen on this control, the
“Source” property is the one which is manipulated in order to point the system to different pages.
With the NavigationWindow established, we can switch out the “Source” property and use any of the implemented “Page” objects to allow the user to navigate through the system. An example of a single “page”
54
Login Page
Using a simple username and password textbox combination the login screen is
implemented as shown in Figure 16.
Figure 16: Login screen
Using the input of a username, a function requests a record from the database where this value appears. This value is case sensitive and will only return a valid record if the input exactly matches a record in the database. When this has been found, the password which is also part of this record is assessed to determine if it matches the user’s input. This is done using an MD5 hashing class 4 . This is a one-way hashing algorithm which creates an encoded version of the user’s password which can be compared with the encoded one held in the database.
4 MD5 algorithm class http://msdn.microsoft.com/en-us/library/system.security.cryptography.md5.aspx
55
This “GetMd5()” function is used to create a string from the MD5 object and the user inputted password string. The code for this function is located below and allows the string located in the database to be directly compared to the MD5 output from the user’s entry.
Should the user enter an incorrect combination, they will be prompted with an error which
will allow them to re-enter their information - Figure 17.
Figure 17: Login information error
However, if this information is correct, the navigation object will be used to direct the user to the next stage of the system; the “History” page. First, an instance of the “History” class is created and then passed to the “NavigationService” class.
Since this creates a new instance of the page, all user properties on this page must be set before navigating away from the login page. Here, the users ID value is passed to the history page as this will be used to find and store data in the subsequent pages.
56
History Page
For the history page, two “DatePicker” controls have been implemented. This allows the
user to select a date range and view the appropriate data. An example of this is seen in Figure 18.
Figure 18: History page with date pickers
Since both selections in the DatePicker controls can be any date, the system must check to ensure that the date range implied is valid. This means checking that the date given by the “to” control is greater than the “for” DatePicker.
This code will determine that the selected date range is valid and will allow the actions to
57
Figure 19: Invalid date selection
With a valid range selected, the user will now see the “gridView” control. This control is made available to the WPF project by using a “Windows Form Host” wrapper control. This piece of
XAML code enables this grid to be shown and interacted with as part of the native system.
By giving these namespace references the additional wrapper is now accessible as shown in the following code.
We can now implement this grid within the history page to display sortable information to the user. This information is all of the values that are saved to the database elsewhere in the system
and are relevant to the current user - Figure 20.
Figure 20: DataView control filled with sample data
58
The table in Figure 20 is populated with the results from a function which retrieves data
from the MySQL database. In order to match the values coming out of the date selections with the format in which MySQL accepts dates, an additional function was used. The code below converts the string-based date property of the DatePicker into the MySQL “YYYY/MM/DD HH:MM:SS” format 5 .
Finally, using this information we are able to request the desired information from the database. With this data returning in a “MySQL Reader” object; we iterate through the returned rows to populate the DataView.
Each record in the “reader” object is here used to create a new “DataGridViewRow” object which is then used in the implementation of the DataView.
5 Date/Time format as shown in the MySQL documentation http://dev.mysql.com/doc/refman/5.1/en/datetime.html
59
Measurements Page
The page which contains the interface for measuring finger dimension data is shown in
Figure 21. While containing selections for hand measurements it also contains navigation back to
other sections of the system.
Figure 21: Finger measurements page
The “Begin Test” button shown in this page will begin the finger measurement process.
Before it starts the functions necessary to measure the hand, several checks are performed to make sure that the system is accepting all the necessary data. Firstly, it will establish whether or not the user has selected a “Hand to test” so that the fingers on the hand can be correctly identified later. If
this selection has not been made, the user is prompted with an error box as shown in Figure 22.
60
Figure 22: Hand selection error
With this check completed, the system must also ensure that the Kinect sensor is attached to the testing computer. However, the OpenNI framework has no way of polling the Kinect to ensure its presence. As a result, the first indication that it is not present is when the OpenNI framework attempts to access the data coming back from the device. In order to combat this eventuality, a
“try..catch” method was implemented around the code which would cause the error. In the eventuality shown in the code below, this error is gracefully caught and will throw an error while allowing the system to continue running.
In this fashion, the system will wait for the Kinect to be attached and will still run if it is not present at run-time but added later, allowing for plug-and-play functionality. In order to present this
error to the user, the error box in Figure 23 is employed.
Figure 23: Kinect not connected error
With the necessary error checking completed, the system can begin to show information to
the user in the form of the raw depth image previously shown in Figure 10. From here, the finger
recognition and dimension determination begins.
61
Section 6.2 notes the process whereby the system will establish whether the hand is tilted and if the data being read from it is valid. In order to convey this process to the user, in the event that their hand is tilted more than the system can compensate for, they will be prompted with the
Figure 24: Hand orientation notice
As shown previously, the location of the finger on the X axis is used along with the user selection of “hand to be tested” to determine which finger is which in the hand object. Furthermore, the sampling method documented in section 6.4 is utilised here in order to ensure the accuracy of the information is as high as it can be with the sensor data. When the data is constructed by the system it is displayed on screen for the user to see
Figure 25: Finger measurements results
Here, the user is given the option to save the data if they feel it is representative of the dimensions of their hand. If the data is determined to be flawed; i.e. if the user has placed their hand at an incorrect range and the results have been skewed because of it - the user may clear the data and attempt to run the measurements again.
62
Exercise Page
The final page object in the system is the exercise page. It is here that the user will be able to perform flexion and extension exercises with the guidance of the system and have the data from these exercises be recorded for reference at a later date.
Figure 26: Exercise page
Figure 26 shows the exercise page. This page has all of the same error checking documented
for the measurements page. It utilises the functionality documented in section 6.3 to determine whether the user has completed the exercises they are being prompted to follow. When the actions suggested by the system have been completed, their information is once again saved to the database and can be viewed at a later date.
63
These database interactions are all dependent on the database implemented for the system.
The database is based on the designs proposed in section 5 and consists of a MySQL database connecting with the C# system via the Connector/NET plugin. The MySQL database has two connected tables, “readings” which stores the records of the finger dimensions and “users” which contains the users allowed access to the system. These two tables are shown in their relational form
Figure 27: MySQL table design view with relations
Connecting to this database is handled as part of the “DatabaseConnection” class. Here, information that is needed to access the server and the database are stored. This is also where values would be changed should the system be scaled up and moved to a production server and database environment.
With the “SERVER” property in the string set to “localhost” we are able to implement a local testing environment. However, in a production environment, this would be changed to the IP address of the server where the database is located; i.e. an IP address such as “203.0.113.0”.
64
7. Testing
Thorough testing will ensure the system is implemented and functions the way it was designed to. The purpose of this section is to document that each area of functionality in the system performs its task effectively and in a manner expected of a well-finished product. While ensuring the system meets requirements is important, it is also vital that any incorrect data, crashes, undesired functions or general erroneous behaviour is eliminated from the end product.
7.1 Functional tests
Functional testing refers to the determination that the specified functionality of the system works as documented. This implies that each of the processes that were designed in section 5 will work effectively in an environment with a user who is unfamiliar with the system. The end result will be that any elements of the system which are designed for user interaction should work in the way they are supposed to and without causing faults or crashes (Basili & Selby, 1987).
Initially, test-cases are used to establish typical usage patterns and assess the system based
on these interactions. These test-cases can be found in Appendix 1. These should indicate any errors
or issues the system has and allow them to be fixed.
Through these test-cases several areas of functionality which produced unexpected results were fixed. One such area was the lack of proper control over absent internet connectivity. In this case, were the system not allowed internet connectivity it would cause the system to crash without prompting the user and gave no indication of what was causing the problem. Now, the user is presented with an error box detailing the problem so that they may provide connectivity and use the system properly.
Next, tests were performed to document the data readings taken by the sensor for each finger. These tests assessed the real-world dimensions of the digits on each finger independently and then compared these to the readings output by the system. The tests were then performed at optimal range, too close to the sensor and then far way to determine the scale of results depending
on distance to the sensor. The results of these tests can be found in Appendix 2.
65
7.2 Non-functional tests
Non-functional tests are ones which are dependent on factors such as system scaling or usage constraints which are not directly related to user input. Testing system output compared to real-world, physical values is an important factor in these tests. The system Is used to determine height and width of digits on a specified hand.
To compare this set of results, physical measurements were taken of each digit using a ruler.
These results were tabulated as “expected” results and measured against results output by the
system. The results for these tests are shown in Appendix 2.
The results of these tests are put together as a series of graphs in order to visually display the effect of range on the data taken from the sensor. The first set of results, measuring the width of
digits on the right hand, is shown in Graph 1.
Width - right hand
3,5
3
2,5
2
1,5
1
0,5
0
50 >
60
50 >
60
50 >
60
50 >
60
50 >
60
60 >
70
60 >
70
60 >
70
60 >
70
60 >
70
70 >
90
70 >
90
70 >
90
70 >
90
70 >
90
Expected Actual
Graph 1: Width of digits on right hand
As can be seen the results from this test, system readings vary only slightly at the optimum range of 50 > 60cm from the sensor. With some outliers, the readings remain consistent until outer limits of range at ~80cm. Furthermore, as a result of this test it can be determined that most variance occurs within the recognition of the pinky finger on the right hand. The results of this test remained more constant than were initially expected to be. Consequently, this adds to the integrity of the data at all readable ranges from the Kinect sensor.
66
Next tested was the width measurements taken of the left hand. Once again, the physical measurements of the left hand were taken and compared against the width measurements
established through the system. These were documented in a table which can be found in Appendix
2. The variation of these results is shown on Graph 2.
Width - left hand
6
5
4
3
2
1
0
50 >
60
50 >
60
50 >
60
50 >
60
50 >
60
60 >
70
60 >
70
60 >
70
60 >
70
60 >
70
70 >
90
70 >
90
70 >
90
70 >
90
70 >
90
Expected Actual
Graph 2: Width of digits on left hand
Variation in these results is similar to the results shown on the right hand. As can be seen, at optimal range the data interpreted by the system is very reliable. With the distance between the sensor and hand increasing, so does the level at which the results fluctuate. Once more, the readings are most varied in the mind range (60 > 70cm). This was found to be true of the readings for the widths on the right hand also.
When the variations in the digit widths had been determined, the next step was to compare the results of the finger height measurements. This was done by measuring the fingers on each hand in a similar fashion to what the system is intended to do. By taking the centre point of the digit then using a ruler to go from base to fingertip, a physical, real-world value was determined. This value was then laid out in a table and used for comparison with system-generated values. The table
holding these values is contained in Appendix 2.
First tested was again the right hand. Similar to the width tests, the hand was placed at different distances from the sensor to test effects of distance on reading accuracy. The values from
the right hand are shown in Graph 3.
67
Height - right hand
8
7
6
5
4
3
2
1
0
50 >
60
50 >
60
50 >
60
50 >
60
50 >
60
60 >
70
60 >
70
60 >
70
60 >
70
60 >
70
70 >
90
70 >
90
70 >
90
70 >
90
70 >
90
Expected Actual
Graph 3: Height of digits on right hand
While the variance in width measurements remained relatively consistent throughout the distance tests, finger height measurement showed much more varied results. As seen in these results; at optimal range, digit height is established very accurately. However, these tests prove that for digit height, distance is a very important factor.
These tests must be compared with the results of the left hand testing in order to be
substantiated. Graph 4 shows the results of these tests.
Height - left hand
8
7
6
5
4
3
2
1
0
50 >
60
50 >
60
50 >
60
50 >
60
50 >
60
60 >
70
60 >
70
60 >
70
60 >
70
60 >
70
70 >
90
70 >
90
70 >
90
70 >
90
70 >
90
Expected Actual
Graph 4: Height of digits on left hand
68
Through this second set of height results, we can determine that the behaviour of the first test is replicated. As the distance from the sensor increases, the data variance increases. This variation becomes very high at the extreme ranges such as 70 > 90cm and renders information invalid.
From these tests we can determine that the optimal range for the Kinect when finding digit height falls between 50 > 60cm distance from the device.
Figure 28 shows this range in a visual diagram. Within a range of 0 to 50cm, the information
from the Kinect sensor is not efficient enough to rely in for accurate data. At 50cm the data becomes usable and reliable for accurate readings. However, as the distance from the Kinect increases, the data becomes less and less accurate; finally resulting in the data being unusable again.
Figure 28: Kinect optimal range
Lastly, the ability for the system to determine individual fingers was tested. The aim of this test is to ensure that a)
The system can pick up a hand without needing to identify finger elements (i.e. a fist) b) The system can pick up a hand showing only a sub-set of fingers (i.e. a raised index finger)
To accomplish these tests, the measurement process was run with the right and left hand testing scenarios. Through this raw data window we can determine when the hand has been recognised by the system as it overlays a cluster image on the raw data. Furthermore, when a finger has been recognised, a “fingertip point” is added to the overlay. By monitoring this when the sensor is presented with a hand clenched into a fist and a hand which presents only the index and middle
69
fingers, we can assess whether the system will work with only partial readings. __ shows this functionality being tested.
Figure 29: Hand clenched into a fist
It was found that the system worked very efficiently in determining that a hand is present even without all of the fingers showing. This functionality is utilised in the hand exercise section to establish the time a user takes to complete tasks. This testing validates the readings returned by the exercise section of the system.
70
8. Evaluation
With the system completed, an evaluation can be conducted. The level to which the current system meets the requirements specified in section 3, as well as how the design was realised in implementation will be assessed.
8.1 Evaluation of functional requirements
The functional requirements documented in chapter 3 are listed below in Figure 30. What
follows is an assessment of how each requirement was met.
ID Functional requirement Success level
1 Determine hand measurements of the patient Succeeded
2 Determine initial joint angles at rest
3 Monitor maximum flexion of joint
4 Monitor maximum extension of joint
5 Assess time taken to perform exercise
Failed
Failed
Failed
Succeeded
6 Establish connection to database to save data Succeeded
7 Give real-time feedback of finger dimensions Succeeded
8 Run on Windows-based computers
Figure 30: Function requirements successes
Succeeded
One key aspect of the system at the time it was design was the ability to carry out measurements of joint angles to determine the range of movements the patient is capable of. Due to limitations in the capabilities of the Kinect sensor it was determined later that this functionality lay outside of the scope of the system. Instead, attention for this system is focused on the hand recognition and digit dimension analysis in order to provide a more functional system.
While the system was in progress, it was determined that the facilities intended for development initially – the Kinect SDK – would prove ineffective at accomplishing the desired goal.
For this reason, the ability to recognise and acknowledge a user’s hand became more of a challenge than initially thought. With the process of hand recognition completed, the remaining requirements were met to a high standard. The finger dimensions are recognised at a variety of ranges from the
Kinect sensor which allows for most users to utilise the device without much direction. Furthermore, implementation of the dimension functionality was constructed in such a manner as to negate the effects of a majority of positioning problems on the part of the user. By a process of sampling,
71
assessing hand and finger orientation and distance, the system is able to factor these aspects and provide an accurate reading of the user’s hand.
The time taken to complete exercises as specified by the system was also implemented to completion. The end result of this requirement is also implemented to a very high level of functionality. In terms of usability, the process does not require the user to “start” a timer. Instead, a timer is begun in the event that their hand starts to move. This extends the functionality, making the system more approachable to the user and more reliable in terms of data for the physician.
All values which are determined by the system are provided to the user in real time as well as storing in a database, completing the last of the functional requirements set out in chapter 3.2.
8.2 Evaluation of non-functional requirements
The non-functional requirements as specified in chapter 3.3 included Interface,
Performance and Operating requirements. Among these, an important factor was the performance requirements. It was specified that the system be able to handle a number of adverse conditions such as poor lighting and range diversity in user location (too close, too far from sensor). In terms of these requirements the system performs well. Since the Kinect itself works with infra-red light rather than standard image recognition, it is functional in very dimly lit scenarios. The only adverse condition in terms of lighting is when the Kinect sensor is introduced to a room with direct sunlight which can interfere with the infra-red recognition. Similarly, the system has been developed in such a manner that the range of user motion is accounted for programmatically; handling poor hand orientation on-the-fly where possible and displaying notices to the user otherwise.
In terms of the operating and performance requirements, the system has been implemented to a high standard in both of these areas. Many of the key components of these requirements were involving ease of use. The system has been implemented using HCI best-practices (as noted in chapter 3.3) and utilises colour schemes and layout which adhere to these practices where possible.
Further, it was determined that the interface design would use custom background and icon sets in order to enhance to professional and approachable feel of the system. These were created in Adobe
Photoshop and integrated into the system; achieving a high level of success in these requirements.
72
8.3 Summary of evaluation
Development of this project presented many obstacles which needed to be overcome in order to complete the system as it was specified. One aspect of development which presented issues at the beginning was determining that the Kinect SDK for Windows would be ineffective at establishing hand and finger recognition in the system. This necessitated the creation of a separate method of finger and hand recognition to be built into the system, independently of the Kinect SDK.
As a result of this functionality being implemented, the validity of the measurements and exercises is enforced. The increased level of understanding required by this process led to other sections of the system being completed to a standard otherwise unlikely to have been achieved.
The system was specified and designed to provide patients and physicians a means of testing joint dimensions and movements of the hand in patients with rheumatoid arthritis. By combining the
Kinect sensor and a software interface, the main principles of the system would be ease-of-use and a high level of functionality.
In using the Kinect sensor, an affordable and commercially available device was used to perform a specialised and highly sensitive task. In doing so, the project succeeded in producing a solution which would be usable by either health professionals or patients using it in their homes.
8.3 Future work and enhancements
Through completion of this project, significant investigation into the uses and possible functions of the Kinect sensor hardware has been conducted. As a result, the possibilities for future development of the system and proposed additional elements have been formulated.
Firstly, it is recommended that any future work on this project be conducted on and utilise the Kinect for Windows hardware. This hardware was released during the lifecycle of this project and has been integrated so that in the presence of the device, the system will perform to a much higher standard. By utilising the additional processing power and higher accuracies of the device across a range off distances, the system can have its capabilities increased each time new hardware iterations are released.
One main feature of the system is that it connects to the database for storing information via the internet. This allows many aspects of development which may be completed in the future.
Since the user interacting with the system and the Kinect sensor saves their data to the web-based database server, all of the information stored there is automatically available for any other webconnected device. Following this, an additional system could be constructed as a web-based portal giving access to the information stored in the database. The patient or physician would then be able
73
to access their personal data from any internet connected device like smartphones, laptops PCs among others. This would also eliminate the need for the patient and physician both to have their own Kinect sensor – they would no longer need the C# system if their objective is to read data only.
An internet-based implementation would also pave the way for a much deeper level of interaction between the patient and physician. With this set up, a physician would be able to add exercises or instructions to a database which the system would then pick up and display to the user.
Thus, a physician could view a patient’s information day-to-day and prompt them with specific actions or tasks in order to highlight key areas of their condition; allowing the physician to build a much more personalised treatment plan.
Furthermore, this type of implementation revolves around a unified data source in the form of a web-based database. With this in mind it is quite easily conceived that further developments could come in the form of any type of internet-connected technology. For example; an app may be developed which allows the physician instant access to patient records and exercise building functionality. Similarly, information may be shared between physicians in the form of URLs – though this would mean dramatically increasing the level of security in the system.
For developments such as these, the database and Kinect system both would need to be safe guarded in order to protect user information. The database may be encrypted and the connections with the system secured, providing the patient peace of mind that their information is safe.
This system computes the dimensions of the digits on a hand in a patient suffering rheumatoid arthritis. In the case of expanding upon the work already completed, there are some areas in which it could be made more effective in the treatment of these patients. One such area is monitoring the resting angles of joints in the hand and comparing to the fully flexed angles of joint movement. While the system handles patient exercises and provides useful data; this could be extended to accommodate the treatment plans provided by a physician. The user could perform joint flexion and extension exercises day to day and over a period of time the physician could build a model representing their range of movement. This would be greatly beneficial to the patient as it would result in a full documentation of their condition, rather than the physician taking measurements weeks apart at different appointments.
74
Appendix 1: Test-Cases
ID Description
1 Is the user made aware that they may have entered incorrect login information?
2 Does the system handle the Kinect not being present?
Test Actions
Attempt log in with invalid details
Expected Result
Error box displayed with notification that combination is invalid
Attempt to measure hand without Kinect
Error box displayed with notification that Kinect is absent
3 Will the system handle a lack of internet connection?
Attempt to access login internet without connectivity
Error box displayed to notify user that internet connection is not present
Actual Result
Error box is displayed until details have been updated
Date Pass/Fail Comments
28/04/2012 PASS
Error box shown; however, will not appear second time if user clicks and Kinect is still absent
Error is not displayed, system crashes when login is attempted
28/04/2012 FAIL
28/04/2012 FAIL
Fixed: will not attempting ensure absence of Kinect cause system to crash
Fixed: system will wait on internet connection before database actions
4 Can user retrieve historical information?
5 Can user sort retrieved historical data
6 Are invalid date ranges excluded from searches?
7 Are valid date ranges which have yet to occur excluded from results?
Select date range and click “Show”
Click on header in
GridView to sort by that property
Select a range where the start is later than the end date
Select a date range in the future
GridView control will be populated with results
List will arrange in either alphabetical or numerical order depending on field
Error box displayed to user to notify them that their selection is invalid
Error box displayed to user to notify them that their date range has not occurred yet
GridView is shown and contains all results.
Each column in GridView sorts properly to arrange by date, size or hand measured
Error box shown until user selects a different range of dates
28/04/2012 PASS
28/04/2012 PASS
28/04/2012 PASS
Error box is not shown; instead, no results are returned. This an acceptable result
29/04/2012 PASS Result was unexpected; it still followed a logical process and provided necessary information to the user
75
ID Description
8 Can user begin an exercise without specifying the hand tested?
9 Can user begin finger dimension measurements without specifying the hand tested?
10 Can user clear results from hand tests and rerun?
11 Can user save their measurements for viewing later?
12 Is user data cleared upon exiting the system
Test Actions
Click “Begin
Exercise” before selecting a hand to be exercised
Click “Begin
Measurements” before selecting a hand to be measured
Click “Clear” when values have been stored from the sampling process
Click “Save” when values have been stored form sampling process
Expected Result
Error box displayed to user to notify them that they have not specified which hand is being exercised
Error box displayed to user to notify them that they have not specified which hand is being measured
Values clear from interface and are reset in the sampling functions so that hand is retested
Values clear from interface and are saved to the database
Exit the program User must re-enter log in to ensure security of information
Actual Result
Error box is displayed. User cannot begin exercise until selection is made
Date Pass/Fail Comments
29/04/2012 PASS
Error box is displayed. User cannot have digits measured until selection is made
29/04/2012 PASS
Values are cleared and hand is retested
Values are left displayed on interface until page is navigated away from. Values are stored in database
User information is only cleared upon system exit. No logout functionality exists
29/04/2012 PASS
29/04/2012 PASS
30/04/2012 PASS
Fixed: clears after saving results
Fixed: interface logout function added to ensure security
76
ID Range
1 50 > 60
2 50 > 60
3 50 > 60
4 50 > 60
5 50 > 60
6 60 > 70
7 60 > 70
8 60 > 70
9 60 > 70
10 60 > 70
11 70 > 90
12 70 > 90
13 70 > 90
14 70 > 90
15 70 > 90
Appendix 2: Sensor data comparisons
Width - right hand
ID Range (cm)
1 50 > 60
2 50 > 60
3 50 > 60
4 50 > 60
5 50 > 60
6 60 > 70
7 60 > 70
8 60 > 70
9 60 > 70
10 60 > 70
11 70 > 90
12 70 > 90
13 70 > 90
14 70 > 90
15 70 > 90
Digit
Thumb
Index
Middle
Ring
Pinky
Thumb
Index
Middle
Ring
Pinky
Thumb
Index
Middle
Ring
Pinky
1.7
2.6
2.3
2.1
1.9
1.7
2.6
2.3
Expected
2.6
2.3
2.1
1.9
2.1
1.9
1.7
Width - left hand
Digit
Thumb
Index
Middle
Ring
Pinky
Thumb
Index
Middle
Ring
Pinky
Thumb
Index
Middle
Ring
Pinky
1.8
2.5
2.1
2.0
1.9
1.8
2.5
2.1
Expected
2.5
2.1
2.0
1.9
2.0
1.9
1.8
Actual
2.5
2.55
2.4
1.79
1.9
2.19
4.89
2.06
1.77
3.27
1.66
3.22
1.65
1.6
1.84
Actual
2.9
2.1
2.2
2.0
2.6
2.8
2.24
1.9
2.16
3.12
2.85
1.71
1.64
1.64
2.13
77
Height – left hand
ID Range (cm)
1 50 > 60
2 50 > 60
3 50 > 60
4 50 > 60
5 50 > 60
6 60 > 70
7 60 > 70
8 60 > 70
9 60 > 70
10 60 > 70
11 70 > 90
12 70 > 90
13 70 > 90
14 70 > 90
15 70 > 90
Height – right hand
ID Range (cm)
1 50 > 60
2 50 > 60
3 50 > 60
4 50 > 60
5 50 > 60
6 60 > 70
7 60 > 70
8 60 > 70
9 60 > 70
10 60 > 70
11 70 > 90
12 70 > 90
13 70 > 90
14 70 > 90
15 70 > 90
Digit
Thumb
Index
Middle
Ring
Pinky
Thumb
Index
Middle
Ring
Pinky
Thumb
Index
Middle
Ring
Pinky
Digit
Thumb
Index
Middle
Ring
Pinky
Thumb
Index
Middle
Ring
Pinky
Thumb
Index
Middle
Ring
Pinky
6.7
7.1
7.4
7.2
6.0
6.0
6.7
7.1
7.4
7.2
6.0
Expected
6.7
7.1
7.4
7.2
6.8
7.0
7.4
7.0
6.1
6.8
Expected
6.8
7.0
7.4
7.0
6.1
7.0
7.4
7.0
6.1
5.3
3.95
5.35
6.1
5.63
3.76
Actual
6.2
6.48
6.8
7.0
6.44
2.85
3.71
3.69
3.69
Actual
6.2
6.7
6.8
7.0
6.1
5.8
5.95
5.77
6.02
5.55
3.81
3.6
3.8
4.0
3.35
78
Appendix 3: Source code
XAML interface code
MainWindow.XAML
< NavigationWindow x : Class ="Hand_Recognition_Interface.MainWindow"
xmlns ="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns : uc ="clr-namespace:CCT.NUI.Visual;assembly=CCT.NUI.Visual"
xmlns : x ="http://schemas.microsoft.com/winfx/2006/xaml"
Title ="Hand Recognition" Height ="725" Width ="905" Source ="Login.xaml"
ResizeMode ="NoResize" WindowStartupLocation ="CenterScreen"
ShowsNavigationUI ="False">
< NavigationWindow.Background
>
< ImageBrush ImageSource ="/Hand_Recognition_Interface;component/bin/background.png" />
</ NavigationWindow.Background
>
</ NavigationWindow >
Login.XAML
< Page x : Class ="Hand_Recognition_Interface.Login"
xmlns : x ="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns ="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
Height ="700" Width ="900">
< Page.Background
>
< ImageBrush ImageSource ="/Hand_Recognition_Interface;component/bin/splash.png" />
</ Page.Background
>
< Grid Loaded ="Grid_Loaded">
< TextBlock Height ="23" HorizontalAlignment ="Left" Margin ="298,201,0,0" Name ="textBlock1"
Text ="Please enter your username and password: " VerticalAlignment ="Top" Foreground ="White" />
< Button Content ="Login" Height ="23" HorizontalAlignment ="Left" Margin ="534,296,0,0"
Name ="btnLogin" VerticalAlignment ="Top" Width ="75" Click ="btnLogin_Click" />
< TextBox Height ="23" HorizontalAlignment ="Left" Margin ="456,230,0,0" Name ="txtUser"
VerticalAlignment ="Top" Width ="153" />
< PasswordBox Height ="23" HorizontalAlignment ="Left" Margin ="456,259,0,0" Name ="txtPass"
VerticalAlignment ="Top" Width ="153" PasswordChar ="*" />
< TextBlock Height ="23" HorizontalAlignment ="Left" Margin ="289,12,0,0" Name ="txtNotice"
Text ="Please note: the Kinect sensor works best in well lit room away from direct sunlight."
VerticalAlignment ="Top" FontSize ="16" Foreground ="#FF003B72" />
</ Grid >
</ Page >
79
HistoryPage.XAML
< Page x : Class ="Hand_Recognition_Interface.HistoryPage"
xmlns ="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns : uc ="clr-namespace:CCT.NUI.Visual;assembly=CCT.NUI.Visual"
xmlns : wfi ="clrnamespace:System.Windows.Forms.Integration;assembly=WindowsFormsIntegration"
xmlns : wf ="clr-namespace:System.Windows.Forms;assembly=System.Windows.Forms"
xmlns : x ="http://schemas.microsoft.com/winfx/2006/xaml"
Title ="HistoryPage"> Height ="700" Width ="900"
< Grid >
< DatePicker Height ="25" HorizontalAlignment ="Left" Margin ="53,100,0,0" Name ="dateFrom"
VerticalAlignment ="Top" Width ="115" />
< DatePicker Height ="25" HorizontalAlignment ="Left" Margin ="244,100,0,0" Name ="dateTo"
VerticalAlignment ="Top" Width ="115" />
< Label Content ="to" Height ="28" HorizontalAlignment ="Left" Margin ="193,100,0,0"
Name ="label1" VerticalAlignment ="Top" />
< Button Content ="Exercise hand" Height ="43" HorizontalAlignment ="Left"
Margin ="242,33,0,0" Name ="btnMeasure" VerticalAlignment ="Top" Width ="161"
Click ="btnExercise_Click"/>
< Button Content ="Take new measurements" Height ="43" HorizontalAlignment ="Left"
Margin ="51,33,0,0" Name ="btnExercise" VerticalAlignment ="Top" Width ="161"
Click ="btnMeasure_Click" />
< Button Content ="Show readings" Height ="28" Margin ="0,100,0,0" Name ="btnShow"
VerticalAlignment ="Top" Click ="btnShow_Click" Width ="90" />
< wfi : WindowsFormsHost Height ="400" Margin ="12,145,12,155" Visibility ="Hidden"
Name ="wfhContainer">
< wf : DataGridView x : Name ="gridResults" ColumnHeadersHeightSizeMode ="AutoSize"
AllowUserToAddRows ="False"
AllowUserToDeleteRows ="False" AllowUserToResizeRows ="False"
AllowUserToResizeColumns ="False"
AutoSizeColumnsMode ="Fill" Visible ="True" />
</ wfi : WindowsFormsHost >
</ Grid >
</ Page >
80
ExercisePage.XAML
< Page x : Class ="Hand_Recognition_Interface.ExercisePage"
xmlns ="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns : uc ="clr-namespace:CCT.NUI.Visual;assembly=CCT.NUI.Visual"
xmlns : x ="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns : mc ="http://schemas.openxmlformats.org/markup-compatibility/2006"
Height ="700" Width ="900" Title ="Exercise">
< Grid >
< uc : WpfVideoControl Name ="videoControl" Width ="640" Height ="480"
HorizontalAlignment ="Left" Margin ="23,160,0,0" VerticalAlignment ="Top" BorderBrush ="#FF2D2D2D"
BorderThickness ="1" Background ="White" />
< Button Content ="Begin Exercise" Height ="38" HorizontalAlignment ="Left"
Margin ="462,116,0,0" Name ="btnDepth" VerticalAlignment ="Top" Width ="201" Click ="btnDepth_Click"
/>
< TextBlock Height ="23" HorizontalAlignment ="Left" Margin ="609,16,0,0" Name ="txtNotice"
Text ="" VerticalAlignment ="Top" />
< GroupBox Header ="Please select a hand to exercise" Height ="55"
HorizontalAlignment ="Left" Margin ="33,99,0,0" Name ="groupBox1" VerticalAlignment ="Top"
Width ="284" Foreground ="White" FontWeight ="Bold">
< Grid >
< RadioButton Content ="Left hand" Height ="16" HorizontalAlignment ="Left"
Margin ="6,11,0,0" Name ="radioLeft" VerticalAlignment ="Top" Foreground ="White" />
< RadioButton Content ="Right hand" Height ="16" HorizontalAlignment ="Left"
Margin ="167,11,0,0" Name ="radioRight" VerticalAlignment ="Top" Foreground ="White" />
</ Grid >
</ GroupBox >
< Button Content ="History" Height ="43" HorizontalAlignment ="Left" Margin ="51,33,0,0"
Name ="btnMeasure" VerticalAlignment ="Top" Width ="161" Click ="btnMeasure_Click" />
< Button Content ="Take new measurements" Height ="43" HorizontalAlignment ="Left"
Margin ="242,33,0,0" Name ="btnExercise" VerticalAlignment ="Top" Width ="161"
Click ="btnExercise_Click" />
</ Grid >
</ Page >
81
DataWindow.XAML
< Page x : Class ="Hand_Recognition_Interface.DataWindow"
xmlns ="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns : uc ="clr-namespace:CCT.NUI.Visual;assembly=CCT.NUI.Visual"
xmlns : x ="http://schemas.microsoft.com/winfx/2006/xaml"
Title ="DataWindow" Height ="700" Width ="900" Foreground ="Black">
< Grid >
< uc : WpfVideoControl Name ="videoControl" Width ="640" Height ="480"
HorizontalAlignment ="Left" Margin ="23,160,0,0"
VerticalAlignment ="Top" BorderBrush ="#FF2D2D2D"
BorderThickness ="1" Background ="White" />
< Button Content ="Begin Test" Height ="38" HorizontalAlignment ="Left" Margin ="462,116,0,0"
Name ="btnDepth" VerticalAlignment ="Top" Width ="201" Click ="Begin" />
< GroupBox Header ="Please select a hand to test" Height ="55" HorizontalAlignment ="Left"
Margin ="33,99,0,0" Name ="groupBox1" VerticalAlignment ="Top" Width ="284" Foreground ="White"
FontWeight ="Bold">
< Grid >
< RadioButton Content ="Left hand" Height ="16" HorizontalAlignment ="Left"
Margin ="6,11,0,0" Name ="radioLeft" VerticalAlignment ="Top" Foreground ="White" />
< RadioButton Content ="Right hand" Height ="16" HorizontalAlignment ="Left"
Margin ="167,11,0,0" Name ="radioRight" VerticalAlignment ="Top" Foreground ="White" />
</ Grid >
</ GroupBox >
< Grid Visibility ="Visible" Name ="resultsGrid">
< Rectangle Height ="285" HorizontalAlignment ="Left" Margin ="681,160,0,0"
Name ="rectangle1" Stroke ="Black" VerticalAlignment ="Top" Width ="200" Fill ="#CDFFFFFF" />
< TextBlock Text ="Width (cm)" TextWrapping ="Wrap" Height ="46" TextAlignment ="Right"
HorizontalAlignment ="Left" Margin ="765,188,0,0" Name ="labelWidth" VerticalAlignment ="Top"
FontSize ="16" Width ="53" />
< TextBlock Text ="Height (cm)" TextWrapping ="Wrap" Height ="46" TextAlignment ="Right"
HorizontalAlignment ="Left" Margin ="820,188,0,0" Name ="labelHeight" VerticalAlignment ="Top"
FontSize ="16" Width ="53" />
< Label Content ="Results" Height ="36" HorizontalAlignment ="Left" Margin ="741,160,0,0"
Name ="labelResults" VerticalAlignment ="Top" FontSize ="20" FontWeight ="Bold" Width ="81" />
< Label Content ="Thumb: " Height ="30" HorizontalAlignment ="Left" Margin ="689,233,0,0"
Name ="label1" VerticalAlignment ="Top" FontSize ="16" />
< Label Content ="Index: " Height ="30" HorizontalAlignment ="Left" Margin ="689,273,0,0"
Name ="label2" VerticalAlignment ="Top" FontSize ="16" />
< Label Content ="Middle: " Height ="30" HorizontalAlignment ="Left" Margin ="689,313,0,0"
Name ="label3" VerticalAlignment ="Top" FontSize ="16" />
< Label Content ="Ring: " Height ="32" HorizontalAlignment ="Left" Margin ="689,353,0,0"
Name ="label4" VerticalAlignment ="Top" FontSize ="16" />
< Label Content ="Pinky: " Height ="32" HorizontalAlignment ="Left" Margin ="689,393,0,0"
Name ="label5" VerticalAlignment ="Top" FontSize ="16" />
82
< Label Content ="" Height ="30" HorizontalAlignment ="Right" Name ="lblThumb"
VerticalAlignment ="Top" FontSize ="16" FlowDirection ="RightToLeft" Margin ="0,233,90,0" />
< Label Content ="" Height ="30" HorizontalAlignment ="Right" Name ="lblIndex"
VerticalAlignment ="Top" FontSize ="16" FlowDirection ="RightToLeft" Margin ="0,273,90,0"/>
< Label Content ="" Height ="30" HorizontalAlignment ="Right" Name ="lblMiddle"
VerticalAlignment ="Top" FontSize ="16" FlowDirection ="RightToLeft" Margin ="0,313,90,0"/>
< Label Content ="" Height ="32" HorizontalAlignment ="Right" Name ="lblRing"
VerticalAlignment ="Top" FontSize ="16" FlowDirection ="RightToLeft" Margin ="0,353,90,0"/>
< Label Content ="" Height ="32" HorizontalAlignment ="Right" Name ="lblPinky"
VerticalAlignment ="Top" FontSize ="16" FlowDirection ="RightToLeft" Margin ="0,393,90,0"/>
< Label Content ="" Height ="30" HorizontalAlignment ="Right" Name ="lblThumb_height"
VerticalAlignment ="Top" FontSize ="16" FlowDirection ="RightToLeft" Margin ="0,233,30,0" />
< Label Content ="" Height ="30" HorizontalAlignment ="Right" Name ="lblIndex_height"
VerticalAlignment ="Top" FontSize ="16" FlowDirection ="RightToLeft" Margin ="0,273,30,0"/>
< Label Content ="" Height ="30" HorizontalAlignment ="Right" Name ="lblMiddle_height"
VerticalAlignment ="Top" FontSize ="16" FlowDirection ="RightToLeft" Margin ="0,313,30,0"/>
< Label Content ="" Height ="32" HorizontalAlignment ="Right" Name ="lblRing_height"
VerticalAlignment ="Top" FontSize ="16" FlowDirection ="RightToLeft" Margin ="0,353,30,0"/>
< Label Content ="" Height ="32" HorizontalAlignment ="Right" Name ="lblPinky_height"
VerticalAlignment ="Top" FontSize ="16" FlowDirection ="RightToLeft" Margin ="0,393,30,0"/>
< TextBlock Height ="23" HorizontalAlignment ="Left" Margin ="474,87,0,0"
Name ="txtNotice" Text ="" VerticalAlignment ="Top" />
</ Grid >
< Button Content ="Clear" Height ="49" HorizontalAlignment ="Left" Margin ="677,459,0,0"
Name ="btnClear" VerticalAlignment ="Top" Width ="92" Click ="btnClear_Click" Visibility ="Hidden" />
< Button Content ="Save" Height ="49" HorizontalAlignment ="Left" Margin ="785,459,0,0"
Name ="btnSave" VerticalAlignment ="Top" Width ="92" Click ="btnSave_Click" Visibility ="Hidden" />
< Button Content ="Exercise hand" Height ="43" HorizontalAlignment ="Left"
Margin ="242,33,0,0" Name ="btnExercise" VerticalAlignment ="Top" Width ="161"
Click ="btnExercise_Click" />
< Button Content ="History" Height ="43" HorizontalAlignment ="Left" Margin ="51,33,0,0"
Name ="btnHistory" VerticalAlignment ="Top" Width ="161" Click ="btnHistory_Click" />
</ Grid >
</ Page >
83
C# code
MainWindow.xaml.cs using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Windows; using System.Windows.Controls; using System.Windows.Data; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Navigation; using System.Windows.Shapes; namespace Hand_Recognition_Interface
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary> public partial class MainWindow : NavigationWindow
{ public MainWindow()
{
InitializeComponent();
}
}
}
84
Login.xaml.cs using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Windows; using System.Windows.Controls; using System.Windows.Data; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Navigation; using System.Windows.Shapes; using System.Security.Cryptography; namespace Hand_Recognition_Interface
{
/// <summary>
/// Interaction logic for Login.xaml
/// </summary> public partial class Login : Page
{ public Login()
{
InitializeComponent();
} private void Grid_Loaded( object sender, RoutedEventArgs e)
{
DatabaseConnection .ConnectToDB();
} private void btnLogin_Click( object sender, RoutedEventArgs e)
{
Array user = DatabaseConnection .FindUser(txtUser.Text);
MD5 md5Hash = MD5 .Create(); string hash = GetMd5Hash(md5Hash, txtPass.Password); if (hash == ( String )user.GetValue(2))
{
//password is confirmed
HistoryPage history = new HistoryPage ();
history.user_id = ( String )user.GetValue(0); this .NavigationService.Navigate(history);
} else
{
//password is invalid
MessageBox .Show( "Invalid username and password combination" , "Notice" ,
MessageBoxButton .OK, MessageBoxImage .Error);
}
} static string GetMd5Hash( MD5 md5Hash, string input)
{
// Convert the input string to a byte array and compute the hash.
85
byte [] data = md5Hash.ComputeHash( Encoding .UTF8.GetBytes(input));
// Create a new Stringbuilder to collect the bytes
// and create a string.
StringBuilder sBuilder = new StringBuilder ();
// Loop through each byte of the hashed data
// and format each one as a hexadecimal string.
for ( int i = 0; i < data.Length; i++)
{
sBuilder.Append(data[i].ToString( "x2" ));
}
// Return the hexadecimal string.
return sBuilder.ToString();
}
}
}
HistoryPage.xaml.cs using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Windows; using System.Windows.Controls; using System.Windows.Data; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Navigation; using System.Windows.Shapes; using System.Windows.Forms; using System.Data; using MySql.Data.MySqlClient; namespace Hand_Recognition_Interface
{
/// <summary>
/// Interaction logic for HistoryPage.xaml
/// </summary> public partial class HistoryPage : Page
{ public string user_id; public HistoryPage()
{
InitializeComponent();
} private void btnMeasure_Click( object sender, RoutedEventArgs e)
{
86
DataWindow dw = new DataWindow ();
dw.user_id = user_id; this .NavigationService.Navigate(dw);
} public String formatDateForMySql( String date)
{ return date.Substring(6, 4) + "/" + date.Substring(3, 2) + "/" + date.Substring(0, 2) + " " + date.Substring(11, 8);
} private void btnShow_Click( object sender, RoutedEventArgs e)
{ if (dateFrom.SelectedDate <= dateTo.SelectedDate)
{
// logic to find entries from the database
btnShow.Content = "Add results to table" ;
btnShow.Width = 115;
//convert dates to format readible in mysql
String from = formatDateForMySql(dateFrom.SelectedDate.ToString());
String to = formatDateForMySql(dateTo.SelectedDate.ToString());
MySqlDataReader reader =
DatabaseConnection .FindReadings(from,to,user_id);
gridResults.ColumnCount = 7;
gridResults.Columns[0].HeaderText = "Hand" ;
gridResults.Columns[1].HeaderText = "Thumb" ;
gridResults.Columns[2].HeaderText = "Index" ;
gridResults.Columns[3].HeaderText = "Middle" ;
gridResults.Columns[4].HeaderText = "Ring" ;
gridResults.Columns[5].HeaderText = "Pinky" ;
gridResults.Columns[6].HeaderText = "Time" ; while (reader.Read())
{
DataGridViewRow newRow = new DataGridViewRow ();
newRow.CreateCells(gridResults); for ( int i = 0; i < reader.FieldCount; i++)
{
newRow.Cells[i].Value = reader.GetValue(i).ToString();
}
gridResults.Rows.Add(newRow);
}
wfhContainer.Visibility = Visibility .Visible;
DatabaseConnection .CloseConnection();
} else
{
System.Windows.
MessageBox .Show( "Please select a valid date range" ,
"Notice: invalid date selection" , MessageBoxButton .OK, MessageBoxImage .Asterisk);
}
} private void btnExercise_Click( object sender, RoutedEventArgs e)
{
ExercisePage ex = new ExercisePage ();
ex.user_id = user_id; this .NavigationService.Navigate(ex);
87
}
}
}
ExercisePage.xaml.cs using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Windows; using System.Windows.Controls; using System.Windows.Data; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Navigation; using System.Windows.Shapes; using CCT.NUI.Core; using CCT.NUI.Core.OpenNI; using CCT.NUI.Core.Video; using CCT.NUI.Visual; using CCT.NUI.HandTracking; using CCT.NUI.KinectSDK; using CCT.NUI.Core.Clustering; using System.Timers; using System.Diagnostics; namespace Hand_Recognition_Interface
{
/// <summary>
/// Interaction logic for ExercisePage.xaml
/// </summary> public partial class ExercisePage : Page
{ private IDataSourceFactory dsFactory; private IClusterDataSource cluster; private IHandDataSource hand; private IImageDataSource rgbImage; private IImageDataSource depthImage; private int handToTest; private Stopwatch timer = new Stopwatch (); public string user_id; public ExercisePage()
{
InitializeComponent();
} private void btnDepth_Click( object sender, RoutedEventArgs e)
{ try
{ if ((radioLeft.IsChecked == true ) || (radioRight.IsChecked == true ))
{
88
if (radioLeft.IsChecked == true ) handToTest = 1; else if (radioRight.IsChecked == true ) handToTest = 0;
dsFactory = new OpenNIDataSourceFactory ( "config.xml" );
cluster = dsFactory.CreateClusterDataSource( new
CCT.NUI.Core.Clustering.
ClusterDataSourceSettings { MaximumDepthThreshold = 900 });
hand = new HandDataSource (dsFactory.CreateShapeDataSource(cluster, new CCT.NUI.Core.Shape.
ShapeDataSourceSettings ()));
hand.NewDataAvailable += new
NewDataHandler < HandCollection >(hand_NewDataAvailable);
ProcessDepth();
hand.Start(); //begin analysing hand outline data
OverlayImageData(); //overlay image processing data
btnDepth.Visibility = Visibility .Hidden;
} else
{
MessageBox .Show( "Please select a hand to be tested" , "Notice: invalid hand selection" , MessageBoxButton .OK, MessageBoxImage .Asterisk);
}
} catch (OpenNI.
GeneralException ) { MessageBox .Show( "Kinect not connected" );
} // this catch statement will ensure Kinect is connected before attempting to use it
} void hand_NewDataAvailable( HandCollection data)
{ for ( int index = 0; index < data.Count; index++) // for each hand do the following
{ var hand = data.Hands[index]; if (hand.FingerCount == 0) //if user has clenched whole hand
{ if (timer.IsRunning == true )
{
timer.Stop();
MessageBox .Show( "Time to flex in seconds:
" +(( float )timer.ElapsedMilliseconds/1000) + "sec" );
// use message box later to display time to see if it works ok
}
} else if (hand.FingerCount == 5)
{
timer.Reset();
timer.Start(); // Start the timer
}
}
} void Exercise_NewDataAvailable( ImageSource img)
{
videoControl.Dispatcher.Invoke( new Action (() =>
{
videoControl.ShowImageSource(img);
}));
89
} void ProcessDepth()
{
depthImage = dsFactory.CreateDepthImageDataSource();
depthImage.NewDataAvailable += new
NewDataHandler < ImageSource >(Exercise_NewDataAvailable); //run method when new data is found on the camera input
depthImage.Start(); //begin analysing depth data
} void OverlayImageData()
{ var layers = new List < IWpfLayer >();
layers.Add( new WpfHandLayer (hand)); //display data showing outline of hand information
layers.Add( new WpfClusterLayer (cluster)); //display data showing image point clustering
videoControl.Layers = layers; //add this information to the video control
} private void btnMeasure_Click( object sender, RoutedEventArgs e)
{
HistoryPage hist = new HistoryPage ();
hist.user_id = user_id; this .NavigationService.Navigate(hist);
} private void btnExercise_Click( object sender, RoutedEventArgs e)
{
DataWindow dw = new DataWindow ();
dw.user_id = user_id; this .NavigationService.Navigate(dw);
}
}
}
DataWindow.xaml.cs using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Windows; using System.Windows.Controls; using System.Windows.Data; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Shapes; using CCT.NUI.Core; using CCT.NUI.Core.OpenNI; using CCT.NUI.Core.Video; using CCT.NUI.Visual; using CCT.NUI.HandTracking; using CCT.NUI.KinectSDK; using CCT.NUI.Core.Clustering; using MySql.Data; using MySql.Data.MySqlClient;
90
namespace Hand_Recognition_Interface
{
/// <summary>
/// Interaction logic for DataWindow.xaml
/// </summary> public partial class DataWindow : Page
{ private IDataSourceFactory dsFactory; private IClusterDataSource cluster; private IHandDataSource hand; private IImageDataSource rgbImage; private IImageDataSource depthImage; public string user_id; double [] thumb_average = new double [10]; double [] index_average = new double [10]; double [] middle_average = new double [10]; double [] ring_average = new double [10]; double [] pinky_average = new double [10]; double thumb_width; double index_width; double middle_width; double ring_width; double pinky_width; double thumb_height; double index_height; double middle_height; double ring_height; double pinky_height; double [] thumb_average_height = new double [10]; double [] index_average_height = new double [10]; double [] middle_average_height = new double [10]; double [] ring_average_height = new double [10]; double [] pinky_average_height = new double [10]; private int handToTest; // define whether hand being tested is "left" or
"right" (1, 0) public DataWindow()
{
InitializeComponent(); for ( int i = 0; i < 10; i++ )
{
thumb_average[i] = 0;
index_average[i] = 0;
middle_average[i] = 0;
ring_average[i] = 0;
pinky_average[i] = 0;
thumb_average_height[i] = 0;
index_average_height[i] = 0;
middle_average_height[i] = 0;
ring_average_height[i] = 0;
pinky_average_height[i] = 0;
}
91
} private void Begin( object sender, RoutedEventArgs e)
{ try
{ if ((radioLeft.IsChecked == true ) || (radioRight.IsChecked == true ))
{ if (radioLeft.IsChecked == true ) handToTest = 1; else if (radioRight.IsChecked == true ) handToTest = 0;
btnDepth.Visibility = Visibility .Hidden;
dsFactory = new OpenNIDataSourceFactory ( "config.xml" );
cluster = dsFactory.CreateClusterDataSource( new
CCT.NUI.Core.Clustering.
ClusterDataSourceSettings { MaximumDepthThreshold = 900 });
hand = new HandDataSource (dsFactory.CreateShapeDataSource(cluster, new CCT.NUI.Core.Shape.
ShapeDataSourceSettings ()));
hand.NewDataAvailable += new
NewDataHandler < HandCollection >(hand_NewDataAvailable);
ProcessDepth();
hand.Start(); //begin analysing hand outline data
OverlayImageData(); //overlay image processing data
} else
{
MessageBox .Show( "Please select a hand to be tested" , "Notice: invalid hand selection" , MessageBoxButton .OK, MessageBoxImage .Asterisk);
}
} catch (OpenNI.
GeneralException ) { MessageBox .Show( "Kinect not connected" );
} // this catch statement will ensure Kinect is connected before attempting to use it
} void hand_NewDataAvailable( HandCollection data)
{
//this is hand frame recognition for ( int index = 0; index < data.Count; index++) // for each hand do the following
{ var hand = data.Hands[index];
IList < FingerPoint > fingers = hand.FingerPoints;
FingerPoint thumb = null ;
FingerPoint indexFinger = null ;
FingerPoint middle = null ;
FingerPoint ring = null ;
FingerPoint pinky = null ; if (hand.FingerCount == 5)
{
//full hand has been recognised
//begin assigning fingers float most = 0;
92
if (handToTest == 1){ foreach ( var finger in fingers)
{ if (finger.Fingertip.X > most)
{
thumb = finger;
most = finger.Fingertip.X;
}
}
fingers.Remove(thumb);
most = 0; foreach ( var finger in fingers)
{ if (finger.Fingertip.X > most)
{
indexFinger = finger;
most = finger.Fingertip.X;
}
}
fingers.Remove(indexFinger);
most = 0; foreach ( var finger in fingers)
{ if (finger.Fingertip.X > most)
{
middle = finger;
most = finger.Fingertip.X;
}
}
fingers.Remove(middle);
most = 0; foreach ( var finger in fingers)
{ if (finger.Fingertip.X > most)
{
ring = finger;
most = finger.Fingertip.X;
}
}
fingers.Remove(ring);
most = 0; foreach ( var finger in fingers)
{ if (finger.Fingertip.X > most)
{
pinky = finger;
most = finger.Fingertip.X;
}
}
fingers.Remove(pinky);
most = 0;
} else if (handToTest == 0)
{ float rightmost = 10000;
93
foreach ( var finger in fingers)
{ if (finger.Fingertip.X < rightmost)
{
thumb = finger;
most = finger.Fingertip.X;
}
}
fingers.Remove(thumb);
rightmost = 10000; foreach ( var finger in fingers)
{ if (finger.Fingertip.X < rightmost)
{
indexFinger = finger;
rightmost = finger.Fingertip.X;
}
}
fingers.Remove(indexFinger);
rightmost = 10000; foreach ( var finger in fingers)
{ if (finger.Fingertip.X < rightmost)
{
middle = finger;
rightmost = finger.Fingertip.X;
}
}
fingers.Remove(middle);
rightmost = 10000; foreach ( var finger in fingers)
{ if (finger.Fingertip.X < rightmost)
{
ring = finger;
rightmost = finger.Fingertip.X;
}
}
fingers.Remove(ring);
rightmost = 10000; foreach ( var finger in fingers)
{ if (finger.Fingertip.X < rightmost)
{
pinky = finger;
rightmost = finger.Fingertip.X;
}
}
fingers.Remove(pinky);
rightmost = 10000;
}
//ensure that hand orientation is within 4 % +/- horizontally if ((pinky.BaseLeft.Z <= (thumb.BaseRight.Z + (thumb.BaseRight.Z *
0.04))) &&
(pinky.BaseLeft.Z >= (thumb.BaseRight.Z - (thumb.BaseRight.Z
* 0.04))))
{
94
ChangeTextBox(txtNotice, "" );
thumb_width = FindFingerWidth(thumb);
index_width = FindFingerWidth(indexFinger);
middle_width = FindFingerWidth(middle);
ring_width = FindFingerWidth(ring);
pinky_width = FindFingerWidth(pinky);
//testing finger height code
thumb_height = FindFingerHeight(thumb);
index_height = FindFingerHeight(indexFinger);
middle_height = FindFingerHeight(middle);
ring_height = FindFingerHeight(ring);
pinky_height = FindFingerHeight(pinky);
//end test bool filled = false ; for ( int i = 0; i < 10; i++)
{ if ((thumb_average[i] == 0) && (filled == false ))
{
thumb_average[i] = thumb_width;
index_average[i] = index_width;
middle_average[i] = middle_width;
ring_average[i] = ring_width;
pinky_average[i] = pinky_width;
thumb_average_height[i] = thumb_height;
index_average_height[i] = index_height;
middle_average_height[i] = middle_height;
ring_average_height[i] = ring_height;
pinky_average_height[i] = pinky_height;
filled = true ;
}
} if (thumb_average[9] != 0) // if sample data has been filled
{
// jump out of thread to access UI elements
thumb_width = Math .Round((thumb_average.Sum() / 10), 4);
index_width = Math .Round((index_average.Sum() / 10), 4);
middle_width = Math .Round((middle_average.Sum() / 10), 4);
ring_width = Math .Round((ring_average.Sum() / 10), 4);
pinky_width = Math .Round((pinky_average.Sum() / 10), 4);
thumb_height = Math .Round((thumb_average_height.Sum() /
10), 4);
index_height = Math .Round((index_average_height.Sum() /
10), 4);
middle_height = Math .Round((middle_average_height.Sum() /
10), 4);
ring_height = Math .Round((ring_average_height.Sum() / 10),
4);
pinky_height = Math .Round((pinky_average_height.Sum() /
10), 4);
95
resultsGrid.Dispatcher.Invoke(
System.Windows.Threading.
DispatcherPriority .Normal, new Action ( delegate ()
{
resultsGrid.Visibility = Visibility .Visible;
btnClear.Visibility = Visibility .Visible;
btnSave.Visibility = Visibility .Visible;
lblThumb.Content = Math .Round((thumb_width /
10), 2);
lblIndex.Content = Math .Round((index_width /
10), 2);
lblMiddle.Content = Math .Round((middle_width /
10), 2);
lblRing.Content = Math .Round((ring_width /
10), 2);
lblPinky.Content = Math .Round((pinky_width /
10), 2);
lblThumb_height.Content =
Math .Round((thumb_height / 10), 2);
lblIndex_height.Content =
Math .Round((index_height / 10), 2);
lblMiddle_height.Content =
Math .Round((middle_height / 10), 2);
lblRing_height.Content =
Math .Round((ring_height / 10), 2);
lblPinky_height.Content =
Math .Round((pinky_height / 10), 2);
}));
}
} else
{
ChangeTextBox(txtNotice, "Please adjust hand orientation" );
}
//end finger assigning
} //end if count is 5
}
} void ChangeTextBox( TextBlock txt, String text)
{ if (!txt.Dispatcher.CheckAccess())
{
txtNotice.Dispatcher.Invoke(
System.Windows.Threading.
DispatcherPriority .Normal, new Action ( delegate ()
{
txt.Text = text;
}
));
} else
{
96
txt.Text = text;
}
} double FindFingerWidth( FingerPoint finger)
{ double finger_width = 0; float A = finger.BaseRight.X - finger.BaseLeft.X; float B = Math .Abs(finger.BaseLeft.Y - finger.BaseRight.Y); return finger_width = Math .Sqrt((A * A) + (B * B));
} double FindFingerHeight( FingerPoint finger)
{ double midpointX = (finger.BaseLeft.X + finger.BaseRight.X) / 2; double midpointY = (finger.BaseLeft.Y + finger.BaseRight.Y) / 2; double A = Math .Abs(midpointX - finger.Fingertip.X); double B = Math .Abs(finger.Fingertip.Y - midpointY); double C = Math .Sqrt( (A * A) + (B * B) ); return C;
} void Login_NewDataAvailable( ImageSource img)
{
videoControl.Dispatcher.Invoke( new Action (() =>
{
videoControl.ShowImageSource(img);
}));
} void ProcessRGB()
{
rgbImage = dsFactory.CreateRGBImageDataSource();
rgbImage.NewDataAvailable += new
NewDataHandler < ImageSource >(Login_NewDataAvailable); //show RGB image
rgbImage.Start();
} void ProcessDepth()
{
depthImage = dsFactory.CreateDepthImageDataSource();
depthImage.NewDataAvailable += new
NewDataHandler < ImageSource >(Login_NewDataAvailable); //run method when new data is found on the camera input
depthImage.Start(); //begin analysing depth data
} void OverlayImageData()
{ var layers = new List < IWpfLayer >();
layers.Add( new WpfHandLayer (hand)); //display data showing outline of hand information
layers.Add( new WpfClusterLayer (cluster)); //display data showing image point clustering
videoControl.Layers = layers; //add this information to the video control
} private void btnClear_Click( object sender, RoutedEventArgs e)
{
lblThumb.Content = "" ;
lblIndex.Content = "" ;
lblMiddle.Content = "" ;
lblRing.Content = "" ;
lblPinky.Content = "" ;
97
lblThumb_height.Content = "" ;
lblIndex_height.Content = "" ;
lblMiddle_height.Content = "" ;
lblRing_height.Content = "" ;
lblPinky_height.Content = "" ;
Array .Clear(thumb_average, 0,thumb_average.Length);
Array .Clear(index_average, 0, index_average.Length);
Array .Clear(middle_average, 0, middle_average.Length);
Array .Clear(ring_average, 0, ring_average.Length);
Array .Clear(pinky_average, 0, pinky_average.Length);
Array .Clear(thumb_average_height, 0, thumb_average_height.Length);
Array .Clear(index_average_height, 0, index_average_height.Length);
Array .Clear(middle_average_height, 0, middle_average_height.Length);
Array .Clear(ring_average_height, 0, ring_average_height.Length);
Array .Clear(pinky_average_height, 0, pinky_average_height.Length);
} private void btnSave_Click( object sender, RoutedEventArgs e)
{
String handToTest_string = "" ; if (handToTest == 1) { handToTest_string = "left" ; } else if (handToTest == 0) { handToTest_string = "right" ; }
DatabaseConnection .SaveReadings(
Convert .ToInt32(user_id),
handToTest_string,
thumb_width,
index_width,
middle_width,
ring_width,
pinky_width,
thumb_height,
index_height,
middle_height,
ring_height,
pinky_height
);
} private void btnHistory_Click( object sender, RoutedEventArgs e)
{
HistoryPage hist = new HistoryPage (); this .NavigationService.Navigate(hist);
} private void btnExercise_Click( object sender, RoutedEventArgs e)
{
ExercisePage ex = new ExercisePage (); this .NavigationService.Navigate(ex);
}
}
}
98
DatabaseConnection.cs using System; using System.Collections.Generic; using System.Linq; using System.Text; using MySql.Data; using MySql.Data.MySqlClient; using System.Windows; using System.Windows.Controls; using System.Windows.Data; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Navigation; using System.Windows.Shapes; namespace Hand_Recognition_Interface
{ class DatabaseConnection
{ private static MySqlConnection connection; public static void ConnectToDB()
{ string MyConString = "SERVER=localhost;" +
"DATABASE=hand_tracker;" +
"UID=hand_system;" +
"PASSWORD=hand_user;" ;
connection = new MySqlConnection(MyConString);
} public static void ReadData()
{
MySqlCommand command = connection.CreateCommand();
MySqlDataReader Reader;
command.CommandText = "select * from readings" ;
connection.Open();
Reader = command.ExecuteReader(); while (Reader.Read())
{ string thisrow = "" ; for ( int i = 0; i < Reader.FieldCount; i++)
thisrow += Reader.GetValue(i).ToString() + "," ;
MessageBox .Show(thisrow);
}
} public static Array FindUser( String user)
{
String [] user_credentials = new String [3];
connection.Open();
MySqlCommand command = connection.CreateCommand();
MySqlDataReader Reader;
command.CommandText = "SELECT `user_id`,`username`,`password` FROM `users`
WHERE `username` = '" + user + "'" ;
99
Reader = command.ExecuteReader(); while (Reader.Read())
{
// 0, user_id
// 1, username
// 2, password for ( int i = 0; i < Reader.FieldCount; i++)
user_credentials[i] = Reader.GetValue(i).ToString();
}
connection.Close(); return user_credentials;
} public static MySqlDataReader FindReadings( String from, String to, String user_id)
{
connection.Open();
MySqlCommand command = connection.CreateCommand();
MySqlDataReader Reader;
command.CommandText = "SELECT
`left_or_right`,`thumb`,`index`,`middle`,`ring`,`pinky`,`timestamp` FROM `readings`
WHERE `user_id` = '" + user_id + "' AND `timestamp` >= '" + from + "' AND `timestamp`
<= '" + to + "'" ;
Reader = command.ExecuteReader(); return Reader;
} public static void SaveReadings( int user_id, String left_or_right, double thumb, double index, double middle, double ring, double pinky, double thumb_height, double index_height, double middle_height, double ring_height, double pinky_height)
{
connection.Open();
MySqlCommand command = connection.CreateCommand();
command.CommandText = "INSERT INTO `readings`
(`thumb`,`index`,`middle`,`ring`,`pinky`,`thumb_height`,`index_height,`middle_height`,
`ring_height`,`pinky_height`,`left_or_right`,`user_id`) VALUES ('"
+ thumb + "','"
+ index + "','"
+ middle + "','"
+ ring + "','"
+ pinky + "','"
+ thumb_height + "','"
+ index_height + "','"
+ middle_height + "','"
+ ring_height + "','"
+ pinky_height + "','"
+ left_or_right + "','"
+ user_id + "'"
+ " )" ;
command.ExecuteNonQuery();
connection.Close();
} public static void CloseConnection()
{ connection.Close(); } } }
100
MySQL dump of database
-- phpMyAdmin SQL Dump
-- version 3.4.5
-- http://www.phpmyadmin.net
--
-- Host: localhost
-- Generation Time: May 02, 2012 at 10:31 PM
-- Server version: 5.5.16
-- PHP Version: 5.3.8
SET SQL_MODE="NO_AUTO_VALUE_ON_ZERO";
SET time_zone = "+00:00";
--
-- Database: `hand_tracker`
--
-- --------------------------------------------------------
--
-- Table structure for table `readings`
--
CREATE TABLE IF NOT EXISTS `readings` (
`id` int(21) NOT NULL AUTO_INCREMENT,
`timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`thumb` double NOT NULL,
`index` double NOT NULL,
`middle` double NOT NULL,
`ring` double NOT NULL,
`pinky` double NOT NULL,
`thumb_height` double NOT NULL,
`index_height` double NOT NULL,
`middle_height` double NOT NULL,
`ring_height` double NOT NULL,
`pinky_height` double NOT NULL,
`left_or_right` enum('left','right') NOT NULL,
`user_id` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=10 ;
-- --------------------------------------------------------
-- Table structure for table `users`
CREATE TABLE IF NOT EXISTS `users` (
`user_id` int(11) NOT NULL AUTO_INCREMENT,
`username` text NOT NULL,
`password` varchar(60) NOT NULL,
PRIMARY KEY (`user_id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=2 ;
101
References
Arthritis: Rheumatoid Arthritis. (2008). Retrieved December 08, 2011, from American Society for
Surgery of the http://www.assh.org/Public/HandConditions/Pages/ArthritisRheumatoidArthritis.aspx
Hand:
Handout on Health: Rheumatoid Arthritis. (2009). Retrieved 12 5, 2011, from National Institute of
Arthritis and Musculoskeletal and Skin Diseases: http://www.niams.nih.gov/Health_Info/Rheumatic_Disease/default.asp
DAS Booklet - Quick reference guide for Healthcare Professionals. (2010, February). Retrieved
December 08, 2011, from National Rheumatoid Arthritis Society: http://www.nras.org.uk/includes/documents/cm_docs/2010/d/das_quick_reference.pdf
GigE Vision for 3D Medical Research. (2010, May 13). Retrieved from Allied Vision Technologies: http://www.alliedvisiontec.com/us/products/applications/application-casestudy/article/gige-vision-for-3d-medical-research.html
Limitations of the Kinect. (2010, 12 17). Retrieved from I Heart Robotics: http://www.iheartrobotics.com/2010/12/limitations-of-kinect.html
Rheumatoid: Hand Exam. (2011). Retrieved December 08, 2011, from Clinical Exam: http://clinicalexam.com/pda/r_hand.htm
Ackerman, E. (2011, 6 17). Microsoft Releases Kinect SDK, Roboticists Cackle With Glee. Retrieved from IEEE Spectrum - Automaton: http://spectrum.ieee.org/automaton/robotics/diy/microsoft-releases-kinect-sdk-roboticistscackle-with-glee
Arnett, F., Edworthy, S., Bloch, D., Mcshane, D., Fries, F., Cooper, N., . . . Hunder, G. (1988). The
American Rheumatism Association 1987 Revised Criteria for the Classification of Rheumatoid
Arthritis. Arthritis & Rheumatism, 315-324.
Ashton, L., & Myers, S. (2004). Serial Grip Testing- Its role In Assessment Of Wrisy And Hand
Disability. The Internet Journal of Surgery, Vol. 5.
Basili, V., & Selby, R. (1987). Comparing the Effectiveness of Software Testing Strategies. IEEE
Transactions on Software Engineering, 1278-1296.
Black, S., Kushner, I., & Samols, D. (2004). C-reactive Protein. The Journal of Biological Chemistry,
48487-48490.
Boehm, B. (1988). A Spiral Model for Software Development and Enhancement. IEEE Computer, 61-
72.
Bradski, G., & Kaehler, A. (2008). Learning OpenCV: computer vision with the OpenCV library.
Sebastapol: O'Reilly Media, Inc.
Carmody, T. (2010, 11 5). How Facial Recognition Works in Xbox Kinect. Retrieved from Wired: http://www.wired.com/gadgetlab/2010/11/how-facial-recognition-works-in-xbox-kinect/
102
Chaczko, Z., & Yeoh, L. (2007). A Preliminary Investigation on Computer Vision for Telemedicine
Systems using OpenCV. Swinburne: 2010 Second International Conference on Machine
Learning and Computing.
Chen, Y., Cheng, T., & Hsu, S. (2009). Ultrasound in rheumatoid arthritis. Formosan Journal of
Rheumatology, 1-7.
Chung, L., Cesar, J., & Sampaio, J. (2009). On Non-Functional Requirements in Software Engineering.
Lecture Notes in Computer Science, 363-379.
Condell, J., Curran, K., Quigley, T., Gardiner, P., McNeill, M., Winder, J., . . . Connolly, J. (2010). Finger
Movement Measurements in Arthritic Patients Using Wearable Sensor Enabled Gloves.
Londonderry: University of Ulster.
Conger, S. (2011). Software Development Life Cycles and Methodologies: Fixing the Old and
Adopting the New. International Journal of Information Technologies and Systems Approach,
1-22. de Kraker, M., Selles, R., Molenaar, T., Schreuders, A., Hovius, S., & Stam, H. (2009). Palmar
Abduction Measurements: Reliability and Introduction of Normative Data in Healthy
Children. The Journal of Hand Surgery, 1704-1708.
Dipietro, L., Sabatini, A., & Dario, P. (2008). A Survey of Glove-Based Systems and Their Applications.
IEEE Transactions On Systems, Man, And Cybernetics—Part C: Applications And Reviews,
461-482.
Eberhardt, K., Malcus-Johnson, P., & Rydgren, L. (1991). The Occurence and Significance of Hand
Deformities in Early Rheumatoid Arthritis. British Journal of Rheumatology, 211-213.
Fees, E. (1987). A method for checking Jamar dynamometer calibration. Journal of Hand Therapy,
28=32.
Fess, E. (1995). Documentation: Essential elements of an upper extremity assessment battery. In J.
Hunter, E. Mackin, & A. Calahan, Rehabilitation of the hand: Surgery and therapy (4th edn)
(pp. 185-214). St. Louis: Mosby.
Fries, J., Spitz, P., Kraines, R., & Holman, H. (1980). Measurement of patient outcome in arthritis.
Arthritis & Rheumatism, 137-145.
Hamilton, A., Balnave, R., & Adams, R. (1994). Grip strength testing reliability . Journal of hand
Therapy, 163-170.
Majithia, V., & Geraci, A. (2007). Rheumatoid Arthritis: Diagnosis and Management. The American
Journal of Medicine, 936-939.
Panayi, G. (2003). What is RA? National Rheumatoid Arthritus Society.
Richards, L., & Palmiter-Thomas, P. (1996). Grip strength measurement: A critical review of tools, methods and clinical utility. Critical reviews in Physical and Rehabilitation Medicine, 87-109.
103
Roman, G. (1985). A Taxonomy of Current Issues in Requirements Engineering. IEEE Computer, 14-
21.
Schofield, P., Aveyard, B., & Black, C. (2007). Management of Pain in Older People. Keswick: M&K
Publishing.
Tölgyessy, M., & Hubinský, P. (2011). The Kinect Sensor in Robotics Education. Bratislava: Slovak
University of Technology.
Wiegers, K. (2003). Software Requirements. Redmond: Microsoft.
Worden, J. (2011). Rheumatoid Arthritis. Retrieved from BBC Health: http://www.bbc.co.uk/health/physical_health/conditions/in_depth/arthritis/aboutarthritis_ rheumatoid.shtml
104
