Assistive Human-Machine Interfaces via
Artificial Neural Networks
Wei Tech Ang and Cameron N. Riviere
The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
Abstract - This paper reports the study and results of modeling and
online compensating of movement disorder stemming from multiple
sclerosis (MS) via artificial neural networks. We trained and tested a
cascade-correlation neural network with Kalman filtering on data
collected from 11 subjects with MS. The test subjects use headcontrolled mouse emulators to move a cursor to a series of random
targets on screen. Simulated real-time testing of the trained neural
networks shows that the networks successfully make the cursor
trajectories of all the 11 subjects less chaotic, and hence more
controllable. The neural networks also reduce the time needed to
reach the targets by an average of 31.8%. The neural network
approach can be easily applied to other human-machine interfaces
such as computer mice and joysticks, or powered wheelchairs. This
technique is also applicable to movement disorders resulting from
certain geriatric diseases such as Parkinson’s disease and essential
tremor.
I. INTRODUCTION
Multiple sclerosis is a serious progressive disease of the
central nervous system, occurring mainly in young adults and
thought to be caused by a malfunction of the immune system.
It leads to the loss of myelin in the brain or spinal cord and
reduces and even loses bodily control. Movement disorders
associated with MS include tremor, myoclonus and
spasticity. These movement disorders are among the most
difficult of MS symptoms to treat, and many do not respond
well to medication [1]. These symptoms can, in the least
severe form, hamper effective control of the patients’
environment, and in more acute cases obscuring most daily
voluntary activities. For many of these patients, the
interference of these symptoms make the use of interfaces
such as computer mice and joysticks or powered wheelchairs
a challenging and daunting task. This reduction in
independence is detrimental to patients’ quality of life, and is
a significant factor contributing to the enormous lifetime cost
of MS, estimated at $2.2 million per patient [2]. Similar
observations could be made regarding other common
diseases such as essential tremor and Parkinson’s disease.
The most well known type of involuntary motion
affecting patients with MS is pathological tremor. Tremor is
defined as any involuntary, approximately rhythmic, and
roughly sinusoidal movement [3]. Tremor stemming from
MS is typically a cerebellar intention tremor. Most types of
tremor are characterized by having a higher frequency band
than voluntary motion. Many of the previous work in the area
of modeling and suppressing erroneous motion during
human-machine control have taken advantage of this
frequency separation. Riley and Rosen [4] have investigated
lowpass filtering, while Prochazka et al. [5] have shown
some success with the approach of closed-loop functional
electrical stimulation. Gonzalez et al. [6] has also proposed a
signal equalizer technique to suppress pathological tremor.
Pledgie et al. [7, 8] have presented methods that suppress the
power of pathological tremors through force feedback and
impedance control. Riviere et al. [9] have developed adaptive
noise canceling of tremor based on a dynamic sinusoidal
model, offering better performance than low-pass filtering,
but distortion of the voluntary motion still has not been
completely eliminated. Performance improvement beyond
this adaptive noise canceller is likely to require modeling the
dynamics of the specific type of tremor treated. Several
others have looked into application of viscous damping to
suppress tremor [10, 11].
Besides tremor, other types of involuntary motions such
as pathological myoclonus can also cause incapacitation of
human motor control. Myoclonic movements are twitches or
jerks resulting from sudden muscle contractions that can
occur alone or in a sequence [12]. Non-tremulous types of
erroneous movement are aperiodic, erratic, unpredictable at
our current state of understanding, and can overlap in
frequency with voluntary motion. Effective online canceling
is therefore difficult.
Efforts to develop assistive interfaces for persons with
the most severe non-tremulous movement disorders have
generally taken the low signal-to-noise ratio as given, and
then followed one of the two approaches. One approach is to
develop interfaces of sufficiently low bandwidth and input
gain such that the tasks are slowed down to a level that the
users are capable of handling them. One example is
specialized keyboards with larger key pitch (effectively a low
input gain) and “sticky key” operation – a keystroke is
registered only if the finger dwells on the key for a few
seconds. Another approach is to develop interfaces to be
controlled with different body parts which may be less
affected by the movement disorder in question, such as head
motion tracker, chin joysticks, mouthsticks, tongue-touch
keypads, electroencephalogram and electrooculogram-based
systems, and an electrogalvanic intraoral access device [13][17].
Although these approaches have often been successful in
many ways, they have intrinsic drawbacks. The low
bandwidth systems are slow and inefficient, while control via
alternative body parts is often awkward. A technique to
specifically estimate and suppress nonlinear In this paper, we
present a neural network approach to simultaneously
modeling and canceling both tremulous and non-tremulous
types of movement disorder. Section II explains the
motivation of the neural network approach and introduces the
characteristics of a cascade-correlation neural network with
Kalman filtering. Section III describes the experimental
methods involve in the training and testing of the neural
networks. Section IV presents the results of our experiments.
Section V discusses the significance of the results and future
work, and finally concluding remarks in Section VI.
collected from 11 subjects with MS, wearing a HeadMaster
PlusTM computer head control system (Prentke Romich
Company, Wooster, OH). The test subjects had no prior
experience in using head controls to operate computers at the
time of the experiment. Each subject underwent an icon
selection exercise where they were asked to move the cursor
into a series of randomly positioned circular target of 30pixel radius, on a 14” screen of 1024 x 768 resolution. A test
was completed when the cursor dwelled in the circle for more
than 500 ms, or when the 10-second time limit expired. The
sampling rate of the system is 32 Hz [17].
II. ERROR MODELING AND CANCELING VIA ARTIFICIAL NEURAL
NETWORKS
Proper or physically parameterized modeling of nontremulous involuntary motion is problematic due to the
largely unknown nature of the dynamics of the process [18].
Given this situation, along with the fact that biological
signals and involuntary movements are frequently nonlinear
[19, 20], nonlinear black-box modeling via artificial neural
network has been utilized. For the sake of versatility, instead
of the traditional fixed network architecture with
backpropagation, we used a cascade-correlation neural
network with Kalman filtering proposed by Nechyba [19].
Nechyba combines (i) flexible cascade-correlation neural
networks, which dynamically adjust the size of the neural
network as part of the learning process, and (ii) nodedecoupled extended Kalman filtering (NDEKF), a fastconverging alternative to backpropagation. This technique
has been used successfully for modeling of human control
strategies in driving vehicles [21], and modeling and
canceling of physiological tremor and myoclonic jerk in
microsurgery [22].
In this method, when training starts, the network has no
hidden nodes, only linear connections between the input and
the output nodes. During training, each time the error
performance stagnates, a new hidden node is added to the
network from a pool of candidate units (transfer functions),
such as sigmoid, Gaussian, sinusoidal, and Bessel. After a
brief period of being trained independently and in parallel
with different random initial weights using the quickprop
algorithm, the candidate unit with the best performance is
selected. Once a new hidden unit is installed, the hidden-unit
input weights are frozen, while weights to the output units are
retrained. The process is repeated until the algorithm
succeeds in reducing the root mean square error (rmse)
sufficiently for the training set or the number of hidden units
reaches a specified maximum number.
In neural network training, learning is cast as an
identification problem for a nonlinear dynamic system. The
neural network weights represent the state of the nonlinear
system. The NDEKF theory is then used to derive a recursion
for the weight updates [23]. This method improves the
convergence of the network weights over backpropagation
and significantly reduces the computational complexity.
III. EXPERIMENTAL METHODS
The experimental data, made available by Dr. Ed LoPresti of
the University of Pittsburgh, consisted of selected recordings
Raw trajectory
Low-passed trajectory

□
Start position
End position
*
Target
Fig. 1. The dotted line depicts a typical cursor trajectory of a MS patient
using HeadMaster PlusTM computer head control system. The solid line is
the low-passed version of the original trajectory, corrected for phase shift
and extrapolated to end at the same point as the raw trajectory.
To make the training process easier and more effective,
particularly given the relatively small volume of training
data, the training was organized in the following way. The
screen was divided into eight equal bearing sectors, namely
north, northeast, east, southeast, south, southwest, west, and
northwest. For each bearing sector, the two-dimensional
network training problem was split into two one-dimensional
ones, each for the x and y direction respectively. Each
network had 15 input nodes and 1 output node. The input to
each network was a moving window of data in the time
series. To ensure that the networks were trained to be
independent of the starting position on the screen, the
forward difference (velocity) instead of position coordinates
were used as the inputs. The training targets were the forward
difference of the phase-compensated low-passed trajectories.
Each neural network had one output node, representing the
forward difference of the error-compensated motion in that
dimension. For each bearing sector, 6 to 8 icon selection tests
randomly chosen from the 11 subjects were used for training.
The organization of the neural net training is illustrated in
Fig. 2.
N
SE
for more than 500 ms or 16 sample points (32 Hz sampling
rate). The offline simulated real-time testing of the trained
neural networks is shown in Fig. 3.
NE
2D cursor
trajectory
data file
E
W
SW
Cursor movement data read
in sequentially
SE
S
Yes
Split X & Y
trajectories
No
2D cursor
trajectory
No
Forward difference
Read in 15 pairs of
X & Y data samples
Split X & Y trajectories
X
net
Low-pass &
phase correction
Forward difference
X target
Y target
Y
net
Forward difference
X input
X net
In circle with
10-pixel
radius?
Y input
Y net

Trained
Neural
Networks:
N, NE, E,
SE, S, SW,
W, NW
Most likely
bearing
detection
14th X & Y coordinates
Error compensated cursor positions
Training
Fig. 3. Offline simulated real-time testing of the trained neural networks
Fig. 2. Organization of the neural net training
The data presented here represent offline tests using
recorded input data. The cursor recordings used for testing
were again randomly selected from the 11 subjects. These
recordings were not used in the training of the networks. To
simulate real-time testing conditions, the cursor coordinate
data was read in from the data file sequentially. To account
for the synchronization mismatch between the beginning of
data recordings and the test subjects, and to eliminate
equipment noise in the experimental setup, no action was
taken when the cursor was within a radius of 10 pixels from
its starting position. Beyond that, the forward difference of
the first 15 samples was computed as they were read in, and
from these information the most probable bearing was
estimated based on maximum likelihood criterion. The fact
that these cursor trajectories often have a reasonably accurate
onset helped the system to maintain a high success rate in
identifying the bearing. The pair of trained neural networks
that corresponded to the bearing then read in these 15 pairs of
forward difference samples, and put out the error
compensated x and y forward differences. The compensated
cursor position for the 15th point was obtained by summing
the network outputs to the position of the 14th point. There
was no compensation for the first 14 points. The neural
networks performed real-time error compensation of the raw
cursor movement until the cursor dwelled in the target circle
In our experiment, we trained a pair of neural networks
for each of the northeast, south, and west directions. Eight to
ten cursor trajectories randomly selected from the icon
selection exercise of 11 MS subjects were used in training the
neural networks for each bearing direction. A total of 29
cursor trajectories that are not used in the training, randomly
selected from the 11 MS subjects, are chosen to test the
trained neural nets. Of the 29 tests, we have 12, 8, and 9
targets located in each of the northeast-, south- and westbearing sectors respectively.
IV. RESULTS
Our bearing detection algorithm managed to correctly
identifying the bearing direction in 56 of the 69 cases tested,
or a success rate of 81.2%.
Experimental results show that the neural networks
successfully produce smoother trajectories for all the 29 tests.
The neural networks also bring the cursor of all the tests into
the target circle and keep it in the circle for more than 500 ms
or 16 sampling cycles, the criteria for the system to register
the test as complete. On the average, the neural networks
complete the test 31.8% faster than the MS subjects. A
comparison of test completion time is presented in Table II.
Fig. 4(a)-(c) shows typical results for the northeast-, south-,
and west-bearing sectors respectively.


TABLE II
Comparison of test completion time with and without the assistive
interface
NE
12
S
8
W
9
Raw data: Mean
completion time (s)
NN output: Mean
completion time (s)
Mean time saved (s)
4.52
3.34
4.05
2.89
2.39
2.44
1.62
0.95
1.60
Mean time saved (%)
35.8
28.4
39.5
Standard deviation of
% time saved (%)
14.0
11.3
23.8
No. of tests
Target circle


□
*
Target circle
□
*
Raw trajectory
NN output
Start position
End position
Target circle
10-pixel
circle
(c)
Fig. 4. Sample results for various sectors. The dashed line is the raw
cursor trajectory of the MS subjects. The dotted line is the combined
output of trained X and Y neural networks (NN). (a) Northeast-bearing
sector. (b) South-bearing sector. (c) West-bearing sector.
While we are compensating for tremulous and chaotic
cursor trajectories, we also want to make sure that the neural
networks do not overcompensate or worsen those that are
already good. The trained neural networks are tested on 4
tests with relatively smooth trajectories, 1 each from southand west-, and 2 from northest bearing sectors. On the
average, task completion time is 18.4% less with the assistive
interface than without. Fig. 5 shows the results of one of the
four trajectories tested.
Raw trajectory
NN output
Start position
End position
Target circle
10-pixel
circle
Target circle
(a)
10-pixel
circle


□
*
Raw trajectory
NN output
Start position


End position
Target circle
□
10-pixel
circle
*
Raw trajectory
NN output
Start position
End position
Target circle
Fig. 5. Sample result of the effect of neural networks(NN) on less tremulous
cursor trajectory.
Target circle
V. DISCUSSION
(b)
The experimental results indicate the feasibility of the
neural-network approach to modeling and canceling of
movement disorders of MS patients in assistive computer
interfaces.
In many cases encountered in our experiment, the cursor
trajectories become tremorous and sometimes chaotic when
the cursors get close to the target circle. As a result, the
subjects do not have sufficient control of the cursor to let it
dwells in the target circle for more than the required 500 ms.
The neural networks have evidently captured this
characteristic and compensated for the erroneous movements,
allowing sufficient dwelling time in the target circle. The
bearing detection algorithm, based on maximum likelihood
criterion, has worked reasonably well. This shows that the
assumption on the MS subjects’ ability to control cursor to
approach the general target directions is not an audacious
one. We are currently working toward implementation of the
neural network approach for online error compensation
experiment. It would be interesting to see the interactions
between the test subjects and the neural networks in action at
the human-machine interface.
The progress made in our preliminary experiment has
been encouraging, but in some ways the present tests do not
exploit the full extent of the nonlinear capabilities of the
method. MS can exhibit a variety of movement disorder
symptoms, including myoclonic and spastic involuntary
movements that are aperiodic and may not be as easily
separable in frequency from voluntary components of
motion. As this work continues it will involve data from a
larger number of subjects with a wider variety of movement
disorder symptoms, likely including those that are not
amenable to linear modeling techniques. The present results
show the feasibility of the approach; its full advantages in
comparison with, e.g., linear filtering, are expected to
become more evident as the work continues with further
exploration and additional experimental data.
The method presented here, based on the cascadecorrelation learning architecture, is a versatile technique
capable of modeling and canceling a variety of different
types of nonlinear involuntary motion. In the future we plan
to apply it also to other types of input interfaces (e.g., handoperated devices) and different medical disorders, including
severe conditions such as athetoid cerebral palsy that exhibit
extremely low signal-to-noise ratio. We also intend to design
and conduct our own data collection exercise involving hand
movement data (controlling mouse or joystick).
Foundation, and by the National Institutes of Health (grant 1
R21 RR13383) and the National Science Foundation (grant
EEC-9731748).
The authors thank Dr. Edmund LoPresti of AT Sciences
and Koester Performance Research for supplying test data,
and Ms. Ashley Holtgraver for assistance with data analysis.
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
VI. CONCLUSION
The feasibility of a novel artificial neural network
technique for modeling and online correction of erroneous
movement in an assistive computer interface has been
demonstrated using data from subjects with multiple
sclerosis. The trained neural networks successfully make the
cursor trajectories of all the test subjects less chaotic, and
hence more controllable. The neural networks also reduce the
time needed to reach the targets by an average of 31.8%.
[11]
[12]
[13]
ACKNOWLEDGEMENT
This work was supported in part by the R. Green Annan Fund
and the Lettie B. Trognitz Fund of The Pittsburgh
W. J. Weiner and A. E. Lang Tremor., eds., Movement
Disorders: A Comprehensive Survey. New York:
Futura, pp. 22 1-256, 1989.
K. Wbetten-Goldstein, F.A. Sloan, L.B. Goldstein, and
E.D. Kulas, “A comprehensive assessment of the cost
of multiple sclerosis in the United States”, Muit Scier 4
(5): 41 9-25, 1998.
R. J. Elble and W. C. Koller, Tremor. Baltimore: Johns
Hopkins, 1990.
P. O. Riley and M. Rosen, “ Evaluating manual control
devices for those with tremor disability,” J.
Rehabilitation Research Dev., vol24, pp.99-110, 1987
A. Prochazka, J. Elek, and M. Javidan, “Attenuation of
pathological tremors by functional electrical
stimulation 1:method”, Ann Biomed Eng 20:205-24,
1992.
J. G. Gonzalez, E. A. Heredia, T. Rahman, K. E.
Barner, S. K. Basu, and G. R. Arce, “ A new approach
to suppressing abnormal tremor through signal
equalization,” Proc. RESNA Annu. Conf., pp.707-709,
1995.
S. Pledgie, K. E. Barner, S. K. Agrawal, and T.
Rahman, “Tremor Suppression Through Impedance
Control”, IEEE Transactions on Rehabilitation
Engineering, Vol. 8, No. 1, 2000, 53-59.
S. Pledgie, K. E. Barner, S. K. Agrawal, and T.
Rahman, “Tremor Suppression Through Force
Feedback”, Proceedings of the Sixth International
Conference on Rehabilitation Robotics, July 1-2, 1999,
Stanford, CA, pp. 16-26, 1999.
C. N. Riviere, R. S. Rader, N. V. Thakor, “Adaptive
canceling of physiological tremor for improved
precision in microsurgery”, IEEE Trans Biomned Eng
45(7): 839-46, 1998.
Beringhause, M. Rosen, and S. Haung, “Evaluation of
a damped joystick for people disabled by intention
tremor. Proc. of the RESNA 12th Annual Conf, 41-42,
1989.
M. Rosen, A. Arnold, I. Baiges, M. Aisen, and S.
Eglowstein. Design of a controlled energy-dissipationorthosis (CEDO) for functional suppression of
intention tremors. J. Rehabilitation Research and
Development, 32(l): 1 — 16, 1995.
J. Caviness, “Myoclonus”, Mayo Clinic Proc 71:679688, 1996.
J. R. Wolpaw, D. Flotzinger, G. Pfurtscheller and D. J.
McFarland, “Timing of EEG-hased cursor control”, J
Clin Neurophysiol 14 (6): 529-38, 1997.
[14] C. Lau and S. Oteary, “Comparison of computer
interface devices for persons with severe physical
disabilities”, Am J Occup Ther 47(ll): 1022-30, 1993.
[15] C. J. Clayton, “An intra-oral access device”, JMed Eng
Technol l6 (5): 204-9, 1992.
[16] J. R. LaCourse and F.C. Hiudik Jr., “An eye movement
communication-control system for the disabled”, IEEE
Trans BiomedEng 37: 1215-20, 1990.
[17] E. F. LoPresti and D. M. Brienza, “Automatic
Customization Software for Computer Head Controls”,
Proceedings of the RESNA 2002 Annual Conference.
June 2002. (in press).
[18] J. B. Hampshire, “A differential theory of learning for
efficient statistical pattern recognition”, Ph. D.
dissertation, Carnegie Mellon University, Pittsburgh,
Pa., ch. 3, 1993.
[19] C. N. Riviere, and N. Thakor, “Modeling and
canceling tremor in human-machine interfaces”, IEEE
Engineering in Medicine and Biology, 15(3): 29- 36,
1996.
[20] C. Gantert, J. Honerkamp, and J. Timmer, “Analyzing
the dynamics of hand tremor time series”, Biol Cybern
66:479-84, 1992.
[21] M. C. Nechyba and Y. Xu, “Learning and transfer of
human
real-time
control
strategies”,
JAdv
Computational Intell 1(2): 137-54, 1997.
[22] W. T. Ang and C. N. Riviere, “Neural Network
Methods for Error Canceling in Human-Machine
Manipulation”, Proceedings of the 23rd IEEE EMBC
Annual Conference, pp. 7.6.1-5, 2001.
[23] G. V. Puskorius and L. A. Feldkamp, “Decoupled
extended Kalman filter training of feedforward layered
networks. Proc. Intl. Joint Conf. Neural Networks,
1:771-7, 1991.