On-screen Target Acquisition
On-screen Target Acquisition through Mouse Movement Recognition
Hyun W. Ka, M.S; Richard C. Simpson, PhD; Vinod K. Sharma, PhD
Department of Rehabilitation Sciences and Technologies, University of Pittsburgh
ABSTRACT
In this paper we present a new design idea as an alternative to improve on-screen target
acquisition for individuals who have difficulty pointing by inferring the user’s intended target from the
user’s mouse movements. The intended target is inferred based on real-time mouse movement
information and the properties of all user interface elements currently displayed on the computer screen.
During preliminary testing of the prototype application, the software reduced pointing errors and the
physical effort needed to operate a pointing device. The software is compatible with a range of existing
pointing technologies.
Keywords: pointing; on-screen target acquisition; computer access; mouse movement recognition; intent
recognition
BACKGROUND
Since the graphical user interface (GUI) was introduced, the ability to acquire on-screen targets
has been a critical component of effective computer access. However, this emphasis on pointing has
made it more difficult for some individuals with motor impairments to access a computer. Problems with
on-screen target acquisition include difficulty positioning the mouse cursor within a confined area,
slipping off targets, and trouble moving the mouse cursor in a straight line [1]. These types of problems
can cause the user to restart or undo subsequent actions, which usually increases the total time needed to
complete the task. There have been a variety of research efforts to address this problem, such as area
cursors [2], bubble cursors [3], bubble targets [4], force fields [5], goal crossing [6], gravity wells [7],
multiple monitors [8], object pointing [9], semantic pointing [10], and sticky icons [11]. However,
because user needs and abilities are extremely diverse, not only between users but also for the same user,
the existing technologies are not able to cover all issues. This research presents another alternative to
improve on-screen target acquisition for individuals who have difficulty pointing by inferring the user’s
intended target from the user’s mouse movements.
METHOD
On-screen Target Acquisition
Design Criteria
Four design principles were used to evaluate the quality of our design idea. 1) Ease of use: The
proposed design idea should minimize the efforts needed to acquire on-screen targets of interest. 2)
Specificity: The design should be able to keep the user from acquiring unwanted targets. 3) Efficiency:
The number of steps to acquire a target of interest should be minimized. 4) Flexibility: The design
should be compatible with a range of existing pointing technologies to ensure maximum adaptation to
user needs.
Underlying Design Technologies
Two types of information are required to infer the intended target: (1) real-time mouse
movement information and (2) the properties of all user interface (UI) elements currently displayed on
the screen. In order to get mouse movement data, we employ a system events hooking technique to
monitor and intercept mouse events. The information retrieved includes all details associated with
mouse movements and operations. In order to get the properties of UI elements, we employ the
accessibility application programming interface (API) provided by Microsoft Windows. The technology
allows software developers to programmatically make all of the standard UI elements (e.g., button, edit
box, list control, combo box, icon, scroll, and etc.) expose their properties to the client software. The
properties of UI elements can be represented by a hierarchical tree structure as shown in Figure 1. To
effectively manage the information retrieved from the above two technologies, we devised a dynamic
virtual screen object model (SOM). Each time the user moves the mouse cursor, the SOM updates its
record of mouse operations, and whenever the top level window is changed, the UI tree is reorganized.
----------------------------------Figure 1 Goes Here
-----------------------------------
Design algorithm
Mouse movement information allows the software to calculate motion parameters (e.g. distance,
direction, speed), which are used to trigger the decision making process. The information included in
each node of the UI tree serves as a point of reference. Each motion parameter plays its own role in the
algorithm. Distance determines how far the target of interest is away from the current mouse position.
Direction, which is evaluated from the movement angle, determines the relative position of the target,
and speed along with distance acts as a threshold for whether or not the decision making process should
be activated. Each node in the UI tree corresponds to a UI element on the screen. It can be used not only
to indicate what kind of UI element a particular control represents (such as a button, icon, or menu), but
also to determine what types of controls are available in the current context and how to interact with
On-screen Target Acquisition
them. When the user makes any kind of mouse movement, the agent for decision making interprets the
mouse event and characterizes the pointer’s motion. If the parameters describing the pointer’s motion
exceed a threshold, the agent starts the decision making process, navigating the UI tree to find the target
of interest based on the motion parameters. When reaching the target, the agent provides information
about it to the client application.
RESULT
In order to evaluate the feasibility and performance of the presented design idea, we developed a
prototype application for the Microsoft Windows operating system. The UI Automation API included in
MS .Net Framework 3.5 was employed, and Microsoft Visual Studio 2008 Professional edition was used
as the main development tool. The software was implemented using two programming languages: C++
for low level event hooking and C# for the client application.
The prototype application infers the user’s intended target from the user’s mouse movements.
Thus, the user is not required to keep moving the mouse cursor until it reaches the intended target.
Instead, the user determines the direction where the target of interest is and makes a short mouse
movement. Then, as shown in Figure 2, the software predicts the target of interest (surrounding it with a
yellow rectangle) and automatically moves the mouse cursor toward it. When the mouse cursor reaches
the target, the yellow rectangle turns red, as shown in Figure 3.
----------------------------------Figure 2 Goes Here
--------------------------------------------------------------------Figure 3 Goes Here
----------------------------------Involuntary movement, as one of the most frequent errors associated with mouse operation, can
cause the user to get unwanted targets. In order to prevent that kind of error, the application allows the
user to adjust the threshold for triggering the on-screen target acquisition process. The process does not
start until the mouse movement exceeds given speed and distance thresholds. When the result of
prediction represented by a yellow rectangle does not point to the intended target, the user can cancel the
process by making a random mouse movement to which the same threshold mechanism is applied. Then,
the yellow rectangle disappears and mouse movement is stopped. In addition to this, the application also
provides a sticky function snapping a mouse cursor to the center of the target to prevent slipping out of it.
Blocking and occlusion, where surrounding targets interfere with the cursor’s approach to the
intended target, can increase the number of steps to get access to the target. To address this issue, the
presented design utilizes the distance of current movement. Based on the distance, the design algorithm
On-screen Target Acquisition
determines how far the target should be away from the current mouse position. For the prototype, the
determination is made just for two cases: the nearest and the farthest one.
Considering maximum adaptation to user needs, as shown in Figure 4, the prototype application
provides options for the user to customize the functionalities of it: change the threshold of trigger and
cancellation, adjust movement speed, and predefine the types of target of interest by selecting or
unselecting each of them.
----------------------------------Figure 4 Goes Here
----------------------------------During a preliminary test of the prototype application with common pointing devices (including
a standard mouse, trackball, touchpad, and a head-mounted mouse emulator), it was observed that the
software is compatible with them, reduces pointing errors and difficulties with acquiring on-screen
targets, and reduces the physical effort needed to operate the pointing devices.
DISCUSSION
Our goal is to improve on-screen target acquisition for people with motor impairments by
inferring the intended target from their short mouse movements. We have made an algorithm for
inferring the target of interest by integrating the current mouse events with properties of UI elements
currently displayed on the screen. This approach allows the users to use their familiar pointing devices
without additional settings or modifications, and minimizes the physical effort needed to reach the
intended target. Since the current work is focused specifically on acquiring the target of interest, the
succeeding mouse operations with the acquired target like clicking and dragging are not considered here.
Future directions include holding focus groups and collecting empirical data from clinical trials.
ACKNOWLEDGEMENT
This study is funded by the Quality of Life Technology Engineering Research Center, National Science
Foundation (EEC-0540865).
REFERENCES
1.
HWANG F, KEATES S, LANGDON P, CLARKSON J. Mouse movements of motion-impaired
users: a submovement analysis. ACM New York, NY, USA, 2004: 102-09.
On-screen Target Acquisition
2.
KABBASH P, BUXTON W. The "Prince" Technique: Fills' Law and Selection Using Area
Cursors. ACM New York, NY, USA, 1995: 273-79.
3.
GROSSMAN T, BALAKRISHNAN R. The bubble cursor: enhancing target acquisition by
dynamic resizing of the cursor's activation area. ACM New York, NY, USA, 2005: 281-90.
4.
COCKBURN A, FIRTH A. Improving the acquisition of small targets. People and Computers.
2004:181-96.
5.
AHLSTRÖM D, HITZ M, LEITNER G. An evaluation of sticky and force enhanced targets in
multi target situations. ACM, 2006: 67.
6.
WOBBROCK J, GAJOS K. A comparison of area pointing and goal crossing for people with
and without motor impairments. ACM, 2007: 10.
7.
HWANG F, KEATES S, LANGDON P, CLARKSON P. Multiple haptic targets for motionimpaired computer users. ACM New York, NY, USA, 2003: 41-48.
8.
BAUDISCH P, CUTRELL E, HINCKLEY K, GRUEN R. Mouse ether: accelerating the
acquisition of targets across multi-monitor displays. ACM New York, NY, USA, 2004: 1379-82.
9.
GUIARD Y, BLANCH R, BEAUDOUIN-LAFON M. Object pointing: a complement to bitmap
pointing in GUIs. Canadian Human-Computer Communications Society School of Computer Science,
University of Waterloo, Waterloo, Ontario, Canada, 2004: 9-16.
10.
BLANCH R, GUIARD Y, BEAUDOUIN-LAFON M. Semantic pointing: improving target
acquisition with control-display ratio adaptation. ACM New York, NY, USA, 2004: 519-26.
11.
WORDEN A, WALKER N, BHARAT K. Making computers easier for older adults to use: area
cursors and sticky icons. ACM New York, NY, USA, 1997: 266-71.
Author Contact: Hyun W. Ka
Department of Rehabilitation Sciences and Technologies, University of Pittsburgh
853 Bakery Square, 6465 Penn Avenue, Suite 401 Pittsburgh, PA, 15206
Hyk21@pitt.edu
(412) 390-8241