Multiple-Mouse Multiple-Monitor
Applications
CS 691
Project Documentation
James Murithi
Academic Supervisor:
Dr. Duke Hutchings
Spring 2008
Department of Computer Science
Bowling Green State University, Ohio
Abstract
With the advent of multiple monitors it becomes interesting to build applications that support
multiple users simultaneously interacting with mice. This project explored the existing software
development kits used to build multiple mouse applications. Limited support was found for the
multi-user, multi-mice, multi-monitor scenario this paper discusses these limitations. The project
focuses on one multi-mouse toolkit whose implementation has some interaction limitations. This
paper presents general ideas that were developed to counter these limitations. These ideas can be
used to improve the toolkit or to build a new toolkit. This paper presents the ideas developed
during this project and provides an analysis of the solutions that were built.
2
Table of Contents
Introduction ..................................................................................................................................... 4
Goals and Objectives .................................................................................................................. 4
Contribution ................................................................................................................................ 4
Project Overview ............................................................................................................................ 6
Related Work .................................................................................................................................. 7
Framework for enabling collaboration........................................................................................ 7
CPN Mouse ................................................................................................................................. 7
Windows Multipoint ................................................................................................................... 8
SDGToolkit ................................................................................................................................. 9
Project Details ............................................................................................................................... 10
Drag and click ........................................................................................................................... 10
Project Analysis ............................................................................................................................ 12
Future Work .................................................................................................................................. 14
Bibliography ................................................................................................................................. 15
3
Introduction
There are a number of reasons why multiple mice would be desired on a system. Michael
Westergard talks about several where one mouse would be better than one. In graphic
applications it would be helpful to have different mice draw different types of objects; a user
could also do two handed resizing of objects using two mice [Westergaard. 2002]. It could also
be helpful to have multiple mice if one has huge screen space, this would significantly reduce
mouse movement and the probability of losing track of the mouse cursor. These scenarios show a
single user with multiple mice. Multiple mice could also be useful in a group setting. Two
collocated users can share a system and complete tasks collaboratively or competitively. A good
example of a competitive multi mouse task is a two player tic-tac-toe game. Each player clicks
on items in the screen using their mouse when it is their turn to play. Collaborative tasks require
cooperation between users. When the two players finish playing their tic-tac-toe game and want
to start a new one, who will activate the new game? This task requires cooperation, the system
may be designed to support this kind of collaboration and if not the users will learn to collaborate
without the computers’ support for them to proceed with the task.
To write applications that support multiple mice there has to be support for two or more cursors
on the screen. Microsoft Windows does not natively support this. When more than one mouse is
connected to the system a distinction cannot be made between the different input streams. The
result is that both devices control the same cursor on the screen. To distinguish between mouse
input streams it becomes necessary to first identify the different devices, draw cursors for each
then associate hardware input with the cursors. I explore different techniques of enabling
multiple mice on the screen. The raw input API introduced in Windows XP provides rich
information regarding hardware devices; this can be used to separate hardware input streams and
draw cursors for each. Another option would be to use multi mouse toolkits. These toolkits come
with built in ability to distinguish between mice and support for rapid prototyping of multi
mouse applications. Using a toolkit eliminates the need for developing a custom solution for
each application a programmer would like to develop. This project explores existing multi mouse
toolkits. Applications were written to support multiple monitors.
Goals and Objectives

To explore existing Multiple Mouse toolkits for the Windows Operating System, use and
improve the toolkit where possible.

To write applications that allow interaction using more than one mouse and also allow
users to collaborate when completing a task
Contribution
I found existing Windows multiple mouse toolkits and explored them giving their strengths and
weaknesses. Only one of the toolkits allowed me to write applications that span more than one
monitor. This was a key requirement for this project. However this toolkit does not have widgets
that the multiple mice could interact with. I implemented a technique that allows the toolkit
4
cursors to interact with windows toolkit widgets. Using this technique it is now possible to write
applications that use windows widgets which respond to the toolkits mouse cursors.
5
Project Overview
Out of three toolkits found two of them could be installed and used with programming tools to
rapidly prototype multi mouse applications. Windows multi point was found to lack multi
monitor support and therefore no further work was done using this toolkit. SDGToolkit (Single
Display Groupware Toolkit) was found to have multi-monitor support but it lacked user interface
components. However it has a default style of interaction that lets the toolkits cursors interact
with windows toolkit components in a limited way. This default behavior is slow when there is
more than one user on the system because it utilizes the system cursor. The system cursor has to
keep switching its position to the location of the then active toolkit cursor for any interaction to
be possible. Improving this default behavior was one of the major things that I focused on during
this project.
6
Related Work
Framework for enabling collaboration
A study was done that focuses on support of multiple mice with an aim of determining what
differences exist when collaborative tasks are done using mice versus when using laser pointers.
During this study a framework was built for enabling multiple mice in an application. This
framework is a client server architecture where a pair of socket connections is created to handle
events from different mice. The sockets keep track of user interface component states for each of
the connected devices and transmit these events to higher level client applications for processing.
[Vogt, et al. 2004]
This framework is a good idea for supporting multiple mice because the authors of this paper
proved the concept and it served their experimental purpose. However to use this technique the
framework would have to customized for each application that needs to support multiple mice.
One of the conclusions drawn for from this work is that it is necessary to distinguish between
cursors on the screen either by coloring them differently or labeling the cursors. This would
enhance the interaction process by allowing users to recognize their cursor. The relevance of this
work to my project is its emphasis on the importance of distinguishing between cursors on the
screen, there is also a limit to the number of cursors that can be used effectively in a certain
screen size. When building a new toolkit it will be necessary to include functions for changing
mouse cursor colors or labeling the cursors with an identity tag.
CPN Mouse
CPN Mouse is a filter driver that enables multiple mice on Windows operating systems; the
project was motivated by the ongoing CPN tools project that seeks among other things to
investigate advanced interaction techniques using multiple pointing devices. The solution to
enabling multiple pointing devices that existed before the CPN Mouse project was a non generic
method of probing a secondary mouse for events and transmitting them to an application. The
solution did not scale well with increasing number of mice and different mouse interfaces. The
CPN Mouse created a generic mouse interface and an application programming interface (API)
to access it. The CPN Mouse filter driver is generic in that it allows different mouse interfaces to
make use of the driver [Westergaard. 2002]. The driver also solved the problem of scaling to
many mice. Further a user does not need to have an idle mouse that is only active within CPN
applications. By running programs from a special folder the toolkit enables multiple mice. The
driver comes with an application programming interface that allows a programmer to write multi
mouse applications.
This toolkit has some limitations. To start the driver is built to support a framework specific to
CPN tools and it may not be suitable for other frameworks or applications. My project aimed to
write windows applications that do not necessarily follow the CPN tools framework; therefore I
was not guaranteed support. Little documentation was found for this toolkit and there exists no
7
examples on how to use it. The only reference to the API is found on the toolkits website. The
reference is code for the various headers that the toolkit is made up of, this was not helpful. It
made it hard for me to get started using the toolkit. Besides that I needed to use a language that
allows rapid development of windows applications. Such a language should provide user
interface components like buttons and text boxes. CPN Mouse restricts me to using C which does
not support rapid application development; I therefore opted against using this toolkit. Further
the latest updates to the toolkit were done in 2004, it is documented to support windows 2000
and XP, for my project I had Windows Vista therefore I was not guaranteed it would work. This
toolkit would have caused incompatibility issues for my project.
Windows Multipoint
Windows Multipoint is a multiple input device-enabling toolkit that was the result of a study
done concerning the high ratio of students to computers in developing countries. This study
looked at how having a single input device impacts the learning process among children using
educational software. And according to the study children immediately understand the concept of
multiple mice and using multiple mice enables sharing of expensive hardware [Pawar, et al.
2006]. This toolkit has several desirable features for multiple mouse interaction, one of which is
the ability to distinguish between multiple mice and draws cursors for each on the screen without
the programmer having to write any code. Further the toolkit uses color in a good way that
allows a programmer or user to differentiate between cursors belonging to different mice. The
toolkit comes with a template for rapidly writing multiple mouse enabled applications in C#.
This template speeds up development. Another feature of this toolkit is its user interface
components. These components include a readily made Multipoint button that can receive a
Multipoint event from the different mice and distinguish between them. This feature of the
toolkit makes it possible to write applications that can be used by different users all of who
interact with components while distinguishing between different user inputs. The button is the
only ready made Multipoint control but the toolkit makers included a template for rapidly
prototyping other Multipoint controls. The toolkit comes with a sample two player tic-tac-toe
game and compiled documentation which was valuable in introducing basic concepts of
programming using the toolkits event model and controls.
For several reasons I found this toolkit unsuitable for my project. First and foremost this toolkit
has a known issue of disallowing movement of cursors across extended monitors. The toolkit
was made to work for a single display. For my project I wanted to explore how interaction would
proceed with multiple users each with their input device and with my application spanning
multiple monitors. Also the toolkit was made to use the newer Windows Presentation Foundation
(WPF) style of interface design over the old Windows Forms style. A major limitation of WPF to
my project is its lack of support for the traditional multiple document interface applications
approach of writing applications that allow independent movable and re sizable windows to exist
within a container window. With WPF multiple windows in an application need to be tabs within
the container. The toolkit comes with a compiled documentation file which teaches how to use
and write Multipoint controls, no on line documentation exists, this impacted on my project
negatively since it took me a long time to figure out that multiple monitor support was a known
issue. Finally from experiments conducted during a study it shows that applications built using
this toolkit are good for competitive activities like games, little has been done that is focused on
8
collaboration but rather applications that model the turn taking model are favored. A good
example was children playing an educational game, when it was time to start a new one it was
not clear who was supposed to start the new game since all children had control.[Pawar, et al.
2006]
SDGToolkit
Before this toolkit was written several studies had been done on single display groupware.
However there was no easy way of managing simultaneous use of multiple input devices or
rapidly prototype applications that make use of multiple input devices. SDG toolkit was built to
solve this problem; it allows rapid prototyping of applications that use several input devices.[Tse,
et al. ] The toolkit comes with excellent on line documentation and more than a dozen sample
applications that illustrate the various features of the toolkit. The documentation also clearly
states the toolkits’ limitations.
I found this toolkit suitable for my project because it allowed me to write applications without
restricting cursors to a single screen. This toolkit also uses windows forms interface design style
which was a desired feature for my project because I could have multiple independent windows
within a container. The toolkits major limitation is lack of user interface components that can
distinguish between mice. Since one of my project objectives was to improve a toolkit where
possible I decided it was worthwhile to do it for this toolkit because the documentation allowed
me to easily understand how the toolkit works. Another limitation of this toolkit is in its native
style of interaction, the windows system mouse remains active for the duration of the interaction,
depending on how a programmer implements their application the system mouse can be a
problem.
9
Project Details
Drag and click
Dragging was one of the first behaviors I choose to implement for the SDG toolkit. The default
drag behavior was not appropriate. While it was possible to move a component like a button
around using the SDG cursors I noticed that once the user let go of the component it did not drop
down to the exact position of the SDG toolkit mouse cursor. From reading the toolkit
documentation manuals I found that the toolkit cursors are in fact little windows, this dropping
behavior could be explained by this fact. Its possible that once the object is dropped it is dropped
to the SDG cursor windows top left coordinate where as the cursor drawn is somewhere within
the containing window and not exactly at the top left coordinate. To rectify this behavior I made
API calls that give me the power to capture a windows image using its unique identifier. Instead
of moving the actual object I move its representation and on dropping the object I replace its
picture with the real object.
Interaction with windows toolkit user interface components was not supported by SDG mouse
cursors therefore there is no way to click a standard graphical user interface component using an
SDG cursor. I tried several techniques of to implement a click behavior. Firstly I sub classed the
toolkit and made a component that can be clicked by the toolkits cursors, the object was meant to
be a button that can be clicked by the toolkits cursors. Then instead of writing special
components I enabled interaction with standard windows toolkit components since these are
ready made. The premise was, if I could get it to work for one then there is a chance my method
can be extended to the other windows toolkit components. I implemented the click method by hit
testing, a method of determining if and where an object is hit by a mouse cursor. By making API
calls to windows its possible to get a windows bounding rectangle and I can tell a windows
location by its coordinates relative to its container or to the screen. By doing some calculations I
can tell if the SDG cursor is within the bounding rectangle of a given window. Subsequently I
can activate any function or even call an objects methods when this condition is met. Once the
click method was done I wrote a simple application that allows text entry facilitated by clicking
windows toolkit labels using SDG cursors.
I then built an application to investigate how collaboration can be achieved by two people trying
to accomplish a task like browsing the Internet. The application is a web browser extended into
two monitors and with multiple mice enabled. The idea is to have two people interact with their
browser component without interfering with each other. The approach to enabling multiple mice
that I implemented used the system cursor to achieve interaction. The toolkit cursors in this
approach were powerless, interaction with the web browsers was achieved by allowing the
system cursor to invisibly follow the currently active toolkit cursor. When the toolkit performs
an operation like clicking it causes the underlying system cursor to perform the operation. In this
way I can interact with windows toolkit objects using the SDG toolkit cursors. Since it is
undesirable to have the system cursor follow both mice at the same time I use a visual technique
to try enforce a turn taking model where one user has to wait until the other used is done for
them to perform an operation. I also wrote code to enable interaction by just SDG cursors, this
10
can be done by hit testing a component and then activating a methods to manipulate the
component only when its hit by an SDG cursor. I got to the level where I was able to scroll a
browser component by using SDG cursors.
The toolkits native behavior did not allow dragging objects in an acceptable manner. On
dragging and dropping a component the drop location is not the exact cursor position. I solved
this problem by first detecting if a component has been clicked. After determining whether the
toolkit cursor is on the object I take the components’ image and hide the component. I can then
move the image around and when a user lets go of the mouse place unhide the component and
place it exactly where its’ image is on the screen. Simultaneous dragging led to a flickering
image in the native behavior. By identifying components and distinguishing mice I was able to
keep certain mice from interacting with specific components. This makes sure that one user does
not interfere with another users’ component.
11
Project Analysis
I implemented a dragging technique. Since the technique makes use of SDG toolkit cursors I
conclude that SDG cursors can be empowered to be as useful as the system cursor is. The
method also shows that two users can interact with an object without one person’s action
hindering the others; both collaboration and turn taking can be built into applications. In a demo
application I wrote two users can drag their unique objects around using different mice. I did not
observe any interference or undesirable behavior when the objects were being moved around.
Further while the default toolkit behavior resulted in a misplaced drop when dragging the method
we came up with does not. It solved the problem of a misplaced drop that existed as the toolkits
default behavior. The shortcoming of this technique is that it is hard to apply it to larger
components like windows. When I applied the technique to a window there are some parts of the
window that give an immediate result informing me that the window has received a hit. There
are sections of a window that do not return a hit notably the title bar. This can be explained by
windows design, the title bar is a special section of the window referred to as the non client area.
The non client area of a window shows when a window is active by having a dim color or a
brighter one if the window has focus. This is also the portion of the window used to drag a
window around. I could not get hits on this portion of the window because toolkit cursors cannot
send messages to this portion of a window.
On testing the click technique it was found to improve interaction speed. While previously the
toolkit allowed clicking windows toolkit objects solely by allowing the system cursor to invisibly
follow the SDG cursors using this technique it is now possible to click a windows toolkit
component. The default style of interaction was slow because each time one of the SDG cursors
was moved the system cursor had to move to perform an action on behalf of the SDG cursor.
Using the method we developed its no longer necessary to use the system cursor, since no
relocation of the system cursor is needed interaction speed improves. The button made by subclassing the SDG toolkit could be used in place of windows toolkit buttons, this button does not
need the system cursor to invisibly click it. However because of toolkit restrictions it was not
possible to include standard properties like text in this object. Having text on a button is a feature
that would be desired to communicate the buttons purpose, the SDG button would need to have
an image with text explaining its function as its background. Using such buttons in an application
would thus be tedious because the programmer needs to make an image with a suitable
background for each button needed. The hit test method I made can be applied to any component
which has a window handle, one of the problems encountered was that when it came to some
level some components were not well documented possibly due to the assumption that nobody
would need to use them. An example is the scroll bar button, the windows documentation was
insufficient in helping me determine how I can access the button there was no example code
either. My hit test function works for every window that has a handle, if the window does not
have a handle this technique cannot be used. Since I could not uniquely identify the scroll bar
button the only way toolkit cursors can scroll a document is by holding anywhere within the
document.
The visual technique used to enable collaboration does not restrict users from using their mouse.
With this technique we can only hope that the visual busy indicator stops the user from
12
attempting any interaction. The technique simply suggests to the user that they should not move
their mouse; they still have the ability to move their mouse. This would interfere with the other
person’s interaction. Finding a way of disabling currently inactive mice when another person is
performing an operation using their mouse would be a better way of enforcing turn taking. The
locking mechanism I implemented for buttons and smaller objects could possibly be abstracted
to suit entire windows. However the aim was to allow collaboration and not turn taking.
13
Future Work
The work done so far enables better faster interaction with windows toolkit components using
the toolkits cursors which were not meant to do this. One of the problems that needs more work
is extending these ideas to more windows toolkit components like scroll bars, window title bars
and hyperlinks are good examples. Currently there is no way the SDG cursors can sense that a
link therefore though a user can browse what is currently on the browser the only way they can
move to another link is by typing an address in a text box then navigating to it.
The visual technique of enabling collaboration needs to be more restrictive. Though the
technique works for users with some knowledge of how the system works it would not be
appropriate for standard users to work with such an application. This is because accidental
movement of a mouse when another mouse is active results in delayed interaction or inexplicable
feedback for example when dragging. If a second user moves their mouse when another user is
dragging an object then the object switches back and forth between the two mouse cursors.
Disabling other mice when a user is actively engaged in some interaction would eliminate this
problem.
The windows system is made with a single user and single mouse in mind. Therefore its hard for
applications designed for two people when trying to enable two users to interact seamlessly to be
built because of problems like change of focus or sharing of focus. It would be worthwhile to
write applications that spawn multiple windows sessions then send messages back and forth as a
work around to the design that was made for a single user with a single mouse.
14
Bibliography
BIER, E.A. AND FREEMAN, S. 1991. MMM: a user interface architecture for shared editors on
a single screen. In UIST '91: Proceedings of the 4th annual ACM symposium on User interface
software and technology, Hilton Head, South Carolina, United States, Anonymous ACM, New
York, NY, USA, 79-86.
HOURCADE, J.P. AND BEDERSON, B.B. Architecture and Implementation of a Java Package
for Multiple Input Devices (MID). Human-Computer Interaction Laboratory, University of
Maryland, College Park, MD 20742.
KIM, Y.B., SHIM, M.S., SONG, C.G. AND KIM, Y.S. 2007. A First Person Shooter with Dual
Guns Using Multiple Optical Air Mouse Devices. Artificial Reality and Telexistence, 17th
International Conference on 306-307. .
MORRIS, M.R. AND HORVITZ, E. 2007. SearchTogether: an interface for collaborative web
search. Proceedings of the 20th annual ACM symposium on User interface software and
technology 3-12. .
PAL, J., PAWAR, U.S., BREWER, E.A. AND TOYAMA, K. 2006. The case for multi-user
design for computer aided learning in developing regions. In WWW '06: Proceedings of the 15th
international conference on World Wide Web, Edinburgh, Scotland, Anonymous ACM, New
York, NY, USA, 781-789.
PAWAR, U.S., PAL, J. AND TOYAMA, K. 2006. Multiple Mice for Computers in Education in
Developing Countries. International Conference on Information Technologies and Development,
May .
TSE, E. AND GREENBERG, S. SDGToolkit: A Toolkit for Rapidly Prototyping Single Display
Groupware. screen 91, 79-86. .
VOGT, F., WONG, J., PO, B., ARGUE, R., FELS, S. AND BOOTH, K. 2004. Exploring
collaboration with group pointer interaction. Computer Graphics International,
2004.Proceedings 636-639. .
WESTERGAARD, M. 2002. Supporting Multiple Pointing Devices in Microsoft Windows.
Microsoft Summer Workshop for Faculty and PhDs.Cambridge, England, September .
15