X10 Workshop on Extensible Virtual Worlds
Position Paper
SOFT CONTROLLER – A UNIVERSAL REMOTE CONTROL
FOR THE INTERNET OF THINGS AND FOR VIRTUAL WORLDS
Jonathan McCrary <jmccrary@uark.edu>, Craig Thompson <cwt@uark.edu>
Computer Science and Computer Engineering Dept., University of Arkansas, Fayetteville AR
Context
Humans directly view and touch things in their environments. Man, as a tool user, has developed intermediaries
(tools) for some tasks. Today, most devices are still controlled by a human operating a physical interface attached to
a device. Increasingly, we use “universal remote controls” to operate devices at a distance, including TVs, stereos,
gas fireplace logs, security systems, and PowerPoint presentations. If we aim for a world where many more objects
are smart objects that can be controlled remotely, then there are some problems to solve [1].
Problem
The term Internet of Things refers to a future time (actually happening now with RFID) where we can address and
communicate with real world things, which have become networked objects.
To build this Internet of Things, we will need to understand what makes an object smart – see companion paper on
reifying interfaces to make objects smart(er) [2].
If we separate some of these smart object interfaces from the object itself so they can be controlled remotely through
an intermediary device, we need to better understand the (eventually standardizable) architecture of these remote
control devices.
How many remote controls do you own? Most of today’s remotely controls have special case architectures that only
work in one application. Very often, the physical buttons on the remote reflect the functions of that device directly
and cannot be reprogrammed to control different classes of devices. A general purpose universal remote control
would need a more general architecture with an ability to learn about new smart devices and a flexible (soft)
interface that is reprogrammable for the target smart object(s) being controlled.
As more and more objects become smart, it will be increasingly important for devices to operate in ensembles that
humans can control. Managing complexity will become an issue. Similarly, managing security will become an
issue as not everyone should have the same access permissions to control every smart object. But at the same time,
humans won’t want to waste their lives managing and programming hundreds to thousands of smart objects.
Finally, the problem of a universal remote control for smart objects is not limited to the real world. It is just as
relevant to smart objects in virtual worlds. In virtual worlds, many objects can be scripted but a human-controlled
avatar can pass by an object without knowing that the object is scripted or how to communicate with the object even
if they do know it is a smart object so objects need a way to expose their interfaces.
Objective
The objective of this project is to develop a family of architectures for soft controllers that can communicate with
real or virtual smart objects.
Background - Related Work
The Star Trek communicator [3] can be taken as an ideal of a personal device that a human can use to communicate
with other humans and with things. While many remotes are programmable via training, Harmony remote controls
are near the high end of programmable remotes, with USB interfaces to a web database of device descriptions that
can be downloaded [4]. But, these are still special purpose. More general purpose would be a smart phone albeit
augmented with additional capabilities. Indeed, smart phones already are programmable with new apps and are
increasingly location aware. Where much of the world missed the personal computer revolution, countries like India
and China are joining in using smart phones. Smart phones (proprietary iPhones, open source Android [5) can
already act as TV remote controls [6]. RFID-enabled smart phones (and other near field technologies) are a logical
next step [7]. Meanwhile, some smart home applications can already use iPhones to control a suite of security and
entertainment devices [8].
Family of Architectures
In a pre-remote-control architecture, a person communicates by touching the interface (I/F) controls embedded in an
object:
1
X10 Workshop on Extensible Virtual Worlds
Position Paper
Human -------operates--------I/F + object
With a remote control, a person communicates with the intermediary device which communicates with the object.
Of course, not uncommonly, both architectures are usable together. You can use control your TV via its built-in
controls or its remote controls. But in this architecture, the remote is bound (dedicated) to the specific object it
controls.
Human ---------operates----------remote including I/F----------object
In a more general architecture, the object interface can parameterize the remote. Here, the I/F is loaded from the
object from a web-cloud while remote is docked or wireless.
Human ---------operates----------remote [I/F*]----------object*
In this more flexible case, there are a family of architecture alternates

The remote could need to communicate with different smart objects using a variety of near field
communication protocols (RF, RFID, …).

The remote could need to acquire the object’s I/F from the object itself or by using a manual
downloading protocol or dynamically from the web-cloud. It might require docking to the Internet and a
website (like Harmony remotes) or wireless connections (smart phones).

Finally, the form of the user interface the remote uses could include any or all of the following: GUIs
(which may look like physical remotes or 3D virtual world models), pull down menus, completion menus,
English, speech, others TBD).
Higher level interfaces will be needed as well:

A variety of interfaces – so people can choose the way they want to talk to objects. (Many people
would benefit from a remote a lockable Input button as that can wedge their TV).

Security (access control, authentication, encryption, grant operations) to make sure only approved
users can control the object

Plugin interfaces – so we can extend smart objects with additional capabilities like scheduling and
logging plugins. Presumably, you could download updates and additional plugins.

Rules, policies, workflows, and other means for controlling and communicating with assemblies and
collections of smart devices.
Where Virtual Worlds fit in
It is relatively easy to experiment with turning virtual world objects into smart objects because they have 3D
sharable representations, identity, access control, and can be scripted. So, it makes sense to experiment with
universal soft controllers for smart objects in virtual worlds.
At the same time, Second Life (SL) is missing a general capability to control smart devices. As mentioned, human
controlled avatars can walk right past a highly scripted object but may not know it is controllable or how to control
it. SL does support a PIE menu interface – point at an object and you can select it, sit on it, or buy it. But the PIE is
static with no ability to upload object-specific interfaces. And in general, there is no reflection so one cannot ask an
object what commands it accepts. On the other hand, virtual worlds are rich in objects and it makes as much sense
to control and experiment with assemblies of smart objects in a virtual world as in the real world. In fact, it might
even be the case that, done right, the same interface that controls virtual world objects could control real world
objects especially in mirror world applications where the virtual world mirrors the real world [9].
In summary, SL supports some of the capabilities needed to experiment with next generation universal soft
controllers:

Object Identity – objects have unique IDs (at least as long as they are rezzed)

3D visual interface, a large number of object types, and the ability for users to create more objects

Script-ability

Access control
2
X10 Workshop on Extensible Virtual Worlds
Position Paper
But SL is missing other capabilities that it would need to make it easier to build soft controllers for smart objects:

Persistent identity – so individual object identity survives past being returned to an inventory

Type and ontology – the label on an object is an initial step toward a type. There needs to be a way to
associate a potentially rich ontology for recording knowledge about objects. See companion paper on
ontologies [10]

Interface reflection – so we can ask an object for its control interface(s). This information could be
stored directly with the object and/or indirectly via the ontology service.

Dynamic in-world text – if a soft controller is modeled as an in world object inside SL, then it may
need to dynamically display text. There is no direct support beyond importing textures to make this easy.

Alternatively, the soft controller display could replace the PIE to dynamically display commands – see
[11]. Other displays are possible - SL supports maps, mini-maps and other views and interfaces including
Skype access so you use cell phones to can talk to in-world friends. For assemblies, commands can operate
on the head object. For collections, one could use a menus or a dashboard or a 3D envelope or a plural noun
(e.g., the lights in the bedroom) to select and command specific groups of smart objects.

We’d also need interfaces for specifying the rules, policies, and workflow mentioned earlier. See [12].
The ontology service deserves special mention because, if there are thousands of smart object types, then the soft
controller will need a standardized way to discover and talk to them all. See [10] for more discussion on this topic.
Prototype
One of the authors, McCrary, developed an architecture for the interaction between a soft controller and a generic
smart device in SL. There are two parts to the architecture: scripts in the soft controller and scripts in the smart
devices. The scripts in the soft controller use a keyword to ask for all smart devices to respond with their names and
unique ID. The smart devices recognize the keyword in the message from the soft controller and then send chat
messages broadcasting their names, unique ID, and another keyword that signifies them as smart devices.
A soft controller’s script then takes all of the smart device names that it received and arranges them into a graphical
display so that the user can choose which smart device to control. Meanwhile, the smart devices listen for their
name and unique ID. The smart device that is selected then sends a list of commands to the soft controller. The soft
controller arranges these commands into another graphical interface. The user selects the desired command; this
sends a message to that smart device triggering the appropriate action.
Other soft controllers were developed in our smart devices project [2] and in our next generaion PIE project [11].
About the author
Jonathan McCrary is a first year Computer Science graduate student in the Computer Science and Computer
Engineering Department at the University of Arkansas. He began work on a soft controller project in Second Life in
Fall 2009 [13]. Craig Thompson heads the Everything is Alive pervasive computing project at the University of
Arkansas which initially focused on agent architectures, then RFID middleware and more recently on virtual worlds
as a window to understand the future world of pervasive computing [14].
References
[1]
C. Thompson, “Smart Devices and Soft Controllers,” Architectural Perspective Column, IEEE Internet
Computing, January-February 2005, http://csce.uark.edu/~cwt/DOCS/2005-01--PAPER--IEEE-InternetComputing--Smart-Devices-and-Soft-Controllers.pdf
[2]
A. Eguchi, C. Thompson, , “Smart Objects in a Virtual World,” X10 Workshop on Extensible Virtual
Worlds, venue: Second Life, 29 March – 2 April 2010, http://vw.ddns.uark.edu/X10.
[3]
“Star Trek Communicator,” Wikipedia, http://en.wikipedia.org/wiki/Communicator_(Star_Trek)
[4]
“Harmony Remote Controls,” http://www.logitech.com/index.cfm/remotes/universal_remotes/&cl=roeu,en
[5]
“Android,” http://en.wikipedia.org/wiki/Android_(operating_system)
[6]
“Turn your iPhone into a Universal Remote Control,” http://l5remote.com/
3
X10 Workshop on Extensible Virtual Worlds
Position Paper
[7]
A. Heining, “RFID on the next iPhone? Why?”, Christian Science Monitor,
http://www.csmonitor.com/Innovation/Horizons/2009/1110/rfid-on-the-next-iphone-why
[8]
M. Duba, “Extending an iPhone to Control a Smart Home,” Term project, Spring 2009, Fall 2009, CSCE
Dept., University of Arkansas, http://www.csce.uark.edu/~cwt/COURSES/2009-01--CSCE-5043-AI/TERM-PROJECTS/2009.htm#_Using_an_iPhone
[9]
S. Krishna et al., “Mirror Worlds,” X10 Workshop on Extensible Virtual Worlds, venue: Second Life, 29
March – 2 April 2010, http://vw.ddns.uark.edu/X10
[10]
T. Censullo, “Semantic World: Ontologies for the Real and Virtual World,” X10 Workshop on Extensible
Virtual Worlds, venue: Second Life, 29 March – 2 April 2010, http://vw.ddns.uark.edu/X10
[11]
T. Kumar, “Beyond the PIE: Communicating with Smart Objects using Menu-based Natural Language
Interfaces,” X10 Workshop on Extensible Virtual Worlds, venue: Second Life, 29 March – 2 April 2010,
http://vw.ddns.uark.edu/X10
[12]
Business Rule Management System, Wikipedia, http://en.wikipedia.org/wiki/BRMS
[13]
Jonathan McCrary et al., “Soft Controller,” Term project in Artificial Intelligence, Fall 2009, CSCE Dept.,
University of Arkansas, http://www.csce.uark.edu/~cwt/COURSES/2009-08--CSCE-4613--AI/TERMPROJECTS/FINAL-REPORT--Soft Controller--McCrary-Jones-Estorninos-Zantow.doc
[14]
Everything is Alive project website (aka Modeling Healthcare Logistics in a Virtual World), University of
Arkansas, http://vw.ddns.uark.edu
Potential Discussion Topics


If we humans could talk to (smart networked) objects (like cars, thermostats, chairs, pets, hospital
equipment)
o
what would we talk about?
o
how would we express ourselves? e.g., English, chat commands, GUIs, menus, dashboards, …
o
what kinds of responses could we expect?
What would we need to add to a smart phone to make it into a universal remote control capable of
controlling an entertainment system, RFID tagged items, sensors, actuators (networked objects that have an
API), collections of items, … any object the controller discovers?

What standards would be needed to make all soft controllers interoperable with all smart objects?

What are the R&D issues and directions related to soft controllers?
4