COMP327
Mobile Computing
Lecture Set 7 - Context Aware Systems
1
In this Lecture Set
•
•
Context Aware Systems
•
•
•
•
Modelling Context Awareness
Context Dependent Mobile Scenarios
Spatial Awareness
Augmented Reality
Services and Service Discovery
•
Jini, UPnP, etc
2
Context Aware (CA)
Systems
•
Systems that are aware of their own situation in the physical,
virtual, and user environment.
•
•
e.g. a “travel guide” that only provides information on buildings near to the user’s
location, or a “memories” application that identifies and focusses on familiar faces when
photos are taken
Typically adapt their operation or goal based on contextual cues
from the environment or the user’s actions
•
•
•
Implicit behaviour, rather than explicitly directed by the user
Lessen the (cognitive) load on the user
Requires ability to not only sense environment, but to determine what is
relevant to the system’s task(s)
•
•
May require additional processing to convert raw sensor data into relevant information
May also need a representation of the relevant knowledge model used to comprehend
contextual cues and direct behaviour
3
Classifying CA systems
•
Various ways of classifying CA software:
•
•
•
•
•
•
Proactive Triggering
•
e.g. performing some interaction based on environmental perceptions
•
Reducing irrelevant information - e.g. travel guide for current location
•
Contextual retrieval, e.g. based on spatial or feature-based cues
•
e.g. tagging details regarding current context for future access
•
e.g. changing interface based on situation (from screen to text-to-voice)
•
Social networking based on shared contexts
Streamlining Interaction
Memories for past events
Reminders for future contexts
Optimising patterns of behaviours
Sharing Experiences
4
Classifying CA systems
•
•
Passive context-aware systems
•
New context is presented to the user, to inform them of change.
•
•
User can then explicitly determine if the use of an application should change
Examples include
• Changing information regarding environment (weather meters)
• Context-based tagging (e.g. by camera)
Active context-aware system
•
Behaviour of the applications change automatically
•
Examples include
•
•
•
Task filtering (e.g. information filtering, based on current wireless network speed)
Context-base task activation (e.g. routing)
Content adaptation (e.g. for people with disability)
5
Context Creation
•
New contexts can be created based on sensor
data (captured by the device or nearby sensors)
•
Lower-level raw contexts may need to be processed
into higher level contexts
•
Raw data may need to be scaled or transformed
•
•
•
e.g. electrical signal on a temperature gauge should be converted into a
Celsius value...
...or absolute geo-location position should be converted into an address
or identification of a building
Often this abstraction is more useful
•
e.g. “this photo was taken at my parents home last christmas”
6
Context Creation
•
Some contexts can be derived from other contexts
•
•
•
•
By combining several contextual cues, a better understanding of the
context can be determined, than considering the cues in isolation
The same cues might mean different things
•
A user in front of a building may be admiring the architecture, or waiting for a friend
Good for implicit HCI - iHCI systems
•
e.g. knowing the current location, current time, and having access to the
user’s calendar, an app can infer the user’s context
•
i.e. waiting for a flight, in a meeting, or even en-route, but running late.
Cues may be heterogenous in format
•
•
Currently over 100 location coordinate systems are in use
Need mechanisms to resolve heterogeneity within open environments
7
Approaches for
determining user contexts
•
Combining several simpler contexts - Context Composition
•
•
•
•
•
The joint inference of simpler contexts can imply other contexts
•
May require interoperation and mediation
•
Getting several independent values about the same thing
•
Additional cues may disambiguate contexts suggested by other cues
•
e.g. by knowing someone’s weight and location, an identity may be derived
•
e.g. a physical location may be derived from more general information such as
the street name and building name given by a third party.
Combining homogeneous contexts
Combining heterogeneous contexts
Deriving high-level context from lower-level ones
Deriving low-level contexts from higher-level ones
8
Approaches for
determining user contexts
•
Combining several simpler contexts - Context
Composition
•
•
•
(see previous slide)
Consulting a user profile or preferences
•
Checking the user’s calendar - not always reliable as the user
may deviate...
Using historic data
•
Observing user behaviour and building an implicit user profile
9
Context Representation
•
To identify contexts, one needs knowledge about a
scenario, domain, or environment
•
•
•
Supports reasoning based on individual cues
•
Inference may be processor heavy
Generally application or task specific
•
•
•
Cues can be limited
Granularity of knowledge often restricted to the problem
Scalability is very challenging
Six separate forms of context representation were
identified by Strang and Linnhoff-Popien (2004)
10
Different knowledge representation
schemes for contextual knowledge
•
•
•
Key-Value pairs
•
•
•
•
Simple, flat data structures
Accessed through linear search
•
•
•
•
Hierarchical data structure (e.g. XML)
Accessed using a markup query language
Pros: Distributed model, uses W3C standards, XML web services are becoming pervasive
Cons: weak formalism
Pros: Easy to manage and parse in embedded systems
Cons: Uses Exact matches, lacks expressive structure, has weak formalisms
Mark-up Scheme
Graphical
•
•
•
•
Graph structures, such as UML.
Accessed through transformation algorithm
Pros: More expressive than key-value and hierarchies
Cons: lack of formalism for online automated access
•
Caveat - graphical representation languages such as RDF support ontologies (see next slide)
11
Different knowledge representation
schemes for contextual knowledge
•
•
•
Object-oriented
•
•
•
•
Context processing is encapsulated. Access is through well-defined interfaces
Accessed through accessor methods
•
•
•
•
A logic defines the conditions in which a concluding expression or fact may be derived
Accessed through logical reasoning
Pros: Strong formalism, expressive structuring
Cons: handling uncertainty, heterogeneity and temporal elements is challenging. High complexity
Pros: Distributed OO is mature. Reuse supported through inheritance and composition
Cons: handling incompleteness
Logic based
Strong Ontology
•
•
•
•
A combined expressive conceptual model with an axiomatic representation (logic, rules, etc)
Accessed through ontological reasoning (may include rule execution or logical inference)
Pros: expressive structuring, managing heterogeneity
Cons: handling uncertainty, and scalability issues
12
Context Architectures
• A basic context-aware model consists of
four components:
1.
2.
3.
4.
Current context capture (context determination)
Goal context creation
Adaptation of the current context to goal context
Context management
• This framework can be further decomposed
13
1. Context Determination
•
•
•
•
Acquisition of environmental cues
•
may require configuration, e.g. frequency of samples
Acquisition of user contexts
•
e.g. user identity, stereotype recognition, preference retrieval
Encapsulation and Abstraction
•
•
convert raw data into meaningful information
harmonise heterogeneous sources into homogenous data
Filtering of environmental context to user context
•
Identify what is relevant for further use!
14
2. User Context Acquisition
•
•
•
Determine user’s goal context
•
Could be based on a list of the user’s application
tasks or desired goals
•
e.g. todo list
Policy or constraint acquisition
•
Based on user’s tasks, what should be mediated
based on the environmental context (from point 1)?
Encapsulation and abstraction of user contexts
15
3. Context Processing
•
Determine any synergies or compositional implications of the
contextual cues
•
•
•
An application may govern the use of multiple environmental contexts
from different sources
Mediate and interoperate between difference contextual
representations
•
Identify a suitable representation to facilitate application-based reasoning
•
Passive: use the context to constrain user input or queries, or what
information is presented to the user.
Adapt the application with respect to the context
•
Active: adapt application to the context, proactively performing tasks
(e.g. pulling data and/or presenting information to the user).
16
4. Context Management
•
•
•
•
Discovery / Repository
•
Directory services enable context stores, sources, and users, to be
registered and discovered
Storage
•
Store context data in a repository (local or otherwise). May be
historical, and used to further adapt the application or user profile in
future
Sharing of environmental and goal contexts
•
For distribution and access
Access Control
•
Protect the privacy of any user-oriented data, or contextual
information that could be used to identify a user
17
Challenges in Context
Awareness
•
There are many challenges in modelling contexts
•
•
Environmental cues may be inaccurate or erroneous
•
Lack of alignment with cues and the internal
representation of contexts
•
•
The user contexts may be incorrectly determined or
predicted, or just ambiguous
The use of contexts may reduce user privacy
Awareness of context shifts or changes in application
may overload or distract the user
18
Examples of Context
Awareness for Mobile Devices
•
•
The Active Badge Location System
•
Call Routing for Mobile Users
•
•
Initially in 1989, published by Want et. al. 1992
•
Calls could then be routed to users through the nearest telephone extension
Users were tracked around a building using active badges and embedded
sensors in the environment
Electric Elves
•
Agent-based Mobile Personal Assistant
•
•
•
Deployed for 6 months in 2000 (Tambe et al, 2005)
Agents would act on behalf of their users coordinating routine office tasks
Exploited an awareness of factors such as user’s location and calendar to
make decisions (e.g. nominating speakers for seminars, or delaying meetings)
19
Spacial Awareness
•
•
•
•
Often considered one of the main drivers for mobile services
•
•
Often referred to as Location-Aware Systems
However, often the notions of orientation and of nearby objects (as
polygons) are also significant
Services can be related spatially
•
•
e.g. ordered in terms of proximity on a list
located on a localised map
Peer to Peer based services can also relate the proximity of
users to each other, rather than to any fixed geographical point
Services may also be discovered locally, and then use a local
PAN (rather than internet-based) communication
20
Examples of Spatially
Aware Applications
•
•
•
•
•
Navigation
•
Where is the nearest coffee shop? Give me directions...
•
My route is congested - is there a better route?
•
Tell me about the building in front of me
•
I’m having a heart attack! My car has broken down!
•
Download Starbucks tune of the week. Special offer on restaurant X
Context Change
Query Location Context
Personal Emergency
Location/Time based offers
21
Examples of Spatially
Aware Applications
•
Asset Tracking:
•
•
•
•
•
Enterprise:
•
Why does it always take so long to deliver to customer X?
•
How long before the next bus will arrive at this bus-stop?
•
Where has Smudge the cat gone? My car has been stolen - can it be tracked?
Public:
Personal:
Location Tagging
•
Where were those photos taken?
Location and time synchronisation
•
Used can share and synchronise their context (location, activity, mood)
to arrange ad-hoc meetings
22
Location Acquisition
•
Various approaches based on:
•
•
•
•
triangulation
•
based on angulation (bearing) given two or more fixed points
lateration
•
determines distance from fixed points - e.g. determining Time of
Arrival
proximity
•
recognising RFID tags or being within a WIFI cell
scene analysis
•
image recognition of iconic buildings, or text recognition of street
names
23
Triangulation
•
•
•
The process of measuring the location of a point by measuring the angles to it from
known points.
•
Uses law of sines (trigonometry)
If A and B are fixed points, and the location of O is known if the following are
known:
O
Distance AB
Angles at A and B
a
Given that:
Y?
b
sin A = Y/a
sin B = Y/b
•
cos A = X/a
Can calculate:
A
Y = a * Sin A which is also equal to Y = b * Sin B
B
X?
X = a * Cos A
X = AB - b * Cos B
24
Lateration - by
Attenuation
•
Attenuation can also be used to determine distance
based on known transmitter strength (i.e. Received
Signal Strength - RSS)
•
•
Uses square law of attenuation (i.e. 1/d2)
•
Used with WLAN access points to get an accuracy of 1-7m
•
However, attenuation can be variable due to moisture,
reflections, spot interference etc
•
Depending on access point density, reflections etc
Bluetooth has also been explored as a comms system for
determining location
•
Very noisy, but can be used to segment a space into overlapping cells.
25
Multilateration
•
Multilateration (hyperbolic positioning) determines the location of an object
•
Computes the time difference of arrival (TDOA) of a signal to an object from
three or more synchronised receivers
•
•
•
•
•
•
A pulse sent from a transmitter will arrive at slightly different times at two different receivers
If the locations of the receivers are known, then the time difference can be used to determine possible
locations of the receivers (depending on original signal strength, etc).
Two signal case
•
The locus of two transmitter locations forms a two sheeted hyperboloid (see opposite)
Three signal case
•
•
•
Provides a second TDOA measurement and hence locate the emitter on a second hyperboloid
The intersection of this and the previous hyperboloid gives a curve on which the the emitter lies
Four signal case
A third TDOA determines a third hyperboloid, and a unique position in 3D space
By using synchronised transmitters at known locations, a receiver can
determine its location.
Image taken from Wikipedia, at http://en.wikipedia.org/wiki/File:HyperboloidOfTwoSheets.png
Creative Commons Attribution ShareAlike 3.0 License
26
Mobile Phone
Localisation
•
Determines the location of a phone based on power levels of signals and
antenna patterns
•
•
•
Doesn’t need active calls to work
Can achieve a precision of 50 meters in urban areas with high density of cells
•
Lower precision in rural areas, where cell towers are few and far between!!!
Three approaches
•
•
•
Network Based
•
•
•
•
Relies on the Provider’s network to (non-intrusively) identify the location of the handset.
Cell identification (i.e. knowing which cell the phone is in) gives a course-grained precision.
GSM uses trilateration to determine a better location precision.
Used by legislative frameworks such as E911
Handset Based
•
•
•
Relies on applications running on the phone itself to determine position using various methods
Including GPS, cell identification, signal strengths of different towers, etc.
These are then processed, and sent to the receiver
Hybrid Systems
•
Combines both network and handset-determined positioning to get the best accuracy
•
May vary the mechanism used depending on local conditions and available signals
27
iPhone Location Services
•
iPhone makes use of a combination of GPS, GSM
and Wi-Fi positioning methods
•
•
GPS is used whenever possible
•
Can be inaccurate indoors or due to urban canyons (e.g. in New York)
•
This relies on a large database of known WiFi access points
Exploits Skyhook’s WiFi Positioning System (WPS)
•
•
•
•
•
See www.skyhookwireless.com for more details
The MAC address and Received Signal Strength is used from each detected
Wifi Point to generate an approximate location through trilateration
Good accuracy, but dependent on high density of Wifi access points
Prone to error if Access Points are moved (e.g. consumers change
addresses!!!)
Uses GSM localisation based on cell towers
28
Trilateration - by TOA
•
Determines the intersection of three sphere surfaces given the centres and
radii of three spheres!
•
•
Correlates the time the signal was sent with the time it arrives
•
Assuming the signal propagation can be estimated; d = time * signal propagation speed
Assuming that spheres are translated so that P1 is on the origin, and P2 is on
the x axis, and assuming the equations for three spheres:
r12 = x2 + x2 + z2
r22 = (x-d)2 + y2 + z2
•
r32 = (x-i)2 + (y-j)2 + z2
Can determine a point (x,y,z) satisfying the above eqns.
r2
x = (r12 - r22 - d2) / 2d
y = ((r12 - r32 + i2 +j2) / 2j) - x(i/j)
z = ±√(r12-x2-y2)
(0,0) P1
d
r1
P2
(d,0)
j
P3
r3
(i,j)
29
Global Positioning
System
•
Positioning based on satellites in medium earth orbit
•
•
•
•
Between 24-32 satellites, launched between 1989 and 1994
Used to determine latitude, longitude, altitude (and time)
Each satellite transmits:
•
•
•
the time the signal was sent
precise orbital information of that satellite
general information of all the GPS satellite status (the almanac)
Trilateration is used to combine the distances to determine
location
•
Due to possible error propagation, positioning is improved by using
additional satellite signals
30
Assisted GPS
• A network carrier dependent GPS system
•
Improves startup performance of GPS within
mobile phones by using additional information
sent by the carrier
•
•
•
Orbital data and/or almanac for GPS satellites, to help lock
onto signals more rapidly
Atomic time information
Accurate surveyed coordinates of the cell tower location
can be used to determine ionosperic conditions, and
correct for errors
31
Proximity Analysis
•
Makes use of short-range transceivers at fixed
positions
•
•
Determines when tags such as RFIDs are within range
Typically determine relative distance, as opposed
to absolute positions
•
•
Typically used for asset tracking
Can trigger location events for tourist guides
•
e.g. in museums or galleries
32
Scene Analysis
• Relies on the abstraction of image markers
from terrain scenes
•
•
These are then cross-correlated from a database
of known markers
•
E.g. church spires, canals, railway lines etc
Good for orientation, given a rough location
•
Can be used for absolute positioning, but requires
significant image information on unique buildings!
33
Orientation and
Magnetometers
•
The most common used in mobile devices are solid-state Hall
Effect sensors
•
These sensors produce a voltage proportional to the applied
magnetic field and also sense polarity.
•
•
•
The movement of electrons is curved when a magnetic field is perpendicular to the
direction of the charge through a conductor
The AK8973 (used by the iPhone) is a 3 axis magnetometer
•
Thus, can be used to determine the earth’s magnetic field
irrespective of orientation
Orientation can be used to determine orientation of the
phone
•
Compass, rotate maps, or determine the orientation of the camera
34
Augmented
Reality
•
Combining live direct (or indirect) view of a physical realworld environment with virtual CGI
•
•
Typically in real-time using the semantics of environmental elements
Identify digital “cues” (e.g. 2D barcode) and overlay with a digital
image
•
•
Video camera identifies orientation of the cue, and replaces it with the image,
transforming it in 3 dimensions
Current phone applications exploit location, device
orientation (using accelerometer) and compass direction
(using magnetometer) to overlay information on a video feed
•
Future applications could capture and process live video images
35
Service Discovery
•
•
Historically, devices have required specific interfaces or
drivers to connect to other devices
•
e.g. offering printing or visualisation services
With the increasing number and diversity of device
types, managing individual services is no longer feasible
•
•
Discovery architectures have emerged to support the
dynamic (run-time) discovery and use of services offered by
other devices
Particularly important with standardised bus services (e.g.
firewire or USB), or wireless (PAN) communication
•
e.g. Bluetooth
36
Capabilities supported
by discoverable devices
•
Ability to:
•
•
•
•
•
make other devices aware of its own existence and
presence
advertise the services it offers, and to provide
additional information when necessary
dynamically search and use other services
Requires zero administration
Interaction with other devices in the
environment fulfils some goal
37
Context and Service
Discovery
•
Services (and/or devices) may be contextual, and usable
within those contexts
•
•
•
Location or availability of services, bound to a physical device
•
e.g. printers, projectors, scanners etc
•
e.g. tourist information services, support for augmented reality
Services that provide location information to a mobile device
Many existing services rely on determining location,
then searching for services relevant to that location
•
Alternatively, devices could locate localised services
•
This is already done for communication services such as WiFi
38
Universal Plug
and Play
•
UPnP originally emerged from Microsoft's Plug
and Play mechanism
•
•
•
Uses established standards such as TCP/IP, UDP and
HTTP for networking, communication and broadcast
XML and SOAP are used for higher level comms.
Devices can:
•
•
•
dynamically join a network and obtain an IP address
announce themselves and convey their capabilities upon request
learn about the presence and capabilities of other devices
39
UPnP features
•
•
•
•
Media and device independence
•
•
over ethernet, firewire, IrDA, RF (including Bluetooth and WiFi) etc.
•
use and configuration is via a web browser
no device drivers necessary; common protocols are used instead
User Interface Control
OS and programming language independence
•
no constraints on the design for the APIs
Extensibility
•
in addition to the basic architectures, vendors can define their own
devices and services
40
UPnP Discovery
•
Assumes that one or more control points (centralised registries)
•
•
These store information about devices and their services
When a new device appears on the network, it broadcasts the
services it offers
•
•
A control point can then choose whether or not to register these services
New control points search for services by sending messages to
devices
•
Devices that match the service discovery request responds to the
new control point
•
Defined by the Simple Service Discovery Protocol (SSDP)
41
UPnP Description
• During discovery, a URL to a description of
a device is sent
•
•
•
Includes:
•
•
•
vendor specific details (name, model, manufacturer)
serial number
etc
Also provides URLs for details on:
•
Control, Eventing and Presentation
Forms part of the UPnP Device Template
42
UPnP Control
• The control point retrieves details of the
devices services from the control URL
•
•
Each service description includes
•
•
•
•
A list of actions which the service offers
The interfaces for each action
•
i.e. the arguments it takes
Service states are also represented as a set of variables
Control messages are expressed in XML over SOAP
Defined by the UPnP Service Template
43
UPnP Eventing
•
The state variables of each service can change
•
•
•
Notification of these changes can be obtained by
sending a subscription request to the device
The device returns the duration of the subscription
•
The control point is responsible for renewing the subscription
when it expires
Some state variables are not evented
•
•
Due to rapid variations or high dynamicity
Values can be polled on demand
44
UPnP Presentation
• Device information and control can be
loaded from a URL
•
•
It is then presented to the user
The user can also control the device through a
browser
• No direct support for service invocation is
provided by UPnP
45
Digital Living
Network Alliance
•
DLNA is a consumer electronics standard to support
discovery and sharing of content across a home network
•
Various types of devices
•
•
•
•
•
Digital media servers (DMS) - e.g. home PC, Network Attached Storage (NAS)
•
Serve content to other devices over the network
•
Discover content from DMS and provide playback and control
•
Control media sent to a DMR
Digital media players (DMP) - e.g.TV, stereo, projector
Digital media controlers (DMC) - e.g. mobile device, digital cameras
Digital media renderers (DMR) - e.g. video displays, remote speakers
• Render content that is controled by a DMC
Upload/Download capabilities also exist for mobile devices
46
Jini
•
Service Discovery system based on Java
•
Assumes a number of distributed devices running a Java virtual
machine
•
•
RMI (Remote Method Invocation) ius used to call objects which reside on
other devices.
Discovery and Join mechanism (similar to UPnP) allow
services to advertise themselves
•
•
Lookup services act as service repositories (similar to control
points)
Devices advertise services for a limited time; to remain they
need to renew their lease
•
Ensures that the repository does not become stale
47
Jini Discovery
• A service requester client uses the lookup
protocol to find a service
•
•
When one is found, its corresponding service
object is downloaded into the client
This object is typically a thin interface that
supports RMI calls to the service itself
• Provides better service lookup than UPnP,
as well as service invocation
48
Exercises...
•
•
•
Describe why context can be relevant for mobile
applications, and detail two different scenarios where mobile
applications could be used within a tourist environment.
Discuss the privacy and ethical issues of location
determination. Is it acceptable for a provider to be able to
determine the location and context of a customer when an
incident occurs?
Why is there a need for service discovery by mobile
devices? Discuss how devices could discover other services
within a home environment, and speculate how a mobile
device might be used to control the output of a rendering
device
49
To Recap...
• In this lecture set, we covered:
•
•
•
The advantages and usage of context
•
•
How context can be represented or constructed
The challenges of developing context aware systems
Spatial Awareness
•
•
•
How it can be used
How location can be determined
Other spatial modalities (e.g. compass direction)
Service Discovery
•
UPnP, DLNA and Jini
50
Further Reading
• Ubiquitous Computing: Smart Devices,
Environments and Interactions
Stefan Poslad (Wiley, 2009)
Chapter 7
•
Computing
• Pervasive
Uwe Hansmann, Lothar Merk, Martin S. Nicklous and
Thomas Stober (Springer, 2003)
Chapter 15
•
• Wikipedia !!!
51