JASS `04
2nd Joined Advanced Student School
Ubiquitous
Tracking based
AR-Setups
Benjamin Fingerle
Christian Wachinger
Benjamin Fingerle, Christian Wachinger
1
JASS `04
Procedure
• We will present a scenario in which a user - Gerhard gets into the benefits of an AR-enabled Environment
• First the scenario will be presented from the user’s
perspective
• In a 2nd stage we’ll look behind the surface and see how
the AR-environment can be modeled using the DWARF
Service concept
• Finally we’ll go even deeper into detail and present how
spatial relationships of objects can be represented using
the Ubiquitous Tracking Framework
Benjamin Fingerle, Christian Wachinger
2
JASS `04
The User’s Equipment
• Gerhard wears an Optical See Through Head Mounted
Display (OST-HMD)
• Up on this HMD two 6DOF markers - one optical, one
magnetic - are rigidly mounted
• Additionally a stereo-vision camera is installed on top of
his head
• Gerhard wears special gloves with attached optical
6DOF markers
Benjamin Fingerle, Christian Wachinger
3
JASS `04
A Day in Gerhard’s Life …
• Gerhard strolls down the TUM hallway greeting
colleagues of his through closed doors
• Eventually he reaches his own office and steps in
• Sitting down in front of the desk he reads a virtual
message of his Russian friend Vladimir from St.
Petersburg asking for a chess game
• He starts the RemoteChess application whereupon a
virtual chess board appears aligned on the desk and the
virtual counterpart of Vladimir takes a seat across from
him
Benjamin Fingerle, Christian Wachinger
4
JASS `04
… A Day in Gerhard’s Life
• Suddenly an emergency request for instant help by his
befriended Spanish colleague José shows up
• José asks Gerhard - a renowned surgeon - for advises
regarding a difficult surgery José is currently conducting
• Just where a minute ago the chess board was visible a
3D - Model showing the patient crops up
• Gerhard studies the patient while having a look from
different sides and getting computer tomography images
registered on the patient on demand
• In vivid discussion with José a life is saved
Benjamin Fingerle, Christian Wachinger
5
JASS `04
The Scenario Can Be Modeled With Services
• Services
– Have needs
• Refined by predicates
– Offer abilities
• Refined by attributes
• Connectors
– Offer interfaces for data
exchange
• Service Managers
<service name=“OpticalTracker”>
<need name=“video” type=“VideoStream”>
<connector protocol=“sharedMemory”>
</need>
<need name=“marker” type=“MarkerData”>
<connector protocol=“ObjectReference”>
</need>
<ability name=“markerPose” type=“PoseData”>
<attribute name=“location”
value=“$(marker.location)”>
<attribute name=“identity”
value=“$(marker.identity)”>
<connector protocol=“NotificationPush”>
</ability>
</service>
– One for each network
node
– Detect mutually
satisfying services
– Provide services with
connectors
Context = {location,
identity, activity, time}
can be modeled using Predicates and Attributes
Benjamin Fingerle, Christian Wachinger
6
JASS `04
Services are Specified Using XML
<service name=“HMDOpticalTracker”>
<need name=“
...
</need>
<ability name=“HeadPose” type=“PoseData”>
<attribute name=“location”
value=“$(landmark.location)”>
<attribute name=“identity”
value=“Gerhard”>
<connector protocol=“NotificationPush”>
</ability>
<ability name=“markerPose” type=“PoseData”>
...
</ability>
</service>
Benjamin Fingerle, Christian Wachinger
7
JASS `04
Hallway Services
Service Name
Needs
Abilities
GerhardConfigData
--
landmarkDescription
HallwayConfigData
--
landmarkDescription
HMDCamera
--
videostream
HMDOpticalTracker
videostream
markerPose, HMDPose
HMDVideoShow
videostream
--
WhatsBehind
pose
videostream
ContextEstimator
landmark
context
Benjamin Fingerle, Christian Wachinger
8
JASS `04
Office Services
Service Name
Needs
Abilities
GerhardConfigData
--
landmarkDescription
HMDCamera
--
videostream
HMDOpticalTracker
videostream
markerPose, HMDPose
HMDVideoShow
videostream
--
ContextEstimator
landmark
context
RoomConfigData
--
landmarkDescription
RoomCamera
--
videostream
roomMagneticTracker
landmark
markerPose
roomOpticalTracker
videostream
markerPose
Table
3D-content
--
VirtualChess
chessPartner
3D-content
VirtualSurgery
handPose
3D-content
VirtualCommunication
communicationPartner
3D-content
Benjamin Fingerle, Christian Wachinger
9
JASS `04
Mutually Satisfying Services are Found by ServiceManager
• For each network node (e.g. Room, Hall, …) one
ServiceManager exists
• This ServiceManager observes needs and abilities of
those services belonging to his particular network node
• When a match is found the ServiceManager provides all
involved parties with connector-objects
• These connectors offer push-, pull-, or shared memory
access to abilities via various interfaces
Benjamin Fingerle, Christian Wachinger
10
JASS `04
Matching of Mutually Satisfying Services
Benjamin Fingerle, Christian Wachinger
11
JASS `04
Ubiquitous Tracking - The Formal Layer
• The crucial problem of Augmented Reality
applications is the correct tracking of objects
• It is common to integrate the tracking procedure
into the application
• Different tracking technologies are combined to
get better results
• To release the AR application of tracking and to
enable a seamless integration of new tracking
devices a formal layer is introduced
• The formal framework, called UbiquitousTracking,
forms the formal layer
Benjamin Fingerle, Christian Wachinger
12
JASS `04
Ubiquitous Tracking – The Formal Framework
• Requests to the framework about the spatial relationship of objects
can be send
• The answer delivers the optimal relationship available
Graph-model:
• nodes == objects
• edges == spatial relationships
Benjamin Fingerle, Christian Wachinger
13
JASS `04
Ubiquitous Tracking - The Underlying Graph Model
• Properties of a spatial relationship
– Represents the transformation and translation of the source
coordinate system to the target coordinate system
– Attributes characterizing the quality of the relationship
• Three different types of graphs
– Real relationship graph
– Measured relationship graph
– Inferred Relationship
Benjamin Fingerle, Christian Wachinger
14
JASS `04
Real Relationship Graph
Each pair of objects has at every point of
time a geometric relationship
Benjamin Fingerle, Christian Wachinger
15
JASS `04
Measured Relationship Graph
•
•
•
Estimates of relationships are just available for certain objects
for discrete points of time
Additionally Attributes characterizing the quality of the
measurement exist
Error function describing
the quality of a relationship
Benjamin Fingerle, Christian Wachinger
16
JASS `04
Inferred Relationship Graph
• Knowledge about spatial relationships not just for discrete points of
time is necessary
• Knowledge has to be inferred about the relationship of objects
• Error functions help to find optimal inferences
Benjamin Fingerle, Christian Wachinger
17
JASS `04
What Is a Good Error Function?
Aim: Finding optimal paths in the graph!
= { Latency,
Update frequency
Confidence value
Pose accuray
Time to live }
Benjamin Fingerle, Christian Wachinger
19
JASS `04
Optimisation of Graph Algorithms
•
Precomputing the Data Flow Graphs
– Infrequently changing structure of the spatial relationships
– Infrequently changing Attributes
– No dependency on the pose measurements
•
Grouping Nodes
– Several nodes can be represented by a single supernode
– Enabling level of detail hierarchies
 Faster graph search
What is the coordinate system of the new supernode?
Benjamin Fingerle, Christian Wachinger
20
JASS `04
Security & Safety Aspects
• What if the “WhatsBehind” Service can be used in front
of every door?
• What if your boss knows what you have done this
weekend?
• What if other users know about every place where you
have been this week?
• What if Indra the intern uses the AR setup of Gerhard?
• What if Eve corrupts the data showing you the false
distance to the oncoming bus?
Privacy
• What if you can’t see the car because the latest sports
Restrictions
news occluded it?
Authenticity
Benjamin Fingerle, Christian Wachinger
21
JASS `04
Finding Matches Forms Crucial Performance Issue
Matching problem of high computational complexity
• One Service Manager for each Network Node
• Number of potential matchings grows exponentially with
the number of services available inhibiting scalability
Different strategies for coping with this issue could include
• Heuristics, based on context information, realized as
graph searches on the spatial relationship graph
• Interpretation of matching problem as predicate logic
formula, applying specialized algorithms known from
model checking
• …
Benjamin Fingerle, Christian Wachinger
22
JASS `04
Issues Concerning Graph Representation
Possible ways of storing the spatial relationship graph
include
• One graph-service holding the complete Graph
+ all graph algorithms applicable
contradicts the distributed computing paradigm
• Each Service knows its adjacency
+ complies with distributed computing paradigm
not all graph algorithms applicable
Benjamin Fingerle, Christian Wachinger
23
JASS `04
Issues Concerning Access to the Graph Information
Possible ways of accessing spatial relationships include
• Request formulated as need for spatialRelationship with predicates
source and target, abilities offered by
– Certain GraphInformationServices
+ Graph can be split up between different such services
- different graph search strategies for the same relation cannot coexist
– One service for each relationship instantiated by special
relationShipGenerators
+ relationships instantly available
- different graph search strategies for the same relation cannot coexist
- number of services raises dramatically
• Requesting a graphInformerObject providing an interface
getSpatialRelationShip(source, target)
+ Objects implementing different graph searches can coexist
- centralization
Benjamin Fingerle, Christian Wachinger
24
JASS `04
Open Questions
• How much information about the world do we have to put
in the error function?
• Which graph algorithms have to be applicable?
• Which graph representation allows theses algorithms?
• How to perform context changes?
• How to enable new users to enter AR environments?
• What has then to be calibrated?
Benjamin Fingerle, Christian Wachinger
25
JASS `04
Critical success factors
•
•
•
•
•
Number of AR ready buildings
Development and acceptance of standards
Prices and convenience
Legal issues
Products and applications urging the user to buy
Benjamin Fingerle, Christian Wachinger
26