Telepresence:
An Umbrella Research Topic
Jim Gray
Microsoft Research
Gray@Microsoft.com
http://research.Microsoft.com/~Gray/
1
Federal Research Support
Nerve Center of Science
If it’s not broke, don’t fix it.
But….




US Science is the engine of progress
BUT…..
Best and brightest are spending
increasing time fund raising
Seems excessive to me.
Venture capital community is
THE
LONG
richer and
BOOM
more generous
than Federal Support
2
Cyberspace is a New World.


We have discovered a “new continent”.
It is changing how we learn, work, and play.
– 1 T$/y industry
– 1 T$ new wealth since 1993
– 30% of US economic growth since 1993



There is a gold rush to stake out territory. THE
LONG
But we also need explorers:
BOOM
Lewis & Clark expeditions
Universities to teach the next generation(s)
Governments, industry, and philanthropists
should fund long-term research.
3
Research Investments Pay Off
CSTB –NRC Evolving the High-Performance Computing and Communications Imitative to Support the nations Information Infrastructure, NA Press,
Washington DC, 1995.
1960
1970
Time-sharing
1980
1990
Government funded CTSS, Multics, SSD
Unix
Industrial
Billion Dollar/year Industry
SDS 940, 360/67 VMS
Graphics
Sketchpad, Utah
GM/IBM, LucasFilm
E&S, SGI, PIXAR,..
Networking
Arpanet, Internet
Ethernet, Pup, Datakit
DECnet, LANs, TCP/IP
Workstations
Windows
Lisp machine, Stanford
Xerox Alto
Apollo, Sun
Englebart, Rochester
Alto, Smalltalk
4
Star, Mac, Microsoft
Research Investments Pay Off
1970
1980
Relational Data Bases
1990
2000
Berkeley, Wisc,…
IBM
Oracle, IBM,…
Parallel DBs
Tokyo,Wisconsin, UCLA
ICL, IBM
ICL, Teradata, Tandem
Data Mining
(complex queries)
Wisc, Stanford, …
IBM, Arbor,…
IRI, Arbor, Plato, …
5
Why Can’t Industry Fund IT Research?

It does: IBM
(5.8%),
Intel
(13%),
Lucent
(12%),
Microsoft
(14.%)
, Sun
(12%)
, ...
– R&D is ~5%-15% (50 B$ of 500 B$)

AD is 10% of that (5 B$)
– Long-Range Research
is 10% of that 500 M$
2,500 researchers and university support
– Compaq: 4.8% R&D (1.3 B$ of 27.3 B$).AOL: 3.7% D, ?R (96 M$ of 2.6 B$)
– Dell:1.6% R&D
EDS, MCI-WorldCom, ….
(204 M$ of 12.6 B$),

To be competitive, some companies
cannot make large long-term research investments.
The Xerox/PARC story:
created Mac, Adobe, 3Com…
6
PITAC Report
Presidential IT Advisory Committee
http://www.ccic.gov/ac/report/

Findings:
– Software construction is a mess: needs breakthroughs.
– We do not know how to scale the Internet 100x

Security, manageability, services, terabit per second issues.
– USG needs high-performance computing (Simulation)
but market is not providing vector-supers – just providing processor arrays.
– Trained people are in very short supply.

Recommendations:
–
–
–
–
Lewis & Clark expeditions to 21st century.
Increase long-term research funding by 1.4B$/y.
Re-invigorate university research & teaching.
Facilitate immigration of technical experts.
7
Outline (ambitious!)

Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence

What if you could record everything you see & hear?

The architecture revolution:
processing moves to transducers


8
Microsoft Research -- 1991



Founded in 1991
Goal:
pursue strategic technologies
for Microsoft
Original research groups:
– Natural Language Processing
– Operating Systems
– Programming Languages

Overall size < 20 at the end of 1992
9
Microsoft Research -- 1999

400 Researchers in 25 areas
– Operating systems to Statistical Physics

Research lab locations:
– Redmond, Cambridge, San Francisco, Beijing

Internationally recognized research teams
– Hundreds of publications, presentations
– Leadership roles in professional societies,
journals, conferences
10
MS Research Areas



Operating systems, languages,
compilers, virtual machines, networking,
wireless computing, fault-tolerance,
large scale servers, security
Natural language, speech, vision,
graphics, decision theory, information
retrieval, UI, collaboration, statistics,
signal processing
Cryptography, statistical physics and
discrete mathematics
11
Growing Fast


Grew 20x from ‘92 to ‘99
Decided in ‘97 to grow by a 3x in 3 years
– 200 in FY97 => 600 in FY00,
primarily in Redmond

Major impact on MS products
– Virtually all MS products shipped today use
technology from MS Research

Key role in MS growth
– Pioneering research in software that allows
computers to see, hear, speak and understand
12
Microsoft Research
Philosophy

University organizational model
– Flat structure, critical mass groups

Open research environment
– Aggressive publication of research results
in literature and on world wide web
– Frequent visitors, daily seminars
– Over 100 visiting professors and interns in
1998
– Over 110 visiting researchers in 1998
13
What I Do.

Work for the government!
– CSTB, PITAC(software, ngi), LoC study, ....

Work on scaleable systems:
– 1 Billion Transactions Per Day Cluster
– TerraServer
– New: Sloan Digital Sky Survey
17
Outline (ambitious!)

Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence

What if you could record everything you see & hear?

The architecture revolution:
processing moves to transducers


18
Gordon Bell on
Tele Presentations
http://research.microsoft.com/barc/GBell/
19
Motivation:
Telepresentations
• Presenter and/or
audience
telepresent
NOT: meeting or collaboration
settings
Forget the nasty social
issues!
Mostly one-way
20
Telepresentation
Elements
Slides
 Audio
 Video
 Script,
text
comments,
hyperlinks,
etc.

21
Telepresentations:
The Essentials



Slide and audio a must
Add some video
(low quality)
to make us feel good
Storage and
transmission costs low
22
Telepresentations:
The Killer App



Increased attendance & lower
travel costs
Practical and low-cost NOW
e.g. ACM97 - 2,000 visitors in real
space, 20,000 visitors on Internet
http://research.microsoft.com/acm97
23
Today’s
Experiment


Would you like to pause, rewind, browse?
Do you wish you could have seen this
– At home?
– At another time?

How much does a present speaker add?
How much would you pay for real
presence?
24
University Lectures Online




Research lectures on-line & on-demand
http://murl.microsoft.com/
Will get UVC content
Available to anyone anywhere
– T1 good, 28.8 OK


Generated by CMU, MIT, MSR, Stanford, UW, Xerox
Hosted by MSR
25
Outline (ambitious!)





Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence
What if you could record everything you see &
hear?
The architecture revolution:
processing moves to transducers
26
Changing role of computation

Past: Computers for:
– computing (Cray)
– business data processing (IBM)
– “document” creation (PC)

Future: Computers for:
– understanding & learning
– communicating
– consuming & entertaining

Requires new User Interface to machines
27
Flows
Making “Flows” a Reality

Computer Graphics
– Creating realistic looking environments,
people

Computer Vision
– Analyzing posture, gaze, gestures


Speech input/output
Natural Language
– Analysis, IR

Implicit requests for information
29
Building life-like human
characters
How to fail at Tele-Conferences
1.
2.
3.
4.
Eliminate gaze awareness and
sense of space of a normal group setting
Have long audio latencies &
poor audio quality
Use incompatible equipment
Make it much harder to initiate the call
to make a phone call
31
Gaze Awareness &
Sense of Space
Is anyone
paying attention?
 Who is talking
(where is sound
coming from?

32
Gaze Awareness

Looking at screen:
the forehead shot

Looking at camera:
the glowering shot

Looking at YOU.
33
You can’t just move the eyes

Glowering

Surprise

Boredom

Interest
34
Mona Lisa Effect

Eyes and nose indicated gaze
35
Spatialized Audio & Video
Pointing “nose vector” at target




Map video onto
wire frame
Rotate frame to
point in space
Move (fake) eyes
in frame (>30°)
to point at target
Project voice
on that vector.
36
Recognizing gestures
Live video
Area of
motion
H flow
V flow
Generating life-like speech
from textual data

Data-driven stochastic speech
– Natural sounding
– Rapid, automatic customizability

Examples
– Synthetic voice w/ transplanted speech
contours
38
Artificial singing

AT&T Voder, 1962, by Homer Dudley
– Daisy (Inspiration for HAL’s voice in 2001)

Microsoft Research Whistler, 1997
– Scarborough Fair
39
Analyzing language


Language recognition shipped in Word 97
General purpose text-critiquing,
summarization, Japanese word-breaking
40
Inside The Office
Grammar Checker
41
Understanding language:
MindNet





A huge language
knowledge base
Automatically created
from dictionaries
Words (nodes) linked
by relationships
Millions of links
Recently added
(Encarta)
encyclopedia
knowledge
42
MindNet -- “Going to the birds”
chicken
Is_a
clean
Is_a
smooth
Is_a
poultry
Means
chatter
hen
Is_a
gaggle
Is_a
Is_a
beak
strike
Is_a
bill
egg
bird
sound
Is_a
goose
Part_of
face
Locn_of
feather
wing
Is_a
claw
hawk
Is_a
Is_a
Is_a
Typ_obj
leg
turtle
mouth
limb
Is_a
Part
Typ_subj_of
Is_a
creature
Is_a
Part
fly
Typ_subj
plant
Not_is_a
Is_a
Is_a
Typ_subj_of
Means
meat
Is_a
Is_a
Typ_obj
keep
Typ_0bj_of
Purpose
animal
Typ_subj
make
peck
duck
Cause
quack
Is_a
Typ_obj
Quesp
Typ_obj
Typ_subj
preen
supply
Purpose
catch
Is_a
opening
arm
Changing balance between
user & software systems

Yesterday:
– Applications were single programs running in
isolation
– Users used to (more or less) understand systems
that they used

Today:
– Componentized applications operate in concert
– Sophisticated users understand only small
percentage of systems they use
44
Tomorrow’s Systems and
Applications

Users will not be able to predict
– where computations will be performed,
– when they will be performed or
– by what software components

Gap between system capabilities and
user understanding will grow to the
point that the only way user will be able
to use system is through assisting
agents
45
Examples of user agents &
implicit actions

Lumiere (Office 97)
– Monitoring user and program events to
provide user help and assistance

Implicit queries
– Inferring information needs from browsing

Lookout/SpamKiller
– Monitoring mail activity to auto-categorize it
46
User Modeling

Models of a user’s informational goals
– User’s query (when available…)
– User’s background
– Acute and long-term search activity
– Acute actions with objects and documents
– Program data structures

Explicit and implicit information access
and display
Outline (ambitious!)





Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence
What if you could record everything you see &
hear?
The architecture revolution:
processing moves to transducers
48
A novel
Kilo
A letter
Mega
Library of
Congress (text)
LoC
(sound +
cinima)
All Disks
All Tapes
Giga
Tera
Peta
Exa
A
Movie
LoC
(image)
All
Photos
Zetta
All Information!
Yotta
51
Alan Newell’s & Michael Lesk’s
Points
www.lesk.com/mlesk/ksg97/ksg.html

Soon everything can be recorded and kept

Most data will never be seen by humans

Precious Resource: Human attention
Auto-Summarization
Auto-Search
will be a key enabling technology.
52
Outline (ambitious!)





Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence
What if you could record everything you see &
hear?
The architecture revolution:
processing moves to transducers
53
Put Everything
in Future (Disk) Controllers
(it’s not “if”, it’s “when?”)
Acknowledgements:
Dave Patterson explained this to me a year ago
Kim Keeton
Erik Riedel
Helped me sharpen
these arguments
Catharine Van Ingen
54
Remember Your Roots
55
Kilo
Mega
Giga
Tera
Peta
Exa
Technology Drivers: Disks




Disks on track
100x in 10 years
2 TB 3.5” drive
Shrink to 1” is 200GB
Disk replaces tape?
Zetta
Yotta

Disk is super
computer!
56
Data Gravity
Processing Moves to Transducers



Move Processing to data sources
Move to where the power (and sheet metal) is
Processor in
– Modem
– Display
– Microphones (speech recognition)
& cameras (vision)
– Storage: Data storage and analysis
57
It’s Already True of Printers
Peripheral = CyberBrick


You buy a printer
You get a
– several network interfaces
– A Postscript engine




cpu,
memory,
software,
a spooler (soon)
– and… a print engine.
58
Basic Argument for x-Disks

Future disk controller is a super-computer.
– 1 bips processor
– 128 MB dram
– 100 GB disk plus one arm

Connects to SAN via high-level protocols
– RPC, HTTP, DCOM, Kerberos, Directory Services,….
– Commands are RPCs
– Management, security,….
– Services file/web/db/… requests
– Managed by general-purpose OS with good dev
environment

Apps in disk saves data movement
– need programming environment in controller
60
The Slippery Slope
Nothing =
Sector Server
If you add function to server
 Then you

add more function to server

Function gravitates to
data.
Everything =
App Server
61
Why Not Everything?
Allow Everything on Disk Server
(thin client’s)

Tried and true design
– Mainframes, Minis, ...
– Web servers,…
– Encapsulates data
– Minimizes data moves
– Scaleable


It is where everyone ends up.
All the arguments against are short-term.
66
Disk = Node




has magnetic storage (100 GB?)
has processor & DRAM
has SAN attachment
has execution
Applications
environment
Services
DBMS
RPC, ...
File System
SAN driver
Disk driver
OS Kernel
68
Technology Drivers:
System on a Chip

Integrate Processing with memory on one chip
– chip is 75% memory now
– 1MB cache >> 1960 supercomputers
– 256 Mb memory chip is 32 MB!
– IRAM, CRAM, PIM,… projects abound

Integrate Networking with processing on one
chip
– system bus is a kind of network
– ATM, FiberChannel, Ethernet,.. Logic on chip.
– Direct IO (no intermediate bus)

Functionally specialized cards shrink to a chip.
69
Technology Drivers: What if
Networking Was as Cheap As Disk IO?
 Disk
 TCP/IP
– Unix/NT
100% cpu @ 40MBps
– Unix/NT
8% cpu @ 40MBps
Why the Difference?
Host does
TCP/IP packetizing,
checksum,…
flow control
small buffers
Host Bus Adapter does
SCSI packetizing,
checksum,…
flow control
70
DMA
Technology Drivers:
The Promise of SAN/VIA:10x in 2 years
http://www.ViArch.org/

Today:
– wires are 10 MBps (100 Mbps Ethernet)
– ~20 MBps tcp/ip saturates 2 cpus
– round-trip latency is ~300 us

In the lab
– Wires are 10x faster Myrinet, Gbps Ethernet,
ServerNet,…
– Fast user-level communication


tcp/ip ~ 100 MBps 10% of each processor
round-trip latency is 15 us
71
SAN:
Standard Interconnect
Gbps Ethernet: 110 MBps

PCI: 70 MBps


UW Scsi: 40 MBps

LAN faster than
memory bus?
1 GBps links in
lab.
100$ port cost
soon
Port is computer
FW scsi: 20 MBps
scsi: 5 MBps
72
Technology Drivers
Plug & Play Software

RPC is standardizing: (DCOM, IIOP, HTTP)
– Gives huge TOOL LEVERAGE
– Solves the hard problems for you:
 naming,
 security,
 directory service,
 operations,...

Commoditized programming environments
–
–
–
–


FreeBSD, Linix, Solaris,…+ tools
NetWare + tools
WinCE, WinNT,…+ tools
JavaOS + tools
Apps gravitate to data.
General purpose OS on controller runs apps.
75
Basic Argument for x-Disks

Future disk controller is a super-computer.
– 1 bips processor
– 128 MB dram
– 100 GB disk plus one arm

Connects to SAN via high-level protocols
– RPC, HTTP, DCOM, Kerberos, Directory Services,….
– Commands are RPCs
– management, security,….
– Services file/web/db/… requests
– Managed by general-purpose OS with good dev
environment

Move apps to disk to save data movement
– need programming environment in controller
76
Summary





Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence
What if you could record everything you see &
hear?
The architecture revolution:
processing moves to transducers
77