Telepresence:
An Umbrella Research Topic
Jim Gray
Microsoft Research
Gray@Microsoft.com
http://research.Microsoft.com/~Gray/
1
NSF: Nerve Center of Science
If it’s not broke, don’t fix it.
But….




US Science is the engine of progress
BUT…..
Best and brightest are spending
increasing time fundraising
Seems excessive to me.
Venture capital community is
richer and
more generous
than NSF
2
Outline (ambitious!)

Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence

What if you could record everything you see & hear?

The architecture revolution:
processing moves to transducers


3
Microsoft Research -- 1991



Founded in 1991
Goal:
pursue strategic technologies
for Microsoft
Original research groups:
– Natural Language Processing
– Operating Systems
– Programming Languages

Overall size < 20 at the end of 1992
4
Microsoft Research -- 1998

280 Researchers in 25 areas
– Operating systems to Statistical Physics

Research lab locations:
– Redmond, Cambridge, San Francisco

Internationally recognized research
teams
– Hundreds of publications, presentations
– Leadership roles in professional societies,
journals, conferences
5
MS Research Areas



Operating systems, languages,
compilers, virtual machines, networking,
wireless computing, fault-tolerance,
large scale servers, security
Natural language, speech, vision,
graphics, decision theory, information
retrieval, UI, collaboration, statistics,
signal processing
Cryptography, statistical physics and
discrete mathematics
6
Growing Fast


Grew 4x from ‘94 to ‘97
Decided in ‘97 to grow by a 3x in 3 years
– 200 in FY97 => 600 in FY00,
primarily in Redmond

Major impact on MS products
– Virtually all MS products shipped today use
technology from MS Research

Critical role in MS growth
– Pioneering research in software that allows
computers to see, hear, speak and understand
7
Microsoft Research
Philosophy

University organizational model
– Flat structure, critical mass groups

Open research environment
– Aggressive publication of research results
in literature and on world wide web
– Frequent visitors, daily seminars
– Over 70 visiting professors and interns in
1997
– Over 110 visiting researchers in 1998
8
Some Key Senior Researchers




Systems
– Rick Rashid, Butler Lampson, Gordon Bell
– Anoop Gupta, Roger Needham, Chuck Thacker
Databases & Data Mining
– David Lomet, Jim Gray, Usama Fayyad
Graphics
– Jim Kajiya, Jim Blinn, Alvy Ray Smith, Michael Cohen
Speech & Language
– Karen Jensen, George Heidorn, X.D. Huang, Alex
Acero, Hsiao-Wuen Hon, Scott Meredith
9
Some Key
Senior Researchers




UI Design, Intelligent Systems, IR
– George Robertson, Linda Stone, Susan Dumais, David
Heckerman, Eric Horvitz, Jack Breese
Computer Vision & Signal Processing
– Steve Shafer, Rick Szeliski, P. Anandan, Rico Malvar
Cryptography & Theory
– Yacov Yacobi, Jennifer Chayes, Christian Borg,
Michael Freedman
Languages & Compilers
– Daniel Weise, Chris Fraser, Amitabh Srivastava, Luca
Cardelli, David Hanson, Charles Simonyi, Todd
Proebsting
10
Microsoft Research

1997 BusinessWeek Poll of Academia:
– Voted #7 lab (overall) in Computer Science
– Voted #3 industrial research lab
(after Bell Labs and IBM Research)
– Voted #2 most desirable lab to work
(after Stanford)
11
Outline (ambitious!)

Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence

What if you could record everything you see & hear?

The architecture revolution:
processing moves to transducers


12
Gordon Bell on
Tele Presentations
http://research.microsoft.com/barc/GBell/
13
Motivation:
Telepresentations
• Presenter and/or
audience
telepresent
NOT: meeting or collaboration
settings
Forget the nasty social
issues!
Mostly one-way
14
Telepresentation
Elements
Slides
 Audio
 Video
 Script,
text
comments,
hyperlinks,
etc.

15
Telepresentations:
The Essentials



Slide and audio a must
Add some video
(low quality)
to make us feel good
Storage and
transmission costs low
16
Telepresentations:
The Killer App



Increased attendance & lower
travel costs
Practical and low-cost NOW
e.g. ACM97 - 2,000 visitors in real
space, 20,000 visitors on Internet
http://research.microsoft.com/acm97
17
Today’s
Experiment


Would you like to pause, rewind, browse?
Do you wish you could have seen this
– At home?
– At another time?

How much does a present speaker add?
How much would you pay for real
presence?
18
Outline (ambitious!)





Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence
What if you could record everything you see &
hear?
The architecture revolution:
processing moves to transducers
19
Changing role of computation

Past: Computers for:
– computing (Cray)
– business data processing (IBM)
– “document” creation (PC)

Future: Computers for:
– understanding & learning
– communicating
– consuming & entertaining

Requires new User Interface to machines
20
Flows
Making “Flows” a Reality

Computer Graphics
– Creating realistic looking environments,
people

Computer Vision
– Analyzing posture, gaze, gestures


Speech input/output
Natural Language
– Analysis, IR

Implicit requests for information
22
Building life-like human
characters
Recognizing gestures
Live video
Area of
motion
H flow
V flow
Generating life-like speech
from textual data

Data-driven stochastic speech
– Natural sounding
– Rapid, automatic customizability

Examples
– Synthetic voice w/ transplanted speech
contours
25
Artificial singing

AT&T Voder, 1962, by Homer Dudley
– Daisy (Inspiration for HAL’s voice in 2001)

Microsoft Research Whistler, 1997
– Scarborough Fair
26
Analyzing language


Language recognition shipped in Word 97
General purpose text-critiquing,
summarization, Japanese word-breaking
27
Inside The Office
Grammar Checker
28
Understanding language:
MindNet





A huge language
knowledge base
Automatically created
from dictionaries
Words (nodes) linked
by relationships
Millions of links
Recently added
(Encarta)
encyclopedia
knowledge
29
MindNet -- “Going to the birds”
chicken
Is_a
clean
Is_a
smooth
Is_a
poultry
Means
chatter
hen
Is_a
gaggle
Is_a
Is_a
beak
strike
Is_a
bill
egg
bird
sound
Is_a
goose
Part_of
face
Locn_of
feather
wing
Is_a
claw
hawk
Is_a
Is_a
Is_a
Typ_obj
leg
turtle
mouth
limb
Is_a
Part
Typ_subj_of
Is_a
creature
Is_a
Part
fly
Typ_subj
plant
Not_is_a
Is_a
Is_a
Typ_subj_of
Means
meat
Is_a
Is_a
Typ_obj
keep
Typ_0bj_of
Purpose
animal
Typ_subj
make
peck
duck
Cause
quack
Is_a
Typ_obj
Quesp
Typ_obj
Typ_subj
preen
supply
Purpose
catch
Is_a
opening
arm
Changing balance between
user & software systems

Yesterday:
– Applications were single programs running in
isolation
– Users used to (more or less) understand systems
that they used

Today:
– Componentized applications operate in concert
– Sophisticated users understand only small
percentage of systems they use
31
Tomorrow’s Systems and
Applications

Users will not be able to predict
– where computations will be performed,
– when they will be performed or
– by what software components

Gap between system capabilities and
user understanding will grow to the
point that the only way user will be able
to use system is through assisting
agents
32
Examples of user agents &
implicit actions

Lumiere (Office 97)
– Monitoring user and program events to
provide user help and assistance

Implicit queries
– Inferring information needs from browsing

Lookout/SpamKiller
– Monitoring mail activity to auto-categorize it
33
User Modeling

Models of a user’s informational goals
– User’s query (when available…)
– User’s background
– Acute and long-term search activity
– Acute actions with objects and documents
– Program data structures

Explicit and implicit information access
and display
Outline (ambitious!)





Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence
What if you could record everything you see &
hear?
The architecture revolution:
processing moves to transducers
35
Some Tera-Byte Databases







The Web: 1 TB of HTML
TerraServer 1 TB of images
Several other 1 TB (file) servers
Hotmail: 7 TB of email
Sloan Digital Sky Survey:
40 TB raw, 2 TB cooked
EOS/DIS (picture of planet each week)
Mega
Federal Clearing house: images of checks
Exa
– 15 PB by 2007
– 15 PB by 2006 (7 year history)

Kilo
Nuclear Stockpile Stewardship Program
– 10 Exabytes (???!!)
Giga
Tera
Peta
Zetta
Yotta
36
Info Capture



You can record
everything you see
or hear or read.
What would you do
with it?
How would you
organize & analyze
it?
A letter
A novel
Kilo
Mega
A
Movie
Giga
Library of
Congress (text)
Tera
LoC (image)
Peta
All Disks
Exa
Video
8 PB per lifetime (10GBph)
All Tapes
Audio
30 TB (10KBps)
Read or write: 8 GB (words)
Zetta
See: http://www.lesk.com/mlesk/ksg97/ksg.html
Yotta
37
A novel
Kilo
A letter
Mega
Library of
Congress (text)
LoC
(sound +
cinima)
All Disks
All Tapes
Giga
Tera
Peta
Exa
A
Movie
LoC
(image)
All
Photos
Zetta
All Information!
Yotta
38
Michael Lesk’s Points
www.lesk.com/mlesk/ksg97/ksg.html

Soon everything can be recorded and kept

Most data will never be seen by humans

Precious Resource: Human attention
Auto-Summarization
Auto-Search
will be a key enabling technology.
39
Outline (ambitious!)





Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence
What if you could record everything you see &
hear?
The architecture revolution:
processing moves to transducers
40
Put Everything
in Future (Disk) Controllers
(it’s not “if”, it’s “when?”)
Acknowledgements:
Dave Patterson explained this to me a year ago
Kim Keeton
Erik Riedel
Helped me sharpen
these arguments
Catharine Van Ingen
41
Remember Your Roots
42
Kilo
Mega
Giga
Tera
Peta
Exa
Technology Drivers: Disks




Disks on track
100x in 10 years
2 TB 3.5” drive
Shrink to 1” is 200GB
Disk replaces tape?
Zetta
Yotta

Disk is super
computer!
43
Data Gravity
Processing Moves to Transducers



Move Processing to data sources
Move to where the power (and sheet metal) is
Processor in
– Modem
– Display
– Microphones (speech recognition)
& cameras (vision)
– Storage: Data storage and analysis
44
It’s Already True of Printers
Peripheral = CyberBrick


You buy a printer
You get a
– several network interfaces
– A Postscript engine




cpu,
memory,
software,
a spooler (soon)
– and… a print engine.
45
All Device Controllers will be Cray 1’s

TODAY
– Disk controller is 10 mips risc engine
with 2MB DRAM
– NIC is similar power

SOON
– Will become 100 mips systems
with 100 MB DRAM.

Central
Processor &
Memory
They are nodes in a federation
(can run Oracle on NT in disk
controller).

Advantages
–
–
–
–
–
Uniform programming model
Great tools
Security
economics (CyberBricks)
Move computation to data (minimize
traffic)
Tera Byte
Backplane
46
Basic Argument for x-Disks

Future disk controller is a super-computer.
– 1 bips processor
– 128 MB dram
– 100 GB disk plus one arm

Connects to SAN via high-level protocols
– RPC, HTTP, DCOM, Kerberos, Directory Services,….
– Commands are RPCs
– Management, security,….
– Services file/web/db/… requests
– Managed by general-purpose OS with good dev
environment

Apps in disk saves data movement
– need programming environment in controller
47
The Slippery Slope
Nothing =
Sector Server
If you add function to server
 Then you

add more function to server

Function gravitates to
data.
Everything =
App Server
48
Why Not a Sector Server?
(let’s get physical!)


Good idea, that’s what we have today.
But
– cache added for performance
– Sector remap added for fault tolerance
– error reporting and diagnostics added
– SCSI commends (reserve,.. are growing)
– Sharing problematic (space mgmt, security,…)

Slipping down the slope to a 2-D block server
49
Why Not a 1-D Block Server?
Put A LITTLE on the Disk Server

Tried and true design
– HSC - VAX cluster
– EMC
– IBM Sysplex (3980?)

But look inside
– Has a cache
– Has space management
– Has error reporting & management
– Has RAID 0, 1, 2, 3, 4, 5, 10, 50,…
– Has locking
– Has remote replication
– Has an OS
– Security is problematic
– Low-level interface moves too many bytes
50
Why Not a 2-D Block Server?
Put A LITTLE on the Disk Server

Tried and true design
– Cedar -> NFS
– file server, cache, space,..
– Open file is many fewer msgs

Grows to have
– Directories + Naming
– Authentication + access control
– RAID 0, 1, 2, 3, 4, 5, 10, 50,…
– Locking
– Backup/restore/admin
– Cooperative caching with client

File Servers are a BIG hit: NetWare™
– SNAP! is my favorite today
51
Why Not a File Server?
Put a Little on the Disk Server

Tried and true design
– Auspex, NetApp, ...
– Netware

Yes, but look at NetWare
– File interface gives you app invocation interface
– Became an app server

Mail, DB, Web,….
– Netware had a primitive OS

Hard to program, so optimized wrong thing
52
Why Not Everything?
Allow Everything on Disk Server
(thin client’s)

Tried and true design
– Mainframes, Minis, ...
– Web servers,…
– Encapsulates data
– Minimizes data moves
– Scaleable


It is where everyone ends up.
All the arguments against are short-term.
53
The Slippery Slope
Nothing =
Sector Server
If you add function to server
 Then you

add more function to server

Function gravitates to
data.
Everything =
App Server
54
Disk = Node




has magnetic storage (100 GB?)
has processor & DRAM
has SAN attachment
has execution
Applications
environment
Services
DBMS
RPC, ...
File System
SAN driver
Disk driver
OS Kernel
55
Technology Drivers:
System on a Chip

Integrate Processing with memory on one chip
– chip is 75% memory now
– 1MB cache >> 1960 supercomputers
– 256 Mb memory chip is 32 MB!
– IRAM, CRAM, PIM,… projects abound

Integrate Networking with processing on one
chip
– system bus is a kind of network
– ATM, FiberChannel, Ethernet,.. Logic on chip.
– Direct IO (no intermediate bus)

Functionally specialized cards shrink to a chip.
56
Technology Drivers: What if
Networking Was as Cheap As Disk IO?
 Disk
 TCP/IP
– Unix/NT
100% cpu @ 40MBps
– Unix/NT
8% cpu @ 40MBps
Why the Difference?
Host does
TCP/IP packetizing,
checksum,…
flow control
small buffers
Host Bus Adapter does
SCSI packetizing,
checksum,…
flow control
57
DMA
Technology Drivers:
The Promise of SAN/VIA:10x in 2 years
http://www.ViArch.org/

Today:
– wires are 10 MBps (100 Mbps Ethernet)
– ~20 MBps tcp/ip saturates 2 cpus
– round-trip latency is ~300 us

In the lab
– Wires are 10x faster Myrinet, Gbps Ethernet,
ServerNet,…
– Fast user-level communication


tcp/ip ~ 100 MBps 10% of each processor
round-trip latency is 15 us
58
SAN:
Standard Interconnect
Gbps Ethernet: 110 MBps

PCI: 70 MBps


UW Scsi: 40 MBps

LAN faster than
memory bus?
1 GBps links in
lab.
100$ port cost
soon
Port is computer
FW scsi: 20 MBps
scsi: 5 MBps
59
Technology Drivers:
100 GBps Ethernet replaces SCSI

Why I love SCSI
– Its fast (40MBps)
– The protocol uses little processor power

Why I hate SCSI
– Wires must be short
– Cables are pricey
– pins bend
60
Functionally Specialized Cards

Storage
P mips processor
ASIC
Today:
P=50 mips

Network
M MB DRAM
M= 2 MB
In a few years
ASIC
P= 200 mips
M= 64 MB

Display
ASIC
61
Technology Drivers
Plug & Play Software

RPC is standardizing: (DCOM, IIOP, HTTP)
– Gives huge TOOL LEVERAGE
– Solves the hard problems for you:
 naming,
 security,
 directory service,
 operations,...

Commoditized programming environments
–
–
–
–


FreeBSD, Linix, Solaris,…+ tools
NetWare + tools
WinCE, WinNT,…+ tools
JavaOS + tools
Apps gravitate to data.
General purpose OS on controller runs apps.
62
Basic Argument for x-Disks

Future disk controller is a super-computer.
– 1 bips processor
– 128 MB dram
– 100 GB disk plus one arm

Connects to SAN via high-level protocols
– RPC, HTTP, DCOM, Kerberos, Directory Services,….
– Commands are RPCs
– management, security,….
– Services file/web/db/… requests
– Managed by general-purpose OS with good dev
environment

Move apps to disk to save data movement
– need programming environment in controller
63
Summary





Microsoft Research (census)
Tele-Presentations (Gordon Bell, Jim Gemmell)
Microsoft Research initiative on Telepresence
What if you could record everything you see &
hear?
The architecture revolution:
processing moves to transducers
64