Systems to Capture Everything:
Beyond cameras and desktops
www.MyLifeBits.com
Gordon Bell, Jim Gemmell, Roger Lueder
Outline: past & futre

MyLifeBits aka Memex
How has the project evolved?
 How do we use MyLifeBits?
 How is it built?
 Shape of the database?


CARPE- Continuous archiving and
recording of personal experiences
What is the vision?
 Relevance for devices and software?


A Personal Health Record…
The Rs: Recording and remember and much more:
Recollecting, reminiscing, retrieving, …reflecting












Recording and remembering….. (make a permanent record of)
Registering as in registrar = Filing, classifying, organizing
Relegating (assign to a class), classify
Retaining as in record retention
Retrieving - accessing memory info, regaining something lost..
Recalling, recollecting (remembering, retrieving, calling back,
calling up)….
Relationship…relating. A state of connectedness.
Recreating, restoring, reliving, receipt, resonating, repurposing,
replaying,
Regurgitating, reproducing (repeat after memorization)
Reminding! Put in the mind of someone, prompting, cuing .
Reflecting
Realizing or rematerialize as in digital immortality;
understanding, recognizing
I am data
History: Telepresence c1995
Tele-presentations
Tele-meetings
Ambience and Presence:
Being there while being here
Dining at home on the “Orient Express”
History: The remote worker rediscovers the PERSONAL
computer
Oct 1998
Can we
scan your
books and
put them
online?
Raj Reddy
Sure! Don’t worry
about copyright
stuff. Microsoft
has lots of
lawyers
1999 – Scanning starts in earnest
“we” start to scan,
put content into
folders & files
My docs and archive
Library/file cab
X- Employer
Active Employer
Library/file cab
Employer
Self
..
..
Biographical
Project
Employer
Project
Project
Employer
Business
Invests,
family $s,
& Legal
Library/file cab
Library/file cab
Library/file cab
Library/file cab
X-Employer
Library/file cab
Library/file cab
Library/file cab
Library/file cab
Library/file cab
<1980s Library/file cab
Project
Project
Personal,
including
Medical
Now that it’s in Cyberspace
How do you remember the 20,000+ file names?
Or in which of 1500 folders they live?
What’s about a tool for finding stuff?
2001 Capture goes beyond paper
Gordon, You
should be using
a database.
Jim, I don’t
need no stinkin’
database!
Re-discovery of Memex
As We May Think, Vannevar Bush, 1945
“A memex is a device in which an individual stores all
his books, records, and communications, and which
is mechanized so that it may be consulted with
exceeding speed and flexibility”
 Full-text search, text & audio annotations, and
hyperlinks
Even more capture

Telephone calls, more video, all web pages
visited, keyboard and mouse usage logging,
radio, TV…
Steve Mann timeline
“I sensed”
Clarkson MIT c2001
Visually impaired
UW 2004
2003 - SenseCam
Feb 2005
Epiphany!
Memex is a database
&
personal TP system
Demo Clips & Screens
747 Screen…
Timeline
Pivoting:
contact>
call>
t>
web page
GPS Photo location
Reports
The Stew family tree
Copyright Mark Stewart, 2004
Vibe report
Quindi Meeting Capture
SenseCam
SenseCam around Cambridge
Gordon with
SenseCam
& Voice
Recorder
MyLifeBits Software
Everything goes in a database

MyLIfeBits need all the features of a database
(Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup,
Replication)



If we didn’t use one, we’ll eventually create one!
Files as blobs; sync with file system for legacy apps
We are part of Jim Gray’s Bay Area Research Lab
SQL
MyLifeBits Software
Room
Capture
GPS import &
Map display
SenseCam
Import files
VIBE
logging
MyLifeBits
Shell
Text
annotation
tool
Voice
annotation
tool
Screen saver
MyLifeBits
store
Radio
capture
& EPG
Internet
Browser
tool
Legacy
applications
database
IM capture
MAPI
interface
files
PocketPC
transfer
tool
Outlook
interface
TV capture
tool
PocketRadio
player
Telephone
capture tool
TV EPG
download tool
Legacy
email client
The Shape & Size of
Gordon’s LifeBits
Money, 35
Vicki Video,
200
.xls, 1470
NULL&blank,
985
.ZIP, 80
MyLifeBits
Sept 2006
SQL, 1
Pictures, 66138
eMail, 101541
302,700
Audio, 6411
Video, 1842 items
167.4 GB
Doc&Rtf, 15375
by number
of Items.
Web pages,
Tiff, 3084
98813
PPT, 2133
PDF&PS, 4682
SQL, 25400
Pictures, 18462
Audio, 17455
MyLifeBits
Sept 2006
Video, 10241
Size (MB) by Type
Web pages, 8803
Vicki Video,
65000
167.4 GB
302,700 items
PDF&PS, 8250
PPT, 5758
By Size (GB)
Tiff, 5712
Doc&Rtf, 1383
eMail, 387
Money, 210
NULL&blank, 136
.ZIP, 89 .xls, 120
Bell Growth:
1GB/month
1.1 TB/lifetime
Monthly & Lifetime Storage Use
Item
1 MB Books|reports
5KB Emails
0.1
100
Total*
MB|GB
Month|Life
3
13
100 KB Image scans
0.4 MB Photos
75 KB Web pages|docs
100 MB Music
5
10
100
0.1
13
100
188
250
40,000
1,000
1,000
4
1,250
200,000
1 KB/s Listened audio, speech
50 KB Daily photos
2 GB/hr TV
Daily
number
Observations about use(rs)
1.
2.
3.
4.
5.
6.
7.
Cell phone sized device (CPSD) will be the platform!
On Applications… think about CPSD as the platform and context

Search is the “killer app” pretty much as Bush described.

Screen savers “memory refreshers” also provide ambience

Where did my day to?
Users are unwilling to spend time managing their computers or data.

Meta-data, classification, etc. must be automatic

User-input meta-data e.g. Dublin Core – naïve’ Librarian’s dream.

We have nice scheme for classification using facets. It requires work.
Time is the most important meta-data. Photos: place (GPS), subject.
Folders are a good and bad idea.

Most users don’t know what they are or how they work

If used, over time, they become useless: too many, miss-file, etc.
User should put “every” information fragment into the system. e.g., to dos,
call backs, business cards numbers, attention events. It pays.
Same information in multiple places always becomes obsolete.
Capturing Everything:
Phone calls in context of cell phone as a
platform for communication and capture
 Formal Meetings
 Rooms
 Everything in daily life
 Personal health and medical monitoring
 Memex for scientists and engineers

The vision…
SkyDrive UX
Hotmail
Office Live
Spaces
Abby’s
Camera
Memex
Store
Microsoft Storage
Toby’s
Pocket
PC
Toby’s PC
Abby’s
Home PC
Abby’s Laptop
Phone
The killer app…
Health?
Capturing every heartbeat
72.6 beats/min; 38.16 Million beats/year
 3.13 billion beats year
 The important number is 4-4.5 years, or ETS
 Battery life: the expected trip to surgery!

BodyMedia Output
PHR Schema
MyHealthBits
Courtesy of B. Parmanto, U. of Pittsburgh
B Parmatto,
U. Pittsburgh
Record View
Courtesy of B. Parmanto, U. of Pittsburgh
Defining Timeframe
Courtesy of B. Parmanto, U. of Pittsburgh
Preview
Courtesy of B. Parmanto, U. of Pittsburgh
Preview: web page
Courtesy of B. Parmanto, U. of Pittsburgh
SenseCam University Grant Program
MSFT supplies money, software, SenseCams





Memex vision: Notebook for engineers & scientists
Medical & health: observations & memory recall,
including diet and exercise
Education: How do people learn?
Help me learn/remember!
Tourist e.g. museum experience
Plumbing


Security
Filtering many images, voice & location annotation
The Agenda for the Tbyte(s), Lifetime, PC:
The killer app after office and mail.


Guarantee that data will live forever! “dear appy”
problem
Cheap, easy, and data-rich (e.g. time, place)
capture:







GPS and time everywhere
Paper capture has to be as easy as discarding
(scanner/shredder)
Personal meeting capture...
E-book…e-magazines & journals need to have critical
mass!
Telephony and audio capture with indexing
Media Center compatible for entertainment (photos,
video, TV, radio)
Content analysis (critical for photo & video!)
more






One dbase for everything (articles, books,
conversations, ... financial transactions) …vs. longterm use of hierarchical files. Is dbase intuitive?
Information control: privacy, security, expungeability
deniability,…
Having to be schizophrenic or have a lobotomy when
leaving a “life” i.e. a company
Annotations/meta-data add every-increasing value
Easy annotation aids search and becomes the content
The “killer apps”: Alzheimer, immortality, surrogate
memory?
GUI’s to improve use e.g. time to learn, use, retention
www.MyLifeBits.com