Digital Memories (Memex)
Jim Gemmell and Ken Wood
MSR San Francisco and Cambridge
Digital Memories (Memex) RFP
Digital Memories Memex RFP
Solicits research proposals that build
on the “memex” vision for personal
lifetime storage
Offers funding, software, and hardware
Memex
As We May Think, Vannevar Bush, 1945
“A memex is a device in which an individual stores
all his books, records, and communications,
and which is mechanized so that it may be
consulted with exceeding speed and flexibility”
Text and audio annotations, trails (hyperlinks),
speech recognition, tablet-style input,
wearable camera
Storage
“Yet if the user inserted
5000 pages of material a
day it would take him
hundreds of years to fill
the repository, so that he
can be profligate and
enter material freely”
-Vannevar Bush, 1945
Doc&Rtf 1.0
.PPT 4.0
Other 1.0
Web pages 4.1
.PDF 4.4
Size (MB) by Type
Pictures 7.1
101 GB
206 K items
By Size (GB)
Tiff 7.5
Video 58.9
Audio 10.9
MyLifeBits
3/26/05
Bell Growth:
1GB/month
=1 TB/lifetime
“A record if it is to be useful … must be
continuously extended, it must be stored,
and above all it must be consulted”
“The difficulty seems to be, not so much that
we publish unduly … but rather that
publication has been extended far
beyond our present ability to make real
use of the record”
- Vannevar Bush
Buzz
Press Coverage
Keynotes
Workshops
..Workshops
CARPE ’04
The First ACM Workshop on
Continuous Archival & Retrieval of Personal Experiences
October 15th 2004
Columbia University, New York, NY, USA
Exciting Research
Request For Proposals (RFP)
Solicited Research
Broad themes
Capture
Annotation
Links between items
Metadata
Proposals should be in the context of
personal lifetime storage
Other cases considered where cross-over
potential high
Solicited Research
Suggested areas
Capture/sensors
E.g., scanning, wearable, embedded, different kinds of sensors,
robotic assistance
Data
Storage, management, organization, and retrieval
Insight
Content analysis and data mining
User interface issues
Visualization, authoring, story-telling, annotation, user-studies
Applications
E.g., personal museum, health-support, childcare, research
tools, meeting capture
Security, privacy, and legal issues
Successful Applicants Get
Funding
Hardware (SenseCams)
Software
Hardware
SenseCam
Designed by Lyndsay Williams,
MSR Cambridge lab
Sensecam
Essentially a “Black Box” data and image recorder for the
human body
First wearable prototype can be worn for a day and
captures up to 2000 images in 12 hours
Image capture is triggered by sensors, e.g. motion, light,
temperature, people in field of view, ...
Sensor data is also recorded for later presentation,
analysis, and correlation
Taking A Trip With Sensecam
How does SenseCam work?
Passive Capture
Accelerometers for motion detection
Passive Infrared (PIR) sensors triggered by people or other living
beings in front of camera
Digital sensors for light level and temperature
Transaction detection, e.g. +100% or -50% light change –.door transition
Images captured automatically on sensor events caused by, e.g.
Turning around, sitting down, standing up, running, falling over, ...
Moving between locations (rooms, car, bus, indoor/outdoor, etc.)
Meeting a friend in the street
Simple hand gesture allows intentional image capture
Accelerometers also allow image stabilisation
Image Stabilisation
Accelerometer measures motion of the device
When any sensor indicates image capture, we
wait a few tens of milliseconds to capture image
when movement is less than 10 degrees/second
if possible
Greatly reduces the number of blurred images
Before
After
SenseCam Ultra Wide-Angle Lens
(132 degrees)
Captures everything regardless of exactcamera orientation
Sensecam Scenarios
Family
SenseCam could:
• Help grandad find his
glasses
• Let mum see what the kids
did today
• ...
Tourism
SenseCam could:
• Help friends recount visits
to festivals, museums, etc
• Produce annotated photo
blogs automatically
• ...
SenseCam
• Human-wearable “black box”
• 2000 images per day,
captured automatically
• Sensors trigger image
capture
• Sensor data is recorded too
Medical
SenseCam could:
• Help consolidate
memories for the memoryimpaired*
• Provide vital information
for paramedics at an
accident
• Help evaluate compliance
• ...
*Clinical trials underway
HCI research tool
SenseCam could:
• Augment reflective practice
• Supplement and/or validate
diary studies as an
alternative to full
ethnography
• ...
SenseCam
Family scenarios
SenseCam could be used in daily family life, both for practical and
social activities
Sample Scenarios
The kids drop their SenseCams into the docking station when they come
home from a trip to the zoo with Dad
Mum switches to “Channel SenseCam” on the TV and quickly flips through
images of the trip. The kids come in and tell Mum more about the giraffes
showing on the screen
Grandad docks his SenseCam and flips back through the last half hour,
noticing that he put his glasses down on the desk in the study
He goes and retrieves them
Mum goes to the study, navigates to last Saturday night’s SenseCam log
and flips through the images looking for a wine bottle
She finds it, sees that it was Chateau Coutet that she liked so much, and
makes a note to buy some
FridgeCam
SenseCam
Tourism scenarios
SenseCam could enhance tourist visits and help generate mobile
photo blogs automatically
Sample Scenario
Friends visit the edinburgh festival together for a few days
Each wears a sensecam as they attend various events
In the late evenings or over breakfast, they get together and replay their
sensecam logs to each other on their laptops or pdas, prompting
discussion on what they saw
Selected images are uploaded to a weblog, annotated with sensor
data and audio comments (captured with sensecam or added during
viewing sessions)
Family and friends elsewhere can experience the visit through the blog
Two years later they could find particular images by recalling that they
occurred on a cold but sunny day in edinburgh
With integrated wireless connectivity, the friends could also create their
blogs in real time (and with no effort) as they attend the events
A Walk Around Cambridge
SenseCam
Medical scenarios
Currently targetting non-progressive memory-loss
patients, e.G. Stroke and accident victims
Working with cambridge doctors on clinical trials now
DG, 32-year old stroke victim
F, 19-year old with frontal-lobe impairment
S, 62-year old limbic encephalitis sufferer
Ethics approval for 20 patients over 3 years
Initial results show that sensecam can help with
consolidation and even improve general memory
function in formal tests (!)
Doctors also evaluating potential use with
other conditions
E.G. Alzheimer’s, pediatrics, …
SenseCam V2
V2 now being designed
First boards back, debugging and software
development underway
Many general usability improvements
battery, data sync, programmability, form-factor,
low-light image quality, ...
Additions include
GPS, continuous audio recording, IRDA for id
beacon, buttons, and indicators, …
Software
Evolution
From Silo Applications on DB islands to the
universal store
Items And Links
Photo of Event
Caller in Phone Call
Annotates
Software
SenseCam
GUI
logging
Extensible
Shell
Text
annotation
tool
GPS import &
Map display
Screen saver
database
Browser
tool
Media player
logging
NTFS file
sync
Outlook
interface
Voice
annotation
tool
SenseCam Software
Item Types
Refining An E-mail Search
Web Capture
Archived Web Page Search
Map UI
Event Log Of A Photo
GUI Logging
…GUI Logging
Extending The Shell
_LinkTypes2
_EventTypes2
PK
event_type_id
link_type_id
name
description
ntfs_files2
PK,FK1,I2,I3
PK
_ItemSubTypes2
item_id
_Item-Events2
I1
ntfs_root_id
volume_serial_number
frn
parent_frn
Extension
Length
Created
Modified
Accessed
Attributes
IndexedContent
I3
I2
PK,I1
PK
event_id
item_id
event_type_id
occurred
PK,FK3,I1,I2
PK,FK1,I1,I2
PK,FK2,I1,I2
PK,FK1
PK
PK
name
openwith
small_icon_id
large_icon_id
delete_stored_procedure
_Links2
FK2,I1
FK1,I1
I1
_FullTextColumns2
subtype_id
link_type_id
source_item_id
target_item_id
Outlook_Tasks2
_Item-ItemSubTypes2
PK,FK1
ntfs_dirs2
TAPI_PhoneCalls2
PK,FK1,I1
Phone
Call Type
CID
CID Name
CID #
Begin
End
Seconds
Connected
Ended
Roaming
Trimmed
Recorded
Transcript
I1
item_id
I1
ntfs_root_id
volume_serial_number
frn
parent_frn
Created
Modified
Accessed
Attributes
IsRoot
SyncOnStartup
item_id
I2
PK,FK1
I1
_Items2
PK,I1,I2,I3,I4,I5
item_id
I5
I1
I4
I2
name
creation_date
last_modified_date
interval_begin
interval_end
is_collection
is_annotation
is_hidden
thumbnail
thumbnail_width
thumbnail_height
collection_cnt
annotation_cnt
descendants_cnt
annotates_cnt
child_cnt
large_icon_id
small_icon_id
rank
Radio_Recordings2
item_id
Title
Date Recorded
Duration
Station
unc_path
cached_local
cached_path
item_id
Title
Genre
Year
Rating
Description
Date Recorded
Duration
Channel
Channel Description
Bytes
unc_path
cached_local
cached_path
Title
Author
Album
Genre
Duration
Bitrate
I3
PK,FK1
item_id
subtype_id
Outlook_CalendarItems2
PK,FK1
item_id
Subject
Start
End
Description
Location
Creation Time
Modified
PK,FK1,I1,I2
MSMSGS_logs2
location_data2
I1
I2
I3
Width
Height
Date Taken
Camera Make
Camera Model
Latitude
Longitude
Elevation
PK,FK1,I1,I2
U1,I1,I2
I1
I2
U1
PK,FK1
item_id
Time
Latitude
Longitude
Elevation
Source
BLOB_Data2
PK
item_id
blob
blob_file_extension
blob_size
PK,FK1
item_id
Body
Creation Time
Modified
PK,FK1
item_id
Domain
Page Url
Title
Visited
Top Level
Secure
IndexedContent
big_thumbnail
IMG_Images2
item_id
Outlook_StickyNotes2
Outlook_MailItems2
IE_Web_Pages2
I1,I2
I1,I2
I2
PK,FK1,I1,I2,I3
item_id
Subject
Start Date
Due Date
Percent Complete
Status
Priority
Notes
Creation Time
Modified
item_id
I3
TV_Recordings2
PK,FK1,I1
PK,FK1,I1
PK,FK2,I1
Music_Songs2
PK,FK1,I2
subtype_id
table_name
column_name
item_id
Handle
Logon Name
Buddy Handle
Buddy Logon
Begin Time
End Time
I1
item_id
From Name
Subject
Received
Sent
Body
From Address
From Type
To
Cc
Bcc
Attachments
Read
Modified
Outlook_Contacts2
PK,FK1
item_id
Title
Display Name
First Name
Middle Name
Last Name
Suffix
Email
Email2
Email3
IM Address
Company
Department
Job Title
Business Street
Business City
Business State
Business Postal Code
Business Country
Home Street
Home City
Home State
Home Postal Code
Home Country
Other Street
Other City
Other State
Other Postal Code
Other Country
Assistant's Phone
Business Fax
Business Phone
Business Phone 2
Callback
Car Phone
Company Main Phone
Home Fax
Home Phone
Home Phone 2
ISDN
Mobile Phone
Other Fax
Other Phone
Pager
Primary PHone
Radio Phone
TTY/TDD Phone
Telex
Account
Anniversary
Assistant's Name
Birthday
Location
Manager's Name
Notes
Office Location
Profession
Sensitivity
Spouse
Web Page
Creation Time
Modified
Extending…
Adding new item types
Just add your table + an entry to the
_ItemSubTypes table
Adding new link types
Just add an entry in the _LinkTypes table
Add your own side table if you need one…
docsim_doclen1
PK
item_id
ntfs_sync1
SenseCam_Raw1
I2,I3
I3
ntfs_sync_id
item_id
sample_time
_Agent_Work_Items1
doclen
sample_no
hours
mins
tilt_x
tilt_y
light
temp
trigger
image_no
red
green
blue
NTFS_ChangeLog1
PK
entry_id
I1
I1
ntfs_root_id
sequence_no
priority
URI
Type
Submitted
Updated
Status
Comments
Arg1
ErrArg1
I2
classif_topics1
PK
topic_id
name
description
facet_id
parent_topic_id
I2
I2,I3
query_result_id
table_or_view
I2
ntfs_root_id
file_id
volume_serial_number
frn
parent_frn
name
extension
length
attributes
created
modified
accessed
is_dir
is_new
is_modified
VIBE_Activity1
Time
Keyboard
Mouse
BLOB_Data1
PK
Time
HWND
Title
Foreground
Document
Keyboard
Mouse
Application
item_id
PK,I2
item_id
I2,I1
Read Time
Booth Number
Demo Title
Full Name
Email
Title
Reports To
Function
Cost Center
PK
work_item_id
I1
I2
I3
sequence_no
item_id
work_item_class
work_item_type
submit_time
completion_time
priority
status
message
retry_step
retry_next
retry_counter
item_id
blob
blob_file_extension
blob_size
_QueryResults_Info1
PK
query_result_id
created
accessed
delete_req
sql
cardswipe_data1
_QueryResults_Schema1
PK
PK
VIBE_KMTimeline1
ntfs_roots1
PK
item_id
path
crawl_needed
number
PK
PK
PK
PK,I1
Extension
I2,I3 item_id
I3
sample_time
sample_no
hours
mins
tilt_x
tilt_y
light
I2
temp
I2,I3 trigger
image_no
red
green
blue
PK
PK
store_id
entry_id
crawl_date
is_root
should_crawl
last_full_crawl
last_quick_sync
last_slow_sync
quick_sync_n
quick_sync_units
slow_sync_n
slow_sync_units
is_syncing
docsim_item-terms_prox1
PK
PK
PK
PK
I1
I2
item_id
Media Start
Media Stop
Record Begin
Record End
video_item_id
_custom_queries1
PK
item_id
Description
create_results_table
query
query_where_clause
override_select_sql
default_order_by
PK
classif_aliases1
topic_id
alias
classif_facets1
PK
current_position
old_position
new_position
PK
timestamp
msg
source
PK
key
value
ntfs_volume_info1
extension
docsim_idf1
volume_serial_number
PK
PK
term_id
idf
drive_letter
usn_journal_id
next_usn
view_state_id
view_state
PK
site_id
I1
domain
rule
IE_Popup_Log1
FK1
site_id
date
url
allowed
_hashValues1
docsim_terms1
PK
icon_id
PK
term_id
U1
class
image
width
height
U1
term
occurrences
IE_Popup_Rules1
facet_id
name
xmlns
_msgs1
_config1
_Icons1
PK
PK
term
event_id
cliplets_raw1
PK,I1,I2
docsim_noise1
item_id
term_id
position
Shell_SavedViewState1
tf
SenseCam_Raw21
query_id
term
tf_idf
file_id
description
item_id
term_id
Assembly
Outlook_CrawlInfo1
PK
PK
_promotion_agent_handlers1
num
docsim_item-terms1
item_id
term_id
Wmp_EventArguments1
timestamp
nestlevel
sender
description
time
numrecords
query_result_id
item_id
instance
NTFS_Promotion_Handlers1
PK
PK,I1
docsim_query1
_log1
_QueryResults1
PK,FK1,I1
PK,I1
PK
docsim_tfidf1
item_id
hashvalue
_event_queries1
PK
item_id
Outlook_Ids1
I1,I2
I1,I2
I1
store_id
entry_id
delete_processor_is_valid
I1,I2
item_id
event_type_id
topn
column_name
sort_table_or_view
sort_column
sort_dir
description
Hitting The Database Directly
Facetmap
Time/location Greg Smith, MSR VIBE lab
FacetMap
Refining text search
Conclusion
Digital Memories is a hot research
area requiring contributions from
many disciplines
The Digital Memories (Memex) RFP offers
financial, hardware, and software support
Researchers can hit the ground running
with many kinds of capture, database
storage, extensible software, and logging
that supports user studies
© 2005 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
Taking A Drive With Sensecam