Some Reflections on
Augmented Cognition
Eric Horvitz
ISAT & Microsoft Research
November 2000
Toward Augmented Cognition
Challenge: Volume of information, overall complexity of
defense tasks continue to grow at a rapid pace, in stark
contrast to human cognitive abilities, which remain static:
Attention
 Memory
 Learning
 Comprehension
 Sensory bandwidth
 Visualization abilities
 Judgment and decision making

In contrast, computational capabilities have continued
to grow rapidly
Apply computation to support / augment human cognition
Augmented Cognition
Developing new powers of understanding,
remembering, and decision making via new
technologies custom-tailored for human—
computer collaboration, symbiosis
Methods, designs that harness computation and
explicit knowledge about human limitations to open
bottlenecks, address biases, deficits in cognition.

Techniques and interfaces that allow for fluid mixedinitiative interaction to solve problems.

Continual background sensing, learning, inference to
understand trends, patterns, situations relevant to a
user’s context and goals.

Modern Psychology:
Cognition as information processing



Human cognition: Vast abilities coupled with
highly characterized / characterizable
limitations & bottlenecks
Mind as an architecture composed of
different subsystems that form, remember,
transmit representations of the world.
Work on distinct dimensions of cognition:
Attention, memory, learning, concept attainment,
visualization, judgment
Timing: Is Now the Time to
Undertake Such an Effort?
Understanding of human as resource-limited reasoner,
well-characterized bottlenecks, capabilities, biases
•
Limitations in serial processing
•
Attention
•
Concept attainment
•
Learning and memory
•
Visualization
•
Judgment and decisions
•
Collaboration, group decisions
•
Psychophysics
•
HCI expertise, results
Results in Cognitive Psychology
Timing: Is Now the Time to
Undertake Such an Effort?
Leaps in computation, memory, algorithmic prowess
•
Raw computational resources (CPU, memory)
•
Connectivity
•
Sensing
•
Sensor fusion
•
Inference and classification
•
Learning
•
User modeling
•
Speech recognition
•
Vision
•
Interaction, input
•
UI, display design
Computational power
Augmented Cognition Efforts




Characterizing inefficiencies, problems, opportunities re:
performing critical tasks based on fundamental cognitive
limitations
Identify, apply results from cognitive psychology to build
computation solutions
Identify new cognitive psychology research
Identify new joint CS—cognitive psychology efforts
Cognitive Tasks
Existing
Psychological
Results on
Cognitive Limitations
Memory
Concept
Computation
Divided
Attainment
Attention
Abilities & efficiencies
Target
Augmented Cognition Efforts




Characterizing inefficiencies, problems, opportunities re:
performing critical tasks based on fundamental cognitive
limitations
Identify, apply results from cognitive psychology to build
computation solutions
Identify new cognitive psychology research
Identify new joint CS—cognitive psychology efforts
Cognitive Tasks
* NewExisting
Cog. Psych.
Research
Psychological
Results on
* HCI, Aug. Cognition
Cognitive
Limitations
Research
?
Memory
Concept
Divided
Attainment
Attention
Abilities & efficiencies
Target
Augmented Cognition: Opportunities

Enhance learning and memory via reminder
systems & methods

Automate specific aspects of problem solving &
filtering

Modulate / triage communications

Develop new visualizations, other info. rendering
to increase rate of “concept attainment”---raising
effective human-computer bandwidth
Augmented Cognition: Opportunities

Support sensory fusion, judgment, action
under uncertainty

Provide vigilant monitoring for situations
requiring attention

Monitoring, knowledge about human and
machine errors

Extend abilities to coordinate, collaborate
with other people and systems

Design multimodal, context-aware systems
for more efficient, natural human—computer
collaboration
Low-Hanging(?) Fruit









Information filtering and triage
Mixed-initiative interaction
Context-sensitive UI / computing
Intelligent reminding
Managing attention and disruption
Visualization & comprehension
Human-error--aware systems
Automated sensor fusion
Judgment de-biasing systems
Risks in Building a Strong,
Valuable Program

Timing: Too early…?



Strength of research



Need more complete understanding of human
bottlenecks in real world?
Need more sophisticated computational
procedures?
Rigorous, foundational work may be needed
Poor initial stabs may lead to premature
discontinuation, disappointment
Focus


Choosing some foci for early traction
Avoiding replication of work in commercial sector
Several Sample Projects

Vista: Models for limiting, controlling
information based on coarse value of revealed
information.

Handsfree decision support: Fluid
interaction for providing recommendation,
updates; context-sensitive speech.

Lumiere: Context- and competence-based
assistance

Lookout: Mixed-initiative interaction.

Priorities: Automated assignment of urgency,
alerting, communication, caching by value.
Several Sample Projects

Notification Platform: Attention in HCI,
notification architectures, UI, psych studies of
disruption, reminders.

Continual computation: Use background
computation to forecast goals, attention; guide
precomputing, caching. Background query,
document tracking.

Learning information goals: Learning about a
user’s goals from behavior, log files.

Qualia: Models of attention, perception in
multimedia rendering and communication

Context-aware devices: Enhance UI with
implicit gestures, context-sensing.
Project Drill Down & Demos