When-to-act: Evidence for Action Bias
in a Dynamic Belief-Updating Task
Michael Hildebrandt
Human-Computer Interaction Group
Department of Computer Science
University of York
Joachim Meyer
Department of Industrial
Engineering and Management
Ben Gurion University
DIRC annual workshop
Edinburgh, 15-17 March 2005
Time Design
Time as designable property of systems
When-to-act problem
Managing action/judgement [timeliness/accuracy] trade-offs
Modelling
Bayesian updating, SDT, normative / empirical
When-to-act experiment
Diagnostic task, belief updating [alarm], action bias
When-to-act: Evidence for action bias in a dynamic belief-updating task
Theme activities: Time Design workshops
CHI’04: Subjective (psychological, social time), representation
ECCE’04: Control of dynamic systems, social time
HFES’04: Temp. decision, interruption scheduling, case studies
Conclusion: From Newtonian view to functional view
[Slides, wiki, mailing list: www.timeDsn.net]
‘Newtonian’, descriptive view of time
’Absolute, true and mathematical time, of itself and from its own
nature, flows equably and without relation to anything external.’
Descriptive view of time. Problem: ubiquity. Events necessarily
happen in time, but this does not make time an interesting or
relevant property in design/operation
KSLM, Newell’s “bands”, task representation, time-and-motion
Time Design – When-to-act
Time Design:
problem
An interdisciplinary
– Model – When-to-act
view
experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Time Design: Functional view of time
Time as designable property of systems
Psychological time, social time
Designing the experience of time
Designing the display of temp. inf., supporting temp. awareness
Tempo: Increase speed / availability / flexibility;
results in fast / dynamic / complex systems;
creates need for machine support;
creates need to design effective human control of system
Designing / appreciating temp. patterns
Supporting synchronisation / coordination
Design space: Time as property of environment, task,
physical system / interface, user behaviour
Functional view: Exploring degrees of freedom, trade-offs
Time Design – When-to-act
Time Design:
problem
An interdisciplinary
– Model – When-to-act
view
experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Representation, analysis, design
From [focus on] temp. description of human behaviour (KSLM)
To Human reaction to / use of time
Time in HCI / Human Factors: Research domains
System Response Time
Temporal awareness, temporal reference systems
Temporal error; decision lockout
Control of dynamic systems; automation, scheduling, control
Trust in automation / dynamic systems; long-term automation
Time perception & stress; affordance model of dynamic system
TA Time Design tutorial
Time Design – When-to-act
Time Design:
problem
An interdisciplinary
– Model – When-to-act
view
experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Example: Dynamic Function Scheduling research
Misperception / misjudgement of when-to-act
Misperception of time available / time required
Duration neglect in assessing control strategies
Inappropriate / rigid Function Scheduling
Benefit of temporal awareness and anticipative control behaviour
Multiple and conflicting temporal reference systems
Scheduling and allocation based on dynamic value functions
Late and premature action; pacing and decision lock-out
Effect of time pressure on automation use and trust
Human scheduling performance; sched. as operation strategy
TA Dynamic Function Scheduling
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
When-to-act problems: Simple cases
Event-driven, self-paced, periodicity, predictability, triggers
Simple heuristics: The sooner, the better
When-to-act dilemmas
Motivation to act early [reduce uncertainty, increase prob. of execution]
Motivation to defer action [improve solution, gather additional inf.]
Risk: Action- or judgement-bias leads to premature / late decision
Contributing factors: Reliability, predictability, timing, age of info.
Temporal uncertainty [reliance on time perception vs. visualisation]
When-to-act problems in the real world
ATC, medical treatment [A&E, long-term], piloting [V2, pattern],
rear-end CW, supervisory control [manufacturing], emergency C&C
Time Design – When-to-act
Time Design:
problem
An interdisciplinary
– Model – When-to-act
view
experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
When-to-act experiment
System-1
information
Start
of trial
Alarm (Sys-2
Decision
information)
Decision
End
of trial
Investment,
rating
Trial duration:18s
100 trials, IVs: Timing and reliability of alarm
Alarm timing
early (5s) late (13s)
Alarm
high (.9)
reliability low (.7)
Time Design – When-to-act problem – Model – When-to-act experiment
Feedback
t
When-to-act: Evidence for action bias in a dynamic belief-updating task
Modeling and paradigm
Diagnostic DM, monitoring & control [Kerstholt]
Response to alarms [over-/under-use, compliance/reliance, reliability]
Signal detection theory
Bayesian belief updating
Heuristics & biases
Time perception
Human Factors approach
Normative / engineering model as standard
Compare with empirical data
Model empirical data (ideal: in same formalism)
Identify systematic deviations, biases, behavioural disposition
Express these using formal models or design heuristics
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Modeling temporal aspects of the system
p(overallSuccess) = p(decisionCorrect) * p(decisionInTime)
Alarm timing: Early (mean 5s), late (mean 13s); SD=1.5
Prob. action execution: f(x)=1.0875-x/18*0.675
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Modeling pre-trial information: SDT
Noise distribution (“no problem”): Mean=4, SD=1
Noise distribution (“problem”): Mean=5, SD=1;
d’=.5
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Modeling alarm reliability, combining evidence
High reliability: p(miss)=p(FA)=.1; Low: p(miss)=p(FA)=.3
Decision based on pre-trial inf.: p(turb|x)=p(x|turb)*p(turb)/p(x)
Decision accuracy after updating with additional information:
p(turb|x∩alarm)=p(turb)*p(x|turb)*p(alarm|turb)/p(x∩alarm)
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Modeling strategies
p(overallSuccess) = p(decisionCorrect) * p(decisionInTime)
Decision based solely on pre-trial information could [should]
be taken at the start of the trial, where p(decisionInTime)=1
and p(decisionCorrect)=p(turb|x)
Only wait for alarm if
p(turb|x) < p(turb|x∩alarm) * p(decisionInTime[alarmTime])
AlarmTime is unknown at the start of the trial, but mean
alarm time [5s, 13s] can be used as an approximation
Alarm makes biggest improvement at indifference point [4.5]
Strategies
With a low reliability alarm, never wait for the alarm unless in the early
alarm condition with a system-1 value close to the indifference point.
With a high reliability alarm and an early alarm, wait for the alarm even
for system-1 values around the distribution means (3.7-5.3).
With late alarms, only wait if the value is near the indifference point.
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Results: Alarm usage
Data suggests participants in the late
over-relied on low-reliability alarm and
under-relied on high-reliability alarm
alarm
condition
Proportion of trials where participants waited for alarm
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Results: Overall earning
Alarm timing and reliability affected overall earning
Main effect of reliability more pronounced in late condition
Need to compare with normative model [esp. p(turb|x)]
Need to analyse investment strategy
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Results: Participant feedback
Evidence for action bias
“Timeliness is the important characteristic, especially when even
some decisions made in 5-6sec are late! So, more important to send
the data quickly upon current estimates than wait for accuracy. [After
80 trials] Timeliness is important but accurate data again may not be
available in time, need to balance the two. [After 100 trials] I waited
long enough, about 5seconds for the alarm to occur. If it did, I almost
invariably based my decision on it, otherwise I made my decision on
Sensor1 data.”
A few late decisions may make participants over-cautious
“[after 20 trials] At first I waited to see system 2, but realised it was
too slow, so based my decisions on system 1.”
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Results: Overall success (trend)
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Results: Decision timeliness (trend)
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Results: Decision accuracy (trend)
Time Design – When-to-act problem – Model – When-to-act experiment
When-to-act: Evidence for action bias in a dynamic belief-updating task
Summary
Identified an important but under-researched temporal
reasoning problem that may severely compromise dependability
Provided a formal model (SDT + Bayesian updating) against
which empirical data can be compared
Conducted experiment manipulating alarm timing & reliability
Contrary to previous studies, obtained evidence for action bias
Feed into Timing book, TA Time Design tutorial, TA DFS
Future work
More detailed analysis of data [invest. strategy, time perception]
Modeling of empirical data
Additional experiments investigating alternative w2a
scenarios, different motivations for deferring/promoting
decisions, levels of uncertainty, visualisation of time
Derive design heuristics?
Time Design – When-to-act problem – Model – When-to-act experiment