User Satisfaction:
Illusion or Paradox?
Charlene R. Weir, PhD
Professor
Department of Biomedical Informatics
3515B
OVERVIEW

Current state of the literature (2015 only)
 Instruments used
 Correlation with adoption

Purpose and Related Theories
 Proxy
 Proxy

for adoption
for effectiveness
Alternative View
 Focus on Behavior
 Motivation
 Task-based
(usage)
assessments of effectiveness
EXAMPLE (VA)
Recent Studies(2015)
Assessing User Satisfaction
# of studies 32
 Intention to use 12
 Adoption Behavior (only usage) 7
 Information Performance 5
 Theories 15

What is the question that User
Satisfaction is supposed to answer?

A proxy for adoption?

A proxy for effectiveness?
A Proxy for Adoption

What IS Adoption?
 Intention
to Use
 Usage/Behavior
 Customization

What are relevant theories?
 Technology
Adoption Model
 Universal Theory of ***
DIFFUSION OF INNOVATION
A staged communication process
Early
Innovators
Innovation Characteristics

Relative Advantage – compared to other things

Compatibility - How consistent the innovation is with the
values, experiences of adopters.

Complexity - How difficult the innovation is to understand

Triability - The extent to which the innovation can be
tested or experimented with

Observability - The extent to which the innovation
provides tangible results.
MOTIVATIONAL THEORIES
Theory of Planned Behavior





Behavioral belief / Perceptions of Outcomes:
beliefs regarding outcomes of XXX PLUS the
subjective probability that the behavior will produce
that outcome. (CPOE decreases errors, improves
communication, improves quality of care)
Attitude toward behavior: Perceived value for the
outcome (good thing/bad thing)
Normative beliefs: perception of social influence
Perceived behavioral control: beliefs about it
actually working (e.g. support, mandated, resources).
Control beliefs: The concept of perceived behavioral
control is conceptually related to self-efficacy.
UTAUT (Unified Theory of Acceptance and Use of Technology)
Venkatesh, V., Morris, M.G., Davis, F.D., and Davis, G.B.(2003) “User Acceptance of Information Technology: Toward a Unified View,”
MIS Quarterly
MODERATORS
TAM
Technology Acceptance Model
SUS (System Usability Scale)






I think that I would like to use this system frequently
I found the system unnecessarily complex
I thought the system was easy to use
I think that I would need the support of a technical
person to be able to use this system
I found the various functions were integrated
I thought there was too much inconsistency in the
system


I would imagine that most people would learn to use this
system very quickly
I found the system very cumbersome to use
 I felt very confident using the system
What is WRONG with this picture?
Why do we care?

Most usage is mandated in clinical information systems

Adoption measures are not correlated with outcomes

Adoption is not correlated to implementation strategies

Models developed to PREDICT usage do not predict
“making decisions better,” “workflow more efficient”
or “enhanced coordination”

Can’t be used for design

Results are always in the 3.2 to 4.2 range (1 to 5 scale)

We already KNOW the theories are empirically validated
A Proxy for Effectiveness

What is effectiveness?
 System
performance
 IT support
 Information Management tasks
 Documentation Goals
 Communication/Coordination

What are relative theories?
 DeLone
and McLeon
 Task/Technology Fit
 Information Theory
DeLone W, McLean E. (2003) The DeLone and McLean Model
of Information Systems Success: A Ten-Year Update
Task/Technology Fit
Goodhue & Ammenworth
WHAT ARE the NET
BENEFITS of Clinical IT?
Example:
Assessing Satisfaction and User
Behavior for CPRS Adoption
Methods – Task Instrument
Interviews of 24 Primary Care Practitioners
 Week-long observation of 2 primary clinics
 Action Identification Interviews

 “Describe
what you are doing”
Qualitative Analysis
 Generation of 34 items
 Factor Analysis with 360 providers
 Reduction to 22 items (TASK)

Theory of Action Identification
Vallacher R, Wegner D. What do people think they are doing? Action
identification and human behavior. Psych Rev. 1987;94:3–15

Action identities
“what are you doing?”
 “ You are doing XX in order to do XX?”
 “ You are doing XX BY DOING YY”?


Hierarchical construction
 Goals/Values
on top
 Actions at bottom

Experts report high levels; novices low
levels
Typical Action “Identities”









Reporting required information
Making a decision about the best course of
treatment
Looking for patient data
Checking over the patient status
Becoming familiar with the patient
Finding info to teach pt.
Looking for the evidence on the best treatment
Determining the # of procedures done per day
Looking up local policies
Results
FACTOR NAME
# Items
% Var
Patient Assessment
5/6
38.8%
Data Integration
4/4
17.6%
Communication
4/5
8.3%
Work Compliance
2 /4
7.2%
Tracking Workload
3/5
5.8%
Learning New Information
None
NA
Methods – Validation

N=124





26 Primary Care Practitioners
18 Other specialists
14 Pharmacists
44 RNs
22 Residents
Pre/Post CPRS Implementation
 92% response rate

Measures of User Satisfaction

Task Effectiveness (TASK; n= 22 items)
“Information Tasks”
 Three questions each on ratings of effectiveness




Communication
Decision-Making
Policy Compliance
 Total

• Documentation
• Workload Tracking
• Data Review
Summed: alpha = 0.90
Usability (3 items)
 Easy to use, minimal effort; easy to navigate

Total Summed: alpha = 0.93
Measures of User Satisfaction

Intrinsic Motivation (AFFECT)
 Three
items assessing affective experience
Interest
 Enjoyment
 Fun

 Items

summed to form scale: alpha = 0.68
User Satisfaction (GENERAL)
9
items (Doll, et al) (comprehensive, accurate,
easy to use, useful, improves my work)
 1 factor (alpha= 0.84)
Correlation Results
RESULTS:
User Satisfaction Measures
5
4.5
4
3.5
GENERAL (9)
TASK (22)
USABILITY (3)
AFFECT (3)
3
2.5
2
1.5
1
MEAN (1-5)
Early vs Late Adopters:
Progress Notes in Primary Care
5
4.5
4
3.5
EARLY
LATE
3
2.5
2
1.5
1
GENERAL
TASK
USABILITY
AFFECT**
Level of Adoption
Personal Customization
Sum of 12 customization options used
MEASURES
GENERAL
TASK
USABILITY
AFFECT
LEVEL OF ADOPTION
0.25
0.35**
0.23
0.26
RESISTANCE RATINGS

Taken from Reactance Theory
 Reactance
results from a threat to control
 Results in demeaning the messenger; dismissing the
message, avoiding the threat, and reasserting control
(refusals, hording paper)
Four members of the implementation team
rated 24 primary care providers on:
1 (none) to 7 (very high) scale
 Ratings were averaged.

RESISTANCE RATINGS
Correlation Results
MEASURES
GENERAL
TASK
USABILITY
AFFECT
RESISTANCE RATINGS
-0.18
-0.49*
-0.20
-0.41*
STAGES OF CHANGE
Prochaska & DiClemente


Combines many motivational theories
Proposes four stages in change process,
characterized by attitude, intentions and
behavior.
 Pre-Contemplation
- not concerned, little attention
 Contemplation - thinking about the pros & cons
 Action - formed an intention; making plans
 Maintenance - behavior in place; well-learned
STAGES OF CHANGE
They checked one (out of 5)
 I use the system only when I have to and I do not intend


to try to expand my use in any way. I see no reason to
learn much more about it.
I have begun trying to integrate using the computer in my
daily work processes ad am now partially successful. I
am learning more about it all the time.
Even if the system were not mandated, I would use it.
My work is extensively dependent on the computer.
Stage of Change
Mean values across stage for each measure (ANOVA)
A good measure “picks up” the change across stages
5
Early Cont.
Cont.
Action
Main
4.5
4
3.5
3
2.5
2
1.5
1
GENERAL
TASKS**
USABILITY
AFFECT**
Involvement/Volunteer



Asked if they wanted to get more information and
would like to volunteer.
Involvement Group = either question (n=57)
Non-involvement group (n=20) = checked none
INVOLVEMENT
5
NON-INVOLVE
INVOLVE
4.5
4
3.5
3
2.5
2
1.5
1
GENERAL
TASK
USABILITY
AFFECT**
So What?

These results had NO impact on developers

Did not inform implementation

Might be used to compare across settings

Need to develop metrics that can relate Usability
measures to User Satisfaction

ANSWER:




Focus on Tasks – relate them to goals
Link Usability Assessments to these Tasks
Link tasks to implementation strategies
Link tasks to clinical outcomes
Questions?