Research you can use
Judith Olson
University of California Irvine
Three recent events that inspired this
“call to arms”
 Your role in this discussion
 What it means to have impact
 Kinds of impact


Recap of what it means to have impact
 Scope, Cost, Timeline…

Your pledge about making an impact

Theory based on
 The literature on teams
 Own own observations
and interviews of over 50
▪ Science Collaborations
▪ Corporate virtual teams

To verify theory
 Need data
 Online survey with advice to motivate participation
 They get the help and we get the data


Web accessible
assessment
tool
Assesses
 Strengths
 Challenges
 How to
overcome the
challenges

NSF
 Had us give a talk to Federal
funders in general
 “I have needed this for the last 10
years! Thank you.”

Teams who were assessed
welcomed advice
 “It drew out patterns in the way our
members work that we were not
conscious of, confirmed some of our
impressions, and allowed us to hear
frankly from our members.
 …useful as an independent
evaluation tool not tied to a funding
agency or other review panel”.

Object of study

…”to speed the translation
of laboratory discoveries
into treatment for patients.”
 “from bench to bedside.”

National Institute of Health:
 60 CTSA awards in 30 states plus DC
National Institutes of Health
Since 2006
$733 M
“While most researchers know
what is meant by Intellectual
Merit, experience shows that
many researchers have a less
than clear understanding of the
meaning of Broader Impacts.”

Many of us came to this field to change the
digital world
 Technology had gone awry
 Many early people attracted
to HCI were
“Children of the 60s”

Our careers were caught up in the reward
structures
 Industry
▪ Create new products
▪ Disincentive to make
findings available to others
 Academia
▪ Publish new findings
▪ Stay on topic, build a reputation

Where have all the impacts gone
 Long time passing



I will describe what I think it means to have
impact
I will list a number of ways we do and can
have an impact
You pledge…
 The card on your seat
▪ What other ways can you have impact
▪ How are you going to have an impact
▪ Collected by SVs at the door as you leave

What “counts”

Who is impacted?
 Theory gets used
 Students
 Downloads/views
 Developers
 Profits
 Consultants
 Degrees/Education
 Technologies
 Specific populations
 Lives changed
 The general public
 …..

Scopes differ
 You affect some people directly
▪ Interventions, teaching
 You enable others to be better at making better
products
▪ Toolkits
 You set policy
▪ Affect a large number of people

Time scales differ
 Now
▪ e.g. Action research
 1-3 years
▪ e.g., Publications
 20-30 years
▪ e.g., Theory  Assessment Tools
 40-50 years
▪ e.g., Cyberinfrastructure development
 ?
▪ e.g., Policy (like SOPA/PIPA)

Access?
 Free
▪ Toolkits…
▪ Wizard
▪ ….
 Fees
▪ Commercial Assessment Tools
▪ Products
▪ Educational degree
▪ ….


Theories
Assessment tools
Technological innovations
Guidelines, templates, patterns, toolkits and
standards
 Policies
 New media dissemination
 Action research
 Teaching and teaching materials
 …


What else?
Which kinds of impacts will YOU make?

“There is nothing so practical as a
good theory”
Kurt Lewin

“He who loves practice without
theory
 is like the sailor who boards ship
without a rudder and compass and
 never knows where he may cast”
Leonardo Da Vinci

Who
 Other researchers
 Consultants
 Tool developers

How
 Read and build on/test theory

Scope
 Small at first

Time scale
 1-3 or more years

Access
 Free


Theories
Assessment tools


Technological innovations
Guidelines, templates, patterns, toolkits and
standards
Policies
New media dissemination
Action research
Teaching and teaching materials




What else?
Which kinds of impacts will YOU make?

Collaboration Success Wizard

Globesmart
Myers-Briggs Personality Assessment
CogTool
…




Based on academic theories of
cultural differences
▪ David Matsumoto
▪ Handbook of Culture and Psychology

Like the Wizard, they collect
data to adjust their
assessments
▪ Recent upgrade used data from
400,000 users from over 60 countries

Based on the work of Carl Jung
 Developed further by Myers
and Briggs

Like the GlobeSmart you can see differences
in values and habits with people you interact
with
 Dimensions of discussion
 Some professional help

Based on work of Bonnie John
▪ Based on Card, Moran, & Newell
▪ GOMS and the Model Human Processor

A general purpose UI
prototyping tool
 It automatically evaluates your
design with a predictive human
performance model
▪ A “cognitive crash dummy”
▪ “You can compare expert use task time without recruiting
participants…An excellent choice for completely new systems
that don’t already have experts.”

Who
 General public

How
 Take the assessment

Scope
 Could be huge

Time scale
 Immediate

Access
 Some are free; some cost money


Theories
Assessment tools


Technological innovations
Guidelines, templates, patterns, toolkits and
standards
Policies
New media dissemination
Action research
Teaching and teaching materials




What else?
Which kinds of impacts will YOU make?


3-D programming environment
 For telling a story
 Playing an interactive game

Teaching tool for introductory
programming
 Formally shown to improve
learning and performance
Randy Pausch
Caitlin
Kelleher,
2006
Caitlin Kelleher

Using storytelling to make
computer programming
attractive to middle school girls

Storytelling Alice users
 spent 42% more time programming
 were more than three times as likely
to sneak in extra time to continue
working on their programs

10% of the nation’s colleges now use Alice
 An accompanying textbook, lessons, test banks

88% of “at risk” students who had Alice in a
pre-CS1 course were retained through CS2
 3.03 GPA

iMuse
 A requirements engineering
environment where both
developers and stakeholders
could understand the flow
Kristina Winbladh

Hypertext Transfer Protocol HTTP
 HTTP/1.1 spec
▪ Fielding, Gettys, Mogul, Frystyk and Berners-Lee
 WebDAV extension
 “Architecture of the Web”
▪ Fielding and Taylor

Aspect Oriented Programming
▪ Difference lies in the power, safety
and usability of the constructs provided
Crista Lopes
 Original article downloaded
6,681 times
 16,600 articles in Google Scholar
with “Aspect Oriented
Programming”

Who
 Students
 The general public
 Other developers

How
 Use the technology that makes things possible

Scope
 Huge

Time scale
 5-10 years

Access
 Often free (though products cost money)


Theories
Assessment tools


Technological innovations
Guidelines, templates, patterns, toolkits
and standards
Policies
New media dissemination
Action research
Teaching and teaching materials




What else?
Which kinds of impacts will YOU make?

All provide conventions
 So there is little new to learn
 Where things go, what they
look like
 Sometimes task flow guide

What are they based on? Are they consistent?
(Human Interface Guidelines)

Principles, patterns and
practices for improving
use experience
Christopher Alexander

Early instance:
Christian Crumlish &
Erin Malone
 Their effectiveness
depends on
▪ The research they are
based on
▪ The context in which
they arose
▪ Their fit to the context
they are being applied
to

UI Development environments
 With extra features
▪ Highly interactive
▪ Graphical
▪ Direct manipulation
▪ Automatic undo
▪ Support for animation
▪ Gesture recognition
 Amulet - C++
 Garnet – Common Lisp, X11, and Mac
Brad Myers
Keeping these up to date….

Who
 Developers
 End users

How
 Find and use relevant templates….

Scope
 Speeds development, makes software consistent

Time scale
 Immediate

Access
 Free


Theories
Assessment tools


Technological innovations
Guidelines, templates, patterns, toolkits and
standards
Policies
New media dissemination
Action research
Teaching and teaching materials




What else?
Which kinds of impacts will YOU make?






Stop Online Piracy Act (SOPA)/Protect
Intellectual Property Act (PIPA)
Network neutrality
Participatory design in Scandinavia
Open access vs. commercial production of
educational materials
Data sharing policies
……

Who
 Everyone

How
 Dictates what’s possible

Scope
 Huge

Time scale
 ?

Access
 Who gets to be in the conversation?


Theories
Assessment tools


Technological innovations
Guidelines, templates, patterns, toolkits and
standards
Policies
New media dissemination
Action research
Teaching and teaching materials




What else?
Which kinds of impacts will YOU make?

Total views = 14,660,471
Hans Rosling
A Swedish medical doctor, academic,
statistician and public speaker. He is
Professor of International Health
at Karolinska Institute[2] and co-founder and
chairman of the Gapminder Foundation,
which developed the Trendalyzer software
system.

All videos viewed 6,813,795 times

Who
 The public
 Students

How
 YouTube, TedTalks….

Scope
 Huge

Time scale
 Immediate

Access
 Free


Theories
Assessment tools


Technological innovations
Guidelines, templates, patterns, toolkits and
standards
Policies
New media dissemination
Action research
Teaching and teaching materials




What else?
Which kinds of impacts will YOU make?

Helping teachers of autistic
children assess behavioral
incidents
Gilllian Hayes

Helping caretakers
and clinicians of
preterm infants
monitor their
movement and
other key factors

Who
 Target population starting with a small
group

How
 New technologies to help critical situations

Scope
 Small at first, larger as results are
generalized

Time scale
 Immediate

Access
 Free


Theories
Assessment tools


Technological innovations
Guidelines, templates, patterns, toolkits and
standards
Policies
New media dissemination
Action research
Teaching and teaching materials




What else?
Which kinds of impacts will YOU make?

Undergraduate teaching
 6,970 students in a career

Ph.D. students
~40
▪ A multiplier because they go on to teach

Teaching materials
 Books for classes
 Cases, exercises

Online resources
that educate
Useit.com
UsabilityFirst.com
Hcibib.org

Teaching or action kits
National Center for Women
in Information Technology
NCWIT

NCWIT
David Evans &
Sebastian Thrun

Who
 Students

How
 Exposed to lectures, exercises, assessments

Scope
 Digital media is the multiplier

Time scale
 1-2 years

Access
 Sometimes free; sometimes requires tuition


Theories
Assessment tools
Technological innovations
Guidelines, templates, patterns, toolkits and
standards
 Policies
 New media dissemination
 Action research
 Teaching and teaching materials
 …


What else?
Which kinds of impacts will YOU make?

Who is impacted
How
Scope
Time Scale
Access

Decisions you have to make…





What “counts”

Who is impacted?
 Theory gets used
 Students
 Downloads/views
 Developers
 Profits
 Consultants
 Degrees/Education
 Technologies
 Specific populations
 Lives changed
 The general public
 …..

Scopes differ
 You affect some people directly
▪ Interventions, teaching
 You enable others to be better at making better
products
▪ Toolkits
 You set policy
▪ Affect a large number of people

Time scales differ
 Now
▪ e.g. Action research
 1-3 years
▪ e.g., Publications
 20-30 years
▪ e.g., Theory  Assessment Tools
 40-50 years
▪ e.g., Cyberinfrastructure development
 ?
▪ e.g., Policy (like SOPA/PIPA)

Is it free?
 Yes
▪
▪
▪
▪
Khan Academy
Open Knowledge
Standards, toolkits, patterns
Wizard
 No
▪
▪
▪
▪
▪
▪
Udacity
Meyers Briggs
GlobeSmart
Textbooks
Degree programs
Products

How to translate our research to have
broader impacts?

How to guarantee quality?
 E.g. evidence based medicine

How to make it accessible?

How to evaluate impact?

How careers are advanced now
 Product innovation
 Publications

Future
 + Impact

It takes the evaluators to change the system
 Promotion policy


Theories
Assessment Tools


Popular technologies that become standards
Guidelines, templates, patterns, toolkits and
standards
Policies
New media dissemination
Action Research
Teaching and teaching materials
…





Student Volunteers will collect on the way out

In the interest of potential impact
 A video of this will appear on the ACM website
and the ACM-W website

jsolson@uci.edu