Development of a Measure of
Systems Thinking:
A Key Component of the
Advancement of the Science of CQI
Shirley M. Moore, RN, PhD, FAAN
Frances Payne Bolton School of Nursing
Case Western Reserve University
Cleveland Ohio
Acknowledgements

Team:
Co-PI: Mary A. Dolansky, RN, PhD
Brian Ragan, PhD, Psychometrician
Mamta Singh, MD, MS, Internal Medicine
Physician, VA Quality Scholar
Farrokh Alemi, PhD, Systems Engineer

Funding:
Robert Wood Johnson Foundation
9/1/08-2/28/10
Background


Ability to engage in Systems Thinking
is viewed as a key component in the
success of CQI initiatives
Systems Thinking is one of the 5 core
knowledge areas needed for learning
organizations (Senge, 1990)
Definition of Systems Thinking
An individual’s or team’s ability to
consider a group of interdependent
people, items, processes and products
and services that have a common
purpose or aim
Nolan & Provost, 1990
Purpose
To develop and conduct psychometric testing
on a new measure of systems thinking –
Systems Thinking Scale (STS)
We envision that the STS will:
Consist of 12-20 items
Use a Likert-type response scale
Be completed in less than 5 minutes.
Research Questions
1)
What is the validity (content, construct,
concurrent criterion-related, discriminate,
and predictive) of the STS?
2) What is the reliability (internal consistency
and test-retest) of the STS?
3) What is the feasibility (acceptability, ease
of use) of the STS?
Phase I



Identify domains (grounded theory) using
CQI systems experts in a series of electronic
focus groups
Develop question items
Establish content validity
Phase II


Preliminary field testing for item clarity
and feasibility of the instrument (15
healthcare professionals)
Revisions will be made to the items and
the instrument will be formatted into a
provisional instrument
Phase III
Construct validity is the degree to which a
measure performs in accordance with
theoretical expectations and is the best way
to assess validity for abstract constructs
(Carmines & Zeller, 1979)
------------------------------------------------------Construct validity will be assessed using
exploratory factor analysis (N=300)
Phase III
Concurrent criterion-related validity Confirmed
when scores on an instrument are correlated
to a related criterion at the same point in
time (Carmines & Zeller, 1979)
--------------------------------------------------------Scores on the STS will be compared with
scores on the QI Knowledge Application
Tool (QIKAT)
Phase III
Discriminate validity (Known-group difference
validity) Confirmed when scores on an
instrument are able to discriminate among
groups (Carmines & Zeller, 1979).
------------------------------------------------------The known groups will be a group of students
who receive systems thinking education
(N=90) and a group of students in a general
health professionals course (N=90) who do not
receive education on systems thinking
Phase III
Predictive validity Degree to which test scores
predict performance in the future (Schultz &
Whitney, 2005)
------------------------------------------------------Predictive validity will be supported if we
find that higher scores on the STS are
found in individuals on CQI teams that
have greater success than CQI teams that
are less successful
Reliability


Internal consistency
Test-retest
Partners





Case Western Reserve University
University Hospitals Case Medical Center
Metrohealth Medical Center
Cleveland VA Medical Center
Cleveland Clinic
Progress to Date


Expert panel developed a working
definition of Systems Thinking and its
essential domains were identified
Draft of instrument items developed
Working Definition:
Systems Thinking




The ability to recognize, understand, and synthesize the
interactions, and interdependencies in a set of components
designed for a specific purpose.
This includes the ability to recognize patterns and repetitions
in the interactions and an understanding of how actions and
components can reinforce or counteract each other.
These relationships and patterns can be temporal, spatial,
social, technical or cultural.
It is fundamental to undertaking specific methodology or
strategies to explore and redesign a set of components
comprising a whole.
Domains








Sequence of events
Causal sequence
Multiple causations possible
Variation of different types (random-special)
Interrelations of factors
Patterns of relationships
Feedback loops
Mechanisms that explain the cause and effect
Mechanisms that explain the cause and effect
Response Format
Instructions:
Please read each of the statements and place an “x” in the
answer box that indicates frequency of agreement with the
statement:
When I want to make an improvement…
Never
1. I believe the harder I
work the more
improvement I can
make.
2. ……...
Seldom
About
Half the
Time
Often
Most of
the
Time
X
X
Examples of Items (8 of 30)




I believe the harder I work the more
improvement I can make
I think recurring patterns are more
important than any one specific event
I believe understanding how the chain of
events occur is crucial
I think that lasting change relies on personal
effort and motivation
Examples of Items




I focus on the opinions of a champion in the
system
I believe lasting change involves more than
personal commitment
I think my first impressions turn out to be very
useful
I believe small changes can produce important
results
Challenges



Representativeness of the initial item bank of
the STS
The lack of a current “gold standard”
measure of systems thinking for comparison
Score the STS to obtain a “team” measure of
systems thinking?
Challenges



Context of system thinking – only in regard
to quality improvement or in other contexts,
i.e., systems biology, engineering
Systems thinking regarding process
improvement vs. any kind of change
Can it be taught? Changed?