Verity Questionnaire Development
The truth about
outcomes questionnaires!
 All patient self report outcome questionnaires tend to
load on a common factor: “global distress”.
 Due to the high degree of correlation between items,
well constructed questionnaires of 10-15 items can
have coefficients of reliability and construct validity
comparable to measures of 30 or more items.
 Even ultra brief questionnaires of 4-9 items may have
adequate reliability and validity for must measurement
needs.
Measurement 2.0
Measurement 1.0 0
Measurement 2.0
 Reliance on copyrighted and
published questionnaires
 Item banks and resulting
questionnaires belong to community
of users.
 Copyright holder may charge
fees for the use of
questionnaires
 Copyright holder may place
conditions or restrictions on
the use of questionnaires
 No fees for questionnaires
constructed from items in the shared
item bank
 Each organization is responsible for
their own measurement methods and
determines the appropriate content
and use of questionnaires
Measurement 2.0
Questionnaire Development
Measurement 1.0
0
Measurement 2.0

A pool of items are tested in various samples

A pool of items are tested in various samples


Item analysis used to select items for final
questionnaire
Item analyses employ both Classical Test Theory
and Item Response Theory

Item analysis used to select items for multiple
versions of the questionnaires, depending on
the needs of the users




Questionnaire validated, usually in correlation
studies with questionnaires measuring the same

construct
Questionnaire published in final form

Various organizations may conduct correlation
studies to satisfy internal skeptics.

Multiple versions available, with the community
of users determining which versions offer the
greatest utility

Online manual constantly updated as data
accumulates and needs of users dictate
Manual published
Many years may pass before a new version is
published
Construct validity determined by factor analysis
and comparison of results to known constructs.
Item Pool
 Item content drawn from multiple sources
 Items reworded for consistent format, brevity and ease
of use
 Item formats tested to identify optimal format for brief
questionnaires
Item pool
Item Format
5 point Likert scale found to produce best results
Items scored 0 – 4
Mean item response tends to be in middle of range
4 or 3 point Likert scales tend to produce responses
skewed toward on end of the scale
Liker scales with more than 5 points provide no
additional benefit
Ease of use
Items written to 4th grade reading level
Simple to understand frequency anchors
 Never, Hardly Ever, Sometimes, Often, Very Often
Common sentence structure aids rapid completion
How often in the past two weeks did you
…feel unhappy or sad?
…have little or no energy?
Reliability
Reported as coefficient of correlation (r)
Test-retest reliability- correlation between
measure at two points in time
Less relevant to measures of change
Coefficient alpha - measure of internal
consistency; reflects degree of inter-correlation
between items on the same scale.
Useful for measures designed to be sensitive
to change.
Low reliability
Low validity
Validity is a function of reliability!!!!!
Low reliability leads to low correlations with
other measures
Validity
You better have face validity!!!!!
Construct validity based on correlation to a
known “construct” such as Global Distress
Factor analysis used to identify “constructs”
Concurrent validity based on correlations
between measures of a common construct
ACORN correlation with Beck Depression=.78
ACORN correlation with Patient Health
Questionnaire-9=.82
Reliability & item count
More items=greater reliability
Assuming they correlate with the same factor
(construct)
Each additional item adds less additional reliability
Little additional gain beyond 15 well chosen
items
Coefficient alpha =>.9 recommended for
measures of individual change
Slope of change
Outcome questionnaires need items that
change in therapy.
Items vary in their likelihood to show change.
Linear regression used to calculation the slope
and intercept for each item
Intake score used to predict last score
Final score = Intake Score * slope + intercept
Regression lines
 Feel unhappy or sad (Slope=.45; Intercept=.88)
 Hear voices when alone (Slope=.63; Intercept=.06)
Item score at last session
3
2.5
feel unhappy or
sad
2
1.5
here voices when
I was alone
1
0.5
0
0
0.5
1
1.5
2
2.5
Item score at intake
3
3.5
Item Change Calculator
Excel tool with normative data on all items
Mean intake, slope & intercept
Used to predict psychometric properties of new
questionnaires
Used to assure equivalency of alternative forms
of questionnaires
Item Response
Theory
Permits us to evaluate how well items works
for clients at varying levels of distress
Identify items sensitive for clients with very
high levels of global distress scores
Thoughts of suicide; Worthless
Outpatient questionnaires ideally targeted
for moderate to severe cases with a mix of
items
Youth and child measures
Different mix of items for children or adolescents
More complex factor structure
Relatively weak sub factors for “externalizing” and
“internalizing” items
 Items tapping ADHD & conduct problems
 Items tapping symptoms of depression and anxiety
 Subscales both correlate with Global Distress
Child/Youth measures tend to require more items to
achieve .9 reliability on Global Distress scale
Child or parent completed?
Questionnaires may be completed by the child
or by an adult who knows the child well.
Different norms depending on who completes
the questionnaire.
Adolescent & child measures produce similar
scores for self or parent completed forms
“Recovery” questionnaires
Designed for clients in substance abuse
recovery programs.
Contains both Global Distress Scale and
Substance Abuse Recovery Scale
Items on Substance Abuse Recovery Scale have
high face validity for recovery settings.
Substance Abuse Recovery
Sample items
…feel accepted by your recovery support group
…feel sad or angry that you can’t use drugs or
alcohol like other people
…participate in activities that support a clean
and health lifestyle.
…feel a need to use alcohol or drugs
Client registration form
Completed at start of treatment
Used to collect case mix variables such as:
Age
Sex
Diagnosis

History of physical
and/or sexual abuse

Prior treatment history
Language & ethnicity
Health status

Focusing on the relationship
“…decades of research consistently demonstrate that
relationship factors correlate more highly with client
outcome than do specialized treatment techniques…
therapists need to remember that the development and
maintenance of the therapeutic relationship is a primary
curative component of therapy and that the relationship
provides the context in which the specific techniques
exert their influence.”
- Lambert & Barley (2001)
Psychotherapy, 38(4):357-361
Relationship building is an
Evidence Based Practice!
Practitioners are encouraged to routinely monitor
patients’ responses to the therapy relationship and
ongoing treatment. Such monitoring leads to
increased opportunities to repair alliance ruptures,
improve the relationship, modify technical strategies,
and avoid premature termination.
- Norcross & Lambert (2006) in Evidence-Based Practices
in Mental Health, Norcross, Beutler & Levant (Eds), p. 218
Therapeutic alliance
as a key ingredient
 Large body of research on therapeutic
relationship and working alliance show that it is
an important factor in the outcome of
psychotherapy
 Horvath and Symonds, 1991: Meta-analysis of
24 studies attributed 26% of the difference in
case outcomes to differences in alliance
Concept of Therapeutic Alliance
Three Components:



Tasks: Behaviors and processes within the therapy
session that constitute the actual work of therapy
Bonds: The positive interpersonal attachment between
therapist and client of mutual trust, confidence, and
acceptance
Goals: Objectives of therapy that both client and
therapist endorse
Therapeutic Alliance Scale
Sample items
I felt like we talked about the things that were
important to me
I felt like the therapist liked and understood me
I felt the session was helpful.
I felt confident that the therapist and I worked
well together
Did you feel that the clinician understood what
it was like to be you
Alliance Scale Psychometrics
Items are heavily skewed in positive direction.
Scale scores are not normally distributed.
Cannot calculate reliability & validity using
parametric statistics that assume normality of
distribution
Items are only as “valid” as clinician’s
ability to illicit honest and frank
responses!
Alliance Results
Measurement make a difference
1.2
Highly effective range
Effect Size
1
0.8
0.6
Effective
Range
0.4
0.2
0
Alliance items completed No items alliance at start
at start of treatment
of treament (n=1192)
(n=1924)
High praise for clinicians
Alliance scores at start of treatment
10%
Perfect (Alliance=0)
39%
Almost Perfect
(Alliance<1)
51%
Room for improvement
(Alliance=>1)
Alliance changes in treatment
8%
Alliance Change for
Better
No Change
47%
45%
Alliance Change for
Worse
Alliance change & outcome
1.20
Highly effective range
Effect Size
1.00
0.80
Effective
Range
0.60
0.40
0.20
0.00
Alliance Change for
Worse
No Change
Alliance Change for
Better