Rohling’s Interpretive Method:
How Can a Flexible Battery
Perform Like a Fixed Battery
Martin L. Rohling, Ph.D.
Associate Professor
Department of Psychology
University of South Alabama
Clinical vs. Mechanical Diagnosis

Much research has been conducted since Meehl
(1954) found clinical judgment to be less accurate
than mechanical or “actuarial” judgement



2
e.g., Dawes, Faust, & Meehl (1989); Filskov (1981); Garb (1989); Garb
(1994); Garb (1998); Grove et al. (2000); Sawyer (1966); and Wedding
& Faust (1989)
Such results influential in causing NPs to turn to
different versions of the HRB (Russell, 1998).
“Batteries” have been defined as the method by
which one can avoid the clinical errors highlighted
by Meehl an others, using “actuarial” rules for
diagnosis (Russell, 1995; Russell et al., 2005).
January 13, 2007
Rohling - CCPN Orlando, FL
Rohling’s Interpretive Method
(RIM): Development History


Conducted several meta-analysis with Dr.
Laurence Binder at the Portland, OR – VA
The last of these focused on the residual cognitive
effects of mild head injury.


Binder et al. grouped effect sizes (ES) into
domains of neuropsychological functioning based
on factor analytic studies.

3
Binder, Rohling, & Larrabee (1997)
e.g., Leonberger, Nicks, Larrabee, & Goldfader (1992)
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Generated from
Meta-Analytic Procedures

Meta-analysis (MA) combines effect sizes (ES)
across samples assuming that they all sample the
population M for the particular effect of interest.



4
Common method ES calculation is a standardized
mean difference score (e.g., Glass’ delta).
delta = difference between con. & exp. group’s M’s
divided by con. group’s SD.
delta analogous to Z score - linear equivalent of T
score used in clinical neuropsychology
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Generated from
Meta-Analytic Procedures


Binder et al. (1997) combined ES’s generated
from various tests into cognitive domains.
Why not similarly combine ES’s, or T scores, from
a single patient into cognitive domains in the
same way that it is accomplished in MA.


5
Each test score is treated as a ES that reflects the
individual’s ability within a domain.
ES can be combined based on homogeneity of
variance, so as to avoid combining apples and
oranges.
January 13, 2007
Rohling - CCPN Orlando, FL
Introduction to the RIM Analysis

Flexible battery (multiple measure) use:


Is the most frequently cited model of assessment
among neuropsychologists.
 Only 7% of neuropsychologists use a fixed
battery (Rabin et al, 2006, ACN).
Regarding the suitability, practicality, and
usefulness of any fixed battery:

“We know of no batteries that fully satisfy these
criteria.”
(Lezak, Howieson & Loring 2004, Neuropsych. Assess., 4th
ed, p 648.)
6
January 13, 2007
Rohling - CCPN Orlando, FL
Advantages of Flexible Battery





7
Dynamic & responsive to clinician’s needs
Covers 1 or many domains
“Flexible”, can be adapted for each patient
Can “oversample” domains
Well suited for hypothesis-driven approach
January 13, 2007
Rohling - CCPN Orlando, FL
Potential Problems with a
Flexible Battery





8
Inflated error rates
Multicollinearity
Weighting decision problems
Unknown veracity/reliability of sets of tasks
Human judgment errors
January 13, 2007
Rohling - CCPN Orlando, FL
Human Judgment Errors
(Wedding & Faust, 1989, ACN)





9
Hindsight bias
Confirmatory bias
Overreliance on salient data
Under-utilization of base rates
Failure to take into account co-variation
January 13, 2007
Rohling - CCPN Orlando, FL
Potential Benefits with
Rohling’s Interpretive Method (RIM)

Judgment errors can threaten reliability &
validity of multiple measure test batteries.

RIM was designed to reduce these effects.
10

Based on meta-analytic techniques.

Uses a linear combination of scores placed on a
common metric.
January 13, 2007
Rohling - CCPN Orlando, FL
Potential Benefits of RIM
11

A strategy that produces summary results
analogous to those generated in a fixedbattery approach (e.g., HII, GNDS, AIR).

Takes advantage of psychometric
properties of same metric data, e.g., T
Scores.
January 13, 2007
Rohling - CCPN Orlando, FL
Today’s Presentation - Intent

Present a set of procedures that allows for a
quantitatively-based comparison of an
overall battery of measures.



Demonstrate importance and practicality of
use of established statistical indices.

12
Non-specific to battery measures
themselves.
Can be used for any individual patient.
(e.g., alpha, beta, effect size).
January 13, 2007
Rohling - CCPN Orlando, FL
Today’s Intent (cont’d)

Present a data format for any set of
measures to be inspected at:



13
Global level (OTBM)
Domain level (DTBM)
Test measure level (ITBM)

Present a series of calculations to assist in
the generation of these indices.

Present Steps in conjunction with clinical
judgment from an informed position.
January 13, 2007
Rohling - CCPN Orlando, FL
Common RIM Domains of
Functioning





Symptom Validity (SV) Tests
Emotional / Personality (EP) Measures
Meta-Cognition, Pain, or other self-ratings
Estimated Premorbid General Ability (EPGA)
Test Battery Means


Cognitive Domains:


VC, PO, EF, AML, VML, AW, PS
Non-Cognitive Domains:

14
Overall (OTBM), Domain (DTBM), & Instrument
(ITBM)
PM, LA, SP
January 13, 2007
Rohling - CCPN Orlando, FL
Sample RIM: Summary Table
RIM Summary Statistics: T-scores, variances, ES, & power.
Column #
1
2
3
4
5
6
7
8
9
10
11
12
Cognitive Domain
M
sd
n
Hetero.
p value
Classify
% TI
ES
CI
PreM
Nec.
1-sample
t test
anova
s&w
Power
(1-b)
1 Symptom Validity (SV)
46.8
9.8
8
---
Average
25%
-.32
+ 5.8
52.6
---
---
.2045
2 Emotional Personality (EP)
30.6
17.2
10
.0033
Mld-Md
60%
-1.38
+ 9.0
39.6
.0060
---
---
---
---
---
---
---
---
---
---
---
---
---
---
4 Est. Pre. Gen. Ability (EPGA)
45.2
5.7
8
---
Average
13%
---
+ 3.3
41.9
---
---
---
5 Overall TBM (OTBM)
36.6
12.4
69
.0063
Mild
57%
-.89
+ 2.5
39.1
<.00012
---
---
6 Domain TBM (DTBM)
38.0
6.6
7
---
Mild
57%
-1.17
+ 3.1
41.1
.02721
.0068
---
7 Instrument TBM (ITBM)
33.0
13.5
12
.0855
Mld-Md
67%
-1.18
+ 6.5
39.5
.00961
---
---
8 Verbal Comprehension (VC)
38.6
7.1
6
---
Mild
67%
-1.03
+ 4.8
43.4
---
---
.6989
9 Perceptual Organization (PO)
33.9
10.2
6
---
Mld-Md
50%
-1.37
+ 7.0
40.9
.04321
---
---
10 Executive Functioning (EF)
40.1
12.3
12
---
Blw Avg
50%
-.53
+ 5.9
46.0
---
---
-.5306
11 Auditory Mem. & Learn (AML)
32.0
11.5
20
---
Mld-Md
70%
-1.45
+ 4.2
36.2
<.00011
W
---
12 Visual Mem. & Learn (AML)
29.5
13.4
11
---
Moderate
82%
-1.52
+ 6.7
36.2
.00311
W
---
13 Attention/Work Memory (AW)
45.0
9.0
9
---
Average
22%
-.03
+ 5.0
50.0
---
---
.0591
14 Processing Speed (PS)
46.9
10.6
6
---
Average
17%
.20
+ 7.1
54.0
---
---
---
---
---
---
---
---
---
---
---
---
---
---
---
16 PsychoMotor (PM)
44.7
11.6
4
---
Blw Avg
25%
-.49
+ 7.5
52.2
---
---
.1919
17 Language/Aphasia (LA)
13.0
---
1
---
Severe
100%
---
---
---
---
---
---
18 Sensory Perception (SP)
---
---
---
---
---
---
---
---
---
---
---
---
R
o
w
3 Meta-Cognition (MC)
15 Global/Miscellaneous (GM)
15
January 13, 2007
Rohling - CCPN Orlando, FL
Sample RIM: Graphic Display
100
90
80
70
60
50
40
30
20
16
January 13, 2007
SP
LA
PM
PS
AW
VML
AML
EF
PO
VC
ITBM
DTBM
OTBM
EPGA
EP
0
SV
10
Rohling - CCPN Orlando, FL
Brief of RIM Steps:
• There are 24 steps to the RIM process
 17 calculation steps:
 Advice on design of the battery
 Calculation of summary statistics
 Generation of graphic displays
 7 interpretative steps.
• Detail a systematic procedure for use of the
statistical summary table and graphic displays to:




Assess and verify summary data.
Identify strengths/limitations of current data.
Obtain a reliable diagnosis.
Develop tx plans based on sound judgments.
• We briefly review each step in just a moment.
17
January 13, 2007
Rohling - CCPN Orlando, FL
Support for the RIM Process


18
Rational support/reasoning: Reduce clinical
judgment errors.
 The RIM is a Process, not a program.
 Rather, the RIM is a way of formalizing
thinking & interpretation of individual
case data.
This is operationalizing what many flexible
battery clinicians are already doing in their
head.
January 13, 2007
Rohling - CCPN Orlando, FL
Support for the RIM Process:
Specific Advantages

Psychometric properties at level with fixed, conormed batteries, without their limitations.

Flexibility of test selection.

Flexibility of theoretical view of cognition
(domain structure)
19
January 13, 2007
Rohling - CCPN Orlando, FL
Support for the RIM Process:
Specific Advantages


20
Quantitatively support your conclusions and
interpretations
 Statistical evaluation
 Measure of confidence in findings
 Measure of limitations of findings
 Ability to present data at different
levels of interpretation
Greater defensibility
January 13, 2007
Rohling - CCPN Orlando, FL
The RIM has a Set of
Procedure or Specific Steps
21
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Steps 1-4: Summary Data
1)
Design & administer battery.

2)
Estimate premorbid general ability.

3)
We recommend T scores, but z or SS OK too.
Assign scores to domains.

22
Use Reading (WTAR), Regression (OPIE-III), &
academic records (rank, SAT, ACT).
Convert test scores to a common metric.

4)
Use well standardized recently normed tests.
Factor analysis to support assignment
(Tulsky et al., 2003)
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Steps 5-8: Summary Data
5)
6)
Calculate domain M, sd, & n.
Calculate test battery means (TBM).


Overall TBM – All scores, large N & high power.
Domain TBM – Avoids domain over weighting.


7)
Have you put “apples & oranges” together?
Determine categories of impairment.

23
Instrument TBM – One score per norm sample.
Calculate p for heterogeneity.

8)
(e.g., attention & memory).
Recommend using of Heaton et al. (2003).
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Steps 9-12: Summary Data
9)
Determine % of test impaired.

Analogous to Halstead Impairment Index

10)
Calculate ES for all summary stats.

11)
90% CI = 1.645 x SEM
Upper limit of performance for impair.

24
Use Cohen’s d = (Me – Mc) / SD pooled
Calculate confidence interval for stats.

12)
# scores below cutoff / total # of of scores
Look for overlap between 90% CI of EPGA
(lower) & Summary Stats (upper)
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Steps 13-17: Summary Data
13)
Conduct one-sample t tests.

14)
Conduct a between-subjects ANOVA.

15)
17)
25
Looking for strengths & weaknesses
Conduct power analyses.

16)
Use EPGA as reference point
Only needed for those NS differences
Sort scores for visual inspection.
Graphically display summary statistics.
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Steps 18-20: Interpretation
18)
Assess battery validity.



Examine the Symptom Validity scores.
Caution in accepting low power results.
Look at heterogeneity of summary stats.
1.
2.
3.
19)
Examine influence of psychopathology.

20)
26
Normative sample unrepresentative of patient.
Scores assigned to wrong domain.
Inconsistent performance on construct measures.
Examine scores for heterogeneity.
Check OTBM, DTBM, & ITBM impaired.
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Steps 21-24: Interpretation
21)
Examine strengths/weaknesses looking at:
Confidence intervals overlap.
2. Results from one-sample t tests.
3. Results of ANOVA.
4. %TI show differences otherwise not evident.
Determine if pattern existed premorbidly.
1.

22)
Examine non-cognitive domains.

23)
24)
Explore Type II errors –need more tests?
Examine sorted T-scores

27
Psychomotor, Lang/Aphasia, Sensory Percept
Look for patterns missed by summary stats.
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Sample Case 1: Obvious TBI
Reason
for Referral:
TBI in head-on boat
accident. Propeller hit pt in
right parietal-occipital lobe
(LOC = 7 days; GCS = 3).
Eval. to determine capacity
for medical & financial
decisions, parenting skills,
occupational prognosis, &
disability status. Significant
emotional, behavioral,
occupational, and social
problems pre-TBI.
28








January 13, 2007
Age: 37
Handed: Left
Race: Euro-American
Sex: Female
Ed: 14 years
Occup: Nursing
Marital: Sep. 10 yrs
Living: Camper in
parent’s backyard
Rohling - CCPN Orlando, FL
RIM Sample Case 1: Obvious TBI
100
90
80
70
60
50
40
30
20
29
January 13, 2007
SP
LA
PM
PS
AW
VML
AML
EF
PO
VC
ITBM
DTBM
OTBM
EPGA
EP
0
SV
10
Rohling - CCPN Orlando, FL
RIM Sample Case 1: Obvious TBI
RIM Summary Statistics: T-scores, variances, ES, & power.
Column #
1
2
3
4
5
6
7
8
9
10
11
12
Cognitive Domain
M
sd
n
Hetero.
p value
Classify
% TI
ES
CI
PreM
Nec.
1-sample
t test
anova
s&w
Power
(1-b)
1 Symptom Validity (SV)
46.8
9.8
8
---
Average
25%
-.32
+ 5.8
52.6
---
---
.2045
2 Emotional Personality (EP)
30.6
17.2
10
.0033
Mld-Md
60%
-1.38
+ 9.0
39.6
.0060
---
---
---
---
---
---
---
---
---
---
---
---
---
---
4 Est. Pre. Gen. Ability (EPGA)
45.2
5.7
8
---
Average
13%
---
+ 3.3
41.9
---
---
---
5 Overall TBM (OTBM)
36.6
12.4
69
.0063
Mild
57%
-.89
+ 2.5
39.1
<.00012
---
---
6 Domain TBM (DTBM)
38.0
6.6
7
---
Mild
57%
-1.17
+ 3.1
41.1
.02721
.0068
---
7 Instrument TBM (ITBM)
33.0
13.5
12
.0855
Mld-Md
67%
-1.18
+ 6.5
39.5
.00961
---
---
8 Verbal Comprehension (VC)
38.6
7.1
6
---
Mild
67%
-1.03
+ 4.8
43.4
---
---
.6989
9 Perceptual Organization (PO)
33.9
10.2
6
---
Mld-Md
50%
-1.37
+ 7.0
40.9
.04321
---
---
10 Executive Functioning (EF)
40.1
12.3
12
---
Blw Avg
50%
-.53
+ 5.9
46.0
---
---
-.5306
11 Auditory Mem. & Learn (AML)
32.0
11.5
20
---
Mld-Md
70%
-1.45
+ 4.2
36.2
<.00011
W
---
12 Visual Mem. & Learn (AML)
29.5
13.4
11
---
Moderate
82%
-1.52
+ 6.7
36.2
.00311
W
---
13 Attention/Work Memory (AW)
45.0
9.0
9
---
Average
22%
-.03
+ 5.0
50.0
---
---
.0591
14 Processing Speed (PS)
46.9
10.6
6
---
Average
17%
.20
+ 7.1
54.0
---
---
---
---
---
---
---
---
---
---
---
---
---
---
---
16 PsychoMotor (PM)
44.7
11.6
4
---
Blw Avg
25%
-.49
+ 7.5
52.2
---
---
.1919
17 Language/Aphasia (LA)
13.0
---
1
---
Severe
100%
---
---
---
---
---
---
18 Sensory Perception (SP)
---
---
---
---
---
---
---
---
---
---
---
---
R
o
w
3 Meta-Cognition (MC)
15 Global/Miscellaneous (GM)
30
January 13, 2007
Rohling - CCPN Orlando, FL
TBI Dose Response Curves
Dikmen ES’s
Meyers’ T Scores
60
0
55
50
Max y = -2.1x + 56.2
Overall Test Battery Mean-T
-.5
-1
-1.5
45
90%ile y = -2.2x + 53.5
75%ile y = -2.3x + 50.4
40
50%ile y = -2.6x + 47.6
35
25%ile y = -3.1x + 44.6
30
10%ile y = -3.7x + 42.2
25
-2
Min y = -3.5x + 36.4
20
15
-2.5
10
< 1 hr
-3
31
hrs < 1
hrs 1-23
day 1-6
day 7-13
day 14-28
day > 28
January 13, 2007
1-23 hrs
1-6 days
7-13 days
14-28 days
> 28 days
Severity of TBI based on LOC
Rohling - CCPN Orlando, FL
Combined Dikmen & Meyers
Estimates: ES, T, & Difference
LOC
n
ES
T
Mean Diff.
(EPGA – OTBM)
32
Trauma Control
G1: < 1hr
121
299
.00
-.11
48.1
46.5
--1.5
G2: 1-24 hr
G3: 1-6 day
G4: 7-13 day
G5: 14-28 day
152
99
56
46
-.22
-.33
-.68
-1.29
44.2
42.7
38.1
29.4
3.8
5.6
10.2
19.1
G6: > 28 day
43
-1.49
27.7
20.6
January 13, 2007
Rohling - CCPN Orlando, FL
Return to Work Study: OTBM’s for
4 Groups of TBI Survivors
Group
n
M
SD
ES
Disabled
17
32.8
6.4
-2.29
Unemployed
96
39.5
6.1
-1.01
Below Previous
32
43.3
4.6
-.36
At Previous
137
45.1
5.2
-.45
33
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Sample Case 1: Obvious TBI
Normal Distribution of T Scores
25
20
Percent
15
10
5
0
0
34
10
20
30
40
50
OTBM
January 13, 2007
60
70
80
90
100
Rohling - CCPN Orlando, FL
RIM Sample Case 2: Subtle Diabetes
Reason
for Referral:
2 yrs dangerous work habits.
Eval to see if atrial fib & Type
II diabetes impairs cognition.
Hospitalized “TIA-like” Sx.
Admitted to problems for 20
yrs, cardiac dysrhythmia &
bradycardia, pacemaker,
blood sugar difficult to
manage, & family Hx of heart
disease & diabetes.
35








Age: 55
Handed: Right
Race: Euro-American
Sex: Male
Ed: 13 years
Occup: Mechanic
Marital: Married 20 yr
Living: at home w/wife
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Sample Case 2: Subtle Diabetes
100
90
80
70
60
50
40
30
20
36
January 13, 2007
SP
LA
PM
PS
AW
VML
AML
EF
PO
VC
ITBM
DTBM
OTBM
EPGA
EP
0
SV
10
Rohling - CCPN Orlando, FL
RIM Sample Case 2: Subtle Diabetes
RIM Summary Statistics: T-scores, variances, ES, & power.
Column #
1
2
3
4
7
8
9
10
11
12
Cognitive Domain
M
sd
n
Hetero.
p value
Classify % TI
ES
CI
PreM
Nec.
1-sample
t test
anova
s&w
Power
(1-b)
1 Symptom Validity (SV)
45.3
2.2
7
---
Average
0%
-.65
+ 1.3
46.6
---
.0014
---
2 Emotional Personality (EP)
37.4
14.7
8
.0814
Mild
50%
-1.00
+ 8.5
45.9
---
.0458
---
---
---
---
---
---
---
---
---
---
---
---
---
4 Est. Pre. Gen. Ability (EPGA)
52.1
3.1
8
---
Average
0%
---
+ 1.7
50.4
---
---
---
5 Overall TBM (OTBM)
44.6
9.7
57
---
Blw Avg
25%
-1.04
+ 2.1
46.7
---
<.0001
---
6 Domain TBM (DTBM)
44.4
4.9
7
---
Blw Avg
14%
-1.88
+ 3.0
47.4
.1978
.0058
---
7 Instrument TBM (ITBM)
41.2
9.5
12
---
Blw Avg
58%
-1.54
+ 5.6
46.8
---
.0023
---
8 Verbal Comprehension (VC)
49.6
9.0
5
---
Average
20%
-.37
+ 6.6
55.2
---
---
.1710
9 Perceptual Organization (PO)
45.7
9.3
6
---
Average
17%
-.92
+ 6.2
51.9
---
---
.6155
10 Executive Functioning (EF)
40.8
6.8
11
---
Blw Avg
36%
-2.14
+ 3.2
44.0
---
.0003
---
11 Auditory Mem. & Learn (AML)
48.6
14.1
15
.0176
Average
20%
-.34
+ 6.0
54.6
---
---
.3476
12 Visual Mem. & Learn (AML)
47.4
5.2
8
---
Average
0%
-1.10
+ 2.9
50.3
---
.0377
---
13 Attention/Work Memory (AW)
42.8
7.3
9
---
Blw Avg
33%
-1.66
+ 3.9
46.7
---
.0051
---
14 Processing Speed (PS)
36.0
4.6
4
---
Mild
50%
-4.10
+ 3.7
39.7
W
.0061
---
---
---
---
---
---
---
---
---
---
---
---
---
16 PsychoMotor (PM)
19.0
2.8
2
---
Severe
100%
-4.22
+ 3.3
22.3
W
.0410
---
17 Language/Aphasia (LA)
36.8
4.5
2
---
Mild
50%
-1.70
+ 5.3
42.1
---
---
.2935
18 Sensory Perception (SP)
45.5
1.0
2
---
Average
0%
-.63
+ 1.1
46.6
---
---
.1237
R
o
w
3 Meta-Cognition (MC)
15 Global/Miscellaneous (GM)
37
5
January 13, 2007
6
Rohling - CCPN Orlando, FL
RIM Sample Case 2: Subtle Diabetes
Normal Distribution of T Scores
25
20
Percent
15
10
5
0
0
38
10
20
30
40
50
OTBM
January 13, 2007
60
70
80
90
100
Rohling - CCPN Orlando, FL
RIM Critiques: Concern 1

The method of calculating the standard
deviations (SDs) for summary statistics
and domain scores is incorrect.


39
Since many of the remaining steps of
the RIM depend on the use of these SDs,
this error is magnified in the subsequent
steps.
SDs statistically can not exceed 9.99 and
are more likely to be around 6.4
January 13, 2007
Rohling - CCPN Orlando, FL
30
30
25
25
20
20
Percent
P ercent
Response 1: RIM Ms 4 Datasets
15
15
10
10
5
5
0
25
30
35
40 45 50
55 60
Mean
M W1A(WAIS-R)
IS -R
Mean OTBM
for OTB
Dataset
65
0
70
20
30
30
25
25
20
20
15
10
5
5
0
40
25
30
35
40
45
50
55
60
M
e
a
n
O
TB
M
G
re
e
n
Mean OTBM for Dataset 2 (Green)
30
35
40
45
50
55
60
65
70
15
10
20
25
Mean
OTBM Meyers
Mean OTBM
for Dataset
3 (Meyers)
Percent
P e rc e nt
20
65
70
January 13, 2007
0
20
25
30
35
40
45
50
55
60
65
70
Mean
OTBM4HRB
Mean OTBM for
Dataset
(HRB)
Rohling - CCPN Orlando, FL
Inter-Individual Ms & SDs
N
Mn
SD
WAIS-R
457
43.2
7.2
WAIS
150
45.0
9.1
904
44.8
7.3
1,734
42.0
7.3
Dataset 4 (HRB)
114
42.8
6.8
Total 4 Samples
3,359
43.1
7.4
Dataset 1 Psych Pts
Dataset 2 (Green)
Dataset 3 (Meyers)
41
January 13, 2007
Rohling - CCPN Orlando, FL
25
25
20
20
15
Percent
Percent
Response 1: RIM SDs 4 Datasets
10
5
0
2
4
6
OTBM-sd for
8
10
12
14
16
WAIS-R OTBM-SD
Dataset
1 (WAIS-R)
18
0
20
0
2
4
6
8
10
12
14
16
18
20
OTBM-sd for Meyers
DatasetOTBM-SD
3 (Meyers)
25
25
20
Percent
20
Percent
10
5
0
15
10
15
10
5
5
0
0
2
4
6
OTBM-sd
42
15
8
10
12
14
16
Green's
OTBM-SD
for Dataset
2 (Green)
18
20
0
0
January 13, 2007
2
4
6
8
10
12
14
HRB
OTBM-SD
OTBM-sd for Dataset 4 (HRB)
16
18
20
Rohling - CCPN Orlando, FL
Intra-Individual Ms & SDs
N
Mn
SD
% > 9.99
457
150
904
1,734
114
3,359
6.8
7.4
11.4
11.9
10.6
10.8
2.0
2.2
2.9
2.9
2.4
2.8
7%
10%
65%
56%
61%
50%
Dataset 1 Psych Pts
WAIS-R
WAIS
Dataset 2 (Green)
Dataset 3 (Meyers)
Dataset 4 (HRB)
Total 4 Samples
43
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Critiques: Concern 2


More false-positives then clinical judgment.
Palmer et al. (2004) expressed concern that


44
We failed to distinguish “statistical” from
“clinical” significance.
This failure is a critical error that precludes
the prudent clinician from using the RIM.
January 13, 2007
Rohling - CCPN Orlando, FL
Response 2: RIM vs. Manual
Detecting Differences – Overall %
% of Total S’s
RIM t TEST
MANUAL METHOD
VIQ-PIQ: NS VIQ-PIQ: Sig.
Marginal M’s
VIQ-PIQ: NS
54%
23%
78%
VIQ-PIQ: Sig.
1%
21%
22%
Marginal M’s
55%
45%
100%
45
January 13, 2007
Rohling - CCPN Orlando, FL
Response 2: RIM vs. Manual
Detecting Differences – ESs
Means (SDs)
RIM t TEST
VIQ-PIQ: NS
46
MANUAL METHOD
VIQ-PIQ: NS VIQ-PIQ: Sig.
.38 (.30)
.80
VIQ-PIQ: Sig.
1.58 (.82) 1.70
Marginal M’s
.40 (.37) 1.22
January 13, 2007
(.41)
Marginal
M’s
.50 (.39)
(.86) 1.69 (.85)
(.80)
.90 (.71)
Rohling - CCPN Orlando, FL
Response 2: RIM vs. Manual
Detecting Differences Scores
Means (SDs)
47
MANUAL METHOD
Marginal M’s
RIM t TEST
VIQ-PIQ: NS VIQ-PIQ: Sig.
VIQ-PIQ: NS
3.9
(2.5)
13.2
(3.7)
6.7 (5.2)
VIQ-PIQ: Sig.
6.7
(0.8)
19.0
(6.5)
16.9 (8.5)
Marginal M’s
4.0
(2.5)
15.9
(6.0)
9.3 (7.4)
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Critiques: Concern 3

Clinicians who use the RIM will:


48
Idiosyncratically assign scores to cognitive
domains.
This will result in low inter-rater reliability in
analysis & diagnosis.
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Critiques: Concern 4


Scores on domains are unit weighted,
which introduces error.
Willson & Reynolds (2004) said scores
load on multiple domains. Assignment to
domains & weights depend on:


49
Battery of tests administered.
Patients whose test scores are being
examined.
January 13, 2007
Rohling - CCPN Orlando, FL
Response 4: Cross-Valid. Unit Wts

Conducted 4 multiple reg. on 457 pts’ WAIS-R.


Regressed patients’ verbal subtests onto PIQ.


Use Pre-PIQ’s regress on actual PIQs 1st ½ sample.
Repeated, except performance subtests predict VIQ

50
Pre-PIQs regressed on actual PIQs 2nd ½ sample.
Also, generated weights for the 2nd ½ of sample.


Generated ideal weights for the 1st ½ of sample.
Used wts to predict PIQs in the 2nd ½ of sample.


Split sample in ½ - assess shrinkage.
split sample ½ & generate same statistics as before.
January 13, 2007
Rohling - CCPN Orlando, FL
Response 4: Cross-Valid. Unit Wts

Purpose of these procedures:




How much variance in wts. is sample specific.
Amount of shrinkage using cross-validated wts.
Shrinkage error compared to error introduced by
using “unit wts” vs. “ideal wts.”
Results: 98% of the variance accounted for with
unit wts. Compared to ideal weights.

Support use of unit wts. Rather than ideal wts.

51
See, Dawes, R. M. (1979).
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Critiques: Concern 5

Multiple measures used to generate
composite scores:

52
Results in less accurate estimates of
the cognitive domains.
January 13, 2007
Rohling - CCPN Orlando, FL
Response 5: Estimate FSIQ Using
Scaled Score Means’s
53
4
3
2
January 13, 2007
Diff 11
Diff 10
Diff 9
Diff 8
Diff 7
Diff 6
0
Diff 5
1
Diff 4
(.47)
(.49)
(.45)
(.36)
(.25)
(.26)
(.21)
(.18)
(.14)
(.12)
(.06)
Diff 3
4.99
4.42
3.58
3.01
2.42
2.53
2.11
2.06
1.31
1.36
1.27
Diff 2
--51%
70%
70%
81%
79%
81%
88%
93%
91%
93%
5
Diff 1
1 predictor (Voc)
2 predictors (1 +OA)
3 predictors (2 + BD)
4 predictors (3 + Sim)
5 predictors (4 + DSp)
6 predictors (5 + Info)
7 predictors (6 + PA)
8 predictors (7 + Comp)
9 predictors (8 + DSy)
10 predictors (9 + Arith)
11 predictors (10 + PC)
6
M
%
Absolute (SEM of
Improved Value of Improved)
Diff
Cell Mean
Diff. between 1 predictor
and additional predictors
Rohling - CCPN Orlando, FL
RIM Critiques: Concern 6


A general ability factor is used to
represent premorbid functioning for all
domains.
This not supported by the literature.

54
This results in inaccurate conclusions
regarding degree of impairment suffered by
a patient in each cognitive domains
assessed.
January 13, 2007
Rohling - CCPN Orlando, FL
Domain Means Correlations
All were Significant ( p < .001 )
1
1- Premorbid
2
3
4
5
6
7
8
9
10
.76 .71 .62 .56 .79 .68 .53 .54 .30 .28
2- OTBM
.76
3- DTBM
.71 .98
4- Attention/Working Mem
.64 .81 .77
5- Proc. Spd/Mental Flex
.62 .82 .79 .64
6- Verbal Reasoning
.79 .84 .78 .69 .72
7- Visual Reasoning
.68 .81 .81 .54 .64 .64
8- Verbal Memory
.53 .77 .78 .68 .50 .54 .51
9- Visual Memory
.54 .77 .80 .53 .55 .55 .70 .62
10- Dom Motor/Sensory
.30 .54 .62 .37 .44 .36 .41 .34 .37
11- Non-Dom Motor/Sens
.28 .53 .62 .31 .44 .30 .45 .32 .40 .53
55
11
.98 .81 .82 .84 .81 .77 .77 .54 .53
.77 .79 .78 .81 .78 .80 .62 .62
January 13, 2007
.64 .69 .54 .68 .53 .37 .31
.72 .64 .50 .55 .44 .44
.64 .54 .55 .36 .30
.51 .70 .41 .45
.62 .34 .32
.37 .40
.53
Rohling - CCPN Orlando, FL
RIM Critiques: Concern 7


56
Norms used come from samples that are
of undocumented comparability.
Furthermore, even when norms used
were generated from different but
comparable samples, their format
prohibits ready comparisons.
January 13, 2007
Rohling - CCPN Orlando, FL
Response 7: Split-Half Reliability


Analyze Dataset 2: OTBM’s from 42 DV’s
Individuals’ data split into two sets



57
21 test variables for each OTBM (1 & 2)
2 independent OTBMs created for pt.
Split DV’s intentionally-separated so no
normative sample included both OTBM’s
January 13, 2007
Rohling - CCPN Orlando, FL
Response 7: Split-Half Reliability

Results r = .81, 66% of variance accounted





Mean OTBM-1 = 45.0 (sd = 7.3)
Mean OTBM-2 = 43.6 (sd = 7.2)
Results simulate worse case scenario.


58
Slope of the regression line was .82 (SE = .027)
Intercept 9.2 (SE = 1.20).
used an entirely different set of norms.
Est. test-retest r for OTBM 42 DV’s increased
r from .81 to .88 using the Spearman-Brown
correction).
January 13, 2007
Rohling - CCPN Orlando, FL
Response 7: Split-Half Reliability

No overlap in normative samples.

Worst-case condition, generally administer
instruments (e.g., WAIS-III) with OTBMs generated
from “co-normed” variables.




When different norms used, often gave same
instruments (e.g., AVLT or RCFT)
No instrument used OTBM-1 included OTBM-2
Heaton et al.’s (2001) - schizophrenic pts.


59
Meyers & Rohling test-retest reliability of .86.
Obtained a test-retest reliability of .97.
Comparing 2 identical batteries, not worst-case.
January 13, 2007
Rohling - CCPN Orlando, FL
RIM Critiques: Concern 8

The RIM will result in an undue inflation of
clinicians’ confidence.

60
Such overconfidence results in more error
in a interpretation, not less.
January 13, 2007
Rohling - CCPN Orlando, FL
RIM vs. Tulsky et al. (2003): Case 1
WAIS & WMS Battery
Disc.
Model
RIM
Model
#
RIM Model
#
S EPGA1 (WTAR)
103
102
1
106
5
U Overall TBM2 (FSIQ)
(112)
104
18
96 I
70
M Domain TBM2
---
105
6
98
7
M Inst TBM2 (FSIQ/GMI)
---
104
2
93 I
14
1 Verbal Comp (VCI)
120 *
118 *
3
110
6
2 Percept Organ (POI)
121 *
117 *
3
107
5
3 Attent/Work Mem (WMI)
105
105
2
95 I*
9
Index or Domain Scores
61
Full Battery
4 Process Speed (PSI)
93 **
95 **
2
86 I*
5
5 Aud Mem & Learn (AMI)
97 **
97 **
4
81I*****
18
6 Vis Mem & Learn (VMI)
94
95
4
98
13
7 Executive Function (EF)
---
---
0
100 I*
15
8 Psycho-Motor (PM)
---
---
0
111
6
January 13, 2007
Rohling - CCPN Orlando, FL
RIM vs. Tulsky et al. (2003): Case 2
WAIS & WMS Battery
Index or Domain Scores
RIM Model
#
RIM Model
#
117
5
S EPGA1 (WTAR)
125
120
1
U Overall TBM2 (FSIQ)
(119)
103
18
96 I
53
M Domain TBM2
---
105
6
95 I
7
M Inst TBM2 (FSIQ/GMI)
---
106
2
93 I
11
124
122
3
117
5
95
97
3
97
3
108
108
2
102
7
98
92
2
83
5
5 Aud Mem & Learn (AMI)
111
110
4
99 I
9
6 Vis Mem & Learn (VMI)
104
103
4
89 I
10
7 Executive Function (EF)
---
---
0
94 I
14
8 Psycho-Motor (PM)
---
---
0
77
6
1 Verbal Comp (VCI)
2 Percept Organ (POI)
3 Attent/Work Mem (WMI)
4 Process Speed (PSI)
62
Disc. Model
Full Battery
January 13, 2007
Rohling - CCPN Orlando, FL
Summary of the Rohling
Interpretive Method of Statistical
Analysis of Individual
Neuropsychological Test Data
63
January 13, 2007
Rohling - CCPN Orlando, FL
Summary of RIM Steps
24 total steps to the process

17 calculation steps




7 interpretative steps

64
Battery Design
Calculation of summary statistics
Generation of graphic displays
Use of summary table and graphic displays to:

Assess and verify summary data

Identify strengths/limitations of current data

Obtain a reliable diagnosis

Develop tx plans based on clinical judgments.
January 13, 2007
Rohling - CCPN Orlando, FL
Summary of RIM Advantages

Formulize thinking interpretation of data:

65
Operationalize what you already do.

Reduce judgment errors thru RIM Process.

Take advantage of psychometric properties
at level with fixed, co-normed batteries.

Allows flexibility of test selection.

Allows flexibility of theoretical view of
cognition (e.g., domain structure)
January 13, 2007
Rohling - CCPN Orlando, FL
Summary of RIM Advantages cont’d


66
Gives Quantitative support for your
conclusions and interpretations

Statistical evaluation

Measure of confidence in findings

Measure of limitations of findings

Ability to present data at different levels of
interpretation
Equals greater defensibility
January 13, 2007
Rohling - CCPN Orlando, FL
Our RIM Cautions/Concerns

Does not “replace” clinical judgment, rather,
informs clinical judgment.

This still means CJ errors are possible.

Susceptibility T-Scores to distrib. deviance

Process, not program
67

Pre-morbid ability estimates

Domain selection, test placement
January 13, 2007
Rohling - CCPN Orlando, FL
RIM is Not Alone Out There!

Dawn Flanagan, Ph.D., at St. Johns University in
New York independently developed a similar
method


Second edition of Essentials of Cross-Battery
Assessment (Flanagan, Ortiz, & Alfonso, in press)
is due out in March, which explains her method,
along with co-authors

68
The Cattell-Horn-Carroll (CHC) Cross Battery
Approach.
Some of her work can also be found on the website by
Dumont-Willis.
January 13, 2007
Rohling - CCPN Orlando, FL
Published Research Findings
Using the RIM
1)
2)
3)
4)
RIM vs. HRB
Variance Accounted for by SVT
Effect of Depression on NP Results
Prediction of Employment after TBI
RIM of HRB: OTBM vs. HII

Heaton et al.’s (1991)
HRB norms for OTBM
T Score (M=50, sd=10)
OTBM r with HII = -.79
 (p < .0001)
 62% variance account.
 Over predicts low
 Under predicts high
1.0
0.9
0.8

0.6
HII

0.7
0.5
0.4
0.3
0.2
0.1
0
20
25
30
35
40
45
50
55
60
OTBM
70
January 13, 2007
Rohling - CCPN Orlando, FL
RIM of HRB: OTBM vs. GNDS
 OTBM

71
70
60
50
GNDS
r with GNDS = -.87
 76% variance acc.
 OTBM neither under
nor over predicts
across range of GNDS
 Intercept impairment
is T Score = 46.0
80
40
30
20
10
Reitan & Wolfson (1993)
(GNDS = 29)
January 13, 2007
0
20
25
30
35
40
45
50
55
60
OTBM
Rohling - CCPN Orlando, FL
RIM of HRB: OTBM’s
Relationship to Global Indices
INDICES OF FUNCTION
72
Correlation Coefficient
Halstead Impairment Index
.79
Average Impairment Rating
.90
Global Neuro. Deficit Scale
.87
RIM: Domain TBM
.99
RIM: Instrument TBM
.95
RIM: % Tests Impaired
.96
January 13, 2007
Rohling - CCPN Orlando, FL
RIM of HRB: Diagnostic
Classification Using the HII
73
BR 65%
Sens.
Spec.
PPV
NPV
% Corr.
HII
.64
.66
.77
.51
65%
AIR
.58
.78
.82
.51
65%
GNDS
.78
.63
.79
.62
73%
OTBM
.90
.32
.70
.65
69%
ITBM
.86
.37
.71
.60
69%
%TI
.85
.56
.78
.68
74%
January 13, 2007
Rohling - CCPN Orlando, FL
RIM of HRB: Cross-Validation of
RIM using HRB in 2 Samples

Regressed Dikmen & Meyers TBI data



74
Generated a predicted HII for pts in OK dataset.
Correlation actual & predicted HII = .95
 Sen = .60, Spec = .77, PPV = .78, NPV = .59
 Overall % Correct Classification = 71%
Predicted HII from MNB’s OTBM & got a
more accurate indicator of impairment than
actual HII
January 13, 2007
Rohling - CCPN Orlando, FL
Factor Loadings of Domain Scores
Genuine Normal
75
Genuine Neuro
Exag Normal
Obj
Perf
SelfReport
Obj
Perf
SelfReport
Obj
Perf
SelfReport
NPT
.57
-.03
.64
-.01
.89
.11
SVT
.58
-.08
.63
-.02
.87
.12
MCI
-.03
.56
.04
.55
.33
.81
PSX
-.07
.58
-.06
.55
.03
.91
Eigen
1.30
1.83
1.22
1.67
2.02
1.14
% Var.
33%
46%
30%
42%
51%
29%
January 13, 2007
Rohling - CCPN Orlando, FL
M & SDs of Composite Z scores
Genuine
Exaggerate
Normal
Neurologic
Normal
Neurologic
Neuropsych
Test Scores
.33
.19
-.60
-.79
(.62)
(.64)
(.80)
(.65)
Symptom
Validity
.51
.50
-1.25
-.50
(.38)
(.30)
(.94)
(.52)
Memory
Complaints
.14
.41
-.62
.49
(.93)
(.92)
(.93)
(.54)
Psychiatric
Symptoms
.10
.46
-.39
.19
(.95)
(.96)
(.85)
(.99)
76
January 13, 2007
Rohling - CCPN Orlando, FL
Mean Z score on Objective Tests
 Small diff. between
Gen. Normal & Gen.
Neuro. on NPT
.5
.3
.1
Cell Mean
-.1
Gen-Normal
Gen-Neuro
Exag-Normal
Exag-Neuro
-.3
-.5
-.7
-.9
-1.1
-1.3
-1.5
77
Neuropsych Ability
Symptom Validity
 No diff. between Exag.
Normal & Exag. Neuro
on NPT
 Deficits for Exag. Neuro
were more modest than
for Exag. Normals on
SVT
 Interaction between
Validity & Neuro Status.
January 13, 2007
Rohling - CCPN Orlando, FL
Mean Z score Self-Report
 No diff. between Gen.
Neuro. & Exag. Neuro on
Memory Complaints
.5
.3
.1
Cell Mean
-.1
Gen-Normal
Gen-Neuro
Exag-Normal
Exag-Neuro
-.3
-.5
-.7
-.9
-1.1
-1.3
-1.5
78
Memory Complaints
Psychiatric Symptoms
 No diff. between Gen. &
Exag. Neuro on Psych.
symptoms
 Deficits for Exag. Normal
on Psych. symptoms &
Memory Complaints, the
latter is larger
 Interaction between
Validity & Neuro Status.
January 13, 2007
Rohling - CCPN Orlando, FL
Depression Study: Reference
Rohling, M. L., Green, P., Allen, L. M., & Iverson,
G. L. (2002). Depressive symptoms and
neurocognitive test scores in patients passing
symptom validity tests. Archives of Clinical
Neuropsychology, 17, 205-222.
79
January 13, 2007
Rohling - CCPN Orlando, FL
Mood Group Assignment


Patients classified into 2 subgroups
From entire sample, 420 passed all SVTs


Sample split based on BDI
Low-Depressed 25%ile on BDI (< 10)


High-Depressed 75%ile on BDI (> 25)

80
n = 178, M = 6 (3)
n = 187, M = 31 (6)
January 13, 2007
Rohling - CCPN Orlando, FL
Depression Study Participants


All 365 patients referred for evaluation for
compensation-related purposes
All diagnostic groups included





81
53% Head injury referrals
22% Medical referrals
14% Psychiatric referrals
11% Other neurological
Age = 42 (11); Ed = 13 (3); Sex = 64% males;
Non-English = 18%; Handedness = 9% Left
January 13, 2007
Rohling - CCPN Orlando, FL
Results Mood & Validity Status
SVT Status
Mood BDI
Genuine
Exaggerating
Depress 75%ile
107
(30%)
68
(19%)
175
(48%)
NonDep 25%ile
159
(44%)
27
(7%)
186
(52%)
266
(74%)
82
January 13, 2007
95
(26%)
Rohling - CCPN Orlando, FL
Z-scores
Results: Sample Split by Validity
83
1.3
1.1
.9
.7
.5
.3
.1
-.1
-.3
-.5
-.7
-.9
-1.1
-1.3
-1.5
EPT MCI OTBM
EPT MCI OTBM
Low-Dep
Low-Dep
High-Dep
High-Dep
Total Sample
Gen Pts
January 13, 2007
EPT
MCI
OTBM
1.3
1.1
Low-Dep
.9
.7
.5
.3
.1
-.1
-.3
-.5
-.7
-.9
-1.1
High-Dep -1.3
-1.5
Exag Pts
Rohling - CCPN Orlando, FL
Effect of Mood Depends on Effort

Exaggerating patients accounted for



Mood & Effort used as IVs and Cognition DV


Effects for both effort and mood
Also, when other Emotion/Personality DV

84
Effect for effort, no effect for mood
However, when Memory Complaints DV


39% of High-Dep group
14% of Low-Dep group
Effects for both effort and mood
January 13, 2007
Rohling - CCPN Orlando, FL
Effect of Mood Depends on Effort
When both mood groups were included in
regression analysis, as predicted:
 Memory ratings related to mood

(r = .60; p < .0001)
 Mood

not correlated with cognition
(r = .10; p > .10)
 Memory

85
ratings not related to cognition
(r = .13, p = .06)
January 13, 2007
Rohling - CCPN Orlando, FL
Mood Replication
Gervais’ pain sample findings (n = 177)

Exaggerating patients accounted for





Memory ratings related to mood (r = .55)
Mood not correlated with cognition (r = .06)
Memory ratings related to cognition (r = .15)
Group means correlated with Green’s .94

86
55% of High-Dep; 33% of Low-Dep group
all patient (High-D, Low-D, Gen, & Exag).
January 13, 2007
Rohling - CCPN Orlando, FL
Effect if Pain on OTBM
SVT Results
87
WMT
CARB
TOMM
Pass
Pass
Pass
Test Score
N
Mean
OTBM
Mean
Pain
Pass
340
0.24
-0.30
Fail
Pass
14
-0.06
-0.09
Fail
Pass
Pass
74
-0.54
0.14
Fail
Fail
Pass
35
-0.74
0.57
Fail
Pass
Fail
17
-0.96
0.39
Fail
Fail
Fail
38
-1.39
0.67
January 13, 2007
Rohling - CCPN Orlando, FL
Effect if Pain on OTBM
Pain Rating Status
Low Rating
(n = 41)
Objective Performance
Mean SVT
OTBM
Demographic Comparisons
Pain Rating (0 – 5)
Age in Years
Years of Education
WAIS-R/III or MAB VIQ*
WRAT-III Reading Subtest
English 1st Language (% total)
Male (% total)
88
High Rating
(n = 80)
M
(SD)
M
(SD)
g (ES)
.43
.38
(.08)
(.54)
.40
.14
(.12)
(.68)
-.03
-.38
.1103
.0596
.59
36.8
11.9
101.3
44.4
95.1
63.4
(.50)
(10.0)
(2.9)
(12.3)
(5.8)
(---)
(---)
4.31
40.0
11.4
93.0
41.8
93.6
62.5
(.47)
(10.2)
(2.5)
(13.0)
(6.0)
(---)
(---)
-7.75
.32
.19
.65
.44
-----
<.0001
.0317
.3466
.0029
.1537
.7320
.9215
January 13, 2007
p
Rohling - CCPN Orlando, FL
Return to Work after Injury

Three main hypotheses using MNB-RIM



OTBM will predict return to work level
Cognitive domain that will be most
predictive will be executive function
Adding the Patient Competency Rating
Scale will improve work prediction

89
PCRS is by Prigatano (1985)
January 13, 2007
Rohling - CCPN Orlando, FL
Return to Work: ANOVA of OTBM
Group
n
M
SD
ES
Disabled
17
32.8
6.4
-2.29
Unemployed
96
39.5
6.1
-1.01
Below Previous
32
43.3
4.6
-.36
137
45.1
5.2
-.45
At Previous
90
January 13, 2007
Rohling - CCPN Orlando, FL
Logistic Regression Using OTBM
Predicted
Disable
Unemploy
Below
Prev
At
Prev
% Corr
Disabled
2
12
0
3
12%
Unemployed
1
48
0
47
50%
Below Previous
0
9
0
23
0%
At Previous
0
25
0
112
82%
Observed
91
January 13, 2007
Rohling - CCPN Orlando, FL
Return to Work: Summary




OTBM differences between groups
Disabled /Unemployed not able to
separate
Below/At Previous not able to separate
Collapsed groups result in 71% correct

92
above base rate of 52% correct
January 13, 2007
Rohling - CCPN Orlando, FL
Return to Work: Domain Analysis

Executive function not the most predictive


Using Cognitive Domains


OTBM increases % Correct from 71% to 74%
Incremental validity of PCRS very low.

93
Most of variance carried by Perceptual
Organization & Working Memory
7% of the variance
January 13, 2007
Rohling - CCPN Orlando, FL
Return to Work: Domain Analysis

By including premorbid variables,
increases diagnostic accuracy; most
helpful being:


Including acute measures increases
accuracy; most helpful being:


94
Premorbid IQ, level of occupation, & education
LOC group
Time since injury
January 13, 2007
Rohling - CCPN Orlando, FL
Depression Study Conclusions

Memory complaints not synonymous with
impairment in compensation sample


Effort accounts for more variance in selfratings of cognition & objective
performance than mood

95
Findings replicated
Findings replicated
January 13, 2007
Rohling - CCPN Orlando, FL
What’s wrong with this patient-1? (Key: RCPS)
100
90
80
70
Cell Mean
60
50
40
30
20
96
January 13, 2007
PM
PS
AW
VML
AML
EF
PO
VC
ITBM
DTBM
OTBM
EPGA
MC
EP
0
SV
10
Rohling - CCPN Orlando, FL
What’s wrong with this patient-1? (Key: RCPS)
RIM Summary Statistics: T-scores, variances, ES, & power.
Column #
R
o
w
Cognitive Domain
1 Symptom Validity (SV)
2
3
4
Classify
% TI
Mld-Mod
50%
M
sd
n
Hetero.
p value
31.7
23.1
10
<.0001
5
6
7
8
9
10
11
12
ES
CI
PreM
Nec.
1-sample
t test
ANOVA
s&w
Power
(1-b)
-.90
+ 12.0
43.7
.03381
---
---
1
2 Emotional Personality (EP)
40.9
12.1
11
---
Low Avg
55%
-.64
+ 5.9
46.8
.0319
---
---
3 Meta-Cognition (MC)
20.3
12.1
9
---
Mod-Sev
100%
---
+ 6.6
26.9
< .0001
---
---
4 Est. Pre. Gen. Ability (EPGA)
50.5
5.1
7
---
Average
0%
---
+ 3.2
47.3
---
---
---
5 Overall TBM (OTBM)
42.1
10.3
50
---
Low Avg
42%
-.85 + 2.5
44.6
<.00013
.0100
---
3
6 Domain TBM (DTBM)
43.7
6.8
7
---
Low Avg
43%
-1.13 + 4.2
47.9
.0394
---
---
7 Instrument TBM (ITBM)
37.7
12.7
12
---
Mild
50%
-1.20 + 6.1
43.8
.00513
---
---
-2.45 + 3.6
41.7
3
.0050
W
---
.99 + 1.8
56.6
---
8 Verbal Comprehension (VC)
38.1
5.0
5
---
Mild
60%
9 Perceptual Organization (PO)
54.8
2.2
4
---
Average
0%
S
---
3
10 Executive Functioning (EF)
37.3
10.1
12
---
Mild
58%
-1.52 + 4.8
42.1
.0009
W
---
11 Aud. Mem. & Learn (AML)
37.4
6.1
9
---
Mild
67%
-2.30 + 3.3
40.7
.00013
W
---
3
---
12 Visual Mem. & Learn (VML)
45.1
2.6
5
---
Average
0%
-1.26 + 2.0
47.1
.0102
---
13 Attent/Work Mem. (AW)
45.2
12.9
10
---
Average
30%
-.50 + 5.8
51.9
---
---
14 Processing Speed (PS)
48.4
11.8
5
---
Average
40%
-.25 + 8.7
57.1
---
---
---
---
---
---
---
---
---
---
---
---
42.2
8.3
7
---
Low Avg
43%
47.3
.03783
---
---
---
---
---
---
---
---
---
---
---
---
---
---
33.4
47.2
2
50%
-.49
+ 54.9
88.3
---
---
---
15 Global/Miscellaneous (GM)
16 PsychoMotor (PM)
17 Language/Aphasia (LA)
18 Pain Complaints (PC)
97
1
< .0001 Mld-Mod
January 13, 2007
---
---
-.85 + 5.1
Rohling - CCPN Orlando, FL
What’s wrong with this patient-2? (Key: JSVD)
C e ll Po in t C h ar t
Er r o r Bar s : 90% C o n fid e n ce In te r val
100
90
80
Cell Mean
70
60
50
40
30
20
98
January 13, 2007
PS-T
A W-T
ML-T
EF-T
PO-T
V C-T
ITBM-T
DTBM-T
OTBM-T
EPGA -T
EP-T
0
SV -T
10
Rohling - CCPN Orlando, FL
What’s wrong with this patient-2? (Key: JSVD)
Column #
1
2
3
4
7
8
9
10
11
12
Cognitive Domain
M
sd
n
Hetero.
p value
Classify % TI
ES
CI
PreM
Nec.
1-sample
t test
anova
s&w
Power
(1-b)
1 Symptom Validity (SV)
2 Emotional Personality (EP)
49.1
8.2
4
---
Average
-.09
+ 9.6
58.7
---
---
---
46.5
9.5
9
---
Average 33%
-.35
+ 5.9
52.4
---
---
.42
3 Est. Pre. Gen. Ability (EPGA)
62.0
6.0
7
---
0%
1.20
+ 4.5 57.5
.002
---
---
4 Overall TBM (OTBM)
5 Domain TBM (DTBM)
50.9 15.4
57
18%
-1.10 + 3.4
54.3
<.0001
---
---
53.5
7.9
6
---
Average
0%
-.85
+ 6.5
60.0
.05
.02
---
6 Instrument TBM (ITBM)
49.6 11.3
14
---
Average
14%
-1.24 + 5.3
54.9
.001
---
---
7 Verbal Comprehension (VC)
8 Perceptual Organization (PO)
62.3
4.0
6
---
Excel
0%
.03
+ 3.3
65.6
---
S
---
55.2
5.6
6
---
Ab Avg
0%
-.68
+ 4.6
59.8
.05
---
---
9 Executive Functioning (EF)
10 Memory & Learning (ML)
45.5
7.4
14
---
Average
21%
-.65
+ 4.5
49.0
<.0001
W
---
44.7 22.6
15
.01
W
---
11 Attention/Working Mem (AW)
12 Processing Speed (PS)
63.1 17.3
7
.0132
Excel
14%
49.6
7.5
9
---
Average
11%
13 Global/Miscellaneous (GM)
---
---
0
---
---
---
---
14 Language/Aphasia (LA)
15 Sensory Perception (SP)
---
---
0
---
---
---
---
---
0
---
---
---
R
o
w
99
5
Excel
<.0001 Average
6
0%
<.0001 Blw Avg 33% -1.73 + 10.3 55.0
January 13, 2007
.11
+ 12.7
75.8
---
S
---
-1.24 + 4.7
54.3
.001
---
---
---
---
---
---
---
---
---
---
---
---
---
---
---
---
---
---
---
Rohling - CCPN Orlando, FL
What’s wrong with this patient-3? (Key: NPAD)
100
90
80
70
Cell Mean
60
50
40
30
20
10
0
SV
100
EP
EPGA OTBM DTBM ITBM
VC
January 13, 2007
PO
EF
AML
VML
AW
PS
PM
Rohling - CCPN Orlando, FL
What’s wrong with this patient-3? (Key: NPAD)
RIM Summary Statistics: T-scores, variances, ES, & power.
Column #
R
o
w
1
2
3
4
5
6
Classify
% TI
7
8
9
10
11
12
CI
PreM
Nec.
1-sample
t test
ANOV
A
s&w
Power
(1-b)
M
sd
n
Hetero.
p value
1 Symptom Validity (SV)
13.3
18.6
8
.0021
Severe
88%
-2.46 + 10.0
23.3
.0008
---
---
2 Emotional Personality (EP)
20.8
19.5
8
.0008
Moderate
75%
-1.88 + 10.5
31.3
.0038
---
---
---
---
---
---
---
---
---
---
---
---
---
---
4 Est. Pre. Gen. Ability (EPGA)
41.5
3.1
5
---
Blw Avg
60%
---
+ 2.3
39.2
---
.0576
---
5 Overall TBM (OTBM)
40.2
12.3
42
.0364
Blw Avg
55%
-.11
+ 3.1
43.3
---
---
.1727
6 Domain TBM (DTBM)
39.9
6.8
7
---
Mild
57%
-.28
+ 3.3
43.2
---
---
.1623
7 Instrument TBM (ITBM)
38.4
16.4
10
.0079
Mild
60%
-.23
+ 8.6
47.0
---
---
.1657
8 Verbal Comprehension (VC)
35.6
1.9
3
---
Mild
100%
-2.14
+ 1.9
37.5
.0332
---
---
9 Perceptual Organization (PO)
47.8
13.9
3
---
Average
33%
--- + 13.3
61.1
---
---
---
10 Executive Functioning (EF)
46.9
13.4
12
---
Average
33%
---
+ 6.5
53.4
---
S
---
11 Auditory Mem. & Learn (AML)
34.8
10.6
9
---
Mld-Md
78%
-.76
+ 5.8
40.6
---
---
---
12 Visual Mem. & Learn (VML)
39.2
3.2
4
---
Mild
50%
-.73
+ 2.6
41.8
---
---
.3074
13 Attention/Working Mem (AW)
45.0
9.0
5
---
Average
20%
---
+ 6.6
51.6
---
---
---
14 Processing Speed (PS)
30.3
13.0
6
---
Mld-Md
83%
-1.13
+ 8.8
39.1
---
W
---
---
---
---
---
---
---
---
---
---
---
---
14.0
2.8
2
---
Severe
100%
+ 3.4
17.4
.0353
---
---
17 Language/Aphasia (LA)
---
---
---
---
---
---
---
---
---
---
---
---
18 Sensory Perception (SP)
---
---
---
---
---
---
---
---
---
---
---
---
Cognitive Domain
3 Meta-Cognition (MC)
15 Global/Miscellaneous (GM)
16 PsychoMotor (PM)
101
January 13, 2007
ES
---4.90
Rohling - CCPN Orlando, FL
What’s wrong with this patient-4? (Key: SMAA)
Ce ll Point Chart: Error Bars 90% Confide nce Inte rval
100
90
80
70
Cell Mean
60
50
40
30
20
102
January 13, 2007
LA-T
GM-T
PS-T
AW-T
ML-T
EF-T
PO-T
VC-T
ITBM-T
DTBM-T
OTBM-T
EPGA-T
EP-T
0
SV-T
10
Rohling - CCPN Orlando, FL
What’s wrong with this patient-4? (Key: SMAA)
RIM Summary Statistics: T-scores, variances, ES, & power.
1
M
2
sd
3
n
5
Classify
6
% TI
7
ES
8
80%
CI
9
10
11
12
PreM
Nec.
1-sample
t test
anova
s&w
Power
(1-b)
1 Symptom Validity (SV)
2 Emotional Personality (EP)
53.5
0.6
5
---
Abv Avg
0%
.35
.5
54.0
---
---
---
43.3
4.0
10
.0001
Low Avg
%
-.67
6.0
49.3
---
---
---
3
58.6
2.7
6
---
Abv Avg
0%
.86
1.7
56.9
---
---
---
4 Overall TBM (OTBM)
5 Domain TBM (DTBM)
50.4 10.8
86
.0001
Average
14%
-.80
1.8
52.2
.0001
---
---
47.9
5.3
7
---
Average
0%
-.99
3.0
50.9
.0019
---
---
6 Instrument TBM (ITBM)
7 Verbal Comprehension (VC)
46.6
8.9
20
---
Average
25%
-1.11
3.0
49.9
.0009
---
---
51.3
6.9
5
---
Average
0%
-.68
4.7
56.6
.0785
---
---
8 Perceptual Organization (PO)
9
Executive Function (EF)
10 Memory & Learning (ML)
44.6
8.6
8
---
Low Avg
13%
-1.30
4.5
49.1
.0024
---
---
46.8
3.8
10
---
Average
0%
-1.09
1.8
48.6
.0001
---
---
57.8 11.7
29
---
High Avg
7%
-.07
3.3
61.1
---
S
---
11 Attention/Working Mem (AW)
12
Processing Speed (PS)
13 Global/Miscellaneous (GM)
48.2
9.3
16
---
Average
25%
-.96
3.5
51.7
.0004
---
---
45.4
9.4
14
---
Average
36%
-1.22
3.8
49.2
.0002
---
---
41.3
2.3
3
---
Low Avg
0%
-1.60
2.0
43.3
.0059
---
---
14
38.5
---
1
---
Mild
100% -1.86
---
---
---
---
---
---
---
---
---
---
---
---
---
---
---
R
o
w
Column #
Cognitive Domain
Est. Pre. Gen. Ability (EPGA)
Language/Aphasia (LA)
15 Sensory Perception (SP)
103
4
Hetero.
p value
January 13, 2007
---
---
Rohling - CCPN Orlando, FL
Rohling’s Interpretive Method: Use
of Meta-Analytic Procedures for
Single Case Data Analysis
Martin L. Rohling
Questions & Comments Welcome!
104
January 13, 2007
Rohling - CCPN Orlando, FL
CT/MRI Data

Participant Demographic Information


Variable Sample Sizes (N = 124)
Gender
Male
 Female


Ethnicity
Caucasian
 Other

105
82
42
119
5
January 13, 2007
Rohling - CCPN Orlando, FL
CT/MRI





106
Diagnostic Groups Sample Size
MVA/TBI
47
Blow to Head
32
LCVA
24
RCVA
21
January 13, 2007
Rohling - CCPN Orlando, FL
CT/MRI
1.
2.
3.
4.
107
All were Right Handed.
All were followed by Dr. Meyers through
hospitalization and rehabilitation.
None were involved in litigation.
All passed internal validity checks.
January 13, 2007
Rohling - CCPN Orlando, FL
CT/MRI









108
CT/MRI Location
Left Frontal
Left Parietal
Left Temporal
Left Occipital
Right Frontal
Right Parietal
Right Temporal
Right Occipital
January 13, 2007
59
37
34
6
40
42
31
3
Rohling - CCPN Orlando, FL
CT/MRI


All were given MNB
CT/MRI data coded for injury reported on
MRI/CT at the time of injury


109
Present = 1
Absent = 0
January 13, 2007
Rohling - CCPN Orlando, FL
110
January 13, 2007
Rohling - CCPN Orlando, FL
111
January 13, 2007
Rohling - CCPN Orlando, FL
CT/MRI


Independent Sample 1-tailed t-test on
each lobe
On CT/MRI report


112
Present = 1
Absent = 0
January 13, 2007
Rohling - CCPN Orlando, FL
CT/MRI Data
113
January 13, 2007
Rohling - CCPN Orlando, FL
Brain Regions Involved in the
Performance of WAIS-III Arithmetic
114
January 13, 2007
Rohling - CCPN Orlando, FL
Brain Regions Involved in the
Performance of the Boston Naming Test
115
January 13, 2007
Rohling - CCPN Orlando, FL
Brain Regions Involved in the
Performance of the Rey-CFT Copy
116
January 13, 2007
Rohling - CCPN Orlando, FL
Brain Regions Involved in the
Performance of the AVLT Total Score
117
January 13, 2007
Rohling - CCPN Orlando, FL
CT/MRI



118
NP tests generally behaved as expected
A more “Systemic” or “Domain” like
approach better at explaining results
Construct of “Executive Function” not
supported.
January 13, 2007
Rohling - CCPN Orlando, FL
Domains used by the MNB





119
Attention/Working
Memory:
Digit Span
Forced Choice
Animal Naming
Sentence Rep
AVLT 1





Processing
Speed/Mental
Flexibility:
Digit Symbol
Dichotic Both
Trails A
Trails B
January 13, 2007
Rohling - CCPN Orlando, FL
Domains used by the MNB









120
Verbal Reasoning
Similarities
Arithmetic
Information
COWA
Dichotic Left
Dichotic Right
Boston Naming
Token Test
Visual Reasoning
 Picture Completion
 Block Design
 JOL
 Category
 RCFT Copy
January 13, 2007
Rohling - CCPN Orlando, FL
Domains used by the MNB





121
Verbal Memory
AVLT Total
AVLT Immediate
AVLT Delayed
AVLT Recognition




Visual Memory
RCFT Immediate
RCFT Delayed
RCFT Recognition
January 13, 2007
Rohling - CCPN Orlando, FL
Domains used by the MNB

Motor and Sensory




122
Finger Tapping Dominant Hand
Finger Tapping Non-Dominant Hand
Finger Localization Dominant Hand
Finger Localization Non-Dominant Hand
January 13, 2007
Rohling - CCPN Orlando, FL
123
January 13, 2007
Rohling - CCPN Orlando, FL
124
January 13, 2007
Rohling - CCPN Orlando, FL
125
January 13, 2007
Rohling - CCPN Orlando, FL
126
January 13, 2007
Rohling - CCPN Orlando, FL
Commonality of Reduced O2
127
January 13, 2007
Rohling - CCPN Orlando, FL
Domain Consistency

N = 936





128
Passed all validity checks
No missing data
Not involved in litigation
Calculated Domain M’s
Regression used to predict Domain M’s
using all on other Domain M’s
January 13, 2007
Rohling - CCPN Orlando, FL
Domain Means Correlations
1 – Premorbid
2 - OTBM
3 - DTBM
4 - Attent/Work Mem
5 – Pro Spd/Mental Flex
6 - Verbal Reason
7 - Visual Reason
8 - Verbal Memory
9 - Visual Memory
10 - Dom Motor/Sensory
11 - Nond Motor/Sensory

129
1
.76
.76
.71
.64
.62
.79
.68
.53
.54
.30
.28
2
.71
.98
.81
.82
.84
.81
.77
.77
.54
.53
3
.62
.98
.77
.79
.78
.81
.78
.80
.62
.62
4
.56
.81
.77
.64
.69
.54
.68
.53
.37
.31
5
.79
.82
.79
.64
.72
.64
.50
.55
.44
.44
6
.84
.78
.69
.72
.64
.54
.55
.36
.30
All were Significant p < .001
January 13, 2007
Rohling - CCPN Orlando, FL
Domain M’s Correlations (cont.)
7
.68
.81
.81
.54
.64
.64

1 - Premorbid
2 - OTBM
3 - DTBM
4 - Attent/Work Mem
5 - ProcSpd/Ment Flex
6 - Verbal Reasoning
7 - Visual Reasoning
8 - Verbal Memory
9 - Visual Memory
10 - Dom Motor/Sen
11 - Nond Motor/Sen

All were Significant p < .001










130
.51
.70
.41
.45
8
.53
.77
.78
.68
.50
.54
.51
.62
.34
.32
9
.54
.77
.80
.53
.55
.55
.70
.62
.37
.40
January 13, 2007
10
.30
.54
.62
.37
.44
.36
.41
.34
.37
11
.28
.53
.62
.31
.44
.30
.45
.32
.40
.53
.53
Rohling - CCPN Orlando, FL
Domains Regression Equations

Attention & Working Memory
(Verbal Reasoning) * .315
 (Verbal Memory) * .273
 (Processing Speed) * .193
 Constant = 10.972

131
January 13, 2007
Rohling - CCPN Orlando, FL
Domains Regression Equations

Processing Speed/ Mental Flexibility




132
Verbal Reasoning * .401
Visual Reasoning * .284
Attention & Working Memory * .230
Constant = 2.434
January 13, 2007
Rohling - CCPN Orlando, FL
Domains Regression Equations

Verbal Reasoning




133
Processing Speed * .361
Attention & Working Memory * .354
Visual Reasoning * .243
Constant = 2.5
January 13, 2007
Rohling - CCPN Orlando, FL
Domains Regression Equations

Visual Reasoning




134
Visual Memory * .322
Processing Speed/Mental Flexibility * .213
Verbal Reasoning * .208
Constant = 11.813
January 13, 2007
Rohling - CCPN Orlando, FL
Domains Regression Equations

Verbal Memory



135
Attention & Working Memory * .738
Visual Memory * .388
Constant = -7.615
January 13, 2007
Rohling - CCPN Orlando, FL
Domains Regression Equations

Visual Memory




136
Visual Reasoning * .698
Verbal Memory * .311
Processing Speed * .0909
Constant = -5.517
January 13, 2007
Rohling - CCPN Orlando, FL
Regression






137
Attent/Working Memory
Processing Speed
Verbal Reasoning
Visual Reasoning
Verbal Memory
Visual Memory
R
.79
.77
.80
.78
.75
.77
R2
.63
.60
.64
.61
.56
.59
January 13, 2007
Adjusted SE
of the Estimate
4.88
5.31
5.04
4.88
7.96
7.11
Rohling - CCPN Orlando, FL
Review




138
Took a battery of well known tests
Developed Norms
Identified Validity, Reliability, Sensitivity
and Specificity.
Internal Validity Checks and Internal
Consistency
January 13, 2007
Rohling - CCPN Orlando, FL