Psychometric analysis of ADNI data

Psychometric analyses of

ADNI data

Paul K. Crane, MD MPH

Department of Medicine

University of Washington

Disclaimer

• Funding for this conference was made possible, in part by Grant R13 AG030995 from the National Institute on Aging.

• The views expressed do not necessarily reflect official policies of the Department of Health and Human Services; nor does mention by trade names, commercial practices, or organizations imply endorsement by the U.S. Government.

Outline

• ADNI neuropsychological battery

• Latent variable approaches

– SEM and IRT

• ADAS-Cog in ADNI

• Memory in ADNI

• Executive functioning in ADNI

ADNI Neuropsychological Battery

Handout

• There is a handout that summarizes these tests and provides the variable names for the variables in the dataset

The Boston Naming Test presents the participant with 30 black and white line drawings; the participant is asked to identify the object in the drawing. Only the odd numbered drawings from the original 60 item version are administered (30 points max). Each drawing is accompanied by 1) semantic and 2) phonemic cues that are provided to the participant if they do not spontaneously produce the name. The item is scored correct if the participant either spontaneously produces the correct name or if they do so after the semantic clue. (Both scores are in the data set.) Words provided after the phonemic cue are provided are not counted as correct. bntspont "Number of spontaneously given correct responses" bntstim "Number of semantic cues given" bntcstim "Number of correct responses following a semantic cue" bntphon "Number of phonemic cues given" bntcphon "Number of correct responses following a phonemic cue" bnttotal "Total Number Correct (1+3)"

ADNI Neuropsychological Battery

MCI

AD

Alternate Word Lists

There are also two versions of the Rey AVLT that are alternated

NL

MCI

AD

BL

A

A

A

M6

B

B

B

M12

A

A

A

M18

--

B

--

M24

A

A

A

M30

--

--

--

M36

B

B

--

Summary

• Repeated administration of a rich neuropsychological battery at 6 month intervals for 2 (AD) or 3 (NC, MCI) years

• How do we drink from that fire hose?

Strategies for analyzing these data

• Pick a couple of tests and ignore the others

– ADAS-Cog and MMSE

– CDR and CDR-SB

• Modifications of those tests

– ADAS-Tree

– ADAS-Rasch

• Composite scores for specific domains

– Z score

– Something fancier using latent variable approach

Outline



– SEM and IRT


• Memory in ADNI


Latent variable approach

• “Items” not intrinsically interesting, only as indicators of the underlying thing measured by the test

• Many nice properties follow

Parallel development 1: SEM

• “Measurement part” of the model specifies how latent constructs are modeled

• “Structural part” of the model specifies relationships between latent constructs and each other, and between latent constructs and other covariates

http://sites.google.com/site/lvmworkshop/home/downloads-general/2010-downloads

limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl

Bunch of indicators

ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6

“Memory”

Memory

Underlying single factor with many indicators

LM story

Rey word list

ADAS word list

MMSE words ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6

Memory

LM story

Rey word list

ADAS word list


HC volume

Memory

This is what we care about!

ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6

HC volume

Memory*


… and typically don’t care whether memory is modeled this way or with all that secondary structure

LM story

Rey word list

ADAS word list


HC volume

Memory


And sometimes we care about it for

600,000 SNPs, or for voxels – we need to move outside of an SEM package for some of the analyses we want to do

Parallel development 2: IRT

• Models are nested within SEM

• Single factor confirmatory factor analysis model

• Initially worked out with binary indicators

• Extended 1960s to polytomous items

(Samejima)

• It’s only the measurement part

• Attention to the quality of measurement and the quality of scores

Typical SEM example

Indicator 1

Indicator 2

Indicator 3

Indicator 4

Construct

Beck

Zung

CESD

PHQ-9

Depression

Depression

Beck

Zung

CESD

PHQ-9

PHQ-9

Depression

A closer look at PHQ-9

• A 9 item depression scale

• Standard scores totalled up

• Typical SEM model would take that total score and treat it as a continuous indicator by using a linear link (single loading parameter)

PHQ-9 measurement properties

Beck

Zung

CESD

PHQ-9

IRT approach to PHQ-9

Depression Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Item 7

Item 8

Item 9

Depression*

SEM and IRT, then and now

• SEM was initially about total scores as indicators of constructs measured in common across tests

• IRT was initially about item level data that had to satisfy assumptions

• More recently: merging the strengths of the two approaches

– Computational rather than conceptual advances, or maybe computational advances have fueled conceptual advances

Categorical data in SEM

• Runmplus code:

• That little code snippet tells Mplus to treat all of the elements in the local `vlist’ as categorical data

• Mplus default is WLSMV

– Other appropriate ways of handling

• Major reason Mplus is dominant SEM software used at FH

What about IRT?

• Array of tools to address measurement precision

• Explicit focus on measurement properties and measurement precision differentiates it from SEM

Pretend for a moment that a single factor model was appropriate…

• Item response theory (IRT) developed middle of last century

– Lord and Novick / Birnbaum (1968)

• Polytomous extension Samejima 1969

• Lord 1980

• Hambleton et al. 1991

• XCALIBRE, Parscale, Multilog

– All variations on a single factor CFA model

4 items each at 0.5 increments

30

25

20

15

10

5

0

50

45

40

35

-3 -2 -1 0 1 2 3

Comments on that test

• Essentially linear test characteristic curve

– Immaterial whether the standard score or the

IRT score is used in analyses

• No ceiling or floor effect

– People at the extremes of the thing measured by the test will get some right and get some wrong

• Pretty nice test!

2 items each at 0.5 increments

50

45

40

35

30

25

20

15

10

5

0

-3 -2 -1 0 1 2 3

Comments on that test

• Essentially linear test characteristic curve

– Immaterial whether the standard score or the IRT score is used in analyses

• No ceiling or floor effect

– People at the extremes of the thing measured by the test will get some right and get some wrong

• Pretty nice test!

• But that’s what we said about the last one and it had twice as many items!

Why might we want twice as many items?

• Measurement precision / reliability

• CTT: summarized in a single number: alpha

• IRT: conceptualized as a quantity that may vary across the range of the test

• Information

• Mathematical relationship between information and standard error of measurement

• Intuitively makes sense that a test with 2x the items will measure more precisely / more reliably than a test with 1x the items

Test information curves for those two tests

16

14

12

10

8

6

4

2

0

-3 -2 -1 0 1 2 3

Standard errors of measurement for the two tests

0.5

0.4

0.3

0.2

0.1

0

-3 -2 -1 0 1 2 3

Comments about these information and SEM curves

• Information curves look more different than the

SEM curves

– Inverse square root relationship

– TIC 100  SEM 0.10

(1/10)

– TIC 25  SEM 0.20

(1/5)

– TIC 16  SEM 0.25

(1/4)

– TIC 9  SEM 0.33

(1/3)

– TIC 4  SEM 0.50

(1/2)

• Trade off between test length and measurement precision

These were highly selected “tests”

• It would be possible to design such a test if we started with a robust item pool

• Almost certainly not going to happen by accident / history

• What are more realistic tests?

Test characteristic curves for 2 26item dichotomous tests

30

25

20

15

10

5

0

-3 -2 -1 0 1 2 3

Comments on these TCCs

• Same number of items but very different shapes

• Now it may matter whether you use an IRT score or a standard score in analyses

• Both ceiling and floor effects

14

12

10

8

6

4

2

0

-3 -2 -1

TICs

0 1 2 3

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

-3 -2 -1

SEMs

0 1 2 3

Comments on the TICs and SEMs

• Comparing the red test and the blue test: the red test is better for people of moderate ability (more items close to where they are)

– For people right in the middle, measurement precision is just as good as a test twice as long

• Items far away from your ability level don’t help your standard error

• The blue test is better for people at the extremes

(more items close to where they are)

Where do information curves come from?

• Item information curves use the same parameters as the item characteristic curves (difficulty level, b , and strength of association with latent trait or ability, a )

(see next slides)

• Test information is the sum of all of the item information curves

– We can do that because of local independence

I( θ) = D 2 a 2 *P( θ)*Q(θ)

1

0.75

0.5

0.25

0

-3

I( θ) = D 2 a 2 * P( θ) *Q( θ)

-2 -1 0 1 2 3

I( θ) = D 2 a 2 *P( θ)* Q( θ)

1

0.75

0.5

0.25

0

-3 -2 -1 0 1 2 3

I( θ) = D 2 a 2 * P( θ)*Q(θ)

1

0.75

0.5

0.25

0

-3 -2 -1 0 1 2 3

1

0.75

0.5

0.25

0

-3

I( θ) = D 2 a 2 *P( θ)*Q(θ)

-2 -1 0 1 2 3

More thoughts on IICs

• The b parameters shift the location of the curve left and right (where is the bead on the string?)

• The a parameters modify the height of the curve

– But not tons; location location location

• IRT gives us tools to evaluate and manage measurement precision

What about cognitive tests?

Cognitive screening tests

Typical SEM example

Indicator 1

Indicator 2

Indicator 3

Indicator 4

Construct

Transformation of the scale of the indicator only works if it happens to be the inverse of the function from Cecile’s model! (Blom comment)

How about we fix the tools?

• Testlet models in IRT

– Wainer, Bradlow, Wang (2007)

– Li, Bolt, Fu (2006)

• Embed everything in hierarchical Bayesian framework or

SEM framework

– Curtis (2010)

• Modify IRT tools to integrate over nuisance factors for measurement precision

– Curtis and Crane (2011 under review)

• Sample from posterior

– Mislevy (1992)

– Crane, Feldman, et al., (2011 under development)

Outline



– SEM and IRT


• Memory in ADNI


ADAS-Cog in ADNI

• Brief cognitive test

• Commonly used outcome in drug trials

• Described in handout

• Two parts: cognitive part, rater part

– Ignored in several manuscripts I have reviewed

• Eliminating rater items does not reduce respondent burden!

Does it meet IRT assumptions?

6

5

4

3

2

1

0

1 2 3 4 5 6 7 8 9 10 11 12 13

q1

ADAS Cog 1 q5 q6 q7 q8 q2 q3 q4 q9 q10 q11 q12 q14

ADAS 1

More on assumptions: 1 factor CFA

TESTS OF MODEL FIT

Chi-Square Test of Model Fit

Value 273.016*

Degrees of Freedom 39**

P-Value 0.0000

Chi-Square Test of Model Fit for the Baseline Model

Value 4011.878

Degrees of Freedom 29

P-Value 0.0000

CFI/TLI

CFI 0.941

TLI 0.956

Number of Free Parameters 63

RMSEA (Root Mean Square Error Of Approximation)

Estimate 0.086

WRMR (Weighted Root Mean Square Residual)

Value 1.515

More on assumptions

TESTS OF MODEL FIT


Value 273.016*


P-Value 0.0000


Value 4011.878


P-Value 0.0000

CFI/TLI

CFI 0.941

TLI 0.956


RMSEA (Root Mean Square Error Of Approximation)

Estimate 0.086


Value 1.515

0.90 / 0.95

0.90 / 0.95

0.08 / 0.05

BUT the details

• Item 1 is an average across three learning trials

– Already assumes we can just total them up

• What if we treat the data as more granular, as 3 different indicators

Scree

4

3

2

1

0

1

8

7

6

5

2 3 4 5 6 7 8 9 10 11 12 13 14 15

Scree

2

1

0

1

5

4

3

8

7

6

2 3 4 5 6 7 8 9 10 11 12 13 14 15

q1a q1b q1c q5 q6 q7 q8 q2 q3 q4 q9 q10 q11 q12 q14

ADAS Cog 2

ADAS 2

ADAS Cog 2 vs standard scores

-4 -2 0 single

2 4

Typical SEM example

Indicator 1

Indicator 2

Indicator 3

Indicator 4

Construct

Transformation of the scale of the indicator only works if it happens to be the inverse of the function from Cecile’s model! (Blom comment)

Broad range of scores for each standard score

-4 -2 0 single

2 4

Typical SEM example

Indicator 1

Indicator 2

Indicator 3

Indicator 4

Construct

1. Transformation of the scale of the indicator only works if it happens to be the inverse of the function from Cecile’s model! (Blom comment)

2. Transformations will only work at the level of granularity included in the model

Returning to the ADAS story

Item 1 Item 1a, 1b, 1c

TESTS OF MODEL FIT


Value 273.016*


P-Value 0.0000


Value 4011.878


P-Value 0.0000

CFI/TLI

CFI 0.941

TLI 0.956



Estimate 0.086


Value 1.515

TESTS OF MODEL FIT


Value 372.634*


P-Value 0.0000


Value 7559.487


P-Value 0.0000

CFI/TLI

CFI 0.957

TLI 0.975



Estimate 0.091


Value 1.522

• Lot more parameters (20 more)

0.90 / 0.95

0.90 / 0.95

• CFI, TLI a bit better

• RMSEA a bit worse

0.08 / 0.05

Secondary domain structure

• Methods effects

– Same word list multiple times

– Rater items

• Subdomain effects

Word list

ADAS Cog 2 bi-factor q1a q1b q1c q5 q6 q7 q8 q2 q3 q4 q9 q10 q11 q12 q14

ADAS 2

Fit for single vs bi-factor

Chi squared

DF

Single Bifactor

372.634* 277.210*

48** 46**

Baseline Chi squared 7559.487 7559.487

DF 28 28

CFI

TLI

RMSEA

0.957

0.975

0.091

0.969

0.981

0.078

0.90 / 0.95

0.90 / 0.95

0.08 / 0.05

PQ01A

PQ01B

PQ01C

PQ02

PQ03

PQ04

PQ05

PQ06

PQ07

PQ08

PQ09

PQ10

PQ11

PQ12

PQ14

Single

Loadings

Bifactor difference % difference

-0.84

-0.91

-0.89

0.43

0.44

0.86

0.51

0.46

0.65

0.61

0.58

0.63

0.46

0.65

0.55

-0.72

-0.75

-0.75

0.46

0.46

0.80

0.55

0.49

0.70

0.66

0.63

0.68

0.49

0.69

0.59

0.12

0.16

0.14

0.03

0.03

-0.06

0.03

0.03

0.05

0.05

0.05

0.05

0.04

0.04

0.04

-17%

-22%

-19%

6%

6%

-7%

6%

7%

7%

8%

8%

7%

7%

6%

7%

Word list


ADAS 2

Single factor vs. secondary factor structure

• Single factor easier to explain but not consistent with theory, may or may not fit data as well

– Single factor model has a nice history of use in educational testing

– Nice tools to understand the quality of measurement

• Bi-factor harder to explain, more consistent with theory, may fit data better

– Tools not widely available to understand the quality of measurement

– Scores often highly HIGHLY correlated between single factor and bi-factor models

Scores correlate 0.95

-2 -1 0 bifact2

1 2 3

Word list

Ratings


ADAS 3

Further improves fit

Chi squared

DF

Single Bifactor 2 Bifactor 3

372.634* 277.210* 145.486*

48** 46** 46**

Baseline Chi squared 7559.487 7559.487 7559.487

DF 28 28 28

CFI

TLI

RMSEA

0.957

0.975

0.091

0.969

0.981

0.078

0.987

0.992

0.051

0.90 / 0.95

0.90 / 0.95

0.08 / 0.05

Differences in loadings

PQ01A

PQ01B

PQ01C

PQ02

PQ03

PQ04

PQ05

PQ06

PQ07

PQ08

PQ09

PQ10

PQ11

PQ12

PQ14

0.46

0.65

0.61

0.58

0.63

0.46

0.65

0.55

Single Bifactor 2 difference % difference Bifactor 3 Difference % difference

-0.84

-0.72

0.12

-17% -0.75

0.09

-13%

-0.91

-0.89

-0.75

-0.75

0.16

0.14

-22%

-19%

-0.77

-0.78

0.14

0.11

-18%

-14%

0.43

0.44

0.86

0.51

0.46

0.46

0.80

0.55

0.03

0.03

-0.06

0.03

6%

6%

-7%

6%

0.46

0.46

0.83

0.54

0.03

0.03

-0.03

0.03

6%

6%

-4%

6%

0.49

0.70

0.66

0.63

0.68

0.49

0.69

0.59

0.03

0.05

0.05

0.05

0.05

0.04

0.04

0.04

7%

7%

8%

8%

7%

7%

6%

7%

0.49

0.70

0.65

0.59

0.56

0.40

0.56

0.59

0.03

0.05

0.05

0.02

-0.07

-0.06

-0.08

0.04

7%

7%

7%

3%

-12%

-15%

-15%

6%

Information content: single factor

12 1.2

10

8

6

1

0.8

0.6

4

2

0

-3 3

0

0.4

0.2

-2 -1 0 1 2

ADAS Single vs. Bi-factor

• Single factor is sure nice

– Easy to explain

– But it sure fits less well

– And over-estimates measurement precision somewhat

• Bi-factor is sure nice

– Not so hard to explain

– More consistent with our theory

– Fits better

• Measurement precision appropriately estimated at least within the SEM framework

I( θ) in presence of nuisance factors

So we integrate over the nuisance dimensions….

What about this?

Multiple versions longitudinally

• A bit of a challenge

• But nothing we can’t handle!

• Does require some assumptions

4

0.79

0.79

0.90

0.77

0.78

1_3

0.86

Different versions of the word list

1_2

0.77

0.77

0.88

-2.5

1_1

-2 -1.5

-1 -0.5

0 0.5

1 1.5

2

0.71

0.72

0.81

2.5

4

0.79

0.79

0.90

0.77

0.78

1_3

0.86


1_2

0.77

0.77

0.88

-2.5

1_1

-2 -1.5

-1 -0.5

0 0.5

1 1.5

2

0.71

0.72

0.81

2.5


4

1_3

0.79

0.79

0.90

0.77

0.78

0.86

1_2

0.77

0.77

0.88

-2.5

1_1

-2 -1.5

-1 -0.5

0 0.5

1 1.5

2

0.71

0.72

0.81

2.5

Mplus ADAS model constraints 1

ADAS model constraints 2





And there’s a cost

• 320 lines of code

Beginning Time: 14:09:56

Ending Time: 15:01:36

Elapsed Time: 00:51:40

Limitations

• Only tried this with the single factor model

– Thresholds likely to be unaffected

– But loadings may vary

• Also relies on a crucial assumption that may not hold

Essentially co-calibration over time

• Cross sectional co-calibration assumption: people are people. Relationship of items to the underlying domain, controlling for the latent ability level, is invariant across people

• This seems problematic with memory vs. other domains when we have people with MCI and AD

– Memory declines faster

– Maybe not hugely so over 6 months

Outline



– SEM and IRT


• Memory in ADNI



Scr/0


Scr/0 avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl

6 avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6


Scr/0 avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl

6 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl

12 limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6


Scr/0 avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl


12 avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl

18 avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6


Scr/0 avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl






Scr/0 avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl




24 limmtotal avtot1 avtot2 avtot3 avtot4 avtot5 avtotb avtot6 ldeltotal avdel30min avdeltot cot1scor cot2scor cot3scor cot4totl mmballdl mmflagdl mmtreedl


Strategies

• Ignore (likely a bad idea)

• Use flexible approach we used for ADAS-Cog

– Mplus to Paul: “That model looks hard. Perhaps you could give me an easier model.”

– Paul to Mplus: “No, computers don’t date.”

– Mplus: “Fine. I’m not talking to you anymore.”

• Use dummy variables for study visit and regress versions on visit (Rich Jones)

– But: single parameter does not address all differences in ADAS-Cog versions

• ???

The “ignore” strategy

Executive functioning in ADNI

• Report is in the documentation

• Bifactor scores

Acknowledgements

• R01 AG 029672, “Improving cognitive outcome precision & responsiveness with modern psychometrics”

• UW

– Betsy

– Elena

– Elizabeth

– Joey

– Jonathan

– Laura

– McKay

– RJ

• Hebrew SeniorLife, UC Davis, Washington U

– Alden

– Chengjie

– Dan

– Danielle

– Doug

– Frances

– Rich

Item 5

Item 6

Item 7

Item 8

Item 9

Item 1

Item 2

Item 3

Item 4

Depression*

Psychometric analysis of ADNI data

Related documents

Products

Support

Psychometric analysis of ADNI data

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib