USING THEORY AND FACTOR
ANALYSIS TO GUIDE INSTRUMENT
REFINEMENT
Xiaoying Xu, Jennifer Lewis
Department of Chemistry
University of South Florida
ASSESSMENT IN CLASSROOM

Why to assess

An essential component of education
Classroom use: diagnosis, placement, and grading
 Data-driven reform in education
 Performance standard setting


What to assess

Cognitive: subject content knowledge & skills


SAT, ACT, GRE and NEAP
Non-cognitive: attitude, motivation, self-efficacy
PSYCHOMETRIC QUALITY FOR
ASSESSMENT
Why psychometric quality?
What evidence for psychometric
quality?
Reliability
Validity
RELIABILITY
•
Is the assessment providing consistent
results?
•
•
Random scores with poor reliability cannot be
valid.
Two different angles
test-retest reliability: two administrations of
the same test for the same sample
 internal consistency reliability:
 acceptable reliability: Cronbach’s alpha > .7

Nunnally, 1978)
VALIDITY
 Validity
is about the degree to which scores
from a measurement measure what it purports
to measure.
 Validity is an integrated evaluative judgment
of the degree to which empirical evidence and
theoretical rationale supports the adequacy
and appropriateness of inference and actions
based on test scores or other models of
assessment. Messick ,1989

Validity is a process, not for an instrument itself.
ASPECTS OF VALIDITY


Content validity: developing and reviewing the content clarity and
representative of construct domain.
Construct Validity: a mutual verification of the measuring
instrument and the theory of the construct it is meant to
measure (Angoff, 1988).




Factorial validity: each item should load on the assumed factors from factor
analysis, thus provides strong confirmation of construct validity.
Convergent validity: the results of one specific construct from different
methods should be similar.
Discriminant validity: the results of different constructs should be different.
Criterion-related validity



Predictive validity: predict later performance
concurrent validity: correlate with other constructs measured at the same time
Nomological network: lawful network among concepts
FACTOR ANALYSIS TECHNIQUE FOR
CONSTRUCT VALIDITY
Using covariance matrix manipulation, it identifies factors
to explain most variance in the measured data set
•
E.g., 20 items in 2 scales, 10 items correlated, 2 factor score
report, instead of 20 item score, make the results simple and
easy!
CFA has offers more advantages over EFA
•
•
•
Both provide item loading, factor correlation
CFA provide how data fit conceptual model, and
measurement error
“… the most rigorous test of factorial structure is
through CFA techniques”. Greenbaum & Dedrick, 1998
RESEARCH PURPOSE
1 To evaluate quality of the attitude instruments
in terms of reliability and validity
2 To use theory and factor analysis to guide
instrument refinement
WHY ASCI (ATTITUDE TOWARD
THE SUBJECT OF CHEMISTRY
INVENTORY)?
Offered a clear definition and conceptual framework
 Provided the evidence for reliability and validity using
factor analysis
 High quality (18 out of 28 by Blalock ) & could be better

Bauer, C. J. Chem. Educ. 2008, 85, 1440
Blalock, C. L. etc Int. J. Sci. Educ. 2008, 30, 961
BLALOCK’S RUBRIC USED FOR 66 SCIENCE
ATTITUDE INSTRUMENTS EVALUATION
Theoretical background for instrument development (3 pts)
 Reliability: internal consistency, test–retest, and standard
error of measurement (9 pts)
 Validity: content, discriminant, congruent, contrasting
groups, and factor analyses (9 pts)
 Dimensionality (6 pts)
 Development and usage (1 pt)

Dimensionality of ASCI
• The 20 items are to measure students’
attitude toward chemistry in general on a 7point semantic format.
• Sample item: chemistry is difficult vs. easy
• Five subscales:
•
•
•
•
•
Intellectual accessibility (5 items)
Interest and utility (5)
Fear (1)
Emotional satisfaction (4)
Anxiety producing (5)
Attitude Framework in Psychology
• Definition: a psychological tendency that is
expressed by evaluating a particular entity
with some degree of favor or disfavor
Eagly & Chaiken, 1993
• Framework
• Think?
• Feel?
• Behave?
Flowchart for ASCI Refinement
ASCI DATA SCREEN & ANALYSIS
Any response with multiple answers, missing data, missing
UID or wrong response was excluded
 Pattern of missing data was checked for whether the
missing data may bias the findings
 Descriptive statistics:

M, SD, normality, Cronbach’s alpha >.7
 Factor score and correlation


EFA & CFA for model fit
 CFI < .95 and SRMR < .08
Hu, L.; Bentler, P. M. In Struct Equation Model: Concepts, Issues and Applications,1995, p 76
Mean of ASCI V1 at SE, GC labs I & II
M (lab I, n=405)
5.13
4.00
4.54
4.68
4.86
3.95
3.95
M (lab II, n=509)
5.14
4.05
4.56
4.56
4.92
4.06
3.92
relaxed
secure
5.82
5.10
4.14
5.65
4.94
4.16
hard
simple
clear
incomprehensible
2.60
2.80
3.22
4.02
2.98
2.91
3.38
4.26
dangerous
2.31
3.88
2.57
3.78
frustrating
unpleasant
uncomfortable
organized *
3.26
3.63
3.76
4.03
3.44
3.69
3.91
3.97
Interest &
utility
2
3*
6*
12*
15*
8* Anxiety
13*
Worthless (1)
exciting
good
interesting
worthwhile
scary
disgusting
Beneficial (7)
boring
bad
dull
useless
fun
Attractive
Work (7)
Play (1)
tense
insecure
accessibility
easy
complicated
confusing
comprehensible
16*
19*
20*
Intellectual
1*
4
5
9*
10
18 (Fear)
Emotional
7*
11*
14*
17
challenging unchallenging
safe
Satisfaction
satisfying
pleasant
comfortable
chaotic
recoded
INTERNAL CONSISTENCY AND
TEST-RETEST RELIABILITY FOR ASCI V1
Scale (Items)
Interest&Utilit
y (15,2,6,12,3)
Anxiety
Internal consistency
Lab I
0.82
Lab II Bauer
0.85 0.83
Test-retest reliability
Lab II (n=10) Bauer
0.91
0.74
0.71
0.79
0.77
0.96
0.64
0.79
0.82
0.78
0.96
0.71
0.74
0.78
0.79
0.96
0.72
(19,16,8,20,13)
Accessibility
(4,5,1,10,9)
Emotional
satisfaction
(11, 14,17, 7)
FACTOR SCORE AND CORRELATION
Mean
SD
Int&use Anxiety Access Fear Emotion
General Chemistry I lab (n=405)
54.9
59.9
33.2
47.9
44.5
13.3
15.7
17.6
23.2
19.8
Factor correlation
Interest
&Use
-0.55
Anxiety
Access
Fear
0.40
-0.64
-0.18
0.17
-0.17
0.63
-0.75
0.68
-0.25
Inter&use Anxiety Access Fear Emotion
General Chemistry II lab (n=509)
54.6
59.1
37.0
46.3
45.9
13.7
17.4
19.0
24.0
20.1
-0.63
0.38
-0.66
-0.15
0.19
-0.18
0.64
-0.79
0.67
-0.28
FIT STATISTICS
FOR ASCI V1, 4-FACTOR CFA
Fit Statistic
2
df
p
N
CFI
SRMR
Lab I
618.5
146
.00
405
.86
.076
Lab II
692.9
146
.00
509
.89
.079
Problem: CFI < .95, data doesn’t fit the
hypothetical model decently well.
FACTOR CORRELATION FOR ASCI V1 FROM CFA
Interest&utlity
Anxiety
Intell. Access.
Emotion
Anxiety Access satisfac.
Lab I
Emot.
Anxiety Access satisfa.
Lab II
-.72
-.77
-.46
.83
.80
-.98
-.85
-.49
.82
.84
-.97
-.82
Problem: "anxiety" and “emotional satisfaction” subscales are
strongly correlated and redundant
Suggestions for refinement: emotional satisfaction is a better
fit for the affective component of the attitude framework
CFA BASED ON ASCI V1 SINGLE
SUBSCALE: INTELLECTUAL ACCESSIBILITY
Subscale
Items
intellectual 1, 4, 5, 9, 10
accessibility 1, 4, 5, 10
(without 9)
4
5
1*
10
9*
Model fit
CFI
0.94
0.99
Comments
Item 9 harms the validity of intellectual
accessibility scale; removing it produces a good
fit
Intellectual Accessibility
complicated
simple
confusing
clear
easy
hard
challenging
unchallenging
comprehensible incomprehensible
* recoded
PROBLEM WITH THE “INTEREST AND UTILITY”
SUBSCALE
15*
2
6*
12*
3*
worthwhile
worthless
good
interesting
exciting
useless
beneficial
bad
dull
boring
The “interest and utility” subscale has more than one concepts.
* recoded
Proposed ASCI V2
V2
Items
1, 4, 5, 10
7, 11, 14, 17
Model fit
CFI
0.96
Comments
final version captures both intellectual
accessibility and emotional satisfaction,
congruent with two-component attitude theory
Intellectual Accessibility
4
5
1*
10
Emotional Satisfaction
11* pleasant
unpleasant
complicated simple
14* comfortable uncomfortable
confusing clear
17 chaotic
organized
easy
hard
challenging unchallenging 7* satisfying
frustrating
* recoded
DATA COLLECTION & ANALYSIS
FOR ASCI V2
Given to the Peer-Led (PL) classes at the beginning of
the class at SE, 11th week Spring 09
 375 data set returned, 354 complete set
 EFA and CFA

THE ITEM LOADING
FOR ASCI V2 FROM EFA
Item # in each scale
V1
V2
Intellectual
1
1*
4
2
5
3
10
6
Emotional
14
4*
7
5*
11
7*
17
8
(N=354)
Accessibility
easy
complicated
confusing
challenging
Satisfaction
comfortable
satisfying
pleasant
chaotic
loading
F1
F2
hard
simple
clear
unchallenging
0.41
0.34
0.45
0.00
0.66
0.75
0.68
0.86
uncomfortable
frustrating
unpleasant
organized
0.74
0.72
0.75
0.71
0.29
0.36
0.38
0.02
* recoded
INTERNAL CONSISTENCY RELIABILITY
FOR ASCI V2 SCORE, PL CLASS
Internal consistency by Cronbach’s alpha
PL(N=354), V2
Intellectual Accessibility
Emotional Satisfaction
.82
.79
Literature
V1
.78
.79
(Bauer, 2008),
FIT STATISTICS OF ASCI V2 SCORE
FOR 1- AND 2-FACTOR CFA
Fit Statistic
Model
2
df
p
N
CFI
SRMR
Value
one-factor
156
20
.00
354
.89
.056
two-factor
77
19
.00
354
.95
.042
IMPLICATIONS FOR FUTURE RESEARCH

V2 in different population
V2 for longitudinal study to track attitude change, pre- and
post- test
 For curriculum effect, quasi-experimental design
 Further evidence for validity under different test situations


Limitation
V2 does not captures all potential aspects of attitude
 It is not the case that t a single measure is desirable

ACKNOWLEDGEMENT

The NSF project DUE Grant No. 0817409



Thomas Holme, Melanie Cooper, Marcy Towns, Jennifer
Lewis
Alicia, Bo, Janelle, Karla, Keily, Sachel, Teresa, and
Ushiri, Drs. Rick Moog, Santiago Sandi-Urena
Dedrick, Ferron and Kromrey from education
department, USF for conversation for data analysis