Longitudinal Approaches to
Research and Evaluation
in Adult Education
Professor Stephen Reder
Portland State University
Charles Darwin University
October 9, 2014
Outline of Session




Flexible & negotiable depending on your
needs and interests
Longitudinal Research Methods
Applications to Research and Program
Evaluation in Adult Education
Example applications in my own longitudinal
study of adult learning
Longitudinal Methods Texts



Singer, J. D. & Willett, J. B. (2003). Applied
longitudinal data analysis: Modeling change
and event occurrence. Oxford & New York:
Oxford University Press.
Magnusson, D., Bergman, L. R., Rudinger, G.,
et al. (Eds.) (1994). Problems and methods in
longitudinal research: Stability and change.
New York: Cambridge University Press.
Saldaña, J. (2003). Longitudinal qualitative
research: Analyzing change through time.
Lanham, MD: AltaMira Press.
Longitudinal Adult LLN
Research Text

Reder, S. & Bynner, J. (Eds.) (2009).
Tracking adult literacy and numeracy skills:
Findings from longitudinal research. New
York & London: Routledge.
Many Fundamental Research Questions in
Adult Education Involve Change Over Time




How adults learn
How participation in programs influences
adults’ acquisition of knowledge and skills
How adults’ educational development interacts
with their social and economic performance
How adults’ private and public situation in the
family and community may enhance or
obstruct their progress in learning
Methodological Concepts and Issues
in Longitudinal Research



Sample attrition and missing data
Longitudinal stability of measures and scales
Methods for analyzing change in longitudinal
data
Sample Attrition and Missing Data




Missing item and unit data
Ignorable and Non-Ignorable Missing
Missing Completely at Random (MCAR)
Missing at Random (MAR)
Constructing Measures and Scales
That Are Longitudinally Stable



Essential to know that the same construct is being
measured at different time points
Scales having good psychometric properties for
measurement on a single occasion do not
necessarily have good psychometric properties
for measuring change across multiple occasions
Cannot determine longitudinal stability in
advance, must use longitudinal data to examine
measurement stability
Some Methods For Analyzing
Change in Longitudinal Data





Growth Curve Modeling
Event Occurrence Modeling
Structural Equation Modeling
Treatment Effects Models
Dynamic Panel Models
The Longitudinal Study of Adult Learning
(LSAL)
Portland State University
funded by U.S. Department of Education
funded by
U.S. Department of Education
National Institute for Literacy
LSAL Perspective


Look at how programs fit into the lifelong and
life-wide landscape of adults’ learning, rather
than at how adults fit into LLN programs as
students
We’ll see that things look considerably
different from this vantage point
LSAL Design

Decade-long panel study of Portland (Oregon) early school leavers,
age 18-44 at the beginning of the study

Representative sample of ~1,000 drawn from local rather than
national population of dropouts

Includes both program participants and nonparticipants

Examines program participation and other learning activities, social
and economic changes, and changes in literacy skills, literacy
practices & technology use over time

Periodic in-home interviews and literacy assessments and SSNlinked administrative data (with individuals’ permission)

Smaller-scale qualitative components
LSAL Realized Sample

N = 940





496 from RDD Frame
444 from Student Frame
High level of diversity in sample
90% sample retention over 8 years
39 additional pilots for instrument development,
training & qualitative studies
Some LSAL Demographics








Average age is 28 (at Wave 1)
50 % female and male
35 % minority
9 % foreign-born
34 % live in poverty
29 % report a learning disability
34 % took special education
Broad range of assessed basic skills
LSAL Timeline

wave 1
1998 – 1999

wave 2
1999 – 2000

wave 3
2000 – 2001

wave 4
2002 – 2003

wave 5
2004 – 2005

wave 6
2006 – 2007
Repeated Measures of Engagement in Literacy Practices
use comp at home
do math
read manuals, ref mats
write note, letter, email
write diary,story,poem
read mags, comic bks
read non-fiction
read fiction
read news section
read entertain,tv guides
read street maps
use ATM
do financial math
read instructions
0%
never
rarely
20%
< weekly
40%
weekly
60%
> weekly
80%
daily
100%
Scaling of Literacy Practices
•Started with 14 parallel questions, such as:
How often do you read the news section of the newspaper?
never / rarely / less than once a week / once a week
a few times a week / every day
• Confirmatory factor analyses of wave 1 data were used to
examine internal reliability & factor structures
• Two factors emerged, one assessing frequency of
engagement in literacy practices and one in numeracy
practices
• Structural equation models indicate these factors have
stationary measurement properties over time
Literacy Growth Curves





With multiple time points and repeated measures, we
can develop and test models of individual change in
literacy
We can examine how individual characteristics (e.g.,
years of schooling, nativity), key life events (e.g.,
birth of a child) and time-varying experiences (e.g.,
program participation) affect the growth process
Linear growth curve models closely fit LSAL data on
literacy proficiency and literacy practices
Individuals’ literacy proficiency and practices do
change substantially after leaving school
Heterogeneity of individual change is crucial
Person-specific slope
(“rate of change”)
Person-specific intercept
(“starting level”)
Change in Proficiency (Wave 1-4) by
Age at Wave 1
10
8
6
Change in 4
Document
2
Literacy
0
-2
-4
18-24
25-31
32-38
Wave 1 Age
39-44
NALS Document Literacy by Age
(1992 national data corrected for education & disabilities)
Proficiency
300
275
250
225
200
16-24 25-34 35-44 45-54 55-64 65+
Age
Similar results in ALLS & PIAAC
data from Australia
Key Findings
• Measures of literacy proficiency and literacy
practices show systematic change over time across the
adult lifespan
• Growth curve models of these literacy measures show
impressive heterogeneity of change: some adults show
literacy growth, others little change, and others literacy
loss over time
• Age, birthplace, parental education, intergenerational
reading practices, K-12 schooling experiences, and
health systematically influence adult literacy
development
Key Findings (con’t)
• Key life history events – such as changes in family
composition and employment changes -- influence adult
literacy development
• The dynamics of change are quite different for literacy
proficiency and literacy practices: e.g., program
participation directly affects literacy practices measures
but not literacy proficiency
Outcomes & Program Impact





Literacy proficiency growth over relatively short periods of time
is not affected by program participation
Pre-post test accountability data, that apparently show
systematic gains in participants’ proficiency, do not contrast
participants’ gains with those of comparable non-participants;
LSAL indicates those gains are equivalent
Literacy practices growth over relatively short periods of time is,
on the other hand, directly affected by program participation
These findings are reinforced by cross-sectional research (e.g.,
Smith & Sheehan-Holt) and by classroom studies (e.g., PurcellGates, Jacobson & Degener)
Proficiency measures thus do not reflect the impact that
programs have or support evidence-based program and policy
improvement processes
Identifying Causal Relationships




Temporal ordering
Including all relevant variables
Analytical methods for experimental designs
Analytical methods for non-experimental designs

Dealing with selection bias




Standard regression methods
Propensity score matching methods
Instrumental variables methods
Dealing with omitted variables bias

Fixed effects panel regression
Estimating Participation Impact

Adults decide whether to participate in basic skills
programs, so participants and nonparticipants are not
usually comparable (selection bias)

Several analytical methods can be used to address
selection bias in comparing program participants &
nonparticipants:



Treatment effects (propensity score matching)
Difference-in-differences (propensity score matching)
Fixed effects panel regressions
Propensity Score Matching

Compares participants and nonparticipants
matched on their likelihood of participating
based on observed pre-participation
characteristics:



Age Gender Race/Ethnicity Education
Immigration status Income
Learning disabilities Parents’ education
Income Growth in Matched Participants (100+ hours) & Nonparticipants
Treatment Effects Models

Estimates average treatment effect on treated by comparing 2007
incomes of matched participants and nonparticipants

With participation defined as any amount of attendance, there is
no significant difference between groups

With participation defined as 25 or more hours of attendance,
there is no significant difference between the groups’ 2007
incomes

With participation defined as 75 or more hours of attendance,
there is a nearly significant (p=0.053) difference between the
groups’ 2007 incomes

With participation defined as 100 or more hours of attendance,
there a statistically significant difference: participants average
$9,621 more in annual income over what they would have
received if they had not participated (in 2013 USD)
Treatment Effects of Participation on 2007 Earnings
Model
Minimum
Number
Average
Number of
Hours of
of
Treatment
Propensity
Attendance Propensity
Effect on
-Scoreto Be
-Scorethe
Matched
Considered Matched
“Treated”
“Controls”
“Treated” “Treated”
(ATET)
t
p
Effect
Size
A
1
396
105
1.111
1.411 0.159
n.s.
B
25
216
82
0.789
0.953 0.342
n.s.
C
75
216
80
2.014
1.942 0.053
n.s.
D
100
197
68
2.142
2.469 0.014 0.45
E
150
154
62
2.272
2.085 0.039 0.48
Difference-in-Differences (DID) Model

Compares income changes over a decade (1997-2007)
between matched participants and nonparticipants

There is no statistically significant DID between groups if
participation is defined as any amount of attendance

If participation is defined as 100 or more hours of attendance,
there is a statistically significant DID

Despite different statistical assumptions, estimates 2007
incomes to average $10,179 more because of participation,
comparable to the treatment effects estimate of $9,621 (in
2013 USD)
Difference-in-Differences
16000
14000
Annual Income (1997 $)
12000
10000
8000
6000
4000
2000
0
Before (1997)
Nonparticipants
After (2007)
Participants
DIFFERENCE-IN-DIFFERENCES ESTIMATION
100+ hours
--------------------- ------------ BASELINE --------- ----------- FOLLOW-UP ---------- --------Outcome Variable
| Control | Treated | Diff(BL) | Control | Treated | Diff(FU) |
DID
---------------------+---------+-----------+----------+---------+-----------+----------+--------Log Income
| 7.651 | 6.126
| -1.525
| 5.386
| 6.935
| 1.549
| 3.074
Std. Error
| 0.513 | 0.468
| 0.695
| 0.617
| 0.513
| 0.802
| 1.061
t
| 14.91
| 4.39
| -2.19
| 3.98
| 9.85
| 2.31
| 2.90
P>|t|
| 0.000 | 0.000
| 0.029** | 0.000
| 0.000
| 0.054*
| 0.004***
------------------------------------------------------------------------------------------------* Means and Standard Errors are estimated by linear regression.
**Inference: *** p<0.01; ** p<0.05; * p<0.1
Fixed Effects Panel Regression Model

Within-subject models of year-to-year variations in
income in relation to year-to-year program
participation and other life events

Eliminates selection bias due to observed and
unobserved time-invariant individual characteristics

Reveals how temporal details of participation -intensity, duration and elapsed time – are reflected in
observed changes in economic status
Fixed Effects Panel Regression (con’t)

Results consistent with other models

Only when participation involves about 100 or more
hours of attendance does it have a significant &
substantial impact on future earnings

Concentrated hours have a larger impact on earnings
than hours distributed over years

The impact of participation on earnings takes several
years to develop after program exit
Fixed Effects Panel Regressions of Log Annual Earnings
Model
Participation
Variables
Coeff. in
Panel Regr.
Robust Std.
Err.
t
A
CUMHOURS
.0032
.0014
2.39*
B
YEARS
.1221
.1756
0.70
C
YRSCUM100
.4005
.1510
2.65**
D
YRSCUM100
CUMHOURS
.3106
.0020
.1411
.0013
2.20*
1.55
Pulse, Step, Growth:
The Shape of Program Impact
Proficiency
PULSE
<=
• High
Time
burst, short-lived impact
• Example in LSAL data: effects of receiving
GED credential shows a short-lived “brushing
up” of proficiency
Proficiency
STEP
<=
• Abrupt,
Time
qualitative & lasting impact
• Some changes in literacy practices seem to
have this temporal shape
Proficiency
GROWTH
<=
Time
• Slow, steady & progressively accumulating impact
• This is the shape of program impact on proficiency
• This is the shape of program impact on earnings
• Life history events -- such as the birth of children, taking on or losing a
partner, or a significant change in employment -- have similarly shaped
impacts on the course of literacy development
Summary: Impact of Participation on Earnings

Multiple methods of controlling for selection bias all indicate that
participation in LLN programs has a significant positive impact on
adults’ future earnings

The significance of the impact requires a minimum amount of
program attendance, about 100 hours in the LSAL data

The earnings premium grows over time and becomes substantial 5-6
years after program exit: the annual premium was nearly half (0.45) a
standard deviation of 2007 incomes

The impact of participation is not at all evident in short-term followups to program participation

Post-program learning, proficiency growth, and postsecondary
education and training may all play a role mediating the continuing
impact of participation on labor market outcomes
Fixed Effects Panel Regressions of Log Annual Income
Coeff. in
Panel
Regr.
Robust Std.
Err.
t
E
CUMHOURS
HAVEGED
.0030
.4793
.0013
.4648
2.25**
1.03
F
CUMHOURS
YEARSGED
.0028
.4160
.0013
.1394
2.17*
2.98**
G
YRSCUM100
HAVEGED
.3848
.5360
.1478
.4542
2.60**
1.18
H
YRSCUM100
YEARSGED
.3025
.3821
.1437
.1364`
2.10*
2.80**
Model
Participation
Variables