Online Research and
Comprehension Assessment:
How Connecticut Students in Rich and Poor School Districts
Perform on New Aspects of the Common Core State
Standards
Donald J. Leu, Elena Forzani, Clint Kennedy, and Cheryl Burlingame
Neag School Of Education
University Of Connecticut
Transition to the Connecticut State Standards and System of Assessments
Third Annual Connecticut Assessment
Crown Plaza, Cromwell
Rocky Hill, CT
August, 2012
PowerPoint available at: HTTP://WWW.EDUCATION.UCONN.EDU/ASSESSMENT/
1
Common Core State Standards:
Three Fundamental Changes to
English Language Arts



Emphasis on Non-Fiction
Emphasis on Higher-Level Thinking
The First Appearance of Online Research
and Comprehension Skills
2
Online Research and
Comprehension are Blended Into
the ELA Standards
“To be ready for college, workforce training, and life in a
technological society, students need the ability to gather,
comprehend, evaluate, synthesize, and report on
information and ideas, to conduct original research in
order to answer questions or solve problems, and to
analyze and create a high volume and extensive range of print
and nonprint texts in media forms old and new.”
(Introduction, p. 4)
3
Online Research and Comprehension
Skills Appear in at Least Half of the
Reading and Writing Anchor Standards
4
Why?
The Nature of Work Has Changed
The “General Motors” Model of Economic
Management
CEO
Upper Level Management
Upper Middle Level Management
Middle Level Management
Line Supervisors
Workers
5
Wasted
intellectual
capital
In a Flattened World: Opportunities
Expand but Competition Increases
How do economic units increase productivity?
Flatten The Organization into Problem Solving Teams
Team
Team
Team
Team
Team
Greater Intellectual Capital Use = Greater Productivity
These teams take full
advantage of their intellectual
capital to the extent their
education system has
prepared them for this.
6
1.
2.
3.
4.
Define problems
Locate information
Critically evaluate information
Synthesize and solve
problems
5. Communicate solutions
Which Tool Has Been Used By
Economic Units To Increase
Productivity And Compete?
The Internet
Team
Team
Online Research and
Comprehension
1. Define problems
2. Locate information
3. Evaluate information
4. Synthesize - solve problems
5. Communicate solutions
7
Team
Team
Team
Recent productivity gains are due
to using the Internet to share
information, communicate, and
solve problems (van Ark, Inklaar, &
McGuckin, 2003; Friedman, 2005;
Matteucci, O’Mahony, Robinson,
& Zwick, 2005).
These New Literacies Appear at
Every Grade Level in ELA in the
CCSS
Kindergarten (WS 6)
With guidance and support from adults, explore a variety
of digital tools to produce and publish writing, including
in collaboration with peers.
Grade 3 (WS 8)
...gather information from print and digital sources; take brief
notes on sources and sort evidence into provided categories.
8
These New Literacies Appear at
Every Grade Level in ELA in the
CCSS
Grade 6 (RS 7)
Integrate information presented in different media or
formats (e.g., visually, quantitatively) as well as in words
to develop a coherent understanding of a topic or issue.
Writing Anchor Standard 6
Use technology, including the Internet, to produce
and publish writing and to interact and collaborate
with others.
9
An Important Challenge:
How Do We Assess Online
Research and Comprehension?
10
PIs
Donald J. Leu, The University of Connecticut
Jonna Kulikowich, The Pennsylvania State University
Nell Sedransk, National Institute of Statistical Sciences
Julie Coiro, University of Rhode Island
Graduate Research Assistants
Elena Forzani, Clint Kennedy, and Cheryl Burlingame, The University
of Connecticut
Scientific Advisory Board
P. David Pearson, The University of California, Berkeley
Irwin Kirsch, Educational Testing Service
Rand Spiro, Michigan State University
Elizabeth Stage, Lawrence Hall of Science, Berkeley
Glenn Kleimann,
Friday Institute, NCSU
11
Project Goals
1) Develop assessments of online research and
reading comprehension in three different
formats (ORCA-MC, ORCA – Open, ORCAClosed).
2) Evaluate each instrument's internal
assessment characteristics.
3) Evaluate the extent to which performance on
each format is associated with various student
variables.
4) Evaluate the practicality of each assessment
format in the eyes of key education decision
makers.
12
ASSESSMENT TOPICS AND CONTEXTS
13
SCORE POINT SYSTEM
14
SCORE POINT SYSTEM
15
16
17
18
Reading to Evaluate
Information Online
19
Reading to Synthesize
Information Online
20
21
Corresponding Multiple Choice
Format
22
Research Question

Is there an achievement gap in the ability
to read on the Internet and conduct
research between students who attend
rich and poor school districts?
23
Reading Achievement Gaps
Between Rich and Poor (NAEP)
% At or Above Proficient
55%
23%
245
275 250
% Below basic
32%
10%
24
90/10 Income Achievement & Black-White Gaps
in Reading, 1943-2001 Cohorts
Average Difference in S.D. Units on National
Assessments
*Adapted from: Reardon, S.F. (2011). The widening academic achievement gap between the rich and the poor:
New evidence and possible explanations. In R. Murnane & G. Duncan (Eds.), Whither Opportunity? Rising
Inequality and the Uncertain Life Chances of Low-Income Children. New York: Russell Sage Foundation Press.
Years
25
A Central Problem for New Literacies
Research:
The Stability of Assessment Contexts
?
26
Research in Online Reading
Comprehension



A problem-based research and reading
process
Elements: locate, evaluate, synthesize,
communicate
Online reading ≠ offline reading



Leu, Castek, Hartman, Coiro, Henry,
Kulikowich, & Lyver, 2005.
Coiro & Dobler, 2007
27 2011.
Coiro,
Research on Internet Access at
Home and at School: Poor Students
May Be Doubly Disadvantaged

Less home access among poor:


32.1% adoption rate when family income less than
$15,000
89.6% when more than $150,000


(US Dept. of Commerce, 2011)
Instructional access at school may be affected
by state tests:

Greater pressure in poor schools to teach to state
tests - with no aspects of online reading
comprehension included.
28
Purpose


To determine if a significant online
research and reading achievement gap
existed between students attending rich
and poor school districts.
Evaluate a new, more stable assessment
format: ORCA-Closed

Part of the ORCA Project (IES)
29
METHODS
30
Participants



256 7th-grade students attending a rich
and a poor school district in CT.
West Town: a wealthy school district
(Level B in CT District Reference Groups)
East Town: a poor school district (Level
H in CT District Reference Groups)
31
School District Differences
West Town
DRG B
East Town
DRG H
Median Family Income
$119,338
$58,981
% of Families Below
Poverty Line
2.1%
11.8%
% of Students Eligible for
Free/Reduced Price
Lunches
4%
67%
32
Population Sample
West
East
Total
Town
Town
(Richer) (Poorer
)
108
148
256
Seventh Grade
Students
Students who completed both research activities:
Asthma (wiki) and Energy Drinks (email)
33
Internet Use Survey

Home Access



Do you have a computer at home? (Yes, No)
How many computers in your home are
connected to the Internet? (0, 1, 2, 3 or more)
School Instruction

How often have you been required to use the
Internet for a school assignment? (Never, Less
than once a week, Once a week or more.)
34
Offline Reading Comprehension


Connecticut Mastery Test (CMT)
Subtests





Reading for literary experience
Reading for information
Reading to perform a task
Degrees of Reading Power
Reliability = .94
35
ORCA-Closed

Two types of topics in science (human
body systems)
Learn more about: Are energy drinks heart
healthy?
 Take a position: Can Chihuahua dogs cure
asthma?

36
Scoring




4 score points in each of four areas: locate,
evaluate, synthesize, communicate.
Each area had 3 process score points and 1
product score points.
Process points were subjectively scaled prior
to the assessment by a panel of experts.
A total of 16 score points for each topic. 32
points were possible.
37
RESULTS
38
The ORCA-Closed Is a Reliable
Instrument
Combined Cronbach’s Alpha = .89
 Energy Drinks = .83
 Asthma = .79

39
A Significant Achievement Gap
Existed in Offline Reading
(CT Mastery Test: Reading)
300
225
150
75
0


West Town (Rich)
East Town (Poor)
★
Means:
CMT
Reading
t = 13.81 p = .000

eta squared = .511 (large)
West Town (Rich) Mean = 282.60 (SD = 41.54)
East Town (Poor) Mean = 215.10 (SD = 31.07)
40
A Significant Achievement Gap Also
Existed in Online Reading and
Research: ORCA-Closed
20
15
10
5
0
West Town (Rich)
East Town (Poor)
✴
Means:
ORCAClosed


t = 11.22, p = .000

eta squared = .394 (large)
West Town Mean = 15.00 (SD=5.69)
East Town Mean = 7.65 (SD=4.39)
41
...Even When an ANCOVA
Analysis Was Conducted
Covariates: Offline Reading + Prior Knowledge
16
12
8
4
0


West Town (Rich)
East Town (Poor)
ORCAClosed:
Adjusted
Means
★F
(1,234) = 15.84, p = .000

partial eta squared = .063

(medium)
West Town (rich) adjusted mean = 12.96
East Town (poor) adjusted mean = 10.27
42
West Town Students Had More
Computers Connected to the
Internet at Home
How many computers in your home are connected to the
Internet?
West Town
East Town
0
1% (1)
8% (12)
1
15% (15)
31% (45)
2
23% (23)
27% (39)
3 or more
61% (61)
33% (48)
43
★
Pearson ChiSquare = 23.40,
df = 3, p = .000
Differential Access to Instruction
Appeared at School
How often have you been required to use the
Internet for a school assignment?
West Town
East Town
Never
4% (4)
25% (35)
Less than
once a week
52% (52)
39% (39)
Once a week
or more
44% (44)
37% (52)
★
44
Pearson ChiSquare = 18.67,
df = 2, p = .000
DISCUSSION


Developed a reliable, new form of
assessing online reading comprehension,
ORCA-Closed, with greater stability.
The gap in reading between students
attending rich and poor school districts
appears even greater for online reading
than for offline reading.

Mean scores nearly 2x greater for students in
the richer
district.
45
DISCUSSION

An indication that the gap may be due to
home access as well as, perhaps,
unintended consequences of current state
assessment practices, especially in poorer
school districts.
46
The Take Aways?



There appears to be an additional
achievement gap, central to CCSS -- the
ability to research and read online.
Any society that professes egalitarian
ideals should be concerned about this gap
in the 21st century.
We continue work on the ORCA Project
and value partnerships with CSDE and
school districts for this work.
47
Online Research and
Comprehension Assessment:
How Connecticut Students in Rich and Poor School Districts
Perform on New Aspects of the Common Core State
Standards
Donald J. Leu, Elena Forzani, Clint Kennedy, and Cheryl Burlingame
Neag School Of Education
University Of Connecticut
Transition to the Connecticut State Standards and System of Assessments
Third Annual Connecticut Assessment
Crown Plaza, Cromwell
Rocky Hill, CT
August, 2012
PowerPoint available at: HTTP://WWW.EDUCATION.UCONN.EDU/ASSESSMENT/
48