Statistical Validation of
Prerequisites and
Corequisites: Approaches
from the Field
RP Conference
April 11, 2014
D AY L E N E M E U S C H K E , D I R E C T O R , I N S T I T U T I O N A L R E S E A R C H ,
COLLEGE OF THE CANYONS
J I M F I L L P O T, D E A N , I N S T I T U T I O N A L R E S E A R C H & R E S O U R C E D E V E L O P M E N T , C H A F F E Y
COLLEGE
KEITH WURTZ, DEAN, INSTITUTIONAL EFFECTIVENESS, RESEARCH & PLAN NING, CRAFTON
HILLS COLLEGE
A N N E D A N E N B E R G , R E S E A R C H A N A LY S T P L A N N I N G , R E S E A R C H A N D I N S T I T U T I O N A L
EFFECTIVENESS OFFICE, SACRAMENTO CITY COLLEGE
Session Objectives
Briefly review the Suggestions for CCC institutional Researchers
Conducting Prerequisite Research
Review approaches for locally validating prerequisites preimplementation and post-implementation
◦ Research methodologies
◦ Data sources
◦ Statistical techniques
Provide strategies for communicating prerequisite findings to internal
audiences and facilitating evidence-based decision making
Overview of the Prerequisite
Validation Guidelines
Intended to help researchers
◦ Execute the statistical analyses
◦ Support faculty
Developed by the RP Group with input from the field
Input provided by various groups: matriculation, faculty, researchers
Incorporated information from Academic Senate and Chancellor’s office
guidelines
The Prerequisite Research
Framework
Post-Implementation
Pre-Implementation Content
Analysis
The “Warm Up”
(a.k.a. Preliminary Work)
Content Review/Validity (Pre-Implementation)
◦
◦
◦
◦
◦
Develop a systematic but manageable approach
Researchers can help facilitate, if needed
Identify what is acceptable level of success
Ensure faculty have sufficient experience with the pre-requisite and target courses
Consider using a rating template
Content Review:
Sample Rating Template
Skill: Ability to…
#1
#2
Rater
#3
#4
Skill 1: Solve radical,
4
3
4
4
quadratic equations.
Skill 2: Solve logarithmic
4
4
5
4
equations.
Skill 3: Solve Number
exponential
of skills with3 a mean
4 rating
4 of ≥ 4.0:
5 4
equations.
Skill 4: Solve Percent
a varietyofofskills with a mean rating of ≥ 4.0: 80%
problems by applying the
5
4
4
3
definitions, postulates and
theorems of plane geometry.
Skill 5: Graph linear,
quadratic, simple polynomial,
5
4
5
4
exponential logarithmic
functions and conic sections.
#5
Mean
Rating
4
3.8
5
4.4
4
4.0
5
4.2
4
4.4
The “Warm Up”
(a.k.a. Preliminary Work)
Preliminary Analyses (Pre-Implementation)
◦ Is the pre-requisite reasonably likely to improve student success?
◦ What enrollment/access issues might arise?
◦ What impact might this have on other programs?
Information researchers need from faculty
◦ Changes in curriculum
◦ Inclusion/exclusion of different delivery modes
◦ Outcome measures
Pre-Implementation Statistical Analysis
Initiating a Statistical Validation Request
Local Discussion Among:
◦
◦
◦
◦
◦
Discipline Faculty
Faculty from Prerequisite Discipline
Student Services (Matriculation) Personnel
Curriculum Committee
Administration
•Knowing What the Process Entails
•Institutionally Approved Requestor/Process
◦ Curriculum Committee
◦ Approved Research Request Form with Appropriate Sign-Off
Some Useful Data Elements for
Statistical Validation of
Prerequisites
MIS Data Elements
◦ GI03 (Term Identifier)
◦ CB01 (Course-Department-Number)
◦ SX01 (Enrollment-Effective-Date)
◦ SX04 (Enrollment-Grade)
◦ SB, SD, STD Data Elements (for Disproportionate Impact)
Assessment Data
◦ Placement Recommendation (Communication and
Computational Skills Courses)
Sample Statistical Options
for Researchers
Test
Questions Answered with Prerequisite Validation Approach
T-Test
Is their a statistically significant difference between the average
GPA/success rate in the prerequisite and target courses? The
observed difference may not be substantial.
Chi-Square
Is there a statistically significant difference between GPA in the
target course and completion of the prerequisite course? The
observed difference may not be substantial.
Pearson
Correlation
Is there a significant relationship between GPA in the prerequisite
and target courses?
Sample Statistical Options
for Researchers
Test
Questions Answered with Prerequisite Validation Approach
Odds Ratio
How likely is it that students who meet the prerequisite will succeed
in the target course compared to those who do not meet the
prerequisite?
Effect Size
What is the strength of the relationship between successfully
completing the prerequisite course and successfully completing the
target course?
Average
Percent
Gain
What is the average percent gain in success in the target course of
students who met the prerequisite over those who did not meet the
prerequisite?
Chaffey College
Prerequisite Validation Approach
Three-Pronged Approach:
1.
Comparison of Performance in the Target Course of Students Who Did
and Did Not Meet the Prerequisite
2.
Effect Size (accounts for influence of sample size) and Average Percent
Gain
3.
Restricted Bivariate Correlation Coefficient and Corrections for
Restriction of Range
◦
Pearson’s r (Rule of Thumb: r ≥ .35, assuming p < .05)
◦
Chaffey also recalculates to correct for restriction of range
Chaffey College
Prerequisite Data Table
Examination of Disproportionate Impact
Prior to Prerequisite Enforcement
Disproportionate impact occurs when “the percentage of persons from a
particular racial, ethnic, gender, age or disability group who are directed to
a particular service or placement based on an assessment instrument,
method, or procedure is significantly different from the representation of
that group in the population of persons being assessed, and that
discrepancy is not justified by empirical evidence demonstrating that the
assessment instrument, method or procedure is a valid and reliable
predictor of performance in the relevant educational setting.” [Title 5
Section 55502(d)]
When there is a disproportionate impact on any such group of students,
the district shall, in consultation with the Chancellor, develop and
implement a plan setting forth the steps the district will take to correct
the disproportionate impact.” [Title 5 Section 55512(a)]
Disproportionate Impact Guide (CCCCO and RP Group, 2013)
Examination of Disproportionate Impact
Prior to Prerequisite Enforcement
Classification and Regression Trees (CART)
◦ divide a population into segments that differ with respect to a
designated criterion.
◦ identifies the best predictor variable, conducting a splitting
algorithm that further identifies additional statistically significant
predictor variables and splits these variables into smaller
subgroups
◦ ensures that cases in the same segment are homogeneous with
respect to the segmentation criterion, while cases in different
segments tend to be heterogeneous with respect to the
segmentation criterion
Classification and Regression Tree (CART)
Example Tree
Chaffey College
Decision to Implement a Prerequisite
Green – Sufficient evidence exists to enforce prerequisite (at
least two out of three measures are supported)
Yellow – Although evidence exists, only one out of three
measures supports enforcement of the prerequisite. Further
discussion should occur within the department and the
Curriculum Committee before the prerequisite is enforced
Red – Data does not exist to support enforcement of the
prerequisite. None of the measures explored meet preestablished criteria
Insufficient Data – While evidence may point to the efficacy of
the prerequisite, the sample size is too small to render a reliable
decision
Post-Implementation
Statistical Analysis
Post-Implementation Statistical
Analysis: Research Questions
The following questions were examined to determine the impact of
implementing READ-078 as a prerequisite for EMS-020:
◦ Did the EMS-020 course success rate increase after the READ-078
prerequisite was implemented?
◦ What is the racial/age/gender/disability makeup of the course post
implementation compared to pre implementation?
◦ Does the increased success of students in each protected category support
the implementation if indeed the percentages of students in each group
have changed?
◦ Was there disproportionate impact?
◦ What effect did the implementation have on overall course enrollment?
Post-Implementation Statistical
Analysis: Methodology
An effect size statistic was used to indicate the size of the difference
between course success for students who met and did not meet the
prerequisite
At the time of the study the prerequisite had been enforced from Spring
2011 to Spring 2013 (i.e. 5 primary terms)
The performance of students who had to meet the prerequisite prior to
taking EMS-020 was compared to students who earned a GOR in EMS020 from Fall 2008 to Fall 2010
Post-Implementation
Statistical Analysis: Findings
1.
Did the EMS-020 course success rate increase after the READ-078
prerequisite was implemented?
a.
Yes, students who met the reading prerequisite were statistically significantly
(p < .001) and substantially (ES = .21) more likely to successfully complete
EMS-020 (62%) than students who had not completed the prerequisite (51%).
Success Rate
Course
PreImplementation
#
EMS-020
440
N
%
PostImplementation
#
857 51.3 349
N
%
ES
P
Value
565
61.8
.21
< .001
Post-Implementation
Statistical Analysis: Findings
2. What is the racial/age/gender/disability makeup of the course post
implementation compared to pre implementation?
a.
Gender, ethnicity, age, and disability status were not substantially different
prior to or after the implementation of the READ-078 prerequisite.
Gender
Female
Male
Unknown
Total
Pre-Implementation
#
N
142
16.6
710
82.8
5
0.6
857
100.0
Post-Implementation
%
#
103
18.2
462
81.8
0
0.0
565
100.0
Total
N
245
1,172
5
1,422
%
17.2
82.4
0.4
100.0
Post-Implementation
Statistical Analysis: Findings
To identify disproportionate impact can use the 80% rule or segmentation
modeling (see page 17 in Wurtz & Riggs, 2010 for an example of segmentation
modeling).
80% Rule
.80 * .828 (proportion of males pre-implementation) = .6624 or 66.2%. Is the
proportion of males less than 66.2% post-implementation?
.80 * .166 (proportion of females pre-implementation) = .1328 or 13.3%. Is
the proportion of females less than 13.3% post-implementation?
Gender
Female
Male
Unknown
Total
Pre-Implementation
#
N
142
16.6
710
82.8
5
0.6
857
100.0
Post-Implementation
%
#
103
18.2
462
81.8
0
0.0
565
100.0
Total
N
245
1,172
5
1,422
%
17.2
82.4
0.4
100.0
Post-Implementation
Statistical Analysis: Findings
3. Does the increased success of students in each protected category support
the implementation, if indeed the percentages of students in each group have
changed?
a.
b.
Yes, male students, Hispanic Students, and students 24 years old or younger
were substantially (ES >= .20) and statistically significantly (p < .01) more likely
to successfully complete EMS-020 if they had met the reading prerequisite
than students who had not met the prerequisite.
In addition, female, African American, and Native American students were
slightly more likely to successfully complete EMS-020 post-implementation.
Ethnicity
Asian
African American
Hispanic
Native American
Caucasian
Unknown
Total
Success Rate
Pre-Implementation
Post-Implementation
#
N
%
#
N
%
11
22
50.0
11
24
45.8
30
57
52.6
14
26
53.8
116
304
38.2
121
211
57.3
14
24
58.3
9
14
64.3
258
429
60.1
191
286
66.8
11
21
52.4
3
4
75.0
440
857
51.3
349
565
61.8
ES
-.08
.02
.39
.12
.14
.44
.21
P Value
.783
.919
< .001
.726
.070
.425
< .001
Post-Implementation
Statistical Analysis: Findings
4. Was there disproportionate impact?
◦ Used Classification and Regression Tree (CART) modeling to analyze
disproportionate impact by gender, ethnicity, age, and disability status
Post-Implementation
Statistical Analysis: Findings
5. What effect did the implementation have on overall course enrollment?
◦ The overall course enrollment in EMS-020 did not decrease as a result of the
implementation of the prerequisite
◦ The decline in enrollments and section offerings was due to the statewide budget
cuts and comparable to the cuts that occurred college wide.
Section
1
2
3
4
5
6
Total
Fall
2008
40
40
39
40
37
33
229
Pre-Implementation
Spring
Fall
Spring
2009
2009
2010
40
40
40
39
42
30
45
37
41
36
41
41
35
0
0
0
0
0
195
160
152
Fall
2010
42
43
36
0
0
0
121
Spring
2011
39
38
43
0
0
0
120
Post-Implementation
Fall
Spring
Fall
2011
2012
2012
39
43
39
37
39
37
40
35
29
0
0
0
0
0
0
0
0
0
116
117
105
Spring
2013
42
42
23
0
0
0
107
Sacramento City College
Example
History Department, 2011—2014
17 courses
40-50 sections per semester offered pre-implementation, with 2,300 to
2,700 Census enrollments from 2009-2011
Studied largest-enrolled (U.S. HIST) courses based on Fall 2009 HIST
enrollments and prior English preparation. A department-wide pre-req.
of English 1-level-below-transfer was implemented in Fall 2012, based
on a number of factors, including:
Same SLOs for all SCC HIST courses
Same minimum writing word count requirement for all SCC HIST courses
Sacramento City College
Example (continued…)
Enrollment impact estimation=16% (based on 2009 data)
Able to calibrate estimates against actuals to inform planning for other
departments. How close were we? From Fall 2011 to Fall 2012 =
16.37%
No measurable change is success rates
How does enrollment composition look?
SCC Example (continued…)
Now that prerequisite has been in place
for 3 semesters…
SCC History Department Fall and Spring Fill Rate (Fall 2004 to Fall 2013)
120.0
99.8
100.0
89.9
86.1
83.0
80.0
Percentage 60.0
40.0
20.0
0.0
Fall 2012:
Pre-requisite
implementation
SCC Example (continued…)
Ethnicity before and after prerequisite
35
30
25
African American
Asian
Filipino
20
Latino
MultiNative American
15
Other, non-wht
Pacific Islander
Unknown
10
White
5
0
Fall 2009 Spring 2010 Fall 2010 Spring 2011 Fall 2011 Spring 2012 Fall 2012
Communicating Results and
Evidence-Based Decision Making
Tips for Communicating
Prerequisite Results
Affirm your common interest in supporting and enhancing student
success
Affirm that you are there to support and assist faculty
Affirm that the research is not meant to substitute for faculty’s
professional judgment (i.e. evidence-based decision making)
Making data accessible (i.e. do not use big statistical term, know your
audience)
Recognize and communicate the limitations of the data/research in the
beginning
Consider different approaches for how data is presented
Questions
Resources: College Examples
Cabrillo College
◦ Cabrillo College. (2002). Validation of English 1A as a prerequisite for Psychology 1A. Aptos, CA: Borden, R. C.
Chaffey College
◦ Chaffey College. (2011). Prerequisite Validation Studies: Impact of a Reading Prerequisite on HIST-1, HIST-2,
and HIST-7. Rancho Cucamonga, CA: Institutional Research
◦ Chaffey College. (2010). Philosophy 76 Prerequisite Validation Study: English 1A Prerequisite. Rancho Cucamonga, CA:
Institutional Research.
Sacramento City College
◦ Danenberg, A. (2011). Methodological and Data Considerations for a Communication or Computation Prerequisite
Implementation Study. Sacramento City College: Planning, Research, & Institutional Effectiveness.
Crafton Hills
◦ Wurtz, K. A. (2014). Relationship of the EMS-020 Reading Prerequisite to EMS-020 Course Success. Retrieved March
25, 2014 from
http://www.craftonhills.edu/~/media/Files/SBCCD/CHC/About%20CHC/Research%20and%20Planning/Research%20
Briefs/Academic%20Success%20Studies/2013_July_EMS20_PrereqEval_Post2.pdf
◦ Wurtz, K. A., & Riggs, M. (2010). Prerequisite Validation Study: Examination of Reading as a Prerequisite to EMS-020
(Emergency Medication Technician-I / EMT – Basic). Retrieved November 16, 2012 from
http://www.craftonhills.edu/~/media/Files/SBCCD/CHC/About%20CHC/Research%20and%20Planning/Research%20
Reports/0910_EMS_Read_PrerequisiteStudy.ashx
RP Group for California Community Colleges
◦ RP Group. (2013). SUGGESTIONS FOR CALIFORNIA COMMUNITY COLLEGE INSTITUTIONAL RESEARCHERS
CONDUCTING PREREQUISITE RESEARCH. Retrieved March 25, 2014 from
http://www.rpgroup.org/sites/default/files/RPGroupPreqreqGuidelinesFNL.pdf