The Assessment of Higher Education
Learning Outcomes (AHELO)
OECD’s Study of the Scientific and Practical
Feasibility of Assessing Baccalaureate-Level
Student Learning Outcomes Across Nations
and Institutions
SHEEO Higher Education Policy Conference
August, 2012
Charlie Lenth
clenth@sheeo.org
AHELO Context and
Development
• OECD’s education studies, surveys and analyses
• Near universal recognition of changing education
needs and objectives, particularly at higher levels
• 2006 Council of Education Ministers’ request to plan
postsecondary assessment
• OECD commissioned papers and convened groups of
experts to develop AHELO design
Basic Project Objectives and Design
• Feasibility study—to determine scientific and
practical feasibility of assessing college learning
across nations
• Assess learning across and within the disciplines
or program areas at degree completion
• Minimum five diverse nations with non-scientific
(convenience) sample of 10 institutions each
• Small, random sample of graduating students
with institutions as primary unit of analysis
• Student, faculty, institutional contextual data to
allow meaningful interpretation and analysis
U.S. Involvement in AHELO
• Considerable U.S. role in research design
and development (CAE, Peter Ewell, ETS)
• Invitation from OECD to SHEEO, and then
to SHEEO members
• U.S. participation in Generic Skills Strand
since early 2010 and through many
months of delays and uncertain funding
(and several trips to Paris!)
U.S. Financial Support
for AHELO Participation
• To OECD for assessment development:
Lumina Foundation (early and largest foundation
donor), Spencer Foundation, Teagle Foundation
• For U.S. participation from Hewlett Foundation
and U.S. Department of Education:
– Payment of national participation fees
– Discretionary grant by U.S. Secretary of
Education in September 2011
– In-kind contributions by institutions, states,
NCHEMS and SHEEO
Country Participation in AHELO
Generic Skills
Economics
Engineering
Columbia
Egypt
Finland
Korea
Kuwait
Mexico
Norway
Slovakia
United States
Belgium
Egypt
Italy
Mexico
Netherlands
Russia
Slovakia
Australia
Canada
Columbia
Egypt
Japan
Mexico
Russia
Slovakia
Total countries: 17 plus several EU country observers
Overview--U.S. Participants’ Roles
• National level
– US Dept of Education—member of OECD Education Governing Board
– SHEEO—National Project Manager (NPM) and representative on project
advisory board--Group of National Experts (GNE)
• State level
– SHEEO agency provides project leadership, coordination, and
oversight in Connecticut, Missouri and Pennsylvania
• Institutional level
–
–
–
–
–
–
AHELO Institutional Coordinator and “team”
IR office prepares student/faculty files and institutional survey
Test Administration—recruitment, scheduling, set up, test monitoring
Academic leadership—faculty involvement, organizational support
President, media relations, governing board, others
Anticipated analysis and use of returned assessment data
U.S. Higher Education Institutions
Participating in AHELO Feasibility Study
•
•
•
Connecticut Board of Regents for Higher Education
– Southern Connecticut State University (public regional university;
9,000 undergraduates)
Missouri Department of Higher Education
– Central Methodist University (private university; 3,500 students)
– Missouri State University (public institution; 16,000 undergraduates)
– Truman State University (public institution; 6,000 undergraduates)
– University of Central Missouri (public regional; 12,000 students)
– Webster University (private, not-for-profit; 4,000 undergraduates)
Pennsylvania State System of Higher Education (PASSHE)
– Cheyney University of Pennsylvania (public historically black
university; 1,300 undergraduates)
– Clarion University (public institution; 5,100 undergraduates)
– Edinboro University (public institution; 6,600 undergraduates)
– Lock Haven University (public institution; 5,000 undergraduates)
– Millersville University (public institution; 7,200 undergraduates)
Institutional Sampling Plan
http://www.sheeo.org/misc/ahelo/27samplingmanual.pdf
• AHELO Sampling Manual provides international
guidelines for both student and faculty samples
• Basic components:
1. Institutions provide census of “in scope” students and
faculty to NPM (SPSS.sav or Excel .xls/.xlsx preferred)
2. NPM will draw equal probability samples using
prescribed routines and submit for ACER review
(generally 200 students and 40 faculty)
3. Flexibility for stratified or “augmented” sample design
if justified and documented by institution (e.g., strata
for major program areas, distinguishing student
characteristics, special accommodations, etc.)
4. Review and approval by ACER Sampling Team
Generic Skills
Assessment Framework/Design
• Two rotating performance tasks adapted
from Collegiate Learning Assessment
(CLA) – 90 minutes
• Selected response items from tested
Australian item bank – 30 minutes
• Brief student experience survey
• Faculty/institutional web-based
questionnaires
• All computer-administered on secure
international testing sites
Fieldwork, Analysis and Reporting
• Test administration by U.S. institutions using
secure international testing websites
• Lead scorer trained on international scoring
rubrics recruited and managed five U.S scorers
• Scored assessment results analyzed by ACER and
reported to OECD; later to institutions
• Preparation, review and release of Feasibility
Study findings and recommendations
• Final project conference March 2013
Context and Consequences
• AHELO is part of a larger postsecondary
education agenda with implications for students,
institutions, nations and the global environment.
• Much has already been learned about both
scientific and practical feasibility for assessment
within an international setting.
• OECD and other major organizations will shape
future decisions and financial commitments based
in large part on this study.