HEQCO LEARNING
OUTCOMES ACTIVITIES
MARY CATHARINE LENNON
SENIOR RESEARCH ANALYST
HIGHER EDUCATION QUALITY COUNCIL OF ONTARIO
PRESENTATION TO:
OISE/HEQCO/TCU SYMPOSIUM – KEY ISSUES IN ONTARIO HIGHER EDUCATION REFORM: A
POLICY RESEARCH SYMPOSIUM
MAY 16, 2013
Informing the Future of Higher Education
MEASURING THE VALUE OF A
POSTSECONDARY EDUCATION
Conference and book
• What do we mean by ‘quality’
education?
• What do courses and programs
promise to deliver and do they
succeed?
• What do we know about
improving learning outcomes,
and is reform possible?
2
Informing the Future of Higher Education
CONTRIBUTORS INCLUDE…..
• Background and national initiatives
–
–
–
–
Tuning - Robert Wagenaar (University of Groningen)
Degree profiles - Holiday Hart McKiernan (Lumina Foundation)
Australia - Michael Gallagher (Group of Eight)
Ontario - Virginia Hatchette (Postsecondary Education Quality Assessment
Board),
• Measurement activities
– NSSE - Jillian Kinzie (Indiana University)
– CLA - Roger Benjamin (Council for Aid to Education)
– AHELO - Diane Lalancette (Organisation for Economic Co-operation and
Development)
• Organisational activities
– Cultures of assessment - Lorne A. Whitehead (University of British
Columbia
– Institutional activities - Jeana Abromeit (Alverno College),
3
Informing the Future of Higher Education
LEARNING OUTCOMES PROJECTS
• HEQCO initiated a number of projects to examining
generic, sector-level, and discipline-specific
learning outcomes
– Assessment of Higher Education Learning
Outcomes (AHELO): Civil Engineering Strand
– Collegiate Learning Assessment (CLA)/College
Collegiate Learning Assessment (CCLA)
– ‘Tuning’: Identifying and Measuring SectorBased Learning Outcomes in Postsecondary
Education
• Learning Outcomes Assessment Consortium
Informing the Future of Higher Education
ASSESSMENT OF HIGHER EDUCATION
LEARNING OUTCOMES (AHELO)
• Feasibility Study
– AHELO seeks to determine if there are
internationally appropriate standard tests which
measure what final-year undergraduate students
know and can do.
– Primary questions
• Is it possible to have agreement on expected learning
outcomes?
• Is it possible to implement a test across cultures and
languages?
• Are the assessments valid and reliable?
5
Informing the Future of Higher Education
AHELO
• Three strands:
– Generic: assess analytical skills, written
communication, leadership ability, etc.
– Discipline specific: Economics and Civil Engineering
Pilots. Assess context knowledge and students’
capacity to use or apply the knowledge
– Context: Surveys of student, faculty and institution
characteristics
6
Informing the Future of Higher Education
CANADA IS 1 OF 9 COUNTRIES
PARTICIPATING IN ENGINEERING
Australia
Canada
Columbia
Egypt
Japan
Mexico
Russia
Slovakia
United Arab Emirates
7
Informing the Future of Higher Education
9 OF 10 ONTARIO UNIVERSITIES
PARTICIPATED
8
Informing the Future of Higher Education
TRANSMISSION OF RESULTS
• 3 OECD reports: background, implementation and analysis,
conclusions and next steps
• ‘Institution Reports’: Institutions receive reports detailing
the performance of the cohort of students from their
individual institution
• ‘Jurisdictional Reports’: outline the performance of Ontario’s
Civil Engineering programs compared to the Ontario and
international aggregate
– Australia, Japan and Canada have agreed to share data for
comparative purposes
– Current analysis is primarily descriptive, and includes a regression
analysis indicating some of the factors which seem to be
associated with student success on the AHELO assessment
9
Informing the Future of Higher Education
PRELIMINARY FINDINGS FROM AHELO
• In Ontario
– There is considerable interest from programs in jurisdictional and
international comparison
– Institutions have been prompted to reflect on their curriculum design and
delivery, and on the assessment techniques they employ
– Faculty members have been prompted to reflect on their own methods
and re-evaluate what they require of students
• At the OECD
– Theoretically and practically possible to assess common student learning
outcomes internationally
– Conversations continue on the ultimate goals, purpose and possibilities of
such information
– Determining the value of the data is key to assessing the success of the
feasibility study
10
Informing the Future of Higher Education
COLLEGIATE LEARNING ASSESSMENT
BACKGROUND
• Currently in 700 institutions in the US and
around the world
• Examines critical thinking, analytic reasoning,
problem solving, and written communication
skills gained from first to final year.
• Provides information on the ‘value-add’ of the
institution
11
Informing the Future of Higher Education
THE ASSESSMENT
• Administered to students at the beginning of first- and
end of fourth-year to test abilities in several cognitively
demanding activities:
– analytic reasoning and evaluation
– problem solving
– writing effectiveness and mechanics
– articulating complex ideas
– examining claims and evidence
– supporting ideas with reasons and examples
– sustaining a coherent discussion
– using standard written English
12
Informing the Future of Higher Education
SAMPLE TASKS
Sample CLA tasks
Measure
Analytic
Writing
Description
Time on task
Make an
argument
Take and justify a position on 45 minutes
an issue
Critique an
argument
Evaluate an argument for
how well reasoned it is
35 minutes
Complete a task using a set
of provided materials
90 minutes
Performance task
• CLA + introduced for Fall 2013.
• Modified system and updated questions
• Multiple choice questions
• Provides student level data on skill sets
13
Informing the Future of Higher Education
CLA/CCLA
IN ONTARIO
• 8 projects across universities and colleges
– Primarily examining programs rather than entire
institution
– Sample of approximately 100 – 200 students per
institution
– Testing began in spring 2012, 5 institutions
completed in Fall 2012
14
Informing the Future of Higher Education
INITIAL OUTCOMES AND PRELIMINARY
FINDINGS
• Challenges associated with student recruitment
and administration
• Mixed results on value of data
– Useful in benchmarking against peer institutions in
generic skills
– Needs to be ‘Canadianised’
• Mixed results on recommendation to use the CLA
– Other less expensive/cumbersome tests
– Institutional activities may be more appropriate
15
Informing the Future of Higher Education
LEARNING OUTCOMES CONSORTIUM
• Durham College - Student success e-portfolio: Essential
employability skills project
• Humber College - Develop rubrics to assess skills
• George Brown College - Building capacity to measure
essential employability skills
• University of Toronto - Development of analytic rubrics
for learning outcomes assessment
• University of Guelph - A pilot project for the
development of an online learning outcome assessment
strategy
• Queen’s University - Learning outcomes assessment and
program improvement
16
Informing the Future of Higher Education
For more information please contact
Mary Catharine Lennon
mlennon@heqco.ca
17
Informing the Future of Higher Education