Direct vs Indirect Assessment
of Student Learning:
An Introduction
Dr. Sheila Handy, Chair Business Management and Co-Chair University
Assessment Committee
Sponsored by University Assessment Committee
February 24, 2014
Assessment
• “The most important purpose of assessment should be not
improvement or accountability but their common aim: Everyone
wants students to get the best possible education. Everyone wants
them to learn what’s most important. A college’s mission statement
and goals are essentially promises that the college is making to its
students, their families, employers, and society. Today’s world needs
people with the attributes we promise. We need skilled writers,
thinkers, problem-solvers and leaders. We need people who are
prepared to act ethically, to help those in need, and to participate
meaningfully in an increasingly diverse and global society. Imagine
what the world would be like if every one of our graduates achieved
the goals we promise them! We need people with those traits, and
we need them now. Assessment is simply a vital tool to help us
make sure we fulfill the crucial promises we make to our students
and society. “
Linda Suskie, 2010
Continuous Improvement of Student Learning
Use Results
for
Improvement
Articulate
Goals
Collect
Assessment
Information
Offer
Programs,
Services and
Initiatives
Setting the Stage
• Middle States Commission on Higher Education
• “…each program should employ direct
measures of student learning…”
• “…each stated learning outcome should be
aligned to a measure, a performance goal, and
a finding. Measures can be used to evaluate
more than a single learning outcome.”
Assessment Measures - Definitions
Direct
• Direct examination or
observation of
student knowledge,
skills, attitudes or
behaviors to provide
evidence of learning
outcomes.
Indirect
• Perceived extent or
value of learning
experiences
Assessment Measures - Examples
Direct
• Exam
▫ Standardized
▫ Locally developed
▫ Embedded questions
• Juried review
• Portfolios with rubrics
Indirect
• Surveys
▫ Students
▫ Alumni
• Focus Groups
• Student Records
▫ Class Attendance
Short Table Activity
• Introductions
• Envelopes
▫ Match Assessment Measure and Description
▫ Sort into Direct and Indirect Measures
Standardized Tests
• Licensing Exams
▫ Education - Praxis, PAPA
▫ Nursing - NCLEX
▫ Accounting - CPA Exam
• Major Field Test
▫ Developed by ETS
▫ Administered in a proctored environment as paper
and pencil or online
MFT subjects
▫
▫
▫
▫
▫
▫
Biology
Business
Chemistry
Computer Science
Criminal Justice
Literature in English
▫
▫
▫
▫
▫
Math
Physics
Political Science
Psychology
Sociology
Standardized Tests
Advantages




Carefully developed
Highly reliable
Professionally scored
Nationally normed
Disadvantages
▫ If tests don’t match LOs of
program, scores will be low
▫ MC questions that test facts
– faculty may focus on
higher order skills
▫ May be expensive
▫ Students may not take
seriously
Locally Developed Exam
Advantages
Disadvantages
• Can be tied-in to specific LOs
• Can be administered as part of
regular semester testing
• Can use publisher developed
questions
• Reliability and generalizability
not known
• Time required to prepare
questions
• Scoring may take a long time
• Norms not available
Embedded Questions
Advantages
• Can be conducted as part of
regular student testing
• Does not require collection of
additional data
• Can be used for both grading
and assessment
Disadvantages
• Faculty may think they are
being assessed
• Takes time to develop and
grade
Embedded Questions
• One way to assess learning without faculty
“angst”
▫ Add assessment questions as a quiz administered
at the beginning of the next course in the
sequence
▫ Effective if different faculty teach multiple sections
of an introductory course
Juried Review
• Suitable for assessing
▫ Performing arts projects
▫ Written assignments
▫ Oral presentations
Juried Review
Advantages
Disadvantages
• Provides experience and
practice for student performers
• Can be formative and
summative – indicating “value
added”
• Can also introduce students to
the standards they will be
expected to achieve
• Achieving consensus with
regard to the evaluation rubric
• May not be sufficient to
determine if students have
achieved the LOs set by the
faculty
Portfolios With Rubrics
• Suitable for many majors
▫
▫
▫
▫
▫
Art
English
Computer Science
Business
Education
Portfolios With Rubrics
Advantages
• Provides students with
evidence to submit to potential
employers
• Tracks student work over time
• Can assess higher levels of
learning
• Allows students to reflect on
their learning
Disadvantages
• Faculty must agree on
assessment rubric
• Time consuming
• Storage of evidence
Case Study
Your goal is to develop a plan to assess this
student learning outcome for the Engineering
Program for the upcoming academic year. This
plan should include two direct measures.
Wrap Up
• Resources
▫ Materials in Folders
▫ University Assessment Committee
▫ Assessment Consulting Team
▫ Website