Overview of Types
of Measures
Margaret Kasimatis, PhD
VP for Academic Planning & Effectiveness
Location in Assessment Cycle
IDENTIFY
SPECIFIC
OUTCOMES
DETERMINE
PRACTICES USED
TO ACHIEVE
OUTCOMES
GATHER
EVIDENCE
ARTICULATE
MISSION/
GOALS
RECOMMEND
ACTIONS
REVIEW &
INTERPRET
RESULTS
Measures Within the Context
of Program Assessment
• For program assessment, measurement tools
should capture student learning that occurs as
a result of the program curriculum
• Many measurement tools can be used for
multiple levels of assessment
– But how you use them differs
• Some assessment tools are not appropriate
for program-level assessment
Categories of Measures
• Direct Measures
– Look at student work products or
performances that demonstrate level of
learning
• Indirect Measures
– Capture students’ perceptions of their
learning and the educational environment
that supports learning
Categories of Measures
• Direct Measures
– Published, standardized tests (e.g., GRE Subject Test,
ETS Major Field Test)
– Locally developed tests
– Systematic evaluation of student work (papers,
presentations, creative work, performances)
• May or may not be embedded within courses
• Usually involves scoring rubrics
• Indirect Measures
– Published surveys
– Locally developed surveys and interviews
– Alumni surveys
Properties of Good
Assessment Techniques
• Reliable – internally consistent; consistent across raters
• Valid – measures what it is supposed to; appropriate
• Actionable – results point reviewers toward challenges
to address (and how to address them)
• Efficient and cost effective in time and money
• Interesting and meaningful – people care about the
results and are willing to act on them
• Convergence – multiple lines of evidence point to the
same conclusion
Evaluating Measures: Direct
PUBLISHED TESTS
PROS
• Provide direct evidence of student
mastery of content
• Some are designed specifically to
assess major programs
• Generally highly reliable
CONS
• May be difficult to motivate students to
perform at their best level
• May not align with program outcomes
• Validity established (within a
specific context)
• Often focus more on content knowledge
than higher-order skills
• Usually easy to implement and
obtain results
• Can be expensive
• Norms/comparisons available
Evaluating Measures: Direct
LOCALLY DEVELOPED TESTS
PROS
• Provide direct evidence of student
mastery of content or skills
• More flexible in terms of content
and format; easier to align with
program outcomes
• If embedded in courses, student
motivation is higher
• Faculty are more likely to be
interested in and use results
CONS
• Likely to be less reliable
• Validity unknown
• Norms/comparisons not available
• Can take several iterations (and several
years) to work out the “bugs”
• Scoring tests and tabulating results can
be cumbersome
Evaluating Measures: Direct
EVALUATION OF STUDENT WORK
PROS
• Provides direct evidence of student
mastery of content or skills
• If embedded in course, student
motivation is higher
• Faculty are more likely to be
interested in and use results
• Data collection is usually
unobtrusive to students
CONS
• Requires time to develop, conduct
training, implement
– Creating flexible rubric, at appropriate for
program-level assessment can be tricky
• Validity unknown; takes time to
establish reliability
• Requires faculty trust that the program
will be assessed, not individual
instructors
• Norms/comparisons not available
Evaluating Measures:
Indirect
PUBLISHED SURVEYS
PROS
CONS
• Minimal effort to implement and
tabulate results
• Provides indirect evidence of student
learning
• Can be administered to a large
group of respondents
• May not be aligned with program
outcomes
• Demonstrated reliability and
validity
• Potential for biased results if sample is
not representative
• Can address a variety of outcomes
• Can be expensive
• Norms/comparisons available
Evaluating Measures:
Indirect
LOCALLY DEVELOPED STUDENT SURVEYS & INTERVIEWS
PROS
CONS
• Flexible in terms of content and format;
easy to align with program outcomes
• Provide indirect evidence of student
learning
• Usually have face validity
• Their validity depends on the quality of
questions and response options
• Can add open-ended questions that
allow you to flesh out quantitative results
– More actionable
• Can be administered to a large group of
respondents
• Can address a variety of outcomes
• Relatively easy to implement
• Potential for biased results if sample is
not representative
• Can be time-consuming to construct,
implement, and tabulate results
• Norms/comparisons not available
• Open-ended responses can be difficult
and time-consuming to analyze
Evaluating Measures:
Indirect
ALUMNI SURVEYS
PROS
CONS
• Same advantages as student surveys • Many of the same disadvantages of
student surveys
• Can gather more “direct” evidence
– It’s particularly difficult to get a good
than current student surveys
response rate
– E.g., employment, enrollment in
graduate programs
• Can ask questions of alumni that are
not appropriate for current students
– E.g., the extent to which the program
prepared them for their career
• The timing can be tricky
– Alumni should be far enough out to see
an impact of the program on their
life/career but not so far out that the
program they experienced is very different
from current one
A Word on Embedded
Assessment
• Various types of measurement tools can
be embedded within courses
• Only carefully constructed measures,
used in certain types of courses, are
appropriate for program-level assessment
– Must go beyond individual course content
– Some should occur in upper-level courses that are
taken only after several other courses in the major
QUESTIONS?