Increasing Assessment Effectiveness in
a Time of Decreasing Budgets
Ken Stanton
Conference - Spring 2009
Engineering
Education
PhD in Engineering Education
at Virginia Tech
 First cohort began Fall 2008
 16 total students
 Coursework in engineering,
education, and engineering
education
 Teaching and research
assistantships available
 Graduate Recruitment Days
every March and October–
travel grants available!
 www.enge.vt.edu
Engineering
Education
Introduction – The Problem
 ABET EC2000 standards are a lot of work for engineering
departments
 Purposes: assessment and accountability
 Problems: some faculty less familiar with assessment,
requiring more and more of programs
 Solutions: faculty development, assessment experts
 Economic downturn affects education budgets
 Problem: difficult to implement the solutions above
 Enrollment, and therefore class size, increases
 Time and resources limited
 Any solutions: must be efficient, use what we have, be simple
and easy to implement
Engineering
Education
Introduction – Proposed Solution
 Address the need to increase assessment effectiveness
from three angles
1. Utilize lean assessment techniques
2. Implement a process that maximizes intrinsic motivation
3. Take advantage of efficiency of assessment methodology
Lean
Assessment
Techniques
Increasing
Intrinsic
Motivation
Increased
Assessment
Effectiveness
Assessment
Method
Efficiency
Engineering
Education
Innovative Solution – Lean
Assessment
 Goal of assessment
Typical process for an assignment
Teaching toward
learning
objectives
Assignment
created for
student
Student
completes
assignment
Grade and
feedback given to
student
Engineering
Education
Innovative Solution – Lean
Assessment
Typical assessment process
Teaching toward
learning
objectives
Assessment
created for
student
Use to improve
teaching and
assessments
Student
completes
assessment
Feedback given
to student, may
be graded
Collect and analyze
evidence of learning
Engineering
Education
Innovative Solution – Lean
Assessment
 Definition
 simple, efficient, and effective assessment techniques
 based in the principle of lean manufacturing
 productive and simple methods
 still attain useful data
 Methods
 Some existing
 Angelo and Cross (1993)
 Some modified
 Some novel
Engineering
Education
Innovative Solution – Lean
Assessment
Technique:
Time Requirement: Purpose:
Description:
Notes:
Rubrics (for
various activities)
MED;
LOW if shared
development
Categorize
important
assessment and
grading criteria; set
up structure to
summarize
performance;
provide clear
expectations to
students
Important criteria
are categorized and
assigned point
values in a table,
then performance
levels (e.g. novice,
expert) described
One-minute
problem summary
LOW
Evaluate student
understanding of
assigned problems
and underlying
concepts
At the end of a
problem, the
student takes one
minute to explain
what major
concepts were used
and what was
learned
Many resources
online for predefined or custom
rubrics;
coordinating with
other faculty can
decrease time
investment and
increase
consistency of
course/material
Good to prompt
students with
leading question to
answer in problem
summary
 Use to assess teaching and learning
Engineering
Education
Innovative Solution – Lean
Assessment
Technique:
Time Requirement: Purpose:
Description:
Individual project
reports (in team
project)
MED
Assess and improve When team reports
individual reportare required, have
writing abilities
each student submit
individual report at
least one week
preceding
Oral exams
LOW and can
reduce time
Assess oral abilities
and allow for
deeper probing of
key topics; mimic
business meeting
Students present
and discuss project
instead of written
exam/project, oneon-one with
instructor; oral
questions provide
richer evidence of
knowledge and
learning; grade is
given before
leaving meeting
Notes:
Reviewing 3-5
times as many
reports can be time
consuming; a
rubric should be
used and specific
aspects of reports
focused on
A form of authentic
assessment,
abilities are tested
in a more realistic
manner that closely
reflects real-world;
time can be less
than administering
and grading exams/
projects
Engineering
Education
Innovative Solution –
Implementation
 These assessment methods alone do not solve the
problem
 Consider motivation theory in implementation
 Self-Determination Theory (SDT) by Deci & Ryan (2000)
has 3 needs
1. Competence – confidence, self-efficacy
2. Autonomy – choice that aligns with identity / values
3. Relatedness – connection to others identify with
 Satisfaction of these leads to internalization of motivation –
move from extrinsic to intrinsic
Engineering
Education
Innovative Solution –
Implementation
 Competence
 Lacking due to inexperience with assessment
 Lean assessment techniques – simple, effective
 Faculty helping each other
 Autonomy
 Challenged by requirements being handed down
 Helped somewhat by flexibility of ABET criteria
 Use assessment to address faculty’s teaching concerns
 Relatedness
 Can take opportunity to connect with students through
feedback
Engineering
Education
Innovative Solution –
Implementation
1. Review program educational objectives (PEOs),
program learning outcomes, and course outcomes
established for the program and course(s) of interest.
2. Interview faculty for input on assessment and what they
would like to improve in their teaching and students’
performance. Review current assignments and
assessments being used in their course(s) of interest.
3. Identify lean assessment techniques that
A. are good for measuring the outcomes from #1
B. meet goals that faculty name as important to them
C. can complement/replace existing assignments and activities
Engineering
in the course(s).
Education
Innovative Solution –
Implementation
4. Briefly illustrate how the lean assessment techniques
match the faculty member’s goals and can be used
alongside or in place of existing course materials.
5. Assist with familiarization and use of the assessments,
if necessary, and encourage sharing the expectations
and motives with students.
 Likely to be effective seeing as studies found faculty
generally hold positive views of assessment
Engineering
Education
Innovative Solution –
Assessment Methodology
 Many times we develop courses by deciding material
then assessments
 Less efficient and less productive
 Wiggins & McTighe (2005) coined “Backward Design”
 Work from learning objectives to ways to measure them
(assessments) to how to teach it
 Ensures efforts are goal-oriented and assessments can be
used for accreditation
 Using this process can increase efficiency of teaching
and assessment efforts
Engineering
Education
Introduction – Proposed Solution
 Address the need to increase assessment effectiveness
from three angles
1. Utilize lean assessment techniques
2. Implement a process that maximizes intrinsic motivation**
3. Take advantage of efficiency of assessment methodology
Lean
Assessment
Techniques
Increasing
Intrinsic
Motivation
Increased
Assessment
Effectiveness
Assessment
Method
Efficiency
Engineering
Education
Future work
 Dissertation study
 From a motivational standpoint, how do engineering faculty
members engage in assessment of student learning?
 Also look at ways to integrate assessment into the
culture of teaching and learning
Engineering
Education
THANK YOU
QUESTIONS?
Engineering
Education