College of Engineering
Outcome Assessment:
A Seven-Year Study
Professor David G. Meyer
School of Electrical and Computer Engineering
Introduction
• A proven method for satisfying the
Accreditation Board for Engineering and
Technology (ABET) “Criterion 3” requirements
is the formulation of outcomes specific to “core”
courses in a curriculum, which are tied to the
program outcomes.
• This presentation summarizes a 7-year study
on formulating and assessing course-specific
outcomes
School of Electrical and Computer Engineering
Basic Questions Addressed
• Formulation of content-specific learning outcomes that
can be consistently and quantitatively assessed
• Formulation of effective outcome assessment
instruments along with mechanisms to determine
outcome demonstration thresholds
• Formulation of grading strategies that incorporate
outcome demonstration thresholds yet produce results
consistent with prior (accepted) grading practices (i.e.,
course GPA in range of approximately 2.5 – 2.8 on a
4.0 scale)
• Formulation of outcome remediation strategies that are
both fair and efficient
School of Electrical and Computer Engineering
Edict from the ABET Visitors
“We want to see evidence of failing grades
assigned for cases in which students
[who would otherwise be passing but]
failed to successfully demonstrate one or
more course outcomes.”
Interpretation: We can’t just say that, because a
student earned a total score above the “course passing
threshold”, that he/she has successfully demonstrated
all the course outcomes. Also, application of outcome
assessment should produce some form of meaningful
and measurable difference.
School of Electrical and Computer Engineering
Major Issues
• Balance
– fairness to students
• providing students with a sufficient number of
opportunities to demonstrate course outcomes
• ensuring that students’ own work is used as the
basis for outcome demonstration success
– overhead for instructor
• keeping the incremental workload associated with
outcome assessment and tracking to a minimum
• keeping the outcome assessment process
“contained” within a given term/semester
School of Electrical and Computer Engineering
Major Issues
• Evaluation Instruments
– exams (whole or question subsets)
– quizzes (written/oral)
– homework assignments
– labs
– papers/projects/presentations
• Passing Thresholds
– static (what absolute value?)
– dynamic (what algorithm?)
– consistent with grading cutoff (A-B-C-D)
thresholds
School of Electrical and Computer Engineering
Major Issues
• Assigning course grades consistent with
proven prior practice, yet reflecting meaningful
application of outcome assessment thresholds
• Achieving consistency
– semester-to-semester
– professor-to-professor
– course-to-course
– school-to-school
– institution-to-institution
School of Electrical and Computer Engineering
Sample Course-Specific Outcomes
ECE 270 – Introduction to Digital System Design
1. an ability to analyze static and dynamic behavior of digital
circuits
2. an ability to represent Boolean functions in standard forms, to
map and minimize them, and to implement them as
combinational logic circuits
3. an ability to use a hardware description language to specify
combinational logic circuits, including various “building blocks”
such as decoders, multiplexers, encoders, and tri-state buffers
4. an ability to design and implement arithmetic logic circuits
5. an ability to analyze, design, and implement sequential circuits
and use a hardware description language to specify them
6. an ability to design and implement a simple computer based
on combinational and sequential building blocks
KEY: Modularization of course material
School of Electrical and Computer Engineering
Sample Outcome Assessment Exam
Primary Assessment of Outcome 2 for CN: 0000-X
X 1. Most minimization methods are based on a generalization of…
2. The fact that any combinational logic circuit can be realized using only...
3. The complement of this function is represented by the ON SET...
4. The dual of this function is represented by the ON SET...
5. This expression is an example of...
6. The XOR property listed below that is NOT true is...
7. A NOR gate is logically equivalent to...
8. A circuit consisting of _ is logically equivalent to...
9. This circuit exhibits the following type of hazard...
10. The number of prime implicants is...
11. The number of essential prime implicants is...
12. The number of non-essential prime implicants is...
X 13. The cost of a minimal sum of products realization of this function is...
X 14. The cost of a minimal product of sums realization of this function is...
X 15. The fewest number of 2-input NAND gates needed to realize this function is...
X 16. The fewest number of 2-input NOR gates needed to realize this function is...
17. The fewest number of 2-input OD NAND gates needed to realize this function is...
18. The number of pull-up resistors required is...
19. The simplest realization of the function depicted by this circuit is...
X 20. The minimum SoP function realized by this circuit is...
X 21. The ON-SET of the function realized by this circuit is...
X 22. The minimum PoS function realized by this circuit is...
X 23. Realization of this same function using only 2-input NOR gates would require...
X 24. Steady-state analysis of the function realized by this circuit predicts...
25. The dynamic behavior of the circuit does NOT match the steady-state analysis...
Number of correct answers: 15 (out of 25 questions)
School of Electrical and Computer Engineering
Sample Outcome Demonstration Report
ECE 270 Course Outcome Report for: 0000-X, enrolled in Lab Division 5
Outcome 1: An ability to analyze static and dynamic behavior of digital circuits
Outcome 1: Primary Assessment Score = 9/ 25 = 36.0%
Passing Threshold = 52.0%
Outcome 1: Sorry - this outcome was not demonstrated
Outcome 2: An ability to represent Boolean functions in standard forms...
Outcome 2: Primary Assessment Score = 15/ 25 = 60.0%
Passing Threshold = 45.0%
Outcome 2: Congratulations - this outcome was successfully demonstrated
Outcome 3: An ability to use a hardware description language to specify combinational...
Outcome 3: Primary Assessment Score = 15/ 25 = 60.0%
Passing Threshold = 50.0%
Outcome 3: Congratulations - this outcome was successfully demonstrated
Outcome 4: An ability to design and implement arithmetic logic circuits
Outcome 4: Primary Assessment Score = 12/ 25 = 48.0%
Passing Threshold = 54.0%
Outcome 4: Sorry - this outcome was not demonstrated
Outcome 5: An ability to analyze, design, and implement sequential circuits...
Outcome 5: Primary Assessment Score = 12/ 25 = 48.0%
Passing Threshold = 52.0%
Outcome 5: Sorry - this outcome was not demonstrated
Outcome 6: An ability to design and implement a simple computer
Outcome 6: Primary Assessment Score = 15/ 25 = 60.0%
Passing Threshold = 46.0%
Outcome 6: Congratulations - this outcome was successfully demonstrated
You will need to retake the outcome(s) you have missed on the Final
School of Electrical and Computer Engineering
Outcome Demonstration Summary Report
ECE 270 PRIMARY OUTCOME ASSESSMENT REPORT
Outcome 1 -- Avg Score: 67.4% Threshold: 52.0% Took: 205/206 = 99.5%
Passed: 179/205 = 87.3% Failed: 26/205 = 12.7%
Outcome 2 -- Avg Score: 59.0% Threshold: 45.0% Took: 203/206 = 98.5%
Passed: 167/203 = 82.3% Failed: 36/203 = 17.7%
Outcome 3 -- Avg Score: 64.9% Threshold: 50.0% Took: 203/206 = 98.5%
Passed: 175/203 = 86.2% Failed: 28/203 = 13.8%
Outcome 4 -- Avg Score: 67.6% Threshold: 54.0% Took: 203/206 = 98.5%
Passed: 171/203 = 84.2% Failed: 32/203 = 15.8%
Outcome 5 -- Avg Score: 66.2% Threshold: 52.0% Took: 200/206 = 97.1%
Passed: 171/200 = 85.5% Failed: 29/200 = 14.5%
Outcome 6 -- Avg Score: 64.0% Threshold: 46.0% Took: 200/206 = 97.1%
Passed: 169/200 = 84.5% Failed: 31/200 = 15.5%
Summary of Outcome Demonstration:
Number of students who demonstrated 0 outcome(s) on the primary assessments: 2
Number of students who demonstrated 1 outcome(s) on the primary assessments: 5
Number of students who demonstrated 2 outcome(s) on the primary assessments: 9
Number of students who demonstrated 3 outcome(s) on the primary assessments: 12
Number of students who demonstrated 4 outcome(s) on the primary assessments: 22
Number of students who demonstrated 5 outcome(s) on the primary assessments: 51
Number of students who demonstrated 6 outcome(s) on the primary assessments: 105
School of Electrical and Computer Engineering
Refinements
• Early (discarded) attempts
– fixed passing threshold (60%) on weighted sum of
(selected) lab, homework, and exam scores
– fixed passing threshold (60%) on primary assessment,
remediation homework, and final assessment
• Main problems
– basically everyone “passed” – strategy did not prove
to be effective
– impossible to ensure that students’ own work was
being evaluated on labs and (remediation) homework
School of Electrical and Computer Engineering
Refinements
• Restricting outcome assessment to exams only
– fixed passing threshold (60%) on primary and final
assessments
– use of “conditional failure” grade for those who would
otherwise be passing, with 3rd attempt for remediation
of failed outcome(s) the following semester
• Main problems
– fixed passing threshold became a significant factor:
hard (“impossible”) to write exams that produce a
predictable mean/distribution
– lack of containment within term (“overhead leakage”)
– grade inflation due to allowing final assessment to be
used as replacement score for all N outcomes
School of Electrical and Computer Engineering
Refinements
• Incorporation of dynamic thresholds
– used on each primary and final assessment,
calculated as mean – standard deviation, with
bounded range of 40% to 60% (choice based on
empirical study of data sets in multiple courses over
multiple terms)
– must pass at least 50% of N (total) outcomes on the
primary assessments to qualify for a passing grade
– final assessment can be used for remediation and/or
improvement (replacement) of score on up to N/2
outcomes
– self-contained (no “overhead leakage” to following
semester)
School of Electrical and Computer Engineering
Refinements
• Controlling grade inflation
– added limited opportunity for grade improvement on
final assessment
• effective outcome score = weighted average of
primary and final assessments (60-40)
• number of outcomes re-taken limited to half
• must have passing grade to qualify for final
– grade inflation associated with grade improvement on
final reduced by about an order of magnitude
School of Electrical and Computer Engineering
Refinements
• Reducing outcome-induced failures
– students with grades as high as “B” were still failing
course due to failure to demonstrate all course
outcomes on two attempts (primary + final)
– award grade of “I” or “E” if otherwise have passing
grade after final but still need to satisfy one or more
outcomes
– remediation exam(s) taken subsequent semester of
enrollment at scheduled exam times with rest of class
– if fail this third remediation attempt on any given
outcome, receive failing grade and re-take course
– some “overhead leakage” reintroduced, but minimal
School of Electrical and Computer Engineering
Results
Percent Successful
ECE270 Outcome Demonstration Success Rate
100
90
80
70
60
50
40
30
20
10
0
Primary
Final
4
5
6
Trial Number
7
Trial 4 – Static thresholds, N replacement, 3 tries
Trial 5 – Dynamic thresholds, N/2 replacement
Trial 6 – Dynamic thresholds, no replacement
Trial 7 – Dynamic thresholds, weighted average
School of Electrical and Computer Engineering
Results
Percent Successful
ECE362 Outcome Demonstration Success Rate
100
90
80
70
60
50
40
30
20
10
0
Primary
Final
4
5
6
Trial Number
7
Trial 4 – Static thresholds, N replacement, 3 tries
Trial 5 – Dynamic thresholds, N/2 replacement
Trial 6 – Dynamic thresholds, no replacement
Trial 7 – Dynamic thresholds, weighted average
School of Electrical and Computer Engineering
Results
Course Grade Point Average
4
GPA (4.0 Scale)
3.5
Desired range, based on
proven prior practice
3
2.5
ECE 270
ECE 362
2
1.5
1
0.5
0
4
5
6
Trial Number
7
Trial 4 – Static thresholds, N replacement, 3 tries
Trial 5 – Dynamic thresholds, N/2 replacement
Trial 6 – Dynamic thresholds, no replacement
Trial 7 – Dynamic thresholds, weighted average
School of Electrical and Computer Engineering
Lessons Learned – A “Delicate Balance”
• Assigning course grades consistent with proven
prior practice
• Providing incentives for students to successfully
demonstrate outcomes, yet not causing grade
inflation and/or excessive instructor workload
• Establishing reasonable, meaningful thresholds
for outcome demonstration success that are
decoupled from “exam difficulty”
• Determining a fair level of pass/fail “filtering”
based on outcome demonstration success
School of Electrical and Computer Engineering
Some Remaining Questions
• What is an appropriate amount of outcomebased “filtering” for various kinds of courses?
– currently averaging about 10-15% in lower
division courses
– is this too much, too little, or about right?
• What is an appropriate balance between
number of attempts allowed to demonstrate an
outcome, and the incremental workload
associated with managing this effort?
– currently offer three attempts
– is this too much, too little, or about right?
School of Electrical and Computer Engineering
Summary and Conclusions
• What may appear to make the most sense
“on paper” may not be the “best” strategy
when all factors are considered
• Finding an “optimal” strategy is non-trivial
• Different kinds of courses (e.g., “contentintensive” vs. “design-intensive”) require
different outcome assessment instruments,
thresholds, and remediation strategies
School of Electrical and Computer Engineering
Senior Design Outcomes and Assessment
1. An ability to apply knowledge obtained in earlier
coursework and to obtain new knowledge necessary
to design and test a system, component, or process to
meet desired needs – design component report
2. An understanding of the engineering design process –
individual lab notebook
3. An ability to function on a multidisciplinary team –
project specific success criteria (functionality)
4. An awareness of professional and ethical
responsibility – professional component report
5. An ability to communicate effectively, in both oral and
written form – formal design review, final written
report, and final presentation
School of Electrical and Computer Engineering
Outcome Tracking Results for
Digital Systems Senior Design
2 Tablet PCs
per team
100.0%
90.0%
80.0%
Cohort Average
70.0%
HTML
notebooks
initiated
60.0%
50.0%
TCSP sessions
initiated
Outcom
1 Tablet PC
per team
Outcome 1 – design component report
Outcom
Outcom
Outcom
Outcom
40.0%
Outcome 2 – individual lab notebook
30.0%
Outcome 3 – project specific success criteria
20.0%
Outcome 4 – professional component report
10.0%
Outcome 5 – technical communication skills
0.0%
Spr-03
Fall-03
Spr-04
Fall-04
Spr-05
Fall-05
Spr-06
Fall-06
Spr-07
Fall-07
Spr-08
School of Electrical and Computer Engineering
Summary and Conclusions
• Have the “basic questions” been answered?
– Formulation of content-specific learning outcomes
that can be consistently and quantitatively assessed
YES
– Formulation of effective outcome assessment
instruments along with mechanisms to determine
outcome demonstration thresholds YES
– Formulation of grading strategies that incorporate
outcome demonstration thresholds yet produce
results consistent with prior (accepted) grading
practices (i.e., course GPA in range of
approximately 2.5 – 2.8 on a 4.0 scale) YES
– Formulation of outcome remediation strategies that
are both fair and reasonably efficient YES
School of Electrical and Computer Engineering
Discussion Questions
Should outcome assessment strategies be
more widely deployed across the ECE
curriculum?
What are some successful assessment
methods used in industry that could
translate to the academic environment?
School of Electrical and Computer Engineering