Tracking and Assessing Instructional Effectiveness
Using EvalTools®
Fong Mak, Ph.D., P.E.
PETE
& C 2010
Febuary 2010
1/20
Objectives:
•
Gain knowledge on effective tracking and assessing
teaching at course level
•
Understand how the assessment process can be
automated
Febuary 2010
2/20
Common Questions:
•
How do I meet state standard?
•
How do I know my teaching is ok?
Febuary 2010
3/20
Secondary Standards – Math Snapshot:
Program
outcomes
Course
outcomes
Febuary 2010
4/20
Planning-Teaching-Assessment Process
Example: Math101 Algebra Ia…one-term activities
Math 101
Learning
Plan
Teaching and Learning
FCAR
Promise of what to be learned
Building Course Portfolio
Evaluation of what delivered and learned
Febuary 2010
5/20
Planning-Teaching-Assessment Process
Example: Learning Plan
Learning
Plan
Teaching and Learning
FCAR
•Course Description
•Course Outcomes
•Course Outline
•Course Assessment Methods
•Grading Policy
•Course Outcome Assessment Criteria
•Relationship to Student Outcomes, Course Outcomes, and Objective Evidence
Learning Plan Details
Febuary 2010
6/20
Planning-Teaching-Assessment Process
Example: Building Course Portfolio
Learning
Plan
Teaching and Learning
FCAR
•Lessons --- reading material, lecture notes, etc.
•Assignments --- homework, projects, oral presentation, etc.
Key Assignments
Identify assignments as objective evidence to justify if course outcomes
can be met
Use justifiable language to associate with course outcomes
Sample of key assignments
Febuary 2010
7/20
Planning-Teaching-Assessment Process
Example: Faculty Course Assessment Report
Learning
Plan
Teaching and Learning
• Course Description
• Course Outcomes
• Course Outline
• Course Assessment Methods
• Grading Policy
• Course Outcome Assessment Criteria
• Relationship of Student Outcomes,
Course Outcomes and Objective
Evidence
Febuary 2010
FCAR
• Course Description
• Course Outcomes
• Grade Distribution
• Relationship of Student Outcomes,
Course Outcomes and Objective
Evidence
• Reflection on Course Delivery
• Old Action Items
• New Action Items
• Course Outcomes Assessment
• Student Outcomes Assessment
8/20
Sample of FCAR…
What is FCAR?
Consists of…
Course outcomes
Course description
Pre-requisites, Co-requisites
Grade distribution
Relationship to student outcomes, course objectives,
and objective evidence
Old action items
Reflection on course delivery
New action items
Course outcomes assessment
Student outcomes assessment
Febuary 2010
9/20
Performance Vector (EAMU) for
Teaching/Learning Effectiveness
Excellent (E) – student applies knowledge with virtually no conceptual
or procedural errors
90% - 100%
Adequate (A) – student applies knowledge with no significant
conceptual errors and only minor procedural errors
75% - 90%
Minimal (M) – Student applies knowledge with occasional conceptual
errors and only minor procedural errors
60% - 75%
Unsatisfactory (U) – Student makes significant conceptual and/or
procedural errors when applying knowledge
0% - 60%
Febuary 2010
10/20
Calculation of EAMU vectors…
Course outcome
Key assignment
EAMU for each
outcome
* generated by EvalTools®
Febuary 2010
11/20
Heuristic Rule 1…
Red flag – any performance vector with an average
below 3.3 (out of 5) and a level of unsatisfactory
performance shown as U that exceeds 10%
Yellow flag – any performance vector with an average
below 3.3 or a level of U that exceeds 10%, but not both
Green flag – any performance vector with an average
that is at least 4.6 and no indication of U performance
No flag – any performance vector that does not fall into
one of the above categories
Febuary 2010
12/20
Course Outcomes Assessment…
consistent performance criteria
* generated by EvalTools®
Justification of key assignment used
EAMU tabulated for each outcome
Febuary 2010
13/20
Performance Vector Table (PVT)…
Correlation of course outcomes
to student outcome produces
EAMU vector at program level!
* generated by EvalTools®
PVT is evaluated through use of two heuristic rules
Febuary 2010
14/20
Meeting Standards
Program Level Assessment: correlating all FCARs
• 2.1.A1.A Model and compare values of
irrational numbers
• 2.1.A1.B.
• 2.1.A1.C.
• 2.1.A1.D.
• 2.1.A1.E
• 2.1.A1.F
• 2.2.A1.C
FCAR1
FCAR2
FCARn
Febuary 2010
15/20
Course/Student Outcomes Mapping…
* generated by EvalTools®
Provide means to gauge strength and weaknesses of how each
student outcomes are met by a metric of courses
Febuary 2010
16/20
Conversion of Assessment Plan to
Assessment Report…
heuristic rule 2 applies here
Febuary 2010
17/20
* generated by EvalTools®
Heuristic Rule 2…
Below Expectations – when there are
Any red or yellow flags raised in the final year of data, or if
Multiple red flags are raised prior to the final year of data, or if
A majority of red and yellow flags are raised prior to the final year of
data
Exceeding Expectation – when there are a majority of green
flags raised
Meeting Expectations – the default condition when the
application of the heuristic rules does not fall into either of the
above two categories
Febuary 2010
18/20
Excerpt from an Assessment Report…
heuristic
rule 2
Febuary 2010
* extracted from “A heuristic approach to assessing program outcomes using performance vectors” by John Estell
19/20
Excerpt from an Assessment Report…
Febuary 2010
* extracted from “A heuristic approach to assessing program outcomes using performance vectors” by John Estell
20/20