Project Evaluation
Don Millard
John Yu
dmillard@nsf.gov
zyu@nsf.gov
March 27, 2012
Guy-Alain Amoussou
gamousso@nsf.gov
Lou Everett
leverett@nsf.gov
March 28, 2012
Handout 1

Before you leave the session please complete the
assessment survey:
http://www.nsflsu.com
Note: please correct this URL in your handouts

Most of the information presented in this workshop represents
the presenters’ opinions and not an official NSF position

Local facilitators will provide the link to the workshop slides at
the completion of the webinar.

Participants may ask questions by “raising their virtual hand”
during a question session. We will call on selected sites and
enable their microphone so that the question can be asked.

Responses will be collected from a few sites at the end of each
Exercise. At the start of the Exercise, we will identify these
sites in the Chat Box and then call on them one at a time to
provide their responses.
3

Learning must build on prior knowledge
◦ Some knowledge correct
◦ Some knowledge incorrect – Misconceptions

Learning is
◦ Connecting new knowledge to prior knowledge
◦ Correcting misconceptions

Learning requires engagement
◦ Actively recalling prior knowledge
◦ Sharing new knowledge
◦ Forming a new understanding
4

Effective learning activities
◦ Recall prior knowledge -- actively, explicitly
◦ Connect new concepts to existing ones
◦ Challenge and alter misconceptions

Active & collaborative processes
◦ Think individually
◦ Share with partner
◦ Report to local and virtual groups
◦ Learn from program directors’ responses
5

Coordinate the local activities

Watch the time
◦ Allow for think, share, and report phases
◦ Reconvene on time -- 1 min warning
◦ With one minute warning, check Chat Box to see if you will be
asked for a response

Ensure the individual think phase is devoted to
thinking and not talking

Coordinate the asking of questions by local
participants and reporting local responses to
exercises
6
The session will enable you to collaborate
more effectively with evaluation experts in
preparing credible and comprehensive
project evaluation plans…. it will not
make you an evaluation expert.
7
After the session, participants should be able to:
 Discuss the importance of goals, outcomes, and
questions in the evaluation process
◦ Cognitive and affective outcomes

Describe several types of evaluation tools
◦ Advantages, limitations, and appropriateness

Discuss data interpretation issues
◦ Variability, alternative explanations

Develop an evaluation plan in collaboration with an
evaluator
◦ Outline a first draft of an evaluation plan
8

The terms evaluation and assessment have many meanings
◦ One definition
 Assessment is gathering evidence
 Evaluation is interpreting data and making value judgments

Examples of evaluation and assessment
◦ Individual’s performance (grading)
◦ Program’s effectiveness (ABET and regional accreditation)
◦ Project’s progress and success (monitoring and validating)

Session addresses project evaluation
◦ May involve evaluating individual and group performance – but in the
context of the project

Project evaluation
◦ Formative – monitoring progress to improve approach
◦ Summative – characterizing and documenting final accomplishments
9

Think about your favorite course. What types of in-class
activities could be called:
◦ Assessment versus Evaluation?
◦ Formative versus Summative Evaluation?

Exercise ---- 6 min
◦ Think individually -------- ~2 min
◦ Share with a partner ----- ~2 min
◦ Report in local group ---- ~2 min



Watch time and reconvene after 6 min
Use THINK time to think – no discussion, Selected local
facilitators report to virtual group
With one minute warning, check Chat Box to see if you will be
asked for a response
10




Handout 2
An in-class quiz could be called assessment.
Using the in-class quiz results to realize the students are not
achieving a learning outcome is an evaluation.
Using your evaluation results to change your approach as the
course progresses is a formative evaluation.
Integrating your evaluation over the whole semester to
determine if you achieved your objective is summative
evaluation.
11
Project Goals, Expected
Outcomes, and Evaluation
Questions

Effective evaluation starts with carefully defined
project goals and expected outcomes

Goals and expected outcomes related to:
◦ Project management
 Initiating or completing an activity
 Finishing a “product”
◦ Student behavior
 Modifying a learning outcome
 Modifying an attitude or a perception
13

Goals provide overarching statements of
project intention
What is your overall ambition?
What do you hope to achieve?

Expected outcomes identify specific observable
or measureable results for each goal
How will achieving your “intention” be reflected by
changes in student behavior?
How will it change their learning and their attitudes?
14

Goals → Expected outcomes

Expected outcomes → Evaluation questions

Questions form the basis of the evaluation process

The evaluation process consists of the collection and
interpretation of data to answer evaluation questions
15


Read the abstract -- Goal statement removed
Suggest two plausible goals
◦ One on student learning
 Cognitive behavior
◦ One on some other aspect of student behavior
 Affective behavior

Focus on what will happen to the students
◦ Do not focus on what the instructor will do



Long Exercise ---- 6 min
◦ Think individually -------- ~2 min
◦ Share with a partner ----- ~2 min
◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min
Use THINK time to think – no discussion, Selected local facilitators
report to virtual group
16
The goal of the project is …… The project is developing computer-based
instructional modules for statics and mechanics of materials. The project
uses 3D rendering and animation software, in which the user manipulates
virtual 3D objects in much the same manner as they would physical objects.
Tools being developed enable instructors to realistically include external
forces and internal reactions on 3D objects as topics are being explained
during lectures. Exercises are being developed for students to be able to
communicate with peers and instructors through real-time voice and text
interactions. The project is being evaluated by … The project is being
disseminated through … The broader impacts of the project are …
Two goals: one for student learning and one for student behavior
Non engineers should substitute:
“Organic chemistry” for “statics and mechanics of materials”
“Interactions” for “external forces and internal reactions”
17
GOAL: To improve conceptual understanding and
processing skills
 In the context of course
◦ Draw free-body diagrams for textbook problems
◦ Solve 3-D textbook problems
◦ Describe the effect(s) of external forces on a solid object orally

In a broader context
◦
◦
◦
◦
◦
Handout 3
Solve out-of-context problems
Visualize 3-D problems
Communicate technical problems orally
Improve critical thinking skills
Enhance intellectual development
18
GOAL: To improve
◦
◦
Self- confidence
Attitude about engineering as a career
19
Write SMART outcomes for your goals

•
•
•
•
•

Specific
Measurable
Attainable
Realistic
Timely
Achieving outcomes brings you closer to your goal, e.g.
◦ Goal: My students will be life long learners
◦ Outcomes (some of these lack SMART components):
 reading an unassigned technical article,
 attending a professional society meeting
 attending a non required seminar or talk

Outcomes provide observable effects that goals are
being achieved
20
Write expected measurable outcomes for each of the
following goals:

◦
◦



Improve the students’ understanding of the fundamental
concepts in statics (cognitive)
Improve the students’ self confidence (affective)
Long Exercise ---- 6 min
◦ Think individually -------- ~2 min
◦ Share with a partner ----- ~2 min
◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min
Use THINK time to think – no discussion, Selected local facilitators
report to virtual group
Non engineers may substitute:
“Organic chemistry” for “statics”
21
Understanding of the fundamentals
◦



By the end of the class, 70% of the students will be able to:
Correctly draw freebody diagrams of 2D truss structures
Correctly write Newton’s laws when given a FBD
Describe the effects on member force when one angle in a
2D truss is changed
Self-Confidence
◦
By the end of the semester:




Handout 4
30% of the class volunteers to show the solution to any
homework problem on the board
Self reported test anxiety reduces to 50% of the initial amount
80% will say the class was easier than they expected it would be
50% report they are excited about taking the follow-on course
22
Understanding of the fundamentals
◦
◦
Are the students better able to describe the effects of
changing some variable in a simple problem
Are the students better able to describe the effects of
changing some variable in a simple problem as a result of the
intervention
Self-Confidence
◦
◦
Do the students express more confidence in their solutions
Do the students express more confidence in their solutions as
a result of the intervention
23
Questions
“Hold-up your virtual hand” and you will
be called upon after we unmute your
mike.
BREAK
15 min
BREAK
1 min warning
Tools for Evaluating
Learning Outcomes

Surveys
◦ Forced choice or open-ended responses

Concept Inventories
◦ Multiple-choice questions to measure conceptual understanding

Rubrics for analyzing student products
◦ Guides for scoring student reports, tests, etc.

Interviews
◦ Structured (fixed questions) or in-depth (free flowing)

Focus groups
◦ Like interviews but with group interaction

Observations
◦ Actually monitor and evaluate behavior
Olds et al, JEE 94:13, 2005
NSF’s Evaluation Handbook
28
Observations
Surveys
 Time & labor intensive
 Efficient
 Accuracy depends on
 Inter-rater reliability must
subject’s honesty
be established
 Difficult to develop reliable
 Captures behavior that
and valid survey
subjects are unlikely to
 Low response rate
report
threatens reliability, validity
 Useful for observable
& interpretation
behavior
Olds et al, JEE 94:13, 2005
29

Use interviews to answer these questions:
◦
◦
◦
◦
◦
What does program look and feel like?
What do stakeholders know about the project?
What are stakeholders’ and participants’ expectations?
What features are most salient?
What changes do participants perceive in themselves?
The 2002 User Friendly Handbook for Project
Evaluation, NSF publication REC 99-12175
30



Originated in physics -- Force Concept Inventory (FCI)
Several are being developed in engineering fields
Series of multiple choice questions
◦ Questions involve single concept
 Formulas, calculations or problem solving skills not required
◦ Possible answers include distractors
 Common errors -- misconceptions

Developing CI is involved
◦
◦
◦
◦
Identify misconceptions and detractors
Develop, test, and refine questions
Establish validity and reliability of tool
Language is a major issue
31

Pittsburgh Freshman Engineering Survey
◦ Questions about perception
 Confidence in their skills in chemistry, communications,
engineering, etc.
 Impressions about engineering as a precise science, as a
lucrative profession, etc.

Validated using alternate approaches:
◦ Item analysis
◦ Verbal protocol elicitation
◦ Factor analysis

Compared results for students who stayed in engineering
to those who left
Besterfield-Sacre et al , JEE 86:37, 1997
32

Levels of Intellectual Development
◦ Students see knowledge, beliefs, and authority in
different ways
 “ Knowledge is absolute” versus “Knowledge is
contextual”

Tools
◦ Measure of Intellectual Development (MID)
◦ Measure of Epistemological Reflection (MER)
◦ Learning Environment Preferences (LEP)
Felder et al, JEE 94:57, 2005
33

Suppose you were considering an existing tool (e. g., a
concept inventory) for use in your project’s evaluation
of learning outcomes

What questions would you consider in deciding if the
tool is appropriate?

Long Exercise ---- 6 min
◦ Think individually -------- ~2 min
◦ Share with a partner ----- ~2 min
◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min
Use THINK time to think – no discussion
Selected local facilitators report to virtual group



34

Nature of the tool
◦ Is the tool relevant to what was taught?
◦ Is the tool competency based?
◦ Is the tool conceptual or procedural?

Prior validation of the tool
◦
◦
◦
◦

Has the tool been tested?
Is there information concerning its reliability and validity?
Has it been compared to other tools?
Is it sensitive? Does it discriminate between a novice and an expert?
Experience of others with the tool
◦ Has the tool been used by others besides the developer? At other
sites? With other populations?
◦ Is there normative data?
Handout 5
35
Questions
Hold up your “virtual hand” to ask a
question.
Question or
Concept
1
2
3
-
Percent w ith Correct
No. of Students
Answ er
Com parision Experim ental Com parision Experim ental
Group
Group
Group
Group
25
24
25
-
30
32
31
-
29%
34%
74%
-
23%
65%
85%
-
Data suggest that the understanding
of Concept #2 increased
38

Data suggest that the understanding of Concept #2
increased

One interpretation is that the intervention caused the
change

List some alternative explanations
◦ Confounding factors
◦ Other factors that could explain the change




Long Exercise ---- 6 min
◦ Think individually -------- ~2 min
◦ Share with a partner ----- ~2 min
◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min
Use THINK time to think – no discussion
Question or
Selected local facilitators report
to virtual group
Concept
1
2
3
No. of Students
Com parision
Group
Experim ental
Group
25
24
25
30
32
31
Percent w ith Correct
Answ er
Com parision Experim ental
Group
Group
29%
34%
74%
23%
65%
85%






Students learned the concept out of class (e. g., in another
course or in study groups with students not in the course)
Students answered with what they thought the instructor
wanted rather than what they believed or “knew”
An external event distorted the pretest data
The instrument was unreliable
Other changes in the course and not the intervention was
responsible for the improvement
The characteristics of groups were not similar
Handout 6
40

Data suggest that the understanding of the concept
tested by Q1 did not improve

One interpretation is that the intervention did cause
a change that was masked by other factors
Think about alternative explanations
How would these alternative explanations
(confounding factors) differ from the previous list?


No. of Students
Question or
Concept
Com parision
Group
Experim ental
Group
1
2
3
25
24
25
30
32
31
Percent w ith Correct
Answ er
Com parision Experim ental
Group
Group
29%
34%
74%
23%
65%
85%
Evaluation Plan





List the topics that need to be addressed in the
evaluation plan (a.k.a. summarize)
Long Exercise ---- 6 min
◦ Think individually -------- ~2 min
◦ Share with a partner ----- ~2 min
◦ Report in local group ---- ~2 min
Watch time and reconvene after 6 min
Use THINK time to think – no discussion
Selected local facilitators report to virtual group
43
Name & qualifications of the evaluation expert

◦







Get the evaluator involved early in the proposal development phase
Goals, outcomes, and evaluation questions
Instruments for evaluating each outcome
Protocols defining when and how data will be collected
Analysis & interpretation procedures
Confounding factors & approaches for minimizing their
impact
Formative evaluation techniques for monitoring and
improving the project as it evolves
Summative evaluation techniques for characterizing
the accomplishments of the completed project.
Handout 7
44

Workshop on Evaluation of Educational Development
Projects
◦ http://www.nsf.gov/events/event_summ.jsp?cntn_id=108142&org=NSF

NSF’s User Friendly Handbook for Project Evaluation
◦ http://www.westat.com/westat/pdf/projects/2010ufhb.pdf

Online Evaluation Resource Library (OERL)
◦ http://oerl.sri.com/

Field-Tested Learning Assessment Guide (FLAG)
◦ http://www.wcer.wisc.edu/archive/cl1/flag/default.asp

Student Assessment of Their Learning Gains (SALG)
◦ http://www.salgsite.org/

Science education literature
45
Identify the most interesting, important, or surprising
ideas you encountered in the workshop on dealing
with project evaluation

Take ---- 4 min
◦ Think individually -------- ~2 min
◦ Report in local group ---- ~2 min

Watch time and reconvene after 4 min

Use THINK time to think – no discussion, Selected local facilitators
report to virtual group
46
Questions
Hold up your “virtual hand” to ask a
question.
Proposal & Award Policies & Procedures Guide
http://www.nsf.gov/pubs/policydocs/pappguide/nsf11001/

To download a copy of the presentation- go to:
http://www.nsflsu.com

Please complete the assessment survey-go to:
http://www.nsflsu.com