Case Study #9
Jackie Adams: Evaluating a Federally Funded
Faculty Training Program
Melody Murphy
AIL 606
The University of Alabama
Part One: Creating an Evaluation
Plan
• Jackie Adams has a instructional
design position with a large
university for a federally funded
project.
– Her main responsibilities were to work
with technical subject matter experts in
the development, delivery, and
evaluation of in-service faculty education.
• The name of the project Jackie was
working on was the Advanced
Manufacturing Technology
Education (AMTE).
Part One: Continued
• The goal of the AMTE project was
one of many projects funded by the
Advanced Technology Education
(ATE) program.
– Its goal was to improve science and
engineering technician education at the
undergraduate and secondary school
levels.
– Also, it focused on curriculum
development, instructional materials
development, teacher/faculty
enhancement, and/or student
recruitment.
Main Goal
• Its main goal was to focus on teacher
enhancement as a means of
advancing technology education.
– The rationale behind AMTE was to
provide educators with state-of-the-art
knowledge in the technical disciplines, so
that their students would in turn benefit
from their teachings.
• Over a three-year grant, the AMTE
project was to provide faculty
development to 100 science, math,
and engineering technology
educators each year.
Progress
• Jackie had been working on the
project for about four months when
Ray DeMilo (her boss) asked her to
develop an evaluation plan for the
project.
• In order for them to receive funding
for the project an evaluation plan
was going to be required to assess
the performance of their project.
Jackie’s Plan for the Evaluation
Plan
1. Give it to her boss.
2. Send it to the funding agency.
3. File it with the university grants
office.
4. Use it to conduct the evaluation.
Jackie’s Planning Evaluation
Notes
1. Why Evaluate?
2. Evaluate What?
3. How should the evaluation be
conducted?
4. Who should be involved?
5. When should it be done?
Preliminary Analysis Questions
for Part One – Question #1
Based on Jackie’s questions, what does
she need to do to complete the
evaluation.
First of all Jackie decided that on why the
program must be evaluated. It should be
evaluated for the ongoing funding of the
program and to measure whether or not learning
is taking place in the workshops
Preliminary Analysis Questions
for Part One – Question #2
How can Jackie determine whether short- and longterm outcomes are being achieved? What evaluation
data does she need to collect?
Short-term
•
Jackie can conclude by receiving the surveys
from the faculty about how they are using what
they have learned in the workshops to conclude
that by what they have learned they can use in
their classrooms.
Long-term
•
Jackie can conclude from the surveys that if
the faculty can continue or use the information
now that they can use then the faculty can use it
for long range planning in their classes.
Preliminary Analysis Questions
for Part One – Question #3
How should Jackie design the
instruments for this evaluation?
Based on the material taught, she wants to know
what the learners have gained from the
workshops in order to report to the team that is
reviewing her project in order to keep the grant
money.
Part Two: Conducting a MetaEvaluation
Conducting the Evaluation
• The funding agency sent out a sixmember team to review the project’s
progress and impact.
• The team would report the success of
the project to key administrators at
the university and the funding
agency.
– The team included four post-secondary
educators with a background in science,
engineering, and technology and two
engineering practitioners.
Team Review
• During the two-day meeting, Hank
spent the entire first day with Jackie.
• He reviewed the progress and impact
of the project.
• At the end of their meeting, Hank
felt that confident that he had
enough information to start the
review.
Preliminary Analysis Questions
Question #1
Part A: Questions Related to the Evaluation Plan
What are the main evaluation
questions Jackie appears to be
addressing?
Why – why evaluate the project?
What – what to evaluate in the project? What should
be measured?
How – How should it be evaluated?
Who – Who should be evaluated?
When – When should it be evaluated? Periodically?
Preliminary Analysis Questions
Question #2
What are the strengths and
weaknesses of the evaluation design
for answering the questions?
Strengths – the evaluation plan answers why
evaluate the objectives (reason for evaluation),
purpose, when it should be done which is
periodically.
Weaknesses – How it should be done. It says
documentation but it does not specify the
documentation used or the learners in Jackie’s
evaluation plan.
Preliminary Analysis Questions
Question #3
Given the goals stated in the ATME
proposal, are these questions
sufficient?
Yes, it does give the basic why, what, how, who, and
when.
Preliminary Analysis Questions
Question #1
Part B: Questions Related to Instrumentation
How appropriate are the instruments
for the goals of the evaluation?
I question the validity. This is a lot of Jackie’s opinion
on the evaluation plan. I would probably recommend
incorporating interviews and surveys to the
evaluation plan.
Preliminary Analysis Questions
Question #2
Are the evaluation questions answered
in enough detail, or are other
instruments needed?
• I felt like there could have been more detail, for
example, when stated “determine how
disciplined and effective the organization’s
operations are.” How will these be determined?
Preliminary Analysis Questions
Question #3
What, if anything, would you change
about the instruments and why?
I would add a survey to be given to the faculty/staff.
For reinforcement for what is stated in the
evaluation plan.
What are the implications for
instructional design practice in this
case?
• The instructional design of high-tech
areas using computers.
– For example computer numerical control,
programmable logic controllers, robotics,
electromechanical controls, lasers, solid
modeling, and rapid prototyping in the
workshops conducted by Jackie.
As an instructional designer,
what did I learn?
• In order to keep your
program or project funded
it is important to regularly
evaluate it.
– Ask questions such as:
–Why, what, how, who, and
when.
My Questions
1. How important do you think
evaluation plans are to instructional
design projects?
2. What types of instruments would
you recommend for Jackie to use in
helping with the evaluation of her
project?