IST@IUB
R561: Evaluation in the Instructional Development Process (Section 7463)
Spring 2011
“We improve human learning and performance in diverse contexts”
Class Time:
Thursdays at 4:00 – 6:45 pm (1/13 – 4/28) (No class – 2/24 and 3/17)
Location:
Education 2275
Instructor:
Dr. Yonjoo Cho (Education 2232)
Office Hours:
Wednesday at 1:30 pm – 3:00 pm (appointment requested for other times)
Communications: choyonj @indiana.edu; 812-856--8144
Teaching Assistant: Ingu Kang (ingukang@indiana.edu) and Minkyoung Kim (kimmink@indiana.edu)
Course Access:
http://oncourse.iu.edu; https://www.indiana.edu/~istr561/cho11spring/
Course Description
R561: Evaluation centers on evaluation as an integral element of the instructional technology (IT),
human performance technology (HPT), and human resource development (HRD) processes. Training,
performance improvement, and HR professionals need information about the impact and effectiveness
of programs in terms of: (1) the degree to which program results achieve intended objectives, (2)
whether results are desirable, and (3) evidence that results are achieved in a cost-effective manner.
Principles and methods for evaluating instructional and performance improvement programs during the
stages of analysis, design, development, implementation, and utilization are covered. Frameworks and
models for planning and conducting evaluations are also discussed and applied.
Course Objectives
At the completion of the course, students will be able to:
1. Understand basic concepts and terminology associated with instructional, performance
improvement, and HRD evaluation.
2. Explain the purposes and uses of evaluation within different instructional, performance
improvement, and HRD environments.
3. Use qualitative and quantitative data collection techniques in evaluation activities.
4. Analyze and interpret evaluation data and information.
5. Report the results of evaluation activities.
Course Outline
The course is divided into six units to reflect the importance of major evaluation perspectives.
Unit 1: Basics of Evaluation
Develop common understandings of basic concepts and definitions, underlying principles and
theories, and perspectives of the field of evaluation.
1
Units 2 to 5: Four Levels of Evaluation
Address four well known perspectives for evaluating instruction and performance improvement:




Gain knowledge of purposes, concepts, theories, cases, and major issues in using different
levels of evaluation in varied types of organizations.
Gain skills of developing instruments required for data collection in each level of evaluation,
including interview questions, questionnaires, test items, and observation checklists.
Gain skills of analyzing collected data using both qualitative and quantitative methods.
Gain field work experience by applying knowledge and skills to various organizations including
business, education, military, non-profit organizations, and simulated situations.
Unit 6: Evaluation Synthesis
Discuss evaluation frameworks and ethics that are applicable to a variety of organizational settings.
This is developed through the student group’s evaluation project and a review of recent innovative
approaches to evaluation.
Course Assignments
Weekly Participation
Students are expected to: (1) post discussion questions on required readings (i.e. one on each
reading) by Thursdays at 9:00 am and the same number of comments on other students’ questions
by the end of Sunday each week in the Oncourse Forum; (2) actively participate in class activities
such as evaluating a movie (12 Angry Men); and (3) honestly respond to my check-in asking “where
are you at right now?” either at the beginning of each class or near the end of class to see if
students are doing okay in class. The instructor and TAs will assign grades by reviewing the quality
AND the quantity of the participation.
Unit Exercises
Students will complete three unit exercises including levels one to three evaluation: survey, test
development, and transfer scenario. Each exercise will be worth 10 points, assessing students’
abilities to analyze and synthesize what they have learned in each level of evaluation. Peer reviews
are part of unit exercises evaluating other groups’ surveys and test development.
Case Study Presentation
Groups of students will select a case study related to their contexts and prepare for a presentation.
In their presentations, students are expected to show their mastery of basic concepts and
knowledge of evaluation. The case study presentation should include: (1) title (2) table of contents
(3) introduction (4) explanation of the evaluation process in the case (5) results of the case (6)
conclusion/implications (7) discussion points and (8) references. Your presentation is limited to 30
minutes while you effectively use the next 10 minutes after for a Q&A session. Your group is
evaluated both for the content you deliver and for the effectiveness of your presentation.
Final Evaluation Project
Students will work as a team to complete an organization-specific evaluation project (e.g., IST’s
Distance Education). Students are expected to utilize three unit exercises and add on the impact
level of evaluation to this capstone project in order to display their ability to apply what they’ve
learned to a real organization. Teams are required to do a pilot presentation on their draft of the
report in the second to last week (April 21) and based on feedback they will have one more week
2
to finalize their evaluation report. Instructor will evaluate progress in their final report.
Individual (Personal) Evaluation Project
Students are required to develop an individual (personal) evaluation project. Their evaluation
topics can be anything to practice an individual level of evaluation to see progress based on the
intended goal. Students work in pairs to monitor progress throughout the semester. We will have
three informal presentations (proposal, mid-term, and final) on students’ progress in their
individual evaluation project. For their final presentation, they are required to write a one-page
report on what they’ve done in their individual evaluation project.
Reflection Paper
Individual students are required to write a (single-spaced, 3 page-limit) reflection paper. This endof-class reflection paper reflects on the lessons learned from a perspective of what learning and
changes have occurred with respect to the field and team work experiences as well as class
activities and an individual evaluation project experience. The reflection paper should include: (1)
title (2) introduction (3) key points and anecdotes (4) suggestions and (5) conclusions.
Grading Criteria and Due Dates
Assignment
1. Weekly Participation
2. Unit Exercises 2 to 4 (10 x 3)
(Peer Reviews included)
3. Case Study Presentation
4. Final
Proposal
Project
Mid-term Presentation
Pilot Presentation
Final Report
5. Individual
Proposal
Evaluation
Mid-term Presentation
Project (10)
Final Presentation
6. Reflection Paper
Unit
Individual
Points
10
Team
30
Group
Team
10
10
20
Individual/Pair
Individual
10
10
100 points
Due
weekly
2/10
3/3
3/31
your choice
2/3
3/24
4/21 (a draft)
4/28
1/20
3/24
4/28
5/1 (Sun)
Grading Policy
The following grading policy has been adopted for graduate courses in the School of Education
(http://www.indiana.edu/~bulletin/iu/educ_grad/2005-2007/policies.shtml#grading).
A (95%)
= Outstanding achievement. Unusually complete command of the course content.
A- (90%)
= Excellent achievement. Very thorough command of course content.
B+ (85%)
= Very good achievement. Thorough command of course material.
B (80%)
= Good achievement. Solid, acceptable performance.
B- (75%)
= Fair achievement. Acceptable performance.
C+ (70%) = Not wholly satisfactory. Marginal performance on the course requirements.
C (65%)
= Marginal achievement. Minimally acceptable performance on course assignments.
C - (60%) = Courses with a grade of C- or lower may not be counted in graduate programs.
3
Be certain that you understand the evaluation criteria before you begin any of the projects. The
evaluation guideline sheets are the checklists for content to be included in all project assignments.
Plagiarism and Original Work
We expect that you will turn in original work for every part of every deliverable in this course. We also
expect that you make every effort to acquaint yourself with both the IU Code of Student Rights,
Responsibilities and Conduct, the concept of plagiarism (start with the required departmental tutorial
"Understanding Plagiarism"), and the ways in which you must both credit the work of others and avoid
presenting that work as your own (start with the resources from the Campus Writing Program and
reference the APA style guide).
Both individual and group/team project work containing plagiarized material will be awarded a grade
of F. At the discretion of the instructor, your original work may be turned back to the group/team for
correction of the problem before a specified deadline and re-graded for a grade equivalent to or lower
than the grade the project would have otherwise received. If your individual/team work is discovered
to be plagiarized or to contain plagiarized material, you will receive a failing grade for the course.
These policies cover written and graphical work, and all work assigned in the course.
Required Textbook
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.).
San Francisco: Berrett-Koehler.
Required Readings
Week 2
Cho, Y., Park, S., Jo, S. J., Jeung, C.-W., & Lim, D. H. (2009). Developing an integrative evaluation
framework for e-learning. In V. C. X. Wang (Ed.), Handbook of research on e-learning applications
for career and technical education (pp. 707-722). Hershey, PA: IGI Global.
Shrock, S. A., & Geis, G. L. (2010). Evaluation. In J. L. Moseley & J. C. Dessinger (Eds.), Handbook of
improving performance in the workplace, vol. 3: Measurement and evaluation (pp. 185-209). Silver
Spring, MD: International Soceity for Performance Improvement.
Week 2 - optional
Arthur Jr., W. A., Bennett, W. Jr., Edens, P. S., & Bell, S. T. (2003). Effectiveness of training in
organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology,
88(2), 234-245.
Christie, C. A., & Fleischer, D. N. (2010). Insight into evaluation practice: A content analysis of designs
and methods used in evaluation studies published in North American evaluation-focused journals.
American Journal of Evaluation, 31(3), 326-346.
Holton, E. F. III. (1996). The flawed four-level evaluation model. Human Resource Development
Quarterly, 7(1), 5-21.
Week 3
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). The four levels; evaluating reaction. Evaluating training
programs: The four levels (3rd ed.) (pp. 21-41). San Francisco: Berrett-Koehler.
Lee, S. H. (2006). Constructing effective questionnaires. In J. A. Pershing (Ed.), Handbook of human
performance technology (3rd ed.) (pp.760-779). San Francisco: Pfeiffer.
4
Week 3 – optional
Maher, J. B. Jr., & Kur, C. E. (1983, June). Constructing good questionnaires. Training and
Development Journal, 100-110.
Ritter, L. A., & Sue, V. M. (2007). Introduction to using online surveys. New Directions for evaluation,
115, 5-14.
Week 4
Gilmore, E. R. (2006). Using content analysis in human performance technology. In J. A. Pershing (Ed.),
Handbook of human performance technology (3rd ed.) (pp.819-836). San Francisco: Pfeiffer.
Thomas, M. N. (2006). Quantitative data analyses. In J. A. Pershing (Ed.), Handbook of human
performance technology (3rd ed.) (pp. 837-872). San Francisco: Pfeiffer.
Week 4 – optional
Lehrer, J. (2010, December 13). The truth wears off: Is there something wrong with the scientific
method? The New Yorker, 52-57.
Moss, P. A., Phillips, D. C., Erickson, F. D., Floden, R. E., Lather, P. A., & Schneider, B. L. (2009).
Learning from our differences: A dialogue across perspectives on quality in educational research.
Educational Researcher, 38(7), 501-517.
Week 5
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating learning. Evaluating training programs: The
four levels (3rd ed.) (pp. 42-51). San Francisco: Berrett-Koehler.
Morrison, G. R., Ross, S. M., Kemp, J. E., & Kalman, H. K. (2007). Instructional objectives. Designing
effective instruction (5th ed.) (pp. 102-129). Hoboken, NJ: John Wiley & Sons.
Week 5 – optional
Morrison, G. R., Ross, S. M., Kemp, J. E., & Kalman, H. K. (2007). Developing evaluation instruments.
Designing effective instruction (5th ed.) (pp. 264-305). Hoboken, NJ: John Wiley & Sons.
Week 6
Shrock, S., & Coscarelli, W. (2007). Create cognitive items; create rating instruments. Criterionreferenced test development (2nd ed.) (pp. 121-194). San Francisco: Pfeiffer.
Week 7: No class! I will present my study at the 2011 AHRD Conference in Chicago.
Week 8
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating behavior. Evaluating training programs: The
four levels (3rd ed.) (pp. 52-62). San Francisco: Berrett-Koehler.
Marrelli, A. F. (2010). Data collection. In R. Watkins & D. Leigh (Eds.), Handbook of improving
performance in the workplace, vol. 2: Selecting and implementing performance interventions (pp.
792-816). Silver Spring, MD: International Society for Performance Improvement.
Week 8 – optional
Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research.
Personnel Psychology, 41, 63-105.
5
Hutchins, H. M., Burke, L. A., & Berthelsen, A. M. (2010). A missing link in the transfer problem?
Examining how trainers learn about training transfer. Human Resource Management, 49(4), 599618.
Week 9
Pershing, J. L. (2006). Interviewing to analyze and evaluate human performance technology. In J. A.
Pershing (Ed.), Handbook of human performance technology (3rd ed.) (pp.780-794). SF: Pfeiffer.
Pershing, J. A., Warren, S. J., & Rowe, D. T. (2006). Observation methods for human performance
technology. In J. A. Pershing (Ed.), Handbook of human performance technology (3rd ed.) (pp. 795818). SF: Pfeiffer.
Week 9 – optional
Rodriguez, H., Trainor, J., & Quarantelli, E. L. (2006). Rising to the challenges of a catastrophe: The
emergent and prosocial behavior following Hurricane Katrina. The Annals of the American Academy
of Political and Social Science, 604, 82-101.
Week 10: Spring Break – Enjoy!
Week 11: Individual and final evaluation projects: Mid-term evaluation
Week 12
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating results; implementing four levels. Evaluating
training programs: The four levels (3rd ed.) (pp. 63-74). San Francisco: Berrett-Koehler.
Phillips, J. A., & Phillips, P. P. (2010). The business of program evaluation: ROI. In J. L. Moseley & J. C.
Dessinger (Eds.), Handbook of improving performance in the workplace: Measurement and
evaluation (pp. 219-239). Silver Spring, MA: ISPI.
Week 12 – optional
Parry, S. B. (1996). Measuring training’s ROI. Training & Development, 50(5), 72-77.
Russ-Eft, D., & Preskill, H. (2005). In search of the Holy Grail: Return on investment evaluation in
human resource development. Advances in Developing Human Resources, 7, 71-85.
Week 13
Kaplan, R. S., & Norton, D. P. (1992). The balanced scorecard-Measures that drive performance.
Harvard Business Review, 70(1), 71-79.
Parsons, J. G. (1997). Values as a vital supplement to the use of financial analysis in HRD. Human
Resource Development Quarterly, 8, 5-13.
Week 14
Patton, M. Q. (2008). Utilization-focused evaluation: Processes and premises. In Utilization-focused
evaluation (4th ed.) (pp. 559-582). Thousand Oaks, CA: SAGE.
Schwandt, T. A. (2007). Expanding the conversation on evaluation ethics. Evaluation and Program
Planning, 30, 400-403.
6
Week 14 - optional
Chouinard, J. A., & Cousins, J. B. (2009). A review and synthesis of current research on cross-cultural
evaluation. American Journal of Evaluation, 30(4), 457-494.
Greene, J. C. (2007). Contested spaces: Paradigms and practice in mixed methods social inquiry. In
Mixed methods in social inquiry (pp. 49-65). San Francisco: Jossey-Bass.
Liu, Y., Combes, J. G., Ketchen, D. J., & Ireland, R. D. (2007). The value of human resource
management for organizational performance. Business Horizons, 50, 503-511.
Ostrom, E., & Nagendra, H. (2006). Insights on linking forests, trees, and people from the air, on the
ground, and in the laboratory. PNAS, 103(51), 19224-19231.
Case Studies
Education
Keim, J., Warring, D. F., & Rau, R. (2001). Impact of multicultural training on school psychology and
education students. Journal of Instructional Psychology, 28(4), 249-252.
Kellogg, D. L., & Smith, M. A. (2009). Student-to-student interaction revisited: A case study of working
adult business students in online courses. Decision Science Journal of Innovative Education, 7, 433456.
Lee, Y.-F., Altschuld, J. W., & Hung H.-L. (2008). Practices and challenges in educational program
evaluation in the Asia-Pacific region. Evaluation and Program Planning, 31, 368-375.
Mabry, L. (2008). Consequences of No Child Left Behind on evaluation purpose, design, and impact. In
T. Berry & R. M. Eddy (Eds.), Consequences of No Child Left Behind for educational evaluation.
New Directions for Evaluation, 117, 21-36.
Nemanich, L., Banks, M., & Vera, D. (2009). Enhancing knowledge transfer in classroom versus online
settings: The interplay among instructor, student, content, and context. Decision Science Journal
of Innovative Education, 7, 123-148.
Corporate Sector
Guerci, M., Bartezzaghi, E., & Solari, L. (2010). Training evaluation in Italian corporate universities: a
stakeholder-based analysis. International Journal of Training and Development, 14(4), 291-308.
Simmonds, D., & Tsui, O. (2010). Effective design of a global leadership programme. Human Resource
Development International, 13(5), 519-540.
Tomlinson-Clarke, S. (2000). Assessing outcomes in a multicultural training course: a qualitative study.
Counseling Psychology Quarterly, 13(2), 221-231.
Social Services Sector
McLinden, D., Phillips, R., Hamlin, S., & Helbig, A. (2010). Evaluating the future value of educational
interventions in a health care setting. Performance Improvement Quarterly, 22(4), 121-130.
Sleezer, C. M., & Spector, M. (2006). Assessing training needs of HIV program providers: A mixed
methods evaluation. Performance Improvement Quarterly, 19(3), 89-105.
Non-Profit Organization
Hendricks, M., Plantz, M. C., & Pritchard, K. J. (2008). Measuring outcomes of United Way-funded
programs: Expectations and reality. In J. G. Carman & K. A. Fredericks (Eds.), Nonprofits and
evaluation. New Directions for Evaluation, 119, 13-35.
7
R561 Course Schedule
Intro:
Unit 1:
Unit 2:
Unit 3:
Unit
Intro
Unit
1
Unit
2
Unit
3
Introduction and Overview
Basics of Evaluation
Reaction: Level 1 Evaluation
Learning: Level 2 Evaluation
Week
Topics
Reading
Assignment
1
• Introduce yourself
1/13 • Course overview
• Guidelines for individual
evaluation project proposal
• Movie: 12 Angry Men
2
• Basics of Evaluation:
1/20
1. concepts and definitions
2. models and frameworks
• Individual evaluation project
proposal presentation
Cho et al. (2009); Shrock • Case study selection
& Geis (2010)
• individual evaluation
project proposal due
3
• Reaction: Level 1 Evaluation
1/27
1. purposes and uses
2. developing questions
• Guidelines for final evaluation
project proposal
Kirkpatrick & Kirkpatrick
(2006); Lee (2006)
4
2/3
Gilmore (2006); Thomas
(2006)
• Reaction: Level 1 Evaluation
1. assessing questionnaire
2. questionnaire data
analysis
• Final evaluation project
proposal presentation
Final project proposal due
5
• Learning: Level 2 Evaluation
2/10
1. fundamentals of testing
2. instructional objectives
• Case study presentation 1
Kirkpatrick & Kirkpatrick • U2 Exercise due
(2006); Morrison et al.
• Case study 1
(2007)
6
• Learning: Level 2 Evaluation
2/17
1. practical tests
2. guidelines for measuring
learning (Ingu)
• Case study presentation 2
Shrock & Coscarelli
(2007)-chapters 7-8
7
2/24
8
3/3
Unit
4
Unit 4: Transfer: Level 3 Evaluation
Unit 5: Impact: Level 4 Evaluation
Unit 6: Evaluation Synthesis
Case study 2
No Class (I will be presenting my study at the AHRD Conference)
• Transfer: Level 3 Evaluation
1. data collection methods
• Case study presentation 3
9
• Transfer: Level 3 Evaluation
3/10
1. interviews
2. observation
• Case study presentation 4
Kirkpatrick & Kirkpatrick • U3 Exercise due
(2006); Marrelli (2010)
• Case study 3
Pershing (2006);
Pershing et al. (2006)
8
Case study 4
10
3/17
Spring Break (No class!)
11 Individual and final evaluation projects: Mid-term evaluation Mid-term peer evaluation
3/24
(due 9:00 am)
Unit
5
Unit
6
12 • Impact: Level 4 Evaluation
3/31
1. implementing four levels
2. return on investment
Kirkpatrick & Kirkpatrick U4 Exercise due
(2006);Phillips & Phillips
(2010)
13
4/7
Kaplan & Norton (1992);
Parsons (1997)
• Impact: Level 4 Evaluation
1. balanced scorecard
2. values
14 • Evaluation Synthesis
4/14
1. utilization-focused
evaluation
2. evaluation ethics
15
4/21
Patton (2008); Schwandt
(2007)
Pilot Presentations
Wrapup
16
4/28
Reflections on individual and final evaluation projects
9
• Draft of final project
• presentation file
• Individual project report
(due 9:00 am)
• Final report (due 9:00 am)
• Reflection paper
(due May 1 at 9:00 am)
• Final peer evaluation (due
9:00 am)
• Course evaluation
R561: Evaluation
INDIVIDUAL EVALUATION PROJECT GUIDE
(10 points)
“You cannot understand a system until you try to change it.” (Kurt Lewin)
Purpose: Students are required to develop an individual (personal) evaluation project to fully
understand the organization level of evaluation. Students’ struggles with themselves in the individual
project are expected to give them insights into the problems of evaluation in the team projects.
Students’ evaluation topics (e.g. stopping smoking, losing weight, overcoming shyness, learning to talk
more in large classes, improving a relationship with a family member, overcoming chronic lateness) can
be anything to practice an individual (personal) level of evaluation to see what you’ve done to see
progress throughout the semester.
Outcome:
Report of what you’ve done to evaluate progress in your individual project
Process: Once you choose a topic and lay out a proposal, you work in pairs to evaluate progress in the
project throughout the semester. We will have three presentation sessions (proposal, mid-term, and
final) on your progress incorporating your partner’s feedback. For your final presentation, you are
required to write a one-page evaluation report of what you’ve done to accomplish your intended goal
for the individual project.
Reminder:
 Post a single-spaced, one-page report in Assignments by April 28 at 9:00 am.
Step 1: You are expected to pick an individual (personal) evaluation topic that you want to work on
throughout the semester.
Step 2: You are required to write a one-page proposal of your individual evaluation project (due Jan.
20) and revise it twice in the mid-term (Mar. 24) and in the final presentations (April 28) including:





Why did you choose the topic?
What is the purpose/goal of your project?
How will you achieve the intended goal?
What are the evaluation methods that you will implement to achieve the goal?
Others if any
Step 2: You regularly discuss this with your partner explaining what to do to evaluate progress in your
project. You also play the role of peer reviewer for your partner which is very important to help your
partner succeed in the project.
Step 3: Proceed with a step-by-step process to accomplish your intended goal by implementing
evaluation methods and procedures as you learn in class. In the process, you consult with your partner
and if necessary ask for a help from the instructor/TAs.
Step 4: We will have three informal presentations in class in weeks 2 (Jan. 20), 11 (Mar. 24), and 16
(April 28). For the mid-term presentation, you must submit a one-page, updated summary of evaluating
progress. For the final presentation, prepare for a finalized one-page evaluation report including two
more items: (1) what are the results of the project? (2) What did you learn from this project? And (3)
what would you do differently?
10
R561: Evaluation
UNIT 2 EXERCISE GUIDE
Questionnaire Development
(10 points)
Purpose: Develop and evaluate survey questions. Unit 2 Exercise assesses students’ abilities to
synthesize what they have learned. Students are also required to peer-review the other team’s work.
Deliverable: A short evaluation report includes the following:
1. Cover page and table of contents
2. Main body:
o A brief introduction to the context you chose for your survey.
o A summary description of the feedback you received from your peer review team and
how you incorporated their suggestions into your final survey.
3. Appendices:
o The “final” draft of your survey.
o A copy of the assessment documents (attached in Assignments) you sent to your peer
review team.
o A copy of your “first” draft of your survey.
REMINDER: Your team leader is required to post a file (i.e. a professionally developed set of
materials) in Assignments by Feb. 10 at 9:00 am. Do not exceed a single-spaced five-page limit for
the main body.
Process: Work in a team to carry out each unit exercise. Your work will be assigned a letter grade.



Check schedule for due date (Feb. 10).
Your team is playing the role of a survey designer hired by an organization to evaluate the survey
and make recommendations for improvements.
Your team will create a survey and send it for feedback from peers, assess your peer team’s
survey, and revise your own survey based on your peer review team’s feedback. Here is the
scenario:
1.
2.
3.
4.
5.
6.
You have just finished developing a training program for 100 employees in the client’s
organization. This training covered a range of content including cognitive, behavioral, and
affective objectives. You choose the content area.
Your first task is to create a survey that measures participant reactions to this training.
Your survey should be one to three pages in length, meet all the criteria discussed in the
readings, and be formatted in a way that is user-friendly. Create your first draft. In your
introduction to the questionnaire, include information that addresses the content area you
chose for this simulated activity.
Email your survey as an attachment to your peer review team. You need to do this at least
one week before the final due date for the assignment.
Your team will also serve as a reviewer of your peer team’s survey. To help you provide
useful feedback, use “assessment documents” (attached in Assignments) which are a job
aid for your use in evaluating the survey.
Send a completed, filled-out copy of these documents to your peer team.
Once your team receives feedback from your peer review team, your final task is to revise
your survey and post a short evaluation report in Assignments by Feb 10 at 9:00 am.
Attached to your revised survey should be a description outlining the feedback you
received and what steps you took to remedy the noted problems or an explanation as to
why you chose to ignore the advice.
11
R561: Evaluation
UNIT 3 EXERCISE GUIDE
Test Development
Purpose: Develop test questions to evaluate what students have learned in a course.
Deliverable: A short evaluation report includes the following:
1. Cover page and table of content
2. Main body:
a. A brief introduction to the context you chose for your test questions.
b. A summary description of the feedback you received from your peer review team and
how you incorporated their suggestions into your final test questions.
3. Appendices:
a. The “final” draft of your test questions.
b. A copy of the assessment document (attached in Assignments) you sent to your peer
review team.
c. A copy of the “first” draft of your test.
REMINDER: Your team leader is required to post a file (i.e. a professionally developed word
document) in Assignments by March 3 at 9:00 am. Do not exceed a single-spaced five-page limit
for the main body.


Check schedule for due date (March 3).
Your team will create test questions, assess your peer team’s test questions, and revise your own
test based on peer team’s feedback. Here is the scenario:
1.
2.
3.
4.
5.
6.
7.
Your team is playing the role of a consultant hired by IST to create test questions for R511:
Instructional Technology Foundations and make recommendations for improvements.
This IST course covers a range of topics related to instructional technology, human
performance technology, and career and professional development. You may have to figure
out how you would know whether students have mastered an understanding in the course
contents.
Your test should be about three pages in length, meet all of the criteria discussed in the
readings, and be formatted in a way that is user-friendly and to the point. Create a first
draft. In your introduction to the test, include information that addresses the specific
contents for this simulated activity.
Email your test questions as an attachment to your peer review team. You need to do this
at least one week before the final due date for the assignment. Your test questions will be
reviewed by your peer team and you will also review your peer team’s test development.
To help you provide useful feedback, use the one-page assessment document (attached in
Assignments) which is a job aid for your use in evaluating the test questions.
Send a completed, filled-out copy of the document to your peer team of the test
development you reviewed.
Once your team receives feedback from your peer review team, your final task is to revise
your test questions and post a short evaluation report in Assignments. Attached to your
revised test questions should be a description outlining the feedback you received and
what steps you took to remedy the noted problems or an explanation as to why you chose
to ignore the advice.
12
R561: Evaluation
UNIT 4 EXERCISE GUIDE
Transfer Evaluation Case Scenario
Purpose: Using a professional context of your choice, write a case using a transfer evaluation scenario
for a simple and straightforward performance improvement intervention.
Deliverable: Your case scenario includes the following:
1. Cover page
2. A brief introduction to the context and intervention
3. Key transfer evaluation questions are:
(1) stated and (2) accompanied by an explanation of how they were answered including:
a. data collection (e.g. interviews and/or observation)
b. data analysis (i.e. specifics on how the data gathered are transformed to useful
information)
c. accompanied with sample instruments
4. Summary of findings and implications
5. General conclusions
6. References (in the APA format)
7. Appendix if any
REMINDER: Post a case scenario in Assignments by March 31 at 9:00 am. Do not exceed a singlespaced ten-page limit.


Check schedule for due date (March 31).
Your team’s case scenario will be evaluated based on the following criteria:
(1) Context: Clear, simple, and straightforward transfer scenario

Intervention clearly explained

Organization type clear: sector and/or setting

Subjects described: who would be impacted

Transfer evaluation approach chosen clearly explained
(2) Transfer Evaluation

Transfer evaluation techniques used clearly explained

Techniques match well with context

Data collection/analysis explained and match with techniques

Findings clear and fit with transfer evaluation methodology
(3) Communications

Clear closing and general conclusions

Succinct and pointed

Logical flow and structure

Grammar does not interfere with messages

References in the APA format
13
R561: Evaluation
FINAL EVALUATION PROJECT GUIDE
(30 points)
Purpose: Partake in in-depth activities to develop a more complete understanding of the evaluation
process as applied in training and performance improvement. Students are expected to utilize three
unit exercises and add on the impact level of evaluation to this capstone project in order to display
their ability to apply what they’ve learned to a real organization.
Criteria: Criteria for choosing the final evaluation project are:
 Something that the students are involved in and familiar with (e.g. somewhere in the SOE)
 Something that would make a contribution to the target organization
 Something that is realistic in terms of being doable in the time allocated
Outcome: Report of a comprehensive organization-specific evaluation project
Reminder: This final evaluation project is ongoing throughout the semester.
 Post a one-page proposal of your final evaluation project by Feb. 3 at 9:00 am.
 Post a one-page outline of the mid-term presentation (due Mar. 24 at 9:00 am).
 Post a draft of your final evaluation report in Assignments by April 21 at 9:00 am. Do not
exceed a limit of ten single-spaced pages.
 Submit the revised final evaluation report by April 28 at 9:00 am.
Step 1.




A one-page proposal of your final evaluation project (due Feb. 3) including:
Project title and a description of organization and contact person(s)
Training, PI, and HRD issues/concerns to be addressed
General explanations of processes or methods
Division of labor, rough timelines, and others if there is any
Step 2. Project objectives: What will be accomplished?
 Outline of major project steps and activities, including detailed timelines
 Letter of agreement on project goals and processes with the client organization
Step 3. Completed draft of instrumentation: Means of collecting data
 Data collection strategies: details on what, when, how, and where
 A detailed description of the population (who)
Step 4. Completed draft of data analysis plan: Methods used to analyze and synthesize the data
 Outline of project report: topical outline
 Samples of tables and graphics that are used to present findings
Step 5. Implementation of instrumentation
 Data collection, coding of data, data analyses and syntheses, follow-ups and refinements
Step 6.



Draft of the project report
A one-page outline of the mid-term presentation (Mar. 24)
Pilot presentation (April 21)
Feedback and Revision
Step 7. Final evaluation report (due April 28)
 Executive summary
 Evaluation report
14