EDMS 647: Introduction to Program Evaluation
Section 0101 WEB Online
Fall 2010
Dr. Kathryn Anderson Alvestad
1226 Benjamin Building
301-405-3622
alvestad@umd.edu
Catalog Course Description
“Overview of the program evaluation process; problems encountered in the practice of program evaluation.”
General Course Information
This course covers both theory and application of program evaluation methods. During the fall semester, 2010, the
course is being conducted completely online. Although there is some use of lecture-based information transfer in the form
of narrated PowerPoint presentations (which will be available both synchronously and asynchronously), the course is
predominantly delivered in a seminar (1) format. As such, it relies heavily on active student participation and group work
facilitated by discussion boards, a group journal, group wikis, and individual online assessments.
The University’s Enhanced Learning for Maryland Students (ELMS) learning management system will be used
exclusively. If you are not accustomed to an online instructional format, and especially if you have never used ELMS, you
should – at your earliest convenience – spend several hours going through the online tutorials that are provided by the
University. Access them on the home page of ELMS under “Student Resources.” Contact the OIT student help desk
(301-405-1400 or oit-helpdesk@umd.edu) immediately if you encounter difficulties. Once the semester begins, you are
expected to log on to the ELMS website several times each week and to participate completely in all online activities.
Communication
Open Office Hours: This semester my “walk-in” office hours will be on Tuesdays from 9:00 AM until 12:30 PM and
Thursdays from 12:30 until 3:00 PM. On these days at the designated times you may stop by without an appointment.
Appointments: I ‘m also available by appointment on Monday and Thursday mornings from 9 AM to 12:30 PM, and I can
make special arrangements to meet with you later in the day on some Mondays, Tuesdays, and Thursdays.
Online Conferences: Most of the time we will be meeting via online conferences using the WIMBA Live Classroom on our
ELMS website. Online conferences give us flexibility for scheduling during the evenings and on some weekends. We can
share documents on our computer desktops using this application, so I can look at your work, review it with you, and give
you immediate suggestions for improving it.
Email: I’m always happy to respond to email messages from students. Please follow these guidelines:
 You will receive faster and better answers to questions about technology problems by calling (301-405-1400) or
emailing (oit-helpdesk@umd.edu) the University of Maryland College Park OIT Helpdesk.
 All of the email messages you receive from me will be sent directly from ELMS, so you should check your
umd.edu email account regularly or set up your profile to automatically forward messages to the email account
you use daily. (Go to www.myumd.edu to change your default email address.)
Twitter: I will occasionally post short messages on twitter. These may include reminders, updates, and information about
helpful resources I come across during the semester. Follow me at 410DrA.
REQUIRED MATERIALS
Royse, D., Thyer, B. A. & Padgett, D. K. (2010). Program Evaluation: An Introduction (5th Ed.). Belmont, CA:
Wadsworth. ISBN 978-0-495-60166-1
Morris, M., Editor. (2008). Evaluation Ethics for Best Practice: Cases and Commentary. New York: The Guilford Press.
ISBN 978-1-59385-569-7
Please note that the on-campus bookstore has two versions of both of these texts listed on Testudo. One is the traditional
paperback book and the other is an e-book. You do not need both!
Page 1 of 6
Accessing ELMS
To access the course website, use the following URL: http://www.elms.umd.edu. You will be asked to provide your
Username (Directory ID) and a Password. If you have difficulty logging in, look below the login box where there is a link to
a self-help page that takes you step-by-step through the process. Click on “Student: Logging into your Course(s).”
Once you are successfully logged in, a list of the courses in which you are registered that have ELMS websites will
appear on the right side of your screen. Click on “EDMS 647” and then select from the list of menu items on the left of the
screen.
Prerequisite Skills
EDMS 645 Quantitative Research Methods I is a prerequisite for EDMS 647. A list of the topics covered in EDMS 645 is
available on the EDMS website (www.education.umd.edu/EDMS). There will be no direct instruction on these topics in
EDMS 647. Please note that a complete data analysis plan will be part of the research proposal that you are required to
produce for this course, so if you have not taken the prerequisite course or if your skills are weak in this area, it is your
responsibility to rectify the situation so that you can meet the course requirements for EDMS 647.
Course Objectives
The following are the principle course objectives for EDMS 647. They are presented in a standard format that includes
the conditions under which the objective will be assessed, the course objective that will be assessed, and the level of
accuracy that is necessary to certify mastery. Note that the criteria stated in the objectives are associated with a
grade of “B.” To earn a grade of “A“ in the course, students must exceed the criteria stated.
There are three categories of course objectives for EDMS 647. They follow the structure of Bloom’s Taxonomy of
Educational Objectives, Revised (Anderson & Krathwohl, 2001), namely factual knowledge that you are expected to
remember, conceptual knowledge that you are expected to understand, and procedural knowledge that you are expected
to apply. Each objective is referenced in the Evaluation section of this syllabus by category and number.
Remembering Factual Knowledge
1. Given two lists, SWBAT(2) recognize the definitions and characteristics of the terms cost-effectiveness evaluation,
client satisfaction evaluation, mixed-method evaluation, needs assessment, outcome evaluation, process
evaluation, and single system research design by matching each term with its definition with 80% accuracy.
2. Given two or more options from which to choose, SWBAT recognize the definitions and characteristics of the
terms audience, bias, case study, convergent analysis, data reduction, Delphi technique, descriptive data,
external evaluation, field test, focus group, formative evaluation, goal, informed consent, internal evaluation,
meta-evaluation, participant, pilot test, program, program drift, qualitative evaluation, quantitative evaluation,
sponsor, stakeholder, strategy, summative evaluation, and treatment fidelity by selecting the correct response
with 80% accuracy.
3. Given definitions for the terms anonymity, assessment, attrition, confidentiality, dependent variable, evaluation,
external validity, independent variable, internal validity, logic model, measurement, operational definition,
outcome, program monitoring, quality assurance, test, triangulation, and verification, SWBAT recall each definition
by supplying the term with 80% accuracy.
Understanding Conceptual Knowledge
1. Given several authentic scenarios, SWBAT compare and contrast the scenarios, determine the purpose of the
evaluation that each scenario depicts, explain their reasoning, and make a judgment about the appropriateness of
the planned strategies, meeting the criteria described on a rubric with a score of 4 out of 5.
2. Given a series of articles and internet resources relevant to the course content, SWBAT collaboratively contribute
to a group wikis and/or a group journal relevant to the content topics listed in the syllabus, meeting the
participation criteria described in the course syllabus.
3. Given a series of case studies related to concepts and ideas relevant to the course content, SWBAT respond
collaboratively in asynchronous discussions to questions posed by peers, meeting the participation criteria
described in the course syllabus.
Applying Procedural Knowledge
1. Given a list of specifications and a research report of their choice, SWBAT complete a meta-evaluation of the
research described in the report, using the JCSEE Program Evaluation Standards as a guide and meeting the
criteria described on a rubric with a score of 4 out of 5.
2. Given a scaffolded performance-based task, SWBAT develop a research proposal for a hypothetical program
evaluation, meeting the criteria described on a rubric with a score of 8 out of 10.
.
Recommended Resources
Recommended resources and electronic links to full-text documents and/or websites can be found in the folders in the
Topical Materials section of the course website and also in the Course Resources section.
Evaluation Plan
Selected-Response Exam
A summative evaluation of vocabulary knowledge and other concepts specified in the course objectives will be
administered via the course website. (Remembering Factual Knowledge, Objectives 1, 2, and 3)
Essay Exam
A summative evaluation of concepts specified in the course objectives will be administered via the course website. A
rubric describing performance criteria for the exam is posted in the Assessment section of the course website.
(Understanding Conceptual Knowledge, Objective 1)
Collaborative Activities
Students are expected to read the textbook assignments and selected or assigned articles, and to review relevant internet
sites to prepare for each course topic. Students are expected to participate in collaborative activities continuously
throughout the semester. Up to 15 points of the final grade will be awarded for participation in collaborative activities.
(Understanding Conceptual Knowledge, Objective 2)
Asynchronous Online Discussions
Case studies from the Morris text will be assigned in conjunction with relevant topics. Students will then use the
Discussion Board on the course website for asynchronous discussion of the case studies. Students are expected to
participate in the online discussions continuously throughout the semester. Students should set a goal of a minimum of
30 postings overall, or 5 per discussion. The responses themselves will not be graded, but up to 20 points will be
awarded for participation as outlined above. (Understanding Conceptual Knowledge, Objective 3)
Metaevaluation
You will select an article describing a completed program evaluation project and write a meta-evaluation of the project. A
detailed description of the assignment, including performance criteria, is posted in the Assignments section of the course
website on ELMS. (Applying Procedural Knowledge, Objective 1)
Course Project
You will be expected to develop a three-part research proposal for a program evaluation. A detailed description of the
project, including performance criteria, can be found in the Assignments section of the course website on ELMS. As you
develop your plan, you will have the opportunity to receive formative feedback through a series of instructor reviews.
Summative feedback will be provided by grading following the final submission of your plan. (Applying Procedural
Knowledge, Objective 2)
Grading
This course is not graded on a curve. Projects and exams will be graded on a 1 to n-point scale. Each activity will be
associated with n points, for a total of 120 possible points. The final grade will be based on the ratio of the total number of
points earned to the total number of points possible, according to the following criteria:
Students who earn 90 - 100% of the total number of possible points will receive a grade of A;
Students who earn 80 - 89% of the total number of possible points will receive a grade of B;
Students who earn 70 - 79% of the total number of possible points will receive a grade of C;
Students who earn 60 - 69% of the total number of possible points will receive a grade of D; and
Students who earn 0 - 59% of the total number of possible points will receive a grade of F.
The number of points available for each assignment and assessment will be as follows:
Course Project
30 points
Online Discussions
20 points
Meta-evaluation
15 points
Collaborative Activities 15 points
Essay Exam
20 points
Selected-Response Exam
20 points
“Incomplete” Grades: Unless a student can demonstrate that a catastrophic event has led to extreme hardship, a grade of
“Incomplete” will NOT be given.
.
POLICIES
Academic Accommodations
In compliance with and in the spirit of the Americans with Disabilities Act (ADA), I want to work with you if you have a
documented disability that is relevant to your work in this course. If you need academic accommodation by virtue of a
documented disability, please contact me immediately to discuss your needs. Students with documented needs for an
accommodation must meet the same achievement standards required of all other students, although the exact way in
which achievement is demonstrated may be altered.
Our student population at the University of Maryland is quite diverse, and we have many students whose primary
language is one other than English. Although the same standards of performance apply to all students in EDMS 647, I
want to work with you to make sure that you are able to demonstrate what you know and can do without being overly
restricted by a language barrier. Please make an appointment to talk with me within the first two weeks of the semester
so that we can discuss requirements for written assignments.
Academic Integrity
The University of Maryland, College Park has a nationally recognized Code of Academic Integrity, administered by the
Student Honor Council. This Code sets standards for academic integrity at Maryland for all students. As a student you
are responsible for upholding these standards for this course. It is very important for you to be aware of the
consequences of cheating, fabrication, facilitation, and plagiarism. For more information on the Code of Academic
Integrity or the Student Honor Council, please visit http://www.studenthonorcouncil.umd.edu/whatis.html. To demonstrate
your commitment to academic integrity, remember to sign the Honor Pledge on all examinations and assignments.
Participation
Congruent with expectations for a graduate level course, students are expected to share expertise, discuss ideas, and
work together to accomplish course objectives. It is therefore expected that students will attend all classes, participate
fully in class discussions, and contribute equally to group activities. Students are also expected to check their email
frequently and respond expeditiously to communications from the instructor and/or other members of the class.
Make-Up Examinations
Both of the exams for this course will be administered online, and I generally give students 24 hours in which to complete
online exams. Nevertheless, occasionally a student has an excusable reason why they can’t take an exam on the
designated date. (For instance, one student was on her honeymoon, another was in a foreign country completing
research for her dissertation.) Students anticipating an excused absence for the week in which an exam is scheduled
must follow this procedure: (1) Notify me in writing no later than two weeks prior to the scheduled exam; (2) be prepared
to present the requisite documentation for the absence per UMD requirements; and (3) be prepared to take an alternative
examination prior to the scheduled date if necessary.
Religious Observances
It is the policy of UMCP that students not be penalized in any way for participation in religious observances. You will be
allowed to make up academic assignments that are missed due to such absences. It is your responsibility to contact me
and make arrangements for make-up work or examinations. You must provide written notification of a projected absence
to me within the first two weeks of the semester. The notification must identify the religious holiday(s) and the date(s) and
should not include travel time. The process will be confidential.
REFERENCE
Anderson, L. W. & Krathwohl, D. R., Eds. (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of
Bloom’s Taxonomy of Educational Objectives. New York: Addison Wesley Longman, Inc.
FOOTNOTES
(1) Seminar: A form of academic instruction. It has the function of bringing together small groups for recurring meetings, focusing each
time on some particular subject, in which everyone present is requested to actively participate. This is often accomplished through an
ongoing Socratic dialogue with a seminar leader or instructor. The idea behind the seminar system is to familiarize students more
extensively with the methodology of their chosen subject and also to allow them to interact with examples of the practical problems that
always crop up during research work. It is essentially a place where assigned readings are discussed, questions can be raised and
debates conducted. It is relatively informal, at least compared to the lecture system of academic instruction. (Wikipedia)
(2) SWBAT = Students will be able to
.
EDMS 647 Introduction to Program Evaluation
COURSE SCHEDULE for Fall Semester 2010
WEEK
Aug. 30
Sept. 6
TOPICS and ACTIVITIES
Course Overview
Program Evaluation Standards
Guiding Principles for Evaluators
Institutional Review Boards
ONLINE: “Introduction to the
Course” in the WIMBA Live
Classroom, Sept. 1 @ 7 PM
Discussion Board (DB): Please go
online to the ELMS course website
and contribute to the DB titled
“Introduce Yourself.”
Evaluation Proposals
Metaevaluation
Pragmatic Issues
Collaborative Activity (CA):
Pragmatic Issues
Evaluation Ethics
Sept. 13
Sept. 20
RESOURCES
Royse: Chapters 1 & 2
Morris: Chapter 1
Program Evaluation Standards
http://www.jcsee.org/programevaluation-standards
AEA Guiding Principles
(Morris: Appendix A)
UMD IRB Process
http://www.umresearch.umd.ed
u/IRB/
Royse: Chapters 13 & 15
Metaevaluation Online
Resource
http://www.bhummel.com/Meta
evaluation/index.html
Morris: Chapters 2 & 3
DB: Case Studies
Program Evaluation Designs I:
Needs Assessment
Process/Formative Designs
CA: R3QD*
Program Evaluation Designs II:
Single System Research Designs
Client Satisfaction Designs
ASSIGNMENTS DUE
Draft of Course Project, Part 1
is due Friday, Sept. 17 if you
wish to have a FORMATIVE
FEEDBACK review.
Royse: Chapters 3 & 5
Royse: Chapters 6 & 7
Course Project, Part 1 is due
on Friday, Oct. 1.
Royse: Chapters 9 & 10
Turn in the citation and abstract
ONLY for the article or report
you plan to use for your
metaevaluation paper.
Morris: Chapters 4 & 5
Draft of Course Project, Part 2
is due Friday, Oct. 15 if you
wish to have a FORMATIVE
FEEDBACK review.
Royse: Chapters 4 & 8
Course Project, Part 2 is due
on Friday, Oct. 29.
Sept. 27
Oct. 4
ONLINE: “Program Evaluation
Designs,” WIMBA Live
Classroom, Sept. 30 @ 7 PM
Program Evaluation Designs III:
Group Research Designs
Cost-Effectiveness Designs
CA: R3QD*
Evaluation Ethics
Oct. 11
Oct. 18
Oct. 25
DB: Case Studies
ESSAY EXAM
Program Evaluation Methods I:
Qualitative Methods
Mixed-Method Evaluations
Sampling
ONLINE: “Program Evaluation
Methods,” WIMBA Live
Classroom, Oct. 26 @ 7 PM
Page 5 of 6
Nov. 1
Nov. 8
Program Evaluation Methods II:
Measurement Tools & Strategies
Data Analysis (Review)
CA: Data Reduction Activity
Royse: Chapters 11 & 14
http://www.eval.org/Resources/
instruments.asp
(Explore the electronic links to
Instrument Collections)
Evaluation Ethics
Morris: Chapters 6 & 7
DB: Case Studies
Participatory Evaluation
Nov. 15
Nov. 22
Nov. 29
CA: Semester Summary Activity
NO ACTIVITIES
(Thanksgiving Week)
Evaluation Ethics
DB: Case Studies
ONLINE: “Review,” WIMBA Live
Classroom, Dec. 1 @ 7 PM
FINAL EXAM
Dec. 6
*R3QD =
.
Research
Read
Review
Question
Discuss
http://www.hfrp.org/evaluation/t
he-evaluation-exchange/issuearchive/participatory-evaluation
(Review this issue of the
Evaluation Exchange)
Morris: Chapter 8
Draft of Course Project, Part 3
is due Friday, Nov. 12 if you
wish to have a FORMATIVE
FEEDBACK review.
Metaevaluation is due on
Friday, Nov. 19.
Course Project, Part 3 is due
on Friday, Dec. 3.