PAUS 8521:
Dates:
Time:
Location:
Credits:
Pre-requisite:
EVALUATION RESEARCH (cat #4663)
Fall 1999, Wednesdays, August 25 – December 8
7:15-9:45 p.m.
324 - Sparks
3
PAUS 8131 Applied Research Methods and Statistics
Professor:
Office:
Phone:
e-mail:
Office hours:
Judith M. Ottoson, Associate Professor
Urban Life, 1044
404-651-0605
JudithOttoson@gsu.edu
by appointment
Why Evaluation Research?
Policy makers, practitioners, administrators, program funders, and intended beneficiaries
are often faced with decisions about determining the “success” or value of social policies and
programs. Are policies and programs achieving intended results? Any results? Should existing
policies or programs be abandoned, replaced, or modified in whole or in part? The answers to
these questions are determined by who is involved in the process, how the process evolves,
which values are used, what constitutes the policy or program, and how results are used. Program
evaluation is a field of research and practice that attempts to address these issues.
Course Objectives
The overall purpose of the course is to develop an understanding of and skill in
evaluation research as a means of informing public policy and program development. Specific
course objectives include:

to understand alternative evaluation concepts, approaches, and models used in
contemporary evaluation

to analyze the social, political, legal, ethical, organizational, and interpersonal problems
which often undermine technically sound evaluation studies and gain skill in anticipating
and coping with such pressure and constraints

to identify ways to facilitate evaluation use as a means of improving public programs and
organizational processes

to develop a competent and comprehensive evaluation proposal

to choose evaluation approaches appropriate to the context and your values and expertise
1
Course Approach
Evaluation is understood to be a complex process of placing value on something. In this
course, that something is a policy or program. This course combines theory and practice to
explore the why, who, and when of evaluation as a determinate of the how. Rather than applying
a one-theory-fits-all approach, the course begins with an overview of multiple perspectives on
evaluation research and the standards for program evaluation. Following this foundation, we will
work our way through the text to develop and apply an understanding of the evaluation research
process. The knowledge gained from this work will be applied to case studies of evaluation
practice and an educational policy or program of the student’s own choosing.
Course Readings
The following pages of this syllabus designate the required readings for each week. It is
expected that students will complete all readings before each session so that the course can be
conducted as a lively seminar.
Required readings

Course text: Weiss, C.H. (1998). Evaluation: Methods for studying programs and
policies. (2nd ed.) Englewood Cliffs: Prentice-Hall.

A packet of supplemental readings for weeks 2 -5 is available at The Print Shop by 9/1/99

A packet of supplemental readings for weeks 6-12 will be available at the Print Shop by
9/15/99
Recommended reading

The Joint Committee on Standards for Educational Evaluation. (1994). The program
evaluation standards (2nd ed.). Thousand Oaks: Sage.
Course Assignments and Grading
The course grade will be based on an exam, evaluation proposal, and class presentation.
The following is a summary of the components of the course grade. More detail on assignments
and grading are provided on page 7.
Date
September 22
September 29
October 6
November 10
Nov 17 or Dec 1
December 8
What
Pre-proposal, 1 page
Mid-term exam, 60"
Proposal: part I, 8-10 pages
Proposal: part II, 8-10 pages
In-class presentation
Final proposal: parts I, II, & III, 30 pages
2
% of grade
Na
25%
15%
15%
5%
40%
100%
Course Expectations
It is expected that the course will be a collaborative learning process for all involved. You
are encouraged to share your experiences and thinking about evaluation research and to
encourage others to do the same. It is expected that you will attend all course sessions. If you
cannot do so, it is expected that you will contact the professor prior to class and notify her by
phone or e-mail of your absence. If you miss a class, you will be held responsible for the content
and materials of that week. Please make arrangements with a fellow class member to acquire
these. It is expected that the work you do in this class will be original and not plagiarised from
another other source. You are responsible for understanding what plagiarism means. If in doubt,
ask. Lastly, it is expected that you just might become interested in and skilled at doing evaluation
research. Con cuidado!
COURSE SCHEDULE, READINGS, AND ASSIGNMENTS
SESSION 1: August 25, 1999
TOPICS: Overview
- course intent, content, schedule, and assignments
- interests and experiences in evaluation
- current debates in evaluation research
SESSION 2: September 1, 1999
TOPICS: multiple purposes and stakeholders
- evaluation as research or evaluation vs research?
- multiple purposes and stakeholders in evaluation
READINGS:
- Text Chapter 1: Setting the Scene, pp 1-19
- Text Chapter 2: Purposes of Evaluation, pp 20- 45
SESSION 3: September 8, 1999
TOPICS: evaluation theories
- comparative framework
- meanings of theory and connections to practice
- theories of programming, valuing, utility, knowledge construction, and practice
READINGS:
- Shadish, W.R., Cook, T.D.,& Leviton, L.C. (1991). Good theory for social program
evaluation. In Shadish, W.R., Cook, T.D.,& Leviton, L.C., The foundations of
program evaluation, pp. 36-67. Newbury Park: Sage.
- Ottoson, J.M., (in press) Evaluation of continuing professional education: Towards a
theory of our own. New Directions in Adult and Continuing Education. Newbuy Park:
Sage.
3
SESSION 4: September 15, 1999
TOPICS: multiple approaches to evaluation
READINGS:
- Posavac, E.J., & Carey, R.G. (1992). Planning and evaluation. In Posavac, E.J., &
Carey, R.G. Program evaluation: Methods and case studies. (4th ed.). pp. 23- 27.
- Stufflebeam, D.L., & Webster, W.J. (1983). An analysis of alternative approaches to
evaluation. In Stufflebeam, D.L., & Webster, W.J. (eds). Evaluation models:
Viewpoints on educational and human services evaluation. (pp 23-43). Boston:
Kluwer-Nijhoff Publishing.
- Patton, M.Q. (1997). Utilization-focused evaluation. Thousand Oaks: Sage, pp. 192194.
SESSION 5: September 22, 1999
TOPICS: program evaluation standards
- utility, feasibility, propriety, accuracy
- trade-offs and application
READING:
- The Joint Committee on Standards for Educational Evaluation. (1994). The program
evaluation standards (2nd ed.). Thousand Oaks: Sage, pp 1-20.
- Summary of program evaluation standards
ASSIGNMENT: pre-proposal due
SESSION 6: September 29, 1999
TOPICS: focusing on what is being evaluated
- “the evaluand:” policy, program or project
- social problem intended to be ameliorated
- social program theory
READINGS:
- Text Chapter 3: Understanding the Program, pp. 46-71
EXAM: mid- term exam at the beginning of class
SESSION 7: October 6, 1999
TOPIC: planning the evaluation
- evaluation questions
- practical guidelines for practice
- evaluator roles
READINGS:
- Text Chapter 4: Planning the Evaluation, pp. 72- 96
- Text Chapter 5: Roles for the evaluation, pp. 97 – 113.
ASSIGNMENT: Part I of the evaluation proposal is due
4
SESSION 8: October 13, 1999
TOPIC: evaluation design – quantitative
- developing measures
- data collection
- data analysis
READINGS:
- Text Chapter 8: Design of the Evaluation, pp. 180-214
- Text Chapter 6: Developing Measures, pp. 114-151
SESSION 9: October 20, 1999
TOPIC: evaluation design – qualitative
- foreshadowed problems
- data collection
- data analysis
READINGS:
- Text Chapter 11: Qualitative Methods, pp. 252-270
- Le Compte, M.D. & Preissle, J. (1993). Summary Table. Ethnography and qualitative
design in educational research. (2nd ed.). San Diego: Academic Press.
- Text Chapter 7: Collecting Data, pp. 152-179
SESSION #10: October 27, 1999
TOPIC: evaluation design – mixed
- informing measures with foreshadowed problems
- data collection
- data analysis
READINGS:
- Steckler, A.,McElroy, K.R., & Goodman, R.M. Bird ST | McCormick L (1992).
Toward Integrating Qualitative and Quantitative Methods: An Introduction. Health
Education Quarterly, 19(1), 1-8.
- Reading to be determined
SESSION #11: November 3, 1999
TOPIC: analysis and reporting
- writing reports
- disseminating results
- evaluator stance toward utilization
READINGS:
- Text Chapter 12: Analyzing and Interpreting the Data, pp. 271- 293
- Text Chapter 13: Writing the Report and Disseminating Results, pp. 294-319
5
SESSION # 12: November 10, 1999
TOPIC: management and utilization of findings
- practice of evaluation
- managing the evaluation
READINGS:
- Text Chapter 14: Evaluating with integrity, pp 320-326
- Reading to be determined
ASSIGNMENT: Part II of the evaluation proposal is due
SESSION #13: November 17, 1999
ASSIGNMENTS:
student presentations
NO CLASS NOVEMBER 24, 1999:
HAPPY THANKSGIVING
SESSION #14: December 1, 1999
ASSIGNMENTS:
student presentations
SESSION #15: December 8, 1999
TOPIC: synthesis of course content
- review of course content and process
- revisting the evaluation debates: where do you stand
ASSIGNMENT: final proposal (Parts I, II, and III) due
6
COURSE ASSIGNMENTS: ’99
The following assignmenst are intended to help you gain the knowledge and skill this
course hopes to facilitate in evaluation research.
Mid-term exam
A 60 minute mid-term exam will explore your knowledge of the foundational material in
evaluation research covered in the first five sessions of the course. Both closed-ended and essay
questions will be used. The exam will be held in class on September 29th and count as 25% of the
course grade. Grading will consider the accuracy of your answers, the depth of your
understanding of concepts, your ability to apply ideas, the connection among ideas, and insight.
Evaluation Proposal
To apply your understanding of course content, you will be asked to develop a proposal to
evaluate a policy or program of interest and relevance to you. It is best that you pick a policy or
program in which you are, have been, or will be involved. In other words, pick something you
know something about. On September 22nd, a one-page pre-proposal of the policy or program to
be evaluated is due. The pre-proposal should describe the nature of the policy or program to be
evaluated, your relationship to it, a ball park budget, and the inquiry approach anticipated.
Instructor approval of the proposal is necessary before moving to other parts of the proposal.
Part I of the proposal is due on October 6th. It is expected that this paper will be
approximately 8-10 pages in length. The work in this paper lays the foundation for subsequent
parts of the proposal. Part I should contain the following elements based on the materials covered
in the course to date:

purposes of the evaluation - who wants the evaluation and why? Include an identification
of stakeholders and their interests, like uses of the evaluation, kind of evaluation
(formative vs summative), and your position in the process.

description of the policy or program - what are the relevant characteristics of the policy or
program to be evaluated, including the social problem intended for amelioration. What is
the program "theory?" What services are offered? by whom? for whom?

budget- What is the approximate overall budget for the evaluation? (no details, just the
parameters for funding) Who funds the program? Who funds the evaluation?
Part II of the proposal is due on November 10rd. It is expected that this paper will be
approximately 8-10 pages in length and build on the work initiated in Part I. This part focuses on
evaluation design and data collection. It should contain the following elements based on
materials covered in the course to date:

evaluation questions – What are the key questions for the study?
7




family of inquiry- Which family (families) of inquiry will be used to meet the purposes of
the evaluation and answer the questions posed? Defend your choice for its
appropriateness and feasibility.
data – If this is a quantitative study, which measures will be used? If this is a qualitative
study, what will guide data collection?
data collection – From what sources, sites, and/or samples will data be collected? How
will it be collected? How will the quality of the data be assured?
data analysis – What approaches will be used to analyze the data? Justify these decisions
in light of the evaluation purpose and questions?
Final proposal and Part III are due on the last day of class, December 8.th It is expected
that this final paper will be build on and include Parts I and II (they will count for 20 of the 30
pages expected.) It will also contain Part III, which is intended to analyze and summarize the
proposal. Part III should be 10 pages in length. The final paper should include the following:

Parts I and II of the proposal should be resubmitted after having been strengthened by the
professor’s previous review, class feedback during your presentation, and your increased
understanding of the topic and issues. Make changes as necessary to strengthen these
sections. It is critical that you include a summary sheet to highlight changes made in these
parts.

Part III of the final proposal should build on final sessions of the course and include
- time line – What is the overall time line for the evaluation? What is the time line for
specific components of the evaluation?
- budget- Provide a budget of sufficient detail to show expenses for personnel,
materials, travel, and other major items
- reporting and dissemination – What form will the report(s) take? How will results be
disseminated?
- evaluation standards – Critically analyze your own proposal using the standards for
program evaluation. Describe how this evaluation has taken into consideration the
standards for program evaluation, including accuracy, propriety, feasibility, and
utilization. What kind of trade-offs were necessary to complete the final evaluation
proposal?
Grading of Parts I and II of the proposal will consider the inclusion (at a minimum) of the
components requested, the clarity of your presentation, the connection among the components
and parts, the use of ideas from the readings and class discussion, and insights applicable to your
own circumstances. Grading of the final proposal will look for evidence of learning in revisions
to Parts I and II, the connection of part III to the preceding parts, the quality and clarity of the
writing and presentation of ideas, the ability to think critically about your own proposal, and
accuracy, feasibility, propriety, and utility of the proposal.
In-class presentation
You will be asked to present your evaluation proposal – parts I and II – in-class on either
November 17th or December 1st. It is anticipated that you will have approximately 30 minutes for
8
a presentation and discussion of your proposal. This will be an opportunity to get feedback on
these parts before their inclusion in the final proposal. The grade will consider the completeness
of your presentation, the ability to stimulate and respond to class suggestions, and coherency of
the presentation.
Grading
Following the policies of GSU, straight letters grades will be given as follows. For more
information on grades, see page 62 of the GSU Policy Studies Graduate Bulletin, 1999-2000.
A
B
C
D
F
W
90 – 100
80 – 89
70 – 79
60 – 69
59 or less
Excellent
Good
Average
Poor
Failure
Withdrawal
Before 10/15/99)
9