University of Alaska Anchorage
School of Social Work
Master of Social Work Program
Course Title:
Program Evaluation
Course Number:
SWK A628A
Section:
651
Prerequisites:
None
Co-requisites:
None
Course Credits:
3 Semester Credits
Semester:
Fall 2010
Meeting Day & Time:
Wednesdays, 2:30 pm-5:15 pm
Location:
ESH 206
Course Instructor:
Randy Magen, Ph.D.
Office:
106F Gordon Hartlieb Building
Telephone:
(907) 786-6901
FAX:
(907) 786-6912
E-mail:
magen@uaa.alaska.edu
Office Hours:
Office hours will be Wednesdays from 11:00 pm to 12:30
pm. A sign-up sheet is located at the front desk of the
School of Social Work (Suite 106 Gordon Hartlieb
Building). Other times are available by appointment.
Support Staff:
Suzanne Dvorak, Sharie Field, Ken Schultz, Eva Wilson
786-6900
Department Home Page: http://www.uaa.alaska.edu/socialwork/
Blackboard:
http://www.uaa.alaska.edu/classes/
Course Description
Theory and practice of agency or community-based research and evaluation. Course topics
include commonly used evaluation models and research designs, politics and ethics of
conducting and using research in an applied setting, communicating findings.
Many human services professionals have voiced a need for more training in evaluation research.
Funding sources, legislative bodies and accreditation agencies are requiring service providers to
demonstrate the effectiveness of their services and develop responsive programs. Agencies also
recognize the value of program evaluation in improving the quality of the services delivered. Yet
while the need for program evaluation is increasing, many health and human service providers
are not familiar with program evaluation methods. This course will provide the student with
knowledge of purposes, designs, and implementation of social services program evaluation. The
acquisition of knowledge and skills will enable practitioners to critically evaluate programs and
practice effectiveness, create and disseminate empirically derived knowledge, and apply the
knowledge for more effective practice and service. The student will develop the ability to
conceptualize, design and implement program evaluations. This course covers both formative
and summative evaluation processes. The necessity for evaluation in the current political climate
and issues of cultural competence in evaluation will be discussed.
1
By taking the Program Evaluation class, students learn to produce research in social work
practice at the agency level. (Indeed, the course might be better called Agency-Based
Research & Evaluation to clarify that research is required as part of good program
development (e.g., needs assessments), not only for monitoring after the fact.) Some
content here may be seen as duplicative insofar as many of the concepts from the
Foundation course (SWK 624) will be reviewed; however, in this class the concepts are
applied to as-close-to-real-life-as-possible situations. Part of the class discussion will
focus on the compromises necessary to conducting agency-based research (“design X
would allow us to draw firmer conclusions, but the realities of recruiting from this
population make design Y the best we can do”). In addition, the course will address
situations when the quality of a project has been compromised too much to be of value.
The class also addresses new content specific to program evaluation.
Instructional Goals and Student Outcomes
The Instructor will:
1. Instill an appreciation of the role of program evaluator.
2. Describe types of program evaluation designs, their use, and their relative advantages and
disadvantages. Projects will include summative and formative evaluations, outcome
evaluation, and other common evaluation models such as logic models, cost-benefit
analyses, and goals-based evaluation.
3. Provide an overview of various methodologies and criteria for selecting a particular
methodology used to collect information for use in program evaluation, including
surveys, interviews, document review, observation, focus groups, and case studies.
4. Create an awareness of the political and ethical issues that confront program evaluators
and program evaluations.
5. Implant sensitivity to cultural issues in conducting program evaluations.
6. Educate students on reporting program evaluation findings.
Upon completion of this course, the student will be able to:
Outcomes and Assessment Measures
Outcomes
Measures
1. Comprehend the role of program
Class discussions
evaluator.
Written assignments
2. Estimate and evaluate the
Class discussions
compromises often necessary in
conducting program evaluation.
3. Demonstrate integrated knowledge Written assignments
of various approaches to program
Class discussions
evaluation and their related
methodologies, including
limitations and constraints to
effective program evaluation.
4. Demonstrate the ability to apply
Written assignments
knowledge in critical analysis and
Class discussions
problem-solving during program
2
planning, implementation and
evaluation.
5. Assess program evaluation design
and implementation.
6. Demonstrate an awareness of and
sensitivity to cultural issues in
conducting program evaluations
and the necessity for cultural
competence among evaluation staff.
7. Demonstrate an awareness of and
sensitivity to political and ethical
issues in conducting program
evaluations and the necessity for
protecting human subjects through
ethical research procedures.
8. Develop and present an evaluation
plan, and thoughtfully critique the
evaluation plans and products of
others.
Critique of published program evaluations
Case examples
Class discussions
Critique of published program evaluations
Case examples
Class discussions
Critique of published program evaluations
Case examples
Class discussions
Critique of published program evaluations
Case examples
Class discussions
Place in the social work curriculum: This course is offered in the fall semester of the
concentration year of the MSW program.
Vertical Integration: For full-time students, SWK 628 is taken simultaneously with the
following required social work courses:
SWK 608: Social Policy for Advanced Generalist Practice
SWK 633: Direct Practice II
SWK 634: Social Work Practice IV Organizational Practice (may also have been competed
in the preceding summer)
SWK 6346: Advanced Generalist Practicum I
For part-time students, the following social work course should be taken concurrently:
SWK 608: Social Policy for Advanced Generalist Practice
Horizontal Integration: SWK 628 is the second of three required research courses in the
MSW curriculum. It builds upon foundation knowledge acquired by students in SWK 624.
Whereas SWK 624 enabled students to develop skills to become critical consumers of
research, SWK 628 continues that theme and enables students to gain knowledge and skills
that will allow them to become designers and implementers of evaluation research. It will
also assist students in integrating research into practice. In the following semester in SWK
698, students will conduct their research and disseminate the findings.
Assignments
3
Gains in students’ knowledge and skills will be evaluated through written assignments,
involvement in class and posting of materials in Blackboard. There are 6 graded
assignments. Assignments are due on the stated date, should be composed in a wordprocessing program, and be submitted digitally via Blackboard’s “assignment” feature.
Good academic English is expected. Grades will be lowered for poor grammar, syntax,
and/or spelling, as well as for late papers. Late papers, with or without the instructor’s
permission, will not be eligible for a grade of "A".
Required naming conventions for documents
When creating documents for this class
 Name the file with your first and last name and the name of the assignment
(e.g., RandyMagenEvaluationCritique.docx)
 Always save your file in Rich Text Format (.rtf) or Word (.doc or .docx)
The instructor checks all assignments using a plagiarism checking website. You should
check your assignments using Blackboard’s Safe Assign feature prior to submitting the
assignments for grading. In the assignment section in Blackboard you will find two
submission options for each assignment, one is labeled draft, the other is labeled final.
Use the draft option for checking your work via the plagiarism software.
The assignments require you to apply course content to your social work practice, to
strive toward the integration of theory and practice. All papers should be typed and
double-spaced. Bibliographies may be single-spaced. References must conform to A.P.A.
(6th Edition) bibliographic style.
Assignments are graded on an A-F basis, not points. Converting assignment A-F grades
onto a 4-point scale and multiplying by the pre-determined percentage for that
assignment determine final grades.
Assignment
1. Class Participation
2. Reading quizzes
3. Research Ethics
Training
4. Critique of Evaluation
Report
5. Lecture Wiki
6. Evaluation proposal
Percentage of Final
Grade
15%
10%
5%
Due Date
Every session
Variable
September 14, 2011
25%
October 26, 2011
5%
40%
Variable
December 7, 2011
Assignment #1: Class Participation
Students are expected to be on time, participate actively in class discussions and
exercises; to present program evaluation illustrations from their current work, their past
experience, and from the readings. The grade for class participation (15% of the final
grade) is comprised of four components: attendance, promptness, quantity of
participation, and quality of participation.
4
Assignment #2: Reading Quizzes
Periodically during the semester the instructor will use quizzes, blackboard quizzes, quick writes,
and other short assignments to confirm that students have completed the required reading.
Assignment #3: Training on Protecting Human Subjects
Go to the website http://www.citiprogram.org/ and complete the training on "Basic Course in
Human Subjects Research Curriculum." Choose the course for “University of Alaska Anchorage
Faculty, Staff and Students,” (this can be a little confusing so make sure you choose the correct
course). You will need to register and log in. You can return to the tutorial at any time. You will
need to sign-in using your user name and password each time. It will take you approximately 4-6
hours to complete the 19 modules in this tutorial. When you finish, you will receive a certificate
of completion.
This assignment is fulfilled when you turn in a copy of your certificate from the on-line training.
Be prepared for classroom discussion of the training content. NOTE: It is essential that you
retain a copy of your training certificate in order to complete the research project in SWK 698
(MSW Research Project). Without a copy of the certificate you will be required to complete the
on-line training again prior to initiating your SWK 698 research project.
Completing the training and submitting the certificate by September 14, 2011 will result in a
grade of “A” for this assignment. Training completed or certificates submitted after September
14th will be graded a “C” for this assignment. Failure to complete the training or not turning in
the certificate by the end of the semester will result in an “F” for this assignment. This
assignment counts 5% toward the final grade.
Assignment #4: Critique of Evaluation Research Report
Students will read and write a critique of an evaluation research report. An evaluation
report selected by the instructor will be placed on Blackboard. In writing the critique of
the report students should comment on the following areas (the organization of your
paper may differ from this outline, but you must cover these points):

Literature used to support the evaluation

Provisions for human subjects protection, informed consent, and other
ethical issues

Cultural considerations

Evaluation question (s) and hypotheses

Evaluation methodologies, include threats to reliability and validity of the
methodology

Identification and discussion of evaluation stakeholders

Results and the utility of the results to the program and stakeholders

Clarity and adequacy of presentation
5
The completed critique should be no longer than 10 pages and is due on October 26,
2011. This assignment is worth 25% of the final grade. A grading rubric for this
assignment is posted on Blackboard.
Assignment #5: Lecture Wiki.
(n.) A collaborative Web site comprises the perpetual collective
work of many authors. Similar to a blog in structure and logic, a
wiki allows anyone to edit, delete or modify content that has been
placed on the Web site using a browser interface, including the
work of previous authors. In contrast, a blog, typically authored by
an individual, does not allow visitors to change the original posted
material, only add comments to the original content.
The term wiki refers to either the Web site or the software used to
create the site.
Wiki means “quick” in Hawaiian. Ward Cunningham created the
first wiki in 1995. (Retrieved 8/27/2007 from:
http://www.webopedia.com/TERM/W/wiki.html)
In this course we will use wikis within Blackboard to construct an archive of the
content of class sessions. My hope is that these lecture wikis will evolve over
time, rather than remain a static summary. The lecture summary will evolve
through comments by other students and editing by the students responsible for
the weekly summary.
The class session wiki will be a summary of the main points from the Wednesday
afternoon class session. This wiki should offer a summary of the instructor’s and
students’ comments. Examples and unresolved issues should also be included in
the class session wiki. The class session wiki should be completed by noon on the
Saturday following the Wednesday evening class session. The wiki posting
should be a summary and no longer than 2 pages (about 600-800 words).
In the first class session students will sign-up for one class session wiki. The
lecture wiki counts 5% toward the final grade.
ASSIGNMENT #6: Evaluation Proposal
Students have two choices for the final assignment:
1. Write an evaluation proposal in response to the modified Alaska Mental Health Trust
RFPs posted in Blackboard, or;
2. Develop a program evaluation proposal based upon actual program evaluation needs
identified in your practicum setting, other community-based programs, or a program that
you identify based on your interests. This evaluation proposal must utilize quantitative
and qualitative research methodologies (you must use mixed methods) and include a
measure of consumer satisfaction.
Your completed evaluation proposal, whether choosing #1 or #2 above, should be no longer than
25 pages (double-spaced), excluding the cover sheet and references. This proposal is due on
December 7, 2011.
6
Please follow the following format for your proposal (choice #2) above (the organization of your
paper may differ from this outline, but you must cover these points):
A. Title Page
B. Problem and Objectives (an overview of the evaluation’s goals)
a. Background and need for the evaluation
i. Anticipated use of the results of the evaluation
b. Stakeholder involvement.
c. Dissemination plans and anticipated evaluation products
C. Evaluation question(s) and aims. Identify at least two evaluation questions that could
guide program improvement.
a. Operational definitions
D. Literature Review
E. Provisions for human subjects protection and informed consent
F. Provisions for addressing cultural issues
G. Design and Data-Collection Methods
a. A description of proposed evaluation methodologies
i. Specific qualitative methodology and approach (grounded theory, case
study, etc.)
ii. Quantitative methodology (survey, quasi-experimental design, etc.)
b. Sampling (may be more than one method relating to particular methodologies)
i. Description of setting and proposed participants
c. Measurement and data collection procedures
i. Data sources (identify at least two possible data sources for each
evaluation question and provide a rationale for their selection).
ii. Identify both process and outcome measures. (Remember one
measure must be focused on consumer satisfaction.)
H. Data Management and Analysis
a. Strategies to maximize reliability and rigor
I. Strengths and limitations of the evaluation plan
J. Schedule
a. Timetable for the evaluation
b. Staffing
c. Description of services to be provided
K. Budget (including personnel, equipment, software, supplies, transportation costs,
etc.).
a. Spreadsheet with anticipated costs
b. Budget justification
Your completed proposal should be no longer than 25 pages (double-spaced), excluding the
cover sheet and references. This proposal is due on December 7, 2011.
The following is a tentative scoring rubric for the program evaluation research proposal
assignment (option #2 above). The final scoring rubric may deviate from this system.
Students are urged to not write their papers for the scoring system but for the purpose of
providing as complete an answer as possible.
7
Presentation
Grammar, Syntax & Spelling
Clarity and adequacy of presentation (including organization)
Content
Problem and objectives
Evaluation questions & definitions
Literature review
Provisions for Ethical and Cultural issues
Design and data collection methods
Data management & analysis
Strengths and limitations
Schedule & Budget
Total
3
2
5
5
5
5
10
5
5
5
____
50
The scoring for option #1 is included with the RFP.
Periodically throughout the semester students will have the opportunity to evaluate this
course and the instructor. For many years I have used a version of Rose’s (1984) postsession questionnaire as a way to collect student feedback about the class session. The
post-session questionnaire is a flexible tool that can be used to collect data on a number
of different topics. For at least the past four years I have included a question that asked
students, “What percentage of the readings for today’s class did you complete?” Data
from the anonymous questionnaire are collected at the end of every class session,
compiled and reported to the class at the next meeting.
While it is logical to expect that increased reading leads to greater knowledge and
achievement of course objectives, at this point there is no direct evidence to support or
refute this connection. In this class I will be conducing inquiry process to collect
evidence on the connection between reading required materials and course outcomes.
You will receive a separate consent form to opt-in to allowing me to use your
anonymised data for this evaluation. You should feel free to decline to participate in this
evaluation.
The instructor adheres to University, College, and School policies regarding
accommodations for students with disabilities, religious holidays, plagiarism,
incompletes, and other issues as stated in the University of Alaska Anchorage Catalog,
UAA Student Handbook and School of Social Work Field Education Manual and Student
Handbook. If you experience a disability and would like information about support
services, contact Disability Support Services, located in RH 105, at 786-4530.
Courtesy & Comportment
8
As a courtesy to your classmates and to the instructor, please attend all classes on time
and please turn off all cell phones.
The ringing of cell phones in class can be distracting and interferes with our education. If your cell
phone goes off during class, you are required in the next class session to feed a snack (the professor
prefers cookies) to the entire class. In the event of an emergency situation where there is a need to
keep your cell phone on during class, you must inform the class of this need at the beginning of the
class session; informing the class exempts you from the need to feed the class.
Non-academic use of laptops and other devices are distracting and can disrupt the
learning process for everyone. Neither computers nor other electronic devices are to be
used in the classroom for non-academic reasons. This includes emailing, texting, social
networking, and use of the internet.
(I have adopted the following from Professor Steve Johnson, Associate Professor in the
Communications Department)
Realize that in any communication setting you may encounter people with opinions very
different from yours. While I do not expect you to necessarily accept their opinions as
your own, I demand that you offer them the same opportunity to express their opinions
that you expect. In other words, life in a democracy is one based on tolerance and respect
for the expression of diverse opinions and positions; this class is no different.
“Disruptive behavior” includes but is not limited to talking when someone else has the
floor, being rude to other people, destructive (rather than constructive) criticism of
another, arriving late repeatedly, sleeping during class, and so on. These behaviors have
no place in a college classroom. You have a right to maintain your own opinions and to
disagree with others, but you must do so in a fashion that is conductive to learning and
does not take the form of a personal attack. Minimally, you are expected to treat your
classmates and your instructor in a respectful and professional fashion and they will
return the favor. Any behavior that obstructs or disrupts the classroom teaching and
learning environment will be addressed. Such behavior may result in a reduction of your
final grade in the course and may be referred to UAA authorities if warranted.
In this class we will encounter controversial issues. These issues are controversial
precisely because there exists no “absolute” right or wrong; the fact that people disagree
about these issues means that the issues are unresolved and warrant discussion.
To facilitate discussion of issues that promotes an honest inquiry into the controversy and
avoids degenerating into unproductive attacks, participants in this class will adhere to the
following code of communication conduct.
1. Extend to your fellow classmates the respect and consideration you believed you
deserve.
2. Disagree, but don’t be disagreeable.
3. Do not engage in personal attacks.
9
4.
5.
6.
7.
8.
Do not interrupt others.
Refrain from using abusive language.
Listen to understand, not to rebuke.
Do not engage in side conversations while others are speaking.
Deal with issues in class, not outside of class.
Adapted from: Isenhart, M. W. & Spangle, M. (2000). Collaborative approaches to
resolving conflict. Thousand Oaks, CA: Sage.
Disclaimer
Sometimes it is necessary or advisable to alter the policies and procedures of a
course and/or the nature, number, and value of the assignments or readings in a
course after a course is underway. I reserve the right to make changes such as
these in this course when appropriate.
Texts:
Required texts:
Friedman, M. (2009). Trying hard is not good enough: How to produce
measurable improvements for customers and communities. Charleston, SC:
Booksurge.
Smith, M. J. (2010). Handbook of program evaluation for social work and
health professionals. New York, NY: Oxford University Press.
W. K. Kellogg Foundation (1998). W. K. Kellogg Foundation evaluation
handbook. Battle Creek, MI: Author. (available in Blackboard)
Recommended text:
American Psychological Association (2010). Publication Manual of the American
Psychological Association (6th ed.). Washington, DC: Author
Supplemental Texts: The supplemental texts are resources for obtaining
additional information related to program evaluation.
Altschuld, J. W. (2011). The needs assessment kit. Thousand Oaks, CA: Sage
Publications.
Bloom, M., Fischer, J., & Orme, J. G. (2003). Evaluating practice: Guidelines
for the accountable professional. (4th ed.). Boston, MA: Allyn & Bacon.
Chen, H. T. (2005). Practical program evaluation: Assessing and improving planning,
implementation and effectiveness. Thousand Oaks, CA: Sage.
10
Donaldson, S. I., Christie, C. A., & Mark, M. A. (2009). What counts as credible
evidence in applied research and evaluation practice? Thousand Oaks, CA: Sage
Publications.
Donaldson, S. I. & Scriven, M. (2003). Evaluating social program and
problems. Mahwah, NJ: Lawrence Erlbaum Associates.
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program evaluation:
Alternative approaches and practical guidelines (3rd ed.). Boston, MA: Allyn
& Bacon.
Knowlton, L. W., & Phillips, C. C. (2009). The logic model guidebook: Better strategies
for great results. Thousand Oaks, CA: Sage Publications.
Patton, M. Q. (2008). Utilization focused evaluation (4th ed.). Thousand Oaks,
CA: Sage Publications.
Posavac, E. J. & Carey, R. G. (2006). Program evaluation: Methods and case studies (7th ed.).
Old Tappan, NJ: Pearson.
Royse, D., Thyer, B. A., & Padgett, D. K. (2006). Program evaluation: An
introduction (5th ed.). Belmont, CA: Wadsworth.
Russ-Eft, D., Bober, M. J., da la Teja, I., Foxon, M. J., Koszalka, T. A. (2008).
Evaluator competencies: Standards for the practice of evaluation in
organizations. San Francisco: Jossey-Bass.
Unrau, Y. A., Gabor, P. A., & Grinnell, R. M., Jr., (2001). Evaluation in the
human services. Itasca, IL: F. E. Peacock.
Weinbach, R. W. (2005). Evaluating social work services and programs.
Boston, MA: Allyn & Bacon.
Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (Eds.). (2004). Handbook of
practical program evaluation (2nd ed.). San Francisco, CA: Jossey-Bass.
Yarborough, D. B., Shula, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The
program evaluation standards: A guide for evaluators and evaluation users (3rd
ed.). Thousand Oaks, CA: Sage Publications.
AEA 365: A tip-a-day by and for evaluators blog: http://aea365.org/blog/
Topical Course Outline
1. Introduction to Program Evaluation in Social Work Settings
a. Purpose and Uses of Program Evaluation
b. Cultural Considerations
11
c. Politics & Ethics in Program Evaluation
2. Approaches to Program Evaluation
a. Summative Evaluation
b. Formative Evaluation
c. Outcome Evaluation
d. Goals-Based Evaluation
e. Logic Models
f. Cost-benefit Analyses
g. Participatory evaluation
h. Other relevant Evaluation Models
3. Program Evaluation Methodology
a. Criteria for Choosing Methodology
b. Methods
i. Surveys
ii. Interviews
iii. Document review
iv. Observation
v. Focus groups
vi. Case studies
vii. Analysis of available data
4. Reporting Writing Program Evaluations
12
Course Outline
Note: Items not in the textbooks can be found in Blackboard
Class Date
Reading
Session
1
August 31
2
September 7
3
September 14
4
September 21
5
September 28
6
October 5
Smith chapter 1
Friedman Introduction & chapter 1
Kellogg chapters 1 & 2
Review Program Evaluation Standards
(about 52 pages)
Smith chapter 2 (p. 29-41 only) & chapter 3
Kellogg chapter 3
Read Health Families Alaska RFP (pages
28-51 only)
Anchorage Coordinated Resources Project
RFP (starting at section 12 on page 10
through page 17 and pages 19-20)
(about 73 pages)
Smith chapter 2 (p 41-55)
Article by Caldwell et al.
Report “Advocates perspectives on
evaluation”
(about 55 pages)
Smith chapter 4 & 6
Article by Hodges & Hernandez
(about 114 pages)
Smith chapters 5 & 7
Kellogg chapter 4
(about 96 pages)
Topics & Assignments
Introductions. Overview of course.
What is program evaluation?
Purposes and uses of program evaluations, program
evaluation process, evaluation questions, utility/uses
of program evaluation,
Purposes and uses of program evaluations (continued),
planning evaluation, evaluator roles,
stakeholders/clients, types of evaluation, evaluation
questions (continued), Responding to RFPs
Ethics of program evaluations
Training on protecting human subjects due
Politics of program evaluations
Program theory, logic models, evaluability assessment
Process evaluation, Implementation evaluation,
performance monitoring, needs assessment,
measurement, process measurement
13
7
October 12
8
October 19
9
October 26
10
November 2
11
November 9
12
November 16
13
November 23
November 30
Smith chapter 10
Kellogg chapter 5 (pages 69-96 only)
Dore article
(about 98 pages)
Smith chapter 8.
Read evaluation of Health Families Alaska
including the separate document “Appendix
B.”
(about 138 pages)
Friedman chapters 2 & 3
Kellogg (chapter 5 pages 47 – 68)
Article by Knickman & Hunt
(about 83 pages)
Friedman chapters 4, 5, & 6
(about 60 pages)
Friedman chapters 8, 9 and Epilogue
Article by Scott & Dixon
(about 15 pages)
RFP for evaluation of family care court
Hornby Zeller proposal
Evaluation report on the family care court
No Class Thanksgiving holiday.
Smith chapter 9.
Friedman chapter 7
Kellogg chapter 5 (pages 96-104)
Scan the report “Locating the dropout crisis”
and the newspaper article “dropout label
dismissed…”
(about 77 pages)
Evaluation reports, meta-analyses & systematic
reviews, consumer satisfaction
Paradigms: quantitative & qualitative, quantitative
approaches – quasi-experimental and experimental
designs.
Accountability, costs of evaluations
Evaluation critique due
ACRP evaluation, observation, surveys, qualitative
methods, measurement and consumer satisfaction
revisited
Expert judgment, role-playing, cost-benefit and cost
effectiveness.
Start to finish, the evaluation of the family care court.
Analysis of available data, implementing evaluations,
statistical analysis and reporting.
14
14
December 7
None
Review and course wrap-up.
Evaluation proposal due by midnight
15