Graduate Degree Program Assessment Progress Report Cover Sheet:
Degree: Master of Public Administration For Calendar Year: 2014
(Date submitted to college committee: 2/26/2015_By: Dr. Jerry Stevenson
(Date posted on college assessment website:__________)
Overall Rating:__________________________________
(1) Student learning goal(s) addressed this year:
The policy analysis paper assessment seeks to provide data to inform the MPA
program’s faculty about the extent to we are successful in meeting our program’s
academic objectives, including NASPAA’s universal competencies. In turn, this data
informs our process for reviewing and updating the program’s curriculum, teaching
approaches and methods, academic advising and other student support services.
In particular, the policy analysis assessment provides data on aggregate student
learning in four areas:
Goal 1A: Knowledge of public service principles
Goal 2A: Developing, implementing and evaluating policies, programs, and
projects
Goal 2B: Analyze, synthesize, think critically, solve problems and make decisions
Goal 4A: Written communication skills
(2) Learning Outcomes/objectives for those goals addressed this year.
Goal 1A: Knowledge of Public Service Principles: Learning outcomes
1.5 Understand the relationship between politics and administration;
1.6 Examine conflicting public values;
1.9 Understand the role of efficiency, equity, accountability and transparency in the
public and nonprofit environment; and,
Goal 2A: Developing, implementing and evaluating policies, programs, and projects:
Learning Outcomes
2.1 Develop and implement a policy and/or procedure;
2.2 Examine the relationships between public policies and the political process;
2.3 Identify and analyze relevant information to assess the effectiveness of policies;
2.4 Examine ethical and legal perspectives and other underlying assumptions that
might impact the policy process;
Goal 2B: Analyze, synthesize, think critically, solve problems and make decisions: Learning
Outcomes
2.9 Identify appropriate research methods;
2.11 Employ analytical tools and statistical techniques appropriate to problem;
2.12 Collect, analyze, interpret data for addressing a problem;
2.13 Draw conclusions and make recommendations for a policy, organizational, or
managerial situation, issue, or problem; and,
2.14 Synthesize information to produce innovative alternatives and solutions.
2.26.15 jgs
-1-
Goal 4A: Written Communication Skills: Learning Outcomes
4.1 Write a professional paper that reflects an appropriate structure and format for the
type of paper selected, i.e. research, policy analysis, or case study;
4.2 Write a clear, concise, comprehensive, grammatically correct, free from spelling
errors, and properly referenced paper;
4.3 Write an executive summary that presents briefly the purpose of the paper,
supporting literature, methodology and analyses and major results;
4.4 Use illustrations, tables, and graphics effectively.
(3) Courses & Activities where assessed:
The written papers represent student learning at a point midway through a typical
student’s curriculum and is based on an assignment that all students complete in our
policy analysis core course (PADM 7363). Hence it informs a review of curriculum
covering the first year of the program and the second year required policy analysis
course.
(4) Assessment method
Two faculty members evaluated each policy analysis paper (31 papers in all) completed
for PADM 7363 (Public Policy Analysis) in Fall 2012 and Fall 2013 using the rubric
shown in Appendix A:
(Score: Clearly evident (2); Partly evident (1); Not evident (0); Not applicable (N/A)
Each item in the rubric is matched with a MPA program learning goals and NASPAA
universal competencies. Though our policy analysis paper assessment primarily focuses
on Goals 2 and 4a (Research Skills and Written Communication Skills), it also provides
an opportunity to assess aspects of Goal 1A (Knowledge of Public Service Principles).
The MPA faculty adopted the following criteria to identify rubric items in which overall
student performance is deficient and/or superior:
Significantly deficient: 30% or more of student papers (10 for this survey) were
rated as 0 (not demonstrated) on the item by at least one faculty member
reviewing the paper.
Deficient: 20% or more of student papers (8 for this survey) were rated as 0 (skill
not demonstrated) on the item by at least one faculty member reviewing the paper.
Superior: 40% or more of student papers (12 for this survey) were rated as 2
(skill clearly evident) by both faculty members reviewing the paper AND at least
80% of student papers were rated as 1 or 2 by both faculty reviewers (i.e we are
not “deficient” on the category).
In addition, paper reviewers were asked to indicate whether a criterion was “not
applicable” to the paper. A high number of such ratings may indicate that the paper
assignment is not adequate for assessing that outcome and hence a need for a review of
the relevance of this learning outcome or whether the paper assignment itself should be
reconsidered in future years. Hence:
Unclear: 20% or more of student papers were rated as NA by at least one faculty
member.
2.26.15 jgs
-2-
Rubric items for which aggregate student performance meet none of these standards may
be taken to be “satisfactory”.
(5) Assessment Findings:
SUMMARY RESULTS
Rubric item
Overall student
performance
Goal 1A: Knowledge of Public Service Principles
1.5 Understand the relationship between politics and
administration
Addresses the concerns and questions of relevant
stakeholders and policymakers
Design of policy recommendation reflects political
feasibility
1.6 Examine conflicting public values
Paper is organized effectively to make an argument about
the nature, scope and causes of a public problem; evaluate
outcomes from a public policy; and/or make a policy
recommendation.
Policy recommendation includes clearly stated goals
Design of policy recommendation (i.e. policy instruments
and institutional arrangements) adheres to best practices
1.9 Understand the role of efficiency, equity,
accountability and transparency in the public and
nonprofit environment
Uses appropriate values criteria to evaluate policies and
make recommendations
Demonstrates transparency in the assumptions and values
judgments on which it bases recommendations
Demonstrates transparency in choices concerning
measurement
Demonstrates transparency in choices of analytic
approaches and methods (i.e. by assessing strengths and
weaknesses of the research design)
Goal 2A: Developing, implementing and evaluating
policies, programs, and projects
2.26.15 jgs
-3-
Satisfactory
Seriously deficient
Superior
Satisfactory
Deficient
Satisfactory
Seriously deficient
Seriously deficient
Unclear
2.1 Develop and implement a policy and/or procedure
Policy recommendation includes clearly stated goals
Design of policy recommendation (i.e. policy instruments
and institutional arrangements) adheres to best practices
2.2 Examine the relationships between public policies and
the political process
Design of policy recommendation reflects political
feasibility
2.3 Identify and analyze relevant information to assess the
effectiveness of policies
Evaluates policy outcomes and recommendations using
appropriate evidence
2.4 Examine ethical and legal perspectives and other
underlying assumptions that might impact the policy
process
Uses appropriate values criteria to evaluate policies and
make recommendations
Satisfactory
Deficient
Seriously deficient
Seriously deficient
Satisfactory
Goal 2B: Analyze, synthesize, think critically, solve
problems and make decisions
2.9 Identify appropriate research methods
Uses relevant scholarly and public policy literature to
support arguments and analysis
Constructs tables, charts and illustrations with adherence to
best practices
Overall, uses appropriate quantitative and qualitative
evidence to support arguments and analysis
2.11 Employ analytical tools and statistical techniques
appropriate to problem
Interprets tables, charts and illustrations competently
Uses other descriptive methods of data analysis correctly
and as appropriate
Uses and interprets inferential statistical methods (e.g.
regression analysis) correctly and as appropriate
2.12 Collect, analyze, interpret data for addressing a
problem
2.26.15 jgs
-4-
Satisfactory
Unclear
Deficient
Unclear
Seriously deficient
Unclear
Uses appropriate data sources and data gathering methods
to develop measures
2.13 Draw conclusions and make recommendations for a
policy, organizational, or managerial situation, issue, or
problem
Conclusions and recommendations logically follow from
arguments and evidence
2.14 Synthesize information to produce innovative
alternatives and solutions
Overall, arguments and analysis generalize from the
evidence and literature to generate original ideas,
conclusions or recommendations
Deficient
Satisfactory
Deficient
Goal 4A: Written Communication Skills
4.1 Write a professional paper that reflects an appropriate
structure and format for the type of paper selected, i.e.
research, policy analysis, or case study
Paper is organized effectively to make an argument about
the nature, scope and causes of a public problem; evaluate
outcomes from a public policy; and/or make a policy
recommendation.
Superior
4.2 Write a clear, concise, comprehensive, grammatically
correct, free from spelling errors, and properly referenced
paper
Paper is free of mechanical problems (i.e. grammatical and
spelling errors)
Paper's writing style (word choice, paragraph and sentence
construction) effectively communicates its argument and
analysis to an average reader
Superior
Superior
4.3 Write an executive summary that presents briefly the
purpose of the paper, supporting literature, methodology
and analyses and major results
Paper's executive summary communicates the paper's goals, Unclear
methods, and conclusions concisely and effectively to an
average reader
4.4 Use illustrations, tables, and graphics effectively
2.26.15 jgs
-5-
Constructs tables, charts and illustrations with adherence to
best practices
Interprets tables, charts and illustrations competently
Unclear
Unclear
(6) What conclusions were drawn and what decisions were made as a result? How
were stakeholder groups involved?
Overall, the results of the assessment point to some learning outcomes in which
students in our program appear to meet significant success by the time they complete the
policy analysis paper assignment. But at the same time, they highlight several learning
outcomes where too many students fall short of our expectations. In particular:
a) Our program is meeting its objectives with respect to helping student to develop
professional-quality writing, at least with respect to mechanics and style. This suggests
that the writing-intensive nature to our curriculum and the feedback we provide students
on writing mechanics and style have been largely effective in helping students to improve
their writing quality.
b) The policy analysis assignment itself has had uneven expectations with respect to
providing some expected components, particularly:
--- An executive summary
--- Use of descriptive data analysis, particularly charts and tables
--- Use of inferential statistical analysis (e.g. regression analysis)
Because of this, it is not yet possible to fully assess student learning on these criteria.
This points to a need for an agreement among those teaching PADM 7363 about skills
that each student should be expected to demonstrate as part of the assignment.
c) Students in our program are falling short in showing that they can apply some types of
public service principles in professional-style analysis. In the policy analysis paper
assignment, students have an opportunity to demonstrate that they understand the
relationship between the political process, mass public and public administration in
whether and how their analysis takes into account the interests of differing stakeholders
and in the extent to which they consider political feasibility in analyzing policy options.
Though most students succeeded at considering stakeholders, a large number of students
did not incorporate political feasibility into their analysis. Very likely this reflects the fact
that most students do not take the Politics and Bureaucracy course until the program’s
second year and so may not need to be addressed here.
In addition, students have an opportunity to demonstrate that they have internalized
values of transparency in governance through the extent to which they are transparent in
the construction and sources for their measures; their methods; and the assumptions and
values they adopt in interpreting data and drawing conclusions. Since transparency is a
value we expect students to adhere to when they graduate, we should expect that they
demonstrate this consistently in writing for our program. The absence of transparency in
many student papers likely reflects a lack of understanding over how to be transparent
when it comes to one’s research methods and policy analysis. That is, students may
understand the value of transparency in the abstract but are unclear how to operationalize
2.26.15 jgs
-6-
it in their work. One approach may be to review curriculum in the research methods and
policy analysis courses with respect to how one should describe methods of analysis. In
addition, some students may not yet grasp the extent to which value judgments underlay
policy evaluation and recommendation arguments. These are topics that could be
reinforced in policy-related writing assignments across the curriculum.
d) The results also suggest that too many of our students fall short in their capacity to
develop, implement and evaluate policies, programs and projects. Significant numbers of
students fell short in their ability to support analyses with appropriate types of evidence
and to identify, describe and analyze policy designs (i.e. institutional arrangements and
policy instruments) as part of evaluating policy recommendations. These are skills
students should be well developed by the time they finish the policy analysis course.
This may indicate a need to integrate evidence-based argumentation across the
curriculum, beyond the methods and policy analysis courses.
e) In addition, the results indicate that too many students fall short in demonstrating depth
of analysis: critical thinking, synthesis, use of evidence and capacity to generalize from
specific instances. This could reflect first the lack of comfort students have with using
quantitative evidence as part of a research project. More generally, however, it suggests
that while students are writing well in terms of mechanics and style, their writing often
lacks analytic depth. Given the writing-intensive nature of our curriculum, this is a bit
puzzling. But it does suggest that we may need to provide additional feedback to students
concerning the depth of their writing, as well as additional instruction in how to think and
write critically.
f) Finally, it is worth considering the extent to which the policy analysis rubric and the
overall assessment process produces useful measures of student learning outcomes. As
our first time with this process, we should take time to evaluate the process itself and
make appropriate revisions to the rubric and how we interpret the data we collect with it.
At our September 30, 2014 faculty meeting, we decided to focus on helping students
improve skills in those areas deemed to be either “deficient” or “seriously deficient.”
Consequently, we are taking steps to develop student competency in using quantitative
evidence by integrating qualitative analysis across other courses in the MPA curriculum.
We will also increase our students’ exposure to critical and argumentative readings (as
opposed to textbooks and content-centered readings) in our introductory course, PADM
7301, Profession of Public Administration. This change is made to improve the student’s
depth of analysis and critical thinking skills
Stakeholder Involvement
1) Alumni Survey: An alumni survey is conducted every other fall semester. The
results of our last survey lead to the creation of a MPA Student Lounge and
working with the Dean’s Office to update the equipment and technology available
in Ross Hall 213 that is used frequently by MPA faculty.
2.26.15 jgs
-7-
2) Joint IOG/MPA Advisory Board: This joint advisory board in in the process of
being reconstituted. Names for recommended board members have been solicited
and plans are underway to have a preliminary meeting during Summer 2015.
2.26.15 jgs
-8-
Appendix A
Policy Analysis Paper Assessment Rubric
MPA Program Methods and Writing Assessment
Spring 2014
Evaluator: _______________________________________
___________
Year/Paper #:
The policy analysis paper assignment in PADM 7363 (Policy Analysis) serves as an opportunity for
students to demonstrate professional-level competence at making arguments about a public problem and
policy supported with original data analysis, including problem definition and analysis, policy evaluation
and analysis of policy recommendations. Each student draws on his/her skills and knowledge in research,
quantitative analysis, critical thinking and writing from the first half of their MPA experience to complete
this paper. Please indicate the extent to which the paper demonstrates professional-level competence in the
following ways:
Score: Clearly evident (2); Partly evident (1); Not evident (0); Not applicable (N/A)
Learning
outcome
Criterion
Score
Holistic criteria
1.6 and 4.1
Paper is organized effectively to make an argument about the nature, scope and
causes of a public problem; evaluate outcomes from a public policy; and/or make
a policy recommendation.
4.2
Paper is free of mechanical problems (i.e. grammatical and spelling errors)
4.2
Paper's writing style (word choice, paragraph and sentence construction)
effectively communicates its argument and analysis to an average reader
4.3
Paper's executive summary communicates the paper's goals, methods, and
conclusions concisely and effectively to an average reader
1.5 and 2.2
Addresses the concerns and questions of relevant stakeholders and policymakers
2.13
Conclusions and recommendations logically follow from arguments and evidence
2.14
Overall, arguments and analysis generalize from the evidence and literature to
generate original ideas, conclusions or recommendations
Criteria on analytic methods
2.9
Uses relevant scholarly and public policy literature to support arguments and
analysis
1.9
Demonstrates transparency in choices concerning measurement
2.26.15 jgs
-9-
1.9
Demonstrates transparency in choices of analytic approaches and methods (i.e. by
assessing strengths and weaknesses of the research design)
2.12
Uses appropriate data sources and data gathering methods to develop measures
4.4 (and 2.9)
Constructs tables, charts and illustrations with adherence to best practices
4.4 (and 2.11)
Interprets tables, charts and illustrations competently
2.11
Uses other descriptive methods of data analysis correctly and as appropriate
2.11
Uses and interprets inferential statistical methods (e.g. regression analysis)
correctly and as appropriate
2.9
Overall, uses appropriate quantitative and qualitative evidence to support
arguments and analysis
Criteria on policy recommendation and design
1.6 & 2.1
Policy recommendation includes clearly stated goals
1.6 & 2.1
Design of policy recommendation (i.e. policy instruments and institutional
arrangements) adheres to best practices
1.5 & 2.2
Design of policy recommendation reflects political feasibility
2.3
Evaluates policy outcomes and recommendations using appropriate evidence
1.9 and 2.4
Uses appropriate values criteria to evaluate policies and make recommendations
1.9
Demonstrates transparency in the assumptions and values judgments on which it
bases recommendations
Overall, the paper demonstrates professional-level competence in policy analysis
2.26.15 jgs
- 10 -