Western Carolina University
Master of Public Affairs
Annual Assessment Report for 2007-2008
Primary Contact Name/Info:
Chris Cooper (ccooper@email.wcu.edu)
In our 2006-2011 Assessment Plan, we listed 8 learning outcomes. This year we will assess our
first outcome.
1. Assess research design and determine the quality and reliability of research used in public
service.
2. Understand how to organize people to maximize productivity in the public sector.
3. Analyze the advantages and disadvantages of policy choices.
4. Understand various types of public budgeting.
5. Understand the legal issues that affect public affairs.
6. Have a basic command of the concepts and theories in the field.
7. Communicate effectively through oral and written means.
Overview
We address student’s ability to assess research design and determine the quality and reliability of
research used in public service in three ways—two are within course assessments and one is an
out-of-course assessment. In the policy analysis class, each student is required to evaluate a
research article and discuss how it relates to public sector decision-making. All students are
required to take this course and all students in the program must complete this activity. Second,
students are required to conduct an original research project in their research methods in public
affairs course. Part of this project requires students to assess current research on the topic, and
another part requires the students to discuss how their original research relates to public affairs.
Finally, all students must pass comprehensive exams which assess how well students understand
research design and how well they are able to assess the quality and reliability of research used in
public service. Our comprehensive exams are 6 hour closed book, closed note exams. Students
either receive a Fail, Low Pass, Pass, or High Pass on each section. Students who receive a
Failing grade on any section are required to retake that section and receive a passing grade before
graduation. Our comprehensive exams are a rare opportunity for us to get together as an MPA
faculty and see how much our students are learning and how well they meet our learning
outcomes. The within course assessments provides excellent opportunities to see how students
are progressing while taking our courses; our comprehensive exams provide an excellent
opportunity for out-of-course assessment to determine how well our students are able to apply
their knowledge after finishing their core coursework.
Changes
For the second year in a row, an MPA faculty committee has graded the research methods
comprehensive exams. Our old system only required one student to grade the section. This new
system gives each MPA faculty member an opportunity to evaluate how well our students
understand and evaluate research and research design. Similar to a National Science Foundation
grant committee, we all grade the section separately, then we get together and discuss the results
together and decide on one final grade. Students who fail receive detailed feedback on their
answers.
Results
Most of our students did quite well in their research methods comprehensive exams. In the fall 5
passed, 3 low passed and two failed. In the spring, 2 received high passes, 2 passed, 2 low
passed and 2 failed. All students who failed were given extensive feedback and given the
opportunity to take the exams again approximately one month after failing. Thus far, two of the
students have retaken and passed their exams (one of whom has graduated) and the other two are
studying to retake their exams in the future. Overall, we were quite satisfied with the
substantive content our students are learning and we feel that they reflect well on most of our
intended learning outcomes.
As for the within course assessments, the students in the policy analysis course performed quite
well in their research evaluation project. All students received As or Bs on this assignment,
demonstrating reasonable proficiency with the material. On the research methods papers,
students once again demonstrated excellent proficiency with the material—approximately ½ of
the students received As on the assignment, and just under ½ received Bs. Only two students
received Cs on that assignment. Overall, our within course and out of course assessment suggest
that our students are able to “Assess research design and determine the quality and reliability of
research used in public service.”
Potential Problems and Implementation Plan
Our assessment techniques raise three problems and three potential solutions. First, assessing
research design should be a skill developed in each of our courses—not just two. It is quite
likely that this is already taking place. Anecdotally, we know that professors in many of our core
classes require students to evaluate research similar to how they do so in policy analysis. Next
year we will ask each of the professors in our core classes to assess research skills in some way
and report back to the MPA Director how this occurs.
The second problem is that although many of our students are intellectually capable of assessing
research, many to work on communicating this information to a wider audience. To help address
this problem, we will work hard to get more of our students involved in the Graduate Research
Symposium on campus. This Symposium provides an excellent opportunity for our students to
present their research—demonstrating that they can communicate research findings to an
interdisciplinary audience.
Third, some of our faculty who do not teach the research methods class are uncertain how to
grade the exam. This leads to some inconsistent grading and assessment. Next year we will
require the faculty member who writes this section of the exam to distribute a grading rubric to
all faculty who grade the comprehensive exams. This should lead to more consistent grading and
a more open process.