SPNHA Learning Assessment 2008
Outcomes/Objectives
MPA
PNA major
• Good Communication
Skills
• Good Critical Thinking
Skills
• Post Graduation
Employment
• Successful &
Valuable Internship
Experience
• Post graduation
employment /
graduate school
• Meaningful Internship
• Communication Skills
• Critical Thinking
Measures
MPA
PNA major
• Writing Skills Analysis
• Critical Thinking
Analysis
• Internship Analysis
• Job Attainment Rate
• Internship Approval
Rate
• Writing Skills Analysis
• Critical Thinking
Analysis
• Internship Analysis
• Job Attainment Rate
Writing Skills Analysis
Actions
MPA
PNA major
• Writing Skills Analysis
Rubric
• Applied to papers
from PA619
• Writing Skills Analysis
Rubric
• Applied to papers
from PA307
Communication Skills Rubric
Criteria
3 = Good
2 = Satisfactory
1 = Needs Improvement
0 = Absent
Content
Thoughtful and insightful
analysis of a clearly present
thesis or purpose. Reader
gains good insight
Basic analysis of a thesis or
purpose. Reader gains
sufficient insight.
Some analysis of a thesis
or purpose. Reader gains
few insights.
Paper does not identify
thesis or purpose. Analysis
vague or missing. Reader is
confused or misinformed.
Organization
Ideas arranged logically.
Flow smoothly and are
topics links. Reader can
follow reasoning.
Ideas are, for the most part,
arranged logically and
linked. Reader can follow
most of the reasoning.
Writing is not logical and
ideas sometime fail to
make sense. Reader needs
to work to figure out
meaning.
Little semblance of logical
organization. Reader
cannot identify reasoning.
Tone
Tone is consistently
professional and
appropriate of audience.
Tone is generally
professional and mostly
appropriate for audience.
Tone is occasionally
professional or occasional
appropriate for audience.
Tone is not professional. It
is inappropriate for
audience and purpose.
Mechanics
Writing is free or almost
free of errors.
Occasional errors in writing,
but they do not represent a
major distraction.
Writing has numerous
errors and distracts the
reader.
Errors are so numerous
that they obscure meaning.
References
Sources of presented
evidence are clearly and
fairly represented
Complete references are
generally present.
Occasional and/or
incomplete references are
provided.
References are not or
mostly not presented.
Format
A recognized format is
correctly followed
A recognized format is
generally followed; a few
mistakes, however.
Format of document
reflects incomplete
knowledge of standard.
No standardized format
followed.
Writing Skills Analysis
Goals
MPA
• Papers in each
assessment category rated
satisfactory or better: 75%
• All papers rated
satisfactory or better: 75%
• Average total
communication skills score:
12 points
PNA major
• Papers in each
assessment category rated
satisfactory or better: 65%
• All papers rated
satisfactory or better: 65%
• Average total
communication skills score
of 9
Writing Skills Analysis
Findings
MPA
• All of communication skill
targets are achieved with
exception of “content” and
“mechanics.”
• “Content” is close, but
“mechanics” is substantially
lower than the established
target.
• Average paper’s total score
12.7 points
PNA major
• All of the communication
skills targets fall short,
except "organization.“
• “Mechanics," "references,"
and "format" are worst.
• Average paper’s total score
11 points
Writing Skills Analysis
Actions
MPA
PNA major
• A "Writing Mechanics" module
• A module should be developed
will be developed and introduced
for PA420 that focuses the
in all sections of PA520.
students attention on:
– making an argument
– using good information
– citing that information.
• More interaction is needed
among the faculty to develop
greater consistency in evaluating
critical thinking and writing skills.
The next evaluation rounds will
require faculty to compare their
scores on the same paper.
Writing Skills Analysis
Committee Comments
MPA
none
PNA major
“You did a very good job
developing a rubric to standardize
the evaluation process. It may be
useful in the future to expand the
scale to allow for more subtle
differences between students, for
example a 1-5 or 1-10 scale.
There were several typographical
and grammatical errors in this
entry that may have occurred
during importing of the text.”
Critical Thinking Analysis
Measure
MPA
PNA major
• Critical Thinking
Analysis Rubric
• Applied to papers
from PA619
• Critical Thinking
Analysis Rubric
• Applied to papers
from PA307
Assessment Rubric for Critical Thinking Skills
Criteria
3 = Good
2 = Satisfactory
1 = Needs Improvement
0 = Absent
Quality of
evidence
Completely evaluates
the evidence and
sources of evidence.
Can distinguish
between fact, inference,
opinion, and value
judgment.
Adequately evaluates
the evidence and
sources of evidence.
Can usually distinguish
between fact, inference,
opinion, and value
judgment.
Superficially evaluates
evidence and sources.
Often substitutes
opinion and value
judgment for fact and
inference.
Merely repeats
information provided or
denies evidence with no
justification. Confuses
facts with inferences,
opinions, and value
judgments.
Supports
arguments
Able to support a
position with a
substantial amount of
information, little or no
bias, and valid
arguments.
Able to support a
position with adequate
information and few, if
any, logical fallacies.
Able to support a
position with some
evidence.
Cannot articulate a
position or offers no
supporting evidence.
Used models
Accurately and
completely analyzes
purpose using
appropriate models;
finds connections
between the material
and the models.
Satisfactorily analyzes
purpose using
appropriate models;
misses minor
connections.
Attempts to use
appropriate models, but
gives attention to only
the most significant or
obvious connections.
Models are misapplied
or not used.
Recognizes
alternatives
Recognizes all plausible
alternative viewpoints or
solutions; completely
considers each one
before choosing.
Recognizes alternative
solutions or viewpoints;
gives them some
consideration.
Recognizes a few
alternative solutions or
viewpoints; but
dismisses them without
justification.
Unable to recognize
alternative solutions or
viewpoints.
Critical Thinking Analysis
Goals
MPA
• Papers in each
assessment category rated
satisfactory or better: 70%
• All papers rated
satisfactory or better: 70%
• Average total
communication skills score:
7 points
PNA major
• Papers in each
assessment category rated
satisfactory or better: 65%
• All papers rated
satisfactory or better: 65%
• Average total
communication skills score
of 6
Critical Thinking Analysis
Findings
MPA
PNA major
• We met our target of 70 percent • With exception to "recognizing
of all papers rated as satisfactory
alternatives," our target of a 65
or better in all critical thinking
percent satisfactory rating was
areas.
achieved or nearly achieved.
• All of the papers assessed
• While two categories (“quality of
received a satisfactory or better
evidence” and “supports
rating.
arguments”) are two points lower
than the established target, the
• Average paper’s total score: 7.9.
“use of models” is substantially
lower than the target of 65
percent.
• Average paper’s total score: 6.8.
Critical Thinking Analysis
Actions
MPA
PNA major
• A module should be developed
for PA420 that focuses the
students attention on:
– making an argument
– using good information
– citing that information.
• Faculty are not satisfied with the
critical thinking rubric used. They
will adopt a modified rubric at the
Fall retreat
• More interaction is needed
among the faculty to develop
greater consistency in evaluating
critical thinking and writing skills.
Critical Thinking Analysis
Committee Comments
MPA
none
PNA major
• “Again, it may be useful in the future to expand the scale
to allow for more subtle differences between students,
for example a 1-5 or 1-10 scale. How you plan to apply
the specific rubric citeria (sic) was also not clear.”
• “Several measures appeared to be based on rubric
which is to be commended. The criteria for applying
these rubrics and the scale chosen were somewhat
confusing. Perhaps an example of how the rubric was
applied to arrive at the final score for each student would
help clarify the process. Measures seems to rely heavily
on one course PA307 and on writing samples. Are there
other measures that could provide information regarding
critical thinking skills?”
Internship Analysis
Measure
MPA & PNA majors
Internships were evaluated on:
• Internship provided exposure to management decision
making
• Internship provided opportunity to participate in various
parts of the organization's operation, including
budgeting, core work and outcome assessment
activities.
• The host organization's work matched with the intern's
primary career objectives.
• Student reported a good internship experience.
• Host organization reported satisfaction with the intern's
knowledge, attitude and reliability.
Internship Analysis
Goals
MPA & PNA majors
• 70% of internships should be satisfactory
or better in all five categories.
• 80% of host organizations should report
that the students were satisfactory interns.
Internship Analysis
Findings
MPA
PNA major
• 65% of internships had
• All host organizations were
satisfactory scores on all five
satisfied with their interns.
criteria. One issue was that
• However, only 50 percent of
some host organizations,
internships had satisfactory
although well intentioned, were
scores on all five criteria. Well
not prepared to use their interns
short of the 70% desired. One of
effectively.
the most obvious problems was
• Although all host organizations
that students were taking
reported being satisfied with their
convenient internships rather
interns, both students and host
than strategic internships. Thus
feedback indicated that students
they did not match well with their
needed better preparation in
career objectives.
budgeting, finance and
fundraising. .
Internship Analysis
Actions
MPA
PNA major
• Students taking internships for
credits should have at least on
year of course work(9 credits)
including finance(PA615 or
PA632)
• We will establish a relationship
with organizations that can
provide quality internship
experiences on a regular basis.
we will place interns there on a
rolling basis. We will initiate 4
such pipeline internships per
year for the next few years.
• Beginning in Fall 2008 SPNHA
and Career Services will offer
each semester a workshop
exploring internship and volunteer
options. Governmental and
Nonprofit employers and
agencies will be invited to
campus in the fall and spring to
conduct informational and
recruitment sessions. These will
be organized opportunities for
local governmental and nonprofits
to meet with undergraduates for
internship, volunteerism and
entry-level career placement. And
for undergraduates to get tips on
seeking out, applying for and
getting the most out of internship
experiences.
Internship Analysis
Committee Comments
MPA
none
PNA major
“The criteria applied to
evaluate this measure is
not clear. Will you be
polling.”
Job Attainment Rate
Measure
MPA & PNA majors
• Ever year, GVSU Career Services surveys
recent graduates. Taking the most recent
two years of data available, we looked at
the rate at which recent MPA and PNA
graduates were employed in the public or
nonprofit sector, or where attending
graduate school. .
Job Attainment Rate
Goals
MPA
PNA major
We expect that our recent MPA
graduates will be employed in the
public or nonprofit management
field (or attending graduate school)
at a rate equal to or higher than
– graduates from the MBA
program
– equal to or higher than all
graduates from GVSU`s
Master`s and Doctoral
programs.
We expect that our recent PNA
graduates will be employed in the
public or nonprofit management
field or attending graduate school at
a rate
– equal to or higher than
graduates from business
management
– equal to or higher than
graduates from political
science
– equal to or higher than all
GVSU undergraduates
Job Attainment Rate
Findings
MPA
PNA major
• 84% of recent MPA
• 80%of recent PNA
graduates are employed in
graduates are employed in
their field or in graduate
their field or in graduate
school. That is on par with
school. That is on par with
the MBA program.
Political Science (more
However, it is short of the
working, less in grad
95% rate for all GVSU
school) and with all GVSU
Masters and Doctoral
BS/BA graduates. It is
graduates
higher than graduates in
business management.
Job Attainment Rate
Actions
MPA
none
PNA major
none
Job Attainment Rate
Committee Comments
MPA
none
PNA major
“This measure seems like a good one for evaluating
success after graduation. Changes over time will be very
important as well as consitency (sic) in polling of
alumni.”
“Some of your objectives are worded broadly to the point
that some sound more like program goals rather than
student learning objectives. For example, objective 1 “to
have graduates able to obtain a job in public or nonprofit
management, or enter graduate school” may be more
appropriate in a self study. For assessment report
purposes the objective should be more student learning
focused and provide student learning outcomes that are
measurable. Can you identify specific, measurable,
skills and knowledge that prepare your students to be
successful in locating employment or pursuing graduate
school.” (from memo)
The genesis of learning assessments
Or “How Shit Happens”
In the beginning was the Final Exam.
And then came the Student Learning Assessments.
And the Student Learning Assessments were without form.
And darkness was upon the faces of the Faculty.
And they spoke among themselves, saying, "It is a crock of shit, and it stinketh."
And the Faculty went unto their Unit Heads and said, "It is a pail of dung, and none
may abide the odor thereof."
And the Units Heads went unto their Deans, saying, "It is a container of
excrement, and it is very strong, such that none may abide by it."
And the Deans went unto the Executive Committee of the Senate (ECS), saying,
"It is a vessel of fertilizer, and none may abide its strength."
And the Senators spoke amongst themselves, saying one to another, "It contains
that which aids plant growth, and it is very strong."
And the ECS Chair then went to the Provost, saying unto her, "It promotes growth
and is very powerful."
And the Provost went unto the President, saying unto him, “Our great and wise
Faculty has sent us word that these new Student Learning Assessments will
actively promote the growth and vigor of our grand university, with powerful
effects."
And the President looked upon the Student Learning Assessments, and saw that
they were good.