Assessing Assessment:
Examining the Assessment Plans of
70 Political Science Programs in the
United States
John Ishiyama
Professor of Political Science
University of North Texas
• Two parts of my talk today
– Provide a description of the context in the United
States that frames the current discussions over
assessment (particularly program assessment)
• Explains focus in the US on program assessment as
opposed to course based assessment
– An empirical examination of the kinds of
techniques used in program assessment in
political science
Understand the context in US
• Higher education accreditation system
• Trends toward greater federal intervention
into accreditation and assessment
• Reaction of the Political Science discipline and
APSA.
Accreditation System
• The accreditation system in higher education in
the United States Decentralized– several regional
accrediting organizations
• Resulted from renewals of Higher Education Act
of 1965 (especially the renewal of 1974)
• The Federal Department of Education does
accreditation– however, it is general, largely
based on spending
• General higher education accreditation is
conducted by the regional organization
• Accreditation is coordinated by the Council
for Higher Education Accreditation (CHEA)
• Purposes: national advocacy for selfregulation of academic quality
• Self policing as opposed to direct government
regulation
Seven regions
•
•
•
•
•
•
1) Middle States Association of Colleges and Schools
2) New England Association of Schools and Colleges
3) North Central Association of Colleges and Schools
4) Northwest Commission on Colleges and Universities
5) Southern Association of Colleges and Schools
6) Western Association of Schools and Colleges
Accrediting Commission for Community and Junior
Colleges
• 7) Western Association of Schools and Colleges
Accrediting Commission for Senior Colleges and
Universities
• The regional associations are relatively
autonomous--CHEA does not direct their
activities
• Standards are different across regional
associations
• A loss of accreditation could prevent a school
from participating in federal financial-aid
programs.
Political environment has also changed
• Before, university wide summative data was considered
sufficient, and research universities were given a “pass”
• Texas Tech probation in 2007 for lack of providing data that
students were acquiring skills sent a chill though higher
education
• course based, but only core classes
• HEA reauthorization 2008– wanted to impose externally
summative goals for educational assessment (much like “no
child left behind” for higher education)
– This was removed from the legislation
• However accreditation associations are now pushing
hard for department program assessments
• Motivation: Desire for fiscal accountability by
political leaders.
• Both sides of the aisle (Democrats and
Republicans) support this
• Political science has been under particular
pressure recently, apart from assessment
• Greater pressure on Political Science
– Example of Senator Tom Coburn, only saved by
Elinor Ostrom winning the Nobel prize
• Reaction in political science mixed. Four groups
– Actively hostile
• Two groups – those who don’t care about teaching
• Those who see threats to academic freedom
– - positively supportive (political scientists who work in
higher education administration and are largely
outside the field)– very small
– Passively resigned– resigned to the inevitable
– Growing number who are sensitive to the pressures
and believe we should take charge of the process,
instead of leaving others to impose alien standard
•
•
•
•
APSA has begun to address the directly
Assessment Track at APSA TLC
Assessment handbook (published 2009)
Standing Teaching and Learning Committee
(established 2007)
• Assessment Task Force (established 2010)
• No guidance on assessment (but lots of
inquiry)
Our study
• Study we conducted in 2007- what existed?
• Examined 70 program assessment plans across
the United States
• Coupled with study by Kelly and Klunk (2003)
which employed as survey of department chairs
(n= 213)
• Interested in techniques used (what they
reported and what they say they did)
• And whether these were related to institutional
types
Learning Outcome
Table 2: Learning Outcomes
% reported in
% reported in Kelly
Ishiyama (2008)*
and Klunk
65.2
54.0
63.8
Not asked in study
Knowledge of Theories
Knowledge of political
institutions and processes
Knowledge of Fields in Political
Science
Critical thinking
Methods/Research Skills
Written Communication Skills
Oral Communication/
Presentation Skills
Citizenship
Career Goals
Cultural Diversity
Ethics/Values
66.7
46.0
68.1
62.3
66.7
53.6
55.7
Not asked in study
57.1
30.7
24.6
23.2
17.4
11.6
Not asked in Study
Not Asked in Study
26.5
Not Asked in Study
• Distinguish between what Barbara Wright (2005)
calls internal and external methods of program
assessment
• Internal refers to technques that can be
accomplished without any additional work by
faculty members beyond the day to day
operation of a class
• E.g. analysis of grades, reviewing existing
coursework for signs of student learning,
examinations of existing course syllabi, or student
course evaluations.
• external approaches require effort (often
collaborative between faculty members in a
department) outside of classroom activities
• Includes comprehensive examination, exit
interviews, alumni surveys, portfolio analysis,
a capstone experience, or graduating student
surveys/questionnaires.
Table 3: Most Frequently Mentioned Assessment Techniques
Assessment Technique
% reported in % reported in Kelly
Ishiyama
and Klunk (2003)
(2008)
Graduating Student
50.0
22.2
Survey/Questionnaire
Analysis of Student
45.7
Not asked in study
Grades/Performance
Senior Seminar/Capstone
35.7
39.6
Comprehensive Exam
34.3
Not asked in study
Senior Thesis
32.9
20.3
Senior Exit Interview
24.3
24.1
Portfolio
22.9
17.9
Random reading of
17.1
Not asked in Study
student papers
Student Course
17.1
Not asked in study
Evaluations
Alumni survey/interviews
21.4
Not asked in study
Syllabi Analysis
7.1
Not asked in study
Pretest/Post-test
Not in study
9.9
Post-test only
Not in study
14.2
Faculty Observations
Not in study
25.0
Institutional Characteristics
• the distribution of external assessment techniques
used, by frequency of schools.
• Nine programs employed none of the external
assessment techniques (i.e they only employed
internal techniques)
• 16 employed at least one,
• 22 at least two,
• and eight at least three.
• Only fifteen programs used four or more of the
external techniques (with three political science
programs using all six).
Figure 2: External Assessment Techniques Score
By Selectivity of Institution
6
5
4
3
2.28
2.17
2
1
0
selective/less selective
more selective/most selective
Figure 3: External Assessment
Techniques Score by Private or Public Institution
6
5
4
3
2.27
2.25
public
private
2
1
0
Figure 4: External Assessment Techniques Score
By Student Faculty Ratio
6
5
4
3
2.41
2.11
2
1
0
student faculty ratio more
student faculty ratio less
than 17
or equal to 17
Figure 5: External Assessment Techniques Score
By Highest Degree Offered by Department
6
5
4
2.71
3
2
1.81
1
0
Graduate Degree Offered
Bachelor's Degree Only
Figure 6: External Assessment Techniques
Score by Size of Student Population
6
5
4
2.85
3
1.74
2
1
0
student population 9000
student population less
and over
than 9000
Conclusions
• wide variety of learning outcome expressed and
assessment tehniques employed; not a single model to
conduct student learning assessment.
• discernable patterns-- which is all the more
remarkable given that there has been little discipline
wide guidance as to how to construct an assessment
program.
• many political science departments have hit upon
similar learning outcomes and similar kinds of
assessment techniques.
• Most of the learning outcomes expressed deal directly with
the content of the political science discipline and the
promotion of critical thinking, writing and oral
communication skills as opposed to the promotion of
citizenship and ethics/values.
• internal techniques more likely used than external
techniques
• the use of external assessment activities does not vary by
the level of institutional selectivity, whether they were
public or private institutions, or the institution’s studentfaculty ratio
• Primarily undergraduate departments and smaller
institutions are more likely to employ external assessment
techniques than larger graduate degree granting
departments.
• This might indicate that the external techniques are more
easily implemented at smaller institutions that have smaller
class sizes and fewer majors than the larger state
institutions.
• However, it might also reflect the greater emphasis smaller,
primarily undergraduate institutions place on teaching and
learning, and these departments seek to go beyond merely
employing the easiest assessment techniques to meet the
“letter of the law.”