_--
.
Memorandum
To: Dr. Rosemary Sutton, Director of Assessment
Office of Planning, Assessment, and Information Resource Management
From: David R. Elkins
Associate Professor & Interim Chair
Date: June 10,2004
Re:
2004 Annual Assessment Report - Format A: Narrative
c ~
Goals and Outcomes: The goals for the Political Science Department’s Political Science Major
and International Relations Major assessment are to have students demonstrate critical and
analytic thinking, to engage in proper research, and effective communication. The outcomes of
these goals are as follows:
~
Dimension
Thesis Component
Expectations
Critical and Analytical Component
I Clearly articulated thesis.
I
Hypothesis Component
Research question or hypothesis is clearly formulated. Evidence Component
Evidence is generally appropriate. Conclusions Component
Draws appropriate conclusions. Research Component
Five to ten scholarly sources cited or combinationof
scholarly sources, government documents, interviews,
foreign news sources, and articles from newspapersof
I record.
Appropriate citations (footnotes, endnotes, I
-
Sources Component
I
Citations Component
~~
embedded) ~~
Bibliography Component
Properly organized bibliography. Organization Component
Paragraphs Component
Sentence Structure Component
Diction Component
Grammar Component
Good organization. Consistently developed paragraphs. Concise sentence structure. Accurate diction. Infrequent grammatical errors. The goals and outcomes were developed in a subcommittee consisting of Dr. Schulz, Dr.
Charlick, and Dr. Elkins. The goals and outcomes were presented to the entire Department for
approval. The goals and outcomes were refined by the full department and approved by the
I
department. The goals and outcomes have not been modified since the approval (See Appendix
A).
Research: The method of assessment is based on students demonstrating outcomes as indicated
by their final papers in the Department’s senior seminars. The faculty members teaching a senior
seminar submit unmarked and anonymous final papers from each senior seminar to the chair.
The chair randomly selects from these papers a representative sample to distribute to paired
teams of reviewers. The reviewers are fulltime faculty members in the Political Science
Department that did not teach senior seminars (faculty members that did teach senior seminars
are excluded from the pool of reviewers). The reviewers assess each paper using a instrument
measuring the outcomes of the discrete components of the goals (Appendix B). The
measurement instrument was modified between the spring and the fall to increase the level of
intercoder reliability by increasing the number of measurement categories:
Figure 1: Illustration of Spring 2003 and Fall 2003 Student Assessment Instruments
Spring 2003 Measurement
Exceeds Expectations
Meets Expectations
Does Not Meet Expectations
3
2
1
I Fall 2003 Measurement
5
4
3
2
1
-
Findings: The Political Science Department has produced three reports based on assessment In
general, the findings indicate that the majority of Political Science Majors and International
Relations Majors are meeting or exceeding Departmentally established expectations (For more
detailed information see Appendix C: Student Assessment Results, Spring 2003, Student
Assessment Results, Fall 2003, and Influence of the Data Analysis Course on Senior Seminar
Grades). The Department is currently conducting its review of senior seminar final papers for
the spring 2005 semester. The report will be distributed in the fall of 2005 and discussed at the
first departmental faculty meeting.
Review: All fulltime faculty members are involved in the review process either as instructors in
senior seminars or as reviewers for the purposes of student assessment. In addition, reports were
distributed to faculty members and discussed in subsequent Departmental faculty meetings.
Action: The Department has determined that the guidelines for the International Relations
majors does not clearly enough indicate the proper sequencing of the senior seminar.
International Relations majors may take the seminar, according to current guidelines, as if it were
a regular course. The idea, however, is that the senior seminar is a capstone course. The
Department has directed the chair to take steps to articulate clearly that the senior seminar is a
capstone course and should only be taken near the completion of the degree, specifically after the
student has completed the core and track requirements.
Appendix A Assessment Outcome Expectations Senior Seminar Student Assessment
Criteria Explanations
Critical and 1
Dimension
Does Not
Meet
Expectations
Meets
Exceeds
Expectations
Expectations
I
fiesis
Sbmponent
Clearly articulated thesis linked to
either theory or a cogent statement Clearly articulated thesis.
of the nature and significance of
the problem being investigated.
Hypothesis
Component
Theoreticallyappropriate
hypothesis or explanation of a
significant political problem.
Research question or hypothesis is Research question or hypothesis is
unclear or nonexistent.
clearly formulated.
Evidence
Component
Superior evidence appropriate for
the hypothesis or problem being
investigated.
Evidence is generally appropriate.
Conclusions
Component
Unclear or nonexistent thesis
statement.
Conclusions linked to theory or
Draws appropriate conclusions.
problem being investigated.
Research Criteria
I
Dimension
Meets
Expectations
Exceeds
Expectations
I
(lo Or more) scholarlY
Sources
Component
sources cited or combination of
scholarly sauces, news sowces,
and
. .
O
'm
newspapers Of
record.*
1
Five to ten scholarly sources cited
or
of scholarly
souTces,govement documents,
interviews, foreign
sources,
and articles from newsuauers of
record.*
.~~~
a
.
Evidence does not test hypothesis
of answer research question.
Unclear or no conclusions.
Does Not
Meet
Expectations
Less than five scholarly sources
cited or combination of scholarly
sources, government documents,
interviews, foreign news sources,
and articles from newspapers of
record.*
I
Appropriate citations
(footnotes, endnotes,
embedded)
Citations
Component
Consistent use of appropriate
citations (footnotes, endnotes,
embedded)
Bibliography
Component
Consistent,
and properly properly organized bibliography.
organized bibliography.
Poor, inconsistent use of style or
no use of formal style.
Senior Seminar Student Assessment
Criteria Explanations
Akticulate an( Communicate Criteria
Dimension
Exceeds
Expectations
Meets
Expectations
Does Not
Meet
Expectations
Uneven and ineffective
organization.
Organization
Component
Excellent organization.
Good organization.
Paragraphs
Component
Thoroughlydeveloped paragraphs
with clean transitional phrases.
Consistently developed
paragraphs.
Sentence Structure
Component
Interlinked and concise sentence
structure.
Concise sentence structure.
Diction
Component
Accurate and appropriate
diction.
Accurate diction.
Vague and inexact diction.
No grammatical errors.
Infrequent grammatical errors.
Isolated serious mechanical errors
and occasional minor mechanical
errors.
Grammar
Component
Incompletely developed
paragraphs and inconsistent
transitional phrases.
Coherent but vague sentence
structures.
Appendix B Assessment Measurement Instrument Senior Seminar
Student Assessment Form
Evaluator’s Comments
Organization
5
4
3
2
I Paragraphs
5
4
3
2
1
Sentence
Structure 5
4
3
2
1
Diction
5
4
3
2
1
Grammar
5
4
3
2
I Appendix C
Student Assessment Results, Spring 2003 Student Assessment Results, Fall 2003 Influence of the Data Analysis Course on Senior Seminar Grades MEMORANDUM
To:
PSC Faculty Fr:
D. Elkins, Interim Chair
& Date: September 25,2003 Re:
Student Assessment Results, Spring 2003 Overview
The results of the first student assessment have been calculated. In brief, the reviewing
faculty found that over three-quarters (76.3%) of the papers assessed either met or exceeded the
established expectations. However, the faculty, paired as reviewers, had a low degree of intercoder reliability (r = .22). The paired faculty reviewers agreed on only two-fifths (40.0%) of the
paired reviews.
Specific Findings
On the attached pages, you will find tables depicting the results of the assessment. I have
divided this section into two parts. The first part relates to the student assessment, and the
second part relates to the inter-coder reliability issue.
Student Assessment. The findings of the assessment indicate that the reviewers had
their most Cerious reservations about the papers in the Critical and Analytic Criteria. As
illustrated in Table 1, well over a third of the assessed papers did not meet established
expectations (37.5%). The components that the reviewers found most troublesome were
related to the Hypothesis Component and the Conclusions Component.
Despite the relatively low scores on the Critical and Analytic Criteria, the reviewers
found that the vast majority of the assessed papers either met or exceeded established
expectations for the Research Criteria and the Articulate and Communication Criteria. The
Research Criteria was a particularly noteworthy result with over two-fifths of the reviewers
scoring the papers in this area as exceeding established expectations, the Sources Component
and the Bibliography Component led this criteria.
Inter-Coder Reliability: This was a non-trivial problem with the assessment. As
Table 2 illustrates, the reviewing faculty agreed on only 40.0% assessed papers’ dimensions.
1
The area of most frequent agreement was in the Research Criteria. The reviewers agreed over
half the time. By contrast, the Critical and Analytical Criteria and the Articulate and
Communicate Criteria pose had low levels of agreement. The reviewers were more likely to
disagree than agree in these two criteria. However, the nature of disagreement was unique to
each criterion.
The Critical and Analytic Criteria had the more difficult and troubling forms of
disagreements. There were more Major Disagreements, one reviewer scores a paper’s
dimension as Exceeds Expectations and the other scores the paper on this dimension as Does
Not Meet Expectations, in this section than in any other section of the assessment. The
reviewers were as likely to disagree in the Articulate and Communicate Criteria as they were
in the Critical and Analytic. However, the disagreement was almost as likely to be whether
the paper exceeded expectations instead of meeting expectations.
Proposed Suggestions
My suggestions are divided into the Substantive Outcome of Assessment and Inter-coder
Reliability sections. In the first, 1observe that most students write case studies and that the
instruction that I use in the Data Analysis class has a quantitative assumption bias. In the next
section, I suggest that we modify the instrument, based on the suggestion of a colleague, to
minimize hopefully
the inter-coder reliability problem.
Substantive Outcome of Assessment: This is the Department’s first attempt at
conducting student assessment with this instrument and this process and these results are
obviously preliminary. My chief observation is that most of the students write case studies and
not quantitative-based papers. Given that the reviewers of the papers indicate the greatest
reservations in the Critical and Analytic Criteria, I surmise that this is a weakness that should be
discussed.
I can only speak for the section of PSC 251 Data Analysis that I teach, but I do not spend
much time at all instructing students on proper techniques of case study analysis. Indeed, I
instruct students with a quantitative assumption bias. That is to say, I instruct students
presuming that they are going to use quantitative methods. Still, there are at least two
components to this usage: consumption and production. On the one hand, the instruction in
quantitative methods is important to aid students as critical consumers of quantitative research.
2
On the other hand, as is evident in this round of assessment, the students produce papers that are
fundamentally qualitative in orientation. There are obvious overlaps between the two forms
(quantitative and qualitative) of methods in the development of research questions, hypothesis
formation, use and import of theory. It is my sense that in my class my emphasis on the
quantitative may implicitly bias students into believing that these critical areas of overlap are
exclusive to quantitative research and perhaps not relevant to qualitative research.
To the extent that my colleagues that teach Data Analysis have this problem (to varying
degrees) I think it is important that we insist and demonstrate the importance of the development
of research questions, hypothesis formation, use and import of theory in qualitative research as
well as with quantitative research.
Inter-coder Reliability: The inter-coder reliability must be increased. There are two
potential remedies. The first is a minor modification to the existing instrument’s coding
structure. One faculty member suggested that we create middle categories between the Exceeds
Expectations and Does Not Meet Expectations. For instance,
Exceeds Expectations
Meets Expectations
Does Not Meet
Expectations
By modifylng the instrument, this may eliminate the number of Minor Disagreements and
increase the number of Agreements. Two reviewers may not feel compelled to a judgment of
“either-or” and settle for a “sort-of’ category when scoring a dimension on a paper. The
dilemma is that this may increase the number and magnitude of Major Disagreements.
The potential remedy for Major Disagreements is neither easy nor simple. One choice is
to re-work and hrther clarify the descriptions and instructions of each dimension in the Student
Assessment. The transaction cost of getting agreement among the faculty on this issue is
formidable. The other option is to provide training to the faculty reviewers regarding the use of
the Student Assessment. This would be time consuming and as difficult as the previous option.
My suggestion is twofold. First, adopt the coding structure modification for the Fall 2003
assessment. Analyze the Fall 2003 results and determine if a significant problem remains with
3
inter-coder reliability. If inter-coder reliability remains a problem, identify those substantive areas that are creating the greatest problem, most likely the Critical and Analytical Criteria, and address those problems at that time. Attachments: Table 1: Frequency Distribution of Scores and Average of Scores for Assessment Papers, Spring 2003; Table 2: Frequency Distribution of Agreements and Disagreements of Paired Assessment Reviewers, Spring 2003; Table 3 : Frequency Distribution of Reviewer Agreements; Table 4: Student Assessment Spring 2003 Evaluator’s Comments 4
Frequency of Scores’
Does
Not
Meet
Exceeds
Meets
Dimension
Expectations Expectations
Average2
Expectations
Thesis component
5
18
7
1.93
Hypothesis Component
4
12
14
1.67
Evidence Component
7
15
8
1.96
163
1.53
3 7.5%
45
I . 78
Conclusions Component
3
11
I5.8%
46.7%
19
56
Research Criteria
Criteria Subtotal
Sources Component
Citations Component
~~
-
Bibliography Component
_
_
.
Criteria Subtotal
16
13
1
2.5
10
15
5
2.17
1
11
14
43.3%
44.4%
40
39
Articulate and Communicate Criteria
I
5
I
2.3
I2.2%
II
2.32
Organization Component
5
18
7
1.93
Paragraphs Component
6
19
5
2.03
Sentence Structure Component
7
17
6
2.03
Diction Component
Grammar ComDonent
8
16
6
2
8
17
58.0%
87
50.5%
5
2
Criteria Subtotal
TOTALS
22.7%
34
25.8%
93
182
19.3%
29
23.6%
85
2.03
2.02
In this portion of the analysis the scores are treated as discrete and not paired. That is to say, though each paper had two reviewers (paired reviewers), I recorded in these columns the score that each reviewer would have given on the various dimensions. For example, two colleagues reviewed Assessment Paper #2. If the first colleague scored the Thesis Componentas “Meets Expectations”and the second colleague scored it as “Does Not Meet Expectations,”those scores would be represented in two separate columns in the Frequency of Scores. N = 360 ((12 Dimensions . 15 Papers) . 2 Reviewers} 2 = The arithmetic average was derived by establishing a mean for each dimension for each paper. I then created an average of these averages. 3 = One reviewer coded a paper as “0”. I recoded this to “1” correspondingto the appropriate categories. Table 2: Frequency Distribution of Agreements and Disagreements of Paired Assessment
Reviewers, Spring 2003 Disagreements2
Agreements’
Dimension
MinorHigh3
I
I
I
I
Hypothesis Component
~~~~ I
~
Evidence Component
Conclusions Component
8
1
3
6
I
I
I
0
36.7%
18.3%
22
I1
Research Criteria 5
Criteria Subtotal
I
I I
I
1
3
3
3
7
28.3%
I7
3
7
5
9
53.3%
24
5
35.6%
16
Organization Component
5
5
Paragraphs Component
6
Sentence Structure Component
4
6
5
5
3
6
Diction Component
5
6
4
6
34.6%
26
40.0%
72
4
5
33.3%
25 25.0%
45
~
Bibliography Component
Criteria Subtotal
Grammar Component
Criteria Subtotal
TOTAL
I
32.0%
24
28.3%
51
3
16.7%
IO
I
I
I
6
Citations Component
Major
4
8
Sources Component
1
MinorLow4
I
Critical and Analytical Criteria ,
6
4
Thesis Component
1
I
1
2
0
I
o
1
I
6.7%
3
1
4.4% 2
I
o
0
0
I
o
0
0
6.7% 12 Table 3: Frequency Distribution of Reviewer Agreements
Exceeds
Expectations
Meets
Expectations
Does Not Meet
Expectations
Thesis Component
0
5
1
Hypothesis Component
0
4
4
Evidence Component
0
2
1
Conclusions Component
0
2
3
3
0
Dimension
~
Sources Component
I
Research Criteria
5
I
Citations Component
2
4
1
Bibliography Component
4
3
2
Organization Component
0
4
1
Paragraphs Component
0
0
Sentence Structure Component
2
4
4
Diction Component
1
3
2
22.2%
16
4
~~
___
Grammar Component
TOTAL
58.3%
42
0
I
I
1
0
19.4%
14
I
1
Table 4. Student Assessment Spring 2003 Evaluator’s Comments
Evaluator’s Comments
Dimension
0
Thesis Component 0
0
0
0
0
Hypothesis Component 0
0
0
0
Evidence Component
0
No political science or political economy thesis - purely descriptive (2)
Too many of them (5)
No purpose apparent (11)
Application of a theory-based theory thesis or hypothesis is not made clear (13)
Must it be so trite? (20)
Thesis appears on pg. 2. That conflict in entire region is dominated by “economic
approach” - vague and not operational (21).
No clear hypothesis (2)
The student’s hypothesis struck me as a “virtual” hypothesis (4)
Never stated - mostly about Japan’s effect (5)
(descriptive?) (11)
Paper comes down to: does case fit paradigm (13)
More the application of a theory than its explanation power (14)
There is no hypothesis. No systematic use of rational choice theory either (21) Meets test
of theoretically appropriate.. .explanation of a significant political problem (23)
Evidence present but not widely sourced (4)
Somewhat difficult to assess this. Student has an unclear thesis but has evidence
generally appropriate to demonstrate a potential thesis (12)
For what = in the end the contention is that symbolic politics explains the “ethnic war” (I
3)
Most of actual data is a chronicle account not consciously linked to the theory (14)
Needed to tie evidence to theory more (19)
There is no evidence or conclusion relative to this contention (13)
Are asserted but far from demonstrated. Still a credible effort (14)
Odd conclusions section (17)
Over reach and not tied to rational choice theory contentions (21)
z
0
0
0
Conclusions Component
0
0
0
Many cites but no political science sources (2)
I count 4-5 academic sources actually used (13)
0
Few scholarly sources (21)
0
No year in embedded citation (4)
0 Does not include year in embedded citation form. Sometimes in sentence, sometimes not
(7) I have some reservations about student’s citations. For instance, cites Rabuskla and
-Shepsle’s notion from a separate source (12)
0
But largely unused (13)
0
Sources Component
0
Citations Component
Bibliography Component
.
..
. ~ . I
.-.i*...;:
Y.
,
:*:.,.:
0
Organization Component
1
Paragraphs Component
0
0
Sentence Structure 0
0
0
Diction Component 0
0
0
Grammar Component 0
0
Highly redundant - argument goes nowhere (2)
Some problems with all these, but better than many CSU students (4)
Long. Long historical section does not advance argument much (14).
Not clear how the sections contribute to the economic argument (21).
Some problems with all these, but better than many CSU students (4)
Endless repetition (20)
Student‘s writing is making errors consistent with non-native speaker (4)
Some problems with all these, but better than many CSU students (4)
Student’s writing is making errors consistent with non-native speaker (4)
Some problems with all these, but better than many CSU students (4)
Generally OK with some minor problems (21)
Student’s writing is making errors consistent with non-native speaker (4)
Some problems with all these, but better than many CSU students (4)
Some mistakes (21)
One evaluator submitted the following:
“Overall comment :
With the exception of paper #7, they weren’t particularly good. They were mostly descriptive, not using theoretical material
very effectively. On the other hand, there was some description of theory, they had obviously done a fair amount of work,
and they presented their stories in a fairly coherent manner - certainly better than many CSU students.
So, even though I was unsatisfied on some level, I gave them mostly “2”s.”
Memorandum To:
PSC Faculty
From: David R. Elkins
Associate Professor and Interim Chair
Political Science Department
Date: February 11,2004
Re:
Student Assessment Results, Fall 2003
Overview
This report summarizes the Department’s second student assessment. In brief, the four
members of the faculty that reviewed the twelve randomly selected senior seminar papers found
that two-thirds (66.6%) either met or exceeded departmentally established expectations. As
anticipated, altering the assessment instrument appears to have improved inter-coder reliability.
The paired faculty reviewers had a moderate level of agreement among their assessments (r =
.54), which represents an improvement over the Spring 2003 assessment results (r = .22).
Student Assessment Process
The student assessment process required the two faculty members teaching senior
seminars in the Fall 2003 semester to submit copies of all senior seminar term papers to the
student assessment coordinator. The senior seminar papers were un-graded, un-marked, and
anonymous versions of the papers submitted to the instructor of record for a grade. A total of
twenty-nine papers from the two Fall 2003 senior seminars (PSC 420 American Politics and PSC
422 International Relations) were submitted to the student assessment coordinator for review.
Twelve of the twenty-nine papers were randomly selected for review (41.3%). Four
faculty members reviewed the papers. The faculty member reviewers were paired, and the
1
pairings represented the identical pairing from the Spring 2003 student assessment. Each paired
reviewer received six randomly assigned papers, three from PSC 420 and three from PSC 423.
Each reviewer received a student assessment packet that included six seminar papers, six Senior
Seminar Student Assessment forms and one Senior Seminar Student Assessment Criteria
Explanation form on December 17,2003. The last set of reviews was submitted February 11,
2004.
Findings
This report describes two issues. It describes the changes that were adopted since Spring
2003 assessment, the outcome of those changes, and it describes the results from this iteration of
student assessment.
Assessment Modifications: The last round of student assessment identified a problem
with inter-coder reliability. The Spring 2003 inter-coder reliability was very low (r = .22) with
only about 40% of the paired reviewers agreeing on a dimensional score.
The Department decided to modify the measurement instrument itself.
Figure 1 illustrates the changes in the measurement instrument. The strategy was to
expand the number of possible categories from three to five. The presumption was that it would
minimize the number of minor disagreements and increase inter-coder reliability.
2
I Figure 1: Illustration of Spring 2003 and Fall 2003 Student Assessment Instruments
Exceeds Expectations
Meets Expectations
Does Not Meet Expectations
3
2
1
5
4
3
2
I
1
A number of variables are different from the spring and fall student assessments (six
reviewers in the spring and four reviewers in the fall, fifteen papers in the spring and twelve
papers in the fall, different seminars with different instructors) but the pairings of the four
reviewers remained the same from the spring and the fall. Though it remains to be seen in future
iterations of the student assessment process, the inter-coder reliability increased substantially (r =
.54) from the Spring 2003 to Fall 2003. However, the improvement is a result of creating more
categories than the responses in those categories. Table 2 illustrates the percent of agreements
and disagreements between Spring 2003 and Fall 2003 student assessment.' There were
proportionately fewer points of agreement in the fall assessment than in the spring assessment
and the percent of disagreements declined marginally. However, the percent of major
disagreements doubled. A major disagreement is defined as a difference in paired reviewers
scores by two or more points. Because the number of categories in the measurement instrument
increased by two-thirds from spring to fall, it is perhaps no surprise that the proportion of major
'
A complete display of diagnostics of agreement or disagreement for the fall 2003 assessment are attached to this
document (Tables 6 and 7).
3
disagreements increased. Still, the change in the instrument was a positive change increasing the
magnitude of inter-coder reliability appropriately.
Student Assessment: Table 3 (see attached) illustrates the results of the Fall 2003 student
assessment. The reviewers found that two-thirds (66.6%) of the senior seminar papers met or
exceeded departmental expectations. Indeed, with an overall mean score of 2.94 (s=.904) and a
median of 3, the department can be reasonably comfortable that students that submitted senior
seminar papers are meeting its expectations. However, there remains variation in the
dimensional components.
The reviewers were most satisfied with the Research Criteria and the Articulate and
Communicate Criteria. In general, the reviewers found that four out of five (79.1YO)seminar
papers met or exceeded Research Criteria expectations and that three-quarters (75%) of the
seminar papers met or exceeded Articulate and Communicate Criteria expectations. Individual
4
dimensions within these two categories varied, but even there the results were likely to meet or
exceed expectations. However, this result does not hold for the Critical and Analytical Criteria.
The majority (51.3%) of the Fall 2003 senior seminar papers did not meet the Critical and
Analytical Criteria expectations. This represents a substantial increase from the previous student
assessment and is likely accounted for in the changed metric of the instrument. For instance, if
two of the spring papers had been found not to meet expectations the results would have been
similar to this fall’s results. These results are disappointing and consistently so across all
dimensions of this criterion.
In light of the recent report on data analysis and its association with seminar grades, I
examined the rosters and transcripts of the Fall 2003 Semester’s senior seminars. Table 4 depicts
the frequency of students that completed seminars, and it shows that just under half had not taken
data analysis, and three of those that are counted as having taken data analysis took it during the
Fall 2003 Semester. Clearly, the probability of a reviewer reading a paper written by a student
that had not taken data analysis was very high. Still, this does not necessarily mean that the
missing data analysis class was the contributing factor. However, given that there is a clear
difference in the number of students having taken (or taking) the data analysis class by seminar
type, one way to examine whether completing the data analysis course has an impact is by
looking at the student assessment results across the three criterions by senior seminar type.
5
Table 4: Frequency of Students Completing PSC 420 American
Politics and PSC 422 International Relations Senior Seminar, Fall
2003
I
I
Senior Seminars
Data Analysis
Total
PSC 420
PSC 422
1
Yes
12
No
3
11
I
14
Each paired reviewer assessed six papers, three from PSC 420 and three from PSC 422.
Though the anonymous papers were randomly selected and assigned for assessment, I kept a
record of the paper assignments by reviewer and by senior seminar. If data analysis has a role to
play here I would expect two outcomes. First, I would expect that reviewers would be much
more likely to rate papers from PSC 420 as either meeting or exceeding expectations on the
Critical and Analytical Criteria than papers from PSC 422. Second, if it were truly a meaningful
outcome, I would expect very little difference between the ratings of papers on the other two
criteria (Research Criteria and Articulate and Communicate Criteria). I would expect these two
outcomes because more students in PSC 420 took data analysis than in PSC 422, and I would
expect the review outcomes for other two criteria to be reasonably close because there is no
systematicevidence to suggest that either the quality of the students, the seminars’ substance, or
the instructors’ demands and instructions differ in important ways.
6
Meets or Exceeds Expectations
(3 - 5 Rating)
CRITERIA
Critical and Analytical Criteria
Research Criteria
AND
Articulate and Communicate Criteria
PSC 420
PSC 422
58.3%
35.4%
76.0%
(73)
76.0%
(73)
I
Note: There were a total 48 possible observations for the Critical and Analytical
Criteria and there were a total of 96 total possible observations for the Research
Criteria and the Articulate and Communicate Criteria.
Table 5 illustrates the proportion and frequency of reviewers indicating senior seminar
papers either met or exceeded expectations by criteria and seminar type. Nearly three-fifths
(58.3%) of the reviews of seminar papers emerging from PSC 420, where virtually all (80%) of
the students had taken or were taking data analysis, indicated the papers either met or exceeded
expectations. By contrast, just over a third (35.4%) of the reviews for papers from PSC 422,
where only a small minority ( 2 1YO)
of the students had taken data analysis, met or exceeded
expectations. Alone the first finding is interesting, but coupled with the findings for the other
two criteria the evidence becomes striking. There was no difference between the total number of
reviews in either seminar that met or exceeded expectations, none. What appears to separate the
quality of seminar papers from these two seminars is the demonstration of critical and analytical
thinking. And, though it is far from conclusive, the available evidence suggests that the data
analysis course may be an important variable in improving this quality.
Conclusion
7
The results of the Fall 2003 student assessment demonstrate that the alteration made to
the assessment instrument has a positive outcome for the problem of inter-coder reliability. The
Department may wish to review this further in order to increase it more, but for now it seems that
the modification has been successful. In addition, the results also indicate that overall the
reviewers are satisfied that the majority of the papers meet or exceed departmentally established
expectations.
The strength of the papers remains with the Research Criteria and the Articulate and
Communicate Criteria. The evidence suggests that many of our students are meeting and
exceeding our expectations. One question, however, should be asked: Are these standards too
low? Should we increase the Department’s expectations for these two standards? Currently, I
believe the answer should be no, but it is something to consider. The reason not to increase
expectations now is because of the weaknesses revealed in this assessment.
The Critical and Analytical Criteria remain problems. The outcome was disappointing in
the Spring 2003 Semester student assessment and it is even more apparent in this assessment
(largely due to the change in the measurement instrument). The analysis presented here coupled
with an earlier report on seminar grades, indicates that there may be more reason to suggest that
some students are ill-prepared for the rigors of the seminar if they have not taken the data
analysis course. Though not depicted in Table 4, the students in PSC 422 that had not taken the
data analysis course were International Relations majors and the three that had were Political
Science majors (the three in PSC 420 that had not taken data analysis were Political Science
majors). Though it will pose some distinct challenges for the International Relations major, the
Department may wish to consider requiring IR majors to some type of analysis course and
require it be taken prior to the seminar.
8
Frequency of Scores'
Exceeds
Meets
Does Not
Dimension
Expectations Expectations
Meet
Expectations
(5 or 4)
(3)
(2 or 1)
Average'
Thesis Component
4
8
12
2
Hypothesis Component
4
8
8
12
3
13
2.4
2
5
21.9%
(21)
5
25.0%
(24)
14
53.I %
(50
10
6
11
13
3
7
31.9%
(23)
10
47.2%
(34)
7
20.8%
(15)
Organization Component
8
8
8
2.5
Paragraphs Component
8
14
2
3.3
Sentence Structure Component
Diction Component
Grammar Component
7
13
4
3.5
8
9
7
3
6
9
44.2%
l53)
38.5%
(111)
9
25.0%
2.75
Evidence Component
Conclusions Component
Criteria Subtotal
Sources Component
Citations Component
Bibliography Component
Criteria Subtotal
30.8%
(37)
28.1 Yo
(81)
Criteria Subtotal
TOTALS
LL, L 1 I . Y
"
'
"
J
f
"A
L A X " &. .
'..',"."-.-...----- ---
--
.~-1u
I."&
1.5
2.6
3.5
3.2
5
4 3.29
3.5 (30) 33.3%
(96)
fJ'Y'LUU.
A l l U C 1 0
2.94
C
"
CLl"U
6"
each paper had two reviewers (paired reviewers), I recorded in these columns the score that each of the paired reviewers would have given on the various dimensions. For example, two colleagues reviewed Assessment Paper #2. If the first colleague scored the Thesis Component as "Meets Expectations" and the second colleague scored it as "Does Not Meet Expectations," those scores would be represented in two separate coIumns in the Frequency of Scores. N = 288 ((12 Dimensions . 12 Papers) . 2 Reviewers} 2 = The arithmetic average was derived by establishing a mean for each dimension for each paper. I then created an average of these averages. Disagreements’
Dimension
Agreements’
-
Minor High3
Minor Low4
Major
Thesis Component
4
3
5
0
Hypothesis Component
2
3
5
2
Evidence Component
7
3
1
1
5
3 7.5%
2
22.9%
3
29.2%
(18)
(1 1)
(14)
2
10.4%
(5)
Sources Component
5
3
1
3
Citations Component
6
2
3
1
5
44.4%
(16)
1
16.7%
(6)
4
22.2%
(8)
2
16.7%
(6)
Organization Component
3
3
4
2
Paragraphs Component
Sentence Structure Component
6
4
1
I
6
2
2
2
2
4
35.0%
6
2
2
3
20.0%
(12)
23.6%
2
ConclusionsComponent
Criteria Subtotal
BibliographyComponent
Criteria Subtotal
Diction Component
Grammar CornDonent
3
30.0%
(18)
24.3%
I
15.0%
(9)
(2 1)
13.9%
38.2%
TOTAL
(20)
(55)
(35)
* (34)
1 = Agreement means that the paired reviewers agree on the paper’s score for a discrete dimension. 2 = There are two types of disagreements: Minor and Major. A minor disagreement means that the paired reviewers differed by one point for a discrete dimension. A major disagreement means that the paired
reviewers disagreed by two or more points. 3 = A “Minor High” Disagreement indicates that one reviewer indicated that a paper at a minimum Meets Expectations and the other reviewer indicated that the paper Exceeded Expectations. 4 = A “Minor Low” Disagreement indicates that one reviewer indicated that a paper at a maximum Meets Expectations and the other reviewer indicated that the paper Did Not Meet Expectations. Criteria Subtotal
-
Distribution
Exceeds
Does Not Meet
Meets
Xmension
5
4
3
Paragraphs Component
0
1
5
Sentence Structure ComDonent
0
2
4
ul.A---___
~
Diction Component
Grammar Component
I o
l
o
l
o
I
1
1
2
1
2
U
0
I
1
0
0
1
I
1
I
0
Influence of the Data Analysis Course on Senior Seminar Grades
Summary
David R. Elkins Associate Professor & Interim Chair Political Science Department February 2,2004 Over the last few years a number of colleagues have expressed reservations about
the quality of preparation of students for the intellectual challenges of senior seminars.
The Spring 2003 student assessment pinpointed, to an even greater degree, that one chief
problem area was regarding students’ analytic capability. This raised questions about the
data analysis course. This paper attempts to address some of those questions.
Specifically, it addresses the following:
0
0
0
Does taking a data analysis course effect a seminar grade?
Does the data analysis course have a different effect on seminar grades by
seminar type, by major, and by high and low student educational skill?
Does the timing of taking and performance in a data analysis class predict
seminar grades?
Drawing on a sample of 169 of the 256 students (66.6%) that took one (or more) of
the twenty senior seminars taught between fall 1998 and spring 2003, the analysis
demonstrates:
0
Data Analysis Course’s Effect on Seminar Grades - Students that had either
completed a data analysis course prior to their seminar semester or enrolled and
completed the data analysis course during their seminar semester did statistically
better than students that did not. On average, a student that took data analysis
improved her letter grade by roughly one-half letter grade.
1
Student Enrollment in Data Analysis Course - Most Political Science majors have taken PSC 25 1 Introduction to Data Analysis prior to enrolling in a senior seminar. However, the majority of International Relations majors have not because they do not have to per the degree program. Timing of Taking Data Analysis Course - In general, students take the data analysis course between ten and twelve months prior to taking a seminar, but this has no impact on seminar grades. Still, a substantial proportion of students do not take the course early enough. For instance, nearly one in five of the students that completed a seminar took the data analysis course during their seminar semester, and another fifteen percent took the data analysis course a semester before their seminar semester. This poses a problem to the extent that the material taught in data analysis should bolster and deepen the understanding of social science material taught in the Department’s baccalaureate-level courses. For a third of students this is not happening. Performance in Data Analysis and Its Effect on Seminar Grades - The results of this analysis indicates that how a student did in the data analysis course has no bearing on how the student will do in a seminar. 0
Grades in Senior Seminars - Overall, students do satisfactorily in senior seminars. The average grade earned is equivalent to a B. To the extent that assessment is about measuring students’ ability to meet educational targets, the faculty members of this Department are indicating, through their grades, that students in political science undergraduate seminars are meeting departmental expectations. 2
0
Quality of Seminar Students - Students that complete senior seminars have
cumulative GPAs of 2.92 (s = .62) and this is roughly equivalent to a B. A
student’s cumulative GPA is a strong and consistent predictor of seminar
performance.
0
Student Substantive Preparation for Senior Seminar - The vast majority of
Political Science majors take the senior seminar after having taken eight political
science courses. By contrast, because of degree differences International
Relations majors take roughly five political science courses prior to a seminar
semester. Ironically, as measured in this analysis, substantive preparation does
not have an effect on seminar performance.
0
Non-Completion of Seminars, Repeating Seminars, and Multiple Enrollments This sample indicates that very few students that enroll in a seminar do not
complete the seminar for grade. In general, about 5% do not complete for grade.
However, this does not mean a student will not attempt another seminar. In
addition, a very few students seem to like the seminar type. Two students
enrolled in two separate seminars and completed for grades.
3
REPORT Influence of the Data Analysis Course on Senior Seminar Grades David R. Elkins Associate Professor & Interim Chair Political Science Department February 2,2004 Beginning in the last academic year, this Department began a systematic attempt
to assess the ability of its students to meet departmentally established expectations though
its student assessment process. The first student assessment report, released in the fall of
2003, indicated that, overall, students completing political science seminars do so to
expectations. However, a significant weakness was detected in the assessment exercise.
The results demonstrated that a non-trivial proportion of students preformed under
departmental expectations in the analytical area. This raised an important question about
the preparation students have prior to enrolling in senior seminars. Some of this concern
was directed at the Department’s data analysis course, PSC 251 Introduction to Data
Analysis. Indeed, there was a significant shadow of doubt about its efficacy in my mind
that a thorough analysis of its effectiveness was warranted. Of central concern to this
Department is whether the data analysis course has an effect, particularly an effect on
student performance in senior seminars.
In this report I examine this issue. I address three broadly related questions:
0
0
0
Does taking a data analysis course effect a seminar grade?
Does the data analysis course have a different effect on seminar grades by
seminar type, by major, and by high and low student educational skill?
Does the timing of taking and performance in a data analysis class predict
seminar grades?
4
I address these questions using a random sample of two-thirds of the students that have
taken the seminar between fall 1998 and spring 2003. In the broadest of terms, I find that
the data analysis course has a positive and statistically significant effect on grades
students earn in the Department’s senior seminars.
PURPOSES OF DATA ANALYSIS INSTRUCTION
PSC 25 1 Introduction to Data Analysis is a three-credit hour course. Its catalog
description states it provides “Sources of information for research in political science, the
use of computers as a research tool, and elementary statistical analysis” (Cleveland State
University, Undergraduate Catalog 2002-2004:237). Currently, three fulltime faculty
members teach the Department’s data analysis course’ and at least one part-time faculty
member has taught the data analysis course infrequently (Dr. Watson).
The intent of the data analysis course is to introduce students to the use of
research methods and empirical analysis in social science research. It attempts to make
them reasonably sophisticated consumers of scientific papers and effective producers of
papers for baccalaureate-level political science courses. The course is required for all
Political Science majors, but it is not required for International Relations majors. In
general, the course instructs students in basic areas of research methods and empirical
observation. For instance, the course instructs students in how to generate research
questions, define concepts and variables including ideas of validity and reliability, write
proper hypotheses, and to limit conclusions to an appropriate extent. In addition, it
provides students with a basic introduction to forms of empirical observation, specifically
as it relates to quantitative forms of analyses. In this regard, the course provides a basic
I
Between fall 1998 and spring 2003, Dr. Govea taught PSC 25 1 five times, Dr. Elkins taught the course
three times, and Dr. Hasecke has taught the course once.
5
understanding of statistical techniques. This, however, does not come without some
instructional challenges.
For me the greatest challenge is altering the way students think about research
issues. It is now second nature for many of us, through our professional training and
research experience, to recognize potential areas of research and parse them into discrete
and highly specific components. The scientific research orientation that we adopt as
professicnal social scientists is sometimes a daunting challenge to convey to students. In
some cases, they falter by simply not recognizing what is an appropriate researchable
issue. A second issue for some students is they have a distinct mathematics phobia and
they do so to an extent that even the most simple of equations chills their analytic
abilities.2 For too many students the stumbling block is, among other things, related to
distinguishing between measures of association and statistical significance. Finally, some
students resent having to take the course. Many question the need for data analysis
course. This obstacle to learning is never easy to overcome, particularly when faced with
the other two noted above. As will be illustrated below, some students avoid taking the
data analysis until it is absolutely necessary for them to do so to graduate with a political
science degree. However, this may mean that the effectiveness of the course - its ability
to inform and aid the student’s education in substantive courses - is undermined.
I do not speak for my colleagues that share teaching responsibilities in this course, but I ask
students to do mathematics with the hope that it helps some understand differences in some techniques.
Still, I underscore that the heavy lifting of statistical analysis is most frequently done via computer
statisticalpackages. In addition, I present statistical techniques that are, in today’s terms, very
unsophisticated. If I can move through the material sufficiently, I can introduce students to multivariate
forms of linear regression, and it is the most cursory of introductions.
6
Despite the challenges of teaching the data analysis course, it provides the benefit
of refreshing and solidifjmg, at least for me, core ideas related to social scientific
research. Like teaching any course, you have to have more than expertise with the topic.
You also have to be able to communicate that expertise to students that are not only
unfamiliar with the material, but also potentially resistant to learning. However, the
expectation of the course is that it will provide basic introduction to key ideas of
scientific inquiry. The assumption is that students that have taken this course will be
more sophisticated consumers and producers of political science research, and ultimately
that those students that have taken it will successhlly complete a senior seminar.
SAMPLE
The sampling frame for this analysis was taken from the day-one rosters of four
senior seminars taught between fall of 1998 and spring of 2003. The day-one rosters are
kept on file in the Political Science Department. The start point of fall 1998 was selected
because it is the first semester after semester conversion. The four senior seminars
include PSC 420 Seminar in American Politics, PSC 42 1 Seminar in Comparative
Politics, PSC 422 Seminar in International Relations, and PSC 423 Seminar in Legal and
Political Theory. Twenty (20) seminar classes were offered during this time period, with
a total of 256 students enrolled on the first day.3 With the exception of one PSC 421
Seminar in Comparative Politics seminar taught during the summer of 1999, all were
conducted during fall and spring semesters.
3
Six PSC 420 American Politics, eight PSC 421 ComparativePolitics, five PSC 422 International
Relations, and one PSC 423 Legal and Political Theory seminar classes were conducted during the period
between the fall of 1998 and spring of 2003. One PSC 420 American Politics seminar was taught by a parttime instructor (Dr. Plax) and one PSC 422 InternationalRelations was taught by a term appointment (Dr.
Lavelle).
7
A random sample of seminar students was drawn roughly equal to two-thirds of
all students enrolled in the four seminars. Table 1 depicts the comparison of the
population with the sample by seminar type, and it indicates that the sample is a
reasonable approximation of the population. The sample slightly over-represents the
number of students in the American Politics and International Relations seminars and
slightly under-represents the number of students in the Comparative Politics and Legal
and Political Theory seminars.
Table 1 : Population and Sample by Seminar Type, Fall
1998 to Spring 2003.
Population Sample
Seminar
Percent
Percent
PSC 420 - American Politics
PSC 421 - Comparative Politics
PSC 422 - International Relations
PSC 423 - Legal and Political
Theory
Total
(58)
23.1%
(39)
37.9%
(97)
36.7%
(62)
34.4%
(88)
35.5%
(60)
5.1%
(13)
4.7%
(8)
100.0%
(256)
100.0%
(169)
22.7%
Though the sample is a reasonable representation of the population, it is important
to note that the chief drawback to using day-one rosters as the sampling frame is the
number of students that did not complete the course. Because this is an analysis of the
impact PSC 25 1 Introduction to Data Analysis might have on the course grade a student
receives in a senior seminar, students that did not complete the course are eliminated
from the majority of this study. However, this data source provides an opportunity to
8
examine the extent to which students do not complete seminars and perhaps why. The
majority of this analysis is based on an examination of 160 cases (62.5%).
VARIABLES
The variables for this analysis are drawn from the unofficial transcripts of
sampled students that were enrolled in senior seminars between the fall of 1998 and
spring 2003. The data is available via the university’s administrative system. Because of
student confidentiality requirements, the data used in this paper does not indicate a single
student and the data access is restricted and confidential.
Dependent Variable: The dependent variable for this analysis is the grade a
student received in the senior seminar. The grades are posted in a traditional letter-grade
format ranging from A to F. Beginning in the Fall of 1999, the university moved to a +/grading scheme, and this change is reflected in the assigned of values to each letter grade:
A = 100; A- =93; B+ = 88; B = 85; B- = 83; C+ = 78; C = 75; D = 65; F =50.
Table 2 illustrates the percent, frequency, and means of the grades by type of
senior seminar. Overall, the grade distribution suggests that most students complete the
seminar in a satisfactory manner. Though some seminar instructors may feel that some
students are ill prepared for the seminar, the grades earned by students in seminars depict
that the majority of seminar students are completing the seminar in a more than
satisfactory fashion. Indeed, with a total average of a B (85.4%) and with less than a
quarter (23.8%) of all sampled students getting a C or less, seminar students do very well
9
in their respective ~ e m i n a r Still,
. ~ a sufficient level of variation among the grades exists
to conduct an appropriate analysis.
Table 2: Percent, Frequency, and Means of Student Grades by Senior
Seminar
Senior Seminar
Legal and
Total
American
Comparative
International
Political
Grade
Relations
Politics
Politics
Theory
420
421
422
423
A
42.1%
(16)
33.9%
(19)
32.2%
(19)
28.6%
(2)
35%
(56)
B
47.4%
(18)
39.3%
(22)
37.3%
(22)
57.1%
(4)
41.3%
(66)
C
7.9%
(3)
14.3%
(8)
18.6%
(1 1)
14.3%
(1)
14.4%
(23)
D
2.6%
(1)
3.6%
(2)
5.1%
(3)
..-
3.8%
(6)
89.6
(8.6)
83.8
(13.8)
84
(13.3)
87.9
(9.1)
85.4
(12.5)
Mean
Note: The numbers in parentheses in the letter grade component is the frequency. The number
in parentheses in the means componentrepresents the standard deviation.
Independent Variables: There are three independent variables: Data Analysis
Course, Cumulative GPA, and Substantive Preparation.
Data Analvsis Course: The Data Analysis Course variable is treated as a dummy
variable. If a student took PSC 25 1 Introduction to Data Analysis either before or during
the seminar semester, the student is scored a 1, otherwise the student is scored a 0. Table
It is assumed that the quality of seminar instruction and grading is uniform and that no grade inflation is
occurring
10
3 depicts the percent of seminar students that have taken the data analysis course either
during or prior to the seminar semester.
Table 3: Proportion and Frequency of
Seminar Students In Data Analysis Course
by Seminar Type, Fall 1998 to Spring 2003.
Seminar
Percent
PSC 420 - American Politics
81.6%
(31)
PSC 42 1 - Comparative Politics
46.4%
(26)
PSC 422 - International Relations
40.7%
(24)
PSC 423 - Legal and Political
Theory
Total
N
85.7%
(6)
54.4%
(87)
160
According to the sample data, over half of all seminar students have taken the
data analysis class. However, it is clear that the distribution of students having taken the
data analysis course varies among the seminar types. Over four out of five seminar
students in the PSC 420 American Politics and PSC 423 Legal and Political Theory have
taken the data analysis course whereas a little less than half of the PSC 42 1 Comparative
Politics students have taken the course and only about two-in-five of PSC 422
International Relations students have taken the course.’ Part of the disparity in the
proportions is associated with students’ majors.
It is worth noting that 10% (16) of the students in the total sample (n=169) took the data analysis course
the same semester that they took a seminar.
11
Table 4: Proportion and Frequency of Seminar Students
In Data Analysis Course by Major, Fall 1998 to Spring
2003.
Major
Political Science
Proportion
82.1%
(78)
International Relations
12.5%
(7)
Other*
11.1%
(1)
* Includes History, Economics, Spanish, Social Work, and Undeclared.
As Table 4 illustrates, over four-fifths of Political Science majors have taken the
data analysis course either during or prior to their seminar semester. By contrast, only
one-in-eight International Relations majors have taken the data analysis course prior to
(or during) their seminar semester. This presents a point worth emphasizing. First,
International Relations majors are not required to take the data analysis course as part of
the degree program and thus it is not surprising that the proportions are low. Given that
the data analysis course is not required it is interesting to see the number of International
Relations majors that, in fact, take the course. By contrast, Political Science majors are
required to take PSC 251 Introduction to Data Analysis, and are encouraged to do so
prior to their seminar semester. Though the timing of taking the data analysis course
varies, it is clear that most Political Science majors come to the seminar after having
taken the data analysis course. Still, this does raise the issue of the distribution of majors
by seminar type.
12 Table 5: Majors by Type of Senior Seminar, Fall 1998 to Spring 2003
Type of Senior Seminar
Major
American
Politics
PSC 420
Comparative International
Politics
Relations
PSC 421
PSC 422
Political
Science
97.4%
(37)
42.9%
(24)
45.8%
(27)
International
Relations
2.6%
(1)
48.2%
(27)
47.4%
(28)
8.9%
5.1%
(5)
(4)
Other*
Legal and
Political
Total
Theory
PSC 423
100%
(7)
--
60.1?Ao
(95)
35.4%
(56)
2.5%
(9)
* = Includes Historv. Economics. SDanish, Social Work, and Undeclared.
Table 5 illustrates the sample’s proportion of majors by seminar type. With the
exception of one International Relations major, all International Relations majors have
taken either PSC 42 1 Comparative Politics or PSC 422 International Relations. This is
not surprising given that either the Comparative Politics seminar or International
Relations seminar is required as part of the International Relations major’s degree
program. By contrast, PSC 420 American Politics students are almost exclusively
Political Science majors (the lone International Relations major being the exception).
However, Political Science majors make up roughly half of the Comparative Politics and
International Relations seminars.
Cumulative GPA: Cumulative grade point average (GPA) is introduced as a
control variable. Cumulative GPA is used here as a crude proxy for the educational skill
of the student. For the purposes of this analysis, educational skill is defined as a bundle
of skills that are essential to be an effective student. This bundle includes dimensions
such as cognitive ability, time management, writing skills, and institutional diligence
13 (following directions, class attendance). Ideally, it would be more effective to have
discrete measures for these bundled concepts, but (a) that data is not readily available and
(b) those issues are not the direct focus of this analysis. I expect that students with
greater educational skills will be better able to address the complex issues involved in a
senior seminar. The measure is the student’s cumulative GPA in the semester prior to the
student’s seminar semester. It is measured in the university’s established 4.0 scale
system.6
Table 6: Mean of Cumulative GPA by Type of Senior Seminar,
Fall 1998 to Spring 2003
Senior Seminar
American
Politics
PSC 420
Comparative International
Politics
Relations
PSC 421
psc 422
Legal and
Political
Theory
Total
PSC 423
Note: The number in parentheses is the standard deviation. The numbers in
brackets represent the ffequency.
Table 6 depicts the mean cumulative GPA for the entire sample and by seminar
type. According to these data, the grade distribution of the senior seminars (see Table 2
above) is consistent with the quality of the students taking the seminars. For instance, a
2.7 grade point average is equivalent to a B- letter grade and 3.0 is equivalent to a B. The
sample’s student seminar grades have a mean of 85.4% (B) and this is nearly equivalent
to the total Cumulative GPA of 2.92 (B- > B). In this regard, it perhaps should be
expected that the grades students earn in seminars are so seemingly high. The students
6
The quality points associated with the 4.0 scale is described in the Cleveland State University
Undergraduate Catalog 2002-2004, page 3 1.
14
that take a seminar have scored above average throughout their academic career, and it
stands to reason that they would continue to do well. I expect the variable Cumulative
GPA to be positively associated with the dependent variable, senior seminar grade.
Substantive Preparation: The final variable is a control for the substantive
preparation a student may have before taking a senior seminar. The assumption is that
the more exposure a student has had to substantive areas of political science the better
that student will perform in a senior seminar. This variable is measured as the total
number of political sciences courses completed prior to the student’s seminar semester.
This additive variable includes any course that was taken at another college or university
and transferred in as political science course credit. There are some obvious limitations
with this measurement.
One limitation is the measure counts all courses as equal whether the course is an
introductory course or a baccalaureate-level course directly related to the seminar’s
subject. It is reasonable to assume that courses specifically related to seminar will have
greater impact on the seminar grade. Second, it is a count and thus does not take into
consideration the quality of a student’s performance in the course. Presumably, students
that perfonned better in a political science course are likely to have greater understanding
of the material and, all things being equal, perform better in a seminar. Still, some of this
quality issue is likely accounted for in the cumulative GPA measure. Third, the measure
does not take into account the time that may lapse between taking any particular course.
Any decay that might occur in substantive preparation is not accounted for in this
measure. Finally, it treats the quality of instruction as uniform when in fact the quality of
that instruction may vary from types of instructors to types of institutions.
15 Tables 7 and 8 depict the mean number of political science courses students have
taken prior to the seminar semester by seminar type and major, respectively. On average
a seminar student has taken seven courses prior to the seminar semester (MEAN = 6.77).
However, there is variation in the number of political science courses students have taken
by seminar type. Students in the PSC 420 American Politics and PSC 423 Legal and
Political Theory seminars have taken, on average, eight political science courses
(MEANS = 8.02 and 8.57, respectively). By contrast, students in the PSC 421
Comparative Politics and PSC 422 International Relations seminars have taken between
six and seven political science courses (MEANS = 5.64 and 6.81, respectively). The
difference in the means by seminar type and major provide an interesting insight into
student preparation for the seminar, and much of this variation is likely attributable to
major guidelines.
A Political Science major should have taken PSC 111, PSC 25 1, either PSC 221
or PSC 23 1, two American sub-field courses, two Cornparative/Intemational sub-field
courses, and one Legal and Political Theory sub-field course prior to taking a seminar - a
sum of eight courses. By contrast, an International Relations major need only take three
non-seminar political science courses as part of the major’s core - PSC 23 1, PSC 328,
and one PSC International Relations/Comparative elective - with the potential to take
more political science courses in specific areas of concentration. In addition, the seminar
is considered part of the International Relations major core whereas the seminar is
considered a capstone course in the Political Science major. Because two separate majors
16 with distinctly different requirements take senior seminars, coupled with the distribution
of majors in seminars, variation is not only evident, but should be expected.
Table 7: Mean of Number of Political Science Classes
CompletedPrior to Seminar Semester by Seminar Type, Fall
1998 to Spring 2003
Senior Seminar
American
Politics
PSC 420
8.02
(3.1 1)
~381
Comparative International
Politics
Relations
PSC 421
psc 422
5.64
(2.81)
[561
6.81
(3.31)
[591
Legal and
Political
Theory
PSC 423
Total
8.57
(3.51)
c71
6.77
(3.23)
C1601
Note: The numbers in parenthesesrepresent the standard deviation and the
numbers in brackets represent frequency.
Table 8: Mean of Number of Political Science Classes
Completed Prior to Seminar Semester by Major, Fall
1998 to Spring 2003
Major
Proportion
Political Science
8.17
(3.03)
P51
International Relations
(2.26)
4.70
~561
Other*
4.88
(2.76)
[91
* Includes History, Economics, Spanish, Social Work, and Undeclared.
Note: The number in parentheses in the means represents the standard
deviation. The numbers in bracket represent the frequency.
'
As noted above, Political Science majors almost exclusively enroll in the American Politics and Legal and
Political Theory seminars and virtually all of the International Relations majors enroll in either the
Comparative Politics or International Relations seminars per degree program requirement.
17
Though there is variation in the number of courses by seminar type and major,
there are some interesting observations to be made based on these data. On average and
at a very basic level, students appear to be following programmatic guidelines before
taking a senior seminar. As noted above, a Political Science major should have taken
eight courses prior to taking a seminar and this is, on average, what they are doing.
Though further analysis would be needed to determine the extent to which Political
Science majors are following precisely the major’s guidelines, it is evident that most
students have completed a minimum of eight political science courses. In fact, in an
examination of the fiequency distribution of Political Science majors and the number of
courses taken prior to the seminar semester, nearly two-thirds (65.3%) of the sample’s
students receiving seminar grades have completed eight or more political science courses
prior to their seminar semester. By contrast, a little more than a third (35.7%) of
International Relations majors have completed at least political science courses, which is
the minimum required by this degree program. As a multidisciplinary degree program,
International Relations does not require students to complete a specific number of
courses, political science or otherwise, prior to taking the seminar. In addition, the
seminar is considered part of its core and not its capstone. It remains to be seen whether
taking more political science courses affects the grade outcome in senior seminars.
I treat the fall 1998 to spring 2003 as cross-sectional pooled time-series data. The
data is analyzed using multivariate least squares regression. In the following section, I
examine the entire sample to test whether having taken (or taking) the data analysis
course effects seminar grades, then I examine separate cuts of the data to examine
18 specific issues related to student skill, seminar type, arid major. Next, I examine whether
the quality of a student’s performance in the data analysis class affects the senior seminar
grade. Finally, I provide a brief analysis of students that either repeated or did not finish
a seminar (or both).
FINDINGS
Table 9 presents the results of the multivariate linear regression for student’s
seminar grades. The results indicate that the data analysis dummy variable has a positive
and statistically significant relationship with a student’s seminar grade. Those students
that have taken the data analysis course receive, based on these findings, about five and
one-half points more relative to those students that did not take the data analysis course,
all other things being equal. It is also worth noting that Cumulative GPA has a positive
Table 9: Multivariate Regression Results for Seminar Grades, Fall 1998 to
Spring 2003
Variable
Slope
Standard Error
t
Constant
55.55
4.38
12.69”
Data Analysis
5.49
2.08
2.64*
Cumulative GPA
9.89
1.38
7.17*
Number of Political
Science Courses
-.29
.32
,.91
Adjusted R2= .25
N = 160
* = Statisticallysignificant at the p
SO5
and statistically significant relationship with a student’s senior seminar grade. For each
one full point in a student’s cumulative GPA, all things being equal, that student will
19
receive nearly ten points for her seminar grade. In general, students with greater
educational skills do better in senior seminars. Ironically and somewhat troubling, the
number of political science cowses, a measure of preparation, is inverse but not
statistically significant.
Though it is evident that the data analysis course has a positive influence on a
student's seminar grade, three separate questions remain. First, does the data analysis
course have the same influence for students with high and low GPAs? Second, does the
data analysis course have the same influence across seminar types? Finally, does it have
the same influence for the two majors taking political science seminars?
To answer the first of these questions I divide the sample at the 3.0 cumulative
GPA mark. This number was selected primarily for convenience reasons, it's a good
round number, and secondly it is a value that is close to the overall mean of the sample's
cumulative GPA (2.92). The findings are depicted in Table 10.
Table 10: Multivariate Regression Results for Seminar Grade by High and
Low Cumulative GPAs, Fall 1998 to Spring 2003
GPAs S3.0
GPAs > 3.0
Variable
Standard
Standard
'lope
Error
'lope
Error
Constant
54.53
8.01
6.80"
49.16
13.02
3.77"
Data Analysis
9.64
3.16
3.05"
.76
2.55
.30
Cumulative GPA
9.35
3.16
2.95"
12.34
3.80
3.25"
Number of Political
Science Courses
-.27
.51
-.54
-.27
.39
-.70
Adjusted R2 = .18
N=85
* = Statistically significant at the p
Adjusted R2= .13
N=75
s.05
20 As is indicated, the influence of the data analysis course is positive for both low
and high categories, however, it is statistically significant for those students that have a
cumulative GPA less than 3.0. As with the entire data set, the cumulative GPA variable
is in the predicted direction and is statistically significant. In addition, the preparation
variable (number of political science courses) is inverse and not statistically significant.
These findings indicate that taking the data analysis class has more of an influence for
seminar students with lower cumulative GPAs, specificallybelow 3.0. Though the
results are not statistically significant for students with GPAs above 3.O, it does not mean
that the data analysis course does not influence the grade outcomes in other political
science courses.
Table 11 : Multivariate Regression Results for Seminar Grades b y Type of Seminar, Fall 1998 to Spring 2003
Senior Seminar
American Comparative International
Variable
Relations
Politics
Politics
Constant
60.22"
(6.26)
54.62"
(8.12)
51.10"
(7.32)
Data Analysis
7.65"
(2.94)
-2.81
(3.96)
12.19"
(3.82)
Cumulative GPA
8.35"
(2.09)
10.13"
(2.42)
12.67"
(2.36)
Number of Political
Science Courses
.001
(.374)
.I5
(-71)
-1.43"
(57)
56
.2 1
59
.34
N
Adjusted R2 3%
.38
* = Statisticallysignificant at the p 1.05
21 The next question relates to the influence of the data analysis course relative to
specific types of senior seminars. Because the sample number of students that completed
the Legal and Political Theory seminar is so small (n=7), it will be excluded from this
analysis. Table 11 illustrates the multivariate regression results for three types of senior
seminars.
Because the sample sizes are so small for each seminar type, caution must be
exercised in generalizing about these results. The data analysis dummy variable is
positive and statistically significant for the PSC 420 American Politics and PSC 422
International Relations seminars. However, it has an unexpected inverse relationship for
PSC 42 1 Comparative Politics seminar grades, though not statistically significant. Given
its statistically significant and positive relationship, cumulative GPA remains a reliable
predictor of seminar grades. Finally, the preparation variable takes still another curious
turn in these findings. Its slope is positive for both PSC 420 American Politics and PSC
421 Comparative Politics, but is not statistically significant. However, the number of
political science courses a student has taken prior to the PSC 422 International Relations
seminar is inverse and statistically significant. To the extent that this small sample size is
accurate, this finding indicates that the more political science courses a student has taken
hurts, not helps, a student’s grade in the PSC 422 International Relations seminar.
Next, I turn to the final question regarding the effectiveness of taking the data
analysis course on seminar grades by major. The senior seminar is required for both the
Political Science major and the International Relations major, although as noted above,
there are key differences. In brief, the data analysis course is required for Political
Science majors and the senior seminar is a considered the capstone course for the major
22 whereas the data analysis course is not required for International Relations majors and the
senior seminar is part of the core for International Relations majors.
Table 12: Multivariate Regression Results for Seminar Grade by Major, Fall 1998 to Spring
2003
Type of Major
Political Science Majors
Variable
International Relations
Majors
Standard
'lope
Error
'lope
Standard
Error
Constant
58.35
5.7
10.24"
58.62
6.97
8.42"
Data Analysis
8.97
2.75
3.26*
SO
4.67
.lo6
Cumulative GPA
8.77
1.71
5.14"
9.80
2.11
4.64"
Number of Political Science
Courses
-.55
.35
-1.56
-.37
.68
.58
Adjusted R2= .26
N=95
Adjusted R2= .27
N = 56
* = Statistically significant at the p 1.05
Table 12 depicts the regression results for the two majors separately. For Political
Science majors, the hypothesis that taking the data analysis course improves seminar
performance is verified. The data analysis course provides an advantage to Political
Science majors taking a senior seminar. Controlling for a student's cumulative GPA and
the number of substantive of political science courses taken, the Political Science major
that has taken the data analysis course has nearly a letter grade improvement over her
non-data analysis seminar colleagues. However, this was not the case.for International
Relations majors.
23 According to these findings, the decision not to include the data analysis course as
part of the International Relations major degree program may have been appropriate.
Still, there are two problems. The first problem is that there may not be a large enough
variation in the International Relations majors category data to render an effective
analysis. As noted above (see Table 8) only seven International Relations majors took
the data analysis course. The second problem is that for students enrolled in PSC 422
International Relations, and roughly half of this sample’s International Relations majors
(47.5%)did (see Table 5 ) , the data analysis course variable had a positive and statistically
significant result (see Table 11). Given these cautions, the findings of the International
Relations major category may be too inconclusive to form any concrete generalizations.
Attempting to c1ariQ this issue, I examine the seminar grades for the two seminars
International Relations majors most frequently enroll, PSC 42 1 Comparative Politics and
PSC 422 International Relations.
In this cut of the data, I adopt a descriptive strategy. Throughout the analysis a
lingering issue has been the extent to which the data analysis variable was also capturing
variance associated with International Relations majors. Conducting another linear
regression analysis adding a dummy variable for International Relations major will not
necessarily resolve this issue. By doing so, in effect, the analysis will measure the set of
students that were not International Relations majors or had not taken the data analysis
course. Instead, I retreat here to a descriptive analysis of the two majors and the data
analysis course. Since my chief concern is the performance of International Relations
majors, I eliminate from consideration students in either PSC 420 American Politics or
PSC 423 Legal and Political Theory leaving 115 cases.
24 Table 13: Means of PSC 421 and PSC 422 Seminar Grades by Major and
by Data Analysis, Fall 1998 to Spring 2003
Seminar Course
Data Analysis
Number
No
Yes
International
Relations
42 1
422
86.7 (23)
86.4 (25)
77.5 (4)
86.0 (3)
Political
Science
42 1
422
84.8 (4)
72.2 (6)
85.2 (20)
86.8 (21)
Major
Note: P.mbers in parentheses are ,.zquencies
Table 13 presents a two-by-two of seminar grade means for the two courses by
major and data analysis course. The means depict a puzzle. The few International
Relations majors that have not taken the data analysis course perfonn on average worse
in the two seminars than those students that did not take the data analysis course. And,
the few Political Science majors that did not take the data analysis course performed
more poorly in the two seminars than did those Political Science majors that did.
curious.
So far, the results of this analysis indicate that taking the data analysis course
matters. It matters particularly for those students that have overall lower cumulative
GPAs, but not so much so for students with higher GPAs. For those students that have
generally lower GPAs the data analysis course provides the kind of training that improves
their prospects of attaining a better grade than their colleagues that have not had the data
analysis course. In addition, the data analysis course is positively associated with
seminar performance for students taking PSC 420 American Politics or PSC 422
International Relations. However, it not only has no statistically significant effect for
25 seminar students in PSC 421 Comparative Politics, the inverse slope suggests that it is
likely to do harm to their performance. Finally, the data analysis course provides for
Political Science majors a boost in seminar performance, but not so for International
Relations majors. After having determined that the data analysis course has a positive
effect for most students, I now turn to the question of whether the performance in the data
analysis course and when the course was taken has any effect on senior seminar grades.
In this section of the analysis, I examine only those students that have taken or
were taking the data analysis course and completed a senior seminar. This cut of the data
provides a sample of 86 students. With nearly 90% (89.7%, n= 78) of this slice of the
sample, this analysis is almost exclusive to Political Science majors. Still, the questions
is worthy of exploration. Given the nature of this analysis, I drop the dummy variable for
data analysis and include two new variables. The first variable is the grade a student
received in the data analysis course. The assumption is that students that perform better
in this course will be better able to understand the analytical components of senior
seminars. This variable was measured by translating the letter-grade metric into a
numeric metric as follows: A=l.O, A-=.93; B+=.88; B=.85, B-=.83; C+=.78; C=.75;
Dz.65; F=.50. The mean for this variable is ,858 (s = .12). It is expected that this
variable will be positively associated with seminar grades.
The second additional variable is a timing measure for the data analysis course.
The presumption is that students benefit most fkom the data analysis course if they take it
early after they have declared the major. This affords students the opportunity to
understand and take advantage of the substance of the data analysis course more fully by
aiding them in understanding the process of social science research. This variable is
26 measured in months. For example, if the student took the data analysis course during the
same semester as the seminar, the student was scored a 0. If a student took it in the fall
semester prior to taking a spring semester seminar, the student was scored a 1. If the
student took the data analysis course in the spring and then took the seminar in the
following fall, the student was scored a 3. In short, students were scored based on the
number of months that transpired between completing the data analysis course (end of the
semester) and beginning the seminar semester.
The mean for the data analysis course-timing variable is 12.1 (s = 18.43) with a
median of 10. Though the mean suggests a relatively ideal time period, the reality is
quite different.
In fact, nearly one-fifth of the sample’s students (18.6%, n = 16) took the
data analysis course the same semester they took and completed the senior seminar.
Overall, a third of the students took the data analysis course within three months of
enrolling in a senior seminar.8 There is strong reason to believe that a substantial
proportion of students do not take the data analysis course early enough in their major to
prepared them for the substantive baccalaureate-level courses. Still, the expectation is
that this variable will be positively associated with seminar grades.
In addition to the new variables, I continue to include the control variables of
cumulative GPA and preparation. Table 14 depicts the results of the regression analysis
for those students that have completed data analysis. The statistical results did not bear
out the hypothesis. Though both slopes were, as anticipated, positive the variables were
not statistically significant. It appears that neither how a student performs in the data
Sixteen students (18.6%) took the data analysis course during the seminar semester, eight students (9.3%)
took it within one month of the seminar semester, and five students (5.8%) took it within three months.
27
Table 14: Regression Results for Seminar Grades of
Students Completing Data Analysis Course, Fall 1998 to
Spring 2003
Standard
Variable
Slope
Error
Constant
59.78
8.52
7.02"
Data Analysis Grade
9.32
11.49
.8 1
Data Analysis Course-Timing
.001
.062
.18
Cumulative GPA
6.17
2.40
2.57'
Number of PSC Courses
.17
.41
.41
Adjusted R2= .12
N=86
* = Statisticallv simificant at the D 5.05
analysis course nor how recently or long ago a student took the course impacts the
seminar grade. However, based on previous findings, regardless of how a student did in
the class or when the student took the class, for many students in most seminars having
taken (or taking) the data analysis course positively affects their seminar grades.
Finally, there remain two issues to examine. There is the issue of those students
that did not complete the senior seminar and then there are the seminar repeaters. I
define a seminar repeater as a student that takes a seminar more than once. Ten students
fall into either one or both of these categories. This class of student is highly
idiosyncratic and does not appear to have any overarching pattern. For instance, two
students enrolled in a seminar, did not receive grades, and did not re-enroll in a seminar.
Another student enrolled in two separate seminars but did not receive a grade in either
seminar. Two students completed two separate types of seminars for passing grades, and
one student enrolled in three separate types of seminars but received grades for two. A
28
couple of students appeared to be seminar shopping, one appears to have erred in
enrolling in a seminar (this student completed a seminar about a year earlier for a grade
and dropped), and a couple of students seemed to like the seminar format. Half of the
students were Political Science majors and the other half International Relations majors.
All of the Political Science majors have taken the data analysis course and all of the
International Relations majors had not. Calculating the students’ cumulative GPAs based
on the most recent entry in the data set indicates a mean of 2.39 (s=.54), a few points less
than the overall GPA (2.92) for students that completed a seminar. In general, to the
extent this sample is a relatively accurate reflection of students enrolling in seminars and
assuming an enrollment of fifteen students, most faculty members teaching a seminar are
likely to not report a grade for at least one student in the seminar.’ This does not
necessarily mean, however, that the student will not attempt another seminar.
CONCLUSION
The central question posed in this analysis was whether PSC 25 1 Introduction to
Data Analysis had an impact on the outcomes of students’ seminar grades. Though the
answer is not without qualifications, the answer is that for most students taking the data
analysis course it has a positive and statistically sigmficant effect. After controlling for
educational skill and substantive preparation, students that have taken the data analysis
course are likely to improve their senior seminar grades by about one-half of one letter
grade, but depending on the specific student, it could be more or not at all. However, like
most empirical analyses this comes with several qualifications. First, there is the issue of
the proportion of variance what I can statistically claim to be explained.
The data indicate that some students had either a W, an X, or a ** on transcripts. In one case, a student’s
name was on the day-one roster, but the course did not appear on the student’s transcript likely indicating
the student dropped the first week of classes.
29
Table 15: Summary of Proportion of Variance Explained by Regression
Results
Adjusted
Description
Table Number
N
R2
Total Sample of Students
Completing All Seminars
Sample of Students
by Cumulative GPA
Sample of Students
by Seminar Type
Sample of Students
by Major
9
GPA 13.0
10
GPA > 3.0
American
Politics
Comparative
Politics
International
Relations
Political
Science Majors
International
Relations
Maiors
Sample of Students
Completing Data Analysis
Course
11
160
.25
85
.18
75
.13
38
.38
56
.21
59
-34
95
.26
56
.27
86
.12
12
13
Table 15 provides a summary of the Adjusted R2 ’s for the various regression
findings depicted in this paper. The answer that can be most confidently given, based on
the number of cases analyzed, is the first regression finding. The three variables, Data
Analysis Course, Cumulative GPA, and Number of Political Science Courses, explains
about a quarter of the variance in a student’s seminar grade. In some regression results,
most notably discrete analyses of specific seminars and the two majors, greater variance
is explained but the sample sizes are so small in some, I have less confidence in the
accuracy. Still, if we accept that a quarter of the variance is being explained, there is
three-quarters of the variance left unexplained.
30 On the one hand, this result is disappointing. I would like to think that the
variables presented here had a better predictive quality. Still, some of the control
variables have such puzzling results, most notably the measure for preparation, that there
may be substantial questions regarding measurement validity. On the other hand, the
results are enlightening and encouraging. Faculty members teaching a senior seminar
know with qualified confidence that those students taking the data analysis course will be
better prepared for the seminar. And, faculty members that teach the data analysis course
can now be more confident that what is taught is having a tangible and positive effect.
31 References
Cleveland State University, “Administrative Internal Homepages,”
https://viking.csuohio. edu/wdb/-internal-his-/admin
Cleveland State University, UndergraduateBulletin 2002-2004, Cleveland, OH: 2002.
Political Science Department, “Day-One Rosters,” Cleveland State University, Rhodes
Tower 1744.
32