Teaching Critical Thinking and Writing through Debates: An Experimental Evaluation
Author(s): Charles S. Green, III and Hadley G. Klug
Source: Teaching Sociology, Vol. 18, No. 4 (Oct., 1990), pp. 462-471
Published by: American Sociological Association
Stable URL: http://www.jstor.org/stable/1317631
Accessed: 30-09-2015 16:09 UTC
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at http://www.jstor.org/page/
info/about/policies/terms.jsp
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content
in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship.
For more information about JSTOR, please contact support@jstor.org.
American Sociological Association is collaborating with JSTOR to digitize, preserve and extend access to Teaching Sociology.
http://www.jstor.org
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions
TEACHING CRITICAL THINKING AND WRITING THROUGH
DEBATES: AN EXPERIMENTAL EVALUATION*
III
CHARLES
S. GREEN,
University
of Wisconsin-Whitewater
HADLEY
G. KLUG
ofWisconsin-Whitewater
University
Faced with the desire to maintain high standards of scholarship in a context of large
introductorylevel classes and limitedresources, the authorsexperimentedwith several active
learning techniques designed to develop critical thinkingand writing skills. The tentative
conclusionsdrawnare thatthe debateformatin the contextof large classes is an effectiveway
to modifystudents' opinions on social issues and to teach critical thinkingand writing skills.
Also discussed are several ethical issues involvedin teaching critical thinking.
BACKGROUND
Probably all faculty think that the teaching
of critical thinking and writing is a "good
thing,"just as motherhood and apple pie are
good things. The problem, of course, is that
unless one has plenty of teaching assistants,
the conventional term paper isn't feasible
in mass classes of 50, 100, or more because
the grading load is simply too high. Moreover, if one has ambitions of having students not only write papers but rewrite them
several times during a semester, the grading
task becomes virtually impossible. So, perhaps reluctantly, we give up the goals of
teaching critical thinking and writing and
fall back on lecturing and multiple choice
tests. But what if our consciences bother
us?
Our consciences botheredus, so for over
ten years we have tinkeredaroundin a variety
* Charles S. Green Ilmwas awarded the Hans O.
MaukschAwardfor DistinguishedContributionsto UndergraduateSociology in 1989 by the ASA Section on Undergraduate Education. Award recipients are required to
providea talkatthe following year's meetings. Green'stalk
before the 1990 meetings was based on this article. The
article is a revised version of a paper presentedby both
authorsto the 1990 annualmeetings of the Midwest Sociological Society in Chicago.
The authorswish to acknowledgethe help theircolleagues
renderedto thisprojectby gradingstudents'essays: Ronald
Berger, Eugene Grigsby, Lanny Neider, LawrenceNeuman, and Richard Salem. We also wish to thank our
colleague PatriciaSearles as well as Dean Dom and BarbaraShermanHeyl for providinguseful commentsduring
various stages of the project, William Rau and Jean C.
Karlen for comments during the Midwest meetings, and
the three reviewers of this version: Louis E. Anderson,
EugeneM. Labovitz,and NormaJ. Shepelak. This article
was copy edited by Donna Perdue.
of ways to try to give our students at least
some writing experience. Green, for example, gave his students the choice of doing
short papers on selected and well-defined
topics or of doing reviews of books from a
limited list of suitable books. Students were
also offered a "revision provision": if they
submitteddraftsof their shortpapersor book
reviews by a deadline early in the semester,
they were provided with constructive criticisms for use in revising their work. The
strategy worked reasonably well in that both
writing style and substance improved markedly from rough to final drafts. But the grading load was very demanding and after several semesters he got numbreadingpaperson
the same small number of topics over and
over again! Moreover, in large classes it is
impossible to give studentsthe opportunityto
improve their writing thr?ugh several revision and critique cycles. In addition, the
proliferationof paperson a limited numberof
topics semester after semester invites plagiarism.
Finally, it is not at all clear that papersand
reviews stimulate much critical thinking. In
fact, the natureof critical thinking was not
clear in our minds when we were first tinkering aroundwith our courses and there wasn't
muchliteratureavailablethenon teachingcritical thinking in sociology. We did share the
notion that critical thinking involved such
thingsas the abilityto createlogical arguments
basedon the "sociologicalimagination"and to
supportthose argumentsempirically. Therefore, we concluded thatone way to overcome
the shortcomings of short papers and book
reviews was to have small groupsof students
collaborate in preparinglibrary research papers. By collaborating, students would be-
TeachingSociology, 1990, Vol.18 (October:462-471)
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions
462
463
TEACHINGSOCIOLOGY
come more responsiblefor their own learning
and would teach one another,improving their
thinkingand writing skills (see Billson 1986;
Cohen, Lotan, and Leechor 1989; Rau and
Heyl 1990). In addition we anticipatedthat
since there would be a smaller numberof papers to grade, we could afford the time to
critique them and allow students to revise
them. We also expected that studentswould
acquirea sense of the vast arrayof empirical
and theoreticalliteraturethat exists in sociology andacquirethe skills for finding thatliteraturein the library. Finally, we believed oral
skills could be developed by having students
presenttheirresults in class.
Afterusing such projectsfor several years,
we concludedthattheydid indeedhave numerous advantages over short papers and book
reviews. However, the presentations the
groupsmadein class tendedto be stupefyingly
dull, for the students presented their ideas in
much the same fashion thatprofessorspresent
papersat the meetings of professionalassociations! A far more serious problem was that
therewas no directchallengeto the ideas being
presentedand, therefore,no need for students
to defend theirown argumentsand rebutthose
of others,both of which we thoughtshould be
crucialcomponentsin the developmentof critical thinkingskills.
In orderto overcome these difficulties we
decided to try a debate format. Starting in
1985, studentsin Green'sIntroductorySociology and Marriageand the Family classes were
given the choice of five debatetopics, with the
expectation that every student would participate in a debate. Students were assigned at
randomto the "pro"and "con" sides of their
debate topics. All students were given a detailed handout containing directions for researching their topic, starter bibliographies,
suggestions for coordinating their efforts in
preparingfor both the oral in-class debateand
the writtenversion of theirarguments,and the
standardsthat would be used in grading de1
The writing-across-the-curriculumapproach has
promised that writing skills can be improved if students
have frequentwriting assignmentsand that little increase
in gradingloads need resultfrom frequentwritingassignments because not all assignments need to be graded
rigorously. However,Day (1989, p. 458) in an experimental evaluation of the approachfound that ". .. numerous
writing assignments are not sufficient in themselves to
produce better writing, but that rigorous grading of the
entireassignmentis the variablewhich producesimprovement."
bates and papers. As had been done with the
earlierprojects,rough draftsof the group papers were submittedrelatively early in the semester and subjected to detailed critiques of
logic, evidence, grammar,and composition.
Students were then permitted to revise and
resubmitthese draftsby the end of the semester. Both the oral debate and the group paper
received group grades to emphasize the importanceof cooperation. However,in orderto
deter"free-riding,"groupswere empoweredto
eject any studentwho was not doing her or his
shareof the work.
We expected to gain a numberof advantages from this debateformat. First,the added
drama of debating a controversialtopic was
expected to generatemore interestin the presentationsand more post-presentationdiscussion than had been the case for the projects.
Second, it was expected that students would
not only develop researchskills in the process
of developing and finding evidence for their
argumentsbut also skills in oral and written
argumentationin the process of working out
defenses for those argumentsand rebuttalsto
opposition arguments. Third,it was expected
that, as was the case with projects, students
would learn more effectively when working
cooperatively than when studying alone.
Fourth,it was expected thatthe groupsformed
for the debates could be used for other class
discussions, thus providingfurtheropportunities for collaborativelearning.
Earlyexperiencein using debatesrevealed
that the first expectation was fully realized:
debateswere lively and elicited a greatdeal of
post-debate discussion on the part of class
membersnot debating. Moreover,it was clear
2 In
decidingupondebatetopics carefulconsideration
must be given to selecting topics which are currentand
controversial,but not so emotionally engaging that students are sharplypolarizedin theirinitial opinions, and to
selecting topics for which a substantialand reasonably
high quality body of literatureexists for both sides. Students must be informedthat gradingof both oral debates
anddebatepapersfocuses on the qualityof theirreasoning
andevidence,noton the side of the debatetopic which they
have been assignedto defend.
3Relatively few studentshave been ejected by their
teams, perhapsbecause the sanctionhas been so severe.
Recently, on students' advice, a less severe sanction has
been added. Now at the end of each semesterstudentsin
each teamare asked to evaluateeach other's contributions
to the team effort;if there is a consensus that a student's
performancehas been inferior,the instructorlowers that
student'sgrade.
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions
464
TEACHINGCRITICALTHINKING
from both the debatesand post-debatediscussions that some crucial critical thinking skills
were being used: students challenged each
other on such mattersas definition of terms,
unstatedvalue premises, the lack or misuse of
evidence, reliance on "authorities"whose expertise was questionable,the pertinence,reliability, and validity of evidence, and the adequacy of logic. It was not, however, clear
whetherthose skills had been learned in preparing for debates or learned previously in
other classes such as speech, a subject which
ourstudentsarerequiredto takeas partof their
generaleducationprogram. Finally, a number
of unexpectedfindings turnedup.
One surprisewas the extent of the impact
of the debateson multiplechoice exam scores.
Typicallyfoursuchexams were given, with the
thirdexam given afterthe firsttwo debatesand
the fourthexam given after the last three debates. In all classes, the scores (percentageof
items correct) on the last two exams were
significantly higher and had lower standard
deviations than did the first two exams. In
contrastto what we had expected, this effect
was not limited to the groupsdebatingbefore
eachexam butratherwas characteristicof most
students in each class. Thus, debating and
even hearingandlaterdiscussingthose debates
reinforcedstudents' learningof the text material parallelingthe content of the debates. A
second surprise was that the student evaluations of the course improvedsignificantlyand
the standarddeviations of evaluation scores
declined significantly comparedto the scores
from Green's previous courses, taught with
lecturescombinedwith groupprojectsor with
short papers and book reviews, and Klug's
courses, taughtin a conventional lecture for-
askedto writeon the studentcourseevaluation
instrument"Whatwere the best andworst features of this course?"about 40 percent chose
to do so. Of the latter,50 percent wrote specific and favorable comments about debates
while just 11 percentwrote unfavorablecomments. No other feature of the course was
mentioned as favorably.
Before analyzing data on exams and student evaluations, thereby becoming fully
awareof the firsttwo surprises,Greendecided
thatperhapshe had been too lenient in assigning letter grades to multiple choice exam
scores. So he raised the standards,hoping
therebyto achieve a Pygmalioneffect. In the
five course sections taught since these standards were raised, exam scores (percentage
correct, not letter grades) for the first two
exams have gone up significantly. However,
the higher expectations had no impact on
scores for exams three and four. Apparently
the impactof debatingoverwhelmedthe effect
of higherstandards.Incidentally,althoughthe
higher standardsresulted in sharplyreduced
proportionsof As given as final course grades,
thishadno noticeableeffects on students'evaluations of the course. In short,the majorimpact on those evaluations has come from organizing classes into small groupsfor debates
and otherdiscussion andclass exercises.
In order to explore the potential of the
debate formatfurther,we designed a series of
experimentsto determinethe impactof debates
on students' opinions about social issues and
on changes in thinkingand writingskills.
CRITICALTHINKINGAND
WRITING: EXPERIMENTS
mat. Whenstudentsin Green'ssectionswere
4
We regressed the grades from all four multiple
choice tests againsttime. Before debates were introduced
the slopes of all four lines were essentially zero, intercepts
of all four lines were nearly identical at about 60 percent
correctanswers,and Pearson'sR equalled roughly .70 (p
< .001) for all four lines. After debates were introduced
slopes were again found to be near zero for all four lines.
Intercepts for tests one and two remained at about 60
percentbut those for tests threeandfourincreasedto about
70 percentcorrectanswers. Rs remainedat about .70 for
tests one and two but increasedto about .80 (p < .001) for
tests three and four. Attemptsto fit a single line to all test
scores were not successful: neitherlinear nor several curvilinearregressionsproducedRs greaterthan.30 andnone
were significant. Gradesin Klug's courses remainedconstant,withregressionlines for his tests comparableto those
for Green's tests one and two.
We have already noted the rough-and-ready
definition of critical thinkingthat we used in
5 Students
takingcoursesin ourdepartmentareasked
to rate instructorson fourteen items, the last of which
requires an overall evaluation: "Taking everything into
account,how would you ratethis instructor?"The rating
scale varies from a high of one to a low of five. The mean
scores on this item were regressedagainst time. Before
debates were introduced, the regression intercept (i.e.,
mean evaluationscore) was 1.9 with a slope of zero and R
= .62 (p < .01). Afterdebateswere introduced,the regression interceptwas 1.6 with a slope of zero and R = .79 (p
< .005).
Attempts to fit linear and curvilinearlines
throughthe mean scores of all classes yielded non-significant Rs of less than .40. Student evaluationsin Klug's
classes were comparableto those in Green's classes before
debateswere introduced.
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions
TEACHINGSOCIOLOGY
465
our early tinkering with our courses. But a
review of the literaturesuggests that a more
precise and certainlymore sophisticateddefinition is in order. For example, Dean Dor
(1987) defines critical thinking as ". . . the
process of reasonably deciding what to do
and/orbelieve. This means individualsshould
be able not only to assess theirown andothers'
arguments but also construct good arguments."6 Dom points out that most experts
believe that critical thinking involves acquiring a set of micro and macro level skills and
dispositions:
1. Micro level intellectual abilities and
skills. Examples:abilityto clarify issues,
ability to identify fallacies in an argument, ability to find value assumptions
embedded in an argument, ability to
know when statistics are misleading or
absent, ability to judge whethera statement follows from a premise.
2. Macro level dispositions. Examples: an
interestin seeking reasonsfor what to do
or believe, skepticism about others' beliefs, a predispositionto ask for evidence,
a predispositionto be creative, to think
of counter-argumentsand examples, to
be sensitive to one's own biases and values.
3. Macrolevel values. Examples:a commitment to open-mindednessand fairness,
empathyfor others' positions, openness
to self-criticism, appreciation of the
value of looking at events from multiple
points of view.
lectures can be used to challenge students to
question the beliefs and values they take for
granted.Similarly,objective tests can be constructed to measure higher order cognitive
skills (see Howery 1987); whereas, essay
exams may tapmererecallof facts, figures,and
ideas. As Baker(1981)concludes "... critical
thinkingis not associatedwith any single patternof teachingand testing."
Mayer (1986) argues that ". . . critical
thinking cannot be accomplished unless the
goal is specifically built into course objectives
and methods." Teaching critical thinkingrequires what Goldsmid and Wilson (1980) call
"benign disruption":a set of techniques by
which the taken-for-grantedis challenged.
Moreover,such techniquescannotbe effective
if addressed solely to intellectual concerns.
Teaching techniquesmust instead link reason
and emotion (see Baker 1981; Nelson 1989).
In establishing this link "The key . . . lies in
creating conditions for participation rather
thanpassivity, and in providingopportunities
for emotionalengagementwith the materials"
(Mayer 1986). Debates appear to fit all the
criteriamentionedabove (see also Broderick
1982 and Huryn 1986). But do they actually
work? Can it be shown thatchangestakeplace
in studentsduringa semesterthatcan be attributed in whole or partto the debateexperience?
EXPERIMENT1:
HOW DOES DEBATINGAFFECTOPINIONSON
ISSUES?
We startedoff with a rathernaive hypothesis:
With a betteridea of what criticalthinking thatdebatingwould give studentsgreaterintelis, can we specify what are the best methods lectual flexibility (see Kohn 1971; Kohn and
for conveying these skills, dispositions, and Schooler 1983). Specifically,we predictedthat
values? The conventionalwisdom in the aca- those who had debated and had listened to
demic world is that lecturing and multiple othersdebatevariousissues pertainingto marchoice exams are useful only for impartingthe riage and the family (Green's students)would
recall of facts, while Socratic dialogue, essay become less extreme in their opinions than
exams, and term papers are best for teaching students who had also taken a marriageand
higher ordercognitive skills. But this dichot- family class but under more conventional
omy is far too simple. Socraticdialogues can teaching methods (Klug's students). To test
be used to intimidatestudents, teaching them these expectationswe administeredan opinion
only what to think, not how to think, whereas survey to two of Klug's and three of Green's
classes at both the beginning and the end of
the semester. To see if reactive effects were
6
Dom's approachtends to focus on the development
(for example if takingthe survey early
present
of skills of formalargumentationas developed by philosoon later opinions), we also adhad
an
impact
phersand rhetoricians.Otherapproachesinclude those of
the
ministered
survey only at the end of two
cognitive psychology, epistemology, and Marxist critical
of Green's, one of Klug's).
classes
other
(one
Nelson
1989).
theory (see
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions
TEACHINGCRITICALTHINKING
466
Table 1. Comparisonof OpinionChangeAmong Non-Debatersin ExperimentalandControlClasses, by Debate Topic
DebateTopic:
Classes:
Experimental
1. Enoughlove maintainsmarriage
2. Singlehoodpreferable
3. Trainedparentsproducemodelchildren
4. Women's liberationgonetoo far
5. Familyhas no future
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
ControlClasses
1
2
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
* Theproportion
of studentschangingopinionsin theexperimentalclass is significantlyhigher(p < .05, one-tail)thantheproportionof studentschangingopinionsin the controlclass;in no comparisonis theproportionchangingopinionsin the control
classsignificantlyhigher.
Binomialtests:
- Forall 30 comparisons,19 as predicted,p = .100 one-tail
- ExcludingDebate2: of 24 comparisons,21 as predicted,p = .001 one-tail
- ExcludingDebate2 andExperimental
Classl: of 16 comparisons,all as predicted,p = .000 one-tail
Thus we had a Solomon four-groupresearch
design (Selltiz, Wrightsman,Cook 1976, pp.
140-48 ). Studentswere asked to express their
opinions by indicating on a five item Likert
scale how strongly they agreed or disagreed
with each of the following debate topic statements:
1. "If two people love each other enough,
they can maintain their marriage until
deathdo them part."
2. "Singlehood is much preferableto marriage."
3. "Ifparentsaretrainedin modem methods
of child-rearing, they should have no
problemproducingmodel children."
4. "The women's liberationmovement has
gone too far."
5. "Thefamily has no future."
What were the results of this experiment?
Contraryto our initial naive hypothesis, very
few opinions among either controls or nondebatingexperimentalsshifted towardneutral
("I can see merit to both sides of this"), the
positionwe first thoughtindicativeof intellectualflexibility. In fact, substantialproportions
of students were neutralto begin with, especially with respectto topics 2, 3, and4. On the
other hand,roughly 60 percent of students in
the experimental classes and 40 percent of
studentsin controlclasses changed theiropin-
ions. Among those whose opiniondid change,
the changetypicallyinvolveda shiftof just one
categoryon the Likertscale (for example from
strongly agree to agree). Further,it is clear
from the resultsin Table 1 thatthe experimental "treatment"
did not lead to a shift of opinion
for debate topic 2 nor within experimental
class 1. Interestingly,it was only in the latter
class that-despite directionsto thecontrarystudentsin several debates issued disclaimers
to theeffect that"Ireallydon'tbelieve this, I'm
just defendingthis positionbecauseI have to."
Whenwe examinedopinionchangeamong
debaterswe found something quite different.
Those who debated a side ("pro"or "con")
opposite to their initial opinion changed their
opinionsharply,typicallyswitchingtheiropinion to the side uponwhich they debated.Those
who were initially neutral were even more
likely to switch theiropinions to coincide with
the position debated. In contrast, those who
debated a side coinciding with their initial
opinion either did not change theiropinion or
strengthenedit (see Table 2). These differing
patternsof opinion shift reflect differences in
the extent to which studentsexperiencedcognitive dissonance. Furthermore,those who experienceddissonancechose one of threepossible modes of resolving it (see Brehm and
Cohen 1962; Michener,DeLamater,Schwartz
1986, pp.184-86). When asked to defend in
public views with which they disagreed,stu-
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions
TEACHINGSOCIOLOGY
467
Table 2. Changeof OpinionsAmong Debatersin ExperimentalClasses, by Section and by CorrespondenceBetween
Initial Opinionand Position Debated
InitialOpinionRelativeto PositionDebated,by Section
Section 1 (n = 55)
OppositeNeutralSame
OpinionChange
Section2 (n = 50)
OppositeNeutralSame
Switched*
Shiftedto neutral
Weakened**
No change
Strengthened***
3
4
4
10
0
14
0
0
3
0
1
1
0
10
5
8
3
2
3
1
Sub-totals
Gamma
Chisquare****
P<
21
17 17
.25
16.97
.001
17
*
**
***(
**1(*
Section3 (n = 43)
OppositeNeutralSame
1
0
4
7
6
6
7
0
1
0
15 18
.51
14.48
.001
14
13
0
0
2
0
11
0
0
4
0
0
0
0
10
4
15
14
.65
2754
.001
or vice versa;or fromneutralto positiondebated.
Fromagreementto disagreement
Fromstronglyagreeto agreeor fromstronglydisagreeto disagree.
Fromagreeto stronglyagreeor fromdisagreeto stronglydisagree;or fromneutralto positionoppositethat
debated.
Forpurposesof calculatingChi square,sub-tableswerecollapsedby dichotomizingthe opinionchangevariable
between"weakened"
and"nochange."
dentsfaced threechoices: 1) they could change
theiropinionto conformto the view defended,
which most students did; 2) they could keep
their initial opinion, claiming publicly that
they "don'treally " feel or think"thatway"which several in experimentalsection 1 did,
despite instructionsto the contrary;or 3) they
could keep their initial opinion, excusing to
themselves and possibly to their significant
others the apparent inconsistency between
opinion and behaviorby saying "I was forced
to defend thatside." Several studentsconfided
to Green that this is just what they did. Incidentally, we were able to find no reactive effects of takingthe opinion survey at the beginning of a class: among both the experimental
(debate) and control classes, the distribution
of opinions at the end of a semester had no
relation to whetheror not a class had taken a
survey at the beginning of thatclass.
pers producedand the tape recordingsof selected oral in class debates. However, this
would be quite time consuming,and so far we
have found no one willing to do it. Furthermore,such a strategywould leave unanswered
the more importantquestionof whetheror not
studentsacquiredskills which would transfer
to new situations. In fact, the literatureis quite
pessimistic about such a possibility (Baker
1981, p. 341) for it appearsthatthe teachingof
critical thinking and writing succeeds only
when it is taught across the curriculum
(Browne and Litwin 1987).
In order to examine the transferabilityof
skills, the authors had every student in their
spring 1989 MarriageandFamily classes (one
section each) write an essay at the beginning
of the semesterdefendingtheirposition on one
of the five debate topics outlined earlier.
(Green'sstudentswere askedto pick one of the
four topics they were not assigned to debate.)
We also had them write on those same topics
at the end of class. Thus, we had"before"and
EXPERIMENT2:
"after"essays from each student which we
HOW DOES DEBATINGAFFECTTHINKINGAND
WRITINGSKILLS?
planned to have gradedblindly by other colleagues in our department.
In orderto keep the gradingload for these
As we noted earlier,debateshave a numberof
over
other
teaching techniques, faculty within some reasonable bounds, we
advantages
even including those techniqueswhich place each picked a 25 percent sample of our stusome emphasis on writing (such as research dents at random: 11 from Green's class, 10
projects,book reviews, shortpapers). But, so from Klug's. Thus from these 21 students
far we have not demonstratedwhetheror not there were 42 essays to be evaluated(21 "bedebatinghasany impacton thinkingor writing. fore" and 21 "after")by each of two faculty
The best demonstrationwould involve a care- members. We assigned essays to faculty at
ful externalevaluation of both the group pa- randomso thateach would gradeessays from
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions
468
TEACHINGCRITICALTHINKING
Table 3. Reliability of Faculty Evaluationsof the Quality
of Thinkingand WritingDisplayed in Students'
Essays
Thinking
1. Differencesin absolute
scoresbetweenthe
two gradersof each
essay:*
a. Meandifference
b. Mediandifference
2. Rankordercorrelation
(Kendall'stau)
betweeneachgrader's
scoresandthose
assignedby other
gradersof those
sameessays:**
Grader1
Grader2
Grader3
Grader4
Grader5
12.64
10.00
Table 4. FacultyEvaluationsof the Qualityof Thinking
andWritingin Beginningand FinalStudentEssays, by ExperimentalCondition
Writing
7.02
5.00
Experimental
1. TheQualityof Thinking
Meanscoreof two graders:
Beginningessay
Finalessay
Meanchangein scores:
2. TheQualityof Writing
Meanscoreof two graders:
Beginningessay
Finalessay
Meanchangein scores:
.804
.921
.724
.889
.902
.841
1.000
.871
.946
.935
* Scoringwas basedon a 100 pointscale;see Appendix
forscoringdirectionsgiven faculty.
** Forall taus,p < .0000 one-tail.
bothexperimentalandcontrolclasses andeach
wouldgrade"before"and"after"essays. Each
faculty member was provided with a set of
guidelines for grading the quality of both
students' writing and critical thinking (these
guidelines may be found in the Appendix).
Before turningto the results of the faculty
gradingefforts, it is importantto discuss
Control
65.6
75.9
10.4
p <.0005
one-tail
63.6
67.1
3.5
p NS
one-tail
71.6
77.3
5.6
p <.0005
one-tail
68.7
72.8
4.1
p NS
one-tail
obtained by averaging for each student the
scores assigned by each faculty grader. The
statistical significance of the results is not
changed if, insteadof averagingthe scores of
the two graders,the scores of just the first or
just the second graders are analyzed, or if
the scores of the gradersassigning the worst
grades are analyzed, or if the scores of the
gradersassigningthe best gradesareanalyzed,
or if the scores of graderspickedat randomare
analyzed. Thereforeit doesn't seem to matter
which set of scores are chosen for analysis.
Part2 of Table4 shows thatstudentsin the
experimentalclass improvedtheir writing by
almost six points, a statistically significant
change; whereas, studentsin the control section improvedby just fourpoints, a difference
which is not a statistically significant result.
Again, tinkeringaroundwith scoring methods
does not change the significanceof the overall
result.
In summary,the results look quite good:
debatesdo seem to work in that they produce
improvementsin both the quality of thinking
and the quality of writing. However, a more
cautious, even pessimistic, interpretationcan
be made of the resultsin Table 4. Even in the
experimentalclass, final essays werejudged to
be about C quality work with respect to the
qualityof thinking(a score of 75.9) and about
the same quality with respect to writing (a
score of 77.3). We would be much more encouragedby B qualitywork.
whetheror not the grading of the essays was
reliable;for example,do facultyagreeon what
is good critical thinking and good writing?
Note the resultsreportedin Table 3. Part 1 of
thatTable suggests thatfaculty have quite differentstandardsin assigning absolutenumerical scores,especially to the qualityof thinking.
They arealmost 13 points apart,the equivalent
of a grade and a half or the differencebetween
an A and a B-. On the otherhand,Part2 of the
Tablesuggestsa very highdegreeof reliability,
for facultydo agreeon the rankorderingof the
quality of students' essays. For example, the
rank ordercorrelationbetween the scores assigned by Grader1 for writing and the scores
assigned by the other graders of l's set of
essays is .804. With some assurancethatthe
grading is reliable, we turn to the actual researchquestionsof interest.
The results are summarized in Table 4.
Part1 of thatTable shows thatthe experimental class improvedthe qualityof its thinkingby
DISCUSSION
a full ten points, a highly significant change,
whereasthe controlclass improvedbarelyfour
points,which is not significant. It is important We have shown in this paperthatdebates can
to note thatthe scores reportedin Table4 were be effective in improving students' perfor-
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions
TEACHINGSOCIOLOGY
469
manceon multiplechoice exams covering materialrelatedto the debates,in raisingstudents'
participationin and enthusiasmfor class discussions, and in raising students' evaluations
of instructors.Furthermore,there is evidence
for modest gains in writing and critical thinking skills and evidence that these skills are
transferable.However, the study does need to
be hedged with several qualifications.
From a methodological perspective, our
experimentsare not true ones: students were
not assigned at random to experimental and
controlclasses. Therefore,thereis the risk that
the results are due merely to differences in
composition between classes. We have tried
to rule out the effects of several potential
sources of bias in the composition of classes.
For example, the distributionsof studentsby
year in college and by majorfield are the same
between our classes. Moreover, initial opinions about the five debate topics are similar.
Butwe cannotruleout moresubtledifferences:
might students have chosen Green and Klug
for different reasons which might have affected their performance in class? In fact,
there is a hint that such is the case, for a
somewhat higher (but not quite statistically
significant)percentageof Green's classes are
composed of females. Differences in the way
we conduct our classes also may have had an
impact on one or more aspects of students'
performance. One such difference, our textbooks, can probablybe ruledout: Greenused
several differenttextbooksbefore introducing
debates with no noticeable effects on student
evaluations or on grades on multiple choice
tests. In addition,we have no way of knowing
to what extent opinion change may have been
influenced by students knowing or guessing
that Green expected such change. Nor do we
know how those debaterswho faced a dissonance arousing situation were influenced by
the presenceof friendsin the audiencefor their
debate.
Our conclusions with respect to improvementsin thinking and writing skills are
based on just 21 students. If we had more
faculty on our staff or could have inducedour
present faculty to grade far more essays, we
would have greaterconfidence in our results.
Thus, the study cries out for replication.
This study provides no evidence concerning the permanenceof either the changes in
opinions induced or of the improvements in
writing and critical thinking skills. We also
do notknow whetherthe latterskills will transfer to other sociology courses, let alone to
course work in otherdisciplines. It should be
obvious thatfollow-up studies are needed.
Debates seem to be like other methods of
stimulating critical thinking in that they 1)
encounterstudentswith views contraryto the.ir
own and,in so doing, 2) inducestudentseither
to change those views or learn to defend their
own views with betterlogic and more substantial evidence. We wish to emphasize, however, that debates are not the only way to get
studentsto thinkmorecritically,andtheyprobably are not the best way-of course the "best
way" has yet to be found. Nevertheless, debates are especially advantageous in large
classes. By subdividing a large class into
smaller groups, instructorscan encouragecooperativelearning,not only for the debatesper
se but also for otherclass discussions or exercises and for preparingfor exams. A final
advantageof debatesis thatinsteadof grading
(or giving up on grading) 50 or more papers,
the instructorhas to gradejust one paperfrom
each debate team.
We wish to close this articleby noting two
ethical issues involved in teaching critical
thinking. First, are we engaged in a form of
brainwashingwhen we get students'opinions
to change sharply,especially underthe condition in which they mustdefendin public views
to which they are initially opposed? The second issue, one raised by a reviewer, concerns
whether our teaching of critical thinking
merelycreatesamoralmonsters:personsadept
at criticismbut lacking both a sense of values
and a commitmentto acting on those values.
Thus, could it be that critical thinking and
commitment are actually incommensurate,
7 The writing of formal, stylistically correct papers
may not be essentialin the teachingof criticalthinking. In
teachingduringsummersessions oftenjust threeweeks in
length, Green has found that it is almost impossible for
studentsto write up their debatepapers;consequently,he
no longer requirespapersduringthe summer. However,
adequatepreparationfor the oral debates requiresconsiderable notetaking,the sharing of those notes with one's
debateteam members, and the reorganizationof data and
potentialpoints of argumentfrom those notes into a logically interrelatedset of arguments. Each debate team
typicallyhas a well developedoutlineandarrangesto have
each member present certain arguments in a specified
sequence. The betterpreparedteams will also have members assigned in advanceto rebuttheiropposition. Therefore, considerablewriting(albeitinformal)does takeplace,
together with much discussion of how to organize arguments and mobilize evidence for best effect.
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions
TEACHINGCRITICALTHINKING
even mutually destructive?Or is it possible
thatstudentscan be taughtto act as responsible
citizens while fully and freely acknowledging
the weaknesses in the basis for thataction (see
Paul 1984)?
470
- valueassumptionsleft unstatedand
unrecognized.
- citesthe opinionsof personswhoseexpertiseis
dubiousorundocumented.
APPENDIX
DIRECTIONS
FORFACULTYGRADINGTHEESSAYS
1. Pleaseassigntwo gradesto eachessay,one forthe quality
of writingandone forthequalityof argumentation
or
criticalthinking.Guidelinesforeachof theseareprovided
below.
a. Use a numericalscale for eachgrade:
NumericalScale:
RoughEquivalency:
A or excellent
B or aboveaverage
C or average
D or below average
F or well below average
b. NOTE-you do nothaveto provideanywritten
comments,critique,or rationaleforyourgrade. In
fact,you shouldnot writeanythingat all-ontheblue
books.
91-100
81-90
71-80
61-70
60 andless
2. Someguidelinesto use in gradingthe essays:
a. Qualityof writing
i. Is theessay well organizedandcohesive-are
argumentsclearlystatedandis anoutlineclearly
discernible,albeitimplicit?
ii. Arethe sentencescompleteor aretheymere
fragments?Aresentenceslinkedtogetherinto
meaningfulparagraphs?Arethereeffective
betweenparagraphs?
transitions
iii. Is grammatical
use
usageproper(e.g., punctuation,
of tense,avoidanceof splitinfinitives)?
iv. Arewordchoicessuitable?Arewordsspelled
correctly?
or of "criticalthinking."
b. Qualityof argumentation
i. Aresociologicalconceptsusedto illuminateand
analyzethe issue?Does the writerindicatean
awarenessof historical,cultural,andsocialstructural
(e.g., class, gender,age, race)contexts?
ii. How adequateis the logic of the argument?Some
commonlogicalgoofs to look for:
- uses evidenceselectively,or uses outof date
evidence,or uses examplesof dubious
pertinence.
- generalizesfrompersonalexperienceor uses
adhominemarguments.
- fails to definekey termsor uses circular
definitions.
in
-insensitive to weaknessesandcontradictions
own logic and/orevidence.
- appealsto numbersor prevailingopinion(e.g.,
"weall knowthat...;" "sincemost people
thinkthat... it followsthat..."-the
bandwagonrationalel)
- not sensitiveto questionsof reliabilityand
validityof evidence.
- oversimplifiesormisrepresents
opponents'
arguments-thestrawpersonfallacy.
REFERENCES
Baker, PJ. 1981. "Learing Sociology and Assessing
CriticalThinking."TeachingSociology 8:325-363.
Billson, J.M. 1986. "The College Classroom as a Small
Group: Some Implications for Teaching and Learning." TeachingSociology 14:143-151.
Brehm, J.W., and A.R. Cohen. 1962. Explorations in
Cognitive Dissonance. New York:Wiley.
Broderick,J.J. 1982. "Using Debateto TeachSociology."
ASA TeachingNewsletter 7:8-9.
Browne, M.N., and J. L. Litwin. 1987. "CriticalThinking
in the Sociology Classroom: Facilitating Movement
from Vague Objective to Explicit Achievement."
Teaching Sociology 15:384-391.
Cohen, E.G., R.A. Lotan, and C. Leechor. 1989. "Can
Classrooms Learn?" Sociology of Education 62:7594.
Day, S. 1989. "ProducingBetter Writers in Sociology
Classes: A Test of the Writing-Across-the-Curriculum
Approach."Teaching Sociology 17:458-464.
Dom, D.S. 1987. "CriticalThinking."Presentationat the
Spring Teaching Seminar, University of WisconsinWhitewater.
Goldsmid, C.A., and E.K. Wilson. 1980. Passing on Sociology. Belmont, CA: Wadsworth.
Howery, C.B. 1987. "Measuring Up to Writing Good
Multiple Choice Tests." ASA Footnotes 15 (February):9.
Huryn,J.S. 1986. "Debatingas a Teaching Technique."
Teaching Sociology 14:266-269.
Kohn,M. M. 1971. "BureaucraticMan:A PortraitandAn
Interpretation." American Sociological Review
36:461-474.
,and C. Schooler. 1983. Workand Personality.
New York: Aldine.
Mayer, J. 1986. "TeachingCritical Awareness in an IntroductoryCourse."Teaching Sociology 14:249-256.
Michener, H.A., J.D. DeLamater, and S.H. Schwartz.
1986. Social Psychology. New York:HarcourtBrace.
Nelson, L. 1989. "CriticalThinkingAboutCriticalThinking." UndergraduateTeachingImprovementCouncil,
University of Wisconsin System. Teaching Forum
11(April):1-3.
Paul, R.W. "CriticalThinking:Fundamentalto Education
for a Free Society." EducationalLeadership42:4-14.
Rau, W., and B.S. Heyl. 1990. "Humanizingthe College
Classroom:CollaborativeLearningand Social Organization Among Students." Teaching Sociology
18:141-155.
Selltiz, C., L.S. Wrightsman,and S.W. Cook. 1976. Research Methods in Social Relations. 3rd ed. New
York:Holt, Rinehart.
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions
TEACHING
SOCIOLOGY
471
Charles S. Green Ill is Professor of Sociology at the
University of Wisconsin-Whitewater. His interests include marriageand the family, critical thinking, and applied sociology. Address correspondanceto Charles S.
Green lm, Departmentof Sociology, University of Wisconsin-Whitewater,Whitewater,WI 53190.
Hadley S. Klug is Associate Professorof Sociology at
the Universityof Wisconsin-Whitewater.He has interests
in marriageand the family, ruralsociolgy, andintentional
communalgroups. He is doing researchon changesin the
curricularofferings of those communalgroups known as
"graduatedepartmentsof sociology."
Por
caicultq ancl Profession
s witk A\utboritq
Recommend Journcis for Tleir LibrarL
to
tosubscribetothisjournal?Ifyouhavetheauthority
Wouldyoulikeyourlibrary
to
to
use
the
form
below
to
a
recommend
order
journals yourlibrary,
complimentary
review.
samplecopyforyourlibrarian's
O Yes, pleasesend a samplecopyof:
(journal
name)
Thenameofmy
Iwillsubmitthecopytoourlibrary
forsubscription
consideration.
andlibrary
are:
institution
Name
Institution
Address
State
Zip
City
AmericanSociologicalAssociation,
Returnformto:PublicationsDepartment,
1722 N Street, N.W.,Washington,DC20036
This content downloaded from 152.20.158.206 on Wed, 30 Sep 2015 16:09:49 UTC
All use subject to JSTOR Terms and Conditions