Bringing up Scientists in the Art of Critiquing Research
Author(s): Barbara J. Kuyper
Reviewed work(s):
Source: BioScience, Vol. 41, No. 4 (Apr., 1991), pp. 248-250
Published by: University of California Press on behalf of the American Institute of Biological Sciences
Stable URL: http://www.jstor.org/stable/1311414 .
Accessed: 02/12/2011 11:07
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.
University of California Press and American Institute of Biological Sciences are collaborating with JSTOR to
digitize, preserve and extend access to BioScience.
http://www.jstor.org
Education
Bringing
in
the
up scientists
critiquing research
art
of
n additionto factualknowledge the structurefor presentingit have Developing skills in
of a given discipline,scientifically variations,but the basic analogy reresearch
need an- mains. Research is conducted and critiquing
literate
collegegraduates
alyticalskills to interpret,apply, and
communicatethe scientific information they have acquired(AAAS1990,
NAS 1989). For research scientists,
analyticalskills are essentialin writing, critiquing,revising,and defending research proposals and articles
and reviewingthe researchof other
scientists.Criticalthinkingand writing are activitiesintegral,ratherthan
peripheral,to scientificresearch.As
Sidney Perkowitz (1989) of Emory
Universitywrites,"Ihavelearnedthat
when I write a researchpaperI do far
more than summarizeconclusionsalready neatly stored in my mind.
Rather,the writingprocessis whereI
carry out the final comprehension,
analysis,and synthesisof my results'
(p. 353).
Butgraduatestudentsrarelyreceive
formaltrainingin thinkingor writing
about research.Many become good
scientists who are nonetheless severely handicappedin communicating theirown researchand in eliciting
useful assessmentsof it from others.
With a good analyticalmind and a
few other tools at hand, however, a
scientistat any careerstage can learn
the art of critiquingresearch.
Criticalassessmentof
researcharticles
Traditionally,the scientific method
involvesformulatinga hypothesis,designingan experimentto test the hypothesis, collecting data, and interpreting the data. The structure of
researcharticles(calledIMRAD)parallels this sequence:introduction,includingstatementof objective;methods; results; and discussion. The
model for conducting research and
by BarbaraJ. Kuyper
248
presented by the scientific method,
and it can also be analyzedby using
the same logical sequenceof steps.
Critical assessment of a research
articleappropriatelyoccursat several
stages. The author critiquesthe first
draft and revises it accordingly.
Friendlycolleaguesreviewthe revised
draft, and the author revises the
manuscriptagain in the light of their
suggestions.Thesepresubmissioncritiques and revisions are intended to
improve the written presentationof
research,short-circuitunfavorablereviews, and decreasetime to publication. On submission,the article undergoes peer review to determine
acceptabilityfor publication. When
an article enters the scientificliterature, it becomesopen to scrutinyby
other scientists,as well as by journalists, politicians,and the generalpublic, and at this stage a scientist'sreputation can be firmly establishedor
irrevocablydamaged.
The value of being able to selfcritiquemanuscriptsand to have confidencein the critiquecannotbe overemphasized.A scientist should ask,
"What was my bias in carryingout
proceduresor in collectingdata?Did I
wantmy resultsto happen?"Scientists
are human and thus subjective,and
awarenessof one's own subjectivityis
essential in preparing objective researchresultsfor presentationto the
scientificcommunity(Harper1990).
Forthe samereason,scientistsneed
to learnhow to elicit useful critiques
from colleagues. "Is my bias showing? Can you tell what I'm most
afraidof? Can you detect any weaknesses in my experimentaldesign or
methodologythat an incisive reader
will most certainly expose if you
don't?As a friendlycolleague,I'd like
you to tell me beforea journalisttells
the world!"
Some tools are needed for training
scientists to critique their own and
their colleagues'researcharticles.An
analyticalmind-setis basic to all facets of scientific research, including
criticalanalysisof the scientificliterature. In editing manuscriptsfor research scientists, I preparea written
summarythat assessesthe articlesection by section.This editorialcritique
is designed to give the author an
overview of the manuscript rather
than gettingboggeddown in editorial
clean-up work or a sentence-bysentenceanalysis.A colleague'swritten critique also provides an overview, but it emphasizesdesign and
interpretationof researchratherthan
presentation.The checklist, a traditional editors' tool, is also useful in
scrutinizing scientific manuscripts
from authors', statisticians',and reviewers' standpoints (Applewhite
1979, CBE Style Manual Committee
1983, Gardner et al. 1986, Squires
1990).
I have developed a checklist for
critiquing a research article at an
earlydraftstage that both the author
and in-house reviewerscan use (see
box page 249). The checklistfocuses
on structure,or organization,and its
interrelationshipwith content. It is
based on the IMRAD structurebut
can be modified for other types of
journal articles. In assessingarticles
with the aid of the checklist,fluorescent color markers are useful tools
that give authorsand reviewerssomething useful (andplayful)to do. I use
a yellow markerto call attentionto
statements of objectives at various
pointsin the manuscript(anddiscrepancies among them) and a rose
markerto identify undefinedor misused terms.
A critiqueof the introductionalone
(steps 1-4) sometimes unravels the
BioScienceVol. 41 No. 4
Checklistfor critiquinga
researcharticle
Title
Author
Introduction
1. Read the statementof purpose at the end of the introduction.
What was the objectiveof the study?
_ 2. Considerthe title. Does it preciselystatethe subjectof the paper?
3. Readthe statementof purposein the abstract.Does it matchthat
in the introduction?
_ 4. Check the sequenceof statementsin the introduction.Does all
informationlead directlyto the purposeof the study?
Methods
_ 5. Review all methodsin relationto the objectiveof the study. Are
the methodsvalid for studyingthis problem?
_ 6. Checkthe methodsfor essentialinformation.Couldthe study be
duplicatedfrom the informationgiven?
7. Review the methods for possible fatal flaws. Is the sample
selectionadequate?Is the experimentaldesign appropriate?
8. Check the sequence of statements in the methods. Does all
informationbelong in the methods?Can the methodsbe subdivided for greaterclarity?
Results
9. Scrutinizethe data, as presentedin tables and illustrations.Does
the title or legend accurately describe content? Are column
headingsand labels accurate?Are the data organizedfor ready
comparisonand interpretation?
_ 10. Reviewthe resultsas presentedin the text while referringto data
in the tablesand illustrations.Does the text complement,and not
simply repeat, data? Are there discrepanciesin results between
text and tables?
_ 11. Check all calculationsand presentationof data.
12. Review the resultsin the light of the stated objective.Does the
study revealwhat the researcherintended?
Discussion
13. Checkthe interpretationagainstthe results.Does the discussion
merelyrepeatthe results?Does the interpretationarise logically
from the data, or is it too far-fetched?Have shortcomingsof the
researchbeen addressed?
_ 14. Comparethe interpretationto relatedstudiescited in the article.
Is the interpretationat odds or in line with other researchers'
thinking?
15. Consider the published research on this topic. Have all key
studiesbeen considered?
16. Reflect on directions for future research.Has the author suggested furtherwork?
Overview
17. Considerthe journal for which the article is intended.Are the
topic and format appropriatefor that journal?
18. Rereadthe abstract.Does it accuratelysummarizethe article?
19. Check the structure of the article (first headings and then
paragraphing).Is all materialorganizedunder the appropriate
heading?Are sections subdividedlogically into subsectionsor
paragraphs?
20. Reflect on the author's thinking and writing style. Does the
authorpresentthis researchlogically and clearly?
entire article. Discrepanciesbetween
the title of the article and the stated
objectiveat the end of the introduction throb in the fluorescentcolor.
The researchermay discover an ambiguityin thinkingabout the purpose
of the researchthat was previously
concealedbut is now glaringlyobvious.
A carefulscrutinyof researchmethods (steps 5-8) may expose fatal
flaws in sample selection or experimental design that invalidatethe results. This disturbingrevelationcan
be beneficialover the long run, however, if it helps the scientist to cut
losses and move on to better-defined
research. A review of methods on
completionof a researchprojectcan
also emphasize the importance of
choosing an appropriateexperimental design at the onset and evaluating
the researchprojectas it develops.
The results, particularly as presented in tables and illustrations,almost inevitablyrequiredrastic redesign and revision.Selecting,aligning,
and labelingdata appropriatelyin tables requireas much thought as does
the textual descriptionof results.Ideally, the author has designedthe tables beforewritingthe resultssection,
and steps 9-12 on the checklist directs reviewersto examine the tables
first. A table should be self-explanatory, with a title that accuratelyand
concisely describes content and column headings that accurately describe information in the cells. Instructions for preparing scientific
tables (CBEStyleManual Committee
1983) and illustrations (CBE Scientific IllustrationCommittee1988) are
invaluabletools in writing and revising researcharticles.
Authors often seem mentally fatigued by the time they have defined
in writing what their researchwas
reallyabout, struggledwith statistical
analysisof data, sortedout meaningful results, and revised tables again
and again. Consequently,the discussion often degeneratesinto a feeble
rewordingof resultsratherthan interpretationof the researchand its status
in relationto otherstudiesin the field.
In critiquing the discussion section
(steps 13-16), the author can easily
detect mere repetitionof results. To
validate and refine interpretation,
however,a colleague'sprobingquestions are probably more fruitful at
April 1991
249
this stage than is self-examination.
The overview section of the checklist (steps 17-20) requires the author
or reviewer to step back and reconsider the manuscript as a whole. Does
the author think and write logically? Is
the organizational sequence of the paper logical and appropriate to content? Are the objectives and results of
the research stated clearly? Does the
article fit the stated purpose of the
journal to which it is being submitted?
Conclusions
After all is said and done, critiquing
research is intellectual fun. The ability
to scrutinize a piece of writing with a
critical eye requires time for leisurely
contemplation, an analytical mind
(the scientific mind?), a zest for arguing with colleagues, and the ability to
set ego aside. If we do not assess our
own research, journal reviewers and
subsequent readers will do it for us,
with the potential for much more
badly bruised egos and scientific reputations.
Acknowledgments
I thank Stephen B. Kritchevsky, Department of Biostatistics and Epidemiology, University of Tennessee,
Memphis, and Jerry M. Williams, Department of Horticulture, Virginia
Polytechnic Institute and State University, for critiquing this manuscript.
References cited
AmericanAssociationfor the Advancementof
Science (AAAS). 1990. The LiberalArt of
Science:Agendafor Action. AAAS, Washington,DC.
Applewhite,L. 1979. Examinationof the medical/scientificmanuscript.Journalof Technical Writingand Communication9: 17-25.
CBE ScientificIllustrationCommittee. 1988.
IllustratingScience:Standardsfor Publication. Council of Biology Editors,Bethesda,
MD.
CBE Style Manual Committee. 1983. CBE
StyleManual:A Guidefor Authors,Editors,
and Publishersin the BiologicalSciences.5th
edition. Council of Biology Editors, Bethesda,MD.
Gardner,M. J., D. Machin,and M. J. Campbell. 1986. Use of checklists in assessingthe
statistical content of medical studies. Br.
Med.J. 292: 810-812.
Harper, A. E. 1990. Criticalevaluation-the
only reliableroad to knowledge.BioScience
40: 46-47.
NationalAcademyof Sciences(NAS),Committee on the Conduct of Science. 1989. On
Being a Scientist.National AcademyPress,
Washington,DC.
Perkowitz,S. 1989. Commentary:can scientists learn to write? Journal of Technical
Writingand Communication19: 353-356.
Squires, B. P. 1990. Statistics in biomedical
manuscripts:what editors want from authors and peer reviewers.Can. Med. Assoc.
J. 142: 213-214.
WordPerfect~itation
Everyacademicwriterhas spent
valuabletime translatingthe
section of a style manual
References
into bibliographiccitations.
It is an academicproblem.We
have a "perfect"solution.
Barbara J. Kuyper is an assistant professor in the Department of Health Informatics, University of Tennessee, Memphis, TN 38163. She is responsible for
developing the scientific writing component of a curriculum for graduate students
planned to include training in information
science, analytical skills, scientific communication, and the roles and responsibilities of scientists in the world community.
She teaches a graduate course on writing
journal articles and a faculty workshop on
critiquing research articles. ? 1991 American Institute of Biological Sciences.
With""'Citation,
you can
generatebibliographiesin 100+
differentpublishingstyles, without
everleavingWordPerfect
or
learninganotherprogram.
is perfectlysimpleto
""'Citation
use andveryefficient.Author
names,titles,pages, etc., are
entered - and proofread-
only
once. Sinceyour""Citation
datafileis a WordPerfectdocument,you use WordPerfectkeys
andcharactersets to enter
informationin the datafile.
WithwPCitation,
generating
bibliographiesin different
publishingstyles is easy. And at
the introductorypriceof $79,
"'Citationwill save you more than
time.
""'Citation
requiresWordPerfect5.1or
higher. WordPerfect is a trademark of
WordPerfect Corporation. ""Citationis a
trademark of Oberon Resources.
For more information,call or write.
147 E Oaklanc Avenue
Columbus. Ohio 43201
1 * 800 * 594 * 9062
250
BioScience Vol. 41 No. 4