Writing, Critical Thinking, and Quantitative Reasoning in PSY 10200 (Fall 2010)
Introduction
An introductory course in psychology confronts college freshmen with a large set of
subfields, each containing a bewildering array of new terms. During the course of the
semester, students are expected to master, think critically, and write coherently about the
concepts, principles, and methodologies used these subfields, which include
neuroscience, child development, perception, clinical psychology, social psychology, and
health psychology. Each subfield uses a distinct empirical approach to tackle
circumscribed theoretical issues. Nonetheless, students must learn to employ a common
methodological toolbox of research design, in which independent variables are
manipulated to measure the effects on designated dependent variables, while controlling
for the confounding influences of nuisance variables.
To gauge how well students in PSY 10200 are able to analyze theoretical problems,
design research protocols, and interpret empirical results, each semester they are asked to
complete three short papers (4-5 pages) targeting a specific topic currently under
discussion in lecture. For example, when considering the role of pseudoscience in the
context of extrasensory perception students in Fall 2010 were asked to write a research
proposal:
Dr. Peter Venkman is interested in testing whether psychics who claim to have clairvoyant
powers actually do. Dr. Venkman believes that real psychics have special clairvoyant
powers that non-psychics do not possess. He wishes to design an experiment, testing his
subjects with a deck of Zener cards. The deck contains 25 cards, each having one of 5
geometric figures, as shown above (5 cards of each geometric figure). You have been hired
as a consultant to submit a research proposal to Dr. Venkman, based on his suggestions,
which you are free to accept or reject. Dr. Venkman wants to compare the ability of psychics
at predicting the cards in the deck to the ability of non-psychics. But he’s not sure how to
select the two groups. He wants an experimenter to shuffle the cards and place them face
down one at a time in front of the subject, who then will name the figure underneath. But
he’s afraid that shuffling may be done poorly and that some subjects may count cards or else
use other card tricks to guess the identity of the card. He’s also not sure whether the
experimenter should know the identity of the subject being tested. Finally, Dr. Venkman
realizes that even non-psychics guess some cards correctly by chance alone, especially if
only a few cards are tested. He wants you to determine how to control for guessing,
experimenter knowledge, card tricks, and poor shuffling.
A set of specific questions guides the student in crafting the paper. In the example
described above, students were led through four different sections of the research
proposal: Introduction, Methods, Results, and Conclusions. Each successive paper
provides students with progressively less formal structure. The aim of this approach is to
scaffold the demands we place on students to defend their claims and interpretations.
Thus, the ultimate learning objectives of the three term papers are fivefold:
a.
b.
c.
d.
e.
Effective written communication
Critical thinking and logical reasoning ability
Ability to formulate questions, hypotheses, and research designs
Proper use of psychological concepts and theories
Competence in quantitative reasoning and analysis of research findings
To assess how well each student meets each of these five outcomes, teaching assistants
complete a 4-point rubric for each of the papers of the students in his or her recitation
section. Thus, each semester, over 2000 rubrics are completed, which are subsequently
returned to the students to provide feedback in meeting the learning outcomes. Some of
the dimensions of the rubric are specific to the particular paper topic. However, each
rubric also includes the five general outcome dimensions listed above. Here is an
example of the rubric categories used for three of the dimensions:
Learning
Outcome
Beginning (1)
Developing (2)
-Progression of thought within -Progression of thought within
A. Effective
and between paragraphs is
paragraphs is clear, but
written
unclear
progression of thought between
communication
B. Critical
thinking
and logical
reasoning
ability
C. Ability to
formulate
questions,
hypotheses
and research
design
Competent (3)
Accomplished (4)
-Progression of thought both
within and between paragraphs is
clear (structure/organization)
-Has a clear organizing
strategy that provides clarity
throughout the assignment
(structure/organization)
(structure/organization)
paragraphs is unclear
(structure/organization)
-The writing is unfocused and
unclear at the sentence level
(mechanics and style)
-The writing is inconsistently
focused and clear (mechanics
and style)
-Most of the writing is focused
and clear (mechanics and style)
-The writing is focused and
clear throughout (mechanics
and style)
-There are major problems in
grammar, punctuation, and
usage that undermine the
communication of ideas
(mechanics and style)
-There are some major
problems in grammar,
punctuation, and usage, but
they do not undermine the
communication of ideas
(mechanics and style)
-There are only minor problems in
grammar, punctuation, and usage
(mechanics and style)
-The writer uses sophisticated
means to express ideas
(mechanics and style)
- Uses fallacious or irrelevant
reasons, and unwarranted
claims. Misinterprets
evidence, statements,
graphics, questions, etc.
(supports conclusions)
- Language is not always
accurate. Does not attempt, or
is unsuccessful, in supporting
conclusions with findings,
theories and facts. (supports
conclusions)
- Uses accurate language.
Generally presents conclusions
that can be supported by findings,
theories, or facts. (supports
conclusions)
- Uses accurate language,
inference, reference,
articulation, and/or selection of
most salient issues. Does not
present conclusions that cannot
be supported by findings,
theories, or facts. (supports
conclusions)
- Does not break down
problems, issues, or questions
into parts. Misinterprets (or
does not interpret) evidence,
statements, graphics,
questions, etc. (analyzing
problems)
- Exhibits no/limited
understanding of how to state
a hypothesis
- Analyzes mainly through
reiteration, chronology, or
form. Describes events,
people, and places with some
supporting details from the
source. (analyzing problems)
- Breaks down problems, issues,
or questions into generally
meaningful parts or segments.
Offers analysis of obvious
alternative points of view.
(analyzing problems)
- Demonstrates some
understanding of a hypothesis,
but is unable to identify the
correct hypothesis to be
evaluated
- Demonstrates understanding of a
hypothesis but somewhat limited
ability to correctly identify
hypothesis to be evaluated
- Demonstrates proficient
understanding of a hypothesis
and correctly identifies
hypothesis to be evaluated
- Exhibits no/limited
understanding of appropriate
research design
- Exhibits some understanding
of appropriate research design
- Exhibits adequate understanding
of appropriate research design, but
does not consider potential flaws
and limitations
Exhibits sophisticated
understanding of appropriate
research design, including
potential flaws and limitations
- Breaks down problems,
issues, or questions into
significant parts or segments.
Thoughtfully analyzes major
alternative points of view.
(analyzing problems)
Results
An analysis was conducted of the three term papers given to students enrolled in PSY
10200 in Fall 2010. The sample included a large subset of the 22 sections of the course.
For each of the five learning outcomes (written communication, critical thinking,
formulate hypotheses, proper use of concepts, and quantitative reasoning), we calculated
for each of the three papers the average rating and its associated standard error (i.e., the
standard deviation of the sampling distribution). These appear graphically below. The
results indicate that, on average, students performed at a competent level on each
dimension (i.e., average of 3.14 on the 4-point scale). Students scored slightly higher in
written communication (3.25) and proper use of concepts (3.26) than on the other three
dimensions. The finding seems to be due primarily to relatively high scores on these two
dimensions students received on the second term paper, which explored a topic in
neuroscience. The relatively small standard errors (depicted by T-bars in the figure)
indicate that ratings were relatively consistent across students and across raters.
PSY 10200 - Analysis of Fall 2010 Rubrics
Paper 1
Paper 2
Paper 3
4
Average Rating
3.5
3
2.5
2
1.5
1
0.5
0
Writt en co m m un ic a tio n
Critic a l think ing
F o rm u la te hypo th e s es
P r oper us e o f c o nc epts
Quan tita tive r ea s o ning
Learning Outcome
Conclusions
The results of the current analysis suggest that students in PSY 10200 are able to
effectively integrate a set of five distinct abilities when developing ideas in the different
subfields of psychology (here, child development, neuroscience, and perception). In each
case, students were able to critique a flawed theoretical idea, propose a research design
that isolated one variable among many, and provide an interpretation of hypothetical data,
while writing effective prose and using psychological terms and concepts appropriately.
Nevertheless, there was little evidence that students improved in any of the five learning
outcomes across the semester. One reason may be the scaffolded design of the three
papers, which included progressively less structure from one paper to the next.
One possible limitation of the conclusions concerns the validity of the ratings themselves:
Although the raters selected categories on the basis of specific criteria included in each
rubric, one cannot be certain whether they interpreted category labels (e.g.,
“accomplished,” “competent,” etc.) in terms of students entering (freshmen) or leaving
(senior) their college careers. Thus, the meaning of “competent” is ambiguous. Ideally,
our results could be compared directly with ratings of the same students as they progress
through the General Education curriculum, providing an indication of the possible
improvements in learning outcomes the college experience brings to students. It is
conceivable, though, that the raters would use the scales quite differently if they were told
the papers being judged were the product of college seniors rather than college freshmen.
Although the ability to integrate among these five learning outcomes is critical to success
in introductory psychology, future assessments may seek to focus on an analysis of more
restricted learning outcomes. One possibility is to consider each term paper only in terms
of the development and structure of the written responses:
a. Critical Response to Text and Writing Task
b. Development of Writer's Ideas
c. Structure of the Response
d. Language Use: Sentences and Word Choice
e. Language Use; Grammar, Usage, Mechanics
A rubric for this purpose is currently under development for use in the 2011-2012
academic year:
Critical
Response to
Text and
Writing Task
(5 pts)
* The response demonstrates
a clear understanding of the
complexity of ideas in the text
and of the writing task. A
thoughtful and insightful
response to the task
effectively integrates analysis
of main ideas in the text and
relevant elements of the
writer's own reading and
experience.
* The response usually
demonstrates a good understanding
of the complexity of ideas in the text
and of the writing task. The
response usually integrates
analysis of main ideas in the text
and relevant elements of the writer's
own reading and experience/
* The response demonstrates an
adequate understanding of the
main ideas in the text and of the
writing task. The response
adequately integrates analysis of
main ideas in the text and relevant
elements of the writer's own
reading and experience.
* The response demonstrates
some understanding of the
main ideas in the text and of
the writing task. The response
includes some integration of
main ideas from the text and
relevant elements of the writer's
own reading and experience.
* The response
demonstrates a weak
understanding of the main
ideas in the text of the
writing task. There is little
integration of ideas from the
text and elements of the
writer's own reading and
experience.
* Approaches to development
Development (e.g., summarizing,
of Writer's evaluating, narrating) are used
Ideas (5 pts) skillfully to support clarity.
* The response maintains an
effective balance between
broad assertions and specific
reasons, details and
examples.
* There is some development of
ideas, but the approaches
used may not always support
* Approaches to development (e.g., * Approaches to development (e.g., clarity.
summarizing, evaluating, narrating) summarizing, evaluating, narrating) * The response may include
are usually used skillfully to support are adequate, but they may not
both broad assertions and
clarity.
always support clarity.
specific examples, but the
* The response usually
* The response maintains an
balance may be uneven and
demonstrates a balance between
adequate balance between broad examples may not be
broad assertions and specific
assertions and specific details and developed enough to support
reasons, details and examples.
examples.
clarity.
* There is little, if any, us of
releveant approaches to
development.
* A balance between broad
assertions and specific
details and examples may
not be evident. If details and
examples are present, they
are brief, general,
inadequately developed, or
not clearly relevant.
* A cohesive structure that
demonstrates a logical, wellStructure of executed progression of ideas
the Response that supports a central focus
(5 pts)
for the response.
* Sophisticated and effective
use of transitions conveys
relationships among ideas
throughout the response.
* The response uses a simple
* An organization structure is
organizational structure that
* Organization is clear and
evident and supports a central
usually supports a central
demonstrates some logical
focus. Ideas are presented in
focus for the response. For the
progression of ideas that supports a logical groupings. There may be
most part, ideas are logically
central focus for the response.
an attempt at logical progression. grouped.
* Transitions clearly convey
* Though often simple and obvious, * Simple and obvious
relationships among ideas
transitions are usually used to
transitions are used to convey
throughout the response.
convey relationships among ideas. relationships among ideas.
* The response shows an
attempt to create a central
focus for the response and
to organize ideas into logical
groupings, but the logic of
these groupings is often
unclear.
* Few, if any, transitions are
used to convey relationships
among ideas.
* Sentences are sometimes
correct but there is little, if
any, sentence variety.
* Word choice is often
unclear and at times
obscures meaning.
* The response
* A basic control of language is demonstrates weak
Language
* Though there may be few
apparent. Grammar, usage
language control. Grammar,
Use;
errors in grammar, usage and * Though there may be a few
* Language use is competent.
and mechanics are usually
usage and mechanics are
Grammar,
mechanics, strong control of distracting errors in grammar, usage Grammar, usuage, and mechanics correct but there are some
sometimes correct but there
Usage,
language is apparent and
and mechanics, good control of
are mostly correct and there may distracting errors. Errors may are many distracting errors
Mechanics (5
meaning is clear throughout
language is apparent and meaning be some distracting errors, but
occasionally impede
and some that impede
pts)
the response.
is clear
meaning is clear.
understanding.
understanding.
Language
Use:
Sentences
and Word
Choice (5 pts)
* Sentences are well
controlled with effective variety
in structure.
* Word
choice is specific, precise,
and supports clarity.
* Sentences are usually well
controlled and their structure is
somewhat varied.
* Word choice is clear and
adequately supports clarity.
* Sentences are adequately
controlled and there is a little
structural variety.
* Word choice is usually clear and
adequately supports clarity.
* Sentences are somewhat
controlled but with little
structural variety.
* Word choice is sometimes
unclear and may obscure
meaning.