Jean
Nicod
Lectures
2007
Moral Theory Meets Cognitive
Science
How the Cognitive Sciences Can
Transform Traditional Debates
Stephen Stich
Dept. of Philosophy
& Center for Cognitive Science
Rutgers University
sstich@ruccs.rutgers.edu
1
Jean
Nicod
Lectures
2007
Lecture 4
Stephen Stich
Daniel Kelly
Joshua Knobe
Debunking Moral Intuition
A Hodgepodge of Multipurpose Kludges
2
Jean
Nicod
Lectures
2007
Lecture 4
Stephen Stich
Joshua Knobe
Daniel Kelly
Debunking Moral Intuition
A Hodgepodge of Multipurpose Kludges
3
Introduction
 Philosophers – and more recently cognitive
scientists – have offered many accounts of the
psychological mechanisms & processes
underlying intuitive moral judgment
 Moral philosophers have always insisted that
sometimes the outputs of those processes –
people’s “moral intuitions” – are not to be
trusted

though they disagree about when skepticism is
warranted
4
Introduction
 Our goal in this talk is to sketch a newly
emerging perspective on the mechanisms
underlying moral intuition …
 and to explore its implications for the hotly
debated issue of whether and when intuitions
should be relied on
5
Introduction
 Philosophers have typically assumed that those
mechanisms were well designed for …
something
 But we now have reasons to think that many of
theses mechanisms are not well designed
for ANYTHING
6
Introduction
Moral Psychology is a
Kludge
A hodgepodge of multipurpose
kludges!
7
Introduction
 Before explaining and defending this claim it
will be useful to consider some of the reasons
that philosophers – both classic &
contemporary – have offered for discounting
moral intuitions
8
Philosophical Background
 When should we be skeptical about
moral intuitions?
 The “Moral Sense” & “Ideal Observer” traditions
 Reflective Equilibrium
 Evolutionary arguments debunking intuition
9
Philosophical Background
 The “Moral Sense” & “Ideal Observer” traditions

Ideal observer theorists maintain that our moral
intuitions are correct (or justified) when made under
ideal conditions

When conditions are not ideal – e.g. when we have
false beliefs about relevant non-moral matters, or we
are irrational – our intuitions are not to be trusted
10
Philosophical Background
 The “Moral Sense” & “Ideal Observer” traditions

For Hutcheson – an important precursor of this
tradition – moral judgments are the product of a “moral
sense” implanted in us by “the Author of Nature”

Thus it can be relied upon when doing its job
properly

But, like other senses, it can mislead when
conditions are unfavorable
11
Philosophical Background
 Reflective Equilibrium

Rawls’ “Decision Procedure for Ethics”
(1951)

Narrow Reflective Equilibrium

Bring intuitions about
 particular cases
 moral principles
into accord

To do this, sometimes an intuition about a particular
case must be rejected
12
Philosophical Background

Wide Reflective Equilibrium

Bring intuitions about
 particular cases
 moral principles
into accord with the rest of our beliefs
 including beliefs about scientific matters, history,
politics – even metaphysics & semantics

Even more of our intuitions about particular cases
will have to be rejected
13
Philosophical Background

Evolutionary arguments debunking intuition

Perhaps the most influential writer in this
tradition is Peter Singer
The
Expanding
Circle
Ethics and Sociobiology
Peter Singer
FARRAR, STRAUS & GIROUX
New York
1981
Updated in “Ethics & Intuition (2005)
Philosophical Background

In The Expanding Circle, Singer focuses on
nepotistic intuitions which maintain that, in various
domains, we ought to value the welfare of our kin and
tribesmen more than the welfare of people outside
these circles

The psychological processes leading to judgments of
this sort were adaptive in ancestral environments
(and perhaps they still are)

But once we see why we have these nepotistic &
tribal intuitions, Singer suggests, we can also see
that there is no good reason to use them in a
“decision procedure for ethics”
15
Philosophical Background

In “Ethics and Intuition” (2005) Singer develops the
argument by focusing on the sort of “trolley problems”
that have loomed large in recent philosophical and
empirical studies
16
Philosophical Background

Singer (following Greene) maintains that the
neuroscientific evidence suggests that
intuitions about the “footbridge” case are the
result of our emotional reaction to cases in
which harm is caused by the sort of
interaction that would have occurred in
ancestral environments
17
Philosophical Background
“The salient feature that explains our different intuitive judgments
concerning the two cases is that the footbridge case is the kind
of situation that was likely to arise during the eons of time over
which we were evolving; whereas the standard trolley case
describes a way of bringing about someone’s death that has
only been possible in the past century or two…. But what is
the moral salience of the fact that I have killed someone in
a way that was possible a million years ago, rather than in
a way that became possible only two hundred years ago?
I would answer: none….
18
Philosophical Background
“At [a] more general level …this … casts serious doubt on the
method of reflective equilibrium. There is little point in
constructing a moral theory designed to match considered
moral judgments that themselves stem from our evolved
responses to the situations in which we and our ancestors
lived during the period of our evolution as social mammals,
primates, and finally, human beings. We should, with our
current powers of reasoning and our rapidly changing
circumstances, be able to do better than that.” (348)”
What I am saying, in brief, is this. Advances in our understanding
of ethics … undermine some conceptions of doing ethics
…. Those conceptions of ethics tend to be too respectful
of our intuitions. Our better understanding of ethics gives
us grounds for being less respectful of them.” (349)
19
Philosophical Background
 We agree with Singer’s skepticism about intuition
 But we also think his skepticism is
not radical enough!
20
Philosophical Background
 Assumptions that Singer and the friends of
intuition share:

The psychological system underlying our moral
intuitions is well designed

Thus there is some point to – or reason for – the
intuitive moral judgments people make when the
system is working properly

Though Singer (unlike the friends of intuition) insists that
the function the system is designed for is of dubious
moral importance, and thus that the intuitions are not to
be taken seriously
21
Philosophical Background
 We believe that the engine of moral intuition is not
well designed at all
 Far from being the sort of “elegant machine”
celebrated in the writings of some evolutionary
psychologists, we think that it is a kludge

a cluster of mechanisms cobbled together rather
awkwardly from bits of mental machinery most of
which were designed for functions that have
noting to do with morality
22
Philosophical Background
 To use a term that may be more common in Paris,
we maintain that the engine of moral intuition is the
result of bricolage
François Jacob
Claude Lévi-Strauss
23
Philosophical Background
 This explains many of the quirks of moral
intuition …
 And provides yet another reason to be
skeptical of their use in moral deliberation
24
Overview of the Rest of the Talk
25
Overview of the Rest of the Talk
 Two examples of the “kludginess” of the
mechanisms underlying moral intuition


Dan Kelly’s work on Moral Disgust
Joshua Knobe’s work on intentionality
judgments & unconscious moral
judgments
 From kludginess to skepticism
26
Kelly on Disgust
 Kelly has constructed a rich,
nuanced, empirically supported
account of the psychological
mechanisms underlying the
uniquely human disgust
system and how that system
evolved
Daniel Kelly
 In this talk I’ll only have time to for a
brief sketch of two central themes
27
Kelly on Disgust
 The Entanglement Thesis

Disgust is itself a kludge – a uniquely
human emotion produced by the merger of
two distinct systems
 The Co-Optation Thesis

After the merger, disgust was co-opted by


the norm system
the ethnic boundary system
which were central elements in the
emergence of human ultra-sociality
28
Kelly on Disgust
 Kelly assembles a vast array of evidence for
these theses, drawn from






neuroscience
social psychology
cognitive psychology
developmental psychology
evolutionary psychology
gene-culture co-evolution theory
 As usual, the devil is in the details

So I join Paul Rozin in urging that you read the
work as it appears in print
29
Kelly on Disgust
The Entanglement Thesis
 Disgust exhibits a puzzling array of
elicitors
which evoke an equally puzzling cluster of
responses
30
Kelly on Disgust
The Entanglement Thesis
 Elicitors include

Foods: dog meat, grubs, insects
31
Kelly on Disgust
The Entanglement Thesis
 Elicitors include







Foods: dog meat, grubs, insects
Substances associated with the body: feces, vomit,
spit
Organic decay
People and objects associated with illness: a shirt
once worn by a person with leprosy
Sexual practices: necrophilia, incest
Some moral transgressions & transgressors: rape,
torture, child molestation
Members of low status outgroups: untouchables,
Jews
32
Kelly on Disgust
Some elicitors
are
pan-cultural
The Entanglement Thesis
 Elicitors include







Foods: dog meat, grubs, insects
Substances associated with the body: feces, vomit,
spit
Organic decay
People and objects associated with illness: a shirt
once worn by a person with leprosy
Sexual practices: necrophilia, incest
Some moral transgressions & transgressors: rape,
torture, child molestation
Members of low status outgroups: untouchables,
Jews
33
Kelly
Disgust
Others
are on
culturally
local
The
(orEntanglement
idiosyncratic)Thesis
 Elicitors include







Foods: dog meat, grubs, insects
Substances associated with the body: feces, vomit,
spit
Organic decay
People and objects associated with illness: a shirt
once worn by a person with leprosy
Sexual practices: necrophilia, incest
Some moral transgressions & transgressors: rape,
torture, child molestation
Members of low status outgroups: untouchables,
Jews
34
Kelly on Disgust
The Entanglement Thesis
 The disgust response includes
 Gape face (occasionally accompanied by retching)
 Feeling of nausea
 Sense oral incorporation
 Quick withdrawal
 A more sustained & cognitive sense of
offensiveness
 A more sustained & cognitive sense of
contamination
35
Kelly on Disgust
The Entanglement Thesis
 How are all of these connected?
 The Entanglement Thesis maintains that the
human emotion of disgust is the result of the
fusion of two distinct mechanisms

each of which has homologous counterparts in
other species

though they have combined only in humans
36
Kelly on Disgust
The Entanglement Thesis
 One mechanism (“the poison avoidance
mechanism”) is directly linked to digestion



It evolved to regulate food intake and protect the gut
against ingested substances that are poisonous or
otherwise harmful
It was designed to expel substances entering the
gastro-intestinal system via the mouth
And to acquire new elicitors very quickly
 As John Garcia famously demonstrated, ingested
substances that induce gut-based distress often
generate acquired aversions
37
Kelly on Disgust
The Entanglement Thesis
 The other mechanism (“the parasite avoidance
mechanism”)



Evolved to protect against infection from pathogens
and parasites, by avoiding them
Not specific to ingestion, but serves to guard against
coming into close physical proximity with infectious
agents
This involves avoiding not only visible pathogens and
parasites, but also places, substances and other
organisms that might be harboring them
38
These elements
of the on
disgust
response are
Kelly
Disgust
traceable to the
poison
avoidance
system
The
Entanglement
Thesis
 The disgust response includes
 Gape face (occasionally accompanied by retching)
 Feeling of nausea
 Sense oral incorporation
 Quick withdrawal
 A more sustained & cognitive sense of
offensiveness
 A more sustained & cognitive sense of
contamination
39
and these
are on
traceable
to
Kelly
Disgust
the parasite The
avoidance
poison
system
Entanglement
Thesis
 The disgust response includes
 Gape face (occasionally accompanied by retching)
 Feeling of nausea
 Sense oral incorporation
 Quick withdrawal
 A more sustained & cognitive sense of
offensiveness
 A more sustained & cognitive sense of
contamination
40
These elicitors
to
Kellyare
ontraceable
Disgust
the poison
system
Theavoidance
Entanglement
Thesis
 Elicitors include







Foods: dog meat, grubs, insects
Substances associated with the body: feces, vomit,
spit
Organic decay
People and objects associated with illness: a shirt
once worn by a person with leprosy
Sexual practices: necrophilia, incest
Some moral transgressions & transgressors: rape,
torture, child molestation
Members of low status outgroups: untouchables,
Jews
41
and these
are on
traceable
to
Kelly
Disgust
the parasite
system
The avoidance
Entanglement
Thesis
 Elicitors include







Foods: dog meat, grubs, insects
Substances associated with the body: feces, vomit,
spit
Organic decay
People and objects associated with illness: a shirt
once worn by a person with leprosy
Sexual practices: necrophilia, incest
Some moral transgressions & transgressors: rape,
torture, child molestation
Members of low status outgroups: untouchables,
Jews
42
Kelly on Disgust
The Entanglement Thesis
 One bit of evidence supporting the Entanglement
Thesis is that different components of that response
are on different developmental schedules


Distaste & gape are present within the first year of life
Contamination sensitivity emerges significantly later
 Once the full system in in place, the components of
the response are produced together – they form a
nomological cluster

Any elicitor of disgust will reliably produce all or most of
those clustered components
43
Kelly on Disgust
The Entanglement Thesis
 A puzzle:
 Why should the sight of a festering sore or a person
with leprosy evoke a gape face and a feeling of
nausea?
 The solution: Disgust is a kludge!
 But it is kludge with features that could be readily co-
opted and put to other uses as humans began living
in larger groups and human ultrasociality emerged
44
Kelly on Disgust
The Co-Optation Thesis
45
Kelly on Disgust
The Co-Optation Thesis
 The Gape Face as a Signal

As group size increased, there was an increasing need
for a perspicuous signal warning of dangerous
foods and risk of infectious disease

In humans, the face and facial expressions provide a
rich source of such social information

The gape face, which clearly has roots in the facial
motions that accompany retching, was co-opted as a
signal, warning others not just against toxic foods, but
also against the presence of parasites and
contagious pathogens
46
Kelly on Disgust
The Co-Optation Thesis
 Co-Optation by the Norm System

As group size increased, there was increased need for
complex social coordination

The norm system – whose structure we considered
briefly in the 2nd Lecture – played an important role in
facilitating this co-ordination

And the disgust system had features that made it an
obvious candidate to be co-opted by the norm system
as it evolved
47
Kelly on Disgust
The Co-Optation Thesis

The S&S model suggests that compliance motivation &
punitive motivation are linked to “the emotion system”
48
Kelly on Disgust
The Co-Optation Thesis
infer contents of
normative rules
identify norm
implicating
behavior
Acquisition
Mechanism
Proximal
Cues in
Environment
Execution Mechanism
norm data base
r1---------r2---------r3---------……
rn----------
Rule-related
reasoning
capacity
compliance
motivation
emotion
system
punitive
motivation
other emotion
triggers
beliefs
judgment
explicit
reasoning
post-hoc
justification
49
Kelly on Disgust
The Co-Optation Thesis

But psychological & neurological evidence indicates
that there are several separate emotion systems –
the disgust system being one of them
50
Kelly on Disgust
The Co-Optation Thesis
infer contents of
normative rules
identify norm
implicating
behavior
Acquisition
Mechanism
Proximal
Cues in
Environment
other emotion
triggers
Execution Mechanism
norm data base
r1---------r2---------r3---------……
rn---------Rule-related
reasoning
capacity
compliance
motivation
beliefs
DISGUST
other
emotions
punitive
motivation
judgment
explicit
reasoning
post-hoc
justification
51
Kelly on Disgust
The Co-Optation Thesis

Disgust is a natural candidate to provide both
compliance & punitive motivation for norms that
involve intrinsically disgusting matters, like the disposal
of corpses & bodily wastes, and other activities that are
antecedently salient to the disgust system, like eating
practices

Compliance is motivated by making norm violating
behavior disgusting & thus aversive

Punitive motivation is provided because the violator
is considered dirty and contaminated and is
avoided or shunned
52
Kelly on Disgust
The Co-Optation Thesis
infer contents of
normative rules
identify norm
implicating
behavior
Acquisition
Mechanism
Proximal
Cues in
Environment
other emotion
triggers
Execution Mechanism
norm data base
r1---------r2---------r3---------……
rn---------Rule-related
reasoning
capacity
compliance
motivation
beliefs
DISGUST
other
emotions
punitive
motivation
judgment
explicit
reasoning
post-hoc
justification
53
Kelly on Disgust
The Co-Optation Thesis
 The norm system is thus a kludge built with kludgy
parts

Not surprisingly, this can lead to some very quirky and
disturbing behavior

Several recent studies have focused on the fact that
the disgust system can be triggered by many things
that have nothing to do with norms

but even when triggered by these non-moral items,
the disgust system can have dramatic and persistent
influence on a person’s judgments about moral
issues
54
Kelly on Disgust
The Co-Optation Thesis
infer contents of
normative rules
identify norm
implicating
behavior
Acquisition
Mechanism
Proximal
Cues in
Environment
other emotion
triggers
Execution Mechanism
norm data base
r1---------r2---------r3---------……
rn----------
Rule-related
reasoning
capacity
compliance
motivation
beliefs
DISGUST
other
emotions
punitive
motivation
judgment
explicit
reasoning
post-hoc
justification
55
Kelly on Disgust
The Co-Optation Thesis
 Wheatley & Haidt have shown that when participants
are hypnotically induced to feel a brief pang of
disgust when they encounter the work “often” and
then presented with the following scenario
“Dan is a student council representative at his school.
This semester he is in charge of scheduling
discussions about academic issues. He often picks
topics that appeal to both professors and students in
order to stimulate discussion.”
many judge that Dan is doing something wrong!
56
Kelly on Disgust
The Co-Optation Thesis
 Schnall et al. have shown participants make more
severe moral judgments when the judgments are
made in a disgusting office:




greasy pizza boxes
sticky chair
a dried up smoothie
a chewed up pen
57
Kelly on Disgust
The Co-Optation Thesis
 Other studies have focused on prima facie irrational
downstream consequences of the disgust system
being triggered in moral deliberation
58
Kelly on Disgust
The Co-Optation Thesis
infer contents of
normative rules
identify norm
implicating
behavior
Acquisition
Mechanism
Proximal
Cues in
Environment
Execution Mechanism
norm data base
r1---------r2---------r3---------……
rn---------Rule-related
reasoning
capacity
Downstream
consequences
other emotion
triggers
compliance
motivation
beliefs
DISGUST
other
emotions
punitive
motivation
judgment
explicit
reasoning
post-hoc
justification
59
Kelly on Disgust
The Co-Optation Thesis
 The Lady Macbeth Effect

Zhong & Liljenquist have shown that recalling an
unethical deed increased the desire for products
related to cleansing, like antiseptic wipes

And that cleaning one’s hands after describing a past
unethical deed reduced moral emotions like guilt &
shame
 and also reduced the likelihood that participants
would volunteer to help a desperate graduate
student!
60
Kelly on Disgust
The Co-Optation Thesis
 The Lady Macbeth Effect

Schnall et al. (unpublished) compared judgments
about moral severity in two groups of participants
 One group had just used an alcohol-based
cleansing gel on their hands
 The other group had just used an ordinary, noncleansing hand cream

The moral judgments of those using the cleansing gel
were significantly less severe!
61
Kelly on Disgust
The Co-Optation Thesis
 Ethnic Boundary Markers
 Boyd & Richerson & their students have argued that
another crucial step in the development of human ultrasociality was the emergence of mechanisms which
allow people to recognize members of their own tribe or
“ethnie”

This is important because in-group members share
beliefs & norms that facilitate coordination
62
Kelly on Disgust
The Co-Optation Thesis

Since different cuisines & eating practices are one of
the more visible correlates of ethnie membership, and
since disgust is heavily involved in regulating food
intake, disgust was a natural candidate to be co-opted
by the emerging system of ethnic identification

Eating practices of out-groups and other readily
detectable signs of out-group membership came to
evoke disgust

And disgust came to provided a significant part of the
motivation to avoid out-group members
63
Kelly on Disgust
The Co-Optation Thesis

Though the evolutionary function of the ethnic
boundary marker system was to facilitate cooperation
by keeping groups apart, the kludgy solution to this
problem has some unfortunate consequences

Out-group members are not simply avoided, they are
also considered offensive & contaminating

People who embrace different norms are often felt to
be disgusting and sub-human!
64
Kludge Meets Kass
65
Kludge Meets Kass
 Leon Kass, M.D., Ph.D.

Conservative bio-ethicist

Chairman of the U. S. A.
President's Council on
Bioethics from 2002 to 2005
66
Kludge Meets Kass
 In his book, Life, Liberty & the Defense of Dignity (2002),
there is a chapter called “The Wisdom of Repugnance”
 Kass maintains that

"in crucial cases...repugnance is the emotional
expression of deep wisdom, beyond reason's power
fully to articulate it.”

“In this age in which everything is held to be permissible
so long as it is freely done, and in which our bodies are
regarded as mere instruments of our autonomous
rational will, repugnance may be the only voice left that
speaks up to defend the core of our humanity. Shallow
are the souls that have forgotten how to shudder."
67
Kludge Meets Kass
 The claims play a central role in Kass’ critique of human
cloning
 Others have adopted the idea to argue against abortion,
pornography & same-sex marriage
68
Kludge Meets Kass
 Some philosophers, most notably
Martha Nussbaum, have
challenged Kass, arguing that
disgust should be discounted in
moral & legal deliberation because
(roughly) it reminds us of our
animal origins
69
Kludge Meets Kass
I think Kelly’s work offers a far more
plausible &
powerful
critique
70
Kludge Meets Kass
 There is no reason to think there is
wisdom in repugnance
because
Disgust is a Kludge
and the psychological system that bases moral judgments
on disgust is a
Kludge twice over!
71
Kludge Meets Kass
 Anti-Jewish Nazi propaganda often invoked the
imagery and language of disgust, purity,
contamination & dehumanization very flagrantly
A poster advertising
the film The Eternal
Jew
Hitler described “the
Jew” as “a maggot in a
festering abscess,
hidden away inside
the clean and healthy
body of the nation”
72
Knobe on Norms and
Intentional Action
 My second example draws
some elegant and exciting
work by Joshua Knobe
which demonstrates the
way in which unconscious
moral judgments –
judgments which an agent
may explicitly reject – can
nonetheless have
significant impact on a
range of morally relevant
intuitions
73
Knobe on Norms and
Intentional Action
 In his new book, Kluge, Gary
Marcus argues that more recently
evolved, computationally slow and
consciously accessible mental
processes – “System 2 Processes”
in the currently fashionable jargon –
were grafted onto older (System 1)
psychological systems designed for
quite different purposes
 The resulting kludgy architecture
accounts for many of the quirks and
shortcomings that contemporary
cognitive science has discovered
74
Knobe on Norms and
Intentional Action
 I think that Knobe’s work provides an
important & disquieting illustration of this
phenomenon in the moral domain
75
Knobe on Norms and
Intentional Action
 The story begins with “the side effect effect” (aka
the Knobe effect) – one of best known and most
surprising finding in the emerging field of experimental
philosophy
 Knobe (2003) reports an experiment in which
participants were presented with a pair of almost
identical vignettes
76
Knobe on Norms and
Intentional Action
The vice-president of a company went to the chairman of
the board and said, ‘We are thinking of starting a new
program. It will help us increase profits, but it will also
harm [help] the environment.’
The chairman of the board answered, ‘I don’t care at all
about harming [helping] the environment. I just want to
make as much profit as I can. Let’s start the new
program.’
They started the new program. Sure enough, the
environment was harmed [helped].
77
Knobe on Norms and
Intentional Action
 In the harm case, participants were asked how much
blame the chairman deserved (on a scale from 0 – 6)
and whether he intentionally harmed the
environment
 In the help case, participants were asked how much
praise the chairman deserved (on a scale from 0 – 6)
and whether he intentionally helped the environment


In the harm case, 82% said the chairman brought
about the side-effect intentionally
In the help case, 77% said the chairman did not bring
about the side-effect intentionally
78
Knobe on Norms and
Intentional Action
 Knobe’s initial hypothesis was that people’s moral
assessment of the side-effect plays a substantial
role in determining whether they are willing to say that
the side-effect was brought about intentionally

A judgment that the side-effect is morally bad makes
it more likely that it will be judged to be intentional

Though this seems incompatible with the widespread
idea that judgments of intentionality are judgments
about a purely factual matter, it does have an
obvious rationale since judgments about whether an
action is intentional play a central role in determining
whether an agent deserves praise or blame
79
Knobe on Norms and
Intentional Action
 Subsequent research showed that, if the hypothesis is
understood as a claim about the effect of moral
judgments that people consciously make, this
hypothesis is mistaken
 The problem emerges clearly in study Knobe ran in
collaboration with David Pizarro & Paul Bloom
80
Knobe on Norms and
Intentional Action
 Liberal university students were given Knobe-style
vignettes in which an advertising executive approves
an ad campaign which has the side-effect of
encouraging interracial sex
or placing gardenias in one’s office
81
Knobe on Norms and
Intentional Action
 None of the participants judged that inter-
racial sex (or placing gardenias) is morally
wrong
 But participants were much more inclined to
say that the executive intentionally
encouraged interracial sex
 Explicit moral judgments cannot explain the
difference in judgments about the intentionality of the side-effects
82
Knobe on Norms and
Intentional Action
 However, (following Pizarro & Bloom) Knobe has
recently proposed that perhaps participants were
making non-conscious normative judgments that
the behavior in question violates a norm that is
made salient by the question or situation, even if it is a
norm that they explicitly reject
83
Knobe on Norms and
Intentional Action
 The picture Knobe now proposes looks like this:
“In reaching a conscious moral judgment, we can
consider a variety of different moral norms, weigh
these norms against each other, perhaps even
determine that some of the norms are themselves
unjustified.”
84
Knobe on Norms and
Intentional Action
 Non-conscious moral judgments are formed
through a much simpler (system-1 style) process

They are formed extremely quickly and therefore
involve very shallow processing

In generating a non-conscious moral judgment, the
only norms we consider are the ones that first
come to mind. We do not search for additional norms;
we do not weigh norms against each other; we do not
ask whether any of the norms might themselves be
unjustified.
85
Knobe on Norms and
Intentional Action

Instead, we simply determine whether the behavior in
question violates any of the norms in the very limited
set we are considering

If it does, we classify it as a transgression. It is this
judgment as to whether or not the behavior is a
transgression that then influences our intuitions
about intentional action.
86
Knobe on Norms and
Intentional Action
 The theory predicts that the most salient norms
evoked by a given case will be the ones used to in
making intentionality judgments, even if subsequent
reflection leads the agent to think that there is nothing
wrong with violating the norm – or that doing so would
be a very good thing.
 Here is a vignette that Knobe has recently used to test
this idea
87
Knobe on Norms and
Intentional Action
In Nazi Germany, there was a law called the ‘racial identification
law.’ The purpose of the law was to help identify people of certain
races so that they could be rounded up and sent to concentration
camps. Shortly after this law was passed, the CEO of a small
corporation decided to make certain organizational changes. The
Vice-President of the corporation said: “By making those
changes, you’ll definitely be increasing our profits. But you’ll also
be violating [fulfilling] the requirements of the racial
identification law.” The CEO said: “Look, I know that I’ll be
violating [fulfilling] the requirements of the law, but I don’t care
one bit about that. All I care about is making as much profit as I
can. Let’s make those organizational changes!” As soon as the
CEO gave this order, the corporation began making the
organizational changes.
 81% of subjects in the violate condition said that he violated
the requirements intentionally; 30% of subjects in the fulfill
condition said that he fulfilled the requirements intentionally.
88
Knobe on Norms and
Intentional Action
 Knobe’s theory is certainly not the last word on how
intentionality judgments are generated
 His work has inspired dozens of other researchers
 there are many studies I have not mentioned
 and many others are underway
89
Knobe on Norms and
Intentional Action
 However, IF Knobe’s theory is on the right track, then
intentionality judgments are a product of a kludgy
architecture which can be influenced by norms and
judgments which the agent
 is not aware of, and

does not endorse
 This raises serious questions about the use of those
judgments in further moral deliberation, or in the law
90
From Kludginess to Skepticism
 Both Kelly’s & Knobe’s work support the hypothesis that
motivates this talk
The psychological mechanism underlying moral
intuition is
A Hodgepodge of Multipurpose Kludges
91
From Kludginess to Skepticism
 Suppose that’s right. What should we conclude about
moral intuition?

The answer is NOT that all moral intuition should be
rejected
 nor even that intuitions that are closely tied to kludgy
features of the mind should be rejected

For, as Shaun Nichols has argued, some of the most
admirable features of the cultural evolution of norms –
including the increased scope and acceptance of
norms prohibiting physical harm – are the products of
kludgy design
92
From Kludginess to Skepticism
 Rather, I suggest, the right conclusion to draw is that
ALL moral intuitions should be viewed with a
healthy dose of skepticism

The mechanisms that give rise to them may not have
been well designed to do anything

So we should be skeptical about moral intuitions for
roughly the same reason that we should be skeptical
of the output of a kludgy piece of computer
software
93
From Kludginess to Skepticism
 Compare and Contrast

The friends of intuition (e.g. moral sense theorists) think the
system producing them is well designed for morally
admirable goals

though it can sometimes misfire when conditions are
unfavorable

Previous enemies of intuition (e.g. Singer) think the system
producing them has been well designed for morally
problematic goals

We believe that the system producing them is a kludge –
much of it has not been well designed at all!
94
From Kludginess to Skepticism
 But if we should be skeptical about all
intuition, how can we go about making
moral decisions?
 That’s a BIG question & a HARD one.

Perhaps I’ll be able to suggest an answer …
95
From Kludginess to Skepticism
…the next time I come to Paris
96