What is the Imitation Game?
Robert Evans and Harry Collins
Centre for the Study of Knowledge, Expertise and Science (KES)
Cardiff School of Social Sciences
http://www.cardiff.ac.uk/socsi/expertise
Overview

Theory and Background



Imitation Games in Practice




Different Kinds of Expertise
Turing Test
Software and data
Identify and Chance conditions
Qualitative and Quantitative Results
Comparative Research

Cross-cultural and longitudinal research
Understanding societies
http://bmrc.berkeley.edu/courseware/cs298/spring99/w9/slides/sld006.htm
Three Kinds of Expertise



Contributory expertise: enables those who have
acquired it to contribute linguistically and practically
to the community through the expertise is
sustained. The most common usage of the word
‘expert’.
Interactional Expertise: expertise in the language of
a specialism in the absence of expertise in its
practice. Like contributory expertise, it requires the
tacit-knowledge acquired by immersion in a form-oflife (i.e. socialisation). It enables individuals to talk
as if they had contributory expertise even though
they lack practical or craft skills.
No Expertise: not even the ability to talk intelligently
about a given domain or topic
Collins and Evans (2002)
Turing Tests and Imitation Games
In the Imitation Game, the
judge must decide which
participant shares their social
group.
In the Turing Test, the judge
must decide which is the
computer and which is the
human.
Modern Imitation Game
R1
‘once a week’
R2
‘not very often, when
they need doing’
Male pretending to
be female
Female judge setting
questions
R2 is female ‘because I expected
How
do women
you
the man
tooften
believe
are
pluck
your eyebrows?
more
regulated
in their beauty
regime than they actually are
Female answering
naturally
Judge compares
answers
Judge’s
verdict
Judge’s
confidence
Judge’s
reason
Judge’s
question
Respondents
Answers
Automatically recorded data

Software records



Questions and answers
 analysis of topics and assumptions used by
judges (i.e. what is distinctive about own
culture)
Guesses and confidences
 what discriminates and what does not (i.e.
what is shared with others and what is not)
Judge’s reasons
 cultural norms and variation (i.e. what counts
as a ‘mistake’ and what as normal variation)
Researcher generated data

Researchers needed to record



Problems with method and software
(e.g. how to use, do participants
understand role etc)
Strategies used by judges and
pretenders (e.g. what is hypothesis
being tested by judge, how does
pretender ‘know’ the answer)
Effort and determination (e.g. judge’s
motives and fears)
Imitation Games with the Blind
Identify Condition
 Judge is blind
 Target expertise is
‘being blind’
 Sighted participant
has to pretend to be
blind
 Prediction: Judge will
succeed in identifying
participants
Chance Condition
 Judge is sighted
 Target expertise is
‘being sighted’
 Blind participant has
to pretend to be
sighted
 Prediction: Blind
person will succeed in
pretending
Sighted dialogs
Judge
R1
R2
Could you tell me
roughly how old
you are and
whether you have
been registered
blind since birth
I'm 50 and have
been blind since I
was 10
I'm 30 and I've
been registered
blind since I was
Do you have any
residual sight and
what mobility aids
do you use?
No I have no
residual sight. I
use a white stick
and have a guide
dog
Twelve. I've got
light and dark and
color perception in
one eye and I use
a guide dog.
Judges’ reasons

Question One


The second person is not black and white and you
do not usually lose your sight over-night, so the fact
they mention being registered suggest that they are
blind. If the first one was blind they would normally
say how they became blind if it was sudden. (level
2)
Question Two

I have both white stick and dog but would never
use both at same time. Therefore if I was
responding I would say something like I use a guide
dog predominantly but sometimes use a white stick
-- but if you are blind you would call it a cane
normally. Also, number 2 was much less black and
white. It's always grades of blindness. (level 4)
Blind dialogs
Judge
R1
R2
So let me start with
sport. Are you interested
in tennis and do you ever
watch it on the
television?
I watch Wimbledon a
little bit on the television
and occasionally the
Australian Open in
January
I like tennis but only
watch big tournaments
like Wimbledon
So tell me what you
think about the HawkEye line judging system
Not being a tennis
professional it is not for
me to say if it should or
should not be used. It
does not really alter
viewing
It adds an other element
to the game which could
make it more interesting
Blind dialogs
Judge
R1
R2
How accurately would
you say a human can
judge the flight of a
tennis-ball? I mean,
would you say they could
tell the difference
between touch the line
and 1mm out 2mm out 1
cm out, 2 cm out, or
what, and what would it
depend on?
I think often a tennis
player is not in a position
to judge accurately as
they are not usually
parallel with the line. I
think that if you set up a
test for a line judge with
two balls one which
landed on the line and
one which landed 1mm
away from the line, I
don't think they could
tell the difference. If you
think how small 1mm is
then it would be so hard
for them to judge.
it would depend on the
speed the ball was
travelling and the
position of the judge
relative to the line and
obviously the closer the
ball is the line the harder
it would be to make a
judgement. So you
would have to judge
each call on an individual
bases as there are a lot
of factors.
Judges’ reasons

R1 is the blind person



‘I think respondent 1 gives himself away when he
discusses the human judgments on the flight of a
tennis ball.’
‘I cannot believe a sighted person saying that
Hawk-eye does not alter the viewing.’
R2 is the blind person


‘The Hawk-Eye questions reveal some quite specific
information that I don’t think was published in audio
media. Also, the story wasn’t that important that I’d
expect it to be picked up by the audio news services
provided to the blind.’
‘person 2 seems really unfamiliar with hawk-eye,
given that they say they watch Wimbledon’
Interpreting Results
The Identification Ratio
measures the excess of
correct identifications
Comparisons across topics
IdentificationRatio
Ratio
Identification
Chance
Identify Conditions
Conditions
1.00
1.00
0.80
0.80
0.60
0.60
0.40
0.40
0.20
0.20
0.00
0.00
0.73
0.86
0.68
0.44
0.33
0.05
colour blind
blind
-0.20
-0.20 colour
0.13
0.00
-0.01
-0.13
pitch
blind
sexuality
sexuality
religion
religion
Topics
ID Ratio
Mean IR for ID
condition approx 0.6
Know
Net Don't Know
Mean IR for Chance
condition approx zero
IG and Comparative Research

Social Change and Social Difference



Imitation Game as ‘ethno-method’





How to standardise data collection?
How to reflect local variations?
Participants’ discourse is always contemporary
Judge as proxy ethnographer
Participants as proxy stranger
Participant errors like breaching experiments
Imitation Game as secondary data


Identification ratios by time, place and topic
Qualitative data based on topics, reasons, errors
Imitation Games on Gender

Hypotheses



Malestream culture: hegemonic
masculinity implies asymmetrical
results
1960s and all that: gender roles and
identities more flexible so knowledge
widely shared
Age as confounding factor


Younger people have less interaction
The 1960s are history?
Research Design

Research Methods module project


Phase One



Recruit students and heterosexual,
cohabiting parents
Students:
Parents:
29 games (15 F, 14 M)
25 games (12 F, 13 M)
Phase Two


Students:
Parents:
70 games (40 F, 30 M)
70 games (50 F, 20 M)
Results by gender
Female Male
Incorrect
11
12
Don't Know
15
10
Correct
29
22
Total
55
44
0.33
0.23
IR
Female Male
Incorrect
21
13
Don't Know
19
9
Correct
22
11
Total
62
33
0.02
-0.06
IR
Phase One and Two, students
Approximates identify condition
for both groups, female judges
appear more successful than
male judges but not statistically
significant (p = 0.37)
Phase One and Two, students
Approximates chance condition
for both groups, no statistically
significant difference between
genders (p = 0.26)
Results by age
Young
Old
Incorrect
6
9
Don't Know
5
4
Correct
18
12
Total
29
25
0.41
0.12
IR
Young
Old
Incorrect
17
25
Don't Know
20
24
Correct
33
21
Total
70
70
0.23
-0.06
IR
Phase One only
Clear difference between
groups; young judges more
successful than older judges (p
= 0.031)
Phase Two only
Clear difference between
groups; young judges more
successful than older judges (p
= 0.007)
Conclusions for gender

Differences by gender



No statistically significant differences in sample
as a whole
Some differences in topics chosen to
‘represent’ and test cultures
Clear differences by age


Older participants much better at pretending
(i.e. have more interactional expertise)I
Impossible to distinguish between ‘1960s’
hypothesis and the ‘learning’ hypothesis
without longitudinal study
Conclusions

Interactional expertise


Imitation Game



Useful concept for explaining social phenomena
Combines qualitative and quantitative
approaches
Opportunities for comparative research within
cultures, across cultures, over time
Want to know more

http://www.cardiff.ac.uk/socsi/expertise
References and reading




Collins, Harry and Robert Evans (2002) ‘The Third Wave of
Science Studies: Studies of Expertise and Experience’, Social
Studies of Sciences, 32 (2): 235-96.
Collins, Harry and Robert Evans (2007) Rethinking Expertise,
Chicago, IL: The University of Chicago Press.
Collins, Harry, Robert Evans, Rodrigo Ribeiro and Martin Hall
(2006), ‘Experiments with Interactional Expertise, Studies In
History and Philosophy of Science, Volume 37, No. 4 (Dec
2006), pp. 656-674.
Evans, Robert and Harry Collins (forthcoming, 2010)
‘Interactional Expertise and the Imitation Game’ in Michael
Gorman (ed) Trading Zones and Interactional Expertise:
Creating New Kinds of Collaboration, Chicago, IL: MIT Press