The Interviewer Fallacy:
Evidence from 10 years of MBA interviews
Photo not
necessary
Uri Simonsohn
Francesca Gino
HBS
Motivation
• How is a journal editor like a venture capitalist?
• Continuous flow of judgments
“random” “daily” subsets.
• Research question: Impact of subsetting?
Narrow bracketing
+Belief in law of small numbers
interviewer fallacy
Definition. Reluctance to create subsets of judgments
that differ too much from expected distribution.
Paper in one slide
• Data: 1-5 Rating of MBA interviewees
– Handful per day.
• corr[avg(so far), this interview]<0
• Ruled out alternatives:
– Contrast effects
– Non-random sequence
Data Description
• A business school gave us data
• 10 years: N=9,323, k=31
***INTERRUPT THIS TALK TO COMMENT ON ANOTHER PROJECT***
False-Positive (PsychScience2011): “list all your variables”
Naysayers: “love to, have too many”
Authors of False-Positive: “really?”
Uri: “watch me.”
Note: another 22 variables are listed in this page
Other side of that single sheet of paper
Note:
The .pdf weighs 13Kb.
The Wharton logo from slide 1:
11kb
A hardliner may say: Only
reason to choose not
to post is to hide information from readers.
Back to this talk
Data Description
• A business school gave us data
• 10 years: N=9,323, k=31*
– Interviews per day M=4.5, SD=1.9
– Cluster SE
[repeated measures]
• Info on:
–
–
–
–
Applicant (e.g, GMAT scores, experience, race, gender)
Interviewer identity
Interview: time, date
Ratings (1-5 likert)
• 5 subscores: communication, leader, etc.
• Overall score (M=2.9, SD=0.9)
• Would like to analyze like gambler fallacy
– HHHHpr(T)↑
• Problem
– Non-binary data
– Covariates
– Different interviewers
Instead:
Scorek,i = OLS(average score so fari , covariates)
<0
k: Interviewee, 1 to N that day.
i : Interviewer
Prediction:
(1)
Average interview score
Given by same interviewer to previous interviewees that day (1-5)
(3)
(4)
Interview Score
(1-5)
Dependent variable:
Specification
(2)
Baseline
Interviewee
controls
Interview
controls
Score (1-5) of
written
application
-0.116***
-0.110***
-0.105***
-0.088**
(0.038)
(0.035)
(0.036)
(0.035)
0.244***
0.250***
0.079**
(0.036)
(0.035)
(0.032)
0.324***
0.319***
0.254***
(0.057)
(0.055)
(0.055)
GMAT score of applicant (/100)
Months of job experience of applicant (/100)
Number of interviews by same interviewer that day
Total
So far
-0.000
0.001
(0.012)
(0.012)
-0.018
-0.010
(0.013)
(0.014)
Score given by reader of application
0.340***
(0.044)
Other controls
Month*year dummies (k=12*9)
Interviewer dummies (k= 21)
Interviewee gender, race (k=9), age & age-squared
Interview's time (k=12) & location (k=4)
Yes
Yes
No
No
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Effect Size
• Average interview 1 point higher,
• Equivalent to losing:
– 40 GMAT points, or
– 30 months of experience.
Alternative Explanations
• Contrast effects
• Non-random sequencing of interviews
Contrast vs. Interviewer Fallacy
Two divergent predictions:
1) Same effect on the interview subscores?
Explanation
Prediction
Contrast:
Int.Fallacy:
yes, and stronger
no, or at least weaker.
Data:
- Every one of five subscores: n.s.
- Average a-la Robyn Dawes: n.s.
- Biggest point estimate, ¼ as big
- one is >0
Contrast vs. Interviewer Fallacy
Two divergent predictions:
2) Effect as end of day approaches.
Explanation
Prediction
Contrast:
Int.Fallacy:
weaker (arguably)
stronger (absolutely)
Data:
Estimate same regressions for:
• last interview of day
• 1 interview left
• 2 interviews left
Effect of previous interviews as
day’s end approaches
Alternative Explanations
• Contrast effects
• Non-random sequencing of interviews
• If better candidates follow bad ones
or vice-versa
 spurious finding.
• Can we predict objective quality with
average-interview-score-so-far?
• Test:
GMAT=OLS(avg.score)
Job Experience = OLS(avg.score)
Same table + 2 new columns
(1)
Average interview score
Given by same interviewer to previous interviewees that day (1-5)
(3)
(5)
(6)
PLACEBOS
GMAT
(250-800)
Experienc
(in months
Same
as (3)
Same
as (3)
Baseline
Interviewee
controls
Interview
controls
Score (1-5) of
written
application
-0.116***
-0.110***
-0.105***
-0.088**
0.085
0.251
(0.038)
(0.035)
(0.036)
(0.035)
(2.063)
(0.959)
0.244***
0.250***
0.079**
--
1.140**
(0.036)
(0.035)
(0.032)
--
(0.495)
0.324***
0.319***
0.254***
10.363**
--
(0.057)
(0.055)
(0.055)
(4.541)
--
GMAT score of applicant (/100)
Months of job experience of applicant (/100)
Number of interviews by same interviewer that day
Total
So far
-0.000
0.001
0.845
0.504*
(0.012)
(0.012)
(0.676)
(0.290)
-0.018
-0.010
-0.461
0.008
(0.013)
(0.014)
(1.275)
(0.360)
0.340***
24.190***
(0.044)
(1.719)
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Score given by reader of application
Other controls
Month*year dummies (k=12*9)
Interviewer dummies (k= 21)
Interviewee gender, race (k=9), age & age-squared
Interview's time (k=12) & location (k=4)
(4)
Interview Score
(1-5)
Dependent variable:
Specification
(2)
Yes
Yes
No
No
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Possible Mechanisms
1) Gambler fallacy + confirmation bias
2) Mental Accounting
3) Accountability
A note on the internal validity
of non-lab data
• In the lab: hard to study interviewer fallacy
• Participants could be learning about
– Scale use
– Distribution of underlying stimuli quality
• Some psychological questions are better
studied outside the lab.
• This seems likes one of them.