Performance Prediction: Truths
and Falsehoods
Dr. Jordan B Peterson
Professor of Psychology
University of Toronto
What do psychologists do?
• Pursue scientific truth
• Pursue careers
How to pursue scientific truth
• MEASUREMENT
– If the phenomenon cannot be measured
• It does not exist
How to pursue scientific truth, continued
• CONSTRUCT VALIDATION
– Multimethod, multitrait
• Analogous to sensory analysis
– Five senses, not one
– Plus intrasubjective measurement
– Plus technological extension of sensory analysis
Use statistics destructively
• Aim at demolishing your effect
How to pursue a career
• Invent a scale (or a construct)
– Or rename a scale that already exists
• Refuse to validate it
– No convergence
– No divergence
– No criterion
– No multimethod, multitrait analysis
• Correlate it with some phenomenon
How to pursue a career, continued
• Publish the paper
• Establish a small primate dominance hierarchy
– Based on the scale
– Climb the hierarchy
• Use statistics selectively
– Very selectively
• Enjoy the fruits of your success!
– But seriously distort the knowledge base
– And spend your time chasing red herrings
NATURE | NEWS FEATURE
• Replication studies: Bad copy
– In the wake of high-profile controversies,
psychologists are facing up to problems with
replication.
• Ed Yong
– 16 May 2012
Things that might not exist
• Self-esteem
– (neuroticism – extraversion)
– 25000 published papers
• Whole political and educational agendas
• Working memory and EF
– IQ?
• Emotional intelligence
– Agreeableness
Things that might not exist
• Grit
• MacArthur Genius Grant (Angela Duckworth)
• Martin Seligman (positive psychology)
• Conscientiousness
• Optimism/Pessimism
– Extraversion/neuroticism
• Promotion/prevention
– Extraversion/conscientiousness
Things that might not exist
• Empathy
– Measures do not correlate well, but distill to
agreeableness
• Psychopathy
– Agreeableness, negative, minus conscientiousness
– Predatory parasite
Things that might not exist
• Positive illusions
– Extraversion, by Taylor’s own analysis
• Practical intelligence
– Sternberg is a crook
• Multiple intelligences
– Gardner is a crook
Things that might not exist
• Any questionnaire measure independent of
the Big Five
• Any measure of cognitive function
independent of IQ
• Any behavioral measure of a trait,
independent of IQ and the Big Five
Things that might exist
Individual Differences
• Contextual and individual factors
– Contextual
• Cultural factors
• Peer networks
• Social support
– Individual
• Personality
• Cognitive ability
Personality
Traits
Measurement
Items
Employment Variables
Economic Value
Traits
Industriousness
Conscientiousness
Orderliness
STABILITY
Volatility (r)
Emotional Stability
Withdrawal (r)
Politeness
Agreeableness
Compassion
Openness
Creativity
Intellect
PLASTICITY
Extraversion
Assertiveness
Enthusiasm
Personality
• Extraversion
– Happiness, optimism, enthusiasm, gregariousness, assertiveness
• Neuroticism
– .2-.3 with major life outcomes
• Particularly anxiety and depression, since that’s what it measures
• Agreeableness
– Compliance, empathy, maternality, kinship
– criminality
• Conscientiousness
– .4 with major life outcomes
• Openness
– Creativity, intelligence
• Disagreeable/neurotic
– Personality disorders
• Gender differences
– 75% classification accuracy
Emotion & Motivation
• Extraversion
–
–
•
Neuroticism
–
–
•
Incentive reward (hope, curiosity, play, enthusiasm)/dopamine
Harvest attention/approach and exploit social situations
Pain, frustration, disappointment, fear, anxiety/GABA, Serotonin
Avoid threat, uncertainty and punishment
Agreeableness
–
–
Empathy, sympathy, compliance, CARE (Oxytocin)
Form intimate relationships and share
•
–
–
•
Pursue individual agenda/defend territory
People vs things
Conscientiousness
–
–
–
Orderliness: DISGUST
Industriousness: GUILT, SHAME
Maintain order
•
–
•
Vs
Stay uncontaminated and sparkly clean
Implement goals
Openness
–
–
–
–
Manipulate abstractions prior to implementation
General Cognitive Ability
Awe, curiosity
Non-social incentive reward/dopamine
Models are grounded in motivation
Capitalize on social groups
Maintain order
Measurement
Items
Measurement
• Trait Measures
• Big Five Aspect Scale
• Unfakeable Big Five
• General cognitive ability
• Fluid
• Crystallized
• Dorsolateral prefrontal
• Creativity
• Creative Achievement Questionnaire
• Divergent thinking and fluency
(BFAS) Conscientiousness
• Industriousness
– Waste my time (r)
– Always know what I am doing
• Orderliness
– Am not bothered by disorder (r)
– Follow a schedule
(BFAS) Emotional Stability
• Volatility
– Get angry easily
– Keep my emotions under control
• Withdrawal
– Seldom feel blue
– Am filled with doubts about things
(BFAS) Agreeableness
• Politeness
– Respect authority
– Believe that I am better than others (r)
• Compassion
– Am not interested in other people’s problems (r)
– Take no time for others (r)
(BFAS) Openness
• Creativity
– Believe in the importance of art
– Love to reflect on things
• Intellect (but see general cognitive ability)
– Am quick to understand things
– Can handle a lot of information
(BFAS) Extraversion
• Assertiveness
– Take charge
– Lack the talent for influencing people (r)
• Enthusiasm
– Make friends easily
– Am hard to get to know (r)
Unfakeable Big Five (+)
Unfakeable Big Five (-)
Cognitive Ability
• The complexity of the world
– Motivated ends must be met
– Modeling of possibility prior to implementation
• Models generate solutions to three problems:
– Where are you?
– Where are you going?
– How are you going to get there?
– Allows testing and failure
• Popper: “Our hypotheses die in our stead”
Conceptions of cognitive ability
– INTELLIGENCES
– Practical vs analytical
– Social, emotional, moral
– Multiple: linguistic, musical, logical-mathematical, spatial,
body-kinesthetic, intrapersonal, interpersonal
– INTELLIGENCE
– Highest strata has the most explanatory power
» “g” measures “success” across broad domains
» r ~ .50
Effect Size
• Relative magnitude
•
•
•
•
r= .50
r= .35-.50
r= .15-.35
r< .15
90-97%ile
75-90%ile
25-57%ile
0-25%ile
• Binomial
– Predictor r = .30
• From 50/50
• To 65/30
– Predictor r = .50
• From 50/50
• To 75/25
•
Hemphill, J.F. (2003). Interpreting the magnitudes of
correlation coefficients. American Psychologist, 58, 78-8.
General Cognitive Ability
Creativity
Creative Achievement Questionnaire
•
•
•
•
•
•
•
•
•
Visual Arts (painting, sculpture)
0. I have no training or talent in this area.
1. I have taken lessons in this area.
2. People comment on my talent in this
area.
3. I won one or more prizes at juried art
shows.
4. I showed my work in a gallery.
5. I sold a piece of my work.
6. My work was critiqued in local
publications.
7. My work was critiqued in national
publications.
•
•
•
•
•
•
•
•
•
Inventions
0. I do not have recognized talent in this area.
1. I find novel uses for household objects.
2. I sketched out an invention and worked on
its design flaws.
3. I created original software for a computer.
4. I built a prototype of one of my designed
inventions.
5. I sold an inventions to people I know.
6. I received a patent for one of my inventions.
7. I sold an invention to a manufacturing firm.
Employment Variables
Innovation
Hi/Lo
Hi/Hi
Entrepreneurial
Lo/Lo
Rote Tasks
Lo/Hi
Managerial/Administrative
Complexity
Who suits which position?
• Cognitive ability
• High levels of fluid and executive intelligence
– Decision making and planning
– Research
• High levels of retrieval
– Creativity and ideational production
– Leadership and public speaking
• High levels of crystallized intelligence
– Information dispensing and storage
– Writing and lecturing
Who suits which position?
• Specific Performance
• Openness (r = .30 - .40)
– Associated with high levels of creativity/entrepreneurial ability
– Extraversion (r = .20)
» Managers
» Sales People
» Leadership
– Agreeableness
» Lower levels associated with creativity/managerial excellence
» Higher levels for public relations, customer service (?)
Openness
• Complexity and Intelligence
• Innovation and Creativity
Conscientiousness
• Duty and Industriousness
• Disgust and Orderliness
Employment Variables II
• Emotional Stability
• High vs Low Stress
• Agreeableness
• People vs Things
• Extraversion
• Solitary vs Gregarious
Economic Value
Complexity and cognitive ability
• 95-86 percentile
– IQ 130 – 116
• 85-73 percentile
– IQ 115-110
•
•
•
•
•
Attorney, Research Analyst
Editor, Advertising Manager
Chemist, Engineer, Executive
Manager, Trainee,
Systems Analyst, Auditor
•
•
•
•
•
•
•
•
•
Copywriter, Accountant
Manager/Supervisor
Sales Manager
Sales, Programmer
Analyst, Teacher, Adjuster
General Manager
Purchasing Agent
Registered Nurse
Sales Account Executive
Complexity and cognitive ability
• 70-60 percentile
– IQ 108-103
• 55-50 percentile
– IQ 102-100
•
•
•
•
•
•
•
•
•
•
•
Administrative Assistant
Store Manager, Bookkeeper
Credit Clerk. Drafter, Designer
Lab Tester/Tech, Assistant Manager
General Sales, Telephone Sales
Secretary, Accounting Clerk,
Medical Debt Collection
Computer Operator
Customer Service Rep
Technician, Automotive Salesman
Clerk, Typist
•
•
•
•
•
•
•
Dispatcher, General Office
Police Patrol Officer
Receptionist, Cashier
General Clerical
Inside Sales Clerk, Meter Reader
Printer, Teller, Data Entry
Electrical Helper
Complexity and cognitive ability
• 45-42 percentile
– IQ 98-95
• 37-21 percentile
– IQ 93-87
•
•
•
•
•
•
•
Machinist, Food Dept Manager
Quality Control Checker
Claims Clerk, Driver/Deliveryman
Security Guard, Unskilled Labor
Maintenance, Machine Operator
Arc Welder, Die Setter, Mechanic
Medical/Dental Assistant
•
•
•
•
•
•
Messenger, Factory Production
Assembler, Food Service Worker
Nurse’s Aide, Warehouseman
Custodian/Janitor
Material Handler
Packer
Economic Value I
• Predictive Power
• r = .6 to r = .2
– Intelligence
– Conscientiousness (managerial)
– Openness Creativity (innovative)
– Emotional Stability
– Agreeableness
– Extraversion
Economic Value II
• Binomial Effect Size
• Price’s Law
• Economic Impact of Prediction
Pareto Distribution
Individual variation and productivity
prefrontal ability
cognitive ability
job tryout
biographical inventory
personality
reference check
experience
interview
training/experience ratings
academic achievement
interest
education
0
0.1
0.2
0.3
0.4
0.5
0.6
Economic value and placement
– Number of years
– Current - Potential Predictor
– Job Complexity
• High complexity = high variability
– Salary
• Theoretically proportionate to output
• Average output must be 2x salary
Process
•
•
•
•
Advertise
Test online
Screen
Interview best candidates
The Founder Institute is the world's largest entrepreneur
training and startup launch program, helping aspiring
founders across the globe build enduring technology
companies.
Based in Silicon Valley and with chapters across 35 countries,
the Founder Institute has helped launch over 1,112
companies in 5 years.
The company's mission is to "Globalize Silicon Valley" and
build sustainable startup ecosystems that will create one
million new jobs worldwide.
• First, only roughly 30% of applicants are accepted
to the program.
• Then, less than 40% of accepted Founders
generally make it through the program to
Graduation.
• To date, over 20,000 people have applied, and the
Founder Institute can predict with 85% accuracy
somebody's potential to become a successful
technology entrepreneur.