Judgment
Removing the Mystery
Toby Groves
tgroves@tgroves.com
@tobylgroves
Linkedin TobyGroves
Understanding Judgment
• The role of contrast and perspective
• A basic understanding of probabilities
• Recognizing the limitations inherent in our thinking and our technology
• An understanding of the invisible forces that influence judgment (biases)
• The neuroscience behind judgment processes
• The role of the the emotional brain in logical judgment
• Judgment under pressure
What is Judgment?
• Logic- Normative theory of how one should reason. Formal logic.
• Mathematics/computer science
• Hard answers-precisely defined operations
• Not descriptive theory of how humans actually reason
• Does not generally consider psychology
• Critical Thinking- Everyday, real life reasoning.
• Using creativity
• Deals with psychology. Beliefs and recognizing biases
• Gray areas, complex, unclear and changing situations
• Reflective and independent thinking
• Judgment- decisions made, and conclusions reached that drive actions
and behaviors in how you deal with a situation or set of facts
Judgment: Logical and Rational thought
• Rational drug design
• The technical aspect of models for drug design are easier than the human
endeavor of interpretation
• Placebo effect without believing in the placebo?
• Unconscious effects
• Cyclosporin A
• Placebo studies using the meaning response
• We are constantly primed- consciously and subconsciously and unconsciously
Judgment
Good Judgment –Including Ethical Judgment
What it is not…
• Whatever society accepts- If this were true, you could take a poll and you would believe
whatever the result is
• Not science- Science deals with how people “ought to act”, not considering biases,
psychology, or emotions
• Not the law- Laws can deviate from what is ethical, and rules have a counterintuitive
effect on ethical thought processes.
• Intuition- We may have formed habits in thinking that bias us. Familiar habits trigger
oxytocin.
(Beaman et al. 1979; Diener et al., 1976)
Judgment
Good Judgment-Including Ethical Judgment
What it is…
Endless number of models, theories, and definitions-for our purposes today we will view
ethics as:
• Well founded standards of critical thinking of fairness and of right and wrong
• Understanding our own reasoning processes, thinking critically about these processes,
and seeking to understand our unique biases
• A continuous effort to understand and develop one’s own reasoning and standards
• Understanding the important role the emotional brain plays in judgment and critical
thinking
(Beaman et al. 1979; Diener et al., 1976)
Judgment: Removing the Mystery
What is judgment?
Perception- Becoming aware of something through the senses
Interpretation- The action of explaining the meaning of something
Judgment- Forming an opinion, estimate, notion or conclusion from
circumstances presented to the mind
(Beaman et al. 1979; Diener et al., 1976)
What is Critical Thinking?
• Critical thinking skills allow us to:
• Understand connections between ideas and concepts
• Construct and evaluate arguments
• See inconsistencies and gaps in reasoning
• Assess the relevance of ideas
• Examine the justification of your own beliefs
• Critical thinking should not be confused with:
• Being critical or argumentative
(Indeed, critical thinking plays an important role in cooperative reasoning)
Critical Thinking
• Halpern Critical Thinking Assessment dimensions of critical thinking
• Argument analysis
Recognizing the difference between conclusions and
assumptions
• Seeking reasons why certain considerations would support
or fail to support certain conclusions
• The difference between uninformed opinions and reasoned
judgment
Critical Thinking
• Halpern Critical Thinking Assessment dimensions of critical thinking
• Thinking as hypothesis testing
• Looking at a sample and drawing conclusions and testing to
see if that conclusion is reasonable
• Hasty generalizations from samples of behavior
• Not considering control conditions (The change you
witnessed may have happened with or without the item
you identified)
Critical Thinking
• Halpern Critical Thinking Assessment dimensions of critical thinking
• Likelihood and uncertainty
• A basic understanding of probabilities
• Likelihood of outcome in uncertain situations
• An extreme event is likely to be followed by a less
extreme event (regression to the mean)
• If a coin flip results in 3 heads in a row, what is the likely
outcome of the 4th flip?
Critical Thinking
• Halpern Critical Thinking Assessment dimensions of critical thinking
• Decision making and problem solving (judgment)
• Decision making -Selecting from alternatives based on
relevant information
• Problem solving -Finding solutions to a situation, requires
seeing connections in data that may not appear to go
together
Research in judgment skills
Judgment: Critical Thinking
• Scientific method
• Ask a question
• Do background research
• Construct a hypothesis
• Test your hypothesis with an experiment
• Analyze your data and draw a conclusion
• Communicate your results
• Limitations-Goal is not prove a rule but to disprove (null hypothesis)
• Randomly assign subjects
• Exactly the same treatment of subjects with no outside influences that bias
the results (remove contextual differences)
Judgment: Critical Thinking
• Important in the scientific method:
• Validity-Are you actually measuring what you intended to
measure? Internal and external validity. The extent to which a
concept, conclusion or measurement is well founded and actually
corresponds to the real world.
• Reliability-The ability to achieve similar results under consistent
conditions.
Judgment: Critical Thinking
• Critical thinking
• Is thinking about your thinking process
• Analyzing the thinking process and the outcome
• Supporting your beliefs and actions with solid reasons
• A meta-thinking skill that requires
• Theory
• Understanding the rules of reasoning
• Laws of logic and methods of scientific reasoning
• Understanding what not to do (not just what to do)
• Knowledge of common mistakes in reasoning
(biases)
Critical Thinking- A meta-thinking skill that requires:
• Attitude
• Openness
• Willingness to be wrong
• Exploring biases and assumptions
• Accepting, even seeking criticism
• Preferring to figure out problems on your own so you really
understand them-not handed the answer
• Analyzing your errors in judgment
• Instead of excuses, curiosity about what biases or other
errors may have resulted in the incorrect judgment
Judgment: Critical Thinking
• You may use:
• Inductive reasoning-From the bottom up approach-Seek strong
evidence for the truth of a conclusion-seeks only to give probable
indication of truth-not absolute-is inherently uncertain
• The old dictionary definition of deriving general principles from
specific observations is considered out of date.
• Deductive reasoning-Top down approach-Start broad and work
towards specific conclusion-Relies on original premise being correct.
Judgment: When Less is More
• The story of Cook County Hospital
Situational Awareness
Attractors and Distractors
The “more information is always better” bias
Research shows that doctors have more confidence in their
diagnoses when more data is available, even though the
accuracy of their diagnoses does not improve with that
additional data-Joseph Kidd
-Stock market performance/weather forecasts
Judgment: When Less is More
Creeping determinism
After an event has occurred, we feel very sure that the outcome is what we would have
predicted
Research had people predict an outcome and the level of certainty of their outcome,
then after the outcome occurred, they asked them again- and people gave much
different answers, either changing their answers (saying they had predicted what ended
up occurring, or changing their level of certainty very dramatically.
Judgment: Complex or ComplicatedUnderstanding the true nature of your problem
Complicated-Many different moving parts that all fit together like a puzzle (An
engine)
Complex-Many similar parts that interact in ways that are not fully predictable (A
group of people). What happens is a mystery.
Does this distinction matter? Absolutely-it changes the way we search for data
Choice Complexity and Professional Judgment
Study on bias in professional decision-making using Doctors
• Decision to send patient-a 67 year old farmer for hip replacement surgery
• First group of doctors told they forgot to try one drug-Ibuprofen. Would
they call patient back from surgery to try the drug? 50% said pull then back
• Second group of doctors told they forgot two drugs-Ibuprofen & Piroxicam.
Would they call patient back from surgery? 72% let patient go on to surgery
Why? Choice complexity increases chance of going with default option
Redelmeier & Shafir (1995)
Judgment: Puzzle or Mystery
• Puzzles
• Source dependent (data)
• Clear conclusion
• Every piece of new information makes answer clearer
• Search will fall in the normal curve
• Mysteries
• Receiver dependent (interpretation skills)
• Answer does not become clearer with additional piece of information
• Making sense of the information we already have makes answer clearer
• Making judgments about situations that are uncertain
• Search is open-may be outlier data, or uncorrelated
Judgment:
Choke or Panic
Choke- Thinking too much-When you are thinking
about the steps and should just be on autopilot
Panic- Thinking too little, when you should be
following the steps that you learned
Time Management
Choking vs. Panic
The part of the brain that handles new learning is different than the part of the
brain that handles memory and memorized activities.
-Memorized tasks-Cerebrum-automatic and extremely fast
-New learning- cerebellum handles motor learning, parts of temporal lobe
handle other types of learning-slower
Judgment:
Research Techniques
Quantitative- Relies on strict assumptions (normal distribution,
no outliers, variables are mutually exclusive), data drivencomplicated-what the data is
Qualitative- Recognize assumptions may be biased, except for
underlying assumption that people can perceive the world
differently-complex-what the data means
Big Data and Judgment
• What makes “BIG” data different from the data we’ve always used?
• Has a different direction
• Usually we have a problem and form a hypothesis and then
gather data
• In big data-we have this pile of data and want to see what
there is to learn
Judgment
What is the nature of your data?
Understanding the nature of the data requires different perspectives
One or a few things
-middleMany things
Judgment: Insight
Is the problem you face one of logic, or one of insight?
• Insight- Understanding the hidden nature of something.
• Requires a new perspective to achieve a solution
Judgment: Insight
There are ten bags, each containing ten weights, all of which
look identical. In nine of the bags each weight is 16 ounces, but
in one of the bags the weights are actually 17 ounces each.
How is it possible, in a single weighing on an accurate weighing
scale, to determine which bag contains the 17-ounce weights?
Judgment: Insight
How is it possible, in a single weighing on an accurate weighing
scale, to determine which bag contains the 17-ounce weights?
Take 1 from the 1st bag, 2 from the 2nd, 3 from the 3rd etc. Then
weigh all those coins. If all the bags weigh 16 ounces you will
have 55 ounces (10+9+8+7+6+5+4+3+2+1). Any amount in
excess of the 55 ounces will determine which bag contains the 17
ounces (two ounces over = bag 2 ,if it is 7 ounces over = bag 7
etc)
Insight-Spatial
Three cards lie face down on a table, arranged in a row from left to right. We have the following
information about them. a. The Jack is to the left of the Queen b. The Diamond is to the left of
the Spade c. The King is to the right of the Heart d. The Spade is to the right of the King. Which
card - by face and suit - occupies each position?
Insight
Three cards lie face down on a table, arranged in a row from left to right. We have the following
information about them. a. The Jack is to the left of the Queen b. The Diamond is to the left of
the Spade c. The King is to the right of the Heart d. The Spade is to the right of the King. Which
card - by face and suit - occupies each position?
Jack and hearts, King of Diamonds, Queen of Spades
Judgment: Insight
2, 4, 6, 30, 32, 34, 36, 40, 42, 44, 46,
50, 52, 54, 56, 60, 62, 64, 66, ___?
Judgment: Insight
Think of a stranger you saw today, perhaps the
person who served you coffee-anyone.
-Picture that person
-Now write a description of that persons face
-You would now have more difficulty picking
that person out of a lineup
Judgment: Insight
Dissecting something-breaking it down into
pieces, damages insight. It loses meaning as a
whole.
Judgment: Insight-Removing barriers to Rapid Cognition
Rules of Improv
• Improvisers get a suggestion of a location, object, or relationship
• Make up everything on the spot
• But…guided by basic rules
Judgment: Insight-Removing barriers to Rapid Cognition
Rules of Improv
Rule #1 Yes and… This is the holy grail of improv-this allows scenes
to be created out of nothing. This rule says we must agree with the
reality created by the other actor and add to it which is agreed to in
turn. Disagreeing “blocks” the scene and this is the end of the scenenothing but frustration comes from this disagreement.
Judgment: Insight-Removing barriers to Rapid Cognition
Rules of Improv
Rule #2 Act/React- Everyone on stage should be making an effort to
contribute to the scene-take responsibility and confront fear. Put an
idea out there and trust that your fellow actors will make the best of
it. No holding back-be bold! If other actors are on a roll-let them
keep going, but be ready as soon as the scene allows. When
everyone doesn’t pull their weight – the scene dies.
Judgment: Insight-Removing barriers to Rapid Cognition
Rules of Improv
Rule #3 Make someone else look good and you look good- Everyone
is a supporting actor and instead of looking out for yourself, look out
for the other actors. This builds trust, acceptance, responsibility and
commitment. Have the other actors back. Never, ever put someone
else on the spot to get yourself out of one. Don’t block, don’t say
something “standard” that stalls the scene.
Judgment: Insight-Removing barriers to Rapid Cognition
Rules of Improv
Rule #4 Be Vulnerable and Honest- Don’t try too hard to be funny, it
will feel forced. Be as natural as possible, this is always most
interesting and moves the scene forward. Don’t hide behind
standard answers and responses-be authentic!
Judgment: Insight-Removing barriers to Rapid Cognition
Rules of Improv
Rule #5 There Are No Mistakes- Everyone is doing the best they canno matter what happens, everything is used-a bad joke, someone’s
frustration, etc. This shows that the reward that comes with the risk
outweighs staying safe.
Judgment: Insight-Removing barriers to Rapid Cognition
Rules of Improv
This basic structure frees you up to think openly, like the doctors at
Cook County hospital when they used the Goldman Algorithm
Emotional Need for Certainty
• Search for level of certainty that doesn’t exist
• We subconsciously assign tremendous weight to our
assumed outcomes and automatic character assessments
• This assessment causes profound biases our
perceptions
• Colors our judgment
• Results in automatic assumptions and predictions
• Causes subliminal susceptibility to seek subliminal
information
(Dan Ariely, Duke University)
Negotiation Research in Auditing
• The more ambiguous the accounting issue, auditor likely to accept clients
preferred treatment.
• The more important/increased pressure surrounding the accounting issue,
auditor more likely to concede
• The greater number of possible alternative treatments, the less likely the
auditor was to insist on their judgment
• Clients perceive they are more likely to persuade auditors if firm has short
tenure
• Auditors are much more likely to waive smaller adjustments that aggregate to a
material amount than those that are individually material (Small Steps Phenomenon)
Brown & Wright (2008)
Judgment and Pressure
Take a “Mental” Walk
The importance of taking breaks
• Heavy concentration causes fatigue
• Fatigue results in diminished attentional control
• Give your mind an easier task every 20-30 minutes for a few minutes
• Take a longer break every 2-3 hours
• Improves control and judgment
• Can allow elusive answers to appear
Our mind will force the breaks if you don’t take them voluntarily
-Research shows that we spend 30%-47% of our waking hours daydreaming
We benefit from taking a walk-even if it is a “Mental Walk” –spurs those spatial parts of our
brain.
Judgment and Pressure
Task Type: Creative vs. Logical
Creative Tasks
Suffer more from pressures- limitations, distractions, and structure/rules
Logical tasks
Can improve under low pressure but suffer increasingly as pressure increases into
moderate/extreme
Physical tasks
Can improve under moderate pressure but suffer under extreme pressure
The Effect of Rules and Sanctions
• Narrows our thinking processes
• Control the tendency to act only in our self interest (Self protection-Good
Samaritan laws)
• The tendency to focus on the letter of the rule rather than the spirit.
• Is tendency to interpret rules narrowly due to the characteristics of the rules
themselves?
• Principles versus Rules based accounting standards
• CFO’s less likely to make aggressive accounting choices under “Principles”
based system (Agoglia, 2009)
• CFO’s came to more similar conclusions under principles than rules based
system
• U.S. accounting standards have become so precise as to invite opportunistic
interpretation by corporate executives (Agoglia, 2009)
• The mention of sanctions gives a different frame
Mood and Professional Judgment
• Can something as simple as mood affect professional
judgment?
• Study:
• Auditors in a “good” mood state found inventory values to
be an average of about 5% less than the client valuation
• Auditors in a “poor” mood state found inventory values to
be an average of about 8% less than the client valuation
(Beaman et al. 1979; Diener et al., 1976)
Recognizing Biases
Confirmation Bias- The tendency of people to favor information that confirms their
beliefs or hypotheses. Studies show we tend to seek information that confirms
rather than properly tests the validity of what we are told
Selective Perception-A cognitive bias wherein individuals are subconsciously
attracted to stimuli that falls in their range of reasonable expectation and are
oblivious to other stimuli.
Belief Perseverance- Our tendency to seek information that is consistent with our
pre-existing beliefs.
Biased Processing of Disconfirming Information- The subconscious bias towards
being more critical of evidence that disconfirms our initial beliefs than evidence
that confirms it.
Judgment
Removing the Mystery
Toby Groves
tgroves@tgroves.com
@tobylgroves
Linkedin TobyGroves