Personality Assessment,
Measurement, and
Research Design
CHAPTER 2
© 2015 M. Guthrie Yarwood
4 Sources of personality data
II. Reliability / Validity in
Personality
III. A little more on observer
reports
I.
Outline
© 2015 M. Guthrie Yarwood
I. Sources of Personality Data
 Self-Report Data (S-Data)
 Observer-Report Data (O-Data)
 Test-Data (T-Data)
 Life-Outcome Data (L-Data)
© 2015 M. Guthrie Yarwood
Self-Report Data (S-Data)
 Information provided by a person, such as through a
survey or interview
 Limitations of S-data?
© 2015 M. Guthrie Yarwood
O-Data
 Information provided by someone else about another
person


Professional personality assessors
People who actually know the target person
 Naturalistic vs. Artificial Observation
 Limitations?
© 2015 M. Guthrie Yarwood
Test-Data (T-Data)
 Information provided by standardized tests or
testing situations
 Situation designed to elicit behaviors that serve as
indicators of personality
© 2015 M. Guthrie Yarwood
Test-Data Creativity Example
 What are unusual uses for common objects – bricks,
knives, newspapers?
 Answers to hypothetical events
 What would happen if people went blind?
 What would happen is people shrank to 12 inches tall?
(Paul Silvia’s work)
© 2015 M. Guthrie Yarwood
Test-Data: Other Examples
 Mechanical recording devices, e.g., “Actometer”
used to assess children’s activity
 Physiological data
 Projective Tests
 Ex: Fairy Tale Test
 Limitations?
© 2015 M. Guthrie Yarwood
Life-Outcome Data (L-Data)
 Information that can be gleaned from events,
activities, and outcomes in a person’s life that is
available for public scrutiny—e.g., marriage,
speeding tickets
 Can serve as important source of “real life”
information about personality
 Ex: implicit egotism: people gravitate toward
people, places, things that resemble the self
© 2015 M. Guthrie Yarwood
(Pelham and colleagues’ work)
Issues in Personality Assessment
 Links among different data sources
 Fallibility of personality measurement
 All sources of data have limitations
 Results that replicate through “triangulation” are most
powerful
© 2015 M. Guthrie Yarwood
You are a personality psychologist and would like to
measure the personality trait risk-taking (i.e., sensation
seeking).
How could you measure risk-taking using each of the four
data sources?
 S-Data
 O-Data
 T-Data
 L-Data
© 2015 M. Guthrie Yarwood
Evaluation of Personality Measures
How do we know whether our scale is a “good scale?”
 Types of Errors
 Reliability
 Validity
 Threats to Reliability and Validity
© 2015 M. Guthrie Yarwood
Extraneous vs. Confounding Variables
 Extraneous variables are variables that may
compete with the independent variable in explaining the
outcome of a study.
 Confounding variable: an extraneous variable
that does indeed influence the dependent variable.
 A confounding variable systematically varies or
influences the independent variable and also
influences the dependent variable.
© 2015 M. Guthrie Yarwood
Random vs. Constant Errors
 Random Errors (unsystematic): extraneous
variables whose average influence on the outcome is
the same in both (or all) conditions (Aronson et al.,
1990)
 Affects reliability AND validity
 Constant (systematic) Errors: influences all the
scores in one condition in the same direction and has
no effect or a different effect on the other
condition.


Affects ONLY validity
Confounding variables
© 2015 M. Guthrie Yarwood
Clarification Check!
 IV: Type of show
 Condition
1: Extraverts watch comedy show
 Condition 2: Extraverts watch neutral show
 DV: Self-reported emotions
 Extraverts
report more positive emotions in
Comedy Condition than in Neutral Condition
© 2015 M. Guthrie Yarwood
Clarification Check! – Random Errors
 Extraverts in conditions 1 and 2 are placed in very hot
rooms. The hot temperature lowers their self-reported
levels of positive emotions.
 But, we still find that extraverts report more positive
emotions when watching the comedy show than the
neutral show.
 Accurate: We find differences in self-reported positive
emotions (on 5-point scale):

Comedy Cond. = 4.8, Neutral Cond. = 2.5
 Random Errors: In presence of hot temperature, we find
differences in self-reported positive emotions

Comedy Cond. = 3.0, Neutral Cond. = 1.5
© 2015 M. Guthrie Yarwood
Clarification Check! – Random Errors
 Extraverts in conditions 1 and 2 are placed in very hot
rooms. The hot temperature lowers their self-reported
Hot temperature lowers the
levels of positive emotions.
scores in both conditions!!
 But, we still find that extraverts report more positive
emotions when watching the comedy show than the
neutral show.
 Accurate: We find differences in self-reported positive
emotions (on 5-point scale):

Comedy Cond. 1 = 4.8, Neutral Cond. 2 = 2.5
 Random Errors: In presence of hot temperature, we find
differences in self-reported positive emotions

Comedy Cond = 3.0, Neutral Cond.2 = 1.5
© 2015 M. Guthrie Yarwood
Clarification Check! – Constant Errors
 Extraverts in Comedy Cond. are in a 70◦ room.
Extraverts in Neutral Cond. are placed in a 78◦ room. In
Neutral Cond. only, the hot temperature lowers selfreported levels of positive emotions.
 We find that the comedy show results in more positive
emotions than the neutral show.
 Accurate: We do not find differences in self-reported
positive emotions

Comedy Cond. = 4.8, Neutral Cond. = 4.6
 Constant Error: We find differences in positive emotions
due to the hot temperature, not due to the manipulation!

Comedy Cond. = 4.8, Neutral Cond. = 2.5
© 2015 M. Guthrie Yarwood
Clarification Check! – Constant Errors
 Extraverts in Comedy Cond. are in a 70◦ room.
Extraverts in Neutral Cond. are placedHot
intemperature
a 78◦ room.
lowersIn
the
scores
in Condition
Neutral Cond. only, the hot temperature
lowers
self- 2, but
not Condition 1!!
reported levels of positive emotions.
 We find that the comedy show results in more positive
emotions than the neutral show.
 Accurate: We do not find differences in self-reported
positive emotions

Comedy Cond. = 4.8, Neutral Cond. = 4.6
 Constant Error: We find differences in positive emotions
due to the hot temperature, not due to the manipulation!

Comedy Cond. = 4.8, Neutral Cond. = 2.5
© 2015 M. Guthrie Yarwood
Reliability
 The ability of a test to measure an attribute
consistently


Does this extraversion scale measure the true level of
each participant’s extraversion (over time)?
Are coders following the same method?
 To achieve reliability we want to reduce random
error.
© 2015 M. Guthrie Yarwood
Threats to Reliability (random error)
 Participant Characteristics:
 fatigue, motivation, boredom
 Testing situation:
 time of day, room temperature
 Testing Instrument:
 instructions, rating scale, items, reading level.
 Experimenter Characteristics and Errors:
 interactions with participants; incorrect observations of
participants; incorrect coding of behavior
© 2015 M. Guthrie Yarwood
Estimating Reliability
 Test-Retest Coefficient
 Parallel-Forms Coefficient
 Internal Consistency Coefficient
 Interrater (interobserver) Reliability
© 2015 M. Guthrie Yarwood
Validity
 Validity: The degree to which a test or measurement
accurately measures or reflects what it claims to
measure.
 Internal Validity: Did the experimental treatments
make a difference in this specific experimental
instance?
 External Validity: Generalizability; To what
populations, settings, treatment variables, and
measurement variables can the effect be generalized?

**Never completely answerable
© 2015 M. Guthrie Yarwood
Internal Validity
 Degree to which test measures what it claims to
measure
 5 types of internal validity





Face validity
Predictive or criterion validity
Convergent validity
Discriminant validity
Construct validity
© 2015 M. Guthrie Yarwood
Threats to Internal Validity (errors)
 Affected by random and constant errors

Random (unsystematic) Errors: same errors that affect
reliability

Constant Errors (systematic): errors that affect measurement
of variable; does not affect reliability
© 2015 M. Guthrie Yarwood
Knowledge Check!
You are conducting a study on the
personality traits associated with the
frequency of exercising.
For your study, which of the following poses a
threat to validity, but not reliability?
A. All the participants are bored.
B. The construction outside the laboratory window
is very loud.
C. You recruited participants from the Rec Hall.
D. The experimenter who greets all participants is
very rude.
© 2015 M. Guthrie Yarwood
Knowledge Check!
You are conducting a study on the personality
traits associated with the frequency of exercising.
For your study, which of the following poses a
threat to validity, but not reliability?
A. All the participants are bored.
B. The construction outside the laboratory window
is very loud.
C. You recruited participants from the Rec
Hall.
D. The experimenter who greets all participants is
very rude.
© 2015 M. Guthrie Yarwood
Threats to Internal Validity – MRS SMITH
 Maturation
 Regression to the Mean
 Selection of Subjects
 Selection by Maturation Interaction
 Mortality
 Instrumentation
 Testing
 History
© 2015 M. Guthrie Yarwood
Regression to the Mean Example
 In an experiment involving reading instruction,
subjects grouped because of poor pre-test reading
scores show considerably greater gain than do the
groups who scored average and high on the pre-test.
© 2015 M. Guthrie Yarwood
Regression to the Mean Example
 In an experiment involving reading instruction,
subjects grouped because of poor pre-test reading
scores show considerably greater gain than do the
groups who scored average and high on the pre-test.
Pre-test
Poor
© 2015 M. Guthrie Yarwood
Average /
Mean
High
Regression to the Mean Example
 In an experiment involving reading instruction,
subjects grouped because of poor pre-test reading
scores show considerably greater gain than do the
groups who scored average and high on the pre-test.
Pre-test
Poor
After reading
Instruction
© 2015 M. Guthrie Yarwood
Average /
Mean
High
Selection by Maturation Interaction
Group
Pre-test
Head Start
Intervention
Middle-class children
65/100
Disadvantaged children
65/100
Control Group – No
Intervention
Middle-class children
65/100
Disadvantaged children
65/100
© 2015 M. Guthrie Yarwood
Post-Test (after head
start)
Selection by Maturation Interaction
Group
Pretest
Head Start
Intervention
Control Group – No
Intervention
© 2015 M. Guthrie Yarwood
Post-Test:
6 months
Post-Test:
12 months
Post-Test:
18 months
65/100 70/100
75/100
80/100
65/100 65/100
65/100
65/100
Selection by Maturation Interaction
Group
Pretest
Head Start
Intervention
Control Group – No
Intervention
Post-Test:
6 months
Post-Test:
12 months
Post-Test:
18 months
65/100 70/100
75/100
80/100
65/100 65/100
65/100
65/100
Ss in Intervention are middle class, while Ss in control group are
disadvantaged. Over time, Intervention Ss show improvement of posttest due to better health care, greater parental support, greater access to
resources, etc.
© 2015 M. Guthrie Yarwood
Identify the Threat to Validity!
 In a short experiment designed to investigate the effect of
computer-based instruction, Ss missed some instruction
because of a power failure at school.
A. History
B. Mortality
C. Testing
D. Instrumentation
© 2015 M. Guthrie Yarwood
Identify the Threat to Validity!
 In a short experiment designed to investigate the effect of
computer-based instruction, Ss missed some instruction
because of a power failure at school.
A. History
B. Mortality
C. Testing
D. Instrumentation
© 2015 M. Guthrie Yarwood
Identify the Threat to Validity!
 In a health experiment designed to determine the effect of
various exercises, those Ss who find the exercise most
difficult stop participating.
A. Selection of Subjects
B. Mortality
C. Testing
D. Maturation
© 2015 M. Guthrie Yarwood
Identify the Threat to Validity!
 In a health experiment designed to determine the effect of
various exercises, those Ss who find the exercise most
difficult stop participating.
A. Selection of Subjects
B. Mortality
C. Testing
D. Maturation
© 2015 M. Guthrie Yarwood
Estimating Internal Validity
 Content Validity
 Criterion-Related Validity
 Concurrent Validity
 Predictive Validity
 Construct Validity
 Convergent Validity
 Discriminant Validity
© 2015 M. Guthrie Yarwood
Content Validity
 Definition: whether the content of a test elicits a
range of responses that are representative of the
entire domain or universe of skills, understandings,
and other behaviors a test is designed to measure.
 To assess: compare tests’ content with an outline of
specifications concerning subject matter to be
covered in test.
© 2015 M. Guthrie Yarwood
?????
?????
Openness to
Experience
?????
?????
© 2015 M. Guthrie Yarwood
Behaviors?
Thoughts/Cognitions?
Openness to
Experience
Feelings/Emotions?
Perceptions?
© 2015 M. Guthrie Yarwood
Criterion-Related Validity
 Definition: procedures in which the test scores of
a group of people are compared with ratings,
classifications, or other measures of performance.
© 2015 M. Guthrie Yarwood
Concurrent Validity –
A type of criterion-related validity
 Concurrent Validity: when a test is
administered to people in various categories, to
determine whether test scores of people in 1
category are significantly different from people in
other categories.


Clinical vs. non-clinical group
Different socioeconomic levels
© 2015 M. Guthrie Yarwood
Concurrent Validity
Group A shows more
openness to experience
Group B shows less
openness to experience
© 2015 M. Guthrie Yarwood
Concurrent Validity
High IQ group
shows more openness to
experience
Low IQ group shows less
openness to experience
© 2015 M. Guthrie Yarwood
Predictive Validity –
A type of criterion-related validity
 Predictive Validity: how accurately test scores
predict criterion scores.
 Indicated by correlation between test score (the
predictor) and a criterion of future performance
(what the test predicts)
© 2015 M. Guthrie Yarwood
Predictive Validity
Openness to
Experience
(Predictor)
© 2015 M. Guthrie Yarwood
?????
(Criterion)
Predictive Validity
Openness to
Experience
College Major
Openness to
Experience
Well-being,
Psychological Adjustment
© 2015 M. Guthrie Yarwood
Construct-Related Validity
 Definition: extent to which scale measures a
particular construct or psychological concept
 To assess: need to determine whether an
assessment instrument that presumably measures a
certain personality variable is actually doing so
© 2015 M. Guthrie Yarwood
Convergent Validity –
A Type of Construct-Related Validity
 Convergent Validity: the measure has high
correlations with other measures or methods of
measuring the same construct
© 2015 M. Guthrie Yarwood
Convergent Validity
Our Openness to
Experience
Self-Report Scale
Our Openness to
Experience
Self-Report Scale
© 2015 M. Guthrie Yarwood
Same Construct,
Same Measurement
Same Construct,
Different Measurement
Convergent Validity
Our Openness to
Experience SelfReport Scale
Our Openness to
Experience SelfReport Scale
© 2015 M. Guthrie Yarwood
Costa & McCrae’s
Self-Report OE Dimension
Observer-Report
OE Dimension
Discriminant Validity –
A Type of Construct-Related Validity
 Discriminant Validity: the measure has low
correlations with measures of different constructs
© 2015 M. Guthrie Yarwood
Discriminant Validity
Our Openness to
Experience
Self-Report Scale
Different Construct,
Same Measurement
OR
Our Openness to
Experience
Self-Report Scale
© 2015 M. Guthrie Yarwood
Different Construct
Different Measurement
Discriminant Validity
Our Openness to
Experience
Self-Report Scale
Our Openness to
Experience
Self-Report Scale
© 2015 M. Guthrie Yarwood
Self-Report
Sensation Seeking Scale
Clinical Diagnosis of
Schizotypal
PersonalityDisorder
Discriminant Validity
Our Openness to
Experience
Self-Report Scale
Self-Report
Sensation Seeking Scale
These correlations should
be significant, but lower
than correlations for
convergent validity.
Our Openness to
Experience
Self-Report Scale
© 2015 M. Guthrie Yarwood
Clinical Diagnosis of
Schizotypal
PersonalityDisorder
To assess convergent/discriminant validity
 The same construct using the same method (c)
 The same construct using different methods (c)
 Different constructs using the same method (d)
 Different constructs using different methods (d)
 Note: c = convergent; d=discriminant
© 2015 M. Guthrie Yarwood
Reliability & Validity
 A test can be reliable, but not valid
 In other words, to be valid a test must first be
reliable.
 NO VALDITY WITHOUT RELIABILITY!
© 2015 M. Guthrie Yarwood
Reliability and Validity: Summary
Internal Validity
Reliability
 Test-Retest Coefficient
 Parallel-Forms Coefficient
 Internal Consistency
Coefficient
 Interrater (interobserver)
Reliability
© 2015 M. Guthrie Yarwood
 Content Validity
 Criterion-Related
Validity


Concurrent Validity
Predictive Validity
 Construct Validity
 Convergent Validity
 Discriminant Validity
A LITTLE MORE ON OBSERVER
REPORTS (O-DATA)
© 2015 M. Guthrie Yarwood
Correlation • Self-report
• Observer report 1
1
Correlation • Observer report 1
• Observer report 2
2
Correlation • Self-report
• Several different observers
3, 4, 5…
© 2015 M. Guthrie Yarwood
Aggregating Scores
 Averaging the self-report and the observer report/s
provides a clearer picture of personality than the self
or observer report alone (Kolar et al., 1996)
© 2015 M. Guthrie Yarwood
Let’s discuss the purpose of observer reports
 What is the purpose of an observer report?
 Compared to a self-report measure, what results
would you expect from an observer report?
 What would you think if you did not obtain these
results?
 Can you think of any threats to validity that observer
reports may pose?
© 2015 M. Guthrie Yarwood
Aggregating Scores
 A positive correlation between an observer report
and a self-report, would provide evidence for which
type of validity?
A. Convergent
B. Predictive
C. Concurrent
D. Discriminant
© 2015 M. Guthrie Yarwood
Recall Reflection Post and TIPI!
1
Extraverted,
enthusiastic
2
Reserved, quiet r
3
Critical,
quarrelsomer
4
Sympathetic, warm
5
Dependable, self-
disciplined
6
Disorganized,
carelessr
7
Anxious, easily
upset r
8
Calm, emotionally
stable
9
Open to new
experiences,
complex
10
Conventional,
© 2015 M. Guthrie
Yarwood
r
uncreative
Disagree
Disagree
Disagree a
Strongly
Moderately
little
(1)
(2)
(3)
Neither
Agree nor
Disagree
(4)
Agree a
Agree
Agree
little
Moderately
Strongly
(5)
(6)
(7)
Self Scores!
7
6
5.71
5.54
5.13
5
4
4.46
3.96
3
2
1
Extraversion
Agreeableness
Conscientiousness
TIPI Dimensions
© 2015 M. Guthrie Yarwood
Emotionally
Stable
Openness to
Experience
Observer Scores!
7
6
4.92
5
5.25
5.13
4.38
4.29
4
3
2
1
Extraversion
Agreeableness Conscientiousness
TIPI Scores
© 2015 M. Guthrie Yarwood
Emotionally
Stable
Openness to
Experience
 Based on the following graphs, which dimension do
you think shows the weakest correlation between self
and observer scores?
A. Extraversion
B. Agreeableness
C. Conscientiousness
D. Emotional Stability
E. Openness to Experience
© 2015 M. Guthrie Yarwood
AGREEABLENEESS
7
6
5
4
3
2
1
1
2
3
4
5
6
7
SELF SCORE
OBSERVER SCORE
OBSERVER SCORE
EXTRAVERSION
7
6
5
4
3
2
1
1
2
OBSERVER SCORE
OBSERVER SCORE
7
5
4
3
2
1
3
4
5
6
7
SELF SCORE
OBSERVER SCORE
6
7
7
6
5
4
3
2
1
1
2
3
4
SELF SCORE
Openness to Experience
7
6
5
4
3
2
1
1
2
3
4
SELF SCORE
© 2015 M. Guthrie Yarwood
5
Emotional Stability
LOW NEUROTICISM
6
2
4
SELF SCORE
Conscientiousness
1
3
5
6
7
5
6
7
 Based on the following graphs, which dimension do
you think shows the weakest correlation between self
and observer scores?
A. Extraversion
B. Agreeableness
C. Conscientiousness
D. Emotional Stability
E. Openness to Experience
© 2015 M. Guthrie Yarwood
LET’S CORRELATE THE SCORES!
TIPI DIMENSION
1
EXTRAVERSION
r
Extraverted, enthusiastic
.74
Reserved, quiet r
2
3
TIPI TRAIT
AGREEABLENESS
Critical, quarrelsomer
4
Sympathetic, warm
5
Dependable, self-disciplined
CONSCIENTIOUSNESS
6
Disorganized, carelessr
7
Anxious, easily upset r
8
(LOW NEUROTICISM)
9
OPENNESS TO
10
© 2015 M. Guthrie Yarwood
EMOTIONAL STABILITY
Calm, emotionally stable
.48
.52
.07
Open to new experiences,
EXPERIENCE
complex
Conventional,
uncreativer
.39
AGREEABLENESS
7
7
6
6
SCORE
SCORE
EXTRAVERSION
5
4
3
5
4
3
2
2
1
1
1
2
3
4
5
6
7
8
9
10
11
1
12
2
4
5
6
7
8
9
10
11
12
STUDENT
STUDENT
EMOTIONAL STABILITY
(LOW NEUROTICISM)
CONSCIENTIOUSNESS
7
7
6
6
5
SCORE
SCORE
3
4
3
2
5
4
3
2
1
1
1
2
3
4
5
6
7
8
STUDENT
9
10
11
12
1
2
3
4
5
6
OPENNESS TO EXPERIENCE
7
SCORE
6
5
4
3
2
1
1
Note. Red = Self; Yellow = Observer
2
3
4
© 2015 M. Guthrie Yarwood
5
6
7
STUDENT
8
9
10
7
STUDENT
11
12
8
9
10
11
12
AGREEABLENESS
7
7
6
6
SCORE
SCORE
EXTRAVERSION
5
4
3
5
4
3
2
2
1
1
1
2
3
4
5
6
7
8
9
10
11
1
12
2
4
5
6
7
8
9
10
11
12
STUDENT
STUDENT
EMOTIONAL STABILITY
(LOW NEUROTICISM)
CONSCIENTIOUSNESS
7
7
6
6
5
SCORE
SCORE
3
4
3
2
5
4
3
2
1
1
1
2
3
4
5
6
7
8
STUDENT
9
10
11
12
1
2
3
4
5
6
OPENNESS TO EXPERIENCE
7
SCORE
6
5
4
3
2
1
1
Note. Red = Self; Yellow = Observer
2
3
4
© 2015 M. Guthrie Yarwood
5
6
7
STUDENT
8
9
10
7
STUDENT
11
12
8
9
10
11
12
Summary and Evaluation
 Decisions about data source and research design
depend on


(1) the purpose of study and
(2) threats to validity/reliability
 There is no perfect data source
 There is no perfect research design
 Assessing threats to reliability and validity will assist
in selecting a data source and research design.
 Observer reports improve validity and reduce social
desirability concerns
© 2015 M. Guthrie Yarwood
Reminder – Paper Topic!!
 Find a group member (can do on ANGEL!)
 Select a topic!
 Submit Paper topic to drop-box
 Due: Friday, January 30th at 9 AM ET
 Paper Topic Outline available on ANGEL.
 Guidance: Meet with Michelle or Celina
 Questions??? 
© 2015 M. Guthrie Yarwood
Chapter 3 (Dispositional Domain) Survey
 HEXACO
 Follow link to access survey
 Score your survey according to the instructions on ANGEL
 Big Five Model
 Access through Chapter 3 Survey on course website
© 2015 M. Guthrie Yarwood