Reliability
Definition: The stability or consistency of a test.
Assumption: True score = obtained score +/- error.
Domain Sampling Model
Item Domain
Test
Reliability Techniques
4. Internal Consistency
1. Test-retest
T1
A
T2
K-R-20
A
1
2
2. Parallel Forms
A
T
Score
B
Score
3. Split-half
1
50 pairs
100
Coefficient
Alpha
Types of Validity
Content Validity [the extent to which test
items represent the domain]
a) Subject Matter Expert Opinions (e.g.,
CVR statistic)
b) Internal consistency reliability
c) Correlation with other similar tests
Types of Validity (cont.)
Criterion-related Validity
Predictive –
Concurrent --
[Correlation between test
scores of applicants and their
performance scores when
some time interval has passed
after they are hired]
[Correlation between test scores
and performance scores of
current employees]
• Range restriction issue on
performance scores
• Time, cost, & pragmatic
concerns
• Job experience factor
• Motivation level
• Guessing, Faking
• Range restriction issue on
performance scores
Types of Validity (cont.)
Construct Validity [the extent to which the test assesses the
construct it intends to measure]
• Correlation between scores measuring a construct (e.g.,
anxiety) with one method (e.g., paper & pencil) with scores on
the same construct using a different method (e.g., interview)
[Convergent validation]
• Correlation between scores measuring a construct (e.g.,
anxiety) using one method (e.g., paper & pencil) with scores
on a different construct (e.g., leadership) assessed with a
different method (e.g., interview) [Discriminant validation]
Types of Selection Tests
• Application Blanks (Weighted Application Blanks,
Biographical Information Blanks)
• Honesty (Integrity) Tests, Drug Tests
• Interviews (e.g., Situational)
• Aptitude Tests (e.g., Mechanical, Clerical)
• Work Samples
• Assessment Centers (Situational Exercises)
Types of Selection Tests (cont).
•Personality Inventories
Self-reports -(e.g., Minnesota Multiphasic
Personality Inventory (MMPI),
California Psychological Inventory
(CPI), Myers-Briggs Type
Indicator (MBTI), Personality
Inventory (HPI), NEO PI-R
NEO (assesses the 5-Factor
Model consisting of: Neuroticism,
Extraversion Openness,
Agreeableness Conscientiousness)
Projective Techniques -(e.g., Thematic
Apperception Test (TAT),
Rorschach Inkblot Test
(RIT)
Weighted Application Blank (WAB)
Job Performance
Absenteeism
Develop test
items
Low
r
_______
_______
_______
_______
r
Select items
that relate to
criterion (e.g.,
absenteeism
High
_______
Correlate items with
performance
Weight items according to
their relationship to job
performance
Weighted Application Blank (WAB)
Job Performance
Absenteeism
Develop test
items
Low
r
_______
_______
_______
_______
r
Select items
that relate to
criterion (e.g.,
absenteeism
High
_______
Correlate items with
performance
Weight items according to
their relationship to job
performance
Frequency of Common Inappropriate Application Blank Questions
Worded
Appropriately
Not Asked
Item
Not
Appropriate
Past salary
98.9%
0%
1.1%
Minimum acceptable salary
72.7%
0%
27.2%
Reference source
59.1%
0%
40.9%
Age
54.5%
37.5%
8.0%
relatives
50.0%
10.2%
39.8%
Conviction records
43.2%
28.4%
28.4%
Health
40.9%
2.3%
56.8%
Military service
30.7%
30.7%
38.6%
Marital status
27.3%
0%
72.7%
25.0%
43.2%
31.8%
residence
23.9%
0%
76.1%
Physical description, photo
19.3%
0%
80.7%
Rent or own car or home
18.2%
0%
81.8%
Handicap
17.0%
6.8%
76.2%
Organizational membership
15.9%
21.6%
12.5%
Work schedule
13.6%
63.6%
22.7%
Information about
Who to notify in case
of emergency
Length of time in
Sample Biographical Information Blank Items
• During high school, how many times did you make the honor roll?
• How much freedom or independence did your parents allow you in grade school?
• How important did your favorite high school teachers stress discipline in the
classroom?
• How many times did you change schools before you were sixteen years old?
• Compared to other people in high school, how many friends did you have?
• How old were you when you spent your first week (or more) away from your
parents?
• How bothered are you if you a job is left undone?
• How often do you read craft and mechanics magazines?
• How quickly do you normally work?
• How well do you feel you can understand the feelings of others?
• How well do you tolerate performing routine tasks?
Honesty Testing
Types of Questions:
1)
Frequency and extent of theft (e.g., What percentage of people take more than $1.00
per week from their employer?)
2)
Punitiveness toward theft (e.g., should a person be fired if caught stealing $5.00?)
3) Thoughts about theft (e.g., Have you ever thought about taking company merchandise
without actually taking any?)
4)
Perceived ease of theft (e.g., How easy would it be for a dishonest person to steal
from an employer?)
5) Likelihood of detection (e.g., What percent of employees thieves are ever caught?
Validity Issues:
a) Correlations with polygraph results
b) Future behavior (e.g., # days with cash shortage, discharges)
c) Admissions of past theft
d) Shrinkage reduction
e) Contrasted groups (e.g., scores by criminals vs. general population scores)
Employment Interview
• Frequently used to make selection decisions (over 90%)
• Social exchange (interpersonal) process
• Search for information
COMMON PROBLEMS WITH THE “TRADITIONAL” INTERVIEW
•Variety of Interviewer Biases
* 1st Impressions
* Expectancy Effect
* Contrast Effect
* Stereotype Matching
•Different Questions Asked to Applicants
(Lack of standardization)
•Disagreement on the Desirability of Interview Responses
•Little Formal Interviewer Training
•Subjective (or no) Scoring System
•Interview Conducted and Scored by One Person
•Poor Reliability, Validity, and Job Relevancy
(Open to Legal Challenge
Overview of Situational Interview Process
1)
Perform a Job Analysis Using the Critical Incident Technique
2)
Place Critical Incidents into Relevant Job Dimensions (e.g., Safety, Responsibility,
Interpersonal Skills)
3)
Reword Critical Incidents Into Question Format
Incident: The employee was married for a year and a half and used any excuse to
stay home. One day the employee’s children got colds and no one was around to care
for them. So, the employee didn’t show up for work and didn’t phone in.
Question: Your two teenage children are home in bed sick with colds. No friends or
relatives are available to watch them. Your shift starts in two hours. What would you
do in this situation?
4)
Decide on the desirability of responses [Think of how good, average, and mediocre workers
would have answered such a question]
_____
1
Stay home
_____
_____
_____
3
Phone in &
explain the problem
_____
5
Go in, they just
have colds
5) Conduct interviews in groups of two or more. Each interviewer scores applicant
independently. A single score is given after group discussion
Differential Aptitude Test (DAT)
Verbal Reasoning
..... is to night as breakfast is to …..
..… is to water as eat is to …..
A.
B.
C.
D.
E.
continue ----- drive
foot ----- enemy
drink ----- food
girl ----- industry
drink ----- enemy
….. is to one as second is to …..
A.
B.
C.
D.
E.
two ----- middle
first ----- fire
queen ----- hill
first ----- two
rain ----- fire
A.
B.
C.
D.
E.
supper ----- corner
gentle ----- morning
door ----- corner
flow ----- enjoy
supper ----- morning
Differential Aptitude Test (DAT)
Numerical Ability
Add
30
20
A. 8
B. 15
C. 16
D. 26
N. none of these
Add
13
12
A. 14
B. 16
C. 25
D. 59
N. none of these
Differential Aptitude Test (DAT)
Abstract Reasoning
PROBLEM FIGURES
ANSWER FIGURES
A
B
C
D
E
Differential Aptitude Test (DAT)
Abstract Reasoning (cont.)
PROBLEM FIGURES
ANSWER FIGURES
A
B
C
D
E
Differential Aptitude Test (DAT)
Mechanical Reasoning
A
B
Which weighs more?
(If equal, mark C.)
Differential Aptitude Test (DAT)
Space Relations
A
B
C
D
Differential Aptitude Test (DAT)
Spelling
A) man
B) Gurl
C) Catt
D) dog
Language
Usage
1) I just / left / my friends / house.
A
B
C
D
2) Ain’t we / going to / the office / next week?
A
B
C
D
3) I went / to a ball / game with / Jane.
A
B
C
D
Differential Aptitude Test (DAT)
Clerical Speed and Accuracy
V. AB AC AD AE AF
W. aA aB
BA Ba Bb
X. A7 7A
B7 7B AB
Y. Aa Ba
bA BA bB
Z. 3A 3B
33
B3 BB
Work Sample Tests Performing (performing a piece of the job
-- Assessment Center Process -Job Candidates
Rate
candidates on
performance
dimensions
Assessors (evaluators)
Sample dimensions
Decisiveness, Initiative,
Judgment, Problem analysis,
Oral communication,
Planning, Leadership
Exercises
Observe and
record candidates
behavior in
exercises
Sample Individual Exercises
Sample Group Exercises
Interview Simulation
Leaderless Group Discussion
Scheduling Exercise
Business Game
In-Basket
Discuss and
give final
ratings to
candidates
Self-Report Inventories
Hogan Personality Inventory
Primary Scales ---
Adjustment (confidence, self-esteem, composure under stressful
situations)
Ambition (competitive, possessing initiative, potential for leadership)
Sociability (extraverted, friendly, enjoys social interactions)
Likeability (warm, charming, capable of maintaining relationships)
Prudence (responsible, possessing self-control, conscientious)
Intellectance (imaginative, curious, creative)
School Success (achievement orientation, keeps current of business
and technical developments)
Hogan Personality Inventory (cont.)
Occupational Scales ---
Service Orientation (attentive, pleasant, courteous to others
such as customers and clients)
Stress Tolerance (ability to handle stress)
Reliability (integrity, good organizational citizen)
Clerical Potential (able to follow directions, attentive to detail,
clear communicator)
Sales Potential (energetic, ability to interact socially, able to
deal with client issues/problems)
Managerial Potential (leadership ability, good at organizing,
capability to make decisions)
Self-Report Inventories (cont.)
Myers-Briggs Type Indicator
Introversion
Extraversion
(Internally motivated; enjoys spending time
prefers one-to-one communication)
(Action-oriented; motivated enjoys alone;
by outside world and social
interactions)
Sensing
Intuitive
(Desires concrete information; practical;
in orientation)
(Imaginative, creative factual and detailed
improvises comfortable
with ambiguity; focuses
on contexts and connections
Thinking
Feeling
Judging
Perceiving
Rorschach Inkblot Test
Ten cards which bilateral and symmetrical inkblots
Scoring --Location: the part of the blot used (e.g., use of the
whole blot, common or unusual detail)
Determinants: form, color, shading, and movement)
Content (human figures, animal figures, anatomical
diagrams, inanimate objects
Thematic Apperception Test
TAT --- 31 pictures that depict a variety of social and interpersonal
situations. Participants are requested to write or tell a story about each
picture to the examiner (e.g., what happened, what Ten pictures are
gender-specific; the others can be used with either sex.
Use: To uncover internal conflicts, dominant drives, interests, and
motives. Specific motives include the need for achievement, need for
power, the need for intimacy, and problem-solving abilities.
Types of Selection Tests (cont).
Ability Tests
Sensory (e.g.,
hearing, vision)
Motor (e.g.,
dexterity,
strength, agility)
Cognitive (e.g.,
Intelligence)