Reliability
Definition: The stability or consistency of a test.
Assumption: True score = obtained score +/- error.
~ Domain Sampling Model ~
Item Domain
Test
Test-retest method
[error is due to changes occurring due to the passage of time]
Test 1
-------------
r
Test 1
-------------
Issues:
•
Length of time between test administrations if crucial (generally, the
longer the interval, the lower the reliability)
• Memory
• Stability of the construct being assessed
• Speed tests, sensory discrimination, psychomotor tests (possible fatigue
factor)
Parallel/Alternate Forms
[error due to test content and perhaps passage of time]
Test 1
-------------
r
Test 2
-------------
Issues:
• Need same number & type of items on each test
• Item difficulty must be the same on each test
• Variability of scores must be the same on each test
Two types:
1) Immediate (back-to-back administrations)
2) Delayed (a time interval between administrations)
KR-20 and Coefficient Alpha [error due to item similarity]
• KR-20 is used with scales that have right & wrong
responses (e.g., achievement tests)
• Alpha is used for scales that have a range of response
options where there are no right or wrong responses (e.g.,
7-point Likert-type scales
KR-20
Rtt = k
k–1
% of those getting
the item correct
Σ pi (1 – pi)
σy
2
# of items variance of test scores
variance of scores
on each item
Alpha
ά = k
1 – Σ σ i2
k–1
# of items
σ y2
variance of
test scores
Factors Affecting Reliability
1) Variability of scores (generally, the more variability, the
higher the reliability)
2) Number of items (the more questions, the higher the
reliability)
3) Item difficulty (moderately difficult items lead to higher
reliability, e.g., p-value of .40 to .60)
4) Homogeneity/similarity of item content (e.g., item x
total score correlation; the more homogeneity, the higher
the reliability)
5) Scale format/number of response options (the more
options, the higher the reliability)
Types
ofof
Validity
Types
Validity
Content Validity [the extent to which test
items represent the domain]
a) Subject Matter Expert Opinions (e.g.,
CVR statistic)
b) Internal consistency reliability
c) Correlation with other similar tests
Types of Validity (cont.)
Criterion-Related Validity
[correlation between test scores and job
performance scores
Predictive –
Concurrent --
[Correlation between test
scores of applicants and their
performance scores when
some time interval has passed
after they are hired]
[Correlation between test scores
and performance scores of
current employees]
• Range restriction issue on
performance scores
• Time, cost, & pragmatic
concerns
• Job experience factor
• Motivation level
• Guessing, Faking
• Range restriction issue on
performance scores
Types of Selection Tests
• Application Blanks (Weighted Application Blanks,
Biographical Information Blanks)
• Honesty (Integrity) Tests, Drug Tests
• Interviews (e.g., Situational)
• Aptitude Tests (e.g., Mechanical, Clerical)
• Work Samples
• Assessment Centers (Situational Exercises)
Application Blanks
• Content of items (use of job analysis)
• Number of application blanks (one for each position or job category)
• Legal issues
• Image of organization (e.g., format, recruitment issue, perceived fairness
of questions)
• Accuracy of data
Education (e.g., 21%), salary (22%), job title (24%), years worked (29%)
College students willing to include one lie on AB (95%); 45% had done so
Yahoo CEO Scott Thompson
caught padding his resume
Yahoo director Patti Hart, who
was in charge of the
company's CEO search, will
not seek re-election at the next
shareholder meeting. An exact
date for the meeting has not
been set, but last year's event
was held in June. Hart, who
joined Yahoo's board in 2010, is
the CEO of International
Game Technology Statements
from both Yahoo and IGT said
that IGT's board asked her not
to seek re-election at Yahoo.
Thompson stepping down as
Yahoo CEO (May 13th,
Washington Post) Yahoo chief
executive Scott Thompson,
hired from eBay in January to
turn around the struggling Web
portal, is stepping down after it
was discovered that his
academic credentials were
misrepresented, according to a
person with knowledge of the
matter
In a series of published biographical statements stretching back for years – including his bio
on Yahoo’s website ---- Thompson has said that he "holds a Bachelor's degree in accounting
and computer science" from Stonehill College. But his degree is actually only in
accounting … Yahoo sent out a statement saying references to Thompson earning a
computer science degree were an "inadvertent error.” … It's an error Thompson made
repeatedly. References to Thompson's nonexistent computer science degree are featured in
his bios on sites for PayPal, the eBay subsidiary where he previously served as president.
Frequency of Common Inappropriate Application Blank Questions
Worded
Appropriately
Not Asked
Item
Not
Appropriate
Past salary
98.9%
0%
1.1%
Minimum acceptable salary
72.7%
0%
27.2%
Reference source
59.1%
0%
40.9%
Age
54.5%
37.5%
8.0%
relatives
50.0%
10.2%
39.8%
Conviction records
43.2%
28.4%
28.4%
Health
40.9%
2.3%
56.8%
Military service
30.7%
30.7%
38.6%
Marital status
27.3%
0%
72.7%
25.0%
43.2%
31.8%
residence
23.9%
0%
76.1%
Physical description, photo
19.3%
0%
80.7%
Rent or own car or home
18.2%
0%
81.8%
Handicap
17.0%
6.8%
76.2%
Organizational membership
15.9%
21.6%
12.5%
Work schedule
13.6%
63.6%
22.7%
Information about
Who to notify in case
of emergency
Length of time in
Years of experience and
previous salary are the
strongest predictors of
starting salary, and starting
salary is the greatest
predictor of current salary.
--- Mickey Silberman,
Jackson Lewis, LLP
Industry Liason Conference
(2011)
Other possible
concerns include:
personal email,
personal web page,
and/or Facebook
accounts
Effect of Name on Resumes and
Interview Rates
Name type
Resume Quality
Low
High
“White”
sounding
name
“Black”
sounding
name
50% less chance of
being invited for
an interview
versus “Whites”
with high
qualifications
(Bertrand & Mullainathan; 2004)
Social Media and Selection
Frequency of Use --•18% indicated that they have used social networking websites to screen
applicants, while 11% planned on using such sites in the future (survey of
over 400 organizations by the Society for Human Resource Management in 2011)
• 45% of employers used social networking sites to investigate job applicants
(survey of over 2,600 hiring managers conducted by Harris Interactive for
CareerBuilder.com)
Consequences?
• 35% of organizations in Harris survey said they did not hire candidates
Protection Act (2012)
due to content available on social networking sites.Password
Most common
examples
•
of negative information included
Provocative attire
Images of drug or alcohol use,
Complaints about previous employers
Password Protection Act (2012)
• Prohibit an employer from forcing prospective or
current employees to provide access to their own
private account as a condition of employment
• Prohibit employers from discriminating or
retaliating against a prospective or current employee
because that employee refuses to provide access to a
password-protected account.
~ Honesty
Testing ~
Two Types
1) Overt integrity tests
2) Covert (Honesty test questions imbedded within personality measures,
e.g., among items assessing drug use/attitudes, tendency toward violence)
~ Honesty Testing ~
Types of Question (Overt):
1)
Frequency and extent of theft (e.g., What percentage of people take more than $1.00 per week
from their employer?)
2)
Punitiveness toward theft (e.g., should a person be fired if caught stealing $5.00?)
3)
Thoughts about theft (e.g., Have you ever thought about taking company merchandise without
actually taking any?)
4)
Perceived ease of theft (e.g., How easy would it be for a dishonest person to steal from an
employer?)
5) Likelihood of detection (e.g., What percent of employee thieves are ever caught?
Other types of questions:
•
Do you think you are too honest to take something that is not yours?
•
Do you believe most employers take advantage of the people who work for them?
•
How often do you tell the truth?
•
I have frequently associated with fellow employees who admitted they were stealing
merchandise from the company.
Validity Issues: a) Correlations with polygraph results, b) Future behavior (e.g., # days with cash
shortage, discharges), c) Admissions of past theft, d) Shrinkage reduction, Contrasted groups
(e.g., scores by criminals vs. general population scores)
Honesty Testing --- Types of Questions (cont.)
Covert Tests (tests about one’s attitudes thought to relate to honesty)
• It would be better if almost all laws were thrown away
[Conventionality, traditionalism]
• Sometimes I am strongly attracted by the personal articles of others
[Self-discipline]
• I like to take chances [Sensation seeking]
• On average, how often during the week do you go to parties?
[Sensation seeking]
• Does it bother you when people ask stupid questions? [Forgiveness]
• How often do you make your bed? [Tidiness]
Drug Testing
• Approximately 57% of companies use pre-employment drug
testing (Society for Human Resource Management survey (2011)
• What does a positive drug test score indicate?
Some contend that drug use does not negatively affect
performance in the majority of jobs (Macdonald, 1997)
Other research indicates that drug use is related to poor job
performance (e.g., increased injuries and involuntary turnover)
(Bass et al., 1996; Normand, Salyards, & Mahoney, 1990).
Drug Testing
Some Issues:
• Those being tested
Applicants, employees or both
• Testing procedure
Random or for cause
• Type of company
Public or private
• Type of test
Sensitivity, Cross-reactivity
(TLC, EIA/RIA, GCMS)
• Type of job
Safety concerns or not
~ Reference Checks ~
(Exceptionally common technique; e.g., 95% usage by organizations)
In-Person (e.g., interview)
• Costly, time consuming
• Used in jobs that involve the concern for risks (e.g., security, $)
• Can elicit different types of information (differences
between in-person and written reference information)
Mail (or e-mail)
• Low return rate using “snail” mail (e.g., 56 – 64%)
• Standardized questions, format
• Written record of responses
• Ensure confidentiality of responses (signed statement by applicant)
~ Telephone Checks ~
(More frequently used than written references)
• Allows follow-up or clarification of answers given
• Less resistance to giving certain types of information can be collected
• Quick process
• Important data can be gleaned from various verbal cues (e.g., pauses,
hesitations, voice inflections, voice level, intonations)
• Relatively high return rate
• Better responsiveness, more interactive nature of the method
• More confidence in the identity of responder
Sample Biographical Information Blank Items
• During high school, how many times did you make the honor roll?
• How much freedom or independence did your parents allow you in grade school?
• How important did your favorite high school teachers stress discipline in the
classroom?
• How many times did you change schools before you were sixteen years old?
• Compared to other people in high school, how many friends did you have?
• How old were you when you spent your first week (or more) away from your
parents?
• How bothered are you if you a job is left undone?
• How often do you read craft and mechanics magazines?
• How quickly do you normally work?
• How well do you feel you can understand the feelings of others?
• How well do you tolerate performing routine tasks?
Classification of Biographical Items
~ Employment Interview ~
• Frequently used to make selection decisions (over 90%)
• Social exchange (interpersonal) process
• Search for information
COMMON PROBLEMS WITH THE “TRADITIONAL” INTERVIEW
• Variety of Interviewer Biases
* 1st Impressions
* Expectancy Effect
* Contrast Effect
* Stereotype Matching
• Different Questions Asked to Applicants
(Lack of standardization)
• Disagreement on the Desirability of Interview Responses
• Little Formal Interviewer Training
• Subjective (or no) Scoring System
• Interview Conducted and Scored by One Person
• Poor Reliability, Validity, and Job Relevancy (Open to Legal
Challenge)
~ Summary of Situational Interview Process ~
•
Perform a Job Analysis Using the Critical Incident Technique
•
Place Critical Incidents into Relevant Job Dimensions (e.g., Safety,
Responsibility, Interpersonal Skills)
•
Reword Critical Incidents Into Question Form
•
Decide on the desirability of responses [Think of how good,
average, and mediocre workers would have answered such a
question]
•
Conduct interviews in groups of two or more. Each interviewer
scores applicant independently. A single score is given after group
discussion
Situational Interview Process (cont.)
Differential Aptitude Test (DAT)
Abstract Reasoning
PROBLEM FIGURES
ANSWER FIGURES
A
B
C
D
E
Differential Aptitude Test (DAT)
Abstract Reasoning (cont.)
PROBLEM FIGURES
ANSWER FIGURES
A
B
C
D
E
Differential Aptitude Test (DAT)
Mechanical Reasoning
A
B
Which weighs more?
(If equal, mark C.)
Differential Aptitude Test (DAT)
Space Relations
A
B
C
D
~ Work Sample Tests ~
(performing a piece, or sample task, of the job)
• Good content validity
• Less/no adverse impact
~ Assessment Center Process ~
Candidates Participate in
Situational Exercises
Sample Group Exercises
Leaderless Group Discussion
Business Game
Sample Individual Exercises
Interview Simulation
Scheduling Exercise
In-Basket
Observed, discussed, and
scored on various
dimensions (e.g.,
communication,
decision making,
planning/organizational
skills) by trained raters
• Overall scores
computed and
ranked for
personnel
decisions
• Indiviudaul
dimension scores
used for
developmental
purposes
Types of Selection Tests (cont).
Ability Tests
Sensory (e.g.,
hearing, vision)
Motor (e.g.,
dexterity,
strength, agility)
ADA concerns
• Reasonable accommodation
• Essential job duties
Cognitive (e.g.,
Intelligence)
Wonderlic Personnel Test
Cognitive Ability Tests --- Pros & Cons
Pros --• Significant predictors of managerial performance (Hunter &
Hunter, 1984), especially scores on verbal and numerical ability
(Grimsley & Jarrett, 1973, 1975).
• Data from of meta-analytic studies indicate they are some of the
best predictors of performance available across an array of jobs
(Bobko, Roth, & Potosky 1999; Schmidt & Hunter, 1998).
Cons ---
• Potential adverse impact (minorities score significantly lower these
tests (Roth, Bevier, Bobko, Switzer, & Tyler, 2001)
• They do not assess other types of intelligence (e.g., analytical,
creative and practical intelligence (Sternberg, 1985, 2000).
~ “Big 5” Personality Factors ~
• Extraversion --- Outgoing, sociable
• Neuroticism (Emotional Stability): Depressed, anxious,
worrisome, etc.
• Agreeableness: Flexible, forgiving
• Conscientiousness: Careful, thorough, persevering
• Openness to Experience: Curious, imaginative
 Overall, conscientiousness and extraversion are best
predictors of managerial performance across jobs
 Personality measures add to prediction above and beyond
other commonly used measure such as cognitive ability
Self-Report Inventories
~ Myers-Briggs Type Indicator ~
Introversion
Extraversion
(Internally motivated; enjoys spending time
prefers one-to-one communication)
(Action-oriented; motivated enjoys alone;
by outside world and social interactions)
Sensing
Intuitive
(Desires concrete information; practical;
in orientation)
(Imaginative, creative factual and detailed
improvises comfortable with ambiguity;
focuses on contexts and connections)
Thinking
(Searches for facts and logic in a decision situation
Focus on tasks and work to be accomplished
Easily able to provide an objective and critical analysis)
Judging
Feeling
(Sensitive to other’s needs, Seeks
consensus, does not like conflict)
Perceiving
Plans before acting, breaks tasks down
Is comfortable to acts without planning,
into subsets, uses deadlines to accomplish tasks
Enjoys working on multiple tasks at the
same time, enjoys flexibility in work.
Sample Myers-Briggs Items
•
•
•
•
When you go somewhere for a day, would you rather:
a)
plan what you will do and when, or
b)
just go
In a large group, do you more often:
a)
introduce yourself, or
b)
get introduced
Is it harder for you to adapt to:
a)
routine, or
b)
constant change
Do you think it is a worse fault to be
a)
unsympathetic
b)
unreasonable
Literal ______________________ Figurative
Forgive _____________________ Tolerate
Impulse _____________________ Decision
Thinking --- Feeling
Very Clear Clear
30
25
Moderate Slight
20
15
10
5
T
Slight Moderate
5
10
Clear
15
20
Very Clear
25
30
F
Holly (7)
Steve (10)
Paul (2)
Bill (5)
Sally (2)
Frank (1)
Betty (19)
Thinking
• Looks at the logical consequences
of actions;
• Examines pros and cons of
approaches objectively;
• Energized by critique and analysis;
• Focuses on tasks
• Desires to find a standard to apply
in all situations; reasonable and fair
Feeling
• Focuses on how others may be affected by
decisions by mentally placing themselves into
situations;
• Guided by personal values;
• Energized by appreciating and supporting
others;
• Focuses on interactions
• Strives for harmony and positive interactions