Fire & Police Selection, Inc.
Hiring the Most Qualified
Firefighters….
…While Avoiding
(and Defending) Lawsuits
Contact Information
Dan A. Biddle, Ph.D.
CEO, Biddle Consulting Group, Inc./Fire & Police Selection, Inc.
193 Blue Ravine, Ste. 270
Folsom, CA 95630
1-888-990-3473
www.biddle.com | www.FPSI.com
Email: dan@biddle.com | stacy@fpsi.com
Copyright © 2011 Fire & Police Selection, Inc.
Biddle Consulting Group, Inc.
•
•
•
•
•
•
Assisted hundreds of clients with cases involving Equal Employment Opportunity
(EEO) / Affirmative Action (AA) (both plaintiff and defense)
EEO Litigation Support / OFCCP (federal contracting) Audit Support
Compensation Analyses / Test Development and Validation
Published: Adverse Impact and Test Validation, 2nd Ed., as a practical guide
for HR professionals
Editor & Publisher: EEO Insight an industry e-Journal
Creator and publisher of a variety of productivity
Software/Web Tools:
– OPAC® (Administrative Skills Testing)
– CritiCall® (9-1-1 Dispatcher Testing)
– AutoAAP™ (Affirmative Action Software and Services)
Industry
– C4™ (Contact Center Employee Testing)
Leader
– Encounter™ (Video Situational Judgment Test)
– Adverse Impact Toolkit™ (free online at www.disparateimpact.com)
– AutoGOJA® (Automated Guidelines Oriented Job Analysis®)
Copyright © 2011 Fire & Police Selection, Inc.
Fire & Police Selection, Inc.
•
•
•
•
•
•
•
•
•
Over 80 Physical Ability Test installations at fire departments across
the U.S.
Over 800 personnel and fire and police clients across the U.S. and
Canada
Entry-level written, physical, interview assessments, manipulative
skills tests, cognitive ability tests, and personality tests available for
selection of public safety personnel
Post-Academy certification tests
Fire Promotional tests and assessment exercises
Work Styles Inventory for predicting firefighter and police officer job
performance
Comprehensive Examination Battery for Entry-Level Firefighters
Situational Judgment/Writing Ability Test for Entry-Level Police
Officers
New National Fire Select Test (NFST™) for Entry-Level Firefighters
Copyright © 2011 Fire & Police Selection, Inc.
Fire & Police Selection, Inc.
•
Overview of Today’s Presentation:
•
•
•
•
The modern litigation landscape
Litigation triggers
Validation 101: Building and using tests
A quick tour through an “ideally weighted” test
for firefighters
Copyright © 2011 Fire & Police Selection, Inc.
Approach to Test Validation
The Quickly Changing Climate of
Personnel Testing…
•EEOC
•DOJ
•DOL (OFCCP—Office of Federal Contract
Compliance Programs)
•State EEO Offices of Fair Employment
•Private Plaintiff Groups
Copyright © 2011 Fire & Police Selection, Inc.
The Quickly Changing Climate of
Personnel Testing…
Title VII Cases Filed with EEOC
34,000
# Cases
30,000
26,000
22,000
18,000
14,000
10,000
FY
1997
FY
1998
FY
1999
FY
2000
FY
2001
FY
2002
FY
2003
FY
2004
FY
2005
FY
2006
Year
Copyright © 2011 Fire & Police Selection, Inc.
FY
2007
FY
2008
FY
2009
FY
2010
Adverse Impact & Test Validation
-Uniform Guidelines & Civil Rights Act of 1991
Amends Section 703 of the 1964 Civil Rights Act (Title VII)
(k)(1)(A). An unlawful employment practice based on
disparate impact is established under this title only if:


A(i) a complaining party demonstrates that a respondent uses a
particular employment practice that causes a disparate impact on the
basis of race, color, religion, sex, or national origin, and the respondent
fails to demonstrate that the challenged practice is job-related for the
position in question and consistent with business necessity; OR,
A(ii) the complaining party makes the demonstration described in
subparagraph (C) with respect to an alternate employment practice, and
the respondent refuses to adopt such alternative employment practice.
Copyright © 2011 Fire & Police Selection, Inc.
How Can Testing Practices be Challenged?
Title VII Disparate Impact Discrimination
Flowchart
Practice,
Procedure,
or Test (PPT)
Adverse
Impact?
“OR”
YES
Is the PPT
Valid?
YES
Alternative
Employment
Practice?
NO
Defendant Prevails
YES
Plaintiff Prevails
NO
END
NO
Plaintiff
Prevails
Copyright © 2011 Fire & Police Selection, Inc.
Validation Standards
Court Precedence
 Title VII: Uniform Guidelines on Employee Selection Procedures
(1978): EEOC, DOJ, CSB, DOL
 Principles for the Validation and Use of Personnel Selection
Procedures (2003)

– Division 14 of the American Psychological Association (the Society
for Industrial and Organizational Psychology, or SIOP)

Standards for Educational and Psychological Testing, published
by:
– American Educational Research Association
– American Psychological Association, and
– National Council on Measurement in Education
Copyright © 2011 Fire & Police Selection, Inc.
What is Validation?
 Traditionally,
it is making sure a
selection procedure (i.e., test) measures
what it is designed to measure
 In a legal realm, a selection procedure is
valid if it can be proven by an employer
that it is “…job related for the position in
question and consistent with business
necessity.”
Copyright © 2011 Fire & Police Selection, Inc.
Uniform Guidelines Style
Validation…
 Content
Validity
– Section14C
 Criterion Validity
– Section 14B
 Construct Validity
– Section 14D
Copyright © 2011 Fire & Police Selection, Inc.
What Types of Validation are
REALLY USED in Practice?
 Content
(80%)
 Criterion (15%)
 Construct (5%)

Why?
– Simplicity
– Defensibility
Copyright © 2011 Fire & Police Selection, Inc.
Content Validity

Cannot alone support inferences about
mental processes
– Cannot measure traits (such as
intelligence, aptitude, personality,
commonsense, judgment, leadership,
spatial ability)

Not appropriate for those KSAs that will
be expected to learn on the job
Copyright © 2011 Fire & Police Selection, Inc.
Content Validity Essential #1: Job
Analysis Research



Level 1: Job Description/Job Analysis
Review
Level 2: Test Customization Survey (TCS)
Level 3: Full Job Analysis (GOJA)
Copyright © 2011 Fire & Police Selection, Inc.
What Ratings are Typically Collected During
a Content Validity Job Analysis?
(Level 2-3)

For Job Duties ratings are typically collected for:
– Frequency
– Importance*
 Guidelines , 14.C[1,2,4]
– Best Worker*
 Guidelines , 14.C[9]
– Fundamental
 Americans with Disabilities Act
– Assignable
 Americans with Disabilities Act
* = required under federal Guidelines for Content Validity
Copyright © 2011 Fire & Police Selection, Inc.
What Ratings are Typically Collected During a
Content Validity Job Analysis? (Level 2-3)

For KSAs ratings are typically collected for:
– Frequency
– Importance*
 Guidelines, 14.C[1,2,4]
– Best Worker
 Guidelines, 14.C[9]
– Minimum v. Desirable Qualifications
 Guidelines, 14.C[4]
– Level needed upon entry
 Guidelines, 5.F, 14.C[1]
– Level needed for success
 knowledge only
* = required under federal Guidelines for Content Validity
Copyright © 2011 Fire & Police Selection, Inc.
Content Validity Essential #2: Targeting Key
KSAs for Test Building

Only certain KSAs can be tested…
– Only “concrete” KSAs can be tested
under content validity (cannot test
abstract KSAs)
– KSAs need to be “operationally defined”
– Important/Critical for the job
– Needed “Day One” on the job
– Duty/KSA Linkages
Copyright © 2011 Fire & Police Selection, Inc.
Firefighter Job Analysis Duties Sample
Firefighter/EMT-I, Firefighter-PM and Fire Engineer
Duties
Domain A
Supervisor Ratings
Frequency
Importance
Best
Worker
A1
Arrives on time, receives and
follows orders, instructions,
regarding the work to be
done.
8.75
4.88
5
yes
no
A2
Recognizes the need and
makes recommendations for
entries in daily log.
8.25
3.13
4
yes
no
A3
Takes inventory and records
and reports equipment and
apparatus missing or needing
repair or replacement.
8.25
4.00
4
yes
yes
Station Duties
Copyright © 2011 Fire & Police Selection, Inc.
Fundamental
Assignable
Firefighter Job Analysis KSAPCs Sample
Supervisor Ratings
Firefighter/EMT-I, Firefighter-PM, and Fire Engineer KSAPCs
Domain O Knowledges,
Skills, Abilities, and
Personal Characteristics
(KSAPCs): The following
This KSAPC is
linked to the
following
duties.
Frequency
O53
Willing to be resourceful, goaloriented, and proactive in
bringing tasks to completion.
A1, A2, A24, C5
O54
Willing to remain calm under
pressure or heavy workloads,
not easily given to hostility,
anxiousness, or vulnerability.
Wiling to make sensible
decisions under pressure.
A5, A20, E1, F8,
G2
O55
Ability to maintain respect for
other’s emotional needs
following a crisis.
knowledge skills, abilities,
and personal characteristics
affect work behavior and
are necessary or important
for successful job
performance.
Importance
MQ
vs.
HQ
Best
Worker
Level
Needed
Upon
Entry
7.25
4.13
MQ
1
4
7.63
4.63
MQ
1
4
HQ
1
3
A5, A20, C6,
6.38
4.13
F11
Copyright © 2011 Fire & Police Selection, Inc.
Selection Plan

Identify appropriate weights for the testing components.
– Completing a selection plan is the step in the validation process
where the key, measurable KSAPCs and Physical Requirements are
laid out as targets to be assessed by the selection process.
– A selection plan distills the complete list of KSAPCs and Physical
Requirements into only those that can and should be tested by one
or more selection procedures in the overall selection process.
– The result is a detailed plan that identifies the critical KSAPCs to
measure, five necessary ratings, and how the device should be
used (e.g., pass/fail, banding, etc.)
– The final step is to identify the weights for the various components.
Copyright © 2011 Fire & Police Selection, Inc.
Selection Plan—Sample
KSAPC/
Phys Req.
Selection Procedure and Use
(Pass/Fail, Rank, or Combine*)
Minimum/Helpfu
l Qualification
(MQ/HQ)
N/A
4
MQ
R
Verbal
communication skills
3.9
3.5
N/A
3.7
MQ
R
Knowledge of Fire
Codes
2.6
3.5
3.4
3.9
MQ
Upper body strength
1.9
3.9
N/A
3.4
MQ
Basic math skills
1.4
3.5
N/A
3.1
MQ
Background
Check
Importance
(1-5)
3.2
Structured
Interview
Level Needed
(Knowledges
Only) (1-4)
4.2
Written Test
Level Needed
Upon Entry
(1-4)
Interpersonal and
team work skills
Application Form
Best Worker
(1-5)
Knowledge, Skills,
Ability, Personal
Characteristic, or
Physical Requirement
KSAPC/Physical Requirement Rating
P/F
P/F
*The score from this selection procedure will be combined with scores from other selection procedures in the selection process.
Copyright © 2011 Fire & Police Selection, Inc.
Content Validity Essential #3: Connecting
the Test to the Job
Other KSAOs
Job
Duties
Operationally
defined KSAOs
Selection
Devices
(e.g., application
form, tests,
interviews)
Copyright © 2011 Fire & Police Selection, Inc.
Content
Valid!
Content Validity Essential #4: (Test
Usage)
Pass/Fail Cutoffs:
– “Normal Expectations of Acceptable Proficiency in the Workplace”
(Guidelines, 5H)
– Modified Angoff (U.S. v. South Carolina, USSC)
 Banding:
– Substantially Equally Qualified Applicants
– Statistically Driven (use Std. Error of Difference)
 Ranking:
– Is there adequate score dispersion?
– Does the test have high reliability? (e.g., >.85)
– Is the KSA performance differentiating?
 Weighted/combined with other tests
– How are the weights related to the job
– Do they come from the job analysis or SME ratings?

Copyright © 2011 Fire & Police Selection, Inc.
Using Test Scores
Applicant
Tom
Stacy
Bob
Frank
Julie
Rozanne
Mark
Luke
Henry
Paul
Peter
Rebecca
Alyssa
Matthew
John
Annette
Ray
Thomas
Julissa
Score
100
100
100
100
99
99
98
98
97
97
96
96
95
94
93
93
92
91
90
• Ranking assumes one applicant is
reliably more qualified than the other
• Banding considers the unreliability of
the test battery and “ties” applicants
• Pass/fail cutoffs treat all applicants
as either “qualified” or “not qualified”
Pass/Fail Cutoffs

Some major considerations:
– NOT ADVISED:
 Using an arbitrary cutoff (e.g., 70%)
 Using applicant scores to benchmark (e.g., setting cutoff scores at
mean-SD of applicant scores)
– ADVISED:
 Consider “Normal expectations of acceptable proficiency in the
workplace” (Guidelines, 5H)
– Usually requires SME-level data or ratings
 # Applicants that can be feasibly processed
– CONSIDER:
 Is the test supported by content validity information or criterionrelated information?
 How critical are the KSAs measured?
 Does the test measure “baseline” or “differentiating” KSAs?
 How would current incumbents perform on this test?
Copyright © 2011 Fire & Police Selection, Inc.
Using Test Scores

Tests can be used on a pass/fail basis to ensure that candidates
possess the minimum competency levels needed for the job.

Tests can be used in a ranked fashion whereby criterion-related
validity evidence supports that higher test performance
statistically predicts better job performance (or, content validity
can also be used per UGESP 14C9).

Tests can be used by weighting/combining multiple test scores
whereby a “compensatory” or combined score is used.

Banding is a recommended “balanced” solution because it uses
the test results in ways that are supportable by the tests
themselves.
Copyright © 2011 Fire & Police Selection, Inc.
Comparison Between “Test Use”
Options
Factor
Ranking
Banding
Pass/Fail
Cutoffs
Validation Requirements
High
Moderate
Low
Adverse Impact
High
Moderate
Low
Defensibility
Low
High
High
Litigation "Red Flag"
High
Moderate
Low
Utility
High
Moderate
Low
Cost
Low
Moderate
High
Applicant Flow
Restrictive/Controllable
Moderate/Controllable
High
Development Time
Low
Moderate
High
Reliability Requirements
High
Moderate
Low
# Item Requirements
High
Moderate
Low
Copyright © 2011 Fire & Police Selection, Inc.
Weighting/Combining Scores
Interpersonal
Job Performance
Personality
Factors
Cognitive
Ability
Experience/
Background
Copyright © 2011 Fire & Police Selection, Inc.
Sample Data—Cognitive and
Personality Test Scores Total Scores
Gender
Ethnicity
Cognitive
Score
M
W
72
175
M
B
71
158
F
W
70
175
M
B
70
161
M
W
69
156
M
B
68
225
M
W
63
156
Copyright © 2011 Fire & Police Selection, Inc.
Personality
Score
Sample Data—Cognitive and Personality
Test Scores Combined Test Scores
Gender
Ethnicity
Cognitive
% Score
Personality
Score
Combined
% Score
M
W
72
175
83%
M
B
68
225
83%
F
W
70
175
81%
M
B
71
158
81%
M
B
70
161
81%
M
W
69
156
79%
M
W
63
156
73%
Copyright © 2011 Fire & Police Selection, Inc.
Steps for Weighting Tests Using
Content Validity Methods
1.
2.
3.
4.
5.
6.
7.
8.
Select a panel of 4-12 Subject-Matter Experts (SMEs) who are truly experts in
the content and “functional areas” of the target position. Ensure diversity on
the panel.
Provide a copy of the Job Analysis for each SME. Be sure that the Job Analysis
itemizes the various job duties and KSAPCs that are important or critical to the
job.
Provide each SME with a copy of each test (or a highly detailed description of
the content of the test if confidentiality issues prohibit SMEs from viewing the
actual test).
Explain the confidential nature of the workshop, the overall goals and
outcomes, and ask the SMEs to sign confidentiality agreements.
Discuss and review with SMEs the content of each test and the KSAPCs
measured by each. Also discuss the extent to which certain tests may be better
measures of certain KSAPCs than others. Factors such as the vulnerability of
certain tests to fraud, reliability issues, and others should be discussed.
Provide a survey to SMEs that asks them to distribute 100 points among the
tests that will be combined. Be sure that they consider the importance levels of
the KSAPCs measured by the tests, and the job duties to which they are linked,
when completing this step.
Detect and remove outlier SMEs from the data set.
Calculate the average weight for each test. These averages are the weights to
use when combining the test into a composite score.
Copyright © 2011 Fire & Police Selection, Inc.
Standardizing Scores
Before individual test scores can be weighted and
combined, they should be standard scored. This
process normalizes the scores and is a necessary
step to place different tests on a level playing field.
 Standardizing test scores is simple:

– Convert raw scores into Z scores by subtracting each
applicant’s score from the average (mean) score of all
applicants and dividing this value by the standard
deviation (of all applicant total scores).
– Multiply the Z scores by the respective weights of the test
components and a final score for each applicant can then
be calculated.
Copyright © 2011 Fire & Police Selection, Inc.
Building a Balanced Hiring Program by Sorting
the Key Firefighter Competencies for Success
In 2008, FPSI obtained input from over 140 chief-level
officers who were asked to weigh the importance of key
critical skills and abilities required for successful entry-level
firefighter job performance
 Developed cutoff score by working with a panel of 12 Chiefs
 Results

 The process identified the skills and abilities that are most critical for
entry-level firefighters. With input from over 140 fire departments,
these ratings are being used to build a written test that properly
weighs the most important qualifications for the entry-level firefighter
job. With this type of input from fire executives across the country,
the National Fire Select Test™ is the first test of its type!
Copyright © 2011 Fire & Police Selection, Inc.
U.S. Fire Department Input &
Opinions
National Fire Select Test™
Personal
Characteristics
55%
Cognitive/Academic
Skills & Abilities
45%
Personal Characteristics Component
Cognitive/Academic Component
Mechanical
Ability
14%
Following Orders
11%
Mathematical
Reasoning
15%
Strategic
DecisionMaking
14%
Teamwork
12%
Thoroughness/
Attention
To Detail
9%
Reading
Ability
18%
Working
Under Stress
10%
Public
Relations
9%
Allegiance/
Loyalty
10%
Sensitivity
8%
Problem
Solving
14%
Writing Ability
15%
Map Reading
10%
Emotional
Stability
10%
Copyright © 2011 Fire & Police Selection, Inc.
Proactive/
Goal-Oriented
8%
Truthfulness/
Integrity
13%
Proposed Solutions for Testing Cognitive/Academic Competencies
for Entry-Level Firefighters
Cognitive/Academic
(45% Overall
Importance)
Weight
Math
10%
Use written or “work sample” format; measure using a limited number of
multiple-choice items. Balance various types of math skills (add/subtract/multiple, etc.).
CV
Reading
14%
Measure using either (1) “Test Preparation Manual” approach (where applicants are given a
manual and asked to study it for a few weeks prior to taking the test based on the Manual);
or (2) a short reading passage containing material at a similar difficulty/context to the job
that applicants are allowed to study during the testing session and answer related test items.
CV
Verbal Communication
15%
While a Structured Interview is the best tool for measuring this skill (because the skill
includes verbal and non-verbal aspects), some level of this skill can be measured using
word recognition lists or sentence clarity items.
CV
Writing
12%
Measure using writing passages or word recognition lists, sentence clarity, and/or
grammar evaluation items.
CV
Map Reading
8%
Measure using maps and related questions asking applicants how they would maneuver
to certain locations. Include directional awareness.
CV
Problem-Solving
15%
Measure using word problems measuring reasoning skills in job-rich contexts.
Strategic Decision-Making
13%
While a Structured Interview is the best tool for measuring this skill (because the applicant
can be asked to apply this skill in firefighter-specific scenarios), some level of this skill can be
measured using word problems or other contexts supplied in written format where applicants
can consider cause/effect of certain actions.
CV
Mechanical Ability
12%
Using CV, measure mechanical comprehension skills such as leverage, force, and mechanical/
physics contexts regarding weights, shapes, and distances. Also can measure spatial
reasoning (when using a CRV validation strategy).
CV/CRV
Proposed Testing Solution
Notes: CV: Content Validity; CRV: Criterion-related validity.
Typical
Validation
Method
CRV
Proposed Solutions for Personal Characteristics
for Entry-Level Firefighters
Personal Characteristics
(55% Overall)
Weight
Proposed Testing Solution
Teamwork
12%
Working Under Stress
10%
Allegiance/Loyalty
10%
Truthfulness/Integrity
13%
Public Relations
9%
Emotional Stability
10%
Sensitivity
8%
Under a CV strategy, a Situational Judgment Test (SJT)
can be used for measuring these skills. Alternatively, a
custom personality test can be developed using CRV.
while these types of assessments can measure if an
applicant knows the most appropriate response (using an
SJT) or the best attitude or disposition (personality test),
they are limited in that they cannot measure whether an
applicant would actually respond in such a way. For these
reasons, measuring the underlying traits that tend to
generate these positive behaviors is typically the most
effective strategy. Structured Interviews can also provide
useful insight into these types of competencies, as well
as background and reference evaluations. However, these
tools are time consuming and expensive, so measuring
these areas in the testing stage is an effective strategy.
Proactive/Goal-Oriented
8%
Thoroughness/Attention to Detail
9%
Following Orders
11%
These competencies can be effectively measured using
either an SJT (using a CV or CRV strategy) or a
Conscientiousness (CS) scale (using a CRV strategy). A CS
test can be developed using just 20-30 items (using likerttype responses). Such tests are typically successful in
predicting job performance in fire settings.
Notes: CV: Content Validity; CRV: Criterion-related validity.
Typical Validation
Method
CV/CRV
CV/CRV
Selecting the Ideal Firefighter
Mechanical Aptitudes
Critical Skills &
Abilities
Test Map
“A”
Reading
Test “B”
Reading Ability
Mathematical Ability
Writing Ability
Interpersonal Skills
NFST™
Decision“C”
Making
Test
Spatial Abilities
Copyright © 2011 Fire & Police Selection, Inc.
Challenges to Building a Balanced Testing
Program and Recommendations for Success

The most significant challenge to building an effective testing
program for entry-level firefighters lies with testing the “soft skills”
(personal characteristics). This is because these skills—such as
teamwork and interpersonal skills—are crucial ingredients for success
but they are the most difficult to measure in a typical testing format.

For example, developing a math test is easy; developing a test for
measuring teamwork skills is not—but the latter was rated as more
important for overall job success!

The reason for this is that many skills and abilities are “concrete”
(opposed to theoretical and abstract). An applicant’s math skills can
readily be tapped using questions that measure numerical skills at the
same level these skills are required on the job. Abstract or soft skills
like teamwork are more difficult to measure during a two-hour testing
session.
Copyright © 2011 Fire & Police Selection, Inc.
Failure to Use a “Balanced Test”

May have limited validity.

Less minority representation in the final eligibility list.

A testing process that leaves out (or underestimates) important
cognitive/academic skills will likely result in a significant number of
cadets washing out of the academy, and can also lead to poor
performance on the job.

A testing process that over-measures cognitive/academic skills and
under-measures personal characteristics could lead to a group of
book-smart firefighters who lack the skills necessary to work
cooperatively in the close living conditions required by firefighters.

Greater potential for lawsuits which may result in lowering
standards.
Copyright © 2011 Fire & Police Selection, Inc.
Fire & Police Selection, Inc.
Contact us at:

Mailing Address
 193 Blue Ravine Rd. Suite 270
Folsom, CA. 95630

Website
 www.FPSI.com
 www.NationalFireSelect.com

Telephone
 916.294.4242 x. 245

E-Mail
 stacy@fpsi.com
Copyright © 2011 Fire & Police Selection, Inc.