National Personal Protective
Technology Laboratory
Variability of Respirator Fit Test Panels
NIOSH Study Results
Science Webinar
Christopher Coffey, PhD
Associate Director For Science
James T. Wassell, PhD
Research Statistician
July 23, 2013
NIOSH PPT / NPPTL
Vision & Mission
The VISION is to be the leading provider of quality, relevant, and timely
PPT research, training, and evaluation.
The MISSION of the PPT program is to prevent work-related injury,
illness and death by advancing the state of knowledge and application
of personal protective technologies (PPT).
PPT in this context is defined as the technical methods, processes, techniques, tools, and materials
that support the development and use of personal protective equipment worn by individuals to
reduce the effects of their exposure to a hazard.
Purpose
 Scientific meeting
 Discuss objectives and preliminary results of the inter-
panel variability study
 Initiate and encourage dialog
Background
 NIOSH certification testing policy since early 1980s
 Based on1960’s anthroprometric data
 1970’s LANL panels
 42 CFR
 Filter penetration
 Resistance
 No measurement of face seal and other leakage (inward
leakage)
Background (Continued)
 Inward leakage important
 Determines amount of unfiltered air in facepiece
 Facepiece-to-face seal leakage potentially substantial
 Benchmark testing
 Evaluate respirator fit characteristics
 101 respirators
Reasons for New Panels
 Concerns about LANL panel
 Used military data
 Applicability to civilian workforce
 Demographic changes
 2003 survey of respirator users
 Found differences in key facial measurements
 Developed new, more representative panels
 Bivariate
 PCA
NIOSH Panel Development Timeline
 Protocol development and review
 2002
 Data collection
 2003
 Peer-review publication
 2007
 Institute of Medicine review
 2007
 Conclusion - both panels an improvement over the
LANL panels
Distribution of US Workers NIOSH Bivariate
Panel
Total = 97.7%
138.5
128.5
Face Width (mm)
120.5
134.5
132.5
146.5
144.5
5.2%
3.5%
21.3%
8.7%
5.7%
118.5
10.5%
25.0%
5.5%
5.3%
7.1%
108.5
98.5
158.5
Principal Component Analysis
 Defines new coordinate system
 linear combinations to describe trends
 Principal components - set of values of linearly
uncorrelated variables
 Number of principal components ≤ number of
original variables
 First principal component has the largest
possible variance
 Sensitive to the relative scaling of the original
variables
PCA Panel Development
 2003 survey – 18 measurements
 Four criteria to reduce number of
measurements required
 Relevant to respirator fit
 Excluded dimensions correlated with included ones
 Reasonable number
 Excluded difficult to obtain and/or highly variable
 10 measurement in categorization algorithms
35-Member Population-Based
Bivariate Panel
Cell # Population %
1
5.5
2
5.3
3
10.5
4
25.0
5
7.1
6
5.7
7
21.3
8
8.7
9
5.2
10
3.5
Calculated subjects needed Subjects tested
1.9 = 2
2
1.85 = 2
2
3.67 = 4
4
8.75 = 9
9
2.49 = 2
2
1.99 = 2
2
7.45 = 7
7
3.04 = 3
3
1.82 = 2
2
1.22 = 1
2
PCA Measurements
 Head Breadth
 Nose Breadth
 Minimum Frontal
 Nose Protusion
Breadth
 Bizygomatic Breadth
(face width)
 Bigonial Breadth
 Nasal Root Breadth
 Sellion-Subnasale
Length
 Menton-Sellion Length
(Face length)
 Interpupillary Breadth
10 Facial Measurements
Head Breadth
Minimal Frontal
Breadth
Nasal Root
Breadth
Nose
Length
Interpupillary
Breadth
Nose
Protrusion
Face Width
Face
Length
Nose Breadth
Bigonial
Breadth
Principal Component Analysis Panel
Long/Narrow Nose
Long Face
Shape
Short Face
Short/Wide Nose
Small
Overall Size
Large
Background (Continued)
 Inward leakage (IL) 2009 NPRM and Test Procedure
 Half-facepieces
 Leakage through face seal and non-filter components
 New human subject panels

Screening - Principal Component Analysis (PCA)

Sizing - NIOSH Bivariate

35-member panel
Statistical basis for TIL testing1
 Simple threshold test
 Sample sizes of 25 to 45 in increments of 5
 Definitions
 Highly effective – sufficient fit for at least 90% of
population, pass at least 90% of time
 Effective – sufficient fit for at least 80% of
population, pass at least 80% of time
 Ineffective – sufficient fit for no more than 60% of
population, pass no more than 5% of time
 Highly ineffective – sufficient fit for no more than
50% of population, pass no more than 1% of time
- D Landsittel, Z Zhuang, W Newcomb, R BerryAnn, Determination of sample size and passing criteria for respirator
fit panels. http://www.cdc.gov/niosh/docket/archive/docket137.html
Statistical basis (Continued)
 Type I errors
 α0.8 = 0.20, α0.9 = 0.10
 Type II errors
 β0.6 = 0.05 (1- β0.6 = 0.95), β0.5 = 0.01 (1- β0.1 = 0.99)
 Results
 Panels ≤ 30, insufficient to meet specified properties
 Panel of 35 met all except power for rejecting ineffective
model (0.942 vs. 0.95)
 Panels ≥ 40, sufficient to meet specified properties
 Decision to use 35-member panel
Panel Variability
 Definition
 Degree of agreement among panels
 Grid variability1

Temporally from one assembly of a panel to another

Between laboratories
 Public comments
 Causes a wide range of results among panels
 Increase number of panel members to 105
 RTI contracted to recommend panel size
1Submission
to NIOSH Docket Number 036 from Birkner (Moldex-Metrics) – August 22, 2007,
http://www.cdc.gov/niosh/docket/archive/docket036.html
2011 RTI Report
 Mathematical/simulation based
 Simulated the distribution of the test statistic
 Verified Landsittel results
 Estimates of fit within a two-member cell
highly variable
 Equal allocation of members to cells
 Any respirator that fits a population should also fit
the subpopulations
 Adequate data for precise cell estimates
2011 RTI Recommendation
 40- or 50-member evenly distributed bivariate panels
Study Objectives
 Address comments
 Determine the variability between different
anthropometric panels
 Experimentally confirm RTI findings
 Investigate the ease of panel implementation
 Allow for statistical analysis
Methodology
 Ten N95 Respirators
 Seven manufacturers
 Five Filtering Facepiece (OSFA and two sizes)
 Five Elastomeric Facepieces (three sizes)
 Inward Leakage by PortaCount® Pro+ 8038
 Ambient concentration within PortaCount
range (0.01 to 2.5 x 105 particles/cm3)
 Two replicates
 Seal completely broken in between replicates
 Experienced and new subjects
Study Methodology (continued)
 Evenly distributed
 40 subjects/panel
 Four subjects/cell
 Study goal recruit
120 subjects
Bivariate Panel - Two Facepiece Sizes
Smaller size
Larger size
Bivariate Panel - Three Facepiece Sizes
Small
Medium
Large
Distribution of Panel Members for
Bivariate Panel
One cell 9, three cell 10 members are outside the limits of the panel.
PortaCount Pro+ and 8026 NaCl Generator
Probe photos
Exercises
 Eight exercise OSHA Fit test protocol
 Normal Breathing
 Deep Breathing
 Turning Head Side to Side
 Moving Head Up and Down
 Reciting Rainbow Passage
 Reaching Floor to Ceiling
 Grimacing (not included in calculations)
 Normal Breathing
Inward Leakage Test
New Subject Recruitment
 NPPTL Website
 NIOSH Facebook page and Twitter
 Community Outreach (Posters, Flyers,
Face to Face meetings)
 Local universities, community colleges, vocational
tech schools, high schools
 Local Fire Departments, Ambulance Services,
Hospitals
 Federal employees in Pittsburgh and
Morgantown
Questions?
Bivariate Implementation Issues
Identified in the Study
 Difficulty in finding members for cells 1, 6
and 10
 Excluded more than expected 3.3% of
potential members
 Measurement concerns consistent with
docket comments
 Multiple certification panels
Current NIOSH Certification Panels for
Non-CBRN Respirators
 Qualitative iso-amyl acetate test
 LANL Panels
 One size fits all

10 member panel

Half-facepiece

Full facepiece
 Multiple size facepieces

6 member panel

Specific cell sizes
Current NIOSH Certification Panel for
Elastomeric Half-Facepiece Respirators
(LANL 10-member Panel number of subjects evaluated per cell)
LANL Full Facepiece Panel
Face Width (mm)
117.5
126.5
135.5
144.5 153.5
133.5
cell 9
cell 10
cell 6
cell 7
cell 8
cell 3
cell 4
cell 5
cell 1
cell 2
123.5
113.5
103.5
93.5
Current NIOSH Certification Panel for
CBRN Respirators
 Quantitative generated aerosol (corn oil) test
 Panel
 Face width and face length
 One size – 25 member panel
 Two sizes – 29 member panel
 Three sizes – 38 member panel
Reasons for PCA/Bivariate Inclusion
 Currently both used for research
 PCA headform development
 Both in 2009 proposed test method
 PCA for screening
 Bivariate for testing
PCA Panel Implementation Issues
 Identified 7/144 (5%) as outliers
 Expected
 Three females

Two excluded

One on the outer oval
 Four males

One from Bivariate cell 9

Three from Bivariate cell 10
 Exclusions not due to expected discriminators
Distribution of PCA Panel Members
Second Principal Component
50
40
30
20
10
240
260
One Outside with First PC = 327
280
First Principal Component
300
320
Pittsburgh PCA Distribution with NIOSH
BV Cell Designations
Cell 6
16/105 15.2%
6,6,6,6,6,6,7,7,7,7,9,9
9,9,9,9
Cell 1
16/105 15.2%
1,1,1,1,1,1,1,2,2,2
3,3,3,3,3,4
Cell 2
17/105
16.2%
1,2,2,3,3,3,3
3,3,3,4,4,4,4
5,5,7
Cell 5
8/105 7.6%
4,4,7,7,7,7,8,9
Cell 4
12/105 11.4%
2,2,4,4,4,4,4,5,5,5,5,8
Cell 3
6/105 5.7%
1,2,2,2,2,2
5/105 outliers were identified, 4.8% (5,8,9,10,10)
Cell 7
7/105 6.7%
5,5,7,7,7,9,10
Cell 8
18/105 17.1%
5,5,5,7,8,8,8,8,8,8,8
8,9,10,10,10,10,10
Morgantown PCA Distribution with NIOSH
BV Cell Designations
Cell 6
5/39 12.8%
6,6,6,9,9
Cell 1
3/39 7.7%
1,1,1
Cell 2
6/39 15.4%
4,4,4,4,4,4
Cell 5
4/39 10.3%
4,6,6,7
Cell 4
4/39 10.3%
4,4,4,4
Cell 3
3/39 7.7%
4,4,4
2/39 outliers were identified 5.1% (9,10)
Cell 7
7/39 17.9%
4,5,7,7,7,7,7
Cell 8
5/39 12.8%
7,7,10,10,10
Implications of using new panels
 Confusion of panel cell designations among
multiple panels
 Limits number of potential panel members
 Smaller women/larger men over 50
 Current IRB age limit ≤50
 Confusion related respirator facepiece sizes
Mid-Study Decisions
 Recruitment issues
 Difficulty in finding 4/cell
 Move to Morgantown
 Physical limits of cell size
 Multiple panel distributions in analysis
 35-member in 2009 NPRM
 25-member in previous NIOSH research1
 Recruited 24 more members
 Smaller panel sizes analyzed
1Zhuang
Z, Bradtmiller B, and Shaffer, RE. New Respirator Fit Test Panels Representing the Current
U.S. Cilivian Work Force. JOEH 4:647-659, 2007.
Distribution of 144 Subjects Tested
Questions?
Data Analysis Methodology
 Nonparametric Bootstrapping
 data-based resampling
 500,000 panels generated and used for statistical
analysis
 Each measurement considered independent
 2880 data points used in this analysis
 144 subjects x
 2 measurements x
 10 respirator models = 2880
Bootstrap – Data-Based Resampling
35-member population-based Bivariate Panel
Cell # Subj
12
1
12
2
12
3
27
4
12
5
12
6
21
7
12
8
12
9
12
10
total 144
Reps data
2
2
2
2
2
2
2
2
2
2
24
24
24
54
24
24
42
24
24
24
288
sample
2
2
4
9
2
2
7
3
2
2
35
Repeat
500,000
times
Calculate ONE mean
for this One Panel
Sampling with replacement yields
ONE bootstrap sample for the 35
member population based Bivariate
Panel
Summary
coefficient of
variation
estimate
Bootstrap – Data-Based Resampling
40-member Equal Distribution Panel
Cell # Subj
12
1
12
2
12
3
27
4
12
5
12
6
21
7
12
8
12
9
12
10
total 144
Reps data
2
2
2
2
2
2
2
2
2
2
24
24
24
54
24
24
42
24
24
24
288
sample
4
4
4
4
4
4
4
4
4
4
40
Repeat
500,000
times
Calculate ONE mean
for this One Panel
Sampling with replacement yields
ONE bootstrap sample for the 40
member equal distribution
Bivariate Panel
Summary
coefficient of
variation
estimate
Data Analysis Methodology
 Variability between panels determined by
computing the mean IL for each panel
 Variability measured as the coefficient of
variation (CV) = SD/mean for the 500,000
bootstrap means.
 Reference: Efron, B and R. Tibshirani, An
Introduction to the Bootstrap. 1993. Chapman
& Hall, Inc.
Results - Equal Distribution Panels
Coefficient of Variation (% change from baseline)
Model (Type)
N=40
N=30
N=20
A (FFR)
0.241
0.278
(15.4%)
0.341
(41.3%)
B (FFR)
0.156
0.180
(15.5%)
0.221
(41.2%)
C (FFR)
0.169
0.196
(15.6%)
0.240
(42.0%)
D (FFR)
0.266
0.308
(15.6%)
0.377
(41.7%)
E (FFR)
0.175
0.202
(15.4%)
0.248
(41.5%)
F (Elastomeric)
0.278
0.321
(15.3%)
0.393
(41.2%)
G (Elastomeric)
0.198
0.229
(15.7%)
0.281
(41.6%)
H (Elastomeric)
0.191
0.221
(15.7%)
0.270
(41.5%)
I (Elastomeric)
0.227
0.262
(15.3%)
0.321
(41.2%)
J (Elastomeric)
0.228
0.264
(15.5%)
0.323
(41.5%)
Results - Equal Distribution Panels
 Reference: Cell size of 4 (n=40)
 Cell size of 3 (n=30) has 15.3% - 15.7% greater
CV than the cell size of 4 (n=40) panel
 Cell size of 2 (n=20) has 41.2% - 42.0% greater
CV than the cell size of 4 (n=40) panel
Results - Population Based Panels
Coefficient of Variation (% change from baseline)
Model (Type)
N=35
N=30
N=25
A (FFR)
0.251
0.271
(7.8%)
0.297
(17.8%)
B (FFR)
0.171
0.184
(7.4%)
0.205
(19.8%)
C (FFR)
0.179
0.193
(7.7%)
0.208
(16.3%)
D (FFR)
0.265
0.290
(9.3%)
0.307
(15.3%)
E (FFR)
0.184
0.199
(8.2%)
0.223
(21.6%)
F (Elastomeric)
0.321
0.343
(6.7%)
0.379
(18.3%)
G (Elastomeric)
0.189
0.209
(10.7%)
0.226
(18.6%)
H (Elastomeric)
0.198
0.213
(7.7%)
0.228
(15.4%)
I (Elastomeric)
0.219
0.242
(9.9%)
0.265
(20.7%)
J (Elastomeric)
0.228
0.248
(8.5%)
0.271
(18.1%)
Results - Population Based Panels
 Reference: Population Based Panel of n=35.
 Panel of n=30 has 6.7% - 10.7% greater CV
than the reference
 Panel of n=25 has 15.3% - 21.6% greater CV
than the reference
Comparison of Population based and Equal
distribution panels for n=30
Coefficient of Variation
Model (Type)
A (FFR)
Pop Based
N=30
0.271
<
Equal Dist
N=30
0.278
B (FFR)
0.184
>
0.180
C (FFR)
0.193
<
0.196
D (FFR)
0.290
<
0.308
E (FFR)
0.199
<
0.202
F (Elastomeric)
0.343
>
0.321
G (Elastomeric)
0.209
<
0.229
H (Elastomeric)
0.213
<
0.221
I (Elastomeric)
0.241
<
0.262
J (Elastomeric)
0.247
<
0.264
8 out of 10 Population Based Panels have smaller CV. Binomial test p-value = 0.11
Summary of Results
 This is a comparison of 6 types of Panels.
 Panels with smaller sample size have greater
variability as measured by the CV. This is
result is consistent for all respirator models
investigated.
 For n=30, Population based panels have
smaller variability than equal distribution
panels
 Population based panels take a larger sample
from the “middle” cells (4 & 7) that are more
homogeneous (smaller area).
Future Analysis
 Within respirator type variability
 Pass/Fail Rates
 Within cell variability
 Use of cell clouds to evaluate “size specific” data
 New subjects v. experienced
 Male/female comparisons
 Inter-rater statistics for analysis of panel variability
Questions for Discussion
 Is CV an appropriate approach for this
analysis? If not, what do you suggest?
 What other ways can the data be used?
 What further research and data analysis is
needed to answer the question of inter-panel
variability?
Potential Topics for Next Webinar
 August 20, 2013 at 1 PM
 Panels issues for certification implementation
 Two LANL panels for certification testing

LANL half facepiece panel
 Based on face length and lip length
 Divided into 10 cells

LANL full facepiece panel
 Based on face length and face width
 Divided into 10 cells
 NIOSH Bivariate
 PCA
Quality Partnerships Enhance Worker
Safety & Health
Visit Us at: http//www.cdc.gov/niosh/npptl
Disclaimer:
The findings and conclusions in this presentation have not been
formally disseminated by the National Institute for Occupational
Safety and Health and should not be construed to represent any
agency determination or policy.
Thank you