Quality Assurance &
Quality Control
University of New Mexico




Kristin Vanderbilt
William Michener
James Brunt
Troy Maddux
University of South Carolina

Don Edwards
Structure of talk:



Types of errors
Define QA/QC
Procedures for reducing and detecting data
contamination




Designing data sheets
Validation rules, filters, lookup tables, control charts
Graphics and statistics
Outlier detection


Samples
Simple linear regression
Instruction Materials

Primary References

Michener and Brunt (2000) Ecological Data: Design, Management
and Processing. Blackwell Science.




Edwards (2000) Ch. 4
Brunt (2000) Ch. 2
Michener (2000) Ch. 7
Dux, J.P. 1986. Handbook of Quality Assurance for the Analytical
Chemistry Laboratory. Van Nostrand Reinhold Company
QA/QC

“mechanisms [that] are designed to prevent
the introduction of errors into a data set, a
process known as data contamination”
Brunt 2000
Errors (2 types)


Commission
Omission
Brunt 2000
Errors of Commission
Incorrect or inaccurate data in a dataset
Common
Can be (relatively) easy to find




Malfunctioning instrumentation





Sensor drift
Low batteries
Damage
Animal mischief
Data entry errors
Identifying Sensor Errors: Comparison of data
from three Met stations, Sevilleta LTER
Identification of Sensor Errors: Comparison of
data from three Met stations, Sevilleta LTER
Errors of Omission
Common
Difficult or impossible to find



Inadequate documentation of data
values, sampling methods, anomalies in
field, human errors
Quality Control

“mechanisms that are applied in advance,
with a priori knowledge to ‘control’ data
quality during the data acquisition process”
Brunt 2000
Quality Assurance

“mechanisms [that] can be applied after the
data have been collected, entered in a
computer and analyzed to identify errors of
omission and commission”


graphics
statistics
Brunt 2000
QA/QC Activities



Defining and enforcing standards for formats,
codes, measurement units and metadata.
Checking for unusual or unreasonable
patterns in data.
Checking for comparability of values between
data sets.
Brunt 2000
Process from raw data to verified data to
validated data (reviewed by a qualified
scientist) is called data maturity, and
implies increasing confidence in the
quality of the data through time
QA/QC should be
considered at all stages
of the research
process……
•Study Design
•Acquisition/Data Entry
•Data documentation
Flowering Plant Phenology – QA/QC
and study design example




What data are we collecting?
How often are we collecting?
Where are we collecting?
What is the experimental design?
Protocols are developed:



Description of the project design
(where, how, when data are collected)
Instructions for collecting the data
Sample data sheet
The plant phenology study has:



Four Sites
Three sites have three transects, one
site has two transects
Every species encountered will have
phenological stage recorded
Data Collection Form Development:
What’s wrong with this data sheet?
Plant
______________
______________
______________
______________
______________
______________
______________
______________
Life Stage
_______________
_______________
_______________
_______________
_______________
_______________
_______________
_______________
PHENOLOGY DATA SHEET
Collectors:_________________________________
Date:___________________ Time:_________
Location: black butte, deep well, five points, goat draw
Notes: _________________________________________
_______________________________________________
Plant
______________
______________
______________
______________
______________
______________
______________
______________
Life Stage
_______________
_______________
_______________
_______________
_______________
_______________
_______________
_______________
PHENOLOGY DATA SHEET
Collectors:_________________________________
Date:___________________ Time:_________
Location: black butte, deep well, five points, rio salado
Notes: _________________________________________
_______________________________________________
Plant
ardi
arpu
atca
bamu
zigr
P/G
P/G
P/G
P/G
P/G
P/G
P/G
V
V
V
V
V
V
V
Life Stage
B
B
B
B
B
B
B
FL
FL
FL
FL
FL
FL
FL
FR
FR
FR
FR
FR
FR
FR
M
M
M
M
M
M
M
S
S
S
S
S
S
S
D
D
D
D
D
D
D
NP
NP
NP
NP
NP
NP
NP
PHENOLOGY DATA SHEET
Collectors:_________________________________
Date:___________________ Time:_________
Location: black butte, deep well, five points, goat draw
Notes: _________________________________________
_______________________________________________
Plant
ardi
arpu
atca
bamu
zigr
P/G = perennating or germinating
V = vegetating
B = budding
FL = flowering
FR = fruiting
P/G
P/G
P/G
P/G
P/G
P/G
P/G
V
V
V
V
V
V
V
Life Stage
B
B
B
B
B
B
B
FL
FL
FL
FL
FL
FL
FL
FR
FR
FR
FR
FR
FR
FR
M
M
M
M
M
M
M
S
S
S
S
S
S
S
M = dispersing
S = senescing
D = dead
NP = not present
D
D
D
D
D
D
D
NP
NP
NP
NP
NP
NP
NP
PHENOLOGY DATA SHEET
Rio Salado - Transect 1
Collectors:_________________________________
Date:___________________ Time:_________
Location: black butte, deep well, five points, goat draw
Notes: _________________________________________
_______________________________________________
Plant
ardi
arpu
atca
bamu
zigr
P/G = perennating or germinating
V = vegetating
B = budding
FL = flowering
FR = fruiting
P/G
P/G
P/G
P/G
P/G
P/G
P/G
V
V
V
V
V
V
V
Life Stage
B
B
B
B
B
B
B
FL
FL
FL
FL
FL
FL
FL
FR
FR
FR
FR
FR
FR
FR
M
M
M
M
M
M
M
S
S
S
S
S
S
S
M = dispersing
S = senescing
D = dead
NP = not present
D
D
D
D
D
D
D
NP
NP
NP
NP
NP
NP
NP
QA/QC and Data
Acquisition:
Train technicians well!!
Data entry:

Use data entry programs that have


Validation rules or constraints on the range or
values that can be entered
Lookup Tables
Other methods for preventing data
contamination


Double-keying of data by independent
data entry technicians followed by
computer verification for agreement
Filters for “illegal” data

Computer/database programs


Legal range of values
Sanity checks
Edwards 2000
Flow of Information in Filtering “Illegal” Data
Raw
Data
File
Table of
Possible
Values
and Ranges
Illegal
Data
Filter
Report of
Probable
Errors
Edwards 2000
Illegal-data filter in SAS
Data Checkum; Set all;
Message=repeat(“ ”,39);
If Y1<0 or Y1>1 then do; message=“Y1 is not on
the interval [0,1]”; output; end;
If Y3>Y4 then do; message=“Y3 is larger than Y4”;
output; end;
…….(add as many checks as desired)
If message NE repeat(“ ”, 39);
Keep ID message;
Proc Print Data=Checkum;
QC in the Lab
Laboratory quality control using
statistical process control charts:

Determine whether analytical system is
“in control” by examining
 Mean
 Variability (range)
All control charts have three basic
components:



a centerline, usually the mathematical
average of all the samples plotted.
upper and lower statistical control limits
that define the constraints of common
cause variations.
performance data plotted over time.
X-Bar Control Chart
UCL
26
UWL
24
Mean
22
20
LWL
LCL
18
16
0
2
4
6
8
10
Time
12
14
16
18
20
Constructing an X-Bar control
chart



Each point represents a check standard
run with each group of 20 samples, for
example
UWL = mean + 2* standard deviation
UCL = mean + 3*standard deviation
Things to look for in a control chart:
The point of making control charts is to look
at variation, seeking patterns or statistically
unusual values. Look for:




1 data point falling outside the control limits
6 or more points in a row steadily increasing or
decreasing
8 or more points in a row on one side of the
centerline
14 or more points alternating up and down
Linear trend
12
10
8
6
4
2
0
0
2
4
6
time
8
10
12
Range control charts – use when check
standards are unavailable.

To construct an R chart:


Run 15-20 samples in duplicate, and
calculate absolute value of range between
duplicate readings, and calculate mean of
ranges (R)
50% of ranges should be above the line
corresponding to 0.845R, 95% above
2.456 R, and 3.27R corresponds to 99%.
Range Control Chart
16
14
12
10
8
6
4
2
0
0
2
4
6
8
10
time
12
14
16
18
QA/QC should be
considered at all stages
of the research
process……
•Study Design
•Acquisition/Data Entry
•Data documentation
QA/QC and metadata




Check data table attribute names—are
they reasonable?
Are variables defined?
Are missing values coded?
Are metadata complete?
Quality Assurance:

Graphical and statistical methods can be
used to detect:


Outliers
Influential points in regression analysis
Outliers


An outlier is “an unusually extreme value for
a variable, given the statistical model in use”
The goal of QA is NOT to eliminate outliers!
Rather, we wish to detect unusually extreme
values.
Edwards 2000
Outlier Detection

“the detection of outliers is an intermediate
step in the elimination of [data]
contamination”


Attempt to determine if contamination is
responsible and, if so, flag the contaminated
value.
If not, formally analyse with and without
“outlier(s)” and see if results differ.
Edwards 2000
Methods for Detecting Outliers

Graphics





Scatter plots
Box plots
Histograms
Normal probability plots
Formal statistical methods


Kolmogorov-Smirnov test of normality
Grubbs’ test
Edwards 2000
X-Y scatter plots
of gopher tortoise
morphometrics
Michener 2000
Example of
exploratory data
analysis using SAS
Insight®
Michener 2000
Box Plot Interpretation
Pts. > upper adjacent value
Upper adjacent value
Interquartile
range
Upper quartile
Median
Lower quartile
Lower adjacent value
Pts. < lower adjacent value
Box Plot Interpretation
IQR = Q(75%) – Q(25%)
Interquartile
range
Upper adjacent value = largest
observation < (Q(75%) + (1.5 X
IQR))
Lower adjacent value = smallest
observation > (Q(25%) - (1.5 X
IQR))
Box Plots Depicting Statistical Distribution of
Soil Temperature
Statistical tests for outliers assume that
the data are normally distributed….
CHECK THIS ASSUMPTION!
Normal density and Cumulative Distribution Functions
0. 0.2 0.4 0.6 0.8 1.0
0. 0.1 0.2 0.3 0.4
m
u
a
ll
a
F
ti v
re
eq
N
u
o
e
r
n
m
cy
a
-3
-2
-1
0
1
2
3
Y
-3
-2
-1
0
1
2
3
Y
Edwards 2000
Y
7 8 9 10 1 12
Normal Plot of 30 Observations from a Normal Distribution
es
-2
-10
of
1
2
Edwards 2000
S tandard
Normal Plots from Non-normally Distributed Data
- 2
1
0
1
2
Y
0. 1.0 2.0 3.0
Y
01234
e
s
ft
k
-t
e
ru
w
e
n
d
ca
d
ti
e
s
d
t ri
N
b
u
o
t
rm
i
o
-2
1
0
1
2
Q
of
uan
S
t
andar
t
iles
of
d
S
N
t
andar
or m
a
- 2
1
0
1
2
Y
2 4 6 8 10 12 14 16
Y-5 0 5
v
o
y
n
-t
ta
a
i
m
l
e
d
i
n
a
d
t
i
s
e
t
d
ri
b
N
u
o
t
rm
i
o
-2
1
0
1
2
Q
of
uan
S
t
andar
t
iles
of
d
S
N
t
andar
or m
a
Edwards 2000
Grubbs’ Text is a Statistical
Test to determine if a point is
an outlier.
Grubbs’ test for outlier detection:
Tn = (Yn – Ybar)/S
where Yn is the possible outlier,
Ybar is the mean of the sample, and
S is the standard deviation of the sample
Contamination exists if Tn is greater than T.01[n]
Example of Grubbs’ test for outliers:
rainfall in acre-feet from seeded clouds
(Simpson et al. 1975)
T26
4.1
32.7
118.3
200.7
274.7
489.1
1656.0
= 3.539 >
7.7
17.5
31.4
40.6
92.4
115.3
119.0
129.6
198.6
242.5
255.0
274.7
302.8
334.1
430.0
703.4
978.0
1697.8
2745.6
3.029; Contaminated
Edwards 2000
But Grubbs’ test is sensitive to non-normality……
Checking Assumptions on Rainfall Data
Contaminated
500
150
0
250
00
Quan
r a in
t
i
f
l
a
es
ll
-2
-1
0
1
2
of S
t
andar
p
l
l
o
g
1
0
(
r
a
i
n
f
)
0.5 1.5 2.5 3.5
l
o
g
1
0
(ra
d
.i
n
N
fa
o
l
rm
l
)
a
l
0 2 4 6 8 10
p
l
r
a
i
n
f
l
0 50 150 250
ra
i
n
b
f
.
a
N
l
l
o
rma
l
0 5 10 15 20
a
.
Normally
distributed
0.
1.
5
2.
5
3.
55
-2
-1
0
1
2
og10
Quan
(r
t
i
ai
les
nf
a
of
ll)
S
t
andar
Edwards 2000
Simple Linear Regression:
check for model-based……


Outliers
Influential (leverage) points
Influential points in simple linear
regression:

A “leverage” point is a point with an
unusual regressor value that has more
weight in determining regression
coefficients than the other data values.
Edmonds 2000
Influential Data Points in a Simple Linear Regression
p
y
50 10 150
l ev
erage
outl i er
and
l ev
erage
p
outl i er
0
10
20
30
40
50
Edwards 2000
x
Influential Data Points in a Simple Linear Regression
y
50 10 150
l ev
erage
p
outl i er
an
l ev
erage
p
outl i er
0
10
20
30
40
50
x
Edwards 2000
Influential Data Points in a Simple Linear Regression
y
50 10 150
l ev
erage
p
outl i er
an
l ev
erage
p
outl i er
0
10
20
30
40
50
x
Edwards 2000
Influential Data Points in a Simple Linear Regression
y
50 10 150
l ev
erage
p
outl i er
an
l ev
erage
p
outl i er
0
10
20
30
40
50
x
Edwards 2000
Brain weight vs. body weight, 63 species of terrestrial
mammals
Leverage pts.
b
r
a
i
n
w
e
g
h
t
(
)
0 10 20 30 40 50
A fri c an
s i an
E l eph
E l ephant
“Outliers”
M
an
0
2000
4000
6000
body
Edwards
2000
wei ght
(
k
g
Logged brain weight vs. logged body weight
l
o
g
1
0
b
r
a
i
n
w
e
h
t
(
g
)
-1 0 1 2 3
E l ephant
M
an
Chi nc hi l l a
“Outliers”
m i s punc hed
10
p
-2
-1
0
1
2
3
4
body
Edwards 2000
wei gh
Outliers in simple linear
regression
Observation
62
250
Phosphatase
200
150
100
50
0
0
50
100
150
200
250
Beta-Glucosidase
300
350
400
450
Outliers: identify using
studentized residuals

Contamination may exist if
|ri| > t
 = 0.01
/2, n-3
Simple linear regression:
Outlier identification
n = 86
t/2,83 = 1.98
Simple linear regression:
detecting leverage points
hi = (1/n) + (xi – x)2/(n-1)Sx2
A point is a leverage point if hi > 4/n,
where n is the number of points used to fit
the regression
Regression with leverage point:
Soil nitrate vs. soil moisture
Regression without leverage point
Observation 46
Output from SAS:
Leverage points
n = 336
hi cutoff
value:
4/336=0.012
The Key to QA/QC:
Make sure all personnel collecting,
entering, and analyzing data have a stake
in data quality!
Questions?