Ελληνικό Στατιστικό Ινστιτούτο
Πρακτικά 18ου Πανελληνίου Συνεδρίου Στατιστικής (2005) σελ.441-449
UNCERTAINTY AND CONFIDENCE IN
MEASUREMENT
Ioannis Kechagioglou
Dept. of Applied Informatics, University of Macedonia
ABSTRACT
Providing estimates of uncertainties related to engineering measurements involves the use of
statistical methods and has become a necessity for all accredited laboratories. Accreditation
bodies including the International Standards Organization (ISO) have introduced standards
that require the above procedure. Various sources and types of uncertainty are present at any
measurement undertaken but the total uncertainty at a certain confidence level has to be
estimated. The choice of appropriate probability distributions for each partial and combined
uncertainty is one of the main issues of concern. In this work the terms of standard
uncertainty, standard combined uncertainty and expanded uncertainty are clarified and a
procedure for obtaining estimates of these uncertainties is introduced. In this way a laboratory
can design its uncertainty estimation procedure, where a compromise between speed and
accuracy has to be made.
1. INTRODUCTION
Obtaining estimates of measurement uncertainty has always been a critical task in
engineering applications. In the past years however, accreditation bodies require that
accredited laboratories should produce estimates of uncertainty of their measurements
using accepted statistical and error analysis methods. Clause 5.4.6 of the ISO/IEC
17025:2000 standard, which focuses on electrical calibration procedures, is just an
example that states clearly the above requirement. Apart from the typical
requirements imposed by various national and international bodies, considering that
measurement procedures may be involved in safety critical applications, one can
understand the significance of this task.
In this work a procedure is introduced with the aim of providing guidance with the
estimation of measurement uncertainties in laboratories. Although electrical
measurements were considered at all stages of this work, the results obtained can be
easily modified to include cases of other types of measurement.
- 441 -
2. EVALUATION OF TYPE A UNCERTAINTY COMPONENTS
The uncertainty of measurement can be defined as "a parameter associated with
the result of a measurement that characterizes the dispersion of the values that could
reasonably be attributed to the measurand" [International Vocabulary of basic and
general standard terms in Metrology (1993)]. This parameter is usually a standard
deviation or the width of a confidence interval.
Experimental uncertainties that can be revealed by repeating measurements are
called random or Type A. Statistical methods can give reliable estimates of random
uncertainties through analysis of a series of observations. The relative importance of
the random component of uncertainty is high, in relation to other contributions, if
there is a significant spread in a sample of measurement results [Davis (2000)]. In this
case the arithmetic mean or average of the results should be calculated. For n repeated
independent values x1,…, xn for a quantity x, the mean value x is given by:
x=
1 n
∑xj
n j =1
The spread in the results is dependent on the instrument used, the method and
probably the person taking the measurements and can be quantified by the standard
deviation s(x) of the n values that comprise a sample of measurements taken.
In general, the standard deviation σ of a finite population with N elements can be
calculated as:
σ=
1 N
∑ ( x j − x) 2
N j =1
Given the statistical results of a single sample of n measurements, an estimate s(x)
of σ is:
s(x)=
1 n
( x j − x) 2
∑
n − 1 j =1
(1)
If more samples are obtained, each will have a different value for the arithmetic
mean and standard deviation. It follows from the central limit theorem that for large
n, the sample mean approaches the normal distribution with mean the population
mean.
If samples of n measurements are taken after the estimation of the standard
deviation of the population s, the standard deviation of the sample mean x is given
by:
s( x ) = s(x) / n
(2)
also called the standard error.
In many cases it is not practical to repeat the measurement many times during a
calibration or other measurement process. However, a prior measure of s can be used
for obtaining an estimate of the random component of uncertainty [Hurll, Willson
(2000)]. An experiment which involves a series of observations is run and an estimate
of the standard deviation is obtained and used for all subsequent measurements. The
- 442 -
instrument and method employed are therefore assessed through statistical analysis
before any measurement is performed.
In this way the standard deviation of the mean s( x ) can be obtained even if only
one measurement is taken during the actual process. Equations (1) and (2) can be used
for this purpose. Although reducing the number of measurements taken during the
measurement process is beneficial for obtaining fast throughput times, it leads to
greater standard errors therefore greater uncertainty values. Thus the sample sizes
have to be determined according to the degree of accuracy and speed required for the
process, where usually a compromise between these two parameters has to be made.
When considering electrical measurements, for example, repeatability or random
uncertainty components are mainly present due to electromagnetic interference and
other kinds of electrical noise entering the system from the environment.
3. EVALUATION OF TYPE B UNCERTAINTY COMPONENTS
Systematic or Type B components of uncertainty are associated with errors that
remain constant while a sample of measurements is taken. These are usually
determined based on scientific judgment using all the available information. This may
include manufacturer’s specifications, data provided in calibration reports and
knowledge of the measurement process.
While the normal distribution is used to describe random uncertainties, rectangular
and triangular distributions are usually assumed for systematic uncertainties. If the
upper and lower limits of an error are ±α without a confidence level and there is
reason to expect that extreme values are equally likely, the rectangular distribution is
considered to be the most appropriate (see Fig. 1). In case that extreme values are
expected to be unlikely, the triangular distribution is usually assumed [Bean (2001)].
When an estimate of the maximum deviation which could reasonably occur in
practice is determined, the value of the standard deviation has to be calculated in
order to obtain the standard uncertainty value. The standard deviations are equal to
α
3
for the rectangular and
-α
α
6
for the triangular distribution respectively.
μ-σ
μ
μ+σ
+α
Fig. 1: The rectangular distribution used to describe a Type B uncertainty
based on the maximum expected error range ±α
4. COMBINED UNCERTAINTY
Usually more than one parameter affects significantly the accuracy of a
measurement being taken. All random or systematic sources of uncertainty associated
with a measurement have to be quantified and then combined all together for the
- 443 -
calculation of total uncertainty. Before combination all uncertainty contributions have
to be expressed as standard uncertainties, which may involve conversion to the
standard deviation from some other measure of dispersion [Guide to the Expression
of Uncertainty in Measurement (1995)].
Uncertainty components evaluated experimentally from the dispersion of repeated
measurements are usually expressed in the form of a standard deviation, as described
in section 2 and therefore no conversion is needed. When an uncertainty estimate is
taken from specification data it may be in a form other than the standard deviation. A
confidence interval is often given with a level of confidence (±α at p%) in
specification sheets or calibration certificates. In this case the value α has to be
divided by the appropriate divisor that corresponds to the given confidence interval in
order to calculate the standard deviation. For example, a confidence level of 95%
corresponds to two standard deviations, for a normally distributed variable. Therefore
a division of the confidence interval by two will give us the value of the standard
deviation. Uncertainties given in the form described above are called expanded
uncertainties and will be further investigated in the next section.
For some uncertainty components, error limits in the form ±α may be estimated. In
this case the divisors 3 and 6 are used to determine the standard deviation for
rectangular and triangular distributions respectively, as described in the previous
section [UKAS (1997)]. The divisors for the most commonly used probability
distributions are summarized in Table 1.
Probability Distribution Type Divisor
Normal (confidence interval 68%)
1
Normal (confidence interval 95%)
2
Normal (confidence interval 99%)
3
Rectangular
3
Triangular
6
Table 1: Common Probability Distributions and corresponding
Di i
Once the uncertainty components have been identified, estimated and expressed as
standard deviations, the next stage involves calculation of the combined standard
uncertainty u(y). The relationship between u(y) of a value y and the uncertainty of the
independent parameters x1, x2, …, xn upon which it depends is
n
u(y) =
∑ c u( x )
i =1
2
i
2
(3)
i
where ci is a sensitivity coefficient calculated as ci= ∂ y/∂ xi, the partial derivative
of y with respect to each of the standard uncertainty components xi. This coefficient
describes how the value of y varies with changes in the parameters xi and therefore
compensates for the contribution of a certain uncertainty component in the total
uncertainty of a measurement [Barr, Zehna (1983)] Equation (3) is also known as the
law of propagation of uncertainty.
- 444 -
5. EXPANDED UNCERTAINTY
Usually a statement of confidence associated with a calculated total uncertainty is
required. In practice this is the probability that a measured value and the
corresponding uncertainty will define a range of values within which the true value of
the measurand is included. The combined standard uncertainty is in the form of one
standard deviation and therefore may not provide sufficient confidence [Laboratory
Accreditation Bureau (2001)].
For this reason the expanded uncertainty U is calculated by multiplying the
standard uncertainty by a coverage factor k as follows:
U = k u(y)
(4)
Expanded uncertainties provide intervals that encompass a larger fraction of the
measurand value distribution, compared to that of the combined uncertainty. The
result of a measurement is then reported in the form y ± U along with the confidence
level, which depends on the coverage factor. According to the requirements of
dominant accreditation bodies, the expanded uncertainty should provide an interval
with a level of confidence close to 95% [ISO/IEC (2000)].
The choice of the coverage factor k is of critical importance as it determines the
level of confidence associated with an uncertainty statement. Apart from the
confidence level required, knowledge of the underlying distributions and the number
of repeated measurements used for the estimation of random effects are issues to
consider.
In many cases the probability distribution characterized by the measurement result
y and its combined standard uncertainty u(y) is approximately normal and u(y) itself
has negligible uncertainty. If the above conditions are satisfied, a coverage factor of k
= 2 defines an interval having a level of confidence of approximately 95 percent, and
k = 3 corresponds to an interval having a level of confidence greater than 99 percent.
This situation is encountered when the standard uncertainty estimates contribute
comparable amounts to the combined standard uncertainty, random uncertainties are
estimated using a sufficient number of observations and systematic errors are
evaluated with high reliability. In this way the conditions of the Central Limit
Theorem are satisfied and normality may be assumed.
There are cases however that the combined standard uncertainty is dominated by a
single contribution with fewer than six degrees of freedom. This usually happens
when the random contribution to uncertainty is large in comparison to other
contributions and the sample size used for its estimation is small [UKAS (1997)]. It is
then probable that the probability distribution will not be normal and the value of the
coverage factor k will result in a confidence level that is smaller than the expected
one. If this is the case, the value of the coverage factor that corresponds to the
required confidence level has to be derived by considering the effective degrees of
freedom veff of the combined standard uncertainty. These can be evaluated using the
Welsh-Satterthwaite equation:
- 445 -
u4 ( y)
veff =
n
∑
c 4j u 4j ( y )
(5)
vj
The effective degrees of freedom veff are based on the degrees of freedom vj, which
are equal to the number of measurements n taken during the prior evaluation of the
variance, less 1. For systematic uncertainties, it is assumed that vj→∞ since the
probability for the quantity considered laying outside the limits is very small. The
combined standard uncertainty u(y), the sensitivity coefficient ci and the individual
component uncertainties uj(y) are also included for the calculation of veff as described
in the above equation [Guide to the Expression of Uncertainty in Measurement
(1995)].
After the value of veff is calculated using the above formula, the coverage factor kp,
where p is the confidence probability in percentage terms, can be determined based
on a t-distribution rather than a normal distribution. In general, the t-distribution is
less peaked at the center and higher in the tails compared to the normal distribution,
but it approaches it as the number of samples increases [Daniel, Terrell (1995)].
Using the t-distribution tables, the entry value that is immediately lower than the
calculated value of veff is found in the degrees of freedom section and the
corresponding value of t95 is looked-up. The coverage factor k95 is then taken equal to
the t95 value to ensure a 95% confidence level.
j =1
6. UNCERTAINTY ESTIMATION PROCEDURE
Once all uncertainty components associated with an instrument are evaluated,
manipulations have to take place on each of these values in order to obtain the
combined and expanded uncertainty of a measurement. The procedure of calculating
the expanded uncertainty from the individual component uncertainties is described in
the flowchart of figure 2.
After calculating the standard deviation of the repeatability uncertainty and
obtaining tolerance limits for uncertainties related to systematic errors, the type of the
probability distribution for each source of uncertainty has to be determined. All
uncertainties can then be expressed as standard uncertainties using a divisor from
table 1. The sensitivity coefficients ci are determined and formula (3) is then used to
find the combined standard uncertainty. In case that the standard uncertainty of the
repeatability component is more that 50% of the combined standard uncertainty, it is
assumed that the latter is described by the t-distribution rather than the normal one
[Davis (2000)]. Therefore the value of the coverage factor k is determined by looking
at the t-distribution tables for the t95 value that corresponds to the 95% level of
confidence for the effective degrees of freedom of the combined standard uncertainty,
as calculated using formula (5).
Otherwise the value of the sample size of measurements has to be increased so that
the standard deviation and hence the standard combined uncertainty of the
repeatability component is reduced. If the standard uncertainty of the repeatability
component is less than 50% of the combined standard uncertainty, normal distribution
- 446 -
Determine probability
distribution for each source
of uncertainty
Find corresponding divisor
from table 1
Determine
coefficient ci
Calculate all standard
uncertainty components
Calculate Combined
Standard Uncertainty
Is repeatability uncertainty
> 50% of the Combined
Standard Uncertainty?
Yes
Assume t-distribution for
the Combined Standard
Uncertainty
Calculate deg. of freedom
for Combined Standard
Uncertainty
No
Assume that Combined
Standard Uncertainty is
normally distributed
Find t95 value from tdistribution tables and
hence determine k
Use k=2 to calculate
normally distributed
Expanded Uncertainty
Use k= t95 to calculate tdistributed Expanded
Uncertainty
Fig. 2: Flowchart for the Expanded Uncertainty calculation procedure.
- 447 -
may be assumed and therefore a coverage factor of k= 2 may be used for a 95%
level of confidence.
Having determined the value of the coverage factor k, the expanded uncertainty for
a confidence level of 95% is then obtained by substituting the values into formula (4).
Its distribution type is assumed to be the same as the one for the combined standard
uncertainty.
Consider for example the uncertainty associated with a digital voltmeter. Type-A
uncertainty components can be quantified by taking a series of ten measurements on a
constant voltage source. Calculating the standard deviation we find the standard
uncertainty related to repeatability u(x1) = 0.002Volts. Suppose that the error
introduced due to limited resolution is ±0.01Volts. Also the manufacturer states an
inaccuracy of ±0.05 Volts due to environmental effects. Since the distributions of
both systematic inaccuracies may be assumed to be rectangular, the maximum errors
have to be divided by 3 in order to find the standard errors u(x2) = 0.0057Volts and
u(x3) = 0.029Volts. All three sensitivity coefficients can be set equal to one since all
uncertainty components contribute equally. Formula (3) is then used to estimate the
combined standard uncertainty u(y) = 0.0296Volts. Since the random component
u(x1) is less than 50% of u(y) normal distribution may be assumed. If we assume that
a confidence interval of 95% is required, we use a coverage factor k = 2 in equation
(4) and the estimated expanded uncertainty is U = 0.0592Volts.
7. CONCLUSIONS
Estimation of measurement uncertainty is a time consuming and demanding task
for laboratories. A large number of experiments as well as research are required
before all required uncertainties are obtained. This task however should not cause
delays to other primary processes and operations. The requirement for uncertainty
estimation should be considered even at the stage of work procedure and
measurement method design so that all operations are performed in such a way as to
reduce potential uncertainties and simplify their calculation. Since uncertainty
estimation is based to a large extent on information provided by instrument
manufacturers, only equipment with adequate information on associated uncertainties
should be considered for use. It was also found that the calibration and drift since last
calibration are much easier and more accurately calculated when the calibration of
instruments is contracted to the same laboratory. The evaluation of both random and
systematic uncertainties associated with measurement equipment becomes much
easier once instruments of the same type are used by all bench engineers. Therefore
consistency is required as far as subcontracting of calibrations and purchase of
equipment is considered.
The uncertainty estimation procedure presented in this paper has been
implemented using an Excel spreadsheet supported by VBA code. The procedure was
designed to facilitate its programming by setting a threshold for the decision variable.
The estimation process can therefore be automated within any laboratory and become
fully integrated within a measurement process.
- 448 -
ΠΕΡΙΛΗΨΗ
Η εκτίμηση της αβεβαιότητας που σχειτίζεται με μετρήσεις σε εφαρμογές μηχανικής
περιλαμβάνει τη χρήση στατιστικών μεθόδων και καθιστάται απαραίτητη για κάθε
διαπιστευμένο εργαστήριο. Τα σώματα πιστοποίησης, συμπεριλαμβανομένης της Διεθνούς
Οργάνωσης Προτύπων (International Standards Organization, ISO), έχουν εισαγάγει
προδιαγραφές που απαιτούν την παραπάνω διαδικασία. Διάφορες πηγές και τύποι
αβεβαιότητας παρευρίσκονται σε κάθε μέτρηση που πραγματοποιείται. Η συνολική όμως
αβεβαιότητα για συγκεκριμένο επίπεδο εμπιστοσύνης πρέπει να προσδιοριστεί. Η επιλογή
των κατάλληλων πιθανοτικών κατανομών για κάθε μερική και συνδυασμένη αβεβαιότητα
είναι απο τα κύρια ζητήματα αυτής της διαδικασίας. Σε αυτή την εργασία διευκρινίζονται οι
έννοιες της τυπικής αβεβαιότητας, της συνδυασμένης αβεβαιότητας και της διευρυμένης
αβεβαιότητας και εισάγεται μια διαδικασία για την εκτίμηση αυτών. Κατ’ αυτό τον τρόπο
παρέχεται καθοδήγηση στα εργαστήρια ώστε να σχεδιάσουν μια κατάλληλη διαδικασία
εκτίμησης της αβεβαιότητας, όπου αντιμετωπίζεται ένας συμβιβασμός μεταξύ ταχύτητας και
ακρίβειας των εκτιμήσεων.
REFERENCES
Barr D. Zehna P. (1983). Probability: Modeling Uncertainty. Addison-Wesley.
Bean M. A. (2001). Probability: The Science of Uncertainty. Brooks/Cole.
Daniel, Terrell (1995). Business Statistics, 7th Edition. Houghton Mifflin.
Davis P. W. (2000), ERA Technology Course Notes on Quality Systems in UKAS
Accredited Laboratories.
General requirements for the competence of testing and calibration laboratories
(2000). ISO/IEC 17025:2000.
Guidance for Documenting and Implementing ISO/IEC 17025 (2001), Revision 3.
Laboratory Accreditation Bureau.
Guide to the Expression of Uncertainty in Measurement (1995). Vocabulary of
Metrology, Part 3. BSI.
Hurll J. Willson S. (2000), Uncertainty of Measurement Training Course Material,
UKAS.
International Vocabulary of basic and general standard terms in Metrology. ISO,
Geneva, Switzerland 1993 (ISBN 92-67-10175-1)
Mandel J. (1984). The Statistical Analysis of Experimental Data. Dover Publishers,
New York, NY.
Taylor J. R. (1997). An Introduction to Error Analysis, 2nd Edition. University
Science Books.
The Expression of Uncertainty and Confidence in Measurement (1997). UKAS
Publications.
- 449 -