“Misrepresenting Quality Data through Incorrect
Statistical Applications– A Statistical Quality Control
(SQC) Case Study”
Tim McCoy: McNair Scholar
Your
Picture
Here
Dr. Richard.Yearout: Mentor
Department of Mathematics
Abstract
Statistical Process Control (SPC) is an essential element for quality assurance. A major SPC tool is the
control chart. The investigator examined a western North Carolina manufacturer who was using incorrect
control chart methodology. Client specifications required strict adherence to specified tolerances. Initial
observations uncovered malpractice that appeared to be the result of ignorance and economic constraints.
It was noted that systems used to measure quality performance had misapplied accepted statistical
applications. Recognizing the risk of misapplication of SPC, the investigator examined the tradeoff between
continuing the present practice versus adhering to statistically sound 6 control. The principle error was
constructing control charts without statistically calculated control limits, which makes the current state of
statistical stability not only unknown, but also noninferable. Thus, validation of process capability cannot
be assured. After a full discussion explaining why “specification” charts with sample averaging are
inappropriate for process monitoring, proposed solutions with appropriate examples are recommended for
quality and industrial engineers.
Introduction
Since international trade has become an essential component of manufacturing strategy, quality
has surfaced as one of production's main foci. In today’s international environment, Statistical Process
Control (SPC) is a fundamental element of any organization’s quality master plan. One of the main tools of
SPC is the control chart. Originally introduced by Dr. Walter Shewhart in the 1920’s, the control chart has
revolutionized the way companies in both manufacturing and service industries monitor quality 7. This
study examines a specific case where a western North Carolina (wNC) manufacturer was utilizing incorrect
control chart methodology.
Client product specifications require all vendors, including this wNC manufacturer, to implement
the Quality System 9000 (QS-9000), which is the automotive industry’s supplement to the International
Standards Organization series 9000 (ISO-9000). In other words, QS-9000 certified suppliers are more than
compliant with ISO-9000 guidelines. Initially this organization’s malpractices appeared to be the results of
the manufacturer’s ignorance and economic rationale for training and sampling costs.
Correctly applying statistical control charts has been shown in saving resources by reducing scrap
and re-work, not to mention saving potential external cost from product failure and contract loss. The
critical error in this case study is the manufacturer using control charts without statistically calculated
control limits. This makes the current state of statistical stability not only unknown, but non-inferable. Thus
validation of process capability cannot be assured. “The accepted practice in the automotive industry is to
calculate capability only after a process has been demonstrated to be in a state of statistical control.” 1
In addition to the process capability issue, it appeared that the several internal and external
systems used to measure the organization’s quality performance were distorting the applications of
accepted statistical practices. Ambiguous requirements within the company’s quality system allowed the
organization to incorrectly implement and guarantee obligatory quality levels. The specific customer
requirements in question will be analyzed according to QS-9000 criteria to determine the degree of
misinterpretation from the data derived from quality inspections.
First, an introduction to control charts and their various types is needed. There is a dichotomy
among type of control charts, being variable charts and attribute charts. Variable charts are used for
numeric data; characteristics that can be measured. Attribute charts are used for categorical data; pass/fail,
ok/nok characteristics. The formulation of the calculated control limits differ for either type and become
further differentiated depending on the situation. Variable control charts were of interest in this
investigation. The basic idea of variable charts is to compare process characteristics such as the mean (in
the x-bar chart) and the spread (in the R chart) to its natural variability in the format of a visual display. The
idea is to produce a signal that indicates when to take action on the system and when to leave the process
alone. By taking samples of subgroups and plotting their means (or range) over time, the charts can use
statistical theory to signal when an event with low probability happens, such as when a mean is outside
3from the average mean. It should be noted this is exactly where the calculated control limits are placed
on an x-bar chart so that when a point crosses over a control limit a signal is made that the process has
shifted. Refer to the QS-9000 form in Figure 5 for examples of an x-bar and R chart and methodology of
data collection.
Of critical importance for this investigation is the nature of specific product being made. Type II
error (consumer's risk), as it applies to the specific industry cannot be overlooked. In strict statistical terms,
Type II error is the probability of failing to reject a null hypothesis, when in fact the null hypothesis was
false6. In practical terms, it is “keeping and shipping a bad part” (as opposed to Type I error which is
“throwing away a good part”). Assumed probabilities do not match up with actual practiced probabilities of
Type II errors for this situation. How exactly was the control chart being misused here? What are the
implications of such a malpractice? After analysis and discussion, the proposed solutions are given to
consider more appropriate control chart methods such as the standard x-bar and R charts, and alternative
control charts for individual measurements.
Observations of Incorrect Charting Practices
In midst of daily operations, the investigator monitored charts in the facility. While recording limit
changes and chart attributes, the investigator noticed that the variable charts had no calculations of control
limits that are used in the typical control chart, depicted in Figure 1 3. The charts were being used with
specification limits, thus nullifying the investigative effect on the variation in the process, the central goal
of the control chart. Also, using sample averaging and plotting averages against specification limits for
characteristics is hazardous.
Figure 1: Control Chart vs. Observed Specification Chart Situation
Control charts are formulated such that “too much” or “special cause” variability in the process
causes an out-of control situation. Using charting based on limits with specifications does not allow similar
answers to be justifiably formed from an out-of-specification situation. True, both are undesired and while
specifications are what the customer requires in the end, warning flags are raised in cases of non-controlchart monitoring. One of which is the validity of Process Capability studies, where assumptions of “in
control” are required.
By using specification (spec) charts instead of control charts, this cannot be assumed. Thus
validity of requalifications and other various Q.A. efforts are in question. Monitoring charts that raise
warning signals only when near or cross over specifications is dangerous because: 1) Charts with
specifications as limits and sample averaging allow out-of-specification parts to be made while the charts
do not reflect this, 2) there may exist an out of control situation as far as too much variation within the
process, regardless of closeness to specifications, 3) and assumptions required for other SPC tools like
capability studies may be violated.
Discussion of Process Capability
Process capability is defined to be the capability of the process to meet design specifications 4.
Process capability can only be determined and hold validity once a process has demonstrated a state of
statistical control, via correct control chart methods with calculated control limits. The two most common
measures of process capability are the Process Capability Ratio, C p, and the Process Capability Index, or
Cpk5. Their equations are shown here:
Cp = (USL – LSL)/ 6






Cpk = Minimum of: [ ( x - LSL)/3, (USL - x )/3]


(1)
(2)
The population (process) standard deviation is , the Upper Specification Limit is USL, the Lower
Specification Limit is LSL, and the average of the subgroup means is x-double-bar, which is the centerline
of the x-bar control chart. Note that the process standard deviation is rarely known and is replaced by
estimates. A commonly used estimate is R-bar/d2 for the process standard deviation (although other
estimates exist) 2. The R-bar and x-doublebar are taken from the control charts where the process has
demonstrated statistical control and thus can be used for process capability analysis. This is the standard
procedure for performing process capability studies.
In the present case, the state of statistical control is unknown with use of specification charts.
Thus, the validity of using R-bar/d2 as an estimate of is unfounded. In cases where the process is not in
statistical control, the Rbar/ d2 estimate will not reflect well due to the presence of special variation. This
is another crucial misapplication on behalf of the manufacturer in question, whose capability performance
is a major part of the customer’s requirements.
Motivation for Chart Revision
At the center for this motivation for revision is that specification limits do not work well with
sample averaging, because out-of-specification parts can be produced without being caught by the monitor
if the average of the sample “averages” to be in specification. It is the individual piece that needs to be
upheld to specifications, not the average of a sample of pieces. The reason the average is used on control
charts with statistically calculated limits (UCL’s and LCL’s and not USL’s and LSL’s) is because
calculating the average and plotting it against the calculated control limits allow for unbiased monitoring of
variation due to common causes. Once a sample average crosses the calculated control limit then assignable
cause should be sought. This knowledge is based upon sound statistical theory like the Central Limit
Theorem. However, in the case of using specifications as limits for charts, out-of specification parts can be
made and the chart looks in specification! “If specification limits are drawn on the average(s) chart, there is
a natural tendency to compare the subgroup averages, plotted on the chart, with the specification limits.
This sometimes leads to the false conclusion that whenever an average is within specification limits, all the
product is within specifications.”3
Consider Figure 23 where the situation depicts individual pieces being out-of-specification and the
average plots inside specifications on the chart.
Figure 2: Plots of Individual measurements vs. Averages of Five to Specification Limits
In Figure 2 there exist sample points that are in specification and out-of-specification on the low
side at the same time, which allows the average to be plotted in specification. Another potential case exists
when a group of points is outside a specification limit on the high side and out of specification on the low
side at the same time, and the average plots in. The monitor is unaware of the individual parts being out of
specification if they are only monitoring the average, especially if the chart does not explicitly disclose
engineering specifications on the chart.
Impacts of Misleading QS-9000 Requirements
How could a situation like this evolve under the strict quality system of QS-9000? Reviewing the
requirements placed on suppliers by QS-9000 may reveal the rationale. Shown in Figure 3 are the QS-9000
requirements that suppliers are expected to follow regarding process monitoring 1.
Figure 3: QS-9000 Guidelines, 3rd Edition
In no part do the requirements specifically state that control charts with calculated control limits
must be employed. Instead, the requirements leave gray areas of interpretation open, letting the supplier
themselves determine appropriate statistical techniques. QS-9000 does remark, “The supplier shall identify
the need for statistical techniques required for establishing, controlling and verifying process capability and
product characteristics.” Regarding appropriate statistical techniques, the manufacturer has not
implemented control charts but rather specification charts, which cannot verify process capability. Also in
the SPC book of QS-9000, no model is given where specification limits are being used 2. From the
introduction and throughout, control charts are being correctly demonstrated with calculated control limits.
Thus, it can be deduced that QS-9000 correctly dictates the use of control charts with calculated control
limits and not the observed situation here. But one may see how these requirements can be misinterpreted
to lead to the observed situation. Therefore, the impacts of these ambiguous requirements may lead
suppliers required to follow QS-9000 guidelines to incorrectly implement and monitor with improper SPC
techniques and practices.
Scope of Misapplication
The charting practices that were discovered did not conform to QS-9000 requirements and did not
allow the state of statistical control to be known. The manufacturer was using charts with specification
limits on every numeric characteristic being monitored (not with attribute or pass/fail characteristics). Since
a malpractice in the initial formulation of the control chart was detected, data were collected on the number
of control charts that could be included in this misapplication. Seen in Table 1 is the quantity of control
charts used at the facility, noting this was happening to all charts that used samples of three and five
individuals. Variable charts with one and two individuals were also not constructed with calculated control
limits but were not quite as bad, since they did not use sample averaging (impractical for samples of one
and not employed for two).
Table 1: Frequency of occurrence of specification charts with sample averaging
Frequency type
number of charts
% with averaging
5 times per shift
67
46.5%
3 times per shift
2
1.4%
2 times per shift
6
0.0%
1 time per shift
69
0.0%
Total
144
47.9%
The total number of characteristics under this scope is one hundred and forty-four (144). This
misapplication of the control charts occurred in situations where a sample average was computed on
monitoring characteristics. This is alarming because the supplier in question was using these kinds of charts
for their most important characteristics. Less important variables were being monitored using attribute
charts with pass/fail monitoring. Thus, this misapplication is being applied to the most critical
characteristics, since they require more than simple pass/fail monitoring. They require variable chart
monitoring, and commonly utilize destructive testing and manual off-line tests that are more costly. Thus, if
the type of monitoring is misleading insofar that the theoretical statistical foundations are not being applied
properly; this monitoring is an economic waste. This misapplication is occurring on all characteristics in
the facility that require samples of three and five. Thus, the manufacturer has deemed these characteristics
important enough to spend extra cost to ensure detection of special variation. However, this is nullified in
that the charts will allow for potential false or no alarms to be sounded when possible special variation is
present. This is because the control chart has been incorrectly implemented into the quality system.
Application of Correct Practices
The correct control chart formulation with calculated control limits was illustrated in Figure 1.
Notice the calculated control limits that are estimates of +/- 3. These limits follow correct statistical
theory by use of the Central Limit Theorem. The Central Limit Theorem discloses that a sample average
follows Normal distribution probabilities (despite whatever distribution its individuals follow) once a large
enough sample size has been achieved 6. Thus, the chance that a sample average exceeds 3 standard
deviations from the mean of sample averages is approximately 27 out of 10,000 (.0027). This probability
represents the chance that a single point on the x-bar chart goes outside a control limit. This is further
emphasized by QS-9000 in their manuals designed for QS- 9000 compliant suppliers. On their following
control chart form in Figure 4 2, the analyst is given a complete and fundamentally sound template for
producing both x-bar and R control charts for numeric characteristics. It is this template that was
misapplied in the manufacturer’s quality system. QS-9000 manuals also divulge other various control
charts for numeric variables such as x-bar and s charts, median charts, and control charts for individuals,
but none with specification limits. Directly on their x-bar and R chart form QS-9000 states process
capability (which analyzes performance to specifications) can only be determined when the process is in
(statistical) control.
QS-9000’s adoption of a sound control chart is presented in Figure 4:
Figure 4: QS-9000 form for x-bar and R control charts
Notice the QS-9000 form does include calculations of control limits for both x-bar chart and R
chart. It further discloses “rules for suspicion” such as “runs of 7” derived from probability theory which
are only valid in the use of control charts with calculated control limits, since these rules have been
formulated according to probability laws. These and other rules have no basis for corrective action on
charts without calculated control limits, which is the situation under scope here. The form assists in the
proper plotting and analysis by listing all prudent information such as applicable calculations and constants
for particular subgroup sizes, the rules for suspicion besides out of control points, and other information
like engineering specifications and sample size and frequency. By putting both x-bar and the R charts
together into one form, QS-9000 had made it easy for the two charts to be pair wise monitored, as they
should be.
Discussion
It can be deduced that using the right rules with the wrong analysis will yield inconclusive results.
This is the essence of the situation here. Engineering specifications cannot replace control limits on control
charts and result in the same benefits. The use of Shewhart control charts results in a different kind of
analysis than plotting subgroup averages against specifications. The Type II error cannot be inferred in the
latter case, which is unacceptable risk since preventing Type II error is a high priority considering the
particular industry. Performing to specifications and being in a state of statistical control are two different
goals manufacturers strive for. Customers require specifications to be met; yet manufacturers can predict
better on their process when in a state of statistical control. When quality analysis of data misleads the
manufacturer into a false confidence of satisfactory quality levels, the customer’s assurance of quality is
nullified.
In summary, the charting method with specification limits and sample averaging was happening
on important numeric variables. SPC tools were being misapplied which lead to not only the QS-9000
guidelines being misinterpreted but also to process capability being nullified. For this specific case,
inference on Type I and II errors have been misdirected and the standard “runs rules” for corrective action
are invalid since control chart limits formulated with probability laws are not being applied. It has been
shown where use of specification limits with sample averaging is bad for producing product to blueprint
tolerances and given evidence that the correct standard is adherence to correct control chart methodologies.
Control charts are only beneficial when properly applied with understanding and precision. Only then can
the benefits described by Shewhart and all subsequent authors be enjoyed.
Conclusions and Recommendations
When sound control chart methods are not used, the state of statistical control is unknown. Thus,
the manufacturer nullified any validation of obligatory process capability levels. Also, the manufacturer has
allowed a non-inferable level of Type II error to exist. This is a most dangerous practice, since the
importance of Type II error, or consumer’s risk, cannot be underemphasized due to the nature of the
industry.
For the immediate short term it is recommended that all specification charts with sample averaging
be switched to charts for individuals to avoid making a higher percentage of potential Type II errors. This is
an immediate temporary fix, with the recommended course of action being to implement x-bar and R
control charts for all numeric characteristics and determine the economic and quality level effects on scrap
and rework. Also, it is recommended that training for operators and management be provided on correct
SPC procedures to ensure no repeat of this situation. Another macro/long-term recommendation would be
revision of the QS-9000 guidelines to resolve ambiguity in interpretation and application of control chart
methods that could be dangerously misleading. Proper training and careful attention to correct procedures
can then lead to all the fruitful benefits of sound SPC practices.
References
[1] Automotive Industry Action Group (August 1994). Quality System Requirements – QS-9000 Third
Edition, AIAG, MI.
[2] Automotive Industry Action Group (1991). QS-9000 Statistical Process Control – SPC, AIAG, MI.
[3] Grant, E.L. and Leavenworth, R.S. (1996). Statistical Quality Control, 7 th Edition, McGraw-Hill
Companies, Inc., New York.
[4] Krajewski, L.J. and Ritzman, L.P. (1999). Operations Management Strategy and Analysis 5th edition,
Addison-Wesley, Reading.
[5] Lyda, M., Blackwelder, C., Yearout, R., Nelms, L. (2000). Process Capability Ratio and Process
Capability Index: When is Each Appropriate? (Two Case Studies to Illustrate Appropriate Use,
International Journal of Industrial Engineering Theory, Application, and Practice, 2000, pp. 341347.
[6] McCabe, G.P. and Moore, D.S. (2000). Introduction to the Practice of Statistics, W.H. Freeman and
Company, New York.
[7] Shewhart, W.A. (1931). Economic Control of Quality of Manufactured Product, D. Van Nostrand
Company, Inc., Princeton, N.J. Republished by American Society for Quality Control, Milwaukee,
Wis., 1980.