Machine Characterization for Precision Lathes
Accuracy and Precision Analysis
Brent Baugh
March 15th, 98
Summary
Tolerances drive many of the costs associated with the production of cells. Understanding
tolerances has lead to the statistical analysis of several features which indicated that high end
precision machines are capable of repeatedly producing parts that form a normal distribution
with standard deviations on the order or 1.5 to 2 microns. However, the mean dimension fell far
outside the nominal target for various features. This essential indicates a systematic error in the
process. These shifts appear to be most strongly linked to the geometry of the feature in
question, as well as the choice of inspection methods.
Background
This study was initiated because the tolerances of the NLC DDS cells are driven largely by
assumptions made concerning the ability of a machine to accurately and precisely machine a
feature repeatedly. Very little information is available in the literature concerning modern day
machining capabilities in this respect. This lack of information is the result of machining
performance being a function of the machine, the environment, the material, the tool, the CNC
code and control systems, and finally the feature itself. Due to the high number of parameters
that affect the repeatability, a new study must be conducted for nearly every feature.
Although extensive data does not exist for the DDS 3 or DDS 5 cells, a comparable part is the
NLCTA .9m Injection Section cells. These cells were produced at Robertson Precision, Inc. and
were extensively checked for dimensional accuracy here at SLAC. The Leitz coordinate
measuring machine (CMM) was used to evaluate nearly every feature on all the cells. Data was
collected for 211 cells.
Three separate features have been statistically characterized for this study; the cell’s outer
diameter (OD), the inner diameter of the cavity (IDC), and the inner diameter of the iris (IDI).
Please note that for the following discussion, the dimensions will be listed in English units
followed by metric in parentheses.
Outer Diameter of Cell
The outer diameter of the cells was held constant for all cells in the stack. The nominal
diameter was 1.9750 inches (50.165 mm) with a tolerance of plus zero, minus 0.0005 inches
(12.7 m). Bill Robertson indicated that during production this feature was measured while on
the machine with a set of digital micrometers that were regularly calibrated on a gauge block.
He also indicated that in all cases, he targets the center of the tolerance band for production.
This implies that for this particular feature, his machines and machinist were attempting to
produce parts that had an offset of -0.00025 inches (6.4 m) from the nominal.
The following chart is a histogram indicating the range of diameters found on the parts. The
charts also indicate the “target” dimension, and the upper/lower tolerance limits.
Tolerance Offset from Nominal Cell OD
(1.975 inches +0.0000, -0.0005)
40
Average: -0.00027 in. (-7 m)
Offset from Target: -0.00002 in (-0.5 m)
Standard Deviation: 0.00006 in. (1.5 m)
35
# of Cells
30
25
20
15
10
5
-0.00002
-0.00006
-0.0001
-0.00014
-0.00018
-0.00022
-0.00026
-0.0003
-0.00034
-0.00038
-0.00042
-0.00046
-0.0005
0
Tolerance Band
Nominal Target
Tolerance Limits
The data is a relatively well behaved Gaussian distribution which is centered almost perfectly on
the target. In fact, the mean offset from this target is only 20 millionths of an inch (0.5
microns). Additionally, the standard deviation is quite small. It should be noted that all parts
are well within the tolerance band.
Inner Diameter of the Cavity
The diameter of the cavities varied across the length of the section. A typical dimension for the
cavity is about 0.85 inches (21.6 mm). Regardless of the nominal diameter, the tolerance for all
cavities was plus or minus three ten-thousandths, 0.0003, of an inch (7.6 m). Bill Robertson
indicated that this feature was qualified using go/no-go pins chosen to correspond to the upper
and lower limits of the tolerance band.
Tolerance Offset from Nominal Cavity ID
(Cavity approx. 0.85 inches +0.0003, -0.0003)
35
30
Average: -0.00012 in. (-3 m)
Standard Deviation: 0.00008 (2 m)
# of Cells
25
20
15
10
5
0.0003
0.00026
0.00022
0.00018
0.00014
0.00010
0.00006
0.00002
-0.00002
-0.00006
-0.0001
-0.00014
-0.00018
-0.00022
-0.00026
-0.0003
0
Tolerance Band
Nominal Target
Tolerance Limits
Again, a normal distribution is revealed. However, this curve is offset from the nominal target
by a little more than one ten-thousandth of an inch (3 microns). This shift occurs in the negative
direction indicating that, on average, the cavities are smaller than nominally desired.
With only one exception, all parts fall inside the tolerance band.
Inner Diameter of the Iris
The iris diameter also varied across the length of the section. In this case, a typical iris
dimension is 0.40 inches (10 mm). The tolerance on all iris diameters was plus or minus three
ten-thousandths, 0.0003, of an inch (7.6 m). Again, pin gauges were used for quality
assurance during production.
Tolerance Offset from Nominal Iris ID
(Iris approx. 0.40 inches +0.0003, -0.0003)
35
30
Average: 0.00036 in. (9 m)
Standard Deviation: 0.00008 (2 m)
# of cells
25
20
15
10
5
More
0.00058
0.00054
0.0005
0.00046
0.00042
0.00038
0.00034
0.0003
0.00026
0.00022
0.00018
0.00014
0.00010
0.00006
0.00002
-0.00002
-0.00006
-0.0001
-0.00014
-0.00018
-0.00022
-0.00026
-0.0003
0
Tolerance Band
Nominal Target
Tolerance Limits
The resulting normal distribution is offset from the nominal target by .00036 inches (3 microns).
This shift occurs in the positive direction indicating that, on average, the cavities are larger than
nominally desired. Note that this is the opposite of the previously noted trend with respect to
cavity diameters. The shift is also large enough in magnitude to cause most of the inspected
irises to fall outside the tolerance band.
Discussion
The systematic error noted in the preceding statistical analyses must be understood to insure the
quality of cells produced for the NLC. Several different factors could be, and probably are, at
work to cause the error.
A DFM tool known as the Cause-Effect diagram was employed to identify potential sources of
the offset.
Geometry
Human
Error
Machine
Tool wear
Internal
Rectilinear
Choose Wrong
Target
Spindle
Eccentricity
External
Curvilinear
Systematic
Error
+ value, - 0
+ 0, - value
+ or - value
Tolerance
Digital
Go/No Go
Calibration error
In-Process
Humidity
Analog
Temperature
QC Machine Error
QC Technique
Environment
Cause - Effect Diagram
The Cause - Effect diagram is a tool used to help brainstorm and map all the potential sources of
a given outcome. In this case, the major sources of the shift have been identified as feature
geometry, human error, machine error, tolerance impact, QC technique, and the environment.
Internal to all of these, more specific details are identified.
Noting that the best results come from the OD we can identify a number of important differences
in production that might result in systematic error. These differences are summarized in the
following table.
Geometry
QC Technique
Tolerance
OD
External
Digital
+ 0, - 0.0005 inches
ID’s
Internal
Go/No-go
+ or - 0.0003 inches
In this case, Robertson maintains that the center of a tolerance band is always his goal, so the
tolerance difference should not skew the process in one direction. Also, the OD has a tighter
tolerance than the IDs so we would expect if this were the only difference, to see problems
reversed (i.e. out of tolerance parts for the OD, not the IDs). The geometry on the other hand
has actually influence the choice of QC technique, indicating that the Cause-Effect diagram may
not be as ‘linear’ as shown above. Regardless, it seems likely that the cause of the systematic
error is probably due to the QC method. Obviously, digital micrometers are superior for
identifying proper depths of cut to achieve the desired diameter, and the data supports this.
In trying to decipher the cause of the cavity diameter being undersize while the iris was
oversized, the major difference appears to be in geometry. Here, the cavity is essentially
rectilinear and entirely machined from one side, while the iris has a curvilinear profile which
drives the necessity of machining both front and back faces. Machining errors probably
increase due to this additional operation.
The pin inspection process is most definitely affected by the different geometries. When
inspecting the cavity, it is known that if the ‘go’ pin is inserted (‘go’ pin being the lower limit of
the tolerance band) and the no-go does not fit, the part is acceptable. If small passes are being
taken, and the pin is tried repeatedly until it fits, it seems natural that most of the diameters will
be just larger than the smallest acceptable diameter. Therefore, we naturally see a shift toward
the ‘go’ gauge.
In the case of the iris, the pin technique again functions the same way. However, it may be
easier to lodge the pin in the iris. This would drive larger diameters and perhaps reverse the
skew. Alternatively, when inspected with the CMM it is imaginable that the probe tip was
either slightly above or below the cusp of the curve. If this were the case, the reading from the
CMM would indicate an overly larger diameter.
Case One: Pin gauge inspection of cavity
Case Two: Pin gauge inspection of iris
Case Three: CMM inspection of iris
However, these are just a few ideas, not necessarily the true source of the problem.
Conclusions
It appears the largest source of systematic error results from the choice of QC methods.
Geometry is closely tied to the QC selection and may prohibit certain options for inspection. If
good methodologies for inspection may be developed, it seems likely that existing precision
machines will be able to consistently generate parts very accurately, and with a precision of a
few microns.
Raw Data
Cell
OD Offset
Iris Offset
49
-0.00025
-0.000143
0.000354
50
-0.00029
-0.000121
0.000385
51
-0.00027
-0.000151
0.000375
52
-0.00023
-0.000168
0.000324
53
-0.00024
-0.000155
0.000321
54
-0.00026
-0.000075
0.000355
55
-0.00029
-0.000019
0.000439
56
-0.0003
-0.000093
0.000349
57
-0.0003
-0.000052
0.000289
58
-0.00032
-0.000023
0.000356
59
-0.00033
0.000008
0.000501
60
-0.00032
-0.000116
0.000327
61
-0.0003
-0.000093
0.000344
62
-0.0003
-0.000019
0.000384
63
-0.00026
-0.000113
0.000421
64
-0.00031
-0.000116
0.000376
65
-0.00027
-0.000137
0.000388
66
-0.00023
-0.000143
0.000336
67
-0.0004
-0.000197
0.000425
68
-0.00031
-0.000198
0.00032
69
-0.00029
-0.000201
0.000331
70
-0.00029
-0.000196
0.000362
71
-0.00036
-0.00002
0.000517
72
-0.00035
-0.000138
0.000358
73
-0.00015
-0.000098
0.000351
74
-0.00021
-0.000108
0.000348
75
-0.0003
-0.000011
0.000391
76
-0.00036
0.000068
0.000488
77
-0.00025
-0.00015
0.000268
78
-0.00025
-0.000157
0.000138
79
-0.00017
-0.000198
0.000244
80
-0.00026
-0.000225
0.00037
1
-0.00029
Cavity
Offset
-0.000132
2
-0.00033
-0.000081
0.000268
3
-0.00019
-0.000048
0.000348
4
-0.00035
0.000018
0.000349
5
-0.00026
-0.00005
0.000248
6
-0.00025
-0.000046
0.00021
7
-0.00027
-0.000201
0.000128
8
-0.00026
-0.000178
0.000198
9
-0.00032
0.000022
0.000367
10
-0.00036
0
0.000394
11
-0.00038
0.000037
0.000448
12
-0.0003
0.000096
0.000414
13
-0.00035
0.000093
0.000433
14
-0.00031
0.000139
0.000496
15
-0.00039
0.000139
0.000522
16
-0.00033
0.000005
0.000338
17
-0.00039
0.000014
0.000332
18
-0.00034
0.00003
0.000354
19
-0.00039
-0.000066
0.000249
20
-0.00037
-0.000102
0.000341
21
-0.00036
-0.000118
0.000292
22
-0.00037
-0.000131
0.000274
23
-0.00038
-0.000064
0.000397
24
-0.00037
-0.000153
0.000283
25
-0.00036
-0.000077
0.000351
26
-0.00036
-0.000076
0.000339
27
-0.0004
-0.000049
0.000423
28
-0.00033
-0.000029
0.000329
29
-0.0004
-0.000055
0.000345
81
-0.0002
-0.000161
0.000392
30
-0.00034
-0.000068
0.000313
82
-0.00024
-0.000137
0.000444
31
-0.00015
-0.00017
0.00007
83
-0.00018
-0.000135
0.000421
32
-0.0001
-0.000078
0.000407
84
-0.00024
0.000252
0.000677
33
-0.00022
-0.00001
0.00039
85
-0.00021
-0.000137
0.000497
34
-0.00036
-0.000123
0.000357
86
-0.00025
-0.000032
0.000493
35
-0.00039
-0.000134
0.000401
87
-0.00021
-0.000108
0.000406
36
-0.00028
-0.000046
0.000452
88
-0.00028
-0.000057
0.000512
37
-0.00032
-0.00003
0.000437
89
-0.00023
-0.000069
0.000475
38
-0.00028
-0.000055
0.000375
90
-0.00026
-0.000135
0.00047
39
-0.00026
-0.000079
0.000498
91
-0.0002
-0.000137
0.000259
40
-0.00023
-0.000026
0.000371
92
-0.00024
-0.000195
0.000295
41
-0.0003
-0.000043
0.000394
93
-0.0002
-0.000091
0.000402
42
-0.0003
-0.000087
0.000428
94
-0.0002
-0.000067
0.000324
43
-0.00022
-0.000104
0.000379
95
-0.0002
-0.000121
0.00025
44
-0.00032
-0.00007
0.000338
96
-0.00025
-0.000131
0.000382
45
-0.00032
-0.000084
0.000294
97
-0.00025
-0.000146
0.000411
46
-0.00025
-0.000107
0.000365
98
-0.00024
-0.00016
0.000371
47
-0.00022
-0.000141
0.000339
99
-0.00026
-0.000114
0.000403
48
-0.00025
-0.000161
0.000341
100
-0.00027
-0.000153
0.00047
0.00034
101
-0.00023
-0.000105
0.000228
153
-0.00019
-0.000214
0.000301
102
-0.00026
-0.00019
0.000398
154
-0.00019
-0.00022
0.000183
103
-0.00029
-0.000151
0.000441
155
-0.00019
-0.000227
0.000252
104
-0.00031
-0.000167
0.000361
156
-0.00015
-0.00024
0.000222
105
-0.0003
-0.000089
0.000506
157
-0.0002
-0.000146
0.000213
106
-0.00032
-0.000136
0.000417
158
-0.0001
-0.000142
0.000274
107
-0.00033
-0.000119
0.000414
159
-0.00016
-0.000144
0.000352
108
-0.0003
-0.000146
0.000414
160
-0.00019
-0.00012
0.00035
109
-0.00013
-0.000117
0.000432
161
-0.00014
-0.000308
0.00022
110
-0.0002
-0.000207
0.000073
162
-0.00018
-0.000093
0.000352
111
-0.00018
-0.000213
0.000097
163
-0.00022
-0.000046
0.000353
112
-0.0002
-0.0002
0.000103
164
-0.00025
-0.000116
0.000393
113
-0.00025
-0.000239
0.000327
165
-0.00022
-0.000135
0.000372
114
-0.00024
-0.000173
0.000309
166
-0.00024
-0.000159
0.000334
115
-0.00026
-0.000118
0.000459
167
-0.00023
-0.000133
0.000342
116
-0.00027
-0.000136
0.000378
168
-0.00026
-0.000151
0.000388
117
-0.00027
-0.000116
0.000421
169
-0.00025
-0.000149
0.000346
118
-0.00026
-0.000099
0.00049
170
-0.00024
-0.000195
0.000406
119
-0.00028
-0.000116
0.000422
171
-0.00027
-0.000157
0.000386
120
-0.00026
-0.000192
0.000286
172
-0.00018
-0.000152
0.000385
121
-0.00027
-0.000255
0.000323
173
-0.00019
-0.000053
0.000421
122
-0.00029
-0.000149
0.000354
174
-0.00016
-0.000112
0.000221
123
-0.00031
-0.00015
0.000362
175
-0.00021
-0.00025
0.000315
124
-0.0003
-0.000146
0.000339
176
-0.0002
-0.000251
0.000382
125
-0.00029
-0.000103
0.000394
177
-0.00028
-0.000142
0.000416
126
-0.00028
-0.000085
0.000405
178
-0.00023
-0.000124
0.00022
127
-0.00024
-0.000135
0.000327
179
-0.00026
-0.000229
0.000396
128
-0.00026
-0.000119
0.000438
180
-0.00026
-0.000103
0.00043
129
-0.0003
-0.000121
0.000441
181
-0.00027
-0.000154
0.000343
130
-0.00032
-0.000132
0.00046
182
-0.00025
-0.000165
0.000364
131
-0.00027
-0.000109
0.000451
183
-0.00027
-0.000155
0.000441
132
-0.00023
-0.000196
0.000304
184
-0.00025
-0.000166
0.00041
133
-0.0003
-0.000004
0.000459
185
-0.00028
-0.000161
0.000441
134
-0.00032
-0.000176
0.000378
186
-0.00025
-0.000204
0.000434
135
-0.00027
-0.000111
0.000338
187
-0.00029
-0.000211
0.000426
136
-0.00022
-0.000129
0.000341
188
-0.00028
-0.000196
0.000349
137
-0.00023
-0.000223
0.000278
189
-0.00029
-0.000153
0.000313
138
-0.00021
-0.00015
0.000327
190
-0.00024
-0.000169
0.000208
139
-0.00021
-0.000069
0.000399
191
-0.00027
-0.000222
0.000374
140
-0.00038
-0.000099
0.000293
192
-0.00023
-0.000132
0.000195
141
-0.00027
-0.000006
0.000364
193
-0.00027
-0.000253
0.000336
142
-0.0002
-0.000193
0.000371
194
-0.00028
-0.000185
0.000396
143
-0.00022
-0.000158
0.000226
195
-0.00026
-0.000253
0.000278
144
-0.00021
-0.000147
0.000268
196
-0.00028
-0.000074
0.00032
145
-0.00026
-0.000197
0.000363
197
-0.00026
-0.00008
0.000373
146
-0.00025
-0.000142
0.000339
198
-0.00023
-0.000096
0.000451
147
-0.00032
-0.00024
0.000358
199
-0.00023
-0.000091
0.000395
148
-0.00033
-0.000204
0.00037
200
-0.00022
-0.000218
0.000288
149
-0.00034
-0.00017
0.000375
201
-0.00026
-0.000056
0.000346
150
-0.00028
-0.000152
0.000436
202
-0.00019
-0.000063
0.000354
151
-0.00015
-0.000251
0.000264
203
-0.00033
-0.000207
0.000367
152
-0.00038
-0.000173
0.000422
204
-0.00031
-0.000097
0.000401
205
-0.00022
-0.000173
0.000317
206
-0.00026
-0.00017
0.000307
207
-0.00018
-0.000175
0.000376
208
-0.00015
-0.000276
0.000358
209
-0.00025
-0.000198
0.000348
210
-0.00026
-0.000152
0.000275
211
-0.00029
-0.000045
0.000346