Computer Experiments to Predict Propagation of Variation
An Aircraft Engine Blade Assembly Case Study
Presented at the Fall Technical Conference
King of Prussia, PA
October 2002
By
David Rumpf
and
GE Aircraft Engines
(781) 594-5508
fax (781) 594-0954
1000 Western Ave.
Lynn, MA 01910
USA
David.Rumpf@ae.ge.com
William J. Welch
Department of Statistics and Actuarial Science
(519) 888-4567 x5545
fax (519) 746-1875
University of Waterloo
Waterloo, Ontario N2L 3G1
Canada
wjwelch@uwaterloo.ca
Special thanks to Robert Shankland, GEAE Engineering for his patience and
expertise in running the analytic computer stress model.
David Rumpf, Statistician
GE Aircraft Engines
Page 1
Blade Assembly Stress Study
X1=interference
fit
Y1a,b= notch mean
and alternating
stress
X2=CP rabbet
interference fit
Y2a,b = rabbet
fillet mean and
alternating
stress
X3=CP drop
a function of
three drops
David Rumpf, Statistician
Goals:
1) A DFSS (design for six sigma) design
which meets LCF life requirements
2) Tolerance requirements for X1, X2, X3
What was available:
•
A computer model which evaluates
stress for any specific set of X1, X2 and
X3 values. Run time ~ 1 hour
•
Two stress points, Y1 and Y2 as shown.
•
Two outcomes for each location, mean
stress and alternating stress.
•
Alternating is bigger driver for part
low cycle fatigue life.
What was needed:
•
Non-linear transfer functions for Y1a,b
and Y2a,b versus X1, X2 and X3 which
could be used for multiple Monte Carlo
models, ~1000 iterations each, for stress
versus tolerances on X1, X2 and X3.
GE Aircraft Engines
Page 2
Six Sigma, Producibility and Robust Design
“sigma level”
Typical
Historical
Situation
Process
Tolerance meets
Customer Need
2
xbar +/- 3 s
only +/- 2 s fit within tol
Quality plan relies on inspection. Expect to have rework, scrap and MRB activity.
Reaching six sigma goal requires combined Manufacturing/Design Effort
“sigma level”
Combined
6
Engineering,
Manufacturing
6 s Goal
Improved
Manufacturing
Process
Robust Design allows
Wider Tolerance to
meet Customer Need
xbar +/- 3 s
+/- 6 s will fit within tol
Quality plan focus on parameter control and process monitoring. First time yield 100%.
David Rumpf, Statistician
GE Aircraft Engines
Page 3
Statistical/Design of Experiment Opportunities
Manufacturing Process Improvement:
• Screening designs
• Factorial designs
• Leveraging
• EVOP
• Quality Improvement metrics
• Review by Vice-president
Engineering, meeting Customer Needs and improving producibility:
• Quality Function Deployment
• Voice of the Customer
• Robust Design:
• Screening
Focus of this presentation
• Factorial
• Response Surface Designs
• Producibility Scorecards
• Review by Vice-president
David Rumpf, Statistician
GE Aircraft Engines
Page 4
Robust Design
Y = Stress or Useful Life
X’s are parameters which impact Y which could include:
• Part Key Characteristic values
• Environment
• Mating part Key Characteristic values
• Customer usage pattern
X’s are typically a combination of controllable and noise (uncontrollable) factors
Y = f(XC , XN)
Goals:
• Target Y
• Minimize sY, that is, variation in Y
David Rumpf, Statistician
GE Aircraft Engines
Page 5
Why Robust Design
Statistical/Engineering method for product/process improvement (Taguchi’s idea)
Two types of factor, control (Xc) and hard to control (Xn or noise)
• Control factor levels can change target
• Hard to control factors have variation during normal process or usage
Robust design: Set Xc to take advantage of non-linearity in Y = f(Xn)
• Design space is typically non-linear
Non-linear Response
14
Response
12
10
8
6
4
2
0
1
David Rumpf, Statistician
2
3
4
Xn
GE Aircraft Engines
5
6
7
Page 6
Wu and Hamada recommend a two step process
Obtain Transfer Functions: Ybar = f1(XC)
sY = f2 (XC)
Typically one finds different sets of X’s in the two transfer functions
If Target is goal:
• Minimize variation in Y, the harder objective
• Minimize Ybar distance from target
If maximum or minimum is the goal:
• Optimize Ybar
• Minimize variation in Y
An alternative approach is non-linear optimization of Z where
Z = |Target – Ybar|/ sY
Experiments: Planning, Analysis and Parameter Design Optimization, CFJ Wu and M Hamada, Wiley 2000
David Rumpf, Statistician
GE Aircraft Engines
Page 7
Statistical Issues with Analytic Models
Designing a new part:
• Typically done analytically
• Often a complex, time consuming process to obtain a result for a single set of
parameter values
• Examples include finite element analysis models, system models, etc
• Leads to serious optimization and simulation issues
Recommended approach:
• Run designed experiment, typically Response Surface, to capture non-linear
effects
• Use RSM transfer function for
• Optimization
• Simulation to estimate effect of variation in X’s on Y
Statistical Problem:
• Analytic models have no random variation, always the same answer for a set of X
values
• RSM assumes normally distributed error in residuals from model fit
• Residuals from analytic model are entirely lack of fit.
David Rumpf, Statistician
GE Aircraft Engines
Page 8
Case Study was a Learning Process for the GEAE Author
Initial approach: (Note: All results coded for proprietary reasons)
• Full Factorial with center-point, 9 computer runs, ~ 9 hours run time
• Y’s = life required log transformation
• Choose to use Y’s = stress
• Interactions and curvature were significant, see Y1a graph below
Centerpoint5
05
-0 .0
Interaction Plot (data means) for notch
0 .0
04 5
-0 .0
07
- 0 .0
03
1
-0 .0
0
30
RetArm
20
0.0045
-0.0055
10
30
CP Rab
20
-0.003
-0.007
10
CP Drop
30
20
0
-0.01
David Rumpf, Statistician
Largest interaction and
curvature effects
Results led team to an
RSM design in 3 factors.
10
GE Aircraft Engines
Page 9
RSM Design
• Face centered central composite design, illustrated for two factors below
• We ran 15 runs, no repeated center-points since computer model has no random variation
Factor B
RSM analysis requires 9+
runs for a 2 factor design,
15+ runs for a 3 factor
design.
Factor A
David Rumpf, Statistician
GE Aircraft Engines
Page 10
RSM Results
Analysis plan and results:
• Chose most parsimonious model via backwards selection based on p values
Response
Y1a
Y1b
Y2a
Y2b
Main
3
3
3
3
Two-way
1
1
0
1
Quadratic
1
2
1
0
R-sq adjusted
98.2
95.0
98.7
94.3
Residual versus Standard Order Run Number
Concerns:
1) High residuals, especially for
Rabbet stress, cause concern.
2) R-sq not as high as desired
5
Y1a
Y1b
Y2a
Y2b
4
Questions:
1) Does this transfer function fit well
enough for engineering need?
2) Is there a better way to fit
analytic/computer model results
Residual
3
2
1
0
-1
-2
-3
0
5
10
15
Run-Num
David Rumpf, Statistician
GE Aircraft Engines
Page 11
Enter Professor William Welch and a Space Filling Design
GEAE author looked for help. Jeff Wu suggested Professor W. Welch of the
University of Waterloo
New Experimental Plan:
• Space filling design instead of DOE or RSM design
• Recommended for computer/analytic experiments
• Multiple levels to provide better estimate for non-linear and interaction effects
• 33 runs for 3 factors
• Doubled the number of runs
• Spaces levels for each factor at 1/32nd of the range
• Plots below show experimental grid, 2 factors at a time
• Computer experiment was run for the 33 sets of conditions (~30 hours of run time)
0.005
0.000
-0.003
0.000
CP Drop
CP Rabbet
RetArm
-0.004
-0.005
-0.005
-0.006
-0.005
-0.010
-0.007
-0.007
-0.006
-0.005
-0.004
CP Rabbet
David Rumpf, Statistician
-0.003
-0.005
-0.010
-0.005
0.000
0.000
0.005
RetArm
CP Drop
GE Aircraft Engines
Page 12
Approximating Random Variable Function Model
Treat Deterministic output Y(x) as a realization of a random function
(stochastic process)
Y(x) = Ybar(x) + Z(x)
Sacks et al, Statistical Science, 1989
Intuition:
• Model correlation between Z(x) and Z(x’) for any two input
vectors x and x’
• x close to x’ – correlation large
• x far from x’ – correlation small
• Leads to a distribution of Y(x) at any x given the Y’s at the
design points
Perform diagnostic tests on model
• Accuracy of prediction? Standard error of prediction?
David Rumpf, Statistician
GE Aircraft Engines
Page 13
Accuracy comparison (e.g., Notch-Alt or Y1b)
Error = Y – fitted Y
Fitted Y is leave-one-out cross validated
(take observation out and predict it)
RMSE = 1/n sum (Y – fitted Y)2
-------------------------------------------------Approximating
Cross-Validated
Model
RMSE
--------------------------------------------------Polynomial – 2nd degree
0.71
Polynomial – 3rd degree
1.04
Random function
0.48
3rd degree polynomial fits even worse than 2nd degree!
David Rumpf, Statistician
GE Aircraft Engines
Page 14
Diagnostic Checking of Random-Function Model
Accuracy assessment:
Plot Y versus fitted Y
Standard error (se) assessment: Plot (Y – fitted Y) / se(fitted Y) versus fitted Y
(Fitted Y is leave-one-out cross validation)
David Rumpf, Statistician
GE Aircraft Engines
Page 15
Visualization of Input-Output Relationships
e.g. Y1b as a function of RetArm
Other two inputs (CPRabbet and CPDrop) averaged out
David Rumpf, Statistician
GE Aircraft Engines
Page 16
Propagating Variation Through the Random Function Model
CPRabbet, CPDrop, RetArm have independent N(mu, sigma) distributions
e.g., set mu = center of range
Sample CPRabbet, CPDrop, RetArm and pass through model to get a
distribution of e.g., Y1b values
David Rumpf, Statistician
GE Aircraft Engines
Page 17
Conclusions
RSM approach:
• Good starting point
• Will work fine for Ybar and simple underlying Physics
Space Filling Design:
• Allows us to model responses with very nonlinear underlying
Physics
Random-function model:
• Provides valid standard errors of prediction
• Can adapt to nonlinearities in data
• Fast, so can quickly propagate variation
inputs => outputs
via Monte Carlo
David Rumpf, Statistician
GE Aircraft Engines
Page 18