Applications of the Modern Design of Experiments
at NASA Langley Research Center
Richard DeLoach
NASA Langley Research Center
46th Annual Fall Technical Conference
Learning and Discovery Building the Future Through Quality and Statistics
October 17-18, 2002
Valley Forge, PA
Langley Research Center
OUTLINE
• What is the “Modern” Design of Experiments (MDOE)?
• Conventional wind tunnel test practices
– Brief description
– Some shortcomings
• Reasons for Langley’s growing interest in statistically
designed experiments
– Resource savings
– Enhanced quality and productivity
– Deeper insights into underlying physics of flight
• Summary and parting thoughts
Langley Research Center
Why the “M” in MDOE?
(How does Modern DOE differ from “ordinary” DOE?)
• Naming convention to distinguish between modern and
conventional experiment design methods.
– LaRC researchers “design experiments” all the time
– Emphasis on modern experiment design overcomes initial
perception that this is nothing new.
• Effort to highlight differences between industrial and
scientific applications of DOE
– Some unsatisfactory prior experiences with industrial DOE
– Perceptions that designed experiments inappropriate for
complex research applications
Langley Research Center
Four Eras in the
History of Designed Experiments
(Credit: Douglas Montgomery)
• The agricultural origins, 1918 – 1940s
– R. A. Fisher & his co-workers
– Profound impact on agricultural science
– Factorial designs, ANOVA
• The first industrial era, 1951 – late 1970s
– George Box and others
– Response surface methods
– Applications in the chemical & process industries
• The second industrial era, late 1970s – 1990
– Quality improvement initiatives in many companies
– Taguchi and robust parameter design, process robustness
• The modern era, beginning circa 1990
Langley Research Center
Conventional Wind Tunnel Testing
Characteristics of a typical wind tunnel test
• Design
– Test matrix changes One Factor At a Time (OFAT).
– Test matrix generally specifies greatest combination of
factor levels that resources permit.
• Execution
– Change variable levels monotonically to maximize data
acquisition rate.
– Make small incremental changes in each factor.
– Physically set all factor combinations of interest.
– Data collection ends when resources are exhausted.
• Analysis
– Tabulate measured responses (generate database).
– Graphically display selected response relationships.
Langley Research Center
Three Shortcomings in OFAT Testing
MDOE is not simply an incremental improvement in an
otherwise satisfactory process.
• Design
– Changing one factor at a time is very time-consuming.
– Designing for maximum data volume also ensures
maximum time, labor, and expense.
• Execution
– Monotonically changing factor levels confounds true
factor effects with ubiquitous systematic variations.
– Fastest test matrix not the highest quality test matrix.
• Analysis
– There are generally more factor effects of interest than
can be displayed efficiently.
– Important relationships, especially factor interactions,
are often not quantified.
Langley Research Center
Two Competing Views of Wind Tunnel Testing
(“Knowledge vs. Data”)
• OFAT: An industrial process
– Model: Wind tunnel as data factory
– Product: High quality data
– Goal: Maximize data volume, given fixed tunnel time.
• MDOE: A research process
– Model: Wind tunnel as laboratory
– Product: Insights leading to reliable response prediction
– Goal: Minimize expense and time to achieve specific
technical objectives
Langley Research Center
Importance of Cycle Time Reduction
in Aerospace Testing
• Dollars are often perceived by researchers as the most
important resource, but…
• “Cycle Time” can be more critical.
– The cost of bringing a major aerospace product to
market can be in the billions of dollars.
– Cost of capital can therefore be in the millions of
dollars per day.
– Cost of getting to the market late can be very high.
Langley Research Center
OFAT Thinking Maximizes Cycle Time
• Emphasis on direct measurement of all factor
combinations results in high-volume data collection.
• Seldom sufficient time to set all conditions of interest.
• Productivity associated with high data volume, but…
• Data acquisition rate is limited so “high productivity”
translates into tests that use all available time.
• The “Bernoulli Factor”
Langley Research Center
High Data Volume – A Very Old Idea!
Even the most stupid of men, by some instinct of nature, by
himself and without any instruction, is convinced that the
more observations that have been made, the less danger there
is of wandering from one’s goal.
Jacob Bernoulli (1654-1705)
Langley Research Center
Synthetic Jet Lift Enhancement Study
An illustration of why no amount of time is “too much” for an
OFAT wind tunnel test.
Langley Research Center
Leading Edge Synthetic Jets
Enhanced Lift
Lift
Without Synthetic
Jet Flow Control
Langley Research Center
With Synthetic Jet
Flow Control
Synthetic Jet Configuration Variables
Four jet variables, 3 flap variables each side
Starboard Jet Banks
SJ4
SJ3
SJ2
SJ1
Starboard Flaps
Outboard (Mean + Diff)
Inboard
Port Flaps
Port Jet Banks
PJ1
PJ2
PJ3
PJ4
Langley Research Center
Inboard
Outboard (Mean + Diff)
Synthetic Jet Data Volume Considerations
• There are 14 configuration variables
– Four banks of synthetic jets each wing
– Three flap variables each wing
• Inboard flap deflection
• Outboard flap deflection
• Differential outboard flap deflection (clamshell mode)
• At least three levels of each factor required for any
response that is non-linear in the factors.
• Total number of candidate configurations: 314=4,782,969.
Langley Research Center
Synthetic Jet Data Volume Considerations
• Total number of candidate configurations: 314=4,782,969.
• Required time to measure all configurations assuming two
configurations/minute, two shifts/day: 9.6 years
• Allocated test time: two weeks
– There is time to set a theoretical maximum of 19,200 configurations
– This would leave 4,763,769 configurations unexamined (99.6% of
the cases). In practice, many fewer configurations could be set.
• Any realistic allocation of wind tunnel time is likely to be
considered insufficient.
• Conventional solution is to rely on judgment and “instinct”
to select some subset of configurations to study, but…
– This assumes we already know much of the answer.
– Much must go unexamined in any case.
Langley Research Center
Minimum Number of Candidate Configurations (two levels per factor)
Grows exponentially with number of factors
Configurations
100,000
16,384
10,000
1,000
100
10
14
1
0
5
10
Factors
Langley Research Center
15
What Should Determine Data Volume?
The MDOE approach to developing a test matrix.
• Not practical to set every factor combination, even with a
moderate number of factors.
• How can we rationally decide how many and which factor
combinations to set?
– Number of factor combinations driven by signal/noise ratio, inference
error risk tolerance, and expected complexity of response.
– Selection of factor combinations driven by specific experiment design.
Langley Research Center
Factors That Drive
Data Volume Requirements
• Size of smallest effect that must be resolved, d
• Variance in individual measurements, s2
• Acceptable inference error risk
– Probability of inferring the wrong size of an effect, a
– Probability of failing to detect a significant effect, b
• Parameters in model, p
2
s
2


n  p za  z b
d2
Note Bene: There are variations on this formula, depending on the details of the
experiment, but they all feature the same elements: S/N, risk, model complexity
Langley Research Center
Cost/Benefit Tradeoff
Synthetic Jet Lift Enhancement Study
100%
95
%
95%
Confidence
90%
d = 0.01
85%
s = 0.005
80%
75%
p = 20
70%
b = 0.01
92
65%
60%
50
75
100
Points
Langley Research Center
125
150
MDOE Data Volume Estimate
Synthetic Jet Lift Enhancement Study
• We estimated that 92 or more factor combinations
would satisfy our inference error risk tolerance.
– For the smallest signal to noise ratio expected
– For the most complex response model expected
• Sufficient volume of data to ensure 99% probability
of detecting lift changes large enough to be
important to us.
• Sufficient volume of data to ensure 95% probability
that indicated lift change is within an acceptably
small range.
• Selection of factor combinations driven by design.
Langley Research Center
Impact of Experiment Design on Data Volume
Synthetic Jet Lift Enhancement Study
• A two-level factorial design would reveal flap/jet
interaction effects, which are of special interest, but…
• A full factorial design in 14 factors would require more
than the two weeks allotted.
• Fractional factorial design a possible solution
– Confounds low- and high-order interactions (aliasing).
– Good solution if high-order interactions are negligible.
– Unfortunately, high-order interactions were anticipated.
• Solution was to exploit expected independence of portside and starboard-side factor effects.
• This reduces design to two additive full factorial
designs in seven factors each.
Langley Research Center
Minimum Number of Candidate Configurations (two levels per factor)
Grows exponentially with number of factors
Configurations
100,000
16,384
10,000
1,000
128
100
10
14
7
1
0
5
10
Factors
Langley Research Center
15
Synthetic Jet Lift Enhancement Study
Initial MDOE Experiments
• Minimum of 92 factor combinations to satisfy inference
error risk tolerance.
• Two full factorial designs (port wing and starboard wing)
in seven factors
– 27 = 128 factor combinations each
– Sufficient to drive inference error risk below acceptable levels.
• All main effects and interactions through 7-way on each
wing, clear of any aliasing.
• Assumes negligible port/starboard interactions
– In harmony with views of aerodynamic subject-matter specialists
– Assumption easy to test
• Less than 5 wind-on hours.
Langley Research Center
Summary of Single-Wing Results
• There were over a dozen lift effects significant at the
0.05 level (95% confident they are real)
• Flap effects dominated, as expected, however…
• There were significant jet main effects.
• There were significant interactions up to 6-way.
• There were substantial jet/flap interactions.
• Some asymmetry between wings noted in that port and
starboard lift effects were not identical.
– Some airframe manufacturing differences
– Some differences in jet performance
Langley Research Center
Synthetic Jet Lift Enhancement Study
Follow-On MDOE Experiments
• Initial port-side and starboard-side experiments were
extended to include angle of attack and replicated for two
synthetic jet waveforms.
• These extended experiments were augmented to permit
higher-order response modeling.
– 2nd-order terms in jet/flap configuration variables
– Up to 4th-order in angle of attack
• Model believed capable of adequately predicting lift for all
factor combinations in range tested.
• Total wind-on time for all experiments: 22 hrs (3 shifts)
• Approximately 25% of wall-clock time is wind on, so…
– This spanned roughly 11 eight-hour shifts
– Less than six two-shift days (60% of allocated tunnel time)
Langley Research Center
Current Status of MDOE
• Over 40 MDOE tests designed, executed, and analyzed since
concept introduced to Langley production wind tunnels in 1997.
• Over 100 researchers have been intimately exposed to these
methods through courses and/or scientific collaboration.
• Employees at Langley and from private industry have taken
graduate leave to obtain university training in formal experiment
design.
• The American Institute of Aeronautics and Astronautics (AIAA) now
includes MDOE instruction in its Professional Development Series of
short courses.
• MDOE recognized as an emerging technology in two AIAA
Recommended Practices documents (wind tunnel testing and force
balances)
• Technical sessions on experiment design now a part of major AIAA
conferences (Aerospace Sciences, Ground Testing, Applied
Aeronautics)
Langley Research Center
A Parting Thought
• Based on balance of trade data, the world especially values
two American products (Credit: Dennis Bushnell, LaRC
Chief Scientist)
– Aeronautical systems
– Wheat
• The one we most efficiently cultivate is wheat.
• There is an important role for the statistics and quality
community in aeronautical research, because...
• Statistically designed experiments are key to cultivating
aeronautics as efficiently and productively as wheat.
Langley Research Center