For my Father, whom we have to assume would have
wanted me to get a degree at MIT; for my sister Frances,
who never doubted; and for my Mother, who saw us all
through.
1
Objective Comparison of Design of Experiments Strategies in
Design and Observations in Practice
by Ion Chalmers Freeman
Submitted to the Engineering Systems Division
on May 7, 2004, in partial fulfillment of the
requirements for the degree of
Master of Science in Engineering and Management
Abstract
Design of Experiments (DoE) strategies in robust engineering determine which prototypes and how many of each are created and tested. A better strategy is one that
delivers a closer-to-optimal performance at a lower experimental cost. Prototype
testers who may use statistical DoE, design-build-test, or one-at-a-time methods in a
wide variety of industries were sought out and interviewed to examine the strategies
used in practice and how they fit into a proposed five-layer process support model.
From these interviews, we see that DoE are competently and widely practiced.
Some improvements to the state of the practice may include
• contracts to specify and reward quality engineering among suppliers to complex
product systems
• wider use in light of new computing power of system level mathematical models
for experimentation on complex systems
This thesis also examines the relative value of strategies in a particular response
surface using a software-based comparator. The data is modified to simulate data
environments with other levels of repeatability and interactions, and the way that
these variables effect the performance of strategies is examined. The concept of an
optimal design of experiments strategy is developed by abstracting the characteristics
of a generic strategy and letting it develop in a genetic algorithm in that comparator.
The framework for the evaluation of DoE strategies is one significant output to
come out of this work that may be of use in future research. Further, the particular
abstraction chosen for DoE strategies is offered to other researchers as an exemplar
of a particular perspective, to help engender dialogue about methods for optimizing
prototype testing policy.
Thesis Supervisor: Christopher L. Magee, Ph. D.
Title: Professor of the Practice of Engineering Systems and Mechanical Engineering
Acknowledgments
I would like to thank Professor Chris Magee, my supervisor, for his vision and constant
guidance during this research.
Other MIT and Sloan faculty contributed greatly to this work. Professor Dan
Frey supplied me with preprints of his papers, as well as the source data used in
much of the work. Professor Dan Whitney shared with me his enthusiasm and perspective. Professor Emeritus Don Clausing shared with me Antje Peters’ thesis, and
his time, contributing several points about the practice of engineering experimentation to this thesis. Professor Diane Burton was generous with her advice, and at
Harvard, Professor Stefan Thomke was a great help in navigating his own work in
this area.
Several of my System Design and Management classmates, notably Wei Shen and
Frank Lanni, and alumni, notably Scott Ahlman, contributed substantially to this
material.
My graduate studies were supported by the Center for Innovation in Product
Development at MIT. I have been lucky enough to enjoy the feedback and encouragement of my officemates Jagmeet Singh, Ryan Whitaker, Tom Speller, Mo-Han Hsieh
(who provided the ANOVA analysis in Appendix A,) especially Mike Gray, and other
CIPD students including Victor Tang and Katja Höllta. The CIPD staff, Nils Nordal,
Kathleen Sullivan, Susan MacPhee and Michael Mack, provided valuable assistance
and information at various times, and encouraged me to create a seminar on this
research. The SDM staff, notably Dennis Mahoney, Ted Hoppe, Jeff Shao and Bill
Foley have rendered me great assistance throughout my stay at MIT.
Finally, I wish to thank the following: my roommate Tim Baldwin, for always
pulling me back into the world; Tabitha Baker, for giving me hope; and Marisa
Joelson, for letting me work on my thesis when I should have been at the gym.
Cambridge, Massachusetts
May 7, 2004
I. Freeman
Contents
1 Problem Statement
12
1.1
Product Development . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
1.2
Performance and Variability . . . . . . . . . . . . . . . . . . . . . . .
13
2 Background
15
2.1
Taguchi Methods & Robust Engineering . . . . . . . . . . . . . . . .
15
2.2
Parameter Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
2.3
Design of Experiments . . . . . . . . . . . . . . . . . . . . . . . . . .
17
2.3.1
The Resistance of Statisticians . . . . . . . . . . . . . . . . . .
19
2.4
Strategies in Design of Experiments . . . . . . . . . . . . . . . . . . .
19
2.5
Adoption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
2.5.1
Morrow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
2.5.2
Peters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
2.5.3
Thomke . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
3 GABackground
23
3.1
History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
3.2
Appropriateness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
3.3
Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
3.4
Design Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
26
3.4.1
Population Size . . . . . . . . . . . . . . . . . . . . . . . . . .
26
3.4.2
Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
26
3.4.3
Crossover . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
27
5
3.4.4
Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 Practice
4.1
4.2
29
Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
4.1.1
Conversations . . . . . . . . . . . . . . . . . . . . . . . . . . .
30
Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
30
4.2.1
Vehicle Handling at a High Performance Car Development Interest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.2
30
A Master Black Belt at an Automatic Transmission at an Automotive Manufacturing Company
. . . . . . . . . . . . . . .
33
4.2.3
Engine Design at an Automotive Design Concern . . . . . . .
34
4.2.4
Consulting at a Corporate Engineering Center . . . . . . . . .
34
4.2.5
Black Belt at an auto parts manufacturer . . . . . . . . . . . .
35
4.2.6
Software project manager . . . . . . . . . . . . . . . . . . . .
37
4.2.7
Decision Support at a Large Commercial and Retail Loan Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
Naval Marine Propulsion . . . . . . . . . . . . . . . . . . . . .
39
Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40
4.3.1
43
4.2.8
4.3
28
Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Comparator
45
5.1
Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
45
5.2
Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
46
5.2.1
Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47
5.2.2
Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47
5.2.3
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
48
The Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
49
5.3.1
Paper Airplane Game . . . . . . . . . . . . . . . . . . . . . . .
49
5.3.2
Normal Data Games . . . . . . . . . . . . . . . . . . . . . . .
52
5.3.3
Manipulating the nonlinearity . . . . . . . . . . . . . . . . . .
54
5.3.4
Manipulating the noise . . . . . . . . . . . . . . . . . . . . . .
55
5.3
6
5.4
The Central Command . . . . . . . . . . . . . . . . . . . . . . . . . .
56
5.5
The Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
59
5.5.1
Validating the experimental designs . . . . . . . . . . . . . . .
62
5.5.2
Guess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
63
5.5.3
Full Factorial . . . . . . . . . . . . . . . . . . . . . . . . . . .
63
5.5.4
Adaptive One At A Time . . . . . . . . . . . . . . . . . . . .
64
5.5.5
Orthogonal Array . . . . . . . . . . . . . . . . . . . . . . . . .
66
5.6
Payoff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
67
5.7
Evaluating the experimental designs . . . . . . . . . . . . . . . . . . .
67
5.8
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
68
5.9
Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
71
6 Abstraction
73
6.1
Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
73
6.2
Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
74
6.3
Available optimization methods . . . . . . . . . . . . . . . . . . . . .
75
6.4
Genetic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . .
76
6.4.1
The Codings . . . . . . . . . . . . . . . . . . . . . . . . . . . .
77
6.4.2
The Fitness Function . . . . . . . . . . . . . . . . . . . . . . .
78
6.4.3
Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
79
6.4.4
Crossover . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
86
6.4.5
Mutation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
87
6.5
Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
89
6.6
Results and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . .
90
6.6.1
Strategy Competence . . . . . . . . . . . . . . . . . . . . . . .
90
6.6.2
Two model runs . . . . . . . . . . . . . . . . . . . . . . . . . .
92
Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
97
6.7
7 Conclusion
7.1
98
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
98
7.1.1
98
Problem Statement . . . . . . . . . . . . . . . . . . . . . . . .
7
7.2
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7.3
Further Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
A Grewen Data
99
103
A.1 The Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
B Interview Questions in Design of Experiments
112
C Tables
115
C.1 Selected Configurations for the Strategies . . . . . . . . . . . . . . . . 116
C.1.1 Full Factorial . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
C.1.2 Adaptive One At A Time . . . . . . . . . . . . . . . . . . . . 123
C.1.3 Orthogonal Array . . . . . . . . . . . . . . . . . . . . . . . . . 125
C.2 Performance of the Strategies . . . . . . . . . . . . . . . . . . . . . . 127
C.2.1 Interpreting this Table . . . . . . . . . . . . . . . . . . . . . . 127
D Five Layer Process Support Model
141
D.1 Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
D.2 Effecting Process Change . . . . . . . . . . . . . . . . . . . . . . . . . 143
D.3 Describing Current Processes . . . . . . . . . . . . . . . . . . . . . . 143
D.4 Accountability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
D.5 Fads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
D.6 An Example: Balancing the Registers . . . . . . . . . . . . . . . . . . 144
D.6.1 A process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
D.6.2 Conflict with other corporate needs . . . . . . . . . . . . . . . 145
8
List of Figures
5-1 Class Structure for Comparator . . . . . . . . . . . . . . . . . . . . .
46
5-2 Comparator Functions . . . . . . . . . . . . . . . . . . . . . . . . . .
56
5-3 Stacked Histogram of Mean Payoff . . . . . . . . . . . . . . . . . . .
69
5-4 Stacked Histogram of Mean Wealth . . . . . . . . . . . . . . . . . . .
69
5-5 Strategy values at different nonlinearity values . . . . . . . . . . . . .
70
5-6 Strategy values at different signal-to-noise ratios . . . . . . . . . . . .
71
6-1 Evolution of Payoff over ten generations(Run 1) . . . . . . . . . . . .
90
6-2 Evolution of Payoff over ten generations(Run 2) . . . . . . . . . . . .
90
6-3 Evolution of Diligence(Run 1) . . . . . . . . . . . . . . . . . . . . . .
92
6-4 Evolution of Diligence(Run 2) . . . . . . . . . . . . . . . . . . . . . .
93
6-5 Evolution of Profligacy(Run 1) . . . . . . . . . . . . . . . . . . . . . .
93
6-6 Evolution of Profligacy(Run 2) . . . . . . . . . . . . . . . . . . . . . .
94
6-7 Evolution of Planning(Run 1) . . . . . . . . . . . . . . . . . . . . . .
94
6-8 Evolution of Planning(Run 2) . . . . . . . . . . . . . . . . . . . . . .
95
6-9 Evolution of Care(Run 1) . . . . . . . . . . . . . . . . . . . . . . . .
95
6-10 Evolution of Care(Run 2) . . . . . . . . . . . . . . . . . . . . . . . .
96
6-11 Evolution of Fairness(Run 2) . . . . . . . . . . . . . . . . . . . . . . .
96
6-12 Evolution of Fairness(Run 1) . . . . . . . . . . . . . . . . . . . . . . .
97
9
List of Tables
2.1
An orthogonal array experimental design of order two for a four-parameter,
three-level experiment . . . . . . . . . . . . . . . . . . . . . . . . . .
18
5.1
Variance Components in the Grewen Data . . . . . . . . . . . . . . .
50
6.1
Canonical strategies as phenotypes . . . . . . . . . . . . . . . . . . .
78
6.2
80-Goodness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
91
6.3
Selections, Run 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
91
6.4
Selections, Run 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
91
A.1 Grewen Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
A.2 Grewen ANOVA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
C.1 Strategy Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
10
Listings
5.1
Player.run() method . . . . . . . . . . . . . . . . . . . . . . . . . . .
57
5.2
The Strategy Interface . . . . . . . . . . . . . . . . . . . . . . . . . .
59
5.3
Listing the Configurations . . . . . . . . . . . . . . . . . . . . . . . .
62
5.4
AOAT Parabolic update() . . . . . . . . . . . . . . . . . . . . . . . .
64
5.5
SQL Query to retrieve comparative value . . . . . . . . . . . . . . . .
67
11
Chapter 1
Problem Statement
This thesis examines the application of prototype testing approaches in a number of
enterprises, and considers the drivers and perceived utility of these design of experiments strategies. It then creates a framework for comparing strategies, and presents
an method for developing a testing approach within that framework.
1.1
Product Development
In the course of a product life cycle, decisions are made and commitments created.
Each of these decisions has a cost. The earlier in the product development life cycle
a good decision is made, the greater the benefits reaped[1].
The spread of time from the first articulation of need that becomes a product
requirement, to the final collection of the last retired product is the product life time.
However, as there are multiple products in a typical family, it is useful to speak of a
product life cycle.
A product life cycle comes about through iteration. One product will feed information and infrastructure the next, making subsequent versions better suited to
customer needs, as well as cheaper and easier to design and produce.
The life cycle, generically, has four phases: Envision, Design, Develop and Deploy.1 It is during the design phase that decisions regarding the implemented form
1
This breakdown, and the alternate Conceive, Design, Implement, Operate, come from [2, Slide
12
and included features are made.
1.2
Performance and Variability
When the system design is complete, a component can be expected to deliver a
particular function. The measure of the degree to which this function is provided is
the component’s performance. However, the measured performance will vary from
measurement to measurement, and from prototype to prototype. These are both
types of variability that come out of prototype testing.
It is during the validated design phase that prototype testing takes place. A
component’s performance will indicate how well it suits your need, and its variability
will tell you how much you can trust it.
All of this activity lies within a single one of the four Ps of marketing, Product2 .
The only way to affect profitability in the detailed design phase is through product
changes. We can change the product in two ways – we can change its cost, or we can
change its quality. Reducing the cost of goods sold (COGS) increases profit to first
order.
The definition of quality is somewhat flexible; it should be clear here that we are
speaking of ’manufacturing-based’ quality[3], which can be considered as performance
minus variability, and can be measured in terms of the total loss to society due to
functional variation and harmful side effects[4].
An experimental design can reveal to the experimenter the effects on quality of
design changes including cost reduction, and uncover ways that quality can be improved. To manipulate the quality of a designed product, and to see how proposed
cost reductions might change the quality, a product designer engages in parameter
design by selecting modifiable characteristics of the system as control factors and
creating prototypes that have particular configurations, which is to say some specific
combination of chosen levels for the control factors. This process is referred to as pro8]
2
The other three being Place, Promotion and Price
13
totype testing, and the specific configurations to build and the number of experiments
to run on those configurations are together known as Design of Experiments(DoE).
Design of Experiments strategies determine how configurations will be selected for
experimentation, and how many times an experiment will be repeated.
Design of Experiments strategies are the central topic of this thesis. We examine
the question of how DoE is practiced in a company, why product designers engage in
it, what causes the practice to vary and its perceived benefits.
In the first part, we investigate the adoption and practice of DoE in product
development concerns. While Some work has investigated how widely adopted DoE is,
and how rational it is for a company to pursue it, as discussed in section 2.5.3. While
the drivers and inhibitors of Taguchi method adoption have been well explicated[5],
the activity of adopting the practice in a corporation has not been researched to the
same extent.
We felt the practice and understanding of DoE would benefit from some standard
way to compare these four types of strategies. In the second part, we therefore created
a comparative framework within which different strategies and response surfaces can
be evaluated.
Given a generic comparative framework for DoE strategies, we naturally asked in
the third part how one would use it to find the optimal strategy in a particular data
environment under a particular set of rules. So, whereas with the framework we could
determine the best of any set of strategies, our new goal was to find the best DoE
strategy for a given experimental environment.
14
Chapter 2
Background
Taguchi methods have become progessively better adopted since their initial introduction in the 1940s, and have focused attention on robustness and effects of noise
in product development. They have come to dominate statistical performance enhancing methods at many American corporations, and continue to be a well-regarded
method which engineering firms can adopt to improve the quality of their goods[7].
The selection of control factors and the levels which they can take frames the
subsequent work on DoE. This series of tasks occupies the parameter design phase.
Taguchi Methods, Robust Engineering and Design of Experiments can contribute
meaningfully to a company’s profitability. DoE is most effective if introduced as part
of a meaningful quality initiative[5].
However, the performance of one system may have nothing to do with the performance of another. The proper choice of a set of experiments to do, and how often to
repeat each member of that set, depends quite strongly on what preconceptions the
designer has about the system’s response to changes in parameter levels.
2.1
Taguchi Methods & Robust Engineering
Genichi Taguchi, an electrical engineer charged by Nippon Telephone and Telegraph
to develop a methodology to produce high-quality products and continue to improve
quality in a scarcity environment(Japan) in the 1950s and 1960s, devised an accessi15
ble way to involve reduction of variability in the design phase1 . A narrow interest in
reducing variability in response for launching munitions[4, p.2] spread through various industries, and eventually Dr. Taguchi was invited to America by Bell Labs in
1980. Ford Motor Company created what is now the American Supplier Institute to
elaborate on and promulgate his ideas.
Simply put, Taguchi methods seek to maximize the signal-to-noise ratio, the performance’s mean over its standard deviation. They do this by creating a Quality Loss
Function. The Quality Loss Function seeks to capture the total cost to society of
variation from an optimum, and is generally modeled as a parabola. The width of
the parabola is difficult to gauge, and is set so that components in a system that have
less need of precision lose quality more slowly than others, and so that the value does
not drop below zero inside the acceptable bounds.
2.2
Parameter Design
Parameter design in the second step in Taguchi methods, which consider all that
has come before as system design. In the validated design subphase, the shape and
specific characteristics of the product are well described, and the designer is seeking
information on fine tuning to improve performance and reduce variability.
She does this by selecting parameters or control factors from several aspects of the
product design. Parameters are selected with criteria including expected influence on
the objective signal-to-noise function and ease of manipulation2 , and are kept few
enough that the number of tested configurations does not grow too high. Hirschi
and Frey[6] note that in a coupled system, more than a few parameters become very
difficult for people to consider.
These parameters each have a number of values that they can take, or levels,
1
Most of the history here comes from [5, Appendix A], [8], [4, p.2] and the websites for Leeds
Metropolitan University and the American Supplier Institute. A more complete history and explanation is given in [7]
2
Influence on signal to noise and ease of manipulation can be difficult to reconcile. Taguchi
specifically recommends transformations in the control factors to maximize their impact; transformed
control factors are rather difficult to manipulate as they are composed of real control factors
16
defined. A collection of parameters with levels set to particular values is referred to
herein as a configuration. The number of tested configurations depends on both the
experimental design and the total number of configurations, and parameter design
reduces the latter.
The total number of configurations T in the parameter space is
T =
Y
np
(2.1)
1≤p≤P
where P is the number of parameters and np is the number of levels for the pth
parameter. So, the number of configurations to test increases geometrically with the
number of levels of each parameter. Chapters 5 and 6.5, constrain themselves to
rectangular parameter spaces, which is to say systems where each parameter has the
same number of settings as any other. This simplifies the equation somewhat to
T = NP
(2.2)
where N is the number of levels for any parameter.
To keep the total number of configurations low, parameters are frequently chosen
to be ’two-level’, especially for screening DoE (as discussed in section 4.2.4), or ’threelevel.’ Three-level parameters are chosen with the nominal value that comes out of
system design and reasonable minimum and maximum values.
In a full experimental regime for optimizing performance and reducing variability,
a designer might choose many more than three levels, or run tests repeatedly on the
same system making finer distinctions. It is useful, therefore, to be able to test many
fewer than the total number of configurations.
2.3
Design of Experiments
The creation and testing of a prototype with a particular configuration is an experiment. The prospect of testing every possible configuration can be daunting as
discussed in section 2.2 and in any case one must decide how many prototypes she
17
Table 2.1: An orthogonal array experimental design of order two for a four-parameter,
three-level experiment
Parameter
Configuration
Configuration
Configuration
Configuration
Configuration
Configuration
Configuration
Configuration
Configuration
1
2
3
4
5
6
7
8
9
A
1
1
1
2
2
2
3
3
3
B
1
2
3
1
2
3
1
2
3
C D
1 1
2 2
3 3
2 3
3 1
1 2
3 2
1 3
2 1
will build, or how many times – called repetitions – she will repeat an experiment,
with a given configuration. These decisions are referred to as Design of Experiments
(DoE).
The origin of Design of Experiments has nothing to do with product development.
Ronald Fisher developed a statistical method at Rothamsted Agricultural Station to
investigate the main effects of varying influences on crop yield. The central idea is
that a parameterized statistical model links expected response with the values of the
control factors. The DoE strategy tries the least number of configurations necessary
to establish the parameters in the statistical equation.
The Orthogonal Array is a particular type of DoE useful for pulling out main
effects and first order interactions. We can speak of DoE strategies in terms of their
experimental design, which configurations the strategy visits and in which order. We
shall now illustrate the orthogonal array strategy with an example.
The paper airplane game is more fully described in section 5.3.1; briefly, it is a
3-parameter, 4-level rectangular parameter space with an easily measurable response.
We build an experimental design using an orthogonal array strategy.
The order of an orthogonal array design is the number of columns for which –
and for less than which – there is no repetition (or, strictly, no more repetition for
any one grouping of a that number of columns than for any other.)[9] In Table 2.1,
18
no two columns have any repetition in them, so this is an orthogonal array of order
two.
The statistical model that corresponds to an orthogonal array of order two is a
simple main effects model, which is to say, one that assumes that the response is the
sum of some constant and a value for each parameter according to its level.
2.3.1
The Resistance of Statisticians
Jeffery Morrow’s 1988 thesis[5] discusses a tension in its first appendix. Dr. Taguchi
employed the orthogonal array in teaching his methods, and Robust EngineeringTM is
tied to the use of the orthogonal array in many people’s minds. Statisticians can
therefore be antagonistic toward the adoption of Taguchi methods, as they perceive
it to underplay the complexity of the response surface.
The signal-to-noise objective function and the quality loss function are therefore
ignored as part of a statistically naı̈ve endeavor. The quality loss function is rejected
in part on its own merits with the complaint that the coefficient is difficult to establish.
2.4
Strategies in Design of Experiments
The most natural and intuitive way to do product testing is to design the product,
make it, and see if it works, adjusting the design if the product fails. This approach
is herein called design-build-test and is still popular, as we shall see in section 4.2.
Over the course of the Twentieth Century, however, the balanced orthogonal array
developed by Fisher has gained wide currency among enlightened product developers,
as later described in section 4.3. Box, Hunter and Hunter were great advocates of
statistical approaches in experimentation.
As we shall see in section 4.3, statistical DoE approaches are often driven by
a corporate Design for Six Sigma (DFSS) policy framework. DFSS is essentially
a recognition ”that upstream decisions made during the design phase profoundly
affect the quality and cost of all subsequent activities[10],” and enacts a Define,
Measure, Analyze, Improve and Control model[11]. Policies created within the DFSS
19
framework in a company which has created an environment amenable to devoting
significant resources to quality improvements therefore specify instructions for actions
that reduce variability in later stages of the product development process explained
in section 1.2.
Policies in DFSS are set by Green Belts, managers trained in process improvement;
Black Belts, trained statistical process improvement specialists; and Master Black
Belts, teachers and leaders in service to the rest of the organization. While they can
be hired in, their training is frequently provided as part of a corporate push for quality
improvement in the company’s products and processes. The presence of belted DFSS
staff at a company is a clear signal that some attempt at a framework for statistical
quality improvement has been adopted, although not necessarily sign of a conducive
environment – which can be developed through Senior Executive Six Sigma training
– or a harbinger of successful polity.
Neither Fisher nor Taguchi invented the idea of changing parameter settings to
find the best effect. A more traditional way to determine the best configurations is to
choose control factors to be parameters after the system design has been done, and
to vary each one in turn from its nominal position. In a system with no interactions,
this nominal one at a time strategy would give the correct influences and point to the
best answer.
Recent work[12] has reexamined the one-at-a-time strategy and found that it
can beat statistical methods under particular circumstances in systems with high
repeatability and strong interactions between parameter settings. The more intuitive
adaptive one at a time (AOAT), in which a parameter has to be left at its best setting,
not its nominal setting, can work better than Orthogonal Array[13]. Vary everything,
then one at a time (VEOT)[14], is a similar strategy where in a rectangular parameter
space the setting for every parameter is changed with each new configuration until
every setting has been tried, then the best is selected and the strategy proceeds as in
adaptive one at a time.
20
2.5
Adoption
Several MIT theses have considered the adoption of Taguchi methods at American
companies. This section reviews some of them.
2.5.1
Morrow
In 1988, Jeffrey Morrow surveyed several five large American companies – Ford Motor Company, Davidson Rubber, AT&T, and ITT – for his Master of Technology
degree[5]. From this he developed an ’impressionistic and anecdotal’ idea of to what
extent Taguchi methods had penetrated these enterprises, and what factors had determined the success of that process implementation.
He found some factors common to process implementation in general, that they
require strong upper management support, are encouraged by market pressure to
change business methods, and that they are more likely to be successful in stably
structured companies; some factors common to process improvement methods, that
they are hindered by poor training materials and less effective when applied piecemeal;
and some special factors, which were excessive ’segmentalism,’ which Morrow defines
as ’tightly drawn lines of responsibility between functionally organized departments,’
and the resistance of statisticians.
2.5.2
Peters
In 1992 and 1993, Antje Peters worked with Don Clausing[7] to inquire of several US
corporations how involved they were in the use of Taguchi methods. She used the
determinants laid out by Morrow[5] and surveyed engineers regarding the adoption
of Taguchi methods at the three organizations that first brought them to America,
Xerox, AT & T and Ford Motor Company, as well as ten others3 .
She found that management buy-in, internal experts and reinforced training using
the case method were the promoters most often cited by her respondents.
3
comprising Polaroid, General Motors Corporation, Hewlett Packard, Hughes Aircraft Company,
Packard-Hughes Interconnect, LSI Logic, Eastman Kodak Company, Lexmark International, Analog
Devices Corporation and the Boeing Company
21
2.5.3
Thomke
Thomke[15] examined the decisions that stakeholders have to make in the design
process around expending resources on experimentation.
Thomke references Middendorf’s[16] division of experimentation into three types:
device evolution, repeated analysis and synthesis. Device evolution is parameter manipulation in the absence of a statistical model, an adaptive optimization like the
adaptive-one-at-a-time strategy. Repeated analysis is experimentation in well developed statistical models, such as orthogonal array or analytic modeling in automatic
transmissions. Synthesis recalls the final step in the Taguchi DoE strategy, the confirmation trial, as it is a simple affirmation of an existing mathematical model.
Thomke examines mode switching, the tendency of a product design entity to stop
doing practical prototype testing and move to some other experimental mode, which
is to say computer simulation, rapid prototyping or mass screening. Mass Screening
is the creation of a particular test for fitness, which is then applied to many possible
design candidates. This experimental mode is largely applied in pharmaceutical drug
discovery. Rapid Prototyping is popular in easily configurable or simple to make
products. Thomke uses the example of a programmable application-specific integrated
circuit (ASIC) chip, and further explores the concept in his case study on IDEO[17].
Computer Simulation, as we will see in section 4.2.1, is a useful substitute for
practical prototype building when the parameters’ settings effects are well-known and
subject to modeling. While simulation can be a fast and cheap way to get results, it
does not supply variability information.
In a follow up paper, Thomke, et al.[18] ”propose that strategies and modes of
experimentation can be an important factor in the effectiveness of a firms innovation
processes and its relative competitive position.”
22
Chapter 3
GABackground
Genetic algorithms belong to empirical mathematics, and are a powerful way to solve
a problem that is unlikely to yield to analytic methods. The central idea is to allow
ensembles of agents to perfect mutable strategies to solve a problem over the course
of generations.
Genetic algorithms will be applied in Chapter 6.5 to optimize an abstracted design
of experiments. This chapter introduces the notion of genetic algorithms and provides
some background.
3.1
History
The notion of genetic algorithms – solutions that are perfected by iterated selection
– may be as old as digital computers themselves, and go back at least as far as
Baricelli[19] and Box[20] (separately) in 1957.
University of Michigan researchers, notably John Holland, initially developed the
Genetic Algorithm approach leaning on Rechenberg’s evolutionary strategy and Fogel,
Owens and Walsh’s evolutionary programming 1 . The missing element is these was the
notion of crossover or structured recombination discussed below in section 3.4.3.
With the 1975 release of both Holland’s Adaptation in Natural and Artificial Sys1
the history of genetic algorithms is more completely explored in Marczyk[21] and in chapter 4
of Goldberg[22]
23
tems and Kenneth De Jong’s An Analysis of the Behavior of a Class of Genetic
Adaptive Systems[23], genetic algorithms were first defined as an area of study. De
Jong found that genetic algorithms could be used to study nonconvex, stochastic processes. He defined the on-line performance xe (s) of strategy s in environment e over
the course of T generations as[24, section 4.6]
T X
n
1 X
xe (s) =
fe (dti )
T · n t=1 i=1
(3.1)
where n is the population size, f (d) is the fitness of agent d and dti is the ith agent
in generation t. The on-line performance is thus the running average of the objective
function value. De Jong also defined an off-line performance
x∗e (s) =
1X ∗
fe (t)
T
(3.2)
where
fe∗ (t) = max (fe (1), fe (2), fe (3), ..., fe (t))
(3.3)
and is the running average of the highest fitness attained by an agent in each generation. The off-line performance retains the best achieved value in a generation, the
online performance averages across all agents with the strategy in the environment.
While smaller populations have better on-line performance, larger populations are
able to find a better answer, and have ultimately better off-line performance[22, p.
111].
Genetic Algorithms were first implemented to try to reproduce nature, and slowly
gained ground in illustrating the shapes of solutions to nonconvex problems, problems
in noisy environments, and discontinuous problems.
Some recent applications of genetic algorithms have been in machine learning and
design optimization, as well as in generic nonlinear optimization techniques. Genetic
algorithms are a rather broad field, and are today readily encapsulated in platforms
such as MatLab or Swarm. The genetic algorithm optimization scheme discussed in
Chapter 6.5 was written ’from the ground up’ in a fourth generation language.
24
3.2
Appropriateness
Genetic Algorithm models are a technique for the optimization of non-linear systems.
They do not require that the fitness function, which for this study will be the payoff an
agent gets less its experimental cost, or its wealth, be continuous between adjacent
points. Nor do they require that a particular parameter choice get the same answer
every time, particularly important here because not only might two players with
the same DoE strategy behave differently, but due to variability in the response
surface, two players with the same experimental design may select different final
configurations, and two players with the same final configurations may get different
payoffs!
This variability represents the intrinsic variability in Product Development, where
payoff and response surfaces are inconsistent and unpredictable, and where making
the right decisions in a rational framework does not guarantee a good result.
3.3
Application
It is useful when thinking of genetic algorithms to understand certain terms[22]. A
gene is a bit in a genome or chromosome, which is a series of true or false values
called a bit string. The gene’s locus is simply its position, which we will describe
with numerical indices. The value of a particular bit string is referred to as an allele.
Patterns found in the alleles of different objects are referred to as a schema (pl.
schemata.) The genotype is the collection of all alleles associated with an object, and
the phenotype is how the genotype affects the fitness function, which is the measure
of the object’s success. Agents, or players in this work, are run simultaneously with
many other players in a generation. They are then selected and bred to produce new
players for the next generation.
The schema theorem states that the patterns in successful agents receive exponentially more representation as the agents iterate through more generations[25]. This,
along with implicit parallelism, the idea that n3 schemata are being tested with any n
25
agents[22, p. 41], is what gives the genetic algorithm approach its processing power.
3.4
Design Decisions
In the creation of a genetic algorithm scheme, certain design decisions have to be made
to distinguish one model from another. This section outlines what the decisions are,
and attempts to point toward implications. The decisions as made are discussed in
section 6.4.
3.4.1
Population Size
Population size determines how subject the population is to premature convergence
through gene starvation. Too large a population size will slow the emergence of a
best answer.
3.4.2
Selection
The fitness function, or objective function, comes into play when the selection mechanism is enacted. Fitness is determined through some scalar measure of the strategy’s
performance, and is generally scaled in some way.
Brindle[22, p. 121] suggests six schemes for selecting the strategies to be propagated to the next generation. A generic agent was to create a strategy by
1. Deterministic Sampling: probability of selection is fi /
P
f , the agent’s fitness
over all the sum fitness for all agents.
2. Remainder stochastic sampling with replacement: fi /µf , the ratio between one
agent’s fitness and the population’s mean fitness, is calculated. The strategy is
awarded the whole part, the remainder is used as a probability for additional
selection.
3. Remainder Stochastic Sampling without replacement: As in the previous scheme,
but with the probability, which is less than one, zeroed out once the strategy is
26
selected.
4. Stochastic sampling with replacement: The ”roulette wheel.” All of the fitness
evaluations fi are placed end to end, and a uniformly distributed random number U [0,
P
fi ] determines which strategy is added to the selection pool. If for
example, the number 0.0132 was selected, the first agent had a fitness of 0.0072,
the second 0.0031, and the third 0.0049, the third agent would be selected. This
is referred to as a ’with replacement’ strategy because the same agent would be
selected again if another random number u s.t. 0.0103 <= u < 0 is picked.
5. Stochastic sampling without replacement: Like the roulette wheel, but with
the probability of selection dropping by 0.5 (or to 0)2 every time a strategy is
selected as a parent
6. Stochastic tournament: The stochastic tournament selects a pair of agents using
a roulette wheel scheme, then compares them to one another, discarding the one
with the lower value.
Goldberg[22] reports that Brindle found little difference in performance between genetic algorithms using these different methods, except for deterministic selection,
which performed poorly.
3.4.3
Crossover
In order to generate new strings, agents are mated by splitting the genome and
pasting together different alleles from different agents. So, therefore, at which point
the parental genomes are split and how they are constructed are important design
considerations for the genetic modeler.
A model could use a single or multiple genomes to represent a player, could take
advantage of diploidy to allow alleles that are currently unhelpful to persist in a
dynamic environment.
2
De Jong developed this scheme, and selected the parents separately, which is why this number
is 0.5 and not 1.
27
Crossover has the greatest impact on how well the phenotypes of the children
represent those that were successful in their parents.
3.4.4
Mutation
However, mutation can also have a great impact. Mutation is the easiest concept in
genetic algorithms to express or conceive, as it is essentially the chance that any given
bit, once determined by generation, selection and crossover, will flip. A mutation of
0.1 reduces a genetic algorithm to little better than a random search, and at a rate
of 0.5 they are literally the same[22, p.111][23].
28
Chapter 4
Practice
4.1
Method
To assess the actual usefulness and status of Design of Experiments in various enterprises, a survey was constructed and delivered by phone to four or five practitioners
in several industries. We found these DoE practitioners by use of three techniques:
1. Through personal connections with professional engineers
2. Through academic ties, using fellow students, alumni or faculty
3. Through professional associations
We were able to recruit seven of these practitioners into confidential conversations
regarding their practice of DoE. The interview notes were then edited for clarity and
sent to the subjects for review and approval.
These conversations were meant neither to provide a holistic look at industrial
use of DoEs nor to follow up in detail on any previous study. Our intent is merely
to ask practitioners close to hand how they use design of experiments, what are their
approaches to it and issues with it, and how well they see it as working for them.
29
4.1.1
Conversations
The interview guide is included in Appendix B. Interviews took place at a time of
the subject’s choosing over the phone, and took approximately an hour.
The interview guide investigates such concerns as mode switching, parameter design, policy implementation, response to management time pressure, testing budget
and DoE philosophy. Interview subjects were forthcoming about each of these topics,
except where their experience did not help them answer the questions.
Conversations did not follow the guide exactly. Often, the questions that were in
the guide had been answered or rendered irrelevant in the answer to an earlier section,
and interesting comments the interviewee made were followed up on, rendering the
conversations at times discursive, and at times digressive.
4.2
Profiles
The following profiles correspond roughly one to one with the interviewees. In one
case, an interviewee was willing to speak to DoE strategies in two different industries,
and has been split into two profiles.
4.2.1
Vehicle Handling at a High Performance Car Development Interest
Respondent #1 is engaged in design and development of the chassis his company
produces. While his corporation has no official policy or process for design of experiments in quality engineering, his division has a semi-formal policy set around the use
of DoE. He uses D-Optimal Designs/citeneubauer:dopt to get the coefficients of his
quadratic model for parameter effects, which he has been using for the last seven of
the ten years he has been doing design of experiments.
Respondent #1’s experiments are analytical primarily, with no variance in response. He uses DoE to unwind the strength of a control factor in a particular mix.
The DoE experiments, of which he typically does between 100 and 120 for his 1030
factor, 3-level system can take up to an hour of computing time each. Once he has run
them all, he can very quickly see what the effects of changing particular parameters
would be.
Typically, his statistical model has an R-squared adjusted fit of 0.98 or better
with the response of his finite element analytic model. Once the model is in place,
Respondent #1 uses genetic algorithms to find what he calls the ”best balance”
configuration for his objective function. He does not call it ”optimum” because it
uses weighting factors based on engineering judgment. Respondent #1 chooses his
three levels for the parameter design pragmatically. Knowing that his decisions will
go into the final detailed design for an actual product, he selects the experiment run
order in physical DoE’s that varies the factor which is hardest to change the least
amount.
The range for his factor levels are based on experience with the system and what
is judged feasible within design and development and many constraints including the
laws of physics, package, design for manufacture and cost. Vehicle dynamics, the
performance category Respondent #1 works in, heavily relies on subjective ratings
of vehicle attributes. DoEs are used at times in a physical system on the road and
track to match this subjective feeling to measurable performance objectives like various vehicle accelerations, gains and displacements. Various parametric and detailed
vehicle dynamic models are also utilized heavily for this task.
In addition, Respondent #1 and his team completes these DoEs in labs on hydraulic ride simulators, otherwise known as a four-post rig because the vehicles four
tires sits on four hydraulic rams that move vertically to simulate the road. The fourpost rig is a more controlled environment than the road and track as it does not suffer
ambient weather changes, resultant track changes, driver line changes and ultimately
tire behavior change. Therefore, it reduces the amount of noise to deal with in the
system experiment.
Initially, he starts with a two-level screening DoE to find out which of around 15
factors are important, then returns with a three-level DoE on a few selected factors.
It has been about three years since he has felt a need to do a practical DoE, because
31
of the high r-squared fit of the model; he estimates that he has only done twenty in
his ten-year career.
As noted above, the outer array of the test track, with rain and lain rubber, can
be hard to control, and so he prefers to do his testing in the lab. He can do lab
experiments on components, which have fewer control factors and typically fewer interactions, using a quarter-car or a whole-car test rig. He will do five or six repetitions
on the track, and only three to five in the lab.
Respondent #1 is trying to improve twenty or more vehicle responses. Most of
them are improved through minimization, although others are best maximized, like
Aerodynamic downforce and average vertical tire loading, and some are ”optimum”
at set values. Respondent #1 has also worked in designing automotive race cars, for
which the vehicle dynamic models were not as well developed. That experience, he
feels, is more typical of the industry. ”80
Respondent #1 feels that most try to build ”mental models” and engineering
knowledge through design-build test and one-factor-at-a-time strategies. He believes
this is insufficient in complex dynamic systems, like a automobile, which invariably
exhibit significant interactions, non-linearities and far too many factors and responses
to keep track in a person’s head. Respondent #1 believes in a balance of strong
theoretical laws of physics knowledge, analytical modeling, physical design-build-test
iterates and DoE application. Otherwise, one tends to end up at local ”optima” at
best and have spent far more resources and time to get there. Ultimately, he has no
doubt in the value of design of experiments. As one example, he points to how he
completed four major kinematics and compliance DoEs recently on a project that he
estimates would have taken six to eight months longer with lower performance using
design-build-test.
He defines his test budget in terms of scheduling; he’s constrained by prototype
availability, parts availability and project timing. He uses a mostly intuitive process
for determining the cost-benefit, driven by his confidence in the model he is using.
”All models have limitations. Models are just abstractions of reality. You have
to understand any model’s strengths and weaknesses to gauge which information you
32
can use when and where.”
4.2.2
A Master Black Belt at an Automatic Transmission at
an Automotive Manufacturing Company
Respondent #2 performs DoEs on automotive transmissions, seeking ways to decrease
costs in the automatic transmission in future production runs of the same vehicle.
While other engineers in her group use analytic DoEs, her DoEs are purely practical.
The automatic transmission is a complex system in and of itself. In order to
create realistic models for the transmission, Respondent #2’s team had to consider
both the theory of gear sets for the hardware component, and of hydraulic stability,
which came out of analytic work on the origins of shift quality. As her main concern
is reducing the vibrations due to shifting, all of her optimizations are of the ”smalleris-better” variety. She works with her six-person group to decide what the objectives
should be, but typically measures gear noise, ’XYZ’ noise, torsional access measure
and airborne noise.
In order to provide as pleasant a ride as possible, the vehicle handling department
at the product developer visited in section 4.2.1 needs to be able to predict the
vibrations of components like the automatic transmission. It is therefore important
to Respondent #2 that she be able to shift the mean of the transmission’s vibration
as little as possible when the gear changes, as well as minimize its variation. To do
this she uses design of experiments.
The first step in her experimental design is to do some probe testing, choosing
well spread configurations as in uniform design, and repeating them until the standard deviation bounds. A desire for a 90 % confidence level gives her the notion of
variability she uses on her DoEs.
While Respondent #2 assumes constant variability across her design space in the
design of her experimental strategy, she acknowledges that this is sometimes clearly
not the case, ”Sometimes in a solenoid coupled to a hydraulic package, the variability
is quite large in one part of the parameter space compared to another.” In these cases,
33
where variability grows extremely large at particular configurations, she and her team
have to do many repetitions in order to establish the variability. It is only in such
systems that she sees DoEs granting no measurable result.
Respondent #2 typically limits her parameter selection to three or four control
factors, or ’knobs,’ and prefers to select knobs that do not have a large impact on
cost, recalling ” I ran a gear noise DoE a few years ago, and burned through a halfmillion dollars pretty quickly.” The expense of her DoEs are driven by time; which is
consumed in setting up and breaking down the experiments as well as by performing
and analyzing them.
While her experiments generally use two-level parameters, many of the physical
systems do have curvature to them.
She sees one great advantage in working on the continuous improvement of a
released product, instead of its development. As the product has a warrantee history
and a product scrap history. This lets her do realistic cost-benefit analyses when
designing an experimental design. However, she suggests greater focus on variability
reduction in product design.
4.2.3
Engine Design at an Automotive Design Concern
Respondent #3 does practical and analytic DoEs on engines in new product development. The last product she worked on was designed from the inside out using Robust
Engineering principles.
The experimental design of the engines is driven by the deep engineering understanding of the system built up over decades, and by reverse engineer and X-ray
analysis of competitors products. Analytic models using finite element analysis represent the engine, and the experiments are run against these models.
4.2.4
Consulting at a Corporate Engineering Center
Respondent #4 has been working at a center for engineering excellence (CEE) at a
large corporation with diverse products since graduating college in 1997. He took a
34
DFSS course in 1998, and perceives DoE as being part of his employer’s six sigma
push.
As an engineer at a CEE, Respondent #4’s responsibility is to help other business
units fill gaps in their expertise to bring new ideas to market. He spends much of
his time in concept selection using voice-of-the-customer methods like Kj to capture
needs and translating those needs into requirements using tools like Triz, QFD and
the Pew selection matrix. However, he is occasionally called upon to do DoEs.
To do practical DoEs, Respondent #4 uses minitab to generate a list of experiments once he has input a presumed statistical model. He last used DoEs in the year
2000 to optimize the performance of a guidance system for a ground vehicle.
Magnetic beacons were lain in the vehicle’s path, and Respondent #4 varied
around a half-dozen parameters, all of which, like the type of sensor, had discrete
values except for the speed of the vehicle, which could vary continuously. He used
DoEs to determine how far apart the beacons could be placed and still define a path.
Other engineers in his group run DoEs against finite element models. In Respondent #4’s team, they use their embedded engineering knowledge to decide when to
run a DoE and which control factors to use.
4.2.5
Black Belt at an auto parts manufacturer
Respondent #5 was trained as a black belt by her employer, which has its own
training program. She has started to move into a Master Black Belt role, training
other staff on statistical methods. She works with design of experiments frequently
in three regimes:
1. Manufacturing process improvement
2. Manufacturing trouble shooting
3. New Product Development
Respondent #5 recently worked on a muffler assembly. Her company formed an
interdisciplinary team to develop a new muffler assembly. She was the six sigma
35
technical expert and was charged with creating the experimental design. After a
few meetings, she proposed doing multiple repetitions on the experimental design in
order to capture some of the variability, and was ”was laughed at in the meeting.”
Respondent # 5 ruefully concludes ”One [experiment] is better than nothing.”
Respondent #5 is sanguine about the lack of repetitions. While her statistical
training recommends them, she feels the benefit would not be worth the cost, and
that the testing environment and manufacturing processes are so well-controlled that
any variability would be unimportant.
The prototypes took hundred of dollars to build; the product was eventually canceled because of its cost. Respondent #5 and her team had six months for the project.
A larger driver than prototype cost in the non-repetition of their experiments was
perhaps the minimum standards they had to pursue. Generally in new product development, Respondent #5’s company is given specifications by a systems integrator
and stop testing once the meet the standard. They have no incentive for reducing
variability or delivering performance better than specified.
Respondent #5’s team started off identifying important control factors an interactions using their physical intuition, physical analysis, knowledge and experience.
They created a list of about 30 control factors this way. After considering them analytically, they drop to about 20, thence to around 12 when the uncontrollable or
insignificant ones are dropped.
Then the team ran an L12 Plackett-Burman design, which is a kind of orthogonal
array[9, Section 7.4] for 11 factors at 2 levels. This screening DoE allowed Respondent
#5 and her team to focus on the four most important factors, on which they ran a
full factorial with a center point, for a total of 29 prototypes.
The center point was to seek evidence of curvature, which was not forthcoming.
If she had found curvature, she would have had to review the results from the other
prototypes and see if any of them were already good enough. If they weren’t, she
would have had to go back to her customer to negotiate for more money, perhaps
enough to do a response surface design.
36
Respondent #5 is chary of aspects of Taguchi methods, calling the quality loss
function ”just a teaching tool” and indicating that three-level parameter design gives
up much of the advantage of DoEs by requiring so many more experiments. ”Taguchi
provided a simple, practical method but sometimes other methods give better results.
It all depends on the applications, situations, and how people use them.”
4.2.6
Software project manager
Respondent #6 is a project manager at a transit consultancy. The consultancy provides a customizeable software platform to transit authorities, and performs safety
inspections on transit system elements. In neither endeavor does the consultancy use
DoE.
Respondent #6 exclusively engages in design-build-test. ”We are not so sophisticated that we test the products against different platforms.” If his company sells
additional components or reports with the system, they will test them. If the components do not work, they return and debug them.
The testing budget is decided from experience with similar types of products for
the same or similar customers. They stay on budget about 15 % of the time. ”Usually
the smaller the task is, the easier it is for us to come in under budget.”
4.2.7
Decision Support at a Large Commercial and Retail
Loan Generator
While replacing terminology like ’performance’ and ’variability’ with ’reward’ and
’risk,’ Respondent #7 believes her work is fundamentally the same as that of a quality
engineer working on an engineered product. She handles a diverse array of issues using
DoEs, including capital allocation for the bank, process improvement, and individual
loan decisions for retail and commercial customers. These regimes are all treated
differently as far as the decision making process; they all use design of experiments,
but somewhat differently.
Belted DFSS specialists as discussed in section 2.4 work out statistical recipes to
37
use for evaluating applicants for different types of lending products. These recipes are
adjusted by each process owner by his assigning weights to the different components.
It is this customized recipe that is built into software that loan officers run on their
desks.
The lender does do some practical DoEs in their new product group, adjusting
product parameters and testing the market response. However, in Respondent #7’s
area, the testing is all on analytic models. Unlike in engineering, where parameters are considered independent to first order per the hierarchical ordering principle
(c.f. Wu and Hamada[9, section 3.5]), the parameters in her models are strongly
interdependent.
However, the statistical model is not developed enough to create a fractional
factorial to serve it, and Respondent #7 favors uniform design. ”Orthogonal array
can require too many experiments, and not give the best coverage. I don’t need
complete orthogonality; I need the best information. Uniform design techniques are
far more popular in the practice.”
In another distinction from physical engineering, whereas in something like Respondent #1’s work in vehicle handling quite complex finite element models can take
hours to run, in financial modeling a complete experimental design with repetitions
can be run in two hours.
In capital allocation, Respondent #7 attempts to maximize value-at-risk. The
stakes for the lender are much higher in a capital allocation model than in a lending scenario, and the choice to use design of experiments is made by the corporate
leadership. First a model is built using the loss database, market information, historical context, and whatever other information is appropriate to a particular scenario.
These models may include analytic models particular to a scenario – a valuation might
use a Merton or Black-Scholes model. This is an analog to the physical analysis of
an engineering system.
Once the model is developed, it is used to predict the probability distribution
of the output. Since the inputs are correlated and interact in a very complex way,
normally distributed input parameters do not create normally distributed output
38
parameters, and the only way Respondent #7 is able to establish the distribution for
her objective function is to perform a monte carlo simulation.
In credit risk applications, Respondent #7 has some ten objective functions she
attempts to optimize with hundreds of parameters. Typically, she will select a score
or so of knobs to tweak. She uses this model and a monte carlo simulation to establish
rules for making credit decisions. The model does not behave linearly – credit ratings
of 41 or below will be treated the same, while above four the value of the loan will
drop steeply.
Respondent #7’s parameter level selection is driven by nonlinearity and competitive information. She needs to capture as much curvature as possible, and needs to
see the implications of matching the other offers in her market. For example, price
elasticity is a parameter that effects probability of default or mean time to default
in strongly nonlinear fashion, and for which other lenders will have diverse values.
To do a stochastic assessment, she makes her number of levels equal to her number
of runs for uniform coverage. If she uses a latin hypercube, she can distinguish 27
parameter levels in a 27-run DoE.
Respondent #7 has no formal methodology for balancing the number of repetitions
she does with the number of configurations that she tests. However, she does need
to do many repetitions to establish expectation values and variances in any of her
models, and almost always has a large number of control factors.
Finally, Respondent #7 uses DoEs to assess operational risk, which includes risk
to the bank’s reputation or credit rating, risk of legal actions like law suits, and so
forth.
4.2.8
Naval Marine Propulsion
Respondent #8 develops shafts and rotors for sea-going vessels. As he is a defense contractor, the specifications that his group works with specify minimum performance,
reliability and safety standards for his product. Surpisingly, given naval architecture’s
proud engineering history, DoE is not generally practiced nor attended to.
1
Credit ratings vary from one to ten, getting worse as they get higher
39
In the same industrial sector, on the other hand, Power generation uses DoEs extensively and has a well-developed list of tradeoffs defined. According to Respondent
#8, The power generation side of the business has become involved in DFSS and
employes quality engineering tools.
While Respondent #8 is aware of some DoE work that has gone on in his division,
he maintains that it is ”rarely very formal, and tends to be full-factorial in nature.”
4.3
Analysis
DoE is alive and well in American companies. One surprising result wwas how much
each practitioner felt his or her experience represented all DoE practitioners in design.
While the development of statistical models and the policy of experimentation vary
widely, there are some broad similarities.
DoE practitioners typically do a two-level balanced screening DoE – often one of
the Plackett-Burman designs – to select control factors, then investigate curvature
with three or more levels on those knobs, replacing real-system experimentation with
numerical models where possible. Experimentation and theory eventually lead to
reliable statistical models, at which point the experimental design is determined by
a software tool.
Based on these interviews, we can define a five layer model for intentional process
support in an organization. At the broadest level, an environment is created that is
amenable to the activity. This allows the creation of a framework in which policies
can be articulated. Next come the policies themselves. Once the policies are in
place, instructions can take place inside that policy, and on the most immediate level,
actions are performed that enact the process. In business process change, they would
enact the transformation, or, rather, the new way of doing things.
These layers are all dependent, each on its neighbors. Actions without instructions
are ill-coordinated, and very possibly harmful. A policy is moribund if not followed
up with instructions, and suboptimally effective if not articulated within a framework.
We can see from our investigations in 4.2 that the introduction of formal design
40
of experiments strategies into product development processes follows this model.
• An environment is created when the senior management decides that reducing variability and improving quality are important goals to which company
resources should be allocated.
• A framework is put in place by working with consultants, academics, publications and peers to create an operation goal of Design for Six Sigma, Robust
Design or some quality improvement program. This framework is supported by
seminars, classes and incentive structures.
• A policy of employing design of experiments to discover and manipulate control
factors is articulated within that framework.
• An instruction comes from the engineer, and engineering manager, or through
software code to a practitioner to perform a DoE.
• The action of performing the DoE is completed.
Most of the interviewees were involved in stable DoE practices where these levels
were fully realized and harder to see. However, we can look at Respondent #4’s
experience with the ground vehicle as examples of instructions without a policy that
did not persist. Respondent #2 and Respondent #1’s experiences differ because one
is in an organization that has an explicit and defined policy in DoE, and the sets
policy on the same level as instruction, whereas the company provides a framework
in which these policies can be created.
DoE was in many cases a ’Design for Six Sigma’ (DFSS) framework push. The
main implementation mechanism were classes in experimental design, and the adoption of the DFSS Black Belt hierarchy. The DoE practice was introduced into organizations by a specific DFSS vendor, who taught classes to create Green Belts of
design engineers. These Green Belts generally decide locally the best way to apply
these statistical practices. From the ground-level view of this survey, this strategy
works impressively well in creating statistical control in product design.
41
Tools such as minitab are useful to the statistical experimenter in creating the
experimental design, and their availability and usability may explain why orthogonal
array designs were not common in our survey group. GASolver can help determine
the best configuration in a nonconvex response surface once the statistical model has
been performed. Microsoft Excel is used by nearly all practitioners for some part of
the process – tracking experiments, statistical analysis, or plotting the output.
At what organizational level the policy was created depended on the stakes for
the company – the lender specified DoE for capital allocation at a department level,
but slipped it into loan approval in tools development – and on what level of the
organization variability reduction had an advocate. These drivers are more explicitly
lain out in Morrow[5].
As expected by Thomke[15], computer simulation has come to dominate design
of experiments. The combination of a believable system level mathematical model,
which allows experimentation to take place in a few hours, and a reliable statistical
model, which obviates the need for running a full factorial with that system level
mathematical model, creates a powerful ability to perfect well-understood product
systems.
However, in some cases, such as Respondent #7’s coworkers in new product marketing for financial instruments, the statistical model is not well-developed and the
underlying mechanisms are not well-described. In these cases, DoE practitioners
tend to stick with statistical models that handle main effects and, perhaps, first order
interactions, depending on the time and money they have to spend.
That time and money is afforded them gladly. In this study, we did not look at
DoE evangelists who are working to overcome policy impediments in their organizations, and found that the engineers had enough space and money to complete DoEs
that they found were worthwhile. To some extent the number of experiments was
budget limited, but in no case did a practitioner express frustration with management
pressure to cut schedule or cost by reducing the scope of their experimental design.
The above is not to say they expressed no frustrations whatsoever. While the
practioners all felt that reducing variability was worthwhile, and that their design of
42
experiments strategies were well chosen, most of them had a message that they wanted
to include. These represented frustrations with their industries, tools or companies,
and I have listed a sample here.
• Concept selection is now driven by preconceived business needs whereas in the
past our corporation has gotten a lot of benefit by creating new products and
then finding needs for them.
• The industry could be doing more optimization of subsystems
• Variability reduction in product design could go a long way. We spend a lot of
money on premium production processes when we could design sensitivity out.
4.3.1
Improvements
We will follow Frey, et al.[12] in suggesting that the quest for a statistical model is not
always worthwhile. In product development – especially in a case like Respondent
#4’s, where the products are very diverse, or Respondent #2’s, where the performance is determined by complex interactions, an experimental design that implies a
statistical model is not obviously preferable to one that follows some other type of
optimization algorithm.
The assumption of homoscedascity, which is to say, that variability will be constant in all parts of the parameter space, persists even in physical environments where
the practitioner knows the assumption is false. This results in as many repetitions
being performed on configurations where the variance is more accessible as on configurations where it is difficult to establish. One way to improve the cost and quality
characteristics of an experimental strategy would be to distribute the number of repetitions according to the repeatability of the response. Indeed, as the variability of
the response is one of the measure parameters, one can envision an iterative process
where measured variability determines continued repetition.
Many engineers at the system level were contacted and do not use design of experiments. While they generally insist that DoEs are necessary for component development, they generally feel that the state of the art in systems engineering is not
43
well developed enough for DoEs to be worth the considerable investment in time and
money required to do repeated prototype testing. Further the benefit of doing DoEs
is not expected to outweight the difficulty and expense of building the detailed system
level mathematical analytic models that it would require.
This stance seems to be belied somewhat by the fact that computers are now powerful enough that engineers like Respondent #3 can do DoEs against analytic models
for automotive engines. While it would be facile to recommend that all systems engineers start performing designed experiments on analytic models, we would suggest
that the decision to do so or not take into account the current quite powerful state
of the art in computing and the demonstrable benefit some companies have obtained
by developing these models.
Respondent #5 points to a breakdown in the supplier relationship. To the extent
that her specifier had a quality engineering framework in design, it clearly did not
extend into her enterprise. While Respondent #5’s company had adopted quality
engineering for its manufacturing process, its capability appears to lag in design. As
they do not seem to be much engaged in the business of new product development,
this may be the optimal place to be. However, her experience does suggest that tiered
or relative reward structures be worked out to encourage vendors to overdeliver.
Lastly, Respondent #8’s experience is somewhat worrisome. While some quality
engineering recommendations are starting to be deployed in military – and notably
United States Air Force[27] – specifications, the notion that defense contractors are
complacently meeting minimum standards and eschewing opportunities for continuous quality improvements points to avoidable waste for which the tax payer is held
to account. We would suggest that the other forces follow the Air Force’s lead and
begin to incorporate quality engineering requirements in their contracts.
44
Chapter 5
Comparator
The main output of this work is a software-based comparison framework for design
of experiments strategies. The software assigns any codeable1 design of experiments
strategy a particular value for a particular well-defined response surface.
5.1
Concept
Before prototype testing can begin, the product designer must decide on his experimental strategy. Some analyses have been done to attempt to rationalize the decision
making process by creating computer representations of experimental strategies and
running them against different response surfaces[13]. We believed it would be useful
to extend the concept to create a generic framework with pluggable interfaces for any
arbitrary strategy run against a defined response surface, with variable parameters
for repetitions, payoff structures, including targets, payoffs, balance and any creative
response one wishes to test.
As the enactment of a product design strategy will be quite variable in the payoff
that it obtains, the model employs a monte carlo simulation and allows statistical
moments of large ensembles of agents performing the coded strategy against the
same data set.
1
as described in section 5.2.2 under ’Strategy’
45
Figure 5-1: Class Structure for Comparator
5.2
Architecture
The Comparator consists of an object oriented computer application, with a Player
object that provides an interface for a DoE Strategy object. The central AirplaneWorld object also provides an interface for a Game object, which in turn
would provide an interface to a Goal object. Its basic structure of the Comparator
is shown in figure 5-1.
46
5.2.1
Platform
This model runs on Windows 2000. It was developed in Java using Borland JBuilder
8 personal edition, 9 personal edition and X foundation edition. Data was reported
by Microsoft Excel from the data store in Microsoft Access. The Comparator reached
the data store using JDBC; the reporting tool used ActiveX Data Objects.
5.2.2
Modules
There are five types of modules snappable into the framework: goals, games, data
stores, parameter spaces and strategies. Each of these has a corresponding Java
interface, and are implemented by the Goal, Game, DataStore, ParameterSpace
and Strategy interfaces respectively.
Goal This allows different sorts of goals to be set for the players. Whereas one game
might have a parabolic quality loss function around a particular point, another
might give full credit for any response below a particular level. The Goal
interface lets the designer distinguish between less-is-better, more-is-better, and
optimal-is-best environments, and to specify a shape for the payoff function.
The data discussed in this chapter uses a ParabolicDistanceGoal, which
delivers maximum value at a particular point then parabolically decaying value
further from that point.
Game The game provides the interface to the response surface. Any configuration
passed to a Game.experiment() will deliver a result, which will be specified by
data or a Game’s number generator. The game also serves to connect players to
the goal object. The data discussed in this chapter uses a NormalDataGame,
described below in section 5.3.2.
DataStore This interface allows the development of data access modules for other
types of data stores. The data store interface a player uses to interact with the
Microsoft Access database is a ConcreteDataStore.
47
ParameterSpace This interface was intended to allow flexibility in the specification
of parameters and their level settings. However, some of the software in the
framework can only handle RectangularParameterSpaces, which have the
same number of levels for each parameter, and the framework has not been used
with anything other than a 4-parameter, 3-level parameter space.
Strategy This is the main interface of interest, and defines what is mean by a codeable design of experiments strategy. It allows objects to report to the player
which configuration to try next, and how many times to try it. It also provides
a reporting mechanism for the player to record what response a particular configuration got. Finally, it takes this information and instructs the player which
configuration to use to try for the final payoff.
5.2.3
Tables
The Microsoft Access database stores game, flight, reward and player data in a number
of tables. The Game table records the game identifier, the type, target and tolerance
for the goal, the number of reward attempts (here always three), the signal to noise
ratio and the interaction strength. The Player table lists the players, how many
repetitions each does on a configuration, its strategy and the wealth the player has
earned.
The Flight table links to the game and player tables, and lists when a plane was
thrown, the settings of parameters A, B, C, and D, and the distance the plane flew,
as well as when the flight was recorded. The Reward table also lists the parameter
settings and distances, as well as linking to the player and game tables. Instead of the
time, however, it records how much payoff its player got for that reward experiment.
The database also has a store for the Grewen data.
To support the genetic algorithms, there is an alleles table, which lists the allele
indices, their value, and to which player it belongs. A table named ZeroToFifteen
links each allele index with an integer equivalent.
48
5.3
The Data
Data can be selected from a predefined set of responses from any configuration, a
pseudorandom generator that uses a configuration as a parameter, or some other
number generator that satisfies the Comparator’s Game interface. For the results
given in section 5.8, we used a parameterized NormalDataGame object.
5.3.1
Paper Airplane Game
Eppinger[28] developed a paper airplane experiment to illustrate methods in prototype testing and robust engineering. The experiment uses a sheet of paper with a
number of lines drawn on it. The lines correspond to design choices one can make
by folding over a particular line or another according to whether the flaps should
be up, down, or flat, how long the wings should be, how tightly they should angle,
and where an ancillary paper clip should be placed for weighting. The simple format
defines four design variables or control factors and 3 levels of each of these variables.
Jennifer Grewen employed this airplane template in her undergraduate thesis[29].
She threw 10 airplanes in each of the 81 possible configurations as a full factorial
design with 10 repetitions. This data set is hereafter referred to as the Grewen data.
The Grewen data is reproduced in Appendix A. This set is the test bed for the
model and all strategies that are encoded in the model. The original flight distance
ranged from 5 to 27 feet; configuration means varied from 8.15 to 21.8 feet. In order
to make the data easier to interpret, it was scaled such that the smallest mean (for
configuration 32132 ) was zero and the largest mean (for configuration 2111) was 100.
That scaling for the unadulterated data was
x0 = 7.326(x − 8.15)
(5.1)
where x is the measured flight distance in feet, and x0 the same measure in scaled
distance.
2
configurations here are referred to by their A-setting, their B-setting, their C-setting and their
D-setting, each of which can be 1, 2 or 3
49
A1
C1
Parameter A:
Weight Position
:
ter C h
e
m
t
Para e Leng
s
o
N
C2
A2
C3
A3
Expt.
#
1
2
3
4
5
6
7
8
9
Weight
A
A1
A1
A1
A2
A2
A2
A3
A3
A3
D3
D1
D2
Stabiliz.
B
B1
B2
B3
B1
B2
B3
B1
B2
B3
MIT Design of Experiments Exercise v2.0
D3
D1
D2
D3
D1
D2
Wing Ang
___
_
_
_
t #_ _ _ _ _ _
n
e
_
erim _ _ _ _ _____
p
x
E ance
___
_
t
s
_
i
D
e __
m
a
Parameter D:
N
le
Nose
C
C1
C2
C3
C2
C3
C1
C3
C1
C2
Wing
D
D1
D2
D3
D3
D1
D2
D2
D3
D1
B1 (up)
B2 (flat)
Parameter B:
Stabilizer Flaps B3 (down)
Table 5.1: Variance Components in the Grewen Data
Input variables
Constant term
C
D
A*B*D
C*D
B*D
A*C*D
B
A*B*C
A*B
A*D
A
A*C
B*C*D
B*C
A*B*C*D
Coefficient Std. Error
14.7127161
0.10078103
-1.01203704
0.12343105
-0.82537037
0.12343105
0.78458333
0.18514659
0.5138889
0.15117155
0.48277777
0.15117155
-0.45416668
0.18514659
-0.29759258
0.12343105
0.43958333
0.18514659
-0.35888889
0.15117155
0.28277779
0.15117155
0.10611111
0.12343105
0.11111111
0.15117155
-0.09166667
0.18514659
-0.05
0.15117155
-0.0625
0.22675732
p-value
0
0
0
0.00002694
0.00072983
0.00149454
0.01450977
0.01627129
0.01796453
0.01797388
0.0619926
0.39038122
0.46268675
0.62074649
0.74097294
0.78295088
SS
175335.8594
553.0782471
367.8675842
147.7370453
95.06944275
83.90677643
49.50416565
47.82313156
46.37604141
46.36844635
28.7867775
6.08016634
4.44444466
2.01666665
0.89999998
0.625
The individual data points thus vary from -23.1 to 138.1 in the scaled form; in
the unmodified Grewen Data, the unscaled global mean of 14.7 feet corresponds to
a scaled length of 48.1. As the reader will note in section 5.3.3, the data is modified
to accommodate investigations of the effects of nonlinearity, and the scaling at that
time is shifted to keep the means between 0 and 100.
An analysis of the variance in the data set shows that two of the third order
interactions are more significant than two of the main effects, as shown in table
5.3.13 . This table is sorted by p-value, the probability that a component’s effect
differs from zero. The bottom three components, B*C*D, B*C, and A*B*C*D, have
a better-than-even chance of not existing.
This sorting roughly corresponds to magnitude, but differs because some components, like the main effect of B, have lower variability than their larger neighbors,
and so a higher p-value. Sorting by size, B, the third largest main effect, would be
the tenth-largest of the sixteen terms, not the seventh. This ordering completely
3
This table was generated by Mohan Hsieh using XLMiner from Cytel Statistical Software.
50
undercuts the hierarchical ordering principle introduced in section 4.2.7, one of the
assumptions underlying the orthogonal array.
As explained in section A.1, the tokens represent the control factor interactions;
”A*B*D” can be thought of as the effect that A has on how the level of B changes
the contribution of the level of C to the response (the functions are commutative, so
this could just as well read ”B*D*A”.)
These sixteen components – the mean, the four main effects, the six cross terms,
the four three-way terms and the singe four-way term – were selected for simplicity
and clarity. There are a total of 81 terms, enough to explain all of the differences
between the means of the configuration. The missing terms contain the effect of a
parameter on itself.
The first column of table A.2 is labled ’Dif,’ and shows the difference between the
flight distance we would have expected based on our model, and the actual configuration mean. The Dif column is small, but significant. It is possible that interactions
with low p-values (that are definitely different from zero) are among these higherorder terms and are dropped. These missing 55 terms, which is to say, the ’Dif’
column, together crest five at one point, but are more typically less than one; there
is no reason to think the unexpressed terms will be larger than A ∗ C ∗ D. In any
case, the point that the Grewen Data is violates the hierarchical ordering principle
and will not easily yield to an orthogonal array would still hold true if it were.
In addition, this data set is very noisy. The final column in the table A.1 is the
signal-to-noise ratio for each configuration, which is calculated as
ST Nt = abs(
ȳt − ȳ
)
σt
(5.2)
and gets as high as 21.88 for configuration 1233, but is for 51 of the 81 configurations
less than 1, and for 26 less than
1
.
2
As we shall see in section 5.3.4, an average
datum’s difference from the global mean is only about twice the noise. The outer
array – effect external to the design decisions that affect the objective function –
are quite important in the throwing of paper airplanes. While the thrower’s style
51
and height were constant in this case, we know nothing about the constancy of the
attitude or thrust of the throws, nor of the folding of the airplane.
Even the most disciplined paper airplane maker and thrower will have to deal with
a large outer array – air is a turbulent medium and subject to all sorts of unrepeated
drafts. The paper from which the airplanes are constructed will equilibrate with the
atmospheric humidity over time, but only at its edges or when it is at the top of
the stack. Add to these sorts of uncontrollable effects the poor discrimination in
mensuration, and repeatable data becomes an unlikely affair.
5.3.2
Normal Data Games
We created a class of number generators called DataGame objects, data pickers
which selected data from predefined sets as discussed below in section 5.3.1. However,
this presented two problems.
1. The EMV players described in section 5.5 presumed a normal data set, and
could not adapt to the discrete distribution the actual 10 data given by the
Grewen data for each configuration.
2. the Grewen data has only 0.5 foot (scaled to 3.664 ) precision, and results generally in the 12’ - 17’ range. This results in much repetition in the reported data,
making statistical moments difficult to calculate. The agent can not in these
cases get a sense of the variability associated with a configuration.
We therefore created a NormalDataGame object, which calculated the means and
populations standard deviations for the response associated with each configuration.
It used these parameters to generate a gaussian pseudorandom response.
The difficulty in striking a target can be varied in this comparator by moving the
target. In the work thus far, two settings have been used, 60 and 80. 60 was very
simple to hit, as the scaling tended to place the global mean very near there and thus
many configurations had significant probabilities of going this distance. The 80 point,
4
One foot equals 7.326 scaled units
52
on the other hand, was very difficult to hit as only few configurations had significant
probabilities with that distance. This is somewhat artificial, and it is unlikely a
product developer would be required to find such an unlikely goal. This may unfairly
impede AdaptiveOneAtATime’s performance, as OrthogonalArray will be better at
precise results.
The parabolic players were rewarded some amount every time they threw for an
award. If the game had a payoff P = AR, where A = 3 is the number of attempts the
players get at a payoff, then each throw could have a maximum payoff R. The quality
loss function was constructed as a symmetric parabola so that its value was R at the
target D = 80, and 0 at the minimum 05 , so that the player would only very rarely
get a negative payoff. This differs somewhat from Frey, et al,frey:role) who preferred
to consider larger-is-better problems.
Once it had calculated a mean µ and population standard deviation σ from its
experiments on a configuration t, a player would then expect a payoff from any throw
of
< P >t = R
Z inf
− inf
(D − x)2 exp
µt − x 2
dx
σt2
(5.3)
and the quality Q of the configuration t is (by Wu & Hamada [9, equation 3.2]
Q = P [(D − µt )2 − σ 2 ]
(5.4)
This is the quantity the player uses to select the configuration to throw. While
OrthogonalArrayParabolic players attempt to predict this for every configuration
and take the best one, FullFactorialParabolic and AdaptiveOneAtATimeParabolic
players just use the best one that they have experienced. Guess players, of course,
just generate a random configuration, and provide a baseline for comparison.
5
the value went to 0 at the minimum as it was further from the target than the maximum
53
5.3.3
Manipulating the nonlinearity
The nonlinearity of a system was defined as by Frey[12], as the sum of the square
of the means for each configuration less the contributions due to global average and
main effects, which are the linear components, over the total variability. The sum
square of the factor effects SSF E is calculated for m configurations t each tested N
times for a mean ȳt by
SSF E =
m
NX
(y¯t − ȳ)2
m t=1
(5.5)
where ȳ is the global mean of the experimental strategy.
Each of the configurations has an associated setting for each parameter, and we
assume that the setting of every parameter can have some effect on the outcome of
the response. We extract these effects with a main effects model. For a rectangular
parameter space with s level settings in each of q parameters, the number of configurations t with a particular level set for parameter p will be m/s. The effect due to
that level setting is
M Ep(t)
X
s m/s
(y¯t − ȳ)
=
m t=1
(5.6)
We can then compose an expectation value for a configuration by summing the global
mean yt and the main effects M Ep(t) calculated for each parameter at that configuration t’s level setting s. Having accomplished this, we can define the sum square
variations due to main effects as
SSM E =
m
NX
(yM E (t) − ȳ)2
m t=1
where
yM E (t) = ȳ +
q
X
M Ep(t)
(5.7)
(5.8)
p=1
which allows us to calculate the sum squared variation from interactions
SSIN T = SS F E − SSM E
54
(5.9)
and thence the Interaction Strength I
I=
SSIN T
SSF E
(5.10)
For the Grewen data, this interaction strength was quite large
IGrewen = 0.527
(5.11)
meaning that more than half the variation was not explained by main effects.
To examine the relative performances of strategies at different levels on nonlinearity, we took the Grewen data and made its nonlinearity tunable, by defining a scaling
of the difference between the value in the main effects model and the statistical mean.
The main effects of each parameter setting were stored in a database table, and a
new mean generated for a desired Interaction Strength Id for each configuration by a
simple scaling
ȳt0 = yM E (t) +
Id
(ȳt − yM E (t))
I
(5.12)
This allows us to compare the same strategies at different levels of nonlinearity.
5.3.4
Manipulating the noise
The ’signal’ in the Grewen data is less than twice the noise. It is calculated in the
following way, again for m configurations t each tested N times for a mean ȳt and
global mean ȳ, as in section 5.3.3.
√
Pm
(ȳt −ȳ)2
qP
t=1
N
n=1
ST N =
(ytn −ȳt )2
m
ST NGrewen = 1.93
(5.13)
(5.14)
As does the nonlinearity, this extreme value would press any strategy hard for success.
While this hard problem may be of interest at times, to get variable solvability prob55
Figure 5-2: Comparator Functions
lems for the agents to work on, we scale the noise in a similar way to the nonlinearity.
Instead of the sigma calculated from the data, we generate normally distributed pseudorandom numbers using
σt0 =
ST N
σt
ST NGrewen
(5.15)
where STN is defined by the model.
Now that we can manipulate the interactions and noise of a particular data set, we
are ready to see how changes in those values affect the performance of the strategies.
5.4
The Central Command
The operation of the framework is diagrammed in figure 5-2. The basic activity
of enacting the DoE strategy and performing the experimental design is handled
in the Player.run() method, reproduced below. The player asks the strategy for a
configuration and number of repetitions, performs the repetitions, reports the results,
and asks if it should continue. As this is the central piece of code in the framework,
it may be useful to walk through it bit by bit. The first thing the agent does is
56
update the number of players currently in the body of the run method. This is to aid
higher-level thread management.
Listing 5.1: Player.run() method
public void run ( ) {
p l a y e r s r u n n i n g o n t h i s v m ++;
The player starts performing its experiments by retrieving its strategy and acknowledging its experimental cost. The experimental cost in these experiments was always
set to one, but flexibility was built into the model. The strategy here provides a list
of configurations to the player, and the player begins to work through them. The
hasNext() method returns true if there are more configurations the strategy wishes
the player to try.
S t r a t e g y how = t h i s . g e t S t r a t e g y ( ) ;
int c o s t = t h i s . getGame ( ) . g e t E x p e r i m e n t a l C o s t ( ) ;
try {
// g i v e up i f t h i s t h r o w s an ArrayIndexOutOfBoundsException
I t e r a t o r e x p e r i r a t o r = how . i t e r a t o r ( ) ;
i f ( null != e x p e r i r a t o r ) {
Game game = t h i s . getGame ( ) ;
while ( e x p e r i r a t o r . hasNext ( ) ) {
Request the configuration and number of repetitions for the experimental design.
// g e t t h e n e x t bunch o f parameter s e t t i n g s
HashMap s e t t i n g b y p a r a m = ( HashMap ) e x p e r i r a t o r . next ( ) ;
int t r i a l c o u n t = how . g e t T r i a l C o u n t ( ) ;
double [ ] r e s p o n s e s = new double [ t r i a l c o u n t ] ;
// r e p e a t f o r a c e r t a i n number o f t r i a l s
f or ( int t r i a l = 0 ; t r i a l < t r i a l c o u n t ; t r i a l ++) {
We are now inside the repetitions loop. The configuration and number of repetitions
have been decided. All that remains is to perform the experiment, store the results
and account the cost.
57
// throw t h e a i r p l a n e
double r e s p o n s e = game . e x p e r i m e n t ( t h i s . hashCode ( ) ,
settingbyparam ) ;
if ( storeflights ) {
// u p d a t e t h e d a t a s t o r e
try {
d a t a s t o r e . s t o r e d a t a ( t h i s . gameid , setti ngbyparam ,
response ) ;}
catch ( OutOfMemoryError o ) {
System . e r r . p r i n t l n ( "Player , line 106 ,
OutOfMemoryError " ) ; } }
10
responses [ t r i a l ] = response ;
t h i s . w e a l t h −= c o s t ; }
Update the strategy by passing back the configuration and all of the results it engendered. The strategy object will decide how to handle it – whether to get statistical
moments of the responses, take an extremum, ignore them altogether, or some other
approach.
// l e t t h e s t r a t e g y know how i t d i d
i f (0 < t r i a l c o u n t ) {
how . update ( settin gbyparam , r e s p o n s e s ) ; } } }
After all of the experiments in the experimental strategy have been performed with all
of their repetitions, the strategy object has enough data to decide which configuration
it is going to use to get the payoff.
At this point, it starts performing the experiments, getting rewarded, and adding
those rewards to its wealth.
// Throw t h e p l a n e s
Goal d e s i r e = t h i s . getGame ( ) . g e t G o al ( ) ;
Map b e s t s e t t i n g s = how . g e t B e s t S e t t i n g s ( t h i s . g e t D a t a S t o r e ( ) )
;
58
for ( int attempt = 0 ; attempt < d e s i r e . getAttemptCount ( ) ;
attempt++) {
double d i s t a n c e = game . e x p e r i m e n t ( t h i s . hashCode ( ) ,
bestsettings ) ;
double p a y o f f = d e s i r e . g e t P a y o f f ( d i s t a n c e ) ;
t h i s . w e a l t h += p a y o f f ;
datastore . recordTrial ( bestsettings , distance , payoff ) ;}
this . getDataStore ( ) . s t o r e w e a l t h ( this . wealth ) ; }
10
catch ( IndexOutOfBoundsException e ) {
e . printStackTrace () ;
System . e r r . p r i n t l n ( "run failed for player " +
S t r i n g . v a l u e O f ( t h i s . hashCode ( ) ) ) ; }
Finally, the player clears up some resources and announces its exit.
this . getDataStore ( ) . clearCache ( ) ;
S t r i n g n o t i c e = " finished Player .run for " +
S t r i n g . v a l u e O f ( t h i s . hashCode ( ) ) + " at " +
j a v a . t e x t . DateFormat . g e t T i m e I n s t a n c e ( j a v a . t e x t . DateFormat
.SHORT) .
format ( j a v a . u t i l . Calendar . g e t I n s t a n c e ( ) . getTime ( ) ) ;
p l a y e r s r u n n i n g o n t h i s v m −−; // decrement t h e semaphor }
5.5
The Strategies
Each of the strategies is an implementation of a generic Strategy interface reproduced
here.
Listing 5.2: The Strategy Interface
/∗
∗ Strategy . java
∗/
package edu . mit . c i p d . a i r p l a n e g a m e ;
59
import j a v a . u t i l . ∗ ;
/∗ ∗ ∗/
public i n t e r f a c e S t r a t e g y {
/∗ ∗ l e t t h e c l i e n t i t e r a t e t h r o u g h t h e d e s i g n s p a c e
∗ r e l e a s e a new parameter s e t t i n g e v e r y time
10
∗/
public I t e r a t o r i t e r a t o r ( ) ;
/∗ ∗ i n c r e a s e t h e a c c u r a c y o f t h e s t r a t e g y by u p d a t i n g i t −− t h i s
is
∗ required for adaptive s t ra te gi e s
∗ @arg s e t t i n g s b y p a r a m which s e t t i n g s were used
∗ @arg d i s t a n c e t h e r e s u l t o f t h o s e s e t t i n g s
∗/
public void update ( HashMap s e t t i n g s b y p a r a m , double [ ] d i s t a n c e s )
;
/∗ ∗ use t h e d a t a s t o r e t o e v a l u a t e t h e e f f e c t s o f t h e
parameters , and
20
∗ e s t i m a t e t h e b e s t s e t o f v a l u e s f o r t h e d e s i r e d outcome
∗ @arg d a t a s t o r e t h e a c c e s s o b j e c t f o r t h e s t o r e d d a t a −−
a d a p t i v e s t r a t e g i e s may not need i t
∗ @return t h e s e t t i n g s t o g i v e t h e b e s t r e s u l t hashed by t h e
parameters
∗/
public Map g e t B e s t S e t t i n g s ( DataStore d a t a s t o r e ) ;
/∗ ∗ A d a p t i v e s t r a t e g i e s need t o know t h e i r g o a l
∗ @arg g o a l t h e t a r g e t t o s h o o t f o r
∗/
public void s e t G o a l ( Goal g o a l ) ;
/∗ ∗
30
∗ A l l o w s t h e c a l l e r t o s u g g e s t how many t i m e s an e x p e r i m e n t
s h o u l d be r e p e a t e d
60
∗ @param T r i a l C o u n t −− how many t i m e s t o r e p e a t an e x p e r i m e n t
∗/
public void s e t P r e f e r r e d T r i a l C o u n t ( int T r i a l C ou n t ) ;
/∗ ∗
∗ Allow t h e s t r a t e g y t o d i c t a t e how many t i m e s t h i s p a r t i c u l a r
experiment i s repeated
∗ @return t h e number o f t i m e s t o r e p e a t t h e n e x t e x p e r i m e n t
∗/
public int g e t T r i a l C o u n t ( ) ;
/∗ ∗
40
∗ r e t u r n t h e parameter s p a c e t h e s t r a t e g y i s w o r k i n g with ,
which s h o u l d have
∗ been s e t i n t h e c o n s t r u c t o r
∗/
public ParameterSpace g e t S p a c e ( ) ;
}
Each of the examples below is from the ’parabolic’ payoff structures. Strategies
were each implemented three times. In the first version, they were merely meanseeking. In the second version, they used the error function to create an expectation
value of their reward. In these first two versions, the players got full reward for
landing in between two values, or goalposts. In the third version, the payoff structure
was changed to be parabolic. There was one optimum point, and the payoff decayed
parabolically from it.
We will return to a goalposts structure in Chapter 6.5. For the remainder of this
chapter, we will only consider the parabolic payoff agents.
We distinguish four types of DoE strategies:
1. impulsive strategies, where parameter design is abandoned and a random configuration tried,
2. complete strategies (the full factorial strategy), where every possible combination of parameter levels are tested,
61
3. statistical strategies, which gather information to satisfy a statistical model and
thereby predict a best setting, and
4. adaptive strategies or search routines, which move through a parameter space
seeking an optimal result.
For each of these types of strategies, we built a canonical example, respectively
Guess, FullFactorial, OrthogonalArray and AdaptiveOneAtATime, described below.
5.5.1
Validating the experimental designs
The function of the Strategy objects is to determine the configurations explored by
the Player objects. So, the first step in validating the model is to ensure that the
strategies are providing the correct configuration lists to the Players.
Each time a player performs an experiment, it records the configuration and result
in a database, as shown above in line 7 of listing 5.4. This data helps the strategy
make a decision about which configuration the player should use in the payoff round.
So, it is important that the strategy deliver exactly the configurations it purports to.
Appendix C shows the results of the following query with @Strat set to the name
of each strategy.
Listing 5.3: Listing the Configurations
Select Sample . PlayerID , A & B & C & D AS C o n f i g u r a t i o n , avg (
D i s t a n c e ) As Mu, count ( d i s t a n c e ) As Frequency , s t d e v p ( d i s t a n c e
) As Sigma FROM (FLIGHT INNER JOIN (SELECT TOP 10 mid ( P l a y e r .
S t r a t e g y , l e n ( ’edu.mit.cipd. airplanegame .’ ) + 1 ) AS S t r a t e g y ,
PlayerID , T ri a l Co u n t FROM PLAYER Where P l a y e r . S t r a t e g y = ’edu
.mit.cipd. airplanegame .’ + @Strat AND T ri a l C o u n t =7) as Sample
ON FLIGHT . PlayerID = Sample . PlayerID ) GROUP BY Sample . PlayerID
, A, B, C, D, Sample . S t r a t e g y , Sample . T r i al C o u n t ORDER BY
Sample . PlayerID , MIN( F l i g h t .WHen)
Each player is listed in the Player table with its playerid and strategy, which is
prepended by the JavaTM package qualifier ’edu.mit.cipd.airplanes.’.
62
5.5.2
Guess
The simplest strategy is Guess. Its hasNext() method6 always returns false, its
getBestConfiguration() method returns a randomly selected configuration, and
its update() method is not implemented. It is this last point that explains why
there is no GuessParabolic class. The parabolic strategies are coded to employ
their strategies against a parabolic reward. As guess is not strategic, it needed no
extension.
The flight data query for Guess comes up empty, as it should. Guess does no
flights before entering the reward phase. We expect that guess will do equally well as
far as payoff in all sorts of situations. Guess provides an implicit null hypothesis for
considering the effectiveness of the other strategies.
5.5.3
Full Factorial
Next clearest in the set of canonical strategies is full factorial.
FullFactorial-
Parabolic visits every possible configuration. It’s iterator’s next() function retrieves
the next array of settings created by a function getAllSettingsCombinations().
public s t a t i c I n t e g e r [ ] [ ] g e t A l l S e t t i n g C o m b i n a t i o n s ( Parameter
[ ] parameters , Map s e t t i n g s b y p a r a m ) {
Iterator seterator
= settingsbyparam . values () . i t e r a t o r () ;
int s e t t i n g c o u n t = 1 ;
while ( s e t e r a t o r . hasNext ( ) ) {
s e t t i n g c o u n t ∗= ( ( C o l l e c t i o n ) s e t e r a t o r . next ( ) ) . s i z e ( ) ; }
// a l l o c a t e an a r r a y t o h o l d t h e s e t t i n g s
Integer [ ] [ ]
r e t v a l = new I n t e g e r [ s e t t i n g c o u n t ] [ p a r a m e t e r s .
length ] ;
// i t e r a t e a c r o s s t h e p a r a m e t e r s
10
int p r e v c o u n t = 1 ;
fo r ( int i n d e x =0; i n d e x < p a r a m e t e r s . l e n g t h ; i n d e x++){
6
strictly, the hasNext() method of its associated iterator object
63
L i s t s e t t i n g s = ( L i s t ) settingsbyparam . get ( parameters [
index ] ) ;
int mycount = s e t t i n g s . s i z e ( ) ;
for ( int prev = 0 ; prev < p r e v c o u n t ; prev++){
fo r ( int s e t t i n g = 0 ; s e t t i n g < mycount ; s e t t i n g ++){
fo r ( int r e p e a t = 0 ; r e p e a t < s e t t i n g c o u n t / p r e v c o u n t /
mycount ; r e p e a t++){
int pos = r e p e a t + prev ∗ s e t t i n g c o u n t / p r e v c o u n t +
s e t t i n g ∗ s e t t i n g c o u n t / p r e v c o u n t /mycount ;
r e t v a l [ pos ] [ i n d e x ] = ( I n t e g e r ) s e t t i n g s . g e t ( s e t t i n g
) ;}}}
p r e v c o u n t ∗= mycount ; }
20
return r e t v a l ; }
The update() function allows the strategy to calculate the Quality of each configuration, and getBestConfiguration() simply returns the highest-valued configuration.
We expect full factorial to always find a reasonably good answer, assuming one exists,
and to be as invariant as guess to noise and nonlinearity.
5.5.4
Adaptive One At A Time
The AdaptiveOneAtATimeParabolic strategy goes through the parameters and
tries each of the settings in turn, leaving the visited parameters at their best settings.
The hasNext() method returns true until each parameter has had each setting visited. The update method calculates the quality loss function of the configuration
given the test results. If the quality of the current configuration is highest, it preserves
it and discards the previous best.
Listing 5.4: AOAT Parabolic update()
// use t h e q u a d r a t i c l o s s f u n c t i o n and t h e s t a t i s t i c a l moments
to get the best expected
64
public void update ( HashMap s e t t i n g s b y p a r a m , double [ ] d i s t a n c e s )
{
float l o s s = 0;
double [ ] moments = A n a l y s i s . s t a t i s t i c a l m o m e n t s ( d i s t a n c e s ) ;
//The f i r s t moment i s t h e mean . The second i s t h e s t a n d a r d
deviation
double mu = moments [ 0 ] ;
double sigma = moments [ 1 ] ;
/∗ c a l c u l a t e t h e q u a l i t y l o s s f u n c t i o n −− t h e c o e f f i c i e n t , t h e
b r e a d t h o f t h e p a y o f f curve , doesn ’ t m a t t e r . we j u s t need
t h e sigma−s q u a r e d + (mu − t a r g e t )−s q u a r e d ∗/
10
l o s s = ( f l o a t ) ( Math . pow ( sigma , 2 ) + Math . pow ( (mu − t h i s . od )
, 2) ) ;
// r e p l a c e t h e b e s t answer
// n o t e t h a t b e s t answer h e r e i s t h e s m a l l e s t l o s s f u n c t i o n ,
not t h e b e s t d i s t a n c e
i f ( l o s s < this . bestanswer ) {
Parameter param = orderedparams [ l a s t p a r a m ] ; // g e t t h e
c u r r e n t l y v a r y i n g parameter
b e s t s e t t i n g s . put ( param , s e t t i n g s b y p a r a m . g e t ( param ) ) ; // s a v e
its setting
this . bestanswer = l o s s ;
Subsequent calls to next() will use this new best setting to set the parameters that is
not currently being varied. The call to getBestConfiguration() will simply return
the bestsettings variable value.
It is hard to develop an intuition about what to expect from adaptive one at
a time, but we expect it to behave like a continuously improving search algorithm.
It should be reasonably good in a situation we know nothing about, but somewhat
subject to being trapped.
65
5.5.5
Orthogonal Array
This is the statistical strategy, and as such is the least obvious in its operation. It
is tied to a particular statistical model. As it supplies a symmetric second order
orthogonal array as an experimental design, this strategy demands a main effects
model.
The OrthogonalArrayParabolic object does not calculate its own Orthogonal
Array. It uses a particular orthogonal array taken from a lecture by Dan Frey, generalizing it by swapping the meanings of the parameters and levels in a randomized
fashion.
/∗
10
A1
B1
C1
D1
A1
B2
C2
D2
A1
B3
C3
D3
A2
B1
C2
D3
A2
B2
C3
D1
A2
B3
C1
D2
A3
B1
C3
D2
A3
B2
C1
D3
A3
B3
C2
D1
∗/
This strategy does some number of experiments at each visited configuration, then
uses that information to estimate the main effects of parameter levels on flight distance
and standard deviation. With that information, it then constructs a predicted quality
as per equation 5.4 for all 81 configurations, then submits its best choice to the Game
for the reward.
We expect orthogonal array to not do terribly well against the Grewen data,
as it depends on the hierarchical ordering assumption. Orthogonal array is fairly
insensitive to noise, which is important here. Main effects are still fairly important,
so we expect it to beat adaptive one at a time, but not to approach full factorial in
66
payoff.
5.6
Payoff
After the agents had precessed through their experimental designs, they entered performance trials, were they would receive some payoff if they landed between two
values (in the goalposts mode) or according to their distance from an optimum (in
the parabolic mode.)
In all cases, the agents had three tries for a payoff. Only the parabolic agents are
discussed here.
5.7
Evaluating the experimental designs
So, now the strategies can be compared with their success levels at different points,
and for different noise and nonlinearity levels. The SQL Query
Listing 5.5: SQL Query to retrieve comparative value
SELECT Game . Payoff , Count ( ∗ ) AS Frequency , TrialCount , Noise ,
N o n l i n e a r i t y , mid ( P l a y e r . S t r a t e g y , l e n ( ’edu.mit.cipd.
airplanegame .’ ) + 1 ) AS S t r a t e g y , AVG( Wealth ) AS [ $ \mu { Wealth
}$ ] , STDEVP( Wealth ) AS [ $ \ s i g m a { Wealth } $ ] , [ $\mu { Wealth } $ ] / [
$\ s i g m a { Wealth }$ ] AS [ $STN { Wealth }$ ] , Game . AttemptCount ∗AVG(
P a y o f f s . V i r t u e ) /Game . P a y o f f AS [ $ \mu { P a y o f f } $ ] , Game .
AttemptCount ∗STDEVP( P a y o f f s . V i r t u e ) /Game . P a y o f f AS
[ $\ sigma {
P a y o f f } $ ] , [ $\mu { P a y o f f }$ ] / [ $\ s i g m a { P a y o f f } $ ] AS [ $STN {
P a y o f f } $ ] FROM (GAME INNER JOIN [SELECT GameID , PlayerID , SUM(
P a y o f f ) AS V i r t u e FROM Reward GROUP BY GameID , PLAYERID ] . AS
P a y o f f s ON GAME.GAMEID = P a y o f f s .GAMEID) INNER JOIN PLAYER ON
PLAYER.PLAYERID =P a y o f f s . PLAYERID WHERE GOAL=’edu.mit.cipd.
airplanegame . PARABOLICDISTANCEGOAL ’ AND AttemptCOunt=3 AND
TARGET=80 GROUP BY Game . AttemptCount , Game . Goal , Game . Target ,
67
Game . T o l e r a n c e , S t r a t e g y , Game . Payoff , TrialCount , Noise ,
Nonlinearity ;
gives us a table of results that allow us to compare the value of the DoE strategies
at different settings, as shown in table C.1. The standard deviation and mean for the
payoff is pulled from the preceding table to be shown in the following graph.
5.8
Results
The query 5.5 creates the OLAP table C.1. This data forms the data source for both
Figure 5-3, a stacked histogram for the mean value of payoff – the number of hits out
of three the agent got – and Figure 5-4 for the mean value of wealth, which is the
number of payoffs times the payoff amount (which is constant for every player in a
particular game) less the number of experiments (which is the same for every player
using a particular strategy and trial count).
The charts therefore can be generated one from the other according to the relationship7
W =
A
RX
p(a) − Cs
A a=1
(5.16)
Where W is the wealth, R is the reward for the goal, A is the number of attempts
a player is given, a is the attempt index, p(a) is the payoff portion (up to 1) for a
particular attempt and Cs is the experimental cost for a strategy s.
On these following plots ’noise’ is the signal to noise ratio, and nonlinearity is
the data scaled as explained in section 5.3.3. Figures 5-4 and 5-3 show success data
for agents with each of the four canonical strategies and 2, 4 or 7 repeats on each
experiment, operating in games with payoffs of 63, 567 or 3000, signal-to-noise ratios
of 100, 10 or 1.94 (the level of the Grewen data,) and interactions of 0, 0.1, 0.527 (the
level of the Grewen data) and 0.8.
The model run to generate the high interaction data was incomplete, creating
more unevenness in the height of the stacked columns than exists in the data.
7
see also equation 5-3.
68
45
Sum of $\mu_{Payoff}$
40
Nonlinearity
Noise
1.79769313486232E+308 - 0
0.8 - 100
0.8 - 10
0.8 - 1.93717750485
0.52705102742087 - 100
0.52705102742087 - 10
0.52705102742087 - 1.93717750485
0.1 - 100
0.1 - 10
0.1 - 1.93717750485
0 - 100
0 - 10
0 - 1.93717750485
35
30
25
20
15
10
5
0
2 4 7 2 4 7 4 2 4 7 2 4 7 2 4 7 2 4 7 2 4 7 2 4 7 4 2 4 7 2 4 7 2 4 7 4 2 4 7
63
567 972 3000
Guess
63
567
3000
63
AdaptiveOneAtATime
567 972 3000
63
FullFactorial
567 972 3000
OrthogonalArray
Strategy Payoff TrialCount
Figure 5-3: Stacked Histogram of Mean Payoff
35000
Sum of $\mu_{Wealth}$
30000
Nonlinearity
Noise
1.79769313486232E+308 - 0
0.8 - 100
0.8 - 10
0.8 - 1.93717750485
0.52705102742087 - 100
0.52705102742087 - 10
0.52705102742087 - 1.93717750485
0.1 - 100
0.1 - 10
0.1 - 1.93717750485
0 - 100
0 - 10
0 - 1.93717750485
25000
20000
15000
10000
5000
0
2 4 7 2 4 7 4 2 4 7 2 4 7 2 4 7 2 4 7 2 4 7 2 4 7 4 2 4 7 2 4 7 2 4 7 4 2 4 7
-5000
63
567 972 3000
Guess
63
567
3000
AdaptiveOneAtATime
63
567 972 3000
63
FullFactorial
567 972 3000
OrthogonalArray
-10000
Strategy Payoff TrialCount
Figure 5-4: Stacked Histogram of Mean Wealth
69
Noise 1.93717750485 Payoff 567 TrialCount 4
Nonlinearity vs. Strategy Performance
3
Sum of $\mu_{Payoff}$
2.8
2.6
Payoff
Strategy
AdaptiveOneAtATimeParabolic
FullFactorialParabolic
Guess
OrthogonalArrayParabolic
2.4
2.2
2
1.8
0
0.1
0.52705102742087
0.8
Interaction Strength
Nonlinearity
Figure 5-5: Strategy values at different nonlinearity values
The stacked histogram is difficult to interpret, in part due to the missing 0.8
nonlinearity data for many of the 3000 payoff runs. We can use this data to look at
strategy vs. signal-to-noise ratio or strategy vs. nonlinearity for any payoff value or
number of repetitions. Figure 5-5 pulls out a little of this data. We see from this that
adaptive one a time does outperform orthogonal array at the enhanced linearity level,
597 payout, four-trail case. The source table C.1 tells us that the standard deviations
in payoff for AdaptiveOneAtATime and OrthogonalArray in this regime are 0.400 and
0.431 respectively, whereas the difference is only 0.1. The result does assure us that
AdaptiveOneAtATime is not uniformly worse. Resetting the nonlinearity to the level
of the Grewen data and varying the signal-to-noise ratio, we see that, as we expect,
guess is insensitive to the noise level and full factorial essentially becomes perfect at
reasonably low noise. From left to right, the payoff values for adaptive one at a time
are (2.512, 2.646, 2.681) and for orthogonal array (2.640, 2.753, 2.771). Orthogonal
array is somewhat less senstive to noise, improving its performance by only half as
much as adaptive one at a time moving from an STN of 10 to an STN of 100.
70
Nonlinearity 0.52705102742087 Payoff 567 TrialCount 4
Signal To Noise Ratio vs. Strategy Performance
3
Sum of $\mu_{Payoff}$
2.8
2.6
Payoff
Strategy
AdaptiveOneAtATimeParabolic
FullFactorialParabolic
Guess
OrthogonalArrayParabolic
2.4
2.2
2
1.8
1.93717750485
10
100
SIgnal To Noise Ratio
Noise
Figure 5-6: Strategy values at different signal-to-noise ratios
We see that, as we expect, the Orthogonal Array does well in noisy, linear environments. Full Factorial wins when the experimental cost becomes small compared with
the payoff. Adaptive One At A Time loses to Orthogonal Array until the interactions
become very strong.
5.9
Analysis
Guess does well on wealth, but is beaten by the Full Factorial in the high payoff
regime and by the Orthogonal Array and Adaptive One At A Time strategies in the
lower payoff regimes. Full Factorial always wins on payoff, but loses in the areas of
lower payoff.
Adaptive One At A Time, here, never convincingly beats Orthogonal Array, only
tying it in the enhanced-interaction Grewen data with the noise extracted, where we
would expect it to do better. We believe this is due to the unrealistic constraint
imposed on AOAT by the ’nominal is best’ target.
We have shown that strategies are codeable, and perform relative to one another
71
in line with out expectations. We can therefore have some confidence in moving
forward and applying this evaluative framework to other tasks.
72
Chapter 6
Abstraction
With a framework for modeling an arbitrary design of experiments (DoE) strategy
in place, it becomes possible to talk about what makes a DoE strategy. There is an
essential value to being able to categorize a strategy along particular and familiar
parameters.
Formally, by an experimental design we mean a ordered list of configurations to
test, and by an experimental strategy we mean an experimental design with repetitions
assigned to each configuration. We use ’DoE strategy’ here to mean the means by
which the experimental strategy is constructed.
6.1
Motivation
In section 6.5, we discuss optimization of a DoE approach. It would be nice to be able
to say of a particular response surface that you were using the best DoE. Abstraction
is a necessary precursor.
Tools like minitab, as discussed in section 4.3 will build the best experimental
design for a known statistical model. When a response surface comports well with
a particular statistical model, it would be hard to argue that a better experimental
design existed. However, the presumption that a statistical model is known and
available is an extreme one for DoE in general.
It is therefore incumbent upon us to decide how to extract the essential nature
73
from a DoE strategy and parameterize it. It is in this way that we will be able to
create novel or particularly appropriate DoE strategies.
6.2
Method
As discussed in section 5.5, design of experiments strategies fall in four categories:
arbitrary, complete, statistical and adaptive. Part of the challenge of abstracting and
generalizing DoE is to create a measuring scheme under which any of the forms could
be more or less expressed.
Abstraction of DoE strategies is not straightforward, and there are several possible
routes one could take. We could model learning or bias, as agents take successful
configurations from a previous game or a preprogrammed list of configurations to try.
We could grant each agent a statistical model and let those compete. We could allow
agents to pick configurations at random for ’experimental designs’ of different lengths
and just compare the effectiveness of a particular design length.
We decided to establish certain tendencies in the agents that distinguish strategies that we had tested from one another. The four canonical strategies discussed
in section 5.5 – Guess, Orthogonal Array, Adaptive One At A Time and Full Factorial – differ in the number of configurations that they test, the way they choose
configurations to test and whether or not they’ve decided these questions before they
start.
We came up with these five tendencies
• How far ahead do the players look? It takes a certain amount of discipline to
not just assume that the best answer you’ve gotten so far will be quite close to
the best answer possible. We call this the planning.
• How much do the players want to spend in total? The plans are the atomic unit
of experimental design, so the profligacy is the number of plans through which
the player goes.
• How seriously does this player regard variability in response? How many times
74
will it repeat an experiment? We call this the care.
• The order 2 orthogonal array carefully balances the pairs of parameter settings.
A1 B2 will show up as often as A2 B1 The concept of fairness captures this. Unlike
the other concepts, this one does not vary between two values, but is just a set
of decisions as to whether to balance groups of various sizes. A strategy might
want to balance singletons, seeing A3 as often as D2 , pairs, triplets or quartets.
• Finally, a tendency it made sense to include was that of Diligence. Diligence
would be the earnestness with which a player approached its task, and its tendency to quit early when it felt it had a good answer.
The investigation of diligence is left to further work, as the results with the first
four alleles were not robust enough to include diligence. As diligence was intended to
capture the practice of varying the number of trials between configurations, a Bayesian
approach to noise inspired by the Bayesian approach to interaction in Chipman, et
al.[30], it was deemed of a greater level of subtlety than the other inclinations, and
its implementation was postponed until the rest of the model had settled.
In short, all configurations tried by an agent were tried an equal number of times.
As many of our investigations were done with ’goal posts,’ and not quality loss functions, the lazy players in this paradigm would gain an unfair and unrealistic advantage.
6.3
Available optimization methods
What we proposed was a set of decision criteria for creating a DoE strategy. To create
an objective function, the DoE strategy itself would have to be created and run against
the data and payoff criteria. Therefore, we have a highly variable response, and there
is no gradient available in the objective function. The first point argues for some sort
of ensemble calculation, and the latter against numerical search or other traditional
optimization schemes.
Our options for optimization schemes are therefore somewhat limited. Genetic algorithms, however, require only that our parameters be codeable, the payoff function
75
be defined and our problem amenable to large populations of solvers. Our environment is so provided, as described in section 6.4
6.4
Genetic Algorithm
Once we had decided to proceed with genetic algorithms, and therefore to code our
five tendencies as genes, we had further design decisions to make.
On top of the question of how a coding for care of ’1010,’ for example, would be
interpreted, there are questions of initial population and number of generations to
run, as well as design decisions – selection, crossover, and mutation – that separate
genetic algorithm models one from another as described in section 3.3.
A generic agent was to create a strategy by
1. Deciding on the length of a plan. A plan is a portion of an experimental design,
which is to say, a list of configurations to try,
2. Creating the plan by populating the list with configurations selected according
to some scheme,
3. Enacting the plan by visiting each configuration and testing it some determined
number of times,
4. Reviewing the information obtained through the previous tests and iterating,
creating a new plan, and
5. Continuing to iterate until some number of plans had been completed and
merged into the experimental design.
In a nod to the confirmation trial required in statistical models, the last configuration was determined by a simple main effects model, as explained in section
2.3.
76
6.4.1
The Codings
A four-bit gene, either 0 to 16 or 4 yes/no decisions, was decided to give enough
granularity so that effects could be considered, but not inappropriately complex.
Each of the tendencies interprets the four-bit gene and lets it effect the phenotype,
that is, the DoE strategy, in its own way.
profligacy allele determined how many plans the agent goes through. The Adaptive One At A Time strategy goes through 4 plans (with one additional configuration,) and the Orthogonal Array goes through only one. The number of
plans to test is allowed to vary from 0 to 15. The Guess strategy would have a
profligacy of 0, and a zero profligacy renders the other alleles irrelevant.
planning allele decided how many configurations would be in a particular plan. In
the four-parameter, three-level parameter space of the paper airplane game,
Adaptive One At A Time, for instance, would have two configurations in its
plan, as it knows only the next two configurations it will test when it starts an
allele. Orthogonal Array would have nine, as it knows which nine configurations
it will test when it begins. The expression of the planning allele varied from 1
to 16.
care allele told the agent how many times it should experiment with a given configuration. An agent with a care genotype of 0011, for instance, would throw
0 · 23 + 0 · 22 + 1 · 21 + 1 · 20 + 11 = 2 + 1 + 1 = 4 airplanes of each design it chose
to test.
fairness allele determined how those configurations were chosen. It created most of
the complexity of this model. It is the seeking for balance of parameter settings
in the experimental design. As a plan is constructed, an agent takes some
care to balance the appearance of settings in a particular number of columns
according to each yes or no gene. If the first bit was ’flipped,’ that is, true,
1
The choice is to add one to the binary equivalent or to make 0000 equal ’16.’ We chose the
former in all cases.
77
Table 6.1: Canonical strategies as phenotypes
Strategy
Guess
Orthogonal Array
Adaptive One at a Time
Full Factorial
Profligacy
0000
0001
0101
1001
Planning
any
1001
0010
1001
Care
any
gt 0000
gt 0000
any
Fairness
any
1100
0001
1111
then the groupings of size one were balanced. This means that for every A1,
the player was inclined to include as many A2s and B1s. If the second bit was
flipped, then the groupings of size two were balanced – A1B3 would tend to
appear no more often than B2D1. Adaptive One At A Time balances four of a
kind: only its last bit would be flipped.
diligence allele was never implemented, and was used for sizing the populations
and correcting the generation rule. If the diligence converged more quickly
than another allele or converged to a particular value, this pointed out some
problem in the model itself.
6.4.2
The Fitness Function
At the end of its experimental strategy, a player would then try to get a payoff by
running tests, as explained in section 5.6. The fitness function was simply the payoff
less the experimental cost; alternately, we could have considered the payoff as the
fitness function, and defined a penalty function to be the experimental cost; further
work may find that distinction useful. It may be of interest to examine the signalto-noise ratio instead, as per the Taguchi method, but this would involve running
multiple players with the same strategy. In the interest of simplicity, we ran only
single players with single strategies.
f = N (P )/T − Cost = N (P )/T − P r(P l)Ca
(6.1)
Where N is the number of successful trials, P is the total possible payoff, T is the
78
number of trials, and Pr, Pl and Ca are the profligacy, planning and care respectively.
In all the trials we ran, the player had three chances to make the payoff, and the
payoff was delivered in thirds. Therefore the T in the payoff equation was always ’3’,
and N varied from ’0’ to ’3.’
6.4.3
Selection
The selection criteria are simple – we simply allow the strategies working for more
successful players a greater chance of reproduction. This is the stochastic remainder
selection mechanism described in section 3.4.2.We can look at some sample code from
GAAirplaneWorld.java.
/∗ ∗
∗ // u p d a t e d 1/26/04 t o use t h e s t o c h a s t i c remainder s e l e c t i o n
without
∗ r e p l a c e m e n t mechanism i n G o l d b e r g ’ s
Genetic
Algorithms ,
f i g u r e 4.24
∗
∗ Gives t h e p l a y e r s new s t r a t e g i e s , and z e r o e s t h e i r w e a l t h
∗ @param p l a y e r s −− t h e l i s t o f p l a y e r s ( s h o u l d be a p e r f e c t
s q u a r e i n number )
∗ @return −− t h e same l i s t o f p l a y e r s w i t h z e r o w e a l t h and new
strategies
∗ @todo −− r e v i s i t t h e w e a l t h c a l c u l a t i o n . Maybe a v e r a g e
mother and f a t h e r ?
∗/
10
protected void g e n e r a t e ( P l a y e r [ ] p l a y e r s ) throws
PlayerCreationException ,
j a v a . s q l . SQLException , IOException {
// do a l i t t l e sigma t r u n c a t i o n and l i n e a r s c a l i n g
// g e t t h e l i s t o f w e a l t h s
int p o p s i z e = p l a y e r s . l e n g t h ;
double [ ] w e a l t h s = new double [ p o p s i z e ] ;
79
f o r ( int pdex = 0 ; pdex < p o p s i z e ; pdex++) {
w e a l t h s [ pdex ] = ( double ) p l a y e r s [ pdex ] . w e a l t h ; }
// g e t t h e mean and l o w e r c u t o f f f o r v a l i d i t y
double [ ] s t a t s = A n a l y s i s . s t a t i s t i c a l m o m e n t s ( w e a l t h s ) ;
20
double mean = s t a t s [ 0 ] ;
double d e v i a n t = mean − 2 ∗ s t a t s [ 1 ] ;
// i f t h e r e i s no d e v i a t i o n ( c o u l d happen ) a r b i t r a r i l y move
t h e c u t o f f down
i f ( 0 == s t a t s [ 1 ] ) {
d e v i a n t = mean − 1 ; }
double max = Double . MIN VALUE ;
double min = Double .MAX VALUE;
// change t h e w e a l t h t o z e r o i f i t ’ s more than two s t a n d a r d
d e v i a t i o n s b e l o w t h e mean
f or ( int pdex = 0 ; pdex < p o p s i z e ; pdex++) {
w e a l t h s [ pdex ] = w e a l t h s [ pdex ] > d e v i a n t ? w e a l t h s [ pdex ] −
deviant : 0;
30
// c a p t u r e t h e g r e a t e s t s c a l e d w e a l t h
i f ( w e a l t h s [ pdex ] > max) {
max = w e a l t h s [ pdex ] ; }
i f ( w e a l t h s [ pdex ] < min ) {
min = w e a l t h s [ pdex ] ; } }
Now that we have zeroed the underacheivers, we calculate the new mean. Keeping
that mean, we rescale the values of the objective function so that the greatest value
is twice the new mean. We’ll limit that scaling a little to avoid negative numbers.
Dividing by the mean gives us a scaled fitness function.
// r e c a l c u l a t e t h e mean
mean = A n a l y s i s . s t a t i s t i c a l m o m e n t s ( w e a l t h s ) [ 0 ] ;
i f ( 0 == mean ) {
mean = 1 ;
80
// so , I want max t o be s c a l e d t o t w i c e t h e mean w i t h o u t
c h a n g i n g s a i d mean}
double s l o p e = 1 . / (max / mean − 1 ) ;
i f ( Double . i s I n f i n i t e ( s l o p e ) ) {
s l o p e = 0 ; // a d j u s t f o r t h e non−v a r i a n t c a s e }
10
double i n t e r c e p t = ( 1 . − s l o p e ) ∗ mean ;
// u n l e s s t h i s would p u t t h e l e a s t one l e s s than ze r o , i n
which c a s e s c a l e i t t o z e r o
i f ( 0 > s l o p e ∗ min + i n t e r c e p t ) {
s l o p e = 1 . / ( 1 . − min / mean ) ;
i n t e r c e p t = ( 1 . − s l o p e ) ∗ mean ; }
f or ( int pdex = 0 ; pdex < p o p s i z e ; pdex++) {
w e a l t h s [ pdex ] = w e a l t h s [ pdex ] ∗ s l o p e + i n t e r c e p t ; }
At this point we have a reasonably well-scaled set of values for the objective
function (the wealth, which is the sum payoff less the experimental cost,) with a
dispersion approaching the magnitude of the largest value. This was necessary to more
easily differentiate between fine performance differences[22]. Note that it preserves
players with a negative objective function value, and may let the best of a bad lot
dominate, preventing the growth of a more effective strain.
Now, we preselect the players that have a scaled fitness function one or greater by
moving them into a selection list. We attempted to scale the greatest value, recall, to
twice the mean. If that was sucessful, it gets preselected twice here.
//OK, so , I ’m a l l s c a l e d . P r e s e l e c t t h e p l a y e r s .
boolean [ ] [ ] [ ]
p r e s e l e c t e d = new boolean [ p o p s i z e ] [ 5 ] [
GAStrategy .
ALLELE LENGTH ] ;
double [ ] f r a c t i o n = new double [ p o p s i z e ] ; // t h i s i s t h e
residual part
int ndex = 0 ;
81
f o r ( int pdex = 0 ; pdex < p o p s i z e ; pdex++) {
f r a c t i o n [ pdex ] = w e a l t h s [ pdex ] / mean ;
/∗ f o r however many t i m e s t h e mean t h e y a r e ( which s h o u l d be
bet we e n 0 and
10
2) , add one t o t h e p r e s e l e c t i o n .
The remainder i s t h e chance t h e y g e t a n o t h e r e n t r y i n t h e
preselection .
∗/
while ( f r a c t i o n [ pdex ] >= 1 ) {
p r e s e l e c t e d [ ndex++] = ( ( GAStrategy ) p l a y e r s [ pdex ] .
getStrategy () ) .
getAlleles () ;
f r a c t i o n [ pdex ]−−;} }
Every player2 qualified for preselection subtracted one from its scaled fitness. This
left a fraction for any non-integer multiples of the mean. We now interpret the fraction
as the probability that the player will be moved into the selection list.
Pi = Wi /µ0 − s
(6.2)
Where Pi is the probability the genotype will be selected, Wi is the player’s wealth, µ0
is the mean calculated after the removal of the negative values, and s is the number
of times the genotype was already transferred to the select list.
For example, if a player’s wealth was 372 and the recalculated mean after the
negative values were removed was 248, the odds that it would end up on the selection
list a second time after being preselected once (for being greater than the mean) is
P372 = 372/248 − 1 = 0.5 = 50/
2
or, more accurately, the set of alleles defining the player’s strategy
82
(6.3)
If another player had come out of the same game with a wealth of 124, its probability
of ending up on the selection list for the first time would be
P124 = 124/248 = 0.5 = 50/
(6.4)
try { // s t o p t h i s when t h e a l g o r i t h m t r i e s t o w r i t e p a s t t h e
end o f t h e a r r a y
//Now, we a s s i g n more a l l e l e s t o t h e p r e s e l e c t e d a r r a y
a c c o r d i n g t o chance
while ( ndex < p o p s i z e ) {
// i t e r a t e t h r o u g h t h e p l a y e r s u n t i l we have enough
alleles
fo r ( int pdex = 0 ; pdex < p o p s i z e ; pdex++) {
i f ( 0 < f r a c t i o n [ pdex ] ) {
i f ( rand . nextDouble ( ) <= f r a c t i o n [ pdex ] ) {
f r a c t i o n [ pdex ]−−; // t h a t ’ s a l l f o r you !
10
p r e s e l e c t e d [ ndex++] = ( ( GAStrategy ) p l a y e r s [ pdex ] .
getStrategy () ) .
getAlleles () ;} } } } }
catch ( ArrayIndexOutOfBoundsException e ) {}
An element is chosen randomly from the select list, and the model
• makes a new strategy object from it (line 17)
• adds it, once to the fathers list and once to the mothers list (lines 18 and 19)
• replaces the genotype with the last accessible item in the list (line 21)
• decreases its range of selectable items by one (line 22.)
// s e l e c t them two−by−two
ParameterSpace s p a c e = p l a y e r s [ 0 ] . ge tP ar amet e rS pace ( ) ;
83
/∗ l e t me t a k e t h e t o p h a l f as p a r e n t s . So , I can l e t each
winner be a mother
and a f a t h e r
∗/
GAStrategy [ ] f a t h e r s = new GAStrategy [ p o p s i z e / 2 ] ;
GAStrategy [ ] mothers = new GAStrategy [ p o p s i z e / 2 ] ;
10
int end = p o p s i z e / 2 ;
f or ( int pdex = 0 ; pdex < p o p s i z e / 2 ; pdex++) {
int c h o i c e = rand . n e x t I n t ( end ) ;
GAStrategy how = new GAStrategy ( space , p r e s e l e c t e d [ c h o i c e
][0] ,
preselected [ choice ] [ 1 ] ,
preselected [ choice ] [ 2 ] ,
preselected [ choice ] [ 3 ] ,
preselected [ choice ] [ 4 ] ) ;
mothers [ pdex ] = how ;
f a t h e r s [ pdex ] = how ;
20
// copy t h e l a s t ( now i n a c c e s s i b l e ) c h o i c e o v e r t h e j u s t
s e l e c t e d one
p r e s e l e c t e d [ c h o i c e ] = p r e s e l e c t e d [ end ] ;
end −−;}
When I have chosen half the population to reproduce, I let them do so by selecting a
mating partner for each strategy object randomly and letting them reproduce, first
with one as the ’left parent,’ then with the other.
The mechanics of mating are discussed more fully below in section 6.4.4, ’Crossover.’
j a v a . s q l . Connection con = t h i s . con ; // g e t t h e c o n n e c t i o n f o r
p l a y e r −making
84
//OK, so , now I have t h e s e p a r e n t a l s t r a t e g i e s . Let ’ s p a i r
them o f f
// g e n e r a t e new s t r a t e g i e s and p l a c e them i n p l a y e r s
f o r ( int pdex = 0 ; pdex < p o p s i z e / 2 ; pdex++) {
// l e t each winner be a f a t h e r
GAStrategy f a t h e r = f a t h e r s [ pdex ] ;
int c h o i c e = rand . n e x t I n t ( p o p s i z e / 2 − pdex ) ;
10
GAStrategy mother = mothers [ c h o i c e ] ;
GAStrategy son = new GAStrategy ( space , f a t h e r , mother ) ;
p l a y e r s [ pdex ] = new P l a y e r (new C o n c r e t e D a t a S t o r e ( con ,
t h i s . getNextPlayerCode ( son . g e t C l a s s ( ) . getName ( ) ,
I n t e g e r . MIN VALUE) ) ) ;
p l a y e r s [ pdex ] . s e t S t r a t e g y ( son ) ;
GAStrategy d a u g h t e r = new GAStrategy ( space , mother , f a t h e r )
;
p l a y e r s [ p o p s i z e − pdex −
20
1 ] = new P l a y e r (new C o n c r e t e D a t a S t o r e ( con ,
this .
getNextPlayerCode
(
d a u gh t e r . g e t C l a s s ( ) . getName ( ) ,
I n t e g e r . MIN VALUE) ) ) ;
p l a y e r s [ p o p s i z e − pdex − 1 ] . s e t S t r a t e g y ( d a u g h t e r ) ;
i f ( pdex < p o p s i z e / 2 − 1 ) { // don ’ t do t h i s t h e l a s t time
mothers [ c h o i c e ] = mothers [ p o p s i z e / 2 − pdex − 1 ] ; } } }
I now have an output population equal in size to the input population, with a parentage represented by how well it had done in the objective function compared to the
mean. Let us next look at how the child strategy objects differ from the parents.
85
6.4.4
Crossover
Alleles are simply split in half, and the left half from the left parent is joined to the
right half of the right parent. As with the care phenotype and many other things
in the model, more complex and fruit-promising structures were employed, modified
and discarded in the process of understanding the model. Variable crossover could
be profitably reemployed here.
/∗ ∗
∗ G e n e r a t i o n a l C o n s t r u c t o r . Let a s t r a t e g y be t h e c h i l d o f two
parents
∗ @param s p a c e t h e parameter s p a c e i n which t h e s t r a t e g y
operates
∗ @param l e f t t h e p a r e n t t h a t c o n t r i b u t e s t h e l e f t p a r t o f t h e
allele
∗ @param r i g h t t h e p a r e n t t h a t c o n t r i b u t e s t h e r i g h t p a r t o f
the a l l e l e
∗
∗/
public GAStrategy ( ParameterSpace space , GAStrategy l e f t ,
GAStrategy r i g h t ) {
10
this . space = space ;
/∗ r i g h t , ok . Go t h r o u g h t h e a l l e l e s , d e t e r m i n e a s p l i t t i n g
p o i n t , and j o i n
’em up
∗/
boolean [ ] [ ] mother = l e f t . g e t A l l e l e s ( ) ;
boolean [ ] [ ]
father = right . getAlleles () ;
boolean [ ] [ ]
a l l e l e s = new boolean [ mother . l e n g t h ] [ ] ;
f or ( int genome = 0 ; genome < a l l e l e s . l e n g t h ; genome++) {
// copy t h e mother ’ s a l l e l e
boolean [ ]
a l l e l e = new boolean [ mother [ genome ] . l e n g t h ] ;
86
20
// f i n d t h e s p l i t t i n g p o i n t
int s p l i t = 2 ;
// r e p l a c e t h e gene ’ s w i t h t h e f a t h e r ’ s a f t e r t h e s p l i t t i n g
point
fo r ( int gene = 0 ; gene < a l l e l e . l e n g t h ; gene++) {
a l l e l e [ gene ] = gene < s p l i t ? mother [ genome ] [ gene ] :
f a t h e r [ genome ] [ gene ] ; }
a l l e l e s [ genome ] = a l l e l e ; }
//OK, so now I have t h e same l i s t o f a l l e l e s as my p a r e n t s .
Mutate them .
a l l e l e s = mutate ( a l l e l e s ) ;
setAlleles ( alleles ) ;
30
initializeStartingConfiguration () ;}
6.4.5
Mutation
The mutate method call in listing 6.4.4 simply looks as each gene of an allele in turn,
tests its probability of mutating a gene against a uniform random number between 0
and 1, and flips the bit on the gene if the random number is less than the mutation
probability. If it has flipped any of the bits on an allele, it replaces it. This code
follows.
/∗ ∗
∗ Unpredictably a l t e r s the strategy ’ s a l l e l e s
∗ @param a l l e l e s −− two d i m e n s i o n a l a r r a y o f b o o l e a n s
∗ @return −− argument a r r a y w i t h randomly s e l e c t e d b i t s
f l i p p e d . The
∗ p r o b a b i l i t y o f s e l e c t i o n i s t h e parameter
probabilityofmutation
∗/
public boolean [ ] [ ] mutate ( boolean [ ] [ ]
87
alleles ) {
// go t h r o u g h each b i t . I f i t ’ s u n l uc k y , f l i p i t
10
f or ( int genome = 0 ; genome < a l l e l e s . l e n g t h ; genome++) {
boolean [ ]
a l l e l e = a l l e l e s [ genome ] ;
boolean d i r t y b i t = f a l s e ;
for ( int gene = 0 ; gene < a l l e l e . l e n g t h ; gene++) {
i f ( rand . n e x t F l o a t ( ) < p r o b a b i l i t y o f m u t a t i o n ) {
d i r t y b i t = true ;
a l l e l e [ gene ] = ! a l l e l e [ gene ] ; } }
// i f you changed a b i t , swap i t o u t
if ( dirtybit ) {
a l l e l e s [ genome ] = a l l e l e ; } }
20
return a l l e l e s ; }
88
6.5
Optimization
To apply the genetic algorithm search method to the Comparator, we took the tendencies outlined in section 6.2, and expressed each as a four-bit allele3 as described
in section 6.4.1.
We then allowed populations to evolve along the lines described in section 6.4,
passed the strategies to the players described in Chapter 5 and sought lessons in the
result.
The investigations with the genetic agents differed from the investigations of Chapter 5:
1. The rewards are not parabolic, and these agents operate without a quality loss
function. They obtain one quantum payoffs for every payoff attempt for which
the experimental result is within 5 % of the target, which is again 80. This
corresponds to a range 18.524 to 19.616. Table A.1 is the data source yet
has relatively few data in this range, making it difficult to hit with normally
distributed pseudorandom numbers generated around each configuration’s mean
according to its standard deviation.
2. Only the scaled Grewen data is used. The nonlinearity and noise, discussed in
section 5.3, are not adjusted.
3. Neither adjusted in this study is the payoff. While the model is certainly capable
to this sort of adjustment, and such model runs were made, the output was not
instructive and is not presented here.
As a nod to the hierarchical ordering principle, we instruct the strategy to construct an optimal configuration from a main effects model using the data it has
gathered. As the configuration submitted to payoff testing is the best configuration
the player has experienced, this creates some reward for seeking balance to the extent
that main effects are important.
3
In Java, this was implemented as a four member boolean array
89
Average Payoff By Generation
800
0.35
600
0.3
0.25
200
0.2
0
1
2
3
4
5
6
7
8
9
10
0.15
Payoffs
Value Units
400
-200
Cost
Wealth
Payoff
0.1
-400
0.05
-600
0
-800
Generation
Figure 6-1: Evolution of Payoff over ten generations(Run 1)
Average Payoff By Generation
800
0.35
600
0.3
0.25
200
0.2
0
1
2
3
4
5
6
7
8
9
10
0.15
Payoffs
Value Units
400
-200
Cost
Wealth
Payoff
0.1
-400
0.05
-600
0
-800
Generation
Figure 6-2: Evolution of Payoff over ten generations(Run 2)
6.6
Results and Analysis
In figures 6-1 and 6-2, the players can be seen to suffer exponentially decaying cost
with a more affine decay in payoff until the last few generations, wehre payoff suddently recovers after cost reduction has stopped increasing wealth. The agents have
generations of sacrificing payoff for reduced experimental cost; they never get very
good at finding the best configurations.
6.6.1
Strategy Competence
Let us introduce a concept of strategy competence. In the NormalDataGame the
genetic agents are playing each configuration t has a mean µt and a standard deviation
90
Table 6.2: Ten 80-best configurations
Plane # 1311 3121 3112 3321 2112 1313 1321 3231 3313 2312
mean 83.15 84.25 86.45 73.63 87.91 71.43 83.88 69.23 90.84 80.95
stdevp 10.99 10.62
9.04 15.30 11.47
8.79 21.03 18.88 11.16 31.90
80 goodness
0.36
0.35
0.30
0.24
0.24
0.21
0.21
0.17
0.16
0.14
Table 6.3: Ten modal configurations, run 1, 200 players ten generations
MODE
2
2111 3111
3
1223
4
1311
5
6
1313 1323
7
3211
8
1233
9
1211
10
1222
σt . The player will receive a full payoff of p(a) = 1 if experiment a yields a performance
within tolerance of the target, which here means between 76 and 84.
Since the performance (or flight distance) is a pseudorandom normal variable, p’s
expectation value < p > is given by the error function.
2
2 Z 84 (µ−x)
e σ2 dx
< p >= √
π 76
(6.5)
This expectation value for payoff varies from 0 to 1, and is called the ’80-goodness.’
The 80-goodness for the top ten configurations are listed in table 6.2. To measure
the competence of the algorithm, let us run our model twice with populations of 200
agents for ten generations. The top ten selected configurations among all agents in
these models are shown in table 6.3 and table 6.4.
We get a sense of the competence of the populations by looking at how well the
top ten selections congrue. The first, second and seventh most popular selections for
the first run are respectively the second, first and ninth configurations for the second,
Table 6.4: Ten modal configurations, run 2, 200 players ten generations
MODE
2
3111 2111
3
1211
4
1213
5
6
1322 1212
91
7
1321
8
1221
9
3211
10
2211
Diligence Allele By Generation
40
35
Generation 1
30
Generation 2
25
Generation 3
20
Generation 4
Generation 5
Generation 6
Generation 7
Generation 8
Generation 9
15
10
5
0
0000
0011
Generation 10
0110
1001
1100
1111
Figure 6-3: Evolution of Diligence(Run 1)
but none of them contain any of the top ten configurations for the space.
These algorithm, then, is not terribly competent.
6.6.2
Two model runs
Figures 6-5 and 6-6illustrate how profligacy evolves over the course of several generations. The agents tended to reduce costs for the first several generations, making
their payoffs low. This is, we posit, because the noise and interaction levels were so
high no viable strategy ever arose. That we allowed agents with negative values to
propagate may have bred agents unable to operate in the data environment.
We can now look at the evolution in two runs of the five alleles. Let us consider
diligence first, as it has no operational role and will tell us when spurious convergence
has taken place, either through some correlation with a useful allele – this is an
artifact of our crossover rule – or through gene starvation.
The figures show a clear, slowly growing convergence around 1001 in run 2 (figure
6-6) and a bimodal one around 0010 and 1111 in run 1 (figure 6-5). The model
has been validated in that the convergences are around different numbers. A larger
population size would converge more slowly.
Since from the payoff chart we know what happens to profligacy, let us look at that
next. Fully half the population is at 0000 Profligacy, which means the agent enacts 1
92
Diligence Allele By Generation
45
40
Generation 1
Generation 2
Generation 3
Generation 4
Generation 5
Generation 6
Generation 7
Generation 8
35
30
25
20
15
10
5
0
0000
Generation 9
0011
Generation 10
0110
1001
1100
1111
Figure 6-4: Evolution of Diligence(Run 2)
Profligacy Allele By Generation
120
100
Generation 1
Generation 2
80
Generation 3
60
Generation 4
Generation 5
40
Generation 6
20
Generation 7
Generation 8
Generation 9
Generation 10
0
0000
0011
0110
1001
1100
1111
Figure 6-5: Evolution of Profligacy(Run 1)
plan, by generation 6. While a full 30 % of run 2’s agents go to 0000 profligacy, there
is another attractor at 0100 (which enacts 1 · 22 + 1 = 5 plans.) We are starting to
see evidence of adaptive behavior, which requires multiple plans.
Planning, though, is where adaptive and rigid behavior are distinguished. In run
1 (figure 6-7), 30 % of the population is at 0 – which is equivalent to Guess – at
generation 6 and 40 % by generation 10, with as many again experimenting on only
one or two configurations. As guess plays havoc with selected configurations, this
explains the lack of competence of the populations. In run 2 (figure 6-8), we see a
residual population slowly decaying at 1000 Run 2 (figure 6-10) has another mid-field
93
Profligacy Allele By Generation
90
80
Generation 1
Generation 2
Generation 3
Generation 4
Generation 5
Generation 6
Generation 7
Generation 8
70
60
50
40
30
20
10
0
0000
Generation 9
0011
Generation 10
0110
1001
1100
1111
Figure 6-6: Evolution of Profligacy(Run 2)
Planning Allele By Generation
90
80
Generation 1
Generation 2
Generation 3
Generation 4
Generation 5
Generation 6
Generation 7
Generation 8
Generation 9
Generation 10
70
60
50
40
30
20
10
0
0000
0011
0110
1001
1100
1111
Figure 6-7: Evolution of Planning(Run 1)
94
Planning Allele By Generation
140
120
Generation 1
Generation 2
Generation 3
100
80
Generation 4
60
Generation 5
40
Generation 6
20
Generation 7
0
Generation 8
0000
Generation 9
0011
Generation 10
0110
1001
1100
1111
Figure 6-8: Evolution of Planning(Run 2)
Care Allele By Generation
100
Generation 1
Generation 2
80
60
Generation 3
Generation 4
40
Generation 5
Generation 6
Generation 7
Generation 8
Generation 9
20
0
0000
0011
Generation 10
0110
1001
1100
1111
Figure 6-9: Evolution of Care(Run 1)
population in Care, this one holding steady with about an eighth of the population.
Otherwise, the drive to zero is very strong, capturing half the population of run 1
(figure 6-9) and 5/8 of run 2 by the tenth generation. Repetitions are not found to
be adaptive in this environment.
Fairness does not converge convincingly, in fact converging dramatically less than
diligence in run 2 (figure 6-11, compare to (figure 6-12)). We would expect that fairness, which propels an agent though the parameter space, would only be a competitive
advantage if the agents were doing long experimental designs. Since they tend to do
so little experimentation, they can find no advantage in doing them evenly.
95
Care Allele By Generation
100
Generation 1
Generation 2
80
60
Generation 3
Generation 4
40
Generation 5
Generation 6
20
Generation 7
0
Generation 8
0000
Generation 9
0011
Generation 10
0110
1001
1100
1111
Figure 6-10: Evolution of Care(Run 2)
Fairness Allele By Generation
35
30
Generation 1
Generation 2
Generation 3
25
20
Generation 4
15
Generation 5
10
Generation 6
Generation 7
Generation 8
Generation 9
Generation 10
5
0
0000
0011
0110
1001
1100
1111
Figure 6-11: Evolution of Fairness(Run 2)
96
Fairness Allele By Generation
50
40
Generation 1
Generation 2
30
Generation 3
Generation 4
20
Generation 5
10
Generation 6
Generation 7
0
Generation 8
0000
Generation 9
Generation 10
0011
0110
1001
1100
1111
Figure 6-12: Evolution of Fairness(Run 1)
6.7
Analysis
The lessons in these results are minimal. However, the concept and framework for
abstracting and optimizing design of experiments strategies here presented can provide a springboard for further work, and can be run in the Comparator against other
data sets or with other abstractions and codings.
97
Chapter 7
Conclusion
Statistically driven experimental strategies display a healthy diversity in practice.
The relative qualities of experimental strategies are quantifiable within the supplied framework.
We propose that it is possible to optimize an experimental design itself.
7.1
7.1.1
Summary
Problem Statement
Design of Experiments (DoE) is a fundamental part of Robust Engineering. Deciding
which prototypes to build, how many of each to test and how to interpret the results
is part of what makes Product Development an art.
The goal of Robust Engineering is to minimize the effect of variability on performance and thus to optimize quality. The goal of DoE is to do this at the least
possible cost by choosing and interpreting experiments wisely. However, it is not at
all obvious what the correct set of configurations to select for experimentation are,
or how to select the number of experiments to do on each one.
With the rise of the use of statistics in product design through the twentieth
century, and especially with Taguchi methods in the last half, the practice of fitting
a statistical model to the reality of product performance, doing a predicted number
98
of experiments, and selecting a best configuration based on the model coefficients has
become widely adopted for design projects.
As with any business transformation movement, there is a risk that the core messages will be lost and the selection of configurations will become rote. Or that configurations selected in one product performance space will be inappropriately applied to
another. So, we ask, ”what are people doing when they do design of experiments?”
Further, ”How is management trying to influence such choices?” And, ”How effective
are management interventions?”
We also partially develop a method for asking how Design of Experiments can
be improved to create good answers on an arbitrary design surface. To do this, we
have to develop a way for objectively evaluating one strategy against another in a
particular design space. Further, we have executed an optimization algorithm against
that framework to demonstrate how one might characterize and perfect a design of
experiments strategy.
7.2
Conclusion
This thesis is organized in 7 chapters, dealing with three broad themes. The first is
the contextualized prototype testing interviews; the second is the comparator; and
the third is the optimization algorithm.
In the first section, we discussed the practice of Design of Experiments in Robust
Engineering. From previous work, we had some idea of the popularity of the concept,
and we pursued a few in depth interviews to get some insight into the structure of
prototype testing decisions made in individual companies.
This has some implications. Our expectation was that the position of prototype
testing in validated design, the last phase of detailed design, itself the last phase of
the design phase of the product development time line, that is, the very last thing you
do before implementing, would create a pressure to cut into the time spent carefully
assembling and testing the prototypes.
We did not find that practitioners tended to return to earlier parts of the design ef99
fort in response to information generated during the Design of Experiments endeavor.
This suggests that the parameter design was done carefully enough to isolate it from
the system design, and that the parameters themselves were not pushed up against
their constraints.
Analytic models provide some DoE opportunity at the system level. They are
useful when the governing equations are well known and the statistical model has
a good fit to the data, and can give valuable guidance around sensitivity. We were
impressed by the confidence shown by analytic prototypers, and the clarity and definition around their belief they were saving their organizations time and money.
We acknowledge system level mathematical analysis is difficult, and that the machines and expertise they require can be prohibitively expensive. Yet, as tools are
always improving and computers becoming more powerful[31, fig. 1-1], we urge designers of complex product systems to periodically revisit the issue of whether to
create such a system.
We discovered a very real difference between product owners and subcontractors
that indicates a supply chain issue with quality engineering. Neither the defense
contractor in section ch4:prof-nmp nor the parts manufacturer in section ch4:profapm seem particularly concerned with delivering the best possible product, because
there is no contractual reward for overdelivering on performance or variability. In
section 4.3.1, we recommend that proper incentives for quality be written into parts
and subsystem contracting.
Design of experiments in robust engineering has come to be closely linked with
design for six sigma (DFSS). DFSS would be considered a framework in the five-layer
model, and the warning explicated in section D.5 may pertain. DFSS may start
to lose its core meaning, and seems to lack coordination mechanisms such that any
one implementation will comport with every other. This puts DFSS in danger of
being used as a generic label for any kind of statistical control, which will eventually
discredit it. An industry movement to create consistency in the certification of belted
staff would be helpful in delaying that.
In the second section, we showed that a comparator could be built, and would give
100
expected and consistent results for particular payoff structures and response surfaces.
The comparator is adaptable to different response surfaces, and can operate as a data
picker or a number generator, with or without underlying data.
We showed that it was possible to code a variety of DoE strategies into this framework, and that it would generate a metric with which strategies could be compared
objectively.
In the third section, we illustrated how one might abstract the differences between design of experiments strategies, abstracting generic concepts in design of experiments. We use a Genetic Algorithm model to optimize a DoE strategy for a
particular response surface.
7.3
Further Work
Our intention here has been to articulate and develop ideas for further rigorous investigations into design of experiments, and we have some suggestions for how that
work can proceed.
1. Ahlman[32] discusses some ways to apply design of experiments in robust engineering. We have discovered a variety of methods of creating statistically driven
experimental designs, including Uniform Design, D-Optima, Latin Hypercubes,
Orthogonal Arrays and other Fractional Factorials, as well as Full Factorial and
One-At-A-Time methods. It may be of interest to find out how popular each of
these methods are and what factors outside of the statistical characteristics of
the system of interest – like DFSS vendor, industry, educational institution or
region – drive the selection of a particular DoE strategy.
2. The model for process support introduced in section 4.3 can be further elucidated and applied to various situations. It could perhaps be recast as a prescriptive or diagnostic model.
3. We have suggested that the practice of DoE is generally adopted locally by
design engineers when the corporation enrolls them in a design for six sigma
101
(DFSS) initiative. The mechanisms by which this occurs could be examined
and codified, either as case studies or within an industry.
4. The framework for comparing design of experiments strategies can be employed
to seek nonlinear optimization schemes in DoE that can beat or compare to the
optimal statistical scheme in a number of different circumstances.
5. Frey, et al.[12] suggests that there is a time to use more search-oriented experimental designs over statistically oriented ones. What the nature and characteristics of this boundary are can be investigated using the Comparator, with
the intent of developing a practical heuristic for strategy switching.
6. The optimization scheme for DoE strategies themselves outlined in Chapter 6
gave us answers limited in scope, as elucidated in section 6.6. However, we
believe that this thesis does show the concept of abstracting design of experiments strategies and creating optimization schemes in the supplied framework
is essentially sound, and we encourage others to follow this work by exploring
different response and payoff spaces.
102
Appendix A
Grewen Data
On the following pages are the data from the paper airplane game as they appeared
in Grewen[29]. This was used in chapters 5 and 7. The data measures flights of
airplanes with different configurations and is discussed in section 5.3.1.
103
PLANE #
Throw 1
Throw2
12
23
15
19
11.5
13
15.5
12
13
14
18.5
13
14
15
13
13
14
15
12
13
14.5
14
10.5
12
13
17
14
17
16
15.5
15.5
16.5
14
11
21
20
15.5
17.5
12
15
13
12
13
13
15
1121
1122
1123
1131
1132
1133
1211
1212
1213
1221
1222
1223
1231
1232
1233
1311
1312
1313
1321
1322
1323
1331
1332
1333
2111
2112
2113
2121
2122
2123
2131
2132
2133
2211
2212
15
18
9
13
15
19
20
16
14.5
19
13
12
12.5
11
16
15
11
8
14.5
13
14
13.5
15
13.5
17.5
15
14
1113
19
15.5
1112
2.20
Table A.1: 10 throws for each of 1111
81 configurations
18
104
15
16
15
14
14.5
11
11.5
16.5
17
20
22
13
13
13
14
18
24
18
16.5
19
13
11
13.5
10
16.5
13
9
7.5
14.5
12.5
14
15
16
14.5
16
14.5
17
16.5
Throw 3
14.5
15
13
14
14
14
14.5
16
17
20.5
23
16
12
14
16
17
23
20
16
18
13.5
12
20
12.5
14
14
9
14.5
13
16
15
15
16
13.5
15
14
18
16.5
Throw 4
12
15
15
13
15
13
12
15
16
17.5
22
11
14
15
16
17
20
18.5
15
20
13
12
10.5
10.5
14
14
10
15
13
14
15
18
16.5
15.5
17
14
14.5
17.5
Throw 5
11
13
14
15
14
11
13.5
18
16
20
21
12
15.5
13.5
15
18
21
19
16
21
13
12
13.5
9
13
14
11.5
15
14
13
13.5
16
16
15
14.5
15.5
13
17.5
Throw 6
11
14
13.5
13.5
15
13
12
17
17
21
19
14
14
15.5
15
14
21
17
17
20
12.5
14
11.5
11
14
15
10
13
14
16
15
16.5
15.5
15
16
15
18
17.5
Throw 7
13
14.5
15
15
15
11.5
12
19
16.5
23
21
10
14
14
16
18
18
18
15
18
13
13
16.5
9
14
14
5
13.5
16
16.5
14.5
14.5
15
14
15
15
17.5
18
Throw 8
12
10.5
17
13
14
11.5
14
19
18
22
22
13
12
13
15
17
19
19
17
22
12.5
14
14.5
10.5
13
16
7
15.5
16
15
15
14.5
14
15
16
14
16
15
Throw 9
16
12.5
15
16
13
12
13
15
15
19.5
24
12
12
15.5
16.5
20
14
16.5
16
21
13
15
12.5
12
13.5
16.5
11.5
13
14
19
14
15
19
14.5
17
25
14.5
17.5
Throw 10
19
13.35
13.65
14.55
13.75
14.3
12.5
12.6
17.2
16.3
20.15
21.8
12.4
12.95
14.3
15.4
17.35
19.6
17.9
15.7
19.5
12.95
12.7
13.55
10.95
14.25
14.45
9.6
13
14.3
14.8
14.3
15.3
15.7
14.35
16.25
15.6
16.3
Mean Distance
1.80
1.58
1.21
1.32
0.86
1.33
1.07
1.57
0.98
1.65
1.40
1.71
1.80
1.25
0.74
1.70
3.03
1.26
1.03
1.58
0.28
1.25
2.91
1.55
1.16
1.17
2.21
2.91
1.03
2.08
0.71
1.25
1.44
0.82
1.25
3.35
1.92
17.3
Std Dev
3.23
2.50
1.47
1.74
0.73
1.78
1.16
2.46
0.96
2.73
1.96
2.93
3.25
1.57
0.54
2.89
9.16
1.60
1.07
2.50
0.08
1.57
8.47
2.41
1.35
1.36
4.88
8.44
1.07
4.34
0.51
1.57
2.07
0.67
1.57
11.21
3.68
1.09
Variance
0.42
0.42
0.11
0.55
0.56
1.24
1.83
1.01
1.66
2.00
3.62
0.79
0.54
0.26
1.26
0.91
0.53
1.99
0.93
1.91
21.88
1.28
0.14
1.56
0.34
0.19
1.05
0.20
0.39
0.02
0.81
0.37
0.48
0.54
0.98
0.08
0.43
1.18
S/N Ratio
105
14
14
14
13
21
11
14
18
22
18
17
13
13
11
11
12
12
17
16
12
13
14
14.5
9
10
12
15
17
18
19
15.5
13
15
14
13
14
8.5
9
14.5
13.5
12
20
12
11
13.5
15
22
13
15
18
2221
2222
2223
2231
2232
2233
2311
2312
2313
2321
2322
2323
2331
2332
2333
3111
3112
3113
3121
3122
3123
3131
3132
3133
3211
3212
3213
3221
3222
3223
3231
3232
3233
3311
3312
3313
3321
3322
3323
18.8
13.5
7
11.5
14
13
14.5
13
14.5
15
19
17
19
14
14
13
11.5
14.5
14
14
10.5
15
19
11.5
12
11
13
14
11
7.5
2213
Throw2
Throw 1
PLANE #
18
18
20
22
16
13
11
11
14
13.5
13
15
9
9
16
12
14.5
16
14.5
17.5
20.5
16
20
13
12
11
10.5
15
13
14.5
13
17
15
16.5
12
10
10
14
14
6
Throw 3
17
15
18
20
12
14
12
11
18
11
12
15
5
9
14
13
16
17.5
15
17
18.5
17
20
16
15
12.5
11
13.5
14
18.5
12
24
16
11.5
11.5
9
12
14
13.5
12.5
Throw 4
19
17
18
20
17
14.5
12
13
14
11
12.5
15.5
8.5
14
16.5
14
15
14.5
15.5
17
19.5
19
22
17
14
16
9.5
16.5
13
14.5
14
16
18
10.5
12.5
11
13
14.5
12.5
12
Throw 5
17
17.5
20
19.5
16
14
12
10
14
13
12
11.5
5
14
16
14
16
14.5
16
16.5
22.5
14
20.5
15
14
12
11
17.5
12
14.5
14
18
18
13
12
11.5
9
15.5
13.5
7
Throw 6
16.5
17.5
19
20
17
14
12
11.5
19
14
11
13
9
12
13
13.5
14
13
17
17.5
19
16
20
20
13.5
13
12
15.5
15.5
15.5
7
27
17.5
13
15.5
11
13.5
15
15
12
Throw 7
14
14
17
22
18
14
14
12
17
13.5
10
15
9
14
15
12.5
16
15
14
17
20.5
17
21
22
14
14
14
16
12
13
12
26
18
12.5
10.5
12.5
12
12
15
6
Throw 8
17
16.5
18
21
16.5
13
14
11
19
13.5
11
14
10
14
14
13
15
15
16
17.5
21
20
20
20
16
13
12
17
13
15
12
17
18.5
11
13.5
12
11
15
14
8
Throw 9
15.5
14
21
17
13
8
13
17
20
14
12
14.5
10
10
13
13.5
16
15.5
22
18
17
19
20
15
13
12
10.5
16
14
16
14
16
12.5
16
12
10
10
13.5
14
12
Throw 10
17.08
16.15
18.2
20.55
15.85
13.15
12.5
11.95
17.6
12.85
12.1
14.2
8.15
11.6
14.55
13.15
15.1
14.9
15.75
16.85
19.65
17.3
19.95
16.7
13.75
12.65
11.1
15.6
13.45
14.85
12.05
19.2
16.95
12.75
12.35
10.9
11.45
14.05
13.85
9.4
Mean Distance
1.52
1.51
2.20
1.61
2.00
1.87
1.18
1.95
2.72
1.13
1.22
1.18
1.86
2.33
1.26
0.63
0.84
1.33
2.47
0.94
1.53
1.77
1.30
2.98
1.23
1.63
1.41
1.24
1.07
1.60
2.11
4.59
1.96
2.02
1.33
1.02
1.50
1.04
0.78
2.73
Std Dev
2.30
2.28
4.84
2.58
4.00
3.50
1.39
3.80
7.38
1.28
1.49
1.40
3.45
5.43
1.58
0.39
0.71
1.77
6.13
0.89
2.34
3.12
1.69
8.90
1.51
2.67
1.99
1.54
1.14
2.56
4.47
21.07
3.86
4.07
1.78
1.04
2.25
1.08
0.61
7.43
Variance
1.03
0.63
0.72
2.26
0.28
0.45
1.59
0.73
0.39
1.45
1.75
0.37
1.90
0.57
0.10
3.99
0.54
0.11
0.17
2.40
2.11
0.83
3.10
0.22
0.64
0.77
1.82
0.57
1.11
0.05
0.60
0.21
0.58
0.48
1.33
3.65
1.45
0.61
1.41
0.71
S/N Ratio
106
14
12
13
3332
3333
16.5
12.5
12.5
3331
Throw2
Throw 1
PLANE #
14
13
13
Throw 3
14.5
10.5
11
Throw 4
16
13.5
12
Throw 5
18
10.5
14.5
Throw 6
17
7.5
13
Throw 7
13.5
13.5
13
Throw 8
17
12.5
16
Throw 9
16.5
13
13
Throw 10
15.6
12
13.05
Mean Distance
1.71
1.99
1.36
Std Dev
2.93
3.94
1.86
Variance
0.30
0.69
0.89
S/N Ratio
A.1
The Contributions
This data was extracted in preparation for calculating the components in section
5.3.1. Whereas in table A.1 and in most discussions in this thesis the parameter
levels are labled 1 to 3, two was subtracted from each level index to generate table
A.2.
107
Dif
measure
A
0.106
14.713
Y
A
Constant term
19.50
15.70
17.90
19.60
17.35
15.40
14.30
12.95
12.40
21.80
20.15
16.30
17.20
12.60
12.50
14.30
0.58
3.97
3.52
2.68
2.14
0.77
0.11
0.18
3.91
4.18
2.24
0.88
2.41
1.20
0.45
1.26
12.95
9.60
14.45
4.05
0.87
13.00
2.73
0.39
14.30
3.51
12.70
14.80
0.70
13.55
14.30
0.39
0.78
15.30
1.58
0.07
15.70
1.96
10.95
14.35
0.20
14.25
16.25
0.90
2.55
15.60
2.22
0.36
16.30
1.12
1
14.75
13.70
15.01
16.32
14.06
15.97
17.89
12.58
13.06
13.53
13.26
14.67
16.08
13.93
16.28
18.63
13.34
13.48
13.62
13.50
14.61
15.71
13.65
15.73
17.81
14.10
13.91
13.72
13.74
14.55
15.35
13.38
15.18
0
0
0
0
0
0
0
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
-1
Table A.2: Variance due to linear components
0.32
in 17.30
the Grewen Data
16.98
108
C
-1
-1
-1
-1
-1
-1
-1
1
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
-1
-1
-1
-1
-1
-1
-1
-1
1
0
0
0
-1
-1
-1
1
1
1
0
0
0
-1
-1
-1
1
1
1
0
0
0
-1
-1
-1
1
1
1
0
0
0
-1
-1
-1
B
-1
-1.012
C
-0.298
B
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
D
-0.825
D
0
0
0
0
0
0
0
-1
-1
-1
-1
-1
-1
-1
-1
-1
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
-1
A*B
-0.359
A*B
0
0
0
0
0
0
0
-1
-1
-1
0
0
0
1
1
1
-1
-1
-1
0
0
0
1
1
1
-1
-1
-1
0
0
0
1
1
1
A*C
0.111
A*C
0
0
0
0
0
0
0
-1
0
1
-1
0
1
-1
0
1
-1
0
1
-1
0
1
-1
0
1
-1
0
1
-1
0
1
-1
0
1
A*D
0.283
A*D
-1
0
0
0
1
1
1
1
1
1
0
0
0
-1
-1
-1
0
0
0
0
0
0
0
0
0
-1
-1
-1
0
0
0
1
1
1
B*C
-0.050
B*C
1
-1
0
1
-1
0
1
1
0
-1
1
0
-1
1
0
-1
0
0
0
0
0
0
0
0
0
-1
0
1
-1
0
1
-1
0
1
B*D
0.483
B*D
-1
0
0
0
-1
0
1
1
0
-1
0
0
0
-1
0
1
1
0
-1
0
0
0
-1
0
1
1
0
-1
0
0
0
-1
0
1
C*D
0.514
C*D
0
0
0
0
0
0
0
-1
-1
-1
0
0
0
1
1
1
0
0
0
0
0
0
0
0
0
1
1
1
0
0
0
-1
-1
1
A*B*C
0.440
A*B*C
0
0
0
0
0
0
0
-1
0
1
-1
0
1
-1
0
1
0
0
0
0
0
0
0
0
0
1
0
-1
1
0
-1
1
0
-1
A*B*D
0.785
A*B*D
0
0
0
0
0
0
0
-1
0
1
0
0
0
1
0
-1
-1
0
1
0
0
0
1
0
-1
-1
0
1
0
0
0
1
0
-1
A*C*D
-0.454
A*C*D
1
0
0
0
1
0
-1
1
0
-1
0
0
0
-1
0
1
0
0
0
0
0
0
0
0
0
-1
0
1
0
0
0
1
0
-1
B*C*D
-0.092
B*C*D
0
0
0
0
0
0
0
-1
0
1
0
0
0
1
0
-1
0
0
0
0
0
0
0
0
0
1
0
-1
0
0
0
-1
0
-1
A*B*C*D
-0.063
A*B*C*D
109
11.10
12.65
13.75
16.70
19.95
17.30
19.65
16.85
15.75
14.90
15.10
13.15
14.55
11.60
2.17
0.70
0.32
2.09
3.18
2.56
2.36
1.37
2.08
0.88
0.92
0.56
1.77
4.12
15.85
15.60
1.53
1.18
13.45
0.97
13.15
14.85
0.09
12.50
12.05
2.66
0.70
19.20
3.72
0.94
16.95
0.71
11.95
12.75
0.64
1.97
12.35
1.35
17.60
10.90
3.11
3.20
11.45
2.44
12.85
14.05
0.66
12.10
13.85
1.69
1.43
9.40
4.99
2.72
13.35
2.37
8.15
13.65
3.41
14.20
14.55
1.20
1.16
13.75
6.97
measure
Dif
0.30
Y
14.67
13.85
13.44
13.92
14.40
14.28
14.82
15.36
15.12
15.72
16.32
12.59
14.18
15.78
13.67
15.48
17.29
14.74
16.77
18.79
13.43
13.35
13.27
14.07
14.42
14.76
14.71
15.48
16.24
13.39
13.70
14.01
13.89
14.71
15.54
14.39
15.72
17.06
13.35
14.05
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
A
B
1
1
0
0
0
0
0
0
0
0
0
-1
-1
-1
-1
-1
-1
-1
-1
-1
1
1
1
1
1
1
1
1
1
0
0
0
0
0
0
0
0
0
-1
-1
-1
-1
1
1
1
0
0
0
-1
-1
-1
1
1
1
0
0
0
-1
-1
-1
1
1
1
0
0
0
-1
-1
-1
1
1
1
0
0
0
-1
-1
-1
1
1
C
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
D
1
1
0
0
0
0
0
0
0
0
0
-1
-1
-1
-1
-1
-1
-1
-1
-1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
A*B
-1
-1
1
1
1
0
0
0
-1
-1
-1
1
1
1
0
0
0
-1
-1
-1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
A*C
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
1
0
-1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
A*D
-1
-1
0
0
0
0
0
0
0
0
0
-1
-1
-1
0
0
0
1
1
1
1
1
1
0
0
0
-1
-1
-1
0
0
0
0
0
0
0
0
0
-1
-1
B*C
0
-1
0
0
0
0
0
0
0
0
0
-1
0
1
-1
0
1
-1
0
1
1
0
-1
1
0
-1
1
0
-1
0
0
0
0
0
0
0
0
0
-1
0
B*D
0
1
1
0
-1
0
0
0
-1
0
1
1
0
-1
0
0
0
-1
0
1
1
0
-1
0
0
0
-1
0
1
1
0
-1
0
0
0
-1
0
1
1
0
C*D
-1
-1
0
0
0
0
0
0
0
0
0
-1
-1
-1
0
0
0
1
1
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
A*B*C
0
-1
0
0
0
0
0
0
0
0
0
-1
0
1
-1
0
1
-1
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
A*B*D
0
1
1
0
-1
0
0
0
-1
0
1
1
0
-1
0
0
0
-1
0
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
A*C*D
0
1
0
0
0
0
0
0
0
0
0
-1
0
1
0
0
0
1
0
-1
1
0
-1
0
0
0
-1
0
1
0
0
0
0
0
0
0
0
0
-1
0
B*C*D
0
1
0
0
0
0
0
0
0
0
0
-1
0
1
0
0
0
1
0
-1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
A*B*C*D
110
measure
20.55
18.20
16.15
17.08
13.05
12.00
15.60
Dif
5.06
4.76
1.99
2.19
0.03
1.65
1.32
Y
14.28
13.65
13.02
14.89
14.16
13.44
15.49
1
1
1
1
1
1
1
A
1
1
1
1
1
1
1
B
C
1
1
1
0
0
0
-1
1
0
-1
1
0
-1
1
D
1
1
1
1
1
1
1
A*B
1
1
1
0
0
0
-1
A*C
1
0
-1
1
0
-1
1
A*D
1
1
1
0
0
0
-1
B*C
1
0
-1
1
0
-1
1
B*D
1
0
-1
0
0
0
-1
C*D
1
1
1
0
0
0
-1
A*B*C
1
0
-1
1
0
-1
1
A*B*D
1
0
-1
0
0
0
-1
A*C*D
1
0
-1
0
0
0
-1
B*C*D
1
0
-1
0
0
0
-1
A*B*C*D
This table was generated by listing the means for a configuration in the column
’Y’, and then indicating which elements were present. Let us illustrate the approach
with the A2B1C3D1 configuration. We subtract two from each level so that A = 0,
B = −1, C = 1 and D = −1. Then, for example, B × D = −1 · −1 = 1 and
A × B × C = 0 · −1 · 1 = 0, meaning that this configuration shows the effect of
parameter B on the effect of parameter D, but does not contain A’s effect on B’s
effect on C.
On the first page of the table, the contribution terms are along the top. To
compose the mean of each configuration, these are added, subtracted or not depending
on whether the corresponding column contains a 1, -1 or 0 for that configuration’s
row. The values in the header for each component are XLMiner’s best solution for
this system of 81 equations for 16 unknowns. The composed value is in the Y column,
the actual mean for the configuration in in the ’measure’ column, and the difference
– resulting from higher order terms like A2 C – is in the ’Dif’ column.
This table can then be used to generate table 5.3.1, discussed in section 5.3.1.
111
Appendix B
Interview Questions in Design of
Experiments
”Design of Experiments” (DoE) herein refers to the approach your design group takes
to deciding on which experiments to perform in parameterized product design.
1. Do you have a formal methodology in Design of Experiments? If so, please
answer each of the following questions for each methodology, indicating where
your practice deviates from the formal methodology. If not, please answer each
question according to your tendency.
(a) How did you come up with your methodology?
(b) Have you validated your methodology?
2. How do modelling and prototyping balance in your methodology?
3. How do you determine your testing budget?
(a) How does this vary between different types of product design endeavors?
(b) How flexible is your testing budget?
(c) How often do you come in under budget? Over?
(d) How much does each of these experiments cost you?
112
4. How do you choose your parameter levels?
(a) Do you generally work with product iterations, or with novel products?
(b) How is your system design accomplished? Is it performed with parameter
design in mind?
(c) How well do you feel you know the approximate optimal parameter settings
before you begin?
5. Please describe your level setting
(a) How many parameters can your methodology handle?
(b) Does each parameter need to have the same number of levels?
(c) How many levels can each parameter have?
(d) How do you proceed to fine tune your parameter settings?
6. How do you navigate through the configurations?
(a) Do you know which configurations you will test before you begin?
(b) Under what conditions will you add configurations to test?
(c) Under what conditions will you not test configurations you had originally
planned to?
7. Please annunciate your methodology’s philosophy
(a) Does it try to balance particular groupings, for example, to make sure that
every pair of settings is represented equally? Why?
(b) Is it concerned with covering a large area of the design space?
8. How many experiments do you do on a configuration?
(a) Do you do the same number of experiments on each configuration? Is there
a minimum number that you will do?
113
(b) How do you decide when to stop doing experiments on a particular configuration?
9. Target and variance
(a) How many measurements does your response surface have? What are they?
(b) Do you combine these into a single metric? How?
(c) Are you primarily concerned with value, variance, or some combination of
the two?
(d) Do do use DOE to minimize a response value, maximize a response value,
or try to bring the response value to a particular measure? Do you have
different goals for different responses?
10. Taguchi methods
(a) How do you compare different deviations from an optimal response? Do
you use a parabolic quality loss function to compare different response
values?
114
Appendix C
Tables
115
C.1
Selected Configura-
PlayerID
Configuration
Mu
Frequency
Sigma
10826
2223
19.75543
7
0.337543
10826
3132
61.14312
7
0.095518
10826
3211
57.37536
7
0.170007
10826
3212
44.72732
7
0.776722
10826
3213
36.18656
7
1.127652
10826
3221
51.27432
7
0.196692
10826
3222
38.43142
7
0.325064
10826
3223
30.07707
7
0.202549
Representatives of the four canonical DoE
10826
3231
31.79037
7
0.637585
10826
3232
18.6933
7
0.462875
types were run against the Grewen Data
10826
3233
10.32495
7
0.229383
10826
3123
72.37437
7
0.630645
described in Appendix A, as described in
10826
1222
35.67104
7
0.180602
10826
1211
54.81157
7
0.257605
section 5.5. The Guess strategy had no
10826
3133
52.87849
7
0.13108
10826
2222
27.91374
7
0.228272
data, as it does no experiments before try-
10826
1212
41.28829
7
0.817457
10826
1221
48.50138
7
0.262994
ing for the reward. But, listed below are
10826
1223
27.45756
7
0.417111
10826
1231
28.63735
7
0.400477
ten exemplar players for each strategy.
10826
1232
16.21678
7
0.202462
10826
1233
7.730179
7
0.077896
10826
1311
89.73102
7
0.401164
10826
1312
76.82089
7
0.188109
10826
1313
68.44617
7
0.322122
10826
1321
83.28073
7
0.498183
10826
1322
70.52987
7
0.216
10826
1213
33.80961
7
0.371004
tions for the Strategies
C.1.1
Full Factorial
The list of full factorial agents illustrates
that each such agent visits every configuration.
10826
1331
63.6701
7
0.261361
[toll]wunderbar height 10826
3111
100.2155
7
0.476937
10826
1332
50.57786
7
0.310212
10826
2113
68.36128
7
0.222182
10826
2221
40.88418
7
0.15749
10826
2121
83.38096
7
0.458565
10826
2213
26.3259
7
0.727618
10826
3131
74.11078
7
0.321494
10826
2212
34.10137
7
0.301946
10826
2122
70.4573
7
0.318062
10826
1323
62.13096
7
0.135455
10826
2123
62.07166
7
0.212633
10826
2211
47.18546
7
0.43047
10826
2131
63.77643
7
0.214037
10826
1333
42.36598
7
0.311944
10826
2132
50.75678
7
0.479546
10826
3312
79.60071
7
0.467187
10826
2111
89.48107
7
0.174434
10826
3313
71.20297
7
0.188753
10826
2112
76.64826
7
0.186993
10826
3322
73.26717
7
0.444756
10826
3121
93.72876
7
0.259797
10826
3332
53.23598
7
0.603127
10826
3112
87.21328
7
0.132382
10826
3321
86.05909
7
0.677151
10826
2133
42.46152
7
0.267301
10826
3333
45.3722
7
0.371612
10826
1111
97.23322
7
0.149476
10826
3331
66.5451
7
0.214546
10826
1133
50.24109
7
0.685311
10826
2331
56.25953
7
0.490401
10826
1132
58.50793
7
0.119559
10826
2333
34.94829
7
0.436146
10826
1131
71.34159
7
0.198802
10826
3323
64.78235
7
0.226452
10826
1123
69.87565
7
0.16048
10826
3311
92.23012
7
0.433479
10826
1122
78.12189
7
0.238391
10826
2332
43.09297
7
0.329486
10826
1121
91.31952
7
0.373196
10826
2323
54.5053
7
0.250316
10826
1113
76.08479
7
0.819885
10826
2322
62.72095
7
0.158909
10826
1112
84.30915
7
0.468066
10826
2313
60.53102
7
0.334882
10826
3122
80.83533
7
0.071054
10826
2312
68.63959
7
0.93714
10826
2233
-0.06581
7
0.321205
10827
3111
100.0464
7
1.072997
10826
2232
8.258003
7
0.365448
10827
2112
76.67161
7
0.265895
10826
2231
21.09231
7
0.333574
10827
3112
87.02136
7
0.225619
10826
2311
82.10878
7
0.219928
10826
2321
75.74433
7
0.64096
10826
3113
78.66255
7
0.203463
116
PlayerID
Configuration
Mu
Frequency
Sigma
PlayerID
Configuration
Mu
Frequency
Sigma
10827
3122
80.8941
7
0.207776
10827
1333
42.38289
7
0.493227
10827
2111
89.38625
7
0.295761
10827
2211
47.14543
7
0.234344
10827
3131
74.08672
7
0.36862
10827
2223
19.70233
7
0.334384
10827
3121
93.76518
7
0.430431
10827
2222
28.02398
7
0.122266
10827
2113
68.52522
7
0.207912
10827
2221
40.77921
7
0.191947
10827
2121
83.42163
7
0.373431
10827
1332
50.58789
7
0.436083
10827
2122
70.51149
7
0.392169
10827
2213
25.57597
7
0.746787
10827
2131
63.59153
7
0.10322
10827
1221
48.4994
7
0.173895
10827
2132
50.86116
7
0.287367
10827
3313
71.16184
7
0.480815
10827
2133
42.39767
7
0.154328
10827
3322
73.41985
7
0.385665
10827
1111
97.16582
7
0.267775
10827
3312
79.38481
7
0.417309
10827
3132
61.21471
7
0.096123
10827
3332
53.70172
7
0.598834
10827
2123
62.22394
7
0.373695
10827
3321
86.17372
7
0.555143
10827
1133
50.03784
7
0.36937
10827
3333
45.09119
7
0.482313
10827
1112
84.61602
7
0.37374
10827
3331
66.32618
7
0.332175
10827
1121
91.0857
7
0.259083
10827
2333
34.88251
7
0.255912
10827
1122
78.24737
7
0.142405
10827
2312
69.17097
7
1.114411
10827
1123
69.82908
7
0.19136
10827
2313
60.81978
7
0.639111
10827
1113
76.29016
7
0.527992
10827
2322
62.69747
7
0.252641
10827
1132
58.27875
7
0.250715
10827
2323
54.59292
7
0.254655
10827
1131
71.2232
7
0.245375
10827
2332
43.00685
7
0.285261
10827
3221
51.26542
7
0.186864
10827
3311
92.30759
7
0.540487
10827
2233
-0.10512
7
0.246222
10827
3323
64.86583
7
0.26247
10827
2311
82.08967
7
0.28914
10828
3121
93.80603
7
0.393306
10827
2321
75.79582
7
0.39998
10828
3112
87.00746
7
0.236622
10827
2331
56.14681
7
0.510558
10828
2111
89.35407
7
0.253764
10827
1312
76.71937
7
0.308011
10828
3111
100.1738
7
0.615781
10827
2232
8.127993
7
0.198358
10828
2112
76.51951
7
0.246493
10827
3212
44.69136
7
0.629669
10828
2113
68.3949
7
0.263706
10827
3133
52.87122
7
0.125717
10828
2121
83.18535
7
0.383105
10827
3223
30.17953
7
0.121166
10828
2122
70.37558
7
0.276277
10827
3222
38.6346
7
0.150281
10828
2131
63.54766
7
0.206926
10827
3123
72.88921
7
0.509285
10828
2132
50.5979
7
0.270097
10827
3213
36.48044
7
0.572734
10828
2133
42.42251
7
0.216862
10827
3233
10.47756
7
0.168211
10828
1111
97.33272
7
0.225144
10827
3232
18.58516
7
0.673925
10828
3131
74.33652
7
0.34517
10827
3113
78.79769
7
0.535579
10828
2123
62.25202
7
0.387976
10827
3231
31.33284
7
0.321116
10828
1133
50.08459
7
0.494421
10827
3211
57.40009
7
0.318674
10828
1112
84.64597
7
0.659324
10827
1223
27.59778
7
0.334483
10828
1121
90.83905
7
0.313651
10827
1211
54.80719
7
0.286653
10828
1122
78.14566
7
0.241066
10827
1321
83.62698
7
0.562736
10828
1123
69.92576
7
0.276111
10827
2231
21.1476
7
0.17727
10828
1131
71.36943
7
0.184849
10827
1212
42.16889
7
0.473908
10828
1132
58.42131
7
0.13331
10827
1213
33.3326
7
0.289037
10828
1113
76.22013
7
0.641218
10827
1222
35.56694
7
0.110984
10828
2232
8.26113
7
0.210221
10827
1231
28.55128
7
0.474492
10828
2233
-0.09573
7
0.378231
10827
1232
16.00803
7
0.27603
10828
2311
81.83687
7
0.356894
10827
1233
7.687431
7
0.043323
10828
2231
21.24329
7
0.222391
10827
1311
89.66518
7
0.337454
10828
2221
40.88772
7
0.147584
10827
1313
68.50299
7
0.196558
10828
3232
18.47211
7
0.502869
10827
1322
70.6245
7
0.330486
10828
3222
38.54215
7
0.271337
10827
1323
62.24637
7
0.119609
10828
3113
78.79086
7
0.397972
10827
1331
63.72681
7
0.310501
10828
3231
31.39952
7
0.462361
10827
2212
34.3596
7
0.45069
10828
3122
80.91971
7
0.296848
117
PlayerID
Configuration
Mu
Frequency
Sigma
PlayerID
Configuration
Mu
Frequency
Sigma
10828
3123
72.05493
7
0.433959
10829
2113
68.42635
7
0.229825
10828
3223
30.1779
7
0.190135
10829
2121
83.37988
7
0.396071
10828
3132
61.1019
7
0.136813
10829
2122
70.53217
7
0.241134
10828
3133
52.81496
7
0.094685
10829
2131
63.57518
7
0.190191
10828
3211
57.46619
7
0.154391
10829
2133
42.20438
7
0.298835
10828
3212
44.4349
7
0.666139
10829
2123
62.11723
7
0.15984
10828
3213
36.1785
7
0.613405
10829
1133
49.84422
7
0.465432
10828
3221
51.37692
7
0.161595
10829
3131
74.14424
7
0.208033
10828
3233
10.3248
7
0.259028
10829
1111
97.13857
7
0.349898
10828
1231
29.0472
7
0.614265
10829
1113
75.93712
7
1.038271
10828
1212
42.22875
7
0.653827
10829
1121
91.0866
7
0.317662
10828
1213
33.74085
7
0.417505
10829
1122
78.10229
7
0.138469
10828
1221
48.53656
7
0.277772
10829
1123
69.71775
7
0.340329
10828
2223
19.82986
7
0.290246
10829
1131
71.36426
7
0.16032
10828
1223
27.54512
7
0.406632
10829
1112
84.16204
7
0.47462
10828
1232
15.96498
7
0.234367
10829
1132
58.31867
7
0.18258
10828
1233
7.747209
7
0.036397
10829
3211
57.60604
7
0.30808
10828
1311
89.748
7
0.425798
10829
2232
8.220903
7
0.318173
10828
1312
76.75586
7
0.189069
10829
2233
-0.10856
7
0.297907
10828
1313
68.43218
7
0.195884
10829
2311
81.84451
7
0.289558
10828
1321
83.60116
7
0.916878
10829
2321
75.88627
7
0.295308
10828
1322
70.74817
7
0.188007
10829
3222
38.54431
7
0.312567
10828
1323
62.25103
7
0.11926
10829
3221
51.13753
7
0.236433
10828
1331
63.43309
7
0.140925
10829
2231
21.22055
7
0.162575
10828
1333
42.60045
7
0.378945
10829
2331
55.8943
7
0.323035
10828
2211
46.91662
7
0.363682
10829
3212
44.62272
7
0.376303
10828
1222
35.90616
7
0.206562
10829
3232
18.91144
7
0.675194
10828
1332
50.60954
7
0.534786
10829
3231
31.73651
7
0.415566
10828
2222
27.90685
7
0.127609
10829
3132
61.25321
7
0.166006
10828
2212
34.49342
7
0.399072
10829
3123
72.1469
7
0.55068
10828
2213
26.12851
7
0.544093
10829
3122
80.72006
7
0.14568
10828
1211
54.77262
7
0.175844
10829
3223
30.07654
7
0.136968
10828
3312
79.13236
7
0.486374
10829
3233
10.46892
7
0.173252
10828
3332
53.35021
7
0.650831
10829
3113
78.73872
7
0.301163
10828
3331
66.39347
7
0.385022
10829
3213
36.13676
7
0.445381
10828
3323
65.02283
7
0.305242
10829
1213
33.84972
7
0.378237
10828
3322
73.2647
7
0.25665
10829
2223
19.85847
7
0.48785
10828
3313
71.19716
7
0.273125
10829
3133
52.93364
7
0.068246
10828
3311
92.3657
7
0.55173
10829
1212
42.03442
7
0.340972
10828
2332
42.78661
7
0.467654
10829
1221
48.71138
7
0.223077
10828
3321
86.07349
7
0.340321
10829
1222
35.76385
7
0.263479
10828
2312
69.14748
7
0.812801
10829
1223
27.4276
7
0.48903
10828
2333
34.97653
7
0.212578
10829
1231
29.30435
7
0.352877
10828
2331
56.08963
7
0.339541
10829
1232
15.84157
7
0.115863
10828
2323
54.53473
7
0.300318
10829
1233
7.710956
7
0.05459
10828
2322
62.69022
7
0.183232
10829
1311
89.56654
7
0.241554
10828
2321
75.44436
7
0.247539
10829
1312
76.86509
7
0.114859
10828
2313
60.70046
7
0.385388
10829
1313
68.3949
7
0.254993
10828
3333
45.46196
7
0.415543
10829
1321
83.40976
7
0.559438
10829
2132
50.54586
7
0.355816
10829
1322
70.49447
7
0.208862
10829
2112
76.63599
7
0.122751
10829
1211
54.85568
7
0.272788
10829
3112
87.04453
7
0.17298
10829
2213
26.45385
7
0.663329
10829
3111
100.1194
7
0.831769
10829
2212
34.15749
7
0.201556
10829
2111
89.45248
7
0.179542
10829
1323
62.13889
7
0.137433
10829
3121
93.73225
7
0.238512
10829
2211
47.27901
7
0.267671
118
PlayerID
Configuration
Mu
Frequency
Sigma
PlayerID
Configuration
Mu
Frequency
Sigma
10829
1333
42.61804
7
0.338711
10830
3212
44.70443
7
0.672588
10829
2221
40.95322
7
0.143111
10830
3213
36.48177
7
0.88677
10829
2222
27.99571
7
0.321374
10830
3221
51.46459
7
0.211801
10829
1332
50.63676
7
0.386874
10830
3133
52.91907
7
0.129992
10829
1331
63.61836
7
0.242451
10830
3222
38.39098
7
0.256834
10829
3333
45.27511
7
0.357936
10830
3223
30.23152
7
0.281034
10829
3322
73.19034
7
0.310072
10830
3232
18.72676
7
0.367823
10829
3312
79.48028
7
0.215265
10830
1312
76.77289
7
0.285286
10829
3323
64.94426
7
0.421552
10830
1311
89.87053
7
0.28437
10829
3321
85.81243
7
0.443739
10830
1211
54.74304
7
0.176669
10829
3313
71.22746
7
0.18962
10830
1212
42.17279
7
0.532576
10829
3332
53.67342
7
0.427327
10830
1213
33.58316
7
0.454091
10829
2332
43.20728
7
0.54548
10830
1221
48.59795
7
0.243622
10829
2333
34.69185
7
0.315795
10830
1222
35.60726
7
0.18594
10829
3331
66.23613
7
0.284805
10830
1223
27.45589
7
0.465687
10829
2312
69.41299
7
0.575349
10830
1231
28.59167
7
0.635704
10829
2313
60.68389
7
0.56372
10830
1321
83.40703
7
0.300325
10829
2322
62.81876
7
0.22435
10830
1233
7.729514
7
0.054726
10829
2323
54.39
7
0.35928
10830
2232
8.345623
7
0.220114
10829
3311
92.34354
7
0.461426
10830
1313
68.56356
7
0.213635
10830
3113
78.90729
7
0.380806
10830
1322
70.30972
7
0.230426
10830
3121
93.8101
7
0.231744
10830
1323
62.16175
7
0.194553
10830
3122
80.9111
7
0.158111
10830
2222
27.9744
7
0.15567
10830
3112
87.11371
7
0.28043
10830
1232
16.11223
7
0.248567
10830
3111
99.61813
7
0.339405
10830
2223
19.61998
7
0.28086
10830
2122
70.20121
7
0.1364
10830
1331
63.81635
7
0.195803
10830
3131
73.8601
7
0.353376
10830
2221
40.69039
7
0.042032
10830
3123
72.38722
7
0.730463
10830
2213
25.70403
7
0.603318
10830
2111
89.54611
7
0.176062
10830
2211
47.24354
7
0.290535
10830
2112
76.79443
7
0.189098
10830
1333
42.67951
7
0.653917
10830
2121
83.26451
7
0.267939
10830
1332
50.82532
7
0.588542
10830
2123
62.16327
7
0.226129
10830
2212
34.12938
7
0.208798
10830
2131
63.65349
7
0.208887
10830
2231
21.28191
7
0.173132
10830
2132
50.82017
7
0.436155
10830
3332
53.79562
7
0.501553
10830
2133
42.61168
7
0.268116
10830
3333
45.07963
7
0.138849
10830
1111
97.40122
7
0.169438
10830
3331
66.31069
7
0.352753
10830
3132
61.32208
7
0.121381
10830
3323
64.84398
7
0.217084
10830
2113
68.32832
7
0.283481
10830
3322
73.25379
7
0.299733
10830
1131
71.42875
7
0.137415
10830
3321
86.22778
7
0.614224
10830
1112
84.71107
7
0.325278
10830
3313
71.27266
7
0.277575
10830
1121
90.86902
7
0.20009
10830
3311
92.20105
7
0.573016
10830
1113
76.55897
7
0.835328
10830
2313
60.43956
7
0.640373
10830
1123
69.97334
7
0.417367
10830
2333
34.90762
7
0.33017
10830
1132
58.44314
7
0.144604
10830
2332
42.92384
7
0.187344
10830
1133
50.43431
7
0.409837
10830
2323
54.44863
7
0.209853
10830
1122
78.14103
7
0.213604
10830
3312
79.69797
7
0.33297
10830
2233
-0.27198
7
0.32512
10831
3112
87.17229
7
0.243995
10830
2311
81.38113
7
0.49345
10831
2112
76.80838
7
0.395032
10830
2312
68.84609
7
0.875206
10831
2113
68.3733
7
0.18972
10830
2321
75.78298
7
0.366654
10831
2121
83.28729
7
0.479507
10830
2322
62.92185
7
0.225048
10831
2122
70.55648
7
0.261219
10830
2331
55.87395
7
0.392158
10831
2123
62.19016
7
0.247272
10830
3233
10.33219
7
0.191124
10831
2131
63.64033
7
0.249082
10830
3211
57.49945
7
0.255353
10831
2132
50.6995
7
0.296479
10830
3231
31.37577
7
0.529126
10831
2111
89.60247
7
0.176781
119
PlayerID
Configuration
Mu
Frequency
Sigma
PlayerID
Configuration
Mu
Frequency
Sigma
10831
3111
99.90105
7
0.459171
10831
2213
25.21279
7
0.529769
10831
3132
60.9975
7
0.134099
10831
1221
48.62773
7
0.211897
10831
3113
78.71602
7
0.274139
10831
2211
47.11778
7
0.380225
10831
3121
93.69053
7
0.212757
10831
1212
41.52864
7
0.757695
10831
3122
80.79605
7
0.281617
10831
1213
33.12627
7
0.316219
10831
3123
72.82056
7
0.430675
10831
2221
40.82144
7
0.175991
10831
3131
73.99756
7
0.367367
10831
3333
45.4242
7
0.391126
10831
3133
52.81248
7
0.156604
10831
3331
66.01542
7
0.427685
10831
1111
97.11059
7
0.225245
10831
3322
73.26866
7
0.353002
10831
2133
42.29832
7
0.277542
10831
3332
53.68726
7
0.544591
10831
1121
91.09289
7
0.317869
10831
3323
64.7581
7
0.242555
10831
1112
84.22855
7
0.265823
10831
3313
71.14196
7
0.206847
10831
1133
50.26167
7
0.33258
10831
3311
92.27003
7
0.619633
10831
1132
58.50069
7
0.140939
10831
3312
79.43226
7
0.428234
10831
1131
71.33755
7
0.23996
10831
2333
34.90447
7
0.26302
10831
1122
78.08654
7
0.17796
10831
2323
54.63933
7
0.172822
10831
1113
76.21231
7
0.76633
10831
3321
86.1577
7
0.318422
10831
1123
69.85782
7
0.191814
10832
2113
68.40362
7
0.222663
10831
3232
18.69926
7
0.225855
10832
3132
61.08162
7
0.120964
10831
3233
10.38109
7
0.247021
10832
3131
74.02567
7
0.30336
10831
2313
60.83324
7
0.389832
10832
3123
72.80607
7
0.313241
10831
2322
62.91726
7
0.419552
10832
3122
80.81831
7
0.204578
10831
3211
57.66638
7
0.320898
10832
3121
93.85092
7
0.30128
10831
2331
55.86751
7
0.478578
10832
3113
78.96974
7
0.322489
10831
2332
43.33577
7
0.358462
10832
3112
86.99945
7
0.225746
10831
1312
76.73959
7
0.188589
10832
2112
76.66304
7
0.346459
10831
3221
51.17442
7
0.30447
10832
3133
52.89075
7
0.139325
10831
3231
31.48594
7
0.379468
10832
2121
83.18633
7
0.12304
10831
2311
81.8816
7
0.51378
10832
2122
70.42481
7
0.179653
10831
3223
30.31445
7
0.270962
10832
2123
62.18343
7
0.26331
10831
2312
69.65491
7
0.982871
10832
2131
63.64576
7
0.133321
10831
3222
38.2463
7
0.195386
10832
2132
50.7137
7
0.374002
10831
3212
44.61051
7
0.35207
10832
2133
42.29165
7
0.44963
10831
3213
36.46266
7
0.787308
10832
3111
100.1509
7
0.695568
10831
1211
54.9416
7
0.106248
10832
2111
89.46871
7
0.215786
10831
1222
35.58253
7
0.087267
10832
1122
78.18839
7
0.187005
10831
1313
68.67613
7
0.254182
10832
1111
97.12239
7
0.189915
10831
1321
83.10537
7
0.347521
10832
1112
84.52212
7
0.17442
10831
1322
70.33526
7
0.405192
10832
1121
91.30848
7
0.316979
10831
1323
62.20039
7
0.106325
10832
1123
69.98372
7
0.287424
10831
1331
63.87246
7
0.193005
10832
1131
71.37551
7
0.45588
10831
1332
50.92549
7
0.336416
10832
1132
58.40558
7
0.117805
10831
1333
42.46745
7
0.406972
10832
1133
50.09111
7
0.382325
10831
1311
89.5734
7
0.23989
10832
1113
75.89634
7
0.925779
10831
1233
7.73327
7
0.031245
10832
3213
36.68833
7
0.530683
10831
1232
16.0395
7
0.271401
10832
2312
68.97422
7
1.051159
10831
2321
75.98762
7
0.514487
10832
2313
60.66234
7
0.435669
10831
1223
27.75266
7
0.280185
10832
2321
75.67904
7
0.307874
10831
2233
0.054588
7
0.21107
10832
2322
62.66666
7
0.241763
10831
2212
34.03542
7
0.489673
10832
2331
56.15659
7
0.279158
10831
2232
8.333177
7
0.249077
10832
2332
42.65186
7
0.611532
10831
2231
21.05024
7
0.20004
10832
1313
68.44726
7
0.207236
10831
2223
19.72947
7
0.562634
10832
3233
10.53975
7
0.214409
10831
2222
27.98408
7
0.137417
10832
3221
51.2957
7
0.281915
10831
1231
29.32489
7
0.586065
10832
3211
57.60323
7
0.147392
120
PlayerID
Configuration
Mu
Frequency
Sigma
PlayerID
Configuration
Mu
Frequency
Sigma
10832
3222
38.293
7
0.279266
10833
2123
62.07111
7
0.17471
10832
3212
44.80703
7
0.292171
10833
2131
63.76487
7
0.145416
10832
3223
30.13561
7
0.177837
10833
2132
50.74604
7
0.234017
10832
3231
31.11738
7
0.212423
10833
1113
76.26339
7
0.517793
10832
3232
18.78078
7
0.273744
10833
3132
61.16893
7
0.158128
10832
2311
82.03939
7
0.337814
10833
2113
68.27166
7
0.190204
10832
1213
33.62377
7
0.505646
10833
1132
58.34217
7
0.155048
10832
1322
70.27236
7
0.349613
10833
1111
97.24384
7
0.294656
10832
2233
0.065081
7
0.340894
10833
1112
84.12903
7
0.393489
10832
1212
41.74627
7
0.842067
10833
1121
91.05544
7
0.195962
10832
1221
48.67998
7
0.217601
10833
1122
78.12577
7
0.113184
10832
1222
35.78949
7
0.205518
10833
1131
71.39219
7
0.38571
10832
1223
27.23152
7
0.359362
10833
3133
52.884
7
0.156001
10832
1231
28.99931
7
0.658745
10833
1133
50.10009
7
0.472719
10832
1232
16.00144
7
0.208288
10833
1123
69.76405
7
0.241678
10832
1233
7.724811
7
0.045514
10833
3212
45.17632
7
0.600919
10832
1311
89.69058
7
0.255808
10833
2233
0.182081
7
0.191368
10832
1312
76.84608
7
0.243965
10833
2311
82.02853
7
0.477471
10832
1321
83.85058
7
0.496037
10833
2312
68.85326
7
1.144537
10832
1211
54.78781
7
0.264928
10833
2321
75.76204
7
0.15949
10832
2212
34.13162
7
0.465116
10833
2322
62.77925
7
0.221258
10832
1323
62.18919
7
0.095341
10833
2331
56.16179
7
0.19951
10832
2223
19.64114
7
0.347986
10833
3211
57.45451
7
0.162099
10832
2222
28.08657
7
0.128733
10833
3232
18.92404
7
0.241477
10832
2221
40.75434
7
0.241415
10833
3213
36.35159
7
0.557376
10832
2213
25.85545
7
0.75291
10833
3221
51.20824
7
0.364657
10832
2231
21.30451
7
0.294104
10833
3222
38.30105
7
0.400999
10832
2211
46.98158
7
0.501255
10833
3223
30.33347
7
0.358597
10832
1333
42.23854
7
0.316322
10833
3231
31.5103
7
0.248989
10832
2232
8.289994
7
0.326262
10833
3233
10.50977
7
0.172747
10832
1332
50.83936
7
0.227835
10833
2221
40.91567
7
0.149656
10832
1331
64.04377
7
0.30812
10833
1231
28.99633
7
0.697732
10832
3323
65.0787
7
0.134032
10833
2232
8.042673
7
0.311906
10832
3322
73.23531
7
0.182539
10833
1211
54.84414
7
0.194406
10832
3311
91.90997
7
0.530978
10833
1212
42.10362
7
0.863173
10832
3331
66.37592
7
0.446575
10833
1213
33.49769
7
0.571707
10832
3333
45.29214
7
0.21529
10833
1221
48.73291
7
0.107133
10832
3313
71.31025
7
0.160091
10833
2223
20.14907
7
0.228818
10832
3312
79.53823
7
0.475524
10833
1223
27.44008
7
0.390827
10832
2323
54.66835
7
0.323337
10833
1232
16.17081
7
0.454539
10832
2333
34.6482
7
0.152965
10833
1233
7.71829
7
0.071014
10832
3321
86.01485
7
0.750701
10833
1311
89.42849
7
0.351894
10832
3332
53.58568
7
0.390011
10833
1312
76.77124
7
0.216688
10833
2122
70.49502
7
0.190054
10833
1313
68.49481
7
0.254006
10833
3131
73.85473
7
0.150772
10833
1321
83.47631
7
0.343992
10833
3123
72.68801
7
0.66082
10833
1322
70.56938
7
0.226542
10833
3122
80.86396
7
0.134326
10833
2212
34.13141
7
0.428131
10833
3121
93.79636
7
0.264531
10833
1331
63.62496
7
0.353116
10833
3113
78.79256
7
0.385325
10833
1332
50.82781
7
0.311623
10833
3112
87.13972
7
0.239028
10833
1333
42.62422
7
0.675669
10833
3111
99.70079
7
1.091695
10833
2231
21.1088
7
0.121161
10833
2111
89.47625
7
0.275939
10833
1323
62.25266
7
0.166031
10833
2112
77.15942
7
0.238663
10833
2211
46.84528
7
0.350518
10833
2121
83.35256
7
0.299531
10833
2222
27.95342
7
0.102299
10833
2133
42.41301
7
0.194516
10833
1222
35.77136
7
0.142757
121
PlayerID
Configuration
Mu
Frequency
Sigma
PlayerID
Configuration
Mu
Frequency
Sigma
10833
2213
25.96627
7
0.792677
10834
1331
63.6641
7
0.098315
10833
3321
85.94551
7
0.535432
10834
1222
35.62074
7
0.116474
10833
3322
73.2008
7
0.319693
10834
1321
83.31855
7
0.677845
10833
3331
66.43243
7
0.219449
10834
1221
48.53687
7
0.280819
10833
3311
92.38447
7
0.660527
10834
1312
76.62685
7
0.205917
10833
3332
53.43254
7
0.524678
10834
1311
89.65325
7
0.167469
10833
3323
65.03515
7
0.262094
10834
1213
33.47576
7
0.882419
10833
3312
79.82158
7
0.250803
10834
1232
15.89561
7
0.243786
10833
2313
60.76944
7
0.472018
10834
1211
54.75498
7
0.232955
10833
2323
54.30314
7
0.250908
10834
1223
27.49401
7
0.505131
10833
3333
45.44838
7
0.294586
10834
2221
40.86389
7
0.151335
10833
2332
43.25981
7
0.246818
10834
3211
57.69107
7
0.344703
10833
2333
34.8617
7
0.227242
10834
2133
42.44379
7
0.130485
10833
3313
71.29952
7
0.14146
10834
2211
47.27825
7
0.224434
10834
2132
50.47305
7
0.290073
10834
2212
34.25534
7
0.491531
10834
2131
63.75272
7
0.284692
10834
2213
26.34719
7
0.845452
10834
2123
62.33169
7
0.30957
10834
1233
7.725874
7
0.039263
10834
2122
70.2387
7
0.35904
10834
3312
79.20107
7
0.439196
10834
2121
83.67291
7
0.488455
10834
3332
53.62585
7
0.478848
10834
2113
68.27001
7
0.190733
10834
3313
71.18868
7
0.295293
10834
2112
76.4726
7
0.339104
10834
3321
85.95688
7
0.417492
10834
2111
89.56639
7
0.222745
10834
3322
73.36442
7
0.238717
10834
1132
58.42755
7
0.168134
10834
3323
64.88466
7
0.242456
10834
1133
49.85813
7
0.23641
10834
3331
66.40392
7
0.305236
10834
1123
69.88329
7
0.368646
10834
3333
45.28919
7
0.212368
10834
1122
78.18725
7
0.169759
10834
3311
92.34938
7
0.588727
10834
1121
91.06208
7
0.228141
10834
2321
75.69878
7
0.479295
10834
1131
71.269
7
0.275228
10834
1313
68.46498
7
0.340697
10834
1112
84.34074
7
0.375115
10834
2311
81.79343
7
0.644058
10834
1113
76.07886
7
0.342506
10834
2312
68.93652
7
0.812724
10834
1111
97.19336
7
0.252561
10834
2313
60.58484
7
0.28656
10834
3121
93.62454
7
0.287439
10834
2322
62.57677
7
0.178891
10834
3132
61.28121
7
0.169288
10834
2323
54.49472
7
0.29715
10834
3233
10.3918
7
0.2704
10834
1323
62.25024
7
0.097494
10834
2232
8.09371
7
0.176603
10834
2332
43.09377
7
0.242704
10834
2231
21.08036
7
0.237552
10834
2333
34.90591
7
0.384819
10834
2223
19.62078
7
0.201046
10834
1322
70.60084
7
0.409037
10834
2233
0.14219
7
0.25399
10834
1332
50.752
7
0.355351
10834
2222
27.86845
7
0.216297
10834
2331
56.17172
7
0.353474
10834
3111
100.1123
7
0.718791
10834
1333
42.46641
7
0.399426
10834
3113
78.81596
7
0.321198
10835
2131
63.72573
7
0.129522
10834
3122
80.83402
7
0.264625
10835
1111
97.19196
7
0.309009
10834
3123
72.69082
7
0.494675
10835
2122
70.46117
7
0.462251
10834
3131
73.85627
7
0.294792
10835
2121
83.51424
7
0.384618
10834
3133
52.91208
7
0.197442
10835
2112
76.79217
7
0.316283
10834
3212
44.29851
7
0.348922
10835
2132
50.71526
7
0.436743
10834
3213
36.39924
7
0.744555
10835
2111
89.49915
7
0.211209
10834
3221
51.32401
7
0.331497
10835
1122
78.15615
7
0.132919
10834
3222
38.46995
7
0.264423
10835
2113
68.43544
7
0.220246
10834
3223
30.23315
7
0.233302
10835
1133
49.91835
7
0.646261
10834
3231
31.39133
7
0.421542
10835
1132
58.45909
7
0.090398
10834
3232
18.34314
7
0.173939
10835
1123
69.92408
7
0.31911
10834
3112
86.97728
7
0.231497
10835
1121
90.83871
7
0.29813
10834
1231
28.90641
7
0.709323
10835
1113
76.34321
7
0.793536
10834
1212
42.29036
7
0.80796
10835
1112
84.49454
7
0.350642
122
PlayerID
Configuration
Mu
Frequency
Sigma
PlayerID
Configuration
Mu
Frequency
Sigma
10835
1131
71.20642
7
0.353861
10835
1323
62.24549
7
0.104837
10835
2233
0.029805
7
0.315164
10835
2332
43.44515
7
0.298474
10835
3113
78.57753
7
0.181977
10835
2333
34.75154
7
0.274637
10835
3111
99.79826
7
0.593261
10835
2312
69.13031
7
0.72645
10835
3222
38.59023
7
0.247788
10835
2313
60.94599
7
0.678969
10835
2232
8.211241
7
0.268513
10835
2321
75.83721
7
0.480822
10835
2231
21.16143
7
0.260698
10835
2322
62.83502
7
0.25058
10835
2223
19.54255
7
0.378813
10835
1313
68.50339
7
0.179811
10835
3112
87.15465
7
0.146613
10835
2323
54.59581
7
0.301123
10835
3212
44.67289
7
0.409404
10835
2331
56.20111
7
0.402349
10835
3233
10.48446
7
0.273738
10835
1333
42.52949
7
0.32294
10835
1311
89.92522
7
0.435716
10835
2222
28.02773
7
0.148226
10835
3232
18.55074
7
0.289976
10835
3231
31.89949
7
0.34886
10835
3213
36.41008
7
0.616274
10835
3221
51.46614
7
0.193247
10835
3121
93.8832
7
0.221719
10835
3211
57.66559
7
0.278383
10835
3133
52.78071
7
0.108418
10835
3132
61.17678
7
0.195756
10835
3131
73.72182
7
0.33517
10835
3123
72.24507
7
0.254928
10835
3122
80.89652
7
0.244182
10835
3223
30.24506
7
0.345469
10835
1321
83.1899
7
0.366674
10835
1211
54.85667
7
0.19959
10835
1212
41.45912
7
0.56364
10835
1213
33.59543
7
1.299008
10835
1221
48.53083
7
0.311036
10835
1222
35.69783
7
0.21123
10835
1223
27.52718
7
0.398143
10835
1231
28.92846
7
0.613124
10835
1232
16.06236
7
0.185072
10835
1233
7.728351
7
0.053098
10835
1312
76.98637
7
0.143635
10835
2221
40.92842
7
0.105723
10835
1331
63.84923
7
0.38561
10835
2123
62.4187
7
0.328936
10835
2133
42.59919
7
0.295678
10835
2213
26.11802
7
0.442408
10835
2212
34.40497
7
0.271632
10835
2211
46.96041
7
0.362813
10835
1322
70.39979
7
0.284561
10835
3332
53.53088
7
0.426573
10835
3331
66.3836
7
0.413966
10835
3323
64.93469
7
0.33279
10835
3322
73.08386
7
0.303778
10835
3321
86.14012
7
0.522964
10835
3313
71.17992
7
0.303282
10835
3312
79.5723
7
0.666377
10835
3311
92.88952
7
0.464714
10835
3333
45.27531
7
0.204391
10835
2311
81.92651
7
0.459555
10835
1332
50.82785
7
0.580373
C.1.2
Adaptive One At A Time
The Adaptive One At A Time players may
not list their configurations in order. This
is because they store their flights in local
memory, and put them into the database
with a Java Iterator which does not guarantee order. However, it should be possible to recreate the path along with they
optimized response.
123
PlayerID
Configuration
Mu
Frequency
Sigma
PlayerID
Configuration
Mu
Frequency
Sigma
26590
2212
34.19665
7
0.519765
26602
1222
30.90775
7
2.714101
26590
2222
28.10085
7
0.226637
26602
1312
60.45085
7
2.409952
26590
2232
8.464541
7
0.212538
26602
1321
84.62143
7
8.885429
26590
2211
46.92297
7
0.404632
26602
1322
67.19152
7
3.88737
26590
2213
25.84148
7
0.569421
26602
1323
51.90408
7
0.946307
26590
3111
100.0967
7
0.655665
26602
1332
28.42597
7
4.235913
26590
3211
57.5896
7
0.161021
26602
3321
79.27343
7
2.914844
26590
3311
92.44719
7
0.418913
26605
1221
45.98021
7
0.941654
26590
1211
54.80807
7
0.174184
26605
1311
83.06292
7
1.916766
26593
1312
61.104
7
0.201656
26605
1321
85.32993
7
5.175007
26593
3131
55.43405
7
0.246968
26605
1322
67.47119
7
2.562254
26593
3121
90.89119
7
0.199314
26605
1323
53.32884
7
0.636825
26593
3113
70.71259
7
0.253346
26605
1331
44.06813
7
2.275396
26593
3112
88.8697
7
0.281097
26605
2311
65.37017
7
3.71897
26593
3111
78.69616
7
0.375592
26605
3311
34.85719
7
2.800078
26593
2112
84.66041
7
0.338924
26605
1121
59.22335
7
1.518183
26593
1112
67.99615
7
0.242706
26608
1311
90.02925
7
12.96746
26593
1212
28.8151
7
0.672718
26608
1322
71.31015
7
18.92207
26596
3223
34.49221
7
0.163734
26608
1313
73.01483
7
8.06266
26596
3212
25.34481
7
0.296593
26608
1312
80.58723
7
8.076769
26596
3321
73.66032
7
0.127792
26608
1122
79.03247
7
6.862447
26596
3232
27.74172
7
0.156968
26608
1332
70.88285
7
24.04639
26596
1121
59.47275
7
0.17884
26608
2122
69.59454
7
17.44098
26596
3121
84.14812
7
0.106297
26608
3122
76.65061
7
13.89588
26596
2121
66.26704
7
0.218082
26608
1222
37.63453
7
12.85921
26596
3222
28.95659
7
0.110123
26611
2312
41.717
7
41.13201
26596
3221
44.3419
7
0.112176
26611
3322
63.03962
7
11.73747
26599
2131
63.68947
7
2.454898
26611
2332
19.68985
7
20.21053
26599
3121
93.55434
7
3.469804
26611
2323
49.97717
7
10.24318
26599
3331
66.60938
7
4.232919
26611
1322
66.08525
7
11.54335
26599
3131
72.55791
7
2.831388
26611
1222
25.78537
7
11.46969
26599
3111
103.3269
7
4.660926
26611
1122
62.20121
7
7.462624
26599
2133
41.71143
7
1.825812
26611
2322
34.39164
7
13.89453
26599
2132
52.06717
7
2.577272
26611
2321
63.98955
7
20.08557
26599
1131
69.82673
7
1.910776
26614
3121
86.8485
7
9.007366
26599
3231
27.80773
7
6.182551
26614
3131
48.49215
7
7.218236
26602
2321
58.78123
7
1.63597
26614
3122
64.8474
7
8.678333
26602
1122
55.81358
7
2.09833
26614
3111
52.71932
7
10.68481
26614
2122
28.73908
7
9.368227
26614
1322
68.64058
7
15.54882
26614
1222
47.18818
7
4.566915
26614
1122
43.21876
7
5.933681
26614
3123
68.0132
7
6.043964
26617
3322
73.29396
7
0.386114
26617
1322
70.35325
7
0.19215
26617
2322
62.83205
7
0.369178
26617
3111
100.2234
7
0.684279
26617
3112
86.94305
7
0.211954
26617
3113
78.82093
7
0.457979
26617
3122
80.8174
7
0.207597
26617
3132
61.25669
7
0.201653
26617
3222
38.54138
7
0.326939
124
C.1.3
Orthogonal Array
Each of the following orthogonal array agents
will execute an orthogonal array of order
two. The experimental design is created
swapping parameters and levels in the orthogonal array in section 5.5.5
125
PlayerID
Configuration
Mu
Frequency
Sigma
10946
1221
48.60667
7
0.219172
10946
1333
42.30141
7
0.341348
10946
3123
72.31511
7
0.370889
10946
3232
18.77591
7
0.288191
10946
3311
92.3299
7
0.702735
10946
1112
84.55836
7
0.557416
10946
2131
63.69005
7
0.195152
10946
2213
25.86446
7
1.035251
10946
2322
62.48865
7
0.138834
10947
1211
54.63951
7
0.163444
10947
3331
66.47012
7
0.193775
10947
3223
30.18015
7
0.210722
10947
3112
87.15736
7
0.304219
10947
2313
60.9307
7
0.593196
10947
2232
8.270327
7
0.307079
10947
1322
70.56702
7
0.275226
10947
1133
50.12974
7
0.32591
10947
2121
83.32415
7
0.366318
10948
2322
62.95481
7
0.276815
10948
1333
42.64687
7
0.460137
10948
3311
92.52408
7
0.416048
10948
3232
18.78873
7
0.486353
10948
2213
26.07395
7
0.9538
10948
1112
84.37748
7
0.330648
10948
1221
48.53873
7
0.336891
10948
3123
72.73286
7
0.610879
10948
2131
63.7177
7
0.083133
10949
2212
33.94948
7
0.518402
10949
3311
92.70265
7
0.146939
10949
3223
30.0827
7
0.229913
10949
2333
34.91587
7
0.212555
10949
2121
83.24281
7
0.274399
10949
1322
70.40214
7
0.196062
10949
1231
29.20707
7
0.447071
10949
1113
76.23515
7
0.654299
10949
3132
61.13782
7
0.111627
10950
3211
57.49271
7
0.232589
10950
1223
27.44253
7
0.374749
10950
1312
76.68832
7
0.21716
10950
2113
68.31138
7
0.147938
10950
2232
8.243985
7
0.324666
10950
3122
80.89701
7
0.049578
10950
3333
45.09368
7
0.399337
10950
2321
75.41604
7
0.301748
10950
1131
71.29367
7
0.207039
10951
1212
41.70221
7
0.911915
10951
1331
63.54337
7
0.30642
10951
2132
50.79972
7
0.309857
10951
2221
40.77909
7
0.157527
10951
2313
60.86375
7
0.759014
10951
3111
100.0469
7
1.012175
10951
3233
10.43045
7
0.34522
10951
3322
73.23975
7
0.399345
10951
1123
69.75229
7
0.312474
PlayerID
Configuration
Mu
Frequency
Sigma
10952
1131
71.12201
7
0.301926
10952
3332
53.33533
7
0.603991
10952
3221
51.11489
7
0.288656
10952
3113
78.96729
7
0.33525
10952
1212
41.93867
7
0.472821
10952
1323
62.31452
7
0.234483
10952
2233
-0.00602
7
0.304873
10952
2311
82.18706
7
0.390574
10952
2122
70.41951
7
0.124714
10953
1331
63.79408
7
0.266561
10953
3211
57.37337
7
0.200819
10953
3133
52.98102
7
0.112555
10953
3322
73.25708
7
0.20161
10953
2313
60.72863
7
0.485394
10953
1223
27.61456
7
0.271328
10953
1112
84.29851
7
0.445592
10953
2232
8.271014
7
0.299866
10953
2121
83.4801
7
0.353013
10954
3121
93.79588
7
0.301625
10954
3213
36.15413
7
0.613488
10954
2323
54.48621
7
0.346442
10954
2112
76.64522
7
0.242826
10954
1311
89.58607
7
0.281955
10954
1222
35.84078
7
0.397913
10954
1133
50.406
7
0.421711
10954
3332
53.4614
7
0.544836
10954
2231
21.19276
7
0.18202
10955
3311
92.52495
7
0.455538
10955
1121
91.15273
7
0.282324
10955
1213
33.22885
7
0.647565
10955
1332
50.68105
7
0.348434
10955
2112
76.55203
7
0.312418
10955
2231
21.30874
7
0.322138
10955
2323
54.63653
7
0.325799
10955
3133
52.93525
7
0.117066
10955
3222
38.24516
7
0.198092
126
C.2
Performance of the Strategies
Listed below are the results of listing 5.5. Each instance of the parabolic form of each
of the strategies has some payoff value (payoff equals 1 at the target [80] and decays
parabolically to 0 at the minimum of means [0]) and wealth. The wealth is determined
by the payoff count, the game’s total payoff, and the number of experiments the
strategy ran.
C.2.1
Interpreting this Table
This table displays the effects of all the varied parameters for the ParabolicDistanceGoal, target = 80 games. The ’number of payoffs’ for these agents is not integral.
For every flight of distance x, they get some payoff p of
p(x) = 1 − (80 − x)2 /(80 − 0)2
(C.1)
which 1 at a scaled length of 80 and less everywhere else. The coefficient was chosen
such that p for the mean of any configuration would be zero or greater, making
negative results relatively rare.
As each player gets three tries t at the target, every attempt has a possible payout
P/3. Every agent a has an experimental cost ca determined by its strategy, so ends
up with a wealth wa of
wa = −ca +
3
pX
p(xt )
3 t=1
(C.2)
The first and second statistical moments are displayed in the table. While changing values of payoff should not effect the performance of a strategy, they will balance
the experimental cost in different ways and have a direct effect on the wealth.
127
128
Frequency
400
400
400
200
400
400
400
200
1400
400
400
200
400
400
400
200
400
400
400
200
1400
400
400
200
400
400
400
200
400
400
400
200
1400
400
400
200
400
Payoff
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
567
2
7
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
2
2
2
2
2
2
TrialCount
Noise
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
Nonlinearity
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
Strategy
476.3025
-5.555
-6.215
-5.095
-2.865
-6.31
-5.9175
-5.1525
-3.3175
-7.15
-7.1725
-9.375
-7.0525
19.015
19.765
19.5575
22.39714286
20.51
19.48
19.4325
22.4625
17.915
17.835
15.8525
17.4925
36.89
36.625
36.96
39.16
36.765
35.7975
37.43
39.1225
34.89
34.885
32.155
34.375
µW ealth
96.84720437
7.811976382
8.982693082
10.43819788
6.668769698
9.067188098
9.309709649
10.05928644
6.960365921
9.817713583
9.729991971
13.45843137
11.14561545
8.121870166
8.155045984
9.798045405
5.857058536
7.794863693
8.565313771
10.73896847
5.492594446
8.875684481
9.30874723
13.7965845
11.11710141
8.521613697
7.664814088
8.679769582
6.095089358
6.758681454
9.118744088
8.459024766
6.276742288
9.280511839
9.348089377
14.47483938
12.41871068
σW ealth
4.918082077
-0.71108766
-0.69188605
-0.48811108
-0.42961448
-0.69591586
-0.6356267
-0.51221327
-0.47662724
-0.72827547
-0.73715374
-0.69658935
-0.63276003
2.34120955
2.423652796
1.996061377
3.82395749
2.63121984
2.274289129
1.80953134
4.089597406
2.018435878
1.915939874
1.149016266
1.573476696
4.328992291
4.778328552
4.258177553
6.42484428
5.439670481
3.925705081
4.424859961
6.232930747
3.759490921
3.731778612
2.221440885
2.768000711
ST NW ealth
Table C.1: Wealth and Payoff for players; ParabolicDistanceGoal, Target = 80
µP ayof f
2.62363458
2.67504289
2.64891529
2.71657199
2.82329195
2.64153377
2.66506928
2.71209731
2.80004863
2.60110788
2.60007
2.49472974
2.61011981
2.63489766
2.68011732
2.68510038
2.82789197
2.71830791
2.66246956
2.6797797
2.83179582
2.58340687
2.57726554
2.48707621
2.57090649
2.6752263
2.67333297
2.70467926
2.81770641
2.67150466
2.63042875
2.72715265
2.81213163
2.5837261
2.58430361
2.45217077
2.56238783
σP ayof f
0.513582552
0.381883033
0.438102414
0.513337095
0.333833055
0.441422933
0.453480766
0.493288983
0.345320874
0.476233939
0.470253021
0.648333149
0.539500023
0.424653347
0.424970903
0.512800285
0.313359291
0.402366796
0.441940656
0.553156383
0.291555118
0.453913598
0.476115982
0.687866154
0.561994708
0.442855816
0.401874046
0.456065235
0.323378895
0.353648475
0.47026118
0.437796115
0.331597686
0.471567179
0.471499567
0.72405687
0.622323784
5.10849633
7.00487493
6.04633803
5.29198458
8.45719711
5.98413352
5.87691801
5.49798881
8.10854148
5.46182803
5.52908729
3.8479133
4.83803466
6.2048202
6.30659018
5.23615227
9.02443951
6.75579581
6.02449566
4.84452459
9.7127289
5.69140665
5.41310445
3.61563975
4.57460979
6.04085168
6.65216627
5.93046576
8.71332811
7.55412464
5.59354856
6.22927558
8.48055265
5.47902019
5.48103071
3.38671017
4.11745123
ST NP ayof f
129
Frequency
400
400
200
400
400
400
200
400
400
400
200
400
400
400
200
400
400
400
200
400
400
400
200
400
400
400
200
400
400
400
200
400
400
400
200
400
400
200
Payoff
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
3000
3000
3000
2
2
2
7
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
2
2
2
2
2
TrialCount
Noise
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
Nonlinearity
Strategy
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
2561.58
2418.865
2559.41
442.89
441.1175
438.6625
464.245
443.125
440.005
454.365
465.3275
421.19
431.5825
415.425
430.245
470.72
469.2175
470.285
497.35
472.475
462.46
471.345
494.4125
463.155
437.205
434.46
461.015
487.405
495.7975
500.1275
514.3975
478.305
484.17
485.575
508.535
472.805
464.6675
445.09
µW ealth
σW ealth
470.3871635
660.6695935
550.2143191
73.45902191
84.38426212
103.8806459
67.44456965
77.39579688
83.83128876
78.41719693
59.63275311
94.02677225
74.93365862
113.3271564
100.7834559
71.78838068
69.51510767
86.47877644
59.21952803
74.00161738
82.11119534
89.84089255
58.89072375
75.46867546
111.1768095
121.4493244
84.01145621
69.14087774
61.91454994
81.84101199
62.00281843
84.47929909
84.03366052
99.6516401
68.26517249
85.07771139
104.7097748
160.7063841
5.445684319
3.661232519
4.65166011
6.029075646
5.227485421
4.222754837
6.88335625
5.725440112
5.248696597
5.794200989
7.803220139
4.479468878
5.759527934
3.665714496
4.269004236
6.557049979
6.749863674
5.43815511
8.398412087
6.384657751
5.632118715
5.246441644
8.395422378
6.13704954
3.932519757
3.577294498
5.487525402
7.049447677
8.007770395
6.110964269
8.29635673
5.661801236
5.761619773
4.872724619
7.449406212
5.557330966
4.437670702
2.769585057
ST NW ealth
µP ayof f
2.58116897
2.43836926
2.57893256
2.68580218
2.67544655
2.66373053
2.79882785
2.68625359
2.66970579
2.74682221
2.80426845
2.56989102
2.62498628
2.53929052
2.6179536
2.68974609
2.68105537
2.68784416
2.83104306
2.69892549
2.64576266
2.69344843
2.81578157
2.64916238
2.51199368
2.49755792
2.63820822
2.68270269
2.72672671
2.75103341
2.82630733
2.63423506
2.66510351
2.67354263
2.79492152
2.6051544
2.56198407
2.45817791
σP ayof f
0.470455696
0.660845665
0.550421557
0.389650264
0.447096808
0.552075146
0.357612671
0.410181468
0.444562418
0.415875787
0.31634861
0.499166077
0.397334669
0.601130391
0.534643741
0.380819992
0.367935611
0.458682133
0.314322893
0.392086307
0.43522616
0.476431043
0.312237236
0.399936592
0.589655176
0.643879133
0.445439239
0.366464239
0.327967395
0.434225717
0.328687323
0.447654639
0.445310172
0.528344749
0.361751849
0.451092481
0.555092198
0.852125158
5.48652932
3.68977114
4.6853771
6.89285347
5.98404304
4.82494196
7.8264225
6.54893943
6.005244
6.60491015
8.86448797
5.14836872
6.60648689
4.22419255
4.89663191
7.06303805
7.28675151
5.85992776
9.00679881
6.88349847
6.07905247
5.65338567
9.01808384
6.62395598
4.26010621
3.87892354
5.92271175
7.32050335
8.31401764
6.33549166
8.59877193
5.88452534
5.98482514
5.06022371
7.7260739
5.77521133
4.61542079
2.88476157
ST NP ayof f
130
Frequency
400
400
400
200
400
400
400
200
400
400
200
400
400
400
200
400
400
400
200
400
400
200
400
400
400
200
400
400
400
200
720
240
240
240
200
280
280
240
Payoff
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
63
63
63
63
63
63
63
63
2
2
2
2
2
2
2
2
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
2
2
TrialCount
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
Noise
0.527051027
0.1
0
0.8
0.527051027
0.1
0
1.7977E+308
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
Nonlinearity
Strategy
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
AdaptiveOneAtATimeParabolic
-99.0041667
-99.0071429
-99.025
-100.145
-100.975
-102.479167
-102.254167
-99.9708333
2650.26
2598.2025
2653.095
2757.0525
2609.965
2599.025
2696.5775
2764.4525
2490.155
2462.325
2527.755
2652.065
2623.9525
2669.935
2794.57
2654.175
2637
2660.745
2777.765
2540.4
2409.265
2552.415
2567.775
2686.09
2669.51
2795.5625
2669.675
2686.68
2674.9775
2791.755
µW ealth
σW ealth
0.064415103
0.084213044
0.23010091
1.908919852
3.350279839
5.490862043
6.908779388
3.726106752
334.5310335
455.0747647
498.6711301
330.2867311
328.9568418
431.3485243
418.2387464
279.8372701
539.6477101
701.7661572
557.3101515
355.8488595
445.0630408
520.5337028
339.8408379
341.5968594
437.1715224
532.1510828
312.4615013
484.8968859
675.4838153
625.5024682
532.6394225
413.4654543
500.6959156
340.4722398
346.9817133
347.0739152
541.4255138
330.2423958
-1536.97132
-1175.67468
-430.354666
-52.4616054
-30.1392734
-18.6635843
-14.8006125
-26.8298361
7.922314328
5.709397008
5.320330053
8.347451594
7.934065106
6.025348074
6.44745979
9.878785978
4.614408536
3.508754269
4.53563423
7.452784882
5.895687262
5.129225995
8.223172993
7.769904572
6.031957401
4.99998043
8.889943205
5.239052
3.566724983
4.080583418
4.820850451
6.49652824
5.331599314
8.21083828
7.693993365
7.740944745
4.94061959
8.453654151
ST NW ealth
µP ayof f
2.98695531
2.99016041
2.98981249
2.90281027
2.86122315
2.7816386
2.79377746
2.93378972
2.71471676
2.66272178
2.71785457
2.82163547
2.67452996
2.66353655
2.76118564
2.82904827
2.55456867
2.52678239
2.59233892
2.68960538
2.66148654
2.70777213
2.83217426
2.69172692
2.67455135
2.69832986
2.81537941
2.57787841
2.44677236
2.58991377
2.58723575
2.70559755
2.68929207
2.81518326
2.68923209
2.70625057
2.6945721
2.81137996
σP ayof f
0.011121619
0.012768112
0.017801708
0.097871026
0.167165642
0.268928109
0.335776302
0.189179608
0.334482666
0.455176999
0.49889514
0.330330726
0.329015936
0.43146534
0.41835193
0.279928888
0.539852356
0.701990504
0.55744793
0.355842454
0.445149597
0.520747595
0.339902644
0.341685987
0.437285868
0.532319937
0.312536326
0.485104035
0.675688557
0.625655902
0.532869294
0.41354223
0.500885657
0.340551499
0.347068543
0.347082488
0.541611586
0.330272338
268.571993
234.189713
167.950877
29.6595469
17.1160959
10.3434283
8.32035331
15.5079596
8.11616578
5.84986013
5.44774714
8.5418499
8.12887665
6.17323409
6.60015036
10.1063105
4.73197652
3.5994538
4.65036961
7.55841625
5.9788587
5.19977847
8.33231017
7.87777967
6.1162538
5.06900018
9.00816698
5.31407332
3.62115406
4.13951785
4.85529149
6.54249397
5.36907382
8.26654197
7.74841782
7.79713947
4.97510055
8.51230827
ST NP ayof f
131
240
240
240
200
280
280
240
200
1280
280
280
200
360
240
240
240
200
240
240
63
63
63
63
63
63
63
63
63
63
63
567
567
567
567
567
567
567
240
63
63
280
63
720
280
63
63
200
63
200
240
63
63
240
63
280
240
63
63
720
63
280
200
63
63
280
63
200
280
63
1280
1280
63
63
200
63
63
Frequency
Payoff
2
2
2
2
2
2
2
7
7
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
TrialCount
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
100
100
100
100
10
Noise
0.1
0
0.8
0.527051027
0.1
0
1.7977E+308
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
1.7977E+308
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
1.7977E+308
0.8
0.527051027
0.1
0
0.8
Nonlinearity
Strategy
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
400.7625
401.1458333
385.74
378.0833333
363.5791667
360.725
390.5611111
-504
-504
-504
-504
-504
-504
-504
-504
-504.38
-504.820833
-505.941667
-505.4
-504.433333
-261
-261
-261
-261
-261
-261
-261.003571
-261
-261.585
-262.170833
-263.404167
-263.05
-261.501389
-99
-99
-99
-99
-99
µW ealth
2.621595014
2.051316481
24.37462615
30.30761272
46.71895118
57.65334083
31.97910149
0
0
0
0
0
0
0
0
0.797245257
1.086461829
2.590996698
2.658006772
1.749920633
0
0
0
0
0
0
0.059654618
0
1.316349118
2.250921879
3.529515921
3.025860759
1.443375005
0
0
0
0
0
σW ealth
152.8697216
195.5553114
15.82547349
12.47486355
7.782263032
6.256792665
12.21301078
-632.653497
-464.646635
-195.269128
-190.14248
-288.260692
-4375.24503
-198.720078
-116.472649
-74.6289782
-86.9339408
-181.173561
ST NW ealth
µP ayof f
2.98941121
2.99128497
2.90700248
2.86619062
2.78881039
2.7737703
2.93527203
2.99780184
2.99937156
2.99987537
2.99974065
2.99546546
2.99148535
2.99625768
2.9961965
2.94515923
2.91767876
2.85911194
2.89248767
2.96276033
2.99785707
2.99942802
2.99985849
2.9997255
2.99495592
2.98945554
2.99387245
2.99417776
2.93616751
2.90205125
2.83656857
2.85654692
2.95854764
2.99782265
2.9993801
2.99981298
2.9997066
2.99335242
σP ayof f
0.01512462
0.011681085
0.12985362
0.161133095
0.248411023
0.306284565
0.171119237
0.000461392
0.000388759
6.79129E-05
0.000160268
0.003461784
0.006301019
0.003835565
0.004622597
0.046171356
0.060800979
0.131175611
0.134582083
0.094207114
0.000425703
0.000328385
0.000105357
0.000168727
0.004794955
0.008616128
0.010119251
0.007466256
0.071732424
0.114617036
0.175462549
0.154341755
0.084507306
0.000411179
0.00034316
0.000214824
0.000190741
0.007111573
197.651983
256.079382
22.3867651
17.7877215
11.2265968
9.05618703
17.1533726
6497.30058
7715.25317
44172.3916
18717.0165
865.295389
474.762128
781.177701
648.16305
63.7875838
47.9873648
21.7960633
21.492368
31.4494332
7042.14079
9133.86338
28473.2622
17778.5606
624.605682
346.960438
295.85911
401.028007
40.932222
25.3195454
16.1662337
18.5079334
35.0093713
7290.79049
8740.47214
13964.0365
15726.5577
420.912853
ST NP ayof f
132
Frequency
240
200
240
240
240
200
150
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
40
40
40
40
40
40
40
Payoff
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
972
972
972
972
972
972
972
4
4
4
4
4
4
4
7
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
TrialCount
100
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
100
100
100
100
10
10
Noise
0
0.527051027
0.1
0
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
1.7977E+308
0.8
0.527051027
0.1
0
0.8
0.527051027
Nonlinearity
Strategy
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
646.05
644.3
645.125
644.2
601.35
581.75
601.525
0
0
0
0
-0.105
-0.7125
-0.12083333
-0.17083333
-10.86
-13.3375
-22.7625
-15.7416667
241
241.0041667
241.0166667
241.0125
240.905
240.1541667
240.4291667
240.6125
228.83
226.4958333
211.7666667
217.1791667
240.9666667
402
402.0083333
402.025
402.0208333
401.56
400.9125
µW ealth
σW ealth
0.217944947
1.913112647
1.552216158
3.05122926
100.6224503
92.20486701
42.94647104
0
0
0
0
0.306553421
1.315235245
0.454128439
0.532274339
12.15279392
12.36555405
26.08651172
16.54891529
0
0.064415103
0.128019096
0.11110243
0.382066748
1.401332451
1.867257161
1.030700611
18.23406428
13.44029945
27.14950378
26.32911062
0.354338194
0
0.090905934
0.15612495
0.142826138
1.013114011
1.817922922
2964.280697
336.781005
415.6154391
211.1280225
5.976300501
6.309319875
14.00638948
-0.34251779
-0.54172818
-0.26607744
-0.32094978
-0.89362167
-1.07860108
-0.87257738
-0.95122045
3741.423264
1882.661842
2169.281982
630.5311865
171.3755836
128.7606076
233.4455781
12.54958832
16.85199308
7.800019786
8.248632846
680.0471158
4422.24522
2575.020841
2814.756741
396.3621029
220.5332774
ST NW ealth
µP ayof f
2.99970702
2.99129848
2.99529526
2.99154167
2.85814922
2.7980277
2.85853348
2.9978092
2.99938172
2.99987096
2.99975062
2.99570884
2.99111857
2.99636827
2.99609186
2.93618743
2.92250757
2.87252098
2.90972136
2.99777706
2.99935997
2.99986292
2.99969524
2.9955078
2.99003297
2.99386938
2.99438336
2.92906309
2.91615787
2.83782743
2.86681673
2.99881662
2.99777125
2.9994044
2.99982006
2.99970038
2.99324905
2.98888085
σP ayof f
0.000162138
0.006424679
0.005200894
0.009760888
0.31101822
0.284975352
0.132706428
0.00048549
0.000352148
7.17655E-05
0.000174764
0.002822918
0.007566641
0.003833669
0.003806988
0.064471457
0.065946765
0.138099081
0.087989119
0.000456597
0.000355399
0.000102284
0.000185598
0.003429336
0.008127034
0.010740175
0.006702987
0.097237962
0.071618641
0.144702298
0.140811179
0.002679595
0.000449691
0.00035501
0.000208166
0.000186167
0.006592456
0.010221742
18500.9899
465.595027
575.919341
306.48252
9.18965206
9.81849017
21.5402789
6174.80855
8517.39224
41801.0207
17164.5746
1061.20984
395.303359
781.592838
786.998049
45.5424395
44.3161625
20.8004351
33.0691043
6565.47157
8439.41782
29328.8406
16162.3683
873.494966
367.911952
278.754234
446.723749
30.1226292
40.717861
19.6114883
20.3592979
1119.13071
6666.29479
8448.7835
14410.7115
16112.9553
454.041593
292.404249
ST NP ayof f
133
Frequency
40
40
240
240
40
240
240
240
200
240
240
240
200
240
240
40
240
240
240
200
240
240
240
200
240
113
40
240
240
240
200
240
240
240
200
360
240
240
Payoff
972
972
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
63
63
63
2
2
2
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
2
2
2
2
2
4
4
TrialCount
1.9371775
1.9371775
0
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
Noise
0.1
0
1.7977E+308
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.527051027
0.1
Nonlinearity
Strategy
Guess
Guess
Guess
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
FullFactorialParabolic
38.42916667
45.6875
46.56944444
2429.745
2429.995833
2430.016667
2430.041667
2426.44
2422.554167
2426.629167
2427.620833
2327.5
2314.345133
2336.8
2672.81
2673.004167
2673.033333
2673.016667
2668.795
2664.5875
2668.404167
2668.691667
2591.875
2497
2526.975
2834.62
2834.991667
2835.033333
2835.0125
2828.05
2823.945833
2825.616667
2825.375
2721.15
2607.595833
2612.683333
646
646.025
µW ealth
σW ealth
22.79499176
17.10282171
14.4399392
0.583073752
0.193604336
0.128019096
0.199826313
3.526528038
6.61037563
6.338373817
2.990607403
106.3628225
108.1315791
99.47035572
0.483632092
0.193604336
0.179505494
0.128019096
5.626986316
7.657284794
8.758280804
6.55400111
67.85948257
243.729341
205.450905
0.682348884
0.223451461
0.2013841
0.11110243
10.0407918
11.41751137
12.92393344
20.26822739
128.0764518
223.2707045
261.8619065
0
0.15612495
1.685860082
2.671342821
3.225044358
4167.131504
12551.35027
18981.67341
12160.76914
688.0535116
366.477535
382.8472786
811.7484197
21.88264608
21.40304573
23.49242629
5526.535659
13806.53049
14891.09486
20879.82773
474.2849636
347.9807232
304.6721413
407.1851105
38.19473568
10.24497088
12.29965378
4154.209181
12687.28184
14077.74169
25517.1061
281.6560741
247.3346197
218.6344181
139.3992156
21.2462944
11.67907738
9.97733259
4137.871622
ST NW ealth
µP ayof f
1.89749394
2.25212464
2.29395421
2.99785554
2.99936418
2.99987793
2.99973081
2.99529356
2.99119745
2.99552725
2.99648947
2.89597913
2.88287942
2.90542937
2.99790871
2.99939922
2.99986832
2.99971501
2.99462283
2.99025118
2.99434387
2.9945582
2.91751326
2.8225315
2.85250233
2.99781583
2.9993491
2.99982244
2.99968639
2.99186885
2.98761067
2.98938995
2.98926033
2.88475414
2.77114531
2.77628428
2.99934774
2.99987317
σP ayof f
1.115038951
0.833117085
0.701119432
0.000425526
0.000385781
7.01218E-05
0.000168855
0.003573669
0.006666972
0.006475695
0.003157087
0.106378724
0.108240328
0.099554579
0.000399672
0.000354815
9.09848E-05
0.000192909
0.005729981
0.007737358
0.008904183
0.006713692
0.067937253
0.243879937
0.205442216
0.000501175
0.000367287
0.000181018
0.000182243
0.010162416
0.011491916
0.013025445
0.020320879
0.128270786
0.223330482
0.26189141
0.000388675
7.63476E-05
1.70172884
2.70325105
3.27184515
7045.06407
7774.77897
42780.9622
17765.148
838.156511
448.659099
462.580005
949.131033
27.2232925
26.6340604
29.1842866
7500.92898
8453.42338
32971.0796
15549.8872
522.623557
386.469298
336.285061
446.03744
42.9442338
11.5734469
13.8846942
5981.57111
8166.23784
16571.9191
16459.8173
294.405283
259.974984
229.503856
147.102907
22.4895646
12.4082718
10.6008986
7716.86004
39292.2936
ST NP ayof f
134
Frequency
240
200
280
240
240
200
280
280
280
200
360
240
240
240
200
280
240
240
200
280
280
280
200
360
240
240
240
200
280
240
240
200
280
280
280
200
240
240
Payoff
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
567
567
2
2
7
7
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
2
2
2
2
TrialCount
Noise
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
100
100
100
100
10
10
10
10
1.9371775
1.9371775
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
1.7977E+308
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
1.7977E+308
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
Nonlinearity
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Strategy
µW ealth
382.7
416.8416667
47.785
46.82857143
44.01428571
50.625
48.215
46.275
44.8625
49.74285714
45.53
45.54583333
37.01666667
46.38333333
45.76944444
48.265
48.25357143
43.57857143
50.12857143
48.435
47.12916667
45.5875
49.78928571
47.315
46.33333333
40.34583333
46.92916667
45.96388889
48.24
46.34642857
43.50357143
50.66428571
47.78
47.27916667
43.64583333
50.58928571
47.115
45.61666667
σW ealth
166.2212431
164.0933167
11.85701375
12.74751676
16.98612279
13.01312867
11.12873645
12.5289415
16.2332558
14.66671228
13.97208288
14.3232759
24.15194379
16.48645067
16.75193719
11.20601513
11.86787326
17.41574323
12.94236833
10.77616699
12.66704975
16.7886572
14.31913902
12.51861714
14.20993393
20.9127449
15.50024137
14.83881271
12.25407687
12.28666473
17.18034554
13.61570035
9.677892333
11.85648483
17.4022144
12.71722227
11.87315354
13.06023439
2.302353134
2.540272053
4.030104124
3.673544604
2.591190837
3.890301961
4.332477475
3.693448486
2.763616896
3.391547895
3.258640848
3.179847518
1.532657867
2.813421413
2.732188159
4.307061829
4.06589878
2.502251604
3.873214713
4.494640817
3.720611161
2.715374998
3.477114485
3.779570817
3.260629752
1.929246186
3.027641024
3.097544917
3.936649044
3.772091906
2.532170923
3.721019443
4.937025373
3.987620895
2.508062039
3.978013801
3.968195967
3.492790812
ST NW ealth
µP ayof f
2.03250238
2.21320573
2.35013686
2.30377368
2.17306373
2.49565799
2.37021575
2.27592251
2.21729089
2.45195837
2.24193965
2.24286252
1.83431429
2.28290499
2.25710058
2.37549285
2.37610148
2.15330607
2.47065018
2.38029294
2.31841767
2.2496404
2.45865442
2.32917633
2.28360857
1.99175083
2.31402349
2.26703408
2.37002414
2.28581009
2.15167494
2.4981975
2.35092705
2.32451061
2.15557371
2.49265261
2.31728381
2.24728687
σP ayof f
0.881358075
0.869768142
0.577466979
0.61928836
0.818887049
0.63359647
0.537893373
0.604192843
0.782255137
0.716560431
0.682566722
0.697120891
1.174766962
0.802555339
0.810170434
0.544438708
0.572422513
0.841650767
0.631602577
0.520652487
0.610791042
0.813492963
0.694028384
0.607400157
0.690212104
1.021593409
0.75287173
0.717409454
0.593978023
0.596835282
0.828260498
0.657857815
0.464044981
0.5767566
0.841439836
0.620793636
0.575563347
0.630977916
2.30610286
2.54459278
4.06973377
3.72003387
2.65367945
3.9388761
4.40647881
3.76688095
2.83448556
3.42184451
3.28457215
3.2173222
1.56142822
2.84454526
2.78595773
4.3631961
4.15095742
2.55843178
3.91171644
4.57174987
3.7957624
2.76540856
3.54258482
3.83466533
3.3085606
1.94965121
3.07359594
3.16002817
3.99008726
3.82988432
2.59782392
3.79747333
5.066162
4.03031471
2.56176807
4.01526765
4.02611428
3.56159354
ST NP ayof f
135
Frequency
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
40
40
40
40
Payoff
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
972
972
972
972
4
4
4
4
7
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
2
2
2
2
TrialCount
Noise
10
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
0
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
Nonlinearity
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Strategy
799.975
795.825
639.175
752.95
433.27
449.525
404.7208333
459.9666667
447.885
441.8958333
417.4458333
470.4625
430.135
430.1375
386.7583333
426.4291667
448.375
436.9083333
400.0958333
477.7458333
435.685
436.8541667
410.4833333
454.9041667
457.425
430.8083333
367.3083333
423.0208333
433.545
450.7875
387.7166667
477.925
449.335
433.025
401.5333333
472.55
441.74
439.775
µW ealth
σW ealth
212.4249853
126.1823854
299.7336224
299.3309999
109.6447313
98.43106408
158.7883063
130.2939263
92.79688451
109.266692
151.959168
123.9702865
117.7542219
117.4000011
176.3891293
180.9552292
106.6775252
115.6304816
164.6007644
112.7842242
113.430092
116.2384026
154.8926232
128.7118033
94.25085875
119.2577947
198.5822582
149.3201384
118.5368212
111.0240995
160.0399785
121.8218072
105.0206778
119.9441163
151.7370221
129.5710455
104.6869734
110.6413849
3.765917643
6.306942112
2.132476814
2.515442772
3.951580663
4.566901762
2.548807545
3.530223394
4.826509019
4.044195218
2.747092123
3.794961788
3.652820197
3.663862827
2.192642681
2.356545144
4.203087757
3.778487536
2.430704588
4.235927823
3.841000146
3.758260239
2.650115447
3.534284773
4.85327143
3.612412374
1.849653321
2.832979114
3.657471119
4.060267112
2.422623836
3.92314817
4.278538372
3.610222938
2.646244981
3.647033935
4.219627196
3.974778521
ST NW ealth
µP ayof f
2.47358433
2.46135546
1.97696835
2.32798815
2.30018634
2.38707999
2.1493462
2.44193504
2.37800668
2.34632279
2.21702891
2.49758372
2.28358663
2.28411077
2.05347117
2.26418438
2.38041615
2.31980475
2.12488479
2.53594271
2.31309765
2.31900678
2.1795769
2.41533925
2.4281511
2.28731203
1.95074074
2.24608073
2.30182524
2.39350444
2.05923576
2.53698133
2.38562563
2.29914637
2.13252861
2.50870798
2.34511553
2.33482762
σP ayof f
0.655484714
0.389698448
0.92604765
0.926017508
0.580502521
0.521490144
0.841022398
0.689849819
0.491484917
0.578412317
0.804691745
0.656905188
0.624110695
0.622199137
0.935239552
0.959531129
0.564673383
0.612451682
0.870990383
0.596921921
0.600894773
0.615723626
0.820398182
0.682013454
0.499494391
0.632395999
1.052912555
0.791185088
0.628383798
0.588080296
0.847823759
0.64535175
0.55610983
0.635439132
0.803551288
0.686124256
0.555224695
0.586429953
3.77367203
6.31605149
2.13484516
2.51397854
3.96240543
4.5774211
2.55563491
3.53980674
4.83841233
4.05648829
2.75512819
3.80204596
3.65894489
3.67102851
2.19566331
2.35967788
4.21556288
3.78773513
2.43961911
4.24836585
3.84942215
3.76631119
2.65673053
3.54148329
4.86121795
3.61689833
1.85270917
2.83888153
3.66308815
4.07003
2.42884885
3.93116054
4.28984618
3.61820078
2.65387989
3.65634644
4.22372339
3.98142628
ST NP ayof f
136
Frequency
40
40
40
40
40
240
40
40
240
240
240
200
240
240
240
200
240
40
40
240
240
240
200
240
240
240
200
240
40
40
240
240
240
200
240
240
240
200
Payoff
972
972
972
972
972
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
2
2
2
2
2
4
4
4
4
4
TrialCount
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
10
10
Noise
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.527051027
0.1
0
0.527051027
0.1
Nonlinearity
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Guess
Strategy
2320.05
2301.108333
2128.633333
2550.925
2371.14
2302.683333
2183.65
2464.358333
2269.7
2198.85
2255.5125
2401.81
2345.6
2085.495833
2497.645833
2369.815
2304.745833
2202.0875
2479.883333
2037
2205.525
2257.741667
2359.385
2342.679167
2164.058333
2524.304167
2384.93
2270.016667
2236.1125
2457.591667
2228.775
1987.9
2286.929167
738.9
715.3
822.65
720.2
684.425
µW ealth
σW ealth
500.763245
556.346599
847.8570431
596.5506218
545.4923376
577.7423299
827.7871068
699.5246516
613.5645932
846.4155466
839.1423398
470.7621309
568.6841889
900.8633637
694.9932521
526.0697014
558.6667294
834.9165007
673.3411986
983.9329245
887.8823117
836.3098458
608.5787433
549.234529
822.5631769
647.9497113
503.583732
573.8096735
726.3098511
710.2148442
619.6429814
785.1950968
784.7505755
227.2265169
263.6784216
171.0847086
189.8867031
267.10802
4.63302773
4.136105689
2.510604058
4.276124954
4.34678883
3.98565799
2.637936714
3.5229042
3.699203026
2.597837444
2.687878317
5.101960932
4.124609134
2.314996832
3.593769905
4.504754776
4.125439572
2.637494286
3.682952029
2.070263073
2.484028537
2.699647359
3.876877111
4.265353037
2.630871882
3.895833462
4.735915497
3.956044611
3.078730788
3.460349621
3.596869596
2.531727475
2.914211519
3.251821178
2.712774127
4.80843675
3.792787954
2.562352863
ST NW ealth
µP ayof f
2.32156232
2.30258761
2.13016689
2.55243868
2.37274957
2.30423128
2.18517854
2.46593037
2.27109475
2.20014493
2.2569835
2.40333583
2.34706946
2.08691571
2.49911949
2.371294
2.30626088
2.20353459
2.48140903
2.03825747
2.20682684
2.25919054
2.36091181
2.34411844
2.16557588
2.52576966
2.38639513
2.27148496
2.2376022
2.45914163
2.23020199
1.98932941
2.28832789
2.28504773
2.21237778
2.54359762
2.22734138
2.11748343
σP ayof f
0.500810923
0.556289354
0.848006405
0.59662397
0.545443796
0.577788418
0.827844162
0.6996072
0.613778982
0.846861028
0.839425546
0.470794795
0.568769019
0.901012161
0.69516821
0.526146899
0.558703553
0.834952483
0.6734074
0.984786127
0.888276537
0.836547984
0.608649111
0.549309782
0.822617006
0.648013942
0.50354122
0.573841085
0.726396872
0.710327415
0.619880221
0.785226932
0.785003242
0.701305774
0.813765811
0.527960292
0.586197493
0.824781971
4.63560641
4.1391905
2.51197028
4.27813633
4.35012661
3.98801915
2.63960132
3.52473555
3.70018331
2.59799997
2.68872386
5.10484792
4.12657754
2.31619039
3.59498529
4.50690482
4.12787939
2.63911376
3.6848556
2.06974633
2.48439168
2.70061083
3.87893742
4.26738884
2.63254451
3.89770883
4.73922499
3.95838677
3.08041276
3.46198328
3.59779505
2.53344521
2.91505533
3.25827594
2.71869099
4.81778206
3.79964331
2.56732506
ST NP ayof f
137
Frequency
720
240
240
240
200
280
280
240
200
280
280
280
200
720
240
240
240
200
280
265
240
200
280
280
280
200
720
240
240
240
200
280
240
240
200
280
280
280
Payoff
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
63
7
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
2
2
2
2
2
2
2
TrialCount
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
0
Noise
Nonlinearity
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
1.7977E+308
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
1.7977E+308
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
1.7977E+308
Strategy
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
-4.425
-1.20714286
0
-4.485
-4.2625
-0.94583333
0
-6.735
-5.66666667
-5.48333333
-4
-3.09444444
21.11
21.87857143
23.975
25.02142857
21.285
21.09583333
23.64150943
24.99642857
18.24
19.03333333
19.17916667
21.3
22.39861111
37.28
38.95357143
40.79642857
42.025
37.045
38.18333333
40.63571429
42.00357143
34.465
36.17083333
35.44166667
37.43333333
38.9375
µW ealth
σW ealth
6.863054557
2.45093693
0
6.090137519
6.034091515
2.338353974
0
7.764971024
6.736509152
6.644525733
5.800862005
5.493882681
6.117017247
4.984214879
2.007864004
0.144808107
5.269134179
5.039343473
3.017523285
0.179248716
7.200166665
7.172788083
7.73770418
5.625240736
4.753741014
7.655821315
5.379324316
2.415276107
0.15612495
6.405698635
4.96820446
2.992158118
0.059654618
8.846964169
7.860236381
8.394736281
7.049507469
5.8266852
-0.64475664
-0.49252302
-0.73643657
-0.70640294
-0.40448681
-0.86735675
-0.84118741
-0.82524074
-0.68955269
-0.56325273
3.45102836
4.389572272
11.94054974
172.7902474
4.0395631
4.186226529
7.834739685
139.4510889
2.533274693
2.653547423
2.478663725
3.786504614
4.711786159
4.869497141
7.24135024
16.89099994
269.1754265
5.783131882
7.685539845
13.58073761
704.1126524
3.895686627
4.6017488
4.221891609
5.310063646
6.682616044
ST NW ealth
µP ayof f
2.72476534
2.90535461
2.99972906
2.72628953
2.73774554
2.91215617
2.99118522
2.6187764
2.67355167
2.68534266
2.75656001
2.80991822
2.74619109
2.78271336
2.9106239
2.99969989
2.7593282
2.74831234
2.89142532
2.98856826
2.60079363
2.64383883
2.65272222
2.76868227
2.82833479
2.70346049
2.78782393
2.90195038
2.9997119
2.68973127
2.75062018
2.89081239
2.98764558
2.55911036
2.64524257
2.61192184
2.71921195
2.80428014
σP ayof f
0.33516027
0.142533747
0.000158308
0.297140859
0.300474688
0.123663392
0.010574527
0.375623614
0.329297536
0.32170976
0.282417322
0.278061917
0.315134118
0.264108481
0.125039143
0.000179199
0.279035441
0.263945132
0.167193724
0.01526028
0.373644596
0.367398099
0.391074314
0.283159695
0.255705578
0.394590944
0.281899759
0.147114163
0.000216917
0.329396326
0.263679426
0.164641009
0.011716893
0.45220936
0.399726776
0.424446465
0.35339794
0.306900324
8.12973847
20.3836261
18948.7232
9.17507454
9.11140156
23.5490563
282.867043
6.97180982
8.11895438
8.34709727
9.76059113
10.1053688
8.71435661
10.5362514
23.2777018
16739.4483
9.88880907
10.4124381
17.2938628
195.839674
6.96060819
7.19611461
6.78316658
9.77781202
11.0609038
6.85129887
9.88941581
19.7258395
13828.87
8.16563836
10.431683
17.5582766
254.986159
5.65912737
6.61762667
6.15371325
7.69447596
9.1374297
ST NP ayof f
138
Frequency
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
240
240
240
200
40
Payoff
63
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
567
972
4
7
7
7
7
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
2
2
2
2
2
2
7
TrialCount
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
1.9371775
100
Noise
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
Nonlinearity
Strategy
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
869.2
458.975
457.4958333
483.3375
501.0375
448.52
458.2125
484.2833333
500.0375
427.89
435.3458333
437.2125
454.45
477.48
486.1333333
512.0791667
528.0333333
474.565
482.8333333
507.8708333
526.8541667
456.395
457.5041667
460.25
480.7916667
498.115
501.4208333
529.5458333
546.0333333
493.005
504.8833333
527.0958333
544.5166667
470.32
474.7416667
480.8
496.9791667
-4.58
µW ealth
σW ealth
78.22889492
47.24356438
57.32240675
29.21255884
0.210777331
74.70735974
47.35786465
25.14845367
1.99024967
77.87597768
63.30739477
59.81276907
59.4484861
67.57336458
55.05102683
27.51904612
0.2013841
56.83023645
47.54547531
32.10445165
1.991540268
81.19894688
76.03491511
77.35128204
54.32278464
56.27398844
54.55969574
24.93136377
0.179505494
69.55591258
50.74998905
27.63789637
2.173642616
76.42805506
79.18890872
60.33208102
59.84719765
6.934233916
11.110984
9.715079843
7.981099525
16.54553792
2377.093861
6.003692294
9.675531265
19.25698254
251.243604
5.494505658
6.876697974
7.309684985
7.644433522
7.066097759
8.8305952
18.60817284
2622.020976
8.35057233
10.15518996
15.81932745
264.5460778
5.620700976
6.017027388
5.950127624
8.850644714
8.851602913
9.190315791
21.24014708
3041.875335
7.087894928
9.948442212
19.07148888
250.5088291
6.153761202
5.995052519
7.969226188
8.304134299
-0.66049113
ST NW ealth
µP ayof f
2.799298
2.76978542
2.7620232
2.90146497
2.99973312
2.71425235
2.76607019
2.9047195
2.99072942
2.60594054
2.64532021
2.65532089
2.74635682
2.72498132
2.77075034
2.91035586
2.99973331
2.70965572
2.75349318
2.88640034
2.9893442
2.61345425
2.61968071
2.63427613
2.74294249
2.73916609
2.7562923
2.90786966
2.9997274
2.71190174
2.774912
2.89324477
2.98735944
2.5918442
2.61504961
2.64759324
2.73343084
2.72434992
σP ayof f
0.241311487
0.25023487
0.303591313
0.156321741
0.000174393
0.396987836
0.250950509
0.133766585
0.011781835
0.412426107
0.335529444
0.317066867
0.315275491
0.358302179
0.291062317
0.147058003
0.000145311
0.301172332
0.251805328
0.170400721
0.011855443
0.430694305
0.402975627
0.409862701
0.287785424
0.297995618
0.288791143
0.133742207
0.000159737
0.368973817
0.268880826
0.147097372
0.013198658
0.404613015
0.419544812
0.319561187
0.317388424
0.339388206
11.6003512
11.0687428
9.09783345
18.5608537
17200.9974
6.83711718
11.0223733
21.7148364
253.84242
6.31856348
7.88401812
8.37464007
8.71097468
7.60526027
9.51944025
19.7905302
20643.4883
8.9970274
10.9350076
16.9388974
252.149506
6.06800281
6.50084158
6.42721605
9.53120715
9.19196768
9.54424111
21.7423484
18779.2159
7.34984873
10.3202301
19.6689086
226.33812
6.4057361
6.23306386
8.28509015
8.61225752
8.02723804
ST NP ayof f
139
Frequency
40
40
40
40
40
40
40
40
240
43
40
240
240
240
200
240
240
240
200
240
40
40
240
240
240
200
240
240
240
200
240
40
40
240
240
240
200
240
Payoff
972
972
972
972
972
972
972
972
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
3000
7
7
7
7
7
7
7
7
4
4
4
4
4
4
4
4
4
4
4
2
2
2
2
2
2
2
2
2
2
2
4
4
4
4
4
4
4
4
TrialCount
Noise
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
100
10
10
10
10
1.9371775
1.9371775
1.9371775
100
100
100
10
10
10
1.9371775
1.9371775
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.8
0.527051027
0.1
0
0.527051027
0.1
0
0.527051027
0.1
0
0.527051027
0.1
0
0.527051027
0.1
Nonlinearity
Strategy
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
2934.033333
2629.35
2674.754167
2831.720833
2926.0625
2555.975
2649.75
2694.025
2646.64
2693.479167
2871.095833
2961.020833
2673.575
2705.954167
2865.708333
2951.154167
2572.625
2597.675
2724.858333
2714.3
2787.55
2898.341667
2979.029167
2706.195
2769.979167
2881.245833
2966.945833
2634.775
2498.093023
2736.020833
835.7
914
933.025
873.65
914.375
930.025
811.25
799.175
µW ealth
σW ealth
0.179505494
344.6716807
284.4157498
144.4046729
11.40980253
376.0750249
201.650905
329.5359607
446.197199
299.3575892
139.8720367
0.142826138
303.3748908
274.8882689
132.5452373
13.90283182
349.5953152
428.5359604
246.4828864
322.2342781
248.8143836
134.4859036
0.191440083
336.6273265
245.2297978
157.8836926
13.05793115
401.7544329
524.2303689
259.3416095
96.149935
39.34399573
0.15612495
69.00998116
24.89245619
4.251396829
128.2592589
134.014717
16345.08936
7.628564071
9.404381328
19.60962049
256.4516338
6.796449726
13.1402832
8.175207933
5.931547768
8.997530925
20.52658917
20731.64538
8.812776143
9.843832831
21.62060585
212.270004
7.358865776
6.061743331
11.05495953
8.423374496
11.20333141
21.55126738
15561.15685
8.039142359
11.29544285
18.24916675
227.2140815
6.558172814
4.765258122
10.54987219
8.691633541
23.23099073
5976.142828
12.65976291
36.73301635
218.7575137
6.325079427
5.963337593
ST NW ealth
µP ayof f
2.99972945
2.6938528
2.73920285
2.89633402
2.99089427
2.62050613
2.71434214
2.75854174
2.68408486
2.7309589
2.9089622
2.99972407
2.71114221
2.7434739
2.90325738
2.98896765
2.61003307
2.63506937
2.76240818
2.73380361
2.80702245
2.91821121
2.99970718
2.72572058
2.78953604
2.9008534
2.98671176
2.65434198
2.51778574
2.75561643
2.69490795
2.93821309
2.99973677
2.81265888
2.93851068
2.9880099
2.61985055
2.58209804
σP ayof f
0.000166752
0.344781948
0.284406153
0.144446961
0.011521721
0.376109276
0.201680101
0.329702081
0.446463284
0.299394804
0.140160275
0.000159453
0.303384618
0.274897456
0.132609341
0.014017915
0.349641017
0.428657568
0.246517859
0.322399835
0.248897283
0.134774811
0.000175984
0.336663922
0.245281622
0.157920098
0.013175291
0.401873062
0.524310921
0.259394762
0.296885806
0.122220451
0.000128274
0.213230472
0.076812827
0.013892916
0.395638129
0.414151133
17989.1602
7.8132072
9.63130658
20.0511938
259.587463
6.96740626
13.4586512
8.36677079
6.01188263
9.1215975
20.7545411
18812.578
8.93632059
9.97999015
21.8933098
213.224841
7.46489382
6.14725963
11.2057122
8.47954408
11.2778349
21.6524971
17045.3151
8.09626575
11.3727886
18.369121
226.690391
6.60492637
4.80208524
10.6232539
9.07725427
24.0402737
23385.3391
13.1906986
38.2554686
215.074351
6.62183536
6.23467577
ST NP ayof f
140
Frequency
240
240
200
Payoff
3000
3000
3000
7
7
7
TrialCount
100
100
100
Noise
0.8
0.527051027
0.1
Nonlinearity
Strategy
OrthogonalArrayParabolic
OrthogonalArrayParabolic
OrthogonalArrayParabolic
2646.14
2660.370833
2839.325
µW ealth
σW ealth
393.8919146
329.3002809
136.7570633
6.717934291
8.078859897
20.7618161
ST NW ealth
µP ayof f
2.71059652
2.7248916
2.9041494
σP ayof f
0.394082534
0.329348037
0.137023061
6.87824575
8.27359298
21.1946032
ST NP ayof f
Appendix D
Five Layer Process Support Model
To attempt to understand the connection between the individual or local adoption of
design of experiments and quality engineering practices, division-level DFSS implementations and a top-level corporate push to improve quality and lower costs due to
detailed design prototype testing and warrantee returns, we propose in section 4.3 a
five-layer process support model.
We take this opportunity to lay out some of our thinking about it in an appendix.
The model is nascent, and our investigations in Chapter ch4 did look at the organizations from enough perspectives to apply it fully. However, it does reconcile their
apparent autonomy with their alignment with a corporate goal, and explains why
some practitioners have faltered in their development of DoE practices while others
have flourished.
D.1
Layers
The layers as introduced in section 4.3 were
Environment The environment is the responsibility of the top management in an
organization. While a process can survive in a hostile environment, it would
typically need a strong champion, and would evanesce should that champion
leave.
141
An environment is created when the senior management decides to devote significant resources to a goal, and effectively communicates that decision to those
who create frameworks for polity.
Framework A framework is the bridge between environments and policies. The
framework happens in accordance with the set environment, but typically closer
to the policy making level. A framework specifies and articulates corporate goals
vaguely rendered by the environment, and describes necessary policies by type.
Policy A policy is a bundled set of rules that are intended to direct instruction
toward meeting the goals set up in a framework. A policy might indicate who
does what when.
Instruction Instruction enacts policy. Only when an order is given is it clear that
a policy must be followed. Without a reinforcing environment, instructions are
unlikely to follow policy.
Action On the operational level, actions, of course, must complete the intent. Actions without instructions can be helpful to an enterprise, but will not generally
be inspired by appropriate context, and will suffer from all the ill effects of poor
coordination.
The levels can crowd together somewhat. If a software lead sets a coding standard, and follows it when he writes software elements, then he embodies three levels.
Without a framework, however, that policy will be moribund even with an auspicious
environment, and it may be difficult enrolling other team members into abiding by
it.
This is explicitly a layered architecture. There are conflicting drives and diverse
needs at the corporate level, warring frameworks trying to fill some number of those
needs (for example, moving the corporate headquarters while implementing Business
Process Restructuring,) attempts to fit favored policies into as many current frameworks as possible, instructions that balance the demands of various policies with
142
available resources, and actions driven at least in part by the instructions of every
person with instructing authority over a particular actor.
D.2
Effecting Process Change
Clearly, this model can be applied in process change as a top down approach, or even
with a leading from within method. A new process can start at any layer. However,
the central point of this appendix is that all five layers must be whole and active for
a process to persist. We therefore recommend that process change identify what the
five layers would be for a particular process change, and start campaigns on all of
those levels.
The five-layer model can be compared to a theoretical treatment of Policy Deployment or Hoshin Management[33, Chapter 14]. The toolset built around Policy
Deployment may be useful in adopting the five-layer model, but there are some differences, notably that processes in the five layer model can exist comfortably with
other processes. The layered architecture of the five-layer model allows influences of
one layer on another to balance each other somewhat, whereas Policy Deployment
argues for monolithic action.
D.3
Describing Current Processes
The true power, however, of the model is in describing current processes. If you know
what your process is, you can tend to its health on all five levels, and coordinate
among the five levels to retire it. Particularly, obsolesced frameworks create a danger
of undercutting internal authority. This model may help a process owner detect such
things, and preserve corporate resources for current intents.
143
D.4
Accountability
This concept of the five-layer model came out of a discussion of the accountability toplevel decision makers have in the moral failings of ground-level workers. We proposed
that each act had a certain moral weight, that would accrue to the actor only insofar
as he had the option of not doing it. The remainder of the act would flow up the
chain of command.
The use of the model for partial and accretive accountability will have to wait until
reliable methods of measuring moral weights and willfulness have been established.
D.5
Fads
Over the course of the practice of scientific managment, innovations have grown into
movements, evolved into fads, and then been discredited in the public perception.
The five-layer process support model suggests that when a corporate commitment is
not made to the underlying principles of a movement, it is being introduced into a
hostile environment.
Currently, DFSS shows some signs of turning into a fad. The five-layer model
suggests that this happens when DFSS is adopted as a policy framework with a
corporate need of saving money or staying abreast of industry – needs for which
other frameworks will yield better metrics – not of improving quality through early
analysis.
D.6
An Example: Balancing the Registers
In one over the counter tax preparation service, a new process was put in place
whereby the registers would be fully closed out every night with particular documentation.
144
D.6.1
A process
All of the layers worked together to create this new process. Once a corporate need
for tighter accounting was annunciated, a framework was created to try to satisfy it.
This daily-register-closing policy came out of that framework. The instructions were
left up to the branch managers.
The business of tax preparation requires mathematically competent seasonal specialists to work odd hours for little pay. Some of these specialists balked. One
specialist refused to close out the registers, but the instruction was flexible enough
that she could perform all the activities required to close out the register without
calling it by its proper name. Another maintained that she was, in fact, closing out
the registers, but pushed the action back on her manager.
As the actions were largely competent and successful, this was, for the 2004 tax
season, a successful process implementation. However, if the second tax preparers
failure to comply with the new policy had been more common, the manager would
have been overwhelmed in trying to perform the functions left by her direct reports,
and the process would have failed.
D.6.2
Conflict with other corporate needs
Another corporate need is to reduce costs. The environment around this can engender a framework of cost-reduction, and particular cost-cutting policies such as the
reduction of overtime. This policy conflicts with the proper register closing policy, as
the latter will sometimes push the operator into overtime.
In this kind of situation, both environments have to be maintained and balanced,
and management has to enforce both needs at the operational level.
145
Bibliography
[1] J. D. Musa and A. F. Ackerman Quantifying software validation: when to stop
testing? Software, IEEE, vol. 6, no. 3, May, pp. 19-27, 1989.
[2] E. F. Crawley, System Architecture, MIT course ESD.34J, Lecture 1, Fall, 2003.
[3] W. J. Hopp, Factory Physics : Foundations of Manufacturing Management,
Boston, Massachusetts: Irwin/McGraw-Hill, 2001.
[4] M. S. Phadke, Quality Engineering using Robust Design, Engelwood Cliffs, New
Jersey: Prentice Hall, 1989.
[5] J. A. Morrow, An investigation of the penetration of Taguchi Quality Engineering in five U.S. companies Thesis of Master of Science in the Management of
Technology, MIT, June 1988.
[6] N.W. Hirschi and D.D. Frey, Cognition and complexity: An experiment on the
effect of coupling in parameter design, Research in Engineering Design vol. 13,
pp 123-131, 2002.
[7] A. Peters, An investigation of the implementation and application of Taguchi’s
Quality Engineering method in US corporations Thesis for the Diplomarbeit, TU
Berlin, October 1994.
[8] G. Taguchi, Quality Engineering in Production Systems, New York, New York:
McGraw-Hill, 1988.
[9] C. F. J. Wu, Experiments: Planning, Analysis, and Parameter Design Optimization, New York, New York: J. Wiley, 2000.
146
[10] QFD Institute, Design for Six Sigma and QFD Workshop by the QFD Institute,
[Online document], 2004, Available HTTP:
http://www.qfdi.org/tut dfss.htm
[11] Six Sigma Qualtec, Six Sigma Products and Services web site [Online document],
2004, Available HTTP:
http://www.sixsigmaqualtec.com/six-sigma-products/
[12] D. D. Frey, F. Engelhardt, E. M. Greitzer, A role for ”one-factor-at-a-time”
experimentation in parameter design, Research in Engineering Design, Volume
14, Number 2, pp. 65 - 74, 2003.
[13] D. D. Frey and Rajesh Jugulum, How one-factor-at-a-time experimentation can
lead to greater improvement than orthogonal arrays, Proceedings of DETC ’03,
ASME 2003 Design Engineering Technical Conference and Computers and Information in Engineering Conference, DETC2003/DTM-48646, pp. 505 - 513.
[14] D. D. Frey Personal Conversation, June, 2003.
[15] S. H. Thomke, The economics of experimentation in the design of new products
and processes, Thesis for Doctor of Philosophy in Electrical Engineering and
Management, MIT, February 1995.
[16] W. Middendorf, Design of Devices and Systems, New York: Marcel Decker, 1986.
[17] S. H. Thomke and A. Nimgade, IDEO Product Development, Harvard Business
School Case No. 600-143 (2000), 21 p.
[18] S. H. Thomke, E. von Hippel and R. Franke, Modes of experimentation: an
innovation process–and competitive–variable, Research Policy, Volume 27, Issue
3, July 1998, Pages 315-332.
[19] N. A. Barricelli, Symbiogenetic evolution processes realized by artificial methods,
Methodos, no. 9, pp. 35-36, 1957.
147
[20] G. E. P. Box, Evolutionary operation: A method for increasing industrial productivity, Applied Statistics, vol. 6, no. 2, pp. 81–101, 1957.
[21] A. Marczyk, Genetic Algorithms and Evolutionary Computation, [Online document], 2004, Available HTTP:
http://www.talkorigins.org/faqs/genalg/genalg.html#history
[22] D. E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine
Learning, Reading, Massachusetts: Addison-Wesley, 1989.
[23] K. A. De Jong, An analysis of the behavior of a class of genetic adaptive systems,
Thesis for Doctor of Philosophy in Computer Science, University of Michigan,
1975.
[24] I. Faybish, Applying the genetic algorithm to the game of Othello, Thesis for
Master of Science degree in Computer Science, Vrije Universiteit Brussel, 1999.
Available HTTP:
http://www.geocities.com/Athens/Delphi/2402/othello.html
[25] D. Beasley, D. R. Bull and R. R. Martin, An Overview of Genetic Algorithms:
Part 1, Fundamentals, University Computing, vol. 15, no. 2, pp. 58 - 69.
[26] M. G. Neubauer, W. Watkins and J. Zeitlin, Notes on D-optimal designs, Linear
Algebra and its Applications, vol. 280, no. 2-3, Sep., pp. 109-127, 1998.
[27] D. Clausing, personal conversation, March, 2004.
[28] K. T. Ulrich, and Steven D. Eppinger Product Design and Development, Boston,
Massachusetts: Irwin/McGraw-Hill, 2000.
[29] J. T. Grewen, An experimental study of the effects of interactions upon parameter design with Taguchi Methods, Thesis of Bachelor of Science in Mechanical
Engineering, MIT, 1999.
148
[30] H. Chipman, M. Hamada and C. F. J. Wu, A Bayesian variable selection approach for analyzing designed experiments with complex aliasing, Technometrics,vol. 39, Feb., pp. 372-381, 1997.
[31] S. H. Thomke, Experimentation matters: unlocking the potential of new technologies for innovation, Boston, Massachusetts: Harvard Business School Press,
2003.
[32] S. M. Ahlman, Complex dynamic system architecture evaluation through a hierarchical synthesis of tools and methods, Thesis of Master of Science in Engineering
and Management, MIT, 2003.
[33] S. Shiba, D. Walden, A. Graham, A new American TQM: four practical revolutions in management, Norwalk, Connecticut: Productivity Press, 1993
[34] Epstein, Joshua M., Growing Artificial Societies: Social Science from the Bottom
Up, Washington, D.C.: Brookings Institution Press, 1996.
149