EXPERIMENTAL
DESIGN
Instructor: Lê Ngọc Liễu
Email: lnlieu@hcmiu.edu.vn
Office: A1.706
Content
Chapter 1: Introduction and Basics
Chapter 2: Comparative Experiments
Chapter 3: Factorial Design
Chapter 4: Modeling
Chapter 5: Optimization
Chapter 6: Design of experimental flowchart
Reference
Gary W. Oehlert, A First Course in Design and Analysis of Experiments, University
of Minnesota, 2010
A guide to experimental design
https://www.scribbr.com/methodology/experimental-design/
STAT 503
https://online.stat.psu.edu/stat503/home
Chapter 1: Introduction and Basics
Content
What is research?
Type of research
What is experiment?
Terms and Concepts
Three basic principles of statistical experimental design
Correlation and Regression Analysis
What is research?
A research can be defined as a systematic investigation to establish novel facts
usually using scientific methods.
The scientific method is the means by which researchers are able to make
conclusive statements about their studies with a minimum of bias.
In order to minimize the influence of personal stakes and biased opinions,
a standard method of testing a hypothesis is expected
Hypothesis
A hypothesis is a 'small' cause and effect statement about a specific set of
circumstances.
It represents a belief that a researcher possesses before conducting a
satisfactory number of experiments that could potentially disprove that belief.
Null or alternative hypothesis?
Hypothesis example
Usage of preservatives increases the shelf life of food
Vitamin C can protect the color of wine from degradation
Type of solvents determines the yield of phenolic extraction
Elevated temperatures inactivate enzyme activity
Steps of the scientific methods
Ask question
Do background
research
Construct
hypothesis
Think it
again
Test with
experiment
Analyze results
Draw conclusion
Hypothesis is
True
Hypothesis is False
or Partially True
Report results
Steps of the scientific methods
Identification of the researchable problem (or question that needs to be
answered)
Making observation and researching the topic of interest
Formulation of hypothesis or prediction of possible answers to the
problem
Experimentation or developing and following a procedure for testing as
to what causes the happening of phenomena. The outcome of an
experiment must be measurable (quantifiable).
Steps of the scientific methods
Recording and analyzing results (in tables, graphs, photographs)
Making conclusion whether to accept or reject the hypothesis and
forwarding recommendations for further studies as well as possible
improvement to the procedure
Communicating the finding or result to the scientific world (writing
publishing research papers, presenting scientific papers).
Types of Research
Based on the Purpose of Research
Based on the Methods of Research Conclusion
Based on the Status of Experimental Units under Study
Types of Research: Based on the Purpose of Research
Basic Research (Pure or Fundamental Research)
• It is undertaken for increase in knowledge.
• There is no direct benefit as it is a research for the sake of research
• It is conducted to satisfy any curiosity
• It is the source of most new theories, principles and ideas
Types of Research: Based on the Purpose of Research
Basic Research (Pure or Fundamental Research)
Types of Research: Based on the Purpose of Research
Applied Research
• It is use of past theories, knowledge and methods obtained from
basic research for solving an existing (specific) problem.
• It deals with practical problems
Types of Research: Based on the Methods of Research
Conclusion
Deductive Research
• “Top-down” or “from general to specific”
• Start from a theory and try to prove it right with the help of available
information.
Deductive Research
1. Start with an existing theory
2. Formulate a hypothesis based on existing theory
3. Collect data to test the hypothesis
4. Analyze the results: does the data reject or support the hypothesis?
Limitations of a deductive approach
The conclusions of deductive reasoning can only be true if all the
premises set in the inductive study are true and the terms are clear.
Types of Research: Based on the Methods of Research
Conclusion
Inductive Research
• “bottom-up” in nature or from specific to general approach of drawing
conclusions from research findings
• most widely applied by taking representative samples from a large
population.
Inductive Research
1. Observation
2. Observe a pattern
3. Develop a theory
Limitations of an inductive approach
A conclusion drawn on the basis of an inductive method can never be
proven, but it can be invalidated.
Types of Research: Based on the Status of Experimental
Units under Study
Observational studies
• The researcher is not changing or manipulating the factors or treatments, but
grouping the existing factors that result in the responses of interest
• Example: research carried out on human beings and survey studies.
Types of Research: Based on the Status of Experimental
Units under Study
Experimental studies
• The researcher intentionally changes or manipulates the levels of the factors
or treatments to see the effect on the responses or parameters being
measured
• Widely used in optimization of industrial production processes by trying
different levels of the factors (input variables) so as to improve the quality and
quantity of the output variables
What is experiment?
An experiment is characterized by the treatments and experimental units to
be used, the way treatments are assigned to units, and the responses that are
measured.
An experiment is a type of research method in which you manipulate one or
more independent variables and measure their effect on one or more dependent
variables.
Example: Will an ice cream manufactured with a new kind of stabilizer be as
palatable as our current ice cream?
Terms and Concepts
Treatments are the different procedures we want to compare
Experimental units are the things to which we apply the treatments.
Factors/Independent variables combine to form treatments.
Responses/dependent variables are outcomes that we observe after applying a
treatment to an experimental unit.
Terms and Concepts
Control is a “standard” treatment that is used as a baseline or basis of
comparison for the other treatments.
Placebo is a null treatment that is used when the act of applying a treatment—
any treatment—has an effect.
Example: GRAS status of MGS
Terms and Concepts
Experimental Error is the random variation present in all experimental results.
Measurement units (or response units) are the actual objects on which the
response is measured.
Blinding occurs when the evaluators of a response do not know which treatment
was given to which unit.
Randomization is the use of a known, understood probabilistic mechanism
for the assignment of treatments to units.
Other aspects of an experiment can also be randomized: for example, the
order in which units are evaluated for their responses.
Advantages of experiments
Experiments allow us to set up a direct comparison between the treatments of
interest.
We can design experiments to minimize any bias in the comparison.
We can design experiments so that the error in the comparison is small.
Most important, we are in control of experiments, and having that control
allows us to make stronger inferences about the nature of differences that we
see in the experiment. Specifically, we may make inferences about causation.
Experiment vs. observational study
Observational study
Also has treatments, units, and responses, but we don’t
get to control that assignment.
Its mechanism is usually unknown
Still useful and can produce important results
Unique when experiments are not feasible due to ethics
constrain
A good experimental design must be:
Avoid systematic error
Be precise
Allow estimation of error
Have broad validity.
Compromise often needed
Independent and dependent variables
To research a cause-and-effect relationship, you need to define your
independent and dependent variables.
Independent variable (factor): the cause.
o Its value is independent of other variables in your study.
Dependent variable (response): the effect.
o Its value depends on changes in the independent variable.
Control variable:
o Variables that are held constant throughout the experiment.
Independent and dependent variables
Research question
Independent variables
Dependent variables
Do tomatoes grow fastest
under fluorescent,
incandescent, or natural
light?
Does pasteurization keep fruit
juice safe for consumption?
In experimental research, the independent variable is manipulated or
changed by the experimenter to measure the effect of this change on the
dependent variable.
Control variables?
Visualizing independent and dependent variables
Researchers often use charts or graphs to visualize the results of their studies.
• Independent variable: “x” or horizontal axis
• Dependent variable: “y” or vertical axis
Research question:
Does a new medication affect blood
pressure?
https://www.scribbr.com/methodology/experimental-design/
Visualizing independent and dependent variables
Researchers often use charts or graphs to visualize the results of their studies.
• Independent variable: “x” or horizontal axis
• Dependent variable: “y” or vertical axis
140
Could pre-blanching preserve antioxidants
of basil leaves during hot-air drying?
No blanching
Total phenolic content
Research question:
Blanching
120
100
80
60
40
20
0
50
60
70
Drying temperature (oC)
80
Qualitative vs. quantitative variables (data)
Qualitative, or Attribute, or Categorical, Variable:
• A variable that categorizes or describes an element of a population.
Quantitative, or Numerical, Variable
• A variable that quantifies an element of a population.
Example
Identify each of the following examples as attribute (qualitative) or numerical
(quantitative) variables.
1. The residence hall for each student in a statistics class.
2. The amount of gasoline pumped by the next 10 customers at the local Unimart.
3. The amount of radon in the basement of each of 25 homes in a new
development.
4. The color of the baseball cap worn by each of 20 students.
5. The length of time to complete a mathematics homework assignment.
6. The state in which each truck is registered when stopped and inspected at a
weigh station.
Qualitative vs. quantitative variables (data)
Binomial
Qualitative
Nominal
Ordinal
Variable
Quantitative
Discrete
Continuous
Qualitative variable
Binomial: place things in one of two mutually exclusive categories:
right/wrong, true/false, or accept/reject.
Nominal: assign individual items to named categories that do not have an
implicit or natural value or rank. Example: record color of candies
Ordinal: items are assigned to categories that do have some kind of
implicit or natural order. Example is a survey question that asks us to rate
an item on a 1 to 10 scale, with 10 being the best. This implies that 10 is
better than 9, which is better than 8, and so on.
Quantitative variable
Discrete: data is a count that can't be made more precise. Typically it
involves integers. For instance, the number of children (or adults, or pets)
in your family is discrete data, because you are counting whole, indivisible
entities: you can't have 2.5 kids, or 1.3 pets.
Continuous: data could be divided and reduced to finer and finer levels.
For example, you can measure the height of your kids at progressively
more precise scales—meters, centimeters, millimeters, and beyond—so
height is continuous data.
Case study 1.1: observational study
Bingham, Sheila A., Nicholas E. Day, Robert Luben, Pietro Ferrari, Nadia Slimani,
Teresa Norat, Françoise Clavel-Chapelon et al. "Dietary fibre in food and protection
against colorectal cancer in the European Prospective Investigation into Cancer
and Nutrition (EPIC): an observational study." The lancet 361, no. 9368 (2003):
1496-1501.
Case study 1.2: experimental study
Leidy, Heather J., Cheryl LH Armstrong, Minghua Tang, Richard D. Mattes, and
Wayne W. Campbell. "The influence of higher protein intake and greater eating
frequency on appetite control in overweight and obese men." Obesity 18, no. 9
(2010): 1725-1732.
Confounding variables
In research that investigates a potential cause-and-effect relationship,
a confounding variable is an unmeasured third variable that influences
both the supposed cause and the supposed effect.
• It must be correlated with the independent variable. This may be a
causal relationship, but it does not have to be.
• It must be causally related to the dependent variable.
It’s important to consider potential confounding variables and account
for them in your research design to ensure your results are valid.
Confounding variables
https://www.scribbr.com/methodology/experimental-design/
Why confounding variables matter?
To ensure the internal validity of your research
• If not, your results may not reflect the actual relationship between
the variables that you are interested in
Example
You find that babies born to mothers who smoked during their pregnancies
weigh significantly less than those born to non-smoking mothers.
However, if you do not account for the fact that smokers are more likely to
engage in other unhealthy behaviors, such as drinking or eating less healthy
foods, then you might overestimate the relationship between smoking and low
birth weight.
Accuracy vs Precision
Accuracy is how close a measured value is to the actual (true) value
Precision is how close the measured values are to each other
Reliability vs validity
Reliability refers to how consistently a method measures something. If the
same result can be consistently achieved by using the same methods
under the same circumstances, the measurement is considered reliable.
Validity refers to how accurately a method measures what it is intended to
measure. If research has high validity, that means it produces results that
correspond to real properties, characteristics, and variations in the physical
or social world.
Reliability vs validity
https://outmatch.com/resources/blog/pre-employment-assessments-validity-vs-reliability/
Reliability vs validity
Example: reliable but not valid
You measure the temperature of a liquid sample several times under
identical conditions. The thermometer displays the same temperature
every time, so the results are reliable.
The thermometer that you used to test the sample gives reliable
results. However, the thermometer has not been calibrated properly, so
the result is 2 degrees lower than the true value. Therefore, the
measurement is not valid.
How to ensure validity and reliability in your research
Ensuring validity
Choose appropriate methods of measurement
Use appropriate sampling methods to select your subjects
How to ensure validity and reliability in your research
Ensuring reliability
Apply your methods consistently
Standardize the conditions of your research
Internal vs external validity
Internal validity (the design of experiment): refers to the degree of
confidence that the causal relationship being tested is trustworthy and not
influenced by other factors or variables.
External validity (the generalizability of the results): refers to the extent
to which results from a study can be applied (generalized) to other
situations, groups or events.
Generalizability
Truth in
the Study
Truth in
Real Life
Internal validity
External validity
Trade-off between internal and external validity
Better internal validity often comes at the expense of external validity (and vice
versa). The type of study you choose reflects the priorities of your research.
Factor
Laboratory experiment
Field experiment
Artificial
Realistic
Control
High
Low
Internal validity
High
Low
External validity
Low
High
Time required
Short
Long
Number of subjects
Small
Large
Ease of administration
High
Low
Cost
Low
High
Environment
Solution: conduct the research first in a controlled (artificial) environment to
establish the existence of a causal relationship, followed by a field experiment
to analyze if the results hold in the real world.
How to reduce the impact of confounding variables
Three basic principles of statistical experimental design
Replication
Blocking
Randomization
Replication
What?
• Replicates are multiple experimental runs with the same factor settings
(levels).
Why?
• Ensure validity
• Account for error
• Define the outlier
Difference between replicates and repeats
• Repeat measurements: taken during the same experimental run or
consecutive runs
• Replicate measurements: are taken during identical but different
experimental runs, which are often randomized
Replication
How many replicates?
The more the better?
What is the average height of a man in Germany?
What is the average height of a man in Germany?
Blocking
Blocking is the arrangement of experiments into groups (called also
blocks)
• A block is characterized by a set of homogeneous plots or a set of
similar experimental units.
• Failure to block is a common flaw in designing an experiment.
Blocking
Blocking is a technique for dealing with nuisance factors.
• A nuisance factor is a factor that has some effect on the response, but
is of no interest to the experimenter; however, the variability it
transmits to the response needs to be minimized or explained
• Typical nuisance factors include batches of raw material if you are in a
production situation, different experimenters or subjects in studies, the
pieces of test equipment, when studying a process, and time (shifts, days,
etc.) where the time of the day or the shift can be a factor that influences
the response.
Nuisance vs. confounding factors/variables
Confounding variable: Variable that changes concomitantly with the
independent variable. The dependent variable is affected by both the
independent variable and the confounding variable.
Nuisance variable: Variable that causes dependent variable scores to
be more variable
Blocking principle
If the nuisance variable is known and controllable, we use blocking and
control it by including a blocking factor in our experiment.
If you have a nuisance factor that is known but uncontrollable, sometimes
we can use analysis of covariance to measure and remove the effect of the
nuisance factor from the analysis.
If nuisance factors are unknown and uncontrollable (sometimes called a
“lurking” variable), we use randomization to balance out their impact.
The general rule is:
“Block what you can; randomize what you cannot.”
Randomization
Randomization is an insurance against a systematic bias due to a
nuisance factor
• For example, selection bias (where some groups are
underrepresented) is eliminated and accidental bias (where chance
imbalances happen) is minimized.
Example: choose panelists for food sensory tests
Correlation and Regression Analysis
https://byjus.com/maths/correlation-and-regression/
Correlation
Correlation is a measure of the degree to which two random variables,
(which are independent of each other), vary together or are associated.
Example:
• Determining the association of a food quality attribute measured by two
methods (e.g. volumetric and instrumental methods)
• Association between sensory textures obtained from panelists and
instrumental measurements.
Note: Correlation is not about a cause-and-effect relationship
Correlation
The association between only two random variables is called simple
linear correlation.
The degree of this association is measured by a quantity known as
Pearson correlation coefficient (r).
∑
=
∑
−
∑
−
∑ ∑
× ∑
−
∑
Correlation
r always lies between -1 and +1
r close to absolute 1 indicates that a very strong linear relationship
exists between the two variables
When r is equal to zero then there is no linear relationship between the
two variables.
Positive value of r indicates that the two variables tend to increase
together
Negative value of r on the other hand shows that large value of one
variable is associated with small value of the other variable.
Correlation
https://ecstep.com/correlation-regression-analysis/
Correlation example
The nutrient retention of different pre-processing treatments on carrot and sweet potato samples was
studied. The research team was interested in finding out the association between the nutrient retention of
the two roots. The following data was obtained from the two roots.
Sample
Nutrient retention of root samples
Sweet potato (X)
Carrot (Y)
1
14
7
2
12
8
3
11
7
4
10
8
5
9
9
6
9
11
7
8
12
8
7
14
9
6
16
1. What could be the hypotheses that will be helpful in
testing the relationship of nutrient retention of the
two roots?
2. Compute the coefficient of correlation
3. Determine the kind and significance of the
association
4. What practical meaning does this relationship
have?
20
Correlation example
Nutrient retention of root samples
Carrot
Sample
16
12
8
Sweet potato (X)
Carrot (Y)
1
14
7
4
2
12
8
0
3
11
7
4
10
8
5
9
9
6
9
11
7
8
12
8
7
14
9
6
16
Homework:
Draw a graph in excel
Data input in SPSS
Output for correlation in SPSS
0
4
8
12
Sweet potato
Result from SPSS
16
Extra correlation example
The β-carotene content of carrot samples subjected to different treatments was determined by two
different methods i.e.: UV vis. Spectrophotometer and HPLC. The researcher wanted to know if the trends
in the β-carotene obtained by the two methods have some association. The researcher arranged as the
one in the Table below.
β-carotene content (mg/100 g)
Spectrophotometer(X)
HPLC (Y)
1
8.4
12.5
2
8.1
12
3
7.6
11.3
4
6.9
10.7
5
6.3
9.8
6
5.8
8.4
7
5.2
7.8
8
4.7
6.6
1. State hypothesis
2. Determine the coefficient of correlation
3. Test the significance of the correlation of there is any
and identify the kind.
4. What is the practical meaning of the relationship?
HPLC
Sample
14
12
10
8
6
4
2
0
0
2
4
6
8
10
Spectrophotometer
Regression Analysis
In regression, we express the relationship of one variable to another by
an equation (regression model) that describes one as a function (linear
in the simplest case) of the other variable.
In regression analysis, the dependent variable is denoted "y" and the
independent variables are denoted by "x"
The most basic assumption underlying regression analysis is that there
is unilateral casual relation between X and Y. That is, variation in X
results in variation in Y. However, variation in Y does not result in
variation in X.
Example:
=
+
=
+
+
+⋯+
+
Regression Model Adequacy checking
Coefficient of determination (R2 and R2adj).
Distribution of residuals (residual plots)
Root mean square error (RMSE)
Percentage of root mean square error
Coefficient of determination
Coefficient of determination explains proportion of variability
The value of R2 is between 0 and 1. R2 value of 1 means the model
perfectly predicts the experimental data whereas a 0 value indicates a
totally faulty prediction.
In practical application R2 value less than 0.75 usually indicates
an insufficiently precise description of the experimental data.
Coefficient of determination
R2 overestimates the goodness of fit and it is usually modified into
adjusted R2 (R2adj) by taking the degree of freedom of the model into
account
R2adj is very important in comparing adequacy of different models.
Distribution of residuals (residual plots)
Residual: difference between the predicted and experimental values
Plots close to 0 means the experimental and the estimated values are
equal and this indicates the goodness or adequacy of the model
Root mean square error (RMSE)
What is the unit of RMSE?
Correlation and Regression Analysis