EXPERIMENTAL DESIGN Instructor: Lê Ngọc Liễu Email: lnlieu@hcmiu.edu.vn Office: A1.706 Content Chapter 1: Introduction and Basics Chapter 2: Comparative Experiments Chapter 3: Factorial Design Chapter 4: Modeling Chapter 5: Optimization Chapter 6: Design of experimental flowchart Reference Gary W. Oehlert, A First Course in Design and Analysis of Experiments, University of Minnesota, 2010 A guide to experimental design https://www.scribbr.com/methodology/experimental-design/ STAT 503 https://online.stat.psu.edu/stat503/home Chapter 1: Introduction and Basics Content What is research? Type of research What is experiment? Terms and Concepts Three basic principles of statistical experimental design Correlation and Regression Analysis What is research? A research can be defined as a systematic investigation to establish novel facts usually using scientific methods. The scientific method is the means by which researchers are able to make conclusive statements about their studies with a minimum of bias. In order to minimize the influence of personal stakes and biased opinions, a standard method of testing a hypothesis is expected Hypothesis A hypothesis is a 'small' cause and effect statement about a specific set of circumstances. It represents a belief that a researcher possesses before conducting a satisfactory number of experiments that could potentially disprove that belief. Null or alternative hypothesis? Hypothesis example Usage of preservatives increases the shelf life of food Vitamin C can protect the color of wine from degradation Type of solvents determines the yield of phenolic extraction Elevated temperatures inactivate enzyme activity Steps of the scientific methods Ask question Do background research Construct hypothesis Think it again Test with experiment Analyze results Draw conclusion Hypothesis is True Hypothesis is False or Partially True Report results Steps of the scientific methods Identification of the researchable problem (or question that needs to be answered) Making observation and researching the topic of interest Formulation of hypothesis or prediction of possible answers to the problem Experimentation or developing and following a procedure for testing as to what causes the happening of phenomena. The outcome of an experiment must be measurable (quantifiable). Steps of the scientific methods Recording and analyzing results (in tables, graphs, photographs) Making conclusion whether to accept or reject the hypothesis and forwarding recommendations for further studies as well as possible improvement to the procedure Communicating the finding or result to the scientific world (writing publishing research papers, presenting scientific papers). Types of Research Based on the Purpose of Research Based on the Methods of Research Conclusion Based on the Status of Experimental Units under Study Types of Research: Based on the Purpose of Research Basic Research (Pure or Fundamental Research) • It is undertaken for increase in knowledge. • There is no direct benefit as it is a research for the sake of research • It is conducted to satisfy any curiosity • It is the source of most new theories, principles and ideas Types of Research: Based on the Purpose of Research Basic Research (Pure or Fundamental Research) Types of Research: Based on the Purpose of Research Applied Research • It is use of past theories, knowledge and methods obtained from basic research for solving an existing (specific) problem. • It deals with practical problems Types of Research: Based on the Methods of Research Conclusion Deductive Research • “Top-down” or “from general to specific” • Start from a theory and try to prove it right with the help of available information. Deductive Research 1. Start with an existing theory 2. Formulate a hypothesis based on existing theory 3. Collect data to test the hypothesis 4. Analyze the results: does the data reject or support the hypothesis? Limitations of a deductive approach The conclusions of deductive reasoning can only be true if all the premises set in the inductive study are true and the terms are clear. Types of Research: Based on the Methods of Research Conclusion Inductive Research • “bottom-up” in nature or from specific to general approach of drawing conclusions from research findings • most widely applied by taking representative samples from a large population. Inductive Research 1. Observation 2. Observe a pattern 3. Develop a theory Limitations of an inductive approach A conclusion drawn on the basis of an inductive method can never be proven, but it can be invalidated. Types of Research: Based on the Status of Experimental Units under Study Observational studies • The researcher is not changing or manipulating the factors or treatments, but grouping the existing factors that result in the responses of interest • Example: research carried out on human beings and survey studies. Types of Research: Based on the Status of Experimental Units under Study Experimental studies • The researcher intentionally changes or manipulates the levels of the factors or treatments to see the effect on the responses or parameters being measured • Widely used in optimization of industrial production processes by trying different levels of the factors (input variables) so as to improve the quality and quantity of the output variables What is experiment? An experiment is characterized by the treatments and experimental units to be used, the way treatments are assigned to units, and the responses that are measured. An experiment is a type of research method in which you manipulate one or more independent variables and measure their effect on one or more dependent variables. Example: Will an ice cream manufactured with a new kind of stabilizer be as palatable as our current ice cream? Terms and Concepts Treatments are the different procedures we want to compare Experimental units are the things to which we apply the treatments. Factors/Independent variables combine to form treatments. Responses/dependent variables are outcomes that we observe after applying a treatment to an experimental unit. Terms and Concepts Control is a “standard” treatment that is used as a baseline or basis of comparison for the other treatments. Placebo is a null treatment that is used when the act of applying a treatment— any treatment—has an effect. Example: GRAS status of MGS Terms and Concepts Experimental Error is the random variation present in all experimental results. Measurement units (or response units) are the actual objects on which the response is measured. Blinding occurs when the evaluators of a response do not know which treatment was given to which unit. Randomization is the use of a known, understood probabilistic mechanism for the assignment of treatments to units. Other aspects of an experiment can also be randomized: for example, the order in which units are evaluated for their responses. Advantages of experiments Experiments allow us to set up a direct comparison between the treatments of interest. We can design experiments to minimize any bias in the comparison. We can design experiments so that the error in the comparison is small. Most important, we are in control of experiments, and having that control allows us to make stronger inferences about the nature of differences that we see in the experiment. Specifically, we may make inferences about causation. Experiment vs. observational study Observational study Also has treatments, units, and responses, but we don’t get to control that assignment. Its mechanism is usually unknown Still useful and can produce important results Unique when experiments are not feasible due to ethics constrain A good experimental design must be: Avoid systematic error Be precise Allow estimation of error Have broad validity. Compromise often needed Independent and dependent variables To research a cause-and-effect relationship, you need to define your independent and dependent variables. Independent variable (factor): the cause. o Its value is independent of other variables in your study. Dependent variable (response): the effect. o Its value depends on changes in the independent variable. Control variable: o Variables that are held constant throughout the experiment. Independent and dependent variables Research question Independent variables Dependent variables Do tomatoes grow fastest under fluorescent, incandescent, or natural light? Does pasteurization keep fruit juice safe for consumption? In experimental research, the independent variable is manipulated or changed by the experimenter to measure the effect of this change on the dependent variable. Control variables? Visualizing independent and dependent variables Researchers often use charts or graphs to visualize the results of their studies. • Independent variable: “x” or horizontal axis • Dependent variable: “y” or vertical axis Research question: Does a new medication affect blood pressure? https://www.scribbr.com/methodology/experimental-design/ Visualizing independent and dependent variables Researchers often use charts or graphs to visualize the results of their studies. • Independent variable: “x” or horizontal axis • Dependent variable: “y” or vertical axis 140 Could pre-blanching preserve antioxidants of basil leaves during hot-air drying? No blanching Total phenolic content Research question: Blanching 120 100 80 60 40 20 0 50 60 70 Drying temperature (oC) 80 Qualitative vs. quantitative variables (data) Qualitative, or Attribute, or Categorical, Variable: • A variable that categorizes or describes an element of a population. Quantitative, or Numerical, Variable • A variable that quantifies an element of a population. Example Identify each of the following examples as attribute (qualitative) or numerical (quantitative) variables. 1. The residence hall for each student in a statistics class. 2. The amount of gasoline pumped by the next 10 customers at the local Unimart. 3. The amount of radon in the basement of each of 25 homes in a new development. 4. The color of the baseball cap worn by each of 20 students. 5. The length of time to complete a mathematics homework assignment. 6. The state in which each truck is registered when stopped and inspected at a weigh station. Qualitative vs. quantitative variables (data) Binomial Qualitative Nominal Ordinal Variable Quantitative Discrete Continuous Qualitative variable Binomial: place things in one of two mutually exclusive categories: right/wrong, true/false, or accept/reject. Nominal: assign individual items to named categories that do not have an implicit or natural value or rank. Example: record color of candies Ordinal: items are assigned to categories that do have some kind of implicit or natural order. Example is a survey question that asks us to rate an item on a 1 to 10 scale, with 10 being the best. This implies that 10 is better than 9, which is better than 8, and so on. Quantitative variable Discrete: data is a count that can't be made more precise. Typically it involves integers. For instance, the number of children (or adults, or pets) in your family is discrete data, because you are counting whole, indivisible entities: you can't have 2.5 kids, or 1.3 pets. Continuous: data could be divided and reduced to finer and finer levels. For example, you can measure the height of your kids at progressively more precise scales—meters, centimeters, millimeters, and beyond—so height is continuous data. Case study 1.1: observational study Bingham, Sheila A., Nicholas E. Day, Robert Luben, Pietro Ferrari, Nadia Slimani, Teresa Norat, Françoise Clavel-Chapelon et al. "Dietary fibre in food and protection against colorectal cancer in the European Prospective Investigation into Cancer and Nutrition (EPIC): an observational study." The lancet 361, no. 9368 (2003): 1496-1501. Case study 1.2: experimental study Leidy, Heather J., Cheryl LH Armstrong, Minghua Tang, Richard D. Mattes, and Wayne W. Campbell. "The influence of higher protein intake and greater eating frequency on appetite control in overweight and obese men." Obesity 18, no. 9 (2010): 1725-1732. Confounding variables In research that investigates a potential cause-and-effect relationship, a confounding variable is an unmeasured third variable that influences both the supposed cause and the supposed effect. • It must be correlated with the independent variable. This may be a causal relationship, but it does not have to be. • It must be causally related to the dependent variable. It’s important to consider potential confounding variables and account for them in your research design to ensure your results are valid. Confounding variables https://www.scribbr.com/methodology/experimental-design/ Why confounding variables matter? To ensure the internal validity of your research • If not, your results may not reflect the actual relationship between the variables that you are interested in Example You find that babies born to mothers who smoked during their pregnancies weigh significantly less than those born to non-smoking mothers. However, if you do not account for the fact that smokers are more likely to engage in other unhealthy behaviors, such as drinking or eating less healthy foods, then you might overestimate the relationship between smoking and low birth weight. Accuracy vs Precision Accuracy is how close a measured value is to the actual (true) value Precision is how close the measured values are to each other Reliability vs validity Reliability refers to how consistently a method measures something. If the same result can be consistently achieved by using the same methods under the same circumstances, the measurement is considered reliable. Validity refers to how accurately a method measures what it is intended to measure. If research has high validity, that means it produces results that correspond to real properties, characteristics, and variations in the physical or social world. Reliability vs validity https://outmatch.com/resources/blog/pre-employment-assessments-validity-vs-reliability/ Reliability vs validity Example: reliable but not valid You measure the temperature of a liquid sample several times under identical conditions. The thermometer displays the same temperature every time, so the results are reliable. The thermometer that you used to test the sample gives reliable results. However, the thermometer has not been calibrated properly, so the result is 2 degrees lower than the true value. Therefore, the measurement is not valid. How to ensure validity and reliability in your research Ensuring validity Choose appropriate methods of measurement Use appropriate sampling methods to select your subjects How to ensure validity and reliability in your research Ensuring reliability Apply your methods consistently Standardize the conditions of your research Internal vs external validity Internal validity (the design of experiment): refers to the degree of confidence that the causal relationship being tested is trustworthy and not influenced by other factors or variables. External validity (the generalizability of the results): refers to the extent to which results from a study can be applied (generalized) to other situations, groups or events. Generalizability Truth in the Study Truth in Real Life Internal validity External validity Trade-off between internal and external validity Better internal validity often comes at the expense of external validity (and vice versa). The type of study you choose reflects the priorities of your research. Factor Laboratory experiment Field experiment Artificial Realistic Control High Low Internal validity High Low External validity Low High Time required Short Long Number of subjects Small Large Ease of administration High Low Cost Low High Environment Solution: conduct the research first in a controlled (artificial) environment to establish the existence of a causal relationship, followed by a field experiment to analyze if the results hold in the real world. How to reduce the impact of confounding variables Three basic principles of statistical experimental design Replication Blocking Randomization Replication What? • Replicates are multiple experimental runs with the same factor settings (levels). Why? • Ensure validity • Account for error • Define the outlier Difference between replicates and repeats • Repeat measurements: taken during the same experimental run or consecutive runs • Replicate measurements: are taken during identical but different experimental runs, which are often randomized Replication How many replicates? The more the better? What is the average height of a man in Germany? What is the average height of a man in Germany? Blocking Blocking is the arrangement of experiments into groups (called also blocks) • A block is characterized by a set of homogeneous plots or a set of similar experimental units. • Failure to block is a common flaw in designing an experiment. Blocking Blocking is a technique for dealing with nuisance factors. • A nuisance factor is a factor that has some effect on the response, but is of no interest to the experimenter; however, the variability it transmits to the response needs to be minimized or explained • Typical nuisance factors include batches of raw material if you are in a production situation, different experimenters or subjects in studies, the pieces of test equipment, when studying a process, and time (shifts, days, etc.) where the time of the day or the shift can be a factor that influences the response. Nuisance vs. confounding factors/variables Confounding variable: Variable that changes concomitantly with the independent variable. The dependent variable is affected by both the independent variable and the confounding variable. Nuisance variable: Variable that causes dependent variable scores to be more variable Blocking principle If the nuisance variable is known and controllable, we use blocking and control it by including a blocking factor in our experiment. If you have a nuisance factor that is known but uncontrollable, sometimes we can use analysis of covariance to measure and remove the effect of the nuisance factor from the analysis. If nuisance factors are unknown and uncontrollable (sometimes called a “lurking” variable), we use randomization to balance out their impact. The general rule is: “Block what you can; randomize what you cannot.” Randomization Randomization is an insurance against a systematic bias due to a nuisance factor • For example, selection bias (where some groups are underrepresented) is eliminated and accidental bias (where chance imbalances happen) is minimized. Example: choose panelists for food sensory tests Correlation and Regression Analysis https://byjus.com/maths/correlation-and-regression/ Correlation Correlation is a measure of the degree to which two random variables, (which are independent of each other), vary together or are associated. Example: • Determining the association of a food quality attribute measured by two methods (e.g. volumetric and instrumental methods) • Association between sensory textures obtained from panelists and instrumental measurements. Note: Correlation is not about a cause-and-effect relationship Correlation The association between only two random variables is called simple linear correlation. The degree of this association is measured by a quantity known as Pearson correlation coefficient (r). ∑ = ∑ − ∑ − ∑ ∑ × ∑ − ∑ Correlation r always lies between -1 and +1 r close to absolute 1 indicates that a very strong linear relationship exists between the two variables When r is equal to zero then there is no linear relationship between the two variables. Positive value of r indicates that the two variables tend to increase together Negative value of r on the other hand shows that large value of one variable is associated with small value of the other variable. Correlation https://ecstep.com/correlation-regression-analysis/ Correlation example The nutrient retention of different pre-processing treatments on carrot and sweet potato samples was studied. The research team was interested in finding out the association between the nutrient retention of the two roots. The following data was obtained from the two roots. Sample Nutrient retention of root samples Sweet potato (X) Carrot (Y) 1 14 7 2 12 8 3 11 7 4 10 8 5 9 9 6 9 11 7 8 12 8 7 14 9 6 16 1. What could be the hypotheses that will be helpful in testing the relationship of nutrient retention of the two roots? 2. Compute the coefficient of correlation 3. Determine the kind and significance of the association 4. What practical meaning does this relationship have? 20 Correlation example Nutrient retention of root samples Carrot Sample 16 12 8 Sweet potato (X) Carrot (Y) 1 14 7 4 2 12 8 0 3 11 7 4 10 8 5 9 9 6 9 11 7 8 12 8 7 14 9 6 16 Homework: Draw a graph in excel Data input in SPSS Output for correlation in SPSS 0 4 8 12 Sweet potato Result from SPSS 16 Extra correlation example The β-carotene content of carrot samples subjected to different treatments was determined by two different methods i.e.: UV vis. Spectrophotometer and HPLC. The researcher wanted to know if the trends in the β-carotene obtained by the two methods have some association. The researcher arranged as the one in the Table below. β-carotene content (mg/100 g) Spectrophotometer(X) HPLC (Y) 1 8.4 12.5 2 8.1 12 3 7.6 11.3 4 6.9 10.7 5 6.3 9.8 6 5.8 8.4 7 5.2 7.8 8 4.7 6.6 1. State hypothesis 2. Determine the coefficient of correlation 3. Test the significance of the correlation of there is any and identify the kind. 4. What is the practical meaning of the relationship? HPLC Sample 14 12 10 8 6 4 2 0 0 2 4 6 8 10 Spectrophotometer Regression Analysis In regression, we express the relationship of one variable to another by an equation (regression model) that describes one as a function (linear in the simplest case) of the other variable. In regression analysis, the dependent variable is denoted "y" and the independent variables are denoted by "x" The most basic assumption underlying regression analysis is that there is unilateral casual relation between X and Y. That is, variation in X results in variation in Y. However, variation in Y does not result in variation in X. Example: = + = + + +⋯+ + Regression Model Adequacy checking Coefficient of determination (R2 and R2adj). Distribution of residuals (residual plots) Root mean square error (RMSE) Percentage of root mean square error Coefficient of determination Coefficient of determination explains proportion of variability The value of R2 is between 0 and 1. R2 value of 1 means the model perfectly predicts the experimental data whereas a 0 value indicates a totally faulty prediction. In practical application R2 value less than 0.75 usually indicates an insufficiently precise description of the experimental data. Coefficient of determination R2 overestimates the goodness of fit and it is usually modified into adjusted R2 (R2adj) by taking the degree of freedom of the model into account R2adj is very important in comparing adequacy of different models. Distribution of residuals (residual plots) Residual: difference between the predicted and experimental values Plots close to 0 means the experimental and the estimated values are equal and this indicates the goodness or adequacy of the model Root mean square error (RMSE) What is the unit of RMSE? Correlation and Regression Analysis