Introduction to Research Methods Q560: Experimental Methods in Cognitive Science Lecture 2 A CogSci major named White, an Informatics major named Black, and a Psyc major named Red meet for a coffee. One of the three says “I’m wearing a black hat, and you two are wearing a red and a white hat, respectively, but none of us has a hat color that matches his name.” White responds “you are quite correct.” What color is the Psyc major’s hat? Harvard Medical Review (1971): Hypothesis: Coffee consumption causes urinary tract cancer Data were collected from a group of patients, some with urinary tract cancer, some with other diseases. Also, family history, age, gender, etc. Not everyone who drinks coffee gets urinary tract cancer, and not everyone with urinary tract cancer is a coffee drinker…so it would seem that the evidence contradicts the hypothesis However, 25% of cancer patients habitually drink coffee, compared to only 10% of those without cancer…so there would also seem to be some evidence in favor of the hypothesis Experimental Methods in CogSci “Experimental methods” is a bit overly-restrictive Exploratory research is a fundamental component of theory building; experiments test theory Some early experiments: Very early: statisticum collegium and statista 1747: James Lind’s scurvy “experiment” 1920: Fisher’s classic “lady tasting tea” Lind’s Scurvy “Experiment” 1747: as surgeon on HMS Salisbury, Lind conducted a controlled experiment to develop a cure for scurvy At the time, the concept of vitamins was unknown Selected 12 men “as similar as I could have them” A quart of cider every day 25 drops of sulphuric acid 3x day ½ pint of seawater every day Mixture of garlic, mustard, and horseradish 2 spoons of vinegar 3x a day 2 oranges + 1 lemon every day Fisher’s Lady Tasting Tea 1920: Hanging out on the weekend at Cambridge bored… The famous “lady” in question was Dr. Muriel Bristol Can she tell, by taste alone, whether the tea or the milk was added first to the cup? ↳ ↳ ↳ ↳ 4 cups of each type….how to present them? Blindfolded Need to control portions Could 8/8 be due to guessing alone? ↳ ↳ How many need to be correct to reject this assumption? Using Fisher’s exact test, 1/70 chance of guessing ● Led to Statistical Methods for Research Workers (1925) and The Design of Experiments (1935) ● Salsburg, D. (2001). The lady tasting tea: How statistics revolutionized science in the twentieth century. Owl Books Research in Cognitive Science 1. The role of theory 2. The role of exploratory research 3. The role of hypotheses 4. The role of experiments 5. The role of statistics 6. Iterative science 1. The Role of Theory Theories have two roles in CogSci: 1. As a framework to tie together isolated findings and to understand cognitive phenomena 2. They allow us to generate new hypotheses to test Armchair vs. verbal conceptual vs. formal theories: CogSci focuses on formal models One is not “better” …they should be different stages Formal models: Descriptive Predictive Explanatory 1. The Role of Theory “But I have observed over the years that there is a tendency for even the best cognitive scientists to lose sight of large issues in their devotion to particular methodologies, their pursuit of the null hypothesis, and their rigorous efforts to reduce anything that seems interesting to something else that is not. An occasional reminder of why we flash those stimuli and measure those reaction times is sometimes useful.” --George Miller, Psychological Science, 1990. Estes (1975): 2. The Role of Exploratory Research There is nothing wrong with: Introspection /self report Naturalistic observation/case studies Surveys Correlational Designs …as long as you don’t stop there Exploratory methods are a necessary component to theory building, and in designing contrastive experiments 3. The Role of Hypotheses Inductive method: In the past, theories were tested by gathering data that supported them; now, we try to prove ourselves wrong. NSHT: Hypothesis is not a hunch…it is a testable prediction from theory Probabilistic statement tied to a model of chance Mommy seems to call everything with four legs a doggie: H1: All four-legged things are doggies H0: At least one four-legged thing is not a doggie Note that most Bayesians hate NSHT! 4. The Role of Experiments Experiment is designed to test the hypothesis: Requires manipulation, control, and causality The manipulated factor is the only free variable that could produce the change in the outcome variable Observed outcome is either due to chance (H0) or the manipulation (H1) Cf. Our shark beacon researcher… 5. The Role of Statistics Statistical models allow us to efficiently summarize outcomes, and determine whether the outcome is due to chance Two main branches of Statistics: 1. Descriptive Statistics: Summarizing and communicating information about a group of numbers (data) 2. Inferential Statistics: Drawing conclusions based on the data collected, and making predictions that go beyond the immediate data 6. Iterative Science The results of the experiment are used to reject, prune, or refine the theories used to generate the hypotheses i.e. ....back to step 1 Research Approaches Field vs. Laboratory Studies Nomothetic vs. Ideographic approaches: Criticisms: Frankfurt school, Estes, etc. Hypothesis is not a hunch…it is a testable prediction from theory Non-experimental (desc), quasi, experimental The quest for causation Descriptive Techniques Non-experimental methods describe behavior, but do not let us identify the causes or reasons for the behavior They are useful when you can’t ethically manipulate a variable, or want to simply describe/predict behavior Descriptive techniques are the backbone of many fields (e.g., medical science, meteorology, political science, etc.) Descriptive Techniques 1. Case Studies 2. Observational Methods: I observed squirrels masturbating for 2000h with 10x50 binoculars from trees orObservation a vehicle situated within 40 m of the perimeter of a Naturalistic (Qualitative) burrow cluster. Activities and masturbation frequency among • males Requires wereimmersion recorded using scan sampling at 5-min intervals… • Goal is to provide complete + accurate picture, not to test hypotheses • Issues of reactivity Systematic Observation (Quantitative) • Requires a predefined coding system Descriptive Techniques 3. Self-Report Measures: What people say rather than what they do If interested in opinions or social perceptions Demand characteristics (cues) + biases Note: We often don’t know how/why we do things Interviews vs. questionnaires + surveys: only as good as questions 4. Correlational Research: Correlation is not causation? Direction and 3rd variable problems Quasi-Experimental Techniques A true experiment requires random assignment Often, this is impossible or unethical Some examples: Non-equivalent control groups design NECG-pretest-posttest design Interrupted time-series design Control series design Experimental Techniques Goal is to establish causal relation between two variables Requires manipulation, randomization, and control Experimental method: A constant is a characteristic that is fixed across conditions A variable is a characteristic that changes across conditions To make inferences, we manipulate a variable of interest, and observe the effect on an outcome variable, holding all other variables constant (cf. shark beacon study) Independent variable is manipulated, dependent variable is measured Types of IVs: Between-subjects (independent measures) Within-subjects (repeated-measures) Components of the Classic Experiment Control condition (no treatment) Experimental condition (receives treatment) Random assignment of subjects to conditions Extraneous variables: Random and systematic Confounding variables (cf. shark beacon)….source of error in interpretation Goal: reduce randoms and eliminate confounds via systematic control, design, and randomization An example w/ Gordon Bower’s classic imagery experiment Extraneous Variables Situation and participant variables: E.g., timing, systematic selection, etc. Systematic and random (cf. shark beacon study) Hawthorne Effects: You behave differently when being observed Dangers of hypothesis guessing Transparency and single-blind conditions Experimenter Effects (Rosenthal): Confirmation bias, self-fulfilling prophecy Double-blind studies Between-Subjects Designs Independent groups of participants are assigned to the different levels of the IV The variable takes on different values across individuals Pros: Controls for timing variables, can test everyone at once Do not need equivalent sized groups Cons: Variability due to individual differences Need more subjects (more than twice as many!) Within-Subjects Designs Each participant is measured under all levels of the IV The variable takes on different values within individuals Pros: More sensitive, less random noise Fewer subjects are needed Cons: Timing confounds: History and maturation Carryover effects Practice, sensitization, fatigue Solutions: Counterbalancing, randomized blocks, Latin squares Fundamentals of Measurement Qualitative (categorical) and quantitative (numerical) Discrete vs. continuous variables Stevens’ scales of measurement: Nominal Ordinal Interval Ratio Fundamentals of Measurement A discrete variable consists of separate, indivisible categories. No values can exist between two adjacent categories Example: political party, dog breed, number of children in household, etc. A continuous variable has an infinite number of possible values between two adjacent values Example: time, weight, pupil diameter, etc. Scales of Measurement 1. Nominal Scale: Values are be categories that differ only in name Example: Gender, ethnicity, eye color, numbers on race cars Values on a nominal scale only represent different categories, and may not be averaged Scales of Measurement 2. Ordinal Scale: Values are categories organized in an ordered sequence (ranks) Example: Places in a race, Olympic medals, etc. Values on an ordinal scale are nominal, but also contain a greater than/less than relationship between values on the scale. However, you cannot determine the magnitude of the relationships Scales of Measurement 3. Interval Scale: Ordered categories that are all intervals of exactly the same size Example: Temperature in Fahrenheit, IQ scores Differences between intervals are meaningful, but ratios are not (because there is no absolute zero) Scales of Measurement 4. Ratio Scale: An interval scale with an absolute zero point Example: Reaction time, height, errors on a test, temperature in Kelvin All the qualities of an interval scale, but ratios of numbers reflect ratios of magnitude (because the zero reflects a true absence of the variable being measured) Scales of Measurement 1. Do different values denote different categories? 2. Are larger values really “more” of the variable than smaller values? 3. Are there equal intervals between values? 4. Is zero on the scale really an absence of the variable? NOIR Scales of Measurement 1. Numbers on basketball jerseys Nominal Discrete 2. Sizes of McDonald’s drinks/fries Ordinal Discrete 3. Weight Ratio Continuous 4. Scores in Jeopardy Interval Continuous 5. Golf scores Interval Discrete Bar Graphs and Histograms Frequency Frequency 40 0 1 2 3 4 5 6 7 8 9 10 30 20 10 0 Seconds blue grey green brown Eye color Histogram (left) for continuous variables Bar graph (right) for discrete variables Populations and Samples Observations are usually made on individuals. A population is the set of all the individuals of interest. Populations are often so large that it is impossible to obtain measurements from all the individuals A sample is a set of individuals selected from a population – we usually want samples to be representative (not biased) Chris H Praful Justin Trinity Mary Ellen Nicole Kim Sean Frank George Sarah John S Greg Erin Jim James Sorab Ruben Trevor Ruben Tank Rich Ji David Will Cory Steve Sean Justin John L Ricky Art Matt Chris J Royce June Ruben Tom Pete Alex T Dennis Alex K Grant Sue Jhung Ruben Ruben Chen Jullian Bubbles Nathan Gillian Brad Chuck Vera Amanda Tessa Sample Sampling Dennis James Sue Erin June Xiangen Brenda Xiangen Hillary Sample should be All CogSci Students (Population) • Representative • Generalizable The Population All Individuals of Interest Sampling Results from the sample are generalized back to the population The Sample Individuals selected for study Parameters and Statistics A parameter describes a population A statistic describes a sample Parameter • Average GPA for all U.S. university students • Average height of all CogSci students Statistic • Average GPA for IU students • Average height for this class • We use a statistic to estimate a parameter • Generally, Greek letters denote parameters, and Roman letter denote statistics The Population Average Height = 5’9’’ Inferential Statistics: Sampling How good an estimate of the parameter is the statistic? The Sample (n=60) Average Height = 5’6’’ The Population Average Height = 5’9’’ Inferential Statistics: Sampling How good an estimate of the parameter is the statistic? The Sample (n=120) Average Height = 5’10’’ Sampling Error: The discrepancy between the sample statistic and the true population parameter it is estimating Sampling Error Sampling Error: The discrepancy between the sample statistic and the true population parameter it is estimating To reduce sampling error: • Use a sufficiently large sample • Use random selection: selecting individuals from the population at random for your sample to create an unbiased sample (sometimes bias is subtle— telephone survey example) Statistical Truth: We can only “prove” something if we can measure the population. As a result of sampling error, we can only ever determine “beyond a reasonable doubt” Sampling Error Random Sampling Systematic Sampling Quota Sampling Opportunity Sampling Volunteer Sampling Snowball Sampling