The tradeoff between internal and external validity

• In natural sciences, most experiments have been undertaken in carefully controlled laboratory conditions
– Seeking laws of physics, etc.
• Inanimate objects are unlikely to react differently in controlled and uncontrolled conditions
• The main goal is to avoid contaminating the experiment via the introduction of third variables
• Far more exacting and finer measurement is possible under controlled conditions

• Closely controlled social science experiments do have the advantage of limiting the impact of third variables, but the unnatural situation presents problems for applying the findings outside the laboratory
– “Artificiality” of laboratory setting

• Human behavior is sensitive to the environment within which it occurs
– People act differently in a laboratory than in the natural world
• Several characteristics of laboratories are thought to influence behavior
• The very third variables controlled in the lab may be the ones that determine behavior in the real world
• So, findings from laboratory experiments may only be valid for laboratory environments

• New commercials are tested in controlled conditions
– Eye tracking
– Liking for commercials
– Influence on purchase interest
• May try to provide less artificial conditions for study
– Simulated living room
• Commercials that test high in lab experiments often do not work very well when used in real marketing campaigns

• Researchers want to retain some of the advantages of the experiment:
– Ability to manipulate/introduce the independent variable and to control how much of it is presented
– Time order—which comes first
• While sacrificing some of their ability to control third variables
• The goal is to improve our ability to generalize our findings to the real world

• One way to do so is to carry out a ‘field experiment’
– The researcher still manipulates the independent variable, but she does so within the natural world
– For example, police try randomly stopping motorists and testing for alcohol to see if alcoholrelated crashes decline

• BehaviorScan system

• Greatly reduced ability to prevent third variable contamination of results
– The crusading doctor in MHHP
– History effects
• Cycles
• What can you do?
– Measure/monitor likely alternative explanatory variables
– Question subjects about sources of influence
– Multiple manipulations of the independent variable over time
– Multiple measurement waves
– Multiple dependent measures
• None will be perfect

• Expense
– Tradeoff between extensive and intensive study
– Budget constraints on number of sites, etc.
• Access/permission
– Some research may present concerns to authorities, citizens, etc.
• Gain authorization/support prior to entering the field
• Maintain good relationships with community leaders, etc. throughout the intervention/research

• Sometimes unusual or unique events occur
– ALAR scare
– Service disruptions
– Political campaigns
– School shooting
– Institution of Wi-Fi in an area

• Because most such events are unplanned, the ability to prepare for them is limited
– May keep a research group, materials and resources ready for certain types of events
• Must engage in ‘firehouse research’ gathering as much data as possible in a short time
– Inefficient, and may miss important data
– However, real-world events, etc. may provide very valuable data—may have both internal and external validity

• Use statistics, research analysis models that are appropriate to experimental studies