Foundations of Research Descriptive research This is a PowerPoint Show Click “slide show” to start it. Click through it by pressing any key. Focus & think about each point; do not just passively click. To print: Click “File” then “Print…”. Under “print what” click “handouts (6 slides per page)”. © Dr. David J. McKirnan, 2014 The University of Illinois Chicago McKirnanUIC@gmail.com Do not use or reproduce without permission. 1 2/29/16 Foundations of Research Forms of descriptive research Qualitative or Observational Quantitative Describe an issue via valid & reliable numerical measures Study behavior “in nature” (high Simple: frequency Qualitative counts of key behavior “Blocking” by other variables Correlational research: “what relates to what” Complex modeling 2 ecological validity). In-depth interviews Focus (or other) groups Textual analysis Qualitative quantitative Observational Direct Unobtrusive Existing data Use existing data for new quantitative (or qualitative) analyses Accretion Study “remnants” of behavior Wholly non-reactive Archival Use existing data to test new hypothesis Typically nonreactive Foundations of Research 3 Forms of descriptive research Quantitative Qualitative or Observational Existing data Describe an issue via valid & reliable numerical measures Study behavior “in nature” (high ecological validity). Use existing data for new quantitative (or qualitative) analyses Simple: Qualitative Accretion frequency counts of key behavior “Blocking” by other variables Correlational research: “what relates to what” Complex modeling In-depth interviews Focus (or other) groups Textual analysis Qualitative quantitative Direct Unobtrusive Wholly non-reactive Archival Use existing data to test new hypothesis Typically non-reactive Observational Study “remnants” of behavior Foundations of Research 4 Examples of descriptive data Simple description: how much alcohol and drugs do gay/bisexual men consume? 70 60 Very high rates of simple use Much lower rates of heavy use Drugs increasing, alcohol decreasing 1999 -> 2001 50 40 30 20 10 0 Ma Cra Alc C Me MD He Oth Da riju oca r yu t c oh M o h e k i r . A n sed ana ine ol i nto xic atio n Any, 6 mo. McKirnan, D., et al., 2001 community sample > 3 days / month Foundations of Research 5 Examples of descriptive data; blocking variable More complex description: blocking alcohol & drug use by ethnicity. 40 % of participants 35 All ps <.01 except alcohol use. 30 25 Whites show more alcohol & drug use on most measures. Other ethnic differences vary by drug. 20 15 10 5 0 An ys ub s Al co tan ho l ce African-Am., n=430 Ma rij u Ot h an a er d ru g Latino, n = 130 Al -d ru g s s+ se x White, n = 183 2001 Community data: Ethnic differences in frequent (> 3 days/month) drug & alcohol use. McKirnan, D., et al., 2001 community sample Foundations of Examples of descriptive data; Simple correlation of measured variables. Research Testing exploratory hypotheses in descriptive data: Drug use by Quasi - Depression Groups 60 Participants are blocked (post-hoc) on a standard measure of depression: 0 or 1 symptom v. 3 or more symptoms. Men with more symptoms use all forms of drugs more often. All effects p<.005 % of participants 50 40 30 20 10 0 Any use Any freq. use Freq. Alch. Intox. 0 - 1 symptom Freq. Maj. Freq. 'Hard' drugs > 2 symptoms 0/1 symptoms n = 391, > 2 symptoms n = 289. “Frequent” > 3 days / month. 6 Foundations of Research 7 Forms of descriptive research Quantitative Qualitative or Observational Existing data Describe an issue via valid & reliable numerical measures Study behavior “in nature” (high ecological validity). Use existing data for new quantitative (or qualitative) analyses Simple: frequency Qualitative Accretion counts of key behavior “Blocking” by other variables Correlational In-depth interviews Focus (or other) groups Textual analysis Qualitative quantitative research: “what relates to what” Observational Direct Complex modeling Unobtrusive Study “remnants” of behavior Wholly non-reactive Archival Use existing data to test new hypothesis Typically non-reactive Foundations of Research Testing hypotheses with simple correlations: 8 Naturally occurring events: Correlational designs Brain basis for addiction: test correlation between “sensation seeking” and drug abuse Trauma theory of depression: correlate reports of childhood abuse with scores on a depression measure… Procedures: Careful selection of sample to reflect target population Systematic development of measurements: Reliability Core virtues: Validity “Natural” look at how variables relate Less control = less reactivity than experimental designs Can model very complex phenomena Foundations of Research Does ice cream cause people to drown? Drownings This shows a simple correlation. How might you interpret these data? Ice cream consumption (scoops / day). 9 Foundations of Research 10 Correlation designs: Drawbacks & fixes Causality; a simple correlation may confuse cause & effect. Negative affect ? Marijuana consumption Confounds!; an unmeasured 3rd variable may influence both observed measures. Levels of the neurotransmitter anandamide? ? Negative affect Marijuana consumption Dealing with confounds: Use complex measurements or samples to eliminate alternate hypotheses. Learn about anandamide. Foundations of Research Complex correlation design: Mothers’ earnings Does having a child at an earlier age cause a woman earn less? Having a 1st child @ age 24 v. 25 correlates with 10% lower lifetime earnings. Lower base salary Smaller raises x earning lifetime What causes this? Main hypothesis: Getting pregnant earlier has a big financial cost, due to the simple burden of motherhood. Child at an earlier age. Slate.com, The Price of Motherhood. Burden of earlier motherhood. Poorer lifetime earning. 11 Foundations of Research 12 Motherhood and income. Is there an alternate hypothesis? Why else might women who have a child earlier earn less? Could a 3rd variable explain both earlier childbirth and earning potential? Alternate explanation Child at an earlier age Burden of earlier motherhood Poorer lifetime earning. Foundations of Research 13 Motherhood and income. A major alternate hypothesis would be that certain personal characteristics of women lead them to both early birth and less $... …instead of the disadvantage of getting pregnant earlier in life. To support the original hypothesis the researchers tested – and eliminated – two alternate explanations. Alternate explanation Child at an earlier age Burden of earlier motherhood Poorer lifetime earning. Foundations of Research 14 Alternate hypothesis 1: Less personal ambition or skills: …may lead women to get pregnant earlier and have less earnings. Personal characteristics (less ambition / skills) Decision to have a child earlier. Burden of earlier motherhood Poorer lifetime earning. The researchers tested this alternate hypothesis by examining different subgroups within their data Foundations of Research 15 Test: Compare women who started a family at 24 to women who tried to start at 24, miscarried, started at 25. Alternate hypothesis 1: Women who tried to start at 24 but failed should have the same characteristics as those who started at 24; If the alternate hypothesis is correct, this specific comparison should eliminate the 10% differential. Data: Comparison still showed a 10% earnings decrement; the alternate hypothesis was not supported. Foundations of Research 16 Alternate hypothesis 2: Personal importance of motherhood: Women with strong motherhood values may both get pregnant early and not value a career. Personal importance of Motherhood Decision to have a child earlier. Burden of earlier motherhood Poorer lifetime earning. The researchers again used subsets of the data to test if this explanation is better than the original hypothesis Foundations of Research Alternate hypothesis 2: Test: women who had been trying to get pregnant since they were 23. Some succeeded at 24; others at 25. 17 Age of pregnancy was random, so “motherhood value” should be the same in each group. If the alternate hypothesis is correct, this specific comparison should eliminate the 10% differential. Data: Comparison still showed a 10% difference; 3rd variable “value” hypothesis was not supported. Foundations of Research 18 Bottom line: testing causality in correlational data: The simple correlation between age at 1st pregnancy & income suggests that the simple burden of having children earlier costs. Alternate 3rd variable hypotheses question whether the age of 1st pregnancy really caused lower economic performance. It could be women’s job skills or commitment …or the value she places on motherhood By comparing specific sub-samples from her data… ...she was able to test & refute alternate hypotheses about women’s personal characteristics. Alternate explanations Child at an earlier age Burden of earlier motherhood Poorer lifetime earning. Foundations of Research 19 Forms of descriptive research Quantitative Qualitative or Observational Existing data Describe an issue via valid & reliable numerical measures Study behavior “in nature” (high ecological validity). Use existing data for new quantitative (or qualitative) analyses Simple: frequency Qualitative Accretion counts of key behavior research: “what relates to what” Interviews, focus groups, textal analysis Qualitative quantitative Complex modeling Observational “Blocking” by other variables Correlational Direct Unobtrusive Study “remnants” of behavior Wholly non-reactive Archival Use existing data to test new hypothesis Typically non-reactive Foundations of Research Experimental v. Observational research https://statswithcats.wordpress.com/2015/01/01/how-to-tell-if-correlation-implies-causation/ 20 Foundations of Research Qualitative research 21 Key feature: Data are unstructured or “natural” Assess participants’ own thoughts or descriptions Interview / collect data in participants’ own environment, using field studies Less influenced by researchers’ hypotheses or structured measures Key uses: “Ground” research in the every-day reality of people. Describe the social or physical context of a behavior Generate hypotheses Provide a deeper understanding of lab or quantitative findings. Foundations of Research Approaches to qualitative data 22 Structured / guided description Qualitative / semi-structured interviews face to face, telephone, “Street intercept” Open-ended questions Guided analysis of behavior: “deconstruct” an event… Take me through the last time you drank any alcohol… What day was it? Time? Where were you?, what was the place like? Who were you with … family? Friends? Boy/girl friend? Strangers? What were you doing / what was going on… …etc. Foundations of Research Qualitative data: guided description Approaches to qualitative data Structured / guided description Qualitative / semi-structured interviews Focus groups Computer programs & raters categorize and count specific types of responses within the text. Textual analysis Use computer or expert raters to analyze existing text e.g., political writings, therapy transcripts, correspondence Analyze “found text” e.g., diary entries, suicide notes 23 Foundations of Example of qualitative - quantitative research: Research Rafael Diaz’s study of Latino meth use. Empirical questions: What % of Latino gay men use stimulants? Methamphetamine Cocaine Other What does stimulant use “mean” for men? – what are their motives or understandings? How does the meaning of drug use differ for meth v. cocaine? How do these concepts and attitudes affect drug use? Sexual or other risks & harms? Amount of drugs? 24 Foundations of Research Diaz study: Qualitative quantitative approach 1. 2-hour qualitative semi-structured interview with 70 drug-using Latino gay men: Detailed qualitative description of drug use & sexual activity behavior social contexts reasons for use perceived effects Narratives on specific episodes of drug use with and without sexual activity with and without condom use. 2. Used qualitative findings to develop and test a survey instrument Different dimensions of stimulant use Relationship between stimulant use and HIV risk. 3. Administered revised survey to random sample of Latino gay men (n=300) who reported stimulant use. 25 Foundations of Research Diaz study: qualitative findings Reported positive reasons for using meth / speed: Energy We each did a line of crystal because I was feeling sleepy. I was yawning. It wasn’t that I didn’t want to go out, I think I was physically just exhausted from the week. It was just long, and so that kind of gave me a boost of energy. Youthfulness, attractiveness [With crystal] I find that I am no longer pudgy and plump. I feel that I’m a little bit more physically attractive because I’m not overweight. 26 Foundations of Research Diaz study: qualitative findings, 2, Positive Reasons Reported sexual effects of meth / speed: Sexuality I felt like it rushed to my brain, I felt my skin get hot and I felt the desire to have sex with whomever was around… Sexual disinhibition I become even more hardcore. Sexual risks and inhibitions are totally gone. I become empowered in feeling, like I can take on the world or anyone that f_ _ _ed with me. It can be an euphoric rush. Sexual risk With drugs you start degenerating and you no longer are satisfied with one person…you want another and you want more and you want them all at the same time. So I do see a relationship, drugs do lead to becoming infected with diseases 27 Foundations of Research Diaz study: qualitative findings Reported negative consequences of meth / speed: Paranoia That also makes me want to stop because I have been feeling this horror of someone who is following me, uh… who wants to kill me or that is hiding but is following me. Social isolation … I was in another world... where at times you lose all shame, you lose friends, family, you lose... everything. Sometimes I wouldn't even make a phone call, all I cared about was getting high and that was it. Physical depletion I feel so gross that I can’t wash it off anymore. It’s like you feel like this inside dirty, like because there’s no food in your stomach for the past days, you’ve been just like running on empty and like you’re really gaunt now because you’ve been in a constant workout. 28 Diaz study: Quantitative analysis of qualitative findings Foundations of Research Develop conceptual categories by coders using the qualitative data. Then go back and have the computer search each interview for key words to count the % of men who mentioned each topic We can then use quantitative analyses to test hypotheses about differences between drugs… 29 Foundations of Diaz study: Qualitative findings, 2 Examining many categories of impacts weQuantitative can see that: Research Many stimulant users have important negative life effects, Significantly more so for meth. than for cocaine. 30 Diaz study: Quantitative phase Foundations of Research 1. Using key words from the qualitative phase, create quantitative closed-ended survey items. 2. Administer the quantitative survey to a much larger sample of men. 3. Use statistical tests to: a. Ensure the items are reliable and internally valid; b. Test theory-driven hypotheses about drug use and personal harms. 4. Publish the survey instrument for replicating studies by other researchers. Meth makes me not feel left out… Meth makes me feel better emotionally… 31 Foundations of Research Typically uses direct interviews, focus groups.. Structured: specific questions driven by research topic or hypothesis Semi-structured: general / probing questions guided by general topic Summary: Qualitative research Unstructured: “personal biography”; completely person centered. Important primary data source: Direct, in-depth measure of behavioral process Less biased by researcher’s hypothesis than a survey Important step in quantitative research: Generate hypothesis or theory of new phenomenon Produce externally [ecologically] valid qualitative assessments 32 Foundations of Research 33 Forms of descriptive research Quantitative Qualitative or Observational Existing data Describe an issue via valid & reliable numerical measures Study behavior “in nature” (high ecological validity). Use existing data for new quantitative (or qualitative) analyses Simple: frequency Qualitative Accretion counts of key behavior “Blocking” by other variables Correlational research: “what relates to what” Complex modeling Interviews, focus groups, textual analysis Qualitative quantitative Observational Direct Unobtrusive Study “remnants” of behavior Wholly non-reactive Archival Use existing data to test new hypothesis Typically non-reactive Foundations of Research Assess behavior directly rather than by participants’ selfreports or recall Typical data collection is highly reactive: participants know they are being studied, and react to that Observational methods are often less (or non-) reactive. Observational Research 34 Directly observe the social & physical settings or environments of behavior Similar to qualitative research: “ground” a research approach in the every-day reality of people. describe the social or physical context of a behavior generate hypotheses deeper understanding of a set of lab or quantitative findings, Foundations of Research Observational research: methods 35 Direct observation; visual observation & note taking or recording. Sitting in on classroom discussion, therapy session… Ethnographic studies (human or animal) Relatively direct data collection method Potentially strong reactive effects (Jane Goodall’s Chimpanzee studies were criticized because she fed and interacted with her subjects) Foundations of Research Observational research: methods Unobtrusive observation; participants unaware of data collection Major advantage: Eliminate reactive effects of data collection Less direct data, more difficult to gather & interpret One-way mirror & therapy research “Stake out” drug scene Focus group observation Participant observation; become part of social phenomenon to describe it e.g., joining political organization or cult, posing as prostitute (c.f.; Hunter S. Thomson Hells Angels; NY Times Down Low article here). Highly immediate and compelling description High potential bias in reporting and description Potential ethical concerns 36 Foundations of Research 37 Forms of descriptive research Quantitative Qualitative or Observational Existing data Describe an issue via valid & reliable numerical measures Study behavior “in nature” (high ecological validity). Use existing data for new quantitative (or qualitative) analyses Simple: frequency Qualitative Accretion counts of key behavior “Blocking” by other variables Correlational research: “what relates to what” Complex modeling Interviews, focus groups, textual analysis Qualitative quantitative Observational Direct Unobtrusive Study “remnants” of behavior Non-reactive Archival Use existing data to test new hypothesis Non-reactive Foundations of Research Existing data 38 Accretion; Study remnants of behavior Data wholly unobtrusive Campbell & Webb: Field Museum studies: determine popularity via linoleum flooring, nose-prints on glass… HIV prevention studies: # used condoms in “lovers lane” area after a public health media campaign. Indirect; may only partially map onto phenomenon. Archival; data collected for other purposes Often in highly reliable, large & rich data sets Provide unbiased correlations, but most be adapted to new purpose or hypothesis (may not “map on” fully..). Northern European health records; effectiveness of mammography in lowering breast cancer Correlation of suicide rate and publicity about prominent suicides to test modeling effects. Foundations of Research 39 Archival research example. Archival descriptive data; Standardized Illinois Board of Ed. drop-out data Chicago Tribune, 1 in 5 blacks drop out, 11/11/03; full article here. Very high drop-out rates in late 90s Gradual decrease to 2000 Significant increase again in 2000 – 2001, coinciding with No Child Left Behind legislation. Possible pressure to raise scores by ushering lower performing students out? Foundations of Research 40 Archival research example. Archival descriptive data; Standardized Illinois Board of Ed. drop-out data Chicago Tribune, 1 in 5 blacks drop out, 11/11/03; full article here. Archival data test the effect of recent educational policy, even though they were not collected for that purpose. Cannot clearly answer the “why?’ question. Article presents qualitative data from individual interviews. Archival + qualitative data can be used to generate important and testable hypotheses. Foundations of Research Weird archival research example. Do frustrated people view pornography to feel better? Data from the 2014 Seattle – Denver Super Bowl. Baseline porn traffic is similar for the 2 cities Traffic lessens in both cities as the game begins After the game traffic is much higher among Denver fans Long after the game traffic evens out As Denver begins losing badly traffic increases, particularly for Denver fans 41 Foundations of Research 42 Weird archival research example. Do frustrated people view pornography to make themselves feel better? Data from the Super Bowl. The overall viewing patterns suggest that more fans of a badly losing team view porn as the game goes on… To make themselves feel better? As a simple distraction? An alternate hypothesis is that people in Denver simply watch more porn. This is not plausible: traffic in the two cities was the same before and after the game. Foundations of Research 43 Attractiveness & Desirability What makes a women attractive to men? On OKCupid… Does simple attractiveness lead to more messages? Or is there something more complicated? The women in these two pictures get similar attractiveness ratings, 3.4 v. 3.3 The picture on the left has a normal distribution, peaking at ‘4’. The picture on the right has a bimodal distribution: lots of both ‘1’s and ‘5’s. Foundations of Research 44 Attractiveness & Desirability What makes a women attractive to men? On OKCupid… Simple attractiveness does not by itself lead to more messages ✓ The woman with more complex or diverse ratings gets 2.3 times the average number of messages… The women with less diverse ç ç ratings gets only .8 times the average. This is despite their being rated as similarly attractive. Foundations of Research 45 Attractiveness & Desirability This finding is tested more scientifically by deriving the Standard Deviation (S) of 8 women’s attractiveness ratings, that is, the variance in how she was rated. All the women in this chart were about the 80th percentile in attractiveness. The amount of variance in each women’s ratings (not her overall attractiveness) is correlated with the number of messages she got. Foundations of Research 46 Attractiveness & Desirability All the women in this chart were about the 80th percentile in attractiveness. Women with higher deviation scores, i.e., both ‘1’s and ‘5’s … … elicited more messages than did women with more consistent scores, i.e., mostly ‘3’s and ‘4’s Perhaps simple attractiveness is not as interesting as being challenging. Foundations of Research Archival / “found” data What is common to these examples is that the data were not collected for research. They stem from tracking customers, uniform drop-out rates, etc. The data are “repurposed” to answer a research question. 47 Foundations of Research 48 Overall Descriptive Design Issues Time frame Cross sectional Simultaneous measure of all study variables. Good for simple description Major problem for correlations: Longitudinal Causal direction: Which caused which? Major 3rd variable threat (ice cream and drowning). Cohort or panel study; follow participants over time. Best for testing hypotheses; assessing over time helps determine cause & effect. With archival data powerful description of behavior (e.g., crime rates, health status in population x time). Case study Single or multiple n = 1, cross-sectional or longitudinal Foundations of Research Descriptive methods: design issues, 3 Reactive measurement Participants (people or animals) react to the knowledge that they are being measured. Represents confound if responses are reaction to measurement rather than process under study Reactive bias increases with.. Clarity (face validity) of measures Face-to-face interview methods Often lessened with computer interviews 49 Foundations of Research Evaluating our measures: Reliability and Validity. Reliability If we are assessing a stable characteristic (IQ, personality, temperament, core values…) a good measure will give about the same result each time we administer it and for different sections of the measure. Validity Our survey or scale must actually measure what we designed it to. There are several ways we think about validity, each getting at a different element… 50 Foundations of Research Test - retest; Reliability similar responses over time? Assume stable attribute; e.g., “personality” disposition If measure is reliable, should show similar scores across time, e.g., at baseline and after a year. Split-half; similar responses across item sets? Assume redundant / converging items or scales. If scale is reliable, each half should yield similar scores. Chronbach’s alpha; overall internal reliability 51 Converging items should inter-correlate. Foundations of Research Scale appears to measure what it is designed to E.g., interview item; “How dependent are you on heroin?” Simple skill index; assess computer skills by writing program Intuitively valid; clearly addresses topic May yield socially desirable responses. Content validity Validity Face validity Descriptive research: Assesses all key components of a topic or construct: e.g., the various components of complex political attitudes… Mid-term; test all core skills for research design… Predictive validity Validly predicts a hypothesized outcome: e.g., I.Q. is a moderately good predictor of college success, criminality, etc. A measure may be predictive valid without being face or content valid: the MMPI. 52 Foundations of Research Descriptive research: Validity (2) 53 Construct validity Test whether the hypothetical construct itself is valid (differs from other constructs, corresponds to measures or outcomes it should..). Test if the Measure addresses the construct it was designed for E.g.; “anxiety” and “depression” and “anger” may not be separate constructs, but may all be part of “negative affectivity”. e.g., measures of social support (“do you have people who care for you”) often strongly influenced by depression, a separate construct… “Ecological” validity Measure corresponds to how the construct “works” in the real world External validity of assessment device. Foundations of Research Clicker Interpret this important data graph a. More drownings cause people to eat ice cream. b. Sharks like people who just ate ice cream. c. Who knows? It is just a correlation. d. Eating ice cream causes you to drown. e. In summer people both eat ice cream and swim. 54 Foundations of Research Qualitative or Observational Quantitative Describe an issue via valid & reliable numerical measures Study behavior “in nature” (high Simple: frequency Qualitative counts of key behavior “Blocking” by other variables Correlational research: “what relates to what” Complex modeling 55 Forms of descriptive research ecological validity). In-depth interviews Focus (or other) groups Textual analysis Qualitative quantitative Observational Direct Unobtrusive Existing data Use existing data for new quantitative (or qualitative) analyses Accretion Study “remnants” of behavior Wholly non-reactive Archival Use existing data to test new hypothesis Typically nonreactive Foundations of Research 56 Descriptive Research: Overview Basic design issues: Time frame Cross sectional Longitudinal Case study Reliability Test – retest Split – half Alpha (internal) Validity Face Content Predictive Construct Ecological