Conducting Research by Lawrence T. Orcher This text is for those doing their first empirical study Chapters 1-10 go over setting up to prepare a proposal Model literature reviews through basic stats are included in this book Chapter 1 Selecting Tentative Topic Research is process of systematically collecting and interpreting information Library research – collect/interpret info others wrote Empirical research – making observations to collect new info (data) which is analyzed to assist in making interpretations Observations can be overt (direct) or indirect (interviews, surveys) Chapter 1 (cont.) Several broad areas of interest identified based on Everyday observations Theories (test a theory) Beware of availability of participants, ethical considerations (no phys/psych harm), audience (my approval), personal needs (your interests) Can replicate published studies Use demographics to narrow OR broaden a topic Start searching the literature after broad area identified then may narrow or even change topic Chapter 2 - Locating Literature and Refining the Topic Desire SCHOLARLY/PEER REVIEWED/REFEREED articles (most recent and are checked) Databases of published research available through paid subscriptions (by TC) ERIC; Infotrac I – Onefile; Proquest–Education Journals Use the database thesaurus to better define topic and find related terms Topic should narrow, become better defined after literature search is completed Chapter 2 (cont.) Goal is 50-100 abstracts to examine at one time Use Boolean operators to narrow (AND) or widen (OR) search Review both negative and positive studies Focus on recent studies (last 10 years) Tutorial on APA citations on TC webpage, select Library Homepage, then Tutorials Good idea to print the APA Quick Reference Guide 1 For federal statistics, searching by topic, go to www.fedstats.gov Chapter 3–Preparing (writing) a Literature Review Lit review establishes need for your study and gives context to it See table in text for organizing sources Lit review is an essay about the literature Synthesize studies about a topic Do a topic outline first (see example in text) Group sources together under given topic NOT a source-by-source listing/discussion Use APA Level 4 subheadings if lit review is long Review examples in book Chapter 3 (cont.) Critical review is needed…if there is a flaw in a study, point it out (see book examples of before/after) Avoid statements of fact/truth/proof…all research is flawed and so use tentative language (Research implies, suggests, etc.) Point out gaps in literature, and how your study will address the gaps Chapter 3 (cont.) Placement of surnames for in-text citations Use Harvard method (surnames, year) Author can be subject of sentence (draws attention to who the author is) Author can be parenthetical (putting the source in parenthesis puts focus on the content of the sentence…this is preferred) Discuss relevant theories in detail before studies Reader should see a connection between lit. rev. and RQ/purpose/hypotheses…logically RQs come at end See 3 model lit reviews at end of book Chapter 4 - Writing Research Hypotheses, Purposes, and Questions These guide your study and identify the variables included They are based on previous research or theories and logically follow the literature review Can be given in one of three formats Hypothesis – a statement of a predicted outcome from the study Purpose – a declarative statement that identifies the variables of interest, what was studied (no expected outcome is given) If cause/effect the independent and dependent variables must be included Do not give numerical values (i.e. Group X will score 50% higher than Group Y…this means you’d be wrong if it was 49% or less OR if 51% plus) Should not be a value judgment…must be something observable Often this is used instead of a statement of the problem Question (RQ) – alternate wording of the purpose, given at the end of the lit review (numbered as RQ1, 2 RQ2…same for hypotheses H1, H2…) Chapter 4 (cont.) Clarifying the Purpose/Hypotheses/Questions Often include information about the population, but not usually about the methods of measurement Sometimes have research questions and hypotheses that link to the research purpose statement Reviewing the literature helps clarify the RQ etc. See text examples, especially before/after cases Chapter 5 - Selecting a Research Approach Choose a design (approach) after form RQ/Hyp Experimental v. Non-Experimental Experimental explores cause and effect relationships Has at least one independent variable (IV)/cause which is under control of the researcher Has at least one dependent variable (DV) or outcome/effect Generally form two groups, one (experimental or treatment group) gets the new treatment and the other (control group) does not Random assignment to trtmt/control key element of true experimental designs Quasi-experimental is used when random assignment is not possible (see Ch. 22) Chapter 5 (cont.) Major types on non-experimental research Causal-Comparative (aka ex post facto) Used when random assignment to trtmt/control is not possible Compare existing groups to identify causal sequence Often matching is used to ensure the comparison groups are equivalent on variables that might affect the DV Chapter 5 (cont.) Major types on non-experimental research (continued) Surveys (provide a description of what exists, aka descriptive research) Survey a population (census) or a sample on attitudes/opinions Mailed questionnaires often used (see App. A for ways to improve response rates…use incentives, keep it short, preaddressed/stamped envelopes, standard vocab/grammar) Interviews (more labor intensive/expensive, but get more in-depth data) Achievement surveys – measure knowledge levels Groups’ opinions are often compared based on demographic factors (such as age, gender, race, etc.) Chapter 5 (cont.) Major types on non-experimental research (continued) Correlational Studies (Appendix B has hints) Use a correlation coefficient statistic to assess relatedness of two quantitative variables (ACT and college GPA) Coefficient is a numerical measure from 0 to 1 (plus or minus) that tells how related the two variables are 3 • • Closer to zero, less relationship; further from zero more Sign of the coefficient tells if the relationship is positive/direct (hi/hi and lo/lo) or negative/inverse (hi/lo and lo/hi) Chapter 5 (cont.) Major types on non-experimental research (continued) Document/Content Analysis (examine existing records to look for patterns/themes) Good samples available Documents are secondary sources…may not be accurate Program Evaluation (hybrid of experimental & non) Program is a treatment, but usually not random assgnmt Sometimes focus on implementation process (formative) rather than outcome (summative) – App. D info Long timeframe needed Chapter 5 (cont.) Quantitative v. Qualitative (or breadth v. depth) Quantitative Qualitative Numbers based, standardized procedures to obtain data, rather rigid with no personalized interactions with subjects Generalization is important See example list of projects in text Not structured nor laid out in advance Direct involvement with participants increases chances of bias, so usually best done by trained researchers Hints on which to use: Some topics are inherently quantitative (including those w/ hypothesis since need structured study & measuremt); some topics can be studied with either approach; emerging topics may need qualitative Chapter 6 - Looking Ahead to Participant Selection How many and how selected Size Smaller numbers often allowed in research class projects Experimental and qualitative usually smaller samples than quantitative and descriptive studies Either way a minimum of 30 and maximum of about 1000 Selection method (most important…representation is essential) Sampling methods are common criticisms of research studies Chapter 6 cont. Random sampling (for quantitative research) – all members of the population must be identified and have an equal chance of being selected A random sample is an UNBIASED sample Use names in a hat (or equivalent w/ table of random #s) Often hard to identify all population members Sometimes those selected do not participate (low response rates on mailed questionnaires in particular) Stratified sampling used to ensure subgroups are represented (usually proportional…using volunteers w/in strata not ok) 4 Chapter 6 (cont.) Sampling Methods (continued) Convenience sampling (aka accidental sampling) Presumed to be biased; using volunteers is a bad idea Often used to pilot test instruments/materials or for exploratory studies Findings open to criticism (should be interpreted with caution) Purposive sampling Often used in qualitative to gain depth in understanding a group that has information of interest (such as women officers in large corporations) Criteria for inclusion must be clearly identified and based on specific reasons; then select a clear method to identify participants…do not use convenience OR volunteers Chapter 6 (cont.) Using the literature to plan sampling Note how many tend to be used in studies reviewed in the literature and how they were selected Pay attention to the Limitations sections of published research reports…avoid the sampling errors identified there; also can identify needed populations to study Ch. 11/12 will cover this in more detail Chapter 7 - Looking Ahead to Instrumentation Methods of measurement (tests, questionnaires, interview schedules, observation forms, etc.) Can use ones from the literature Have established validity Allows comparisons between studies Any errors in them will apply to your study Can modify ones from previous studies Devising new instruments Must be able to defend validity/reliability Pilot test these Chapter 7 (cont.) Validity – extent to which an instrument measures what it is supposed to (Ch. 13) Construct validity (correlate the new measure with some other measure related to it) Expert judgment (content/facial validity) Qualitative (Ch. 15) focuses on credibility (member checks, reviews by participants, triangulation of data sources) Reliability – extent of consistency in results (Ch. 17) Number of items is important Objectivity of items is necessary Qualitative (Ch. 15) focuses on dependability (use more than one person to interpret data, triangulate measures…use interviews plus surveys) Chapter 8 - Looking Ahead to Quantitative Data Analysis Levels of data (nominal, ordinal, interval, ratio) dictate how statistical analysis is 5 done Nominal (named categories) Still just names even if numbers assigned as name tags (such as 1=male, 2=female); the numbers have no quantitative meaning For a single variable, use frequency/percent tables and picture with pie or bar charts (percent=part divided by whole) For relationship between two variables, use cross-tabulation (also called a contingency table) For inferences from sample to population, use chi-square test Ordinal (named categories with relative order, such as class rank: freshman, sophomore, junior, senior) Analysis is similar to nominal Chapter 8 (cont.) Interval (continuous, quantitative measures) For one variable, use mean and standard deviation and picture with a histogram) For two variables (both interval) calculate correlation coefficient to see relationship between them; look at direction (positive/direct v. negative/indirect) and strength (weak, r=0 ….. strong, r=1.0) For two variables (one nominal and one interval), compare means and standard deviations of the two groups For inferences to pop. use t-tests, ANOVA, significance of correlation • With experiments can compare the net gain/loss from pre to post test for the control v. experimental group Ratio (continuous, quantitative but with an absolute zero) Analysis is similar to interval Chapter 9 - Looking Ahead to Qualitative Data Analysis Only interviews are covered here Qualitative has data collection and data analysis intermingling Memo writing during data collection (journal of researchers ideas/impressions) Reflection and reframing of interview questions Collect until reach data saturation (no new info is being gained with successive interviews) Chapter 9 (cont.) Two general approaches Grounded Theory Approach – inductive reasoning (theories developed based on data collection) Open coding (track themes) Axial coding (temporal, causal, associational, valence, spatial) Core categories developed Describe process used in analysis Consensual Qualitative – several people review data and come to consensus about results (auditor checks) Analysis techniques Chapter 9 (cont.) Enumeration (count times theme/construct is mentioned) Selecting quotations (to support findings) 6 Inter-coder agreement (reliability check) Diagramming (see text example) Peer debriefing (other researchers review/concur) Auditing (independent review) Member checks (go back to participants for review) Identify range of responses Analyze discrepant cases Chapter 10 – Preparing a Preliminary Research Proposal Proposal is a plan for conducting research (the proposal allows feedback on its accuracy) – see TC guidelines Title – brief statement that names the major variables investigated (and subjects sometimes) as given in the RQ/purpose/hypothesis (see text examples) Introduction chapter introduces the topic/study by giving the Problem area/background info (use limited number of references here) Definitions (conceptual and operational…see text examples) Significance of study (why it’s important to do the study) Note that qualitative generally has some info about how the researcher is directly related to the problem (personal experiences and perspectives) since this may influence collection/interpretation of data) Literature review RQs/hypotheses Chapter 10 (cont.) Methods is the second chapter and as a minimum covers participant selection and instrumentation Participants – discuss population, how subjects will be selected, number selected, permission, etc. Instrumentation – describe measurement tools used and cite sources (see text examples and TC guidelines) Procedures section (explains how the research will be carried out) Data analysis section (explains how the data will be analyzed statistically) Threats to validity (limitations/problems with the data) References/Appendices follow the last chapter Chapter 11 – Participant Selection in Quantitative Research Population is the group the researcher is interested in Accessible population differs from the entire pop, but must be cautious with generalizations from an accessible pop. Sample is a subgroup drawn from a population Biased sampling occurs when everyone does not have equal chance of being selected as a participant Bias can be subtle or obvious Volunteers, convenience sampling most common biases Need a system to assure equal chance to everyone – called RANDOM SAMPLING Even random sampling can have random (chance) errors, but not systematic ones (calculate and report margins of error) Larger samples have less error Allows inferences 7 Simple random sampling Chapter 11 (cont.) Names in a hat or using table of random numbers (in book) Number each pop member with equal digits, then select random starting place in table Stratified sampling Systematic sampling…select every nth one from list Divide population into subgroups (and draw from each group…usually proportional) Need to make selection through entire list Beware ordered lists…best to use randomly order lists Cluster sampling Population is in groups, ea group is a cluster (Girl Scouts/troops) Select enough clusters and select them randomly Chapter 11 (cont.) Sample size – influenced by several things Pilot study to help determine return rates Number of subgroups to be examined (more needed, increases the sample size) Importance of precise results/need to find a small effect (such as heart attacks…occurrence is small so need larger sample size) Statistical significance – larger samples have greater chance of significance See table of suggested sample sizes in book Chapter 12 – Participant Selection in Qualitative Research Purposive Sampling (not so interested in unbiased samples so can generalize) – hand pick participants since they possess the needed info Criterion sampling – very specific criteria must be met to be included; can purchase lists of those meeting criteria or get from professional associations Random purposive sampling – random selection from a list of those who meet the criterion Typical case sampling – select what are believed to be normal or typical cases (reduces complexity of data…good for data analysis; beware data don’t apply to most cases) Chapter 12 (cont.) Extreme/deviant sampling – identify the criteria (define them) and take from one or both extremes Useful to refine measurements for later typical case sampling Intensity sampling – select those with intense feelings or experiences Maximum variation sampling – making sure participants have full range of characteristics (age/income/gender/etc.) Homogeneous sampling – opposite of above; want participants to be similar on characteristics Opportunistic – individuals selected to participate as the opportunity arises since they may have info of interest to the study Chapter 12 (cont.) Stratified purposive – identify subgroups then select from them so all subgroups are represented Snowball (aka chain/network) sampling (also used in quantitative) – for hard to find participants…identify one and ask for referral to others, i.e. drug addicts Combination purposive sampling – combine any of the above Improving convenience/accidental sampling – if must be used 8 Use a larger group selected from multiple locations Collect demographic data so readers can describe the participants – readers can then judge representativeness Chapter 12 (cont.) Sample size in qualitative research Much smaller than in quantitative (in one published report average N in qualitative was 14 versus 432 in quantitative) Focus is more on in-depth information gained from participants and requires more time per participant Saturation sampling – stop when new insights and info are no longer being provided by participants (point of redundancy) Chapter 13 – Instrumentation in Quantitative Research General rules for good measurements: Standardization – use same methods for administering tests to each subject Objectivity – Use scoring that does not allow subjective interpretation Social desirability bias – Eliminate this tendency of people to give what they believe is an acceptable answer by using anonymity Chapter 13 (cont.) Reliability and Internal Consistency Reliability refers to consistency in measurement (getting the same reading with multiple measures) Test/retest reliability – administer test to same group twice (a week or two apart) and should get consistent scores Measure this using reliability coefficient (0 to 1.0 score and the closer to 1.0 the more consistent); Look for 0.85+ Guessing/random answers; inconsistent administration of tests; changes in subjects; ALL can affect reliability so try to limit these Internal Consistency from one part of the test to another Can calculate split-half reliability (odd/even) or Cronbach alpha On surveys ask positively and negatively worded questions Chapter 13 (cont.) Both test/retest and internal consistency provide info that helps in understanding how well a test works Best to report a couple of reliability measures Inter-observer Reliability – extent to which 2 observers get the same score, especially important when scoring is not objective Chapter 13 (cont.) Validity and its relationship to reliability Validity is the extent to which an instrument measures what it is intended to measure Measures can be highly reliable but still not valid Judgment vailidity Content – based on expert opinion of the appropriateness of the contents of a test/scale – does it cover the whole content Facial – on its face does it appear to measure what it’s supposed to 9 Best to report a couple of reliability measures Criterion validity – how well measure correlates with some criterion Concurrent – criterion occurs at same time as measure Predictive – criterion occurs at a future time (ACT score & college GPA) Chapter 13 (cont.) Construct validity – cannot observe some things directly (love, depression, anxiety), thus construct their existence based on observation of behaviors that indicate they exist Construct validity – refers to the extent to which an instrument yields scores that are consistent with what is known about the construct Calculate it by correlating scores with other things that indicate the construct ( correl. happiness scale scores to an anxiety scale you developed, should get negative correl.) Need to have several measures of validity Chapter 14 – Writing Objective Instruments Often can find and use existing instruments (ETS Test Collection database) For this program generally develop your own to develop the related skills…text covers three most common ones Develop a plan Have plan reviewed Revise plan Write items based on the plan Have items reviewed Revise items, put into instrument form Pilot test instrument Revise after pilot test Chapter 14 (cont.) Attitude scales - Attitudes are orientations toward something and they affect actions Planning Identify the components of the object being studied (i.e. job satisfaction made up of working conditions, pay, benefits, etc.) For each component write 3-5 items plus 1-2 on overall attitude (I like my job Review/feedback at this stage Writing Likert scales are used to measure attitudes by providing descriptive statements and asking whether respondents agree/disagree (SA to SD) Each statement has only one point (see text example) Use favorable/unfavorable (neg/pos) wordings Review items/revise Pilot testing Use sample similar to popul. to be tested; think aloud/written feedback Review/revise Chapter 14 (cont.) Observation Checklists – List of behaviors/characteristics for which observations should be made Planning Writing Identify the behaviors/characteristics and context for observing them Have these reviewed, identify new ones/revise Each item refers to only one single discrete behavior (no ANDS); see Text Also often ask about speed, duration, and success of behaviors Pilot testing 10 Have at least two people do observations independently and see if they agree on items checked Review/revise (may need to clearly define behaviors on checklist) Chapter 14 (cont.) Achievement tests – Usually objectively scored, multiple choice Planning Writing Base test items on objectives – try to cover all or at least most important ones Have these reviewed/revise Multiple choice – one correct choice and at least two plausible distractors Beware of ambiguous items (see example in text) Review/revise Pilot testing Try to include some high and low achievers Review carefully and compare performance (of high/low performers) Revise items Chapter 14 (cont.) Note these are only the beginning steps in writing up instruments It’s a complex process You can get a doctorate in tests and measurments, so obviously it is a much more complex task than we have just reviewed You will learn a lot by developing your own instrument for your project Chapter 15 – Instrumentation in Qualitative Research Qualitative researchers strive for credibility Member checks used to verify findings (feedback from participants on correctness of results) Prolonged engagement in the field gives credibility Triangulation of data sources (multiple sources give same results) Triangulation of instrumentation (multiple instruments used for each participant) Chapter 15 (cont.) Interviews are most common data collection method, particularly for beginning qualitative researchers Good source: The Long Interview by Grant McCracken Pay attention to interviewer selection and behavior so results are not contaminated Often match interviewers with participants on characteristics, behaviors, etc. (to gain access, trust) Interviewer self disclosure (interviewer needs to give self-disclosure in both collecting and analyzing data as well as reporting) Chapter 15 (cont.) Interview protocols (steps and questions used in the interview) Simi-structured/open-ended (core questions with some probing allowed) Initial questions used to establish rapport Can adopt previously used protocols or develop, review, pilot test your own Demographic questions should be standardized Recording responses/note taking – taping is best, but may inhibit responses; make notes immediately, describe process in reporting 11 Chapter 20 – Experimentation and Threats to External Validity Experiments: researchers administer treatment to part of a population to determine effects on an outcome variable Experiments explore cause and effect relationships Independent variable (IV) in an experiment is cause Dependent variable (DV) in an experiment is the effect Simple experiments have one IV and one DV; complex ones have several of either (see text chart) Chapter 20 (cont.) Hawthorne effect – the result changes just because subjects are being observed, not because of the experimental treatment Control groups are used to make sure the HE is not occurring See text examples before and after use of control groups Chapter 20 (cont.) External validity of experiments (generalizability of results to outside the experimental setting) Threats to external validity Selection bias – failure to use true random sampling in choosing participants so they are not representative of the population Reactive effects of exper. setting – a lab setting used in an experiment might differ from the field situation (so different reactions occur in people outside the exp. setting) Reactive effects of testing/pretest sensitization – subjects in experiment perform differently because sensitive to what effect you are looking for Obtrusiveness of measurements – try to use unobtrusive measures so no effect on experiment (or do field experiments) Multiple treatment interference – if several different treatments given to same subjects, can interact Chapter 21 – Threats to Internal Validity and True Experiments Internal validity – Did the independent variable really cause a change in the dependent variable (see text example) True experiments have a treatment and control group…random assignment is essential Graphically described as: ROXO RO O Chapter 21 (cont.) Threats include other things besides IV that might have caused a change in the DV History – any external event that might cause a change in the DV (control for this by having a control group to compare the treated group to) Maturation – natural changes that occur in subject (control and treatment groups should mature at same rate) 12 Instrumentation – refers to possible changes in the instrumentation (how the test was administered from pre to post testing); control/trtmt groups should have same problems Chapter 21 (cont.) Threats (cont.) 4. Testing (practice effect) – control group should take care of this Statistical regression/regression toward the mean – those really high (or low) should move down (or up) on subsequent tests; random selection and control group take care of this Selection – nonequivalent groups compared due to nonrandom assignment (as opposed to random selection) Can use a Solomon 4-Group design which adds two more groups (one control and one treatmt) that do not have a pretest (to avoid pretest sensitization) ROXO RO O R XO R O Chapter 21 (cont.) True experiments have random assignment – it’s what makes groups equivalent (not the pretest) Need high external and internal validity to trust results! Chapter 22 – Pre-Experiments and Quasi-Experiments Random assignment is the hallmark of true experiments since it controls all threats to internal validity Sometimes cannot randomly assign so go to the next best Pre-experimental designs Single group pretest-posttest design (O X O) - all threats apply Single group posttest only (X O) - cannot measure change Static group design – 2 groups, no pretest, no random assignment (dashed line indicates intact groups used) X O O Chapter 22 (cont.) Quasi-experimental designs Non equivalent control group design, 2 intact groups O X O (note that equal pretest scores needed) O O Time series design – one group, baseline then treat O O O O O X O O O O O (observe several, treat, observe) Equivalent time samples design (X1 trmt and X0 none) X1O X0O X1O X0O X1O X0O X1O X0O Quasi- and pre-experimental sometimes used when cannot do true experiments, but need to be cautious with interpretations Chapter 16 – Descriptive Statistics for Quantitative Qualitative Research Descriptive statistics summarize data and are used to present data Quantitative studies – all variables are analyzed statistically Qualitative studies – demographic variables are analyzed statistically The level of data determines what kinds of statistical analysis is done (nominal, ordinal, 13 interval, ratio) The higher levels of data (interval/ratio) allow more analysis techniques than lower levels Use measures of center and variation/spread/dispersion Center: mean, median, mode Variation: range, IQR, standard deviation Chapter 16 (cont). Nominal data – named categories or labels (gender, race, location) For univariate analysis: Report frequency/percent (% calculated with part divided by whole); pie charts Report most frequently given answer (Mode) For bivariate analysis: Report frequency/percent by category using a contingency (aka crosstabs/crossbreak) table Chapter 16 (cont). Ordinal data – named categories that have relative order (places in a race, class rank) For univariate analysis: Report frequency/percent; bar/column charts Report most frequently given answer (Mode) and the middle answer (Median) Note: If odd number of scores the median is the middle one, if even number of scores, average the two middle answers Report the range (highest – lowest) and the innerquartile range (IQR) For bivariate analysis: Report frequency/percent by category using a contigency (aka crosstabs/crossbreak) table Compare medians for different groups Chapter 16 (cont). Interval and ratio data – quantitative numbers that tell how much with equidistance between points; ratio also has an absolute zero, so no negative numbers (test scores, age in years, number of minutes late) For univariate analysis: Picture distribution with histograms, stemplots Report mode and median plus the Mean (mathematical average, calculated: ∑x ÷ N) M=population mean; m=sample mean or x-bar Report the range and SD (average distance of all points from the mean using formula to calculate) SEE TEXT EXAMPLES FOR 3 GROUPS… larger SD means more variation/spread/dispersion For bivariate analysis: Compare distributions Compare means, SD for different groups Correlate Chapter 16 (cont). Normal distributions – bell shaped and symmetrical Centerline is the mean (and if normal, also median and mode) 68% of cases within plus or minus 1SD from the mean; 95% of cases within plus or minus 2SD from the mean; 99.7% of cases within plus or minus 3SD from the mean When have skewed distributions, mean and SD are warped, so use median and range (or IQR) instead See IQR example in text 14