IS 4800 Empirical Research Methods for Information Science Class Notes Feb. 17, 2012 Instructor: Prof. Carole Hafner, 446 WVH hafner@ccs.neu.edu Tel: 617-373-5116 Course Web site: www.ccs.neu.edu/course/is4800sp12/ Hypothesis Testing and Inferential Statistics 2 Basic Process of Hypothesis Testing ■ H1: Research Hypothesis: ■ Population 1 is different than Population 2 ■ H0: Null Hypothesis: ■ No difference between Pop 1 and Pop 2 ■ The difference is “null” ■ Compute p(observed difference|H0) ■ ‘p’ = probability observed difference is due to random variation ■ If p<threshold then reject H0 => accept H1 ■ p typically set to 0.05 for most work ■ p is called the “level of significance” 3 Examples ■ Research Question: ■ Which is more popular, Guitar Hero or Rock Band? ■ Research Question: ■ Is the ownership of Wii vs. Xbox consoles significantly different for NU students compared to ownership in the general US population? ■ Research Question: ■ Are Wii owners more likely to own Guitar Hero vs. Rock Band, compared to Xbox owners? 4 Type of Errors in Hypothesis Testing “The Truth” H0 True H0 False Decide to Reject H0 Type I Error Correct Decision Do not Reject H0 Correct Decision Type II Error ‘p’ = p(?) Probability of Type I Error 5 Procedure for Hypothesis Testing with Chi-square 1. Formulate your research hypothesis (including statement of expected frequencies) 2. Determine hypothesis test parameters – significance threshold 3. Collect your data 4. Compute Chi-Square statistic and draw conclusion 6 Chi-Square for Goodness of Fit ■ Assumes 1. You have a nominal variable • Values are exhaustive & mutually-exclusive 2. You have an Expected Frequency table for the nominal variable 3. Representative sample 7 Chi-Square for Goodness of Fit ■ Form of null hypothesis H0? ■ Observed frequency = Expected frequency ■ Populations (expected, observed) are actually the same ■ Form of hypothesis H1? ■ Observed frequency ≠ Expected frequency ■ Populations (expected, observed) are different 8 Formula for Chi-square statistic (O E ) E 2 2 ■ O = Observed frequency for a given category ■ E = Expected frequency for a given category ■ Note: “statistic” is a function you apply to a set of data (in a statistical analysis) 9 Example ■ You go gambling in a shady casino, and suspect that the games are rigged. ■ You focus your attention on one 6-sided die being used in a game and keep track of 60 rolls: Roll 1 Count 6 2 5 3 7 4 9 5 3 6 30 ■ Is the die “loaded”? 10 Example ■ OS market share in US is: ■ 96% PC; 3% Macintosh; 1% Linux ■ You do a survey in your company and find the following user breakdown: ■ 475 PC; 25 Mac; no Linux ■ Is your company weird? 11 Formula for Chi-square statistic (O E ) E 2 2 ■ O = Observed frequency for a given category ■ E = Expected frequency for a given category ■ Note: “statistic” is a function you apply to a set of data (in a statistical analysis) 12 Computing Chi-square ■ SPSS: ■ run NonParametric/ChiSquare ■ See if significance<threshold • Yes => reject H0 • No => inconclusive ■ Manually: ■ Determine df (= num categories – 1) ■ Compute Chi-square using formula ■ Lookup to see if statistic>table entry for threshold-significance, df • If yes => reject H0 • If no => inconclusive 13 Reporting result Where, 2 (df ) chisq , p sigthresh – df = degrees of freedom – sigthresh = pre-defined significance threshold • Note: if p<<sigthresh, can report that as well, e.g., “p<.01”, “p=.001” For example: 2 (2) 11.89, p 0.05 If not significant, than use “n.s.” instead of “p<…”. Usually also report expected and actual frequencies, or at a minimum, the total number of cases considered (aka “n”). Measure 475 25 0 500 O E 95 5 0 100 X^2 term 96 0.010417 3 1.333333 1 1 100 2.34375 15 What do you do if you don’t have any information to base Expected frequencies on? 16 Chi-Square Test for Independence Statistical independence: P(A| B) = P(A | ~B) = P(A) 17 Chi-Square Test for Independence ■ Are two variables related, or are they independent? ■ Assumptons ■ Both variables must be nominal (or treated as if) ■ Cannot be related in a ‘special’ way (i.e., repeated measures) ■ Representative samples assumed ■ Normal distribution NOT assumed 18 Example ■ Morning & night people using different modes of transportation. ■ What kind of study is this? Bus Morning Night Carpool Own Car 60 30 30 20 20 40 19 Expected frequencies if variables are independent ■ E = (R x C)/N for each cell ■ R = row count ■ C = column count ■ N = total number in all cells Bus Morning Night Carpool Own Car 60 30 30 20 20 40 20 Expected frequencies if variables are independent ■ Step 1 – compute row & col totals Bus Morning Night Carpool Own Car 60 30 30 20 20 40 80 50 70 21 120 80 Expected frequencies if variables are independent ■ Step 1 – compute row & col totals ■ Step 2 – compute row % Bus Morning Night Carpool Own Car 60 30 30 20 20 40 80 50 70 22 120 80 60% 40% Expected frequencies if variables are independent ■ Step 1 – compute row & col totals ■ Step 2 – compute row % ■ Step 3 – ea cell = (R x C)/N Bus Morning Night (48) (32) Carpool Own Car (30) 60 30 (42) 30 (20) 20 20 (28) 40 80 50 70 23 120 80 Formula ■ df = (NumRows-1)x(NumColumns-1) (O E ) E 2 2 24 Written Study Reports ■ Objectives (also critiques) ■ Describe what your study is about ■ Motivate your study ■ Assure reader you have conducted a sound study • Research Methods – often presented in small font ■ Present results in an objective manner ■ Discuss implications ■ Discuss future work ■ Enable replication 25 Typical Study vs. IS/CS/HCI Paper Structure Astract Introduction Method Results Discussion Motivation Related work Hypotheses Limitations Implications Future work References 26 Typical Study vs. IS/CS/HCI Paper Structure ■ Abstract ■ Introduction ■ Motivation ■ Related work ■ System design ■ Evaluation ■ ■ ■ ■ Hypotheses Method Results Discussion – summary, limitations ■ Conclusion ■ Implications ■ Future work ■ References 27 The Abstract ■ Concise summary ■ Abstract for an empirical study should include ■ Information on the problem under study ■ The nature of the subject sample ■ A description of methods, equipment, and procedures ■ A statement of the results ■ A statement of the conclusions drawn ■ Often the last thing you write 28 The Introduction ■ Part of paper giving justification for study ■ Usually has the following information ■ Introduction to the topic under study ■ Brief review of research and theory related to the topic ■ A statement of the problem to be addressed ■ A statement of the purpose of the research ■ A brief description of the research strategy ■ A description of predictions and hypotheses ■ CS/IS papers often put Related Work as a separate section after Introduction ■ For each, describe how your work is different 29 Organization of the Introduction: General to Specific Present a general introduction to your topic Review relevant literature Link literature review to your hypotheses State your hypotheses 30 The Method Section ■ Includes information on exactly how a study was carried out ■ Subsections ■ Participants or subjects • Describe in detail the participant or subject sample • Human participants go in a Participants subsection, and animal subjects in a Subjects subsection ■ Apparatus or materials • Describe in detail any equipment or materials used • Equipment is usually described in an Apparatus subsection and written materials in a Materials subsection 31 The Method Section ■ Procedure ■ Describe • Exactly how the study was carried out • The conditions to which subjects were exposed or under which observed • The behaviors measured and how they were scored • When and where observations were made • Debriefing procedures ■ Enough detail should be included in all sections so that the study could be replicated 32 The Results Section ■ Objective, dry, boring – just the facts ■ All relevant data and analyses are reported in the results section ■ Do not present raw data ■ Data should be reported in summary form ■ Descriptive statistics ■ Inferential statistics ■ Results of descriptive and inferential statistics must be presented in narrative format ■ Describe the source of any unconventional statistical tests 33 Commonly Used Statistical Citations Statistical Test Format Analysis of variance F (1,85) = 5.96, p < .01 Chi-square χ2(3) = 11.34, p < .01 t test t (56) = 4.78, p < .01 34 Abbreviations for Statistical Notation Abbreviation Meaning df Degrees of freedom F F ratio M Arithmetic average (mean) N Number of subjects in entire sample p p value SD Standard deviation t t statistic z Results from a z test or z score μ Population mean (mu) s Population stddev 35 The Discussion Section ■ This is where you can take some liberties with describing what the results mean ■ Results are interpreted, conclusions drawn, and findings are related to previous research ■ Section begins with a brief restatement of hypotheses ■ Next, indicate if hypotheses were confirmed ■ The rest of the section is dedicated to integrating findings with previous research ■ It is fine to speculate, but speculations should not stray far from the data 36 Organization of Discussion: Specific to General Restate your hypotheses or major finding Tie your results with previous research and theory State broad implications of your results, methodological implications, directions for future research 37 Example 38 39 40 41 42 43 44 45 46 47 48 49 50 Citations ■ Liberally cite previous & related work. ■ If you copy passages you must cite and, depending on length, format to indicate it is copied. ■ Suggest using EndNote, BibTex or similar. 51 Ethical Issues ■ Report all of your findings (not just the ones you like) ■ Adhere to your original plan ■ Report any deviations and why ■ Power analysis, statistics, measures ■ Do not drop subjects or data points without rigorous justification ■ If your hypothesis test was not significant you cannot say anything about difference in means (example). ■ If you did not do an experiment, attempting to control for extraneous variables, you cannot mention or imply causality. 52 Oral Presentation of Study Results 53 Oral Presentation ■ Main concepts and ideas ■ Do not go into great detail on experimental methods – just enough so people understand roughly what you did ■ Focus on motivation, results, implications ■ If listener wants details they can read the paper or ask questions 54 Oral Presentation Don’t do this… Change From To Measure Day1 Day2 WAI/COMP 7 27 WAI/BOND 7 27 WAI/TASK 7 27 WAI/GOAL 7 27 CONTINUE LAURA 30 44 MIN/DAY -6-0 22-30 1-7 22-30 22-30 38-44 DAY/WK>30MIN -6-0 22-30 1-7 22-30 22-30 38-44 STEP/DAY 1-7 22-30 DAY/WK>10KSTEP 1-7 22-30 STAGE Intake 30 30 44 SELF-EFFICACY 1 29 29 44 PROS 1 29 29 44 CONS 1 29 29 44 CONTINUE FT 30 44 ALL CONDS df t p 54 0.205 0.838 54 0.519 0.606 54 0.134 0.894 54 0.155 0.877 54 0.868 0.389 81 1.470 0.145 81 0.691 0.492 81 3.626 0.001 81 6.653 0.000 81 6.272 0.000 81 8.990 0.000 81 1.778 0.079 77 3.986 0.000 81 6.988 0.000 81 2.019 0.047 81 4.782 0.000 81 2.770 0.007 81 1.998 0.049 81 0.393 0.695 81 0.902 0.370 81 0.740 0.462 81 1.520 0.133 CONTROL df t p 26 26 26 26 26 26 26 25 26 26 26 26 26 26 26 26 26 1.274 0.758 2.480 2.323 2.401 4.043 1.197 1.355 3.403 1.185 0.872 1.525 1.418 1.147 1.124 0.386 1.442 0.214 0.456 0.020 0.028 0.024 0.000 0.242 0.188 0.002 0.247 0.391 0.139 0.168 0.262 0.271 0.703 0.161 NON-REL df t p 24 0.014 0.989 24 0.376 0.710 24 0.409 0.686 24 0.081 0.936 24 0.625 0.538 24 0.124 0.903 24 0.109 0.914 24 1.959 0.062 24 5.284 0.000 24 3.818 0.001 24 5.322 0.000 24 2.366 0.026 23 3.591 0.002 24 4.000 0.001 24 1.000 0.327 24 3.314 0.003 24 4.550 0.000 24 0.456 0.653 24 0.225 0.824 24 0.499 0.622 24 0.611 0.547 24 1.163 0.256 RELATIONL df t p 29 0.361 0.720 29 1.489 0.147 29 0.661 0.514 29 0.329 0.745 29 0.619 0.541 29 1.104 0.279 29 0.358 0.723 29 1.804 0.082 29 4.347 0.000 29 4.597 0.000 29 6.530 0.000 29 0.236 0.815 27 2.055 0.050 29 4.738 0.000 29 1.409 0.169 29 4.750 0.000 29 0.085 0.933 29 1.540 0.134 29 0.308 0.760 29 0.823 0.417 29 0.339 0.737 29 0.000 1.000 55 Oral Presentation Do use as many figures as possible 7 WEEK 1 WEEK 4 6 5 4 NON-REL RELATIONAL 3 2 CO M P W K1 BO ND W K1 TA SK W K1 G O AL W K1 CO M P W K4 BO ND W K4 TA SK W K4 G O AL W K4 1 56 Oral Presentation Guide for Visuals ■ Visuals should be exhibits that you talk about ■ Do not put lots of text on charts ■ Do not read your charts for your presentation ■ Use interactivity, video, images to keep your audience awake 57 Common Questions ■ How did you evaluate that? ■ How did you measure that? ■ How did you control for extraneous variable X? ■ Why didn’t you use statistic Y? ■ Isn’t that a biased sample? ■ What was your control group? ■ How did you do study procedure Z? 58 Tips ■ Describe your sample ■ Minimal demographics – number of subjects, broken down by gender ■ Better: age, occupation, major, year ■ Minimize text on your charts ■ If you use a novel measure (e.g., new survey) you must give details on the measure ■ Actual questions asked ■ Any reliability/validity/psychometrics done ■ If you do interviews, include actual quotes ■ Build from data to conclusions ■ Practice your timing/delivery with your project team 59