Survey Research Slide 1 Data Collection Design: Survey Research T. Ramayah School of Management, Universiti Sains Malaysia ramayah@usm.my, http://www.ramayah.com Slide 2 Learning Objectives Understand . . . 1. The process for selecting the appropriate and optimal survey approach. 2. Factors that affect participation in survey research. 3. Sources of error in communication studies and how to minimize them. 4. Major advantages and disadvantages of the three survey methods. 5. Why an organization might outsource a communication study. 6. Ethical issues with survey research and their solutions. Slide 3 “We saw that our customers required help beyond the data sets they had and that they could benefit from a wider opinion. So we built SurveyMonkey Audience, and we’ve now got 4 million users who signed up to take surveys. Our clients can choose the demographic they want to hear from, and we can provide that sample.” David Goldberg CEO SurveyMonkey Slide 4 Data Collection Approach Slide 5 Communication Approach • The communication approach involves surveying people, through the use of a questionnaire or a personal interview. • People often make the mistake of seeing survey research in business as the only appropriate data collection design. Rather, it should be seen as a strong complement to observation research. • When is it useful: • Observation cannot give you the information needed • Articulation of ideas • Speed • Cost Slide 6 Social Media Analytics vs. Survey If social media is the temperature check, surveys are the taste test to validate that the meal (kale salad) is cooked properly. Slide 7 Selecting a Communication Data Collection Approach Slide 8 Communication Approach Strengths • Versatility • Efficiency • Geographic coverage Weaknesses • Error • Inaccessible populations Slide 9 Sources of Error Error Sources Instrument Participant Interviewer Situation Slide 10 1. Instrument • Wording of the question • Interpretation of some words • Jargons Slide 11 2. Participant • Possess the information being targeted by the investigative questions • Understand his or her role in the survey as the provider of accurate information • Must have adequate motivation to cooperate (see the next slide) Slide 12 Participant Motivation Slide 13 3. Interviewer Error 1. Sampling Error ▪ Failure to secure full participant cooperation 2. Data Entry Error ▪ Failure to record answers accurately and completely 3. Process Errors ▪ Failure to consistently execute interview procedures ▪ Failure to establish appropriate interview environment ▪ Physical presence bias ▪ Inappropriate influencing ▪ Falsification of individual answers or whole interviews. Slide 14 4. Situation • When data was collected • Where the data was collected Slide 15 Response Terms Noncontact rate Refusal rate Incidence rate Slide 16 Response Terms 1. The noncontact rate is a ratio of potential but unreached contacts to all potential contacts. A contact may be unreachable due to no answer, busy signal, answering machine or voice mail, and disconnects). 2. The refusal rate refers to the ratio of contacted participants who decline the interview to all potential contacts. 3. The incidence rate refers to the ratio of contacted people who actually qualify for the survey to all contacts. Total number of qualified respondents divided by the total number of respondents who were screened for the study (qualified plus nonqualified). Slide 17 Communication Approaches SelfAdministered Survey Telephone Survey Survey via Personal Interview Slide 18 Surveys Slide 19 Self-Administered Surveys Mobile service provider Mailed/Faxed Messenger App. Delivery Modes Courier Computer delivered Intercept Slide 20 Self-Administered Surveys – When? Costs Sample Accessibility Topic Coverage Systematic Anonymity Time Constraints Slide 21 Advantages of Self-Administered Study • Access inaccessible participants • Rapid data collection • Incentives for higher response rates • Visuals possible • Lowest-cost • Multiple sampling possible • Geographic coverage • Minimal staff needed • Perceived anonymity • Reflection time • Question complexity Slide 22 Disadvantages of Self-Administered Study • Low response rates in some modes • Skewed responses by extremists • No interviewer intervention • Participant anxiety possible • Need for low-distraction environment • Security • Requires accurate list • Cannot be too long • Directions necessary • Cannot be too complex Slide 23 Designing Questionnaires (TDM Method) • The TDM method refers to Don Easy to read Dillman’s Total Design Method. Offer clear directions • He proposes that surveys be based upon social exchange Include personalization theory. • One suggestion that flows from Notify in advance social exchange theory is that the burden to participants should be Encourage response minimized. Slide 24 Improving Response Rates • • • • • • • Advance notification Reminders Return directions and devices Monetary incentives Deadlines Promise of anonymity Appeal for participation Slide 25 Options for Web-based Surveys Fee-Based Service Surveying Software Slide 26 Surveying Software Slide 27 Advantages of Surveying Software • Questionnaire design in word processing environment • Question and scale libraries • Automated publishing to the Web • Real-time viewing of incoming data • Rapid transmission of results • Flexible analysis and reporting mechanisms Slide 28 The Web as a Survey Research Venue Advantages Disadvantages • • • • • • Cost savings • Recruitment Short turnaround • Coverage Use of visual stimuli • Difficulty developing probability Access to participants samples Perception of anonymity • Technical skill Access to data and experiences • System compatibility issues otherwise unavailable • Possible self-selection bias Slide 29 Telephone Survey Traditional CATI systems Computeradministered Slide 30 Advantages of the Telephone Survey • Lower costs than personal interview • Wide geographic coverage • Fewer interviewers • Reduced interviewer bias • Fast completion time • Random Dialing • CATI Slide 31 Disadvantages of the Telephone Survey • Lower response rate iPhone • Early termination • Higher costs if geographically dispersed sample • Limited Interview length • Inaccessible populations • Limited complexity of scales Voice-over IP 32 Slide 32 Survey via Personal Interview • A survey via personal interview is a quantitative study using an individual depth interview (IDI) between a trained interviewer and a single participant taking anywhere from 20 minutes to an hour or more. Pre-scheduled Intercept • This structured interview uses a measurement instrument similar to a questionnaire which specifies the questions and the order of the questions. Slide 33 Personal Interview Survey Advantages Disadvantages • Good cooperation rates • Interviewer can probe and • High costs explain • Visual aids possible • Illiterate participants can be reached • Interviewer can prescreen • CAPI possible • Need for highly trained interviewers • Time consuming • Labor-intensive • Some unwilling to invite strangers into homes • Interviewer bias possible Slide 34 Outsourcing Data Analysis Sampling via Panels Survey Tasks Omnibus • Individual questions in omnibus studies are used to capture everything from people’s feeling about the rise in gasoline prices to the power of a celebrity spokesperson in an advertising campaign or the latest teenage fashion trend. • Questions of more than one company are fielded in the same survey, but where the results are separated Survey Fielding Ethical Issues Deception Confidentiality Quality Safety Slide 36 Ethical Issues 1. Deception occurs when participants are told only part of the truth or when truth is fully compromised. Survey research requires an estimate of time commitment for the participant. If researchers aren’t careful in questionnaire design, surveys are often longer than they need to be. If a researcher fails to correctly estimate time, he or she may falsely gain a participant’s cooperation. 2. The right to confidentiality—to protect their identity and specific responses from being attached to their identity. Researchers address these rights by prescreening participants to ensure willingness, prescheduling survey experiences, limiting time requirements, restricting access to participant identification, restricting access to completed data instruments, and revealing participant information only with signed consent. Slide 37 Ethical Issues 3. Quality relates to avoidance of error, which translates to strict adherence to questionnaire structure and instructions. It also relates to not asking unnecessary questions to achieve the research objective or making claims about data that cannot be supported by the survey protocols employed. Interviewers have the responsibility to not falsify survey completions. 4. Safety refers to not putting the interviewer in harm’s way due to the environment where the interview takes place (e.g., night interview, high crime location) or the people they must interview. Researchers, who are insensitive to interviewers’ fears of bodily harm, face potential legal consequences as well as research quality issues (falsified measurement instruments). The solution involves choosing a different interviewer or a different survey method. Slide 38 Ethical Issues and their Solutions • Adequately pretest the survey to determine a solid estimate of the time needed for completion. Underestimates of completion time also lead to higher drop-out rates that, for any given survey, cause poorer-quality data. Informed Consent In the broader sense, such underestimates discourage people from participating in surveys, increasing the likelihood of nonresponse error. • To protect participants from the consequences of deception, researchers should explain the benefits of the study (through introducing themselves or Pretest their company, explaining participants’ rights and protections, and obtaining informed consent—a full disclosure of research procedures). • Researcher solutions to quality issues relate to interviewer training and to Training following appropriate industry standards for measurement—including selection and development of measurement scales and measurement instruments—and for data analysis. Slide 39 Informed Consent Slide 40 Institutional Review Board (IRB) Process (JEPEM) Purpose IRB Actions Applications Process Revision Initial Review Continuing Review Prepare IRB Materials Slide 41 What is important in this chapter? • Types of communication • Sources of error • Ethical Issues and Solution Slide 42 Thank you for listening Slide 43 43