Survey Research Prying into Your Personal Life in Ways Telemarketers Could Never Dream Of Appropriate Research Questions for Surveys • Self-reported beliefs and behaviors • Ask things at one time, measure many variables, and test several hypotheses Appropriate Research Questions for Surveys • Behavior – How often do you brush your teeth? • Attitudes/Beliefs/Opinions – What is the biggest problem facing the nation today? • Characteristics – Do you belong to a union? • Expectations – Do you plan to buy a new car in the next year? • Self-classification – Are you liberal or conservative? • Knowledge – Who was elected mayor in the last election? Key Limitation • Surveys ONLY provide data on what a person or organization says • This can differ from what they actually do or think • Survey research is DEDUCTIVE Steps in Conducting a Survey • Step 1: – – – – – Develop hypotheses Decide type of survey Write survey questions Decide on response categories Design layout Steps in Conducting a Survey • Step 2: – Plan how to record data – Pilot test survey instrument • Step 3: – – – – Decide on target population Get sampling frame Decide on sample size Select sample Steps in Conducting a Survey • Step 4: – Locate respondents – Conduct interviews – Carefully record data • Step 5: – Enter data into computers – Recheck all data – Perform statistical analysis on data Steps in Conducting a Survey • Step 6: – Describe methods and findings in research report – Present findings to others for critique and evaluation Constructing the Questionnaire • Introductory remarks and instructions • Keep it clear, keep it simple, keep respondent’s perspective in mind • Question writing is more of an art than a science – Skill, practice, patient, and creativity Questionnaire Hints • 1) Avoid jargon, slang, and abbreviations – No abbreviations – Only use this if you’re surveying a specialized population – “Did you drown in brew until you were totally blasted last night?” Questionnaire Hints • 2) Avoid ambiguity, confusion, and vagueness – “What is your income?” – Confusion causes inconsistencies • 3) Avoid emotional language – Implicit connotative as well as explicit denotative – Use neutral language – “What do you think about a policy to pay murderous terrorists who threaten to steal the freedoms of peaceloving people?” Questionnaire Hints • 4) Avoid prestige bias – Avoid associating statements with them – “Most doctors say that cigarette smoke causes lung disease for those near a smoker. Do you agree?” • 5) Avoid double-barreled questions – Each question = ONE QUESTION ONLY – “Does this company have pension and health insurance benefitted?” Questionnaire Hints • 6) Do not confuse beliefs with reality – Don’t mix up respondents beliefs with what we want to measure – “Do you rate a teacher higher if the teacher tells many jokes?” • 7) Avoid leading questions – Don’t lead them to an answer – “You don’t smoke, do you?” Questionnaire Hints • 8) Avoid asking questions that are beyond respondents’ capabilities – Frustrates respondents – Produces poor-quality responses – “Two years ago, how many hours did you watch TV every month?” Questionnaire Hints • 9) Avoid false premises – Can’t put a premise that respondent could disagree with without risking alienating them – “When did you stop beating your girl/boyfriend?” Questionnaire Hints • 10) Avoid asking about intentions in the distant future – Hypotheticals are bad – Not specific or concrete – “After you graduate college and have kids, will you invest in the stock market?” Questionnaire Hints • 11) Avoid double negatives – Grammatically incorrect and confusing – “Do you disagree with those who do not want to build a new city swimming pool?” Questionnaire Hints • 12) Avoid overlapping or unbalanced response categories – Mutually exclusive: categories do not overlap – Exhaustive: every respondent has a choice – Balanced: Offer polar opposites at each end of a continuum – “Did you find the service at our hotel to be, Outstanding, Excellent, Superior, or Good?” Aiding Recall • Ability to recall accurately declines over time • Memory is less trustworthy than once assumed • Give respondents special instructions and extra thinking time to guide them Types of Questions • • • • Threatening Questions Socially Desirable Questions Knowledge Questions Skip or Contingency Questions Threatening • Sensitive issues…people don’t want to talk about them • Underreport socially undesirable answers • Context and wording are how to fix this • Issue examples: – Masturbation, sexual intercourse, illicit drug use, income Socially Desirable • Over report socially desirable answers (Social desirability bias) – Giving money to charity, voting etc. • Have to make the norm violation seem less objectionable then it really could be Knowledge • We can’t answer elementary questions as Americans… Are You Smarter than a 5th Grader?? • Pilot-test to make sure they’re at an appropriate level of difficulty Skip or Contingency • Two or more part question – Answer to one part determines the next question you’ll receive – Sometimes called screening questions Open v. Closed Questions • Open-ended: Respondent can give any answer they want • Close-ended: Gives respondents fixed answers from which to choose • Big surveys use close-ended…quicker and easier • Can mix both methods • Open-ended best in early or exploratory stages of research • Partially open questions leave final answer choice • as other Advantages of Closed • • • • Easy and quick to do Easy to compare Easier to code and analyze Response choices can clarify meaning of question • Replication is easier Disadvantages of Closed • Suggest ideas that respondent may not have thought of • No opinion people can answer anyways • Frustrating if desired answer is not a choice • Simplistic responses to complex ?s • Force us to make choices we wouldn’t make in the real world Advantages of Open • Unlimited responses • Answer in detail and clarify • Unanticipated feelings can be discovered • Creativity and self-expression • Reveal logic Disadvantages of Open • • • • • • • Different degrees of detail in answers Can be irrelevant Hard to compare/analyze Coding is tough Hard to write verbatim in interviews More time necessary Take up lots of space on questionnaire Nonattitudes and Middle Positions • Debate whether to include choices for middle, neutral, or nonattitudes • Two errors: – Accept nonattitude when respondent has one – Force choice when respondent doesn’t have one • Standard-format v. quasi-format v. fullformat Agree/Disagree, Rankings, or Ratings? • • • • • Agree/Disagree Rankings Ratings Best to offer respondents explicit alternatives Avoid the response set bias (just check agree to everything) • Respondents can rate several items equally high, but will put them in hierarchy if asked to Word Issues • Need to use simple vocabulary and grammar to minimize confusion • Need to watch for effects of specific words or phrases – Forbid v. not allow – Certain words trigger an emotional reaction – “To help the poor” v. “for welfare” Length of Survey • Researchers prefer long because they’re cost effective • Phone: 5-20 minutes • Mail questionnaire: 3-4 pages • Face-to-face: Around an hour Order Effects • Strongest for people who lack strong views, less educated, elderly • May not perceive each element of a survey as isolated and separate • Influence through content and response Context Effects • Funnel sequence: – Start general and get more specific • Split sample – Half get questions in one order, other half different • More ambiguous the question, the more context effects matter Survey Layout • • • • • • Clear, neat, and easy to follow Number questions Identifying info Cover sheet Professional appearance Provide instructions Survey Layout • • • • Crucial in mail questionnaires Use letterhead Always thank them for participation Circle response v. check box v. fill in dots v. put an X in a blank • List answer categories down, not across Survey Layout • Use arrows and instructions for contingency questions • Matrix questions – Compact way to present a series of questions with the same response categories – Box 7.5 Nonresponse • If not enough people respond, we may not be able to generalize • Many people are burnt out with surveys these days • Improve eligibility rates through careful respondent selection Nonresponse • Decrease refusals through sending letters before calling, small incentives, adjusting behavior, using alternative interviewers, using alternative interview methods • Initial contact is HUGE in determining likelihood for success Surveys • Survey research is research based on direct or indirect interview methods. • Many types of surveys: – – – – – In person (including interviews) Mail Telephone E-mail and Internet Group surveys Mail and Self-Administered Questionnaires • Advantages: – – – – – Cheaper Wide geographical area Completed when convenient Respondent can check personal records Anonymity Mail and Self-Administered Questionnaires • Disadvantages • Low response rate • Can’t control conditions under which its filled out • Someone other than intended can fill out • Can’t observe reactions • Ill suited for those who don’t have good grasp of English Mail and Self-Administered Questionnaires • How to Increase Response – – – – – – – – Address to specific person Careful cover letter on letterhead Postage-paid, addressed return envelope Neat, attractive layout Send two follow-up reminders Nothing during major holidays No questions on back page Include a small monetary inducement Web Surveys • Advantages – – – – Very fast Inexpensive Flexible design Can use visuals or audio Web Surveys • Disadvantages – – – – Coverage (75% have internet) Multiple e-mail addresses Self-selection Protecting privacy and confidentiality • Encrypt – Complexity of design depends on software • Screen-by-screen questions • Progress indicator • Needs to be easy to move back and forth across questions Phone Interviews • Advantages – – – – 95% coverage rate Can interview about 1500 in a few days Half the cost of face-to-face interviews CATI Phone Interviews • Disadvantages: – Higher cost – Limited interview length – Use of interviewer reduces anonymity and introduces potential interviewer bias Face-to-Face • Advantages: – – – – Highest response rates Permit the longest questionnaires Can observe surroundings Ask all types of questions Face-to-Face • Disadvantages: – High cost • Training, travel, supervision, personnel costs – Interviewer bias Comparing the Methods • Cost: – I, M, P, F2F • Speed: – I, P, F2F, M • Length: – F2F, M, I, P • Response rate: – F2F, P, I, M Interviewers • Obtain cooperation and build rapport, but remain neutral and objective • Interviewer must treat everything in a matter-of-fact manner • Pleasant, honest, accurate, mature, responsible, moderately intelligent, stable, and motivated • Poise and tact Interviewer Bias • Interview expectations can create significant bias • Social setting can affect answers – Students answer differently at home and at school • Visible characteristics can play a role…such as race and gender CATI • Computer-assisted telephone interviewing • Read questions from the screen and enter answers immediately • Automatically makes calls for you • Automatically stores answers in a database The Ethical Survey • Invasion of privacy – That’s what ANY survey is!!!! • Voluntary participation – Informed consent • Use of push polls—AKA “pseudosurveys” • Sloppy reporting of surveys by the media – No quality control standards Surveys • Five important factors influence the choice of survey type: – – – – – Cost Completion rates Sample-population congruence Questionnaire length Data-processing issues Surveys • Cost: – All surveys share basic expenses for materials. – Cost of a survey can be quite variable, based on • • • • • • • Professional questionnaire design Length of the questionnaire Geographic dispersion of the sample Callback procedures Respondent selection rules Availability of trained staff Method-specific costs like long distance rates or postage Surveys • Completion rates: – Low response rate on a survey limits the ability to make statistical inferences about the target population – Must make an effort to reach those selected for inclusion in the sample to maximize the representativeness of the sample…or risk bias Surveys • Sample-population congruence: – If those who fail to participate in the survey share common characteristics, the sample will be wrought with systematic sampling error and the conclusions made with the data could be incorrect. Surveys • Questionnaire length: – Keeping respondents’ attention during a survey helps achieve the best possible response quality. – A lengthy questionnaire can lead respondents to lose focus or speed through questions to finish sooner, resulting in low response quality. – Survey length ranges between three and four dozen items, depending on complexity and format. Surveys • Data-processing issues – A good deal of time and effort are required to process data. – Surveys using more elaborate or complex designs or a large number of questions will require more time and effort. Surveys • Each type of survey has advantages and disadvantages when it comes to the five factors.