TEL 300 Fall 2010 Introduction to Surveys According to MSN: Encarta survey the verb ‘survey’ is defined: sur·vey [ sər váy ] transitive verb (past and past participle sur·veyed, present participle sur·vey·ing, 3rd person present singular sur·veys) Definition: 1. consider something generally: to look at or consider something in a general or very broad way 2. look at something carefully: to look at or consider somebody or something closely, especially in order to form an opinion 3. plot map of somewhere: to make a detailed map of an area of land, including its boundaries, area, and elevation, using geometry and trigonometry to measure angles and distances 4. question people in poll: to do a statistical study of a sample population by asking questions about age, income, opinions, buying preferences, and other aspects of people's lives 5. U.K. inspect building: to inspect a building in order to determine its structural soundness or assess its value The social science research method used to survey a population incorporates several of these ideas. In a social science survey, we seek a broad, general understanding of a population. We may also wish to look carefully at some features of that population or of some subpopulation. The strength of the method is its ability to gather a large amount of information on individuals, groups, etc. that can then be used to identify relationships among several concepts. As an example, we might measure respondents’ religious beliefs, media choices, political attitudes and behavior, gender, income, race, ethnicity, state of residence, and a number of other variables. Hypotheses regarding relationships among these many factors can be tested statistically using the data collected via this method. The basics of the method are fairly simple. A series of measures are developed and included in a questionnaire. A group of people, couples, groups, organizations, etc. is chosen to be measured and recruited to respond to the questionnaire. The questionnaire is presented in one of several ways. Respondents provide responses to the measures. The responses either are in numeric form or are translated into numbers. Statistical analyses of the data allow for testing of hypotheses concerning relationships among variables or for investigation of research questions. If we wanted to determine how demographic variables (gender, race, religion, household income) and news media use (newspaper reading, television news viewership, etc.) relates to political behavior and beliefs we would ask a series of questions meant to measure each of these variables and then run statistical analyses on the numbers generated. Most surveys are sample surveys, gathering data from a subset of the population we are interested in studying. Careful construction of the sample used will allow us to project our findings from a few hundred respondents to estimate with surprising accuracy the characteristics of populations of hundreds of millions of people. Surveys of less than one thousand adults are used to estimate the candidate choice of over 100 million voters in the United States. The point is that a relatively small sample can be measured using survey techniques and yield results quite close to what we would have found had the entire population been measured. For the results of survey studies to be valid, both the sample of respondents who generate data, and the measures used to generate the data must be of high quality. The final group of respondents can deviate from a representative sample of the population under study in two ways: 1) the original sample drawn may be biased and unrepresentative, or 2) those who choose to respond to the questionnaire may differ from those who refuse in some significant way. Surveys begin with a set of questions the researcher wishes to address. If the questions can be effectively studied using survey methods she may choose this method of research. Should a survey be chosen as the best method to address the research concerns then a questionnaire must be developed. Concepts and their relationships are operationalized and a set of concrete measures developed that can be presented to respondents. Survey development Questionnaire construction Method of delivery Analysis Questionnaire format Most questionnaires follow a set format. To start, an introduction providing information on the purpose of the research, its social significance and information about the study sponsor/investigator is provided. The contact is asked to participate and assured of confidentiality, etc. If the respondent provides affirmative consent, the interviewer continues or if the respondent is filling out the questionnaire she continues to the first section. The early questions are usually general in nature and simple. They are meant to get the respondent to focus on the topic of interest and to gain basic information that will often be used to determine what questions are appropriate for the respondent or, in some cases, whether the respondent fits a quota group yet to be filled. The first few questions should probably be closed-ended so that the respondent gets used to the interaction and to providing information before being asked to complete this more complicated and demanding task. The early middle portion of the questionnaire focuses in on more specific types of information. It often is organized to work through topic areas sequentially. Also, category questions precede specific brand questions, network questions precede local station questions, etc. The late middle portion includes the most specific questions on the topical issues, and may ask open-ends for more exploratory research. In interview formats the greatest level of rapport should have been reached, with the respondent feeling most comfortable with the interviewer and the types of questions being asked. In self-administered surveys the information previously collected may indicate professionalism on the part of the researcher and ease the concern. A second influence may be that once someone has already given a fair amount of information he will have a tendency to give more, as we find with small donations predicting significantly greater donations later on. The last questions in the main body of the instrument are likely to include open-ends that ask for summary-like statements, closed-ends calling for evaluations or conclusions, and the most sensitive of all questions to be asked. In most surveys, demographic questions are saved for the end as a number of respondents find them intrusive and may refuse to answer them or may quit the interview at this point. This is often the most difficult part of an interview for the interviewers, calling for the greatest amount of subtlety in both demonstrating empathy with respondents and at the same time encouraging them to respond. Questions concerning the reason and need for the information are common at this time and interviewers should have practiced answers for those questions. During the closing section, the interviewer thanks the respondent for her time, lets her know that the interview is over, and asks if any questions remain. Any debriefing that is appropriate based on deception or vagueness that was included in the study should take place at this time. The respondent is often asked for her first name for callbacks should it be necessary. Question formats A variety of question types can be used in survey research. The quality of the questions is probably the single most important influence over the validity of the survey. Small changes in wording can lead to large differences in results. Question formats also have a significant impact on responses obtained for any given topic, as do the response options available. These issues are addressed in another method introduction. Survey instruments or ‘questionnaires’ may either be administered by having an interviewer ask respondents questions and record the responses or by distributing the questionnaire to a sample and have them fill out and return the instrument. A number of advantages and disadvantages are tied to this distinction. Interviewing—a trained set of questioners gathers data from a sample of people. They may be responsible for securing respondent cooperation as well. The interviewer is expected to maintain a neutral attitude with regard to the content of responses while being encouraging and courteous toward the respondents themselves. Interviewers develop an ability to get reluctant respondents to answer questions without upsetting those respondents. Interviews usually are carried out either face-to-face or over the phone. Face-to-face interviews may be quite expensive, especially when the interviewer must travel to meet with the respondent. The cost climbs as the geographic distance between respondents increases. If a random sample of a large population is desired, the travel time between interviews can be extensive and the costs high—perhaps prohibitive. Costs can be lowered significantly if the sample is concentrated or congregated in one spot (as in a convention or a prison). If the sample does not need to be random for the purposes of the study, then the costs can be cut further. Face-to-face interviews have several advantages and a few disadvantages. The FTF format allows interviewers to observe the respondent and to make notes of nonverbal information encountered. The rapport built between the respondent and the interviewer helps ease concerns about respondent privacy and to encourage completion of all questions. In-person contacts are more likely to elicit a response than are the less personal methods. They also are more effective in getting respondents to complete the instrument rather than dropping out partway through. Phone interviews are just as one might expect. The interviewer calls homes from a sample list, identifies the appropriate respondent in the home, secures informed consent, and administers the questionnaire over the phone. Phone interviewers are usually trained prior to contacting respondents in order to make sure that they do not stumble over the words or misread the questions. They are coached on how to respond to questions interviewees may ask or problems they may encounter. General training improves their ability to develop rapport with the respondents, to encourage those who answer the phone to complete the survey and those who begin the survey to complete it. Another area of concern is how closely the interviewer sticks to the script. In order to encourage respondents to answer questions or stay on the line, a number of interviewers will go off script, engage the respondent in conversation, etc. The impact of these behaviors on the validity of the information gathered is not well known, but the belief is that close adherence to the formal questionnaire and scripted answers to common questions from respondents is the best policy. Although phone interviews are somewhat less intimate than face-to-face interviews, much of what is valued in FTF is retained in phone interviews. The human voice does help to generate rapport between the interviewer and the respondent, to encourage people to provide data and to complete surveys, and to provide good data. Phone interviewers can provide guidance when respondents do not understand a question or need help in providing a suitable answer, though to a lesser extent than in FTF interviews. Phone interviews do not allow for the presentation of physical materials or visual content for respondent reaction. Telephone surveys do generate fairly high response rates, though the rates have been dropping over time. Technological change—answering machines, cell phones, call blocking and other developments over time have reduced interviewer abilities to reach respondents. Phone interviews increase respondents’ feeling of confidentiality and can increase their ease in answering embarrassing questions. Phone surveys, though not cheap, are a good deal less expensive than are FTF interviews in the great majority of cases. Paper questionnaires, once the most common means of gathering information, are still in common use. Rather rare now, the mailed questionnaire was quite common in the past. It is still used when the researcher wants to reach a geographically dispersed and specialized population, or has some other special needs to fill. Questionnaires of this sort are usually self-administered. That is, the respondent herself reads the questions and fills in the answers. With the mailed questionnaire the researcher usually provides a stamped, self-addressed envelope that the respondent can place the completed questionnaire in in order to return it. Questions and instructions must be clear and self-explanatory for the most part so that the data collected and the conclusions drawn are valid. Limited visual materials and video on discs can be distributed along with the questionnaires, providing the method with a level of flexibility. Rapport suffers in comparison to interview methods, and the response rate for most respondent groups drops sharply. The potential for incorrect answers/missed skip patterns, etc. rises. Although careful instructions can improve the performance along these lines, the lack of an interviewer leads to mistakes. The need to limit such mistakes leads most researchers to avoid complicated or difficult questions when developing the survey instrument. Advantages of mailed questionnaires include, foremost, a significant decrease in cost compared to interviews. The potential for bias introduced by interviewers disappears, but the advantages of interviewing disappear as well. For some groups of people it may be difficult to generate a large sample of respondents via phone and be too expensive to reach an adequate number face to face, so the mailed questionnaire may be the only practical option. As Internet access becomes nearly universal, this method (outlined below) may provide a more popular option to reach these groups. Another means for distributing paper questionnaires remains viable. When groups of people with special characteristics of interest are concentrated geographically, especially when they congregate for some reason, paper questionnaires may be distributed en mass. That is, if one wanted to contact veterans concerning their health needs, it might be cost-effective to hand out questionnaires at meetings of local veteran groups or visit a national convention for that purpose. If you wished to ask members of minor political parties about their beliefs and experiences, traveling to their conventions or meetings might be an appropriate means to do so. Group administration can generate high levels of return at low overall cost. The samples are likely to be biased in some ways, though. Internet-distributed questionnaires have become very common over recent years and are likely to be the most common means of surveying widespread populations and even perhaps specialized populations. Questionnaires distributed over the Internet are of two main types—either attached to e-mails or situated on a website that respondents are directed to somehow. The former are declining in favor of the latter as Web survey providers are developing software that makes questionnaire development and survey hosting simple and relatively advanced. High-quality questionnaires can be easily written using commonlyavailable programs and a number of advantages of website hosting of surveys have been found. Web survey answers are automatically entered into a database, eliminating the time and potential errors of coding and transferring answers from paper surveys or the potential mistyping of interviewers using software while asking questions. Respondents can take their time to answer without increasing the cost of the survey and can revisit earlier answers if the survey is set up in that way. Samples can be much larger than any other method—once the website is set up, additional respondents add very little to the cost of the study. Very sophisticated skip patterns can be used. The medium allows for the inclusion of multimedia and online help of several types. The main disadvantage of online questionnaires is that the response rate is quite low. Most people ignore the pop-up requests, e-mail solicitation, etc. that is meant to get them to complete questionnaires online. The ubiquity of such requests also reflects the ease of developing these studies and sending out requests over the Internet. It may be too easy to develop online surveys. When bombarded with requests, potential respondents will respond to fewer and fewer. Analysis Researchers usually apply statistical analyses to survey data. Most survey data is in numeric form and those that are not are often coded into numeric form. That is, trained coders score open-ended questions and any other non-numeric data so that they can be used in statistical analyses along with the numeric data. Most analyses begin with descriptive statistics for sample demographics as well as the variables to be used in the analysis. When the sample deviates from the population on demographic characteristics (should that information be known) the deviation is noted and under some circumstances corrected for statistically. Descriptive analyses of the variables to be used in the analyses is used to determine whether the distribution of scores allows for parametric statistics to be applied. In some cases, the distribution of characteristics within the sample is considered important information in itself. Hypothesis testing is usually carried out using measures of association such as correlation coefficients, regression betas, etc. Advanced statistics allow researchers to evaluate the interrelations among a number of variables at once, developing complicated models of association.