TEL 300 Fall 2010 Introduction to Surveys
According to MSN: Encarta survey the verb ‘survey’ is defined:
sur·vey [ sər váy ]
transitive verb (past and past participle sur·veyed, present participle sur·vey·ing, 3rd person
present singular sur·veys)
Definition: 1. consider something generally: to look at or consider something in a general or
very broad way
2. look at something carefully: to look at or consider somebody or something closely,
especially in order to form an opinion
3. plot map of somewhere: to make a detailed map of an area of land, including its boundaries,
area, and elevation, using geometry and trigonometry to measure angles and distances
4. question people in poll: to do a statistical study of a sample population by asking questions
about age, income, opinions, buying preferences, and other aspects of people's lives
5. U.K. inspect building: to inspect a building in order to determine its structural soundness or
assess its value
The social science research method used to survey a population incorporates several of these ideas. In a
social science survey, we seek a broad, general understanding of a population. We may also wish to look
carefully at some features of that population or of some subpopulation. The strength of the method is its
ability to gather a large amount of information on individuals, groups, etc. that can then be used to
identify relationships among several concepts.
As an example, we might measure respondents’ religious beliefs, media choices, political
attitudes and behavior, gender, income, race, ethnicity, state of residence, and a number of other
variables. Hypotheses regarding relationships among these many factors can be tested
statistically using the data collected via this method.
The basics of the method are fairly simple. A series of measures are developed and included in a
questionnaire. A group of people, couples, groups, organizations, etc. is chosen to be measured and
recruited to respond to the questionnaire. The questionnaire is presented in one of several ways.
Respondents provide responses to the measures. The responses either are in numeric form or are
translated into numbers. Statistical analyses of the data allow for testing of hypotheses concerning
relationships among variables or for investigation of research questions.
If we wanted to determine how demographic variables (gender, race, religion, household income)
and news media use (newspaper reading, television news viewership, etc.) relates to political
behavior and beliefs we would ask a series of questions meant to measure each of these variables
and then run statistical analyses on the numbers generated.
Most surveys are sample surveys, gathering data from a subset of the population we are interested in
studying. Careful construction of the sample used will allow us to project our findings from a few
hundred respondents to estimate with surprising accuracy the characteristics of populations of hundreds of
millions of people. Surveys of less than one thousand adults are used to estimate the candidate choice of
over 100 million voters in the United States. The point is that a relatively small sample can be measured
using survey techniques and yield results quite close to what we would have found had the entire
population been measured.
For the results of survey studies to be valid, both the sample of respondents who generate data, and the
measures used to generate the data must be of high quality. The final group of respondents can deviate
from a representative sample of the population under study in two ways: 1) the original sample drawn
may be biased and unrepresentative, or 2) those who choose to respond to the questionnaire may differ
from those who refuse in some significant way.
Surveys begin with a set of questions the researcher wishes to address. If the questions can be effectively
studied using survey methods she may choose this method of research. Should a survey be chosen as the
best method to address the research concerns then a questionnaire must be developed. Concepts and their
relationships are operationalized and a set of concrete measures developed that can be presented to
respondents.
Survey development
Questionnaire construction
Method of delivery
Analysis
Questionnaire format
Most questionnaires follow a set format. To start, an introduction providing information on the purpose
of the research, its social significance and information about the study sponsor/investigator is provided.
The contact is asked to participate and assured of confidentiality, etc. If the respondent provides
affirmative consent, the interviewer continues or if the respondent is filling out the questionnaire she
continues to the first section.
The early questions are usually general in nature and simple. They are meant to get the respondent to
focus on the topic of interest and to gain basic information that will often be used to determine what
questions are appropriate for the respondent or, in some cases, whether the respondent fits a quota group
yet to be filled. The first few questions should probably be closed-ended so that the respondent gets used
to the interaction and to providing information before being asked to complete this more complicated and
demanding task.
The early middle portion of the questionnaire focuses in on more specific types of information. It often is
organized to work through topic areas sequentially. Also, category questions precede specific brand
questions, network questions precede local station questions, etc.
The late middle portion includes the most specific questions on the topical issues, and may ask open-ends
for more exploratory research. In interview formats the greatest level of rapport should have been
reached, with the respondent feeling most comfortable with the interviewer and the types of questions
being asked. In self-administered surveys the information previously collected may indicate
professionalism on the part of the researcher and ease the concern. A second influence may be that once
someone has already given a fair amount of information he will have a tendency to give more, as we find
with small donations predicting significantly greater donations later on.
The last questions in the main body of the instrument are likely to include open-ends that ask for
summary-like statements, closed-ends calling for evaluations or conclusions, and the most sensitive of all
questions to be asked. In most surveys, demographic questions are saved for the end as a number of
respondents find them intrusive and may refuse to answer them or may quit the interview at this point.
This is often the most difficult part of an interview for the interviewers, calling for the greatest amount of
subtlety in both demonstrating empathy with respondents and at the same time encouraging them to
respond. Questions concerning the reason and need for the information are common at this time and
interviewers should have practiced answers for those questions.
During the closing section, the interviewer thanks the respondent for her time, lets her know that the
interview is over, and asks if any questions remain. Any debriefing that is appropriate based on deception
or vagueness that was included in the study should take place at this time. The respondent is often asked
for her first name for callbacks should it be necessary.
Question formats
A variety of question types can be used in survey research. The quality of the questions is probably the
single most important influence over the validity of the survey. Small changes in wording can lead to
large differences in results. Question formats also have a significant impact on responses obtained for
any given topic, as do the response options available. These issues are addressed in another method
introduction.
Survey instruments or ‘questionnaires’ may either be administered by having an interviewer ask
respondents questions and record the responses or by distributing the questionnaire to a sample and have
them fill out and return the instrument. A number of advantages and disadvantages are tied to this
distinction.
Interviewing—a trained set of questioners gathers data from a sample of people. They may be
responsible for securing respondent cooperation as well. The interviewer is expected to maintain a
neutral attitude with regard to the content of responses while being encouraging and courteous toward the
respondents themselves. Interviewers develop an ability to get reluctant respondents to answer questions
without upsetting those respondents.
Interviews usually are carried out either face-to-face or over the phone. Face-to-face interviews
may be quite expensive, especially when the interviewer must travel to meet with the respondent. The
cost climbs as the geographic distance between respondents increases. If a random sample of a large
population is desired, the travel time between interviews can be extensive and the costs high—perhaps
prohibitive. Costs can be lowered significantly if the sample is concentrated or congregated in one spot
(as in a convention or a prison). If the sample does not need to be random for the purposes of the study,
then the costs can be cut further.
Face-to-face interviews have several advantages and a few disadvantages. The FTF format
allows interviewers to observe the respondent and to make notes of nonverbal information encountered.
The rapport built between the respondent and the interviewer helps ease concerns about respondent
privacy and to encourage completion of all questions. In-person contacts are more likely to elicit a
response than are the less personal methods. They also are more effective in getting respondents to
complete the instrument rather than dropping out partway through.
Phone interviews are just as one might expect. The interviewer calls homes from a sample list,
identifies the appropriate respondent in the home, secures informed consent, and administers the
questionnaire over the phone. Phone interviewers are usually trained prior to contacting respondents in
order to make sure that they do not stumble over the words or misread the questions. They are coached
on how to respond to questions interviewees may ask or problems they may encounter. General training
improves their ability to develop rapport with the respondents, to encourage those who answer the phone
to complete the survey and those who begin the survey to complete it. Another area of concern is how
closely the interviewer sticks to the script. In order to encourage respondents to answer questions or stay
on the line, a number of interviewers will go off script, engage the respondent in conversation, etc. The
impact of these behaviors on the validity of the information gathered is not well known, but the belief is
that close adherence to the formal questionnaire and scripted answers to common questions from
respondents is the best policy.
Although phone interviews are somewhat less intimate than face-to-face interviews, much of what is
valued in FTF is retained in phone interviews. The human voice does help to generate rapport between
the interviewer and the respondent, to encourage people to provide data and to complete surveys, and to
provide good data. Phone interviewers can provide guidance when respondents do not understand a
question or need help in providing a suitable answer, though to a lesser extent than in FTF interviews.
Phone interviews do not allow for the presentation of physical materials or visual content for respondent
reaction.
Telephone surveys do generate fairly high response rates, though the rates have been dropping over time.
Technological change—answering machines, cell phones, call blocking and other developments over time
have reduced interviewer abilities to reach respondents. Phone interviews increase respondents’ feeling
of confidentiality and can increase their ease in answering embarrassing questions. Phone surveys,
though not cheap, are a good deal less expensive than are FTF interviews in the great majority of cases.
Paper questionnaires, once the most common means of gathering information, are still in common use.
Rather rare now, the mailed questionnaire was quite common in the past. It is still used when the
researcher wants to reach a geographically dispersed and specialized population, or has some other
special needs to fill. Questionnaires of this sort are usually self-administered. That is, the respondent
herself reads the questions and fills in the answers. With the mailed questionnaire the researcher usually
provides a stamped, self-addressed envelope that the respondent can place the completed questionnaire in
in order to return it. Questions and instructions must be clear and self-explanatory for the most part so
that the data collected and the conclusions drawn are valid. Limited visual materials and video on discs
can be distributed along with the questionnaires, providing the method with a level of flexibility. Rapport
suffers in comparison to interview methods, and the response rate for most respondent groups drops
sharply. The potential for incorrect answers/missed skip patterns, etc. rises. Although careful instructions
can improve the performance along these lines, the lack of an interviewer leads to mistakes. The need to
limit such mistakes leads most researchers to avoid complicated or difficult questions when developing
the survey instrument.
Advantages of mailed questionnaires include, foremost, a significant decrease in cost compared to
interviews. The potential for bias introduced by interviewers disappears, but the advantages of
interviewing disappear as well. For some groups of people it may be difficult to generate a large sample
of respondents via phone and be too expensive to reach an adequate number face to face, so the mailed
questionnaire may be the only practical option. As Internet access becomes nearly universal, this method
(outlined below) may provide a more popular option to reach these groups.
Another means for distributing paper questionnaires remains viable. When groups of people with special
characteristics of interest are concentrated geographically, especially when they congregate for some
reason, paper questionnaires may be distributed en mass. That is, if one wanted to contact veterans
concerning their health needs, it might be cost-effective to hand out questionnaires at meetings of local
veteran groups or visit a national convention for that purpose. If you wished to ask members of minor
political parties about their beliefs and experiences, traveling to their conventions or meetings might be an
appropriate means to do so.
Group administration can generate high levels of return at low overall cost. The samples are likely to be
biased in some ways, though.
Internet-distributed questionnaires have become very common over recent years and are likely to be
the most common means of surveying widespread populations and even perhaps specialized populations.
Questionnaires distributed over the Internet are of two main types—either attached to e-mails or situated
on a website that respondents are directed to somehow. The former are declining in favor of the latter as
Web survey providers are developing software that makes questionnaire development and survey hosting
simple and relatively advanced. High-quality questionnaires can be easily written using commonlyavailable programs and a number of advantages of website hosting of surveys have been found. Web
survey answers are automatically entered into a database, eliminating the time and potential errors of
coding and transferring answers from paper surveys or the potential mistyping of interviewers using
software while asking questions. Respondents can take their time to answer without increasing the cost of
the survey and can revisit earlier answers if the survey is set up in that way. Samples can be much larger
than any other method—once the website is set up, additional respondents add very little to the cost of the
study. Very sophisticated skip patterns can be used. The medium allows for the inclusion of multimedia
and online help of several types.
The main disadvantage of online questionnaires is that the response rate is quite low. Most people ignore
the pop-up requests, e-mail solicitation, etc. that is meant to get them to complete questionnaires online.
The ubiquity of such requests also reflects the ease of developing these studies and sending out requests
over the Internet. It may be too easy to develop online surveys. When bombarded with requests, potential
respondents will respond to fewer and fewer.
Analysis
Researchers usually apply statistical analyses to survey data. Most survey data is in numeric form and
those that are not are often coded into numeric form. That is, trained coders score open-ended questions
and any other non-numeric data so that they can be used in statistical analyses along with the numeric
data.
Most analyses begin with descriptive statistics for sample demographics as well as the variables to be
used in the analysis. When the sample deviates from the population on demographic characteristics
(should that information be known) the deviation is noted and under some circumstances corrected for
statistically. Descriptive analyses of the variables to be used in the analyses is used to determine whether
the distribution of scores allows for parametric statistics to be applied. In some cases, the distribution of
characteristics within the sample is considered important information in itself.
Hypothesis testing is usually carried out using measures of association such as correlation coefficients,
regression betas, etc. Advanced statistics allow researchers to evaluate the interrelations among a number
of variables at once, developing complicated models of association.