Bias in Surveys and Polls Bias occurs when a poll or survey produces results that do not reflect the true opinions or beliefs of the general population. This is often a result of the methods used to conduct the survey or the wording of the questions asked. Non-Response Bias 1. Voluntary Response In phone-in surveys and internet polls, people are not likely to participate in the survey unless they are already likely to agree with the questions asked or are likely to give opinions related to the subject of the survey. 2. Lack of Response In some surveys, members of the survey group may refuse to answer certain questions or may refuse to participate in the survey. This can particularly happen with phone surveys or mail-in surveys. This also can include when a group of people is excluded from participating in the poll, intentionally or not. For example, if a phone survey uses only randomly chosen land-line phone numbers, those who have only a cell phone would not have a chance to be included in the survey. Response Bias 3. Leading Questions How questions are asked is very important in surveys. A survey question can be worded in such a way as to direct a person to answer in only one way. 4. Social Acceptance Often, when questions about controversial issues are asked, survey respondents may give answers contrary to their true beliefs in order to conform to a societal standard they believe is acceptable. Formulating Survey Questions When conducting a survey or poll, what type of questions and the wording of questions are important considerations. Poorly worded questions can invalidate the results of a survey. Questions should be asked in a manner that is balanced. Consider the questions: “Do you feel that the increasing cost of the high speed rail project is too expensive for California?” “Do you feel that high speed rail will be important to the future economy of California?” “Do you approve or disapprove of building a high speed rail system in California?” Questions should not be vague. For example, the question “What’s wrong with the economy?” is vague. It is unclear what the question is trying to determine. The type of question you ask is important: o In an open question, respondents are free to formulate their own response. o In a closed question, respondents choose from a list of predetermined responses. For example, the question: “What is the most important problem facing California today?” is an open question, whereas: “Which of the following do you feel is the most important problem facing California today? a) Violence in schools b) Use of drugs c) Gun control laws d) Balancing the State Budget is a closed question. When asking closed questions, the order that the choices are listed should be rotated. This is because earlier choices in a list are more likely to be chosen than later choices. The option “No opinion” should be omitted, since this does not allow for meaningful analysis. Here are some questions from recent polls and surveys regarding same sex marriage. Discuss the issues of bias and fairness in these questions: Should states continue to discriminate against couples who want to marry and who are of the same gender? Do you support marriage equality? Should states be forced to legalize homosexual marriage over the wishes of a majority of the people? Do you think marriages between same-sex couples should or should not be recognized by the law as valid, with the same rights as traditional marriages?