Module 4
Objectives
1. Describe the possible issues that researchers may face when developing a questionnaire.
2. Compare and contrast validity and reliability.
3. Identify and describe the various types of the data collection technique.
4. Describe the primary steps in writing up the research.
Lecture Note
Chapter 9 - Questionnaire Development
I. Surveys and Questionnaires
A. Survey and questionnaire are interchangeable and frequently used
1. In quantitative research design, questionnaires rely on numerical assessment
of data
2. In qualitative research, open-ended questions are used
3. Types: personal interviews, mail questionnaires, and telephone surveys
B. Previously used versus new questionnaire
1. If appropriate for your research, using a questionnaire that has previously been
developed and tested reduces the likelihood of validity and reliability issues
2. If previously used questionnaires don’t exist or meet the specific needs, create
a research-specific questionnaire
3. An existing instrument can also be used with adjustment
C. Things to consider when creating a survey
1. Reliability
2. Validity
3. Questions’ level of measurement
II. Rules for Questionnaire Construction
A. Rule One
1. Start with a list of all the items you are interested in knowing about the group,
concept, or phenomenon
2. “Grocery list” mentality—not committing to the items on the list, but gathers
items that will be easy to remove or modify later
3. Ask for identifying characteristics and responses to the questions
4. What is asked depends on the type of comparisons or analysis
5. Decide what is of interest and what is needed
B. Rule Two
1. If an existing questionnaire is not available for use within the study,
establishing validity and reliability is critical
1
2. If the questionnaire is not valid and reliable, others will not place much value
in the results
3. Validity: whether questionnaire is measuring what it claims to measure
a. Face validity: researcher believes questionnaire is measuring what is being
attempted to be to measured (judgment call)
b. Content validity: researcher must be able to explain why each item
measures what is expected
c. Construct validity: questions measure what they are designated to measure
by examining the relationship that measure has to other variables
(theoretical and operational aspects of the item)
d. Criterion validity: relationship between questionnaire’s results and an
external criterion; provides validity coefficient
4. Reliability
a. Stability: under similar conditions, a respondent will provide the same
answers to the same questions on a second testing
b. Consistency: set of questions is strongly related and provides predictable
results
c. Three ways to test reliability
i. Test and retest (pretesting): distribute questionnaire to the same
population twice; best method, though inconvenient
ii. Split-half technique: divide questionnaire; give both sections to the
same or similar groups; similar scores support reliability
iii. Multiple forms: use variations of the same questionnaire; similar
scores prove reliability
C. Rule Three
1. Wording in questionnaire must be appropriate for the target audience.
2. Confusion and misunderstanding can lead to tainted data
3. Keep in mind audience reading level
D. Rule Four
1. Who should answer the question must be clearly identified
2. Clarity in directions is important
E. Rule Five
1. Avoid asking questions that are biased, leading, or double-barreled in nature
2. Findings are not valid with:
a. Questions that are designed to elicit a specific response; these convey the
researcher’s bias
b. Double-barreled questions that actually contain more than one response;
results in ambiguous responses
c. Preceding questions can influence the responses to later questions
3. Questionnaire format and wording can influence responses
2
F.
G.
H.
I.
J.
4. Be skeptical of findings from very ideologically polarized groups
Rule Six
1. Before the construction of a questionnaire, a decision must be made whether
to use open- or closed-ended questions or a combination of both
2. Open-ended questions: complicate data analysis; in-depth responses
3. Closed-ended questions: provides sufficient, easier to analyze
4. Combination allows collection of the most pertinent information; mixed
methods becoming more popular
Rule Seven
1. Keep in mind that the respondents may not have all the information needed to
complete the questionnaire
2. Making assumptions results in few responses or responses based on
assumptions
3. Provide an “escape” response such as “unknown” or “no prior knowledge”
Rule Eight
1. Whenever possible, pretest the questionnaire before it is officially used.
2. Allows you to find errors in construction, language, or other flaws that could
render the data useless if not corrected
3. Can be used to assess reliability
4. A few individuals from the target population should fill out questionnaire and
provide feedback
Rule Nine
1. Set up questions so that the responses are easily recognizable whether the
questionnaire is self-administered or interview.
2. Provide adequate, clear directions and establish recognizable means for
responding
3. Avoid a busy format and small print
Rule Ten
1. Questionnaire should be organized in a concise manner that will keep the
interest of the respondent, encouraging him or her to complete the entire
questionnaire
2. Ask interesting questions and limit survey to two pages
III. Scales
A. Construction of Scales
1. Scale can be
a. Measurement device for responding to a question or statement
b. Compilation of statements or questions used to represent the concept
studied
2. Scales as compilations are relevant to research because
3
B.
C.
D.
E.
a. Allow collapsing of several variables into a single variable that produces a
representative value
b. Offer quantifiable measures that are open to precision and statistical
manipulation
c. Can increase the measurement’s reliability
3. Principle of Unidimensionality
Scaling Procedures
1. Scaling connects phenomena with numbers for analytic purposes
2. Scale should explain a phenomenon
3. Arbitrary scales and attitudinal scales
Arbitrary Scales
1. Designed to measure what the researcher believes it is designed to measure
2. Based on face validity and researcher’s professional judgment
3. Starting point for exploratory research; lacks substantive support
Attitudinal Scales
1. Thurstone Scales
a. Use of other individuals (“judges”) to indicate what items best fit the
concept
i. Paired comparisons
ii. Equal appearing intervals
iii. Problems: time consuming and finding qualified judges
2. Likert Scales
a. Bipolar, five-point response range
b. Questions where all respondents provide similar responses are usually
eliminated during data analysis
Advantages and Disadvantages of Scales
1. Advantages
a. Clearer or more precise measure of the concept
b. Can be replicated and used as longitudinal methods
c. Require more thought
2. Disadvantages
a. Concern as to whether true attitudes can be measured
b. Question of validity and reliability
IV. Summary
A. Questionnaires
1. Used to collect data
2. Using a previously tested questionnaire is preferable to designing a new one
B. Suggested Rules for Constructing Questionnaires
1. Start with a list of all items you are interested in knowing about the group,
concept, or phenomenon
4
2. Be prepared to establish validity and reliability
3. The wording in the questionnaire must be appropriate for the targeted
audience
4. Be sure that who should answer the questions is clearly identified
5. Avoid asking questions that are biased, leading, or double-barreled in nature
6. Prior to construction, a decision must be made whether to use open- or closedended questions, or a combination
7. Consider that the respondents may not have the general information needed to
complete the questions
8. Whenever possible, pretest the questionnaire before it is officially used
9. Set up questions so that the responses are easily recognizable whether the
questionnaire is self-administered or a interview
10. The questionnaire should be organized in a manner that will keep the interest
of the respondent
C. Other issues
1. Reliability
2. Validity
3. Measurement level
4. Scales
a. Arbitrary
b. Attitudinal
i. Thurstone
ii. Likert
Chapter 10 - Data Collection
I. Data Collection
A. Data Collection
1. One of the most crucial aspects of the research effort
2. Must choose the right data collection technique based on the selected
sampling strategy and research design
3. Experimental design: best method; not for social science research
B. Data Collection Techniques
1. Survey
2. Interview
3. Observation
4. Secondary data methods
II. Survey Research
A. Survey Research
1. Most frequently used method
5
2. Excellent for gathering primary data and is useful for descriptive and
correlational
3. In criminal justice and criminology, for measuring attitudes, fears,
perceptions, and victimizations
4. Data collection device: questionnaire
5. Useful because it can be self-administered
B. Mail distribution of surveys
1. Most commonly used method for distributing a self-administered
questionnaire
2. Allows for larger samples, broader area coverage, and minimized cost
compared to others
3. Advantages: no field staff, eliminate bias, allow privacy, fewer time
constraints, and high percentage of returns
4. Disadvantages: lack of responses, possible differences between respondents
and nonrespondents, lack of uniformity, misinterpretation, and slow return
rates
5. Response rates
a. Divide the number of responses by the total number of possible responses
b. Reminders increase possibility of subject participation
5. Differences between respondents and nonrespondents
a. If the nonresponse has to do with the survey itself, there is bias
6. Ways to increase response rate
a. Use an attractive, shortened format
b. Offer remuneration or reward for completing the survey, like a gift
certificate
c. Appealing to the altruistic side
d. Showing legitimate sponsorship of the survey, such as by a university
e. Personalizing the survey
f. Sending the survey at the right time
7. Acceptable Mail Survey Response Rates
a. Oversample the population to meet sample-size requirements
b. Suggested expectations for a response rate: 40% within 2 weeks, 20%
within 2 weeks of follow-up letter, 10% within 2 weeks of final contact
c. 50% is adequate, 60% is good, and 70% is very good
C. Surveys and the Internet
1. Technology provides other means of delivery
2. Researchers can reach targeted populations throughout the world
3. Provide quick access and convenience for respondents
6
4. Same advantages as mail surveys, including large samples, wide geographic
areas, cost efficiency, and ability to address a variety of topics, as well as easy
data processing.
5. Same disadvantages as mail surveys, including low response rate,
nonresponse, and misinterpretation of the questions. Response rate could be
even lower because of spam mail.
D. Mail and Internet surveys are not always accessible to all populations, and a paper
questionnaire might be required
III. Interviews
A. Interviewing
1. Interaction between two individuals where one of the individual’s goals is to
obtain recognizable responses to specific questions
2. Three types: structured, semi-structured, unstructured
3. Usually completed face to face, or via telephone
B. Face-to-face interviews
1. Contact between researcher and respondent can be positive reinforcement,
leading to higher response rates
2. Easy to clarify misunderstandings
3. Other advantages: audiovisual aids, nonverbal cues, follow-up interviews,
easy language, discretion
4. Disadvantages: time consuming, costly, interviewer may impact response,
interviewer’s individual skill
C. Structured Interviews
1. Most common type used in criminal justice research
2. Closed-ended questions that are always asked in the same order, associated
with quantitative research
a. Advantages: easily administered, high response rates, and easier data
processing
b. Disadvantages: does not allow for further exploration of the responses
and limits types of responses
2. Standardized list of open-ended questions that allow for detailed responses
a. Disadvantages: interviewer cannot ask follow-up questions, difficult to
analyze, smaller sample size
b. Advantages: more detailed information
D. Semi-structured interviews
1. Original set of open-ended questions
2. Differs from structured: the interviewer can further explore why the response
was given
3. Advantage: lots of follow up and clarification can happen
E. Unstructured Interviews
7
1. Open-ended questions developed during actual interview
2. Can lead to unexpected but important topics
3. Susceptible to intervening or biasing elements
4. Interviewer must be experienced
F. Telephone Interviews
1. Advantages
a. Eliminate the need for field staff
b. Easier to monitor interviewer bias; eliminates nonverbal cues
c. Unstructured interviews can be done via telephone too, rarely
d. Less expensive and can be quick
2. Disadvantages
a. Limited scope of research
b. Difficulty in obtaining in-depth responses
c. Elimination of anyone without a telephone from sample parameters
d. Possible high refusal rates
3. Increase of cell phones makes it impossible to define the sampling frame
4. Many people can identify and block calls
IV. Other Data Collection
A. Field observation
1. Four approaches
a. Full participant
b. Participant researcher
c. Researcher participant
d. Complete researcher
2. To conduct observational data collection, the researcher makes decisions
including
a. Deciding where the observations are to be done
b. Deciding on the focus of the observations
c. Determining when the observations will be done
3. The structure differs between qualitative and quantitative, and quantitative
will require more preparation, like a checklist
4. The timeframe should be determined to provide the best possible opportunity
of collecting the desired data
5. Advantages: direct collection, legitimized observations
6. Disadvantage: misinterpretation of data
B. Secondary Data
1. Efficient way to conduct research
2. Data are available through many sources like government agencies, research
groups, or other researchers
3. Researcher obtains and analyzes existing data
8
4. Must verify that the data are appropriate for their research
5. Form of “unobtrusive research”
a. Doesn’t require researcher to be directly involved with subjects
b. Doesn’t involve interaction because data are already collected
6. Sources of Secondary Data
a. Use of official statistics or records
b. Governmental agencies collect a considerable amount of information,
which can be released for public use
c. Uniform Crime Reports (UCR) published by the FBI includes data on
arrests, clearances, and law enforcement employees
d. Other federal agencies that give data include the Bureau of Labor
Statistics and Bureau of Justice Statistics, National Longitudinal Survey of
Youth, and National Crime Victimization Survey
e. Data are also available through state and local agencies and other research
groups (the General Social Survey from the University of Chicago)
f. Inter-university Consortium for Political and Social Research
7. Content Analysis
a. Study of social artifacts to gain insights into an event or phenomenon
b. Uses unobtrusive methods but differs because it focuses on a particular
medium (books, magazines, etc.)
c. Public (actuarial, political, judicial, governmental, media) and private
(diaries, letters, autobiographies) records
d. Advantage: many possible sources of information
e. Qualitative (verbal) or quantitative (use a quantifiable instrument)
f. Researchers must properly choose method
VI. Summary
A. Data Collection Forms
1. Surveys
a. Self-administered (mail, Internet) or interviews
b. Face to face, or over the telephone
c. Structured, semi-structured, or unstructured
2. Field observation
3. Analysis of secondary data
4. Content analysis
Chapter 13 – Writing up the Research
I. The Research Paper
A. Writing the Research Paper
1. The final goal of the research process that must be taken as seriously as the
research itself
9
B.
C.
D.
E.
2. Procrastination is a problem at the beginning and end of the project
3. Basic order: title page, abstract, introduction, methodology, results,
conclusions, references, and appendices
The Title Page
1. The title explains in clear and concise terms what the research is about
2. The title draws the reader’s attention
3. Information found on the title page
a. Title
b. The author’s identity and affiliation
c. External organization that funds the research, if applicable
Abstract
1. A summary or synopsis of the information being presented
2. Starts with the research title
3. Presents the paper’s major argument
4. Describes methods that were used
5. May provide key points of analysis results (if applicable)
6. Limited to a paragraph, between 100 and 250 words in length
The Introduction
1. Establishes
a. The research problem
b. Literature that supports its existence
c. The reason to research the problem
2. Presents the topic and the research question to the reader
3. Parts of the introduction
a. Introductory paragraph tells the reader what to expect
b. Main portion or literature review supports the research
c. Summary paragraph indicates what research is being reported and how it
was accomplished
4. Must acknowledge if there is very little support
Methodology
1. Considered to be the mainstay of the research paper
2. Discusses
a. The hypothesis or hypotheses
b. The research design
c. The data-gathering technique
3. This section describes
a. The research population
b. The sampling frame
c. The questionnaire or other methods used to gather the data
4. Where, when, how, and why the data was acquired and analyzed
10
F.
G.
H.
I.
J.
5. May be written up in one complete section, or it may be subdivided with
different sections focused on specific elements
Results
1. Presented in separate section or part of the methodology
2. Describes
a. The sample’s characteristics
b. Statistical techniques used
c. The analysis results
d. Whether the results supported the hypothesis or hypotheses
3. Any tables, graphs, or charts should be placed in this section
Conclusions
1. Usually the last section of the research paper
2. Offers insights about the research and whether it did what was expected
3. Discusses any problems with the research
4. Discusses implications of the research and suggestions for future related
research
References and/or Bibliography
1. Unless the topic has never been researched prior to your project, there are
sources to support the research
2. Each source should be recognized, whether you are quoting, paraphrasing, or
using the source to affirm what is already known
3. Citations not only throughout text but also in a reference list or bibliography
or risk charges of plagiarism
4. Most papers for criminal justice and criminology follow the APA style
a. Two formats, one for paraphrasing and one for direct quotes
b. Include only the sources cited within the text in the reference list
5. Special note: additional formats for referencing interviews, Web sources, and
court cases
Tables and Figures
1. May be placed
a. In the text
b. Separate and noted where they should be inserted
c. At the end of the paper
2. Need to be clear as to their content and readily understandable
Appendices
1. Not a requirement in every paper
2. Often useful to include a copy of the questionnaire in the appendix
3. No limitation to the number of appendices
II. Summary
A. The Research Paper
11
1. An important and possibly required task
2. Formatting the paper as suggested makes the process easier
3. Correct grammar and spelling is important along with using language that the
intended audience can understand
4. The goal is to submit the most efficiently written paper possible
12