An Overview of Survey Techniques,
Sampling Strategies, Modes of
Observation, Interviewing , Focus
Groups
1
Lecture outline: Overview of material covered in chapters
6, 8-12 in text
Modules created for each chapter in text
Ch6: Measuring Crime
Ch. 8:Sampling
Ch. 9: Survey Research
Ch. 10: Qualitative Interviewing
Ch. 11 Field Observation
Ch 12: Agency Records, Content Analysis, and secondary Data
2
Module 1: Surveys( see Chapter6, 9
in text)
1.
Survey data collection methods:A look at UCR,
Self Report , and Victimization Surveys
2.
The Basics of Survey Design
3.
Types of Surveys
3
Topics appropriate to survey research
 Counting crime – asking people about
victimization counters problems of data collected
by police
 Self-reports – dominant method for studying the
etiology of crime
 Frequency/type of crimes committed
 Prevalence (how many people commit crimes)
committed by a broader population
4
Topics appropriate to survey research
 Perceptions and attitudes – to learn how people feel
about crime and CJ policy( e.g. Death Penalty)
 Policy proposals – search for ways to respond to
crime that are supported by the general public( e.g
Downsizing Prisons)
 Targeted victim surveys – used to evaluate policy
innovations & program success( e.g. domestic
violence)
 Other evaluation uses – e.g., measuring community
attitudes, citizen responses, etc.( e.g. criminal thinking
scales)
5
The Basic Structure of a Survey
Instrument
 Introduction: state the general purpose of survey and




emphasize voluntary nature of survey,confidentiality and
protection of human subjects.
Section1: Factual non-threatening questions(
demographics here)
Section 2: Core questions( ordering questions;
contingency questions)
Section 3: Scales( Likert, Gutmann)
Example of Matrix questions--Likert Scale: TCU Criminal
Thinking Scale used by Taxman, et. al. in recent study of
criminal thinking( class handout, Nov. 26th)
6
Types of questions: Open- and
close-ended questions
Open-ended – respondent is asked to provide his or her own
answer
 What were your initial expectations of the program?
________________________________________________
Closed-ended – respondent selects an answer from a list or
scale
 On a scale of 1 to 5, where 1 is low and 5 is high, rate your initial
expectations of the program.
1
2
Low
3
4
5
High
7
Open versus closed?
 Choice between these question types will depend
on the aim of your research, the study design, the
population being study, etc.
 Advantages and disadvantages to both.
 Possible gains in reliability (closed-ended)
balanced with possible losses in breadth of
information and validity.
 One strategy is to combine open- and close-ended
questions.
8
Designing questions…
 Make items clear – avoid ambiguous questions; do not ask
“double-barreled” questions
 Short items are best – respondents like to read and answer
a question quickly
 Avoid negative items – leads to misinterpretation
 Avoid biased items and terms – do not ask questions that
encourage a certain answer
 Closed question response categories must be exhaustive
and mutually exclusive.
9
Survey data collection methods
1. Self-administered mail surveys,
computer based surveys, and
response rates
2. In-person interview: best for complex
questions ( role of interviewer is
key)
3. Telephone interviews; fast, low cost;
CATI- computer assisted telephone
interviewing
4.A comparison of three methods
Focus: Racial Profiling Research( see text
pp 256, 257)
10
Focus: Australia’s National Crime Victimization Survey
Methodology
 The MPHS was conducted as a supplement to the monthly LFS.






Each month one eighth of the dwellings in the LFS sample were rotated out of the
survey.
In 2012-13, all of these dwellings were selected to respond to the MPHS each month
. In these dwellings, after the LFS had been fully completed for each person in scope and
coverage, a person aged 15 years and over was selected at random (based on a computer
algorithm) and asked the various MPHS topic questions in a personal interview.
If the randomly selected person was aged 15–17 years, permission was sought from a
parent or guardian before conducting the interview. If permission was not given, the
parent or guardian was asked the crime questions on behalf of the 15–17 year old.
Questions relating to sexual assault, alcohol or substances contributing to the most
recent physical or face-to-face threatened assault were not asked of proxy respondents.
Only those persons aged 18 years and over were asked questions on sexual assault.
Data was collected using Computer Assisted Interviewing, whereby responses were
recorded directly onto an electronic questionnaire in a notebook computer, usually
during a telephone interview.
11
A Link to the USA National Crime
Victimization Survey
 http://www.bjs.gov/index.cfm?ty=dcdetail&iid=245
12
In Review: Comparing UCR and
NCVS—See Text Chapter
 UCR data are based on reported criminal acts
(offender characteristics)
 NCVS data based on individuals actually victimized
(characteristics of victims)
 Whenever possible, review both data sources and
compare findings
Assessment of NCVS
 Document a massive amount of crime that goes
unreported
 Underestimate crime rate
 Insignificant crimes tend to be forgotten
 Victims of several crimes may also forget about all the
crimes
 Females do not report victimization if her abuser live in
the same household
 Whites and college graduates are more likely to report
being victimized
Assessment of NCVS
 NCVS respondents are interviewed every six months (7
interviews)
 Reported victimization rates usually decease with each
interview (awareness of victimization)
 Overestimation of some crimes
 Respondents might mistakenly interpret some
noncriminal events as crimes
 “Telescoping “ effect
Self-reports data
 Created to complement UCR and NCVS
 Interviews or questionnaires
 Demonstrate the prevalence of offending (the proportion
of respondents who have committed a particular offence)
 Incidence of offending (the average number of offences
per person in the study)
Samples for self-reports
 Target: Adult inmates of jails and prisons:
PREA findings on sexual assault
 Target: Adolescents, usually high school
students
 The most important finding: delinquency is
very common
 Middle-class youth commit as much crime as
working-class youth
 Testing criminological theories: Travis Hirschi’s
Research Study testing Control Theory( compared to
subculural & Strain)
Assessment of self-report studies
 Focus on minor and trivial offenses (truancy,
running away from home, minor drug and alcohol
use)
 Although recent studies (NYS) asked subjects about
rape and robbery
 Respondents might not to tell the truth (reliability
issues)
If respondents lie….
 Self-report data can be checked against police records,
school records, interviews with teachers and parents
 The use of, or threat of , polygraph validation (20%
change their initial responses when threatened with a
“lie detector”)
 Subsequent interviewing of subjects permits probing
regarding the details and context of acts
 Use of “lie scales”
Focus: Self Reported Substance Use Among
Detainees in Australia
 http://www.aic.gov.au/media_library/conferences/eval
uation/mcgregor.pdf
 http://192.190.66.70/documents/7/E/8/%7B7E8D4A8E
-A5AF-4D3B-8821-ED8A1BA489B6%7Drpp93.pdf
21
Focus: Research on Extent of Substance Use by
High school students in USA
 Question: Can we collect reliable estimates from
survey questionnaires distributed to high school
students across the country?
 The Survey: Lets take a close look:
 http://www.casacolumbia.org/addictionresearch/reports/national-survey-american-attitudessubstance-abuse-teens-2012
 http://www.drugabuse.gov/publications/drugfacts/hig
h-school-youth-trends
22
UCR, NCVS, and self-reports
 None of the three is perfect
 For the best estimates of the actual number of crimes,
NCVS data are preferable
 For the best estimates of offender characteristics, selfreports and NCVS are preferable
 UCR are superior for understanding the geographical
distribution of crime
Self-administered questionnaires
 Can be home-delivered
 Researcher delivers questionnaire to home of sample
respondent, explains the study, and then comes back
later
 Mailed (sent and returned) survey is most
common
 Researchers must reduce the trouble it takes to return a
questionnaire
 Goal is High Response Rate
24
25
 Via email, website
 Issues:
» representativeness
» mixed in with, or mistaken for, spam
» requires access to Web
» sampling frame?
What we will Cover:
1.Interviewing techniques and Focus Groups
2. A Comparison of Survey Methods: phone/ computer( survey monkey), mail,
and in-person
3. Field Research: The good, the bad, and the ugly
4. Agency Records, Content Analysis and Secondary Data
26
27
 Typically achieve higher response rates than
mail surveys (80-85% is considered good)
 Demeanor and appearance of interviewer
should be appropriate; interviewer should be
familiar with questionnaire and ask questions
precisely
 When more than one interviewer administers,
efforts must be coordinated and controlled
 Practice interviewing
28
•Two variations:
» General interview guide – less structured, lists
issues, topics, questions you wish to cover; no
standardized order
» Standardized open-ended interview – more
structured, specific questions in specific order;
useful in case studies, retrieves rich detail in
responses
29
 94% of all households now have telephones
 Random-Digit Dialing
» Obviates unlisted number problem
» Often results in business, pay phones, fax lines
 Saves money and time, provides safety to
interviewers, more convenient
 may be interpreted as bogus sales calls; ease of
hang-up
30
 Self-administered questionnaires are generally
cheaper, better for sensitive issues than interview
surveys
 Using mail – local and national surveys are same cost
 In person or phone Interviews – more appropriate
when respondent literacy may be a problem, produce
fewer incompletes, achieve higher completion rates
 Validity low in survey research; reliability high
 Surveys are also inflexible, superficial in coverage
31
 Convince subjects you will guarantee
confidentiality and anonymity
 Minimize possible social undesirability
you are asking respondents to admit
 Phrase questions in non-judgmental
manner
 Bear in mind “fading memory” when
setting time frame
32
 12-15 people brought together to engage in guided
group discussion of some topic( e.g. addicts& recovery)
 Members are selected to represent a target population,
but cannot make statistical estimates about population
 Most useful when precise generalization to larger group
is not necessary
 May be used to guide interpretation of questionnaires
following survey administration
 Examples: Drug addicts and reentry, homeless sex
offenders, teenagers and sexting
Module 3: Field Research
33
•Field research encompasses two different methods of
obtaining data:
•Direct observation( example-bar room behavior;
seat belt use, community disorder indices)
•Asking questions; modes of recording
observations
•May yield qualitative and quantitative data
•Often no precisely defined hypotheses to be tested
•Gives comprehensive perspective – enhances validity
•Go directly to phenomenon, observe it as completely as possible
•Especially appropriate for topics best understood in their
natural setting
•How street-level drug dealers distinguish customers
•Studies of ‘vice’, e.g., prostitution and drug-use.
•Aspects of physical settings, Disney World, social control
•Complete participant – participates fully; true identity and
purpose are not known to subjects
•E.g., posing as a bar patron; becoming a police officer; or corrections worker
•Participant-as-observer – make known your position as
researcher and participate with the group
•E.g., study of active drug users; Julie Mueler and the Guardian Angels
•Observer-as-participant – make known your position as a
researcher; do not actually participate
•E.g., Observational study of police patrol—”ride-along” ; research on gangs
•Complete observer – observes without becoming a participant
•E.g., court observation, Chicago neighbourhood study( windshield studies)
Observer status
 Be aware of, and document role of researcher
(extent of participation)
 Be aware that all observation is subjective.
 Be aware of the possible “effect of participation”.
 Be aware of often competing ethical and ‘scientific’
values related to all observational studies.
37
•Field research is often a matter of going where
the action is and simply watching and listening
•Also a matter of asking questions & recording
answers
•Field research interviews are much less
structured than survey interviews
•Ideally set up and conducted just like a normal,
casual conversation
•Access to formal organizations
•Find a sponsor, write a letter to executive
director, arrange a phone call, arrange a meeting
•Access to subcultures
•Find an informant (e.g., person who works with
offenders), use that person as your “in”
•Snowball sampling is useful as informant
identifies others, who identify others, etc.
An Overview of Sampling
Techniques: See Ch.8 in text
 Sampling Distributions
 The Normal Curve
 Sampling Error
 Probability Sampling
 Non-Probability Sampling
 Illustrations: NCVS, British Crime Survey, Australian
Crime Survey
41
Key Terms to Remember: Sampling
 Simple random sampling
 Table of random numbers
 Confidence intervals
 Stratified sampling
 Population
 Probability vs. non-probability sampling
 Equal probability of selection method( EPSEM)
 Focus: random sampling from probation case files
42
•Note taking, tape recording when interviewing and
when making observations
•Videotaping or photographs can make records of
“before” and “after” some physical design change
•Field notes – observations are recorded as written notes,
often in a field journal; first take sketchy notes and then
rewrite your notes in detail
•Structured observations – observers mark closed-ended
forms, which produce numeric measures
•Useful to combine field research with surveys or
data from official records
•Baltimore study of the effects of neighborhood
physical characteristics on residents’ perceptions of
crime problems (Taylor, Shumaker, & Gottfredson,
1985)
•Perceptions – surveys;
•Physical problems – (1) observations, (2) actual population
and crime information - census data & crime reports from
police records
•Provides great depth of understanding
•Flexibility (no need to prepare much in advance)
•More appropriate to measure behavior than surveys
•High validity; quant. measures – incomplete picture
•Low reliability – often very personal
•Generalizability – personal nature may produce findings
that may not be replicated by another
•Precise probability samples can’t normally be drawn
Module 7: Secondary Data, Content Analysis
46
•Data from agency records – agencies collect
a vast amount of crime and CJ data
•Secondary analysis – analyzing data
previously collected
•Content analysis – researchers examine a
class of social artifacts (typically written
documents) See Text example; p 329 Terrorist
Recruitment
•Most commonly used in descriptive or
exploratory studies
•Content analysis often center on links between
communication, perceptions of crime problems,
individual behavior, CJ policy
•Published Statistics – gov’t organizations routinely
collect and publish compilations of data (e.g., ABS,
BOCSAR, AIC) often available in libraries and online
•Nonpublic Agency Records – agencies produce
data not routinely released (e.g., police departments,
courthouses, correctional facilities)
•New Data Collected by Agency Staff – collected
for specific research purposes; less costly + more
control
Criminal Activity
•Incidents
•Crimes violated
•Victims
•Offenders
Court Activity
•Defendants
•Filings
•Charges and Counts
•Cases
•Appearances
•Dispositions
•Sentences
Apprehension
•Arrests
•Offenders
•Charges
•Counts
Corrections
•Offenders
•Admissions
•Returns
•Discharges
•Virtually all CJ record keeping is a social process –
“social production of data”
•Records reflect decisions made by CJ personnel
as well as actual behavior by juveniles and adults
•Discretion factors in to recordkeeping
•CJ organizations are more interested in keeping
track of individual cases than in examining patterns
•Potential for clerical errors due to volume of data
•Systematic study of messages – can be applied to
virtually any form of communication
•Decide on operational definitions of key
variables
•Decide what to watch, read, listen to & time
frame
•Analyze collected data
•Well suited to answer “who says what, to whom,
why, how, and with what effect?”
•Reminders:
•Remember operational definition of
variables, and their mutually exclusive &
exhaustive attributes
•Pretest coding scheme
•Assess coding reliability via intercoder
reliability method and test-retest method
•Sources – websites, libraries
•AIC, http://www.aic.gov.au/
•BOCSAR, http://www.bocsar.nsw.gov.au/
•BJS, NCVS, ICPSR, NACJD
•Advantages – cheaper, faster, benefit from work of skilled
researchers
•Disadvantages – data may not be appropriate to your
research question; least useful for evaluation studies
(which are designed to answer specific questions about
specific programs)