Introduction to Research
Methods
Research Methods for Public
Administrators
Dr. Gail Johnson
Dr. G. Johnson,
www.researchdemystified.org
1
Numbers and Facts
Any policy debate is awash in numbers that
are presented as absolute fact.
But are they?
Or are we being tricked into “premature
certainty” because numbers appear
concrete?
Dr. G. Johnson,
www.researchdemystified.org
2
In the News
 According to a 2009 study done for the American
Health Insurance Plans by Price Waterhouse and
Coopers:
 The average family health insurance coverage
costs approximately $12,300 today
 Key finding: The overall impact of the proposals
for health care reform will be to increase the cost
of private insurance coverage for individuals,
families, and businesses above what these costs
would be in the absence of reform.
Dr. G. Johnson,
www.researchdemystified.org
3
Their Numbers
 Health Insurance premiums could be
expected to increase to approximately:
 $15,500
in 2013 under current law and to
$17,200 if these provisions are implemented.
 $18,400 in 2016 under current law and to
$21,300 if these provisions are implemented.
 $21,900 in 2019 under current law and to
$25,900 if these provisions are implemented.
Dr. G. Johnson,
www.researchdemystified.org
4
Policy Debate
 What does this data mean in terms of
Congressional proposals to require everyone
purchase health insurance?
 What would you conclude based on this research
result?
 What do you want Congress to do based on these
numbers?
 What do you want Congress to do based on your
beliefs?
Dr. G. Johnson,
www.researchdemystified.org
5
Paradox
 Research: the search for rationality
But:
“We must be aware that our knowledge and
experiences are finite, and always
imperfect.”
Hitoshi Kume
Dr. G. Johnson,
www.researchdemystified.org
6
Assessing Credibility
Requires
 Basic knowledge of research methods



Maintaining a research perspective
Knowing how data was collected is necessary to help
assess whether the results are believable
Critical thinking skills


A hint of detachment and skepticism
Being aware of our beliefs yet setting them aside to
see things clearly
Dr. G. Johnson,
www.researchdemystified.org
7
The Tough Questions
 Is it true?
 How do you know?
 What is the logic of the argument?
 What is the evidence?
Remember:
Much of what is presented as fact melts away
under scrutiny.
Beware of premature certainty.
Dr. G. Johnson,
www.researchdemystified.org
8
Critical Thinking
 Ask Yourself:
 “How do I know that the things I believe are true?”
 “What would it take to convince me otherwise?”
 The greatest challenge comes when looking at
research that reflects what we already believe
 Critical thinking challenges our most cherished
beliefs
 To see clearly, we must be aware of our beliefs
while also remaining detached from them
Dr. G. Johnson,
www.researchdemystified.org
9
The Role of Research in the
Public Sector
 Guide policy
 Target programs
 Find out what works and what doesn’t
 Convince funders
 Provide feedback from customers
 Help plan interventions
 Advocate for change
Dr. G. Johnson,
www.researchdemystified.org
10
Research in the Public Sector
 The Intersection of Data, Values and Beliefs
and Politics
 Does
welfare help or hurt people?
 Should oil companies be prevented from
earning too much profit?
 Is investment in primary prevention more
effective in reducing crime than spending on
law enforcement?
Dr. G. Johnson,
www.researchdemystified.org
11
Different Research Approaches
 Qualitative:
 Stories
 Observations
 No
numbers
 Quantitative:

What you can count.
Dr. G. Johnson,
www.researchdemystified.org
12
Qualitative Data: Stories
 Anecdotes
 Example:
Michael Moore’s “Sicko”
Example: Barbara Erhen “Nickled and
Dimed”
 Paints a picture, makes it feel real
 Limited in scope:
 Is
this a small or large problem?
Dr. G. Johnson,
www.researchdemystified.org
13
Quantitative Data: Numbers
 Percents, Counts, Correlations
 Example:
Study: U.S. Ranking for Preventable
Deaths:
 U.S. lowest of industrialized countries
 But need to check how they defined and
measured preventable deaths
Dr. G. Johnson,
www.researchdemystified.org
14
Quantitative Data: Numbers
 Be aware that some things are not really
knowable—so how did they come up with the
numbers?

Number of illegal immigrants
 Amount of money spent on illegal drugs
 10-year projection of annual U.S. budget deficits (even
1-year is not precise)
 The true cost of mandatory health insurance
 Percent of carbon from raising animals
Dr. G. Johnson,
www.researchdemystified.org
15
As An Advocate
Which would be more effective?
Voice your concerns about a problem?
Or
Show data documenting the problem?
Dr. G. Johnson,
www.researchdemystified.org
16
As a Citizen
What do you base decisions on?
 Data?
 Emotional
appeals?
 Personal opinion?
 Political demands?
Do some work better than others in different
situations?
Dr. G. Johnson,
www.researchdemystified.org
17
As a Decision Maker
What do you base decisions on?
 Data?
 Emotional
appeals?
 Personal opinion?
 Political demands?
Do some work better than others in different
situations?
Dr. G. Johnson,
www.researchdemystified.org
18
The Tough Question
 Generally, we prefer data, facts, hard
evidence
 But the question is:
 How
good is the research?
 Meaning: does the research yield accurate,
reliable, valid, and unbiased information?
Dr. G. Johnson,
www.researchdemystified.org
19
What is Research?
 A systematic search for answers to
questions.
 Search: to uncover, examine, find by
exploration, to investigate, to inquire.
 Research: "the systematic inquiry into a
subject in order to discover or revise facts,
theories
Dr. G. Johnson,
www.researchdemystified.org
20
What is Research?
Research: Root of work means “to know.”
Re---Search:
conveys the idea of searching again, from a
different perspective, using different
approaches, looking at different data.
Dr. G. Johnson,
www.researchdemystified.org
21
What is Research?
 Empirical: derived from experience or
experiment; observation and experience.
 Science: "a branch of knowledge or study
dealing with a body of facts or truths
systematically arranged and showing the
operation of general laws; systematic
knowledge"
Dr. G. Johnson,
www.researchdemystified.org
22
This Thing Called Science
 Scientific Method

Curiosity
 Systematic observation
 Systematic experimentation
 Theories and Hypotheses:
 Used
to Explain Relationships
Dr. G. Johnson,
www.researchdemystified.org
23
Different Goals
Scientists:
 More
interested in creating knowledge
 Not concerned about utility or application of
the results
 Practitioners:
 More
interested in application and problem
solving
 If the research isn’t useful, then so what?
Dr. G. Johnson,
www.researchdemystified.org
24
Research Process:
Simplified 3-step
 Plan
 Do
 Report
Dr. G. Johnson,
www.researchdemystified.org
25
Research Planning Process
 Determine the issues
 Decide on the research question(s)
 Select measures
 Identify the ideal design given the type of
research question
 Develop data collection methods and
instruments
Dr. G. Johnson,
www.researchdemystified.org
26
Research Planning Process
 Decide on sampling frame and strategy
 Develop analysis plan
 Articulate your entire research plan
 Review research plan and test all data
collection instruments
 Prepare work plan with resource and time
requirements
Dr. G. Johnson,
www.researchdemystified.org
27
Planning Matters
 When people think about research, they
tend to immediately focus on statistics
 But no amount of statistical wizardry
can save mistakes made in the planning
process
 Planning is harder and takes longer
than most people expect
Dr. G. Johnson,
www.researchdemystified.org
28
Doing Research
 Gathering the data
 Preparing data for analysis
 Analyzing and interpreting the data
Dr. G. Johnson,
www.researchdemystified.org
29
Reporting
 Executive Summary
 Reports
 Charts and Tables
 Oral Briefings
Dr. G. Johnson,
www.researchdemystified.org
30
Research Planning Process
 Presented as linear but in practice, it is not
 The word is iterative: meaning researchers make
some initial assumptions about how to proceed
and then go back and make changes to the plan as
new information shows that their initial plans will
not work
 There is a lot of back and forth before it all clicks
into place (and researchers live with the
limitations of the situation)
Dr. G. Johnson,
www.researchdemystified.org
31
Ethical Research
 Do No Harm
 Protect participants from being harmed by the research
 Guard the confidentiality of participants
 Do not coerce people to participants: they must freely
consent
 No one should be denied benefits to which they are
entitled because they refuse to participate in research
project
 Do not quickly conclude a program does not work just
because there does not appear to be an impact
Dr. G. Johnson,
www.researchdemystified.org
32
Objective: Strive to See Clearly
 Objectivity is a worthy goal even if not 100%
attainable
 Think in terms of being free of ideological or
political blinders or desired outcomes
 Be aware of biases
 Researchers should state biases and build in
quality controls to minimize bias
 Quality Control: Have others with different
perspectives review the research design,
analysis and results.
Dr. G. Johnson,
www.researchdemystified.org
33
Principles of Good Research
 Honest
 Do
not lie
 Do not distort or “spin” the data
 Fully disclose methods, definitions,
assumptions, biases
 Fully disclose limitations of the research and
implications for making any conclusions
Dr. G. Johnson,
www.researchdemystified.org
34
Principles of Good Research
 Accurate: Strive to be error free
 Build
quality control procedures into data
collection, analysis, interpretation, and written
product
 Verify data entry and analysis.
 Have someone check your numbers.
Dr. G. Johnson,
www.researchdemystified.org
35
Principles of Good Research
 Technically Correct
 Use
appropriate designs, data collection
methods, analysis, statistics and charts
 The right data collection instruments.
 The right statistics.
Dr. G. Johnson,
www.researchdemystified.org
36
Principles of Good Research
 Reliable and Valid Measures
 Use
reliable and valid measures
 Measure what matters using systematic
approaches with fixed measurement rules
 Ask: do they measures actually measure what
they claim to measure?

Are they measuring reported crime but claiming to
be measuring all crime?
Dr. G. Johnson,
www.researchdemystified.org
37
Research Flaws
 Perfection is not the standard to assess the
credibility of research results.
 All
research is flawed, so the trick is to
distinguish minor flaws from major ones.
 Skill: to assess the relative strengths and
limitations of research to determine the
credibility of the results.
Dr. G. Johnson,
www.researchdemystified.org
38
Research Flaws
 It means working in the gray area.
 If the research is the best given the
circumstances, flawed though it may be, it
should be considered.
 However, decisions “to do something”
based on that data should be made with
caution, and implemented incrementally
with built-in feedback.
Dr. G. Johnson,
www.researchdemystified.org
39
Seriously Flawed Research
 While you might not want to make
decisions based on seriously flawed
research, it still may actually be correct,
provide useful insights, or some guidance
about how to do the next study.
 It
might provoke important conversations.
 Remember: perfection is not a useful
standard.
Dr. G. Johnson,
www.researchdemystified.org
40
Takeaway Lesson
Sophisticated Users of Research Question
Numbers!
“Many a statistic is false on its face. It gets
by only because the magic of numbers
brings out a suspension of common sense."
---How to Lie With Statistics, Huff, p. 138.
Dr. G. Johnson,
www.researchdemystified.org
41
Ask Tough Questions
 Does the researcher have axe to grind?
 Particular
political ideology? Desire to
get more funding?
 Who was included and excluded in the
study?
 Is the data really knowable?
 Number of actual crimes is not the same
as the number of reported crimes.
Dr. G. Johnson,
www.researchdemystified.org
42
Ask Tough Questions
 Is reported behavior based on self-reports?
 How
many people used illegal drugs in
the past week?
 Does the research make a giant leap to
conclusion?
 Did a study with all male participants but
generalizes to everyone?
Dr. G. Johnson,
www.researchdemystified.org
43
Creative Commons
 This powerpoint is meant to be used
and shared with attribution
 Please provide feedback
 If you make changes, please share
freely and send me a copy of changes:
 Johnsong62@gmail.com
 Visit www.creativecommons.org for
more information
Dr. G. Johnson,
www.researchdemystified.org
44