Community-Based
Evaluation Methods &
Data Analysis
Dr JABBARIFAR 20007
Different Types of Data

Opinions, priorities

Aspirations, motivations

Level of awareness,
knowledge, attitudes or
beliefs


Behaviors, practices


Assets, skills

Networks, associations



Needs, fears, problems,
concerns
Demographic
characteristics

Services or resources
provided
Resident utilization of
services or resources
provided
Numbers or rates of disease,
illness, disability, injuries
Sales transactions,
purchases

Policies

Pictures, other visuals

Maps
Quantitative vs. Qualitative
Data
Quantitative:






Collected in the form of
numbers or percentages
Closed-ended questions
Answers who?, what?,
when? and where?
Can demonstrate cause and
effect
Can “represent” a
population
Cannot collect new ideas or
responses, only those
considered ahead of time
Qualitative:






Collected in the form of
words, concepts, themes,
or categories
Open-ended questions
Answers how?, why?
Can provide richer, more
in-depth data
Can provide data in a
respondent’s own words
Can explore new ideas in a
dynamic and unstructured
way
Common Community-Based
Evaluation Methods







Secondary data
Activity tracking
Client tracking
Mapping/ Photovoice
Surveys
Focus groups
Key informant interviews
Which of these methods collect
quantitative data?
Which collect qualitative data?
Secondary Data
•
•
•
•
•
•
•
•
•
County health department
Vital statistics  Birth & death certificates
City or county government
Schools (including school nurse records)
Hospital or clinic records
Local health promotion, health ed or advocacy
agencies
Private foundations which fund community programs
Universities or local community colleges
Local surveys or research efforts going on in your
community??
Data you already collect!
Activity Tracking




Tracking and documenting program activity
implementation
Tracking and documenting number of
people reached
Process evaluation method, but important
for outcome evaluation
May include: weekly/monthly program
implementation log, sign-in sheets at
classes or health fairs, counts of materials/
brochures distributed, etc.
Client Tracking
•
Tracking and documenting encounters with
clients; describing the encounter and client
characteristics
•
More than just counting
•
•
Can be used for process and outcome
evaluation
May include: outreach encounter form, home
visit form, clinic intake form, etc.
Mapping/ Photovoice
•
Data can also include maps and photos  don’t just
have to be numbers or words
•
Maps and photos can provide visual imagery to give
meaning to quantitative data
•
They can show graphically what cannot be described
with numbers or words, such as the conditions of a
neighborhood or program facilities, or a community’s
proximity to park space or health services.
•
Make sure your maps and photos are clear and illustrate
the points you would like to make
Surveys

Surveys are composed of carefully crafted
questions on a topic or issue for the purpose
of revealing information about community
residents or services they utilize.

The most commonly used survey methods
are:
1. Mailed Surveys
2. Hand-out Surveys
3. Face to Face Surveys
4. Telephone Interview Surveys
Planning A Survey
•
•
•
•
•
•
•
Determine your purpose
Determine if the survey data collection is
appropriate
Select your target audience
Select a survey method
Design survey questionnaire
Pilot test survey
Publicize and distribute surveys
Determine the Data You Need
•
Draft your research questions – what you
want to learn from these interviews.
•
Identify the data you will need to collect in
order to answer these questions.
•
The type of data you are looking for will
help you identify the best persons in the
community to survey, and the best survey
questions to ask them.
Target Population
The following are two steps in determining
your survey target population.
1) Define your target population --identify the
group of individuals from your community
from who you want to collect information
(single mothers, high school students, the
uninsured, etc.)
2) Identify the geographic area of your target
population.--Where is your group of interest
located? What are the geographic
boundaries of your target population?
Select A Survey Method
Self Administered: These surveys are filled out by
respondents themselves without the assistance of
trained interviewers.
A. Mail Surveys
B. Drop-off Surveys
Administered by Interviewer: These surveys are
filled out with the assistance of trained
interviewers.
A. Face to Face Surveys
B. Telephone Surveys
Design Survey Questionnaire
•
Your survey questions should be in direct relationship with
what you want to learn from your target population.
•
Asking unrelated questions will interrupt the flow of the
survey and tire out your respondents.
•
Be brief and strategic when designing questions!
•
•
Open-ended questions are great for answering “why”
questions, but only include one or two.
Motivating people to fill out your survey will help you
collect the data you need – this can include a benefits
statement or incentives such as entry into a lottery or
drawing.
Pilot Test Survey
•
It is always a good idea to pilot test the
survey with a small group of people from
your target population.
•
The purpose for piloting the survey is to get
feedback on the survey question and
structure before implementing it.
This will ensure you identify potentially
confusing, offensive, leading, or loaded
questions from your survey.
•
Publicize and Distribute
Surveys



Advertise your survey a couple of weeks before
distribution in the community.
Publicizing what you are trying to do and how
the information collected will benefit the
community may help increase your response
rate.
Create a survey distribution timeline and stick
to it - postponing survey distribution date may
limit the time you have set aside for data
analysis.
Advantages and Disadvantages
of Surveys
Advantages





Data can be collected from
a lot of respondents easier
than any other method
Can get a large enough
sample that can be
representative of the larger
population
Findings can be generalized
to the larger population
Can cover a lot of topics
Can easily compare
different groups’ data to
each other
Disadvantages



Survey instrument must be
carefully constructed to
avoid leading questions,
and to make sure the
appropriate responses are
available
Response rates can be low
for self-administered
surveys, especially mailed
ones
Response will be low if
survey is too long
Focus Groups
Focus group are a qualitative research
method designed to learn more about
how people think, feel, or make
decisions (attitudes, perceptions,
opinions, experiences) through
focused discussions.
Focus Groups (cont.)


A series of discussions involving 8-12 people,
selected to share their perceptions of a defined
topic
Stimulates participants to share their opinions
openly in a group discussion

Captures rich data in participants’ own words

Flexible to capture new ideas and issues


Can be difficult to summarize and interpret results
across groups
Need to be creative when recruiting busy people
Planning a Focus Group
•
Determine the data you need
•
Determine and select your target audience
•
Develop a focus group discussion guide
•
Schedule focus groups
•
Recruit focus group participants
•
Recruit moderator and note-taker
•
Plan and conduct focus groups
•
Compile and organize focus group data
Determine the Data You Need
•
Draft your research questions – what you
want to learn from these focus groups.
•
Identify the data you will need to collect in
order to answer these questions.
•
The type of data you are looking for will
help you identify the best persons to recruit
for your focus groups, as well as the best
questions to develop for your focus group
discussion.
Target Audience
•
•
•
•
•
To whom do you want to talk?
What types of experiences and perspectives
you are hoping to capture in your
discussion?
Design and use a screening tool to recruit
participants: ask questions that qualify
people to participate.
It is important that you include participants
who will feel comfortable talking to each
other.
Participants should not know each other.
How Many Focus Groups?
•
Conduct 2-3 focus groups for each subpopulation of interest (i.e. gender, age,
race/ethnicity, language, etc.)
•
2-3 ensures that the perspectives and
experience you capture are more
representative of the larger population and
not just by “chance”.
•
Length: 1.5 hrs.-2 hrs.
•
Focus Group Size: 6-12 participants
Focus Group Discussion Guide

Discussion guide: An outlined script to help
guide the moderator and focus the discussion

Organization:
Introduction
Purpose
Ground rules
Focus group questions
Closing statements
Discussion Guide Questions
•
•
•
•
•
Introductory questions: getting people to think
about the general topic.
Transition questions: questions that get
participants to logically transition between the
introduction topics and the ideas contained in the key
questions.
Key questions: 2-5 questions important to getting
the information you have set out to collect.
Ending questions: questions that provide closure
and reflection.
Summary question: note-taker summarizes the
major themes heard throughout the discussion in a
short oral summary (2-3 minutes), then will ask the
participants if the summary covered all the major
points.
Scheduling Focus Groups
•
Location: The focus groups should take
place in a convenient and accessible
location. It is also important to select a
neutral environment.
•
Time of day: most convenient for your
target audience. Set time earlier than
intend to start.
•
Incentives: Food or refreshments, money
or gifts, drawings, child care, transportation,
anything that is meaningful to those
community members.
Conducting & Moderating

The discussion guide will help guide the moderator
with the questions you want answered.

Moderator Responsibilities: setting up ground
rules, keeping the discussion focused and on track,
drawing out comments from quiet participants, and
asking for clarification or additional information from
participants when needed.

Note-taker Responsibilities: takes comprehensive
notes, operates audio equipment (tape-recorder),
keeps track of time, handles environmental
conditions and logistics, unexpected interruptions,
and incentives.
Skills of the Moderator
Extremely important to select moderators based on
skills and experience:
•
•
•
•
•
•
•
•
•
•
Comfortable/familiar with leading meetings or groups
Able to exercise mild control over the group
Able to guide conversation back on target
Maintain group enthusiasm and interest
Curious about the topic
Respects participants and their comments
Creates and maintains a comfortable environment
Adequate background knowledge
Good oral and written skills
Remains neutral
Advantages and Disadvantages
of Focus Groups
Advantages





Flexible
Captures rich, in-depth
data
Immediate results
Encourages and
stimulates individuals to
share more openly
Data can be combined
with quantitative data to
provide a complete
picture about an issue
Disadvantages




May be challenging to
recruit participants
Need to schedule at least
2-3 focus groups to
capture diversity
Difficult to generalize
results to the larger
population because of
small numbers of
participants
Difficult to compare
results across groups
Key Informant Interviews
•
Key informant interviews are qualitative, in-depth
interviews with key informants
•
Key informants are individuals who know what is
going on in the community (community leaders,
professionals, residents, advocates, elected
officials, doctors, etc.)
•
The following are two common techniques used to
conduct key informant interviews:
1.
2.
Telephone interviews
Face-to face interviews
Planning Key Informant
Interviews
•
•
•
•
•
•
•
•
•
Determine the data you need
Determine target population and brainstorm
possible key informants
Select key informants
Select interview technique
Develop an interview guide
Determine documentation method
Select designated interviewer(s)
Conduct key informant interviews
Compile and organize key informant interview
data
Determine the Data You Need
•
Draft your research questions – what you
want to learn from these interviews.
•
Identify the information you will need to
collect to answer these questions.
•
The type of data you are looking for will
help you identify the best persons in the
community to interview, and the best
questions to ask them.
Target Audience and Key
Informants
• Determine your target population (program
clientele, racial/ethnic minority groups, adolescents,
women, the elderly, HIV+ persons, etc.)
• Brainstorm and create a list of possible key
informants who are knowledgeable and closely
involved with your target population.
• Select key informants:
• Who have first-hand knowledge about the target
population, the program, and the issues you are trying to
investigate.
• Who represent different experiences, backgrounds, and
viewpoints.
Select Interview Technique
Telephone interviews (15-25 min):
•
•
May be the most convenient and least time intensive
way to interview busy key informants.
Disadvantage: not having the personalized interaction
otherwise possible through a face-to-face interview.
Face-to-face interviews (20-30 min):
•
•
Most frequently used format. But time intensive.
Advantages: provides a free-exchange of ideas and
lends itself to ask more complex questions.
Develop Interview Guide
Introduction- Introduce yourself and your purpose
and/or benefits statement.
Key questions- Draft 5-10 questions important to
getting the information you have set out to collect.
Probing questions – Probing questions encourage
participants to reflect more deeply on the meaning
of their comments.
Closing question—Provide an opportunity for the
key informant to give any additional information or
comments.
Summary—Quickly summarize the major comments
made.
Determine Documentation
Method


Note-taking: manually write the key
informant’s comments while conducting the
interview, on a copy of the interview guide.
Tape-recording: use a tape recorder to
document what key informants say. This
approach may allow the interviewer to freely
engage in the conversation without
worrying about note-taking.
Conduct Interviews
•
Thank respondents for their time and stress
the importance of the interview.
•
Interviewers should listen carefully for
recurring and new opinions or beliefs.
•
In order to compare with other interviews
and identify emerging themes, it is
important to get answers from every person
interviewed on key questions.
Advantages and Disadvantages
of Key Informant Interviews
Advantages




Rich data can be gathered
relatively cheaply & easily
Allows interviewer to
establish rapport with the
respondent, clarify
questions, and draw out
responses
Allows for discussion of topic
without group dynamic of
focus groups
Provides an opportunity to
build relationships with
important community
stakeholders
Disadvantages



Selecting the “right” key
informants may be difficult so
they represent diverse
backgrounds and viewpoints
May be challenging to reach and
schedule interviews with busy
and/or hard-to-reach
respondents
Difficult to generalize results to
the larger population unless
interviewing many key
informants
Things to Consider Before You
Select Your Methods
•
What data collection method(s) would best collect
the type of data you need from your target
population
•
What data collection method(s) would help you to
best answer your research question(s)?
•
Think of your available resources: time, staffing,
skills, funding, computer technology
•
Prior experience
•
Potential benefits
•
Potential challenges
How to Understand the
Data You Collect
(Data Analysis)
Why It’s a Good Idea to Plan
Ahead for Data Analyses




To determine if the data you collect are
practical for analyses
To review the appropriateness of your
chosen data collection method(s)
To inform the data collection instrument you
develop
To decide if you will need help with analyses
1: Check the Data

Make sure it’s all there

Make sure it makes sense



Catch any mistakes that happened while filling out
the survey or entering the data
Perform data quality checks throughout your data
collection process
Common mistakes:
– A missing or incomplete response
– An impossible response
– A response unrelated to the question
– A response that contradicts an earlier response
2: Go Back to the Research
Questions




Be sure to analyze only the data that will help you
answer your research question(s)
Don’t get side-tracked into analyzing other pieces of
data with your limited time and resources
It is normal to collect more data than you will need
for your evaluation
You can return to any other interesting data after
completing your evaluation
3: Reduce the Amount of Data




Save your data in its original form so that
you can return to it if you need to
Look at it in small chunks at a time
Eliminate any irrelevant data from your
analysis
Create summary documents to help you
“eyeball” the data more easily to look for
patterns or themes:
– Numeric spreadsheet
– Qualitative summary document
4: Analyze the Data
Quantitative Data Analysis: Looking for
Patterns in the Data

Calculate averages
– Used with “continuous” data: infinite number of
values
– Add numbered responses
– Divide by number of responses
– 2 + 4 + 6 + 10 = 22 22/4 = 5.5

Count frequencies
– Used with “discrete” data: set number of response
categories
– Count number of responses
– Number “yes”, number “no”
– Number of choice “A”, Number of choice “B”, etc.
4: Analyze the Data
Quantitative Data Analysis: Looking for
Patterns in the Data


Calculate proportions
– Is standardized
– Equation: % = # of responses/ total # of respondents
X 100
– Example: 45 females/ 100 respondents X 100 = 45%
of the respondents were female
Calculate rates
– Similar to proportions; also standardized
– Easier to compare
– Equation: Rate = freq of event in pop/ total pop X
100,000
– Example: 15 cases of lung cancer/ population of 3,500
X 100,000 = 428 persons per 100,000 have lung cancer
4: Analyze the Data
Quantitative Data Analysis: Looking for
Patterns in the Data


Compare averages, frequencies, proportions, rates
Compare data from different populations:
– County vs. County
– County vs. State
– State vs. US

Compare data from different segments of a population:
–
–
–
–
–
–
–
Male vs. female
African American, white, Latino, Asian, American Indian, etc.
Children, teens, adults, seniors
Clientele of your program vs. those who are not
Different income or health insurance levels
Married vs. single
Smokers vs. not
4: Analyze the Data
Quantitative Data Analysis: Looking for
Patterns in the Data

Present the data in different ways to see additional
patterns and relationships:
– Chart/ table – shows averages, counts, proportions, or
rates side-by-side
– Pie graph – demonstrates percentages of the whole
– Bar graph – compares quantities
– Line graph – shows trends over time

Determine your findings
– Interesting results?
– Interesting patterns or relationships?
– These are your findings
Exercise 5.4A
(Using Step 5 of Performing a
Community Assessment)


Look at sample survey and sample
spreadsheet in Appendix C
Answer questions on page 51 and 52
Step 4: Analyze the Data
Qualitative Data Analysis: Looking for
Themes in the Data

Qualitative data analysis can be deceptively trickier
– Qualitative data is by nature “bigger”
– Analysis requires more analytical thinking and
interpretation

Qualitative data is more open to “bias”
– Numbers don’t lie
– Analysts bring their values, assumptions and opinions
– Analysts may think they “know” how people feel


Include at least 2, preferably 3 people in each stage
of data analysis to avoid bias
Create as structured of a process as possible to
avoid bias
Step 4: Analyze the Data
Qualitative Data Analysis: Looking for
Themes in the Data
1.
Read through all of the data at least twice
–
2.
Create categories
–
–
3.
–
5.
6.
Focus group or interview questions
Or specific assessment health issues
List themes that emerge from discussion
–
4.
Stay “close” or “grounded” in the data
Sub-categories
Ideas
Rank order according to frequency of appearance
Summarize discussion around each theme
Determine your findings
Exercise 5.4B
(Using Step 5 of Performing a
Community Assessment)


Look at sample focus group discussion
transcript in Appendix C
Answer questions on page 56
5: Verify Findings

You now have:
– Some patterns and relationships you found in your
quantitative data
– Some themes and categories you found in your qualitative
data



Based on the results you tallied and summarized, pull
out the main findings of each method you used
Verify these findings by re-tallying and re-summarizing
the data to make sure you get the same results
You can also verify your findings by comparing the
results reached by 2 analysts independently
6: Interpret Findings and Draw
Conclusions






Determine what interpretations can be drawn from each finding
Invite different perspectives from different Partners and
community members, as this will make your conclusions
stronger
Are the results similar to what you expected? If not, discuss
why you think they are different
Brainstorm alternative explanations for your results to make
sure you have considered all possibilities
Make sure the conclusions answer the original assessment
questions
Draw conclusions and recommendations that can be shared
with external audiences
Exercise 5.6
(Using Step 5 of Performing a
Community Assessment)


Look at the secondary data tables in
Appendix C
Answer questions on page 59