WRITING GREAT TOBACCO CONTROL
FINAL EVALUATION REPORTS
WEBINAR TRAINING
APRIL 4 & 5, 2013
Welcome!
2
AGENDA
Thursday, April 4th
10:00-10:10 Welcome!
10:10-10:45 The Project Description Section::
Overview, scoring the report, discussion
10:45-11:30 The Methods Section
Overview, scoring the report, discussion
Friday, April 5th
10:00-10:45 Writing Up Results & Conclusions, Overview
10:45-11:30 Scoring and Discussion
3
WHY FINAL REPORTING?
Organizational
History
Sharing with
colleagues
idi.org
4
STATEWIDE & NATIONAL RELEVANCE
TCEC publishes summary reports and articles
5
Reporting Guidelines
6
FER RATING FORM
Report Section
(maximum 32 points)
Maximum Points
Title Page
2
Abstract
6
Project Description
6
Evaluation Methods
8
Evaluation Results
4
Conclusion &
Recommendations
6
PROVIDING SOME CONTEXT:
PROJECT DESCRIPTION SECTION
8
PROJECT DESCRIPTION SCORING
(maximum 6 points)
Project
Description
Background
Objective
Intervention
Maximum
Points
2
2
2
9
PROJECT DESCRIPTION ELEMENTS
 Provide context for the objective
 Explain rationale for choosing the
objective
 State the objective and indicator
 Describe what you did and why
 Identify activity timeframes
 Explain any changes to plan
10
IT’S YOUR TURN TO SCORE
10-15 minutes
• Read and score the project description
of this report
• Write on the report any questions you
have
5-10 minutes
• Discuss scores as a group.
EVALUATION METHODS
12
DESIGN TYPE
PROCESS AND OUTCOME
 Experimental (control group,
random assignment)
 Quasi-experimental (at least
three waves, comparison group)
 Non-experimental (no
comparison, only two waves)
 Process and/or Outcome
13
DESIGN
LIMITATIONS
14
SAMPLES
15
DATA COLLECTION INSTRUMENTS
AND PROCEDURES
16
DATA ANALYSIS
17
EVALUATION METHODS SCORING
(MAXIMUM 8 POINTS)
Evaluation Methods
Evaluation design
2
Samples from which data
were collected
2
Data collection instruments
and procedures
2
Data analysis
2
18
IT’S YOUR TURN TO SCORE
10-15 minutes
• Read and score the Evaluation
Methods section in pairs
• Write on the report any questions you
have
5-10 minutes
• Discuss scores as a group.
19
SCORING METHODS SECTION
20
REMEMBER TO
JOIN US TOMORROW!
PART 2 begins at 10:00 a.m. Same login info:
1. Call in for audio at 1.866.740.1260 and enter access
code 2974659, AND
2. Log in for the visuals at www.readytalk.com and enter
the access code 2974659
21
 Our team — Diana Cassady, Jeanette Treiber, Robin Kipke,
Catherine Dizon, Diana Dmitrevsky
 Our website — http://tobaccoeval.ucdavis.edu
 Our email — tobaccoeval@ucdavis.edu
 Our main line — 530.752.9951
We are the statewide technical assistance center on
evaluation for all Prop. 99-funded projects in California
WRITING GREAT TOBACCO CONTROL
FINAL EVALUATION REPORTS
WEBINAR TRAINING
APRIL 4 & 5, 2013
Part 2
Welcome!
23
AGENDA
Thursday, April 4th
10:00-10:10 Welcome!
10:10-10:45 The Project Description Section
Overview, scoring the report, discussion
10:45-11:30 The Methods Section
Overview, scoring the report, discussion
Friday, April 5th
10:00-10:45 Writing Up Results & Conclusions, Overview
10:45-11:30 Scoring and Discussion
WRITING UP RESULTS
25
BRAINSTORM
 What is the purpose of the results
section of a report?
 What function(s) should it serve?
26
PURPOSE OF THE RESULTS SECTION
 Present the findings of data analysis
 Explain what the data reveal
 Point out what is important
 Describe how the data informed your
project
 Justify report conclusions
27
FER SCORING CRITERIA
Evaluation Results
Maximum
Points
Findings are clearly & logically presented.
Tables and figures are used when needed
and are clearly labeled.
Findings are objective (include both
positive and negative when relevant)
Section total
2
2
4
28
WHAT TO INCLUDE
Report on every activity
But don’t report every result
29
QUANTITATIVE DATA
202 (79%) Alta Vista
residents were in
favor of the policy
 Provide totals and percentages
 Note statistical significance (where
appropriate)
30
EXCEL-GENERATED TABLE
A paired T-test comparing 2010 and 2013 shows a statistically significant difference
in the number of cigarette butts between 2010 and 2013 (p<0.05)
31
VISUALLY IMPROVED TABLE
Table 1. Number of Cigarette Butts in Sacramento Area Parks, by Year
Location
2010
2011
2012
2013
Arden
68
72
81
37
Sunrise
282
343
314
45
Fair Oaks
121
180
196
58
McKinley
312
369
231
111
Goethe
216
248
239
222
*999
1212
1061
*473
Total Citywide
*Statistically significant difference p<0.05
32
LINE GRAPH
33
BAR CHART
34
QUALITATIVE DATA
 Common themes
 Contrasts
 Indicative quotes
 Avoid percentages
35
SUMMARIZING CONTENT ANALYSIS
Figure 3. Incentives and Barriers to Smoke Free Casino Policy
36
BALANCING ACT
Focus on what’s most important
Report both + and - results
Make results transparent
37
ATTENTION GETTERS




Supplement text with
visual elements
Draw focus to key
findings
Make data digestible
Match format to data
type
CONCLUSIONS &
RECOMMENDATIONS
39
FER SCORING CRITERIA
Conclusions & Recommendations Maximum
Points
Discuss outcome/process evaluation
2
Conclusions are supported by data
2
Includes recommendations for future work
2
Section total
6
40
INTERPRETING THE FINDINGS
 Confirm what you expected to find, and did
 Dispel what you thought was true, but isn’t
 Bring to light what you didn’t know, but should
41
LEAVING A ROADMAP
What strategies worked well or poorly?
What would you do differently next time?
What steps should follow?
42
IT’S YOUR TURN TO SCORE
10-15 minutes:
•Read and score the results and
conclusions/ recommendations section
•Note any questions you have
10-15 minutes:
•Reconvene to share scores
43
AS YOU REVIEW THE RESULTS…
1. What questions does it leave unanswered?
2. Do the results address what the project wanted
to know?
3. Did the section provide enough information for
the reader to assess the findings?
4. Are the most important findings brought out in
the narrative and/or the tables?
5. Do the findings appear to be objective?
6. Does the section adequately interpret the results
by explaining how the data informed the project?
44
AS YOU REVIEW CONCLUSIONS
1. Do the conclusions evaluate the process
and outcome findings in relation to the
achievement of the objective?
2. Are the conclusions supported by the
data?
3. Do the recommendations provide a clear
roadmap for next time?
45
REPORT RATING
 Total score: ___/32 possible points
 Rating: ____ High (24-32)
____ Medium (16-23)
____ Low (0-15)
 Overall Assessment:
 Recommend report for intervention activities?
 Recommend report for evaluation?
46
WORKSHOP RECAP
 Why final evaluation reports matter
 Essential components of a report
 Provide enough detail about your path
 Make rationale and linkages transparent
 How reports are scored
 Use ratings & feedback to improve
47
PARTICIPANT SELF-ASSESSMENT
Help us learn how we’re doing.
Complete the self-assessment form
and rate your understanding
of writing final evaluation reports
before and after this training.
Thank you!
48
 Our team — Diana Cassady, Jeanette Treiber, Robin Kipke,
Catherine Dizon, Diana Dmitrevsky
 Our website — http://tobaccoeval.ucdavis.edu
 Our email — tobaccoeval@ucdavis.edu
 Our main line — 530.752.9951
We are the statewide technical assistance center on
evaluation for all Prop. 99-funded projects in California