Web-Based Evaluation of
Communities That Care
2003
Mark Feinberg and Mark Greenberg
Prevention Research Center, Penn State
Sponsored by
Evaluation of CTC in PA
In the beginning…
Evaluation Resources
• CTC Evaluation I: 4 year project, 21 CTC sites
• Until that point:
– No unified evaluation instrument
– Little understanding of key factors behind CTC or
general cmt’y coalition success
– Little prior evaluation of CTC beyond case study
Over Four Years of Evaluation…
Prevention Center at Penn State developed
• an interview for Prevention Board members and
mobilizers
• reliable scales to measure CTC functioning
• an understanding of needs of CTC sites (and
made recommendations to PCCD)
• a greater understanding of factors underlying
CTC success
Prevention
Knowledge
Training
and
T.A.
Cmt’y
Barriers: lack resources,
infighting, turnover
CTC
Project
functioning
Pcv’d
CTC
Effectiveness
Change in
Community
Outcomes
(internal and
external)
Readiness
EXPANDED MODEL of CTC
Sustained
Prevention
Planning
CTC Evaluation II?
• Leveraging PCCD’s investment
• Action-research: support local CTC Boards
• CTC Philosophy: Science supporting practice
¾Develop online mechanism for brief
version of the evaluation instrument.
Coordination with RSC’s essential.
Web-based
Self-Assessment System
• Mobilizer sends board member email
addresses to Prevention Center
• PSU programs survey and sends out personal
web link and password
• Board members fill out survey on-line
• Results securely downloaded and tabulated by
Prevention Center
• Report generated for each CTC site
• RSC meets with board to present report and
follow with action planning
Example of Web Survey Item…
32. Consider the following statements about the leadership of your CTC
Prevention Board. For each statement, check how much you agree or
disagree with that statement.
The CTC leadership...
...has the political
knowledge and
competence necessary to
support CTC
...is skillful in resolving
conflict
...adheres to decisionmaking procedures that
the Prevention Board has
adopted (for example, bylaws, voting procedures,
member roles and
positions)
Strongly Somewhat Somewhat Strongly
Agree
Agree
Disagree Disagree
Communities That Care
Web-Based Survey, 2003
SAMPLE REPORT
Community Report
CTC Site : ANONYMOUS
Cycle: XX
Number of respondents
:9
CTC Research Team, Prevention Research Center
Pennsylvania State University
402 Marion Place, University Park, PA 16802
Tel: 1-800-228-5690, Fax: 814-865-6004
Sponsored by
CTC-Web Domains
The areas that were studies in CTC-Web include the following:
Board Membership
Board
Relationship
Board Work Style
Board Leadership
CTC Process from
Readiness to
Sustainability
Barriers Experienced
Technical Assistance
Interpreting Results of CTC-Web
The following charts display results for each domain that
we studied. Charts display results from a scale of 1Æ10
(unless otherwise indicated) where:
10= Excellent
7= Good
4= Poor
1= Very Poor
Example
Under Board Relationship, a “10” for Board Cohesion
would mean that there is an excellent level of strong
cohesion and group spirit.
Board Membership
1
0
Y
ou
rC
T
Csite
Board Stability
A
v
erag
eacrossC
T
Csites
Stability of board membership (Higher
scores indicate lower turnover)
Board Recruitment
7
Ease in recruiting people with skill,
talents, or political connections into the
board
4
1
B
oardS
tab
ility
R
ecru
itm
en
t
Board Relationship
10
Y
ou
rC
T
Csite
Board Cohesion
A
v
erag
eacrossC
T
Csites
Unity and group spirit felt within the
board
7
Board Conflict Management
Conflict managed in constructive manner
and/or low level of conflict.
CTC Staff-Board Communication
4
Frequency and productiveness of
communication between staff and board
members
1
B
oardC
oh
esion
B
oardC
on
flict
M
an
agem
en
t
S
taff-B
oard
C
om
m
u
n
ication
Board Work Style
Board Directedness
Board has clear vision, goals, &
community plan. The board has agreed on
how it will function and the leadership
adheres to the decision-making
procedures adopted by board members.
10
Y
ourC
TCsite
A
verageacrossC
TCsites
7
Board Efficiency
Board members work hard and are highly
efficient with little time wasted due to
inefficiencies.
4
1
B
oardD
irected
n
ess
B
oardE
fficien
cy
Board Leadership
10
Y
ourC
TCsite
A
verageacrossC
TC
Board Leadership Style
The CTC Leadership provides praise &
recognition, seeks out members’ views,
and approaches members to assist with
specific tasks
7
Board Leadership Competence
The CTC Leadership is respected in the
community, able to mobilize resources,
has political knowledge and competence,
is skillful in resolving conflict, and
provides a strong leadership
4
1
BL
eadershipStyle
BL
eadershipC
om
petence
Summary: Interpreting T-scores
Slides 5-14 showed actual scores. Slides 16 – 20 will summarize your
coalition’s ratings using a way of presenting data called t-scores.
7
0
T-scores are useful ways of
looking across domains
because the average score is
assigned a 50, and about
two-thirds of scores are
between 40 and 60, and
about 95% of scores are
between 30 and 70.
over 60 is
very good
6
0
5
0
41-60 is right in
the middle of the
pack – but scores
in the low 40’s
may indicate some
difficulty.
4
0
3
0
2
0
50 is
average
under 40 is
substantially
below others
do
m
ai
n
The t-score method is the
same way results of the
CTC Youth Survey for risk
and protective factors are
presented.
Bo
ar
d
St
ab
Bo
ar
d
R
ec
ru
i
Bo Bo
ar ard
d
C Co
on he
fli
ct
St
M
af
f-B
oa
rd
C
om
m
Bo
ar
d
D
ire
ct
e
Bo
ar
d
Ef
fic
i
70
20
Summary: CTC Ratings (t-scores), Part I
60
50
40
30
B
Le
ad
B
er
Le
ad ship
er
sh
ip
C
om
C
om
m
un
ity
Im
R
e
pl
em
en
ta
tio
n
20
TA
ne
e
TA de
ex
pe
ri
en
Su
st
ai
na
bi
l
Summary: CTC Ratings (t-scores), Part II
70
60
50
40
30
Summary of Findings
Domains
Compared to other sites
Board Membership
Within middle range
Board Relationship
Possible area of concern
Board Work Style
Board directedness needs
further reflection
Board Leadership
Possible area of concern
CTC Process from
Readiness to Sustainability
Possible area of concern
TA needed & experienced
Within middle range. TA
experienced needs further
reflection.
9Good
Area worth further
reflection & problem solving
Area requiring attention &
action plan
Comments
9
Process Evaluation of
Presenting Reports to CTC Sites
Presenting Reports to
CTC Boards
• Regional Strategic Consultants (technical
assistance providers) attend local CTC meetings to
deliver and interpret reports.
• N=27 to date
• Average of 12 members at each meeting
• Average of over 40 min. spent on the report
(range of 15 to 105 minutes)
• After meeting, RSC fills out feedback form on
how the board reacted to the report.
RSC Feedback Reports on
Board Member Engagement
Scale:
– 1=not really, 2=a little, 3=some, 4=a lot
• Were board members interested in the report?
Avg. Score: 3.6
• Did board members disagree with results?
Avg. Score: 2.0
• Were the results perceived as helpful?
Avg. Score: 3.5
• Were the results used to problem solve and plan?
Avg. Score: 2.5
Number of sites
Did Board Members find the Survey
Informative and Helpful?
16
14
12
10
8
6
4
2
0
1
2
3
Informative/Helpful (1=no)
4
Number of sites
Did Board Members Disagree with
the Survey Results?
14
12
10
8
6
4
2
0
1
2
3
Disagreement (1=none)
4
Employing Web Survey for Problem Solving
Number of sites
10
8
6
4
2
0
1
2
3
Level of Problem Solving (1=none)
4
Time Spent by Level of Problem Solving
100
Minutes
80
60
40
20
0
1
2
3
Problem Solving (1=low)
4
Communities That Care
Web-Based Survey, 2003
State-wide Scores
67 CTC Sites
Number of respondents
: 552
A
v
e
ra
g
eS
c
o
re
s
:6
7C
T
CS
ite
s
1
0
7
4
1
1
Sustainability
FidtoModel
Recruitment
BdStable
Ldr Comp
Ldr Style
ConMgmt
BdCohesion
BdEffic
St-BdComm
BdDir
ComSupp
Latestvs.M
iddleC
ycles
10
7
avglate
4
avgm
id
A
vgTA
:67P
AC
TCS
ites
3.0
2.5
2.0
1.5
1.0
Scale
1=Not needed
2=A little
3=A fair degree
4=Need a lot
Barriers:67PACTCSites
2.5
2
1.5
Scale
1
1=Not a problem at all
2=A small & manageable problem
3=A significant problem at times
0.5
4= A significant, chronic problem
Right People
on Bd
Bd Member
Commitment
Program
Enrollment
Strong
Leaders
CTC Model
Fidelity
Community
Support
Cmt'y
Agency
Coop
Resources
T.A.
Quality Staff
Group
Divisions
Infighting
0
Unresolved
Conflict
5= A huge & overwhelming problem