From Sea to Shining Sea:
Evaluating Title II TQE Partnership
Strategies to Ensure Quality
Susan Ann Tucker, Evaluation & Development Associates LLC
Nancy Shapiro, University System of Maryland
Letitia H. Fickel, University of Alaska, Anchorage
Shirley Lal, California State University, Dominguez Hills
Roy Weaver, Ball State University
Tomas Morales, College of Staten Island
Agenda








Partnership challenges
Core TQE assumptions and charges
5 questions of focus today
Overview of case study projects
Findings: causal network analysis
Recommendations & follow-up strategies
Small group discussion
Open Q&A
Core TQE assumptions





Partner approach is a vehicle for human and institutional
development
Facilitates knowledge creation and diffusion
Moves the NCLB agenda forward
Makes possible better cost-sharing
Contributes to teacher education capacity development
TQE charges

Creating collaborative partnerships and learning communities in
high need P-12 schools to enhance student achievement

Improving quality teacher preparation—integrating standards-based
assessment strategies and interdisciplinary training and expanding
clinical experiences

Articulating a continuum of services from recruitment to induction
that serves preservice to beginning teachers in collaborative (and
sometimes not so collaborative) settings of school districts,
community colleges, and universities
Partnership challenges





How to identify premises, tacit and overt, embedded in
“partnership”?
What are sources & methods to document partnering
contexts, processes, outcomes?
What model(s) to use for evaluation design, data
collection, and analysis?
What evidence do you have that partnership is “improving”
partners?
How is data being used?
Evaluation Questions
Contexts and Inputs:
- How did your partnership begin? Were there clear rules for membership?
- What is the purpose of this partnership?
- What is the range of partner members? Any linkers?
- How do organizational structures support/inhibit partnering?
- What resources exist for partnering: Incentives? PD? Resources? IT? Are they used?
Processes and Hard Talk:
- What do you talk about? Any patterns? How regularly? Any networks?
- Who decides what to talk about? How are agendas formed?
- How does data guide your talk? How do you collect this data?
- How satisfied are members with partnership? How do you know when it is working?
- What are methods of handling conflict?
Outputs: Decisions and Actions:
- Identify significant decisions. Who participated in this?
- How did your decisions relate to grant objectives?
- Who made these decisions and by what method?
- How are actions of partnership coordinated?
- What were benefits & successes of partnership? Success story?
- Constraints you could not fix? Challenges? ? Surprises?
- Recommendations for follow-ups
Discussion questions today…
1.
2.
3.
4.
5.
Who are the partners? What is the setting where
partnerships operate?
What were moderators & mediating variables?
What partnering actually occurs?
What are the outcomes of the partnerships?
What are needed follow-ups and recommended
replication strategies?
Five TQE projects in study
Ball State University
6 districts (23 PSD school
network), 2 museums, Apple
Cal Poly Pomona
5 districts, 3 CC’s, Southern
Edison, Boeing
California State UniversityDominguez Hills
1 district, 2 CC’s, 2 other CSUs,
NSF SCALE, Boeing, AgileMind
University of Alaska-Anchorage
9 districts, 3 CC’s,
GCI (telecom company)
University System of Maryland
1 district, 1 CC, 3 USM
universities, Maryland Business
Roundtable
Mixed methods design & strategies


Utilization-oriented evaluation model
Mixed data sets

Qualitative:
• observations, interviews and self-report surveys (eg, candidates, faculty, district
staff, leaders, mentors), critical friend protocols, context analysis of
district/building/classrooms, electronic portfolios, collaboration analysis using
NSDC tools, NCATE PDS standards, participant action research,

Quantitative:
• performance-based and standardized achievement tests, teacher & administrator
retention and turnover rates, costs and budgeting allocations, GIS data,
candidate enrollment and completion rates, policies formalized by partners,
publications and grants generated, new partnerships
Contexts to consider




The five case studies represent a convenience sample unified by
having the same external evaluation
Three are in their fourth year of operation
Two fundees have received a second cycle of TQE five-year funding
and are referred to as exemplars by the federal staff
Each grantee has at least the following partners:




IHE where both SCDEs and colleges of arts and sciences participate
public school districts serving high need, diverse populations
for-profit private sector organizations
Examples of partners:

SEA, PDS networks, State K-16 Committee, Chancellor’s Office of a state’s university
system, Superintendents’ Roundtable, Provost/Academic Affairs Office, Tech/telecom
companies, Museums, Business Roundtable, CC’s
Analysis methods


Demographic analysis
Content analysis


Performance analysis of content & pedagogy



Open-ended survey items, interviews, extant docs, portfolios
Multifactor matched pair method
Pre-post score analysis
Management and partnership analysis





Data team protocol analysis
Group development rubric (Wellman & Lipton, 2004)
NSDC IC rubrics
Datawise step analysis (Boudett et al, 2005)
Causal network analysis
Learnings & Recommendations

Causal network analysis
Contextual variables
 Mediating variables
 Outcome variables

Contextual variables
External
influence
No systematic STEM faculty
training in teaching; no PD
facilitator training;
Awareness of state and national
standards
Internal
structure
COE programs separate from A&S
programs; TED programs divided
from Admin programs; demanding
faculty workload, RTP policies
Leadership verbally “supports” TQE
reform; value of COE for field service vs
A&S; mid-career and senior IHE faculty
participation
Interactions
& resources
STEM/COE faculty interaction
competitive; lack 360 degree
involvement in grant prep
Networks of discipline based faculty;
PDS networks vs “partners”; leveraged
resources by PI’s over time
Shared
meanings
Lack negotiated definitions by
A&S/COE/CC/LEA re: COP,
PDS,inquiry, partnership,assmt
Verbal support for diversity and
culturally responsive teaching, student
centered instruction by COE/LEA
Individual
beliefs
Instructional practice not
consistently conscious at all levels
Some A&S individuals interested in
pedagogy improvement
Practice
Improvements in instruction not
guided by data; teacher & admin
turnover in high need schools
Role of data enhanced decision making
valued “verbally”
Mediating variables
External influence
Logic maps tied to $ and SIP; Research-based PD for LEA
teachers & STEM IHE faculty, NCATE, shifting state $
Internal structures
Workload reduced by grant monies; RTP polices and
workload reviewed by administration; LEA release time
Interactions &
resources
Grant & LEA interactions become driven by data and
action agendas; systematic attention to trust building;
partnership networks created/enhanced, national & state
standards; leveraging multiple grants/partners across grant
Shared meanings
Intentionally built by trained facilitators: CRT, PDS, COP,
inquiry, pedagogy techniques, protocols for partnering
Individual beliefs
Facilitators uncover tacit assumptions and make them
overt
Practice
Use interdisciplinary teams of facilitators, A&S go beyond
“expertise”, shifting mental models, trained data teams;
trained IT integration across COE & A&S, curric mapping
Outcome variables
External
influence
Admin PD models elusive
Developed IHE & LEA faculty PD
models;
Internal
structure
Workloads unchanged
Institutionalized novice IHE faculty PD;
partnerships between COE/A&S/LEA
faculty in PD/research
Interactions &
resources
Dependency on grant funding
Personality referent leadership
Transcending match difficult
New networks created; operationalized
for partnering and “divorce”; RTP
policies mitigated by personal
intervention,strengthened principal
involvement; enabled new grants
Shared
meanings
A&S discipline currency
discourages pedagogical
improvement across depts
Changes in A&S/COE views of one
another,actions tied to SIP’s; codified in
action plans and logic maps
Individual
beliefs
Short term visions predominate
Conscious connecting of beliefs with
practice; increased valuing of research
& pedagogy & alignment with practice
Practice
Private sector involvement limited;
Changes in IHE/LEA instruction;
datawise cycles increase; cohorts/COL
develop across partners; Increased
research-based instructional practice &
assessment
LINC Project-wide Spending
6 years - $10,732,825 total spending
TU, $793,677,
7%
BSU,
$497,895, 5%
MBRT,
$40,000, 0%
UMCP,
$1,566,082,
15%
PGCPSS,
$5,747,950,
53%
PGCC,
$515,179, 5%
USM Office,
$1,572,042,
15%
Partnering strategies


Write the original proposal with adequate collaboration among all partners
Plan for sustainability & institutionalization across partners from year 1 as reflected in logic
maps








Expect incentives to partnering to evolve over time in terms of specificity, collaboration, and
resourcing
Overtly plan for changing leadership in IHE & districts & districts
Be open and/or actively recruiting new partners
Tie partner debriefings to evidence-based action agendas
Make funding contingent upon a review process that involves specific requirements for
collaboration across multiple stakeholders and evidence of successful collaboration
Invest in “facilitation” training across partners


Tie budgets for PD and collaborative activities to SIP and “real” school needs
Deal with the politics of cost-sharing
Example: train STEM faculty in pedagogy using Understanding by Design
Plan preservice & induction collaboratively & use cross disciplinary teams
Success depends on…

Incentives to partners evolving over time

Genuine collaboration between COE and A&S

Recognizing patterns of resistance & constraints you can’t fix

Leveraging resources:

umbrella enabled moderating and mediating that could not be done at
single partner level and harnessed competition to be productive

Collaborations between R-1 and comprehensive universities

Using partners and enhanced resource capacity to solicit new monies

Customizing partnership to harness unique contexts & inputs

Investing in developing networks of facilitators

teachers as leaders, data teams, COL, mini-networks, PDS networks
MENU of topics to explore
1.
Testing range of certification options



2.
3.
4.
Redesigned innovative undergraduate programs
Producing STEM graduates
Piloted 2+2 program between university and CC
Developing Mentoring & Induction to improve retention rates


Created Orientation Program as part of new teacher contract
Trained Job-Alike mentors working with new teachers

Developing PDS, mini-networks and communities of practice
Leveraging resources with partners



Dealing with increasing match requirements
Embedding IT and Culturally responsive teaching
Soliciting and maintaining partners--building shared meanings


Short-term
Long-term
Institutionalization and dissemination strategies
Interactive discussion

Discuss 1-2 topics of most import to your
group


Notetakers summarize main points to share out in
large group
Identify clarifying questions for panel to discuss in
more detail
Panel discussion
 Open Q&A
