Teacher Quality Activities Grant-FGCU
Professional Development System
Assistance,
Third Cycle Development, 2009-10
Submitted by
Constance C. Bergquist, Ph.D.
Evaluation Systems Design, Inc.
5145 Pimlico Drive
Tallahassee, FL 32309
850-893-9504 (phone)
850-893-9569 (fax)
850-508-8087 (cell)
email: esdi@talstar.com
website: www.esdi.cc
May 5, 2009
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
Teacher Quality Activities Grant-FGCU
Professional Development System Assistance,
Third Cycle Development, 2009-10
Introduction
For almost a decade, the Florida Department of Education (DOE) has been implementing
the Professional Development System Evaluation Protocol, a system for evaluating the
quality of district professional development systems in accordance with legislative
requirements stipulated in 1012.98 – School Community Professional Development
Act (F.S.). Presently, the Department is concluding the Second Cycle of reviews of all
67 school districts and university research school.
State law specifies that “The purpose of the professional development system is to
increase student achievement, enhance classroom instructional strategies that
promote rigor and relevance throughout the curriculum, and prepare students for
continuing education and the workforce. The system of professional development
must align to the standards adopted by the state and support the framework for
standards adopted by the National Staff Development Council.” (s. 1012.98 (1) F.S.).
The purposes of the Professional Development System Evaluation Protocol are to:
1. Ensure the highest quality district, school, and faculty Professional Development
Systems in Florida to support instructional programs throughout the state.
2. Provide the Commissioner of Education, State Board of Education, and
Legislature with information each year on the quality of the district Professional
Development Systems.
3. Provide Florida school districts with the methods and protocols needed to conduct
ongoing assessments of the quality of professional development in their schools.
Since 2001, the Department has contracted with Evaluation Systems Design, Inc., (ESDI)
of Tallahassee, Florida, to continue development and assist in implementing the
Professional Development System Evaluation Protocol. Presently, the Department has
conducted initial development activities in revising the Protocol System for the Third
Cycle to update the standards and integrate recent national and state developments in
professional development. Florida Gulf Coast University, through the Teacher Quality
Activity Grant from the Florida Department of Education and in conjunction with
Evaluation Systems Design, Inc., is interested in contracting for services to continue to
Evaluation Systems Design, Inc.
1
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
support the development of the Third Cycle standards, assist in generating Third Cycle
documents, and assist in planning and conducting dissemination activities to inform all
local school districts of the new standards and documents. Activities will be concluded
by May 31, 2010.
Detailed Tasks and Products
ESDI will conduct six major tasks that will culminate in the Third Cycle Professional
Development System Evaluation Protocol materials and documents for use in the 201011 school year. Detailed descriptions of these tasks and products are presented below
and followed with a timeline for completion of all tasks:
1. Assist FGCU/DOE in Conducting Review Activities for New Standards
2. Assist FGCU/DOE in Generating Supporting Documents for the New System
3. Assist FGCU/DOE in Organizing and Conducting a Field Test of New
System
4. Assist FGCU/DOE in Organizing and Conducting Awareness Sessions on
New System
5. Assist FGCU/DOE in Generating a New Computer Data System for New
Protocol System
6. Assist FGCU/DOE in Generate a Report Documenting the Development
Process
1. Assist FGCU/DOE in Conducting Review Activities for New Standards
The final standards will be disseminated to the professional development community
initially through the September, 2009 meeting of the Florida Association for Staff
Development to be held in September. Dr. Kathy Hebda, Bureau Chief for the Bureau of
Educator Recruitment, Development, and Retention will present the standards and the
new vision for professional development for Florida. The DOE website will be updated
to reflect the new standards, and a cross-walk will be generated to display the relationship
to the previous standards with a rationale for inclusion of new content areas.
FGCU and ESDI, with the assistance of Dr. Joellen Killion, will further disseminate the
draft standards to all district staff development directors and a larger group of
stakeholders in professional development for further comments and suggestions.
Additional suggestions for revisions will be solicited from school districts and other
organizations involved in planning, delivering, and evaluating professional development
for school systems. The Third Cycle Standards will be revised yet again and submitted to
the DOE professional development staff. Final review and approval will be conducted by
the DOE professional development staff.
(Completion by September. 30, 2009)
Evaluation Systems Design, Inc.
2
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
2. Assist FGCU/DOE in Generating Supporting Documents for the New System
Following the final approval of the Third Cycle standards, FGCU and ESDI with the
assistance of Dr. Joellen Killion, will draft new supporting documents for the Third
Cycle. These documents will include, but not be limited to:




New Protocol System document – containing
i. an overview of the Third Cycle standards and their development,
ii. a listing of all standards, and
iii. sections describing the checklists and judgment scales,
1. district site visit process,
2. review cycle and logistics,
3. awareness and preparation
4. documentation and data analysis
5. reporting and action planning
6. district and school use
7. web-based assistance
8. action plans for improvement
9. district preparation for a review, and
10. commonly asked questions
New Reviewer’s Guide – containing for each standard
i. a rationale for inclusion,
ii. an explanation for the standard with descriptions of the various ways
in which the standard may be met in school districts
iii. factors that may inhibit or restrict implementation
iv. rating rubrics for reviewers to use in making judgments that will
increase the consistency of judgments made across reviewers.
New Checklists to be used by reviewers that display the standard and the
documents, questions, and probes to be used in judging the standard
Reporting formats for use in conducting reviews
(Completion by October 31, 2009)
3. Assist FGCU/DOE in Organizing and Conducting a Field Test of New System
FGCU and ESDI will assist the DOE in organizing and conducting six district pilot tests
of the Third Cycle Professional Development System Evaluation Protocol in the fall of
2009. Selected districts will include 2 small, 2 medium and 2 large sized districts. We
anticipate that the reviews will be conducted in November and December. Each site visit
will be a 2-day visit for 2 people with one day spent in a school and one day at district
level. One ESDI staff member will participate on each team visit as the Team Leader.
We will summarize the results from each site visit into a mock-up of a report as well as
summarizing the “lessons learned” and suggestions for revisions/improvements in the
Evaluation Systems Design, Inc.
3
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
Third Cycle standards and review process. We will transfer the quantitative data from
the reviews of the pilot districts into SPSS and analyze the data using descriptive
statistics of means, medians, modes, ranges, and percents as appropriate for the data. We
will use a variety of methods for analyzing the qualitative data gathered through
interviews with district administrators, principals, teachers, and other school staff. As
appropriate, we will also compare results on standards held constant across the three
cycles to compare with results from the First and Second Cycles. Based on these results
and in consultation with Dr. Joellen Killion and the DOE professional development staff,
we will revise standards a final time and the supporting documents and materials to
provide a final complete set of standards and procedures for the Third Cycle Professional
Development System Evaluation Protocol.
(Completion December 15, 2009)
4. Assist FGCU/DOE in Generating a New Computer Data System for New
Protocol System
At the present time, the DOE is using an ACCESS-based computer data collection and
analysis system to generate draft reports from each site visit. The system employs
computers that are now four years old and must be deployed on computers with the same
configuration of operating system and version of ACCESS. The Department anticipates
using another contractor (FCIM perhaps) to create a web-based data collection and
analysis system that will incorporate and reflect the new standards, Reviewer’s Guide,
and rating rubrics. ESDI, working with FGCU; will assist in this process by providing
technical assistance to the developers, meeting as needed to clarify the process, reviewing
and testing components of the system while under development, and using a Beta version
of the system in a field testing setting.
5. Assist FGCU/DOE in Organizing and Conducting Awareness Sessions on New
System
Following reviews and approvals within the DOE and the State Board of Education,
FGCU and ESDI will conduct six regional meetings to increase awareness of the new
standards and the ramifications for local school districts. These sessions will be one-day
sessions held throughout the state with possible locations at Panama City (or PAEC),
Tallahassee, Gainesville, Orlando, Ft. Lauderdale, and Sarasota. We also will assist the
Department as needed in preparing and conducting webinars for the Third Cycle system
as well as participating as needed in the creation of podcasts and from the awareness
sessions. We anticipate that the awareness and dissemination phase of the development
process will extend through May, 2010 and include presentations at statewide groups
such as FASD.
(Completion by May 15, 2010)
Evaluation Systems Design, Inc.
4
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
6. Assist FGCU/DOE in Generate a Final Report Documenting the Development
Process
Based on and using the quantitative and qualitative results from the field testing for the
Third Cycle Professional Development System Evaluation Protocol, FGCU and ESDI
will generate a final report that presents the development process for the Third Cycle.
The final report will contain an introduction section that explains the rationale of the
Third Cycle Professional Development System Evaluation Protocol and the process used
to develop and field test the system. The final report will be written in simplified
language and avoid education jargon and acronyms to increase the readability of the
document.
(Completion by May 31, 2010)
Evaluation Systems Design, Inc.
5
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
Project Tasks and Budget
Dr. Constance C. Bergquist, the President of Evaluation systems Design, Inc., will be the
project manager and will guide and direct all contract activities.
Evaluation Tasks and Timetable
Task
ESDI
Days
1. Assist FGCU/DOE in Conducting Review Activities for
New Standards
2. Assist FGCU/DOE in Generating Supporting
Documents for the New System
3. Assist FGCU/DOE in Organizing and Conducting a
Field Test of New System
4. Assist FGCU/DOE in Generating a New Computer
Data System for New Protocol System
5. Assist FGCU/DOE in Organizing and Conducting
Awareness Sessions on New System
6. Assist FGCU/DOE in Generate a Report Documenting
the Development Process
Total
11
Completion
Date
9/30/09
30
10/31/09
37
12/15/09
Evaluation Systems Design, Inc.
4
9
5/15/10
6
5/31/10
97
6
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
Evaluation Systems Design, Inc.
Evaluation Systems Design, Inc. (ESDI) of Tallahassee, Florida, is a private consulting firm
whose purpose is to provide expert assistance in assessing, designing, developing,
implementing, and evaluating programs in education and human services. Areas of
expertise and interest include: program evaluation, instructional systems and design,
technology implementation and use, training, needs assessment, tests and measurement,
educational research, statistics, data analysis, curriculum development, and information
systems. Incorporated in 1979, ESDI has successfully completed more than 100 contracts
with local and state agencies and with private industry, including contracts with the
FGCU/DOE to evaluate staff development programs and the regional Area Centers for
Educational Enhancement. Our first Florida contract was a statewide evaluation of the
Teacher Education Centers, and we also conducted a major statewide study of staff
development for the Illinois Department of Education. ESDI has many years of experience
in developing and conducting scientifically sound evaluation methodologies and work
plans that are closely monitored.
ESDI staff members have had extensive experience in program evaluation and program
development at the national, state, and local levels for many education programs. We have
completed national studies in the private sector for IBM, Motorola, and the National
Association of Realtors. We also have had major statewide contracts in Illinois, Georgia,
Louisiana, Virginia, and Connecticut, and have completed projects of a similar nature and
duration to the proposed work effort. Because of its evaluation and staff development
expertise, ESDI is uniquely equipped to address the critical issues for this proposal.
Examples of contracts successfully completed are listed below:
•
assisting the Florida Department of Education in the first two full cycles
of implementing the Florida Professional Development System Protocol in
67 school districts, and four university schools
•
generating the 1999-2000 Final Report and Executive Summary of the
Florida Professional Development System Protocol for the Florida
Department of Education that established the system
•
generating the 1998-99 Florida School Recognition Program Executive
Summary and Final Report for the Florida Department of Education
•
conducting research for and generating for the Florida Department of
Education two sets of documents entitled Successful Schools: A Closer
Look, including onsite visits to many school districts and analyses of factors
associated with higher performing, low poverty schools in Florida
•
developing a training program for the Florida Department of Education on
Vital Signs for School Improvement
Evaluation Systems Design, Inc.
7
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
•
conducting five training sessions for a total of over 400 Florida educators on
Vital Signs for School Improvement
•
designing and implementing an evaluation of the Florida Area Centers for
Educational Enhancement
•
designing and implementing an evaluation of the Florida At-risk
Prekindergarten programs
•
designing an evaluation of the Florida Statewide Initiative in Science,
Mathematics, and Technology
•
designing a comprehensive evaluation for a statewide systematic evaluation
of early intervention programs for young children
•
designing and producing an annual report of Chapter I services in Florida
•
conducting a needs assessment and evaluation of the Madison County
Chapter I Program
•
designing and implementing an evaluation of the Florida Reading
Improvement programs
•
designing and implementing an evaluation of two Goals 2000 programs in
Florida
•
designing and implementing a national evaluation of the Writing to Write
Program for IBM including national surveys and onsite visits to 27 sites
nationwide
•
designing and conducting a comprehensive evaluation of the Gifted
Education Program for the State of Florida
•
developing a state-wide handbook for the evaluation of gifted education
programs
•
designing and conducting a comprehensive evaluation of gifted education for
the Commonwealth of Virginia
•
developing self-instructional materials and group training for exceptional
student education (ESE) administrators across the state of Florida
•
designing and conducting a comprehensive evaluation of the special
education programs and compliance monitoring system for the
Commonwealth of Virginia
Evaluation Systems Design, Inc.
8
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
•
designing and implementing an evaluation of the Florida Staff Development
programs
•
designing and implementing training for more than 500 Florida Department
of Corrections staff members on racial, cultural, and gender diversity
•
designing and implementing an evaluation of the Florida Teacher Education
programs
•
designing and implementing a comprehensive statewide evaluation of the
Illinois Accelerated Schools Program for the Illinois State Board of
Education
•
designing and implementing a comprehensive statewide evaluation of the
Illinois Gifted Education Program for the Illinois State Board of Education
•
designing and implementing a comprehensive statewide evaluation of the
Illinois Reading Program for the Illinois State Board of Education
•
designing and implementing a comprehensive statewide evaluation of the
Illinois Pre-kindergarten At Risk Program for the Illinois State Board of
Education
•
designing and implementing a comprehensive statewide evaluation of the
Illinois Staff Development Program for the Illinois State Board of Education
•
developing and field-testing a comprehensive program audit system for the
Louisiana Department of Education
•
designing and conducting a statewide evaluation of the child abuse and
neglect prevention programs in Florida
•
evaluating the Head Start program for Palm Beach County, FL
•
conducting validation visits for NAEYC
•
conducting the third-party evaluation of Florida's State Plan Grant for Prekindergarten Handicapped Students
Evaluation Systems Design, Inc.
9
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
Project Personnel
Dr. Constance C. Bergquist, President, Evaluation Systems Design, Inc., serves as project
director of all ESDI projects. She received a Ph.D. in Evaluation and Measurement from the
Department of Education Research, Development, and Foundations, Florida State
University. She has extensive management experience as the contract manager for
numerous projects in education, including those which have by necessity been completed
under very short time frames. For example, Dr. Bergquist supervised the evaluation of the
Illinois Staff Development Program, the Illinois Accelerated Schools Program, and the
evaluation of Area Centers for Educational Enhancement in Florida. She also was Project
Director for the National Evaluation of Writing to Write for IBM and the development of
the teacher's guide and all supplementary materials for the Writing to Read 2000 product.
Dr. Bergquist has spearheaded ESDI’s contracts since 2000 to study professional
development, was responsible for the development of the First Cycle for the Protocol
System, and has had successive contracts to implement the First and Second Cycles across
all 67 school districts.
Dr. Bergquist has been the Committee Chair for Research and Evaluation for the American
Evaluation Association as well as President of the Florida Educational Research Association
and the Southeast Evaluation Association. Dr. Bergquist has conducted numerous formative
and summative program evaluations for state and local education and human service
agencies. She designed, collected, and presented five evaluations of exemplary education
projects which were approved by the National Joint Dissemination Review Panel in
Washington as exemplary education programs for national dissemination. She also was a
lead researcher for the Florida Evaluation of the Child Abuse and Neglect Prevention
Programs, and has conducted training on case management for Florida's community mental
health case managers. She was a trainer for two years on Appreciating Racial, Cultural, and
Gender Diversity for the Department of Corrections. She has classroom teaching
experience in Program Evaluation and Statistics. She is a certified NAEYC validator. Dr.
Bergquist will be responsible for the implementation all contract activities.
Dr. Joellen Killion is Deputy Executive Director for Strategic Initiatives for the National
Staff Development Council. Her areas of expertise are NSDC's standards, strategic
alliances and partnerships, NSDC's academy, NSDC affiliates, the Big 35 Network, the
SEA PD Network, and projects related to demonstrating the impact of staff development
efforts. Joellen is also responsible for overseeing communications and publications
initiatives for NSDC. She has been directly involved in professional development in
Florida through extensive training on the NSDC standards and on evaluating professional
development, as well as consulting with several of the school districts in Florida.
Dr. Cindy L. Bigbie received her Ph.D. in Instructional Systems Design from Florida
State University. Dr. Bigbie has been a classroom teacher and has extensive experience
in conducting educational evaluations involving at-risk youth and in designing and
implementing educational programs in a range of areas. She has directed the evaluation
Evaluation Systems Design, Inc.
10
Teacher Quality Activity Grant-FGCU
Professional Development System Assistance – Third Cycle Development 2009-10
of The Florida On-line High School for five years and completed a range of other
regional and district evaluations of innovative programs. Dr. Bigbie was one of the
original developers of the Protocol System and has conducted two or more school district
reviews every year since initiation of site visits, serving as a Team Leader. Dr. Bigbie
will assist in the development of the new standards and supporting documents, serve on the
field testing site visits, and assist in generating reports associated with the study.
Carol Johnson served on the original study group overseeing the development of the
Protocol System and has conducted school district reviews every year since initiation of
site visits, serving as a Team Leader or Assistant Team Leader for many visits. Carol
will assist in the development of the new standards and supporting documents, serve on the
field testing site visits, and assist in generating reports associated with the study.
Debbie Cooke served on the original study group overseeing the development of the
Protocol System and has conducted school district reviews every year since initiation of
site visits, serving as a Team Leader or Assistant Team Leader for many visits. Debbie
will assist in the development of the new standards and supporting documents, serve on the
field testing site visits, and assist in generating reports associated with the study.
Facilities and Equipment
ESDI is a drug-free workplace located in a four-office suite appropriately equipped with
resources necessary for survey development, production, distribution, and storage, as well as
for the analysis of survey and case study data. Our technical capability includes four
networked micro-computers with sophisticated word processing, statistical, graphics, and
desktop publishing software, Hewlett-Packard LaserJet printers, copy machine, fax
machine, collator, comb-binder, desks, tables, and chairs necessary to support the staff.
ESDI has conference space available if needed. Additionally ESDI has immediate access to
optical scanning equipment and item analysis programs for scanning answer documents and
conducting traditional, Rasch and three-parameter model item analyses. Our in-house
computer capabilities easily will accommodate the analyses planned for this project and can
include parametric and nonparametric statistics, analysis of covariance, and multivariate
analyses.
Evaluation Systems Design, Inc.
11