An Evaluation of the
Massachusetts Pilot Teachers’
SEI Endorsement Course
Erin Haynes, American Institutes for Research
Diana August, American Institutes for Research
Christine Paulsen, Concord Evaluation Group
DECEMBER 2012
December 2012
An Evaluation of the Massachusetts Pilot
Teachers’ SEI Endorsement Course
December 19, 2012
Erin Haynes
Researcher, American Institutes for Research
Diane August
Managing Researcher, American Institutes for Research
Christine Paulsen
Principal Research Scientist, Concord Evaluation Group
1000 Thomas Jefferson Street
Washington, DC 20007
Phone: (202) 403-5000
Fax: (202) 403-5001
Copyright © 2012 American Institutes for Research. All rights reserved.
3280_12/19
Contents
Executive Summary ........................................................................................................................ 1
Description of Evaluation ........................................................................................................... 1
Research Questions ..................................................................................................................... 1
Recommendations ....................................................................................................................... 2
Introduction ..................................................................................................................................... 8
Research Questions ....................................................................................................................... 10
Quality of Delivery ................................................................................................................... 10
Fidelity of Implementation ....................................................................................................... 10
Coherence ................................................................................................................................. 10
Teacher Outcomes .................................................................................................................... 10
Research Context .......................................................................................................................... 11
Data Collection Methods .............................................................................................................. 13
Activity 1: Course Observations ............................................................................................... 13
Activity 2: Instructor Interviews ............................................................................................... 14
Activity 3: Teacher Surveys...................................................................................................... 14
Findings and Recommendations ................................................................................................... 16
Quality of Delivery ................................................................................................................... 16
Fidelity of Implementation ....................................................................................................... 30
Coherence ................................................................................................................................. 31
Teacher Outcomes .................................................................................................................... 34
Next Steps ..................................................................................................................................... 38
References ..................................................................................................................................... 39
Appendix A: Data Collection Instruments .................................................................................... 43
Face-to-Face Session Observation Protocol ............................................................................. 43
Online Session Observation Protocol ....................................................................................... 51
Instructor Interview Protocol .................................................................................................... 57
Participant Focus Group Protocol ............................................................................................. 60
Participant Survey ..................................................................................................................... 61
Appendix B: Participant Survey Results....................................................................................... 72
Tables
Table 1. RETELL Pilot Sessions .................................................................................................. 12
Table 2. Research Questions Addressed by Each Data Collection Activity................................. 13
Table 3. RETELL Pilot Session Observations .............................................................................. 14
Table 4. Usefulness Ratings for Readings Assigned in the RETELL Course .............................. 18
Table 5. Topics for Which RETELL Participants Reported Greater Understanding .................. 35
Table 6. Topics in Which RETELL Participants Need Additional Information ......................... 35
Figures
Figure 1. RETELL Course Participants’ Responses to Survey Questions About Course
Readings ........................................................................................................................................ 17
Figure 2. RETELL Course Participants’ Responses to Survey Questions About Instructor
Quality........................................................................................................................................... 21
Figure 3. RETELL Course Participants’ Responses to Survey Questions About Online
Sessions ......................................................................................................................................... 22
Figure 4. RETELL Course Participants’ Responses to Survey Question About Course Pace .... 25
Figure 5. Average Time Spent on Activities During RETELL Course Face-to-Face Sessions ... 26
Figure 6. RETELL Course Participants’ Responses to Survey Questions About Time
Allocation ...................................................................................................................................... 27
Figure 7. RETELL Course Participants’ Response to Survey Question About Course
Instructions .................................................................................................................................... 28
Figure 8. RETELL Course Participants’ Response to Survey Question About Time
Allocated to Discussions of Classroom Strategy Application ...................................................... 33
Figure 9. RETELL Course Participants’ Response to Survey Question About Usefulness of
Online Illustrations of Strategy Application ................................................................................. 34
Executive Summary
Description of Evaluation
In order to improve instruction for English language learners (ELLs) in Massachusetts, the
Massachusetts Department of Elementary and Secondary Education (ESE) has introduced the
Rethinking Equity and Teaching of English Language Learners (RETELL) initiative. Under this
initiative, all core academic teachers in the state will be required to complete comprehensive
professional development (PD) in sheltered English instruction (SEI) methods by July 1, 2016.
For simplicity in this report, this PD will be referred to as the RETELL course pilot, although the
course will be called the Teacher SEI Endorsement Course in the future.
Implementation of the full program will begin across the state in February 2013. In order to
ensure that this program is ready for complete rollout by February, a pilot of the PD was
implemented during fall 2012, ending in November. The American Institutes for Research
(AIR), in collaboration with Concord Evaluation Group (CEG), has completed an evaluation of
the pilot to provide feedback on the quality of the RETELL PD and RETELL instructors’ fidelity
to course materials in time to make any necessary changes to the program before the February
rollout. A preliminary report of the findings of the evaluation was submitted to ESE in
November. This document constitutes a full report of the RETELL pilot evaluation.
Research Questions
The RETELL pilot evaluation report responds to nine research questions, grouped into four
critical categories for success of the PD program: quality of delivery, fidelity of implementation,
coherence, and teacher outcomes. The research questions are as follows.
Quality of Delivery
1. What is the overall quality of delivery of the online and face-to-face courses, in terms of
content, process, and context, as well as pace, allocation of time, and clarity of PD
instruction? How can the overall quality of delivery of the online and face-to-face
courses be improved?
2. In what ways could the support for RETELL course instructors be improved to enhance
their instruction?
3. What ongoing feedback loops can be established to ensure the quality of the RETELL
instruction and course delivery?
Fidelity of Implementation
4. What is the level of implementation of RETELL components during the online and faceto-face courses? How can the level of implementation of RETELL components during
the online and face-to-face courses be improved?
5. What ongoing feedback loops can be established to ensure the fidelity of the RETELL
instruction and course delivery?
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—1
Coherence
6. What is the level of coherence of the RETELL sessions?
7. What is the level of coherence of the course with what teachers experience in their
educational contexts?
Teacher Outcomes
8. How could the RETELL course better contribute to key understandings of teachers of
ELL students in regular classroom settings, in terms of both the content of the course and
the course delivery?
9. What additional information do teachers need to be successful in the RETELL course?
Recommendations
The following is the complete list of recommendations made in this report in each of the four
identified critical areas: quality of delivery, fidelity of implementation, coherence, and teacher
outcomes.
Quality of Delivery
Quality of Content

Goals, objectives, and expected outcomes appear in the course syllabi and presentation
slides. Participants seem capable of assimilating the information from course materials.
Therefore, no change is recommended.

No changes are recommended to the topics covered in the course syllabi. If changes are
made to the syllabi for other purposes (e.g., to ensure that participants have more
opportunities to engage with specific strategies), make sure that key areas of ELL
instruction (see description and sources in full report) are retained.

Teacher buy-in is essential in order to motivate teachers to adopt practice, especially
when the PD does not already align with the beliefs of the teacher (Kubitskey & Fishman,
2005). Because some recipients of the RETELL PD may not be convinced that they can
successfully impart content to students who are not proficient in English, consider adding
a case study or video to the first session of the course that illustrates how sheltering
instruction can help ELLs access content at the same time as they acquire English
language skills.

To reduce printing burdens for teachers, provide a CD or printed copies of the required
readings.

Reduce the reading load by carefully selecting readings that are the most relevant to the
successful implementation of SEI and that are clearly written. Consider readings that
provide practical advice about helping ELLs meet high content standards.
Quality of Process

The pilot instructors expressed knowledge of the course participants’ backgrounds. This
knowledge is important to the quality of process, and instructors of the main course
should learn about their participants’ backgrounds and contexts. Consider including some
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—2
short activities in the course that explore what participants already know about various
topics and what they would like to learn. Provide sufficient flexibility for instructors to be
responsive to participants’ needs.

Identify instructional strategies that are aligned with each course topic. Have PD
instructors model these strategies using elements of model lessons and give participants
an opportunity to apply what they have learned in a new context. Consider providing
participants with a rubric to record each strategy they use, with information about the
classroom context, how the strategy worked, and recommendations for adjustments.

Ask participants to briefly summarize what they learned at the end of each session, either
as a whole class, in small groups or partners, or as a warm-up during a subsequent online
session. As part of their summary, ask them to link the course materials to specific
experiences or examples from their own educational contexts, or describe methods they
have used related to the session.

Pilot instructors appeared to be skilled at leading collaborative discussions and engaging
participants. Ensure that instructors in future RETELL courses are prepared to use good
pedagogical practice to facilitate collaboration and encourage participation.

Consider developing professional learning communities (PLCs) as part of the RETELL
course so that collaboration among participants can extend beyond the duration of the
course. A review of relevant research on PLCs by Vescio et al. (2008) finds that welldeveloped PLCs have positive effects for both teacher practice and student outcomes.

Consider methods for increasing discussions and interactions online. Some options
include the following:
o Establish weekly “office hours” during which the instructor is online and available for
chatting (consider making attendance at some office hours a course requirement).
o Create a blog or wiki in which participants can report on the implementation of
strategies or share ideas for teaching that came to them as a consequence of a reading
or session.
o Group participants and provide tasks that require them to check in with each other
regularly, either in person or via the online platform.
o Set up a collaborative project online to which all participants are required to
contribute. Ideally, such a project would be best facilitated by the online platform
(such as posting videos of instruction and eliciting constructive feedback or
conducting an analysis of student work using graphic tools; see Little et al., 2003).

Encourage instructors to actively monitor online activity and send reminders and prompts
to participants as needed.
Quality of Context

It will be essential to select highly qualified instructors for a course that is not heavily
scripted. Unless activities are added and scripted, only RETELL instructors who are
prepared to develop and incorporate their own activities should be selected.

Consider developing and providing answer guides for questions posed in the course
materials.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—3
Pace of Course

Carefully monitor the semester-long course to make sure that the pace is not too fast. For
example, the schedule should allow reasonable time between sessions for participants to
complete readings and assignments, and to test the methods and instructional strategies
they are learning.
Allocation of Time in Course

Because participants found course time allocations appropriate for lectures and in-class
interactions with colleagues (including small-group work and whole-class discussion),
maintain these activities at their current levels.

In order to address instructor concerns and participant feelings that not enough time was
devoted to strategies, consider replacing some of the small-group discussions with smallgroup strategy practice; whole-class discussions also could be used to discuss how theory
is realized in practice and how to apply instructional strategies successfully.

Replace significant out-of-class assignments with activities that require collaboration
with colleagues, particularly on the practice of strategies. Be sure to incorporate enough
time between assignments to allow participants to engage in these activities
meaningfully.

To address time allocation during the self-paced, self-directed online sessions, consider
creating an activity that helps participants understand how they learn best. Provide tips
for time management as part of the introduction to the online sessions.
Clarity of Instruction

Consider including some short activities in both the face-to-face and the online portions
of the course that explore participants’ conceptions and misconceptions to ensure that
they are addressed.

For the online sessions, provide a single list of required readings, materials, and activities
for each session with hyperlinks to each.

Include summaries of readings and information about possible questions or concerns in
online materials.

Provide a list of assignments and readings ahead of the course so that participants have
additional time to complete the work.
Support for Course Instructors

Make sure that course classrooms are easily accessible, can be accessed ahead of each
session for setup, and have Internet access. If possible, designate a person at each site to
ensure that these logistics are taken care of.

Provide participant contact information to instructors as early as possible (ideally, at least
two weeks before the first session).

Establish a clear policy about material distribution and provide it to all instructors.
Include information about which materials and services ESE provides and which it does
not.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—4
Establishment of Feedback Loops

Provide information to instructors about which aspects of the RETELL course will need
to be consistent across the state, such as participant grading or evaluation information.

Consider providing a training session for RETELL course instructors during which they
can practice elements of the PD with each other before delivering it to course
participants.

Establish a consistent feedback system for all RETELL PD courses. The feedback system
might consist of an observation by an ESE official, formal participant surveys (e.g., short
surveys delivered via Survey Monkey after each class), or an outside evaluation of
teacher products (e.g., lesson plans) or outcomes.
Fidelity of Implementation

Strategically group participants with similar backgrounds and educational contexts into
the same courses so that instructors can focus on their needs efficiently, increasing the
likelihood of covering all course content. As an added advantage, grouping participants
from similar educational contexts can facilitate the formation of PLCs (see, e.g.,
Newmann & Associates, 1996).

Check online sessions to ensure that they cover all topics and assignments in the syllabus,
especially if there have been changes to the sessions.

Provide instructors with information about how long each activity should take.

Facilitate regular meetings among instructors (by region and/or participant type—e.g.,
secondary STEM) throughout the course semester to discuss RETELL delivery and
fidelity to the syllabus. These meetings would also be a good opportunity for ESE to
collect feedback from instructors about recommended changes.

Make necessary changes to the course between, not during, rollout semesters.
Coherence
Level of Coherence of the RETELL Sessions

Revise course materials to provide explicit connections between topics covered in each
session, including revising the online materials to include explicit connections to previous
and upcoming sessions.

Explain to instructors the importance of communicating expectations for the online
course during the face-to-face sessions, even if there is only a little time to do so.

Consider implementing the following instructor-recommended changes:
o Incorporate session 4 into session 5.
o Consolidate information about academic language into a single session.
o Incorporate information about the Common Core State Standards throughout the
sessions rather than introducing it in session 15.
o The topic of assessment should come at the end of the course, after reading and
writing have been covered.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—5
o Provide more grade-level differentiation in Module C [within the secondary courses].
Level of Coherence With What Teachers Experience in Their Educational Contexts

Connections to participants’ educational contexts are crucial. Ensure that RETELL
course instructors are as prepared as the pilot instructors to connect material to real-world
contexts, use stories to link theory to practice, and lead participants in discussions of how
strategies would work in their educational contexts.

Consider incorporating the practice of instructional strategies into small-group work and
whole-class discussions in order to increase its impact.

Update the online materials to make better connections to participants’ educational
contexts. In each session, include
o Stories or videos that model practical strategies
o Discussions of possible challenges that teachers might face in applying strategies
o Prompts that encourage participants to post and discuss examples of how strategies
would work in their educational contexts
Teacher Outcomes

Conduct a brief survey at the outset of each course to determine areas in which
participants are already well versed and areas where they need additional instruction.

Consider providing (additional) instruction in the following areas, as determined by
participant needs:
o Creating lessons and practical strategies for the classroom
o Effective practice with ELLs who do not speak Spanish
o Standardized assessments and ELLs
o ELLs with special needs
o Grade-specific strategies
o Supporting language development versus helping students access content
o Students with interrupted formal education (SIFE)
o Using data

Fishman et al. (2003) propose an iterative PD research design model, in which feedback
about teacher outcomes is incorporated into future PD sessions; a similar model has been
proposed by Kubitskey & Fishman (2007). The current evaluation gathers self-reported
data from teachers, but it also is important to measure teachers’ skills and knowledge.
The following formative and summative evaluation activities are recommended both to
provide teachers with information about their progress in the course and to improve
future iterations of the course:
o Develop and incorporate protocols for assessing participants’ plans for and reports of
how they implemented course strategies.
o Develop and conduct pre- and post-course assessments of participants’ knowledge
about ELLs and ELL instruction.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—6
o Include a classroom observation component in the course. Teachers might observe
each other and provide peer-to-peer feedback; a school or district ESL coordinator or
coach might observe participants and provide feedback as part of the RETELL
course; or RETELL course instructors might observe each participant teaching.
Observations are recommended for teachers in high- and medium-incidence schools;
they are not recommended if participants do not have ELLs in their classrooms at the
time of the RETELL PD.

Make sure that instructors can provide timely feedback to participants using the Moodle
online platform. For instructors who are unfamiliar with Moodle, offer training on how to
use it effectively.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—7
[This page intentionally left blank.]
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—8
Introduction
English language learner (ELL) enrollment in PK–12 grew nationwide by more than 51 percent
between the 1998–99 school year and the 2008–09 school year, while the growth of total student
enrollment increased by only 7 percent (NCELA, 2011). This growth trend is reflected in
Massachusetts, where the proportion of ELLs in public schools has grown 57 percent since 2000,
while enrollment of English-proficient students has remained steady; by 2021, it is estimated that
20 percent of all Massachusetts students will be ELLs (Massachusetts Department of Elementary
and Secondary Education [ESE], 2009; ESE, 2012). Furthermore, the number of districts in
Massachusetts enrolling at least one ELL has nearly doubled between 2000 and 2011; in 2011,
324 of the state’s 400 districts had at least one ELL, and 63 districts had 100 or more ELLs
(ESE, 2012). In light of this growth, all Massachusetts educators can expect to work with ELLs
in their careers.
However, there is a significant achievement gap between ELLs and their English-proficient peers
in the state. Results from the Massachusetts Comprehensive Assessment System (MCAS) for
English language arts and mathematics in Grades 4, 7, 8, and 10 reveal that in both subject areas
at all grade levels, the proportion of ELLs scoring at Warning (the lowest rating for Grades 4‒8)
or Failing (the lowest rating for Grade 10) was significantly higher than the proportion of
English-proficient students receiving those ratings (Owens, 2010). Further, ELLs in
Massachusetts are more likely than English-proficient students to repeat a grade and were 25
percent more likely to be suspended, according to 2008 data, up from 16 percent more likely in
2006 (Owens, 2010).
In order to improve instruction for ELLs in Massachusetts and ultimately improve their
achievement, ESE has introduced the Rethinking Equity and Teaching of English Language
Learners (RETELL) initiative. Under this initiative, all core academic teachers in the state will
be required to complete comprehensive professional development (PD) in sheltered English
instruction (SEI) methods by July 1, 2016.1 This PD will incorporate 16 online and face-to-face
sessions during the course of an academic semester. The PD’s main purpose is to prepare
teachers to shelter their instruction by increasing their knowledge of ELL demographics and
cultural backgrounds, second-language acquisition theory, literacy, English language
development standards and assessments, and effective practices in ELL instruction.
Implementation of the full program will begin across the state in February 2013. In order to
ensure that this program is ready for complete rollout by February, a pilot of the PD was
implemented at five sites for 8.5–13.5 weeks during fall 2012 (the five sites were Springfield,
Boston, Worcester, Somerville, and Lowell). The pilot was completed at all five sites by the end
of November. The American Institutes for Research (AIR), in collaboration with Concord
Evaluation Group (CEG), completed an evaluation of the pilot at four of the sites (Boston,
Worcester, Somerville, and Lowell).2 The purpose of the evaluation was to provide feedback on
the quality of the RETELL PD and RETELL instructors’ fidelity to course materials in time to make
any necessary changes to the program before the February rollout. A preliminary report of the
findings of the evaluation was submitted to ESE in November. This document constitutes a full
report of the RETELL pilot evaluation.
1
For simplicity in this report, this PD will be referred to as the RETELL course pilot, although the course will be
called the Teacher SEI Endorsement Course in the future.
2
Note that the Springfield site PD is not evaluated as part of this report because it received its PD earlier than the
other sites, and because it served as a test case for the pilot PD materials. One course observation was conducted in
Springfield, but the results of that observation were provided to ESE separately and are not included here.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—9
Research Questions
The RETELL pilot evaluation report responds to nine research questions, grouped into four
critical categories for success of the PD program: quality of delivery, fidelity of implementation,
coherence, and teacher outcomes. These categories will be described in greater detail in the
Findings and Recommendations section. The research questions are provided here:
Quality of Delivery
1. What is the overall quality of delivery of the online and face-to-face courses, in terms of
content, process, and context, as well as pace, allocation of time, and clarity of PD
instruction? How can the overall quality of delivery of the online and face-to-face
courses be improved?
2. In what ways could the support for RETELL course instructors be improved to enhance
their instruction?
3. What ongoing feedback loops can be established to ensure the quality of the RETELL
instruction and course delivery?
Fidelity of Implementation
4. What is the level of implementation of RETELL components during the online and faceto-face courses? How can the level of implementation of RETELL components during
the online and face-to-face courses be improved?
5. What ongoing feedback loops can be established to ensure the fidelity of the RETELL
instruction and course delivery?
Coherence
6. What is the level of coherence of the RETELL sessions?
7. What is the level of coherence of the course with what teachers experience in their
educational contexts?
Teacher Outcomes
8. How could the RETELL course better contribute to key understandings of teachers of
ELL students in regular classroom settings, in terms of both the content of the course and
the course delivery?
9. What additional information do teachers need to be successful in the RETELL course?
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—10
Research Context
Data to address the nine research questions were collected in four pilot districts: Boston,
Worcester, Somerville, and Lowell. Seven instructors provided PD at these sites; each site had
two instructors, but one instructor taught at two of the sites (Boston and Worcester). These
instructors were selected for their expertise in English as a second language (ESL) and
experience delivering PD to educators in Massachusetts. Three of the instructors reported
delivering the SEI category trainings, which were the previous PD effort aimed at preparing
teachers to work with ELLs. All the instructors had at least a master’s degree in ESL or a related
field, and all had provided PD to teachers in Massachusetts districts.
Instructors spent one week with the developers of the RETELL course curriculum in the summer
before the pilot implementation to review the course content. This week was structured as a
working group, during which the instructors gave input and feedback about the course. One
instructor for the four evaluation sites also taught the course at Springfield, beginning two
months prior to the other sites. She made some changes to the course materials on the basis of
her experiences, in consultation with ESE. Some of these changes were provided to the other
instructors during the delivery period at the other sites. The RETELL pilot course was not
scripted, so all the instructors had latitude in its delivery, and they all provided their own
activities. One instructor said, “We were instructed to just go ahead of them [the trainees] and
pave the sidewalk as they walked on it.”
The instructors delivered 16 sessions, divided among three modules. The first module develops
participants’ knowledge of ELLs, including ELL demographics, the diversity within populations
of ELLs, and their educational context. It emphasizes the importance of integrating students’
background knowledge and skills into instruction, while building the foundational skills they
need to succeed in a U.S. academic setting. The second module introduces the theoretical
principles of second-language acquisition and their impact on the instruction of second-language
learners. The third module builds on the second module principles with a focus on teaching ELLs
academic language and enabling them to master content. This module illustrates how curriculum
can be scaffolded for ELLs at different levels of English-language proficiency and for different
content areas.
Ten of the courses were delivered face-to-face, including the first and last sessions, and six of the
sessions were delivered online via the Moodle platform. All face-to-face sessions were three
hours long, except the first and last sessions, which were two hours each. (Table 1 provides
information about each session’s delivery type and length, by module.) PD participants were
expected to complete two or three readings for each session, including the online sessions. They
also were asked to complete several assignments, which fell into four categories: journal writing
(nine entries); critical analyses of challenges in teaching ELLs (five analyses); adaptations of
lesson plans for ELLs (11 lesson plans); and short formal papers (three papers). The course was
delivered over 8.5–10 weeks at each site (it was delivered over 13.5 weeks in Springfield, but
that site is not included in the evaluation).
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—11
Table 1. RETELL Pilot Sessions
Module
A
B
C
Session
Type
Length
1
Face-to-face
2 hours
2
Online
varies
3
Face-to-face
3 hours
4
Online
varies
5
Face-to-face
3 hours
6
Face-to-face
3 hours
7
Online
varies
8
Online
varies
9
Face-to-face
3 hours
10
Online
varies
11
Face-to-face
3 hours
12
Online
varies
13
Face-to-face
3 hours
14
Face-to-face
3 hours
15
Face-to-face
3 hours
16
Face-to-face
2 hours
Participants were content area teachers in the respective districts. The Boston course was
intended for elementary teachers, Worcester and Somerville were for secondary English
language arts/humanities teachers, and the Lowell course was for secondary STEM teachers. A
total of 103 participants were enrolled in the four sites (25 in Boston, 28 in Lowell, 24 in
Somerville, and 26 in Worcester), but participation dwindled at all four sites during the course.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—12
Data Collection Methods
Research activities to address the nine research questions included course observations,
interviews with instructors, and participant surveys. Each of these research activities provided
information for a subset of the research questions, as presented in Table 2. More detailed
descriptions of the activities follow.
Table 2. Research Questions Addressed
by Each Data Collection Activity
Data Collection
Activity
Research
Questions
Addressed
Course observations
1, 4, 6
Instructor interviews
2, 3, 5
Participant surveys
1, 7, 8, 9
Activity 1: Course Observations
Fourteen course observations were completed for this evaluation. Nine were completed during
face-to-face sessions, and five during online sessions. The sites, sessions, and types are presented
in Table 3. Sessions were selected for observation across the four sites with the goal of
maximizing the variety of sessions, session types (online or face-to-face), and participant type
(elementary teachers, secondary English language arts/humanities teachers, or secondary STEM
teachers); as a secondary consideration, sessions were selected that did not have an observer
from ESE in order to make sure that all pilot sessions were observed. Sessions earlier in the
sequence occurred before the evaluation period and could not be observed. Likewise, Session 16
could not be observed at any of the sites because observations had to be completed by November
16 in order to finalize analyses and provide timely feedback to ESE. The final online and face-toface observation protocols are in Appendix A.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—13
Table 3. RETELL Pilot Session Observations
Site
Boston
Worcester
Lowell
Somerville
Date
Session
Type
Teachers
October 13
5
Face-to-face
Elementary
October 13
6
Face-to-face
Elementary
October 19
8
Online
Elementary
November 3
13
Face-to-face
Elementary
November 3
14
Face-to-face
Elementary
October 9
6
Face-to-face
ELA/humanities
October 16
9
Face-to-face
ELA/humanities
November 15
2
Online
STEM
October 12
7
Online
STEM
November 7
14
Face-to-face
STEM
November 14
15
Face-to-face
STEM
October 23
10
Online
ELA/humanities
October 26
11
Face-to-face
ELA/humanities
October 30
12
Online
ELA/humanities
Activity 2: Instructor Interviews
Interviews were completed by phone with six of the seven RETELL pilot instructors, including
the two co-instructors at the Lowell site, the two co-instructors at the Somerville site, the primary
instructor at the Boston and Worcester sites (the same person), and the co-instructor at the
Worcester site. Interviews were designed to elicit the instructors’ perspectives about the structure
and quality of content of the RETELL PD, information about the support they needed and the
feedback they received, and their perspectives on the outcomes of the PD for participants. The
instructor interview protocol is in Appendix A.
Activity 3: Teacher Surveys
A survey was adapted from a midcourse survey that had been designed and administered by
ESE. During the first week of the evaluation period, volunteers from the Springfield PD site
were asked to form a focus group to discuss the survey questions. A total of five participants
were asked about each survey question in order to ensure that the questions were valid measures
of the constructs (quality of delivery, course coherence, and teacher outcomes).3 The survey was
revised to reflect their comments before it was administered to all sites. The focus group protocol
and final survey are included in Appendix A.
Participants at all four evaluation sites were asked to complete the survey during the week of
November 12, during their second-to-last session. A total of 57 participants completed the
survey, including 18 participants from Somerville, 16 participants from Lowell, and a combined
total of 23 participants from Boston and Worcester (these two sites had the same instructor, who
3
Participants from the Springfield site formed the focus group because they had already completed most of the
training early in the evaluation period.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—14
combined the completed surveys, so no differentiation is possible). Compiled survey results are
in Appendix B.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—15
Findings and Recommendations
This section presents the findings of the evaluation and provides recommendations to improve
the RETELL course. The findings and recommendations are grouped by category: quality of
delivery, fidelity of implementation, coherence, and teacher outcomes.
Quality of Delivery
Quality of delivery, examined through research questions 1, 2, and 3, encompasses several
factors, including content, process, context, pace, allocation of time, and clarity of PD instruction.
The first three factors are described in the draft Massachusetts Standards for Professional
Development (SPD). Also included in the analysis of quality of delivery are ways in which the
support for RETELL course instructors could be improved to enhance their PD delivery and
feedback loops that could be established to ensure the quality of the PD instruction and course
delivery.
Quality of Content
Quality of content is examined through research question 1. According to the Massachusetts
SPD, high-quality PD has clear goals and objectives that are relevant to participant outcomes and
aligned with state priorities. RETELL course goals and objectives were included in the syllabi
and presentation slides, so despite the fact that instructors at all four sites were rarely observed
articulating goals, objectives, or expected outcomes, a large majority (86 percent) of participants
at all four sites indicated that instructors always or frequently made this information available,
presumably as part of course materials.
It is beyond the scope of this evaluation to examine the alignment of the RETELL course goals
with state priorities. An analysis of the course syllabi reveals, however, that the RETELL course
materials cover critical aspects of ELL instruction, and they address key areas of concern for
Massachusetts educators (such as serving subgroups of ELLs), as reported in Perez-Selles,
Cazabon, & Mello (2011). For example, the first module develops participants’ knowledge of
ELLs, including ELL demographics, the diversity within populations of ELLs, and their
educational context. It emphasizes the importance of integrating students’ background
knowledge and skills into instruction, while building the foundational skills they need to succeed
in a U.S. academic setting (cf. Lee, Maerten-Rivera, Penfield, LeRoy, & Secada, 2008). The
second module introduces the theoretical principles of second-language acquisition and their
impact on the instruction of second-language learners. The third module builds on the second
module principles with a focus on teaching ELLs academic language and enabling them to
master content as specified in the Common Core State Standards (National Governors
Association Center for Best Practices & Council of Chief State School Officers, 2010). The third
module illustrates how curriculum can be scaffolded for ELLs at different levels of English
language proficiency and for different content areas (cf. Saunders, 1999; Saunders &
Goldenberg, 1999). The last module is of particular importance because, according to the syllabi,
it emphasizes strategies that have been found to be effective with ELLs in high-quality research
studies, such as providing multicomponent linguistic support for ELLs, including explicit
vocabulary instruction, peer assistance, and the use of visual support (Carlo et al., 2004; Franken
& Haslett, 1999; Vaughn et al., 2009; see also sources cited in Gersten et al., 2007), as well as
providing differentiated instruction to capitalize on ELLs’ strengths (cf. August & Shanahan,
2010).
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—16
The RETELL course instructors were generally pleased with the scope of the curriculum and the
division into three modules. One instructor, however, expressed concern over course buy-in; she
did not feel that the course sufficiently established why SEI is important for all participants.
Overall, participants in the PD were not observed disputing the factual material included in the
course or explicitly questioning the usefulness of the course. They were able to complete
activities correctly (though online activities often were completed late). It was sometimes
unclear, however, whether participants had completed the readings. Though they sometimes
referred to readings during face-to-face discussions or online comments, they almost never asked
or posted clarifying questions about the main ideas of the reading selections. In all sessions,
participants were observed complaining that the reading load was too high and that they did not
have money to print or make copies of the articles. Forty-nine percent of the participants who
completed surveys reported that they did not find the readings easy to understand. In addition,
some of the instructors expressed the opinion that the focus on the reading and theory was very
heavy, to the detriment of other activities like modeling strategies and experiential activities for
the participants.
Nonetheless, in surveys almost all participants agreed (77 percent) or strongly agreed (14
percent) that the readings were related to course content, and 80 percent agreed or strongly
agreed that the readings helped them better understand SEI. The majority also agreed (67
percent) or strongly agreed (6 percent) that the readings were applicable to instruction of ELLs.
See Figure 1.
Figure 1. RETELL Course Participants’ Responses to
Survey Questions About Course Readings
90%
77%
80%
67% 67%
70%
60%
50%
40%
26%
30%
20%
10%
5%
14% 13%
11%
9%
2%
6%
4%
0%
Strongly disagree
Disagree
Agree
Strongly agree
Readings related to course content
Readings led to better understanding of SEI
Readings were applicable to instruction of ELLs
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—17
This apparent disparity in participants’ reactions to the readings (on the one hand they found the
readings difficult and thought there were too many, but on the other hand they found the readings
useful) is likely due to the diversity of articles that they were asked to read. In order to explore
this issue further, participants were asked to rate individual course readings as either problematic
or useful. In recognition of the fact that they may not recall or have a reaction to some readings,
participants were asked to rate only the readings they found problematic or useful. A score was
assigned to each reading by subtracting the number of problematic ratings from the number of
useful ratings, presented as a fraction of the total number of readings. The score for each reading
is presented in Table 4, also provided as a percentage. A percentage above 50 percent indicates
that if a reading elicited a response, the response was more heavily positive. A percentage below
50 percent, on the other hand, indicates that if a reading elicited a response, the response was
more heavily negative.4
Table 4. Usefulness Ratings for Readings Assigned in the RETELL Course
Session
Reading
Useful/
Total
%
3
Harper & de Jong (2004)
33/39
85%
2
DeCapua & Marshall (2011)
29/37
78%
3
Colorin Colorado (2007)
27/37
73%
2
Robertson & Lafond (2008)
23/33
70%
7
Silbold (2011)
26/38
68%
1
ESE (2012)
21/31
68%
6
Calderon et al. (2005)
24/38
63%
DelliCarpini (2009)
20/32
63%
de Jong & Harper (2005)
20/32
63%
12
Gottlieb (2006)
23/37
62%
11
Misco & Castaneda (2008)
16/26
62%
Dilg (2003)
20/34
59%
12
Abedi (2002)
19/33
58%
15
WIDA (2012)
16/30
53%
9
Himmel (2012)
15/31
48%
15
Echevarria et al. (2004)
14/30
47%
14
Brisk et al. (2007)
14/32
44%
10
Coleman & Goldenberg (2010)
12/30
40%
7
Carlo et al. (2005)
12/30
40%
8
Echevarria & Graves (2011)
12/30
40%
5
Peregoy & Boyle (2008)
13/33
39%
9
Varela (2010)
9/25
36%
14
8
3
4
No interpretation is available for the number of responses that a reading elicited. Participants may have failed to
rate readings for a number of reasons, including that the reason was not memorable, that they did not read it, or that
they read it, but found it neither useful nor problematic. These options were not included in the survey in order to
reduce the burden of participating in the research.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—18
Session
Reading
Useful/
Total
%
5
Finegan (2004)
11/35
31%
13
Singhal (2005)
9/29
31%
Lems et al. (2010)
9/29
31%
13
Cohen (2007)
9/31
29%
10
Gottlieb et al. (2009)
9/33
27%
4
Brown (2000)
8/32
25%
4
Crawford (2004)
0/32
0%
4
Meyer (2009)
–1/31
–3%
Schleppegrell (2001)
–8/30
–27%
6
11
The following are recommendations to improve the quality of the RETELL course content on the
basis of the findings just described:

Goals, objectives, and expected outcomes appear in the course syllabi and presentation
slides. Participants seem capable of assimilating the information from course materials.
Therefore, no change is recommended.

No changes are recommended to the topics covered in the course syllabi. If changes are
made to the syllabi for other purposes (e.g., to ensure that participants have more
opportunities to engage with specific strategies), make sure that key areas of ELL
instruction (see description and sources discussed earlier) are retained.

Teacher buy-in is essential in order to motivate teachers to adopt practice, especially
when the PD does not already align with the beliefs of the teacher (Kubitskey & Fishman,
2005). Because some recipients of the RETELL PD may not be convinced that they can
successfully impart content to students who are not proficient in English, consider adding
a case study or video to the first session of the course that illustrates how sheltering
instruction can help ELLs access content at the same time as they acquire English
language skills.

To reduce printing burdens for teachers, provide a CD or printed copies of the required
readings.

Reduce the reading load by carefully selecting readings that are the most relevant to the
successful implementation of SEI and that are clearly written. Consider readings that
provide practical advice about helping ELLs meet high content standards.
Quality of Process
Quality of process is examined through research question 1. According to the Massachusetts
SPD, high-quality PD promotes collaboration among educators and provides opportunities for
participants to apply what they are learning to their own educational context. An important first
step in helping participants tie their coursework to their own contexts is learning what those
contexts are. Indeed, during interviews, instructors expressed knowledge of participants’
backgrounds and their general knowledge about ELL instruction. But the instructors were almost
never observed asking participants what they would like to learn about a topic, or what they
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—19
already knew. (It is possible that these conversations happened during sessions that were not
being observed.)
The Massachusetts SPD also states that high-quality PD models good pedagogical practices,
including being problem-centered, capitalizing on learners’ experiences, and allowing choice and
self-direction. It is especially important to link theory presented in PD to practice (e.g., DarlingHammond & McLaughlin, 1995; Garet, Porter, Desimone, Birman, & Yook, 2001; Kubitskey,
2006; Kubitskey & Fishman, 2006; Penuel, Fishman, Yamaguchi, & Gallagher, 2007). Most
observed sessions, however—with only a few exceptions—did not focus heavily on strategies.
Those sessions that did, spent little time reviewing them in depth. Though participants
sometimes asked or posted questions about how something could be used in the classroom, or
made suggestions for applying ideas in the classroom, they seldom were given clear steps they
could apply to improve their skills. The instructors made similar observations; some expressed a
desire for more time to model strategies and discuss practical strategies rather than focusing on
theory. One stated, “I remember one of the courses on oral linguistics.… It was so overwhelming
and so technical and unnecessary.… I think heavy theory about linguistics is just a lot for
teachers. They’re practitioners and they’re practicing right then. You do have to wrap it up with
research and theory to convince people along with explaining it to people but I also think if it’s
way above people’s heads, not their field specifically, and it’s hard.”
It also was observed that instructors did not have sufficient time to give overviews when
introducing new topics and activities, or summaries at the end of discussions, presentations, and
activities (though overviews and summaries were almost always provided in online sessions).
Overviews and summaries could have been a good place to effectively link theory to practice
during PD sessions.
On the other hand, many examples of good pedagogical practices were observed. At all sites,
instructors frequently asked whether there were questions and then waited for participants to
respond. They used phrases like “good point,” “I can understand your interpretation of that,”
“great question,” and “thanks for bringing that up.” They also frequently asked participants to
“think about this,” or “what would you do if…?” et cetera. In face-to-face sessions, participants
appeared actively engaged during presentations (though there were occasional instances of
people checking e-mail or engaged in side conversations). Most participants reported that
instructors facilitated discussions and responded to questions well or very well (82 percent and
86 percent, respectively). See Figure 2.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—20
Figure 2. RETELL Course Participants’ Responses to
Survey Questions About Instructor Quality
50%
45%
40%
42%
42%
44%
40%
35%
30%
25%
20%
16%
15%
12%
10%
5%
2% 2%
0% 0%
0%
Poorly
Adequately
Well
Very well
Did not observe
How well the instructor facilitated discussions
How well the instructor responded to questions
In addition, in face-to-face sessions, instructors frequently promoted collaboration. During
whole-group discussions, participants answered questions from other participants and
commented on each other’s statements. Participants largely felt that interactions with colleagues
during face-to-face sessions were pretty relevant (30 percent) or very relevant (58 percent). Their
feelings about interactions are supported by research studies by Garet et al. (2001) and Kubitskey
& Fishman (2007), who find that activities that encourage professional communication enhance
teachers’ knowledge, skills, and instructional practices. Other research indicates that
collaboration is important both in terms of implementation of new skills (Bryk & Schneider,
2002) and in terms of giving educators an opportunity to discuss and resolve issues of practice
and initiate broader systemic changes in their schools (Frank, Zhao, & Borman, 2004; Lieberman
& McLaughlin, 1992; Little, 1993; Penuel et al., 2007).
The online portion of the course, however, was less engaging and collaborative. Though
participants had the opportunity to post questions and answer other participants’ questions or
comment on what they had said, few took this opportunity. (There was one exception in which a
discussion forum had multiple responses and some participants had an online dialogue going.)
Some participants were observed to be highly engaged in posting online, but many did not post
anything, even to fulfill a session requirement. In some online sessions, the instructors did not
participate in the discussion forums by commenting on any posts, nor did they utilize the news
feature. One instructor expressed frustration with using the online platform to generate
discussion. She had difficulty using it and also was disappointed in the quality of posts that
participants were generating. Overall, participants felt that interactions with colleagues during
online sessions were not relevant (43 percent) or only a little relevant (27 percent), and most
participants felt that the online portion of the course was not relevant (25 percent) or only a little
relevant (39 percent). Only 9 percent of participants found it very relevant. See Figure 3.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—21
Figure 3. RETELL Course Participants’ Responses to
Survey Questions About Online Sessions
45%
43%
39%
40%
35%
30%
25%
28%
27%
25%
20%
20%
15%
9%
10%
5%
5%
5%
0%
0%
Not relevant
A little relevant Pretty relevant
Very relevant
Relevance of online interactions with RETELL participants
Did not occur
Relevance of the online portion of RETELL
The following are recommendations to improve the quality of the RETELL course process on the
basis of the findings just described:

The pilot instructors expressed knowledge of the course participants’ backgrounds. This
knowledge is important to the quality of process, and instructors of the main course
should learn about their participants’ backgrounds and contexts. Consider including some
short activities in the course that explore what participants already know about various
topics and what they would like to learn. Provide sufficient flexibility for instructors to be
responsive to participants’ needs.

Identify instructional strategies that are aligned with each course topic. Have PD
instructors model these strategies using elements of model lessons and give participants
an opportunity to apply what they have learned in a new context. Consider providing
participants with a rubric to record each strategy they use, with information about the
classroom context, how the strategy worked, and recommendations for adjustments.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—22

Ask participants to briefly summarize what they learned at the end of each session, either
as a whole class, in small groups or partners, or as a warm-up during a subsequent online
session. As part of their summary, ask them to link the course materials to specific
experiences or examples from their own educational contexts, or describe methods they
have used related to the session.

Pilot instructors appeared to be skilled at leading collaborative discussions and engaging
participants. Ensure that instructors in future RETELL courses are prepared to use good
pedagogical practice to facilitate collaboration and encourage participation.

Consider developing professional learning communities (PLCs) as part of the RETELL
course so that collaboration among participants can extend beyond the duration of the
course. A review of relevant research on PLCs by Vescio et al. (2008) finds that welldeveloped PLCs have positive effects for both teacher practice and student outcomes.

Consider methods for increasing discussions and interactions online. Some options
include the following:
o Establish weekly “office hours” during which the instructor is online and available for
chatting (consider making attendance at some office hours a course requirement).
o Create a blog or wiki in which participants can report on the implementation of
strategies or share ideas for teaching that came to them as a consequence of a reading
or session.
o Group participants and provide tasks that require them to check in with each other
regularly, either in person or via the online platform.
o Set up a collaborative project online to which all participants are required to
contribute. Ideally, such a project would be best facilitated by the online platform
(such as posting videos of instruction and eliciting constructive feedback or
conducting an analysis of student work using graphic tools; see Little et al., 2003).

Encourage instructors to actively monitor online activity and send reminders and prompts
to participants as needed.
Quality of Context
Quality of context is examined through research question 1. According to the Massachusetts
SPD, high-quality PD is facilitated by knowledgeable instructors and makes use of relevant
resources. Kubitskey & Fishman (2007) find that in order for participant discussions to be
fruitful arenas for deep conversations, it is important to have a well-trained local PD provider.
For the RETELL pilot course, every participant across all four evaluation sites either agreed (35
percent) or strongly agreed (65 percent) with the statement that their instructor was
knowledgeable about SEI. As additional evidence of the instructors’ knowledge, it was observed
that the instructors answered all the questions posed in the course materials, though no answer
guide was provided.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—23
The instructors reported that they had a great deal of flexibility in the delivery of the course. For
example, the course was not heavily scripted, and so they determined which activities and even
which strategies to model. One instructor noted that the lack of scripting would make this a very
difficult course to deliver without an advanced degree and extensive teaching experience: “I just
don’t see honestly how someone without a Ph.D. or really advanced work in linguistics can take
this and run with it. So I think that the idea is nice to let people have their own input into it, but I
also think that you have to really have a strong command of the content to be able to do that. I
mean this is after seven years of teaching college courses, developing courses…, so I just don’t
know if they [ESE] have enough people to do that.”
The following are recommendations to improve the quality of the RETELL course context on the
basis of the findings just described:

It will be essential to select highly qualified instructors for a course that is not heavily
scripted. Unless activities are added and scripted, only RETELL instructors who are
prepared to develop and incorporate their own activities should be selected.

Consider developing and providing answer guides for questions posed in the course
materials.
Pace of Course
The pace of the course is examined through research question 1. Participants need adequate time
to assimilate information that they are learning, but the pace of the PD should be brisk enough to
maintain participants’ energy and to avoid boredom. During observations, the instructors
appeared to be very skilled at gauging participants’ need to speed up or slow down. But the pace
of the sessions was always very fast, with lots of content and little time for reflection. Indeed,
instructors generally viewed the pace as much too fast, describing it as “overwhelming” and
“problematic.” Several described the pace as “ridiculous.” One instructor noted that the course
would work if each session were delivered every two weeks, giving participants time to apply
what they’re learning in their classrooms in order to successfully complete course assignments.
Participants agreed that the pace of the RETELL course was problematic; a large majority of
participants thought that the pace of the course was a little too fast (40 percent) or much too fast
(45 percent). Only 15 percent thought that the pace was just right, and no one found it too slow.
See Figure 4.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—24
Figure 4. RETELL Course Participants’ Responses to
Survey Question About Course Pace
50%
45%
45%
40%
40%
35%
30%
25%
20%
15%
15%
10%
5%
0%
0%
Much too slow
A little too slow
0%
Just right
A little too fast
Much too fast
Pace of the RETELL course
Note that plans are already in place to deliver the RETELL course over 16 weeks (one semester)
rather than the 8.5–10 weeks of the pilot courses. ESE would prefer to not extend it any longer
because of potential problems with implementation, such as scheduling conflicts.
The following recommendation to improve the pace of the RETELL course is on the basis of the
findings just described:

Carefully monitor the semester-long course to make sure that the pace is not too fast. For
example, the schedule should allow reasonable time between sessions for participants to
complete readings and assignments, and to test the methods and instructional strategies
they are learning.
Allocation of Time in Course
Related to course pacing, the allocation of the time spent in the course can have important
implications for learning. For example, adults need time to reflect on what they are learning in
terms of their own classroom experiences (cf. Kubitskey, 2006; Penuel et al., 2007). It also is
important that they have adequate opportunities for collaboration with peers, through whole-class
discussions, small-group work, and interactions outside session time. The allocation of time in
the RETELL course is examined through research question 1.
On average, time during observed face-to-face sessions was allocated as presented in Figure 5,
with small-group work and whole-class discussions taking up the majority of class time.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—25
(Participant questions and discussions of how to apply strategies happened throughout the
sessions, but not continuously.)
Figure 5. Average Time Spent on Activities During RETELL Course Face-to-Face Sessions
Summarizing and review
7
Participant questions
Whole-class discussion
[throughout]
50
Practice applying strategies
24
Discussions of strategy application
Individual work
[throughout]
9
Small-group work
58
Instructor presentation/lectures
22
0
20
40
60
80
100
120
140
160
180
200
Average Minutes
Most participants expressed the opinion that the right amount of time was spent on instructor
lectures (71 percent), small-group work (77 percent), whole-class discussions (84 percent),
interactions with colleagues (79 percent), individual work in class (53 percent), and videos (63
percent). They thought that too little time was spent on discussions of how to apply strategies (55
percent) and practice applying strategies (61 percent), and too much time was allocated to
homework (58 percent) and reading (57 percent). See Figure 6.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—26
Figure 6. RETELL Course Participants’ Responses to
Survey Questions About Time Allocation
90%
84%
79%
77%
80%
71%
70%
63%
60%
53%
50%
44%
41%
38%
40%
61%
58%
55%
39%
33%
30%
20%
10%
57%
21%
20%
23%
20%
19%
9%
9% 7%
2%
9%
2%
4%
0%
4%
0%
0%
Too little time
Right amount
Too much time
For online sessions, participants reported devoting the most time (on average) to assignments (60
percent more than one hour) and reading (52 percent more than one hour). They devoted the least
time to interactions with the instructor (90 percent less than one hour), learning how strategies
can be applied in the classroom (69 percent less than one hour) and reading the slide
presentations (68 percent less than one hour).
Almost all the instructors expressed frustration at the allocation of time in the course. They were
concerned that they were trying to cover too much material, without time to focus on strategies
and classroom applications. One instructor recommended condensing the slide presentations into
very short overviews of the theory, devoting the rest of the session to practicing strategies.
The following are recommendations to improve the allocation of time in the RETELL course on
the basis of the findings just described:

Because participants found course time allocations appropriate for lectures and in-class
interactions with colleagues (including small-group work and whole-class discussion),
maintain these activities at their current levels.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—27

In order to address instructor concerns and participant feelings that not enough time was
devoted to strategies, consider replacing some of the small-group discussions with smallgroup strategy practice; whole-class discussions also could be used to discuss how theory
is realized in practice and how to apply instructional strategies successfully.

Replace significant out-of-class assignments with activities that require collaboration
with colleagues, particularly on the practice of strategies. Be sure to incorporate enough
time between assignments to allow participants to engage in these activities
meaningfully.

To address time allocation during the self-paced, self-directed online sessions, consider
creating an activity that helps participants understand how they learn best. Provide tips
for time management as part of the introduction to the online sessions.
Clarity of Instruction
It is crucial that instruction be delivered in a manner that participants can easily understand.
Clear definitions and descriptions, particularly of instructional strategies, and information about
how to use the instructional strategies for teaching are important factors in influencing teachers’
instruction (Kubitskey & Fishman, 2007). Clarity of instruction in the RETELL course is
examined through research question 1.
Overall, participants felt that instructors gave instructions well (44 percent) or very well (28
percent). See Figure 7. In the survey, 70 percent of participants reported rarely or never being
unable to receive clarification from the instructor about something that confused them. This can
be interpreted to mean that a large majority of participants received clarification when they
needed it. In observations of face-to-face sessions, instructors at all four sites always responded
directly to questions and comments, offering additional instruction or clarification when needed.
In all sites except Worcester, they identified prior conceptions and misconceptions (one
instructor made conceptions and misconceptions a topic for an activity and discussion). It was
always easy to hear the instructor and see the presentation during face-to-face sessions.
Figure 7. RETELL Course Participants’ Response to
Survey Question About Course Instructions
50%
44%
45%
40%
35%
28%
30%
25%
20%
20%
15%
10%
7%
5%
0%
0%
Poorly
Adequately
Well
Very well
Did not observe
How well the instructor gave instructions
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—28
For the online sessions, the evaluators observed that the presentation slides were extremely easy
to navigate, but online course materials were less so. Furthermore, in online sessions, instruction
was never observed to address feedback from previous sessions nor to identify prior conceptions
and misconceptions. In all cases but one, online instruction did not summarize the readings or
provide information about possible questions or concerns.
Finally, instructors noted that the assignments changed frequently throughout the course, which
was stressful for participants. They also noted that some participants needed extra time for the
readings. One instructor stated, “I’ve had people in class say, ‘I have ADHD’…just like with our
ELLs, you have to be able to differentiate and meet them three weeks ahead of time. If they don’t
read it till the night before, well that’s their choice but some people like to have it [in advance].”
The following are recommendations to improve clarity of instruction in the RETELL course on
the basis of the findings just described:

Consider including some short activities in both the face-to-face and the online portions
of the course that explore participants’ conceptions and misconceptions to ensure that
they are addressed.

For the online sessions, provide a single list of required readings, materials, and activities
for each session with hyperlinks to each.

Include summaries of readings and information about possible questions or concerns in
online materials.

Provide a list of assignments and readings ahead of the course so that participants have
additional time to complete the work.
Support for Course Instructors
In addition to examining how the RETELL course was delivered, it is important to examine how
instructors were supported in their delivery of the PD. Support for RETELL course instructors is
examined through research question 2. During delivery of the pilot courses, some of the
instructors experienced logistical issues, like lack of Internet access, locked bathrooms, inability
to access the classroom ahead of a session to set up, and being placed in classrooms where
people were unable to eat during three-hour evening sessions. Other logistical issues included
making sure participants had access to Moodle and having participants’ e-mail addresses in
advance. In addition, instructors reported that ESE was good about making copies for instructors
if they requested them in advance. It sometimes, however, proved difficult to obtain other
materials, like chart paper and markers.
The following are recommendations to better support RETELL course instructors:

Make sure that course classrooms are easily accessible, can be accessed ahead of each
session for setup, and have Internet access. If possible, designate a person at each site to
ensure that these logistics are taken care of.

Provide participant contact information to instructors as early as possible (ideally, at least
two weeks before the first session).
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—29

Establish a clear policy about material distribution and provide it to all instructors.
Include information about which materials and services ESE provides and which it does
not.
Establishment of Feedback Loops
Feedback is an important aspect of course quality. The establishment of feedback loops to ensure
course quality is examined through research question 3. Instructors sought feedback from
participants, both informally (talking to participants after class) and formally (surveys). They
also sought feedback from each other. In terms of ensuring course quality in the future, most
instructors expressed the need for more scripted instruction so that the RETELL message is
consistent across all courses. In addition, they wanted uniform information about grading. One
instructor also suggested that they should have the opportunity to practice the trainings with
other instructors to see what it should look like in practice before attempting to deliver it.
The following recommendations to establish feedback loops to ensure RETELL course quality
are based on the findings just described:

Provide information to instructors about which aspects of the RETELL course will need
to be consistent across the state, such as participant grading or evaluation information.

Consider providing a training session for instructors during which they can practice
elements of the PD with each other before delivering it to RETELL course participants.

Establish a consistent feedback system for all RETELL PD courses. The feedback system
might consist of an observation by an ESE official, formal participant surveys (e.g., short
surveys delivered via Survey Monkey after each class), or an outside evaluation of
teacher products (e.g., lesson plans) or outcomes.
Fidelity of Implementation
During full-scale rollout, the strength of the RETELL PD will depend on the fidelity of its
implementation. Course observations were used to evaluate the level of implementation during
face-to-face and online sessions (research question 4) and instructor interviews examined how
feedback loops could be established to ensure greater fidelity of implementation (research
question 5). Note that because the course is still in its pilot stage, evaluation of fidelity of
implementation was largely limited to alignment with the planned syllabus.
In every observed face-to-face session, instructors’ goals and objectives were the same as those
stated in the syllabi, and at least one of the required readings was explicitly incorporated into the
activities at each session. Participants were required to complete the homework in nearly every
session. Most face-to-face sessions, however, did not cover all the content. Some sessions did not
cover even half the content in the syllabi. The failure to cover all the content appeared to be
caused by differences in participant backgrounds and educational contexts; instructors had to
modify content, discussions, and activities to accommodate everyone, slowing the pace of each
session.
Online sessions exhibit almost universal fidelity to the syllabi, with a few exceptions. In session
7, the Halliday assignment in the syllabus was eliminated. In session 10, the assignments in the
syllabus for session 10 were included in the slides but were marked as assignments for session
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—30
11. And finally, in session 12 the slides did not address ELL assessment through MCAS
accommodation, as specified in the syllabus.
Instructors reported using a clock or stopwatch to ensure that there was sufficient time for each
activity in the materials that were provided. They also met regularly with the other RETELL
pilot course instructors to discuss course delivery in order to ensure fidelity to the syllabi and
course materials. They reported difficulty with fidelity, however, because the course was
continuously changing while they were delivering it.
The following recommendations for fidelity of RETELL course implementation are based on the
findings just described:

Strategically group participants with similar backgrounds and educational contexts into
the same courses so that instructors can focus on their needs efficiently, increasing the
likelihood of covering all course content. As an added advantage, grouping participants
from similar educational contexts can facilitate the formation of PLCs (see, e.g.,
Newmann & Associates, 1996).

Check online sessions to ensure that they cover all topics and assignments in the syllabus,
especially if there have been changes to the sessions.

Provide instructors with information about how long each activity should take.

Facilitate regular meetings among instructors (by region and/or participant type—e.g.,
secondary STEM) throughout the course semester to discuss RETELL delivery and
fidelity to the syllabus. These meetings would also be a good opportunity for ESE to
collect feedback from instructors about recommended changes.

Make necessary changes to the course between, not during, rollout semesters.
Coherence
A national study by Garet et al. (2001) found that coherence was one of the most important
aspects of effective PD; PD that connects to teachers’ previous knowledge and training is more
likely to have an impact on teacher practice. In this evaluation, two types of coherence are
examined. The first is the level of coherence of the RETELL sessions (research question 6), or
how well the sessions build on one another. The second is the level of coherence of the course
with what teachers experience in their educational contexts (research question 7).
Level of Coherence of the RETELL Sessions
Because the RETELL course is a hybrid course with nearly a third of the sessions online, it is
important that the sessions explicitly build on each other such that participants can recognize the
relevance of what they are learning outside the face-to-face time. In addition, each session should
help teachers augment their overall knowledge of effective instruction for ELLs by building on
what was learned about the topic before. The level of coherence of the RETELL course is
addressed in research question 6 by examining the ways in which explicit connections between
sessions are made.
Instructors at all four sites were only occasionally observed giving a brief overview of the
previous session and explaining how the current session built on what was learned before. Only
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—31
one instructor was observed connecting the material in a face-to-face session with material from
a previous online session. They reported that there was insufficient time to make connections
across sessions, in the interest of covering as much of each session’s content as possible. The
majority (65 percent) of participants, however, reported that their instructor made a clear
connection between the online and face-to-face content, so perhaps this occurred more often at
sessions that were not observed for this evaluation. Observed online sessions did not explicitly
build on previous sessions or provide explanations of how the online content would link to faceto-face sessions (except in one occasion when the instructor added comments to this effect).
The following recommendations to increase the level of coherence of the RETELL sessions are
based on the findings just described:

Revise course materials to provide explicit connections between topics covered in each
session, including revising the online materials to include explicit connections to previous
and upcoming sessions.

Explain to instructors the importance of communicating expectations for the online
course during the face-to-face sessions, even if there is only a little time to do so.
In addition, instructors recommended the following changes to increase coherence of the
RETELL course:

Incorporate session 4 into session 5.

Consolidate information about academic language into a single session.

Incorporate information about the Common Core State Standards throughout the sessions
rather than introducing it in session 15.

The topic of assessment should come at the end of the course, after reading and writing
have been covered.

Provide more grade-level differentiation in Module C [within the secondary courses].
Level of Coherence With What Teachers Experience in Their Educational Contexts
Garet et al. (2001) noted the importance of ensuring that what teachers experience in PD be
applicable to their own educational contexts. This type of coherence is examined through
research question 7. The RETELL pilot course instructors also seemed to be aware of the
importance of making connections to participants’ contexts; instructors at all sites used stories to
link theory to practice, discussed problems that might occur with classroom application of
strategies and how one might address them, and led participants in discussions of how a strategy
would or would not work in their classroom. The majority of participants at all sites found it
pretty useful (34 percent) or extremely useful (45 percent) to discuss application of strategies; no
participant responded that it was not useful at all. Instructors were not, however, able to engage
in these activities as often as they felt was necessary, and participants appeared to agree; overall,
more participants felt that too little time was allocated to discussions of how to apply strategies
to classroom instruction (55 percent) than felt that the right amount of time was allocated (41
percent). This difference was greatest in Boston and Worcester, where 68 percent of participants
felt that too little time was allocated to discussion of strategy application. See Figure 8.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—32
Figure 8. RETELL Course Participants’ Response to Survey Question
About Time Allocated to Discussions of Classroom Strategy Application
80%
68%
70%
60%
55%
50% 50%
50%
44% 44%
41%
40%
32%
30%
20%
11%
10%
0%
4%
0%
0%
Somerville
Lowell
Too little time
Boston/Worcester
Right amount
Total
Too much time
In addition, few examples of connections to classroom practice were evident in the online
sessions. Though participants sometimes posted examples of how something would or would not
work in their classrooms, the online instruction seldom discussed problems that might occur with
classroom application and only sometimes incorporated stories or videos to link theory to
practice. Most participants (70 percent) reported that less than an hour was devoted to illustration
of how strategies could be applied to the classroom in the online sessions, but all reported that
such illustrations were at least a little useful, and most found them pretty useful (29 percent) or
extremely useful (38 percent). See Figure 9.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—33
Figure 9. RETELL Course Participants’ Response to Survey Question About
Usefulness of Online Illustrations of Strategy Application
40%
38%
35%
29%
30%
29%
25%
20%
15%
10%
5%
5%
0%
0%
Not useful at all A little useful
Pretty useful Extremely useful
Unknown
Usefulness of online illustrations of strategy application
The following recommendations to increase the level of coherence of the RETELL sessions with
what teachers experience in their educational contexts are based on the findings just described:

Connections to participants’ educational contexts are crucial. Ensure that RETELL
course instructors are as prepared as the pilot instructors to connect material to real-world
contexts, use stories to link theory to practice, and lead participants in discussions of how
strategies would work in their educational contexts.

Consider incorporating the practice of instructional strategies into small-group work and
whole-class discussions in order to increase its impact.

Update the online materials to make better connections to participants’ educational
contexts. In each session, include
o Stories or videos that model practical strategies
o Discussions of possible challenges that teachers might face in applying strategies
o Prompts that encourage participants to post and discuss examples of how strategies
would work in their educational contexts
Teacher Outcomes
The purpose of the RETELL initiative is to improve instruction for ELLs in Massachusetts by
increasing mainstream teachers’ knowledge about ELLs and informing their instructional
practices. This evaluation therefore examined how the course could better contribute to teachers’
key understandings of ELL students in regular classroom settings through research question 8
and additional information that teachers need to be successful with ELLs through research
question 9.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—34
The majority of participants in all sites agreed or strongly agreed that the course enhanced what
they already knew about SEI (88 percent overall), but 61 percent also reported that some topics
repeated what they already knew about SEI. The majority of participants reported that the course
had contributed a fair amount or a great deal to their understanding in the areas shown in Table
5. It is impossible to determine whether the course contributed less to participants’ understanding
in some areas because of their previous knowledge, or because the course did not adequately
cover these areas.
Table 5. Topics for Which RETELL Participants
Reported Greater Understanding
Percentage for Whom
Course Increased
Understanding
82%
SEI Topic
Vocabulary instruction for ELLs
How ELLs acquire a second language
81%
How to meet the needs of ELLs in the classroom
78%
Social and cultural issues that affect ELLs
78%
How to teach academic language to ELLs
78%
How to apply SEI to reading and writing instruction
72%
How to address the needs of diverse ELL populations
68%
How to differentiate instruction for ELLs
64%
How to assess ELLs in the content areas
64%
As part of the survey, participants were also asked to list areas in which they felt they needed
additional information to be successful with ELLs. These areas are given in Table 6.
Table 6. Topics in Which RETELL Participants
Need Additional Information
SEI Topic
Creating lessons and practical strategies for the classroom
Effective practice with ELLs who do not speak Spanish
Standardized assessments and ELLs
ELLs with special needs
Grade-specific strategies
Supporting language development
Students with interrupted formal education (SIFE)
In addition to the topics in Table 6, one instructor felt that teachers needed more information
about the difference between helping ELLs develop language and helping them access content.
Other instructors expressed the opinion that there needed to be additional grade-level
differentiation, and one felt that participants needed to be more immersed in data.
Of course, additional information that teachers need to be successful will vary from course to
course, and teachers and instructors may not be aware of some of these needs without specific
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—35
participant feedback structures. For example, the RETELL pilot course instructors reported
several anecdotes of individual participants applying particular strategies or knowledge with
their ELL students, but one instructor pointed out that it is impossible to tie the RETELL
instruction to specific outcomes because there is no evidence to assess the outcomes. She
explained that though there is a monitoring tool for participants’ lesson plans early in the course,
there is no such tool for revisiting the plans at the end. She also suggested adding tools to get
feedback about how participants are implementing strategies. Another instructor recommended a
pre- and post-assessment of the content of the course.
The following recommendations to enhance the teacher outcomes of the RETELL course are
based on the findings just described:

Conduct a brief survey at the outset of each course to determine areas in which
participants are already well versed and areas where they need additional instruction.

Consider providing (additional) instruction in the following areas, as determined by
participant needs:
o Creating lessons and practical strategies for the classroom
o Effective practice with ELLs who do not speak Spanish
o Standardized assessments and ELLs
o ELLs with special needs
o Grade-specific strategies
o Supporting language development versus helping students access content
o Students with interrupted formal education (SIFE)
o Using data

Fishman et al. (2003) propose an iterative PD research design model, in which feedback
about teacher outcomes is incorporated into future PD sessions; a similar model has been
proposed by Kubitskey & Fishman (2007). The current evaluation gathers self-reported
data from teachers, but it also is important to measure teachers’ skills and knowledge.
The following formative and summative evaluation activities are recommended both to
provide teachers with information about their progress in the course and to improve
future iterations of the course:
o Develop and incorporate protocols for assessing participants’ plans for and reports of
how they implemented course strategies.
o Develop and conduct pre- and post-course assessments of participants’ knowledge
about ELLs and ELL instruction.
o Include a classroom observation component in the course. Teachers might observe
each other and provide peer-to-peer feedback; a school or district ESL coordinator or
coach might observe participants and provide feedback as part of the RETELL
course; or RETELL course instructors might observe each participant teaching.
Observations are recommended for teachers in high- and medium-incidence schools;
they are not recommended if participants do not have ELLs in their classrooms at the
time of the RETELL PD.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—36

Make sure that instructors can provide timely feedback to participants using the Moodle
online platform. For instructors who are unfamiliar with Moodle, offer training on how to
use it effectively.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—37
Next Steps
ESE plans to roll out the first phase of the RETELL course in February 2013. Before that time,
many revisions will be made to the course materials on the basis of what has been learned from
the pilot course.
This evaluation has made 44 recommendations in four major categories (quality of delivery,
fidelity of implementation, coherence, and teacher outcomes) for improvement of the quality of the
course. As a final recommendation, if these or other changes are implemented, it will be important to
monitor their effects on the success and quality of the course. It also will be important to explore how
the course functions when it is rolled out to a much larger group of teachers across many more
districts and schools. Of especial interest is the online portion of the course. If it can be successfully
revised to increase participant engagement and collaboration, it will be an important tool for ensuring
that teachers across the state are prepared to provide effective instruction for the ELLs in their
classes.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—38
References
August, D., & Shanahan, T. (2010). Effective English literacy instruction for English learners. In
F. Ong (Ed.) (with V. Aguila), Improving education for English learners: Researchbased approaches (pp. 209–237). Sacramento, CA: CDE Press.
Bryk, A. S., & Schneider, B. (2002). Trust in schools: A core resource for improvement. New
York: Russell Sage.
Carlo, M. S., August, D., McLaughlin, B., Snow, C. E., Dressler, C., Lippman, D., et al. (2004).
Closing the gap: Addressing the vocabulary needs of English-language learners in
bilingual and mainstream classrooms. Reading Research Quarterly, 39(2), 188–215.
Darling-Hammond, L., & McLaughlin, M. W. (1995). Policies that support professional
development in an era of reform. Phi Delta Kappan, 76(8), 597–604.
Fishman, B. J, Marx, R. W., Best, S., & Tal, R. T. (2003). Linking teacher and student learning
to improve professional development in systemic reform. Teaching and Teacher
Education, 19(6), 643–658.
Frank, K. A., Zhao, Y., & Borman, K. (2004). Social capital and the diffusion of innovations
within organizations: The case of computer technology in schools. Sociology of
Education, 77(2), 148–171. doi: 10.1177/003804070407700203
Franken, M., & Haslett, S. J. (1999). Quantifying the effect of peer interaction on second
language students’ written argument texts. New Zealand Journal of Educational Studies,
34(2), 281–293.
Garet, M. S., Porter, A. C., Desimone, L., Birman, B. F., & Yook, K. S. (2001). What makes
professional development effective? Results from a national sample of teachers.
American Educational Research Journal, 38(4), 915–945.
Gersten, R., Baker, S. K., Shanahan, T., Linan-Thompson, S., Collins, P., & Scarcella, R. (2007).
Effective literacy and English language instruction for English learners in the elementary
grades: A practice guide (NCEE 2007–4011). Washington, DC: U.S. Department of Education,
Institute of Education Sciences, National Center for Education Evaluation and Regional
Assistance. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/practice_guides/20074011.pdf
Kubitskey, B. (2006). Extended professional development for systemic reform. Unpublished
doctoral dissertation, University of Michigan, Ann Arbor.
Kubitskey, B., & Fishman, B. (2005, April). Untangling the relationship(s) between professional
development, practice, student learning, and teacher learning. Paper presented at the
Annual Meeting of the American Educational Research Association, Montreal, Canada.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—39
Kubitskey, B., & Fishman, B. J. (2006). A role for professional development in sustainability:
Linking the written curriculum to enactment. In S. A. Barab, K. E. Hay, Songer, N. B., &
D. T. Hickey (Eds.), Making a difference: Proceedings of the seventh International
Conference of the Learning Sciences (Vol. 1, pp. 363-369). Mahwah, NJ: Lawrence
Erlbaum.
Kubitskey, B., & Fishman, B. J. (2007, April). A design for using long-term face-to-face
workshops to support systemic reform. Paper presented at the American Educational
Research Association, Chicago, IL.
Lee, O., Maerten-Rivera, J., Penfield, R. D., LeRoy, K., & Secada, W. G. (2008). Science
achievement of English language learners in urban elementary schools: Results of a firstyear professional development intervention. Journal of Research in Science Teaching,
45(1), 31–52.
Lieberman, A., & McLaughlin, M. W. (1992). Networks for educational change: Powerful and
problematic. Phi Delta Kappan, 73(9), 673–677.
Little, J. W. (1993). Teachers’ professional development in a climate of educational reform.
Educational Evaluation and Policy Analysis, 15(2), 129–151.
Little, J. W., Gearhart, M., Curry, M., & Kafka, J. (2003). Looking at student work for teacher
learning, teacher community, and school reform. Phi Delta Kappan, 85(3), 184–192.
Massachusetts Department of Elementary and Secondary Education. (2009). Report to the
legislature: English language acquisition professional development. Malden, MA:
Author.
Massachusetts Department of Elementary and Secondary Education. (2012, May). The state of
the state: A report on English language learners in Massachusetts. Paper presented at the
Massachusetts Association of Teachers of Speakers of Other Languages Conference,
Framingham, MA.
National Clearinghouse for English Language Acquisition and Language Instruction Educational
Programs. (2011). The growing number of English learner students. Washington, DC:
Author. Retrieved from http://www.ncela.gwu.edu/files/uploads/9/growingLEP_0809.pdf
National Governors Association Center for Best Practices & the Council of Chief State School
Officers. (2010). Common core state standards for English language arts. Washington,
DC: Authors.
Newmann, F. M., & Associates. (1996). Authentic achievement: Restructuring schools for
intellectual quality. San Francisco, CA: Jossey-Bass.
Owens, A. (2010). In the aftermath of Question 2: Students with limited English proficiency in
Massachusetts. Boston: Massachusetts Office for Refugees and Immigrants. Retrieved
from www.doe.mass.edu/research/reports/0610ori.doc
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—40
Penuel, W. R., Fishman, B. J., Yamaguchi, R., & Gallagher, L. P. (2007). What makes
professional development effective? Strategies that foster curriculum implementation.
American Educational Research Journal, 44(4), 921–958.
Perez-Selles, M., Cazabon, M., & Mello, D. (2011). Needs assessment report: Sheltered English
immersion (SEI) category trainings, Massachusetts Department of Elementary and
Secondary Education. Woburn, MA: WestEd. Retrieved from
http://www.doe.mass.edu/retell/1011SEIPD_NeedsAssess.pdf
Saunders, W. M. (1999). Improving literacy achievement for English learners in transitional
bilingual programs. Educational Research and Evaluation, 5(4), 345–381.
Saunders, W. M., & Goldenberg, C. (1999). Effects of instructional conversations and literature
logs on limited- and fluent-English-proficient students’ story comprehension and
thematic understanding. Elementary School Journal, 99(4), 277–301.
Vaughn, S., Martinez, L. R., Linan-Thompson, S., Reutebuch, C. K., Carlson, C. D., & Francis,
D. J. (2009). Enhancing social studies vocabulary and comprehension for seventh-grade
English language learners: Findings from two experimental studies. Journal of Research
on Educational Effectiveness, 2(4), 297–324.
Vescio, V., Ross, D., & Adams, A. (2008). A review of research on the impact of professional
learning communities on teaching practice and student learning. Teaching and Teacher
Education, 24(1), 80-91.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—41
[This page intentionally left blank.]
American Institutes for Research
MA SEI Endorsement Pilot Evaluation—42
Appendix A: Data Collection Instruments
Face-to-Face Session Observation Protocol
RETELL In-Class Observation Protocol
Date
Location
Instructor Name
Observer
Observation Start Time
# of Participants (15 min in)
Observation End Time
Topic
Complete this section immediately after your observation. Rate each item below.
Frequency
Session
Introduction
Never
Sometimes
Quality
Almost
Always
Poor
Acceptable
Excellent
Notes
N/A
Instructor articulates goal of session and the
effect the session should have on participants’
practice.
Instructor asks what participants would like to
learn/know about the topic (or how much they
already knew)
Instructor adjusts schedule to address these
topics
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —43
Frequency
Quality
Notes
Instructor can be heard in the back of the room
Instructor responds directly to questions and
comments
Instructor identifies prior conceptions and
misconceptions about content
Instructor appears knowledgeable about content
Instructor offers additional instruction and/or
clarification when a participant asks a question
Instruction
Instructor gives a brief overview of the
previous session and explains how the current
session will build on what was learned.
If the course follows an online session,
instructor gives an overview of the online
component and summary of what should have
been learned
If the course follows an online session,
instructor summarizes the readings and asks if
there are questions or concerns
Instructor uses clues from participants to speed
up, slow down, or reengage
If the course follows an online session,
instructor explains how the online content links
to the content to be covered in the face-to-face
session
Instructor gives overviews (when introducing
topic, idea, activity)
Instructor gives summaries (at the end of a
discussion, presentation, activity)
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —44
Pedagogy/Transferring Ideas to Practice
Struggling
Pushing Back
Quality of Curriculum
Frequency
Quality
Notes
Most participants complete the activities
correctly (as the instructor intended)
Multiple participants ask for clarification on the
same topic
Participants suggest that the factual material
included in the syllabus (the course content) is
flawed in some manner (e.g. “I don’t think this
is the case.” “In my experience, this usually
works differently.” “I don’t see how that would
be useful!”)
Participants question the usefulness of the
course (e.g. “This is good information but I’m
not sure it’s applicable to me/us/the real world.”
“I don’t know. I think the old category training
was sufficient.”)
Instructor discusses problems that might occur
with classroom application and addresses them
Instructor promotes collaboration among
participants
Instructor connects content to real-world
contexts
Instructor uses stories to link theory to practice
Engagement with important ideas relevant to
classroom practice
Participants discussed an example of how
something would/would not work in their
classroom
Participants asked clarifying questions about
how something could be used in the
classroom
Participants offered suggestions on how
ideas could be applied in the classroom
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —45
Frequency
Quality
Notes
Participants answered questions from other
participants or commented on other
participants’ questions/comments on
classroom application
Participants reference readings during
discussions
Participants state reading selections were useful
or helpful
Vocabulary Activities
Participants ask clarifying questions about what
the main ideas of reading selections were
PowerPoint
Accessibility of Material
Readings
Most participants completed the readings for
the session
When introducing activities, instructor states
how the activity is connected to course goals
Participants seem to have trouble following the
instructions for activities
Participants readily adopt the phrases used in
the course to describe phenomena or ideas (e.g.
“culturally responsive” or “affective filter”)
Participants are engaged during PowerPoint
presentations (see Overall Engagement for how
to measure this)
Participants ask questions during or after
PowerPoint presentations
Participants can see the presentation clearly
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —46
Content
Frequency
Quality
Notes
Participants demonstrate some familiarity with
much (30-40%) of the course content (e.g. if
participants have heard of “culturally
responsive practice” or “affective filter” they
can understand what it is because they have
witnessed or learned about it before--albeit by a
different name)
Most participants look at the instructor as
he/she is speaking
Overall Engagement
Most participants nod their heads in agreement
as the instructor is speaking
Most participants take notes as the instructor is
speaking
When instructor asks general questions to
gauge understanding (e.g. “Is this clear?” “Are
there any questions?”), most participants nod or
answer verbally
Participants turn to the appropriate
pages/sections of the instructional materials
when instructed to do so
Most participants are sitting upright in their
seats
Most participants are checking email,
voicemail, texting, or surfing the internet for
content unrelated to the course
Participants ask clarifying questions
Pacing
Participants are flipping through later sections
of the materials ahead of the instructor
Participants are still reading earlier sections of
materials (or discussing with another
participant) although the instructor has moved
on
Course activities are distributed evenly
throughout session.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —47
Frequency
Quality
Notes
Next Steps
Classroom Culture
Participants and instructor challenged each
other’s ideas
Instructor asks if there are questions and waits
for participants to respond
Instructor uses phrases like “good point,” “I can
understand your interpretation of that,” “great
question,” “thanks for bringing that up,” or
others
Instructor asks participants to “think about
this,” or “what would you do if…,” “what’s a
good way to handle…,” “why do you think that
is,” or “what else would you like to know?” or
others
Participants presented with clear next steps they
could apply to continue to improve their skills
Participants provided with list of resources
(websites, contact info, etc.) to help implement
what was learned in the classroom
Fidelity of
Implementation
Session covers all topics listed in the syllabus.
Instructor’s goals and objectives are the same
as those stated in syllabus.
Instructor discusses and/or incorporates all
required readings for session.
Participants are required to complete all
homework for session as described in the
syllabus.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —48
Complete this section during your observation. When you observe a listed action, note the start and end for each occurrence (as many
as needed). You do not need to describe the activity itself unless it is of particular importance, in which case, you may use the notes
section.
Start
End
Start
End
Time
Start
End
Start
End
Start
End
Instructor presenting/lecturing
Allocation of Time
Small group work/discussions
Individual work
Discussions of how to apply strategies being
learned to classroom instruction
Practice applying strategies to classroom
instruction
Whole class discussion
Participant questions
Summarizing and review
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —49
Complete this section during your observation. List examples of each that you observed. You may use the notes section to describe
each in more detail.
Notes
Accessibility
Ah-ha moments
American Institutes for Research
Demonstrating
understanding of key
concepts and ideas during
course
Notes
MA SEI Endorsement Pilot Evaluation —50
Online Session Observation Protocol
RETELL Online Observation Protocol
Instructor Name
Session #
Observer
Observation Start Time (if applicable)
# of Participants
Average # of Posts per Participant
Observation End Time (if applicable)
Topic
Complete this section immediately after your observation. Rate each item below.
Occurrence
Yes
No
N/A
Quality
Poor
Acceptable
Excellent
Notes
N/A
The goal of session and the effect the session should
have on participants’ practice are provided
Instruction
A brief overview of the previous session and an
explanation of how the current session will build on
what was learned are provided
Instruction explicitly addresses feedback from
previous sessions
Instruction identifies prior conceptions and
misconceptions about content
Instructor’s paper exhibits knowledge about content
Instructor uses the online platform effectively
Instructor offers additional instruction and/or
clarification when a participant posts a question
Instructor’s paper summarizes the readings and
provides information about possible questions or
concerns
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —51
Occurrence
Yes
No
Quality
N/A
Poor
Acceptable
Excellent
Notes
N/A
Explanations of how the online content links to the
content to be covered in the face-to-face sessions are
provided
Overviews are provided (when introducing topic, idea,
activity)
Summaries are provided (at the end of a discussion,
presentation, activity)
Frequency
Pedagogy/Transfer
ring Ideas to
Practice
Struggling
Pushing Back
Quality of Curriculum
Never
Sometimes
Quality
Almost
Always
Poor
Acceptable
Excellent
Notes
N/A
Participant posts/comments are irrelevant to the stated
requirements.
Participant comments suggest that the factual material
included in the syllabus (the course content) is flawed
in some manner (e.g. “I don’t think this is the case.”
“In my experience, this usually works differently.” “I
don’t see how that would be useful!”)
Participant comments question the usefulness of the
course (e.g. “This is good information but I’m not sure
it’s applicable to me/us/the real world.” “I don’t know.
I think the old category training was sufficient.”)
Instruction discusses problems that might occur with
classroom application and addresses them
Instruction connects content to real-world contexts
Instruction incorporates stories/vides to link theory to
practice
Engagement with important ideas relevant to
classroom practice
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —52
Frequency
Never
Sometimes
Quality
Almost
Always
Poor
Acceptable
Excellent
Notes
N/A
Participants posted an example of how something
would/would not work in their classroom
Participants posted clarifying questions about how
something could be used in the classroom
Participants posted suggestions on how ideas could be
applied in the classroom
Activities
Materials Vocabulary
Accessibility of Material
Readings
Participants answered questions from other
participants or commented on other participants’
questions/comments on classroom application
Most participants completed the readings for the
session
Participants reference readings in their comments
Participants posted clarifying questions about what the
main ideas of reading selections were
Activities are introduced with information about how
the activity is connected to course goals
Participants complete the activities correctly (as the
instructor intended)
Participants readily adopt the phrases used in the
course in their comments (e.g. “culturally responsive”
or “affective filter”)
The PowerPoint slides are easy to follow.
Course materials are easy to navigate.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —53
Frequency
Classroom
Culture
Pacing
Overall
Engagement
Participation
Never
Sometimes
Quality
Almost
Always
Poor
Acceptable
Excellent
Notes
N/A
Participants have an opportunity to ask clarifying
questions
Participants have an opportunity to response to
discussion questions
Participants use online tools to indicate participation
Participants complete session requirements by posting
online and responding to other participants’ posts
Participants can cover all materials in the allotted
session time.
Course activities are distributed evenly through session
topics.
Participants and instructor challenged each other’s
ideas in posts
Comment prompts encourage participants to “think
about this,” or “what would you do if…,” “what’s a
good way to handle…,” “why do you think that is,” or
“what else would you like to know?” or others
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —54
Occurrence
Next Steps
Yes
No
Quality
N/A
Poor
Acceptable
Excellent
Notes
N/A
Participants presented with clear next steps they could
apply to continue to improve their skills
Participants provided with list of resources (websites,
contact info, etc.) to help implement what was learned
in the classroom
Fidelity of
Implementation
Session covers all topics listed in the syllabus.
Session goals and objectives are the same as those
stated in syllabus.
Session discusses and/or incorporates all required
readings for session.
Participants are required to complete all homework for
session as described in the syllabus.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —55
Complete this section as you are completing the online session activities. When you observe a listed action, note the duration, if
applicable. You do not need to describe the activity itself unless it is of particular importance, in which case, you may use the notes
section.
Duration
Duration
Duration
Duration
Duration
PowerPoint presentation
Allocation
of Time
Participant discussions
Individual work
Practice applying strategies to classroom instruction
Participant questions
Summarizing and review
Notes
Accessibility
Ah-ha moments
American Institutes for Research
Demonstrating
understanding of key
concepts and ideas during
course
Notes
MA SEI Endorsement Pilot Evaluation —56
Instructor Interview Protocol
_____________________________________________________________________
District:
Interviewer:
Interviewee(s):
Date/Time:
Background Information
1. Please briefly describe your background as a PD provider.
Prompts: What is your ESL background? Have you previously provided PD on ELLs?
2. Could you please describe your role in RETELL? When did you first become involved as
an instructor?
3. Were you involved in the development of the RETELL course content?
[If yes] In what capacity?
[If no] What process did you use to become familiar with it?
4. What is different in the RETELL course from PD you have offered before? What is
similar?
Structure and Content of the RETELL PD
5. How much leverage do you have in how you structure your RETELL course?
Prompts: Is it heavily scripted? Are you expected to fill in a lot of the information?
6. How is the pace of the RETELL course?
Prompts: Is it easy or difficult to cover all of the topics in the syllabus in each session? Are
there topics you’ve had to skip, and if so, how have you chosen which topics to skip? Do the
topics seem evenly distributed throughout the sessions?
7. How is the sequence of the RETELL topics?
Prompts: Are the topics sequenced in a way that makes sense to you? If not, how would you
rearrange them? Have you re-sequenced any of the topics? Is there too much, too little, or
adequate repetition of important information?
8. What is the content focus of the RETELL PD?
Examples: linguistic training particular to English language acquisition; explicit curricular
and instructional strategies/best practices for teaching English to ELLs; cultural or
community sensitivity/culturally responsive pedagogy
9. Could you describe the content in more detail?
Examples/prompts: Is there training with models or approaches that have been shown to be
effective with ELLs (e.g., modeling, using graphic organizers, contextualizing, using
multiple modes of communication, using SIOP)? What instructional strategies are
emphasized as important for ELLs’ acquisition of subject matter? Is data-based assessment of
learner needs covered? Differentiated instruction? Are materials of relevance to ELLs
incorporated?
10. What steps, if any, do you take to ensure the quality of the RETELL course you deliver?
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —57
11. What steps, if any, do you take to ensure that the RETELL sessions are delivered as they
were designed?
12. From your perspective, which aspects of the RETELL content do teachers struggle with
most? What is easiest for them?
13. From your perspective, what are [name of district] teachers’ greatest needs when it comes
to PD on ELLs? Why?
Prompt: What is the capacity of teachers in the district to address the needs of ELLs? What
are the challenges that teachers in the district face in working with ELLs?
14. Does the RETELL PD meet these teachers’ needs? Why or why not?
15. What has been the most successful aspect of delivering the RETELL material?
16. What has been the most challenging aspect of delivering this material?
Instructor Experiences
17. Have you encountered any challenges with course logistics? If yes, please describe.
Examples: internet access, timing, availability of the room, size of the room, communication
with ESE, communication with participants
18. What kinds of support have you received in delivering the RETELL sessions? How has it
helped you, if at all?
19. Do you feel you need additional support from ESE or from the district?
[If yes] What kind of support do you most need?
20. What kinds of support are/would be most useful to you as a RETELL instructor?
Prompt: Are there particular types of support that would improve the quality or fidelity of the
course?
21. Do you receive formal or informal feedback as a RETELL instructor?
[If yes] What kind of feedback do you receive, and how useful is it?
22. What kinds of feedback are/would be most useful to you as a RETELL instructor?
Prompt: Are there particular types of feedback that would improve the quality or fidelity
of the course?
Project Outcomes
23. Have there been any observable effects of the RETELL PD on the teachers? Please
describe what you have observed.
24. [If effects are noted:] What factors or conditions do you think supported these changes?
25. How would you assess the effectiveness of the participants who have completed the
program to work with ELLs?
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —58
Wrap-up
26. Is there anything else you would like to tell me that might help ESE improve RETELL
and increase its impact?
Those are all the questions I have. Thank you very much for your time!
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —59
Participant Focus Group Protocol
This focus group will gather information about the teacher survey. Members of the focus group
will each fill out the survey, and discuss each item as they answer it.
1. Say: “Thank you for agreeing to participate in this focus group. We are trying to gather
information about our teacher survey. Specifically, we’re trying to find out if the questions give
us the information we’re looking for. I have given you each a survey. Please answer the first
item.”
2. When all participants have finished the first item, say, “I’m going to ask you a couple of
questions about the first item.” Ask the following questions:
 What did you think this item was asking you to do?
 Was there anything you found challenging about providing an answer for this item?


If yes, What was challenging, and why?
Were there any other options that should have been included?
Were there any options that didn’t make sense to include? Why?
3. Repeat for each multiple choice item. For each open ended question, ask,
 What did you think this item was asking you to do?
 Was there anything you found challenging about providing an answer for this item?

If yes, What was challenging, and why?
Do you think this could be posed as a multiple choice question?
If yes, What would the options be?
If no, Why not?
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —60
Participant Survey
PART I: Face-to-Face Portion of the RETELL Course
This first set of questions asks about your face-to-face time in class with an instructor.
1. Select the one rating that best reflects your opinion of the pace of the course.
 Much too slow
 A little too slow
 Just right
 A little too fast
 Much too fast
Comments:
2. Select the rating that best reflects how well time was allocated to each of the activities listed below.
Select only one rating for each activity.
Too Little Time
The Right Amount
of Time
Too Much Time
Discussion around how to apply strategies being learned to
classroom instruction
○
○
○
○
○
○
○
○
○
Practice during the course applying strategies to classroom
instruction
○
○
○
Whole class discussion
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Instructor presenting/lecturing
Small group work/discussions
Interacting with colleagues
Individual work (in class)
Individual work (homework)
Reading
Videos
Comments:
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —61
3. Select the rating that best reflects how useful each of the activities listed below was. Select only one
rating for each activity.
Not Useful
At All
A Little
Useful
Pretty
Useful
Extremely
Useful
Unknown
Discussion around how to apply strategies
being learned to classroom instruction
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Practice during the course applying strategies
to classroom instruction
○
○
○
○
○
Whole class discussion
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Instructor presenting/lecturing
Small group work/discussions
Interacting with colleagues
Individual work (in class)
Individual work (homework)
Reading
Videos
Comments:
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —62
PART II: Online Portion of the RETELL Course
This set of questions asks you about your experience with the online portion of the RETELL course.
4. Select the average amount of time each of the following activities took you while completing each online
session. Select only one time range for each activity.
PowerPoint presentations
Activities
Assignments
Interactions with the instructor
Illustration of how strategies can be applied to
the classroom
Reading
Posting comments and responses online
< 30 Min.
per Session
30-60 Min.
per Session
60-90 Min.
per Session
90-120 Min.
per Session
> 120 Min.
per Session
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Comments:
5. Select one rating that best reflects how useful each of the following activities was. Select only one rating for
each activity.
PowerPoint presentations
Activities
Assignments
Interactions with the instructor
Illustration of how strategies can be applied to
the classroom
Reading
Posting comments and responses online
Not Useful
At All
A Little
Useful
Pretty
Useful
Extremely
Useful
Unknown
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Comments:
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —63
6. Select one rating that best reflects how strongly you agree or disagree with each of the statements listed
below. Select only one rating for each statement.
Strongly
Disagree
Disagree
Agree
Strongly
Agree
Not
Applicable
○
○
○
○
○
○
○
○
○
○
I used the online platform to communicate
with other participants when not required.
○
○
○
○
○
○
○
○
○
○
The instructor made a clear connection
between the online and face-to-face content.
○
○
○
○
○
The PowerPoint presentations were easy to
follow.
I felt as engaged in the online content as I was
in the face-to-face session.
I found it easy to navigate the course materials.
Comments:
7. Please tell us what changes you think would make the online portion of the RETELL course more useful for
educators. Please be as candid as possible.
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —64
PART III: Instructor, Course Activities, and Classroom Culture
This set of questions asks you to rate the instructor, materials, and classroom culture of the RETELL
course.
8. Select the rating that best reflects how strongly you agree or disagree with each of the following statements.
Select only one rating for each statement.
It was easy to see the instructor’s presentation.
The room was clean.
The room was comfortable.
The organization of the room facilitated collaboration
with my colleagues.
Strongly
Disagree
Disagree
Agree
Strongly
Agree
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Comments:
9. Select the rating that best reflects how often the lead instructor did each of the activities listed below. Select
only one rating for each activity.
Slowed down or moved on based upon the
mood in the room
Demonstrated excitement about the content
Engaged participants to ask questions and/or
share expertise
Checked for understanding
Posted course goals and objectives
Never
Rarely
Sometimes
Frequently
Not
Applicable
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Comments:
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —65
10. Select the rating that best reflects how well the instructor did each of the following. Select only one rating for
each activity.
Summarized readings
Gave instructions
Facilitated discussions
Responded to questions
Handled difference of opinion
Linked instructional materials (e.g., readings)
to the main ideas being presented
Poorly
Adequately
Well
Very well
Did Not
Observe
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Comments:
11. Select the rating that best reflects whether and how often each of the following occurred. Select only one
rating for each statement.
Never
Rarely
Sometimes
Frequently
Not
Applicable
I did not receive clarification from the
instructor about something that confused me.
○
○
○
○
○
I was uncertain about what to do during an
activity after instructions were given.
○
○
○
○
○
I did not get the opportunity to share my
comments.
○
○
○
○
○
I did not feel the main ideas from a reading
were clear, even after discussing it in class.
○
○
○
○
○
My questions to the instructor were not
answered thoroughly.
○
○
○
○
○
Comments:
12. What could improve the instruction/pedagogy of the course?
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —66
PART IV: The RETELL Readings
This set of questions asks you specifically about the course required readings.
13. Select the rating that best reflects how strongly you agree or disagree with each of the following statements.
Select only one rating for each statement.
Strongly
Disagree
Disagree
Agree
Strongly Agree
○
○
○
○
○
○
○
○
In general, the readings were applicable to my
instruction of English language learners.
○
○
○
○
○
○
○
○
In general, the readings helped me to better
understand sheltered English immersion.
○
○
○
○
In general, the readings were too academic.
In general, the readings were related to the course
content.
In general, the readings were easy to understand.
14. The following is a list of all required course readings. Please mark only the readings you found particularly
useful or problematic. It is not necessary to mark every reading.
Problematic
Useful
Session 1: Massachusetts Department of Elementary and Secondary Education. (2012).
User guide for DART detail: English language learners.
○
○
Session 2: DeCapua, A., & Marshall. H. W. (2011). Reaching ELLs at risk: Instruction for
students with limited or interrupted formal education.
○
○
Session 2: Robertson, K. & Lafond, S. (2008). How to support ELL Students with
Interrupted Formal Education.
○
○
Session 3: Colorin Colorado. (2007). How to Reach Out to Parents of ELLs.
○
○
Session 3: Dilg, M. (2003). From home to school and home again.
○
○
Session 3: Harper, C. & de Jong, E. (2004). Misconceptions about teaching English
language learners.
○
○
Session 4: Brown, H. (2000). Section on “Communicative Competence.” In Principles of
language learning and teaching
○
○
Session 4: Crawford, J. (2004). Chapter 8: Basic Research in Language Acquisition.
○
○
Session 4: Meyer, C. (2009). The study of language.
○
○
Session 5: Peregoy, S. F. & Boyle, O. F. (2008). Second language acquisition.
○
○
Session 5: Finegan, E. (2004). Acquiring first and second languages.
○
○
Session 6: Calderon, M., August, D., Cheung, A., Durán, D., & Madden, N. (2005).
Bringing words to life in classrooms with English language learners.
○
○
Session 6: Lems, K., Miller, L. D., & Soro, T. M. (2010). ELL oracy: Listening
comprehension and oral language development.
○
○
Session 7: Carlo, M. S., August, D., & Snow, C. (2005). Sustained vocabulary-learning
strategy instruction for English Language Learners.
○
○
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —67
Problematic
Useful
Session 7: Silbold, C. (2011) Building English language learners’ academic vocabulary:
Strategies & tips.
○
○
Session 8: de Jong, E. J. & Harper, C. A. (2005). Preparing mainstream teachers for
English language learners: Is being a good teacher good enough?
○
○
Session 8: Echevarria, J. & Graves, A. (2011). Sheltered instruction in the content
areas.
○
○
Session 9: Himmel, J. (2012). Language objectives: The key to effective content area
instruction for ELs.
○
○
Session 9: Varela, E. (2010). Mainstreaming ELLs into grade-level classes.
○
○
Session 10: Coleman, R. & Goldenberg, C. (2010). What does research say about
effective practices for ELLs?
○
○
Session 10: Gottlieb, M., Katz, A. & Ernst-Slavit, G. (2009). Rethinking the language of
school.
○
○
Session 11: Misco, T. & Castaneda, M.E. (2009). Now what should I do for ELLs? :
Reconceptualizing social studies curriculum design for ELL education.
○
○
Session 11: Schleppegrell, M. J. (2001). Linguistic features of the language of schooling.
○
○
Session 12: Abedi, J. (2002). Assessment and accommodation of English language
learners: Issues, concerns, and recommendations.
○
○
Session 12: Gottlieb, M. (2006). Classroom assessment.
○
○
Session 13: Cohen, J. (2007). A case study of a high school English language learner
and his reading.
○
○
Session 13: Singhal, M. (2005). Process, practice, and pedagogy.
○
○
Session 14: Brisk, M.E., Horan, D.A., & McDonald, E. (2007). A scaffolded approach to
learning to write.
○
○
Session 14: DelliCarpini, M. (2009). Writing in the ESL classroom: Confessions of a
guilty teacher.
○
○
Session 15: Echevarria, J., Vogt, M., & Short, D. J. (2004). Introducing sheltered
instruction.
○
○
Session 15: World-Class Instructional Design and Assessment (WIDA). (2012). Focus on
differentiation part 1.
○
○
Comments:
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —68
PART V: All Aspects of the RETELL Course
This final set of questions asks you to consider the course as a whole.
15. Select one rating that best reflects how strongly you agree or disagree with each of the statements listed
below. Select only one rating for each statement.
Strongly
Disagree
Disagree
Agree
Strongly
Agree
My instructor is knowledgeable about sheltered English
immersion.
I was taught skills that I can apply directly to in my own
classroom.
I was given opportunities to practice instructional
strategies.
The content of this course enhances what I already know
about sheltered English immersion.
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
The content of this course repeats what I’ve already
learned about sheltered English immersion.
○
○
○
○
This course was a good use of my time.
○
○
○
○
Comments:
16. Select the rating that most accurately reflects how relevant you found each of the following aspects of the
course for your instruction of English language learners. Select only one rating for each statement.
Not
Relevant
A Little
Relevant
Pretty
Relevant
Very
Relevant
Did not
occur
○
○
○
○
○
○
○
○
○
○
Interactions with colleagues during the online
portion of the course (e.g., e-mails, comments)
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Online interactions with the instructor apart
from the online presentations
○
○
○
○
○
The instructor’s face-to-face presentations
Interactions with colleagues during the face-toface portion of the course
The online portion of the course
Posting comments and responses online
Comments:
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —69
17. The following is a list of all course sessions. Please mark only the sessions you found particularly useful for
providing instruction to English language learners. It is not necessary to mark every session.
Particularly Useful for ELL Instruction
Session 1: Examining Data and Policies Relevant to English Language Learners
Session 2: Diversity within English Language Learner Populations
Session 3: Cultural and Social Aspects of Teaching in the SEI Classroom and
School
Session 4: The Role of Language and Linguistics in Instructing ELLs
Session 5: Second Language Acquisition in the SEI Classroom
Session 6: Oral Language, Listening Comprehension and Vocabulary
Development in the SEI Classroom
Session 7: The Role of Vocabulary Development in Unlocking Content
Session 8: Sheltering Content for ELLs in the Classroom I
Session 9: Sheltering Content for ELLs in the Classroom II
Session 10: Characteristics of Academic Language
Session 11: Specialized Academic Content Language
Session 12: Assessment of ELLs in the SEI Classroom
Session 13: Literacy: Reading in the SEI Classroom
Session 14: Literacy: Writing in the SEI Classroom
Session 15: The Benefits of Standards-Based Teaching for ELLs
Session 16 (if completed): Effective Practices for the SEI Classroom
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Comments:
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —70
18. Select the rating that best reflects how much you think the course so far has contributed to your
understanding of each of the following topics. Select only one rating for each statement.
Not At All
A Little
A Fair
Amount
A Great Deal
Topic not
addressed
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
How to apply sheltered English immersion to
reading and writing instruction in the
classroom
○
○
○
○
○
How to differentiate instruction when working
with ELLs at various English proficiency levels
○
○
○
○
○
How to address the needs of diverse ELL
populations
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
How to meet the needs of ELLs in the
classroom
Vocabulary instruction for ELLs
How ELLs acquire a second language
The social and cultural issues that affect ELL
students
How to assess ELLs in the content areas
How to teach academic language to ELLs
Comments:
19. Given your expertise as a teacher, of the topics presented so far are there any that should have been delved
into deeper even at the expense of skipping other topics? If so, please briefly explain why.
20. What did you like most about this course? What would you have liked to do more of?
21. Is there anything else you would like to share with us to help us improve the RETELL course?
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —71
Appendix B: Participant Survey Results
PART 1: FACE-TO-FACE PORTION OF THE RETELL COURSE
1. Select the one rating that best reflects your opinion of the pace of the course.
Much too
A little too
A little too Much too
Location
n
Slow
slow
Just Right fast
fast
Somerville
18
0%
0%
11%
50%
39%
Lowell
15
0%
0%
0%
40%
60%
Boston/Worcester
22
0%
0%
27%
32%
41%
Total
55
0%
0%
15%
40%
45%
2a. Time allocation to Instructor presenting/lecturing
Too little
Right
Too much
Location
n
time
amount
time
Somerville
18
33%
61%
6%
Lowell
15
0%
93%
7%
Boston/Worcester
23
22%
65%
13%
Total
56
20%
71%
9%
2b. Time allocation to small group work/discussions
Too little
Right
Too much
Location
n
time
amount
time
Somerville
18
33%
67%
0%
Lowell
16
13%
88%
0%
Boston/Worcester
23
17%
78%
4%
Total
57
21%
77%
2%
2c. Time allocation to discussion around how the apply strategies being learned to classroom
instruction
Too little
Right
Too much
Location
n
time
amount
time
Somerville
18
44%
44%
11%
Lowell
16
50%
50%
0%
Boston/Worcester
22
68%
32%
0%
Total
56
55%
41%
4%
2d. Time allocation to practice applying strategies to classroom instruction
Too little
Right
Too much
Location
n
time
amount
time
Somerville
17
65%
35%
0%
Lowell
16
69%
31%
0%
Boston/Worcester
23
52%
48%
0%
Total
56
61%
39%
0%
2e. Time allocation to whole class discussion.
Too little
Right
Too much
Location
n
time
amount
time
Somerville
18
11%
83%
6%
Lowell
16
13%
81%
6%
Boston/Worcester
23
4%
87%
9%
Total
57
9%
84%
7%
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —72
2f. Time allocation to interacting with colleagues.
Too little
Right
Location
n
time
amount
Somerville
18
17%
83%
Lowell
16
38%
63%
Boston/Worcester
23
9%
87%
Total
57
19%
79%
2g. Time allocation to individual work (in class).
Too little
Right
Location
n
time
amount
Somerville
17
47%
53%
Lowell
15
47%
53%
Boston/Worcester
23
39%
52%
Total
55
44%
53%
2h. Time allocation to individual work (homework).
Too little
Right
Location
n
time
amount
Somerville
18
28%
28%
Lowell
16
25%
0%
Boston/Worcester
23
43%
0%
Total
57
33%
9%
2i. Time allocation to reading.
Too little
Right
Location
n
time
amount
Somerville
18
22%
17%
Lowell
16
25%
19%
Boston/Worcester
22
23%
23%
Total
56
23%
20%
2j. Time allocation to videos.
Too little
Right
Location
n
time
amount
Somerville
17
41%
59%
Lowell
10
50%
50%
Boston/Worcester
21
29%
71%
Total
48
38%
63%
3a. Usefulness of instructor presenting/lecturing.
Not useful
A little
Location
n
at all
useful
Somerville
18
0%
17%
Lowell
14
0%
21%
Boston/Worcester
22
5%
5%
Total
54
2%
13%
American Institutes for Research
Too much
time
0%
0%
4%
2%
Too much
time
0%
0%
9%
4%
Too much
time
44%
75%
57%
58%
Too much
time
61%
56%
55%
57%
Too much
time
0%
0%
0%
0%
Pretty
Extremely
useful
useful
Unknown
44%
39%
0%
43%
36%
0%
32%
59%
0%
39%
46%
0%
MA SEI Endorsement Pilot Evaluation —73
3b. Usefulness of small group work/discussions.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Unknown
Somerville
18
0%
17%
33%
50%
0%
Lowell
16
0%
31%
50%
19%
0%
Boston/Worcester
23
0%
9%
52%
39%
0%
Total
57
0%
18%
46%
37%
0%
3c. Usefulness of discussion around how to apply strategies being learned to classroom instruction.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Unknown
Somerville
18
0%
11%
28%
61%
0%
Lowell
16
0%
31%
50%
13%
6%
Boston/Worcester
22
0%
14%
27%
55%
5%
Total
56
0%
18%
34%
45%
4%
3d. Usefulness of practice during the course applying strategies to classroom instruction.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Unknown
Somerville
18
0%
6%
33%
56%
6%
Lowell
16
0%
38%
44%
13%
6%
Boston/Worcester
22
14%
18%
18%
50%
0%
Total
56
5%
20%
30%
41%
4%
3e. Usefulness of whole class discussion.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Unknown
Somerville
18
0%
17%
67%
17%
0%
Lowell
16
0%
25%
63%
13%
0%
Boston/Worcester
22
5%
14%
41%
41%
0%
Total
56
2%
18%
55%
25%
0%
3f. Usefulness of interacting with colleagues.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Unknown
Somerville
18
0%
6%
33%
61%
0%
Lowell
16
0%
25%
56%
19%
0%
Boston/Worcester
23
0%
17%
43%
39%
0%
Total
57
0%
16%
44%
40%
0%
3g. Usefulness of individual work (in class).
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Unknown
Somerville
18
6%
33%
44%
11%
6%
Lowell
15
13%
33%
40%
7%
7%
Boston/Worcester
23
9%
17%
39%
26%
9%
Total
56
9%
27%
41%
16%
7%
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —74
3h. Usefulness of individual work (homework).
Not useful
Location
n
at all
Somerville
17
6%
Lowell
15
13%
Boston/Worcester
22
14%
Total
54
11%
3i. Usefulness of reading.
Not useful
Location
n
at all
Somerville
18
0%
Lowell
16
6%
Boston/Worcester
22
5%
Total
56
4%
3j. Usefulness of videos.
Not useful
Location
n
at all
Somerville
18
0%
Lowell
12
25%
Boston/Worcester
20
10%
Total
50
10%
A little
useful
59%
53%
36%
48%
Pretty
Extremely
useful
useful
Unknown
35%
0%
0%
33%
0%
0%
32%
14%
5%
33%
6%
2%
28%
25%
36%
30%
Pretty
Extremely
useful
useful
Unknown
61%
11%
0%
69%
0%
0%
41%
18%
0%
55%
11%
0%
33%
17%
15%
22%
Pretty
Extremely
useful
useful
Unknown
44%
11%
11%
42%
0%
17%
45%
25%
5%
44%
14%
10%
A little
useful
A little
useful
PART 2: ONLINE PORTION OF THE RETELL COURSE
4a. Average amount of time for PowerPoint presentations
Location
n
<30 min.
Somerville
17
47%
Lowell
16
25%
Boston/Worcester
23
17%
Total
56
29%
4b. Average amount of time for activities.
30-60 min.
41%
50%
30%
39%
60-90
min.
6%
13%
30%
18%
90-120
min.
6%
6%
4%
5%
>120 min.
0%
6%
17%
9%
Location
n
<30 min.
Somerville
18
11%
Lowell
16
0%
Boston/Worcester
21
10%
Total
55
7%
4c. Average amount of time for assignments.
30-60 min.
50%
19%
19%
29%
60-90
min.
22%
31%
14%
22%
90-120
min.
17%
25%
14%
18%
>120 min.
0%
25%
43%
24%
Location
n
Somerville
Lowell
Boston/Worcester
Total
30-60 min.
33%
0%
9%
15%
60-90
min.
44%
7%
18%
24%
90-120
min.
17%
53%
9%
24%
>120 min.
0%
40%
64%
36%
American Institutes for Research
18
15
22
55
<30 min.
6%
0%
0%
2%
MA SEI Endorsement Pilot Evaluation —75
4d. Average amount of time for interactions with the instructor.
60-90
90-120
Location
n
<30 min.
30-60 min.
min.
min.
>120 min.
Somerville
16
81%
13%
0%
6%
0%
Lowell
11
82%
9%
9%
0%
0%
Boston/Worcester
22
68%
18%
5%
0%
9%
Total
49
76%
14%
4%
2%
4%
4e. Average amount of time for illustration of how strategies can be applied to the classroom.
60-90
90-120
Location
n
<30 min.
30-60 min.
min.
min.
>120 min.
Somerville
16
38%
38%
19%
6%
0%
Lowell
12
33%
42%
17%
8%
0%
Boston/Worcester
21
29%
33%
14%
10%
14%
Total
49
33%
37%
16%
8%
6%
4f. Average amount of time for reading.
60-90
90-120
Location
n
<30 min.
30-60 min.
min.
min.
>120 min.
Somerville
17
12%
12%
24%
29%
24%
Lowell
16
0%
6%
44%
31%
19%
Boston/Worcester
23
4%
9%
35%
22%
30%
Total
56
5%
9%
34%
27%
25%
4g. Average amount of time for posting comments and responses online.
60-90
90-120
Location
n
<30 min.
30-60 min.
min.
min.
>120 min.
Somerville
17
12%
76%
6%
6%
0%
Lowell
16
6%
38%
31%
6%
19%
Boston/Worcester
23
13%
22%
30%
13%
22%
Total
56
11%
43%
23%
9%
14%
5a. Usefulness of PowerPoint presentations
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Unknown
Somerville
18
11%
67%
17%
6%
0%
Lowell
16
19%
6%
50%
25%
0%
Boston/Worcester
23
9%
4%
57%
30%
0%
Total
57
12%
25%
42%
21%
0%
5b. Usefulness of activities.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Unknown
Somerville
18
6%
17%
56%
17%
6%
Lowell
15
7%
47%
40%
7%
0%
Boston/Worcester
21
5%
14%
48%
33%
0%
Total
54
6%
24%
48%
20%
2%
5c. Usefulness of assignments.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Unknown
Somerville
18
6%
44%
39%
11%
0%
Lowell
15
13%
60%
27%
0%
0%
Boston/Worcester
23
4%
22%
52%
22%
0%
Total
56
7%
39%
41%
13%
0%
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —76
5d. Usefulness of interactions with the instructor.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Somerville
18
17%
6%
33%
39%
Lowell
15
0%
13%
60%
27%
Boston/Worcester
23
9%
4%
35%
52%
Total
56
9%
7%
41%
41%
5e. Usefulness of illustration of how strategies can be applied to the classroom.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Somerville
18
0%
28%
17%
50%
Lowell
16
0%
50%
38%
6%
Boston/Worcester
22
0%
14%
32%
50%
Total
56
0%
29%
29%
38%
5f. Usefulness of reading.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Somerville
18
6%
33%
56%
6%
Lowell
16
6%
19%
69%
6%
Boston/Worcester
23
9%
26%
43%
22%
Total
57
7%
26%
54%
12%
5g. Usefulness of posting comments and responses online.
Not useful
A little
Pretty
Extremely
Location
n
at all
useful
useful
useful
Somerville
18
22%
50%
17%
11%
Lowell
16
50%
38%
13%
0%
Boston/Worcester
22
27%
32%
32%
5%
Total
56
32%
39%
21%
5%
6a. The PowerPoint presentations were easy to follow.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
18
17%
22%
44%
17%
Lowell
16
0%
13%
63%
25%
Boston/Worcester
23
0%
4%
61%
35%
Total
57
5%
12%
56%
26%
6b. I felt as engaged in the online content as I was in the face-to-face session.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
18
56%
33%
11%
0%
Lowell
16
50%
31%
13%
6%
Boston/Worcester
23
17%
70%
13%
0%
Total
57
39%
47%
12%
2%
6c. I found it easy to navigate the course materials.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
18
28%
56%
11%
6%
Lowell
16
44%
44%
13%
0%
Boston/Worcester
23
39%
39%
17%
4%
Total
57
37%
46%
14%
4%
American Institutes for Research
Unknown
6%
0%
0%
2%
Unknown
6%
6%
5%
5%
Unknown
0%
0%
0%
0%
Unknown
0%
0%
5%
2%
N/A
0%
0%
0%
0%
N/A
0%
0%
0%
0%
N/A
0%
0%
0%
0%
MA SEI Endorsement Pilot Evaluation —77
6d. I used the online platform to communicate with other participants when not required.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
N/A
Somerville
18
50%
39%
11%
0%
Lowell
16
56%
38%
0%
0%
Boston/Worcester
22
45%
36%
9%
0%
Total
56
50%
38%
7%
0%
6e. The instructor made a clear connection between the online and face-to-face content.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
N/A
Somerville
18
11%
17%
61%
11%
Lowell
16
19%
31%
38%
13%
Boston/Worcester
23
4%
26%
61%
9%
Total
57
11%
25%
54%
11%
0%
6%
9%
5%
0%
0%
0%
0%
PART 3: INSTRUCTOR, COURSE ACTIVITIES, AND CLASSROOM CULTURE
8a. It was easy to see the instructor’s presentation.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
18
0%
6%
50%
44%
Lowell
15
0%
0%
73%
27%
Boston/Worcester
23
0%
0%
43%
57%
Total
56
0%
2%
54%
45%
8b. The room was clean.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
18
0%
0%
28%
72%
Lowell
15
0%
0%
93%
7%
Boston/Worcester
23
4%
0%
39%
57%
Total
56
2%
0%
50%
48%
8c. The room was comfortable.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
18
0%
11%
33%
56%
Lowell
16
0%
6%
88%
6%
Boston/Worcester
23
4%
22%
43%
30%
Total
57
2%
14%
53%
32%
8d. The organization of the room facilitated collaboration with my colleagues.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
18
0%
0%
33%
67%
Lowell
16
0%
6%
88%
6%
Boston/Worcester
23
0%
9%
43%
48%
Total
57
0%
5%
53%
42%
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —78
9a. How often the instructor slowed down or moved on based upon the mood in the room.
Sometim
Location
n
Never
Rarely
es
Frequently N/A
Somerville
18
0%
0%
44%
56%
Lowell
16
0%
6%
31%
63%
Boston/Worcester
23
4%
13%
22%
61%
Total
57
2%
7%
32%
60%
9b. How often the instructor demonstrated excitement about the content.
Sometim
Location
n
Never
Rarely
es
Frequently N/A
Somerville
18
0%
6%
0%
94%
Lowell
16
0%
0%
19%
81%
Boston/Worcester
23
0%
0%
0%
100%
Total
57
0%
2%
5%
93%
9c. How often the instructor engaged participants to ask questions and/or share expertise.
Sometim
Location
n
Never
Rarely
es
Frequently N/A
Somerville
18
0%
0%
11%
89%
Lowell
16
0%
0%
25%
75%
Boston/Worcester
23
0%
9%
9%
83%
Total
57
0%
4%
14%
82%
9d. How often the instructor checked for understanding.
Sometim
Location
n
Never
Rarely
es
Frequently N/A
Somerville
18
0%
0%
28%
72%
Lowell
16
0%
6%
13%
81%
Boston/Worcester
23
0%
4%
30%
65%
Total
57
0%
4%
25%
72%
9e. How often the instructor posted course goals and objectives.
Sometim
Location
n
Never
Rarely
es
Frequently N/A
Somerville
18
0%
6%
11%
83%
Lowell
16
0%
0%
13%
81%
Boston/Worcester
23
9%
13%
9%
65%
Total
57
4%
7%
11%
75%
10a. How well the instructor summarized readings.
Did not
Location
n
Poorly
Adequately Well
Very Well
observe
Somerville
18
6%
17%
50%
28%
Lowell
16
0%
63%
19%
13%
Boston/Worcester
22
5%
18%
55%
23%
Total
56
4%
30%
43%
21%
10b. How well the instructor gave instructions.
Did not
Location
n
Poorly
Adequately Well
Very Well
observe
Somerville
18
6%
22%
39%
33%
Lowell
15
0%
27%
53%
20%
Boston/Worcester
21
14%
14%
43%
29%
Total
54
7%
20%
44%
28%
American Institutes for Research
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
6%
4%
4%
0%
6%
0%
2%
0%
0%
0%
0%
MA SEI Endorsement Pilot Evaluation —79
10c. How well the instructor facilitated discussions.
Did not
Location
n
Poorly
Adequately Well
Very Well
observe
Somerville
18
0%
6%
44%
50%
0%
Lowell
16
0%
25%
44%
31%
0%
Boston/Worcester
23
4%
17%
35%
43%
0%
Total
57
2%
16%
40%
42%
0%
10d. How well the instructor responded to questions.
Did not
Location
n
Poorly
Adequately Well
Very Well
observe
Somerville
18
0%
6%
39%
56%
0%
Lowell
16
0%
25%
50%
25%
0%
Boston/Worcester
23
4%
9%
39%
48%
0%
Total
57
2%
12%
42%
44%
0%
10e. How well the instructor handled difference of opinion.
Did not
Location
n
Poorly
Adequately Well
Very Well
observe
Somerville
18
0%
6%
22%
72%
0%
Lowell
16
0%
6%
50%
31%
13%
Boston/Worcester
23
9%
4%
22%
61%
4%
Total
57
4%
5%
30%
56%
5%
10f. How well the instructor linked instructional materials (e.g., readings) to the main ideas being
presented.
Did not
Location
n
Poorly
Adequately Well
Very Well
observe
Somerville
18
0%
22%
22%
56%
0%
Lowell
16
0%
31%
25%
44%
0%
Boston/Worcester
23
4%
17%
22%
57%
0%
Total
57
2%
23%
23%
53%
0%
11a. I did not receive clarification from the instructor about something that confused me.
Sometim
Location
n
Never
Rarely
es
Frequently N/A
Somerville
18
56%
17%
17%
6%
6%
Lowell
16
56%
31%
13%
0%
0%
Boston/Worcester
23
22%
35%
35%
0%
9%
Total
57
42%
28%
23%
2%
5%
11b. I was uncertain about what to do during an activity after instructions were given.
Sometim
Location
n
Never
Rarely
es
Frequently N/A
Somerville
18
17%
33%
39%
11%
0%
Lowell
16
0%
31%
38%
31%
0%
Boston/Worcester
23
9%
22%
48%
17%
4%
Total
57
9%
28%
42%
19%
2%
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —80
11c. I did not get the opportunity to share my comments.
Sometim
Location
n
Never
Rarely
es
Frequently N/A
Somerville
18
89%
11%
0%
0%
Lowell
16
50%
44%
6%
0%
Boston/Worcester
23
35%
43%
4%
4%
Total
57
56%
33%
4%
2%
11d. I did not feel the main ideas from a reading were clear, even after discussing it in class.
Sometim
Location
n
Never
Rarely
es
Frequently N/A
Somerville
18
44%
33%
22%
0%
Lowell
15
20%
40%
33%
7%
Boston/Worcester
23
26%
30%
22%
13%
Total
56
30%
34%
25%
7%
11e. My questions to the instructor were not answered thoroughly.
Sometim
Location
n
Never
Rarely
es
Frequently N/A
Somerville
17
71%
18%
12%
0%
Lowell
16
63%
31%
6%
0%
Boston/Worcester
23
48%
22%
17%
4%
Total
56
59%
23%
13%
2%
PART 4: THE RETELL READINGS
13a. In general, the readings were too academic.
Strongly
Location
n
disagree
Disagree
Agree
Somerville
17
6%
24%
53%
Lowell
15
7%
53%
33%
Boston/Worcester
23
9%
35%
35%
Total
55
7%
36%
40%
13b. In general, the readings were related to the course content.
Strongly
Location
n
disagree
Disagree
Agree
Somerville
18
6%
6%
83%
Lowell
15
7%
7%
73%
0%
0%
13%
5%
0%
0%
9%
4%
0%
0%
9%
4%
Strongly
agree
18%
7%
22%
16%
Strongly
agree
6%
13%
Boston/Worcester
23
4%
0%
74%
22%
Total
56
5%
4%
77%
14%
13c. In general, the readings were easy to understand.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
17
6%
53%
29%
12%
Lowell
15
7%
20%
73%
0%
Boston/Worcester
23
17%
39%
39%
4%
Total
55
11%
38%
45%
5%
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —81
13d. In general, the readings were applicable to my instruction of English language learners.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
16
6%
19%
75%
0%
Lowell
15
0%
27%
67%
7%
Boston/Worcester
23
0%
30%
61%
9%
Total
54
2%
26%
67%
6%
13e. In general, the readings helped me to better understand sheltered English immersion.
Strongly
Agr Strongly
Location
n
disagree
Disagree
ee
agree
Somerville
17
6%
12% 82%
0%
Lowell
15
13%
7% 67%
13%
Boston/Worcester
23
9%
13% 57%
22%
Total
55
9%
11% 67%
13%
14. Usefulness of readings
Reading
S1. ESE (2012)
S2. DeCapua & Marshall
(2011)
S2. Robertson & Lafond
(2008)
S3. Colorin Colorado (2007)
S3. Dilg (2003)
S3. Harper & de Jong (2004)
S4. Brown (2000)
S4. Crawford (2004)
S4. Meyer (2009)
S5. Peregoy & Boyle (2008)
S5. Finegan (2004)
S6. Calderon et al. (2005)
S6. Lems et al. (2010)
S7. Carlo et al. (2005)
S7. Silbold (2011)
S8. de Jong & Harper (2005)
S8. Echevarria & Graves
(2011)
S9. Himmel (2012)
S9. Varela (2010)
S10. Coleman & Goldenberg
(2010)
S10. Gottlieb et al. (2009)
S11. Misco & Castaneda
(2008)
S11. Schleppegrell (2001)
S12. Abedi (2002)
S12. Gottlieb (2006)
American Institutes for Research
n
% Problematic
% Useful
31
16%
84%
37
11%
89%
33
37
34
39
32
32
31
33
35
38
29
30
38
32
15%
14%
21%
8%
38%
50%
52%
30%
34%
18%
34%
30%
16%
19%
85%
86%
79%
92%
63%
50%
48%
70%
66%
82%
66%
70%
84%
81%
30
31
25
30%
26%
32%
70%
74%
68%
30
33
30%
36%
70%
64%
26
30
33
37
19%
63%
21%
19%
81%
37%
79%
81%
MA SEI Endorsement Pilot Evaluation —82
14. Usefulness of readings
Reading
S13. Cohen (2007)
S13. Singhal (2005)
S14. Brisk et al. (2007)
S14. DelliCarpini (2009)
S15. Echevarria et al. (2004)
S15. WIDA (2012)
n
% Problematic
% Useful
31
35%
65%
29
34%
66%
32
28%
72%
32
19%
81%
30
27%
73%
30
23%
77%
PART 5: ALL ASPECTS OF THE RETELL COURSE
15a. My instructor is knowledgeable about sheltered English immersion.
Strongly
Location
n
disagree
Disagree
Agree
Somerville
18
0%
0%
39%
Lowell
16
0%
0%
44%
Boston/Worcester
23
0%
0%
26%
Total
57
0%
0%
35%
15b. I was taught skills that I can apply directly to my own classroom.
Strongly
Location
n
disagree
Disagree
Agree
Somerville
18
0%
11%
61%
Lowell
13
0%
23%
62%
Boston/Worcester
23
4%
4%
39%
Total
54
2%
11%
52%
Strongly
agree
61%
56%
74%
65%
Strongly
agree
28%
15%
52%
35%
15c. I was given opportunities to practice instructional strategies.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
18
6%
22%
56%
17%
Lowell
15
0%
60%
40%
0%
Boston/Worcester
23
4%
17%
39%
39%
Total
56
4%
30%
45%
21%
15d. The content of this course enhances what I already know about sheltered English immersion.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
18
0%
6%
72%
22%
Lowell
16
0%
13%
75%
13%
Boston/Worcester
23
4%
13%
52%
30%
Total
57
2%
11%
65%
23%
15e. The content of this course repeats what I’ve already learned about sheltered English immersion.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
17
0%
29%
47%
24%
Lowell
14
0%
29%
57%
14%
Boston/Worcester
23
13%
39%
22%
26%
Total
54
6%
33%
39%
22%
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —83
15f. This course was a good use of my time.
Strongly
Strongly
Location
n
disagree
Disagree
Agree
agree
Somerville
17
0%
24%
53%
24%
Lowell
16
19%
44%
38%
0%
Boston/Worcester
23
13%
17%
35%
35%
Total
56
11%
27%
41%
21%
16a. Relevance of the instructor’s face-to-face presentations for instruction of ELLs.
Not
A little
Pretty
Very
Did not
Location
n
relevant
relevant
relevant
relevant
occur
Somerville
18
0%
6%
33%
61%
0%
Lowell
16
0%
6%
63%
31%
0%
Boston/Worcester
23
0%
4%
26%
70%
0%
Total
57
0%
5%
39%
56%
0%
16b. Relevance of interactions with colleagues during the face-to-face portion of the course for
instruction of ELLs.
Not
A little
Pretty
Very
Did not
Location
n
relevant
relevant
relevant
relevant
occur
Somerville
18
0%
0%
39%
61%
0%
Lowell
16
0%
25%
44%
31%
0%
Boston/Worcester
23
4%
9%
13%
74%
0%
Total
57
2%
11%
30%
58%
0%
16c. Relevance of the online portion of the course for instruction of ELLs.
Not
A little
Pretty
Very
Did not
Location
n
relevant
relevant
relevant
relevant
occur
Somerville
18
33%
39%
28%
0%
0%
Lowell
16
25%
38%
31%
6%
0%
Boston/Worcester
23
17%
39%
26%
17%
0%
Total
57
25%
39%
28%
9%
0%
16d. Relevance of posting comments and responses online for instruction of ELLs.
Not
A little
Pretty
Very
Did not
Location
n
relevant
relevant
relevant
relevant
occur
Somerville
18
28%
39%
22%
11%
0%
Lowell
16
50%
50%
0%
0%
0%
Boston/Worcester
23
26%
35%
22%
17%
0%
Total
57
33%
40%
16%
11%
0%
16e. Relevance of interactions with colleagues during the online portion of the course for instruction
of ELLs.
Not
A little
Pretty
Very
Did not
Location
n
relevant
relevant
relevant
relevant
occur
Somerville
18
33%
22%
28%
6%
11%
Lowell
15
60%
33%
0%
0%
7%
Boston/Worcester
23
39%
26%
26%
9%
0%
Total
56
43%
27%
20%
5%
5%
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —84
16f. Relevance of online interactions with the instructor apart from the online presentations for
instruction of ELLs.
Not
A little
Pretty
Very
Did not
Location
n
relevant
relevant
relevant
relevant
occur
Somerville
18
28%
17%
28%
11%
Lowell
15
40%
0%
7%
7%
Boston/Worcester
23
13%
30%
26%
17%
Total
56
25%
18%
21%
13%
17. Usefulness of sessions for ELL instruction.
Somerville Lowell Boston/Worcester
Useful
Useful
Useful
Session 1
10
4
12
Session 2
5
4
11
Session 3
12
9
17
Session 4
6
3
14
Session 5
10
11
14
Session 6
11
9
13
Session 7
13
11
16
Session 8
7
11
10
Session 9
9
9
8
Session 10
9
9
11
Session 11
8
3
9
Session 12
13
6
13
Session 13
15
9
13
Session 14
13
10
15
Session 15
5
4
8
Session 16
3
2
5
17%
47%
13%
23%
Total
26
20
38
23
35
33
40
28
26
29
20
32
37
38
17
10
18a. The course contributed to my understanding of how to meet the needs of ELLs in the classroom.
A fair
A great
Topic not
Location
n
Not at all A little
amount
deal
addressed
Somerville
18
6%
11%
61%
22%
0%
Lowell
15
7%
27%
33%
33%
0%
Boston/Worcester
22
0%
18%
23%
59%
0%
Total
55
4%
18%
38%
40%
0%
18b. The course contributed to my understanding of vocabulary instruction for ELLs.
A fair
A great
Topic not
Location
n
Not at all A little
amount
deal
addressed
Somerville
18
0%
22%
28%
50%
0%
Lowell
15
7%
13%
33%
47%
0%
Boston/Worcester
22
0%
14%
14%
73%
0%
Total
55
2%
16%
24%
58%
0%
American Institutes for Research
MA SEI Endorsement Pilot Evaluation —85
18c. The course contributed to my understanding of how ELLs acquire a second language.
A fair
A great
Topic not
Location
n
Not at all A little
amount
deal
addressed
Somerville
18
6%
6%
67%
22%
Lowell
15
0%
20%
40%
40%
Boston/Worcester
22
9%
14%
32%
45%
Total
55
5%
13%
45%
36%
18d. The course contributed to my understanding of the social and cultural issues that affect ELL
students.
A fair
A great
Topic not
Location
n
Not at all A little
amount
deal
addressed
Somerville
18
11%
17%
44%
28%
Lowell
15
7%
20%
40%
33%
Boston/Worcester
22
0%
14%
27%
59%
Total
55
5%
16%
36%
42%
18e. The course contributed to my understanding of how to apply SEI to reading and writing
instruction in the classroom.
A fair
A great
Topic not
Location
n
Not at all A little
amount
deal
addressed
Somerville
18
6%
33%
28%
33%
Lowell
15
0%
20%
60%
20%
Boston/Worcester
21
0%
24%
24%
52%
Total
54
2%
26%
35%
37%
18f. The course contributed to my understanding of how to differentiate instruction when working
with ELLs at various English proficiency levels.
A fair
A great
Topic not
Location
n
Not at all A little
amount
deal
addressed
Somerville
18
0%
39%
39%
22%
Lowell
15
7%
47%
20%
27%
Boston/Worcester
22
9%
14%
32%
45%
Total
55
5%
31%
31%
33%
18g. The course contributed to my understanding of how to address the needs of diverse ELL
populations.
A fair
A great
Topic not
Location
n
Not at all A little
amount
deal
addressed
Somerville
18
11%
22%
39%
28%
Lowell
15
7%
33%
33%
27%
Boston/Worcester
22
5%
23%
32%
41%
Total
55
7%
25%
35%
33%
18h. The course contributed to my understanding of how to assess ELLs in the content areas.
A fair
A great
Topic not
Location
n
Not at all A little
amount
deal
addressed
Somerville
18
6%
28%
39%
28%
Lowell
16
6%
31%
44%
19%
Boston/Worcester
22
9%
27%
23%
41%
Total
56
7%
29%
34%
30%
18i. The course contributed to my understanding of how to teach academic language to ELLs.
A fair
A great
Topic not
Location
n
Not at all A little
amount
deal
addressed
Somerville
18
0%
22%
50%
28%
Lowell
15
0%
27%
60%
13%
Boston/Worcester
22
0%
18%
32%
50%
Total
55
0%
22%
45%
33%
American Institutes for Research
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
0%
MA SEI Endorsement Pilot Evaluation —86
1000 Thomas Jefferson Street NW
Washington, DC 20007-3835
202.403.5000 | TTY: 877.334.3499
www.air.org