RUNNING HEAD: On the Development of Reflective Lessons
FROM FORMAL EMBEDDED ASSESSMENTS TO REFLECTIVE LESSONS: THE
DEVELOPMENT OF FORMATIVE ASSESSMENT SUITES
Carlos Ayala
Sonoma State University
Richard J. Shavelson
Stanford University
Paul Brandon
University of Hawaii
Yue Yin
University of Hawaii
Erin M. Furtak
Max Planck Institute for Human Development
Maria Araceli Ruiz-Primo
University of Colorado at Denver and Health Sciences Center
Donald Young
University of Hawaii
Miki Tomita
Stanford University and University of Hawaii
On the Development of Reflective Lessons
Abstract
The idea that formative assessments embedded in a curriculum could help guide
teachers toward better instructional practices that lead to student learning has taken center
stage in science assessment research (Atkins & Coffey, 2003; Black & Wiliam, 1998). In
order to embed formative assessments in a curriculum, curriculum developers and
assessment specialists need to collaborate to create these assessment tasks. This paper
describes the development of the formal embedded formative assessment suites and
implementation plans that were designed for the Romance Study using a Science
Achievement Framework. It describes the fundamental shift away from “summative
assessment scripts” to reflective lessons scripts. Samples of the assessment tasks and
implementation plans are described along with the rationale for why these tasks were
selected and where these tasks were placed in the curriculum. Finally we conclude about
how to successfully embed formative assessments in new or existing curriculum and how
to help teacher use these assessments. For example, we point out the critical importance
of collaboration, of professional development aimed at enabling teachers to
reconceptualize the role of assessments in their teaching, of linking formative
assessments to overall goals, and of providing a learning trajectory as reference for
teachers to locate students’ ideas developmentally and provide feedback accordingly.
2
On the Development of Reflective Lessons
Introduction
While some empirical evidence suggests that the use of curriculum-embedded
(formative) assessment leads to increased learning (Bell & Cowie, 2001; Black &
Wiliam, 1998; 2004; Shepherd, 2000), how these formative assessments are designed and
used by curriculum developers and then eventually implemented by teachers is poorly
understood. Moreover, assessment specialists and curriculum developers rarely
collaborate on assessment development, let alone on embedding formative assessments in
existing curricula, on preparing end-unit summative assessments, or in thinking about
how teachers might use these assessments.
The “Romance” project, described by Shavelson et al. (this issue), attempted to
fill this knowledge gap. Here we describe how we went about building, refining and
embedding formative assessments into an inquiry science curriculum. Most noteworthy is
the use of an interdisciplinary Assessment Development Team that laid out a blueprint
for formative assessment definition and development. We report the fruits of this
collaboration, and how they “matured” over time. We then describe how we trained
teachers to use formative assessments in their inquiry science teaching. Finally, we draw
conclusions about these activities with the goal of informing collaborative assessment
and curriculum developers with knowledge of how to successfully embed formal
formative assessments in a new or existing curriculum and of the challenges of helping
teachers use this emerging technology.
Background
We define the activities that we developed in this project as embedded formal
formative assessments. Formal because we crafted assessment tasks which would be
3
On the Development of Reflective Lessons
available for teachers to use at critical times in curriculum implementation; this contrasts
with on-the-fly and planned-for formative assessment that capitalize on informal ongoing
clinical observations or create teachable moments for enhancing students’ understanding
(see Shavelson et al., this issue). They are embedded assessments because they are
inserted into a curriculum to be used at a particular time as opposed to the end of a unit.
They are formative assessments because they are developed to give a snap shot to
students and teachers about what students know and are able to do at a particular time
such that this information could be used to make teaching decisions—what does the
teacher or student do next (see Shavelson et al., this issue, for details).
We chose a curriculum for the project, Foundational Approaches in Science
Teaching (FAST), because the Stanford Educational Assessment Laboratory (SEAL) had
collaborated previously with Curriculum Research & Development Group at the
University of Hawaii (CRDG), the curriculum developers; because previous studies have
supported FAST’s efficacy (Pauls, Young, & Lapitkova, 1999; Tamir & Yamamoto,
1977; Young, 1983;1993); and because various national organizations have found it to
be an exemplary program (U. S. Department of Education’s Expert Panel on
Mathematics and Science Education, 2001; National Staff Development Council, 1999;
for more information, See Shavelson et al., this issue)
The content for this project focused on the conception of “why things sink and
float” that was built up from an understanding of mass, volume, density and relative
density (see Table 1). This curriculum develops students’ science understandings
incrementally in a manner that parallels how science knowledge was developed in the
Western world (cf. King & Brownell, 1966), and as such, students’ understandings of
4
On the Development of Reflective Lessons
why things sink and float are developed via explanations of sinking and floating
phenomena sequentially beginning with the concepts of mass and volume and moving to
the concepts of density and relative density.
--------------------------Insert Table 1 Here
--------------------------Assessment Development
A science achievement framework guided the Assessment Development Team in
the conceptualization, development, and evaluation of embedded and end-of-unit
assessments. We begin by describing the framework and then turn attention to the
Assessment Development Team, which turned out to be a very important, collaborative
infrastructure for the project.
Science Achievement Framework
The project’s assessment development was guided by conceptual framework for
science achievement (see Shavelson et al., this issue).
From the framework, we
conceived and developed embedded and end-of-unit assessments. That is, we conceived
of science achievement as, in significant part, the acquisition of and reasoning with
knowledge. More specifically we conceived of science achievement as comprised of (at
least) four different but overlapping types of knowledge and reasoning: declarative,
procedural, schematic and strategic (see Figure 1 in Shavelson et al., this issue).
Declarative knowledge and reasoning is “knowing that”—for example, knowing
that force is a push or pull and light is a form of energy and reasoning with this
knowledge. Declarative knowledge includes scientific definitions and facts, mostly in the
5
On the Development of Reflective Lessons
form of terms, statements, descriptions, or data.
For instance, a statement like,
“combining two or more materials together forms a mixture” is the scientific definition of
“mixture.” A scientific fact would be, for example, “the density of water is 1 gram per
milliliter at 4 degrees centigrade and at one atmosphere of pressure.”
Procedural knowledge and reasoning is “knowing how.” For example, knowing
how to design and reason about a study that manipulates one relevant variable and
controls others, or how to measure the density of an object or how to graph the relation
between the angle of an incline plane and the force needed to move an object up it.
Procedural knowledge includes if-then production rules or a sequence of steps, in the
form of actions or steps that can be carried out to achieve a certain goal leading to task
completion. In a broad sense, students’ statements, such as, “If I have a mixture like
gravel and water, I will use a screen to separate them,” or “After I pour the mixture on the
screen, the water goes through the screen and the gravel is left on it,” reflect their
procedural knowledge of separating mixtures.
Schematic knowledge and reasoning is “knowing why.” For example, knowing
why Delaware has a change of seasons or knowing why we see different phases of the
moon, and reasoning with this knowledge. To know why is to have a scientifically
justifiable “theory” or “model” or “conception” that explains the physical world.1
Schematic knowledge includes principles, schemes, and mental models.
Schematic
knowledge can be used to interpret problems, to troubleshoot systems, to explain what
happened, and to predict the effect that changes in some concepts will have on other
concepts (De Kleer & Brown, 1983; Gentner & Stevens, 1983).
1
A theory or model often called a “mental model” (Gentner & Stevens, 1983).
6
For instance, the
On the Development of Reflective Lessons
schematic knowledge students may develop when they learn about mixtures and solutions
includes: (a) The method used to separate mixtures (screening or filtering) is determined
by the size of the mixture’s particles; (b) evaporation is needed to separate a dissolved
material like salt from a liquid; or more generally, (c) evaporation can be used to separate
a material from a liquid because the liquid is changed into the gas state.
And strategic knowledge and reasoning is “knowing when, where and how” to
use and reason with certain types of knowledge in a new situations. Strategic knowledge
includes domain-specific conditional knowledge and strategies such as planning and
problem-solving as well as monitoring progress toward a goal. People use strategic
knowledge to recognize the situations where some procedures can be carried out, to
examine the features of tasks in order to decide what schematic knowledge can be
applied, to set task goals, or to control and monitor cognitive processing.
Our research has linked certain types of assessment to this science achievement
framework. Briefly put, to measure the structure of declarative knowledge, multiplechoice, short-answer and concept maps provide valid evidence (Ruiz-Primo & Shavelson,
1996a; Shavelson & Ruiz-Primo, 1999). To measure procedural knowledge, performance
assessments are appropriate (e.g., Ruiz-Primo & Shavelson, 1996b; Shavelson, Baxter, &
Pine, 1992). To measure schematic knowledge, multiple-choice and short-answer items,
and performance assessments are appropriate. Strategic knowledge is difficult to measure
directly but is essential, especially with novel assessment tasks. The CRDG participants
confirmed that the knowledge framework fit well with FAST, except for schematic
knowledge, which was not addressed in the unit that was to be the focus of the study (see
Brandon, et al., this issue).
7
On the Development of Reflective Lessons
Assessment Development Team
The Assessment Development Team guided the conceptualization, development
and evaluation of the embedded and end-of-unit assessments. The Team consisted of
Stanford assessment specialists and researchers and CRDG curriculum developers, FAST
trainers and teachers, and researchers. (The collaboration of the Assessment
Development Team towards the implementation of this study is more fully covered in
Shavelson et al. and Brandon et al., this issue.) Once formed, the Team went through
four major tasks to establish rapport, develop a common language, identify assessment
targets, and begin assessment development. First, the team reviewed the project
background, formative and summative assessments, the science achievement framework
for assessment development and corresponding assessment methods, the project goals
and each members’ responsibilities.
Second, the team reviewed the curriculum for the study, FAST Physical Science
(PS) investigations 1 to 14. To this end, the curriculum developers provided abbreviated
hands-on demonstrations of PS 1-14 while other Team members participated as students.
The team discussed what FAST teachers typically used as assessments and student
corresponding responses. In addition, a SEAL member presented storyboards showing
the types of knowledge that were addressed in each of the PS 1-14 investigations. The
group’s discussions about the lessons helped the team members understand the science
achievement framework and helped clarify the purpose of the investigations.
Thirdly, the Assessment Development Team developed a goal statement for the
entire unit: Students individually and in groups, will be able to explain why things sink
8
On the Development of Reflective Lessons
and float using relationships between mass, volume and density; develop, carry out,
report and defend scientific investigations of buoyancy and develop, carry out, report.
The ADT then discussed possible assessments, the knowledge types, target
difficulty levels, and the match of the proposed assessments with the curriculum. Many
assessment items discussed here ended up in the study’s final assessment suites. We
refer to the collection of assessment items and prompts given at a particular point in the
curriculum implementation as an assessment suite (e.g., the multiple-choice, graphing
and lab performance assessment suite at FAST investigation 4).
What and where to embedded assessment? SEAL researchers developed a
comprehensive assessment blue print for the PS 1-14 investigations using the science
achievement framework described above. Using the comprehensive blueprint, the
Assessment Development Team identified the most important end-of-unit concepts
taught in the 14 investigations to be used in a posttest assessment suite, and identified the
points (natural joints) in the instructional sequence in which the formative assessments
were to be embedded. The team came up with three criteria to identify the natural joints:
(1) a subgoal of the end-of-unit goal is achieved, that is, there is a body of knowledge and
skills sufficiently comprehensive to be assessed; (2) teachers need to know about student
understanding before they proceed with further instruction; and (3) feedback to students
is critical to help them improve their understanding and skills of the material already
taught (Shavelson, SEAL & CRDG, 2005, p. 6). Four embedded-assessment natural
joints were identified in the 14 investigation sequence (Figure 1). In this curriculum the
natural joints were clearly identifiable because students move from learning about
sinking and floating phenomena using mass and then move on to another set of
9
On the Development of Reflective Lessons
investigations that are focused on the next concept, volume. For example, a natural joint
in the sequence of physical science investigations occurred between investigations 4 and
5 where students move away from examining why things sink and float using mass
holding volume constant and begin using volume holding mass constant (Table 1).
Deciding what to assess in the embedded and in the end-of-the-unit assessments
was not as straight forward as finding the natural joints. In curriculum that typically takes
eight to ten weeks to implement, there are many important concepts and procedures that
students learn. Since these important topics are clearly identifiable to the curriculum
developers all of these topics became equally important to them as assessment targets.
However, it would impossible for the embedded assessments or the end of unit
assessments to cover all of the topics and so decisions had to be made as to how to select
the most important ones. For example, in the identification of the nine concepts to be
used in the concept map, a SEAL assessment developer collected and then presented a
large list of terms to the curriculum developers and asked them to identify the most
important terms that could be used to in a concept map assessment. The curriculum
developers selected all the terms because all the terms were important. In the case of the
terms to be used in the concept map, the Assessment Development Team narrowed down
the concept list by focusing on the overall goal of the unit and then asked, “which terms
are the most important for explaining why things sink and float?” This list narrowing
still generated a large number of concepts that was then whittled down in pilot testing
using empirical information.
Additionally, a FAST teacher and trainer on the team expressed concern about the
length of time the assessments would add to the unit. One team member felt that student
10
On the Development of Reflective Lessons
progress might be slowed by the embedded assessments. In response to this concern, a
SEAL team member noted that the assessments should help teachers decrease their
teaching time, because they will know the students’ deficiencies and as such the teacher
would more efficiently deliver information to the students. The tensions between time
spent on instruction vs. assessment, and depth vs. coverage of the assessments remained a
concern throughout the project.
Once the important concepts had been selected and the joints identified, the
assessment specialists of the Assessment Development Team began an iterative process
of designing and refining assessments based on the science achievement framework,
piloting of assessments and validation with the rest of the Team. The first complete
iteration of the assessment tasks identified four locations in the curriculum (excluding the
pre-post tests) where formative assessments were to be placed (Figure 1). We reduced
the list again because of the amount of time it would take to complete the assessments.
The final selection of the assessments attended to the main goal of the unit.
----------------------Insert Figure 1
----------------------Assessment refinement. The Assessment Development Team reviewed the first
complete version of the assessments and began a process of refining them to match the
purpose of the project and intent of the curriculum. We exclude a detailed discussion
here of the pre and post-tests because we are focused on the embedded assessments (see
Yin et al., this volume). The Team reviewed the assessments by discussing and/or
11
On the Development of Reflective Lessons
carrying out the assessments. As assessment development and refinement continued, the
assessments went through multiple iterations.
Once all the embedded assessments were reviewed and discussed, the Team
decided to make several important modifications to the project. First the team decided
that the project should only focus on the 12 physical science lessons in FAST. Second
the team had to make hard decisions about what not to include in the embedded
assessments. For example, the first version of the assessments contained performance
tasks for students. These tasks included using a balance, using metric rulers, measuring
volume, etc. A series of “passport” assessments was created to assess these
performances. The passport was a student record book where information is stored about
whether or not a student passed a performance goal (e.g., Can Mary mass the rock? If so
then give her a stamp in her Passport book). The Assessment Development Team
dropped these passports for several reasons. The information that was collected although
important for investigations was indirectly related to the unit’s goal: explaining “Why
things sink and float.” Moreover, the information about student performance could be
verified in the conduct of hands-on investigations. Finally, the amount of teacher time
and effort to carry out these assessments outweighed the value of the information
collected.
This focus on explaining “Why things sink and float” was significant because it
foreshadowed larger changes to come. The Assessment Development Team also dropped
the concept map from the summative tests and moved the concept maps after
investigation 6 and investigation 11. The purpose for this was to reduce the amount of
time in the pre- and posttests, to reduce the amount of time at each major embedded
12
On the Development of Reflective Lessons
assessment joint, and to ameliorate an expected pre-post testing effect with the concept
maps.
From Embedded Assessments to Reflective Lessons
Pilot Study. The Assessment Development Team carried out a pilot study with the
embedded assessments. Three teachers were trained in the use of the embedded
assessments and asked to use these with their classes during the project’s second year.
The pilot study focused on the teachers’ implementation of the embedded assessments,
student performance on these assessments, and professional development. We trained the
pilot study teachers by reviewing the curriculum, carrying out the embedded assessments
with the teachers as students, and then having the teachers use the embedded assessment
with students who were involved in the FAST investigations in summer school. The pilot
study’s findings suggested that the embedded assessments should be 1) short in duration
and tightly focused on the key outcomes of the unit—explaining “why things sink and
float” based on relative density; 2) allow for immediate feedback to teacher and students;
3) provide opportunities for students to test their “why-things-sink-and-float”
explanations with evidence from the assessment event and to take a stand on what they
believe; and 4) set the stage for the next set of investigations
The study’s findings also suggested that teachers treated the embedded
assessments just like any other test they might give. For example, the teachers would
review the material covered in the unit before the embedded assessments even though the
purpose of the formative assessments was to do just that. The teachers treated the
embedded assessments as external to the unit of instruction and did not use the
assessments to inform their teaching. The teachers who participated in this pilot study
13
On the Development of Reflective Lessons
had already been trained in FAST and knew that the assessments were added to the
curriculum after the fact. This may have lead to the assessment externalization.
However, in later studies we found the assessment externalization in non-FAST teachers.
Furthermore, pilot study teachers often would provide feedback to students weeks after
the assessment thus missing the teachable moments provided for by the embedded
assessments—none of which was intended by the project staff. It became clear that how
these assessments were to be used in the classroom was very important and that teachers
preconceived notions about assessments influence their implementation.
These pilot findings led to significant changes in the embedded assessments.
Overall, the teachers believed these embedded assessments to be summative assessments
and would revert from formative assessment pedagogy to a summative assessment
“teaching script” (Shavelson, 1986). A summative assessment teaching script can be
conceptualized as a formalized teaching pattern consisting of a set of expectations of
what events are necessary and about the temporal order of such events. A summative
assessment teaching script might include studying for the tests, taking practice tests, or
reviewing lecture notes with the students prior to actually giving the test.
In order to avoid the usual summative assessment teaching scripts, we changed
the name from embedded assessments to Reflective Lessons. The Reflective Lessons also
evolved from assessment activities to learning activities intended to provide instructional
information to both the student and the teacher by: 1) building on what students already
know; 2) attending to student conceptions and misconceptions; 3) making student
conceptions and misconceptions public and observable; 4) priming students for future
learning, and 5) reflecting on material covered.
14
On the Development of Reflective Lessons
Other pilot study evidence collected suggested that teachers needed increased
structure in order to use the Reflective Lessons and, as such, detailed information about
how to use the “teachable moments” should be provided with the Reflective Lessons.
For example, teachers were able to elicit student conceptions about why things sink and
float, but they did not necessarily use these conceptions to further student learning. This
shift from assessment activities to learning activities represents a fundamental change for
teachers in the way to look at the formative assessment practices.
Finally, pilot-study evidence suggested the need to reduce the number of
Reflective Lessons. If left untouched, the Reflective Lessons would have taken a teacher
about 15 lesson periods to complete (see Furtak et al., this issue). The Assessment
Development Team removed some assessments because they believed that regular
classroom practices allowed for formative assessment of students declarative and
procedural knowledge and because they believed that too much time was taken away
from the regular science curriculum. The tension between time spent on assessments and
time spent on instruction as if the assessment were not instructional continued even
though the Team began to move towards believing that these assessment were
instructional. Consequently, the Team set the goals of adding no more than two lesson
periods at the natural joints and one period each for the concept maps. Further, the Team
set a goal of simplifying the structure of the assessments by identifying two reflective
lesson scripts. These Reflective Lessons and scripts were used in the final Romance
experiment.
15
On the Development of Reflective Lessons
Reflective Lessons
The Reflective Lessons were composed of a carefully designed sequence of
investigations (prompts) that enabled teachers to step back at key points during an
instruction unit (natural conceptual joints) to check student understanding, and to reflect
on the next steps that they must take to move forward (Figure 2). Furthermore, these
assessments were carefully designed to match the content and structure of the existing
FAST investigations. The SEAL team developed drafts of the assessments and, using
talk-aloud methods, refined them in tryout-refine-tryout cycles conducted with students
from a local school near Stanford and the University of Hawaii CRDG Laboratory
School.
Reflective Lesson Goals
In order to promote student understanding of why things sink and float via
relative density concepts, the Reflective Lessons were designed to elicit and make public
student sinking-and-floating conceptions, encourage communication and argumentation
based on evidence from the investigations or assessment tasks, push on students’ why
things sink and float conceptions, help students track their sinking and floating
conceptions and help students reflect about these concepts (e.g., Duschl, 2003).
Elicit and make students’ conceptions public. The Reflective Lessons were
intended to make students’ thinking about different physical phenomena explicit (e.g.,
Asking students, “Why things sink and float?”). In order to achieve this purpose,
Reflective Lessons included a set of activities/investigations that bring forth students’
conceptions of density and why things sink and float (e.g., having students predict
whether a small-sized high density plastic block will sink or float and explain why).
16
On the Development of Reflective Lessons
Moreover, they included different suggested teaching strategies, such as student group
work, small and/or large group discussions, sharing activities, and questions and prompts
that help teachers reveal students’ conceptions and thinking and make these conceptions
public.
Encourage communication and argumentation of ideas using evidence. We
designed the Reflective Lessons to provide opportunities for students to discuss and
debate what they know and understand. By having students take a stand and make their
why-things-sink-and-float conceptions public, students could be provided with the
opportunity to see competing student conceptions, and hear and evaluate the supporting
evidence provided for the different competing views. That is, Reflective Lessons were
concerned with what data counts as evidence for students (Uncle Joe’s tales of the sea vs.
results of an investigation), how this evidence is used to support their predictions,
decisions and explanations, and how these explanations could be generalized to other
similar situations, that is, the universality of the scientific principles. For example, can
the notion that more bbs means more sinking be applied in all sinking and floating
events? The use of evidence for explanations in the Reflective Lessons was an extension
of the pedagogical practices already found in FAST.
Furthermore, the Reflective Lessons provided teachers with scripts to help
students support their why-things-sink-and-float explanations based on scientifically
sound evidence and help test the universality of student claims. That is, the Reflective
Lessons provided for examples and models for setting up the lessons, collecting student
conceptions, establishing discussion events, and providing questions to push students
evidence-based why things sink and float explanations: “How do you know that?” “What
17
On the Development of Reflective Lessons
evidence do you have to support your explanations?” “In what cases does your
explanation apply and what cases does it not apply?” Such discussion allows students to
think about how evidence should be used to generate and justify explanations as well as
think about the universality of their explanations.
Push on students’ why-things-sink-and-float conceptions. The Reflective Lessons
were also designed to push on students’ ideas about why things sink and float. In the
Reflective Lessons, students are presented with problems that seem to be buoyancy
anomalies, but rather the problems provide instances where everyday knowledge about
sinking and floating events cannot be easily applied. For example, the Reflective
Lessons asked students to predict whether a large-sized low-density plastic block
(polypropylene) or a small-sized lightweight high-density plastic block (PVC) would sink
or float. Students focused on the size and weight of the two blocks (decisions based on
everyday life) rather than on the density of the plastic (decisions based on scientific
evidence).
Help students track why-things-sink-and-float conceptions. The Reflective
Lessons were also designed in part to track student why-things-sink-and-float
conceptions. That is, in the Reflective Lesson suites there were assessment items that
were used each time and that allow for teacher and students to track student
understanding. With the goal of tracking students’ conceptual development, each
Reflective Lesson included the Why Things Sink and Float prompt and the same concept
terms were used in both Reflective-Lesson concept map administrations. By comparing
earlier with later assessments, student progress could be observed.
18
On the Development of Reflective Lessons
Reflect with students about their conceptions. The Reflective Lessons were
intended to help teachers pinpoint students’ conceptual development at key instructional
points, and help guide the development of students’ understanding by identifying where
students were going wrong. Through making student conceptions public and then
through discussions and debate based on evidence, teachers and students find the
problems they are having as they move towards the unit goals. Furthermore, the
Reflective Lessons provided teachers with strategies for progress that can help students
achieve these goals.
Connect to the curriculum. The Reflective Lesson goals were derived directly
from the FAST curriculum. The curriculum developers stated the importance of these
types of learning activities (i.e., encourage argumentation about evidence, push on
student buoyancy conceptions, reflect on student’s buoyancy conceptions) in their
instructional materials about group work, class discussion, looking for universality of
student claims. What the reflective lessons did, is to make the whole reflective
discussion explicit in terms of what to do with students, when to do it with students and
how to do it with students.
Types of Reflective Lessons
In order to accomplish these goals, Reflective Lessons provided specific prompts
that have proved useful for eliciting students’ conceptions, encouraging communication
and argumentation based on evidence, and helping teachers and students reflect about
their learning and instruction. These prompts vary according to where the Reflective
Lessons are embedded within the unit. In order to simplify the Reflective Lessons, two
types were identified.
19
On the Development of Reflective Lessons
Type I Reflective Lesson
The Type I Reflective Lessons were designed to expose students’ conceptions of
why things sink and float, to encourage students to bring evidence to support their
conceptions, and to raise questions about the universality of their conceptions when
applied to new situations. The prompts and questions focus students on the use of
evidence to support their conceptions of why things sink and float.
These lessons were embedded at three joints in the unit, after investigations 4, 7
and 10 (Figure 2). Each Type I Reflective Lesson has four prompts for students to: (1)
interpret and evaluate a graph, (2) predict-observe-explain an event related to sinking and
floating, (3) answer a short question, and (4) predict-observe an event related to sinking
and floating.
Graph Prompt. This prompt asks students to use knowledge of investigations,
resulting data and explanations they have collected in different FAST investigations as
evidence to support their responses and conclusions (Figure 3). Students are familiar
with the data representation used because the scatter plots are like those created by the
students in their FAST investigations and reveal the important relationship between two
variables that were just studied. This prompt requires students to interpret, evaluate and
complete graphs that focus on the variables involved in sinking and floating specific to
the FAST investigations that they have just completed. The Assessment Development
Team chose the graphing prompts because of the importance of interpreting graphs in the
curriculum in helping students use data to draw conclusions and support explanations.
Graphs like these are used throughout the 12 physical science investigations in the unit.
---------------------------
20
On the Development of Reflective Lessons
Insert Figure 3
--------------------------Predict-Observe-Explain Prompt. This prompt get directly at schematic
knowledge. It asks students to predict the outcome of a sinking-and-floating event and to
justify their prediction, then observe the event, and then reconcile their predictions and
observations (White & Gunstone, 1992). Students use what they have learned in the unit
to help them make predictions and to explain and justify their ideas. In order to make
students’ thinking explicit, they must write their predictions, observations, explanations
and reconciliations. Students are expected to use evidence in their explanations and
reconciliations. Careful administration is important for this prompt type. For example an
important administration consideration is when and how to collect student predictions
and evidence based explanations without affecting student reconciliations—to make ideas
public and anonymous. The activities that were used in this assessment came from the
literature on student’s understanding about density, developmental psychology literature
of dimensionality (relating two variables together) in student learning, and from the
repertoire of FAST and non-Fast science teachers associated with the project. These
activities focus on the variables that the students have just completed in their FAST
investigations.
------------------Insert Figure 4
------------------Short-Answer Prompt. This prompt was a single question that asks students to
explain why things sink and float with supporting examples and evidence (Figure 5). The
21
On the Development of Reflective Lessons
same prompt is used as is same in all three Type I Reflective Lesson after investigations
at 4, 7 and 10. This prompt is a direct result of the ADT work in clarifying the goal of
the instructional unit. Furthermore, this prompt would eventually serve as a vehicle to
track student understanding across FAST 1 implementations thus providing a link
between the embedded formal formative assessments and the summative assessments
(i.e., the pre and post tests).
----------------------Insert Figure 5
----------------------Predict-Observe Prompt. A slight variation on the Predict-Observe-Explain
prompt, this prompt asks students to predict and observe an event (Figure 6). These
prompts are based on the FAST 1 challenge questions that are provided at the end of each
natural joint investigation. These FAST challenge questions were restructured using a
modified POE model and student worksheets were developed for these POs. Students are
not asked to reconcile predictions and observations. POs act as a launching point for the
next instructional activity of the unit in which the explanation will emerge. POs can be
thought of as a springboard that motivates students to start thinking about the next
investigation might be about.
-----------------Insert Figure 6
------------------Type I Reflective Lesson Implementation. The Team allowed for three class
sessions for Reflective Lesson Type I implementation, although the Team expected
22
On the Development of Reflective Lessons
implementation might take only two or two-and-a-half sessions (see Furtak et al., this
issue). Each session was planned to take about 45 minutes or so to complete. The
implementation is as follows (Figure 7):
-----------------Insert Figure 7
------------------The important components of the reflective lessons are the discussions that occur
after the students have completed a few of the Reflective Lesson prompts. While the
actual Reflective Lessons are important, they in themselves represent only part of what
needs to happen to promote the goals of the Reflective Lessons. The discussions are key
because in the discussions, the teacher makes student conceptions and reasoning public
and where the relevance and importance of supporting one’s explanations with evidence
and pushing for universality are brought to the surface. The teacher’s manual for the
Reflective Lessons suggests that the discussion during the first leg focuses on the
information from the graph and the Predict-Observe-Explain prompt as well as what has
happened in class. In the subsequent short answer question Why do things sink and
float?, the discussion focuses on students’ conceptions and the universality of the ideas
and evidence. Students are asked to extend what they know beyond the context of the
Graph or Predict- Observe-Explain prompt, although both might be used as evidence. If
students rely primarily on the graph or Predict- Observe-Explain prompt as evidence, the
teacher is expected to push them beyond these toward universality and encourage
students to use information learned in class.
23
On the Development of Reflective Lessons
The last Reflective Lesson prompt, the Predict- Observe prompt, is performed in
two parts. First students make a prediction based on recent class investigations,
Reflective Lessons and discussions, and then they observe a demonstration that may
challenge their ideas. The two parts may be carried out on the same day, or divided as
shown in Figure 6 above so that part one is carried out at the end of class session two,
while part two is carried out at the beginning of class session 3. Many teachers carried
out the Predict-Observe prompt on one day and left the explanation and recap for the next
day as an introduction to the next investigation.
Type II Reflective Lesson
This type of reflective lesson focuses on checking students’ progress in their
conceptual understanding of the key concepts across the twelve investigations. Only one
kind of prompt is used in this type of reflective lesson, a Concept Map.
Concept maps provide evidence on how students see the relationships between
ideas/concepts. The concept maps were based on the terms that make up the content of
the entire series of twelve investigations. Concept maps make evident students’
understanding and the evolving sophistication of their thinking as their investigations
progress.
The concept map Reflective Lessons were inserted after Investigations 6 and 11.
These Reflective Lessons are implemented in one session (Figure 8).
-----------------Insert Figure 8
-------------------
24
On the Development of Reflective Lessons
The first box in the implementation model reflects that students need to be trained
to draw concept maps. This training takes about 30 minutes, but it needs to be done
carefully and teachers need to make sure that students know how to construct concept
maps in accordance with as set of rules. The second time students construct the maps
they need only to be reminded of the rules and do not need the entire training.
Based on the pilot study observations, the most important aspect of student
interaction occurs when the students construct a group concept map. While a class
concept map may be desirable, we found in our pilot study that only a handful of students
participated in constructing a whole class concept map and consequently, the whole class
concept map is not desirable. The sharing of main ideas benefits from looking for those
terms that students have not been able to place into the concept maps, looking at the
terms most often used in the group maps, and having students describe how their group
concept maps were constructed. For example, students are not expected to know how to
relate one of the concept map terms (density) with the other concept terms in their whythings-sink-and-float concept map until investigation 9.
FAST Buoyancy Learning Trajectory
While the curriculum was reviewed for natural joints, assessment blueprints were
drawn and the Reflective Lessons refined, a learning trajectory for student understanding
of why things sink and float using a relative density explanation was developed to guide
teachers. That is, the 12 FAST investigations build a student’s understand of why things
sink and float through a series of models. This model is evident in the curriculum and
was made explicit through the development of the Reflective Lessons and the study’s
pre- and posttests.
25
On the Development of Reflective Lessons
-----------------Insert Figure 9
------------------That is, the students in the first three investigations work with alternative
conceptions of sinking and floating events (size, number of bbs, greater or lesser amounts
of something) and then in investigation 4 students focus in on mass as an important
variable for sinking-and-floating events. In Investigations 5 and 6 students learn that
volume, especially displaced volume, plays an important role in explaining sinking and
floating events. In Investigations 4 through 6, students hold mass and volume variables
independent of each other. In Investigation 7-9 students now combine both volume and
mass together to explain sinking-and-floating events. In Investigation 10 (and the in part
in 9), students learn about the density of an object, while in Investigation 11 students
learn that liquids have density too. Finally in Investigation 12, students learn that sinking
and floating events can be explained by comparing the density of an object and the
density of medium in which it is placed—relative density.
The learning trajectory is useful when a teacher is trying to understand where a
student is in her understanding of why things sink and float. Many student responses can
readily be placed in the model and a teacher armed with this knowledge, might fashion a
discussion question that gets at what is important for a student to know to move forward.
For example, if during the Reflective Lesson after investigation 4 a student speaks about
how more sand makes the straw sink more a teacher might ask this student “What do you
think the bbs and sand have in common that affects why things sink and float?” Knowing
where a student’s response is in the FAST learning trajectory may help a teacher think
26
On the Development of Reflective Lessons
about what a student needs to know in order to increase their understanding.
Furthermore, providing teachers with this learning trajectory allowed teachers to identify
where students might be in their conceptions about sinking and floating thus simplifying
teachers’ categorization of student conceptions into five levels.
Conclusions
In order to create efficacious embedded formal formative assessments or
Reflective Lessons in new or existing curriculum, several considerations may be drawn
from this project. First, collaboration of the assessment specialists along with curriculum
developers is vital. In this assessment development, the line between assessment
specialists and curriculum developers was very thin and often blurred because these
Reflective Lessons appeared seamless to students and teachers in terms of instructional
approaches and in content. This helps ameliorate the summative assessment teaching
script and student testing scripts as well as helps teachers and students make use of the
information that they gather from these reflective lessons. If the assessments do not look
like the other lessons it is hard to make the information connect to what the teachers and
students are already and will be doing.
Second, professional development should be provided to teachers to
reconceptualize the value of assessment, especially formative assessment. Assessment
specialists or curriculum developers cannot expect teachers to effectively use reflective
lessons without training in their use of the strength of teachers’ summative assessment
scripts which may defeat the purpose of the reflective lessons.
Third, the reflective lessons must be linked to the overall goal of the curriculum
and not only to the material that the students have just covered. If the reflective lessons
27
On the Development of Reflective Lessons
are only linked to the material just covered, instructional time may be misused, focusing
on concepts or procedures that might not be vital to the overall goal of the instructional
unit (i.e., focusing on students’ abilities to mass an object in a why-things-sink-and-float
unit). This may raise questions about the need for some lessons in established curricula.
Fourth, in the development of these reflective lessons, a useful tool was the
development of the why things sink and float learning trajectory. This trajectory albeit
useful in the reflective lessons proved helpful beyond those lessons because teachers
were expected to use the trajectory as a vehicle to track student understanding throughout
unit implementation.
Fifth, we lack knowledge of how teachers use reflective-lesson information (but
see Furtak et al., this issue). When we consider formative assessments, assessment
specialists and curriculum developers must consider all five important assessment
pedagogies: knowing the content, developing the assessments, implementing the
assessments, collecting information from the assessments, and most importantly, using
the information gained (see Ayala, 2005).
Finally, this project raises questions about the quantity and frequency of formative
assessments in a curriculum. Having some embedded formal formative assessments or
reflective lessons in a curriculum is useful because it reminds teachers to reflect back on
what has been learned and hopefully guide future lessons toward unit goals. The more
efficacious use of these assessments may reside in the application of formative
assessment principles to all instructional activities (including reflective lessons). These
formative assessments should be coherently focused on the overall goal of the unit rather
than each minuscule instructional or assessment target.
28
On the Development of Reflective Lessons
References
Atkin, J. M. & Coffey, J (Eds.) (2003). Everyday Assessment in the science classroom.
Arlington, VA: NSTA Press.
Ayala, C, (2005, April). Assessment Pedagogies, Paper presented at the annual meeting
of the American Educational Research Association, Montreal Canada.
de Jong, T., & Ferguson-Hesser, M. G. M. (1996). Types and qualities of knowledge.
Educational Psychologist, 31, 105-113.
Bell, B., & Cowie, B. (2001). The characteristics of formative assessment in science
education. Science Education, 85, 536–553.
Black, P.J., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in
education, 5(1), 7-73.
Black, P., & Wiliam, D. (2004). Classroom assessment is not (necessarily) formative
assessment (and vice-versa). In M. Wilson (Ed.), Towards a coherence between
classroom assessment and accountability. Chicago: University of Chicago Press.
deKleer, J. & Brown, J.S. (1981). Mental models of physical mechanisms and their
acquisition. In J.R. Anderson (ed.), Cognitive skills and their acquistion.
Hillsdale, NJ: Erlbaum.
Duschl, R. A. (2003). Assessment of inquiry. In J. M. Atkin & J. Coffey (Eds.), Everyday
assessment in the science classroom (pp. 41-59). Arlington, VA: NSTA Press.
Gentner, D. & Stevens, A. (1983). Mental models. Hillsdale, NJ: Erlbaum.
29
On the Development of Reflective Lessons
King, A. R. and J.A. Brownell, (1966). The curriculum and the disciplines of knowledge;
a theory of curriculum practice. Wiley: New York.
Li, M., & Shavelson, R.J. (2001). Examining the links between science achievement and
assessment. A Paper presented at the American Educational Research Association
Annual Meeting, Seattle, WA.
Li, M., Ruiz-Primo, M.A., & Shavelson, R.J. (2006). Towards a science achievement
framework: The case of TIMSS 1999. In S. Howie & T. Plomp (Eds.), Contexts of
learning mathematics and science: Lessons learned from TIMSS.
London:
Routledge.
National Research Council. (2001). Classroom assessment and the national science
education standards. Washington, D.C.: National Academy Press.
Pauls, J., Young, D. B., & Lapitkova, V. (1999). Laboratory for learning. The science
teacher, 66, 27–29.
Pottenger, F. and Young, D. (1992). The Local Environment: FAST 1 Foundational
Approaches in Science Teaching, University of Hawaii Manoa: Curriculum
Research and Development Group.
Ruiz-Primo, M. A., & Furtak, E. M. (2004). Informal assessment of students'
understanding of scientific inquiry. Paper presented at the American Educational
Research Association Annual Conference, San Diego, CA.
30
On the Development of Reflective Lessons
Ruiz-Primo, M.A., & Shavelson, R.J. (1996a). Problems and issues in the use of
concept maps in science assessment. Journal of Research in Science Teaching,
33(6), 569-600.
Ruiz-Primo, A., & Shavelson, R. (1996b). Rhetoric and Reality in Science Performance
Assessments: An Update. Journal of research in science teaching, 33(10), 10451063.
Shavelson, R, Stanford Educational Assessment Laboratory (SEAL) and Curriculum
Research & Development Group (CRDG). (2005). Embedding assessments in the
FAST curriculum: The romance between curriculum and assessment. Final
Report.
Shavelson, R. J., & Ruiz-Primo, M. A. (1999). On the assessment of science
achievement. (English version). Unterrichts wissenschaft, 27(2), 102-127.
Shavelson, R. J., Baxter, G., & Pine, J. (1991). Performance assessment in science.
Applied measurement In education, 4(4), 347-362.
Shavelson, R.J. (1986). Toma de decision interactiva: Algunas reflexiones sobre los
procesos cognoscitivos de los profesores. (Translation: Interactive decision
making: Some thoughts on teacher cognition.) In L.M.V. Angulo (Ed.),
Pensamientos de los profesores y toma de decisiones Universidad de Sevilla,
Servicio de Publicaciones, Seville, Spain.
Shepherd, L.A. (2000). The role of classroom assessment in teaching and learning.
Santa Barbara, CA: Center for Research on Education, Diversity & Excellence,
University of California, Santa Barbara.
31
On the Development of Reflective Lessons
Tamir, P., & Yamamoto, K. The effects of junior high ‘FAST’ program on student
achievement and preferences in high school biology. Studies in Educational
Evaluation, 1977,3(1), 7–17
White, R. and R. Gunstone. 1992. Probing understanding. Falmer Press: London,
England.
Young, D. B. (1982). Local science program makes good: The evaluation of FAST.
Human Sciences, Technology, and Education, 1, 23–28.
Young, D. B. (1993). Science achievement and thinking skills. Pacific-Asian Education,
5, 35–49.
32
On the Development of Reflective Lessons
Figure Captions
Figure 1: Assessment suite timeline.
Figure 2. Final reflective lesson suites and placement.
Figure 3. Reflective Lesson graph after investigation 7.
Figure 4. Type I Reflective Lesson Predict-Observe-Explain prompt after investigation 7
Figure 5. Type I Reflective-lesson short-answer after Investigation 7.
Figure 6. Type I Reflective Lesson Predict-Observe after investigation 7.
Figure 7. Type I Reflective Lesson Suite implementation by session (adapted from
SEAL’s Teacher guide to the reflective lessons, 2003)
Figure 8. Implementation of the Type II Reflective Lesson in one session.
Figure 9. FAST Buoyancy Learning Trajectory.
33
On the Development of Reflective Lessons
Figure 1:
34
On the Development of Reflective Lessons
Figure 2.
35
On the Development of Reflective Lessons
Figure 3
36
On the Development of Reflective Lessons
Figure 4.
37
On the Development of Reflective Lessons
Figure 5.
Figure 6.
38
On the Development of Reflective Lessons
First Session
Second Session
Third Session
Grap h
Interp retation
Pred ict-Observe-Explain
Piece of
Evidence #1
Piece of
Evidence # 2
(POE)
Short-Answ er
Why Things Sink and Float
DISCUSSION
DISCUSSION
Pred ict
(P)
N ext
Investigation
Observe
(O)
Figure 7.
Concept Map
Training
Figure 8.
Individual
Map
Group
Map
39
Sharing Main Ideas
from Group Maps
On the Development of Reflective Lessons
Figure 9.
40
On the Development of Reflective Lessons
Table 1. First twelve FAST 1 investigations, student tasks, and learning goals (Adapted from
Ruiz-Primo & Furtak, 2004)
Lesson
1
Investigation
Liquids and
Vials
2
Sinking Straws
3
Graphing
Sinking Straws
Data
Mass and the
Finding the relationship between total mass
Sinking Straws and depth of sinking of straws
Sinking Cartons Measuring the depth of sinking of different
sizes and of equal mass
4
5
6
7
8
9
10
11
12
Volume and
Sinking Cartons
Floating and
Sinking Objects
Introduction to
Cartesian
Divers
Density and the
Cartesian Diver
Density of
Objects
Density of
Liquids
Buoyancy of
Liquids
Student Tasks
Observing vials of different liquids sinking
and floating in different liquids (a buoyancy
anomaly)
Adding BBs to a straw and measuring the
depth of sinking
Graphing number of BBs versus depth of
sinking
Finding the submerged and total volume of
cartons
Finding the mass and displaced volume of
different objects
Experimenting with Cartesian Divers
Investigating the relationship between the
diver’s mass and volume at different sinking
and floating positions
Finding the relationship between mass and
total volume and sinking and floating;
Determining the density of liquids
Experiment with different objects in liquids
of different densities
41
Learning Goals
Make scientific observations and
test predictions
Predict the number of BBs needed
to sink a straw to particular depth
Represent BB data in line graphs;
more BBs more sinking
Conclude that more mass more
sinking
Discover the relationship between
the amount of ballast, the carton
size and the depth of sinking
Calculate the displaced volume of
different cartons
Graph mass vs. displace volume of
floating and sinking objects
Discover how a Cartesian Diver
works
Find the density of Cartesian divers
of different masses and volumes
Find the density of floating and
sinking objects; density graph.
Discover that different liquids have
different densities
Understand relative density