2015 MCAS-Alt:
How to Submit a
“Complete” Portfolio
Karen Orlando
MCAS-Alt Teacher Consultant
Topics
Top five reasons portfolios were scored Incomplete
Other reasons for an Incomplete score
Checking for “completeness”
Frequently-asked questions
2
Massachusetts Department of Elementary and Secondary Education
Top Five Reasons for
Incomplete Portfolios
3
Massachusetts Department of Elementary and Secondary Education
Top Five Reasons For Incomplete
Portfolios
1. Some or all brief description(s) did not address the
measurable outcome. As a result, student did not address
measurable outcome on at least 8 dates.
2. Brief descriptions were not included or did not provide
enough information on at least 8 dates.
3. Titles and/or samples of text used with student were
missing in the ELAReading strand.
4. At least two pieces of primary evidence were not
submitted that related to measurable outcome listed on
the data chart.
5. No evidence of self-evaluation in the strand.
Massachusetts Department of Elementary and Secondary Education
4
Top Five Reasons For Incomplete
Portfolios
1. Some or all brief description(s) did
not address the measurable
outcome. As a result, student did
not address the measurable
outcome on at least 8 dates.
5
Massachusetts Department of Elementary and Secondary Education
Walter will identify basic forms of energy with 75% accuracy and
75% independence.
6
Massachusetts Department of Elementary and Secondary Education
Student will answer simple comprehension questions to demonstrate the understanding of
an informational text with 80% accuracy and 80% independence.
7
Massachusetts Department of Elementary and Secondary Education
Page 35
2015 Educator’s Manual for MCAS-Alt
Brief Descriptions Unmatched
to Measurable Outcome
Page 35: “…include a brief description beneath
each data point that clearly describes what the
student did and how the student addressed the
skill, taking particular care to document that the
activity matched the measurable outcome .”
Page 35: “…assess and record data only on those
skills directly based on the measurable outcome.
The data will be inconclusive, and the chances of
scoring Incomplete will increase, when unrelated or
multiple skills are included on the same data chart.
Massachusetts Department of Elementary and Secondary Education
8
Creating a Measurable Outcome from Entry
Points that Include Related Skills
Example:
Entry Point as written:
“Solve number sentences that represent one-step
multiplication and division word problems.”
Option 1:
 Create a measurable outcome that addresses both skills.
 All activities must document “solving number sentences
involving one-step multiplication AND division problems.”
Option 2:
 Modify the measurable outcome: “Solve number sentences
that represent one-step multiplication word problems.”
 All activities must document “solving number sentences
involving one-step multiplication problems.”
Massachusetts Department of Elementary and Secondary Education
9
Which descriptions are scorable?
Measurable Outcome:
Student will solve addition and subtraction
word problems with 80% accuracy and
100% independence.
Student
solved
addition
and
subtraction
word
problems
worksheet.
Student
solved
subtraction
word
problems in
math class.
Word
problems
game
with whole
class
Solved addition
and subtraction
word problems
created by
peers on smart
board
Solved
addition
problems
on
computer
with no
help
10
Massachusetts Department of Elementary and Secondary Education
Clear and Specific Descriptions
 Score accuracy and independence only on the skill related
to the measurable outcome. Clearly indicate this on the
work description label.
Example:
 Skill measured in the measurable outcome:
“Answer questions about bar graphs”
 Activity: Student creates a bar graph from the results of
a class survey, then answers questions about the graph.
 Scorable Description: “After creating a bar graph from a
class survey, student answered questions about the
graph. (Accuracy and independence were based on
questions only.)”
 Unscorable Description: “Student created a bar graph
and answered questions.”
Massachusetts Department of Elementary and Secondary Education
11
Top Five Reasons For Incomplete
Portfolios
2. Brief descriptions were not
included or did not provide enough
information on at least 8 dates.
12
Massachusetts Department of Elementary and Secondary Education
Walter will compare fractions using the terms “greater than,” “less than, or
“equal to” with 80% accuracy and 80% independence.
13
Massachusetts Department of Elementary and Secondary Education
Pages 35 and 40
2015 Educator’s Manual for MCAS-Alt
Scorable Brief Descriptions
Page 35: “include a brief description
beneath each data point that clearly
describes what the student did and how the
student addressed the skill, taking particular
care to document that the activity matched
the measurable outcome;
14
Massachusetts Department of Elementary and Secondary Education
Pages 35 and 40
2015 Educator’s Manual for MCAS-Alt
Scorable Brief Descriptions continued
Page 40: “Brief descriptions are provided
below each learning activity that match the
measurable outcome, and describe what the
student was asked to do to address the
measurable outcome, and how they did
it. The following are examples of appropriate
brief descriptions:…
15
Massachusetts Department of Elementary and Secondary Education
Acceptable Brief Descriptions
Measurable Outcome for STE- Technology/Engineering
Walter will identify simple machines with 80% accuracy and
100% independence.
Date
9/13/12
(m/d/y)
10/15/12
What
the
student
did
(skill)
After
listening to
“Simple
Machines,”
identified
simple
machines
in the book
by pointing
Identified
simple
machines
by
labeling
pictures
How did in an
they do adapted
it?
textbook
(approach
materials)
10/17/12 10/24/12 11/4/12 11/15/12 11/20/12
Homework:
Identified
simple
machines
at home,
made a list
of the ones
he found
Worksheet
identified
inclined
planes and
levers as
simple
machines
using
pictures
and text
After
watching
video,
Wheels
and
Axles,
identified
wheels
and axles
around
the
school
Using
Home
Depot
flyer,
Walter
identified
levers and
wedges
using a
bingo
dauber
After
watching
Eduhead
on the
computer,
Walter
identified
inclined
planes by
matching
them to
the
pictures in
the video
12/7/12
Class
created a
poster of
simple
machines
he
identified
the simple
machines
in of each
picture
with a label
Massachusetts Department of Elementary and Secondary Education
16
These brief descriptions DO NOT provide enough
information to know what the student was asked
to do, or how he/she did it.
Brief
Pre- Solve word Home- Digestive Answer
Label
Describe Chap.
Description test problems on work
system vocab
parts of digestive 2 test
worksheet
class
questions digestive system
assignment
system
17
Massachusetts Department of Elementary and Secondary Education
Which descriptions are scorable?
Outcome: After listening to story or article, student will answer simple
comprehension questions to demonstrate understanding of an
informational text with 80% accuracy and 80% independence.
Student
answered
questions
about a
story during
speech
class
Student
answered
questions
on an
information
al text
After
listening to
Boston
Globe
article on
“Pollution,”
student
answered
simple
comp. ques.
After
listening to
chapter 1 of
Helen Keller
by M.
Davidson,
student
answered
simple
comp. ques.
Comprehen
sion
question
quiz,
chapter 2
18
Massachusetts Department of Elementary and Secondary Education
Top Five Reasons For Incomplete
Portfolios
3. Titles and/or samples of text
used with student were missing
in the ELAReading strand
19
Massachusetts Department of Elementary and Secondary Education
Titles of text are missing!
20
Massachusetts Department of Elementary and Secondary Education
Pages 15-19 and 41
2015 Educator’s Manual for MCAS-Alt
Cite the Title or Include Actual Text
 Page 15-19: “The title or a photocopy of the text
used for assessment must be included in all brief
descriptions for ELAReading; a copy of any teachercreated text must be submitted”
 Page 41: “Evidence or brief descriptions in the
ELAReading strand must refer by name to the text
from which words, phrases, or excerpts were selected
for assessment, and must include either:
 the title of the published text, or
 the actual text (e.g., sentence, passage, narrative, etc.), or
 a photocopy or printout, if text is teacher-created, untitled,
or includes selections from a print or digital source
(e.g., websites)
Massachusetts Department of Elementary and Secondary Education
21
Options for Citing the Text
 Provide titles.
 Provide copy of teacher-created text, worksheets,
or selections from websites.
 Attach a list of text used to data chart with dates
and names of text used.
 Include topic of the text if it is from a specific
reading program, rather than just name of
program.
 Include title of text and/or topic if it is from a
specific website, rather than just the name of the
website.
 For untitled texts include a photocopy
Massachusetts Department of Elementary and Secondary Education
22
Top Five Reasons For Incomplete
Portfolios
4. At least two pieces of primary
evidence were not submitted that
relate to the measurable outcome
listed on the Strand Cover Sheet
23
Massachusetts Department of Elementary and Secondary Education
Student will compare and contrast characters in a story with 90%accuracy and
80%Independence
24
Massachusetts Department of Elementary and Secondary Education
EVIDENCE #1
25
Student
Massachusetts Department of Elementary and Secondary Education
EVIDENCE #2
Student
Student is
answering
questions
about
characters,
not
comparing
and
contrasting
them.
26
Massachusetts Department of Elementary and Secondary Education
Pages 12-19 and 33
2015 Educator’s Manual for MCAS-Alt
Two Pieces of Primary Evidence
Pages 12-19 “…two additional pieces
of primary evidence showing the
student’s performance of the same
targeted skill identified on the data
chart.”
Page 33 “A minimum of one data
chart and two pieces of additional
primary evidence (called the “core set
of evidence”)…”
Massachusetts Department of Elementary and Secondary Education
27
Top Five Reasons For Incomplete
Portfolios
5. No evidence of self-evaluation was found
in this strand.
28
Massachusetts Department of Elementary and Secondary Education
Self-Evaluation
Either…
Evidence was either missing altogether, or
Showed no evidence of student choice
Stickers
Stamp
“Student smiled”
“Student traded in for music”
29
Massachusetts Department of Elementary and Secondary Education
Self-Evaluation
Pages 39
2015 Educator’s Manual for MCAS-Alt
 “Self-evaluation activities document the
student’s choices, decisions, and involvement
before, during, and after instruction, including
evidence that the student performed…”
30
Massachusetts Department of Elementary and Secondary Education
Examples of Self-Evaluation include:










planning and goal setting
choosing an activity or next steps in an activity
selecting a problem-solving strategy
monitoring own progress or use of a strategy (e.g., by checking off
steps as each is completed)
deciding when to continue or end participation in an activity
identifying and correcting his/her own errors
graphing own performance or progress on a chart
determining own score using a rubric
selecting work for his/her own portfolio
reflecting on his or her performance; for example, teacher asks
student to respond to:
 What did I do well? What am I good at? Was this too easy?
 What did we do during this activity? What did I learn?
 How could I do better? Where do I need help?
 What should I work on next? What would I like to learn?
Massachusetts Department of Elementary and Secondary Education
31
32
Massachusetts Department of Elementary and Secondary Education
Other Reasons for Incomplete
 Literature and Informational text were combined in a
single strand. The Reading strand must be one or the other.
 Data chart did not document student performance on at least
8 different dates.
 Both Accuracy and Independence began above 80-100%.
 No data chart was submitted.
 Data were unclear or could not be interpreted by the scorer.
 Percent accuracy and/or independence were not included on
primary evidence and/or could not be determined by scorer.
 Student's name and/or date (month, day, year) was missing
on one or more pieces of primary evidence, or was incorrect
on Work Sample Description label.
Massachusetts Department of Elementary and Secondary Education
33
Level of Complexity = 1
34
Massachusetts Department of Elementary and Secondary Education
Page 48
Educator’s Manual for MCAS-Alt
What Does Level of Complexity = 1 Mean?
Portfolio strand reflects little or no basis in,
or is unmatched to, Curriculum Framework
learning standards required for assessment.
 Strand must assess standards required for a student in that
grade.
 Data chart must document student performance of an
acceptable measurable outcome in the required
strand/domain. (Teachers cannot make up or use outdated
entry points!)
 ELA Reading is comprehension of words, phrases, sentences,
passages based on a text.
 Access skill must be taught in the context of an academic
activity.
Massachusetts Department of Elementary and Secondary Education
35
Skill Not Taught in the Context of
an Academic Activity
36
Massachusetts Department of Elementary and Secondary Education
Earth and Space Science
is not assessed in High School
37
Massachusetts Department of Elementary and Secondary Education
Checking for Completeness
38
Massachusetts Department of Elementary and Secondary Education
Pages 35-36,
Educator’s Manual for MCAS-Alt
12 Key Questions to
Check for Completeness
1. Does the strand include a data chart and at least
two pieces of primary evidence
2. Is the first data point on the data chart below
80 percent accuracy and/or independence?
3. Does the data chart have at least 8 different
valid dates?
4. Is the student’s name, date, % accuracy,
and % independence listed on the data chart
and on at least two pieces of primary evidence
(or on a Work Description label)?
Massachusetts Department of Elementary and Secondary Education
39
12 Key Questions to Check for
Completeness (continued)
5. Do brief descriptions on the data chart state what the
student was asked to do (i.e., what was the activity?) and
how the activity was conducted (instructional approach,
materials, etc.) on at least 8 different dates ?
6. Do brief descriptions on at least 8 different dates
address the skill identified in the measurable outcome?
7. Do at least two pieces of primary evidence directly
address the measurable outcome?
Massachusetts Department of Elementary and Secondary Education
40
12 Key Questions to Check for
Completeness (continued)
8. Are all skills listed in the measurable outcome (e.g.,
“addition and subtraction”) addressed in at least 8
activities on the data chart and two pieces of evidence
(all skills must be addressed in each activity).
9. If photographs or video are included as primary
evidence, do they show a final product and are they
clearly labeled?
10. If a teacher-scribed work sample is included as
primary evidence, does it provide sufficient information
41
to determine what the student did and how the student
did it?
Massachusetts Department of Elementary and Secondary Education
12 Key Questions to Check for
Completeness
For ELA-Reading Strands Only:
11. Do brief descriptions include titles of all
texts?
12. Do all activities in the strand include either
Informational or Literary text (but not both)?
42
Massachusetts Department of Elementary and Secondary Education
Common Questions
43
Massachusetts Department of Elementary and Secondary Education
How Do I Determine
Accuracy and Independence?
First, determine what the outcome is.
What are you asking the student to do?
Then, determine the activity the student will perform
to address that outcome.
Next, divide that activity into “items” or steps.
Determine the overall accuracy (correctness) and
independence (i.e., no prompts needed) for the
entire activity:
% Independence = Number of “items” performed independently44
Number of possible responses
Massachusetts Department of Elementary and Secondary Education
Page 38
2015 Educator’s Manual for MCAS-Alt
Examples of “Items”:
Outcome: Andre will answer simple comprehension questions in relation to a text
with 80% accuracy and 100% independence.
Activity: After reading Chapter 1 in “Frog and Toad Are Friends” by A. Lobel, with
support of visuals and a reading checklist, student answered 4 comprehension
questions on a worksheet.
Question#1
Correct (accurate)
Visual prompt
(not independent)
Question#2
Correct(accurate)
No Prompt
(independent)
Question#3
Correct (accurate)
Gestural prompt
(not independent)
Question#4
Incorrect (inaccurate)
No prompt
(independent)
Averages for all
activities
3/4 or 75% accurate
2/4 or 50%
independent
Massachusetts Department of Elementary
and Secondary Education
45
46
Massachusetts Department of Elementary and Secondary Education
• How many
questions? 5
• How many correct
responses? 2
• What is the
accuracy score?
2/5 or 40%
P
I
I
• How many
independent
responses? 4
• What is the
independence
score? 4/5 or 80%
I
I
Massachusetts Department of Elementary and Secondary Education
47
Pages 37-39, 50
Educator’s Manual for MCAS-Alt
Independence:
Prompts vs. Accommodations
The following are cues and prompts since they
guide the student toward the answer:
1.
2.
3.
4.
Verbal assistance (coach or tell: ”look on page…”)
Gesture (point or indicate where to find answer)
Visual cue (provide a sample problem or model)
Full or partial physical assistance (including hand-overhand assistance)
Any Cue or Prompt = a “non-independent response”
48
Massachusetts Department of Elementary and Secondary Education
What is Generalized
Performance?
Page 39,
2015 Educator’s Manual for MCAS-Alt
2015 Educator’s Manual:
 Page 39: “Generalized performance reflects the student’s
ability to transfer knowledge and/or skills to other
learning situations through application of a range of
instructional approaches and activity formats,
including any of the following instructional elements:
 media and materials (e.g., using art materials, written text,
manipulatives, computer)
 activity formats (e.g., classroom projects, small group
discussions, paired research, experiments)
49
 presentation formats (e.g., oral, written, multimedia)
 method of response (e.g., handwritten, word-processed,
oral, creation of a visual display, video)”
Massachusetts Department of Elementary and Secondary Education
Page 37,
2015 Educator’s Manual for MCAS-Alt
What is a Teacher-Scribed Work
Sample and When Should it Be Used?
One method of documenting performance of a
student who does not produce written work
Indicates what the activity was, what response
options were offered to student, and how
he/she responded on each item (accurately,
independently?)
Documents student’s performance of a series
of tasks recorded by the teacher at one time.
Massachusetts Department of Elementary and Secondary Education
50
Teacher-Scribed Work Sample
 What was the
activity?
 What
materials
were used?
 What was the
correct
response?
 What was the
student’s
actual
response?
Student’s
Response
Correct
Response
Massachusetts Department of Elementary and Secondary Education
51
Teacher-Scribed Work Sample:
Not Enough Information to Score
• What was
the activity?
• What was
the correct
response?
• What
materials
were used?
• What was the
student’s
actual
response?
52
Massachusetts Department of Elementary and Secondary Education
Reminders
 Double-check the evidence before submitting
• Have another teacher or supervisor double-check your
evidence and see if they can answer “yes” to the 12
questions.
 Check for clerical errors such as incorrect dates,
names or work of another student, or evidence
submitted in wrong strand.
 Attend Portfolios-in-Progress review sessions
(January and March) and have training specialist
look over your portfolios, answer questions, and
share ideas
 Make sure you are using current Educator’s
Manual, Resource Guides, and forms
Massachusetts Department of Elementary and Secondary Education
53