2.E Developing an Assessment
Framework
Starting from Strand A: Modules
• Cognitive instruments developed under Strand A for
15-year-old students attending educational
institutions in grades 7 and higher that are
comparable to main PISA.
• Cognitive instruments developed under Strand A for
15 year olds that are in grades 6 or 5 and even lower
grades that are part of a national component.
• Cognitive instruments (reading components)
developed specifically under Strand C that are
targeted on the out-of-school 15 year olds who have
never been to school or who have dropped out after
completing only lower grades.
Types of Cognitive Instrument Specific to
Strand C
Need a range of assessment instruments
specifically for Strand C :
• Basic assessment based on PASEC / SACMEQ
items
• Basic literacy and numeracy assessment such
as ASER-type or EGRA early years assessment
• Pictorial / verbal assessment
A set of channels/ filters /gateways will be
needed to identify appropriate instrument
Assessment Experience
• ASER and UWEZO are basic surveys of child
literacy and numeracy, most likely to provide
basic material for those who have never been
to school or dropped out after a few grades
• PASEC and SACMEQ used in SSA are school
based; their material likely to be useful for
group that has completed primary (Strand A)
• LAMP, PIAAC and STEP are oriented towards
adults, but there are some elements which
should be considered by a contractor. In
particular, both LAMP and PIAAC have items
on Reading components
Other Assessment Issues
• None of the other surveys reviewed have paid
specific attention to the plight and treatment
of children with special needs (e.g. disability).
This pilot exercise is seen as an opportunity to
partially remedy those omissions.
• The languages and/or culture of the parent/
caregiver generation may be very different
from that of the 15 year old
ASER
Levels and Categorisation
Letters: Set of commonly used letters.
■ Words: Common familiar words with 2 letters
and 1 or 2 matras.
■ Level 1 (Std 1) text: Set of 4 simple linked
sentences, each having no more than 4-5 words*.
UWEZO
Levels and Categorisation
Recognise Letters
Read selected words
Read one of two paragraphs
SACMEQ
Levels
1: Pre‐reading
2: Emergent
Reading
3: Basic Reading
■ Level 2 (Std 2) text: Short story with 7-10
sentences. Sentence construction is
straightforward, words are common and the
context is familiar to children*.
Procedure for Categorisation
Read a story and answer 2
comprehension questions
4: Reading for
meaning
Categorisation
Children are offered a paragraph to read. If they can do
either of those easily, they are then offered a story to
read; and if they can do that easily they are rated at Story
level; if not they are rated at Paragraph level. If they
cannot read the Paragraph easily, then they are offered a
word list. If they can answer 4 out of 5 items but not read
the paragraph, they are rated at Word level. If they
cannot read 4 out of the 5 chosen words then they are
presented with a list of letters, from which they choose 5
items. If they can answer 4 out of 5 chosen items they are
rated as at Letter level if they cannot, they are rated as at
Children are categorized according to the
highest level attained. For example, if a child
could read the words but not the paragraph,
that child was ranked at word level
5: Interpretive
reading
6: Inferential
Reading
Pass Levels
7: Analytical
Reading
A child “passed” the literacy test if
8: Critical
he/she was able to read the story aloud Reading
and correctly answer the two
comprehension questions
Reading Components in LAMP
1. Vocabulary
2. Digit naming
3. Rapid digit naming
4. Lowercase letter naming
5. Uppercase letter naming
6. Rapid letter naming (lowercase)
7. Word recognition (high frequency words; lowercase)
8. Rapid word recognition (same as previous)
9. Decoding (rare words; non-words)
10. Rapid decoding (same as previous)
11. Sentence processing (oral reading; silent reading,
timed)
12. Passage fluency (oral reading; silent reading, timed)
Language Issues
• Language of instruction for in-school 15 yos
inappropriate medium for testing some of target
group in Strand C. Here, we distinguish four subdivisions of the target group:
1. First, a small group which will have had some
secondary and then dropped out.
2. Second, a minority will have completed primary
and then dropped out.
3. Third, a significant minority will have dropped
out before completing primary
4. Fourth, a small minority (in most countries) will
have never been to school
Language Issues (continued)
• Language of instruction appropriate for first
two groups, except where entire primary cycle
conducted in local language (mother tongue),
possibly for third group, but clearly not for 4th
• Generally agreed that four grades sufficient
for learning language
• Breakdowns of those currently OOS suggest
that very high percentages (70% to 90%) are
less than 5th grade
Testing Procedures/ Filters
• Age estimation: include specific event
calendar. To avoid over-identification of 15
yos, calculate expected number of ‘hits’ in
each area and explain to citizen volunteers
• Status vis-à-vis school: in addition to direct
questions to 15 yo and caregiver, need to ask
about any other learning experiences, reasons
why this 15yo no longer in school and about
education/ schooling of other children in
household
Structure of tests
• For Language EITHER directly ask 15yo which
language they are comfortable with OR use
ASER/ UWEZO approach to identify level of
literacy (letters, words, sentences)
• For mathematics, OOS 15 yo will almost
certainly understand verbally numbers
because of financial transactions, but may not
be able to read them. Use ASER/ UWEZO
approach to identify level of numeracy.
• If 15yo unable to answer simplest item,
possibility of questions based on graphic
representations and non-verbal reasoning;
both would have to be made equivalent
Questions for Discussion
Starting from Strand A:
• Is it just literacy and numeracy?
• What do we additionally learn from other
assessments?
• What can be achieved in 10-30 minutes?
• How can Strand C handle multiple languages?
• What do we call Strand C? Pre-PISA?
• How do we adapt to 15 year olds with special needs?
• How do we adapt to different cultural contexts in the
households?