PISA for Development
1st International Advisory
Group Meeting
Component Terms of Reference for the IC
27 – 28 May 2014
Paris, France
EDU/DCD
PISA for Development
International Advisory Group Meeting
Expected Results from Meeting
• Agreement on the main components of
the Terms of reference for the
International Contractor(s)
-
General approach
Cognitive instruments
Contextual questionnaires
Next steps for managing the tendering
process
2
PISA FOR DEVELOPMENT
General Approach
General Approach - 1
• To focus on the tasks as they relate to the
development of the cognitive instruments and of
the context questionnaires.
• The elements relating to other tasks (e.g. sample
design, translation and verification, survey
operations, analysis and reporting) are not
covered in this discussion and will be added in the
final ToR.
• The elements relating to approach to OOS are also
not covered in this discussion: these will be focus
of a separate tendering exercise
4
General Approach - 2
Focus on:
• the constraints that should be imposed on
the bidders for the International
Contract(s);
• the requirements expected of bidders; and
• questions that the bidders should respond
to as part of the tendering process
5
General Approach - 3
• principles and components of the ToR
would, once agreed by the International
Advisory Group (IAG) of the PISA for
Development project, provide the
framework for the development by OECD
of a complete ToR and associated
tendering documents that would serve as
the basis for a call to tender by the OECD
6
General Approach - 4
ToR and tendering process will:
• reflect the general PISA principles;
• reflect and support the rationale of the countries
for participating in PISA for Development;
• emphasise capacity building;
• emphasise peer-to-peer learning
• emphasise consultation and communication with
and support for participating countries
• emphasise track record of bidders in building
capacity for student assessment in developing
countries
7
PISA FOR DEVELOPMENT
Cognitive Instruments
ToR – Cognitive Instruments
Countries require an assessment that:
• Reports results on the PISA scale and
evidence supports comparability to
international PISA results;
• Allows students to demonstrate the full
range of proficiency levels; and
• Adheres to all PISA standards
– unless certain modifications are agreed for the
implementation of PISA for Development.
9
ToR – Cognitive Instruments
Fundamental constraints for bidders:
• No new cognitive items will be developed
• Review the secure pool to inform the
selection of items based upon their
cultural and contextual suitability
• The targeted test should give a robust and
targeted measure of the country's
strengths and weaknesses
10
ToR – Cognitive Instruments
Fundamental requirements of bidders:
• Assess students' abilities as they are now,
note where it is thought the students’
performance should be.
• Better targeting but also assess across the
full range of the PISA scale.
• Must ensure sufficient PISA framework
coverage and coverage of a full range of
proficiency levels.
11
4 Key elements of the ToR and bidding
documents
• Review of assessment
framework and items
–including cross-cultural validity
and test targeting;
• Test design;
• Review of the proficiency levels;
• Review of scaling models.
12
Review of assessment framework and
items - 1
Constraints for bidders:
• The selection of all assessment items is
based on the PISA assessment frameworks
for reading, mathematics and science
• Any extension of the framework will need
to continue to incorporate the original
13
Review of assessment framework and
items - 2
Requirements of bidders:
• To carry out a complete review of the
assessment frameworks
• Consider the implications of PISA 2015
move to a computer-based assessment
14
Review of assessment framework and
items - 3
Requirements of bidders:
• Need to review PISA’s technical standard
on language of instruction?
• Processes to review item suitability,
translation and verification of test
– Variations needed?
15
Review of assessment framework and
items – 4. Cross-cultural validity
• Item-by-country interactions (country
DIF).
• Need to examine the potential impact of
these interactions on the validity of
reporting.
• Bidders’ approach to this and proposed
solutions.
16
Review of assessment framework and
items – 5. Test targeting
• Cognitive instruments to be developed
from the pool of 337 secure PISA items
• Bidders’ proposals for test design should:
– deal with the challenge of designing a test that
draws upon easier items
– But still matches the framework specifications,
including with regard to item-types
17
Test design - 1
Requirements of bidders:
• To reliably forecast from a targeted test
the proportion of participating students
who would successfully complete the most
difficult items
• Achieves the desired test targeting but
– provides good coverage of the frameworks
– provides results comparable with the main
PISA assessment
• Propose an equating study
18
Target population
• International target population:
– 15-year-old students attending educational
institutions in grades 7 and higher
• Bidders to make provisions to allow a
participating country to supplement the
sample
19
Review of the proficiency levels
• Propose how to ensure these are
meaningful
• How to better describe the proficiency of
students who perform below the lowest
proficiency levels
20
Review of scaling models - 1
Should PISA for Development pursue
alternative scaling models that allow features
such as:
• varying discrimination across items;
• dependencies between items clustered in
units
• guessing; and,
• parameter variation (including difficulty)
across countries?
21
Review of scaling models - 2
• To what extent do the observed item-bycountry interactions undermine
developing a PISA-like learning metric for
developing countries?.
• How can their proposals for scaling
overcome this?
22
Draft Components of the
ToR for the IC(s):
Contextual Questionnaires
Tramonte & Willms
Paris, May 28, 2014
The 7 themes
• early learning opportunities
• language at home and school
• family and community support
• quality of instruction
• (effective) learning time
• student socioeconomic status (SES)
• school resources
• Add items on students’ early
learning experiences,
• Add items on students’
familiarity with the language
of the test
• Measure parental
involvement, social capital
and cultural capital
• Measure the role of other
community members and of
types of community
• Enhance measure of school
attendance
• Add items on participation in
formal and informal labor
market
The last two themes – student SES and
school resources – should be emphasised in
the ToR for the International Contractor(s).
In presenting their proposals, bidders will be
asked to consider the guidance provided by
the expert paper, to elaborate extensively on
the last two themes, and to justify alternative
approaches.
Implications for the Components of the
ToR for the International Contractor(s)
The 7 themes will be addressed with
theoretical, methodological, and technical
discussions in the ToR for the International
Contractor(s).
Requirement in the enhancement of the
questionnaires according to the 7 themes
The revised questionnaires must be:
a) comparable internationally and
b) consistent with the current PISA
frameworks.
Structure of the ToR-contextual
questionnaires
This component of the ToR is articulated in three parts:
• Part One refers to the first 5 themes that require minor
intervention;
• part Two and Three refer to the two critical themes, SES
and school resources, that warrant greater attention;
• finally, the contextual component of the ToR contains
indications on general expectations, specific questions,
and underlying considerations on choice of informants
and synergy with national assessments.
Key elements of the ToR: Part One
Five areas for enhancement and measurement:
early learning opportunities, language at home and school,
family and community support, quality of instruction, learning
time.
Four underlying aspects for bidders to respond to:
the best informants;
the core questionnaires;
PfD and its synergy with national assessments;
the collection of data on quality of instruction that are subject
specific;
Key elements of the ToR: Part Two
One area for enhancement and measurement:
Student SES
Four underlying aspects for bidders to elaborate on:
 offer options to address the extension of SES: adding to PISA ESCS more
items at its lower end and/or develop new “poverty-related” measures;
 capture the experiences of different countries regarding their own variables
for measuring SES;
 create a global measure of poverty that can be applied consistently across
countries for comparative purposes;
 ensure comparability of results of the project on ESCS scale with
international results, even if the scale is extended.
Key elements of the ToR: Part Two
Requirement for bidders: the measure of socioeconomic
status (SES) that is developed must be comparable across
participating countries and with the results of PISA
international assessment, even if the scales are extended.
In addition, a measure of SES for PfD should be:
 a reliable and valid measurement of SES within each country;
 a tool for accurate assessment of low levels of SES and poverty;
within each country and across countries; and
 a comparable measure of SES and its variability across countries.
Key elements of the ToR: Part Three
Area for enhancement and measurement
Measuring school resources
OECD and participating countries agreed on the need to
integrate the PISA questions with few new measures on
school resources that provided data on basic services,
didactic facilities, and didactic materials.
The bidders should enhance the PISA tools to capture:
availability, conditions, and use of basic services, didactic
facilities, and didactic materials;
conditions and quality of school infrastructure;
school infrastructural features, services, and safety.
Key elements of the ToR: Part Three
Expectations and challenges:
Bidders should propose viable options for
maintaining comparability with the main PISA
instruments where essentially the same
construct is being measured. They should
therefore explain the extent to which their
proposal will achieve this.
 Bidders should explain how to capitalise on
potential synergies in data collection with ongoing national/international assessments
and
they should identify who are the optimal
respondents about availability, conditions, and
use of basic services, didactic facilities, and
didactic materials.
Note: this issue relates to the broader discussion on the extension
of PISA core questionnaires to accommodate teacher or parent
questionnaires (see later).
General Expectations:
Bidders should provide a detailed account of:
the methods proposed to measures of SES and school
resources, as well as to conduct pilot testing for the revised
questionnaires;
the strategies to deal with the challenges associated with
pilot testing, obtaining reliable and valid data from school
administrators, and integrating the new content into the
current PISA framework.
General Expectations (continued)
Bidders should provide a detailed account of how to:
ensure international comparability once new measures are
introduced;
maintain comparability with PISA current measures;
allow nationally relevant analyses;
provide data that inform of equity and equality.
Expanding on questionnaires and
informants. Three questions for the bidders
Currently PISA assesses students and schools with two core
questionnaires (student and school); parent and teacher
questionnaires are optional.
1. Given the 7 themes that need to be captured in PfD, are
the core questionnaires enough?
2. If new questionnaires have to be introduced, what should
they be? Why?
3. Should any of the core questionnaires be dropped?
Why?
More specific questions:
Who are the best informants for the 7 themes?
How can bidders insure that the informants provide
effectively their information?
How do bidders ensure that the current PISA questions
and the new questions are relevant for the partner
countries?
How do bidders plan on collecting data on quality of
instruction that are subject-specific?
How do bidders propose to capture students’ learning
time, in and out of school?
On Informants
The bidders should discuss the rationale for choosing their
informants and the need for expanding the set of core
questionnaire to guarantee:
limited number of non-responses or missing data resulting from:
low level of literacy of the respondents;
impoverished living conditions of the respondents
gap between language of the informant and language of the
questionnaire.
representation of the variability in family structures;
comparability between and within participating countries;
triangulation of information from different questionnaires;
contained length of the questionnaires
PfD and National/International Assessments
PfD data should be a tool and a resource for each participating
country. Bidders should propose how to maximize the synergy
between PfD and national assessments.
While maintaining the core PISA content, they should explain how
to ensure:
stability and consistency of data collection within the country;
comparability of data between PfD countries and PISA countries;
validity and reliability of the indices used to conduct meaningful
analyses within each country; and
manageable national datasets that can be analyzed by PfD
national analytical teams.
On sampling and methods
Bidders are expected to design national samples that are
structured using the same stratification variables used in
PISA but also incorporate country specific variables.
In the bid, applicants are expected to propose innovative
methods of data collection and data merging that allow for
effectively linking PfD data to locally-collected national
datasets.
CALL FOR TENDER
PROCESS
Roles, Responsibilities and the Process
Call for Tender process
Set clear
evaluation
criteria
Agree
structure
of CfT
Draft
Terms of
Reference
Independent experts;
Secretariat; IAG; SDG
potential bidders
Deadline for
Submission
Technical
Review
Panel
Finalisation
Identify
potential
bidders
Call for
Tender
issued
Technical Review Process (detail)
•
•
•
•
•
•
•
The technical criteria reviewed prior to the panel meeting to
ensure common understanding among panel members.
Panel members individually score bids on the technical
criteria (without knowledge of the budget proposals) prior
to convening for the panel meeting.
The panel convene in person to share scores and to discuss
reasons for score deductions.
Cost proposals reviewed after all the bids are discussed.
Follow-up questions sent to the top-rated bidders.
The Secretariat drafts the report, incorporating responses to
questions, for the panel’s review.
The panel agrees the report and the recommendations
therein.
Call for Tender process
Agree
structure
of CfT
Deadline for
Submission
Draft
Terms of
Reference
Technical
Review
Panel
Finalisation
Technical
Review
Report
Call for
Tender
issued
IAG/SDG
review
Contract
negotiations
Draft contract
Procurement
Board
Contract signed
Best Value for Money
• OECD awards the tender based on the concept of best
value for money
– Best quality to price ratio
– Bidders offering added value can improve the quality of the
goods and services purchased by OECD
– Adds a level of complexity which makes the Tender Evaluation
Process important
– Evaluation Criteria balances
•
•
•
•
•
Contract compliance
Performance
Delivery
Ability to meet minimum specifications
Price
Timeline and procedures: 2014
Agree
structure of
CfT
Draft Terms
of
Reference
• NJune
• NoJune
Deadline
for
Submission
•
August
Finalisation
• DecJuly
Technical
Review
Panel
• September
• 30-31 May 2011
Draft contract
• 13 October 2011
Technical
Review
Report
• 7September
Procurement Board
• 2October
• 30-31 May 2011
Call for
Tender
• 1July
IAG/SDG
review
• 10 October
Contract signed
• 7November
Development
Capacity Needs
Assessment
Framework
Fernando Cartwright
Objective
Develop a framework to identify any factors or
conditions that are barriers to the successful
implementation of PISA, including the production of
meaningful information and consumption of
information by stakeholders, in participating countries.
Needs Assessment
Dimensions

Enabling environment


relationships,
Organization


Legislative,
political/bureaucratic
cooperating partners
the National Centre and other sub-national institutions
Individual

National Centre staff, data collection teams, local service
providers
Program goals and Project
Management

Intended Program outputs


Project requirements and activities


Defined by PISA scheduled activities
Quality standards


Macro-level goals of PISA for Development
PISA technical standards, SABER – Student
Assessment
Specific project elements

Project management tasks required for successful
implementation
Sources: PISA Technical Standards, NPM Manual, PISA for
Systems Approach for Better
Education Results (SABER) –
Student Assessment
SABER rubrics
Development Level
Dimension
ESTABLISH
LATENT
EMERGING
ADVANCE
ED
(Absence of, (On way to
(Acceptabl
D
or deviation
meeting
e
(Best
from,
minimum
minimum practice)
attribute)
standard)
standard)
Justificati
on
Enabling Environment ratings:
normative definitions
1. Latent: there is no environmental support or there are
environmental obstacles that deter program
implementation.
2. Emerging: there are political, economic or social structures
in place that may be adapted to facilitate implementation.
3.
Established: political, social or economic
structures exist that can support
implementation. MINIMUM REQUIRED
4. Advanced: political, social or economic structures are
currently providing support to similar activities.
Organizational ratings:
normative definitions
1. Latent: there is no capacity to assume this role.
2. Emerging : some capacity exists but it is not institutionalized
in a coherent administrative structure.
3. Established: some capacity exists within a
coherent administrative structure, but may
lack availability or technical skills to assume
responsibilities. MINIMUM REQUIRED
4. Advanced : capacity is institutionalized and has sufficient
resources to assume the responsibilities without developing
additional capacity.
Individual ratings: normative
definitions
1. Individuals do not have the skills and/or are resistant to
developing requisite skills
2. Individuals have foundational knowledge or personal
attributes that will enable them to acquire requisite skills or
attributes
3. Individuals have sufficient knowledge,
interest and aptitude to allow development
of requisite skills or attributes with brief
workplace training and/or independent
training and practice. MINIMUM REQUIRED
4. Individuals already have the required skills or attributes
Operationalization
PISA
Element / Description
Activity
Latent
Adequacy of
Facilities of
transportatio
the National
no vehicles
n for data
Centre
collectors
Recruitment
Commitment
and training
Insufficient
of data
of test
data collection
collection
administrator
staff
staff
s
Recruitment
and training Availability of
of test
training
administrator facilities
s
Recruitment
and training
of test
administrator
s
Commitment
of data
collectors to
training
No facilities
available (selfstudy or oneone-one)
Emerging
Establishe
Advanced
d
public/shared personal
transportation vehicles
part-time
shared with
other
institutions
institutional
dedicated
vehicles
part time with specially hired
same
for this
institution
role/project
Existing
facilities may
be repurposed
to
accommodate
training
Dedicated
training
environment is
available
Training time is
compensated
Data collectors Data collector
and is
must volunteer time is
integrated with
time or training compensated
regular duties
conflicts with but
(or staff are
other
responsibilities
hired
Capacity Building Plan
Prioritization and Trade-offs
1. Meet minimum standards for capacity
2. Prioritize immediate requirements within the PISA cycle
3. Added-value outcomes associated with PISA implementation
4. ‘Wish-list’ items
PISA
Element / Description
Activity
Latent
Adequacy of
Facilities of
transportatio
the National
no vehicles
n for data
Centre
collectors
Recruitment
Commitment
and training
Insufficient
of data
of test
data collection
collection
administrator
staff
staff
s
Recruitment
and training Availability of
of test
training
administrator facilities
s
Recruitment
and training
of test
administrator
s
Commitment
of data
collectors to
training
No facilities
available (selfstudy or oneone-one)
Emerging
Establishe
Advanced
d
public/shared personal
transportation vehicles
part-time
shared with
other
institutions
institutional
dedicated
vehicles
part time with specially hired
same
for this
institution
role/project
Existing
facilities may
be repurposed
to
accommodate
training
Dedicated
training
environment is
available
Training time is
compensated
Data collectors Data collector
and is
must volunteer time is
integrated with
time or training compensated
regular duties
conflicts with but
(or staff are
other
responsibilities
hired
PISA
Element / Description
Activity
Latent
Adequacy of
Facilities of
transportatio
the National
no vehicles
n for data
Centre
collectors
Recruitment
Commitment
and training
Insufficient
of data
of test
data collection
collection
administrator
staff
staff
s
Recruitment
and training Availability of
of test
training
administrator facilities
s
Recruitment
and training
of test
administrator
s
Commitment
of data
collectors to
training
No facilities
available (selfstudy or oneone-one)
Emerging
Establishe
Advanced
d
public/shared personal
transportation vehicles
part-time
shared with
other
institutions
institutional
dedicated
vehicles
part time with specially hired
same
for this
institution
role/project
Existing
facilities may
be repurposed
to
accommodate
training
Dedicated
training
environment is
available
Training time is
compensated
Data collectors Data collector
and is
must volunteer time is
integrated with
time or training compensated
regular duties
conflicts with but
(or staff are
other
responsibilities
hired
PISA
Element / Description
Activity
Latent
Adequacy of
Facilities of
transportatio
the National
no vehicles
n for data
Centre
collectors
Recruitment
Commitment
and training
Insufficient
of data
of test
data collection
collection
administrator
staff
staff
s
Recruitment
and training Availability of
of test
training
administrator facilities
s
Recruitment
and training
of test
administrator
s
Commitment
of data
collectors to
training
No facilities
available (selfstudy or oneone-one)
Emerging
Establishe
Advanced
d
public/shared personal
transportation vehicles
part-time
shared with
other
institutions
institutional
dedicated
vehicles
part time with specially hired
same
for this
institution
role/project
Existing
facilities may
be repurposed
to
accommodate
training
Dedicated
training
environment is
available
Training time is
compensated
Data collectors Data collector
and is
must volunteer time is
integrated with
time or training compensated
regular duties
conflicts with but
(or staff are
other
responsibilities
hired
Capacity Building Tools
1. Organization of existing resources
2. Definition of protocols to increase bureaucratic efficiency
3. Local expertise and service providers
4. International technical assistance and training
5. PISA International Contractor
Some issues for Capacity
Building Plan
Funding agreements and project budgeting
Infrastructure vs human resources
MoE internal capacity vs. service providers
Short-term PISA capacity vs long-term large-scale assessment
programming
PISA for Development
1st International Advisory
Group Meeting
ToR for an independent project review
27 – 28 May 2014
Paris, France
EDU/DCD
PISA for Development: Independent review
Scope and purpose of the review
• focused on the progress of the project in relation to
its five main outputs and extent to which the
delivery of these will achieve the project’s purpose
• purpose of review is to help understand what has
been achieved against the OECD’s original plans,
how practicable those plans were as well as how
relevant and valuable the project’s work is to
developing countries’ and development partners'
evolving education policies.
67
Project’s outputs
– Contextual questionnaires and data-collection
instruments enhanced;
– Descriptive power of cognitive assessments enhanced
– Analytical framework and methodological approach
for including out-of-school 15-year-olds in
assessments developed;
– Country capacity in assessment and analysis
strengthened
– Engagement established with developing countries
and partners for peer-to-peer analysis and learning
opportunities to support the UN-led post-2015
process
68
Main issues to be reviewed
• Impact of the project
• Relevance of the project
• Sustainability of project
achievements
• Management and partnership
arrangements
69
Impact of the project
• Achievement of outputs and the purpose
of the project
• Expected impact on policy makers and the
education systems in the participating
countries
• Evidence and indications that PISA results
will be used in policy-making
• Dissemination and use of project
deliverables
70
Relevance of the project
• Key lessons from the project to inform
work on improving education quality and
improved student learning outcomes
• The role of the project in informing
discussions of education quality and
learning outcomes
• Promotion of evidence-based policy
making
71
Sustainability of project achievements
• Achievement of capacity building outputs and
objectives
• Sustainability of the capacity that has been
built
• Lessons from capacity building
• Success of peer-to-peer learning strategies
• Likely transition of countries from PISA for
Development to main PISA
• Spill-over benefits of the project for student
assessment as a whole
72
Management and partnership
arrangements
• Effectiveness of the governance and
management structures for the project
• Effectiveness of project management
systems and processes
• Roles of PISA GB, the DAC and IAG and
TAG
• Particular successes and challenges in
implementing the project
73
Methodology
• Selection of experts
• Experts to propose a design, plan and
methodology in accordance with ToR
• Collection and analysis of documents, data
and information, interviews with
stakeholders and review of documents
produced by the project
• Possible use of surveys
74
Schedule
Inception Report by
December 2016
Presentation of initial findings and End February 2017
recommendations
Draft Report
Mid-March 2017
Presentation to IAG
Late March 2017
Final Report
May 2017
75
Deliverables
• Final report of 50 pages
• IAG and TAG will have opportunity to
comment on Inception report and Draft
report
• Final report will inform OECD’s final
report on the results of the project
76