Wynne Harlen
OECD Skills Strategy
 “In the context of the OECD Skills Strategy, the concepts
of ‘skill’ and ‘competence’ are used interchangeably. By
skill (or competence) we mean: the bundle of knowledge,
attributes and capacities that enables an individual to
successfully and consistently perform an activity or task,
whether broadly or narrowly conceived, and can be built
upon and extended through learning.”
Towards an OECD Skills Strategy (OECD 2011 p7 foot note)
Agenda
 Starting point: aims of science units and courses (Access and






Higher)
Inquiry as a means to developing scientific literacy
Research into development of understanding through inquiry
Challenges to assessing understanding and skills (particularly
context sampling error )
Examples of questions assessing skills and understanding in
science
Alternatives to external papers
Pros and cons of using teachers’ assessment.
Unit and course outcomes
Unit assessment
 “Learners who complete this Unit will be able to:
 apply skills of scientific inquiry and investigation in the
context of the unit (eg physics of dynamics and
astrophysics)
 demonstrate knowledge and understanding related to the
content of the units (eg physics of dynamics and
astrophysics) “
Scientific literacy as an aim
 “The Course will therefore encourage learners to become
scientifically literate citizens, while developing their
literacy and numeracy skills. It will also develop learners’
investigative and experimental skills in a (physics) context.
Learners will recognise the impact (physics) makes on
their lives, the environment and society. Through this
Course, learners can develop relevant skills for learning,
for use in everyday life and across all sectors of
employment.”
A note on scientific literacy (SL)
 Scientific literacy as a metaphor in which the traditional
sense of literacy assumes another meaning (Bybee).
 SL refers to being well-educated and well-informed in
science. It does NOT mean being able to read and write
about science or being able to understand science
vocabulary.
 Development of SL: scientific illiteracy --> nominal SL -->
functional SL --> conceptual and procedural SL -->
multidimensional SL
 Through scientific inquiry students’ advance their
understanding of scientific concepts and the nature of
science, that is, procedural literacy.
The role of inquiry in science education
 Through inquiry students progressively developing key
scientific ideas through learning how to investigate and
build their knowledge and understanding of the world
around. They use skills employed by scientists such as
raising questions, collecting data, reasoning and
reviewing evidence in the light of what is already known,
drawing conclusions and discussing results.
(based on IAP 2011)
Is it necessary to assess inquiry skills?
 Yes, if inquiry skills and competencies are seen as key
outcomes in their own right, not just a means to
developing understanding (of science and about science)
 Skills needed for continued learning –
 Students cannot learning in school everything they will
need to know in adult life. What they must acquire is the
prerequisites for successful learning in future life….Students
must become able to organise and regulate their own
learning, to learn independently and in groups and to
overcome difficulties in the learning process. This requires
them to be aware of their own thinking and learning
strategies and methods. (OECD 1999)
How does this development of understanding happen?
New experience/question
Possible explanation
Prediction
Plan and conduct
investigation
Interpret data
Conclusion
Alternative
ideas
Existing idea
Bigger
idea
Does inquiry ‘work’?
 Do inquiry-based activities lead to understanding more
than do more direct teaching approaches?
 Inquiry synthesis project
Minner, D.D., Levy, A.J., & Century, J. (2010). Inquiry-based science
instruction – what is it and does it matter? Results from a research
synthesis years 1984-2002.
Journal of Research in Science Teaching, 47 (4), 474-496
Inquiry Synthesis Project
 Synthesis of 138 studies
• 30 experimental; 35 quasi-experimental; 73 non-experimental
 Most (105) conducted in the United States
 Amount of instruction delivered in the interventions varied
widely (mean ~ 12 lessons)
 Most (56%) did not report if the outcome measurement
instruments were new or existing
• Generally low rigour regarding reliability and validity of the
measures used
Inquiry Synthesis Project findings
 The majority of studies (51%, n=71) showed positive impacts of some





level of inquiry science instruction on student content learning and
retention
The extent of ‘inquiry saturation’, or specific elements of inquiry, or
specific components of instruction was not statistically significantly
associated with better learning outcomes
Active thinking was a significant predictor of learning outcomes
Drawing and thinking about conclusions from data was a marginally
significant predictor of learning outcomes
5 studies (of the 6 that were reviewed) of the effect of hands-on
activities on student learning showed a statistically significant
difference
6 studies (of the 9 that were reviewed) of the effect of giving students
greater responsibility showed a statistically significant difference
Course assessment
 “Learners will draw on, extend and apply the skills they
have learned during the Course.
 This will be assessed within
 a question paper (external paper)
 and a case study (stimulus-based assessment of ability to
analyse and draw conclusions)
 requiring demonstration of the breadth of knowledge,
skills and understanding acquired from across the Units
and of how they can be applied in unfamiliar contexts
and/or integrated ways.”
Challenges for assessment
 To show that the skills, knowledge and understanding
can be used in ‘unfamiliar contexts’
 The need for authentic situations, real contexts
 Complexity in presenting the context
 Bias of familiarity with the context
 Preference for ‘unfamiliar contexts’ in order to assessment
application of understanding and capabilities
 Context sampling error
 Impact on reliability and validity
Sources of error in assessment of
understanding
 Requires application of knowledge, in a context
 How to choose – what are ‘unfamiliar contexts’?
 Lack of knowledge or lack of ability to apply?
 Context sampling error (Wiliam 2001).
The context effect in assessment of inquiry
skills
 (Brown et al 1996)
 ‘Little evidence of the generalisability of skills assessment across
science subjects’
 Need for continued assessment of skills within subject domains
 High context sampling error when few contexts can be used.
 Possible solutions:
 Find contexts likely to be unfamiliar to all students
 Increase contexts by assessing smaller, specific aspects of inquiry
 Use the range of contexts encountered in the course for
moderated teacher’s assessment (e.g. portfolio).
Important differences between assessing
populations and assessing individuals
 PISA and national monitoring programmes
 Can minimise the context sampling error by using a bank of items
and random allocation of items to students
 But, restricted in the form of the item on paper or on screen
 For population, high reliability (accuracy) but lower validity
 Assessing individuals
 Using examinations and tests, same items of all, limited range of
contexts
 Large context sample error; low validity
 Using performance in regular and special inquiry-based tasks
raises validity
 Higher reliability tends to lower validity.
What Human activities contribute to climate change?
Read the following information and answer the questions which follow.
The burning of coal, oil and natural gas, as well as deforestation and
various agricultural and industrial practices, are altering the composition
of the atmosphere and contributing to climate change. These human
activities have led to increased concentrations of particles and greenhouse
gases in the atmosphere. The relative importance of the main contributors
to temperature change is shown in the figure
Cooling
Relative Importance
Carbon dioxide
Methane
Particles
Particle effects on clouds
known effect
possible effect
Heating
Bars extending to the right of the centre line indicate a heating effect. Bars
extending to the left of the centre line indicate a cooling effect. The relative effect
of ‘Particles’ and ‘Particle effects on clouds’ are quite uncertain: in each case the
possible effect is somewhere in the range shown by the light grey bar.
Figure 1 shows that increased concentrations of carbon dioxide and methane
have a heating effect. Increased concentrations of particles have a cooling effect
in two ways, labelled ‘Particles’ and ‘Particle effects on clouds’.
Questions
1. Use the information in Figure 1 to support the view that
priority should be given to reducing the emission of
carbon dioxide from the human activities mentioned.
2. Use the information in Figure 1 to support the view that
priority should not be given to reducing the emission of
carbon dioxide from the human activities mentioned.
Early immunisation
 As early as the 11th century , Chinese doctors were
manipulating the immune system. By blowing pulverised scabs
from a smallpox victim into their patients’ nostrils. They could
often induce a mild case of the disease that prevented a more
severe onslaught later on. In the 1770s people rubbed their
skins with dried scabs to protect themselves from the disease.
These primitive practices were introduced into England and
the American colonies. In 1771 and 1772, during a smallpox
epidemic, a Boston doctor named Zabdiel Boylston tested an
ideas that he had. He scratched the skin on his six year old
son and 285 other people and rubbed pus from small pox
scabs into the wound. All but six of his patients survived.
 Questions:
 1. What idea might Zabdiel Boylston have been testing?
 2. Give two other pieces of information that you would need to decide
how successful Boylston’s approach was
Chocolate
 A newspaper article (Daily Mail on March 30, 1998) recounted
the story of a 22 year old student, named Jessica, who has a
‘chocolate diet’. She claims to remain healthy, and at a steady
weight of 50Kg, whilst eating 90 bars of chocolate a week and
cutting out all other food, apart from one ‘proper meal’ every
five days. A nutrition expert commented: “I am surprised
someone can live with a diet like this. Fats give her energy to
live but she is not getting a balanced diet. There are some
minerals and nutrients in chocolate, but she is not getting
enough vitamins. She could encounter serious health
problems in later life.”
Imaginary animal
Threats to validity
 Communicating an unfamiliar context makes demands on




reading, interpretation of representations
The extent to which the assessment specific skills
equates to assessment of capability in inquiry and
investigations
Other features of context (beyond familiarity) may affect
engagement
What was once a ‘novel’ context can become familiar
once used
Science fiction a questionable context for assessing
understanding and skills in science.
Some advantages of assessment by teachers
 Potential for the full range of goals to be included as
teachers collect evidence as part of their normal work
with students
 Can relieve the pressure, on students and teachers, of
terminal tests and examinations
 Teachers can use information about students formatively
as well as summatively
 Can release resources (time and other costs) for
alternative use
Some disadvantages
 Teachers’ judgements often perceived as being unreliable
 Increase in work load for teachers
 Can lead to the same distortion of teaching as testing if the
results are used for high stakes accountability.
.
Dimensions of approaches to assessment by
teachers
2
1
3
4
Greater
specification
of tasks
Assessment criteria more detailed
A portfolio system (eg Queensland)
 School-based
 Constructed over all units in the course
 Selected and assessed by reference to strict criteria
 Full range of evidence – projects, assignments,




observations of practical performance, field work
Mandatory criteria
Up-dated to include most recent relevant evidence
Five levels of achievement
Moderated.
A formative function
 Assessment of any kind should ultimately improve
learning (Harlen, 2010)
 The collection of data during the year informs the
teacher of the areas of more and less progress of
students (formative use of summative information)
 The practice of assessment for learning can inform the
judgement of achievement at a particular time
(summative use of formative information).
References
Brown, CR., Moor J., Silkstone, B.E., and Botton, C. (1996) The construct validity
and context dependency of teacher assessment of practical skills in some preuniversity level science examinations, Assessment in Education, 3 (3) 377-391.
Bybee, R., Fensham, P and Laurie, R (2009) Scientific literacy and contexts in
PISA 2006 science. Journal of Research in Science Teaching 46 (8) 862-864.
Harlen, W. (2001) The assessment of scientific literacy in the OECD/PISA project.
Studies in Science Education, Vol 36, 79-104
Harlen, W. (2005) Trusting teachers’ judgment: research evidence of the reliability
and validity of teachers’ assessment used for summative purposes. Research
Papers in Education 20 (3) 245-270
Harlen, W. (2010) What is quality teacher assessment? In Gardner et al
Developing Teacher Assessment. Maidenhead: Open University McGraw-Hill
IAP (InterAcademies Panel) (2011) Taking IBSE into Secondary Education.
Report on a conference held in York, October 2010. London: Welcome Trust.
Minner, D.D., Levy, A.J., & Century, J. (2010). Inquiry-based science instruction –
what is it and does it matter? Results from a research synthesis years 19842002. Journal of Research in Science Teaching, 47 (4), 474-496
Wiliam, D. (2001) Reliability, validity and all that jazz, Education 3-13 (3) 17-21