Network News
Issue 6
Welcome!
This month we have an article by Vicki Crisp about the ‘Question Writers’ Interactive Learning Tool’. The
material in this has been used for training sessions for OCR and Cambridge International Examinations
(CIE) examiners over some years now, and has also been presented in training sessions in other countries.
Pitching one’s questions correctly, either as a teacher or examiner, is by no means easy and so it is useful
to have some concepts which help us to think about what we are intending as we draft them. Examiners
who work for Cambridge ESOL, CIE or OCR are able to access a special section of our interactive website,
AssessNet (http://AssessNet.org.uk) to see and use the materials. In addition, Vicki is going to run a twohour online question-and-answer session on 31 May based on the QWILT materials. This is our first online
seminar for examiners, so we hope that you will find it useful.
The Cambridge Assessment Network launched AssessNet, an online environment in which to get more
information about the world of assessment, on 31 March 2006. AssessNet is not the Cambridge
Assessment corporate website, but a learning environment in which you can participate and respond to
what you find there. We hope it will grow into an online networking tool that will be of use to anyone who
works in assessment. To become that, it will need the active involvement of people like yourselves, so do
use the opportunities it offers to take part.
Andrew Watts
Director
Cambridge Assessment Network
1
Network Event
The Question Writers’ Online Learning Tool, or
QWILT, was developed by researchers at
Cambridge Assessment. Vicki Crisp, Research
Officer, explains more about QWILT here.
An Online Tool for Question Writers
Over a number of years the Question Difficulty
Project Team at UCLES/Cambridge Assessment
has developed a group of tools for question
writers based on research into the cognitive
processing that goes on when a candidate
attempts a question and the effects that certain
features of questions can have on how difficult a
question is, how students respond and on the
validity of questions. It is important for examiners
to be in control of what is being measured in
exams and hence it is important that questions
are valid and measure the knowledge or skills
that they were intended to measure.
Analysis of scripts and trialling of adapted
versions of questions has led to a defined list of
question features that affect question difficulty.
These are known as ‘Modifiers of Difficulty’
(abbreviated to MoDs). Modifiers of Difficulty can
be valid or invalid depending on whether or not
the added difficulty or easiness was intended as
part of the task. Other literary and practical
research has led to the development of three
other tools for question writers. Firstly, the ‘Scale
of Cognitive Demands’ which allows examiners to
rate the demands of their questions and/or
consider how to adjust the demands of a
question. Secondly, a model of the psychological
processing involved when a student attempts a
question. And thirdly, a way of helping examiners
predict as much as possible the range of answers
(or types of answers) that a question will elicit (the
‘Outcome Space Generator’).
The tools and materials have been used in faceto-face training and development sessions over
the years and have more recently been built into
a web-based computer tool which was completed
in 2005 and is known as QWILT (Question
Writers’ Interactive Learning Tool). The aim of this
computer system is to make this research of
practical use to examination question writers
(Principal Examiners, Revisers, etc.) to help to
promote valid assessment. The system itself has
been designed with ease of use in mind and with
the help of feedback from a number of question
writers. So far QWILT has been available to
delegates who have attended training or
development sessions but it is now available to all
Cambridge Assessment examiners online via
AssessNet. If you would like access please email
Sandra Gibson, the AssessNet administrator, at
gibson.s@cambridgeassessment.org.uk,
with
your name and creditor number.
Vicki Crisp will be online in the ‘Examiners’ area
of AssessNet, from 2 to 4 pm on Wednesday 31
May, to discuss QWILT and its use and to take
any questions you have about it. (This event is
open to Cambridge Assessment examiners only.)
Log on to AssessNet at http://AssessNet.org.uk to
take part in the discussion.
E-list discussions
The main discussion theme this month has been
the issue of including creationism in secondary
science lessons again. Some of the comments
you made included:
‘There is a lot of good scholarship at the science /
religion interface, including a project, based at the
Ian Ramsey centre at Oxford, to bring it alive and
in to the classroom.’
(See http://www.ianramseycentre.org/)
2
‘Verification of theories is rarely as clear cut as is
often believed - and beauty in a theory can be as
highly prized as in any art form!’
Keep the discussions going! Let us know more
about what you think about this issue, or begin a
new discussion thread on your own topic. If you
are a registered list subscriber, you can
see
the
discussion
archives
at
http://lists.ucles.org.uk/lists/private/assessmentnetwork-discuss/
(Please note that all discussion contributions
reflect individuals’ views only, and do not
necessarily represent corporate policy. We
suggest, therefore, that you do not include your
job title and employer with your message, in
order to emphasise that you are expressing a
personal point of view.)
•
•
•
Northcote, M (2003). Online assessment in
higher education: The influence of pedagogy
on
the
construction
of
students'
epistemologies. (From Issues in Educational
Research)
Haughey, M & Muirhead, B (2005). The
pedagogical and multimedia designs of
learning objects for schools.
(From
Australasian
Journal
of
Educational
Technology)
Lester, Nita C (2005).
Assessment in
multiage primary classrooms. (From Issues
in Educational Research)
To access these, or any other articles in the
Cybrary, please go to http://AssessNet.org.uk and
click on the ‘Resources’ tab along the top of the
page, then on ‘Cybrary’. Please enter your
search terms in the blue box. If you
have
any
problems,
please
contact
assessnet@cambridgeassessment.org.uk
Cybrary
The Cybrary is an authoritative resource within
AssessNet which brings together a range of
reference material and resources to provide you
with access to valuable information which can
facilitate learning and contribute to your personal
development.
Items in the Cybrary are drawn from across the
Cambridge Assessment Group and from a range
of external sources, including websites, research
reports, study data and news articles.
We will continually update and increase the
volume of resources available within the Cybrary
to ensure that it remains a relevant and dynamic
resource.
Some of the articles added to the Cybrary this
month include:
AssessNet
AssessNet,
the
Cambridge
Assessment
Network’s online communication tool for building
a community of practice for worldwide
assessment professionals, was launched on 31
March 2006. Since the launch, people from all
over the world have been logging in to network
with their assessment peers and use the exciting
features on offer.
AssessNet is still very new and we would
welcome your contributions and opinions. If
you have anything to say about the
site, or if
you would like to contribute to the discussion
forums,
please
contact
us
at
thenetwork@cambridgeassessment.org.uk or log
in to AssessNet at http://AssessNet.org.uk, click
on the bold ‘discussions forums’ link on the
homepage, and then on ‘General discussion’ and
share your views with the community.
3
Thinking Skills
This month’s article is by Nigel
Upton (left), Product Manager at
CIE. He discusses the recently
developed Thinking Skills tests and
their uses, as well as providing
some student views on the subject.
Why Thinking Skills?
Success in education, the workplace and life in
general depends on our ability to make sense of
the world around us. What we read, what we hear
and what we see influences our interpretation of
events and our perception of the way the world
‘is’. These learning experiences then affect the
views we hold and the decisions we take.
Important decisions – such as what we write in
examinations, what courses we choose to study,
and the value that we attach to particular books
and ideas. Critical decisions – such as how we
settle conflicts and how we vote (if we live in a
democracy). The ability to make decisions such
as these is based on our ability to recognise
options and formulate opinions. So how do we
hone our ability to interpret the information so that
it helps, rather than hinders, our progress? In
many societies formal education begins at school.
Subject teaching involves not only the acquisition
of knowledge but also development of the ability
to think and reason. In the sciences, students
learn about ‘methods’ for solving problems or
understanding
physical phenomena. In the
humanities, students develop the ability to be
‘critical’ of what they read. The ability to think
becomes increasingly important as students
move from school to higher education.
Teaching Thinking Skills
Should
‘thinking
skills’
be
developed
independently of specific subject teaching? It is
an interesting question and one that does not yet
have a fully agreed answer! CIE believes that
students can improve their thinking skills through
the study and practice of well-defined approaches
to problem solving, critical thinking and reasoning.
However, we also believe that students gain when
thinking skills are taught as an integral part of
their learning in other subjects. By combining
contemporary topics with specific types of
thinking skills, teachers can construct some
stimulating examples on which students can
work. Printed media can provide stories that
students will relate to – because they are topical,
or relevant to the students’ age range. Students
can use thinking skills to decide whether the
journalists’ views are rigorous or just biased
opinion. In lessons, teachers can create lively
debate, with free and open expression by the
students. An appreciation of thinking skills can
enable students to have greater confidence when
expressing their views and engaging in
constructive arguments. Teachers can use these
approaches to develop critical and investigative
thinking and to encourage informed and
disciplined reasoning.
Cambridge Experience
Cambridge Assessment’s experience in this area
has resulted in a reputation as leader in the field.
Cambridge Thinking Skills Assessment (TSA) is
used by a number of colleges and departments
as part of the entrance process to the University
of Cambridge. More recently, Cambridge has
developed the Biomedical Admissions Test which
is used in the selection of medical and veterinary
science students at the Universities of Oxford,
Cambridge, London (University College) and
Bristol, as well as by the Royal Veterinary
College. CIE has also developed A and AS Levels
in Thinking Skills. The AS Level has been
established for a number of years; now, with
interest growing in this subject, CIE is launching a
full A Level.
These qualifications aim to:
• Develop a specific set of intellectual skills
independent of subject content
4
•
•
Encourage students to develop more mature
and sophisticated ways of thinking
Improve ability to understand, analyse and
resolve problems
But what do these problems look like? Let’s look
at a typical AS Level example:
AS Level Thinking Skills – Example Question
Mountaineering is a very hazardous sport, and it
takes many years of experience to learn how to
survive in mountainous regions. The weather can
change abruptly when you are a few thousand
feet above sea-level, sometimes with disastrous
consequences. Even if you know what signs to
look for, and what to do if violent weather
descends, you still run life-threatening risks.
Therefore someone without knowledge and
experience has no chance of survival if they are
caught in a bad storm at high altitude.
Which of the following identifies the main flaw
in the argument?
A. Even at sea-level, bad weather can have
disastrous consequences.
B. Even experienced mountaineers can have fatal
accidents.
C. Even inexperienced mountaineers can have
lucky escapes.
D. Even in good weather, climbing high
mountains can be dangerous.
E. Even weather experts can make totally
incorrect forecasts.
The Answer and the ‘Distractors’
The fault in the reasoning is that it draws much
too strong a conclusion. The dangers that face
even experienced climbers do imply that an
inexperienced person has a poor chance of
survival in high altitude storms, but not that such
a person has no chance. C points out one of the
reasons that the conclusion goes too far. A, B and
D all lend weight to the argument, rather than
exposing any deficiency. E misses the point of the
argument.
The problem shown is an example of ‘recognising
flaws in reasoning’. There are actually 16 subskills which are developed under the three main
areas of problem solving, critical thinking and
reasoning skills.
Sixteen Thinking Skills
Critical Thinking
Recognising reasoned arguments
Identifying conclusions
Drawing conclusions
Recognising implicit assumptions
Assessing the impact of further evidence or
information
Recognising flaws in reasoning
Selecting plausible explanations
Recognising the logical functions of key elements
of an argument
Understanding and clarifying key terms and
expressions
Problem Solving
Data selection and processing
Finding methods and procedures
Identifying relationships and hypotheses
Analysing and modelling
Reasoning Skills
Producing further argument
Engaging in inference and deduction
Analysing and evaluating further evidence and
argument
Problem solving requires some of the basic
mathematical skills that are usually learnt at junior
school. Students also need to learn how to
construct their own answers and present their
own reasoning. Therefore at AS level, in addition
to multiple choice questions, students take a
written paper, concentrating on inference and
deduction, analysis, evaluation and further
argument.
But do students enjoy Thinking Skills? Well, here
is some of the feedback that students at
5
Wellspring School in Argentina gave to their
teacher, Susan Hillyard.
‘I think the subject is very good and most
important; very useful for day-today life. It’s useful
for other subjects too.’ Sebastian Fernandez Vigil
‘Thinking skills was quite useful for me. I enjoyed
the lectures a lot … I’d like to continue studying
TS.’ Nicolas Bujak
‘I think it is a productive subject, it is a kind of
exercise that helps you in the development of
your brain skills. This is very useful for the future.’
Anon
A Level in Thinking Skills
The new syllabus provides students with the
opportunity to obtain a full A Level, while building
on the established AS Level syllabus. To obtain
the A Level, candidates will take the same papers
as at AS level (papers 1 and 2), plus two more.
The third paper tests more advanced skills in
problem solving and critical thinking. Questions
are more varied and searching and students may
need to combine several of the skills listed in the
table in order to reach the answer. They will also
need to apply some more mathematical
techniques, such as basic probability. However,
the emphasis will be on the application of such
techniques rather than on a deep knowledge of
mathematics. The fourth paper, Applied
Reasoning, assesses a student’s ability to apply
problem solving and critical thinking skills to reallife situations which may be encountered in higher
education, at university, or in employment. For
instance, in terms of problem solving, students
may be asked to develop a model, carry out an
investigation, or analyse complex data and draw
conclusions.
Students will also have to demonstrate critical
thinking skills by constructing a reasoned case. A
number of ‘documents’ will give differing
perspectives on an issue: information may be
incomplete, superfluous or conflicting. The
student needs to select and synthesize
information in order to present a clear conclusion,
recommendation or decision, supporting this with
appropriate justifications, and dealing with
objections or alternative arguments. In addition to
providing an A Level qualification of value in its
own right, students will also develop skills – rather
than knowledge – that will be of great value in
their university studies or in their future careers.
The first examinations in A Level Thinking Skills
will be taken in June 2007, with the syllabus
available to Centres in 2005. Excellent resources
are available for teachers and students, and an
overview is given in the 2006 AS syllabus reading
list
(which
can
be
downloaded
from
www.cie.org.uk). CIE is fully committed to
supporting this subject. Thinking Skills, published
by Cambridge University Press in May 2005, is
the recommended text. The book’s authors, Geoff
Thwaites and John Butterworth, have long been
involved in the development of CIE’s Thinking
Skills qualifications. There is also an online
discussion group in which teachers can share
information about resources and teaching
approaches.
The Future Place of Thinking Skills in Schools
The strong take-up of critical thinking studies in
UK schools is a trend we believe will also be seen
at Cambridge Centres around the world. This is a
growth subject and one that actively helps
prepare students for higher education and for a
wide range of careers. Thinking skills are of
particular benefit to those considering a career in
law,
scientific
research,
social
science,
journalism, medicine, business, accounting or
engineering. However, an improved ability to think
clearly and decisively would certainly be of value
to us all!
What do you think about Nigel’s comments? Do
you agree with him? Whatever your views on the
subject, why not share your comments with
6
others? Send them to assessment-networkdiscuss@lists.ucles.org.uk
For details of opportunities at Cambridge
Assessment, please visit our website at
http://www.cambridgeassessment.org.uk/workingf
orucles/
Events
3 May/27 June 2006
Harnessing Technology: Beyond Technology
National College for School Leadership (NCSL)
http://www.ncsl.org.uk/programmes/slict/slictconference2006.cfm
23 May 2006
Assessment Conference: Focussing on Feedback
Higher Education Academy
http://www.heacademy.ac.uk/events/List_4399.ht
m
8/9 June 2006
Advancing Evidence-Informed Practice in HE
Learning, Teaching and Educational Development
Staff and Educational Development Association
(SEDA)
http://www.seda.ac.uk/confs/liv06/liv06.htm
15 June 2006
Blended Learning Conference
Higher Education Academy
http://www.heacademy.ac.uk/events/List_4033.ht
m
20/21 September 2006
Scottish Learning Festival
Scottish Education and Teaching with Technology
(SETT)
http://www.ltscotland.org.uk/sett/
Products and projects
This month’s new projects news comes from the
ICT Team, based within the Assessment
Research
and
Development
division
at
Cambridge Assessment. The ICT Team contains
four members of staff who carry out a broad
range of technical work for the company. Each
member of the team has an extensive range of
skills, complementing one another to enable them
to cover the entire lifecycle of a project. The
team:Jeremy Costello - Projects Officer, specializes in
Systems Analysis, Reporting, Database Design
and Cuban cuisine;
George MacKerron - Web Designer/Developer, is
a whizz when it comes to Website Design and
Development and playing Samba drums;
James Ordish - Computer Officer, knows all about
Operating Systems, Hardware, Support and
beating Tom at any sport he wishes;
Tom Pelly - Systems Developer, dedicates his
time to Systems Design and Development, and
wishing he was Tom Cruise.
Vacancies
7
We hope you have enjoyed this issue of Network
News. Issue 7 will be out at the end of May.
Please send any feedback on the newsletter,
and
any
articles
that
you
may
like
to
contribute
to
Issue
7,
to:
thenetwork@cambridgeassessment.org.uk.
We look forward to hearing from you!
The remit of the team is to look at, develop or
create new technologies that can be introduced
within the Division and further afield in the
Cambridge Assessment Group. The team has
been hard at work for the last few months
creating
Campus
(an
internal
Virtual
Learning
Environment)
and
AssessNet
(http://AssessNet.org.uk), projects that are now
live.
Other recent pieces of work include: wireless
networking; SWIFT (Secure Web Interface for File
Transfer) a site for secure exchange of files
between Cambridge Assessment and our
partners; and the uniTEST results site
(http://www.unitest.org.uk/).
Future developments will encompass: Mobile
assessment
and
learning,
Gaming
and
Simulations, Psychometrics, Biometrics, Automarking and Functional Skills Diagnostics.
For
further
information
contact
costello.j@cambridgeassessment.org.uk.
Next issue
8