We need to know about …
Computer Assisted Assessment
(CAA)
Dr Andrew Oliver
LTDU & BLU
University of Hertfordshire
Objective Testing (in StudyNet)


StudyNet Quizzes are ‘objective’
In other words …
–

Student provides a response to a question which
has a pre determined answer
Which means
–
–
Select solution from a list of choices (MCQ) or
Enter short text or numerical responses (matched
to correct answer)
Objective Tests

Diagnostic
–

Formative
–

Measure understanding in-class & adjust
teaching, e.g. Electronic Voting Systems (EVS)
Provide ongoing feedback in-course and
motivation, e.g. StudyNet
Summative
–
Exam type forming part of final course grade, e.g.
QuestionMark
Advantages





Immediate feedback to students (and staff)
Decreased marking time
More frequent assessment
Provide statistics (improve question design)
STUDENTS LIKE IT ! (formative that is)
Challenges



Time investment for design of questions & tests
As above but with technology
Measure high order skills ?
–

Difficult to assess essays (and hence creativity)
–


Yes – but time required to design
Well maybe, may be not
CHEATING
Network stability
Objective Testing at UH


StudyNet: designed for formative use
QuestionMark: summative (conditions apply,
contact r.a.shaw@herts.ac.uk)
StudyNet Quizzes: Question Types
–
–
–
–
–
Multiple Choice
True/False
Multiple Response
Text / Numerical input
Ranking
StudyNet Quizzes: Where are they?

Every Module website
–

Step 1: Question Wizard
–

Create questions with images, text formatting, variable
feedback/scoring etc.
Step 2: Quiz Wizard
–

Teaching Resources > Quizzes & Forms
Create quizzes with using previously created questions,
adding feedback tailored to score
Each question is a single resource & can be used in >
1 quiz
Question features

You can:
–
–
–
–
Include images (stem or option)
Feedback to option chosen
Negative scoring
Confidence based scoring
Quiz features:

You can:
–
–
–
–
–
–
Add extra info at the top & base of quiz
Tailor feedback according to user score
Provide remedial or complex material based on
score
Number questions per page
Allow re – try before viewing detailed results
Anonymous or named submission
Viewing Results




Go to Teaching Resources > Quizzes &
Forms
Find Quiz & click ‘R’
Click ‘Submissions’ to list students
Click student name for details
Statistics!

Identify & remove weak questions
–






e.g. too easy or too difficult
Also find incorrect or poorly worded questions
Pre test questions for future
AND
Problem areas in module
Struggling students
Students who are NOT being challenged
StudyNet Quiz Statistics




Go to Teaching Resources > Quizzes &
Forms
Find Quiz & Click ‘R’
Click ‘Question Statistics’ & ‘More Statistics’
Per questions: mean score, SD, Difficulty
Index, Correlation (Pearsons), Mean for
outcome (MCQ only)
Guessing




Corrective scoring
+1 correct, -1 incorrect, 0 no answer
But contentious
NB relevance decreases with as questions
increase
Confidence Based Scoring





Choose: low medium high
For question correct score 1:
Correct 1 (low) 2 (med) 3 (high)
Incorrect -1 (low) -2 (med) -3 (high)
Tell the students
Why?

Gets students to:
–
–

Think about reliability
Understand issues (not instant)
But also:
–
–
Fairer (confident vs guess)
Wake up students (confident incorrect)
Question Design

Part 1: basic tips
–
–

Question stem
Question distractors
Part 2: Bloom’s taxonomy
Design: the question stem
–
–
–
–
–
–
single, definitive statement
avoid unnecessary or irrelevant material
use clear language
use negatives sparingly
put as much of the question in the stem
Any more?
Bad question I

Morphemes
–
–
–
–
a. are made up of smaller units called phenomes
b. are NOT found in grammatical function words
c. create meaning or have a grammatical function
d. can be described as free or bound
Good question I

Morphemes are the smallest contrastive unit
of
–
–
–
–


a. compound words
b. prefixes
c. grammar
d. intransitive verbs
Clear, shorter, homogenous choices.
Provide single clear statement.
Bad Question II

Paul Muldoon, an Irish postmodern poet who
uses experimental and playful language,
uses which poetic genre in “Why Brownlee
Left”?
–
–
–
–
–
a. sonnet
b. elegy
c. narrative poem
d. dramatic monologue
e. haiku
Good Question II

Paul Muldoon uses which poetic genre in “Why
Brownlee Left”?
–
–
–
–
–


a. sonnet
b. elegy
c. narrative poem
d. dramatic monologue
e. haiku
Contains material irrelevant to the question.
Avoid unnecessary and irrelevant material.
Bad Question III

As mortality rates approach the zenith, what
is the most likely ramification for the citizenry
of an industrial nation?
–
–
–
a. an increase in the labour force participation
rate of older people
b. a dispersing effect of population concentration
c. an upward trend in the youth dependency ratio
Good Question III

A major increase in mortality rates in an
industrial nation is likely to cause
–
–
–


a. an increase in the labour force participation
rate of older people
b. a dispersing effect of population concentration
c. an upward trend in the youth dependency ratio
Too complex – tests reading comprehension
?
Use clear, straightforward language
Bad Question IV

Which of the following is not a symptom of
osteoporosis?
–
–
–
–
a. decreased bone density
b. frequent bone fractures
c. raised body temperature
d. lower back pain
Good Question IV

Which of the following is a symptom of
osteoporosis?
–
–
–
–


a. decreased bone density
b. raised body temperature
c. hair loss
d. painful joints
Use negatives sparingly.
If needed then capitalize, underscore, embolden or
otherwise highlight.
Bad Question V

Maslow’s theory of growth motivation asserts
that
–
–
–
a. humanistic needs are physiological
b. humanistic needs are emotional
c. humanistic needs are intellectual
Good Question V

Maslow’s theory of growth motivation asserts
that humanistic needs are
–
–
–

a. physiological
b. emotional
c. intellectual
Put as much of the question in the stem as
possible, DON’T duplicate in each of the
options
Design: the distractors
–
–
–
–
–
ensure there is only one correct answer
use plausible distractors
avoid clues to the correct answer
base distractors on common student errors
(phenomenographics)
Any more?
Bad Question VI

Which of the following texts is considered to
represent the pinnacle of modernist
achievement?
–
–
–
–
–
a. The Waste Land
b. Middlemarch
c. “Ode to a Nightingale”
d. Ulysses
e. Ethan Frome
Good Question VI

Which of the following texts represents one of the
high points of modernist achievement?
–
–
–
–
–


a. The Waste Land
b. Middlemarch
c. “Ode to a Nightingale”
d. Ethan Frome
e. “My Last Duchess”
ensure that there is only one correct response.
both options ‘a’ and ‘d’ could be considered to be
correct.
Bad Question VII

The Dichotic Listening Test determines
which side of the brain is directly involved in
–
–
–
–
a. tone perception
b. selection attention
c. cognition
d. hearing sounds
Good Question VII

The Dichotic Listening Test determines which side of
the brain is directly involved in
–
–
–
–


a. tone perception
b. selection attention
c. pitch constancy
d. hearing sounds
Use only plausible and attractive alternatives as
distracters.
‘c’ not plausable
Bad Question VIII

A fertile area in the desert in which the water
table reaches the ground surface is called an
–
–
–
–
a. mirage
b. oasis
c. water hole
d. polder
Good Question VIII

A fertile area in the desert in which the water table
reaches the ground surface is called a/an
–
–
–
–


a. mirage
b. oasis
c. water hole
d. polder
Avoid giving clues to the correct answer
uses the article “an” which identifies choice “b” as
the correct response.
Bloom’s Taxonomy est. 1956




hierarchy of educational objectives
divides cognitive objectives into subdivisions
range from the simplest behaviour to the
complex
other systems or hierarchies exist
Cognitive subdivisions






Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Bloom’s Taxonomy: Knowledge




Recall of information
Bringing to mind the appropriate information
Learning objectives: know facts, know
theories, know terms, know events, know
basic concepts, know places, know methods
Question cues: list, define, label, describe,
name
Aim - to recall a specific date
Question: in which year did the American Civil
War end?
a. 1832
b. 1865*
c. 1857
d. 1888
Bloom’s Taxonomy: Comprehension



Ability to grasp meaning of the material – to
interpret, to predict, to estimate
Learning outcomes: understand facts,
interpret graphs, translate verbal to math
formulae, estimate consequences from data.
Question cues: interpret, discuss, predict,
summarise, classify
Aim - to understand and interpret
definitions, relationships & analogies
Question: which one of the following describes what takes place in the socalled PREPARATION stage of the creative process, as applied to the
solution of a particular problem?
1. The problem is identified and defined.
2. All available information about the
problem is collected.
3. An attempt is made to see if the proposed
solution to the problem is acceptable.
4. The person goes through some experience
leading to a general idea of how the problem
can be solved.
5. The person sets the problem aside, and gets
involved with some other unrelated activity.
KNOWLEDGE is recalled and understanding (COMPREHENSION) tested of the
meaning of each term (preparation).
Second Example: comprehension
Question: In the following, a related pair of words is
followed by five more pairs of words. Choose the
response pair that best expresses a relationship
similar to that expressed in the original pair.
QUENCH : THIRST
a. staunch : wound
b. douse : fever
c. antidote : poison
d. extinguish : fire*
Bloom’s Taxonomy: Application




Ability to use learned materials in NEW
situations
Application of rules, methods, concepts and
theories.
Learning outcomes: apply principles /
theories to new situations, solve math
problems, construct graphs.
Question cues: apply, demonstrate, show,
relate, calculate
Aim: to calculate velocity
Question: End A of the cord is moving 6m/s to the left.
Compute the velocity of the block B and select one
of the responses below.
a. 2 m/s (to left)
b. 3 m/s (to left)
c. 6 m/s (to left)
d. 12 m/s (to left)
e. none of the above
Nb there would be a diagram here!
Second Example: application
Which one of the following values
approximates best to the volume of a
sphere with radius 5m?
1.
2.
3.
4.
5.
2000m³
1000m³
500m³
250m³
125m³
Bloom’s Taxonomy: Analysis




Ability to break down material into
components and understand patterns
(underlying structure)
Recognition of parts and their relationship to
each other
Learning objectives: recognise unstated
assumptions, evaluate the relevancy of data
Question cues: analyse, arrange, order,
explain, connect, infer, compare, categorize
Aim: to analyse and infer from a
geological map
Question 1. LANDSLIPS
At which of the following contacts
between strata would you expect
MOST landslips to occur?
Choose 2 of the following options.
a. Chalk above Gault Clay
b. Gault Clay above Lower Greensand
c. Corallian limestone above Oxford Clay
d. Cornbrash limestone above inferior Oolite limestone
Bloom’s Taxonomy: Synthesis




Use old ideas to create new ones
Organise and relate knowledge from several
areas
Learning objectives: writing themes & stories,
formulating a new scheme for classifying
objects (or events, or ideas).
Question words: integrate, modify, invent,
design, compose, plan, formulate, arrange
Aim: to organise and arrange appropriate critical
terms in order to construct a geological analysis of
the following photographic image
Question: Move the appropriate
descriptive terms from the
list to the ‘form’ and
‘attitude’ boxes below.
Question created by
Professor Don MacKenzie,
TRIADS.
Bloom’s Taxonomy: Evaluation





Make judgements of / evaluate data
Assess value of ideas, theories
Compare and discriminate between ideas
Learning outcome: judge the adequacy with
which conclusions are supported by data
Question cues: appraise, evaluate, defend,
rank, conclude, discriminate, recommend
Aim: to assess the condition of a
patient based on various readings
Question: An adult subject breathing air was found to have the following lung volumes:
• Vital capacity - 3.5 litres
• Forced expiratory volume in 1 sec (FEV1) - 2.8 litres
• Functional residual capacity (FRC) - 1.8 litres
• Residual volume (RV) - 0.8 litre
True / False
a. There is no obstruction to airflow.
b. The subject must be abnormal.
c. The expiratory reserve volume is 1 litre.
d. All of these measurements could have been made using only a spirometer.
e. There would be approximately 250 ml of oxygen in this subject’s
lungs at the end of a tidal expiration.
Question by Dr David A. Davies, The University of Birmingham (MEDWEB ComputerAssisted Assessment web site - http://medweb.bham.ac.uk/caa)
References I




Designing and Using Objective Tests, Dr
Joanna Bull, The CAA Centre, 2001
Other resources :
Designing and Managing Multiple Choice
Questions, Carneson, J., Delpierre, G. &
Masters, K., University of Cape Town
Measurement and Evaluation in Teaching,
Gronlund, N. E., 1971 (In the library)
References II

StudyNet online tutorials (animated)
–
–
–

Log in StudyNet
Technical Support section
Online Tutorials (under ‘Using StudyNet’)
Covers ‘how to’ and includes examples,
statistics info
‘How to’ series





How to Blog in 5 minutes (Word)
How to Wiki in 5 minutes (Word)
How to create a Podcast in 5 minutes (Word)
How to convert audio files to MP3 in 5
minutes (Word)
How to use RSS in 5 minutes (Word)
‘You need to know…’ series










You need to know about Podcasting (Slides)
You need to know about Podcasting: a teaching and learning guide
(Word)
You need to know about Online Discussions: a teaching and
learning guide (Word)
You need to know about Blogs (Slides)
You need to know about Wikis (Slides)
You need to know about RSS (Slides)
You need to know about Web 2.0 (Slides)
You need to know about Computer Assessment (Slides)
You need to know about Effective Question Design (Slides)
You need to know about Electronic Voting Systems (Slides)
Also available:
Podcasting: audio talk on the benefits of
podcasting (hi quality)
 Computer Assessment: using statistic
analysis to design effective questions
For copies please contact: Dr Andrew Oliver.
LTDU, College Lane LRC
(a.oliver@herts.ac.uk, ext. 4754).
