Leonardo
Exam question templates, valuation key,
and instructions and methods for
confidentiality control
WP1
Technical Note
November 2007
Leonardo
Agern Allé 5
DK-2970 Hørsholm
Denmark
November 2007
Tel:
Fax:
Client
Client’s representative
Project
Project No
Leonardo
Authors
+45 4516 9200
+45 4516 9292
dhi@dhigroup.com
www.dhigroup.com
80137-01
Date
November 2007
Anke Oberender
Approved by
Christian Grøn
Technical Note
Revision
Description
Key words
By
Checked
Approved
Date
Classification
Open
Internal
Proprietary
Distribution
DHI:
No of copies
CONTENTS
1
AIM OF THIS DOCUMENT ........................................................................................ 1-1
2
2.1
2.2
2.2.1
2.2.2
2.2.3
2.3
2.3.1
2.3.2
2.3.3
PREPARATION OF EXAM QUESTIONS AND PRACTICAL EXAM CASES .............. 2-2
General guidelines ..................................................................................................... 2-2
Multiple-choice test .................................................................................................... 2-4
The stem .................................................................................................................... 2-4
The options ................................................................................................................ 2-5
Evaluation key ............................................................................................................ 2-6
Calculation problems and cases................................................................................. 2-7
Calculation problems .................................................................................................. 2-7
Cases......................................................................................................................... 2-8
Evaluation of calculation problems and cases ............................................................ 2-8
3
3.1
3.2
3.3
EXAM QUESTION DATABASE ............................................................................... 3-10
Document control ..................................................................................................... 3-11
Instructions for confidentiality control........................................................................ 3-12
Revision and development of exam questions ......................................................... 3-12
4
4.1
4.2
4.3
4.4
TEMPLATES ............................................................................................................ 4-14
Formats .................................................................................................................... 4-14
Exam templates ....................................................................................................... 4-14
Evaluation key .......................................................................................................... 4-15
Question database template..................................................................................... 4-15
5
THE EXAM............................................................................................................... 5-16
6
REFERENCES .......................................................................................................... 6-1
APPENDICES
A
B
C
D
E
F
G
H
I
116103806
REQUIREMENTS FOR EXAMINATION TOPICS
EXAMPLE OF EXAM TEMPLATE FOR GENERAL COURSE
EXAMPLE OF EXAM TEMPLATE FOR SPECIALIST COURSE
RANDOM NUMBER TABLE
EXAMPLE OF EVALUATION KEY TEMPLATE FOR GENERAL COURSE
EXAMPLE OF EVALUATION KEY TEMPLATE FOR SPECIALIST COURSE
EXAMPLE OF QUESTION DATABASE TEMPLATE FOR MULTIPLE-CHOICE
QUESTIONS
EXAMPLE OF QUESTION DATABASE TEMPLATE FOR CALCULATION
PROBLEMS
EXAMPLE OF QUESTION DATABASE TEMPLATE FOR CASES
i
DHI
1
AIM OF THIS DOCUMENT
This document is prepared as part of the Sampler education pilot project, which
supports a certification scheme for environmental samplers in the Nordic-Baltic region
(Nordtest sampler certification scheme) and which comprises teaching and examination
material. The direct end users of the project results are the course units in those
countries that are setting up vocational training courses for the certification scheme.
However, the project focuses on the development of an educational form suitable to
provide education and documentation of competencies for a target group with different
basic education and language skills and aiming at certification of their competences.
The aim of this document is thus to develop the methods, templates and formats that can
be employed for the development of examination material:
-
To be used as part of the certification scheme for environmental samplers, as
well as other education systems
-
To be used irrespective of basic education and competences of the target group
-
That can easily be translated and thus used in trans-national education and
regardless of the type of training course.
In the document there is distinguished between methods and recommendations for the
development of examination material and specific methods and information regarding
the development of exam questions, practical problems etc. to support the
implementation of the Nordtest sampler certification scheme. To make this distinction
more clear, the reader will find highlighted sections in the text.
NORDTEST SAMPLER CERTIFICATION SCHEME.
If a section looks like this it illustrates
how certain things are done for the Sampler education pilot project. They can be used as an
example of how to apply the methods and templates etc.
With regards to examination material the following points will be addressed in this
document:
116103806
-
How to prepare exam questions and practical exam cases
-
Development of a questions database, including instructions for document
control and confidentiality
-
Development of exam templates and evaluation key
1-1
DHI
2
PREPARATION OF EXAM QUESTIONS AND PRACTICAL EXAM
CASES
The following section will give some guidelines for the preparation of exam questions.
In order to provide suitable exam questions for a target group with different level of
basic education and language skills, a variation of exam question types should be used
that require different level of knowledge, comprehension and application and give all
examinees an equal chance of passing regardless of their academic background.
Examples of exam question types are multiple-choice questions, true-false-questions,
matching questions, essay type questions, calculation problems or cases.
2.1
General guidelines
In order to assess actual course activity Kehoe (1995) suggests to prepare exam
questions right after a class has been given. Furthermore it is a good idea to construct
exam questions similar to control questions that are used in teaching or in teaching
materials, e.g. exercise sheets or textbooks (Ministry of Refugee, Immigration and
Integration Affairs, 2006). This way you can ensure, that the examinees are familiar
with the type of question that they will meet in the exam, and that mistakes in answering
the test questions are not a result of a lack of familiarity with the type of exams
questions and the structure of the exam.
To ensure that exam questions have the same level of difficulty, e.g. when different
people construct the questions or when questions are updated, it should be considered to
develop so-called item templates (an item is a test question). However, this might only
be appropriate for certain education and training systems. The following example
(Figure 1) is taken from a handbook about “Constructing written test questions for the
basic and clinical sciences” (Case & Swanson, 2001).
116103806
2-2
DHI
Figure 1
Example of an item template that is used to write test questions for the basic and clinical
sciences (Case & Swanson, 2001)
Another aspect to be considered is the length of the test and the time allotted to each
question. This is important to ensure that all examinees and regardless of their level of
reading have sufficient time to think through and answer the questions. Some general
time estimates for different types of exam questions can be found in Table 1.
Table 1
General guidelines regarding time requirements for high school test takers (CIDDE, 2007)
Task
True/False items
Multiple-choice (factual)
Multiple-choice (complex)
Matching (5 stems/6 choices)
Short-answer
Multiple-choice (with calculations)
Word problems (simple math)
Short Essays
Data analysis/graphing
Extended essays
116103806
2-3
Approximate time per item
20-30 seconds
40-60 seconds
70-90 seconds
2-4 minutes
2-4 minutes
2-5 minutes
5-10 minutes
15-20 minutes
15-25 minutes
35-50 minutes
DHI
NORDTEST SAMPLER CERTIFICATION SCHEME.
Exams include a combination of
multiple-choice questions (with 3 answer options) and calculation problems or cases,
respectively. Therefore the following sections describe hoe to develop these 3 types of exam
questions.
Calculation problems are presentations of information and data that shall be used to do
calculations, to do planning or to prepare reports or forms. Cases include the request to give a
general description of the sampling setting and specific questions, where a short answer is
required. General sampling courses are assessed by multiple-choice questions and calculation
problems, whereas specialist sampling courses use multiple-choice questions and cases.
Appendix A shows the requirements for examination topics that are based on training course
requirements. Exam questions shall be developed based on these requirements and shall be
constructed like the control questions given at the end of each chapter in the textbooks.
Appendix B and C show the exam templates. As can be seen, the examinee has 3.5 hours to
complete the exam – divided into 1.5 hours to answer 30 multiple-choice questions and 2 hours
to solve 3 cases and calculations problems, respectively. This should be ample time, even for
examinees with a lower reading level.
2.2
Multiple-choice test
A number of sources are available that offer help and guidelines in multiple-choice test
writing (e.g., Wallace, 2007; Frary, 1995; Kehoe, 1995; Burton et al., 1991, Haladyna
and Downing, 1989, VTECS, 2007, DIIA, 2007). A multiple-choice question consists of
two parts:
-
The stem: The stem is the actual question or an incomplete statement at the
beginning of a multiple-choice question.
-
The options: The options are the alternative answers that consist of
o the correct option and
o the so-called distractors (the incorrect options).
2.2.1
The stem
You start by writing the stem for each multiple-choice question. Make sure to select one
specific point or topic to be tested before writing the stem. Once you have decided what
you want to test, you can phrase the stem as an incomplete statement or a direct
question. Depending on how you construct the stem you can test factual knowledge,
comprehension, analysis or application.
Make sure that the wording is such, that an examinee understands what is being asked
and knows the correct answer without looking at the options. This includes placing as
much information as possible in the stem to clearly define the question and to keep the
options short and easy to read (Figure 2 and Figure 3). At the same time you should
avoid giving irrelevant information in the stem.
116103806
2-4
DHI
NORDTEST SAMPLER CERTIFICATION SCHEME.
Refer to the “Aim of learning” that is
given for each chapter in the textbooks, and the examination requirements to define what you
want to test with each question. Each multiple-choice question has 3 answer options, including
the correct answer. Each correctly answered question is rewarded with 1 point. Use the questions
database template for multiple-choice questions to store your questions.
2.2.2
The options
Once you have written the correct answer to the questions, make sure to write the
distractors in a way that they are comparable in length, complexity and grammatical
form to the correct answer. Differences in length, complexity and grammatical
inconsistencies can be used as a clue to what the correct answer is (Figure 4). The
distractors should be plausible answers as well.
Figure 2
116103806
Example of how to exclude irrelevant information from the stem to keep it short and clear
(Burton et al. 1991)
2-5
DHI
2.2.3
116103806
Figure 3
Example on how to include as much information as possible in the stem to keep the options
short and clear (Burton et al. 1991)
Figure 4
How to avoid giving clues to the correct answer through grammatical inconsistencies – In
the poor examples option A can be spotted to be correct, since the other alternatives start
with a consonant and the correct article to adjective is “an”. (Burton et al. 1991)
Evaluation key
Once you have written the stem and the options, you have to develop the evaluation key
for each question. In general you should try to place the options as randomly as
possible. This can be done by making sure, that the position of the correct answer
appears about the same number of times in a section of questions as the position of the
distractors does.
2-6
DHI
There are some general suggestions regarding the placement of the correct answer
amongst the distractors. One possibility is to arrange the options in a logical order, e.g.
numerical, alphabetical, sequential (see Figure 5). Another option is to flip two or three
coins at a time, and to use each possible head-tail combination for each location of the
correct answer (Kehoe, 1995). Yet another possibility is to use random number tables,
which you can create in Excel. You can see an example in appendix D. Once you have
decided on the position of your options, put the position of the correct answer in the
evaluation key for this set of questions.
Figure 5
Arranging options in logical order (Burton et al. 1991)
2.3
Calculation problems and cases
2.3.1
Calculation problems
Calculation problems can be compared to a short-answer or short essay test. Before you
write the question, you should be sure of what you want to test, e.g. comprehension,
analysis, evaluation, and make sure to focus on one specific learning objective or topic.
Questions should be constructed in such a way that the tasks are clearly defined for the
examinee (CIDDE, 2007). Construct your question in a way that individual parts of the
question are clearly separated and indicate how much time the examinee is expected to
spend on each part or how the individual parts are weighted (Ory, 2007). Make sure to
use the appropriate terms in your question for the response that you are looking for, e.g.
compare, describe, define, illustrate, interpret, etc. (Landsberger, 2007). Once you have
developed the questions, write the model answer. When this is done, put this
information into the evaluation key for this set of questions.
116103806
2-7
DHI
Figure 6
Example of how to separate parts of the questions and to indicate their value (Ory, 2007)
NORDTEST SAMPLER CERTIFICATION SCHEME.
Refer to the “Aim of learning” that is
given for each chapter in the textbooks, and the examination requirements to define what you
want to test with each calculation problem. Write clear questions and give clear directions (see
Figure 6). Use the questions database template for calculation problems to store your calculation
problems.
2.3.2
Cases
It is important that you make sure to create a realistic situation when constructing cases,
e.g. by using relevant figures, illustrations etc., and that you provide the examinee with
all necessary information in the scenario description to be able to answer your
questions.
Provide clear questions or clear directions. It might help to include the main elements
that you wish to be covered. This could be by constructing a separate question for each
element or by listing keywords, ideas, concepts etc. that should be addressed in the
examinees response. Indicate whether you are looking for a single-word answer or a
short response. Figure 6 gives a short example of how the description part of the case
can be divided from the specific questions to the case.
NORDTEST SAMPLER CERTIFICATION SCHEME.
Refer to the “Aim of learning” that is
given for each chapter in the textbooks, and the examination requirements to define what you
want to test with each case. Write clear questions and give clear directions (see Figure 6). Use
the question database template for cases to store your cases.
2.3.3
Evaluation of calculation problems and cases
NORDTEST SAMPLER CERTIFICATION SCHEME.
Write a model answer and list the
necessary elements, e.g. keywords that should appear in the answer, general ideas, concepts,
that should be discussed etc. Assign points to the individual elements of the answer. Where
calculations have to be done, ask the student to write down
-
116103806
the formula to be used,
2-8
DHI
-
to show the method of calculation with the formula and data provided, and
-
to give the calculation result.
These three things shall be necessary elements when doing calculations. Indicate whether your
questions require a single-word answer or a short response. Figure 7 shows how essay-type
questions can be evaluated and gives an example for necessary elements.
Each calculation problem and case has a total possible score of 10 points. Only correct answers
shall be assigned a score. Irrelevant responses or unreadable answers shall not be assigned a
score.
Figure 7
116103806
Suggestion for evaluation of essay questions (Ory, 2007)
2-9
DHI
3
EXAM QUESTION DATABASE
There are a lot of different item banking systems (an item is a test question) available on
the market. Most of these software systems allow for storage, retrieval, and maintenance
of test items, development and printing of examinations, calculation, storage, and
retrieval of item statistics, as well as calculation of test scores. However, a question
database can also be developed in a less technical way, e.g. a document containing a list
of questions and fields in a spreadsheet or word processor.
A question database is a good possibility to avoid the time consuming process of
creating new exam questions every time an exam is to be held. Once an exam question
is created it can be added to the question database. The database can be structured to
contain different fields, e.g. type of question, level of difficulty or comprehension,
course content, aim of learning. This will help to find suitable questions when an exam
is compiled and questions are drawn from the database.
NORDTEST SAMPLER CERTIFICATION SCHEME.
The database for this education
system contains several folders and Microsoft Office documents, and will be stored on a CDROM. The database structure can be seen in Figure 8. The three main folders are templates,
questions, and exams:
116103806
-
The template folder contains the exam question templates, evaluation key templates and
question database template.
-
The questions folder is subdivided into the 5 sampling topics and each subfolder is again
subdivided into folders that contain the different types of question database (multiplechoice, cases, calculation problems). The questions folder is the actual exam question
database, as questions are stored here.
-
The exam folder contains subfolders for each sampling course. The database user shall
create additional folders for the individual exams, where the exam and evaluation key
shall be saved (see the example in Figure 8).
-
Templates and questions can be discontinued, e.g. when templates or questions need to
be revised. The discontinued items shall be saved in the designated folders.
3-10
DHI
Figure 8
3.1
Structure of exam question database
Document control
It is important to establish a system for document control. This ensures that templates,
questions and other documents are stored in an organised way, where revisions and
other changes can be tracked.
NORDTEST SAMPLER CERTIFICATION SCHEME.
Document
control
comprises
numbering of template versions, document titles for exams and evaluation keys, and questions
databases:
-
116103806
The template title is followed by a two digit number: X_Y (e.g.
“database_calculation_problems_1_0.doc”). Major changes to the template will be
reflected by a shift to a higher first digit (X), whereas minor changes will be reflected in a
3-11
DHI
shift to a higher second digit only (Y). Discontinued versions shall be saved in the
assigned folder in the database.
3.2
-
Exams titles comprise the course name or number, initials of the examiner or course
responsible, the date of the exam (e.g. soilsampling_aob_30102007.doc). The title of the
corresponding evaluation key contains the same information (e.g.
evalkey_soilsampling_aob_30102007.doc).
-
The title of the question database contains the type of question, the course name or
number
and
the
date
for
the
last
changes
(e.g.
database_cases_soilsampling_30102007.doc). Discontinued questions shall be removed
from the question database. The questions shall be saved in a new document using the
question database template and with a title containing type of question, the course name
or
number
and
the
date
for
the
last
changes
(e.g.
discontinued_cases_soilsampling_30102007.doc).
Instructions for confidentiality control
The course and exam unit has to ensure that exam questions and evaluation keys are
kept confidential. This could be done by assigning a responsible person at the course
and exam unit, who is in charge of preparing exams and evaluation keys and handing
them out to course instructors on request. Furthermore, the database or database
documents should be pass word protected to ensure confidentiality.
3.3
Revision and development of exam questions
Depending on how many questions there are required per exam and how many times it
is permissible to reuse the same question, there will be a need to revise the stored
questions or to develop new questions. Other reasons for revision and development of
exam questions are the level of difficulty being inappropriate or adaptation of teaching
material, e.g. to technical progress.
In order to ensure that the level of difficulty in the exam is appropriate, and that the
assessment reflects the course activities and content, exam results can be collected and
statistically evaluated (Milanovic, 2002). This is especially important when different
people are involved in exam question writing, or when course activities, teaching
material and exam questions are revised.
NORDTEST SAMPLER CERTIFICATION SCHEME.
For this education system the following
number of multiple-choice questions, cases and calculation problems shall be developed:
-
General sampling course: 120 multiple-choice questions and 12 calculation problems
-
Specialist sampling course: 120 multiple-choice questions and 12 cases
The examination topics shall be covered as far as possible with this number of questions.
Furthermore, a question shall not be used more than three times. The question database
template contains a field to keep track of how many times a particular question has been used in
116103806
3-12
DHI
an exam. Every time a question is used, the title of the corresponding exam is inserted in the
designated column in the question database.
116103806
3-13
DHI
4
TEMPLATES
Templates should be developed to ensure uniformity and coherence in the course and
exam system.
NORDTEST SAMPLER CERTIFICATION SCHEME.
prepared for the development of examination material:
4.1
Seven different templates have been
-
Templates for general sampling course and specialist sampling course exams,
respectively
-
Evaluation key template for general sampling course and specialist sampling course
exams, respectively
-
Question database template for general sampling course and specialist sampling course
exams, respectively.
Formats
In order to use a uniform layout with regard to headings, figures, tables, bulletins and
other formatting elements when preparing exam material, certain rules have to be laid
down and styles have to be defined.
NORDTEST SAMPLER CERTIFICATION SCHEME.
4.2
Refer to:

“Jettes textbook outline” for information on which formats are used in this textbook
series and shall be applied to the examination materials as well.

“Jettes guideline document” in order to learn, how to install the templates on your
computer and to use them.
Exam templates
Templates for exam questions should contain some general information, such as the
name of the course, information about the exam and course unit, document title, which
will help with the document control. Furthermore, the front page of an exam paper
might be used to provide the examinee with some useful information about the exam,
such as duration, how the answers are assessed, the final score etc. The main part of the
exam template is the questions part, which can be arranged in different ways, depending
on the type of exam questions used (e.g. as a list of questions, a table with question and
answer field).
NORDTEST SAMPLER CERTIFICATION SCHEME.
Refer to appendix B and C to see
examples of the exam templates used for this scheme. Note that the templates contain some
general fields (highlighted by italic writing, e.g. placeholders for a logo or document title) and
some information specific for this scheme (e.g. specific topics for the questions).
116103806
4-14
DHI
The exam questions are copied from the question database and saved in the exam question
template. Every time a question is used, the title of the corresponding exam is inserted in the
designated column in the question database.
4.3
Evaluation key
During preparation of the exam questions, the evaluation key is developed. Developing
the evaluation key for multiple-choice questions or simple calculations is rather straight
forward. Where a more detailed response is requested (e.g. essay type questions) it is
important to describe a model answer. By giving a model answer and listing the
necessary elements (e.g. certain key words, concepts, ideas, that should be used in the
answer), both the test developer and any other examiner can objectively asses the exam
results.
The evaluation key should contain a marking scheme, so that it is clear how each
question and the individual parts of each question are weighted in relation to the overall
mark for the exam, and how many percent or points have to be achieved in order to
pass.
NORDTEST SAMPLER CERTIFICATION SCHEME.
Refer to appendix E and F to see
examples of evaluation key templates. When an exam is prepared the evaluation key to each
question is copied from the question database and saved in the evaluation key template. See the
rules for document titles in section 3.1.
For this scheme the multiple-choice questions part of each exam is weighted 50 % of the total
mark, and the calculations problems in the general course exam and the cases in the specialist
course exam are weighted 50 %, respectively. The total mark of each exam is adjusted to 60
points (100%), and a minimum of 42 points (70%) has to be reached to pass the exam:
-
General course
o 30 multiple-choice questions, 1 point per correct answer (total score: 30 points,
50 %)
o 3 calculation problems, 10 points per correctly solved problem (total score 30
points, 50 %)
-
Specialist course
o 30 multiple-choice questions, 1 point per correct answer (total score: 30 points,
50 %)
o 3 cases, 10 points per case (total score 30 points, 50 %)
4.4
Question database template
NORDTEST SAMPLER CERTIFICATION SCHEME.
Refer to appendix G-I to see examples
of the questions database template. Once questions and answers have been developed they are
stored in the respective questions database and shall be copied from the database to prepare
exams and evaluation keys.
116103806
4-15
DHI
5
THE EXAM
There are a number of things that should be considered when planning an exam and that
should be in place before a course and exam unit conducts and exam:
116103806
-
Dates and duration of the exam: Should the exam only be offered when a course
has been held? How many hours lasts the exam and is it only theoretical?
-
Rules for admittance to the exam: Can someone take the exam without attending
the course? Or should the examinee have attended at least X % of the course in
order to be admitted to the exam?
-
Rules for retaking the exam: How many times can an examinee retake an exam,
before he/she might be required to retake the course? Is there a fee for retaking
the exam? What are valid reasons (e.g. illness) for being absent on the
examination day?
-
Aids: Are examinees allowed to use notes, textbooks, calculators etc. during the
exam? Should the course unit provide extra sheets of paper for answering
questions?
-
Exam supervision: Does the course instructor supervise the exam or is this done
by external supervisors? How many supervisors should there be for a certain
number of examinees?
-
Exam rooms and exclusion from the exam: How large should the rooms be?
How close can the examinees sit? What happens if there is a reason to believe,
that an examinee has used forbidden aids?
-
People with disabilities: Are there people with special needs among the course
participants, who might have special requirements to be able to attend the exam?
5-16
DHI
6
REFERENCES
Burton S J, Sudweeks R R, Merrill P F, Wood B (1991) How to prepare better multiplechoice
test
items:
guidelines
for
university faculty.
Available
at
http://testing.byu.edu/info/handbooks/betteritems.pdf , accessed 16/08/2007
Case S M, Swanson D B (2001) Constructing written test questions for the basic and
clinical sciences, National board of medical examiners, Philadelphia, available at:
http://www.nbme.org/PDF/ItemWriting_2003/2003IWGwhole.pdf
,
accessed
24/10/2007
Center for instructional development & distance education (CIDDE) (2007) Writing
essay
items,
available
at:
http://www.pitt.edu/~ciddeweb/fds/lrn_assess_essay_item.htm, accessed 27/08/2007
Division of instructional innovation and assessment (DIIA) (2007) Writing multiplechoice questions. Instructional assessment resources (IAR), available at
http://www.utexas.edu/academic/diia/assessment/iar/students/plan/method/examsmchoice-write.php, accessed 16/08/2007
Frary R B (1995) More multiple-choice item writing do’s and don’ts. Practical
Assessment,
Research
&
Evaluation
4(11),
available
at
http://pareonline.net/getvn.asp?v=4&n=11, accessed 16/08/2007
Haladyna T M, and Downing S M (1989) A taxonomy of multiplechoice item-writing
rules. Applied Measurement in Education, pp. 37-50, available at
http://testing.byu.edu/info/handbooks/MultipleChoice%20Item%20Writing%20Guidelines%20%20Haladyna%20and%20Downing.pdf, .
Kehoe J (1995) Writing multiple-choice test items. Practical Assessment, research &
Evaluation 4(9), available at http://pareonline.net/getvn.asp?v=4&n=9, accessed
16/08/2007
Landsberger J (2007) Terms or directives for essay, reports, and answering questions,
available at: http://www.studygs.net/essayterms.htm, accessed 27/08/2007
Milanovic M (2002) Common European framework of reference for languages:
learning, teaching, assessment; Language examining and test development, language
policy
division,
Strasbourg,
available
at:
http://www.coe.int/T/DG4/Portfolio/documents/Guide%20October%202002%20revised
%20version1.doc
Ministry of Refugee, Immigration and Integration Affairs (2006) Vejledning om prøve i
dansk 3, Ministeriet for flygtninge, indvandrere, og integration, available at:
http://www.nyidanmark.dk/NR/rdonlyres/B4C00E1A-507B-43DB-83DF9BD04C2726C2/0/pd3_vejl_okt_07.pdf
Ory J C (2007) Improving your test questions – EES Memorandum nr. 27, revised and
distributed by evaluation and examination service, the University of Iowa, available at:
116103806
6-1
DHI
http://www.uiowa.edu/~examserv/Level_2/resources/Technical%20Bulletins/Tech%20
Bulletin%2027.pdf, accessed 27/08/2007
VTECS (2007) Multiple choice item writing guidelines, available at http://www.vtecs.org/ccifiles/MULTIPLECHOICE%20ITEM%20WRITING%20GUIDELINES.pdf, accessed 16/08/2007
Wallace L (2007) Guidelines for writing multiple-choice test items, available at
http://ltc.umanitoba.ca/multiple_choice/GuidelinesWritingMCItems.pdf,
accessed
16/08/2007
116103806
6-2
DHI
APPENDICES
116103806
DHI
APPENDIX
A
Requirements for examination topics
116103806
DHI
General courses
General courses are evaluated in exams with a combination of
-
multiple-choice questions (minimum 30 questions, with three optional answers
per question) and
-
3 types of “calculation problems”
Multiple-choice questions shall cover:
-
Theory of sampling
-
Simple sampling statistics
-
Quality control procedures
-
Requirements for certification of samplers
-
Theory of establishment of a quality system for sampling
“Calculation problems” shall cover 3 of the following 5 topics:
116103806
-
Preparation of sampling report from presented data
-
Preparation of chain of custody report from presented data
-
Selection of proper quality control (QC) procedures for a presented purpose
-
Calculation and evaluation of uncertainty estimates for a given case
-
Preparation of elements in a sampling quality system for a presented purpose
DHI
Specialist courses
Specialist courses are evaluated in exams with a combination of
116103806
-
multiple-choice questions (minimum 30 questions, with three optional answers
per question) and
-
cases (3 cases, each with 5 specific answers asked for)
DHI
Specialist course – waste
Main topics
There shall be a minimum of five multiple-choice questions from each of the
following four main topics:
-
Background on sampling of waste from moving streams, stopped belt, stationary
lot
-
Background on sampling of waste for purpose of basic characterization,
compliance testing, on-site verification
-
Background on sampling of waste for chemical investigations, leaching testing,
physical investigations, biological investigations
-
Information of waste characteristics (e.g., waste heterogeneity and
biodegradability) and special features in waste sampling (e.g., production
restrictions in waste sampling, resources available)
Specialist course content
The remaining (minimum) 10 multiple-choice questions are selected among the
specialist course content. All specialist courses shall include the following topics
adapted to the matrix concerned in the specialist course:
116103806
-
Best practice on hygiene and safety issues
-
Purpose of sampling
-
Understanding a sampling plan
-
Principles of sampling techniques
-
Limitations of sampling equipment
-
Preparing for sampling, “mobilization”
-
Equipment control, calibration and maintenance
-
Practical sampling and practical sub-sampling
-
Principles for selection of sampling containers, samples preservation, storage,
and transport
-
Training in quality control procedures in sampling (including checking
equipment status, evaluation of risk for sample contamination, duplicate
samples, bland samples, samples from reference stations, sampler
intercomparison)
-
Information on sampling uncertainty
DHI
-
Documentation of sampling (covering sample identification and labelling, field
records, chain of custody reports, writing sampling reports)
-
The chain from sampling plan over sampling and transport to analysis and
results
Cases
All specialist courses exams shall include two cases from the following types:
-
Selection of appropriate hygiene and safety measures
-
Selection of sampling method and equipment
-
Selection of field- sub-sampling methods
-
Selection of sample containers and handling (sample containers, preservation,
storage and transport)
-
Calculation and evaluation of quality control data
-
Documentation of sampling procedure (sample report with uncertainty and chain
of custody report)
The solid waste course exam shall contain one case from the following types:
116103806
-
Selection of sampling positions in sampling from moving belt, stopped belt and
stationary lot
-
Handling of samples for chemical investigation, leaching test, physical
investigations or biological investigations
-
Description of special features in waste producing process or waste storage
conditions and identification of critical conditions in waste sampling
DHI
Specialist course – soil
Main topics
There shall be a minimum of five multiple-choice questions from each of the
following four main topics:
-
Background on geological characterization of soils (covering topsoils and
geological sediments)
-
Background on sampling of soils for chemical analysis of volatiles and nonvolatiles, biological analysis, analysis of physical parameters)
-
Sampling techniques for contaminated sites and soil piles
-
Equipment handling for hand drilling in soil piles, in situ drilling with hollow
stem or solid flight augers, and dedicated samplers for sampling in open
boreholes or pits
Specialist course content
The remaining (minimum) 10 multiple-choice questions are selected among the
specialist course content. All specialist courses shall include the following topics
adapted to the matrix concerned in the specialist course:
116103806
-
Best practice on hygiene and safety issues
-
Purpose of sampling
-
Understanding a sampling plan
-
Principles of sampling techniques
-
Limitations of sampling equipment
-
Preparing for sampling, “mobilization”
-
Equipment control, calibration and maintenance
-
Practical sampling and practical sub-sampling
-
Principles for selection of sampling containers, samples preservation, storage,
and transport
-
Training in quality control procedures in sampling (including checking
equipment status, evaluation of risk for sample contamination, duplicate
samples, bland samples, samples from reference stations, sampler
intercomparison)
-
Information on sampling uncertainty
-
Documentation of sampling (covering sample identification and labelling, field
records, chain of custody reports, writing sampling reports)
DHI
-
The chain from sampling plan over sampling and transport to analysis and
results
Cases
All specialist courses exams shall include two cases from one of the following types:
-
Selection of appropriate hygiene and safety measures
-
Selection of sampling method and equipment
-
Selection of field- sub-sampling methods
-
Selection of sample containers and handling (sample containers, preservation,
storage and transport)
-
Calculation and evaluation of quality control data
-
Documentation of sampling procedure (sample report with uncertainty and chain
of custody report)
The soil course exam shall contain one case from the following types:
116103806
-
Selection of sampling positions (including, positions within site, depths during
drilling, positions and depths in pile)
-
Selection of samples for analysis (including, evaluation of field measurements,
sample appearance, soil and sediment properties)
-
Elaboration of quality control procedures for soil sampling (including, check of
equipment status, evaluation of risk for sample contamination, QC types)
DHI
Specialist course – groundwater
Main topics
There shall be a minimum of five multiple-choice questions from each of the
following four main topics:
-
Background on hydrogeology and hydrochemical properties of groundwater
-
Background on sampling of groundwater for chemical analysis of volatiles and
non-volatiles, biological analysis and analysis of physical parameters
-
Sampling techniques for investigation wells and drinking water supply wells
-
Equipment handling for sampling with positive displacement pumps for volatiles
and non-volatiles, sampling with suction pumps for non-volatiles, and sampling
with bailers for non-volatiles
Specialist course content
The remaining (minimum) 10 multiple-choice questions are selected among the
specialist course content. All specialist courses shall include the following topics
adapted to the matrix concerned in the specialist course:
116103806
-
Best practice on hygiene and safety issues
-
Purpose of sampling
-
Understanding a sampling plan
-
Principles of sampling techniques
-
Limitations of sampling equipment
-
Preparing for sampling, “mobilization”
-
Equipment control, calibration and maintenance
-
Practical sampling and practical sub-sampling
-
Principles for selection of sampling containers, samples preservation, storage,
and transport
-
Training in quality control procedures in sampling (including checking
equipment status, evaluation of risk for sample contamination, duplicate
samples, bland samples, samples from reference stations, sampler
intercomparison)
-
Information on sampling uncertainty
-
Documentation of sampling (covering sample identification and labelling, field
records, chain of custody reports, writing sampling reports)
DHI
-
The chain from sampling plan over sampling and transport to analysis and
results
Cases
All specialist courses exams shall include two cases from one of the following types:
-
Selection of appropriate hygiene and safety measures
-
Selection of sampling method and equipment
-
Selection of field- sub-sampling methods
-
Selection of sample containers and handling (sample containers, preservation,
storage and transport)
-
Calculation and evaluation of quality control data
-
Documentation of sampling procedure (sample report with uncertainty and chain
of custody report)
The groundwater course exam shall contain one case from the following types:
116103806
-
Selection in response to hydrogeology, groundwater chemistry and
contamination and purpose of investigation of pump position, purge period, well
purge indicator parameters, and stop criteria
-
Handling of samples for parameter types (volatile, adsorbable, redox sensitive)
-
Selection of techniques for sampling from investigation wells and drinking
water supply wells
DHI
Specialist course – waste water and sludge
Main topics
There shall be a minimum of five multiple-choice questions from each of the
following four main topics:
-
Information on the background for water pollution and sludge quality control
-
Theoretical background for proper sampling of wastewater and sludge
(including, physical/chemical and chemical investigations, microbiological
investigations)
-
Principles for sampling of sludge (including, criteria for sampling of sludge and
selection of sampling regime)
-
Principles for sampling wastewater (use of flow- and time-dependent equipment
and demonstration of field measurement equipment)
Specialist course content
The remaining (minimum) 10 multiple-choice questions are selected among the
specialist course content. All specialist courses shall include the following topics
adapted to the matrix concerned in the specialist course:
116103806
-
Best practice on hygiene and safety issues
-
Purpose of sampling
-
Understanding a sampling plan
-
Principles of sampling techniques
-
Limitations of sampling equipment
-
Preparing for sampling, “mobilization”
-
Equipment control, calibration and maintenance
-
Practical sampling and practical sub-sampling
-
Principles for selection of sampling containers, samples preservation, storage,
and transport
-
Training in quality control procedures in sampling (including checking
equipment status, evaluation of risk for sample contamination, duplicate
samples, bland samples, samples from reference stations, sampler
intercomparison)
-
Information on sampling uncertainty
-
Documentation of sampling (covering sample identification and labelling, field
records, chain of custody reports, writing sampling reports)
DHI
-
The chain from sampling plan over sampling and transport to analysis and
results
Cases
All specialist courses exams shall include two cases from one of the following types:
-
Selection of appropriate hygiene and safety measures
-
Selection of sampling method and equipment
-
Selection of field- sub-sampling methods
-
Selection of sample containers and handling (sample containers, preservation,
storage and transport)
-
Calculation and evaluation of quality control data
-
Documentation of sampling procedure (sample report with uncertainty and chain
of custody report)
The waste water and sludge course exam shall contain one case from the following
types:
116103806
-
Safe procedures for working in sewers and wastewater plants
-
Description of proper installation, calibration and sampling in relation to sewer
geometry and sampling purposes for commonly used equipment (time
proportional samplers, flow proportional samplers, random sampling)
-
Description of practical sewage sludge sampling parameters (depth,
subsampling, preparation of composite samples) in response to storage type,
geometry and duration
DHI
APPENDIX
B
Example of exam template for general course
116103806
DHI
To be added as a picture to inlude header and footer.
116103806
DHI
APPENDIX
C
Example of exam template for specialist course
116103806
DHI
To be added as a picture to inlude header and footer.
116103806
A-1
DHI
APPENDIX
D
Random number table
116103806
A-2
DHI
RAND( )- Function in Excel
This function returns an evenly distributed random number greater than or equal to 0
and less than 1. A new random number is returned every time the worksheet is
calculated.
To generate a random real number between a and b, use:
RAND()*(b-a)+a
Table of random numbers produced in Excel.
Function used: =ROUND((RAND()*(3-1)+1);0)
2
2
1
1
1
1
2
3
1
2
1
2
3
1
2
1
3
1
1
2
2
2
2
2
3
2
1
2
116103806
3
2
1
2
2
2
3
1
3
2
1
2
2
3
3
3
2
2
1
3
3
2
2
2
2
1
2
3
2
3
2
3
2
2
2
2
2
1
2
2
2
2
2
2
2
1
2
2
2
3
3
3
2
2
1
2
2
1
3
1
1
2
3
3
2
2
1
3
1
2
2
2
2
3
3
3
3
2
2
2
2
1
2
2
2
2
1
2
2
3
2
2
3
1
2
2
2
2
3
2
2
3
2
1
1
2
1
1
2
2
2
3
2
1
2
2
2
1
3
1
2
3
2
2
2
2
1
2
1
3
2
2
3
2
1
2
2
2
2
3
2
1
2
3
2
3
1
2
2
2
2
2
2
2
1
2
2
1
3
2
2
2
3
2
2
2
2
2
2
3
2
3
3
1
2
2
3
2
3
1
2
2
2
3
1
3
2
2
2
2
1
2
3
3
2
3
2
1
1
3
2
1
2
2
1
2
2
2
3
3
2
3
3
2
1
3
2
2
3
2
2
2
2
2
1
3
3
2
1
1
1
2
3
1
2
2
3
2
3
1
2
3
2
3
2
1
1
2
1
2
2
2
2
1
2
2
1
2
1
2
1
1
2
3
3
2
1
3
3
2
2
3
2
2
1
2
2
2
2
3
1
1
2
1
2
1
2
3
1
2
1
2
2
3
3
2
2
3
2
3
2
1
2
2
2
1
1
3
2
3
2
3
2
2
2
2
1
1
2
2
2
1
2
2
3
3
3
2
3
3
3
2
1
2
3
3
3
2
2
2
2
2
1
1
1
2
3
2
2
2
1
1
2
1
2
1
2
3
3
3
3
2
2
2
1
2
3
3
2
1
2
3
3
2
1
1
2
1
3
2
1
3
2
2
2
2
1
2
3
2
1
3
3
2
3
3
1
2
2
2
2
2
2
1
3
2
2
3
3
2
1
3
1
3
2
1
2
2
2
3
2
1
2
2
2
2
1
1
1
2
2
3
1
2
3
2
1
3
1
2
3
2
1
3
2
1
2
2
2
2
3
3
2
2
2
2
2
2
2
2
3
2
2
3
2
3
2
1
2
2
3
1
2
2
3
2
2
2
3
3
1
2
1
2
1
2
3
2
2
2
2
3
1
2
3
2
3
3
1
1
2
3
2
2
3
2
2
2
2
1
1
1
2
1
3
2
2
2
2
2
2
2
1
3
3
3
1
2
2
2
2
2
1
1
3
2
2
1
3
2
3
2
2
1
2
1
2
2
2
2
2
2
2
3
1
1
3
1
2
2
2
3
2
1
3
1
1
2
1
1
2
3
2
2
3
2
2
2
2
2
2
1
1
3
2
2
2
3
2
2
2
2
3
2
1
3
1
2
2
3
2
2
2
2
3
2
2
2
1
2
1
1
1
1
3
3
2
2
1
1
1
1
2
3
1
3
3
2
3
3
2
3
2
2
2
3
2
2
3
2
2
2
1
2
3
1
3
2
1
2
3
3
3
2
3
2
2
3
2
3
1
3
1
3
3
2
2
2
2
2
2
1
2
2
3
2
3
2
2
2
1
3
2
2
1
2
3
1
3
3
1
1
2
2
2
2
2
2
2
3
1
3
2
2
2
1
2
2
1
3
2
2
2
1
3
2
3
1
3
2
2
1
3
2
1
2
1
2
DHI
2
3
3
3
3
3
1
2
3
2
3
1
1
2
1
2
1
2
2
2
2
2
3
2
3
1
3
1
3
3
1
2
3
1
1
2
2
2
1
3
2
2
2
2
2
1
3
2
2
1
2
2
2
3
3
3
1
1
3
3
3
2
3
2
2
3
3
2
2
1
1
2
1
3
3
1
2
3
1
1
2
2
1
2
APPENDIX
E
Example of evaluation key for general course
116103806
DHI
To be added as a picture to inlude header and footer.
116103806
DHI
APPENDIX
F
Example of evaluation key for specialist course
116103806
DHI
To be added as a picture to inlude header and footer.
116103806
A-1
DHI
APPENDIX
G
Example of question database template for multiple-choice
questions
116103806
A-2
DHI
116103806
A-1
DHI
APPENDIX
H
Example of question database template for calculation
problems
116103806
A-1
DHI
116103806
A-1
DHI
APPENDIX
I
Example of question database template for cases
116103806
A-1
DHI
116103806
A-1
DHI