BILC PROFESSIONAL SEMINAR
Stockholm , October 14 - 17, 2013
INNOVATIVE TEST DESIGNS AND FORMATS
Testing in the time of crisis
Lt.Col. F. Gratton
Francesco
Gratton
2013
Summary:
• Past situation (up to Sept 2013)
• New Course Of Actions adopted
• A do-it-all software
• Proposals
Francesco
Gratton
2013
Stanag Proficiency Test 1.0
RECEPTIVE SKILLS (Listening & Reading)
1: Multilevel test (level 1 to 4)
2: Multiple choice questions (60 for L & R)
3: No penalties for wrong answers
4: Duration: R 105’ / L 90’
5: Separate Sections (& levels)
6: # of correct answers multiplied for a coefficient
7: Potential use of “F” factor
Francesco
Gratton
2013
How STANAG levels
were awarded
(Stanag Proficiency Test 1.0)
RECEPTIVE SKILLS
# of correct answers
Multiplied by
fixed coefficient (1,66)
Francesco
Gratton
2013
LEVEL
LEVEL
LEVEL000
LEVEL
LEVEL
LEVEL111
LEVEL
LEVEL
LEVEL222
LEVEL
LEVEL
LEVEL333
LEVEL
LEVEL
LEVEL444
CORRECT
CORRECT
CORRECT
%
%
%
CORRECT
CORRECT
CORRECT
%
%
%
CORRECT
CORRECT
CORRECT
%
%
%
CORRECT
CORRECT
CORRECT
%
%
%
CORRECT
CORRECT
CORRECT
%
%
%
111
1,67
1,67
1,67
11
11
11
18,37
18,37
18,37
26
26
26
43,42
43,42
43,42
41
41
41
68,47
68,47
68,47
56
56
56
93,52
93,52
93,52
222
3,34
3,34
3,34
12
12
12
20,04
20,04
20,04
27
27
27
45,09
45,09
45,09
42
42
42
70,14
70,14
70,14
57
57
57
95,19
95,19
95,19
333
5,01
5,01
5,01
13
13
13
21,71
21,71
21,71
28
28
28
46,76
46,76
46,76
43
43
43
71,81
71,81
71,81
58
58
58
96,86
96,86
96,86
444
6,68
6,68
6,68
14
14
14
23,38
23,38
23,38
29
29
29
48,43
48,43
48,43
44
44
44
73,48
73,48
73,48
59
59
59
98,53
98,53
98,53
555
8,35
8,35
8,35
15
15
15
25,05
25,05
25,05
30
30
30
50,1
50,1
50,1
45
45
45
75,15
75,15
75,15
60
60
60
100,2
100,2
100,2
666
10,02
10,02
10,02
16
16
16
26,72
26,72
26,72
31
31
31
51,77
51,77
51,77
46
46
46
76,82
76,82
76,82
777
11,69
11,69
11,69
17
17
17
28,39
28,39
28,39
32
32
32
53,44
53,44
53,44
47
47
47
78,49
78,49
78,49
888
13,36
13,36
13,36
18
18
18
30,06
30,06
30,06
33
33
33
55,11
55,11
55,11
48
48
48
80,16
80,16
80,16
999
15,03
15,03
15,03
19
19
19
31,73
31,73
31,73
34
34
34
56,78
56,78
56,78
49
49
49
81,83
81,83
81,83
10
10
10
16,7
16,7
16,7
20
20
20
33,4
33,4
33,4
35
35
35
58,45
58,45
58,45
50
50
50
83,5
83,5
83,5
21
21
21
35,07
35,07
35,07
36
36
36
60,12
60,12
60,12
51
51
51
85,17
85,17
85,17
22
22
22
36,74
36,74
36,74
37
37
37
61,79
61,79
61,79
52
52
52
86,84
86,84
86,84
23
23
23
38,41
38,41
38,41
38
38
38
63,46
63,46
63,46
53
53
53
88,51
88,51
88,51
24
24
24
40,08
40,08
40,08
39
39
39
65,13
65,13
65,13
54
54
54
90,18
90,18
90,18
25
25
25
41,75
41,75
41,75
40
40
40
66,8
66,8
66,8
55
55
55
91,85
91,85
91,85
Francesco
Gratton
2013
Proficiency Test 1.0 PRODUCTIVE SKILLS
SPEAKING (holistically assessed )
1: Functional language assessed in a global manner
2: Structured interview
3: Tailored to the candidate
4: Checks & probes
5: 1 to 2 role plays
Francesco
Gratton
2013
JFLT 1.0
PRODUCTIVE SKILLS
WRITING (holistically assessed )
Three tasks (one for each level)
Francesco
Gratton
2013
Summary:
• Past situation
• New COAs adopted
• A do-it-all software
• Proposals
Francesco
Gratton
2013
New COAs adopted
•Specifications
•Cut-off score
•Joint Database
Francesco
Gratton
2013
Francesco
Gratton
2013
• PURPOSE
• LEVELS OF LINGUISTIC KNOWLEDGE
• TEST CONTENT
• TEST FORMAT
• VALIDATION PROCEDURES
• ADMINISTRATION PROCEDURES
Francesco
Gratton
2013
Stanag Proficiency
Test 1.0
TWAS
HIGH & Reading)
RECEPTIVE SKILLS (Listening
TIME!!!
1: Multilevel test (level 1 to 4)
2: Multiple choice questions (60 for L & R)
3: No penalties for wrong answers
4: Duration: R 105’ / L 90’
5: Separate Sections (& levels)
6: # of correct answers multiplied by a coefficient
7: Potential use of “F” factor
Francesco
Gratton
2013
Stanag Proficiency Test 2.0
New Key-factors
• Each section is a mini-test (L & R)
• Plus levels
• Percentages
Francesco
Gratton
2013
New COAs adopted
•Specifications
•Cut-off score
•Joint Database
Francesco
Gratton
2013
JFLT 2.0: RDS (Listening & Reading)
Section 1 –Stanag level 1 (questions from 1 to 15)
No.Correct answ. 1
2
3
4
5
6
7
0
Level awarded
8
9
0+
10 11 12 13 14 15
1
Section 2 –Stanag level 2 (questions from 16 to 30)
No.Correct answ. 1
2
3
4
5
6
7
1
Level awarded
8
9
1+
10 11 12 13 14 15
2
Section 3 –Stanag level 3 (questions from 31 to 45)
No.Correct answ. 1
2
3
4
5
6
7
2
Level awarded
8
9
2+
10 11 12 13 14 15
3
Section 4 –Stanag level 4 (questions from 46 to 60)
No.Correct answ. 1
Level awarded
2
3
4
3
5
6
7
8
3+
9
10 11 12 13 14 15
4
Francesco
Gratton
2013
EXAMPLE
 Level 1 (15 questions)
 Level 2 (15 questions)
 Level 3 (15 questions)
Max score: 45
All candidates answer correctly to 30 questions
With Old test they would all get the same score
Francesco
Gratton
2013
CORRECT
LEVEL 1
LEVEL 2
LEVEL 3
BIANCHI
30
10
10
10
3
ROSSI
30
11
10
9
VERDI
30
13
12
5
2+
2
GIALLI
30
15
8
7
1+
ARANCIONI
30
15
6
9
1
CANDIDATO ANSWERS (15 Questions) (15 Questions) (15 Questions) FINAL LEVEL
Francesco
Gratton
2013
New COAs adopted
•Specifications
•Cut-off score
•Joint Database
Francesco
Gratton
2013
NEW LISTENING & READING ITEMS
JDB (JOINT DATA BASE)
JOINT EFFORT
HOW NOT TO MAKE JOINT EFFORTS
Francesco
Gratton
2013
Test-writers involved
• Accustomed to military environment
• Language Testing Seminar
• Qualified
• Norming Sessions
Francesco
Gratton
2013
ARMY
CARABINIERI
NAVY
AIR
FORCE
Francesco
Gratton
2013
2012
2013
AUG SEP OCT NOV DEC JAN FEB MAR APR MAJ JUN JUL
WG
GdL
GdL
GdL
Phase
1
Phase
2
Phase
3
Phase
4
TIMINGS
Phase
1
AUGUST 2013
60 NEW ITEMS.
INTO JDB
Phase
2
Phase
3
Phase
1
NOVEMBER 2013
80 NEW ITEMS.
INTO JDB
FEBRUARY 2013
20 NEW ITEMS.
INTO JDB
Phase
4
Phase
2
MAJ 2013
40 NEW ITEMS.
INTO JDB
Phase
3
Phase
1
Francesco
Gratton
2013
Summary:
• Past situation
• COA (specs, JDB, cut-off score)
• A do-it-all software
• Proposals
Francesco
Gratton
2013
WHAT’S THE DIFFERENCE ?
Francesco
Gratton
2013
PC-assessment-related terminology
Term
Definition
Stakes
Any systematic method of obtaining
Assessment
evidence (through questions) for a purpose.
Quiz
Survey
Test
Exam
… measures for the purpose of providing
feedback to the student.
Low
… to determine needs required to fulfill a
Low
defined purpose.
… measures knowledge for the purpose of
informing the student on their current level
Medium
… measures knowledge for the purpose of
High
documenting the current level of knowledge
Francesco
Gratton
2013
the software is used for:
• Needs analysis:(surveys)
• Placement test
• Any training activity
• Assessment:
– First level survey
– Post-course
– Pre-certification
– Certification
Francesco
Gratton
2013
Software’s system Schema
Create
questions
And organize
them in
tests using
a windows
based PC
Francesco
Gratton
2013
Assessment ……
… allows to choose:
Question
Manager
Assessment
Manager
On Windows PC
• Time limits
• Feedback to test-taker
• Styles (Template)
Questions
Question
Manager
Assessment
Manager
via Browser
Assessment
Definitions
• Jumps
• Question shuffling
• Instructions to testtakers
Francesco
Gratton
2013
Assessments
• … also created with
Question
Manager
Assessment
Manager
su PC Windows
authoring manager by
selecting Qs previously
Question
Manager
created
• Any question can be
Questions
Assessment
Manager
via Browser
Assessment
Definitions
chosen from the
database
Francesco
Gratton
2013
Software’s system Schema
Result
reports, CIA,
graphs,
gimmicks,
you name it
…..
assessment
published using
any browser,
secure browsers, or
a PC/MAC
Create
questions
And organize
them in
tests using
a windows
based PC
Set security
parameters,
schedule
assessment
and link to
other
(Learning
Management
Systems
Francesco
Gratton
2013
Types of Questions: Multiple Choice
Francesco
Gratton
2013
Likert Scale (for questionnaires)
Francesco
Gratton
2013
Essay Question
1. Candidate can
write free text in
the space
provided
2. Testers will
evaluate later
Francesco
Gratton
2013
Francesco
Gratton
2013
Summary:
• Past situation
• COA (specs, JDB, cut-off score)
• A do-it-all software
• Proposals
Francesco
Gratton
2013
Testing in the time of crisis
•
Wide project
•
Sharing experience & capabilities
•
Optimization of resources
•
No alternative to B.A.T.
Francesco
Gratton
2013
Francesco
Gratton
2013
COMBINED DATABASE
or …
A multilateral-based CDB
or …
A Bilateral-based CDB
Francesco
Gratton
2013
Francesco
Gratton
2013
1 SPECS
2
3
FLOW
TIME SCHEDULE
Francesco
Gratton
2013
Francesco
Gratton
2013
canutesting@sclingue.esercito.difesa.it
Thank you
francesco.gratton@gmail.com
Francesco
Gratton
2013