]
Placement Testing
and Audio Quiz-making
with
Open Source Software
Pacific Association of CALL International Conference
National University of Singapore, 3 December 2004
Don Hinkelman
Sapporo Gakuin University
University of Melbourne
Case Study:
1.
2.
3.
4.
Sapporo Gakuin University
2000 first and second year students.
Two year required general English program.
Students assigned alphabetically by family name.
Mixed ability levels in each class.
Six full-time teachers agreed in a staff workshop that
unstreamed class levels was their greatest problem
- bright students not challenged, lost interest
- weak students over-challenged, lost interest
Mixed levels - “Like herding cats”
QuickTime™ and a
Sorenson Video decompressor
are needed to see this picture.
Case Study:
Sapporo Gakuin University
2. Commercial testing software was expensive and would
take 1-2 years to get budget approval. We had doubts
about its usefulness. We wanted a system where we
could control and improve the the test items.
-
Calus Testing Service
3000 yen/student/year 1000 freshmen/year.
Total budget = 3,000,000 yen/year (USD31,000)
Case Study:
Sapporo Gakuin University
3. We wanted to adapt our current progress test system
(50 questions--30 listening, 20 reading) for placement
of incoming freshmen.
But paper test marking services 300,000 yen /year
We wanted a “testing machine” that would not give an
audio-based test, but also analyze the results so we
could improve the test questions.
Placement Testing
Machine
version 3.1
50 User friendly
questions



Reusable stainless
steel interface
User adaptable hood
harness
Wrist and leg straps
not necessary
I am level 3!
Questions
Case Study:
Sapporo Gakuin University
4. Tool Selection
We had experience administering computer-based m/c tests
in large English classes using an open source CMS with
quiz-making tools (Moodle).
-
10 computer rooms available in beginning of term.
Why not do a computer-based placement test
Rooms had no headphones.
But quizzes could not do audio.
How would we do listening questions?
Case Study:
Sapporo Gakuin University
5. Programming
We went to local colleague-volunteer and the lead developer
of Moodle and asked them to modify Moodle to handle
listening questions.
- One colleague contributed 110,000 yen of personal
research money for programming.
- Sent money to moodle.com to supervise process.
- Local colleague fell through--no programming
- Dropped all bells and whistles requirements
- moodle.com came through with core program:
auto-embedded flash buttons with media filter
\ ]
Core Requirements
for a placement test programming
Play/Stop Button within each Question:
It should be intuitive for student to use
No Popup Player:
Avoid confusion for student
Number of Plays Setting:
It should be easy to create an
online course from existing learner resources,
Simple Login for Student:
It should provide a simple
screen. provide bilingual menus, help, and input.
No Result Displayed at End
\ ]
Actual Accepted Interface
for a placement test programming
OK Play/Stop Button within each Question:
It should be intuitive for student to use
OK No Popup Player:
Avoid confusion for student
CUT Number of Plays Setting: We just set a start and end
time for the whole test. We let students play as many times as
needed, but warned them to finish in time allowed.
CUT Simple Login for Student:
We used standard
Moodle login. Teachers turned on machines, logged into main
school system. Then we assigned seats, and put unique username
and password on a paper on the keyboard of each machine.
CUT No Result Displayed at End
Course Mangement System
Evaluating Tests:




Practicality
Reliability
Validity
Washback
Four Criteria
Types of Tests

Criterion Referenced
 Proficiency
Tests
 Achievement Tests

Norm Referenced *
 Placement
Tests
[Implications for design, validity]


Content can be more arbitrary
Construct based more on listening, less on reading
Validity
Content
● Related to high school curriculum in Japan
● General topics of common interest in Hokkaido
Construct
● Greater emphasis on listening
60%
● Less emphasis on reading
40%
● No speaking, writing
Washback
●
Effect on teachers
–
Informal feedback:
●
●
enthusiastic comments
felt student group cohesion better,
Reliability



Item Facility - 50 questions
X
O
X
0~30% 30~40% 40~60% 60~70% 70~100%
Item Discrimination
X
<1.0
1.0~1.3
Distractor Analysis
O
>1.3
Reliability Results
Item Facili ty
Performance
Number of Items
Level
Good
12
Acceptable
16
Poor: too difficult
6
Poor: too easy
16
Item Discrimi nation
Performance
Level
Good
Acceptable
Poor
Number of Items
20
18
12
Too easy 92% correct
High IF

Item Q-38 (page 3 in handout)

Watching the news always _____ me sad




A. puts
B. changes
C. results
D. makes
Too hard 5% correct
Low IF

Item Q-55 (page 3 in handout)

What does the author feel about human relationships in
the future?




A.
B.
C.
D.
He thinks they will be more convenient
He thinks they will be more controlled by technology
He does not express an opinion on the matter
He thinks they will contain a large amount of information
Random Answers
Low ID = 1.3

Poorer students answer correctly as often as
better students

Item Q-48 (page 3 in handout)

Which word means the same as “task”?
(in the second paragraph)




A.
B.
C.
D.
tool
job
ability
industry
Practicality
●
●
●
●
Organizing and Achieving Consensus
Exam Proctoring and Management
Hardware Test
Software Test
Results
Server 1
Mac OS X, 933 mhz, single
Result: 150 students, server stopped, switched to paper test
80 students, very slow, successful
Server 2
Mac OS X, 1.2 ghz dual
Result: 100 students, server stopped
Finder problems, hardware freezes
Server 3
Linux Fedora 2, 3.2ghz single
Result: 100 students, successful
250 students at one time planned for 2005
2 servers, plus two backup servers
Hardware Specifications/Cost
Server 3 Linux Fedora 2, Pentium 4--3.2ghz single processor
Built from parts, server engineer made LAMP in free time.
Parts
Motherboard
Processor
Memory, 2gb
Hard disk, 120gb
DVD-R drive
Case
Labor
Assembly
Configuration
yen
13,000
30,000
45,000
8,000
6,000
8,000
USD
120
270
410
75
55
70
10,000
40,000
90
360
Total Cost:
160,000 yen
1,450 USD
RAID with second hard disk will automatically clone whole server
+30,000 yen
Time Costs
Test Rooms
4 x 60 computers
Conclusion
●
Future:
–
–
80 students >> 1000 students in one day
Two tests
●
●
●
Higher Level: English/Humanities majors
Lower Level: Economics/Law/Computer majors
Goal:
–
–
–
April 2005
Gradually improve test items
45% problem items >> 10% problem items
Save $30,000 commercial testing fees/year
Superior placement test norming for our student
population
Thank you
Contact:
Testing Operation: Don Hinkelman, Tim Grose
<hinkel@sgu.ac.jp> <grose@sgu.ac.jp>
Statistical Report Module: Tom Robb
<tom@robb.net>