MENO
An admission test experiment
at the Vrije Universiteit Brussel
Raoul Van Esbroeck
Why an admission test in Flanders?
•
•
•
•
•
•
Open admission
High failure rate
Costs for society
Costs for the individual student
Need for more efficiency
Admission test a possible option
Vrije Universiteit Brussel
23/11/2006 Pag.2
Why the MENO-test?
•
•
•
UCLES (University of Cambridge Local Examination Syndicate)
Skills crucial to success in HE
Limited to 50 multiple choice items
•
•
•
•
•
Critical Thinking (CT) with 3 subscales
Problem Solving (PS) with 3 subscales
Good reliability and validity
Limited bias (language, age, ethnic origin)
Gender and SES bias?
Vrije Universiteit Brussel
23/11/2006 Pag.3
Admission test experiment
•
•
OBPWO project
Questionnaire
•
•
•
•
•
(Promoter: Prof. M. De Metsenaere)
Social and cultural variables
MENO- test
Administered in class during first week
Cohort of entering Freshmen in 95-96
Respondents: 85% of the target group (N =
1000)
Vrije Universiteit Brussel
23/11/2006 Pag.4
Does MENO allow correct classification?
•
Errors in classification
•
•
•
False positives (FP)
False negatives (FN)
Total number of errors
% refused
10
25
50
•
%FN
6
18
39
%FP
87
63
39
Conclusion: too many errors
Vrije Universiteit Brussel
23/11/2006 Pag.5
Has the test an added value?
•
Can the test predict academic success better than
SE results?
•
A model with 4 variables:
•
•
SE option, class rank, graduating on time
MENO
•
Explains 18.7% of variance
MENO represents only 1.75% in this model
•
Conclusion: only limited added value
•
Vrije Universiteit Brussel
23/11/2006 Pag.6
Bias?
•
•
•
•
•
•
•
•
There is a gender bias
Girls would be more refused than boys
FN error is higher with girls
FP error is higher with boys
There is a SES bias
Lower SES will be more refused than higher SES
groups
FN error is higher with lower SES groups
FP error is higher with higher SES groups
Vrije Universiteit Brussel
23/11/2006 Pag.7
Increase of success rate?
•
There is limited increase
•
If 25% of the lowest MENO-scores are
eliminated success rate increases from
49% to 58%
•
What about errors?
Vrije Universiteit Brussel
23/11/2006 Pag.8
Conclusion on MENO
•
MENO is not an acceptable
admission test
•
•
•
•
•
Too many errors in classification
Limited added value
There is gender and SES bias
The gains are limited
Using this test would be unethical
Vrije Universiteit Brussel
23/11/2006 Pag.9
Are other tests better?
•
There are indications that the MENO in the UK is also
biased
•
The Chrysostomos test (P. Janssen, KULeuven) also
faces classification errors
•
•
•
A “Pass” (50%) score on the test leads to 37.6% of classification
errors
A 60% score on the test leads to 30.9% of classification errors (with
about 5% of FN)
Similar problems exist in other countries where
admission tests are used
Vrije Universiteit Brussel
23/11/2006 Pag.10
General conclusion
•
There is no need for admission tests because
•
•
•
•
•
•
•
There is limited added value
Too many errors are made
There is bias
The gains in success rate are limited
The costs for the tests are extremely high
There is no real societal benefit if FN are to
be avoided
There will be too many wrong decisions if
you want to have real gains
Vrije Universiteit Brussel
23/11/2006 Pag.11
Personal reflection
•
The use of general admission tests
for selection is unacceptable and
unethical
•
Such tests may be excellent for
guidance purposes
Vrije Universiteit Brussel
23/11/2006 Pag.12