Best practices in continuous evaluation:
Evidence-Based Decision Making and
Admissions Models
Admissions criteria and success criteria
ETS Europe
January 2006
Jenny Dalalakis
Gerben van Lent
jdalalakis@etseurope.org
gvanlent@etseurope.org
Key Messages
• Admissions Criteria are the outcome of a decision
process which must be regularly evaluated
• The process has the following components:
–
–
–
–
Strategic objectives of the institution
Instruments to identify student quality
Student application model
Admissions criteria
Objective of this presentation:
Introduce Best Practices in
Admission Evaluations
How to Admit?
What is Student
Quality?
Why
Admit?
How to Decide?
Admissions: Best Practices
C. How to
Admit?
Admissions
Criteria
Why
Admit?
Consensus on
Strategic
Objectives
A. What is Student
Quality?
Qualities sought in
student body
B. How to Decide?
Evaluation Models,
Fairness, Validity
Outline
Determining Institute’s Strategic Objectives for Admissions
Deciding on Qualities of Student Body
Evaluating Application Models
Deciding on Key Admissions Criteria
Wrap up: Conclusions and Questions
Strategic Objective: Successful fit between Policy
and Admissions Process
College Board research identified best practices as:
•
Consensus on admissions objectives
–
–
•
Consensus on the qualities sought in student body
–
–
–
•
relevance of information, equal access, transparency of admission process
Validity of admissions process
–
•
Institutional goals supported in transparent manner
Institution’s Policy affects weight given to student quality measurements
Profile of student balanced among different qualities sought
Fairness
–
•
Institutional objectives supported by individual and group qualities of student body
Reflected in application components (explicit) and underlying factors (role of informal priorities)
Evaluations and processing models
–
–
–
•
Board, staff, alumni, current student body, labour market representatives
Academic and non-academic qualities of admired students
Acceptable instruments to measure these desired qualities
Admissions criteria
–
–
•
Explicitly affirmed and revised periodically
Quantitative and qualitative criteria of incoming student body
Periodic correlation measurement
Future considerations
–
change is inevitable
Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003)
Admissions: Best Practices
C. How to
Admit?
Admissions
Criteria
Why
Admit?
Consensus on
Strategic
Objectives
A. What is Student
Quality?
Qualities sought in
student body
B. How to Decide?
Evaluation Models,
Fairness, Validity
Strategic Objectives:
Why Admit?
ACA Research identified 6 key objectives
–
–
–
–
–
–
Policy
Revenue
Profile
Sustainability
Non-discrimination
Demand
(Hybrid needs typically exist)
Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)
Why Admit?
Key objective: Development Policy
Institution Policy or Government Policy requires foreign students
– France: Prestige in educating other countries’ elite students and future
leaders
– Netherlands: Manner of providing aid to developing countries
– UK, US: Foreign students become host country’s goodwill
ambassadors when they return home
– EU: Support for mobility (Bologna, Lisbon) means international student
exchanges are vital at Higher Education level
Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)
Why Admit?
Key objective: Revenue
Market-driven need for funds related to foreign students
•
•
•
•
Public and private schools may charge more for all or some international
students directly so the extra income is welcome
– UK, US
– CH (MBA programme students specifically)
– NL (English-taught programme students specifically)
Public university with foreign students may qualify for extra government funds
– Australia
Programme survival in danger and loss of staff so foreign students help numbers
– SE, CH, NL
Disincentives may apply if institution does not qualify for extra funds if they have
foreign students
– CH
Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)
Why Admit?
Key objective: Institutional Profile
International profile is of intrinsic value for Institution
•
•
•
Foreign students enhance innovation and modernization
Global pool better for choosing world-scholar quality
– Minimum or maximum target % may be set: AU
Global recognition of host university for its student quality is very competitive
–
•
•
Competition at level of Institution/ Discipline or at level of Labour Market Demands
Nature of programme requires international approach
– Research literature is in English (MBA, Engineering): FR
– Applied Languages, International Relations, International Law
Foreign students bring the world home to remote areas
– AU, SE
Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)
Why Admit?
Key objective: Sustainability
Securing the future of the programme, staff, research
•
Smaller countries do not have enough local students for keeping some
disciplines alive at their universities
– NL, SE, CH need Science students
•
Students coming for Masters programmes are encouraged to stay on
for PhD programmes to secure enough teaching staff assistants and
research assistants
Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)
Why Admit?
Key objective: Non-discrimination
No discrimination is tolerated between any student groups
• Quality of students comes first regardless of student’s origin
• Important objective for highly selective / private institutions: US
• Some are more equal than others (informally or formally)
– CH: evidence required that foreign student could have studied back
home
– FR: to protect local student quota, local students chosen with earlier
deadline for public university places
– SE: first choose local students and unfilled quota goes to foreign
students
Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)
Why Admit?
Key objective: Students’ Demand
Large number of foreign students want to come to the Institution
• Local, regional and global mobility (economic, academic,
professional) is a growing demand by students, governments,
labour market so students expect international programmes in
countries of interest to them
• Universities receive large number of foreign student applications
and react by starting international programmes and (hopefully)
support programmes
Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)
Admissions: Best Practices
C. How to
Admit?
Admissions
Criteria
Why
Admit?
Consensus on
Strategic
Objectives
A. What is Student
Quality?
Qualities sought in
student body
B. How to Decide?
Evaluation Models,
Fairness, Validity
How to Admit? Look for Quality
What is Student Quality?
ACA research showed a variety of instruments for gathering
evidence on student quality
–
–
–
–
–
–
–
–
–
–
–
Secondary school/ Bachelor diploma
Transcript of courses
List of completed courses
Motivation statement/personal statement
Letter of recommendation
Proof of work experience
Proof of previous academic work
Curriculum Vita
Standardized tests for subject and skill competencies
Proof of knowledge of the teaching language
Right to access Higher Education in home country
Source: The Admission of International Students in Higher Education, ACA Paper on International Cooperation in Education (2004)
How to Admit? Look for Quality
What is Student Quality?
College Board research on factors used in making admissions decisions
– Academic Achievement, Quality, and Potential
• Direct measures (coursework, grades, test scores)
• Caliber of high school/ previous study institution (rank of class, school)
• Evaluative measures (depth and breadth of interests, world awareness,
communication skills)
– Non-academic Characteristics and Attributes
• Geographic (remote or disadvantaged region, local resident)
• Personal Background (minority, socio-economic status, age)
• Extra-curricular activities, service, & leadership (awards, social service, work
experience)
• Personal attributes (talent, character, commitment, social concern, creativity,
determination)
• Extenuating circumstances (family problems, health issues, language difficulties)
This may differ somewhat from what institutions tell students what they are looking for
Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board
(2003)
Where do Student Quality Instruments
fit?
University policy for educating students
S
t
u
d
e
n
t
b
a
c
k
g
r
o
u
n
d
known
Low score on
TOEFL ITP or
TOEIC
inclusive
exclusive
High score on
GMAT
High score on
TOEFL
unknown
Where do Student Quality Instruments
fit?
University policy for educating students
known
S
t
u
d
e
n
t
b inclusive
a
c
k
g
r
o
u
n
d
•Motivation statement/personal statement
•List of completed courses
•Letter of recommendation
•Proof of work experience
exclusive
•Secondary school/bachelor diploma
•Right to access HE in home country
•Proof of knowledge of the teaching language
•Proof of previous academic work
•Standardized tests
unknown
How to decide depends on which
type of error you tolerate more
capable
refused
admitted
not capable
Consider exceptional cases:
(a) Lottery systems (Numerus Clausus)
(b) Admissions and immigration (NL, DE)
S
t
u
d
e
n
t
b
a
c
k
g
r
o
u
n
d
University policy for educating students
known
inclusive
exclusive
unknown
‘ Capable’ = Successful
How do we recognize success?
Popular indicators of evidence:
I. Student quality
II. Attrition
III. Time to degree
• Faculty ratings
• Grade point average
• Results of Exams
Which Student Quality?
Student Quality =
Competencies important for
graduate school success
Some qualities are also important to the labour market
Student Quality?
Student Quality = Competencies important for
graduate school success
Three research studies
•
Two studies asking faculty for competencies of successful
students
•
One scheme for classifying outcome or criterion variables:
“taxonomy of higher order performance components.”
Types of Competencies
linked to Student Quality
I.
II.
III.
IV.
V.
VI.
VII.
VIII.
Job-specific, Study-specific task proficiency
Non-job, Non-study-specific task proficiency
Written & Oral Communication task proficiency
Demonstration of effort
Maintenance of personal Discipline
Facilitation of peer and team performance
Supervision/ Leadership
Management/ Administration
•
•
Some are present when student is admitted
Some are part of the curriculum that the institution
provides (educational value added) and so are expected at
graduation
Source: Standardized Letter of Recommendation, Educational Testing Service
Competencies
I.
Job/study Specific Task Proficiency
1. Research/scholarly experience (amount and quality)
5. Mastery of discipline
11. Ability to read and analyze research in the field
13. Breadth of perspective
C. Explanation (Enright & Gitomer)
G. Synthesis (Enright & Gitomer)
Competencies
II. Non-Job/study specific Task Proficiency
7. Ability to teach (cross-listed in VII and VIII)
8. Independence (cross-listed in IV)
9. Creativity; ability to think “out of the box”
14. Open mindedness
15. Opportunism/Resourcefulness
16. Critical thinking ability, logic, problem solving ability
18. Computer literacy/Understand & Manage Technology
19. Skill in investigation
20. Ability to find the facts: information gathering
B. Creativity (Enright & Gitomer)
E. Planning (Enright & Gitomer)
Competencies
III. Written & Oral Communication Task Proficiency
4. Communication (professional)
6. English-as-a-Second-Language ability
A. Communication (Enright & Gitomer)
IV. Demonstration of Effort
2. Persistence/Tenacity*
8. Independence*
10. Enthusiasm*
17. Drive/Commitment/Motivation/Zeal*
D. Motivation (Enright & Gitomer)
Competencies
V. Maintenance of Personal Discipline
12. Values/Character: integrity, fairness/openness/honesty,
trustworthiness/ consistency – personal and professional*
21. Professional posture
VI. Facilitation of Peer and Team Performance
3. Collegiality/Making professional connections/Networking (Crosslisted with VII)
F. Professionalism
VII. Supervision/Leadership
3. Collegiality/Making professional connections/Networking (Crosslisted with VI.)
7. Ability to teach (Cross-listed with II. and VIII.)
VIII. Management/Administration
7. Ability to teach (Cross-listed with II. and VII.)
Cross reference to Tuning Project
How are competences and learning outcomes related?
• Learning outcomes according to Tuning methodology should
be formulated in terms of competences.
• Learning outcomes are requirements of a unit or a programme
and are expressed in terms what the learner knows and is
able to do at the end of the learning experience.
• Competences may be developed to a greater degree than the
level required by the learning outcome.
Graduates
 Capacity for analysis
and synthesis
 Capacity to learn
 Capacity for applying
knowledge in practice
 Elementary
computing skills
 Capacity to adapt to
new situations
Employers
 Capacity to learn
 Capacity for applying
knowledge in practice
 Capacity for analysis
and synthesis
 Capacity to adapt to
new situations
 Interpersonal skills
Academics
 Basic knowledge of the
field of study
 Capacity for analysis and
synthesis
 Capacity to learn
 Capacity for generating
new ideas (creativity)
 Capacity for applying
knowledge in practice
Staff-centred vs. Student-centred?
On completion of a first cycle degree in Mathematics,
students should be able to
•
•
•
•
•
Show knowledge and understanding of basic concepts,
principles, theories and results of Mathematics
Understand and explain the meaning of complex statements
using mathematical notation and language
Demonstrate skill in mathematical reasoning, manipulation
and calculation
Construct rigorous proofs
Demonstrate proficiency in different methods of mathematical
proof.
Admissions: Best Practices
C. How to
Admit?
Admissions
Criteria
Why
Admit?
Consensus on
Strategic
Objectives
A. What is Student
Quality?
Qualities sought in
student body
B. How to Decide?
Evaluation Models,
Fairness, Validity
How to recognize competencies?
The Admission Challenge
What are the competencies you attribute to successful students?
Which of them have to be present at the start?
Is the aim Inclusive or Exclusive selection?
The Admission Challenge
What evidence do you consider:
Is it fair, valid, reliable?
Have you considered all costs?
What is the legal setting?
Evaluating an Application:
Different models of meeting
the Admissions challenge
Admissions offices may be responsible for
 Combining all individually-rated components and weighing
–
–
–
–
Application information
Transcripts
Essays
Recommendations
 Combining academic and non-academic factors and weighing
– Some factors considered separately e.g., communication
– Balance of academic and non-academic
– Many wide-scope factors vs. few key factors
Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board
(2003)
Evaluating an Application:
Different model examples
Example 1
– Academic
– Personal characteristics
– Suitability for desired field of study
Example 2
– Academic
– Communication
– Character, leadership, initiative
Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate
Students, College Board (2003)
Evaluating an Application:
Different model examples
Example 3
– Exceptional academic Achievement
– Academic Promise
– Potential to Contribute
Example 4
–
–
–
–
Academic Achievement
Academic Qualities
Non-academic Achievement
Personal Qualities
Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select
Undergraduate Students, College Board (2003)
Evaluating an Application:
Different model examples
Example 5
–
–
–
–
–
Quality of Courses
Grades in Core Curriculum
Test results
Activities (extra-curricular or curricular)
Essay
Example 6
–
–
–
–
–
–
Academic Performance
Extra-curricular activities
Teacher / counselor recommendations
Interview
Personal Inventory
Essays
Source: Admissions – Decision-Making Models: How US Institutions of Higher Education Select Undergraduate Students, College Board (2003)
Evaluating an Application:
Different model examples
Example 8
Example 7
 Academic Achievement
– Application and Essay
 Intellectual Curiosity
– Academic Performance
– Level of Challenge of Academic Record  Potential
 Commitment
– Recommendation and Interview
 Communication
– Personal Qualities
 Engagement with others
– Special Talents
 Extra-curricular activities
 Initiative
Source: Admissions – Decision-Making Models: How US Institutions of Higher
Education Select Undergraduate Students, College Board (2003)
Making Decisions: Making sense of what
information comes in from Applications
•
Single readers, multiple readers, group readings, automated readings
– Process x for collecting academic info (formulaic rating) into an index
– Process y for collecting non-academic info (holistic rating) into an index
•
Clear minimum thresholds of entry per index or factor
•
Clear criteria for strong, and weak candidates
– Level descriptors for each level of qualification
•
Clear guidelines for multiple entryways
– Special consideration for minorities, shortage of students
– Lottery for some competitive programmes with limited places
– Work experience
•
Final review of denials
– Conditional acceptance
– Additional information or interview requested
Quality Assurance for
Application Models
•
Setting key current criteria into application form
– Direct and indirect questions
•
Setting dates for re-evaluation of application form criteria
– Selection of staff who represent key constituents affected by
decision
•
Adapting admissions criteria to institutional profile over time
– Respecting changes in strategic objectives
•
Adapting to external changes in education, demographics, labour
market
– Understanding where the student body comes from and goes to
Best Practices in Admissions
How to Select
What to look for
Admissions Criteria
Qualities sought
in student body
Consensus on
Admissions objectives
How to Decide
 Application & Processing models
 Fairness
 Validity
Key Messages
• Admissions Criteria are the outcome of a decision
process which must be regularly evaluated
• The process has the following components:
–
–
–
–
Strategic objectives of the institution
Instruments to identify student quality
Student application model
Admissions criteria
Discussion and insights
1.
Why do you admit students?

What are the 3 main Strategic Objectives of your institution?

What are your main 3 Admission objectives?

What are your 3 most common profiles of admitted students?
2. What is student quality for you?

What are the 3 main student-quality instruments that you use?
Consider how your policy objectives affect the level of detail that
you require from your instruments.

What are the most common instruments of your peers?

Where do your admissions instruments fall on this chart?
Where do your Student Quality
Instruments fit?
S
t
u
d
e
n
t
b
a
c
k
g
r
o
u
n
d
University policy for educating students
known
inclusive
exclusive
unknown
Discussion and insights
3. What is the quality of your added value to the student’s education?

Which 2 key competencies does your institution
expect to be present at admission?

Which 2 key competencies does your institution
expect at the learning process?
4. How do you recognize quality?

What evidence does your institution require?
Evidence from the student directly or other institutions
Student portfolio
High-school/ previous programme transcripts
Letters of recommendation
Personal answers to questions on the application
Candidate’s scores on standardized admissions tests
Language standardized tests


Which formal instruments do you use?
How do you ensure fairness, validity, and reliability?
Discussion and insights
5. What type of student application do you use?



What are the top 3 factors in your qualification evaluation model?
What quality assurance principles to you use to keep your application
model valid?

How frequently do you hold re-evaluation?

How do you ensure validity of criteria/ benchmarking success
factors
What is your process of updating criteria?