Comparison of two
Normalization Procedures
Results of some statistical analysis
Normalization procedure 1
• Get percentile (P) of a student on the basis of 5subject aggregate scores in board.
• Determine the JEE-Main aggregate marks
corresponding to percentile P at the All-India level.
Regard this as the normalized board score of the
student (B1).
• Get JEE-Main aggregate marks (A0) of the student.
• The composite marks used for drawing merit list is
A1 = 0.6 A0 + 0.4 B1.
Normalization procedure 2
• Get percentile (P) of a student on the basis of 5-subject
aggregate scores in board.
• Determine the JEE-Main aggregate marks corresponding to
percentile P from the set of aggregate marks obtained by
different students of that board in JEE-Main. Regard this as
the normalized board score of the student (B2).
• Get JEE-Main aggregate marks (A0) of the student.
• The composite marks used for drawing merit list is
A2 = 0.6 A0 + 0.4 B2.
Common assumptions
• Common assumptions
• aggregate scores of students in a board examination are in the order of
their general merit within that board
(scholastic achievement indicates ability).
• aggregate scores of the students in JEE-Main are in the order of their
merit in respect of engineering admissions
(achievement indicates aptitude).
• Assumptions may be violated if
• a student has a poor exam,
• A student has a special advantage.
• If these effects are confounded with merit, violations become
difficult to rectify.
• No fairer assumption is possible.
Exclusive assumptions
• Procedure 1
• General merit distribution does not vary from one
board to another.
• Procedure 2
• Merit distribution varies from board to board;
this difference is captured by
differences in performance patterns of students of
various boards in JEE-Main .
Data for analysis
One data common data set:
• Aggregate marks in AIEEE 2012 and the corresponding percentiles
computed from the set of all students taking the examination.
Three sets of data in respect of six boards:
• Aggregate marks in board examinations 2012 (expressed as
percentage) and the corresponding percentiles for all students
taking the examination.
• Aggregate marks in AIEEE 2012 and the corresponding percentiles
computed from the set of students from the chosen board.
• Matched pair of AIEEE aggregate marks and board aggregate
percentage.
Issue 1: Extent of ‘compensation’
for board percentile ‘disparity’
350
Transformed board score ( B2)
300
CBSE
AM
JH
MR
MZ
UK
250
200
150
100
50
0
-50
-100
0
20
40
60
Percentile in board (P)
80
100
‘Compensation’ for ‘disparity’:
some specific examples
• Transformed score of a student at the 50th percentile
of CBSE is more than that of
– a student at the 80th percentile of Maharashtra Board,
– a student at the 85th percentile of Jharkhand Board.
• Transformed board scores of top five students of
three boards:
– CBSE: 346,
– Maharashtra board : 331,
– Jharkhand board : 274.
Amount of compensation (by Procedure 2) is LARGE!
What is being ‘compensated’?
• Poor performance of students of some boards in AIEEE may
be due to
– Poor merit/ability (not a tenable explanation for large boards),
– Non-alignment of board examination pattern with AIEEE
(rank correlations indicate this),
– Lack of instruction in English and Hindi
(only available languages for AIEEE/JEE-Main),
– Less access to coaching,
– Load of an extra subject in board (for some boards).
• All these effects are confounded.
• If confounding factors are ignored, solution may be worse
than the problem.
Issue 2: Performance pattern of different boards
(distribution of students in different percentile ranges)
100%
80%
60%
40%
20%
0%
100%
80%
60%
40%
20%
0%
Current
procedure
100%
80%
95 <= P0
60%
75 <= P0 < 95
40%
50 <= P0 < 75
20%
P0 < 50
0%
Procedure 1
95 <= P1
75 <= P1 < 95
50 <= P1 < 75
P1 < 50
Procedure 2
95 <= P2
75 <= P2 < 95
50 <= P2 < 75
P2 < 50
Disparity among the distributions
is the least for Procedure 1.
Issue 3: The question of fairness
• In 2012, five students (4 from CBSE, 1 from Maharashtra
board) were tied at
– AIEEE aggregate marks of 130 ,
– board percentile of 93.1.
• According to Procedure 1, they shared rank 20,352.
• According to Procedure 2
– CBSE students had shared rank 18,077,
– Maharashtra board students had rank 34,175.
• The differential treatment is entirely due to performance of
other students of the two boards in AIEEE.
• This goes against the principle of fairness.
Issue 4: Sampling bias
• Procedure 2 requires the
assumption
• Board to board
difference in merit
distribution is
captured by
differences in
performance patterns
of students of various
boards in JEE-Main .
Proportion of AIEEE candidates in percentile
range (board aggregate)
Board
0-20
20-50
50-80
Above 80
Total
Assam
0%
0%
2%
98%
100%
CBSE
29%
15%
19%
37%
100%
Jharkhand
16%
5%
5%
74%
100%
Maharashtra
11%
8%
11%
69%
100%
1%
1%
2%
96%
100%
14%
3%
4%
79%
100%
Mizoram
Uttarakhand
• For this assumption to hold, sampling of AIEEE candidates from the
boards should follow the same pattern.
• However, 2012 data shows that AIEEE attracts different profiles of
students from different boards.
In summary
• Students of different boards face different types/levels of difficulty in AIEEE.
• The presence of these factors confound any presumed board-to-board
difference in merit distributions.
• Board-to-board difference in performance patterns in AIEEE is vast.
• Use of AIEEE marks for ‘tracking’ would produce drastic amount of
compensation.
• Such levels of unequal treatment of percentiles of different boards, based on
presumptions, is unfair.
• Procedure 2 also penalizes a student for poor performance of peers from
same board.
• For large boards, there is a strong reason to presume a common merit
distribution.
• Procedure 1 produces a more equitable pattern of performance of students
of different boards.
Criteria that are NOT appropriate
for choosing between procedures
• Correlation of new and old ranks.
– How much correlation is ideal?
– The more the merrier?
• Movement of students from one merit list to
another.
– How much movement is ideal?
– The less the merrier?
A benchmark for rank correlations
Rank correlation between
board and AIEEE marks
Spearman rank correlation
Board
with AIEEE aggregate marks
Assam
0.4124
CBSE
0.6891
Jharkhand
0.1926
Maharashtra
0.4322
Mizoram
0.2919
Uttarakhand
0.3235
All matched
samples
0.4841
Observations
• No rank correlation is
more than 0.7.
• Board examinations are
different from AIEEE
(measure something else).
• CBSE marks have highest
rank correlation with
AIEEE:
– Similarity of examination
pattern,
– Same agency administers
CBSE exams and AIEEE.
Correlation of new and old ranks
Rank
Rank
correlation
correlation
between
between
Board
A0 and A1
A0 and A2
Assam
0.7283
0.8347
CBSE
0.9464
0.9302
Jharkhand
0.5782
0.7163
Maharashtra
0.7691
0.8337
Mizoram
0.5936
0.7025
Uttarakhand
0.5996
0.7435
0.8559
0.9106
All
matched
samples
• Not much difference
overall, and for CBSE.
• For some boards,
Procedure 1 produces lower
correlation .
• This is in line with observed
low correlation between
board and AIEEE marks.
• Why should large
correlation be a good thing?
Movement from old to new rank list
• Procedure 1 produces
greater movement.
• This is in line with observed
disparity between board
and AIEEE marks.
Some questions of interest to CBSE
1. What do the correlation patterns and rank
movements indicate about the fate of CBSE students?
2. Will the CBSE students be at a disadvantage under
Procedure 1?
3. Will they get a better deal under Procedure 2?
4. Will smaller boards gain unfairly from Procedure 1?
5. Is there a possibility of ‘board hopping’ if Procedure 1
is used?
Q1: Correlation patterns and rank
movements
• Correlation and movement patterns for the two procedures
are SIMILAR.
• Correlations indicate degree of association/alignment
between new and old ranks.
• Correlations do not reveal ANYTHING about the relative
advantage or disadvantage to CBSE students.
• If all CBSE students are pushed down 10,000 places in AIEEE
merit list, their rank correlation with board scores would stay
the same.
• Scatter-plots of rank movement show that students exchange
fate with one another .
– no clue about advantage/disadvantage to CBSE students.
New simulations to answer Q2-Q5
• Need simulated data for ALL the boards
• Simulation conditions
– 10% Random samples chosen from AIEEE 2012
scores data.
– Presumed correlation between AIEEE and board
scores:
• Observed correlations for AS, CBSE, JK, MR, MZ, UK
• 0.4 for other boards.
– 100 separate simulation runs.
Actual performance of CBSE
students in AIEEE 2012
• 30.1% of all AIEEE candidates are CBSE students.
• However, this groups constitutes
–
–
–
–
66 of the top 100 AIEEE rand holders,
608 of the top 1,000 AIEEE rand holders,
5,977 of the top 10,000 AIEEE rand holders,
54,430 of the top 100,000 AIEEE rand holders.
Share of boards in different
percentile ranges of AIEEE 2012
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
CBSE
MR
AP
BR
UP
RJ
All other
How Procedure 1 or 2 would
change the shares
100%
AIEEE 2012 - Procedure 1
100%
80%
AIEEE 2012 - Procedure 2
80%
CBSE
60%
MR
40%
AP
20%
BR
UP
60%
40%
MR
20%
AP
RJ
0%
UP
All other
CBSE
0%
BR
RJ
All other
AIEEE 2012 - Present scenario
100%
80%
CBSE
60%
MR
40%
20%
0%
AP
BR
UP
RJ
All other
• Procedure 1 reduces marginally the
dominance of CBSE students in
merit list ,
• Procedure 2 increases drastically
that dominance.
…and the numbers behind the charts
Board
CBSE
MR
AP
BR
All other
Procedure 1: Share of boards in different percentile ranges of AIEEE 2012
0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100
15%
18%
21%
24%
26%
29%
33%
38%
44%
53%
17%
16%
14%
13%
12%
11%
10%
8%
7%
5%
7%
8%
8%
8%
8%
8%
9%
9%
10%
13%
10%
9%
9%
8%
8%
7%
6%
6%
5%
3%
51%
49%
48%
47%
46%
44%
42%
39%
35%
26%
Board
CBSE
MR
AP
BR
All other
Procedure 2: Share of boards in different percentile ranges of AIEEE 2012
0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100
11%
14%
17%
21%
25%
29%
35%
42%
48%
59%
20%
18%
16%
14%
12%
11%
9%
7%
5%
4%
6%
6%
7%
8%
8%
9%
9%
10%
11%
13%
11%
10%
10%
9%
8%
7%
6%
5%
4%
2%
53%
51%
50%
49%
47%
44%
41%
36%
31%
22%
Board
CBSE
MR
AP
BR
All other
Current scenario: share of boards in different percentile ranges of AIEEE 2012
10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100
17%
20%
22%
26%
28%
33%
39%
46%
54%
17%
15%
14%
12%
11%
9%
8%
6%
5%
8%
8%
8%
8%
8%
9%
9%
10%
13%
9%
9%
8%
8%
7%
6%
5%
4%
3%
49%
48%
48%
46%
45%
42%
38%
33%
25%
0-10
14%
17%
7%
10%
51%
Overall
30%
11%
9%
7%
43%
Findings from simulation
Present scenario
Procedure 1
Procedure 2
Top 0.01%
65.1%
61.9%
82.7%
Top 0.1%
60.5%
60.4%
80.1%
Top 1%
60.0%
60.0%
69.6%
Top 10%
54.4%
53.4%
58.8%
•
Procedure 1 produces a marginal reduction in CBSE’s share at the top;
2% reduction at the top 0.01%.
•
Procedure 2 brings a great windfall to CBSE students.
•
CBSE students gain more and more from Procedure 2 as one goes up
the merit list (share at top 1% and top 0.1% increases by 10% and
20%, respectively ).
•
If interest of top CBSE students had been the criterion, Procedure 2
would be a clear winner!
How does Procedure 2 dramatically
increase CBSE’s share of top-rankers?
• It does so by practically turning the board score into another
AIEEE score.
• The composite score A2 for each board has same mean as A0,
but has much smaller spread.
• Each set of board scores get more concentrated around their
respective means.
• In effect, every students gets a second chance to perform at
the same mean level.
• As a result, CBSE students gain substantially.
Why does Procedure 1 have only a mild
effect on CBSE’s share of top-rankers?
• For CBSE, the students who do well in AIEEE are
generally those who do well in board exam.
– They mostly hold on to their position in merit list.
• For boards with less rank correlation with AIEEE, this
is not the case.
– Many AIEEE top-rankers from other boards move towards
middle in terms of composite score.
– If their board exams had been better aligned with AIEEE,
they would have gained more from Procedure 1.
Answers to Q2 and Q3
•
Q1: Will the CBSE students be at a disadvantage
under Procedure 1?
–
•
Yes, but marginally so.
Q2: Will they get a better deal under Procedure 2?
–
Yes, the top third of the merit list gets PACKED with CBSE
students.
–
It is not just a better deal, it is a great deal.
–
It would not be fair to other boards!
Q4: Will smaller boards gain unfairly
from Procedure 1?
• A small board may not have same merit distribution as others. It may have
– Stronger students, or
– Weaker students.
• To say that they gain more (from Procedure 1) than larger boards is to
PRESUME that they have weaker students.
• Percentiles generally correct for board size.
• Here, percentile of a student is defined as percentage of candidates strictly
below that student.
• Toppers of a smaller board will always have a smaller percentile than the
toppers of a larger board.
• Students of smaller boards are also more vulnerable to variations of all kinds.
Q5: Will Procedure 1 encourage board
hopping?
• Yes, but TOWARDS CBSE (continuation of current trend)
– CBSE’s loss and other boards’ gain is marginal.
– This loss is only in relation to the present scenario; 2013 students cannot
hop into 2012 system.
– Even after this minor loss, CBSE students will have continued advantage
over students of most other boards in terms of
•
•
•
•
alignment of examination pattern
alignment of language
access to coaching
a well administered examination system (less uncertainty)
– If a CBSE student hops to a smaller board, good marks are not guaranteed
(exam is likely to be of a different type)
• Confounding advantages in AIEEE should not give them false confidence.
Composition of different percentile
groups in 2012 by Procedure 1
100%
90%
80%
70%
60%
CBSE
50%
MR
40%
AP
30%
BR
20%
RJ
10%
UP
0%
All other