REPORT OF RESULTS ON COMMERCIAL COACHING
Barbara Griffin PhD
2013
These results have been published in peer reviewed academic journals1 and presented to
teachers at forums for career advisors held at Monash University and the University of
Western Sydney. Please do not reproduce the material without permission.
BACKGROUND
Current medical student selection in Australia draws on a large body of national and
international research. This significant evidence base indicates that best practice in selection
takes into account a range of skills, including cognitive and academic ability as well as noncognitive skills and abilities. Typically, these are assessed by a combination of the ATAR
score or a University Grade Point Average (GPA), the Undergraduate Medical and Health
Sciences Admissions Test (UMAT) or the Graduate Australian Medical Schools Admission
Test (GAMSAT), and interviews, either with a panel interview or a Multiple Mini Interview
(MMI). Our research has focussed on selection into undergraduate medical degrees.
RESEARCH STUDIES
Results reported here are from the analyses of data from two main sources:
1. Data from applicants to th University of Western Sydney School of Medicine – an
undergraduate medical degree. There are approximately 3000 applicants each year
and from these about 350 are shortlisted for interview.
2. Data from 432 Year 12 students from 31 high schools across Sydney and Melbourne.
 58.5% were females
 51% of the participants were enrolled in a ‘selective’ high school
_____________________________
1
Griffin, B. (in press). The ability to identify criteria: its relationship with social understanding, preparation, and
impression management in affecting predictor performance in a high stakes selection context. Human
Performance.
Griffin, B., Yeomans, N. & Wilson I. (in press). Studnets coached for an admission test perform less well
throughout a medical course. Internal Medicine Journal.
Griffin, B., Carless, S., & Wilson,I. (2013b). The Undergraduate Medical and Health Sciences Admissions Test:
What is it measuring? Medical Teacher.
Griffin, B., Carless, S., & Wilson,I. (2013a). The effect of coahing on selection test performance. Medical
Teacher, 35, 295-300.
Griffin, B., Harding, D. W., Wilson, I. G., & Yeomans, N. D. (2008). Does practice make perfect? The effect of
coaching and retesting on selection tests used for admission to an Australian medical school. The Medical
Journal of Australia, 189(5), 270-273
RESULTS
The data analyses aimed to answer a number of key questions:
1. How widespread is commercial coaching by applicants to medical school?
 Slightly more of the high school students (56.2%) attended commercial
coaching compared to those who were actually offered an interview
(between 51 and 53% most years)
 However, rate of coaching attendance are significantly higher (up to 74%)
among students from academically selective high school.
2. Do coached students do better at selection interviews?
 Results since the beginning of this research project in 2006 have consistently
shown that there is NO statistical difference in the overall interview (MMI)
scores of those who have attended commercial coaching and those who have
not.
 Furthermore, on some of the short MMI interview stations, coached students
perform significantly worse than those who had not been coached.
3. What is the UMAT Test?
 The UMAT is published and administered by the Australian Council for
Educational Research (ACER). Refer to http://umat.acer.edu.au/ for more
details.
 UMAT test results are incorporated into the selection decision for most
undergraduate medical degrees in Australia and New Zealand, and for a
number of other health-related degrees in some universities.
 The UMAT is designed to assess three types of cognitive ability: ‘Logical
reasoning’, ‘non-verbal reasoning’ and ‘understanding people’. Our results
showed that scores were related to other validated measures of verbal
reasoning, numerical reasoning, abstract reasoning and emotional
intelligence (‘understanding people’ section only).
4. Does it matter which coaching company?
 There is NO difference in UMAT score according to which coaching company
students attend. In other words, no company appears to be any better than
the others.
5. Do coached students do better at UMAT ‘logical reasoning’ test questions?
 Coaching has NO effect on performance in the logical reasoning component
of the UMAT. In fact, as can be seen in the graph below, students at nonselective public schools performed significantly worse in this part of the
UMAT if they had coaching.
 Note that these results were the same after controlling for ATAR. In other
words, even when we consider students who did equally well in their Higher
School Certificate, coaching still had no effect.
Mean score
Logical reasoning scores
65
60
55
50
45
40
35
30
25
20
Non government schools
Selective Schools
Coached
Public schools (non
selective)
Not coached
6. Do coached students do better at the UMAT ‘understanding people’ test questions?
 Coaching has NO effect on performance in the ‘understanding people’
component of the UMAT. The small difference illustrated in the graph below
were not statistically significant. The analysis also controlled for ATAR scores.
Mean score
Understanding people scores
65
60
55
50
45
40
35
30
25
20
Non government schools
Selective Schools
Coached
Public schools (non
selective)
Not coached
7. Do coached students do better on UMAT ‘non-verbal reasoning’ test questions?
 After controlling for ATAR scores, there was a significant but small positive
effect for coaching on this type of question, but ONLY for those who
attended a government selective high school.
Non-verbal reasoning
70
Mean score
60
50
40
30
20
10
0
Non government schools
Selective schools
Coached
Public school (non selective)
Not Coached
8. Does coaching have a ‘long-term’ influence on performance in medical school?
 A major concern with coaching is that it trains participants to answer test
questions (“test wiseness”) but cannot alter the underlying ability of the
construct the test measures. For example, even if coaching were to result in a
student getting a higher score on an intelligence test, the student’s actual
level of intelligence is likely to remain unchanged. The risk for coached
students is that they will not have the real ability to cope with a demanding
university degree or manage in their later professional life (when no coaching
is available).
 We compared the academic performance of medical students who had
attended UMAT coaching for selection with the performance of those who
had not attended coaching. As can be seen in the graph below, for the first
cohort of students, those who had attended UMAT coaching had POORER
end-of-year exam results for every year of their medical degree.
Furthermore, 60% of those who failed their exams had attended coaching.
This effect was reduced in later years, after the university altered their
selection metrics in a way that significantly reduced the likelihood of an
applicant who had benefited strongly from coaching being selected.
Mean of Exam Grade (standard errors)
80
75
70
65
60
55
50
1
2
3
4
5
Year of Course
Attended coaching
Did not attend coaching
EXPLANATION
Coaching familiarises students with possible test questions and attempts to reach basic rules
and methods for answering questions. We believe the results above may therefore be
explained by:




Interviews: interviewers are trained to identify faked/rehearsed answers and
questions are subtly altered every year so that a rehearsed answer will not be
sufficiently specific to score well.
Logical reasoning test questions: require students to synthesize information and
draw conclusions about facts. The UMAT test developers have endless possibilities
for new questions and coached students risk misapplying rules.
Understanding people questions: there is plenty of evidence beyond our research
supporting the fact that it is difficult to train people to identify emotions and
interpret social information if they do not have the underlying ability.
Non-verbal Reasoning: it may be possible to learn “rules” for solving some of these
items. However, our results indicate this is only likely under conditions of sustained
practice. Anecdotal evidence suggests that when there is a large group of students in
one high school all applying for medicine a “norm” develops that motivates
individuals to engage in many hours of practice. We suspect it is the practice not the
coaching itself that may produce an advantage, albeit a small one, in this section of
the UMAT.
CAUTION
Students need to consider that:
1. The small positive effect of coaching for one third of the UMAT test questions may
be outweighed by the potential for disadvantage on other types of questions.
2. The need for sustained practice of test items distracts them from potentially more
productive HSC study.
3. Universities use the UMAT in different ways and can adjust these to reduce any
advantage of coaching, which may in turn maximise the potential disadvantage of
coaching.
4. Medicine can be a stressful career and those who artificially raise their test scores
risk being a poor fit even if they do succeed at being selected.
BIO
Barbara Griffin B.Psychology (Hons), PhD
Barbara is an organizational psychologist and Associate Professor in the Department of
Psychology at Macquarie University plus Adjunct Associate Professor in the School of
Medicine at the University of Western Sydney. Apart from teaching into the Master of
Organisational Psychology program, she runs a substantial research program investigating
person-environment fit across the employee career cycle, including projects on selection,
incivility, adaptation during career transition, and culture assessment. In each of these she
has addressed issues of national and international importance, obtained significant
competitive research funding, and published in leading academic journals. Barbara has led
the team that developed, and continues to implement, the Multiple Mini Interview for the
UWS School of Medicine, including an interviewer training package for over 150
interviewers each year.