What are examiners thinking when
they examine a thesis?
Margaret Kiley
CEDAM, The Australian National University
Margaret.kiley@anu.edu.au
Limits of this workshop…
• This is ‘just’ the theory
• To get the practice in your own university you need to
contact your Graduate Studies authority
• This workshop does not specifically focus on doctorates in
art, design and performance where an exhibition or
equivalent plus exegesis is the norm
In pairs/small groups discuss
• One thing I don’t really understand about the examination
process is…
• One of my biggest fears about the examination process
is…
• Something I am hoping to learn from today’s session is…
• Something I hope to contribute to today’s session is…
Preparing for Examination
• What are your University’s expectations of a PhD or
Masters thesis? What is the difference?
• Who selects examiners? Does the candidate have a say?
• Can a candidate include published work? If so, how?
• Is there a word limit?
• What happens re multi-disciplinary research?
• How strategic can supervisors and candidates be with the
selection of examiners e.g. is it a good idea to appoint an
examiner who might be a future employer?
• What are the University’s rules on conflict of interest
between candidate, examiner, and supervisor?
Overview
• Research re the selection of examiners
• How examiners examine theses
• Discussion
Selection of Examination (Kiley, 2009)
• Recent research suggests that the selection of
examiners varies depending on the perceived quality of
the candidate and the dissertation.
• Experienced supervisors report that there are two sorts
of criteria that they use:
• Professional / academic
• Personality attributes.
• The aim is to find examiners who are knowledgeable in
the area and who will give a fair and balanced opinion.
Professional / academic considerations
• Topic/methodology fit
• Understanding of the system: country, type of doctorate
(professional doctorate, exegesis)
• Examiners outside the academy
• Experience versus inexperience (avoid retirees with too
much time on their hands)
• Availability
Personality Issues
• High standards but fair (avoid the pedants)
• Intellectual courtesy and respect (avoid the Smart Alecs)
• Reliability (avoid the ‘gunnas’)
Experienced Examiners report that they…
(Mullins & Kiley, 2002)
• Expect the student to pass as they open the thesis
• Are very reluctant to fail a student with most
experiencing considerable distress if they do so
• Come to a decision about the quality of a PhD by about
the end of Chapter 2
• Have a formative rather than summative view of thesis
examination
• Believe that there is a risk attached to sending theses
to inexperienced examiners
Experienced Examiners appear to…
• Be fiercely independent in their views
• Hold varying views about the purpose of the PhD. (Is it
the thesis or the student being examined?)
• Consider professional duty as the main reason for
examining, followed by the fact that they are going to
be needing examiners for their own students!
• Devote considerable time to examining each thesis
• Have surprisingly inclusive approaches to
methodology/ paradigm
• Demonstrate few discipline differences in their
responses, other than regarding publications
Inexperienced Examiners
(Kiley & Mullins, 2004)
• Report a high level of confidence in their ability to
examine (which is not always reflected in what they
say in response to other questions)
• Suggest experience comes from supervising &
examining Honours students and theses and
reviewing manuscripts
• Adopt a similar approach to the process of examining
as more experienced colleagues, although they are
more likely to focus on the components of a PhD
rather than the whole
Inexperienced Examiners …
• At a surprisingly high rate, wanted to fail their first
thesis or said it was ‘awful’
• Follow institutional criteria more than experienced
colleagues
• Felt (some of them) they were being examined too
• Suggest a main difficulty is inability to benchmark
• See role as maintaining standards and performing
their summative assessment role ‘correctly’
Is there a difference?
• From work of Trafford (2003) from 130 vivas it was
possible to determine that:
• Experienced examiners tended to ask questions
that can be defined as ‘Defending doctorateness,
contributing to knowledge, critique of research,
synthesizing concept’
• Inexperienced examiners tended to ask more
‘technical’ questions
Trafford’s Categorisation
C. Questions generally
related to issues such as
research question, choice
of topics, location of study
D. Defending doctorateness,
contributing to knowledge,
critique of research,
synthesizing concept
A. Types of questions
include resolving research
problems, content,
structure
B. Implications,
awareness of, and
familiarity with wider
literature
LOW
HIGH
Scholarship & Interpretation
LOW
Innovation and Development
HIGH
Strategies for examining
• There is no one way, but most:
• Begin by reading the Abstract, Acknowledgements,
Introduction & Conclusion to gauge the scope of
the work and whether what candidates say they are
going to do is actually done
• Look at the references to see what sources have
been used and whether they need to follow up on
any of them, and to see if they have been
referenced
• Read from cover to cover taking detailed notes,
finally go back over the thesis to check whether
their questions have been answered or whether
their criticisms are justified
Tip
Strategies for examining
• There is no one way, but most:
• Begin by reading the Abstract, Acknowledgements,
Introduction & Conclusion to gauge the scope of
the work and whether what candidates say they are
going to do is actually done
• Look at the references to see what sources have
been used and whether they need to follow up on
any of them, and to see if they have been
referenced
• Read from cover to cover taking detailed notes,
finally go back over the thesis to check whether
their questions have been answered or whether
their criticisms are justified
Analysis of reports demonstrates that…
(Kiley, 2004)
A ‘good’ thesis has:
• Critical analysis & argument
• Confidence & a rigorous, selfcritical approach
• A contribution to knowledge
• Originality, creativity & a degree
of risk taking
• Comprehensiveness &
scholarly approach
• Sound presentation & structure
• Sound methodology
A ‘less than ideal’ thesis has:
• Too much detail with lack of
analysis
• Lack of confidence, energy &
engagement by the candidate
• Lack of argument and rigour
• Shoddy presentation (typos
etc)
• Lack of critique of own
analysis/ sweeping
generalisations based on
opinion rather than analysis
• Inadequate or poorly
expressed methodology &
scope
A very experienced supervisor (Lees, 2008)
 Failing a PhD is an awful experience….It can take
months, and many re-readings, to finally come to that
decision.
 On the other hand…
• “It’s a delight to deal with a thesis which has a
clear, concise, research question; a simple
structure; a research design which was decided
upon before the data was gathered; a literature
review which relates to the topic; and a logical
progression from ‘Results’ through ‘Discussion’
to ‘Conclusion’. These get marked briskly!”
Discussion and Questions
• In small groups list the main qualities of a ‘good’ doctoral
thesis
• Through discussion then decide which are the top three
qualities and why
• QUESTIONS?
Resources
Kiley, M. (2009). You don't want a smart alec: Selecting examiners of doctoral
dissertations Studies in Higher Education 34(8)
Kiley, M. (2004). What examiners' comments can tell us about the postgraduate learning
environment. In C. Rust (Ed.), Improving student learning: Theory, research and
scholarship (pp. 213-222). Hinckley, Leicestershire: The Oxford Centre for Staff and
Learning Development.
Kiley, M., & Mullins, G. (2004). Examining the examiners: How inexperienced examiners
approach the assessment of research theses. International Journal of Educational
Research, 41(2), 121-135
Lees, B. (2007). Theses I have marked. Presentation at the Summer ACT InterUniversity research workshop, ANU.
Mullins, G. & Kiley, M. (2002) ‘It’s a PhD, not a Nobel Prize’: How experienced examiners
assess research theses. Studies in Higher Education 27(4) pp. 370-386
Research Supervision@ANU http://researchsuper.cedam.anu.edu.au
SORTI web site at the University of Newcastle has information on examining theses,
especially in the performing/visual arts
http://www.newcastle.edu.au/centre/sorti/publications.html
Trafford, V. (2003) Questions in doctoral vivas: Views from the inside, Quality Assurance
in Education 11(2) pp 114-122