Appraising academic publications:
tips for effective reading and writing for
quality and safety professionals
Dr Anne Hogden, Dr Deborah Debono,
Dr Janet Long and A/Prof David Greenfield
Centre for Healthcare Resilience and
Implementation Science (CHRIS)
Australian Institute of Health Innovation
(AIHI)
Presenters
David, Deborah, Anne and Janet
2
Australian Institute of Health Innovation
Our mission is to enhance local, institutional and
international
health
system
decision-making
through evidence; and use systems sciences and
translational approaches to provide innovative,
evidence-based solutions to specified health care
delivery problems.
http://aihi.mq.edu.au/
3
Purpose of the webinar
Build on previous presentations given to ISQua
Fellowship:
• Previous webinars - January 2015 and May 2015
• Seminar at 2015 ISQua conference: “Healthcare research for quality
and safety professionals: study design, implementation and
translation.”
Appraising publications is a useful skill to assist you
plan and conduct your work:
• Reading to inform your work or conducting evaluations and research
• Writing up or presenting your project
• Reaching a high standard for publishing your manuscript in peerreviewed journals
• Acting as a reviewer for peer-reviewed journals
4
Agenda
Analyse the overall coherence and
cohesion of publications
David
Appraise the quality and rigour of
each individual section
Anne
Identify limitations and how they
could be improved
Deborah
Apply this knowledge to improve
your own research work and writing
Janet
5
Analysing coherence and
cohesion of the publication
Three approaches you can use, depending on your time
constraints.
Each approach is directed at taking a “big picture” look and
answering several basic questions:
1. Does the aim match the method, leading to the
findings and conclusion?
2. Do you find it credible?
3. Are you convinced?
6
Analysing coherence and
cohesion of the publication
Approach #1
Review the Abstract
Questions to ask:
 Does the aim match the method, leading to the findings
and conclusion?
 Do you find it credible?
 Are you convinced?
7
Analysing coherence and
cohesion of the publication
Approach #2
Examine the Introduction + Discussion and/or Conclusion
Additional questions to ask:
 Are both sections grounded in the relevant, current
literature?
 Is there a clear aim or objective that is answered in the
discussion and conclusion?
8
Analysing coherence and
cohesion of the publication
Approach #3
Consider the whole paper from beginning to end, and back
and forth at the same time
Additional questions to ask:
 Is the paper well structured, written and easy to read?
 Are all sections grounded in the relevant, current
literature?
 Is there a clear aim/objective that is answered in the
discussion and conclusion?
 Do sub-sections add to addressing the research aim?
9
Appraising the quality and
rigour of each section
RATS guidelines:
http://www.biomedcentral.com/authors/rats
Relevance
Appropriateness of method
Transparency of procedures
Soundness of interpretive approach
Reference: Clark, JP. How to peer review a qualitative manuscript. In Peer Review in Health
Sciences. Second edition. Edited by Godlee F, Jefferson T. London: BMJ Books; 2003:219-235
10
Appraising the quality and
rigour of each section
Relevance of study question
ASK THIS OF THE MANUSCRIPT
1. Is the research question interesting?
2. Is the research question relevant to
clinical practice, public health, or
policy?
THIS SHOULD BE INCLUDED IN THE
MANUSCRIPT
• Research question is explicitly stated
• Research question is justified and linked
to the existing knowledge base
(empirical research, theory, policy)
11
Appraising the quality and
rigour of each section
Appropriateness of method
QUALITATIVE STUDIES
Is qualitative methodology the best approach for
the study aims?
• Interviews: experience, perceptions, behaviour,
practice, process
• Focus groups: group dynamics, convenience,
non-sensitive topics
• Ethnography: culture, organizational behaviour,
interaction
• Textual analysis: documents
Study design is described and
justified,
i.e. why was a particular method
chosen?
12
Appraising the quality and
rigour of each section
Appropriateness of method
QUANTITATIVE STUDIES
Is quantitative methodology the best approach for Study design described and
the study aims?
justified, i.e. why was a particular
• Large scale surveys
method chosen?
• Pre-and post-test design
• Longitudinal studies
• Cross-sectional studies
13
Appraising the quality and
rigour of each section
Transparency of procedures
SAMPLING
Are the participants selected the most
appropriate to provide access to the type
of knowledge sought by the study?
Is the sample adequate?
RECRUITMENT
Was the recruitment conducted using
appropriate methods?
Is the sampling strategy appropriate?
Could there be selection bias?
Criteria for selecting the study sample
justified and explained
• theoretical: based on preconceived or
emergent theory
• purposive: diversity of opinion
• volunteer: feasibility, hard-to-reach
groups
Details of how recruitment was conducted
and by whom
Details of who chose not to participate
and why
14
Appraising the quality and
rigour of each section
Transparency of procedures
DATA COLLECTION
Was collection of data systematic and
comprehensive?
Are characteristics of the study group and
setting clear?
Why and when was data collection
stopped, and is this reasonable?
Method(s) outlined and examples given
(e.g., interview questions)
Study group and setting clearly described
End of data collection justified and
described
15
Appraising the quality and
rigour of each section
Transparency of procedures
ROLE OF RESEARCHERS
Is the researcher(s) appropriate?
How might they bias (good and bad) the
conduct of the study, and the results?
Do the researchers occupy dual roles
(clinician and researcher)
Are the ethics of this discussed?
Do the researchers critically examine their
own influence on the formulation of the
research question, data collection, and
interpretation of results?
16
Appraising the quality and
rigour of each section
Transparency of procedures
ETHICS
Was informed consent sought and
granted?
Were participants’ anonymity and
confidentiality ensured?
Was approval from received from an
appropriate ethics committee?
Informed consent process explicitly and
clearly detailed
Anonymity and confidentiality discussed
Ethics approval cited
17
Appraising the quality and
rigour of each section
Soundness of interpretive approach
ANALYSIS
Is the type of analysis appropriate for the
type of study? eg
• Thematic: exploratory, descriptive,
hypothesis generating
• Constant comparison/grounded theory:
theory generating, analytical
Are the interpretations clearly presented
and adequately supported by the evidence?
Was trustworthiness/reliability of the data
and interpretations checked?
Analytic approach is described in-depth and
justified
Indicators of quality: Description of how
themes were derived from the data
(inductive or deductive)
Evidence of alternative explanations being
sought
Analysis and presentation of negative or
deviant cases
Method of reliability check described and
justified, e.g. was an audit trail, triangulation,
or member checking employed? Did an
independent analyst review data and contest
18
themes?
Appraising the quality and
rigour of each section
Soundness of interpretive approach
DISCUSSION AND PRESENTATION
Are findings sufficiently grounded in
empirical, theoretical or conceptual
framework?
Findings presented with reference to existing
theoretical and empirical literature, and how
they contribute
Is adequate account taken of previous
knowledge and what the findings add?
Are the limitations thoughtfully considered? Strengths and limitations explicitly described
and discussed
Is the manuscript well written and
Written for a health sciences audience
accessible?
19
Identifying limitations and
how they could be improved
•
What are study limitations?
•
What are not study limitations?
•
Why is it important to identify study limitations?
Strengths
Limitations
20
Identifying limitations and
how they could be improved
Types of limitations
•
Theoretical limitations
•
Methodological limitations
 Study design limitations
 Data limitations
21
Identifying limitations and
how they could be improved
Presenting study limitations
•
Are they described at all?
•
How well are they described?
 Critical reflection
 Evaluation
22
Identifying limitations and
how they could be improved
The implications of study limitations
•
Identifying areas for additional research
•
Conducting future studies
23
Applying this knowledge to
improve your research and writing
Relevance


Target group
Audience: journal articles, reports,
conference presentations, study
partners, industry
Appropriateness


Best method to answer the question
Give reasons for your choices
Transparency


Keep a log
What does the report look like?
Soundness


Results only
Discussion
24
Applying this knowledge to
improve your research and writing

Report statistics using APA style guide
E.g. http://my.ilstu.edu/~jhkahn/apastats.html
The sample as a whole was relatively young (M = 19.22, SD = 3.45).
The average age of students was 19.22 years (SD = 3.45).
The percentage of participants that were married did not differ by
gender, C2(1, N = 90) = 0.89, p = .35
There was a significant main effect for treatment, F(1, 145) = 5.43, p =
.02, and a significant interaction, F(2, 145) = 3.24, p = .04.
25
Applying this knowledge to
improve your research and writing
• Draft your output
• Check back on RATS to
ensure all the criteria are
addressed
• Authors’ guidelines
• Revise, review and
rewrite
• Critical review partners
• Big picture
“Great work is not written it is
re-written”
26
Conclusions
Effective reading and writing of academic publications, by:
• analysing the overall coherence and cohesion
publications
• appraising the quality and rigour of each individual
section
• identifying limitations and improvements
• applying this knowledge to your own research work and
writing
27
Contact details
Centre for Healthcare Resilience and Implementation
Science, Australian Institute of Health Innovation,
Macquarie University
Dr Anne Hogden
Dr Deborah Debono
Dr Janet Long
A/Prof David Greenfield
anne.hogden@mq.edu.au
deborah.debono@mq.edu.au
janet.long@mq.edu.au
david.greenfield@mq.edu.au
28
Australian Institute of Health Innovation
The Australian Institute of Health Innovation (AIHI) comprises three
centres:
• Centre for Healthcare Resilience and Implementation Science
(CHRIS),
• Centre for Health Informatics (CHI), and
• Centre for Health Systems and Safety Research (CHSSR).
We are a major national resource for many people whose work is
intrinsic to strengthening health systems, organisations and services.
We strive to conduct world-class research with relevance to national
and international research communities, governments, policymakers,
providers of health services, managers, clinicians, patients and the
community.
http://aihi.mq.edu.au/
29
Australian Institute of Health Innovation
Our work underpins health reforms and systems
improvement and provides new tools, perspectives and
evidence to help stakeholders who are interested in
making the health system more effective, efficient and
productive.
Our overarching aim is to produce new, high quality
research evidence to support change and improvement.
We are funded by the National Health and Medical
Research Council, the Australian Research Council,
NSW Health and many other supporters, partners and
stakeholders.
30
31
Centre For Healthcare Resilience
and Implementation Science
The Centre for Healthcare Resilience and Implementation Science
(CHRIS), formerly the Centre for Clinical Governance Research, has a
successful track record of undertaking research and evaluation
projects on health sector issues since 1991.
Our core interest is to investigate issues of policy, governance and
leadership in the health sector.
We are involved in conducting original research providing a scholarly
capability by which to evaluate health sector policies, programs and
projects, and contributing to undergraduate medical, postgraduate
health services management, and public health and al education.
Website:
http://aihi.mq.edu.au/chris
32