Dr Samira Alsenany
Dr SA 2012
Purpose of this seminar
 The purpose of this seminar is to provide a brief
overview of the critical appraisal process.
 A critical review must identify the strengths and
weaknesses in a piece of research and this should
be carried out in a systematic manner (Eachus,
2003)
Dr SA 2012
Critical Appraisal
 “Critical appraisal is the process of systematically
examining research evidence to assess its validity,
results, and relevance before using it to inform a
decision” (Hill and Spittlehouse, 2001, p.1).
Dr SA 2012
Critical Appraisal
Critical appraisal is an essential step in the process of
putting research into practice. Asking questions about
an article’s research methodology, scrutinizing its data
collection and analysis methods, and evaluating how
its findings are presented will help you to determine
whether that article’s conclusions should influence
practical decision-making.
Dr SA 2012
When should you Critically
Appraise?
• Conducting literature reviews for grant proposals.
• Evaluating the effectiveness, costs, and benefits of
health programs.
• Establishing new health programs.
• Implementing policies; and Public health decision
making, especially at the senior management level.
Dr SA 2012
What is critical
appraisal?
 Critical appraisal is an essential part of evidence-based.
 Critical appraisal allows us to make sense of research
evidence and thus begins to close the gap between
research and practice.
 The most appropriate design for studying the
effectiveness of a specific intervention or treatment.
Dr SA 2012
Appraising systematic
reviews
 Once you have conducted a literature search and
obtained full text articles, you can begin the critical
appraisal process.
 Consider the following questions to assess the quality
of the article
Dr SA 2012
Questions to Ask
 What kind of reports do I want?
 Is this article relevant to my issue and setting?
 How much detail do I need?
 How comprehensive do I need to be?
 How far back should I search?
Dr SA 2012
Questions to Ask
 Is it of interest?
 Why was it done?
 How was it done?
 What has been found?
 What are the implications?
 What else is of interest?
Dr SA 2012
Critically Appraise What You Read.
Simple checklists enable the useful information
to be identified.
Dr SA 2012
Checklists
 Checklists for particular types of literature are a
quick and easy way of learning critical appraisal.
 They all have 3 stages:
 Basic questions.
 Essential appraisal.
 Detailed appraisal
Dr SA 2012
Critical Appraisal: Notes and Checklists
Available at:
http://www.sign.ac.uk/methodology/checklists.html
Scottish Guidelines Network (SIGN) has developed a number
of critical appraisal tools in support of their creation of
evidence-based guidelines. These appraisal tools used by
SIGN to conduct systematic reviews have been made
available for public use on their website, including
methodology evaluation checklists for each of the
following 6 types of studies: systematic reviews/meta
analyses, RCTs, cohort studies, case-control studies,
diagnostic studies, and economic evaluations.
Dr SA 2012
Questions to Ask
The abstract
Use the information found in the abstract to answer the
questions below:
• Are your issues discussed there?
• What are the main findings of the research?
• Do you want to know more after reading the abstract?
• Was the research done in a similar setting to yours?
• Does it address a related question? (Even research that
covers your issue indirectly can be useful.)
• Are there reasons to doubt the findings without reading the
whole article?
Dr SA 2012
Questions to Ask
 You may conclude that the study is not reliable merely
by reading the abstract.
 If this is the case, move on to another article! Critical
Dr SA 2012
Questions to Ask
 Why was it done?
Introduction
Should a clear statement of the purpose of the study.
 Focus on the time, place and circumstances of the population and
setting studied.
 How similar or different is the study population or setting to yours? Is a
difference likely to matter for the issue at hand?
 The absence of such statements can imply that the authors had no
clear idea of what they were trying to find out.
Or they didn’t find anything but wanted to
publish!
Dr SA 2012
Questions to Ask
Methods
 How was it done?




Brief but should include enough detail to
enable one to judge quality.
Must include who was studied and how they
were recruited.
Basic demographics must be there.
How the data was collected (interviews,
focus groups, questionnaires, surveys,
observations, etc.).
Dr SA 2012
Questions to Ask
Results
 Three broad issues need to be considered the results of the
study :
 A Are the results of the study valid?
 B What are the results?
 C Will the results help locally
 The major challenge.
 What do I think this really means?
Dr SA 2012
Questions to Ask
Results
 What has it found?
 The data should be there statistics.
 Is the sample large enough to produce significant results? Is the
effect size clinically or operationally relevant?
 Are the aims in the introduction addressed in
the results?
 Look for illogical sequences, bland
statements of results.
 Flaws and inconsistencies.
 All research has some flaws , the impact of
the flaws need to assessed.
Dr SA 2012
Questions to Ask
How good are the measures?
• Do the measures accurately reflect what the researcher
was trying to measure
(validity)?
• How clear and appropriate are these measures? Too
broad? Too narrow?
• Are they actual measures or proxy measures?
• Are the measures well established in either prior
research or through pilot testing?
Dr SA 2012
Questions to Ask
Discussion
 The discussion section is more detailed and precise
than the abstract, and will explain:
 A review of how these results compare or contrast with
prior research? If this report found something
different from.
 The limitations of the research and possible
implications which are not mentioned in
the abstract.
Dr SA 2012
Questions to Ask
implications
 What are the implications?
 Abstract / discussion.





The whole use of research is how far the results can be
generalised.
All authors will tend to think their work is more important
than the rest of us!
What is new here?
What does it mean for health care?
Is it relevant to my speciality?
Dr SA 2012
Recommended Reading
 The following 3 reading series were published in core medical
journals and contain in-depth
 information on critical appraisal and evidence-based practice.
From the British Medical Journal (BMJ):
 How to Read a Paper: The Basics of Evidence-Based Medicine
Available at: http://www.bmj.com/collections/read.dtl
 Evidence-Based Nursing Notebook
Available at: http://ebn.bmj.com/cgi/collection/notebook
From the Journal of the American Medical Association (JAMA):
User Guides to Medical Literature
 Available at: http://www.shef.ac.uk/scharr/ir/userg.html
Dr SA 2012
Online Critical Appraisal Tools
The Centre for Methods and Tools (http://www.nccmt.ca/) is currently developing a critical
appraisal framework that is specifically tailored to
public health research. This guide will be the definitive critical appraisal resource for public
health practitioners in Canada.
• AGREE Instrument
Available at: http://www.agreetrust.org/instrument.htm
The purpose of the Appraisal of Guidelines Research & Evaluation (AGREE) Instrument
is to provide a framework for assessing the quality of clinical practice guidelines. It
originates from an international collaboration of researchers and policy makers who work
together to improve the quality and effectiveness of clinical practice guidelines by
establishing a shared framework for their development, reporting and assessment.
• A Beginner’s Guide to Judging Research Studies
Available at http://www.cihr-irsc.gc.ca/e/34192.html
This is a succinct guide to critical appraisal written by John Frank, Scientific Director at
the Canadian Institutes of Health Research. Although this article focuses on making
sense of media reports of new health research, the astute questions Frank poses can be
applied directly to research studies themselves.
Dr SA 2012
Online Critical Appraisal Tools
• CASP Critical Appraisal Tools
Available at: http://www.phru.nhs.uk/Pages/PHD/resources.htm
The UK-based Critical Appraisal Skills Programme (CASP), a division of the
NHS’s Public Health Resources Unit, has developed a customized critical
appraisal checklist for each common type of research study. Direct links to each
of the 6 appraisal tools are
listed below:
• Qualitative Research
http://www.chsrf.ca/kte_docs/casp_qualitative_tool.pdf
• Review Articles (including Systematic Reviews)
http://www.phru.nhs.uk/Doc_Links/S.Reviews%20Appraisal%20Tool.pdf
• Case Control Studies
http://www.phru.nhs.uk/Doc_Links/Case%20Control%2011%20Questions.pdf
Dr SA 2012
Thanks
Dr SA 2012