STEP 3 - QUANTITATIVE CRITICAL APPRAISAL TOOLS
A PROCESS INSTRUCTION CHART FOR SPECIALISTS AND MANAGERS
Introduction
This process chart for Quantitative Critical Appraisal Tools has been designed to
assist Specialists and Managers/Supervisors within Peel Public Health in
choosing the appropriate critical appraisal tool to assess research literature.
This document describes the hierarchy of evidence, provides recommended
critical appraisal tools and describes the process for starting your critical
appraisal of the evidence. This document does not describe how to critically
appraise grey literature. Critical appraisal of qualitative literature is described
under the “Qualitative Critical Appraisal Tools – A Process Instruction Chart for
Specialists and Managers”.
There are many levels of evidence as shown in Figure 1 entitled the “Hierarchy of
Evidence”. Ideally, it is best to obtain evidence from summaries, synopses of
syntheses, and syntheses. Sometimes, however, there is only evidence available
from either single studies or from synopses of single studies. Searching through
the literature for this type of evidence can be time consuming. The stop sign
shown in this hierarchy of evidence means that staff should consult with their
Manager/Supervisor and the Librarian to discuss whether it is appropriate to
conduct a search and evaluate the research evidence at this level.
It should be noted that research may be published in peer reviewed journals or it
may be found on websites. Some research will have very strong methodological
requirements for document review and some will not.
What is the process for Critical Appraisal?
 Using the literature search you conducted in Step 2 – Searching the
Evidence, obtain the full text articles from your search.
 Discuss the result of the literature search with your Manager/Supervisor to
determine whether you have the right number of studies.
 Critically appraise the research studies starting with the most recent article
by following the instructions in this document – Step 3. Please note, after
you critically appraise your studies, you will be required to provide the
completed critical appraisal documents to your Manager/Supervisor along
with the other documents described in Steps 4 to 6.
 Create a data extraction table as outlined in Step 4 – Data Extraction for
Systematic Reviews using the sample table as a guide. Use RefWorks or
other software if possible. This table is required for the final 1:2:20 Report
– Step 6 – Literature Review Report Template.
Note to Managers/Supervisors: Critical appraisal requires reading the
article at least twice. Please ensure enough time is allowed for this
process.
Figure 1
Hierarchy of Evidence
Start here with
clinical query
Systems
Summaries
Synopses of Synthesis
Synthesis
Synopses of Single Studies
Single Studies
HIERARCHY OF EVIDENCE
SYSTEM
A system provides public health professionals with information to guide an
intervention or action. It can also be defined as a decision support service that
matches information from individual patients with the best evidence from
research that applies (Haynes, 2007).
Examples of systems includes practice guidelines, decision pathways, or
evidence based summaries of a public health practice (NCCMT – compendium
document)
Example of source: Computerized decision support systems (CDSS)
What do I Use to Critically Appraise?
The critical appraisal of systems is not required for Public Health since no such
system currently exists for the public health setting.
SUMMARY
A summary includes clinical pathways or textbook summaries that integrate
evidence-based information about specific clinical problems and provide regular
updating.
Examples of sources include: Clinical Practice Guidelines and Evidence-Based
Textbooks
Tools for Critical Appraisal
AGREE: http://www.agreecollaboration.org/pdf/agreeinstrumentfinal.pdf
SIGN: http://sign.ac.uk/guidelines/fulltext/50/Index.html
SYNOPSIS OF SYNTHESIS
A synopsis of syntheses is also referred to as a summary of systematic reviews.
This type of systematic review summarizes the systematic review with key
methodologic details and results required to apply the evidence (Haynes, 2001)
Examples of sources include: Healthevidence.ca, Evidence-based abstract
journals, CDC ranked sites
Tools for Critical Appraisal
CASP: http://www.sph.nhs.uk/sph-files/S.Reviews%20Appraisal%20Tool.pdf
SYNTHESIS OR SYSTEMATIC REVIEW
A synthesis, also referred to as a systematic review is a systematic consolidation
of the research evidence on a specific topic. It provides public health
professionals with all the evidence that addresses a focused public health
question (NCCMT – compendium document)
Examples of sources of systematic reviews include: Cochrane Library Reviews,
Campbell Collaboration Reviews
Critical Appraisal Tools
CASP: http://www.sph.nhs.uk/sph-files/S.Reviews%20Appraisal%20Tool.pdf
AMSTAR: http://www.biomedcentral.com/1471-2288/7/10
(tool can be downloaded from this article)
Health-Evidence.ca: http://www.healthevidence.ca/downloads/QA%20tool_Doc%204.pdf and Dictionary to go with the
tool: http://www.healthevidence.ca/downloads/QA%20tool%20dictionary_Doc%205.pdf
SYNOPSES (SUMMARIES) OF SINGLE STUDIES
NOTE! The critical appraisal of single studies should only be conducted under
the following circumstances:
 When a new single study has come out and is adding important
information to an existing review that has not been updated with the new
study or evidence.
 When there is not enough or no evidence available from a higher level of
the hierarchy of evidence (i.e., no available or no relevant summaries,
synopses of synthesis or synthesis on your PICO and you have deemed it
appropriate to search for single studies).
A synopses or a summary of a single study provides a brief summary of
individual studies with key methodologic details and results required to apply the
evidence (Haynes, 2001 and NCCMT compendium document).
Examples of sources include: Evidence-based abstract journals
Tools for Critical Appraisal
Randomized Control Trials
CASP: http://www.sph.nhs.uk/sph-files/rct%20appraisal%20tool.pdf
Cohort Studies
CASP: http://www.sph.nhs.uk/sph-files/cohort%2012%20questions.pdf
Case Control Studies
CASP: http://www.sph.nhs.uk/sph-files/Case%20Control%2011%20Questions.pdf
SINGLE STUDIES
NOTE! The critical appraisal of single studies should only be conducted under
the following circumstances:
 When a new single study has come out and is adding important
information to an existing review that has not been updated with the new
study or evidence.
 When there is not enough or no evidence available from a higher level of
the hierarchy of evidence (i.e., no available or no relevant summaries,
synopses of synthesis or synthesis on your PICO and you have deemed it
appropriate to search for single studies).
A single study article describes the methods, analysis and results for one
individual study. Single studies contribute to clinical decisions if no higher level of
preprocessed evidence is available.
Examples of sources for single studies include: PubMed clinical queries, CINAHL
clinical queries
Critical Appraisal Tools
Randomized Control Trials
CASP: http://www.sph.nhs.uk/sph-files/rct%20appraisal%20tool.pdf
Cohort Studies
CASP: http://www.sph.nhs.uk/sph-files/cohort%2012%20questions.pdf
Case Control Studies
CASP: http://www.sph.nhs.uk/sph-files/Case%20Control%2011%20Questions.pdf