Appraisal of an RCT using a
critical appraisal checklist
Spink MJ, et al. Effectiveness of a
multifaceted podiatry intervention to
prevent falls in community dwelling older
people with disabling foot pain: randomised
controlled trial. BMJ 2011;342:d3411.
Is the included population explicitly
described?
Is the setting explicitly described?
• The RCT should clearly explain how it recruited people, and the
inclusion/exclusion criteria it used, so that you can decide if this is
representative of the target population and generalisable to your practice
• Examples of fair recruitment are: random recruitment/recruitment of
consecutive patients in a treatment centre. However, an intervention study
will need to have informed consent from participants, and so it may use
volunteer recruitment/recruitment from advertising to ensure that it recruits
enough people
• Also remember: inclusion and exclusion criteria should be relevant for the
study method and question — i.e., a study shouldn’t exclude people with
certain weight or height, if this is not relevant to the study outcomes or
question posed
In this study:
Is the included population explicitly described?
•
Look in Methods section for details of recruited participants. (The results section should
detail the actual people randomised from this recruited population group.) Here,
participants were recruited from the treatment centre database and by advertisements.
Inclusion and exclusion criteria are clearly defined and sensible for study method and
question
In this study:
Is the setting explicitly described?
• Look in Methods section in this RCT for details of the setting.
Sometimes this information may be in the results section
…
Is the intervention and control explicitly
described?
• The study should describe the intervention and control in
sufficient detail, such that these can be replicated
• Watch out for statements such as “usual care” — the RCT
should define what this involves
In this study:
Is the intervention and control explicitly described?
• Look in Methods section for details of the intervention and
control
etc…
Are the outcomes explicitly described?
• The study may report on several different outcomes
• A primary outcome is the prespecified outcome that is considered
to be the most important outcome of the study. It is best to have
only one primary outcome in a study
• Secondary outcomes are additional outcomes of interest. However,
the study is often not powered to detect differences between
groups in these outcomes
• All outcomes should be clearly described in sufficient detail,
including the timing of measurement, so that they could be
replicated
In this study:
Are the outcomes explicitly described?
• Yes, look in Methods section. This RCT also refers to another
protocol paper with further information, which you should
also read
…
Are the outcome measurements valid?
(objective, using standard/validated
measures)
• Outcomes can be continuous (vary on a continuum or scale, e.g., weight,
pain score), or dichotomous (yes/no outcomes e.g., death, heart attack)
• Clinically relevant outcomes include outcomes that are important to the
patient, e.g., survival, symptom improvement. Ideally an RCT should
report on clinically relevant outcomes
• Proxy measures are sometimes used because these may be
easier/cheaper to measure, or can be assessed in a shorter time frame,
e.g., intraocular pressure in glaucoma trials, or HbA1c in diabetes trials.
However, proxy measures do not always correlate with clinical outcomes
• Also watch out for:
– Reporting significant outcomes only — an RCT should report on all its prespecified
outcomes whether they are significant or not
– Reporting continuous outcomes on unvalidated scales. It can be difficult to
compare results of these with other studies
In this study:
Are the outcome measurements valid?
• Look again at the outcomes section — particularly the primary
outcomes
• There may be further information in the introduction or
discussion sections if you are not familiar with the topic
…
Is the randomisation method described?
Is the study adequately randomised?
• If a study is successfully randomised, then it should be
impossible to predict in advance which group each person will
be assigned to. The ideal is that everyone has a known and
equal chance of being in either group
• The following allocation methods are not true randomisation
methods, as it may be possible to “guess” which group each
person will be in in advance, so watch out for these!
– Date of birth, day of the week, medical record number, month of the
year, or the order in which they are included in the study (alternate)
In this study:
Is the randomisation method described?
Is the study adequately randomised?
• Look in Methods for details of randomisation used
…
Is the level of blinding reported?
Could the level of blinding affect the interpretation of
the results? (observer bias, bias in analysis, use of
objective or subjective outcome measures, etc)
•
Double-blinding — i.e., both patients and investigators are unaware of treatment
allocation is ideal, and minimises measurement bias. However, sometimes this
may not be possible, for example, where the intervention involves a physical
treatment or surgery or where adverse effects of a drug clearly compromise
blinding. Single-blinding (where either one of the patients or the assessors are
unaware of treatment allocation) may, however, be possible and acceptable in
these circumstances, depending on the study design
•
Also remember — if the outcome is highly objective (e.g., death) then the
requirement for blinding may be less critical, because measurement bias should
not really affect the result. However, if the outcome is subjective (e.g., self-rated
symptoms or function) then blinding of the assessor may be crucial to avoid the
influence of bias on the result
In this study:
Is the level of blinding reported?
• Generally check the Methods section for information on blinding applied;
however, sometimes further clarification may be found in other parts of
the review — e.g., this RCT gives further clarification in the discussion
section
• This RCT was single-blinded (assessor blinding). It was not possible to blind
the patients because of the physical nature of the intervention
…
…
In this study:
Could the level of blinding affect the interpretation of
the results?
• To assess whether the level of blinding affects the interpretation of the
results, look again at the outcomes section, and whether you decided if
the outcomes were objective or subjective
• In this study, the primary outcome of proportion of fallers, multiple fallers,
and falling rate are relatively objective measures and so single blinding
applied should not affect the interpretation of these results. Secondary
outcome measures are more subjective, however, and so level of blinding
applied, particularly no patient blinding, may suffer from bias
Are both groups treated in exactly the same
way apart from the intervention?
• Bear in mind that this includes all analyses, tests, and contact with
the study personnel, as well as any other allowed treatments in the
study
• This may not always be possible/ethical depending on the nature of
the interventions; however, you should decide whether any
differences in treatment during the study could affect the results
In this study:
Are both groups treated in exactly the same
way apart from the intervention?
• Look again at the Methods (description of interventions)
section. You should also look at the results section to check
that both groups had similar assessments and follow-up
Are treatment and control groups similar at baseline?
Are any baseline differences explained/accounted for at
analysis?
• If the randomisation process worked, then the groups should
be similar at baseline (remember that the goal of
randomisation is that everyone should have a similar and
equal chance of being in either group)
• The study should specify whether any differences between
groups are statistically significant, because any significant
differences between groups at the beginning of the study
could potentially affect the results
In this study:
Are treatment and control groups similar at baseline?
Are any baseline differences explained/accounted for at
analysis?
• Yes. Look at the results section. RCTs often contain a table
describing baseline characteristics, as in this case
Are all randomised people accounted for at the end of the RCT?
Does the RCT present an intention-to-treat analysis?
Are final results based on more than 80% of people randomised?
• Most trials will suffer from some loss to follow-up for various reasons
(including patient withdrawal due to safety/efficacy/consent, lost contact,
etc). However, to avoid bias and maintain the validity of the initial
randomisation, loss to follow-up should be minimal — generally less than
20% of the randomised population. However, it should be noted that if
few patients have the outcome of interest, then even small losses can bias
the results
• An intention-to-treat analysis is where the groups are analysed based on
the initial treatment to which they were randomised (not by treatment
eventually administered or whether they finished the trial!). This type of
analysis is generally considered optimal for most RCTs (with the possible
exception of non-inferiority or equivalence trials)
In this study:
Are all randomised people accounted for at the end of the RCT?
Does the RCT present an intention-to-treat analysis?
Are final results based on more than 80% of people randomised?
• Yes. Look at the
Methods and Results
sections (to see what
actually happened).
RCTs often contain a
figure which shows
the flow of people
through the study, as
in this case
Are all clinically relevant outcomes
reported?
• This requires your judgement — look at what outcomes were
reported, and how these answer the question posed by the
study
Does the RCT report a direct statistical analysis between
groups?
Does the RCT report appropriate test statistics?
•
•
•
The RCT should compare similar results between groups directly. Indirect comparisons, for
example, comparison with baseline within a group, are difficult to interpret
The appropriate type of test statistic presented will depend on what type of outcome was
measured. For dichotomous outcomes, the results can be expressed in many ways, including:
Relative Risk (RR) = risk of the outcome in the treatment group / risk of the outcome in the
control group
Absolute Risk Reduction (ARR) = risk of the outcome in the control group – risk of the
outcome in the treatment group
Relative Risk Reduction (RRR) = absolute risk reduction / risk of the outcome in the control
group.
Number Needed to Treat (NNT) = inverse of the ARR and is calculated as 1 / ARR.
The study should present a confidence interval or P value so that you can judge the statistical
significance of the result. However, bear in mind that “statistically significant” and “clinically
significant” are different!
In this study:
Does the RCT report a direct statistical analysis between
groups?
Does the RCT report appropriate test statistics?
• Yes, generally look first at the Methods section and then at
the Results section
…
…
…
In this study:
Does the RCT report on the clinical
relevance/importance of the results?
• Yes, generally find this information in the
discussion section of the RCT
In this study:
Are the setting and intervention generalisable to
routine clinical practice/care?
• Again, this requires your judgement — the study may make
suggestions but you will know best what is generalisable to
your area of practice