1
1,2
2
2
1
2
• Clinicians rely on summary estimates from systematic reviews (SR) and metaanalyses for clinical decision making.
• Interpreting a treatment’s effects – large or small – can be difficult.
• When pooling results of trials, authors of SR report the differences between intervention and control groups in standard deviation units of Standardized Mean
Difference (SMD).
• Presenting results as SMD is the longest standing and most widely used approach.
• SMD values 0.20 represents a small difference, 0.50 represents a moderate difference and 0.80 represents a large difference.
• To determine clinicians’ understanding and perceptions of 6 different approaches to the presentation of continuous outcome data in meta-analyses.
• The six include SMD, Minimal Important Difference Units, Natural Units, Relative
Risk, Risk Difference and Ratio of Means
Survey design: Participants received paper-based, self-administered surveys presenting summary estimates of a hypothetical intervention versus placebo for chronic pain, with estimates demonstrating either a small effect or large effect for each of the 6 presentation approaches. We asked 6 questions addressing understanding and 6 questions addressing preferences. We randomized participants to size of effect and order.
Participants : As part of a larger multicenter international study, we invited 66 staff, residents, and trainees in family medicine (FM) and internal medicine (IM) academic programs at University of Missouri-Kansas City (UMKC) to participate, 41 FM resident and staff and 25 IM residents.
We evaluated 6 common presentation approaches found in systematic reviews and meta-analyses.
(1) Standard deviation units – standardized mean difference (SMD)
(2) Minimal important difference units (MID)
(3) Conversion into natural units of the most commonly used instrument.
(4) Conversion to relative effects, calculation of relative effects (e.g. relative risk)
(5) Conversion to absolute effects, calculation of absolute effects (e.g. risk difference and the corresponding number needed to treat)
(6) Ratio of means or ratio of change - ratio of means (RoM) or ratio of change
(RoC) between the intervention and control groups
• 63 clinicians responded (39 FM and 24 IM), all of which provided completed surveys (95.5% response rate)
• Data analysis entailed calculating proportion of participants answering correctly for small and large effects, along with number of participants who favored each approach. 95% Confidence Interval was constructed for each entity.
• Risk Difference was the approach best understood by clinicians, followed by the
Ratio of Means and Relative Risk (Table 2).
• Clinicians generally found dichotomous presentation of continuous outcomes
(Relative Risk; Risk Difference) very useful, and other approaches less useful
(Table 3).
Table 1. Demographic characteristics of respondents , n = 63
Characteristic N (%)
Gender Male
Female
Specialty Internal Medicine
Family Medicine
Professional status Staff
Trainee
32
31
24
39
6
57
(50.8%)
(49.2%)
(38.1%)
(61.9%)
(9.6%)
(90.4%)
Year graduated from medical school
Before 1990
1990-1999
2000-2009
2010and after
3
1
12
47
Training in Health
Research Methods or
Epidemiology
Never completed a formal course
Completed a formal course but no degree
Have master/PhD degree in HRM
47
16
0
Table 2. Understanding of the Presentation of Continuous Outcomes, n = 63
(4.8%)
(1.6%)
(19.0%)
(74.6%)
(74.6%)
(25.4%)
(0%)
Approach N (%) correct (95% CI)
SMD
MID
Natural units
16 (25.4%)
9 (14.3%)
10 (15.9%)
(16.3% - 37.3%)
[7.71% - 25.0%]
[8.85% - 26.8%]
Relative Risk
Risk Difference
Ratio of Means
21 (33.3%)
24 (38.1%)
22 (34.9%
[22.9% - 45.6%]
[27.1% - 50.5%]
[24.3% - 47.3%]
Table 3. Perceived Usefulness of the Presentation of Continuous Outcomes, n = 63
Approach
SMD
Mean (SD)
2.95 (1.43)
(95%CI)
[2.60 – 3.30]
MID
Natural units
Relative Risk
Risk Difference
Ratio of Means
2.92 (1.53)
3.75 (1.61)
4.16 (1.69)
4.30 (1.73)
3.95 (1.61)
[2.55 – 3.30]
[3.35 - 4.15]
[3.74 - 4.58]
[3.87 – 4.72]
[3.55 - 4.35]
1-7 scale with higher numbers indicating higher preference
• UMKC clinicians best understood continuous outcomes when presented as dichotomies (relative and absolute risk differences) and also found these presentations most useful.
• Presenting results as SMD, the longest standing and most widely used approach, was poorly understood and not perceived as useful.
Strengths:
• This is one of the first studies to survey physicians about their understanding and perceived usefulness of different continuous outcome presentations.
• We used both perceived usefulness and percent of correct answer
• We surveyed both internal medicine and family medicine physicians
• We achieved an excellent response rate from the physicians surveyed
Limitations:
• Data is from a single center, which may represent a biased view. However, the preliminary results from other centers support the same findings.
• There may have been a learning effect. However, we did not observe any systematic differences in responses based on forms with different orders.
• Surveys were distributed in an academic institution, so the results may not be generalized to practicing physicians outside academia.
Funding: None
We thank all participant clinicians who answered our survey
References: Cochran Review Handbook 2011