Chapter 13
Evaluation and Diagnosis:
Research Methods and Data
Analysis
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Chapter Overview
• Health care providers use a variety of diagnostic procedures, including
laboratory testing, diagnostic imaging, physical examination,
interviewing, and observation.
• Patient interviews and the performance of physical examination
procedures are the primary tools available to those evaluating an
individual seeking care.
• Overview of the diagnostic continuum and the differences between
research into diagnostic testing and prevention or treatment strategies.
• Define and decrease investigational bias.
• Concepts of sensitivity and specificity.
• Calculation of sensitivity and specificity values, and likelihood ratios.
• Positive and negative prediction values.
• Understand and interpret receiver-operator characteristic (ROC) curves.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Physical Examination Procedures
• Observation, interview, and the performance of physical
examination procedures are the primary methods available
during the injury evaluation process.
• Observation guides the opening questions in an interview
with a patient. Through observation and interview, the list of
diagnostic possibilities shrinks, and one or two conditions
emerge as the most likely culprits of the patient’s complaints.
• The clinician can then select physical examination procedures
that will help confirm or refute the existence of the suspect
conditions.
• Research into the performance of diagnostic tests differs from
research into, for example, prevention efforts and treatment
outcomes.
• In studies of diagnostic procedures, comparison to another
group of patients or subjects is unnecessary. These studies
require that the results of the diagnostic test of interest be
compared to the results of an established standard.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Clinical Research Improving Patient Evaluation
• Challenge in health care: maximize benefit to the patient
without squandering talent, time, and money on procedures
that are of little benefit or that pose more risk than benefit.
• The only way to find answers to clinical questions is through
clinical trials.
• The true magnitude of benefit and risk are revealed once a
diagnostic test is studied in the population it is intended to
benefit.
• The clinician-consumer of clinical research must understand
the relationship between study methods and threats to the
validity of data.
• The methods of studies of prevention, treatment, and
diagnostic procedures, and therefore the steps that minimize
investigational bias, differ.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Design of Studies of Diagnostic Testing
• Estimating the usefulness of any diagnostic procedure
involves comparing the results of a diagnostic test to the
results of an established, “gold” standard.
• Statistical analysis for studies involving tests with
dichotomous outcomes involves positive or negative
results.
• If a test is on a continuous scale, then a different analysis
is required to assess diagnostic test performance; but a
gold standard identifying those with and without the
diagnosis of interest is still needed to estimate the
usefulness of the test of interest.
• The reason a gold standard test is not applied to all
diagnoses in question has to do with the setting, timing,
scope, costs, and risks associated with the diagnostic
process. These issues often overlap.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Physical Examination Used to Narrow
Down Diagnostic Possibilities
• Studies of diagnostic procedures guide clinicians in the certainty
of their diagnoses.
• The less certain one is of a particular diagnosis, the greater the
likelihood that something else is wrong.
• The use of advanced diagnostic procedures (e.g., MRIs) comes at
a cost and in some cases (e.g., spinal tap to work up a patient
who may have meningitis) with a risk.
• The research consumer must decide if the subjects enrolled in a
study of diagnostic tests were sufficiently similar to patients in
their care to support the generalization of the results.
• If after considering the generalizability of research findings into
one’s practice the results of the paper remain of interest, the
research consumer must consider the validity of the data
reported.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Assessing Research of Diagnostic
Instruments
•
14-item assessment tool: assists consumers in evaluating the methodological quality
of research into diagnostic tests. Scores of 10 or more are considered to reflect sound
research methods.
•
Subject selection has the potential to bias results as well as to influence the
generalization of research findings.
•
Studies should include a spectrum of patients to whom the test in question would
typically be applied in a clinical setting.
•
Spectrum bias: exists when only patients very likely to be suffering from the condition
of interest are studied or when patients that clearly do not have a condition are
included.
•
Blinding of investigators performing or interpreting the results of the diagnostic test
being investigated as well as those responsible for the results of “gold standard”
assessment to each other’s finding is essential (items 10 and 11).
•
“Work-up” bias: data are biased towards underreporting of false negative results.
•
Some tests may not yield useful data, and not all subjects can complete all testing
(items 11 and 12).
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
The QUADAS Tool
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
The QUADAS Tool (continued)
Reprinted with permission from Whiting P, et al. BMC Med Res Meth. 2003;3:25.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Statistics and Interpretations
• Research reports from diagnostic tests describe how the data were
analyzed and provide estimates of the diagnostic test performance.
• The results from studies of diagnostic tests are reported in several
forms:
– estimates of sensitivity and specificity
– positive and negative prediction values
– positive and negative likelihood ratios
– receiver operator characteristic curves
• When one compares the results from a diagnostic test with a
dichotomous result (positive or negative) of interest to the results of a
“gold standard” diagnostic test, four subgroups or cells are formed.
• From this table sensitivity and specificity, positive and negative
prediction values, and positive and negative likelihood ratios can be
calculated.
• An expansion of the table vertically is needed to generate receiver
operator characteristic curves.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Diagnostic Test with a Dichotomous
Result Compared to “Gold Standard”
Diagnostic Test Results
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Sensitivity and Specificity
• Sensitivity and specificity are related to the ability of a test to identify
those with and without a condition and are needed to calculate
likelihood ratios.
• Sensitivity is the number of illnesses or injuries that are correctly
diagnosed by the clinical examination procedure being investigated
(cell A) divided by the true number of illnesses/injuries (cells A + C)
(gold standard measure).
# diagnosed as having a condition by clinical examination procedure
# diagnosed as having a condition based upon gold standard
=_A_
A+C
• Specificity is the number of individuals correctly classified as not
having the condition of concern based on the test being investigated
(cell D) divided by the true number of negative cases (cells B + D)
(gold standard measure). *Calculated from the right side of the
table.*
# diagnosed as not having a condition by clinical examination procedure
# diagnosed as not having a condition by gold standard
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
= _D_
B+D
Impact of Sensitivity and
Specificity on Clinical Practice
• Ideally, a diagnostic procedure has high sensitivity and high
specificity. Unfortunately many diagnostic tests lack
sensitivity or specificity or both.
• SNout: tests with high sensitivity are good at ruling out a
condition
• SPin: tests with high specificity are good at ruling in a
condition
• A sensitive test is one with relatively few false negative
findings; thus, a negative examination finding effectively rules
out the condition of interest.
• Tests with high specificity have few false positives; thus, a
positive result using a diagnostic test with high specificity has
identified the target disorder.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Positive and Negative
Prediction Values
• The calculation and interpretation of sensitivity and specificity are predicated
on the effort to identify individuals with and without a target disorder of
interest.
• These values are calculated “vertically” using cells A and C or B and D
respectively:
• Positive prediction value: if a diagnostic test is positive, what is the
probability the target condition is present? Calculated by: PPV= A / A+B
• Negative prediction value: estimates that the target condition is not
present when the diagnostic test is negative. Calculated by: NPV = D / D+C
• Sensitivity and specificity are calculated differently than PPV and NPV.
• As prevalence falls, PPV values also fall and NPV values rise.
• Sensitivity and specificity values are more stable estimates than PPV and
NPV but are difficult to apply in clinical practice.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Likelihood Ratios
• Likelihood ratios (LRs): values derived from estimates of sensitivity and
specificity.
• LRs can be applied by the clinician to the examination of individual
patients.
• Knowledge of the LRs influences the level of certainty that a condition
does or does not exist at the end of the examination.
• A positive likelihood (+LR) ratio: impact of a positive examination
finding on the probability that the target condition exists. For tests with
dichotomous results, an +LR is calculated as follows:
–
(+) LR = Sensitivity / (1 – Specificity)
• A negative likelihood (-LR) ratio: impact of a negative examination on
the probability that the condition in question is present. Negative
likelihood is calculated as follows:
–
(–) LR = (1 – Sensitivity) / Specificity
• A negative result of a diagnostic test with a small likelihood ratio suggests
that the chance that the target condition exists is very low.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Likelihood Ratios:
Broader Categories of Clinical Value
• Jaeschke et al. (1994) summarized likelihood ratios
(positive and negative) into broader categories of
clinical value, as follows:
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Applications
• Diagnosis is a process during which information is gathered to narrow the range
of diagnostic possibilities.
• A carefully conducted interview and observation form the foundation of a
physical examination.
• As the examination proceeds, a narrowing list of diagnostic possibilities is
developed.
• Once the diagnostic possibilities have been narrowed, the physical examination
and other diagnostic assessments that may confirm or rule out specific
diagnoses begins.
• A level of suspicion regarding diagnostic possibilities exists before the
examination procedure is performed.
• This pretest level of suspicion can be quantified as pretest probability.
• Pretest probability values vary among clinicians and the circumstances of the
individual patient.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Probability and Uncertainty
• Probability implies uncertainty. Uncertainty is inherent in the clinical practice
of athletic training.
• Decisions regarding referral, plans of treatment, and a physician’s use of
additional diagnostic studies revolve around the level of certainty (probability)
that a condition does or does not exist.
• Probability can be converted into an odds ratio using the following formula:
Odds = probability / (1 – probability)
• Posttest probability is calculated by first multiplying the pretest odds by the
LR+ to yield posttest odds.
• The conversion to posttest probability is made using the following formula:
Formula posttest probability = posttest odds / (posttest odds + 1)
• “Fair degree” of uncertainty: the more serious the consequences of being
wrong, the broader the definition of uncertainty. Additional testing is indicated
when this exists.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Receiver-Operator Characteristic Curves
• Applications in data management:
–
The assessment of diagnostic procedures in which data from
a procedure of interest are on a continuum rather than
dichotomous (positive or negative).
• ROCcs are an extension of the sensitivity, specificity, and
likelihood ratios.
• ROCcs allow for the identification of critical points along
continuous measures to guide clinical practice.
• ROCcs can also be applied to the analysis of clusters of
diagnostic or prognostic criteria.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins
Chapter Summary and Key Points
• Diagnostic testing should be used as an aid instead of a crutch for a clinician.
• History and observation should help narrow the scope of possible injuries during
the evaluation process.
• Diagnostic tests have limitations.
• The only way to find answers to clinical questions is through clinical trials.
• Special tests do not always deliver clear-cut decisions.
• The amount of confidence a clinician has in assessment results determines the
likelihood of a correct diagnosis.
• Clinical research must be sorted through because of the varying results in the
data.
• Comparing the clinical study with the current standard is important in research.
• Statistics are crucial to research methods because they provide numerical
evidence that can be ranked to form valid conclusions.
Copyright © 2011 Wolters Kluwer Health | Lippincott Williams & Wilkins