External Validation of clinical prediction models
Dr. Nancy Agnes, Head,
Technical Operations,
Statswork info@statswork.com
Validation, particularly external validation,
II. FACTORS INFLUENCES AFFECT
is a crucial part of developing a predictive
EXTERNAL VALIDATION DATA
model. External validation is needed to
ensure
that
a
prediction
model
is
generalizable to patients other than those
in
the
derivative
cohort.
External
The sample size for external validation
data for the implementation of the
prediction model is affected by the number
of events and predictors.
validation can be done by testing the
model's output in data that isn't the same as
External validation of the prediction model
the data used to create the model. As a
requires a minimum of 100 events and/or
consequence, it is carried out after the
non-events,
creation of a prediction model.
studies, and a systematic analysis found
according
to
simulation
that small external validation studies are
I. EXTERNAL VALIDATION
ineffective
and
inaccurate.Example:
External validation can take many forms,
Radiology imaging is often treated as
including validation in the field such as
effective
temporal, geographical and independent
researchers often validate the findings
validation. For external validation studies,
using clinical prediction model. Every
the sample size calculation estimates
prediction
based on statistical power considerations
regression analysis. The most common
have not been extensively investigated.
predictive model or the regression model
However, in order to achieve adequate
used for the clinical prediction model are
model output in the validation set, a large
linear regression if the dependent variable
sample size is needed to validate the
is continuous in nature, logistic regression
prediction model.
model if the dependent variable is binary,
and
predictive
model
is
Cox-proportional
parameters
based
model
on
if
and
the
the
dependent variable is time-to-event in
Copyright © 2021 Statswork. All rights
1
nature. Al-Ameri et al (2020) presented a
identified the validity using the calibration
detailed review on clinical prediction
slope
models for liver transplantation study.
presented
and
the
in
sample
the
articles
following
are
table
Further, Ratna et al (2020) discussed the
quality of clinical prediction model in vitro
fertilisation
and
human
reproduction.
Validation of model has been carried out
using re-sampling technique and measured
the accuracy using AUC, calibration plot
as shown in figure 1, c-index, and HosmerLemeshow test statistic.
Figure 1: Slope of Calibration plot
(Source: Stevens and Poppe (2020))
In addition, Stevens and Poppe (2020)
suggested the Cox- calibration slope using
logistic regression model instead of using
simply the calibration slope for the
predictive model. This suggestion has been
made after the scrutiny of around 33
Table1.Stated Interpretation of the
“Calibration Slope” Source: Stevens and
Poppe (2020)
Arjun et al (2020) considered the
pandemic mortality study of COVID19
and discussed the development and
validation of clinical prediction model.
research articles and found that most of the
validation are external validation and
Copyright © 2021 Statswork. All rights
2
II. FUTURE SCOPE
Though many literature suggests several
validation techniques for the predictive
model, there is no such proper technique
which can be suitable for all the clinical
datasets. Further, proper adjustment has to
be made for the calibration index to
validate the prediction model suitable for
all clinical datasets.
References:
1.
2.
3.
4.
5.
Stevens, R. J. and Poppe, K. K. (2020).
Validation of Clinical Prediction Models: What
does the "Calibration Slope" Really Measure?.
Journal of clinical epidemiology, 118, pp. 93–
99.
Adibi, A., Sadatsafavi, M., Ioannidis, J. P. A.
(2020). Validation and Utility Testing of
Clinical Prediction Models: Time to Change
the Approach. JAMA. 2020; 324(3):235–236.
Ratna, M. B., Bhattacharya, S., Abdulrahim, B.
and McLernon, D. L. (2020). A Systematic
Review of the Quality of Clinical Prediction
Models in Vitro Fertilisation, Human
Reproduction, 35(1), pp. 100–116
Arjun S Yadaw., Yan-chak Li., Sonali Bose.,
Ravi Iyengar., Supinda Bunyavanich., Gaurav
Pandey. (2020). Clinical Features of COVID19 Mortality: Development and Validation of a
Clinical Prediction Model, The Lancet Digital
Health, 2(10), pp. 516-525.
Al‐Ameri, A.A.M., Wei, X., Wen, X., Wei, Q.,
Guo, H., Zheng, S. and Xu, X. (2020),
Systematic review: risk prediction models for
recurrence of hepatocellular carcinoma after
liver transplantation. Transpl Int, 33, pp. 697712.
Copyright © 2021 Statswork. All rights
3