12/2013 WHITE PAPER
Adopting Site Quality
Management to
Optimize Risk-Based
Monitoring
Medidata and other marks used herein are trademarks of Medidata Solutions, Inc.
All other trademarks are the property of their respective owners.
Copyright © 2013 Medidata Solutions, Inc.
12/2013 WHITE PAPER
Adopting Site Quality Management to Optimize
Risk-Based Monitoring
Introduction
As it is now well documented, clinical R&D organizations have been actively
embarking on a paradigm shift in their approach to ensuring quality in the conduct of
clinical trials. In particular, a more targeted, risk-based approach to site monitoring
and data cleaning is being actively pursued by organizations both large and small.
The primary goals of this new paradigm include significant improvements in resource
efficiency associated with clinical R&D, increased quality in trial conduct and
corresponding data, and the maintenance or even reduction of clinical trial timelines.
While the interest in this paradigm has increased steadily in recent years, progress
had been significantly impeded by the following factors:
• Lack of full electronic data capture (EDC) adoption, which severely hampers an
organization’s ability to proactively assess quality across all sites in a clinical trial.
• Concern over the potential of increased regulatory scrutiny, especially in the absence
of clear guidance or endorsement for risk-based approaches from the FDA or EMA.
• Potential for actual decreased trial and data quality, due to less intensive on-site
monitoring during trial conduct.
The first of these issues has been largely overcome, as EDC and similar clinical
technologies have become “business as usual” for most clinical R&D organizations
over the past six or more years. The second issue was also finally addressed last
August in the form of the FDA’s draft guidance on risk-based monitoring, followed
closely by EMA’s reflection paper on risk-based quality management in clinical trials.
These documents not only provide the industry guidance on how to approach a
risk-based monitoring program, they firmly endorse the concept. The following from
the FDA guidance clearly demonstrates this:
“FDA encourages greater reliance on centralized monitoring practices than has been
the case historically, with correspondingly less emphasis on on-site monitoring.”1
The following will help you understand the challenges faced when implementing
risk-based monitoring and how developing a site quality management program—
especially one leveraging real-time site quality analytics—can simplify the adoption
of this strategy.
2
12/2013 WHITE PAPER
Adopting Site Quality Management to Optimize
Risk-Based Monitoring
The Role of Data Quality and Metrics in
the Adoption of Risk-Based Monitoring
For many organizations, the final concern regarding potential impact on data quality
has remained. Medidata has asked one question in this regard: to what extent is
patient electronic case report form (eCRF) data actually corrected during trials,
as a result of all of the intense scrutiny traditionally applied to that data by study
teams in the form of 100 percent source document verification (SDV), extensive data
management reviews and safety/medical reviews? In order to assess this, Medidata
computed the following two measures using the Medidata Insights™ metrics
warehouse, comprising over 2,500 studies from 65 contributing sponsor organizations:
• Total Data Correction Rate: The total percentage of eCRF data found to have one or
more updates (“corrections”) following initial submission of the eCRF form.
• Post-Capture Data Correction Rate: The total percentage of eCRF data found to
have one or more updates following the initial “capture session” at the site, defined
as a four-hour window following initial submission of each eCRF form.
The surprisingly low industry median for these two measures is illustrated in Figure 1.
The total data correction rate comes in at just 4.3 percent, while the post-capture
data correction rate is only 2.7 percent. Putting this in context, the post-capture rate
indicates that greater than 97 percent of eCRF data is already in its final form—ready
for formal reporting, analysis and submission—before any site monitor, data manager
or other sponsor representative has had a chance to scrutinize the data!
Figure 1: eCRF data correction rates
5.0%
4.32%
4.0%
2.68%
3.0%
2.0%
1.0%
Total
Source: Medidata Insights metrics warehouse
Post-Capture
3
12/2013 WHITE PAPER
Adopting Site Quality Management to Optimize
Risk-Based Monitoring
It is relevant to note that these metrics currently account for any updates to data
values in the EDC audit trail, which includes natural patient event-based updates,
such as adverse event (AE) resolution/outcome information that do not represent
actual corrections. While Medidata has not yet filtered out these type of updates from
the metric, initial analysis of AE and concomitant medication forms on various studies
reveals a very high “correction” rate on these resolution data fields (e.g., AE resolution
date, ConMed Stop Date, etc.), typically over 20 percent. Removal of these fields from
the calculation will likely yield a significantly lower post-capture data correction rate,
perhaps as low as two percent overall.
Whatever the final figure, it’s clear that our industry is spending a tremendous amount
of resource effort for a very modest return, in terms of impact on the final clinical trial
dataset. Figure 2 illustrates that a very typical breakdown of clinical trial costs and site
monitoring in particular is one of the two largest cost drivers at 30 percent or more.
If nothing else, the very modest ROI with traditional site monitoring methods should
compel us to seek a more effective, efficient approach to ensuring clinical data quality.
Figure 2: Clinical trial cost breakdown
Site Payments
Other
IVRS and Drug Distribution
Project and Clinical Leadership
Data Management
and Statistics
Monitoring
Source: Medidata internal analysis
4
12/2013 WHITE PAPER
Adopting Site Quality Management to Optimize
Risk-Based Monitoring
The Impact of Regulatory
Guidance on Site Management
The FDA draft guidance identifies two key monitoring approaches that comprise an
effective risk-based approach: on-site monitoring and centralized monitoring.
On-site monitoring is the current de-facto practice, so the introduction of centralized
monitoring now becomes a key enabler to realizing greater overall quality and
resource efficiency in the new paradigm. But what exactly is centralized monitoring
or—more to the point—what should it be? The FDA draft guidance outlines the
primary goals of centralized monitoring that should guide a solution. Among them
are the following:
• “Target on-site monitoring by identifying higher risk clinical sites (e.g., sites with
data anomalies or a higher frequency of errors, protocol violations or dropouts
relative to other sites)”
• “Augment on-site monitoring by performing monitoring activities that can only be
accomplished using centralized processes (e.g., statistical analyses to identify data
trends not easily detected by on-site monitoring)”
• “Conduct aggregate statistical analyses of study data to identify sites that are
outliers relative to others and to evaluate individual subject data for plausibility
and completeness”
• “Conduct analyses of site characteristics, performance metrics (e.g., high screen
failure rates, high frequency of eligibility violations, and delays in reporting data),
and clinical data to identify trial sites with characteristics correlated with poor
performance or noncompliance”2
The items above all reflect a similar theme that calls for leveraging the clinical and
operational data captured through EDC and other electronic systems. Together, these
can help sponsors identify sites with emerging quality-related risks, relative to other
sites in the trial. This in turn enables the attention of a sponsor team’s finite resources
to be directed to those sites with emerging risks as proactively as possible to remediate
any issues, thereby ensuring optimal quality across the trial. Any implementation of
centralized monitoring should therefore include an effective signal detection system
that emphasizes earliest possible detection of emerging quality-related risks at one
or more trial sites.
5
12/2013 WHITE PAPER
Adopting Site Quality Management to Optimize
Risk-Based Monitoring
Identification of the quality measures to use for signal detection should of course be
driven by the key risk areas identified in the monitoring plan for each study. However,
it is very important to note that the vast majority of trials will share similar key risk
categories including the following:
• Patient safety monitoring and reporting, for example AEs and serious AEs (SAEs),
• Patient data quality, and
• Protocol compliance, including eligibility criteria, and visit and dosing schedules.
Key Metrics to Consider for
Adopting Centralized Monitoring
As such, the industry can and should identify a relatively small set of measures that
serve as effective surrogates for the quality of trial conduct by sites in each of these
categories. As an example, one might consider the following measures related to
the quality of patient eCRF data capture at each site, all of which should be readily
computable using available EDC-based information:
• Data correction rate (percent of data corrected after initial entry)
• Rate of eCRF queries from clinical data management
• Rate of eCRF queries and/or SDV findings from site monitors
• Rate of “auto queries” (i.e., queries resulting from programmed data checks within
the EDC system)
• Cycle time from patient visit to entry of eCRF data by the site
• Cycle time for sites to respond to queries from the sponsor team
All of the above measures relate to processes that may have a real impact on the
quality of eCRF data. One approach might be to compute all of these measures in
parallel to help detect sites with emerging issues. This may have some degree of
success; however, it will more likely lead to a proliferation of signal-detection “noise”
that will actually detract from the goal of finding the true high-risk sites. Rather, we
should ask which groups of measures are descriptive of the same site process and
then choose the one or two that enable the most proactive and effective assessment
of site quality around that process. With respect to data quality, for example, it is fair to
say that eCRF query rates (e.g., auto queries, data management queries, site monitor
queries, etc.) and data correction rates should all be good indicators of the level of
quality with which sites are capturing patient data into the sponsor’s EDC system.
6
12/2013 WHITE PAPER
Adopting Site Quality Management to Optimize
Risk-Based Monitoring
In order to confirm this, Medidata conducted an analysis of these measures across
over 2,000 studies in the Medidata Insights metrics warehouse, which reveals a very
strong correlation between each of these query rates and data correction rates. So
it holds that any one of these measures should be a sufficient surrogate to detect
data quality issues. However, one of them—auto query rate—is clearly the most
effective when viewed from the perspective of early signal detection. This is because,
unlike manually generated queries and subsequent data corrections, auto queries
are generated and observable immediately upon entry of the eCRF data at the site.
Therefore, both the numerator and denominator comprising an auto query rate are
always current, whereas the numerator in the other rates always lags behind the
denominator by weeks or months (and yes, even years of delay in query generation
have been observed!).
Thus auto query rate is likely to be not only sufficient to detect potential data quality
issues at sites, but also the most effective among the obvious choices. Note that the
Visit-to-eCRF Entry cycle time metric—while it does overlap with auto query rate as a
surrogate for data quality—also measures an additional aspect of site behavior that
is worth observing. In particular, lack of timely attention to capture of patient data in
the sponsor’s EDC system—and slowness in responding to sponsor queries—may
portend broader issues with site’s attentiveness to and engagement with the trial
overall. Additionally, various sponsor reviews of incoming EDC data begin losing their
value the longer the delay in getting access to that data. So these cycle times provide
an important complement and addition to the auto query rate.
Understanding a Typical Site
Quality Management Scenario
It is indeed important to assess all potential site quality measures in the context of
not only how effectively, but how proactively, they are able to support signal detection.
Getting to a relatively small, focused set of standard quality measures is critical
to a successful centralized monitoring implementation. From a tool perspective,
organizations and study teams should be looking for a centralized dashboard in
which these quality measures are pre-computed across all sites and that provides for
each site a measure-by-measure and overall assessment of risk-level with respect
to quality. The tool should be configured in a way that directs study team members
quickly and effectively to sites with the highest risk levels requiring follow-up. Figure
3 depicts what the tool might display for a given site and measure. In the conceptual
example, the AE rate for the study overall is 8.7 AEs per subject-year, and for site 105
it is 5.6, which puts that site at an elevated (yellow) risk level for potentially
under-reporting AEs.
7
12/2013 WHITE PAPER
Adopting Site Quality Management to Optimize
Risk-Based Monitoring
About Medidata
Figure 3: Example site and study AE rate scorecard
3.7
5.6
Site 105
AE Rate
6.2
8.7
8
11.2
13.5
Study
AE Rate
Note that this would not mean conclusively that the site is under-reporting AEs—only
that it is at an elevated risk of such. The study team would be responsible for deciding
what, if anything, is the appropriate course of action to address this observation. It
could be as simple as having the site monitor take some extra time at the next site
visit to scrutinize the site’s AE reporting process and perhaps to review AE reporting
requirements with the site staff. It could also include increasing the amount of SDV
targeted for patients enrolled at that site—particularly for the AE forms. Or it could
be a combination of the above. Either way, with a site quality management program,
sponsors could better anticipate such risks and identify the root causes more quickly.
Implementing a Site Quality
Management Program
Using a standard set of measures, this type of centralized dashboard could be deployed
very quickly for each new study—a critical requirement given the importance of
early issue detection during trial conduct. Organizations should also plan to employ
a dashboard tool that is turnkey: not only easy to deploy for each study but also with
site quality metrics automatically assessed and presented in a near real-time fashion.
Such a centralized monitoring tool is not only ideal but quite feasible, and will help your
organization revolutionize the way total quality is managed in clinical research.
For more information on how Medidata Solutions can help you fully realize the
benefits of centralized and risk-based monitoring by leveraging the site quality
management concepts in this paper, please contact your Medidata representative.
Medidata Clinical Cloud™
Cloud-based clinical research solutions | Innovative technology | Data-driven analytics
Reduced costs | Improved time to market | Faster decisions | Minimized risk
Medidata Solutions is the leading
global provider of cloud-based
solutions for clinical research in
life sciences, transforming clinical
development through its advanced
applications and intelligent data
analytics. The Medidata Clinical
Cloud™ brings new levels of
productivity and quality to the
clinical testing of promising medical
treatments, from study design
and planning through execution,
management and reporting. We
are committed to advancing the
competitive and scientific goals of
global customers, which include
over 90% of the top 25 global
pharmaceutical companies; innovative
biotech, diagnostic and device firms;
leading academic medical centers;
and contract research organizations.
info@mdsol.com | mdsol.com
+1 866 515 6044
12/2013 WHITE PAPER
Adopting Site Quality Management to Optimize
Risk-Based Monitoring
Endnotes
1. AUG-2011 FDA Draft Guidance: Guidance for Industry: Oversight of Clinical
Investigations — A Risk-Based Approach to Monitoring
2. Ibid
9