REVIEW OF DATA VALIDATION AND STATISTICAL PROCESSES FOR CONSULTANT
OUTCOMES PUBLICATION 2014
(2010-2013 DATA)
SUMMARY REPORT
The Healthcare Quality Improvement Partnership (HQIP) had been requested to provide written
assurance to NHS England Medical Director, Sir Bruce Keogh, that the validation and analytical
processes used to deliver the National Adult Cardiac Surgery Audit (NACSA), managed by NICOR, are
fit for purpose and have been suitably applied.
To support HQIP in undertaking this assurance, NICOR produced a report that detailed the processes
applied within the audit and described how the processes and procedures were consistently and
accurately executed within the data processing period.
The report focussed on a number of key elements:
a)
Governance
The report noted the governance of the audit and explained how it is managed through a number of
mechanisms.
 The NACSA project board; this group meets quarterly and consists of patient representation,
project management, professional and statistical representation.
 The SCTS-NACSA; the NACSA database is a standing item within the SCTS Executive which
meets quarterly.
 NACSA-HQIP; as part of the national Clinical Audit programme, NACSA has regular contract
review meetings with HQIP.
 NICOR-HQIP; NICOR has regular programme review meetings with HQIP.
The project governance of the NACSA is well-defined and transparent, with minutes of the NACSA
project group and the SCTS executive group available on the appropriate websites. The NACSA
project group is appropriately configured.
b) Data Validation
The report explained how data from 2010 – 2013 was used for the Consultant Outcome Publication
(COP) programme for publication in March 2014. Thirty six hospitals across England, Wales and
Northern Ireland submitted their data and this went through the NACSA routine data validation
process.
The initial data validation process involves three rounds of validation which supports hospitals and
consultants involved in verifying their data including the activity, mortality and risk factor data.
The process consists of:
 Export of the data from the NICOR database
 Cleaning/Analysis
 Validation with the units
As part of project management, at each stage of the validation process, the database manager and
individual consultant at each centre were informed of deadlines by email and letter and informed
that they were required to check and resubmit their data before the next stage of analysis could
proceed.
After the third round of validation, when all centres had had opportunities to respond and question
the output, the audit outputs are published on the SCTS website.
Key points to highlight within each of the three validation rounds:



Each centre obtains their information and the consultants receive their individual information
from the data manager.
At each stage of the process, NICOR confirm the deadline for resubmission of data in readiness
for the next round of analysis. All centres are made aware of the process to ensure that they are
involving each individual consultant and requiring them to check their own data.
NICOR sends all files to the data manager of each centre and alerts the consultant that the files
are ready for them to collect and investigate. Each provider receives notes from NICOR that
explain how the analysis was performed.
The report issued to HQIP included a number of supplementary documents and graphs which
included flow charts of the planned application of the routine data validation process, emails from
NICOR to individual database managers and clinical leads pertaining to different stages of the
validation process, extracts of minutes from the Executive Committee and NACSA Project Board
meetings, example letters regarding data revalidation and a selection of funnel plots.
Once the data was published in March 2014 issues were raised by one hospital regarding the quality
of the data about one particular surgeon from that hospital. This surgeon’s data was immediately
withdrawn from publication and a second data validation process was therefore undertaken on all
data in July/August 2014.
c)
Re-Validation
The repeat validation plan was approved by the NACSA steering group in May 2014. It was also
approved by the SCTS executive committee in July 2014.
The process had several aims:

To take further efforts to make it clear to units that data quality was their responsibility.
o

To support units in the repeat validation process.
o

Risk factors that had an incidence outside the expected range were flagged clearly to the
units.
To change the way in which one particular variable, ‘unstable angina’, was derived from the
existing fields in the database to decrease potential variability related to subjective
interpretation of this particular field.
o

This was undertaken by contacting the medical directors and surgeons within each unit,
placing the responsibility for data quality locally.
NICOR attributed EuroSCORE ‘unstable angina’ where a patient is recorded as having all of
the following:
o
Angina Class IV pre-surgery (field 2.01)
o
Urgent surgery (field 2.35)
o
Prescribed IV nitrates or any form of heparin pre operatively (field 2.29)
To check the validity of submitted mortality data by comparison with independent ONS tracked
mortality. This showed a discrepancy rate between patients submitted by the units as ‘alive’ and
tracked by the office of national statistics as ‘dead’ of less than 0.04%.
d)
Risk Adjustment and definition of outliers
Previously the NACSA used the logistic EuroSCORE for risk adjustment. This is a peer reviewed,
internationally accepted risk adjustment model in cardiac surgery first published in 1999. Monitoring
against a continually moving (and improving) standard has made NICOR’s methods more complex,
requiring a contemporary recalibration of the original EuroSCORE model; the risk-adjustment
methodology used to adjust for the diverse case-mix needed to be sophisticated enough to
represent contemporary outcomes or false reassurance may be derived.
In the absence of national guidance on the methodology for individual clinical risk adjustment and
statistical definition of outliers, a process was developed by NICOR in conjunction with the University
of Manchester
Following approval at the SCTS Executive and the NACSA project board, the methodology for a
contemporary recalibrated risk adjustment model was developed, based on the EuroSCORE model,
but calibrated for each year’s performance. This work has received independent statistical review
within NICOR and has been peer-reviewed and published to encourage outside scrutiny; these
references were supplied to the review team.
Centre Assurance
After re-validation had been undertaken; NICOR contacted all the units to seek assurance that the
data submitted for analysis and publication had been reviewed by the units. It was highlighted that,
although all deadlines for making changes to the data had passed, if there were any last minute
issues, there would still be the potential to add narrative alongside the data at the time of
publication.
Assurance was asked to be provided by email. All 36 centres gave written assurance that they were
satisfied that their data could be published.
Future direction & Conclusions
The report issued to NICOR detailed the processes followed to validate and analyse the adult cardiac
surgery audit data for the 2014 round of COP and describes how NICOR has developed and applied
the processes. National clinical audits have responsibility for supporting validation data by the
hospitals and surgeons; hospitals and surgeons have a responsibility to comply with these processes.
Once NICOR were informed that there were potential problems with the data, it responded by
supporting local data validation in a more targeted way, and has supplied a description of the
rational and the processes which were followed to do so. All analysis has been conducted with
appropriate statistical oversight within academic organisations, and the methods have been openly
published through peer review. Additional input has been sought from acknowledged international
experts where necessary.
The report fully recognises that there are a number of different approaches to the issues that NICOR
deals with. However, it set out the procedures and describes the project management, governance
and transparency that are currently in place.
NICOR is committed to providing accurate and timely data on cardiovascular outcomes for the
public, healthcare providers and medical profession. These issues are complex and England is leading
the way on transparency of individual outcomes and as such the processes are constantly evolving.
NICOR has approval from the Professional Liaison Group and the NICOR Executive for a minimum
data quality standard for all audits in the future. This standard operating procedure mandates
quarterly data submission of data to a pre-specified minimum data quality. NICOR is also currently
working on a NICOR-wide data validation process which incorporates many of the robust
mechanisms already in place within the NACSA, and will include unit level sign-off on the validity of
all data submissions from their hospital pertaining to the relevant national audit. NICOR will
complete the development of this process once the new HQIP outlier policy is published, and will
aim to incorporate any new recommendations or guidance given.