AGENDA ITEM: C2 MEETING: RSSB Board Meeting DATE: 14 January 2016 SUBJECT: Data Quality Health Check Report SPONSOR: George Bearfield AUTHOR: Siona Vass 1.
Purpose 1.1
This purpose of this paper is to provide RSSB board with an overview of the data quality health check process for 2015, including the data quality scores of all companies assessed during the process. 2.
Overview of the data quality health‐check process 2.1 The data recorded in SMIS is used by the industry for a range of important purposes: safety performance monitoring; risk modelling; national and international statistics; input into research and development; and support for safety and business decisions. The data also helps build industry models like the Safety Risk Model and the Precursor Indicator Model, ensures we meet our legislative requirements, and informs safety campaigns. For all of these reasons it is vital that the quality of the information is as high as possible. 2.2 RSSB first introduced the data quality health‐check process in 2008. The process has focussed on a number of data quality aspects: timeliness of reporting, accuracy, completeness, and response to identified actions. The approach to assessing each measure has developed over time, following feedback from the industry and as improvements have been made in the process, but the essential approach has remained the same. 2.3 Following discussion with the industry at ATOC Safety & Stats Group and SMIS User Group, the measures for the 2015 data quality health check assessment have been limited to the measures related to timeliness and accuracy. Industry discussions identified that a factor affecting individual scores was the fact that smaller companies with fewer events were more susceptible to bias on the measure of completeness, and that the measure related to response to actions was also too affected by the size or nature of the organisation. It was therefore decided to limit the 2015 assessment to the two most robust measures of timeliness and accuracy. This also allows for more resources to be made available to develop new and better measures within SMIS+. See the Annex B for further information on the indicators which make up the score and for the details of the health checking process for 2015. 2.4 The material in this paper has been circulated to SMIS Programme Board, the governing group for the SMIS system for the duration of the SMIS+ programme. It has also been provided to SSRG for information. In addition, the individual company scores have also been provided to the Managing Director, Safety Manager and data quality contact of each individual company. RSSB Board Meeting Final: 14 January 2016 Page 1 of 3 AGENDA ITEM: C2 3.
Benefits to RSSB members 3.1 The data quality project has seen continuous improvement over the eight years it has been in existence. The continued effort to improve further each year confirms the enormous effort the industry has put in to improve data quality within each organisation. The project has reached a pivotal point in its life span to be adapted further to ensure each measure is considered to be relevant and accurate measure of data quality. This timely adaption to the project to shape how the project will continue in the coming years with the new system as part of the SMIS+ programme. 3.2 The project aims to identify data that is missing or inconsistencies and discrepancies in the data using the indicator report sent to companies weekly. The health checks are a way of highlighting where procedures within individual companies are less complete or need formalising. Such issues highlighted the need for improved data capture during the initial event to improve accurate reporting. This allows for timely entry into the system and more accurate from the onset to minimise the need to chase information long after the event has occurred. The investigation process has also improved in many organisations to ensure the right level of detail is captured after the event. Network Rail has recently formalised their investigation process and has introduced a sign off process for the more serious events requiring an investigating. 3.3 The industry continues to actively engage with RSSB in the data quality programme and the benefits of the initiative appear to be widely recognised. An evaluation of the data quality programme will therefore be scheduled. Feedback to RSSB regarding the programme is encouraged, as any potential improvements that can be recognised and developed will reinforce the data quality and associated outputs, ultimately benefitting the industry. The outcome of the review will be fed into the SMIS+ programme. 4.
SMIS+ and future data quality initiatives 4.1 The strategy to incorporate data quality within the new safety reporting system is fairly new concept for RSSB and the programme team. The idea is in the early stages of planning how it will be captured in SMIS+ is still under consideration. The high level specification ensures that the system should be able to identify any conflicting areas during input and this will reduce the number of data quality errors from the initial input. 4.2 A strategy meeting has been held to review how data quality overlaps with other areas of SMIS+ programme and focus is around these areas: • Data design (defining quality regime, rules logic) • System delivery (interface/workflow design) • Business change (DQ monitoring, governance) 4.3 The current scope for the data quality project will fit within the new system and potentially expand the measures for improved assessment. This will help to explore further the question of what makes good data quality. Specific questions to be explored include: 1. How should the current scope be expanded to cover better data quality monitoring? 2. Does the BI functionality within SMIS provide an opportunity to do DQ more efficiently? 3. How can the system be structured to innately support better quality data? RSSB Board Meeting Final: 14 January 2016 Page 2 of 3 AGENDA ITEM: C2 4.4 An area which has been highlighted during industry feedback on the current approach is that a means for identifying where incorrect or poor quality information has been entered should be considered. To resolve these issues, it is planned that data quality checking capabilities will be incorporated into the design of SMIS+. 4.5 Consideration is also being given in the SMIS+ design as to how to reduce the occasions where disagreement about the content, status or ownership of an event occurs. This should also improve data quality. 5.
Results of the 2015 data quality assessment 5.1 Table 1 in the Annex A shows the data quality scores based on a year’s data from 1 November 2014 to 31 October 2015. Annex B shows how the score is calculated. Timeliness 5.2 Nearly all companies (24 out of 28) fully meet the requirement to enter events into SMIS within five working days, as required by the group standard. This is the same as 2014, when 24 companies achieved 100%. Two companies improved during the year, each increasing their score by 5%. For three companies there was a deterioration in the measure. These companies have had staff changes and/or resourcing issues during the year that are likely to have affected the ability to meet the five‐day requirement. Accuracy 5.3 Nearly all companies (22 out of 28) achieved 100% on the accuracy assessment measure. This is an improvement on 2014, when 10 companies scored 100%. This reflects a general increase in input accuracy of the industry as a whole. Seventeen companies improved their accuracy measure compared with the previous year. On a positive, note, improvements in accuracy were achieved by the three companies for which timeliness has decreased. 5.4 Overall, there was a small decrease of 2% in the average timeliness ranking for the industry as a whole, which was offset by a small increase of 2% in the average accuracy ranking for the industry as a whole. 6.
Concluding remarks 6.1 The data quality programme has been active for eight years, with the aim of introducing consistency in reporting among stakeholders. This has demonstrably taken place when the industry is considered as a whole, and the latest scores once again highlight the improvements that have been made throughout the programme, particularly in the area of input accuracy. 6.2 The SMIS+ programme offers the opportunity to bring about a positive step‐change both in how data quality is assessed, and the quality of the data in the system. 7.
Recommendations 7.1 The RSSB board is asked to NOTE the contents of the report and APPROVE its release to ORR. RSSB Board Meeting Final: 14 January 2016 Page 3 of 3 AGENDA ITEM: C2 Annex A: Data Quality Scores 2015 Table 1.
Data quality scores – November 20151 Organisation
Abellio Greater Anglia
Arriva Trains Wales
c2c Rail Limited
Chiltern Trains
Cross Country Trains
DB Schenker
Direct Rail Services
East Coast ‐ Virgin
East Midlands Trains
Eurostar
First Hull Trains
First TP Express
Freightliner
GB Railfreight
Govia Thameslink*
Grand Central Railways
Great Western Railway
Heathrow Express
London Midland
London Overground
London Underground
Merseyrail
Network Rail
Northern Rail
Scotrail
South West Trains
Southeastern
Virgin Trains WC
National Score
Change in 2015 Number 2015 2014 Timeliness Input of events Timeliness Timeliness
Score
quality
1836
100%
100%
0%
100%
2156
80%
100%
‐20%
100%
255
100%
100%
0%
100%
673
60%
90%
‐30%
99%
961
100%
100%
0%
100%
766
100%
100%
0%
100%
119
100%
100%
0%
100%
1460
70%
95%
‐25%
100%
1250
100%
100%
0%
100%
647
100%
95%
5%
100%
63
100%
100%
0%
100%
636
100%
100%
0%
100%
87
100%
100%
0%
99%
297
100%
100%
0%
99%
799
100%
100%
0%
100%
109
100%
100%
0%
99%
1898
100%
100%
0%
100%
184
100%
100%
0%
100%
1092
100%
100%
0%
100%
1062
100%
100%
0%
100%
15
100%
100%
0%
99%
1071
100%
100%
0%
100%
46386
100%
100%
0%
100%
1610
95%
90%
5%
100%
1552
100%
100%
0%
100%
2145
100%
100%
0%
100%
2086
100%
100%
0%
100%
1233
100%
100%
0%
99%
72448
97%
99%
‐2%
100%
2014 Input quality
99%
98%
100%
96%
97%
98%
99%
91%
93%
100%
100%
98%
100%
88%
99%
99%
100%
100%
100%
100%
98%
100%
99%
97%
100%
97%
97%
98%
98%
Change in Input Quality Score
1%
2%
0%
3%
3%
2%
1%
9%
7%
0%
0%
2%
‐1%
11%
1%
0%
0%
0%
0%
0%
1%
0%
1%
3%
0%
3%
3%
1%
2%
* The Govia Thameslink score is based on a combination of the score for their pre‐merger and post‐
merger operations. 1
There is currently insufficient data to place; MTR Crossrail, DB Regio, Colas and Devon & Cornwall on the table. RSSB Board Meeting Final: 14 January 2016 Page 1 of 1 AGENDA ITEM: C2 Annex B: Data Quality Scores Calculation 2015 The calculation for the two elements of the score in 2015 has remained unchanged from the previous year and can be easily compared. The measure for input quality tracks the error rate every day in records entered, but it won’t consider the quality of any record until five days after the record was entered to allow time for the initial information to be gathered. Input quality The input quality percentage is calculated by measuring the average error rate2 in the following topics: Topic Description Event summary List of events with either a missing narrative, missing location or missing
description. Lists any injury where RSSB have flagged the injury degree as being possibly incorrect and where a response hasn’t been entered into SMIS by the event owner. Lists any injury where RSSB have flagged the person type as being possibly
incorrect and where a response hasn’t been entered into SMIS by the event owner. Lists any injury where RSSB has flagged it as being a duplicate entry.
Lists any event where there is a train object with either a missing train
type, a missing train company or missing unit number (passenger trains only). Lists any event with a component that hasn’t had all the necessary classification questions answered. Lists any event with one component and one line object where the line
type is inconsistent with the line object and classification questions. Lists any event with one component and one train object where the train
type is inconsistent with the train object and classification questions. Lists any operational incidents with a consequence of fail to call or stopping
short that has no train object. Lists any operational incidents event with either a missing activity, a
missing sub activity, a missing role or a missing consequence. List any workforce assault that has no key contributory factor. Injury degree differences
Person type differences Duplicate injuries Missing train information
Classification questions not answered Line type inconsistencies
Train type inconsistencies
Fail to call and stopping short events with no train object Operational incidents with missing information Workforce assaults with no key contributory factor Operational incidents with a missing component Dangerous goods with missing information TPWS with missing information Inquiries and recommendations with missing information Lists any operational incidents where the consequence matches another
SMIS component and that component hasn’t been entered into SMIS. Lists any dangerous goods incident with one of the following fields missing:
Description, location, UN number, alpha code, DG class, DG incident type, hazard risk ranking, event classification, DG reference, train number, train sponsor, rail vehicle sponsor, rail vehicle type or rail vehicle number. Lists any TPWS event with one of the following fields missing: TPWS mode, TPWS classification, Approaching, Loop type, Reset and continue or Speed reduction. Lists any inquiry with one of these fields missing:
Report title, inquiry type, report ref, and inquiry summary (for formal only) AND Lists any recommendations with one of these fields missing: Primary lead, recommendation number, recommendation details, directed to org. The average error rate is an average of the cumulative error rate in records from 1 November 2014, measured every day. This will therefore take into account the number of errors made, and how long they remain in SMIS. 2
RSSB Board Meeting Final: 14 January 2016 Page 1 of 2 AGENDA ITEM: C2 Timeliness The average number of working days to enter the records into SMIS will be measured and assigned a percentage based on the following table: Timeliness % Within 5 working days 100% Between 5 and 7 days 95% Between 7 and 9 days 90% Between 9 and 11 days 80% Between 11 and 13 days 70% Between 13 and 15 days 60% Between 15 and 17 days 50% Between 17 and 19 days 40% Over 19 days 20% The current health check process In 2015 the full and half‐year health check meetings took place via telephone in March and November. The phone calls are used to discuss the latest indicator reports, input quality and to check progress against their previously agreed action plans, and where necessary agree further actions. At each health check, RSSB covers the following main areas: 
A brief update of the local data process employed by the stakeholder in question, highlighting any changes in the process and staff over the last review period. 
A review of the indicator report, highlighting issues requiring action based on the current input quality score. 
Any training requirements. 
How the stakeholders have responded to any open actions from the previous health check and subsequent follow up. 
Any other issues that impact on data quality, such as clarification of reportable events, resourcing changes and local IT performance issues. Following each health check, RSSB produces a note detailing all the key discussion points and the actions agreed at the meeting and share this with the stakeholder contacts for comment/follow up. Any actions are added to the actions log and followed up on a periodic basis by the RSSB data quality contact. On a weekly basis, data quality indicator reports are sent out automatically by SMIS to each stakeholder data quality contact, highlighting records that need attention, based on the input quality assessment. Individual stakeholders have stipulated the frequency that these are sent out according to their preference. This regular communication has ensured that data quality checking is part of their data input process. RSSB Board Meeting Final: 14 January 2016 Page 2 of 2