Testing and trialling
Good practice guide for
testing and trialling new technology
for Britain’s railways
April 2014
’The railway sector is seeing significant investment,
but it comes with the need to deliver greater value for
money – to deliver more for less – so that the sector can
develop and prosper for the benefit of all stakeholders.
Government and the wider rail industry see innovation
as a key enabler for this to happen, and the ability to
efficiently test and trial new technology and products is
essential for this.
The processes for introducing new technology
and products onto the operating railway are often
seen as unclear as are the responsibilities at each
stage of the process. In this guide, the Technical
Strategy Leadership Group (TSLG) aims to improve
understanding of the process of product introduction
and associated testing and to help all those with an
interest in introducing new technology to Britain’s
railways – through sharing good practice on testing and
trialling. The guide is a partner document to the TSLG
website which identifies the testing and trialling facilities
available to the industry.
The guide is intended as the key reference document
for all testing and trialling activities. It provides
a thorough description of the testing and trialling
landscape for Britain’s railways, together with practical
information and experience about how to engage with it
and apply innovation. We trust it will prove a valuable
resource for all those developing equipment and
systems for use on the railway and for those considering
commissioning tests or trials.’
Francis How, Technical Director, Railway Industry Association
Jerry England, Director of Engineering, Network Rail
2
RSSB
About this guide
When compared with other modes of travel, rail
provides substantial benefits in terms of environmental
performance and urban regeneration. However, to
secure these benefits for the longer term and enhance
them, rail has to become more sustainable, particularly
in light of the current economic climate. This means
change, which needs effective innovation, which needs
effective testing and trialling, which ensures risks
are appropriately managed when new technology is
introduced. The first edition of this guide shared good
practice on testing and trialling to promote more efficient
introduction of new technology.
Work has continued since then, instigated by TSLG,
to address concerns that barriers in testing, trialling,
standards, and consistency in the acceptance of
innovation could have a negative impact on the
introduction of innovative technology or products for GB
railways. In particular TSLG felt that it would be worth
investigating if better use could be made of test results
from other sources and environments to speed up the
process of introducing innovation.
A study for TSLG has concluded that barriers to
innovation associated with inappropriate testing are
not as significant as had been suggested, and that this
was not the key issue in bringing innovation to GB rail.
RSSB
3
In most cases, testing requirements for development
or product acceptance are reasonable. Nevertheless,
there are economic impacts associated with defining
and completing testing requirements, and these cause
delays in the introduction of products and technology.
Delays can be significant for small- and mediumsize companies. Given that a significant number of
the issues identified arose out of a failure to identify
stakeholder expectations, TSLG recommended that
additional efforts be made in a number of areas, notably
to improve up front formal negotiation and planning
of acceptance criteria and tests. This updated edition
incorporates the findings of that work and seeks to offer
guidance to successfully implement these.
The guide is aimed at anyone who has an interest in
testing and trialling, whether they have considerable
experience in the rail sector or are new to it. It should
be of particular interest to:
•Those thinking of commissioning tests or trials,
particularly those working for a railway undertaking
or infrastructure manager who haven’t been closely
involved in this vital area of product introduction
before.
•Test house and facility operators, by conveying to a
new client the sort of issues that need to be thought
about when approaching them to discuss potential
tests.
•Those who are developing new technology for
4
RSSB
Britain’s railway, particularly those new to the
industry or seeking to gain a foot hold in it.
To avoid being a large and inaccessible tome that
risks never being opened, this guidance is pitched at a
relatively high level and applies across all asset groups.
It gives advice that is as applicable to engineers seeking
to introduce large complex systems such as new rolling
stock, as it is to those bringing in components such as
new rail fixings. It aims not just to provide advice on
the testing required for product approval, but testing
required throughout the whole product development
lifecycle; many of the principles are as applicable to
concept development as they are to product approval.
Some, perhaps most, of what is in this guide may
seem rather obvious – it’s certainly not rocket science
- and is relatively straightforward to take into account
when conducting testing and trialling. Nevertheless,
it can so easily be forgotten or overlooked with costly
consequences. To those who see the content in this
way, the guide will be a useful aide memoire and they
may even wish to contribute to future revisions. To
others, particularly those just starting out to earn their
spurs in this most important part of the innovation
process, it should provide an invaluable compendium
of what should and shouldn’t be done. Ignore it at your
peril!
While the guide is not intended to replace expert
guidance, it should lead to more efficient dialogue with
RSSB
5
experts. Considering its content may reduce uncertainty, and the risk of
abortive tests and cost, as well as improve the chance of success.
The guide has been compiled by the Railway Industry Association (RIA)
on behalf of the Testing Facilities Steering Group (TFSG), a subgroup of
the Technical Strategy Leadership Group (TSLG). It would not have
been possible without the assistance of the many organisations listed
in Section 11 for which RIA and TFSG are grateful. Notwithstanding
this support, there is probably much that could have been included in
this issue of the guide that would give further value to those involved
in testing and trialling. With this is mind, comments are welcomed and
should be submitted to TFSG via enquirydesk@rssb.co.uk. TFSG will
review this guide and consider the need for further iterations in response
to feedback.
© RSSB Copyright 2014 Rail Safety and Standards Board.
This publication may be reproduced free of charge for research, private
study, or for internal circulation within an organisation. This is subject to
it being reproduced and referenced accurately and not being used in a
misleading context. The material must be acknowledged as the copyright
of Rail Safety and Standards Board and the title of the publication
specified accordingly. For any other use of the material please apply
to RSSB’s Head of Research and Development for permission. Any
additional queries can be directed to enquirydesk@rssb.co.uk. This
publication can be accessed via the RSSB website www.rssb.co.uk.
A PDF version of this document is available to download from www.
futurerailway.org/Pages/testingandtriallingfacilities.aspx. The PDF
has hyperlinks and bookmarks to aid navigation through the different
sections of the document and to other supporting information.
6
RSSB
Contents
About this guide3
Contents7
1 How to use this guide
9
2
Scope of this guide12
3 Definitions
14
4 Why do we need to test?
15
4.1 Testing as part of the technology
development process
16
4.2 Testing as part of the product development lifecycle
18
4.2.1. Testing to demonstrate compliance with legislation 21
4.2.2. Testing to demonstrate compliance with the contract
22
5
Real or virtual?26
6
Legislation and standards32
6.1 Context
32
6.2 European standards and legislation
33
6.3 Domestic standards and legislation
35
7
Where can we test?39
8 How to go about testing
41
RSSB
7
8.1 Preparation
43
8.1.1. Context and stakeholders
43
8.1.2. Planning
46
8.1.3. Objectives and success criteria
50
8.1.4. Roles and responsibilities
52
8.1.5. What is to be tested
55
8.1.6. Facilities and equipment required
56
8.1.7. Facilities and equipment required
60
8.2. Testing
62
8.3. Reporting
63
8.4. Reviewing how it went
64
9 Special requirements when trialling
65
10
For more information70
11
Acknowledgements73
Annex A: Planning checklist74
Annex B: Processes to support on-track
testing and authorisation for placing into
service of rolling stock78
References80
8
RSSB
1 How to use this guide
This guide could be read from beginning to end and has
been written to tell the ‘testing and trialling story’ in a
logical sequence. However, the Test Facilities Steering
Group (TFSG) is acutely aware of the demands on
people’s time and that the value in a guide such as
this is to make the content as accessible as possible.
The guide has therefore been structured in a way that
signposts content to facilitate quick and easy reference,
in the hope that it will become a valued reference
guide for those interested in testing and trialling new
technology for Britain’s railways.
Although the document can be read as hardcopy, it
has been developed to be read in its electronic form
so readers can make use of the many hyperlinks,
which not only take you to other parts of the document
(green hyperlinks), but to other reference material that
can be found on the internet (bold hyperlinks). In an
attempt to illustrate the points being made and add
further interest, anecdotes and case studies are given
throughout the guide in grey boxes.
Briefly, the guide is structured as follows. The scope
is outlined in Section 2 followed by key definitions in
Section 3. Section 4 explores the role of testing and
trialling during the development of new technology
and the product development lifecycle, while Section
5 discusses the ever increasing role of computer
based simulation in this. Section 6 sets out the legal
RSSB
9
framework and relevant industry standards that we need
to comply with when testing to demonstrate compliance
against mandatory requirements. Section 7 signposts
where to get more information on facilities that can be
used for testing and trialling work, while Section 8 sets
out good practice for testing, followed by Section 9
which gives special requirements for trialling. Section
10 signposts where to go for more information and
expert advice, and the guide concludes with Section 11
which acknowledges the many organisations that have
assisted with its compilation.
10
RSSB
RSSB
11
2 Scope of this guide
What this guide covers
12
What this guide does not cover
Testing for Britain’s mainline railway
systems: Network Rail managed
infrastructure. Information for
London Underground and other
systems has been included where
readily available.
Light rail, heritage railways and
foreign rail administrations, although
many of the principles will be
transferable.
Testing and trialling using physical
facilities such as laboratory test rigs,
as well as test tracks and other sites
dedicated to testing and trialling
purposes such as sections of the
operational railway.
Computer simulation, although
a section is included on its ever
increasing role in the testing arena.
Testing of new technology that
exists in a physical form, including
mechatronic systems.
Testing of software when not part
of a physical system, such as
when not part of a mechatronic
system. Further information
relating to software can be found
on the management of change
page of the RSSB website
and EN50128:2011 ‘Railway
applications. Communication,
signalling and processing systems.
Software for railway control and
protection systems’.
RSSB
What this guide covers
What this guide does not cover
Most of the guide concentrates on
testing although there is a section on
trialling, and many of the principles
described for testing will apply to
trialling as well.
Routine testing, such as following a
vehicle exam (see Railway Group
Standard GM/RT2004), or electrical
appliance testing, although many
of the principles may still apply.
Equally, testing following incidents
and accidents is excluded (see
Railway Group Standard
GM/RT2273).
All types of railway asset (such as
rolling stock, track, and signalling)
with asset specific standards
referenced where readily available.
However the principles tend to
be defined at a higher level and
therefore apply across all assets.
Guidance on seeking product
acceptance, except where
acceptance affects testing and
trialling. Further guidance can be
found on the Network Rail website
and in its Company Standard
NR/L2/EBM/029 (see Section 10).
Technologies, products, and
systems.
Manufacturing processes and their
management systems.
RSSB
13
3 Definitions
While the following terms can mean many different
things to different people, this guide uses them with the
following meanings:
•Test – to demonstrate using a physical process (as
opposed to computer-based) that the equipment
being tested is capable of achieving a specified
level of performance for the test environment and
conditions used; the testing using facilities away from
the operational railway. This may include testing on
service lines during a possession.
It may also include commissioning where new
technology is being made ready for operational
service and which might involve testing such things
as the man-machine interface, finalising operational
processes and staff training.
•Trial – to test performance using facilities on the
operational railway in revenue earning service; the
word ‘test’ having the meaning given above.
14
RSSB
4 Why do we need to test?
The dictionary1 definition of the verb ‘to test’ is ‘to
try (something) out to ascertain its worth, safety or
endurance’. In the context of this guide, and indeed
the wider engineering field, this means testing to
demonstrate performance against required functionality.
Such functionality may be nothing more than conceptual
aspirations of a nascent innovation, or it could be
acceptance criteria formally defined by a regulatory
body for a mature technology. Testing is not just about
gaining acceptance and demonstrating that something
is safe to operate, however important that may be;
it is an integral part of the process of technology
development long before it becomes mature enough to
warrant acceptance.
It is perhaps useful to consider testing in the context of
both technology development and product development;
the latter being a subset of the former and taking place
in the latter stages of the technology development
lifecycle.
1. Paperback Dictionary, Collins, 2004
RSSB
15
4.1 Testing as part of the technology
development process
Testing is a vital aspect of the development of new
technology. The box on the following page defines the
technology readiness levels (TRL)1 which describe the
different stages that a new technology goes through
during development. Testing enables a technology
to progress from TRL2 where it is nothing more than
a concept through to TRL9 where it is proven and
accepted into the operational environment.
It would be unwise to wait until full-scale testing
to determine the value of a new technology; less
expensive small case testing (in a laboratory for
instance) is normally done first. Nevertheless, testing
enables the technology to be assessed at each stage,
whether it be in a laboratory or full-scale testing or
trialling in an operational railway environment. The
design can then be corrected or improved if necessary
and the next stage of development informed (or even
shelved).
Testing serves a number of purposes during the
exploration and assessment of a new technology and it
is wise to explicitly agree its purpose. For example, to:
•Understand the behaviours, so that the effect on
adjacent systems can be understood.
•Establish boundaries of performance, so that a
specification can be created.
1. ‘Development of a technology roadmap and action plan for the GB
railways – short-term roadmap’, R&D project T809, RSSB, October
2010.
16
RSSB
•Demonstrate viability and value, by exceeding some required
threshold of performance.
•Explore failure limits and behaviours, to consider risk and safety.
Only when the purpose is clear can the appropriate tests be designed
and the requisite performace targets and thresholds be negotiated.
The next section considers the role of testing during the later TRLs,
during product development.
RSSB
17
4.2 Testing as part of the product development
lifecycle
Consider now a technology that has emerged from
the ‘valley of death’ (TRLs 4-6) and is starting to be
commercialised as a new product and integrated into a
system.
The familiar V lifecycle diagram (next page) is derived
from one published by the Institution of Railway Signal
Engineers (IRSE), and depicts the different stages of
development that a system will undergo throughout its
entire lifecycle; from its birth as an outline definition
(in response to a business need) through gradually
increasingly more detailed design phases, through
to testing phases, commissioning, operation and
maintenance and finally decommissioning. Verification
and validation (proving) activities include testing to
ensure that the requirements defined earlier on in the
lifecycle have been delivered.
Whether we’re talking about rolling stock or
infrastructure components, as the stages of testing
progress, the environment will move from the laboratory,
to test tracks and then onto the operational railway. As
this happens, precise control over input conditions will
gradually deteriorate making way for an environment
that is gradually more representative of the operational
railway but less determinate.
18
RSSB
‘Competence Guidance for Train-Borne Train Control Systems’ © IRSE 2009
RSSB
19
The impact test cannon, one of a
series of factory test rigs for testing
vehicle bodyside windows to Railway
Group Standard
GM/RT2100.
Each stage of the testing process
has its place and there are pros and
cons to each stage. While it can
be difficult in a laboratory or factory
to fully simulate service conditions,
it does enable performance to be
tested in a known and consistently
repeatable (test) environment
that is difficult to replicate in a
demonstrable (measurable) way on
the operational railway. Examples
include:
•Tests of electronic systems in
© Independent Glass, Glasgow
climatic chambers to ensure
that they continue to perform
as required over extreme
temperature ranges.
•Accelerated fatigue testing of sleepers on laboratory
test rigs to demonstrate acceptable life.
•Testing of vehicle bodyside windows.
•Sled testing of anatomical test dummies in seats to
test interior crashworthiness.
In the later stages of the product development lifecycle,
testing becomes an essential part of the acceptance
process (although it is by no means the only part).
Testing at this stage can typically be divided into
two areas: testing to demonstrate compliance with
legislation and testing to demonstrate contractual
compliance.
20
RSSB
4.2.1 Testing to demonstrate compliance with
legislation
This is usually aimed at demonstrating that the new
technology will be safe and compatible with the
system in which it is due to operate. These ‘type
tests’ are normally performed on the first of build
example or prototype; subsequent builds can then
simply be subject to tests that prove they are the
same as those which were type tested.
•Testing to demonstrate safety is usually carried out
in accordance with test protocols and performance
criteria set out in standards. Section 6 gives more
information.
Testing to demonstrate compatibility with the
system in which the technology is expected
to operate may not be subject to such clear
requirements. These tests need to assess the
impact of the new technology on the system in
which it is expected to operate and vice versa.
This is about systems integration, ensuring that
interfaces perform as required. Examples include:
A new sleeper ‘engaging’ with the ballast and
rail retaining mechanism.
New trains interfacing with the permanent way,
power supply, signalling system, the wider
environment and, operationally, with traincrew
and passengers. EMC testing of new trains to
demonstrate compatibility with the signalling
system has in the past been particularly
problematic.
•
•
•
Even technology that has been proven in other
environments may not be suitable for the rail
RSSB
21
environment, for instance, a road/rail crane will
behave differently when on rail wheels to road
wheels and could be more unstable. Another
example is Network SouthEast running a trial on a
commercially available hand-dryer in the toilet of
a train as part of the ‘tractionisation’ process, ie to
ensure that reliability of the hand-dryer would not
be compromised by the accelerations seen in the
rail environment.
Whether testing to demonstrate compatibility or
testing to demonstrate safety, the criticality of these
tests means that the results are usually required to
be assessed by independent third parties to give
the required level of assurance. Choice of the third
parties is governed by the assurance regime that
has been selected and can include Notified Bodies
(NOBOs), Designated Bodies (DEBOs) and Vehicle
Acceptance Bodies (VABs). Section 6 gives more
information.
4.2.2. Testing to demonstrate compliance with the
contract
This is dictated by the requirements set by the
client in the contract and will usually be carried out
as part of type testing. By way of example, ATOC1
suggests the following stages of testing to gain client
acceptance for new trains (notwithstanding those
associated with legislative compliance):
•Fault free test track running to enable preliminary
acceptance at the ‘factory gate’.
•Commissioning both static and dynamic tests to
1. ‘The Twenty Point Plan, Fleet Reliability Focus Group’, CR/TP 1203
Issue 7, January 2011.
22
RSSB
gain enough confidence for trains to be run on
Network Rail managed infrastructure.
•Test runs, accumulating mileage and proving
experience on Network Rail managed
infrastructure to enable provisional acceptance.
•Operation in passenger service under the
operator’s safety management system.
•Final acceptance when modifications arising from
service experience are completed and sufficient
fault free running has been achieved.
Examples of the sort of testing that takes place to
demonstrate compliance with the contract are:
•Vehicle reliability – a period of mileage
accumulation on a test track to demonstrate fault
free running prior to operational service.
Performance – proving, for instance, that a new
tamper can work at a sufficient speed.
Capability – to confirm how far beyond the
required limits a machine can actually continue
to operate safely. For example, the maximum
installed cant a crane can operate safely when
fully extended.
Vehicle ride quality – to confirm that the
accelerations that the passengers will feel as a
result of vehicle movement are acceptable.
•
•
•
The opportunity can also be taken during testing
programmes (or at least while the new equipment
is visiting test facilities) to achieve other objectives
besides testing alone. For example:
•Training those who will use the new equipment in an
RSSB
23
environment where risks can be better controlled.
This could include emergency scenarios which are
difficult if not impossible to simulate elsewhere.
•Proving the man-machine-interface between people
and complex on-track machines to inform production
processes – if people can’t work with it then there’s
no point in commissioning it.
•Fine tuning the maintenance strategy to help
maintenance staff and improve care of the new
equipment.
A ballast regulator used for distributing and profiling
track ballast. An example of one of the complex
on-track machines used to maintain the
infrastructure, which need ‘pre-production’ tests and
trials to prove the man-machine-interface and the
processes required to use the machines effectively.
24
RSSB
Be very careful of other purposes gradually creeping
into a test programme. For example, people might
be interested to ‘see what it will do if …’ Unless the
purpose, justification, protocols, and parameters are
properly designed as part of the test programme
planning, such tests can cause costs and, more
importantly for those seeking to market new products,
delays that are hard to control and that increase
commercial risk.
RSSB
25
5 Real or virtual?
This guide concentrates on testing in the physical
environment; however, advances in technology
during the last few decades now make computational
simulation a worthy alternative. Although the robustness
of most numerical models means that physical testing
will continue to be needed for some time to come,
virtual testing can offer significant benefits. It is not
uncommon in many industries these days, for simulation
to be widely used as a way of avoiding the delays and
costs of traditional approaches to testing and trialling.
But when should the virtual test environment be chosen
over the physical?
In practice the distinction is rarely so marked.
Nowadays a ‘testing’ programme is unlikely to involve
only physical tests or only computer simulation; rather
it is increasingly common for one to inform the other, in
an iterative sequence, throughout the test programme.
For instance, in the automotive industry, testing typically
comprises:
•Laboratory component testing supported by
simulation.
•Full vehicle simulation.
•Only then, full-scale physical crash tests.
A lot of money can be saved by using computer
simulation:
26
RSSB
•At the start, to inform
development of the new
technology, establishing how
performance varies with changes
in design parameters, to optimise
the design.
•Towards the end of the design
process, to assist with problem
solving.
When asked why scale models
were being used for assessing the
impact of wave action on costal
defences, instead of computer
modelling, Professor William Allsop,
Technical Director of the Coastal
Structures Group at HR Wallingford,
said that numerical models aren’t
yet rigorous and robust enough to
commit to building real structures...
to understand how the structures
would work, models were required.
Interview for BBC’s Countryfile
programme broadcast on 20 March
2011.
However computer models will only
ever provide estimates; they are
only as good as the data that goes
into them and that data is likely to
have come from testing, whether
it’s testing to establish material
properties, or testing to establish
boundary conditions that describe
the operational environment.
Furthermore, validation of some
form is usually required and that
ultimately means physical testing at
some stage.
Striking the right balance between
the physical and virtual test
environment is important, based
on the pros and cons of each
technique, but ultimately driven by
the need to make the development
time as short as possible and thus
time to market. Here are a few
RSSB
27
things that can influence that balance:
•Cost – virtual testing is not necessarily cheaper
than physical testing. It can be expensive up front
because computer models need to be developed.
•Time – virtual testing can be quicker because
physical testing requires components to be
manufactured, which takes time particularly for low
production volume prototypes.
•The number of parameters – if the purpose of testing
is to establish the impact of different design features
on performance, then virtual testing is very good at
quickly predicting how performance is affected by
varying large numbers of parameters.
•Availability of the necessary and representative
testing facilities/equipment and engineers with the
necessary competence to manage, plan and conduct
the tests as well as analyse the results.
In their work on whole
train dynamics for RSSB
project T118, Atkins and
AEA Technology Rail
(now DeltaRail) simulated
collisions in a rake of
vehicles studying the
effect of four different
parameters – analysis
and presentation of the
results can be quite
challenging!
28
RSSB
•Availability of suitable computers and, more
importantly, competent engineers to construct the
computer models and analyse the results.
•The availability of suitable numerical models for
virtual testing and how well validated they are – how
well understood are the physical phenomena? Are
the models empirical (based on observations from
previous tests) or deterministic (using mathematical
models derived from theory)? Computer models
have to be validated at some time or other using
physical testing, and some validated models are
available commercially, such as finite element
analysis (FEA) and computational fluid dynamics
(CFD) packages. Validation will only apply over a
specific range of conditions, and by implication these
models will remain unproven outside those ranges,
when further physical testing is likely to be required.
More recently, there has been a trend for parts of
DeltaRail’s VAMPIRE software
is widely regarded as being a
well validated computer model
for simulating the wheel rail
interface and comes with a
number of validated vehicle
models. New vehicle models
are typically validated using
test results from physical
testing of the vehicle as a
whole, or perhaps individual
components.
© DeltaRail Group Ltd
RSSB
29
computer models to be validated by physical testing.
For example, the head of an anthropomorphic test
device (ATD), colloquially ‘crash test dummy’, being
subject to physical testing to validate that part of
a computer model. Remember, the model is only
as good as the data used to build it; garbage in –
garbage out!
.... A computer model of
the vehicle, which has
been validated against
the laboratory ΔQ/Q and
X-factor tests, is used
to predict the Y/Q ratio
as a time history when
negotiating prescribed
track inputs...
Clause 1.2.4.1 compliance method 2 in
GM/RT2141 Issue Three,
June 2009 ‘Resistance
of Railway Vehicles to
Derailment and Roll-Over’
30
RSSB
•Client confidence – the most easily understood
outputs of computer simulation involve watching
animations on a computer screen, which have
seemingly been determined by a grey box with
little human involvement. Intrinsically, this is less
convincing than watching physical tests, even though
the latter will typically cover fewer scenarios and the
test environment may vary. The client needs to buy
in to the approach, particularly if it involves computer
simulation in totality.
More information on computer based simulation can
be obtained from the CFMS consortium, a non-profit
organisation dedicated to driving forward the boundaries
and reducing the cost of simulation-based design
techniques.
RSSB
31
6 Legislation and standards
As explored in Section 4 and in particular Section 4.2,
testing to demonstrate compliance with legislation
and standards is just one of a whole range of reasons
for testing; however it is one of the most important
and high profile. This section has therefore been
devoted to clarifying which legislation and standards
set out relevant requirements; something that can be
particularly confusing for those new to the rail industry.
6.1 Context
In the days of British Rail, requirements for the mainline
railway network were set by the British Railways
Board and much of the testing was done in-house by
that organisation under its Director of Mechanical &
Electrical Engineering or Director of Research. Not
surprisingly things have changed. The railway has
been privatised and now many different organisations
are involved with setting standards and demonstrating
compliance. This has introduced many complexities
with which Section 8 endeavours to assist; however
there has been a more fundamental change.
As part of the European Union, Britain is subject to
European Law. Over the last couple of decades the
European Commission has issued many directives,
standards, and regulations that have a significant
impact on the way we do business. The intent is to free
up markets by reducing barriers to trade and, for the
railway in particular, to work towards an integrated and
32
RSSB
interoperable European railway network. This promises
significant benefits in the longer term, if not sooner,
however in the short term at least it has meant change
and complexity while the railway network is in transition
and gradually becomes more ‘interoperable’. Testing
is no exception and anyone involved in testing on the
railways needs to consider not just domestic standards
and legislation but European also. The rest of this
section elaborates on both of these areas.
6.2 European standards and legislation
The European Commission has a number of statutory
instruments at its disposal to enact legislation and
affect the way industry goes about its business. Further
information on each of these instruments can be found
on the European Commission’s website; however
some of the most relevant to the railway industry are the
following EC directives, the instruments used to align
national legislation:
•The Railway Safety Directive 2004/49/EC, aims
to harmonise the approaches to managing safety
throughout the European Union. Its relevance to
this guide is that it sets out the overall framework in
which testing contributes to proving that something is
safe to operate.
As an EC Directive, the UK government is required
to transpose the requirements into domestic
legislation and this has been achieved through the
Railways and Other Guided Transport Systems
(Safety) Regulations 2006 as amended (commonly
known as ROGS, and last amended May 2013).
RSSB
33
•The Railway Interoperability Directive 2008/57/EC
(RID), sets out the conditions that need to be met to
achieve interoperability of the community’s railway
network. Of particular relevance to this guide is the
amendment of Annexes II, V and VI by Directive
2011/18/EU, which split the verification procedure
for subsystems into two parts: an EC verification
procedure (by a Notified Body (NOBO)) and a
‘verification procedure in the case of national rules’
(by a Designated Body (DEBO)). Further guidance
is provided in Commission Recommendation
2011/217/EU on the authorisation for the placing into
service of structural subsystems and vehicles. As
a recommendation, it is not mandatory but it does
give what is widely recognised to be more clarity and
good practice; member states therefore need to have
a good reason for not following it.
Although the two mentioned above have had the most
significant impact, there are many more directives which
are relevant to the railway industry. Two of relevance to
testing are:
•The Non-Road Mobile Machinery Directive (NRMM)
which sets emissions limits for diesel engines and
of which there are four directives: the ‘mother’
Directive 97/68/EC and three amendments, the most
relevant of which is Directive 2004/26/EC.
•The Machinery Directive 2006/42/EC which
specifies requirements for On-Track Machines when
working on the railway (as opposed to travelling on
the railway between work sites).
34
RSSB
Another significant implication of the RID has been
the development of the Technical Specifications for
Interoperability (TSI). A list of the TSIs and their
current status is provided on the RSSB website along
with more detail on the European legislative arena as it
pertains to rail.
The final pieces of the European legislation and
standards jigsaw are the EuroNorms published by the
European Standards Organisations: CEN (mechanical),
CENELEC (electrical) and ETSI (telecommunications)
which specify standards for components, systems
and processes (including product testing) that are
recognised throughout the European Union. These
standards are not mandatory in their own right, unless
they have been called up by the TSIs, but they may
become mandatory for specific contracts by virtue of
being called up by a contracting entity.
As can be seen, the European legislative and standards
framework is complex. Those needing greater
awareness are encouraged to seek advice from experts
by contacting the relevant organisations listed in Section
10.
6.3 Domestic standards and legislation
We have seen how the European standards system
is gradually developing to improve harmonisation
across member states of the European Community
and ultimately improve efficiency and reduce costs.
However, there are a number of areas where it has not
yet been possible to harmonise practice across member
RSSB
35
states and this is where domestic standards come into
play.
Domestic standards, in particular Railway Group
Standards (RGS), will continue, for the foreseeable
future, to have a number of roles in the context of
the European Standards system. RSSB provides
an explanation of where RGS fit in the European
standards system on its website, and the diagram that
depicts the relationships between the key ingredients
provides a particularly useful summary.
There are many standards that are relevant to testing
on the mainline railway; too numerous to mention in
this guide. However, the high-level standards for each
of the main asset types, are shown in the table on
the following pages. Most of these reference further
relevant standards. Section 10 provides advice on how
to source these standards.
The Railway Industry Association also co-ordinated the
development of a number of ‘RIA Specifications’ prior to
the introduction of CEN and CENELEC standards. Most
have now been superseded by EuroNorms, but more
information can be obtained from RIA’s website.
Network Rail has recently introduced a streamlined
innovation and suggestions scheme which identifies
their challenges and priorities, encouraging those with
relevant ideas to submit an outline of the innovation.
36
RSSB
Asset types
Mandatory
Advisory
Generic (applies
to multiple types of
asset)
GE/RT8270,
Assessment of
Compatibility of
Rolling Stock and
Infrastructure
Not applicable
Rolling Stock
GM/RT2000,
Engineering
Acceptance of Rail
Vehicles (note that, in
response to ROGS,
RSSB has recently
issued a noncompliance against
this standard to allow
alternatives to the
VAB approach that it
mandates)
GM/RC2510, Code of
Practice for Acceptance
Testing of Rail Vehicles
GM/RC2559, Railtrack
Approved Code of
Practice: Safe Testing of
Rail Vehicles on Railtrack
Controlled Infrastructure
GM/RC2515, Engineering
Development of Rail
Vehicles - Code of Practice
GM/GN2688, Guidance
on the Structural Design
of Rail Freight Wagons
including Rail Tank
Wagons
Network Rail Stakeholder
Relations Code of
Practice: Introducing new
vehicles or changes to
vehicles
Annex B gives an overview
of the processes to
support on-track testing
and authorisation for
placing into service
RSSB
37
Asset types
Mandatory
Advisory
On-Track Machines
GM/RT2400,
Engineering Design of
On-Track Machines
Not applicable
On-Track Plant
See relevant NR
company standards
RIS-1530-PLT, Rail
Industry Standard for
Engineering Acceptance
of On-Track Plant and
Associated Equipment
Signalling
GK/RT0045, Lineside
Signals, Indicators and
Layout of Signals
GK/GN0645, Guidance
on Lineside Signals,
Indicators and Layout of
Signals
GE/RT8026, Safety
Requirements for Cab
Signalling Systems
Permanent way
GC/RT5021, Track
System Requirements
GC/RT5033, Terminal
Tracks - Requirements
for Buffer Stops,
Arresting Devices and
End Impact Walls
Civils and
structures
GC/R5112, Rail Traffic
Loading Requirements
for the Design of
Railway Structures
Telecommunications
GE/RT8082, GSM-R
Cab Mobile, Great
Britain Open Interface
Requirements
Electrification
GE/RT8016,
Verification of
Electrification Systems
and Interactions with
Other Systems
38
RSSB
RIS-7700-INS, Rail
Industry Standard for
Station Infrastructure
In an article in the
Evening Standard of
9 March 2011, Peter
Hendy, Commissioner
for Transport for
London, is reported as
saying ‘we will never
give contractors free
access to the railway
at weekends’ adding
that the engineering
work will have to be
done at night, and the
signalling software
tested off-site.
7 Where can we test?
The 24/7 railway and the need to protect performance,
increasingly preclude the possibility of testing in the
normal operational environment. We therefore prefer
to test at dedicated facilities away from the operational
railway. There is an increasing number of cost-effective
facilities available to Britain’s railway industry. These are
provided by Network Rail and a variety of other industry
stakeholders and facility providers, either in Britain or
elsewhere, either specific to the rail industry or other
sectors. These can be found at dedicated test centres,
such as Network Rail’s Innovation & Development
Centre at Melton, or may be single test rigs used by
a supplier or university, but which could be made
available to third parties. Regardless of what equipment
is needed for testing, competent personnel are also
needed; facilities are no good without the expertise
necessary to design and run representative tests,
preferably based locally to minimise logistics and costs.
Not surprisingly, Network Rail also shares TfL’s view.
In the March/April 2010 issue of Rail Magazine,
Jim Morgan, Network Rail’s Principle Programme
Sponsor for ERTMS is reported as saying ‘Anybody
who wants to install ERTMS in the UK on our network
will have to install it on Hertford and prove it will
work’, referring of course to its new Hertford North
Integration Facility.
RSSB
39
Information on facilities available to Britain’s railway
industry for testing and trialling has been collated
by RSSB and made available on the future railway
testing and trialling web page.
Feedback would be welcomed on:
•How to develop this facility page further
•Facilities that are needed but that appear unavailable
Please send any comments to TFSG via
enquirydesk@rssb.co.uk.
40
RSSB
8 How to go about testing
Having explained why we want to test, the legal
framework that applies, and where to find information
on test facilities; this section covers how to go about
testing.
The timeline of any testing programme can be
separated out into four discrete phases as follows:
•Preparation – the most important part of the whole
process.
•Testing – obtaining the results and analysis.
•Reporting – documenting the evidence.
•Reviewing – identifying how it can be done better
next time.
Good practice for each of these phases is outlined on
the following pages.
RSSB
41
New and innovative system of lightweight signals and structures –
delivering high performance and cost savings.
Demanding expectations and short time-scales can be achieved if:
•The supplier has the technology and ability to:
•Think ‘out of the box’
•Apply tested solutions
•The client has a need with driven benefits:
•Bigger rewards/pressures provide increased focus
•Input is captured from stakeholders:
•End product addresses stakeholder needs
•Gets buy-in from users
•Both sides (client and supplier) contribute to the objectives:
•‘Push-pull’ effect
•Don’t do it in one go:
•Get concepts and mandatory requirements right and accepted
•Then attend to the detail
©Variable Message Signs Limited
42
RSSB
8.1 Preparation
Testing is a crucial phase – however it is usually under
time pressure, reputations are often at stake and there
are frequently commercial consequences far in excess
of the value of the item tested hinging on the results.
So it is important that all involved understand the wider
context in which the test is being run and through this it
helps to see why various stakeholders behave as they
do.
So, in the following sections there is continuing
emphasis on the importance of understanding and
addressing the perspectives of all those involved as
tests are conceived, planned, and agreed. During
preparation for the test make sure everybody is clear
whether the test is to open up new understanding or
to converge on a go/no-go decision. As roles and
responsibilities are clarified think also of people’s
incentives as well as their power, personal professional
risk, and their professional capability. The better the
preparation, the better the foundation for cost-effective
and timely success – from the perspective of all parties.
8.1.1. Context and stakeholders
Section 4 discussed ‘why we need to test’, identifying
that testing might take place at any stage of the life
of a maturing technology or product and for differing
purposes. Only at the later stages, when the application
is well-understood and there is a clear specification
does testing for acceptance come to the fore. Before
that there may be broader and less precise questions
RSSB
43
– does this technology or product provide the sort
of performance that would solve our problem at an
acceptable risk? How far can we push the performance
in this particular application? Does this have any
unanticipated consequences? Do we fully understand
the implications of developing or adopting this
technology or product?
Different stakeholders will have different views of the
most important questions to be answered. A supplier
may feel that previous experience abroad should be
evidence of product quality, whereas a customer may
wish to test compatibility with legacy systems unique to
GB rail. A supplier may be anxious to break into a new
market with a reference site, whereas a customer may
not feel the need for yet another supplier to add to their
current portfolio.
Downsides may be different for different people. A
supplier may be sensitive not just to the cost of a test,
but also fear that delay might mean missing a window
of opportunity for lucrative contracts or time sensitive
markets. Hence ‘adding one more test’ might provoke
a reaction disproportionate to the incremental cost.
Similarly, a project manager may be more interested in
limiting the risk to his or her project than in obtaining
system-wide success through innovation or the last few
percent of performance – performance visible only long
after s/he may have been pilloried for an expensive
overrun. The stakes get higher in safety-related
matters. Individuals may be concerned about personal
career risk as well as commercial risk as they decide
44
RSSB
what testing is necessary to demonstrate satisfactory
safety performance. What further testing might they feel
to be necessary to demonstrate their personal diligence
as well as delivering the objective benefits of a test?
Finally there might be the matter of understanding
technology potential. In some instances, customers
may not yet understand the implications of their
requirements – they are still exploring what questions
to ask. Many suppliers are frustrated when they have
not grasped where their customer is on the road to
technology acquisition, or when the customer has not
thought through the system-wide potential of a solution.
Hence the customer and the supplier must share their
perceptions of:
•The maturity of the new technology, product,
subsystem or system.
•The extent to which the interfaces are well
understood or characterised.
•The degree to which the customer understands how
they will use the product or system, and the extent to
which all the requirements have been captured and
specified.
•The extent to which the required performance can be
unambiguously defined in ways that are testable.
•The degree of step change which will need to be
rolled out by the customer, against the commercial
benefit and the risks to either party.
RSSB
45
Only then can the purpose of the test be clarified and
the test be designed to demonstrate what is needed.
8.1.2. Planning
Without exception, all the experienced test engineers,
customers, and suppliers that helped to compile this
guide said that planning is the most important part of the
process.
Testing uses valuable resources; whether stakeholder
time or facility time which can cost thousands of pounds
each day. Effective planning ensures that you use
your resources efficiently in the most expensive phase
of the test cycle – the testing phase itself. As the old
adage goes ‘proper planning prevents particularly poor
performance’!
For the purposes of communication and change control,
the result of the planning process should be captured
in a ‘test plan’. This is the ‘bible’ for the whole testing
programme.
The test plan should be the outcome of discussions
and agreement between supplier(s) and customer(s),
recognising that for complex systems there may be
teams of both supplier’s and customer’s engineers
involved. These discussions should explore the
context for the testing and consider the perspectives
of those involved (Section 8.1). The objective of these
discussions is to negotiate a shared set of expectations
about the tests, their purpose, the criteria for success,
boundaries, the methods and protocols and who is
46
RSSB
going to do what. In other words an agreed way forward
at the outset.
The starting point is to agree what risks the customer
(and also the supplier) wishes to address by testing.
What uncertainties and questions will the test resolve?
What performance needs to be demonstrated – and
why that particular level – and under what operating
conditions or environment?
Knowing this enables the supplier and the customer to
propose and agree constructive and effective tests that
will demonstrate the outcome and deliver the certainty
that both parties seek. What tests will best demonstrate
the performance that addresses the needs (technical
and commercial) of the customer and other stakeholders
that will use the new equipment?
RSSB
47
Difficult though it is, a valuable step is to consider what
the implications might be of a failed test in respect
of a given product. What useful information can be
gathered? What contingencies are available to enable
the product to be taken forward to the benefit of both
supplier and customer and would there be an impact
on price/performance over that originally specified.
Such thinking is particularly valuable during technology
development.
A critical issue, especially for equipment that might have
been used in other applications or on other railways, is
the admissibility of prior test results or prior operational
or maintenance experience. This is where a clear
understanding of the objectives of the tests becomes
so important. If the background concerns are those
illuminated by past test experience – for example
long-term reliability, then past tests or operational
evidence may be useful. But if the concern is about
aspects unique to the application then transfer of past
experience may be harder.
There is increasing interest across the industry in the
philosophy of ‘plug and play’, for example to minimise
field-testing. This shifts the focus of testing from the
performance of the product in the field to the robustness
and consistency of controls over manufacturing – so
testing will move increasingly to the demonstration that
product conforms to type, every time with complete
repeatability and reliability. This shift to testing process
rather than product will require suppliers and customers
to consider anew their approach and the tests they
choose to perform and assess.
Finally, the parties should work together to identify what
48
RSSB
might be the best choice and sequence of tests – what
gives the most insight for the least risk and cost?
And if necessary, loop around the process, revisiting
the questions of benefits and insights from the tests,
understanding the extent to which they address key
concerns and finding the best choice and sequence of
tests, the best use of prior experience and therefore the
most cost- and time-effective approach for both supplier
and customer.
With this negotiated plan in place, all the parties
involved have a view of the future tests, why they’re
needed, their criticality to each party, how they’ll be
assessed and what the implications and impact are on
wider plans.
Without a shared set of expectations, somebody is likely
to get a nasty and possibly expensive surprise during
the test programme.
The agreed test plan should define:
•The objectives of the tests and the success criteria
•What are the concerns and risks to be addressed
by the tests and/or what performance criteria are
being applied and why?
•Roles and responsibilities
•Who is going to do what?
•What is to be tested?
•Consider the boundaries of the equipment to
be tested and give careful consideration to
configuration control.
•Methodology
•Facilities and equipment required.
RSSB
49
The following sections outline many components of the
test plan so, for ease of reference, a checklist has been
provided in Annex A.
8.1.3 Objectives and success criteria
1. Understand why the test is being done, and what the
results of the tests will and will not be used for.
•Resolve concerns and uncertainties
•Explore risks and show them to be under control
•Assess performance and show it meets needs
•Explore and understand boundaries of potential
•Understand insights gained and future potential and
possibilities
•Treat and resolve emergent open points and
unforeseen issues
2. Define the scope of the tests and the objectives;
what does the testing need to demonstrate? Is it
just to demonstrate compliance against mandatory
standards, are there further contractual requirements
to be met, or is it a good opportunity to determine the
safe limits of operation? (See Section 4.) To what
extent can test results and experience from other
environments be used to inform the objectives?
3. Define the requirements that need to be met to pass
the test, including relevant standards that have to be
complied with. Consider the following:
a) The standards may not just pertain to the design
of the new technology and cover approval
50
RSSB
requirements; they may also pertain to its
operation and the delivery of the tests.
b) Ensure that the requirements are fit for purpose.
If they have been defined by the client or a
third party, work with them to ensure they are
appropriate, particularly when integrating new
technology into equipment that may have been
around for decades.
One experienced
testing engineer
referred to a case
where innovative
converter interlacing
on new rolling
stock led to higher
frequency EMC than
that specified in the
requirements. This
led to unexpected
interference with
lineside equipment.
c) Ensure that the requirements satisfactorily
describe how the interfaces with the system/
environment are required to interact with the
equipment or new technology, and vice versa.
Note: The interfaces with the system and
environment may not only be mechanical and
electrical, they may also be human and climatic,
etc. Some may not be controlled so easily when
finally in service operation, such as weather and
vandalism.
d)A further complication is that interface
requirements may not yet be defined, because
the equipment that the technology has to interface
with has not yet been installed. For example
designing electric traction for IEP on the Great
Western Mainline that has not yet been electrified.
4. Define the inputs to the test. Conclusions cannot be
reliably drawn about the behaviour of the test system
if the inputs are not understood; for example: how
can you reliably compare the ride quality measured
over different pieces of track if you don’t know the
track quality and the operating speed over it?
RSSB
51
8.1.4 Roles and responsibilities
Is there an identifiable individual championing the need
for and the progress of the product – someone who is
convinced of the economic and technical merit of the
technology, product, or system, and hence the need
to test it? Experience shows that the best progress
is made when there’s an enthusiastic and committed
person driving the testing forward. And progress is
better still when there are champions inside each of
the main organisations involved so they can push
forward within their own organisations. So identify your
champions – and worry if you don’t have any.
In today’s railway, most testing programmes are going
to involve at least two companies, if not many more.
This isn’t necessarily a bad thing, but it does mean that
there are more risks to manage because there are more
interfaces.
As in any project, stakeholder management and
communication between them is critical for success.
Here are some things to consider when managing
stakeholders, that is, anyone who is affected by the
testing programme, including those who will be doing
the legwork as well the client and approvals bodies:
1. Identify all stakeholders who need to be involved
and think about their interests and concerns (both
corporate and personal). There could be a large
number of different organisations involved: operators,
infrastructure managers, vehicle owners, suppliers
of the equipment, test facility operators, specialist
engineering support, regulatory bodies, neighbours,
52
RSSB
independent assessors, and approvals bodies
(NOBOs), and of course the end clients. They will all
have differing requirements and interests.
Soon after
privatisation in the
mid 90s, South West
Trains fitted sanding
devices to its Class
159 fleet. Resources
from nine different
organisations where
involved in the
development, testing
and fleet wide fitting
on 22 units, yet the
whole programme
took only 10 months.
Large numbers of
stakeholders does
not necessarily mean
extended timescales,
but stakeholders do
need to be managed.
The organisations and individuals involved will
each have different perspectives and concerns.
Explicitly think through the perspectives of these
– as individuals. What’s in it for them? Perhaps
they’re indifferent; for example, is this just another
product test when there are adequate numbers
of alternatives already on the market? Is there a
downside for them? Are they about to take a difficult
decision that might come back to haunt them, and
hence they choose to be more cautious than you
would otherwise expect?
2. Ensure that all stakeholders have a common
understanding of the objectives and the test plan,
especially across disciplines, where the interfaces
typically lie and thus risks. Early engagement with
those authorised to approve the new technology is
invaluable.
3. Obtain agreement from all stakeholders – a word of
caution – this can take more time than the testing
itself. Getting stakeholders engaged in a testing
programme that might not happen for two years may
not be easy – they will have more pressing priorities!
Agreement will be facilitated by obtaining clarity on
how the testing will benefit each of the stakeholders
involved. Getting agreement from any approvals
bodies involved as well as the client is particularly
important.
RSSB
53
4. Allocate roles to all stakeholders – a RASIC matrix is
good practice which identifies those who need to be:
responsible, accountable, supported, informed, and
consulted on each activity. Remember interfaces
present risks and they’re not just confined to trains
and infrastructure; they also occur between people
both within the same company and in different
companies as well. Effective communications are
essential for successful delivery. Something to bear
in mind in particular for testing is whether or not tests
need to be witnessed by the end client, approvals
body or other independent party.
5. Listen and talk to the experts – see Section 10. They
can assist throughout this complex process.
Engage senior management to provide ‘air support’
for the project. Often the decision to introduce an
innovation or new approach is made at a relatively
senior level. It is important for the project managers
to engage with the senior leadership to make sure that
innovative projects are explicitly given additional time
or resources to introduce innovation on the first project
or demonstration – otherwise the innovative approach
will be expected to deliver against the expectations for a
standard approach.
54
RSSB
8.1.5. What is to be tested?
It almost goes without saying, but clearly, any test
programme requires clarity on what is to be tested:
1. Define in an unambiguous way what equipment or
new technology is to be tested; use part numbers
if you can. This may sound straightforward but
when the technology being tested is part of a larger
system with a number of complex and different
interfaces (human, electrical, mechanical) it may
not be straightforward at all – let alone getting all
the stakeholders to agree on it! Large projects use
systems such as DOORS (Dynamic Object-Oriented
Requirements System) for managing requirements.
2. Define the interfaces that the equipment or new
technology has with the system and the environment
in which it is to operate.
3. Maintain configuration control throughout the testing
phase before transferring to an equivalent control in
operational practice
In testing new technology and equipment the issue
of interfaces and environment becomes more critical
because:
•Sometimes
the adjacent systems are not fully
understood or fully characterised because it is an
experimental system.
•Sometimes legacy systems do not provide the full
range of interface functions, signals or data.
•The equipment and the environment may interact in
ways that are not fully predictable.
RSSB
55
8.1.6. Methodology
1. Define the test methodology. Remember standards
are good at specifying such things as controls,
accuracy and error limits, but the correct test must
be done in the first place. There is no substitute for
engaging with the experts (see Section 10).
2. Consider how much testing is required to achieve
the objectives, for example the number of scenarios
that need to be tested (variation in inputs) and the
consistency required from the results (variation in
outputs). Also consider the order and sample size in
which different tests will be done – is this the order
that gives the most important and useful information
for the costs incurred. These are key factors in
planning and project costs. Consider:
a) The risk associated with failure of the new
technology when in its operating environment.
The level of testing should be proportionate to
the risk, for instance, a wheelset will be subject to
much more testing than an interior light.
b) How novel the technology is, the nature of the
novelty, and thus the extent and nature of the
testing needed.
c) Where the technology has been developed and
manufactured.
d) The extent to which the technology has or has not
already been tested outside Britain’s rail industry.
e) The confidence clients have in tests carried
out elsewhere, considering issues such as how
representative such tests might be, the standards
56
RSSB
regime applied, and the familiarity of all involved
with the UK’s operating environment and
practices.
f) The extent to which Britain’s rail operating
environment can be cost-effectively simulated
(whether physically – on a test track for instance,
or virtually – in a computer).
g) The number of tests for each set of input
conditions to get statistically significant results.
3. Consider phasing tests with supplier/customer
reviews between them to allow all those involved to
reflect upon the intermediate findings and to decide
the need for and best direction for completion of
the test programme and any outstanding testing. In
particular, for exploratory testing when the capability
of a new technology or product is being evaluated
such reviews can lead to a much more cost-effective
outcome because lessons and issues are explicitly
managed in context (See Section 8.4).
4. Try to strike the right balance between what could
be foreseeably needed during the testing phase and
what must get done. Also consider:
a) What else could be measured at the same time?
It might not seem essential now, but it could
prove to be later during analysis. For example, if
testing a train’s performance in low adhesion after
applying a suitable lubricant to the track, could it
be worthwhile measuring adhesion between each
train pass and not just at the beginning and the
end?
RSSB
57
b) Combine tests with other tests only when you’re
confident that they won’t affect each other.
5.Identify the risks associated with the test programme
and how they are to be managed. This should
concentrate not only on safety but also the
environment and other business risks; for example,
What might delay the testing programme? What
happens if it doesn’t yield the required results?
Remember also to consider that risks might not
just manifest themselves in the equipment being
tested, but also in the environment. For example,
electromagnetic interference from a new train may
cause station systems, such as lifts, to fail; or even
affect colliery traffic below the railway.
6. Define timescales. Testing programmes do not
always run to plan so contingency and flexibility is
essential:
a) Have all reasonable eventualities been planned for
and variables managed? When testing on service
lines, for instance, be sure that the intended lines
will deliver the required test conditions such as
power supply and track quality. When ride testing,
for example, how will consistent speed and track
quality be delivered?
b) What should be done if the results fall outside
expectations – how will the testing programme be
modified? Such planning might be informed by
prior computer simulation (see Section 5).
c) Build review gates into the plan to prompt
decisions on whether to: proceed, do more
simulation, or revise testing protocols.
58
RSSB
d) If on-track testing is required, consider the amount
of time it will take to get test vehicles to site, take
control of infrastructure and hand it back; these
significantly erode the available testing time. Don’t
underestimate the amount of time it can take to
get approvals to run trains that aren’t already in
the timetable; it can take two years when other
operators need to be involved to give up paths. If
existing train operators are not involved in the test
programme, it can take many months to set up a
track access agreement.
7. Consider dependencies. The tests are likely to
be dependent on completion of several activities
such as supply of the equipment to be tested and
provision of appropriate facilities or equipment.
Consider also what activities are dependent on
successful completion of the tests.
8. Define what evidence is required and needs to
be captured throughout the remaining phases of
the testing lifecycle. Agree the deliverables to be
provided by testing, the pass/fail criteria, and the
reporting requirements.
9. Define how data is to be captured, analysed and
uncertainty established. There are standards for
the instrumentation used to take measurements
during testing and for software used to subsequently
conduct the analysis. Speak to the experts to find
out more (see Section 10).
10. Above all ensure that the test plan is cost-effective,
proportionate to risk and experience, realistic, and
deliverable.
RSSB
59
8.1.7. Facilities and equipment needed
1. Define the facilities and test equipment that are
sufficiently representative of the intended operating
environment to enable the objectives to be met (see
Section 7). Taking test tracks for rolling stock as an
example, considerations will include:
a)Speed of operation.
b)Compatibility of all interfaces – a substantive issue
in many cases.
c)How representative are track quality, and other
factors?
d)Does they need to be representative?
e)Safety of operations, risk management, and
others.
f) Proximity to manufacturing facility.
g)How easily can you demonstrate to the customer
and other third parties that the test track is
sufficiently representative of the intended
operating environment?
Similar principles apply for asset groups other than
rolling stock.
2. Ensure the facilities have appropriate accreditation.
ISO/IEC 17025:2005 ‘General requirements for the
competence of testing and calibration laboratories’
is commonly cited as an appropriate standard,
however it is aimed at laboratories and may not be
appropriate for all testing facilities. It is the highest
level of accreditation for testing and, consequently,
60
RSSB
can be expensive to attain and retain. It might be
more appropriate for facilities to adopt the following
essential principles and be verified by an appropriate
independent party such as a NOBO (Gaskill, 2007):
a) Test facilities are appropriate for the testing
planned.
b) The quality management system complies with
ISO9001:2008 Quality management systems
- Requirements, including procedures for
managing instrumentation and data processing.
c)Staff competency is effectively managed using an
appropriate competency management system.
Note: ‘Competence’ is defined by the Office
of Rail Regulation (2007) as ‘the ability to
perform activities to the standards expected in
employment; it is a combination of practical and
thinking skills, experience and knowledge’. This
has been clarified by the Institution of Railway
Signal Engineers (2009) by reference to work by
Baker & Durrant (2008):
d) The staff who undertake the testing are
independent; have no vested interest in the
outcome of the tests.
The level of assurance associated with the tests, and
thus the level of confidence in them, will dictate how you
can use the results.
RSSB
61
Again, in the case of ‘plug and play’ equipment there will
be a need to test manufacturing facilities and processes
to provide assurance about product consistency. Tests
‘to type’ are a specific domain not covered further here.
8.2. Testing
As has previously been stated, the most important
part of testing is planning. If the planning has been
thorough, the rest of the testing programme stands
the best chance of successful delivery, even when
contingency plans have to be engaged because of
unexpected (but planned for) eventualities.
The testing phase should be all about delivering the
plan set in Section 8.1.2, concentrating on:
1. Checking nothing has changed since the plan was
produced, if necessary, revising the plan accordingly.
2. Ensuring the equipment to be tested arrives on time.
3. Ensuring that the equipment being used to conduct
the test complies with the requirements set in the
plan, and has the necessary approvals.
4. Ensuring everyone is clear on the plan, their roles
and responsibilities.
5. Ensuring that the test conditions and environment
comply with the plan and are adequately described
in the test records.
6. Ensuring that everyone who is taking part in the
testing has the competencies defined in the plan,
including any necessary certification.
62
RSSB
7. Being prepared to be flexible, as even the best plans
cannot cater for everything – be ready to conduct
reviews in the wake of the unexpected.
8. Ensuring data is being collected as planned. As one
experienced test engineer put it, the biggest single
risk of tests failing, is measuring equipment that
performed perfectly well in a lab, not being able to
cope with the harsh railway environment. This can
be due to electrical problems as well as mechanical,
particularly 50Hz interference.
9. Review the emerging results to ensure they are as
expected, and that further tests can benefit from the
experience obtained.
8.3. Reporting
The reporting phase is particularly critical as without
structured, documented evidence, the tests will
have been for nothing. In many cases, it will form
a significant element of the submission made to
third party approvers. In others cases, it will be the
feedback into the product development process. During
technology development the results of tests will be
used to determine the direction and scope of further
development routes.
Reporting should be in accordance with the plan set
in Section 8.1.2 and should set out the following in an
objective and auditable way:
1. What was tested – see Section 8.1.5.
2. The objectives of the test, referencing legislation or
approval requirements – see Section 8.1.3.
RSSB
63
3. How it was tested (the test method), and how it
addresses the objectives – see Section 8.1.6, and
the key parameters and thresholds sought.
4. The facilities used – see Section 8.1.7.
5. The test conditions and environment at the time of
the test.
6. Who performed the tests, citing evidence of relevant
competency.
7. Pass/fail criteria.
8. The test results.
9. How the results compared with the criteria - the
conclusions.
8.4. Reviewing how it went
When the first 3 phases of the test lifecycle have been
completed, the final phase should review how the
whole programme went, to identify learning points,
inform future test programmes, and aid continuous
improvement. Good practice is given in ISO9001:2008
‘Quality management systems – Requirements’ but
this should already be an integral part of all effective
quality management systems.
64
RSSB
9 Special requirements when trialling
Most of the principles that apply to testing will also apply
to trialling; however there are some requirements that
will be specific to trialling.
As explained in Section 3, although the terms can mean
different things to different people, the distinction we
have drawn in this guide between testing and trialling
is that trialling takes place on the operational railway in
revenue earning service.
The level of risk associated with trialling therefore
increases by orders of magnitude, particularly safety
risk, because passengers are involved, not to mention
Not all trials lead to new technology being adopted, as these
remains of Brunel’s atmospheric railway remind us. Carriages
were ‘sucked’ along by a partial vacuum acting on a piston that
ran within the pipe. Leather strips that helped to seal the pipes
proved ineffective leading to overworking of the pumps that
created the vacuum.
RSSB
65
greater numbers of staff. Business risk also increases,
in areas such as operational disruption and loss of
reputation.
Trialling will not therefore take place with new
equipment or technology until the confidence in the level
of benefits to be returned outweighs the risk.
If we consider the development timeline of a new
technology as it progresses through the various
technology readiness levels (see Section 4.1), the
progression from testing through to trialling is achieved
by increasing confidence allowing restrictions in the test
environment to be reduced until it is possible to move
into the (more unrestrained) operational environment.
Trialling does not involve novel equipment unless that
novel equipment has already been subject to testing
that has yielded sufficient confidence.
66
RSSB
‘Note on test tracks – they are
invaluable for getting past first base in
developing the technical safety case
for a new train, and for validating later
changes to the design. Pre-delivery
endurance running on test tracks
is useful as a sophisticated build
quality check, but a true indication
of reliability only emerges from
experience on real infrastructure,
which tends to draw out many more
issues.’
The Twenty Point Plan, Fleet
Reliability Focus Group,
CR/TP 1203 Issue 7, January 2011,
published by ATOC.
You should consider these issues when embarking on
trialling:
•Follow the principles outlined earlier in this guide;
most of them apply equally to trialling as they do
testing. However, bear in mind that because the
risks are different, the controls may also need to be
different.
•The big difference with trialling is that it will be
done on the operational railway in revenue earning
service. That means operation under the safety
management system of the railway undertaking,
RSSB
67
whether it be a train operator or the infrastructure
manager, and approval under ROGS (see Section
6.3). Early engagement with the railway undertaking
and its commitment is essential; without it there is no
point in proceeding.
•The scope of the railway undertaking or the test
facility provider’s insurance may also be an issue,
as trialling new technology may invalidate existing
cover; it has certainly been a problem in the
past following privatisation. Make sure that the
appropriate asset and 3rd party liability cover are in
place.
•Finally, note that trials are new and usually
interesting. They can be a welcome distraction
from delivering the daily routine service, whether
it be operating trains or delivering paths on the
infrastructure. Trials get a lot of a management
attention to ensure they run smoothly and ensure the
maximum chance of success. So if a trial proves to
be successful, bear in mind that when it is rolled out
and integrated into the operational environment there
might not be quite the same level of management
attention and it might not be quite so successful!
68
RSSB
Finally, consider experience from two trials managed by the same
consultancy but on trains operated by two different companies. Both
trials involved adding monitoring and data recording equipment to DC
rolling stock and operating the vehicles in service. One trial involved
developing a complex bespoke distributed data acquisition system,
and the other used more conventional PC-based acquisition with
fewer channels. The former took twice as long to mobilise but not just
because of more complex architecture.
Key messages are:
1 Commitment of the train operator is essential for timely progress.
Their core business is to move passengers reliably from A to B not to
conduct trials per se, therefore any trial that doesn’t contribute to such
aims needs to be incentivised appropriately.
2 Approval processes are there for a reason. The trial which took
longest to mobilise has proven to be reliable in operational service;
design tweaks were required on
the other following an initial period
in-service. Robust independent
review contributed to improving the
design and ensuring it was well
documented.
3 Ensure those involved have
sufficient experience of trialling on
the operational railway, and ensure
that all parties are incentivised to
co-operate with each other and
make timely progress. This needs
to be recognised at the outset and
built into the plans because there
will be an impact on resources
(financial and man-power) as
well as programme. Listen to the
experts!
RSSB
69
10 For more information
The advice given in this guide is only a starting point for those seeking to
test new technology. If more information is needed, most organisations
will offer free initial advice within reason. Contact details are shown
below, along with other information mentioned in this guide that is
available on the internet:
Source
70
Contact details
Advice – CFMS
consortium
More information on computer based simulation can
be obtained from the CFMS consortium:
http://www.cfms.org.uk/
Advice –
Consultancies
Refer to the RIA website for a list of consultancies
that may be able to assist:
http://www.riagb.org.uk
Advice – Rail
Research UK
Association
RRUK-A similarly also has a number of university
based experts experienced in testing:
http://www.rruka.org.uk
Advice – RSSB
RSSB has a number of industry experts experienced
in testing most parts of the railway system and
are particularly well placed to advise on European
issues. RSSB has also prepared advice on the
management of engineering change. Previously
covered in the Yellow Book, this now covered by EC
Common Safety Methods legislation.
http://www.rssb.co.uk
Advice – The
Railway Industry
Association
RIA has a small technical section who may also be
able to assist:
http://www.riagb.org.uk
RSSB
Source
Contact details
Facilities –
Organisations with
facilities available
for testing and
trialling
http://www.futurerailway.org/Pages/
testingandtriallingfacilities.aspx
Legislation –
Domestic
Further advice can be found on the government’s
website:
http://www.legislation.gov.uk
Legislation –
European
Further advice can be found on the European
Commission’s website: http://ec.europa.eu
Network Rail
– Innovation &
Suggestions
Scheme
Outlines of innovations that may provide solutions to
one of Network Rail’s challenges, can be submitted
at http://www.networkrail.co.uk/aspx/12000.aspx
Network Rail
– Product
Acceptance
Information on Network Rail’s product acceptance
process can be found at:
http://www.networkrail.co.uk/aspx/3262.aspx
Standards – British
and European
Standards
Available to purchase from BSI:
http://shop.bsigroup.com
Standards
– European
Standards
Organisations
CEN (mechanical standards):
http://www.cen.eu/cen/pages/default.aspx
CENELEC (electrical standards):
http://www.cenelec.eu/
ETSI (telecommunications standards):
http://www.etsi.org/
Standards –
International
Standards
http://www.iso.org/
See also Section 7.
RSSB
71
Source
72
Contact details
Standards
– London
Underground
Standards
Available by subscription from IHS on line at:
http://www.lulstandards.co.uk/
Standards –
Network Rail
Company
Standards
Available by subscription from IHS on line at:
http://www.ihs.com/en/uk/products/industrystandards/collections/uk-network-rail/index.aspx
Standards –
Railway Group
Standards
Available free of charge from:
Technical
Specifications for
Interoperability
(TSI)
As with many documents of this type, the situation
is continuously changing. RSSB provide a regularly
updated status report on the situation with TSIs
which includes direct links to the TSIs themselves:
http://www.rssb.co.uk/Library/standards-and-therail-industry/TSI_status_summary.pdf
RSSB
http://www.rgsonline.co.uk/default.aspx
11 Acknowledgements
The authors would like to thank many experienced individuals from the
following companies and acknowledge others who preferred to remain
anonymous who have assisted with compiling this guide and without
whom it would not have been possible:
•Alstom Transport
•Angel Trains
•Arthur D. Little
•Association of Train Operating
Companies
•Bombardier Transportation UK
•DeltaRail Group Ltd
•Department for Transport
•Frazer-Nash Consultancy
•Freightliner
•Hitachi Rail Europe
•HR Wallingford
•Independent Glass
•Interfleet Technology
•Kilfrost
•London Undergound
•MIRA
•Network Rail
•Porterbrook
•Railway Industry Association
•Rail Safety and Standards
Board
•Serco
•Siemens
•Technical Programme Delivery
Limited
•Thales
•University of Birmingham
•Variable Message Signs
Limited
•Virgin Trains
RSSB
73
Annex A: Planning checklist
Ref
Element
8.1
Common view of scale of
economic benefits of the
innovation, product, or
technology
8.1.1
Context and stakeholders
1.
Stakeholders and their role
2.
Stance of stakeholders
(up-side or down-side)
3.
Senior staff support and
resources (budget, time) for
the project
4.
Committed project champion
in client organisation?
5.
Existing context and evidence
(existing test data, knowledge)
shared between shareholder
(and agreed)
6.
Stakeholders aligned on what
is needed beyond existing
context and evidence
8.1.2
74
Planning
1.
Performance requirements,
risks, and uncertainties to be
resolved
2.
Tests agreed to meet
requirements and to resolve
risks and uncertainties
RSSB
Covered
Comments
Ref
Element
3.
Agreed scope to use test or
operational experience from
elsewhere
4.
Sequence and timing reviewed
to give best benefits, earliest,
for lowest costs
8.1.3
Covered
Comments
Objectives and success
criteria
1.
Why the test is being done
and what the results will be
used for?
2.
The scope of the tests and the
objectives
3.
The requirements that need
to be met to pass the tests,
including relevant standards
4.
Clarity about timeline for tests
to be done and expectations
about time for client/supplier
to respond, provide feedback
and process results
5.
The interfaces between the
equipment under test and
adjacent systems
6.
The inputs to the test
7.
The evidence required from
the testing
RSSB
75
Ref
Element
8.1.4
Roles and responsibilities
1.
Identify all stakeholders, their
interests and concerns
2.
Ensure a common
understanding of the test plan
3.
Obtain agreement
4.
Allocate roles to all
stakeholders
5.
Listen and talk to the experts
8.1.5
1.
The equipment/new
technology
2.
The interfaces it has with the
system in which it is to operate
3.
Test configuration control
system
8.1.6
76
What is to be tested?
Methodology
1.
The test methodology
2.
How much testing is required
to achieve the objectives
3.
The balance between what
could foreseeably be needed
and what must get done
4.
The risks – not just safety
risks but environmental and
business risks
5.
Timescales taking into account
contingency and flexibility
RSSB
Covered
Comments
Ref
Element
6.
Dependencies – what could
the test programme affect
and what could affect the test
programme
7.
The evidence required from
the testing
8.
How the data is to be
captured, analysed and
uncertainty established
9.
Ensure the test plan is
realistic/deliverable
8.1.7
Covered
Comments
Facilities and equipment
required
1.
Facilities/test equipment that
are sufficiently representative
of the intended operational
environment
2.
Ensure the facilities have
appropriate accreditation
3.
Ensure staff competency
RSSB
77
Annex B: Processes to support on
for placing into serv
78
RSSB
n-track testing and authorisation
vice of rolling stock
© Interfleet Technology Limited 2009
RSSB
79
References
Baker, John and Durrant, Paul, 200 ‘Developing and
Maintaining Staff Competence, Comparisons with Rail
Industry Experience’.
Gaskill, Stephen, 2007, The role of the Notified Body in
the European rail industry, available from http://www.
europeanrailwayreview.com/1525/err-magazine/
past-issues/the-role-of-the-notified-body-in-theeuropean-rail-industry/ Viewed 27/02/2014.
Institution of Railway Signal Engineers, 2009
Competence Guidance for Train-Borne Train Control
Systems.
Office of Rail Regulation, 2007 Developing and
Maintaining Staff Competence .
80
RSSB
Notes
RSSB
81
Notes
82
RSSB
Notes
RSSB
83
RSSB
Block 2 Angel Square, 1 Torrens Street, London EC1V 1NY.
www.rssb.co.uk
Tel:
020 3142 5300
Fax: 020 3142 5663
Email: enquirydesk@rssb.co.uk