Ministry of Defence
Defence Standard 00-42
Issue 3 Publication Date 14 March 2008
Reliability and Maintainability
(R&M) Assurance Guide
Part 3
R&M Case
DEF STAN 00-42 Part 3 Issue 3
Unclassified
Contents
0
Introduction.........................................................................................................................................vi
1
Scope....................................................................................................................................................1
2
Warning ................................................................................................................................................1
3
Related Documents .............................................................................................................................2
4
Definitions ............................................................................................................................................2
5
The Principles of The R&M Case ........................................................................................................3
5.1
Introduction..........................................................................................................................................3
5.2
R&M Strategy Overview ......................................................................................................................6
5.3
R&M Case Review................................................................................................................................7
6
The Application of the R&M Case Through the System Life Cycle..................................................7
6.1
Introduction..........................................................................................................................................7
6.2
Initial R&M Case...................................................................................................................................8
6.3
Tender Stage R&M Case and Case Reports ......................................................................................9
6.4
R&M Case and Case Reports - Post Contract Award .....................................................................10
6.5
Purchaser’s R&M Activities During Procurement ...........................................................................10
6.6
In-Service R&M Case.........................................................................................................................10
6.7
Modifications .....................................................................................................................................11
6.8
Progressive Assurance of R&M........................................................................................................11
6.9
The R&M Programme Plan................................................................................................................12
7
Providing the Evidence .....................................................................................................................13
8
Guidance On How To Present The Evidence...................................................................................13
9
Guidance on How to Assess the Adequacy of Evidence................................................................15
Annex A – General Requirements for the R&M Case and R&M Case Report .........................................18
A.1
R&M Case...........................................................................................................................................18
A.1.1 System Description ...........................................................................................................................18
A.1.2 R&M Requirements............................................................................................................................18
A.1.3 R&M Risk Areas .................................................................................................................................18
A.1.4 R&M Strategy .....................................................................................................................................19
A.1.5 The Evidence Framework..................................................................................................................19
A.1.6 R&M Claims........................................................................................................................................19
A.1.7 Limitations on Use.............................................................................................................................19
A.1.8 Conclusions and Recommendations ...............................................................................................19
A.2
R&M Case Reports ............................................................................................................................19
Annex B – Examples of R&M Risks at the Different Stages of a Systems Life Cycle ............................21
Annex C - CHECKLIST OF POINTS FOR ASSESSING THE ADEQUACY OF EVIDENCE ........................23
Annex D - R&M Risk Reduction Process ...................................................................................................24
ii
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
Annex E - R&M Case Evidence Framework ...............................................................................................27
Unclassified
iii
Unclassified
DEF STAN 00-42 Part 3 Issue 3
Foreword
AMENDMENT RECORD
Amd No
Date
Text Affected
Signature and Date
REVISION NOTE
This standard is raised to Issue 3 to update its content.
HISTORICAL RECORD
This standard supersedes the following:
Defence Standard (Def Stan) 00-42 part 3 Issue 2 dated 6 June 2003
ARRANGEMENT OF DEFENCE STANDARD 00-42
The proposed arrangement of the complete series of Def Stans comprising Def Stan 00-42 is shown below.
Part 1 – One-shot devices/systems
Part 2 – CANCELLED
Part 3 – Reliability and Maintainability (R&M) Case
Part 4 – Testability
Part 5 – In-Service Reliability Demonstrations
Part 6 – Maintainability Demonstrations
a)
This standard provides requirements for accommodating Ministry of Defence (MOD) R&M practices and
procedures throughout the life of the project
b)
This standard has been produced on behalf of the Defence Material Standardization Committee (DMSC)
by the Committee for Defence Equipment Reliability and Maintainability (CODERM). It reflects the
conclusions of consultations among various authorities within the MOD and within industry.
c)
This standard has been agreed by the authorities concerned with its use and is intended to be used
whenever relevant in all future designs, contracts, orders etc. and whenever practicable by amendment
to those already in existence. If any difficulty arises which prevents application of the Def Stan, UK
Defence Standardization (DStan) shall be informed so that a remedy may be sought.
d)
Any enquiries regarding this standard in relation to an Invitation To Tender (ITT) or a contract in which it
is incorporated are to be addressed to the responsible technical or supervising authority named in the
ITT or contract.
e)
Compliance with this Def Stan shall not in itself relieve any person from any legal obligations imposed
upon them.
iv
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
f)
This standard has been devised solely for the use of the MOD and its contractors in the execution of
contracts for the MOD. To the extent permitted by law, the MOD hereby excludes all liability whatsoever
and howsoever arising (including, but without limitation, liability resulting from negligence) for any loss
or damage however caused when the standard is used for any other purpose.
g)
This Part of the standard has been developed to reflect current policy within Defence Acquisition. This
Part of Def Stan 00-42 provides a description of the principles of progressive R&M assurance, and
provides guidance on the content and the ownership of the R&M Case through the life of a system. The
R&M Case is intended to be used throughout the procurement chain from Prime Contractor - Contractor
down to individual sub-contractors and sub-suppliers.
h)
Annex A to this Part of the standard sets out the general requirements for the content of the R&M Case
and R&M Case Report. This report provides a summary of the R&M evidence provided by various
activities and a reasoned argument why the system will satisfy the R&M requirements.
Unclassified
v
DEF STAN 00-42 Part 3 Issue 3
Unclassified
0 Introduction
0.1
Reliability and Maintainability (R&M) are vital performance characteristics that affect the operational
availability, effectiveness, safety and the whole life costs of defence materiel. The provision of defence
materiel with acceptable levels of R&M is therefore essential to the achievement of operational availability
and effectiveness, economy of in-service maintenance support and optimised life cycle costs.
0.2
The R&M achievements of a system depend on all aspects of that system, including components,
processes, hardware, software, people and the man/machine interfaces. R&M engineering is a systems
engineering discipline and so R&M programmes must adequately address all these aspects.
0.3
To satisfy the R&M requirements, they must be fully understood and a suitable strategy with defined
Project Management tasks must be developed. This should include the identification of R&M risk areas and
consideration of how these risks will be managed. The strategy should be flexible in its approach to
providing progressive R&M assurance, in that the results of R&M activities should be reviewed periodically
against the R&M requirements and the R&M programme modified as necessary.
0.4
Def Stan 00-40 Part 1 recognises that different systems and technologies require particular or
unique engineering activities. It now requires that the following objectives shall be satisfied:
a) The R&M requirements of the Purchaser shall be determined and demonstrated to be understood by the
Purchaser and Contractor;
b) A programme of activities shall be planned and implemented to satisfy the requirements, and investigate
the risks identified in 0.3 above;
c) The Purchaser shall be provided with progressive assurance that the R&M requirements are being, or
will be, satisfied and that confidence is building.
The Contractor shall determine and agree with the Purchaser (Procuring Authority) which activities are
required to fulfil the second objective. The third objective is to be satisfied by the provision of progressive
assurance. It is intended that this assurance will be provided by means of a R&M Case.
0.5
The R&M Case is fully defined in Clause 5. It will remain with the system throughout its life and
record all R&M evidence and data from design activities, trials, etc, through to in-service including field data.
0.6
Progress is monitored via R&M Case Reports which are periodic evaluations of the R&M Case and
fully defined in Clause 5.
0.7
This Def Stan provides the Purchaser and Contractors with guidance on how to manage the R&M
Case and also provides guidance on assessing and judging the adequacy of the outputs from R&M methods
used in the programme. The R&M Case is produced by a process that progresses and records achievement
in an Evidence Framework set against the Contractor’s target R&M measures.
0.8
vi
Throughout this standard, the term R&M includes Testability, Durability and Availability.
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
Standards for Defence – RELIABILITY AND MAINTAINABILITY
ASSURANCE GUIDE - PART 3: R&M CASE
1
Scope
1.1
This part of the Def Stan provides a description of the principles of the R&M Case and provides
guidance on its content and application through the life of a system. The R&M Case is intended to be used
throughout the acquisition cycle from concept throughout the Concept, Assessment, Demonstration,
Manufacture, In-Service and Disposal (CADMID) acquisition cycle phases. This part of the Def Stan is not
intended to provide guidance on how to contract for progressive R&M assurance or the R&M Case.
1.2
This Def Stan has five main Sections which describe:
a) The principles of the R&M Case
(Clause. 5)
b) The application of the R&M Case through the system life cycle
(Clause. 6)
c) Providing the evidence
(Clause. 7)
d) Guidance on how to present the evidence
(Clause. 8)
e) Guidance on how to assess the adequacy of the evidence
(Clause. 9)
The activities required for the achievement of R&M will depend on the nature and maturity of the system and
are likely to vary significantly from one project to another.
1.3
Annex A describes the general requirements for the R&M Case and R&M Case Report. The
guidelines in Annex A should not be considered as being prescriptive in nature as they are generic and do
not attempt to be exhaustive.
1.4
Annex B provides examples of R&M risks at different stages of a system’s life cycle.
1.5
Annex C provides a checklist of points for assessing the adequacy of evidence. The checklist
should not be considered to be prescriptive or exhaustive; it is generic and provides guidance to supplement
the generic guidance provided in Clause 9.
1.6
Annex D describes the R&M risk reduction process shown in Figure 8 using illustrative examples
where appropriate.
1.7
4.3.
2
Annex E describes the R&M Evidence Framework, expanding on the information given in Clause
Warning
The Ministry of Defence (MOD), like its contractors, is subject to both United Kingdom and European laws
regarding Health and Safety at Work. All Defence Standards either directly or indirectly invoke the use of
processes and procedures that could be injurious to health if adequate precautions are not taken. Defence
Standards or their use in no way absolves users from complying with statutory and legal requirements
relating to Health and Safety at Work.
Unclassified
1
Unclassified
DEF STAN 00-42 Part 3 Issue 3
3
Related Documents
3.1
The publications shown below are referred to in the text of this standard. Publications are grouped
and listed in alpha numeric order.
Designation
Title
British Standard 4778 Part 3
Glossary of International Terms (Availability, Reliability, Section 3.2
1991 (IEC 50(191 and Maintainability) 1990;
Defence Standard 00-40 (Part 1)
Reliability and Maintainability Management: Responsibilities and
Requirements for Programmes and Plans;
Defence Standard 00-40 (Part 7)
Reliability and Maintainability NATO Terminology ;
Defence Standard 00-49
Reliability and Maintainability MoD Guide to Terminology – Definitions;
Defence Standard 00-60 Part 3
Logistic Support Analysis for Software.
3.2
The following publications may be helpful when using this document:
IEC61508 IEC 61508 (Part 3)
Functional Safety of Electrical/Electronic/programmable electronic
safety-related systems, Part 3: software requirement;
The TickIT Guide
British Standards Institute;
Defence Standard 00-40 through 00-49 Series of R&M Standards and Guides;
Defence Standard 00-56
Safety Management;
Defence Standard 00-60
Integrated Logistic Support.
3.3
A reference in this standard to any related document means that, for any dependent ITT or contract,
the extant edition and all amendments of the document at the date of the ITT or contract applies unless a
specific edition is indicated.
3.4
In consideration of 3.2 above, users of this standard shall be fully aware of the issue and
amendment status of all related documents, particularly when forming part of an ITT or contract.
Responsibility for their correct application rests with those users.
3.5
DStan can advise regarding where related documents are obtained from. Requests for such
information can be made to the DStan Helpdesk. How to contact the helpdesk is shown on the outside rear
cover of Def Stans.
4
Definitions
4.1
The R&M Case is defined as: “A reasoned, auditable argument created to support the contention
that a defined system will satisfy the R&M requirements”. Starting with the initial statement of requirement, it
will subsequently include identified perceived and actual risks, strategies and an Evidence Framework
referring to associated, supporting and mitigating information. The R&M Evidence Framework will include
data from design activities, trials, etc, through to in-service including field data as appropriate, recording any
changes. It will, thus, manifest itself as a top-level control document, summarised periodically through the
issue of R&M Case Reports linked to the Evidence Framework, record progress and remain with the
equipment/system throughout its life until disposal. The R&M Case is, therefore, a progressively expanding
body of evidence (see also Clause 5).
4.2
R&M Case Reports are periodic evaluations of the R&M Case (usually at predetermined points in
the programme as agreed in the Evidence Framework). They report on the evidence, arguments and
conclusions drawn since the last report (ideally referring out to all other forms of evidence including papers
2
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
and data sources wherever possible), providing an assessment of overall R&M achievement/progress and a
review and evaluation of the R&M Strategy and Plan. They may be used to provide sufficient detail to allow
a decision whether to proceed from one phase of a project cycle to the next.
4.3
The R&M Evidence Framework is a matrix of R&M risks that includes:
a) Requirements for evidence to mitigate the risks;
b) Activities necessary to obtain the required evidence;
c) The evidence acceptance criteria;
d) References to the evidence actually provided and confirmation of its acceptance (or rejection).
It provides traceability of the R&M Case process through the life of the system. It is equally applicable to the
Purchaser’s and Contractor’s risks and is typically presented in the form of a matrix.
4.4
A system is defined as “a combination of physical components, software, procedures and human
resources organised to achieve a function”.
4.5
For the purposes of this Def Stan, definitions and clarifications of the terms is provided by:
Defence Standard 00-49: Reliability and Maintainability, MOD Guide to Terminology.
4.6
Supplementary definitions are given in:
British Standard 4778 Part 3 Section 3.2 1991 (IEC 50(191) 1990).
5
5.1
The Principles of The R&M Case
Introduction
5.1.1 As a reasoned, auditable argument created to support the contention that a defined system satisfies
the R&M requirements, the R&M Case provides an audit trail of the engineering considerations from
requirements through to evidence of compliance. It provides the traceability of why certain activities have
been undertaken and how they can be judged as successful. It is initiated at the Concept Phase, revised
progressively during the system life cycle and will be summarised in R&M Case Reports at predefined
milestones. Figure 1 illustrates the concept of building and arguing claims in the R&M Case using the
evidence sources.
5.1.1.1 In practice, the collation of all documentation may be unmanageable, particularly where there are
many and diverse sources of evidence. An acceptable solution is to present the R&M Case as the body of
accumulated R&M Case Reports, which in turn refer out to source evidence (this is illustrated in Figure 2).
5.1.1.2 A number of specialist methods and techniques exist that may be used to generate evidence of
software reliability; these are broadly divided into confidence-building claims based on analysis of the
software development process, and techniques that generate direct evidence of the software’s reliability.
5.1.1.3 All the analyses, strategies, plans, evidence, assumptions, arguments and claims that make up the
R&M Case are to be found within the boundary indicated by the dotted line in Figure 2.
Unclassified
3
DEF STAN 00-42 Part 3 Issue 3
Unclassified
Figure 1 – The Development of Claims
Figure 2 – The Concept of the R&M Case
5.1.1.4 The Evidence Framework captures the current set of compliance and mitigation activities (and their
success criteria) to address the R&M risks. It will typically be presented in the form of a matrix. The number,
4
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
content, objectives and timescales of the R&M Case Reports are determined and prescribed by the
Evidence Framework. This starts with the initial work on the R&M Strategy & Plan and is updated throughout
the project. Each R&M Case Report will reflect the latest state of the Evidence Framework (this is illustrated
in Figure 3). This element of the R&M Case will contain details of the initial requirement and justification of
the proposed solution.
Figure 3 – Establishing and Developing the Evidence Framework
5.1.2 Figure 4 shows the relationship between the R&M activities and the overall process. It also shows
that the evidence produced from the R&M activities may provide input to the Safety Case and the Logistic
Support Analysis Record (LSAR)
Unclassified
5
DEF STAN 00-42 Part 3 Issue 3
Unclassified
General Delivery
Process
R&M
Process
Requirement
R&M
Requirements
Analysis
Analysis
Evidence
Summaries
Requirements
and System
Description
R & M Case
R & M REQUIREMENTS
SYSTEM DESCRIPTION
REASONED ARGUMENT
Strategy
Strategy
Feedback
on Results
EVIDENCE FRAMEWORK
Feedback
on Results
Project Plans:
eg. ILS Plan
Project Risk
Safety
R&M
Quality
R&M
Programme
Plan
Safety Case Report
LSAR
Activities:
eg. Specify
Design
Analyse
Qualify
Produce
R&M
Activities:
eg. Specify
Design
Analyse
Test
Acceptance
Provide
Output
Figure 4 - The Relationship between the R&M Process, the overall Delivery Process and the Evidence
Produced
5.2
R&M Strategy Overview
5.2.1 All projects are to have and maintain an over arching R&M strategy in the R&M Programme Plan.
This is to ensure that the capability being procured demonstrates the required R&M characteristics through
out its life. The strategy should also detail how the R&M characteristics will continue to be monitored inservice.
5.2.2 The Purchaser must determine the R&M requirements and their measurement base. The R&M
requirements should include the anticipated system usage and its environment. In the absence of specific
6
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
direction from the Purchaser, the onus shall be on the Contractor to take the initiative and propose
appropriate R&M design targets and a measurement base.
5.2.3 Through analysis of the R&M requirements, the Contractor should decide upon a robust but
practicable design philosophy for the eventual solution. The consideration of the risks to achievement of the
R&M requirements will result in a strategy for managing the risks and delivering the necessary assurance.
The programme of activities must include verification and 'feedback' in order to mature the R&M strategy and
Programme Plan in response to emerging evidence.
5.2.4
5.3
A description of the R&M strategy is given in Annex D.
R&M Case Review
5.3.1
The R&M Case must be reviewed and updated if:
a) The system is modified.
b) There are changes in how or where the system is used.
c) There are changes to interfacing systems
d) There are changes in the R&M requirements.
e) There is a deviation between actual performance and design intention
6
6.1
The Application of the R&M Case Through the System Life Cycle
Introduction
6.1.1 The balance of effort on the development of the R&M Case during the CADMID life cycle of a typical
system is illustrated at Figure 5.
6.1.2 The lines with arrows indicate how the main effort changes during the life cycle. Information will flow
in the direction of the arrows at that stage; it will also flow backwards and forwards, particularly during the
‘selecting solutions’, ‘demonstrating solutions’ and ‘producing solutions’ phases. This flow of information will
be part of the communications between the Purchaser and the Contractor. Following the “producing
solutions” phase the main effort would remain with the Contractor if contracted to manage support of the
system ‘in-service.’
6.1.3 The role of the R&M Case in the identification and mitigation of R&M risks and in the build up of
confidence in the achievement of R&M has already been discussed. The R&M Case is a living document,
starting with the conception of a system, to meet a capability and continues to be developed through a
number of stages of increasing detail throughout the project. The content and emphasis of the R&M Case
will vary throughout the life cycle depending on the current R&M Risks.
6.1.4 Examples of typical generic risks at the different stages of the system’s life cycle are listed at Annex
B. In addition there will be risks that are specific to individual projects.
Unclassified
7
Identifying
Solutions
Purchaser
Producing
Solutions
Demonstrating
Solutions
Purchaser
Supporting
Solutions
Unclassified
Purchaser
Selecting
Solutions
Concept
DEF STAN 00-42 Part 3 Issue 3
Contractor
Contractor
Purchaser
Contractor
Purchaser
Contractor
Contractor
Purchaser
Disposal
Key:
Purchaser - Main Effort
Purchaser
Purchaser - Monitoring
Figure 5 - Example of Change of Balance of Effort over System Life Cycle
6.2
Initial R&M Case
6.2.1 The Purchaser must determine the R&M requirements and their measurement base. The R&M
requirements should include the anticipated system usage and its environment. In the absence of specific
direction from the Purchaser, the onus will be on the Contractor to take the initiative and propose R&M
8
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
design targets and measurement base. Where necessary references to other documents or evidence will
also be included.
6.2.2 It is essential that all the R&M stakeholders are consulted at this stage to ensure that the
requirements are captured fully. Specialist advice and the use of R&M modelling techniques will probably be
necessary. The Purchaser should also begin identifying R&M risks to be included in the overall project risk
register.
6.3
Tender Stage R&M Case and Case Reports
6.3.1 On receipt of an ITT containing the Initial R&M Case, the Contractor will analyse the R&M
requirements and develop a strategy to satisfy these requirements. Strategy development should consider
the following:
a) Understanding the requirements
b) Analysis of requirements to define system R&M targets
c) Consideration of existing evidence
d) Ensuring an appropriate development process (and/or architecture for software intensive programmes) is
in place.
e) Identification of risk areas (within the overall project risk register).
f)
Pass / fail criteria.
g) Fall-back activities
6.3.2 The development of the R&M strategy will lead to a plan of R&M activities and an outline Assessment
Phase R&M Case Report. The content of the R&M Programme Plan (to be included in the Tender) will be
based on the work to achieve the R&M requirements and mitigate the R&M risks. At this stage the R&M
Case Report should discuss the development of the strategy for providing R&M assurance and document
the justification for the proposed activities within the R&M programme. The objective of developing the
strategy is to provide confidence to the Contractor and the Purchaser that the risk of failing to meet the R&M
requirements is minimised, before committing resources.
6.3.3 It is essential that the Contractor fully understands the requirements. These requirements should be
considered in the widest sense, in that they should not only include R&M requirements, but also other
aspects such as system usage, its environment, the man/machine interfaces, and supportability, all of which
will have an impact on system R&M. To gain this understanding, the Contractor will be involved in dialogue
with the Purchaser. This dialogue will result in a definitive statement of the Purchaser’s requirement and all
the usage and environmental conditions thereof. This statement, articulated in the Contractor’s R&M
Programme Plan, must give confidence that the first objective of the R&M programme (ref. Def Stan 00-40
Pt1 para d) has been fulfilled.
6.3.4 The R&M requirements must be analysed and understood to determine the Contractor’s target R&M
measures. The Contractor’s target value is thus a design aim, to give margin over the R&M requirement. All
the requirements that affect the system R&M must be analysed to determine their impact at system and subsystem level and the results of these analyses form part of the R&M Case. For example, the Purchaser’s
requirements for the environment may be relevant to the system as a whole, but due to their location, some
items will experience very different environmental conditions. (See Def Stan 00-40, Part 1, Annex A.).
6.3.5 The Contractor shall outline the design philosophy and principal design features (referencing and
justifying software development processes where appropriate), and then identify the R&M risk areas for the
proposed design. The key aspects that should be considered are:
a) System maturity (e.g. a new concept versus systems which already exists);
b) The likely R&M performance;
Unclassified
9
DEF STAN 00-42 Part 3 Issue 3
Unclassified
c) Time-scales;
d) Cost.
6.3.6 The risks may be determined using a checklist and techniques such as brainstorming, but relevant
engineering experience and judgement will have a significant input. These R&M risks should be aggregated
with other contract risks into the overall Project Risk Management Plan. The R&M risks and the strategy for
their management form part of the R&M Case.
6.3.7 At the bid stage the R&M Case Report should also include a partially completed
Evidence/Acceptance Framework consisting of the proposed R&M activities, their ‘success criteria’, and the
project milestone(s) at which this evidence will be produced. The Evidence Framework will be updated as
the contract progresses, with references to the outputs from the activities. The ‘success criteria’ will be the
criteria by which the R&M Case Reports will be judged as providing the necessary evidence. The criteria
may be quantitative and/or qualitative evidence, i.e. numeric and/or non-numeric. For example:
a) The success criterion for an improvement in reliability activity might be to grow the system reliability to
0.95.
b) The success criterion for a failure consequence activity might be to identify and eliminate all single point
mission-critical failures.
c) The success criterion for a software development activity might be to demonstrate a robust development
and fault avoidance strategy.
6.4
R&M Case and Case Reports - Post Contract Award
6.4.1 Following Contract Award, the Contractor will review the Assessment Phase R&M Case with a view
to updating the R&M Programme Plan to accommodate any differences between the content of the ITT and
the Contract itself. At this stage the R&M Programme Plan and the success criteria are formalised.
Thereafter, the R&M Case will contain the evidence that the Programme Plan has been followed, reviewed
and evaluated. In addition, it will contain (in the form of data, assumptions and arguments) the evidence that
has been collected and details of the Evidence Framework. At the appropriate milestone in the contract
specified by the Purchaser, the R&M Case should contain sufficient evidence that the applicable
requirements have been achieved and that those still outstanding will be achieved with requisite confidence.
This confidence is based on the provision of progressive assurance that the requirements will be achieved at
the appointed time.
6.4.2 It is essential that the management of R&M is not separated from the design and project
management activities. To this end, it is recommended that the lead for the achievement of R&M should rest
with the Design and Development or Project Manager.
6.5
Purchaser’s R&M Activities During Procurement
6.5.1 During procurement the Purchaser should study the R&M Case Reports produced by the Contractor
and monitor the mitigation of the R&M risks and the progressive achievement of the R&M requirements.
6.5.2 Provision should be made in the contract for an R&M Case Report review and acceptance process,
led by the R&M Panel.
6.6
In-Service R&M Case
6.6.1 Once the system enters service it is important that the R&M performance of the system be
maintained and monitored. Actual performance of the system and changes in the way that it is used or the
environment in which it is used, must be carefully monitored and recorded. Significant changes should be
analysed and corrective action undertaken to restore the levels of R&M identified and incorporated into the
R&M Case where feasible and justifiable. This should be managed using a Data Reporting, Analysis and
Corrective Action System (DRACAS), which in turn forms part of the R&M Case evidence.
10
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
6.6.2 The foundation for the In-Service R&M Case will depend on allocated responsibilities for
management of the system in service. Should the original Contractor be allocated responsibility, it is
expected that the extant R&M Case continues. If the Purchaser is responsible, then it is acceptable that the
In-Service R&M Case develops to the previous interpretation of the procurement R&M Case, but capturing
all evidence from earlier R&M activities and demonstrations.
6.7
Modifications
It is possible that the system will require modifications whilst in-service, involving action by a Contractor.
Should this happen, the Contractor may be required to build a discrete R&M Case using the Purchaser’s
R&M Case as a base line. The Purchaser’s monitoring actions and the results will be captured in the InService R&M Case.
6.8
Progressive Assurance of R&M
6.8.1 In every project there will be potential for shortfalls in R&M performance. The recognition of R&M
risks should prompt the selection of a programme of specific R&M activities as well as the core design
proving activities, which will mitigate the risks. The objective is to build up a body of evidence, which provides
assurance that the R&M requirements are being achieved.
6.8.2 The risks of not achieving the R&M requirements should be evaluated and managed by the
application of risk management practice, which conventionally involves “scoring” of each risk in accordance
with a set of criteria defined at the start of the project. The risk management process will commence at the
bid stage and continue through the development, manufacture and in-service stages. This is an active
process, which will react to changing levels of risk, and the emergence of new risks as the project
progresses.
6.8.3 Figure 6 is presented as an illustration of the manner in which the level of R&M risks change during
the course of a project. The vertical axis represents the level of R&M risks identified at any point in the
project. As the body of R&M evidence increases, the R&M risks reduce and progressive assurance is
obtained. The horizontal axis represents the time into the project, from the bid stage ‘a’, through project start
‘b’, to formal acceptance ‘c’, entry into service ‘d’, mid-life improvement and beyond.
6.8.4 The figure illustrates two types of product development: new development and modified Commercial
Of The Shelf (COTS). At time ‘a’ (Bid Stage) the level of R&M risk is relatively high, but as the project
progresses this level decreases until at ‘c’ (Acceptance) the body of evidence is sufficient to assure the R&M
performance at entry into service. The body of evidence (assurance) should continue to build in-service as
successful trials and usage are recorded and the residual R&M risks can be seen to reduce still further.
6.8.5 It must be recognised that the R&M risks will not always decrease. There will be occasions when
the selection of a different design option, technology insertion or mid-life improvement will render a
proportion of the evidence obsolete, and fresh evidence will need to be generated accordingly. Also there
will be periods when no evidence is being provided, for example during testing, prior to the release of the test
results.
Unclassified
11
DEF STAN 00-42 Part 3 Issue 3
Unclassified
New Development Solution
Modified COTS Solution
Re-design Increases Risk
R&M
Risks
Mid-Life Improvement
(technology insertion)
Increases Risk
plateau (awaiting
further evidence)
EVIDENCE INCREASING
RISKS REDUCING
a
Bid
Stage
b
Project
Start
c
Acceptance
d
Entry Into
Service
e
Mid-Life
Improvement
Time Into
Project
Figure 6 - Illustration of progressive Assurance Process
6.9
The R&M Programme Plan
6.9.1 The R&M Programme Plan and the R&M Case Report are two important documents that support the
achievement of R&M. The R&M Programme Plan should contain a clear description of the management and
organisational structure for R&M and a systematic programme of technical activities for satisfying the
requirements and providing progressive R&M assurance. This should include the maintenance and updating
of the R&M Case.
6.9.2 At the assessment stage, the Contractor’s R&M Case Report may be a section or annex to the R&M
Programme Plan. During the contract, the Contractor’s R&M Case Report may be submitted at predefined
project milestones with a progress report, and finally submitted at the end of the contract as evidence that
the R&M requirements have been satisfied.
6.9.3 The R&M Programme Plan should be considered as a live document. Results from engineering
activities may indicate that one or more R&M measures are not at the expected level and in order to satisfy
the R&M requirements, additional or modified activities must be undertaken. The R&M Programme Plan
should be maintained to reflect these changes necessary to satisfy the R&M requirements.
6.9.4 Some activities traditionally considered to be part of an R&M programme are not included to provide
R&M assurance, but in order to generate information for other disciplines, such as Safety and Supportability.
Although these activities may appear in the R&M programme, they may not necessarily be used to produce
the R&M assurance that appears in the R&M Case.
6.9.5 Alternately, evidence from related activities within the support programme may contribute evidence
for R&M progressive assurance and risk mitigation. The Integrated Logistic Support (ILS) or Supportability
Engineering (SE) programme determines the optimum maintenance strategy for the system. The resulting
maintenance programme and corrective maintenance in particular, generates the data for maintainability
evaluation within the R&M Case. Such evidence should be included in the R&M Case.
12
Unclassified
Unclassified
7
DEF STAN 00-42 Part 3 Issue 3
Providing the Evidence
7.1
During the contract, the R&M Case should bring together all forms of evidence and not just the
results from numerical analysis and testing. These forms of evidence may include:
a) Performance in previous usage;
b) Analysis of Operating Duty Cycle;
c) Design Calculations;
d) Logistic Support Analysis (LSA);
e) Predictions, Modelling;
f)
Testing;
g) Simulation;
h) Expert opinion including previous recorded success of the Contractor;
i)
Correct implementation of best practice;
j)
Contractor’s R&M Cases;
k) Analysis of the software development process.
7.2
During the contract phase, R&M activities will be undertaken in order to generate evidence. It is
essential that the activity results (the evidence) are reviewed progressively against their defined ‘success
criteria’, and also assessed in terms of meeting the ultimate target R&M measures. If the results indicate
that the system R&M performance is not at the expected level for the specific stage of the programme or that
the evidence does not satisfy its ‘success criteria’, then the strategy may have to be modified. The R&M
Programme Plan would be amended to include additional or modified activities.
7.3
As the contract progresses, evidence will be generated, gathered and evaluated in a series of R&M
Case Reports, such that at the end of the contract the Evidence Framework will demonstrate that the R&M
requirements have been met. It is difficult to describe the full range of evidence in its many forms that might
be provided, however, indicative examples of what the R&M evidence might encompass are shown in Annex
E.
7.4
Figure 1 shows that the reasoned arguments in the R&M Case can combine different types of
evidence and also build on assumptions. It is important that these assumptions should be declared openly.
During the R&M programme, the key assumptions should be validated where possible and this effectively
replaces each assumption with evidence. The reasoned arguments enable claims to be made about the
expected R&M performance, and these claims, together with their backing evidence, make up the R&M
Case.
8
Guidance On How To Present The Evidence
8.1
This section provides guidance on how to present the evidence in the R&M Case. The activities
required for the achievement of R&M will depend on the nature and development state of the system and are
likely to vary significantly from one project to another.
8.2
Before undertaking an R&M activity it is essential that the objective of the task is fully understood,
and success criteria defined. The success criteria should be those by which the activity can be judged, and
will substantiate a claim in the R&M Case Report. It should be noted that not all activities lend themselves to
a quantified success criteria, and may require qualitative criteria based on the objectives of the activity. For
example, the success criteria for modelling are not simply that the predictions and modelling show
compliance of the proposed design with the requirements. More important are considerations such as does
Unclassified
13
DEF STAN 00-42 Part 3 Issue 3
Unclassified
the modelling include representative usage of the system or have all system elements (e.g. software) been
included in the modelling.
8.3
Many of the available R&M methods do not deliver R&M assurance directly but when used in
combination with other associated methods. For example, an analysis technique such as Fault Tree
Analysis may identify a design weakness, but improvement and assurance are only provided when the
weakness is addressed at design reviews and corrected with sanctioned follow up action.
8.4
The evidence of R&M assurance does not just result from the generation of the activity results, but
also from the implementation of actions arising from the risk identified by the activity. Undertaking the
activity at the appropriate time such that it will influence the design is very important. Therefore the evidence
from the analysis consists of the documentation showing that actions have been implemented in a timely
manner.
8.5
Activities should be carried out in parallel with the design process, so that the analysis can influence
the design and reflect the final design. If more than one design option is being considered, then each
outcome should be considered separately, so that the implications on reliability and the consequences of
failure can be assessed before deciding on the preferred design option.
8.6
The input to the R&M Case from an activity can be considered to include the following parts:
a) Objective and Success Criteria (what the activity plans to achieve, and defining how and when the
activity has been successful);
b) Outputs (the outputs from the activity);
c) Assumptions;
d) Evidence (how the outputs substantiate claims in the R&M Case Report);
e) Development and Maintenance of Evidence (how will the results of the activity be maintained to reflect
the latest design).
8.7
For software-intensive systems specific techniques may be required to provide assurance of
availability, of the software delivered functions.
8.8
There are two complementary techniques used to achieve software reliability requirements at the
design and implementation phase, the results of which may be used to support the R&M Case. These are:
a) Fault avoidance. This requires that contractors take steps to avoid faults during software development,
and to detect and correct those faults that do occur.
b) Fault tolerance. It is unlikely that fault avoidance will succeed completely, and to achieve high reliability
it is also necessary to design the software to correct or tolerate errors in service.
8.9
With the use of these design and implementation techniques in place, several numerical analysis
techniques may be used to provide evidence of software reliability, which may also be used to support the
R&M Case to show that the system satisfies the R&M requirements:
a) Reliability growth modelling;
b) Statistical testing;
c) Performance testing;
d) Exhaustive testing;
e) Field data and fault data.
14
Unclassified
Unclassified
9
DEF STAN 00-42 Part 3 Issue 3
Guidance on How to Assess the Adequacy of Evidence
9.1
The adequacy of evidence is primarily a function of its practical impact on the reduction of R&M risks,
i.e. progressive assurance. Whilst it is not necessary to assess the adequacy of specific, detailed R&M tasks
in their own right, the visibility, traceability and quality of evidence produced are crucial factors. It is therefore
necessary to confirm that the evidence is generated, managed, validated and used within a closed loop
system of R&M practices and controls, and that it achieves the principal objectives of Def Stan 00-40 Part 1.
Figure 8 illustrates a representative closed loop system (R&M risk reduction process), based upon which,
criteria for assessing the adequacy of evidence are derived in this section.
9.2
Figure 7 comprises four main sections from left to right. The left hand section reiterates the principal
objectives of Def Stan 00-40 Part 1. These represent the highest level criteria for the adequacy of evidence
and need to be borne in mind as key objectives, throughout the risk reduction process. The R&M practices,
which are fundamental to assuring R&M, are high level, ‘objectives-oriented’ processes which depend on the
appropriate traditional R&M activities, e.g. commercial standards (or tailored R&M activities), for specific
analyses, tests and results. The R&M Practices exchange evidence between themselves and output
evidence to the other processes, grouped in Figure 7 as R&M controls. These processes are based on
‘Review the Evidence’ and lead to the ‘R&M Strategy’, ‘R&M Plan’ and ‘Manage R&M’. At any stage,
inadequate evidence should stimulate a review of the ‘R&M Strategy’ and ‘R&M Plan’. ‘Manage R&M’
provides a control feedback path to all of the R&M Practices (and selected R&M activities within them), to
deliver the evidence required. In this way, the R&M Processes are focused at all times on providing
progressive assurance that the risks are being managed. The right hand side of Figure 7 identifies the key
outputs of the risk reduction process, including the ‘R&M Risk Register’ (which is part of the Project Risk
Register), ‘R&M Case & Reports’ and the ‘Assessment of Adequacy’ of the evidence. Note that the latter
does not address adequacy of the R&M Case & Reports intrinsically.
9.3
The R&M risk reduction process is applicable at any project phase, and to any contractual
arrangement. In the diagram, an ‘alpha’ identifier has been allocated to each process block for reference
purposes only. The identifiers do not signify chronological sequence, as the processes are interactive and
often concurrent. Depending on the stage in the acquisition cycle for a new, developing or Off The Shelf
(OTS) system, both the relevance of different practices and the order in which they are conducted will vary.
However, there must always be a Risk Register, R&M Strategy and R&M Plan as core elements of the
process. Wherever possible, the two general practices, ‘Collate Existing Evidence’ and ‘DRACAS/In-Service
Feedback’ should be on-going, spanning different generations and versions of similar systems. All relevant,
available information on the R&M performance and lessons learnt with one system should be used to
provide assurance of R&M in similar systems.
9.4
For a particular development contract, entry into the process is achieved by developing an R&M
Case, using at least the practices (a) to (e) of Figure 7, including R&M modelling/simulation as an activity.
The available evidence (including R&M Risks) is reviewed, an R&M Strategy is determined and R&M
Planning and Management implemented to control and monitor R&M activity. Once the process is
established for a particular project, the R&M Practices and Controls should be implemented and maintained
in the most expedient manner to generate and manage evidence which mitigates R&M risks and provides,
within the acquisition cycle, the earliest possible indication that the R&M requirements have been achieved.
9.5
The principal criteria for assessing the adequacy of evidence are therefore as follows:
a) the evidence as a whole is clearly derived from a closed loop R&M risk reduction process such as
Figure 7, see Annex D for guidance on the principles of specific process blocks;
b) the origin of any specific item of evidence is unambiguously linked to specific R&M Practice(s) and/or
Control process(es);
c) the links between any specific item of evidence and the R&M Risk Register, Strategy and R&M Plan are
shown;
d) the evidence from any particular R&M activity is presented in accordance with Para. 8.6;
Unclassified
15
(b) Assess R&M Requirements
Control
Evidence
(c) R&M Design Philosophy
(d) Allocate R&M Design Targets
(e) Assess R&M Risks
(l) Review the Evidence
Risk Register
(f) Design for R&M
(g) R&M Status/Design Review
(m) R&M Strategy
R&M Case and
Reports
(h) Measure R&M
(i) Plan Transition to Production
(n) R&M Plan
The Purchaser has
assurance that the
requirements have
been satisfied
(k) DRACAS/In-Service Feedback
(o) Manage R&M
Objectives
R&M Practices
R&M Controls
Assessment of
Adequacy
(j) Plan Transition to In-Service
Outputs
Unclassified
A program of
activities has been
planned and
implemented to
satisfy the
requirements
Key:
Insert Annex Title Here(or delete/leave empty if not required)
Unclassified
Figure 7 – R&M Risk Reduction Process
The Purchaser's
R&M requirements
have been
determined and
demonstrated to be
understood
DEF STAN 00-42 Part 3 Issue 3
16
(a) Collate Existing Evidence
Unclassified
DEF STAN 00-42 Part 3 Issue 3
e) the status of each item of evidence, in terms of its relevance, completeness, accuracy and how it has
been used to influence the system and reduce risk, can be readily identified in the Evidence Framework.
9.6
In order to assess the adequacy of evidence, auditable methods/techniques, assumptions, and
detailed results will be sought. Consequently, an open, honest dialogue between partners will be of high
importance. Judgement will be required to assess the evidence presented, including its visibility, traceability
and quality in accordance with the criteria listed above. Annex C provides a checklist of generic points which
are not prescriptive, but which should provide additional guidance on assessing the adequacy of evidence in
appropriate circumstances.
9.7
Annex D outlines the principles of the process blocks in Figure 8 and provides guidance on the
context for assessing the adequacy of evidence against the criteria listed in 9.5. Normally, all of the R&M
Control processes, and the relevant R&M Practice processes, need to be considered for a particular
system/phase, the selection of any particular process being driven by the need to provide evidence against
the R&M risks. Due to the broad scope of the guidance, the Annex does not recommend specific R&M
activities but it identifies examples where appropriate to help illustrate a point.
Unclassified
17
DEF STAN 00-42 Part 3 Issue 3
Unclassified
Annex A – General Requirements for the R&M Case and R&M Case
Report
A.1
R&M Case
The following paragraphs provide the headings and describe the content for sections within the R&M Case.
It is not envisaged that this structure will be suitable for every contract, but it is intended to provide guidance
on the information that should be contained within the reports
A.1.1 System Description
The system description should contain the following items:
a) System description – this should briefly describe the system’s physical or functional characteristics;
b) System boundary – this should describe the system’s physical or functional boundary. Block diagrams
may provide a good method of illustrating the boundary of the system considered in the case;
c) Usage – this should describe the system’s primary role or function, and any secondary roles. It should
include its typical peacetime and wartime mission profiles;
d) Environment – this should describe the system’s operating environments;
e) Interfaces with other equipment/systems – this should define equipment associated with the inputs,
outputs and services to the subject system. Where appropriate, it should also describe such equipment
physically near to the installed system;
f)
Build standard / software version – this should relate to a specific build standard of the system, including
software version(s) where appropriate;
g) Configuration control – to ensure the report reflects the latest build standard/version, the description
should indicate where the latest build standard/version is defined, for example, the Master Record Index;
h) Personnel skill levels and training – the skill level and the training required to operate and maintain the
system should be described;
i)
Maintenance policy – this should describe the support regimes for each of the system’s role or mission
profiles.
A.1.2 R&M Requirements
This should reflect the Purchaser’s requirements and the Contractor’s understanding of those requirements.
The requirements should be considered in their widest context, in that they should include the environment
and usage requirements, as well as the explicitly defined R&M requirements. The Contractor should
describe how the requirements have been interpreted for his proposed design solution and developed into
contract target R&M measures.
A.1.3 R&M Risk Areas
Through analysis of the R&M requirements, the Contractor should identify the risk areas associated with the
system satisfying the R&M requirements, and the how these risks will be, or have been managed during the
contract
18
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
A.1.4 R&M Strategy
Based on the risks identified, the maturity of the proposed design solution and the R&M requirements, the
Contractor should determine a strategy for meeting the requirements and providing the necessary
assurance. This strategy will justify the activities in the Contractor’s R&M programme and identify the
success criteria for these activities.
A.1.5 The Evidence Framework
This section should provide a complete overview or plan of evidence to be provided during the acquisition
phase. It should also show when and by whom R&M Case Reports will be issued. Specific entries in the
Evidence Framework can be selected by the Purchaser and may be matched to payment milestones for
control purposes.
A.1.6 R&M Claims
Typically, the claims will be that the system will satisfy each of the R&M requirements. This section should
provide a reasoned argument why each of the requirements will be met in service, based on the evidence
and any assumptions. All assumptions should be listed explicitly.
A.1.7 Limitations on Use
This section should define the boundaries on system use, which if exceeded will mean the R&M claim may
no longer be valid. These limitations will include the system’s operating envelope, the environment and
important maintenance activities.
A.1.8 Conclusions and Recommendations
This section should contain a diary of the conclusions drawn from the R&M evidence accumulated to date,
including whether the system is likely to satisfy its R&M requirements. In interim issues, it should
recommend whether the project should proceed to its next milestone, or what further work is required to
enable the project to progress. In addition, it should recommend what activities should be conducted in the
future in order to generate the necessary assurance that the R&M requirements will be satisfied.
A.2
R&M Case Reports
A.2.1 The remainder of this annex provides guidance on the content of R&M Case Report. The R&M Case
Report provides R&M evidence at a specific stage within the R&M Case Evidence Framework. The reports
present an argued claim, based on evidence and assumptions that the system will satisfy the R&M
requirements. The report is not expected to contain all the evidence produced for that stage, but will
summarise and act as a ‘signpost’, indicating where the detailed evidence may be found.
A.2.2 This standard jointly refers to R&M, which might be taken to imply that documentary evidence for
Reliability and Maintainability will be summarised in a single report. However, this does not mean that the
R&M Case Reports must be documented in the same report. If the Evidence Framework requires separate
reports, or the Purchaser or Contractor considers that having separate reports will present a clearer picture,
or provide a more focused approach, separate Reliability and Maintainability Case Reports are considered
perfectly acceptable.
A.2.3 Format for R&M Case Report:
a) Introduction. Each R&M Case Report should list and cross reference the parent requirements in the
Evidence Framework, against which the evidence is to be judged, and be traceable to the original
Purchaser’s requirement.
b) Body of Evidence. This should index the existing evidence. Every item of evidence should be crossreferenced to the Evidence Framework, and each of the risks to which it mitigates. The following list
comprises examples of the type of evidence which should be included:
Unclassified
19
DEF STAN 00-42 Part 3 Issue 3
Unclassified
•
Information from relevant similar systems R&M Cases or in-service experience;
•
Contractors’ evidence and R&M Case Reports;
•
Analysis of R&M requirements and operating conditions;
•
Design Philosophy;
•
LSA;
•
Targets;
•
Risks;
•
Strategy, including assumptions, arguments, claims;
•
Plan of R&M Activities;
•
Outputs from R&M Activities; these should include the relevant design R&M calculations, use of best
practice, predictions and modelling, analyses, simulation, testing, DRACAS/In Service Feedback and
expert opinion;
•
On going re-assessment of risks.
The body of evidence should also trace the history of reviews and updates of the R&M design
philosophy, targets, strategy and plan, which keep these in line with the changing status of the original
risks, as well as any new/emerging risks.
c) Review of Evidence To Date. This section should provide a balanced review of the body of evidence in
terms of its completeness, timeliness and acceptability with regard to the criteria contained in the
Evidence Framework.
d) Conclusions. The status of the R&M assumptions, evidence, arguments, claims and residual risks
should be summarised and discussed. Conclusions should be drawn with regard to the status of the
progressive assurance and the activities necessary to mitigate the residual risks.
e) Recommendations. The recommendations should be based on current shortfalls in the evidence
available and propose changes, as appropriate, to the R&M design philosophy, targets, strategy, and
plan in order to maximise the progress towards assuring that the system will satisfy each of the R&M
Requirements.
20
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
Annex B – Examples of R&M Risks at the Different Stages of a Systems
Life Cycle
System Stage
Risks will Include
Concept
Failure of the User to adequately define what they require.
Failure to allocate sufficient financial resource to satisfy the Users
aspirations.
Failure to allocate sufficient time to satisfy the Users aspirations.
Failure of the Purchaser to define the requirements adequately.
Failure of the Purchaser to define the requirements correctly.
Failure of the Purchaser to allocate sufficient resources to achieve
requirements.
Identifying Solutions
Failure of the Purchaser to communicate the requirements to the
Contractor.
Failure of the Contractor to understand the requirements.
Failure of the Contractor to identify all the R&M risks.
Failure of the Contractor to produce an appropriate plan to mitigate
the R&M risks.
Failure of the Purchaser to monitor progress.
Failure of the Purchaser to select the most appropriate solution.
Failure of the Purchaser to modify the requirements where
appropriate.
Failure of the Purchaser to communicate the modified requirements
to the Contractor.
Failure of the Contractor to understand the modified requirements.
Failure of the Contractor to identify all the R&M risks.
Failure of the Contractor to produce an appropriate plan to mitigate
the R&M risks.
Failure of the Purchaser to monitor progress.
Demonstrating the Solution
Failure of the Contractor to work to their plan to mitigate the R&M
risks.
Failure of the Contractor to monitor and review progress and amend
their plan where appropriate.
Failure of the Contractor to capture all available data and process it
appropriately to produce evidence.
Failure of the Contractor to produce sufficient evidence to
demonstrate R&M to the satisfaction of the Purchaser.
Failure of the Purchaser to monitor progress.
Producing the Solution
Failure of the Contractor to identify and manage the R&M risks
associated with the transition from development to production.
Unclassified
21
DEF STAN 00-42 Part 3 Issue 3
System Stage
Unclassified
Risks will Include
Failure of the Contractor to monitor and review progress and amend
their plan where appropriate.
Failure of the Contractor to demonstrate the R&M of their systems to
the satisfaction the Purchaser.
Failure of the Purchaser to monitor progress.
Supporting the Solution
Failure of the Purchaser to ensure that the solution is used,
maintained and supported as specified.
Failure of the Purchaser to monitor changes in use, environment and
support.
Failure of the Purchaser to identify the R&M risks that result from a
change in use, environment and support.
Failure of the Purchaser to mitigate the R&M risks that result from a
change in use, environment and support.
Disposal
Failure of the Purchaser to capture, record and disseminate R&M
‘lessons learnt’ (including evidence) from the project.
Note 1: The Purchaser may choose to delegate some/all of the responsibility for managing these risks to a Contractor.
Note 2: Risks associated with obsolescence have been included in ‘support’.
Note 3: Specific examples of Technical Risk have not been included but should be considered on a case-by-case basis.
22
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
Annex C - Checklist of Points for Assessing the Adequacy of Evidence
C.1
This Annex provides a checklist, which should be considered as a prompt to initiate action where the
checklist points have relevance and does not imply a ‘Yes’ and ‘No’ answer. Judgement is required to
evaluate the evidence presented. The checklist must not be considered as being prescriptive or exhaustive:
it is generic and provides guidance to supplement the general guidance provided in Clause 9 of the
standard.
CHECKLIST:
a) Are the objectives for the activity clearly defined?
b) Has the activity been undertaken in a systematic manner and is it complete?
c) Has the activity been undertaken at a time that will allow influence on the design?
d) Has the usage and environment considered for the activity been documented?
e) Has the physical and functional boundary of the activity been defined?
f)
Are any assumptions defined (e.g. inputs from other systems or services), and are they realistic and
reasonable?
g) Is justification given for the activity method/technique used, and is it reasonable?
h) Who was consulted during the activity, (e.g. user, maintainer, designer etc)? Was this level of
consultation reasonable?
i)
Are the activity recommendations clearly defined, and are they reasonable?
j)
Does documentary evidence indicate that the recommendations have been implemented?
k) Have the activity results been progressively updated to reflect the latest design, and are these being
used as an input to design reviews?
Unclassified
23
DEF STAN 00-42 Part 3 Issue 3
Unclassified
Annex D - R&M Risk Reduction Process
D.1
This Annex presents the principles of the process blocks, which are illustrated in Figure 8 (repeated
below). It contains guidance on the context for assessing the adequacy of evidence against the criteria of
Clause 9.5. The process identifiers and titles from Figure 8 are listed below, followed by an indication of the
principles of what the evidence should comprise and how it would relate to other processes. Illustrative
examples are included, where appropriate. Normally, all of the R&M Control processes, and the relevant
R&M Practice processes, would need to be considered during each acquisition phase:
a) Collate Existing Evidence; information from relevant similar system R&M Cases or in-service R&M
experience, including both successes and failures; arguments and justification for using it to mitigate
risks and generate assurance; confirmation of an on-going commitment to R&M improvement.
b) Assess R&M Requirements; interpretation of operating and maintenance scenarios, environmental
profiles and the understanding of:
•
the Purchaser’s R&M Case;
•
permissible/degraded modes of operation;
•
mission dependent operational success and failure criteria;
•
qualitative and quantitative R&M requirements;
•
storage and transportation needs;
•
design loads.
c) R&M Design Philosophy; a process of continuous improvement and commitment to providing assurance
of robust design, eliminating known failure mechanisms or increasing time to failure to an acceptable
level; fault avoidance techniques for software development; specific R&M design criteria (e.g.
redundancy/fault detection and recovery) necessary to achieve the assessed requirements (b), using
lessons learnt from previous systems (a); necessary R&M resources to implement the philosophy.
d) Allocate R&M Targets; based on appropriate R&M analysis and modelling, quantitative/qualitative
targets allocated to subsystems/components; confirmation that these are practicable, realistic and fully
consistent with the R&M requirements.
e) Assess R&M Risks; formalised, systematic identification and evaluation of R&M Risks; details of
technical and timescale risks with regard to the R&M requirements; input to and maintenance of the
R&M Risk Register detailing the current status, whether open or closed, of all identified R&M risks;
integration with the Project Risk Register; links to the R&M Strategy (m) and re-evaluation of risks in
order to assess the risk reduction available from R&M Practices contained in the Strategy; links with the
Evidence Framework.
f)
Design for R&M; design to meet the R&M Requirements, Design Philosophy, Targets and Strategy (b),
(c) and (d) and (m); system description/justification; design margins; confidence in R&M at internal and
external interfaces.
g) R&M Status/Design Review; evidence of independent assessment and review of design decisions,
analyses and tests that impact on R&M, documented to provide an auditable trail; R&M contributions to
formal Design Reviews; confirmation that the status of R&M processes and activities are in accordance
with the Strategy and Plan.
24
Unclassified
(a) Collate Existing Evidence
The Purchaser's
R&M requirements
have been
determined and
demonstrated to be
understood
(b) Assess R&M Requirements
Control
Evidence
(c) R&M Design Philosophy
(d) Allocate R&M Design Targets
(e) Assess R&M Risks
(l) Review the Evidence
Risk Register
(f) Design for R&M
(g) R&M Status/Design Review
(m) R&M Strategy
R&M Case and
Reports
(h) Measure R&M
(i) Plan Transition to Production
Unclassified
Unclassified
A program of
activities has been
planned and
implemented to
satisfy the
requirements
Key:
(n) R&M Plan
The Purchaser has
assurance that the
requirements have
been satisfied
(k) DRACAS/In-Service Feedback
(o) Manage R&M
Objectives
R&M Practices
R&M Controls
Assessment of
Adequacy
(j) Plan Transition to In-Service
25
DEF STAN 00-42 Part 3 Issue 3
Outputs
DEF STAN 00-42 Part 3 Issue 3
Unclassified
h) Measure R&M; results of R&M tests on specific components/subsystems and a representation of the
final system, as determined by the Strategy and Plan; software performance testing; confirmation of use
and impact of DRACAS (k) to mitigate all recorded failures.
i)
Plan Transition to Production; plan for the design and validation of manufacturing systems and
processes in order to protect critical items, realise the design performance and quality requirements and
safeguard a smooth transition to production.
j)
Plan Transition to In-Service; plan for R&M risk control activities and R&M demonstration plans for the
transition to in-service.
k) DRACAS/In–Service Feedback; all incidents should be subject to DRACAS; relevant lessons learnt and
corrective actions determined from this and previous systems; confirmation that these have been
implemented on the system.
l)
Review the Evidence; this should be a routine review of all evidence arising and conducted,
exceptionally, whenever the findings from any of the R&M Practices need to be considered urgently; all
items of evidence should be specifically linked to, and traceable with regard to, the R&M Strategy (m),
Risk Register (e), R&M Plan (n) and the Evidence Framework.
m) R&M Strategy; this forms the response to the assessment of available evidence (l); creation and
maintenance of a structure of Assumptions, Claims, Arguments and Evidence needed to assure R&M;
definition of appropriate R&M Practices necessary to deliver the evidence; the assumptions must be
justified and the claims must be practicable and realistic, based on what has been achieved in the past;
the Strategy should be linked to the R&M Requirements and justified by a re-assessment of the R&M
Risks (e), assuming the Strategy had been implemented; timescales and acceptance criteria for the
required evidence, should be derived; R&M Test Strategy, covering the R&M functional testing
provisions, should also be derived; specific requirements for the R&M Plan; definition and development
of the Evidence Framework to manage the evidence requirements.
n) R&M Plan; this responds to the R&M Strategy by defining a schedule of specific R&M activities, e.g.
system R&M modelling, FMEA, accelerated testing etc., in context with the appropriate R&M practices;
evidence and justification for the task selection, clearly linked to risk control activities of the strategy;
success criteria, including outputs and milestones; R&M management, monitoring and control activities
and review schedules; links with the Evidence Framework.
o) Manage R&M; to complete the loop, evidence of R&M Management, including status of the plan, should
feed into (l).
26
Unclassified
Unclassified
DEF STAN 00-42 Part 3 Issue 3
Annex E - R&M Case Evidence Framework
E.1
The R&M Evidence Framework is a matrix comprising the R&M requirements, the R&M risks
associated with achieving and satisfactorily delivering the requirements, and one or more programmes of
activities to mitigate such risk and enable the requirements to be delivered. As with the Risk Register the
R&M Evidence Framework is a living document which needs to be maintained throughout the acquisition
cycle to enable assurance and confidence to be developed in the solution and presented through the
population of the R&M Case.
E.2
R&M Programme Plan shall comprise a number of discrete activities that, when integrated together,
will be employed to add value to the project and mitigate these risks. For each activity; the plan will require:
(a) The aim of the activity;
(b) The pass and fail criteria for that activity;
(c) The method by which success will be measured;
(d) The fall back activity in the event of failure.
Together these criteria will form part of the evidence framework and in due course populate the R&M Case.
E.3
In these early stages of the project such plans will be generic recognising the requirement for
development and growth, and later that of prove and demonstrating compliance. Potential contractors will be
required to specify exactly how they intend to mitigate the perceived risks accompanied by detailed
proposals to provide just such evidence. Where resources are limited alternative strategies will be required,
hence the importance of addressing these at the earliest opportunity and recognising that one R&M strategy
may not fit all.
E.4
The R&M Case Evidence Framework is defined in Clause 4.3. Suitable column headings and
contents are described below:
Column No.
Heading
Contents
1
Project Phase
Relevant phase in the product life cycle
2
Risk Ref
Relative to the Project Risk Register
3
Risk Description and
initial Risk Score
Relative to the Project Risk Register
4
Risk Cause
A description of the underlying cause of the risk
5
Evidence Required
Evidence needed to mitigate the risk (information, not deliverable
reports)
6
R&M Practice
Process required to generate the necessary evidence (usually a
combination of traditional R&M and other activities, i.e. not
necessarily an individual R&M Activity or Technique)
Initial Risk Score
This is the R&M Risk Score which, in the
examples, is assumed to be on a scale of 0 to 1, for increasing
risk
Acceptance Criteria
7
Evidence
Deliverable document/contents
8
Target Risk Score
Acceptable risk score
9
Time Due
Time the evidence is due
Acceptance Status
Unclassified
27
DEF STAN 00-42 Part 3 Issue 3
Unclassified
Column No.
Heading
Contents
10
Evidence
References to the latest evidence, including
Issue No, and date delivered
11
Approval
Signature of approving authority and date of acceptance
E5
Two example R&M Case Evidence Frameworks are illustrated below. Each covers a selection of
risks at various stages in the system life cycle, assuming the system involves substantial development
activity. However, the examples are not intended to cover all the R&M Risks with the system, and should not
be used as a template. When creating an Evidence Framework, the system should be considered in its own
right. The risk scores are arbitrary and, in practice, should be used to prioritise the R&M activities.
28
Unclassified
Evidence Framework for System ‘X’
Issue:
Date:
Signature:
Acceptance Criteria
Risk
Ref.
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
Assessment
RM
003
There is a risk that
the system will fail
to achieve its
specified reliability:
99.9% over a 24
hour BFM.
Intrinsic
reliability of
solution
components not
assessed/
understood
Part Failure Rates
Parts count Prediction
using in-service experience
of similar parts, defaulting
to industry standard data
sources, e.g. generic failure
rate data if no other data
available.
Initial Score = 0.25
Failure modes
and criticality of
solution(s) not
assessed/
understood
Mission critical
failure modes and
failure rates for
single and double
faults.
FMECA: this information
will be provided by the
design FMECA, conducted
as design practice (OP
Procedure 21).
Evidence
Acceptance Status
Target
Risk Score
Time Due
System reliability RBD
modelling and analysis for
24 hour BFM, based on 1)
to 3) above.
Solution does not
perform as
predicted in
service
In-Service
Demonstration of
ARM
Monitoring and reporting
of in-service defects
through Defect Reporting
via analysis of DRACAS
database.
28
Draft ISRD Plan,
including acceptance
criteria with regard to the
24 hour BFM
requirement.
ISRD Report showing
compliance with the
requirement.
2 weeks prior
to Final
Design Review
1 year after
entry into
service
DEF STAN 00-42 Part 3 Issue 3
System reliability
over 24 hour BFM
Approval
Signature:
Date:
2 weeks prior
to Preliminary
Design
Review,
update prior to
Critical Design
Review
Development Tests and
DRACAS: a) to support
previous assumptions on
failure modes and failure
rates; b) to trigger further
development and testing of
unsatisfactory items and, c)
to initiate selection of
alternative parts.
Reliability of
solution system
not assessed/
understood
Evidence
Reference:
Issue No:
Date
Delivered:
Unclassified
Unclassified
Project Phase
Issue:
Date:
Signature:
Acceptance Criteria
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
Assessment
(cont)
RM
022
There is a risk that
the system BIT
requirements will
not be achieved.
No clear approach
by the Purchaser
to Testability
Testability Design
Strategy
Testability
Requirements not
verified
System
integration fails to
address wild heat
and maintainer
access
Evidence
Target
Risk Score
Time Due
1) Review BIT Design
Strategy in the light of the
functional hierarchy
developed in the FMECA
(see RM001).
Internal document
providing results of the
Review and showing that
the Testability Design
Strategy is consistent
with the functional
hierarchy and the BIT
requirements for:
Start-up
Continuous checks
Diagnostics
Location.
0.2
6 weeks prior
to Critical
Design Review
Testability
Evaluation
Extension of the FMECA
to provide an evaluation of
BIT coverage.
BIT Evaluation Report
showing that the system
testability is consistent
with BIT requirements
for:
Start-up
Continuous checks
Diagnostics
Location.
0.1
6 weeks prior
to Final
Design Review
Technology
Demonstration
Programme (TDP) to
include R&M
assessments
R&M predictions
conducted to support the
TDP
TDP R&M Prediction
report
0.1
6 weeks prior
to Critical
Design Review
Initial Score = 0.3
Unclassified
RM
023
There is a risk that
the integration of
latest
communications
equipment into the
design will
compromise the
R&M performance
of the complete
system
Initial Score = 0.3
Acceptance Status
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
DEF STAN 00-42 Part 3 Issue 3
Risk
Ref.
DRAFT
Unclassified
Unclassified
Unclassified
Project Phase
DEF STAN 00-42 Part 3 Issue 3
DEF STAN 00-42 Part 3 Issue 3
30
Evidence Framework for System ‘X’
Evidence Framework for System ‘X’
Issue:
Date:
Signature:
Acceptance Criteria
Project Phase
Risk
Ref.
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
Demonstration
RM
032
There is a risk that
sub system X will require
unscheduled
replacement
Initial Score = .35
Wear out
mechanisms not
fully understood
Evaluation of
expected life and
determination of any
changes necessary to
achieve the
requirement.
RM
044
There is a risk that
the chassis will
suffer from
stress/fatigue
during system
assembly
Initial Score =0.28
Assembly
activities include
loading that is
very different
from when
system is
complete
Target
Risk Score
Time Due
Review of life data on
similar items and
environmental evaluation
/stress calculations, to
determine ageing factors
and critical components.
Stress Calculations,
justification of (any)
necessary design changes
and Accelerated Life Test
Plan.
0.2
3 months after
Contract
Award
Accelerated Life Testing
using the Highly
Accelerated Life Test
methodology.
Accelerated Life Test
Report providing
assurance for the final
design.
<0.15
6 months after
receipt of test
model(s)
Evaluation of the Load
Case.
Report, including analysis
and calculation records,
showing stress margins.
The report will highlight
(any) areas of potential
overstress and justify
changes, if needed, to
ensure adequate margins.
0.2
3 months prior
to completion
of
Demonstration
Phase
Production Reliability
Acceptance Test.
PRAT Test Plan.
PRAT Test Results
assuring the integrity of
the chassis for
manufacture.
<0.1
PRAT Plan
required before
start of
Production.
PRAT Test
Results
following
completion of
PRAT.
Final Quality Inspection of
Deliverables.
Quality Inspection
Records.
<0.1
During
production
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
28
DEF STAN 00-42 Part 3 Issue 3
Analysis of loads on
the chassis when
suspended;
determination of
changes to the
chassis design and/or
the manufacturing
fixture(s) to ensure
that the expected life
of the chassis is not
compromised.
Demonstration of
manufacture
Evidence
Unclassified
Unclassified
Manufacture
Acceptance Status
Issue:
Date:
Signature:
Acceptance Criteria
Risk
Ref.
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
Concept
RM
001
Inadequate/Incorrec
t definition of
R&M requirements
by Purchaser.
R&M aspects not
addressed within
Systems
Engineering,
therefore impact
of R&M on
capability not
fully understood.
R&M
requirements are
not SMART.
Critical R&M
System Attributes.
Operational
Availability Targets.
Adequate System
Numbers.
Initial R&M Targets
linked to Operational
Availability.
Assessment of the
impact of R&M on
Operational
Effectiveness.
Failure of the
Purchaser to
allocate sufficient
resources to
achieve
requirements
Purchaser fails to
realise the link
between R&M
and Cost of
Ownership
Initial risk score =
0.7
The Purchaser
fails to consider
the impact of the
requirements on
the need for
complex or novel
technology
Initial risk score =
0.4
Unclassified
RM
002
Acceptance Status
Evidence
Target
Risk Score
Time Due
Capability Gap Analysis.
Operational Analysis.
Needs & Numbers studies
(with R&M input).
Availability modelling.
Operation Availability
Targets within the
Sustainability section of
the User Requirements
Document (URD).
First cut R&M targets for
insertion into the Systems
Requirement Document
(SRD).
0.1
Early in the
concept phase,
prior to Initial
Gate Business
Case.
Assessment of the
impact of R&M on
Funding.
R&M Input into BOI,
COEIA and WLC
modelling.
Realistic estimates of
funding and equipment
numbers by ensuring
Availability and
Reliability are included in
these early studies as cost
drivers.
0.2
Early in the
concept phase,
prior to Initial
Gate Business
Case.
Assessment of the
technology risks.
Feasibility studies
examining the maturity of
the technology likely to be
used in the solution
options.
Reports showing pull
through from research.
Formulation of
technology demonstrator
programmes. Input from
industry through
partnering.
0.2
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
Unclassified
Project Phase
DEF STAN 00-42 Part 3 Issue 3
28
Evidence Framework for System ‘Y’
Issue:
Date:
Signature:
Acceptance Criteria
Project Phase
Risk
Ref.
Risk Description
and initial risk
score
Evidence Required
R&M Practice
Purchaser fails to
understand the
key timescale
risks
Assessment of the
Time-scale risks.
Purchaser fails to
outline the
strategy for R&M
assurance
R&M supporting
evidence is
developed in an
adhoc manner
Target
Risk Score
Comparison with similar,
related or historical
projects.
Analyses to show the
time-scales have been
planned in accordance
with the technology and
technical risks.
0.2
Suitable R&M
strategy
An R&M strategy agreed
by the R&M Concept
Panel.
A draft R&M strategy
paper outlining the
elements work and the
strategy to mitigate the
key risks.
Statements of work for
R&M assessment phase
studies in Initial Gate
Business case.
Draft ITEAP to outline
acceptance criteria.
0.2
RM
003
Failure of the
Purchaser to
communicate the
R&M requirements
to the Contractor
Reliability
predictions based on
similar equipment
failure rates, and any
factors applied due to
differences in BFMs,
usage, complexity
etc.
Details of
Operational Analysis
and Operational
Availability studies.
Initial R&M Case Report
issued to Contractor
providing an audit trail of
how the R&M requirement
was derived. Discussions
between Customer and
Contractor
Clear R&M requirements
with the requisite audit
trail set out at the start of
the R&M Case.
Initial R&M risk register.
0.1
Initial Score = 0.4
Time Due
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
Unclassified
Unclassified
Evidence
Unclassified
Assessment
Acceptance Status
Early in the
assessment
phase, prior to
Main Gate
Business Case.
28
DEF STAN 00-42 Part 3 Issue 3
Risk Cause
DEF STAN 00-42 Part 3 Issue 3
28
Evidence Framework for System ‘Y’
Issue:
Date:
Signature:
Acceptance Criteria
Project Phase
Risk
Ref.
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
RM
004
Failure of the
Purchaser to select
the optimum
solution.
Selection
mechanism fails
to rigorously
address R&M
Reliability estimates
of concept options.
Target
Risk Score
Time Due
Assessment studies
allowing R&M estimates to
be input along with other
attributes into a structured
concept down selection
system.
Concept down selection
reports showing evidence
that the R&M has been
factored into the selection
process.
0.1
In the
assessment
phase, prior to
Main Gate
Business Case.
An adequate ITT/bid
R&M assessment
method with
sufficient weighting
on R&M.
Draft R&M assessment
questions for ITT marking
scheme, ensuring R&M is
given equal weighting to
performance, time & cost.
Final scores from the bid
assessment, plus key
R&M risks and R&M
achievement Milestones
necessary to contract for
the production of R&M
evidence within the R&M
case.
0.1
In the
assessment
phase, prior to
Main Gate
Business Case.
Assessment of the
technological risks
associated with each
option.
Assessment of likely
software complexity
through measurements of
the size and complexity of
the software
Concept down selection
reports include
assessment of software
R&M through life
Project R&M Design
Guidelines and define how
these guidelines are to be
contracted against.
Obtain Stakeholder
acceptance for the Project
R&M Design Guidelines
Unclassified
RM
005
Failure of the
Contractor to
understand the
requirements.
Initial Score = 0.6
Contractor fails to
formally assess
the impact of the
environmental
constraints on
R&M
Analysis of the operational
environment, and mission
and peacetime operation.
Contractors R&M Case
Report showing the
environmental factors and
duty cycle loads have
been understood and will
influence the design.
0.1
Provided with
the Tender or
early in the
demonstration
phase.
Analysis of duty
cycle, loads,
temperature levels,
vibration levels.
Analysis of damage
accumulation effects,
dust, dirt ingress,
moisture etc.
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
Unclassified
Evidence
Initial Score = 0.5
Demonstration
Acceptance Status
DEF STAN 00-42 Part 3 Issue 3
28
28
Evidence Framework for System ‘Y’
Issue:
Date:
Signature:
Acceptance Criteria
Project Phase
Risk
Ref.
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
Demonstration
(cont)
RM
006
Failure of the
Contractor to
recognise all of the
R&M risks.
Contractor fails to
involve R&M
staff within
formal Risk
Identification
Identification of all
R&M risks.
Analysis of the strength of
design of critical
components against duty
cycle loads. Predictions
and modelling to identify
critical systems. Analysis
of interface and integration
issues.
Initial Score = 0.5
Acceptance Status
Evidence
Target
Risk Score
Time Due
Comprehensive risk
matrix supported by
R&M Case reports
showing:-
<0.15
Early in the
design process
to influence
the design of
prototype
equipments.
Approval
Signature:
Date:
Unclassified
Unclassified
Unclassified
28
DEF STAN 00-42 Part3 Issue 3
Modelling using the
measured inputs (loads),
to ensure that the strength
of design of mission
critical sub-systems and
components is adequate
to meet the needs of the
mission, and have the
durability to continue to
function for the design
life of the equipment.
A structured analysis of
potential failure modes to
ensure that all interface
and integration issues are
addressed and are not
overlooked as causes of
unreliability.
Reliability modelling,
predictions and
allocations to determine
criticality.
Evidence
Reference:
Issue No:
Date
Delivered:
DEF STAN 00-42 Part 3 Issue 3
28
Evidence Framework for System ‘Y’
Issue:
Date:
Signature:
Acceptance Criteria
Project Phase
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
RM
007
COTS components
fail to perform as
expected
Unsuitable COTS
components used
within the design
R&M Predictions
supported by inservice data for
COTS sub systems.
Design fails to
make use of
automated usage
and fault
reporting
HUMS to be
implemented
effectively and
efficiently as part of
the design process
Initial Score = 0.5
Unclassified
RM
008
Platform condition
and usage unknown
Initial Score = 0.4
Acceptance Status
Evidence
Target
Risk Score
Time Due
Assessment studies where
R&M estimates for COTS
sub systems consider
existing data and the
impact of differences
between the new
application and that of
applicable to the source
data.
Concept down selection
reports include realistic
predictions for COTS Sub
Systems R&M
Performance.
0.15
Provided with
the Tender or
early in the
demonstration
phase.
HUMS to be an integral
part of the design process.
Maintainability Analysis
Report identifying
functions covered by
HUMS
0.1
Provided with
the Tender or
early in the
demonstration
phase.
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
Unclassified
Risk
Ref.
DEF STAN 00-42 Part 3 Issue 3
28
28
Evidence Framework for System ‘Y’
Issue:
Date:
Signature:
Acceptance Criteria
Project Phase
Risk
Ref.
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
Demonstration
(cont)
RM
009
Failure of the
Contractor to
produce an
adequate plan to
mitigate the R&M
risks
Contractor
expects that R&M
issues will be
addressed by the
Purchaser in
service
Production of a
suitable R&M
Programme Plan
Identification of the R&M
risks and the planned R&M
activities to mitigate those
risks, along with the
technical capability,
resources and controls /
success criteria to ensure it
will happen.
Initial Score = 0.7
Acceptance Status
Time Due
Clear R&M Management
and organisational
Structure.
Systematic programme
of activities for satisfying
the R&M requirements
set against the identified
R&M risks. R&M
activities with clear
objectives and success
criteria. Planned R&M
activities in time to
influence design.
R&M Target Allocations
to subcontractors.
Subcontractors R&M
Plans and Case.
A clear test and
evaluation programme.
Planned R&M Milestones
for R&M achievement
with periodic reviews.
0.1
During
proposals for
ITT and during
early
development /
demonstration
phases.
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
28
DEF STAN 00-42 Part 3 Issue 3
Target
Risk Score
Unclassified
Unclassified
Unclassified
Evidence
DEF STAN 00-42 Part 3 Issue 3
28
Evidence Framework for System ‘Y’
Issue:
Date:
Signature:
Acceptance Criteria
Risk
Ref.
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
Demonstration
(cont)
RM
010
Failure of the
Contractor to
produce sufficient
evidence to
demonstrate R&M
requirements have
been met to the
satisfaction of the
Purchaser.
Test and
evaluation criteria
not formally
agreed between
the Contractor
and the Purchaser
Sufficient design and
test & evaluation
data to provide and
engineering and
statistical confidence
that the preproduction prototype
design has met the
R&M requirements.
Contractor fails to
develop COTS
interfaces that are
compatible with
the system
software
architecture
Integration testing
within the Software
Integration
Laboratory (SIL)
includes Test
Analyse Fix process
reported by formal
DRACAS.
Evidence
Target
Risk Score
Time Due
Execute, monitor and
review R&M Plan
activities, amending where
appropriate.
Contractors R&M Case
Reports showing:-
0.1
During
demonstration
prior to system
acceptance and
manufacture.
DRACAS report from the
SIL testing and
development activities
DRACAS report shows
evidence of:-
0.15
During
demonstration
prior to system
acceptance and
manufacture.
Initial Score = 0.6
RM
011
Reliability of
COTS software
packages is poor
when integrated
into the system.
Initial Score = 0.7
Acceptance Status
Design Changes resulting
from the outputs of
design studies (stress
analysis, FMECAs, etc).
Detailed and Effective
DRACAS.
Component test Results.
Sub-system test Results.
Test rig results.
Reliability Growth Test
results.
Reliability Demonstration
Trials.
User Trial results.
Performance trial results.
Information on design
review action.
Field data from other
users.
Design modifications
leading to Satisfactory
reliability growth
Input to R&M prediction
reports to cover likely
Software failure rates and
PM cycles
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
Unclassified
Unclassified
Project Phase
DEF STAN 00-42 Part 3 Issue 3
2828
28
Evidence Framework for System ‘Y’
Issue:
Date:
Signature:
Acceptance Criteria
Risk
Ref.
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
Manufacture
RM
012
Failure of the
Contractor to
identify and
manage the R&M
risks associated
with the transition
from development
to production.
Contractor
underestimates
the scale of
change between
prototype and
production
system.
Evidence that lessons
learned from preproduction prototype
builds have
influenced the
production process.
Sufficient test and
evaluation data to
provide and
engineering and
statistical confidence
that the production
build standard will
meet the R&M
requirements and
show the reliability
has not been
degraded by the
production process.
Immature
production
facility and
processes
Demonstration of
consistent quality of
manufacture. (PRAT
Test Plan).
Evidence of the
implementation of
effective quality
procedures.
RM
013
Failures of the
Contractor to
monitor the
production run.
Initial Score = 0.4
Evidence
Target
Risk Score
Time Due
Mature production and
quality processes along
side Pre-production
prototype design.
Procedures for the
investigation and
rectification of faults,
failures and defects.
Contractors R&M Case
Reports showing:Production
confirmatory/qualificatio
n trials results.
Production Reliability
Acceptance Testing
(PRAT) first batch results
Evidence of changes in
production and quality
procedures to capture
defects.
0.1
Preceding and
during first off
production
phase.
Production Reliability
Acceptance Test.
Final Quality Inspection of
Deliverables
PRAT Batch Test
Results.
Quality Inspection
Records.
<0.1
At agreed
points during
manufacturing
period.
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
Unclassified
Unclassified
Unclassified
Initial Score = 0.5
Acceptance Status
28
DEF STAN 00-42 Part 3 Issue 3
Project Phase
DEF STAN 00-42 Part 3 Issue 3
28
Evidence Framework for System ‘Y’
Issue:
Date:
Signature:
Acceptance Criteria
Project Phase
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
RM
014
COTS components
fail to perform as
expected
Assembly
information for
COTS equipment
is unsuitable for
application
Demonstration of
consistent quality of
assembly/integration
of COTS
components through
PRAT Test Plan.
Evidence of the
implementation of
effective quality
procedures for
assembly and
integration
Contract does not
fully address
system support
Equipment usage and
failure data along
with the appropriate
analysis to provide
reliability estimates
and failure trends.
Data on repair costs
and resources.
Initial Score = 0.5
Unclassified
In-Service
RM
015
Failure of the
Purchaser to
monitor and
mitigate R&M risks
that result from
change in use,
environment and
support.
Initial Score = 0.5
Acceptance Status
Evidence
Target
Risk Score
Time Due
Production Reliability
Acceptance Test.
Final Quality Inspection of
Deliverables
PRAT Batch Test
Results.
Quality Inspection
Records.
0.1
At agreed
points during
manufacturing
period.
Identification of systematic
failure modes and the
introduction of
modifications through
PDS.
In-service Reliability
Demonstration Trial
Results.
In-Service Reliability
Study results.
In-service data collection
and analysis.
At the start and
through out inservice period.
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
Unclassified
Risk
Ref.
DEF STAN 00-42 Part 3 Issue 3
40
28
Evidence Framework for System ‘Y’
Issue:
Date:
Signature:
Acceptance Criteria
Risk
Ref.
Risk Description
and initial risk
score
Risk Cause
Evidence Required
R&M Practice
Disposal
RM
016
Failure of the
Purchaser to
capture record and
disseminate the
R&M lessons learnt
System ownership
and reporting
responsibilities
are not defined by
the Purchaser.
Full dossier of all
elements of R&M
work from concept
through to in-service
or disposal.
Collation of all
requirements documents,
R&M data and reports into
a corporate data repository.
Production of a final LFE
report providing insight
into effectiveness of the
programme and the final
R&M estimates achieved.
Initial Score = 0.9
Acceptance Status
Unclassified
Evidence
Target
Risk Score
Time Due
Early studies results.
Mature URD and SRD.
Extracts from TLMP.
Outputs from R&M
Panels (R&M Strategy
etc,).
ITEAP and Acceptance
reports
Fully populated R&M
Case with all R&M
evidence reports
(including in-service).
Analysis of lessons
learned.
0.2
Ongoing
through to inservice and
disposal.
Evidence
Reference:
Issue No:
Date
Delivered:
Approval
Signature:
Date:
Unclassified
Unclassified
Project Phase
DEF STAN 00-42 Part 3 Issue 3
28
Evidence Framework for System ‘Y’
DEF STAN 00-42 Part 3 Issue 3
28
©Crown Copyright 2008
Copying Only as Agreed with DStan
Defence Standards are Published by and Obtainable from:
Defence Procurement Agency
An Executive Agency of The Ministry of Defence
UK Defence Standardization
Kentigern House
65 Brown Street
GLASGOW G2 8EX
DStan Helpdesk
Tel 0141 224 2531/2
Fax 0141 224 2503
Internet e-mail enquiries@dstan.mod.uk
File Reference
The DStan file reference relating to work on this standard is . 21/42/3
Contract Requirements
When Defence Standards are incorporated into contracts users are responsible for their correct
application and for complying with contractual and statutory requirements. Compliance with a Defence
Standard does not in itself confer immunity from legal obligations.
Revision of Defence Standards
Defence Standards are revised as necessary by an up issue or amendment. It is important that users
of Defence Standards should ascertain that they are in possession of the latest issue or amendment.
Information on all Defence Standards can be found on the DStan Website www.dstan.mod.uk,
updated weekly and supplemented regularly by Standards in Defence News (SID News). Any person
who, when making use of a Defence Standard encounters an inaccuracy or ambiguity is requested to
notify UK Defence Standardization (DStan) without delay in order that the matter may be investigated
and appropriate action taken.
Unclassified