Technological infrastructural needs
to support third party certification
Certification of Safety-Critical Software-Intensive Systems
First Public Workshop
November 11, 2011
Sushil Birla
Division of Engineering
Office of Nuclear Regulatory Research
(301-251-7660, Sushil.Birla@nrc.gov)
Background & source of vision
Context: U.S. Govt. Inter-agency coordination activities
– NITRD (Networking & IT R&D)
• HCSS (High Confidence Software & Systems)
– Cyber-physical systems
» Focus area: Safety critical systems
Sectors
Health
Energy
Defense
Transportation
National Security
Commonalities
2
Current state – some commonalities
• Safety-critical CPSs are typically too complex to be completely verified and
validated. Remaining uncertainties are significant, but not well understood.
• Safety analysis and evaluation require high competence and judgment, but
these capabilities are very scarce.
• Cyber adversaries’ ability to develop and launch new attack tools and
techniques outpaces the ability to develop and deploy countermeasures.
• The competencecomplexity gap is widening rapidly.
• Similar problems exist in most safety-critical, mission-critical application
domains, but there is little synergy to find a common core set of underlying
solution capabilities.
• The requisite knowledge is not well-systematized
• Commercially available tools, driven by non-critical consumer applications,
are being used in critical applications, but their commensurate verification Is
not feasible economically.
3
Current state: Some complexity issues
• A single defect can make logic wrong, potentially leading to
serious consequences, but the capability to engineer defectfree systems does not exist.
• Networking (wired or wireless) introduces new vulnerabilities
that are not well understood
– Hidden dependencies and couplings
• Latent defects could combine in many scenarios
• Latent defects could cause a high consequence failure
• The more complex a system the more exposure to defects
• Verification of a high-integrity system or component, e.g.
operating system, takes more effort and time than its initial
development.
4
Vision state: Some commonalities
• Systems can be routinely developed with built-in assurance of
safety and security
– “Do it right the first time” becomes the cheapest and fastest way to
realize a system
• Accredited third party services are commercially available for
verification & validation (V&V)
• Accredited third party services are commercially available for
review, attestation, and certification
• Requisite tools are certified
• Requisite competence (knowledge, skills) is certified
• Requisite competence becomes readily available
• Requisite body of knowledge is mature and readily accessible
• Educational and training institutions have mature curricula to
produce and certify the requisite competence
5
ISO 17000 definitions - 1
5.5 certification
Third-party attestation related to products, processes, systems or persons
5.2 attestation
Issue of a statement, based on a decision following review, that fulfillment of
specified requirements has been demonstrated
5.1 review
Verification of the suitability, adequacy and effectiveness of selection and
determination activities, and the results of these activities, with regard to
fulfillment of specified requirements by an object of conformity assessment
6
ISO 17000 definitions - 2
3.1 specified requirement
Need or expectation that is stated. NOTE: Specified requirements may be
stated in normative documents such as regulations....
2.1 conformity assessment
Demonstration that specified requirements relating to a product, process,
system, person or body are fulfilled
2.4 third party
A person or body that is independent of the person or organization that
provides the object, and of user interests in that object
7
ISO 17000 definitions - 3
2.5 conformity assessment body
Body that performs conformity assessment services
5.6 accreditation
Third-party attestation related to a conformity assessment body conveying
formal demonstration of its competence to carry out specific conformity
assessment tasks
2.6 accreditation body
Authoritative body that performs accreditation
NOTE … authority … generally derived from government
8
Some expectations & gaps
Enable certification of safety-critical software
3rd party conformity
assessment bodies
Formally demonstrate competence
Competence criteria
Accreditation bodies
9
Some more gaps
Regulatory
requirements
are abstract
Expert
Judgment
needed
Scarce!
Standards
Interpret
Regulatory guides
Concrete
derived
requirements
missing/incomplete
Review
SW in
safety
system
10
Research needs identified
Questions posed to expert group
• What are sources of uncertainties?
• What evidence do we need to reduce these uncertainties?
• What are the areas that need more research?
11
Uncertainties even after best practices
Residual
Uncertainties?
Appendix A in
RIL-1001
Focus of group
Assume
conformity
NRC’s regulatory guidance
framework
“Good” design practice
Uncertainties and resulting size of potential fault space
12
Some sources of uncertainties
• Validation of Requirements
• Architecture: Complexity
• Verification: Adequacy of coverage
• Impact of change: Hidden/obscure dependencies
• Transformation tools
• Integrating/Combining evidence
13
Current review practice
Perform thread audits of several requirements
• Check for conformance clause-by-clause
Is clause-by-clause review enough?
14
Combined effects of deviations
Charles Perrow in “Normal AccidentsLiving with High Risk technologies” 1984:
– A major failure of a complex system is
typically caused by a combination of
relatively small incidents: Three Mile Island
15
Combined effects in SW
• A single defect can make logic wrong
• Hidden dependencies and couplings
• Latent defects
– Could combine in many scenarios
– Could cause a high consequence failure– The more complex a system the more exposure to defects
16
Combined effects of seemingly insignificant
deviations
High consequence failure of
a complex system

Operators’
Action
Inadequate
Procedures
Incorrect
indicator
Faulty
Equipment
Inadequate
Design
17
Example of evidence gaps
Demonstrate
Uncertainties cannot combine
to produce
more complex uncertainties
Demonstrate
Independence and decoupling
Demonstrate
Compliance with architecture principles & constraints
Inadequate
criteria
18
Architecture: Complexity issues
New I&C architectures overly complex
1. High degree of connectivity between two systems
which are suppose to be independent
2. Safety to non-safety interconnectivity
http://www.hse.gov.uk/newreactors/ri-ukepr-0002.pdf
19
Transformation tool issues
• New (unknown) ways of introducing defects
• Preservation of semantics
20
Integrating effect of uncertainties
in software assurance
V&V results
system
Reqmts
Arch
?
?
?
?
Change
Impact
Analysis
?
Auto
test
gen
Auto
code
gen
software
Arch
Reqmts
Tools
system
D
I
Unit
Test
?
?
?
Integr
Test
?
FAT
?
Safety demonstration
in the presence of
uncertainties
Each anomaly or uncertainty by itself seems to be small
21
V&V: Adequacy of coverage
Some major sources of uncertainties
Environment
Requirements
•Assumptions
•Input validity
•Correct?
•Complete?
•Consistent?
Incomplete
coverage
Coverage evidence
(Diverse complementary)
Interference
Proof of noninterference
Analysis
Model checking
Testing
- Coverage based
…

Safety
Demonstration
(e.g. assurance case)
Evidence about other uncertainties
22
Safety demonstration: Adaptation
of Toulmin’s model
Backing, e.g., theoretical or causal model
basis for
Inference rule
used in
Evidence/ Grounds
Assertion/
Argument
Factors influencing validity of argument
Belief/Claim
affects
Qualifiers
(Strength;
Condition)
Challenges; rebuttals; inconsistencies
23
Recap: NRC areas of interest
Certification infrastructure needed
• Accreditation bodies
• Competence criteria
• 3rd party conformity assessment bodies
Some gaps in assurance technology infrastructure
• Validation of Requirements
• Architecture: Complexity
• Verification: Adequacy of coverage
• Impact of change: Hidden/obscure dependencies
• Transformation tools
• Integrating/Combining evidence
24