From Validating Models
to Validating Systems
Peter Denno
2013-02-25
University of Maryland ISR Colloquium
1
Outline
•
•
•
•
Introduction / Scoping
Requirements for MBSE
Exchange Form Validation
NIST Work
2
Goals
• Describe a “design philosophy” for systems
that assist in systems engineering
– Framework for linking multiple viewpoints
– Framework for research
• Link the design philosophy to NIST work
in exchange form validation, requirements
engineering, supply chain logistics simulation
3
What is special about V&V ? (1)
• IBM Watson
- New techniques – exceed human capability in
knowledge-intensive tasks
- “Machine understanding is not human understanding.”
- “Knowledge is not the destination.”
4
What is special about V&V ? (2)
• Validation & Verification
- Knowledge is the destination. – knowledge, or at least
credible rationale.
• Requirement:
- Minimally: be able to explain how the design space was
characterized and demonstrate that requirements are
being met.
- Ideally: Provide deductive arguments where appropriate
- Show how certain alternatives are indeed incompatible
- Reference principles of operation, functions
5
Outline
•
•
•
•
Introduction / Scoping
Requirements for MBSE
Exchange Form Validation
NIST Work
6
Basis for SE decision making
• SE decision making macro level:
– Trade studies, simulations, risk assessment, etc.
• SE decision making micro level:
– A web (conceptual schema) of information
• Uncertain
• Conflicting
• Isolated
• Uncertainty is quantified
• Conflicts resolved
• Inter-relation revealed
7
Strategy for the micro-level
information
• Characterize elements of rationale for
SE decision making.
• Each research project touches on only a
few of these elements
– No single overarching system design intended
8
9 Elements of Rationale (1)
• Measurement Conditions
– Confidence in the process or environment under which it was measured
– “Capacitance was measured using the AC impedance technique.”
• Logical Consistency
– Confidence due to consistency with theory. Type consistent.
– “P = .05, as we’d expect from the law on conservation of energy.”
• Associativity Across Views
– Individuals: knowledge that two references made from different viewpoints
refer to the same thing
– “The region P on the CAD model corresponds to these elements
in the FEA mesh.” (Individuals)
– Concepts: knowledge that two conceptualizations can be used for the same
purpose.
– “What the supplier is calling ‘rated maximum pressure’ is what we call ‘rated
pressure.” (Concepts)
9
9 Elements of Rationale (2)
• Change process
– Knowledge of precursors and the history of properties that
distinguish them.
– “The value of P that we calculated for this design is close to what we
found in earlier models.”
• Authority
– The power that information has due to an approval that is granted or
an estimate of its maturity
– “Supplier-provided data also suggest P=.05 is obtainable.”
• Origin in Requirements*
– Belief that a requirement is sensitive to it
– “Our ability to achieve requirement x diminishes as P exceeds 0.07.”
10
9 Elements of Rationale (3)
• Origin in organization infrastructure
– Belief because you obtained it in ways consistent with the organization’s
best practices.
– “P was obtained from the aero model in the preliminary design library.”
• Consistency with other belief
– Belief due to consistency with prevailing contingent facts
– “P=0.5 is reasonable in products using component y.”
• V&V Process
– Belief that the system in place to manage the other 8 elements is sound
and comprehensive.
– “The value of P is confirmed through simulation that is routinely
performed in validation of this product line.”
11
9 Elements : Observations
• Coupling and overlap
– Authority / Origin in Organizational Infrastructure
– Associativity across views / measurement conditions
– etc.
• Though these are found in models, they can be
expressed from a more comprehensive viewpoint
where
–
–
–
–
Contradictions can be exposed
Cohesion across views can be noted
Trace to requirements is more evident
(These are all parts of V&V)
12
MBSE Concepts / Logical View
13
Sentence detail / rationale
14
Example Usage Patterns
• V&V
– Origin in requirements
– Automated generation of test cases
• Requirements Engineering
– Origin in other belief, emphasis on tracking
contingent facts and engineering change
– Refinement
15
Outline
•
•
•
•
Introduction / Scoping
Requirements for MBSE
Exchange Form Validation
NIST Work
16
Exchange Form Validation :
Two Methods
1. Axiomatic:




How: Map the exchanged content to sentences
Identify errors: ex falso quodlibet with a reasoner
Advantage: Ontology explains intent
Disadvantage: Proofs hard to interpret
2. Metamodel:




How: Map the exchanged content to objects
Identify errors: Direct structural, with OCL, etc.
Advantage: Constraints relate to exchange form
Disadvantage: Constraints look like code
17
Example use of metamodel
View / Viewpoint: Can be both
consistent with a form (a view),
and the form by which other
conceptualization are stated (a
viewpoint.)
18
Example from the UML Metamodel
19
Example Specification Constraints
20
In MBSE, metamodels play a key role
• Metamodel =
(1) a specification of the form a model can take.
(well-formedness conditions)
(2) a formalization of the viewpoint that
models will express
21
Metamodels also play a key role in
model exchange
• Metamodel =
(1) a specification of the form a model can take.
(well-formedness conditions)
Definition of structure  serialization
(2) a formalization of the viewpoint that
models will express
Illuminate what program structures the elements
of exchange content map to/from.
22
Communication with Exchange Standards
23
Outline
•
•
•
•
Introduction / Scoping
Requirements for MBSE
Exchange Form Validation
NIST Work
– Model Interchange Working Group
– Supply Chain Logistics Simulation
– Collaborative Requirements Engineering
24
Outline
•
•
•
•
Introduction / Scoping
Requirements for MBSE
Exchange Form Validation
NIST Work
– Model Interchange Working Group
– Supply Chain Logistics Simulation
– Collaborative Requirements Engineering
25
OMG Model Interchange Working Group
• Goal: Improve the ability of OMG MOF-based tools
(UML, SysML) to exchange information
– XMI Serialization – common to MOF-based tools.
• Process
–
–
–
–
Group: Produce Test Case diagram and reference file
Tool developers: create diagram in their tool, serialize as XMI
Use NIST tool to identify errors (in files and metamodels)
Correct tools and specifications
26
NIST UML / SysML Validator
Enter below a file to upload:
27
NIST UML / SysML Validator
28
NIST UML / SysML Validator
29
NIST UML / SysML Validator
30
MIWG Results
• Stakeholders witness significant improvement
in interoperability
Elaasar & Labiche, 2012
31
Outline
•
•
•
•
Introduction / Scoping
Requirements for MBSE
Exchange Form Validation
NIST Work
– Model Interchange Working Group
– Supply Chain Logistics Simulation
– Collaborative Requirements Engineering
32
Supply Chain Logistics Simulation
• Goal: Demonstrate integrated use of models
toward enterprise goals
• Design: Map models, guided by metamodels
into sentences that guide compilation of a
discrete event simulation.
• Models
– UML of ordering / logistics objects
– QVT-r mapping of messages to orders
– BPMN “stereotyped” + OCL of business decisions
– Discrete Event Simulation
33
Round Trip Engineering
of Supply Chain Logistics
34
Logistics Processes (1)
35
Logistics Processes (2)
36
Business Rule
37
Simulation Results
38
Outline
•
•
•
•
Introduction / Scoping
Requirements for MBSE
Exchange Form Validation
NIST Work
– Model Interchange Working Group
– Supply Chain Logistics Simulation
– Collaborative Requirements Engineering
39
Collaborative Requirements Engineering
• Goal: Demonstrate Engineering from
Product Data Sheets
• Design: Map product data sheets in to
sentences about requirements. Use these to
guide engineering simulation and reasoning
about alternative designs
40
Conclusions
• Continuing roles for deductive reasoning in
the automation of SE processes
– The nature of V&V, requirements engineering, the
way we think when we engineer, require it.
• Preparing and interpreting macro-level SE
decision processes is aided by the integration
of multi-viewpoint, micro-level information.
• Metamodels facilitate this integration.
41
References
• Welty, C; Inside the mind of Watson, 2nd ESWC Summer School, Kalamaki,
2012, http://videolectures.net/eswc2012_welty_watson
• Denno, P; Thurman, T, Mettenburg, J; Hardy, D; On enabling a modelbased systems engineering discipline – 18th INCOSE International
Symposium (2008)
• Denno, P; Harrison, T; Using Legacy Modeling Artifacts in Supply Chain
Logistics Simulation (in draft, 2013)
• ISO 15288 (2008) – Systems and software engineering – System life cycle
processes. (2008)
42