Design Integrity Concepts
2005 MAPLD
1
Design Integrity Concepts
Unit Agenda
• Consistent terminology, consistent results –
Introduction and definitions
• What does it have to do? –
Specifying the design
• Letting constraints work for you –
Proportional design
• More than the sum of its parts –
Partitioning a design
• You mean we’re still working on it? –
Sustaining a design
• What’s the exit strategy? –
Verifying a design
• What do you mean, it doesn’t do what we thought? –
Validating a design
2005 MAPLD
2
Design Integrity Concepts
Who is this guy?
• John Stone
– Chief Engineer (Department of Space
Systems – SwRI®)
– 18 years experience (Exclusively space-flight)
•
•
•
•
•
•
C&DH design (hardware and software)
Instrument electronics design
Box-level integration and test
Hardware project management
Product assurance (parts, radiation, reliability)
Process improvement
– Broad interest in improving what we do
2005 MAPLD
3
Design Integrity Concepts
What do we want to accomplish?
• Discuss techniques that may help us to do
our jobs better
– Based on general principles
– Broad applicability (logic, board, box, system)
• Foster necessary discussion / brainstorming
– To extent possible (as time allows)
– No one person has all of the answers
– Suggestions appreciated
2005 MAPLD
4
Design Integrity Concepts
Consistent Terminology,
Consistent Results
Introduction and Definitions
2005 MAPLD
5
Design Integrity Concepts
Agenda –
Introductions and Definitions
• Design integrity – A working definition
• Why is this important?
– A tale of two designs
– Has anything changed?
– Why should I care?
• The miracle cloud
• The alternative
– Overview
– Implications
– Additional Definitions
• Summary
2005 MAPLD
6
Design Integrity Concepts
Definition and Goal
• Design – The invention and disposition of the
forms, parts, or details of something according to a
plan. (AH dictionary online)
• Integrity – The state of being unimpaired;
Soundness; The quality or condition of being
whole or undivided; completeness (AH)
• This seminar is intended to talk about techniques
and issues that ensure the soundness and
completeness of both the end product and the
means used to produce it.
2005 MAPLD
7
Design Integrity Concepts
Design 1 – Radarsat 1 ACP
2005 MAPLD
8
Design Integrity Concepts
Radarsat 1 ACP Overview
• Program dates: late 1990 – late 1992
• Specifications
– Processor: 8 MHz 80386/80387
– Memory:128 k x 8 SRAM, 192kx8 EEPROM, 16k x 8
PROM
– Interfaces: A/D (16), D/A (4-12), Synchronous serial (3
input, 3 output), RS-232 GSE
– Function: Attitude control processor for the RADARSAT1
satellite
– Logic Implementation MSI logic + PALs (16L8, 16R6,
16R8)
• Additional functions (cross-strap, control) external
2005 MAPLD
9
Design Integrity Concepts
PAL Reminder
2005 MAPLD
10
Design Integrity Concepts
Design 2 - Command Telemetry
Board (CTB)
2005 MAPLD
11
Design Integrity Concepts
CTB Overview
• Program Dates: early 2001 – late 2003
• Specifications:
– Processor: RTX2010 (16 MHz)
– Memory: 4M x 8 (random), various FIFOs (16k x8) as necessary
– Interfaces
•
•
•
•
•
MIL-STD-1553B
Synchronous Serial (command / telemetry)
Analog input
High-level discrete (output)
Low-level discrete (input / output)
– Functionality: S/C command/telemetry (level 0 and active)
– Logic Implementation: 4 54SX32S FPGAs
• Additional resources (in same box): RAD 750, Mass
Memory, Instrument Interface Card
2005 MAPLD
12
Design Integrity Concepts
What’s Changed?
• Capability / Complexity
• Logic Density
• Specificity
– RADARSAT (1 small specification with interface focus)
– CTB (1 large specification with interface, s/w, operations focus)
• Software Centricity
• Initial Errors
– RADARSAT: 3 jumpers; 1 PAL design change
– CTB: 14 FPGA revisions
• 2 spec change
• 5-6 mistakes
• 6-7 data dependency
2005 MAPLD
13
Design Integrity Concepts
What’s Not Changed?
•
•
•
•
Overall program schedules
Proportional budget
Expectation of correctness
Pain from mistakes
2005 MAPLD
14
Design Integrity Concepts
What Explains the (error)
Difference?
• Engineers aren’t as capable? – Insulting!
• Everything is just more complex? – Copout!
• Methodology?
– Methodology hasn’t changed
• Always inadequate, we just got lucky
• Adequate for old designs, no longer adequate
– Methodology has changed
• Used to be adequate, but we lost the recipe
• Design philosophy of systems has changed?
– Predicated on maximum flexibility
– Expectation of extreme complexity
– Over-specification – almost impossible to meet
2005 MAPLD
15
Design Integrity Concepts
What Do These Examples Illustrate?
• The incidence of initial correctness for designs seems to be
decreasing
– Design changes seem to be more common
– Problems late in the verification/validation cycle seem to be more
frequent
• Perhaps a combination of the factors presented explains
this, but …
– Desired complexity is not going to decrease
– Budgets are not going to get bigger
– The expectation of excellence isn’t going to go away
• The only solution is to develop and improve a consistent
methodology for implementing robustly designed products
– Based on basic principles
– Applicable to a variety of conditions
2005 MAPLD
16
Design Integrity Concepts
Why Should I Care?
• Why do we work?
– Self-actualization (fun, monetary reward, interest)
• Why do people want us to work for them?
– They need what we produce
• What do people want engineers (especially in
Aerospace) to produce?
– A quality product that satisfies the customer’s needs
• How do they want us to produce such a product?
– Consistently and efficiently
2005 MAPLD
17
Design Integrity Concepts
The Layman’s View –
The Miracle Cloud
2005 MAPLD
18
Design Integrity Concepts
The Miracle Cloud Method
• Note that too many engineering schools teach this
approach without meaning to
• Advantages to the miracle cloud method
– Total creative freedom
• Disadvantages to the miracle cloud method
– Product quality is variable
• Team makeup dependent
• Team mood/morale dependent (Monday morning car)
• Luck dependent
– Product is not produced in a repeatable manner
– Product is not produced in an efficient manner
• Result
– Quality low
– Customer Satisfaction Low
2005 MAPLD
19
Design Integrity Concepts
How Do We Replace the
Miracle Cloud?
• Provide structure to the development effort
• Evaluate the effort and the product
produced
• Improve the effort and the product
• Repeat
2005 MAPLD
20
Design Integrity Concepts
Definitions of Importance
• From Q9000-2000
• Process – A set of interrelated and
interacting activities which transforms
inputs to outputs [in our case ideas to
devices]
• Product – The result of a process
2005 MAPLD
21
Design Integrity Concepts
Implications From These Definitions
• If we want a consistent product, we must have a
consistent process
• If we want to improve a product, we must improve
the process
• If our company has no (or inadequate) process and
we must produce a quality product, then we must
establish a process [personal responsibility]
• Developing, imposing, and improving a process is
not an end (in and of itself) it is only a means to
an end
2005 MAPLD
22
Design Integrity Concepts
A Model for Discussing
the Design Process
2005 MAPLD
23
Design Integrity Concepts
Notes on the Model
• Feedback / iteration are not shown for clarity
• Model may be recursive
– Board development process includes FPGA requirement definition,
FPGA development, instantiation, etc.
– Board verification process includes the FPGA validation product
• Successes and failures are cumulative
– Good requirements + successful development => successful
instantiation
– Bad requirements + failed development => failed instantiation
• Complexity multiplies
– Complex requirements increase design complexity which, in turn,
increases verification complexity
• Processes are absolute gates to the next stage of development
2005 MAPLD
24
Design Integrity Concepts
Implications From the Model
• All processes must be addressed at all levels of design
[there are no shortcuts!]
– Does not imply same formality at all levels
– Does imply same rigor at all levels
• Up front work on requirements is essential!
– Must provide adequate time and money
– Must gain team buy-in to the process*
– Benefits compound throughout the rest of the activities
• Simplicity is an essential virtue!
– Complexity inevitably multiplies
• A product is not qualified until both verification and
validation are complete
2005 MAPLD
25
Design Integrity Concepts
Additional Useful Definitions
(courtesy of Q9000-2000)
• Specification – A document* stating requirements, needs, or
expectations that are obligatory
• Quality – The degree to which a set of inherent characteristics fulfill
requirements
• Customer satisfaction – Customer’s perception of the degree to which the
customer’s requirements have been fulfilled
• Verification – Confirmation, through the provision of objective evidence,
that specified requirements have been fulfilled
• Validation – Confirmation, through the provision of objective evidence,
that the requirements for a specific intended use or application have been
fulfilled
• Objective evidence – Data supporting the existence or verity of something
• Continual Improvement – recurring activity to increase the ability to
fulfill requirements
• Note the importance of Requirements
2005 MAPLD
26
Design Integrity Concepts
Summary
• I have no assurance that my product will have consistent
quality without:
– Well-defined requirements
– A well planned approach to implementing the requirements
– A clearly defined plan for verification and validation of the
requirements
– The ability to improve the process that produces the product
• Without quality product, customer satisfaction is
impossible
• Without customer satisfaction, I won’t work!
• Therefore, I must care about ensuring design integrity
2005 MAPLD
27
Design Integrity Concepts
What Does It Have to Do?
Specifying the design
2005 MAPLD
28
Design Integrity Concepts
Agenda – Specifying the Design
•
•
•
•
•
•
Basic concepts and implications
What should we include in a specification?
What should we avoid in a specification?
The role of the design engineer
Summary
FPGA specifications
2005 MAPLD
29
Design Integrity Concepts
Basic Specification Concepts
• #1 A specification has a limited set of purposes:
– It is intended:
• To capture product requirements that are essential to meeting a
need – functionally and interfacially
• To ensure traceability of the requirements from higher levels
• To provide a fixed framework for verifying product
conformance
• To provide a framework for derived requirements
– It is not intended:
•
•
•
•
To impose a detailed implementation for a design
To provide unnecessary theory of operation statements
To become a user’s manual
As a box to check on the schedule checklist
2005 MAPLD
30
Design Integrity Concepts
Basic Specification Concepts (cont.)
• #2 Specification language must meet certain criteria
– Structure
• Similar requirements must be grouped together
• All requirements must stand on their own
– Verbs must have a consistent specific meaning
• Shall, must -> impose imperatives
• Should, might -> define preferences
• Will -> defines objective information
– Phraseology
• Phrases must be unambiguous
• Phrases must define one (and only one) requirement
• Phrases must be short and understandable
– All must agree on the meaning of what is written
2005 MAPLD
31
Design Integrity Concepts
Basic Specification Concepts (cont.)
• #3 Specifications are inherently intrusive
– Requirements exclude options in a design
• The number and specificity of requirements determine the level of
exclusion
– Specifications impose non-negotiable verification requirements
• Every requirement must be verified
• The narrower the requirement, the more specific the verification
• The more requirements, the more work needs to be performed
– A well designed specification assists the design and verification
process
– A poorly designed specification
• Unnecessarily shuts down trade space
• Increases verification time and effort
• Makes for an unhappy engineer
2005 MAPLD
32
Design Integrity Concepts
What is Included in a Specification?
• Interfaces
– Number
– Type
– Timing / Handshaking
• Interrelationships
– Temporary storage
– Data Flow
– Software Support
(careful)
• System impact
– Quality and Reliability
– Parts, materials, and
processes
2005 MAPLD
33
Design Integrity Concepts
Examples of Things to Include
• Interfaces
– The unit shall provide 16 low-level discrete interfaces
– The unit shall configure 8 low-level discrete interfaces as pulsed interface
– Pulse interfaces shall generate pulses between 2 and 20 ms long
• Interrelationships
– The unit shall provide on-board temporary storage sufficient for 2 packets
from the XYZ interface
– The unit shall forward data from the XYZ to QRS interfaces on a packet
by packet basis
– The unit shall provide an interrupt to the S/W upon receipt of a complete
package from the XYZ interface
• System Integration
– The unit shall have a .75 probability of success as calculated per the parts
count method of MIL-HDBK-217
– The Unit shall use only EEE parts that meet the requirements of EEEINST-002
2005 MAPLD
34
Design Integrity Concepts
What Should Not Be Included in the
Specification?
• Implementation details (unless essential to safety or
reliability)
– The unit shall implement standard pyrotechnic fire circuit #2 as
found in …(marginally acceptable)
– The unit shall use 4 7206 FIFOs made by IDT corporation
– 370 ns after the last bit of a packet is received the unit shall raise
bit 3 of interrupt control register 21
• Requirements based on inherent capability of devices used
– The unit shall provide a 14-bit A/D converter for interface X (when
based upon the part being used)
– Note: Decisions on precision or accuracy etc. should be made for
interface characteristics (physical measurement range and
accuracy)
2005 MAPLD
35
Design Integrity Concepts
What Shouldn’t Be Included in the
Specification (cont)?
• Ambiguous, unclear, or interpretive requirements
– The unit shall be constructed using techniques found in generally
accepted space hardware guidelines
– The unit shall interface to other components per figure 2 (picture
based requirements)
– The unit shall provide the best performance possible
• Overly complex requirements
– The unit reset shall be immediately triggered upon expiration of
the watchdog timer unless the watchdog timer has timed-out five
times (8 times if in mode 2) in which case the circuitry shall look
at discrete bit 2 to determine whether to delay 3 s (discrete bit 2 =
1) before timing out
• Extraneous information
– The unit shall interface with the visible CCD used in
spectrographic mode
– The unit shall replace the functionality provided by p/n 276401-01
2005 MAPLD
36
Design Integrity Concepts
Why Shouldn’t These Be Included?
• This type of requirement increases cost [time, money,
emotional] to develop a product
– Increase design effort (less trade space, inefficient design, overly
complex design)
– Increase verification effort [planning, implementing, configuring]
due to design complexity
– Increase probability of re-design or re-build [validation
uncertainty]
• This type of requirement reduces the overall quality of the
product developed
– Implemented design not per intention [interpreted requirements]
– Implemented design is not 100% verifiable to specification
[unclear requirements]
– Implemented design requires significant number of waivers and
deviations
2005 MAPLD
37
Design Integrity Concepts
The Role of the Design Engineer
• The design engineering team must “buy in” to the
specification development process
– They have to deal with the consequences if it is done
poorly
– Lack of “buy in” produces a non-controlled process
• “Buy in” requires early review and participation
by the design team
• Therefore, design engineers have a significant role
to play in the specification development process
• What is this role?
2005 MAPLD
38
Design Integrity Concepts
Role of the Design Engineer (cont.)
• Know how to write a specification
– Allows constructive review of the imposed specification
– Improves qualities of derived requirements
• Try to understand the “why” behind the various
requirements
– Improves efficiency of design trade studies
– Allows intelligent suggestion of alternatives
• Suggest alternatives when necessary
– Expose more efficient ways of producing equivalent functions
– Support trade offs between software and hardware functionality
2005 MAPLD
39
Design Integrity Concepts
Role of the Design Engineer (cont.)
• Think forward
– Take the lead in defining derived requirements
– Ask:
• What implications does this have for the design?
• What implications does this have for testability?
• Will this let me sleep at night?
• Push back
– Seek clarification of ambiguous requirements
– Inform others of impractical or costly driving requirements
– Ask for relief from impossible requirements
• Remain engaged in the process
– Be thorough in review
– Don’t be passive with suggestions
2005 MAPLD
40
Design Integrity Concepts
Summary
• Specifications are critically important to the
design engineer and must not be ignored
• Design teams must be intimately involved
in the specification development process
• Detailed design and implementation will
succeed or fail largely based on successful
application of the specification process
2005 MAPLD
41
Design Integrity Concepts
FPGA Specifications - Rationale
• FPGAs are a major part of modern day spaceflight designs
– Primary control functionality
– Equivalent of multiple boards of traditional circuitry
– Major schedule driver
• FPGAs are impossible to verify completely from external
signals
– Too many buried modes and functions
– Too much dependence on simulation
• Correcting FPGAs is conceptually simple
– Temptation to sloppiness
• “First time right” is an essential virtue
– The FPGA specification is a means to achieve “first time right”
2005 MAPLD
42
Design Integrity Concepts
FPGA Specification Prototype
• Interface Section
– Specific pinout requirements
– I/O type definition (Names, Direction, Logic levels)
– Imposed Software requirements (addressing, etc.)
• Do not include non-imposed addressing / register mapping / software
interfaces
• Functionality Section
–
–
–
–
Interface interaction requirements
Data flow requirements
Exclusion requirements
Do not include
• Implementation details
• Theory of operations
• Usage instructions
2005 MAPLD
43
Design Integrity Concepts
FPGA Specification Prototype (cont.)
• Fine timing section
– All timing imposed by board peripheral circuits
– Include
• Min and max delay between I/Os
• Set-up and hold for controlled peripherals
• Fine timing requirements should be an input
to the FPGA development effort, not
outputs of the concluded design
2005 MAPLD
44
Design Integrity Concepts
Why Include Fine Timing?
• Reduces influence of changes external to FPGA
(encapsulation)
– Board implementation can change significantly
– FPGA implementation can change significantly
– Changes ok as long as interfaces remain stable - but fine timing
must be a controlled item
• Simplifies verification
– Verification between implemented FPGA performance and fixed
requirements defined at the board level can be easily accomplished
– Reduces reliance on complex back-annotated simulations at the
board level for FPGA specific verification
– Increases reliability of static timing analysis
• Note – this only works if a structured design approach is
used such that valid requirements can be imposed on the
FPGA early in the process.
2005 MAPLD
45
Design Integrity Concepts
Stand-Alone or Integral?
• When should an FPGA specification stand alone?
– When the FPGA design engineer is not the board design
engineer
– When there are multiple interrelated FPGAs on a board
– When the FPGA design will be used in multiple board
applications
– When the FPGA will become an ASIC
– When the development schedule between the board and
FPGA are disjoint
– When the complexity of the FPGA is greater than that
of the board support circuitry
2005 MAPLD
46
Design Integrity Concepts
Stand Alone or Integral (cont.)?
• When should an FPGA be specified as part of the
board specification?
– When the FPGA is so intertwined with the peripheral
circuitry that writing a stand-alone specification is not
practical
– When the FPGA functionality is easily expressed by
simple gate level schematics
– When the FPGA and board are designed by the same
person
– When heritage design with adequate specifications exist
2005 MAPLD
47
Design Integrity Concepts
Letting Constraints Work For You
Proportional Design
2005 MAPLD
48
Design Integrity Concepts
Agenda – Proportional Design
•
•
•
•
•
Conceptual background
Types of constraints
Examples
The proportional design mindset
Summary
2005 MAPLD
49
Design Integrity Concepts
Conceptual Background
• Three parts to solving a problem:
– Need, solution set, constraints
• All parts have a role to play in the solution
• Ignoring any of them will lead to problems
2005 MAPLD
50
Design Integrity Concepts
Conceptual Background (cont.)
• Example
– Need:
– Solution set:
means of conveyance to work
Skateboard, bicycle, bus, jogging shoes, mid-size
sedan, luxury car, helicopter
– Constraints: Distance (6 miles), $, not on bus route, $, not in
very good shape, $
– Solution:
1992 Honda Accord (120 kmiles, 4 k$)
– The constraints guide selection of the solution from the solution set
• The particular solution is not necessarily – The cheapest (roller skates)
– The most desired (Lexus LS400)
– What is perceived as best for society (bus)
• But … the best overall fit to the needs
2005 MAPLD
51
Design Integrity Concepts
Conceptual Background (cont.)
• Definitions
– Constraint: the state of being checked, restricted, or compelled to
avoid or perform some action (AH)
– Proportional: corresponding in some degree or intensity (AH)
• Proportional design is design that results in a product
“sized” appropriately to the needs and restrictions of the
specification
• The concept of proportional design:
– Accepts the reality of constraints
– Attempts to optimize the solution given the constraints
– Accepts that the constraints provide benefits (more later)
•
•
•
•
More efficient designs
More thorough designs
More correct designs
Caveat – All other things being equal
2005 MAPLD
52
Design Integrity Concepts
Types of Constraints
• External (mass, power, cost, quality)
• Internal
– Derived (packaging, architecture, component
availability, maximum clock speed)
– Self-imposed
• Design rules/guidelines (free space, clock use, logic structure,
HDL language)
• Documentation style (pre-design, post design)
• Component acceptability (maturity of part, limited use of
various features
2005 MAPLD
53
Design Integrity Concepts
Examples (1)
• Problem: provide decoding logic for memory map
– 0-3FFF = SRAM; 4000-4FFF = Peripheral; E000-FFFF = PROM
• Constraint: use minimum amount of logic
• But what about …
– Unused addresses, future expansion, etc.
• Doesn’t matter – given the constraints
2005 MAPLD
54
Design Integrity Concepts
Examples (2)
• Problem: provide all combinational / sequential logic for
the RADARSAT ACP
• Constraint: Only low density high speed logic available
(16X8 PALs, MSI/SSI logic)
• What was forced by the constraint?
– Careful mapping of peripherals into available address space
– Careful partitioning between:
• Programmable logic and MSI/SSI
• MSI/SSI functionality
– Efficient data bus partitioning (tri-state enable issues)
– Special attention to component delays at the gate level
2005 MAPLD
55
Design Integrity Concepts
The Proportional Design Mindset
• Constraints inevitably foster attention to detail (creativity
“inside the box”)
– With respect to methodology
– With respect to level of planning
– With respect to implementation
• Attention to detail is of inherent value because it produces
carefully structured, well-thought out designs
– Improved up-front correctness
– Decreased design post-processing time (simulation, verification,
validation, lab time)
– Efficient designs that meet the stated requirements
– Increased reliability
• Therefore, constraints are welcomed, whether externally
imposed or self-imposed
2005 MAPLD
56
Design Integrity Concepts
The Proportional Design
Mindset (cont.)
• Examples of self-imposed constraint
– Ignoring achievable flexibility (when not
necessary)
– Removing non-specified capability
– Avoiding gratuitous cleverness (especially with
abstract design techniques)
– Rejecting brute force solutions without analysis
2005 MAPLD
57
Design Integrity Concepts
The Proportional Design
Mindset (cont.)
• Characteristics of the right mind set
–
–
–
–
–
Planning before starting
Reviewing before finalizing
Simplifying ruthlessly
Making the design do only what it must
Viewing resources as precious commodities to be used
only to the extent needed
– Understanding the implication of the design’s level of
abstraction
– Being satisfied with the result
2005 MAPLD
58
Design Integrity Concepts
The Proportional Design
Mindset (cont.)
• Why aren’t self-imposed constraints more common?
– They aren’t absolutely essential because we have:
• Lots of logic space [FPGAs, ASICs]
• Lots of memory space [DOS file systems, complicated operating
systems]
• Lots of bandwidth [fast data busses, general purpose communications
protocols]
– They don’t match the current paradigm
•
•
•
•
Flexibility is all-important [re-use, re-configure, adapt]
Specifications are malleable late in the game
Software changes, why can’t hardware?
We can catch problems in simulation and reprogram the part
– They aren’t fun
– We don’t train people to value constraints and work within them
• This is unfortunate because constraints can make our job
easier without degrading the end product
2005 MAPLD
59
Design Integrity Concepts
Summary
• The proportional design mindset is
important because it:
– Focuses on fulfilling needs, not wants
[specification orientation]
– Deepens understanding of the final design
[ownership oriented]
– Avoids unnecessary effort [efficiency oriented]
– Fosters simplicity that aids verification and
validation [quality oriented]
2005 MAPLD
60
Design Integrity Concepts
More Than the Sum of Its Parts
Design Partitioning
2005 MAPLD
61
Design Integrity Concepts
Agenda – Design Partitioning
• Definitions and examples
• Why partition an electronic design?
• Guidelines for partitioning an electronic
design
• Why isn’t it done more often?
• Summary
2005 MAPLD
62
Design Integrity Concepts
Definitions and Examples
• Partition – to divide into parts or shares
– Examples: budgeting, outlining, WBS development
• Design partitioning refers to the deconstruction of
a design into various sub-designs in an ordered
and logical manner
• Goal – to simplify the whole by optimizing the
parts and thus increase:
– Efficiency
– Reliability
– Maintainability
2005 MAPLD
63
Design Integrity Concepts
Why Partition (General)?
• Complexity interferes with ready comprehension
– Comprehension of a complex system depends on ability to impose order
upon it
– But … a given mind has a finite capability to impose order which depends
on the quantity and structure of the data related to the system
– The more complex the system, the more difficult its comprehension
• Partitioning a design introduces a piece-wise reduction in complexity
– Reduces quantity and complexity of design to manageable “chunks”
– Improves comprehension of the various parts of the design
– Increased comprehension of the parts leads to better comprehension of the
whole
• Better comprehension of the whole increases the ability to
– Verify correctness of the design
– Correct errors in the design
– Update the design when necessary
2005 MAPLD
64
Design Integrity Concepts
Why Partition (cont.)?
• Programmatic advantages
– Refines scope of work
• Identifies unexpected effort
• Identifies reuse possibilities
• Identifies staffing requirements
– Identifies schedule dependencies
• Improves allocation of resources
• Identifies parallelization and schedule enhancement
opportunities
– Promotes management visibility
2005 MAPLD
65
Design Integrity Concepts
Why Partition? - Design
• Improved interface organization / more formal structure
– Interfaces between functions have more predictable characteristics
– Expansion by addition of functions is more controlled
• Enhanced functional encapsulation
– Individual functions have predictable results when invoked
– Functional design enhancements have limited side-effects when
installed
– Effects of faults are more easily predicted and mitigated
• Efficient design implementation
– Functional (fewer functions and types of functions)
– Data flow (fewer, more sensible data busses)
– Control flow (simpler address decoding and state machines)
• All are side effects of additional thought put into “How?”
2005 MAPLD
66
Design Integrity Concepts
Why Partition? - Correctness
• Simpler inspection
– Functionality may be “obvious at a glance”
– Error space is more limited
• Within the function or within the interconnect
• Simpler Qualification
– Verification can begin with encapsulated modules or circuit subsets
– Overall functionality correctness becomes less of a “late” concern
because subset functionality is proven correct “early” in the
process
• Simpler / more thorough review
– Structure provides orientation to peer reviewers
– Encapsulation allows easier review
– Peer input more likely to be useful
2005 MAPLD
67
Design Integrity Concepts
Why Partition? – Maintainability
• Clearer documentation
– Documentation has “smaller” parts to focus on
– Structure of documentation grows from the design structure
• Simpler maintenance
– Changes affect only enhanced area
– Interactions between changed area of design and remainder of
original design is controlled by a formal structure
• Enhanced reuse
– Sub-circuits / functions usable in other applications as long as the
interface structure is observed
• Inherent capability of design better understood
2005 MAPLD
68
Design Integrity Concepts
Guidelines for Partitioning
• Take advantage of organizational strengths
– Expertise (analog, digital, software, etc.) is seldom the
same across organizations
– Partition the design in accordance with organizational
strengths according to primary functions
– Divide auxiliary functions (those that can be assigned
to multiple organizations) so that
• Interfaces are simplified
• Workload is equalized
• Functions are easily tested without requiring all of the
hardware
2005 MAPLD
69
Design Integrity Concepts
Guidelines for Partitioning (cont.)
• Example: Alice UV spectrograph electronics
(Rosetta)
• Core expertise
– Sensor Sciences: Detector and front end electronics
– Mobius Systems: Embedded software
– SwRI® space systems: Digital, low speed data
acquisition
– SwRI® space science: LV and HV power supplies
• Primary partition per core expertise
2005 MAPLD
70
Design Integrity Concepts
Guidelines for Partitioning (cont.)
• Alice example (work breakdown chosen)
– Sensor Sciences: detector electronics through parallel
digital output – simplified interface
– Mobius: instrument control / protocol servicing (some
error decoding partitioned to H/W)
– Space systems: microcontroller; s/c interface; heater,
motor, door digital control; analog high-voltage control
– Space sciences: LVPS; HVPS; motor, door, and heater
drive circuitry
2005 MAPLD
71
Design Integrity Concepts
Guidelines for Partitioning (cont.)
• Alice example – partitioning decisions
– Auxiliary expertise split along sensible
interfaces
• C&DH to detector – analog or digital? (noise, ease
of test) - digital
• C&DH to HVPS – analog or digital? (space, noise
susceptibility, ease of test) - analog
• C&DH to S/W – protocol decoding? (performance
margin, logic capability) – hardware error
decoding, s/w protocol encoding
2005 MAPLD
72
Design Integrity Concepts
Guidelines for Partitioning (cont.)
• Use specific functionalities
– Combine functionality with very similar characteristics
• Low-level analog / discrete bi-level (comparator is difference)
• Example: Spacelab flex interfaces
– Cluster related functionalities
• Shared data-flow direction, shared logical control
• Examples
– Constant level discretes / pulsed discretes
– Low-level discretes / high-level discretes
2005 MAPLD
73
Design Integrity Concepts
Guidelines for Partitioning (cont.)
• Use specific functionalities (cont.)
– Isolate related functional sub-groups from other items
• Ex. – analog data acquisition group (multiplexers, converters,
signal processing, control logic)
• Isolate
– Through appropriate data/address buffering
– Through separate programmable logic sub-sets
– Exploit directionality
• Write functions; read functions; read/write functions –
appropriate buffering
• Examples
– Write: Analog output, digital output, telemetry
– Read: Analog input, digital input, command
– R/W: Memory, bi-directional discretes, GSE
2005 MAPLD
74
Design Integrity Concepts
Guidelines for Partitioning (cont.)
• Exploit operational considerations
– Operational considerations often determine the specific
configuration of a set of common components
– Example: memory system
• Components: memory, write control, read control, sequencing,
buffering
• Application: telemetry system
–
–
–
–
Packetized / unpacketized?
Asynchronous timing / TDM ?
Science data / engineering data?
Pushed / pulled ?
• The type of telemetry system determines the partitioning
2005 MAPLD
75
Design Integrity Concepts
Example – Science Data Storage and Readback
System
• How should the logic be partitioned?
– Is write logic part of science data process or memory process?
– Is read logic part of telemetry system or memory process?
– How complicated is the arbitration?
• How it is partitioned depends on specific operational requirements
2005 MAPLD
76
Design Integrity Concepts
Example - New Horizons Alice Science Data
•
Alice Operation
–
–
•
Slow source accumulation relative to output speed
Push interface initiated by instrument
Alice Implementation (others likely in different circumstances)
–
–
–
–
Block 1: handshake and latch data
Block 2: Ingest data, process, write to memory
Block 3: Read, serialize, send, blank memory
Arbitration simplified to a switched double buffered memory access (no real-time arbitration)
2005 MAPLD
77
Design Integrity Concepts
Guidelines for Partitioning (cont.)
• Ensure encapsulation is reflected in the form of the
engineering documents
– Functions contain many types of operations
– Example: Telecommand interface
• De-serialization, decoding, error determination, re-packetization,
temporary storage
• Real time functionality [level 0], forwarding to software
• Storage access / arbitration, status log maintenance, data-bus handshaking
– Partitioning should ensure that all reasonable aspects of a function are
in one locality (we are ordering data for understanding)
• One (or a few) pages in a schematic
• One module in HDL
• One object in software
– Benefits: readability, error determination, testability, maintainability
2005 MAPLD
78
Design Integrity Concepts
Ordering Example – Bus Structure
• A logical bus structure
–
–
–
–
–
–
–
–
Simplifies data flow
Eases expansion / enhancement
Identifies bottlenecks / opportunities for efficiency
Ensures signal compatibility
Reduces timing uncertainty (capacitive loading)
Reduces power
Simplifies control logic / arbitration
Simplifies analysis
2005 MAPLD
79
Design Integrity Concepts
Ordering Ex. – Bus Structure (cont.)
Before:
After:
2005 MAPLD
80
Design Integrity Concepts
Ordering Ex. – Bus Structure (cont.)
• Directional data flow is clustered
• High-speed access devices (SRAM) not
buffered
• Exclusive functions clustered (PROM/
EEPROM)
• Simplification of
– Timing
– Loading
– Control
2005 MAPLD
81
Design Integrity Concepts
Why is Partitioning Not a Priority?
• It is – at the S/C, sub-system, and box level (sometimes)
• Why not at the board level?
– Requires potentially significant planning effort (schedule / cost)
– The tool syndrome (CAD / CAE)
• Crush creativity by forcing an early start to design
• Primarily a way to communicate between software packages rather
than humans (schematic => PWB)
– Too much junk being placed in too little space (simplification may
not always be space efficient)
– Lack of emphasis from senior engineers
– Boards aren’t where the action is (FPGAs) – less effort placed on
them
2005 MAPLD
82
Design Integrity Concepts
Why is Partitioning Not a Priority? (cont.)
• At the FPGA level
– Lack of solid design methodology
• Methodology must be tailored to a tool
• VHDL / Verilog are functional descriptions
• VHDL / Verilog don’t inherently enable data flow visualization
or block oriented deconstruction
• The synthesizer can understand non-partitioned design
– Perception of expansive resources (no / few constraints)
– The tool syndrome (“I must start coding”)
– Lack of effective design quality gates prior to start of
detailed design
2005 MAPLD
83
Design Integrity Concepts
Summary
• A design is only as solid as its weakest part
• Proper planning and partitioning of a design:
–
–
–
–
–
Ensures individual functions are logical and complete
Ensures interconnects are ordered and efficient
Provides for improved reliability / verifiability
Allows easier modification and enhancement
Enhances detailed understanding of how things work
• Partitioning requires ordering the design by considering
–
–
–
–
The capabilities of the team
The functionality of the modular pieces
How the design will operate
The individual components of a particular function
• Partitioning a design ensures that all parts are solid – resulting in a
solid whole
2005 MAPLD
84
Design Integrity Concepts
You Mean We’re Still Working
On It?
Sustaining a Design
2005 MAPLD
85
Design Integrity Concepts
Agenda – Design Sustainability
• Definitions
– Sustainable – Capable of being supported (AH)
– Sustainability – The characteristic of an item
that allows it to be supported
• Why is this important?
• Suggestions for sustainability
• Summary
2005 MAPLD
86
Design Integrity Concepts
Why Sustainability?
• Missions may be multiple decades long
– Flight systems may develop anomalous behavior
– Ground equivalents may need repair
– An understanding of the design is necessary to ensure
mission success
– Original designers will not be available for debugging
• Other critical assignments
• Working in telecon
• Cruising the Pacific
– Sustainable designs allow analysis and correction
without the access to the original designers
2005 MAPLD
87
Design Integrity Concepts
Why Sustainability? (cont.)
• Many designs are derivative
– Reuse of unmodified circuits essential for similar performance in
modified designs
– Acceptable modification depends on creative incorporation of what IS
• Derivation may take many years
– Example – Alice UVS
•
•
•
•
Design 1 – Rosetta (1997-2001)
Design 2 – New Horizons (2002-2005)
Design 3 – LRO (2005-2008)
Design 4 – Juno (2006 - ???)
– Staffing will not be constant
– Human memory will not be precise
• Sustainability ensures an ability to efficiently build on past
successes
2005 MAPLD
88
Design Integrity Concepts
Why Sustainability (cont.)
• You may not be the person who has to make it work
– Staffing is dynamic
• You may quit
• You may get re-assigned
• Somebody with more clout may be needed to satisfy the customer
– Teams produce a product and share debugging
• Test technician
• S/W designer
• I&T team
– Self-interest and common courtesy
• You don’t want 18 questions per day
• Ethics – Do unto others
– Example (ICB)*
2005 MAPLD
89
Design Integrity Concepts
Suggestions for Sustainability
• Remember the dual nature of design input
– The CAD perspective
• Schematic => PCB layout package => circuit board
• HDL => Synthesizer => Fuse file => Programmed FPGA
– The human perspective
• Schematic
– Interrelationships (time, space, connection)
– Debugging tool
– Functionality description
• HDL
– Describes functions and interaction
– Renders constraints understandable
– Ensure Readability!
2005 MAPLD
90
Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Record the design process
– Keep a notebook (type not vital)
– Describe everything of importance
• Why?
– Is this bus used for this function?
– Is this function implemented like this?
– Etc.
• How?
–
–
–
–
–
Do these things talk to one another?
Does this sequential logic work (state diagrams)?
Is the address map decoded?
Are errors handled?
Etc.
2005 MAPLD
91
Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Record the Design Process (cont.)
– Describe (cont.)
• What?
–
–
–
–
–
Signals are needed to perform this function?
Do the waveforms look like?
Timing do I expect to observe?
Changes have I made?
Etc.
• When? – Record chronology
– Provide a way to reproduce what was done
– Make part of permanent project record
– Example – Radarsat 1 Notebook
2005 MAPLD
92
Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Schematics
– Provide an overview of the design
• Schematic table of contents
• Block diagram (hierarchical design if available)
– Provide consistent naming scheme
• Descriptive of signal direction / function / polarity
• Consistent across logic gates and within various blocks
–
–
–
–
Cluster sub-circuits on contiguous pages
Make connections between components explicit
Add comments where necessary for clarification
Remove unused circuitry (for FPGA schematics)
2005 MAPLD
93
Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Don’t:
2005 MAPLD
94
Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Do:
2005 MAPLD
95
Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Help!
2005 MAPLD
96
Design Integrity Concepts
Suggestions for Sustainability (cont.)
• HDL (Must be done from beginning!)
– Provide overall orientation to design
– Provide top-level comments on
• Level of use (top, intermediate, etc.)
• Overall purpose of function / block / module
• Signal function and origination (external and internal)
– Provide operational comments on
•
•
•
•
State machine purpose and configuration (how? why?)
Transition logic (theory and reasoning)
Function of particular sequences
State to control signal translation
– Clarify obscure references
• Remove superseded code (don’t comment out) and explain uncommon
structures
– Improve readability
• Create logical file names
• Minimize file, logic block, function sizes
• Include related functions together (error generation, data interface, basic
function, etc.
2005 MAPLD
97
Design Integrity Concepts
Suggestions for Sustainability (cont.)
Comments?
2005 MAPLD
98
Design Integrity Concepts
Suggestions for Sustainability (cont.)
Header from same design!
(After the fact documentation)
2005 MAPLD
99
Design Integrity Concepts
Suggestions for Sustainability (cont.)
• Post process documentation
– Theory of Operation / Users Manual
• Generate One
• Include
–
–
–
–
Design concept / features
Operational Constraints
Appropriate Uses
Complete Engineering Documentation
• Update, release and correct as necessary
– Create design archive
• Self-consistent and complete
• Place under revision control
• Control changes
2005 MAPLD
100
Design Integrity Concepts
Summary
• Useful designs will be corrected, modified and
evaluated
– FOR A LONG TIME!
– By people besides you
• Sustainability measures must be implemented to
make this happen efficiently
• Sustainability requires
– Adequate conceptual documentation and records
– Clear and readable implementation records
– Finalized and controlled configuration records
• Ensuring sustainability will preserve your legacy
2005 MAPLD
101
Design Integrity Concepts
What’s the Exit Strategy?
Verifying a Design
2005 MAPLD
102
Design Integrity Concepts
Agenda – Design Verification
•
•
•
•
•
•
•
Concepts and implications
The specification connection
Verification before test
Test thoroughness
Tiered verification
Concluding the process
Summary
2005 MAPLD
103
Design Integrity Concepts
Verification Concepts
• Verification – Confirmation, through the provision of
objective evidence, that the specified requirements have
been fulfilled (Q9000-2000)
• Confirmation – the act of ratifying or corroborating (AH)
2005 MAPLD
104
Design Integrity Concepts
Verification Implications
• The verification process is not intended to be a “craps
shoot”
– Verification is primarily intended to highlight correctness
– Verification is NOT primarily intended to reveal incorrectness
• The mindset is important!
– Doubt that a design will work expresses a lack of confidence in the
designer and the design process
– Waiting until “verification” to guarantee correctness is an excuse
for being sloppy
• We should expect our designs to work! Verification simply
puts the seal of confirmation on the expectation
2005 MAPLD
105
Design Integrity Concepts
Verification Implications (cont.)
• Verification addresses two processes
– Design
• Primarily a one-time, iterative process in which the developed
concept is proven sound
• Fundamental correctness can be proven analytically
– Inspection confirms logical correctness in all circumstances
» Peer reviews are a manual form of inspection
» Functional simulation is an automated form of inspection
» Inspections must be tailored to design
– Analysis verifies correctness in the presence of variability
» Reliability (WC, Derating, FMEA, etc.)
» Timing over environments and age
2005 MAPLD
106
Design Integrity Concepts
Verification Implications (cont.)
• Verification addresses two processes (cont.)
– Instantiation
• Particular instance must be sound
• Particular correctness must be demonstrated experimentally
– Inspection confirms that instructions are followed
» Correct components installed
» Correct configuration selected
» Correct processes performed
– Test and demonstration confirms that operation matches
expectations
» Functionally and parametrically
» Predicted by nominal analytical predictions
2005 MAPLD
107
Design Integrity Concepts
Verification Implications (cont.)
• Verification “after the fact” is too late
– Analysis and test are NOT equivalent
– Design analysis and inspection must proceed in lockstep with
design processes
– Implementation tests and inspections should simply confirm what
is known
• Design efforts fail because they occur in a vacuum
– Creativity takes primacy over correctness
– Functionality comes first with operability being “shoehorned” after
the fact
• The “monster” simulation syndrome
• Conclusion: Verify as you proceed
2005 MAPLD
108
Design Integrity Concepts
Verification Implications (cont.)
• Correctness is a matter of personal integrity
for the designer
– Verification is not simply externally imposed
task
– Verification is something that the designer must
want to do (a part of his/her ethos)
– Verification is confirmation that the designer is
“worth his/her salt”
2005 MAPLD
109
Design Integrity Concepts
The Specification Connection
• “Requirements” are confirmed during verification
– The designer does not have a free hand with respect to how a
design is confirmed
– Specification provides the constraint set under which verification is
prosecuted
– Therefore, verification must be defined prior to start of design
• Not simply in a general manner
• Specifically
– Since specification is a customer and vendor document
• Both customer and vendor must be involved in the verification
process
• “Trust me” is not a reasonable phrase when it comes to verification
2005 MAPLD
110
Design Integrity Concepts
The Specification Connection (cont.)
• Specification contains a complete [ideally] set of requirements for the
design
– Some requirements are very specific
• “All discrete outputs shall have a source impedance of 2 kOhms.
• An adequate test is fairly obvious
– Some requirements are not very specific
• “The unit shall provide 512 kbytes of SRAM”
• Note the implied requirement – The SRAM shall function correctly
• What is an adequate functional test for this requirement? (walking 1’s, 0’s
addressing, etc?)
– Some requirements lend themselves to quantitative analysis – “The unit
shall provide a .1° C accuracy at end of life”
– Some requirements lend themselves to simple inspection – “The unit shall
provide 4 analog output channels”
– When do we decide adequacy of the method of verification and the
individual verification cases?
• – Before we start designing
2005 MAPLD
111
Design Integrity Concepts
The Specification Connection (cont.)
• Therefore, verification planning must begin with
specification creation
– Each requirement must be assigned a method or
methods
– Each requirement must be assigned a test or analysis
case
• Must answer question – What is an adequate verification?
• Must establish answer to question – When are we done?
– Each requirement must be verified at one or more levels
[FPGA, board, box, sub-system, system]
– Customer must concur with decision
• Note that modern verification databases make this
relatively straightforward
2005 MAPLD
112
Design Integrity Concepts
The Specification Connection (cont.)
• Advantages to this type of verification planning
– More verifiable / verified designs
• Up-front focus on programmatically difficult verification can
be instituted
– Earlier analysis
– Simplification of overly complex circuit designs
• Guidance for lower-level verification efforts can be established
– Sub-module vs. module level
– Module vs. unit level
• Integral self-test provisions can be included in design
• Earlier simulation vector definition (FPGAs / logic)
2005 MAPLD
113
Design Integrity Concepts
The Specification Connection (cont.)
• Advantages (cont.)
– More robust test sets
• Early knowledge regarding end-item function communicated
• Capability of test set matched to need
– Precision (timing, voltage, current, etc.)
– Speed
– Test level (component, unit, system)
• Test flow design considerations included in test set design
– Knowing verification requirements early makes the
entire development process more efficient
2005 MAPLD
114
Design Integrity Concepts
The Specification Connection (cont.)
• Defining requirements and test cases at the same
time as the specification clarifies and simplifies
the FPGA verification process
– Allows early start to simulation vector design
– Creates early knowledge of additional functional
models (SRAM, peripherals, etc.) needed
– Clarifies decision regarding what can be functionally
simulated and what must be simulated with backannotation
– Defines levels at which simulations must be run
2005 MAPLD
115
Design Integrity Concepts
Verification Before Test
• It is a relatively bad thing to discover a design flaw during
test (better than nothing)
– Late in the development cycle
– Correction is more complicated / expensive
– Personal stress is higher
• Yet … a surprising lack of verification occurs early (before
test)
• Types of pre-test verification
– Inspection - conformity evaluation by observation and judgment
accompanied, as appropriate by measurement or testing (ISO)
• Personal self-check
• Peer Review
• Basic functional verification (Signal audits, functional simulation,
etc.)
2005 MAPLD
116
Design Integrity Concepts
Verification Before Test (cont.)
• Types of pre-test verification (cont.)
– Analysis – Conformity evaluation of the predictions
produced by realistic models of the system
• Worst case analysis (voltage, current, frequency, time) using
models that incorporate real world degradation of function
• Derating analysis using models that reduce stress
• Other
• Note that pre-test verification is fundamentally
conceptual
– Not tool dependent
– Not design dependent
– Can be accomplished many ways
2005 MAPLD
117
Design Integrity Concepts
Verification Before Test (cont.)
• Two approaches to pre-test verification
– Verify full design at once (one big peer review, code walkthrough,
analysis, simulation)
• Benefits
–
–
–
–
Complete design reduces need to develop assumptions
When proved correct, does not need to be repeated
When design is solid, less work
Reduces final documentation for verification
• Drawbacks
– Allows small design flaws to propagate for a long time
– Complexity reduces visibility (verification more prone to mistakes and
oversights)
– When design is flawed, it increases work
– Leads to corrections that are global (hard to test effectively and predict
side effects)
– May take longer to execute than an aggregate of smaller verification
activities
2005 MAPLD
118
Design Integrity Concepts
Verification Before Test (cont.)
• Two approaches to pre-test verification (cont.)
– Verify incremental designs
• Benefits
–
–
–
–
Supports development of “known good” sub-designs
Meshes with a partitioned design approach
Facilitates visibility (fewer mistakes, clearer goals)
Allows small flaws to be fixed quickly (minimizes downstream
impact)
• Drawbacks
– Is more work in the ideal case (no/few flaws)
– Increases documentation if formality is vital
• Either approach can be made to work, but one or
the other must be chosen and implemented
2005 MAPLD
119
Design Integrity Concepts
Test Thoroughness
• Principles for test thoroughness
– Every “shall” in the specification that meets the following criteria must be
tested
• Items that may be affected by the instantiation process (subject to fabrication
error)
• Items for which the only extant verification is model based
• Items for which the testing does not pose unacceptable risk (radiation,
overstress, etc.)
– Every instance of a “shall” must be tested
• 18 interfaces require 18 specific tests
– Requirements that have an exclusive character
• Must be tested for correct operation
• Should be tested at some level for abnormal operation
• Example: If you write a 5Ah to something to set it on, you should verify that
writing other values does not set it on
– Requirements must be tested over a representative range of conditions
2005 MAPLD
120
Design Integrity Concepts
Test Thoroughness (cont.)
• A thorough test is complicated and time
consuming
– Requires some level of automation to be
comprehensive
– Requires some level of compromise to be
practical
– Requires team agreement to be acceptable
2005 MAPLD
121
Design Integrity Concepts
Test Thoroughness (cont.)
• Ensuring thorough, yet practical, tests
– Define which tests require manual interaction and which can be
automated
• Output impedance or analog fine precision may need to be manual
• Signal level or analog coarse precision may be automatable
– Define which tests must be performed over environmental
conditions and which can be performed at room temperature only
• Make decision based on character of measurement
• Do not decide based purely on practicality
– Define level of complexity for all measurement sets
• Example: 24 analog inputs (3 eight channel muxes)
• Test full range of values for 24, or limited range for 7 of 8 mux inputs
and full range for remaining
– Define need for repetition of measurement at higher level of
integration (tiered verification)
• Example: voltage level / signal quality / clock jitter at board level
• Repeat at box level?
2005 MAPLD
122
Design Integrity Concepts
Test Thoroughness (cont.)
• Test thoroughness issues affect everything
– Customer relationship
– Test set design (board, box, system)
– Schedule / cost
• An adequate test program is not trivial
– Cannot wait until system level
– Must incorporate lowest level of design
– Must be a constant background activity during design process
• All programs must balance perfect and good enough
• A good test program
–
–
–
–
Will ensure that the specification is met
Will verify everything that can be affected by fabrication
Will build on the analysis and inspection effort
Will maximize automation without sacrificing test accuracy
2005 MAPLD
123
Design Integrity Concepts
Tiered Verification
• Tiered verification is the process of matching the correct type of
verification to the appropriate level of integration
• Tiered verification incorporates all types of verification (before and
during testing)
• Tiered verification applied to the test regime attempts to have
thoroughness with practicality
– Test a requirement at the lowest conceivable level
– Determine which tests must be repeated at higher levels by
• Establishing the purpose of a test at a particular level
• Determining whether the next level has the potential to change previous
measured quantities
• Determining what test set capabilities are
• Tiered verification cannot be retrofitted into the process
– Must be planned up front
– Must be implemented throughout the development process
2005 MAPLD
124
Design Integrity Concepts
Tiered Verification - Example
• Need: A set of 16 high-level pulsed discretes to control
latching relays used to change the spacecraft power
configuration
• Basic Requirements
–
–
–
–
–
–
Quantity:
Level:
Load:
Duration of pulse:
Rise / Fall time max:
Command capability:
– S/W requirements:
2005 MAPLD
16
+30V +/- 2 V
2 kOhm, 4 mH
50 ms
1 ms
Unique bit pattern, only one at a time,
arm first
Various (beyond scope of example)
125
Design Integrity Concepts
Tiered Verification – Example (cont.)
• Tier 1 – Basic Verification (functional simulation)
– Verify each channel fires for a specific bit pattern /
address
– Verify cannot be fired without an arm command
– Verify that other complementary patterns don’t
accidentally fire the pulse
– Analyze drive capability of FPGA and input / output
requirement / capability of translation chip
– Evaluate glitch hardness of logic selected
2005 MAPLD
126
Design Integrity Concepts
Tiered Verification – Example (cont.)
• Tier 2 – Board level
– Test to confirm signal level between FPGA and
driver chips is compatible
– Test rise, fall, duration, level with dummy load
– Repeat test for all 16 outputs
– Replace dummy load with relay, verify that the
relay switches for all 16 outputs
– Repeat over temperature if necessary
2005 MAPLD
127
Design Integrity Concepts
Tiered Verification – Example (cont.)
• Tier 3 – Box level
– Attach GSE (verifies with relays)
– Functionally pulse all outputs
• Verify relay switches
• Verify gross drive duration
• Tier 4 – Box and operational software level
– Verify functionality commanded through software
– GSE verifies correct relay switch
• Tier 5 – System
– Verify gross functionality as needed for system
operation
2005 MAPLD
128
Design Integrity Concepts
Tiered Verification - Notes
• Most complex and detailed functionality is verified at
lower levels
– Performance
– Error cases
– Predictive of future operation
•
•
•
•
Higher levels focus on functionality / operation
Lower level verification requires more manual touch
Higher level verification is more automated
STE and test procedure developed as appropriate
– Purpose is maximum test completeness
– As well as “bang for the buck”
2005 MAPLD
129
Design Integrity Concepts
Concluding the Process
• (“… through the provision of objective evidence…”)
• The importance of traceability
– Something is only verified (formally) when it is documented
– The tie between verification item and specification must be made
(verification matrix or database)
• The importance of planning
– Establish the links between requirement, method, level, test case
early
– Use the established structure to develop verification items
• The importance of buy in
– All responsible must complete verification and report results
– Systems engineers must track and close
– People must keep up with the process
• Only personal commitment from all involved will ensure
that the process is successful
2005 MAPLD
130
Design Integrity Concepts
Summary
• Verification is to prove success, not avoid failure
• Verification planning is an integral part of the project lifecycle
– Must be initiated with specification (including customer buy-in)
– Must be included in design effort
– Must be used to develop documentation and appropriate test hardware
• Verification is more than final test
– Begins at lowest possible level
– Incorporates analysis and inspection throughout design process
– Develops proof of compliance at all levels of operation
• A thorough test is complicated
– Requires extensive work
– Must trade perfect for practical (a tiered test approach can help)
• Verification processes must be tracked and documented
– The whole team must buy in
2005 MAPLD
131
Design Integrity Concepts
What Do You Mean It Doesn’t
Do What We Thought?
Validating a Design
2005 MAPLD
132
Design Integrity Concepts
Agenda – Design Validation
•
•
•
•
•
Concepts and implications
Specification mitigations
Design mitigations
Test set mitigation
Summary
2005 MAPLD
133
Design Integrity Concepts
Concepts
• Validation – Confirmation, through the provision of
objective evidence, that the requirements for a specified
intended use or application have been fulfilled
2005 MAPLD
134
Design Integrity Concepts
Implications
• The issues relating to validation encompass those of verification
• Additional concerns with validation
– Caused by the need to match the application to the product
– Desired application has been translated to specification through the
requirements process
– Requirements process is by nature imperfect
– Sometimes the specification does not satisfy the needs of the application
• Result – a verified product might be invalid
– May require significant rework to the product
– May require accepting reduced functionality (waiver)
• A goal of the development process is to minimize validation failures
– Begins in review of the requirements process (hopefully primary point)
– Mitigate by design activities
– Reduce by robust test set design
2005 MAPLD
135
Design Integrity Concepts
The Implication Illustrated
• Alice electronics (detector component and C&DH
component)
• Joint requirement: process > 10 k sustained events per
second
• Individual requirements defined for detector and C&DH
processing
– Both met individual requirements for processing
– When combined only 6-7 k sustained events per second
• Verification of individual units led to invalid system
• What went wrong?
– The overall requirements were not broken down correctly
– The C&DH and DE test sets were not high fidelity
• Verified functionality, not performance
• We got lucky that a waiver was acceptable
2005 MAPLD
136
Design Integrity Concepts
Specification Mitigation
• Only list requirements, not desirements
• State unambiguous performance requirements
• Build in adequate margin
– Not open-ended enhancement, but
– Enough to accommodate performance shortfalls
• Ruthlessly remove TBDs
• Insist on definite test methods for mitigation
• Remember – Unless application needs can be
unambiguously specified, they won’t be met!
2005 MAPLD
137
Design Integrity Concepts
Design Mitigation
• Implement specification exactly
• Isolate various sub-sections
– Minimizes “corner cases” and negative interactions
– Allows correction with minimal impact when things don’t work
right
• Verify complex functions early, thoroughly, and
completely
– Allows early look at potential problems
– Analysis / simulation / what ifs should be as realistic as possible
• Insist on end-user review of implementation
– Allows user community to comment
– Minimizes misunderstandings upon delivery
• Develop test plans that have high fidelity to the end
application
2005 MAPLD
138
Design Integrity Concepts
Test Set Mitigation
• Ensure interfaces are maximally flight-like
– Precludes misunderstandings of characteristics
– Provides early indication of problems
– Don’t emulate only one characteristic of interface (R vs. R+L)
• Make test set reasonably sophisticated
– Sufficient complexity to reproduce operational timing
– Adequate functionality for stress testing
• Run all interfaces at maximum speed with margin
• Don’t let the same group build the tested unit (design) and
the unit tester (test bench)
– Identical assumptions might go into both ends of an interface
– Faithful reproduction is dependent on familiarity (if possible, test
bench should be provided by end user)
2005 MAPLD
139
Design Integrity Concepts
Test Set Mitigation (cont.)
• Make the control interface as application like as
possible
– Forces correct command structures / types
– Allows all test scripts to be reproduced at higher levels
• If at all possible, incorporate early interface tests
of real engineering hardware
• Keep the test (or simulation) environment constant
unless the flight system changes
– Don’t change test equipment hardware configurations
– Apples to apples comparisons during tests vital
– Ensure that flight changes are reflected in test set
design
2005 MAPLD
140
Design Integrity Concepts
Test Set Mitigation (cont.)
• Use the same controls for test set development as for flight
unit development
– Configuration management
– Software development
– Peer reviews
• Build in diagnostics so that anomalies can be traced to test
equipment or unit under test
• Ensure that test results mean something
–
–
–
–
Pass / fail criteria clear
Allowable flight parameter variations included
Reasonable displays (with significant information clearly shown)
Ensure that test set accommodates calibration
2005 MAPLD
141
Design Integrity Concepts
Summary
• Successful verification does not always guarantee
successful validation
• Techniques can be incorporated that improve the
likelihood that validation will succeed
– Careful specification development
– Thorough and cautious design techniques
– Extensive test set fidelity to flight requirements
• Effective techniques for validation are extra effort
– More time consuming
– More expensive
– But, definitely worth it.
2005 MAPLD
142
Design Integrity Concepts