Standards Conformance Testing and
M&S Web Services
Bernard P. Zeigler, Ph.D.,
Arizona Center for Integrative Modeling and
Simulation
and
Joint Interoperability Test Command
Fort Huachuca, AZ 85613-7051
Outline
• Role of Standards Conformance Testing in Net-Centric T&E
–
–
–
–
JITC responsibility for Link-16 standards conformance testing
Link-16: Challenges to implementation and testing
M&S–based Automated Test Case Generation
Application to Link-16 in the IABM SIAP Context
• Service Oriented Architecture: M&S-based Testing Services
– Link-16 standards conformance testing
– certification for GIG deployable services
– exposing M&S-based T&E assets as services
Net-Centric Test and Evaluation:
Some Assertions
Net-Centric T&E
•
places an increased emphasis on standards conformance
testing (SCT)
–
–
•
performed on distributed net-centric test-bed
infrastructure
–
–
•
SCT will be the first gate before more extensive T&E
necessary, but not sufficient, for interoperability and mission
effectiveness
Global Information Grid (GIG)
provide on-site or downloadable SCT packages
emphasizes services rather than systems
–
testing of Service Oriented Architecture (SOA)
– GIG enterprise services
JITC is the Responsible Test Organization
for Tactical Data Link Standards
• Joint Interoperability Test Command (JITC) is responsible
for ensuring systems that implement Tactical Data Link*
(TDL)
– are interoperable and
– in compliance with the applicable joint standards
• This is accomplished by conducting the following types of
tests:
– Joint / NATO /Combined Interoperability
– Performance Assessment in Operational Environments
– Standards Validation
– Standards Conformance
• JITC employs a variety of tools that provide its analysts the
ability to evaluate TDL system performance in both the lab
and live environments.
source: http://jitc.fhu.disa.mil
* Link 11/11B/16 and Variable Message Format (VMF)
Link-16: Challenges to Implementation and Testing
Joint Single Link Implementation Requirements Specification
JSLIRS is an evolving standard (MIL-STD-6016c) for tactical data
link information exchange and networked command/control of
radar systems
•
Presents significant challenges to automated conformance testing:
– The specification document states requirements in natural language
– open to ambiguous interpretations
– The document is voluminous
– many interdependent chapters and appendixes
– labor intensive and prone to error
– potentially incomplete and inconsistent.
•
Problem: how to ensure that a certification test procedure
– is traceable back to specification
– completely covers the requirements
– can be consistently replicated across the numerous contexts
– military service, inter-national, and commercial companies
Benefits of Formalization and Automation
• Provides traceability to original specification
• Reduces ambiguity from textual specification
• Facilitates integrating Modeling & Simulation into
the testing process
• Enables testing of complex:
–
–
–
–
Standards
Systems
Functions
Families of systems
SIAP/IABM —
Successor to Link-16
• SIAP (Single Integrated Air Picture) Goal: Improve
the adequacy and fidelity of information to form a
shared understanding of tactical situation
• Integrated Architecture Behavior Model (IABM)
requires that all sensors utilize a standard reference
frame for conveying information about the location of
targets.
• Developed by the Joint SIAP System Engineering
Organization (JSSEO), Arlington, Va., a sub-office of
the Assistant Secretary of the Army for Acquisition,
Logistics and Technology.
source: http://www.navyleague.org/sea_power/mar_04_18.php
Automated Test Case Generator (ATC-Gen)
• IABM is an extension of Link-16 developed in HLA
environment and requires HLA simulation-based testing
• JITC has taken the initiative to integrate modeling and
simulation into the automation of the testing process
• Funded the development of Automated Test Case Generator
(ATC-Gen) led by ACIMS using DEVS (Discrete Event
System Specification) technology.
• In R&D of two years, proved the feasibility and the general
direction
• First major test of IABM – Config05 – ATC-Gen is the basis
for testing link-16 and extended IABM requirements
Discrete Event Nature of Link-16 Specification
Transaction Level - example P.1.2 = Drop Track Transmit
1
Preparation
Constraints
(Exception)
Rules
2
Processing
Modify C2
Record for TN
3
Transmit
Msg
Validity
checking
Track
Display
Rule
Processing
Time
outs
Operator
decisions
Periodic
Msg
Other ConsequentProcessing
Stop
Stop, Do Nothing,
Alerts, Or jump to other
Transaction
Jumps (stimuli) to other
Transactions of specification
Output from
Input to
system
system
DEVS
t
1
t
2
t
3
t
4
ATC Gen Goals & Approach
Goals:
• To increase the productivity and
effectiveness of standards conformance
testing (SCT) at Joint Interoperability Test
Command (JITC)
• To apply systems theory, modeling and
simulation concepts, and current software
technology to (semi-)automate portions of
conformance testing
Objective:
Automate Testing
Capture Specification
as If-Then Rules in XML
Analyze Rules to
Extract I/O
Behavior
Synthesize DEVS
Test Models
Test Driver Executes
Models to Induce
Testable Behavior in
System Under Test (SUT)
Interact With SUT Over Middleware
Test Driver
DEVS Simulator
SUT
HLA
HLA
Network
ATC Gen Tool
• Standard to XML Translation
– Analyst interprets the requirements text to extract state
variables and rules, where rules are written in the form:
If P is true now
Condition
Then do action A later
Consequence
Unless Q occurs in the interim
Exception
• Dependency Analysis & Test Generation
– Dependency Analyzer (DA) determines the relationship
between rules by identifying shared state variables
– Test Model Generator converts Analyst defined test
sequences to executable simulation models
• Test Driver
– Test Driver interacts with and connects to SUT via HLA or
Simple J interfaces to perform conformance testing
– Validated against legacy test tools
Capturing the Link-16 Standard in XML
Repository
Appendix P
Appendix U
Section 4.11.13
Appendix P
Appendix U
Section 4.11.13
Appendix D
Appendix D
MIL-STD-6061C
XML FILES
Rule Analysis
Extract I/O Behaviors
Appendix P
Appendix U
Section 4.11.13
Appendix D
MIL-STD-6061C
RULE ANALYSIS
RESULTS
Test Sequence
Manually Derive & Create Paths
Automatic Generation of
Path File (Text)
ResponderNnot specified>
D.1.3.1.9ND.1.3.1>
D.1.3.3.10.1a,
D.1.3.3.13.1ND.1.3.1>
D.1.3.12.1ND.1.3.1>
Test Sequence
Validation & Generation
Completed Path File
Repository
Appendix P
Appendix U
Section 4.11.13
Appendix D
XML FILES
DEPENDENCY
ANALYZER
Validates
Test Sequence
GENERATED
TEST SEQUENCE
(XML)
Test Case
Generation
Test Sequence with Assigned Values
TEST CASE
GENERATOR
GENERATED
TEST CASE
(XML)
Test Model
Validation & Generation
Test Case
TEST MODEL
GENERATOR
GENERATED
TEST CASE
MIRROR
(XML)
Test Model
#include "hierSequence.h"
#include "PPLI.h"
#include "RemoteTNdrop.h"
const
const
const
const
port_t
port_t
port_t
port_t
hierSeqDigraph::start=0;
hierSeqDigraph::inJmsg=1;
hierSeqDigraph::pass=2;
hierSeqDigraph::outJmsg=3;
hierSeqDigraph::hierSeqDigraph():staticDigraph()
{
PPLI *pp = new PPLI();
add(pp);
couple(this, this->start, pp, pp->start);
couple(pp, pp->outJmsg, this, this->outJmsg);
RemoteTNdrop *p1 = new RemoteTNdrop();
add(p1);
couple(this, this->start, p1, p1->start);
couple(this, this->inJmsg, p1, p1->inJmsg);
couple(p1, p1->outJmsg, this, this->outJmsg);
}
Test Model
Execution
Test Model (C++)
#include "hierSequence.h"
#include "PPLI.h"
#include "RemoteTNdrop.h"
const
const
const
const
port_t
port_t
port_t
port_t
hierSeqDigraph::start=0;
hierSeqDigraph::inJmsg=1;
hierSeqDigraph::pass=2;
hierSeqDigraph::outJmsg=3;
hierSeqDigraph::hierSeqDigraph():staticDigraph()
{
PPLI *pp = new PPLI();
add(pp);
couple(this, this->start, pp, pp->start);
couple(pp, pp->outJmsg, this, this->outJmsg);
RemoteTNdrop *p1 = new RemoteTNdrop();
add(p1);
couple(this, this->start, p1, p1->start);
couple(this, this->inJmsg, p1, p1->inJmsg);
couple(p1, p1->outJmsg, this, this->outJmsg);
}
TEST DRIVER
SYSTEM UNDER TEST
Extending M&S-Based T&E to the GIG
•
Migrate ATC-Gen capability to the GIG-SOA
– accessible to all developers requiring Link-16 conformance testing
and certification
– enable self-tests for Link-16 standards conformance
– expose Link-16 XML-rule repository as authorized reference model
– register ATC-Gen test capabilities as web services
•
Extend M&S-Based automated testing
– provide mission thread testing for GIG deployable services
– certify transitions through the stages of DISA’s Federated Distributed
Certification Environment
•
Expose M&S-based T&E assets as services
– experimental frame components: generators, acceptors, transducers
– exploit DEVS reusability support
– characterize services for discovery and reuse
Link-16 XML Repository Web-Service Asset
• Organized according to MIL-STD-6016C
macro-structure hierarchy
• Folders store aggregations/abstractions
of lower level rules
• MIL-STD-6016C Reference Model
– Removes ambiguity
– Annotates problems areas, improving the
ability to find and fix issues
– Provides organization for indexing states,
rules, and variables
– Supports test generation and executable
rule construction
Transferring DEVS-based Testing
Methodology to SOA
•
•
•
DEVS-based Testing Methodology readily migrates to SOA
environment as shown below
In particular, the ATC-Gen Test Driver can be offered as a
service for Link-16 conformance self-test
Further, the methodology can be applied to other test and
evaluation contexts
DEVS Model
DEVS Model
DEVS Simulator
DEVS Simulator
HLA
Packaging:FOM
Messaging:Interactions,Updates
SOA
Service Discovery: UDDI
Sevice Description: WSDL
Packaging:XML
Communication: RTI
Messaging:SOAP
Communication: HTTP
Automating Mission Thread/Vignettes for
GES Testing
•
Objective – automate to the maximum extent the definition of mission
threads/vignettes and the generation of test services based on them
•
Enable COI and community wide service users and developers to easily
write threads/scenarios in (restricted) Natural Language form
•
Interacting with such users/developers, implement these scenarios
with DEVS test federation models that can execute over Core ES
•
Mission thread/Vignette scenario testing applies objectives-driven
testing to all service levels and throughout service offering life cycle
•
Repository can support re-use and re-application for regression testing
to preserve service behavior under continual change in SOA
environment
Mission Thread Testing Applies to all Service Levels and
Throughout Service Offering Life Cycle
Mission Threads enable objecti
ves driven testing at every deve
lopment level
Community-wide Enterprise
Services
COI Enterprise Services
CORE Enterprise Services
Mission Threads enable
objectives driven testing
throughout service offering
life-cycle
NLP Mission Thread Scenario Generation
JTAC working with FourthInfDiv
….
…
USNAircraft gets deconfliction from
USMCAircraft
Natural
Language
Parser
Restricted Natural Language
form of Mission Thread Scenario
Specification
Transform
to System
Entity
Structure
Scenario
in XML
(actor = JTAC, action = work, actee = FourthInfDiv)
….
(actor = JTAC, action = requests, message =
ImmediatCAS, actee = DirectAirSuppAirborne)
….
Transform
to DEVS
Parser outputs tagged
equivalent of input
DEVS Test
Driver
for NCES
Mission Threads
Example: Joint Close Air Support
Scenario
JCAS JMT Operational Scenario #1
A. Special Operations Force (SOF) (AFSOC and NSW) JTAC working with
Operational Detachment-Alpha (ODA) is tasked to request Immediate
CAS on a stationary mechanized target in mountainous terrain. A
Predator unmanned aerial vehicle (UAV) is on station for support.
B.
SOF JTAC contacts AWACS with request. AWACS passes the request
to Special Operations Liaison Element (SOLE) in the Combine Air
Operations Center (CAOC).
C. Joint Special Operations Task Force (JSOFT) approves the request and CAOC
assigns a section of USMC F/A-18Ds, F-15Es, and a single B-1B.
Ordnance consists of 20mm, Joint Direct Attack Munitions (JDAMs), and
Laser Guided Bombs (LGBs).
D.
Aircraft get situational brief from AWACS aircraft while in route, then switch
to SOF JTAC for Terminal Attack Control and deconfliction
from orbiting
UAV. A 9-Line brief will be given to each section/single aircraft. JTAC will
continue to execute CAS missions until all weapons are expended.
NLP Test Generation Example:
Joint Close Air Support Scenario
JTAC works with ODA!
JTAC is supported by a Predator!
JTAC requests ImmediateCAS to AWACS !
AWACS passes requestImmediateCAS to CAOC!
CAOC assigns USMCAircraft to JTAC!
CAOC sends readyOrder to USMCAircraft !
USMCAircraft sends sitBriefRequest to AWACS !
AWACS sends sitBrief to USMCAircraft !
USMCAircraft sends requestForTAC to JTAC !
JTAC sends TACCommand to USMCAircraft !
USMCAircraft sends deconflictRequest to UAV!
USMCAircraft gets targetLocation from UAV!!
NCES
DEVS-based Web-Services Testing
DEVS
Test
Player
Service
Under
Test
NCES/GIG
Live
Test
Player
SOAPXML
DEVS Test
Federation
DEVS
Simulator
Node
Demo:
http://acims9.acims.arizona.edu
JCAS JMT Operational Scenario #1
A. Special Operations Force (SOF) (AFSOC and NSW) JTAC
working with Operational Detachment-Alpha (ODA) is tasked to
request Immediate
CAS on a stationary mechanized target in mountainous
terrain. A Predator unmanned aerial vehicle (UAV) is on
station for support.
B. SOF JTAC contacts AWACS with request. AWACS
passes the request
to Special Operations Liaison Element (SOLE)
in the Combine Air Operations Center (CAOC).
C. Joint Special Operations Task Force (JSOFT) approves the
request and CAOC
assigns a section of USMC F/A-18Ds, F-15Es,
and a single B-1B. Ordnance consists of 20mm, Joint Direct
Attack Munitions (JDAMs), and
Laser Guided Bombs (LGBs).
D.
Aircraft get situational brief from AWACS aircraft while
in route, then switch to SOF JTAC for Terminal Attack
Control and deconfliction
from orbiting UAV. A 9Line brief will be given to each section/single
aircraft. JTAC will continue to execute CAS missions
until all weapons are expended.
Web Service:
Find aircraft available
for CAS given coordinates and time frame.
M&S-Based T&E Services on the SOA:
Experimental Frame Repository
• Compose EF from
generator, acceptor,
and transducer
components in
repository
• Characterize
functionalities using
M&S Theory and
WSDL
• Support composability
based on migration of
DEVS technology to
SOA
SUT
Generator
Acceptor
Transducer
Experimental Frame
Request
Generator
Repository
Acceptor
Repository
Transducer
Repository
Example: Synthetic Environment
Repository
Experimental
Frames needed
for Testing
Objectives
Experimental
Frame Partial Order
Experimental
Frame- Model
Applicability Relations
Environmental and
Standards Reference
Models available for
use in Testing
Summary and Conclusions
There is an acute need for a new Net-centric testing paradigm at the
enterprise level where joint and coalition operations are
conducted (Editorial, ITEA Jnl, Sept, 2005)
•
Net-Centric T&E places an increased emphasis on standards
conformance self-testing over the GIG-SOA and emphasizes services
rather than systems
•
Due to its formal and abstract basis, the DEVS functionality migrates quite
directly to execute over the web-services middleware based on XML and
SOAP
•
This extension enables semi-automated construction of DEVS test
federations for verification testing of a wide variety of proposed web
services:
–
–
•
ATC-Gen Link-16 and other TDL standards conformance self testing
Mission thread or vignette scenario specification using natural language input
can replace Link-16 style standards formalization as the source input for test
case generation
M&S-based T&E services can exploit experimental frame methodology
Contact:
Bernard P. Zeigler
zeigler@ece.arizona.edu
More information:
ACIMS
www.acims.arizona.edu
JITC
jitc.fhu.disa.mil
Joint Interoperability, Assured Security, Best Value, Customer Success
Global Net-Centric Solutions -- The Warfighter's Edge