THE HIGH LEVEL ARCHITECTURE
FEDERATE CONFORMANCE TESTING PROCESS
(REVISED)
Margaret Loper, Thom McLean, Margaret Horst, Kyle Crawford
Georgia Tech Research Institute
Georgia Institute of Technology
347 Ferst Drive
Atlanta, Georgia 30332
Keywords
High Level Architecture, Compliance Testing, Test Process, Test Tools
ABSTRACT: Conformance testing is the process of verifying that an implementation performs in
accordance with a particular standard. HLA Conformance testing ensures that a federate performs in
accordance with the Interface Specification and the Object Model Template (OMT) standards, per the HLA
compliance checklist. In order to determine if a federate performs in accordance with the interface
specification and OMT, a series of conformance tests are conducted. This paper describes the current
state of the test process and the tests that implement the process. U.S. Department of Defense (DoD)
federates will follow the process described in this paper to determine compliance with HLA.
1. Introduction
The HLA Compliance process has two parts:
Conformance and Certification. Conformance is
the process of verifying that an implementation
performs in accordance with a particular
standard [1]. HLA Conformance testing ensures
that a federate performs in accordance with the
Interface Specification [2] and the Object Model
Template (OMT) [3] standards, per the HLA
compliance checklist [4]. In order to determine
if a federate performs in accordance with the
interface specification and OMT, a series of
conformance tests are conducted, as described in
the remainder of this paper.
Certification is the process of validating that a
federate has been tested for conformance. This
means that once a federate has completed
conformance testing, the test results must be
validated before the federate can be certified as
“HLA Compliant.” A Certification Agent who
is accredited by the government executes the
certification process. The Certification Agent is
legally responsible for validating a federate’s
conformance test results.
It is important to note that conformance is
specific to a federate, not a federation.
Therefore once a federate has passed
conformance testing and has been certified as
HLA Compliant, it can be reused as often as
needed in federations.
Also note, HLA
Compliance does not guarantee federation
interoperability, rather it is the first step.
2. Overview of the Conformance
Process
Conformance testing is a four-step process, as
shown in Figure 1. The process described below
is the one the DoD will follow to determine if a
federate complies with the HLA. More details
on the conformance tests can be found in the
remainder of this paper.
T h e T e st P ro c ess
F e d e ra te U n d er T e st
STEP 1
C e r t if ic a t io n A g e n t
R e q u e s t I n f o r m a tio n o n T e s t P r o c e s s a n d
S u b m it A p p lic a tio n
C heck
C o m p lia n c e
D a ta b a s e
A d m in P r o c e d u r e s a n d C o n f o r m a n c e G u id e
STEP 2
S u b m it C o n f o r m a n c e N o te b o o k : S O M ,
C o n f o r m a n c e S ta te m e n t, S c e n a r io D a ta
R e tu r n S O M T e s t R e s u lts , I F T e s t D a ta
( F O M a n d T e s t S e q u e n c e ) , a n d T e s t D a te
STEP 3
S u b m it T e s t E n v ir o n D a ta a n d .r id /.f e d file s ;
C o n f ir m T e s t S e q u e n c e a n d D a te
C o n f ir m T e s t I n f o ( D a te , S e q u e n c e , E n v ir o n )
STEP 4
E x e c u te I F T e s t
R e tu r n I F T e s t R e s u lts a n d C S R
C onduct SO M and
C o n fo r m a n c e X - C h e c k
T e s ts ; G e n e r a te T e s t
D a ta
C h eck T est
S c h e d u le
L o g D a ta a n d
A n a ly z e , G e n e r a te
T e s t R e s u lts
Figure 1: HLA Conformance Test Process
In Step 1 of the Conformance process, the
developers of a federate request information on
the test process from the Certification Agent by
completing a HLA test application.
The
Certification Agent checks the official
Compliance Database to determine the federate’s
priority for compliance testing and responds with
administrative procedures and the Conformance
Guide which describes the conformance tests. It
is important to note that the test process is
initiated by the federate developer, not the
Certification Agent, and it is the responsibility of
the federate developer to ensure that the federate
under test (FUT) represents a stable and mature
release of code. Ideally, the test process should
be initiated late in beta testing, so that the actual
tests are performed on the release version of the
code.
In Step 2 of the test process, the federate
developer submits the Conformance Notebook
(CN) to the Certification Agent. The CN
includes the Simulation Object Model (SOM),
the Federate Conformance Statement (CS), and,
as an option, Scenario Data. The CS is a record
of the Interface services a federate can invoke
and respond to (described in section 3.3 below);
Scenario data describes existing scenarios the
federate can execute which invoke the Interface
services asserted in the CS. The Certification
Agent checks the SOM for conformance to the
OMT (“SOM Conformance Test”) and, once
conformant, the Certification Agent checks the
SOM against the CS for consistency
(“Conformance Cross-Check”). Results for the
first two tests are then returned to the federate
developer.
Once the FUT successfully passes the SOM and
Conformance
Cross-Check
tests,
the
Certification Agent uses the SOM, CS and
Scenario data to generate a Test Sequence and
Test FOM to be used by the FUT during the
Interface (IF) test. The Test Sequence and Test
FOM indicate the services the federate must
invoke and the range of data it must use to prove
it implements the Interface services correctly.
The Test FOM and Test Sequence are described
in Section 3.3. In addition to the IF test data, the
Certification Agent sends the federate developer
instructions for downloading the version of the
Run Time Infrastructure (RTI) that is
instrumented for testing. The FUT is required to
use this particular RTI version for IF testing.
The Certification Agent proposes a date and time
for IF testing based on other testing
commitments in a schedule maintained by the
Certification Agent.
In Step 3, the federate developer reviews the
Test FOM and Test Sequence generated by the
Certification Agent, download and install the
RTI to use for testing at the developer’s site, and
submit test environment data to the Certification
Agent. At this point, both the federate developer
and the Certification Agent confirms the test date
and time. The FUT must be able to connect to
the RTI that is instrumented for testing and must
be prepared to conduct the test sequence multiple
times.
In Step 4 of the test process, the IF test is
executed by the federate developer and the
Certification Agent. The IF Test uses two types
of input data to verify the correct implementation
of the Interface services by the federate. First is
Nominal test data, which ensures that the FUT
can invoke and respond to all services for which
it is capable, per its CS. The second type of data
is the Representative SOM (RepSOM), which
ensures that the FUT is capable of invoking and
responding to services using the range of data
contained in its SOM (as suggested by the
Scenario data if supplied). The Certification
Agent logs service data from the test, analyze the
data, generate test results, and return a
Certification Summary Report (CSR) to the
federate developer. The CSR is the official
record of HLA compliance for the specific
version of the federate code tested.
3. Conformance Tests
The conformance processes for the object model
template and interface specification are distinct
but related. For each standard, there are several
distinct tests a FUT must complete. Then to test
the rules, a consistency test is performed
between the OMT and IF. These tests are
described below.
3.1 SOM Conformance Testing
In accordance with Federate compliance
checklist item 1, a federate must have a
Simulation Object Model in the OMT format. In
order to facilitate automated testing, the SOM is
tested using the OMT Data Interchange Format
(DIF) [5]. The DIF is a standard file exchange
format used to store and transfer object models
between OMT development tools. The SOM
conformance test process has three parts:
Parseability: ensure the SOM DIF is readable,
Completeness: ensure that all required data in
tables have been completed,
Consistency: ensure that data are consistent
across tables.
The parseability test is not strictly part of
conformance, but is a requirement of the
automated testing process, which ensures that the
DIF is readable.
The completeness and
consistency checks are conducted in accordance
with the OMT Test Procedures [6].
3.1.1 Parseability
The parseability tests ensure that the Object
Model conforms to the DIF standard [5].
Because of the specificity of the DIF, simply
parsing the DIF file can reveal many potential
problems in an object model. The parseability
test is completed simply through a parse tool,
which is programmed to accept only DIF
conformant data.
3.1.2 Completeness and Consistency
The completeness of an object model ensures
that it is self-sufficient. That is, it goes beyond
what is specifically required in the DIF format
definition to ensure that each element of the
object model is fully defined, and all
dependencies are resolvable. For example, if a
class is defined as having a superclass, the
completeness checks ensure that the definition of
that superclass exists.
Consistency checks
enforce context dependent rules on the object
model, which are not defined in the DIF. The
consistency checks are designed to catch logic
errors in the definition of an object model.
Below are examples of the specific checks for
parseability, completeness and consistency used
in one of the Object Model Development Tools
developed under DMSO sponsorship [7].
Object Model
Check for the existence of at least one class.
Classes
1. The class name is checked in order to make
sure it conforms to the OMT DIF
2.
3.
4.
5.
specification. [REF 5, section 2.2 Basic
BNF Constructs].
The class name is checked to make sure it
does not conflict with another class. [REF 3,
section 3.1.1 Purpose/Rationale].
The number of superclasses is checked to
insure that the class does not inherit from
more than one superclass. [REF 3, section
3.1.1 Purpose/Rationale].
The Publish and Subscribe options are
checked depending on the model type of
FOM or SOM. [REF 3, section 3.12].
Class definitions are checked to insure that
each class has its own textual description.
Attributes
1. The attribute name is checked in order to
make sure it conforms to the C standard for
user declared variables. [REF 5, section 2.2
Basic BNF Constructs].
2. The attribute name is checked to make sure
it does not conflict with another attribute in
its owning class. [REF 3, section 3.3.2]
1. The attribute name is then checked to make
sure it does not conflict with another
attribute that is inherited by its owning class.
Subclass attributes are not checked at this
point because as classes are checked all
superclasses and subclasses are checked and
any duplicate attributes are detected in the
hierarchy. [REF 3, section 3.3.2]
3. Transferable and Acceptable options are
checked. For a SOM the legitimate
combinations are T-Transferable, AAcceptable, TA, and N-Neither. For a FOM,
the legitimate combinations are TA and N.
[REF 3, section 3.3.2 Table Format].
4. Updateable and Reflectable options are
checked. For a SOM the legitimate
combinations are U- Updateable, RReflectable, and UR. For a FOM, the only
legitimate combination is UR. [REF 3,
section 3.3.2 Table Format].
5. Class definitions are checked to insure that
each attribute has its own textual
description.
3.2 Conformance Cross-Check Testing
The conformance cross-check rules, shown in
Table 1, list each OMT Table with its required
interface services, corresponding Interface
Specification [2] reference, and OMT [3]
reference. For the cross-check test, the services
asserted in the FUT’s Conformance Statement
are compared to the services implied by the
FUT’s SOM to determine if the two
specifications are consistent.
3.3 Interface Testing
In accordance with Federate compliance
checklist items 2-6, a federate must be capable of
supporting the services in the Interface
Specification, and as specified in its SOM. The
services a federate can invoke and respond to are
recorded in a Federate Conformance Statement.
The Conformance Statement, in the format
indicated in Table 2, is a list of all the interface
services with their corresponding Interface
Specification [2] reference, OMT [3] reference,
HLA Compliance Checklist [4] reference, and
whether the service is optional or mandatory, per
the Compliance Checklist.
A FUT is required to complete the Conformance
Statement indicating which services the federate
supports when submitting the Conformance
Notebook.
Only the services specifically
asserted in the Conformance Statement are tested
using the procedures outlined in the IF
Specification Test Procedures [8].
Two types of input data are used to verify the
correct implementation of the services asserted
in the Conformance Statement. The Nominal
test data ensures that the FUT can invoke and
respond to all services for which it is capable per
its CS. The Representative SOM (RepSOM)
test data ensures that the FUT is capable of
invoking and responding to services using the
range of data contained in its SOM, as suggested
by Scenario Data (if supplied by the FUT in its
Conformance Notebook).
In order to pass the interface tests, the federate
must execute a Test Sequence (scenario) which
proves it can invoke and respond to the services
as specified in the nominal and RepSOM test
data. The Test Sequence can be based on a
federate’s existing scenario data (if scenario data
submitted) or it can be arbitrarily generated by
the certification agent. These tests are described
more fully below.
OMT Table
Object Class
(P)
(S)
Object Interaction
(I)
(S)
(R)
Attribute/Parameter
(T) Active
(T) Passive
(A) Active
(A) Passive
(U)
(R)
IF Service Name
OMT Reference
IF Reference
Publish Object Class
Register Object
Subscribe Object Class Attributes
Discover Object
sec 3.1.2, page 10
sec 3.1.2, page 10
sec 3.1.2, page 10
sec 3.1.2, page 10
3.1
4.2
3.3
4.4
Publish Interaction Class
Send Interaction
Subscribe Interaction Class
Receive Interaction
Receive Interaction
Update Attribute Values
Publish Object Class
sec 3.2.2, page 19
sec 3.2.2, page 20
sec 3.2.2, page 20
sec 3.2.2, page 20
sec 3.2.2, page 20
sec 3.2.2, page 20
sec 3.2.2, page 20
3.2
4.6
3.4
4.7
4.7
4.3
3.1
Request Attribute Ownership
Divestiture
Attribute Ownership Divestiture
Notification†
Update Attribute Values
Publish Object Class
Request Attribute Ownership
Release†
Attribute Ownership Acquisition
Notification†
Update Attribute Values
Publish Object Class
Request Attribute Ownership
Acquisition
Attribute Ownership Acquisition
Notification†
Update Attribute Values
Request Attribute Ownership
Assumption†
Attribute Ownership Acquisition
Notification†
Update Attribute Values
Update Attribute Values
Publish Object Class
Reflect Attribute Values
Subscribe Object Class Attributes
sec 3.3.2, page 27
5.1
sec 3.3.2, page 27
5.3
sec 3.3.2, page 27
sec 3.3.2, page 27
sec 3.3.2, page 27
4.3
3.1
5.6
sec 3.3.2, page 27
5.4
sec 3.3.2, page 27
sec 3.3.2, page 27
sec 3.3.2, page 28
4.3
3.1
5.5
sec 3.3.2, page 28
5.4
sec 3.3.2, page 28
sec 3.3.2, page 28
4.3
5.2
sec 3.3.2, page 28
5.4
sec 3.3.2, page 28
sec 3.3.2, page 28
sec 3.3.2, page 28
sec 3.3.2, page 28
sec 3.3.2, page 28
4.3
4.3
3.1
4.5
3.3
Table 1: Conformance Cross Check (v0.2, 10 April 1997)
SERVICE
GROUP
Create/Destroy
Join/Resign
Pause/Resume
Save/Restore
SERVICE
Create Federation Execution
Destroy Federation Execution
Join Federation Execution
Resign Federation Execution
Request Pause
Initiate Pause†
Paused Achieved
Request Resume
Initiate Resume†
Resume Achieved
Request Federation Save
Initiate Federate Save†
Federation Save Begun
Federation Save Achieved
Request Restore
Initiate Restore†
Restore Achieved
IF
Ref
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
2.10
2.11
2.12
2.13
2.14
2.15
2.16
2.17
OMT
Ref
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
Check
List
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
Item 6
M/O
M
M
M
M
O
O
O
O
O
O
O
O
O
O
O
O
O
Supported?
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
No
No
No
No
No
No
No
No
No
No
No
No
Table 2: (Partial) Conformance Statement
3.3.1 Nominal Test Data
3.3.2 Representative SOM Test Data
The Nominal test data ensures that the FUT can
invoke and respond to all services for which it is
capable.
As shown in Figure 2, the
Conformance Statement is compared to the
Master Sequence to create the Nominal
Sequence of services that the FUT can support.
The Master Sequence is a dependency tree of all
services defined in the IF Specification [2].
Where true dependencies exist (e.g., Publish
before Update), the Master Sequence shows a
mandatory ordering. If no dependency exists
(e.g., Update and Pause), the Master Sequence
ordering is arbitrary. Therefore, the Nominal
sequence is an ordering of IF services the
federate can invoke. If the FUT can invoke and
respond to all services in the IF specification, its
nominal sequence is equivalent to the Master
sequence.
The RepSOM test data ensures that the FUT is
capable of invoking and responding to services
using the range of data contained in its SOM.
For example, a FUT may be capable of
representing multiple objects, attributes, and
interactions. Instead of attempting an exhaustive
test using all combinations of the objects,
attributes, and interactions, a subset of the SOM
is chosen by the Certification Agent, which
represents a valid range of SOM data. This
“logical subset” forms the FUT’s Test FOM that
is the basis for RepSOM testing.
The RepSOM test data is generated from the
SOM and available Scenario Data submitted
with the Conformance Notebook, as shown in
Figure 2. The logical subset is derived from the
SOM by a set of rules, which select one to three
instances from each object, attribute, and
interaction table.
FUT Conformance Notebook
of FUT’s SOM
Figure 2: NominalSubset
and RepSOM
Tests Used to Create Test Sequence
SOM
RepSOM
Available Scenario
Data
+
Conformance Statement
∩
Master
Sequence
Test FOM
& Test
Sequence
FUT
Nominal
Sequence
RTI
Sequence of services
FUT can support
Sequence of all services in IFSpec
3.3.3 IF Test Sequence
Taking the Nominal Sequence and expanding it
by the RepSOM data generates the final Test
Sequence, as shown in Figure 2. The final Test
Sequence is, therefore, context sensitive as
compared to the nominal sequence. The FUT
should be prepared to execute the Test Sequence
multiple times within the Test Federation, as
specified by the Certification Agent.
The
Certification Agent logs service interactions via
Management Object Model (MOM) interaction
reports, for later analysis and report generation.
4. Test Instructions
This section describes how to prepare for the
Conformance tests.
4.1 Conformance Notebook
The Conformance Notebook is the information a
FUT brings to the test and is comprised of three
items:
the
Simulation
Object
Model,
Conformance Statement, and (optional) Scenario
Data.
The SOM should be submitted in OMT Data
Interchange Format as described in [5].
The CS (shown in part in Table 2) can be
submitted in hardcopy or electronically. If
submitted electronically, the CS must be an ascii
text file which contains the name of every I/F
service, indicating with YES or NO whether the
federate supports each particular service. The
following notation gives an example of the
beginning of a CS file:
createFederationExecution YES
destroyFederationExecution YES
JoinFederationExecution YES
resignFederationExecution YES
requestPause NO
initiatePause NO
…
The last part of the Conformance Notebook is
the Scenario Data. This part of the Conformance
Notebook is optional and is not required for
conformance testing. The purpose of Scenario
Data is to assist the Certification Agent in
developing the RepSOM for the FUT. It is
recognized that developing a scenario which
satisfies the requirements of the Test Sequence
can be both labor intensive and costly for some
federates. It is not the intent of the RepSOM test
to add cost and complexity to conformance
testing. Therefore a federate can submit existing
Scenario Data in its Conformance Notebook to
aid the Certification Agent in developing a Test
FOM test that meets the objectives of the
conformance tests. The format for submitting
the scenario data is shown in Figure 3.
Figure 3: Scenario Data Format, v 0.1, 25 April 1997
(Scenario (meta-data)
(Objects
(Object_name instance_count controlled
(Attrib_name transferred acquired)
(Attrib_name transferred acquired)
.....
)
)
(Interactions
(Interaction_name controlled
(Parameter_name)
(Parameter_name)
....
)
)
)
4.1.1 Interface Test Set-Up
On test day, the FUT shall be prepared to
execute the test sequence using the version of the
RTI instrumented for testing. The federate
developer must show the testing representative
all configuration files relevant to the test.
To conduct a test for the interface specification,
at least two federates are required. The first
federate is the FUT, which contains the
implementation being tested. The second and
subsequent auxiliary federates are present, as
required by the FUT, to demonstrate the test
sequence. For example, the FUT may be
designed to function in a federation with others
like itself, therefore, multiple instances of the
FUT are could be used as auxiliary federates. If,
on the other hand, a FUT is designed to work as
a specialized subsystem within a broader
framework, the FUT would require a sufficiently
sophisticated set of auxiliary federates to provide
all necessary input and output for the test
sequence. In either case, it is the responsibility
of the federate developer to ensure that all
necessary auxiliary federates are present and
operating during each test period.
For the purpose of conformance testing, a
specially configured federate, referred to as the
Test Utility, must also be present and performs
federation management activities during the test
period.
The Test Utility subscribes to
Management Object Model level attributes to
record service interactions between the FUT and
the RTI.
The FUT and Test Utility are
connected via the RTI, as shown in Figure 4.
FUT executes test sequence using
scenario and auxiliary federates,
as required
Test Utility joins federation and
logs service invocations using
MOM
Auxiliary
Federates present
during testing
IF Test
Sequence
Test
Utility
FUT
Service Interaction
Report Log
MOM data is then
analyzed using
Nominal and
RepSOM data
Analysis
RTI
IF Test Report
IF Test Report
is generated
Figure 4: Interface Test Configuration
Along with the environment data submitted in
Step 3, the FUT submits the .rid and .fed
file used to initialize the RTI. The Test Utility is
configured to operate in the execution with the
FUT.
During the start of the test period, the Test
Utility is initialized and joins the execution after
the FUT has created and joined the federation
execution. The Test Utility initializes and begins
logging service invocations via MOM [9] service
interaction reports. Upon completion of a test
sequence, the logging may be stopped at any
time, and the test log is immediately labeled and
stored for post processing.
The final step of the interface testing involves
the post processing of the test logs. The
Interaction Report Post Processor tool is
designed to reduce and analyze Service
Interaction Report Logs to determine whether
each service in the test sequence was
demonstrated and whether the classes,
interactions, and attributes specified in the
RepSOM were demonstrated.
5. Certification Summary Report
Once the conformance tests are completed, the
Certification Agent generates a Certification
Summary Report (CSR), as shown in Figure 5.
The CSR stands as the official record of
conformance testing. Once the Certification
Agent has verified the data as correct, and the
FUT has successfully completed all tests, the
Certification Agent can certify the FUT is HLA
Compliant.
Federate Under Test Information
Name
Version Number
Developer’s Point of Contact for Testing
Test Configuration
RTI Federation Executive (fedex) Host
Application Programmer’s Interface (API) Used
Federate Hardware
Federate Software
.fed and .rid Files
Test Results
Object Model Tests
Pass/Deficiencies with comments
Conformance Cross-Check Tests
Pass/Deficiencies with comments
Interface Tests
Pass/Deficiencies with comments
Supporting Information
SOM (in DIF)
Conformance Statement
Test FOM
Test Sequence
Figure 5: Certification Summary Report
[6]
6. References
[1]
[2]
[3]
[4]
[5]
Knightson,
K.:
OSI
Protocol
Conformance
Testing:
IS
9646
Explained, McGraw-Hill, New York,
1993.
Defense Modeling and Simulation
Office, High Level Architecture
Interface Specification, Version 1.1, 4
February 1997.
Defense Modeling and Simulation
Office, High Level Architecture Object
Model Template, Version 1.1, 12
February 1997.
Defense Modeling and Simulation
Office, High Level Architecture
Compliance Checklist, Version 1.1, 2
March 1997.
Defense Modeling and Simulation
Office, High Level Architecture Object
Model Template Data Interchange
Format, Version 1.2, 10 March 1997.
[7]
[8]
[9]
Defense Modeling and Simulation
Office, High Level Architecture Object
Model Template Test Procedures,
Version 1.0, 5 September 1996.
Aegis Research Corporation, Software
Design Specification for OMDT
Version 1.0 Beta, Module Consistency
Checker, 27 February 1977.
Defense Modeling and Simulation
Office, High Level Architecture
Interface Specification Test Procedures,
Version 1.1, 25 April 1997.
Defense Modeling and Simulation
Office, High Level Architecture
Management Object Model, Version
0.2, 17 October 1996.