INTERNATIONAL TELECOMMUNICATION UNION
COM 12 – D 94 – E
TELECOMMUNICATION
STANDARDIZATION SECTOR
English only
STUDY PERIOD 2005-2008
Original: English
Question:
14/12
Geneva, 17-21 October 2005
STUDY GROUP 12 – DELAYED CONTRIBUTION 94
Source:
Telchemy Incorporated
Title:
Suggested use of TIA-921 for test vector generation for P.VTQ validation
Abstract
This document, previously presented at an interim meeting of Q.14/12 suggests using a parameterdriven network model for test vector generation as a basis for qualification of P.VTQ
implementations.
Introduction
Over the last two-plus years, TIA Committee TR-30.3, Data Communications Equipment
Evaluation and Network Interfaces, has been working to develop an IP network model for use in
test environments, simulations and for algorithm validation as necessary for P.VTQ. From the
introduction of the draft:
Previous network transmission model standards for evaluating modem performance (e.g. TSB- 37A,
TIA/EIA 3700, TIA/EIA 793 and TIA 876) have been statistical models in which likelihood of occurrence
(LOO) values were assigned to all network elements and impairments. Test results using these
statistical models were expressed in terms of Network Model Coverage (NMC). These NMC results
were unconditional – not dependent on the a priori specification of any network elements or
impairments. This is an example of a Statistical Model. Testing to a comprehensive Statistical Model
suggests how communications devices may perform over an IP network in terms of Network Model
Coverage.
This model has undergone over two years of work to ensure that the model accurately depicts how
IP networks behave. The model’s characteristics have been favourably compared to actual network
traces and performance measured by, and provided to the committee by several network operators.
This model has already been used in commercially available test equipment for testing networkconnected devices.
Not only has TR-30.3 worked on this model, but also its work has been also been presented to TIA
Committee TR-41 (User Premises Telecommunications Requirements) and has been introduced to
Study Group 12 for use by the ITU as a potential ITU-T Recommendation. An updated draft will
be available at this meeting of SG12.
Contact:
Alan Clark
Telchemy Incorporated
USA
Tel:
+1 770 614-6944
Fax:
+1 770 614-3951
Email: alan.clark@telchemy.com
Contact:
Bruce Adams
Telchemy Incorporated
USA
Tel:
+1 770 614-6944
Fax:
+1 770 614-3951
Email: bruce.adams@telchemy.com
Attention: This is not a publication made available to the public, but an internal ITU-T Document intended only for use by the
Member States of ITU, by ITU-T Sector Members and Associates, and their respective staff and collaborators in their ITU related
work. It shall not be made available to, and used by, any other persons or entities without the prior written consent of ITU-T.
-2COM 12 – D 94 – E
Characteristics of the model
This model is based on the notion that time series models of known characteristics of network
impairments accurately model these behaviours. In a typical end-to-end network connection, the
system may comprise a Local LAN segment, a Local Access Link segment, a Core Network
Segment, a Remote Access Link Segment and a Remote LAN segment. Each of these segments has
error characteristics that can be modelled using an enhanced form of a two-state Gilbert-Elliot
model and an underlying filter function that is tailored to the particular segment that is being
modelled. Each of the segments can independently change to and from busy or congested states to
low-traffic states.
Each segment also has probability of loss.
Each segment is a model of each successive network element, and adds incrementally to the delay
and loss based on parameters such as link speed, occupancy and probability parameters.
Lastly, a test implementation the model used to create some of the results tables in the draft,
incorporates a simple jitter-buffer model that, using packet arrival times as input, models packet
discards and out-of-sequence arrivals in the end-point equipment.
Combined, these elements create a realistic model of the environment in which VoIP normally
services run.
Network coverage
The standard itself describes a large set of parameters that when combined produce a set of 1064
test cases. These test cases are then combined with three service profiles meant to model wellmanaged networks as well as best-effort and unmanaged networks - all of which VoIP services will
and do run. In testing, it was found that the model produces traces that when measured using PESQ
MOS-LQ scores as well as E-model based (G.107) scores that are well distributed from about 1.5 to
4.3 for all three service profiles, with the higher scores for larger percentages of the model for
increasingly better managed networks.
Use in P.VTQ
So far, the generation of test vectors for P.VTQ has proven to be one of the more difficult tasks in
developing P.VTQ. Issues such as IPR for voice material and consistent test results demand that we
come up with vectors that are realistic. In addition to stressing the algorithms, they must also
represent “real-world” network conditions.
Earlier it was decided to use PESQ (P.862) as a standard against which to measure P.VTQ
compliance. Using a standardized way to produce test vectors seems a prudent thing to do.
Obviously we need to limit the model parameters to a subset that adequately covers the network.
The model can easily be used to produce frame-erasure vectors or to create IP network packet
traces, for use as input to P.VTQ algorithms.
Use in Mid-point scenarios
Because the model is modular, using appropriate input parameters can eliminate various sections
from impacting the data stream. Therefore it is easy to use for coming up with mid-stream test
vectors by eliminating the impairments from there to the end of the connection.
-3COM 12 – D 94 – E
Parameter limits
This section discusses some of the parameters that are inputs to TIA-921 and suggests limited set
for use with P.VTQ. The following tables from TIA-921 draft-14 indicate the suggested limits for
use. We have removed those impairments that are too long-term to be of significance in P.VTQ
testing as such as route flap and link failure. Tables 1 - 4 indicate the range of typical link speeds
for LANs and Access Links for various locations. The LOO percentages are Likelihood-ofoccurrence for each location based on industry data.
Table 1 - LAN Rates for Home Locations
LAN Rate LOO Represents
Mbit/s
%
4 75
802.11b, 10BaseT hub
20 25
802.11g, 100BaseT hub
Table 2 - LAN Rates for Business Locations
LAN Rate LOO
Mbit/s
Represents
%
4
20 802.11b, 10BaseT
20
20 802.11g, 100BaseT hub
100
60 100 BaseT switched, Gbit Ethernet
Table 3 - Access Rates for Home Locations
Access Rate
LOO
Toward core From core
%
kbit/s
Represents
kbit/s
128
768
40 Cable, ADSL
384
1536
50 Cable, ADSL
384
3000
10 Cable, ADSL
-4COM 12 – D 94 – E
Table 4 - Access Rates for Business Locations
Access Rate
LOO
Toward core From core
%
kbit/s
Represents
kbit/s
384
1536
40 ADSL entry
384
3000
15 ADSL premium
1536
1536
40 T1
43000
43000
5 T3
Table 5 - Impairment Severity Combinations
Severity=>
Impairment
A
B
C
E
D
F
H
G
Units
Profile A LOO
%
50
30
15
5
0
0
0
0
Profile B LOO
%
5
25
30
25
10
5
0
0
Profile C LOO
%
5
5
10
15
20
25
15
5
Source Location (A) Parameters
LAN A
Occupancy
%
1
2
3
5
8
12
16
20
Access A
Occupancy
%
0
1
2
4
8
15
30
50
MTU A
bytes
512
512 1508 1508 1508 1508 1508 1508
Core Network Impairments
Delay
ms
4
8
16
32
64
128
256
512
Jitter
ms
5
10
15
20
30
50
75
100
Packet loss
%
0 0.01
0.02 0.04
0.1
0.2
0.5
1
15
30
50
Destination Location (B) Parameters
Access B
Occupancy
MTU B
LAN B
Occupancy
%
bytes
%
0
512
1
1
2
4
8
512 1508 1508 1508 1508 1508 1508
2
3
5
8
12
16
20
Table 5 lists the various impairment severity limits for TIA-921. A subset of these in combination
with the combinations of accesses would cover most situations. Those columns in bold, A, C, E
and H, could cover most of our test surface. In addition, selective use of parameter sets should
eliminate the need to combine test scores at either end of the quality spectrum.
-5COM 12 – D 94 – E
Test methodology
The group has already decided to do testing against PESQ wherever possible, and has for the most
part agreed to the form of the audio signals. Therefore, we must run some set of samples against
the test scenarios and then through PESQ in order to get the target test scores.
Using this method of generating tests eliminates the need for any definitive set of data as input. As
long as the samples meet the other requirements, then they are fine.
In using TIA-921 as a basis for testing P.VTQ implementations, the following test methodology is
suggested:
1) Using a given subset of parameters for the model, generate various sets of frame-erasure
vectors.
2) Take any set of conforming audio data and create a trace of the impaired signals.
3) Optionally create RTCP-XR data to include in the stream.
4) Run PESQ against the test signals to get the target test set.
5) Run the P.VTQ implementation against the test trace to generate the P.VTQ scores.
6) Do the compliance comparison between the target and generated scores.
Conclusions
The ITU has a well-deserved good reputation of making use of the work of other committees and
standards organizations so as not to only avoid duplicating effort, but to allow the most qualified
experts in an area create standards in that area. TR30.3 is one such organization. By making use of
their expertise and experience, we can ensure that we are using reasonable models and scenarios for
testing of our algorithms.
In addition, users can generate their own set of test vectors with their own set of impairment
combinations that will more accurately match conditions and situations that they see, and are most
interesting to them. It not only allows user evaluation based on a recommended set of situations,
but perhaps more importantly a standardized way to further evaluate and configure P.VTQ
algorithms based on the way that they will be used. We therefore recommend that TIA-921, or it’s
possible ITU-T version be used for test vector generation for P.VTQ and further recommend that
Q14/12 define a set of parameters for TIA-921 for inclusion in P.VTQ that cover our required test
surface.
References
[1]
TIA-TR30.3- Editor, TIA Network Model for Evaluating Multimedia Transmission
Performance Over Internet Protocol, PN-3-0062 (TIA-921) – Draft, Document 050618-pn3-0062-Draft14.doc, June 2005, Arlington Virginia.
[2]
TIA-TR30.3-Alan Clark, Telchemy Inc., IP Network Impairment Mode, PN-3-0062,
Document TR-30.3/03-04-13, April 2003, Arlington Virginia.
[3]
SG12/Q14, Paul Barrett and Simon Broom, Psytechnics Ltd., Proposal for a P.VTQ
framework recommendation, May 2005, Santa Barbra California.
____________