International Telecommunication Union
ITU-T
Speech Quality Testing for
VoIP Terminals and
Gateways:
Input from ETSI “Plugtest”
Speech Quality Test Events
H. W. Gierlich
HEAD acoustics GmbH
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
Overview
ITU-T
o Introduction - 1st, 2nd and 3rd
SQTE
o Discussion of Results
•Gateways
•IP phones
o Summary & Outlook
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
2
Overview
ITU-T
o Introduction - 1st, 2nd and 3rd
SQTE
o Discussion of Results
•Gateways
•IP phones
o Summary & Outlook
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
3
Introduction
ITU-T
The ETSI Speech Quality Test Events
• 1st SQTE October 2000
organized and hosted by the ETSI “Bake-off Service”
6 participating companies
test labs HEAD acoustics, Deutsche Telekom
• 2nd SQTE April 2002
organized and hosted by the ETSI “Plugtests Service”
5 participating companies
test labs HEAD acoustics, Deutsche Telekom
• 3rd SQTE June and September 2004, “twin event” EU, US
organized by the ETSI “Plugtests Service” and HEAD acoustics, hosted by
HEAD acoustics
11 participating companies, test lab HEAD acoustics
• 4th SQTE June 2006,
2006 EU – just ongoing
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
4
The Goal
ITU-T
Evaluate all conversational aspects
•
•
•
•
•
Listening speech quality (“status quo”)
Detailed parameter tests
Echo measurements, EC implementation
Double talk performance tests
Quality of background noise transmission
In order to
•
•
•
•
Compare different implementations
Document the state of the arte of technology
Give input to standardization
Help companies to optimize their implementations
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
5
Test Setup Gateways
ITU-T
Packet
MONITOR
Gateway
NISTNet
IP
Gateway
IP
IP
Packet Loss
Delay
E1
E1
PBX
ISDN DSS1
ISDN DSS1
HEAD acoustics
Test System ACQUA
Input
Output
Condition
Packet Loss (Equal)
Additional Delay1
Delay Variation
Condition
Packet Loss (Equal)
Additional Delay1
Delay Variation
1a
0
0
No
1b
0
0
No
2a
1%
0
No
2b
5%
0
No
3a
2%
0
No
3b
0
50 ms
20 ms (2)
4a
3%
0
No
4b
5%
50 ms
20 ms(2)
5a
5%
0
No
6a
1%
50 ms
20 ms (2)
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
6
Test Setup IP-Phones
ITU-T
Packet
MONITOR
Gateway
NISTNet
IP
IP
Packet Loss
Delay
E1
IP
PBX
ISDN DSS1
IP Terminal
(Phone or PC)
Input
HEAD acoustics
Test System ACQUA
Output
Condition
Packet Loss (Equal)
Additional Delay1)
Delay Variation
Condition
Packet Loss (Equal)
Additional Delay1)
Delay Variation
1c
0
100 ms
No
1d
0
100 ms
No
2c
0
100 ms
20 ms(2)
2d
3%
100 ms
No
3c
1%
100 ms
No
3d
0
100 ms
20 ms(2)
4c
1%
100 ms
20 ms(2)
4d
3%
100 ms
20 ms(2)
5c
3%
100 ms
No
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
7
Overview
ITU-T
o Introduction - 1st, 2nd and 3rd
SQTE
o Discussion of Results
•Gateways
•IP phones
o Summary & Outlook
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
8
Listening Speech Quality
PESQ P.862 (MOS(MOS-LQO)
ITU-T
PESQ
max. MOSMOS-LQO
delay
5
600
average
500
min
4
3
300
average
2
max. delay
delay / ms
MOS-LQO
400
200
min
100
1
0
0%, 0ms,
VAD on
0%, 0ms
1%, 0ms
2%, 0ms
3%, 0ms
5%, 0ms
1%, 20ms,
1%, 20ms,
r=0.9 (fix
r=0.5 (fix
delay 50ms) delay 50ms)
network conditions
G.711
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
9
Listening Speech Quality
TOSQA2001 (TMOS)
ITU-T
TOSQA
delay
600
5
500
4
3
300
delay / ms
TMOS
400
200
2
100
1
0
0%, 0ms, VAD
on
0%, 0ms
1%, 0ms
2%, 0ms
3%, 0ms
5%, 0ms
1%, 20ms,
r=0.9 (fix
delay 50ms)
1%, 20ms,
r=0.5 (fix
delay 50ms)
network conditions
G.711
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
10
Listening Speech Quality
(MOS(MOS-LQO vs. TMOS)
ITU-T
PESQ
TOSQA
5
TMOS, MOS-LQO
4
3
For further SQTE results (G.729, G.723) see:
Anonymized Report – “Gateways”
2
SQTE Results IP Phones:
Anonymized Report – “IP Phones”
1
0%, 0ms,
VAD on
0%, 0ms
1%, 0ms
2%, 0ms
3%, 0ms
5%, 0ms
1%, 20ms,
1%, 20ms,
r=0.9 (fix
r=0.5 (fix
delay 50ms) delay 50ms)
network conditions
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
11
Packet Loss Concealment
ITU-T
Highly influencing MOS score - optimization
criteria
Test signal (5s)
• ITU-T Recommendation P.501
• periodical repetition of a voiced sound
• pitch frequency approx. 330 Hz
Relative Approach
• hearing model based forward estimation
• comparison to actual signal
• displays “estimation error“
Cross correlation
vs. Time
• analysis of signal phase
• periodical pattern (approx. 3 ms)
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
12
Packet Loss Concealment
MOS-LQO
5% packet loss
ITU-T
0.4
0.3
0.2
0.2
U
V
-0.2
3.7
U
V
3.6
-0.2
-0.3
-0.4
0.5
1
1.5
2
t/s
3
3.5
4
4.5
5
0.5
1
1.5
2
t/s
3
3.5
4
4.5
5
4k
4k
2k
2k
f
Hz
f
Hz
300
0.5
1
2.5
0
1.5
2
5
t/s
c p/cP a
3
3.5
10
4
4.5
12.5
300
5
0.5
15
5m
1
2.5
0
1.5
2
5
t/s
c p/cP a
3
3.5
10
4
4.5
12.5
5
15
5m
3m
3m
t
s
t
s
-3m
-3m
0.5
-100
1
-75
1.5
-50
2
-25
t/s
3
%
3.5
25
4
50
4.5
75
0.5
5
100
-100
1
-75
1.5
-50
2
-25
t/s
3
%
3.5
25
4
50
4.5
75
5
100
0.5
0.3
0.3
0.2
3.3
U
V
3.1
U
V
-0.2
-0.3
-0.3
-0.5
0.5
1
1.5
2
t/s
3
3.5
4
4.5
0.5
5
1
1.5
2
t/s
3
3.5
4
4.5
5
4k
4k
2k
2k
f
Hz
f
Hz
300
300
0.5
1
2.5
0
1.5
2
5
t/s
c p/cP a
3
3.5
10
4
4.5
12.5
0.5
5
1
2.5
15 0
5m
1.5
2
5
t/s
cp/cPa
3
3.5
10
4
4.5
12.5
5
15
5m
3m
3m
t
s
t
s
-3m
-3m
0.5
-100
1
-75
1.5
-50
2
-25
t/s
3
%
3.5
25
4
50
4.5
75
0.5
5
100 -100
1
-75
1.5
-50
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
2
-25
t/s
3
%
3.5
25
4
50
4.5
5
75
100
13
VAD and Comfort Noise
ITU-T
Gateway
ISDN
ISDN
Simulator
Simulator
PBX
PBX
NISTNet
IP
Gateway
IP
PBX
PBX
Packet Loss
Delay
ISDN
ISDN
Simulator
Simulator
Voice Activity Detection
silence suppression
silence packets
comfort noise generation
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
14
VAD und Comfort Noise
ITU-T
activation thresholds, “hysteresis“
voiced sound,
decreasing and increasing
level vs. time
Level vs. time Manual(5,0 ms)
L/dB[V]
-30
measured signal level
test signal level
Level vs. time Manual(5,0 ms)
L/dB[V]
-30
4.7
4.8
4.9
t/s
5.1
5.2
5.3
4.8
4.9
t/s
-35
-35
-40
-40
-45
-45
-50
-50
-55
-55
-60
-60
-65
-65
Level vs. time Manual(5,0 ms)
5.4 5.5
5.1
5.2
5.3
5.4 5.5
-70
4.5 4.6
L/dB[V]
-30
4.7
4.8
4.9
t/s
5.1
5.2
5.3
Level vs. time Manual(5.0 ms)
5.4 5.5
L/dB[V]
-30
-35
-35
-35
-40
-40
-40
-45
-45
-45
-50
-50
-50
-55
-55
-55
-60
-60
-60
-65
-65
-65
-70
4.7
L/dB[V]
-30
-70
4.5 4.6
4.5 4.6
Level vs. time Manual(5,0 ms)
-70
4.5 4.6
4.7
4.8
4.9
t/s
5.1
5.2
5.3
5.4 5.5
-70
4.5 4.6
4.7
4.8
4.9
t/s
5.1
5.2
5.3
5.4 5.5
adaptive comfort noise
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
15
VAD und Comfort Noise II
ITU-T
comfort noise level slightly too high
Level vs. time Manual(5,0 ms)
Realitsic noise scenario
(students cafe)
L/dB[V]
0
noise contrast
Level vs. time Manual(5,0 ms)
6
7
t/s
6
7
t/s
8
9
L/dB[V] Level vs. time Manual(5,0 ms)
0
8
L/dB[V]
0
-10
-10
-20
-20
-30
-30
-40
-40
-50
-50
-60
-60
-70
-70
-80
5
5
Level vs. time Manual(5,0 ms)
9
10
-80
5
10
L/dB[V]
0
6
7
t/s
8
9
Level vs. time Manual(5.0 ms)
10
L/dB[V]
0
-10
-10
-10
-20
-20
-20
-30
-30
-30
-40
-40
-40
-50
-50
-50
-60
-60
-60
-70
-70
-70
-80
-80
5
6
7
t/s
8
9
10
-80
5
6
7
t/s
8
9
10
quickly adapting comfort noise
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
16
Setup for Echo Canceller Testing
ITU-T
Receive direction
Gateway
ISDN
ISDN
Simulator
Simulator
NISTNet
IP
PBX
PBX
Gateway
IP
PBX
PBX
Packet Loss
Delay
ISDN
ISDN
Simulator
Simulator
Send direction
near end
far end
EC under test
EC off
(Echo path
realized in
MFE VI)
In
Out
RCV SND
In
Out
SND RCV
MFE VI
USB
Test system
„infinite“
ERL 40 dB and
ERL 6 dB
ACQUA
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
17
Double Talk Performance
ITU-T
Gateway
ISDN
ISDN
Simulator
Simulator
PBX
PBX
NISTNet
IP
Gateway
IP
PBX
PBX
Packet Loss
Delay
EC under test
EC off
ISDN
ISDN
Simulator
Simulator
In
Out
RCV SND
In
Out
SND RCV
MFE VI
USB
Testsystem
„infinite“
ERL 40 dB and
ERL 6 dB
ACQUA
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
18
Double Talk Performance
ITU-T
DT_VAR_S.DAT
6 dB ERL
p/Pa
1.25
1
0.75
0.5
0.25
0
-0.25
U/V
0.15
U/V
0.15
0.1
0.1
-0.5
-0.75
-1
echo components
-1.25
0
5
10
t/s
20
25
30
50m
50m
0
0
-50m
-50m
-0.1
-0.1
-0.15
14
14.5
15
15.5
t/s
16.5
17
17.5
-0.15
14
U/V
0.15
14.5
15
15.5
t/s
16.5
17
17.5
U/V
0.15
clipping
0.1
0.1
50m
50m
0
0
-50m
-50m
-0.1
-0.1
-0.15
14
14.5
15
15.5
t/s
16.5
17
17.5
-0.15
14
14.5
15
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
15.5
t/s
16.5
17
17.5
19
Summary Results: „Gateway Pie“
ITU-T
listening speech quality
conversational aspects
like echo behaviour,
double talk performance
and background noise
transmission
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
20
Summary Results: „Gateway Pie“ (ITU-T P.505)
ITU-T
Each “slide“ represents
one parameter
listening speech quality
conversational aspects
like echo behaviour,
double talk performance
and background noise
transmission
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
21
Summary Results: „Gateway Pie“(ITU-T P.505)
ITU-T
Assumption: Independency of
parameters,
no interaction
aspects
listening speech quality
conversational aspects
like echo behaviour,
double talk performance
and background noise
transmission
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
22
Summary Results: „Gateway Pie“(ITU-T P.505)
ITU-T
Minimum requirement
(red circle) represents
“average performance“
of 3rd SQTE
listening speech quality
…or
recommended
numbers
conversational aspects
like echo behaviour,
double talk performance
and background noise
transmission
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
23
Example Results
(4 of 10)
ITU-T
G.711 PLC
“balanced“
implementation
EC inserts high BGN
modulation (NLP/CN)
jitter buffer
performance G.711,
significant
disturbances
BGN modulation
caused by VAD/CN
violation of
requirement only
caused by CN level
(echo attenuation high enough)
enough)
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
24
Overview
ITU-T
o Introduction - 1st, 2nd and 3rd
SQTE
o Discussion of Results
•Gateways
•IP phones
o Summary & Outlook
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
25
Testing IP Phones @ 3rd SQTE
ITU-T
Tests in handset and hands-free mode
For handset
13 N
2N
• Frequency response,
…
• TMOS using
TOSQA2001
• Echo measurements
• Double talk
performance
• Quality of
background noise
transmission
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
26
Testing IP Phones @ 3rd SQTE
ITU-T
Receiving
frequency
responses
and TMOS
L/dB[Pa/V]
30
L/dB[Pa/V]
20
20
10
10
0
0
-10
3.1 TMOS 2N
3.6 TMOS 8N
3.7 TMOS 13N
240 300
400 500 600
f/Hz
1200 1600
2400
600
f/Hz 1200 1600
2400
3.2 TMOS 2N
3.7 TMOS 8N
3.9 TMOS 13N
-20
-30
240 300 400
4000
600
f/Hz 1200 1600
2400
-20
-30
-40
4000
L/dB[Pa/V]
30
L/ dB[Pa/V]
30
20
20
10
10
0
0
2.9 TMOS 2N
3.5 TMOS 8N
3.8 TMOS 13N
240 300 400
-10
4000
-10
-10
3.1 TMOS 2N
3.8 TMOS 8N
4.0 TMOS 13N
-20
-30
240 300
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
400 500 600
f/Hz 1200 1600
2400
-20
-30
4000
27
Listening Speech Quality
ITU-T
TOSQA
TOSQA2001 (TMOS)
delay
5
800
700
600
4
3
400
delay / ms
TMOS
500
300
2
200
100
1
0
sending
2N
8N
13N
0%, 0ms 0%, 20ms, 1%, 0ms 1%, 20ms, 3%, 0ms 1%, 20ms,
r=0.9
r=0.9
r=0.5
network conditions
G.711, sending & receiving direction
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
28
Double Talk Test Hands-Free Terminals
ITU-T
Level vs . t ime Manual(5,0 ms)
Ch. 1 - Ch.2
L/ dB
10
Double talk
test:
Attenuation
range
in sending
direction
(P.340)
Level vs. time Manual(5,0 ms)
Ch.1 - Ch.2
L/dB
10
0
0
-10
-10
-20
-20
-30
-30
-40
-40
-50
-50
-60
0. 25 0. 5
0. 75
1
1. 25 t/ s
Level vs. time Manual(5,0 ms)
1. 75
2
2. 25
2. 5
Ch.1 - Ch.2
-60
0.5
L/dB
10
0.75
1
1.25
t/s 1.75
Level vs. time Manual(5,0 ms)
1
1.25
t/s
1.75
2
2.25
2.5
2.5
2.75
Ch.1 - Ch.2
L/dB
10
0
-10
-10
-20
-20
-30
-30
-40
-40
Reference
-60
0.75
2.25
0
-50
0.25 0.5
2
0.25 0.5
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
0.75
1
1.25 t/s
1.75
2
2.25
-50
-60
2.5
29
Summary Results: “IP-Phone Pies“(ITU-T P.505)
ITU-T
conversational aspects
hands-free
- echo, double talk,
BGN transmission -
handset
listening speech quality
conversational aspects
handset
- echo, double talk,
BGN transmission ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
30
Summary Results: “IP-Phone Pies“
ITU-T
BGN modulation
BGN modulation
ITU-T P.340
“type 3“ implem.
ITU-T P.340
“type 3“ implem.
“balanced“ handset
Implementation…
“balanced“
handset
implementation…
BGN modulation
ITU-T P.340
“type 3“ implem.,
Violation caused
by noise level
(echo attenuation
high enough)
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
G.729 and
hands-free
not tested
31
Some quotes from the 3rd SQTE event:
ITU-T
o
Michael Metzger, Executive Director of Marketing, Mindspeed Technologies:
• "... bringing together the world leaders in VoIP technology in a co-operative
and neutral environment, this prestigious event continues to improve the
quality of VoIP...„
o
Peter Faxel, member of the executive board, AVM;
• "... The ETSI event has helped us towards achieving our goals, and given us
valuable information about the market trend and technological
developments."
o
Daniel Hartnett, Infineon Technologies:
• Plugtests(tm) service of ETSI is an invaluable exercise for a company that
sees voice quality as a key element for the success of our telephone
terminal product families. Infineon will continue to "The VoIP Speech
Quality Test Event (SQTE) organised by the work closely with HEAD
acoustics and ETSI to benchmark their products for voice quality
excellence"
Overall quality of the event: Score 4.6 out of 5
ITU-T Workshop on “End-to-End QoE/QoS“
Geneva, 14-16 June 2006
32