CITEL (PCC.I)/ ITU Forum on Information and
Communication Technology Service:
Quality, Control and Surveillance
(Cartagena de Indias, Colombia, 23-24 September 2013)
QoS assessment for mobile speech services
including user interfaces in vehicles and
hands-free communication
Dr.-Ing. H. W. Gierlich
Head of Telecom Division
Colombia, 23-24 September 2013
Outline
Introduction
The impact of terminal design on
speech quality
Assessment methods and standards
for mobile terminals
Driver distraction and speech
services in cars
Summary
Colombia, 23-24 September 2013
2
Users Expectation (QoE)
Listening
Effort
Sound Quality &
Naturalness
Intelligibility
Speech
Characteristics
Talking
Effort
Individual
Listening
Speech
Quality
Environmental
Conditions
Double Talk
Performance
...
Conversational
Effort
Network
Conditions
Expectation
Transmission of
Background Noise
Colombia, 23-24 September 2013
...
3
Signal Processing in Modern Terminals
Noise
AGC/
cancellatio
companding
n
…
EC
EQ
Mikrofon –
(Array)
Coder
NLP
-
NR
Coder
Modulation
/
Demodulation
H(f )
SE
Decoder
Important:
All
signal
processing
is
Loudspeaker
z.B. GSM/UMTS
non-linear, time variant and
SpeechDecoder
Comfort
targeted“enhancement”
to speech enhamcement
Noise
+
EC
CN
speech recognition/
control
Colombia, 23-24 September 2013
4
Speech Quality
USERS PERCEPTION
…is subjective and generally different between subjects
…is based on experience
…is cultural dependant
TESTING
…in general should be based on users perception
…may be based on simple signals and procedures modelling aspects
of users perception (e.g. loudness)
…may be based on complex signals and advanced procedures
modelling aspects of users perception (e. g. switching and double talk
tests)
…may be based on perceptual models using psychoacoustically
motivated procedures (models of the human hearing)
Colombia, 23-24 September 2013
5
Outline
Introduction
The impact of terminal design on
speech quality
Assessment methods and standards
for mobile terminals
Driver distraction and speech
services in cars
Summary
Colombia, 23-24 September 2013
6
Speech Quality Evaluation
Sending
SpeechQuality
Receiving
Quality of
Echo
Background
PerformanceDouble Talk Noise Transm.
Colombia, 23-24 September 2013
7
Terminal Testing
Bring the world to the lab:
Simulate user
Simulate user behaviour
Simulate user environments
Simulate network and access points
Simulate network conditions
Valdidate correct NTP access
Colombia, 23-24 September 2013
8
The Typical Lab Setup
Background
Noise
Subwoofer
Simulation
Network
Simulation
Testsystem
Colombia, 23-24 September 2013
9
Relevant Standards
GPP TS 51.010 (GSM 11.10)
3GPP TS 26.131 & TS 26.132
3GPP2 C.S0056
TIA 928 (CDMA/TDMA)
CTIA
GSMA HDvoice standard
For more advanced testing:
ETSI TS 103 737 – TS 103740 standard
series
Colombia, 23-24 September 2013
10
Moste Important Test Parameters
Enabling a Good QoS Performance
Basic acoustic tests e.g. Loudness Ratings
(ITU-T P.79), Frequency response tests…
Delay and echo tests
Speech quality tests – with and without
background noise
Conversational test including double talk,
switching…
Important: Test signals are mostly speech
based – ITU-T P.501
Colombia, 23-24 September 2013
11
The Impcat of Delay on Conversational
Dynamics
Best case between fixed & mobile network (no satelite):
~150 ms
Worst case across networks:
>> 500 ms
For superior conversational quality: delay < ~150 ms
(from the users perspective)
Terminal A
Colombia, 23-24 September 2013
Network
0 ms
150 ms
500 ms
Terminal B
12
Conversational/Double Talk
Test Signal
Female
Male
Male
Female
Receive
Send
Short isolated words
Double talk sentences
Word in a speech pause
Word in partially masked
Word fully masked, one
speaker continuosly
talking
Word conicident with the other channel speaker,
Simulating delay and unwanted double talk
Colombia, 23-24 September 2013
13
Summary of Quality Parameters Using
„Quality Pies“ according to ITU-T P.505
D-Value
D-Value
BGNT (far)
5
DT Type
8
6/10
-5
2
Subj. Exp. Assessment
4
3
2
Subj. Exp. Assessment
poor
good
fair
6/-2
1,5
40
4/0
good
40
40
3
50
TCLw max
4
G-MOS Mensa
G-MOS Mensa
N-MOS Mensa
N-MOS CAR
3,4
5
3,4
2
G-MOS CAR
2
3,4
3,5
2
5
1,5
4,3
1,5
S-MOS TRAIN
2
2,3
4,3
N-MOS TRAIN
4,3
S-MOS ROAD G-MOS TRAIN
3,2
2,3
3,4
3,5
1,5
2
2
2,3
S-MOS TRAIN
3
3,2
4,3
1,5
G-MOS ROAD
N-MOS TRAIN
4,3
S-MOS ROAD G-MOS TRAIN
3,2
2,3
4,3
N-MOS TRAIN
4,3
S-MOS ROAD G-MOS TRAIN
2
2,3
3
3
N-MOS ROAD
3,2
S-MOS TRAIN
2
1,5
2
2,3
4,3
5
2
1,5
2
4,3
G-MOS CAR
2
5
2
3
G-MOS ROAD
2
2
S-MOS Mensa
2
3,2
3
3
N-MOS ROAD
3,2
5
1,5
2
2,3
3
3
N-MOS ROAD
3,2
3,5
1,5
2
2,3
3,2
S-MOS TRAIN
2
1,5
G-MOS CAR
2
1,5
1,5
G-MOS ROAD
2
5
2,3
4,3
5
3,5
2
3,4
3,5
3,4
2
2
3
3,2
N-MOS CAR
3,4
5
3,5
S-MOS Mensa
2
2,3
3
5
5
N-MOS Mensa
3,5
5
2
1,5
2
2,3
3,5
2
2
3,2
G-MOS CAR
2
3,4
S-MOS CAR
5
3,4
5
G-MOS Mensa
N-MOS CAR
2
2
S-MOS Mensa
Conv. Quality
5
2
OK
Echo vs. time
S-MOS CAR
3,4
5
3,5
2
5
4
TMOS RCV
TCLw nom
N-MOS Mensa
3,5
5
2
3
TCLw max
OK
5
3,4
3,5
2
G-MOS Mensa
N-MOS CAR
not Ok
50
Conv. Quality
5
3,4
5
40
Echo vs. time
N-MOS Mensa
3,5
RLR nom
2
70
TCLw nom
S-MOS CAR
5
5
2
30
3
4
4/0
1,5
40
50
TMOS RCV
Conv. Quality
S-MOS CAR
5
not Ok
TCLw max
Echo vs. time
Conv. Quality
6/-2
30
2
70
50
OK
TCLw nom
Echo vs. time
G-MOS ROAD
40
4/0
RLR nom
2
TMOS RCV
OK
poor
70
TMOS RCV
S-MOS Mensa
30
70
TCLw nom
fair
8/-4
1,5
40
50
4
70
5
30
2
3
50
TCLw max
good
5
4
3
2
Subj. Exp. Assessment
6/-2
70
not Ok
2c
poor
RLR nom
2
70
not Ok
4/0
30
TMOS SND
2/14
2b
8/-4
1,5
40
50
RLR nom
2
70
fair
-10
2a
5
4
3
2
Subj. Exp. Assessment
6/-2
30
2
30
50
2/14
8/-4
6/10
4/12
10
1
TMOS SND
2c
poor
8/-4
30
8
-5
4/12
2b
5
4
3
0
0
6/10
-5
2c
DT Type
-10
2a
SLR
5
8
10
1
TMOS SND
2/14
5
0
0
4/12
-10
2a
2b
2c
DT Type
6/10
-5
TMOS SND
2/14
SLR
5
8
10
1
-10
2b
BGNT (far)
SLR
0
0
4/12
10
2a
fair
DT Type
D-Value
BGNT (far)
5
0
0
1
good
D-Value
BGNT (far)
SLR
3,2
2,3
3
4,3
3
N-MOS ROAD
N-MOS TRAIN
3,2
4,3
S-MOS ROAD G-MOS TRAIN
Background Noise Performance
Phone 1
Phone 2
Colombia, 23-24 September 2013
Phone 3
Phone 4
14
Outline
Introduction
The impact of terminal design on
speech quality
Assessment methods and standards
for mobile terminals
Driver distraction and speech
services in cars
Summary
Colombia, 23-24 September 2013
15
Driver Distraction
Physical/manual distraction
Manual operation of devices
Visual distraction
Watching information other than road
Auditory distraction
Focusing on auditory events not related to
the driving task
Cognitive distraction
Occupied by non driving related tasks
Colombia, 23-24 September 2013
16
Technology Based Distraction in Cars
Car entertainment systems
Navigation systems
Phones for speech communication
Car information systems
Text messanging systems
„Texting while driving“
:
Colombia, 23-24 September 2013
17
To be Avoided
In general all activities distracting
from the driving task, esp.:
Visual cues which require different focus
than road
Non intuitive manual operation of the car
Loading the human auditory system by
distracting cues
Unnatural and low quality dialogs and
communications
Colombia, 23-24 September 2013
18
Driving Task
The driving task
mostly occupied:
visual system
not involved:
talking
mainly involved:
hands and legs
involved:
auditory system
=> Auditory Channel and Speech “available”
Colombia, 23-24 September 2013
19
Speech Based Services in Cars
o The main speech applications:
• Speech recognition systems
• Speech dialog systems
• Text to speech systems
• Speech enhancement for
communication systems
• Hands-free communication
• Enhanced in-car communication
systems between passengers
Colombia, 23-24 September 2013
20
Communicational Quality –
Drivers Distraction
Superior
Communication
Quality -> Reduced
Drivers Distraction
Perfekt Intelligibility,
High Speech
Naturalness in the
Car
Perfekt Intelligibility,
High Speech
Naturalness at the
Far End
Seamless Interaction
-> Low Delay,
Perfekt Double Talk
Performance
Colombia, 23-24 September 2013
21
Hands-Free in Cars
Why hands-free in
cars?
Reduce phsyical
distraction
(dialing, holding
the phone…)
Reduce visual
distraction
(watching display,
keyboard …)
Colombia, 23-24 September 2013
22
Superior Sound Quality &
Intelligibility in the Car
o Wideband services in mobile networks available soon
->
o Enabling wideband telephony (100 Hz- 8 kHz) in cars
• Fullband
• Narrow band (car)
• Wideband (car)
o Efficient use of the high quality audio systems in
cars:
•
•
•
•
Getting superior sound quality
Increasing speech intelligibility
Increasing naturalness of a conversation
Reduce drivers distraction due to poor speech sound quality
Colombia, 23-24 September 2013
23
Radio Link
Bluetooth Link
Car
audio
system
Hands-free
signal
processing
Colombia, 23-24 September 2013
Phone
adapter &
antenna
amplifier
24
Car Hands-Free Testing
Mobiles Sound Playback
System (can be installed in any
Test System
analysis
vehicle)
analysis
Generator
and Analyzer
ACQUA
Background
Noise
HFT
AES / EBU
(Digital)
USB
(Digital)
USB
Frontend
Programmable
Equalizer
PEQ IV
AES / EBU
(Digital)
MFE VI
Programmable
Equalizer
PEQ IV
AES / EBU
(Digital)
Programmable
Equalizer
(Analog)
PEQ IV
Subwoofer
Radio Tester
Colombia, 23-24 September 2013
25
ITU-T Standards Contributing to
Reduced Drivers Distraction
ITU-T standards for Hands-Free Communication:
• ITU-T P.1100 for narrowband hands-free
• ITU-T P.1110 for wideband hands-free
ITU-T standards work in FG CarCOM:
• New work on subsystem requirements for Hands-Free
Systems in Cars
ITU-T standards for Dialog Systems:
• P.851: Subj. evaluation of dialog systems
• Suppl. 24 to P. Rec.: Parameters describing
the interaction with spoken dialog systems
Colombia, 23-24 September 2013
26
The Main Section in
ITU-T P.1100 & P.1110
Definition of the test arrangement in a car
Description of a digital interface concept for
development and debugging
Microphone test specification for separate
microphones in a car
Measurement parameters and requirements for
hands-free terminals
Bluetooth test interface for validation of
telephone performance
Subjective test strategy for car to car
communication
Colombia, 23-24 September 2013
27
Conclusions and Recommendations
Mobile terminal testing is essential to provide
sufficient or even superior QoE to the user – many
operator have specific terminal standards for their
suppliers
A variety of parameters contribute to the overall
(conversational) speech quality
Network conditions have to be considered in the
testing
Low driver distraction is essential to offer mobile
services
Speech based operation is the key to the success of
mobile services in cars
Both, terminals and networks have to be
thoroughly engineered and adapted to each other
Colombia, 23-24 September 2013
28
Tandeming of Signal Processing in
Terminals & Networks
Mobile
Phone
Mobile
Network
Base
Station
gain
gain
EC
EC
Netw. 1
ES
ES
NR
Colombia, 23-24 September 2013
NR
Netw. 2
29