Broadcom Listening Test
• In December 2010, Broadcom conducted an ITU-R BS.1116-style
subjective listening test comparing different configurations of the
CELT-only mode of the IETF Opus codec along with MP3 and AAC_LC.
AAC LC
• 17 listeners participated.
• 10 diverse full-band audio tracks with 44.1 kHz sampling used:
–
–
–
–
–
–
–
2 pure speech
2 vocal
2 solo instruments
1 rock-and-roll
1 pop
1 classical orchestra
1 jazz
• The goal was to compare different configurations of the latest CELT at
that time (version 0.9.1, as of November 2010), and compare them with
the two reference codecs MP3 and AAC_LC at 96 kbps.
Copyright © 2011, Broadcom Corporation. All rights reserved.
1
Codec Conditions Tested
• 6 codec conditions were tested, all with constant bit-rate (CBR):
–
–
–
–
–
–
Reference 1: MPEG-1, layer 3 (MP3) codec at 96 kbps
Reference 2: AAC Low Complexity profile (AAC_LC) codec at 96 kbps
96 kbps CELT 0.9.1 with pitch prefilter/postfilter on, 23 ms frame size
96 kbps CELT 0.9.1 with pitch prefilter/postfilter on, 5.8 ms frame size
96 kbps
p CELT 0.9.1 with ppitch pprefilter/postfilter
p
off, 5.8 ms frame size
64 kbps CELT 0.9.1 with pitch prefilter/postfilter on, 5.8 ms frame size
• It was later realized that CELT 0.9.1 was only optimized for 48 kHz and
•
•
•
•
not for 44.1 kHz sampling.
After the test, CELT was optimized for 44.1 kHz in addition to 48 kHz.
After re-processing the audio test files with 44.1 kHz-optimized CELT,
there was noticeable audio quality improvement.
improvement
Later, CELT went through further audio quality enhancements.
As a result, the CELT test scores (on a 5-point scale) presented on the
next slide can be considered lower bounds of the scores of the current
version (0.11) of CELT.
Copyright © 2011, Broadcom Corporation. All rights reserved.
2
CELT Listening Test Result (Audio Quality
Degradation Relative to Uncoded Original)
0.00
-0.11
-0.20
-0.17
-0.40
-0.60
-0.80
-0.60
-0.79
-0.86
-1.00
-1.20
-1.40
-1.58
-1.60
-1.80
-2.00
CELT_96kbps_on CELT_96kbps_on CELT_96kbps_off CELT_64kbps_on
_23ms
_5.8ms
_5.8ms
_5.8ms
MP3_96kbps
AAC_LC_96kbps
95% CI-L
-0.61
-0.06
-0.09
-0.47
-0.68
-1.37
95% CI-U
-0.97
-0.16
-0.25
-0.72
-1.04
-1.80
M
Mean
-0.79
0 79
-0.11
0 11
-0.17
0 17
-0.60
0 60
-0.86
0 86
-1.58
1 58
• “on” or “off” in the codec conditions above indicates whether the pitch
prefilter and postfilter were on or off.
• 95% CI-L and 95% CI-U indicate the lower and upper bounds of the 95%
confidence interval.
Copyright © 2011, Broadcom Corporation. All rights reserved.
3
Conclusion
• 96 kbps CELT 0.9.1 with a frame size of 23 ms was rated significantly
better than 96 kbps MP3.
• 96 kbps CELT 0.9.1 with a frame size of 23 ms was rated roughly
equivalent to 96 kbps AAC_LC; with 44.1 kHz optimization and further
enhancements, CELT 0.11 is expected to be no worse than AAC_LC.
• 96 kbps CELT 0.9.1 with a frame size of 5.8 ms was rated slightly better
than 96 kbps MP3, even though its codec delay is much lower than that
of MP3.
• The pitch prefilter/postfilter method provided statistically significant
audio quality improvement for the 96 kbps CELT 0.9.1 with a frame size
of 5.8 ms.
Copyright © 2011, Broadcom Corporation. All rights reserved.
4
!"#$%
&
'((((
&%
)%*+%*,-./
0(
1
(
23456
07,
, (
0
%
8*.
3
59(5#
,
.
): /
*
;
<=
0;>
-%
!)?/
;
,>3=??@A@@B59B8*.
:?=C
8)%85=@*/A('=?C
->(%%,)%D95E/A56
8*.'C
->(%%,)H?B6=/A('=@6BC
3
F
8*.G
%%,(3,)95E/A6#,*.C
!)@/
.
H( 66?-I
6#-I
F
:$1'(
%
%*+JI
%*,(-.J
%
.IF
%*,(-.J)
.!/
66?-I6#-I
->(%%,
%*,-.?@<
9K<=KBL?=B=
)
%?=/
%(H8%3
)
/
;
<=)9/
%
H8%3
.
.
6
<E<
<55
<<9
<@6
!
<
@
?<E
?
=
%->(%%,
8
H->(%%,
3%
2,
!3
%->(%%,
H->(%%,
8
3%
====<?9
=======
=======
=======
%->(%%, H->(%%,
====<?9
=======
=======
=======
======5
=?@56#=
=======
=======
8
=======
======5
=?@56#=
3%
=======
=======
=======
=======
=======
2)
/
(
HL?======
,
2(!.
%->(%%,
H->(%%,
8
3%
B
!
6
<
@
?
=
?
@
<
6
B
5
9
#
E
?=
??
?@
?<
?6
?B
?5
!
?9
?#
?E
@=
@?
@@
@<
@6
@B
@5
@9
@#
@E
<=
2(!!
),
/
! =? =@ =< =6 =B =5 =9 =# =E ?= ?? ?@ ?< ?6 ?B ?5 ?9 ?# ?E @= @? @@ @< @6 @B @5 @9 @# @E <=
%->(%%,
H->(%%,
8
L
L==B
L
L==B
):
/
L!L==B:
LH
L==B
%
-
!)/
%
)/
!"#$%&'()%"*
• +,$--*./01234#56-*#-7#7*6-$8%$9-&*:"*.;$(,*<=>>*;#*
?%%@A-*
• B%#,*#$;:"-&*;"&*'"#$;:"-&*A:7#-"-$7*
• +-7#7*6$-7-"#-&*%"*C:"&%D7*EF*D:#,*,-;&6,%"-7*
+-7#*>*G*H;$$%DI;"&*.%"%*06--(,*
•
•
•
•
•
J*&:K-$-"#*9;A-*;"&*J*&:K-$-"#*8-9;A-*76-;L-$7*
2-8-$-"(-*MA-7*7;96A-&*;#*NO*L1P*:"*A%D*I;(L@$%'"&*"%:7-*
<*;"(,%$7Q*A%D6;774MA#-$-&*;#*JRS*L1P*;"&*TR=*L1P*
>T*A:7#-"-$7U*"%*6%7#47($--":"@*
J*";$$%DI;"&*(%&-(7U*;AA*'7:"@*<=*97*8$;9-7*
– :VBF*;#*>SR<*LI67U*(%"7#;"#*I:#*$;#-*
– 06--W*HB*;#*>>*LI67U*(%"7#;"#*I:#*$;#-*
– X6'7*HB*;#*>>*LI67U*Y;$:;IA-*I:#*$;#-**
*
XY-$;AA*2-7'A#7*G*H;$$%DI;"&*06--(,*
>==R=*
ZZR[*
Z=R=*
O=RO*
O=R=*
T=R=*
[=R=*
SSR[*
SJR>*
NOR[*
S=R=*
NSRS*
N=R=*
J=R=*
<=R=*
>=R=*
=R=*
X$:@:";A*
VE*JRS*
VE*T*
:VBF*>S*
X6'7*>>*
X6'7*;#*>>*LI67*:7*I-\-$*#,;"*:VBF*;#*>S*LI67*
;"&*06--W*;#*>>*LI67*
06--W*>>*
+-7#*<*G*C:&-I;"&*;"&*]'AAI;"&*.%"%*06--(,*
•
•
•
•
•
J*&:K-$-"#*9;A-*;"&*J*&:K-$-"#*8-9;A-*76-;L-$7*
2-8-$-"(-*MA-7*7;96A-&*;#*NO*L1P*:"*A%D*I;(L@$%'"&*"%:7-*
<*;"(,%$7Q*A%D6;774MA#-$-&*;#*JRS*L1P*;"&*TR=*L1P*
>T*A:7#-"-$7U*"%*6%7#47($--":"@*
N*D:&-I;"&*(%&-(7U*;AA*'7:"@*<=*97*8$;9-7*
–
–
–
–
?RT<<R>*;#*<N*LI67U*(%"7#;"#*I:#*$;#-**
06--W*CB*;#*<JRO*LI67U*(%"7#;"#*I:#*$;#-*
X6'7*CB*;#*>ZROS*LI67U*Y;$:;IA-*I:#*$;#-*
3.24CB*;#*>ZROS*LI67U*(%"7#;"#*I:#*$;#-*
• <*8'AAI;"&*(%&-(7U*I%#,*'7:"@*<=*97*8$;9-7*
– ?RT>Z*;#*J<*LI67U**(%"7#;"#*I:#*$;#-*
– X6'7*]B*;#*J<*LI67U*(%"7#;"#*I:#*$;#-*
*
*
XY-$;AA*2-7'A#7*4*]'AAI;"&*;"&*C:&-I;"&*06--(,**
>==R=*
ZZRJ*
ZTR<*
TTRZ*
O=R=*
[ORS*
T<RZ*
[[R[*
[=R=*
NORZ*
NSRS*
JTRZ*
N=R=*
<=R=*
=R=*
X$:@:";A*
?RT>Z*J<*
X6'7*J<*
06--W*<N*
X6'7*<=*
VE*T*
?R<<<R>*<N* 3.24CB*<=*
VE*JRS*
X6'7*;#*J<*LI67*:7*;A9%7#*#$;"76;$-"#*
X6'7*;#*<=*LI67*:7*I-\-$*#,;"*VE*MA#-$-&*76--(,*;#*T*L1P*
+-7#*J*G]'AAI;"&*0#-$-%*.'7:(*
•
*
•
•
•
•
>=*7#-$-%*9'7:(*MA-7*
– 2%(L^2_B**`B%P*0(;@@7a*
– 0%b*$%(L*`0#--A5*c;"a*
– 2%(L*`d'--"a*
– e;PP*`1;$$5*e;9-7*X$(,-7#$;a*
– FA;77:(;A*`*E'$(-AA*0#$:"@*E:-(-a*
– fA-(#$%":(;*`.;#9%7a*
– E:;"%*`.%%"A:@,#**0%";#;a*
– g%(;A7*`0'P;""-*g-@;a*
– ?A%%(L-"76:-A**
– F;7#;"-#7*
2-8-$-"(-*MA-7*7;96A-&*;#*NO*L1P*;"&*NNR>*L1P*
<*;"(,%$7Q*A%D6;774MA#-$-&*;#*JRS*L1P*;"&*TR=*L1P*
Z*A:7#-"-$7U*"%*6%7#47($--":"@*
[*(%&-(7*
– 33F4VF*`H-$%a*;#*[N*LI67U*<>*97*8$;9-*7:P-U*(%"7#;"#*I:#*$;#-*`I:#*$-7-$Y%:$a**
– ?RT>Z*;#*[N*`<*W*J<a*LI67U*<=*97*8$;9-*7:P-U*(%"7#;"#*I:#*$;#-**
– .EJ*`V;9-a*;#*Z[*LI67U*h>==*97U*(%"7#;"#*I:#*$;#-*
– X6'7*;#*[N*LI67U*<=*97*8$;9-*7:P-U*(%"7#$;:"-&*Y;$:;IA-*$;#-*
– X6'7*;#*O=*LI67U*>=*97*8$;9-*7:P-U*(%"7#$;:"-&*Y;$:;IA-*$;#-*
– X6'7*;#*><O*LI67U*S*97*8$;9-*7:P-U*(%"7#$;:"-&*Y;$:;IA-*$;#-*
XY-$;AA*2-7'A#7*]'AAI;"&*0#-$-%*.'7:(*
>==R=*
ZZRN*
Z=RT*
Z>R>*
Z=RT*
Z>RT*
.EJ*Z[*
X6'7*[N*
X6'7*O=*
ZSRS*
TOR>*
O=R=*
[<R=*
[=R=*
JSRZ*
N=R=*
<=R=*
=R=*
X$:@:";A*
VE*JRS*
VE*T*
33F4VF*[N*
?RT>Z*[N*
X6'7*`;#*[N*L6I7^<=97U*O=*LI67^>=*97U*;"&*><O*LI67^S*97a*
:7**
-i';A*:"*i';A:#5*#%*.EJ*;#*Z[*LI67*
-i';A*:"*i';A:#5*#%*33F4VF*;#*[N*LI67*
I-\-$*#,;"*?RT>Z*;#*[N*LI67*
X6'7*><O*
!
!!
"
!
#
$%&'(
%&'(
$,!-
.
/+/012,3/.#4545675.7/8/
9:5;:
+/
<
)*!'+%(
030470=7=.
>?/?06/007
:@
<
A-/
! )
'
<
B
!!"
#
$
% &
' !("
)*
+,%'
-.
/
/
%!!
/#01231241251 %!!01231 /)*266
/%'266
/%!2%(
