Research on Universal
Broadcasting at NHK STRL
Masahiro Shibata
NHK Science & Technology Research Labs.
All Rights Reserved, Copyright © NHK
Background
2
All Rights Reserved, Copyright © NHK
Terminology
Universal Broadcasting
Broadcasting which can deliver information to
everyone including visually or auditorily impaired
people, elder people, and foreigners.
Accessible Technology
Technology to realize universal broadcasting or
human-friendly information presentation.


3
All Rights Reserved, Copyright © NHK
Objectives of Universal Broadcasting
1.
Everyone-equally
2.
To deliver information “extensively in Japan”
To deliver information to “everyone equally”
regardless of impairment/age or language
difference
Three dimensional
“extensiveness”
= Universal Broadcasting
Two dimensional
“extensiveness”
4
All Rights Reserved, Copyright © NHK
History of closed caption (CC)
and audio description (AD) services
Analog broadcasting

Oct. 1983
Character multiplex broadcast (test service)

Oct. 1984
Closed caption (CC) started. (morning drama “Oshin”)

1982
Audio multiplex broadcast

Apr. 1991
Audio description (AD) started.
(morning drama “Rinrin-to”)
Digital broadcasting
ISDB-T standard
Any digital receiver has decoders for CC and AD
⇒ Hardware issues solved

5
All Rights Reserved, Copyright © NHK
Accessible-service guideline
Service guideline set by Japanese government

Guideline for CC
（set in 1997）
Guideline for CC and AD
(set in 2007)
Target year
2007
2017
Scope
Recorded
programs
All programs
from 7AM to 12PM
CC
100% Completed
AD
－
Status in 2010
100%
10% (15% for NHK 2 (Ed.))
NHK 1
NHK 2 (Ed.)
CC
62.2%
52.5%
AD
7.6%
11.2%
（percentage per 7AM to 12PM)
6
All Rights Reserved, Copyright © NHK
Research on accessible technology
Accessible technology for the impaired
AD script
editor
CC by speech
recognition
CG
sign language
Multi-modal
display
Accessible technology for the elderly, foreigners
夏に電力が足りなくなるかもしれ
ない問題で、人々の節電への関
心が高くなったため、扇風機が良
く売れています。
質問：「夏に．．．高くなったため」
の部分はいりますか？
答え：後に同じことが書いてある
ので省略しましょう
現在の難しさ 2500
Speech processing for the elderly
“Easy Japanese ” for foreigners
7
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the impaired
[Closed Caption (CC) Service]
8
All Rights Reserved, Copyright © NHK
Relayed keyboards method
Stenographic keyboard method
http://www.speed-wp.co.jp/laboratory/
Music programs etc.
News show, general news etc.
Methods of producing live CC in NHK
Direct speech recognition
Short news, American baseball, etc.
Re-spoken speech recognition
Sports, information programs etc.
Speech recognition for CC
Recognition of
Spontaneous speech
Non-live CC
Hybrid-type
Drama, Documentary, etc
Current status
of NHK 1
With CC
R
&
D
Without CC
Talk shows, etc.
Keyboard
Speech rec.
Part of general news
Music show etc.
Short news
Re-spoken/Direct
Speech rec.
Sumo, baseball etc
10
All Rights Reserved, Copyright © NHK
Hybrid speech recognition for CC
Direct
Speech of announcer
or reporter
Speech rec.
CC on-air
Error correct
text
voice
Interview
Re-speak
1-2 operators
Male-female recognition, Auto. dictionary update

Recognize speech of announcer and reporter directly, and respoken speech for interview



Recognition accuracy of over 95%
Higher recognition rate and lower operation cost
R&D issue: Spontaneous speech in news show programs


Need to overcome unclear or rapid speech and phrasing in speech
Recognition accuracy of less than 90% for spontaneous conversations
11
All Rights Reserved, Copyright © NHK
Latest CC system by speech recognition
for 4 o’clock News
Output
Mar. 14, 2012〜
Speech recognition
Operators for error correction
display
Research on accessible
technology for the impaired
[Audio Description (AD)
Service]
13
All Rights Reserved, Copyright © NHK
AD production
Script writer
Finished
Time-consuming!
program
Writing
AD script
Script writer
Program director
Narrator
Program director
Narrator
Engineer
Recording
AD voice
(in studio)
Mixing down
to sound track
On Air

Current AD should be considered as a kind of “special radio drama” and its
production is labor intensive.

Script writers and narrators have to be highly trained.
14
All Rights Reserved, Copyright © NHK
AD script editor
Automatically
estimated syllable
counts
Supplemental
direction for narrator
Input script
Script exceeding
the estimation is
indicated by red
Automatically
detected periods of
voice and silence
Detected periods
with stage
direction
Automatically
generated
script draft
Wave form of
program sound
Current script
window
Evaluation of the system
Time to write script(min)
Average of three trials by MS Office Operators
180
160
140
120
100
80
60
40
20
0
(Duration of the
program was 15 min)
165
Work time of
Experienced
AD Writers
101
77
Manual input
手作業
（1人の結果）
Without
電子台本なし
data of
original script
With
電子台本アリ
data of original
script
Using ADシステム利用
script editor system
16
All Rights Reserved, Copyright © NHK
Evaluation of the system
Opinion of Novice Writers

•
•
•
The system is useful.
Indicating silent portion is convenient.
Estimation of syllable count is a good help.
Opinion of Experienced Writers

•
•

The system does not help much.
AD should be inserted to right positions regardless of the presence of
main program voices. This should be done by writers’ skill.
Conclusion
The system can extend the variation of AD programs without
intensively training new writers while AD can be supplied only
for very popular programs with the current production style.
17
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the impaired
[Sign Language (SL) Service]
18
All Rights Reserved, Copyright © NHK
SL service

For deaf people by nature or those who lost hearing in early
childhood, sign language is their first language.



Sign language service (rather than Closed Caption (CC) ) is requested.
Grammar of Japanese Sign Language (JSL) is different from that of the
aural Japanese.
Difficulty to keep sign language interpreters for regular service

Especially for emergency news
CG sign language service by automatic translation
TV
program
Translate CC to SL
SL CG
broadcasting
Station
Home
internet
19
All Rights Reserved, Copyright © NHK
Overview of automatic translation system
Input Japanese sentence: 私の名前は金子です。 (“My name is Kaneko.”)
Japanese - JSL
dictionary
Convert Japanese sentence into sequence of
symbols representing JSL word
[ 私, 名前, 金子, 言う, N ]
( I, name, Kaneko, say, nodding )
Generate signing motion for each symbol
(including hand and finger motions, facial
expressions, and so on).
Japanese - JSL
corpus
Output computer-animated JSL
Generate seamless motion transitions between
each symbol by using motion interpolation.
20
All Rights Reserved, Copyright © NHK
Japanese – JSL CG dictionary system
Input Japanese word
Confidence measure
Found Japanese entry
Entries of synonyms
Thumbnail image of
CG animation
Click entry to display
CG animation of JSL word
21
All Rights Reserved, Copyright © NHK
Evaluation of
Japanese-JSL CG dictionary system
Readability
Naturalness
■Easy to read
■Some part is unreadable but omittable
■Some part is unreadable but not
bothering
■Bothering
■Unreadable
■Natural
■Some part is unnatural but omittable
■Some part is unnatural but not
bothering
■Strange
■Very unnatural
Based on questionnaire survey at Tokyo Federation of Deaf meeting on May 5-6, 2011
22
All Rights Reserved, Copyright © NHK
Japanese-JSL CG translation
prototype for weather information
Screen shot of Japanese-JSL example-based translation prototype.
23
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the impaired
[Multi-modal Display]
24
All Rights Reserved, Copyright © NHK
Digital TV receiver for visually impaired
•Reading-out menu and other
visual information
•Magnifying characters on screen
•(Braille expression of texts)
DTV receiver
for visually
impaired people
Example of data broadcasting screen

Visual information and graphical user interface (GUI) are everywhere!


PC, TV, smart-phone, remote controller, ticketing machine, ATM, etc
But they can be big obstacle for visually impaired people
25
All Rights Reserved, Copyright © NHK
Interactive tactile display

Two dimensional tactile display + touch position sensor

As a GUI device for visually impaired people
Hierarchical expression
Menu expression by Braille
Also applicable for guide terminals in public places such as
railway stations, ATMs in the bank.
26
All Rights Reserved, Copyright © NHK
Information-barrier-free for visually
impaired people
Universality
for
impaired/non-impaired
3D objects
Tactile and force feedback
2D graphics (+ GUI)
texts
Interactive tactile display
Voice and Braille
27
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the elderly
28
All Rights Reserved, Copyright © NHK
Aging of the society

Aging rate and its speed of Japan is highest in the world.

Compensation to auditory and visual ability is social requirement.
40.0
Aging rate (≧65)
35.0
30.0
日本
Japan
イタリア
Italy
ドイツ
Germany
アメリカ合衆国
U.S.A.
韓国
Korea
シンガポール
Singapore
25.0
20.0
15.0
10.0
5.0
Redrawn from “Annual
Report on the Aging
Society: 2009”
(by Cabinet Office Japan)
0.0
29
All Rights Reserved, Copyright © NHK
Problems in information perception of
the elderly

Visual information




Degraded accommodation function (presbyopia)
Low vision
Lower sensitivity of bluish color
Auditory information



Less sensitive to higher tone
Difficult to understand speech with high speed
Difficult to understand speech under background noise
30
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the elderly
[Speech Rate Conversion]
31
All Rights Reserved, Copyright © NHK
Speech rate conversion for the elderly


Started research by responding a claim from an elderly
person; “Recently, speech on broadcasting is too fast for
me to understand”
Slow down speech without degrading sound quality
TV and radio set with “Slow button”
Evaluation experiment
32
All Rights Reserved, Copyright © NHK
Speech rate conversion without
changing voice pitch
Temporal duration of waveform is changed to keep fundamental period
① ② ④ ⑤ ⑥ ⑦ ⑨ ⑩
Faster
×
×
time
① ② ③ ④ ⑤ ⑥ ⑦ ⑧ ⑨ ⑩
Original
time
stop
① ② ③ ③ ④ ⑤ ⑥ ⑦ ⑧ ⑧ ⑨ ⑩
Slower
time
①
②
③
④
⑤
⑥
⑦
⑧
⑨
⑩
Conventional
time
Fundamental period is extended and pitch becomes lower
33
All Rights Reserved, Copyright © NHK
Speech rate conversion without
changing length
Original
Converted
Start is not coincided but… Again it coincides
Start is coincided at
blue line positions
34
All Rights Reserved, Copyright © NHK
Speech rate conversion for visually
impaired people

Understandable high speed speech

Visually impaired people usually understand PC screen
by using screen reader ⇒ they want to hear like quick
reading of sighted people
Important part（speech）
Original
（n times）
Converted
（same length)
BGM
silent
speech
Make slower
Make slower
Make this part easier to understand
35
All Rights Reserved, Copyright © NHK
time
Applications of speech rate conversion

Mobile terminal

iPhone app.


For PC

Plug-in for Windows media player



For language learning and audio book
Variable speed with rip synchronization
Variable speech for audio and visual content
on the internet
DAISY Player for visually impaired
people

Applicable both to CD player and
software
※DAISY：Digital Accessible Information SYstem
36
All Rights Reserved, Copyright © NHK
Research on accessible
technology for the elderly
[Speech Processing]
37
All Rights Reserved, Copyright © NHK
Speech processing for the elderly
Speech
Studio
Home
Speech
enhance
(narration etc.)
Mixed
sounds
Background
sounds
Estimated
speech
Calculating
loudness
input
&
Estimating
favorability of
mixing
balance
Mixing balance meter
Deriving
speech
output
Loudness
control
Estimated
background sounds
Developing as receiver’s
functionalities
38
All Rights Reserved, Copyright © NHK
Research on accessible
technology for foreigners
[“Easy Japanese”]
39
All Rights Reserved, Copyright © NHK
Research on “Easy Japanese” for
foreigners living in Japan
Increase of foreigners population
Registered foreigners : 1.7% of pop. in 2009.
Difficulty of enriching multi-lingual services
(Press-release from the ministry of justice)
Developing Web news services by
“Easy Japanese”:understandable
for foreigners in Japan
Research issues:
・Establish methods to re-write “Easy
Japanese” from news manuscripts in
usual Japanese.
・Develop authoring tools for “Easy
Japanese”
40
All Rights Reserved, Copyright © NHK
An experimental service started on
2nd April 2012
http://www3.nhk.or.jp/news/easy/
Uses short sentences and
easy words
All Kanji characters have
Furigana (phonetic characters)
Underlined words have
dictionary explanation
41
All Rights Reserved, Copyright © NHK
Authoring tool for “Easy Japanese”
Rewriting is done by a team of a
Japanese language teacher and
a reporter.
The tool assists the team work.
Authoring tool can measure the
text difficulty and show it in
index figures and color.
42
All Rights Reserved, Copyright © NHK
Other aspects of accessibility
for broadcasting
[Emergency Warning
Broadcasting System (EWBS)]
43
All Rights Reserved, Copyright © NHK
What is EWBS ?



Remote activation of Radio and TV receivers ready for EWBS
 AM, FM Radio and TV : Control signal and Alert Sound
 Digital TV Broadcasting “ISDB-T” :
EWBS Control (Activation Flag)
In Japan, EWBS has been operated since September 1985.
EWBS on ISDB-T has been operated since December 2003.
EWBS test signals are monthly broadcast in Japan.
Automatic
Activation
Meteorological
Agency
44
Broadcasting
Station
Transmitter
All Rights Reserved, Copyright © NHK
Broadcasting Service Area
Mechanism of EWBS for transmitting

EWBS signal sent by on air signal




start signal ; switches on receivers
end signal ; turns off receivers
Analog system for Radio &TV ; 640Hz & 1024Hz frequency
Digital system TV ; digital data as flag on ISDB-T
45
All Rights Reserved, Copyright © NHK
Functions of EWBS in disaster relief
1. Gathering/receiving disaster information from
authorized organizations
2. Delivering disaster information to the public general
Advantages of EWBS
1. Always connected to everybody, send signal ASAP
There are no congestions like telephone lines
2. Most cost efficient system
One way “Delivery” system is enough.
(No two-way radio communication system is required.)
46
All Rights Reserved, Copyright © NHK
Towards/beyond
Universal Broadcasting
47
All Rights Reserved, Copyright © NHK
service
/functionality
Towards Universal Broadcasting
To support variety of services/functionalities:
Hybridcast ®
(a platform of collaboration of broadcasting
and communication)
Audio description
Closed caption
Sign language
media
Broadcasting
Speech rate conversion
Read-out
….
48
All Rights Reserved, Copyright © NHK
Concept of Hybridcast
®
®
Broadcasting media
(Simultaneous, High quality, Trustworthy)
Enriched services
Program
Hybridcast ® receiver
Communication media
Broadcaster
Service provider
(Applicable to personal request)
Internet service
Program-related inf.
（Cooperation with
broadcasting）
Demand, SNS input
Synchronization of
various information
DRM technology
49
All Rights Reserved, Copyright © NHK
Coop. with mobile term.
Towards Universal Broadcasting
Accessible technology for the impaired
Technology for viewer adaptation
Broadcaster
Content provider
AD script
editor
CC by speech
recognition
CG
sign language
Multi-modal
display
Accessible technology for the elderly, foreigners
夏に電力が足りなくなるかもしれ
ない問題で、人々の節電への関
心が高くなったため、扇風機が良
く売れています。
質問：「夏に．．．高くなったため」
の部分はいりますか？
答え：後に同じことが書いてある
ので省略しましょう
現在の難しさ 2500
Speech processing for the elderly
“Easy Japanese ” for foreigners
50
All Rights Reserved, Copyright © NHK
Context-aware
•Content provision
•AV signal adaptation
Context
Who?
Demand?
Interest?
Situation?
service
/functionality
Beyond Universal Broadcasting
Universal media services
⇒Society free from information barrier
Audio description
Closed caption
Sign language
media
DVD
Broadcasting
Mobile
Web, VOD, BB
Speech rate conversion
Read-out
…
For media-independent data sharing：
Standardization (ITU, SMPTE, W3C, etc.)
51
All Rights Reserved, Copyright © NHK
Closing remarks

Introduced researches on accessible technologies




Technologies for impaired people
Technologies for the elderly, foreigners
Other aspects of accessibility for broadcasting
In order to realize the society free from information
barrier

Towards Universal Broadcasting




Based on accessible technologies
Hybridcast ® as a platform
Viewer adaptation
Beyond Universal Broadcasting

Expansion to other media
52
All Rights Reserved, Copyright © NHK
Thank you very much!!
53
All Rights Reserved, Copyright © NHK