Embedded System:
2
V IP Phone
Design & Development
D96921014
郭人豪
Outline






Introduction
Function List
Design
Obstacles, Workaround and Bug Fix
Rule of Thumb
Conclusion
Introduction

What is V2IP Phone?
Voice and Video over IP Phone
 Extend from H.323

Video use H.263, MPEG4 SP, H.264 BP
 Audio use G.711, G.726, G.729, AMR-NB, AMR-WB,
iLBC, etc.


Large LCD screen with Web Surf AP
3.5” ~ 7”
 Browser, Mail Client, Widget, Media player, etc.

Hardware Design of

2
V IP
Phone
3com, Agere
Separated Codec & AP Processor
 Fast but Complex and High close


TI, Broadcom, Qualcomm
Integrated DSP in AP Processor
 Complex Software Designed and Hard to port


Freescale, ADI, Renesas
AP Processor with Multimedia Accelerator
 Relatively Low Cost but limited capabilities

Separated Codec & Processor
Network I/O
LCD
Keypad I/O
TouchPanel
Hook State
LED Control
CAM
Audio I/O
Modem / Codec
AP Processor
Ethernet Switch
Integrated DSP in AP Processor
LCD
Processor Core
CAM
Internal Bus
Human I/O
Network I/O
Audio I/O
DSP
AP Processor w/ Multimedia Accelerator
LCD
ASIC MM
Accelerator
CAM
AP Processor
Human I/O
Network I/O
Audio I/O
2
V IP

Phone Capabilities
Video
Encoder / Decoder
 Multi format support
 Adapted Buffer Control
 Broken Frame Recovery
 PiP, PoP
 Lips sync
 Bitrate, Frame interval, Framerate, etc.

2
V IP

Phone Capabilities
Audio
Encoder / Decoder
 Acoustic Echo Cancellation
 Acoustic Echo Suppression
 Jitter Control
 Voice Activity Detection
 White Noise Generator
 3-way talking, Muxer, etc.

2
V IP

Phone Capabilities
Dialing Tone
EU, US, JP format
 Tone Generator
 Human interactive tone generator

2
V IP

Phone Capabilities
Protocols
SIP protocol (w/ T.120)
 RTP, RTCP streaming protocol
 RTSP streaming control
 SRTP, SRTCP security
 HTTP for firmware update & Web-base UI Control
 DHCP client, 802.1q VLAN 802.1p, 802.1p QoS
 ICMP ping, ARP dup-host detection, etc.

2
V IP

Phone Capabilities
APPs
Phone Book, Contact List
 Calendar
 Web Browser
 Electric White Board, Screen Cap
 XML support for SIP extension

SoftDSP

Idea from Freescale

Using Hardware ASIC for Video Codec
Too complex for embedded system
 Core2Duo Process only 3-5 fps for 1080p AVC


Audio Codec is relatively easy
Using AP Processor to encode / decode audio frame
 Low cost and can upgrade for upcoming audio codec
 Performance is affected by OS and CPU power

Design Requirement




Audio Heartbeat must be called every 10 or 20 ms
Video Heartbeat must be called every 10, 20, or 40 ms
Hook state must be process at once
TouchPanel must be interactive at once but can be executed later


While streaming, Audio must be always clear and no glitch





i.e. play click sound at once and insert event into list
Video is reasonable smooth
Hook state is always first priority
Web surfing or other AP may have performance penalty while streaming
is on
LED display pattern is always real-time
And on and on and on…..
System Design
Main Phone thread
Generate and Monitor
Hook thread
MultiMedia
thread
ToneGen
thread
UI Interactive
thread
SIP thread
LED control
thread
Keypad
Read thread
……
Create Events and Insert
Event Queue
Execute thread
System Design
Main Phone Process
MultiMedia
state
Hook state
SIP state
LED control
state
ToneGen
state
Dispatch
Idle state
Keypad
Read state
Loop Every 10 ms
UI Interactive
state
System Design
UI Code
Phone Middleware
I/O Routine
LED Control
Hook state
Keypad
(Human Interact)
Virtual Functions
for implementation
Callback
functions
Network I/O
Audio / Video I/O
System Design
UI Code
Keypad
Touch Panel
IPC call
Phone Middleware
Callback
functions
I/O Routine
Network I/O
Audio / Video I/O
Hook State
LED Control
Etc.
Comparison of Interrupt / Poll

High Loading


Low CPU power


How to design queue length?
Mutex, thread overhead, program style


Error resilience or high performance
Resource starvation, implementation fault
Timing

How to make sure it runs on-time?
Comparison of Combine / Separated
UI

Overhead
Need IPC to communicate
 Need CPU to process, waste of already critical CPU
load


Complexity
Need to define internal protocol for communicate
 OS support, performance


Why choose two-tier design?
Results

Code size


Footprint size


2.5MB vs. 8MB (plus C++ lib)
5.5MB vs. 20MB
Client test

500K bulk dial test passed
Audio Problem

DELAY!!!!

Linux 2.4 use OSS audio driver
2.6 starting support ALSA audio driver
 ALSA provide low/zero latency delay


TIA-810A require phone to have 100ms delay from
end to end


Mic → audio driver → audio encoder → packetized →
network → muxer → audio decoder → speaker
How to minimize the delay?
Audio Problem

WHERE IS THE CPU POWER???


VAD use little CPU, so does white noise, most audio codec
except G.726.
AES use moderate CPU but still survive


Not in 4-way talking or above
PEOPLE DON’T WANT AES!!!
AEC is much much more complicated
 According to a pre-Bell Lab engineer, there are two design in
embedded system, 16ms of 1024ms or 64ms of 128ms
 The complexity grows exponentially with detect range


How about a phone without Speaker?
Video Problem

Video Input chip not compatible with AP
Processor
PAL in okay to transform to CIF
 Big problem in NTSC to CIF



640x240 to 352x288 but no upscale (240 → 288 upscale)
The ghostly Preview Windows

Sometimes, the hardware design is just can’t match
the software requirement

Open the preview window 1x1, configure the PrP, close
the window, call UI to flush the bitmap
LED Problem

Use AP Processor’s GPIO port and CPU clock
to emulate LED on/off period
Not accurate
 Actually, totally waste of CPU and out of sync in
such a short period
 You can’t throw the CPU in the LED routing, there
are Audio and Video are waiting for you.


So, how about throw away the LED indicator
and use LCD only?
Extension Board Problem

Custom Chip for ext. board signaling and LED flashing



I2C interface, which is buggy and affect the audio chip
control
Custom protocol, which is also buggy and need to bug on
bug
No-one-can-understand spec, which is told by their senior
through phone and personal guide



The packet with no payload after each command
Time interval between commands
Write-read and read-write-read period
LCD Problem

Are you kidding? LCD problem?


Yes, and it’s really a painful experience
CPU spec is for reference only (the pirate code?)
Demo board comes with QVGA panel
 CPU spec support up to VGA panel
 But not VGA video with PIP!

Amplifier Problem

Not embedded system related but is funny
The gain value write in Audio chip manual is not the
actually measured value on board
 The auto-gain or pre-gain function on chip has the
same name but different implementation with the
one in customer requirement
 Hardware design and power management

Protocol Problem

The SDP in SIP is a RFC standard
But not in the engineer’s mind of the ATA box
designer
 Yes, the compatibility problem


But this time, there are no reference, no guide, no
document, no source code.
System Problem

In Linux 2.4 you have no clock_gettime() but
gettimeofday()




Which is extremely vulnerable while NTP settimeofday()
Embedded system often omit the external RTC clock for cost
issue
How can I do?
The invincible watchdog in embedded system

Phone crash, UI crash, power lost, in-experience user
manipulation, maintenance, Server crash, etc.
Resource Problem

Flash is used widely in embedded system
Read lightningly fast, write damnly slow
 Block all other routine while writing to flash
 Buggy interface in Kernel


Multiple write will crash, use fwrite instead


Fwrite can’t measure time
Configuration lost, Bad bits in flash, etc.
Rule of Thumb

Customer don’t know what they want, just push the
product and ads to them!


Embedded system is customized and extendable


Likewise in sales. Make sure you do not admit every function
what they just think of while stay in toilet.
But they are NOT everything. Think before agree the
requirement from customer
Don’t argue with engineers

Try to help them workaround the bug or negotiate with
customer to have another way
Conclusion



Choosing RIGHT BSP will shorter the development quite a lot
Think about what you’re designing before you design
There are always a workaround there, just you may not find it


Choosing the RIGHT HARDWARE is the same important as
good code-syntax writing


We are developing embedded system! Throw away the compatibility!
Boys, this is not the well-known x86 system
Please always keep in mind that the CPU performance is limited
while coding on embedded system

The RAM is limited too, don’t memcpy() on every stuff!
Thank you

Please have some feedback.