Dolby Australia
Internship Programme
CONFIDENTIAL INFORMATION
May, 2012
Introduction
 Dolby and the new Internship Programme
 Learn about building production software
 Share my experiences working as a software developer
 Please interrupt with questions
CONFIDENTIAL INFORMATION
3
Dolby Voice
 New initiative targeting multi-party audio communication
 Over $4.5B spent annually on audio conferencing, with the user
experiencing poor to extremely poor quality
 We have the right people, technology and audio knowledge to make
it better
CONFIDENTIAL INFORMATION
4
History: Dense Immersive Communications Environment
 We began as a CRC research project at the University of Wollongong
 Funded by Telstra Research Labs and Motorola
 Project was to create scalable immersive audio
 MMO games were considered the hardest practical problem
 Globally distributed user base, all IP connected
 Real time voice for 5,000 player virtual worlds i.e. WoW
 Cost effective to deliver for virtual worlds – even free to play
 Integrated the technology into HL2:DM, CS:S and ET as demonstrators
 Ran game trials with TRL Launceston
CONFIDENTIAL INFORMATION
5
CONFIDENTIAL INFORMATION
6
History: Spatial Voice and Axon
 Spun off into a startup owned by the CRC and UOW in 2006
 Acquired by Dolby in September 2007
 Dolby created Axon
 Voice platform targeting the Gaming segment
 Technology from Spatial Voice
 Enhanced with custom Echo and noise suppression, VAD, AGC, Codec
and Renderer
CONFIDENTIAL INFORMATION
7
Axon API
•
•
•
•
The platform for creation of
other Axon Products
Middleware provided to game
developers
1,000,000 Peak concurrent
users in China (Giant)
Axon Desktop Client
Dolby Axon Network
•
•
•
Wholesale Voice Service
GA Ready November 2010
Infrastructure for 130,000 Peak
Concurrent Users
Customers:
•
•
EA Need for Speed
Jump-gate Evolution
•
•
•
•
•
Consumer service
100,000+ consumer accounts
Hybrid peer-to-peer and hosted
10,000,000+ minutes of voice
served by us each week
99.99% uptime since
launch
CONFIDENTIAL INFORMATION
8
Building Production Software
 Changing requirements
 Security
 Multiple developers
 Long term support of code base
 Diagnosing and fixing bugs in the field
 Minimum downtime
 Supporting legacy platforms
 Time constraints
CONFIDENTIAL INFORMATION
9
Design and Planning
 Keep it simple
 Make it modular
 Think about the tradeoffs between different designs
 Discuss with other team members
 Document your invariants
 Define object ownership and lifetimes
CONFIDENTIAL INFORMATION
10
Development
 Don’t trust you inputs
 Especially from the network or user
 Use asserts to check your state
 Always handle the “Release build” case
 Use logs and state dumps to provide a window into your code
 Test as you develop
 Pause in the debugger and check the state
CONFIDENTIAL INFORMATION
11
Testing
 Unit tests
 System tests
 Negative and Positive testing
 Code reviews
 Playing around and looking for trouble
 Test release builds once you are getting close to shipping
 Dynamic and static analysis





Valgrind
Visual Studio Analyze (2010 Premium or better)
Clang Static Analyzer
CPPCheck
Compiler warnings (Multiple compilers)
CONFIDENTIAL INFORMATION
12
Deployment
 Create procedures around production builds and deployment
 Automate builds and deployment
 If you can’t automate everything, automate what you can
 Test after your deployment
 Put in place mechanisms for informing customers about downtime
and updates
 Have a backup plan, deployment may fail
CONFIDENTIAL INFORMATION
13
Production
 Get feedback from users
 Make it easy/automated
 Include logs and state dumps
 Crashes in the field




Keep debugging files
WER
OSX Crash files
Automated Uploads
 Custom system
 Breakpad + Socorro – Recommended
 Automated handling of feedback and crashes
CONFIDENTIAL INFORMATION
14
WER
 Free* service
 Provides access
to minidumps
 By default any
application crash
on Windows will
be reported here
CONFIDENTIAL INFORMATION
15
Process:
Path:
Version:
Dolby Axon [194]
/Applications/Dolby Axon.app/Contents/MacOS/Dolby Axon
Mac OS X Crash reports
0.2.2.0 (1)
Code Type:
X86 (Native)
Date/Time:
2012-05-01 23:30:54.496 -0700
OS Version:
Mac OS X 10.6.8 (10K549)
 /Library/Logs/CrashReporter/ (10.5 and below)
 /Library/Logs/DiagnosticReports/ (10.6 and above)
Exception Type: EXC_CRASH (SIGABRT)
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Assertion failed: (assert_buffsize(writec, readers[id].playc, readers[id].buffer_size)), function RegisterInputLane, file /DTC_branch/AxonCore/src/dvclient/EchoBuffer.cpp, line 278.
Thread 0 Crashed: Dispatch queue: com.apple.main-thread
0 libSystem.B.dylib
0x9025ec5a __kill + 10
1 libSystem.B.dylib
0x9025ec4c kill$UNIX2003 + 32
2 libSystem.B.dylib
0x902f15a5 raise + 26
3 libSystem.B.dylib
0x903076e4 abort + 93
4 libSystem.B.dylib
0x902f420f __assert_rtn + 252
5 com.dolby.axon.dvclient
0x01191895 AxonClientPrivate::EchoBuffer::RegisterInputLane(unsigned int, unsigned int) + 485
6 com.dolby.axon.dvclient
0x011922e9 AxonClientPrivate::EchoBufferManager::RegisterInputLane(unsigned int, unsigned int) + 553
7 com.dolby.axon.dvclient
0x0118a637 AxonClientPrivate::InputLane::InputLane(AxonClientPrivate::EchoBufferManager*, Axon::Util::LogBackend*, unsigned int, AxonClientPrivate::InputDevice*) + 231
8 com.dolby.axon.dvclient
0x0118a53d AxonClientPrivate::InputLane::InputLane(AxonClientPrivate::EchoBufferManager*, Axon::Util::LogBackend*, unsigned int, AxonClientPrivate::InputDevice*) + 77
9 com.dolby.axon.dvclient
0x0119889d AxonClientPrivate::DeviceManager::AcquireInputLane(unsigned int, unsigned int, AxonClientPrivate::AxonEngine*, AxonClientPrivate::InputLane**) + 573
10 com.dolby.axon.dvclient
0x0112b8a8 AxonClientPrivate::AxonEngine::SetupInputLane(unsigned int, unsigned int, unsigned int*) + 104
11 com.dolby.axon.dvclient
0x012354a1 AxonClientPrivate::DVClientCommon::SetDevice(_DVCL_AUDIO_PATH_TYPE, DVCL_DeviceGuid_, unsigned int, unsigned int) + 737
Custom System
 DTC uses a launcher that acts a crash reporter
 Any uncaught exceptions or crashes are automatically uploaded
 Also detects potential deadlocks and uploads
 Server collects reports and categorizes based on the call stack
 Top 5 Windows crashes from a previous version





1188 ./0xC0000005/iceclient_1/AxonClientPrivate__EchoBufferManager__DeregisterOutputLane/
342 ./0xC0000005/dsound/CThreadBase__StaticThreadProc/
230 ./0xE06D7363/kernel32/RaiseException/
190 ./0xC0000005/winmm/timeGetTime/
169 ./0xC0000005/wdmaud/CWorker___StaticThreadProc/
CONFIDENTIAL INFORMATION
17
Top Windows Crash
iceclient_1!AxonClientPrivate::EchoBufferManager::DeregisterOutputLane Line 274
iceclient_1!AxonClientPrivate::DeviceManager::ReleaseOutputLane Line 299 + 0xF bytes
iceclient_1!AxonClientPrivate::AxonEngine::Destroy Line 270 + 0xC bytes
iceclient_1!AxonClientPrivate::AxonEngine::~AxonEngine Line 201
iceclient_1!AxonClientPrivate::TickManager::destroyhelper<AxonClientPrivate::AxonEngine> Line 94 + 0xE bytes
iceclient_1!AxonClientPrivate::TickManager::~TickManager Line 50 + 0x1C bytes
iceclient_1!AxonClientPrivate::DeviceManager::Shutdown Line 43 + 0x1B bytes
Axon!_DVP_context::~_DVP_context Line 1319
Axon!DVP_destroy Line 89 + 0x6 bytes
Axon!DVPModel::ModelManager::Release Line 42 + 0x1F bytes
Axon!Run Line 92
Axon!wWinMain Line 155 + 0x5 bytes
Axon!__tmainCRTStartup Line 324 + 0x1C bytes
kernel32 + 0x1339A bytes
[Frames below may be incorrect and/or missing, no symbols loaded for kernel32]
ntdll!__RtlUserThreadStart + 0x70 bytes
ntdll!_RtlUserThreadStart + 0x1B bytes
CONFIDENTIAL INFORMATION
18
Support - DAN and DTC
 Monitoring systems are in place
 Key metrics are graphed
 Checked twice daily by a developer on “support duty”
 Automated scripts detect failures and look for problems
 Any failures trigger alerts to the support device
 Support diagnoses and handles it or escalates to module lead
CONFIDENTIAL INFORMATION
19
Ensuring uptime
 Redundancy is you friend
 Dual redundant databases
 Dual internet connections




Dual redundant switches
Dual power supplies
Multiple data centers*
Multiple servers, load based/round robin scheduling
 Standby servers for single points of failure
 Software that can automatically recover from a restart
CONFIDENTIAL INFORMATION
20
General tips
 Never ignore a crash or assert it will happen to a customer (or in a demo)
 If you notice something unusual investigate, it is probably a bug
 You will need diagnostics to help you solve issues in the field. Make life easier on
yourself and write them early so you can use them in development and testing
 Reread your code before committing
 Something can always go wrong even with a “simple fix”
 Prototypes become demos. Demos become products
 Do what works for you and your team
CONFIDENTIAL INFORMATION
21
Questions?
Innovation with Impact