Confidential
© 2009 Teleca AB
Dual Display for Android Applications
Gururaj N
21st Jan, 2011
v0.2
Confidential
Agenda
Introducing Teleca
History of Multi Display
Dual Display Mobile Devices
Teleca’s experience in Dual Display for Android Devices
© 2009 Teleca AB
QA
2
Confidential
© 2009 Teleca AB
Introducing Teleca
Teleca is the only player in segment with comprehensive onshore/offshore
global delivery footprint across all major markets
We have 2000 mobile software engineers available worldwide
3
Confidential
Agenda
Introducing Teleca
History of Multi Display
Dual Display Mobile Devices
Teleca’s experience in Dual Display for Android Devices
© 2009 Teleca AB
QA
4
Confidential
History of Multi-Monitor (1) –
PC and TV
© 2009 Teleca AB
Dual Monitors at Work Place
Picture in Picture
Multi monitor use case
 Engineer needs as bigger screen possible to enhance productivity
 HW Support – two video ports, or add second video card
 OS Support – Adjust display properties
PIP use case
 Two programs side–by–side and audio output of one program going into
speaker and another going to headset
 Two tuner built into TV set OR one tuner being external source like DVD player.
5
Confidential
© 2009 Teleca AB
History of Multi-Monitor (2) –
Mobile Phones
Multi display Mobile Phone usecases
 The secondary display is used typically for notifications ( messages, incoming call ) and
Clock
 HW Support – two display drivers
 OS Support – Adjust display properties
 Separate applications running on two screen
Disadvantages
 Cannot write applications for secondary display
 Only one display is active at any time.
 Low resolutions
6
Confidential
Agenda
Introducing Teleca
History of Multi Display
Dual Display Mobile Devices
Teleca’s experience in Dual Display for Android Devices
© 2009 Teleca AB
QA
7
Samsung Continuum
Acer Iconia
© 2009 Teleca AB
Confidential
Dual Display Mobile Devices (1)
Samsung Continuum Use cases
Acer Iconia
 Dedicated Ticker Display for displaying
Music player controls, RSS feeds,
Social network site updates, Incoming
call notification.
 Concept laptop.
 The idea seems to be use the second
display for dynamically changing data
without affecting the activities user
does on main screen
 Used as e-Reader
 One of the display can become SoftKeyboard.
8
Confidential
Dual Display Mobile Devices (2)
HW Support
 Two different displays
 Two different frame buffers and drivers
 One display virtually split into two displays
 Single virtual frame buffer to support two independent displays
© 2009 Teleca AB
SW Support
 Framework changes to support dual displays
 SDK changes to program applications specific to Dual display
 Currently not available, mostly proprietary
9
Confidential
Agenda
Introducing Teleca
History of Multi Display
Dual Display Mobile Devices
Teleca’s experience in Dual Display for Android Devices
© 2009 Teleca AB
QA
10
© 2009 Teleca AB
Confidential
Teleca’s experience in Dual Display for Android (1)
Teleca
Worked on all
aspects
of
Dual Display on
Android
11
Confidential
Teleca’s experience in Dual Display for Android (2)
Teleca built Dual Display for Android with Texas Instruments
Demonstrated at MWC 2009
Development Environment
 TI Zoom3(OMAP 4430) with Dual Display Support (each display 864x480)
© 2009 Teleca AB
 Android 2.0 ( Éclair )
12
Confidential
Teleca’s experience in Dual Display for Android (3)
- Components
Android Components involved in Dual Display
 Applications - need to define onto which display their Activities should go.
 Activity Manager – launch application’s activities onto the right display.
 Window Manager – needs to properly handle two Activity stacks and two
window stacks
© 2009 Teleca AB
 Surface Flinger – needs to handle layer composition for two displays.
 OpenGL / EGL – needs to support one context for each display
 Linux Kernel / Driver – support for two secondary display for output and
touch input
13
Confidential
Teleca’s experience in Dual Display for Android (4)
- Design Approaches for Window Manager Support
Single Window Manager concept
 This concept is based on an extension of the existing Window Manager for the second
display. The WindowManagerService would be extended to support a second stack of
activities and a second stack of windows.
© 2009 Teleca AB
 The Activity Manager would –based- on the applications preference or the preference
of an Activity create the Activity on the right stack of the Window Manager. Besides the
changes to the ActivityManager the main changes need to be done in the
WindowManagerService and the IWindowManager.aidl.
Pros
Interface to only one Window
Manager
No additional thread
Cons
Very complex 11K lines of code
Massive design changes to central
Android component
14
Confidential
Teleca’s experience in Dual Display for Android (5)
- Design Approaches for Window Manager Support
Second Window Manager
 The concept proposes to duplicate the WindowManagerService code, rename it and
instantiate an instance of this which handles the Activities and Windows for the second
display.
 All Stake holders need to interface with two Window Managers
ViewRoot
WallpaperManagerService
InputMethodManagerService
© 2009 Teleca AB
ActivityManagerService
SensorManager
KeyguardManager
IWindowManager
IWindowManager2
WindowManagerService
WindowManagerService2
15
Confidential
Pros
Simple concept with only minor
changes to the actual
implementation of the
WindowManagerService; Standard
WindowManagerService requires
nearly no changes
© 2009 Teleca AB
Easy to migrate to next version of
Android
Cons
Synchronization of both
WindowManagerService instances
required in some parts
Code duplication
Increased memory usage due to a
second instance of the service
A number of stakeholders of
WindowManagerService need to
be modified
Not too attractive solution from
architecture point of view
16
Confidential
Teleca’s experience in Dual Display for Android (6)
- Design Approaches for Window Manager Support
Second Window Manager Service Instance
© 2009 Teleca AB
 This concept proposes to modify the WindowManagerService and the
IWindowManager.aidl code so that it can be instantiated twice. Each instance of the
service would be responsible for one display then. The required changes haven’t been
analysed in detail as this seemed to be more effort compared to previous approach.
However from an architecture point of view this seems to be preferred.
 The problem with this approach might be that the decision which
WindowManagerService needs to be used might not always be possible and thus
requiring modifications in the stakeholders. As a result the other approach was chosen
17
Confidential
ActivityManagerService
WallpaperManagerService
InputMethodManagerService
IWindowManager
KeyguardManager
© 2009 Teleca AB
SensorManager
ViewRoot
WindowManagerServiceBase
WindowManagerService
WindowManagerService2
18
Confidential
Agenda
Introducing Teleca
History of Multi Display
Dual Display Mobile Devices
Teleca’s experience in Dual Display for Android Devices
© 2009 Teleca AB
 Specific Implementation Details
QA
20
Confidential
Teleca’s experience in Dual Display for Android
- High Level Block Diagram
Applications
Home
Phone
Gallery
App. on second
display
Home 2
OpenGL
Application
…
Application Framework
Activity Manager
…
Window Manager 2
Window Manager
© 2009 Teleca AB
View System
Libraries
Android Runtime
bionic
Surface
Flinger
OpenGL
Hardware Abstraction Layer
Linux Kernel
Display
Driver 1
Modified
Display
Driver 2
New
Pico DLP
Keyboard
Driver
Not feasible
Touch
Input 1
Touch
Input 2
…
21
Confidential
Teleca’s experience in Dual Display for Android Changes across the Android Stack
Applications
 Extensions to the AndroidManifest.xml
The AndroidManifest.xml was extended to support the attribute “android:display” for the
<application> and the <activity> tag. The possible values are integer values, usually
only used when the application or an Activity of the application wants to go on the
© 2009 Teleca AB
second screen
 Sample:
<application android:label="@string/app_label"
android:icon="@drawable/ic_launcher_alarmclock"
android:display="1">
22
Confidential
 Extension to the applications resources
 The interface of the resources was extended to allow an application to set the display
programmatically. This is usually done in the onCreate() of an Activity:
protected void onCreate(Bundle icicle) {
super.onCreate(icicle);
getResources().setApplicationDisplay(Display.DISPLAY2);
© 2009 Teleca AB
}
 The class Resource was extended with two interfaces:
public int getApplicationDisplay();
public void setApplicationDisplay(int display);
 The display values are defined in Display.java as:
Display.DEFAULT_DISPLAY
Display.DISPLAY2
23
Confidential
Activity Manager
 The Activity Manager starts an Activity and as such needs to know onto
which display an application wants to go. The Activity Manager then adds
the Activity onto the Window Manager’s proper Activity stack.
 The ActivityManagerService was extended by a number of changes:
 Additional list of history entries for the second display which stores the stack of
launched activities
 Reference to WindowManagerService for the second display
© 2009 Teleca AB
 Support to start, stop and resume activities for the second display
 Support for separate Home Screen on second display. The second Home Screen
itself was not implemented during the project.
24
Confidential
Window Manager
 The Window Manager needs extensions to support two application/Activity
stacks and two Window stacks. Additionally it needs to be made second
display-aware.
 Touch Events
 Key Events
© 2009 Teleca AB
 Rendering
25
Confidential
Surface Flinger
Applications
Application
1
Application
2
Application
3
Application Framework
© 2009 Teleca AB
Surface
Surface
Surface
Surface
Surface
Libraries
Surface Flinger
Hardware Abstraction Layer
Linux Kernel
Clamshell Driver
Display Driver 1
/dev/graphics/fb0
Display Driver 2
/dev/graphics/fb1
26
Confidential
The standard Surface Flinger was extended two handle two stacks of layers, one for
each display. Internally the Layers contain the buffer to be rendered. Additionally all
the calls to OpenGL set the right OpenGL context before so that the OpenGL calls go
to the right display.
From the applications point of view the decision onto which display is drawn is done
by the display parameter set in the Surface object. The applications don’t do this
directly though. They define this either in the AndroidManifest.xml or by setting the
display in the associated resources by a direct API call.
© 2009 Teleca AB
The Activity Manager then decides onto which display an Activity wants to go and
adds it to the correct Window Manager. The Window Manager then sets the proper
display in the Surface object.
Access permissions
 To access the Surface Flinger the following permission is defined in the platforms
AndroidManifest.xml (in frameworks/base/core/res/):
 android.permission.ACCESS_SURFACE_FLINGER
 The following public constructor of Surface is hidden from the SDK and requires this
permission:
 public Surface( SurfaceSession s, int pid, int display, int w, int h, int format, int flags) throws
OutOfResourcesException;
27
Confidential
Linux Kernel / Drivers
 /dev/graphics/fb0 – Framebuffer for default display
 /dev/graphics/fb1 – Framebuffer for second display
 /dev/input/event0 – HW keyboard
 /dev/input/event1 – Touchable first display
© 2009 Teleca AB
 /dev/input/event2 – Touchable second display
28
Confidential
Agenda
Introducing Teleca
History of Multi Display
Dual Display Mobile Devices
© 2009 Teleca AB
Teleca’s experience in Dual Display for Android Devices
QA
29
© 2009 Teleca AB
Confidential
Resourcefully yours