Implementing a World of GUIs Using the
ARM CortexM3.
Dave Wilson
Luminary Micro, Inc.
GUIs for Embedded Systems
 We are not talking about running Linux or
Windows on a low-cost consumer device!
 We are talking about using CPU-rendered,
on-screen graphical controls in conjunction
with touch or button input to control your
application.
GUI Benefits
 Customizability
 Product features can be defined by the software.
 Offer new features without needing hardware
changes.
 Visually appealing and intuitive
 Show only the information the user needs at that
time.
 Less display clutter.
The Good Old Days
In yesteryears, fewer UI
options were available to
embedded device
developers.
• Graphics-capable displays were
expensive.
• Microcontrollers were too slow to
run an application and handle
graphics processing.
• Graphics took too much storage for
use on a small device.
• Writing graphical interface code
was time consuming and difficult.
LCD Displays Are Too Expensive
• High demand for mobile phones
and the availability of oldergeneration LCD manufacturing
capacity has driven small panel
LCD display prices sharply
downwards.
$40
$35
$30
$25
TFT LCD
CSTN LCD
$20
MSTN LCD
AM OLED
• 128x64, 4bpp OLED
• 128x128, color CSTN
$15
PM OLED
~$4.00
$10
~$3.60
• QVGA color touchscreen ~$14
$5
$0
2004
2005
2006
2007
2008
Source: iSuppli/Stanford Resources, http://www.usdc.org/resources/DisplayTrends_summer2004/biggrowthahead.htm
Graphics Need Too Many MIPs
Consider a worst case:
 320x240x16bpp using an 8bit parallel interface, 2
cycle GPIO operations, 15Hz refresh.
 This requires about 14MIPS.
 28% of the CortexM3 CPU bandwidth at 50MHz.
 72% of the CPU is left for application use.
 BUT - we are not in the video business!
 Updates are seldom at 15Hz and seldom full screen.
 Interface is often 16bits wide with single cycle GPIOs
 15Hz overhead drops to about 9.5% in this case.
To offer a different example…
 Full software
rendering.
 Comparable
resolution to today’s
small displays.
 “DOOM” ran on a
25MHz 80486.
 Less than half the
power of a CortexM3.
“DOOM” from id Software
State of the art 3D
rendering on a PC in 1994.
Bitmapped images are huge!
 A 320x240 16bpp image takes 150KB!
 But…
 The display panels we are looking at include
integrated frame buffers.
 Draw directly into the display panel buffer.
 But what about application images?
 Use the CPU power to your advantage:
 Store images in a compressed format.
 Saves about 80% of the space for typical buttons and
text.
 Use palletized image formats and convert on-the-fly.
 Describe the image using 16 colors rather than 65536.
Graphics library software is big
 True if you want Bezier curves, non-rectangular
clipping regions, support for 15 image formats, pie
charts, metafile generation, complex pens,
customizable line endcaps, double byte fonts…..
 You don’t need the Windows GDI or OpenGL to
implement a very usable, compelling graphical user
interface.
 Luminary Stellaris Graphics Library compiles to about
10KB of code and needs about 400bytes of RAM
workspace.
 ASCII fonts are between 1.5KB and 6KB depending
upon glyph size.
GUI Software Aims
 Develop a minimum subset of graphics primitives that you will
need for your interface.
 Keeps code size to a minimum.
 Keeps work to a minimum!
 Isolate the control definitions and appearance from the code that
manages them.
 Allows rapid prototyping.
 Get the application working with basic controls then spend time
making them pretty.
 Be consistent in how controls are defined and used.
 Learn one set of concepts and use them in different areas.
A widget-based architecture makes a lot of sense.
Widget (noun)
 “A coil-like device placed in the bottom of cans and bottles of
beer to aid in the generation of froth.”
 “A component of a graphical user interface with which a user
interacts.”
 A widget is a control – something on the screen that you read,
press, slide, click, toggle or otherwise fiddle with to set or get
application information.
 Widgets are instantiated as instances of a widget class (don’t
worry – C++ is not needed!).
 Widgets encapsulate the drawing of the control and its
interaction with the user. Interaction with the application is via
messages sent to an application callback function.
Widget Types

Container


Canvas


A Boolean indicator that can be clicked to toggle its
state.
Slider


A clickable control which is grouped with its siblings to
allow “one-of-many” selections to be made.
Checkbox


A circular or rectangular area that the user can press to
initiate some action. May be outlined color or contain an
image.
Radio Button


A “static” control used to display text, image or color.
Canvas widgets ignore user input.
Pushbutton


A convenient way to group other widgets together and,
optionally, place them on a background rectangle.
A draggable horizontal or vertical control that reports its
position.
ListBox

A scrollable collection of text strings optionally allowing
the user to select and highlight one.
Layered Architecture
A layered UI management architecture allows
maximum flexibility and reuse potential.
Application
Widget Manager
Widget
Widget
Widget
Classes
Widget
Classes
Classes
Classes
Input Driver
(e.g. buttons or
touchscreen).
Graphics Functions
Display
DisplayDriver(s)
Driver
Input H/W
Display H/W
Display Driver
A minimal interface to abstract the display
hardware from the graphics rendering
software.
•
•
•
•
•
•
•
•
Initialize the display hardware.
Report display dimensions.
Translate colors from 24bpp RGB to a display-specific value.
Plot a pixel.
Draw horizontal and vertical lines.
Copy a line of pixels from CPU memory to the display.
Fill a rectangle with a given color.
Flush any outstanding operations to the screen.
Display Drivers Without Hardware
 Display drivers may also be written for offscreen
rendering.
 Allows double-buffering for smooth animation.
 Allows composition of images before display.
 Luminary Stellaris Graphics Library includes
offscreen display drivers for 1bpp, 4bpp and
8bpp formats, for example.
 These drivers offer the same interface as their
hardware-supporting siblings (apart from being supplied
a frame buffer image on their Init() call).
A Graphics Library for GUIs
A very visually appealing GUI can be
generated with a small set of graphics
primitives:






Straight lines
Filled and outlined rectangles
Filled and outlined circles
1, 4 & 8bpp images
Bitmap font rendering
Clipping rectangle
The Luminary Stellaris Graphics Library offers this
support and is free for use on Stellaris microcontrollers.
Graphics Context
 All graphics functions take a context pointer as first parameter.
 Graphics operations can take a large number of parameters:




Background and Foreground Colors
Fonts
Clipping rectangle
Display driver to use
 These parameters are used frequently but seldom changed.
 Rather than passing them as parameters on each call, they are
stored in a “graphics context” structure.
 Context parameters are changed via access functions:
 GrContextBackgroundSet()
 GrContextClipRegionSet()
 Context parameters remain active until the application changes
them.
“Hello World”
// Initialize the display driver and turn on the backlight.
Formike240x320x16_ILI9320Init();
Formike240x320x16_ILI9320BacklightOn();
// Initialize the graphics context.
GrContextInit(&sContext,
&g_sFormike240x320x16_ILI9320);
// Fill the screen with dark blue then outline in white
sRect.sXMin = 0; sRect.sYMin = 0;
sRect.sXMax = GrContextDpyWidthGet(&sContext) - 1;
sRect.sYMax = GrContextDpyHeightGet(&sContext) - 1;
GrContextForegroundSet(&sContext, ClrDarkBlue);
GrRectFill(&sContext, &sRect);
GrContextForegroundSet(&sContext, ClrWhite);
GrRectDraw(&sContext, &sRect);
// Say hello using the Computer Modern 40 point font.
GrContextFontSet(&sContext, &g_sFontCm40);
GrStringDrawCentered(&sContext, "Hello World!", -1,
GrContextDpyWidthGet(&sContext) / 2,
GrContextDpyHeightGet(&sContext) / 2,
0);
I can draw stuff. Now what?
 Graphics primitives offer the tools needed to draw the
user interface. Now we need to generate usable
controls.
 The Widget Manager and Widget Classes turn
graphics primitives and user input into working
controls.
 Each widget is defined in terms of a structure defining
properties of the control and a message callback
function.
 The structure contains a standard widget header
used by the Widget Manager and a class-specific
section.
Defining Widgets
Each widget is defined using a C structure:
typedef struct __Widget
{
long
struct __Widget
struct __Widget
struct __Widget
const tDisplay
tRectangle
long (*pfnMsgProc)(
lSize;
*pParent;
*pNext;
*pChild;
*pDisplay;
sPosition;
struct __Widget *pWidget,
unsigned long ulMessage,
unsigned long ulParam1,
unsigned long ulParam2);
} tWidget;
typedef struct
{
tWidget
sBase;
unsigned long
ulStyle;
void (*pfnOnClick)(tWidget *pWidget);
} tTrivialButton;
// Header structure common to all widgets
// TrivialButton-specific style information
// Click handler for this TrivialButton
Widget Tree Structure
WIDGET_ROOT
pParent/pChild
CHILD1
NULL
pNext
pParent/pChild
GRANDCHILD1
NULL
GRANDCHILD2
pNext
pParent/pChild
GGRANDCHILD1
pParent
pNext
pParent/pChild
pParent
GGRANDCHILD2
NULL
NULL
pNext
NULL
pNext
GGRANDCHILD3
NULL
NULL
pNext
An Example
How do we implement this simple
interface as a widget tree?
• 2 Radio buttons
• “Choice 1” & “Choice 2”
• 1 Push button
• “Done”
• 3 Containers
• “What will it be?” border
• Blue border behind “Done”
• Overall black background
Determining Parent/Child Relationships
 Using the right parent/child relationship is vital to ensure correct
repainting and handling of user input.
 The widget tree is painted starting at WIDGET_ROOT and
working downwards – parents are painted before children.
 Parents can be thought of as being below their children on the
screen.
 User input is handled in the opposite direction – clicks are first
passed to children and, if not processed, passed up to the
parents.
 Widgets “on top” get a chance to process input before those
underneath.
 Think in terms of planes - the further back the widget, the closer
it is to WIDGET_ROOT in the widget tree.
Drawing Planes
GreatGrandchildren
Draw Later
Grandchildren
Child
Draw Earlier
Example Widget Tree
WIDGET_ROOT
Background Container
Radio Container
“Choice 1”
Radio Button
“Choice 2”
Radio Button
NULL
NULL
NULL
NULL
NULL
“Done” Container
NULL
“Done”
Pushbutton
NULL
NULL
Handling Repaint
WidgetPaint(WIDGET_ROOT);
1
WIDGET_ROOT
NULL
2
Background Container
NULL
6
3
Radio Container
4
5
“Choice 1”
Radio Button
“Choice 2”
Radio Button
NULL
NULL
7
NULL
“Done” Container
NULL
“Done”
Pushbutton
NULL
NULL
Handling User Input






Pointer drivers generate up, down and (x,y) position messages in
response to touchscreen or pointing device input.
A touchscreen or pointer driver is attached to the Widget Manager by
setting WidgetPointerMessage as the driver callback function.
When a pointer down message (WIDGET_MSG_PTR_DOWN) is received,
the Widget Manager passes it to all widgets, children first.
Each widget checks to see if the pointer is within its area and, if so,
claims the message by returning “true” from the message procedure.
All further pointer move messages (WIDGET_MSG_PTR_MOVE) are sent
to this widget until pointer up (WIDGET_MSG_PTR_UP) is received or the
widget is removed.
Note that the order that pointer messages are passed to widgets is
different from the order in which the widgets are painted. The widgets
“on top” (children) need to see the input first before we offer it to those
which lie behind (parents).
User Input Processing Order
7
WIDGET_ROOT
Search skips to the
bottom of the tree
NULL
6
Background Container
NULL
5
3
Radio Container
1
2
“Choice 1”
Radio Button
“Choice 2”
Radio Button
NULL
NULL
4
NULL
“Done” Container
NULL
“Done”
Pushbutton
NULL
NULL
What does the application have to do?
 Define the widget tree.
 Provide handler functions for each widget.
 void OnDoneBtnPress(tWidget *pWidget);
 Initialize the display driver.
 Formike240x320x16_ILI9320Init();
 Initialize the touch screen/input driver.
 TouchScreenInit();
 Pipe input events to the widget manager.
 TouchScreenCallbackSet(WidgetPointerMessage);
 Process messages.
 while(1) { WidgetMessageQueueProcess(); }
Intelligent Display Module
 240x320 16bpp
touchscreen display
with LED backlight
 10/100 Ethernet
 Relay
 microSD slot
 4 ADC inputs
 Speaker
IDM Example
Security keypad
application
Buttons are arranged
randomly when the keypad
is activated.
Entry of the correct code
closes a relay connected to
the door lock mechanism.
A web server allows the
code to be changed.
Oscilloscope
• Uses a text- and menu-based widget system built on top of the Luminary
Stellaris Graphics Library.
• Runs on an EK-LM3S3748 evaluation board with 128x128 CSTN display.
• User input via 4 directional rocker switch with “press to select.”
Conclusion
 Using the CortexM3,
today’s low cost color
displays and software
such as Luminary Micro’s
Stellaris Graphics Library,
visually appealing, high
function graphical user
interfaces are both
affordable, usable and
straightforward to develop.
Getting hold of the code
www.luminarymicro.com/products/software_updates.html
Software
Documentation
Look under C:\DriverLib\boards\rdk-idm for main examples.
Questions?