Graphics for Embedded GUI on STM32
Agenda
Presentation
Graphics Terminology
GUI Composition and update
Graphics Library
Speaker
T.O.M.A.S.
2
GUI Basic Vocabulary
Pixel / Bitmap
• A pixel (Picture Element) is the smallest element
of a digital image
• It is defined by a combination of primary colors
defining its color value
• A bitmap (or raster image) is an array of pixels
that forms a digital image
Height
• Bitmap properties :
• Width (in pixels)
• Height (in pixels)
Width
• A Pixel Format (RGB888, RGB565… etc.)
4
Raster vs Vector Image
•
•
•
•
•
Bitmap
Color value for each pixels
High ROM (FLASH) usage (stored as C tables)
Limited and known RAM usage
No MCU usage (if no compression)
MCU usage can explode for geometrical
transformations (rotation, scaling, etc) and
require filtering
Zoom
•
•
•
•
Vector graphics
Vector description of image content
Low ROM usage
High MCU usage as image needs to be
generated : a GPU is mandatory
Direct geometrical transformations
Zoom
5
Raster vs Vector Image
•
•
•
•
•
Bitmap
Color value for each pixels
High ROM (FLASH) usage (stored as C tables)
Limited and known RAM usage
No MCU usage (if no compression)
MCU usage can explode for geometrical
transformations (rotation, scaling, etc) and
require filtering
Zoom
•
•
•
•
Vector graphics
Vector description of image content
Low ROM usage
High MCU usage as image needs to be
generated : a GPU is mandatory
Direct geometrical transformations
Zoom
6
Bitmap
• Definition : number of pixels composing an image
• CGA (320 x 200), VGA (640 x 480), SVGA (800x600), XGA (1024 x 768), etc…
• Resolution : number of points forming a given length (Dots Per Inch)
→ Pixels Per Inch in display domain
• A Laptop : 1600x900 – 14 inches → 131 ppi
• Iphone 7 : 1920x1080 – 4.7 inches → 326 ppi
• STM32F746G Disco : 480x272 - 4,3 inches → 128 ppi
1 inch
7
Frame Buffer
• Portion of contiguous memory containing an uncompressed bitmap to
be displayed
• Consists of color values for each and every pixels to be shown on the
display
• Need fast access memory storage (e.g. RAM, SRAM or SDRAM)
• Can be very large : Width x Height x Pixel Size (1,16,24,32 bits)
depending on display
24 bits : 480 x 272 x 3 = 382 KB
16 bits : 480 x 272 x 2 = 255 KB (STM32F746G, 324KB RAM)
→ trade-offs !
8
Display Cycle
Usually
In FLASH
Font
Step
Text 1
Text 2
Frame/Rendering rate
1
Usually
In SDRAM
Prev
Refresh rate (e.g. 60 Hz)
1
: Fill the frame buffer
• The frame buffer is built by composing graphical
primitives
• Need large memory to store input bitmaps, not
necessarily the fastest one
Next
2
Step
2
: Display the frame buffer
• The frame buffer is regularly transferred to the
display through a dedicated hardware interface
• For example at 60 frame per second
• Need fast memory access to store buffer
9
Double Buffering
• Display is refreshed at constant rate (e.g. 60 Hz ~ every 16 ms)
• A complex frame may take more than 16 ms to be rendered
« Hide » the next frame construction using a second buffer
while the current one is sent to the display
• Front buffer : sent to the display every 16 ms (no writing allowed once sent)
• Back buffer : becomes front buffer only when fully rendered
10
Image Viewer Example
Frame N
16ms
Back Buffer
Frame N+1
16ms
Loading…
16ms
Loading…
Prev
Front Buffer
Loading…
Next
Frame N+2
16ms
Loading…
Prev
Back Buffer not ready
Swap Buffers
Loading…
Loading…
Prev
Display
Next
Next
11
• Defines the pixel color representation
Pixel Format
• Color Space : set of color components
1 component
« Luminance only »
« Grayscale »
3 components
« RGB »
• Color Depth : the number of bits to code the color value, defines the color precision
1 bpp
4 bpp
24 bpp
RGB888
𝟐1 = 2 val.
𝟐4 = 16 val.
• Data mapping of values in memory
𝟐24 = +16 M val.
12
Pixel color coding
Direct coding
Indirect coding
32-bit per
pixel
24-bit per
pixel
16-bit per
pixel
16-bits
per pixel
8-bits
per pixel
4-bits per
pixel
ARGB8888
RGB888
ARGB4444
ARGB1555
RGB565
AL88
L8 - AL44
L4
Color LookUp
Table
0x00 A R G B
Px Color Index
0x01 A R G B
...
0xFF A R G B
Px Color Value
Px Color Value
A
A
R
G
B
R
G
B
13
CLUT example
• Acceptable quality while only 8 bits value (index) per pixel color
• But this 256 colors CLUT is almost dedicated for this picture
14
Alpha Channel
• The Alpha channel defines the level of transparency of a pixel
• e.g. ARGB8888 Pixel value
A
R
G
B
Each component value ranges from 0 to (𝟐𝟖 − 1) = 0d255 = 0xFF
• 0xFF = opaque
Red box pixels values →
0xFFFF0000
0x7FFF0000
0x63FF0000
0% transparent
50% transparent
75% transparent
15
Alpha Blending
• Blending consist in mixing a source image with a destination one taking
into account the alpha information of each pixel
• Alpha value [0;255] is converted to a [0;1] value for computation
Source
α = 0x00
Blending
Result
α = 0xFF
Destination
α = 0xFF
Instead of…
Simple
Mixing
Result
Each output pixel color value is computed with:
𝐶𝑜𝑙𝑜𝑟𝑂𝑢𝑡𝑝𝑢𝑡 = 𝐶𝑜𝑙𝑜𝑟𝑏𝑢𝑙𝑏 𝛼𝑏𝑢𝑙𝑏 + 1 − 𝛼𝑏𝑢𝑙𝑏 𝐶𝑜𝑙𝑜𝑟𝑏𝑎𝑐𝑘𝑔𝑟𝑜𝑢𝑛𝑑
16
Anti-aliasing
• Aliasing is a “stair case” effect observed on
oblique edges of digital images
• Anti-aliasing : adapt edge pixels alpha
value to mix with background for :
• bitmaps with non-square-edges content
• anti-aliased font support
Aliased
Anti-aliased
Aliased
Anti-aliased
17
Pixel Format data mapping
Pixel format
32 bits data block organization
31:24
23:16
Color depth (bpp)
7:0
15:8
ARBG8888
𝑨𝒙 [7:0]
𝑹𝒙 [7:0]
𝑮𝒙 [7:0]
𝑩𝒙 [7:0]
32
RGB888
𝑩𝒙+𝟏
𝑹𝒙 [7:0]
𝑮𝒙 [7:0]
𝑩𝒙 [7:0]
24
RGB565
ARGB1555
ARGB4444
𝑹𝒙+𝟏
𝑨𝒙+𝟏
𝑮𝒙+𝟏
𝑹𝒙+𝟏
𝑨𝒙+𝟏
𝑮𝒙+𝟏
𝑹𝒙+𝟏
𝑮𝒙+𝟏
𝑩𝒙+𝟏
𝑮𝒙+𝟏
𝑩𝒙+𝟏
𝑮𝒙+𝟏
𝑩𝒙+𝟏
𝑹𝒙 [7:3]
𝑨𝒙 [7]
𝑮𝒙 [2:0]
𝑹𝒙 [6:2]
𝑨𝒙 [7:4]
𝑮𝒙 [1:0]
𝑹𝒙 [3:0]
𝑮𝒙 [7:5]
𝑩𝒙 [4:0]
16
𝑮𝒙 [7:5]
𝑩𝒙 [4:0]
16
𝑮𝒙 [7:4]
𝑩𝒙 [3:0]
16
L8
𝑳𝒙+𝟑
𝑳𝒙+𝟐
𝑳𝒙+𝟏
𝑳𝒙[7:0]
8
AL88
𝑨𝒙+𝟏
𝑳𝒙+𝟏
𝑨𝒙 [7:0]
𝑳𝒙[7:0]
16
AL44
𝑨𝒙+𝟑
𝑳𝒙+𝟑
𝑨𝒙+𝟐
𝑳𝒙+𝟐
𝑨𝒙+𝟏
𝑳𝒙+𝟏
𝑨𝒙 [7:4]
𝑳𝒙[3:0]
8
18
GUI composition and update
Graphical Elements
• Basic geometrical forms,
• Images,
• Text,
• Buttons (images + state machine + actions)
Advanced graphical controls: “Widgets”
•
•
•
•
Progress bar
Check boxes
Sliders
Customized ones…
20
GUI Management
• Handle each graphical elements appearence and internal logic
• Set of primitives that composes an element
• Transparency
• Element’s state (e.g. button pressed or released)
Button
Button
• Handle their purpose in the overall GUI
• Relation with other elements (Z-order, interactions)
• User interactions (e.g. through touch screen)
• System events (e.g. sensor input, timer)
• Handle the actual refresh of the frame buffer
Button
21
Frame buffer refresh
22
• Rendering rate : e.g. Digital Clock widget
• Frame buffer changes only every second (or more)
• Display still need 60 times per second refresh rate
• Partial Rendering
• Update the entire frame buffer at each frame update is
not always (almost never ?) necessary
« Dirty area » or « invalidated area »: area that need to be updated
• Occlusion culling (Z-ordering)
• Do not render hidden objects (avoid bitmap read)
(e.g. Background behind the clock will be excluded)
« Dirty area »
Graphics Library
Framework
• Provides ready-to-use advanced graphical control elements (widgets)
• Provides an environment (usually object-oriented)
• To handle the position and size of the graphical elements that compose the GUI
• To handle the graphical elements connections to incoming/outgoing event
• to user-based interactions (touch screen handling)
• to other graphical elements or system events
• Provides additional tools
• Composing graphical elements (GUI Editor)
• Build and prototyping environment (gcc toolchain as a minimum)
• Bitmaps/font import tools
24
Graphics Engine
• Save MCU usage
• Deep knowlegde of platform architecture (e.g. FPU availability for M7-based platform)
• Use of available HW accelerator as much as possible (i.e. Chrom-ART in some STM32)
• Save memory bandwidth usage
• Bitmap compression or use of optimized indirect color mode
• Occlusion culling algorithm that skip reading of hidden objects
• Advanced “dirty regions detection” algorithm to update only what is needed
• Must not overload the application
• Low memory footprint (~20 to 40 KB of RAM on ST partners solutions)
• Deterministic memory allocation as much as possible
25
Conclusion
Conclusion
• Understand the vocabulary to deal with Graphical User Interface
Bitmap – Frame Buffer – Alpha Blending – Pixel Format
• Understand what composition/rendering is versus screen refresh
Double buffering – Display cycle
• What needs to be watched to handle a GUI efficiently
Graphical elements – Frame Rendering – Occlusion Culling – Partial rendering
• How a graphics library can help in this process
GUI Framework – Graphics Engine and HW optimizations
27
Questions ?