THE ADVANTAGES OF REAL-TIME TRACE
DEBUG IN COMPLEX EMBEDDED SYSTEMS
E
M
B
E
D
D
E
D
S
O
F
T
W
A
R
E
W H I T E P A P E R
BRAD DIXON, MENTOR GRAPHICS
HUGH O’KEEFFE, ASHLING MICROSYSTEMS
w w w.m e n t o r.c o m
The Advantages of Real-time Trace Debug in Complex Embedded Systems
INTRODUCTION
Working in embedded system design today, product development windows are becoming shorter and shorter
as engineering teams are under more pressure to deliver a product with even greater complexity. Products in
the embedded space have a limited “window” in which to hit. Development slippage of any kind can mean
that the one opportunity for that product might be jeopardized or lost altogether.
Debug and validation are a fundamental, yet time consuming part, of the development process. To assist
developers, trace tools can be used to provide unique functionality and features to ensure successful and
on-time implementation. Without question, trace tools are an essential and integral part of the embedded
engineer’s weaponry.
TWO TYPES OF DEBUG IN USE TODAY:
“STOP ‘N’ STARE” AND REAL-TIME TRACE
Over time, embedded debugging techniques have evolved from the days when printf() was the best available
option, to more sophisticated techniques used today. Today’s debug technology is based on the on-chip
debug logic in the embedded processor.
On-chip debug generally supports two different types of functionality:
Ŷ Run-control debug – Also known as “stop ‘n’ stare” debug which is characterized by the use
of breakpoints to inspect a software state and is considered a type of intrusive debugging.
Ŷ Real-time trace – A far more advanced technique often described as debug “on the fly” and is
considered non-intrusive.
This paper will briefly explain these two approaches and discuss in greater detail the advantages of real-time
trace and why, in most situations, real-time trace is the preferred choice among developers who are validating
and debugging complex systems or multicore architectures. Real-time trace is also the preferred choice even
among debugging systems that do not have hard, real-time requirements.
RUN-CONTROL DEBUG
Run-control debugging requires a fairly simple on-chip debug interface, typically four or five pins using a
JTAG protocol to communicate between the probe and the embedded device being debugged. The system
is fairly straight forward and is best used when the software is of limited complexity or it can tolerate being
halted during execution.
The problem with run-control debug is that it’s an intrusive approach, which means the program must be
halted or stopped to examine its state. Further, visibility into the system is non-existent or drastically limited
when the application is executing. Because of these limitations, run-control debugging is not recommended
for robotics, hard-disk drive head management, or even engine control units – systems which require that the
embedded processor must continuously run to control the feedback loops and maintain mechanical stability.
Run-control debugging has limitations in other areas as well:
Ŷ It cannot detect problems that occur at full clock speed (real-time debug is not possible).
Ŷ It will not be able to detect or locate problems that occur intermittently due to subtle interactions
between the software and hardware, these include multiple processing systems.
Ŷ It will provide limited insight into interrupt handlers that typically run asynchronously and must
complete quickly in order to service hardware deadlines.
w w w. m e nto r. co m
2
The Advantages of Real-time Trace Debug in Complex Embedded Systems
Ŷ It does not have the capability to find problems that occur relative to external events
during real-time execution.
Ŷ When encountering cache related issues, typically halting the program when using run-time
debug often requires flushing of all caches, thus losing cache context.
These many limitations make it very difficult to debug real-time embedded systems. In safety critical and/or
performance critical applications for example, run-control debugging simply does not provide sufficient
depth of test and debug necessary to ensure the product under development meets even the most basic
requirements. This approach is almost useless for real-time embedded systems since stopping the system to
view a problem actually changes the real-time behavior of that system.
REAL-TIME TRACE
Not long ago real-time trace had limited availability. The tools were also difficult to use and quite expensive.
Today, we are finding more and more processors are including real-time trace facilities to encourage access
and ease of use. Most of the ARM processors, for example, now include an embedded trace macrocell, or ETM.
Some of the basic advantages of real-time trace over run-control debugging include:
Ŷ Embedded developers debug in real-time, without having to halt or intrude on the application
under debug.
Ŷ Developers are able to capture and store cycle-accurate information which provides accurate timing
details on program execution time and program data access – from a single core to multiple cores.
Ŷ The ability to capture detail surrounding specific events of interest and ignore everything else. Also,
developers are able to look forward and backward (in real time) from these points of interest to see
what has happened.
Unlike traditional printf() solutions, real-time tracing allows this to be done with very low latency (e.g. suitable
for use in an interrupt handler) and allows system wide, time-stamped correlation of all messages (even if
originating from different cores) into a single trace stream.
HOW REAL-TIME TRACE WORKS
Real-time trace has the ability to capture a “trace” of the instruction sets executed by the processor – while
the processor is running in real time – and then stores these instructions in a buffer to be analyzed at a later
time. In addition to capturing the executed instructions, developers can also choose to capture the data used
by those instructions. Therefore, it is possible to select which instruction and which data are captured for
further analysis.
It’s also possible to select a “trigger condition” such as the execution of a particular instruction, or writing a
particular value to be assigned to specific location in memory. More complicated trigger conditions are
sometimes available depending on the tool and processor. When a trigger condition does occur, the buffer
stops capturing the trace data ensuring that the buffer retains a trace of that system’s behavior around the
time the trigger condition occurred.
w w w. m e nto r. co m
3
The Advantages of Real-time Trace Debug in Complex Embedded Systems
1. Trace data rate limitations can cause stalls or data loss.
2. Once the on-device buffer is full, data must be streamed to a very high-powered PC workstation.
3. Small on-probe trace buffers limit full speed capture to only minutes.
1. Sourcery CodeBench queries Ashling Vitra-XD for data only as it is needed, minimizing workload.
2. 12.8 GB/sec trace data rate - 45% faster than the incumbent.
3. Large trace buffer supplemented by an on-device 500 GB hard drive.
w w w. m e nto r. co m
4
The Advantages of Real-time Trace Debug in Complex Embedded Systems
POWERFUL FEATURES OF REAL-TIME TRACE
Because real-time trace is non-intrusive, it is possible to debug parts of an embedded system that are typically
non-debuggable. Other powerful features include:
Ŷ The ability to determine the program flow up to some event i.e., how did my program get to
this point? Developers can debug a memory exception, data-abort, or stack-overflow.
Ŷ The ability to determine what the program was doing when the event happened. Real-time
trace can show you the program flow up to specific point.
Ŷ Insight within your program to show you where the event happened; what happened before this?
Ŷ The capability to measure the real-time performance of an interrupt-handler i.e., is it meeting
its minimum response time? How is it interacting with other interrupts? What happens when
it interrupts an already executing interrupt?
Ŷ Capturing intermittent or inter-related issues e.g., why is my codec dropping frames occasionally
when I connect a USB device?
Finally, there are two things to keep in mind about real-time trace. First, because it is non-intrusive real-time
trace can give you accurate timing information. Using this timing information allows you to investigate timing
bugs that would otherwise be difficult to investigate in any other way.
Second, with real-time trace, you have to choose in advance what information you want to collect. Unlike
“stop ‘n’ stare’ debug which allows you to look at the state of any part of your system, trace does not stop at
the trigger so you cannot get additional information about the your system outside the parameters the trace
collected.
REAL-TIME TRACE FOR NON REAL-TIME EMBEDDED SYSTEMS
As mentioned earlier, one of the primary advantages of real-time trace is that it gives the software developer
insight into a particular point in the code or state. This characteristic makes real-time trace exceptionally
valuable for solving bugs which are either intermittent or where it’s not clear how the program can reach its
final, yet faulty, state. This is true even when the system is not a real-time embedded system, or the bug is not
related to any real-time aspect.
Another example might be data corruption. If a global variable, or perhaps an element of data structure
becomes corrupted, it is often quite difficult to see how it is being overwritten. Even once a developer has
found the code that is writing to the location, all this typically reveals is that a pointer is incorrect. Using realtime trace, a developer can set a trigger to the corrupted location and then work back from that point to learn
what has happened.
One final example is in the area of debugging problems that cause processor exceptions e.g., data aborts.
Once again, the ability to work backward from the trigger is a critical key difference. Using real-time trace, a
developer can place a trigger on the execution of exception vector instructions, and then work back from that
point to determine how the exception was caused in the first place.
w w w. m e nto r. co m
5
The Advantages of Real-time Trace Debug in Complex Embedded Systems
APPLICATIONS WHERE REAL-TIME TRACE IS USED
The benefits of real-time trace debugging have ensured the wide-spread adoption of trace tools in many
market segments that encompass both performance- and safety-critical applications.
A few of these areas include:
Ŷ Automotive
Ŷ Aerospace
Ŷ Military
Ŷ Medical
Ŷ Mobile Communications
Ŷ Networking
Ŷ Banking
THE VITRA-XD REAL-TIME TRACE PROBE
As discussed in this paper, real-time trace debugging is about
figuring out what the cause is for the effect you are observing.
Only after you have visibility deep inside your system – as it
happens in real-time with no effect to CPU performance – can
you begin to diagnose the problem. Real-time trace is highly
advantageous when compared to traditional methods of
debug because you can actually capture what you need to see.
A particular real-time trace probe recommended by the
software development engineers and system designers at
Mentor Embedded is the Vitra-XD trace probe developed by
Ashling Microsystems. The key advantage of the Vitra-XD when
compared to other real-time trace probes is that it has a large
buffer for storing a vast amount of trace data (as depicted on
page 4). With this particular capability, developers can record
system performance for extended periods of time, increasing
the probability of capturing all of the causes which lead to a
higher probability of identifying the problem.
Highlights of the Vitra-XD trace storage capabilities include:
Ŷ Parallel (up to 16-bits) trace up to 400MHz double-data
rate (DDR).
Ŷ 12.8 Gb/s 16-bit parallel trace data capture rate.
Ŷ Captured data can be independently time stamped using
Vitra-XD’s 50-bits, 5ns resolution timestamp generator.
w w w. m e nto r. co m
The Ashling probe has been integrated with Mentor’s
CodeBench to create non-stop insight with visibility up
to the Linux process level.
6
The Advantages of Real-time Trace Debug in Complex Embedded Systems
Another advantage of the Ashling Vitra-XD probe is fast capture rates. Many other debug solutions require
the device to be slowed down so that the trace probe can keep up. Of course, when you slow the device down
you are no longer debugging in real time. Capture rate highlights include:
Ŷ Over 500GB on-board trace storage memory (2GiB DDR3 RAM and 500GB high-speed hard disk).
Allows extended trace capture for non-intrusive profiling and code coverage.
Ŷ Automatic trace clock/data skew adjustment (“AUTOLOCK”) to ensure integrity of captured
high-speed data.
Ŷ Real-time trace streaming to host PC through Gigabit Ethernet interface (for unlimited trace
capture depending on the host PC available disk space).
CONCLUSION
Today, it is more apparent than ever that semiconductor companies are continuing to enhance and improve trace IP
on their more popular microprocessors. The goal is to enable embedded system developers to perform advanced
levels of real-time trace debugging and performance analysis.
Recognizing this trend, Ashling Microsystems and Mentor Graphics have joined forces to provide advanced tools
for real-time trace debug on Linux embedded systems. Developers now get high-speed, long-duration software
trace with visibility up to the Linux process level. The combination of the Ashling Vitra-XD debug probe and
Mentor’s Sourcery CodeBench provide an unprecedented level of trace, inspect, and debug representing a major
advance in state-of-the-art, real-time trace debug.
Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.
For the latest product information, call us or visit:
w w w.m e n t o r.c o m
©2013 Mentor Graphics Corporation, all rights reserved. This document contains information that is proprietary to Mentor Graphics Corporation and
may be duplicated in whole or in part by the original recipient for internal business purposes only, provided that this entire notice appears in all copies.
In accepting this document, the recipient agrees to make every reasonable effort to prevent unauthorized use of this information. All trademarks
mentioned in this document are the trademarks of their respective owners.
MGC 05-13
TECH 11110-w