Performance Data Collection: Hybrid
Approach
Edu Metz, Raimondas Lencevicius
Software Performance Architecture Group
Nokia Research Center
5 Wayside Road, Burlington, MA 01803, USA
E-mail: Edu.Metz@nokia.com,
Raimondas.Lencevicius@nokia.com
1
© NOKIA
Nokia Research Center / 18.05.04
Motivation
• Performance profiling of complex embedded software is important
• Profiling = monitoring software execution + obtained data analysis
• 2 ways to collect data
• Event tracing
• Sampling
• Both have drawbacks and advantages
• What to use?
2
© NOKIA
Nokia Research Center / 18.05.04
Position
Data collection should be performed using hybrid approach
that combines completeness of event tracing
with low cost of sampling
3
© NOKIA
Nokia Research Center / 18.05.04
Overview
• Event tracing
• Sampling
• Hybrid data collection with example
• Why it will fail
• Why it will succeed
4
© NOKIA
Nokia Research Center / 18.05.04
Event Tracing
• Record events occurring during system execution
• Component entry and exit
• Function calls
• Execution states
• Message communication
• Resource usage
• Complete record of information
• If event was recorded, it did occur; if it was not recorded, it did not occur
• Allows complete analysis of event relationships in time
• Event tracing comes at a cost
• Trace instrumentation takes time
• Changes behavior of software system
• Could violate real-time constraints and timing requirements in real-time
systems
• Disproportional overhead for different software components
• Large data volume
5
© NOKIA
Nokia Research Center / 18.05.04
Sampling
• Intermittent recording of software system’s state
• Program counter
• Call stack
• Scheduled and blocked tasks
• Can be done strictly periodically or with certain randomness
• Overhead may be orders of magnitude less than event tracing
• Has its own downsides
• Granularity of information determined by sampling frequency
• Needs “reasonable” data collection interval to provide accurate
data
• Loss of information completeness
• May complicate application-level information extraction
6
© NOKIA
Nokia Research Center / 18.05.04
Hybrid Approach
• Often a way to minimize drawbacks of two techniques is to use them
both!
• Trace a subset of events
• Sample the rest
• Provide low overhead with adequate information completeness
• But… how not to inherit drawbacks of two techniques?
• Worst case: high overhead and not enough information for
causality determination
• Intelligent selection of what should be traced and what should be
sampled is needed
7
© NOKIA
Nokia Research Center / 18.05.04
Hybrid Approach Example:
Profiling with Tasks
• System contains a number of communicating tasks
• Performance engineer needs to understand what
parts of software consume most of the processor
time and how this influences task messaging and
t2
t1
t3
scheduling
• Using only event tracing would lead to large
Task A
overhead
m1
m2
• Using sampling would not provide complete
information about task messaging
Task B
• Hybrid approach solves the problem
• Low overhead due to sampling of inside task
activities
• Complete information of task messaging
through event tracing - low overhead due to
infrequent tracing
8
© NOKIA
Nokia Research Center / 18.05.04
t4
t (s)
Task A
Hybrid Approach Heuristics
• What should be traced?
• Infrequent events
• Task switches
• Peripheral activations and deactivations
• Events that have causality relationships
• Message sends and receives
• Interrupts (maybe)
• Events for which relative time to other event is necessary
• Video frame display
• Events for which the absolute time is necessary
• Events connected with real world: alarm clock
• What should be sampled?
• Frequent events
• Function calls (maybe)
• Bytecode/CPU op execution
• Events for which causality is not important
• Heap size (maybe)
9
© NOKIA
Nokia Research Center / 18.05.04
Related Work
• We do not know any hybrid profilers
• A lot of sampling profilers
• Intel Vtune – either sample or trace
• Hollingsworth et al. – collect all and then sample the collection
buffers periodically
10
© NOKIA
Nokia Research Center / 18.05.04
Why it will Fail
• Intelligent selection of what should be traced and what should be
sampled is difficult
• Bad tracing/sampling selection may lead to large overhead and
missing information
• Trace instrumentation is still required with all its drawbacks
• Yields two sets of profiling data that need to be reconciled
11
© NOKIA
Nokia Research Center / 18.05.04
Why it will Succeed
• Solves the issue of obtaining complete causality information with low
overhead
• Allows collecting infrequent events even in short program runs
• Limits profiling data volume
12
© NOKIA
Nokia Research Center / 18.05.04
Questions ?
13
© NOKIA
Nokia Research Center / 18.05.04