Kernel Code Coverage
Nilofer Motiwala
Computer Sciences Department
University of Wisconsin
1210 W. Dayton Street
Madison, WI 53706-1685
USA
© 2001 Nilofer Motiwala
March 12, 2001
Motivation
• Code coverage answers the basic question:
How much of my code did I test?
– Workloads exercise only a subset of the
programs functionality
• Code coverage for an OS kernel
– Critical due to complex interactions in kernel
– More difficult to extract information from
kernel (Kerninst makes it easy)
Kernel Code Coverage
2
March 12, 2001
Motivation
• Kernel code coverage is important
– Workload does not guarantee the same
execution pattern over multiple runs
• Networking behavior
• Filesystem behavior
– Error checking code is rarely executed
• Coverage at basic block level
– Make sure we are precise in our analysis
Kernel Code Coverage
3
March 12, 2001
Basic Strategy
• Dynamically instrument all basic blocks
– Increment a counter at beginning of basic block
• Periodically sample counter values
• De-instrument if block has been reached
– Reduces execution overhead
• Similar to DynInst code coverage tool for userlevel applications, but with new challenges…
Kernel Code Coverage
4
March 12, 2001
Basic Strategy
• Dynamically instrument all basic blocks
– Increment a counter at beginning of basic block
• Periodically sample counter values
• De-instrument if block has been reached
– Reduces execution overhead
• Similar to DynInst code coverage tool for userlevel applications, but with new challenges…
First application of mass instrumentation
Kernel Code Coverage
5
March 12, 2001
Previous Applications
• Kernel performance tool (kperfmon)
– Incremental approach
– Mass instrumentation not encouraged
– As a result, no significant issues regarding
allocation of space for large number of
instrumentation code patches
Kernel Code Coverage
6
March 12, 2001
Previous Applications
• Kernel performance tool (kperfmon)
– Incremental approach
– Mass instrumentation not encouraged
– As a result, no significant issues regarding
allocation of space for large number of
instrumentation code patches
However, a code coverage tool calls for
mass instrumentation, and a perf tool can
benefit too
Kernel Code Coverage
7
March 12, 2001
The Problem
• Challenge is mass instrumentation in kernel
– Sparc-Solaris7 kernel is large (~3MB)
• 96 modules
• 13,000 functions
• 188,000 basic blocks
– Jump to instrumentation code using a single
instruction
• Jump range limitation of +/- 8MB
– Finding space within 8MB for large quantity of
instrumentation code is a problem
Kernel Code Coverage
8
March 12, 2001
Instrumentation method
Patch Area
foo()
counter++
(bb1 entry)
displaced code
counter++
(bb2 entry)
displaced code
• Patch area reached via a branch instruction
– Replace one instruction w/ branch to patch area
– However, +/-8MB range limitation of branch
Need 188,000 patch areas within range
Kernel Code Coverage
9
March 12, 2001
Instrumentation method
Springboards
foo()
jmp patch_addr
(bb1 entry)
(bb2 entry)
Patch Area
counter++
displaced code
jmp patch_addr
counter++
displaced code
• Mitigate problem by adding indirection
– Springboard is smaller than a patch area
– Now, only Springboards have to be in 8MB range
• Still, need 188,000 Springboards nearby
Kernel Code Coverage
10
March 12, 2001
Patch Area v/s Springboard
• Kerninst is set up to allocate patch areas
anywhere in kernel virtual address space
• Would like to allocate them close to the
code we are instrumenting
• However, for purposes of code coverage,
Springboards are given preference
• Tradeoff between efficiency and
instrumenting more blocks
Kernel Code Coverage
11
March 12, 2001
How much space is needed?
• Springboard size dictated by patch area
address
– Jump to 32-bit address  three instructions
• Need ~2 MB in 8 MB range
– Jump to 64-bit address  six instructions
• Need ~4 MB in 8 MB range
• Fortunately, we can place all patches in 32
bit address space
Kernel Code Coverage
12
March 12, 2001
How much space is needed?
• Springboard size dictated by patch area
address
– Jump to 32-bit address  three instructions
• Need ~2 MB in 8 MB range
– Jump to 64-bit address  six instructions
• Need ~4 MB in 8 MB range
• Fortunately, we can place all patches in 32
bit address space
Still need 2 MB in 8 MB range
Kernel Code Coverage
13
March 12, 2001
Kernel Address Space(64 bit)
0x000.0000.0000
0x000.1000.0000
Invalid
0x000.1040.0000 Kernel Nucleus
0x000.1080.0000
0x000.7802.0000
0x000.7c00.0000
0x300.0000.0000
0x302.0000.0000
Kernel Code Coverage
32 bit Kernel heap
64 bit kernel heap
14
March 12, 2001
Kernel Address Space(64 bit)
0x000.0000.0000
0x000.1000.0000
Invalid
Kernel Text Segment
0x000.1040.0000
Kernel Data Segment
0x000.1080.0000
0x000.7802.0000
0x000.7c00.0000
0x300.0000.0000
0x302.0000.0000
Kernel Code Coverage
Memory Mgmt
Structures
32 bit Kernel heap
File System Cache
64 bit kernel heap
15
March 12, 2001
Kernel Address Space(64 bit)
0x000.0000.0000
0x000.1000.0000
Invalid
Kernel Text Segment
8 MB Range
0x000.1040.0000
Kernel Data Segment
0x000.1080.0000
0x000.7802.0000
0x000.7c00.0000
0x300.0000.0000
0x302.0000.0000
Kernel Code Coverage
Memory Mgmt
Structures
32 bit Kernel heap
64 bit kernel heap
16
March 12, 2001
Kernel Address Space(64 bit)
0x000.0000.0000
0x000.1000.0000
0x000.1040.0000
Invalid
Trap table
Kernel Text Segment
Kernel Data Segment
0x000.1080.0000
0x000.7802.0000
0x000.7c00.0000
0x300.0000.0000
0x302.0000.0000
Kernel Code Coverage
Memory Mgmt
Structures
32 bit Kernel heap
64 bit kernel heap
17
March 12, 2001
Finding Springboard Space
• Use all available space in nucleus
– Allocate free memory directly
– Overwrite routines that will not be invoked
– Load dummy modules in kernel
• 256 MB invalid region in kernel space
– Unmapped but could it be mapped and used??
• 6 MB of free space will be within our branch range
Kernel Code Coverage
18
March 12, 2001
Kernel Address Space(64 bit)
0x000.0000.0000
0x000.1000.0000
Invalid
Kernel Text Segment
8 MB Range
0x000.1040.0000
Kernel Data Segment
0x000.1080.0000
0x000.7802.0000
0x000.7c00.0000
0x300.0000.0000
0x302.0000.0000
Kernel Code Coverage
Memory Mgmt
Structures
32 bit Kernel heap
64 bit kernel heap
19
March 12, 2001
Current Allocations
• Inside nucleus, able to obtain
– 260 KB via nucleus malloc (mach_mod_alloc)
– 64 KB via dummy module
– and 8 KB by overwriting routines
• 28,000 Springboards for basic blocks inside nucleus
• Can successfully allocate all patch heap space in
32 bit kernel heap
• Able to instrument afs (18,000 basic blocks)
– 210 KB Springboard space in 32 bit kernel heap
• Not yet able to instrument the entire kernel
Kernel Code Coverage
20
March 12, 2001
Information Flow
Code Coverage
Instrumentation request
Sampling request
kerninstd
ioctl()
Patch Heap
/dev/kerninst
Kernel Code Coverage
Data Heap
(counters)
Kernel Space
21
March 12, 2001
Information Flow
Code Coverage
Instrumentation request
Sampling request
Individual counter values
kerninstd
ioctl()
Patch Heap
/dev/kerninst
Kernel Code Coverage
Data Heap
(counters)
Kernel Space
22
March 12, 2001
Experimental Results
• Instrumented afs
– Workload
• Build for a large program (code coverage)
• Find through multiple layers of directories
• Read all files in /usr/include
– Measurements
• Number of functions covered
• Number of basic blocks covered
Kernel Code Coverage
23
March 12, 2001
Experimental Results for AFS
100
90
80
70
60
50
40
30
20
10
0
Basic blocks: 18163
Functions: 922
Covered
Partially
Covered
Not Covered
Functions
Kernel Code Coverage
Basic Blocks
24
March 12, 2001
Current Challenges
• Memory allocation issues
– More springboard space needed to instrument
all basic blocks
• Parsing issues
– 206 of 13,000 functions, currently cannot be
parsed
• Unanalyzable jump instructions
Kernel Code Coverage
25
March 12, 2001
Future Work
• Reduction in number of instrumentation points
needed
– Dominator tree analysis
• Dyninst Code Coverage tool achieves a reduction of 33%-49%
• De-instrumentation policy
– Based on time interval
– Based on de-instrumentation request queue size
• Sampling optimization
– Only samples reflecting a change in value should be
sent
Kernel Code Coverage
26
March 12, 2001
Conclusion
• Kernel Code Coverage
– Mass instrumentation
• Gives the ability to know at a fine level, what kernel
code is exercised by a given workload
– Dynamically de-instrument
– Ongoing work
Kernel Code Coverage
27
March 12, 2001