Kernel Code Coverage Nilofer Motiwala Computer Sciences Department University of Wisconsin 1210 W. Dayton Street Madison, WI 53706-1685 USA © 2001 Nilofer Motiwala March 12, 2001 Motivation • Code coverage answers the basic question: How much of my code did I test? – Workloads exercise only a subset of the programs functionality • Code coverage for an OS kernel – Critical due to complex interactions in kernel – More difficult to extract information from kernel (Kerninst makes it easy) Kernel Code Coverage 2 March 12, 2001 Motivation • Kernel code coverage is important – Workload does not guarantee the same execution pattern over multiple runs • Networking behavior • Filesystem behavior – Error checking code is rarely executed • Coverage at basic block level – Make sure we are precise in our analysis Kernel Code Coverage 3 March 12, 2001 Basic Strategy • Dynamically instrument all basic blocks – Increment a counter at beginning of basic block • Periodically sample counter values • De-instrument if block has been reached – Reduces execution overhead • Similar to DynInst code coverage tool for userlevel applications, but with new challenges… Kernel Code Coverage 4 March 12, 2001 Basic Strategy • Dynamically instrument all basic blocks – Increment a counter at beginning of basic block • Periodically sample counter values • De-instrument if block has been reached – Reduces execution overhead • Similar to DynInst code coverage tool for userlevel applications, but with new challenges… First application of mass instrumentation Kernel Code Coverage 5 March 12, 2001 Previous Applications • Kernel performance tool (kperfmon) – Incremental approach – Mass instrumentation not encouraged – As a result, no significant issues regarding allocation of space for large number of instrumentation code patches Kernel Code Coverage 6 March 12, 2001 Previous Applications • Kernel performance tool (kperfmon) – Incremental approach – Mass instrumentation not encouraged – As a result, no significant issues regarding allocation of space for large number of instrumentation code patches However, a code coverage tool calls for mass instrumentation, and a perf tool can benefit too Kernel Code Coverage 7 March 12, 2001 The Problem • Challenge is mass instrumentation in kernel – Sparc-Solaris7 kernel is large (~3MB) • 96 modules • 13,000 functions • 188,000 basic blocks – Jump to instrumentation code using a single instruction • Jump range limitation of +/- 8MB – Finding space within 8MB for large quantity of instrumentation code is a problem Kernel Code Coverage 8 March 12, 2001 Instrumentation method Patch Area foo() counter++ (bb1 entry) displaced code counter++ (bb2 entry) displaced code • Patch area reached via a branch instruction – Replace one instruction w/ branch to patch area – However, +/-8MB range limitation of branch Need 188,000 patch areas within range Kernel Code Coverage 9 March 12, 2001 Instrumentation method Springboards foo() jmp patch_addr (bb1 entry) (bb2 entry) Patch Area counter++ displaced code jmp patch_addr counter++ displaced code • Mitigate problem by adding indirection – Springboard is smaller than a patch area – Now, only Springboards have to be in 8MB range • Still, need 188,000 Springboards nearby Kernel Code Coverage 10 March 12, 2001 Patch Area v/s Springboard • Kerninst is set up to allocate patch areas anywhere in kernel virtual address space • Would like to allocate them close to the code we are instrumenting • However, for purposes of code coverage, Springboards are given preference • Tradeoff between efficiency and instrumenting more blocks Kernel Code Coverage 11 March 12, 2001 How much space is needed? • Springboard size dictated by patch area address – Jump to 32-bit address three instructions • Need ~2 MB in 8 MB range – Jump to 64-bit address six instructions • Need ~4 MB in 8 MB range • Fortunately, we can place all patches in 32 bit address space Kernel Code Coverage 12 March 12, 2001 How much space is needed? • Springboard size dictated by patch area address – Jump to 32-bit address three instructions • Need ~2 MB in 8 MB range – Jump to 64-bit address six instructions • Need ~4 MB in 8 MB range • Fortunately, we can place all patches in 32 bit address space Still need 2 MB in 8 MB range Kernel Code Coverage 13 March 12, 2001 Kernel Address Space(64 bit) 0x000.0000.0000 0x000.1000.0000 Invalid 0x000.1040.0000 Kernel Nucleus 0x000.1080.0000 0x000.7802.0000 0x000.7c00.0000 0x300.0000.0000 0x302.0000.0000 Kernel Code Coverage 32 bit Kernel heap 64 bit kernel heap 14 March 12, 2001 Kernel Address Space(64 bit) 0x000.0000.0000 0x000.1000.0000 Invalid Kernel Text Segment 0x000.1040.0000 Kernel Data Segment 0x000.1080.0000 0x000.7802.0000 0x000.7c00.0000 0x300.0000.0000 0x302.0000.0000 Kernel Code Coverage Memory Mgmt Structures 32 bit Kernel heap File System Cache 64 bit kernel heap 15 March 12, 2001 Kernel Address Space(64 bit) 0x000.0000.0000 0x000.1000.0000 Invalid Kernel Text Segment 8 MB Range 0x000.1040.0000 Kernel Data Segment 0x000.1080.0000 0x000.7802.0000 0x000.7c00.0000 0x300.0000.0000 0x302.0000.0000 Kernel Code Coverage Memory Mgmt Structures 32 bit Kernel heap 64 bit kernel heap 16 March 12, 2001 Kernel Address Space(64 bit) 0x000.0000.0000 0x000.1000.0000 0x000.1040.0000 Invalid Trap table Kernel Text Segment Kernel Data Segment 0x000.1080.0000 0x000.7802.0000 0x000.7c00.0000 0x300.0000.0000 0x302.0000.0000 Kernel Code Coverage Memory Mgmt Structures 32 bit Kernel heap 64 bit kernel heap 17 March 12, 2001 Finding Springboard Space • Use all available space in nucleus – Allocate free memory directly – Overwrite routines that will not be invoked – Load dummy modules in kernel • 256 MB invalid region in kernel space – Unmapped but could it be mapped and used?? • 6 MB of free space will be within our branch range Kernel Code Coverage 18 March 12, 2001 Kernel Address Space(64 bit) 0x000.0000.0000 0x000.1000.0000 Invalid Kernel Text Segment 8 MB Range 0x000.1040.0000 Kernel Data Segment 0x000.1080.0000 0x000.7802.0000 0x000.7c00.0000 0x300.0000.0000 0x302.0000.0000 Kernel Code Coverage Memory Mgmt Structures 32 bit Kernel heap 64 bit kernel heap 19 March 12, 2001 Current Allocations • Inside nucleus, able to obtain – 260 KB via nucleus malloc (mach_mod_alloc) – 64 KB via dummy module – and 8 KB by overwriting routines • 28,000 Springboards for basic blocks inside nucleus • Can successfully allocate all patch heap space in 32 bit kernel heap • Able to instrument afs (18,000 basic blocks) – 210 KB Springboard space in 32 bit kernel heap • Not yet able to instrument the entire kernel Kernel Code Coverage 20 March 12, 2001 Information Flow Code Coverage Instrumentation request Sampling request kerninstd ioctl() Patch Heap /dev/kerninst Kernel Code Coverage Data Heap (counters) Kernel Space 21 March 12, 2001 Information Flow Code Coverage Instrumentation request Sampling request Individual counter values kerninstd ioctl() Patch Heap /dev/kerninst Kernel Code Coverage Data Heap (counters) Kernel Space 22 March 12, 2001 Experimental Results • Instrumented afs – Workload • Build for a large program (code coverage) • Find through multiple layers of directories • Read all files in /usr/include – Measurements • Number of functions covered • Number of basic blocks covered Kernel Code Coverage 23 March 12, 2001 Experimental Results for AFS 100 90 80 70 60 50 40 30 20 10 0 Basic blocks: 18163 Functions: 922 Covered Partially Covered Not Covered Functions Kernel Code Coverage Basic Blocks 24 March 12, 2001 Current Challenges • Memory allocation issues – More springboard space needed to instrument all basic blocks • Parsing issues – 206 of 13,000 functions, currently cannot be parsed • Unanalyzable jump instructions Kernel Code Coverage 25 March 12, 2001 Future Work • Reduction in number of instrumentation points needed – Dominator tree analysis • Dyninst Code Coverage tool achieves a reduction of 33%-49% • De-instrumentation policy – Based on time interval – Based on de-instrumentation request queue size • Sampling optimization – Only samples reflecting a change in value should be sent Kernel Code Coverage 26 March 12, 2001 Conclusion • Kernel Code Coverage – Mass instrumentation • Gives the ability to know at a fine level, what kernel code is exercised by a given workload – Dynamically de-instrument – Ongoing work Kernel Code Coverage 27 March 12, 2001