Best Practices For
Testing Windows Drivers
Dieter Achtelstetter
Software Design Engineer
Windows Device Experience Group
Microsoft Corporation
About This Presentation
Fulfill a request from OEM and IHV
For best practices on testing
Suggestion for
Ways to reduce support costs for you and your customers
Increasing your customer satisfaction
Finding more bugs earlier in the process, where they cost
a lot less to fix
This advice is based on the experience of many device
testers at Microsoft
While this session attempts to cover most
drivers/devices, some points may not be applicable
to your driver/device
Agenda
Overview
Tools
Test configurations
Testing specific driver aspects
Types of testing
Resources
Call to action
Overview Of Topics
Static tools
PREfast
SDV
Driver aspects
Memory
PO
(Power Management)
Penetration/Security
I/O Cancel
Testing types
Stress tests
Manual testing
Run-time tools
Driver Verifier
PnP
Setup
Concurrency
Long haul testing
Scenario testing
Test configurations
System
Settings
Test strategies
Test early
Plan for testing
Resources
DTM kits
Test Framework
Static Analysis Tools
Tools that run against the source code
“PREfast for Drivers”
“Static Driver Verifier”
Find bugs fast
Quick and inexpensive ways to find bugs
Must pass these tests for driver signatures or logo
Run early/run often
Use these tests early in your development cycle
Resolve issues found by the tools
Run the tests for every new driver drop
For details
At WinHEC: “Static Analysis and Verification of Drivers”
Tools and documentation: WDK
Driver Verifier
Verifies different aspects of drivers at runtime
Built into Windows
Type verifier from the Run menu
Valuable in your regular testing
Use standard settings
Does Special pool, IRP verification, IRQL checking, etc.
Same settings are used for driver signature or logo testing
Only enable settings for drivers under test
To ensure proper testing of your driver
Get high test coverage
By running with a full test pass
You must exercise the code; otherwise, it cannot verify the driver
For details
Tools and documentation: WDK
Many new features for Windows Vista
Test Configuration
Vary your test environment
Don’t just test your device in a static
environment – change things around
System configuration
System environments
Device settings
Make these part of your regular test pass
System Configuration
Different platforms: x86, x64, Itanium
System with S1 and S3 support (for PO testing)
Ensure system has WDDM video drivers
So you get hybrid sleep (S3 with Hiberfile)
Multiprocessor/multi-core machines
Only way to truly test concurrency
Hyperthreading doesn't count for this testing
Different chipsets
Checked build
Skews timing
Has additional assertions
See “Using the Checked Build” in the WDK
System Environments
Low memory
See “Boot Parameters to Manipulate Memory” in the WDK
Large memory
This is important for both for X86 and 64 bit
See “Will I need special hardware for memory top-down testing?”
in the WDK
High CPU utilization
From something other than testing the device
Stress testing does this
Skews timing
Can uncover race conditions in your driver
Device Settings
Test your device with all of its settings
Not just the default settings
Only you know what these settings do
If your device uses hardware resources
(IRQ, I/O Port...), change them
WDK tool “PnP Driver Test”
does rebalancing
Manually in Device Manager
Device Configuration
Test several of the same device
in a system
Topologies
Behind a PCI-PCI bridge
Connected to a large tree of devices
For USB or 1394 device, for example
…
Testing Specific Driver Aspects
Aspects of driver testing that may
not usually be your focus
But tend to have high failure counts
Driver aspects
Memory
PO
Penetration/Security
I/O Cancel
PnP
Setup
Concurrency
Memory
Still a problem area with drivers
Corruption – testing methods
Driver Verifier with special pool setting
Static analysis tools
Code review
Stress testing
Leaks – testing methods
Code review
Manual testing
Use “PoolMon” In the WDK to monitor pool allocations
Generally, do repetitive operations, and look for continued
pool usage increases
Plug And Play – PnP
WDK tests
“Disable/Enable with IO”
“PnP Driver test”
Long haul testing
Set up system that tests PnP for several days
Test the driver/device in docking stations
Test undock/dock
Manual testing
Physically unplug and replug
The device
The bus/media: USB cable, net cables...
Use your device
Then disable/enable it
Power Management
Windows Vista = more aggressive PO management
Not just for laptops anymore
WDK tests
“Sleep Stress with IO” – cycles through sleep states
and does device I/O before and afterward
“Pwrtest” – Logs power events
Long haul testing
Set up system that tests PO for several days
Manual testing
Use the device, and then
Put the system to sleep
Hibernate the system
PnP And Power Management
Test both together
Don’t just test each in isolation
This is a real-world scenario
For example, often when you close the
lid on a laptop, you also unplug devices
WDK tests
Common Scenario Stress with IO
Manual testing
Combine existing automated tests
Or do these manually
Fault Injection
Test driver code paths that are hard to hit
under normal use
Error handling
Corner cases
Hard to test
You may need to add test code to the driver
during testing
Write a test filter driver
Driver Verifier has a Fault Injection setting
For USB, try Device Simulation Framework
Concurrency
Do different things to the driver at the same time
Analysis of our tests show they do not give very good
coverage in this area
System configuration
Use multiprocessor machines
Testing
Use your device in multiple processes
Use your device and
Disable it
Shut down the system
Put the system to sleep
Change the configuration
Get data from it (WMI/IOCTL)
…
Penetration/Security
Be resilient against mischief
Don’t assume that only your user-mode code will be calling
your device
Code review the IOCTL and WMI code path
Who should have access to your devices: Test for it
See “Creating Reliable Kernel-Mode Drivers” in WDK
Errors in buffered I/O, etc.
Test and code review for these pitfalls
Device path exerciser in WDK test – limited but useful
Run with Driver Verifier on target driver enabled
For bugs found with this tool, do a code review
Look for similar bugs in other code paths
Where bug was found, look for other types of bugs
Don’t expect this tool to find other bugs in this area for you
Use static analysis and verification tools
I/O Cancellation
Still a problem area with drivers
Hard to test
I/O must be currently active in the driver
Testing
Use your device, and then kill the process using it
Doesn’t guarantee you tested I/O cancellation
Set a break point or add tracing on cancellation code
Make sure it gets hit during your testing
May need to add test code to your driver
Experiment
Setup Packages
Test for all scenarios
OEM preinstall
During OS installation (for boot drivers)
After OS installation
OS Migration
Install your driver on Windows XP, then upgrade to Windows Vista
and ensure drivers and applets work
Same for Windows Vista to Windows Vista upgrades
For example, Home Basic to Home Premium SKUs
Test with multiple packages
Create 2 packages with 2 different versions of your driver
Install version 1
Upgrade to version 2
Rollback to version 1
Un-install
Tools: ChkInf
Testing Types
Testing types
Stress tests
Scenario testing
Stress Testing
Combines tests to simulate a heavy load
Tests your driver under different system conditions
Can find bugs that can be missed in feature testing
Stress tests
WDK has a stress test
Allows you to add your device-specific test
Build your own
Testing – time and configuration
Run for at least 24 hours at a time
Run on different system configuration
Run with different device settings
Scenario Testing
Test your devices the way customers
will use them
Don’t just rely on synthetic
tests/automation
Focus on corner cases scenarios
Mainstream scenarios tend to not
be a problem
Customers always find new ways
to use your devices
Resources
Driver Test Manager kits – DTM
Test frameworks
Windows Device Testing Framework – WDTF
Device Simulation Framework – DSF
Driver Frameworks
Kernel-mode Driver Framework – KMDF
User-mode Driver Framework – UMDF
Windows Error Reporting
DTM Test Kits
Run early
When software, firmware, and hardware changes are still possible
Use these tools on a daily basis in your testing
Keep in mind that driving issues to a resolution takes time:
Plan for 2-4 weeks
Many useful DTM tests for your regular testing
Use DTM automation features in your lab
Add your own tests
At WinHEC: WDK and DTM presentations
“How to Use the WDK to Develop, Sign and Test Drivers”
“Using the WDK for Windows Logo and Signature Testing”
Sign up for the WDK Beta:
www.microsoft.com/whdc/driver/wdk/betawdk.mspx
See WHDC Web site: www.microsoft.com\whdc
“Index of Windows Driver Kit Tools” in the WDK
WDTF – Windows Device
Testing Framework
Framework for building driver/device tests
Some DTM tests are built using WDTF
PnP and Power Management tests
Write your own tests
For details
At WinHEC: “Using the Windows Device
Testing Framework”
Tools and documentation: WDK
DSF – Device Simulation
Test Framework
Simulate hardware through software
Currently only for USB
For details
At WinHEC: “Using the Device Simulation
Framework for Software Simulation of USB
Devices” session
WDK
DSF USB Loop Back Device Simulation
Windows Driver Frameworks (WDF)
Generally, testing is same as for other drivers
Some features that might help
Features
In Flight recorder
Integrated Verifier
Normal Verifier
Skew execution
Object live-time tracking
Flags leaks on unload
Fault injections
Via registry key settings
Debugger extension
UMDF
X
App Verifier
X
KMDF
X
X
Driver Verifier
X
X
X
X
X
Windows Error Reporting
Database of Windows crashes
Improve experience/reduce costs
Improve your customers’ experience
Reduce your support costs – and your partners’
Engage through Winqual
Sign up to see crash data for your drivers
Provide solutions through Windows Update
Check data regularly
For just-released products, check daily
For products on the market for a while, check monthly
For details
Winqual Web site: https://winqual.microsoft.com
Call To Action
Incorporate the testing points from this
presentation into your regular test pass
Test early
Start in the design and prototyping phase
Take advantage of the resources listed here
Engage with Windows Error Reporting
Engage with driver testers at Microsoft
Drvtest @ microsoft.com
Additional Resources
WDK: http://www.microsoft.com/whdc/wdk
Testing tools on WHDC:
http://www.microsoft.com/whdc/DevTools/tools/
Driver aspects
PO: http://www.microsoft.com/whdc/system/sysperf/resumeperf.mspx
Memory: http://www.microsoft.com/whdc/driver/security
/mem-alloc_tst.mspx
Setup: “Using Driver Install Frameworks (DIFx)” – WDK
Microsoft Symbol Server
All OS symbols for all releases and patches:
http://www.microsoft.com/whdc/devtools/debugging/
Send email: Windows Driver testing alias
Drvtest @ microsoft.com
Questions
Tomorrow’s “Ask the expert” session
© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market
conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.
MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.