Using Binary Decision Diagrams for Combinatorial
1
Simplified Modeling of
Combinatorial Test Spaces
Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
IBM Haifa Research Labs
2
Agenda
• Introduction
• Counters
• Properties
• Implementation
• Summary
3
Background – Restrictions
• Restrictions are a crucial part of real-life combinatorial models
• Effort and expertise required for capturing the restrictions is high
• Existing work concentrates on the ability to state complex restrictions
– Typically as propositional formulas or predicates
– Can be captured using classification trees
4
The Problem
Restrictions are typically:
• Cumbersome to read and review
• Not very generalized (exclude specific conditions)
• Hard to maintain
5
Introduction
• An approach for handling some complex restrictions
• We introduce two new constructs to the
CTD model:
– Counter parameters
– Value properties
• Real-life case studies
6
Counters
7
Counters – Example
• Consider testing an upgrade process for a virtualized server
– Traverses the host and VMs
– Upgrades various out-of-date components
– Ten VMs, each has one of four OS versions:
RedHat 5.7, RedHat 6.1, AIX 6.1.3, AIX 6.1.6
– 2 versions are marked “preferred”
– Triggered when at least half are out of date
8
Counters – Example – Cont.
• Parameters and values:
– VM1 – RedHat5.7 / RedHat6.1 / AIX6.1.3 /
AIX6.1.6
…
– VM10 – RedHat5.7 / RedHat6.1 / AIX6.1.3 /
AIX6.1.6
9
Counters – Example – Cont.
“Triggered when at least half are out of date ”… ?!
10
Counters – Example – Cont.
• (Bad) option I: explicitly exclude all combinations in which less than half are out of date
– All 653,312 of them !
11
Counters – Example – Cont.
• (Bad) option II: choose 5 VMs, and force them to be outdated
– VM1 != “AIX6.1.6” && VM1 != “RedHat6.1” &&
VM2 != “AIX6.1.6” && VM2 != “RedHat6.1” &&
…
VM5 != “AIX6.1.6” && VM5 != “RedHat6.1”
– 252 such (terrible) restrictions
12
Counters
• A counter is a parameter that counts values in other parameters
• Automatically evaluated in each test
• Can be used in restrictions and in coverage requirements
13
Counters – Back to the example
• Define counter “preferred”
– Counts appearances of “RedHat6.1” and
“AIX6.1.6” in parameters VM1 … VM10
• One restriction that excludes: preferred > 4
14
Auxiliary Variables
• The notion can be extended to auxiliary variables
• Parameters that do not define the test
– but are rather a function of the defining parameters
• Examples:
– Sum of other numeric parameters
– Count the number of duplicate values
15
(Touching on) Implementation
• Naïve implementation: Add the counter to the model, and restrictions that correctly map its values to the counted values
• Much better: Represent as functions on other parameters
• More details in the paper
16
Properties
Properties – Example
• Consider testing a Storage Area Network
(SAN)
– A server (one of four types – two x86-based, two PowerPC-based)
– An operating system (one of 17 versions – two Windows-based, three AIX-based, one virtualized, others Linux-based)
– HBA (one of 11 types, of different manufacturers)
17
Properties – Example – Cont.
• Some OSs do not run on some processors
(e.g., AIX on x86, Windows on PowerPC)
• Some HBAs do not have drivers for some
OSs
• Some HBAs cannot connect to some processors
• A total of 299 pairs to be excluded
• Tedious and error-prone
18
19
Properties
• Properties may be defined for parameters
• A value to the property is assigned for each value of the parameter
• Can be used in restrictions and in coverage requirements
• Represent different abstractions of the parameters
Properties – Back to the example
• Define property “Architecture” for parameter “Server”
– The two PowerPC servers will have value
PowerPC
– The two x86 servers will have value x86
• Define property “OSFamily” for parameter
“OS”, with values Windows, AIX, Linux
20
Properties – Back to the example – cont.
• Define properties “X_Compatible”,
“P_Compatible”, “AIX_Compatible” and
“Win_Compatible” for the HBA parameter
– Give property values to the parameter values according to their compatibility with x86,
PowerPC, AIX and Windows, respectively
21
Properties – Back to the example –
Cont.
• The restrictions now disallow:
– Server.Architecture == “PowerPC” &&
OS.OSFamily == “Windows”
– Server.Architecture == “x86” &&
OS.OSFamily == “AIX”
– Server.Architecture == “PowerPC” &&
! HBA.P_Compatible
– Server.Architecture == “x86” &&
! HBA.X_Compatible
– OS.OSFamily == “AIX” && ! HBA.AIX_Compatible
– OS.OSFamily == “Windows” && ! HBA.Win_Compatible
22
23
Simplified Modeling – Benefits
• More readable restrictions
– Easier to review
• More general
– Preserves time, reduces risk of errors
• More maintainable
– e.g., a new Windows-based OS is introduced
24
Implementation
• Our symbolic BDD-based implementation is agnostic to test-space representation
• It merely requires one to symbolically represent the set of valid tests
• Simple to extend to support counters & properties
• More details in the paper
25
Summary
• Correctly capturing restrictions is a significant obstacle in combinatorial testing deployment
• We introduced two constructs that significantly simplify restrictions in some cases
• Examples based on real-life cases
26
