White Box Testing
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Agenda
The need
Benefits
Code review
Unit tests and automation
cppUnit Concepts
Memory leaks
Quantification
Coverage
APIs
Challenges
Best Practices
Don’ts
Metrics
Cyclomatic Complexity
Need for White Box Testing
•
•
•
•
•
•
•
•
•
•
•
•
Better to test anything from outside-in as well as inside-out
Most of the functional and performance issues arise due to bad coding
Ultimately the code matters – hence test the code left right and center
Black box tests are used to check cause and effect without getting into
internals
White box tests do the same by getting into the internals of every program
Every developer is by default a white box tester
Medical field requires general physicians and surgeons
Certain problems can be identified and solved only by surgeons
A person may look healthy but internally the person may have high bp and
sugar; they are not visible externally
So far, the importance of white box testing is not upto the mark
Product companies definitely need white box testing
If your application must scale to a very large extent, white box tests are
inevitable
Black Box Cannot Test These
•
•
•
•
•
•
•
•
•
•
It takes long time to check memory leaks
Any exceptions that happened but unnoticed or unhandled
Which portion of if condition is evaluated
What path the program took to achieve the end result
For every request what is the memory and cpu time taken
Is there any dead and unused code
Is there any extra code that is not needed
What are the potential breaking points in code
Is the code compliant to best practices
In large business apps, the data paths will be too large to test; it will
require senior business analysts; if done thru white box mode,
normal programmers can bring out great bugs
Diagnostics tools
•
•
•
•
•
•
•
•
•
•
•
•
IBM Rational Purifyplus
JProbe
Parasoft tools
HP Mercury Diagnostics
Dynatrace
.Net profiler
AppPerfect
Netbeans
AQTime
All tools work on instrumentation and trace options
Some tools help in analyzing system memory space as well
Usually the system cpu executions are privileged, and hence we
may not get access to those
Business Benefits
• Ensures that the core building blocks are intact, when
adding new features and modules
• Go to market with confidence on product stability
• Provides greater visibility in terms of technical and
internal compliance of the product
• Vulnerability testing always needs a greater amount of
white box testing
• Resource utilization reduction is key for marketing.
Products that use less memory and bandwidth are the
most welcome ones
• Keeps business leaders informed on the underlying
strengths and weaknesses of the application platform
Understand Design
• Honestly speaking, how many of us use proper design before
coding?
• Can the same happen to real estate industry without a proper floor
plan or for pharma industry without proper process and formula for
drugs?
• Design has 4 parts – UI, database, Interface and Logic design
• UI design is key aspect when it comes to usability and compliance
• Database design ensures scalability and storage
• Interface design is key for plug and play during integration
• Logic is important for the business flow of the application
• Logic design can be a flow chart or text algorithm or pseudo code
• Most of the modelers do have pictorial representation of design
• Ideally the code must reflect the translation of pseudo code
• Traceability of design elements to code is one of the key areas we
need to look into
• A change in design may impact several parts of code
Code Review
• Most of the times, this is ignored or not done to its true
spirit
• Code review must have design and code side by side
• Code review can save upto 30% of testing cost at a later
point in time
• Usual problems happen due to cut and paste of code
where it is not required
• When there is a transition from one developer to another,
code goes thru challenges
• Quick fixes on big projects cause enormous amount of
turbulence on production systems
• Ideally senior people must review code
• Spend at least 10% of coding time on code review
Code Review Checklist
•
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Just try these 10 critical points in daily life
No hard coding
Ensure loop termination conditions
No object creation inside loops
Close every object that you open
Give comments to every code block
Use database connection only when you need
Remove unused variables and code portions
Follow a consistent naming convention
Try to reduce overloading methods very often
Check SQL or file status soon after every disk
operation
Unit Testing
• The smallest piece of code that can be independently
run and tested is a unit
• Unit can be a page or a function or a method within a
class or a whole program itself
• It is the developer’s responsibility to unit test the code
• Unit test must focus on data type of every parameter
• Focus on data format of every parameter
• Focus on boundary values of every parameter
• For any given code, what deliverable do we provide as
part of unit testing?
• Most of the times it is just the trust on the developer
• Developer must spend at least 30% of coding time on
unit testing alone
How to draw boundaries
• What is the line that demarcates unit and
other tests?
– One expected output out of a function
– No chaining
• When to stop unit tests?
– When a set of parameter combinations are
tested, stop the same
– Boundaries and equivalence partitions must
be tested before exit
Unit Testing Checklist
•
•
•
•
•
•
•
•
•
•
•
•
•
Are the naming conventions followed?
Are the comments done properly?
Is there any hard coding?
Test case for custom exceptions.
Test case for system exceptions.
Test case for “body” of an if condition.
Test case for “body” of an else condition.
Test case for every loop termination.
Test case for every recursion termination.
Test case for pointers release (for memory leaks).
Test case for every procedure entry and exit.
Test case for every parameter validation for procedures.
Test case for resource release (Closing DB connections, releasing
objects etc.)
Unit Test Matrix
• A single test must make a call to a function with some
parameters combination and expect return value(s)
• If a function has got multiple parameters that are of
different types, then combinations of all parameters are
possible and it will become a Cartesian product table
• As the first step, a matrix needs to be prepared with all
such combinations
• Then the matrix must be optimized to reflect the
practicality of the inputs
• Possibilities of return paths must also be documented
• The samples may go in 100s but not to worry on
• Since automated unit tests run extremely fast without
human inputs, 1000s of tests can be executed in minutes
Automate Unit Tests
• It is not practically possible to give all combinations of the inputs to
programs and run
• To reduce time, we need to write a program, that calls the program
under test and pass parameters
• Ideally the test program must be in the same language as the
program being tested
• This will help in maintaining the homogeneity of the data type of
parameters passed
• Since developers know the programming language, it is easy for
them to write the test program as well
• These are called test suites or test scripts
• The investment on the unit test automation is one time as long as
changes are minimal
• The person who automates test for a program must know the
program well
The core of unit automation
• Class A
– Method m1(p1, p2)
– Method m2(p3, p4)
• TestClass TA
– Testcase tm1
• ta = new Class A
• rc = ta.m1(4, 5)
• If (rc == exp_result)
– Pass
• Else
– Fail
• // destroy ta
Level 1 Automation
• Crude method – quick solution for short
term projects
• Write wrapper callers
• Hard code parameters
• Cut and paste test code with no hesitation
• Spend little time on test creation
• Test code can be throw-away soon
• Simple logs, more manual result analysis
required
Level 2 Automation
• Parameters to tests must be separated
from test code
• You can use csv or xml or any format for
test data
• Useful to run same test on larger test data
sets
• When a mixture of customers use the
product, the data set also will change
largely – this will be helpful in that case
• Use your own test logs with more details
Level 3 Automation
• Use test tools such as cppUnit or jUnit etc.
• Build auto recovery options if tests fail in
between
• Ensure every test has start-run-cleanup
structure
• Use this on long term test engagements
• When tests are to be reusable, this is the
best option
• If you know cppUnit, the concepts are
exactly the same in nUnit, jUnit etc.
Stubs and Drivers
• When the program under test needs some prebuilt data or state, we need to mimic the same
• This is a short-cut approach, but it serves the
purpose
• Stub is the one that mimics a called function
• Driver is the one that mimics a caller function
• When we use stubs and drivers, it is a must that
we ensure that the data created by them are
destroyed soon after the test
Testing APIs
• APIs do exchange of data from one piece to another
• APIs are usually consumed by many independent
programs
• APIs in general are meant to provide a window of
information to 3rd parties
• One change in API will affect all consumers at once
• Most of the current day apps provide APIs
• We can send tweets from our application using APIs
• We can send emails from Amazon SES thru APIs
• API calls must be assumed to be context neutral
• Some APIs do require authentication mechanism
• APIs fall under external interfaces
Wrapper Tools for APIs
• A set of business critical APIs are now being
used in all EAI scenarios
• FIX, SWIFT, News feeds, HL7, HIPAA, Part 11
standards etc
• A variety of commercial tools are available in
each category
• When messages are sent, these tools validate
the structures of XML or EDI against the
templates
• Also, there are different versions of these APIs
exist like html 5 etc.
cppUnit
• Write test wrappers in cpp for cpp programs
• If you are familiar with cpp programming, you may not
require cppUnit !
• There are many versions of cppUnit itself as it is open
source – many people modified it for their convenience
• You can write your own test library and framework
• A set of pre-built classes and methods are already made
available to minimize test coding time
• Testcase, Testsuite, Assert are the important aspects in
cppUnit
• Testcase is the basic block – every single test
• Testsuite contains multiple tests ( a batch)
• Assert is the actual verification point where actual result
is compared with expected result
cppUnit
• Fixture – known set of objects for test
• setUp is the place where we create testcases
• tearDown is the place where we remove all
created tests
• Commonly used asserts
–
–
–
–
–
Assert
Assert_message
Assert_equal
Assert_throw
Assert_equal_message
Instrumentation
• White box tools and profiler tools use one single concept called
Instrumentation
• Source code is compiled and built as binary
• These tools understand the format of these binary files – where
function entry points are made, how they exit, where stack is
maintained etc.
• Before profiling, they instrument the binary – inject their own log
codes in the binary locations
• During the execution, the instrumented code is run and every detail
is logged using these injected code (such as entered function x,
executed loop y, exited function y, destroyed object o etc.)
• Using these logs, the tools provide internal details
• This is very similar to sending a probe inside our body and doing a
scan of that probe
Overheads
• When profilers run along with the instrumented
programs, it adds load to the machine
• This is a necessary overhead we need to take
• The performance overheads are negligible when
compared to actual object size
• The statistics that are collected out of the
programs will vary from time to time; but if the
variance is beyond 5%, then the tool has real
issues
• Also, do not run 2 competing tools on the same
machine. The effects are unpredictable
Memory Leaks
• When a memory pointer is declared and
initialized and it is not released, it is a leak
• Even in managed code sections, the leaks are
there for a time till the garbage collector acts
• Leaked memory locations are useless and
vulnerable
• One program leaking 10 bytes of memory per
call – can bring down the system in a day when
1000s of users use the same for 100s of
transaction
• This is very important when we deal with device
drivers and embedded systems
Memory Profiling
• This happens at runtime
• Profiling is the basic block of performance
engineering
• Concept of instrumentation and tracing happens
• Call graph will tell what is the path the program
touched in terms of functions
• Statistical profilers talk about memory and cpu
• Memory consumed at that instance by variables
and functions
• Memory released at any given point of time
Memory Quantification
• What is the consumption of memory and
cpu is quantified
• Usual quantification happens in basic unit
of measurement
• UOM can be KB or machine cycles
• How many cpu cycles are spent in this
method when this is running
• How many cpus cycles are spent to
initialize the class
Memory Usage
Method
get_call_details logon do_registration
12%
1%
10%
view_bill
view_summary
2%
17%
pay_bill
22%
dispute_bill
36%
logon
do_registration
Kbytes
138
1607
view_bill
254
pay_bill
2897
dispute_bill
4876
view_summary
2346
get_call_details
1345
Buffer Overflow
• When arrays or memory blocks are declared and we try
to access beyond those boundaries it is called overflow
• When program variable space is packed, the overflows
can cause damage to program counters or other stack
areas
• When stack is compromised, the result is program abort
• Dynamic arrays are also vulnerable to overflow when
people do not keep track of the max allocations
• Buffer overflow can result on network buffers also when
the response buffer is underestimated
• Protection of buffer overflow is usually made by
providing padding space to every memory space – but
this needs to be done against memory optimization
Object Issues
• Objects require space and the methods need
cpu cycles
• When objects are declared and not destroyed,
they also cause leaks
• Unused objects are vulnerable areas for attack
• Closure and nullifying objects are essential for
optimized use of memory
• Creating a series of objects in a multi user
environment can cause spiral problems
Function-wise Quantification
• Every function has variables and statements
• Parameters and variables take space from
symbol table as well as stack
• Statements do take cpu cycles based on how
many times they get executed
• When one function calls another, there is a huge
amount of context switch happening at heap and
program registry levels
• How many times these context switches happen
will also determine the complexity of the
programs
Test case for coverage
• A path is defined as a route thru which a specific
functionality is achieved by calling a series of steps
• If programs flow from top to bottom, it is easy
• The more conditions, the more complex it is
• Coverage must ultimately make up to 100% of the tested
function
• Purecoverage will help in achieving this
• Covering statements based on design is the best method
• But when design is not in sync with program, we need to
apply simple logic to hit every statement that is present
in a program
Coverage Output
Line
Code
47
48
49
50
51
52
53
Time
myproc (p1, p2)
7181
if (p1 < p2)
call proc1();
if (p1 > p2)
call proc2();
if (p1 == p2)
call proc3();
2
388
0
0
2
6789
Coverage for conditions
• a < b && c > d
– In the above c>d is evaluated only when a < b comes to true
• a < b || c > d
– In the above c>d is evaluated only when a < b comes to false
• Though part of condition is not evaluated by compiler as
part of optimization, they may be later used in the
program
• Conditions need not be derived based on business case
• Most of the time, the conditions are based on parameter
values of the functions to determine the path to be taken
Coverage for blocks
• This depends on how many conditions we have in the
code
• Write one test for every relational operation of the
condition
• Definitely test db, IO and memory exceptions
• Write one condition for every catch block
• In case the exception cannot be easily triggered, have
conditional compiled statements to raise such exceptions
• Most of the time, people ignore exceptions thinking that it
will be taken care – remember, it will be caught, but how
to treat the exception is up to us
Coverage goals
• Coverage must have 100% at the end of all tests
for every function
• There must be 1 test for every relational
operation in conditions
• There must be 1 test for every loop termination
• There must be 1 test for every exception
• There must be 1 test for every test exit point (if
method has multiple return points)
• There must be one test for successful sql
operation
• There must be one test for unsuccessful sql
operation
Metrics Collection
• Average number of unit test cases per page/component/any specific
unit in the project
• Total number of unit test cases
• Total number of unit test cases failed in test pass-1
• Total number of unit test cases failed in test pass-2
• Total number of unit test cases failed in test pass-n
• Total number of memory leaks found
• Percentage code coverage as per the tool statistics
• Execution time for each test run
• Total test execution time for all units
• Ratio of # of defects found in unit testing to # of defects found in
code review
• Defect distribution across units (asp related, db related, parameter
related etc.)
Metrics Analysis
• Trend analysis helps in seeing whether the error
rate moves up or down
• See pattern of failures and find out the corrective
action
• Metrics vary a lot from developer to developer;
hence find out error rate pattern across
developers
• Analyze failure pattern and time of failure
• It is not easy to catch all errors in one round of
testing; hence do more rounds; more often
Configuration Management
• Since we develop test code, we need to check in
these code sections to VSS or subversion or
PVCS
• Maintain a uniform x.y version code to all test
cases and suites
• Check in the test data along with test code as
well
• Have one single person who can oversee all the
CM operations
• Never allow people to have local copies of test
code
Best Practices
•
•
•
•
•
•
•
•
Keep adding more tests every day
More data preparation is the key
Test often - even twice a day
Never ignore warnings from IDEs
Run unit tests on a clean test bed
Review unit test code as well
Run regression unit tests daily
Run unit tests from other developer’s machine
Best Practices
•
•
•
•
•
•
Run unit tests by a different developer
Make unit test code simple
Never try to have if then else in a test code
Do not have loops in a test code
Run critical tests first
Run all unit tests in a single session; never
stagger
Don’ts
• Do not test entire functionality in one test
• One test must satisfy just one condition
• Never underestimate any parameter
combination
• When skype or sony programs can come down
just by 1 bug, our program too can come down
• If certain validations are performed in other
layers, do not do in unit tests
• Never run tests on a test machine without clean
up
Don’ts
• Never miss the null value test for any
parameter
• Never miss any negative value tests for
database connection or query tests
• Never miss trying to access an object or
record that does not exist, because these
are the places where exceptions get
triggered
• Do not miss to pass a wrong xml structure
to web services
Challenges
• White box testers must be developers
• Hence companies use them to do core development
work and ignore writing white box tests
• White box tests need tools and sometimes they are a
little expensive; hence companies avoid this route
• Main challenge comes in not documenting design. Since
design is the basis, and if that is not present or
incomplete, white box tests do not yield good results
• Project planning never considers the time for white box
testing; especially in service companies, this is out of
scope!
• When changes are made to the programs being tested,
the changes are not documented well and hence unit
tests go out of gear
Challenges
• When same developer runs the unit tests, there
is a chance to ignore issues – so have
developer A testing the code of developer B
• Initial unit test automation code takes some time;
hence do not expect the ROI on day 1
• Results can be seen only after one or 2 quarters.
Many management teams do not have patience
to wait for such a long time
• People try to implement logic in different layers
that they are designated for; instead of coding
some logic in Java, they code in stored
procedure and call that from java – this makes
profiling and coverage difficult
Cyclomatic Complexity
•
•
•
•
•
•
This is to arrive at the testing complexity of code
Nodes, edges are the primary factors
Higher the complexity, higher the number of tests we need
This is based on graph theory
Many physical problems can be solved by graph theory
e.g. traveling salesman theory
•
•
•
•
M = E - N + 2P
E - number of edges (lines)
N - number of nodes (rounds)
P is the connected components (start and destination points)
•
The more if conditions and branches, the nodes and edges increase
•
Call graph of all methods will show the independent connected components
Business Flow Tests
• When we do white box tests, we test individual
components
• Hence it comes to unit tests
• Divide flow into multiple components
• Test them one by one
• Aim at one transaction at a time
• Club white box tools with functional test tool
• Since business transactions need database
setup, ensure you have test stubs that prepopulate records in tables
How to speed-up
• Let a senior create a sample test for every
function
• Let a senior prepare unit test matrix
• For a few tests, senior and junior
developers must do pair test automation
• Then junior takes care of rest of the unit
tests for automation
• Senior reviews the unit test code
• Lead runs all tests every day
Get ready to take blame
• Unit tests are code; hence they may also have
bugs. Developers may start blaming the tests
themselves
• Tests may be redundant if not properly reviewed
• Sequence of unit tests must be carefully done.
Always use bottom up approach on unit tests –
test the smallest one first
• Run unit tests soon after build is released, and
before BVT starts by the black box team
Never Give Up
• Teams give up white box tests after a quarter or so
• This is because, they do not see visible returns
• Remember – these things fail is a feel-good factor on
testing; at the same time, these things work is more
important as well
• Most of the times, great systems are brought down due
to missing white box tests and not black box functionality
• It may take years to find out, one great bug – but it is
essential for big time products
• If you are in product company, be patient and white box
will pay back