What, why and how?
Johnny Bigert, 2012-10-25
johnny@johnnybigert.se
My View on Unit Testing
 Part of something bigger – testing on many levels
1.
Fast and early feedback!


Tight feedback loop, results within seconds
Early implies lower cost
2. Confidence!
 Less fear of refactoring
3. Better code!
 Testability requires decoupling and usable APIs
4. Reliability!
 Possible to remove all non-determinism
Unit test, functional test, system test – what does it mean?
Definitions and Terminology
 There are no definitive answers
 Different dimensions of automated tests:
 Level – Unit, component, integration, system
 API – Developer test, external test
 Characteristic – Functional, non-functional
 Acceptance testing, regression testing
What Level Are We Testing On?
 Unit – The smallest unit of program code, normally a
single function, class with methods or data structure
with functions
 Component – A set of units integrated together,
normally a module or subsystem comprising an
isolated set of classes or files.
 Integration – Several layers of a stack integrated,
normally a stand-alone application (e.g. client or
server)
 System – Several stand-alone applications integrated
(e.g. clients and servers), normally the full system
What API Are We Using For Testing?
 Developer tests – Function calls to an internal or
external interface, normally test and production code
running in the same process
 External tests – Invokation of a non-programming
interface, normally running in separate processes or
machines. For example, UI (simulated user input),
network (HTTP etc), IPC.
What Aspect Are We Testing?
 Functional – The behavior of the system, normally
expressed as a step-by-step instruction with some
alternative paths
 Non-functional – The characteristics of the system,
normally expressed as measurements and numbers.
For example, latency, round-trip time,
memory/CPU/bandwidth consumption, scalability,
availability.
Functional
Unit
Component
Integration
System
Developer
“unit test”
“component test”
N/A
External
N/A
“functional test”
“system test”
“Component test” – personal favorite. Provides stability just like a
“functional test” (test code less brittle than unit tests). Provides
flexibility, reliability and fast feedback just like a “unit test” (quick to
write, easy to mock out problematic parts, results within milliseconds).
Non-Functional
Developer
Unit
Component
e.g. test threading
model of a
component
Integration
System
External
N/A
N/A
e.g. measure CPU
consumption
e.g. “load test”
Choose How to Test
 When choosing, consider
 Cost (of writing, maintaining, execution resources etc.)
 Feedback loop (time)
 Reliability
 Trade-off example: Mocking or using system x?
 Mocking: cost of writing and maintaining, risk of
modeling wrong behavior.
 Using: cost/risk of non-determinism, cost of test
resources and longer execution times.
Regression And Acceptance Testing
 Definition: “regression” or “regression bug”
 Something that was once considered “done” but does
not fulfill requirements anymore
 Acceptance testing: the use of a test to determine if a
feature fulfills the requirements
 Acceptance tests becomes part of the regression test
suite once a feature is “done”
Necessities of effective unit and component testing
Dependencies
 Assumption: we want to achieve fast feedback and
reliable tests
 Note, there are other kinds of tests that also have merit
 Remove non-determinism: we need to be in control
 Randomness, time, file system, network, databases,
multi-threading
 Remove dependencies to non-deterministic
components
Dependencies
 What kind of problematic dependencies to you have in
your code base?
Getting Rid of Dependencies, C++
 Example: Randomness
 class A {
public:
void foo() {
bool coinflip = rand() % 2 == 0;
if(coinflip) …
}
};
 Problem: dependency to non-deterministic
component (randomness)
Getting Rid of Dependencies, C++
 C++ solution: introduce interface and use dependency
injection
 class IRandomness {
virtual bool coinflip() = 0;
};
Getting Rid of Dependencies, C++
 class A {
public:
A(IRandomness &r) : randomness(r) {} // ctor inject
void foo() {
if(randomness.coinflip()) …
}
private:
IRandomness &randomness;
};
What Is a Mock Object?
 Conforms to interface
 Programmable behavior
 For example, “if someone calls your send function with
argument ‘hello’, return 7”
 Can verify function invocations
 Mock frameworks available
 E.g. gmock (Google)
 Much more elegant in reflective languages
Getting Rid of Dependencies, C++
 Test code:
 void testFoo() {
RandomnessMock random; // inherits IRandomness
A a(random);
a.foo();
…
}
Getting Rid of Dependencies, C++
 Production code:
 void main() {
Randomness random; // inherits IRandomness
A a(random);
…
}
Bi-Directional Dependencies
Application
(inherits)
sendData
INetworkReceiver
INetworkSender
onDataReceived
(inherits)
Network
Getting Rid of Dependencies, C
 Function pointers = hard-to-read code
 Link-time tricks
 Link with other translation unit defining e.g. rand()
Getting Rid of Dependencies, C
 RandMock m; // mixed C/C++ test code
RandMock &getRandMock() {
return m; // used by test case to access mock
}
int rand() {
m.rand(); // forward to mock object
}
 Drawbacks: hard to understand test case code, one test
executable per component to test = harder to maintain
 Or clean C solution
Code Coverage
 Don’t focus on high levels of code coverage per se
 Necessary, but not sufficient
 Instead, focus on covering the most important user
scenarios
 After that, use code coverage to determine which code
has not been covered
 Analyze to understand what user scenarios are missing
Unit Test Frameworks
Test runners/frameworks
 GTest
 UnitTest++
 CppUnit
 C Unit Testing Framework
Mock frameworks
 GMock
 Hippo
 Mockpp
 Amop
Agree on what to aim for
Fast Feedback Example
From a previous job:
 Proprietary hardware/software/OS platform
 Very expensive hardware = bottleneck in testing
 Emulator to test software locally on Linux machine
 Ran unit tests in emulator, turn-around time = 15 mins
 Unacceptable
Fast Feedback Example
 Main problem: emulator start-up time
 Challenge: proprietary types used (e.g. string classes)
 Solution: fake proprietary types (string with
std::string, many others just empty implementations)
 Ran unit tests on local machine,
turn-around time < 15 seconds (from 15 mins)
Fast Feedback Example
 Invest to speed up feedback
 Invest to remove testability obstacles
 Unit testing should be easy, fast and fun
Development Excellence
 Defect prevention
 Continuous improvement - feedback from bugs in order to
improve software development process
 Relentless testing
 Regression detection
 Continuous integration running all regression tests often
 Focus on both functional and non-functional requirements
 Always releasable
 Keep CI builds green
 Keep the product working
Continuous Integration
 Staged
 Stage 1 – build, unit tests, code coverage and functional
smoke tests – 10 mins
 Stage 2 – extensive functional tests, smoke nonfunctional tests – 1 hour
 Stage 3 – full functional and non-functional regression
tests , static and dynamic analysis – 12 hours
 Run stages per
 Commit (fine-grained tracability, expensive), or
 Regularly (1. commit, 2. hourly, 3. nightly)
Code Analysis Tools
 Static analysis (run on source code)
 Lint – free, local analysis only, lots of false positives
 Klocwork – commercial, global analysis
 Coverity – expensive, global analysis, good precision
 Dynamic analysis (need representative data)
 IBM Rational Purify – commercial, memory leaks,
double frees, buffer overflow, use of uninitialized
memory. Instrumentation.
 Valgrind – free, memory problems, heap, usage, cache
usage, call graphs, threading problems. No
instrumentation.