Agile Developers Go Testing
Part II
Darcy Casselman @flying_squirrel
Alexei Zheglov @az1
November 2010
What are we talking about today?
• A lot of things…
• They may seem disconnected, but they all:
– improve software quality
– involve testing
– done by developers
• None of them can do the job alone
• We cannot:
– Talk about all of them
– Give them equal time
Unit Testing
• We showed you a very simple unit test
example earlier
• We’ll show you a more difficult example later
• Let’s step back first
Focus on Quality
quotes
"...in 2000, during the dot-com bubble, ... software quality for
North American teams... ranged from 6 defects per function point
to less than 3 per 100 function point, a range of 200 to 1. The
midpoint is approximately 1 defect per 0.6 to 1.0 function
points. This implies that it is common for teams to spend more
than 90 percent of their effort fixing defects.”
http://www.amazon.com/Kanban-David-J-Anderson/dp/0984521402/
http://www.amazon.com/Software-Assessments-Benchmarks-Best-Practices/dp/0201485427/
Logical Connection?
“The midpoint is approximately 1 defect per
0.6 to 1.0 function points. This implies that it
is common for teams to spend more than 90
percent of their effort fixing defects.”
How does he know what they are spending
their time on?
Capacity Allocation To Fixing Defects
Capacity Allocation To Fixing Defects
=
Defect Injection Rate
Bugs, Features
http://www.flickr.com/photos/dratz/1045336659/
Another View
• Root-cause analysis of
established practices
• E.g. unit testing
• Five-whys applied to
the solution, not the
problem
• The link between the
third and fourth whys?
Source: Karl Scotland
http://availagility.co.uk/2010/10/04/a-root-cause-analysis-of-agile-practices/
Compare Two Teams
Team A
• Just finished 40 “features”
(units of work)
• Introduced 10 “bugs” (units
of rework)
Team B
• 10 features
• 8 bugs
Team A Progression
1
40  10  2.5  ...  53
3
•
•
•
•
•
Terms beyond the first three are negligible
Most defects eliminated in the next 2 releases
The 2nd maintenance release scope is very small
Illusion that any team can do it
Time spent fixing defects: 25%
Team B Progression
10  8  6.4  5.12  ...  50
• The sum of the first 3 terms: only 24.4 (less
than half of the total!)
• The sum of the first 10: approx. 45
• Time spent fixing defects: 80%
We want to be more like Team A
(or better)
and certainly not like Team B
Testing
• Is what we are talking about today
• Leaving out other technical practices
• Remembering that testing itself does not
provide quality
Test Matrix
Why?
Unit
Did we build the code right? Does it work as specified?
Test Matrix
Why?
Unit
Acceptance
Did we build the code right? Does it work as specified?
Did we build the right code? Does it do what the user wants?
Test Matrix
Why?
Unit
Did we build the code right? Does it work as specified?
Acceptance
Did we build the right code? Does it do what the user wants?
Integration
Do system components work together? What about performance?
Test Matrix
Why?
Unit
Did we build the code right? Does it work as specified?
Acceptance
Did we build the right code? Does it do what the user wants?
Integration
Do system components work together? What about performance?
System
Is the system installed and configured correctly?
Test Matrix
Why?
Unit
Did we build the code right? Does it work as specified?
Acceptance
Did we build the right code? Does it do what the user wants?
Integration
Do system components work together? What about performance?
System
Exploratory
Is the system installed and configured correctly?
What can we find out about the system that is not specified by these tests?
Test Matrix
Coverage
%
Unit
Acceptance
Integration
System
Exploratory
Auto
?
Tools
Developers’ Role
Test Matrix
Coverage
%
Auto
?
Tools
Unit
Yes
NUnit/JUnit and similar tools
isolation (mocking) frameworks
Acceptance
Yes
acceptance testing frameworks
(Cucumber, FitNesse, Robot etc.)
domain-specific languages
(DSLs)
Integration
Yes
UI testing frameworks (e.g.
Selenium)
System
Yes
same
Exploratory
No
-
Developers’ Role
Sources: Robert C. Martin, Rob Diana http://regulargeek.com/2009/06/12/how-much-testing-is-enough/
Test Matrix
Coverage
%
Auto
?
Unit
90%
Yes
NUnit/JUnit and similar tools
isolation (mocking) frameworks
Acceptance
50%
Yes
acceptance testing frameworks
(Cucumber, FitNesse, Robot etc.)
domain-specific languages
(DSLs)
Integration
20%
Yes
UI testing frameworks (e.g.
Selenium)
System
5%
Yes
same
-
No
-
Exploratory
Tools
Developers’ Role
Sources: Robert C. Martin, Rob Diana http://regulargeek.com/2009/06/12/how-much-testing-is-enough/
Test Matrix
Tools
Developers’ Role
Yes
NUnit/JUnit and similar tools
isolation (mocking) frameworks
take personal
responsibility
50%
Yes
acceptance testing frameworks
(Cucumber, FitNesse, Robot etc.)
domain-specific languages
(DSLs)
Integration
20%
Yes
UI testing frameworks (e.g.
Selenium)
System
5%
Yes
same
-
No
-
Coverage
%
Auto
?
Unit
90%
Acceptance
Exploratory
collaborate
-
Sources: Robert C. Martin, Rob Diana http://regulargeek.com/2009/06/12/how-much-testing-is-enough/
A Very Short Introduction to
Acceptance Testing
Consider This Test Script
• A new customer registers
• The customer places three books in the
shopping cart
• The customer goes to check-out
• The customer fills in their delivery address
(province: Ontario)
• The system offers free delivery to the customer
Source: Gojko Adzic. Bridging the Communication Gap, pp. 78-79
http://www.amazon.com/gp/product/0955683610
Compare the Test Script to This
The system offers free delivery to Ontario
customers when they place their first order
and their order contains at least three books
Convert to the Tabular Form
number of books
order number
ship to
free delivery?
3
1
Ontario
Yes
number of books
order number
ship to
free delivery?
3
1
Ontario
Yes
2
1
Ontario
No
number of books
order number
ship to
free delivery?
3
1
Ontario
Yes
2
1
Ontario
No
3
2
Ontario
No
number of books
order number
ship to
free delivery?
3
1
Ontario
Yes
2
1
Ontario
No
3
2
Ontario
No
3
1
Alberta
No
How To Do It
• Collaborate:
– developers
– business analysts
– testers
– product owner
• Which tool to use is not as important
• Don’t look up terminology in Wikipedia
Books
http://www.amazon.com/gp/product/0955683610
http://manning.com/adzic/
Unit-Testing
When System Under Test (SUT)
Has Dependencies
Break the dependencies
What If We Don’t?
•
•
•
•
•
•
•
Configuration
Database data
Remote services
Permissions
Time to run
Infrequent errors
Dependencies in development
Can You Run Unit Tests Like This?
http://www.flickr.com/photos/theredproject/2878006012/
Code Example
Caller
StudentRepository
GradingEngine
NotificationService
Code Example
Unit Test
Stub
System Under Test
Mock
(Open IDE)
Popular Frameworks (.NET)
0.92%
1.21% 0.57%
3.97%
1.77%
1.99%
3.12%
4.05%
1.35%
37.62%
9.01%
34.42%
Moq
RhinoMocks
Hand-rolled
Typemock Isolator
Moles
JustMock
NSubstitute
NMock2
NMock
FakeItEasy
None
Other
Source http://www.osherove.com/blog/2010/9/10/2010-poll-which-isolation-framework-do-you-use-in-net.html September 2010
Two Camps
Design for Testability:
Moq, RhinoMocks, etc.
Mock Anything, e.g.
Typemock Isolator, JustMock
• Dynamically fake objects
implementing
interfaces/base classes in
their own way
• Require software under test
to be designed to be testable
• Programmers to follow
SOLID principles (esp.
Dependency Inversion
Principle)
• Rely on Profiler API
• Can intercept anything and
run something else instead
• Can fake statics, sealed
classes, etc.
• Don’t force design for
testability
Popular Frameworks (Java)
• “Conventional”:
– Mockito http://code.google.com/p/mockito/
– EasyMock http://easymock.org/
– JMock http://www.jmock.org/
• Alternative:
– JMockit http://code.google.com/p/jmockit/
Roy Osherove
• The Chief Architect at TypeMock
• Wrote: “A customer switches to Moq – and
I’m happy”
• And: “This is pure awesome. This is why I
come to work every day.”
• How is it possible?
• Read it: http://www.osherove.com/blog/2010/9/29/acustomer-switches-to-moq-and-im-happy.html
Back to Our Code Example
• Grading engine operates on Student and
Grade
• But Student has a Department
• And Grade has a Course and Course has a
Semester
• We start mocking
• Soon, we have a lot of mocks
“Every time a mock returns
another mock a fairy dies”
• Who said it?
• Not the guy whose tweet comes up first in
Google search results
• I am not 100% sure
• What the heck does it mean anyway?
“Every time a mock returns
another mock a fairy dies”
• Who said it?
• Not the guy whose tweet comes up first in
Google search results
• I am not 100% sure
• What the heck does it mean anyway?
• We don’t have time for it in this presentation
http://www.amazon.com/Growing-Object-Oriented-Software-Guided-Tests/dp/0321503627/
Creating Objects
•
•
•
•
Constructor injection (as in our code example)
Factory
What if dependencies have dependencies?
What if dependencies of dependencies have
dependencies?
Pretty Soon, Things Begin To Look Like This
(and this is a simple example)
(different box color = different life span)
Source: http://ayende.com/Blog/archive/2010/01/22/rejecting-dependency-injection-inversion.aspx
These Tools Can Help
•
•
•
•
•
StructureMap
Castle Windsor
Unity
NInject
autofac
Conclusions
Quality: More Important Than This
http://www.flickr.com/photos/markhillary/2681349855/
More Conclusions
•
•
•
•
•
•
•
•
You have to test
No single testing strategy is nearly enough
Employ multiple strategies
Each of them is difficult to master
We barely scratched the surface today
Agile brings testing into focus
Combine hard and soft skills
Collaborate
Q&A
http://creativecommons.org/licenses/by-nd/3.0/