Agile Testing
Agile Manifesto
1. people and communication over
processes and tools
2. working software over documentation
3. customer collaboration over contract
negotiation
4. responding to change over following a
plan
Roles and activities on agile team
"Testers have a foot in each world, understanding the customer
viewpoint as well as the complexities of the technical
implementation."
Customer Team
● business experts
● product owners
● domain experts
● product managers
● business analysts
● subject matter
experts
● testers
Developer Team
● programmers
● system
administrators
● architects
● security specialists
● database
administrators
● technical writers
● testers
"Testing"
●
●
●
●
●
●
●
●
●
●
●
Unit and component testing
Functional testing
System testing
Load testing
Performance testing
Security testing
Stress testing
Usability testing
Exploratory testing
End-to-end testing
User acceptance testing
Ten principles for Agile
Testers
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Provide continuous feedback
Deliver value to the customer
Enable face-to-face communication
Have courage
Keep it simple
Practice continuous improvement
Respond to change
Self-organize
Focus on people
Enjoy
What's the difference...?
Traditional testing:
● long iterations, "mini-waterfall"-trap
● testers are not involved until release planning
● testers have requirement documentation, they write long
test plans, wait for
● the developers and then execute tests -> "fast and
furious testing"
● if anything goes wrong it causes rushed testing phase
or/and release is
● usually postponed
● traditionally testers are considered as gatekeepers of
quality -> but they
● have no control over code quality or developer testing
● separated QA team
What's the difference...?
Agile testing:
● short iterations, frequent, small releases
● testers are involved as soon as project planning begins
● testers have access to all knowledge and help from the
team they need,
● they can focus on big picture and small tasks
simultaneously
● if anything goes wrong it can be discovered and fixed in
time
● testable design and code
● team understands that quality is more important than
quantity or speed
● independent, but not separated QA team (everyone is a
tester)
So,
What's the difference...?
"One of the biggest differences in agile development versus
traditional development is the agile 'whole-team' approach.
With agile, it's not only the testers or a quality assurance
team who feel responsible for quality."
● no more quality police mentality - quality is not only the
testers' responsibility
● whole-team approach; in agile teams everyone gets testinfected -> code is designed for testability
● tester team is integrated, there's no "QA-time" (no time to
cut when deadline is close, no rushed QA-phase)
● every team member has equal value to the team
● self-organizing team, continuous improvement
Agile testing Quadrants
Quadrant 1: this represents
test-driven development,
which is a core agile
development practice
Quadrant 2: business-facing
tests, these define external
quality and the features that
the customer wants
Quadrant 3: business-facing
tests to critique the product
(manual)
Quadrant 4: technology-facing
tests
Quadrant 1 Test-driven development TDD
● Unit tests
● Component tests
● internal quality isn't negotiated with the customer, it's
defined by the programmers
● programmer tests make lots of testing activities easier to
accomplish
● TDD means programmers are always consciously
making testable code and it provides immediate feedback
● each unit test is independent and tests one dimension at
a time -> when a unit test fails, it's easy to identify where
and what to fix
● TDD is a design activity, not only testing
Quadrant 2 - External quality
●
●
●
●
●
Functional tests
Examples
Story tests
Prototypes
Simulations
● business-facing tests address business requirements based on
examples
● quadrant 2 tests define and verify external quality
● these are written for each story before coding starts, they help the
team
● understand what code to write
● generally written in executable format, automated, so that team
members
● can run the tests as often as they like
● these become later automated regression tests as well
Quadrant 3 tests that critique the product
●
●
●
●
●
●
●
●
●
●
●
●
●
Exploratory testing
Scenarios
Usability testing
UAT (User Acceptance Testing)
Alpha/Beta testing
critiquing or evaluating the product
difficult to automate because these are based on human intellect,
experience and instinct
this manual testing is not enough to produce high-quality software (without
automation)
trying to recreate actual experiences of the end users
"soap opera testing" - "what's the worst thing that can happen, and how did
it happen?"
Quadrant 4 - Performance, Security
and “-ility” testing
●
●
●
●
●
●
●
●
●
Security
Maintainability
Interoperability
Compatibility
Reliability
Installability
technology-facing tests that critique the product
nonfunctional requirements, configuratiion issues
many of these tasks require specialized knowledge
Why are all of these important?
● help to get emerge technical debt (shortcuts,
quick fixes and skipped automated tests
making the code harder to maintain)
● responsibility is shared between the
quadrants
● quality is not only the testers responsibility
● "When developers don't have an automated
suite of tests acting as a safety net, they
may start viewing the testers themselves as
a safety net."
Why manual testing isn't enough
●
●
●
●
●
●
●
it gets boring very quickly, it's repetitive, so it's easy to overlook simple
bugs
manually testing a number of different scenarios can take a lot of time
especially if tester is keying inputs into a UI -> only a limited number of
scenarios may be tested, and important defects can be missed
automated tests can help with consistency across the application
automated regression tests are the safety net, not the QA person; without
them manual regression testing will grow in scope and eventually may
simply be ignored
automated test frees the QA team for more important work, such as
exploratory testing
"when post-development testing time is occupied with finding and fixing
bugs that could have been detected by programmer tests, there's no time
to find the serious issues"
How to be an agile tester
while release planning
● avoid big design up front, but understand it
correctly
● "create" high-level test plan
● try to identify risk, impact
● think about 'ripple effects' as well
● help correctly sizing stories with correct
questions
"Intuition is something that a
machine cannot learn."
●
●
●
●
●
the tester needs to keep the big picture in mind to create test cases so they
can ask uncomfortable questions
○ how much effort does it require to create a test environment?
○ what's the worst scenario that can happen?
○ what if the end user doesn't understand the application?
○ what's the difference between test and production environment? how
will it affect the performance?
○ what kind of test data should we use and where do we get it?
○ etc.
they always have more different viewpoints (developer-customer team)
they can identify 'thin slice' and 'critical path' to help prioritize stories
they give quick feedback to developers, helping them to create quality
product
UX/UI testing is the most important after quality code -> "if a user has a
choice of applications or websites, and has a bad experience, they likely
won't user your application again"
How to be an agile tester
while story sizing:
● everyone's goal is to deliver real value in each iteration > help identify one path through the functionality to code
and test first, then help adding more features after the
first critical path works
● identify 'small testable chunks' -> testable doesn't
necessarily means GUI
● always think about the big picture, know how each story
affects the whole system (if there's a bug in the
inventory code, what's the worse thing that can
happen?) -> help identify impact
How to be an agile tester
while test planning:
● the biggest benefit of test planning is planning itself
● wait for iteration planning to create detailed tests
● consider automation and what is needed for test
environment
● test planning is a risk mitigation strategy
● consider test infrastructure
● collect test data, real examples and real use-cases
● choose the right testing method, considering:
○ what's in scope
○ what are the impacts, is there any unusual risk
○ do you need test reports (for the customer team? for
your team?)
Test plan alternatives
Lightweight plans
● plan should not cover every
eventuality or every story
● not meant to address traceability
● it's a tool to help thinking about
testing risks
Test Matrix
● it's a list of functionality down the
side and test conditions across the top
● high-level test matrix -> can be used
to show to customer team what has
been tested already and what's left
● more detailed matrix -> can be used
to show to developer team what's
planned for testing and track the
progress
Test plan alternatives
Test spreadsheet
● it's more like a functionality list and
impact analysis in a spreadsheet format,
similar to test matrix but on more tabs
Whiteboard
● sometimes it's enough to list the risks
and assumptions on a whiteboard or
index cards or a wiki page
Automated test list
● provided by tools (at EU Edge:
TestLink)
● it's more like a detailed test plan
● it's only available after test cases are
written
● no added value
How to be an agile tester
while iteration planning:
●
●
●
●
●
●
●
●
●
●
think about how you can test the stories
help the programmers to design the application to be effectively testable
"when testability is an issue, make it the team's problem to solve"
work closely with the customers
think about 'big picture' and detailed tests as well
high-level tests should cover the main functionality behind the story
think both about the desired and undesired behaviour
'What's important as you begin the iteration is that you quickly learn the
basic requirement for each story and express them in context in a way that
works for the whole team."
review the test cases with the customers and developers as well
well-written tests form the core of the application's documentation
How to be an agile tester
while development:
● start simple -> create a happy path to show the core
functionality
● add complexity -> as soon as happy path works start
adding more test cases
● remember the purpose of the tests: they should provide
examples that tell the programmers what code to write
● use risk analysis to help prioritize testing
● identify variations
● use the 'power of three' (disagreement -> programmer +
QA person + Product Owner)
● focus on one story, try to get it 'done'
How to be an agile tester
while development:
● as soon as testable chunks are ready take time to
explore functionality
● if you find 'nice-to-have' stories consult with the
customer, if there's time to add it in the iteration and it
has business value, go ahead (these are cheaper now),
but don't jeopardize other stories by adding 'blind' that
doesn't have big ROI (ROI = Return on investment)
● technology-facing tests to critique the product ("-ility
testing") are often best done during coding
● if the team doesn't have automated tests, everyone
should plan time to address manual regression tests
and manually testing new features
How to be an agile tester
while dealing with bugs:
● bug or feature? -> in the end, does it really matter if it is
a bug or a feature if it needs to be fixed?
● bugs -> technical debt: the longer a bug stays in the
system and goes undetected, the greater is the impact
● sometimes a bug is really a missed requirement and
needs to be handled as a story; estimated and
prioritized for a future iteration
● set rules like "the number of bugs should never get
higher than ten at any one time"
How to be an agile tester
while delivery:
●
●
●
●
●
●
●
●
the goal is to deliver value to the business in a timely manner
plan as much time for the end game as you need (it varies with the
maturity of the team and the size of the application)
use this end-game time to do a final exploratory testing
step back and look at the whole system; do some end-to-end scenarios
find a way to guarantee that all changes made in the test environments will
be done in the staging and production environments during release
'customers have a habit of using the application in weird and wonderful
ways, and the data is not always as clean as we would like' - everyting
needs to be tested again with real data in a real environment
try to avoid high risk -> release in smaller stages or use a system property
to turn on new features and use the old as long as the new one is safe
enough
communicate, have reminders
Summary
Success factor 1: Whole-team approach
●
●
●
●
you have a large variety of skill sets and experience levels
when testing is a team priority, the team designs testable code
remember that quality, not speed, is the goal of agile development
use the "power of three"
Success factor 2: Adopt an agile testing mindset
●
●
●
●
●
●
"Ten principles for agile testers", no more "Quality police mentality"
be courageous
focus on delivering value
be flexible in responding to change
drive to continually find better ways to work
experiment with new practices, tools, and techniques
Summary
Success factor 3: Automate regression testing
●
●
●
if you spend all your time doing manual regression testing, you'll never
have time for the important exploratory testing
remember to start simply
Success factor 4: Provide and obtain feedback
●
●
●
feedback is a core agile value
the short iterations are designed to provide constant feedback
one of the most valuable skills to learn is how to ask for feedback
○ ask the programmers if they get enough information
○ ask the customers if they feel their quality criteria are being met
Summary
Success factor 5: Build a foundation of core practices
●
●
●
●
●
"you can't test quality into the product"
every agile team needs source code management and continuous
integration -> you can't test effectively if you don't know exactly what you're
testing
you can't test effectively without a test environment that you can control
manage technical debt; don't let time pressure get you into hacking, or
"quickfixing" - good test coverage, automated regression tests can help
use small-scaled stories -> develop and test step-by-step to keep quality
"Agile practices were designed to complement each other. Take time to
understand the purpose of each one, consider what is needed to take full
advantage of each practice, and make thoughtful decisions about what works
for your team.”
Summary
Success factor 6: Collaborate with customers
●
●
●
use the "Power of three"
encourage direct communication between customers and developers
help customers clarify requirements, desired behaviour with concrete
examples
Success factor 7: Look at the big picture
●
●
●
●
testers tend to look at the big picture usually from a customer point of view
developers have to focus on the story they're working at
use agile testing quadrants as a guide to help you plan testing that will
cover all angles
make your test environments as similar as possible to production (test data
as well)