Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com
@paul_gerrard
Story-Based Test Automation
Using Free Tools
Overview
• Behaviour-Driven Development and similar
approaches are still developer-focused
– Don’t engage testers as they should
– Don’t support higher complexity requirements
– Don’t scale to a larger requirements bases
• In this session, we’ll cover:
– Acceptance-driven development with free tools
– How Story Platform for QA (a sub-set of Business
Story Manager) and Robot Framework can be used to
implement this approach.
Intelligent Definition and Assurance
Slide 2
Background
• We won’t go into much detail of BehaviourDriven Development
– http://en.wikipedia.org/wiki/Behaviordriven_development for an intro
• Requirements-test hierarchies:
– Go to http://businessstorymanager.com
– and watch the videos.
Intelligent Definition and Assurance
Slide 3
Behaviour-Driven Development
using current tools
Test Fixtures/
Test Step Defs.
+
Story
(structured)
BDD
Tool
Generated
Test Code
Test Outcomes
Test Tool
executes tests
System Under
Test
Write the Code
+ Refactor the
Code
Intelligent Definition and Assurance
Slide 4
BDD and tool support
• A big benefit to developers is that BDD makes
the Test-Driven Development approach easier
• Behaviour-Driven Development:
– A collaborative approach to specification
– Uses Domain–Specific Language in specifications
– Implements Ubiquitous Language
• Currently, tools only support the automated
testing part of the BDD regime.
Intelligent Definition and Assurance
Slide 5
Story Platform QA (BSM subset)
Requirements
Dictionary
Robot Framework
Keyword
Definitions
+
Story
(structured)
Test
Export
Test Status
Red
Amber
Green
Test Cases
Test Library
e.g. Webdriver
executes tests
Intelligent Definition and Assurance
Test Outcomes
System Under
Test
Write the Code
+ Refactor the
Code
Slide 6
Business Story Manager and Test Frameworks
Business Story Manager
STORY: Feature Header
Business Scenario
Requirement
Given
When
And
Then
book”
“Search page”
“I find a book”
“I add to basket”
“basket has my
Just enough to clarify understanding
Test Scenario
Pre-conditions “Search page”
Steps
“enter search text”
“click search button”
“scroll down to find book”
“select book”
“view more information to
verify”
“select add to basket”
Post Conditions/outcomes “Basket has selected book”
Manually maintained
Test
Config.
Technical Code
Per Environment
Test
Log
Generated
Keyword
Definitions
1 per application
Keyword
Definitions
1 per Feature
Story Based
Test Cases
Example: Robot Framework
Drivers
Selenium (Web)
HTTP (web)
SSH
System(s) Under Test
DB Library
Etc.
How Does it Work?
A walkthrough of the process using
http://sp.qa
The detail for the requirements
selected on the left is shown
here.
The feature identifies a
capability of the system
described in the requirement
Intelligent Definition and Assurance
Slide 10
The scenario provides an
illustration of a feature in use.
Examples provide broader
coverage of the feature.
Intelligent Definition and Assurance
Slide 11
NB: The colour coding
highlights terms defined in the
dictionary, and data items in
scenarios. (The dictionary is
available in the full product).
This test scenario has been
edited to provide more detail.
These are the automated test
keywords.
The Lucky Numbers
Application
Lucky numbers features:
• Registration
• Submit ticket
• View my Game Plays
• System Status
• Change my profile
• Reset Password
• etc.
The numbers are submitted and
validated and then the user is
asked to confirm the ‘play’.
Intelligent Definition and Assurance
Slide 14
Intelligent Definition and Assurance
Slide 15
Exporting Test Scenarios
We’ve shown just one test
scenario here for illustration.
Normally, all test scenarios are
exported.
Dropbox asks for confirmation
that you want to post the test
case files to your account.
Intelligent Definition and Assurance
Slide 18
BSM lists all the files that have
been posted to your Dropbox
account.
Intelligent Definition and Assurance
Slide 19
Generated files and resource files
The files
generated
BSM
These
resource
files by
correspond
have
beengenerated
synchronised
to your
testwith
case
your
account
files.Dropbox
You maintain
your
keyword definitions here.
Intelligent Definition and Assurance
Slide 20
Robot script and resource file
The Robot Script files with test
cases are generated by Business
Story Manager
The tester maintains the
resource file with the keyword
definitions.
Intelligent Definition and Assurance
Slide 21
Tests running on a Linux VM that
shares Dropbox
Terminal window
driving the tests
System Under Test
…and the same test on Windows 7
Intelligent Definition and Assurance
Slide 23
Intelligent Definition and Assurance
Slide 24
Results upload service
Use a batch file or shell script to
run the tests, then upload the
results to BSM/SP.QA
loadRobotOutput.py Python
uploads the output file
Intelligent Definition and Assurance
Slide 25
Intelligent Definition and Assurance
Slide 26
Business Story Manager
Requirements
Robot Framework
Dictionary
Keyword
Definitions
+
Story
(structured)
Test
Export
Test Status
Red
Amber
Green
Test Cases
Test Library
e.g. Webdriver
executes tests
Intelligent Definition and Assurance
Test Outcomes
System Under
Test
Write the Code
+ Refactor the
Code
Slide 27
Behaviour-Driven Tooling
• The essential architecture of BDD tools is that
1. A tool parses stories and generates test code
2. The test code can be used for TDD
3. The developer (almost always) writes the step
definitions/fixture code and runs the tests
4. The stories, test code and steps are all sourcecontrolled (in the developers’ library)
• What we have done with SP.QA and Robot is
allow non-techies to own this process and align
it with stakeholder needs properly.
Intelligent Definition and Assurance
Slide 28
Summary
• SPQA and Robot Framework combined:
– Support requirements of any complexity
– Implement keyword-driven test automation
– Manage requirements, stories and tests to any scale
• Acceptance-driven approach:
– Traceable from requirements to tests and back
– Automated story acceptance and regression testing
– Build automated end-to-end acceptance tests with
reusable keywords.
Intelligent Definition and Assurance
Slide 29
http://sp.qa
Story Platform for QA
SP.QA provides a subset of Business Story
Manager functionality and is FREE-TO-USE.
Visit SP.QA to access the free
Story Platform for QA web
service.
Go to the Image Gallery to see
the requirements-test process
in pictures
SP.QA Scope
• What you get
– Requirements module
– Business Stories and Scenarios
– Test Scenarios
– Robot Framework Interface
• FREE Service Limits
– Limited to 100 business stories
– 5 users, 1 application, 1 project
BSM Scope
• What you get
–
–
–
–
–
Stakeholder and Risk Management modules
Lean Task Management module
Dictionary and Library modules
Role-Based Access Control
All SP.QA Scope
• Limitations
– None
• Commercials
– £30 per user per month
Intelligent Definition and Assurance
Slide 34
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com
@paul_gerrard
Story-Based Test Automation
Using Free Tools