Specification By Example Declan Whelan @dwhelan What’s the Problem? Waterfall Source: http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf Waterfall X Source: http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/waterfall.pdf Cost Of Change Cost% Of% Change% Time% AgileAgile&Approach& Specification Cost% of% Change% Itera/on&1& Itera/on&2& Itera/on&3& Itera/on&4& Requirements& Requirements& Requirements& Requirements& Analysis&&&Design& Analysis&&&Design& Analysis&&&Design& Analysis&&&Design& Code& Code& Code& Code& Test& Test& Test& Test& Deploy& Time% Traditional Flow Traditional Flow Specification Traditional Flow Build Specification Traditional Flow Build Specification Tests Cases Traditional Flow Build Specification Test Tests Cases Traditional Flow Build Specification Test Tests Cases Deploy We Found a Bug! How did QA miss this? Developers don't test before they throw it over the wall! That was obvious! I shouldn't have to spell out every detail! Exec Tester Product Manager We did what was in functional spec! Business requirements weren't clear! BA missed it in the technical requirements! Dev BA Architect We Found a Bug! How did QA miss this? Developers don't test before they throw it over the wall! That was obvious! I shouldn't have to spell out every detail! Exec Tester Product Manager We did what was in functional spec! Business requirements weren't clear! BA missed it in the technical requirements! Dev BA Architect We Found a Bug! How did QA miss this? Developers don't test before they throw it over the wall! That was obvious! I shouldn't have to spell out every detail! Exec Tester Product Manager We did what was in functional spec! Business requirements weren't clear! BA missed it in the technical requirements! Dev BA Architect We Found a Bug! How did QA miss this? Developers don't test before they throw it over the wall! That was obvious! I shouldn't have to spell out every detail! Exec Tester Product Manager We did what was in functional spec! Business requirements weren't clear! BA missed it in the technical requirements! Dev BA Architect We Found a Bug! How did QA miss this? Developers don't test before they throw it over the wall! That was obvious! I shouldn't have to spell out every detail! Exec Tester Product Manager We did what was in functional spec! Business requirements weren't clear! BA missed it in the technical requirements! Dev BA Architect We Found a Bug! How did QA miss this? Developers don't test before they throw it over the wall! That was obvious! I shouldn't have to spell out every detail! Exec Tester Product Manager We did what was in functional spec! Business requirements weren't clear! BA missed it in the technical requirements! Dev BA Architect We Found a Bug! How did QA miss this? Developers don't test before they throw it over the wall! That was obvious! I shouldn't have to spell out every detail! Exec Tester Product Manager We did what was in functional spec! Business requirements weren't clear! BA missed it in the technical requirements! Dev BA Architect Accidental Adversaries + Testing + New Bugs Testing Success + - + + Additional Tests Development Success Software Fix - + New Build + Development + Functional Silos Source: http://www.danpontefract.com/images/silo.jpg Functional Silos X Source: http://www.danpontefract.com/images/silo.jpg Build it Right Build the Right Thing Specification By Example Gojko Adzic, 2011 page 4 Build it Right Build the Right Thing Useless Crap Specification By Example Gojko Adzic, 2011 page 4 Build it Right Business Failure Build the Right Thing Useless Crap Specification By Example Gojko Adzic, 2011 page 4 Build it Right Business Failure Build the Right Thing Useless Crap Specification By Example Gojko Adzic, 2011 page 4 Maintenance Nightmare Build it Right Business Failure Business Success Build the Right Thing Useless Crap Specification By Example Gojko Adzic, 2011 page 4 Maintenance Nightmare Build it Right Business Failure Business Success Specification By Example Build the Right Thing Useless Crap Specification By Example Gojko Adzic, 2011 page 4 Maintenance Nightmare What are Specifications By Example? What are Specifications By Example? • Thin slices of system behaviour • that deliver business value • described as concrete examples • that are potentially automatable • without translation • to create executable specifications • captured in live documentation. Agile Testing Quadrants Specification By Example Specification By Example Business Goal Specification By Example Business Goal Derive the scope Scope Specification By Example Business Goal Derive the scope Scope Specify collaboratively Key Examples Specification By Example Business Goal Derive the scope Scope Specify collaboratively Key Examples Refine the specification Specification With Examples Specification By Example Business Goal Derive the scope Scope Specify collaboratively Key Examples Refine the specification Specification With Examples Automate literally Executable Specification Specification By Example Business Goal Derive the scope Scope Specify collaboratively Key Examples Refine the specification Specification With Examples Automate literally Executable Specification Validate frequently Living Documentation Specification By Example Business Goal Derive the scope Specify collaboratively Key Examples Refine the specification Specification With Examples Automate literally Executable Specification Validate frequently Living Documentation Shared Understanding Ubiquitous Language Scope Business Goal Derive the scope Scope Specify collaboratively Key Examples Refine the specification Specification With Examples Automate literally Executable Specification Validate frequently Living Documentation Shared Understanding Ubiquitous Language Source: https://docs.google.com/drawings/d/1cbfKq-KazcbMVCnRfih6zMSDBdtf90KviV7l2oxGyWM/edit Specification By Example Derive the Scope: Story Mapping Source: http://availagility.co.uk/wp-content/uploads/2008/10/user-story-mapping.png Derive the Scope: User Stories Derive the Scope: User Stories As a _______ I want to _______ So that _______ Derive the Scope: User Stories As a _______ I want to _______ So that _______ As a student I want to purchase used books online So that I can save money Specify Collaboratively: Workshops Specify Collaboratively: Workshops Specify Collaboratively: Workshops • Hold regular product backlog workshops Specify Collaboratively: Workshops • Hold regular product backlog workshops • Full team workshops - when starting Specify Collaboratively: Workshops • Hold regular product backlog workshops • Full team workshops - when starting • Three amigo workshops: • One developer • One tester • One analyst Specify Collaboratively: Key Examples Specify Collaboratively: Key Examples Given _______ When _______ Then _______ Specify Collaboratively: Key Examples Given _______ When _______ Then _______ Given “War and Peace” is available as a used book for $2.99 When Susan selects book“War and Peace” Then “Buy used for $2.99” is displayed Refining the Specification Refining the Specification “Specifications with examples are acceptance tests” Gojko Adzic Refining the Specification “Specifications with examples are acceptance tests” Gojko Adzic • Be precise and make sure spec is testable Refining the Specification “Specifications with examples are acceptance tests” Gojko Adzic • Be precise and make sure spec is testable • Avoid “scripts” and “flows” Refining the Specification “Specifications with examples are acceptance tests” Gojko Adzic • Be precise and make sure spec is testable • Avoid “scripts” and “flows” • Focus on business functionality not design Refining the Specification “Specifications with examples are acceptance tests” Gojko Adzic • Be precise and make sure spec is testable • Avoid “scripts” and “flows” • Focus on business functionality not design • Avoid UI details Refining the Specification “Specifications with examples are acceptance tests” Gojko Adzic • Be precise and make sure spec is testable • Avoid “scripts” and “flows” • Focus on business functionality not design • Avoid UI details • Avoid covering every possible combination Refining the Specification: An Example Free Delivery Free delivery is offered to VIP customers once they purchase a certain number of books. Free delivery is not offered to regular customers or VIP customers buying anything other than books. Customer Type Cart Contents Delivery VIP 5 books Free, Standard VIP 4 books Regular 10 books Standard Standard VIP 5 dishwashers Standard VIP 5 books, 1 dishwasher Standard Source: Specification by Example: How successful teams deliver the right software, Gojko Adzic, pg. 116 Automating Examples Automating Examples • Start small Automating Examples • Start small • Select important examples for automation Automating Examples • Start small • Select important examples for automation • Plan up-front to automate Automating Examples • Start small • Select important examples for automation • Plan up-front to automate • Be prepared to go slower at the start Automating Examples • Start small • Select important examples for automation • Plan up-front to automate • Be prepared to go slower at the start • Treat automation code as a first class citizen Automating Examples • Start small • Select important examples for automation • Plan up-front to automate • Be prepared to go slower at the start • Treat automation code as a first class citizen • Avoid record and playback Automating Examples • Start small • Select important examples for automation • Plan up-front to automate • Be prepared to go slower at the start • Treat automation code as a first class citizen • Avoid record and playback • Avoid using pre-populated data Minimize Tests Through the UI Automation Tools FitNesse Cucumber Concordian Green Pepper JBehave Twist Text Test Robot Framework SpecFlow Validate Frequently Validate Frequently • Start with a Continuous Integration system Validate Frequently • Start with a Continuous Integration system • Set up a Continuous Deployment system Validate Frequently • Start with a Continuous Integration system • Set up a Continuous Deployment system • Specify and test business logic separately from end-to-end flows Validate Frequently • Start with a Continuous Integration system • Set up a Continuous Deployment system • Specify and test business logic separately from end-to-end flows • Organize tests along functional lines Validate Frequently • Start with a Continuous Integration system • Set up a Continuous Deployment system • Specify and test business logic separately from end-to-end flows • Organize tests along functional lines • Run all test nightly Validate Frequently • Start with a Continuous Integration system • Set up a Continuous Deployment system • Specify and test business logic separately from end-to-end flows • Organize tests along functional lines • Run all test nightly • Consider an iteration “test pack” Living Documentation • Keep specifications short • Evolve a specification language and leverage in with “common fixtures” • Make documentation accessible - consider a wiki • Organize the documentation • Put specifications under version control A Worked Example Executable Specification Cucumber Organization fixtures are common Features Step System Cucumber Organization Lo ad fixtures are common Features Step System Cucumber Organization Lo ch at Features M ad fixtures are common Step System Cucumber Organization Lo ch at Features M ad fixtures are common Step Call System Feature File Feature: Turn cucumber into beer As a cucumber presenter I want beer after my presentation So I can enjoy the rest of DemoCampGuelph Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Scenario: Ali buys Declan beer Given Ali hosts DemoCampGuelph When Declan demos Cucumber Then Ali should buy Declan 1 beer Step Definitions Given /^(.+) hosts/ do |host| @event = Event.new(host) end When /^(.+) demos/ do |presenter| @event.add(presenter) end Then /^(.+) should buy (.+) (\d+) (.*)$/ do |buyer, drinker, qty, item| perk = @event.perks[0]; perk.buyer.should == buyer; perk.receiver.should == drinker perk.quantity.should == quantity.to_i; perk.item.should == item end System Under Test class Event attr_reader :perks def initialize(host) @host = host; @perks = [] end def add(presenter) @perks.push Perk.new(@host, presenter, 1, "beer") end end class Perk attr_reader :buyer, :receiver, :quantity, :item def initialize(buyer, receiver, quantity, item) @buyer = buyer; @receiver = receiver @quantity = quantity; @item = item end end Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Given /^(.+) hosts/ do | When /^(.+) demos/ do | host| presenter| @event = @event.add(presenter) Event.new(host) end End Then /^(.+) should buy (.+) (\d+) (.*)$/ do |buyer, drinker, qty, item| perk = @event.perks[0]; perk.buyer.should == buyer; perk.receiver.should == drinker Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Given /^(.+) hosts/ do | host| @event = Event.new(host) End Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Given /^(.+) hosts/ do | host| @event = Event.new(host) End “Brydon” Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Given /^(.+) hosts/ do | When /^(.+) demos/ do | host| presenter| @event = @event.add(presenter) Event.new(host) end End Then /^(.+) should buy (.+) (\d+) (.*)$/ do |buyer, drinker, qty, item| perk = @event.perks[0]; perk.buyer.should == buyer; perk.receiver.should == drinker Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer When /^(.+) demos/ do | presenter| @event.add(presenter) end Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer “Declan” When /^(.+) demos/ do | presenter| @event.add(presenter) end Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Given /^(.+) hosts/ do | When /^(.+) demos/ do | host| presenter| @event = @event.add(presenter) Event.new(host) end End Then /^(.+) should buy (.+) (\d+) (.*)$/ do |buyer, drinker, qty, item| perk = @event.perks[0]; perk.buyer.should == buyer; perk.receiver.should == drinker Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Then /^(.+) should buy (.+) (\d+) (.*)$/ do |buyer, drinker, qty, item| perk = @event.perks[0]; perk.buyer.should == buyer; perk.receiver.should == drinker Execution Scenario: Brydon buys Declan beer Given Brydon hosts DemoCampGuelph When Declan demos Cucumber Then Brydon should buy Declan 1 beer Then /^(.+) should buy (.+) (\d+) (.*)$/ do |buyer, drinker, qty, item| perk = @event.perks[0]; perk.buyer.should == buyer; perk.receiver.should == drinker Business Goal Derive the scope Scope Specify collaboratively Key Examples Refine the specification Specification With Examples Automate literally Executable Specification Validate frequently Living Documentation Shared Understanding Ubiquitous Language Source: https://docs.google.com/drawings/d/1cbfKq-KazcbMVCnRfih6zMSDBdtf90KviV7l2oxGyWM/edit Specification By Example Build it Right Build the Right Thing Specification By Example Gojko Adzic, 2011 page 4 Build it Right Build the Right Thing Useless Crap Specification By Example Gojko Adzic, 2011 page 4 Build it Right Business Failure Build the Right Thing Useless Crap Specification By Example Gojko Adzic, 2011 page 4 Build it Right Business Failure Build the Right Thing Useless Crap Specification By Example Gojko Adzic, 2011 page 4 Maintenance Nightmare Build it Right Business Failure Business Success Build the Right Thing Useless Crap Specification By Example Gojko Adzic, 2011 page 4 Maintenance Nightmare Build it Right Business Failure Business Success Specification By Example Build the Right Thing Useless Crap Specification By Example Gojko Adzic, 2011 page 4 Maintenance Nightmare Reading Specification By Example Gojko Adzic The RSpec Book: Behaviour Driven Development with RSpec, Cucumber and Friends David Cheliminksy et al Agile Testing: A Practical Guide for Testers and Agile Teams Lisa Crispin, Janet Gregory Diagram Credits Lisa Crispin and Janet Gregory Agile Testing: A Practical Guide for Testers and Agile Teams Addison-Wesley Professional; January 9, 2009. Mike Cohn Succeeding with Agile: Software Development Using Scrum Addison-Wesley Professional; November 5, 2009.
