Testing the BBC News Website
Mark Micallef
The timeline…
Aug 09
Aug 10
Nov
Feb
May
The timeline…
Aug 09
Aug 10
Nov
Feb
Winter
Olympics
May
UK
Elections
FIFA
World Cup
Starting Point
Product
Manager
Spec
Dev Team
Project
Manager
Tester
“Use the project as a platform
for transitioning to Agile”
The timeline…
Project Setup
Stabilization
Delivery Mode
Post-Launch
Aug 09
Aug 10
Nov
Feb
Winter
Olympics
May
UK
Elections
FIFA
World Cup
Risk mitigation
Aug 09
Rel A
Nov
Rel B
Feb
Winter
Olympics
Rel C Aug 10
May
UK
Elections
FIFA
World Cup
On launch day…
• Developers in at 4:30am
• Very calm launch
• No real major public facing
issues
• Some internal issues
(mostly integration issues
with downstream systems)
• Ongoing public feedback
Let’s talk process
Remember this?
Spec
Scope
Project
Teams
Core
Output Indexes
Output Stories
Site
Building
Project Testers
16
14
Number of testers
12
10
8
6
4
2
0
Initial Process
•
•
•
•
•
•
Agile
Extensive automated testing
TDD
1 week iterations
Daily standups
Weekly Showcases
Process for each user story
Identified
In
Analysis
Ready for
Dev
In Dev
Ready for
QA
In QA
Ready for
Sign Off
Signed
Off
Initial Process
•
•
•
•
•
•
Agile
Extensive automated testing
TDD
1 week iterations
Daily standups
Weekly Showcases
Initial Process
•
•
•
•
•
•
•
•
•
Agile
Extensive automated testing
TDD
1 week iterations
Daily standups
Weekly Showcases
Retrospectives
Twice-weekly QA meetings
Weekly Management Meeting
Process for each user story
Identified
In
Analysis
Ready for
Dev
Review
Acceptance
Criteria
In Dev
Prepare
Cucumber
Feature
Files
Ready for
QA
Dev
Machine
Demo
In QA
Test / Bug
Fix Cycle
Ready for
Sign Off
Signed
Off
Demo to
Product
Manager
Process Change Funnel
Test Automation
Technologies
Technologies
Automated Testing Technology Stack
Analysts
Cucumber
Ruby Models
White
Engineers
Selenium
Grid
W3C
Validator
DB
Libraries
XML
Parsers
JMeter
Cucumber Example
@smoketest
@editions
Scenario: International user accessing the news website
Given I am a user outside the UK
When I visit the News website
Then I should be redirected to the International Edition
And I should see adverts on the page
Given /^I am a user outside the UK$/ do
pending
end
When /^I visit the news website$/ do
pending
end
Then /^I should be redirected to the international
edition$/ do
pending
end
Then /^I should see adverts on the page$/ do
pending
end
Given /^I am a user outside the UK$/ do
@helpers.set_proxy(INTERNATIONAL)
end
When /^I visit the news website$/ do
@browser.open(“/”)
end
Then /^I should be redirected to the international
edition$/ do
assert_equal(true, @page.is_international?)
end
Then /^I should see adverts on the page$/ do
assert_equal(true, @page.ads_present?)
end
Test Environments
Test
(Manual Exploratory Tests)
Stage
Int
(NFR + Config Testing)
(Dev & Unit Testing)
Test XE
(Automated Acceptance Tests)
Test Data Management
Republish
Create Test
Data
QA
Workbench
Test Data
Copied
Over
Test XE
Execute
Automated
Tests
Communication and Measurement
Communication Tools
•
•
•
•
•
Daily standups
Weekly Showcases
Retrospectives
Walls and Boards
Seating arrangements
Bug Management
Bug Management
Signoff Velocity
David McCandless on Visualisation (TED)
Thank you…