Courtesy: Gulliver's Travels (2010)
Agile Testing @ Large
Geoff Meyer, geoffrey_meyer@dell.com
May 2014
Last updated: 25 February, 2014
Session Objectives
IS
• Challenges to expect with Agile test practices in
large-scale, multi-team projects
• Brief overview of Industry Approaches
• Real-world examples − from a large organization
perspective
• Organizational strategies & considerations
• Test automation considerations
IS
•
An In-depth Tools Discussion
NOT
•
Focused on how to develop Test Automation
Geoff Meyer
• Dell Inc, 1998 – present
– Test Architect
› Agile Test & Automation Strategy
– Agile Steering Committee Co-Chair
– Global Projects
› 15 Scrum teams (2 regions)
› 11 Scrum teams (4 regions)
› 9 scrum teams (2 regions)
› 7 Scrum teams (4 regions)
• NCR Corp. 1984 – 1998
– SW developer, Project Lead, SW Manager
• B.S. Computer Science, San Diego State University
• Masters Engineering Management - NTU
3
Agenda
• The Dell Context and Agile @
• Early Agile Testing Challenges
• In Search of… Solutions for Scale
• Adaptations for Testing at Scale
• Automating in an Agile world
• Lessons of Large
4
The Context at
Dell ESG
http://amazngwallpapers.blogspot.com
5
Dell Enterprise Solutions Group
Global Design and Development using Agile Scrum
Silicon Valley
Design Center
Austin
Design Center
6
Noida Design
Center
Bangalore
Design Center
Dell Enterprise Solutions
Software Products
• Server Systems Management
• Converged Infrastructure Systems Management
Compute
• Private Cloud Systems Management
• Console Plug-ins (i.e. for SCCM, vCenter…)
Common Product Characteristics:
Network
• Embedded Software
• Large hardware support test matrix
• Software is installed in the Data Center
Storage
• Enterprise update cycles ~6 months
• Products must function even as underlying HW, FW,
BIOS, and Drivers are upgraded - SUSTAINING
7
The Early
Challenges
?
http://cleaneatingchelsey.com/
8
Culture Transition
• Early commitment on Requirements
• Insufficient Detail in Requirements
• Development vs. Test
• A different “School” of Test1
• Fail Early, Inspect & Adapt
“ “Everybody, every team, every platform, every division,
every component is there not for individual competitive
profit or recognition, but for contribution to the system
as a whole”
W. Edwards Deming
1 – Scott Barber “Approaches to Software Testing: An Introduction “
9
Signs of Trouble
• Thursday’s Build
• Committing to more stories
than team’s sprint velocity
• High % of carry-overs
• Inability to Automate
Acceptance Tests within
Sprint
• “Engineering” Stories
http://www.huffingtonpost.com/2012/03/21/truck-stuck-thomascircle_n_1369545.html?p=truck+and+underpass&vm=r
• Cross-team interdependencies block sprint
progress
• Large HW configuration validation consumes
test resources
• No single team had visibility to the entire product
• Insufficient Product Owner bandwidth
Large-Scale
Test Automation Pitfalls
• Minimal guidance to Test beyond “Go forth and automate”
• Black Box Test Mentality
• Automation was interpreted by many as “Automate the UI”
• Insufficient SW engineering
skills across Test teams
• Architecture(s) not optimized
for Automatability
11
In Search of…
Solutions For
Scale
12
Leverage the Community
Books
Conferences
Professional Groups
Consultants
13
Scaled Agile Framework™ Big Picture
Agile Testing Quadrants
4
4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
15
Automation Tooling
• IDE
• UI Tools
• Scripting Languages
• Services
• Frameworks
16
ALM Tooling
• Multi-team, geographically distributed projects
create a necessity for ALM Tooling
–Requirements Management
–User Story Management
–Test Case Management
–Defect Management
–Test Script Repository
17
Agile @ Dell
Adaptations
18
Agile Testing @ PG Enterprise Solutions Group
How did we go about it?
Process/Project
Non-Functional
Scale, Longevity
In-sprint
Automation
X-Large
project (15)
UI Automation
Rally
2009
Test into
Scrum
Service-level
automation
2010
2011
2012
2013
Maximize
Utilization
Collaborative
Seating
“Automate-First”
19
Automated
Setup & Config
ALM using
TFS
Automation
Architects
Culture/Organization
4 Geography
project (11)
Accelerate
Automation
Skillset
2014
• Defect
Prevention
• Continuous
Testing
Agile @ Dell
Core Activities
• Pre-Sprint activities:
– Staffing
– Training - Project tools and processes
– CI/Build environment
– Automation Framework and BVT
• Establish Project-wide ‘Done’ criteria
• End-to-end, short duration User Stories
• Test Automation is included User Story acceptance
criteria
• Refresher Workshops for new projects
20
2 week Sprint Lifecycle Example
Week 1
Product Owner
(or Proxy)
Scrum Master
Development
Test
Tech Pubs
UI
Week 2
User
Story
Demo
Accept
Demo
User
Story
Sprint
Planning
User
Story
User
Story
Automated
Regression
Demo
Accept
Demo
Retro Sprint
spectiveReview
Accept
Accept
User
Story
Automated
Regression
Automated
Regression
Demo
Accept
Automated
Regression
Manual Regression
Automated
Regression
In-Sprint: Testing
• Design/Develop Test cases and Review with team
• Develop test automation and maintain in TFS
• Execute test cases (manual and automated)
• Immediate Feedback (IM, email, in-person) with team of
sightings… do not automatically enter defects
• Verify fixes for defects; re-test if needed
• Maintain test case status (in MTM)
• Cross-team coordination including Weekly SoS
22
Confidential
Agile @ Dell Roles/Responsibilities
Product
Owner
• Small teams
Product Owner Proxy
1 per Scrum
• Co-located
Scrum Master
1 per Scrum
Development
4-5 per Scrum
• Multi-disciplined
…
Test
• Feature-based
• Distributed Projects
3:1
Tech Pubs
• Scrum of Scrums
1 per 3 scrum team
• Teams formed from
functional silos
UI
1 per Scrum
Scrum teams
23
Release Manager
SW Architect
Test Architect
Automation Architect
Cross-team Test roles
• Test Architect
– Defines Test Strategy across all Scrum
teams
– Establishes non-Functional Test
approach
– Influences consistent Application
Architectures
• Automation Architect
– Leads Analysis and POC activities for
new automation needs
– Ensures automation consistency for
BVT and nightly regression suites
– Safeguards against automation tool
proliferation
24
EXAMPLE - Project-wide “Done” criteria
Code
– Junit Tests
– SONAR
– Code Reviews completed
– All User Story tasks completed
– Demo and accepted by Product
Owner
– All known error conditions caught
and handled appropriately
– Code is Globalized
Test
– Acceptance Tests Reviewed with
scrum team
– Acceptance Tests documented in
Tool
– All planned Acceptance Tests for
User Story has been attempted
– Acceptance Tests executed
against Test system configuration
– Defects that are “must fix” before
releasing must be fixed prior to
story acceptance
– Service-level automated tests
– Service-level automated Tests are
globalized
25
Large-Scale Complexities
Characteristic
Description
Large-Scale
Project consists of more
than 4 Scrum teams
Interdependency
Requirements
implemented across
multiple Scrum teams
Extensive
Configuration
Matrix
Extensive HW or SW
configurations
Waterfall
Intersection
One or more
components are
managed using Waterfall
Geography
Project members are
Geographically dispersed
26
: Agile with Adaptations
Agile @
Usability
Wireframes
Pre-sprints
Development
Architecture
Release Planning
Pre-sprints
HW resources
Test
Automation FW/Tools
Pre-sprints
HW resources
1
1
2
2
1
2
3
N-1
3
N-1
…
… Sprints
3
N-1
N
N
Hardening
Stability
N
Extended Sprint Test
Software System Test
OLP
Define
27
Plan
Develop
Launch
Extended Sprint Test
Extended Hardware Configurations
• Optimizes HW utilization and cost in
support of Scrum teams
• Owns Test Execution against extended
configurations
• Scrum team:
Configurations
Positive
X
Negative
X
X
X
– Establishes Acceptance criteria
Boundary
X
– Identifies all Test scenarios
Stress
X
Scalability
X
– Owns Test Execution against reference
configuration(s)
Concurrency
X
Globalization
X
X
X
X
X
X
X
X
X
X
X
– Develops in-sprint automation tests
Candidate for
Extended Sprint Test
28
X
Software System Test
“Alpha Customer”
IS
Requirements-based
Functional testing
Customer-usage based
First time integration test of workflows
Interoperability coverage of HCL in
specific customer environments
Exhaustive interoperability matrix
coverage of the HCL
Three phased approach
Performance baseline testing
Pre-SST
•
•
•
•
29
IS NOT
SST
Hardening
Workflow Design • Test Automation • Test Execution
Test Analysis
Development
• Regression Test
Test Design
• Test Execution
Test Development • Regression Test
SAFe1 &
System team
Agile @
Scrum teams
30
1- Courtesy of Dean Leffingwell and Al Goerner of Valtech
Agile & Waterfall Converged Model
100% PT Attempted
All Stories Accepted CCB Review
Feature Complete Code Freeze
X-on-A Stories
Accepted
SW
Regression
Test
Sprint1
Sprint2
Sprint1
Sprint2
Sprint1
…
…
Stability
Sprint
Sprint
N-1
Sprint
N-1
Sprint
N-1
Sprint2
…
SprintN
SprintN
Regression
EST Exit
Hardening
SST
HW-Specific User Stories Complete
HW Enablement Features Complete
Vendor Deliverables Required
Dell Confidential
A-Rev
Final
Extended
SSI
31
A-Rev
Candidate
RC-0
SprintN
Extended Sprints
Solution stack with
software focus
Stability
Sprint
SST
Regression
SST
Hardware Product Test
PT
Stability
Regression
EST 1
EST N
PG Enterprise Validation
EST F
Final V
Agile @ Dell
Geography
• Co-location of a Scrum
team is always preferred
• Limit geographical
distribution of Scrum
team to no more than
two time-zones
• Scrum teams within a
project can be distributed
32
PG Enterprise Validation
Automating in
an Agile World
33
Opportunities for Automation
• It’s not just about automating test cases
• Can also provide efficiencies to:
–Infrastructure Automation
› Test Preparation, Setup and Configuration
–Non-Functional Testing (“ility”)
› Longevity, Scale and Performance Characterization
–Compatibility Testing (Solution and Device)
34
Infrastructure Automation
• Bare-metal Deployment
• Setup & Configuration
– OS Provisioning and
Configuration
• Test case staging
• Environment
Cleanup/Baseline
• Virtualization-based test
environments
35
Test Content Development
• UI automation on Customer
Usage workflows
• In-Sprint, Automated
Acceptance Tests
• Web-services (or CLI) Test
Automation for Functional
Testing
• Automated CI (UT, Build,
BVT)
• Unit Test Automation
2
CAUTION: Application Architecture can be an enabler or inhibitor
2 - Agile Testing: A Practical Guide for Testers and Agile teams
36
Application Architecture Matters
• Does it facilitate testing at the API/Services level?
• Does the Business/Error Logic reside below the services
level?
• Does the UI architecture support test automation other
than record-playback?
Large Scale Organizations:
• Standardize UI architectures across the product
portfolio
• HTML5, Flex, Silverlight…
• Standardize Service architectures across the
product portfolio
• SOAP, REST, CLI, API
37
Non-Functional Testing
• Performance Characterization
• Longevity
3
• Stress
• Scale
• Concurrency
Often times analysis of
these areas are simply
too cost-prohibitive to
be done manually
3- http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
38
Performance Characterization Timeline
Sprint 21
Sprint 15
Functional
Validation
Key Dependency:
• 3rd party HW available
• 3rd party user stories
completed
M1 11 12 13 14
Longevity
PerformanceCharacterization and
non-Functional Validation
L
16
…
a
r
g
e
S
c
Baseline a
l
Scalee
17 18 19 M2
SST
Concurrency
Simulation
39
Confidential
L
a
r
g
e
S
c
a
l
e
1.0
The Lessons of
‘Large’
Courtesy: Gulliver's Travels (2010)
40
The Lessons of ‘Large’
1. Co-located Teams
2. Project-wide ‘Done’
Criteria
3. Culture of Automation
4. System Test Team
5. Test Architect
6. ALM Tooling
Questions?
42
Resources
• Agile Manifesto - http://agilemanifesto.org/
• Articles/Blogs:
– http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ - SmartBear
– http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ - Lisa Crispin
– http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=17793&tth=DYN&t
t=siteemail&iDyn=2 – Rajini Padmanaban
– Beyond Agile Testing: http://www.utest.com/int-v1/a/beyond-agile-testing - uTest
– Testing and Checking Refined: http://www.satisfice.com/blog/archives/856 - James Bach and Michael
Bolton
• Books:
– Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell
– How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo
– Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory
– http://www.dorothygraham.co.uk/automationExperiences/index.html - Dorothy Graham
– Unblock! A Guide to the New Continuous Agile – Andy Singleton
• Presentations:
– Approaches to Software Testing: An Introduction – Scott Barber
– Scaled Agile Framework Big Picture – Dean Leffingwell
43
Backup
44