Agile Test Strategies in Practice:
Tales of Large-scale Agile projects
8 May 2015
Geoff Meyer, geoffrey_meyer@dell.com
SW Engineering Director, Dell Inc
Session Objectives
• Introduction to Key Agile Testing concepts
IS
• Real-world examples − from multiple, large-scale
projects
• Organizational strategies & considerations
• Test automation strategies and tooling
considerations
IS
•
Test Data Management
NOT
•
An In-depth Tools Discussion
•
Focused on how to develop Test Automation
Geoff Meyer, SW Director
Dell Commerce Services Group
Business
Personal
Family is paramount; my
wife is a Professional
Networking queen and
my kids make me proud!
Lives
Austin, TX
Lived
San Diego, Twin Cities
Hobbies
Travel, Avid fan of NFL,
NHL, EPL, MLB, Golfer,
Monday Morning
Quarterback
2015
Specialty
Change Agent and
champion for Agile
transformations
2010
2005
2000
1998
1995
1990
1984
My Role @ Dell
Leads
Test and DevOps
functional areas for
Dell.com
Passion
Solving problems with
great people and
enjoying every workday
Mission
Implement Continuous
Testing, Integration and
Delivery practices
Operating Model
Embrace Change and
Continuous Learning
Goal
Transform Dell.com into
a SW Development
engine characterized by
Quality, Predictability
and Continuous Delivery
Roles
Director, Architect,
Program Manager,
Manager, Developer
Agenda
• Agile Testing Concepts
– Organization and
Architecture
– Agile Testing
– Test Automation
– Continuous Testing
– Large-Scale Agile and SAFE
• Agile Projects in Action
Agile Roles/Responsibilities
Program
Product Owner
Scrum Master
Development
Test
Product Owner
Program Manager
SW Architect
Test Architect
DevOps
• Small teams
• Co-located
…
• Multi-disciplined
• Feature-based
• Distributed Projects
• Scrum of Scrums
5
Product Owner
Scrum Master
Development
Test
Agile Sprint Testing Cycle
Week 1
Week 2
User
Story
Sprint
Planning
User
Story
Automated
Regression
Demo
Demo
Accept
Demo
User
Story
User
Story
Accept
Demo
Retro Sprint
spectiveReview
Accept
User
Story
Accept
Automated
Regression
Automated
Regression
Demo
Accept
Automated
Regression
Manual Regression
Automated
Regression
Automation Pyramid
“Automate Within the
Sprint. Automation on a
Scrum project is not
optional.” - Mike Cohn
CAUTION: Architecture (and Organization!) can be an enabler or inhibitor
7
Agile Testing Quadrant
Feature
Team
3
System
Team(s)
UAT
E2E
Performance
More Agile Testing: Learning Journeys for the Whole Team. Gregory, Janet; Crispin, Lisa (2014-09-30).
8
Automation Tooling
• IDE
• UI Tools
• Scripting Languages
• Services
MS-Test
• Unit
• Frameworks
9
Scaled Agile Framework™ Big Picture
© 2008- 2013 Scaled Agile, Inc and Leffingwell, All rights reserved
Agile Projects in Action @ Scale
Project Profiles
Application Architecture
Moe
Presentation Layer
Systems Mgmt
Application
Curley
Business/Service Layer
eCommerce UI
Application
Larry
Data Access Layer
eCommerce
Services
Organizing
Team Roles/Responsibilities
Program
Product Owner
Scrum Master
Development
Test
Product Owner
13
Agile Test Strategy: Moe
All Agile Testing
managed within
the program
Agile Test Strategy: Curley
UAT and ‘ility’
testing conducted
by external teams
UAT Regression
tests automated by
the program team
Agile Test Strategy: Larry
‘ility’ testing
conducted by
external teams
Automation Strategy: Moe
UI Automation limited to
Scenario-based workflows
In-Sprint Automation at the
Unit and Service Levels
‘ility Testing’; Moe
Key Dependency:
• 3rd party HW available
• 3rd party user stories
completed
Sprint 21
Sprint 15
L
Functional
a
Validation
M1 11 12 13 14 …
r 16 17 18
g
SST
e
Longevity
S
Baseline c
Performancea
Characterization and
Scalel
non-Functional Validation
e
Concurrency
19
L
a
M2 r
g
e
S
c
a
l
e
Simulation
18
Confidential
1.0
Automation Strategy: Curley
+UI functional
tests
UI Automation to Scenariobased workflows
Services Virtualization to support
Continuous Testing
Functional Test Automation
done In-Sprint at the Unit and
UI Levels
Automation Strategy: Larry
Services Virtualization to support
Continuous Testing
Functional Test Automation
done in-Sprint at the Unit and
Service Levels
MS-Test
Continuous Integration &
Automated Testing: Moe
Continuous BVT
CI/Build
Automated Testing
Team #1 Feature Test
• Exploratory Testing
• Acceptance Testing
Team #2 Feature Test
TFS TeamBuild
• Exploratory Testing
• Acceptance Testing
Team #N Feature Test
• Exploratory Testing
• Acceptance Testing
21
To Factory
Continuous Integration &
Continuous Testing: Curley
Continuous Testing
Continuous BVT
SIT
App Test VM
Team #1 Feature Test
CI/Build
• Exploratory Testing
• Acceptance Testing
App Test VM
TFS TeamBuild
Team #2 Feature Test
…
• Exploratory Testing
• Acceptance Testing
App Test VM
Team #N Feature Test
• Exploratory Testing
• Acceptance Testing
22
To Staging &
Production
Continuous Delivery
Static Analysis
Auto. Unit Test
CI/Build Gated Check-ins
Dev Test
Baseline
Load
Stress
Dev Test
Unit test
Dev Test
Integration test
Unit test
Dev Test
Integration test
Unit test
Dev Test
BVT
BVT
Tests
Integration test
Unit test
Integration test
Unit test
Integration test
Auto
Depoy
Minutes
•
•
•
•
23
Performance
App Test
App Test
Acceptance Tests
App
Test Tests
Acceptance
Exploratory
Tests
Exploratory
Tests
UITests
Tests
Acceptance
UI Tests
Func.
Regression
Exploratory
Tests
Func.
Regression
regression
UIUAT
Tests
UAT
regression
Func. Regression
UAT regression
Minutes
Automated Smoke, DVT, BVT & Regression Suites
Continuous Integration–Fast feedback/Green Builds
On-demand Environment Provisioning & Deployment
Local Control of Environments: Mocks & Data
Staging
Production
E2E Tests
UAT regression
Automated
Smoke
DVT
UAT
SIT
UAT
Func. regression
UAT regression
E2E
Auto
Deploy
Accept
Auto
Deploy
Zero-Touch
Deploy
The Lessons of ‘Large’
• There is no cookie-cutter Test
Strategy
• Strive for co-located Teams
• Establish Culture of
Automation
• Create Cross-team roles
• Inspect and Adapt
Recommended Readings
25
Dell - Restricted - Confidential
Questions?