Testing, testing
A first-look at the new testing
capabilities in Visual Studio 2010
Mathias Olausson
Callista Knowledgebase | QWise
QWise software engineering – refactored!
www.qwise.se
About Mathias...
QWise software engineering – refactored!
www.qwise.se
About the Presentation...
• What’s on the menu?
– Test planning and execution
– Issue tracking with rich bugs
– Eliminating the "works on my machine / cannot
reproduce" scenario
– Automated UI Testing
– Virtual Lab Management
QWise software engineering – refactored!
www.qwise.se
TEST PLANNING AND EXECUTION
QWise software engineering – refactored!
www.qwise.se
Agile Testing
Agile projects require
different testing
• Incremental = re-test
• Iterative = test often
Development and
test more integrated
QWise software engineering – refactored!
Test documentation
may be scoped out
Focus on automatic
testing
www.qwise.se
Core Practices
Code and Test
as One
Process
Incremental
Work
QWise software engineering – refactored!
Continuous
Integration
Controlled
Test
Environments
www.qwise.se
Agile Testing Quadrants
Business-facing
Manual
Functional Tests
Exploratory Tests
Unit Tests
Component Tests
Performance Tests
Load Tests
Security Tests
”-ilitets” Tests
Critisizing the product
Supporting the team
Automated &
Manual
Automated
Automated
Technology-facing
QWise software engineering – refactored!
www.qwise.se
Test Types
Manual Tests
UI Tests
Acceptance Tests
Unit Tests / Component Tests
QWise software engineering – refactored!
www.qwise.se
Testing in an Iteration
D
E
V
sprint
plan
build 1
T
E
S
T
sprint
plan
implement
Implement
us2
US2
implement
user story ‘1’
(us1)
build 2
fix bugs
fix bugs
build 3
write tests
for us1
build 4
test us1 &
file bugs
write tests
for us2
QWise software engineering – refactored!
build 5
build 6
verify fixes
test us2 &
file bugs
build 7
regress
impacted
tests
verify fixes
www.qwise.se
Microsoft Test Manager and Visual Studio
Testing (Teams)
Development (Teams)
Engineering Support
Unit Testing
Specialist
Generalist
Coded UI Test
Microsoft Test Runner
Web Test
Load Test
Fast Forward For Manual Testing
Test Case Management
Lab Management
Data Collectors (IntelliTrace, Test Impact, Metrics)
Team Foundation Server
QWise software engineering – refactored!
Reporting
www.qwise.se
The Fundamentals
QWise software engineering – refactored!
www.qwise.se
Defining and Running Tests
QWise software engineering – refactored!
www.qwise.se
ISSUE TRACKING WITH RICH BUGS
QWise software engineering – refactored!
www.qwise.se
Should We Do Bug Tracking?
Pros
– Tracability
– Priority of work
– Visibility
Cons
– Overhead
Should we log bugs within an iteration?
QWise software engineering – refactored!
Source: Pixar ”A bugs’ life”
www.qwise.se
Defect Tracking in MTM and MTR
QWise software engineering – refactored!
www.qwise.se
No-more-no-repro
• Automatic data
collection
• Test settings
– Configuration of
diagnostic data
collectors
– Assigned to test plan
• Or per test run
QWise software engineering – refactored!
www.qwise.se
Diagnostic Data Adapters
Data adapter
Description
Action recorder
use to collect each UI action you perform as you run a test (for
client roles only)
Asp.net client proxy for
intellitrace and testimpact
use for web applications when you select intellitrace or testimpact
for a server role (use on client to web server)
Eventlog
use to capture event log data (for client or server roles)
Intellitrace
use to collect exceptions and specific diagnostic tracing
information to help isolate bugs that are difficult to reproduce (for
client or server roles)
Network emulation
use to emulate slower networks when you run your tests (for
client or server roles)
System information
use to collect system information for a machine (for client or
server roles)
Test impact
use to collect information that can help you decide which tests to
rerun based on changes made to an application for a specific build
(for client or server roles)
Video recorder
use to create a video recording of your desktop session while you
run a test (for client roles only)
QWise software engineering – refactored!
www.qwise.se
No-more-no-repro
QWise software engineering – refactored!
www.qwise.se
AUTOMATED UI TESTING
QWise software engineering – refactored!
www.qwise.se
Why Automated Testing?
Källa: Tools for Agility, Kent Beck
QWise software engineering – refactored!
www.qwise.se
Automation Strategy
iteration 1
feature a
feature b
iteration 2
feature c
release iteration
feature d
unit
testing
manual
testing
regression
testing
performance
testing
QWise software engineering – refactored!
www.qwise.se
Coded UI Tests
• Unit test for UI
• High-level framework
• Create test from
– Recording
– Test result (action log)
– Using API
QWise software engineering – refactored!
www.qwise.se
Automation Platform Support
Platform
RTM
RTM+
Notes
IE7/8 – HTML/AJAX
Windows Forms 2.0+
.NET Fx Controls fully supported,
working on 3rd party support
WPF 3.5+
.NET Fx Controls fully supported,
working on 3rd party support
SharePoint 2007
Windows Win32
TBD – Known issues & Workarounds
MFC
TBD – Known issues & Workarounds
FF3 – HTML/AJAX
Currently in Beta
Dynamics (Ax)
TBD – Release Plan
Silverlight
TBD – Release Plan
Citrix/Terminal Services
The client (MTLM or VS) need to be
on remote machine
Office Client Apps
IE 6
Chrome/Opera/Safari
Flash/Java
Opportunity for partners to add this
using the Extensibility support. Docs
and samples around extensibility are
here.
SAP
QWise software engineering – refactored!
Fully supported platform
Partial solution. Further work
required in future to complete
Best efforts with known issues.
Users can workaround these
cases in the code. No major
ongoing investment.
Currently no support but on
the roadmap for future
releases
Currently no support and none
planned for now. Opportunity
for partners to add this using
the Extensibility support. Docs
and samples around
extensibility are here.
For latest info on this, refer this
post.
www.qwise.se
Coded UI Tests
QWise software engineering – refactored!
www.qwise.se
LAB MANAGEMENT
QWise software engineering – refactored!
www.qwise.se
Motivation
Complex
environments
hard to set up
Agile practices
require
shorter cycles
Hard bugs
take long to
reproduce
QWise software engineering – refactored!
Many
configurations
to test
Poor
utilization of
hardware
www.qwise.se
Lab Management Aims To...
Quickly set up new environments
Speed up testing
Improve test effectiveness
Automate deployment
QWise software engineering – refactored!
www.qwise.se
Lab Management Infrastructure
Team
Foundation
Server (TFS)
Lab Management
System Center Virtual
Machine Manager (SCVMM)
Test Case management
Build management
Work Item Tracking
Source Control
QWise software engineering – refactored!
Hyper-V
Hosts
Library
Shares
Virtual
Machine1
Library
Share1
Virtual
Machine2
Library
Share2
Virtual
Machinen
Library
Sharen
www.qwise.se
Environments
• An environment is a set of machines with
different roles
• An environment can
– Be deployed, saved, removed
– Started, stopped
– Snapsshot
QWise software engineering – refactored!
VM1
VM2
Lab Agent
Lab Agent
www.qwise.se
Manual Tests in Lab
• Testing from the outside
– Easy for web applications
– Limitations in data collection
• Testing from the inside
– Run application in lab via environment viewer
QWise software engineering – refactored!
www.qwise.se
Automatic Tests in Lab
• Testing in VM
– Test case connected to test environment
– Test component comes from selected build
• Testing in build
– Test case connected to build definition
– Runs test list or test component
QWise software engineering – refactored!
www.qwise.se
Lab Management Build Workflow
build app.
restore
snapshot
restore
env.
run tests
take env.
snapshot
QWise software engineering – refactored!
deploy
app.
www.qwise.se
Build and Deploy
QWise software engineering – refactored!
www.qwise.se
Summary
•
•
•
•
•
Structured test planning with MTM
Integrated test execution
Effective defect tracking with rich bugs
Seamless integration with automated tests
Powerful lab management
QWise software engineering – refactored!
www.qwise.se
Questions?
mathias.olausson@callista.se
http://msmvps.com/blogs/molausson
QWise software engineering – refactored!
www.qwise.se