Presenter: David A. Watt
Copyright Protected Material
David A Watt, 2010
Database
Server
Client B
LAN
Application
Server
Client A
Peripheral
Systems
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
Other
Components
2
Traditional Development Method
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
3
Load Balancers
Storage Fabric Network
Systems are Getting More Complex
From a Blog by Lori MacVittie on DevCentral
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
4
Enhanced Development Methodologies
V-Model
Development
REQMTS
ACCEPTANCE
ACCEPTANCE TEST PLAN
DESIGN
BUILD MAP
T
SYSTEM
SYSTEM TEST PLAN
DEVELOP
Spiral Model
TEST
TEST
C
INTEGRATION
A
TEST
D
CODE
UNIT
TEST PLAN
UNIT
TEST
Test
Agile
It 0
It 1
It 2
It 4
F
C
A B
Infrastructure Integration Testing:
The Oft Ignored Testing Type
It 3
A B
D
C
A B
Copyright Protected Material
David A Watt, 2010
E
D
C
A B
(Agile Testing; Crispin, Gregory)
5
IT Systems for Mid to Large Sized Corporations
User Population
HR
Finance
End User
Support
Training
Security
Infrastructure
Empl
Portal
Procure
ment
Billing
Infrastructure Integration Testing:
The Oft Ignored Testing Type
MRP
Point of
Sale
Copyright Protected Material
David A Watt, 2010
6
Common Infrastructure Elements
Desktop / Laptop Configuration Variations
User Authentication and Digital Rights Management
Database Services
Web Services – Presentation, Delivery and Authentication
Load Balancing
Computer Hosting Platforms and Operating Systems (OS)
Storage Infrastructure
Network Infrastructure
Security Policies and Operations
People, Processes, and User Support
Operational Management and Oversight
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
7
Statement of the Problem:
Traditional system testing techniques teach and illustrate testing the system as a
“complete” but stand-alone system.
For some deliverable systems this is ok.
But for IT solutions deployed into a medium to large scale corporate environment, this is
naïve and insufficient.
After all traditional system integration testing has been completed, another layer of testing
is required.
Testing in this layer verifies that the solution integrates with the rest of the IT infrastructure
elements and is coined “Infrastructure Integration Testing”.
Development
REQMTS
ACCEPTANCE
ACCEPTANCE TEST PLAN
TEST
INFRASTRUCTURE
INTEGRATION
DESIGN
SYSTEM
SYSTEM TEST PLAN
DEVELOP
CODE
Infrastructure Integration Testing:
The Oft Ignored Testing Type
BUILD MAP
UNIT
TEST PLAN
TEST
TEST
INTEGRATION
TEST
UNIT TEST
Copyright Protected Material
David A Watt, 2010
Test
8
IT Infrastructure Elements
Client
Services
Application
Services
Operating Systems
Desktops
Portals
Laptops
Office Apps
Terminals
End User Apps
Mobile & Wireless Devices
Business Reporting Svcs
Enterprise Applications
Application
Enterprise Applications Suites
Authentication
Ops
Processes
Build Docs
Configuration Mgmt Sys
Software Distribution Sys
Middleware
Database Mgmt Systems
Web Delivery Services
Infra Apps
E-Mail Svcs
Security
Standard Operating Instrs
Blackberry
Security Policy
Authentication Svcs
Terminal Services / Citrix
Storage
Infrastr
Tiered Storage Solutions
Computing
Infrastr
Network
Infrastr
Facility
Infrastr
Disk Array
EPO
International
Windows Forest & AD
Operating Systems: Multi-vendor
Intel Svrs
Document File & Share
DNS, DHCP
Extranet
Encryption
Gateways
Firewalls
Routers
Enhanced Sec OS Settings
LDAP
HVAC
Power
Floor Space
SOA Architecture
Siteminder Svcs
F5 Load Balancers
Fabric Controllers
VM-Based Svr
LAN / WAN
Telecommunications
Vulnerability Mgmt
Virtual Machine Svcs
Message Store
Network Mgmt System
Switches
Development Svcs
Incident Mgmt Collaboration
Disaster Recovery
On-line Backup
Resource Management
Blade Svrs
Provisioning & Health Mon
Web Presentation Services
Remote Access Systems
Network Attached
MP Servers
Application Security
Help Desk, & Training
Patch / Upgrade Mgmt Sys
PGP / PKI
VPN Access
Standard Products & Arch
Data Access Rights Mgmt
Storage Management Systems
Mainframe
Virus Mgmt
Department Applications
Federated AD
Storage Area Net
Encryption
Disability Support Tools
Application & End User Performance
Office Communications Svcs
Compliance
Personal Firewall
Tape Silos
File System: NFS, DFS, CIFS
Open Sys
Peripherals
Backup Sys
Proxy Svrs
VOIP
QOS
VPN
Wireless
IDS
Facility Monitoring
Operations Personnel
IT Infrastructure Elements
IT Infrastructure Elements
IT Infrastructure Elements
IT Infrastructure Elements
Questions To Be Answered
Do your solution’s parameters integrate with the parameters from other
solutions?
How does your solution impact performance of the rest of the environment?
Does your solution comply with corporate security settings?
Does your solution integrate with load balance, firewall, and gateway systems?
Does your solution fit into production security certificate authority? Are there
enough user certificates available?
Is the version and configuration of your solution compatible with that of shared
infrastructure?
Are there adequate installation, operations, and support documents available?
Has the help desk and operations support staff become acquainted with your
solution?
Does your solution require anything to be loaded on the client system? Is it
compatible with all versions and configurations?
Is your solution compatible with health and performance monitoring tools?
Do the backup and disaster recovery requirements of your system integrate with
the existing solutions?
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
14
Types of Testing to be Performed
Functional regression testing of your solution, infrastructure elements, and other
solutions
Capacity, load, and performance testing of your solution
Performance regression testing of the remaining infrastructure and other
solutions
Performance baseline testing
Functional testing of operational monitoring tools and processes
Regression testing of your solution and others when changes are made to the
infrastructure or other solutions
Validation testing of operational SOIs, training materials, help desk
documentation, and other user support processes
Ad Hoc testing and use by support staff
Vulnerability scanning and remediation
Problem isolation and remediation
On-going capacity modeling
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
15
The Challenge of “IIT”
Infrastructure Integration Testing (IIT) may
be a great idea but ……
Q: How and where do you perform IIT?
Not having good answers to this question often leads to one of two
deadly sins:
1.
Perform Infrastructure Integration Testing in the production environment
OR
2.
Test by putting a solution into production and letting the users find the
“bugs”
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
16
Alternate Approach
Build an Enterprise Test Bed:
“An accumulation of separate and distinct test
environments into an integrated and well managed
test bed, which is isolated from the production
environment and where functionality, interoperability,
and performance testing can be performed on many
solutions simultaneously”.
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
17
Enterprise Test Bed
Scale model of production solutions
Physically and logically isolated from production environment
Accessible from anywhere on the corporate intranet
Reciprocal testing: Impact of
System Under Test against the rest of the test bed
Test bed against System Under Test
Test
Article
Client Services
Application Services
Operational Processes
Middleware
Infrastructure Applications
Security & Compliance
Authentication Services
Storage Infrastructure
Computing Infrastructure
Network Infrastructure
The Realities of an Enterprise Test Bed
Logistical
Political
Financial
Operations Management
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
19
Test Bed Management
Experience has shown that test beds must be managed using developed and
enforced processes in order for the value of the investment to be maximized.
Consistency in following processes is the key to success.
Processes should be developed and enforced
in the following areas:
• Configuration management – including change control
• Security compliance monitoring and maintenance
• Problem resolution
• Asset control and management
• Usage coordination
• Test planning and execution
• Test documentation management
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
20
20
Agile and the Enterprise Test Bed
Results of the sprint iterations are tested and pronounced
“Ready for Production use” in the ETB and held there until
a “bundle” of changes are ready for insertion into
production use. New code is then implemented into the
production environment and enabled for use.
(ETB is used as staging area)
Client Services
Application Services
Operational Processes
Middleware
Infrastructure Applications
Security & Compliance
Authentication Services
Storage Infrastructure
Computing Infrastructure
Network Infrastructure
Regression Tests –
on both sprint code as
well as infrastructure
Conclusions
Treating new solutions as “stand alone” systems needs to
change
For IT solutions, “Infrastructure Integration Testing” needs
to be performed
IIT verifies that the new solution integrates successfully
the rest of the computing environment
IIT testing involves such testing as compatibility,
functional, performance, and process validation.
Existing test environments can be consolidated into a
centrally managed Enterprise Test Bed.
Test Bed Management techniques must be employed.
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
22
Questions
Infrastructure Integration Testing:
The Oft Ignored Testing Type
Copyright Protected Material
David A Watt, 2010
23