DevOps for the
Mainframe
Leveraging continuous integration testing, the cloud, and
beyond to deliver z/OS applications
David Myers
February, 2013
Please note
IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment,
promise, or legal obligation to deliver any material, code or functionality. Information about
potential future products may not be incorporated into any contract. The development,
release, and timing of any future features or functionality described for our products remains
at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in a
controlled environment. The actual throughput or performance that any user will experience
will vary depending upon many factors, including considerations such as the amount of
multiprogramming in the user’s job stream, the I/O configuration, the storage configuration,
and the workload processed. Therefore, no assurance can be given that an individual user
will achieve results similar to those stated here.
Why we need DevOps on the Mainframe
•
•
•
“Business software … has become horrifically complex”
“Developers are good at new functions, but bad at ensuring nothing goes
wrong when the new software is added to the existing mix”
“No single group of people, can ever fully understand the structure of key
business transactions in an enterprise”
In 2013 alone
“users to face duplicate charges”
“glitch delaying wage and bill payments”
Major 2012 events
“bungled update to CA-7 batch processing
software costs over £100M”
http://www.bbc.co.uk/news/technology-21280943
“’system issue meant over 400,000
trades had failed to fulfill its bestprice guarantee.”
“flash trading error causes $440M loss – due
to dead code being accidentally triggered”
Mainframe Delivery
Pains…
• Bad code going into
production causes crit sits and
emergency fixes
• Restricted dev and test
capacity lead to conflict,
delays, or bad test results in
shared environments
• Manual dependency
management and
configuration tasks result in
errors and delays
• Bottlenecks due to inefficient
communications between
disparate platforms and
teams (Dev/Test - System
Programmers; mobile –
distributed-mainframe)
4
…solutions from IBM
Play Defense
 Enforce base quality standards
automatically prior to promote
– Continuous Testing (CIz/RTW) with
automated promotion policies
 Provide cheap, isolated,
production-like, fit-for-purpose
development and test
environments for project teams
– RD&T + RTVS Test Virtualization
 Automate build, provision,
configure, and deploy processes
– Cloud deploy of dev/test images
 Improve communication and
collaboration with cross-platform
release planning and monitoring
– Proactive release planning
• Expand RTC planning to System programmers.
• Link RTC to incident management
• Link RQM to application monitoring
Go on Offense
5
DevOps for the common mainframe
delivery cycle
2
1
1
Dev
Dev
Dev
Test
Test
Test
Staging
Prod
4
3
Development
Test
System
Programmer
6
Enterprises want to… deliver end-to-end application enhancements quickly to stay
competitive, trust that complex enterprise systems can be broadly integrated,
and bolster confidence in application quality
Development Environment
Code
Check
In
Change
Request
Queue
Request
Build
Deploy
Queue
Request
Run Tests
Promote
To QA
Developer
Open
Defects
System
Programmer
Tester
Feedback
But…
It takes weeks or even months to test and fix changes due to reliance on manual processes
and limited access to test resources
Fail fast and drive better quality downstream
Reduced delivery time, end-to-end visibility of test activities, safer and faster upgrades (V2V)
Automated,
continuous testing
with isolated
development and
test environment
Continuous Integration for z
Code
Check In + Build + Deploy +
Run Automated Tests
Change
Request
Frequent, rapid
feedback earlier
to improve
quality
Reusable
integration
tests simplify
testing across
layers
Promote
To QA
Developer
Rapid Feedback
Higher quality
going into QA
 Fast, dependable, automatic feedback speeds time to market
 Lower cost of application testing using off-mainframe z/OS test environment
 Enables confidence by automatically tracking and promoting code health
 Rational Team Concert 4.0
 Rational Quality Manager 4.0
 Rational Development and Test Environment for System z 8.5
 Rational Test Workbench (Rational Integration Tester) powered
by Green Hat Technology
Detailed Continuous Integration for System z Scenario
System z
Test
Environment
(RD&T or LPAR)
<Starts Tests>
(Rational Team
Concert)
<Results>
1. Check-in code
2. Build code and zUnit
tests
3. Deploy build results
and test data to Test
Environment
4. Execute zUnit Tests
Test
Execution
Manager
(Rational Quality
Manager)
5. Kick-off Automated
Test Plan
8. Report test results
in dashboard/build
results/defect
records in CI server.
<Drives>
CI
Server
Test
Automation
(Rational Test
Workbench)
6. Run automated
interface tests against
Test Environment
7. Mark execution
records Pass/Fail in
Test Execution
Manager
9
Rational Development and Test Environment for System z
The ultimate in modern application development for System z
COBOL, PL/I, C++, Java, EGL, Batch,
Assembler, Debug Tool
IMS
DB2
CICS
WAS
RDz user
RDz user
MQ
z/OS
x86 PC running Linux
RDz user
ISPF user
RDz & ISPF user




RDz user
Liberate developers to rapidly prototype new applications
Develop and test System z applications anywhere, anytime!
Free up mainframe development MIPS for production capacity
Eliminate costly delays by reducing dependencies on operations staff
Note: This Program is licensed only for development and test of applications that run on IBM z/OS. The Program may not be used to run production workloads of any kind, nor more
robust development workloads including without limitation production module builds, pre-production testing, stress testing, or performance testing.
10
Continuous Testing with DevOps and Green Hat
Improve testing with Green Hat service simulation
Developers &
Testers
Simulate subsystem dependencies
– Test impact of latency
– Test application response to unresponsive
services
Rational Test Workbench
Rational Performance Test Server
Create a stable test environment
– Simulate “public” services
– Reduce capacity requirements on the cloud
infrastructure
– Improve security by reducing access from
external services
App Under Test
Support continuous Test & Delivery
Databases
Mainframe
applications
Third-party
Services
Packaged apps, messaging services, etc.
Rational Test Virtualization Server
Testing with dependency virtualization
Controlled large system testing by isolating components under test
• Easier problem determination
• Lower test environment capacity requirements
• Improved component quality
Phase 1
Test Case
IMS
Data
Access
App
Test Case
3rd Party
Call
CICS
Commarea
Call
Virtual
Services
Phase 3
Phase 2
IMS Data
Access
Test Case
3rd Party
Call
App
CICS
Commarea
Call
Virtual
Services
IMS Data
Access
3rd Party
Call
App
CICS
Commarea
Call
Virtual
Services
Separate tools makes collaboration inefficient or difficult...
Tool A
Dev work Items
Each tool came with its own
• UI - Web and desktop
presentations of views and
tasks
• Logic – Workflow, process,
search, query, scale, security
and collaboration
• Storage – Availability,
traceability
• Privacy, backup/archive
Resulting in...
• Brittle integrations
• Silos everywhere
• High cost to maintain and
administer
• Proprietary API's
DB
Tool B
Release
Planning
UI
LOGIC
DB
Tool E
SCM
DB
UI
LOGIC
UI
LOGIC
DB
DB
Tool D
Incident
Management
UI
LOGIC
UI
LOGIC
Tool C
Reporting
…consolidate, standardize, and simplify to increase collaboration
Engage all stakeholders in early planning, share and make visible
Enterprise Applications have thousands of disparate parts
• Currently maintained in separate systems
• Limited linkage between systems for application dependencies
• Missing assets and information is rampant
DevOps force linkages, automation, and standardized packaging…
Copybooks
HTML
JCL
COBOL
WSDL
Java
DB2
DDL
MQ
Queue
Def
Build, Package,
& Unit Test
Application
Binaries &
Platform
Configuration
DB2
Plans
IMS
DBD/PSB
Procs
Source Management
Deployable Artifacts
Library
Deploy
Environment
Running System
Collaborate
Define
Execute
Report
Rational Team Concert
IBM SmartCloud Provisioning
Agile
Development
V2.0 supports distributed platforms
Deployment of
Virtual Systems
IBM SmartCloud Continuous Delivery
Development
“Cloud”
App Code / Build
Provision / Configure
Virtual Environment(s)
Test
Test Scripts
IT System
Admin
Continuous
Delivery
Validate Build With
Automated Test Suite
Configuration as
Code (Scripts)
Report Test Results
16
Spend less time provisioning and comparing environments!
Development
Benefit: On-demand access to production-like test
environments
… less time comparing configurations and recreating
defects due to configuration shift
Test
Benefit: On demand access to current development build
and configuration
… able to test current function instead of function from 2-4
weeks ago
IT System
Admin
Benefit: Less time spent manually configuring test
environments
… improved response time to delivery demands and
configuration stabilization
17
Internal Success stories
25%
Reduced Development
Cost by 25% with much
improve quality
Accelerated product
development with 100s
of deployments enabling
delivery of stable
deliverables for early
customer feedback
99%
90%
Reduction in Testing
time from 2 Weeks
to 3 hours
accelerating product
delivery with much
improved quality.
Reduction in test
deployment by 90%
and expected to save
over $2.3M / year and
accelerating product
delivery
18
What we are working on in
the lab…
…solutions from IBM (future)
Play Defense
 Enforce base quality standards
automatically prior to promote
– Continuous Testing (CIz/RTW) with
automated promotion policies
 Provide cheap, isolated,
production-like, fit-for-purpose
development and test
environments for project teams
– RD&T + RTVS Test Virtualization
 Automate build, provision,
configure, and deploy processes
– Cloud-style deploy of dev/test images
 Improve communication and
collaboration with cross-platform
release planning and monitoring
– Expand RTC planning to System
programmers. Link RTC to incident
management via OSLC..
Go on Offense
20
Characteristics of Cloud
Shared Resource
Pools
Pay-per-use
Multi-tenant
Elastic and
scalable
Minimal management
Rapid deployment
Self-service
Established mainframe strengths
Standardize z/OS and distributed
images to ease deployment
Configuration
Application
Data
CICS 4.2
• Standardized images which can be
repeatedly deployed are rare
IMS 11.1
Standard Image and Middleware
Z/OS 1.12
• Standard topologies exist today
(production LPARs)
DB2 10.1
• Standardized/Automated deployment of
COMPLETE system is spotty
To adopt the cloud for testing…
Standardize and automate provisioning of
everything …
Standardize z/OS and distributed
Configuration
Application
Application
load modules
JCL and REXX
Data
Test Data
CICS 4.2
IMS 11.1
Standard Image and Middleware
Z/OS 1.12
RTC/CIz
Capabilities
images to ease deployment
DB2 10.1
Pattern/Platform
Evolving to deploying z/OS … cloud-style
“Cloud”
CICS+DB2
RD&T
Or
LPAR
CICS+ IMS
DB
Image/Topology
Patterns
Batch
RD&T
Or
LPAR
RD&T
Or
LPAR
RD&T
Or
LPAR
Provisioning and
Setup Manager
Acknowledgements and Disclaimers:
Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all
countries in which IBM operates.
The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are
provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or
advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in this
presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages
arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is
intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering
the terms and conditions of the applicable license agreement governing the use of IBM software.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they
may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these
materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any
specific sales, revenue growth or other results.
© Copyright IBM Corporation 2013. All rights reserved.

U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with
IBM Corp.

Please update paragraph below for the particular product or family brand trademarks you mention such as WebSphere, DB2, Maximo,
Clearcase, Lotus, etc
IBM, the IBM logo, ibm.com, [IBM Brand, if trademarked], and [IBM Product, if trademarked] are trademarks or registered
trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM
trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols
indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may
also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at
“Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml
If you have mentioned trademarks that are not from IBM, please update and add the following lines:
[Insert any special 3rd party trademark names/attributions here]
Other company, product, or service names may be trademarks or service marks of others.
www.ibm.com/software/rational
© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind,
express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have
the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM
software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities
referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature
availability in any way. IBM, the IBM logo, Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products and services are trademarks of the International Business Machines
Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.
26
DevOps Assessment
Workshop
Objectives
• Assess your current DevOps
level based on practice areas
• Provide recommendations to
achieve goals
• Develop adoption roadmap
Execution
• Two half day sessions
• Discussions to understand
current processes and pain
points
• Discuss trends and directions
focusing on people, process,
and technology
• Present recommendations
and roadmap two to three
weeks after workshop
Assess your current DevOps practice levels
and develop an adoption roadmap to achieve
your goals.
Evolving to Multi-stage Delivery Pipelines
Pipeline dashboards
provide feedback for
the flow of changes
through the pipeline
Library
Changes trigger
pipeline execution
(deployables)
Delivery Pipeline
Reporting/Dashboards/Analytics
QA Stage
SCM
(Source code &
configurations)
Build
Stage
PreProduction
Stage
Dev
Stage
Promote to
Production
Performance
Stage
Each stage fit-forpurpose for task
execution
Each stage defines
tasks and
configuration settings
with entrance and exit
criteria
Approval gates with
automated/manual
approvals provide
compliance checking
Stages can be run in
parallel
28
Extending IBM SmartCloud Continuous Delivery
Optional and alternative Integrations
 Extending CLM with
Continuous delivery
 Deployment to Cloud
& Virtual Systems
IBM SmartCloud Provisioning
Rational
Team Concert
Continuous
Delivery
IBM PureSystems
IBM Workload Deployer
Continuous
Integration
IBM Rational
Build Forge.
Hudson….
Continuous Testing
IBM Rational...
Quality Manager
IBM Rational
Test Workbench
Continuous
Deployment
IBM Rational
Automation
Framework
Continuous
Monitoring
IBM SmartCloud
Application.
Performance
Management
Incident
Management
IBM SmartCloud
Control Desk..
Chef
Jenkins….
29