IBM System z Technical University – Berlin, Germany– May 21-25
Session code zZS25
***** PRELIMINARY VERSION V *****
Software Deployment Customer Experiences
Tim Alpaerts, Euroclear
© 2012 IBM Corporation
Agenda
 Terminology
 The old situation
 The new design
 The Product Catalog in TDSL
 Non-uniform software stacks
 TDSL Product Selection & Generate (build) processes
 The roll-out (restore) process
 VSAM challenges
 zFS on the installation system
2
Terminology
 Installation system: one particular MVS system dedicated to installing software and
installing fixes.
 End-User system: all MVS systems other than the installation system.
 Dead libraries: product libraries created on the installation system by software installation,
(maybe SMP/E installed or non-SMP/E installed) not actively in use by an instance of the
product.
 Retrofit: after making a local correction on an end-user system, it needs to be incorporated
into the dead libraries for future builds.
 Version-less dsnames: product library dsnames that do not imply a particular version or
release of a software, they do not contain a version release qualifier.
 Non-Uniform software stacks: selectively activating only the needed parts of the complete
enterprise software stack on every system, mainly for the purpose of reducing the license
charge.
3
Agenda
 Terminology
 The old situation
 The new design
 The Product Catalog in TDSL
 Non-uniform software stacks
 TDSL Product Selection & Generate (build) processes
 The roll-out (restore) process
 VSAM challenges
 zFS on the installation system
4
Pre-production Brussels
The old situation, overview
Pre-production Paris
Production systems
Development systems
K-systems
each group of systems:
Test systems Paris
Active MVS target & dlibs
Inactive MVS target & dlibs
Test systems Brussels
IBM
MVS SMPE,
2 target zones,
2 dlib zones
5
IBM
SMPE
non-IBM
products
within each group
DASD was shared with CA-MIM
Some issues with the old set-up
 Version-release qualifiers in dsnames of product libraries make system upgrades
risky.
 Enqueue and reserve contention interference between lpars, instability from deadlocks.
 No inventory nor overview of what system is using what software releases.
 No obvious distinction between software product libraries and databases and work
datasets.
 Locally maintained PROGxx parmlib members accumulate errors over time
 Low confidence when doing IPL.
7
Agenda
 Terminology
 The old situation
 The new design
 The Product Catalog in TDSL
 Non-uniform software stacks
 TDSL Product Selection & Generate (build) processes
 The roll-out (restore) process
 VSAM challenges
 zFS on the installation system
9
Objectives for the new design (1)
 Software installation needs to create dsnames (dead libraries) different from those
the driving system is using.
 All software installation work concentrated on one single system dedicated for this
purpose.
 All product libraries with version-less dsnames, to avoid widespread impact from
system upgrades.
 Distinction between R/O product libraries and R/W databases immediately
apparent from the dsname.
 Complete isolation of DASD between lpars in the IODF, no more cross-plex
sharing of dasd.
10
Objectives for the new design (2)
 All software, OS and all software products are upgraded in one go.
– Upfront agreement on scope and planning (IPL slots).
– Integrated testing.
– All the “hold actions” are performed in parallel and are repeatable.
 Provide a ‘perfect’ APF-list and link-list, complete but without any needless entries.
 Ensure at all times we have an SMP perfectly in-sync for every software rolled-out on every
system.
 Upgrade per system if required by a freeze of the environment.
 Complete inventory of software installation and deployment states.
– Visibility of what is used where (licencing, cleanup of old S/W)
 Installation guidelines and documentation is available the same way for all softwares.
11
The new design, overview (1)
K-systems
Production systems
Development systems
Test systems
&SYSR1
&SYSR2
&SYSR3
&OEMR1
&OEMR2
&OEMR3
12
each system:
Active IPL-set
Inactive IPL-set
6 model 9 vols
6 model 9 vols
End-user systems
The new design, the IPL-set
 IPL-set is restored from a (virtual) tape that was created on the installation system.
 Each sysplex has two IPL-sets, the active set and the inactive set.
 First volume of an IPL-set is the load device for the end-user system.
 First volume restored from full volume dump to include the IPL text in the dump, the other
volumes are dumped logically by dataset.
 Second volume contains an ICF user catalog for the VSAM datasets on the IPL-set.
 IPL-set is protected against updates by DFP OPEN exit IFG0EX0B, datasets can only be
opened for INPUT.
13
The new design, the IPL-set, continued
 Datasets on the IPL-set are cataloged with symbolic volsers defined in IEASYMxx parmlib
member with substrings from &SYSR1:
SYSDEF
SYMDEF(&SYSR2.='&SYSR1(1:5)2')
SYMDEF(&SYSR3.='&SYSR1(1:5)3')
SYMDEF(&OEMR1.='&SYSR1(1:5)5')
SYMDEF(&OEMR2.='&SYSR1(1:5)6')
SYMDEF(&OEMR3.='&SYSR1(1:5)7')
When MVS IPL processing sets &SYSR1, the volser of the IPL device to "IPCAA1",
then IEASYMxx sets
&SYSR2.
&SYSR3.
&OEMR1.
&OEMR2.
&OEMR3.
15
=
=
=
=
=
"IPCAA2"
"IPCAA3"
"IPCAA5"
"IPCAA6"
"IPCAA7"
The new design, overview (2)
K-systems
Production systems
Development systems
Test systems
Level-set,
6 model 9 volumes
&SYSR1
&SYSR2
&SYSR3
&OEMR1
&OEMR2
&OEMR3
16
Installation system
each system:
Active IPL-set
Inactive IPL-set
6 model 9 vols
6 model 9 vols
End-user systems
The new design, the Level-set
 An exact image of the software for the end-user systems.
 Generated by the TDSL application on the software installation system.
 A ‘contents’ dataset is added with the list of all products and libraries on the LS.
 The non-VSAM datasets on the LS are not cataloged, VSAM datasets get a
special temporary HLQ.
 Dsnames on the LS have no version-release qualifier (limited exceptions are
allowed),
 Exists only on the software installation system.
17
The new design, overview (3)
K-systems
Production systems
REF-set, SMS storage pool
REFx datasets
Development systems
Test systems
TDSL
Level-set,
6 model 9 volumes
&SYSR1
&SYSR2
&SYSR3
&OEMR1
&OEMR2
&OEMR3
19
Installation system
each system:
Active IPL-set
Inactive IPL-set
6 model 9 vols
6 model 9 vols
End-user systems
The new design, the REF-set
 A large SMS-pool with dead-libraries
 All the REF-datasets created by software installation go in this pool
 ACS routines direct all allocations of REFx datasets to this pool
 All REF-dsnames contain a version-release qualifier
 Mostly vendor product libraries, but also Euroclear ‘config’ libraries
 Exists only on the central software installation system
20
The new design, RENAME
K-systems
Production systems
REF-set, SMS storage pool
REFx datasets
Development systems
Test systems
TDSL
Level-set,
6 model 9 volumes
&SYSR1
&SYSR2
&SYSR3
&OEMR1
&OEMR2
&OEMR3
21
Installation system
each system:
Active IPL-set
Inactive IPL-set
6 model 9 vols
6 model 9 vols
End-user systems
The new design, rename rules (1)
 Dsnames in the REF-set need to be renamed when they are copied to the Level
Set.
 Standard processing: rename removes the version release qualifier, i.e.
REFN.T119A.OPS.LOAD
SYSN.OPS.LOAD
REFN / SYSN dataset:
one qualifier dropped
REF1.T113A.ZOS.LINKLIB
SYS1.LINKLIB
22
REF1 / SYS1 dataset:
two qualifiers dropped
The new design, rename rules (2)
The standard rename does not support two or more releases side by side.
 Keep-Qualifier processing: rename keeps the version release qualifier, i.e.
REFN.T119A.OPS.LOAD
SYSN.T119A.OPS.LOAD
 Keep-Qualifier processing is never done for SYS1 datasets:
REF1.T113A.ZOS.LINKLIB
SYS1.LINKLIB
23
Agenda
 Terminology
 The old situation
 The new design
 The Product Catalog in TDSL
 Non-uniform software stacks
 TDSL Product Selection & Generate (build) processes
 The roll-out (restore) process
 VSAM challenges
 zFS on the installation system
24
The TDSL Product Catalog
 TDSL application handles software deployment on the installation system.
 TDSL Product Catalog describes all products and the datasets belonging to each
product in SQL tables.
 Each product (and version plus maintenance level) is identified by an 8-char
acronym.
 Product Catalog has attributes at the level of the product and at the level of the
dataset.
 Products can be duplicated in the Product Catalog to apply maintenance
25
The TDSL Product Catalog, screenshot
26
The TDSL Product Catalog, screenshot
LR indicator
Filter line provides a fast
navigation facility
More attributes on the right
not in this screenshot
Each product is identified by
an 8-char acronym
27
The TDSL Dataset List screenshot
28
The TDSL Product Catalog, dataset attributes
 Dataset Type:
INSTALL
SMP
DLIB
TARGET
VTARGET
UTARGET
NODIST
product installation jcl
the CSI clusters, SMPPTS, SMPMTS, SMPLOG, etc.
SMP dlibs, 100% inert for TDSL
SMP/E target library or other distributable dataset
VSAM target dataset
z/fs linear, always keeps version-release qualifier for automount
not distributed, but it contributes to the PROGxx member
 APF
dataset should be APF on end-user system
 LNK
dataset should be link-listed on end-user system.
29
The TDSL Product Catalog, dataset attributes, continued
 DSTVOL:
a symbolic volser like &SYSR1, &SYSR2 ...
In the dataset list of the z/OS 1.13 A product:
&SYSR1
…
REF1.T113A.ZOS.NUCLEUS
&SYSR1
REF1.T113A.ZOS.LINKLIB
&SYSR1
REF1.T113A.ZOS.IMW.SIMWSDCK
&SYSR2
REF1.T113A.ZOS.OMEG.TKANMOD
&SYSR3
…
&SYSR2
&SYSR3
Why fix the location of target datasets on the Level-set?
- to ensure SYS1.NUCLEUS etc. on &SYSR1 volume
- fewer catalog updates on the end-user system
31
&OEMR1
The TDSL product catalog, screenshot of z/OS 1.13 dataset list
32
Agenda
 Terminology
 The old situation
 The new design
 The Product Catalog in TDSL
 Non-uniform software stacks
 TDSL Product Selection & Generate (build) processes
 The roll-out (restore) process
 VSAM challenges
 zFS on the installation system
33
Non-uniform software stacks (1)
swap validation
production
systems
test systems
development
systems
production K-systems
Systems using the same software stack:
34
EB systems
EF systems
SSE systems
K systems
Installation system
Non-uniform software stacks (2)
 Why not deploy all software on all systems?
Procurement dept. Licensing all software on all systems needlessly expensive.
Audit dept. libraries in APF list for software products that will never be used.
 Why not deploy software per-system?
Hard to maintain the multitude of parmlib members (i.e. PROGxx members)
Many different combinations to validate before a system upgrade.
 A solution in-between is needed.
All software will be present on all systems, but we make the libraries APF &
linklisted only on those systems where it is licensed.
To do this we add a where-used attribute to each product in the product catalog.
35
Non-uniform software stacks (3)
7 flags attributed to each product, each flag is on (used in this region) or off (not used in region)
SSE
EF
EP
ENL
K
BSOF
SY90
36
PROGEG
PROGEF
PROGEP
PROGEN
PROGEK
PROGEB
PROGES
the libraries of product CORT660A will be included in
PROGEG, PROGEF, PROGEP, PROGEN and PROGEB
but not in PROGEK or PROGES
Creating PROG members
Why create PROG members with a program?
 Software installers know what datasets need to be APF or Link-listed,
it’s in the product installation instructions -> can be entered in TDSL.
 We keep lists of datasets we don’t distribute, but that need to be in the PROG
member.
 TDSL has al the info it needs to create APF lists and Link lists for the End-User
systems.
37
Creating PROG members (2)
38
Creating PROG members (3)
39
Agenda
 Terminology
 The old situation
 The new design
 The Product Catalog in TDSL
 Non-uniform software stacks
 TDSL Product Selection & Generate (build) processes
 The roll-out (restore) process
 VSAM challenges
 zFS on the installation system
40
TDSL Generate steps
 Choose software products to go onto a Level-Set.
 An SQL query finds all unique DSTVOLs
 Engineer specifies the ‘real’ volsers and device numbers for each symbolic
DSTVOLs,
i.e. &SYSR1 will be build on real volser L13A11
 The dialog produces jcl to build the LS with the selected software products.
Only the TARGET datasets will be copied to the LS
41
The TDSL Generate, screenshot
43
The treatment of link list datasets, space allocation
The problem:
 ALL REF-Set libraries have secondary space so SMP APPLY won’t fail with abend
x37.
 On the Level-Set, Link list libraries should have zero secondary allocation, to
prevent an extent being added when someone adds/updates a member.
The solution:
 DFDSS dataset copy will copy every library into a single extent.
 Copy with ALLDATA & ALLEXCP will preserve the allocated space.
 A homegrown utility clears the secondary allocation value in the VTOC of the
Level-Set.
44
Agenda
 Terminology
 The old situation
 The new design
 The Product Catalog in TDSL
 Non-uniform software stacks
 TDSL Product Selection & Generate (build) processes
 The roll-out (restore) process
 VSAM challenges
 zFS on the installation system
45
Roll-out on the end-user system
 Each sysplex has an active and inactive IPL-set with associated master catalogs:
ACTIVE
MASTER CATALOG
ACTIVE
IPL-SET
INACTIVE
IPL-SET
IPCAA1
IPCAB1
CATALOG.PLXP.IPCAA1.MASTER
INACTIVE
MASTER CATALOG
CATALOG.PLXP.IPCAB1.MASTER
 Inactive IPL-set is formatted, ICKDSF INIT
 New master catalog is delete/defined
 A dump is restored onto the inactive IPL-set
 REPRO NOMREGECAT of the active master catalog to the new one
 Inactive master catalog is fixed for new- or changed datasets
 IPL from the inactive IPL-set.
46
This slide just for the notes
47
Agenda
 Terminology
 The old situation
 The new design
 The Product Catalog in TDSL
 Non-uniform software stacks
 TDSL Product Selection & Generate (build) processes
 The roll-out (restore) process
 VSAM challenges
 zFS on the installation system
48
The Challenges of VSAM datasets
 VSAM datasets need to be cataloged for all processing.
Interferece with the driving system when copying VSAM datasets w same name.
 All IDCAMS processing requires a correct VSAM ‘back-pointer’
Catalog volume
Volume with VSAM dataset
forward pointer
ICF catalog
VVDS
SYSV.OMVS.ROOT.ZFS
VVR
back pointer
VSAM dataset SYSV.OMVS.ROOT.ZFS
49
The distribution of VSAM datasets (1)
On the installation system:
 Define a new ICF usercatalog on one of the LS volumes
 Define an alias for a new temporary HLQ with RELATE to the new usercat, i.e. SYSDEF
 Copy the VSAM datasets to the LS with RENAMEUNCONDITIONAL, REFx  SYSDEF
 Level-set is dumped to tape.
On the end-user system:
 Define a new ICF usercatalog on one of the IPLset volumes
 Delete/define the SYSDEF alias
 Restore VSAM clusters with the SYSDEF HLQ from tape (no rename)
 IDCAMS ALTER all SYSDEF dsnames in the user catalog to the normal HLQ for VSAM
50
The distribution of VSAM datasets (2)
 An alias with symbolic relate accomplishes the same ‘volume switching function’
as the symbolic volumes in the catalog do for non-vsam datasets:
DEFINE USERCATALOG –
(NAME(CATALOG.SYSV.IPCAA1) –
ICFCATALOG –
VOLUME(IPCAA2) –
CYLINDERS(1 1)
DEFINE USERCATALOG (NAME(CATALOG.SYSV.IPCAB1) –
ICFCATALOG –
VOLUME(IPCAB2) –
CYLINDERS(1 1)
DEFINE ALIAS(NAME(SYSV) –
SYMBOLICRELATE(CATALOG.SYSV.&SYSR1))
51
IPL-set A
IPL-set B
The distribution of VSAM datasets (3)
 LISTCAT ENTRY(SYSV) ALIAS ALL
ALIAS --------- SYSV
IN-CAT --- CATALOG.PLXP.IPCAA1.MASTER
HISTORY
RELEASE----------------2
ASSOCIATIONS
SYMBOLIC-CATALOG.SYSV.&SYSR1
RESOLVED-CATALOG.SYSV.IPCAA1
52
Agenda
 Terminology
 The old situation
 The new design
 The Product Catalog in TDSL
 Non-uniform software stacks
 TDSL Product Selection & Generate (build) processes
 The roll-out (restore) process
 VSAM challenges
 zFS on the installation system
53
z/FS file systems on the installation system (1)
Risks of APPLY on statically mounted file systems:
 If the wrong file system is mounted, an APPLY job could update USS components
belonging to a different SMP/E zone than the OS libraries.
 Potentially there could be data not hardened in the file system when a R/W
mounted file system is copied with dfDSS.
How automounted file systems can help:
 USS path in dddef’s can contain a zFS qualifier that makes automount mount the
file system corresponding to that zone.
 Automount will unmount file systems that have not been accessed for some
amount of time.
This reduces the time window during which uncommitted data can exist.
54
z/FS file systems on the installation system (2)
/etc/auto.master:
/SERVICE/DB2/DB2BASE
/etc/service_db2_db2base.map
/etc/service_db2_db2base.map
name
type
filesystem
mode
duration
delay
security
setuid
*
ZFS
REFN.<uc_name>.DB2.DB2BASE.ZFS
rdwr
60
10
Yes
no
DDDEF SDSNABIN in target zone TD1010A contains USS path:
'/SERVICE/DB2/DB2BASE/T10101A/bin/IBM/'
56
z/FS file systems on the installation system (3)
The SMP/E APPLY job:
//SMPCNTL DD *
SET BDY(TD1010A) .
APPLY SELECT(…) .
When SMP/E opens a file in the SDSNABIN directory it will cause the automount
daemon to mount
REFN.T1010A.DB2.DB2BASE.ZFS
onto mount point
/SERVICE/DB2/DB2BASE/T10101A
Automount replaces <uc_name> in the map file with the path-qualifier from the
SMP/E dddef path to obtain the zFS dsname:
i.e.
57
Filesystem
REFN.<uc_name>.DB2.DB2BASE.ZFS
becomes
REFN.T1010A.DB2.DB2BASE.ZFS
z/FS file systems on the installation system (4)
 A Loose End
How to distribute large zFS file-systems that need SMS extended format data
class attribute?
An SMS managed volume in the IPL-set is most unpractical.
58
The End
 Question Time.
59
Reporting on changes in the field
 At levelset generation time a hash is created for all datasets.
 Reporting is available to identify all changes between the levelset and a resident set
deployed in a user system.
64