HP-UX
Patch
Program
HP-UX Customer
Patch Panel
HP-UX Successful Patching
Strategies
1
HP-UX
Patch
Program
HP World – HP-UX Customer
Patch Panel
presented by:
HP (Laurie Schoenbaum)
Nestlé (Terri Mando)
Brigham Young University (John Payne)
Philips Research Labs (Donie Collins)
Beckman Coulter ( Chris Maehara)
2
HP-UX
Patch
Program
Nestlé
Presented by: Terri Mando
3
HP-UX
Patch
Program
Nestlé
background
background
Nestlé is Switzerland's largest industrial
company and the world's largest food
company.
Nestlé USA headquartered in Glendale,
CA
technology used to stay competitive in the
market place
135 HP servers
HP-UX 10.20, 11.0 and 11.11
D, K, L, N, and V-class servers
Located in Arizona, California, and Ohio
4
HP-UX
Patch
Program
Nestlé
background
background
(Cont.)
all remote system administration
servers assigned to application
groups
Per application: Test, Development, QA,
and Production servers
SA assigned to application group
24x7x356 support
primary and backup
5-16 servers per SA
CSS support on SAP, mostly PSS
support
customized ASE, no onsite support
5
HP-UX
Patch
Program
Nestlé
patch strategy
patching strategy
twice a year proactive patching
stringent formal change management
process
phased rollout
“12 step program”
patch depot management strategy
one patch depot per OS
use of make_bundles
use of “cleanup”
6
HP-UX
Patch
Program
patching strategy
patch selection
Nestlé
patch strategy
Mission Critical (CSS) support
contract delivers proactive patch
bundles quarterly
“conservative” change strategy
(MCSCM) custom patch bundles are
“delta” bundles
added to existing patch depots quarterly
only select patches applicable to
environment
patch dependencies handled by HP
support (RASE)
7
HP-UX
Patch
Program
patching strategy
patch warnings
Nestlé
patch strategy
HP support personnel track and
provide recommendations on patch
warnings
recommendations are individually
assessed for applicability to environment
rarely has a patch been removed due
to a patch warning
handled in next proactive patch cycle
8
HP-UX
Patch
Program
change management
Nestlé
change
management
documented change management
process
change requests required
formal approval process
business critical systems have a 4 hour
maintenance window
9
HP-UX
Patch
Program
patching strategy
patch application/12 step program
Nestlé
patch
application
pre patching
change management
scheduling conflicts?
health check
commit patches and cleanup SD log files
swinstall –p (review logs, resolve issues)
patch application
stop applications
swinstall (Do it!)
10
HP-UX
Patch
Program
patching strategy
patch application/12 step program
Nestlé
post patching
patch
application
review logs
health check
verify applications
change management
11
HP-UX
Patch
Program
Nestlé
conclusionsrecommendations
written procedures
provides consistency
conclusions
pre-patching, patching, post-patching
plan (12 step program)
provides a framework
allows tasks to be automated
minimize time spent patching
proactive patching!!
definition of success is not having a
problem
12
HP-UX
Patch
Program
conclusionsrecommendations
Nestlé
available on the Interex Patch SIG
website
conclusions
(http://www.interex.org/advocacy/mcgs/patch/
index.html)
“Patching: A 12-Step Program”
patch_preview.sh
patch_do-it.sh
“Patch Depot Management” document
13
HP-UX
Patch
Program
Brigham Young
University
Presented by: John Payne
14
HP-UX
Patch
Program
Brigham
Young
University
background
background
Brigham Young University has grown
from a small pioneer academy to one
of the world’s largest private
universities, with more than 29,000
students from 100 countries
IT supports payroll, student
information, courses online and other
content related to the university
50 HP-UX systems
HP-UX 10.20, 11.0 and 11.11
A500/rp2470s, rp8410, K-class, R-class,
L-class, N-class
15
HP-UX
Patch
Program
background
(Cont.)
Brigham
Young
University
background
1primary system administrator
24x7 with 4 hour response
no on-site HP support
16
HP-UX
Patch
Program
patching strategy
philosophy
Brigham
Young
University
patch strategy
quarterly proactive patching
HP-UX 10.20 exception
based on release of SupportPlus media
goal: no unscheduled downtime
switch from reactive to proactive
maintenance to improve
supportability
3 month test cycle in lab before
rolling to production
17
HP-UX
Patch
Program
patching strategy
patch warnings
Brigham
Young
University
patch strategy
QPK bundles reduce probability of a
patch warning
security patches may be applied
reactively
18
HP-UX
Patch
Program
patching strategy
patch application
Brigham
Young
University
clusters of redundant applications
maximizes system availability
patch strategy
non-redundant applications require
off hour planned outages
problems generally logged with the
ITRC call manager
19
HP-UX
Patch
Program
change management
Brigham
Young
University
change
management
formal change request process
all system changes are logged
20
HP-UX
Patch
Program
Brigham
Young
University
conclusions
conclusionsrecommendations
quarterly proactive patching as
virtually eliminated unscheduled
downtime and reactive patching
quality of patches in QPK helps to
stabilize systems – reduces risk
adequately test before rolling to
production
eliminating the need for system
administers from working nights
would be a plus!
21
HP-UX
Patch
Program
Philips Research Labs
Presented by: Donie Collins
22
HP-UX
Patch
Program
Philips
Research
Labs
background
background
division of Philips Electronics
technical computing support for 3000
users
1600 are researchers of various sciences
1400 are from product division R&D
departments
work in partnership with other IT
departments within Philips
23
HP-UX
Patch
Program
Philips Research ICT Infrastructure:
Server Based Computing (NXA)
Unix batch- and computeservers for compute and
memory intensive CAD
applications
load balancing &redundancy
Unix login-server
(gateway to Unix
for PC desktops)
load balancing &redundancy
Windows Terminals Servers
for PC based applications
load balancing &redundancy
NFS/CIFS
Unix Admin/license servers
load balancing &redundancy
H.A.
GigaBit Ethernet
file
servers
Ethernet
100BaseT/10BaseT
Unix Backup servers
Network
switches
X-terminal
(decreasing)
Windows NT/2000 PC
with X-server
Laptop W2000
with X-server
24
HP-UX
Patch
Program
background
(Cont.)
Philips
Research
Labs
background
150 HP9000 servers and workstations
standard system models and
configurations
10 system administrators
Personalized System Support (PSS)
HP on-site hardware engineer
99.97% uptime goal
25
HP-UX
Patch
Program
patching strategy
philosophy
Philips
Research
Labs
patch strategy
if its not broken, don’t fix it; reactive patch
philosophy
execute security_patch_check weekly
proactive with security patches
keep all systems at same patch level per OS
one patch depot per OS
strive for only 3
patches; highest rated
patches
use QPK bundles to reduce individual point
patches
26
HP-UX
Patch
Program
patching strategy
(cont)
Philips
Research
Labs
patch strategy
moving in direction of proactive
maintenance with a “Enterprise
Technical Server Environment (ETSE)”
includes QPK, HWE bundles and TCOE
6 month delivery cycle
reduces management of point patches
27
HP-UX
Patch
Program
patching strategy
patch selection
Philips
Research
Labs
use IT Resource Center
patch strategy
subscribe to patch digest
use patch database to download patches
ITRC tools identify dependencies
QPK and HWE bundles
28
HP-UX
Patch
Program
patching strategy
patch warnings
Philips
Research
Labs
patch strategy
ITRC tools send proactive notification
of patches with warnings
warnings are examined for applicability
and action is taken
do nothing
turn off functionality
install superseding patch
remove patch
29
HP-UX
Patch
Program
patching strategy
patch application
Philips
Research
Labs
patch strategy
SD-UX tools manage depots and
installation
3 step rollout
install on test system
roll to a few production systems
complete rollout
system redundancy reduces planned
and unplanned downtime
30
HP-UX
Patch
Program
change management
Philips
Research
Labs
proactive configuration management
change
management
cfg2html tool
in-house monitoring tools and EMS
component monitoring
31
HP-UX
Patch
Program
Philips
Research
Labs
conclusions
conclusionsrecommendations
successful with ITRC tools and
security_patch_check tool for
patch selection
patch proactive notifications
looking to ETSE to reduce system
administration time for patch management
make better use of QPK
take advantage of the continuous
improvements with ITRC patch tools
patch installation is labor intensive and time
consuming across 150 systems
32
HP-UX
Patch
Program
Beckman Coulter
Presented by: Chris Maehara
33
HP-UX
Patch
Program
Beckman
Coulter
background
background
Beckman Coulter makes products that are
used in hospital laboratories, physicians'
offices and group practices. The company
provides a variety of systems for medical
research, drug discovery and biotechnology
applications.
business supported by various HP-UX and
NT servers
Oracle applications, SAMBA, and Veritas for
system backup
MC/ServiceGuard used for high availability
and to reduce planned downtime.
34
HP-UX
Patch
Program
Beckman
Coulter
background
(Cont.)
13 HP-UX servers
L-class, N-class, two V2600s
HP-UX 11.0 and 11.11
background
2 system administrators
24x7 Critical System Support (CSS)
No onsite support
100% uptime goal
35
HP-UX
Patch
Program
patching strategy
philosophy
Beckman
Coulter
patch strategy
quarterly proactive patching
rolling upgrades using MC/ServiceGuard
4 stage rollout
master depot of patches for each supported
OS release
cleanup command used
patches kept for 1 year
text file kept in separate directory for all patches
ever applied
standard configurations minimize
complexity
36
HP-UX
Patch
Program
patching strategy
patch selection
Beckman
Coulter
patch strategy
CSS contract delivers proactive patch
bundle
CPM (ITRC tool) delivers proactive
notifications
CPM sends notifications of newly
released patches based on system
configurations
weekly review of CPM notifications
patches added to patch depot
matrix of patch dependencies
use of SD master patch depot minimizes
issues with patch dependencies
37
HP-UX
Patch
Program
patching strategy
patch warnings
Beckman
Coulter
patch strategy
patch warnings reviewed prior to
patch application
generally, patches with warnings left
as is
38
HP-UX
Patch
Program
patching strategy
patch application
Beckman
Coulter
patch strategy
perform rolling upgrades
copy all patches to be applied to a
software depot, regardless of whether
or not the patches are from a
download or a CD.
ensures no corrupted patches
keep two versions of patch in depot
only latest patch will install
39
HP-UX
Patch
Program
patching strategy
patch application
Beckman
Coulter
patch strategy
use SD GUI to install
more user friendly
can make modifications if necessary
without exiting the operation
after installation, review log files
verify successful installation
verify configured
cleanup patches
40
HP-UX
Patch
Program
change management
Beckman
Coulter
change
management
change requests required
sign-off by business leads
MeasureWare and ITO monitors
systems and changes
41
HP-UX
Patch
Program
conclusionsrecommendations
Beckman
Coulter
conclusions
well planned/tested rollouts
regular scheduled proactive patch
applications
proactive better than reactive
always read “special installation”
instructions
do not “force install” a patch
use SD to resolve patch
dependencies
42
HP-UX
Patch
Program
all customers
summary
summary
all customers had some kind change
management process for patching
all customers did some level of testing of
patches prior to rolling into production
all customers are using some level of
proactive patching
customers used a combination of HP
support services, ITRC tools, and
SupportPlus patch bundles (QPK)
HA and/or redundant environments aid with
reducing downtime
security patches are “classed” differently
patches with warnings are rarely removed
from a system
43
HP-UX
Patch
Program
all customers
summary
questions?
HP (Laurie Schoenbaum)
Nestle (Terri Mando)
Brigham Young University (John
Payne)
Philips Research Labs (Donie
Collins)
Beckman Coulter (Chris Maehara)
44