ALM226
SAP Migrations Made Easy
Heiko Zuerker
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
Agenda
Rockwell Automation’s SAP Landscape
Why a different Process?
OS Migration High Level
OS Migration in Detail
Lessons Learned
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
2
Rockwell Automation
•
•
•
•
•
•
•
Manufacturer of industrial automation control and information solutions
World headquarters in Milwaukee, WI
Annual sales about $4.3 billion
Serving customers in more than 80 countries
About 19,000 employees
Over 15,000 internal SAP users
Over 5000 external SAP users (distributors)
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
3
SAP Single Global Landscape
ERP
R/3
(R)
Production
(P*0)
PR0
Internal External TREX
Portal
Portal Ext.Portal CRM
(E)*
(P)*
(C)
(Q)
PE0
PP0
SM1 (Solution Manager)
PQ0
PC0
BW 7.0
(J)
SCM
APO
(A)*
SRM
(S)
PJ0
PA0
PS0
PW0 (WebAS) CUA/CCMS
PCDE
(B)
PB0
XI
(X)
GRC Process
Control
(D)
MDM
(M)
GTS
(T)
SNC
(N)
Access
Control
(V)
PX0
PD0
PM0
PT0
PN0
PV0
LS0 (System Landscape)
Environment
Prod Sup
(P*1)
PR1
PE1
PP1
PQ1
PC1
PJ1
PA1
PS1
PB1
PX1
QA
(Q*0)
QR0
QE0
QP0
QQ0
QC0
QJ0
QA0
QS0
QB0
QX0
QA1
(Q*1)
QR1
QE1
QC1
QJ1
QA1
QS1
QB1
QX1
Dev0
(D*0)
DR0
DE0
DC0
DJ0
DA0
DS0
DB0
DX0
Dev1
(D*1)
DR1
DE1
DC1
DJ1
DA1
DS1
DB1
Training
(T*0)
TR0
TE0
TC0
TJ0
TA0
TS0
Sandbox
(S)
SR0
SE0
SC0
SJ0
SA0
SS0
DP0
SM2 (Solution Manager)
DQ0
PW2 (WebAS) CUA/
CCMS
JD0 (JDI)
APO
Mobile
Live
Infrastr Cache
ucture
(L)
(O)
SCM
TREX
APO
SRM/
Gateway ECC5.0
(G2*)
(U)
TREX
CRM
(H)
TREX
ERP 6.0
(F)
TREX_48
PU0
PH0
PF0
G21
TREX_48
PU1
PH1
PF1
QY0
G22
TREX_48
QU0
QH0
QF0
QL1
QY1
G23
TREX_48
Qu1
QH1
QF1
DL0
DY0
G24
TREX_48
DU0
DH0
DF0
BIA
(Z)
SD
(Y)
PL0
PZ0
PY0
G20
PL1
PZ1
PY1
QL0
LS6 (non Prod SLD)
PM1
PT1
PN1
QM0
QT0
QN0
QM1
QT1
QN1
DM0
DT0
DN0
DX1
DM1
DT1
DN1
DL1
DY1
G25
TREX_48
DU1
DH1
DF1
TB0
TX0
TM0
TT0
TN0
TL0
TY0
G27
TREX_48
TU0
TH0
TF0
SB0
SX0
SM0
ST0
SN0
SL0
SY0
G26
TREX_48
SU0
SH0
SF0
LS2 (System Landscape)
QD0
DD0
SD0
QV0
DV0
SV0
SO0
JD2 (JDI)
Production databases are currently 12 TB with 25% growth per year.
Everything we do in our SAP Environment,
we do 8 times…
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
4
Why Migrate?
• Reached capacity limits on existing hardware
• Reached end of life for existing hardware
• Leveraging commodity hardware decreases maintenance costs
– Easier to reuse/repurpose hardware
– Easier to switch hardware vendors
• Completed full evaluations of different server architectures
– SAP is fully supported on Linux operating system
– x86 based architecture provided a significant price difference in the cost per “SAPS”
• All other components (i.e. Veritas Cluster, Control-M, HPOMS, etc.) supported on Linux
• Not all SAP applications are supported on Solaris x86
– NW6.40 Java stacks are not supported
– Adobe document services are not supported
• TREX 7.10 only supported on Linux x86 and Microsoft Windows
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
5
SAP Production Environment Pre & Post
Sun Solaris 9
SUSE Linux Enterprise Edition 10
Sun V490
4 CPUs (2 Cores each) 1.35 - 1.8 GHz / 32GB RAM
HP Blades BL460c
2 CPU’s (4 Cores each) 2.93 GHz / 96GB RAM
4 Servers for Central Instances (Clustered)
2 Servers for Central Services (Clustered)
(SCS, ASCS, IPC, TREX)
1 Server for TREX’s
24 Servers for Dialog instances
16 Servers for Application Servers
(Central and Dialog Instances)
57 Central and Dialog Instances
39 Central and Dialog Instances
(retired 18 Instances)
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
6
SAP’s Standard OS Migration Process
• SAP recommends and supports the use of the “Heterogeneous System
Copy” for OS/DB Migrations
– Note 82478 - SAP system OS/DB migration
• Procedures shown deviate from SAP’s standard process
• SAP AGS
– Worked with SAP account team to ensure continued support after the OS migration
– Not in violation of any support, maintenance or warranty agreements
• SAP Services
– Scheduled several OS / DB migration checks
• http://service.sap.com/osdbmigration
– Scheduled several Go-Live checks
• http://service.sap.com/goinglivecheck
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
7
Why did we not use the Standard Process?
• Fast track OS Migration
– Estimated time 1 hour executing the migration, plus 3 hours of post steps (mainly SGEN)
– Minimal changes to the system configuration  less risk
• Heterogeneous System Copy  too time consuming
–
–
–
–
–
Estimated time, including post steps, 12 hours per system
Each basis resource is able to complete 2 systems in parallel. More can be dangerous.
12 production systems and only 8 basis resources
Basis resources need to be staffed for post-migration support
Outage of all production systems for at least 24-36 hours
• Requires complete re-installation of dialog instances  too time consuming
– All instance specific settings need to be reapplied (certificates, SSL, proxymap, memory, threads, etc.)
• Lots of “manual” labor  high risk of errors/mistakes, especially during night shifts
• Many of the tasks need to be done anyway, no matter which process
–
–
–
–
–
SAP profile settings
Java parameters (Configtool)
SAP kernel and crypto library updates
SGEN, SPAD, SM69
Architecture changes
• Testing requirements are the same
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
8
What did we do in 16 hours?
•
•
•
•
•
•
Migrated all production SAP instances to Linux
Upgraded the 12 production Oracle databases from 10.2.0.2 to 10.2.0.4
Separated ASCS/SCS (enqueue and message server ) from central instance, even on NetWeaver 6.40
Moved ASCS/SCS into clustered central services layer
Installed enqueue replication server for both SCS and ASCS
Standardized all start profiles
– Process naming
– SAPCPE corrections (mostly for upgraded systems)
– Gateway on SCS/ASCS
• Standardized all instance and default profiles
–
–
–
–
Minimum heap/extended memory
Minimum work processes
Password complexity
Timeouts
• Standardized Java Configtool settings
– Minimum heap
– Minimum threads
– Java VM settings
•
•
•
•
Standardized Kernel and CryptoLib versions
Moved all NetWeaver 6.40 systems to EX2 kernel and Oracle 10 instant client
Changed IP addresses of all SAP systems (each instance got its own VLAN)  DR & virtualization
Validated all business-critical transactions
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
9
Fast track Migration Process High Level
SAP
Kernel
Upgrade
Database
Client
Upgrade
Java
Runtime
Upgrade
Config
Settings
Fasttrack
OS
Migration
Only works when going from one flavor of Linux/UNIX
to another flavor of Linux/UNIX
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
10
Why does it work?
• ABAP code is platform independent (SGEN required)
• Java code is platform independent
• SAP’s Java stacks use some additional binaries (i.e. ADS), but
– The binaries for all supported operating systems are stored in the database
– Instance just needs to be told to extract the correct binaries during bootstrap
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
11
Automate, Automate, Automate
•
•
•
•
•
•
•
•
•
Automate as many tasks as possible
Scripts can be executed by one individual, in parallel, on many systems
Ensures consistency
Ensures quick execution
Ensure the team understands what the scripts do
Every basis resource should do at least one “manual” migration
Almost all steps shown in this presentation are automated
Many of the scripts created can be re-used during patches or upgrades
Focus on the benefits that will be realized during the production
migration
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
12
Main Custom Automation Tools we used
• saprun.pl
– Perl script which allows execution of Unix commands on multiple SAP instances,
systems or environments in parallel, with a minimal amount of effort on the user’s
side. It leverages “expect” to control the SSH sessions.
– Example to shut down all instances in the production environment :
saprun.pl -t appservers -c "stopsap #INSTANCE#" -e PROD
• j2eebatchconfig.sh
– Automated configuration of Configtool settings via templates
– Leverages Configtool’s BatchConfig functionality
– https://cw.sdn.sap.com/cw/docs/DOC-108724
• sap_profiles.sh
– Ensure that the DEFAULT.PFL, instance profiles and instance start profiles follow
company standards (password complexity, file format, variable use)
– Defines minimum values for certain settings (heap, buffers, dialog wp’s)
– Has the optional feature to email the profiles as a CSV file and re-import them later.
This makes mass changes to multiple instance profiles a breeze.
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
13
What we tested?
• Anything I/O related
–
–
–
–
–
File
Email
FTP
Printing
Job Scheduling
• Interface types, not all the interfaces
• All scripts
– Scripts for interfaces
– Administration scripts
• Business-critical transactions (system validation)
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
14
Migration Detailed Timeline
T-2 weeks
T-1 week
Saturday 8:00pm
9:30pm
• Install Hardware
• Install Operating System
• Get SAP Licenses
• Clean queues (daily)
• Initial File Sync
• Export Profiles to CSV
• Disable all PI Communication
Channels
• System Cleanup Tasks
(Queues)
• Lock all Users except Basis
• Shutdown (~ 30 minutes)
10:00pm
10:30pm
Sunday 3:00am
6:00am
• DNS changes (~ 30 minutes)
• Disable Database Archive
Logs
• Database Upgrades
(~ 4 hours)
• OS Migration SAP Systems
(~ 1 hour)
• Post Steps (~ 2 hours)
• Enable Database Archive
Logs
• Final System Restart
(~ 30 minutes)
• Unlock all Test Users
• Enable all PI Comm Channels
7:00am
11:00am
Noon
T+2 weeks
• System Validation
(Business Critical
Transactions)
• Unlock all Users
• Productive Operations
Resume
• Hypercare Support
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
15
The Migration Process – Preparation / Uptime
• Prepare the new hardware
– Install and configure the new operating system
– Create necessary SAP directory structure, mount points, etc.
– Install Java runtime
• Initial file sync
– Copy (rsync) all files from the local file system to the new servers ( /usr/sap/SID )
• Export profiles to CSV
– Clean up the profiles in Excel
– Copy CSV files to staging directory
• SAP Licenses
– Request the licenses for the new hardware-keys
– Import manually via Visual Admin into Java-only systems
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
16
The Migration Process – Downtime
•
•
•
•
Stop all PI communication channels
Clean all queues
Shutdown SAP
Final file sync
– Execute rsync again to get the delta from last rsync
• Re-Import the cleaned-up profiles from the CSV files
• Remove all log files from every SAP instance
–
–
–
–
–
–
–
/usr/sap/SID/*/work/*
/usr/sap/SID/SYS/global/SLOGJ*
/usr/sap/SID/*/log/SLOG*
/usr/sap/SID/*/data/rslgsta
/usr/sap/SID/*/j2ee/cluster/server?/log/*
/usr/sap/SID/*/j2ee/cluster/dispatcher/log/*
/sapmnt/SID/global/*LG
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
17
Replacing Binaries
• Replace database client binaries
• Replace the SAP kernel & crypto libraries
–
–
–
–
–
Delete contents of the /sapmnt/SID/exe directory
Extract SAPEXE.SAR
Extract SAPEXEDB.SAR
Extract IGSEXE.SAR (7.x Systems only)
Extract the crypto library
• Java kernel SAP NetWeaver 6.40 and older
– Delete contents of the /sapmnt/SID/exeU directory
– Extract SCS.SAR (from correct support package)
– Extract SCSCLIENT.SAR (from kernel patch)
• Verify all binaries were replaced
– find -type f -exec file {} \; | grep SPARC
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
18
Application Server (Instance) Migration
• NetWeaver 7.x
– Delete contents of instance exe directory
• NetWeaver 6.40 and older Java stacks
–
–
–
–
–
Extract SCSCLIENT64UC.SAR (from SP) into os_libs directory
Copy os_libs/jstartup.jar into cluster/dispatcher/bin/kernel directory
Copy os_libs/jstartup.jar into cluster/serverX/bin/kernel directory
Copy exeU/libjmon.so into os_libs directory
Extract IGSEXE.SAR into igs/bin directory
• All Java stack versions
– Delete the instance.properties.vmprop file to force a re-generation
– Set element.resynch=force in bootstrap.properties
• All NetWeaver stacks
– Recreate password file for SNC PSE
– Clean up /usr/sap/tmp/dev_*
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
19
Java Configuration on the Central Instance
• Update the dispatcher and Java parameters depending on the JDK
– Note 718901 - How to Change the JDK of the J2EE Engine
– Note 709140 - Recommended JDK and VM Settings for the WebAS630/640/7.0
– Note 723909 - Java VM settings for J2EE 6.40/7.0
• Set new JDK for SDM
– Delete SDM/program/config/sdm_jstartup.properties.vmprop
– Note 743583 - How to change the JDK of SDM
sdm.sh newjdk "javahome=$JAVA_HOME"
• Java stacks older than NW7.1 store the operating system type in the
database
– Update via Configtool
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
20
Updating the Operating System in Configtool
Switch to
Configuration Editor Mode
Where to find os.name values?
Install SAP once on new platform
to compare settings
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
21
ABAP and ABAP+Java Licenses
• Use the command line tools saplicense & saplikey to remove all old
licenses and then add the new ones
• SAP NetWeaver 7.x
– saplikey pf=/path/to/profile -delete \* \* \*
– saplikey pf=/path/to/profile –install /path/to/license
• SAP NetWeaver 6.40
– Need to use “expect” script to remove licenses or just do it manually
– saplicense -install ifile=/path/to/license
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
22
Post Configuration
•
•
•
•
•
Start the systems
Execute SGEN and generate all objects for all components
Import the changed profiles into RZ10
Update custom OS commands in SM69 for new platform
Update OS Commands for your output management system in SPAD
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
23
TREX Migration
•
•
•
•
•
Can not export/import indexes when Endian-ness (byte-order) changes
New installation and full re-indexing required
Re-indexing report does not support parallel processing
TREX tuning can improve re-indexing performance significantly!
ECC TREX had 1.4 million documents, re-indexing took about 24 hours
Postponed TREX migration to 3 day upgrade weekend,
due to time required for re-indexing.
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
24
Migration Done! Time for System Validation!
• Validate business-critical transactions
• Validate interfaces
• Validate printing, emails, faxes
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
25
Lessons Learned
•
•
•
•
•
•
Test and get sign off on all interfaces - Do NOT just test the “types”
Test all I/O related processes
Watch out for character sets, some binaries may have different defaults
Stage as much as you can
Automate as much as you can
Ensure basis has a clear understanding of the underlying concepts
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
26
Closing Comments
• We did this because
–
–
–
–
–
•
•
•
•
•
•
Have the ability to re-use many components of this process
Had the ability to execute the migration within a standard maintenance window
Had the ability to include architecture changes due to time saving
Had the ability to include Oracle upgrades due to time savings
Less risk due to minimal configuration changes
Test, test and then test some more
Don’t be afraid to try out new ideas
Use the opportunity to implement architecture changes
Use the opportunity to standardize
Use the opportunity to clean up the landscape
Schedule SAP OS/DB migration check and Going-Live checks
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
27
Questions ?
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.
28
Copyright © 2010 Rockwell Automation, Inc. All rights reserved.