Banner Virtualization
How Virtualization adds flexibility to
our production environment, and
enables new opportunities
Presented by: Brent Maynard, Myke Watts
Marshall University
April 9, 2013
Session ID 2853
Session ID 2853
1
Session Rules of Etiquette
• Please turn off your cell phone/pager
• If you must leave the session early, please
do so as discreetly as possible
• Please avoid side conversation during the
session
Thank you for your cooperation!
Session ID 2853
2
Introduction
• In this session, we hope to show you the
benefits we have gained by moving from a
physical 32bit architecture to a virtualized
64bit architecture.
Session ID 2853
3
Agenda
•
•
•
•
•
•
•
Where we started
Why a virtualized Environment
64bit benefits
Migration – Source code and database
Infrastructure
Cloning
Performance, Reliability, and Disaster
Planning
Session ID 2853
4
Where we started
.
Session ID 2853
5
Where we are – all virtualized
•
•
•
•
•
•
•
Two transactional database servers
Two ODS/EDW database servers
Four Banner instances
Four Middleware servers (INB, SSB, etc.)
High availability
Increased performance
Business continuity
Session ID 2853
6
Where we are – all virtualized
vSphere 10 node Cluster
Session ID 2853
7
Where we started
PHYSICAL (32 bit)
SSB
INB
BANNER
BANNER
BANNER
BANSTAGE
BANAUX
BANPROD
ODS/EDW
Session ID 2853
ODS/EDW
warehouse
warehouse
COGNOS
ARGOS
COGNOS
ARGOS
8
Background – 32bit Banner
• Redhat Enterprise RHEL5 32 bit
• Dell R710 dual socket quad-core with
12GBytes memory
• Dell Equallogic 6510x storage array
• 1.5 TBytes allocated storage
• Oracle 10g Database and Application
Services
• Typically we have ~250 concurrent Banner
Forms users
Session ID 2853
9
Background - continued
• Due to multi-core expansion over the past few years, we
decided to virtualize our INB/SSB Oracle Application
Services because of Oracle OAS licensing constraints
18 months ago.
Before
Now
MUSSB
MUINB
Dell dual
socket dual
core
Dell dual
socket dual
core
Vmware
Client
MUSSB
Vmware
Client
MUINB
4 cpus
4 cpus
VMware ESX host
Dual socket, quad core
Session ID 2853
10
Background - continued
• We have migrated to the 11g Oracle Fusion infrastructure
• Hopefully by the time of this presentation, Java 7 support.
ESX Host
Session ID 2853
OFM
Classic Domain
INB/SSB/DW
production
OFM
Basic Domain
BEP/WF/BRM
production
OFM
Classic Domain
INB/SSB/DW
preprod
OFM
Basic Domain
BEP/WF/BRM
preprod
11
Why Server
Virtualization?
.
Session ID 2853
12
Why did Marshall University Virtualize?
• $$$$$$
–
–
–
–
–
Session ID 2853
Reduced server expenditures
Reduced power requirements
Reduced cooling requirements
Service availability
Flexibility
13
Why did Marshall University Virtualize?
• At Marshall, the majority of our core
services are now running in the ESX
environment.
– Exchange
– Sharepoint
– Web (www.marshall.edu)
– Banner (Workflow, BRM, DegreeWorks,
ODS/EDW)
Session ID 2853
14
64 bit Benefits
.
Session ID 2853
15
Why 64bit?
• RHEL5 removed highmem constructs that
RHEL4 had allowed us to use to grant Oracle
large chunks of memory.
• RHEL5 32bit caused us some issues when
our pool space was consumed during heavy
use periods.
• With 64bit linux and Oracle, we can now
grant Oracle large chunks of memory (7.5
Gbytes) .
Session ID 2853
16
Migration from
32bit to 64bit
.
Session ID 2853
17
Migration
•
•
•
•
•
Install Oracle 11gR2 64 bit DB server
makefiles
Cobol
library paths
rsync
Session ID 2853
18
Migration – 11g database
• Install Oracle 11gR2 Enterprise Edition
• Allow Oracle Support to apply the latest
patch set
• Configure to connect to Grid Control
• Confirm the latest CPU patch was applied
• Allow the installer to create a test
database
• Install Oracle 11g Client (full client)
Session ID 2853
19
Migration – makefiles
• Download the Banner 8 makefiles from
Ellucian and choose the file that is specific
to your OS
• FAQ CMS-1361 Banner Example
Makefiles
Session ID 2853
20
Migration – Cobol
• The Pro*Cobol pre-compiler is not
included with the Oracle 11g Enterprise
Edition install. It is included with the 11g
Client and must be installed separately
• $COBOL=/opt/FJSVcbl
• $COBOL=/opt/FJSVcbl64
Session ID 2853
21
Migration – rsync
• To move the file system from the 32bit environment to
the 64 bit environment, we used the linux rsync
command.
• It takes approximately 4 hours to sync the file system.
• rsync benefits
– Maintains file dates
– Maintains file permissions and ownership
• File system ACL’s do NOT migrate, so those will need
to be applied manually if you have any set.
Session ID 2853
22
Migration – Database
•
•
•
•
•
•
Create new production database (empty)
Oracle export/import
Duration
Post import cleanup
BAN_ roles replaced by USR_ roles
Database ACL’s (workflow, smtp, http)
Session ID 2853
23
Migration – Database
• Use DBCA to create a new Banner database (general
purpose)
• Give it the same DB name as the previous Banner database
• Configure the new database to connect to Enterprise
Manager Grid Control
• Configure a listener
• Set appropriate locations for datafiles, logfiles, control files
• Choose the correct Unicode character set AL32UTF8.
Session ID 2853
24
Migration – Database
• The Oracle 11g Grid Control does not support
database cloning from 10g 32bit to 11g 64bit.
• Use either the Oracle import/export or datapump
utilities.
• DegreeWorks issues
– Even though the export was successful with no errors, the
subsequent import has missing DegreeWorks objects.
– We had to manually create the DDL to recreate the
missing DegreeWorks objects.
Session ID 2853
25
Migration – Database
• For our instance, it took approximately 11
hours to export the database, copy the
export file to the new server, and import
the file into the new database.
• (Fee Assessment Audit Table)
Session ID 2853
26
Migration – Database
• Oracle 10.2.0.5 and 11g no longer allows
password protected roles such as
BAN_DEFAULT_M to be granted as default
roles
• BAN_DEFAULT roles are replaced by
USR_DEFAULT roles
• Users in your current database with
BAN_DEFAULT roles will have to be granted
the corresponding USR_DEFAULT role
Session ID 2853
27
Migration – Database
• Ellucian supplies three scripts to aid in the role
conversion
gcnvconnectrole.sql
gcnvconnectrole_ban_default_m.sql
gcnvconnectrole_ban_default_q.sql
• These scripts will create the new roles and apply
them to users either by adding the new role or
replacing the existing role.
• Ellucian FAQ’s:
1-XFTOI5
1-H4TMPD
Session ID 2853
28
Migration – Database
• Oracle 11g requires ACL’s to be added for
DegreeWorks, Workflow, and any custom
apps that perform HTTP or SMTP calls
from the database. FAQ# 1-648TQ0
• ACL_and_Banner_11g_database.pdf
Session ID 2853
29
Infrastructure
Environment
.
Session ID 2853
30
Infrastructure
• We have an excellent infrastructure team!
• VMware templates have been created.
• Flexible storage solutions using the Dell
Equallogic storage array.
• OPSCODE Chef provisioning services
– Server configuration (iptables, users, httpd,
profiles, groups, etc.)
– Configuration management
• A 64bit RHEL5/6 server can be provisioned,
configured, and booted in around 15 minutes.
Session ID 2853
31
Infrastructure – Grid control
• We use Oracle 11g Grid Control to
manage our Oracle database
infrastructure.
– Backups
– Alert Monitoring
– Database Cloning
– Performance Monitoring
– Regular Maintenance
Session ID 2853
32
Infrastructure – Grid control
Session ID 2853
33
Infrastructure – Grid control
• For our Banner Environment we have
multiple instances running.
–
–
–
–
BANPROD
BANAUX1 Clone of production for Research Corporation testing
BANAUX2 Clone of production for University testing.
BANSTAGE Staging environment for new releases
• BANAUX1 and BANAUX2 share a
common code tree
Session ID 2853
34
VMware – current versions
• vSphere Software
– Vmware vCenter Server v5
– Vmware ESXi v5 Hypervisor
• vSphere Features
– vSphere HA – clustering ability to hot migrate
VM from one physical server to another
– vSphere DRS – ability to balance performance
across cluster
• vCenter Site Recovery Manager
Session ID 2853
35
VMware Hardware Architecture
• 10 x Dell r710 Aggregate Specs
– 108 CPU (287 GHz, mix of Nahalem and Westmere)
– 1.03 TB 1333Mhz Memory
• 48 TB iSCSI storage (3 tiers)
–
–
–
–
•
Equallogic 6010x Hybrid SSD/SAS 15k 10GBe (8 x SSD, 8 x 15k SAS)
Equallogic 6510x 10k 10Gbe (48 x 10k SAS)
Equallogic 6510e 7.2l 10Gbe (48 x 7.2k SATA)
4 x Dell 8024f SFP+
Networking (per server)
– 2 x 1Gbe – VMkernel & vMotion
– 6 x 1Gbe – Virtual machine data networking
– 2 x 10Gbe – iSCSI network storage
Session ID 2853
36
VMware Hardware Architecture
Session ID 2853
37
VMware Hardware Architecture
Session ID 2853
38
VMware Hardware Architecture
Session ID 2853
39
VMware Hardware Architecture
Session ID 2853
40
Cloning
.
Session ID 2853
41
VMware and SAN cloning
• With VMware, we can now easily clone
– Servers
– Environments
• With Equallogic, we can clone data LUNs
Session ID 2853
42
VMware cloning
Production
Session ID 2853
Isolated vLan
43
Database Cloning – Grid Control
Session ID 2853
44
Performance
Reliability
Disaster Recovery
.
Session ID 2853
45
VMware cluster
• VMware server farm
– No difference in performance compared to standalone
server
– IO throughput is superb with multiple 10GBytes i-scsi
adapters
– vMotion and dynamic load balancing among cluster
members
– Cloud services for DR contingency
• We are talking with cloud services vendors to pilot a DR hot-site
clone of our production environment.
• Sister Institutions within the state of WV we share a 10Gbit fiber
ring.
– Rack swap
– VMware Site Recovery Manager
– Equallogic SYNCREP (byte level SAN replication)
Session ID 2853
46
Summary
• Server virtualization is working very well for
Marshall University.
• We have reduced infrastructure costs,
• Gained a higher level of service for
customers.
• Increased flexibility to respond to customer
needs (multiple Banner instances),
• And created a viable disaster recovery
solution.
Session ID 2853
47
Questions & Answers
Session ID 2853
48
Thank You!
• Brent Maynard
– brent.maynard@marshall.edu
– (304) 696-2591
• Myke Watts
– iwatts@marshall.edu
– (304) 696-3921
• Presentation available
• http://webpages.marshall.edu/~maynard169/live2853.pptx
.
Session ID 2853
© 2013 Ellucian. All rights reserved.
49