CHARON storage options
and clustering
Paris, October 11th – 12th, 2011
Stromasys Product Management
Contents
• Modern storage concepts and layout examples
• Storage layers relevant to CHARON configuration
• Legacy storage technologies and their representation in
CHARON
• Controller emulation, Bypass, and Pass Through modes
• CHARON virtual disks (tapes, CD/DVD) representation
• Container files, PhysicalDrives, and direct LUN access
• Clustering
• Appendix: AXP console operations with KGPSA in Pass
Through mode
Modern storage layouts
Sources: IBM, VMware
Modern storage layout example
Source: Cisco
Working with CHARON storage
CHARON is like 2 faced Janos
• one face looks at the direction of
legacy VAX and AXP hardware.
• Another face looks at the modern
storage technologies, supported
by modern hosting hardware and
OS (Windows, Linux, VMware):
SATA, SAS, NAS, iSCSI, SAN.
Relevant data travel layers
• Layers
Actions
• VMS / Tru64 Applications and layered software
Supplementary: performance
tuning of applications
• VMS / Tru64 Storage Device Drivers
Opportunity: use VMS Bypass
driver
• CHARON: VAX / AXP Storage Controllers
Mandatory: configure
CHARON storage options
• CHARON: Interface to hosting OS storage functions
Mandatory: configure
CHARON storage mapping
• Hosting OS (Windows / Linux / VMware) storage
management
Opportunity: choose optimal
storage allocation
• Storage transport and hardware
Supplementary: performance
tuning of storage options
Mandatory: storage controllers
• VMS / Tru64 Storage Device Drivers
• CHARON: VAX / AXP Storage Controllers
• Storage transport and hardware
Opportunity: use VMS Bypass
driver
Mandatory: configure
CHARON storage options
Supplementary: performance
tuning of storage options
VAX/AXP storage technologies
Legacy storage
technologies
CHARON-VAX
CHARON-AXP
/4100/DS/ES/GS
CHARON-AXP
/SMA(+)/Station
MSCP
Emulated RQDX3,
KDM70, TKQ50, TUK50
-
Bypass in VMS –
presented as MSCP
DSSI
Emulated HSD50
-
Bypass in VMS –
presented as DSSI
CI
Emulated CIXCD, HSJ50
-
-
SCSI
Emulated KZDDA
Emulated KZPBA
Emulated KZPBA;
also bypass in VMS
RAID
-
-
Bypass in VMS –
presented as RAID
IDE
-
Emulated AcerLabs
1543C
Bypass in VMS –
presented as IDE
FDDI Storage
-
-
-
Network Storage
Fiber Channel
Yes, by emulated networking
-
Physical KGPSA, or
compatible Emulex in
Pass Through mode
(Windows platform)
Bypass in VMS –
presented as FC
Emulation and Bypass methods
Emulation
Bypass
VMS / Tru64
VMS
Guest OS
storage controller
driver
VMS
Bypass driver
Up to ~25%faster
Storage controller
No SCSI/RAID/FC
Emulation
Emulation
Windows / Linux
Windows
PCI Pass Through mode
OpenVMS / Tru64
CHARON-AXP
Windows
Host PCI Bus
SCSI
Emulex
driver
Pass-Through
In PCI Pass Through mode
CHARON is taking full control
on a physical PCI adapter,
which is installed in the hosting
server.
Special
CHARON
driver should be installed. In
this case VMS or Tru64 work
with the adapter directly,
passing through hosting OS
and drivers.
Only adapters listed below are
supported in Pass Through
mode.
Currently Pass Through mode
is only supported on Windows
hosting platform.
Emulex
Ethernet
Hints
• Configuration is usually defined based on the original
VAX/AXP hardware layout, in order to avoid or minimize
a need to intervene in the VMS / Tru64 settings
• Some user configurations does not provide flexibility in
selecting the storage method – only one will fit. Go for it
• If there is flexibility, Pass Through provides the best
performance (for a price of additional HBA controller(s)
and relatively complex setup and configuration).
Recommended (sometimes a must) for high
performance system replacements
• Bypass is a really nice feature of CHARON-AXP/SMA(+)
family (provided that requirements are met: Windows
hosting platform, supported VMS version, etc.)
Mandatory: host OS interface
• VMS / Tru64 Applications and layered software
Supplementary: performance
tuning of applications
• CHARON: Interface to hosting OS storage functions
Mandatory: configure
CHARON storage mapping
• Hosting OS (Windows / Linux / VMware) storage
management
Opportunity: choose optimal
storage allocation
• Storage transport and hardware
Supplementary: performance
tuning of storage options
Interface to host OS storage
Storage reliability and performance is defined mostly by the
selected access method
Container
files
Direct disk
addressing
Direct SCSI
addressing
FC Pass
Through
CHARON-VAX
Yes
Yes 2)
Yes
-
CHARON-AXP
SMA(+)/Station
Yes 1)
Yes 2, 3)
Yes
-
CHARON-AXP/xS
for Windows
Yes
Yes 2)
Yes
Yes
CHARON-AXP/xS
for Linux
Yes
Yes
-
-
1)
Booting not always supported from disks if Bypass is used
2) Windows might change the order of the PhysicalDrives in some circumstances.
3) Cannot be used to access disks together with physical Alpha systems
Container files
set dka[0]=“D\CHARON\system.vdisk”
set dka[0]=/opt/charon/disks/system.vdisk”
Simple and fast to configure. Host OS takes care of the I/O, so
any technology could be used: SATA, SAS, SAN, NAS, (i)SCSI
Charon
Host OS
NTFS/ext
file system
Disk controller
emulation: MSCP,
SCSI, DSSI, …
Storage Array
CF.
vdisk
iSCSI or FC
Adapter
iSCSI or FC
network
Disk Volume
Containerfile.vdisk
Directly addressing the disks
set dka[0]=“\\.\PhysicalDrive0”
set dka[0]=/dev/sd0”
Important! Host OS should not access those disk(s)
CHARON talks to SCSI subsystem on the host OS.
Charon
Emulation
Disk controller
emulation: MSCP,
SCSI, DSSI, …
Windows OS
Storage Array
Physical Disk:
\\.\PhysicalDrive0
Logical
Disk
iSCSI or FC
Adapter
iSCSI or FC
Network
Disk Volume
\\.\Physicaldrive1
Direct SCSI addressing
set dka[0]=\\.\SCSIa:b:c:d
Not recommended for disks (designed for generic SCSI
devices)
CHARON talks to SCSI subsystem on the host OS.
Charon
Emulation
Disk controller
emulation: MSCP,
SCSI, DSSI, …
Windows OS
Storage Array
Physical Disk:
\\.\SCSIa:b:c:d
Logical
Disk
iSCSI or FC
Adapter
iSCSI or FC
Network
Disk Volume
\\.\SCSIe:f:g:h
FC Pass Through
Storage configuration for the SAN is done from AXP SRM
console in exactly the same way as for physical Alpha system.
CHARON talks directly to Emulex HBA
Charon
Emulation
FC driver
Windows OS
Storage Array
SAN
infrastructure
Pass through
Driver
Emulex
HBA
Disk Volume
Options
Every device has following options:
• Media_type and disk geometry parameters provide guest
OS with hardware ID and geometry.
• Shared. Allows access from two or more CHARON
instances and other systems. Shared disk devices could
be used for clustering, if the underlying storage also
supports shared access, and the access could be
physically provided.
• Cached. CHARON will enable host OS software chache
is this option is on. Must be switched off, if set to Shared.
• Removable. By default, the tapes, floppies, and
CDROMs are removable. On some emulated controllers
the disks could be made removable too.
Hints
• Reconsider using advanced software options on VMS
and Tru64. For example, VMS shadowing or Tru64 LSM
might have been implemented for fault tolerance. With
the modern storage redundancy could be built in.
Performance boost might be achieved by removing
redundancy.
• Underlying storage should provide sufficient redundancy
and performance for VMS / Tru64 to have access to the
storage at all times. If a timeout happens, the disk could
go offline until VMS/Tru64 reboot. Hence:
– Avoid RAID5 and other slow RAIDs. Mirroring is recommended.
– Consider fastest possible options (SAN, FC Pass Through).
• Modern storage features (snapshots, mirroring) should
be carefully considered.
Clustering
VAX/AXP cluster type
CHARON capability
Cluster traffic
NI (Ethernet)
Mixed clusters
Network
SCSI (Small Computer
System Interface)
Mixed clusters
Emulated SCSI link over
the network
FDDI (Fiber Distributed Mixed clusters
Data Interface)
FDDI link over the
network
DSSI (Digital Storage
Systems Interconnect)
CHARON only clusters
Emulated DSSI link over
the network
CI (Computer
Interconnect)
CHARON only clusters
Emulated CI link over the
network
FC (Fiber Channel)
Mixed clusters
Network
MC (Memory Channel)
Not available
-
Mixed clusters indicates that CHARON can be clustered with physical systems with shared disk
access
Clustering
Applications
Applications
System Utilities
System Libraries
System Utilities
System Libraries
VMS/Tru64
VMS/Tru64
CHARON
CHARON
SCSI disk
controller
emulation
Host
OS
Adapter
Emulation
Adapter
Emulation
Hardware
Hardware
iSCSI or FC
Adapter
Ethernet
Adapter
Ethernet
Adapter
Cluster admin
Storage data
Host
OS
SCSI disk
controller
emulation
iSCSI or FC
Adapter
Ethernet
SCSI, iSCSI or Fibre Channel connection
Storage Array
MSA1000, MSA2100,
iSCSI or EVA etc.
SAN benefits
•
•
•
•
•
•
•
•
Standard in modern environment
Deploy and scale quickly and easily
Simplify management
Use latest storage technologies
Easy migration for existing data on SAN
No local storage / backup needed, centralize it
Reduce footprint of servers
Ideal when using blade servers
Appendix
Fibre Channel PCI
Pass Thruogh
• Avalable since CHARON-AXP/xS 2.3.108
Hardware support - Emulex
LP8000
LP9000
LP9002
LP9802
LP10000
LP10000DC
LP10000-S
LPX1000
LP11002
LPe11002
LPe11
Installation
- Install FC HBA & CHARON-AXP
- Map the driver to the HBA, located at:
..\CHARON\Drivers\EMULEX_2.1.0.108\
Configure CHARON
load KGPSA FGA host_bus_location="PCI Slot 5
(PCI bus 6, device 2, function 0)"
load KGPSA PGA host_bus_location="PCI Slot 3 (PCI bus
4, device 0, function 0)"
load KGPSA PGB host_bus_location="PCI Slot 3 (PCI bus
4, device 0, function 1)"
HSG80
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
HSG80_1L > sho this full
Controller:
HSG80 ZG04404014 Software V88F-2,
Hardware E12
NODE_ID
= 5000-1FE1-000B-6BF0
ALLOCATION_CLASS = 0
SCSI_VERSION = SCSI-3
Configured for MULTIBUS_FAILOVER with
ZG90811283
In dual-redundant configuration
Device Port SCSI address 6
Time: 27-OCT-2009 22:35:00
Command Console LUN is lun 0
(IDENTIFIER = 98)
Host Connection Table is NOT locked
Smart Error Eject Disabled
Host PORT_1:
Reported PORT_ID = 5000-1FE1-000B-6BF1
PORT_1_TOPOLOGY = FABRIC (fabric up)
Address
= 021400
Host PORT_2:
Reported PORT_ID = 5000-1FE1-000B-6BF2
PORT_2_TOPOLOGY = FABRIC (fabric up)
Address
= 011400
NOREMOTE_COPY
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
HSG80_1L > sho other full
Controller:
HSG80 ZG90811283 Software V88F-2,
Hardware E09
NODE_ID
= 5000-1FE1-000B-6BF0
ALLOCATION_CLASS = 0
SCSI_VERSION = SCSI-3
Configured for MULTIBUS_FAILOVER with
ZG04404014
In dual-redundant configuration
Device Port SCSI address 7
Time: 27-OCT-2009 22:35:11
Command Console LUN is lun 0
(IDENTIFIER = 98)
Host Connection Table is NOT locked
Smart Error Eject Disabled
Host PORT_1:
Reported PORT_ID = 5000-1FE1-000B-6BF3
PORT_1_TOPOLOGY = FABRIC (fabric up)
Address
= 011500
Host PORT_2:
Reported PORT_ID = 5000-1FE1-000B-6BF4
PORT_2_TOPOLOGY = FABRIC (fabric up)
Address
= 021500
NOREMOTE_COPY
CHARON SRM
FCMGR
fcmgr rescan
show
set
clear
fcmgr show adapters
ports
devices
fcmgr set boot
dump
FCMGR
•
•
P00>>>fc sho adapter
pga0.0.0.5.1
PGA0
•
•
•
•
•
•
•
P00>>>fc sho port
[0] WWN 5000-1fe1-000b-6bf1 (N0)
via adapter
pga0.0.0.5.1
[1] WWN 5000-1fe1-000b-6bf4 (N1)
via adapter
pga0.0.0.5.1
WWN 1000-0000-C930-490F
FCMGR
•
•
P00>>>fc sho port
run init first
•
•
P00>>>fc sho port
run fcmgr rescan first
FCMGR
• P00>>>fc sho device
• UDID:110 WWID:01000010:6000-1fe1-000b-6bf0-0009-90811283-0081 (ev:none)
•
via adapter via fc_port
con
• [0]
pga0.0.0.5.1 5000-1fe1-000b-6bf1 yes (ev:none)
• [1]
pga0.0.0.5.1 5000-1fe1-000b-6bf4 yes (ev:none)
• UDID:108 WWID:01000010:6000-1fe1-000b-6bf0-0009-0440-40140039 (ev:none)
•
via adapter via fc_port
con
• [12]
pga0.0.0.5.1 5000-1fe1-000b-6bf1 yes (ev:none)
• [13]
pga0.0.0.5.1 5000-1fe1-000b-6bf4 yes (ev:none)
FCMGR
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
P00>>>fc rescan
polling for units on kgpsa0, slot 5, bus 0, hose 1 ...
pga0.0.0.5.1
PGA0
WWN 1000-0000-C930-490F
fabric
WWN 2001-0060-6920-4682
directory
WWN 20fc-0060-6920-4682
port 021400
WWN 5000-1fe1-000b-6bf1
lun 0000000000006e00
DEC HSG80
V86F
……
lun 0000000000006c00
DEC HSG80
V86F
UDID:108
WWID:01000010:6000-1fe1-000b-6bf0-0009-0440-4014-0039
UDID:199
WWID:01000010:6000-1fe1-000b-6bf0-0009-0440-4014-0051
lun 0000000000000000
port 021500
WWN 5000-1fe1-000b-6bf4
lun 0000000000006e00
DEC HSG80
V86F
…….
UDID:108
WWID:01000010:6000-1fe1-000b-6bf0-0009-0440-4014-0039
lun 0000000000006d00
DEC HSG80
V86F
FCMGR (pre-boot)
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
P00>>>fc set boot udid 108
P00>>>sho dev
sys0.0.0.0.0
SYS0
System ROOT Device
pka0.0.0.1.0
PKA0
Q-Logic/ISP PCI SCSI HBA
ewa0.0.0.2.0
EWA0
00-15-17-5F-E8-65
pkb0.0.0.3.0
PKB0
Q-Logic/ISP PCI SCSI HBA
pqa0.0.0.15.0
PQA0
ALi 1553C Integrated IDE Controller
pqb0.0.1.15.0
PQB0
ALi 1553C Integrated IDE Controller
pga0.0.0.5.1
PGA0
WWN 1000-0000-C930-490F
dqa0.0.0.15.0
DQA0
Virtual ATAPI - TEAC DW-224E-V
dka0.0.0.1.0
DKA0
DEC RZ29
(C)DEC
dka200.2.0.1.0
DKA200
COMPAQ RZ1ED-LS
dka300.3.0.1.0
DKA300
DEC RZ1DA (C)DEC
dka400.4.0.1.0
DKA400
DEC RZ28
(C)DEC
dka500.5.0.1.0
DKA500
DEC RZ28
(C)DEC
dga108.1001.0.5.1
$1$DGA108 DEC HSG80
V86F 
dga108.1002.0.5.1
$1$DGA108 DEC HSG80
V86F 
Booting
• P00>>>b $1$dga108
• (boot dga108.1001.0.5.1 -flags 0,1)
• jumping to bootstrap code
VMS
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
CLMB01::SYSTEM>sh dev /multi
Device
Device
Error
Current
Name
Status
Count Paths path
$1$DGA100: (CLMB01) Online
0 3/ 3 PGA0.5000-1FE1-000B-6BF4
$1$DGA108: (CLMB01) Online
0 3/ 3 PGA0.5000-1FE1-000B-6BF4 
$1$DGA110: (CLMB01) Online
0 3/ 3 PGA0.5000-1FE1-000B-6BF1
$1$DGA199: (CLMB01) Online
0 3/ 3 PGA0.5000-1FE1-000B-6BF4
$1$DGA200: (CLMB01) Online
0 3/ 3 PGA0.5000-1FE1-000B-6BF1
$1$DGA208: (CLMB01) Online
0 3/ 3 PGA0.5000-1FE1-000B-6BF4
$1$DGA300: (CLMB01) Online
0 3/ 3 PGA0.5000-1FE1-000B-6BF1
$1$DGA400: (CLMB01) Online
0 3/ 3 PGA0.5000-1FE1-000B-6BF1
$1$DGA500: (CLMB01) Online
0 3/ 3 PGA0.5000-1FE1-000B-6BF4
Device
Device
Error
Current
Name
Status
Count Paths path
$1$GGA98:
Online
0 2/ 2 PGA0.5000-1FE1-000B-6BF1
VMS
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
CLMB01::SYSTEM>sho dev/full $1$DGA100:
Disk $1$DGA100: (CLMB01), device type HSG80, is online, file-oriented device,
shareable, device has multiple I/O paths, served to cluster via MSCP Server,
error logging is enabled.
Error count
0 Operations completed
8
Owner process
"" Owner UIC
[SYSTEM]
Owner process ID
00000000 Dev Prot
S:RWPL,O:RWPL,G:R,W
Reference count
0 Default buffer size
512
Current preferred CPU Id
0 Fastpath
1
WWID 01000010:6000-1FE1-000B-6BF0-0009-0440-4014-0038
Host name
"CLMB01" Host type, avail AlphaServer GS80, yes
Alternate host name "CLMB02" Alt. type, avail AlphaServer GS80, yes
Allocation class
1
I/O paths to device
3
Path PGA0.5000-1FE1-000B-6BF1 (CLMB01), primary path.
Error count
0 Operations completed
4
Path PGA0.5000-1FE1-000B-6BF4 (CLMB01), current path.
Error count
0 Operations completed
4
Path MSCP (CLMB02).
Error count
0 Operations completed
0
VMS
•
•
•
•
•
•
•
•
•
•
….
Volume label
"DISK100" Relative volume number
0
Cluster size
3 Transaction count
1
Free blocks
3093486 Maximum files allowed
4444548
Extend quantity
5 Mount count
1
Mount status
Process Cache name
"_$1$DKA0:XQPCACHE"
Extent cache size
64 Maximum blocks in extent cache
309348
File ID cache size
64 Blocks in extent cache
0
Quota cache size
0 Maximum buffers in FCP cache
3524
Volume owner UIC
[SYSTEM] Vol Prot
S:RWCD,O:RWCD,G:RWCD,W:RWCD
Tru64
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
# scu show edt
CAM Equipment Device Table (EDT) Information:
Bus/Target/Lun Device Type ANSI Vendor ID Product ID
Revision N/W
-------------- ----------- ------ --------- ---------------- -------- --0 1 0 Direct
SCSI-2 Virtual SCSI Disk (C)SRI 0200 W
0 5 0 CD-ROM
SCSI-2 TEAC
DW-224E-V
1.CA W
0 6 0 Sequential SCSI-2 DEC
TZ88 (C) DEC CD50 N
2 0 0 RAID
SCSI-2 DEC
HSG80CCL
V88F W
2 0 1 Direct
SCSI-2 DEC
HSG80
V88F W
2 0 199 Direct
SCSI-2 DEC
HSG80
V88F W 
2 1 0 RAID
SCSI-2 DEC
HSG80CCL
V88F W
2 1 5 Direct
SCSI-2 DEC
HSG80
V88F W
2 1 108 Direct
SCSI-2 DEC
HSG80
V88F W
2 1 109 Direct
SCSI-2 DEC
HSG80
V88F W
2 1 110 Direct
SCSI-2 DEC
HSG80
V88F W
3 0 0 CD-ROM
SCSI-3 TEAC
DW-224E-V
1.CA N
Tru64
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
# hwmr gr =n -view device
HWID: Device Name
Mfg
Model
Location
-----------------------------------------------------------------------------8: /dev/dmapi/dmapi
9: /dev/scp_scsi
10: /dev/kevm
86: /dev/disk/dsk0c
Virtual SCSI Disk (C)SRI bus-0-targ-1-lun-0
96: /dev/ntape/tape0 DEC
TZ88 (C) DEC bus-0-targ-6-lun-0
97: /dev/cport/scp0
HSG80CCL
bus-2-targ-0-lun-0
98: /dev/disk/dsk4c
DEC
HSG80
IDENTIFIER=110
99: /dev/disk/dsk5c
DEC
HSG80
IDENTIFIER=100
100: /dev/disk/dsk6c
DEC
HSG80
IDENTIFIER=200
101: /dev/disk/dsk7c
DEC
HSG80
IDENTIFIER=300
102: /dev/disk/dsk8c
DEC
HSG80
IDENTIFIER=400
103: /dev/disk/dsk9c
DEC
HSG80
IDENTIFIER=108
104: /dev/disk/dsk10c DEC
HSG80
IDENTIFIER=208
105: /dev/disk/dsk11c DEC
HSG80
IDENTIFIER=500
106: /dev/disk/dsk12c DEC
HSG80
IDENTIFIER=199 
Tru64 (Multipath)
•
# hwmgr -show scsi
•
•
SCSI
DEVICE DEVICE DRIVER NUM DEVICE FIRST
HWID: DEVICEID HOSTNAME TYPE
SUBTYPE OWNER PATH
FILE VALID PATH
------------------------------------------------------------------------86: 0
gs160tru64 disk
none 2
1 dsk0 [0/1/0]
92: 4
gs160tru64 cdrom none 0
1 cdrom0 [0/5/0]
97: 7
gs160tru64 raid
none 0
6 scp0 [2/0/0]
98: 8
gs160tru64 disk
none 0
6 dsk4 [2/0/110]
99: 9
gs160tru64 disk
none 0
6 dsk5 [2/0/1]
106: 16
gs160tru64 disk
none 2
4 dsk12 [2/0/199] 
112: 17
gs160tru64 tape
none 0
1 tape1 [0/6/0]
•
•
•
•
•
•
•
•
•
Multipath
•
•
•
•
SCSI
DEVICE DEVICE DRIVER NUM DEVICE FIRST
HWID: DEVICEID HOSTNAME TYPE
SUBTYPE OWNER PATH
FILE VALID PATH
------------------------------------------------------------------------106: 16
gs160tru64 disk
none 2
4 dsk12 [2/0/199]
•
WWID:01000010:6000-1fe1-000b-6bf0-0009-0440-4014-0051
•
•
•
•
•
•
BUS TARGET LUN PATH STATE
-----------------------------2 0
199 valid
2 1
199 valid
4 0
199 stale
4 1
199 stale
Tru64
• # mkfdmn /dev/disk/dsk12c test_dmn
• # mkfset test_dmn test
• # mount -t advfs test_dmn3 #test /test