11/13/2014
IBM UPDATES
OCTOBER 2014
Jaqui Lynch
http://www.circle4.com/forsythe/p8nov2014.pdf
1
AGENDA
Jaqui
• Announcement Summary
• IBM Line of POWER8 Servers
• Upgrading to POWER8
• PDU planning for Power servers and TDAs
• IBM entitlement enforcement for Power
2
1
11/13/2014
ANNOUNCEMENT SUMMARY
Annc
6 Oct
6 Oct
eConfig
Power E870
Power E880
(1-2 node)
Power E880
(3rd/4th node)
Power S824L
NEBS capability for S822/S822L
CAPI card for S821L/S822L
7 Oct
7 Oct
6 Oct
6 Oct
6 Oct
7 Oct
RPQ for 110V S814 in rack
RPQ for bigger S824L memory
23 Sep
Mid Oct
n/a
n/a
6 Oct
GA
18 Nov
18 Nov
Jun
2015
31 Oct
Oct
Dec
SPT availability for E870/E880 planned for
Early November or late October
3
Latest Server Line
4
2
11/13/2014
4Q 2014 PORTFOLIO: POWER8/POWER7/POWER7+
Power
780+
Power
770+
Power
795
Power
750+ / 760+
Power
710+/730+
Power
720+/740+
PowerLinux
7R1+ / 7R2+
PowerLinux
7R4+
IBM PureFlex
System
P460+
p260+
p24L
5
POWER SCALE-OUT SERVERS
(WITH POWER8 TECHNOLOGY)
S814
1 socket 4U
7 PCIe Gen3, full high slots
10 or 12 or 18 SFF-3 SAS bays
S824
2 socket, 4U
11 or 7 PCIe Gen3, full high slots
12 or 18 SFF-3 SAS bays
S822
2 socket 2U
9 or 6 PCIe Gen3, low profile slots
12 or 8 SFF-3 SAS bays
S812L
1 socket 2U Linux only
6 PCIe Gen3, low profile slots
12 or 8 SFF-3 SAS bays
S822L
2 socket 2U Linux only
9 PCIe Gen3, low profile slots
12 or 8 SFF-3 SAS bays
6
3
11/13/2014
POWER8 S824L Linux System
Form Factor:
Sockets:
4U
Dual
Cores:
20 / 24
Memory:
Up to 1TB / 16 DIMMs
Slots: 10 PCI G3 LP (No Hotplug)
Adapters supported:
FC EN0W:
2x 10GBASE-T
FC EN0S:
2x 10Gb SFP+ Fiber SR plus 2x 1GE
IB FDR (x16)
GPU: 1 / 2
Media: DVD
Internal Storage
Up to 12 SFF Bays ( JBOD only / No Cache option)
Power Supplies: 2 / 4
Operating System: Ubuntu (LE)
7
Power Systems NVIDIA GPU support
GPU
Kepler GK110B
Double Precision FP
Performance
1.43 TFLOPS (Peak)
Single Precision FP
Performance
4.29 TFLOPS (Peak)
Memory Bandwidth
288 GB/sec
Memory Size
12 GB
CUDA Cores
2880
Server support: POWER8 S824L
8
4
11/13/2014
POWER8 INTEGRATED PCI GEN 3
POWER8
POWER7
Change off GX impacts
I/O drawer migration …
can move PCIe cards,
but not the drawer itself
GX
Bus
I/O
Bridge
PCIe Gen3
PCIe Gen2
PCI
Device
PCI
Devices
9
POWER8 ENTERPRISE POWER SERVERS: BEST OF BOTH WORLDS
Architectural strengths of Power 795
Modularity & efficiencies of Power 770/780
Power 795 Attributes
Largest scale & most capacity
24” Mid-plane architecture
Largest memory footprint
No primary node
Isolated redundant resources
Service processor
Clock
Oscillators
POWER8 Attributes
Power 795
Power 880
Redundant thermal monitoring
Increase scale & capacity
19” Rack mount
Increase memory footprint/core
New I/O scale & flexibility
No primary node
Isolated redundant resources
Power 770/780 Attributes
19” Rack mount
Modular, Energy efficient
Blind swap I/O Adapters
Integrated Ethernet adapter
Internal disk/media
Power 780
Modular, Energy efficient
Blind swap I/O adapters
Integrated on-chip thermal
monitoring
Enterprise Pool Integration
Power 770
Power 870
Service processor
Clock
Oscillators
POWER7/7+ 770/780/795
POWER8 870/880
10
5
11/13/2014
POWER8 ENTERPRISE SYSTEMS ROLLOUT
Existing
8 Node
256-way
4.0 GHz
16 TB
2Q15
4Q14
8 Node
128-way
4.2 GHz
16 TB
8 Node
192-way
3.7 GHz
16 TB
4 Node
192-way
4.0 GHz
16 TB
2 Node
64-way
4.4 GHz
8 TB
4 Node
128-way
4.4 GHz
16 TB
Power 795
4 Node
128-way
3.7 GHz
4 TB
Power 880
4 Node
64-way
4.2 GHz
4 TB
Power 780+
4 Node
64-way
3.8 GHz
4 TB
2 Node
80-way
4.2 GHz
4 TB
Power 880
2 Node
64-way
4.0 GHz
4 TB
4 Node
48-way
4.2 GHz
4 TB
Power 870
Power 770+
19
TWO POWER8 CHIPS: SCALE-OUT AND ENTERPRISE OPTIMIZED
Scale-Out Chip DCM
Two x
Two x
Two x
Two x
Two x
6 Cores
12 Cores
48 MB L3
96 MB L3
4 Mem Attach
8 Mem Attach
24x PCI G3 48x PCI G3
16x CAPI
32x CAPI
SMP 1st Tier on-DCM
SMP 2nd Tier Multiple DCM sockets
Max SMP Small
Enterprise Chip SCM
12 Cores
96 MB L3
8 Mem Attach
32x PCI G3
16x CAPI
SMP 1st Tier Multi-chip Drawer
SMP 2nd Tier Multiple Drawers
Max SMP Large
12
6
11/13/2014
2014 POWER8 SYSTEMS
Enterprise
PowerLinux
Scale Out
795
Power S824
Power S822L
Power S814
Power S812L
Power S822
770/780
Scale-Out Chip DCM
Enterprise Chip SCM
13
Enterprise Power Scale-up Positioning
Power E880
Power E880
Power E870
Power E870
Power E880
Power 780
Power 795
Power 770
14
7
11/13/2014
POWER8 Scale-up Systems
No Primary Node
Midplane
Service Processors
Clocks
Oscillators
Large Memory
19” Rack
Modular design
Up to 4 CEC drawers/nodes
PCIe slots in the nodes
15
POWER8 Scale-up System
Blend of Power 795 & Power 780/770
Architecturally:
Similar to Power 795
Packaging:
Similar to Power 780/770
One to four nodes (E880 - 9119-MHE)
One to two nodes (E870) - 9119-MME)
19” Rack
Great memory
Up to 16TB (E880) or 4TB (E870)
Faster memory - 1600 MHz
Up to 85.3 GB / Core
PCIe Gen3 Slots
16
8
11/13/2014
POWER E870/E880
System node “building blocks
• 8 PCIe Gen3 low profile slots per node
• 0 SAS bays per node
• Up to 2 nodes in 2014
• Power E880 up to 4 nodes in 2015
• No GX++ … no PCIe Gen1 I/O drawers, no EXP30 Ultra SSD
drawer
PCIe Gen3 I/O Expansion Drawer
12 PCIe full high slots per drawer
uses 2 slots in node to attach
0 or 2 supported per E870/E880 system note in 2014.
SOD for up to 4 drawers per node in 2015.
SOD for drawer attachment to Scale-out servers in future
17
POWER8 ENTERPRISE SYSTEM
CECs
19” Rack
5U Form factor
4 Sockets / CEC
Up to 12 cores per socket
Multiple options
Up to 4TB Memory per CEC
32 DDR3 DIMMs
> 900 GB/s BW
Memory Mirroring (PHYP) / Compression
8 PCIe Gen3 Slots / CEC
Remote IO Drawer support
Up to 4x8 remote drawers
Enterprise
Modular 5U
Up to 16 sockets
Up to 16TB
4 Power Supplies
System Control Unit ( Required )
2U Form factor
FSP, VPD, Clocks
“Midplane”
18
9
11/13/2014
POWER8 ENTERPRISE 870 SOLUTIONS
POWER8 Enterprise 870 Attributes
1 - 2 CEC Drawers
Up to 8 PCIe IO Drawers
Up to 96 PCIe G3 adapters
System Control Unit
No primary CECs
Isolated resources
Service processor
Clock
Oscillators
Architecture of a Power 795
Modularity of a Power 770+
19” Rack mount
Modular
Blind swap IO Adapters
24 x 7 Warranty
Integrated on Chip thermal monitoring
Power Pool support: 770+
19
POWER8 ENTERPRISE 880 SOLUTIONS
POWER8 Enterprise 880 Attributes
1 - 4 CEC Drawers
Up to 16 PCIe IO Drawers
Up to 192 PCIe G3 adapters
System Control Unit
No primary CECs
Isolated resources
Service processor
Clock
Oscillators
Architecture of a Power 795
Modularity of a Power 780+
19” Rack mount
Modular
Blind swap IO Adapters
24 x 7 Warranty
Integrated on Chip thermal monitoring
Power Pool support: Power 795 & 780+
20
10
11/13/2014
POWER8 System Node (CEC Drawer)
1-node
system
2-node
system
3-node
system
4-node
system
32-core
64-core
96-core
(2015)
128-core
(2015)
2015
2015
2015
2015
40-core node
40-core
80-core
n/a
n/a
32-core node
32-core
64-core
n/a
n/a
E880
32-core node
SOD 48-core
E870
21
POWER8 CEC REAR VIEW
PCIe G3 x16
SMP
Buses
SMP
Buses
FSP Clock Clock FSP
Note: There are no disks so you need to add an EXP24 disk drawer
If they want to boot from internal disks for VIOS or other LPAR
22
11
11/13/2014
POWER ENTERPRISE VIEW
5U CEC
5U CEC
System Control Unit
Modular “Midplane” FSP/Clock Drawer
22U
2U Form factor
5U CEC
External FSP
Clock / Oscillator Support
Optional DVD
5U CEC
19” Form Factor
Connect multiple nodes
Required on all systems
23
System Control Unit
(Midplane)
One system control unit per system
2U drawer
Must be immediately physically adjacent
to system nodes
System control unit holds:
Service processors (FSPs)
HMC ports
Master system clocks
Operator panel
VPD (vital products data)
Optional DVD
24
12
11/13/2014
System Control Unit Redundancy
One system control unit per server, HOWEVER
Drawer is basically a holder of redundant system control unit
elements
Redundant system clocks/oscillators
Redundant service processors
Redundant power sources(up to four UPIC power cords and any one of the
four
is all that is needed*)
Redundant HMC ports (two per service processor, four total)
Drawer also includes:
One operator panel shared by the above redundant
components –
not critical due to HMC
One master VPD card – redundant copies of information on the card
One backplane and one midplane providing cross element
connections – connections are passive, therefore incredibly unlikely to fail
One optional DVD (attached separately to nodes, alternate DVDs
available in other drawers)
Note: There is no hot add/remove – CHARM is not supported
* except optional DVD needs at least two UPIC power cords
connected for enough power
25
POWER8 PCIe I/O Drawer Rear
4U 19” Form Factor
PCIe Optical Interface to CEC
Max: 20 Meters
No DASD Support
Dual
Power
Cords
Dual
Modules
4Q14 Support:
• Fan-out Module
• Enterprise Systems
Fan-out Module
6 PCIe G3 Slots
4 x8 & 2 x16
Fan-out Module
6 PCIe G3 Slots
4 x8 & 2 x16
26
13
11/13/2014
E870/880 Racking - “Only Enterprise 42U Rack”
Only the IBM Enterprise rack (7014-T42 or #0553) have been tested/certified
by IBM Development/Test with the E870/E880 as of Oct 2014. Therefore the
7014-T42 or #0553 is the only rack IBM Manufacturing uses with E870/E880
system nodes.
If a different rack (IBM or nonIBM) is desired, work with IBM Service organization
to confirm the rack has the needed strength, rigidity, hole spacing, clearances,
etc. IBM Service does not require certification by IBM Development/Test to be
able to provide service/warranty in other racks.
IBM Manufacturing will always build/test E870/E880 in a 42U rack:
New serial number uses 7014-T42
MES same-serial-number uses #0553
Provides a better experience to client
Chargeable “De-racking” feature #ER21 if client doesn’t want the 7014-T42 or #0553 it was
built/test in.
27
POWER E870/E880 OPERATION SYSTEM SUPPORT
AIX with any valid I/O config:
AIX 7.1 TL3 SP4 and APAR IV63332 or later
AIX 7.1 TL2 SP6 or later
(planned availability January 2015)
AIX 6.1 TL9 SP4 and APAR IV63331 or later
AIX 6.1 TL8 SP6, or later
(planned availability January 2015)
AIX with virtual I/O only:
AIX 7.1 TL2 SP1 or later
AIX 7.1 TL3 SP1 or later
AIX 6.1 TL8 SP1 or later
AIX 6.1 TL9 SP1 or later
IBM i
IBM i 7.2 TR1 or later
IBM i 7.1 TR9 or later
Linux
Red Hat Enterprise Linux 6.5 or later
SUSE Linux Enterprise Server 11 SP3 and later SPs
VIOS
VIOS 2.2.3.4 with ifix IV63331 or later
Firmware 8.2.0
28
14
11/13/2014
RAS
29
POWER8 RPERF - ESTIMATES
30
15
11/13/2014
Upgrading
31
MODEL UPGRADES (SAME SERIAL NUMBER) AS OF 4Q14
795
595
780D
E880
E770
770D
570
750
560
740
550
720 6/8 core
520 4-core
720 4 core
520 2-core
520 1-core
730
710
POWER6
32
POWER7
Withdrawn from marketing
16
11/13/2014
SAME-SERIAL-NUMBER CONVERSIONS/CARRY-OVERS
Processor features (node-to-node)
Processor activation features
Memory features
Memory activation features
CEC enclosure
5250 (IBM i) enablements
Active Memory Expansion feature
Note A: feat conversions are 1:1
So if you upgrade a 4-node
770+ server to a 2-node E870,
there will be more 770 features
to “convert from” than there
are E870 features to “convert
to”.
Note B: you upgrade once to an
E870/E880. That is the only
time these upgrade conversion
apply.
Note C: On an upgrade CEC
components are returned to
IBM whether converted or not.
33
SIDE-BY-SIDE RPQ --- HMC COMMENT
When you need to run two systems in
parallel with the same serial number while
upgrading …
Use two, separate HMCs. HMC can
become confused if use one HMC to handle
two servers with the same serial number
34
17
11/13/2014
CAPI attached Flash
35
CAPI ATTACHED IBM FLASHSYSTEM OPTIMIZATION
• Attach IBM FlashSystem to POWER8 via CAPI
• Issues Read/Write commands from applications to eliminate 97% of code path length
• Saves 20-30 cores per 1M IOPs
Up to 40TB of extended memory with one POWER8 server + CAPI attach FLASH
Application
Application
Read/Write Syscall
FileSystem
strategy ( )
iodone ( )
LVM
strategy ( )
iodone ( )
Disk and Adapter DD
Pin buffers,
Translate, Map
DMA, Start I/O
20K
instructions
reduced to
<500
Posix Async
I/O Style API
aio_read()
aio_write()1
User Library
Shared
Memory Work
Queue
Interrupt, unmap,
unpin,Iodone
scheduling
36
18
11/13/2014
INNOVATIVE “IN-MEMORY” NOSQL/KVS INTEGRATED SOLUTION
4Q14 - Differentiated NoSQL
(POWER8 CAPI + FlashSystem)
Today’s NoSQL in memory (x86)
WWW
WWW
10Gb Uplink
10Gb Uplink
Load Balancer
POWER8 Server
24U
500GB Cache
500GB Cache
Node
500GB Cache
Node
500GB Cache
Node
500GB Cache
Node
Node
4U
Flash Array w/ up
to 40TB
Backup Nodes
Target LoB / solution architects and MSPs
• Supporting or building mobile/web/social apps
• Leveraging Key Value Store (KVS) for fast lookups
• Require high performance in-memory data access
IBM Data Engine for
NoSQL - Power Systems Edition:
• 24:1 physical server consolidation
• 6x less rack space (2U server+2U
FlashSystem vs. 24 1U servers)
37
DEMONSTRATING THE VALUE OF CAPI ATTACHMENT
Identical hardware with 2 different
paths to data
120000
108,438
500
450
FlashSystem 840
100000
466
400
350
80000
300
Conventional
PCIe I/O
60000
250
CAPI
199
200
40000
150
16,150
100
20000
50
0
0
PCIe I/O
POWER S822
CAPI
IOPS per HW Thread
PCIe I/O
CAPI
Latency (us)
38
19
11/13/2014
Other E870/E880
39
OTHER USEFUL DATA
PVU
(E870 and E880)
Software Tier
AIX = Medium
120
IBM i = P30
IBM i QRPCFEAT same
as processor SCM feat code
Warranty
1year …same as Power 795
40
20
11/13/2014
POWER E870 VS POWER 770
2014
9117-MMD
Power 770
9119-MME
Power E870
CPU Sockets per Node
4
Max processor nodes
4
2
Max number sockets
16
8
64
80
Max Cores
4
Max Frequency
3.8 GHz
4.19 GHz
Max Memory
1 TB per node
2 TB per node
Memory per core
31.3 GB
Memory Bandwidth (peak)
I/O Bandwidth (peak)
50 GB
272 GB/s per node
922 GB/s per node
80 GB/s per
256 GB/s per
node (GX)
node (PCIe
Max PCIe I/O drws
16 (4 per Node)
Max PCIe I/O Slots
160 - in IO
drws 24 internal
Gen3)
4 (2 per Node)
48 in IO
drws
8intern
al
41
POWER E880 VS POWER 780
2014
9117-MMD
Power 770
9179-MHD
Power 780
9119-MHD
Power E880
CPU Sockets per Node/Drawer
4
4
4
Max processor nodes/Drawers
4
4
2
Max number sockets
Max Cores
16
16
8
64
128
64
Max Frequency
3.8 GHz
3.7 GHz
4.35 GHz
Max Memory
1 TB per node
1 TB per node
4 TB per node
Memory per core
Memory Bandwidth (peak)
I/O Bandwidth (peak)
64 GB
32 GB
272 GB/s per node
272 GB/s per node
80 GB/s per
nod
Max PCIe I/O drws
Max PCIe I/O Slots
80 GB/s per node
(GX)
128 GB
922 GB/s per node
256 GB/s per
node
e
(PCIe
(GX
Gen3)
)
16 (4 per Node)
16 (4 per Node)
160 - in IO
drws 24 internal
160 - in IO
drws 24 internal
4 (2 per Node)
48 in IO
drws
8intern
al
42
21
11/13/2014
Notes
43
PDUS AND TDAS
D models and HMCs use more Power than people expected – as do the P8 servers
Line cords to PDUs are typically either 6492 (60 amp) or 6654 (30 amp)
In the US those are derated to 48 amp and 24 amp so we can not plan to EVER put more
than 24 or 48 amps on a PDU
A server has 2 power supplies which must go to 2 separate PDUs for redundancy.
Note that redundant power supplies sharing the server load may hide the problem of
accidentally exceeding the 4.8 kVa loading on a PDU in normal operation. However, if one
PDU in a redundant pair happens to fail, for example, if a PDU cord is unplugged or an
input circuit fails, the entire load of all the devices will be forced to the remaining PDU
where it is likely to fail due to the overloading."
The E870 and E880 and all the POWER8 servers have specific power needs – ensure that
you plan for these
Read the E870/E880 Technical Overview Redbook
44
22
11/13/2014
E870 AND E880 POWER NEEDS
Each CEC has 4 power supplies (2 main and 2 redundant)
Per enclosure operating needs:
E870/E880
4.2 kVA
4150 watts
14164 BTU/hr
About 21 amps
SCU draws its power from the nodes
Weight
CEC
SCU
I/O drawer
167lb
52lb
52lb
versus 770D/780D
1.959 kVA
1900 watts
6485 BTU/hr
About 9.8 amps
155lb
N/A
105lb (5877)
2 node + SCU complex requires 12U
Each node is 5U
I/O drawer is 4U
Plus don’t forget to add space for EXP24 if you need internal disks
FROM TDA
Two possible PDU ratings are supported: 60A/63A (orderable in most countries) and 30A/32A:
- The 60A/63A PDU supports four system node power supplies and one I/O expansion drawer or eight I/O
expansion drawers. Basically means 2 nodes per 2 PDUs
- The 30A/32A PDU supports two system node power supplies and one I/O expansion drawer or four I/O expansion
drawers. Basically means 1 node per 2 PDUs
Rack-integrated system orders require two of either feature 7109 or 9188/7188. Each PDU requires one PDU to Wall
Power Cord. Supported power cords include the following features: 6489, 6491, 6492, 6653, 6654, 6655, 6656, 6657,
and 6658.
45
QUESTIONS
If you have questions please email me at:
lynchj@forsythe.com
46
23