Scalable Hierarchical Aggregation
Protocol (SHArP)
Release Notes
Rev 1.0
www.mellanox.com
Mellanox Technologies
NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED
DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY
KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE
THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT
HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S)
AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT
GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED.
IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,
INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT
LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF
ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Mellanox Technologies
350 Oakmead Parkway Suite 100
Sunnyvale, CA 94085
U.S.A.
www.mellanox.com
Tel: (408) 970-3400
Fax: (408) 970-3403
© Copyright 2016. Mellanox Technologies LTD. All Rights Reserved.
Mellanox®, Mellanox logo, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®, CoolBox®, COREDirect®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®, GPUDirect®, InfiniHost®,
InfiniScale®, Kotura®, Kotura logo, Mellanox Federal Systems®, Mellanox Open Ethernet®, Mellanox ScalableHPC®,
Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular Switch®, MetroDX®,
MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, Open Ethernet logo, PhyX®, SwitchX®, Tilera®, Tilera logo, TestX®,
TuneX®, The Generation of Open Ethernet logo, UFM®, Virtual Protocol Interconnect®, Voltaire® and Voltaire logo are
registered trademarks of Mellanox Technologies, Ltd.
All other trademarks are property of their respective owners.
For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks
2
Mellanox Technologies
Table of Contents
Rev 1.0
Table of Contents
Document Revision History.................................................................................................................. 5
1
Overview .......................................................................................................................................... 6
1.1
Supported OS and Platforms .................................................................................................. 6
2
Main Features in This Release ....................................................................................................... 7
3
Limitations ....................................................................................................................................... 8
3
Mellanox Technologies
Rev 1.0
Table of Contents
List of Tables
Table 1: Document Revision History ....................................................................................................... 5
Table 2: New Features, Changes and Fixes ........................................................................................... 7
4
Mellanox Technologies
Scalable Hierarchical Aggregation Protocol (SHArP) Release Notes
Document Revision History
Table 1: Document Revision History
Revision
Date
Description
1.0
June 2016
Initial version of this SHArP release
5
Mellanox Technologies
Overview
1
Overview
Mellanox Scalable Hierarchical Aggregation Protocol (SHArP™) technology improves upon
the performance of MPI operation, by offloading collective operations from the CPU to the
switch network, and eliminating the need to send data multiple times between endpoints.
This innovative approach decreases the amount of data traversing the network as aggregation
nodes are reached, and dramatically reduces the MPI operations time. Implementing
collective communication algorithms in the network also has additional benefits, such as
freeing up valuable CPU resources for computation rather than using them to process
communication.
1.1
Supported OS and Platforms
Distro
Platform
Kernel
RHEL 6.1
x86-64
2.6.32-131.0.15
RHEL 6.2
x86-64
2.6.32-220
RHEL 6.3
x86-64
2.6.32-279
RHEL 6.4
x86-64
2.6.32-358
RHEL 6.5
x86-64
2.6.32-431
RHEL 7.0
x86-64
3.10.0-123
RHEL 7.2
x86-64
3.10.0-327
Fedora14
x86-64
2.6.35.6-45
Fedora16
x86-64
3.1.0-7
Fedora17
x86-64
3.3.4-5
Fedora18
x86-64
3.6.10-4
SLES 11 SP1
x86-64
2.6.32.12-0.7
SLES 11 SP2
x86-64
3.0.13-0.27
SLES 11 SP3
x86-64
3.0.76-0.11
Ubuntu12.04
x86-64
3.2.0-37
Ubuntu13.10
x86-64
3.11.0-12
Ubuntu14.4
x86-64
3.13.0-24
Ubuntu15.10
x86-64
4.2.0-16
CentOS6.3
x86-64
2.6.32-279
CentOS6.0
x86-64
2.6.32-71
6
Mellanox Technologies
Scalable Hierarchical Aggregation Protocol (SHArP) Release Notes
2
Main Features in This Release
SHArP 1.0.0 provides the following new features:
Table 2: New Features, Changes and Fixes
Category
Description
MPI 2.x
Barrier and Allreduce collective operations using SHArP protocol
are supported in Open MPI, MPICH, ScalableSHMEM with HCOLL
library.
HCOLL
Enables running of SHArP collective with the mpirun utility.
For the complete list of flags that can be used when running SHArP,
please refer to the SHArP Deployment Guide.
7
Mellanox Technologies
Limitations
3
Limitations
Module
Limitation
Workaround
Aggregation Manager
Aggregation Manager High
Availability protocol is currently
not supported. Therefore, only a
single instance of Aggregation
Manager can run in the IB fabric.
-
Aggregation Manager
Aggregation manager should run
on the same Host where the
Master Subnet Manager (SM) is
running.
-
Aggregation Manager
Upon Subnet Manager
handover/failover, another
instance of Aggregation Manager
should be started on the Host
where the new Master SM is
running
-
Aggregation. Manager
Upon restart of Aggregation
Manager, ongoing jobs using
SHArP collective operations may
be terminated or get hanged.
Postpone Aggregation
Manager restart all jobs using
SHArP collectives are
completed.
Aggregation Manager
Aggregation Manager should be
started after completion of fabric
configuration by the Subnet
Manager.
-
Aggregation Manager
Switch reboot, fabric extension,
and HCA cards replacement are
currently not supported by the
Aggregation Manager
Restart Aggregation Manager
after the Subnet Manager
completes fabric
reconfiguration followed by
the fabric changes.
Fabric Topology
Only Fat-Tree and Quasi-Fat-Tree
topologies are supported by the
Aggregation Manager.
-
Fabric Topology
Only IB fabrics where all compute
nodes are connected to SHArP
capable switches (SwitchIB-2) are
supported by the Aggregation
Manager.
-
Configuration
Upon changes in configuration
file, Aggregation Manager should
be restarted to deploy new
configuration.
-
SHArP Deamon
Only x86 based servers platforms
are currently supported.
-
8
Mellanox Technologies