NEC and CSCS
A First Ivy Bridge Deployment
Jason Temple, Luc Corbeil, HPC Solutions, CSCS
What is CSCS
Office building
•  5 floors
•  2’600 m2
•  Standard Minergie
•  Offices for 55 people
•  Two conference rooms
© CSCS 2013
2
CSCS in a nutshell
•  An autonomous unit of the Swiss Federal Institute of Technology in
Zurich (ETH Zurich)
•  founded in 1991 in Manno
•  relocated to Lugano in 2012
•  Develops and promotes technical and scientific services
•  for the Swiss research community in the field of high-performance
computing
•  Enables world-class scientific research
•  by pioneering, operating and supporting leading-edge supercomputing
technologies
© CSCS 2013
3
Management of Computing Resources
•  Two groups of highly skilled HPC system administrators
•  National Systems
–  Systems that are operated for the benefit of the Swiss scientific
community
–  Time allowed based on a call for proposal with peer-review
•  HPC Solutions
–  Hosting and management of HPC resources for Swiss organizations
–  Current partners:
–  MeteoSwiss (24/7 operational weather forecast)
–  EPFL/BlueBrain
–  CHiPP (data processing from LHC)
New project for HPC Solutions
•  Community of ETHZ professors needed HPC resources
–  Theoretical physics
–  Seismology
–  Polymer Physics
–  Astrophysics
–  Molecular dynamics
•  CSCS acts as a broker
–  Pooling of resources
–  Reduced overhead for each professor
–  Provider of a turnkey solution
Requirements
•  Common analysis of the requirements
–  Scientific
–  Subset of applications
–  Gromacs
–  VASP
–  AMR
–  SSE
–  Technical
–  Facilities constraints
–  Cluster design
–  Reliability, Availability, Serviceability (RAS)
–  Best solution within budget
Solution
•  NEC’s offer was selected
–  Full competitive WTO process
•  First Intel Ivy Bridge cluster in Switzerland
–  Maximum power consumption was lower than expected
–  Throughput per node improved
System Hardware Specifications
340 Standard Compute Nodes with two Sockets and 20 Cores
–  Two Intel Ivy Bridge EP E5-2660v2 @ 2.2 GHz
–  312 + 4 nodes with 32 GB (DDR3 1600 MHz), 24 with 64GB
© CSCS 2013
8 of 21
System Hardware Specifications
•  FDR fat tree, fully non-blocking, Mellanox SX6036
•  >300TB NEC LXFS v3 (based on Lustre 2.3), >15 GB/s
–  Built with NEC SNA260-FS building blocks
•  <100 kW under typical load
9
System Management, Compilers and Tools
•  LX C3 (LX Cluster Command and Control)
•  Node provisioning
•  Cluster monitoring
•  Slurm scheduler (installed and managed by CSCS)
•  Intel Cluster Studio
•  Intel C, C++, Fortran Compiler
•  OpenMP Version 3
•  Intel MPI (MPI 2.2 for C/C++ and Fortran)
•  Mathematical Libraries
•  Intel VTune XE for Linux
© CSCS 2013
10
•  Video
NEC: solid HPC partners
•  Very professional approach
–  Detailed installation planning
–  Planned followed with very minor variations
–  No bad surprises
•  Great benchmarking expertise
–  Prudent, conservative commitments
–  Performance commitments were all exceeded
•  Efficient support
–  Personalized attention
–  Pro-active replacement of suspicious hardware
Conclusion: Success Story!
Thank you for your attention.
Questions?
13