Understanding I/O performance of data
intensive astronomy applications
with Lustre monitoring tools
Gabriele Paciucci, High Performance Data Division
Intel Corporation
Legal Disclaimer
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY
ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN
INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL
DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR
WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT,
COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or
death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL
INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND
EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES
ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY
WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE
DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.
Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or
characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no
responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without
notice. Do not finalize a design with this information.
The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from
published specifications. Current characterized errata are available on request.
Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.
Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800548-4725, or go to: http://www.intel.com/design/literature.htm
Intel, Look Inside and the Intel logo are trademarks of Intel Corporation in the United States and other countries.
*Other names and brands may be claimed as the property of others.
Copyright ©2013 Intel Corporation.
Agenda
• Lustre* metrics and tools
• Analytics and presentation
• Conclusion
3
Why Monitor Lustre*?
• With the exponential growth of high-fidelity sensor and
simulated data, the scientific community is increasingly reliant
on Exascale HPC resources to handle their data analysis
requirements.
• Lustre is the leading parallel file system in the Exascale Era.
• However, to utilize all the Lustre power effectively, the I/O
components must be designed in a proper way, as any
architectural bottleneck will quickly render the platform
inefficient.
4
The challenge of monitoring Lustre
• Understanding the Lustre metrics in the proc filesystem, gives
the opportunity to Administrators to design a Lustre cluster and
maintain the performance requested.
• But there are thousand of metrics in a mid-size Lustre file
system for each components that include clients, servers and
Lustre networks
 These components are distributed: a problem on one node
can affect multiple nodes and finding the initial source of a
problem can be difficult without an integrated monitor tool.
Metrics and Tools
• What you monitor will depend on what you want to know and
on what you think the problems are.
• What you want to measure will also guide your choice of tools
for collecting, analyzing, and presenting the data.
Tools
python/matplotlib

Matplotlib.org

matplotlib is a python 2D plotting library which produces
plt.xlabel('time')
plt.ylabel(r'$MiB/sec$')
plt.setp( ax.get_xticklabels(), rotation=30, horizontalalignment='right')
plt.title("%s on AWS %s Aggregate OST data rates" % (self.application,
dayStr))
plt.legend()
plt.savefig(plot)
plt.cla()
publication quality figures in a variety of hardcopy formats and
interactive environments across platforms. matplotlib can be
used in python scripts, the python and ipython shell
collectl

collectl.sourceforge.net

CollectL is a tool that can be used to monitor Lustre. You can
run CollectL on a Lustre system that has any combination of
MDSs, OSTs and clients. The collected data can be written to a
file for continuous logging and played back at a later time. It
can also be converted to a format suitable for plotting.
LMT

github.com/chaos/lmt/wiki

The Lustre Monitoring Tool (LMT) monitors Lustre File System
servers (MDT, OST, and LNET routers). It collects data using
the Cerebro monitoring system and stores it in a MySQL
database. Graphical and text clients are provided which display
historical and real time data pulled from the database.
[oss]# collectl –scdl –i 3
#<--------CPU--------><----------Disks-----------><---------Lustre OST--------->
#cpu sys inter ctxsw KBRead Reads KBWrit Writes KBRead Reads KBWrit
Writes
19 19 1930 563
0
0 27211 251
0
0 28701 28
9 8 1346 239
0
0 17269 165
0
0 9225
9
[client]# collectl -sl --lustopts R –oTm
#
<---------------Lustre Client--------------->
#Time
KBRead Reads KBWrite Writes Hits Misses
12:20:50.003 17138
8 12854 13 4100
0
12:20:51.002 18450
9 20500 20 4349
0
12:20:52.003 32735 16 20460 20 8447
0
Intel Manager for Lustre
• Bundled with Intel Enterprise Edition for Lustre
• Simplified installation, configuration, monitoring and
management of Lustre
• Provides plugin interface for integration with storage and
other software tools
• Storage hardware neutral
• Intuitive GUI
• Fully featured CLI
IML details
Monitors
• Read and write throughput to the
file system
• Metadata operations to the file
system
• CPU and RAM usage on MDS and
OSS
• Delve down to individual server and
individual Lustre targets
• Aggregate system log of all of
Lustre servers
• The health of Lustre targets and
servers
• LNET status
• The number of clients connected to
the Lustre file system
• The usage of Lustre file system
Manages
• Install Lustre and IML related
packages
• Automatically setup High
Availability
• Power control Lustre servers
• power down, on, cycle
• Manual failover and failback
option
• Create & Setup new Lustre file
systems
• Manage multiple Lustre file
systems
• Rescan network configuration
changes
• Re-configure Lustre file systems
• Support via GUI or CLI
IML dashboard
Agenda
• Lustre* metrics and tools
• Analytics and presentation
• Conclusion
11
Analytics
http://crd-legacy.lbl.gov/~borrill/cmb/madcap/
http://crd-legacy.lbl.gov/~borrill/MADbench2/
• In the MADbench2 application the problem is to generate
simulations of the cosmic microwave background radiation sky
map.
• Each of those simulations involves a very large matrix inversion
that is solved with an out-of-core algorithm (thus the I/O)
•
That phase of the application can be I/O intensive and scales as n2
for a problem of size n (n is the number of pixels in the map). It also
has a communication phase that scales as n3.
Intel Cloud Edition for Lustre*
• We have used Intel Cloud Edition for Lustre to understand how
the application scales, what its workload looks like and how we
can size the environment to maximize the results
• Intel Cloud Edition for Lustre* or ICEL is a scalable, shared
filesystem for HPC applications in the cloud
• AWS allows you to run Lustre on an Amazon Machine Image
(AMI).
• The Intel Lustre AMI is designed to be used with a
CloudFormation template that defines all the resources needed
by the Lustre file system.
13
ICEL for MADBench2 – Scenario 1
We used 256 cores distributed on
32 compute nodes each with 8 cores
Intel Xeon E5-2670.
The total available memory was 960
GB.
MGS
M1.medium
RAID0
8x 40GB
Standard
32x Clients
M3.2xlarge
The RAW space for the Lustre file
system is 5TB.
The max sequential performance is
limited by the OSS’s network to 16
Gbps (16x1Gbps)
MDS
EBS Optimized
Compute
8x 40GB
Standard
OSS
EBS Optimized
M3.2xlarge
M3.2xlarge
16x OSS
8x 40GB
Standard
OSS
EBS Optimized
1Gbps
M3.2xlarge
14
Analytics – Scenario 1
519.51 sec
• The smaller instances ran
quickly (not shown)
• The 32k and 64k pixel
instances became
communication bound
• The MADbench2 application
reads just as much as it
writes
• Even at this larger scale the
I/O fits entirely in client cache,
so the reads do not generate
any traffic to the servers
(where ltop is listening)
3860.67 sec
Changing the compute nodes –
Scenario 2
The decision was to improve
the inter-process
communication using
instances with 10GbE.
The instances with such
networks have 32 cores, so
we could cut the number of
compute nodes from 32 to 16.
We did not modify the size or
the performance of the Lustre
file system.
MGS
M1.medium
RAID0
8x 40GB
Standard
MDS
EBS Optimized
16x Clients
M3.2xlarge
Compute
8x 40GB
Standard
OSS
EBS Optimized
M3.2xlarge
16x OSS
8x 40GB
Standard
Cc2.8xlarge
10GbE
OSS
EBS Optimized
M3.2xlarge
1Gbps
16
Analytics – Scenario 2
• The analysis allowed us
to decrease the Time To
Run the application by
40% increasing the
network bandwidth (1Gb
to 10GbE)
•
Reduce the cost to run
the application by 50%
by lowering the amount
of the servers (32 to 16)
310.44 sec
2427.64 sec
Conclusion
• There is a wealth of information about the health and
performance of Lustre* available in the Proc filesystem
• Proactively tracking the changes in that information will
allow system staff to anticipate and repair problems or
make a better design of the cluster and save money
• Knowing the tools for gathering, analyzing, and presenting
the information will help with system issues and with the
impacts of user codes.
• In the event that a fault is reported, the monitoring
telemetry can help quickly isolate a specific root cause.
18
Risk Factors
The above statements and any others in this document that refer to plans and expectations for the third quarter, the year and the future are forwardlooking statements that involve a number of risks and uncertainties. Words such as “anticipates,” “expects,” “intends,” “plans,” “believes,” “seeks,”
“estimates,” “may,” “will,” “should” and their variations identify forward-looking statements. Statements that refer to or are based on projections,
uncertain events or assumptions also identify forward-looking statements. Many factors could affect Intel’s actual results, and variances from Intel’s
current expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking statements.
Intel presently considers the following to be the important factors that could cause actual results to differ materially from the company’s expectations.
Demand could be different from Intel's expectations due to factors including changes in business and economic conditions; customer acceptance of
Intel’s and competitors’ products; supply constraints and other disruptions affecting customers; changes in customer order patterns including order
cancellations; and changes in the level of inventory at customers. Uncertainty in global economic and financial conditions poses a risk that consumers
and businesses may defer purchases in response to negative financial events, which could negatively affect product demand and other related
matters. Intel operates in intensely competitive industries that are characterized by a high percentage of costs that are fixed or difficult to reduce in
the short term and product demand that is highly variable and difficult to forecast. Revenue and the gross margin percentage are affected by the
timing of Intel product introductions and the demand for and market acceptance of Intel's products; actions taken by Intel's competitors, including
product offerings and introductions, marketing programs and pricing pressures and Intel’s response to such actions; and Intel’s ability to respond
quickly to technological developments and to incorporate new features into its products. The gross margin percentage could vary significantly from
expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale;
changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; start-up costs; excess or
obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; product manufacturing quality/yields; and
impairments of long-lived assets, including manufacturing, assembly/test and intangible assets. Intel's results could be affected by adverse economic,
social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military conflict and
other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Expenses, particularly
certain marketing and compensation expenses, as well as restructuring and asset impairment charges, vary depending on the level of demand for
Intel's products and the level of revenue and profits. Intel’s results could be affected by the timing of closing of acquisitions and divestitures. Intel's
results could be affected by adverse effects associated with product defects and errata (deviations from published specifications), and by litigation or
regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other issues, such as the litigation and regulatory
matters described in Intel's SEC reports. An unfavorable ruling could include monetary damages or an injunction prohibiting Intel from manufacturing
or selling one or more products, precluding particular business practices, impacting Intel’s ability to design its products, or requiring other remedies
such as compulsory licensing of intellectual property. A detailed discussion of these and other factors that could affect Intel’s results is included in
Intel’s SEC filings, including the company’s most recent reports on Form 10-Q, Form 10-K and earnings release.
Rev. 7/17/13
Virtual hardware available in ICEL
VMs size
vCP
U
Intel CPU
M3.2xlarge
8
Intel Xeon E5-2670 30
Yes
32
Intel Xeon E5-2670 60
N/A
Amazon EC2 instances:
• Spot Instances
• EBS optimized
•
CC2.8xlarg
High network capabilitiese
vRAM
(GB)
EBS
Network performance
**
M3.2xlarge
CC2.8xlarge
Amazon EBS storage:
M3.2xlarge
1.01 Gbps 1.87 Gbps
• Networked storage
CC2.8xlarge
1.87 Gbps 6.18 Gbps
• Max size 1TB per EBS
volume
EBS Storage
IOPS
Size Performance
(WRITE) **
• Not magic
Standard
N/A
100
24+ MB/sec
• Standard, not Provisioned
in ICEL
Provisioned
2000
200
35+ MB/sec
Provisioned
4000
400
50+ MB/sec
** not intended to be authoritative numbers
21
Analytics – Scenario 1
•
•
•
The MADbench application does
appear to scale as n3 for larger
instances, though not necessarily for
the smaller ones.
This is what we expected.
But the scaling study doesn’t illuminate why.
Linear scale
Log scale
Analytics – Scenario 1
•
The ltop utility in the LMT package records
OST stat file contents much like llstat.
•
An ad hoc python script (with numpy and
matplotlib) that parses the output recorded by
ltop can present a variety of special-case
views of the data.