iRODS performance test
and SRB system at KEK
Yoshimi Iida @ KEK
Building data grids with iRODS
27 May 2008
Outline

Performance measurement





Transfer test between CC-IN2P3 and KEK
Scaling test for ICAT
Concurrent test for ICAT
Compare with iRODS and SRB
SRB/iRODS system at KEK
27/05/2008
Building data grids with iRODS
2
Transfer between CC-IN2P3 and KEK
CC-IN2P3, FR

NY, USA
1GB data transfer between CC-IN2P3 and KEK


KEK, JP
Compare with iRODS and bbcp
The network route go through USA

27/05/2008
The RTT is about 285ms
Building data grids with iRODS
3
System used

iRODS system at CC-IN2P3

ICAT enabled iRODS server on Solaris 10





Thumpers Sun x4500 (AMD processors)
Oracle 10g is on a cluster of dedicated machines
Linux file resource from local disk system
iRODS 0.9
iRODS system at KEK

iRODS server on RHEL3



27/05/2008
Intel Xeon 3.0GHz ×4
Linux file resource from local disk system
iRODS 0.9
Building data grids with iRODS
4
From KEK to CC-IN2P3
50
iput

Speed [MB/sec]
40
30
1GB data transfer
during 24 hours
20

10

0
Speed [MB/sec]
50
window size 4MB
number of parallel
streams 16
bbcp
40

30
20
bbcp often fail to
connect
10
0
27/05/2008
Building data grids with iRODS
5
From CC-IN2P3 to KEK
Speed [MB/sec]
40
iput

30
20

10

0
40
Speed [MB/sec]
1GB data transfer
during 12 hours
bbcp
window size 4MB
number of parallel
streams 16
30

20
10
iput is better than
bbcp
0
27/05/2008
Building data grids with iRODS
6
Scaling test

Data



Using the same directory at client machine
1000 files of 1000 bytes each
Measurement


Ingesting directory and listing the collection
Performance measured for every directory
operation

27/05/2008
1 collection and 1000 files
Building data grids with iRODS
7
System used

ICAT based on Oracle system at CC-IN2P3

ICAT enabled iRODS server on Solaris 10



iRODS resource and client on SL4



Thumpers Sun x4500 (AMD processors)
Oracle 10g is on a cluster of dedicated machines
Dual AMD Opteron Processor 848
Linux file resource from local disk system
ICAT based on PostgreSQL system at KEK

ICAT enabled iRODS server on RHEL3



iRODS resource and client on RHEL3


27/05/2008
Dual Intel Xeon 2.8GHz
PostgreSQL 8.2.5 is running on the same machine
Dual Intel Xeon 2.8GHz
Linux file resource from local disk system
Building data grids with iRODS
8
Ingesting up to 1 million files
iput - Oracle
iput-PostgreSQL
100
time in seconds
tme in seconds
100
80
60
40
20
80
60
40
20
0
0
0
200
400
600
800
number of files [x 1,000]
0
1000
200
1000
ils - Oracle
ils - PostgreSQL
5
time in seconds
5
time in seconds
400
600
800
number of files [x 1,000]
4
3
2
1
0
4
3
2
1
0
0
200
400
600
800
number of files [x 1,000]
1000
0
200
400
600
800
number of files [x 1,000]
1000
Running the other process
27/05/2008
Building data grids with iRODS
9
Nested collection test

Data



Registering same directory at client machine
100 files of 100 bytes each
Measurement



Ingesting directory and listing the collection
Making nested collection every 10 collection
Performance measured for every directory
operation

27/05/2008
1 collection and 100 files
Building data grids with iRODS
10
Nested collection test
iput - Oracle
70
60
50
40
30
20
10
0
nest
flat
0
50
100
150
number of files [x 1,000]
time in seconds
time in seconds
ireg - PostgreSQL
70
60
50
40
30
20
10
0
200
nest
flat
0
ils - PostgreSQL
200
ils - Oracle
4
10
3
nest
flat
2
1
time in seconds
time in seconds
50
100
150
number of files [x 1,000]
8
6
nest
flat
4
2
0
0
0
50
100
150
number of files [x 1,000]
200
coll-1220: USER_PATH_EXCEEDS_MAX
27/05/2008
0
50
100
150
number of files [x 1,000]
200
nest-121 : OCI_ERROR
Building data grids with iRODS
11
Concurrent test

Data



Ingesting same directory at client machine
1000 files of 1000 bytes each
Measurement

Runing multi process at the same time





read operation – ils and iget
write operation – iput and ireg
mixed operation – iput, ireg, ils and iget
Performance measured for every directory operation
DB setting

27/05/2008
Set the number of maximum connections to 200
Building data grids with iRODS
12
Concurrent test for reading metadata
ils - PostgreSQL ICAT
ils - Oracle ICAT
1000
Time in seconds
Time in seconds
1000
100
10
1
100
0.1
10
1
0.1
0
50
100
150
200
250
300
0
50
100
Number of clients
250
300
250
300
iget - Oracle ICAT
10000
10000
Time in seconds
Time in seconds
200
Number of clients
iget - PostgreSQL ICAT
1000
100
10
1000
100
10
1
1
0
50
100
150
200
250
300
0
50
Error: connectToRhost failed
100
150
200
Number of clients
Number of clients
27/05/2008
150
10 clients
100 clients
Building data grids with iRODS
200 clients
× 300 clients
13
Concurrent test for writing metadata
Time in seconds
iput - PostgreSQL ICAT

10000
1000
iput – Oracle ICAT
Because of limit of inodes, we
cannot put any more files

100
10
1
0
50
100
150
200
Number of clients
ireg - PostgreSQL ICAT
ireg - Oracle ICAT
10000
Time in seconds
Time in seconds
10000
1000
100
10
1
1000
100
10
1
0
50
100
150
200
0
100
150
200
Number of clients
Number of clients
10 clients
27/05/2008
50
Building data grids with iRODS
100 clients
200 clients
14
Concurrent tests for mix - PostgreSQL
iput
ils
1000
Time in seconds
Time in seconds
10000
1000
100
10
1
100
10
1
0.1
0.1
0
50
100
150
0
200
50
100
200
150
200
Number of clients
Number of clients
ireg
iget
10000
Time in seconds
10000
Time in seconds
150
1000
100
10
1
0.1
1000
100
10
1
0.1
0
50
100
150
200
0
Number of clients
100
Number of clients
10 clients
27/05/2008
50
Building data grids with iRODS
100 clients
200 clients
15
Concurrent test for mix - Oracle
ils
iput – Oracle ICAT

Because of limit of inodes, we
cannot put any more files
Time in seconds

10000
1000
100
10
1
0
50
100
150
200
150
200
Number of clients
iget
10000
1000
Time in seconds
Time in seconds
ireg
100
10
1000
100
10
1
1
0
50
100
150
0
200
100
Number of clients
Number of clients
10 clients
27/05/2008
50
Building data grids with iRODS
100 clients
200 clients
16
Compare with iRODS and SRB

For the scaling test

Data



Measurement



Using the same directory at client machine
1000 files of 1000 bytes each
Ingesting directory and listing the collection
Performance measured for every directory operation
For the nested collection test

Data



Measurement



27/05/2008
Registering same directory at client machine
100 files of 100 bytes each
Ingesting directory and listing the collection
Making nested collection every 10 collection
Performance measured for every directory operation
Building data grids with iRODS
17
System used

iRODS system at KEK

ICAT enabled iRODS server on RHEL3




iRODS resource and client on RHEL3



Dual Intel Xeon 2.8GHz
PostgreSQL 8.2.5 is running on the same machine
iRODS 1.0
Dual Intel Xeon 2.8GHz
Linux file resource from local disk system
SRB system at KEK

MCAT enabled SRB server on RHEL3




SRB resource and client on RHEL3


27/05/2008
Dual Intel Xeon 2.8GHz
PostgreSQL 8.2.5 is running on the same machine
SRB 3.5.0
Dual Intel Xeon 2.8GHz
Linux file resource from local disk system
Building data grids with iRODS
18
Scaling test - iRODS and SRB
Sput
iput
time in seconds
tme in seconds
50
40
30
20
10
0
0
200
400
600
800
number of files [x 1,000]
3000
2500
2000
1500
1000
500
0
0
1000
10
ils
50
Sls
0.5
1
time in seconds
time in seconds
20
30
40
number of files [x 1,000]
0.4
0.3
0.2
0.1
0
0.8
0.6
0.4
0.2
0
0
27/05/2008
200
400
600
800
number of files [x 1,000]
1000
0
Building data grids with iRODS
10
20
30
number of files [x 1,000]
40
50
19
nested collection –iRODS and SRB
Sregister
70
60
50
40
30
20
10
0
nest
flat
0
50
100
150
number of files [x 1,000]
time in seconds
time in seconds
ireg
30
25
20
15
10
5
0
nest
flat
0
200
ils
nest
flat
2
1
0
100
150
200
number of files [x 1,000]
coll-1220: USER_PATH_EXCEEDS_MAX
27/05/2008
time in seconds
time in seconds
3
50
50
Sls
4
0
10
20
30
40
number of files [x 1,000]
0.6
0.5
0.4
0.3
0.2
0.1
0
nest
flat
0
10
20
30
40
number of files [x 1,000]
50
nest-47: Error Problem running command
Building data grids with iRODS
20
SRB system for Belle at KEK
LCG user
Melbourne
LCG user
KU
CYFRONET
APAN
GEANT2
KEK-DMZ
GridFTP
ASGC
NCU
SINET
KEK FW
SRB
KEK-LAN
KEK-2
Nagoya
KEK-1
SRB-DSI
dedicated inside
Pluggable
Extension


Both protocols are authorized by
GSI
Setting up on GridFTP server



NFS
MCAT
3.5PB
HSM
SRB client
grid-mapfile for SRB user
SRB configuration file for
server and resource
Register LCG user DN on MCAT
27/05/2008
Belle Net
SRB server
Enhanced
GridFTP service
LSF
Computing Farm
Building data grids with iRODS
Still not integration with Grid
21
Belle analysis user
Plan for iRODS system at KEK

Data transfer for J-PARC project

Generate huge amount of imaging data
at Tokai


Store the data at Tokai storage once,
then copy to KEK and distribute for
collaborators


27/05/2008
About 1PB data in a year in total
Storage at Tokai are recycled
Tokai
KEK
60km
Bandwidth between 2sites
will be 10Gbps
Building data grids with iRODS
22
Acknowledgements


Special thanks to Jean-Yves Nief from CCIN2P3 for his help to setup iRODS system
and supporting at CC-IN2P3
Thanks to Adil Hasan from RAL for his help
to these tests
27/05/2008
Building data grids with iRODS
23
Back up
27/05/2008
Building data grids with iRODS
24
Bandwidth Lyon-KEK
Speed [MBytes/sec]
120
Lyon to KEK
KEK to Lyon
100
80
60
40
20
0
0
10
20
30
40
50
60
Time [min]

iperf with some options;



27/05/2008
-w 4M : TCP window size [Bytes]
-P 16 : the number of parallel threads
-i 5 : periodic bandwidth reports [sec]
Building data grids with iRODS
25
Summery of iRODS Performance


Transfer from KEK to CC-IN2P3 is not stable but
iput is better than simple transfer software (bbcp)
iRODS can manage 1 million files stability



In case of very nested collection, PostgreSQL ICAT takes
long time to register data
Better performance than SRB
Oracle ICAT can handle more than 300 clients at the
same time, but it takes time according to increase the
number of clients
27/05/2008
Building data grids with iRODS
26