KOLKATA Tier-2@Alice Grid
Kolkata Tier-2 and Network
Shown many data transfers
(few are simultaneously going on)
Site Name :- IN-DAE-VECC-02
VO :- ALICE
Monalisa Name:- Kolkata-Cream
City:- KOLKATA Country :- INDIA
KOLKATA Tier-2@Alice Grid
ALICE Collaboration

~ 1/2 ATLAS, CMS, ~ 2x LHCb
~1100 people
30 countries,

80 Institutes


Total weight
Overall diameter
Overall length
Magnetic Field
10,000t
16.00m
25m
0.4Tesla
Introduction to
ALICE
Lots of data to Process:Solution
Slide from Latchezar Betev PPT
GRID COMPUTING
KOLKATA Tier-2@Alice Grid
Tier 0
ALICE Grid
Structure
Experimental
site
CAF (T1 +T2)
CERN computer centre,
Geneva
Central Analysis Facility
10 -100 Gbps links
Tier 1
8 centres
ASIA
(KISTI)
TAIWAN Germany Italy
France
1 -10 Gbps links
Tier 2 India
Logical, geographical
China
JAPAN
association of regional groups
IN_DAE_VECC_02
100 -1000 Mbps links
Different Universities, Institutes
Tier 3
Individual scientist’s PC
EHEPAG
(VECC)
Jammu U.
Panjab U.
Pakistan
Jaipur U.
KOLKATA Tier-2@Alice Grid
Kolkata Tier-2 at Monalisa
Consistently
800 ALICE
Jobs are
running.
Vikas Singhal, VECC, INDIA
KOLKATA Tier-2@Alice Grid
Grid Computing Facility
Vikas Singhal, VECC, INDIA
KOLKATA Tier-2@Alice Grid
3 M ALICE Job completed Successfully
during last 5 Years
Total :- 888 Cores
DELL Quad Core Blades 38 * 2 * 4 *2 = 608 (HT)
HP Quad Core Blades 8 * 2 * 4 * 2 = 128 (HT)
HP Dual Core Blades 38 * 4 = 156
Equivalent 9K HEPSpec 2006 Computing Resources.
Storage :- 146 TB
XrootD Redirector
3 Disk Servers SAN Based
78TB (IBM) + (36 + 32)TB (HP EVA)
1Gbps WAN Network speed.
10Gbps Backbone Network
Vikas Singhal, VECC, INDIA
KOLKATA Tier-2@Alice Grid
NKN @ INDIA
7
KOLKATA Tier-2@Alice Grid
NKN
TOPOLOGY
April 22, 2009
Dec. 23, 2009
Developing e-inftrastructures in India
8
KOLKATA Tier-2@Alice Grid
Physical Network Connectivity upto Kolkata
CERN LHCONE
10 Gbps
TEIN-3
Network
TIFR
Mumbai
Internet
1 Gbps
POP in Mumbai
10 Gbps
Core
Network
POP in Kolkata
SINP
Kolkata
1 Gbps
1 Gbps
10 Gbps
4-10 Gbps
1 Gbps
x Gbps
VECC
Kolkata
1 Gbps
Kolkata
Tier-2
KOLKATA Tier-2@Alice Grid
10 Gigabit NKN Router at VECC
1Gb
optical
Fiber Backbone
Fiber Link
Grid Computing Facility
1 Gigabit Ethernet Link
XrootD
Redirector
gridce02
Grid-peer
grid
DNS
Server
XrootD
Disk Servers
10 Gigabit Fiber Link
ALICE
Tier-2
Worker
nodes
Tier-3
Nodes
ALICE
Centre
HP
IBM
SAN
Storage
KOLKATA Tier-2@Alice Grid
Kolkata TIER-2 centre Network diagram
1Gbps Copper
Internet
Router
1Gbps Fiber Backbone
Grid-peer
gridse001
naamak
wn001
Computing
Nodes
wn002
Dache-server Installer
4 – Xrootd
Disk Servers
10G Backbone Switch
25 Nodes
Dell and
Wipro Blades
Cluster with
25 TB of
As Tier-3
wn024
wn025
GRID-PEER Tier-3
cluster with 32 & 64 bit
machine
10G Backbone Switch
Computing
Nodes
10 Gbps
Switch-1
Switch
10 Gbps
grid
SINP
gridce02
grid01
Backup-server
130 TB
Backup
wn045
wn046
Switch-2
DELL and HP
Blade Server
with Multi
Core Xeon 3.0
GHz
wn122
wn123
192.168.x.x (Stand by)
144.16.112.xx/27
IN-DAE-VECC-02 Site
with 64 bit machine
Vikas Singhal, VECC, INDIA
KOLKATA Tier-2@Alice Grid
From 128Kbps to 1Gbps WAN Network
Started with
----128Kbps shared link
2002
----512Kbps
2003
----2Mbps Dedicated Link
2004
----4Mbps from Bharti
2006
----30Mbps from Reliance
2008
----100Mbps from VSNL (ERNET)
2009
----300 Mbps from NKN
2011
----1Gpbs
2014
Vikas Singhal, VECC, INDIA
KOLKATA Tier-2@Alice Grid
Kolkata SE Traffic
# cat route-eth0
default via 192.168.1.12 dev eth0
144.16.112.16 via 192.168.1.16 dev eth0
144.16.112.15 via 192.168.1.15 dev eth0
144.16.112.14 via 192.168.1.14 dev eth0
144.16.112.9 via 192.168.1.9 dev eth0
Vikas Singhal, VECC, INDIA
KOLKATA Tier-2@Alice Grid
Tier-3 (grid-peer) cluster
HP c7000e Blade enclosure.
8 Number of HP ProLiant DL585
G7 AMD Blade
Each blade has 2 * 16 Core
2.5GHz AMD CPU
Each blade has 256 GB RAM.
Installed SLC 6.5 x86_64 OS
1U Sliding LCD Monitor with 16
port KVM
Dell PowerEdge M1000e Blade
Server Chassis.
16 Number of Dell PowerEdge
M610 High Performance Intel
Blade
Each blade has Nehalem based 2
* Intel Quad Core E5530 Xeon
2.4GHz CPU with 8MB cache.
Each blade has16GB RAM.
Each blade has 2 * 146GB
Mounted as RAID1.
Installed SLC 6.5 x86_64 OS
IBM SAN Storage
3 * 16 * 2TB SAS hard disks.
3 * 26 TB = 78 TB Usable space
after RAID5 and Hot Spare.
Total 14 Nodes for VECC users
and Collaborating Institutes.
13 Collaborating Institutes.
64bit computing nodes.
7 Non-interactive AMD nodes.
7 Interactive AMD + Intel nodes.
78 TB IBM SAN storage.
Storage mounted via NAS on all
the Compute Nodes.
10Gbps Network backbone.
Managing via ROCKS6.
Using PBS Job Scheduler.
50 + active users (across India.)
30 + active users (in VECC.)
Quota implemented for the users.
Root, Geant3, Aliroot, Alien,
Fairsoft, CBMroot, gFortran etc
user specific software installed.
KOLKATA Tier-2@Alice Grid
Future Plan and Decisions
 Decided to increase Kolkata Tier-2 resources as per ALICE
Fair Share rule.
 10 Cooled Rack Space allotted.
 Indent raised to procure more than 30K HEP Spec 2006
equivalent computing Power.
 Will procure 1200 Physical cores or 2400 Hyperthreaded core.
 12 Core processors are the best suitable as per price
performance point.
 Plan to purchase only bare disks as per requirement.
 3 * 2U servers comprising 12*4TB disks configured under
Open Source EOS storage, will be included into KOLKATA SE.
KOLKATA Tier-2@Alice Grid
Network Backbone Planning
10Gb backbone,
40Gb core switch connection.
Both Fiber and Copper connectivity.
Connection via DAC Cable.
Vikas Singhal, VECC, INDIA
Dual path, Full Redundancy.
KOLKATA Tier-2@Alice Grid
Thank You
DURGA POOJA
Festival of GODESS
DURGA.
Celebrated in Month of
October.
Idol will be submerged into
the Water.
Vikas Singhal, VECC, INDIA