SCIENCE & TECHNOLOGY & INFORMATION TECHNOLOGY DEPARTMENT GOVERNMENT OF KHYBER PAKHTUNKHWA PC-1 Development of Super Computer ADP# 926-110512 APPROVED COST: Rs. 10.00 (MILLION) DIRECTORTE OF SCIENCE AND TECHNOLOGY GOVERNMENT OF KHYBER PAKHTUNKHWA November, 2011 1. Name of the Project Development of Super Computer 2. Location Ghulam Ishaq Khan Institute (GIKI), Topi (23640), K.P.K., Pakistan. 3.Authorities responsible for: Sponsors Directorate of Science and Technology (DOST), Govt. of K.P.K., Pakistan. GIK Institute will fund the operational cost (e.g. electricity, air conditioning, room, installation, administration, maintenance etc.) Execution Directorate of Science and Technology, K.P.K. Ghulam Ishaq Khan Institute (GIKI), Topi (23640), K.P.K., Pakistan. Operation and maintenance Ghulam Ishaq Khan Institute, Topi, K.P.K., Pakistan. Directorate of Science and Technology, Peshawar, K.P.K. Concerned Federal Ministry: N/A 4.Plan provision The project has been included in the ADP 2011-12 with a cost of Rs. 10.00 million. Six million is allocated for the year 2011-12 and the rest of four million will be allocated for year 2012-13. 5.Project objectives The focus of this proposal is to develop the advance super computing infrastructure at K.P.K with a state of the art High Performance Computing (HPC) cluster system. This facility will be installed at Ghulam Ishaq Khan Institute (GIKI), Topi. The facility will be used to carry out research work in data mining, computational biology, computational mechanics, etc. The HPC cluster will provide huge storage capacity (in Tera bytes capacity), fast processing power, reliability and efficient accurate resource for the problems involved million of elements and thousand of time steps including the data over several miles. These computational requirements needs an advance state of the art computing infrastructure to solve and store the data structures for current massively parallel architectures. The proposed state of the art architecture will provide very fast processing power for K.P.K research projects and also for academic research institutions in Pakistan. The main objectives of this project are: 1. To facilitate the Government Organization for the requirements of High Performance Super Computing like Weather forecast, Space Technology, etc. 1 2. Training of undergraduate/graduate students and industry on Super Computer and it usage via workshops/seminars/courses. 3. High performance computational research work for BS, MS and PhD students 4. Research for publication of papers in international research conferences and journals. 5. Human resource development in the field of high performance computing and computational science for K.P.K and as well as for other academic and research institution in Pakistan 6. Lectures, seminars and workshops on HPC related topics to other institutions in Pakistan especially K.P.K academic institutions and industry 7. Availability of the scalable high performance supercomputing facility on 7/24 basis to all over Pakistan for both the academia and industry. This project is mainly focused to support 200 MS/PhD students of GIKI as well as the research projects of the faculty (professors, associate professors, assistant professors and lecturers/research associates) of academic/research institutions in K.P.K and Pakistan. Apart from this, as discussed in the meeting with the secretary ST&IT on Saturday (October 15, 2011), it was decided by PI and his team to extend the facility of supercomputing infrastructure to other departments in the domain of K.P.K. region through a secure remote connection. This connection will be provided by HEC Pern2 network. However, GIKI may charge reasonable operational and other miscellaneous official expenses (like internet/bandwidth, electricity/air conditioning, the employees specially employed for looking after the cluster, etc.) from the users outside the GIK Institute, Topi. 6. Description and justification of project The Scheme “Development of Super Computer” was introduced in the CFY ADP of Khyber Pakhtunkhwa with an estimated cost of Rs.10.00 million for the purpose of promoting Research & Development capabilities in High performance super computing. Extensive Consultative meetings were held with various academic institutions of Khyber Pakhtunkhwa including GIKI(Ghulam Ishaq Khan Institute,Topi), UET Peshawar & University of Peshawar for the development of super computing infrastructure, during the meetings it was decided that GIKI which has previous track record on developing high performance super computing infrastructure and has the requisite highly skilled researchers team will house & develop the High performance super computing infrastructure. Ghulam Ishaq Khan Institute (GIKI) which is the second HEC-recognized institute/university to build the high performance computing facility in Pakistan after Comsats Institute of Information Technology, Islamabad. GIKI developed the computer cluster sponsored by Pak-US project. The project was launched in year 2006 and its aim was to develop the human resource development for computational mechanics program in Pakistan. This program has trained four MS students and one PhD scholar in the said field as well as several under-graduate students. One of the major contributions of this program is to develop the high performance computing (HPC) cluster for training purposes. Four other PhD students also graduated from GIKI, who used HPC computing facility for the research work of their PhD theses. 2 At current, The HPC system was built using dual core AMD Opteron processors. Traditionally HPC is also known as supercomputing. Many scientific fields like Computational Biology, Computational Physics, Reservoir Engineering, Cryptography, Geophysics, Weather Forecasting, etc, need very high computation power of massively parallel computers. Several high performance computer clusters were built in Pakistan to meet HPC requirements of similar kind of computational problems. The subsequent sub-sections give the brief overview of the available computer clusters in Pakistan. 6.1. Cluster at GIK Institute, Topi GIK Institute is the pioneer institute to develop the HPC lab in year 2006. This lab contains initially seventeen AMD dual core Opteron machines (1.6GHz). Initial proposal was to build the 64 nodes cluster [1], but due to limitation of budget only 17 nodes were installed. The cluster was connected using Giga bit Ethernet switch. Currently, the seven systems are in working order, where two of computers are with Dr. Muhammad Abid for the research work on Windows system. The total number of cores in the cluster is ten with 9 GB RAM and connected using Giga bit Ethernet switch. The computer cluster developed using Rocks cluster distribution based on CentOS (a linux based OS). GIK Institute has approved the budget to obtain a computer system consisting of twelve cores to enhance the computing capacity of the cluster with at least 16GB of RAM against the burnt systems. The new system will be used as the master node for existing GIKI HPC cluster. The new computer will be used as the master node and will enrich its processors capacity from 10 to 22 cores. This facility has been used to train both under-graduate and graduate students to familiarize with the current super computing technology. Most of times this computer cluster facility has been used for teaching assignments of Parallel Processing/Computing course, and different graduate and under-graduate final year projects. This training computer cluster facility does not provide the adequate power in terms of speed and memory. However, some intermediate level of parallel computing resource is deliberately needed to perform adequate research in Pakistan for computational science. 6.2. Cluster at KICS, UET, Lahore The High Performance Computing and Networking Lab (HPCNL) at KICS, was established in December 2007 with a number of PIII and PIV processor based systems. It is comprised of a cluster of 30 processors and connected with gigabit Ethernet network topology [2]. The main purpose of the cluster is to provide the training on Hadoop platform, which provides the distributed file system including google’s Map-Reduce programming concept. 6.3. Cluster at SEECS, NUST, Islamabad The High Performance Computing Lab at SEECS, NUST consists of four HPC clusters based on SUN Ultra Spark, AMD Opteron and Intel Xeon based platforms [3]. Very brief information is as under: 1. Raad – 64 cores Solaris cluster with 128 GB RAM and connected with Myrinet 2. Barq – 32 cores Linux cluster with 36 GB RAM and connected with Gigabit Ethernet 3 3. Burraq – 64 cores Linux cluster with 128 GB RAM and connected with Gigabit Ethernet 4. Supercomputing Research & Education Center (ScREC) of 128 TFLOPs The ScREC is the state of the art massively parallel computational facility, which produces 128 TFLOPs. This is the currently most advanced supercomputer in Pakistan using NVDIA as accelerator. It will be put into working condition in a couple of months [4]. This supercomputer is built by Dell using Intel Xeon and NVDIA graphic processors. 6.4. Cluster at NED, Karachi This HPC lab contains 50 nodes Linux Cluster, which is based on Beowulf architecture which calls for dedicated CPUs for computation purposes having their own independent memory, storage and processing power. This system is built using Intel Xeon processors and connected via gigabit Ether network [5]. 6.5. Cluster at CIIT, Islamabad The aim of high performance computing lab at CIIT-Islamabad was to set up a modest (40-50 GFLOPS/sec) computing cluster using commodity-of-the-shelf (COTS) components. For this purpose, the lab was established using normal COTS components by in-house assembling [6] and [7]. 6.6. Justification Initially, supercomputing facility at GIKI has been used for adaptive moving mesh generation algorithm in the Arbitrary Lagrangian and Eulerian (ALE) frame of reference and stabilized mixed finite element methods for Darcy’s law. Numerical experiments are conducted on different hardware platforms with different problems. The results are verified using both OpenMP and Message Passing Interface (MPI) libraries for shared and distributed memory systems, respectively. The software package ANSYS was used for research work. The MS/PhD and under-graduate students of all the faculties at GIKI Institute used the supercomputing facility. In computational domain, most of the computational problems involve millions of elements and thousands of time steps in optimizing the problems for transient analysis. Their analysis over massively parallel architectures can become a formidable task for the problems. Most of the times, the data spans over several miles and involve the data over Giga/Tera bytes for the research analysis. This huge data needs basic infrastructure to solve and store the data structures for today’s massively parallel computers. So an advance HPC facility is deliberately needed including distributed hard disks to store the data. Once this HPC facility is established, researchers at GIKI inside the campus or any researchers from both the academia and business/commercial institutions can connect using secure shell through internet facility. This facility will be easily accessible to all the universities in K.P.K and even in Pakistan for state of the art research using PERN (Pakistan Education Research Network) network established by HEC, Pakistan. GIKI is the best place in Pakistan, where, there is no electricity load shedding problem. Since electric power is mostly available on 7/24 basis at GIKI computing facility for all the 4 Pakistani research institutes as well as industry. The existing GIKI network facility will be going to be upgraded to Pern2 connection, needed for some of the proposed projects for distant academic institutions in K.P.K or elsewhere in Pakistan. GIKI has already applied for the provision of Pern2 connection, which likely to be acquired by the end of March 2012. The application for the provision of bandwidth for Pern2 connection is attached herewith in Annexure A. 6.7. Methodology to up-grade the computer cluster The upgraded HPC cluster will consist of individual multi-core systems and that will be connected with a high performance interconnection to gain maximum speed to solve real world problems. The upgraded HPC cluster will contain one master node including two network cards and several slave nodes using blade technology. This cluster will contain KVM switch to connect all the slave nodes with single monitor, keyboard and mouse. Master node will be connected to outer world externally using World Wide Web (WWW) and internally will be connected to all slave nodes using a high performance interconnection like hybrid version of both the Infiniband and Ethernet switches. The Top 500 supercomputer, June 2011 listing, is established at RIKEN Advanced Institute for Computational Science (AICS), Japan, which is based on Linux OS without acceleration of GPUs [8]. The system is built using SPARC64 VIIIfx processors and based on computer cluster architecture. Thus we would like to establish the proposed supercomputing facility at GIKI using the HPC cluster topology. The Linux flavored Rocks HPC cluster distribution will be installed to put the cluster in working condition. Firstly the rocks will be installed on the master node and then it will be installed on all the slave nodes using network boot options from the master node. Figure 1 shows the system graphically, in which the HPC cluster can be accessed via WWW technology to different academic institutes, business centers and government institutions in Pakistan. Moreover, all the slave nodes will contain the local hard disks and Hadoop file system might be installed and data will be stored in distributed manner to use distributed algorithms efficiently. Finally the whole system will be connected to electric power supply with high precision surge control instruments to avoid the malfunctioning of the system due to high voltage surges, etc. For such, an advanced and state of the art HPC cluster, it is necessary to have adequate maintenance and upgrading budget in the proposal. 6.8. Remote access to GIKI cluster The installed HPC cluster at GIKI would be remotely accessible to all allied partners using the secure shell (ssh) protocol from cluster.giki.edu.pk address. For this purpose a user needs an internet account as described in Figure 1. All the users will be provided a username along with a password for the secure remote access to the master node of GIKI’s HPC cluster using a ssh client for both Windows and Linux operating systems. The user needs to understand the basic of Linux’s command line to use the cluster. Particularly, a user can submit a task using a batch file to the job scheduler program in master node to utilize the high performance supercomputing facility. For any query, a remote user can send an email at hpc@giki.edu.pk regarding the creation of cluster account and as well as the usage. The team members will respond to 5 all the queries and questions raised at the said email address as urgent as possible. HPC training manuals, programs, courses and workshops would be launched at GIKI and other allied institutions/organizations. Master Node Slave 1 Slave 2 HS1 HS2 OK1 OK2 PS 1 2 3 4 5 6 7 8 9 1 01 11 2 CO L- C O N SO LE ACTSTA- Slave 3 Slave n www Government U N IV E R S IT Y Home Business University Figure 1: Proposed Architecture of Computer Cluster 6.9. Hardware Specification 1. 2. 3. 4. 5. RAM: Total 256-512 GB RAM or 16-24 GB RAM per node Processors: Total 128-256 Cores (multi processor systems) Hardisk: 4 TB in master node and 1-2 TB in each Slave node with Raid option External memory: 8-16TB at least with RAID option (Optional) Network: Hybrid hierarchical version of Infiniband/10GB Ethernet for high performance communication 6. Graphic processors due to oil & gas project (Optional) 7. Electric surge devices/UPS to protect the hardware from the high voltage. 6.10. Software Requirements 1. 2. 3. 4. 5. 6. 7. 8. Rocks-Cluster distribution Parallel programming languages (C/C++ and Fortran) Parmetis (A parallel mesh partitioning library) Abaqus (A finite element analysis software) ANSYS (A finite element analysis software) MATLAB (Mathematical and statistical tool box) Mahoot (Scalable machine learning and data mining library) Many others arranged through the courtesy of research groups in Pakistan 7. Capital cost estimates Total cost Date of estimation of project cost Basis of determination of capital cost = = = 6 10 Million Rs. November 2011 Market Survey 7.1 Total Project Cost Breakup S. No. Description of Expenditure Cost 1 Pay & Allowances Annexure reference I 1.107 II 2 Machinery & Equipment 7.5078 III 5 Other Expenditures 1.3852 TOTAL 10.00 7.2 Pay & Allowances S.# 1 Position Project Coordinator(B-17) No. of Posts Salary Package per month per person 1 Total Salary Per month 45,000 Annex-I Annual Increment @5% for 01 years 45,000 Project Allow per Year 27000 Total: 7.3 Total salary for Year 1 Total salary Year 2 540000 567000 1.107 m Machinery & Equipment SNO 1. Item Master node, Network switch, KVM, surge controller, LCD 2. Compute Nodes with surge controller, rack and other parts needed for the connection 3. Color laser printer with scanner Quantity 1 Annex-II Year1 3 6 to10 Year 2 0 2.9378 Total 3 4.4378 1.5 1 0 0.07 0.07 Total: 4.5 7 3.0078 7.5078 7.4 Other Expenditures S# Year 1st ITEMS 1 2 3 4 5 Annex-III Year 2nd 0.18 others T.A/D.A Stationery Medical Charges workshops Total(Rs.m) 0.505 0.155 0.01 0.09 0.2 0.1552 0.01 0.08 0 0.685 0.3102 0.02 0.17 0.2 Grand Total: 1.3852 7.5 Component-wise, year-wise physical activities Items Unit Year-I Year-II A Staff 1 Recruitment will be completed Nil B Master node, Network switch, KVM, surge controller, LCD Compute Nodes with surge controller, rack and other parts needed for the connection Color laser printer with scanner Workshop C D E 1 Procurement and installation Nil 6-10 Partial Procurement and installation Partial Procurement and installation 1 Nil Procurement 1 Nil HPC workshop at GIKI 8 7.6 Year-wise/component-wise financial phasing (Million Rs) Total Year-I Local 0.540 Item A B C D E Total Year-II Total FEC Total Local 0.540 0 0.567 0.567 FE C 0 3 1.5 0 0 3 1.5 0 0 0 0 0 0 2.9378 0.07 0.2 0 2.937 0.07 0.2 0 0 0 0 5.04 5.04 3.7748 3.7748 0 0 0 Total Local 1.107 3 4.4378 0.07 0.2 8.8148 1.107 3 4.4378 0.07 0.2 8.8148 FEC 0 0 0 0 0 0 8. Annual operating cost(To be born by GIK) Annual operating cost like electricity, air conditioning, room, installation, man-power, etc. will be provided by GIK Institute, Topi. Furthermore, GIKI will be responsible for power surge and cooling system for Supper Computer. Table 1. Cost born by GIK Institute against super computing infrastructure Sr. No Name Quantity Rate 1. Bandwidth 5 MB dedicated 2. Electricity charges for 4.5 ton AC per month Electricity charges for one computer 600W** per month 3168 units 80 USD per MB 11 PKR per unit* 3a. 3b. 4. (600W*24hours *30 days)/1000 = 432 units*** Electricity charges for 432*15 = 6480 11 PKR per maximum 15 computers units unit* Employee Expense 1 dedicated 50,000 Total expenses with 5 MB dedicated bandwidth Total price in PKR per month 36000 34848 71280 50000 192128 Total annual operating cost/expenditure with 5 MB dedicated is 2305536 PKR. i.e. 2.3 million PKR. If the whole cluster runs for four and half years then at least GIKI will be paying 10 million PKR. 9 9. Demand supply analysis: N/A 10. Financial plan 100% financed by Provincial Government of K.P.K. Funds will be allocated from the ADP 2011-12. First installment of Rs. Six million will be released in Year 2011-2012. Master node and some of slave nodes including network switch will be purchased from this budget. Second installment of Rs. Four million will be released in Year 2012-2013. In this budget rest of slave nodes will be purchased to put the computer cluster in full working order. GIK Institute will bear the operational/maintenance expenditures. 11. (a) Project benefits and analysis Financial: There is no direct financial benefit to government of K.P.K in short terms, however in the long run the initiative will help in skilled human resource development of the province and Pakistan as a whole. Social: The project will train graduate students and professional from the industry to current state of the art computing environment in K.P.K. region. Later on, this will help Pakistan to become a self reliant country in the world for the supercomputing paradigm. This will also help in human resource development for the potential technologies in up scaling & improving the life standards. Environmental: This project is not hazardous for the environment. (b) Project analysis Quantifiable output of the project 1. 2. 3. 4. Teaching/Training of academic students and industrial personnel Final year projects, MS & PhD theses Research publications Lecture, seminars and workshops on supercomputing related topics to other institutions in Pakistan especially K.P.K organizations 5. Regular courses related to parallel processing and computation at GIKI Unit cost analysis Not applicable 10 Employment generation (direct and indirect): The project will create some direct and indirect employment opportunities for the people of KPK. Impact of delays on project cost and viability Due to high inflation rate in Pakistan, the delay in fund release may impact on the viability of the project at least in short term and can reduce the overall power of supercomputer. 12. a) Implementation of the project The project will be implemented at GIK Institute, Topi. K.P.K. and will start immediately after the release of 1st installments from Finance department, K.P.K. The computer cluster will be fully functional by June 2013. Meanwhile training will be given to allied institutions and both the graduate and undergraduate students of GIKI. The following activities will be done in chronological order for successful implementation of the project: 1. Transfer of 1st Installment 2. Quotations to purchase hardware from different vendors in Pakistan. 3. Training of undergraduate and graduate students in GIKI by offering parallel processing and parallel computation courses to them. 4. Purchasing of Master node, high performance network switch, KVM and miscellaneous items to put the initial computer cluster in working order. 5. Installation of operating system and cluster software package onto the purchased equipments. 6. Installation of software required by allied partners. 7. Transfer of 2nd Installment. 8. Quotations for the rest of equipments and purchasing 9. Connect to the computer cluster to scale up its computational power. 10. Conducting courses/workshops related to parallel computation in allied institutions. 11. Meanwhile the students will use the computing facility for their projects. 12. Find out the international potential collaborator to update the skills of GIKI staff and GIKI staff will transfer the knowledge to other allied institutions through courses/seminars/workshops and co-supervision of the research projects. b) Result Based Monitoring (RBM) Indicators S.No 1 2 Input Workshops/ courses/ seminars Cluster hardware Reports 3 Outcome Targets after Baseline Completion of Indicator Project Output Number of Workshops/ seminars/ courses held Installation of software Published in recognized journals and 11 Limited relevant HPC acitivities Training of students for high performance computing -do- Utilization of cluster -do- Number of published papers Targeted Impact Trained human resource for HPC Skill developmen t HEC approved conference / 4 5 13. Students Online access conference Complete degree Access of the cluster to different institutions limited MS/PhD degree Number of concurrent users accessing the HPC facility journals HRD HPC knowledge to different institutions Management structure and manpower requirements The project will be implemented through a team based in the Ghulam Ishaq Khan Institute, Topi, K.P.K and a Coordinator based in DoST,Peshawar. The human resource provided by GIKI will comprise of the following: In-charge of super computer at GIKI /Principal Investigator o Qualification: PhD/MS System Administrator at GIKI o Qualification: BS/MS Assistant System Administrator o Qualification: BS The following is the list of core team members, who will be mainly responsible for working of the computer cluster and its maximum utilization for their research projects: A. Core Team Members at GIKI 1. 2. 3. 4. 5. Dr. Masroor Hussain, Principal Investigator (PI), FCSE, GIK Institute, Topi Prof. Dr. Mushtaq Ahmad, Adviser, FCSE, GIKI Dr. Fawad Hussain, Member, FCSE, GIKI Dr. Muhammad Abid, Member, Dean-FME, GIKI Dr. Zain ul Abdeen, Member, FMSE, GIKI B. Consultant 1. Prof. Dr. Aftab A. Malik, Co-PI, Dean-FCSE, GIKI 2. Prof. Dr. Ashfaq Khokhar, Honorary Consultant, UIC, USA ToRs Of Project Staff Based in DoST, Peshawar Coordinator B-17: Qualification: M.Sc Computer Science Age: 30-40 Years Salary: As per Project Policy Experience: Five Years Relevant experience in a govt., Public or Private organization of repute. Skill required/Responsibilities: Hands on experience of working as programmer Super vision of data processing operations Develop & write policy & procedures for super computer usage across a wide domain of users. Coordination of development activities with various stakeholders Experience with security mechanism and audit procedures 12 14. Stakeholders/users Jobs coordination Monitoring the utilization of super computer Report and resolve users job processing issues Conduct performance review meetings of development activities Research and improve data processing activities for better utilization of the computing resources Schedule/conduct hardware preventive maintenance meetings with stakeholders Write technical and administrative reports Assist in procurement of computer hardware and software Evaluation of computer hardware and software Additional projects/decisions required It will be of great interest of Govt. of K.P.K. and GIKI to train post-graduate students of the region to high performance computing paradigm by funding couple of PhD (or MS leading to PhD) students at GIK Institute, Topi. It is highly desirable to pay Rs. 20,000 per month stipend to a PhD student for threefour years and three year PhD schooling fee at GIKI as an additional project. It will also be good to induct a few MS students at GIKI with Rs. 10,000 per month stipend per student. ANSYS, Abaqus and MATLAB software products will be purchased from other budgets by allied partners and their cost is not included in this proposal. However, alternative open source softwares may be installed on the HPC cluster as and when required. References [1]. Pak-US Research Project, http://www.giki.edu.pk/go/pakus/index.htm [2]. High Performance Computing and Networking Lab, http://www.kics.edu.pk/hp-labb.php [3]. High Performance Computing Lab, http://hpc.seecs.nust.edu.pk/ [4]. Research Centre for Modeling & Simulation, http://rcms.nust.edu.pk/ [5]. High Performance Computing Center, http://www.neduet.edu.pk/it_department/research_development/MS-Haris.htm [6]. CIIT Computational Cluster Research Project (CCCRP), http://www.ciitisb.edu.pk/Servicesandfacilities/CCCRPLab.aspx [7]. A.F.M. Ishaq and M.I. Khan, Supercomputing: A Roadmap for the OIC Member States, http://init.org.pk/papersandpublications/P8.pdf [8]. Top500 Supercomputer Site, www.top500.org 13 15. Certificate PREPARED BY: Principal Investigator Dr. Masroor Hussain Assistant Professor Faculty of Computer Science & Engineering Ghulam Ishaq Khan Institute, Topi, KPK Email: hussain@giki.edu.pk Phone: (0938) 271-858 Ex. 2237 Fax: (0938) 271-878 Date: Signature: Mr.HALEEM SHAH Planning Officer Directorate of Science & Technology Peshawar, K.P.K. Phone: 091-9214400 Date: Signature: CHECKED BY: Rector Mr. Jehangir Bashar Ghulam Ishaq Khan Institute, Topi, KPK Email: rector@giki.edu.pk Phone: (0938) 271-858 Ex. 2201 Date: Signature: Director Dr. Javed Abbas Bangash Directorate of Science and Technology Govt. of Khyber Pakhtunkhwa Date: Signature: APPROVED BY: Secretary Mr. Muhammad Humayun Science & Technology and Information Technology Department Govt. of Khyber Pakhtunkhwa Date: Signature: 14