A.R.M.S. Active Resource
Management Services
Presentation One
2/21/2013
1
Outline
Introductions
Societal Issue Examined
Michael Rajs
2/21/2013
2
Outline
•
•
•
•
•
•
•
•
•
Group Members and Roles:
slide 4
Introduce Mentor: slide 5
Societal Issue: slide 6
History: slides 7-11
Case Study: slides 12-16
Problem Statement: slide 18
Computer Components
Identified: slides 19 -21
Major Functional Component
Diagram: slide 22
Current Process Flow: slide
23
2/21/2013
•
•
•
•
•
•
•
•
Solution Statement: slide 25
Objectives: slide 26
Improved Process Flow:
slide 27
Competition Identified:
slides 28-30
Benefits of Solution: slide 32
Problems with Solution: slide
33
Conclusion: slide 34
References: slides 35-36
3
Group Members and Roles
• Michael Rajs (Group Manager)
• Adam Willis (Research Specialist)
• Sybil Acotanza (Visualization
Engineer)
• Scott Pardue (Team Leader)
• Jordan Heinrichs (Marketing Analyst)
• David Crook (Documentation
Specialist)
2/21/2013
4
Yaohang Li
•Is an Associate Professor
in the Department of
Computer Science at Old
Dominion University.
•His research interests are
in Computational
Biology, Markov Chain
Monte Carlo (MCMC)
methods and Parallel
Distributed Grid
Computing.
2/21/2013
5
What is the societal issue being faced?
How do researchers handle the
massive amounts of data they are
collecting?
2/21/2013
6
Historical Background
Adam Willis
2/21/2013
7
Collection of Data
• 1890 Census Recorded With an Electric Machine 1
• 1935 Social Security Act 2
• 1974 Privacy Act 3
• 1989 World Wide Web 4
• 1997 Big Data 5
• 2011 IBM’s Watson 6
• Now
“Every day, we create 2.5 quintillion bytes of data —
so much that 90% of the data in the world today has
been created in the last two years alone.”7
2/21/2013
8
Examples of Big Data
• Large Hadron Collider 8
– 150 million sensors report 40 million
times per second
• Facebook 9
– 2.5 billion – content items shared
– 2.7 billion – “Likes”
– 300 million – photos uploaded
• Walmart 8
– 1 million customer transactions
– 2.5 petabytes of data
2/21/2013
9
Big Data Analysis Hardware
• Cluster Computing 10
– A cluster consists of many nodes (computers).
– Big data can be generated and analyzed
quicker by spreading the workload amongst
the nodes.
2/21/2013
10
Managing the Cluster
• Distributed Resource
Management Systems (D-RMS)
–Job management subsystem
–Physical resource management
subsystem
–Scheduling and queuing
subsystem
2/21/2013
11
Case Study
Sybil Acotanza
2/21/2013
12
Dinosolve Case Study
• Bioinformatics
– Disulfide bond prediction program
2/21/2013
(Cronk, 2012)
13
Dinosolve Users
• Who will use it?
– Drug and antibody design
– Bio-energy development
– Genetic mapping11
• Why will they use it?
– 2% accuracy improvement12
2/21/2013
14
Dinosolve Web Site
2/21/2013 (Li & Yaseen, http://hpcr.cs.odu.edu/dinosolve/)
15
Dinosolve Possible Problems
• Hard resources for computation
– CPU cycles
– Memory
– Disk space
– Network bandwidth
• Server crashes
2/21/2013
16
Problem statement
Components of Hardware and Software
Current Process Flow
Scott Pardue
2/21/2013
17
What is the problem?
Processing time on big data sets is
computationally expensive and as
the volume of queries grows the
system will progressively drop in
performance until the system fails.
2/21/2013
18
What are the components of our current
system?
The current system uses the
following software and hardware.
2/21/2013
19
Software
• Unix operating system installed on
the dinosolve cluster
• Dinosolve algorithm
• Sun Grid Engine which will be our
Distributed Resource Management
System (D-RMS) installed on the
cluster.
• MySQL (database software)
• Web based user interface (website)
2/21/2013
20
Hardware
• MySQL database server
• A computer cluster to run the
dinosolve algorithm
• Web server for our web based
user interface
2/21/2013
21
Major Functional Component Diagram
2/21/2013
22
2/21/2013
23
Solution Statement
Objectives
Improved Process Flow
Competition Identified
Jordan Heinrichs
2/21/2013
24
How will we correct the problem?
We aim to configure a distributed
resource management system
(D-RMS), in this case Sun Grid
Engine (SGE), to handle resource
allocation on the dinosolve cluster.
2/21/2013
25
Objectives
• Interpret and visualize current
usage statistics
• Configure, utilize, and optimize
the SGE
• Aesthetically pleasing and
professional user interface
2/21/2013
26
Process Flow with Solution
2/21/2013
27
Competing Distributed Resource
Management Systems
• Sun Grid Engine (SGE)
• Portable Batch System (PBS)
• Load Sharing Facility (LSF)
2/21/2013
28
Competing Resource Management
Systems
Features of
systems
PBS
LSF
SGE
Supported
platforms
Unix
Unix & NT
Unix
Multi-cluster
support
Yes
Yes
No
System level
checkpoint restart
Yes
Yes
Yes
User level
checkpoint restart
No
Yes
Yes
Large
computational grid
support
No
No
No
Massive Scalability
Yes
Yes
Yes
Parallel job
support with Sun
HPC ClusterTools
Loose Integration
Tight Integration
Loose Integration
Distribution format
of end product
Source
Binary only
Binary and Source
Free?
Yes
No
Yes
Posix 1002.2d
compliance
Yes
No
Yes
2/21/2013
Reference 31
29
Competing Protein Prediction Servers
Dinosolve
DiANNA
Scrath Protein
Predictor
Accuracy
90.8%
81%
87%
Usability
X
X
X
508.22
compliance
percentage
67%
85%
67%
Professional
2/21/2013
Reference 19,20 and 21
30
Benefits of solution
Problems with solution
Conclusion
David Crook
2/21/2013
31
What benefits will come from attaining
our goals?
• Efficient utilization of available
resources
• Increased throughput of the cluster
• An intuitive and professional user
interface
• Rise in popularity due to excellent
accuracy, efficiency, and professional
design
2/21/2013
32
Problems with solution
• Improper synchronization of cluster
resources can lead to a deadlock in
the system
• Race conditions between the HPCR
cluster and the MySQL database
2/21/2013
33
Conclusion
With the updated user interface
and correctly configured Sun Grid
Engine we hope to establish a
reputable Disulfide Bonding
Prediction Server.
2/21/2013
34
References for history
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
http://www.columbia.edu/cu/computinghistory/hh/index.html
http://query.nytimes.com/gst/abstract.html?res=F50C11FE385D13728DDDAE0A
94DA415B868FF1D3
http://www.census.gov/history/pdf/kraus-natdatacenter.pdf
http://www.bbc.co.uk/history/historic_figures/berners_lee_tim.shtml
http://dl.acm.org/citation.cfm?id=266989.267068&coll=DL&dl=GUIDE
http://www.nytimes.com/2012/08/12/business/how-big-data-became-so-bigunboxed.html?_r=1
http://www-01.ibm.com/software/data/bigdata/
http://en.wikipedia.org/wiki/Big_data
http://techcrunch.com/2012/08/22/how-big-is-facebooks-data-2-5-billionpieces-of-content-and-500-terabytes-ingested-every-day/
http://en.wikipedia.org/wiki/Computer_cluster
2/21/2013
35
References for case study
11. Li, Y. (2010, September 1). CAREER: Novel Sampling Approaches for Protein
Modeling Applications [Abstract]. National Science Foundation Award
Abstract #1066471.
12. Li, Y., & Yaseen, A. (2012). Enhancing Protein Disulfide Bonding
Prediction Accuracy with Context-based Features. Biotechnology and
Bioinformatics Symposium
13. bioinformatics. 2011. In Merriam-Webster.com. Retrieved February 15, 2013,
from http://www.merriam-webster.com/dictionary/bioinformatics
14. Cronk, J. D. (2012). Disulfide Bond. Retrieved February 15, 2013, from
Biochemistry Dictionary:
http://guweb2.gonzaga.edu/faculty/cronk/biochem/Dindex.cfm?definition=disulfide_bond
15. Yan, Y., & Chapman, B. (2008). Comparative Study of Distributed Resource
Management Systems–SGE, LSF, PBS Pro, and LoadLeveler. Technical ReportCiteseerx.
16. Li, Y., & Yaseen, A. (2012). Dinosolve. Retrieved from
http://hpcr.cs.odu.edu/dinosolve/
2/21/2013
36
References for competition
17. Arvind Krishna, “Why Big Data? Why Now?”, IBM , 2011
URL: http://almaden.ibm.com/colloquium/resources/Why%20Big%20Data%20Krishna.PDF
18. Yonghong Yan, Barbara M. Chapman, Comparative Study of Distributed Resource Management
Systems - SGE, LSF, PBS Pro, and LoadLeveler, Department of Computer Science, University of Houston, May
2005 (pdf)
19. Dr. Li’s site
http://hpcr.cs.odu.edu/dinosolve/
20. Scratch Predictor
http://scratch.proteomics.ics.uci.edu/
21. DiANNA server
http://clavius.bc.edu/~clotelab/DiANNA/
Portable Batch System (PBS)
22. http://resources.altair.com/pbs/documentation/support/PBSProUserGuide12-2.pdf
23. http://www.pbsworks.com/SupportDocuments.aspx?AspxAutoDetectCookieSupport=1
24. http://resources.altair.com/pbs/documentation/support/PBSProRefGuide12-2.pdf
25.http://resources.altair.com/pbs/documentation/support/PBSProAdminGuide12-2.pdf
26.http://www.pbsworks.com/(S(tykrsyqbemmlf3o5zwrmjrgf))/images/solutions-en-US/PBS-Pro_DatasheetUSA_WEB.pdf
27.http://agendafisica.files.wordpress.com/2011/05/pbs.pdf
Moab HPC Suite
28.http://www.adaptivecomputing.com/publication/420/wppa_open/
IBM Platform LSF
29.http://public.dhe.ibm.com/common/ssi/ecm/en/dcd12354usen/DCD12354USEN.PDF
Apache Hadoop with Zookeeper
30. http://zookeeper.apache.org/doc/current/zookeeperOver.html
31. http://www.cloud-net.org/~swsellis/tech/solaris/performance/doc/blueprints/0102/jobsys.pdf
2/19/2013
References
37