Center for
Computational Sciences
University of Tsukuba
筑波大学計算科学研究センター
1
About Tsukuba City




Tsukuba City is about 50 km (31 miles)
northeast of Tokyo and 40 km (25 miles)
northwest of the New Tokyo International
Airport in Narita.
The city is home to the country's national
research facilities encompassing such
research fields as science, industry,
agriculture and forestry, environment and
space development.
It also houses institutions of higher learning
including the University of Tsukuba
campuses. About 19,000 researchers (40%
of the total for the whole country), 5,000 of
whom hold doctorate degrees, are working
for researches.
Tsukuba has a new train that connects
Tsukuba to Akihabara (Tokyo) in 45 minutes.
The train, called Tsukuba Express, started
running on August 24, 2005.
2
University of Tsukuba
History: After the relocation of the Tokyo University of Education,
the University of Tsukuba celebrates its 30th anniversary in 2003.
#faculty 1702
#undergraduates 10382
#graduates 6269




• School of Humanities and Culture
• School of Social and International
Studies
• School of Human Sciences
• School of Life and Environmental
Sciences
• School of Science and Engineering
• School of Informatics
• School of Medicine and Medical
Sciences
• School of Health and Physical
Education
• School of Art and Design
• Graduate School of Humanities and
Social Sciences
• Graduate School of Business
Sciences
• Graduate School of Pure and Applied
Sciences
• Graduate School of Systems and
Information Engineering
• Graduate School of Life and
Environmental Sciences
• Graduate School of Comprehensive
Human Sciences
• Graduate School of Library,
Information and Media Studies
3
Computational science

Large-scale simulations using supercomputers

Critical and cutting-edge methodology in all of science and engineering disciplines

The third “pillar” in modern science and technology
supercomputers
climate
bio
structure
CFD
Nano materials
Particles and universe
Simulation
Science and technology
theory
Experiment/
observation
Spring-8
Protein analysis
Relativity
(Einstein)
Subaru telescope
Meson theory
(Yukawa)
astronomy
Particle accelerators
Renormalization theory
(Tomonaga)
4
Cillision experiments
Development of Massively Parallel Computer Systems in University of Tsukuba




1977
1978
1996
2006
research begins (by Hoshino, Kawai)
1st machine
CP-PACS (top of Top500)
7th machine PACS-CS
1980
2nd PAXS-32
1978
1st
1989
5th QCDPAX
CP-PACS
 First large-scale general-purpose MPP system in Japan


Development supported by ``Research of Field Physics with
Dedicated Parallel Computers‘’ funded by the Ministry of
Education of the Japanese Government.
ranked as No. 1 system in the November 1996 Top 500 List.
PACS-CS
Massive Parallel Cluste
Collaboration by physicists and computer scientists
 Collaboration with industry, and released as Hitachi SR2201

(2006)
1996
6th CP-PACS (top of Top500 list in 1996)
PACS-9
Progress of Supercomputer
TFLOPS
GFLOPS
year
1978
system
PACS-9
1000 =1PFLOPS
6
Performance
10
BlueGene/L
5
7 KFlops
10
4
10
10
1980
PAX-32
50０ KFlops
CP-PACS
PACS-CS
1000
1983
PAX-128
4 MFlops
1984
PAX-32J
3 MFlops
1989
QCDPAX
１.4 GFlops
1996
CP-PACS
614 GFlops
2006
PACS-CS
14340 GFlops
100
Earth Simulator
100
1
0.1
QCD-PAX
10
0.01
1
1PFLOPS
=1000TFLOPS
=1000兆回／秒
の計算
1 TFLOPS
=1兆回／秒の計算
=1000GFLOPS
0.001
CRAY-1
0.1
1975
1980
1985
1990
1995
2000
2005
0.0001
2010
5
CCS, University of Tsukuba

Center for Computational Sciences http://www.ccs.tsukuba.ac.jp/


Mission of CCS is



to enable scientific discovery by computational science through the application of
advanced computing technologies,
and support researches of computational science in Japanese universities by running
leading-edge advanced computing systems
Collaborative researches with Computational Scientists (application) and
Computer Scientists (system)



Founded on 1992 as CCP (Center for Computational
Physics), expanded and reorganized to CCS in 2004
Needs from applications and Seeds from systems
More than 30 faculties and PDs, students: Particle Physics and Astrophysics. Material and Life Sciences,
Global Environment and Biological Sciences, High Performance Computing System, Data base & Data
Mining
Major Computing Facilities



PACS-CS, 14.3TF, Custom-made 2560 node PC cluster with 3D-hyper crossbar network by GbE.
FIRST, 33TF, 256 node hybrid PC cluster with GRAPE accelerators for Astrophysics.
T2K Tsukuba Cluster, 95.4 TF, 648 nodes, four quad-core Opetron processor (Appro XtremeServerX3) and 4 DDR inifiniband network for each node
6
Computing resources in CCS
PACS-CS
(2006～）
FIRST
(2007～）
GRAPE-6
A Special-purpose system to Astrophysics simulation
by hybrid computation of radiation and N-body.
 Each node is equipped by GRAPE-6, which is an
accelerator specialized for N-body Gravity calculation.
 256 nodes
 performance: cluster 3.5TFLOPS＋Grape-6 35TFLOPS





#node 2560 node
(Intel Xeon 2.8GHz, single core /node)
peak performance 14.34 TF
memory 5 TB
network 250MB/s/link x 3 (3D-HXB by GbE)
core
core
core
core
core
core
core
core
core
core
core
Designed by T2K Open Supercomputer
Alliance (U. Tokyo and Kyoto U)
memory
core
core
memory
core
core
memory
(2008～）
memory
T2K-tsukuba
core
IO interface
IO interface
Network (DDR Infiniband x 4)
Spec；
• 648 nodes
(quad Opteron, 4sockets/node)
• 10000 cores
• Peak performance 95.4TF
• total memory 20TB
• total disk capacity 800TB
( 20th in top 500, June, 2008)
Full bi-sectional FAT-tree Network
n
L3 SWs
n
: #Node with 4 Links
: #24ports IB Switch
L2 SWs
L1 SWs
Nodes
Detail View for one network unit
#
Item
1
2
1
3
2
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
2
3
4
5
6
7
8
9 10 11 12
13 14 15 16 17 18 19 20 21 22 23 24
696
144
Level 2 switch
240
Level 1 switch
232
Total switch
616
12
7
1
Node
Level 3 switch
12
※ノード総数696台には
オンラインのスペア
ノード4台を含みます。
25 26 27 28 29 30 31 32 33 34 35 36
x 20 network units
CCS Research divisions and Expertise




More than 30 faculties (31) and PostDocs, students
Research organization of computational sciences, not supercomputer
“service” center
5 Research Division and 11 Groups
Computational Science

Division of Particle Physics and
Astrophysics






Computer Science

Particle physics – Lattice QCD
Astrophysics – Formation of early cosmic
objects and galaxies
Division of Material and Life Sciences


Material science – nano-science, DFT
Life sciences – Molecular dynamics, CPMD
Quantum Many-Body Systems
Division of High Performance
Computing Systems



Division of Computational
Informatics

Division of Global Environment and
Biological Sciences


Global environment – Meteological science,
global climate simulation
Biological science – molecular phylogenetic
analyses
System architecture
Grid computing

Computational Intelligence Data Mining & Knowledge
Discovery, Large scale
database
Computational Media Visualization, Computer
graphics
8
Organization of Education and Research
for Computational Science in University of Tsukuba



CCS integrates the development of supercomputing facilities and
several researches of computational science.
CCS carries out researches in major computational science fields.
The faculty members of CCS “belong” to a Graduate School for
education, and “work” for the Center for research.
Founded in 1996, re-organized in 2004
Center for
Computational Sciences
Division of Particle Physics
and Astrophysics
Division of Materials and
Life Sciences
Division of Global Environmental
and Biological Sciences
Division of High Performance
Computing Systems
Faculty
Professor
Associate Prof.
Assistant Prof.
31
11
12
8
Graduate School of Pure and
Applied Sciences
• Dept. of Physics
• Dept. of Frontier Science
• Dept. of Materials Science
Graduate School of Life and
Environmental Sciences
• Dept. of Geoenvironmental Sciences
• Dept. of Structural Biosciences
Graduate School of Systems and
Information Engineering
Division of Computational
Informatics
• Dept. of Computer Science
• Dept. of Intelligent Interaction Technologies
9
Position and Role of CCS

From the nation-wide viewpoint of development of scientific research

An unique research “inter-university” organization which jointly carry out
“interdisciplinary” researches on computational sciences, even including
development of the leading-edge high performance computer systems




Computational Science in each discipline is done by university lab and institutes specific to its discipline.
National Institute of Informatics (NII) focuses on Information Science, neither computational science nor
development of high performance computer system.
Supercomputer Centers in major university focus on just operation and service on computing facilities, and
not researches on computational science.
From the viewpoint of development of scientific research within our university



The faculty of CCS belongs to graduate schools and contribute to the education of
computational science
Our university supports CCS as one of the most important research organizations with
respect of budgets, faculty position and facilities.
“Academic Computing and Communications Center” (campus computer center)
manages campus information infrastructures such computer systems and network
systems in the university.
10
PACS-CS System (2560 nodes)
・・・
Computation node (CPU)
Route through a single
dimension of switch
・・・
・・・
・・・
Y=16
・・・
・・・
・・・
・・・
・・・
・・・
Z=10
X-dim switch
Y-dim switch
Z-dim switch
Route through multiple
dimensions of switch
(intermediate CPU is
required)
・・・
・・・
in the figure
real connection
(dual link trunking)
X=16
# of nodes
2560 (16 x 16 x 10)
peak performance
14.3 Tflops
node configuration
single CPU / node
CPU
Intel LV Xeon EM64T, 2.8GHz, 1MB L2 cache
memory
2GB/node (5.12 TB/system), DDR2 interleaved
network for parallel
processing
3-dimensional Hyper-Crossbar Network
link bandwidth
one-sided: 250MB/s/dim.
one-sided: 750MB/s (3-D simultaneous trans.)
local HDD
160 GB/node（RAID-1）
total system size
59 rack
power consumption
550 kW
11
Scientific results by using PACS-CS
Electric Field [eV/nm]
100

照射レーザー電場
物質内電場
50
0
-50
Simulation for electron
dynamics which originates
from nonlinear interaction
between the laser and
matter in a femto-second.
Energy [eV/atom]
-100
5
4
3
2
0

laser
レーザー光から電子への
エネルギー移行
1
0
results for the hadron spectrum by the
“complete QCD” (2+1 flavor QCD),
10
20
30
40
matter
Time [fs]
-> QCD at physical point

Simulation by WRF (local regional model)
“hottest day in Aug 16th 2007
-> Real-time Local Weather Forecast

DFT (Density Function Theory) computation for large Si
cluster (4000 atom) -> DFT computation for 10,000 atom
12
T2K Tsukuba System
648 nodes
Peak performance : 95.4TF
(95.4 trillion 1012 operations/sec)
Memory 20TB
Disk 800TB
Linpack Result:
Rpeak = 92.0TF (625 nodes)
Rmax = 76.5TF (83.15 %)
20th position in 2008 Jun Top500
(T2K: Tsukuba, Tokyo and Kyoto Open Supercomputer Alliance)
Full bi-sectional FAT-tree Network
n
L3 SWs
n
: #Node with 4 Links
: #24ports IB Switch
L2 SWs
core
core
core
core
core
core
core
memory
memory
core
L1 SWs
Nodes
Detail View for one network unit
#
Item
1
core
core
core
core
core
core
core
1
IO interface
3
4
5
6
7
8
9
10
11
2
3
4
5
6
7
8
9
10
11
696
Level 3 switch
144
Level 2 switch
240
Level 1 switch
232
Total switch
616
12
※ノード総数696台には
オンラインのスペア
ノード4台を含みます。
IO interface
1
Network (DDR Infiniband x 4)
Node
12
memory
memory
core
2
2
3
4
5
6
7
8
9 10 11 12
13 14 15 16 17 18 19 20 21 22 23 24
25 26 27 28 29 30 31 32 33 34 35 36
13
x 20 network units
T2K Open Supercomputer Alliance


Primary aiming at design of common 
specification of new supercomputers.
Now extending to collaborative work 
on research, education, grid operation,
..., for inter-disciplinary computational 
(& computer) science.
Kyoto Univ.
416 nodes (61.2TF) / 13TB
Linpack Result:
Rpeak = 61.2TF (416 nodes)
Rmax = 50.5TF (82.51 %)
Open hardware architecture with
commodity devices & technologies.
Open software stack with opensource middleware & tools.
Open to user’s needs not only in
FP & HPC field but also IT field.
Univ. Tokyo
Univ. Tsukuba
952 nodes (140.1TF) / 31TB
648 nodes (95.4TF) / 20TB
Linpack Result:
Linpack Result:
Rpeak= 113.1TF (512+256 nodes) Rpeak= 92.0TF (625 nodes)
Rmax = 83.0TF (73.38 %)
Rmax = 76.5TF (83.15 %)
14
Scientific results from T2K
global climate simulation code NICAM
Collision of the Andromeda Galaxy）
observation
"tears" of Andromeda Galaxy
2000M years ago
1000M years ago
present day
15
CCS Inter-University Activity:
Interdisciplinary Computational Science Promotion Programs

Interdisciplinary collaboration program


Support to establish a network of interdisciplinary
research projects and to encourage grand challenges
of interdisciplinary computational science.
To promote interdisciplinary research projects of
different disciplines.
Between Computer science and application fields （e.g.
Applied Math and Particle Physics）
 Between different application fields（e.g. Modeling of
radiation in Astrophysics and Climate simulation）


Field Ａ
Group Ｘ
Proposal
request
coordinate
Field Ｃ・ Field Ｄ
Alliance ＺＷ
proposal
Proposal
request
support
Interdisciplinary collaboration
program support committee
（ review of proposal and coordinate collaboration）
Establish a network to promote
interdisciplinary research projects
Matchmaking of research groups from different fields
Review of Proposals by external reviewer of
Interdisciplinary collaboration coordination committee
 Coordinate research groups from different fields
 Find partners to mach the request
 Advice with expertise of different fields
Field Ｂ
Group Ｙ



Follow up the research activity produced by the
interdisciplinary collaboration
Field Ｃ
Group Ｚ
Field Ａ
Group Ｘ
CCS
Field Ｂ
Group Ｙ
Field Ｄ
Group Ｗ
Large-scale scientific simulation program



Push forward the grand challenge of several fields in computational sciences by providing the
computational power of the PACS-CS.
Review proposals and concentrate our computational power to make new scientific discoveries
Follow up the scientific results
16
Educational Activities in CCS

HPC Seminar



This seminar presents knowledge, methods
and techniques for programming modern
high performance computer systems,
including recent microprocessors, and its
performance turning, parallel programming.
Participants: researchers and users of
computational science (including
researchers in companies)
Periods: 2 or 3 days in summer season






Held since July 2007
Also broadcasted via internet

Campus-wide courses on “computational
sciences” for graduate students



Faculty members of CCS give lectures
Accredited as “unit” in graduate courses.
Courses



Computational Science Dual Degree
(double major) Program
Computational Science Literacy
High Performance Parallel Computing
Technology for Computational Sciences
(overlapped with HPC Seminar)
Started from 2008


Enables a graduate student in a doctoral
program to simultaneously belong to a
masters program of a different Graduate
School, and receive both a doctoral degree
in science and a masters degree in
computer science, or vice versa, upon
graduation.
Design of curriculum and courses for
advanced computational science
Educate researchers who can push forward
new Interdisciplinary computational science
from global viewpoints
Started in 2009 (Physics in Doctor course
and Computer sciences in Master course)
Education in each graduate school

CCS allocates space for some of
students, and create environment to
enable interactions to other fields.
17
Contributions for National Petascale computing
As a leading research organization, it is very important issue for us to
take a leadership concerning to building/operation of the petascale
system in national-wide computing infrastructure.


“Element technology for next-generation supercomputer” Project (2003-2005)

Collaboration with Hitachi Ltd. on low-power on-chip memory architecture
for next-generation systems
Collaboration with RIKEN on the petaflops system development



Formal agreement between U. Tsukuba and RIKEN signed in September
2006
Participation of several CCS faculty in the system design as a concurrent
researchers
Agreement of actual collaboration in tuning of representative petascale
applications (QCD, RS-DFT and FFT)
18
CCS’s Vision:
COE of “Interdisciplinary Computational Science”
Computational Science is a cutting-edge and indispensable multidisciplinary
area for the development of science and technology of the 21st century.



Computational Science as the key to meet the grand challenges in nature and human, environments.
Integration of science and computer/information science to realize the computational science.
Education System to produce the next generation of scientists who carry out research from global perspectives.
Existing “computational science”
• Computational X as a part
of X science
• Short-term individual
projects
• Make use of
“supercomputer” center
New Interdisciplinary Computational Science
• Integration and collaboration of computational science and
computer science
• High performance computing system will be getting bigger and
more complex in configuration. HPC technologies are
indispensable to make the best use of it
• Continuous development and investment are required to enable
the next generation of computational science.
• The common base of computational science brings mutual
intimate connections among various scientific science
disciplines and allows us to approach science as a whole
from a global perspective.
We aim to be an world-wide COE to develop “Interdisciplinary Computational
Science” as a new area to 21st century advances in science and technology,
and support it continuously from global and long-term perspectives.
19
Updates of CCS in 2010


Re-organization for the period 2010-2016
CCS was recognized as a national core-center for the collaborative research
on the Interdisciplinary Computational Science from 2010


Advanced Interdisciplinary Computational Science Collaboration Initiative（AISCI）
CCS was selected as a core organization of NGS (Japanese petaflops
computer) for "Field 5: The origin of matters and the universe"
• Division of Particle Physics and
Astrophysics
• Division of Material and Life
Sciences
• Division of Global Environment
and Biological Sciences
• Division of High Performance
Computing Systems
• Division of Computational
Informatics
• Division of Particle Physics
• Division of Astrophysics and
Nuclear Physics
• Division of Material
• Division Life Sciences
• Division of Global Environment
• Division of High Performance
Computing Systems
• Division of Computational
Informatics
20
Updates of CCS in 2010

Upcoming international events



ACM ICS (international Conference on Supercomputing) in
Tsukuba, June 2010
IWOMP 2010 (International Workshop on OpenMP) in
Tsukuba, June 2010
ICCS 2011 (International Conference on Computational
Science) in Tsukuba, June 2011
21
International collaborations





International Lattice Data Grid in particle physics (Edinburgh University (UK),
DESY(Germany), Univ. di Roma(Italy), FNAL/BNL/JLAB(USA))
Astronomical Radiation Hydrodynamics Research (National Astronomical
Observatory(Japan), SISSA(Italy))
Nano-Bio Science Collaboration (ETH(Switzerland), Princeton
University/Vanderbilt University(USA), Advanced Institute for Science and
Technology(AIST), National Institute for Materials Science (NIMS)),
Quantum Structural Biological Science Collaboration (AIST, Hyogo
Prefectural University(Japan), ETH/IBM Zurich(Switzerland)
Quantum Dynamics Collaboration (RIKEN(Japan), Washington University
(USA))

Warming of Arctic Research (IARC of Alaska University/Hawai University(USA),
Pusan University(Korea), Academia Sinica(China))

France-Japan Grid Computing (National Institute of Informatics(Japan),
INRIA(France), Pennsylvania State University(USA))

Information integration/Knowledge discovery Research (Carnegie-Mellon
University/Georgia Institute of Technology(USA), Hong-Kong Chubun University).
22
Organization for Inter-university Facilities Operation
Researchers in University and Institute
Proposal
Acceptance
Inter-university activity
committee
Director
Operation of inter-university facilities
Execute Interdisciplinary Computational
Science Promotion Programs
Computing facility
administration committee
Inter-university research
review committee
Allocate
resources
Review of proposal (including external reviewer)
Interdisciplinary collaboration
program support committee
Inter-university activity
support team
Support
staff
from company
Steering committee
Steering council



In order to execute “Interdisciplinary Computational Science Promotion
Programs”, we organize Inter-university research review committee, including
external members, to evaluate the proposal.
We also organize “Interdisciplinary collaboration program support committee” to
review the proposals and coordinate collaborations between different fields.
We report the policy, results and status of inter-university activities in Steering
council held annually.
23