OptIPuter FY2004 Revised Program Plan
NSF ITR Cooperative Agreement ANI-0225642
Larry Smarr, Principal Investigator
Cal-(IT)2, University of California, San Diego
smarr@cs.ucsd.edu
For the period October 1, 2003 – September 30, 2004
DRAFT COMPLETED January 30, 2004
SUBMITTED Monday, May 17, 2004
1.
Preface
On December 22, 2003, we submitted our Response to the OptIPuter Year 1 Project Review to NSF. Reviewers
noted (item 1b) that: By reducing scope and reallocating resources, or minimally making sure that a few key
research problems are addressed, the project will greatly enhance its opportunity for success and impact.
In our response, we stated the following:
We agree with the reviewers that defining 3-4 research topics each year, in order to provide quantitative metrics for
evaluating the success and impact of the project, is a good approach. We are now working with the team members
to define these research focus areas for year 2, proposed below:




How to we control lambdas and how do protocols influence their utility?
How is a LambdaGrid different from a Grid in terms of middleware?
How can lambdas enhance collaboration?
How are applications quantitatively helped by LambdaGrids?
These four areas represent goals to help focus OptIPuter team research efforts in year 2 (and beyond). Hence, our
teams must review their deliverables in the Cooperative Agreement for year 2 and in the Annual Progress Report
and Program Plan (PPP) (pp. 33-38) and then hone their activities to make sure these topics are addressed. We do
not advocate changing the deliverables, but tailoring them; i.e., making them more specific…This will happen by the
OptIPuter All Hands Meeting (AHM) in January 2004…and submitted shortly thereafter.
We held our AHM January 14-16, at which time all funded partners had a chance to work with OptIPuter Team
Leaders to address their Year 2 deliverables. Attached is an updated 2004 Program Plan, in which the deliverables
have been restated so that they emphasize the research topics defined above.
2.
OptIPuter FY2004 Program Plan
2.A. Year 2 Milestones
Our Year 2 Milestone, as described in the OptIPuter Cooperative Agreement:
All Southern California OptIPuter partners (UCSD, UCI, USC and SDSU) will have at least one fully
functioning OptIPuter node(s) connected to California’s CENIC experimental network. Chicago partners (UIC
and NU) will have OptIPuter nodes connected to I-WIRE. The nodes will all be based on UIC/NU’s “Mod 1”
work in Year 1.
2.B. Testbed Infrastructure (Networking, Clusters, Data Storage, Signaling Protocols) – Tom
DeFanti and Phil Papadopoulos, Team Leaders
Year 2 tasks: UCSD had hoped to install a DWDM core system on campus and interconnecting the Southern
California OptIPuter sites; however, this activity may overlap with year 3, given the limited capital money in the
budget. USC, UCI and SDSU, in addition to UCSD, will install OptIPuter nodes and link them to the CENIC
dark fiber experimental network. UIC, however, will deploy OptIPuter nodes connected via I-WIRE DWDM and
other dedicated lambdas; the nodes will all be based on UIC’s “Mod 1” work in Year 1.
UIC, with input from the application teams, will begin building the OptIPuter “Mod 2” using 64-bit processors
The OptIPuter FY2003 Annual Progress Report and FY2004 Program Plan – UPDATED January 30, 2004
1
and upgraded storage, visualization and optical networking components. UIC will work with NU to test and
document the metro-area performance of clusterPC-to-clusterPC using up to 8 GE parallel connections through
the Glimmerglass and Calient O-O-O switches. Signaling protocols for switching lambdas will be designed,
tested, and refined. Similar tests will be done between UIC and UvA over the transatlantic links using VLANs and
groomed GigE circuits, also passed through the O-O-O switches. The APIs will be documented in papers and
manuals.
UIC will work with NU to develop optical network control technologies, incorporating network performance and
protocol parameters into new signaling protocols. The first-generation LambdaRAM will be designed and
implemented.
Southern California Testbed Infrastructure (Networking and Computing)
In Year 2, UCSD (Papadopoulos, Hidley) will:






Complete deployment of a campus-wide distributed testbed consisting of compute, storage, visualization,
and instrument endpoints. Management policies will be put in place so that experiments can schedule
physical hardware and use specialized (experiment-specific) software loaded on assigned nodes.
Provide account creation, cataloguing of resources, MRTG-based network monitoring and ganglia resource
monitoring on a web site; a preliminary version exists on the OptIPuter Intranet <http://web.optiputer.net>
The UCSD campus testbed will have a best-case bisection bandwidth of 5:1 and a worst-case of 8:1.
Monitoring will be put in place to measure when these links are saturated.
A donated IBM 48-node, 21 TB storage cluster will be deployed
Three 24-node dual Opteron clusters will be operational at UCSD campus nodes located at JSOE
(Engineering), SIO, and the School of Medicine. One of these clusters will be storage oriented
IPv6 capability will be made a standard part of the base OptIPuter software stack by the end of Year 2
Storage and compute cluster donations to UCSD, combined with a modest monetary investment from project funds,
have enabled ~150 OptIPuter nodes on campus. The OptIPuter project does not have the financial flexibility to
match the network to this large number of endpoints. To this end, UCSD will judiciously expand the network in the
most useful places


The storage endpoint will be connected to the Chiaro router over a 10 GigE link connected to a single 48port GigE aggregration switch, to be purchased by the project. Bisection will be 5:1.
The Chiaro router will have a 1GigE connection to the UCSD border router. A production virtual router
will be instantiated on the Chiaro to protect the campus network.
Effort will be made to create a Southern California OptIPuter testbed (UCSD, SDSU, UCI, and USC/ISI), connected
via CENIC CalREN HPR 1Gbps circuits. At UCSD, connection from the campus border router will be made to the
existing Chiaro OptIPuter router. Completion of this deliverable is dependent on CENIC’s internal deliverables.
Chicagoland Testbed Infrastructure (Networking and Computing)
UIC (DeFanti) will install a Force10 electronic switch and a 128x128 optical switch at the StarLight facility in
Chicago to connect 1GigE and 10GigE metro, regional, national and international research networks and lambdas.
UIC will also install a 64x64 optical switch at EVL which connects to StarLight over 8 GE circuits.
UIC (DeFanti, Leigh, Renambot) will evaluate AMD 64-bit clusters. Built and installed will be:





Mod 1 (32-bit) and Mod2 (64-bit) OptIPuter clusters as application production nodes
A production Terabyte raid at StarLight for Scripps caching,
“Reflector” PCs for achieving facile testbed subsets
A cluster for LambdaRAM
An Itanium PC with 10GigE NIC (provided by UvA)
In addition, UIC will monitor and benchmark various SOTANs (State-Of-The-Art Nodes) for proof of concept,
notably Itanium, Opteron and IBM G5. SOTANs are, by definition, the newest available technology – from
networking , architecture, and middleware points of view – but may not be usable by applications for several years.
The OptIPuter FY2003 Annual Progress Report and FY2004 Program Plan – UPDATED January 30, 2004
2
Prototype/Evaluate High-Speed NIC Hardware and Driver/Software Stack
UCSD (Papadopoulos) will conduct an Infiniband test, connecting 4 interface cards (at 10Gbps each) to the IBM
Storage cluster and a Topspin switch. The Topspin switch will have 4 channel-bonded GigE interfaces for direct
Infiniband-to-GigE connections into the UCSD campus’ OptIPuter fabric. We will conduct experiments to measure
how protocol offloading of IP over Infiniband can improve performance. It will also be used to explore how highperformance cluster-area connections can be exploited to provide storage services.
Optical Signaling Protocols
UIC (Yu), NU (Mambretti), UCSD (Chien) and UvA (de Laat) will refine their optical backplane architecture. They
will evaluate techniques for optical control and switching. AAA will be developed by UvA. Signaling protocols are
being developed by UIC and NU, working with UvA and others.
Photonic Multicasting
UIC (Leigh, Renambot) will explore the extension of photonic multicast capabilities across the wide area.
LambdaRAM
LambdaRAM takes advantage of multiple-gigabit networks to pre-fetch information before an application is likely
to need it (similar to how RAM caches work in computers today). This technique collects memory in a compute
cluster and then allocates it as a cache to minimize the effects of latency over long-distance, high-speed networks.
LambdaRAM (UDP blasting) has been tested locally. In Year 2, UIC will design a second LambdaRAM prototype.
2.C. System Software and Middleware – Andrew Chien, Team Leader
Year 2 tasks: USC will begin designing system software and middleware between the cluster communication
fabric and the wide-area optical network. Design of a transparent (MAN/WAN) lambda  OS-bypass (local
cluster communication) messaging transport framework will be undertaken. UCSD will collaborate with UCI to
begin exploring security models using packet-level redundancy and multi-pathing. Elements of prototype
OptIPuter system software will begin to be released to partner sites for application development and evaluation.
UCI will begin design of the Time-Triggered Message-Triggered Object (TMO) programming framework for the
OptIPuter. TAMU will include network protocols and middleware in its performance analysis work.
System Software Architecture
UCSD (Chien) will prototype Distributed Virtual Computers (DVCs) for the OptIPuter, including the development
of DVC Interface/Services. These service interfaces constitute the primary presentation of OptIPuter capabilities to
applications. DVCs also require resource descriptions, which are used to allocate resources using Globus resource
management mechanisms as well as novel new tools that allocate light paths and directly control optical networks.
As stated in Section 2.B (Optical Signaling Protocols), UIC, NU, UCSD and UvA are working together to define
and implement a shared API to allow applications to access these capabilities, and to perform the necessary network
configurations to make these control capabilities useful. This capability, while far from a complete solution, will
enable basic experimentation with on-demand optical network configuration.
Real-Time Operating System
UCI (Kim) will work with UCSD to integrate the TMO middleware subsystem into OptIPuter middleware (DVC),
enabling demonstrations of current LAN capabilities on optical network WANs. A TMO-based application could
then run across distributed OptIPuter resources, with DVC support for allocation, setup, etc. Based on
experimentation in Year 3, we will design a real-time programming framework for the OptIPuter.
Data Storage
UCSD (Chien) will complete OptIPuter storage benchmarks that characterize the requirements for BIRN and
EarthScope applications; specifically, he will document baseline configurations (what they do in the demonstrations
today) and scaled, distributed versions (what we anticipate they will do when the distributed optical network
infrastructures and the applications are mature). Chien will develop analytical and simulation models for novel
approaches to managing distribution and replication of data using erasure codes, and use these models to explore the
design space. These explorations will help characterize and quantify the benefits that can be achieved, as in
The OptIPuter FY2003 Annual Progress Report and FY2004 Program Plan – UPDATED January 30, 2004
3
performance, robustness, scalability, and will provide approaches for how to achieve them.
Security
UCI (Goodrich) will work with UCSD (Chien, Karin) to define distributed virtual computer security models that
simplify application access and cooperation across distributed sites. These models, in contrast to traditional Internet
and client-server security protocols (e.g., SSL), frame a new set of security problems. We anticipate framing and
initial characterization of these problems in this year, with deeper protocol work in subsequent years.
Application Performance Modeling
TAMU (Taylor) is working with the Data, Visualization and Collaboration Team (Leigh) and the Applications Team
in order to characterize their computing, communications, and storage access behaviors. We will pursue a particular
focus on how these applications are expected to behave in optical network environments. The TAMU researchers
will also help diagnose current performance bottlenecks and problems in the OptIPuter’s visualization software.
High-Performance Transport Protocols
UCSD (Chien) will simulate a Group Transport Protocol (GTP) that efficiently manages receiver contention
(efficient bandwidth handoff) likely to arise in optical switched networks. He will also pursue evaluation via
simulation, emulation, and ultimately implementation during Year 2. The GTP implementation will be integrated
with the concept of DVC for presentation to applications.
UIC (Leigh) will extend his work on RBUDP to Streaming RBUDP, which provides improved jitter control for
gigabit streaming applications (e.g, graphics and remote visualization) over dedicated light paths. Streaming
RBUDP employs flow-rate estimation and early loss recovery to reduce jitter in high-bandwidth environments. UIC
(Leigh) and UCSD (Chien) will work towards integrating LambdaRAM and photonic multicasting in DVCs. UIC
(Grossman) will continue to improve SABUL/UDT, moving to 10GigE interfaces and networks; this effort is
expected to produce an IETF draft on transport protocols as part of the OptIPuter’s technology transfer.
USC (Bannister) will document XCP’s design, develop a protocol specification, and publish XCP header format and
other protocol parameters as Internet Drafts. XCP performance will be benchmarked using BSD end-systems and
PC routers. XCP will be compared to TCP to determine how fast XCP will go on a PC router. The effects of XCP on
TCP and other background traffic will also be considered. USC will use a prototype implementation to explore
research issues, such as: How does XCP behave when there are L2 queues? How does XCP behave over variable
bandwidth links? USC will continue exploration of router implementation constraints in protocol design; e.g.,
header format and number representation.
The XCP PC-router design will be made available to OptIPuter collaborators. USC will help OptIPuter applications
migrate to XCP by creating instructions on how to modify code and (possibly) writing wrapper/shim software to
adapt TCP binaries to use XCP. USC will collaborate with router vendors on XCP router implementations.
2.D. Data, Visualization and Collaboration – Jason Leigh, Team Leader
Year 2 tasks: UIC, UCSD and SDSU will deploy data management, visualization and collaboration software
systems developed in Year 1 to OptIPuter application teams, both for testing and for feedback. UIC will continue
to enhance computing and visualization cluster hardware/software to achieve very-large-field real-time volume
rendering. UIC, UCSD and UCI will integrate the data middleware and high-level data services into an
OptIPuter application and begin using the data middleware to develop data-mining algorithms for clustering data
streaming over the OptIPuter. In addition, high-level software abstractions for remote data retrieval and for
accessing bundled computing resources as high-level computational units or peripherals will be studied.
Data
UCSD will deploy a large storage-intensive cluster funded by an IBM SURS grant to enable new OptIPuter
Storewidth research. See Section 2.B (Southern California Testbed Infrastructure).
UIC (Grossman) will achieve data transport rates of 5Gbps, improve the I/O interfaces of his DSTP servers, and
begin development of wide-area storage and database services over lambdas using SIO and/or NCMIR datasets.
UCI (Smyth) and SDSU (Frost) will continue to develop new statistical modeling and clustering algorithms capable
of handling high-bandwidth, spatio-temporal scientific data streams, which allow interactive visualization and
The OptIPuter FY2003 Annual Progress Report and FY2004 Program Plan – UPDATED January 30, 2004
4
detection of clusters, outliers, and other structural properties of streaming datasets. UCI/SDSU will begin to examine
relevant geoscience data – notably MODIS satellite data – as a potential data source for this data mining research.
Photonic Multicasting
UIC/EVL will test Glimmerglass Networks’ photonic multicast option as a means of supporting collaboration over
lambdas.
Visualization/Collaboration Tools (General)
Further evaluation by UIC/EVL and UCSD/SIO of the effectiveness of seismoglyphs is needed.
UIC (Leigh) and USC (Thiébaux) will continue development of visualization techniques for data streaming, from
servers to tiled displays driven by an OptIPuter node. UCSD/SIO (Nayak) will tailor SIO’s visualization cluster
(purchased late in Year 1 for SIO) to enable Fledermaus to run over the cluster using Chromium.
UCSD/CRCA (Brown) is upgrading his multi-user, interactive graphic environment authoring toolkit (SceneEditor)
with OptIPuter-based applications in mind. Enhancements will include flexible scalability to heterogeneous display
configurations, as well as balancing storage, computer and display for very large, interactive datasets.
SDSU will help university and corporate partners connect visualization centers over high-speed networks and will
help data fusion and applications, using both wireless and optical communication, especially with IPv6 components.
Volume Visualization Tools
UIC and USC/ISI will integrate performance monitoring capability into their existing visualization tools to begin to
understand where current bottlenecks lie. This will inform the design of a new adaptive visualization architecture to
support both high-resolution digital montages and volume visualization.
NCSA (Cox), a new member of the OptIPuter team, will start working with SIO oceanographers to understand their
visualization requirements, with the goal of determining how to translate visual idioms from atmospheric science
and similar domains into oceanographic/earth science visualizations.
2.E. Applications and Education Activities – Mark Ellisman, Team Leader
Year 2 tasks: UCSD and SDSU will conduct full-scale visualization and collaboration trials using sample
geoscience and bioscience datasets over updated OptIPuter testbeds in order to identify key system bottlenecks
and usability problems with the software developed in Year 1. Education efforts will continue with the Preuss
School and the Lincoln Elementary School in Chicago, testing first-draft modules in the experimental section of
its core curriculum, involving grade-school children in observational science activities. The Sixth College will
select a group of students, from its distinctive Honors Students, to join OptIPuter application teams and work on
the OptIPuter node located within the College. In addition, an annual OptIPuter meeting/workshop will be held,
with some days dedicated to OptIPuter partners and some days open to the wider e-Science and highperformance computing and communications communities.
Application Codes Running on OptIPuter Clusters
SDSU will continue to work with UCSD/SDSC (Mike Bailey) and others at UCSD/SIO to develop content and
applications across the optical connectivity between the two sites as well as further develop connections to other
universities both in US and internationally. This will involve large datasets from both satellite imagery and seismic
cubes. Seismic data cubes, such as the Beta field in the offshore Long Beach area, which SDSU provided to Mike
Bailey, will be acquired where possible and satellite imagery over the top of the fields will be incorporated into the
visualizations. SDSU will also work to develop a high-resolution image of the San Diego county region for use by
first responders during crisis management situations.
All UCSD application teams will continue to provide sample datasets to technologists and will conduct a series of
collaboration trials with OpIPuter infrastructure and visualization groups to further elicit user requirements and test
emerging technologies in practicum.
UCSD/NCMIR will conduct a series of targeted collaboration trials with:


UCI to explore interactive volume rendering and annotation system
Osaka University to explore HDTV over IPv6 featuring future decoder/compression technologies
The OptIPuter FY2003 Annual Progress Report and FY2004 Program Plan – UPDATED January 30, 2004
5



NetherLight and UK eScience in Edinburgh to explore optical projection tomography and the mouse
embryo database as integrated large data sources
EVL on newly constructed NCMIR Geowall, and
ISI to explore distributed rendering application.
UCSD/Preuss School
UCSD/SDSC (Steckler) will harden the visualization facility; i.e., it will go from a pilot facility to one easily used
by all teachers in multiple classrooms. Emphasis will be on using the OptIPuter network to bring data to Preuss for
analysis and visualization by the classes. Steckler will start exploring the use of the OptIPuter resources for other
areas outside of earth science.
UCSD/SIO (Kilb) will continue to provide Preuss School with educational material.
UCSD/Sixth College
The Sixth College will select a group of students, from its distinctive Honors Students, to join OptIPuter application
teams and work on the OptIPuter node located within the College.
UCSD/SIO (Kilb) will work with UCSD (Steckler, Wienhausen) to bring the education programs similar to what
was done in year 1 at the Preuss School to the UCSD Sixth College.
Lincoln Elementary School in Oak Park, IL
During Year 2, UIC (Moher) will (a) finalize, deploy and assess the effectiveness of the RoomQuake system that
was developed during Year 1 in collaboration with UCSD/SIO (Kilb), and (b) develop of a new application,
RoomBrain. These applications are part of a series of RoomWare systems designed to map scientific phenomena
into/onto the physical space of the classroom. Moher’s working hypothesis is that the classroom holds a special
place in the minds of students and teachers – apart from their bedroom, it is likely the place that a child spends most
of his or her time. By using the physical space of the classroom as a “stage” populated with props and a “thin layer
of technology”, he hope to provide settings in which elementary students can “play-act” their roles as scientists in
“participatory simulations.” In these settings, thin clients (e.g., PDAs, tablet displays) and large shared displays
serve as “activity points” for spontaneously aggregated groups of students.
UCSD/SIO Outreach Activities
SIO hopes to hold another teacher workshop at the SIO Visualization Center. SIO will work in tandem with the
Regional Workbench Consortium (RWBC), as needed. SIO will continue development of 3D visualizations for use
by members of the GeoWall Consortium in educational venues.
Annual OptIPuter meeting
Our second year annual OptIPuter meeting will take place January 14-16. All partners and interested parties will be
invited.
The OptIPuter FY2003 Annual Progress Report and FY2004 Program Plan – UPDATED January 30, 2004
6