Fall 2002 Internet2 Member Meeting
NSF Middleware Initiative:
Update and Overview of Release 2
18 March 2016
Alan Blatecky, National Science Foundation
John McGee, USC/ISI
Ken Klingenstein, Internet2
& University of Colorado-Boulder
Mary Fran Yafchak,
Southeastern Universities Research Association
Ann West,
EDUCAUSE & Internet2
Session Topics
NSF Middleware Initiative Overview
GRIDS Center
NMI-EDIT
NMI Integration Testbed
NMI Outreach and Participation
2
NSF Middleware Initiative
Purpose
To design, develop, deploy and support a set
of reusable, expandable set of middleware
functions and services that benefit
applications in a networked environment
3
NMI Organization
Core NMI Team
• GRIDS Center
– ISI, NCSA, U Chicago, UCSD & U Wisconsin
• EDIT Team (Enterprise and Desktop Integration Technologies)
– EDUCAUSE, Internet2 & SURA
Grants for R & D
• Year 1 -- 9 grants
• Year 2 -- 9 grants
4
A Vision for Middleware
To allow scientists and engineers the ability to transparently use and
share distributed resources, such as computers, data, and
instruments
To develop effective collaboration and communications tools such as
Grid technologies, desktop video, and other advanced services
to expedite research and education, and
To develop a working architecture and approach which can be extended
to Internet users around the world.
Middleware is the stuff that makes “transparently use” happen, providing
persistency, consistency, security, privacy and capability
5
NMI Goals
a) facilitate scientific productivity,
b) increase research collaboration through shared data,
computing, code, facilities and applications,
c) support the education enterprise,
d) encourage the participation of industry, government labs and
agencies for more extensive development and wider adoption
and deployment,
e) establish a level of persistence and availability so that other
applications developers and disciplines can take advantage of
the middleware,
f) encourage and support the development of standards and open
source approaches and,
g) enable scaling and sustainability to support the larger research
and education communities.
6
NMI Process
Experimental Software & research applications
Early Implementations
- GRID services, directories, authentication, etc
Research &
Education
Early Adopters
Consensus
- disciplines
- communities
- industries
MiddlewareTestbeds
- experimental, Beta, scaling & “hardening”
Dissemination & Support
Middleware deployment
7
First Deliverables: NMI Release 1
Software
•
(Globus, Condor, Network Weather Service,
KX.509, CPM, Pubcookie)
Object Classes
• (eduPerson, eduOrg, commObject)
White Papers (Shibboleth, video directories, etc)
Best Practices (Directories, LDAP)
Policies (campus certificates, account management)
Services (certificate profile registry)
www.nsf-middleware.org
8
NMI Release 2
Release 2 shipped on Oct-25-2002
New versions
• Globus Toolkit, Condor-G, Network Weather Service, Pubcookie, etc
New components and best practices
• OpenSAML 1.0, Shibboleth 1.0, etc
• GSI-OpenSSH, Gridconfig Tools
• LDAP Analyzer, Metadirectory Practices for Enterprise Directories, etc
Two releases each year: April/October
Release being adopted by projects, agencies
International interest in releases
9
3rd Year Program of NMI
Program Announcement in process
March 3, 2003 Proposal deadline
$7M available for FY’03
http://www.nsf-middleware.org
10
GRIDS Center Overview
John McGee
USC, Information Sciences Institute
mcgee@isi.edu
11
GRIDS Center, Part of the
NSF Middleware Initiative
One of two NMI teams, the GRIDS Center
(Grid Research, Integration, Development &
Support)
In late 2001, GRIDS created to:
• Define, develop, deploy, and support an integrated national
middleware infrastructure for 21st Century S&E
• Create robust, tested, packaged, & documented middleware for
S&E, including large NSF projects (e.g., NEES, GriPhyN, TeraGrid)
• Work with middleware research community to evolve architecture &
integrate other components
• Provide dedicated operations capability for 24x7 support and
monitoring of Grid infrastructure
12
GRIDS Center Participants
The Information Sciences Institute (ISI), University of Southern
California (Carl Kesselman)
The University of Chicago (Ian Foster)
The National Center for Supercomputing Applications (NCSA) at
the University of Illinois at Urbana-Champaign (Randy Butler)
The San Diego Supercomputer Center (SDSC) at the University
of California at San Diego (Phil Papadoupolus)
The University of Wisconsin at Madison (Miron Livny)
13
Elements of Grid Computing
Resource sharing as a fundamental pursuit
• Computers, storage, sensors, networks
• Sharing is always conditional, based on issues of
security, trust, policy, negotiation, payment, etc.
Coordinated problem solving
• Beyond client-server: distributed data analysis,
computation, collaboration, etc.
Dynamic, multi-institutional “virtual
organizations”
• Community overlays on classic org structures
• Large or small, static or dynamic
14
Grid-Oriented
Projects in eScience
15
Grid Applications
Science portals
• Help scientists overcome steep learning curves of installing and
using new software
Distributed computing
• High-speed workstations and networks as aggregated
computational resources
Large-scale data analysis
Computer-in-the-loop instrumentation
• Grids permit quasi-real-time analysis of data from telescopes,
synchrotrons, and electron microscopes
Collaborative work
• Grids enable collaborative problem formulation, data analysis,
and discussion
16
Grid Portals
17
Mathematicians Solve NUG30
Looking for the solution to the
NUG30 quadratic assignment
problem
An informal collaboration of
mathematicians and computer
scientists
Condor-G delivered 3.46E8 CPU
seconds in 7 days (peak 1009
14,5,28,24,1,3,16,15,
processors) in U.S. and Italy (8
10,9,21,2,4,29,25,22,
sites)
13,26,17,30,6,20,19,
8,18,7,27,12,11,23
MetaNEOS: Argonne, Iowa, Northwestern, Wisconsin
18
Home Computers
Evaluate AIDS Drugs
Community =
• 1000s of home
computer users
• Philanthropic computing
vendor (Entropia)
• Research group
(Scripps)
Common goal =
advance AIDS
research
19
Sloan Digital Sky
Survey Analysis
Size distribution of
galaxy clusters?
Galaxy cluster
size distribution
100000
Chimera Virtual Data System
+ iVDGL Data Grid (many CPUs)
10000
1000
100
10
20
1
1
10
Number of Galaxies
100
iVDGL: International Virtual
Data Grid Laboratory
Tier0/1 facility
Tier2 facility
Tier3 facility
10 Gbps link
2.5 Gbps link
622 Mbps link
21
Other link
Network for Earthquake
Engineering Simulation
NEESgrid: US national
infrastructure to couple
earthquake engineers with
experimental facilities,
databases, computers,
and each other
On-demand access to
experiments, data
streams, computing,
archives, collaboration
NEESgrid is a partnership of
Argonne, Michigan,
NCSA, UIUC, USC
22
The 13.6 TF TeraGrid:
Computing at 40 Gb/s
26
Site Resources
HPSS
4
Site Resources
HPSS
24
8
External
Networks
Caltech
HPSS
5
Argonne
External
Networks
External
Networks
Site Resources
External
Networks
SDSC
4.1 TF
225 TB
NCSA/PACI
8 TF
240 TB
TeraGrid: NCSA, SDSC, Caltech, Argonne
Site Resources
UniTree
www.teragrid.org
23
Grids and Industry
Grid computing has much in common with
major industrial thrusts to decentralize (e.g.,
B2B, P2P, ASP, etc.)
Sharing issues are not adequately addressed
by existing technologies
Companies like IBM, Platform Computing and
Microsoft are now substantively involved
with the open-source Grid community (e.g.,
OGSA, which combines Web services and
Grid services)
24
GRIDS Software for NMI
GRIDS Center Software Suite in the first
release (NMI-R2) is a package of:
•
•
•
•
•
•
•
•
Globus Toolkit™
Condor-G
Network Weather Service
KX.509 & KCA
GSI-OpenSSH
Gridconfig Tools
Grid Packaging Tools
For RedHat 7.2/7.3 on IA32, Solaris 8 on 32-bit Sparc
25
Enterprise and Desktop
Integration Technologies (EDIT)
Consortium
Ken Klingenstein
Director, Internet2 Middleware Initiative
kjk@internet2.edu
26
NMI-EDIT Consortium
Enterprise and Desktop Integration
Technologies Consortium
• Internet2 – primary on grant and research
• EDUCAUSE – primary on outreach
• Southeastern Universities Research Association (SURA)
– primary on NMI Integration Testbed
Grant funding is ~$1.2 million a year:
• about ½ to short-term partial hiring of campus IT staff to
develop and document required standards, best
practices, etc.
• about ½ to testbeds, dissemination and training sessions
Almost all funding passed through to
campuses for work
27
NMI-EDIT: Goals
Much as at the network layer, create a
ubiquitous common, persistent and robust core
middleware infrastructure for the R&E
community
In support of inter-institutional and inter-realm
collaborations, provide tools and services (e.g.
registries, bridge PKI components, root
directories) as required
28
NMI-EDIT: Objectives
• Foster the development of campus enterprise
middleware to leverage both the academic and
administrative missions.
• Coordinate a common substrate across higher ed
middleware implementations that would permit interinstitutional efforts such as Grids, digital libraries, and
collaboratories to scale and leverage
• In some instances, build collaboration tools for
particularly important inter-institutional and government
interactions, such as web services, PKI and video.
• Insure that distinctive higher-ed requirements, from
privacy and academic freedom to multi-realm portals, are
served in the marketplace.
29
A Map of Middleware Land
30
NMI-EDIT:
Core Middleware Scope
Identity and Identifiers – namespaces, identifier
crosswalks, real world levels of assurance
Authentication – campus technologies and policies,
inter-realm interoperability via PKI, Kerberos
Directories – enterprise directory services
architectures and tools, standard object classes, interrealm and registry services
Authorization – permissions and access controls,
delegation, privacy management
Integration Activities – common management tools,
use of virtual, federated and hierarchical organizations
31
NMI-EDIT:
Organization
Overall technical direction set by MACE
• Middleware Architecture Committee for Education
(MACE)
• Bob Morgan, University of Washington, Chair
• Campus IT architects and representatives from Grids and
International Communities
Directions set via
• NSF and NMI management team
• Internet2 Network Planning and Policy Advisory Council
• PKI and Directory Technical Advisory Boards
• Internet2 members
32
Sample NMI-EDIT Process
(Directories )
MACE-DIR Working Group prioritizes needed materials
Subgroups established:
•
•
•
revision of basic documents (LDAP Recipe)
new best practices in groups and metadirectories
standards development for eduPerson 1.5 and eduOrg 1.0
Subgroups work in enhanced IETF approach: scenarios,
requirements, architectures, recommended standards stages
Working group deliverables announced; input and
conference call review/feedback processes start; work
groups reconvene as needed
Process takes around 4-6 months, depending on product
6-8 people drive the process with 15-50 schools
participating
33
NMI-EDIT: Participants
Higher Ed – 15-20 leadership institutions, with 50
more campuses represented as members of
working groups; readership around 2000
institutions
Corporate - (IBM/Metamerge, Microsoft, SUN,
Liberty Alliance, DST, MitreTek, Radvision,
Polycom, EBSCO, Elsevier, OCLC, Baltimore)
Government – NSF, NIST, NIH, Federal CIO
Council
International – Terena, JISC, REDIRIS, AARnet,
SWITCH
34
A Few Year-One
NMI-EDIT Milestones
Sept 1, 2001 – Grant awarded
Oct 2001– eduPerson 1.0 finalized; outreach begins with multiple workshops
Jan 2002 – HEBCA tested; first CAMP workshop held
Feb 2002 – PKI Lite CP/CPS; e-Gov and Management and Leadership Best
Practice Awards
April 2002 – Shibboleth alpha ships; NMI testbed selected; NIST/NIH PKI
workshop
May 2002 – NMI release, with eduPerson 1.5, pubcookie, KX.509, groups
and metadirectories, video white papers
June 2002 – affiliated directories begins; Base CAMP; testbed kickoff
July 2002 – Shibboleth alpha v 2 ships; Advanced CAMP
August 2002 – LDAP Analyzer testing begins; Shibboleth pilot-sites selected;
Work with content providers begins
September 2002 – Grant renewed; supplemental grant awarded for
outreach; Shibboleth beta ships
35
NMI-EDIT: Release 1
Deliverables
Software
KX.509 and KCA, Certificate Profile Maker,
Pubcookie
Object Classes
eduPerson 1.0, eduPerson 1.5, eduOrg 1.0,
commObject 1.0
Service
Certificate Profile Registry
36
NMI-EDIT: Release 1
Deliverables
Conventions and Practices
• Practices in Directory Groups 1.0, LDAP Recipe 2.0
• Metadirectory Practices for the Enterprise Directory in Higher
Education 1.0
White Papers
• Shibboleth Architecture v5
Policies
• Campus Certificate Policy for use at the Higher Education
Bridge Certificate Authority (HEBCA)
• Lightweight Campus Certificate Policy and Practice
Statement (PKI-Lite)
• Sample Campus Account Management Policy
37
NMI-EDIT: Release 1
Deliverables
Works in Progress
•Role of Directories in Video-on-Demand
•Resource Discovery for Videoconferencing
•Directory Services Architecture for Video and
Voice Conferencing over IP (commObject)
38
NMI-EDIT: Release 2
New/Revised Deliverables
Software
Programs and Libraries
–OpenSAML 1.0
–Shibboleth 1.0
–Pubcookie 3.0
Directory Schemas
–eduPerson
–eduOrg
39
NMI-EDIT: Release 2
New/Revised Deliverables
Conventions and Practices
•LDAP Recipe
•Metadirectory Practices for Enterprise Directories
•Practices in Directory Groups
Architectures
•Inter-domain Data Exchange (Draft)
Services
• LDAP Analyzer
40
The pieces fit together…
Campus infrastructure
• Name space, identifiers, directories
• Enterprise authentication and authorization
Inter-realm infrastructure
• edu object classes
• Exchange of attributes
Inter-realm Upperware
• Grids
• Digital libraries
• Video
41
NMI Integration Testbed
Mary Fran Yafchak
Testbed Manager,
Southeastern Universities Research Association
maryfran@sura.org
42
NMI Integration Testbed
Focus on the integration of released middleware
components with real life use and conditions
Elements: Sites, Manager, Workshop
Integration is the point - could think of it as…
• Where “EDIT” meets “GRIDS”
• Where enterprise needs meet research needs
• Where NMI components meet reality
43
NMI Integration Testbed
Planning and management by SURA
Participating Sites:
• University of Alabama at Birmingham
• University of Alabama in Huntsville
• University of Florida
• Florida State University
• Georgia State University
• Texas Advanced Computing Center (U Texas/Austin)
• University of Virginia
• University of Michigan
44
NMI Integration Testbed
NMI Participation
USERS
Implementers
Target Communities
NMI Integration
Testbed
CONTRIBUTORS
DEVELOPERS
SUPPORTERS
NMI Integration Testbed
Core
Testbed
Sites
UAB
UAH
UFL
FSU
GSU
UMich
TACC
UVA
future
expansion
?
45
NMI Integration Testbed Recent Activities
Testbed Kickoff June 10 - 12, 2002 at GSU
Site Integration Plans completed in July 2002
Testing of Release 1 completed
Press release & Web site announced 9/4/02
• See http://www.nsf-middleware.org/testbed
Open Testbed BoF here at I2 Members’ Meeting
• Wednesday, October 30, 11:45AM-1:15PM
46
NMI Integration Testbed Some Highlights from the Sites
• Twenty-six very real institutional projects and applications
“on board” for NMI integration - with more to come...
• Ten projects targeting increased access to their existing or
planning scientific grids (including emerging TeraGrid)
through NMI Globus
• Five sites actively implementing enterprise scale
directories, with centralized authentication and integrated
applications
• Active PKI efforts, from PKI Lite to PKI “heavy”
(maintaining HIPPA/FERPA compliance)
• New collaborative tools also represented, such as click-todial desktop video conferencing and shared calendaring
47
NMI Integration Testbed From R1 to R2
• Summarizing evaluation results from R1 - to be made
available on the Testbed Web site
• Working with Outreach to disseminate lessons learned
thus far
• R2-specific Component Testing Guidelines under
development
• Testbed Sites actively refreshing site plans and project
sets with respect to R2
• R2 evaluation soon to be underway...
48
NMI Integration Testbed Potential for Expansion
Already on our minds...
• Increase opportunities for both sponsored and
unsponsored participation
• Define a role and means of involvement for
international participants
• Define a role and means of involvement for
corporate participants
• Develop “hot topic” or application-specific testbeds
– E.g., Digital Libraries, Digital Video, Medical middleware,
Discipline-specific grids
49
NMI Outreach and
Participation
NMI Participation
and Outreach
Ann West
NMI-EDIT Outreach,
EDUCAUSE/Internet2/Michigan Tech
awest@educause.edu
50
NMI Outreach and Participation
Targeted Communities
Outreach Strategy
Results from PHASE I – Building
Awareness
Activities and Plans for PHASE II –
Delivering on Promise
51
NMI Outreach – Targeted
Communities
TERTIARY CIRCLE - General Interest
(Targeted NEW User Communities – Press/PR)
SECONDARY CIRCLE
Participating/User Communities
(NEES, GriPhyN, TeraGrid, Campuses, etc.)
PRIMARY CIRCLE
NMI Project Team
GRIDS
Center
NMIEDIT
Corporat
e
Partners
(vendors,
other
industry)
NMI
Advisory
Council
52
NMI Outreach Strategy
Sep 2001
Phase I
Building Awareness (Months 19)
We are HERE in time
Creating effective intellectual capital
building blocks for the general program
Broad information dissemination
Aggressive outreach/marketing to
higher-ed, industry, research
communities
Phase II
Delivering on promise (Months 6-20)
Interactive communication with primary user
communities
Building and disseminating “success stories”
Development of detailed documentation,
targeted events/activities and specific
intellectual capital
Phase III
Extending the reach (Months 12-24)
Refining engagement strategy –
independent and sustainable
Implementing “reference library” – evidence
of program credibility.
Establish regular schedule of
events/activities
53
Phase I – Building Awareness
GOAL: Creating effective intellectual capital
building blocks regarding the general
program –
• Websites
– www.nsf-middleware.org
– www.grids-center.org
– www.nmi-edit.org
• Logo(s) created
• Initial NSF Award Announcement Press Release issued
(24 September 2001)
• Three Campus Architecture and Middleware Planning
workshops
54
Phase I – Building Awareness
GOAL: Broad information dissemination –
• E-lists created encouraging “virtual” involvement:
– nmi-developer - Discuss NMI releases and development activities with
the open source community.
– nmi-supporter - Find out about Initiative-supporting opportunities and
how you can incorporate components from and NMI release into your
products.
– nmi-user - Find out about new project and participation opportunities
with the NMI.
– news - Receive NMI and Initiative-related press information
– nmi-announce - Keep informed about new developments in the NMI.
• Internet2
– mw-discuss – Discuss topics in enterprise middleware.
– mw-announce – Receive information on upcoming educational and
participation opportunities
55
Phase I – Building Awareness
GOAL: Aggressive outreach/marketing to
higher-ed, industry and the research
community
• Selected speaking opportunities – Internet2, EDUCAUSE,
Coalition for Networked Information, GGF-3 and GGF-4,
SC2001
• Defined and implemented a strategy for involving selected
vendor/industry partners – NMI Participation Model
56
NMI Participation
USERS
CONTRIBUTORS
DEVELOPERS
Targeted User
Communities
Other Interested
implementers
- Develop NMI-related or derived
components
- Support NMI components
- campuses
- GriPhyN, NEES, etc
- campuses
- industry
- government
NMI Testbed
Participants
SUPPORTERS
- Repackage NMI components and
distribute under own label
- determined by Call
for Participation
57
Phase II – Delivering on the
Promise
GOAL: Interactive communication with
primary user communities –
• NMI Documentation Team formed – working on plans for
appropriate Technical and other documentation to support
NMI Releases
• Coordinated with Training and Support Team to ensure
consistency and accuracy of messages to the users.
58
Phase II – Delivering on the
Promise
GOAL: Building and disseminating “success
stories” and other achievements –
• NMI Release 1 successfully delivered (7 May 2002)
• Announcements of NMI R1 distributed via information
distribution channels established – press lists, e-lists,
website.
• NMI Testbed Participants engaged in Outreach to share
their stores
– Internet2 tutorials and upcoming EDUCAUSE regional meetings
– NMI SC2002 theme – “Sharing NMI User Experiences” (related
activities and documentation)
59
Phase II – Delivering on the
Promise
Development of detailed documentation, targeted
events/activities, and specific, in-depth
intellectual capital
• Articles
– “Middleware: The New Frontier” – EDUCAUSE Review, July/August
2002
– “Middleware: Addressing the Top IT Issues on
Campus” – EDUCAUSE Quarterly, Vol. 25, No. 4 2002
• Presentations planned – GGF5, I2 Fall Meeting, SC2002,
EDUCAUSE Regionals, Coalition for Networked Information, and
others
• Tutorials/workshops planned – GRIDS Center, EDIT
– SC2002, EDUCAUSE, Internet2 meetings, and others
• Schedule for Year 2 deliverables in progress
– Base CAMP February 5-7, 2003 - Tempe, Arizona
60
Interested in NMI?
Visit the NMI web – http://www.nsf-middleware.org
Subscribe to NMI discussion and awareness e-lists
Potential User?
• Visit the NMI Release site and “test drive” NMI packaged
releases.
Potential Contributor?
• Send email to nmi-supporter@nsf-middleware.org indicating
interest in contributing components to the next release.
Press?
• Contact outreach@nsf-middleware.org about newsworthy
activities and achievements.
61
Questions?
Alan Blatecky –
ablatecky@nsf.gov
Ken Klingenstein –
kjk@internet2.edu
John McGee –
mcgee@isi.edu
Mary Fran Yafchak –
maryfran@sura.org
Ann West –
awest@educause.edu
62
www.internet2.edu
63