NSF Middleware Initiative
Building for the Future
Kevin Thompson, NSF
Ken Klingenstein, Internet2
Tom Garritano, Grids Center
Mary Fran Yafchak, SURA
Ann West, EDUCAUSE/Internet2
19 March 2016
Topics
NMI Overview and New Awards
NMI-EDIT
GRIDS Center
NMI Integration Testbed
NMI and NMI-EDIT Outreach
10 April 2003
2
NSF Middleware Initiative
 Purpose
 To design, develop, deploy and support a
set of reusable, expandable set of
middleware functions and services that
benefit applications in a networked
environment
10 April 2003
3
A Vision for Middleware
To allow scientists and engineers the ability to transparently use and
share distributed resources, such as computers, data, and
instruments
To develop effective collaboration and communications tools such as
Grid technologies, desktop video, and other advanced services
to expedite research and education, and
To develop a working architecture and approach which can be
extended
to Internet users around the world.
Middleware is the stuff that makes “transparently use” happen,
providing persistency, consistency, security, privacy and
capability
4
10 April 2003
NMI Status
21 Active Awards (Prior to new awards)
• 3 Cooperative Agreements
• 18 individual research awards
Focus on integration and deployment of grid and
middleware for S&E
• Near-term deliverables (working code)
• coordination and persuasion rather than standards
• Significant effort on interoperability, testing, inclusion
NMI Software Releases, best practices, white papers
•
•
•
•
NMI Release 1 – May, 2002
NMI Release 2 – Oct, 2002
NMI Release 3 – Apr, 2003
NMI Release 4 - est. Dec, 2003
10 April 2003
5
NMI Organization
Core NMI Team
• GRIDS Center
– ISI, NCSA, U Chicago, UCSD & U Wisconsin
• EDIT Team (Enterprise and Desktop Integration Technologies)
– EDUCAUSE, Internet2 & SURA
• Several additions in 2003
Grants for R & D
• Year 1 -- 9 grants
• Year 2 -- 9 grants
• Year 3 (New) -- 10 grants
10 April 2003
6
2003 NSF Middleware Initiative
Program Awards
 20 awards totaling $9M
• 10 “System Integrator” awards
– Focus – to further develop the integration and support
infrastructure of middleware for the longer term
– Themes - extending and deepening current activities,
and expanding into new areas
• 10 smaller awards focused on near-term
capabilities and tool development
– Focus – to encourage the development of additional
new middleware components and capabilities for the
NMI program
10 April 2003
7
2003 New System
Integrator Awards
Butler (UIUC)
Disseminating and Supporting Middleware
Infrastructure: Engaging and Expanding Scientific
Grid Communities
Kesselman
(USC/ISI)
Designing and Building a National Middleware
Infrastructure (NMI-2)
Klingenstein
(UCAID)
Extending Integrated Middleware to Collaborative
Environments in Research and Education
Livny (U
Wisc)
An Integrative Testing Framework for Grid
Middleware and Grid Environments
McMullen
(Indiana)
Instruments and Sensors as Network Services:
Instruments as First Class Members of the Grid
Pierce, Alameda,
Severance,
Thomas, and
von Laszewski
Collaborative Proposal: Middleware for Grid Portal
Development
10 April 2003
8
Other New Awards in 2003
Chase (Duke),
Ramakrishnan (MCNC)
Collaborative Research: A Grid Service for Dynamic
Virtual Clusters
Gemmil (UAB)
NMI-Enabled Open Source Collaboration Tools for
Virtual Organizations
Karonis (Northern Illinois)
Critical Globus-enabled Implementation of the MPI-2
Standard
Lumsdaine (Indiana)
Scalable Fault Tolerance for MPI
Ramachandran (Ga Tech)
Exploration of Middleware Technologies for Ubiquitous
Computing with Applications to Grid Computing
Saltz (Ohio St. URF)
GridDB-Lite: Database Support for Data-Driven
Scientific Applications in the Grid
Wright (U Wisc.),
Linderoth (Lehigh)
Collaborative Research: MW: Master-Worker
Middleware for Grids
Arzberger (UCSD)
Pacific Rim Application and Grid Middleware Assembly
10 April 2003
9
Looking Ahead
 There will be an NMI solicitation in 2004
 Exact funding level not set
 NMI program is expected to be a primary focus area
under CISE’s new division - Shared
CyberInfrastructure
 October 23 Review at NSF for existing activities
among the Grids Center and EDIT teams
10 April 2003
10
Enterprise and Desktop Integration
Technologies (EDIT) Consortium
Ken Klingenstein
Director, Internet2 Middleware Initiative
kjk@internet2.edu
10 April 2003
11
Overview
 NMI-EDIT Overview
 Research Findings
 NMI Release Components
 Building on the Future
10 April 2003
12
NMI-EDIT Consortium
 Enterprise and Desktop Integration
Technologies Consortium
• Internet2 EDUCAUSE, SURA
• Almost all funding passed through to campuses for work
 Project Goals
• Create a common, persistent and robust core middleware
infrastructure for the R&E community
• Provide tools and services in support of inter-institutional
and inter-realm collaborations
10 April 2003
13
A Map of Middleware Land
10 April 2003
14
NMI-EDIT Findings
Consensus on inter-institutional middleware
standards and maturing architecture to
support collaborative applications
Widespread interest in Shibboleth within R&E
communities
Credential mapping from core enterprise to
Grid service
Grid adoption of SAML in Open Grid Services
Architecture (OGSA)
10 April 2003
15
NMI-EDIT Findings (cont.)
Creation and maintenance of a heavily
referenced set of design and best practices
documents
Effective linkages with International research
communities
Discovery and development of campus IT
staff, leading to
Influence on both federal and commercial
standards
Direct outreach to over 320 institutions
10 April 2003
16
NMI-EDIT Components from
Three NMI Releases
 Authentication:
• WebISO solution, credential mapping from Kerberos to
PKI, policy documents, registry service
 Enterprise Directories:
• Schemas; operational monitoring and schema analysis
tools; practices in design, groups, and metadirectories
 Authorization:
• Architecture and related software and libraries for multiinstitution collaboration
10 April 2003
17
NMI-EDIT Components from
Three NMI Releases (cont.)
 Integration Activities:
• Credential mapping from campus to Grid environment,
GLUE schema analysis tool
 Education:
• Venues for learning about enterprise middleware
including CAMPs and on-line deployment materials for
directories
10 April 2003
18
NMI-EDIT:
Building for the Future
 Authorization infrastructure
• Develop authorization architecture to support
– campus applications, both legacy and productivity
– network services
– inter-realm collaboration efforts
– virtual organizations
• Deliverables
– An Enterprise Authorization Design and Implementation
Guide, with case histories
– A set of tools for use with Authz, including registry
templates for policies, roles, and authorities, a
GROUPER group manager, end user interfaces, etc.
10 April 2003
19
NMI-EDIT:
Building for the Future (cont.)
Middleware end-to-end diagnostics
• How do I resolve a permission denied problem?
– From the user perspective
– From the user’s help-desk perspective
– From the technical support perspective
• Deliverables first year
– Develop an expert group to design an interim
architecture and functionality
– Build liaisons to adjacent efforts by other groups
– Design standard, extensible logging formats
– Create a harvesting tool that can assemble end-end
middleware flows from logs
– Work with KX.509, Shibboleth, etc…
10 April 2003
20
New Partnerships
Work with JISC on Virtual Organizations
• A key cross-stitch among enterprises for interinstitutional collaborations
• VO’s range from Grids to digital libraries, from
earthquake engineering to collaborative curation,
from managing observatories to managing rights
Interworkings with Australian, Swiss,
French universities
Corporate interactions with MS, Sun,
Liberty Alliance, etc
10 April 2003
21
NMI-EDIT:
Next Generation Architecture
10 April 2003
22
The pieces fit together…
 Campus infrastructure
• Developing and encouraging the deployment of identity
management components, tools, and support services
 Inter-realm infrastructure
• Leveraging the local organizational infrastructure to enable
access to the broader community though
–Building on campus identity management infrastructures
–Extending them to contain standard schemas and data definitions
–Enabling the exchange of access information in a private and
secure way
–Developing diagnostic tools to make complex middleware
interactions easier to understand
10 April 2003
23
The GRIDS Center:
Defining and Deploying
Grid Middleware
presented by
Tom Garritano
University of Chicago and Argonne National Laboratory
10 April 2003
24
NSF Middleware Initiative (NMI)
 GRIDS is one of two original teams,
the other being EDIT
 New NMI teams just announced
(Grid portals and instrumentation)
 GRIDS releases well-tested,
deployed and supported middleware based on
common architectures that can be extended to
Internet users around the world
 NSF support of GRIDS leverages investment by
DOE, NASA, DARPA, UK e-Science Program, and
private industry
10 April 2003
25
GRIDS Center
 GRIDS = Grid Research Integration Development &
Support
 Partnership of leading teams in Grid computing
•
•
•
•
•
•
University of Chicago and Argonne National Lab
Information Sciences Institute at USC
NCSA at the University of Illinois at Urbana-Champaign
SDSC at the University of California at San Diego
University of Wisconsin at Madison
Plus other software contributors (to date: UC Santa Barbara, U. of
Michigan)
 GRIDS develops, tests, deploys and supports standard
tools for:
• Authentication, authorization, policy
• Resource discovery and directory services
• Remote access to computers, data, instruments
26
The Grid: What is it?
 “Resource-sharing technology with software
and services that let people access
computing power, databases, and other tools
securely online across corporate,
institutional, and geographic boundaries
without sacrificing local autonomy.”
 Three key Grid criteria:
• coordinates distributed resources
• using standard, open, general-purpose protocols
and interfaces
• to deliver qualities of service not possible with
pre-Grid technologies
27
Virtual Organizations
• Distributed resources and people
R
R
R
R
R
R
R
R
R
R
R
R
10 April 2003
28
Virtual Organizations
• Distributed resources and people
• Linked by networks, crossing administrative domains
R
R
R
R
R
R
R
R
R
R
R
R
10 April 2003
29
Virtual Organizations
• Distributed resources and people
• Linked by networks, crossing administrative domains
• Sharing resources, common goals
R
R
R
R
R
R
R
R
R
R
R
VO-A
R
VO-B
10 April 2003
30
Virtual Organizations
•
•
•
•
Distributed resources and people
Linked by networks, crossing administrative domains
Sharing resources, common goals
Dynamic
R
R
R
R
R
R
R
R
R
R
R
VO-A
R
VO-B
10 April 2003
31
Virtual Organizations
•
•
•
•
•
Distributed resources and people
Linked by networks, crossing administrative domains
Sharing resources, common goals
Dynamic
Fault tolerant
R
R
R
R
R
R
R
R
R
R
R
VO-A
R
VO-B
10 April 2003
32
GRIDS Center
Software Suite
 Globus Toolkit®. The de facto standard for Grid
computing, an open-source "bag of technologies" to
simplify collaboration across organizations. Includes
tools for authentication, scheduling, file transfer and
resource description.
 Condor-G. Enhanced version of the core Condor
software optimized to work with GT for managing
Grid jobs.
 Network Weather Service (NWS). Periodically
monitors and dynamically forecasts performance of
network and computational resources.
 Grid Packaging Tools (GPT). XML-based
packaging data format defines complex
dependencies between components.
10 April 2003
33
GRIDS Center
Software Suite (cont.)
 GSI-OpenSSH. Modified version adds support for
Grid Security Infrastructure (GSI) authentication and
single sign-on capability.
 MyProxy. Repository lets users retrieve a proxy
credential on demand, without managing private key
and certificate files across sites and applications.
 MPICH-G2. Grid-enabled implementation of the
Message Passing Index (MPI) standard, based on
the popular MPICH library.
 GridConfig. Manages the configuration of GRIDS
components, letting users regenerate configuration
files in native formats and ensure consistency.
 KX.509 and KCA. A tool from EDIT that bridges
Kerberos and PKI infrastructure.
10 April 2003
34
E-Science Benefits Substantially
from GRIDS Components
 Large-scale IT deployment projects rely on GRIDS
components and architecture for core services
• BIRN, the Bioinformatics Research Network
• GEON, the Geoscience Network
• GriPhyN, Particle Physics Data
Grid, International Virtual Data
Grid Laboratory
• NEESgrid, part of the Network for
Earthquake Engineering and Simulation
• International projects such as the UK
e-Science Program and EU DataGrid
BIRN MRI Data for
Brain Imaging
 GRIDS standard tools let projects avoid building
their own infrastructure
• Increases interoperability, efficiency
• Prevents “balkanization” of applications
35
Industrial and International
Leaders Move to Grid Services
 GRIDS leaders engage a worldwide community in
defining specifications for Grid services
• Very active working through Global Grid Forum
• Over a dozen leading companies (IBM, HP, Platform) have
committed to Globus-based Grid services for their products
 NMI-R4 in December will include Globus Toolkit 3.0
• GT3 is the first full-scale deployment of new Open Grid
Services Infrastructure (OGSI) spec
• Significant contributions from new international partners
(University of Edinburgh and Swedish Royal Institute of
Technology) for database access and security
• UK Council for the Central Laboratory of the Research
Councils (CCLRC) users rank deployment of GT3 as their #1
priority
36
Acclaim for GRIDS Components
 On July 15, the New York Times
noted the “far-sighted simplicity”
of the Grid services architecture
 The Globus Toolkit has earned:
• R&D 100 Award
• Federal Laboratory Consortium Award
for Excellence in Technology Transfer
 MIT Technology Review named Grid
one of “Ten Technologies That Will
Change the World”
 InfoWorld list of 2003’s top 10 innovators
includes two GRIDS PIs
 GRIDS co-PI Ian Foster named “Innovator
of the Year” for 2003 by R&D Magazine
37
Future GRIDS Plans
 GRIDS is completing its second year in October
• Original three-year award, through Fall 2004
• Very successful in establishing processes, meeting twice/year release
schedule, defining broadly accepted Grid middleware standards, and
increasing public awareness of Grid computing
 GRIDS Center 2 plans
•
•
•
•
•
Further develop and refine core NMI releases and processes
Deploy tools based on Open Grid Services Architecture
Expand testing capability
Create a federated bug-tracking facility
Public databases: Grid Projects and Deployments System and Grid
Technology Repository
• Increase outreach to communities at all levels:
– Existing major Grid projects (e.g., TeraGrid, NEESgrid)
– Major projects that should use Grid more (e.g., SEEK, NEON)
– New communities not yet using Grid (e.g., Computer-Aided 10
Diagnosis)
38
April 2003
Upcoming Tutorials
 GRIDS is extremely well-represented at
SC03, the supercomputing conference
• Tutorials, technical papers, BoFs, demonstrations
• Phoenix, AZ, November 15-21
• http://www.sc-conference.org/sc2003
 GlobusWORLD 2004 conference
•
•
•
•
Co-sponsored by GRIDS
San Francisco, CA, January 20-23
Academia and Industry both well-represented
http://www.globusworld.org
10 April 2003
39
For More Information
The GRIDS Center
http://www.grids-center.org/
NSF Middleware Initiative
http://www.nsf-middleware.org/
The Globus Alliance
http://www.globus.org/
10 April 2003
40
NMI Integration Testbed
NMI Integration Testbed
Mary Fran Yafchak
maryfran@sura.org
NMI Integration Testbed Manager
SURA IT Program Coordinator
Southeastern Universities Research Association
10 April 2003
41
About the NMI Testbed
• Developed and managed by SURA
NMI Participation
USERS
CONTRIBUTORS
Implementers
Target Communities
DEVELOPERS
NMI Integration
Testbed
SUPPORTERS
• Evaluate NMI components upon release
• Real life contexts - research projects,
enterprise applications and infrastructure
NMI Integration Testbed
Sites
UAB
UAH
UFL
FSU
GSU
UMich
TACC
UVA
future
expansion
?
(USC)
http://www.nsf-middleware.org/testbed
10 April 2003
42
Activities this year
1)
2)
3)
4)
5)
Evaluation of NMI Releases 2 & 3
Continued project & enterprise integration
Addition of REU student positions
Workshops & Presentations
Firing up of intra-Testbed grid
10 April 2003
43
Evaluation of NMI R2 & R3
Once upon a time…
 NMI R1, completed September 2002. 18 components, 61
reports, focused “across the board”
This past year…
 NMI R2, completed February 2003. 25 components, 59 reports,
focused “across the board”
 NMI R3, completed August 2003. 30 components, 57 reports,
heavy on grids and newer authn/authz components
Trend towards more “practical” evaluation
10 April 2003
44
Project/Enterprise Integration
Catalyzing advanced infrastructure
• Seven sites providing centralized identity & authentication
for multiple active applications
– Examples: UAB Faculty/Class emailer, UFL Wireless VPN
control
• Four sites implementing campus grids (various stages)
– MGrid, USCGrid, UTGrid (w/TIGRE), GridGroup@GSU
10 April 2003
45
Project/Enterprise Integration
Extending access for existing projects
 Examples:
 UAH: Grid-based applications with NCSA MEAD expedition and NASA
Marshall Space Flight Center
 UMich, GSU: Access for physicists to DOE experiments (Particle Physics
Data Grid, ATLAS, CERN LHC)
Expanded access for new audiences
 Examples:
 GSU: Student access to graphic rendering capability
 GSU: Distributed muon detector in collaboration with GSU physicists and
Georgia state high schools.
 NSF REU (Research Experiences for Undergraduates)
10 April 2003
46
NSF REU in NMI
 SURA provides admin & management; sites
provide experience and mentoring
 5 positions at 4 sites: GSU, UMich, TACC/UVa
• GSU
– Muon particle detector GRID for K-12
– GRID-enabled Applications Catalog
• UMich
– NMI components in ATLAS and MGRID
• TACC/UVA
– Grid Portals for the NMI Integration Testbed
10 April 2003
47
Intra-Testbed Grid
 Rationale - Testbed sites’ interest, expertise and
position to contribute
 Two foci - Demonstration (showcase) & Research
• Showcase lead: Art Vandenberg at GSU
• Research leads; Marty Humphrey & Jim Jokl at UVa
 Key deliverables (through August 2004) • Working showcase grid and application catalog
• Identification and documentation of cross-campus issues
(barriers to scalability) and recommendations for resolution
10 April 2003
48
NMI Testbed Workshops
 1st Testbed Results workshop
• April 2003 at Internet2 Spring Meeting
 SURA NMI PACS workshops
• Small group training in enterprise directories and related
applications (Aug & Sept. 2003)
 2nd Testbed Results workshop
• Monday, Nov. 3, preceding EDUCAUSE 2003
• See Testbed Web site for more details
• Registration closes this Friday! (October 17)
10 April 2003
49
NMI Testbed Presentations
 At I2 meetings (Spring 2003, Fall 2003)
Next
Tomorrow
– “Taking Grids out of the Lab and onto the Campus,”
Wednesday, 10/14, 10:30 - 11:45 a.m.
– “Advanced Directory Services and Applications,” Thursday,
10/15, 8:45 - 10 a.m.
 At EDUCAUSE 2003 annual conference
– “Experiences in Middleware Deployment: Teach a Man to
Fish…,” Thursday, 11/6, 3:55 - 4:15 p.m.
 At GlobusWorld 2004?
– Proposed presentation on the intra-Testbed grid. Waiting to
hear…
10 April 2003
50
NMI and
NMI-EDIT Outreach
Ann West
EDUCAUSE/Internet2
10 April 2003
51
Topics
NMI participation model
NMI-EDIT goals, products, and results
Education opportunities
10 April 2003
52
NMI Outreach
Participation Overview
10 April 2003
53
NMI-EDIT Outreach Goals
 Informing higher-education constituents about
enterprise middleware and available resources.
 Guiding a broad selection of campuses interested in
implementing enterprise middleware towards
deployment and service integration, and assisting
them in joining the emerging middleware community
to benefit from peer support and developments in that
community.
 Serving the technical community researching and
developing enterprise middleware infrastructures.
 Establishing information and support persistence.
10 April 2003
54
NMI-EDIT Outreach Products
 Awareness presentations
• 22 presentations last year
 Workshops and tutorials
• 1060 total attendance.
• 718 distinct participants
• 323 distinct organizations
 Content development
• Articles
• Directory Roadmap
 Community work
• Minority serving institutions and small colleges
• Registrars (AACRAO) and CFOs (NACUBO)
• Higher-education systems
 Persistence
10 April 2003
55
10 April 2003
56
10 April 2003
57
Education Opportunities
CAMPs
• Directory CAMP Feb 3-6, 2004 in Tempe AZ
• CAMP/Research CAMP June and November 2004
EDUCAUSE regional and annual
meetings
Getting started?
• Check out getting started link on www.nmi-edit.org
10 April 2003
58
More Information…
NMI
• nsf-middleware.org
NMI-EDIT
• www.nmi-edit.org
• middleware.internet2.edu
GRIDS Center
• www.grids-center.org
10 April 2003
59
10 April 2003
60