Instrumentation and Measurement
Background
GENI I&M Motivation
• To conduct useful and repeatable experiments, GENI
experimenters require extensive and reliable instrumentation
and measurement capabilities to gather, analyze, present and
archive Measurement Data (MD).
• In addition, the GENI operations staff require extensive and
reliable instrumentation and measurement capabilities to
monitor and troubleshoot the GENI suite and its constituent
entities. Some of this data will be made available to
experimenters, to help them conduct useful and repeatable
experiments.
FIBRE KOM - 29/10/11
2
Relationship to GENI Architecture
The Instrumentation and
Measurement WG focuses
on the instrumentation and
measurement
infrastructure that will be
deployed and used in
GENI.
FIBRE KOM - 29/10/11
3
GENI I&M Services Goals (1/3)
• Provide broad data gathering, analysis and archival capability
that is sufficient for scientific mission, operations, and success
of the infrastructure.
• Remove the burden on researcher to become a system and
network measurement infrastructure expert so that researcher
can better focus on the science in the experiments
• Measure details of GENI behavior with high precision and
accuracy in a ubiquitous, extensible, highly available, secure,
and integrated manner without adversely impacting
experiments.
FIBRE KOM - 29/10/11
4
GENI I&M Services Goals (2/3)
• Provide drill-down performance transparency of system and
network resources at hop, link, path and slice levels in terms
of availability, health status, and diagnosis of perceived as well
as impending problems.
• Allow and make-it-easy for various user groups to access and
control functions involving interactions between I&M subservices encompassing resources such as instrumentation taps
in the network, time sensors, software-based and hardwarebased measurement probes, router/switch MIBs, and (shortterm/long-term) measurement data archives.
FIBRE KOM - 29/10/11
5
GENI I&M Services Goals (3/3)
• Provide performance transparency of the status of the
individual I & M sub-service components and their interfaces
with other sub-services to ensure correctness of
measurements provisioned.
• Through tight integration with the CFs, provide mechanisms to
handle security, privacy and access control of measurement
data archives to allow access only to authorized users, and
also provide different data views based on authorization
privileges.
• Make it easy for data providers (e.g., instrumentation device
designers, experiments who implement custom
instrumentation devices, etc) to supply data (within legal and
ethical limits) to other entities.
FIBRE KOM - 29/10/11
6
GENI I&M User Groups
• Experimenters
• Opt-In Users: users who participate or “opt-in” to a GENI
experiment to utilize resources, applications or services.
• Central (i.e., GMOC) Operators
• Aggregate Providers and Operators
• Archive Providers and Operators: Groups that catalog indexes
of GENI-related measurement datasets in a repository and
provide tools for users to share, annotate, search and cite the
measurement datasets.
• Researchers that use Archived Measurement Data: Users that
utilize the measurement datasets provided by Archive
Providers in order to test hypotheses, and promote
reproducible research.
FIBRE KOM - 29/10/11
7
GENI Key I&M Services
•
•
•
•
•
•
Measurement Orchestration (MO) service
Measurement Point (MP) service
Measurement Information (MI) service
Measurement Collection (MC) service
Measurement Analysis and Presentation (MAP) service
Measurement Data Archive (MDA) service
FIBRE KOM - 29/10/11
8
GENI I&M Services Overview
Experimenter
Operator
Wide range of
MP Services
MO
MAP
MP
MI
MP
MC
MAP
MDA
MPs
MC
Researcher
Measurement Orchestration (MO) Service
FIBRE KOM - 29/10/11
10
Measurement Point (MP) Services
FIBRE KOM - 29/10/11
11
Measurement Point (MP) Services
FIBRE KOM - 29/10/11
12
Measurement Point (MP) Services
FIBRE KOM - 29/10/11
13
Measurement Information (MI) Service
FIBRE KOM - 29/10/11
14
Measurement Collection (MC) Services
FIBRE KOM - 29/10/11
15
Measurement Analysis and Presentation
(MAP) Service
FIBRE KOM - 29/10/11
16
Measurement Data Archive (MDA) Service
FIBRE KOM - 29/10/11
17
Experimenter Gathering MD from his/her Slice
Experimenter
MO
MP
MC
MP
MAP
MC
MDA
Experimenter Gathering MD from his/her Slice
FIBRE KOM - 29/10/11
19
Operator Gathering MD from GENI
Infrastructure
Operator
MAP
MI
MP
MP
Operator Gathering MD from GENI
Infrastructure
FIBRE KOM - 29/10/11
21
Experimenters Gathering MD from their Slice
and from GENI Infrastructure
FIBRE KOM - 29/10/11
22
Experimenters Gathering MD from their Slice
and from GENI Infrastructure
FIBRE KOM - 29/10/11
23
GENI I&M Architecture Overview:
Types of Services
• Type 1 Service: Service contained within a slice.
• Type 2 service: Common service platform plus multiple slivers
dedicated to multiple experiments.
• Type 3 service: Common service with shared data provided to
multiple experiments.
• Type 4 Service: MDA service with a portal for sharing data
FIBRE KOM - 29/10/11
24
perfSONAR
PERFormance Service Oriented
Network monitoring ARchitecture
perfSONAR as a Middleware
Analysis &
Visualization
Analysis &
Visualization
API
Measurement
Infrastructure
Measurement
Infrastructure
API
Data
Collection
Performance
Tools
FIBRE KOM - 29/10/11
26
perfSONAR Architecture Overview
Infrastructure
Data Services
Measurement
Points
Measurement
Archives
Information Services
Analysis/Visualization
Service
Lookup
User GUIs
Topology
Service
Configuration
Web Pages
NOC
Alarms
Transformations
Auth(n/z)
Services
FIBRE KOM - 29/10/11
27
perfSONAR
• Base network measurement schema
– OGF Network Measurement Working Group (NM-WG)
• Topology Schema
– OGF Network Markup Language (NML-) WG
– Includes Topology Network ID
• perfSONAR Protocol Documents
– OGF Network Measurement and Control (NMC-) WG
FIBRE KOM - 29/10/11
28
MonIPÊ Infrastructure
FIBRE KOM - 29/10/11
29
Visualização dos Resultados
30
Results visualization
Results visualization
• Backbone transition from 2.5 to 10Gbps
FIBRE KOM - 29/10/11
32
perfSONAR-MDM
• Developed within the scope of the GÉANT project
• Written in Java, was designed as a monitoring solution for LHC
(Large Hadron Collider).
• Available in Debian or RPM packages
• https://forge.geant.net/forge/display/perfsonar/Home
FIBRE KOM - 29/10/11
33
perfSONAR-MDM 3.3 Components
•
•
•
•
•
•
•
RRD Measurement Archive
SQL Measurement Archive
SSH/Telnet Measurement Point
Lookup Service
Authentication Service and Authorization Service
OPPD (previously called BWCTL) Measurement Point)
E2EMon Measurement Point
FIBRE KOM - 29/10/11
34
perfSONAR-PS
• http://psps.perfsonar.net/
• Collaboratively developed by:
– ESnet, Fermilab, Georgia Tech, Indiana University, Internet2,
SLAC, The University of Delaware
• Written in Perl. Available for independent deployment or
through the Performance Toolkit (pS-PT).
FIBRE KOM - 29/10/11
35
pS-PT
• perfSONAR Performance Toolkit (pS-PT)
• Current version: 3.2.1
– LiveCD or Net-Install based on CentOS 5.5
– Easy to deploy
• Boot with the instalation CD
• http://psps.perfsonar.net/toolkit/
– User-friendly Web Interface
• https://pspt.nuperc.unifacs.br
FIBRE KOM - 29/10/11
36
FIBRE KOM - 29/10/11
37
perfSONAR-OS Services
• SNMP Measurement Archive
• perfSONAR-BUOY
• perfAdmin GUI
• Lookup Service
• Topology Service
• Status Measurement Archive
• PingER Services
• PingER GUI
• LS Registration Daemon
FIBRE KOM - 29/10/11
38
pS-PT
• Metrics
– Throughput
– Delay
– Ping
– SNMP Utilization
– Cacti graphs with host performance information
– Scheduled tests:
• Throughput, one-way delay, and ping.
FIBRE KOM - 29/10/11
39
LAMP
• Leveraging and Abstracting Measurements with perfSONAR
• perfSONAR is a multi-domain performance monitoring
framework, which defines a set of protocols standards for
sharing data between measurement and monitoring systems.
• Objectives
– Collaborate on defining a common but extensible format for
data storage and exchange.
• Use perfSONAR NM-WG schema as starting point
– Develop a representation of GENI topolgy to be used to
describe measurements and experiment configuration
– Collaborate on a common GENI I&M architecture
FIBRE KOM - 29/10/11
40
perfSONAR-based GENI I&M
• Operations & Management:
– Federated GENI is perfect match to perfSONAR
– perfSONAR SNMP MA setup to query ProtoGENI switches on
Internet2
• Experiments:
– perfSONAR architecture is modular and extensible
– Integration with GENI will need
• Measurement Portal
• Measurement Orchestration
FIBRE KOM - 29/10/11
41
Onelab Monitoring
MySlice
FIBRE KOM - 29/10/11
43
TopHat (Topát)
•
•
•
•
http://www.top-hat.info/
PlanetLab Europe’s active measurement component
Part of OneLab’s effort: global federated environment
TopHat provides PlanetLab applications with a topology
monitoring service for the entire lifecycle of an experiment.
• Uses DIMES and ETOMIC for specialized Services
FIBRE KOM - 29/10/11
44
Experiment Lifecycle Support
• Experiment set up:
– Assist users in choosing the nodes for their experiments
based on measured characteristics of the network
• Run time:
– Provides info to support adaptative applications and
experiment control.
• Post-morten data:
– Data are archived for further analysis.
FIBRE KOM - 29/10/11
45
Topology Information
• Sources at IP level:
– BGP feeds:
• RouteViews, Team Cymrum, and PWhoIs
– Active measurements:
• ScriptRoute (PL native)
• Other: eg, perfSONAR
FIBRE KOM - 29/10/11
46
TopHat and perfSONAR
• Measurement tools:
– “perfSONAR offers a uniform set of tools from a set of peer entities”
– TopHat federates heterogeneous systems
• Focus:
– perfSONAR focus on troubleshooting across network boundaries
– TopHat focus on experiment support
• TopHat could benefit from accessing router information via
perfSONAR (queue lengths and packet drop rates).
FIBRE KOM - 29/10/11
47
TopHat Architecture
perfSONAR
FIBRE KOM - 29/10/11
48
Core API
• Get:
– request information and measurements about nodes and
paths
• Filter:
– filters a ser of nodes or paths
• SetCallback:
– Configures conditions on which the system will react by
triggering a callback function
FIBRE KOM - 29/10/11
49
DIMES
• www.netdimes.org
• distributed scientific research project, aimed at studying the
structure and topology of the Internet, with the help of a
volunteer community (similar in spirit to projects such as
SETI@Home).
• Traceroute measurement module
• Large number of vantage points (ca. 1700 measurement
agents)
FIBRE KOM - 29/10/11
50
ETOMIC
• Infrastructure with GPS synchronized servers.
• With special cards for a precision of 10 ns
• Performs one way delay and available bandwidth
measurements pre-scheduled tests
• Deployment on PlanetLab and other points
• Similar to the results produced by OWAMP and BWCTL MPs in
perfSONAR.
FIBRE KOM - 29/10/11
51
SONOMA
• On-demand tests
• Precision on the order of ten’s of microseconds
• Uses web services for test requests
FIBRE KOM - 29/10/11
52
OML
The Orbit Measurement Library
OML
• A client library (liboml2) for instrumenting your applications;
plus
• A measurement server for collecting and storing
measurements, remotely.
• Tutorial:
http://groups.geni.net/geni/attachment/wiki/Gec9InstMeasW
orkshopAgenda/OML_tutorial.pptx
FIBRE KOM - 29/10/11
54
Monitoring in OFELIA
Monitoring in OFELIA (D5.1)
• The control framework should perform monitoring tasks, for
both the components conforming the control framework, and
the resources on the testbed.
• Main focus: monitoring resource usage in order to avoid
congestion and guarantee slices isolation.
FIBRE KOM - 29/10/11
56
Monitoring in the FIBRE-BR testbed
Initial ideas/proposals
Initial Ideas
• Ideally we would monitor:
– Experiments (slices)
– Infrastructure (Testbed)
• User groups:
– Experimenters
– Central Operators
– Aggregate (Island) Managers
• Initial proposals:
– Leverage on existing monitoring
– Federation through perfSONAR
FIBRE KOM - 29/10/11
58
Existing Monitoring by Aggregate
• OF Aggregate:
– Flow information:
• SNMP, Controller (and FlowVisor) info, directly from OF API
• PlanetLab Aggregate:
– TopHat, other tools
• OMF Aggregate:
– OML
• ProtoGENI Aggregate:
– LAMP
FIBRE KOM - 29/10/11
59
Initial Ideas
• Adapting LAMP to the FIBRE CMF
• Collect and Export OpenFlow Data
• Integration of OML to perfSONAR
FIBRE KOM - 29/10/11
60
Adapting LAMP to the FIBRE CMF
• LAMP was developed for ProtoGENI
• We plan to study and possibly adapt it for use with the FIBRECMF
FIBRE KOM - 29/10/11
61
Collect and Export OpenFlow Data
• Information about slices, queues, response time, flow
aggregation and flow setup time, not limiting the monitoring
framework to these parameters.
• OpenFlow switch table and port statistics can be collected at
its controller, e.g., NOX.
• The collected data will be made available through a
measurement archive (MA) and shared to other monitoring
tools.
FIBRE KOM - 29/10/11
62
Integration of OML to perfSONAR
• OML main feature is the possibility to add Measurement
Points (MP) to applications or services and it is a researcher’s
decision to enable it at run-time [White, 2010].
• OML focuses mainly on mobile networks and FIBRE proposed
monitoring framework could benefit from this expertise and
could integrate this service to perfSONAR, at least through a
MA interface to OML’s server.
• It is also interesting to analyze and monitor the communication
between the client and the server in FIBRE’s federated
environment.
FIBRE KOM - 29/10/11
63
Interconnection Monitoring
• connections among FIBRE-BR Islands, FIBRE-EU islands, and
intercontinental link
• routers SNMP data (wherever available).
• [passive measurements ?!]
FIBRE KOM - 29/10/11
64