Substrate Control:
Resource Manager Proxy
(See Types.ppt for a description of data types used in
this presentation)
Fred Kuhns
fredk@arl.wustl.edu
Applied Research Laboratory
Washington University in St. Louis
Defining Slices: GNM → RMP
• Message context ID = 0 (privileged operation)
• retCode_t add_slice(plabID_t, name_t plName)
Message context ID = 0
plabID: PLC Globally unique Slice ID.
plName : unique string naming slice
– must be received on a separate, privileged control socket
– return 0 for success or an error code
• retCode_t rem_slice(plabID_t)
– delete any/all defined fast paths
– remove all slice state from rmp
•
On startup RMP must read the passwd file for a list of currently active slices ... but this
will not provide the plabID!!??
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
2
Reserving Fastpath Resources
See Types.ppt for type definition and mapping to substrate API, message context ID = 0.
Reserve resources for 0..N fastpaths (on NPEs) and 0..1 GPEs
• retCode_t make_reservation(rsvRecord_t)
Update an existing reservation, note because all operations need to be idempotent the
update argument includes all required resources.
• retCode_t update_reservation(rsvRecord_t)
Cancel any reservation overlapping date_t
• retCode_t cancel_reservation(date_t)
Return any reservation (belonging to slice) that overlaps with date_t
• rsvRecord_t? get_reservation(date_t)
Return a list of all reservations associated with calling slice
• rsvRecord_t* get_reservations()
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
3
XML For a Fastpath Reservation Record
<rsvRecord>
<rDate start=“Date” end=“Date” />
<fpRSpec >
<fpName>name</fpName>
<fpParams>
<copt> code_option_tag </copt>
<bwspec firm=“bw” soft=“bw” />
<rcnts fltrs=“N” queues=“N” buffs=“N” stats=“N” />
<mem sram=“N”, dram=“N” />
</fpParams>
<ifParams> <ifRec bw=“N” ip=“ipAddr” /> ... </ifParams>
</fpRSpec>
<plRSpec>
<ifParams> <ifRec bw=“N” ip = “ipAddr” /> ... </ifParams>
</plRSpec>
</rsvRecord>
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
4
Allocate Code Option
• fpInfo_t alloc_fastpath(copt_t, bwspec_t, rcnts_t, mem_t)
copt_t code option (see Types.h)
bwspec_t: Aggregate BW requirements for fastpath across all interfaces
rcnts_t: Max resource counts
mem_t: Requested size of SRAM and DRAM in Bytes (currented fixed by code option)
fpInfo_t: NP instance information
•
RMP must:
1. send alloc_fastpath request to SRM. SRM performs admissions control, allocates
resources and notifies SCD of allocations and new fastpath instance number. Returns the fp
parameters
fp alloc_fastpath(copt, bwspec, rcnts, mem)
2. open logical interface and assign it the allocated fp VLAN id, then add NPE’s MAC address
to the arp table
vconfig add ethX vlan
ifconfig ethX.vlan up fpIP netmask fpNM -arp
route add -net fpNet netmask fpNM gw npeIP
arp -i ethX.vlan -s npeIP npeMAC
3. send set_gpe_info command to scd, using the last two valid qid’s, set context id to fpid,
retCode_t set_gpe_info(exPort, ldPort, exQID, ldQID)
4. open two UDP sockets using fpIP and two ephemeral ports for local delivery and exception
packets. These sockets are passed back to the client as ancillary data.
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
5
Removing a Fastpath
•
void free_fastpath(fpID_t) :
•
RMP must:
1.
remove logical interface for fastpath
route del -net fpNet netmask fpNM gw npeIP
ifconfig ethX.vlan down fpIP netmask fpNM -arp
vconfig rem ethX.vlan
arp -i ethX.vlan -d npeIP
2.
3.
4.
send set_gpe_info command to scd, setting entries to 0, set context id to fpid,
retCode_t set_gpe_info(0, 0, 0, 0)
send free_fastpath to srm with context ID set to fpid
retCode_t free_fastpath()
remove remaining fp state and send reply to client
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
6
Instantiate Fastpath and/or Slowpath reservations
• All resource allocation methods operate on the currently active reservation
(i.e. must be used during the interval of time when a reservation is valid).
– If free is not called then all resources are automatically freed when reservation
expires.
allocate any GPE/Slice resources reserved in the current interval
• retCode_t alloc_plspec()
instantiate the “current” fastpath reservation with name fpName
• fpInfo_t alloc_fpspec(name_t fpName=“”)
– if only one fastpath then the fpName parameter can be omitted.
free any GPE specific resources currently allocated to slice.
• retCode_t free_plspec()
Destroy fastpath instance, releasing meta-interfaces and NP resources.
• retCode_t free_fpspec(fpID_t)
– rmp converts slices specific fpid to global fpid
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
7
System Interfaces, Address and Ports
Reserve Aggregate Interface BW for slice (traffic to/from GPE).
• retCode_t resrv_pl_ifbw(ifn_t, bw_t)
– Request to reserve bw Kbps on interface ifn for the calling slice
Release slice interface BW reservation for GPE
• retCode_t reles_pl_ifbw(ifn_t, bw_t): release bw on ifn
Reserve aggregate Interface BW for indicated fastpaths
• retCode_t resrv_fpath_ifbw(fpID_t, ifn_t, bw_t)
– Request to reserve bw Kbps on interface ifn for the calling slice’s fast path
Release Aggregate Interface BW allocation for indicated fastpath
• retCode_t reles_fpath_ifbw(fpID_t, ifn_t, bw_t)
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
8
System Interfaces, Addresses and Ports
Get list of available interfaces
• if_list get_interfaces()
– Message context ID = 0.
if_list = {ifAttr_t, ...}
ifAttr_t {
ifn_t
ifn; // Interface number
ifType_t type; // {Multi-Access = 0, Point-to-Point = 1}
ipAddr_t ipaddr;
// Interface/Link IP address
ipAddr_t linkBW;
// Raw (Gross) interface (link) BW
bw_t
availBW;
// Available (Net) interface (link) BW
}
– if type = point-to-point then may call get_ifpeer()
Map interface IP address to interface number.
• ifn_t get_ifn(ipAddr_t)
Get updated interface attributes
• ifAttr_t get_ifattrs(ifn_t)
Get the IP address of point-to-point link’s peer node (SPP peer).
• ipAddr_t get_ifpeer(ifn_t)
– If interface ifn is type peering then this returns the IP address of the peer. Otherwise
returns null.
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
9
Allocating EndPoints on the GPE
• Message context ID set to 0
Allocate a local endpoint for slice (linecard directs matching traffic to
the corresponding GPE)
• epInfo_t alloc_endpoint(epInfo_t)
– Allocate new endpoint, installs filter in line card to direct matching traffic to
the GPE.
– if proto is TCP or UDP and port == 0 then the substrate will allocate a port
number.
– If user requests a port and its not available then an error is returned.
– If a slice has not reserved sufficient BW on an interface then an error is
returned.
– If bw is set to 0 then share BW.
Release allocated endpoint (linecard filters removed and address added
back to available pool of addresses tuples).
• retCode_t free_endpoint(epoint_t)
– Removes the endpoint’s filter and bandwidth allocation.
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
10
Adding Meta-Interfaces
Allocate local Address for Slice’s fastpath (MI) and forward to NPE
uint8_t tunnel_type : must be one of {UDP = 0}
• [mi, ep] alloc_tunnel(fpid, bw, ipaddr, tunnel_type, params)
Not implemented
• [miID_t, epInfo_t]
alloc_udp_tunnel(fpID_t, bw_t, ipAddr_t, ipPort_t)
– Adds one end of a UDP tunnel, treated as a meta-interface for a fast path instance.
– if port = 0 then one is allocated from the free pool
– Returns the meta-interface number representing the new endpoint.
• status free_tunnel(ipaddr, tunnel_type, params) Not
implemented
• retCode_t free_udp_tunnel(fpID_t, miID_t)
• epoint_t get_endpoint(fpID_t, miID_t)
– Only valid if mi is a valid meta-interface identifier returned from a call to
alloc_udp_tunnel or alloc_tunnel
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
11
Adding Meta-Interfaces
Allocate local Address for Slice’s fastpath (MI) and forward to NPE, will also
reserve sufficient aggregate BW on corresponding internal and external
interfaces.
• [miID_t, epInfo_t]
alloc_udp_tunnel(fpID_t, bw_t, ipAddr_t, ipPort_t)
–
–
–
–
It is an error if the sum of requested BW exceeds the reserved fastpath BW.
Adds one end of a UDP tunnel, treated as a meta-interface for a fast path instance.
if port = 0 then one is allocated from the free pool
Returns the meta-interface number representing the new endpoint
Release BW and Address associated with corresponding meta-interface
• retCode_t free_udp_tunnel(fpID_t, miID_t)
Map meta-interface ID to corresponding externally visible address tuple
• epoint_t get_endpoint(fpID_t, miID_t)
– mi must be a valid meta-interface ID returned from a call to alloc_udp_tunnel
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
12
Manipulating Queues and Attributes
•
All fastpath queues must have a bandwidth assigned and be associated with an interface.
The model is that a user allocates BW on each meta-interface. They may then allocate this
BW amongst their queues as they please; the substrate will enforce total BW restrictions.
Associate one or more queues with a meta-interface
• retCode_t bind_queue(fpID_t, miID_t, list_type, qid_t[])
– uint8_t list_type: explicit list (0) or range specification (1)
– qid_t[]: list of queue identifiers to associate with meta-interface mi. In C or C++ this would
be an array of integers {0, ..., n}. If using a range then the list has the first and one past the last id
of the range. For example {4, 8} is the same as {4, 5, 6, 7}.
Set Queue parameters (bandwidth in Kbps and threshold in packets)
• bw_t set_queue_params(fpID_t, qid_t, cnt_t threshold, bw_t)
Get the current queue parameters for qid.
• [cnt_t threshold, bw_t] get_queue_params(fpID_t, qid_t)
Get the current queue length in packets and bytes
• {cnt_t pktCnt, cnt_t byteCnt} get_queue_len(fpID_t, qid_t)
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
13
Managing the Lookup table
•
Common definitions (N = 14 Bytes), Message context ID = 0.
Add new filter with ID fid for fastpath fpid (smaller fids have higher priority)
• retCode_t write_fltr(fpID_t, fid_t, fpFltr_t)
–
–
Can we overwrite existing valid entries?? Don’t think so.
OK to call with an invalid fid, returns an error.
Update result vector for filter ID fid
• retCode_t update_result(fpid, fid, fpFltrResult_t)
–
If fid is a valid entry then updates result vector. Otherwise no change to the DB
Lookup filter by ID or Key returning the filter entry
• fpFltr_t get_fltr_byfid(fpid, fid)
fpFltr_t get_fltr_bykey(fpid, fpKeyWrap_t)
–
–
returns the filter or an error code.
returns error if called with an invalid FID or Key
Lookup filter by key and return the result vector
• fpFltrResult_t lookup_fltr(fpid, fpKeyWrap_t)
–
returns the result field if key/fid matches a valid entry, otherwise NULL
Remove filter with indicated FID or Key
• retCode_t rem_fltr_byfid(fpid, fid)
retCode_t rem_fltr_bykey(fpid, fpKeyWrap_t)
–
–
returns success if the entry exists (and was invalidated) or an error code.
Returns error if called with an invalid fid or key.
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
14
Stats (accessing FP instance specific counters)
One-time read of one of the stats counters for fastpath FPID
• statRec_t
read_stats(fpID_t, statID_t, flags)
–
–
Uses flags L and W to indicate which counter
Returns the counter valus and a timestamp in NBO
Set stats counter to 0
• retCode_t
clear_stats(fpID_t, statID_t, flags)
–
result : success or error code
X X X X X T L W
7
6
5
4
3
2
1
0
uint8_t flags:
W : {0 - Bytes, 1 - Packts}
L : {0 - PreQ, 1 - PostQ}
T : {0 - Push, 1 - Pull}
X : Don’t Care
Instruct system to create a periodic event to read a stats counter storing values in a circular buffer of size cnt.
•
handle_t create_periodic(fpID_t, statID_t, cnt_t P, len_t cnt, flags)
–
–
–
–
–
uint32_t P : periodic interval in milliseconds
uint16_t cnt : Number of samples to keep in a history buffer
flags : One of {push, pull}. In the push model the substrate will periodical send the requested stats data to a registered UDP port (see
set_callback()) . The pull model requires the VM to explicitly issue the get_periodic command (though it need not be periodic).
uint32_t handle : 32-bit opaque handle associated with periodic operation.
returns the last cnt values read
Delete period event allocated from a call to create_periodic
•
retCode_t delete_periodic(fpID_t, handle_t)
–
Deletes periodic event and removes callback state.
Register a UDP port number for the RMP to call periodically (implements the push model). Results sent every cnt*P msecs.
•
retCode_t set_callback(fpID_t, handle_t, ipPort_t)
–
–
–
–
callback (push) model for stats collection.
statRec_t vector periodically sent to the UDP port xport using the client VM’s control IP address
If xport is 0 then the callback is canceled.
We need to verify that user has this port already open so we don’t risk flooding an unsuspecting vm.
Returns the cnt element stats buffer created from a call to create_periodic (this implements the pull model)
•
statRec_t* get_periodic(fpID_t, handle_t)
–
Returns a list of 32-bit statRecs (count plus time stamp)
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
15
Accessing Code Option Data
• Each code option is expected to have a block of SRAM allocated for its
exclusive use.
• VM control application may read/write to any location within this block
• Specific code options may provide libraries to manipulate control structures
located in SRAM.
• Common types
– offset : Byte offset within fastpath instance’s SRAM block. The substrate control
software verifies offset. Type uint32_t.
– uint8_t data[cnt] : byte array of data to read/write.
• rmpCode_t mem_write(fpID_t, cnt_t offset, data)
– [len] : Number of Bytes in data vector. A library function should include a
length field, however one is not required in the substrate message since it may be
determined from the overall message length
• data mem_read(fpID_t, cnt_t offset, cnt_t)
– len : Number of Bytes to read from SRAM. The substrate control software must
verify the read does not access memory outside of the instances block. it is not an
error to set len = 0, in this case no data is returned.
– Returns a byte array of data (which may be of 0 length) or error.
Fred Kuhns - 3/22/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
16