MediaBroker++
Created: 2005/05/23
Last Updated: 2005/05/26
Coding Conventions (1)
• C++ API, C API
= Java coding convention
•
C API
–
Generic rule
–
–
–
Function name ::= { (a-z,0-9,“_”)+ }
Structure name ::= { (a-z,0-9,”_”)+ “_t” }
Variable/Parameter name ::= Function name
•
Combine words by an underscore “_”.
Coding Conventions
• Comments
– Write comments for everything!
• file, class, method, function, structure, parameter,
and variable!
– Format
• Free, but include your name (and possibly e-mail
address, too).
• C++ codes will be treated with CppDoc
(http://www.cppdoc.com/)
Version Control
• CVS
– Nate
– 1-2 week(s)
Editor
•
•
•
•
Vim
Vi
Emacs
Eclipse + CDT
Common
Data Structure
• Status structures contain dynamically
changeable/modifiable information of a MB++ entity.
Static/public information is stored in attributes structures.
• struct mb_sockaddr_t
– char *name
: represents a host name in a host byte order
– int port : represents a port number in a host byte order
• struct mb_server_stat_t
– struct mb_server_attr_t *attr
information of this server
: a pointer to the static
MB++ Server
Overview
Naming and
Lookup
jin
Naming and Lookup API
•
int mb_advertise_server ( mb_server_attr_t *attr)
•
int mb_advertise_producer ( mb_producer_attr_t *attr)
•
int mb_advertise_consumer ( mb_consumer_attr_t *attr)
•
int mb_stop_advertisement ( int ad)
•
int mb_find_server ( long timeout_msec, struct mb_server_attr_t ** attrs)
•
int mb_resolve_server (long timeout_msec, struct mb_server_attr_t *template, struct mb_server_attr_t **attrs)
•
int mb_find_producer ( long timeout_msec, struct mb_producer_attr_t ** attrs)
•
int mb_resolve_producer ( long timeout_msec, struct mb_producer_attr_t *template, struct mb_producer_attr_t ** attrs)
•
int mb_find_consumer ( long timeout_msec, struct mb_consumer_attr_t ** attrs)
•
int mb_resolve_server ( long timeout_msec, struct mb_consumer_attr_t *template, struct mb_consumer_attr_t ** attrs)
–
–
–
–
–
–
–
–
–
–
Advertise the existence of a server to a MB domain.
Advertise the existence of a producer.
Advertise the existence of a consumer.
Stop advertisement for the specified advertisement descriptor, which is returned by one of advertisement functions.
Locate MB++ servers in a MB domain in the given amount of time.
Locate an MB++ producer in a MB domain that match the template in the given amount of time.
Locate MB++ producers in a MB domain in the given amount of time.
Locate an MB++ producer in a MB domain that match the template in the given amount of time.
Locate MB++ consumers in a MB domain in the given amount of time.
Locate an MB++ consumer in a MB domain that match the template in the given amount of time.
Overview
• Naming and lookup layer of MB++ mutually advertises
entities on it. The advertisement is not necessarily be
periodic or reliable, thus we might require explicit lookup
operation between hosts.
• The attribute of an entity can change, so we need (a kind
of) the observer-observable pattern between the attribute
and the naming/lookup layer implementation. This
means when a producer/consumer/server changes a
part of its attribute, the naming/lookup layer must be
aware of the event to affect the change for later lookups
across hosts.
Figure
producer and consumer
Host 2
Host 1
server
client
notify
lookup
advertise
MB++
Naming/Lookup
Layer
advertise
lookup
advertisement
MB++
Naming/Lookup
Layer
advertisement / lookup
Data Structures
•
•
Attributes structure contains static information of a MB++ entity. Dynamically
changeable/modifiable information is stored in status structures.
struct mb_server_attr_t
–
–
–
–
•
•
struct mb_producer_attr_t
–
–
char *name
struct mb_stream_attr_t *stream
–
struct mb_server_attr_t *server
: represents the name of this producer
: represents the attributes of the stream produced by
this producer.
: represents the MB++ server which this producer is
connecting to.
struct mb_consumer_attr_t
–
•
struct mb_sockaddr_t *mb : represents a socket address where MB++ server is opening a socket
struct mb_sockaddr_t *spd : represents a socket address where D-Stampede server is opening a socket
char *name
: represents the name of a MB++ server
Bla
char *name
struct mb_stream_attr_t
–
–
long start
long end
: the (real) time in msec since the epoc when this stream started.
: the (real) time in msec since the epoc when this stream ended. If this stream is an ongoing
live, this must be 0 meaning unfinished.
MB++ Server Advertisement
•
int mb_advertise_server(mb_server_attr_t *attr)
–
Description
•
–
Parameters
•
–
attr <out> : a pointer to a buffer that includes a server attribute.
Returns
•
–
Advertise the existence of a server to a MB domain. Every MB++ server must advertise itself using this
function when it started running.
An integer representing an advertisement descriptor, or an error code (UNDEFINED yet).
Constraints
•
•
•
Advertisement descriptor is unique in each instance of Naming / Lookup Layer. In other words, the
uniqueness may not be guaranteed in a MB domain-wide.
[MEMORY] No memory allocation across functions.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
MB++ Producer Advertisement
•
int mb_advertise_producer(mb_producer_attr_t *attr)
–
Description
•
–
Parameters
•
–
attr <out> : a pointer to a buffer that includes a producer attribute.
Returns
•
–
Advertise the existence of a producer. Every MB++ server must advertise itself using this function
when it started running.
An integer representing an advertisement descriptor, or an error code (UNDEFINED yet).
Constraints
•
•
•
Advertisement descriptor is unique in each instance of Naming / Lookup Layer. In other words, the
uniqueness may not be guaranteed in a MB domain-wide.
[MEMORY] No memory allocation across functions.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
MB++ consumer Advertisement
•
int mb_advertise_consumer(mb_consumer_attr_t *attr)
–
Description
•
–
Parameters
•
–
attr <out> : a pointer to a buffer that includes a consumer attribute.
Returns
•
–
Advertise the existence of a consumer. Every MB++ server must advertise itself using this function
when it started running.
An integer representing an advertisement descriptor, or an error code (UNDEFINED yet).
Constraints
•
•
•
Advertisement descriptor is unique in each instance of Naming / Lookup Layer. In other words, the
uniqueness may not be guaranteed in a MB domain-wide.
[MEMORY] No memory allocation across functions.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
MB++ Stop Advertisement
•
int mb_stop_advertisement(int ad)
–
Description
•
–
Parameters
•
–
ad <in>
: an advertisement descriptor.
Returns
•
–
Stop advertisement for the specified advertisement descriptor, which is returned by one of
advertisement functions. If no advertisement exists for the descriptor, this returns an error code
(UNDEFINED yet).
An integer representing an advertisement descriptor.
Constraints
•
•
•
Advertisement descriptor is unique in each instance of Naming / Lookup Layer. In other words, the
uniqueness may not be guaranteed in a MB domain-wide.
[MEMORY] No memory allocation across functions.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
MB++ Server Finding (1)
•
int mb_find_server(
–
long timeout_msec,
struct mb_server_attr_t ** attrs
)
Description
Locate MB++ servers in a MB domain in the given amount of time. Returns the number of servers, or an
error code (UNDEFINED yet). If no server exists in the domain when this function is invoked, the caller
will be blocked for timeout_msec milliseconds or until a new server appears in the segment. If
timeout_msec is 0, this call immediately returns whether or not a server exists. If timeout_msec is less
than 0, this call will never timeout.
–
Parameters
•
•
–
: specifies the amount of time that the caller can wait for.
: a pointer to a buffer, malloc’ed in this function, where server attributes will be
stored.
Returns
•
–
timeout_msec <in>
attrs <out>
An integer representing the number of servers running in the domain, or an error code (UNDEFINED
yet) when an error occurs.
Constraints
•
•
[MEMORY] This function malloc’s one or more struct mb_server_attr_t. Callers must free them.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
MB++ Server Finding (2)
•
int mb_resolve_server(
–
timeout_msec <in>
template <in>
attrs <out>
: specifies the amount of time that the caller can wait for.
: a pointer to a buffer containing a template attribute.
: a pointer to a buffer, malloc’ed in this function, where server attributes will be
stored.
Returns
•
–
Locate an MB++ server in a MB domain that match the template in the given amount of time. When
one or more servers are found, the attrs buffer are allocated, otherwise NULL will be set to attrs. If no
server exists in the domain when this function is invoked, the caller will be blocked for timeout_msec
milliseconds or until a new server appears in the domain. If timeout_msec is 0, this call immediately
returns whether or not a server exists. If timeout_msec is less than 0, this call will never timeout.
Parameters
•
•
•
–
)
Description
•
–
long timeout_msec,
struct mb_server_attr_t *template,
struct mb_server_attr_t **attrs
An integer representing the number of servers in the domain that match the template, or an error code
(UNDEFINED yet) when an error occurs.
Constraints
•
•
[MEMORY] This function malloc’s one or more struct mb_server_attr_t. Callers must free them.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
MB++ Producer Finding (1)
•
int mb_find_producer(
–
long timeout_msec,
struct mb_producer_attr_t ** attrs
)
Description
Locate MB++ producers in a MB domain in the given amount of time. Returns the number of producers, or
an error code (UNDEFINED yet). If no producer exists in the domain when this function is invoked, the
caller will be blocked for timeout_msec milliseconds or until a new producer appears in the domain. If
timeout_msec is 0, this call immediately returns whether or not a producer exists. If timeout_msec is
less than 0, this call will never timeout.
–
Parameters
•
•
–
: specifies the amount of time that the caller can wait for.
: a pointer to a buffer, malloc’ed in this function, where producer attributes will
stored.
Returns
•
–
timeout_msec <in>
attrs <out>
be
An integer representing the number of producers in the domain, or an error code (UNDEFINED yet)
when an error occurs.
Constraints
•
•
[MEMORY] This function malloc’s one or more struct mb_producer_attr_t . Callers must free them.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
MB++ Producer Finding (2)
•
int mb_resolve_producer( long timeout_msec,
struct mb_producer_attr_t *template,
struct mb_producer_attr_t ** attrs
–
Description
•
–
timeout_msec <in>
template <in>
attrs <out>
: specifies the amount of time that the caller can wait for.
: a pointer to a buffer containing a template attribute.
: a pointer to a buffer, malloc’ed in this function, where server attributes will be
stored.
Returns
•
–
Locate an MB++ producer in a MB domain that match the template in the given amount of time. When
one or more producers are found, the attrs buffer are allocated, otherwise NULL will be set to attrs. If
no producer exists in the domain when this function is invoked, the caller will be blocked for
timeout_msec milliseconds or until a new server appears in the segment. If timeout_msec is 0, this call
immediately returns whether or not a producer exists. If timeout_msec is less than 0, this call will never
timeout.
Parameters
•
•
•
–
)
An integer representing the number of producers in the domain that match the template, or an error
code (UNDEFINED yet) when an error occurs.
Constraints
•
•
[MEMORY] This function malloc’s one or more struct mb_producer_attr_t . Callers must free them.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
MB++ consumer Finding (1)
•
int mb_find_consumer(
–
long timeout_msec,
struct mb_consumer_attr_t ** attrs
)
Description
Locate MB++ consumers in a MB domain in the given amount of time. Returns the number of consumers,
or an error code (UNDEFINED yet). If no consumer exists in the domain when this function is invoked,
the caller will be blocked for timeout_msec milliseconds or until a new consumer appears in the
domain. If timeout_msec is 0, this call immediately returns whether or not a consumer exists. If
timeout_msec is less than 0, this call will never timeout.
–
Parameters
•
•
–
: specifies the amount of time that the caller can wait for.
: a pointer to a buffer, malloc’ed in this function, where consumer attributes will
stored.
Returns
•
–
timeout_msec <in>
attrs <out>
be
An integer representing the number of consumers in the domain, or an error code (UNDEFINED yet)
when an error occurs.
Constraints
•
•
[MEMORY] This function malloc’s one or more struct mb_consumer_attr_t . Callers must free them.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
MB++ consumer Finding (2)
•
int mb_resolve_server(
–
timeout_msec <in>
template <in>
attrs <out>
: specifies the amount of time that the caller can wait for.
: a pointer to a buffer containing a template attribute.
: a pointer to a buffer, malloc’ed in this function, where server attributes will be
stored.
Returns
•
–
Locate an MB++ consumer in a MB domain that match the template in the given amount of time.
When one or more consumers are found, the attrs buffer are allocated, otherwise NULL will be set to
attrs. If no consumer exists in the domain when this function is invoked, the caller will be blocked for
timeout_msec milliseconds or until a new server appears in the segment. If timeout_msec is 0, this call
immediately returns whether or not a consumer exists. If timeout_msec is less than 0, this call will
never timeout.
Parameters
•
•
•
–
)
Description
•
–
long timeout_msec,
struct mb_consumer_attr_t *template,
struct mb_consumer_attr_t ** attrs
An integer representing the number of consumers in the domain that match the template, or an error
code (UNDEFINED yet) when an error occurs.
Constraints
•
•
[MEMORY] This function malloc’s one or more struct mb_consumer_attr_t . Callers must free them.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
Note for Implementation
• Use of Bonjour (a.k.a. Rendezvous from Apple) could
make implementation and its operation easier.
– http://developer.apple.com/networking/bonjour
– Bonjour limits the amount of traffic per second, so that
implementer should be aware of that.
• Maximum of 10 lookup/advertisement/update traffic per second
• I (Jin) want to avoid use of any centralized server-based
directory/lookup servers, since it brings us additional
work load for operation.
– LDAP, etc.
Client
Common
implementor: Nate
Client Common API
•
int mb_connect( struct mb_server_attr_t *server)
•
int mb_disconnect( struct mb_server_attr_t *server)
–
–
•
Counts the number of unread server commands.
int mb_read_command ( int clientDescriptor, struct mb_command_t *cmd )
–
•
Disconnects from a MB++ server’s bare socket interface.
int mb_listen_command ( int clientDescriptor, long sleepMsec )
–
•
Connects to a MB++ server’s bare socket interface.
Reads the next server command from command queue 1.
int mb_write_reply ( int clientDescriptor, struct mb_producer_status_t *status )
–
Updates a producer status stored in the status register 0.
Data Structures
•
struct mb_server_stat_t
–
The number of consumers chained to this producer
–
The chain of transformers acting on this stream
–
Business
–
What else?
•
•
•
•
long timestamp
: a long integer representing the time
int dataType
: an integer representing the type of the data.
int bufSize
: an integer representing the size of this data
void *command_reply
: a buffer containing a command or a reply
long timestamp
int dataType
int reply
Int bufSize
void *data
: a long integer representing the time
: an integer representing the type of the data
: an integer representing SUCCESS, or FAIL
: an integer representing the size of this data
: a buffer containing a producer’s/consumer’s/server’s status info.
struct mb_data_t
–
–
–
–
•
How busy the server is.
struct mb_reply_t
–
–
–
–
–
•
How many transformers are working, and what order.
struct mb_command_t
–
–
–
–
•
How many consumers are currently connected.
long start
long end
int bufSize
void *data
: a long integer representing the start (real) time of this data
: a long integer representing the end (real) time of this data
: an integer representing the size of this data
: the data
struct mb_qos_params_t
–
int priority
:an integer representing a priority value between 1 and
9. Default is 5.
Figure
-Clients Stateresource
unallocated
attributes
unadvertised
mb_{producer|consumer}_create
resource
allocated(inactive)
attributes
unadvertised
resource
allocated(active)
attributes
advertised
inactive
mb_{producer|consumer}_destroy
mb_{producer|consumer}_stop
mb_{producer|consumer}_start
mb_{producer|consumer}_destroy
active
Upstream/Downstream Queues
A producer’s case
item = readFromQueue;
If(item is a producer command){
process the command;
write a server reply:
}else{ // item is a producer reply
check the reply;
update my status;
}
If(need to dispatch command){
write a server command;
}
mb_read(stream);
upstream
PC1 PC2 PC3 PR1 PC4 PC5 PR2 PR3 PC6 PR4 PR5 PR6
SC6 SC5 SC4 SR6 SC3 SC2 SR5 SR4 SC1 SR3 SR2 SR1
downstream
item = mb_read_command;
If(item is a server command){
process the command;
write a producer reply:
}else{ // item is a server reply
check the reply;
update my status;
}
If(need to dispatch command){
write a producer command;
}
mb_write(stream);
Downstream Queue
• A producer reads server commands or replies from this
queue. This queue is FIFO ordered.
– Server replies contains the same timestamp as the
corresponding producer commands.
– The producer creates a producer reply for each server command
with the timestamp of the command.
• This queue must be read entirely by the producer before
dispatching commands through the upstream command
queue.
• Server-to-producer commands
– Find LUB data type
• the LUB of data types requested by consumers
– Find [LUB or LCD] transmission rate
• the LUB/LCD of the requested transmission rates
– Receive QoS requests from MB server
Server Command
-Check/Wait Server Command•
int mb_listen_command (
–
Counts the number of unread server commands or replies. Parameter sleepMsec can be used to control blocking/nonblocking policy. If sleepMsec is 0, this function returns as quick as possible after counting the unread server items. If it is
more than 0, it blocks as maximum as that time until valid items arrive. If it is less than 0, it blocks for unlimited time until
valid items arrive. “Well-behaved” producers read every item and write replies each time before writing a chunk of data to
their data channels. “Well-behaved” consumers read every item and write replies each time before reading a chunk of data
from their data channels.
Parameters
•
•
–
clientDescriptor <in>
sleepMsec <in>
: an integer representing a producer in “active” state.
: a long integer representing an amount of time to wait for the next command.
Returns
•
•
An integer representing the number of unread command. Can be 0.
Errors
–
–
–
)
Description
•
–
int clientDescriptor
long sleepMsec
Invalid descriptor
Server not responding
Constraints
•
•
•
•
This function must not be invoked before starting the producer or consumer.
The client descriptor is unique in a process.
[MEMORY]
[THREAD]
Server Command
-Read Server Command•
int mb_read_command(
–
Reads the next server command from downstream queue. If no command is found in the queue, this call waits for the next
command available. The producer writes a producer reply to the upstream queue for each server command.
Parameters
•
•
–
clientDescriptor <in>
cmd <out>
: an integer representing a producer in “active” state.
: a pointer to a pre-allocated command/reply buffer.
Returns
•
•
On success ?.
Errors
–
–
–
)
Description
•
–
int clientDescriptor,
struct mb_command_t *cmd
Invalid descriptor (not yet started, not a producer)
Server not responding
Constraints
•
•
•
•
•
This function must not be invoked before starting the producer.
The client descriptor is unique in a process.
[NOTE] we may need an up-call interface to process commands and replies.
[MEMORY]
[THREAD]
Producer Status
•
int mb_write_reply(
–
Write a reply, through a producer’s or consumer’s upstream queue, for a command issued by the server. This updates a
producer/consumer status stored in the server.
Parameters
•
•
–
clientDescriptor <in>
reply <in>
: an integer representing a producer in “active” state.
: a pointer to a status buffer.
Returns
•
•
On success ?.
Errors
–
–
–
)
Description
•
–
int clientDescriptor,
struct mb_reply_t *reply
Invalid descriptor (not yet created, not a producer)
Server not responding
Constraints
•
•
•
•
This function must not be invoked before starting the producer.
The client descriptor is unique in a process.
[MEMORY]
[THREAD]
Producer
design: jin
implementor: Eric
Producer API
•
int mb_create_producer ( struct mb_server_attr_t *server, char *name, void *func, struct mb_producer_attr_t *producer )
•
int mb_start_producer ( int clientDescriptor, int argc, void **argv )
–
–
•
Stop running a producer specified by a client descriptor.
int mb_destroy_producer ( int clientDescriptor )
–
•
Start running a producer specified by a client descriptor.
int mb_stop_producer ( int clientDescriptor )
–
•
Allocates resources required to run a new producer.
Destroy a producer specified by a client descriptor.
int mb_set_access_key ( int clientDescriptor, char *key )
–
Sets an access key to require consumers on connection.
•
int mb_request_producer_qos ( int clientDescriptor, struct mb_producer_qos_params_t *qos )
•
int mb_listen_command ( int clientDescriptor, long sleepMsec )
•
int mb_read_command ( int clientDescriptor, struct mb_command_t *cmd )
•
int mb_write_reply ( int clientDescriptor, struct mb_producer_status_t *status )
–
–
–
–
•
Request a certain set of QoS parameters for this producer.
Counts the number of unread server commands.
Reads the next server command from command queue 1.
Updates a producer status stored in the status register 0.
int mb_write ( int clientDescriptor, struct mb_data_t *data )
–
Write a chunk of stream to the channel.
Overview
• Producer is an MB entity that creates a stream,
transmits the stream to the server, answers
requests for lower/higher fidelity of types issued
by the server, and possibly inject transformers to
the server.
• Producer is required to create a type map that
represents relationships between types in terms
of their fidelity. The transformers should be
located between two different fidelities of types.
• Virtual Time Stamp
Figure
MB++
Type Server
MB++
server
producer command
clientDescriptor
upstream queue
producer reply
consumer
server command
data channel
downstream queue
T
producer
server reply
T
P
profiler
data channel
Per-producer resources
channelDescriptor
Producer Behavior Sequence
producer
mb_connect
mb_create_consumer
mb_start_consumer
mb_stop_consumer
mb_destroy_consumer
mb_disconnect
server
Data Structure
•
struct mb_producer_stat_t
– struct mb_producer_attr_t *attr : a pointer to the static information of this
producer
– char *securityKey
: a pointer to an array of char containing the key
– mb_qos_params_t *qos
: a structure holding the current QoS
parameters
Upstream Queue
•
•
•
•
•
The producer commands, except for “Join the system”, are written into the upstream (producer to
server) queue. The server reads producer commands from the queue in FIFO order.
Commands must be ordered with the non-over wrapping timestamp.
The server will write replies for the commands with the same timestamp as the commands to the
downstream (server to producer) queue. Note that the downstream queue will also contains server
commands dispatched to the producer, so the producer is responsible to switch the operation
based on the item’s type, namely a reply or a command.
These commands will fail if the downstream queue contains items, meaning the existence of
unprocessed server commands or replies.
Producer command
–
Join the system
–
Get server status
–
Set access rights
makes producer available for consumer to connect.
Requests the current server status related to this producer.
modify who is allowed to connect to this producer.
–
Declare type set
–
Request QoS
the set of types this producer is capable of producing, and the min/max transmission rate for each.
Issues a certain value of a QoS parameters set, so that the producer be treated with by the server nicely. The parameter set
includes priority.
–
Leave the system
makes producer unavailable.
Producer Command
-Join the System•
int mb_create_producer(
–
Allocates resources required to run a new producer, if the specified server found, and it’s not too busy, and also the
specified name doesn’t exist in the MB++ domain. Resources allocated by this function includes (1) mb_producer_attr_t,
(2)command queues, (3)status registers, (4)a data channel, and (5) an infrastructure (D-Stampede in the initial impl) thread.
Parameters
•
•
•
•
–
server <in> : a set of attributes of a server to connect to.
name <in> : a string representing the name of this producer. The name must be unique in a single MB++ domain, so that if
there is another producer who has the same name, this call will fail.
func <in> : a pointer pointing the entry to this producer’s body.
producer <out>: a pointer to a producer attribute created in this function. When this function returns, producer will contain
the specified name and server attributes. To start the producer, the caller must fill in the rest of the attributes.
Returns
•
•
An integer representing a client descriptor, or an error code.
Errors
–
–
–
–
–
–
)
Description
•
–
struct mb_server_attr_t *server,
char *name,
void *func,
struct mb_producer_attr_t *producer
There is already another producer having the same name as this one.
Server not found
Server too busy
Server not responding
No such function (func is NULL or less than 0, probably)
Constraints
•
•
•
•
•
A return value (client descriptor) represents the producer just created, so the descriptor must be mapped to the
mb_producer_attr_t during in this function.
The client descriptor is unique in a process.
This function cannot rely on Stampede’s channels/queues/registers since they are not allocated yet. So we need a
bare socket interface to accommodate this.
[MEMORY] Param producer is allocated in this function. The caller must free the memory after leaving from MB++ system.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
•
int mb_start_producer(
–
Description
•
–
Start running a producer specified by a client descriptor, by invoking the specified function with arguments. Before invoking
the function, this invokes mb_advertise_producer and other necessary functions.
Parameters
•
•
•
–
clientDescriptor <in>
argc <in>
argv <in>
: an integer representing a producer in “inactive” state.
: an integer representing the number of arguments passed to this producer
: an array of pointers passed to this producer
Returns
•
•
An integer representing a client descriptor, or an error code.
Errors
–
–
–
–
–
–
int clientDescriptor,
int argc,
void **argv )
Invalid thread (caller thread different from the creation thread)
Invalid descriptor (not yet created)
Specified producer is already started.
Server too busy.
Server not responding
Constraints
•
•
•
•
On success, the return value is equal to the parameter clientDescriptor.
The client descriptor is unique in a process.
[MEMORY] No memory allocation across functions.
[THREAD]
Producer Command
-Leave the System•
int mb_stop_producer(
–
Stop running a producer specified by a client descriptor. This causes the threads in the MB++ server stop reading/writing
through the allocated resources. But the resources are not released yet, and they are reusable. If this producer is already
stopped, this function does nothing.
Parameters
•
–
clientDescriptor <in>
: an integer representing a producer in “active” state.
Returns
•
•
An integer representing a client descriptor, or an error code.
Errors
–
–
–
)
Description
•
–
int clientDescriptor
Invalid descriptor (not yet created, or already destroyed)
Server not responding
Constraints
•
•
•
•
•
On success, the return value is equal to the parameter clientDescriptor.
The client descriptor is unique in a process.
Call the mb_stop_advertisement
[MEMORY] No memory allocation across functions
[THREAD]
•
int mb_destroy_producer(
–
Destroy a producer specified by a client descriptor. This causes the resources allocated by the producer to be released. If
the producer specified by clientDescriptor is not yet stopped, this function first stops it. If this producer is already destroyed,
this function does nothing; it is not an error.
Parameters
•
–
clientDescriptor <in>
: an integer representing a producer in “active” state.
Returns
•
•
An integer representing a client descriptor, or an error code.
Errors
–
–
–
)
Description
•
–
int clientDescriptor
Invalid descriptor (not yet created)
Server not responding
Constraints
•
•
•
•
•
On success, the return value is equal to the parameter clientDescriptor.
The client descriptor is unique in a process.
Call the mb_stop_producer internally.
[MEMORY] No memory allocation across functions
[THREAD]
Producer Command
-Set Access Rights•
int mb_set_access_key(
–
Description
•
–
Sets an access key to require consumers on connection. This function must be called before starting the producer. The key
first is transmitted to the server through the producer command channel, and then it’s successful, this function updates the
status structure of the producer. The structure will be written to the producer command queue. The MB++ server actually
changes the producer’s key, and updates the producer’s state structure in the ??? register with the new key.
Parameters
•
•
–
clientDescriptor <in>
key <in>
: an integer representing a producer in “inactive” state.
: a pointer to a pre-allocated array of char containing a key. Can be NULL.
Returns
•
•
An integer representing a client descriptor, or an error code.
Errors
–
–
–
–
int clientDescriptor,
char *key )
Invalid descriptor (not yet created)
Already started
Server not responding
Constraints
•
•
•
•
•
On success, the return value is equal to the parameter clientDescriptor.
The client descriptor is unique in a process.
Wrap the parameter into mb_command_t.
[MEMORY] Copy the string specified by key, so that the caller can free the memory after this.
[THREAD]
Producer Command
-Request QoS•
int mb_request_producer_qos(
–
Request a certain set of QoS parameters for this producer. The only parameter currently supported is priority. The priority
affects on how well the particular producer is treaded by the MB++ server. This function can be called before or after
starting the producer. The MB++ server actually changes the producer’s priority, and updates the producer’s state structure
in the ??? register with the new priority value. If the specified parameter set is semantically the same as current, this
function does nothing.
Parameters
•
•
–
clientDescriptor <in>
qos <in>
: an integer representing a producer.
: a pointer to a pre-allocated QoS parameter structure.
Returns
•
•
An integer representing a client descriptor, or an error code.
Errors
–
–
–
–
)
Description
•
–
int clientDescriptor,
struct mb_producer_qos_params_t *qos
Invalid descriptor (not yet created, not a producer)
Invalid priority value
Server not responding
Constraints
•
•
•
•
•
On success, the return value is equal to the parameter clientDescriptor.
The client descriptor is unique in a process.
Wrap the parameter into mb_command_t.
[MEMORY]
[THREAD]
Producer Command
-Declare Type Set-
• TBD
– Based on the type server API
Data Channel
• A producer writes a chunk of stream to the
channel. The chunk is asynchronously
read by the MB++ server.
• Constraints
– The producer must keep its rate requested by
the server through server commands.
Write Channel
•
int mb_write(
int clientDescriptor,
struct mb_data_t *data )
–
Description
–
Parameters
•
•
•
–
Write a chunk of stream to the channel. The producer must entirely read the downstream queue before writing the data.
clientDescriptor <in>
data <in>
Returns
•
•
On success ?.
Errors
–
–
–
–
: an integer representing a producer in “active” state.
: a pointer to a data buffer.
Unprocessed items in the downstream queue
Invalid descriptor (not yet created, not a producer)
Server not responding
Constraints
•
•
•
•
•
This function must not be invoked before starting the producer.
The client descriptor is unique in a process.
Increment the Stampede timestamp to trigger garbage collections.
[MEMORY]
[THREAD]
Miscellaneous
•
•
Get server status
Get consumers connected to this producer
Consumer
Design: jin
Implementor: Eric
Consumer API
•
int mb_create_consumer ( struct mb_server_attr_t *server, char *name, void *func, struct mb_consumer_attr_t *consumer )
–
Allocates resources required to run a new consumer, if the specified server found, and it’s not too busy, and also the specified name doesn’t
exist in the MB++ domain.
•
int mb_start_consumer ( int clientDescriptor, int argc, void **argv )
•
int mb_stop_consumer ( int clientDescriptor )
–
–
Start running a consumer specified by a client descriptor, by invoking the specified function with arguments.
Stop running a consumer specified by a client descriptor. This causes the threads in the MB++ server stop reading/writing through the allocated
resources.
•
int mb_destroy_consumer ( int clientDescriptor )
•
int mb_get_access_token ( int clientDescriptor, struct mb_producer_attrs_t *producer, char *key, int *renewalFunc )
•
int mb_release_access_token ( int clientDescriptor, struct mb_producer_attrs_t *producer, char *key )
•
int mb_setup_channel ( int clientDescriptor, int token, struct ??? *type )
–
–
–
–
Destroy a consumer specified by a client descriptor. This causes the resources allocated by the consumer to be released.
Gets a token to consume from a particular producer. Instance of param producer can be obtained using mb_lookup_producer.
Releases an access token. After calling this function, the consumer cannot consume any stream from the producer.
Request a type to a producer. The consumer invokes this function multiple times varying the token and the type corresponding to the number of
producer/type to consume from.
•
int mb_discard_channel ( int clientDescriptor, int token, int channelDescriptor )
•
int mb_request_consumer_qos ( int clientDescriptor, struct mb_consumer_qos_params_t *qos )
•
int mb_read ( int clientDescriptor, struct mb_data_t *data )
–
–
–
Distruct a channel from the producer through transformers to the consumer, which has been established by mb_setup_channel.
Request a certain set of QoS parameters for this consumer. The only parameter currently supported is priority.
Read a chunk of stream to the channel.
Overview
• Consumer is an MB entity that reads a stream from the server,
answers requests issued by the server, and possibly inject
transformers to the server.
• Consumer can create a type map that represents relationships
between types in terms of their fidelity. The transformers should be
located between two different fidelities of types. If the server
includes a transformer between given two types, and if a consumer
injects another transformer between the types, the transformer
injected by the consumer precedes the existing one just for the
consumer.
• Consumer can request a type to consumer, or a particular producer
to consume a stream from. The identifier of the producer can be
obtained via the Naming/Lookup API.
• Virtual Time Stamp
Figure
MB++
Type Server
MB++
server
consumer command
clientDescriptor
upstream queue
consumer reply
producer
server command
data channel
downstream queue
T
consumer
server reply
T
P
profiler
data channel
Per-producer resources
channelDescriptor
Consumer Behavior Sequence
consumer
server
mb_connect
mb_create_server
mb_start_server
mb_get_access_token
mb_setup_channel
mb_setup_channel
mb_setup_channel
mb_stop_server
mb_destroy_server
mb_disconnect
work!
• struct mb_consumer_stat_t
– struct mb_consumer_attr_t *attr
:a pointer
to the static information of this consumer
Command Queue 0
• The server reads consumer commands from command queue 0.
This queue is FIFO ordered.
• Consumer command
– Get access tokens
Get tokens to connect to producers. The tokens will be passed to later
connection requests.
– Connect to producer(s)
Establishes a channel between producers and this consumer.
May connect to multiple producers in one call, which indicates a multiplexed
stream
– Get the current item lag
the difference in real time between the GET_NEXT item and the GET_LATEST
item – so this should be 0 when these are the same item
– Request QoS
Issues a certain value of a QoS parameters set, so that the producer be
treated with by the server nicely. The parameter set includes priority.
– Disconnect from producers
releases a channel between producers and this consumer.
Consumer Command
-Join the System•
int mb_create_consumer(
–
Allocates resources required to run a new consumer, if the specified server found, and it’s not too busy, and also the specified name
doesn’t exist in the MB++ domain. Resources allocated by this function includes (1) mb_consumer_attr_t, (2)command queues, (3)status
registers, (4)a data channel, and (5) an infrastructure (D-Stampede in the initial impl) thread.
Parameters
•
•
•
•
–
server <in> : a set of attributes of a server to connect to.
name <in> : a string representing the name of this consumer. The name must be unique in a single MB++ domain, therefore if there is
another consumer who has the same name, this call will fail.
func <in>
: a pointer pointing the entry to this consumer’s body.
consumer <out>: a pointer to a consumer attribute created in this function. When this function returns, consumer will contain the
specified name and server attributes. To start the consumer, the caller must fill in the rest of the attributes.
Returns
•
•
A consumer descriptor representing this consumer, or an error code.
Errors
–
–
–
–
–
–
–
)
Description
•
–
struct mb_server_attr_t *server,
char *name,
void *func,
struct mb_consumer_attr_t *consumer
There is another consumer already having the same name as this one.
Server not found
Server too busy
Server not responding
not a consumer
No such function (func is NULL or less than 0, probably)
Constraints
•
•
•
•
A return value (client descriptor) represents the consumer just created, so the descriptor must be mapped to the mb_consumer_attr_t
during in this function.
The client descriptor is unique in a process.
[MEMORY] Param consumer is allocated in this function. The caller must free the memory after leaving from MB++ system.
[THREAD] Multi-thread safe. Multiple threads can call this concurrently or parallelly.
•
int mb_start_consumer(
–
Description
•
–
Start running a consumer specified by a client descriptor, by invoking the specified function with arguments. Before invoking
the function, this invokes mb_advertise_consumer and other necessary functions.
Parameters
•
•
•
–
clientDescriptor <in>
argc <in>
argv <in>
: an integer representing a producer in “inactive” state.
: an integer representing the number of arguments passed to this producer
: an array of pointers passed to this producer
Returns
•
•
An integer representing a client descriptor, or an error code.
Errors
–
–
–
–
–
–
int clientDescriptor,
int argc,
void **argv )
Invalid thread (caller thread different from the creation thread, or not a consumer)
Invalid descriptor (not yet created)
Specified consumer is already started.
Server too busy.
Server not responding
Constraints
•
•
•
•
On success, the return value is equal to the parameter clientDescriptor.
The client descriptor is unique in a process.
[MEMORY] No memory allocation across functions.
[THREAD]
Consumer Command
-Leave the System•
int mb_stop_consumer(
–
Stop running a consumer specified by a client descriptor. This causes the threads in the MB++ server stop reading/writing
through the allocated resources. But the resources are not released yet, and they are reusable. If this consumer is already
stopped, this function does nothing. If this consumer is consuming one or more streams from producers, this function stops
the consumption and the dependencies to the producers. Therefore, when this consumer is restarted, it must request
consumption again.
Parameters
•
–
clientDescriptor <in>
: an integer representing a consumer in “active” state.
Returns
•
•
An integer representing a client descriptor, or an error code.
Errors
–
–
–
)
Description
•
–
int clientDescriptor
Invalid descriptor (not yet created, or already destroyed , or not a consumer)
Server not responding
Constraints
•
•
•
•
•
On success, the return value is equal to the parameter clientDescriptor.
The client descriptor is unique in a process.
Call the mb_stop_advertisement
[MEMORY] No memory allocation across functions
[THREAD]
•
int mb_destroy_consumer(
–
Destroy a consumer specified by a client descriptor. This causes the resources allocated by the consumer to be released. If
the consumer specified by clientDescriptor is not yet stopped, this function first stops it. If this consumer is already
destroyed, this function does nothing; it is not an error.
Parameters
•
–
clientDescriptor <in>
: an integer representing a consumer in “active” state.
Returns
•
•
An integer representing a client descriptor, or an error code.
Errors
–
–
–
)
Description
•
–
int clientDescriptor
Invalid descriptor (not yet created, or not a consumer)
Server not responding
Constraints
•
•
•
•
•
On success, the return value is equal to the parameter clientDescriptor.
The client descriptor is unique in a process.
Call the mb_stop_producer internally.
[MEMORY] No memory allocation across functions
[THREAD]
Consumer Command
-Get Access Tokens•
int mb_get_access_token(
–
Description
•
•
–
–
Gets a token to consume from a particular producer. Instance of param producer can be obtained using
mb_lookup_producer. This function must be called after starting the consumer.
A token returned by this function expires after a certain period of time, so that the consumer is required to give a function
that decides whether to renew that or not. That function returns 1 to renew, or 0. If param renewalFunc is NULL, the token
will be renewed automatically until this consumer explicitly invokes mb_release_access_token, or until this consumer is
destroyed.
Parameters
•
clientDescriptor <in>
: an integer representing a consumer in “active” state.
•
Producder <in>
: A pointer to the data that specified a producer
•
•
key <in>
renewalFunc <in>
: a pointer to a pre-allocated array of char containing a key. Can be NULL.
: a pointer to a function that renews the token. This function will be invoked periodically when
the token expires.
Returns
•
•
An integer representing an access token, or an error code.
Errors
–
–
–
–
int clientDescriptor,
struct mb_producer_attrs_t *producer,
char *key,
int *renewalFunc
)
Invalid descriptor (not yet started)
Producer not exist
Server not responding
Constraints
•
•
•
The client descriptor is unique in a process.
[MEMORY] Copy the string specified by key, so that the caller can free the memory after this.
[THREAD]
•
int mb_release_access_token(int clientDescriptor,
struct mb_producer_attrs_t *producer,
char *key )
–
Description
•
–
–
Releases an access token. After calling this function, the consumer cannot consume any stream from the producer. If a
stream is ongoing, all the resources (such as transformers) allocated for the stream is released.
Parameters
•
clientDescriptor <in>
: an integer representing a consumer in “active” state.
•
Producder <in>
: A pointer to the data that specified a producer
•
key <in>
: a pointer to a pre-allocated array of char containing a key. Can be NULL.
Returns
•
•
An integer representing an access token, or an error code.
Errors
–
–
–
–
Invalid descriptor (not yet started)
Producer not exist
Server not responding
Constraints
•
•
•
The client descriptor is unique in a process.
[MEMORY]
[THREAD]
Consumer Command
-Connect to producers•
int mb_setup_channel(
–
Request a type to a producer. The consumer must have a valid token. The consumer invokes this function multiple times
varying the token and the type corresponding to the number of producer/type to consume from. This function sets up all the
channels from the producer-side through transformers to the consumer-side, but doesn’t start reading the producer-side
channel.
Parameters
•
•
•
–
clientDescriptor <in>
token <in>
type <in>
: an integer representing a consumer in “active” state.
: a valid token obtained via mb_get_access_token
: a type of data to consume
Returns
•
•
A channel descriptor on success, otherwise an error code.
Errors
–
–
–
–
)
Description
•
–
int clientDescriptor,
int token,
struct ??? *type
Invalid descriptor (not yet started)
Producer not exist
Server not responding
Constraints
•
•
•
•
The client descriptor is unique in a process.
The channel descriptor is unique in a MB++ server.
[MEMORY]
[THREAD]
Consumer Command
-Disconnect from producers•
int mb_discard_channel(
–
Description
•
–
–
Distruct a channel from the producer through transformers to the consumer, which has been established by
mb_setup_channel.
Parameters
•
clientDescriptor <in>
: an integer representing a consumer in “active” state.
•
token <in>
: A valid token obtained via mb_get_access_token
•
channelDescriptor <in>
: an integer representing a channel descriptor returned by mb_setup_channel.
Returns
•
•
The client descriptor on success, otherwise an error code.
Errors
–
–
–
–
–
–
int clientDescriptor,
int token,
int channelDescriptor )
Invalid client descriptor (not yet started)
Invalid token
Invalid channelDescriptor
Producer not exist
Server not responding
Constraints
•
•
•
•
The client descriptor is unique in a process.
The channel descriptor is unique in a MB++ server
[MEMORY]
[THREAD]
Consumer Command
-Request QoS•
int mb_request_consumer_qos(
–
Request a certain set of QoS parameters for this consumer. The only parameter currently supported is priority. The priority
affects on how well the particular consumer is treaded by the MB++ server. This function can be called before or after
starting the consumer. The MB++ server actually changes the consumer’s priority, and updates the consumer’s state
structure in the ??? register with the new priority value. If the specified parameter set is semantically the same as current,
this function does nothing.
Parameters
•
•
–
clientDescriptor <in>
qos <in>
: an integer representing a consumer.
: a pointer to a pre-allocated QoS parameter structure.
Returns
•
•
An integer representing a client descriptor, or an error code.
Errors
–
–
–
–
)
Description
•
–
int clientDescriptor,
struct mb_consumer_qos_params_t *qos
Invalid descriptor (not yet created, not a consumer)
Invalid priority value
Server not responding
Constraints
•
•
•
•
On success, the return value is equal to the parameter clientDescriptor.
The client descriptor is unique in a process.
[MEMORY]
[THREAD]
Data Channel
• A consumer reads a chunk of stream from the
channel. The chunk is asynchronously written by
the MB++ server.
• Get a single item (pull model) specified by:
–
–
–
–
GET_NEXT
GET_LATEST
Specify a virtual time stamp
Specify a real time
• Get a live feed of items (push model)
– Command socket request, then data channel transport
– Specify which ones to send (i.e. always GET_NEXT, always
GET_LATEST, some specified relative quantity)
• Get the real time of a data item that has already been
retrieved
Read Channel
•
int mb_read(
int clientDescriptor,
struct mb_data_t *data )
–
Description
–
Parameters
•
•
•
–
Read a chunk of stream from the channel.
clientDescriptor <in>
data <out>
Returns
•
•
On success ?.
Errors
–
–
–
: an integer representing a producer in “active” state.
: a pointer to a data buffer. This buffer is allocated
inside this function, so that the consumer must free it after use.
Invalid descriptor (not yet created, not a consumer)
Server not responding
Constraints
•
•
•
•
This function must not be invoked before starting the consumer.
The client descriptor is unique in a process.
[MEMORY] This function allocates mb_data_t, which must be free’d by the caller
[THREAD]
Type
Server
Bin
Overview
Transformer
Bin
Overview
Development Plan
Development Phases
time
•
Client-side implementation
–
•
End of July
Client-side debug
–
–
•
–
•
End of ?
Debug the server using very simple transformers (like text capitalization) with random amount
of processing latencies.
Transformer implementation
–
•
We’ll do detailed design and implementation of the server. Issues will include QoS
management, load balancing, parallel transformation, redundant transformation, intertransformer synchronization, etc.
Evaluate the server performance.
Server debug
–
End of ?
Debug the client-side implementation using “NULL” media stream.
Evaluate the client-side implementation.
Server implementation
–
•
We’ll implement client APIs, namely producers and consumers. MB++ server will be left as a
blank implementation, meaning a media stream goes through the server without any
transformation. Also, at this moment, we don’t have any detailed transformer specification nor
type definitions, so client can deal with abstract “NULL” media stream that contains void* data
as well as the real-/virtual- timestamps.
We’ll implement actual/useful transformer to realize concrete applications.
Application implementation
–
–
We’ll combine these into an application
evaluate the overall system performance.