ReGS - Reporting Grid Services
IBM Haifa Research Lab:
Yariv Aridor, Oleg Frenkel, Dean Lorenz, Benny Rochwerger
IBM T.J. Watson Research Center:
Bill Horn
IBM Software Group:
Hany Salem
IBM Server Group:
Jim Warnes, Dave Elko
IBM Haifa Research Lab.
Reporting as Persistent Messaging
Producers generate log messages
Consumers receive/retrieve log messages
H Problem determination (synchronous or post-mortem)
L
Store for messages
Filtering
Message selection criteria
Persistent
Queriable repository for persistent log messages
Staging
Filter
New services
R
Monitoring (events)
Accounting, auditing, trace
Transactional logging ?
...
Message
Producer
Persistent
Stager
Pub/sub message queue
Message
Consumer
The Reporting Services
Producer
Filter Grid Service
HBasket Factory
R
Creates baskets for the message consumer
Filter Service
Filters messages for each basket
L Basket Registry
Producer Registry
Allow consumer to find what types of messages exist
Basket Grid Service
Message repository
Configurable behavior (eg size, deletion policy)
Supports two modes of accessing messages
Basket Service
query
notification
Query via Service Data (synchronous)
Pub/sub via NotificationSource Topics (asynchronous)
Consumer
What's in a basket?
Filtering Rule
Message selection criteria
H
Producer
Predicate on message fields
May be based on application specific fields
RFiltering done by filtering service
Message Repository
Filter service
LMessage stored in basket's service data
Support queries (through FindServiceData)
Storage mechanism is implementation specific
Files / Views on a Database / Entity Beans / Other
Basket Profile
Notification topics and constraint types
Filtering
Rule
Based on message flow rather than message content
Every x messages, every n minutes, none (wait for polling)
Basket characteristics
Owner, ACL, size
Lifecycle
eg, how long we keep a basket after producer dies
Message policies
Message
Repository
Basket
Profile
Deletion
Based on any time field: creation / arrival / delivery / expiration
Overflow
Which messages to delete when out of space
Consumer
ReGS Architecture
Producer
Producer
Producer
H
R
Filtering Service
L
Filtering
Rule
Filtering
Rule
...
Filtering
Rule
- Message types?
- Create Basket
Basket
Service
Basket
Service
Query
Notification
Rule
Consumer
...
...
Basket
Service
Consumer
The "Recycle" Basket
Motivation
HPlace holder for all recent messages
R Allow consumer to access history
even if they didn't define a basket in
advance
L
Has no filter
Is a basket
Producer
Producer
Filtering Service
Filtering
Rule
Has a profile
Message deletion policy
E.g., keep messages for 24 hours
Access control
Queriable
Can be cloned
New baskets can be populated by
filtering the recycle basket content
Cloning applicable to any basket
Recycle
Basket
Service
Basket
Service
Query
Consumer
The "System" Basket
OGSA
Producer
Wrapper for existing logging systems
HA basket for "system" related events
R Repository implemented using
Legacy
Producer
Filtering
Service
platform-specific logging mechanism
L way interaction
Two
OGSA consumers can access the system
logs through the basket interfaces
Filtering
Rule
Access to data generated by legacy
producers
OGSA producers can export system related
errors to system logs
Filtering rule selects these messages
Legacy consumers can access data
generated by OGSA producers
System
Basket
Query
NT
Events
OGSA
Consumer
Legacy
Consumer
ReGS "Port-type Diagram"
H
GridService
Factory
Registry
+findServiceData()
+createService()
+registerService()
+unregisterService()
NotificationSource
+subscribeToNotificationTopic()
NotificationSink
+deliverNotification()
R
L
-producers
0..*
Producer
+setTraceLevel()
Filter Service
+writeMessage( priority, category, msg )
-baskets
Basket Service
1..* -filteringRule
-messages
-profile
Consumer
+changeProfile( newProfile )
+deleteMessages()
+markMessages( filter, marker )
Basket portType
Operations
Producer
Filtering Service
F
Basket Service
Q
N
Consumer
Basket
ServiceData
SystemBasket
Reporting Components and Interactions
H
R
L
Producer
Producer
Producer
Producers
Change
Granularity
Producer
Producer
Producer
Consumers
Create Basket
Find
Write Message
Filtering Service
Write Message
Publish
Subscribe
Create
Get Messages
Storage & Delivery Service
Wrapping log4j / JSR47
H
R
L
Producer
Producer
Producer
Producers
Producer
Producer
Producer
Consumers
Factory::
createService
Change
Logger.setLevel()
Granularity
NotificationSink::
deliverNotification
GridService::
findServiceData
Write
Logger.log()
Message
org.apache.log4j.Logger
Filtering Grid Service
NotificationSource::
subscribeToTopic
Create
Write Message
Appender.doAppend()
GridService::
findServiceData
Storage & Delivery Grid Service
org.apache.log4j.Appender
System
Basket
Recycle
Basket
Basket
...
Basket
Wrapping (distributed) syslog
H
Producer
Producer
Producer
Applications
R
Producer
Producer
Producer
Consumers
Factory::
createService
L
syslog()
NotificationSink::
deliverNotification
GridService::
findServiceData
Local syslog daemon
syslog protocol
NotificationSource::
subscribeToTopic
GridService::
findServiceData
Remote syslog daemon
Basket separate from Filter
H
Producer
Producer
Producer
Producers
R
L
Producer
Producer
Producer
Consumers
Factory::
createService
Change
Granularity
NotificationSink::
deliverNotification
GridService::
findServiceData
Write Message
Filtering Grid Service
NotificationSource::
subscribeToTopic
Factory::
createService
NotificationSink::
deliverNotification
GridService::
findServiceData
Storage & Delivery Grid Service
Legacy Producer
Legacy Consumer
System
Basket
Recycle
Basket
Basket
...
Basket
Producer Message Pseudo-schema
<log-record>
H <log-level> ERROR | WARNING | FATAL | INFO | DEBUG | ... </log-level>
R <category> module.component.subcomponent </category>
<common-fields>
L
<creation-time> t </creation-time>
<source> producer-handle </source>
<thread-id> id </thread-id>
<localization> key </localization>
...
</common-fields>
<application-fields>
<!-- extensibility elements -->
</application-fields>
</log-record>
Basket Record Pseudo-schema
<basket-record>
H <basket-fields>
R
L
<expiration-time></expiration-time>
<read marks></read marks>
</basket-fields>
<log-record>
...
</log-record>
<basket-record>
Basket Profile Pseudo-schema
<basket-profile>
H <message-filter>
R
<!-- Query based on engine notification topics -->
</message-filter>
L
<message-format>
<!-- XPath selection of log-record fields to store -->
</message-format>
<basket-profile>
Next Steps
Drill down into existing logging systems
HPartially done
R Transactional logging
Continue iterations on portTypes and schema
L
Query semantics for findServiceData
Relationship to other messaging efforts
Notification topics vs. filtering
GGF5 submission:
Grid Monitoring Architecture WG or OGSA WG?
END