TIBCO’s Complex Event Processing (CEP) Offering TIBCO BUSINESS EVENTS AGENDA Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards COMPLEX EVENT PROCESSING “CEP is a set of technologies that allows “events” to be processed on a continuous basis ” COMPLEX EVENT PROCESSING Typical Complex Event Processing Area HAS Situation awareness Sense and respond Track and trace “These aspects may overlap in actual business situations” SITUATION AWARENESS KNOWING the state of product, person, document or entity of interest at any particular time Requires continuous MONITORING of events EXAMPLE The REAL TIME DASHBOARD indicates all the performance indicators for a runtime production process. SENSE AND RESPOND DETECTION of significant fact about the product, person, document or entity of interest and RESPOND accordingly To achieve this, SYSTEM performs a) MONITORING OF EVENTS b) DETECTION OF SOMETHING SIGNIFICANT c) EXECUTION OF REQUIRED RESPONSE EXAMPLE Monitor credit card usage and detect that the credit card is being used consecutively at locations that are too far apart for real time person to business transactions. Fraud is detected and the transactions are denied. TRACK AND TRACE TRACKING the product, person, document or entity of interest over time TRACING pertinent facts like location, owner or general status EXAMPLE Tracking events from an RFID-enabled inventory control system where at any point in time you need to know the status of the delivery of goods at a particular location BUSINESS EVENTS MAJOR COMPONENTS TIBCO Administrator TIBCO Designer WORKBENCH Project Files Browser EAR File STATE MODELER TIBCO Designer RUN TIBCO Business Events Server Runtime Data Store WORKBENCH TIBCO Administrator Browser Project Files MODEL DRIVEN APPROACH IN BUSINESS EVENTS “ TIBCO Business Events enables CEP problems to be solved through a MODEL DRIVEN approach, in which the developer defines the event, rule, concept(class) and state models which are then compiled so that run time incoming events are processed efficiently as possible “ Describes INPUTS in BE Describes BEHAVIORAL MECHANISM in BE Describes DATA in BE Describes STATES, TRANSITION & CONDITIONS in BE AGENDA Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards BUSINESS EVENTS PALETTES Business Events State Modeler Palette mainly consists of : Business Workbench Palette mainly consists of : a) State Events Machine a) b) Channel Call State– Destination Machine b) Simple Concept c) c) Composite Concept View d) d) Event & Time Event e) Simple Concurrent e) Score Card f) Annotation Business Events Activities Palette mainly consists of : f) Set –Event Rule g) End a) Rule Receive g) b) Rule SendFunction Event c) Wait for Event d) Invoke Rule Function e) Rule Service Provider Configuration AGENDA Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards CHANNELS & DESTINATIONS “CHANNELS represent physical connections to a resource” DESTINATIONS CHANNEL MESSAGES “DESTINATIONS in a channel represent listeners to MESSAGES from that resource” ARCHITECTURE : CHANNELS & DESTINATIONS Channel Subject: orders Destination TIBCO EMS orders Default Event map to new_order Deserialization Subject: credit Default Destination Event credit_timeout credit Serialization TYPES OF CHANNELS TIBCO Rendezvous Channels JMS Channels Connects BE to RV sources & sinks. Connects BE to EMS sources & sinks. Local Channels Connects multiple rule sessions at runtime. AGENDA Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards Example WHAT IS AN EVENT? ACTIVITY that happens. EXAMPLE: Kicking a ball. Instance In Complex Event processing the term Event is overloaded. It means the definition of object that represents the activity (Event type) and also an Instance of that event type. WHAT HAPPENS WHEN AN EVENT INSTANCE IS CREATED? 1) Memory Assertion Events that are created from incoming messages, are automatically asserted into working memory. At runtime, event instances that are created using rules are not automatically asserted into working memory. You must explicitly assert such events, for example using the Event.assertEvent() function. Automatic Business Events External Environment Working Memory Create Event instance Explicit WHAT HAPPENS WHEN AN EVENT INSTANCE IS CREATED? 2) Memory Acknowledgement Depending upon the object management type BusinessEvents acknowledges the message. Some messages do not require acknowledgement. e.g.: Reliable Rendezvous messages. TYPES OF EVENTS SIMPLE EVENT Life of an Event Time To Live (TTL) Event Expired! Zero After completion of first RTC Cycle. Positive After specified time period has elapsed. Negative Does not expired. Must be explicitly consumed. Expiry Action -Any Action(s) possible to define in Rule language, to take when a simple event expires. -Expiry actions can be inherited from event’s parent. -If an event is explicitly consumed in the rule, BusinessEvents does not execute the expiry action. SIMPLE EVENT Event Inheritance/Inherits from Parent Event can NOT have payload. Can NOT have distinct properties with same name. Expiry Actions set in child Event overrides the parent event Expiry Actions. SIMPLE EVENT- HOW THE EVENT DATA IS CARRIED? Properties has data types. e.g.: String,int,boolean Payload has complex data structures. e.g.: XML Schema Extended Properties are used internally & it is reserved for future release. TYPES OF EVENTS TIME EVENTS TIMERS, that are used to trigger rules. Repeat Every -Creates a new time event after every specified time interval. -Creates specified number of events at each time interval. Rule Based -ScheduleTimeEventName() function is used inside Rule, to create a new time event. TYPES OF EVENTS ADVISORY EVENTS Advisory Events are asserted into the memory when certain conditions occur. It has attributes for category, type & message. Exception Category: Exception Type: Exception class name Message: Message in class. Engine Activated Advisory Events Category: Engine Type: engine.primary.activated Message: Engine <EngineName> activated. BusinessEvents-ActiveMatrix BusinessWorks Integration Category: Engine Type: INVOKE BW PROCESS Message: Error message from failed BW Process. AGENDA Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards CONCEPTS Concepts are descriptive entities similar to the object-oriented concepts of a class. Department Name Code Manager Employee Event Code Manager Employee -Concepts are automatically asserted into working memory when created, except when concepts are returned by Database query & in the context of in-process integration. Working Memory Depar tment Car Pen Insert.createInstance() Insert.deleteInstance() -Concepts needs to be explicitly deleted. CONCEPTS Historical Values for a concept property are kept in a ring buffer. CONCEPTS History Policy 0—0—0—36—25—10—10—0 CONCEPT RELATIONSHIP CONCEPT RELATIONSHIP Customer Refers Car Wheel CustomerID OrderID OrderDetails Color Vendor Make Model Engine Wheel:Contains Diameter Vendor Rim Type Material Inherits Car Contains Bike Wheel Car Wheel ----- ----------- --------------- Car Car Wheel Car Wheel Motorbike AGENDA Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards RULES “A Rule includes a declaration of entity types, one or more separate conditions , which evaluate to true or false, and an action, which is eligible to execute only when ALL the conditions evaluate to true” RULE Rule Priority = Declaration of Entity Types + Conditions + Actions “ Use priorities prudently” A TYPICAL RULE RULE SETS A rule set is a container for rules. All rules exist within a rule set. Grouping rules into rule sets enables you to deploy a selection of rule sets in a Business Events Archive (BAR). EXAMPLE : As seen in the picture, different colored pebbles are contained by different mesh bags. So mesh bag becomes the rule set that contains pebbles which are the rules. RULE FUNCTIONS Function for a use at a project level. Not contained in a rule set. Mainly 4 types of rule functions Event Preprocessor Functions Start Up Functions Virtual Functions Shut Down Functions AGENDA Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards SCORECARDS Serves as a static variable which is available throughout the project. Unlike concepts and events, each scorecard resource is itself a single instance. Use a scorecard resource to track Key Performance Indicators (KPI) or any other information. AGENDA State Modeler Database Concepts Out & In Process Integration STATE MODELER UML-compliant application. Initiation Used to model the life cycle of an instance of concept. To model the life cycle a state machine resource is used. Within a state machine resource you can configure the states and transitions. Example:Project Life Cycle. Classify Orders. Closure Project Life cycle Execution Defines transition from state to state based on rules that apply. Planning STATE MODEL Each State model begins with start state and ends with one or more end states . Between these states may be simple, composite and concurrent states connected by transitions. STATE MACHINE RESOURCE Exists Within one At most one Instance of Main state Machine State calls Machine Call State machine MODELING ORDER PROCESS Accounting performs credit check Customer passes Credit check Customer places Order Warehouse checks Inventory Item sent for Shipment Warehouse finds the item in stock Customer receives the Item MAIN STATE MACHINE Parent State Machine CALLS CHILD STATE MACHINE Child State Machine State Machine Recursive calls to state Machines not Allowed! STATE MACHINE STATES STATES START Entry Action Entry Action Exit Action END SIMPLE Exit Action COMPOSITE STATE × Composite States are like nested folders. × Composite States can contain simple states, other composite states and concurrent states. Credit Check (Composite State) Fulfillment Process COMPOSITE STATE × Composite States are like nested folders. × Composite States can contain simple states, other composite states and concurrent states. Credit Check (Composite State) Fulfillment Process FAILS Composite State State1 State2 FAILED CONCURRENT STATE × Allows multiple state flows to operate at one time. × A state machine cannot exit a concurrent state until all its region have finished processing. × Can contain composite , simple states. Multiple processing lanes AGENDA State Modeler Database Concepts Out & In Process Integration DATABASE CONCEPTS Database Concepts enable you to manipulate the database using a rule or rule function. Business Events DB Import Utility Tables Views Insert Update Delete Query DB Concept Object to Relational Mapping AGENDA State Modeler Database Concepts Out & In Process Integration OUT-OF-PROCESS ACTIVE MATRIX BUSINESS WORKS INTEGRATION Enables you to send and receive Business Events events in an Active Matrix Business Works engine. Both engines run in separate JVMs. Business Events Engine JMS/RV channel Enterprise and External Resources Business Works Engine OUT-OF-PROCESS ACTIVE MATRIX BUSINESS WORKS INTEGRATION ACTIVITIES IN-PROCESS ACTIVE MATRIX BUSINESS WORKS INTEGRATION Enables you to integrate ActiveMatrix BusinessWorks and Business Events functionality in one JVM. Enables each product to take advantage of the strengths of the other product. ActiveMatrix BusinessWorks can use Business Events as a light-weight rules engine. BusinessEvents can use transports available in ActiveMatrix BusinessWorks. JVM ActiveMatrix Business Business Events Works All input and output is done through Business Works. The BusinessEvents engine cannot communicate with resources outside of the BusinessWorks container. Less management overhead Simplifies maintenance All input and output is done through Business Events. The BusinessWorks engine cannot communicate with resources outside of the BusinessEvents container. BUSINESS WORKS FEATURES Passes concept/event /object Business Works process Uses To call Business Events Rule Function Invoke Rule Function Activity Rule service Provider Configuration (Required only when BW container) Used to identify the location of Business Events application at runtime BUSINESS EVENTS FUNCTIONS BusinessWorks.startProcess() BusinessWorks.invokeProcess() -- Invokes a BusinessWorks process in -- Invokes a BusinessWorks process in asynchronous mode and returns the process ID. synchronous mode. -- Upon completion, the BusinessWorks process -- Waits for completion of the process before passes an event to the rule function specified in an returning to the rule or rule function. argument of -- Starts the process engine startProcess(). BusinessWorks.init -- Starts the process engine if it is not already started. -- Returns an event, or null. if it is not already started. () (Optional) BusinessWorks.cancelProcess() -- Cancels the specified BusinessWorks process. -- Useful for canceling a long running job . -- Used with BE container only. -- Initializes BW Engine. BusinessWorks.shutdown() (Optional) --Shuts down the BusinessWorks process engine. DESIGN CONSIDERATIONS RELATED TO CONTAINER AGENDA Introduction to Object Management And Fault Tolerance In Memory Object Management Persistence Object Management Cache Object Management OBJECT MANAGEMENT OM is used to manage the state of Ontology objects (concepts, state machines, scorecards, simple events and time events) that are created and used by each Rete network. Three main Options for Object Management In Memory Persistence Cache FAULT TOLERANCE Fault tolerance mechanism differs with each of the OM options. In Memory FT is provided at the engine level. Configuration uses various engine properties to define primary and secondary engines. Persistence Fault tolerance requires a custom solution. Cache FT is provided at the inference agent level. With multi-engine features enabled, fault tolerance and load balancing are provided by the same set of features. OVERVIEW Default option Persistence of Objects : Enables objects to be available for reuse, either in memory caches or in databases. Data Recovery : Ability to survive failures. Partitioned : The ability to partition the objects among multiple JVMs. Clustered : The ability to maintain multiple copies of each object in different nodes such that if one node fails, another node can take over. AGENDA Introduction to Object Management In Memory Object Management Persistence Object Management Cache Object Management IN MEMORY OBJECT MANAGEMENT Rete n/w Primary server Fault Tolerance Cluster of two engines using in memory OM Objects are managed in memory Rete n/w Total System Failure FT Secondary server NO Recovery Possible FT Cache Object state is not maintained FT FAULT TOLERANCE FOR IN MEMORY OM BusinessEvents offers priority-based fault tolerance to provide high availability of the BusinessEvents engine process. Fault tolerance provides transitioning between inactive and active states. If two servers have the same priority setting, then the server that joined the group first takes priority in determining the failover and failback order. FAILOVER BEHAVIOR(IN MEMORY OM FT) When a node fails, the node with the next highest priority assumes responsibility for that node’s work. Rete n/w Primary server Fault Tolerance Cluster of two engines using in memory OM Rete n/w FT Primary Secondary server Server FT Cache FT FAILBACK BEHAVIOR(IN MEMORY OM FT) When a node restarts , it assumes responsibility from the node with the next lowest priority. Rete n/w Last Primary server Fault Tolerance Cluster of two engines using in memory OM Rete n/w FT Primary Secondary server Server FT Cache FT STEPS TO CONFIGURE FT IN IN MEMORY OM Open the BE_HOME\bin\be-engine.tra file in text editor. Configure following properties with same values for all engines. Engine.FT.UseFT true Engine.FT.GroupName group name In each engine property file provide the unique engine name be.ft.nodename Configure the weight properties to define priorities among servers. Secondary Servers will have lowest priorities. Engine.FT.Weight integer AGENDA Introduction to Object Management Memory Object Management Persistence Object Management Cache Object Management PERSISTENCE OBJECT MANAGEMENT The PERSISTENCE object management option persists a snapshot of the working memory for each inference agent in the deployed system. Data for each inference agent is persisted to a data store at specified intervals. PERSISTENCE OBJECT MANAGEMENT CACHE “ Small cache for each inference agent ensures that currently used objects are available in memory for improved performance. “ ”YOU CAN CONTROL THE SIZE OF THE CACHE” PERSISTENCE OBJECT MANAGEMENT WORKING Provides data recovery in the case of a complete system failure. When system comes up, BE restores the working memory to the last check point state. Receives all of the previously unacknowledged messages. Data in memory at the time of failure and not yet written on this disk is LOST. PERSISTENCE OBJECT MANAGEMENT Affects performance due to disk writes required. Fault Tolerance features are not provided by BusinessEvents. Parameters such as checkpoint interval and property cache size helps us to tune performance. Can also determine how many objects to keep in the data cache, in order to manage JVM memory usage for the application for better performance. PERSISTENCE OBJECT MANAGEMENT PROPERTIES A checkpoint is the point in time at which the working memory data is written to disk. Database operations include creations, deletions. The checkpoint interval is theobject time, in seconds,updates, betweenand writes to disk. Outstanding database operation is one that is held in working memory only. It has not yet been written to the disk. Defines the maximum number of concept properties keptnumber in the JVM memory for this ruleoccurs session. When the number of outstanding DB ops exceedsthat thatare of the specified, a checkpoint When the persistence layer performs cleanup, the least recently used(LRU) properties are moved to the When objects are retracted(deleted) from the memory, they are marked with a retraction flag. persistence store, to reduce the number of properties in memory to the specified number. You can delete these objects from the persistence DB, or you can leave them in the DB. Recommended that you delete these retracted objects to avoid accumulating large numbers of Uncheck this field toobjects use theinpersistence database tothem recover unplanned system shutdown. retracted DB, but you can keep for from data mining purposes. Performance INCREASES, but data is lost in event of SYSTEM FAILURE. DEFAULT PATHS where PERSISTENCE FILES are stored BE_HOME/db/session_name : TIBCO Administrator working_directory/db/session_name : BE Engine USING CHECKPOINT INTERVAL & OUTSTANDING DB OPS OUTSTANDING DATABASE OPERATIONS DATABASE OPERATIONS AGENDA Introduction to Object Management Memory Object Management persistence Object Management Cache Object Management CACHE OBJECT MANAGEMENT Object Management refers to managing the state of ontology object instances created by each inference agent. Cache Server abcde Inference Agent Rete n/w - Manage data objects - Handles recovery CACHE OBJECT MANAGEMENT Distributed Cache Cache Server a 1 b c Cache Server 2 d e Inference Agent Rete n/w CACHE OBJECT MANAGEMENT Cache Cluster with Load Balancing requires point to point communication! Cache Server a 1 b c Cache Cluster Cache Server 2 d e Event z Inference Agent 1 Inference Agent 2 Inference Agent 3 Rete n/w Rete n/w Rete n/w CACHE OBJECT MANAGEMENT Cache Cluster with Fault Tolerance Cache Server a 1 b c Cache Cluster Cache Server 2 d e Event z Inference Agent 1 Inference Agent 2 Inference Agent 3 Rete n/w Rete n/w Rete n/w CACHE OBJECT MANAGEMENT Cache Cluster with Fault Tolerance Cache Cluster Cache Server 2 d e Cache Server a 1 b c Event z Inference Agent 1 Inference Agent 2 Inference Agent 3 Rete n/w Rete n/w AGENDA Understanding Run to Completion (RTC) Cycle & Conflict Resolution Startup & Shutdown Rule Functions Event Preprocessor Rule functions Virtual Rule Functions Rule Analyzer & Debugger Rule Profiler UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE External action changes working memory BusinessEvents executes the first rule action on the agenda and removes it from the agenda RTC ends Is the agenda empty ? BusinessEvents refreshes the agenda Does the action change working memory ? UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE Begins when an external action causes changes to working memory One RTC cycle ends when there are no more rule actions to execute as a result of that initial change. This is also known as forward chaining, or inferencing. During one RTC no new external actions can affect the working memory. An RTC is composed of one or more conflict resolution cycles. UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE External action changes working memory UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE Working Memory changes When a message arrives at a destination, the working memory changes. First conflict resolution cycle begins. Business Events builds the agenda Business Events examines all rules that are newly true because of the change to working memory and compares them with rule dependencies. The agenda’s entries are ordered according to rule priority and other criteria. UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE External action changes working memory BusinessEvents executes the first rule action on the agenda and removes it from the agenda BusinessEvents refreshes the agenda Does the action change working memory ? UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE Business Events executes the first rule on the agenda and removes it from the agenda The rule action does not change working memory and Business Events executes the next rule entry in the agenda (if there is one). OR The rule action does change working memory and Business Events refreshes the rule action agenda to account for the changes. UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE Business Events refreshes the agenda Rules that have become newly true are added to the agenda. OR Rules that have become false are dropped from the agenda. OR Rules that were newly true at the last conflict resolution cycle and are still true remain in the agenda. UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE External action changes working memory BusinessEvents executes the first rule action on the agenda and removes it from the agenda RTC ends Is the agenda empty ? BusinessEvents refreshes the agenda Does the action change working memory ? UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE An empty agenda ends the RTC At some point, no more actions remain to be executed. The conflict resolution has run to completion. At the end of one RTC, the following happens Events are sent to destinations. Cache OM: Changes are saved to the cache and written to the backing store. Cache OM, cache-only cache mode: At the end of the RTC, all cache-only objects are removed from working memory. Persistence OM: One transaction is completed and saves changes (enabling rollback in case of failures). Profiler: profiler data is updated. HOW A RULE BECOMES NEWLY TRUE A rule is newly true if it has become true due to a change in working memory. A rule that was already true can also become newly true. A rule remains newly true until it is executed or it is removed from the agenda, or the RTC ends. Runs.scored = 90 NEWLY TRUE Runs.scored = 95 NEWLY TRUE ALREADY TRUE HOW CONFLICT RESOLUTION USES RULE DEPENDENCIES Before any data enters into the system, BE builds a Rete Network, which has all the rule dependencies, using the rule conditions. All dependencies in a rule are called its dependency set, which is the only thing needed to determine the truth of all the conditions. customer.name == “James”; person.name == “John”; hasAllAccess(customer); DEPENDENCY SET TESTING THE TRUTH OF A RULE’S CONDITION USING DEPENDENCY SET Business Events tests each rule’s dependency set against new set of facts. If facts match the rule dependencies, the rule action is added to the rule action agenda ORDER OF EVALUATION OF RULE CONDITIONS • Conditions that only involve one scope element FILTERS 1 • Conditions that compare two expressions using == or != EQUIVALENT JOIN CONDITIONS NON EQUIVALENT JOIN CONDITIONS 2 • Conditions involving two or more scope elements other than equivalent joins 3 AGENDA Understanding Run to Completion (RTC) Cycle & Conflict Resolution Startup & Shutdown Rule Functions Event Preprocessor Rule functions Virtual Rule Functions Rule Analyzer & Debugger Rule Profiler STARTUP & SHUTDOWN RULE FUNCTIONS Configured to execute during normal system startup and shutdown. Used to initialize the system & perform more “expensive” operations so that the system is more efficient at runtime. TAKE NO ARGUMENT VALIDITY ACTION STARTUP & SHUTDOWN RULE FUNCTIONS Configure these functions in the BUSINESS EVENTS ARCHIVE