Basic

CA-ESP Workload Automation

Course – Agent Overview

What You Will Learn

-

What is an ESP System Agent

-

Basic Agent Architecture

-

How ESP System Agents Interact With CA ESP

Workload Automation Engine

-

Basic Configuration Options

-

Defining non-mainframe workload

-

Value that ESP Agents can provide

-

ESP Agent for Databases

2

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Product Components

3

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

ESP Agents

-

Lean, non-invasive conduits (Approx 50MB disk space)

-

Extends CA ESP Workload Automation’s solutions across a variety of operating systems and ERP environments

-

Unlimited scalability, throughput

-

Each platform has unique ESP Agent

-

Manage through a single point of control, will be integrated with CA Job Management products (CA-7, CA-Autosys)

4

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

ESP System Agents - Scalability

-

Volumes and Concurrency

-

Large number of concurrent processes

-

Lab tested at 1,000 concurrent jobs, largest known field implementation 500 concurrent jobs

-

Highly scalable

-

Can keep up with volumes manager can handle

5

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

What is an ESP Agent?

-

Processes work

-

Notifies Manager of job status

-

Started

-

Running

-

Complete/Failed

-

Stores output from jobs in a spool file

-

Allows users to control workload

-

Waits for work

-

Persistence

6

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

7

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Workload Objects

Distributed OS Integration

File

Management

OS Resource Integration

ERP Integration

Agent Monitoring

Database Integration

Web App Integration

Mainframe Integration

8

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Who Controls the ESP Agent?

-

Schedulers

-

Schedule jobs regardless of platform & Have control over entire enterprise

-

Operations

-

Monitor entire enterprise & Control every production job

-

System Administrators

-

Install ESP Agent & Maintain ESP Agent

9

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

ESP System Agent Architecture

-

ESP System Agents for UNIX Based Platforms are started processes.

-

ESP System Agents for Microsoft Windows are installed as

Windows Services

-

All ESP System Agents Utilize SUN Microsystems JAVA

Runtime Environment as the Base Architectural Component

-

All ESP System Agent Functions are Created as JAVA Plug-

In’s

10

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

ESP System Agent Architecture

Spawns &

Tracks

Submitted

Commands

& Scripts

Job Execution

JAVA Plug-In

Processes &

Monitors File

Trigger &

FTP

Requests

File

Monitoring

JAVA Plug-In

Processes

Machine

Resource

Monitoring

Requests

Object

Monitoring

JAVA Plug-In

Processes

EJB & JMS

Publish

Subscribe

Requests

Processes

TCP/IP

Communications

& SNMP

Requests

J2EE

Execution

JAVA Plug-IN

Agent

Management

JAVA Plug-In

SUN Java Runtime Environment Base Code

11

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Communication

-

ESP Managers and ESP System Agents communicate asynchronously using message queues.

-

ESP Managers and Agents communicate by sending Automated Framework Messages or AFMs.

-

Most Agent commands deal with the control of these AFMs.

12

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Communication

-

ESP Managers and ESP System Agents have

Sender and Receiver Ports.

-

The receiver listens on a predefined TCP/IP port.

-

When the sender has messages to transmit, it connects to the receiver’s port, sends the messages, and then closes the connection.

13

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Communication

-

ESP Managers and Agents have sender and receiver ports.

-

Each Agent has one dynamic sender port and one receiver port.

-

The ESP Host can have multiple receiver ports (for example, to separate encrypted and non-encrypted message traffic) and has one dynamic sender port for each connected Agent.

14

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

ESP Agent Basic Communications

15

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Communication

-

Automated Framework Messages include:

-

ESP System Agent Name

-

Command or Script to Execute

-

UserId to Execute the Workload Under

-

Arguments for the Script or Command

-

Environment Variables

-

Job Execution Status

16

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Communications

-

Sample AFM Messages:

20050908 12060405+0500 CM_DE54 DLEIGH_WIN_AGENT

WINNT1/WLMDEMO5.20/MAIN State EXEC SetStart Status(Executing at

DLEIGH_WIN_AGENT) Jobno(1860) User(DLEIGH_WIN_AGENT) Host(DLeigh)

20050908 12063416+0500 CM_DE54 DLEIGH_WIN_AGENT

WINNT1/WLMDEMO5.20/MAIN State COMPLETE Cmpc(0) SetEnd

User(DLEIGH_WIN_AGENT) Host(DLeigh)

20050908 12290376+0500 CM_DE54 DLEIGH_WIN_AGENT

CYBDL01K/WLMDEMO3.3/MAIN State SUBERROR Failed SetEnd

Status(Command file not found) Cmpc(20007) User(DLEIGH_WIN_AGENT)

Host(DLeigh)

17

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Basic Configuration

-

All ESP Agent Configuration Settings are stored in a single file called agentparm.txt

-

This file is created by the installation process and can be manipulated using any text editor

-

This Configuration File Defines the Name of the

Agent, Ports Used, and Other Configurations that

Pertain to Other Functions (FTP, J2EE, SNMP, etc.)

18

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Basic Configuration

-

Sample agentparm.txt File:

# ESP System Agent for Microsoft Windows parameters

#

# Log

# log.level=5 log.maxsize=1024000

#

# Agent name

# agentname=DLEIGH_WIN_AGENT

#

# Communications

# communication.managerid_1= CM_DE54 communication.manageraddress_1= lparc communication.managerport_1= 6666 communication.monitorobject_1=DLEIGH_WIN_AGENT/AGENTMON1.0/MAIN communication.managerid_2= ESPRESSO_CALYPSO_47500 communication.manageraddress_2= calypso communication.managerport_2= 47507 communication.monitorobject_2=DLEIGH_WIN_AGENT/AGENTMON2.0/MAIN

19

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Basic Configuration

-

Sample agentparm.txt File: communication.managerid_3= ESPRESSO_ELVIS_7500 communication.manageraddress_3= elvis communication.managerport_3= 7507 communication.monitorobject_3=DLEIGH_WIN_AGENT/AGENTMON3.0/MAIN communication.inputport= 9900 communication.prefixlevel=2

#

# Security

# security.filename=D:/Cybermation/ESP System Agent R6.1/security.txt

security.keystorage=D:/Cybermation/ESP System Agent R6.1/keys.txt

security.cryptkey=0x 3132333435363738 security.level=off

#

# Initiators

# There will be separate line for each pair of <class, number of initiators>

# initiators.class_N, where N is number running from 1 to ...

# Class is user definable, (should be the same here and in AFM)

# Soft shutdown mode waits for all jobs to be completed

# initiators.class_1=Default,1000

20

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Basic Configuration

21

-

Sample agentparm.txt File:

# Persistence

# persistence.gcinterval=10000 persistence.level=2

#

# Core parameters (for plugins)

# core.address=localhost core.port= 35800

#

# General characteristics

# spooldir=./spool

COLD_START=false

#

# Runner plugin parameters

# runnerplugin.managerport= 35801 runnerplugin.requestport= 35802

#

# Start JVMs

# oscomponent.attachjvm=true

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Basic Configuration

-

Sample agentparm.txt File:

#Path to the JRE oscomponent.javapath=D:/Cybermation/ESP System Agent R6.1/jre/bin

#type of jvm (used only if attachjvm=true) oscomponent.jvm=client

#What plugins we want to start by the core Java agent plugins.start_internal_1=runner plugins.start_internal_2=fileMon plugins.start_internal_3=objmon plugins.start_internal_4=management plugins.start_internal_5=ftp plugins.start_internal_6=j2ee plugins.start_internal_7=router

#

# SNMP specific

# management.snmp.mibfile=D:/Cybermation/ESP System Agent R6.1/cybermib.txt

management.snmp.host=DLeigh management.snmp.port=162 management.snmp.community=public

22

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

23

Basic Configuration

-

Sample agentparm.txt File:

#

# FTP specific

#

# ftp.noserver=false ftp.serverport=21

#ftp.client.ssl=true

#ftp.client.ssl.truststore=

#ftp.client.ssl.truststore.password=055A55EB863D2A5D

#ftp.server.ssl=true

#ftp.server.ssl.keystore=

#ftp.server.ssl.keystore.password=C8B98BFA6652520BC0

#

# Windows Service name

# oscomponent.servicename=ESP System Agent for Microsoft Windows R6.1

oscomponent.servicedisplayname=ESP System Agent for Microsoft Windows R6.1

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Successful Communication depends on…

Agent Information that must match…

Agentparm.txt Topology in ESP dSeries agentname communication.inputport

communication.managerid

communication.manageraddress

coomunication.managerport

security.cryptkey

Name

Agent Port number

Manager Instance Host name

DNS name or IP Address of dSeries Server

ESP Server Manager Port

Encryption key used from server to agent

24

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Your Memory

-

What Type of Workload Can an ESP System Agent

Execute?

Windows Scripts, Windows Commands, FTP Processes,

Machine Resource Monitors, J2EE Based Processes, etc…

-

How Does the ESP System Agent Communicate with an ESP Server?

Utilizing Automated Framework Messages over TCP/IP

25

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Your Memory

-

What is the Receiver Port Used for?

The receiver listens on a predefined TCP/IP port. When the sender has messages to transmit, it connects to the receiver’s port, sends the messages, and then closes the connection.

-

What Job States Can the ESP System Agent for

Windows Send to an ESP Server?

READY, EXEC, COMPLETE, FAIL, SUBERROR, &

SUBDELAY

26

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Additional Automation capabilities

Approaches for use in your environment

-

Event-Driven Workload Automation

-

“Batch” Environment Monitoring

-

Run Book Automation

28

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Event-Driven Automation

-

Trigger off new applications based on monitoring capabilities of the R7 Agent, using ESP Alerts:

-

Error message being written to a log file

-

A service coming online

-

Server’s CPU stuck at 100%

-

ESP:dSeries currently implemented as Jobs, also several monitors implemented as Event Triggers

29

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

“Batch” Environment Monitoring

-

The ability to monitor critical environmental dependencies of an ESP Application to ensure successful completion

-

Example: Find out you are going to run out of disk space hours before the server dies

-

Example: If you know you’re going to start running a job on your SAP server at 2am, and the database server for that job crashes at 10pm, why wait 4 hours for everyone to be surprised?

-

(and getting called in the middle of the night)

-

Simply another step towards better managed SLA’s

30

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Run Book Automation

-

Latest fad in IT Operations that many of you, have been doing for years

-

Focus on managing (executing, controlling, monitoring) IT Operations tasks in a controlled, workflow-style manner much like workload automation

-

Example: Shutting down certain processes on a server, running maintenance, starting back up the processes

-

You can, naturally, do much of this with your existing ESP Scheduler and

ESP Agents

31

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

FTP Workload Objects

Automating File Transfers

-

To schedule FTP workload, use the FTP_JOB job type.

-

You can automate file transfer with an FTP job. The job can use an existing FTP server or the Agent’s in-built FTP server.

-

Note: To use the Agent as an FTP server, you need to configure the Agent during installation or set the Agent parameter ftp.noserver to false. The Agent configured as an

FTP server does not support anonymous file transfers

33

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Automating File Transfers

-

In this sample, a file called /temp/cyberftp181006.txt is downloaded from a UNIX machine (rem_unix) then copied to a local machine, a Windows PC. Note that the two locations include a complete path statement. After the download is complete, the job completes in ESP dSeries:

34

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Properties for ‘Monitor’

Workload Objects

Adding Monitoring Capabilities to ESP

-

Starting with Release 6 of the ESP System Agent, new monitoring capabilities were introduced:

-

Windows

-

Windows Event Log Monitoring

-

Windows Services Monitoring

-

Windows/UNIX/OS400

-

Disk Space Monitoring

-

Process Monitoring

-

TCP/IP Address/Port Monitoring

-

CPU Monitoring

-

Text File Monitoring

-

J2EE

-

JMS Publish/Subscribe to Queues and Topics

36

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Windows Services

-

What is it?

-

This job type allows you to monitor Windows

Services on a local machine

-

Sample Definition:

37

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Windows Services

-

Service Name corresponds to the name of the Service as identified in the Services

Application

38

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Windows Services

-

Status refers to the state you with the Service to be in for the monitor to complete

-

Valid Statuses include: RUNNING, STOPPED,

CONTINUE_PENDING, PAUSE_PENDING, START_PENDING,

STOP_PENDING, EXISTS, NOTEXISTS

-

Can be coded with either WAIT or NOW. Now will return a

COMPLETE/FAIL immediately, while WAIT will remain until the condition is met or the JOB is forced complete

39

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Windows Event Log

-

What is it?

-

A job to monitor the Event Log of a Local

Windows Server. It can monitor any one of three types of logs

-

Application Log (Programs)

-

System Log (System Components, e.g. a Driver)

-

Security Log (Security Events like an invalid login, file access)

40

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Windows Event Log

41

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Windows Event Log

-

Sample Job Definition

42

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Windows Event Log

43

-

EVENTLOG values are generally Application, Security, or System

-

EVENTTYPE values may be ERROR, WARN, INFO, AUDITS, AUDITF

(AUDITS and AUDITF related to EVENTLOG Security only)

-

EVENTSOURCE value is typically generated by software vendor.

Values with spaces requires quotes

-

EVENTCATEGORY represents a classification by the

EVENTSOURCE, in this case, ‘Norton AntiVirus’ is catagorizing this as a Disk event.

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Disk Space

-

What is it?

-

Allows you to find out how much disk space has been used or is free

-

Can be expressed in MB, GB, or as a Percentage

-

Can be CONTINUOUS monitored using an ESP Alert

44

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

IP Monitoring

-

What is it?

-

It allows you to monitor specific IP addresses or IP Address/Port combinations to validate that network resources are accessible and listening ports are available

Valid Status include RUNNING and STOPPED and are monitored for immediate state NOW or monitored until condition is met WAIT

45

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

CPU Monitoring

-

What is it?

-

Allows you to monitor CPU utilization to determine success/failure criteria of your job monitor

-

Why would I want to do this?

-

Provides you with a means of ensuring that the machine has sufficient CPU available before submitting workload to it

-

If a machine is in a continuously busy state, allows you to fire off notifications

-

CPU Monitoring is also tied into Physical Resources, which can be used for physical load balancing (required HPO for ESP:mSeries)

-

How is CPU utilization measured?

-

Windows uses CPU as recorded by the system

-

UNIX uses Load Average

46

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

CPU Monitoring

-

This example completes if the CPU Usage is greater than

80%.

-

Can be used to send warnings to System Administrators.

47

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Text File Monitoring

-

Allows you to monitor INSIDE a text file for a specific string of text for a matching value

-

Monitoring has a great deal of flexibility with

Regular Expressions

48

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Text File Monitoring

-

This example monitors the file by lines for a text string, but only the first 20 lines

49

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Text File Monitoring

Example using Regular Expressions:

50

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Properties for JMS

Workload Objects

-

JMS Monitoring

JMS Subscribe

-

What is it?

-

Java Messaging Service

-

A Java-based standard for connecting to, and communicating with, message brokers such as

MQ Series, Sonic, WebLogic, or Tibco

-

Allows you to monitor a JMS Queue or Topic for a specific message that matches your criteria

52

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

JMS Subscribe

ESP Agent

-

JMSS_JOB job type

-

Subscribe to Topics or Queues

-

Filter results

Receive

Acknowledge

JMS Provider

Topic or

Queue

53

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

JMS Subscribe

54

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

JMS Subscribe

-

A few uncommon terms

-

INITIAL_CONTEXT

-

CONNECTION_FACTORY

-

JNDIUSER

55

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

ESP Agent for

Databases

ESP Database Agent

-

Provides integration to a variety of database platforms

-

Utilizes the agent plug-in architecture to adapt to the

System Agent core

-

Written entirely in JAVA and the

JDBC API

57

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

ESP Database Agent Features

ESP Database

Agent

ESP Database

Agent

ESP Database

Agent

ESP Database

Agent

SP

SP

Stored

Procedures

+

Table Monitor

SQL Updates and Queries

58

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

TRG

TRG

Data Triggers

Event Driven SQL Statements

-

Event-driven execution of SQL commands i.e. Insert, update, delete, etc.

-

Success criteria specified through regular expressions

-

Output stored to file

ESP Agent for

Databases

59

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Invoking Stored Procedures

-

Event-driven execution of stored procedures

-

Retrieve output values generated by the stored procedure

ESP Agent for

Databases

SP

SP

60

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Monitor Database to Trigger an

ESP Event

-

Monitor a table for a net change in size

i.e. number records added or deleted which satisfy a user defined condition

-

Complete or selected (columns) records can be defined

-

One-time or continuous alerts

ESP Agent for

Databases

+

61

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Database Triggers

-

Very granular notification of changes to a database table

-

Database generates the events as they occur

-

Excellent performance, with very low overhead

-

Generate an event for every record added, deleted, or modified which satisfies a user defined criteria

-

One-time or continuous alerts

ESP Agent for

Databases

TRG

TRG

62

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Database Agent Technical

Details

-

Supported databases

-

Oracle 9i & up

-

Microsoft SQL Server 2000 & up

-

IBM DB2

63

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Database Agent Examples

-

This example uses SQL to Insert a line into a Table called ‘stores’ in the ‘pubs’ database

64

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Questions?

65

Copyright ©2006 CA. All rights reserved. All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.