Slide 1
Oracle Enterprise Manager Cloud Control 12c:
Best Practices for Middleware Management
Mary Peek, Senior Principal Curriculum Developer
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Hello, and welcome to this online, self-paced course entitled Oracle Enterprise Manager Cloud
Control 12c: Best Practices for Middleware Management. My name is Mary Peek, and I’ll be
your guide for approximately the next two and a half hours of interactive lectures, videos, review
sessions, and optional demonstrations. The goal of this course is to teach you best practices
when using Oracle Enterprise Manager Cloud Control 12c for managing your WebLogic and
SOA, applications and infrastructure.
Slide 2
Using the Player
Narration and
Best Practices
Summary
Course Outline
Player Controls
Change View
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Before we begin, take a look at some of the features of this Flash-based course player. If you've
attended a similar self-paced course in the past, then feel free to skip this slide.
Let's start with the Outline. It's set up so that you can go at your own pace. Feel free to skip over
topics you already feel confident about, or jump right to a topic that really interests you, or go
back and review topics that were already covered. Just click a slide title in the outline to display
its contents. By default we'll automatically walk you through the slides without requiring you to
use the outline. Use the Transcript tab to view the audio transcript for each slide, and use the
Search tab to find specific information in the course.
Next, let's look at the Player Controls. Use these controls to Pause, Play, or move to the
Previous or Next slides. You can also use the interactive Progress Bar to fast forward or rewind
the current slide. Some interactive slides in this course may contain additional navigation and
controls.
This icon on the bottom right corner allows you to change the View of this player, for example to
go to Full Screen. You can control the view at any time using this button.
A downloadable, audio narration document for this course as well as the Best Practices
Summary are available from the Attachments button.
This course will now pause, so feel free to take some time and explore the interface. Then when
you're ready to continue, click the Next button below or alternatively click About This Course in
the outline on the left.
Slide 3
PROPERTIES
Allow user to leave interaction:
Show ‘Next Slide’ Button:
Completion Button Label:
Anytime
Show always
Next Slide
Welcome!
So, you know the title of the course, but you may be asking yourself, “Am I in the right place?”
To help you answer this question, you can access information here regarding the course
objectives, the target audience, and the prerequisites. When finished, click the Next Slide
button.
What skills will I learn?
What can you expect to get out of this course? The core learning objectives are listed here.
After this course you should be able to list the most important tasks for successful management
of WebLogic Server and SOA applications and infrastructure using Oracle Enterprise Manager
Cloud Control 12c. You should also know when to perform each task in the middleware
development lifecycle. You should be able to outline why certain management tasks are more
important than others. And you should be able to provide examples of the most important
management tasks.
Who is the target audience?
This course is intended for System Administrators and SOA Architects.
What are the prerequisites?
Before taking this course, you should have experience using Oracle Enterprise Manager Cloud
Control 12c as well as the Fusion Middleware product stack. In Particular Oracle WebLogic
Server and Oracle SOA Suite.
From now on I’ll refer to Oracle Enterprise Manager Cloud Control as just Cloud Control.
Slide 4
Why Take This Course?
What’s in it for me?
What challenges do I face on the job?
How can this help me do my job better?
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
So, why take this course? Start by asking yourself, “What’s in it for me?” You will learn Oracle
recommended best practices for managing your Fusion Middleware applications and
infrastructure using Cloud Control.
What challenges do you face on the job? Enterprise middleware applications are becoming
increasingly powerful and are also more challenging to manage. How do you keep track of all
your WebLogic Server configurations throughout the enterprise? How do you get visibility into
your Java EE applications or your SOA composites? How do you track transactions across
multiple tiers? This course will answer all those questions and more.
And, finally, how can this course help you do your job better? This course shows you what tasks
to prioritize in terms of middleware management so you will know what to do at each stage of
the enterprise application development process. Knowing these management best practices will
help your business achieve peak performance and avoid costly downtime.
Slide 5
Road Map
Plan
Set
Operational
Goals
Improve
Processes
Introducing
Middleware
Management
Track &
Control
Changes
Monitor
Proactively
Diagnose
Problems
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Let's look at how this course is structured. First, I'll give you an introduction to middleware
management at a high level – we'll look at the challenges you face in managing middleware
applications and infrastructure, I'll introduce Cloud Control, and I'll take you through the
middleware development lifecycle which is what you see here. Then each section of the course
will focus in on each stage in the lifecycle in detail.
Slide 6
Challenges in Middleware Management
•
Monitoring multiple domains across the enterprise
•
Monitoring applications across tiers
•
Tracking and maintaining configurations
•
Managing capacity to changing workload
•
Managing multi-tier transaction flow
•
Obtaining visibility into SOA services
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Today’s IT organizations are increasingly adopting Java EE, SOA, composite application, and
cloud computing that enable them to quickly connect disparate applications and fulfill ever
changing business needs. Although these applications offer unprecedented flexibility and agility,
they’re more challenging to manage.
Particularly, how do you centrally monitor multiple WebLogic domains across the enterprise?
And how do you monitor your enterprise applications when they are deployed across several
tiers and containers?
And how do you track and maintain the configurations of numerous servers and applications?
And how do you manage capacity to changing, typically increasing, workload patterns?
And how do you get visibility into, and manage transaction flow that spans shared components
and services and is deployed across several tiers?
And how do you get visibility in to the performance of SOA and Oracle Service Bus services?
And so on…
Let’s hear from two Product Managers at Oracle on these challenges, first for WebLogic Server
management and then for SOA management.
 Glen Hawkins, Director, Product Management for WebLogic Server discusses the
challenges in WebLogic Server management.
 James Kao, Director, Product Management for SOA discusses the challenges in SOA
management.
Slide 9
Introducing Enterprise Manager Cloud Control
•
•
•
•
Application performance management
JVM diagnostics
Configuration & lifecycle management
Business transaction management
•
•
•
Performance & diagnostics
Configuration & lifecycle management
Business transaction management
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
All of these middleware management challenges can lead to more business downtime, higher IT
costs, and less agility. Oracle Enterprise Manager Cloud Control helps you meet these
challenges. Cloud Control is Oracle's single, integrated solution for managing all aspects of the
Oracle Cloud and traditional IT environments and the applications running on them. It couples a
potent, top-down monitoring approach with a cost-effective automated configuration
management, provisioning, and administrative solution. This powerful combination lets you to
stay focused on business priorities, while reducing the effort and cost of managing sophisticated
applications built on Oracle WebLogic Server and Oracle Fusion Middleware.
Oracle offers several management packs that enhance the capabilities of Cloud Control for
specific purposes. In this course we'll be looking at two specific management packs – the
WebLogic Server Management Pack Enterprise Edition and the SOA Management Pack
Enterprise Edition.
The WLS Management pack provides application performance management, JVM diagnostics,
configuration and lifecycle management, business transaction management, and more for
Oracle WebLogic Server application environments.
The SOA Management pack provides , performance and diagnostics, configuration and lifecycle
management, and business transaction management, for a SOA-based environment.
So, Cloud Control, licensed with these two packs, gives you visibility across all your middleware
infrastructure and applications so you can manage them more effectively. It also reduces the
cost and complexity of middleware management, thereby increasing your ROI on middleware
investments.
Slide 10
PROPERTIES
Allow user to leave interaction:
Show ‘Next Slide’ Button:
Completion Button Label:
Anytime
Show always
Next Slide
Here you can see the Middleware Development Lifecycle. Different aspects of middleware
management are used at each stage of this lifecycle. This course will cover the most important
middleware management best practices at each stage. Click a label for more information.
Plan
In the planning stage you need to look at what your middleware management requirements are
in terms of what you are going to monitor and what operational procedures your organization
has. Here I’ll give you best practices on what parts of Cloud Control you need to install and
where you need to install them to meet these requirements.
Set Operational Goals
Before you go into production you need to set your operational goals. Here I’ll give you best
practices on defining your monitoring standards.
Monitor Proactively
Once you are in production you need to ensure that your Oracle Fusion Middleware and
WebLogic deployments are up and performing as expected. Here I’ll give you best practices on
monitoring your targets proactively so you can detect potential problems before they become
severe.
Diagnose Problems
You need to diagnose problems at all stages in the development lifecycle – through
Development, Test, QA and production but it is vitally important in production. When your
critical business applications go down or are not performing well, your business suffers. Here I’ll
give you best practices on diagnosing problems quickly and efficiently.
Track & Control Changes
Tracking configurations is probably one of the most time-consuming and difficult tasks you as an
administrator face on a daily basis. Do you know how each of your managed targets is
configured and whether they comply with your corporate or regulatory standards? Here I’ll give
you best practices on tracking and controlling configuration changes.
Improve Processes
You probably spend a large amount of time performing various lifecycle management
operations - such as installing, patching, and configuring WebLogic servers, as well as
deploying Java applications and SOA composites, and so on. Here I’ll give you best practices
on improving your processes and automating common lifecycle management operations.
Slide 11
Road Map
Plan
Improve
Processes
Set
Operational
Goals
Track &
Control
Changes
Monitor
Proactively
Diagnose
Problems
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
In the planning stage, before you even install Cloud Control, you need to look at what your
Middleware management requirements are in terms of what are you going to monitor and what
operational procedures your organization has. Here I’ll give you best practices on what parts of
Cloud Control you need to install and where you need to install them to meet these
requirements.
Slide 12
Planning Best Practices
Deploy Cloud Control universally
Expand with business-driven needs
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
So what should you plan for in terms of installing Cloud Control to meet your middleware
management requirements?
First you should deploy Cloud Control universally for plug-and-play core capabilities with
Management Agents on each of your monitored hosts. Then you expand with your businessdriven needs.
You should deploy Application, Dependency & Performance, or ADP, and JVM Diagnostics
where you need deep diagnostic visibility into your applications. And you should deploy
Business Transaction Management where you need visibility into your business processes.
Slide 13
Monitor & Manage Everything With One Console
User
User
Load
Balancer
DB
DB
Cloud Control
Console
DB
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Your current and planned environment will no doubt comprise many hosts running multiple
WebLogic Servers, the Java EE, SOA, web applications, and business processes deployed on
these servers, Oracle databases, and so on. Here you see an example production environment
with multiple databases on Exadata machines, WebLogic Servers on Exalogic machines, some
with a mixture of SOA and Java EE applications, Web services, Oracle Service Bus, and
business processes deployed to them. A load balancer is also used to improve performance.
You can, of course, use the individual product consoles to monitor the status of each of these
targets, but it becomes cumbersome to shuttle between multiple console windows and track the
performance of each of these targets using so many windows.
Cloud Control offers a solution that allows you to monitor and manage the complete Oracle IT
infrastructure from a single console. Cloud Control gives you the core infrastructure and
component management as well as the metrics, configuration, lifecycle, and operational
infrastructure.
Slide 14
Deploy Cloud Control Universally
User
User
Load
Balancer
DB
DB
OMS
OMS
Management
Repository
DB
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
So what should you plan for in terms of installing Cloud Control to meet your middleware
management requirements? First, deploy Cloud Control universally for plug-and-play core
capabilities then expand with business-driven needs.
The Oracle Management Service, or OMS, collects and stores data and renders it on the Cloud
Control Console. In large production environments, as you see here, deploy multiple OMSs on
WebLogic servers to spread the load and improve efficiency of data flow so you can meet your
high availability requirements.
Then you can use OMS to push Oracle Management Agents to all hosts that contain targets you
want monitored. Agents can be pushed in a bulk manner and don't require manual effort for
each install. An Agent is responsible for monitoring all services and components on its host.
Core monitoring needs to be deployed and available everywhere so you can use it without
worrying about scaling or overhead.
The Agents send their monitoring data to the OMS which stores the collected information in the
single management repository, a database, for future reference and analysis.
When you install Cloud Control, a set of mandatory Oracle Management plug-ins are also
deployed by default. These plug-ins include special management capabilities customized to suit
Fusion Middleware target types and work in conjunction with the OMS and the Management
Agents to monitor every Middleware target in your environment. The WebLogic Server
Enterprise Edition and SOA Management Enterprise Edition Management Packs are also
installed by default when installing Cloud Control.
Slide 15
Expand with ADP and JVM Diagnostics
User
User
ADP
ADP
ADP
JVMD
JVMD
Load
Balancer
JVMD
Deploy ADP for deep visibility
into SOA, Oracle Service Bus,
Portal & ADP applications.
DB
DB
OMS
JVMD
OMS
JVMD
ADP
ADP
Management
Repository
DB
Deploy JVMD for deep
diagnostics into Java
applications.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Once you have your Cloud Control infrastructure in place, expand it with your business-driven
needs. Cloud Control also comes with some other managers and agents that are critical for
middleware management.
Application Dependency and Performance, or ADP, analyzes SOA, Oracle Service Bus, Portal,
and ADF applications. ADP collects performance measurements, tracks contextual
relationships, and summarizes data in real-time while introducing as little overhead as possible.
So with ADP, you can understand the complex relationships among the business functions,
associated interconnected components, and the underlying runtime environments.
Deploy the ADP Manager to a managed server in the Cloud Control domain. Or multiple
domains, as you see here, for load balancing and high availability. Then deploy the ADP agents
on the production WebLogic servers wherever you need deeper visibility into SOA, OSB, Portal,
and ADF applications.
Another critical functionality in Cloud Control is JVM Diagnostics, or JVMD, which allows you to
diagnose problems in Java applications in your production environment. It provides deep
diagnostics into Java applications - to the Java thread level and can even provide line-of-code
visibility. JVM Diagnostics can help resolve issues with applications related to java locking,
memory leaks, hung threads, and so on. And it does this with near-zero overhead, making it
ideal to be always-on in production to catch intermittent issues that are extremely hard to
reproduce and diagnose.
Deploy the JVMD Manager to a managed server in the Cloud Control domain. Or multiple
domains, as you see here, for load balancing and high availability. Then deploy the JVMD
agents on the production WebLogic servers wherever you need deeper diagnostic visibility into
Java applications. That could include systems that are also involved in SOA, Oracle Service
Bus, Portal or ADF because those are also Java based, if you need that visibility.
Note, you should deploy each of the Diagnostics Managers to their own managed server with no
other applications running on those servers.
So Oracle does not recommend deploying both ADP and JVMD agents universally. Only deploy
them where you need that deep diagnostic visibility into your applications.
Slide 16
Expand with Business Transaction Management
User
User
ADP
BTM
BTM ADP
BTM ADP
JVMD
JVMD
JVMD
Load
Deploy Balancer
BTM for real-time
monitoring and instance
tracking of transactions.
BTM
DB
DB
OMS
JVMD
OMS
JVMD
ADP
BTM
ADP
DB
Management
Repository
DB
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
If you need visibility into your business transactions, further expand your Cloud Control
infrastructure with Business Transaction Management, or BTM.
This visibility is an important piece of middleware management as these transactions span
multiple tiers and applications. BTM provides real-time monitoring and instance tracking of cross
tier and cross application transactions. With it, you can non-invasively view transaction
performance and conditions or faults centered on transaction performance. You can also
perform payload capture, search and correlation. BTM gives you understanding of your logical
transaction flow. Both the WebLogic Server and the SOA Management Packs come with BTM.
In the case of the WebLogic Server Management Pack, BTM monitors Java EE services
running on WebLogic Server. In the case of the SOA Management Pack, BTM monitors SOA
services running on the SOA Suite. So they do not overlap. Note, however, that BTM is a
separate install.
The Central BTM Servers manage the BTM environment and contain the service-level and
transaction management components. Deploy these three BTM server applications to a
WebLogic server. They also require an Oracle database for storing performance data, logging
messages, and so on.
BTM Observers monitor messages and calls between the components of your applications.
Decide which business transactions need to be managed then deploy BTM Observers onto the
WebLogic Servers involved in those transactions.
BTM Monitors collect application performance and usage measurements from observers then
pass this data onto the Central BTM servers. Deploy the monitor application to a WebLogic
Server.
Slide 17
Real User Experience Insight
Install RUEI to monitor
and manage your end
user experience.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Another management tool that you may want to install is Real User Experience Insight, or RUEI.
Install RUEI to proactively monitor and manage your end user experience. You can see who
your users are, where are they coming from, what they’re buying or not buying, what kind of
user experience they are getting, whether they’re running into errors, etc. RUEI collects,
processes and presents every detail of your end user experience.
This is a separate product from Cloud Control but there is tight integration between the two so
you can drill down directly from RUEI into Cloud Control to view the deeper monitoring and
diagnostics data or into BTM to view transaction data.
Slide 19
Planning Summary
•
•
Deploy Cloud Control universally
Expand with business-driven needs by selectively
deploying:
–
–
–
–
ADP for SOA, OSB, Portal, and ADF applications
JVMD for Java applications
BTM for transactions
RUEI for user experience management
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Now let's summarize the best practices we covered in this planning stage of the development
lifecycle.
First deploy Cloud Control universally and then expand with your business-driven needs.
•
•
•
•
Deploy ADP where you need deep visibility into SOA, OSB, Portal, and ADF applications.
Deploy JVMD where you need deep diagnostic visibility into Java applications.
Deploy BTM where you need visibility into your transactions.
And install RUEI to monitor and manage your user experience.
Slide 20
Road Map
Plan
Improve
Processes
Set
Operational
Goals
Track &
Control
Changes
Monitor
Proactively
Diagnose
Problems
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Before you go into production, you want to set your operational goals. This is where you decide
what to monitor and when you want to be alerted if a target is not performing as expected. Here
I’ll give you best practices on defining your monitoring standards so you can effectively monitor
the whole range of middleware targets in your enterprise.
Slide 21
Setting Operational Goals Best Practices
Define Metric Thresholds
Create Monitoring Templates
Create Template Collections
Create Administration Groups
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
First you should define your metric thresholds so you can be proactively alerted of problems
before they occur.
Next you should create monitoring templates to group together thresholds by target type and
simplify managing and monitoring many targets.
Then you should group together monitoring templates of different types to create template
collections.
Also you should create Administration groups to group together different target types. This is
what you associate the monitoring template collections with in the next stage to further ease
managing and monitoring a large number of components in your enterprise.
These are the most important tasks at this stage. Let’s look at each of these in turn.
Slide 22
What are Metric Thresholds?
Threshold
crossed
Alert
Email /
Page
Administrator
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Metric thresholds are very important for any type of management and monitoring and, in fact,
are often called the "bread and butter of monitoring". So defining these is an extremely
important task. You can specify metric thresholds for each performance metric that Cloud
Control monitors. A metric threshold causes an alert to be triggered when a particular
performance metric value exceeds the limit. For example, when the response time on a critical
application, becomes very slow. Often you're not aware of performance problems until users call
to complain. Using rules and notifications, which we will discuss later, you can be sent an email
and/or be paged when a threshold is crossed. This enables you to be more proactive in
monitoring the availability and performance of your Middleware and the applications it supports.
So thresholds let you detect impending IT problems.
At this stage, you define the operational goals, that is, the thresholds, and then in the next
stage, as you go into production, you'll set up what to do with the alerts by creating rules and
notifications.
Slide 23
Metric Threshold Example
100
Critical Alert
90
Warning Alert
80
JVM 70
Heap 80
Usage
70
%
60
50
40
30
20
10
1 PM
2 PM
3 PM
4 PM
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
You can set metric threshold values for two levels of alert severity – warning or critical. For
example, you want to monitor the JMV Heap Usage on a critical WebLogic Server. So you
define a warning threshold of greater than 80%. Warning means that attention is required but
the area is still functional. And you define a critical threshold of greater than 90%. Critical means
immediate action is required because the area is either non-functional or there is an imminent
problem. When the JVM heap usage exceeds 80% a warning alert is created and you should
take corrective action. When the JVM heap usage exceeds 90% another alert is created and
you need to take immediate action.
Development, Test and Production systems will have different thresholds. In Development and
Test you want the thresholds to be achievable so you set them based on the data you have in
those environments. But you don't want to wait until you go into Production to define your
thresholds. You need to set them up in Test, based on test data, then refine them as you go into
production.
Slide 24
PROPERTIES
Allow user to leave interaction:
Show ‘Next Slide’ Button:
Completion Button Label:
Anytime
Show upon completion
Next Slide
So how do you decide what to set metric thresholds on and what to set them to? Follow this 5step procedure to define your metric thresholds. Click through the numbered steps at the bottom
to proceed.
Determine Critical Components
To decide what components to set your thresholds on, determine your critical components
involved in delivering your services. These are your WebLogic Servers, your Java EE
applications, SOA composites, and so on which are all target types in Cloud Control.
Choose Metrics
Look at the metrics available in Cloud Control for each of the target types for your critical
components and decide which are the important metrics that you need to be alerted on if they
go out of normal range. For example, you may want to be alerted if your WebLogic Server
‘Datasource Failed Waiting Connection Requests’ exceeds a certain limit. And you probably
want to know if the average response time on a critical SOA composite becomes too long.
Choose only the metrics you care about. You should only set thresholds for metrics whose
events you will manage. You do not want to generate unnecessary metric alerts. You may also
want to disable metric collection for those metrics you do not need to avoid unnecessary
collection and storage of those metric values.
Determine Acceptable Performance
To determine what to set the threshold values to, analyze your environment during load testing
and determine when performance is acceptable for a typical workload. That is when it meets
your service level agreements.
Define Thresholds
While performance is acceptable, you can identify the threshold values for key performance
indicators by viewing performance metrics in Cloud Control. Here you’re looking at your critical
business applications. You decide on the service level you require and determine your
thresholds from that. When defining thresholds, the key is to choose acceptable values to avoid
unnecessary alerts, while still being notified of issues in a timely manner. Be conservative with
critical thresholds – reserve these for high signals of a serious problems! Note that a few metric
thresholds come predefined out of the box. Although these values are acceptable for most
monitoring conditions, your environment may require customized threshold values to reflect the
operational norms of your environment accurately.
Fine-Tune
Once thresholds are set, you can then fine-tune these values over time until your system meets
or exceeds your performance goals. And Cloud Control can also make recommendations for
thresholds based on historical data, however, this would only apply once you’ve been in
production for a while.
Slide 25
Create Monitoring Templates
Production Template
SOA
Monitoring
Composites
Avg Response
Time > 10/20
Threshold
1
Business Faults
> 10/15
Threshold
2
Errors > 5/10
Threshold
3
Production
Servers
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
To actually set your thresholds, you create monitoring templates. These allow you to set one or
more threshold values for a particular target type. Then later you can apply them to one or more
targets of that type. For example as you see here, you want to set performance thresholds on all
your production SOA composite applications. You decide that ‘Average Response Time’,
‘Business Faults’ and ‘Errors’ are what you want to monitor and you define the thresholds for
each of these. Then you create a monitoring template called Production SOA Composites and
actually set the threshold values. Then you can apply these thresholds to all your production
SOA Composite applications running on your production servers. This simplifies the task of
standardizing monitoring settings across your enterprise because you set specific thresholds
once and then apply the template to the appropriate WebLogic Servers, applications, and
services.
You want to avoid setting thresholds individually, server by server or application by application,
instead create templates that reflect your overall business Service Level Agreements (or SLAs)
and then mass apply them to your whole environment. (Here we’re talking about simple
business SLAs that translate into thresholds in Cloud Control, for example, you want response
time to be less than a minute. There are also more complex SLAs that are used for service
provider contracts. These map to the SLA feature in Cloud Control which is beyond the scope of
this course so I won’t be going into those.)
Note that you can create thresholds at different levels of granularity. Generally you want to
create broadly applicable thresholds that can be templatized. The templates are the base from
which additional fine-grained thresholds may be set. You may need to add in fine-grained
thresholds as you encounter specific issues, but typically this will be rare.
Also Cloud Control does include predefined templates for monitoring Oracle target types, such
as Oracle WebLogic Domain, Oracle Fusion Middleware Farm, Application Deployment, Oracle
WebLogic Server, etc. You can use predefined templates as a starting point or create a new
monitoring template.
At this setting operational goals stage, you create the monitoring templates and you apply them
in the next stage as you go into production.
Slide 26
Create Template Collections
Monitoring Templates
Production
Production
SOA
Composite Application Production
Deployment WebLogic
Threshold 1
Threshold 1 Server
Threshold 2
Threshold 1
Threshold 2
Threshold 3
Threshold 2
Threshold 3
Threshold 3
Template Collection
Production Template
Collection
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Taking it a another step, you can further ease the administration of large enterprise
environments by creating template collections.
A template collection is a group of settings used to monitor and manage different types of
targets within Cloud Control. A template collection contains a collection of different monitoring
templates. You can only have one monitoring template per target type in the template collection.
So for example, as you see here we have one for SOA Composites, one for Application
Deployments, and one for WebLogic Servers. You could have others as well. And in this
example the template collection contains the settings for production targets.
Template collections can also contain compliance standards and cloud policies. But they are
beyond the scope of this course and so I won’t be going into those here.
Slide 27
Create Administration Groups
Administration Group
Template Collection
Production
Group
Production Template
Collection
Associate
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The reason you create template collections is so you can associate them with Administration
Groups.
Today’s IT operations can be responsible for managing a great number of components, such as
WebLogic domains, SOA infrastructure, databases, hosts, or other components, which can be
time consuming and impossible to manage individually. You can combine targets into logical
sets, called Administration groups and then collectively apply monitoring and management
settings to those groups. This enables you to organize, manage, and effectively monitor the
potentially large number of targets in your enterprise. So you should put together the targets
that are monitored and managed in the same way into a group – for example all your Production
systems could be monitored and managed in one way, versus your non-Production systems. Or
you could group together all the targets that support the same application.
Then you associate a template collection with an Administration Group so all the monitoring
settings in the template collection are applied to the members of the group.
Then, when a new target is added to the group, Cloud Control automatically propagates
monitoring and other management settings using the Template Collection. This completely
eliminates the need for administrator intervention. This level of automation simplifies the target
setup process and also enables a datacenter to easily scale as new targets are added to Cloud
Control. (Administrator privileges also propagate in Administration groups when new targets are
added.)
At this setting operational goals stage, create the template collections and, in the next stage as
you go into production, you associate them with Administration groups.
Oracle recommends using template collections to apply your thresholds. Applying monitoring
templates directly to targets is recommended only for ad hoc operations.
Slide 29
Setting Operational Goals Summary
•
•
•
•
Define metric thresholds
Create Monitoring Templates
Create Template Collections
Create Administration Groups
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Now let’s summarize the best practices we covered in this setting operational goals stage of the
development lifecycle.
•
•
•
•
Define metric thresholds on your critical components based on performance that meets your
Service Level Agreements.
Create monitoring templates to group together thresholds for targets of the same type.
Create template collections to group together thresholds for groups of targets of different types.
Create Administration Groups to monitor and manage different types of targets in the same way.
Slide 30
Road Map
Plan
Improve
Processes
Set
Operational
Goals
Track &
Control
Changes
Monitor
Proactively
Diagnose
Problems
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
An integral part of your daily administrator duties is to ensure that your Fusion Middleware and
WebLogic deployments are up and performing as expected. So once you are in production it is
highly advantageous to proactively monitor your targets for potential problems before they
become severe. Here I’ll give you best practices on monitoring your targets proactively.
Slide 31
Monitoring Proactively Best Practices
Apply monitoring thresholds to your
targets
Set up notifications
MANAGE BY EXCEPTION
Create reports
Customize and build monitoring
dashboards
Monitor diagnostic findings
Monitor transactions with BTM
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Oracle’s recommended general best practice here is to manage by exception. You don’t have
time to sit all day staring at the Cloud Control console, especially since a lot of the data is quite
low level. That’s not an effective use of your time. We’ll talk more about diagnostics and when
to view low-level metrics later but for now you should set up proactive monitoring so you can be
notified of impending problems.
You’ve defined your thresholds for key middleware performance metrics in the Setting
Operational Goals stage by creating monitoring templates and template collections, now you
need to apply those thresholds to your targets.
Then you should set up notifications so you’ll be alerted if any of your thresholds are violated.
You do that by doing some initial set up then creating rules and rule sets. These are the two
most important tasks at this stage.
But you should also create reports so you can confirm that your metrics are good.
And you should customize and build specific monitoring dashboards in Cloud Control. As I said
earlier, it’s not an effective use of your time to spend all day looking at the Cloud Control
Console but it’s a good idea to have a few key dashboards customized to your particular
monitoring needs that you can monitor routinely.
Also you want to check your important targets for diagnostic findings and view these to be
alerted of potential issues.
Lastly, to monitor your transactions, use Business Transaction Management or BTM.
Slide 32
Apply Thresholds to Targets
•
Define thresholds
•
Apply thresholds to targets
– Applying monitoring template to targets directly
– Associating template collection to administration group
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
In the previous stage, you defined your monitoring thresholds. You did this by creating
monitoring templates and adding monitoring templates to template collections to make
management and monitoring in large environments easier. Now you actually apply those
thresholds to your targets by either directly applying monitoring templates to targets or
associating template collections with Administration groups.
Slide 33
Apply Monitoring Templates
Deployed
Applications
SOA
Composites
WebLogic
Servers
Thresholds
Monitoring
Templates
Targets
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
So you've defined all your thresholds of the same target type into monitoring templates.
One way you can set your thresholds is by applying these monitoring templates directly to
multiple targets of the same type. Here you see how a deployed applications monitoring
template can be applied to multiple deployed applications, a SOA composites monitoring
template can be applied to multiple SOA composites, and a WebLogic Server monitoring
template can be applied to multiple WebLogic Servers. However, Oracle recommends this only
for ad hoc operations.
Slide 34
Associate Template Collections with Groups
Deployed
Applications
SOA
Composites
WebLogic
Servers
Thresholds
Monitoring
Templates
Template
Collection
Administration
Group
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
As a best practice, you should use template collections to set your thresholds.
By this stage, you've grouped your monitoring templates into template collections. Each of these
template collections contains only one type of each monitoring template. For example as you
see here, the template collection contains one deployed application template, one SOA
composite template, and one WebLogic Server template.
Now you associate a template collection with an Administration group to apply all the thresholds
contained in the template collection to actual targets. You'll recall that administration groups are
a way to group together targets of different types that are managed and monitored in the same
way, for example, all your production systems and applications as you see here.
By associating a template collection to an Administration group you collectively apply all the
monitoring and management settings to the group in one step rather than having to apply them
to each target individually. Then, when a new target is added to the group, Cloud Control
automatically propagates the monitoring and management settings using the Template
Collection. This allows you to manage many targets as one and is scalable as your enterprise
grows. As new targets are added there is now minimal effort to monitor those new targets. So
Oracle recommends using template collections and Administration groups as the preferred way
to apply your thresholds across your enterprise.
Slide 35
Set Up Notifications
•
Define thresholds
•
Apply thresholds to targets
•
Set up notifications
– Set up for notifications
– Create rules and rule sets
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
You’ve defined your thresholds for key middleware performance metrics in the Setting
Operational Goals stage by creating monitoring templates. And you’ve applied these thresholds
to your targets using template collections.
Now you need to set up notifications so you’ll be notified when your warning and critical
thresholds are crossed. This allows you to be proactive in your middleware management and
monitoring. You’ll know about a problem before it affects users negatively. For example, when
SOA applications are failing, proactive alerting lets you initiate the diagnostic process before
your phone starts ringing. To set up notifications you need to perform some technical setup
tasks and then you need to create rules and rule sets. Let’s look at these in more detail.
Slide 36
Types of Notifications
SNMP
•
Email / page Administrator
•
OS command
•
PL/SQL procedure
•
SNMP third-party applications
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Notifications are typically performed by sending an e-mail and or a page to an Administrator as
you see here. For example, you may choose to email an Administrator if a warning threshold is
crossed whereas you page them if a critical threshold is crossed.
But you can also set up more advanced notification methods. You can execute operating
system commands (including scripts), you can also execute PL/SQL procedures, and you can
send SNMP traps to SNMP-enabled third-party applications. So, for example, if a critical
WebLogic Server goes down, you may want to automatically open an in-house trouble-ticket
using an operating system script so that the appropriate IT staff can respond in a timely manner.
Slide 37
Set Up for Basic Notifications
Email / page Administrator:
•
Set up Mail Server
•
Define Administrators’ email addresses
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
To set up for basic notifications, that is email and page, you need to set up your mail server and
define email addresses for each administrator who needs to receive notifications. And for email
notifications, you can customize the email format. You can add target properties such as Line of
Business, Owner, Contact, etc in the email to provide additional operational information that can
be very helpful when a notification occurs.
More set up is required for the advanced notification methods which is beyond the scope of this
course and I won’t go into here.
Then to start receiving notifications, you create rules and rule sets. But before we go into that,
let’s talk about the different kinds of occurrences that can trigger notifications.
Slide 38
What can Trigger Notifications?
Performance
Space
Target Down
INCIDENTS
Metric
Alerts
Job
Events
Standards
Violations
Availability
Events
Other
Events
EVENTS
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
As you now know, when a threshold is crossed this creates a metric alert. A metric alert is one
type of event in Cloud Control. An event is a significant occurrence on a managed target that
typically indicates something has occurred outside normal operating conditions. For example, in
addition to threshold violations creating a metric event, a database target down creates an
availability event, a job failure creates a job event, a compliance standards violation creates a
standards violation event, and so on. Cloud Control monitors the software stack from
applications, databases to hosts and the operating system. When Cloud Control detects issues
in any of this infrastructure, it raises events.
You can manage events using incidents. Generally, an incident is a situation or issue you need
to act on. Of all the events raised within your managed environment, there is likely only a subset
that you need to act on because they impact your business applications. An incident is,
therefore, a significant event (such as a target down event), or a combination of events that
relate to the same issue (for example, running out of space can be detected as separate events
raised from the database, host, and storage targets). You manage incidents using Incident
Manager which we'll discuss later.
You can set up notifications to be triggered by either events or incidents.
Slide 39
Create Rules to Automate Notifications
Targets
Rule
Set
Notifications
Administrator
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
So now you need to automate the notifications related to events and incidents. This automation
in support of operational processes is an important part of any scalable monitoring solution. To
do so you create rules and rule sets. These enable you to choose the targets and the conditions
for which you want to receive notifications from Cloud Control.
Basically, a rule is an instruction for Cloud Control to take action on an event, incident, or
problem. (A problem is a special type of incident raised by Cloud Control that is outside the
scope of this course so I won’t cover them here.) As we saw earlier, actions include sending
email and page notifications, opening helpdesk tickets, creating incidents, and so on. To create
rules you create rule sets. These allow you to logically combine different rules that operate on a
common object such as a group of targets, as you see here, into a single manageable unit.
Slide 40
Rules Example
Rule Set for PROD-GROUP
Applies to: PROD-GROUP
Target Down Rule
Criteria:
SOA Composite Average Response Time,
Business Faults metric alert events of warning
or critical severity
Action:
Create incident, Set Priority = High
Email and Pager Rule
Criteria:
All Incidents with severity= Critical or Warning
Action:
If severity = Warning, email
If severity = Fatal or Critical, page
Escalation Rule
Criteria:
All Incidents open > 24 hours
Action:
Escalate to Level 1
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
For example, you can create a rule set for your production targets, that is, all the targets in your
Production Administration Group. You can then create the rules.
You can set a rule on a number of metric alerts, say, if SOA composite 'Average Response
Time' and 'Business Faults' cross warning or critical thresholds then an incident is created.
Oracle recommends monitoring and managing events using incidents. Therefore use rules to
automatically create incidents for events that are important to be managed. In a rule set, the
rules that create incidents should typically be the first set of rules in the rule set. There is an outof-the-box rule set that does this which you should review to see if it meets your requirements.
Then you can create another rule on incidents of warning or critical severity. If the severity is
warning you can be notified by email, if the severity is critical you can be notified by page.
Finally you can create another rule on incidents with critical severity. If they are open more than
24 hours you can escalate the incident to level 1.
So this rule set automates a business process to create an incident, notify you of the impending
problem and escalate it if no action is taken.
Slide 41
PROPERTIES
Allow user to leave interaction:
Show ‘Next Slide’ Button:
Completion Button Label:
Anytime
Show upon completion
Next Slide
Here are a few tips on setting up your rules. Click on each of the segments within the circle to
view Oracle’s recommendations.
Rules
Recall that you create rules to automate the notifications related to events and incidents.
Plan
You can see setting up your rules is very important so take time to plan the rule structure before
implementing them.
Use Groups
Specify an Administration Group as the object of the rule set rather than individual targets. It’s
much easier to manage and monitor your targets in groups. The group you use in your rule set
should contain targets that have common requirements for notification as well as common
requirements for actions on events and incidents. As the group later expands and additional
targets are added, the rule set will automatically apply to the newly added members without
further modification of the rule set.
Also put all the rules that pertain to an Administration group into one rule set instead of across
multiple rule sets. Doing so makes it much easier to track and manage all actions on the group
because it’s centrally defined in one place. And it avoids potential duplication of actions across
rule sets.
Create Rules
Set the rules for production groups different than the rules for non-production groups. Your
production operations are critical and so you should be notified immediately if anything is going
wrong before it becomes a major problem. Non-production targets are not as critical so you
don’t need to have such stringent requirements.
Create Incidents
Create incidents for the events specified in the rule. Since there are operational requirements to
send notifications for these events, then this means these events are actively managed and
thus incidents should be created for these. Rules that create incidents for events should
typically be the first set of rules in the rule set.
Act on Incidents
Once incidents are created, perform any other subsequent actions in the rule set, such as
notifications, on the incident instead of on the event. One exception would be in the case where
interested parties are interested in receiving notifications for specific events. For example,
business owners might want to be notified of ‘target down events’ for targets that impact their
business applications.
Remind
Set up reminders using repeat notifications so administrators are notified repeatedly until an
important incident is acknowledged. When critical thresholds are crossed you need to ensure
the problem is taken care of by an Administrator.
Escalate
If notifications are not being responded to, for example, the recipient may be out sick for a day,
escalate these unattended, important notifications. Send a page to a different person, say at the
manager level, if an incident is open too long.
Review
If you are receiving too many or not enough notifications, review your thresholds. You may need
to fine-tune threshold values over time until your system meets or exceeds your performance
goals. You can use as a guide recommendations made by Cloud Control for thresholds based
on metric data for recent periods of time, for example the last 31 days.
Slide 42
Create Reports
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
You’ve set up notifications to be notified when things are changing. However, you may still want
confirmation that your metrics are good. Cloud Control comes with a build in reporting
infrastructure that you can use to generate reports. So to check that everything is OK, generate
reports on your managed targets. For example, you may want to run weekly reports on the
performance of your critical WebLogic Server, SOA or OSB performance. The report shown
here shows a Summary, Availability State and Availability history for a host target over the past
24 hours.
You can use predefined report definitions to generate fully formatted HTML reports, which show
critical operations and business information, without any additional configuration or setup. Or
you can make your own customized reports that suit your specific operational needs. And you
can set up automatic email delivery so you don’t need to run them manually.
Slide 43
Customize and Build Monitoring Dashboards
•
•
•
Proactively monitor a few summary dashboards
Customize default dashboards
Create new dashboards
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
It's also a good idea to have at least a few Cloud Control summary dashboards that are targeted
to your critical applications that you can proactively monitor. These can give you a quick
overview of the health, status, and performance of your middleware targets and Cloud Control
provides many default dashboards to do just that. However you should customize these and
create new ones to meet your specific monitoring requirements for your mission critical
applications. This lets you more efficiently analyze, correlate and diagnose performance
problems.
Slide 44
Customize and Build Monitoring Dashboards
WebLogic
Server
Home Page
Middleware
Performance
Summary
SOA
Home
Page
Composite
Application
Group
Home
Page
Incident
Views
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Here you see some of the most useful dashboards to customize or build. Click on each of these
dashboards to find out more details. When you are finished exploring, click Next to continue on
with the course.
Slide 45
Customize WebLogic Server Home Page
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Home pages make it easy to locate the most important monitoring data and the most commonly
used administrative functions. However, you may want to go beyond the default views to see
monitoring data particular to your environment.
You can customize any target home page in Cloud Control but two important home pages for
Middleware management and monitoring are the WebLogic Server Domain and the WebLogic
Server home pages. The default view, as you see here, gives you an overview of the state of
your WebLogic Servers but you may want to see additional metrics, for example, you can add
particular metrics you are interested in using the Performance Metric Chart. You can also reorganize the regions displayed to help you monitor your servers more easily. Do this by
personalizing the page - you can customize the format of the page as well as choosing what
data to display on the page, taking off some content and adding other content.
Slide 46
Customize SOA Home Page
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The SOA target home page is the main landing page and jump-off point for users to interact with
Cloud Control functionality related to the Oracle SOA Suite. Here you see a comprehensive
overview across the entire SOA Infrastructure - aggregated throughput metrics for the whole
soa-infra, summary metrics for the service engines, and summaries for the SOA composites.
You can use this screen to get a quick overview of the health, status, and performance of the
SOA domain.
Add specific additional metrics and re-organize the regions which are displayed to suit your
particular needs.
Slide 47
Customize Group Home Page
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
An important home page worth customizing is the Group Home Page. When you create an
Administration group to group together all your targets for a particular purpose, for example, all
your production targets or all your WebLogic targets associated with a particular set of
applications as you see here, you get a group home page so you can visually monitor them
together.
Customize this home page so you see just the monitoring information you need for your
particular requirements.
Slide 48
Customize Middleware Performance Summary
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
An important Middleware monitoring dashboard you can customize is the Performance
Summary page. Here you see all the relevant performance data for a particular middleware
target. Using this page you can ensure your Fusion Middleware and WebLogic Server
deployments are performing as expected and meeting your targets.
You can customize this page to monitor exactly the performance metrics you want. Here we've
added 'Deployment Work Manager Requests' to the default view.
You can also select multiple metrics at a time for correlation, overlay metrics for comparison and
better analysis, compare performance impact on a periodic basis, and create performance
summary baselines which is like a snapshot of performance that can be used as a benchmark.
Slide 49
Create Composite Application Dashboards
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
As well as customizing existing dashboards in Cloud Control you can create new ones.
In complex datacenter environments administrators need a single view of top level metrics and
information related to critical business services and applications. Composite Applications are a
way for you to create your own dashboards for your custom applications. This lets you build a
comprehensive view representing a multi-tier composite application composed of multiple
application deployments and SOA composites. You can easily include all additional components
(such as databases, service buses, Coherence clusters, and other middleware and nonmiddleware targets). The Composite Application dashboard provides full visibility across the
composite application with access to key monitoring and diagnostics regions, which can be
easily customized and personalized. The overall result is a single dashboard view providing not
only health information about the application, but also deeper visibility into component health
and incidents at a glance.
Each logical application environment in your enterprise can have its own composite application
dashboard and these dashboards are fully customizable. You can personalize any of the
regions to display any relevant metric.
Slide 50
Create Incident Views
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
As well as monitoring performance metrics on various dashboards, you want to monitor your
incidents by creating incident views.
Now that monitoring has been set up, events will be raised and incidents created for these
(using the rules you created). Incident Manager, what you see here, gives you a centralized way
to manage all these different incidents. It’s a central location from which you can view, manage,
diagnose and resolve your incidents as well as identify, resolve and eliminate the root cause of
disruptions.
On the left side are views that allow you to look at specific incidents of interest. Cloud Control
comes with default views for the most common tasks, for example, there is a view for all your
open incidents and problems, another for unassigned incidents, etc. Create your own custom
views to filter incidents based on your specific search criteria. A good example of this is to
create a view that shows all incidents for an important group of targets, for example your
Production Group.
Slide 51
Monitor Diagnostic Findings
Metrics
Configuration
Settings
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Another monitoring task you should perform regularly is to check for Diagnostic Findings on
your WebLogic Server targets.
The Middleware Diagnostics Advisor takes advantage of having all the configuration information
– for servers, applications, and hosts – as well as all the performance metrics and component
dependency data that Cloud Control collects automatically. It analyzes the entire stack of
targets from the WebLogic Server down to the database. Based on this information it generates
findings, displaying all of this correlated performance management data on a single console.
More importantly, instead of just providing raw metrics and configuration details, it helps you to
resolve problems and improve performance by giving actual advice. For example, it can help
you identify slow SQL statements as you see here or that a JDBC connection pool is causing a
performance bottleneck.
We'll talk more about diagnosing problems later but you can also use the the Middleware
Diagnostics Advisor page to locate a potential problem by focusing on the most common finding
based on the number of occurrences in the past 6 and 24 hours. Examine the summary
information at the top of the finding, including the charts. Also examine the data in the Analysis
Information table to understand the finding better, and also examine the related targets table to
determine which targets are impacted by this finding. Respond proactively by performing the
action referenced in the Suggestion column. Here it suggests tuning the SQL statements might
improve performance.
Note, that currently, there are no SOA specific findings for the Diagnostics Advisor.
Slide 52
When to Monitor Diagnostic Findings
Monitor findings during:
•
Peak load time
•
High CPU utilization
•
High disk I/O
•
High memory utilization
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
So when is it advisable to monitor diagnostic findings? First, monitor the findings during peak
load time to check for potential problems that the load might have caused. Also, monitor findings
during high CPU utilization, high Disk I/O, or high memory utilization to check for problems that
might have caused the high utilization.
Slide 53
Monitor Transactions Using Business Transaction Management
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Lastly, use Business Transaction Management to monitor your transactions. The operational
health summary dashboard gives you a roll-up of key metrics for your system and transactions.
The indicators will quickly tell you if something is wrong, and the associated number will tell us
the actual number of issues. For example, here you can see that one service and one endpoint
are down.
Slide 55
Monitoring Proactively Summary
•
•
•
•
•
•
•
Generally, manage by exception
Associate Template Collections with Administration
Groups
Set up notifications
Create reports
Customize and build monitoring dashboards
Monitor Diagnostic Findings
Monitor transactions with BTM
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Now let’s summarize the best practices we covered in this monitoring proactively stage of the
development lifecycle.
•
•
•
•
•
•
•
Generally, Oracle recommends to manage by exception.
Associate template collections with Administration Groups to quickly apply standardized
monitoring settings to many targets of different types and have them applied automatically if a
new target is added to the group.
Set up notifications by creating rules and rule sets so you’ll be notified of impending problems.
Create reports to confirm that your metrics are good.
Customize and build monitoring dashboards to meet the specific monitoring requirements of your
mission critical applications.
Monitor diagnostic findings during peak load, high CPU utilization, high disk I/O and high memory
utilization times.
Monitor transactions with BTM to view the operational health of transactions and the systems
involved in those transactions.
Slide 56
Road Map
Plan
Improve
Processes
Set
Operational
Goals
Track &
Control
Changes
Monitor
Proactively
Diagnose
Problems
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
You need to diagnose problems at all stages in the development lifecycle – through
Development, Test, QA and production, but it’s vitally important in production. When your
critical business applications go down or are not performing well, your business suffers. Here I’ll
give you best practices on diagnosing problems quickly and efficiently.
Slide 57
Diagnosing Problems Best Practices
Narrow down the problem
Drill down to find the root cause
View Diagnostic Findings
Create Diagnostic Snapshots
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Diagnosing problems is a very challenging area as there can be so many different types of
problems with so many different causes. And middleware applications introduce a new order of
complexity in tracking down failures as they are very complicated with multiple tiers within the
application stack. There's no one master methodology to finding the root cause of a problem as
it depends on the symptoms you’re experiencing. So for this course, I’ll be covering diagnostics
at a very high level and making general recommendations about how to use Cloud Control to
solve your middleware performance problems.
Oracle’s recommended approach is to first asses your notifications and/or user reports to
narrow down the possible cause of the problem to a particular part of the infrastructure, for
example, is it a problem with a WebLogic server, the SOA infrastructure, or a particular SOA
composite? Then, once you’ve narrowed it down, drill-down using the appropriate customized
dashboard to find the root cause.
You should also view diagnostic findings from the Middleware Diagnostics Advisor.
And, if you’re in the middle of a critical outage but don’t have time to diagnose the root cause,
create diagnostic snapshots for later root cause analysis.
Slide 58
Narrow Down the Problem
1. Assess notifications / user reports
2. View customized summary pages
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
So first you need to narrow down the possible cause of the problem to a particular part of the
middleware infrastructure. Typically you find out about a problem by either receiving an alert
notification or getting a phone call from someone. This initiates the diagnostic process. Ideally
alert notifications are received before the problem has escalated too far and users are starting
to notice. When you get one or more notifications, you know what metric thresholds were
violated so this should help you to narrow down the cause and guide you to where to start
looking. For example, notification of excessive CPU consumption by a WebLogic Server would
lead you to view the WebLogic Server's home page to see what applications are running on that
instance and investigate further.
When you are notified of a problem by a person, this can be more challenging to narrow down
as the symptoms they are seeing, for example, slow response time of an application, could be
caused by so many factors, many outside of the middleware layer, for example network
issues. For this course we'll assume all non-middleware infrastructure issues have been ruled
out. So if you don't have a clear idea of where to start, this is where your high-level customized
dashboards come in useful. For example if the complaints center on the performance of one
SOA composite application you can view your composite application dashboard that you
created which will show all the composite application dependencies in real time. You then may
be able to see that the problem is in an underlying WebLogic Server so you can then investigate
further. So the notifications and/or user reports help you to form an initial hypothesis.
Then you want to view the appropriate customized summary page in Cloud Control. These
provide a view from the “4,000 foot” perspective and they should point you in the direction of
the problem.
Slide 59
Drill Down to Find the Root Cause
1. Assess notifications / user reports
2. View customized summary pages
3. Drill down into problem area
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Then from your customized dashboards, drill down into the problem area to find the root cause
of the problem.
This root cause analysis is handled by drilling down through multiple views that are
automatically discovered by Cloud Control. These detailed models of the application
infrastructure allow you to maintain your context throughout the entire drill-down process from
multiple entry points ensuring that you are able to quickly get at the application metrics required
to quickly resolve a performance bottleneck or other application issue.
The goal is to resolve the problem as fast as possible with as little impact on the overall
application as possible.
Slide 60
Drill Down from Summary Dashboards
Target
Home
Page
Middleware
Performance
Summary
Incident
Management
Console
Application
Dependency &
Performance
Composite
Application
Business
Transaction
Management
Log
Viewer
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
You can drill down into deeper diagnostics from just about any page in Cloud Control. Here you
see some of the most useful dashboards you can use to drill down into specific components to
find the root cause of a middleware problem. Some of these you’ve seen before in this course.
Click on a dashboard to find out how you can use it to drill down for root cause analysis. When
you are finished exploring these dashboards, click Next to continue on with the course.
Slide 61
Drill Down From Composite Application Dashboard
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
You’ll recall that the Composite Application dashboard, which you can create, provides full
visibility across all components from the applications to the middle-tier, hosts, services, and
databases that those applications rely on. This is a useful dashboard to drill down from for
problem analysis. Let’s look at an example.
Here you can see a composite application comprised of multiple SOA and Java EE components
that we are monitoring from a central dashboard. You can see a variety of metrics selected from
each of the members including the WebLogic Server and database targets. In addition, you can
see the health of the overall members and a specialized thread state table on the JVMs.
• You can see that some of the JVMs have a lot of locks. The database waits are somewhat
concerning and it makes sense to explore at least the health of the JVM. A large number of
locks can be a huge red flag that your application is having problems. As the JVM is the
most likely culprit, we’ll drill in from that perspective while still keeping the context of the
overall Fusion Middleware and WebLogic Server target states. If we click on this
MedRecServer JVM we can drill down in-context to the JVM to examine its health in more
detail.
• This takes us to the JVM Performance Diagnostics page which shows both JVM and related
database data. We’ve scrolled down to view the general tab where we can see Top
Requests. In this case, we’re concerned with our application health, so let’s filter on a key
request such as one of the patient registration requests which is a requirement of our
application. This will filter the entire page by the request allowing us to examine everything
in context including all of the thread states.
• Now on the Threads tab, we can analyze our thread state over the period in question and
also look at key categories in relation to those threads such as the thread state of method
and request calls. At this point, we see a lot of stuck threads and in particular a lot in the
Database Wait state which could indicate a database problem. Let’s click on the Live
Thread Analysis link to determine how threads are impacting each other live.
•
Let’s explore this by clicking on one of the threads to drill into the root cause of why it may
be stuck in a Database Wait state which is a definite indicator of a hung application. We can
see the thread stack, browse local variables, and examine how it impacts other threads.
However, in this case, we want to determine the root cause of why it’s stuck in the
Database Wait state in the first place. Therefore, let’s drill down to the actual database
session that is stuck by clicking on the DB Wait link.
• This takes us to the database session where we can see all the details of the session that’s
being blocked or running extremely slow. Assuming Cloud Control is monitoring your
WebLogic Server, JVM, and database, you can do this type of cross-tier analysis in
production environments with less than 1% overhead so it won’t impact the applications
themselves. In this case, we can now see that the session is stuck within a table lock
contention state. Let’s click on the Blocking ID to determine which database session is
blocking our applications session.
• We can see that our composite application’s database session was blocked by a database
session kicked off from SQL Plus outside our application. This likely represents a
maintenance routine or another process or application outside of ours. We can drill in even
further, to see exactly what the session is doing or we could kill the session that is blocking
ours (assuming that we had the necessary DBA privileges) or notify the appropriate DBA to
take action and resolve the problem.
This cross-tier analysis between the JVM diagnostics and the database diagnostics allows
issues in relation to JDBC calls to underlying databases to be easily diagnosed without all of the
finger pointing that sometimes occurs when trying to troubleshoot issues between the
middleware and database tiers.
Slide 62
Drill Down From Middleware Target Home Pages
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The middleware target home pages are used to get a quick overview of the health, status and
performance of your WebLogic and SOA targets. You can also use these pages to drill down to
find the root cause of problems. For example, from the WebLogic Server home page you can
easily drill into key metrics associated with the JVM of your applications in the context of the
WebLogic Server domain you are analyzing for performance trends or a potential bottleneck.
Here’s another example. You receive an alert notification of excessive CPU consumption by
WebLogic Server. This leads you to investigate the applications running on that instance. By
using the Servlets and JSPs tab in the Most Requested section of the WebLogic Server Home
page as you see here, you can quickly identify the highest volume or least responsive
application. Then you can drill down and diagnose the application's servlets, Java Server
Pages, or EJBs to identify the bottleneck. And, if necessary, you can drill all the way down to the
JDBC/database layer. Even if you’re unfamiliar with the application design, you can quickly
pinpoint problematic bottlenecks and analyze the metrics and architecture associated with them
in order to facilitate a quick turnaround to get the application up and running at the required
service level.
Slide 63
Drill Down From Middleware Performance Summary
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Performance Summary pages are where you see all the relevant performance data for a
particular middleware target. We looked at customizing these pages earlier to show just the
metrics you’re interested in. Here you can drill down to help diagnose and resolve problems. If
you click on the metric name, you get an Additional Information popup. From here you can view
a Problem Analysis page which shows other related infrastructure metrics which are affecting
the metrics being analyzed.
Inspect the charts for unusual increases in recorded metrics such as request processing time,
CPU usage in storage units or percent utilized, number of requests per minute, Java Virtual
Machine heap memory usage, or server memory usage. If you find that request processing time
increased due to a high number of requests per minute, you may need to increase the capacity
of your system. If the metric charts do not indicate the cause of the problem, scroll down to
the Related Information pane and inspect the List of Related Targets To Analyze table for
information about target health (status) and recent configuration changes.
If you want to see the topology of the components for which data is being displayed, click
the Related Targets Topology tab.
If the List of Related Targets to Analyze table does not indicate the cause of the problem, scroll
up and click a link adjacent to one of the metrics and then click Analyze Logs in the Additional
Information pop-up to display log messages for the selected target and its members during the
selected time period.
When used together, Problem Analysis and Analyze Logs can help you inspect metrics, target
status information, and logs during troubleshooting.
Slide 64
Drill Down From Incident Management Console
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Typically you will have incidents created automatically when your critical thresholds are crossed.
From the Incident Management Console you can view and manage your incidents. You can also
diagnose and resolve incidents here. It’s even integrated with “My Oracle Support” for
accelerated resolution of problems.
Earlier you created views to filter your important incidents. Now you can zero in on the critical
ones and drill down to resolve the problem. In this example, I created a Production Incidents
view, and here we need to investigate the SOA composite that is down. When we select this
incident, detailed information is show in the bottom portion of the screen.
In the bottom-right portion of the page we see links to the guided diagnostics and resolution
area to assist you in resolving the issue. For example, if I click the View Topology link I see the
topology view where I can see the health of other targets the SOA composite depends on. And I
can further drill down, in-context, into relevant lower level metrics to further investigate
performance behaviors.
Slide 65
Drill Down From Log Viewer
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The Log Viewer lets you see critical, or even just suspicious, errors across all log files,
including WebLogic and Fusion Middleware logs. So you can gain access to log files
regardless of where they reside or what product they relate to. Here you can search and
correlate messages across log files based on time, severity or Execution Context ID (ECID).
Then you can use the ECID to drill down to the middleware tier for deep-dive details to help
you diagnose and resolve problems.
Slide 66
Drill Down From Application Dependency & Performance UI
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The Application Dependency and Performance, or ADP, pages within Cloud Control analyze
SOA, Oracle Service Bus, Portal, and ADF applications to capture the complex relationships
among the various application building blocks. ADP delivers a holistic, service-oriented view
across heterogeneous environments. Here we see the ADP page for a specific Oracle Service
Bus proxy service. We can see a summary of the proxy service’s performance in tabular and in
graphical form, as well as a summary of the performance of each of the pipeline nodes. We can
further drill down and see individual pipeline nodes within the proxy service allowing us to
diagnose problems within the service.
Slide 67
Drill Down From Business Transaction Management
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Business Transaction Management, or BTM, gives you a central view of distributed transactions
that you can drill down into. For example, two useful predefined dashboards are the Top 10
Services and Top 10 Transactions in terms of load, uptime issues, response time and faults.
Here you see the Top 10 Services dashboard. These dashboards are good starting points for
further analysis, since they give you a good view over the worst performers. Then you can drill
into transaction content and context. This gives a unified view of transactions even across
different keys, IDs, etc.
Let's look at resolving an example problem with BTM. A customer, Epsilon Enterprises, has
called in, complaining about some orders that don't seem to be going through and are getting
lost somewhere. Since we don't know exactly where the issue is, but we do know the customer
name, we'll start by searching through our messages using the customer name. We can see a
few messages related to this customer and a couple stick out as having a much longer
response time than the others.
Let's take a look at one of them. Now that we've found a suspicious message, we can use this
to look at the relevant transaction.
In the transaction inspector, we can see that something is wrong with the PurchasingService.
Drilling down into this service, we can now see in the response that something is wrong with the
outbound service endpoint URL, most likely a typo in the service name.
Slide 68
View Diagnostic Findings from the Middleware Diagnostics Advisor
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Earlier we talked about monitoring diagnostic findings from the Middleware Diagnostics Advisor
to locate potential problems. You should also consult the diagnostic advisor real-time to resolve
current problems quickly and with guided help.
For example, on this WebLogic Server home page, we see Diagnostic Findings which is a clear
indicator that Cloud Control has discovered a key issue with the overall server by analyzing the
performance and configuration data of the entire stack including the middleware and database.
If we click the Diagnostic Findings link, this starts the Middleware Diagnostics Advisor. We can
see that there are at least two findings. One related to locked threads and one related to the
SQL execution under the covers of our Medrec application. We can see these occurred a
number of times with an hourly interval indicating it’s a regular problem that’s occurring
constantly as opposed to being intermittent. Let’s take a look at the details of the findings for
this hour.
If we click on the SQL finding, we see the details associated with the finding and we can see the
associated metrics and configuration data that helped determine the cause of this finding. You
can see that it’s found that SQL execution is taking a long time causing requests to pile up and
increasing response time and reducing throughput. And you can also see the recommendation
which is to tune SQL statements to improve performance. It also lists the SQL statements
themselves which are taking a significant time to run, in this case just one. You could even drill
down into the top one which is the slowest, to look at the SQL in the SQL Analyzer which would
allow the DBA to analyze it more fully and tune the SQL.
If we look at the Thread Lock finding, we can see that there are threads in the WebLogic Server
that are holding locks for a long time and other threads that are waiting and can’t proceed until
these locks are acquired. This is causing server performance to degrade. The recommendation
is to drill down into JVMD diagnostics by clicking on the Lock object to determine the threads
holding and waiting on these locks. And it also recommends to improve the design of the code
to prevent these situations from occurring.
So the Middleware Diagnostics Advisor with it’s proactive root-cause-analysis helps you save
valuable time when it comes to identifying performance bottlenecks in production applications.
Slide 69
Diagnose Problems Using Diagnostic Snapshots
Quickly capture
information about
a problem.
Share snapshots
with Oracle Support
and colleagues.
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Problems often arise suddenly and need to be remedied immediately with no time to capture the
details for later analysis of the root cause. For example, your WebLogic Server and the
underlying JVM enter some locked condition which causes your end users to hang. You need to
get your users up and running as soon as possible, and so you don’t have time to figure out
why the problem occurred. You would like to keep data on the problem, and diagnose the issue
later, in case it happens again.
In these situations, you should immediately capture the state of the JVM, overall metrics for the
WebLogic Server, and log data, as the problem occurs, into a single packaged diagnostic
snapshot for later analysis. These diagnostic snapshots can then be exported and shared with
others to help diagnose the root cause of the problem. Oracle Support can also use these
snapshots to diagnose problems, allowing you to communicate your problems to support much
easier. You can also use these snapshots to validate that a fix you put into place for a problem
has in fact resolved it. This allows you to build in preventatives to better protect your
environment against outages and bottlenecks in the future similar to the one that impacted your
system.
Note that currently there are no SOA metrics in the Diagnostics Snapshot.
Slide 71
Diagnosing Problems Summary
•
•
•
•
Assess notifications/user reports to narrow down the
problem
Drill down using customized dashboards
View Diagnostic Findings
Create Diagnostic Snapshots
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Now let’s summarize the best practices we covered in this diagnosing problems stage of the
development lifecycle.
•
•
•
•
Assess notifications and/or user reports to narrow down the problem to a particular part of your
enterprise infrastructure.
Drill down from the appropriate customized summary dashboard to find the root cause of the
problem.
View Diagnostic Findings from the Middleware Diagnostic Advisor to resolve problems quickly
with guided help.
Create Diagnostic Snapshots in the midst of resolving problems for later root cause analysis.
Slide 72
Road Map
Plan
Improve
Processes
Set
Operational
Goals
Track &
Control
Changes
Monitor
Proactively
Diagnose
Problems
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Tracking configurations is probably one of the most time-consuming and difficult tasks you, as
an administrator, face on a daily basis. For example, do you know how each and every one of
your managed targets is configured and whether they comply with your corporate or regulatory
standards? Here I'll give you best practices on tracking and controlling configuration changes.
Slide 73
Tracking & Controlling Changes Best Practices
View configuration history
Compare configurations
Create “Gold Standard”
Configuration Snapshots
Customize Comparison Templates
Schedule comparisons with
notifications
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Cloud Control automatically collects configuration information for all the hosts and the managed
targets on those hosts that have a running management agent. Using this data it lets you easily
track and control configuration changes in your middleware environments.
As part of configuration management you should view configuration history to pinpoint why a
system may be behaving differently today that was working well yesterday.
Or compare configurations of two targets that were identical to check for configuration drift.
You should also create “Gold Standard” configuration snapshots that you can compare to your
live configurations.
And you should customize configuration comparison templates to specify what to ignore and
what to alert on.
And finally you should schedule your comparisons so they run regularly and set up notifications
so you're notified automatically of important differences.
All of these extensive configuration management capabilities that Cloud Control provides
facilitates compliance with various regulatory and business standards.
So let's look at each of these in more detail.
Slide 74
View Configuration History
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Having recent configuration changes for a particular Middleware target readily available is very
valuable. For example, when an application that was performing well yesterday isn’t performing
well today, often, the performance degradation can be attributed to a recent configuration
change. So if you notice a sudden change in performance, check the target home page for any
Configuration Changes, as you see here. Clicking the link will show you all the configuration
changes in the past week, including the old value and the new value. In this example here, we
see that the login timeout setting changed from 25 seconds to 27 seconds in the past week.
This can be invaluable information to help resolve problems quickly.
Slide 75
Compare Configurations
STAGING
PRODUCTION
ThreadCount
MaxCapacity
StatementCacheSize
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Another way to figure out why a particular middleware target is not performing as well as it was,
is to compare its configuration with another target of the same type. Cloud Control’s
Comparison wizard allows you to compare configurations to quickly and easily pinpoint the
differences between them. This helps to keep systems synchronized and to reduce
“configuration drift”. In addition, it simplifies investigations into why systems that are presumed
to be identical, are behaving differently.
A common use case for this is when you are comparing a WebLogic Server in a staging
environment with another WebLogic Server in a production environment. In this scenario,
there’s a performance problem with the production environment but not the staging, and you
want to compare the configurations as, originally, the two configurations were identical. Upon
comparison you discover that a configuration change was made that introduced the poor
performance in the production environment. For example, a key tuning parameter such as
ExecuteQueue ThreadCount, JDBC Connection Pool MaxCapacity or StatementCacheSize
may have been changed.
And when configuration differences are detected between targets that were presumed to be
identical, you can use Cloud Control to quickly provision configurations such that there are no
longer any differences. We’ll look at doing this in the next section of the course.
Note, that as well as comparing WebLogic Server, domains, and cluster target types, you can
also compare other middleware targets such as SOA composites, SOA Infra, JVMs, and more.
Slide 76
Create “Gold Standard” Configuration Snapshots
•
•
Create snapshots of a well configured
target
Compare current configurations to
“Gold Standard” configurations to:
– Monitor drift
– Quickly identify changes
during critical outages
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
An important best practice is to create “gold standard” point-in-time configuration snapshots for
later analysis and comparison. For example, save the configuration of a WebLogic Server you
know is configured correctly for your production environment with a label such as “Gold
Standard Production WebLogic Server Configuration’.
Then at any time you can compare the current configurations of any of your production
WebLogic servers with this saved configuration to determine if any configuration drift has
occurred from the gold standard. This ensures that your WebLogic Servers are all configured
consistently with this known reference. In fact, your IT management may request that such
configuration consistency be checked on a regular basis and a summary generated that
identifies any important differences.
Also during critical outages, you can compare current configuration settings of the target to the
gold standard and quickly identify changes that could be the root cause.
Slide 77
Customize Comparison Templates
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
When you perform configuration comparisons, use Comparison Templates. These allow you to
specify which property differences to ignore, and which property differences should trigger an
alert. Out-of-the-box, Cloud Control provides comparison templates for WebLogic Server,
domain, and cluster target types as well as SOA Infra, JVM, and more. These out-of-the-box
templates can be used as is, or as a guideline that you can customize. For example, here I’m
customizing the default SOA template. It’s already set up to ignore differences between URLs in
the different environments as these are expected to be different across SOA targets. But I may
be interested in tracking any changes in Audit Level from the standard so I’ll check that to be
notified on any differences.
So, you should review the out-of-the-box comparison templates and customize them as
necessary to meet your specific requirements. In most cases, you may only need to make a few
tweaks to get what you need.
Slide 78
Schedule Comparisons with Notifications
Schedule
comparison
+
Enter email
address
=
Automatic
notification
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Also when you perform configuration comparisons you can schedule the comparison to be run
immediately or performed later. It’s a good practice to set up these comparisons with your “Gold
Standard” configuration to be performed on a regular basis, for example, at the end of each
week or the first of the month. This may even be a requirement of your IT management.
In addition, when performing configuration comparisons you can enter notification email
addresses for who to notify if differences are found. You should enter your email address (as
well as any other administrators) so you can be automatically notified.
So by scheduling the comparison job regularly and entering email addresses you can be
automatically be notified on a regular basis if any configuration drift occurs in your live
configuration from your gold standard.
Going further, you can also validate your target configurations against industry best practices,
Oracle recommendations, and internal best practices by using the Cloud Control compliance
standard framework. However, I won’t be going into that here as this is beyond the scope of this
course.
Slide 80
Tracking & Controlling Changes Summary
•
•
•
•
•
View configuration history
Compare configurations
Create "Gold Standard" configuration snapshots
Customize comparison templates
Schedule comparisons with notifications
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Now let’s summarize the best practices we covered in this tracking and controlling changes
stage of the development lifecycle.
•
•
•
•
•
View configuration history to pinpoint why a system may be behaving differently today that was
working well yesterday.
Compare configurations to check for configuration drift.
Create “Gold Standard” configuration snapshots that you can compare against your live
configurations.
Customize comparison templates to specify what to ignore and what to alert on.
Schedule regular comparisons with automatic notifications of important differences.
Slide 81
Road Map
Plan
Improve
Processes
Set
Operational
Goals
Track &
Control
Changes
Monitor
Proactively
Diagnose
Problems
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
You probably spend a large amount of time performing various lifecycle management
operations - such as installing, patching, and configuring WebLogic servers, as well as
deploying Java applications and SOA composites, and so on. These are very important tasks
but can be very time-consuming and error-prone. Here I’ll give you best practices on improving
your processes and automating common lifecycle management operations. This will allow you
to perform these tasks much faster and more reliably.
Slide 82
Improving Processes Best Practices
Clone Middleware components from
the Software Library
Create Gold Images
Create WebLogic Provisioning
Profiles
Customize default Deployment
Procedures
Automate the application of
WebLogic Server patches
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Rather than installing or deploying Middleware components manually, which can be error-prone
and introduce inconsistency in your environment, you should clone them using Cloud Control.
And the best way to do that is to clone from Cloud Control’s Software Library. This ensures
consistent, standardized images and components are deployed across your environment.
You should also create gold images of tested Middleware Homes and SOA Artifacts in the
software library for later provisioning.
As well as creating WebLogic Domain provisioning profiles to act as templates for WebLogic
domain provisioning.
You should also customize the default Cloud Control deployment procedures to meet your
unique provisioning requirements.
And lastly, you should automate the application of WebLogic Server patches to make this whole
process simple, reliable, fast, and one you can do without any downtime!
You can automate all these routine tasks using Cloud Control’s Software Library and
Provisioning Framework. This lets you improve quality of service by minimizing downtime due to
configuration change. And it reduces your IT operational cost through automated deployment
procedures to provision software. Let’s look at each of these in more detail.
Slide 83
Clone Middleware Components
WebLogic Domain
•
Admin Server
WebLogic Domain
Manual installations are slow
and error-prone
Admin Server
•
Cloning is fast and eliminates
errors
Cluster
Cluster
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Administrators often rely on manual installation methods in order to provision new middleware
environments or add capacity to existing servers which support their production applications.
However manual installation methods can take too long and are error-prone. A much more
efficient means to provision new middleware environments and add capacity is to clone
middleware components in Cloud Control. Cloning reduces time and eliminates errors in
building environments.
You can clone directly from a source domain to create a new domain or scale up or out an
existing domain. “Scaling up” refers to adding managed servers to a domain on existing
hardware, while “scaling out” means adding managed servers to a domain on new hardware.
You would add capacity to existing domains or clusters to accommodate an increase in load.
For example, if your deployed application is incurring high load and not performing adequately
then you need to scale up or out to improve application performance.
You can also clone directly from a source domain to deploy Java EE applications, or BPEL
processes or from a SOA domain to provision SOA artifacts such as SOA composites, web
services policies, JPS policies, and so on. You can also clone the binaries in the Middleware
home directory.
Slide 84
Clone Middleware Components from the Software Library
WebLogic Domain
•
•
•
WebLogic Domain
Ensure consistent, standardized images are deployed
Rapidly
scale
Admin
Serverout or revert to known good stateAdmin Server
Simplify lifecycle management
Cluster
Software Library
Cluster
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
You can clone domains and their components directly from an existing domain, but Oracle
recommends cloning from Cloud Control’s software library to ensure consistent, standardized
images and components are deployed across your environment. This also allows you to rapidly
provision middleware environments or quickly revert to a known good state. For example, you
may need to scale out an application on another environment very quickly after a new version is
introduced that fixes a bug or revert an application to a known good state when a problem is
encountered. Cloning Middleware components from the software library simplifies lifecycle
management.
The software library serves as a central repository for metadata and binary content of
application software, software patches, reference gold images, etc. Here you can store various
revisions of WebLogic Server domains (both the binaries and the domain configuration),
Middleware home binaries, Java EE applications, SOA Artifacts, BPEL processes, Oracle
Service Bus Resources, Java Platform Security configuration, and so on.
You can create software library components in the context of your reference domain. And, when
you clone a component, for example a WebLogic domain, you can make changes to the
destination domain configuration as necessary, such as host names, ports, and so on.
So you should clone components from staging or reference servers into the software library
then deploy those components centrally from the software library to new hardware.
Slide 85
Create Gold Images
Gold images are:
• Based on corporate standards
• Compliant and tested
• Stored in the Software Library
Create Gold Images for:
• Middleware Home
• SOA Artifacts
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Gold images are created by component designers based on corporate standards from reference
deployments. They are compliant and tested images, stored in the software library, from which
they can be used by all operators in a consistent manner.
You can create gold images for Middleware Homes and SOA Artifacts. Middleware Home Gold
Images contain only the WebLogic binaries. SOA Artifacts Gold Images contain SOA Artifacts
such as SOA Composites, Oracle WebLogic Server Policies, Assertion Templates, JPS Policy
and Credential Stores.
So create these tested gold images in the software library for later provisioning.
Slide 86
Create WebLogic Domain Provisioning Profiles
Create a Template for WebLogic Domains
=
+
Middleware
Home Binaries
Domain
Configuration
Provisioning
Profile
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
You should also create WebLogic Domain Provisioning Profiles from existing reference
domains. A Provisioning Profile consists of both the Middleware Home binaries and the domain
configuration. You create a profile, save it in the Software Library, and then use the saved
profile as the source for creating new WebLogic domains. This will ensure that future WebLogic
installations follow a standard, consistent configuration. It also saves time and reduces the risks
of making mistakes. You can think of the provisioning profile as a template for WebLogic
domains provisioning.
Slide 87
Customize Default Deployment Procedures
Deployment
Procedure
Deployment
Procedure
Deployment
Add environment variables
Add custom steps
Procedure
Disable unwanted steps
Add custom scripts
Email notifications
Use authentication tools
Use different error handlers
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
To actually perform provisioning you use a deployment procedure. This contains a hierarchical
sequence of provisioning or patching steps that need to be performed for a particular lifecycle
management activity.
Cloud Control comes with a set of default Deployment Procedures that help you accomplish
common provisioning and patching-related tasks. These default procedures have been created
considering all the best practices in the industry. And each Deployment Procedure is unique,
and is designed to perform a particular operation according to the source being provisioned. For
example, the Deployment Procedure to deploy a Java EE application differs from the one to
deploy a SOA composite.
Understandably, your provisioning and patching-related requirements will vary from one
environment to another, or from a test installation to a production installation. So use the default
Deployment Procedures and customize them according to your needs. For example, you can
add environment variables, include additional custom steps, disable unwanted steps, add
custom scripts, and use authentication tools to run some steps as another user. You can even
implement different error handling methods. And you can reference any of the Middleware
entities in the software library to automate the patching, provisioning or deployment of the
associated software.
To further automate provisioning, set up email notifications to report the status of deployment
procedure execution so you know immediately if it was successful or not, or action is required.
Slide 88
PROPERTIES
Allow user to leave interaction:
Show ‘Next Slide’ Button:
Completion Button Label:
Anytime
Show upon completion
Next Slide
Here you see a list of the default Middleware deployment procedures that you can customize to
meet your needs. Click the tabs to your left to get an overview of each deployment procedure.
Provision Middleware
This procedure clones and configures a Middleware Home and/or WebLogic Domain from the
Software Library.
Scale Up/Scale Out Middleware
This procedure clones existing WebLogic Managed Servers within a domain or adds new
managed servers to an existing domain in order to add capacity to the domain or cluster. Use
this procedure to scale up on existing hardware within the domain or scale out on new
hardware.
Deploy/Undeploy Java EE Applications
This procedure deploys, redeploys, and undeploys Java EE applications to and from WebLogic
domains. You can deploy to multiple servers or multiple domains and you can also specify the
archive files to deploy, any pre or post deploy WLST scripts, and so on.
SOA Artifacts Provisioning
This procedure clones and configures SOA Artifacts such as SOA composites, Web Service
policies, Java Platform Security configuration, Human Workflow configuration and B2B
configuration from a reference SOA Domain environment or gold image to a destination SOA
Domain.
Deploy SOA Composites
This procedure deploys SOA Composites on a selected SOA domain. The composite and the
plan can be selected from a file system accessible from the destination domain host or from the
software library.
BPEL Process Provisioning
This procedure deploys BPEL Processes on a selected Oracle BPEL Process Manager. To do
so, select BPEL Process suitcase files which are BPEL process JAR files stored as generic
components in the Software Library.
Oracle Service Bus Resource Provisioning
This procedure deploys Oracle Service Bus resources to an Oracle Service Bus Domain. You
can deploy the resources as part of a project or individually. Resources include Business
Services, Proxy Services, WS-Policies, WSDL schemas, XQuery transformations, and JARs.
Slide 89
PROPERTIES
Allow user to leave interaction:
Show ‘Next Slide’ Button:
Completion Button Label:
Anytime
Show upon completion
Next Slide
Here you can explore the recommended workflow for provisioning WebLogic domains. Click
the steps along the bottom to view more details.
• In this first step, you manually install the WebLogic software, patch WebLogic, configure
the WebLogic domain and then fully test the environment until you consider it a ‘reference
installation’ that all future installations should conform to.
• In this step you create a WebLogic Domain provisioning profile of the reference
installation; where the WebLogic binaries and the domain configuration are stored in the
software library for use as the source for future cloning operations.
• In this step you use the provisioning profile to create a customized deployment procedure.
Here you can set new inputs, like ORACLE_HOME, ORACLE_SID, and other
configuration parameters. This prevents you from having to repeat identical configuration
inputs for each deployment procedure. You can also lock the procedure so that any
operators running the procedure cannot modify it. This deployment procedure is now a
best practice procedure.
• In this step you use the best practice deployment procedure to clone WebLogic domains
with a standard, consistent configuration quickly and without errors.
Slide 90
Automate Application of WebLogic Server Patches
Patch Plans
•
•
•
•
List patches to apply
Specify targets
CC performs validation checking
Specify deployment procedure
Patch Templates
• Are created from Patch Plan
• List patches to apply + deployment options
• Do not specify targets
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Patching is an important phase of the WebLogic Server lifecycle that enables you to keep your
WebLogic Servers updated with Oracle bug fixes. However, patching has always been a very
challenging phase of the lifecycle because it is complex, risky, time consuming, and involves
downtime. With Cloud Control you can easily automate the application of patches across
WebLogic Servers.
You can automate applying one-off patches and critical patch updates across WebLogic
domains by creating Patch Plans. These plans help you create a consolidated list of patches
you want to apply as a group to one or more WebLogic targets. As you create the plan and
specify the actual targets, Cloud Control performs prerequisite checks and analyzes the patches
for conflicts and potential problems. This automated validation checking minimizes errors.
Also when creating the patch plan you specify the deployment procedure to use. WebLogic
Server has two default deployment procedures for patching, which of course you can customize
to meet our needs. The default plans let you apply patches in either parallel or rolling mode.
Applying patches in rolling mode eliminates downtime as the nodes of the cluster are patched
individually, that is, one by one. For example, if you are patching a cluster that has five nodes,
then the first node is shut down, patched, and restarted, and then the process is rolled over to
the next node until all the nodes are patched successfully. In contrast, when patching in parallel
mode, all the nodes are shut down and the patch is applied on all of them at the same time.
You can save a patch plan as a patch template. This template contains the predetermined set of
patches and deployment options saved from the source patch plan, but without the targets
specified. This can then be used to create new plans to roll out already tested patches to your
enterprise.
So automating the application of patches with patch plans and patch templates makes the
whole process simple, reliable, and fast, and avoids downtime!
Slide 91
PROPERTIES
Allow user to leave interaction:
Show ‘Next Slide’ Button:
Completion Button Label:
Anytime
Show upon completion
Next Slide
Here you can explore the recommended workflow for automating the application of WebLogic
Server patches. This process allows you to create predesigned plans based on an existing
successfully analyzed and deployable patch plan which you use to select a completely new set
of targets. Click the steps along the bottom to view more details.
• In this first step, identify and download WebLogic Server patches through My Oracle
Support which is integrated into Cloud Control.
• In this step, create a patch plan that contains all the patches you wish to apply and the
targets to apply them to. If you’re rolling out patches in a mass scale over a period of time,
apply the patches on a few WebLogic Servers to test if the patches are being applied
successfully. Cloud Control validates the patches can be applied to those targets. Also
select the deployment plan to use and then execute the deployment plan to apply the
patches.
• In this step, save the tested patch plan as a patch template which contains everything in the
patch plan except the targets.
• In this step, create patch plans from the patch template to roll out the same set of tested
patches to the rest of the WebLogic servers in your enterprise.
Slide 97
Improving Processes Summary
•
•
•
•
•
Clone components from the Software Library
Create Gold Images
Create WebLogic Provisioning Profiles
Customize default Deployment Procedures
Automate application of WebLogic Server patches
Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Now let’s summarize the best practices we covered in this improving processes stage of the
development lifecycle.
•
•
•
•
•
Clone Middleware components from the software library to ensure consistent, standardized
images and components are deployed across your environment.
Create Gold Images of tested Middleware Homes and SOA Artifacts in the software library for
later provisioning.
Create WebLogic Provisioning Profiles to act as templates for WebLogic domain provisioning.
Customize default Deployment Procedures to meet your unique provisioning requirements and
add automatic status notification.
Automate the application of WebLogic Server patches by creating patch plans and patch
templates to make the process simple, reliable, and fast and avoid downtime.
Slide 98
PROPERTIES
Allow user to leave interaction:
Show ‘Next Slide’ Button:
Completion Button Label:
Anytime
Show upon completion
Next Slide
This self-study covered the most important Middleware management best practices at each
stage of the middleware development lifecycle. Click a label for the Best Practices at each stage
of the lifecycle. You can also download a summary of the Best Practices by clicking the
Attachment link.
Plan
• Deploy Cloud Control universally and then you expand with your business-driven needs.
• Deploy Application Dependency & Performance where you need deep visibility into
SOA, Oracle Service Bus, Portal, and ADF applications.
• Deploy JVM Diagnostics where you need deep diagnostic visibility into Java
applications.
• Deploy Business Transaction Management where you need visibility into your
transactions.
• And install Real User Experience Insight to monitor and manage your user experience.
Set Operational Goals:
• Define metric thresholds on your critical components based on performance that meets
your Service Level Agreements.
• Create monitoring templates to group together thresholds for targets of the same type.
• Create template collections to group together thresholds for groups of targets of different
types.
• Create Administration Groups to monitor and manage different types of targets in the
same way.
Monitor Proactively:
• Generally, Oracle recommends to manage by exception.
• Associate template collections with Administration Groups to quickly apply standardized
monitoring settings to many targets of different types and have them applied
automatically if a new target is added to the group.
•
Set up notifications by creating rules and rule sets so you’ll be notified of impending
problems.
• Create reports to confirm that your metrics are good.
• Customize and build monitoring dashboards to meet the specific monitoring
requirements of your mission critical applications.
• Monitor diagnostic findings during peak load, high CPU utilization, high disk I/O and high
memory utilization times.
• Monitor transactions with BTM to view the operational health of transactions and the
systems involved in those transactions.
Diagnose Problems:
• Assess notifications and/or user reports to narrow down the problem to a particular part
of your enterprise infrastructure.
• Drill down from the appropriate customized summary dashboard to find the root cause of
the problem.
• View Diagnostic Findings from the Middleware Diagnostic Advisor to resolve problems
quickly with guided help.
• Create Diagnostic Snapshots in the midst of resolving problems for later root cause
analysis.
Track and Control Changes:
• View configuration history to pinpoint why a system may be behaving differently today
that was working well yesterday.
• Compare configurations to check for configuration drift.
• Create “Gold Standard” configuration snapshots that you can compare against your live
configurations.
• Customize comparison templates to specify what to ignore and what to alert on.
• Schedule regular comparisons with automatic notifications of important differences.
Improve Processes:
• Clone Middleware components from the software library to ensure consistent,
standardized images and components are deployed across your environment.
• Create Gold Images of tested Middleware Homes and SOA Artifacts in the software
library for later provisioning.
• Create WebLogic Provisioning Profiles to act as templates for WebLogic domain
provisioning.
• Customize default Deployment Procedures to meet your unique provisioning
requirements and add automatic status notification.
• Automate the application of WebLogic Server patches by creating patch plans and patch
templates to make the process simple, reliable, and fast and avoid downtime.
Slide 99
PROPERTIES
Allow user to leave interaction:
Show ‘Next Slide’ Button:
Completion Button Label:
Anytime
Show upon completion
Next Slide
More Information
Oracle offers a variety of additional channels from which you can learn more about Enterprise
Manager Cloud Control or about any Oracle products.
We at Oracle Education know your time is valuable and limited, and so we thank you for
participating in this self-paced training. We hope this course has met your expectations and
learning objectives, and wish you the best of luck in all of your endeavors.
Please read through all the tabs on this page to learn about all the Oracle resources available
for you.
Self-Study Courses
Oracle Enterprise Manager Cloud Control 12c Self-Study training is aimed specifically at
administrators who are familiar with the current releases of Enterprise Manager. It is designed
to bring existing users up to speed on the new features of Oracle Enterprise Manager Cloud
Control 12c.
The series is broken into a number of tracks, each based around one of the major themes of the
new release, and is available from Oracle University.
Instructor-led Courses
Several instructor-led courses are also included in the Oracle Enterprise Manager Cloud Control
12c curriculum, available through Oracle University.
Demos
Many free demonstrations on Oracle Enterprise Manager Cloud Control 12c are available
through the Oracle Learning Library. Here’s a list of the demonstrations that are most relevant
to this self-study.
Other Resources
The Oracle Learning Library offers many free demonstrations and tutorials.
And, of course, the Oracle Enterprise Manager Cloud Control 12c documentation and online
help embedded within the product are also valuable resources.