Enhancing
Application
Performance
Root Causes and Quick Solutions
www.regoconsulting.com
Phone: 1-888-813-0444
Enhancing Application Performance:
Root Causes and Quick Solutions
Application performance always has the potential to burden a
Clarity instance and discourage users. In this session, Rego’s
team of Clarity experts will review the root causes of Clarity
performance issues and help you identify steps to take that will
improve performance within the application and supporting
infrastructure.
2
www.regoconsulting.com
Phone: 1-888-813-0444
Enhancing Application Performance:
Root Causes and Quick Solutions
Application performance always has the potential to burden a Clarity
instance and discourage users. In this session, Rego’s team of Clarity
experts will review the root causes of Clarity performance issues and
help you identify steps to take that will improve performance within
the application and supporting infrastructure.
●
●
●
●
Importance of solid performance
Components impacting performance
Addressing performance problems
Real-world problems
○
○
○
○
Application sizing
Database tuning & I/O
User portlets & configuration
Reports
3
www.regoconsulting.com
Phone: 1-888-813-0444
Performance is critical to Clarity
deployment and usage
Performance issues impact the usability, dependability, and
scalability of a system
Usability
Dependability
Scalability
4
www.regoconsulting.com
Phone: 1-888-813-0444
Key levers we can use to improve
performance
Items driving performance are typically resolved by adjustments
to the application configuration, application server, and database
environment.
Functional & Technical
App Configuration
• Job timing
• Job conflicts
• Page layout
• Security model
• Slice settings
• Period settings
• Code optimization
• Inactivate resources
• Purge investments
Hardware Investment
App Server
• Memory
• Cores
• JVM Architecture
• JVM Memory
• JVM GC Settings
• Maintennance
Process Changes
DB Environment
• Memory
• Cores
• Shared usage
• Storage speed
• Query performance
• Long sessions
• Maintenance jobs
• Parameters
• Reporting demand
5
www.regoconsulting.com
Phone: 1-888-813-0444
Application configuration tends to force
functional settings
Application configuration issues typically require a mix of both functional and technical changes. The
changes are often caused by suboptimal configurations.
●
Job timing
○
○
○
●
Job conflicts
○
●
Minimize slice time ranges
Period settings
○
●
Browsers and the application have ‘real limits’ to what they can digest
Slice settings
○
●
Use global rights if possible
Minimize the use of instance rights
Page layout
○
●
Avoid conflicts between slicing, posting, etc.
Security model
○
○
●
Leverage evening downtime window but coordinate with DBA
Run database and system maintenance jobs after Clarity jobs complete
Avoid large data changes (e.g. post timesheets regularly)
Close time and fiscal periods.
Code optimization
○
Suboptimal SQL and Gel is the number one issue we see!
6
www.regoconsulting.com
Phone: 1-888-813-0444
Application server issues are hardware
related
Application server issues typically an increase in the capacity of the application
servers. Key indicators of an application server issue are high application cpu and heap
dumps.
● Memory
○ 6 GB for the app JVM is great – but 7 GB is worse
○ Never less than 2 GB for the BG – typically 2.25 GB is good
● Cores
○ 1 Core per JVM or BG
○ 1 Core for the OS
○ 1 Core for GC
● JVM Architecture
○ 1 or 2 BGs
○ 50 to 150 concurrent users per JVM
○ Leverage a XOG / Admin / Scheduler JVM
● JVM GC Settings
○ Parallel garbage collection if more than 1 core
● Maintenance
○ Weekly restarts may be necessary, but can typically be avoided if architecture sized
correctly.
7
www.regoconsulting.com
Phone: 1-888-813-0444
Database server issues are typically
tuning related
Database issues are often a mix of hardware, customization, and maintenance.
● Memory
○ Most common problem. Clarity LOVES memory!
● Cores
○
Uncommon problem – start with tuning and memory
● Shared usage
○
Applications need to ‘play nice’
● Storage speed
○
Clarity demands a high speed SAN. Memory compensates for IOPs….
● Query performance & Long running session
○
○
Second most common problem! Efficient SQL is critical.
Sessions persist after there are not visible on the application!
● Maintenance jobs
○ Do not run standard Oracle database stats job
● Parameters
○
Contact Rego for environment specific database parameters
● Reporting demand
○
Monitor and plan for Crystal and Webi reporting
8
www.regoconsulting.com
Phone: 1-888-813-0444
Performance data exists in a variety of
data sources
There are a variety of sources providing key information on
performance.
Rego Exchange has portlets that help monitor jobs, processes,
and database (Oracle) performance.
Log & Config Files
• App Logs
• BG Logs
• App Access Logs
• Process engine
Infrastructure
• Diagrams
• Confirm Cores
• Confirm Memory
• Confirm IO
• Confirm Network
• GC Performance
Database Reports
• Oracle advantages
• Deadlocks
• Waits
• Physical IO
• Cache Memory
• Long sessions
• Top SQL by CPU
• Top SQL by IO
• Table fragmentation
• Table configuration
9
www.regoconsulting.com
Phone: 1-888-813-0444
Performance is an iterative process
Addressing performance issues is an iterative process!!!
Start with the low hanging fruit. Issues often ‘mask’ the more
systemic issues.
Critical
Configuration
• JVM Config
• Query Tuning
• DB Health
• DB Locking
• Job Schedule
Hardware &
Tuning
• App HW
• DB HW
• DB Config
• Query Tuning
Systemic
• Performance
Assessment
• Maintenance
Program
• Configuration
Assessment
10
www.regoconsulting.com
Phone: 1-888-813-0444
Discussion & Questions
Any questions before we continue?
11
www.regoconsulting.com
Phone: 1-888-813-0444
Real-world Case: Application Sizing
Symptom
● Application heap dumps, out of memory warnings, and periodic slowness
● Database appears to be healthy
Clues
● Clarity 13 requires additional capacity (memory & cores)
● JVMs heaps were 1.5 GB or the app and 1.0 GB for the BG
● Application JVMs spending significant time garbage collecting
● Symptoms appear during peak usage time
● Database is healthy
Resolution
● Increase memory for JVMs to 6 GB for the apps and 2 GB for the BB
● Increase cores to 1 core per BG, 1 core per APP, 1 core for OS, and 1 core for GC.
● Add JVM for XOGs / Admin / Scheduler
Result
● Stable Dev environment
● Pushing to production
● If cores are limited, try memory increase first and watch GC performance
12
www.regoconsulting.com
Phone: 1-888-813-0444
Real-world Case: Database I/O and
Tuning
Symptom
● Frequent & persistent application slowness
● Database CPU and application CPU are inconsistent
Clues
● Long running sessions on the database (more than a few minutes)
● Blocking sessions
● High physical I/O
Resolution
● Moved to the Clarity stats job (Oracle) nightly & disabled the Oracle cron job
● Increased the size of the database cache (PGA and SGA)
● Rebuilt indexes associated with long running SQ
Result
● Stable and consistent production environment
● Exposed additional queries that needed to be tuned
● Exposed need to establish tuning and maintenance process
13
www.regoconsulting.com
Phone: 1-888-813-0444
Real-world Case: User Portlets and
Configuration
Symptom
● Select users long waits with select portlets and pages
● Periodic system slowness
● Oddly high database CPU
Clues
● Long running sessions on the database (more than a few minutes)
● Top SQL includes portlet queries with poor SQL
Resolution
● Tuned the queries causing the issue
● Changed the page configurations to render on filter only
● Set default filter for the portlets
Result
● Improved portlet speed
● Eliminated long running sessions (led to system issues)
● This can also apply to WebI and reports
14
www.regoconsulting.com
Phone: 1-888-813-0444
Agenda
● Integration Basics
○ Triggers
○ Methods
○ Comparing Methods
●
●
●
●
Quiz
Keys to Success
Exercise – Develop an interface
Common Interfaces
15
www.regoconsulting.com
Phone: 1-888-813-0444
Integration Basics – Triggers
● Event Based
○ This type of Interface is triggered by event in the system. (Either something got created
or updated or deleted)
● Batch
○ This type of interface is scheduled and triggered at a set time (nightly or at certain
interval, etc.). Since, batch interfaces will handle multiple instances, you want to
address transaction managements (what happens when a record fails – one fail, all
fail?)
● Manual
○ This type of Interface is manually started by the user when they are ready for data
transmittal.
16
www.regoconsulting.com
Phone: 1-888-813-0444
Integration Basics – Methods
● Flat File
○ A .CSV file ftp'd onto a server can be pulled into Clarity. The file can be processed by custom
GEL script that can be scheduled or started manually. (This is CA’s preferred method of
integration for the On-Demand Clients)
● Web Services
○ XML based messaging that makes a call via URLs, or over HTTP to request data from or push
data into clarity. This method could leverage GEL scripts, Java classes, or Stored Procedures
in the DB
○ This is the most common approach used by any industry for Integrating different systems.
Most of the big software vendors like SAP, Oracle, HP, CA have web service API’s developed
for bi-directional data exchange with their systems.
● Database Links
○ Establish a link from the Clarity DB to another system database and just pull data from one
system to another using a stored procedure or SQL statement. Best practice in this form is to
create a “view” in the source system vs. the core tables. (This is not an option for the On
Demand Clients)
● Third Party Tools
○ Leverage a third party integration tool like ITROI, etc. to build integrations.
○ Leverage an integration service – Pervasive or Task Top
17
www.regoconsulting.com
Phone: 1-888-813-0444
Comparing Methods
Available in SaaS
Rough Effort*
Examples
•
•
Components
•
•
Flat File
Web Services
DB Link
Yes
Yes
No
40-60 Hours
80-120 Hours
40-60 Hours
Send journal entry for
capital entry
Read data from any
legacy system.
Process to read
Error object
•
•
•
•
•
Create SAP project
from new Clarity
project
Auto create support
tickets in Clarity by
reading data from
ticketing system
•
Process for outbound
Java for inbound
Error Object
•
•
•
•
•
Pull non-labor
financials to Clarity
Pull resources into
Clarity
Push assignment
data from Clarity
View to read
Process to move
from view to object
Error object
*With an integration, effort is needed on both the sending and receiving application. This means that any Clarity integration will
require some effort form the support team of the system you are integrating to. The level of effort depends on the type of interface
18
www.regoconsulting.com
Phone: 1-888-813-0444
Quiz
1.
2.
3.
4.
5.
You are an On-Demand customer and you want to bring in Actuals
from your financial system.
You are an On-Premise customer and have a million Resource
Assignments that have to be loaded into your HR System for
populating Resource timesheets.
You want to import all your support-inbox emails into Clarity as
incidents or some other custom object
You want Clarity and your Project Accounting system to be in sync and
you want the changes to be sent over instantly
A project created in Clarity needs to be sent over to your Financial
System for Budget Approval. Upon approval in your Financial System,
Clarity needs to be updated so that the Project Plan is sent over to
your Timekeeping system. How would you implement this?
19
www.regoconsulting.com
Phone: 1-888-813-0444
Keys to Success
● Simpler is Better
○ With integrations, the more complex the interface is the more difficult it will be to
build and maintain. One Direction vs. Bi-Direction is simpler.
● Get it Right the First Time
○ We love agile and iterative development, but not when building an interface. Interfaces
are best done with solid waterfall requirements and signoffs.
● Integrations are recurring jobs.
○ Integration are not for performing one time data loads. Integrations are for exchanging
data between two systems on a regular basis.
● Data Ownership is Key
○ You must determine which system is the “master” vs. the “slave” of the data. One
source must be the owner of the data in case there is a conflict. Do not make the
mistake to think Clarity will be the “source” of everything. Leverage other systems to
pull summarized data vs. all detail.
20
www.regoconsulting.com
Phone: 1-888-813-0444
Keys to Success
● Error Handling / Transaction Management
○ Errors are inevitable when two different systems are being integrated. Therefore plan
to developing an error handing mechanism to handle data errors, connectivity errors,
and system outages.
○ Equally important is transaction management and performance considerations
● Trial First to Avoid Errors
○ Before you build the complete interface – try a semi-automated load to ensure the
“process” you have defined is correct.
○ Have a Testing Environment. It is really important to have test environments that
mirror the productions as much as possible and that the data is representative of
actual production data.
21
www.regoconsulting.com
Phone: 1-888-813-0444
Exercise
1. Create a csv file on your server with data for the following
columns
○
Project ID, Amount Approved, Amount Requested, Date Approved,
Approved By
2. Create a custom object in Clarity (can be a Master Object or a
sub-object of Project ) with the following attributes
○
Project ID, Amount Approved, Amount Requested, Date Approved,
Approved By
3. Create a Workflow with GEL script that will read the .csv file
on the server and load data into the custom object using XOG.
22
www.regoconsulting.com
Phone: 1-888-813-0444
Common Interfaces
● Single Sign On
○ Using a common ID, authenticate users in Clarity for login.
● Time
○ Push time from Clarity into an HR or time pay system like Ceridian, PeopleSoft, etc.
● Financials
○ Pull actuals, budgets, or rates from a financial system into Clarity
● Human Resources
○ Pull resource related data from an HR system into Clarity. This may include manager,
cost center, country, rates, skills, OBS, RBS, etc.
● Help Desk
○ Escalate tickets from a help desk system into Clarity for work management and
assignments. Potentially a feed from Clarity back to the help desk system upon
resolution.
● Sharepoint
○ Push data from Clarity to sharepoint, including project information to allow users in
sharepoint to see high level information – status, fields, milestones, etc.
23
www.regoconsulting.com
Phone: 1-888-813-0444
Common Interfaces
24
www.regoconsulting.com
Phone: 1-888-813-0444
Questions
Contact US
Thank you.
888.813.0444
Presenters:
Raghu Bongu
Bishnu Chattopadhyay
Email Contact
info@regoconsulting.com
Web Site
www.regoconsulting.com
25
www.regoconsulting.com
Phone: 1-888-813-0444