Oracle® EPM, Fusion Edition Infrastructure Performance Tuning Guide APPLIES TO EPM RELEASE 11.1.2.2, 11.1.2.3 An Oracle White Paper October 2013 Change History Date Author Version 03/2012 Ahmed Awan (Product Assurance Engineering) 1.5 10/2013 CEAL Team (Customer Engineering & Advocacy Laboratory) 2 Change Reference Disclaimer: All tuning information stated in this guide is only for orientation, every modification has to be tested and its impact should be monitored and analyzed. Before implementing any of the tuning settings, it is recommended to carry out end to end performance testing that will also include to obtain baseline performance data for the default configurations, make incremental changes to the tuning settings and then collect performance data. Otherwise it may worse the system performance. 1 TABLE OF CONTENTS INTRODUCTION ................................................................................................................. 4 1.0 PERFORMANCE OVERVIEW ................................................................................... 4 1.1 INTRODUCTION TO ORACLE® EPM SYSTEM PERFORMANCE ................................. 4 1.2 PERFORMANCE TERMINOLOGY ................................................................................ 4 1.3 UNDERSTANDING KEY PERFORMANCE DRIVERS ..................................................... 5 2.0 TOP TUNING RECOMMENDATIONS FOR EPM SYSTEM ............................. 6 2.1 TUNE OPERATING SYSTEMS PARAMETERS.................................................................... 7 2.2 TUNE ORACLE WEBLOGIC SERVER (WLS) PARAMETERS .......................................... 20 2.3 TUNE IBM WEBSPHERE SERVER (IBM WAS) PARAMETERS ..................................... 24 2.4 TUNE 64BIT JAVA VIRTUAL MACHINES (JVM) .......................................................... 33 2.5 TUNE 32BIT JAVA VIRTUAL MACHINES (JVM) .......................................................... 38 2.6 TUNE HTTP SERVER PARAMETERS ............................................................................ 39 2.7 TUNE HTTP SERVER COMPRESSION / CACHING ....................................................... 43 2.8 TUNE ORACLE DATABASE PARAMETERS ................................................................... 53 3.0 PERFORMANCE MONITORING EPM SYSTEM ............................................... 55 4.0 TUNING FINANCIAL PM APPLICATIONS ....................................................... 56 4.1 HYPERION FINANCIAL MANAGEMENT (HFM) ......................................................... 56 4.2 HYPERION PLANNING ................................................................................................ 58 4.3 ESSBASE APPLICATION LINK FOR HFM ..................................................................... 64 4.3 FINANCIAL MANAGEMENT ANALYTICS .................................................................... 68 4.5 HYPERION PROFITABILITY AND COST MANAGEMENT (STANDARD COSTING) ........ 69 5.0 TUNING FOUNDATION SERVICES .................................................................... 70 5.1 ENTERPRISE PERFORMANCE MANAGEMENT WORKSPACE ....................................... 70 5.2 ENTERPRISE PERFORMANCE MANAGEMENT ARCHITECT (EPMA) .......................... 70 5.3 HYPERION SHARED SERVICES ..................................................................................... 71 5.3 SMARTVIEW FOR OFFICE ............................................................................................ 73 6.0 TUNING ESSBASE ..................................................................................................... 75 6.1 ESSBASE AGGREGATE STORAGE (ASO) TUNING ....................................................... 75 6.2 ESSBASE BLOCK STORAGE (BSO) TUNING ................................................................. 81 2 6.3 TROUBLESHOOTING .................................................................................................... 86 7.0 TUNING REPORTING AND ANALYSIS ............................................................. 89 7.1 TUNE REPORTING AND ANALYSIS FRAMEWORK (RAF) SERVICES ........................... 89 7.2 FINANCIAL REPORTING .............................................................................................. 91 7.3 WEB ANALYSIS ............................................................................................................ 95 7.0 TUNING DATA MANAGEMENT .......................................................................... 96 7.1 TUNE DATA RELATIONAL MANAGEMENT (DRM) ................................................... 96 7.2 TUNE FINANCIAL DATA QUALITY MANAGEMENT (FDQM).................................... 98 8.0 WEB BROWSER OPTIMIZATIONS ..................................................................... 100 8.1 TUNE INTERNET EXPLORER ...................................................................................... 100 8.2 TUNE FIREFOX ........................................................................................................... 105 9.0 DISK TUNING .......................................................................................................... 107 9.1 RAID CONSIDERATION FOR EPM ............................................................................ 107 9.2 FILE SYSTEM CONSIDERATION FOR EPM .................................................................. 107 9.3 RECOMMENDED IOPS FOR EPM .............................................................................. 108 10.0 ORACLE VIRTUAL MACHINE TUNING ...................................................... 109 10.1 CONSIDERATIONS WITH OVM V3 ........................................................................ 109 10.2 REDUCE THE NUMBER OF DOM0 CPUS................................................................. 109 10.3 TUNE GUEST VM .................................................................................................. 109 10.4 TUNE PROCESS AFFINITY ...................................................................................... 109 3 INTRODUCTION This document is written for people who monitor performance and tune the components in an EPM environment. It is assumed that readers know server administration and hardware performance tuning fundamentals, web servers, java application servers and database. 1.0 PERFORMANCE OVERVIEW This chapter discusses performance and tuning concepts for Oracle EPM. This chapter contains the following sections: 1.1 1.2 1.3 Introduction to Oracle® EPM System Performance Performance Terminology Understanding Key Performance Drivers 1.1 Introduction to Oracle® EPM System Performance To maximize Oracle® EPM System performance, you need to monitor, analyze, and tune all the components. This guide describes the tools that you can use to monitor performance and the techniques for optimizing the performance of Oracle® EPM System components, for example Financial Management, Essbase, Reporting and Analysis and Planning. 1.2 Performance Terminology This guide uses the following performance terminologies: • • • • • Scalability o System's ability to perform within specification under increasing user load, data load and hardware expansion. Latency o Time between the issuing of a request and the time when the work actually begins on the request. Think time o The time a real user pauses to think between actions. Resource utilization o A consumption metric, for example, the percent of CPU usage. Response time o A time metric, for example round-trip time it takes the server to deliver a Web page. 4 • Throughput o A rate metric (requests per unit of time), for example, requests per second, bits per second. For example, if an application can handle 20 customer requests simultaneously and each request takes one second to process, this site has a potential throughput of 20 requests per second. 1.3 Understanding Key Performance Drivers To optimize your deployment, you must understand the elements that influence performance and scalability. A factor that dictates performance is called a key performance driver. Knowing how the drivers behave in combination further enhances your ability to deploy Oracle EPM system optimally, based on the unique requirements of each deployment. - Hardware Capacity - Factors such as number of servers, quantity and speed of processors, available RAM, network speed etc. - Technical Platforms Tuning – Fine tuning other third party software required for installing and running Oracle EPM products; for example: relational databases, Java application servers, Web servers, Server / Client Operating System and browsers. - Business Application Design - Application design is an important factor in system performance i.e. structure, size, and use of product features in designing applications’ databases, reports, Web data entry forms, calculations and consolidations. - Business process usage - Activities carried out by users in the normal flow of your business cycle. o Business process usage has three components: User activity—Activities available to users for data load or data entry, database processing (consolidations, copy, clear, and so on), and reporting and analysis. Rate of user activity – A number of transactions executed by one user per one hour. User concurrency—Number of users for each activity being carried out simultaneously. 5 2.0 TOP TUNING RECOMMENDATIONS FOR EPM SYSTEM Performance tuning Oracle® EPM system is a complex and iterative process. To get you started, we have created a list of recommendations to help you optimize your Oracle® EPM system performance. This chapter includes the following sections that provide a quick start for performance tuning Oracle® EPM products. Note these performance tuning techniques are applicable to nearly all Oracle® EPM products such as Financial PM Applications, Essbase, Reporting and Foundation services. 2.1 Tune Operating Systems parameters. 2.2 Tune Oracle WebLogic Server (WLS) parameters. 2.3 Tune IBM WebSphere Application Server (IBM WAS) parameters. 2.4 Tune 64bit Java Virtual Machines (JVM), including ADF parameters. 2.5 Tune 32bit Java Virtual Machines (JVM). 2.6 Tune HTTP Server parameters. 2.7 Tune HTTP Server Compression / Caching. 2.8 Tune Oracle Database Parameters. 2.9 Tune Reporting And Analysis Framework (RAF) Services. 2.10 Tune Oracle ADF parameters. Note: While the list in each of the above stated section is a useful tool in starting your performance tuning, it is not meant to be comprehensive list of areas to tune. You must monitor and track specific performance issues within your EPM application to understand where tuning can improve performance. See "Performance Monitoring EPM System" for more information. 6 2.1 Tune Operating Systems parameters Proper OS tuning might improves system performance by preventing the occurrence of error conditions. Operating system error conditions always degrade performance. The following sections describe issues related to operating system performance: 2.1.1 Linux Tuning Parameters This topic describes how to tune the Linux operating system to optimize the performance of your Oracle® EPM System. Parameter Default Value Suggested Value tcp_fin_timeout By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. 60 30 1024 4096 (32 bit) 32000 (64 bit) 1024 4096 9000-65535 1024-65535 Consult your Linux (OEL / RHEL / SLES) documentation for more information on how to permanently change the value for this parameter. Important Note: It is hardcoded into Linux kernel directly sometimes, which will imply that it is not possible to modify even with root. File descriptors (open files=ulimit –n value) If the value set for this parameter is too low, a file open error, memory allocation failure, or connection establishment error might be displayed. Important Note: When you increase the soft limit, make sure to increase the hard limit as well. The hard limit is the maximum value which can be reached before the user gets the error messages Out of file handles. Consult your Linux (OEL / RHEL / SLES) documentation for more information on how to permanently change the value for this parameter. Backlog connections queue (tcp_max_syn_backlog) When the server is heavily loaded or has many clients with bad connections with high latency, it can result in an increase in half-open connections. Consult your Linux (OEL / RHEL / SLES) documentation for more information on how to permanently change the value for this parameter. Local Port Range (ip_local_port_range) netstat is the tool to use to determine what ports are in use currently. As to the limits, the number of ports available is a 16bit unsigned integer which gives you the range 0-65535. The ports that are available for applications to bind to are 7 the reserved privileged/root ports (0-1024) plus whatever is not covered by your ephemeral port range. Important Note: cat /proc/sys/net/ipv4/ip_local_port_range will display actual values. To modify it, you’ll have to choose between 3 methods: - echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_range - sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535" - You may need to edit /etc/sysctl.conf file, to make changes to /proc filesystem permanently i.e. append the following to your /etc/sysctl.conf file: # increase system IP port limits net.ipv4.ip_local_port_range = 1024 65535 Recycling of Time Wait connections (tcp_tw_recycle) It enables fast recycling of TIME_WAIT sockets. The default value is 0 (disabled). The sysctl documentation incorrectly states the default as enabled. It can be changed to 1 (enabled) in many cases. Important Note: Known to cause some issues with host stated (load balancing and fail over) if enabled, should be used with caution. 0 1 Reuse of Time Wait connections (tcp_tw_reuse) This allows reusing sockets in TIME_WAIT state for new connections when it is safe from protocol view point. Default value is 0 (disabled). Important Note: It is generally a safer alternative to tcp_tw_recycle, especially when lot of TIME_WAITs are generated 0 1 For more information about Linux tuning, you should consult your Linux documentation. Note the above TCP/kernel tunable parameters and their corresponding values are provided as examples and rough guidelines only. You can re-adjust these and other parameters based on actual system load, usage patterns such as the number of concurrent users and sessions and so on. 8 2.1.2 Windows 2008 Tuning Parameters This topic describes how to tune the Windows 2008 operating system to optimize the performance of your Oracle® EPM System. Parameter Default Value Suggested Value MaxUserPort Under heavy loads it may be necessary to adjust the MaxUserPort. This parameter determines the availability of user ports requested by an EPM application e.g. SmartView, Planning etc. 16383 65532 normal If your user experience slow network performance (e.g. server sends data very slowly or drops the data completely) then only disable the TCP window scaling feature on Windows Server 2008. Tip: By default, the start port is 49152, and the default end port is 65535, this means there are 16383 usable dynamic ports. *Use the following “netsh” command to configure start port and the range: netsh int ipv4 set dynamicport tcp start=1025 num=64508 Important Note: If you are using firewalls to restrict traffic on your internal network you will need to update the configuration of those firewalls to compensate for the new port range. You can view the current dynamic port range on the server by using the following “netsh” command: netsh int ipv4 show dynamicport tcp Receive Window Auto-Tuning Level Windows Server 2008 has a TCP window scaling feature. The operating system can adjust the TCP receive window size to maximize the network performance. However, some old routers, firewalls and operating systems that are incompatible with the window scaling feature may cause slow data transfer or lost of connectivity. At the command prompt, run the following command: netsh interface tcp show global The default value for the autoTuningLevel parameter is listed as Receive Window Auto-Tuning Level: Important Note: In Windows 2008 SP1/SP2, If you try to disabled (auto-tuning) and then again set it to normal, then it is strongly recommended to read MS KB article: http://support.microsoft.com/kb/967224 9 Set the power option setting to "High performance" With High Performance power scheme, processors are always locked at the highest performance state. Refer to below section 2.1.2.1 for performance benchmark on two power plans. Important Note: For more details, read MS KB article: http://support.microsoft.com/kb/2207548 Tip: How-to change a power plan: 1. Under Control Panel click on Power Options. 2. Choose the “High Performance” option 3. Close the Power Option window. Balanced High Performance Disable Data Execution Prevention (DEP) On Windows Vista and later, DEP and PAE are enabled at boot time and are configured by setting values for the nx and pae parameters using the BCDEdit /set command. To disable DEP on Windows Server 2008 type the following on an elevated Command Line. bcdedit /set nx AlwaysOff /pae bcdedit /set nx AlwaysOn bcdedit /set AlwaysOff /pae Scan all Exception for ORACLE_HOME folder nx Important Note: Available documentation is here : http://msdn.microsoft.com/en-us/library/ff542202.aspx http://msdn.microsoft.com/en-us/library/ff542275.aspx Scanning mechanism for ORACLE_HOME folder When using EPM or java based products, you should turn off scanning mechanisms (for example, anti-virus, Windows Automatic Updates). Running these in conjunction with EPM or java based products may result in unexpected behavior. When enabled noticed very negative impact on performances by giving extra CPU time (up to 50%) and slowness on processes running. 10 2.1.2.1 Performance benchmark using “Balanced” versus “High Performance” HFM consolidation test showed 25% performance improvement when using the "High Performance" power plan versus the "Balanced" (default) power plan. Note on HFM application / database servers it has been observed CPU core down-clocking from 2.66GHz to 1.5GHz. This means that “Balanced” power management is in effect on the Xeon processor, which has lowered its clock speed to save power. 11 2.1.3 Windows 2003 Tuning Parameters This topic describes how to tune the Windows 2003 operating system to optimize the performance of your Oracle® EPM System. Parameter Default Value Suggested Value TcpTimedWaitDelay This parameter controls the amount of time the OS waits to reclaim a port after an application closes a TCP connection, has a default value of 4 minutes. During a heavy users load, these limits may be exceeded resulting in an address in use: connect exception. Tip: In registry set this parameter using following: HKLM\System\CurrentControlSet\Services\Tcpip\Para meters Value: TcpTimedWaitDelay Value Type: dword Data: 30 (decimal) 240 30 Disable Data Execution Prevention (DEP) Data Execution Prevention (DEP) is a set of hardware and software technologies that perform additional checks on memory to help prevent malicious code from running on a system. In Microsoft Windows 2003 Server, DEP is enforced by hardware and by software. The problem is that java and EPM programs are considered as malicious codes by default, which will have a very negative impact on performances by giving extra CPU time (up to 50%) and slowness on processes running. To completely disable DEP in Windows Server 2003, perform the following with administrative credentials: 1. Open Windows Explorer 2. Tools > Folder Options > View 3. Uncheck Hide Protected operating system files (Recommended) and Hide extensions for known file types 4. Click apply > OK 5. Browse to C:\ 6. Right click on boot.ini, select properties and ensure the “read-only” tab is unchecked and click OK 7. Edit boot.ini 8. Modify the /noexecute= For example, set /noexecute=AlwaysOff to disable DEP entirely 9. File > Save, close boot.ini file 10. Right click on boot.ini, select properties and ensure the “read-only” tab is checked and click OK 11. Reboot the computer Important Note: Available documentation is here : /noexecute=alwayso n /pae /noexecute=always off /pae 12 http://msdn.microsoft.com/en-us/library/ff542275.aspx http://support.microsoft.com/kb/875352 Scanning mechanism for ORACLE_HOME folder When using EPM or java based products, you should turn off scanning mechanisms (for example, anti-virus, Windows Automatic Updates). Running these in conjunction with EPM or java based products may result in unexpected behavior. When enabled we saw very negative impact on performances by giving extra CPU time (up to 50%) and slowness on processes running. Scan all Exception for ORACLE_HOME folder MaxUserPort The number of user-accessible ephemeral ports that can be used to source outbound connections is configurable using this parameter. Tip: In registry set this parameter using following: HKLM\System\CurrentControlSet\Services\Tcpip\Para meters Value: MaxUserPort Value Type: dword Data: 65534 (decimal) 5000 65534 Tune Windows 2003 (x86 32 bit) /3GB switch Important Note: Using only the /3GB switch allocates 1 GB to the kernel and 3 GB to the User-mode space. Therefore it is strongly recommended to implement following settings when /3GB switch is applied, in order to make sure Essbase, HFM and Windows stability: 1. Using a range of memory for the /userva=xxxx switch that is within the range of 2900 to 3030. The following sample boot.ini file demonstrates how to use the new switch to tune a server to allocate 2,900 MB of User-mode virtual memory and 1,196 MB of Kernel-mode virtual memory. This increases the available kernel space by 172 MB: [boot loader] timeout=30 default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS [operating systems] multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise" /fastdetect /NoExecute=OptOut /3GB /Userva=2900 2. Increase the SystemPages in registry: The setting being updated controls the allocable memory for the operational system caches, file caches among others, and the value being set will tell Windows to calculate the value itself on the side of the maximum allowable resources. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management Open the Data: SystemPages Enter its new value (hex): FFFFFFFF 3. After applying above settings, reboot the server. 13 2.1.4 AIX Tuning Parameters This topic describes how to tune the AIX (6.1 / 7.x) operating system to optimize the performance of your Oracle® EPM System. Parameter Default Value Suggested Value tcp_timewait By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. 1 1 14400 (2 hours) 600 (5 minutes) 150 10 150 40 1024 8192 16384 For optimal value for your Ethernet adapter, consult IBM support. 16384 For optimal value for your Ethernet adapter, consult IBM support. Tip: Set using command no -r -o tcp_timewait=1 tcp_keepidle KeepAlive packet ensures that a connection stays in an active/ESTABLISHED state. Tip: Set using command no -r -o tcp_keepidle=600 tcp_keepintvl Specifies the interval between packets that are sent to validate the connection. Tip: Set using command no -r -o tcp_keepintvl=10 tcp_keepinit Specifies the connection. initial timeout value for TCP Tip: Set using command no -r -o tcp_keepinit=40 Listen Backlog Increase this to handle peak incoming connection rates under heavy load. Tip: Set using command no -r -o somaxconn=8192 Socket send buffer size Tip: Set using command no -r -o tcp_sendspace=16384 Socket receive buffer size Tip: Set using command no -r -o tcp_recvspace=16384 User Limit Parameter It is highly recommended to apply the following values to set for user limits when running Essbase (x64bit) Server on a 64-bit AIX platform. Tip: ulimit -a command displays all the ulimit limits. time (seconds) Default unlimited file (blocks) Default unlimited data (kbytes) Default unlimited stack (kbytes) Default unlimited on 64-bit and 64 MB on 32-bit. memory (kbytes) Default unlimited coredump (blocks) Default unlimited nofiles (descriptors) Important Note: In majority of deployments Essbase does not need too many files to be open. It is pure waste of resources. Default 8192 on 64 bit and 4096 on 32 bit. 14 AIXTHREAD_COND_DEBUG Maintains a list of condition variables for use by the debugger. If the program (e.g. Essbase) contains large number of active condition variables and frequently creates and destroys condition variables, this might create higher overhead for maintaining the list of condition variables. Setting the variable to OFF disables the list. AIXTHREAD_COND_ DEBUG=OFF AIXTHREAD_COND_DEB UG=OFF AIXTHREAD_MUTEX_DEBUG Maintains a list of active mutexes for use by the debugger. If the program (e.g. Essbase) contains a large number of active mutexes and frequently creates and destroys mutexes, this might create higher overhead for maintaining the list of mutexes. Leaving the variable set to OFF disables the list. AIXTHREAD_MUTEX_ DEBUG=OFF AIXTHREAD_MUTEX_DE BUG=OFF AIXTHREAD_MNRATIO Controls the scaling factor of the library. This ratio is used when creating and terminating pthreads. Two positive values (p:k), where k is the number of kernel threads that should be employed to handle the number of executable pthreads defined in the p variable. Might be useful for applications with a very large number of threads. However, always test a ratio of 1:1 because it might provide better performance. AIXTHREAD_MNRATI O=8:1 AIXTHREAD_MNRATIO= 1:1 AIXTHREAD_MUTEX_FAST Enables the use of the optimized mutex locking mechanism. Setting the variable to ON forces threaded applications to use an optimized mutex locking mechanism, resulting in increased performance. If the program experiences performance degradation due to heavy mutex contention, then setting this variable to ON will force the pthread library to use an optimized mutex locking mechanism that works only on process private mutexes. These process private mutexes must be initialized using the pthread_mutex_init routine and must be destroyed using the pthread_mutex_destroy routine. AIXTHREAD_MUTEX_ FAST=OFF AIXTHREAD_MUTEX_FA ST=ON AIXTHREAD_SCOPE Controls contention scope. A value of P signifies process-based contention scope (M:N). A value of S signifies system-based contention scope (1:1). If fewer threads are being dispatched than expected, try system scope. AIXTHREAD_SCOPE= P -AIXTHREAD_SCOPE=S 15 YIELDLOOPTIME Controls the number of times to yield the processor before blocking on a busy lock (only for libpthreads). The processor is yielded to another kernel thread, assuming there is another runnable kernel thread with sufficient priority. If threads are going to sleep often (lot of idle time), then the YIELDLOOPTIME might not be high enough. Increasing the value from the default value of 0 may benefit you if you do not want the threads to go to sleep while they are waiting for locks. - YIELDLOOPTIME=0 -YIELDLOOPTIME=20 SPINLOOPTIME Controls the number of times the system will retry a busy lock before yielding to another process. The default value is 40. This should be increased to 500 or higher because a busy lock retry is inexpensive compared to the alternative. Use the tprof command to determine if the check_lock routine has high CPU usage. If it does, you can increase the value even more. - SPINLOOPTIME=40 -SPINLOOPTIME=500 MALLOCOPTIONS This environment variable enables the use of multiple memory heaps. In a highly threaded environment (e.g. Essbase), many threads can request memory at the same time. With only one heap, these requests can serialize and create wait situations. With this variable, multiple heaps are generated that are used to serve memoryallocations requests and reduce memoryallocation latency Disabled MALLOCOPTIONS=multi heap:16,considersize Tip: Malloc multiheap is not enabled by default. It is enabled and configured by setting the MALLOCOPTIONS environment variable. To enable malloc multiheap with the default settings, set MALLOCOPTIONS=multiheap in Essbase startup environment file. For more information about AIX tuning, you should consult your AIX vendor’s documentation. Note the above TCP/kernel tunable parameters and their corresponding values are provided as examples and rough guidelines only. You can adjust these and other parameters based on actual system load, usage patterns such as the number of concurrent users and sessions and so on. 16 Important Note to remember for Essbase only: In some cases for AIX 5.3/6.1 the default Virtual Memory Manager (VMM) settings are not optimal for an Essbase environment. Common symptoms include high paging, high system CPU time and slow overall system throughput. Therefore in this scenario, it is recommended to lower maxperm/maxclient to get good performance. Important Note to remember for Essbase BSO only: In some cases for AIX 5.3/6.1/7.x the disks are configured in JFS2 for Essbase. AIX classifies memory into two categories: file cache and working storage. Over time, memory use grows with both types of memory. AIX will eventually need to free memory and it can do so by freeing either file cache or working storage. To maximize performance, you would want AIX to release the type of memory not being used by the application. So for databases, which use working storage, you would want AIX to release file cache memory first. To do so in the past, you would put a strict limit on JFS2 file cache memory using maxclient. It is not unusual to see database servers with file cache limited to 15% of total memory. However, a strict limit is not always desirable. The new "lru_file_repage" option solves this problem. By setting "lru_file_repage=0", AIX only frees file cache memory. This guarantees working storage stays in memory, and allows file cache to grow. So in the past you might have set maxclient at 20% for database servers. Today you could set maxclient at 90% and "lru_file_repage=0". The exact setting will vary based on your application and amount of memory. Contact IBM Support if you need help determining the optimum setting. Full example on AIX for Essbase: vmo -p -o maxperm%=90 -o maxclient%=90 -o minperm%=3 -o lru_file_repage=0 Useful documentations: • http://www-01.ibm.com/support/docview.wss?uid=swg21328602 • https://www.ibm.com/developerworks/community/blogs/aixpert/entry/ne w_aix_memory_tuning_option?lang=en 17 2.1.5 Oracle Sun Solaris Tuning Parameters This topic describes how to tune the Solaris operating system to optimize the performance of your Oracle® EPM System. Note: Solaris default network tuning parameters should be sufficient in most cases. For heavy, network centric workloads, it’s always best practice to increase the size of key TCP-related parameters to improve system performance and reduce dropped connections. Parameter Default Value Suggested Value File descriptors (nofiles) Important Note: On any box that hosts an Oracle HTTP Server / WebLogic Server / Essbase server it is recommended to increase nofiles (descriptors) to optimal value. On Solaris this means setting the “rlim_fd_cur” in the /etc/system file and rebooting the box. 256 16384 60000 30000 49152 This parameter is calculated based on the network bandwidth and latency. You could follow this rule to calculate: <numbers of bits per second> * <roundtrip latency> * 8 [bits/byte] Please make sure to check your operating system documentation and release notes; there are some negative performance implications on some older versions of Solaris if you set these numbers too high. Tip: The syntax for adjusting this parameter in the /etc/system file is shown below: set rlim_fd_cur=16384 tcp_time_wait_interval It is recommended to reduce this setting to 60,000 milliseconds, or 1 minute; this is the default value starting in Solaris 9/10. By reducing the value of this entry, TCP/IP can release closed connections faster, providing more resources for new connections. Tip: The command “netstat -a | grep TIME_WAIT | wc -l” will count all of the TCP connections that are in the TIME_WAIT state. As this number approaches the maximum number of file descriptors per process, your application’s throughput will suffer dramatic degradations because new connection requests may have to wait for a free space in the application’s file descriptor table. Tip: Set using command: ndd -set /dev/tcp tcp_time_wait_interval 30000 tcp_xmit_hiwat (Solaris 9/10) / send_buf (Solaris 11) The tcp_xmit_hiwat (tcp_send_buf) and tcp_recv_hiwat (tcp_recv_buf) parameters control the default size of the send window and receive window for each TCP connection. On very fast networks, you should make sure that the values are set to at least 32K. By default, Solaris 9 changes the default settings for 18 both parameters to 48K. tcp_recv_hiwat (Solaris 9/10) / recv_buf (Solaris 11) The tcp_xmit_hiwat (tcp_send_buf) and tcp_recv_hiwat (tcp_recv_buf) parameters control the default size of the send window and receive window for each TCP connection, respectively. On very fast networks, you should make sure that the values are set to at least 32K. By default, Solaris 9 changes the default settings for both parameters to 48K. 49152 (in Solaris tcp_conn_hash_size (Solaris 9) / ipcl_conn_hash_size (Solaris 10/11) This parameter controls the size of a hash table that helps quickly locate the TCP socket’s data structure in the kernel. If the size is too small, it will result in long hash chains in each bucket that force the operating system into a linear search for the socket entry of interest, and performance will suffer accordingly. 512 (in Solaris 9) 0 (in Solaris Connection backlog This parameter controls the maximum allowable number of completed connections waiting to return from an accept call (that have completed the three-way TCP connection handshake). You should increase this parameter only if you notice that your system is dropping connections. 128 (max_q) 1024 (max_q0) 9/10) 128000 (in Solaris 11) 10/11) This parameter is calculated based on the network bandwidth and latency. You could follow this rule to calculate: <numbers of bits per second> * <roundtrip latency> * 8 [bits/byte] 1024 (in Solaris 9) In Solaris 10, the default value of 0 means that the system automatically sizes an appropriate value for this parameter at boot time, depending on the available memory. 1024 (max_q) 4096 (max_q0) Increase these values if you do see non-zero values for tcpListenDrop in the output for the following command: netstat –s | fgrep –i listendrop Increase the values in steps of 256 starting from the default value Tip: Set using commands: ndd -set /dev/tcp tcp_conn_req_max_q 1024 ndd -set /dev/tcp tcp_conn_req_max_q0 4096 Important Note: It is important to keep systems up to date with Solaris patches because the patches provide fixes for security vulnerabilities, performance, and other improvements of the OS. Certain patches specifically target the Java platform, which benefits WebLogic Server in regards to performance and functionalities. 19 2.2 Tune Oracle WebLogic Server (WLS) parameters This topic describes how to tune the WebLogic Server to optimize the performance of your Oracle® EPM System. 2.2.1 Tuning JDBC Data Sources You can improve EPM system performance and stability by properly configuring the attributes in JDBC data sources in your EPMSystem domain. 2.2.1.1 Increase the Number of Connection Pool If EPM data sources are running out of connections to the relational database then you need to set the maximum number of connections to high value for the following EPM data sources: Important Note: Reason why the default value initial and maximum capacity size are set to lower value because it is important to protect database resources. So make sure you have enough resources on the database server (available memory is a constraint). If you increase initial / maximum capacity to very high value, you may exceed the limit of PROCESSES / OPEN_CURSORS on Oracle database server. EPM Data Source Name Default Value Suggested Value EPMSystemRegistry Initial Capacity: 1 Maximum Capacity: 150 Initial Capacity: 10* Maximum Capacity: 150 *Note: Each managed server target to EPMSystemRegistry data source creates 1 pool on start-up. Ensure you have enough Oracle processes; refer to Oracle Database Tuning section of this guide. raframework_datasource Initial Capacity: 1 Maximum Capacity: 15 Initial Capacity: 1 Maximum Capacity: 150 planning_datasource Initial Capacity: 1 Maximum Capacity: 15 Initial Capacity: 1 Maximum Capacity: 150 financialclose_datasource (FCM) Initial Capacity: 1 Maximum Capacity: 30 Initial Capacity: 1 Maximum Capacity: 150 eas_datasource (Essbase Admin Services) Initial Capacity: 1 Maximum Capacity: 15 Initial Capacity: 1 Maximum Capacity: 30 calc_datasource (CalcMgr) Initial Capacity: 1 Maximum Capacity: 15 Initial Capacity: 1 Maximum Capacity: 30 profitability_datasource (Profitability) Initial Capacity: 1 Maximum Capacity: 15 Initial Capacity: 1 Maximum Capacity: 30 hps_datasource (HpsAlerter, HpsWebReports) Initial Capacity: 1 Maximum Capacity: 15 Initial Capacity: 1 Maximum Capacity: 30 discman_datasource (DisclosureManagement) Initial Capacity: 1 Maximum Capacity: 15 Initial Capacity: 1 Maximum Capacity: 30 aif_datasource (ErpIntegrator) Initial Capacity: 1 Maximum Capacity: 120 Initial Capacity: 1 Maximum Capacity: 200 Important Note: You need to determine the optimal value for the Initial and Maximum Capacity as part of your pre-production performance testing. Once optimal values are 20 determined then set the value of Initial Capacity equal to the value for Maximum Capacity in order to boost performance of JDBC in WebLogic Server web applications. Tip: Here is the example on how to size open cursors value on Oracle database, if you have a data source with 150 connections deployed on 2 servers and if you use default Statement Cache Size 10, you may open 3000 (150 x 2 x 10) cursors on your database server for the cached statements. You can increase the open cursors limit in Oracle database server. See section “Tune Oracle Database Parameters” in this guide. Tip: To increase this value: 1. Log in to the WebLogic Server administration console. 2. Select your data source (for example “EPMSystemRegistry”), and then click on Connection Pool tab, and then Maximum Capacity. Tip: Monitor the EPM data source connections from Console (Deployments>Monitoring->JDBC) i.e. “Active Connections Count” and if “wait for connections” has a greater than 0 value then increase the pool size. 21 2.2.2 Tune Connection Backlog Buffering Tune the number of connection requests that a WebLogic Server instance will accept before refusing additional requests. The Accept Backlog parameter specifies how many Transmission Control Protocol (TCP) connections can be buffered in a wait queue. Important Note: In WebLogic Server 10.3.x, it is not required to increase Accept Backlog as default listen thread Accept Backlog value is increased to 300 which is a better default value. 2.2.3 Stuck thread detection behavior Tuning WebLogic Server automatically detects when a thread in an execute queue becomes “stuck.” Because a stuck thread cannot complete its current work or accept new work, the server logs a message each time it diagnoses a stuck thread. • Some operations (for example, long running large FR / WA reports, Database / Essbase call, computation loop inside an application) can get into an infinite loop. This can cause the thread to be stuck with that operation for whatever time it takes to complete that operation. • Performance of the server can degrade significantly if many such threads get stuck. Tip: Changing the stuck threads values will only change the detection of stuck threads. This does not mean that there are no stuck threads any more. Only the reporting is changing. Meaning how long usually a specific operation should take in an EPM application. If such an operation takes longer than 600 seconds then you should increase the stuck thread value to a higher one to prevent the reporting of a stuck thread. The performance is not impacted or controlled through the stuck thread detection. WebLogic Server will not "delete" or "kill" a thread in case it becomes stuck. Use the below Stuck Threads suggested settings as a starting point, and then, after careful testing of your long running transactions, adjust as needed. EPM Managed Servers Default Value Suggested Value Planning server Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 Foundation Services server Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 EPMA Web & DataSync server Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 22 Financial Reporting server Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 Web Analysis server Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 Provider Services server Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 Profitability and Cost Management server Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 Essbase Administration Services Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 RA Framework Web Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 Calc Mgr Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 ErpIntegrator Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 FM Web Services Stuck Thread Max Time=600 Stuck Thread Timer Interval=60 Max Stuck Thread Time=600 Stuck Thread Max Time=1200 Stuck Thread Timer Interval=1200 Max Stuck Thread Time=1200 Tip: To configure stuck thread detection behavior, click on each EPM Managed Server and then click on Tuning tab: 2.2.4 Enable Native IO Performance Pack Performance can be improved by enabling native IO in production mode. WebLogic Server benchmarks show major performance improvements when native performance packs are used on machines that host Oracle WebLogic Server instances. Important Note: Always enable native I/O, if available, and check for errors at startup to make sure it is being initialed properly. Tip: The use of NATIVE performance packs are enabled by default in the configuration shipped with your distribution. You can use the Administration Console to verify that performance packs are enabled by clicking on each managed server and click on Tuning tab. 23 2.3 Tune IBM WebSphere Server (IBM WAS) parameters This topic describes how to tune the IBM WebSphere Server to optimize the performance of your Oracle® EPM System. Required documentation to read before proceeding: http://docs.oracle.com/cd/E40248_01/epm.1112/epm_install/ch08.html http://www.oracle.com/technetwork/middleware/bi-foundation/epm-webspheremanual-deploy11122-1887166.pdf http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/topic/com.ibm.websphere.base.doc/i nfo/aes/ae/welc6toptuning.html http://pic.dhe.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=%2Fcom.ibm.websp here.express.doc%2Finfo%2Fexp%2Fae%2Ftprf_hotspot_jvm.html 2.3.1 Tuning JDBC Data Sources You can improve EPM system performance and stability by properly configuring the attributes in JDBC data sources in your EPMSystem domain. 2.3.1.1 Increase the Number of Connection Pool If EPM data sources are running out of connections to the relational database then you need to set the maximum number of connections to high value for the following EPM data sources: IBM WAS behavior is the following: Each time an application attempts to access a back-end store (such as a database), it requires resources to create, maintain, and release a connection to that data store. To mitigate the strain that this process can place on overall application resources, the application server enables you to establish a pool of back-end connections that applications can share on an application server. Connection pooling spreads the connection overhead across several user requests, thereby conserving application resources for future requests. Important Note: Reason why the default value initial and maximum capacity size are set to lower value because it is important to protect database resources. So make sure you have enough resources on the database server (available memory is a constraint). If you increase initial / maximum capacity to very high value, you may exceed the limit of PROCESSES / OPEN_CURSORS on Oracle database server. EPM Data Source Name Default Value Suggested Value EPMSystemRegistry Minimum connections: 1 Maximum connections: 10 Minimum connections: 10* Maximum connections: 150 24 raframework_datasource Minimum connections: 1 Maximum connections: 10 Minimum connections: 1 Maximum connections: 150 planning_datasource Minimum connections: 1 Maximum connections: 10 Minimum connections: 1 Maximum connections: 150 financialclose_datasource (FCM) Minimum connections: 1 Maximum connections: 10 Minimum connections: 1 Maximum connections: 150 eas_datasource (Essbase Admin Services) Minimum connections: 1 Maximum connections: 10 Minimum connections: 1 Maximum connections: 30 calc_datasource (CalcMgr) Minimum connections: 1 Maximum connections: 10 Minimum connections: 1 Maximum connections: 30 profitability_datasource (Profitability) Minimum connections: 1 Maximum connections: 10 Minimum connections: 1 Maximum connections: 30 hps_datasource (HpsAlerter, HpsWebReports) Minimum connections: 1 Maximum connections: 10 Minimum connections: 1 Maximum connections: 30 discman_datasource (DisclosureManagement) Minimum connections: 1 Maximum connections: 10 Minimum connections: 1 Maximum connections: 30 aif_datasource (ErpIntegrator) Minimum connections: 1 Maximum connections: 10 Minimum connections: 1 Maximum connections: 200 Important Note: You need to determine the optimal value for the Minimum and Maximum connection as part of your pre-production performance testing. Once optimal values are determined then set the value of Minimum connection equal to the value for Maximum connection in order to boost performance of JDBC in IBM Websphere Application Server web applications. Tip: To increase this value: 3. Log in to the IBM Websphere Application Server administration console. 4. Select Resources => JDBC => Data Sources => data_source (for example “EPMSystemRegistry”) => Connection pool properties and then input some values in Minimum and Maximum connection boxes. 25 Tip: Monitor the EPM data source connections from IBM WAS Console (Tivoli Performance Viewer) i.e. “Free Pool Size” and if “Wait Time” has a greater than 0 value then increase the pool size. 2.3.2 Tune Thread Pool Each task performed by the server runs on a thread obtained from one of WebSphere Application Server’s many thread pools. A thread pool enables components of the server to reuse threads, eliminating the need to create new threads at run time to service each new request. Three of the most commonly used (and tuned) thread pools within the application server are: Web container: Used when requests come in over HTTP. Default: Used when requests come in for a message driven bean or if a particular transport chain has not been defined to a specific thread pool. ORB: Used when remote requests come in over RMI/IIOP for an enterprise bean from an EJB application client, remote EJB interface, or another application server. Use the below Thread Pool suggested settings as a starting point, and then, after careful testing of your long running transactions, adjust as needed. EPM Managed Servers Web Container Default ORB Planning server Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 Foundation Services server Minimum Size =50 Minimum Size =20 Minimum Size =10 26 Maximum Size = 300 Maximum Size = 100 Maximum Size = 50 EPMA Web & DataSync server Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 Financial Reporting server Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 Web Analysis server Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 Provider Services server Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 Profitability and Cost Management server Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 Essbase Administration Services Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 RA Framework Web Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 Calc Mgr Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 ErpIntegrator Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 FM Web Services Minimum Size =50 Maximum Size = 300 Minimum Size =20 Maximum Size = 100 Minimum Size =10 Maximum Size = 50 Tip: Thread pool settings can be changed in the administrative console by navigating to Servers => Application Servers => server_name => Thread Pool 27 Tip: Monitoring Thread Pool Size using IBM WAS Console 2.3.3 Increasing JVM Size of Admin Server In order to make sure successful deployments of several EPM web applications. It is highly recommended to tune the Admin Server java heap size. Tip: This can be done in the WebSphere Application Server administrative console by navigating to Servers => Application servers => server_name => Process definition => Java Virtual Machine. 28 Then Then You will see here that you can modify Initial and Maximum heap size. (Should be set both to 2048) 2.3.4 Tuning Servlet Caching WebSphere Application Server’s DynaCache provides a general in-memory caching service for objects and page fragments generated by the server. The DistributedMap and DistributedObjectCache interfaces can be used within an application to cache and share Java objects by storing references to these objects in the cache for later use. Servlet caching, on the other hand, enables servlet and JSP response fragments to be stored and managed by a customizable set of caching rules. Servlet caching can be 29 enabled in the administrative console by navigating to Servers => Application servers => server_name => Web container settings => Web container. The URI path to the servlet or JSP to be cached must be defined in a cachespec.xml file, which is placed inside the Web module's WEB-INF directory. 30 2.3.5 Tuning HTTP Transport persistent connections Persistent connections specify that an outgoing HTTP response should use a persistent (keep-alive) connection instead of a connection that closes after one request or response exchange occurs. In many cases, a performance boost can be achieved by increasing the maximum number of persistent requests that are permitted on a single HTTP connection. SSL connections can see a significant performance gain by enabling unlimited persistent requests per connection because SSL connections incur the costly overhead of exchanging keys and negotiating protocols to complete the SSL handshake process. Maximizing the number of requests that can be handled per connection minimizes the impact of this overhead. Also, high throughput applications with fast response times can realize a performance gain by keeping the connections open, rather than building up and closing the connection on each request. When this property is set to 0 (zero), the connection stays open as long as the application server is running. However, if security is a concern, then careful consideration should be placed on this setting, as this parameter can help prevent denial of service attacks when a client tries to hold on to a keep-alive connection. HTTP transport persistent connections settings can be set in the administrative console by navigating to Servers => Application servers => server_name => Ports. Once there, click on View associated transports for the port associated with the HTTP transport channel settings you want to change: 31 2.3.6 IBM WebSphere PMI (Performance Monitoring Infrastructure): Available via IBM WAS Console, browsing to Application servers > <App Server Name> > Performance Monitoring Infrastructure (PMI) 2.3.7 IBM WebSphere PMI (Performance Monitoring Infrastructure): Useful to monitor and analyze your application performances Available here: http://www.ibm.com/developerworks/websphere/downloads/peformtuning.html#F ORUM144 32 2.4 Tune 64bit Java Virtual Machines (JVM) Newly deployed web application server instances use default memory heap settings, which are often too small to accommodate EPM System 64bit requirements. Optimal JVM settings can be determined only by close monitoring of application server performance under peak realistic load. 2.4.1 Tuning 64bit JRockit JVM In JRockit JVM (R28), the heap grows faster than before. The JVM also ensures that the heap size grows up to the maximum Java heap size (-Xmx) before an OutofMemory error is thrown. This table lists the memory setting, the suggested value (depending on available physical memory on each server), and the Oracle JRockit Java argument: Setting Suggested Value Java Argument* Minimum heap 1024 MB -Xms1024m Maximum heap 4096 MB -Xmx4096m Important Note: Ensure that the sum of the maximum heap size (-Xmx) of all the EPM JVMs running on your server does not exceed the amount of available physical RAM. If this value is exceeded, the Operating System starts paging and performance degrades significantly. For SOA, increase the size of Compressed References to resolve native OutOfMemory (OOM) error. 32 GB -XXcompressedRefs:size=32GB min=128k preferred=512k wasteLimit=128k XXtlaSize:min=128k,preferred=512k ,wasteLimit=128k Tip: This can be done for other EPM web components besides the SOA, only if you encounter a native OutOfMemory error despite memory being available. Thread-local areas (TLA) is used for object allocation. This option (-XXtlaSize) can be used to tune the size of the thread-local areas. Important Note: Only apply this setting if you see “java.lang.OutOfMemoryError: getNewTla” error in web app logs. You may also need to consider increasing the physical memory on server to permanently resolve the “getNewTla” error occurrences. Note: It is recommended to use the default garbage collection mode i.e. -Xgc:throughput. Also large heap reduces the garbage collection frequency and fragmentation, improving the throughput of the application; however, a large heap increases the memory footprint of the Java process. 33 *Warning: Make sure you have enough physical memory allocated / available on servers (deployed with several EPM 64bit web applications) and then use the above settings as a starting point, and after load testing, adjust as needed. For a stable / an even performance over time, you should set the initial heap size (-Xms) to the same value as the maximum heap size (-Xmx). 2.4.2 Tuning 64bit Sun JVM This table lists the memory setting, the suggested value, and the Sun’s Java argument: Setting Suggested Value Java Argument* Minimum heap 1024 MB -Xms1024m Maximum heap 4096 MB -Xmx4096m Important Note: Ensure that the sum of the maximum heap size (-Xmx) of all the EPM JVMs running on your server does not exceed the amount of available physical RAM. If this value is exceeded, the Operating System starts paging and performance degrades significantly. Minimum permanent generation heap 256 MB -XX:PermSize=256m Maximum permanent generation heap 512 MB -XX:MaxPermSize=512m 512 KB -Xss512k If you receive java.lang.OutOfMemoryError: PermGen space errors, you may also need to increase the permanent generation space. Stack size for Java Threads *Warning: Make sure you have enough physical memory allocated / available on servers (deployed with several EPM 64bit web applications) and then use the above settings as a starting point, and after load testing, adjust as needed. For a stable / an even performance over time, you should set the initial heap size (-Xms) to the same value as the maximum heap size (-Xmx). 34 2.4.3 Tuning 64bit AIX JVM This table lists the memory setting, the suggested value, and the AIX’s Java argument: Setting Suggested Value Java Argument* Minimum heap 1024 MB -Xms1024m Maximum heap 4096 MB -Xmx4096m Important Note: Ensure that the sum of the maximum heap size (-Xmx) of all the EPM JVMs running on your server does not exceed the amount of available physical RAM. If this value is exceeded, the Operating System starts paging and performance degrades significantly. Garbage collector policy gencon -Xgcpolicy:gencon Nursery size 768 MB -Xmn768m Stack size for Java Threads 2 MB -Xss2m *Warning: Make sure you have enough physical memory allocated / available on servers (deployed with several EPM 64bit web applications) and then use the above settings as a starting point, and after load testing, adjust as needed. For a stable / an even performance over time, you should set the initial heap size (-Xms) to the same value as the maximum heap size (-Xmx). 2.4.4 Tuning Oracle ADF We can tune default Application Module settings for Application Module (AM) pools, connection pools & data sources as per our requirements. It's mostly depends on use or access of an Application Module. As a performance optimization, when an instance of an AM is returned to the pool in "managed state" mode, the pool keeps track that the AM is referenced by that particular session. The AM instance is still in the pool and available for use, but it would prefer to be used by the same session that was using it last time because maintaining this so-called "session affinity" improves performance. Application Pool Parameters to take into consideration: 1. Pool Behavior Parameters • Failover Transaction State Upon Managed Release (jbo.dofailover) • Row-Level Locking Behavior Upon Release (jbo.locking.mode) 35 • Disconnect Application Module Upon Release (jbo.doconnectionpooling) • Enable Application Module Pooling (jbo.ampool.doampooling) • Support Dynamic JDBC Credentials (jbo.ampool.dynamicjdbccredentials) • Reset Non-Transactional State (jbo.ampool.resetnontransactionalstate)) Upon Unmanaged Release 2. Pool Sizing Parameters • Initial Pool Size (jbo.ampool.initpoolsize) • Maximum Pool Size (jbo.ampool.maxpoolsize) • Referenced Pool Size (jbo.recyclethreshold) • Maximum Instance Time to Live (jbo.ampool.timetolive) 3. Pool Cleanup Parameters • Pool Polling Interval (jbo.ampool.monitorsleepinterval) • Maximum Available Size (jbo.ampool.maxavailablesize) • Minimum Available Size (jbo.ampool.minavailablesize) • Idle Instance Timeout (jbo.ampool.maxinactiveage) • Maximum Instance Time to Live (jbo.ampool.timetolive) See that for more details: http://docs.oracle.com/cd/E15523_01/web.1111/b31974/bcampool.htm This table lists the memory setting, the suggested value (depending on available physical memory on each server), and the Oracle Java argument: Setting Suggested Value Java Argument* Maximum Available Size (jbo.ampool.maxavailablesize) The ideal maximum number of available application module instances in the pool when not under abnormal load. When the pool monitor wakes up to do resource cleanup, it will try to remove available application module instances to bring the total number of available instances down to this ideal maximum. Instances that have been not been used for a period longer than the idle instance time-out will always get cleaned up at this time, then additional available instances will be removed if necessary to 400 -Djbo.ampool.maxavailablesize=400 36 bring the number of available instances down to this size. The default maximum available size is 25 instances. Configure this to leave the maximum number of available instances desired after a resource cleanup. A lower value generally results in more application module instances being removed from the pool on a cleanup. Referenced Pool Size (jbo.recyclethreshold) The maximum number of application module instances in the pool that attempt to preserve session affinity for the next request made by the session which used them last before releasing them to the pool in managed-state mode. The referenced pool size should always be less than or equal to the maximum pool size. The default is to allow 10 available instances to try and remain "loyal" to the affinity they have with the most recent session that released them in managed state mode. Configure this value to maintain the application module instance's affinity to a user's session. A general guideline is to configure this to the expected number of concurrent users that perform multiple operations with short think times. If there are no users expected to use the application with short think times, then this can be configured to 0 zero to eliminate affinity. Maintaining this affinity as much as possible will save the CPU processing cost of needing to switch an application module instance from one user session to another. 320 -Djbo.recyclethreshold=320 Best Practice: When you specify the length of time between application module pool cleanup passes, set all application modules to use the same Pool Polling Interval value. Since there is only a single application monitor pool monitor per Java VM, the value that will effectively be used for the application module pool monitor polling interval will be the value found in the application module configuration read by the first application module pool that gets created. Setting all application modules to use the same value ensures that this value is set in a predictable way. 37 2.5 Tune 32bit Java Virtual Machines (JVM) Newly created application server instances use default memory settings, which are often too small to accommodate EPM System requirements. For 32bit JVM in order to allocate a heap of more than 1024 MB, you need to use a 64bit JVM in order to allocate more than 1024 MB of maximum heap. See section “Tune 64bit Java Virtual Machines (JVM)” in this guide. 2.5.1 Tuning 32bit JRockit JVM This table lists the memory setting, the suggested value, and the Oracle JRockit Java argument: Setting Suggested Value Java Argument Minimum heap 1024 MB -Xms1024m Maximum heap 1024 MB -Xmx1024m Important Note: By setting minimum heap size = maximum heap size gives you a controlled environment where you get a good heap size right from the start and also to minimize minor/major garbage collections. Important Note: By default, JRockit JVM uses the dynamic garbage collection strategy to optimize for throughput. For EPM component, it is not recommended to change to other available dynamic strategies. 2.5.2 Tuning 32bit Sun JVM This table lists the memory setting, the suggested value, and the Sun’s Java argument: Setting Suggested Value Java Argument Minimum heap 1024 MB -Xms1024m Maximum heap 1024 MB -Xmx1024m Minimum permanent generation heap 128 MB -XX:PermSize=128m Maximum permanent generation heap 256 MB -XX:MaxPermSize=256m Important Note: By setting minimum heap size = maximum heap size gives you a controlled environment where you get a good heap size right from the start and also to minimize minor/major garbage collections. Important Note: Also for application servers that use Sun's Java Virtual Machine (JVM), make sure that the size of the permanent generation heap must be increased (as stated in above table). When the permanent area of the heap is too small, the JVM will do a full garbage collection of the entire heap before resizing the permanent area. 38 2.6 Tune HTTP Server parameters This topic describes how to tune the HTTP server to optimize the performance of your Oracle® EPM System. 2.6.1 IIS 7.0+ HTTP Server Tuning Parameters Default Value Suggested Value Disable “Idle Time-out (minutes)” setting for all the EPM Application Pools. 20 0 1740 0 240 50 Tip: In IIS manager, on Application Pool page, selection an application pool and then click on “Advanced Settings ...” to set this configuration. Disable “Regular Time Interval (minutes)” setting for all the EPM Application Pools. Tip: On the Application Pools page, select an application pool, and then click Recycling in the Actions pane and deselect this option. minBytesPerSecond property How-To: Navigate to Path: C:\Windows\System32\Inetsrv\ , Run the Command: Tip: If “Timer_MinBytesPerSecond” error message noted in httperr1.log file on the IIS server. It is recommended to reduce or disable the minBytesPerSecond property in IIS 7.0+ to prevent a slow client connection from being closed prematurely. Connection Timeout Tip: In order to avoid random disconnections for remote users, it is recommended to increase this value. appcmd.exe set config -section:system.applicationHost/webLimits /minBytesPerSecond:"50" /commit:apphost 120 7,200 How-To: Navigate to Path: C:\Windows\System32\Inetsrv\, Run the Command: appcmd.exe set config -section:system.applicationHost/sites "/[name='Default Web Site'].limits.connectionTimeout:02:00:00" /commit:apphost 39 Tune the following memory limits for all the EPM (64bit) Application Pools: Private Memory Limit (KB) Virtual Memory Limit (KB) = Default = Default =0 =0 Tune the following memory limits for all the EPM (32bit) Application Pools: Private Memory Limit (KB) Virtual Memory Limit (KB) = Default = Default = 1048576 = 1572864 ASP Session Timeout (hfm) Tip: Set to longer than the longest consol. 20 360 ASP Response Buffering Limit (hfm) To allow Downloads e.g. Data Extracts and avoid truncated files. 4194304 1073741824 which equates to 1GB maxAllowedContentLength (hfm) 30 MB 1073741824 which equates to 1GB Important Note: It is recommended to set no memory limits (0) only for 64bit IIS. Only force memory limits for 32bit IIS. Tip: On the Application Pools page, select an application pool, and then click Recycling in the Actions pane and deselect all the options. In HFM, during loading of the files if you see error “HTTP Error 404.13 - Not Found” then increase this value. How-To: Navigate to Path: C:\Windows\System32\Inetsrv\, Run the Command: appcmd set config "Default Web Site/hfm" /section:system.webServer/security/requestfiltering /requestlimits.maxallowedcontentlength:1073741824 Note: You may need to further increase this value depending on the total size of all the HFM files. Important Note: Use the above suggested settings as a starting point, and then, after careful testing, adjust as needed. To avoid potential performance issues, values for IIS 7.0+ parameters should be set only after considering the nature of the workload and the system capacity (i.e. CPU and memory). 40 2.6.2 Oracle HTTP Server (ohs) Tuning Oracle HTTP Server uses directives in httpd.conf file. The directives for each MultiProcessing Module (MPM) type are defined in the MIDDLEWARE_HOME/user_projects/epmsystem1/httpConfig/ohs/config/OHS/ohs_component/httpd.conf file. The default MPM type is Worker MPM. Parameters Default Value Suggested Value MaxKeepAliveRequests 100 0 KeepAliveTimeout 5 100 KeepAlive On On Timeout 300 6000 = 512 = 511 =0 = 1048 = 1000 =0 =2 = 512 = 511 = 25 = 75 = 64 =0 =2 = 1024 = 1000 = 25 = 75 = 64 =0 # WinNT MPM (this MPM is for Windows Only <IfModule mpm_winnt_module> ThreadsPerChild ListenBackLog MaxRequestsPerChild </IfModule> #worker MPM (by default OHS use multithreaded mode in UNIX) <IfModule mpm_worker_module> StartServers MaxClients ListenBackLog MinSpareThreads MaxSpareThreads ThreadsPerChild MaxRequestsPerChild AcceptMutex fcntl LockFile "${ORACLE_INSTANCE}/diagnostics/logs/${COMPONENT_TYPE}/${COMPONENT_NAME}/http_lock" </IfModule> SSLSessionCache 1. Open ssl.conf and find SSLSessionCache. It will be a line like: SSLSessionCache "shmcb:${ORACLE_INSTANCE}/diagnostics/logs/${COMPONENT_TYPE}/${COMPONENT_NAME}/ssl_scache(512000)“ 2. Update SSLSessionCache as follows: SSLSessionCache none Important Note: Use the above suggested settings as a starting point, and then, after careful testing, adjust as needed. To avoid potential performance issues, values for OHS parameters should be set only after considering the nature of the workload and the system capacity (i.e. CPU and memory). 41 2.6.3 IBM HTTP Server (IHS) Tuning IBM HTTP Server uses directives in httpd.conf file. The httpd.conf configuration file is located in the conf directory of your server installation. There is also an httpd.conf.default file, if you need to use another copy of the original file. The product provides a sample configuration file called httpd.conf.sample that illustrates basic IBM module directives and advanced security options. IBM HTTP Server also provides the admin.conf.default, magic.default, and mime.types.default configuration files. IBM HTTP Server configuration files are located in [IBM HTTP Server installation directory]\(/)conf. Parameters Default Value Suggested Value MaxKeepAliveRequests 100 0 KeepAliveTimeout 5 10 for high network bandwidth 100 for low network bandwidth KeepAlive On On Timeout 300 900 = 64 = 1920 =0 = 1024 = 4096 =0 =3 = 400 = 511 = 75 = 250 = 25 = 64 =0 = 16 = 10 = 1024 = 2048 = 75 = 250 = 64 = 64 =0 = 16 # WinNT MPM (this MPM is for Windows Only) <IfModule mpm_winnt> ThreadsPerChild ThreadLimit MaxRequestsPerChild </IfModule> #worker MPM <IfModule worker.c> StartServers MaxClients ListenBackLog MinSpareThreads MaxSpareThreads ThreadsPerChild ThreadLimit MaxRequestsPerChild ServerLimit AcceptMutex fcntl LockFile [IBM HTTP Server installation directory]/logs/accept.lock </IfModule> Important Note: Use the above suggested settings as a starting point, and then, after careful testing, adjust as needed. To avoid potential performance issues, values for IHS parameters should be set only after considering the nature of the workload and the system capacity (i.e. CPU and memory). 42 2.7 Tune HTTP Server Compression / Caching Why use Web Server Compression / Caching for Oracle® EPM products? 1. Bandwidth Savings: Enabling HTTP compression can have a dramatic improvement on the latency of responses, while improving the throughput capacity of the system. By compressing static files and dynamic application responses, it will significantly reduce the remote (high latency) user response time. 2. Improves request/response latency: Caching components (i.e. images, js and css) provide an accurate control of every cache in the proxy chain including the browser's one. Such an accurate use of the cache makes it possible to suppress the payload of the HTTP reply using the 304 status code. Minimizing round trips over the Web to revalidate cached items can make a huge difference in browser page load times. Important Note: For only LAN based usage of Oracle EPM applications, it is not recommended to enable compression / caching for HTTP servers. Therefore it is only recommended to enable compression / caching when Oracle® EPM applications are used over the WAN and remote users are accessing from high latency remote locations. Once compression is enabled in http server, it is strongly recommended to conduct series of tests with multiple concurrent users in test environment and during the test it is important to evaluate how much of CPU is typically being utilization for http server. 2.7.0.1 Web Server Compression Flow To better understand compression flow, below are the possible compression enabled scenarios that shows handling of compressed http requests / responses on Oracle and IIS web server. 43 Scenario 1 – Compression enabled on Oracle HTTP server (ohs) level: HTTP IE / Firefox Decompression Layer (gizp, deflate) Web Browser Rendering VPN (WAN) HTTP HFM:80 :19000 Compression Layer (gzip, deflate) Reporting and Analysis Web:45000 Foundations Services:28080 Planning:8300 Scenario 2 – Compression enabled on IIS HTTP server level: 44 2.7.1 Oracle HTTP Server (ohs) This topic describes how to enable compression in Oracle HTTP Server of your Oracle® EPM System. Tip: The module named "mod_deflate.so" that can compress outgoing data is now added in current Oracle HTTP Server release included in Oracle Fusion Middleware platform. a. How to Enable Compression 1. Create file “deflate.conf” and copy & paste the following lines in this file: <Location /> <IfModule mod_deflate.c> SetOutputFilter DEFLATE #Don't compress images SetEnvIfNoCase Request_URI \ \.(?:gif|jpe?g|png)$ no-gzip dont-vary #Don't compress PDFs,doc,ppt,xls SetEnvIfNoCase Request_URI \.(?:pdf|doc?x|ppt?x|xls?x)$ no-gzip dont-vary #Don't compress compressed file formats SetEnvIfNoCase Request_URI \.(?:7z|bz|bzip|gz|gzip|ngzip|rar|tgz|zip)$ no-gzip dont-vary <IfModule mod_headers.c> Header append Vary User-Agent </IfModule> </IfModule> </Location> #check whether compression ratio is working as expected <IfModule mod_log_config.c> <IfModule mod_deflate.c> DeflateFilterNote Input instream DeflateFilterNote Output outstream DeflateFilterNote Ratio ratio LogFormat '"%r" %{outstream}n/%{instream}n (%{ratio}n%%)' deflate CustomLog "|${ORACLE_HOME}/ohs/bin/odl_rotatelogs ${ORACLE_INSTANCE}/diagnostics/logs/OHS/ohs_component/deflate 43200" deflate </IfModule> </IfModule> #Important Note: Above “deflate.log” can grow large in size under high users load. Logging #degrades performance because of the (possibly significant for large file) I/O overhead. Therefore it is #recommended to disable the generation of deflate log. To disable this log, comment the section #“<IfModule mod_log_config.c>”. 2. Copy “deflate.conf” file to the “ohs_component” folder of the OHS folder (path: MIDDLEWARE_HOME/user_projects/epmsystem1/httpConfig/ohs/config/OHS/ohs_component/) 3. Enable mod_deflate module, to do this add the following to “httpd.conf” (path:MIDDLEWARE_HOME/user_projects/epmsystem1/httpConfig/ohs/config/OHS/ohs_component/) file: LoadModule deflate_module "${ORACLE_HOME}/ohs/modules/mod_deflate.so" 45 4. Add the following line to “httpd.conf” file after LoadModule lines: Include /Oracle/Middleware/user_projects/epmsystem1/httpConfig/ohs/config/OHS/ohs_component/deflate.conf Important Note: In UNIX deployment, ensure the above path is correct and update accordingly. 5. Save the changes in “httpd.conf” and Restart the Oracle HTTP Server (ohs). 6. To check that compression is enabled, open “deflate” file at ${ORACLE_INSTANCE}/httpConfig/ohs/diagnostics/logs/OHS/ohs_component Tip: In above steps the default EPM Oracle instance location is used i.e. “MIDDLEWARE_HOME/user_projects/epmsystem1”. Use your EPM Oracle instance location in case if it is different. Tip: After applying all the above steps, if OHS server isn’t starting then verify all the paths (e.g. modules) are valid For details about all directives in http://httpd.apache.org/docs/2.2/mod/mod_deflate.html . mod_deflate, see at 46 Performance Gain Test (Oracle HTTP Server) The test with/without HTTP compression enabled was conducted in internal lab to measure the transactions response time / throughput for Workspace / Planning web applications. Below table summarizes the performance improvement for Workspace / Planning transactions over a 1 Mbps bandwidth link for one remote user (ran with multiple iterations for 30 minutes). Average time (in sec) Average time (in sec) Before (Non-Compressed) After (Compressed) Workspace Logon page 56 25 55% Workspace Home page 49 32 35% Open Planning application 18 10 44% *Open Planning very large web form (approx 50,000 cells). 590 200 66% 7 5 29% Total Throughput (bytes) (Total generated network traffic) 21,587,996 2,272,578 89% Average Network Delay (ms) 400 408 - 0 0 - Transaction Name Workspace Log off Packet Loss % % Improvement *Note: The large web form size used in testing does not represent the real world situation. The form is much larger than Planning users can realistically work with. It is only used to test the maximum strength of compression ratio in OHS. Below tables shows the total response bytes (by Content-Type) for above Workspace / Planning web transactions with and without HTTP compression. Non-Compressed (Before) Compressed (After) 2,679,163 Bytes 507,839 Bytes text/css: 523,365 Bytes 58,591 Bytes text/xml: 307,959 Bytes 59,420 Bytes ~headers: 136,104 Bytes 141,638 Bytes 17,624,524 Bytes 1,264,910 Bytes image/png: 24,628 Bytes 20,891 Bytes text/javascript: 2,602 Bytes 555 Bytes Content-Type application/x-javascript: text/html (Web Form): image/gif 193,873 Bytes 193,321 Bytes image/jpeg: 18,651 Bytes 18,651 Bytes image/x-icon: 4,150 Bytes 805 Bytes text/plain: 134 Bytes 60 Bytes text/x-json: 75,835 Bytes 5,897 Bytes Note: Testing was done using single Oracle HTTP server with an Oracle Sun Solaris SPARC 4*1.2 dual core box and 16 GB RAM. In remote user machine testing was done using Internet Explorer 8.0 (IE cache was disabled) on Windows 7. 47 2.7.2 IBM HTTP Server (IHS) This topic describes how to enable compression in IBM HTTP Server of your Oracle® EPM System. Tip: The module named "mod_deflate.so" that can compress outgoing data is now added in current IBM HTTP Server release. b. How to Enable Compression 7. In file “httpd.conf” copy & paste the following lines in this file: <Location /> <IfModule mod_deflate.c> SetOutputFilter DEFLATE #Don't compress images SetEnvIfNoCase Request_URI \ \.(?:gif|jpe?g|png)$ no-gzip dont-vary #Don't compress PDFs,doc,ppt,xls SetEnvIfNoCase Request_URI \.(?:pdf|doc?x|ppt?x|xls?x)$ no-gzip dont-vary #Don't compress compressed file formats SetEnvIfNoCase Request_URI \.(?:7z|bz|bzip|gz|gzip|ngzip|rar|tgz|zip)$ no-gzip dont-vary <IfModule mod_headers.c> Header append Vary User-Agent </IfModule> </IfModule> </Location> #check whether compression ratio is working as expected <IfModule mod_log_config.c> <IfModule mod_deflate.c> DeflateFilterNote Input instream DeflateFilterNote Output outstream DeflateFilterNote Ratio ratio LogFormat '"%r" %{outstream}n/%{instream}n (%{ratio}n%%)' deflate CustomLog "/u01/IBM/HTTPServer/logs/deflate.log" deflate </IfModule> </IfModule> #Important Note: Above “deflate.log” can grow large in size under high users load. Logging #degrades performance because of the (possibly significant for large file) I/O overhead. Therefore it is #recommended to disable the generation of deflate log. To disable this log, comment the section #“<IfModule mod_log_config.c>”. 8. Enable mod_deflate module, to do this add the following to “httpd.conf” (path: /u01/IBM/HTTPServer/conf) file: LoadModule deflate_module "/u01/IBM/HTTPServer/modules/mod_deflate.so" 48 9. Save the changes in “httpd.conf” and Restart the IBM HTTP Server (IHS). 10. To check that compression is enabled, open “deflate.log” file at /u01/IBM/HTTPServer/logs Tip: Use your EPM Oracle instance location in case if it is different. Tip: After applying all the above steps, if IHS server isn’t starting then verify all the paths (e.g. modules) are valid For details about all directives in http://httpd.apache.org/docs/2.2/mod/mod_deflate.html . mod_deflate, see at 49 2.7.3 IIS 7.0+ HTTP Server This topic describes how to enable compression and caching of your Oracle® EPM System. Tip: IIS 7.0+ involves a new configuration system. The metabase.xml is no longer used. Now in IIS 7.0 compression settings are configurable from IIS Manager or via “appcmd.exe” or using the Configuration Editor. a. How to Enable Compression: 1. HTTP Compression should be enabled on the site level as performance may hindered if enabled globally. Use below commands to enable compression at the individual site (i.e. for HFM and Workspace). Tip: Run the following command line options to enable static / dynamic compression on the Workspace and hfm site levels: Workspace > Navigate to Path: C:\Windows\System32\Inetsrv\ Run the command: Appcmd.exe set config "Default Web Site/workspace" -section:urlCompression -doStaticCompression:true -doDynamicCompression:true HFM > Navigate to Path: C:\Windows\System32\Inetsrv\, Run the command: Appcmd.exe set config "Default Web Site/hfm" -section:urlCompression -doStaticCompression:true -doDynamicCompression:true 2. Only static compression is installed by default. Ensure dynamic compression feature is installed via Web Server (IIS) > Add Role Services Wizard; for step by step details refer to IIS 7.x documentation. Increase the compression level of static and dynamic files. Note: dynamic compression level default value is 0 and for static compression level default value is 7. As stated below change the level of compression (in gzip) for dynamic files to 7 and for static files to 9. Tip: Run the following command line option to increase GZip Compression Level: Navigate to Path: C:\Windows\System32\Inetsrv\ Run the command: Appcmd.exe set config -section:httpCompression -[name='gzip'].staticCompressionLevel:9 -[name='gzip'].dynamicCompressionLevel:7 Important Note: Raise the dynamic compression level above seven only if you need more network bandwidth and have sufficient CPU capacity to handle the extra load. 3. Restart the IIS HTTP Server. 50 b. How to Enable Caching in IIS: 1. Enable content expiration and set the expiration after to at least “30 Days”. 2. Run following commands via command line to configure the expires response header for the following directories (highlighted in bold): • HFM Navigate to Path: C:\Windows\System32\Inetsrv\ Appcmd.exe set config "Default Web Site/hfm/Common/theme_tadpole/images_global" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 Appcmd.exe set config "Default Web Site/hfm/Common/theme_tadpole/images_hfm" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 Appcmd.exe set config "Default Web Site/hfm/Common/theme_tadpole_hc/images_global" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 Appcmd.exe set config "Default Web Site/hfm/Common/theme_tadpole_hc/images_hfm" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 Appcmd.exe set config "Default Web Site/hfm/Common/theme_tadpole_hc/images_hfm" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 Appcmd.exe set config "Default Web Site/hfm/Images" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 • Workspace Navigate to Path: C:\Windows\System32\Inetsrv\ Appcmd.exe set config "Default Web Site/workspace/static/images" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 • raframework Navigate to Path: C:\Windows\System32\Inetsrv\ Appcmd.exe set config "Default Web Site/raframework/img" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 Appcmd.exe set config "Default Web Site/raframework/images" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 Appcmd.exe set config "Default Web Site/raframework/wsmedia/art" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 Appcmd.exe set config "Default Web Site/raframework/wsmedia/images" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 Appcmd.exe set config "Default Web Site/raframework/wsmedia/publish" /section:staticContent /clientCache.cacheControlMode:UseMaxAge /clientCache.cacheControlMaxAge:30.00:00:00 51 c. Performance Gain Test (IIS 7.0+) The test with/without HTTP compression enabled was conducted in internal lab to measure the transactions response time / throughput for Workspace / HFM web applications. Below table summarizes the performance improvement for Workspace / HFM transactions over a 1 Mbps bandwidth link for one remote user (ran with multiple iterations for 30 minutes). Average time (in sec) Average time (in sec) Before (NonCompressed) After (Compressed) Workspace Logon page 58 26 55% Workspace Home page 51 30 41% HFM_Open_Application 20 13 35% HFM_Open_Large_WDEF 42 22 48% HFM_Change_POV_Select_Entities 20 11 45% HFM_Open_Grid 26 12 54% Workspace Log off 8 6 25% Total Throughput (bytes) (Total generated network traffic) 5,523,842 1,191,405 78% Average Network Delay (ms) 405 401 - 0 0 - Transaction Name Packet Loss % % Improvement Below tables shows the total response bytes (by Content-Type) for above Workspace/HFM web transactions with and without HTTP compression. Non-Compressed (Before) Compressed (After) 3,662,585 Bytes 697,108 Bytes text/css: 404,492 Bytes 48,609 Bytes text/xml: 276,032 Bytes 56,611 Bytes text/html: 935,452 Bytes 140,035 Bytes image/png: 35,727 Bytes 41,227 Bytes text/javascript: 1,130 Bytes 429 Bytes 208,424 Bytes 207,386 Bytes Content-Type application/x-javascript: image/gif Note: Testing was done using single HFM web server (IIS 7.5) with an Intel Quad-core 3.4-GHz web server and 8 GB RAM. In remote user machine, testing was done using Internet Explorer 8.0 (IE cache was disabled) on Windows 7. 52 2.8 Tune Oracle Database Parameters This topic describes how to tune the Oracle Database to optimize the performance of your Oracle® EPM System. 2.8.1 Tuning “init.ora” Database Parameters The following table provides important “init.ora” parameters to tune for EPM products. Consider following these basic recommendations to set the database parameters. Ultimately, however, the DBA should monitor on periodic basis the database health and tune parameters. The following table provides information on some important performance-related database initialization parameters to tune for Oracle 11g R1 or above Database (64bit): Products (in an Oracle Instance) Database Parameter Shared Services, Reporting and Analysis, Financial Management (HFM), EAS, HPCM, Performance Scorecard, Planning. PROCESSES=1500 - whatever is needed to support the expected number of applications and application servers. Important Note: Further increase this value to 2500, if you have large deployment with many managed servers running e.g. 22 or 27. Set OPEN_CURSORS=7000 - the default value for this parameter is too small for systems such as WebLogic Server / HFM. - It is important to set the value of OPEN_CURSORS high enough to prevent system from running out of open cursors. - Note: assuming that a session does not open the number of cursors specified by OPEN_CURSORS, there is no added overhead to setting this value higher than actually needed. Set CURSOR_SHARING - changing this parameter to the suggested value will increase the performance, and will not impact any data results. Important Note: Suggested setting for CURSOR_SHARING: For all HFM releases prior to 11.1.2.2.300 – FORCE For all HFM releases 11.1.2.2.300 and later - EXACT Set SESSION_CACHED_CURSORS parameter. - the default value is 50. To evaluate the accuracy of the value, set to 50 and then evaluate if this is enough. If it’s not enough then increase to 100. Set OPTIMIZER_INDEX_COST_ADJ to 50 - setting of 50 makes the index access path look half as expensive as normal. 53 Financial Data Management (FDM) Quality DBA must read and follow the tuning instructions included in the FDM DBA Guide. Failure to tune the database per the FDM DBA Guide will result in poor performance. Important Note: The FDM DBA Guide is available on the Oracle Technology Network. Important Note: Above are the some good minimum values for Oracle Database instance parameters, during load testing process, adjust as needed. For more information on tuning the Oracle database, see the Oracle Database Performance Tuning Guide. Important Note: It is highly recommended to read the section “Preparing a Database” in Oracle Hyperion Enterprise Performance Management System Installation Start Here Guide. 2.8.2 Periodic Database Monitoring DBA to perform database monitoring and tuning on a periodic basis to ensure that you get the best performance from your EPM applications. It is recommended that DBA should further tune the Oracle parameters using following utilities and commands available to perform this task: • an Automatic Work Repository (AWR) report covering a sufficiently long period of heavy user activity. • an Automatic Database Diagnostic Monitor (ADDM) report that analyzes the data within the AWR report. • an Active Session History (ASH) report. • operating system Performance Monitor logs which monitor sufficient memory and CPU objects to adequately review server performance. 54 3.0 PERFORMANCE MONITORING EPM SYSTEM Oracle Fusion Middleware provides a Fusion Middleware Control that can be used to monitor all elements of the farm - including deployed applications and Fusion Middleware components such as: • WebLogic Domain • EPM Managed Servers • Oracle HTTP Server Important Note: See also "Getting Started Using Oracle Enterprise Manager Fusion Middleware Control" in Oracle Fusion Middleware Administrator's Guide @ http://download.oracle.com/docs/cd/E14571_01/admin.htm Tip: For Enterprise Performance Management 11.1.2.3 using Oracle Enterprise manager 12c: http://www.oracle.com/technetwork/middleware/bi-foundation/epm11123oem12c-monitor-1955582.pdf Installation instructions and deployments are mentioned inside the PDF directly. Example for Planning below: Important Note: Each operating system has its own native tools and utilities that can be useful for performance monitoring purposes. Native operating system commands enable you to gather and monitor for example CPU utilization %, process memory, paging activity, swapping, and other system activity information. For details on operating system performance monitoring commands, refer to the documentation provided by the operating system vendor. 55 4.0 TUNING FINANCIAL PM APPLICATIONS This chapter includes the following sections that provide a quick start for tuning Financial PM applications. 4.1 Hyperion Financial Management (HFM) Performance tuning HFM is a complex and iterative process. For detailed information about tuning the HFM, see topic "Tuning Financial Management Applications" in HFM tuning guide at http://www.oracle.com/technetwork/middleware/financialmanagement/overview/hfmperformancetuning-1934817.pdf 4.1.1 Financial Management Records and Subcubes A record in Financial Management holds the data for all base periods for a given intersection of dimension members, and a subcube is a collection of records that all belong to the same Entity, Scenario, Year, and Value (currency). Within a subcube there are a minimum of 4 dimensions: Account, ICP and Customs. (An HFM application needs a minimum of two custom dimensions starting with 11.1.2.2.000 release down from 4 in previous releases. Also the upper limit of four has been lifted). Because the subcube is a natural unit of data for the purposes of consolidation, data movement and processing are carried out on a subcube basis in many places in Financial Management. 4.1.1 Database Tuning The following housekeeping practices are recommended when using Oracle / SQL Server databases with Financial Management: • For HFM tables <appname>_DATA_AUDIT, <appname>_TASK_AUDIT and HFM_ERRORLOG, it is recommended to implement the following housekeeping best practices: • Quarterly - Business to review the Audit logs, archive and delete. • Half-Yearly - Archive System Messages and truncate table. Tip: Put alerts in place so that action can be taken if these tables grow beyond the recommended number of records (> 500,000). Note large audit tables will have severe impact on HFM performance. Important Note: For detailed information about tuning the Oracle database for HFM, see topic "Tuning Guidelines for Oracle 11g Databases" in HFM tuning guide at https://support.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=1083460.1 56 4.1.2 Basic Design Considerations 1. If Data Audit feature is not part of business requirements then it is recommended to turn off auditing of data. It has been observed degradation in performance for HFM application with Data Audit table more than 10GB. Tip: To turn off auditing of members, change the EnableDataAudit attribute to N for all members in your metadata file. 2. Rules must always be tested prior to loading in a production environment to avoid any pitfalls of poorly designed rules (may cause data explosion from rules). So efficient rules are critical for acceptable system performance. 3. For attaching multiple documents to Data Grid / Process Unit, Oracle recommends that you attach no more than three documents to a cell. Each document should be smaller than 100K to limit the performance effect on the database. Tip: You can set a size limit for document attachments and a maximum number of document attachments by user when you create an application. You can set the limits in the AppSettings attribute for the application metadata. 4. Make use of Consolidate All option only under appropriate circumstances. If this option is used the system does not skip entities with NODATA, which can have a significant impact on consolidation performance. 57 4.2 Hyperion Planning This topic describes how to tune the Oracle Hyperion Planning to optimize the performance of your Planning applications. Descriptions here are referenced also into this documentation: http://docs.oracle.com/cd/E40248_01/epm.1112/planning_admin/launch.html 4.2.1 Planning Properties You can add or change application or system properties via “Administration, then Manage Properties” For further information on Planning properties, please see the Oracle Hyperion Planning Administrator’s Guide at http://docs.oracle.com/cd/E40248_01/epm.1112/planning_admin/properts.html 4.2.1.1 Add “SMARTVIEW_COMPRESSION_THRESHOLD” system property In order to optimize large queries performance for Smart View, the optimal compression setting should be determined based on the load testing and set in the Planning System Properties accordingly: "SMARTVIEW_COMPRESSION_THRESHOLD" Important Note: By default the SmartView compression is enabled and the compression threshold is set to default value of 64K. Transmitting 64 Kbytes of data over fast network is not noticeable. Reason to set default value to too low so that it will not create unnecessary CPU overhead to server side in multi user access (with limited server side system resources). It is highly recommended to only adjust this to any value based on network speed, sufficient system resources (i.e. CPU, memory) and amount of data (forms size) to be compressed. 4.2.1.2 JDBC Connections The JDBC connection settings are set by default to a minimum of 1 and a maximum of 10. The Planning application does not function correctly if you lower the maximum JDBC_MAX_CONNECTIONS to less than 2. Tip: To reconfigure JDBC connection pool parameters, change the minimum and maximum property values: JDBC_MIN_CONNECTIONS = 10 JDBC_MAX_CONNECTIONS = 45 Important Note: Once the connection pool is increased, you may also need to increase the number of available connections the database allows. For example, you may want to increase the number of processes in the Oracle Database. 58 4.2.1.3 Allocate memory for supporting detail cache You can specify the amount of memory for the supporting detail cache. This improves performance when users change the planning unit status. To change the default memory allocation for supporting detail cache, change the SUPPORTING_DETAIL_CACHE_SIZE property. The default is 20. If the Supporting Detail Detection Cache reaches 75% or greater, Oracle recommends increasing SUPPORTING_DETAIL_CACHE_SIZE to decrease the Supporting Detail Detection Cache to 60%. 4.2.1.4 OLAP connection settings The OLAP connection settings are set by default to a minimum of 2 and a maximum of 10. Tip: Set the Planning property OLAP_MAX_CONNECTIONS to 20 if you have 100 users. If you need to after careful testing, increase the setting to 100 for 500 users 4.2.1.5 About Runtime Prompts By default, the values for processed runtime prompts in the application are stored in the database and available for viewing from the Job Console (select Tools, then Job Console). Tip: If many users are running business rules with runtime prompts, tracking these values consumes significant system resources. To improve performance, you can turn off this function so Planning does not capture runtime prompt values. To do so, add the CAPTURE_RTP_ON_JOB_CONSOLE property to the properties table, with the property value of FALSE (to turn it back on again, change its value to TRUE). 4.2.2 Application Design Considerations For further information on Planning Application Design considerations, please see the Oracle Hyperion Planning Administrator’s Guide at http://docs.oracle.com/cd/E40248_01/epm.1112/planning_admin/ch04s09.html 4.2.2.1 Clear unnecessary/historical data Create a Historical application for archiving data after the current planning cycle has completed to reduce db size in the Planning application. 59 4.2.2.2 Separate Planning and Reporting Requirements Identify requirements that are strictly for reporting purposes and consider moving the reporting requirements to a separate database to reduce the database size and improve the calculation times in the Planning application. 4.2.2.3 Dense Sparse Dimension Settings The following is the typical Planning application dense sparse configuration. Note that some Planning applications may not follow the typical configuration depending on the functional requirements of the application. • Accounts (dense) - In Planning applications, the Accounts dimension members are usually dense due to the calculation intensive nature of the dimension. Dense dimensions will calculate faster than sparse dimensions. • Time (dense) - Most Planning applications have forms set up to enter data across 12 months. Therefore, the Time periods are usually highly populated with data and thus are a good candidate for a dense dimension. Remaining dimensions (sparse) - The Year dimension might appear to be dense in cases where customers plan over multiple years. However, since years can only be added and not removed, the Year dimension is typically sparse to avoid increasing the block size over time. 4.2.2.4 Dense Members Calculated set to Dynamic Calc Set all Upper Level Members and members containing a formula in a Dense Dimension to Dynamic Calc to reduce the block size and reduce the amount of data that needs to be calculated using a script or rule. Dense dynamic calculations do not cross blocks and have a minimal impact on retrieval times. 4.2.2.5 Dimension Order Use Planning order for sparse dimensions for applications with large index files or if calculations are primarily formula calculations and partial aggregations that are run on save. In the Plannng order, dense members are in order from largest to smallest and aggregating sparse dimensions are next (smallest to largest) then nonaggregating sparse dimensions on the bottom (could be largest to smallest or smallest to largest). Use hourglass order for sparse dimensions if calculations are run in batch and are primarily full dimension aggregations using the CALC DIM or AGG statements. In hourglass order, dense members are in order from largest to smallest and sparse members are in order from smallest to largest. 60 4.2.2.6 Typical Planning dense block size The recommended block size ranges from 8 kb to 100 kb for 32 bit Essbase installations and from 8 kb to 200 kb for 64 bit Essbase installations. The block size for Planning applications is usually between 50 kb and 200 kb, although the block size can be larger or smaller depending on the application requirements. 4.2.2.7 Currency Conversion Use out of the box currency conversion only for customers where multiple reporting currencies are required. In cases where only one reporting currencies are required (e.g. all currencies converted to USD), the currency conversion functionality should be customized to minimize the number of dimensions. 4.2.2.8 Label Only Use label only member property wherever possible. This will reduce the number of blocks on disk if a block is not needed as well as the block size for dense dimensions. 4.2.2.9 Recursive Formulas Make sure there are no recursive formulas in the outline. If there are recursive formulas it typically means the formulas are not efficient. If you cannot avoid recursion then may need to use the CALCLIMITFORMULARECURSION Essbase.CFG setting to limit recursion to 31 level executions. For further information on the CALCLIMITFORMULARECURSION setting, please see the Essbase Technical Reference Guide at http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_tech_ref.pdf. 4.2.3 Database Level Settings For further information on Planning Database Design considerations, please see the Oracle Hyperion Planning Administrator’s Guide at http://docs.oracle.com/cd/E40248_01/epm.1112/planning_admin/ch04.html 4.2.3.1 Essbase database fragmentation Oracle recommends you to monitor Average clustering ratio on a regular basis to see if your database, after several calculations, is not experiencing highly fragmented situation. If so, then make sure to put in place a defragmentation process for your database. 4.2.3.2 Uncommitted Access Planning applications are set to Uncommitted Access not Committed Access. It is recommended to stay away from Committed Access with Planning applications because of the locking. The transactional overhead is too high for databases that are 61 being updated constantly. For further information on uncommitted access, please see the Essbase Database Administrator’s Guide at http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db.pdf. 4.2.3.3 Buffered I/O Access Buffered I/O, where the OS managers the cache, is typically used in Planning applications. For further information on Buffered I/O, please see the Essbase Database Administrator’s Guide at http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db.pdf. 4.2.4 Essbase CFG Settings Setting PIPEBUFFERSIZE in essbase CFG file to 65534, and Retrieval Buffer Size and Retrieval Sort Buffer Size to 100k (instead of 20 by default on 64 bits platform) could improve Web Form performances. Tip: Check those URLs for more details: For PIPEBUFFERSIZE: http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_tech_ref/pipebuffersize.html For Retrieval Buffer Size and Retrieval Sort Buffer Size: http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db/drpoptim.html#drpopti m998656 4.2.5 Data Forms For further information on Planning Database Design considerations, please see the Oracle Hyperion Planning Administrator’s Guide at http://docs.oracle.com/cd/E40248_01/epm.1112/planning_admin/ch04.html 4.2.5.1 Use Grid Diagnostics Use the grid diagnostics functionality within the Planning Tools menu to identify long running forms that can have scalability issues in a multi-user environment. The functionality will provide a diagnostics summary showing the forms that load in less than one second, between 1 second and 5 seconds and greater than 5 seconds. Forms loading in more than one second should be considered for re-design. For further information on grid diagnostics, please see the Oracle Hyperion Planning Administrator’s Guide at http://docs.oracle.com/cd/E40248_01/epm.1112/planning_admin.pdf. 4.2.5.2 Use Suppress Missing Blocks Using Suppress Missing Blocks can improve performance. Before using this feature, it is recommended that you test the impact on performance. 62 4.2.5.3 Split Larger Forms into Smaller Forms Split single larger data forms into multiple smaller data forms with fewer rows and columns. Organize data forms in folders. 4.2.5 Troubleshooting 4.2.5.1 Fragmentation Recommend scheduling a defragmentation process on a regular basis, especially on high processing applications like planning apps. During tuning exercise, run GetDBStats for Average Fragmentation Quotient, which shows value as a % from 0 100%. You should monitor to gauge when defragmentation process is needed. The following shows the fragmentation quotient levels that can cause performance issues: • Small Databases (<200 MB) – 60 % or greater • Medium Databases (<2 GB) – 40 % or greater • Large Databases (> 2GB) – 30 % or greater For further information on the Average Fragmentation Quotient, please see the Essbase Technical Reference Guide at http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_tech_ref.pdf. 4.2.5.2 Clear options when creating or refreshing application databases If data forms have a large number of members, performance may be slow when creating or refreshing applications or opening data forms. You can improve performance by clearing the Shared Members option in the Manage Database page. 63 4.3 Essbase Application Link for HFM This topic describes how to tune Oracle Essbase Analytic Link for Hyperion Financial Management (HFM) to optimize your applications. 4.3.1 EAL Tuning Documentation Please review the following articles/guides: https://blogs.oracle.com/pa/entry/essbase_analytics_link_eal_performance 4.3.2 Data Synchronization Service (DSS) Tuning Few requirements before proceeding: • EAL is installed on a machine in which there’s 4 to 8 cores and 5to 8 GB RAM available. • EAL is installed on Windows 2003 SP2 or 2008 R2 64 bits • EAL is installed using high speed disks When configuring DSS, as shown, below: Important field to define for good performances: • "Job Units" as <Number of Cores dedicated to DSS> * 1.5 • "Max Memory Size" ~1G for each N.B: If DSS machine is 32-bit - max memory size is 2600 MB. • "Data Store Size" - depends on number of bridges and volume of HFM applications, but in most cases 50000 MB is enough. This volume should be available in defined "Data Store Dir" driver. Job Unit. 64 4.3.3 Monitoring EAL DSS: In order to monitor especially what’s going on into EAL Data Synchronization Service, a command line is directly available onto the machine hosting EAL: Open a DOS Command window and type the following: hrutil help Usage: hrutil command [command parameters] [general options] Any command may use the following general options: [-host:<host>] [-port:<portnumber>] [-user:<user>] [-password:<password>] List of commands: version Shows the hrutil version information. sqlquery <qid> Print a Query Statement queryinfo [-noconnections][-mode][-state][-sql][-dsn][-mem] List connections and their SQL queries. killquery <qid> [-q] Kill a running query. If keyword 'all' is used instead of qid, then all queries will be killed. If option [-q] is used, no questions will be asked. updatecol <hdf> The updatecol command tells HyperRoll that a HDFÆs column types have changed. This feature will allow the user to update the columns names, type, and width. This action requires setting hrcmd to the off command and then to the resume command on the relevant cubes. Note: After column type or width has been changed, data validation on registration layer becomes invalid. help Prints this help. path hyperini hyperlog odbcini odbclog oeslog esslog. Provides the full path of the file for each option.. debug <on|off> [facility] Turns ON or OFF the specified debug facility. Supported debug facilities: api, select, source, mirror, sql, sqlscan, sqlprep, reserv, grid, alg. 65 printhier <HierarchyID> <FileName> Prints the structure of the designated hierarchy. The value for <hierID> is found in printids. printdim <DimesionID> <FileName> Prints the structure of the designated dimension in a file. The value for <dimID> is found in printids note <text note> Inserts a note to HyperRoll Log. printids ([all] | dbs | cubes) [-short] When set to all, this command prints the entire path of the IDs of all objects, including databases and cubes. When set to dbs, this command prints only the IDs of the databases. When set to cubes, this command prints only the IDs of the cubes. Added the option ûshort will print the name of the cube without the whole path. getconf [K] Provides relevant information on the configuration of HyperRoll. By default, this information is given in MB. If [k] is added, the information is given in KB. monitorjobs [detailed] Status of jobs that are running in HyperRoll. proclist Shows a list of processes and their status. pidof dbmanager|fbs|named|misc|auth Display process id of a process. start dbmanager|fbs|named|misc|auth Start the designated process. stop <proc> Stops the designated process. Examples: Hrutil getconf should provide the following: ======= HyperRoll information ======= Memory: PhysicalRAM(8191M 3125M) HR(3988M 2862M) VMem(1015M) Query Limit(5000M) Res(70M) RegRes(70M) CPU time: 1175 +1175 Calendar time: 1380115648 +1380115648 66 CPU usage is 0.0% in last 1380115648 seconds IO: 212926 reads, 5567 writes server version: 2.2.13.155 LLP64(P64) client version: 2.2.13.155 ILP32 Max Clients: SQL=0 QSR=0 Queries: peak=1 current=0 past=2807 Stores: id limit size avail name 1 10000 29234 14855 D:/Oracle/EssbaseAnalyticsLink/Work/Store Maximum number of init jobs: 4 Workers:speed working mode treeSz / treeSzB vldTime 0.0000 suspend query-only 0G 0G 0 0.0000 running all-jobs 0G 0G 0 0.0000 suspend all-jobs 0G 0G 0 0.0000 suspend query-only 0G 0G 0 0.0000 suspend light-weight 0G 0G 0 0.0000 suspend light-weight 0G 0G 0 0.0000 suspend all-jobs 0G 0G 0 0.0000 suspend query-only 0G 0G 0 0.0000 suspend all-jobs 0G 0G 0 0.0000 suspend query-only 0G 0G 0 0.0000 suspend query-only 0G 0G 0 0.0000 suspend query-only 0G 0G 0 0.0000 suspend query-only 0G 0G 0 0.0000 suspend query-only 0G 0G 0 0.0000 suspend query-only 0G 0G 0 0.0000 suspend query-only 0G 0G 0 0.0000 suspend query-only 0G 0G 0 0.0000 suspend query-only 0G 0G 0 ===================================== Where, all-jobs workers – are JOB_UNITS query-only workers – are QUERY_ONLY_UNITS light-weight workers – are responsible for EAL – DSS APIs execution (not relevant in your case) 67 4.3 Financial Management Analytics This topic describes how to tune Oracle Hyperion Financial Management Analytics (OFMA) to optimize your applications. 4.3.1 OFMA Tuning Documentation Please review the following official documentation: http://docs.oracle.com/cd/E40248_01/epm.1112/ofma_11123000_admin/apa.html 4.3.2 Setting Caching for BI Server Please review official documentation section: http://docs.oracle.com/cd/E40248_01/epm.1112/ofma_11123000_admin/apas02.html 4.3.3 Setting caches for BI Server in a clustered environment Please review official documentation section: http://docs.oracle.com/cd/E40248_01/epm.1112/ofma_11123000_admin/apas03.html 4.3.4 Turn BI Server logs Off in PRODUCTION Please review official documentation section: http://docs.oracle.com/cd/E40248_01/epm.1112/ofma_11123000_admin/apas04.html 68 4.5 Hyperion Profitability and Cost Management (Standard Costing) This topic describes how to tune Oracle Hyperion Profitability and Cost Management (Standard Costing) to optimize your applications. 4.5.1 HPCM Tuning Documentation Please review Appendix F of the Hyperion Profitability and Cost Management Administrator’s Guide (http://docs.oracle.com/cd/E17236_01/epm.1112/hpm_admin.pdf) 4.5.2 Outline Optimization for Calculation Performance Outline optimization for calculation performance was posted in the CEAL Blog in June 2012 (https://blogs.oracle.com/pa/entry/hpcm_11_1_2_x) A few other tuning techniques can be used to optimize HPCM: • Calculation optimization for multiple POV was posted in the CEAL Blog in December 2011: https://blogs.oracle.com/pa/entry/hpcm_11_1_2_optimising • Commit Block Tuning related to HPCM was posted in the CEAL Blog in January 2012: https://blogs.oracle.com/pa/entry/essbase_11_1_2_commit 69 5.0 TUNING FOUNDATION SERVICES This chapter describes how to tune the foundation services to optimize the performance of your EPM applications. 5.1 Enterprise Performance Management Workspace This topic describes how to tune the Foundation Services to optimize the performance of your EPM system. • The new Foundation Services Web application consolidates to one JVM and includes Shared Services and EPM Workspace web applications. Increase the Foundation Services JVM memory settings as stated in section "Tuning Java Virtual Machines (JVM)"of this guide. • To resolve slow logon to EPM Workspace web, ensure that all integrated applications are started or disable integrated applications that are not started, on the Workspace Server Settings panel. To access Workspace Server Settings, select Navigate, then Administer, and then Workspace Server Settings. See Oracle Enterprise Performance Management Workspace Administrator's Guide. 5.2 Enterprise Performance Management Architect (EPMA) This topic describes how to tune the following EPM Architect components to optimize the performance of your EPM applications. • Performance Management Architect Web application (EPMA Web) and Data Synchronizer Web application: Increase the JVM memory settings as stated in section "Tuning Java Virtual Machines (JVM)" of this guide. 5.2.1 Basic Performance Consideration • EPMA creates transactions in the Performance Management Architect database. Since the database size can increase over time, you can use the “Transaction History Purge Utility” to remove transactions from the database and reduce the database size. For using “Transaction History Purge Utility” refer to “EPMA Administrator Guide” on page 118. (http://docs.oracle.com/cd/E40248_01/epm.1112/epma_help/epma_trans_purge.html). Tip: Optimize EPMA relational databases size by deleting unnecessary transactions greatly improves response time. Important Note: Using this utility can generate a large number of native database log entries. A large number of these entries can significantly increase the storage used by your database. You can use a variety of methods to manage transaction logging space, depending on your database type (Oracle, SQL Server). Note: Reclaiming logging space is a task that should be performed by a database administrator. 70 5.3 Hyperion Shared Services This topic describes how to tune the Shared Services to optimize the performance of your EPM system. For 11.1.2.x: • The new Foundation Services Web application consolidates to one JVM and includes these Web applications: Shared Services and EPM Workspace. Increase the Foundation Services JVM memory settings as stated in section "Tuning Java Virtual Machines (JVM)"of this guide. • Use the default log level <WARN> in usual circumstances and <DEBUG> only for debugging. • Login performance is better when the number of nesting’s within the group membership hierarchy is less than 5. Note the performance has improved for the native groups in the 11.1.2 release because native groups are now cached along with external groups. There are no more extra look-ups to Native Directory for the native group membership. • Set up a filter to retrieve only the required users. Oracle recommends that you set the group URL and tune the group filter to decrease the number of groups that Shared Services must parse to build the cache. Doing so improves runtime performance significantly. • Use native groups, not external groups, to provision external users, and clear the use group’s option on the groups tab of LDAP/MSAD provider configuration panel. • Limit the number of users with EPM System access. Always define a User URL and set it as deep as possible and also set a user filter, if possible. • Consider to fine tune more parameters i.e. MaximumSize, Max Connections, settings, Cache Refresh Interval etc during "Configuring User Directories" process (for more information refer to Oracle EPM User and Role Security Guide). • For the first item in the search order, specify the directory from which the greatest number of users logs in. Find an optimal starting point for searches or create a custom group hierarchy. If you see error “LDAP response read timed out timeout used: 120000ms” in SharedServices_SecurityClient.log then lower the default value of this timeout for external providers. This can cause Essbase Agent process to hang so lowering this timeout value may help to resolve the Essbase agent hang issue. • 71 For 11.1.2.3: • Login performance is improved when users login after the EPM System is idle for some time, by recreating the master context for the LDAP/MSAD provider periodically. So, this connection is closed even before it is terminated at the Firewall or Load Balancer • Set <masterContextTimeToLive> attribute within the LDAP/MSAD provider in the CSSConfig file. Default value is 15 minutes. Best practice is to set this value to 1 or 2 minutes less than the Firewall or Load Balancer timeout. Time is in minutes <ldap name=“MyLDAP”> … <url>ldap://myldap.oracle.com:389/dc=oracle,dc=com</url> <masterContextTimeToLive>10</masterContextTimeToLive> … </ldap> • It is possible that group cache could be empty if there is some error while building it. Thus it is empty till the next cache refresh interval. During this time if a user is logged in who is provisioned via groups, won’t get proper roles. • Now group cache is attempted every time a request comes to group cache and gets built eventually 72 5.3 SmartView for Office This topic describes how to tune the SmartView for Office to optimize the performance of your EPM system. • SmartView uses browser cache to store locally information in XML format. If you are performing constantly different requests from the same Excel Sheet, performances could slow down because the cache is growing. A quick test should be to open a new Excel sheet, which should open a new cache area and could improve performances, because access to cache will be quicker. • Select the option to Use Excel Formatting. Without this option, Smart View will reformat each cell based on the grid operations you perform. It will also mark cells as “to calc” each time you change data values. These activities add system overhead to all Refreshes, making them taking longer. • The most efficient performance will result from small, focused queries on a specific subset of data. Zooming in on extremely large dimensions will cause slow performance. If you absolutely must zoom in on larger dimensions, suppressing rows that have no data or zero data will help. • Keeping the default options for displaying No Data, No Access, and Invalid cells will decrease the time needed for a retrieve. The difference in retrieval timing is much more noticeable when using very large (10,000+ rows) queries. Tip: This is due to XML storage on the client browser that will be less large than expected so the cache is locally quicker to build, and retrieve. 73 • The bigger your Excel workbook, the slower your Smart View queries will run. This is why running a query in an Excel workbook that has many other worksheets can take a long, long time. Retrieval performance will be optimized if the Smart View query is the only tab in the workbook. If other worksheets are being used, the overall size of the file itself will have the biggest impact on performance. Queries used in larger files (over 10 MB) are very likely to run slowly. Tip: It is better to build links between workbooks that between Excel sheets if you need to manage different set of information. • Turn off the preserve formulas option when retrieving against an ASO Essbase cube. This has been known to make retrieves taking much longer than normal. It also can dramatically increase the size of the Excel file itself when this setting is on. • The Excel AutoRecover option can create performance issues or even cause the query to crash. For example, if the AutoRecover option is set to every 3 minutes, a query that takes longer than 3 minutes to retrieve will cause Excel to crash. Disable this option if you are using large queries. 74 6.0 TUNING ESSBASE This chapter includes the following sections that provide a quick start for tuning Essbase BSO / ASO applications. 6.1 Essbase Aggregate Storage (ASO) Tuning For full design, tuning and limits details refer to the Oracle EPM Essbase Documentation Release 11.1.2.3 http://docs.oracle.com/cd/E40248_01/nav/portal_3.htm The key Essbase guides that should be reviewed for all design, tuning and limits considerations are: • Oracle Essbase Database Administrator's Guide • Oracle Essbase Technical Reference • Oracle Essbase Administration Services (Online Help) The following topics are additional design, tuning and limits considerations to optimize Essbase Aggregate Storage (ASO) performance; however, for full details see the guides referenced above. 6.1.1 Outline See the Oracle Essbase Database Administrator's Guide for complete details regarding Aggregate Storage Outline design and tuning. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db.pdf The following sections are to help guide with tuning areas. Only methodical performance and load testing can determine the optimal settings for a given application design and process. 6.1.1.1 Compacting the Outline File It is recommended to include an ASO outline compacting process as part of a normal maintenance process. When you delete members from an aggregate storage outline, the corresponding records of members in the outline file (.otl file) are marked as deleted but remain in the file. NOTE: Compacting the outline file will not cause Essbase to clear the data. 6.1.1.2 Compression Dimension The choice of compression dimension can significantly affect performance. A good candidate for a compression dimension is one that optimizes data compression while maintaining retrieval performance. 75 Another consideration when choosing a compression dimension is how well it is expected to compress the database. The size of the compressed database changes depending on which dimension is tagged as compression. Selecting the best compression dimension requires methodical testing to ensure the appropriate setting is selected for a given design, process and data distribution. NOTE: Changing the compression dimension triggers a full restructure of the database. 6.1.1.3 Dynamic Calculations Log Messages The Essbase ASO application log has some informative messages related to dynamic calculations that can be helpful with tuning. The following message indicates that the ASO optimization member needs more memory to process the MDX member formulas. In this case, consider adding MAXFORMULACACHESIZE configuration setting to the Essbase.cfg file. • Not enough memory for formula execution. Set MAXFORMULACACHESIZE configuration parameter to [xxxx] KB and try again. For additional information on ASO dynamic calculation log messages, see the blog post https://blogs.oracle.com/pa/entry/epm_11_1_2_understanding 6.1.1.4 MDX Outline Formulas See the Oracle Essbase Technical Reference for complete details regarding Aggregate Storage MDX Outline Formulas. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_tech_ref.pdf The NONEMPTYMEMBER and NONEMPTYTUPLE properties enable MDX in Essbase to query on large sets of members or tuples while skipping formula execution on non-contributing values that contain only #MISSING data. Because large sets tend to be very sparse, only a few members contribute to the input member (have non #MISSING values) and are returned. As a result, the use of NONEMPTYMEMBER and NONEMPTYTUPLE in calculated members and formulas conserves memory resources, allowing for better scalability, especially in concurrent user environments. Example: Tuning MDX formula using a NON EMPTY directive: 1. The measure member [Units] is a loaded value 2. The measure member [Avg. Units/Transaction] is calculated using formula 76 [Avg. Units/Transaction] = [Units]/[Transactions] 3. The value of [Avg. Units/Transaction] is empty if value of [Units] is empty 4. The calculation of [Avg. Units/Transaction] can be skipped if value of [Units] is empty 5. We can add the NON EMPTY directive for [Avg. Units/Transaction] by rewriting the formula as [Avg. Units/Transaction] = NONEMPTYMEMBER [Units] [Units]/[Transactions] The application log will contain hints on which MDX formulas could benefit with using the NON EMPTY directives. The query performance improvements obtained from defining NON EMPTY directives properly in each MDX outline member formulas can be very significant. The performance improvements increase with the complexity of the formula and sparseness of the data. 6.1.2 Database Level Settings See the Oracle Essbase Database Administrator's Guide for complete details regarding Aggregate Storage Database Settings tuning. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db.pdf The following sections are to help guide with tuning areas. Only methodical performance and load testing can determine the optimal settings for a given application design and process. 6.1.2.1 Pending Cache Size Limit (MB) Aggregate storage Pending Cache Size Limit (MB) memory is utilized during data loads, aggregations, and retrievals. Pending Cache Size Limit (MB) is not used during dimension building or outline restructuring. Aggregate storage Pending Cache Size Limit (MB) is allocated on demand. Typically, ASO databases with > 20 million input level cells, default of 32 MB cache is sufficient. ASO databases with < 20 million input level cells, 64 to 128 MB cache may be more appropriate. ASO databases with 1 billion cells or more, the Pending Cache Size Limit may be set as high as 512 MB or 1 GB if the available memory permits it. However, it is not recommended to set Pending Cache Size Limit (MB) to more than half of the machine's total physical memory. The reason is that it is beneficial to leave space for the operating system's file cache. Also, there is a diminishing benefit to making the Pending Cache Size Limit (MB) larger and larger. For example, there is a huge benefit to increasing the Pending Cache Size Limit (MB) from the default 32 MB to 64 or 128 MB; however, beyond about 1GB, the effect of 77 increasing the cache will probably not be noticeable unless running parallel data loads. Tuning Pending Cache Size Limit (MB) requires methodical testing to ensure the appropriate setting is selected for a given design, process and data distribution. 6.1.2.2 Retrieval Buffer Size The retrieval buffers are a server buffer per database that holds extracted row data cells per user query. When the retrieval buffer is full, the rows are processed and the buffer is reused. If these buffers are too small, frequent reuse of the area can increase retrieval times. If these buffers are too large, too much memory may be used when concurrent users perform queries while also increasing retrieval times. Important Note: It is recommended to start with default values and incrementally increase during load testing using the largest user reports. Since this setting is per user report request, it may lead to wasted memory so only set to what is needed. Also note the retrieval buffers are allocated per thread. Oracle recommends that you do not exceed 100 KB, although the size limit is set at 100,000 KB. 6.1.3 Data Load See the Oracle Essbase Database Administrator's Guide for complete details regarding Aggregate Storage Data Load tuning. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db.pdf The following sections are to help guide with tuning areas. Only methodical performance and load testing can determine the optimal settings for a given application design and process. 6.1.3.1 Incremental Data Updates When performing incremental updates to an Essbase Aggregate Storage (ASO) outline, depending on the nature of the incremental updates, could trigger either a full or light outline restructure. For additional information on ASO incremental data updates, see the blog post https://blogs.oracle.com/pa/entry/essbase_aggregate_storage_aso_incremental 6.1.3.2 Merge Data Slices When performing an Aggregate Storage (ASO) application copy, it is necessary to merge all data slices first so that all data in the ASO application will be copied properly. For additional information on ASO merging data slices before copying an ASO application, see the blog post https://blogs.oracle.com/pa/entry/merge_data_slices_before_copying 78 6.1.3.3 Table Space The ASO kernel stores data in two different table spaces, “default” and “temp.” The “default” table space is used to store cube cells, both level zero input cells and aggregated cells. The “temp” table space is used for intermediate storage of cells during data load, aggregation, and large queries. There could be data load improvement by separating the two table spaces into two difference physical drives. 6.1.4 Aggregation See the Oracle Essbase Database Administrator's Guide for complete details regarding Aggregate Storage Aggregate Views tuning. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db.pdf The following sections are to help guide with tuning areas. Only methodical performance and load testing can determine the optimal settings for a given application design and process. 6.1.4.1 Aggregation Methods The default selection of aggregate views proposed by Essbase provides excellent performance for most ASO applications. To optimize aggregations for different database retrieval situations, such as for generating reports or user queries, you may need to repeat the tuning process, creating an aggregation script for each situation. There are several aggregate view options in EAS Console and MaxL that should be methodically tested to ensure the appropriate aggregation view option is selected for a given design, process and data distribution. Ref to the Oracle Essbase Database Administrator's Guide and Oracle Essbase Technical Reference to review these areas: • Default Aggregation Option • Stopping When Total_Size Exceeds a Certain Size Option • Enable Alternate_Rollups Option • Query Tracker Enabled Option • ASOSAMPLESIZEPERCENT Essbase.cfg setting to control the views generated more efficiently 6.1.5 Essbase.cfg Settings See the Oracle Essbase Technical Reference for a complete list of configuration settings that apply to Aggregate Storage applications. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_tech_ref.pdf 79 The following sections are to help guide with tuning areas. Only methodical performance and load testing can determine the optimal settings for a given application design and process. 6.1.5.1 Suggested ASO CFG Settings to Review: CFG Setting AGENTTHREADS AGTSVRCONNECTIONS SERVERTHREADS For additional information on these settings, see the blog post https://blogs.oracle.com/pa/entry/essbase_11_1_2_agtsvrco nnections NETDELAY NETRETRYCOUNT For additional information on these settings, see the blog post https://blogs.oracle.com/pa/entry/epm_11_1_2_in2 NO_HOSTNAME_LISTCON NECT For additional information on this setting, see the blog post https://blogs.oracle.com/pa/entry/epm_11_1_2_add ASOSAMPLESIZEPERCENT For additional information on this settings, see the blog post https://blogs.oracle.com/pa/entry/epm_11_1_2_aasosample sizepercent MAXFORMULACACHESIZE http://docs.oracle.com/cd/E10530_01/doc/epm.931/html_es b_techref/config/maxformulacachesize.htm It is strongly recommended that the default value be used than incrementally increase the value during methodical performance and load testing to find the best setting. It is strongly recommended to not randomly set Essbase configuration setting values high without understanding what the setting does and how it works. Also, methodical testing will uncover issues with higher then needed settings. For instances, it is recommended to set the NETDELAY / NETRETRYCOUNT timeout settings to default values, which is 2 minutes. If need be, the max amount of timeout that is recommended should not be above 5 minutes. The reason is because there have been customers reporting Essbase “hangs”; However, technically there is no “hang” until this “hang” exceeds the NETDELAY / NETRETRYCOUNT settings. Meaning, if timeout is set to 17 minutes and the “hang” resolved in 10 minutes, this would not be considered a “hang” until the hang was longer than 17 minutes. At this point, an investigation into what processes are taking that long should be done. 80 6.2 Essbase Block Storage (BSO) Tuning For full design, tuning and limits details refer to the Oracle EPM Essbase Documentation Release 11.1.2.3 http://docs.oracle.com/cd/E40248_01/nav/portal_3.htm The key Essbase guides that should be reviewed for all design, tuning and limits considerations are: • Oracle Essbase Database Administrator's Guide • Oracle Essbase Technical Reference • Oracle Essbase Administration Services (Online Help) The following topics are additional design, tuning and limits considerations to optimize Essbase Block Storage (BSO) performance; however, for full details see the guides referenced above. 6.2.1 Outline See the Oracle Essbase Database Administrator's Guide for complete details regarding Block Storage Outline design and tuning. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db.pdf The following sections are to help guide with tuning areas. Only methodical performance and load testing can determine the optimal settings for a given application design and process. 6.2.1.1 Sparse/Dense Configuration Determine the best dimensions for sparse/dense configuration. Typically, the best dense dimensions are found to be the Accounts and Period dimensions. Data distribution and process are typically the driving force with determination the best sparse/dense configuration. The sparse/dense configuration drives the size of the compressed on disk and uncompressed block size in memory. This can impact several areas of performance including disk IO performance, which is why methodical testing is needed to determine the best configuration. 6.2.1.2 Dimension Order General rule of thumb for any Essbase BSO application is to start in hour-glass order, which is dense dimensions first most to least than sparse dimensions least to most. The “most” and “least” are typically based on total number of stored members. 81 General rule of thumb for any Planning application is to start with dense dimensions first but then sparse dimensions with the aggregating dimensions on top than nonaggregation dimensions on the bottom. Dimension order is critical because it impacts calculator cache usage and determines how parallel threads are split into independent tasks. 6.2.1.3 Member Properties Determining the right Member Properties is critical in that it can impact both query and calculation performance. Placing the right Member Property flag i.e. Dynamic Calc, Label Only or Never Shares will impact the size the database thereby impacting performance. For example, implied share is enabled by default in Essbase so when “Never Share” is used on the parent member, this causes a parent and child to have their own blocks, which will increase the size of the number of blocks therefore the database size. The more blocks that are read into memory and written back to disk, the more impact there is to performance. 6.2.2 Database Level Settings See the Oracle Essbase Database Administrator's Guide for complete details regarding Block Storage Database Settings tuning. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db.pdf The following sections are to help guide with tuning areas. Only methodical performance and load testing can determine the optimal settings for a given application design and process. 6.2.2.1 Index Cache If there is enough memory on the Essbase server, it is recommended to set Index Cache to hold all of the ess*.ind files. It is not recommended to set Index Cache higher than needed to cache all ess*.ind files. There will be no performance improvement and only wasting memory allocation. 6.2.2.2 Data Cache Data Cache holds uncompressed blocks i.e. “expanded blocks”. The expanded or uncompressed block does not include dynamically calculated cells. For Buffered IO, at the time a block is read into Essbase memory, the block is immediately uncompressed or expanded and put into the Data Cache. 82 For Direct IO, at the time a block is read into Essbase memory, the block remains compressed and put to the Data File Cache which will later be uncompressed and put to the Data Cache. Sizing Data Cache should be done based on the expanded block size, which can be calculated from the number of dense cells that are not virtual members i.e. Actual Block Size in GetDBStats. See Commit Blocks for another option to size Data Cache. Sizing Data File Cache is virtually the sum of the sizes of the ess*.pag files. 6.2.2.3 Compression Type Typically, in most Essbase BSO applications, depending on data type, bitmap works best. The best compression type depends on the data pattern, therefore, it is recommended methodical testing be done to find the best compression type. For example: Jan 50 10 Feb 50 1000 Mar 50 200 <---- most likely will compress best under RLE <---- most likely will compress best under bitmap 6.2.2.4 Commit Blocks For some BSO applications, default implicit commit setting (3,000) is not appropriate because compressed block size is so small that transactions are committed too often, which produces too much overhead. Given the current state of transaction, the application log provides a hint on a better commit threshold by adjusting the commit block intervals automatically. Commit block adjustment interval is based on a range from 3,000 to 100,000. Meaning, commit blocks will not adjust higher than 100,000. Adjustment intervals can happen with any setting if the algorithm determines block size is small and commit blocks occurs too often. For additional information on BSO Commit Block tuning, see the blog post https://blogs.oracle.com/pa/entry/essbase_11_1_2_commit Important Note: On some Planning apps where user concurrency is high, then the commit block setting should not be set too high in order to avoid locking issues. 83 6.2.2.5 Retrieval Buffer Size The retrieval buffers are a server buffer per database that holds extracted row data cells per user query. When the retrieval buffer is full, the rows are processed and the buffer is reused. If these buffers are too small, frequent reuse of the area can increase retrieval times. If these buffers are too large, too much memory may be used when concurrent users perform queries while also increasing retrieval times. Important Note: It is recommended to start with default values and incrementally increase during load testing using the largest user reports. Since this setting is per user report request, it may lead to wasted memory so only set to what is needed. Also note the retrieval buffers are allocated per thread. Oracle recommends that you do not exceed 100 KB, although the size limit is set at 100,000 KB. 6.2.3 Calculation Commands (used in calc scripts or business rules) See the Oracle Essbase Database Administrator's Guide for complete details regarding Block Storage Calculation Commands tuning. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db.pdf See the Oracle Essbase Technical Reference for a complete list of Calculation Commands that apply to Block Storage applications. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_tech_ref.pdf The following sections are to help guide with tuning areas. Only methodical performance and load testing can determine the optimal settings for a given application design and process. 6.2.3.1 Suggested BSO Calculation Commands to Review: AGG vs. CALC DIM CALCMODE SET CACHE SET CALCPARALLEL SET CALCTASKDIMS SET FRMLBOTTOMUP SET FRMLRTDYNAMIC SET LOCKBLOCK SET MSG SET NOTICE 84 SET REMOTECALC SET RUNTIMESUBVARS SET UPDATECALC 6.2.3.2 General Best Practices for Calculation Commands • Unary calculations are the fastest • Calculate dense members before sparse members • Calculate only what is needed • Typically, IF on dense members and FIX on sparse members • Minimize the number of passes on the database • Best to use parallel threads on aggregations and serial on formulas. Setting all calc scripts to parallel thread making Essbase force serial with some formulas/calc commands adds overhead to the calculation. It is best to set parallel threads only where it is used and not initiate the overhead when serial thread is forced. Be aware of user concurrency when using CALCPARALLEL. • Make sure Schedules Tasks exceed Empty Tasks at least by 50% when using parallel threads. If Empty Tasks are close or equal to Scheduled Tasks, parallel thread overhead is not worth initializing. For additional information on other BSO Calculation Commands, see the blog post https://blogs.oracle.com/pa/entry/essbase_11_1_2_optimisation 6.2.4 Essbase.cfg Settings See the Oracle Essbase Technical Reference for a complete list of configuration settings that apply to Block Storage applications. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_tech_ref.pdf . The following sections are to help guide with tuning areas. Only methodical performance and load testing can determine the optimal settings for a given application design and process. 6.2.4.1 Suggested BSO CFG Settings to Review: CFG Setting AGENTTHREADS AGTSVRCONNECTIONS SERVERTHREADS For additional information on these settings, see the blog post https://blogs.oracle.com/pa/entry/essbase_11_1_2_agtsvr connections NETDELAY NETRETRYCOUNT For additional information on these settings, see the blog post https://blogs.oracle.com/pa/entry/epm_11_1_2_in2 NO_HOSTNAME_LISTCONNECT For additional information on this setting, see the blog post https://blogs.oracle.com/pa/entry/epm_11_1_2_add 85 It is strongly recommended that the default value be used than incrementally increase the value during methodical performance and load testing to find the best setting. It is strongly recommended to not randomly set Essbase configuration setting values high without understanding what the setting does and how it works. Also, methodical testing will uncover issues with higher then needed settings. For instances, it is recommended to set the NETDELAY / NETRETRYCOUNT timeout settings to default values, which is 2 minutes. If need be, the max amount of timeout that is recommended should not be above 5 minutes. The reason is because there have been customers reporting Essbase “hangs”; However, technically there is no “hang” until this “hang” exceeds the NETDELAY / NETRETRYCOUNT settings. Meaning, if timeout is set to 17 minutes and the “hang” resolved in 10 minutes, this would not be considered a “hang” until the hang was longer than 17 minutes. At this point, an investigation into what processes are taking that long should be done. 6.3 Troubleshooting 6.3.1 EAS Launching Multiple Essbase Sessions If a job initiated from EAS takes longer than 5 minutes, a new Essbase connection and a new Essbase session will be launched before the first session has completed. An Essbase connection and a new session will be launched at 5 minute intervals while the job is running in Essbase, creating multiple sessions per job and unnecessarily creating a heavier load on the Essbase agent. For additional information on EAS Launching Multiple Essbase Sessions, see the blog post https://blogs.oracle.com/pa/entry/eas_jobs_taking_longer_than 6.3.2 Essbase Agent / LDAP Timeout in Shared Services For additional information on Essbase Agent / LDAP Timeout in Shared Services, see the blog post https://blogs.oracle.com/pa/entry/epm_11_1_2_essbase 86 6.3.3 Essbase Java Heap Size Increase the Java heap size only during peak periods with a large number of concurrent logins when Essbase agent runs out of memory or terminates abnormally. For additional information on Essbase Java Heap Size, see the CEAL blog post https://blogs.oracle.com/pa/entry/epm_11_1_2_how1 Important Note: When tuning the heap size for Essbase, there are two JVM_OPTIONS settings available for Essbase - one for the Essbase Agent and one for the Essbase applications that are using Custom Defined Functions (CDFs), Custom Defined Macros (CDMs), data mining, triggers or external authentication. For additional information on these two Essbase Java Heap Settings, see the blog post https://blogs.oracle.com/pa/entry/jvm_option_settings_for_essbase 6.3.4 Essbase “Virtual Cube” BSO Tuning An Essbase “virtual cube” is a specific design setup where transparent partitioning is used and consists of multiple source databases that are partitioned to a single target Block Storage (BSO) database. For additional information on Essbase “Virtual Cube” BSO Tuning, see the blog post https://blogs.oracle.com/pa/entry/essbase_virtual_cube_bso_tuning 6.3.5 Fragmentation on BSO Applications See the Oracle Essbase Database Administrator's Guide for complete details regarding Block Storage Fragmentation tuning. http://docs.oracle.com/cd/E40248_01/epm.1112/essbase_db.pdf Recommend scheduling a defragmentation process on a regular based especially on high processing applications like planning applications. For additional information on fragmentation on BSO applications, see the blog post https://blogs.oracle.com/pa/entry/essbase_bso_data_fragmentation 6.3.6 IMPLIED_SHARE Essbase CFG Setting If you need to disable all implied shares in a given outline, there is an Essbase configuration setting that can be set in the Essbase.cfg file. As part of the supported “best practice” steps, it is recommended to rebuild the outline after the IMPLIED_SHARE <ApplicationName> FALSE setting is in place. However, if rebuilding the outline is challenging, there is a way to set an existing outline to have implied share disabled; however, there are risks associated with these steps and no 87 guarantee the metadata and data implied share setting are in sync. If there are any issues after following the “unsupported” steps, Oracle Support will require that the supported steps be followed first. For additional information on these “unsupported” steps, see the blog post https://blogs.oracle.com/pa/entry/disabling_implied_share_for_an1 6.3.7 Transaction Logging Essbase BSO applications with high end user processing should use the transaction logging feature as part of the disaster recovery processes. For additional information on these Essbase BSO transaction logging, see the blog post https://blogs.oracle.com/pa/entry/using_transaction_logging_to_recover 6.3.8 Upgrading Existing Essbase Outline After upgrading an existing Essbase outline to 11.1.2.3 and there are unexpected performance issues i.e. refreshing the outline, data loads, query performance, etc., there could have been a migration issue with the outline after the upgrade. For additional information on Essbase outline migration issues, see the blog post https://blogs.oracle.com/pa/entry/performance_issues_after_upgrading_essbase 88 7.0 TUNING REPORTING AND ANALYSIS This chapter includes the following sections that provide a quick start for tuning the performance of your reports. 7.1 Tune Reporting And Analysis Framework (RAF) Services This topic describes how to tune the Reporting And Analysis Framework Services (a.k.a. Workspace Service) to optimize the performance of your Oracle® EPM System. 7.1.1 Tune RAF Properties Parameter Default Value Suggested Value -Dmax_db_pool_size Increase the DB connection pool for repository metadata connections Tip: To change the value to 200, for example, add “Dmax_db_pool_size=200” to the list of JAVA_OPTS for the Reporting and Analysis Framework Properties. Note that increasing this value increases the amount of memory that the RAF Service uses, regardless of whether the connections in the pool are used. 50 200 Reporting And Analysis Java Heap size Xms (MB) = 128 Xmx (MB) = 750 Xms (MB) = 1024 Xmx (MB) = 2048 Tip: To apply above settings, in EPM Workspace you need to Select Navigate, then Administer, then Reporting and Analysis, and then Services. Right-click the Reporting and Analysis Framework service and select Properties. Modify the desired properties as per screenshot below and then click OK. 89 7.1.2 Tune Max Connections Tune maximum number of connections allowed for RAF services. When increasing the maximum connections consider the memory allocation for the connections you allow. You must increase the maximum number of file descriptors on some systems, such as UNIX. Important Note: The maximum connection to each service agent is 100 by default. Under load, this value may not be sufficient. Use the below suggested settings as a starting point, and then, after careful testing, adjust as needed Parameter Default Value Suggested Value Service Broker 100 500 Job Service 100 500 Event Service 100 500 Repository 100 500 Tip: To apply above settings, in EPM Workspace you need to Select Navigate, then Administer, then Reporting and Analysis, and then Services. Right-click the Reporting and Analysis Framework service and select Properties. Increase the Max Connections to “500” for each service highlighted in screenshot below and then click OK. 90 7.2 Financial Reporting This topic describes how to tune the Financial Reporting to optimize the performance of your reports. 7.2.1 Tune JVM Heap Size The following three Financial Reporting server components have been combined into one server component: FR Reports server, FR Web Applications server, and FR Scheduler server. The Printer server still remains a separate server component. Increase the FR Web Applications server JVM memory settings as stated in section "Tuning Java Virtual Machines (JVM)"of this guide. 7.2.2 Tune Server parameters For tuning Server parameters, see chapter “Property Information” in Oracle Financial Reporting Workspace Administrator's Guide. (http://docs.oracle.com/cd/E40248_01/epm.1112/fr_webadmin/ch03.html) Especially, the following parameters had been proven in modifying concurrency on Financial Reporting: Name Default Value Description 10 The number of parallel threads that a particular Scheduled Job will spawn to handle the scenario where a batch is being burst. true Use this property to enable caching of ADM connection (for connecting to Oracle Essbase, Oracle Hyperion Planning and Oracle Hyperion Financial Management) in a connection pool for each user session. CachePDFForSnapshots false Specify whether all snapshots and snapshot books produced by the scheduler generate and cache PDFs in Workspace. This will improve initial PDF retrieval times for snapshots for Web viewers. Note: This will require additional space in EPM Workspace. Note: Regardless of this setting, PDFs are cached in Oracle Hyperion Enterprise Performance Management Workspace after the first request for a snapshot or snapshot book in PDF format. The default is “false”. Valid values are “true” and “false” (case sensitive). ClassicPrintServerPrinterPool 5 Specify the number of printers available for PDF output. By default the install creates 5 BatchBurstingThreads CacheADMConnectionBasedOnSession 91 printers. Additional printers can be created by editing HRCreatePrinters.ini and running HRCreatePrinters.exe in financialreporting\bin. CleanUpThreadDelay 300000 Specify the frequency (in milliseconds) at which the MinimumConnectionInactiveTime cleanup thread runs. The default is to check for inactive connections every 300000 milliseconds (5 minutes). DbConnRetryInterval In Annotations, if you have a database rack and failover happens, how long, in milliseconds, do you want to wait for a successful attempt to the database. DbMaxConnectAttempts In Annotations, if you have a database rack and failover happens, how many retries to connect should be made. HRWebKeepAliveInterval Specify, in minutes, the frequency at which the web Application Server sends replies back to the browser client. This is useful if http traffic passes through a proxy server that imposes timeouts. The default is one minute 1 HTTPProxyHost Proxy server hostname for any http connections required by Financial Reporting Books. HTTPProxyPort Proxy server Port number for any HTTP connections required by Financial Reporting Books. MaximumCalculationIterations 5 Report Server Calculation Setting. Specify the maximum number of calculation iterations for all grids and cells. During the calculation process of a grid, it may be necessary to evaluate a cell multiple times due to reference precedence. This mostly occurs in grids with references to other grids. If there are no circular references and calculation cells are retuning Error, increasing the value may resolve the problem. The default is 5. Note: Making the number very high may degrade grid execution performance. 300000 Connection Manager Setting. The Connection Manager is a subcomponent of the Report Server, Web Application and Financial Reporting Studio. The lifetime of a MinimumConnectionInactiveTime 92 data source connection is determined by MinimumConnectionInactiveTime. Specify the minimum time (in milliseconds) before an inactive data source connection is closed by the MinimumConnectionInactiveTime cleanup thread. The default is to close connections that have remained inactive for 300000 milliseconds (5 minutes). PrintServerResultCacheTime 6000000 Specify the time (in milliseconds) that completed Reports, Snapshots, Books, and Snapshot books remain on the print server (for generating PDF or printing) before they are removed (the period of time before which uncollected, or "orphaned", results are deleted). The default is to remove results that run longer than 6000000 milliseconds (1 hour). PrintServerResultsCleanUpThreadDelay 36000000 Specify the frequency (in milliseconds) at which the PrintServerResultCacheTime cleanup thread runs. Setting this too low could cause running requests to be terminated and removed. The default is to check for uncollected results every 36000000 milliseconds (10 hours). PrintServers Comma separated list of print servers available to the report server in the ‘server:port’ format. For example: printserver1:10999, printserver2:10999. PrintingMaxThreads 10 Specify the maximum number of threads created for printing/generating PDF files at one time. The default is based on the number of CPUs in the host machine. Five threads are allowed for each CPU. For example, the default for a system with two CPUs is ten threads. Specifying 0 allows an unlimited number of threads to be created ReportServerMaxThreads 5 Specify the maximum number of threads created for running reports at any time. The default is based on the number of CPUs in the host machine - 5 threads per CPU. 1 CPU = 5 threads 2 CPUs = 10 threads 3 CPUs = 15 threads 4 or more CPUs = 20 threads Specifying 0 allows an unlimited number of threads to be created. ReportServerReportCacheTime 72000000 Specify the time (in milliseconds) that 93 completed Reports, Snapshots, Books, and Snapshot books will remain on the Report Server before they are removed. (The period of time before which uncollected, or "orphaned", results are deleted. A result may become orphaned, for example, if a user requests a report, then closes his web browser before the report finishes running.) The default is to remove results that run longer than 72000000 milliseconds (20 hours). ReportServerReportCleanUpThreadDelay 36000000 Specify the frequency (in milliseconds) at which the ReportServerReportCacheTime cleanup thread runs. Setting this too low could cause running requests to be terminated and removed. The default is to check for uncollected results every 36000000 milliseconds (10 hours). ScheduledBatchCacheTime 0 Configuration setting for the Financial Reporting scheduler server. Specify the time in milliseconds to retain completed batch results. For example, to delete batch results older than 5 days specify 432000000 milliseconds. The ScheduledBatchCleanUpThreadDelay property must be set to a value greater than 0 to enable this cleanup option. Pending batches will not be removed. The default is 0 milliseconds which means that batch results will not be removed. 0 Specify how often the scheduler should check for batch results to remove (in milliseconds). For example, to check every 24 hours specify 86400000 milliseconds as the value. The default is 0 milliseconds which means that the scheduler will not remove batch results. ScheduledBatchCleanUpThreadDelay Tip: Suggestion here is to fine tune each parameter one by one starting thread parameters first, then time-outs one, then cache and interval ones. A valid stress test is around 4 hours in which all concurrent users are tested including all Financial Reporting scenarios. Tip: Database tuning (Essbase, Hyperion Financial Management and Hyperion Planning) will affect report performance. Refer to the appropriate section for each product for proper tuning guidelines. 94 7.3 Web Analysis This topic describes how to tune the Web Analysis to optimize the performance of your reports. 7.3.1 Tune JVM Heap Size Increase the Web Analysis Web JVM memory settings as stated in section "Tuning Java Virtual Machines (JVM)"of this guide. Improving Web Analysis Studio Responsiveness To improve the performance and applet load time of Web Analysis Studio through Windows Control Panel, increase the maximum amount of memory allocated to the plug-in. a. Select Start, then Control Panel, then Java Plug-In to open the Java plug-in console. Important Note: If multiple Java Plug-in versions are installed, select the version that Web Analysis Studio uses. b. Select the Advanced tab, and add these parameters to Java Runtime Parameters: -Xms64m –Xmx256m –Xss6m (for information on parameters, see the JRE documentation) 95 7.0 TUNING DATA MANAGEMENT This chapter includes the following sections that provide a quick start for tuning the performance of your data management. 7.1 Tune Data Relational Management (DRM) This topic describes how to tune the Data Relational Management (a.k.a. DRM) to optimize the performance of your Oracle® EPM System. 7.1.1 CPU recommendations The DRM Application servers are generally constrained by processing power. Multiple CPUs and higher CPU speed is essential. All DRM Servers benefit from having multiple CPUs due to the multithreaded nature of the engines. In general it is recommended not to have significantly more engines than processors, especially considering that the engines can multithread as well. The recommended minimum number of CPUs for DRM is equal to the number of Engines – 1 This means that for a 5 engine system there will be the following engine types: • 1 Read Write Engine • 1 Short Read Engine • 1 Request API Engine (Version 9.3.2 and Higher) • 0 to 2 long read Engines The recommended number of CPUs would be 4. Depending on the usage you may be able to increase the max number of engines by 1 or 2 to get more long read engines available, especially if you are not using the Request API as that engine will then remain idle. However there are circumstances (Long Write Operations like import and blend and running large automates) that can prevent the LRO Engines from running even if there is processing power available. Note: Hyper threading processors make it look like there are more processors. However these are not physical processors and as such they do not increase the capability of DRM. Due to the large amount of memory usage by DRM, hyper threading can actually decrease performance. 7.1.2 Memory recommendations In 32-bit mode, each engine can theoretically use up to 2 GB of RAM. Especially if the versions in DRM contain a large amount of data or there are going to be a large number of versions open at the same time. The memory used by an engine is 96 primarily based on the number of versions opened in that engine times the amount of memory used per version. If the amount of memory used exceeds the physical memory of the server then paging will occur and will severely impact performance. Even when it does not exceed the actual physical memory the Windows OS will force some of the DRM memory down to disk to ensure that there is physical memory available for the operating system and other system processes. It is recommended to have enough physical RAM for the engines to each take 2 GB and still have an addition 2 GB of ram for IIS and an additional 2 GB for general OS and other processes. So for a 5 engine system that would be: o 10 GB for the engines o 2 for IIS o 2 for OS and Misc o Total of 14 GB (Rounded up to 16 usually since this is a more common configuration) Note: if the Request API is not being used that engine can be removed from the equation. 7.1.3 Database recommendations It is recommended that the network connection between the DRM server and the database server be on a high speed LAN and not across a WAN or slow connection. If network issues are slowing down due to the WAN between the DRM server and the Client then, if feasible, the long operations should be done using the DRM Batch utility on the DRM server and then the results pushed out. 7.1.4 Network recommendations In the past there have been issues with DRM on Servers where there are dual Network Interface Cards (NICs). There can be issues with DRM if the NICs are not configured identically or are on different subnets. To get past this, either disable one of the NICs, or configure them to be identical. 97 7.2 Tune Financial Data Quality Management (FDQM) This topic describes how to tune Financial Data Quality Management (a.k.a. FDQM) to optimize the performance of your Oracle® EPM System. Setting Configuration Options allow you to set database performance as well as tuning parameters. Configuration options should be used when network infrastructure requires tuning of the database engine I/O activity. Please follow this link for more details: http://docs.oracle.com/cd/E40248_01/epm.1112/fdm_admin/ch04s04.html 7.2.1 Oracle DB Tuning Configuration Options: A few options are available in order to make you quicker with Oracle DB processing. The most common are: Oracle Work Pct Free Oracle Work Table Bitmap Index Switch Oracle Work Pct Free Oracle Work Init Trans Oracle Work Max Trans Oracle Data Map Seg Pct Free Oracle Data Map Seg Init Trans Oracle Data Map Seg Max Trans Oracle Data Seg Pct Free Oracle Data Seg Init Trans Oracle Data Seg Max Trans For deeper explanation, please follow this link: http://docs.oracle.com/cd/E40248_01/epm.1112/fdm_admin/ch04s04s03.html 7.2.2 SQL Server Tuning Configuration Options: Please review the following options especially: Data Seg Table File Group Data Map Seg Table File Group Work Table File Group Work Table Index File Group For deeper explanation, please follow this link: http://docs.oracle.com/cd/E40248_01/epm.1112/fdm_admin/ch04s04s02.html 98 7.2.3 Creating applications for optimal performances: Note: During Application creation if override table spaces is not selected, all tablespaces default to Users. Use of the default can severely degrade performance. See the Oracle Hyperion Financial Data Quality Management DBA Guide for detailed tuning instructions prior to creating the FDM application. Make sure Data partitioning and RDBMS Disk I/O Optimization is done before using your application. Further more details are available on these links: http://docs.oracle.com/cd/E40248_01/epm.1112/fdm_dba/ch03.html Specific to Oracle DB: http://docs.oracle.com/cd/E40248_01/epm.1112/fdm_dba/ch04.html http://docs.oracle.com/cd/E40248_01/epm.1112/fdm_dba/apas01.html Specific to SQL Server: http://docs.oracle.com/cd/E40248_01/epm.1112/fdm_dba/ch05.html 7.2.4 System Options: System options global values that control the behavior of FDM applications are used as control and performance-tuning mechanisms. Three types of system options: - Application settings: http://docs.oracle.com/cd/E40248_01/epm.1112/fdm_dba/ch02.html - Integration settings: http://docs.oracle.com/cd/E40248_01/epm.1112/fdm_dba/ch01s04.html - Configuration options: http://docs.oracle.com/cd/E40248_01/epm.1112/fdm_dba/ch04s07s03.html 99 8.0 Web Browser optimizations This chapter includes the following sections that provide a quick start for tuning the performance of your data management. 8.1 Tune Internet Explorer This topic describes how to tune Internet Explorer to optimize the performance of your Oracle® EPM System. 8.1.1 Registry Key changes For more information, please follow this link: http://msdn.microsoft.com/en-us/library/cc304129%28v=vs.85%29.aspx For Internet Explorer 7 and earlier: HKEY_LOCAL_MACHINE (or HKEY_CURRENT_USER)\SOFTWARE\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_MAXCONNECTIONSPERSERVER Add iexplore.exe DWORD Value, with 16 as decimal value HKEY_LOCAL_MACHINE (or HKEY_CURRENT_USER)\SOFTWARE\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_MAXCONNECTIONSPER1_0SERVER Add iexplore.exe DWORD Value, with 16 as decimal value HKEY_LOCAL_MACHINE (or HKEY_CURRENT_USER)\SOFTWARE\Policies\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_MAXCONNECTIONSPERSERVER Add iexplore.exe DWORD Value, with 16 as decimal value HKEY_LOCAL_MACHINE (or HKEY_CURRENT_USER)\SOFTWARE\Policies\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_MAXCONNECTIONSPER1_0SERVER Add iexplore.exe DWORD Value, with 16 as decimal value 100 For Internet Explorer 7, there are 3 registry keys: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings Add MaxConnectionsPerServer DWORD Value, with 16 as decimal value Add MaxConnectionsPer1_0Server DWORD Value, with 16 as decimal value Add KeepAliveTimeout DWORD Value, with 120000 as decimal value in For Internet Explorer 8 and 9, there are 3 registry keys in HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Internet Settings Add MaxConnectionsPerServer DWORD Value, with 16 as decimal value Add MaxConnectionsPer1_0Server DWORD Value, with 16 as decimal value Add KeepAliveTimeout DWORD Value, with 120000 as decimal value 8.1.2 Tools configuration For all Internet Explorer releases mentioned above, please apply he following option changes: 1. Turn off Phishing Filter. This feature helps to prevent your computer users to be tricked into revealing personal or financial information through an e-mail message or website. If you think you can live without the extra protection of IE anti-phishing, you can disable it by starting IE and then going to: Tools->Phishing Filter And then clicking on Turn Off Automatic Web Site Checking. Then click OK on the subsequent popup window: 101 2. Automatically checking for RSS feeds is reported to slow IE performance. You can turn this off by going to: Tools->Internet Options->Content->Feeds->Settings And then uncheck all boxes shown: 3. Use of ClearType with IE has been reported to slow IE performance. You can disable Cleartype in IE by going to: Tools->Internet Options->Advanced And then moving the slider on the right until you see Always use ClearType for HTML*, at which point you can uncheck that option. 102 4. Use of automatic image resizing with IE has been reported to slow IE performance. You can disable automatic image resizing in IE by going to: Tools->Internet Options->Advanced And then moving the slider on the right until you see Enable automatic image resizing*, at which point you can uncheck that option. 5. Use of page transitions with IE has been reported to slow IE performance. You can disable page transitions in IE by going to: Tools->Internet Options->Advanced And then moving the slider on the right until you see Enable page transitions*, at which point you can uncheck that option. 103 6. Use of smooth scrolling with IE has been reported to slow IE performance. You can disable smooth scrolling in IE by going to: Tools->Internet Options->Advanced And then moving the slider on the right until you see Use smooth scrolling, at which point you can uncheck that option. 104 8.2 Tune Firefox This topic describes how to tune Firefox to optimize the performance of your Oracle® EPM System. 8.2.1 Firefox configuration changes For HTTP pipelining: Enter ‘about:config’ in the URL bar, and click ‘I’ll be careful’. Search for ‘pipe’. Enable all the pipelining options, if not already. Set ‘network.http.pipelining.maxrequests’ to 100 or some large number. For HTTP keep-alive: Enter ‘about:config’ in the URL bar, and click ‘I’ll be careful’. Search for ‘alive’. Enable all the keep-alive options, if not already (they should be). You can increase ‘network.http.keep-alive.timeout’, but not much need on latest Firefox versions. 105 For HTTP keep-alive: Enter ‘about:config’ in the URL bar, and click ‘I’ll be careful’. Search for ‘network.http’. Enable all the keep-alive options, if not already (they should be). You can increase ‘network.http.max-persistent-connections-per-proxy’ ‘network.http.max-persistent-connections-per-server’ to 15. and 106 9.0 Disk tuning This chapter includes the following sections that provide a quick start for tuning the performance of your disks. 9.1 RAID consideration for EPM This topic describes how to tune RAID to optimize the performance of your Oracle® EPM System. We can define EPM products as I/O intensive or more memory based. 9.1.1 I/O intensive EPM products For I/O intensive products listed below, recommendation is to go for RAID 0, RAID 1, RAID 0+1 or RAID 1+0: - Essbase Server - RM1 Data folder for RAF Agent - EAL repository 9.1.2 Memory intensive EPM products For the rest of EPM products, recommendation is to go for RAID 1, RAID 5 could be done especially at Web Application level. 9.2 File System consideration for EPM This topic describes how to tune File systems to optimize the performance of your Oracle® EPM System. There are 3 types of file system that can be used with EPM products, based upon the level of integrity you would like to reach with your application. Tip: NFS and CIFS protocols are not recommended for EPM, and especially Essbase and R&A Framework. 9.2.1 Local File systems Here are the quickest and most stable local file systems per OS: - XFS for Linux. - UFS for Sun Solaris - NTFS for windows. - JFS2 with NOLOG for IBM AIX. 9.2.2 Shared File systems Here are the quickest and most stable shared file systems per OS: - GFS2 for Linux - QFS and ZFS for Sun Solaris - MPFS for Windows 107 - CXFS for IBM-AIX 9.2.2 Clustered File systems Here are the quickest and most stable clustered file systems per OS: - OCFS2 for Linux - VxFS for Sun Solaris - SFS for Windows - GPFS for IBM AIX 9.3 Recommended IOPS for EPM This topic describes how to tune IOPS to optimize the performance of your Oracle® EPM System. 9.3.1 - Recommended RPM should be the following: Local Disk: minimum 7200 RPM SAN/NAS Disks: minimum 15000 RPM 9.3.2 - Recommended latency for EPM: Local Disk: maximum 3 ms SAN/NAS Disks: maximum 2 ms 9.3.3 - Recommended seek time for EPM: Local Disk: maximum 0.2 ms SAN/NAS Disks: maximum 0.1 ms 9.3.4 - Conclusion: recommended IOPS Local Disk: minimum of around 300 IOPS SAN/NAS Disks: minimum of around 450 IOPS 108 10.0 Oracle Virtual Machine tuning This chapter includes the following sections that provide a quick start for tuning the performance of Oracle Virtual Machines. 10.1 Considerations with OVM v3 This topic describes how to tune the network to optimize the performance of your Oracle® EPM System hosted on OVM. For more details, see this documentation: http://www.oracle.com/technetwork/serverstorage/vm/ovm3-10gbe-perf-1900032.pdf 10.2 Reduce the number of dom0 CPUs The most important change made starting OVM 3.2.2 (this has to be applied on 3.0.3) is to make sure dom0 is assigned to 20 VCPUs MAX to provide better out of the box performance for large systems. Tip: To pin and change dom0's VCPUs add: “dom0_vcpus_pin dom0_max_vcpus=X” to the Xen kernel command line, and reboot. For example, the complete line from /boot/grub/grub.conf would be kernel /xen.gz dom0_mem=5G dom0_vcpus_pin dom0_max_vcpus=20 10.3 • • Tune Guest VM Don’t use small memory /OVS/vms/your_vm/vm.conf: memory = '102400' maxmem = '102400‘ limited VMs. Example from Be sure to adjust Essbase throttle semaphores to correspond to OVM limited CPU capacity. Example for OVM with 20 vCPUs: MAXTOTALACTIVETRANSACTIONS 20 MAXACTIVEUPDATETRANSACTIONS 8 Tip: For more details on above parameters refer to the doc http://docs.oracle.com/cd/E17236_01/epm.1112/esb_tech_ref/frameset.htm?ch04s05s98.html 10.4 Tune Process Affinity In NUMA systems it is advisable to run interacting processes (in this case Xen’s Netback and Netfront) on the same CPU socket. This helps reduce memory latency for accessing shared data structures. For a VM with 2 VCPUs, choose VCPUs on the same socket where dom0 runs, and pin the VM’s VCPUs to them. The recommended way to do this is to use the Oracle VM 3 Utilities ovm_vmcontrol command to obtain and change the virtual CPU settings. The Oracle VM 3 Utilities are provided as an optional add-on to Oracle VM 3. 109 Oracle® EPM, Fusion Edition Performance Tuning Guide, Release 11.1.2.2, 11.1.2.3 Revised Edition – October 2013 Authors: CEAL Team Customer Engineering & Advocacy Laboratory Team, BI/EPM/Cloud Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A. Worldwide Inquiries: Phone: +1.650.506.7000 Fax: +1.650.506.7200 www.oracle.com Oracle is a registered trademark of Oracle Corporation. Various product and service names referenced herein may be trademarks of Oracle Corporation. All other product and service names mentioned may be trademarks of their respective owners. Copyright © 2013 Oracle Corporation All rights reserved. 110