2/19/2021
SAP e-book
HA215
SAP HANA 2.0 SPS05 - Using
Monitoring and Performance Tools
.
.
PARTICIPANT HANDBOOK
INSTRUCTOR-LED TRAINING
.
Course Version: 17
Course Duration: 2 Day(s)
e-book Duration: 3 Hours 45 Minutes
Material Number: 50155423
1/1
2/19/2021
SAP e-book
1/1
2/19/2021
SAP e-book
SAP Copyrights, Trademarks and
Disclaimers
© 2021 SAP SE or an SAP affiliate company. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the
express permission of SAP SE or an SAP affiliate company.
SAP and other SAP products and services mentioned herein as well as their respective logos are
trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other
countries. Please see http://global12.sap.com/corporate-en/legal/copyright/index.epx for additional
trademark information and notices.
Some software products marketed by SAP SE and its distributors contain proprietary software
components of other software vendors.
National product specifications may vary.
These materials may have been machine translated and may contain grammatical errors or
inaccuracies.
These materials are provided by SAP SE or an SAP affiliate company for informational purposes only,
without representation or warranty of any kind, and SAP SE or its affiliated companies shall not be liable
for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate
company products and services are those that are set forth in the express warranty statements
accompanying such products and services, if any. Nothing herein should be construed as constituting an
additional warranty.
In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business
outlined in this document or any related presentation, or to develop or release any functionality
mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’
strategy and possible future developments, products, and/or platform directions and functionality are
all subject to change and may be changed by SAP SE or its affiliated companies at any time for any
reason without notice. The information in this document is not a commitment, promise, or legal
obligation to deliver any material, code, or functionality. All forward-looking statements are subject to
various risks and uncertainties that could cause actual results to differ materially from expectations.
Readers are cautioned not to place undue reliance on these forward-looking statements, which speak
only as of their dates, and they should not be relied upon in making purchasing decisions.
© Copyright. All rights reserved.
iii
1/1
2/19/2021
SAP e-book
Typographic Conventions
American English is the standard used in this handbook.
The following typographic conventions are also used.
This information is displayed in the instructor’s presentation
Demonstration
Procedure
Warning or Caution
Hint
Related or Additional Information
Facilitated Discussion
User interface control
Example text
Window title
Example text
© Copyright. All rights reserved.
iv
1/1
2/19/2021
SAP e-book
Contents
vi
Course Overview
1
Unit 1:
Emergency Analysis and Troubleshooting
2
Lesson: Handling System Offline Situations
26
Lesson: Handling System Hang but Reachable Situations
39
Lesson: Analyzing a Suddenly Slow System
60
Unit 2:
Structural System Performance Root Cause Analysis
61
Lesson: Analyzing Memory Issues
75
Lesson: Analyzing CPU Issues
83
Lesson: Analyzing Expensive Statement Issues
90
Lesson: Analyzing Disk and I/O Issues
102
Unit 3:
Proactive Monitoring and Performance Safeguarding
103
Lesson: Configuring the SAP HANA Alerting Framework
114
Lesson: Setting up SAP HANA Workload Management
142
Lesson: Using SAP HANA Capture and Replay
© Copyright. All rights reserved.
v
1/1
2/19/2021
SAP e-book
Course Overview
TARGET AUDIENCE
This course is intended for the following audiences:
●
Support Consultant
●
Developer IT Adminstrator IT Support
●
System Administrator
© Copyright. All rights reserved.
vi
1/1
2/19/2021
SAP e-book
UNIT 1
Emergency Analysis and
Troubleshooting
Lesson 1
Handling System Offline Situations
2
Lesson 2
Handling System Hang but Reachable Situations
26
Lesson 3
Analyzing a Suddenly Slow System
39
UNIT OBJECTIVES
●
Handle system offline situations
●
Handle system hanging but reachable situations
●
Analyze a suddenly slow system
© Copyright. All rights reserved.
1
1/1
2/19/2021
SAP e-book
Unit 1
Lesson 1
Handling System Offline Situations
LESSON OBJECTIVES
After completing this lesson, you will be able to:
●
Handle system offline situations
Important Monitoring Information: Using the Command Line
Like any other system, an SAP HANA database system can go offline for various reasons. This
lesson looks at scenarios where the database unexpectedly went offline, and how the
database administrator could handle this situation.
Figure 1: Course Content Overview
The following issues can cause a SAP HANA system to go offline (that is, from the end-user
perspective, the SAP HANA system seems to hang):
Question: What can cause a SAP HANA system to go offline?
●
........................................
●
........................................
●
........................................
© Copyright. All rights reserved.
2
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
●
........................................
●
........................................
Answer: What can cause a SAP HANA system to go offline?
●
Power failure in the data center
●
Hardware failures at the server level (CPU/memory)
●
Hardware failures at the storage level (disk)
●
Hardware failures at the network level (switches/router)
●
Software errors at the operating system level (Linux)
●
Software errors at the storage system level (SAN/NAS)
●
Software errors at the database level (SAP HANA)
●
Human error causing downtime (Server, router, storage, Linux, and SAP HANA)
Usually, in a system-down scenario the system cannot be accessed through SQL and/or any
another connection method. This makes analyzing the root cause a bit more difficult, but not
impossible. Several small tests, in the right order, can help you quickly exclude areas that
aren't causing the problem. Such a workflow should become a standard way of approaching a
system that is down.
Because SAP HANA cockpit might only be able to partially connect to the SAP HANA system,
you should use the following quick tests to roughly determine the area that causes the
problem. As soon as you have found the problem area you should investigate more deeply,
but not forget that getting the system up and running again has the highest priority.
Question: What tests can you perform to find the problem area?
●
........................................
●
........................................
●
........................................
●
........................................
●
........................................
Answer: What tests can you perform to find the problem area?
●
Check the network. Ping some hosts in the data center.
●
Check the hosts. Log on using SSH and verify that the OS is running without issues.
●
●
●
Check the storage. Create, read, or delete a file to test the connection to the storage
system.
Check SAP HANA. Use sapcontrol to test if all SAP HANA database services are running.
Check SAP HANA. Use hdbsql to test if the SQL database is working for the application
user(s).
© Copyright. All rights reserved.
3
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Caution:
The following checks will help you to quickly identify parts that are broken or
working. With there tests, you are not supposed to do a deep root cause analysis.
For a deep root cause analysis there are other and probably better tools
available.
Check the Network
Figure 2: Check the Network
In today's world, where almost every device is connected to the network, it's extremely
important that the network is up and running correctly. In an SAP HANA database system, the
network is important as well. End users connect to the database to execute all kind of queries.
This can be done directly using SQL or via a middleware application. The SAP HANA database
itself can be set up as a multi-host scale-out system that distributes the data over several
servers. Without a network, external end-user connections and internal server-to-server
connections would fail.
Because external and internal network connections are important for a SAP HANA system,
you should test both by pinging SAP HANA and non-SAP HANA hosts in your network. If all
the hosts can be reached, then the network is available and can be excluded.
ping <SAP HANA host>
ping <internal host>
ping <external host>
Using a ping, you can test that the remote hosts are reachable, but maybe the network
packages are taking the long way home due to a routing problem in the network. You can
check the network path to the remote host using the following command:
traceroute <SAP HANA host>
traceroute <internal host>
© Copyright. All rights reserved.
4
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
Hint:
If in your company the end users are connecting to the network using a virtual
desktop infrastructure (VDI) solution or are in a dedicated network, then you
should test the network connections from within these infrastructures as well.
Check the SAP HANA Hosts on OS Level
As soon as you know that the network is up and running, you can start testing if the SAP
HANA hosts are functioning within normal parameters. Connect to the SAP HANA host(s)
using your preferred method (SSH, XRDP, VNC, and so on) and check the Linux system logs.
Figure 3: Check the Linux Host and Logs
As the SAP HANA hosts are normally up and running 24/7, check whether there were
unplanned and unexplained restarts. You can check this with the following command:
last | grep boot
Looking at the Linux system log files to analyze the system is one of the most important tasks
when troubleshooting a system. Since the move from syslog to systemd, kernel messages
and messages of system services are handled by systemd.
Systemd was introduced in SLES 12 and RHEL 7 and replaces the traditional init scripts.
Systemd also introduced its own logging system called journal.
Systemd manages the journal as a system service under the name systemd-journald.service
and it is switched on by default. In a systemd-enabled Linux system, the systemd-journald
service collects all messages from the kernel, boot process, syslog, user processes, standard
input, and system service errors in a centralized location.
You can check the last 50 boot error messages in the journal with the following command:
journalctl -n 50 -p err -b
-n = number of messages to display
-p = message priority
-b = display boot messages
© Copyright. All rights reserved.
5
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Hint:
You can check the last 50 kernel error messages in the journal with the following
command:
journalctl -n 50 -p err -k
-k = display kernel messages
Check the Storage
Storage problems can result in severe database problems, database standstill or, even worse,
data loss.
Figure 4: Check the Storage
Avoiding storage problems is part of every layer in the Linux software and hardware stack.
Modern hard drives are capable of detecting and correcting minor errors in block reads. SAN
and NAS have built-in error correction and redundancy to handle power and hardware failure.
Modern Linux file systems are all journal-based and can correct errors created due to power
failures. Last but not least databases also support many different techniques to survive power
failures and incorrect service shutdown situation.
If the SAP HANA database system 'stopped' due to power, hardware, or software failures, you
should check if all file systems are available again after the server has restarted. Depending
on the storage system used you can investigate the storage problem more deeply.
Note:
In the scope of this course we will not investigate storage system problems. For
this you need to contact your storage vendor and get the support information you
need.
Check the SAP HANA Database System
Checking if the database is up and running sounds like a good plan, but with all the services
running it might still be the case that the end user or middleware application cannot connect.
Such a situation can happen when, for example, the SAP HANA system is up and running, but
the network doesn't allow SQL connections due to a firewall having been reconfigured.
© Copyright. All rights reserved.
6
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
Figure 5: Check SAP HANA from the Command Line
To check if all the SAP HANA services and hosts are available on the Linux host, you can
execute the following commands:
As <sid>adm user:
sapcontrol -nr <instance number> -function GetProcessList
sapcontrol -nr <instance number> -function GetSystemInstanceList
You also need to check whether or not the system can be reached over the SQL interface.
When you are already connected to the SAP HANA host via the SSH session, check the SQL
interface with the following command:
Note:
The default port number range for tenant databases is 3<instance>40 3<instance>99.
As <sid>adm user:
hdbsql -n localhost -i <instance number> -d <Tenant name> -u <your
database user>
Enter your password when requested. You are now in the HDBSQL terminal. From the
HDBSQL terminal you can get SAP HANA connection information by executing the command:
\s
Caution:
It's important to test all your tenants, because the tenants have different SQL
ports and can be stopped independently of a running SAP HANA database
system.
© Copyright. All rights reserved.
7
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Checking the SQL connection only from the local host isn't sufficient as it could be the case
that SAP HANA SQL is blocked on the network. To make sure that this isn't the case, you
should perform a HDBSQL connection test also via the network from the end-user LAN and
the application server network.
From the SAP S/4HANA ABAP application server, as <sid>adm user:
hdbsql -n <SAP HANA host> -i <instance number> -d <Tenant name> -u
<your database user>
Enter your password when requested. You are now in the HDBSQL terminal. From the
HDBSQL terminal you can get SAP HANA connection information by executing the command:
\s
If the issue is due to a hardware or a software failure, it is important to save log files on the
Linux operating system or at the storage system level for later analysis.
For further specific steps and guidance on pro-active or reactive actions you can take, see
SAP Note 1999020 — SAP HANA: Troubleshooting when the database is no longer reachable.
Important Monitoring Information: Using SAP HANA Cockpit
SAP HANA Cockpit 2.0 Architecture
The SAP HANA cockpit 2.0 provides a single point of access to a range of tools for
administration and detailed monitoring of SAP HANA databases. It also integrates
development capabilities required by administrators through the SAP HANA Database
Explorer.
The SAP HANA cockpit 2.0 is a web-based HTML5 user interface that you access through a
browser. It runs on SAP HANA extended application services, advanced model (XS
advanced). The cockpit handles single-container and multi-container systems as of SAP
HANA 1.0 SPS 12. You can use the cockpit to monitor and manage multiple SAP HANA
database systems and tenants.
Figure 6: SAP HANA Cockpit 2.0 Architecture
© Copyright. All rights reserved.
8
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
The cockpit can be installed separately on dedicated hardware, shared hardware or in an
existing SAP HANA database tenant. This provides greater flexibility, because it allows you to
manage more than one SAP HANA system in a single administration environment.
What Can SAP HANA Cockpit Do?
SAP HANA cockpit provides centralized system and database administration features, such
as database monitoring, user management, and data backup. Administrators can use the
cockpit to start and stop services, to monitor the system, to configure system settings, and to
manage users and authorizations. The cockpit provides applications that allow you to manage
SAP HANA options and capabilities (for example, SAP HANA dynamic tiering). These
applications are only available if the options or capabilities are installed.
SAP HANA Cockpit Main Tools
The cockpit consists of the following tools, each of which has a dedicated function:
●
Cockpit Manager
●
SAP HANA cockpit
●
SAP HANA database explorer
The Cockpit Manager is used by the cockpit administrator to register databases and to create
groups and cockpit users for accessing SAP HANA cockpit.
The first step for administrating a tenant or systemdb is to register the new tenant in the
Cockpit Manager. As soon as a tenant is registered in the Cockpit Manager, the database
administrator can start to use the tenant in the SAP HANA cockpit.
The SAP HANA cockpit home screen shows a high level aggregated overview of all registered
systems. From this aggregated landscape overview level you can quickly drill down to a
detailed overview of an individual database. In the database overview screen you find cards
for all important parts of the SAP HANA database. On a card, you see a mini graph of an
important KPI that belongs to the monitoring area the card displays. On these cards you will
also find links that start cockpit applications to analyze the measured KPIs further. Through
this drill-down you can easily find the cause of the problem.
The SAP HANA database explorer tool is integrated into the SAP HANA cockpit. The database
explorer allows you to query information about the database using SQL statements, and to
view information about your database's catalog objects.
Opening SAP HANA Cockpit 2.0
To start the cockpit, open the following URL in our training landscape: https://
wdflbmt7288.wdf.sap.corp:51026. After logging on, you are presented with an overview of the
databases assigned to your user account.
© Copyright. All rights reserved.
9
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Figure 7: The SAP HANA Cockpit - Home Screen
In the overview, you can select the Database Directory tile or a dedicated group tile, like the
Group01 in this screenshot, to quickly see the status of the SAP HANA systems.
From the Database Directory tile, you can navigate to your SAP HANA system overview page,
where a detailed status of the selected SAP HANA system is displayed.
Database Directory
The Database Directory gives an aggregated overview of each database for which you are
responsible. In the Database Directory, you can see that a system or tenant is in trouble when
the status Stopped, Running with issues, or Unknown is displayed. To investigate these
problems in more detail, you start the cockpit System Overview page for this system or tenant
by selecting the corresponding line.
When the Database Directory shows that the whole SAP HANA database system is having
problems, it is important to quickly investigate the root cause of the problem so that you can
get the system up and running again.
Even when the SAP HANA database system is down, the cockpit can be used to investigate
the root cause of the problem. This system-down analysis is done via the SAP start service
connection.
© Copyright. All rights reserved.
10
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
Figure 8: Database Directory
When a important system is down, you want to start it as soon as possible. This is a logical
course of action, but it could make the root cause analysis more difficult, because during a
restart, important low-level log or trace files can be overwritten. So a best practice is that
before starting the system, you save all the important log and trace files for later
investigation.
To support this, SAP HANA provides a full system information dump. This information dump
lets you control which logs to save, so you can use these saved logs to troubleshoot the issue
after you have restarted the SAP HANA database.
In the Database Directory, you can also specify the database user credentials required to drill
down to an individual database, which is necessary unless single sign-on is in effect for that
database.
Database Overview
The Database Directory shows a high-level status overview of all the databases belonging to
groups to which you have been granted access. For each database, you can drill down for
more information.
When you open the cockpit's Database Overview page for a system that shows the status
Stopped, No SQL access, or Unknown, it is very likely that you cannot connect to the SAP
HANA database using a SQL connection. The cockpit starts, but cannot retrieve the
monitoring data using SQL. This results in almost all cards showing the text Cannot load
data.
It is best that you start the Database Overview page of SYSTEMDB instead, because from this
Database Overview page you can retrieve some information via the SAP host agent
infrastructure. Via the SYSTEMDB, you can get information on the status of the SAP HANA
services.
© Copyright. All rights reserved.
11
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Figure 9: SAP HANA System Not Available
This means that you cannot use the default monitoring cards to further investigate the
problem. Depending on the error situation, the SAP HANA Cockpit will present you the
relevant cards that can be useful during the investigation. In a system down you will find the
Manage full system information dumps application to be available, but the Troubleshoot
unresponsive systems isn't because the SAP HANA Cockpit can't connect to the SAP HANA
index service.
Figure 10: System Down vs Unresponsive System
© Copyright. All rights reserved.
12
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
The Manage Full System Information Dumps Application
To collect the diagnosis information, choose the Manage full system information dumps
application. This application is found in the Alerts and Diagnostics card.
Figure 11: Manage Full System Dumps
To collect the diagnosis information, perform the following steps:
1. Search for the Alerts and Diagnostics card.
2. Choose the Manage full system information dumps link.
3. Choose Collect Diagnostics, and in the dropdown list choose Collect from Existing Files or
Create from Runtime Environment .
4. In the pop-up window choose the information items you want to collect. In the bottomright corner, choose Start Collecting.
5. When all the data is collected, the
fullsysteminfodump_<SID>_<DBNAME>_<HOST>_<timestamp>.zip file is displayed in
the collections table.
© Copyright. All rights reserved.
13
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Figure 12: Collect Diagnosis Information
Collect from Existing Files
Choose this option if you want to collect diagnosis information for one or more file types for a
specific time period - by default the last seven days. If you also want information from system
views, select Include System Views .
Note:
If you are connected to the system database of a multiple-container system, only
information from the system views of the system database is collected.
Information from the system views of tenant databases is not collected,
regardless of this option setting.
Information from system views is collected through the execution of SQL statements, which
may impact performance. In addition, the database must be online, so this option is not
available in diagnosis mode.
Create from Runtime Environment
Choose this option if you want to restrict the information collected to one or more runtime
environment (RTE) dump files. You can configure the creation and collection of dump files by
specifying the following additional information:
●
●
The number of sets to be collected (that is, the number of points in time at which RTE
dump files are collected). Possible values are 1-5.
The interval (in minutes) at which RTE dump files are to be collected (possible values are 1,
5, 10, 15, and 30). The default value is 1.
●
The host(s) from which RTE dump files are to be collected.
●
The service(s) for each selected host from which RTE dump files are to be collected.
●
The section(s) from each selected service from which RTE dump files are to be collected.
© Copyright. All rights reserved.
14
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
The system collects the relevant information and saves it to a ZIP file. This may take some
time and can be allowed to run in the background.
If you are connected to the system database of a multiple-container system, information from
all tenant databases is collected and saved to separate ZIP files.
Download the Collected Diagnosis Information
Once the collected diagnosis information ZIP files are available, you can download them by
choosing Download. The files are saved locally in your browser’s download directory.
Figure 13: Download the Diagnosis Information
The diagnosis information is collected by the Python support script
fullSystemInfoDump.py, which collects a range of information from your system for
diagnosis purposes. It can be triggered from SAP HANA cockpit, or directly from the
command line.
Collect Diagnosis Information from the Command Line
The fullSystemInfoDump.py script allows you to collect information from your system,
even when it is not accessible using SQL. You can then add this information to a support
message, or use this information to investigate the root cause on your personal computer.
The script is part of the SAP HANA server installation and can be executed directly from the
command line.
© Copyright. All rights reserved.
15
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Figure 14: Command Line: fullSystemInfoDump.py
If you are logged on as the operating system user, <sid>adm, the fullSystemInfoDump.py
script is part of the server installation and can be run from the command line. It is located in
the directory $DIR_INSTANCE/exe/python_support.
Hint:
You can use the predefined shell alias cdpy to quickly navigate to the
python_support directory.
Start the script from its location with the command:
python fullSystemInfoDump.py
You can modify the command with several command line options. To see the available
options, specify the option --help. All options related to getting a system dump are fully
described in SAP Note 1732157.
If the system can be reached by SQL (and you have not specified the option --nosql), the
script starts collecting diagnosis information. If the system cannot be reached by SQL, the
script starts collecting support information, but does not export data from system views.
The script creates a ZIP file containing the collected information and saves it to the directory
$DIR_GLOBAL/sapcontrol/snapshots. $DIR_GLOBAL typically points to /usr/sap/
<SID>/SYS/global.
The name of the ZIP file is structured as follows:
fullsysteminfodump_<SID>_<DBNAME>_<HOST>_<timestamp>.zip.
© Copyright. All rights reserved.
16
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
The time-stamp in the file name is Coordinated Universal Time (UTC). The HOST and SID are
taken from the sapprofile.ini file.
The output directory for the ZIP file is shown as console output when the script is running, but
you can look it up with the command hdbsrvutil -z | grep DIR_GLOBAL= .
Diagnosis Information Collected
The Python support script fullSystemInfoDump.py collects the following information from
your system for diagnosis purposes.
Note:
All of the following file types are collected unless the option --rtedump is
specified, in which case only runtime environment (RTE) dump files are created
and collected.
Log File
All information about what has been collected is shown as console output, and is written to a
file named log.txt, which is stored in the ZIP file.
Trace Files
Each of the following trace files is put into a file with the same name as the trace file. For
storage reasons, only the trace files from the last seven days are collected unabridged. Older
trace files are not collected. This behavior can be changed by using the option --days or with
the options --fromDate and --toDate.
●
●
●
●
●
●
●
●
●
●
●
$DIR_INSTANCE/<SAPLOCALHOST>/trace/
compileserver_alert_<SAPLOCALHOST>.trc
$DIR_INSTANCE/<SAPLOCALHOST>/trace/
compileserver_<SAPLOCALHOST>.<...>.trc
$DIR_INSTANCE/<SAPLOCALHOST>/trace/daemon_<SAPLOCALHOST>.<...>.trc
$DIR_INSTANCE/<SAPLOCALHOST>/trace/
indexserver_alert_<SAPLOCALHOST>.trc
$DIR_INSTANCE/<SAPLOCALHOST>/trace/
indexserver_<SAPLOCALHOST>.<...>.trc
$DIR_INSTANCE/<SAPLOCALHOST>/trace/
nameserver_alert_<SAPLOCALHOST>.trc
$DIR_INSTANCE/<SAPLOCALHOST>/trace/nameserver_history.trc
$DIR_INSTANCE/<SAPLOCALHOST>/trace/
nameserver_<SAPLOCALHOST>.<...>.trc
$DIR_INSTANCE/<SAPLOCALHOST>/trace/
preprocessor_alert_<SAPLOCALHOST>.trc
$DIR_INSTANCE/<SAPLOCALHOST>/trace/
preprocessor_<SAPLOCALHOST>.<...>.trc
$DIR_INSTANCE/<SAPLOCALHOST>/trace/
statisticsserver_alert_<SAPLOCALHOST>.trc
© Copyright. All rights reserved.
17
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
●
$DIR_INSTANCE/<SAPLOCALHOST>/trace/
statisticsserver_<SAPLOCALHOST>.<...>.trc
●
$DIR_INSTANCE/<SAPLOCALHOST>/trace/xsengine_alert_<SAPLOCALHOST>.trc
●
$DIR_INSTANCE/<SAPLOCALHOST>/trace/xsengine_<SAPLOCALHOST>.<...>.trc
Configuration Files
All configuration files are collected unabridged and stored in a file with the same name as
the .ini file:
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/attributes.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/compileserver.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/daemon.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/executor.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/extensions.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/filter.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/global.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/indexserver.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/inifiles.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/localclient.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/mimetypemapping.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/nameserver.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/preprocessor.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/scriptserver.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/statisticsserver.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/validmimetypes.ini
●
$DIR_INSTANCE/<SAPLOCALHOST>/exe/config/xsengine.ini
Database System Log Files
The following backup files are collected unabridged:
●
$DIR_INSTANCE/<SAPLOCALHOST>/trace/backup.log
●
$DIR_INSTANCE/<SAPLOCALHOST>/trace/backint.log
RTE Dump Files
For each index server, an RTE dump file containing information about threads, stack contexts,
and so on is created and stored in the file
indexserver_<SAPLOCALHOST>_<PORT>_runtimedump.trc. These files are stored
unabridged.
Crashdump Information
Crashdump files for services are collected unabridged.
© Copyright. All rights reserved.
18
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
Performance Trace Files
Performance trace files with the suffix *.tpt are collected unabridged.
Kerberos Files
The following Kerberos files are collected:
●
/etc/krb5.conf
●
/etc/krb5.keytab
System Views
If the collection of system views is not excluded (the option --nosql is specified), all rows of
the following system views (with the exceptions mentioned) are exported into a CSV file with
the name of the table:
Note:
If you are connected to the system database of a multiple-container system, only
information from the system views of the system database is collected.
Information from the system views of tenant databases is not collected,
regardless of this option setting.
Note:
If you trigger the collection of diagnosis information from the SAP HANA cockpit
for offline administration, information from system views cannot be collected
because it does not use an SQL connection.
●
SYS.M_CE_CALCSCENARIOS WHERE SCENARIO_NAME LIKE '%_SYS_PLE%'
●
SYS.M_CONNECTIONS with CONNECTION_ID > 0
●
SYS.M_DATABASE_HISTORY
●
SYS.M_DEV_ALL_LICENSES
●
SYS.M_DEV_PLE_SESSIONS_
●
SYS.M_DEV_PLE_RUNTIME_OBJECTS_
●
SYS.M_EPM_SESSIONS
●
SYS.M_INIFILE_CONTENTS
●
SYS.M_LANDSCAPE_HOST_CONFIGURATION
●
SYS.M_RECORD_LOCKS
●
SYS.M_SERVICE_STATISTICS
●
SYS.M_SERVICE_THREADS
●
SYS.M_SYSTEM_OVERVIEW
●
SYS.M_TABLE_LOCATIONS
●
SYS.M_TABLE_LOCKS
© Copyright. All rights reserved.
19
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
●
SYS.M_TABLE_TRANSACTIONS
●
_SYS_EPM.VERSIONS
●
_SYS_EPM.TEMPORARY_CONTAINERS
●
_SYS_EPM.SAVED_CONTAINERS
●
_SYS_STATISTICS.STATISTICS_ALERT_INFORMATION
●
_SYS_STATISTICS.STATISTICS_ALERT_LAST_CHECK_INFORMATION
Note:
Only the first 2,000 rows are exported.
●
_SYS_STATISTICS.STATISTICS_ALERTS
Note:
Only the first 2,000 rows are exported.
●
_SYS_STATISTICS.STATISTICS_INTERVAL_INFORMATION
●
_SYS_STATISTICS.STATISTICS_LASTVALUES
●
_SYS_STATISTICS.STATISTICS_STATE
●
_SYS_STATISTICS.STATISTICS_VERSION
The first 2,000 rows of all remaining tables in the schema _SYS_STATISTICS are exported,
ordered by the SNAPSHOT_ID column.
Additional Information Collected if SQL Connection is Not Available
All available topology information is exported to a file named topology.txt. It contains
information about the host topology in a tree-like structure.
© Copyright. All rights reserved.
20
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
Figure 15: Example: The Topology.txt File
Execute a System Restart
Now that all important diagnosis information is collected in the
fullsysteminfodump_<SID>_<DBNAME>_<HOST>_<timestamp>.zip file(s), you can restart
the SAP HANA database system. Before you restart the system, consider if you need to
restart the whole SAP HANA database system, or just the a tenant in the SAP HANA database
system. How to perform a database restart and how to restart a SAP HANA tenant are
described below.
© Copyright. All rights reserved.
21
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Start a Tenant Database
Figure 16: Start a Tenant Database
In the SAP HANA cockpit - Home screen, select the Database Directory or your personal group
tile. In the Database Directory screen, choose the Manage Databases link. To start the tenant
database, in the Manage databases screen, choose the Start button. This will perform a
normal tenant database start.
© Copyright. All rights reserved.
22
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
Start the SAP HANA Database System
Figure 17: Start the SAP HANA Database System
In the SAP HANA cockpit - Home screen, choose the Database Directory tile and choose the
SYSTEMDB that is stopped. In the Database Overview screen, search for the Services card. To
start the database system, in the Services card choose the Start Database button. This will
perform a normal database start of the SAP HANA database.
Note:
In newer version of the SP HANA Cockpit. you can choose the Start Database
button directly from the Database Overview screen.
Verify the Database System Status
The SAP HANA system restart sequence quickly restores the system to a fully operational
state.
When you restart an SAP HANA system, the following activities are executed by the restart
agent of the persistence layer:
1. The data volume of each service is accessed to read and load the restart record.
2. The list of open transactions is read into memory.
3. Row tables are loaded into memory.
4. Open transactions are processed using the redo log, as follows:
© Copyright. All rights reserved.
23
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
●
●
Write transactions that were open when the database was stopped are rolled back.
Changes to committed transactions that were not written to the data area are rolled
forward.
The first column tables start being reloaded into memory as they are accessed for roll
forward.
Note:
Because a regular or "soft" shutdown writes a savepoint, there are no replay
log entries to be processed in this case.
5. Aborted transactions are determined and rolled back.
6. A savepoint is performed with the restored consistent state of the database.
7. Column tables that are marked for preload, and their attributes, are asynchronously
loaded in the background (if they have not already been loaded as part of log replay).
The preload parameter is configured in the meta-data of the table. This feature is useful,
for example, to make certain tables and columns that are used by important business
processes available more quickly.
8. Column tables and their attributes that were loaded before restart, start reloading
asynchronously in the background (if they have not already been loaded as part of log
replay or because they are marked for preload).
During normal operation, the system tracks the tables that are currently in use. This list is
used as a basis for reloading tables after a restart.
Reloading column tables, as described in steps 7 and 8, restores the database to a fully
operational state more quickly. However, it does create performance overhead and may not
be necessary in non-production systems. You can deactivate the reload feature in the
indexserver.ini file by setting the reload_tables parameter in the sql section to false. In
addition, you can configure the number of tables whose attributes are loaded in parallel using
the tables_preloaded_in_parallel parameter in the parallel section of the
indexserver.ini file. This parameter also determines the number of tables that are preloaded in
parallel.
© Copyright. All rights reserved.
24
1/1
3/1/2021
SAP e-book
Lesson: Handling System Offline Situations
Figure 18: Is the SAP HANA Database Running?
Now that the SAP HANA database system is up and running, you can continue to investigate
the failure. If you cannot find the root cause of the failure, open an SAP support message and
attach the diagnosis information collected in the SAP HANA cockpit - Full system information
dump application.
LESSON SUMMARY
You should now be able to:
●
Handle system offline situations
© Copyright. All rights reserved.
25
1/1
3/1/2021
SAP e-book
Unit 1
Lesson 2
Handling System Hang but Reachable
Situations
LESSON OBJECTIVES
After completing this lesson, you will be able to:
●
Handle system hanging but reachable situations
Troubleshoot Unresponsive Systems
Figure 19: Urgent Analysis and Troubleshooting
There are various reasons for a system to hang, or seem to be hanging from an end-user
perspective. The database is said to be hanging when it no longer responds to queries that are
executed against it.
The source of the system standstill might be related to any of the components involved, for
example, the storage, OS and hardware, network, SAP HANA database or the application
layer. For troubleshooting it is essential to collect information about the context of the active
threads in the SAP HANA database.
Question: How can a system get into a hanging state?
●
........................................
© Copyright. All rights reserved.
26
1/1
3/1/2021
SAP e-book
Lesson: Handling System Hang but Reachable Situations
●
........................................
●
........................................
●
........................................
●
........................................
The following list of issues can cause a system hang state, or a state where the system seems
to hang from the end-user perspective:
Answer: How can a system get into a hanging state?
●
Log volume full caused by either a full disk, a quota setting or failed log backups
●
Savepoint lock conflict with long-running update
●
Wrong configuration of transparent huge page or OS page cache
●
The Translation Lookaside Buffer (TLB) shootdown
●
High context switches caused by many SqlExecutor or JobExecutor threads
●
Huge Multiversion Concurrency Control (MVCC) versions
●
High system CPU usage caused by non-HANA applications
●
Frequent Out of Memory (OOM) situations that lead to a performance drop
Note:
What does "Translation Lookaside Buffer (TLB) shootdown" mean?
A Translation Lookaside Buffer (TLB) is a cache of the translations from virtual
memory addresses to physical memory addresses. When a processor changes
the virtual-to-physical mapping of an address, it needs to tell the other processors
to invalidate that mapping in their caches.
As SQL statements cannot usually be executed for analysis, you should perform the following
steps if it is still possible to log on to the OS of the master host (for example, as the <sid>adm
user). Also see SAP Note 1999020: SAP HANA: Troubleshooting when database is no longer
reachable for further specific steps and guidance on proactive or reactive actions you can
take.
Question: What and where can you check?
●
........................................
●
........................................
●
........................................
●
........................................
Answer: What and where can you check?
●
Use SAP HANASitter
●
Check if the SAP HANA file systems still have free space
© Copyright. All rights reserved.
27
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
●
Collect call stack and runtime information
●
Analyze the current Operating System information
●
Analyze an Unresponsive System in SAP HANA Cockpit
Use SAP HANASitter
SAP HANASitter is a tool developed by SAP Support that allows you to monitor SAP HANA
and to automatically create dump files in certain scenarios. For more information, see SAP
Note 2399979 - How-To: Configuring automatic SAP HANA Data Collection with SAP
HANASitter
Figure 20: Check System using SAP HANASitter
SAP HANASitter Features:
●
Database online check
●
CPU, ping, and critical feature check
●
Recording mode for RTE dumps, stack calls, kernel profiler trace, and GStack
●
Scale-out monitor
●
Critical session killer
The SAP HANAsitter checks by default once an hour, if SAP HANA is online and primary. If so,
it starts to track. Tracking includes regularly (by default, every minute) checking if SAP HANA
is responsive. If it is not, it starts to record.
Recording can include writing call stacks of all active threads, recording run time dumps,
index server gstacks, and/or kernel profiler traces. By default, nothing is recorded.
If SAP HANA is responsive, it checks many of the critical features of SAP HANA. As standard,
the script checks if there are more than 30 active threads. If there are more than 30 active
threads, the script starts to record.
When the script has finished recording, it exits. The script can be configured to restart using
the command line.
© Copyright. All rights reserved.
28
1/1
3/1/2021
SAP e-book
Lesson: Handling System Hang but Reachable Situations
When the script has finished all the tests successfully, it sleeps for one hour, before it starts
all the checks again.
Setup Steps Overview
1. Create an SAP HANA user (for example, HANASITTER, but you can use a different name)
and assign the CATALOG READ privilege.
2. Create a user key (for example, SYSTEMKEY, but you can use a different name) in the
hdbuserstore.
3. Download the hanasitter.py script attached to SAP Note 2399979.
4. Store the script in, for example, the python_support directory.
5. As <sid>adm, change to the python_support directory with the command cdpy.
6. Execute the script with the command python hanasitter.py -ng 1.
Check if the SAP HANA File Systems Still Have Free Space
In a system hanging situation the execution of SQL statements is probably not possible
anymore. If you still can log on to the operating system, you should try to perform the
following steps on the OS of the master host.
Figure 21: Check an Unresponsive System
In cases where logs cannot be written, all DML statements will fall into wait status. This can
lead to a failure of opening new connections because the system internally executes DML
statements during the process. Typically, a full log volume is the cause for this.
The root cause of the "log volume full" situation is either caused by disk being full or hitting
the quota setting. To investigate more deeply, perform the following steps:
1. Check for the Internal Disk-Full Event (Alert 30) in the indexserver trace.
2. Check if the system is running out of disk space using the command df -h on the OS ssh
shell.
© Copyright. All rights reserved.
29
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
3. Check if the system is running out of inodes (NFS) using the command df -i.
4. Check the quota setting in file system.
5. Check SAP Note 1679938 - Log Volume is full.
6. Check SAP Note 2083715 - Analyzing log volume full situations.
Once you have resolved the issue (for example, freed up disk space), you may need to
manually mark the internal event as handled. You can do this on the Overview tab of the
Administration editor in the SAP HANA studio, or by executing the following SQL statements:
ALTER SYSTEM SET EVENT ACKNOWLEDGED '<host>:<port>' <id>
ALTER SYSTEM SET EVENT HANDLED '<host>:<port>' <id>
Log Backup Failure
The "log volume full" situation can also be caused by failing log backups. A SAP HANA log
segment is freed for reuse as soon as a the segment is written by the backup process to the
backup location. If the log backups fail, the log segments are not reused and new log
segments are created. Eventually this can fill up the data volume file system resulting in the
"log volume full" situation. To investigate more deeply, perform the following steps:
1. Check the backup.log (located at /usr/sap/<SID>/HDB<Instance#>/<Host>/trace )
to see whether it includes ERROR in the log backup. Check M_BACKUP_CATALOG and
M_LOG_SEGMENTS.
2. If log backup uses backint, check backint.log (located at /usr/sap/<SID>/
HDB<Instance#>/<Host>/trace ) to see whether it includes ERROR information, and
contact backint vendor support.
Savepoint Lock Conflict with Long-running Update
With certain revision and conditions, the conflict between the savepoint lock and the DML lock
blocks subsequent statements when long-running update/insert statements exist. To
investigate more deeply, perform the following steps:
1. Use SAP Note 813020: How to generate a runtime dump on SAP HANA to collect a
runtime dump. In the generated dump, look for the following combination of call stacks in
many threads.
…
DataAccess::SavepointLock::lockShared(…)
DataAccess::SavepointSPI::lockSavepoint(…)
…
And one of the following call stacks that is in the savepoint phase.
…
DataAccess::SavepointLock::lockExclusive()
DataAccess::SavepointImpl::enterCriticalPhase(…)
…
2. If you are running SAP HANA 1.0 (rev 97 or older) check whether the symptoms match the
description in SAP Note 2214279 - Blocking situation caused by waiting writer holding
consistent change lock. If so, apply the parameter cch_reopening_enabled as
described in the SAP Note.
Collect Call Stack and Runtime Information
For accurate root cause analysis, it is very helpful to have call-stack and runtime information
available from the time of the hang. For this, you can use the approaches outlined in SAP Note
© Copyright. All rights reserved.
30
1/1
3/1/2021
SAP e-book
Lesson: Handling System Hang but Reachable Situations
2313619. It is useful to capture one or several runtime dumps (SAP Note 1813020) so that an
accurate root cause analysis can be done later.
If you consider the possibility of an SAP HANA internal deadlock, you can also run the
deadlock detector functionality of hdbcons (SAP Note 2222218): hdbcons
'deadlockdetector wg -w -o <file_name>.dot'.
Note:
The generated DOT file can be converted to a PDF or a GIF file using the following
commands:
●
●
To generate a PDF file: dot -Tpdf -o <pdffile> /usr/sap/<SID>/
HDB00/work/HA215_<SID>_DeadlockCheck.dot
To generate a GIF file: dot -Tgif -o <giffile> /usr/sap/<SID>/
HDB00/work/HA215_<SID>_DeadlockCheck.dot
Analyze the Current Operating System Information
Use OS commands like top to identify the amount of CPU consumption, the main CPU
consuming processes, and the main CPU utilization component. As SAP HANA is a
multithreading system it's more useful to have an overview of the consumed CPU per thread.
To display the CPU used per thread, use the top -H command. Its output shows the CPU
consumption per thread.
Check /proc/interrupts to understand if certain interrupts happen very often. Ensure that
the SAP HANA-related file systems contain sufficient free space (for example, using tools like
df or mmdf (for GPFS)) and that there is no quota defined that could result in a file system
becoming full. Check if the file systems are still reachable, for example, using df /usr/sap/
<sid>. Inspect OS log files, such as /var/log/messages, for suspicious error messages
that are issued at the time of the problem (but not at times of normal operation).
Check the OS Configuration
On Linux, you must ensure that Transparent Huge Pages (THPs) are disabled. If THPs are
disabled properly, the cat /sys/kernel/mm/transparent_hugepage/enabled
command returns always madvise [never].
See further recommended OS, firmware, and hardware settings described in SAP
Note 2000003.
Use the OS Command "sar" for Historic Analysis
Use operating system commands like sar to identify further relevant operating system
information (for example, memory, network, interrupts, and further CPU and load details). For
more information, see SAP Note 1999670.
Execute Hardware Checks
On IBM hardware, you can use the hardware check tool provided in SAP Note 1661146 to
collect more information about the system environment.
For SAP HANA 1.0, use the SAP HANA hardware configuration check tool (SAP
Note 1943937) to determine bottlenecks on the infrastructure side.
For SAP HANA 2.0, use the SAP HANA Hardware and Cloud Measurement Tools (SAP Note
2493172) to determine bottlenecks on the infrastructure side.
© Copyright. All rights reserved.
31
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
See SAP Note 1999020: SAP HANA: Troubleshooting, when the database is no longer
reachable for further specific steps and guidance on the proactive or reactive actions you can
take.
Analyze an Unresponsive System in SAP HANA Cockpit
The Troubleshoot Unresponsive Systems function uses the SAP host agent. It can collect the
most important diagnosis information from the SAP HANA database, even when the system
is stopped or cannot be reached by SQL due to performance problems.
Figure 22: Troubleshoot an Unresponsive System
To troubleshoot a system in a hang state, there is a function in SAP HANA cockpit called
Troubleshoot unresponsive systems . When using this function, information is collected
through the SAP host agent. The communication between the web browser and the SAP host
agent is always done over HTTPS, which requires that the SAP host agent has a Secure
Sockets Layer (SSL) certificate (PSE) in its security directory.
The information is collected into a file named emergency_info_<SID>.zip by the Python script
emergencyInfo.py. This script connects to the index server, using the hdbcons interface.
The script tries to collect information about the open connections, running transactions, and
threads. It also shows blocked transactions. If the index server is unavailable, no information
is shown.
The information is collected by a Python script that connects to the index server, using the
hdbcons interface. The script tries to collect information about the open connections,
running transactions, and threads. It also shows blocked transactions. If the index server is
unavailable, no information is shown.
© Copyright. All rights reserved.
32
1/1
3/1/2021
SAP e-book
Lesson: Handling System Hang but Reachable Situations
Figure 23: Troubleshoot Unresponsive Systems
The Troubleshoot Unresponsive System function organizes information about the system by
tab. You can diagnose the following:
●
Connections
●
Transactions
●
Blocked transactions
●
Threads
© Copyright. All rights reserved.
33
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Connections Tab
Figure 24: Connections Tab
Analyzing the sessions connected to your SAP HANA database helps you identify which
applications, or which users, are currently connected to your system, as well as what they are
doing in terms of SQL execution.
On the CONNECTIONS tab, you can see information about the current connections to the SAP
HANA server. This information includes connection start time, ID, user name, and status. If
there are many connections open to the server, it can lead to congestion and may result in the
server becoming unresponsive.
On the CONNECTIONS tab, you can use the Cancel Connection button to stop a single
connection. To do this, select the connection that you want to cancel, and choose Cancel
Connection.
You can stop all the transactions that are currently running by choosing Cancel All
Transactions .
© Copyright. All rights reserved.
34
1/1
3/1/2021
SAP e-book
Lesson: Handling System Hang but Reachable Situations
Transactions Tab
Figure 25: Transactions Tab
On the TRANSACTIONS tab, you can see information about the current transactions in the
SAP HANA system. This information includes connection and transaction ID, allocated
memory, and user name. The information shown in the transactions table gives you a good
insight into current activity on the system.
Via the connection ID and the primary connection tab, you can even link the transaction to the
corresponding connection.
You can stop all the transactions that are currently running by choosing Cancel All
Transactions .
Blocked Transactions Tab
Figure 26: Blocked Transactions Tab
On the BLOCKED TRANSACTIONS tab, you can investigate if there are blocked transactions in
your system.
© Copyright. All rights reserved.
35
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Blocked transactions are transactions that cannot be processed further because they need to
acquire transactional locks (record or table locks) that are currently held by another
transaction. Transactions can also be blocked while waiting for other resources, such as the
network or disk access (database or metadata locks).
The type of lock held by the blocking transaction (record, table, or metadata) is indicated in
the Lock Type column.
The lock mode is indicated in the Transactional Lock Type column.
Exclusive: Row-level locks prevent concurrent write operations on the same record. They are
acquired implicitly by update and delete operations, or explicitly with the SELECT FOR
UPDATE statement.
Table-level: Locks prevent operations on the content of a table from interfering with changes
to the table definition (such as drop table or alter table). DML operations on the table content
require an intentional exclusive lock, while changes to the table definition (DDL operations)
require an exclusive table lock. There is also a LOCK TABLE statement for explicitly locking a
table. Intentional exclusive locks can be acquired if no other transaction holds an exclusive
lock for the same object. Exclusive locks require that no other transaction holds a lock for the
same object (neither intentional exclusive nor exclusive).
For more detailed analysis of blocked transactions, information about low-level locks is
available in the columns Lock Wait Name, Lock Wait Component, and Thread ID of Low-Level
Lock Owner. Low-level locks are locks acquired at the thread level. They manage code-level
access to a range of resources (for example, internal data structures, network, or disk). Lock
wait components group low-level locks by engine component or resource.
By choosing Cancel All Transactions , you can stop all the current running transactions.
Because the Delta Table Merge needs to lock tables to proceed, it is a common cause of
blocked transactions. Another job displayed by this monitor is the savepoint write, which
needs to pull a global database lock in its critical phase. A common issue is a flaw in the
application coding that does not commit a write transaction. Such a transaction will block any
other transaction that needs to access the same database object. To remedy the situation,
close the blocking transaction.
In the UI table, the blocked transactions are displayed directly beneath the blocking
transaction.
First, you must determine whether there is only one, or a few transactions, blocking many
other transactions. To do this, open the Blocked Transaction tab and check the amount of
blocking transactions. If there are only a few blocking transactions, there is probably an issue
on the application side. To resolve the problem, use the following techniques:
1. If only one transaction is blocked, contact the application user and the developer. Firstly,
ask the user to close the application and secondly, to check if there is a general issue with
the application code.
If you are not able to contact the user, you can kill the transaction or kill the client process
that opened the session. The transaction is rolled back. The session cancellation may take
some time to succeed. If it takes longer than 30 seconds, consider this as a bug and
contact development support.
If the session cancellation takes too long or does not complete at all, you can kill the client
process that opened the session. This also terminates the blocking transaction. As a
prerequisite, you must have access to the client machine.
© Copyright. All rights reserved.
36
1/1
3/1/2021
SAP e-book
Lesson: Handling System Hang but Reachable Situations
2. If a large amount of transactions are blocked, you must find out whether a specific access
pattern is causing the issue. If multiple transactions are trying to access the same
database objects with write operations, they block each other. To check if this is
happening, open the Blocked Transaction Monitor and analyze the Waiting Schema Name,
Waiting Object Name, and Waiting Record ID columns. If you find a lot of transactions that
are blocking many other transactions, you must investigate whether you can do the
following:
a. Change the client application(s) to avoid the access pattern.
b. If a background job that issues many write transactions (for example, a data load job)
is running, reschedule it to a period with a low user load.
c. Partition tables that are accessed frequently to avoid clashes. See the SAP HANA
Administration Guide for more details on partitioning.
3. If you cannot identify specific transactions or specific database objects that lead to
transactions being blocked, you can assume that there is a problem with the database
itself or with its configuration, an issue with the delta merge (for example, mass write
operation on a column table), or a long savepoint duration.
Threads Tab
Figure 27: Threads Tab
In the THREADS tab, you can identify which statements or procedures are being executed and
at what stage they are, who else is connected to the system, and if there are any internal
processes running as well. The information shown includes thread type, ID, and thread status.
You can also find information of system user and waiting time. The information shown in the
table helps you to identify transactions with high average wait times. With the user name,
thread ID, and wait time columns, you can identify which thread is causing problems.
In this tab, you can identify long-running threads and those threads that are blocked for an
inexplicable long period of time.
In the case of an emergency, choose Cancel All Transactions .
© Copyright. All rights reserved.
37
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
LESSON SUMMARY
You should now be able to:
●
Handle system hanging but reachable situations
© Copyright. All rights reserved.
38
1/1
3/1/2021
SAP e-book
Unit 1
Lesson 3
Analyzing a Suddenly Slow System
LESSON OBJECTIVES
After completing this lesson, you will be able to:
●
Analyze a suddenly slow system
Analyze a Suddenly Slow System
There can are various reasons why an SAP HANA system suddenly becomes very slow. This
lesson tackles issues where the database performance unexpectedly become very slow.
Figure 28: System Suddenly Slow
The following issues can cause a system to suddenly become slow (that is, from the end-user
perspective the SAP HANA system seems to be performing slowly):
●
Hardware failures at the server level (read errors on bad memory)
●
Hardware failures at the storage level (read errors on disk)
●
Hardware failures at the network level (package collisions on switches or a router)
●
Software errors at the OS level (OS command on Linux is using 100% CPU/memory
swapping on disk)
© Copyright. All rights reserved.
39
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
●
●
Software errors at the storage system level (software errors on SAN/NAS)
Software errors at the database level (long running queries on SAP HANA or too many
sessions open)
Figure 29: Collect OS Logs
If the slow system is caused by something at the OS level, it is important to save log files on
the Linux OS or at storage system level for later analysis.
Usually, in a slow system scenario, the system can still be accessed through SQL, but it takes
longer to investigate. The normal SQL access should be used to further analyze the possible
root cause.
What Can and Should You Do?
As SQL statements can still be executed for analysis, you must perform the following steps
using the SAP HANA Cockpit 2.0:
●
●
Look at the Overall Database Status card to see if all the services are running correctly.
Look at the Memory Usage card to see if the SAP HANA database has enough memory
available.
●
Look at the CPU Usage card to see who is using the CPU resources.
●
Look at the Disk Usage card to see who is using the disk resources.
●
Look at the Thread card to see which threads are using the resources.
●
Look at the Sessions card to see which sessions are running.
●
Look at the Monitor Statements card to see which SQL statements are running on the
system.
If the slow system is caused by something in the SAP HANA database, is it important to
investigate the SQL query and save log and trace files at the SAP HANA database level for
further analysis, before terminating sessions or threads.
© Copyright. All rights reserved.
40
1/1
3/1/2021
SAP e-book
Lesson: Analyzing a Suddenly Slow System
Database Overview Application
The Database Overview for a system displays important metrics and available functions,
regardless of whether it is a single-host or multi-host system, or a tenant database.
In the SAP HANA Database Overview, you can view key health indicators for this specific
database, such as services status, alerts, and resource utilization. You also have access to
tools that allow you to perform database administrations tasks, such as performance
analysis, and SQL statement execution. Different parts of a single card can link to different
views or applications. This way, you can see various components in a single view and decide
whether to further examine issues by drilling down.
Figure 30: Database Overview
Details also include the following:
●
Information such as operational status, system usage type, whether the system has
multiple hosts, the number of hosts (if distributed), and database version
●
The SAP HANA version history
●
Information about the plug-ins that are installed
●
The status of replication from your productive system to a secondary system
This information is only available and applicable if you are operating a secondary instance
of your database (for example, in a high availability (HA) scenario). If this is the case,
content from the primary or productive instance of your database is replicated to the
secondary instance.
To launch the overview, drill down to the name of the database from the Database Directory
or from a group. Unless your administrator has enabled single sign-on, you must connect to
© Copyright. All rights reserved.
41
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
the database with a database user that has the system privilege CATALOG READ and choose
_SYS_STATISTICS .
Overall Database Status Card
To identify problems early and avoid disruptions, you must monitor your SAP HANA database
continuously.
You can monitor the overall status and resource usage of the SAP HANA database at a glance
on the homepage of the SAP HANA cockpit. Then, drill down for more detailed monitoring and
analysis.
The overall database status can be running, running with issues, or stopped. Choosing this
status brings you to Manage Services, where you can stop or kill a service, and start or stop a
system. Also, in the Overall Database Status , you receive alerts related to services, and, in
order of priority, you see:
●
The number of services not running
●
The number of services running with issues (if there are no stopped services)
●
The number of services running
Manage Services
Manage Services provides you with detailed information about database services for an
individual database.
Note:
Not all the columns listed in the following figure are visible by default. You can add
and remove columns in the table personalization dialog, which you open by
choosing the personalization icon in the table toolbar.
© Copyright. All rights reserved.
42
1/1
3/1/2021
SAP e-book
Lesson: Analyzing a Suddenly Slow System
Figure 31: Manage Services
The following list gives an overview of the information available on each service:
●
Host: The name of the host on which the service is running
●
Status: The status of the service. The following statuses are possible:
-
Running
-
Running with Issues
-
Stopped
-
Not Running
To investigate why the service is not running, you can navigate to the crashdump file,
created when the service stopped.
Note:
The crashdump file opens in the Trace tool of the SAP HANA Web-based
Development Workbench. For this, you need the role
sap.hana.xs.ide.roles::TraceViewer or the parent role
sap.hana.xs.ide.roles::Developer.
●
●
Service: The service name, for example, indexserver, nameserver, xsengine, and so on
Role: The role of the service in a failover situation. Automatic failover happens when the
service or the host on which the service is running fails. The following values are possible:
© Copyright. All rights reserved.
43
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
-
Master: The service is the active master worker.
-
No entry: The service is a slave worker.
-
●
●
●
●
Standby: The service is in standby mode. It does not contain any data and does not
receive any requests.
Port: The port that the system uses for internal communication between services
Start Time: The time at which the service started. The time is given in the timezone of the
SAP HANA server.
CPU: A mini chart visualizing the CPU usage of the service. To open the Performance
Monitor for a more detailed breakdown of CPU usage, choose the mini chart.
Memory: A mini chart visualizes the memory usage of the service as follows:
-
Dark green shows the service's used memory.
-
Light green shows the service's peak memory.
-
The gray stroke represents the effective allocation limit.
-
The light gray background represents the physical memory.
To open Memory Analysis for a more detailed breakdown of memory usage as follows,
choose the mini chart:
-
-
-
-
-
Used Memory (MB): The amount of memory currently used by the service. Choosing
the mini chart opens the Memory Analysis app for a more detailed breakdown of
memory usage.
Peak Memory (MB): The highest amount of memory ever used by the service
Effective Allocation Limit (MB): The effective maximum memory pool size that is
available to the process considering the current memory pool sizes of other processes
Memory Physical on Host (MB): The total memory available on the host
All Process Memory on Host (MB): The total used physical memory and swap memory
on the host
-
Allocated Heap Memory (MB): The heap part of the allocated memory pool
-
Allocated Shared Memory (MB): The shared memory part of the allocated memory pool
-
Allocation Limit (MB): The maximum size of the allocated memory pool
-
CPU Process (%): The CPU usage of the process
-
CPU Host (%): The CPU usage on the host
-
Memory Virtual on Host (MB): The virtual memory of the host
-
Process Physical Memory (MB): The process physical memory used
-
Process Virtual Memory (MB): The process virtual memory
-
Shrinkable Size of Caches (MB): The memory that can be freed in the event of a
memory shortage
© Copyright. All rights reserved.
44
1/1
3/1/2021
SAP e-book
Lesson: Analyzing a Suddenly Slow System
-
Size of Caches (MB): The part of the allocated memory pool that can potentially be
freed in the event of a memory shortage
-
Size of Shared Libraries (MB): The code size (including shared libraries)
-
Size of Thread Stacks (MB): The size of the service thread call stacks
-
Used Heap Memory (MB): The amount of the process heap memory used
-
Used Shared Memory (MB): The amount of the process shared memory used
-
SQL Port: The SQL port number
-
Process ID: The process ID
Operations on Services
As an administrator, you may need to perform certain operations on all, or selected services for example, start missing services, or stop or kill a service.
You can perform several operations on database services from Manage Services. You can
trigger these operations by selecting the service, and choosing the required option in the
footer toolbar.
Choose Start Missing Services to start inactive services. This can only be performed on a
tenant database if you drill down to Manage Services, through the system database.
Choose Stop Service to stop the selected service normally. The service is then typically
restarted.
Choose Kill Service to stop the selected service immediately and, if the related option is
selected, create a crashdump file. The service is then typically restarted.
Choose Add Service to add the service you selected from the list. This can only be performed
on a tenant database if you drill down to Manage Services, through the system database.
Services cannot be added to the system database itself. To add a service, you must have the
EXECUTE privilege on the stored procedure SYS. UPDATE_LANDSCAPE_CONFIGURATION.
Choose Remove Service to remove the selected service. This can only be performed on a
tenant database if you drill down to Manage Services, through the system database.
You can only remove services that have their own persistence. If data is still stored in the
service's persistence, it is re-distributed to other services.
You cannot remove the following services:
●
Name server
●
Master index server
●
Primary index server on a host
To remove a service, you must have the EXECUTE privilege on the stored procedure SYS.
UPDATE_LANDSCAPE_CONFIGURATION.
Choose Reset Memory Statistics to reset all memory statistics for all services. This can only
be performed on a tenant database if you drill down to Manage Services application, through
the system database.
Peak used memory is the highest recorded value for used memory since the last time the
memory statistics were reset. This value is useful for understanding the behavior of used
memory over time and under peak loads. Resetting peak used memory allows you, for
© Copyright. All rights reserved.
45
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
example, to establish the impact of a certain workload on memory usage. If you reset peak
used memory and run the workload, you can then examine the new peak used memory value.
Choose Go To Alerts to display the alerts for this database.
The SAP HANA database provides several features in support of high availability, one of which
is service auto-restart. In the event of a failure, or an intentional intervention by an
administrator that disables one of the SAP HANA services, the service auto-restart function
automatically detects the failure and restarts the stopped service process.
Memory Statistics per Service
Analyzing the memory allocation of the SAP HANA database can help you to investigate such
situations as out-of-memory incidents, memory corruptions, and memory leaks.
Figure 32: Memory Analysis – Per Service
The Memory Analysis app enables you to visualize and explore the memory allocation of every
service of a selected host during a specified time period. If you notice an increase in overall
memory usage, you can investigate whether it is due to a particular component,
subcomponent, or table.
The upper chart provides the following data:
●
●
Global Allocation Limit: This is the global_allocation_limit for the host (as set in the
global.ini configuration file).
Allocated Memory: This is the pool of memory pre-allocated by the host for storing inmemory table data, thread stacks, temporary results, and other system data structures.
© Copyright. All rights reserved.
46
1/1
3/1/2021
SAP e-book
Lesson: Analyzing a Suddenly Slow System
●
Total Used Memory: This is the total amount of memory used by SAP HANA, including
program code and stack, all data and system tables, and the memory required for
temporary computations.
Move the vertical selection bar in the upper chart to populate the data in the lower chart. The
vertical selection bar snaps to the closest time for which there is collected data for the
components. When you select the Components tab, the lower chart displays the Used
Memory by Component.
The Components tab provides the following detailed information:
●
●
●
Used Memory by Component: For the specific time (chosen by the vertical selection bar in
the upper chart), the components of the selected service are listed in descending order of
the amount of used memory.
Used Memory by Type: This donut chart displays a visual representation of the types of
used memory for the specified time.
Components Used Memory History: If you select the checkbox of one or more
components, the used memory history chart is populated.
The Subcomponents tab displays more detailed memory use. You can filter by component
type. You can move through the collected data points by using the arrow buttons. The
following information is displayed:
●
●
●
Used Memory by Subcomponent: Subcomponents of the selected component are listed in
descending order of used inclusive memory for the specific time (chosen by the vertical
selection bar in the upper chart). By choosing a subcomponent, you can expand the list.
Filter by Component Name: To further refine the displayed subcomponent data, select the
filter icon to specify one or more component names.
Subcomponents Used Memory History: Selecting the checkbox of one or more
subcomponents populates the used memory history chart.
The Tables tab shows detailed statistics on the memory used by data tables. The Tables tab
shows the following information:
●
●
Top Ten Tables by Size: This displays the breakdown of memory usage of the 10 highest
consuming tables for the specific time (chosen by the vertical selection bar in the upper
chart).
Top Ten Tables by Growth: This displays the memory usage of the 10 tables with the
largest change in consumption for the selected time period. By hovering over the data, you
can see the Previous Size memory usage value from the beginning of the time period and
the Growth during the time period (where the current size of the table is the sum of
Previous Size and Growth).
The following system views provide information from which the current and historical
memory allocation is calculated:
●
HOST_RESOURCE_UTILIZATION_STATISTICS
●
HOST_SERVICE_MEMORY
●
HOST_SERVICE_COMPONENT_MEMORY
●
HOST_HEAP_ALLOCATORS
●
GLOBAL_ROWSTORE_TABLES_SIZE_BASE
© Copyright. All rights reserved.
47
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
●
HOST_COLUMN_TABLES_PART_SIZE
All views are in the _SYS_STATISTICS schema. For more information about these views, see
the SAP HANA SQL and System Views Reference Guide.
Usage and Performance Metrics
You can monitor key database metrics through the Memory Usage, CPU Usage, and Disk
Usage cards, as well as the Threads, Sessions, and Monitor Statements cards.
In a multi-host system, each host is represented by a selectable bar, with the selected host
displaying a time graph to the right of the bar chart. Hover over the bars to see details for the
selected host.
If a bar is highlighted, there is an associated high (red) or medium (yellow) alert. With singlehost databases, as there is only one host, no bar graphs are displayed. By viewing this highlevel information, you can decide whether to drill down to the Performance Monitor .
Performance Monitor
Analyzing the performance of the SAP HANA database over time can help you to pinpoint
bottlenecks, identify patterns, and forecast requirements.
Use the Performance Monitor to visually analyze historical performance data across a range
of key performance indicators related to memory, disk, and CPU usage.
Open the Performance Monitor by choosing the chart or the Show All link on the Memory
Usage, CPU Usage, or Disk Usage card on the homepage of the SAP HANA cockpit.
Figure 33: Performance Monitor
The Performance Monitor can be reached through the Memory Usage, CPU Usage, and Disk
Usage cards. All three cards point to the same Performance Monitor , but the displayed data
© Copyright. All rights reserved.
48
1/1
3/1/2021
SAP e-book
Lesson: Analyzing a Suddenly Slow System
(memory, CPU, or disk) depends on the selected database. The general working of the
Performance Monitor is the same for all three of the cards.
The Performance Monitor opens displaying the load graph for the selected database
(memory, CPU, or disk). The load graph initially visualizes resource usage of all hosts and
services listed on the left, according to the default key performance indicator (KPI) group of
the selected database.
You can customize the information displayed on the load graph in several ways, for example:
●
●
Define the monitored time frame.
Use the Add Chart button to create custom charts displaying the host and services
selection, as well as selected KPIs. For a list of all available KPIs, see Key Performance
Indicators.
●
Update the displayed data by the selected refresh rate.
●
Zoom into a specific time by changing the duration.
●
In the Settings menu, customize your graphs by including hosts and services as well as
additional KPIs in the Charts tab. In the Alerts tab, configure alerts per category and
priority status.
Monitoring and Analyzing Sessions
Analyzing the sessions connected to your SAP HANA database can help you identify which
applications or which users are currently connected to your system, as well as what they are
doing in terms of SQL execution.
To use the Sessions card to monitor all sessions in your landscape, choose the mini chart.
Figure 34: Monitoring Sessions Card
© Copyright. All rights reserved.
49
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
The Sessions card displays the number of active and total sessions.
Open the Sessions card. The Sessions page allows you to monitor all sessions in the current
landscape. You can see the following information:
●
Active/inactive sessions and their relation to applications
●
Whether a session is blocked and, if so, which session is blocking it
●
The number of transactions that are blocked by a blocking session
●
●
Statistics such as average query runtime and the number of DML and DDL statements in a
session
The operator currently being processed by an active session
To support monitoring and analysis, you can perform the following actions from the Sessions
page:
●
To cancel a session, choose Cancel Sessions.
●
To save the data sets as a text or HTML file, choose Save As.
Monitoring and Analyzing Threads
Analyzing the threads running in the SAP HANA database can be helpful when analyzing the
current system load.
This analysis can help you identify which statements or procedures are being executed and at
what stage they are, who else is connected to the system, and if there are any internal
processes running also.
Use the Threads card to monitor the longest-running threads active in your system. It may be
useful to see, for example, how long a thread has been running, or if a thread is blocked for an
inexplicably long time.
Figure 35: Monitoring Threads Card
© Copyright. All rights reserved.
50
1/1
3/1/2021
SAP e-book
Lesson: Analyzing a Suddenly Slow System
The Threads card provides you with information about the number of currently active and
blocked threads in the database.
To open the Threads card, choose either the number of active or blocked threads on the card.
The 1,000 longest-running threads currently active in the database are listed. By default,
threads are listed in order of longest runtime. For each statement, you can see the duration,
as well as the name of the service that is executing the thread. You can identify the host, the
port, and the thread type, and whether the statement is related to a blocking transaction.
If a thread is involved in a blocked transaction, or is using an excessive amount of memory,
cancel the operation executing the thread by choosing Cancel Operations in the footer
toolbar.
Thread Details
The Threads card provides you with detailed information about the 1,000 longest-running
threads currently active in the database.
Note:
Not all of the columns listed in the following table are visible by default. You can
add and remove columns in the table personalization dialog, which you open by
choosing the personalization icon in the table toolbar.
The following table lists the information available for threads.
Detail
Description
Blocking Transaction
Blocking transaction
Duration (ms)
Duration (ms)
Host
Host name
Port
Internal port
Service
Service name
Hierarchy
Thread grouping information. Filled with Connection ID/Update Transaction ID/Transaction ID or left empty for inactive threads
Connection ID
Connection ID
Thread ID
Thread ID
Calling
The thread or service which the thread calls
Caller
The thread or service which called this thread
Thread Type
Thread Type
Thread Method
Thread method
Thread Detail
Thread detail
User
User name
Application User
Application user name
CPU Time
CPU time of thread
Cumulative CPU Time
CPU time of thread and associated children
© Copyright. All rights reserved.
51
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Detail
Description
Transaction ID
Transaction ID
Update Transaction ID
Update Transaction ID
Thread Status
Thread State
Connection Transaction ID
Transaction object ID
Connection Start Time
Connected Time
Connection Idle Time (ms)
Time that the connection is unused and idle
Connection Status
Connection Status: 'RUNNING' or 'IDLE'
Client Host
Host name of client machine
Client IP
IP of client machine
Client PID
Client Process ID
Connection Type
Connection type: Remote, Local, History (remote), History (local)
Own Connection
Own connection: TRUE if own connection,
FALSE if not
Memory Size per Connection
Allocated memory size per connection
Auto Commit
Commit mode of the current transaction:
TRUE if the current connection is in autocommit mode, FALSE otherwise
Last Action
The last action done by the current connection: ExecuteGroup, CommitTrans, AbortTrans, PrepareStatement, CloseStatement,
ExecutePrepared, ExecuteStatement, FetchCursor, CloseCursor, LobGetPiece, LogPutPiece, LobFind, Authenticate, Connect, Disconnect, ExecQidItab, CursorFetchItab, InsertIncompleteItab, AbapStream, TxStartXA,
TxJoinXA
Current Statement ID
Current statement ID
Current Operator Name
Current operator name
Fetched Record Count
Sum of the record count fetched by select
statements
Sent Message Size (Bytes)
Total size of messages sent by the current
connection
Sent Message Count
Total message count sent by the current
connection
Received Message Size (Byte)
Total size of messages/transactions received by the current connection
Received Message Count
Total message/transaction count received
by the current connection
© Copyright. All rights reserved.
52
1/1
3/1/2021
SAP e-book
Lesson: Analyzing a Suddenly Slow System
Detail
Description
Creator Thread ID
Thread ID who created the current connection
Created By
Engine component that created the connections: Session, Planning, Repository, CalcEngine, Authentication, Table Exporter, Loader,
LLVM, JSVM, IMS Search API, OLAP Engine,
Mergedog, Ping Status, Name Server, Queue
Server, SQL Stored Procedure, Authorization, TrexViaDbsl from ABAP, HybridTable
Reorganizer, Session external
Is Encrypted
Encrypted: TRUE if the secure communication is enabled (SSL enabled), FALSE, otherwise
Connection End Time
The time when the connection is closed for
history connections
Blocked Update Transaction ID
Write transaction ID of the write transaction
waiting for the lock
Blocking Transaction ID
Transaction object ID of the transaction holding the lock
Thread ID of Lock Owner
Connection ID associated with the blocked
write transaction
Blocking Update Transaction ID
Write transaction ID of the write transaction
holding the lock
Transactional Lock Type
Transactional lock type
Transactional Lock Mode
Transactional lock mode
Lock Wait Component
Waiting for lock component
Lock Wait Name
Waiting for lock ID
Timestamp of Blocked Transaction
Timestamp of the blocked transaction
Waiting Record ID
ID of the record on which the lock is currently
placed
Waiting Object Name
Name of the object on which the lock is currently placed
Waiting Object Type
Type of the object on which the lock is currently placed
Waiting Schema Name
Name of the schema on which the lock is currently placed
Monitoring and Analyzing with the Top SQL Statements
Analyzing the current most critical statements running in the SAP HANA database can help
you identify the root cause of poor performance, CPU bottlenecks, or out-of-memory
scenarios. Enabling memory tracking allows you to monitor the amount of memory used by
single statement executions.
© Copyright. All rights reserved.
53
1/1
3/1/2021
SAP e-book
Unit 1: Emergency Analysis and Troubleshooting
Use the Top SQL Statements card to analyze the current most critical statements running in
the database.
Figure 36: Monitoring Statements Card
The Top SQL Statements card displays the number of long-running statements and the longrunning blocking situations currently active in the database. Statements are ranked based on
a combination of the following criteria:
●
The runtime of the current statement execution.
●
The lock wait time of the current statement execution.
●
The cursor duration of the current statement execution.
Open the Top SQL Statements card to list the 100 most critical statements currently active in
the database. By default, statements are listed in order of the longest runtime. For each
statement, you can see the full statement string, as well as the ID of the session in which the
statement is running. You can identify the application, the application user, and the database
user running the statement and whether the statement is related to a blocking transaction.
Optionally, you can activate monitoring of the memory consumption of statements by
choosing Enable Memory Tracking in the footer toolbar. Detailed information about the
memory consumption of statement execution is collected and displayed.
If a statement is involved in a blocked transaction or using an excessive amount of memory,
cancel the session that the statement is running in (or the blocking session) by choosing
Cancel Session in the footer toolbar.
© Copyright. All rights reserved.
54
1/1
3/1/2021
SAP e-book
Lesson: Analyzing a Suddenly Slow System
LESSON SUMMARY
You should now be able to:
●
Analyze a suddenly slow system
© Copyright. All rights reserved.
55
1/1
3/1/2021
SAP e-book
Unit 1
Learning Assessment
1. What kind of query language statements can be used in the SAP HANA Database
Explorer?
Choose the correct answer.
X
A SQL
X
B MDX
X
C XML
X
D TCL
2. In a system-down scenario, SQL can be used to access the system to further analyze the
problem.
Determine whether this statement is true or false.
X
True
X
False
3. Which analysis steps can be performed when the SAP HANA database cannot be reached
using SQL?
Choose the correct answer.
X
A Check the /var/log/messages for SAP HANA error messages.
X
B Use the deadlock detector functionality of hdbcons.
X
C Use the Heath Monitor of SAP HANA Cockpit 2.0.
X
D Check the hardware with the SAP HANA Hardware Configuration Check Tool.
4. The Troubleshoot Unresponsive Systems function collects its data through the
fullSystemInfoDump.py python script.
Determine whether this statement is true or false.
X
True
X
False
© Copyright. All rights reserved.
56
1/1
3/1/2021
SAP e-book
Unit 1: Learning Assessment
5. Which monitoring KPIs are provided in the Manage Services application?
Choose the correct answers.
X
A CPU usage
X
B Service status
X
C Disk usage
X
D Lock status
6. Because SAP HANA is running in-memory, a slow system situation is always caused by a
problem at OS level.
Determine whether this statement is true or false.
X
True
X
False
© Copyright. All rights reserved.
57
1/1
3/1/2021
SAP e-book
Unit 1
Learning Assessment - Answers
1. What kind of query language statements can be used in the SAP HANA Database
Explorer?
Choose the correct answer.
X
A SQL
X
B MDX
X
C XML
X
D TCL
Correct! The SAP HANA Database Explorer supports only SQL. Read more about this in
the lesson "Handling System Offline Situations" of the course HA215.
2. In a system-down scenario, SQL can be used to access the system to further analyze the
problem.
Determine whether this statement is true or false.
X
True
X
False
Correct! When the SAP HANA database is down, you should use the Troubleshoot
Unresponsive System application to analyze the problem. Read more about this in the
lesson "Handling System Offline Situations" of the course HA215.
3. Which analysis steps can be performed when the SAP HANA database cannot be reached
using SQL?
Choose the correct answer.
X
A Check the /var/log/messages for SAP HANA error messages.
X
B Use the deadlock detector functionality of hdbcons.
X
C Use the Heath Monitor of SAP HANA Cockpit 2.0.
X
D Check the hardware with the SAP HANA Hardware Configuration Check Tool.
Correct! The deadlock detector functionality is part of the hdbcons tool, and can be used
for analyzing an unreachable system. Read more about this in the lesson "Handling
System Hang but Reachable Situations" of the course HA215.
© Copyright. All rights reserved.
58
1/1
3/1/2021
SAP e-book
Unit 1: Learning Assessment - Answers
4. The Troubleshoot Unresponsive Systems function collects its data through the
fullSystemInfoDump.py python script.
Determine whether this statement is true or false.
X
True
X
False
Correct! The Troubleshoot Unresponsive Systems function collects its data through the
SAP Host Agent. Read more on this in the lesson "Handling System Hanging but
Reachable Situations" of the course HA215.
5. Which monitoring KPIs are provided in the Manage Services application?
Choose the correct answers.
X
A CPU usage
X
B Service status
X
C Disk usage
X
D Lock status
Correct! The CPU usage and Service status KPIs are shown in the Manage Services
application. Read more on this in the lesson "Analyzing a Suddenly Slow System" of the
course HA215.
6. Because SAP HANA is running in-memory, a slow system situation is always caused by a
problem at OS level.
Determine whether this statement is true or false.
X
True
X
False
Correct! A slow system can be caused by OS, network, hardware, other software, or the
SAP HANA database. The problem could be with any of these components. Read more on
this in the lesson "Analyzing a Suddenly Slow System" of the course HA215.
© Copyright. All rights reserved.
59
1/1