A Comprehensive Guide to
Building or Improving Your
Enterprise Data Warehouse
with SAP NetWeaver BW
7.3
Dr. Bjarne Berg
COMERIT
© 2012 Wellesley Information Services. All rights reserved.
In This Session …
•
Take a look at the latest enterprise data warehousing capabilities
in SAP NetWeaver® BW 7.3
•
Examine new data modeling capabilities
•
Understand SAP’s vision for SAP NetWeaver BW, and its
evolution into an in-memory EDW solution. Gain insight into the
deployment of SAP HANA data marts.
•
Get insight into the new ETL interface, new management
monitoring tools, the hybrid provider options for real-time data
warehousing in SAP NetWeaver BW 7.3, and the capabilities of the
Layered Scalable Architecture (LSA)
•
Understand how in-memory technology impacts data
warehousing capabilities, existing SAP NetWeaver BW customers,
and the SAP data warehousing roadmap
1
What We’ll Cover …
•
•
•
•
•
•
•
•
New Administrative Cockpit Capabilities
The New Semantic Partition (SPO) Options
The HybridProvider and Real-Time Data
Faster Data Load and New Design Options
Front-End and Security Benefits
Using SAP NetWeaver BW 7.3 for Performance - Real Examples
New In-Memory Possibilities with SAP NetWeaver BW 7.3 and SAP
NetWeaver BW Accelerator 7.2
Wrap-up
2
New 7.3 Performance and Admin Monitoring Capabilities
•
New monitors include:
 Monitor of database usage and object sizes (i.e., InfoCubes, DSOs)
 Query usage statistics are more visible (similar to RSRT, RSRV, RSTT)
 We can see more of the use of SAP NetWeaver BW Accelerator and
sizes
 Monitor for the actual use of OLAP/MDX Cache and hit ratios
 You can now selectively delete internal statistics in RSDDSTATWHM by
date through the updated RSDDSTAT_DATA_DELETE ABAP program
 There is also a new MDX Editor for coding and syntax assistance
Solution Manager has
been updated to take
advantage of these
new monitors.
3
New 7.3 Performance and Admin Monitoring Capabilities
(cont.)
•
Other new monitors include:
 DEAMON update information (i.e., RDA capacity status, usage)
 A performance monitoring workbench for performance trends
 Process chain monitoring (new transaction: RSPCM) with error
and active chain monitoring, user specific displays, and
performance threshold monitoring (i.e., for SLAs)
NLS: In SAP NetWeaver
BW 7.3, the Near Line
Storage has been
enhanced to include
archiving, support for
write optimized DSOs,
and database support for
tools such as DB2 Viper
(v9.5) and more
44
New: Specific Database Support — IBM DB2
•
IBM DB2: 7.3 supports specific database features:
 DB2 v9.5 supports PSA, DSO, and fact table compressions for
reduced disk volume (integrates with DB2 storage
management)
 Support for MDC clustering in the DB Cockpit. This is available
in v9.5.2. or higher and is the default for all DSO tables and PSA
in version 9.7.
 Much faster request deletion if MDC clustering is used
 DB2 v9.7 supports Index compressions for reduced disk
volume
 Support for IBM DB2’s DPF (Database Partitioning Feature)
In the past, SAP NetWeaver BW has been unable to take advantage of
vendor specific database features. This is changed, and fewer
workarounds are needed.
55
What We’ll Cover …
•
•
•
•
•
•
•
•
New Administrative Cockpit Capabilities
The New Semantic Partition (SPO) Options
The HybridProvider and Real-Time Data
Faster Data Load and New Design Options
Front-End and Security Benefits
Using SAP NetWeaver BW 7.3 for Performance - Real Examples
New In-Memory Possibilities with SAP NetWeaver BW 7.3 and SAP
NetWeaver BW Accelerator 7.2
Wrap-up
6
Semantic Partitioned Objects (SPO)
•
•
•
When data stores and InfoCubes are allowed to grow over time, the data load
and query performance suffers
Normally objects should be physically partitioned when the numbers of records
exceed 100 million
 However, this may be different depending on the size of your hardware and
the type of database you use
In SAP NetWeaver BW 7.3 we get an option to create a Semantic Partitioned
Object (SPO) through wizards
 You can partition based on fields such as calendar year, region, country, etc.
7
Semantic Partitioned Objects (SPO) (cont.)
•
When an SPO is created, a reference structure keeps track of the
partitions. The structure is placed in the MultiProvider for querying.
Source: SAP AG, 2011
SPO Wizards create all Data Transfer Processes (DTP), transformations,
filters for each data store, and a process chain automatically.
8
What We’ll Cover …
•
•
•
•
•
•
•
•
New Administrative Cockpit Capabilities
The New Semantic Partition (SPO) Options
The HybridProvider and Real-Time Data
Faster Data Load and New Design Options
Front-End and Security Benefits
Using SAP NetWeaver BW 7.3 for Performance - Real Examples
New In-Memory Possibilities with SAP NetWeaver BW 7.3 and SAP
NetWeaver BW Accelerator 7.2
Wrap-up
9
The New HybridProvider and Real-Time Data
•
•
The “HybridProvider” (HP) is new in SAP NetWeaver BW 7.3. The
core idea is to link the historical data inside BW with real-time
data.
There are two ways of implementing an HP:
 HP based on a DSO
 HP based on a Virtual InfoCube
Real-Time Data
Acquisition
HybridProvider
Transaction
Data
Indexing
ECC
and
External
Systems
BWA based
InfoCube
Direct
reads
Virtual
InfoCube
RDA
Real-Time
Data
Acquisition
Query
(real time)
Direct
Access
10
Option 1: The DSO-Based HybridProvider
•
Core features:
 Real-time data is in the DSO and historical data in the SAP NetWeaver BW
Accelerator-based InfoCube
 The DSO use real-time data acquisition (RDA) to load data
 SAP NetWeaver BW automatically creates a process chain for the
HybridProvder’s data flow
 The process chain is executed for every closed request
Real-Time Data
Acquisition
HybridProvider
Transaction
Data
Indexing
ECC
and
External
Systems
BWA based
InfoCube
Direct
reads
RDA
Real-Time
Data
Acquisition
Query
(real time)
Direct
Access
Virtual
InfoCube
This solution provides for really fast queries, but delta logic has to be custom designed
11
Option 1: The DSO-Based HybridProvider (cont.)
•
This solution provides for really fast queries, but delta logic has to
be custom designed and may be complex. However, the solution
allows for high-frequency updates and very rapid query response.
This is a good option if
you have a low volume of
new records and a high
number of queries or
operational dashboards
12
Option 2: The Virtual Cube-Based HybridProvider
•
Core features:
 Data is read in real-time from SAP ECC, while historical data is read from
SAP NetWeaver BW Accelerator
 The difference depends on how often SAP NetWeaver BW Accelerator is
loaded
 Non-complex data logic can be applied
 DTP is permitted if you do not filter the data set
Real-Time Data
Acquisition
HybridProvider
Transaction
Data
Indexing
ECC
and
External
Systems
BWA based
InfoCube
Direct
reads
RDA
Real-Time
Data
Acquisition
Query
(real time)
Direct
Access
Virtual
InfoCube
Virtual cubes with many users may place high-stress on the ERP system
13
What We’ll Cover …
•
•
•
•
•
•
•
•
New Administrative Cockpit Capabilities
The New Semantic Partition (SPO) Options
The HybridProvider and Real-Time Data
Faster Data Load and New Design Options
Front-End and Security Benefits
Using SAP NetWeaver BW 7.3 for Performance - Real Examples
New In-Memory Possibilities with SAP NetWeaver BW 7.3 and SAP
NetWeaver BW Accelerator 7.2
Wrap-up
Hint for Experts: The search feature in the SAP
NetWeaver BW 7.3 Admin Workbench has been
improved so that it is easier to search more items
such as process chains.
14
Faster Data Load and New Design Options — Activation
•
•
•
During activation, SAP NetWeaver BW 7.0 has to lookup in the
NIRV table to see if the object already exists
 This can be a slow process
In SAP NetWeaver BW 7.0 we may buffer the number ranges to
compare the data load with records in-memory
 This speeds up data activation
However, in SAP NetWeaver BW 7.3, the data activation is
changed from single lookups to package fetch of the active table,
resulting in faster activation and less locks on the lookup tables
 The new method may result in 15-30% faster data activation (2040% in lab tests)
15
Faster Data Load and New Design Options — Activation
(cont.)
•
•
For data transformations, the option “Read from DataStore” for a
faster data lookup is also available
Additionally, the use of navigational attributes as sources in
Masterdata transformations reduce overhead for lookups
 Combined, this may lead to an additional 10-20% improvement
The 7.3 initial load runtime option “Insert only” and the “Unique
data records only” prevents all lookups during activation
16
New Data Design Options — Delta, Init, and Web Services
•
•
•
In SAP NetWeaver BW 7.3 you can
create generic delta extraction for
the Universal Data (UD) and
Database Connect (DB) options,
as well as for flat files
Additionally, you can use the
new DataSource adapter “Web
Service Pull” to load data from
external Web services
 You can even create generic Web services delta loads and load the new data
straight into the staging area of SAP NetWeaver BW 7.3
While Web services does not support hierarchies yet, there is now integration
of hierarchies into the standard process flow such as transformation and DTPs,
as well as being able to load hierarchies from flat files using a new DataSource
When you use delta loads, the first time SAP NetWeaver BW 7.3
automatically defines it as an “init load.” After that, it
automatically switches to “delta” as the InfoPackage mode (no
need to define it anymore!)
17
New Options — ETL Interface and Shut-Downs
•
SAP NetWeaver BW 7.3 has a
new interface to create and
manage data flows graphically in
a “drag-and-drop” manner,
including:
 Data Flow Copy to copy data
flows and process chains
 Data Flow Migration to
migrate, or reverse migrate,
data flows with automatic
adoption of InfoPackages and
process chains
Source: SAP AG, 2011
Use “RS_SYSTEM_SHUTDOWN” to restart an application server and
suspend RDA and process chains and re-start them in a controlled manner.
Data loads are not in “flux” and you do not have to clean up partial loads.
18
The New DataFlow Generation Wizard
•
•
SAP NetWeaver BW 7.3 has a new, step-by-step
wizard that allows you to generate data flows from
flat files or existing data sources
A great benefit is that
the wizards work
against any
InfoProvider; i.e., you
can use the wizards
to create loads from
DSOs to DSOs or
InfoCubes
This wizard reduces the number or manual steps needed to load data. It also
simplifies the development process and makes ETL work much easier.
19
Layered Scalable Architecture (LSA) – New Templates
•
The LSA consists logically of:
 Acquisition layer
 Harmonization/quality layer
 Propogation layer
 Business transformation layer
 Reporting layer
 Virtualization layer
SAP BW 7.3 SP-3 has a set of 10
templates to help build a layered
data architecture for large-scale
data warehousing
Source: sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/24800
20
What We’ll Cover …
•
•
•
•
•
•
•
•
New Administrative Cockpit Capabilities
The New Semantic Partition (SPO) Options
The HybridProvider and Real-Time Data
Faster Data Load and New Design Options
Front-End and Security Benefits
Using SAP NetWeaver BW 7.3 for Performance - Real Examples
New In-Memory Possibilities with SAP NetWeaver BW 7.3 and SAP
NetWeaver BW Accelerator 7.2
Wrap-up
21
Security Changes and New Mass Change Feature
•
When companies upgraded from SAP
NetWeaver BW version 3.5 to 7.0, security
migration was optional and many did only a
technical upgrade and kept the “obsolete
authorization” concept. A security
migration is now required.
Great BW 7.3 Security Admin
Feature: You can make mass
changes to authorizations
instead of one-by-one. This can
be done by cut-and-paste in a
worklist, hierarchy nodes, and
you can also add users to
multiple analysis authorizations.
SAP recommends you migrate to the new concept before
upgrading a 7.0 system. (Notes: 931898; 938871; 946724, 958665; 1001652)
22
Security Features in SAP NetWeaver BW 7.0 and 7.3
•
•
Some companies did not do a security conversion in SAP NetWeaver BW 7.0
since items such as 0TCTAUTHH did not migrate (manually reassigned for the
hierarchies) and passwords became case sensitive. Now it is required.
SAP NetWeaver BW 7.0 still has the ABAP migration tool. It can be used in
SA38 (RSEC_MIGRATION), and migration can occur before the upgrade.
Authorizations are building
blocks of the 7.0/7.3
reporting concept and
security contains both the
data value and hierarchy
restrictions.
Security can be built using
the “RSECADMIN” transaction
SAP BW 3.5 and BW
7.0 without security
conversion
BW 7.0 and 7.3 with
security conversion
Technical
Foundation
Maintenance
Number of
InfoObjects
Navigational
attributes
Hierarchy
authorization
Authorization Objects
Composition of
Authorizations
Only intersection of
authorization objects are
permitted
Per InfoCube and
Per InfoObject only
InfoObject
Authorization
Relevance
Analysis authorizations
Not changeable
Changeable
10 fields at 10 characters Unlimited
On global basis
Individually
Via GUID 0TCTAUTHH
Equivalent to value
authorizations
Union ('as expected')
23
23
BW 7.0 and 7.3 Authorization Objects
Authorization objects for the Data Warehousing Workbench:
After the required
S_RS_DS: For the DataSource or its sub objects (BI 7.x)
S_RS_ISNEW: For new InfoSources or their sub objects (NW BI 7.x) security conversion you
S_RS_DTP: For the data transfer process and its sub objects
have access to lower
S_RS_TR: For transformation rules and their sub objects
security granularity for
S_RS_CTT: For currency translation types
developers and users.
S_RS_UOM: For quantity conversion types
S_RS_THJT: For key date derivation types
S_RS_PLENQ: Authorizations for maintaining or displaying the lock settings
S_RS_RST: Authorization object for the RS trace tool
S_RS_PC: For process chains
S_RS_OHDEST: Open Hub Destination
Authorization objects for the Business Explorer:
S_RS_DAS: For Data Access Services
S_RS_BTMP: For BEx Web templates
S_RS_BEXTX: Authorizations for the maintenance of BEx texts
Authorization objects for the Admin of analysis authorizations
S_RSEC: Authorization for assignment and administration of analysis authorizations
S_RS_AUTH: Authorization object to include analysis authorizations in roles
Changed Authorization Objects:
S_RS_ADMWB (Data Warehousing Workbench: Objects):
New sub objects:
CONT_ACT – Installing Business Content; USE_DND - Drag & Drop to InfoAreas and
application components; CNG_RUN - Attribute change run
24
What We’ll Cover …
•
•
•
•
•
•
•
•
New Administrative Cockpit Capabilities
The New Semantic Partition (SPO) Options
The HybridProvider and Real-Time Data
Faster Data Load and New Design Options
Front-End and Security Benefits
Using SAP NetWeaver BW 7.3 for Performance - Real Examples
New In-Memory Possibilities with SAP NetWeaver BW 7.3 and SAP
NetWeaver BW Accelerator 7.2
Wrap-up
25
How To Leverage SAP NetWeaver BW 7.3 for Performance
•
•
In this section we will explore post implementation reviews from a
few real companies and see how version 7.3 may help in
achieving faster load, query, and response times
All examples come from real companies, but their names have
been removed
SAP NetWeaver BW 7.3 has many performance
benefits that can be retrofitted to earlier
implementations.
The trick is to use the upgrade as an opportunity
fix older designs.
26
Background – BW System Overview
•
This system has the following characteristics:
Area
•
Trend
Active Users
106
steady
Avg. Availability per Week
100 %
up
DB Size
Last Month DB Growth
3514.24 GB steady
131.88 GB up
Database:
BW system:
BPC system:
Operation systems:
Kernel:
Value
Oracle version 10.2.0
BW version 7.0 with content 7.0.4; Patch level 7
BPC content 7.0; Patch level 8
Linux 2.6 for BWA; AIX 6.1 for one app server and Solaris
10 for two app servers.
701, patch level 94 (newer patches existed)
The system is medium sized in terms of data volume, but has a
substantial hardware configuration
This company should upgrade to SP-9, it contained performance
improvements. Lessons learned: Keep an eye on new ServicePacks
27
More System Background
•
•
The system has been
in production for
several years and has
was upgraded two
years ago
Currently, there are
over 700 InfoProviders
in the system
InfoProviders
400
353
350
300
250
200
150
113
101
95
100
50
39
11
0
9
10
0
Most InfoCubes followed standard development guidelines, but some had
abnormalities such as InfoCubes feeding DSOs.
Lessons learned: structured design review sessions should be undertaken
as part of every project to assure that this does not continue.
28
28
Users, Security, and Waits
•
•
•
In January, there were 474 users logged on to the system
(including developers and basis staff)
Of these, 16 executed more than 400 navigation steps (high
usage)
 90 had moderate activity
 368 had low activity
2,271 users are allowed to see all reports in the system, and can
also list the content of all database tables. This is an unusual
security configuration that should be revisited as soon as
possible.
The customer should work with Network admin to determine capacity
towards the servers. During dialog with the servers, the average user
experienced a GUI wait time of almost 220 ms, indicating slow network
connectivity to the servers.
29
Database Growth and Cleanup
•
•
The database has grown
between 12 and 344 Gb each
month for the last year
Three months of the year saw
data, logs, and PSA cleaned.
Data volume declined
between 63 and 275 Gb those
months
Oracle DB (Mb)
3600
3500
3400
3300
3200
3100
3000
2900
2800
2700
2600
2500
The database has grown by
732Gb (26%) in the last year,
and the growth is uneven.
Oracle DB (Mb) Growth
400
300
200
100
0
Feb-2010 Mar-2010
-100
-200
-300
Apr-2010 May-2010 Jun-2010
Jul-2010
Aug-2010 Sep-2010
Oct-2010
Nov-2010 Dec-2010
Jan-2011
Feb-2011
To do: Schedule
“housekeeping” jobs. Better
management of cleanup
would result in more
predictable patterns.
(i.e. we found PSA data that had 10
30
months of load history).
30
Database Performance — RSA1  Manage InfoCubes  Performance
•
Database statistics are used by the
database optimizer to route queries.
Outdated statistics leads to
performance degradation.
•
Outdated indexes can lead to very poor search performance in all
queries where conditioning is used (i.e., mandatory prompts)
The current sampling rates (3% for some cubes) are too low, and
statistics should only be run after major dataloads, and be
scheduled weekly
•
Most database statistics are outdated and cause database performance to
perform significantly poorer than otherwise would be the case. Sampling
should be changed and process chains should be re-scheduled.
31
Many of the very large DSOs
are not partitioned, and
several objects have over
250 million records
Additionally, 101 DSO
objects are flagged as being
reportable. This results in
System IDs (SIDs) being
created during activation.
Millions
DSO Design and Locks on Large Oracle Tables
425
400
375
350
325
300
275
250
225
200
175
150
125
100
75
50
25
-
DSO Number of Records
I.e., 87 million SIDs for 0CO_ITEM_NO
and 65 million SIDs for
0DOC_NUMBER
Combined, these result in
frequent locks on the Oracle
database and failed parallel
activation jobs
FIX: Partition DSOs. The lock on very large
DSOs during parallel loads are well known and
SAP has issued several notes on the topic:
634458 'ODS object: Activation fails - DEADLOCK'
84348 'Oracle deadlocks, ORA-00060.'
32
•
•
•
•
Many InfoCubes are very large
and not partitioned
Several have over 100,000,000
records and one is approaching
0.5 billion (a stats cube where
much of the data should be
removed)
We strongly recommend that
most of these cubes are
physically partitioned, and that
a multiproviders with hints are
used instead
This can also be implemented
in SAP NetWeaver BW
Accelerator 7.2 if the relevant
cubes are indexed.
Millions
BW 7.3 InfoCube Design
425
400
375
350
325
300
275
250
225
200
175
150
125
100
75
50
25
-
InfoCubes Number of Records
InfoCubes should be performance
tuned if the number of records
exceeds 30 million and partitioned
before they are approaching 100+
million records. This creates faster
loads, better query performance,
and easier management.
33
Recommended Use of MultiProvider Hints in 7.3 (non-SPO)
Problem: To reduce data volume in each InfoCube,
data is partitioned by Time period.
2002
2003
2004
2005 2006
2007
2008
A query must now search in all InfoProviders to find
the data. This is very slow.
Solution: We can add “hints” to guide the query execution. In the
RRKMULTIPROVHINT table, you can specify one or several
characteristics for each MultiProvider, which are then used to
partition the MultiProvider into BasicCubes.
•
If a query has restrictions on this characteristic, the OLAP processor is already
checked to see which part of the cubes can return data for the query. The data
manager can then completely ignore the remaining cubes.
An entry in RRKMULTIPROVHINT only makes sense if a few
attributes of this characteristic (that is, only a few data slices)
are affected in the majority of, or the most important, queries
(SAP Notes: 911939. See also: 954889 and 1156681). Alternatively,
some of the MP joins may be included on BWA v. 7.2
34
InfoCube Design — High Cardinality Flags
•
High-Cardinality flag for large InfoCubes with more than 10
million rows
InfoCube
FIUC_C03
ZGAT_C01
FIIF_C02
FIGL_C01
•
•
•
Number of rows
12,859,780
20,793,573
68,090,967
156,738,973
Entries in dimension
compared for F table
37%
46%
102%
88%
There are currently 11 InfoCubes with a ratio of more than 20% of the records in
the dimensions vs. fact table
SAP recommends for Indexing and performance reasons to flag these as “highcardinality” dimensions. However, it has minor impact to smaller cubes.
Currently there are four medium and large InfoCubes that are not following the
basic SAP design guidelines, and subsequently exhibit slow performance
Redesign large InfoCubes with high-cardinality to take advantage
of the standard performance enhancements available.
35
SAP NetWeaver BW 7.3 System Statistics — Data Volume
Reduction
•
There are many statistical cubes with significant volume and no real benefits.
During the BW upgrade, most of these were not cleared and are now creating
poor system performance. For example:
 0TCT_C02 has 408 million rows; others also have millions of rows
 Stats are collected for over 1,900 objects, queries, infoProviders,
templates, and workbooks
 There are 7 aggregates with over 2 million rows on the stats cubes
 Creating aggregates on stats cubes inserts 1.9 million rows and takes
35.6 minutes for refresh each night
 High-cardinality flags are set for small cube with only one million rows
(0TCT_C21)
 Run-time stats are not collected for all InfoCubes and cannot be analyzed
To do: goto RSDDSTAT and select “Delete Data” for old stats and
also schedule periodic jobs using standard process chains.
36
36
Use of Line Item Dimensions for 7.3 InfoCubes not in SAP
NetWeaver BW Accelerator
•
•
•
•
Line item dimensions are basically
fields that are transaction oriented
Once flagged as a line item
dimension, the field is actually
stored in the fact table and has no
table joins
This may result in improvements
to query speeds for cubes not in
SAP NetWeaver BW Accelerator
Today, there are very few
InfoCubes that used this feature
(mostly TCT cubes)
BW 7.3 hint: Explore the use of line item dimensions for fields that are
frequently conditioned in queries. This model change can yield faster queries.37
37
Re-Design 7.3 Data Loads
•
•
ETL Statistics (average per day)
On an average daily data
Load time (sequential) - hrs
44.8
extraction, transformation, Total
- Time spent on source extraction - hrs
11.1
and load process takes
- Time spent on error filtering - hrs
0.5
22.9
44.8 hours if run sequentially - Time spent on transformations - hrs
- Time spent on target - hrs
10.4
A substantial amount of the Number of records extracted from source
371,407,881
Number of records written to target
125,102,791
time is spent on data
transformation (51%) and
• Of the 371 million records
lookups are often done on
extracted from the source, only
large DSOs without secondary
33.7% are written to disk
indexes.
• This is due to lack of ability to do
delta processing for some files
and also a substantial amount of
transform and lookup logic in
some of the ABAP rules
100%
24.7%
1.1%
51.1%
23.1%
33.7%
Consider upgrading as soon as possible to a newer ECC version that
allows for better integration of standard BW 7.x extractors. Also, revisit
extractor design for lookups on source system instead of inside BW.
38
B-Tree Indexes on Large Objects
•
•
•
•
InfoCubes that are not flagged as high-cardinality use a Bitmap
index instead of a classical b-tree index for the joins.
This type of index does not get “unbalanced” since it uses
pointers instead of “buckets”
When updating these large InfoCubes, dropping and recreating
Indexes in the process chain can be very time consuming and
actually take longer than the inserts
It can also result in locks when the objects are very large (100
million+ records) and when attempting to do this in parallel (see
ORA-0060)
Rebuilding bitmap indexes in load processing for large
objects should not be a default answer for all designs.
The process chains that do that, need to be revisited.
39
Legacy SAP BW 7.0 Performance in 7.3 - Data Activation
•
Table
/BIC
/BI0
During activation, there is limited
use of buffering of number
ranges for dimensions and
InfoObjects, even when the
number of entries are large.
InfoObject
SGOBJKEY
SCO_ITEM_NO
SDOC_NUMBER
SAC_DOC_NO
SREF_DOC_NO
SBILL_NUM
SBA_BELNR
SCO_DOC_NO
SMAT_DOC
STCTSTEPUID
Buffer range
(thousands) level (thousands)
12,792
18,699
87,423
93,099
65,184
71,713
16,814
19,693
15,321
18,663
10,389
14,704
9,433
13,421
11,468
12,282
10,427
12,040
9,951
11,793
Rows
Buffer range
(thousands) level (thousands)
FIGL_C012 138,286
163,271
FIGL_C01
FIGL_C016
47,005
53,133
FICP_C053
10,172
29,655
FICP_C05
FICP_C059
6,268
18,808
3,281
15,740
FICP_C02 FICP_C021
FICP_C013
9,149
46,308
3,804
64,388
FICP_C01 FIAR_C014
FIAP_C017
19,139
25,844
69,264
79,758
FIIF_C02 FIIF_C022
InfoCube Dimension
Rows
Objects: BID0003554; 4020; 3531; 4016; 2364; 2313; 2450; 3834 and 2309
This results in many reads on the
NRIV table that slows down data
activation and process chains
(see notes:
857998, 141497, 179224 and 504875)
Start buffering of number ranges of dimensions and
InfoObjects (see tables above) Or use 7.3 data activation instead
40
PSA Cleanup in SAP NetWeaver BW 7.3
•
For this company there were over 1.6 billion rows in the PSA
AGE of PSA
<1 MONTH
1-3 MONTHS
3-6 MONTHS
6-12 MONTH
1-2 YEARS
Total
•
•
Number of Records
591,774,063
738,204,015
144,476,834
173,651,469
35,421,883
1,683,528,264
Percent
35%
44%
9%
10%
2%
100%
Estimated
Gb
14.9
18.6
3.6
4.4
0.9
42.3
An estimated 27Gb could be freed up if data older than one month
is removed
A formal retention policy that is communicated and enforced
should be implemented ASAP
Lessons learned: Start PSA cleanup ASAP and
schedule regular jobs that take care of this in the future
41
Aggregates Are Not Needed with BW 7.3 and BWA 7.2
•
•
•
•
At this company, there are 11 aggregates in the system
Number of records in
Four are related to the cube ZIR_C01
Infocube
Aggregate
and seldom used and (two has never
0TCT_C01
895,070
been used by any query)
0TCT_C21
420,314
0TCT_C01
313,540
The 7 other aggregates are used only
0TCT_C01
188,875
by the statistical cubes.
0TCT_C21
87,771
0TCT_C22
82,051
Every day, 1.9 million records are
0TCT_C23
69,725
inserted into the aggregates and take
35.6 minutes of processing time
To do: delete unused aggregates. Also, by reducing the data
volume in the underlying statistical cubes (cleanup), the remaining
aggregates will reduce in size and processing time. If ZIR_C01
was in BWA, all four related aggregates can be deleted.
42
The Memory Cache Size Utilization
•
The OLAP Cache is by default 100 MB for local and 200 MB for
global use
•
The current system at this company was consuming no more than
80MB on average
This means that most queries were re-executing the same data
•
43
SAP NetWeaver BW 7.3 OLAP Cache — Turned Off?
•
•
•
At one client, the OLAP cache was turned off for 131 out of 690
queries (excluding 4 planning queries in BW-IP)
The cache was also turned off for 24 out of 256 InfoCubes
The OLAP Cache mode for their IP queries should have been
stored as “Binary Large Objects (BLOB),” that would speed up
caching and very large reads, as is done in these queries
Since many of the queries are using CKF and sums
and sorts extensively, the cache read mode for most
queries should be turned on ASAP at this company
44
Broadcast to Pre-Fill the Cache — MDX Based SAP
BusinessObjects Reporting
•
•
This company’s Java Stack did not
communicate properly with SAP NetWeaver
BW, and multiple logons were required
As a result, broadcasting could not be used
until the connectivity was set up correctly
To do: Set up Java connectivity ASAP and use the Broadcasting
feature to prefill the MDX cache for BI analytical processing
intensive functions such as CKF, Sorts, Exceptions, Conditions
45
45
BPC on SAP NetWeaver BW 7.3 — Current System
•
•
This BPC system was experiencing
substantial load performance issues
Some of this was due to underlying SAP
BW configuration, while some was due to
the technical configuration of the data
store architecture and data flow inside SAP
BW
Consolidation
Cube
(OC_CON)
Consolidation Processes:
1) Clearing
2) Load
3) Foreign Exchange
4) Eliminations
5) Optimizations
BPC Staging
Cube
(BPC_C01)
GL Summary
Cube
(FIGL_C03)
Production Issues include:
1) Dependent jobs not running
sequentially, i.e., load from
Summary cube to Staging cube is
sometimes executed before the
summary cube data is loaded and
activated, resulting in zero
records in the staging cube.
2) Long latency with 6 layers of
PSA, DSOs, and InfoCubes
before consolidation processes
can be executed.
Conformed
Reportable
DSO
Write
Optimized
DSO
FIGL_D21
FIGL_D15S
FIGL_D20
FIGL_D17
FIGL_D14
FIGL_D18
FIGL_D13S
FIGL_D10S
FIGL_D08
FIGL_D11S
Persistent Staging Area (PSA)
ECC 6.0
AsiaPacific
ECC 6.0
NorthAmerica
ECC 4.7
LatinAmerica
R/3 3.1i
EU will
ECC 4.7
ASIA Shut
move to ECC6.0
down in 2011
46
BPC on SAP NetWeaver BW 7.3 — Possible Future System
•
•
Long-term the BPC system on SAP NetWeaver BW 7.3 should be
simplified. This includes removing the conformed DSO layer, with
BEx flags for data stores that are never reported on.
Also, the BPC staging cube serves
little practical purpose since the data is
Consolidation Processes:
1) Clearing
already staged in the GL Summary cube
2) Load
3) Foreign Exchange
and the logic can be maintained in the
4) Eliminations
5) Optimizations
load from this cube directly to the
consolidation cube.
Consolidation
Cube
(OC_CON)
GL Summary
Cube
(FIGL_C03)
Long-term benefits include
reduced data latency, faster
data activation, less data
replication, smaller system
backups as well as simplified
system maintenance. Take
advantage of the 7.3 features!
Write
Optimized
DSO
FIGL_D15S
FIGL_D13S
FIGL_D10S
FIGL_D08
FIGL_D11S
Persistent Staging Area (PSA)
ECC 6.0
AsiaPacific
ECC 6.0
NorthAmerica
ECC 4.7
LatinAmerica
R/3 3.1i
EU will
ECC 4.7
ASIA Shut
move to ECC6.0
down in 2011
47
What We’ll Cover …
•
•
•
•
•
•
•
•
New Administrative Cockpit Capabilities
The New Semantic Partition (SPO) Options
The HybridProvider and Real-Time Data
Faster Data Load and New Design Options
Front-End and Security Benefits
Using SAP NetWeaver BW 7.3 for Performance - Real Examples
New In-Memory Possibilities with SAP NetWeaver BW 7.3 and SAP
NetWeaver BW Accelerator 7.2
Wrap-up
48
In-Memory Processing — SAP NetWeaver BW Accelerator
7.2 and SAP NetWeaver BW 7.3
•
•
BIG DEAL! - In SAP NetWeaver BW
7.3, you can also build analytical
indexes
 These are simply “data
containers” from APD that is
stored in SAP NetWeaver BW
Accelerator
 You can build queries on them
Using the Index Designer in
SAP BusinessObjects Data
Services, you can also index data
from external data files and data
bases and query it in SAP
NetWeaver BW 7.3
It is important to note that
currently, the SAP
BusinessObjects indexes are
on dedicated hardware
blades.
3. Queries are routed to BWA
by the Analytical engine
SAP BW
BI Analytical Engine
2. Indexes copied in
to RAM on blades
InfoCubes
1. Indexing and
compression
stored on a file
system
DSOs
Business Object
Data Services
External data
Data
Integrator
Index Designer
Files
Databases
49
CompositeProviders: Unions of Indexes in SAP NetWeaver
BW Accelerator
•
•
With 7.3 you get a tool to create a new “CompositeProvider” to join indices
inside the SAP NetWeaver BW Accelerator system
This works similar to MultiProviders, but can be modeled ad-hoc and the joins
are executed inside SAP NetWeaver BW Accelerator and not inside the BW
Analytical Engine (this is much faster)
Source: SAP AG, 2011
All BW 7.3 CompositeProviders, DSOs, MultiProviders & InfoCubes are available
as InfoProviders for querying. Even queries can be defined as InfoProviders.50
50
The New SAP NetWeaver BW 7.3 Features — Modeling in
SAP NetWeaver BW Accelerator 7.2
With SAP NetWeaver
BW 7.3 and SAP
NetWeaver BW
Accelerator 7.2, you can
complete many of the
joins and the aggregation
that previously were
conducted in the BI
Analytical engine, and
was therefore hard to
performance tune
It also reduces the data
being transferred to the
app server and therefore
is much faster as well
51
The New SAP NetWeaver BW Accelerator 7.2/SAP
NetWeaver BW 7.3 — Faster Performance
•
•
The major benefit of SAP NetWeaver BW Accelerator 7.2 on SAP NetWeaver
BW 7.3 is that you can move more of the calculation over to the accelerator and
materialize the result in-memory
This improves the reads, and calculations in the OLAP/BI analytical engine on
the SAP NetWeaver BW side are substantially reduced
Source: SAP AG, 2011
The following calculations is included in BW 7.3/BWA 7.2
SUM
MIN
MAX
CNT (count all detailed values)
CN0 (Count all values not zero or null) AVG (average)
AV0 (Average for all values not zero or null)
52
The SAP NetWeaver BW 7.3/SAP NetWeaver BW Accelerator
7.2 Limitations and Query Settings
•
There are still some limitations. For example, you cannot use the
exception aggregation for single key figures in SAP NetWeaver
BW Accelerator if it uses:
 Virtual key figures
 Conversion before aggregation
 Formula calculation before aggregation
 Non-cumulative key figures
 Key figures with elimination of internal business volume
You also get more detailed control on
each query as to how it reads SAP
NetWeaver BW Accelerator. Think of it
as similar to Query read modes.
(more details at SDN :
http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/7095000
3-f7ef-2d10-b1bc-ee483800b25c?QuickLink=index&overridelayout=true
With BW 7.3 SP-5 is able to run completely on the HANA database (in-memory
processing database instead of traditional RDBMSs)
53
Looking Inside SAP HANA — In-Memory Computing Engine
(IMCE)
Metadata
Authorization
Transaction
Manager
Manager
Manager
Relational
Engine
SQL Script
SQL Parser
-Row Store
-Column Store
Calculation
Disk Storage
Data
Volumes
Log
Session
Manager
MDX
Engine
Volumes
Load
Controller
Replication Server
BusinessObjects Data Services
Inside the Computing Engine of SAP HANA, we have many different components
that manage the access and storage of the data. This includes MDX and SQL
access, as well as Load Controller (LC) and the Replication Server.
54
SAP HANA — Loading the Application and Performance
•
You can load the application based on the logs in the source
system, ETL-based (Extract Transform and Load) loads, and SAP
trigger-based replication
Tool
Purpose
BusinessObjects Data Services 4.0
Sybase replication server & Load Controller
SAP Landscape Transformation (LT)
ETL-based replication
Log-based replication
Trigger-based replication
Some Reported SAP HANA Performance Achivements
Compression
Query speed improvements
Project
(data)
Max
Min
Average
Bank
1 : 6.3
521.6
258.8
369.5
HANA POC (SAP)
1 : 5.2
484.3
301.4
350.3
Note: a large Oil and Gas company reported
compression of 3.1 times from an Oracle 11g database
55
SAP HANA — Virtual Marts and Applications
•
Virtual data marts and new applications were built that run on SAP
NetWeaver BW, which is again enabled by SAP HANA in-memory
processing
HANA (in-the works)
ERP
Database
Virtual
Data
Marts
Virtual
Data
Marts
Virtual
Data
Marts
Virtual
Data
Marts
Applications
Databases
Files
Applications developed by SAP
1. Profitability analysis
2. Dynamic cash management
3. Strategic workforce planning
4. Smart meter analytics (power companies)
5. Planning & consolidation
6. Customer revenue performance mgmt
7. Predictive segmentation & targeting
8. Trade promotion management
9. Merchandise & assortment planning
10.Sales & operations planning (SOP)
11.Demand signal repository
This provides much tighter integration with the source system, (less data
latency) and much faster query response time for high-volume analysis
56
What We’ll Cover …
•
•
•
•
•
•
•
•
New Administrative Cockpit Capabilities
The New Semantic Partition (SPO) Options
The HybridProvider and Real-Time Data
Faster Data Load and New Design Options
Front-End and Security Benefits
Using SAP NetWeaver BW 7.3 for Performance - Real Examples
New In-Memory Possibilities with SAP NetWeaver BW 7.3 and SAP
NetWeaver BW Accelerator 7.2 and HANA
Wrap-up
57
Additional Resources
•
•
•
•
•
SAP NetWeaver 7.3 on SAP Developer Network
 www.sdn.sap.com/irj/sdn/nw-73
LSA Templates and Architecture in SAP NetWeaver BW 7.3
 www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/24800
Features list for SAP NetWeaver BW 7.3
 www.sdn.sap.com/irj/sdn/edw?rid=/library/uuid/304444f7-e02d2d10-9c97-d5e3ecf09882
SAP NetWeaver 7.3 Business Content
 http://help.sap.com/saphelp_nw73/helpdata/en/ca/6fbd35746db
d2de10000009b38f889/frameset.htm
Roadmap – SAP NetWeaver BW 7.3
 www.sdn.sap.com/irj/sdn/bw-73?rid=/library/uuid/300347b59bcf-2d10-efa9-8cc8d89ee72c
58
7 Key Points to Take Home
•
•
•
•
•
•
•
SAP NetWeaver BW 7.3 is a stable version you can use now
The upgrade is somewhat technical, and strong skills are required
When you start, plan for how much of the new functionality you
will deploy. Include redesign and training as well.
You will need to spend time on security conversion if you have
not already done so
The new features are unique, so don't rely on past experiences
Complete a technical review before you start leveraging 7.3
Make sure you have access to experts and decide early how much
risk you can live with before starting “open heart surgery”
59
Your Turn!
How to contact me:
Dr. Bjarne Berg
Bberg@Comerit.com
Continue the conversation! Post your
questions in the BI-BW Forum on
Insider Learning Network*
*bit.ly/BI-BWForum
60
Disclaimer
SAP, R/3, mySAP, mySAP.com, SAP NetWeaver®, Duet™®, PartnerEdge, and other SAP products and services mentioned herein as well as their
respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product
and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by
SAP.
61