Using SQL Server 2014 In-Memory Optimized

Columnstore with SAP BW

Applies to:
SAP Business Warehouse 7.0 and higher running on Microsoft SQL Server 2014 and higher

Summary
SQL Server 2014 In-Memory Optimized Columnstore dramatically decreases disk space usage and can
improve BW query performance by factors. You can use the columnstore index for SAP BW f-fact and efact tables, for SAP BW basic cubes, and aggregates. SQL Server 2014 columnstore support has been
integrated into SAP BW and was down-ported to all common SAP BW releases:

SAP BW 7.0
SAP BW 7.01
SAP BW 7.02
SAP BW 7.11
SAP BW 7.3
SAP BW 7.31
SAP BW 7.4

(SAP NetWeaver 7.0)

(SAP enhancement package 1 for SAP NetWeaver 7.0)
(SAP enhancement package 2 for SAP NetWeaver 7.0)
(SAP enhancement package 1 for SAP NetWeaver 7.1)
(SAP NetWeaver 7.3)
(SAP enhancement package 1 for SAP NetWeaver 7.3)
(SAP NetWeaver 7.4)

Author(s):

Martin Merdes

Company:

Microsoft

Created on:

February 2015

Version:

1.01

Author Bio
Martin Merdes is a Senior Software Engineer at Microsoft. Since 1996 he has been working in different
technical functions for Microsoft at SAP Germany: Support Engineer, Senior Consultant, Technical
Account Manager, Premier Field Engineer and Software Development Engineer. Since 2009 he is
responsible for the porting of SAP BW on Microsoft SQL Server.

Table of Contents
Overview....................................................................................................................................................... 3
Benefits of Using the Columnstore ............................................................................................................. 3
Space Usage ............................................................................................................................................................... 3
Query Performance...................................................................................................................................................... 3
Faster BW Processes .................................................................................................................................................. 4

Restrictions ............................................................................................................................................... 4
Prerequisites ................................................................................................................................................. 5
SQL Server ............................................................................................................................................... 5
SAP BW .................................................................................................................................................... 5
Applying the Columnstore ............................................................................................................................. 6
Step 1: Configure Cube ............................................................................................................................. 6
Step 2: Convert Cube ................................................................................................................................ 7
Running MSSCSTORE as a Batch Job ....................................................................................................................... 8
Aggregates................................................................................................................................................................... 8
Checking Cube Status ................................................................................................................................................. 9

Loading Data .............................................................................................................................................. 10

Manually Running a DTP ......................................................................................................................... 10
Running DTPs in BW Process Chains ..................................................................................................... 10
BW Cube Compression ........................................................................................................................... 11
Inventory Cubes ......................................................................................................................................................... 11
Aggregates................................................................................................................................................................. 12
Basic Cubes ............................................................................................................................................................... 12

Best Practices ............................................................................................................................................. 14

Copyright .................................................................................................................................................... 16

Overview
SQL Server 2012 introduced an additional storage format for data. Besides the normal row-oriented
store, there is a column-oriented store as well. To make it as transparent as possible for database
administrators and applications, the new columnstore was implemented as a new index type:
nonclustered columnstore index. Therefore, a single table can use both stores side-by-side.
The nonclustered columnstore index has one major drawback: It is a read-only index. Therefore an
additional columnstore index type was introduced in SQL Server 2014: the writeable clustered
columnstore index. In SQL Server 2014 you can use all index types: rowstore, clustered and
nonclustered columnstore indexes.
The columnstore is often called In-Memory Optimized DW in Microsoft Knowledge Base Articles and
BLOGs. Here you often see the acronym NCCI for the (read-only) Nonclustered Columnstore Index and
CCI for the (writeable) Clustered Columnstore Index.
Benefits of Using the Columnstore
Space Usage
Indexes can be compressed much better when stored by columns rather than by rows. Furthermore, the
number of indexes for the fact tables is reduced when using the columnstore. Therefore, we see
consistent space savings when using the columnstore in SAP BW. The space savings are dependent
from the number of BW dimensions, number of key figures, and the data distribution.
The following example shows the total disk space needed for a 100,000,000 row cube having 11
dimensions and 8 key figures. The total space contains the disk space for the f-fact, e-fact and dimension
tables. Note that the dimension tables still use rowstore indexes in SAP BW on SQL Server 2014.
40000 MB
35000 MB
30000 MB
25000 MB
20000 MB
15000 MB
10000 MB
5000 MB
0 MB
SQL 2005

SQL 2005
Series1

38179 MB

SQL 2008
PAGE
compression

SQL 2008 PAGE

compression
18354 MB

SQL 2012 NCCI

SQL 2014 CCI

SQL 2012 NCCI

SQL 2014 CCI

8501 MB

4340 MB

Query Performance
The column-oriented store is optimized for aggregating mass data, while the row-oriented store can be
used optimally when accessing just a single row. Query performance is typically by factors better when
using the columnstore compared with the rowstore. However, the performance improvements depend on
the individual query. BW aggregates also benefit from using the columnstore.
The columnstore is optimized for mass aggregation. Therefore, BW cubes and aggregates having only a
few thousand rows hardly benefit from the columnstore. On the other hand, query performance is
typically not an issue anyway for such small cubes.

In SAP NetWeaver (and BW) systems you typically configure the intra-query parallelism to 1 by setting
the SQL Server configuration option max degree of parallelism. In SAP BW you can overwrite this
global setting for SAP BW queries. You simply have to set the SAP RSADMIN parameter
MSS_MAXDOP_QUERY in SAP report SAP_RSADMIN_MAINTAIN. This results in adding a SQL
Server optimizer hint MAXDOP for all BW queries. The default value of MSS_MAXDOP_QUERY is 2 for
SQL Server 2012 and 4 for SQL Server 2014. We do not recommend changing the default value on a
productive server with 16 CPU threads or less.
For powerful DB servers you may increase the value of MSS_MAXDOP_QUERY, but you should never
set it to MAXDOP=1. We typically do not see a benefit for the columnstore when increasing
MSS_MAXDOP_QUERY above 8. Increasing the parallelism too much is not a good idea. This might
result in a resource bottleneck during high workload. In this case, SQL Server might be forced using
actually only MAXDOP =1 for some concurrently running queries. Furthermore, some columnstore
optimizations only work with MAXDOP >1 in SQL Server 2012. However, with SQL Server 2014 these
optimizations also work with MAXDOP =1.
The overall runtime of an SAP BW query consists of the time consumed on the database server, the SAP
BW application server and the client. The columnstore is an optimization on the database server.
Therefore the columnstore has no impact on the runtime consumed on the SAP BW application server,
which can be a significant part of the overall BW query runtime
An SAP BW query may run more than one SQL query. This is always the case when using BW
Semantically Partitioned Cubes or BW Multi-Provider. Both are logical cubes, which consist of
several physical cubes. When running a BW query against the logical cube, separate SQL queries run
against each physical cube. Afterwards, the result sets of all queries have to be aggregated on the SAP
BW application server. However, the SAP BW application server does not use parallelism. Each SAP
work process is executed single-threaded. Furthermore, aggregation on the application server requires
much more rows, which have to be transferred from the database to the application server.
Faster BW Processes
Many BW processes like DTPs, BW cube compression and DB index creation are much faster when
using the writeable columnstore. One reason is the reduced number of indexes you have to maintain
during data load. Re-creating a columnstore index (or converting a cube to columnstore) can use SQL
Server parallelism much better than rowstore indexes.
SAP BW uses a separate RSADMIN parameter to configure parallelism for index creation:
MSS_MAXDOP_INDEXING. The default value is 8. Index creation scales quite well even with a higher
degree of parallelism. Therefore, you can increase MSS_MAXDOP_INDEXING as long as sufficient CPU
and memory resources are available.
Restrictions
SAP BW does not allow the creation of a writeable columnstore index on the f-fact table of a Real-Time
cube. A Real-Time cube does not close a data load request before running queries against the cube.
Therefore loaded data may still be in the SQL Server Delta Store, which then results in even worse query
performance (compared with the rowstore).
In SQL Server 2014 you cannot create an additional b-tree index, unique index or primary key constraint
when already having a writeable columnstore index. Therefore, the writeable columnstore is restricted to
the f-fact and e-fact tables of SAP BW for SQL Server 2014. Additional support for other SAP BW tables,
like master data tables, dimension tables and DSO tables is planned for future versions of Microsoft SQL
Server.

Prerequisites
The writeable columnstore index of SQL Server 2014 cannot be used out of the box with SAP BW. It
requires additional SAP BW code. SAP Note 2114876 always contains the up-to-date prerequisites.

As at December 2014, you have to install the following software components:

SQL Server
The minimum required SQL Server version is Cumulative Update 5 (CU5) of SQL Server 2014. However,
we recommend installing the newest service pack, once available. You can download the latest SQL
Server patches from http://technet.microsoft.com/en-us/sqlserver/ff803383.aspx.
SAP BW
To use the columnstore on SQL Server 2014, you have to apply the following SAP Support Package
Stacks (SPS). Afterwards you should apply SAP Note 1949486, which will contain necessary bug fixes in
the future.

SAP BW

NW SPS

BW SP

7.0

32 *)

34

7.0 EHP1

17

17

7.0 EHP2

17

17

7.1 EHP1

14

14

7.3

13

13

7.3 EHP1

15

15

7.4

10

10

* NW SPS 32 includes SAP Basis SP 32 and SAP BW SP 34

Alternatively, you may apply an older SAP SPS and additional correction instructions. For more
information, see SAP Note 2114876.
The required SAP BW code for using SQL Server 2014 writeable columnstore also contains many
improvements for SQL Server 2012 read-only columnstore. Therefore, we recommend applying the new
code also for SQL Server 2012. When discussing features of SQL Server 2012 below, we assume that
the newest SAP BW code is already applied.

Applying the Columnstore

Applying the columnstore is a two-step process. In the first step you configure the BW cube for using the
columnstore in the SAP report MSSCSTORE. This step only changes the property of the cube in the
SAP BW dictionary. In the second step you convert the cube to fit the columnstore configuration. In this
step the indexes are created on the database. You can perform the second step either in report
MSSCSTORE or run an index repair in SAP transaction RSA1 or a BW process chain (see section Data
Load below).
Step 1: Configure Cube
To configure a single cube, start report MSSCSTORE, enter the name of the cube, choose the desired
cube type, make sure that the checkbox change cube definition is set, and start the report by pressing
Execute (F8).

Configure Cube Type for a Single Cube

You can also configure all BW cubes at the same time. For this you have to set a global cube type in the
tab Process all cubes and start the report by pressing execute (F8). Note that this is not simply a default
for new cubes (as it used to be in older versions of report MSSCSTORE). By setting a global cube type,
the cube type of all existing cubes is changed and you will not be able any more to set the cube type
individually for a single cube. Once you turn the global cube type off, all existing cubes keep their cube
type. Afterwards you can change the cube type individually in tab Process single cube.

Configure Global Cube Type for all Cubes

Step 2: Convert Cube

You can convert a cube from rowstore to columnstore (and vice versa) by repairing the database indexes
of this cube in SAP BW. This is done automatically the next time you run a BW process chain, which
contains the process chain step Index Repair for this particular cube. However, the easiest way to repair
a cube is using SAP report MSSCSTORE. Here you can either configure and repair the cube in separate
steps or perform both steps together.

Configure and Repair a Cube at the Same Time

The columnstore configuration is stored in the SAP BW dictionary independently of the installed SQL
Server release. However, when repairing a cube, the SQL Server release is taken into account. The
following table shows the index types on the database for e-fact and f-fact table of a cube after it has
been repaired:

SQL Server 2008

and lower
SQL Server 2012
SQL Server 2014
and higher

f-fact table
e-fact table
f-fact table
e-fact table
f-fact table
e-fact table

Row-Store (no CS)

Basic CS

Column-Store (CS)

b-tree
b-tree
b-tree
b-tree
b-tree
b-tree

b-tree
b-tree
b-tree
read-only CS
b-tree
read-only CS

b-tree
b-tree
b-tree
read-only CS
writeable CS
writeable CS

Indexes on DB Dependent on Cube Configuration and SQL Server Release

There are some exceptions to the above rules: The f-fact table of a real-time cube is always using b-tree
indexes. A cube on SQL Server 2012, which has a high precision key figure, and all APO cubes always
use b-tree indexes.
All b-tree indexes of the fact-tables are dropped on the database when converting a BW cube from
rowstore to writeable columnstore (on SQL Server 2014). However, these indexes might still be visible in
the SAP data dictionary for the fact tables (with the flag not for Microsoft SQL Server). Therefore always
check the index state on the database, not in the SAP data dictionary.

Running MSSCSTORE as a Batch Job

Once you have chosen your options, you can run report MSSCSTORE interactively by pressing the F8
button. By pressing F9 you can schedule a batch job instead. This is recommended when repairing the
indexes or when processing all cubes rather than a single cube. You can check the job logs in SAP
transaction SM37 or by pressing the SAP function key SM37 (Ctrl+F1). Report MSSCSTORE also
writes a detailed log into the SAP application log. This is the case when running in batch and when
running interactively. To check the application log, press the SAP function key Logs (Ctrl+F2).

Running MSSCSTORE in Batch

Aggregates
MSSCSTORE always configures the aggregates of a cube the same way as the cube itself. When
repairing a cube all aggregates are also repaired. When configuring or repairing a partition of a
semantically partitioned cube, all partitions are processed at the same time. In many cases BW query
performance is fast enough even without aggregates, once you have applied the columnstore. Therefore,
we implemented the option to deactivate aggregates in report MSSCSTORE. However, by default we
recommend keeping the aggregates. Once you have deactivated them, you can Activate and Fill them in
report MSSCSTORE again. Activating simply creates the database tables. You cannot use aggregates in
BW queries without filling them. Simply activating the aggregates makes sense, if you already have a BW
process chain for filling the aggregates.

Activating or Deactivating all Aggregates of a Cube

Checking Cube Status

You can check the status of a single cube by pressing the SAP function key Status of cube (Ctrl+F5) in
report MSSCSTORE. This performs a full index check of all tables of the cube and its aggregates. In
column DDIC you can see the BW cube configuration, which is always the same for all tables. In
column DB Expected you can see the expected indexes on the database according to the cube
configuration and the SQL Server release. Column DB Check shows whether all indexes are correctly
created on the database. You can also see this in column Exception as an icon. If it is not green, you
should run an index repair as described above.

Status of a Single Cube

You can get an overview of all cubes when using the F4 help in the cube input field (Configure single
cube:). In the column Exception you can easily see all cubes for which the database indexes do not
match their cube definition. In the example below, cube 0CCMDAHGR has a red status, because the
cube is defined as columnstore while the indexes on the database are still using b-tree indexes. For
performance reasons, the F4 help does not perform a full index check. It only checks the columnstore
index. Therefore, a single missing b-tree index of a rowstore cube is not detected here. The status is then
still green.

F4 help: Status of all Cubes

Loading Data
SAP BW uses Data Transfer Processes (DTPs) for loading data into a cube. If the cube is using a
writeable columnstore index, the loaded data is not fully in columnstore format after the DTP has
finished. Some rows might still be in the Delta Store. The procedure of moving all rows from the Delta
Store to the columnstore is called rowgroup compression. SAP has integrated the rowgroup
compression into the BW Index Repair (also known as Create Index or Repair Index) function for
cubes. For best query performance you have to run the BW Index Repair after each DTP.
SAP recommends running a database Update Statistics (also known as Refresh Statistics) after
each DTP. This recommendation is not only related to writeable columnstore indexes. It is also valid for
read-only columnstore or rowstore indexes. However, Update Statistics is not necessary when running a
BW cube compression after the DTP. The BW cube compression automatically performs an Update
Statistics.
Manually Running a DTP
Typically you are using SAP BW process chains for running DTPs. However, you might start a single
DTP in SAP transaction RSA1. In this case you should also run an Index Repair and an Update Statistics
once the DTP has finished.

Transaction RSA1 Modeling InfoProvider Manage Performance

Running DTPs in BW Process Chains

You should always include the process chain steps Index Repair and Update Statistics in all your SAP
BW data load process chains. The process chain step Index Delete is only used for rowstore indexes (of
the BW f-fact table and the BW dimension tables). Therefore, it only has an impact on cubes using the
rowstore or the read-only columnstore (for e-fact table). Index Delete has no impact on cubes using the
writeable columnstore. Therefore, you do not need the process chain step Index Delete once you convert
the cube to writeable columnstore.

10

Index Repair (Create Index) and Update Statistics (DB Statistics) within a BW Process Chain

BW Cube Compression
BW cube compression (also known as Condense or Collapse) has several benefits. It reduces
the total number of rows in a BW cube by aggregating the request ID. This also results in reduced space
consumption on the database. Cube compression reduces the number of database partitions in the f-fact
table. For inventory cubes it creates reference points, which results in much faster inventory queries. For
cubes using the read-only columnstore index, cube compression is mandatory. Without cube
compression all data is stored in the f-fact table, which cannot use the read-only columnstore index at all.
However, cube compression also has some drawbacks. It is an additional step during data load, which
takes some time. BW queries cannot run during BW cube compression of the cube or aggregate used by
the query. Once a faulty loaded request has been compressed, it cannot be easily deleted any more.
For Microsoft SQL Server SAP recommends regularly compressing BW cubes. This could be part of the
daily DTP process chain. If the daily process chain runs concurrent with SAP BW queries, you can only
include the BW cube compression in those process chains, which are running at night or weekend.
Dependent on the cube type, we recommend the following:
Inventory Cubes
Compress most of the requests of an inventory cube, ideally all of them. This may significantly increase
query performance on inventory cubes. In SAP BW process chains you can specify that all requests are
compressed except a given number of requests or all requests loaded within the last few days.

11

Property of BW Process Chain Step Collapse

Aggregates
Always fully compress all aggregates. This is done automatically during aggregate load or rollup, once
you have defined the relevant cube property. A typical scenario is keeping all requests loaded within the
last few days or weeks uncompressed in the basic cube, while compressing all requests in its
aggregates. In this case you can easily delete recent faulty loaded requests. Recent requests are not
compressed in the basic cube. During request deletion, SAP BW automatically rebuilds (the fully
compressed) aggregates.

Setting Cube Property: Always Fully Compress all Aggregates

Basic Cubes
Each uncompressed BW request uses its own database partition in the f-fact table. Although you can
have up to 15,000 partitions, we recommend not having more than 1,000 partitions. Even a few hundred
partitions might result in reduced BW query performance. The only way to get rid of these partitions is to
compress the BW requests. Therefore, we strongly recommend keeping the number of uncompressed
requests low. This is independent from the cube type and columnstore type.

12

For cubes using the rowstore you might still have a few hundred uncompressed requests. Cube
compression takes quite a long time for rowstore cubes. Furthermore, cube compression might not result
in the expected space savings: For rowstore cubes, the e-fact table has no clustered index at all. This
results in less space efficiency compared with tables having a clustered index (like the f-fact table).
Inventory cubes store additional rows in the e-fact table, the reference points. This results in faster BW
queries on BW compressed inventory cubes. On the other hand it increases the size of the e-fact table.
For all these reasons many customers perform BW cube compression seldom (or not at all) when they
are using rowstore cubes. However, in any case you should make sure that the number of partitions in
the f-fact table does not exceed about 1,000.
For cubes using the read-only columnstore, you have to compress as many requests as possible. BW
cube compression moves data from the f-fact table (having rowstore indexes) to the e-fact table (having
read-only columnstore index). Without BW cube compression you simply cannot benefit from the readonly columnstore.
For cubes using the writeable columnstore, you can also use the columnstore on the f-fact table.
However, you still should not have more than a few hundred uncompressed requests. Cube compression
on a writeable columnstore cube is much faster compared with cube compression on a rowstore cube.
When using cube compression, you do not need to include the process chain step Update Statistics in
your DTP process chains (because update statistics is performed during BW cube compression). A
typical process chain looks like this:

Typical DTP Process Chain for Cubes Using Writeable Columnstore

13

Best Practices
For using the columnstore with SAP BW we recommend the following:

Keep the database index statistics up-to-date. You can achieve this by adding one of the
following process chain types at the end of your SAP BW cube load (DTP) process chain:
Compression of the InfoCube or Construct Database Statistics. (You can manually update the
database index statistics of a cube using SAP transaction RSA1 > Modeling > InfoProvider >
Manage Cube > Performance > Refresh Statistics)

Ensure that the database options Auto Create Statistics and Auto Update Statistics are
turned on. For SAP BW we typically turn off the database option Auto Create Statistics Async.
Keep in mind that all index statistics are dropped once you convert a table from rowstore to
columnstore. They are automatically created once the first queries are running (because of the
database option Auto Create Statistics). Therefore the queries may take longer than expected,
when running immediate after the conversion to columnstore.

Make sure that all columnstore rowgroups are compressed after each data load. Therefore,
you have to add the process chain type Generate Index after a DTP in a process chain. (You
can manually compress all columnstore rowgroups of a cube using SAP transaction RSA1 >
Modeling > InfoProvider > Manage Cube > Performance > Repair DB Indexes)

Run BW Cube Compression regularly. You can keep those requests uncompressed, which
were loaded within the last few weeks. However, for best query performance, the majority of all
requests in a cube should be compressed. For inventory cubes, ideally all requests should be
compressed.

BW Partitioning of the e-fact table by time dimension can further increase query performance.
However, these performance improvements are often overrated.

BW Aggregates can increase query performance to an optimum. However, the administrative

overhead of aggregates can be significant. Therefore keep the number of aggregates low.
Configure all cubes to always use fully compressed aggregates (option Compress After Rollup)
Use BW Exceptional Aggregations only when there is a business need. Exceptional
aggregations are typically performed on the BW application server (which is single-threaded).
This might consume a significant proportion of the whole BW query time and cannot be
accelerated by any database technology like the Columnstore.

You can further decrease the needed disk space for the columnstore by changing the data
compression type COLUMNSTORE_ARCHIVE using report MSSCOMPRESS. This results in
additional disk space savings of roughly 20%. However, this also might result in slower query
performance of 20%. Therefore we do not recommend using COLUMNSTORE_ARCHIVE for
SAP BW.
Initially keep the default values of the RSADMIN parameter MSS_MAXDOP_QUERY (2 for SQL
Server 2012, 4 for SQL Server 2014). Increasing this value can result in a memory bottleneck,
which is counterproductive. On database servers having plenty of CPU threads and RAM you
may increase MSS_MAXDOP_QUERY to 8.

You can speed-up Data Transfer Processes (DTPs) into a cube by setting the RSADMIN
parameter RSDD_ENABLE_MASS_CUBE_WRITE=X (for SAP BW 7.30 and higher).

Make sure that the recommended trace flags for SQL Server are set, in particular trace flag 9481
for SQL Server 2014.

Configure sufficient memory for SQL Server and make sure that Locked Pages In Memory is
used for SQL Server
Create many tempdb data files (at least 16) and store them on a fast disk (SSD). A single
tempdb log file is sufficient.

For SAP BW, use the SQL Server recovery model BULK LOGGED whenever possible (as long
as your backup strategy allows it and you are not using database mirroring). Converting the
different cube types and any other index operation can be significant faster when using BULK
LOGGED.

14

Related Content
Columnstore Indexes Described: http://msdn.microsoft.com/en-us/library/gg492088.aspx
Using SQL Server 2012 Column-Store with SAP BW: http://scn.sap.com/docs/DOC-33129
SAP Note 1771177 SQL Server 2012 column-store support for SAP BW
SAP Note 1951490 SQL Server Column-Store for SAP BW Aggregates
SAP Note 1949486 SQL Server 2014 column-store support for SAP BW
Update Center for Microsoft SQL Server: http://technet.microsoft.com/en-us/sqlserver/ff803383.aspx
SAP BW issues in blog "SAP On SQL Server": http://blogs.msdn.com/b/saponsqlserver/archive/tags/bw/
SAP Community Network "SAP On SQL Server": http://scn.sap.com/community/sqlserver

15

Copyright
2015 SAP SE SE or an SAP SE affiliate company. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the
express permission of SAP SE.
The information contained herein may be changed without prior notice.
Some software products marketed by SAP SE and its distributors contain proprietary software
components of other software vendors. National product specifications may vary.
These materials are provided by SAP SE and its affiliated companies (SAP SE Group) for informational
purposes only, without representation or warranty of any kind, and SAP SE Group shall not be liable for
errors or omissions with respect to the materials. The only warranties for SAP SE Group products and
services are those that are set forth in the express warranty statements accompanying such products
and services, if any. Nothing herein should be construed as constituting an additional warranty.
SAP SE and other SAP SE products and services mentioned herein as well as their respective logos are
trademarks or registered trademarks of SAP SE in Germany and other countries.
Please see
http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark
for additional trademark information and notices.

16