SAP Business Planning and Consolidation Data Manager: Now and the Future
Murali Chandrasekaran, EPM RIG (Americas) Pravin Datar, EPM RIG (Americas) Dennis Schwerer , EPM RIG (EMEA)
Disclaimer
This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or noninfringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
BPC Versions
Previously
BPC 5.x
NetWeaver
MS Version
Today
BPC 7
Migration Utility
BPC 7 M
Future
BPC 7.5
BPC 7.5 M
Microsoft based
SAP NW based
Data Manager
Data Manager is an SAP Business Planning and Consolidation (BPC) module that helps you manage loading, transforming, and manipulating data The focus of this session is to discuss Data Manager capabilities in general, however, when examples are given, the details will be described for the NetWeaver version of the product As opposed to using typical ETL tools such as Microsoft SQL Server Integration Services (SSIS), or NetWeaver BI ETL (Process Chains, InfoPackages, DTPs, etc), the Data Manager provides basic ETL operations to an business user (end-user).
This means that loading a flat file isnt an Administrator or IT function. The premise is such that the data is owned by the business.
The BPC User has no idea that this underlying technology is being used when they are using the Data Manager
Data Manager includes a delivered with a set of packages to import, copy, clear and move data within and across applications Data means transactional data as well as master data (no meta data)
In addition, Data Manager supports mapping and complex transformations of transactional data files as well as master data files*
Data Manager
Within the two platforms, there are two different underlying base technologies employed
BPC 7.0 (M), uses SQL Server Integration Services (SSIS) as the base technology
BPC 7.0 (NW), uses the SAP NetWeaver BI Process Chain Framework as the base technology
Data Manager
Package Task / Process Type
DM
Process Chain
PC
PC is a framework, which is provided by BW, that helps to schedule packages, maintain status of running packages, wrap tasks into packages and so on. PC is the foundation of a package. An SSIS package is roughly equivalent to a NW BI Process Chain. The Multi Dimensional Data Cube in BW. (ROLAP) Data Transfer Service or SQL Server Integration Services, provided by Microsoft. These are no longer relevant for BPC 7.0. Is a BI InfoObject that contains dimension members (Master Data) Is a BI Object that stores the value (Key Figure) This is an Excel Workbook that defines mappings for data transformations This is an Excel Workbook that defines conversions within data transformations.
InfoCube DTS (SQL 2000) SSIS (SQL 2005) Dimension Measure Transformation File Conversion File
DTS or SSIS
From
the Launch Page, click SAP Business Planning and Consolidation for Excel. From the Action Pane, click Manage Data.
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
Data
Manager packages are implemented allow you to do common data manipulation activities The packages that come with BPC 7.0 are designed to be dynamic so that you do not need to modify the packages in order for them to work with your applications and dimensions.
You might want to modify packages to behave differently or to perform additional tasks. You can make modifications to the packages supplied with BPC (not best practice) or use those packages as templates to build packages as per your requirements
A typical activity a user would do would be to load a flat file into a BPC application (cube) To do so, the user would first upload data to the BPC File Service as described below
Within the BPC 7.0NW product, files are stored within the NetWeaver database, whereas within the BPC 7.0M product, files are stored on the .NET File Server (file system) NOTE: The following web admin parameters affect the behavior of file upload
ALLOWEXTENSIONS: Defines allowed file extensions for upload ALLOW_FILE_SIZE: This is the maximum file size a user is allowed to upload. The default value is 100 MB. DEFAULT_EXTENSIONS: This is the list of default extensions that are allowed. This list includes: XLS,XLT,DOC,DOT,PPT,POT,XML,MHT,MHTML,HTM,HTML,XLSX,XLSM,XLSB,ZIP,PDF,PPTX,PPT M,POTX,POTM,DOCX,DOCM,DOTX,DOTM,CDM,TDM,PNG,GIF,JPG,CSS,MRC
In some cases, a user may want to download a file from the BPC File Service To do so:
The files are stored in a special generated table, not as a compressed file within file service.
Data Preview
After
uploading a file, it is a good idea for users to preview the data just to ensure that you havent made any mistakes with your file and the file is in the expected format Data Preview is accessed by clicking Preview data file from the Action Pane. It allows you to preview the first 200 lines of a data file.
The
source file for a data import can be a text file containing delimited strings. source file can be in either of three formats:
The
Delimited
Fixed - requires a first row defining the name and position of all dimensions.
The
Variant - allows the user to define the name and position of the dimensions using multiple lines beginning with an asterisk (*). Can not be viewed in delimited data type only fixed width
While the above example is very simple, some extra features of the variable format can demonstrate its potential as a more useful tool.
Repetitive field values can be moved into the header section. For example, the above file could also be read in the following format, where the value of the category (ACTUAL) and time (1999.Jan) are set in the header section and removed from the value rows
*<DIMENSION>=<VALUE> specify the value of the dimension, this has the same effect with *NEWCOL command in mapping section. The line start with is the comment line which will be skipped during process.
If one of the pre-defined values changes along the body of the file, a redefinition of the current member can be inserted anywhere in the file and as many times as desired, like in this example:
If the *<Dimension> =<value> header occurs in the middle of data file, all the records following this header are affected, but the records before the header will not.
ACTUAL, 1999.Jan, UK, Sales, Sedan, 1234 ACTUAL, 1999.Jan, UK, Sales, Coupe, 2345
A set of members of one or more dimension can span across multiple columns (Typically 12 months of budget values). The following example of header section will accept 12 value records:
*<Dimension> =<value1>,<value2> has the similar effect with *MVAL command in mapping section, this type of header can only occurs in the beginning of the data file but not in the middle of data file.
Essentially, the variable format allows the user to specify which dimensions are in the page (CATEGORY=ACTUAL), which dimensions are in column (TIME=1999.Jan, 1999.Feb, etc.) and which dimensions are in row (ENTITY,ACCOUNT,PRODUCT,PERIODIC). Irrespective of the format of the source file, the output file will always be in the "fixed format" described for the input file, i.e: The first line will contain the delimited list of all dimension names with their corresponding positions for the data fields. All subsequent lines will define one data cell per line. For example, the above 12 periods variable format file will generate the following output (if we exclude any conversion of the members labels):
users who are familiar with menus instead of the action pane, BPC has an eData dropdown menu to choose all the actions available within Data Manager action pane
Menu item Run Package Description Opens the Run Package dialog box, where you can run a package for a user, team, or company. Package Schedule Status Opens the Package schedule status dialog box, where you can view and edit scheduled packages. View Status Opens the View Status dialog box, where you can see the status of currently running or previously run packages. Organize Package List Opens the Organize Package List dialog box, where you can add packages and assign them to teams. Manage Team User Package Access Allows administrators to assign packages in certain teams to the users who belong to these teams. New Transformation File Creates a new transformation file Excel workbook. Manage Transformation File Show file list and allow you to edit, save, copy, and validate transformation files. Validate and Process Checks the structure and syntax of transformation files and reports any potential Transformation File problems. Copy Transformation File Creates new transformation files based on existing ones that you can then modify for your needs. New Conversion File Creates new conversion files. Manage Conversion File Allows you to edit, save, copy, and validate conversion files. Validate and Process Conversion Checks the structure and syntax of conversion files and reports any potential File problems. Copy Conversion File Creates new conversion files based on existing ones that you can then modify for your needs. Add Conversion Sheet Adds a sheet to an existing conversion file. You can have multiple sheets in one conversion file in order to use different sheets depending on the context of the transformation. Clear Saved Prompt Values Clears saved values for all Data Manager dialog boxes. Values include references to folders and lists of members. Data Manager Help Opens the help (Only available in BPC 7.0M, not BPC 7.0NW) About Data Manager Displays version information about Data Manager.
For
users who dont like the action pane or menu, BPC also has a Data Manager toolbar
Icon Task Run Package View Status New Transformation File Manage Transformation File Validate and Process Transformation File Copy Transformation File New Conversion File Manage Conversion File Validate and Process Conversion File Copy Conversion File Add Conversion Sheet <Selected Dimension> Description Opens the Run Package dialog box, where you can run a package for a team and application. Opens the View Status dialog box, where you can see the status of currently running or previously run packages. Creates a new Transformation File Excel workbook. Sub menus allow you to edit, save, copy, and validate transformation files. Checks the structure and syntax of transformation files and reports any potential problems. Creates new transformation files based on existing ones that you can then modify for your needs. Creates new conversion files. Sub menus allow you to edit, save, copy, and validate conversion files. Checks the structure and syntax of conversion files and reports any potential problems. Creates new conversion files based on existing one that you can then modify for your needs. Adds a sheet to an existing conversion file. You can have multiple sheets in one conversion file in order to use different sheets depending on the context of the transformation. Works with the Show Member List button to show the Data Manager member selector for the selected dimension. Clicking this button shows the Data Manager member selector.
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
Choose Manage Data -> Run a Data Management Package From the Run Package dialog box, select a team from the left, if desired, select a folder under the Package Group folders, then select a package from the list on the right. Click the Run button at the bottom of the dialog box. The screens that follow differ depending on the package chosen. Each package allows you to set options for running the package, scheduling the package, or both. Follow the instructions on the screen. After supplying the required information the package is sent to the server and executed. You can view the status of the package.
When scheduling Data Manager Packages, you have the following toolbar to help
Icon
Task View Status New Automation Edit Automation Run Automation Data Upload Data Download
Description View the status of all scheduled and finished packages Allows you to create an automation for reuse of scheduling packages Edits an existing automation Run an automation Upload a file to file service Download a file from file service
Refresh the list of packages (discards all changes that havent been saved) Exits the scheduling package functionality.
Data Manager Automation allows you to save particular schedules to reusable macro files To create a new automation, choose New Automation in the scheduling toolbar
NOTE: To refresh the view, you can choose Refresh status every and specify the polling time to refresh the job list.
The detailed log will give you detailed information for success or failed jobs:
The runtimes for each process type within the process chain The successful and failed records and package status
All user input for data management prompts are automatically saved This improves usability for repetitive DM tasks A user can clear the saved prompts by selecting the appropriate menu item
DEMO
Data Manager Running Packages
SAP 2008 / SAP TechEd 08 / <Session ID> Page 30
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
Organizing Packages
Packages are arranged for the convenience of the customer and their end-users. All packages in company are visible and available to all users with Data Manager task access Packages can also be organized by defined security teams. In the image below, admin is a team of users. The list is NOT a reflection of available packages, it is a reflection of what packages are available to the users of an application. These packages are copied from an application when you create a new application and are delivered by default for all applications
Organize package
Organize package is controlled by administrators and team leaders (only for their teams) Package options
Add - allows authorized users to select from available packages on the server Modify allows authorized users to change the variables or behavior of a package, it is best practice to first copy a pre-delivered package prior to modification to ensure the original package remains as shipped with the product. Copy creates a duplicate package with a new package ID Move moves a package between teams and/or package groups, it is common to remove rarely used or potentially damaging packages from the company team to the admin team
Within the Company area or within a Team, you can create Package Groups which act like folders to organize your packages within a team. To do this, right click on any folder and choose Add Package Group
In BPC 7.0 NW, a Data Manager Package can be created from a BPC Process Chain To Create a New Package, go to Manage Data -> Maintain Data Management -> Manage Packages -> Add Package
In BPC 7.0NW, only one package type is available (called Process Chain) You can then choose which process chain to use for Data Manager Only Process Chains in the /CPMB/ namespace are displayed In BPC 7.0M, the package types allow you to choose SSIS packages on the file system You can then name your BPC Data Manager Package (ie. Import into Financial Application) NOTE: The following screenshots are from the BPC 7.0NW version of the product
You can assign your package to a Group. This is the folder to which the package will be added. You can add a new group by typing a new group name. This is just used for organization and categorization. The group field has no functional impact. You can populate a description as well to describe your package
The Task Type can either be a User Package or an Admin Package. This allows you to control who has access to run the package: A user or an administrator. It only makes sense to flag either User Package or Admin Package but not both. If you flag Admin Package, this will take precedence, and ignore that User Package is flagged If the task type is flagged as Admin Package only users who are BPC Administrators will be able to run this package. This package can still be saved to any team (even non-Admin) teams, but only Administrators can run this package A BPC Administrator is defined by a user who has access all the DM Tasks assigned see the list below User Package can be seen by all BPC Users that are assigned Execute Data Management task security Packages can be assigned to teams using the menu Manage team user package access by a team leader or as part of organize packages if an administrator The primary purpose is to restrict a user from running a package that could damage to the data like clearing all data (zero values) by accident. NOTE: BPC 7.0M has two additional package types that are not used in BPC 7.0NW for special actions or packages that process the fact or cube Fact or Cube Process Package Special Package
Modify package
BPC 7.0 allows for the modification of a package via the BPC Excel eData interface
Package type Name of the process chain or SSIS package Package name Location Descriptive name to display Task type
User Admin
In BPC 7.0, a Data Manager Package can be assigned to teams To assign a package to a team, go to Manage Data -> Maintain Data Management -> Manage team userpackage access This will allow you to flag packages as available to be executed Only packages that are flagged as Task Type = User Package and not flagged with Task Type = Admin Package will be allowed to be executed by BPC Users that have the Execute You will not see any packages within this list that have been flagged as Admin Package
In addition to using the context menu to organize package, BPC provides a toolbar to organize packages After changes are complete, be sure to save your changes to the DM package list
Icon
Task Add Package Modify Package Copy Package Move Package Remove Package Add Package Group Modify Package Group Remove Package Group Save Package List Refresh Package List Exit
Description Used to add a new data manager package (choose the Process Chain to assign) Allows you to update package definition Gives you the ability to copy packages between teams and package groups Allows you to move packages between teams and package groups Delete a pacakge Create new package groups (folders) for organizing your packages Allows you to rename package groups (folders) Delete package groups (folders) Saves all changes made to packages. None of the changes are committed to NW until this button is clicked Refresh the list of packages (discards all changes that havent been saved) Exits the organize package functionality.
Administrators have access to all packages for the AppSet. You may want to customize the list of packages you run often so you do not have to find the packages each time. You do this using the Selected List button on the Run Package dialog box. You can also use the Selected List dialog box for the following: Use the data file transfer button to upload packages to the server (see Transferring data files) or use the view package status button to view package status. Users can create a package list using the Selected List functionality
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
SAP 2008 / SAP TechEd 08 / IM106 Data Manager - Page 42
Transformation files
Process
Transformation Transformation
file is selected during the run time of the import package. definition are in Excel files that contain one worksheet named
Instructions. The file is divided into 3 parts (*OPTIONS; *MAPPING, *CONVERSION) The transformation file references conversion files that may contain multiple worksheets.
Transformation Files
SAP 2008 / SAP TechEd 08 / <Session ID> Page 44
DEMO
Transformation Files allow application administrators or team leaders to set up the rules for reading data from a source and putting it in the proper form within your SAP BPC 7.0 target application. To create a new Transformation file,
select eData > New Transformation File. To edit an existing Transformation file, select eData > Manage Transformation Files, choose a team in the left column of the dialog box, choose a folder, and then double-click an existing Transformation file name. Fill out or define the options, mapping, and conversion file sections. *OPTIONS: Set up Mapping Options *MAPPING: Map Dimensions to source *CONVERSION: Setup Conversion Files for dimensions Save the Transformation file.
AMOUNTDECIMALPOINT = <text_character>
Optional Parameter Allows you to specify a non-period decimal point for countries that use a different character, such as a comma. Note: the character specified in this option must differ from the character specified for the DELIMITER. Default Value: Period CONVERTAMOUNTWDIM = <dim_name> Optional Parameter Specifies which dimension to look at for value calculations. You must specify a dimension conversion sheet using the Amount *Conversion option If there is no formula in the Formula column of the Amount conversion sheet this parameter has no affect at all. For example, the Formula column has the following formula: Value*1.10. All accounts will be increased by 10% during the conversion. Default Value: Account CREDITPOSITIVE = YES | NO Required Parameter If No, all amounts referring to an ACCOUNT type (LEQ, INC) will have their signs reversed. Default: YES DELIMITER = <text_character> | SPACE | TAB Required Parameter If the FORMAT option is set to DELIMITED, this option defines the single character that is the delimiter between columns. use the keywords SPACE or TAB if space- or tab-delimited. Default: , (comma)
Required Parameter The format of the data in the input or output file. You must enter a format type. Delimited means that there is a special character between each column of data, defined by the Delimiter option (below). Fixed means that the data is in a fixed field format. Use the FCOL mapping option.
Required Parameter If YES, then your input file contains one header row that defines the fields. If you do have a header row, you can refer to a field by name in the MAPPING section. Default: YES MAXREJECTCOUNT = <empty_string> | -1 | positive number Required Parameter If validating records, the number of rejected records at which to stop processing. A value of -1 implies that DM should keep processing no matter how many rejected records exist. The default value is 500, which can be represented by an empty string. Default: <empty_string> SUPPRESSCHARACTER = <text> This option is used to remove certain text in records, the text is specified by <text>, for example
SUPPRESSCHARACTER = abc will remove all abc occurs in the data records.
Default: <empty>
ROUNDAMOUNT = <integer>
Required Parameter Specifies the amount of decimal places to round values during the transformation. When specifying integer, the integer can be zero or any positive integer. Default: no rounding occurs SKIP = <integer> Required Parameter Number of lines to skip at the top of a data file. If your data file has a header, set this value so that those lines are skipped during transformation. Default: 0 (zero) SKIPIF = <text string1> | <text string2> Required Parameter Skip a line in the data file if it begins with the specified strings the strings are separated by |,in the above case. If a record contains <text_string> or <text_string2>, the record will be skipped. Setting this option to an empty string means that no lines are skipped in the body of the data file. Header lines can still be skipped using the SKIP command, above. Default: <empty_string> VALIDATERECORDS = No | Yes Required Parameter Default: No If YES, validate the mapping and that members exist, and map data in the proper order for Business Planning and Consolidation. For example, the Amount field is always moved to the end of the records even though it might be written in the first line of the mapping section. If NO, will not validate. In this case, if the mapping section is out of order or just wrong, fields will be skipped.
SELECTION = <Dimension1_techname>,<Dimension1_value>;<Dimension2_techname>,<Dimension2_value>
This option is only used for importing transaction data from infoprovider, it is illegal when running other packages. The SELECTION option set the select rules when exacting transaction data from infoprovider.<Dimension1_techname> is the technique name of the dimension, <Dimension1_value> is the value of Condition statement set to this dimension. For example 0SX_ACCS, US;0SX_CSLC, 1 represents the selection statement 0SX_ACCS=US and 0SX_CSLC= 1 NOTE: This is used for loading transactonaldata as is new in BPC 7.0 Default: <empty> FORMULA = LIB.LGF|LIB1.LGF This option is only used for importing master data, it is illegal when running other packages. The FORMULA option set the K2 functions referenced by the master data members formula attribute, because all The libraries are syslib, the *syslib prefix is not needed here. Different libraries is separated by | . NOTE: This is used for loading master data as is new in BPC 7.0 Example: DAVY_KP1_DEV7A2_Q_Mdxlib_1.LGF| DAVY_KP1_DEV7A2_Q_Mdxlib_2.LGF Default: <empty> CONVERT_INTERNAL = YES | NO If NO, the input member names will be compared with external column in conversion file. If YES, the input member names will be compared with internal column in conversion file. Default: NO BEST PRACTICE: Always keep this set to NO. There is really no functional reason to set this to Yes,
While this is supported, it will just cause confusion by setting this to yes This is there for legacy reasons
The *MAPPING section of the transformation file defines how data is mapped to your Business Planning and Consolidation Application The following options are available:
*COL(A) *COL(A,B:C) *FCOL(A:B) *MVAL(A:B) *NEWCOL(A) *Str(string) *Pad(A,B,C,D) *If(Condition1 then Action1;Condition2 then Action2;Default Action)
By default, Data Manager associates fields to header names when you have a header row in your data file. This makes it easier to define mappings in this section of the transformation file. For example, if you have a header row and it defines a column named "AccountVal," you can map the Account dimension to the Account field from the data file this way:
Account = AccountVal
A = column index in the data file Defines a dimension for a field in the data file when the data file does not have a header row. See the note below about automatic field names when you do have a header row. NOTE: This command is only valid in delimited format Example:
Account =*COL(2)
A = column index in the data file B:C = start/end position within the column Defines a dimension for a subset of a field in the data file. NOTE: This command is only valid in delimited format Example:
Account=*COL(3,1:4)
A:B = start/end position within the data row When using Fixed format data files, this option allows you to define the start and end character columns for each field. In the example below, the columns 4 through 7 represent the Account dimension. Example:
Account=*FCOL(4:7)
A:B = start/end column of time period Use this command if your data file has multiple period columns. For example, if you have columns that are like this:
Account,Category,DATASRC,entity,IntCo,RptCURRENCY,JAN,FEB,MAR,APR,MAY,JUN ,JUL,AUG,SEP,OCT,NOV,DEC
You would use Time = *MVAL(7:18), since columns 7 through 18 contain data based on time periods. Using this command requires that you use a Time.xls conversion sheet. See the sample Time.xls conversion file, located in the ConveresionFiles folder, for more information.
If your data does not have a header row, the time periods will be named P1, P2, P3,...,PN by default.
Example:
Time = *MVAL(7:18)
A = dimension member Creates a new field with the given value. Example:
Account=*NEWCOL(Revenue)
Adds a text string to the members of a column. Useful if you need to map data file fields to fields in your application that have the same names, except they have extra characters appended or in front of the field name. Example:
*Pad(A,B,C,D) Arguments:
A = Field name - (the field name is header name in data file) B = Total length C = Padding direction D = Padding character
Use to format exported data into fixed-width records. Use in conjunction with the "None" delimiter. The Padding character is optional and is a space by default.
NOTE: This command is only valid in delimited format and with header turned ON Example:
Account=*Pad(Account,10,L)
Account=*Pad(Account,10,L,0)
*IIf() Example 1
Example 1:
Entity, SEntity U1000000, US01 Z2000A01, CA03 K3430000, JP04 Entity=*IF(*col(1,1:1)=U then SEntity;*col(1,1:1)=Z then *col(1,3:6); *STR(ERR)) Entity, SEntity US01, US01
A01, CA03
ERR, JP04
*IIf() Example 2
Example 2:
ENTITY 52 EF KL
Product=*IF (product+ID+Entity=*Str(148552) then *str(MHarn);ID(1:1) = *str(C) then *str(XX) ; *str(YY)) PRODUCT, ID, ENTITY
MHarn,
AB, YY,
85,
XX, IJ,
52
EF KL
Besides the simple header(the header with only one line), the data file can also contain variant header, the header can have several lines, each line must start with *(star), here is four cases for variant header
*CATEGORY, TIME *ENTITY, ACCOUNT, PRODUCT *PERIODIC ACTUAL, 1999.Jan, UK, Sales, Sedan, 1234 ACTUAL, 1999.Jan, UK, Sales, Coupe, 2345 ACTUAL, 1999.Jan, FR, Sales, Sedan, 986 ACTUAL, 1999.Jan, FR, Sales, Coupe, 945
Case 2--*<DIMENSION>=<VALUE> specify the value of the dimension, this has the same effect with *NEWCOL command in mapping section. The line start with is the comment line which will be skipped during process.
*CATEGORY=ACTUAL *TIME=1999.Jan *ENTITY, ACCOUNT, PRODUCT *PERIODIC '----------------------------------this is the comment line, will be skipped when processing UK, Sales, Sedan, 1234 UK, Sales, Coupe, 2345 FR, Sales, Sedan, 986 FR, Sales, Coupe, 945
ACTUAL, 1999.Jan, UK, Sales, Sedan, 1234 ACTUAL, 1999.Jan, UK, Sales, Coupe, 2345 ACTUAL, 1999.Jan, UK, Sales, Sedan, 986 ACTUAL, 1999.Jan, UK, Sales, Coupe, 945
Case 3If the *<Dimension> =<value> header occurs in the middle of data file, all the records following this header are affected, but the records before the header will not.
*CATEGORY, TIME *ENTITY=UK *ACCOUNT, PRODUCT, PERIODIC ACTUAL, 1999.Jan, Sales, Sedan, 1234 ACTUAL, 1999.Jan, Sales, Coupe, 2345 *ENTITY=FR ACTUAL, 1999.Jan, Sales, Sedan, 986 ACTUAL, 1999.Jan, Sales, Coupe, 945 The results of Case 3 is : ACTUAL, 1999.Jan, UK, Sales, Sedan, 1234 ACTUAL, 1999.Jan, UK, Sales, Coupe, 2345 ACTUAL, 1999.Jan, FR, Sales, Sedan, 986 ACTUAL, 1999.Jan, FR, Sales, Coupe, 945
Case 4--*<Dimension> =<value1>,<value2> has the similar effect with *MVAL command in mapping section, this type of header can only occurs in the beginning of the data file but not in the middle of data file.
*CATEGORY=ACTUAL *TIME=1999.Jan, 1999.Feb, 1999.Mar, 1999.Apr, 1999.Jun, 1999.Jul, 1999.Aug, 1999.Sep, 1999.Oct, 1999.Nov, 1999.Dec *ENTITY, ACCOUNT, PRODUCT *PERIODIC UK, Sales, Sedan, 100, 200,320,450,600,700,850,900,230,200,340,560 UK, Sales, Coupe, 1120, 1230,1220,470,1600,7900,1850,1900,1230,3200,3340,2560
CATEGORY, TIME, ENTITY, ACCOUNT, PRODUCT, PERIODIC ACTUAL, 1999.Jan, UK, Sales, Sedan, 100 ACTUAL, 1999.Feb, UK, Sales, Coupe, 200 ACTUAL, 1999.Mar, UK, Sales, Sedan, 320 ACTUAL, 1999.Apr, UK, Sales, Coupe, 450
Here is an example of loading data from an existing NW BI EDW Cube (this is available in the NW BI version only) All filters are specified within the SELECTION keyword. Conversion Files can be used to convert data as well. All loads are full loads using this package. Merge, Replace, and Append options are all available!
You can validate the transformation file by itself, or by supplying a live data file.
If you are an application administrator, select eData > Validate & Process Transformation File.
When you choose to validate, you can choose the type of transformation file youd like to validate against:
Transaction data
Master Data
Once you choose to validate, you will get a log telling you any issues validating the transformation file.
You can choose Copy Transformation File to create a new transformation or use save as with an open transformation file You will be prompted you to copy the file to a target location Transformation files can be saved without validation
When you want to copy dimension files between applications, but the dimension names are different, you can create a transformation file to map the dimensions. For example, if the dimensions in the source application are:
Category, Account, Entity, Time, Rptcurrency, Intco, and Datasrc Cat, Acc, Ent, Tim, Currency, Int, and Dat
then you can use a transformation file to map the source to the target.
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
* Conversion
Conversion files define the mappings by Dimension Define the mapping from external to internal member names (source to target)
Within BPC 7.0NW, Data Manager Supports Javascript in Conversion Files Within BPC 7.0M, Data Manager Supports VBScript in Conversion Files
Conversion Files
SAP 2008 / SAP TechEd 08 / <Session ID> Page 70
DEMO
Conversion files define how to map the master data in input files (external) to application master data (internal) Also supports formulas to convert the data *CONVERSION
The transformation file section directs the usage of a conversion: Dimension Name = [COMPANY]WorkbookName[!SheetName] or [TeamName]WorkbookName[!SheetName] - Items in brackets are optional. Workbook name is the name of the conversion file. Sheet Name = is the name of the worksheet to use within the conversion file. If not specified, Data Manager assumes the sheet name is: Conversion If [COMPANY] is defined or doesnt exist, then DM will try to obtain the conversion file from the Main company Data Manager folder; if [TEAM_NAME] is defined, Data Manager looks into the corresponding team folder. Dimension Name is either the dimension name to correlate with the conversion file or the keyword Amount. When using DimensionName to specify the conversion file to be used, the external member names of the specified dimension name will be converted according to the rules defined in the referenced conversion file. When using the keyword Amount as DimensionName to specify the conversion file to be used, the value of ConverAmountWDim option in the *Options section will be considered together when doing conversion. This allows you to use a formula to do calculations to the input values.
Examples:
Where AccountConv.xls is the name of the Account dimension conversion file. Account = AccountConv.xls
the input member names of dimension Account will be converted according to the conversion rules defined in the conversion file AccountConv.xls.
Amount = AccountConv.xls!Calculus
Account = AccountConv.xls!newaccount
Account = [COMPANY]AccountConv.xls!newaccount
This conversion file should be in the company folder This conversion file should be located in the corresponding folder of AdminTeam
Account = [AdminTeam]AccountConv.xls!newaccount
To create a new conversion file, go to Manage Data -> Maintain Conversions -> Create New Conversion
When working with this conversion file you have three columns:
EXTERNAL, INTERNAL, and FORMULA NOTE: When transformation option CONVERT_INTERNAL = NO, the external column refers to the source value, the internal column refers to the target dimension member you want to load into the application. When Convert_internal = YES, the external column refers to the target and the internal column refers to the source value. The following example is for when CONVERT_INTERNAL = NO
Refers to the source value since CONVERT_INTERNAL = NO You can use wildcards to match source values
The Internal Column Refers to the target dimension member you want to load into the application (since CONVERT_INTERNAL = NO) You can use formulas within this column Use %EXTERNAL% in your formula to refer to the source value for derivation of the target value You can use wildcards to replace target values
You can use the keyword VALUE to refer to the signeddata value in the source file
When working with conversion files, you can directly map external values to internal values
This is similar to what a VLOOKUP formula in Excel would do Anytime the External value appears in the source, Data Manager would low the Internal value into the target
Special Options: *SKIP: You can ignore some data records and just skip them without loading them into the target cube.
The below example means that all the data records will be skipped and not be loaded into target cube, if the member of dimension ACCOUNT is ACCPAY.
Wildcards: You can use the asterisk (*) and question mark (?) wildcards in the External and/or Internal columns. asterisk (*) stands for ANY character, while question mark (?) stand for any SINGLE character. You also can use colon mark (:) for a specified range.
For example, if you want to reference all members, use the asterisk (*). This would be useful to apply a formula to updated signeddata for all members of a particular dimension:
Using Wildcards
Wildcards can be used within both the external and internal columns!
This will allow for very dynamic replacements (convert_internal = NO, the input member name will be compared with External column)
Correct examples:
Input member name abcdef abcdef abcdef abcdef abcdef abcdef abcdef abcdef abcdef abcdef abcdef abcdef 23 abs c100 Wrong examples: Input member name abcdef abcdef abcdef External a*f abc* *f a*f abc* *f a*c*f a*f a*c?*f a*c?? abcdef abc??? 1:100 aba:abz c001:c999 External a*c*f a*d*f abc??? Internal c*g c*g c*g cg* cg* *cg cg c*g* cg c*g c*g ???cg abc def def Internal *cg *c*g ????g Result cbcdeg cdefg cabcdeg cgbcde cgdef abcdecg cg cbcdegbcde cg cbg cg defcg abc edf def Result Error, the meaning of asterisk (*) for replacement is vague. Error, the meaning of asterisk (*) for replacement is vague. Error, the number of ? is not consistent
Case 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
17 18 19
Formulas for Dimensions To write a formula for dimensions, you will write this formula within the INTERNAL column
This formula will do the following: If the value for category in the source file begins with act, then it will map this value to Actual. Otherwise, it will check if the value for category in the source is 4 characters long. If it is, it will write the value Budget. If not, it will write the value Actual
Formulas for Measures To write a formula to update SIGNEDDATA, you will write this formula with the FORMULA column
This formula will take multiple the value of signeddata in the source by 100 and round it to two places for all time periods and then load this into the target.
VALUE
This can only be used in the formula column of a conversion file It refers to the signeddata in the source file you are trying to load from. The Keyword VALUE is treated as a formula variable in a conversion file The keyword AMOUNT can only be used in the conversion section in transformation files;
AMOUNT
CONVERT_INTERNAL If transformation option Convert_internal = NO, the javascript can be used in Internal column. If transformation option Convert_internal = YES, the javascript can be used in External column; Precede the Java Script code with string js:, which is mandatory. You can use Java Script code in IF statements, too. Examples: js: %external%.substring(0,2) Return a Long containing the number of characters in a string js: %external%.length Return the length of the string contained in external. js: %external%.replace("apples", "oranges") Return a string in which a specified substring has been replaced with another substring a specified number of times.
Currently, we do not allow creating custom javascript includes (This is being evaluated for a feature pack)
js:Math.round(VALUE*Math.pow(10,2))/Math.pow(10,2);
You cannot refer to another dimension or dimension property within a formula when trying to derive the value of a dimension (writing formulas in the INTERNAL column) within a conversion file Example 1:
formula
Formulas - Example 2
Example 2:
The input member names are Acc123 and bcdef. The rules in AccountConv.xls are:
formula
Formulas - Example 3
Example 3:
ConvertAmountWDim = Account The input member names of Account are Acc123 and bcdef with values 100 and 200 respectively. The converted member names of Account are Accpay and Accrec.
To add a conversion sheet to a conversion file, choose Add Conversion Sheet within the eData menu
To validate a conversion file, choose Validate & Process Conversion File within the eData menu
Check if the excel sheet format (the header line should be External, Internal, Formula) is correct.
check formulas --- this will be checked at runtime. check dimension names --- dimension name will not appear in conversion file, so it will not be checked. check valid dimension members --- dimension members will be checked at runtime if the value of the transformation option VALIDATERECORDS is yes.
Choose Manager Data -> Manager Existing Conversion Definition Open a Conversion File Choose Copy Conversion Click Save
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
The UI Script is used to display a dialog to pass user parameters back to the BPC Packages Within BPC 7.0NW, this would be Process Chain Templates Within BPC 7.0M, this would be SSIS packages The User Interface of Data Manager Packages is dynamically generated from the UI Script To get to the UI Script Choose Organize Package and selected the package name Right click for menu to choose Modify Package Then choose the View Package button
The UI Script can be updated from the Dynamic Script Editor for BPC Data Manager The tasks will display the BPC Process Types or SSIS tasks that are assigned to the BPC Package Within the Dynamic Package, you will be able to define 3 sections:
Prompt: This command generates customized dialog boxes, and, by answering the requests of these dialogs, the user tells BPC things like: what file to load, what selections of data to export, in which application to copy the selected data, etc. The syntax for the PROMPT( ) command is: PROMPT([prompt type], [variable name], [second variable name], [parameters,]) Info - The INFO instruction is not an executable instruction, in the sense that it does not 'do' anything specific to the package. Its purpose, as stated earlier, is only to define and assign a value to a script variable.
The syntax for the INFO instruction is: INFO([variablename],[value]) Task The Task displays all the values that can be passed to each BPC Process Type or SSIS Task. The syntax for the TASK instruction is: TASK([taskname],[PropertyName],[PropertyVal ue])
The Dynamic Constants display any system parameters from BPC that can be sent to the backend Process Types or SISS Tasks
In Advanced mode, all code for the dynamic script can be directly edited and saved.
The prompt, info, and task have template syntax here which can aide in writing the dynamic script
The package options can be set in a script using the advanced option (rather than navigate each option in previous dialog.
DEMO
UI Script
SAP 2008 / SAP TechEd 08 / <Session ID> Page 94
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
Loading Large Files from AppServer instead of Database (NW version only)
When the user wants to execute this package (i.e. package runtime), he will be asked to select the file they want to use in their Data Manager package. The user will be given a dialog box containing an F4/Value Help function, to select from a list of files.
Instead of choosing a file, the user can specify a NetWeaver application server file system location to import a file from the NW Application Server.
This may be useful for any files that are already being stored on the NW Application Server from 3rd party applications
/uscikp1_KP1_77/tmp/FINANCE2250000.txt
For example:
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
are four broad types of delivered Process Chain Templates which can be used to build BPC Packages:
Standard Data Loading Packages Financial Packages Administrative Packages Example Packages
These
process chain templates can be displayed and edited in ABAP transaction RSPC
Admin Task Audit Archive Clear BPC Table Clear Comments Clear Cube Convert Convert for Copy and Move Create Dimension Dynamic Modify Email FTP Finalize Optimize Journals Load Master Data Load Master Data Description Load Optimize Cube Ownership Calc Run Logic OnLine Status
Standard Data Manager Packages perform their function while the system is live (i.e. users can be online).
For example, you can use a standard package to load monthly data. Allow you to move data into and out of SAP BPC databases.
The packages that come with BPC 7.0NW are designed to be dynamic so that you do not need to modify the packages in order for them to work with your applications and dimensions.
You might want to modify packages to behave differently or to perform additional tasks. You can make modifications to the packages supplied with BPC or use those packages as templates to build packages as per your requirements (best practice) We recommend that you use standard packages whenever possible, because they work within the SAP Business Planning and Consolidation user ID security framework and there is little risk of data integrity issues.
7.0 (NW) delivers 7 process chain templates that are standard data manager packages
Description This package allows you to load dimension member data directly into a BPC Dimension's Member Table (BW InfoObject's master data table) TechName /CPMB/IMPORT_MASTER
Process Chain Template BPC: Import Master Data BPC: Import Master Data Description
This package allows you to load dimension member texts directly into a /CPMB/IMPORT_DESCRIPT EVDESCRIPTION (BW InfoObject's text table) BPC: Import Transaction Data This package imports one or more ASCII files into the current application. /CPMB/IMPORT You are prompted for one or more import file and associated transformation files. The signs of the amounts are reversed based on the Account type. You can use the import.xls transformation file with this package. BPC: Load Transaction Data from Other Cube This package allows you to load data from one BI Cube (EDW or BPC) /CPMB/LOAD_INFOPROVIDER into a BPC Application. You can associate a tranformation file to do the mapping. BPC: Move Transaction Data in one cube This package lets you move any selection of data within an application. /CPMB/MOVE You are prompted for source and destination data selections for all dimensions. The destination selection's fact table records are cleared before the source's records are moved. The source's fact table records are cleared after they are moved to the destination. BPC: Clear Transaction Data in One Cube This package clears the data in the BPC Application by importing zero /CPMB/CLEAR values. The optimize process removes the zero values, which clears all values from the table. Running this package also clears comments from the comments table. You can use the import.xls transformation file with this package. You must select at least one dimension in order to run this package. If you select [All], the package will not run. BPC: Copy Transaction Data in One Cube This package copies data between dimensions within BPC application. /CPMB/COPY BPC: Append Transaction Data This package imports data into the application by appending the data to /CPMB/APPEND the fact table, and then runs default logic against the imported data set. You can use the import.xls transformation file with this package.
This package allows you to load dimension member data directly into a BPC Dimension's Member Table (BW InfoObject's master data table) If the source dimension member (the member record in data file) already exists in the AppSet (ID is the only key to identify if the member already exists), the source dimension member will overwrite the member exists in AppSet If the source dimension member does not exist in the AppSet, the source dimension will be inserted into AppSet as a new one The dimension members in AppSet but not in source file will stay untouched
This package allows you to load dimension member texts directly into a EVDESCRIPTION (BW InfoObject's text table)
Example:
Example: Dimension: Account. Member: ACCPAY Description before: ACCPAY E Account Payable ACCPAY F Account Payable in French Descriptions in data file: ACCPAY E Account Payable in English ACCPAY C Account Payable in Chinese Description after: ACCPAY E Account Payable in English ACCPAY F Account Payable in French ACCPAY C Account Payable in Chinese
Replace lets the user reset that region of the Application similar reloading the file as if it was a new data set
It is the same as inputting a new financial statement for a specific region of the Application The business requirement for replace within the import package is the same as inputting a new financial statement for a specific region of the cube
If your GL only provided a new export for actual and May on its entirety, you would not be able to determine accounts that have been reclassified, changed or remain the same
Replace lets the user reset that region of the cube similar reloading the file as if it was a new data set It is really an option for what the import file represents
The merge option (alternative to replace) changes only like data between source and database
In BPC 7.0, using the Merge option, the output would be as follows:
ACCOUNT FRINGEBENEFIT OTHEREXP SALARIES.CAT1 DIRECTLABOR FRINGEBENEFIT OTHEREXP SALARIES.CAT1 CATEGORY ACTUAL ACTUAL ACTUAL ACTUAL BUDGET BUDGET BUDGET DATASRC INPUT INPUT INPUT INPUT INPUT INPUT INPUT ENTITY SALESITALY SALESITALY SALESITALY SALESITALY SALESITALY SALESITALY SALESITALY INTERCO NON_INTERCO NON_INTERCO NON_INTERCO NON_INTERCO NON_INTERCO NON_INTERCO NON_INTERCO RPTCURRENCY LC LC LC LC LC LC LC TIME 2006.JAN 2006.JAN 2006.JAN 2006.JAN 2006.JAN 2006.JAN 2006.JAN SIGNEDDATA 1600 500 250 160 15 20 30
In BPC 7.0, using the Replace option, the output would be as follows, because it will first delete all CATEGORY=ACTUAL, DATASRC=INPUT, ENTITY=SALESITALY and then add the data to the cube It would not delete entries where Category is budget
ACCOUNT FRINGEBENEFIT DIRECTLABOR FRINGEBENEFIT OTHEREXP SALARIES.CAT1 CATEGORY ACTUAL ACTUAL BUDGET BUDGET BUDGET DATASRC INPUT INPUT INPUT INPUT INPUT ENTITY SALESITALY SALESITALY SALESITALY SALESITALY SALESITALY INTERCO NON_INTERCO NON_INTERCO NON_INTERCO NON_INTERCO NON_INTERCO RPTCURRENCY LC LC LC LC LC TIME 2006.JAN 2006.JAN 2006.JAN 2006.JAN 2006.JAN SIGNEDDATA 1600 160 15 20 30
Standard Data Management Packages - BPC: Load Transaction Data from Other Cube
BPC:
This package allows you to load data from one BI Cube (EDW or BPC) into a BPC Application
You are prompted for one or more source and associated transformation files The signs of the amounts are reversed based on the Account type Security is required to be granted to the BPC service users to allow NW BI access to any source EDW cube if an EDW cube is used
BPC:
BPC:
Clears the data in the Application by importing zero values Running this package also clears comments from the comments table
BPC:
Copies data between members within the same dimension This package imports data into the application by appending the data to the fact table
BPC:
All packages that manage data can run default logic, must will unless a prompt is included within the package or the administrator changes the behavior of the package.
SALESITALY SALESITALY
RPTCURRENCY LC LC
In BPC 7.0, using the Append capabilities, the result would be as follows:
ACCOUNT FRINGEBENEFIT OTHEREXP SALARIES.CAT1 DIRECTLABOR FRINGEBENEFIT OTHEREXP SALARIES.CAT1 CATEGORY ACTUAL ACTUAL ACTUAL ACTUAL BUDGET BUDGET BUDGET DATASRC INPUT INPUT INPUT INPUT INPUT INPUT INPUT ENTITY INTERCO NON_INTERCO NON_INTERCO NON_INTERCO NON_INTERCO NON_INTERCO NON_INTERCO NON_INTERCO
RPTCURRENCY LC LC LC LC LC LC LC
Data Manager packages for loading dimension members Updates Member Sheet Can load master description from flat file into a NW text table Language codes can be converted to NW required codes
Example:
ID NetIncome
NetIncome NetIncome
LANG EN
FR DE
In the underlying table, EVDESCRIPTION isnt stored in the member table A foreign key relationship to this language specific table Logon will read the dimension member table, and join to this language specific table Text returned based on end-user setting Users member sheet will only show this text Updating EVDescription will update this language specific table. The user can maintain there language view My Settings using BPC Web
Process Chain Template BPC: Default Formulas Logical BPC: Allocation BPC: Calculate Ownership BPC: FX Restatement
Description This package executes default formulas stored in your default.xls file. The package runs the Allocation logic. The package runs the CalcOwnership logic. This package is used for currency translation. The package runs the FXTrans logic. BPC: IC Elimination This package is used to Perform Inter-Company eliminations. The Package runs the ICElim logic. BPC: ICBooking The Package runs the ICBooking logic. BPC: ICData The Package runs the ICData logic. BPC: Legal Consolidation The Package runs the LegalConsolidation logic. BPC: Opening Balances The Package runs the OpeningBalances logic. BPC: Run CalcAccount The Package runs the CalcAccount logic. BPC: Clear the Journal Clears Journal tables and creates an output file. Tables BPC: Export the Journal Exports Journal tables to an output file Tables BPC: Restore Journal Allows you to load Journal tables from a File Tables
TechName /CPMB/DEFAULT_FORMULAS
For example, you can use an administrative package to include back-end data manipulation where you are loading bulk data for the first time, exporting an entire database, and running default logic formulas on your data.
Take
Can
BPC: Admin Task Validate Validates the logic file. BPC: Validate Transformation Validates the Transformation Files for Data Manager File BPC: Validations Runs BPC Validation of Business Rules Table BPC: Archive Activity Archives Audit Activity BPC: Archive Data Archives Audit Data BPC: Clear Comments Clears comments from the comments table. BPC: Full Optimization Runs the BPC Admin Full Optimization Process - this chain can be run from Admin Console BPC: Light Optimize Cube Runs the BPC Admin Light Optimization Process - this chain can be run from Admin Console
Process Chain Template BPC: Import and Send Mail BPC: Import Using FTP
Description TechName Import an ASCII file and send the result to specific user. /CPMB/IMPORT_A_SEND_EMAIL This package allows you to import an ASCII file to download from an FTP /CPMB/IMPORT_USING_FTP site into the current application. You can use modify the server name within the package. After you modify the server name, you can run this sample package at any time.
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
In
Today, there is a how-to guide available for the MSFT version already Use Data Manager packages to natively transform NW BI data to BPC format data Possibly via Data Integrator/BICS/UD Connect
Tighter integration to NW BI
In
the future:
Enhanced parallelization and performance enhancements Enhanced user experience and even simpler user interface
This slide outlines our general product direction and should not be relied on in making a purchase decision. This slide is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this slide or to develop or release any functionality mentioned in this slide. This slide and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This slide is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this slide, except if such damages were caused by SAP intentionally or grossly negligent.
SAP 2008 / SAP TechEd 08 / IM106 Data Manager - Page 117
Design-time
Transformation Files Conversion Files UI Script Security Large Files Delivered Objects
Future Wrap-up
Further Information
Thank you!
Thank You !
SAP 2008 / SAP TechEd 08 / IM106 Data Manager - Page 121