No more unknown members!

Smart data load validation for Hyperion Planning

using ODI

Ricardo Giampaoli
Rodrigo Radtke
About the Speakers
Giampaoli, Ricardo
Oracle Ace Associate
Master in Business Administration
and IT management
Founder of TeraCorp Consulting
EPM training instructor
Essbase/Planning/OBIEE/ODI
Certified Specialist
Blogger @ devepm.com
Radtke, Rodrigo
Oracle Ace Associate
Graduated in Computer
Engineering
Software Developer Advisor at Dell
ODI, Oracle and Java Certified
Blogger @ devepm.com
About TeraCorp

TeraCorp is a company specialized in products and services

focused on EPM
TeraCorp mission is to create innovate solutions that helps
people, businesses and partners to exceed their goals reaching
their full potential.
Learn more @ www.teracorp.com.br/en
Hyperion Planning Architecture
Planning uses RDBMS to store its metadata
information
One main repository is used for Planning system
One additional repository for each Planning App
Planning uses Essbase as Database
The Planning App repositories has the same
Metadata information than Essbase Outline
The Metadata sync is done by Refreshing the
metadata from Planning to Essbase
Hyperion Essbase Architecture
Store all Data used in Hyperion
Planning
One Essbase App for each
Planning App
One Essbase cube per each
Planning Plan Type
Each cube has its own outline that
matches the Planning metadata
Each cube has its own Data Files
that stores the Planning data
Essbase Data Load

In order to load data in Essbase, its mandatory to inform a

combination of members from all dimension and a data value
Essbase reads data sources starting at the top and proceeding from left
to right
Essbase gets the sparse members combination informed and
brings/creates the block in memory
Essbase tries to load this member combination data into the block
If all members from that combination are valid, the data is successfully
loaded
Essbase Data Load

Three ways to load data:

Using Essbase Load Data
If a member is not valid, the process is aborted and no data is
loaded
Using Essbase Load Rules
If a member is not valid, the process continues (depending of
the configuration) and all data but the erroneous ones is loaded
Using ODI
ODI is a powerful ETL tool used to integrate all different data
sources with Essbase using Java Essbase API to load data
Essbase Data Load using ODI

ODI sends chunks of data to Essbase based on a Commit Interval

defined in the Knowledge Module
If a member is not valid, Essbase cancel the entire Chunk of data and
ODI API switches to Cell mode, which sends one individual row to
Essbase per time, until the commit interval ends, making the process
extremely slow
Invalid member combination causes an error in the data load
2015-03-30 18:00:52,234 DEBUG [DwgCmdExecutionThread]: Error occurred in sending
record chunkCannot end data load. Analytic Server Error(1003014): Unknown
Member [ACT001] in Data Load, [0] Records Completed 2015-03-30 18:00:52,234
DEBUG [DwgCmdExecutionThread]: Sending data record by record to Essbase
Essbase Data Load using ODI

In order to prevent Cell mode to happen, we can:

Check the metadata before the load process to guarantee that all
members are valid
We can do this dynamically and easily using ODI Constraints
(CKM) and Planning Metadata Repository
ODI Architecture

ODI uses Knowledge Modules (KM) to interact with different

technologies
KM Type Description
LKM Loading Loads heterogeneous data to a staging area
IKM Integration Uses a given strategy to populate the target datastore from the Staging Area
CKM Check Checks data in a datastore for errors statically or during an integration process
RKM Reverse- Retrieves the structure of a data model from a database. It is needed only for
engineering customized reverse-engineering.
JKM Journalizing Creates the Change Data Capture framework objects in the source staging area
SKM Data Services Deploys data services that provide access to data in datastores
ODI Constraints (CKM)

ODI Constraints checks automatically if the data inside a column

is valid according to a specific condition
Constraints are used by the CKMs to log all invalid data to a E$
table
Need to validate all metadata columns before load to Essbase to
guarantee that no invalid member will be sent (preventing Cell
mode to trigger)
ODI Constraints (CKM) and Essbase

ODI interacts with Planning/Essbase using diverse KMs

For Planning ODI has only a Load Metadata KM
For Essbase ODI has a load and extract KMs for both Metadata and
Data
ODI Knowledge Modules manipulates data to Planning/Essbase using
Java API
ODI does not have a CKM for Essbase
Data validation needs to happen in the inbound tables, before the data is
sent to Essbase
ODI CKM and Inbound Tables

CKM validation is tied to the DataStore (Inbound tables)

For each inbound Table we need to create a set of constraints to
validate the data
This makes difficult when we have more planning applications to load
data since we will have multiple inbounds jobs
If we have only one generic inbound table that contains one
column for each planning dimension (Distinct of all dimensions
from all Applications) we can have dynamic ODI Constraints to
validate any number of Hyperion Planning applications
Inbound Generic table design

For each dimension column in this table, an

ODI constraint will be created to guarantee that
the data is valid for that dimension
Create the table respecting the Sparse dimensions first and
dense dimension later, this way we can load a entire block
each time
The dense members are set before the data columns to
improve performance
If the job loads an entire year the Period members could go
into columns (Loads an entire block per time)
APP_NAME, CUBE and INTERFACE_NAME is used for
multiple application purposes
ODI Constraints (CKM) Estrategy

In order to validate against Essbase outline, we would need to

extract all outline members first
One ODI interface per dimension
Can be very slow in some large dimensions
Easier if we validate against Planning Repository (RDBMS)
All metadata information from all dimensions is inside the Planning App
repository
A lot faster to extract the metadata since it uses SQL
Can be used even without the need of extracting the metadata
Data Validation Architecture

Option 1:
Inbound table is in the same Database as Planning App Repository or
using DB Link
Uses the Planning Repository SQL inside the ODI Constraints querying the
Planning App repository on the fly
Option 2:
Inbound table in a different Database as Planning App Repository
Needs to export all existing metadata from Planning App Repository to a temporary
table first and then create the ODI Constraints to query this temp table
Planning Repository (HSP_OBJECT)

All Planning App metadata is stored in only one table

Column Name Description

OBJECT_ID Object ID for all objects in planning.

OBJECT_NAME Stores all metadata description in Planning (e.g. Alias, Members)

OBJECT_TYPE Type of the Object (e.g. Entity, Account, Attribute)

PARENT_ID Parent ID of the object. Used for build the parent/child relationship with OBJECT_ID

GENERATION Inform which generation that object belongs.

HAS_CHILDREN Inform if the object has or not a child

Repository Query

All existing metadata can be retrieved by a single query on

HSP_OBJECT

This query will return

all Accounts Hierarchy

If we change the query a little we can bring all members at once

This query will return

all Hierarchies
Planning Repository

Not all members in HSP_OBJECT will be loaded to all Essbase

Cubes
The members Plan Type defines in which Cube it will exist
Each Essbase Cube is represented by a Plan Type in Planning
Each Planning member may belong to multiple Plan Types
Data load occurs in one cube at a time
Thats why we must find out which Plan Types a member
belongs in order to validate the data for the correct cube
Planning Repository (HSP_MEMBER)

The properties of the members are stored in this table

Column Name Description

MEMBER_ID Same as Object ID.
Defines in what Plan Types (Cubes) that member will belong.
In case of a member exists in more than one Plan Type, Planning Sums the value
of all plan types its belongs
USED_IN
Plan Types: 1 2 3 4 5

Used In: 1 2 4 8 16
Planning Repository (HSP_PLAN_TYPE)

This table stores the Plan Types created in the Planning App

Column Name Description

PLAN_TYPE ID of the Plan Type

TYPE_NAME Name of the Plan Type

 Planning Repository

Plan Type Used In For each plan type that the member belongs
Plan Type 1 1 Planning SUMs the Used In
Plan Type 2 2 If a Member exists in Plan Type 1 and 2 Used In 1 + 2 = 3
Plan Type 3 4 All Plan Types = 1 + 2 + 4 + 8 + 16 = 31
Plan Type 4 8 To figure out if a member belongs to a plan type we
Plan Type 5 16 need to verify if its Used In is included in the SUM
Plan Type 5 (Used in 16) exists in 31?
Plan Type 1 (Used in 1) exists in 30?
Which Plan Type exists in 20?
Planning Repository

MOD returns the remainder

(modulus) of argument 1 divided
by argument 2
If MOD of Used In per actual Plan
Type * 2 is >= actual Plan Type then
it exists in the SUM of Plan Types.
Plan Type 5 exists in 31
Plan Type 1 not exists in 30
Plan Type 3 and 5 exists in 20
Repository Query

All necessary validation data will come from these three tables
A member that belongs to two Plan Types will appear two times
MOD technique is used to discover which Plan Type that member
belongs to and OR is used to combine more than one Plan Type

This query multiplies all

members depending of
how many Plan type it
belongs
Planning Repository

Another way to select a

Plan Type is using
BITAND
BITAND computes an
AND operation of the bits

Plan Type Used In Binary

Plan Type 1 1 00001
Plan Type 2 2 00010
Plan Type 3 4 00100
Plan Type 4 8 01000
Plan Type 5 16 10000
ODI Constraints (CKM) Implementation

Three types of ODI constraints:

Key: Checks if there are duplicated records or NULL
values in the data
Reference: Checks the relationship (Foreign key) between two tables.
Both tables needs to be reverse-engineered inside ODI models
Condition: Checks for any other specific rule using custom queries
Better to use Condition:
No additional ODI models required (just the inbound table)
More SQL flexibility
Allow complex logic checks
Option 1: Constraints Against Planning App

Validates directly against Hyperion Planning repository

One constraint per dimension
No need of any extra metadata extract
Inbound table and Planning Repository schemas needs to be in
the same database or using DB Link
ODI needs to execute a SQL that access both tables (Inbound table and
Planning repository) in the same connection
Option 1: Constraints Against Planning App
2 changes in the
query for all
constraints
2 ODI variables
that dynamically
switches between
Planning
Applications and
Plan Types
Option 2: Constraints Against a Temporary Table

Need to extract the metadata to a temporary table every time

One constraint per dimension
Inbound table and Planning Repository schemas may exist in
different Databases
Temp table will be created on the same Inbound table schema
Option 2: Constraints against a temporary table
Gets all Dimensions at
same time
1 ODI variables that
dynamically switches
between Planning
Applications
Option 2: Constraints Creation
2 changes in the
query for all
constraints
2 ODI variables
that dynamically
switches between
Planning
Applications and
Plan Types
ODI Constraints Execution

When we open ODI

Operator, each constraint
validation will be shown
in the Control steps
Each step will show how
may invalid members
was removed from the
inbound data
Invalid data is sent to the
E$ error table
Generic E$

With only one inbound generic table, we will have only one
generic E$ table
Stores all the POV and the data that fails the validation
ODI_Cons_name, Interface_Name, App_Name, Cube and
ODI_Sess_NO identifies what was the error, from which package that
error came from and to which Planning app it should have loaded
Generic Structure Benefits

Easy to add new Planning Applications

Generic Inbound tables makes easier to create Generic
components
Send Email
Text Logs (export from E$ table)
Error Handling
Fewer ODI components to Maintain
Overall Architecture
Stage Area Generic Components Planning
Sources
Error Send
Excel Table 1 Handling Email
Cube
1

Oracle Table 2
Inbound Table Cube
2
Sql Inbound
Table 3
Server Generic
Cube
3
Teradata Table 4
E$ Table
Cube
E$ Inbound N
XML Table N Generic
QUESTIONS?
Ricardo Giampaoli TeraCorp
Rodrigo Radtke de Souza - Dell

Thank you!