Big Data and ODAG

Qlik Sense and QlikView – On-Demand App
Generation
Version: June 2017
©2017 QlikTech International AB. All rights reserved. Qlik®, Qlik Sense®, QlikView®, QlikTech®, Qlik® Cloud, Qlik® DataMarket, Qlik® Analytics Platform and
the Qlik logos are trademarks of QlikTech International AB which have been registered in multiple countries. Other marks and logos mentioned herein are
trademarks or registered trademarks of their respective owners. Use of any and all of the foregoing is subject to the written approval of Qlik.
Contents
1 Overview 3
2 Use Cases 4
3 On Demand App Generation - Qlik Sense 7
3.1 QMC Controls and Security 9
3.2 Sample Script Details – Changes 10
3.3 Sample Script Details - Process 12
3.4 App Navigation Links 16
4 On Demand App Generation - QlikView 17
4.1 QlikView Architecture Overview 18
4.2 Selection App 18
4.3 Detail App 20
4.4 Custom ASPX using QMS API 23
2
1 Overview
This document provides a technical overview of the various options we have deploying Qlik
Sense and/or QlikView in a “Big Data” environment. The technical focus of the document will
discuss one approach in detail namely “on-demand app generation”, details of other options
namely document chaining and segmentation have been covered extensively in other
documents.
The on-demand app generation approach expands the potential use cases for Business
Discovery, enabling business users to conduct associative analysis on larger data sources. It
provides a shopping list experience to allow users to first select data they are interested in
discovering insights on which interactively an analysis app with full Qlik in-memory
capabilities.
The following part of the paper provides a technical overview of implementing and using “on-
demand app generation” with the built-in capabilities in Qlik Sense and the options available
to deploy with QlikView.
3
2 Use Cases
A determination of which method or indeed methods to apply (as they are not mutually
exclusive) can be thought of at a high level on a few factors, namely data volume and
complexity of analysis required.
4
Below is a pro/con analysis of the various approaches:
Method Details Data Pro Con

Volumes
In Memory Highly compresses 100s millions - No limit on Qlik Limited to memory on single
data into memory. functionality server
Methods for data - Customizable
load can extend this SQL/script generation
even further. - Can be used for non-
SQL sources
Segmentation Users move 100s millions - No limit on Qlik Overt data replication
& Chaining between multiple to billions functionality
related segmented - Customizable
apps (e.g. by SQL/script generation
region). - Can be used for non-
SQL sources
On Demand User selections Multi billions - No limit on Qlik - Multiple apps required
App spawn generation of functionality - Drill paths and app
Generation a filtered data set - Customizable content predefined by
(Built-in) for analysis via built SQL/script generation app developers
in navigation in Qlik - Can be used for non-
Sense sq. sources e.g. SAP
Bex and PLACEHOLDER
function in HANA
- Built in security rules
to control access
On Demand User selections Multi billions - No limit on Qlik - Higher implementation
App spawn generation of functionality time
Generation a filtered data set - Customizable - JavaScript and API
(API’s) for analysis via SQL/script generation knowledge required
multiple API’s in Qlik - Can be used for non-
Sense or QMS API sq. sources e.g. SAP
/EDX in QlikView Bex and PLACEHOLDER
function in HANA
- Mashup creation
possible to have s
single UI experience
5
This document will focus on the in-built on-demand app generation capability and below is a
high-level summary of the approach and how this helps the business user and IT to deliver
value from a Big Data environment:
• Provides a “shopping list” experience allowing users to choose a series of dimensional

filters e.g. time periods, customer segments or products from a “selection” app which
drives the data content of a subsequent detail app.
• Provides full QlikView/Qlik Sense functionality on a relevant subset of the Big Data
store that is hosted in memory delivering associative sub second analysis.
• Provides an extension to customer investment in existing Big Data technology e.g. a
Hadoop accelerator or an in-memory database such as SAP HANA by allowing the app
developer to generate custom logic pertinent to the underlying source.
• Provides governance and control to the IT user as to how large and when a detail app
can be invoked based on data volume or dimensional selections.
6
3 On Demand App Generation - Qlik Sense
Sample applications are installed with Qlik Sense Server 2017 June release and above. The
sample files can be found C:\ProgramData\Qlik\Examples\OnDemandApp\sample include the
following:
• Selection application containing sample flight data showing high level aggregated
metrics and dimensions for selections.
• A detail level app which gets populated with data driven from the selection app.
• Source qvd files which act as the data source for the on-demand apps.
Below is a high-level architecture of the process flow:
1. The user selects dimensional criteria from a selection app containing an aggregated view
of the underlying Big Data source.
2. After the predefined governed limit is reached the on-demand navigation object which
can be restricted using built in security rules becomes active.
7
3. The user can then invoke the population of detail level app which will be published to the
personal work stream or nominated stream and only contain filtered data based on the
dimensional selections in step 1.
1. Data items are selected from charts/filter boxes in the aggregated selection app in
memory which is populated with dimensional data and an aggregated view of the source.
2. Once a governed limit is reached (based on an expression the on-demand button becomes
active
3. The selections have generated a filtered list of associated/selected data items and these
are dynamically passed into one of the detail apps which contain charts and the reload is
initiated
4. The detail app contains SET statements which indicate which fields are used for filtering
and creates an INLINE LOAD statement dynamically with these values.
5. The detail app contains script to convert these into a WHERE clause variable and the app is
reloaded with the where clause and published to a stream with an optional timeout.
8
3.1 QMC Controls and Security
The on-demand capabilities can be enabled and secured via additional options in the QMC
and via additional security rules to provide granular control. The QMC has a menu item:
which can enable/disable the on-demand service and provide a global option to control the
number of apps which can be produced per on-demand link:
Several security rules are available to control the behaviour of the on-demand feature:
CreateOdagLinkUsage Non-anonymous users with read access to the selection app and
read access to the link can create OdagLinkUsages
CreateOdagLinks Non-anonymous users with read access to the ODAG template app
can create links and it is possible to create a link without first
knowing the template app
9
CreateOdagRequest Non-anonymous users with read access to the link can create new
Requests using that link
ReadOdagLinks Non-anonymous users can read ODAG links
ReadOdagLinkUsage Non-anonymous users with read access to the selection app can
read its OdagLinkUsage
3.2 Sample Script Details – Changes
The examples delivered include:
• ODAG Sample Selection.qvf

• ODAG Sample Detail.qvf
• Source qvd files
The “ODAG Sample Selection” app contains an aggregated view of the data and the
dimensions which will be incorporated into the subsequent WHERE clause for the “ODAG
Sample Detail” app.
The “ODAG Sample Detail” app contains the script logic which interprets the dimensional
selections and dynamically formulates the WHERE clause dependent on the selections made.
A few changes are required to the sample script to tailor to your environment, some are
optional and some may not be required at all pending the data source:
10
3.2.1 Change #1
Modify the set statements and references to the columns which will be required to be
included from the selection app into the where clause
Change the column name to be included in SET ORIGIN = ‘’;

the selection filters
Change the column name references and OdagBinding:

LOAD * INLINE [
add a prefix:
VAL
• ods - selected values $(odso_Origin Code){"quote": "", "delimiter": ""}
• odo - associated values ];
• odx - excluded values
• odso - selected/associated
Additionally, the quoting and delimiters
can be specified with the default being a
single quote and comma delimited
Change the source data column name for SET ORIGIN_COLNAME = 'Origin Code';
inclusion in where clause
Finally change the column name in the CALL BuildValueList('ORIGIN','OdagBinding', 'VAL', 39);
CALL statement
3.2.2 Change #2
Add the connection string to source and any dimensional load statements.
3.2.3 Change #3 - Optional
Allows an additional WHERE clause to be specified in addition to the dynamic WHERE clause
generated.
11
3.2.4 Change #4
Add the columns to be included in the on- FOR EACH fldname IN 'ORIGIN','DEST'.........
demand process to the for/next loop, which
will match those specified in the SET
statements
3.2.5 Change #5 - Optional
Create new folder connections to the source files if appropriate.
3.2.6 Change #6/7
Add the relevant load scripts to be used in the apps.
3.3 Sample Script Details - Process
Several script processes are involved to interpret the selections dynamically passed into the
detail app:
3.3.1 Stage 1 – SET Statements
The first stage in the process involves specifies the columns which will be involved in the on-
demand process for inclusion in the subsequent WHERE clause generation. Options exist to
specify the type of selections which will be dynamically passed into the app and the
delimiters:
Clearing any previous selections SET ORIGIN = ‘’;
12
The list of data items to be included will be OdagBinding:
LOAD * INLINE [
generated in an INLINE LOAD statement.
VAL
The columns to be included are defined $(odso_Origin Code){"quote": "", "delimiter": ""}
and the selection state assigned: ];
• ods - selected values
• odo - associated values
• odx - excluded values
• odso - selected/associated
Additionally, the quoting and delimiters
can be specified
Specifying a column name for inclusion in SET ORIGIN_COLNAME = 'Origin Code';

where clause
The next stage of this process calls the CALL BuildValueList('ORIGIN','OdagBinding', 'VAL', 39);
subroutine to construct the INLINE load
Stage 2 – Constructing the INLINE LOAD
This part of the script creates the list of values in an INLINE LOAD which will participate in the
WHERE clause.
Checks the quoting specified in the IF ($(QuoteChrNum) = 0) THEN

previous step
Constructs a list of the values (VAL) from LET LOADEXPR = 'Concat($(ColName),' & chr(39) & ',' &
chr(39) & ') AS CombinedData';
the Odagbinding (INLINE) table and
concatenates them with no quotes
Constructs a list of the values (VAL) from ELSE

LET CHREXPR = ' chr(' & '$(QuoteChrNum)' & ') ';
the Odagbinding (INLINE) table and
LET LOADEXPR = Concat( $(CHREXPR) & $(ColName) &
concatenates them with quotes $(CHREXPR)' & ',' & chr(39) & ',' & chr(39) & ') AS
CombinedData';
13
The concatenated list of values is loaded _TempTable:
LOAD $(LOADEXPR) Resident $(TableName);
into a temporary table to construct the
LET NoOfRows = NoOfRows(‘_TempTable’);
final INLINE LOAD statement IF $(vNoOfRows) > 0 THEN
LET $(VarName) = Peek(‘CombinedData’,0,‘_TempTable’);
END IF
drop table _TempTable;
drop table ’$(TableName)’;
3.3.2 Stage 3 – Constructing the value list
The FOR/NEXT loop constructs the value list of the field selections and calls the
ExtendQVDWhere subroutine to construct the WHERE clause:
Specifying the field list assigned to a variable FOR EACH fldname IN 'ORIGIN',
'DEST'.........
(fldname)
Specifying a new variable (vallist) to check LET vallist = $($fldnname);

length of the fldname list and invoking the
WHEN (IsNull(vallist)) LET vallist = '';
ExtendQVDWhere sub routine if >0 IF len(vallist) > 0 THEN
CALL ExtendQVDWhere('$(fldname)','vallist');
3.3.3 Stage 4 – Constructing the WHERE clause
The value list is passed into the ExtendQVDWhere subroutine which constructs the WHERE
clause to be applied to the Fact table SQL query.
NOTE: The examples delivered are a series of qvd files which do not support IN clauses so the
mixmatch() function is used, the sample script also contains an IN clause example for SQL
compliant sources (ExtendWhere):
14
Specifying input parameters for the sub SUB ExtendQVDWhere(Name, ValVarName)
routine:
• Name = field name from the for loop
• ValVarName = values from field
Creates a variable (T) to construct the on- LET T = Name & '_COLNAME';
LET ColName = $(T);
demand field name with _COLNAME suffix
LET Values = $(ValVarName);
and assigning ValVarName variable to the
Values variable (this allows for the database
column names to be different from their
association Qlik fields).
Checking if any (Values) are available and IF len(Values) > 0 THEN

IF len(WHERE_PART) > 0 THEN
construct the where clause variable using the
LET WHERE_PART = ' $(WHERE_PART) AND
mixmatch() function (WHERE_PART) mixmatch([$(ColName)],$(Values) )';
ELSE WHERE
mixmatch([$(ColName)],$(Values))
ENDIF
ENDIF
15
3.4 App Navigation Links
The on-demand feature has a several properties which can be set to control the detail app
behaviour which include the detail app to navigate to, conditional expression to invoke the
app, retention times, stream and sheet options and row limits:
16
4 On Demand App Generation - QlikView
An example of a QlikView deployment for this methodology is described in the following

section along with an example in this pack. QlikView relies on construction of the script and
EDX tasks for a server based deployment. Qlik Sense makes use of the API’s in the platform to
achieve the same result.
Deployment in a desktop and server environment will use the same methodology but a
different indexing technique, one for the server is described in the flow below:
Selection • Dimensions in list boxes

App • Conditional show applied to a button to limit amount of data which can be selected
• Action to invoke analysis app with parameters
Index • ASPX page invoked with selection criteria/user name

Technique • EDX parameters passed separated in a list and pipe terminated
Detail • Detail app produced by EDX task

App • Contains data slice limited by populating a
WHERE clause with the parameters
The complexity for the server environment is when the user interacts with the apps via a
browser. In a server deployment, only the QlikView Distribution Service - ”Publisher” can
execute the QlikView Script to materialise data into the subsequent QlikView detail app.
The Web server and QlikView server services deliver content from the QlikView app but
cannot modify the data in it. A combination of techniques are required for the user to be able
to influence the resultant detail app content.
17
4.1 QlikView Architecture Overview
The following diagram describes the architecture of an on-demand app generation process:
1. The selection app is populated with dimensional data on schedule.

2. User selects dimensional criteria. After the governed limit is reached an ASPX page is
invoked.
3. QMS API and EDX indexes the analysis app with the most recent data from the Teradata
database with only the data slice relevant to the user.
4. The analysis app deployed to access point with user security.
4.2 Selection App
The concept behind the selection app is to provide the user with a shopping list of dimensions
from which they can select a range of data values (It is recommended to limit the amount of
data with a conditional show calculation on the button which will invoke the reload task).
18
The procedure should follow the steps below:
1. Create a selection app that offers a selection of Dimensions but no detail level
facts, this app typically contains a summary view of the underlying source.
2. Place a button on a sheet. This should be enabled only when sufficient selection
criteria have been met essentially to minimise the reload time based on the data
volumes.
an example of a conditional show condition:
GetPossibleCount( [OrderDate]) * GetPossibleCount( [ProductID]) > 0

AND
GetPossibleCount( [OrderDate]) * GetPossibleCount( [ProductID]) < 20
This would force the user to select at least one value in each of these dimensions
before the button was enabled. They can then control the breadth (Segment
Name) and depth (Dates) of data to be loaded into the analysis app.
3. Assign an action on this button which would invoke an ASPX page, passing it the
selection criteria and the current Username. The selection criteria must be labelled
and delimited.
The example below constructs the EDXparms variable which consists of repeated
instances of the various field names (ProductID, OrderDate and TerritoryID)
combined with a comma-delimited value list with each set delimited by a pipe.
='http://WIN-006FPHODF1N/OnDemand/Default.aspx'
& '&user=' & replace( OSuser(), '\' , '#')
& '&EDXparms='
& if( GetSelectedCount( [Product Name]) >0,

'ProductID=' & concat( [ProductID], ',')
& '~')
& if( GetPossibleCount( [OrderDate]) >0,

19
'OrderDate=' & concat(distinct
date([OrderDate],'YYYY-MM-DD'), ',') & '~')
& if( GetSelectedCount( [Territory Name]) >0,

'TerritoryID=' & concat( [TerritoryID],
',')& '~' )
4.3 Detail App
The detail app will contain the relevant chart and objects the user requires for analysis along
with the script elements which will consume the variables produced from the selection app.
The reload task will begin with an initial value for the variable EDXparms. The script in the app
must parse the fields [delimited by pipe] and value lists [delimited by comma] from EDXparms
then use this to modify the data source query that will be performed by the script.
The modification of the query may be as simple as string replacement (using concatenation)
or could be construction of a WHERE clause for an SQL query.
The value lists could be in different formats for example distinct values or values containing
wildcards, the variable manipulation will cater for that.
For example, a combination of ProductID (with wild cards) and distinct TerritoryID’s could
produce the following string:
EDXparms=ProductID=prod1*,prod2*|TerritoryID=1,2,3
which would need to resolve to a where clause as follows:
WHERE (ProductID LIKE ‘prod1*’ OR ProductID like ‘prod2*) AND (TerritoryID IN (1, 2, 3))
(with appropriate delimiters pending on datatype numeric/string).
An example script is shown below which will produce a list of ProductID’s to pass into a where
clause based on some Product Name selections.
Firstly, the delimiters are set for the fields and values
20
SET FieldDelimiter = '|';
SET ValueDelimiter = ',';
Set the conditions parameter to 0 and populate the field name and field values variables
LET nConditions = 0; // number of conditions in WHERE clause
LET nEDXparms = SubStringCount( '$(EDXparms)', '$(FieldDelimiter)' );
FOR i = 1 to nEDXparms +1;
LET sEDXparm = SubField('$(EDXparms)', '$(FieldDelimiter)', i);
LET sField = SubField( '$(sEDXparm)', '=', 1);
LET sValues = SubField( '$(sEDXparm)', '=', 2);
Enter the logic to parse the Product Name selections
SWITCH '$(sField)'
// Derived Table query upon Product table

CASE 'ProductName', 'ProductID'
IF Len('$(sValues)') >0 THEN
Identify any wild cards to build a LIKE statement to be passed into the where clause if required
// there are Values to search for
IF Index('$(sValues)', '*') > 0 OR Index('$(sValues)', '%') > 0 OR
Index('$(sValues)', '?') > 0 OR Index('$(sValues)', '_') > 0
THEN
// Build a LIKE clause with OR between each term

LET sCondition = '';
LET nValues = SubStringCount( '$(sEDXparm)', '$(ValueDelimiter)' )+1;
FOR j = 1 to nValues;
LET sValue = SubField( '$(sValues)', '$(ValueDelimiter)', j);
IF j > 1 THEN
LET sCondition = '$(sCondition) OR $(sField) LIKE "$(sValue)"';
ELSE
LET sCondition = '$(sField) LIKE "$(sValue)"';
ENDIF;
NEXT j; // value
For multiple values an OR clause is added to the variable

LET sCondition = '($(sCondition))'; // put the OR clauses in brackets
LET sCondition = replace('$(sCondition)', '*', '%');
LET sCondition = replace('$(sCondition)', '?', '_');
If wildcards are not detected a distinct value IN clause will be generated to pass into the
resultant WHERE clause variable
21
ELSE
// Build an IN clause
LET sCondition = '';
LET nValues = SubStringCount( '$(sEDXparm)', '$(ValueDelimiter)' )+1;
LET sValueList = '';
FOR j = 1 to nValues;
LET sValue = SubField( '$(sValues)', '$(ValueDelimiter)', j);
IF j > 1 THEN
LET sValueList = '$(sValueList), "$(sValue)"';
ELSE
LET sValueList = '"$(sValue)"';
ENDIF;
NEXT j; // value
LET sCondition = '$(sField) IN ($(sValueList))';
ENDIF
Populate the variable which will go to make up the first elements of a subsequent where
clause
// replace stringDelimiter with SingleQuote
LET sCondition = replace('$(sCondition)', '"', chr(39));
IF nConditions > 0 THEN // multiple clauses separated by AND condition

LET sWhere = '$(sWhere) AND $(sCondition)';
ELSE
LET sWhere = '$(sCondition)';
ENDIF;
LET nConditions = nConditions + 1;
ELSE
// No search terms
TRACE # No values in field $(sField);
ENDIF;
After the EDXparms have been parsed the modified query can be executed in the analysis
application using the populated WHERE clause variable.
22
4.4 Custom ASPX using QMS API
A button in the selection app will cause a custom page to be invoked, passing it the user
identity and the selection criteria. The following procedure is followed:
•Detail application is deployed on the Qlik Server and referenced in the ASPX
•User name added into the parameters from the Index app and extracted from
Identify the URL
•ASPX clones the detail application template and appends User name to
resultant app
Clone •The app is available for EDX publisher tasks
• An EDX task is constructed with extracted parameters from the URL including
the filters to be applied
•Loop until timeout or task completion
Invoke •Publish the URL to allow user to launch analysis app in access point
23

Big Data and ODAG

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Big Data and ODAG

Diunggah oleh

Hak Cipta:

Format Tersedia

Qlik Sense and QlikView – On-Demand App

Version: June 2017

3 On Demand App Generation - Qlik Sense 7

3.1 QMC Controls and Security 9

3.2 Sample Script Details – Changes 10

3.3 Sample Script Details - Process 12

3.4 App Navigation Links 16

4 On Demand App Generation - QlikView 17

4.1 QlikView Architecture Overview 18

4.2 Selection App 18

4.3 Detail App 20

4.4 Custom ASPX using QMS API 23

Method Details Data Pro Con

• Provides a “shopping list” experience allowing users to choose a series of dimensional

Below is a high-level architecture of the process flow:

ReadOdagLinks Non-anonymous users can read ODAG links

3.2 Sample Script Details – Changes

The examples delivered include:

• ODAG Sample Selection.qvf

Change the column name to be included in SET ORIGIN = ‘’;

Change the column name references and OdagBinding:

3.2.3 Change #3 - Optional

3.2.5 Change #5 - Optional

Create new folder connections to the source files if appropriate.

3.2.6 Change #6/7

Add the relevant load scripts to be used in the apps.

3.3 Sample Script Details - Process

3.3.1 Stage 1 – SET Statements

Clearing any previous selections SET ORIGIN = ‘’;

Specifying a column name for inclusion in SET ORIGIN_COLNAME = 'Origin Code';

Stage 2 – Constructing the INLINE LOAD

Checks the quoting specified in the IF ($(QuoteChrNum) = 0) THEN

Constructs a list of the values (VAL) from ELSE

3.3.2 Stage 3 – Constructing the value list

Specifying a new variable (vallist) to check LET vallist = $($fldnname);

3.3.3 Stage 4 – Constructing the WHERE clause

Checking if any (Values) are available and IF len(Values) > 0 THEN

An example of a QlikView deployment for this methodology is described in the following

Selection • Dimensions in list boxes

Index • ASPX page invoked with selection criteria/user name

Detail • Detail app produced by EDX task

1. The selection app is populated with dimensional data on schedule.

4.2 Selection App

an example of a conditional show condition:

GetPossibleCount( [OrderDate]) * GetPossibleCount( [ProductID]) > 0

& '&user=' & replace( OSuser(), '\' , '#')

& if( GetSelectedCount( [Product Name]) >0,

& if( GetPossibleCount( [OrderDate]) >0,

& if( GetSelectedCount( [Territory Name]) >0,

4.3 Detail App

which would need to resolve to a where clause as follows:

(with appropriate delimiters pending on datatype numeric/string).

Enter the logic to parse the Product Name selections

// Derived Table query upon Product table

IF Len('$(sValues)') >0 THEN

// Build a LIKE clause with OR between each term

For multiple values an OR clause is added to the variable

LET sCondition = replace('$(sCondition)', '"', chr(39));

IF nConditions > 0 THEN // multiple clauses separated by AND condition

Anda mungkin juga menyukai