Category Archives: QlikView

Productivity, Qlik Sense, QlikView

Understanding and Using Subset Ratio

8 Comments

Summary: If you are familiar with subset ratios in Qlik, you may not find much new in this post. But if you are new to Qlik or are unfamiliar with subset ratios in your data model, please read on. 

When loading data into Qlik and building a new data model, inspecting the Subset Ratio of key fields is an important exercise to ensure data quality.

Subset Ratio is displayed in the Preview Pane (Qlik Sense Data model viewer) or Table Viewer (QlikView) when a key field (field linking two or more tables) is selected.

After clicking a key field, in the Preview pane you will see three important numbers:

Total distinct values:  The count of all distinct values for this field (CustomerId) from all tables (Orders” and “Customers”) in the model.

Present distinct values:  The count of distinct values for this field (CustomerId) in the currently selected table (Customers).

Subset ratio: “Present distinct values” divided by  “Total Distinct Values”.  What percentage of total field values are represented in this table?  In this case, 100% of all CustomerId values in the data model are represented in the Customers table.  This is good. We will typically expect to see 100% Subset ratio in a dimension table.

Let’s take a look at the Subset ratio for the same field in our fact table — the Orders table,

It’s less than 100%.  Our Customer table is our “customer master” and represents all of our potential customers.  Our Orders table represents a limited time period, perhaps 12 months.  Only 44 distinct customers, or 22%,  are represented in the set of Orders we have loaded.

Less than %100 Subset ratio is a normal condition for a Fact table.  If we don’t want to include Customer data for those customers who have no orders, we can filter the Customer load with a “Where Exists(CustomerId)” clause.

So far we’ve seen “normal” subset ratios.  Let’s look at some exceptions.  What does it mean when the dimension table (Customers) has less than 100% subset ratio?

It means we have an order(s) that has no link to a Customer row.  That’s a data quality problem. In the example above we can see that we have 1 missing CustomerId (201 – 200).

Why do we have a missing Customer?  We would have to dig into the data to find out why.  It could be that we have loaded historical orders and “inactive” customers are archived from the Customers table.  It could be that we have some bad data due to a bug.  We have to analyze the data and decide on the best path to remediate.

By the way, what is the specific value(s) of the missing CustomerId?  A simple way to make this determination is to create a table chart with two columns — The key field and a field that has 100% density (every row has a value) from the Customers table.  I’ll use the CustomerName field. Sort the table by CustomerName and the key value in question will show at the bottom of the table with a null value for CustomerName.

What does it mean when the sum of the subset ratios for two tables equals 100%? It means there are no matching values between the two tables.   This can happen for instance when the keys come from two different systems that use slightly different nomenclature.  Perhaps in your ERP all ProductId values start with “P” but in the spreadsheet that someone provided for additional part info the “P” is excluded because none of the humans use the “P” when identifying parts.

Examining Subset ratio as you build up your data model is an important quality step.  Validating the quality of your data model will make the process of creating visualizations go much smoother.

-Rob

 

Share
Qlik Sense, QlikView, Scripting

Loading Varying Column Names

11 Comments

Summary:  A script pattern to wildcard load from multiple files when the column names vary and you want to harmonize the final fieldnames.  Download example file here.

I’m sometimes wondering “what’s the use case for the script ALIAS statement?”.  Here’s a useful example.

Imagine you have a number of text files to load; for example extract files from different regions.  The files are similar but have slight differences in field name spelling.   For example the US-English files use “Address” for a field name, the German file “Adresse” to represent the same field and the Spanish file “Dirección”.

We want to harmonize these different spellings so we have a single field in our final loaded table.  While we could code up individual load statements with “as xxx” clause to handle the rename, that approach could be difficult to maintain with many variations.  Ideally we want to load all files in a single load statement and describe any differences in a clear structure.  That’s where ALIAS is useful.  Before we load the files, use a set of ALIAS statements only for the fields we need to rename.

ALIAS Adresse as Address;
ALIAS Dirección as Address;
ALIAS Estado as Status;

The ALIAS will apply the equivalent “as” clause to those fields if found in a Load.

We can now load the files using wildcard “*” for both the fieldlist and the filename:

Clients:
LOAD *
FROM addr*.csv (ansi, txt, delimiter is ',', embedded labels, msq)
;

It’s magic I tell you!

What if the files have some extra fields picked up by “LOAD *” that we don’t want?  It’s also possible that the files have different numbers of fields in which case automatic concatenation won’t work.  We would get some number of “Client-n” tables which is incorrect.

First we will add the Concatenate keyword to force all files to be loaded into a single table.   As the table doesn’t exist, the script will error with “table not found” unless we are clever.  Here is my workaround for that problem.

Clients:
LOAD 0 as DummyField AutoGenerate 0;
Concatenate (Clients)
LOAD *
FROM addr*.csv (ansi, txt, delimiter is ',', embedded labels, msq)
;
DROP Field DummyField;

Now let’s get rid of those extra fields we don’t want.  First build a mapping list of the fields we want to keep.

MapFieldsToKeep:
 Mapping
 LOAD *, 1 Inline [
 Fieldname
 Address
 Status
 Client
 ]
 ;

I’ll use a loop to DROP fields that are not in our “keep list”.

For idx = NoOfFields('Clients') to 1 step -1
  let vFieldName = FieldName($(idx), 'Clients');
  if not ApplyMap('MapFieldsToKeep', '$(vFieldName)',   0) THEN
    Drop Field [$(vFieldName)] From Clients;
EndIf
Next idx

The final “Clients” table contains only the fields we want, with consistent fieldnames.

Working examples of this code for both  Qlik Sense and QlikView  can be downloaded here.

I hope you find some useful tips in this post. Happy scripting.

-Rob

 

 

Share
QlikView, Scripting

Loading Variables From Another QVW

9 Comments

I just read a good post by Kamal Kumar Sanguri on QlikCommunity.  Kamal’s post reminded me that managing variables in QlikView has always presented some challenges and over the years various techniques and code snippets have been shared to address those challenges.

Most folks quickly find that maintaining variables in external files  loaded with a script loop is a good approach and resolves common concerns regarding shareability, dollar sign escaping and so on.

Sometimes you encounter a need for an adhoc import or export of variables. Kamal’s post offers some useful code snippets for that.  Several years ago my colleague Barry Harmsen  wrote a post on QlikFix.com that shows some useful macros to manipulate Variables.  Barry and I  subsequently collaborated on a desktop utility that handles both import and export from a menu.  While the download link on the blog is dead, I’ve reposted the utility on the QlikViewCookbook Tools section for download.

Kamal said that there is no way to directly load variables from one QlikView document to another.  That got me to thinking.  It is possible, but gee, no one has ever asked for it.

What’s the use case?  When I want to do an adhoc copy, I use the desktop utility referenced above.  I have seen a number of customers who generate complex calendars into QVDs followed  by  generation of variable for use with the calendar.  Calendar QVD consumers incorporate the variable generation logic with an include file. It works.

Would it be any better if calendar consumers loaded the variables directly from the calendar generating QVW?  I think not because there is possibility of a mismatch between the QVW source and the Calendar QVD.

All that said, maybe you have a use case for loading variables from a QVW?  No one asked, but for the record here is the script to load variables directly from another QVW.

VariableDescription:
LOAD 
 Name,
 RawValue
FROM [..\..\data\StudentFile.qvw] 
(XmlSimple, Table is [DocumentSummary/VariableDescription])
Where IsConfig = 'false' and IsReserved = 'false' // Exclude system vars
// Any addtional filtering here
;

FOR idx=0 to NoOfRows('VariableDescription')-1
 LET vVarname = Peek('Name',$(idx),'VariableDescription');
 LET [$(vVarname)] = Peek('RawValue',$(idx),'VariableDescription');
NEXT idx

SET idx=;
SET vVarname=;
DROP Table VariableDescription;

Happy Scripting!

-Rob

 

 

Share
QlikView, Visualization

Distribution Plot in QlikView

2 Comments

Qlik Sense added a Distribution Plot visualization in the June 2017 release.   QlikView does not have a specific chart type for distribution plot, but you can achieve the same with a scatter plot.

The trick is to set the Y value (Expression #2) to a constant value such as “1”.  Here’s a distribution of Life Expectancy by Country (source: WHO 2017).

Dimension: Country
 X-Axis: =[Life Expectancy]
 Y-axis: =1

It works, but it’s difficult to understand how many points overlap.  You can switch the Style to the outlined ball similar to Qlik Sense and that helps.

I find a more effective technique is to add some transparency into the color.  Overlapped points will result in a darker color.

You can also highlight points using set analysis or alternate states.

 

Adding reference lines such as  Quartiles can provide additional understanding.

To add a second dimension e.g.  “Sex” (values: Male,  Female, Both)  replace the fixed Y-axis expression with an expression that generates an index number for the values.

=Dual(Sex, FieldIndex('Sex', Sex))

That will assign Y-values 1,2,3 to the Sex values.  The Dual() will ensure the text value will show in the popup. The Y-axis  will still display a numeric value so I’ve hidden the axis.  That leaves us without labels for the three lines.  We can either create labels using text-in-chart or use a color coding scheme.

Distribution plots can be oriented vertically by using a fixed X-axis. If you’ve used my ScriptRepository tool, you may recognize that the search results scroll-guide (the yellow dots) are a narrow scatter plot.

-Rob

Share
Qlik Sense, QlikView, Scripting

LET, SET, Quotes

6 Comments

Summary: In Qlik script SET is often a better choice than LET, even when the value contains quotes. 

I sometimes see the LET script statement used when SET would be syntactically  easier and more readable.

A brief review:  SET assigns the given parameter as-is to the variable,  LET treats the parameter as an expression and assigns the evaluated result to the variable.

SET x = 1+3;  // x is "1+3"

LET x = 1+3; // x is "4"

I frequently see a variable assignment like this:

LET eSales='sum(Sales)';

eSales stores an expression that will be used later in charts.  It could also be written (simpler in my estimation) as:

SET eSales=sum(Sales);

So far just a matter of style, but the difference becomes clear when we have quotes as part of the string, for example, “Region={‘US’}”.   As LET requires a quoted string,  embedded quotes require some sort of escaping.  In QV10 and earlier, a common way to write this with LET would be:

LET x = 'Region={' & chr(39) & 'US' & '}';

Not real pretty. Many people carry over this style even though QV11 introduced two single quotes to represent an embedded single quote.

LET x = 'Region={''US''}';

Easier to read for sure.  But I think it’s even easier with SET.

SET x = Region={'US'};

That’s it. No special escaping required, just type it as it should be.  What about those quotes? Shouldn’t SET strings be enclosed in quotes?

I find the documentation on SET to be thin, but here is the rule as I understand it.

Single or Double quotes in a SET statement require no special treatment as long as they are balanced (even number of quotes).

SET x = Region={'US'},Product={'Shoe'};  // Valid

SET x = Region={"U*"},Product={'Shoe'}; // Valid

SET x = I won't go;   // Invalid

If the quotes are unbalanced (odd number), then the entire string needs to be enclosed in quotes or brackets.  Use double quotes if we are enclosing single quotes.

SET x = "I won't go";

SET x = [I won't go];

I always favor SET over LET unless I truly want an evaluation.  An exception to this is the string “$(” which will trigger an Dollar Sign Expansion, even in SET.

-Rob

For more on character escaping in Qlik from HIC see https://community.qlik.com/blogs/qlikviewdesignblog/2015/06/08/escape-sequences

Share
Productivity, QlikView, Tools

Mass Editing of QVW Script

6 Comments

Summary: In this post I describe a process using freely available tools to apply changes to all scripts in a set of QVW files.

So you have a lot of QVWs. And now you are asked to identify and make updates to all scripts to support changes such as:

  • A change in QVD or other file naming.
  • Changes in file paths due to a server move or directory restructuring.
  • Updating file paths to use variables.

If you are super cool, all those items are represented by variables, changes are handled by updating a single include file and you can relax and stop  reading this post. For the rest of us, read on.

You can scan and search all your script using my Script Repository tool.  That will allow to identify where changes are required, but do you have time to edit every QVW and make the changes?  Easy enough for a few, but what about when you have dozens?

QlikView Desktop has a useful facility we can leverage for mass changes; the “-prj” folder.  If a folder named qvwname-prj  (case sensitive) exists in the same folder as the QVW, when the QVW is saved, QV Desktop will write a set of text files representing the structure of the QVW to the -prj folder.  One of those files is “LoadScript.txt” which contains the load script.

When QV Desktop opens a file, it checks for the existence of a companion -prj.  If found, it populates the QVW with the contents of the files in the -prj.  If we change one of those files, for example “LoadScript.txt”,  that change will be inherited by the QVW.

Let’s walk thorough a scenario where we can utilize this feature to update the scripts of an entire set of QVWs.  I’ll utilize free tools that will make the process easier.

My sample problem is this: I have inconsistent QVD naming conventions. We’ve decided that “DimCustomer.qvd” shall henceforth be known simply as “Customer.qvd”. I’ll need to update the script that generates the QVD as well as all readers of the QVD.

I will accomplish this update in four steps:

  1. Create -prj folders for all QVWs.
  2. In the “LoadScript.txt” files replace “DimCustomer.qvd” with “Customer.qvd”.
  3. Rebuild the QVWs with the updated -prj.
  4. (Optional) Delete the -prj files.

The sample I’ll use for this post is relatively small to keep the demo simple.  But I’ve used this technique to process hundreds of QVWs at a time incorporating several different script edits.

I have a directory of QVWs that looks like this:

 

 

In the SubFolder “Loaders”, there are additional QVWs.

 

 

I’ll need a -prj folder for each QVW. I  can create the -prj manually, but this is where I can leverage the PrjTool to make life easier.  You can download the PrjTool from the Tools section of this site.  (Note: If you received a copy of PrjTool from the Masters Summit, please download this newer version as it contains important updates.)

PrjTool requires a Directory as input and the selection of one of  three functions:

  • BuildPrj: For all QVW files found in the specified Directory, create a -prj folder.  This includes opening and saving the QVW to populate the -prj.
  • CreateFromPrj: For all -prj folders found in the specified Directory, open and save the QVW to update the QVW with contents of the -prj.  If no QVW exists, a new one will be created.
  • DeletePrj: Delete all -prj folders found in the  specified Directory.

I’ll start by specifying the Directory that holds our QVWs and selecting the BuildPrj function.  Press the Execute button and the script will launch. The execution may take some time as each QVW has to be opened and saved. Good time to go for a coffee.

When the execution completes the log window will be filled with messages listing the -prj folders created by the tool.

 

If we examine the directory again we will see the new -prj folders.

 

Our next task is to edit the LoadScript.txt files. We can use any editor capable of searching and replacing across multiple files.  For this demo I will use the free NotePad++ editor.   From the NotePad++ menu, launch “Search” , “Find in File”.  In the search dialog I specify the Directory  and  the search and replace strings. I’ll also limit the search to the LoadScript.txt files.

 

After pressing “Find All”,  I’ll get a list of search results.  I can double click any of the results to open the file for further examination.

 

When I’m satisfied that I’m going to make the correct updates, I again launch “Find in Files” and press “Replace in Files”  to perform the update.

Now I’ll use the PrjTool again to update the QVWs with the updated -prj files.  I run the tool again, this time selecting the “CreateFromPrj” function.  Again, if you have a lot of large QVWs, be patient while the tool runs.  The resulting log messages will inform me of the updates.

We’re done!  All QVWs now contain the updated load script.  Optionally we can run the tool again with the “DeletePrj” function to delete the generated -prj folders.

You should always perform this kind of mass update activity on copies of QVWs and audit the results.  Also, never use -prj folders in production.  Server reloads do not recognize -prj folders.

-Rob

 

Share
QlikView, Visualization

QV12 Variables with Alt State Fixed

10 Comments

Summary:  QV11 contained an inconsistency in how variables with equal signs were evaluated when using Alternate States.  This has been fixed in QV12.10.  Read on if you want the details. 

QlikView V12.10 includes an important fix to variable evaluation when using alternate states. A quote from the Release Notes:

In QV11.20 the variable was expanded in the first state encountered and this resulted in a random behavior when more than one alternate state was being used. Whereas in version 12 and up, the variable always belongs to a specific alternate state and this results in different behavior.

The random behavior described in QV11.20 has generated several interesting posts to QlikCommunity with responses of “I can reproduce” / “I can’t reproduce” and few clear answers.

I find the problem confusing and interesting enough to warrant an explanation and example beyond the Release Note.

What I am describing in this post only affects variables with a leading “=” in the definition, e.g. “=Sum(Sales)”.  These variables are calculated once in the context of the entire document. They are not calculated per row in a chart.

Let’s consider a variable named “vSumX” with a definition of “=Sum(X)”.  The expression simply sums all selected values of X.   Suppose we have two States in our document — “Default” and “State1”.  There could be two different selections for “X”.  Which set of X should the variable sum?

If we consider the variable definition in isolation, the answer is “Default set”  as there is no set identifier in the expression.  But what if the variable is referenced in an object in State1.  Should the State1 values of X be used?

No matter what you think the rules should be, here’s what was happening in QV11.20.  The variable was expanded (evaluated) in the first state encountered.  First state encountered means first state in the calculation chain, not something the developer directly controls.

Let’s look at some examples.   I’ve created a sample app (you can download here) with three States — Default,  State1 and State2.  The variable “vSumX” is defined as “=Sum(X)”.

With all objects on sheet in the Default state, selections in X would yield results like this.  (Note “$” indicates default state).

The first text box contains the expression “Sum(X)”.  The second text object contains the reference to variable vSumX.  The two values are what we might expect, summing the selected values of X in this state.

Let’s switch to a sheet that contains objects in the state named “State1”.

No selections in X and the first text object shows the expected result.  The second object shows the value of vSumX as previously calculated from the default state.  If we make selections on this State1 sheet, that will cause vSumX to be recalculated and both State1 and the Default sheet will reflect the State 1 number.  Is that correct?  Is it useful?  It’s at least consistent and comprehensible.

My next example is where the aforementioned “random” comes into play.  Let’s put objects from three states on the same sheet.

I’ve selected some values in the Default state of X and the results are what I might expect.  The value of vSumX is calculated from my last selections and the variable value is consistent across objects — there is only ever one value for a variable at a given point in time.

Now I select some X values in State1 and expect to see a new value (19) for vSumX.  But no change! The variable was expanded (evaluated) in the first state encountered which happened to be Default ($).

Now I select some X values in State2.  If the vSumX calc used my last selection I would expect to see 7.  But no, I see 19.  The State1 values were used.  If I repeat the exercise, it may use a different state to calc vSumX.  If you test you may get different results.  In this last example, State1 was used because it was the first state encountered in the calculation chain.  The order is not consistent.  It will be influenced by factors such as number of available processors and the order in which the objects were created.

Now that we’ve established that QV11,20 is broken in this regard, how was it fixed in QV12.10?  Simple.  QV12 uses  set identifiers as specified in the expression, without inheritance.

=Sum(X)

will use the Default State as there is no identifier.   If you want to Sum from a specific state, use it in the expression:

=Sum({State1} X)

Variables do not belong to any State.  Aggregation functions used in a variable may specify a Statename, just as chart expression do.  The difference is that the absence of a set identifier in a chart expression means “inherit the state from the containing object”.  In a “=” variable, no set identifier means “use the default state”.

A reminder that end of standard support for QV11 comes on Dec 8, 2017.  If you haven’t yet upgraded to QV12.10, I encourage you to do so.  Download my  QV12 Upgrade Considerations Doc as part of your planning process.  Feel free to contact me if you want some assistance with your upgrade.

-Rob

Update:  Qlik has extended support for QV11.20 to March 31, 2018. 

Share
Performance, QlikView, Tools

Document Analyzer Batch Analysis

13 Comments

I’ve received several requests to provide a batch interface to the popular QV Document Analyzer tool that will provide for analyzing multiple QVWs with a single command.  It’s now available for download here.

The script is a windows cmd file.  Because many browsers block download of cmd files, I’ve provided it with a “txt” extension. Rename to “DABatch.cmd” after downloading.

The usage from the command line is:

DaBatch.cmd somedir
 where “somedir” is a directory of QVWs to be analyzed.   Each QVW in the directory will be processed by Document Analyzer and the results will be saved for later review.
Before running, there are a number of configuration variables in DABatch.cmd you will want to review and modify as necessary.

 

REM *** Path to QV.exe executable ***

SET qvDir=C:\Program Files\QlikView\Qv.exe

This is location of the QV Desktop executable. The provided value is the default location for most users and is typically appropriate as-is.

REM *** Path to DocumentAnalyzer.qvw. Note that v3.6 or later is required! ***

SET DaPath=DocumentAnalyzer_V3.6.qvw
Where is the Document Analyzer.qvw to be found?  Note that DA V3.6 or later is required by DABatch.

 

REM *** Directory to store DocumentAnalyzerResults QVDs and QVWs. Will be created if it doesn't exist *** SET DaResultsDir=C:\temp\MyDaResults
Specify the directory where analysis results will be saved.  If this directory does not exist, it will be created.

 

REM *** Should the analyzer results be stored in a QVD (YES/NO)? ***
SET SaveResultsInQVD=YES
Do you want to save the DA results in a QVD for later analysis by the DaCompareTool.qvw?  The default of “YES” is usually appropriate here.   QVD result files include a timestamp so you will always get new files for each run. Change to “NO” if you don’t want result QVDs.

 

REM *** Should the analyzer results be stored in a QVW (YES/NO)? ***

SET SaveResultsInQVW=YES
If “YES”, a DA QVW will be saved for each analysis and named “DocumentAnalyzer_your_qvwname.qvw”.  If a file exists with this name, it will be overwritten. If you don’t want individual DA QVWs, change this variable to “NO”.

 

After launching DABatch, you will receive one prompt:
Analysis Title? <ENTER> for 'Baseline'
The prompt is requesting a title to be assigned to the Result QVDs that will be consumed by DaCompareTool.  To accept the default of “Baseline”,  press <Enter>.  Otherwise type a new value and press <Enter>.


If you have set “SET SaveResultsInQVD=NO” as a configuration option, the title value is irrelevant.  (Perhaps I should not prompt in that case; next version?).


While the script is running Document Analyzer windows will be launched for each QVW and progress message displayed.  It’s best to keep your hands off the keyboard to get proper timings.  Good time to get that coffee.

When execution is complete you’ll see a summary message.
*****************************************************
Batch Analysis complete. 3 QVWs analyzed.
*****************************************************


You can now review each”DocumentAnalyzer_your_qvwname.qvw” file or load the result QVDs into DaCompareTool.qvw for comparative analysis.


Please let me know in the comments section if you can think of enhancements that support your use case for DA batch analysis.


-Rob
Share
QlikView, Scripting

QV12 REM Logging Change

5 Comments

Summary: QV12 no longer prints the REM statement to the Document Log.

In QV11 “//” and “/*” comments do not appear in the Document Log, but “REM” comments do appear in the log.  I found REM useful to provide some documentation in my logfile or record which branch was taken in an IF-THEN-ELSE.

QV12 has changed the logging of REM.  The REM statement will now appear in the log obfuscated as a series of asterisks  For example, the statement

REM  Beginning of weekly load;

will appear in the log as

2017-05-18 14:24:34 0005 *** *** ******

My understanding is that the change was made to support security standards, as someone could REM a CONNECT string or other sensitive data thereby exposing it in the log.

To provide log documentation, use the TRACE statement instead.  I don’t find TRACE as pretty as REM because TRACE generates double lines, but it will do the job.

I maintain a collection of upgrade notes for QV12 that you can download here to assist in your planning.  Please do read all the Qlik doc — release notes and help site — as well.

-Rob

 

Share
Qlik Sense, QlikView, Tools, Uncategorized

QlikView to Qlik Sense Convertor

7 Comments

Are you migrating QlikView Apps to Qlik Sense?  Have you tried the new QlikView Convertor tool in QS 3.2? 

The QV Convertor tool is available from the Dev Hub in QS 3.2+.  It’s a pretty slick tool that converts QV Variables, Dimensions and Expressions to Master items and converts QV charts to Master QS Visualizations.  It does not attempt to replicate the sheets,  but expects you to place the visualizations on sheets yourself as required.

It’s a very useful tool with a good UI that allows for filtering and limiting what gets converted.

At the Atlanta Qlik Dev Group meeting on July 13 I’ll be demonstrating the tool and presenting some tips and considerations for doing conversions.   They’ve given me two hours (!) to speak so I’ll be covering several other topics as well.

-Rob

Share