All posts by Rob Wunderlich

A Common SSE Plugin Project

Summary: I introduce qcb-qlik-sse, a community Server Side Extension to share custom Qlik functions. 

At the Masters Summit for Qlik, I dive into several different methods of creating reusable script and custom functions.  In QlikView we have the ability to write custom functions using VbScript/Jscript in the qvw Module.

Custom functions have been useful for things like regular expressions, geo calculations, url encoding, encryption and others.  I’ll call them “edge functions” — some of us need them, some of us don’t.

Qlik Sense does not have the module facility. How can we satisfy the requirement for custom function in Qlik Sense?  The Server Side Extension (SSE) facility can fill the need and is available to both Qlik Sense and QlikView.

An SSE Plugin runs as a separate task and provides communication with a Qlik Script or chart Expression via a TCP port. The same SSE Plugin can serve both QS and QV.

Anyone can write an SSE. The Qlik team provides the SSE base and you write a plugin that wires your new functions to Qlik. The new functions can be used in both Script and Charts.  A number of plugins have already been produced.

SSE seems to be the ideal place to provide a collection of edge functions.  Rather than a bunch of one-offs, I’m thinking a good idea would be to pool resources into a single effort that could be shared, much like QlikView Components did for Script.

I’ve implemented this idea as qcb-qlik-sse.  This server uses as it’s base the qlik-sse package created by Miralem Drek.

qcb-qlik-sse is written in javascript and runs in node.js.  At startup, the server scans it’s “/functions” directory and discovers what functions are available.  The general idea is that you can add new function by creating a new js file.  You can remove function you don’t want available by deleting the corresponding js file.

See what functions I’ve already implemented in the doc here. I’ve also provided an example qvf and qvw that exercise the functions.

If you want to try it out,  download the project and define the plugin to QS or QV as documented here.  You will also need node.js 8 or later installed.

Defined functions will show up in the suggestion list in both the script and expression editors.

 

If you want to add functions, some javascript skills are required.  Follow the directions in the readme and submit a PR.

I’ve labeled the project as “experimental” at this stage because I anticipate there could be some significant restructuring as I get feedback.

Let me know your ideas and if you find this useful!

-Rob

Want to kick the tires on reusable code and make your Qlik team more efficient? Come to the Masters Summit for Qlik, a three day advanced training event for Qlik Developers. 

 

 

Share

Script Interpretation and Evaluation

Summary: Qlik Script is both interpreted and evaluated. Understanding the meaning of these terms is useful in writing advanced script and understanding why some script “tricks” work and others don’t.

In my Advanced Scripting session at the Masters Summit for Qlik, I sometimes begin with the question “What does $() do in script?”.  A common answer I hear is “it causes the variable contents to be treated as an expression and evaluated”.  While in some cases this is the practical result, the answer is incorrect.

$() “Dollar Sign Expansion or DSE”, which takes place during interpretation, replaces the variable name with the value of the variable.  This process is called “expansion” or “substitution”. The new value may subsequently get evaluated as an expression during the evaluation phase, but only if an expression is expected in that specific script statement.  Let’s take a look.

SET vVal = 1 + 1; // expecting "1 + 1"
LET v1 = $(vVal); // expecting "2"
SET v2 = $(vVal); // expecting "1 + 1"

$(vVal) will substitute the contents of the vVal variable which is the string “1 + 1”.   After execution, we expect v1 to be “2” because the behavior of a LET statement is to evaluate what is to the right of the equal sign as an expression.  We expect v2 will be “1 + 1” because the documented behavior of the SET statement is to treat the value to the right of equals as a literal string, not an expression.

The individual pieces of a script statement are carved up by the script parser into “symbols” or tokens. The script language definition defines what types — Expressions, Literals, Numbers, etc. — are valid for each symbol.  Only symbols defined as Expressions will be evaluated as such.

The TRACE statement expects a String as it’s symbol.  We can’t calculate a dynamic value within the TRACE statement itself.  We must calculate the value into a variable and then reference the variable like this:

LET vRows = NoOfRows('Orders');
Trace Orders has $(vRows) rows;

It would be useful if script supported the “$(=)” DSE form, like Set Analysis. Then we could do something like this.

Trace Orders has $(=NoOfRows('Orders')) rows;

Simple enough so far.  Let’s look at some more subtle examples.  A user on Qlik Community posted  a requirement to dynamically form several fieldnames in a load statement based off the current date.  She wanted to load a specific field as ‘Month_MMM’ where MMM is the current month-1, another field as current month-2 and so on.  She created a reusable (good idea) variable-with-parameter to create the names:

SET vMonthFieldname = 'Month_' & Date(AddMonths(today(),$1),'MMM');

If called in July as $(vMonthFieldname(-1)) the expected return value would be “Month_Jun”.  The LOAD statement would look something like:

LOAD
  Key,
  field1 as [$(vMonthField(-1))],
  field2 as [$(vMonthField(-2))]

This script executed without error, but to her surprise the final table had unexpected field names:

The variable expansion took place during interpretation, returning the expression string that was intended to form the fieldname.  However, the “as aliasname” clause only expects a literal so no evaluation took place and the string was used as-is for the fieldname.  Like the TRACE example, the workaround would be to first build the values  using LET:

LET vMonth1=$(vMonthField(-1));
LET vMonth2=$(vMonthField(-2));
Data:
LOAD
  Key,
  field1 as [$(vMonth1)],
  field2 as [$(vMonth2)]

How about if we used vMonthField  as the fieldref parameter (to the left of the “as” keyword)?

LOAD
  Key,
  $(vMonthField(-1)) as Month1,
  $(vMonthField(-2)) as Month2

The symbol to the left of “as” may be an Expression, so this would generate expected values. (Admittedly, this is not what the poster asked for, I include it only for illustration).

Let’s visit another illustration of interpretation and evaluation.  How many times will this loop execute?

SET vCounter = 1;
Do While $(vCounter) <= 3
  TRACE Counter is $(vCounter);
  LET vCounter = $(vCounter)+1;
Loop

If your answer is “forever”, you are correct.  Looking at the progress window, we can see  the counter increments beyond 3 but the script continues running.

The script execution log is where we can see what script looks like after interpretation and DSE.

2019-07-25 00:48:17 0021 Do While 1 <= 3
2019-07-25 00:48:17 0022 TRACE Counter is 1
2019-07-25 00:48:17 0022 Counter is 1
2019-07-25 00:48:17 0023 LET vCounter = 1+1
2019-07-25 00:48:18 0025 Loop
2019-07-25 00:48:18 0022 TRACE Counter is 2
2019-07-25 00:48:18 0022 Counter is 2
2019-07-25 00:48:18 0023 LET vCounter = 2+1
2019-07-25 00:48:19 0025 Loop
2019-07-25 00:48:19 0022 TRACE Counter is 3
2019-07-25 00:48:19 0022 Counter is 3
2019-07-25 00:48:19 0023 LET vCounter = 3+1

The Do While condition is fixed at “1 <= 3” which will always be true!  According to the documentation for Do While:

The while or until conditional clause must only appear once in any do..loop statement, i.e. either after do or after loop. Each condition is interpreted only the first time it is encountered but is evaluated for every time it is encountered in the loop.

The doc tells us our $() expansion — which is interpretation — will happen only once.  Evaluation, on the other hand, will be done repeatedly.  The correction to the loop will be to remove $() so our statement looks like this:

Do While vCounter <= 3

This is a valid expression and the loop will execute only 3 times.

So where do we need $() in this loop? We need it in the TRACE because we do not have evaluation, only interpretation.  The While condition and the LET symbol both allow Expressions, therefore we can count on evaluation to get the current variable value.

Do While vCounter <= 3
  TRACE Counter is $(vCounter);
  LET vCounter = vCounter + 1;
Loop

I hope this discussion and examples help you to understand script  interpretation and evaluation, especially in the context of $().

Share

Form Input and Commenting in Qlik

A common desire from Qlik customers over the years has been the ability to interactively edit data on a QV sheet and persist the changes in Qlik or maybe write back to another system. As Martin Mahler of VizLib describes it “turning a one-way data street into a two-way street”.

Sometimes the term “writeback” is used to describe the idea of inputting data in a Qlik app and persisting that data — writing back  the data — to some other system.  The new data is saved and shared with other users of the app. Importantly,  in a form/writeback application, the added data is associated with some row of the data model. For example, in a warranty claims app we may have a dropdown that allows the user to categorize each claim.  The assigned category is used in further analysis.

I think of “commenting” as making annotations on a chart, or chart data point, and a given set of selections.  Ideally, comments may turn into a discussion and use some sort of notification mechanism.

These are not pure terms. There are overlaps to be sure.

I’ve seen some interesting bespoke implementations by partners.  There is also a growing list of off the shelf products that enable writeback and or commentary within Qlik Sense or QlikView.

Some products are Qlik Sense only, some work with both QlikView and Qlik Sense.

Most products allow you to create table sheet objects that mix Qlik DImensions and Expressions with additional input fields such as freeform text, dropdowns or checkboxes.  They all persist the data to some type of backend store such as a database.

When evaluating a product for your requirements, here are some items to consider:

  • Do you have requirements for read-only and update users?
  • Are you looking to add additional data in a single chart or do you need to reference the new data from multiple charts?
  • Are you looking to add one to one new data or build complex workflow apps?
  • Does your business objective require structured form data,  free-form commentary or both?

There are an interesting range of products and capabilities out there.  Klikins and Emark Forms for example let you add new fields to a straight table.  One of the more interesting approaches is K4 Analytics, which embeds Excel into Qlik. This provides the full range of Excel formula and formatting functions. You can build some pretty powerful aps this way.

There are products that focus on finance reports. TrueChart creates a set of functional and beautiful IBCS compliant reports along with a nice navigation interface that can be reused throughout the Qlik app.

Both TrueChart and Climber Finance Report support the type of commenting and annotations user require in finance reporting.  I’m excited that the Climber commenting is being expanded and released as a generic commenting & collaboration product by VizLib. Qommentary is another global commenting solution.

Here in no particular order are some Writeback / Commenting products I’m aware of. The headings are links to the vendor site.

I’m sure I’ve missed some products.  Feel free to leave me a comment if you have something to share.  It would also be good to hear about use cases where you have found value in implementing an input/writeback solution.

Inform Write 

QlikView and Qlik Sense

K4 Analytics

QlikView and Qlik Sense

TrueChart

Qlik Sense

Emark Forms

Qlik Sense

Klikins

QlikView and Qlik Sense

Pomerol Writeback

Qlik Sense

Qommentary

Qlik Sense

VizLib 

Qlik Sense

 

Share

Install QS Document Analyzer on Server

Some people wish to run QS Document Analyzer (QSDA) on the Qlik Sense Enterprise server rather than Qlik Sense Desktop.  While the current installer only supports desktop, you can manually install to the server.

QSDA installs three components; a connector, an application (qvf) and an extension.

Install the latest QSDA on a desktop machine to unpack the components.  You don’t even need QS Desktop installed.  Take note of the folders where the components were installed.

Using the QMC,  upload the qvf application to the server.  Zip up the qsda-ribbon extension folder and upload as an extension through the QMC.

Copy the QsAppMetadataConnector folder from the install folder to this folder on the server:

C:\Program Files\Common Files\Qlik\Custom Data

If you are running a multi-node cluster, the connector must be installed on the central and rim nodes.

That’s it!

 

Share

Masters Summit Fall 2019

Fall 2019 dates and location for the Masters Summit for Qlik have been announced .  Our goal at this  hands on education event is to “Take your Qlik skills to the next level”  to make you more productive and increase the business value of your QlikView or Qlik Sense applications.

When registering for either location, choose one of two tracks:

Our traditional “Qlik Sense and QlikView” track is designed for Qlik Developers who create applications using the out-of-the box clients: Qlik Sense Hub or the QlikView Desktop. Through lecture,  hands on activities and takeaway code samples,  the Summit will expand your knowledge with advanced techniques and a deeper understanding of the core skills required in all Qlik application development:

  • Data Modeling
  • Advanced Scripting
  • Advanced Aggregation & Set Analysis
  • Visualization.
  • Performance

See the complete agenda here.

Our Qlik API track is designed for Web Developers who create Qlik applications and mashups using the Qlik Engine API, Qlik Core, QAP and custom visualizations.

  • Qlik Core
  • Generic Objects
  • Loading & Visualizing Data
  • Architecture and Performance

The API track is led by Nick Webster and Speros Kokenes. You can see the agenda for the API track here.

Our evening guest speakers, networking events and optional lunchtime lectures fill out the program with additional content and lively discussion.

Our panel of six presenters are well known as authors, educators,  Qlik experts and members of the Qlik Luminary and MVP programs.

Learn more about determining if the Summit is right for you and choosing a track in our Frequently Asked Questions.  If you have any unanswered questions or want to learn more, reach out to registration@masterssummit.com .

I hope you can join us in Amsterdam or Washington D.C. to take your Qlik skills to the next level.  Event details and online registration.

See you there!

-Rob

Share

Catwalk — The Alternative Data Model Viewer

Summary: I introduce “catwalk”, an alternative data model viewer for Qlik Sense from the Qlik oss team.  Don’t like reading?  Go to https://catwalk.core.qlik.com/ and give it a spin. 

I’ve been having a lot of fun with catwalk, a fairly new tool from the Qlik oss team.   I call catwalk an “alternative data model viewer” for Qlik Sense.

I’m going to start by showing a few screenshots and then tell you how to get started using catwalk.  It’s easy to try out.

After selecting an app to view, you’ll get a graphical table & field layout.  In addition to a visual depiction of linkage, you’ll get some rich information about cardinatility, relationships  and some nice explanations of subset ratios.

You can also make field selections and see how those selections impact the other tables.  A nice little tool in the lower right corner lets you build hypercubes (straight tables) on the fly to visualize aggregations.

I’m not going to tour all the features here because the first time you enter catwalk you’ll be offered a walkthrough guide.  I highly recommend you take this brief guide.  You can return to the guide at any time from the … menu in the upper right.  A tip on the guide: When it says “you can do X, try X” it won’t let you continue until you try actually try X.  Clever.

 

So how do you get access to all this goodness?  Go to the github page https://github.com/qlik-oss/catwalk for instructions.  Don’t like reading instructions?  Make sure your QS Desktop is started and go to the hosted version at  https://catwalk.core.qlik.com/.  Have fun.

The connection to your Qlik Sense server is from your local browser. No data is passed to the server hosting catwalk.

A really cool way to invoke catwalk is to set up a bookmark with the javascript shown here https://github.com/qlik-oss/catwalk/tree/master/bookmark.  Click the bookmark while in any app in the hub and you’ll open catwalk on that app.   Simple.

So while catwalk may have been conceived  as a data model explorer for Qlik Core (which has no built-in viewer) it’s just as valuable for Qlik Sense Enterprise or Desktop.

Have fun!

Share

Understanding and Using Subset Ratio

Summary: If you are familiar with subset ratios in Qlik, you may not find much new in this post. But if you are new to Qlik or are unfamiliar with subset ratios in your data model, please read on. 

When loading data into Qlik and building a new data model, inspecting the Subset Ratio of key fields is an important exercise to ensure data quality.

Subset Ratio is displayed in the Preview Pane (Qlik Sense Data model viewer) or Table Viewer (QlikView) when a key field (field linking two or more tables) is selected.

After clicking a key field, in the Preview pane you will see three important numbers:

Total distinct values:  The count of all distinct values for this field (CustomerId) from all tables (Orders” and “Customers”) in the model.

Present distinct values:  The count of distinct values for this field (CustomerId) in the currently selected table (Customers).

Subset ratio: “Present distinct values” divided by  “Total Distinct Values”.  What percentage of total field values are represented in this table?  In this case, 100% of all CustomerId values in the data model are represented in the Customers table.  This is good. We will typically expect to see 100% Subset ratio in a dimension table.

Let’s take a look at the Subset ratio for the same field in our fact table — the Orders table,

It’s less than 100%.  Our Customer table is our “customer master” and represents all of our potential customers.  Our Orders table represents a limited time period, perhaps 12 months.  Only 44 distinct customers, or 22%,  are represented in the set of Orders we have loaded.

Less than %100 Subset ratio is a normal condition for a Fact table.  If we don’t want to include Customer data for those customers who have no orders, we can filter the Customer load with a “Where Exists(CustomerId)” clause.

So far we’ve seen “normal” subset ratios.  Let’s look at some exceptions.  What does it mean when the dimension table (Customers) has less than 100% subset ratio?

It means we have an order(s) that has no link to a Customer row.  That’s a data quality problem. In the example above we can see that we have 1 missing CustomerId (201 – 200).

Why do we have a missing Customer?  We would have to dig into the data to find out why.  It could be that we have loaded historical orders and “inactive” customers are archived from the Customers table.  It could be that we have some bad data due to a bug.  We have to analyze the data and decide on the best path to remediate.

By the way, what is the specific value(s) of the missing CustomerId?  A simple way to make this determination is to create a table chart with two columns — The key field and a field that has 100% density (every row has a value) from the Customers table.  I’ll use the CustomerName field. Sort the table by CustomerName and the key value in question will show at the bottom of the table with a null value for CustomerName.

What does it mean when the sum of the subset ratios for two tables equals 100%? It means there are no matching values between the two tables.   This can happen for instance when the keys come from two different systems that use slightly different nomenclature.  Perhaps in your ERP all ProductId values start with “P” but in the spreadsheet that someone provided for additional part info the “P” is excluded because none of the humans use the “P” when identifying parts.

Examining Subset ratio as you build up your data model is an important quality step.  Validating the quality of your data model will make the process of creating visualizations go much smoother.

-Rob

 

Share

Qlik Sense Document Analyzer V1.5

Version 1.5 of the Qlik Sense Document Analyzer (QSDA) tool is now available for download here.

If you are not familiar with QSDA, it’s a free application profiling tool for Qlik Sense that can help you identify items such as unused fields, poorly performing expressions and data model problems.

Here are the significant changes in V1.5:

The Installer allows editing of all install paths and installation will create a log file in the User’s temp directory.

The  Summary ribbon at top of each sheet provides for sheet navigation by clicking a cell.

 

Improved error handling in the connector. Certain types of errors, such as a chart calculation timeout, will not terminate the script. Error count is reported on Summary sheet and detailed error messages can be viewed on the “Extract Log” sheet.

Items that should be considered incomplete due to analysis errors will highlighted in yellow.  This highlighting is a work-in-progress as I discover new possible errors.

Field widths (Symbol width) are no longer estimated! Field sizes are obtained directly from the engine.  Due to this change, this version will report different values for field sizes than previous releases, generally +/- 10%.

The connector edit dialog has been improved.  Error messages now appear properly in their own window and a progress bar displays when the list of applications is fetching.

If you have general usage questions on QSDA, please use the comments section here or Qlik Community.

If you have a bug to report, use the issue tracker.

The only installation supported by the installer is QS Desktop. You can use QS Desktop to analyze applications on a server.  It is possible to manually install on a server by uploading the application (qvf), the qsda-ribbon extension and the connector directory if you have the need and want to give it a shot.  I do plan to create a supported server installer in the future.

Have fun!

-Rob

Share

Web Dev for Qlik Devs in US & UK March 2019

March 2019 brings our popular boot camp course to both the US and UK. Here’s an opportunity to fast track your Qlik team in using Qlik Sense APIs to create extensions, mashups, portal integrations and custom content pages that leverage data and visualizations from Qlik Sense.

In this four-day hands-on course you will learn:

  • The fundamentals of HTML/Javascript/CSS as they apply to QS Development and how to get started with popular frameworks and libraries including bootstrap and enigma.js.
  • Creating Visualization Extensions.
  • The differences and use cases for the various QS APIs e.g. Capability, Visualization, Engine.
  • Key QS API concepts such as the generic object model.
  • Connecting to the QIX engine to retrieve existing content or generate aggregations (hypercubes) on the fly.
  • Visualizing data using third party libraries.

See the course syllabus here.

Students will come away with example code and completed exercises giving them the confidence to move ahead on their own.

No prior experience with web programming is required as the course will provide an intro to web dev technologies and how they are used in Qlik Sense Web Development.

The US session will be taught by Rob Wunderlich and the UK session by Nick Webster.

Even if you don’t have a specific project in mind, I recommend taking this course to understand the power and potential of the QS APIs.  You’ll be surprised and inspired!

Cost for the four days is $2,600 / £ 2,000 and includes all course materials and lunch each day. Register at http://websy.io/training/web-dev-4-day

Questions? Reach out to us. See you there!

Share

Masters Summit Stockholm

The 16th edition of the Masters Summit for Qlik will take place 1-5 April 2019  at the Stockholm City Conference Centre .  Our goal at this  hands on education event is to “Take your Qlik skills to the next level”  to make you more productive and increase the business value of your QlikView or Qlik Sense applications.

 

New for this summit, you can choose from two tracks:

Our traditional track, 1-3 April, is designed for Qlik Developers who create applications using the out-of-the box clients: Qlik Sense Hub or the QlikView Desktop. Through lecture,  hands on activities and takeaway code samples,  the Summit will expand your knowledge with advanced techniques and a deeper understanding of the core skills required in all Qlik application development:

  • Data Modeling
  • Advanced Scripting
  • Advanced Aggregation & Set Analysis
  • Visualization.
  • Performance

See the complete agenda here.

Our new API track, 3-5 April, is designed for Developers who will create Qlik applications and mashups using Qlik APIs, Qlik Core, QAP and custom visualizations.

  • Qlik Core
  • Generic Objects
  • Loading & Visualizing Data
  • Architecture and Performance

The API track is led by Nick Webster and Speros Kokenes. You can see the agenda for the API track here.

Our evening guest speakers, networking events and optional lunchtime lectures fill out the program with additional content and lively discussion.

Our panel of six presenters are well known as authors, educators,  Qlik experts and members of the Qlik Luminary and MVP programs.

Learn more about determining if the Summit is right for you and choosing a track in our Frequently Asked Questions.  If you have any unanswered questions or want to learn more, reach out to registration@masterssummit.com .

I hope you can join us in Stockholm to take your Qlik skills to the next level.  The early bird registration discount is available until 28 February.  Event details and online registration.

See you there!

-Rob

Share