All posts by Rob Wunderlich

A Journey to My First Published Extension

Like Karl Pover, I’m curious to learn more about writing Qlik Sense extensions and other opportunities to use the Sense APIs.   I’ve created some throwaway examples in class, usually working directly in the Sense/Extensions directory.

I found I reached the point where I wanted to get a bit more serious about the mechanics of writing and maintaining Sense code. This post discusses some of my journey in discovering and implementing a structure for writing and publishing Qlik Sense extensions. (For tutorials on writing the actual extension code see Qlik Help or websy.io)

First, my extension project.  I like the script export/import function in QlikView script editor and have missed this function in Sense.  So I created a extension that provides buttons to Export and Import script to text files.

 

 

 

 

Yes, it’s true — a “Visualization Extension” that visualizes nothing. You can click Export to send the current script to text file, and Import to replace the current script from a text file of your choosing. You can also drop a text file on the Import button.  You can find the extension with download links here.

On to the focus of this post, which is “how-to”.

Naming Conventions

As the extension landscape expands, how many extensions named “Super Bar Chart” will be created and published on GitHub? We need a naming convention that allows everyone to co-exist and avoid collisions.  I adopted the namespace convention of prefixing my assets with my domain name “com.qlikcookbook.com”.   So the formal name of my extension is:

com-qlikcookbook-ExportScript

All my files; js, css, etc are prefixed with this namespace. The “name:” property of the required .qext file provides a friendly name that will display in the Assets panel.  I chose the shorter “Export/Import Script” for the name.

An extension lives inside a larger application and must play well and share with others.  It’s not a good idea to name your html Div “output” as you may collide with others who use the same name.  I used the same prefix for any elements that may have a scope outside my extension, specifically html ids and css class names. The Qlik Help has some recommendations on this topic.

Directory Structure

I looked a a number of different directory structures on github that folks had used for existing extensions.  I settled on the layout recommended by Stefan Walther in the documentation for his sense-go tool.  The design made sense and I was also interested in using the sense-go tool for building.

Loading External Libraries

I used a couple of external libraries in the project to handle the file download and the drag&drop function. Instead of referencing those libraries with html links, I learned how to use require.js, an integral tool in the coding patterns of Qlik Sense.  I also  used require.js to load my css and html files. It’s a great tool.

Building and Deploying

The standard way of writing code is to write your source code in one location and then prep and package the files into an installation or runtime bundle.  There are many advantages to following this pattern.  We also want a way to automatically redeploy the updated code to the Sense desktop or Server for testing, and upload a release package when ready.   As a starter build & deploy process, I chose Stefan Walther’s sense-go tool.  In addition to automating the process, I found  the task chain to be a good knowledge transfer from an experienced Sense developer.

Other Tools

Everyone has a favorite editor, I tried a few on this project.  I found I liked vscode best.

For managing the git repository,  I used GitHub Desktop.  You may like another tool or be a command line fan.

If you are starting out writing extensions, I hope these notes help you out to some direction.

Share

Boston Masters Summit

The next  Masters Summit for Qlik event will be in Boston on October 23-15, 2017.

Designed for Qlik Developers who have basic skills and experience,  the Summit presents three days of intense hands-on sessions in topics such as Advanced Scripting, Data Modeling, Advanced Aggregation and Set Analysis, and Visualization Techniques, applicable to both QlikView and Qlik Sense.

You have some experience with Qlik, have taken the beginning courses. How do you ramp up to create more success with your Qlik program? Learn from seasoned experts, authors and world class presenters Rob Wunderlich, Barry Harmsen, Oleg Troyansky and Bill Lay.

In addition to the hands-on exercises, you’ll come away with many valuable sample files and documents. You’ll also get a chance to meet and network with Qlik Developers from around the world.

Our 2017 program features an expanded Performance Tuning section and additional content specific to Qlik Sense.

In four years over 900 Qlik Developers have attended twelve Summits around the world. Their feedback is overwhelming positive. Read about their experience here.

I hope you can join us in Boston to take your Qlik skills to the next level! Read about the details of registration here.

-Rob

Share

Cookbook Tools Updates

Just a quick note about some recent updates to the Tools available on QlikViewCookbook.com

  • QV Document Analyzer V3.5 
    • Added new computed field, “Expression Table Count” that identifies how many tables are involved in a given expression.  Expressions that use data from more than one table typically run slower then those with all data in a single table.
    • Added “Like Objects Count” attribute for Objects, identifying candidates for linked objects.
    • Bug fixes.
  • Copy Groups Utility V2 allows for copying groups within the same QVW.
  • Script Log Analyzer V1.6 can analyze reload logs from both QlikView and Qlik Sense, Desktop and Server versions.  Interface is available in four languages.

-Rob

Share

Vizlib: Innovation and Agility

Summary:  As a provider of Qlik Sense visualization extensions Vizlib promises to blend the innovation and agility of open source with the reliability of commercial software.. 

The pace of delivery for visualization in Qlik Sense has been disappointing to some.  Thanks to rich open APIs in QS, much of the slack has been taken up by the community in the form of open source visualizations as seen on branch.qlik.com.   There you can see some remarkable innovation and responsiveness to community requests.

Should you use open source visualizations?  An open source extension may provide just what you need in your project.  But you should consider the unsupported nature of open source and evaluate the risk and consequences of the visualization possibly failing at some point.

When I worked as  a Java developer  my team used the open source Eclipse IDE as our main tool.  We also used a dozen or more open source plugins.  As our plugin library grew, we found that testing and updating plugins between releases was taking an inordinate amount of time, sometimes making us afraid to upgrade the base Eclipse product.   We  turned to a vendor and purchased a bundled version of Eclipse with dozens of plugins tested, supported and verified. Problem solved.

Vizlib is a new company is promising to blend the innovation and agility of open source with the commercial support that many customers demand.  Vizlib is partnering with the best extension authors to produce a library of fully supported high function visualization extensions. Check out some of what they have published so far:

  • A Table object that delivers the functionality of  a QlikView table and much more.
    • Rich Formatting Options just like in Excel/QlikView
    • Conditional show & hide of columns
    • Dynamic Labels
    • Minicharts & Progress Bars

  • An Advanced Text Object that provides the full functionality of the classic QlikView text object  (including actions!) plus additional functionalities like HTML code and icons. 

To date Vizlib has released five visualizations with more on the way. Check out what’s available on the Vizlib download page.  A free trial license is offered that allows you to try any of the extensions in your own installation.  Take a look!

-Rob

Share

Dual Storage vs Dual Behavior

Summary: The Dual() function stores both string and numeric representations of a value.  “Implied Duals”  such as Dates, store only the numeric portion and apply the string mask as needed. In some circumstances such as un-optimized QVD loads, implied duals can get converted to “full duals” using storage unnecessarily.

In QlikView and Qlik Sense you can create a Dual field using the Dual() function such as:

if(ShipDate = OrderDate, Dual('Yes',1), Dual('No', 0)) as SameDayShip

Dual fields have both string and numeric representations and Qlik is smart about using the correct representation based on context.

In a listbox or filter, SameDayShip will show the string values as:

Yes
 No

We can also write expressions such as:

Sum(SameDayShip)

which will smartly and automatically use the numeric value of SameDayShip.

Internally, the values will be stored in the symbol table like this:

Y e s 1
N o 0

The numeric portion, 1 or 0 in this case, will always occupy 8 bytes. The average symbol length will be 10.5 —  (11 + 10) / 2 values. You can display the symbol length by using a tool like Document Analyzer.

What about Date() or Num() fields, which are also Dual fields?  When properly scripted, these are what I call “Implied Dual fields”. They have dual behavior, but do not occupy the full dual storage.

Dates are represented as the number of days since Dec 31, 1899.  Today’s date (March 12, 2017) number is 42806.  A properly optimized date stores only the numeric value and does not store the  string value. Instead , the format mask is stored once as an attribute of the field.

Format: M/D/YYYY
ShipDate
42804
42802
42800

On demand, when the string representation is required (like in a listbox) the format mask is applied.  The symbol length in this case is always 8, only the numeric value.

Sometimes — such as in an un-optimized QVD load — the field is converted to what I call a “full dual” (like the “SameDayShip ” example) and both the string and numeric values are stored in the symbol table.  This can greatly increase the storage used for the symbol table.

3/10/2017 42804
3/8/2017 42802
3/6/2017 42800

 

An example of an un-optimized load that will create the “full dual” representation:

LOAD
 DateField
 FROM Dates.qvd (qvd)
 Where Year(DateField) >= 2016;

In QlikView, you can “fix” this problem by going into the Document Properties, Number pane and changing the field format from “Mixed” to to “Date” format.  QV will immediately release the string storage.

Qlik Sense does not provide a Number Format pane, so you must apply corrections in the script like this:

LOAD
 Date(Num(DateField)) as DateField
 FROM Dates.qvd (qvd)
 Where Year(DateField) >= 2016;

To be fair, this is usually not a big deal for something like Dates, which have a relatively small number of values.  It can become more significant with something like Timestamps or other numeric fields that have many unique values.

The “Recommendations” sheet of Document Analyzer identifies these “Numeric Size” opportunities and quantifies the memory savings if you were to apply a correction.

-Rob

 

 

 

Share

Evaluating a Data Story

I’m midway through Alberto Cairo’s new book “The Truthful Art” and finding it very stimulating.  It’s an interesting time to be a data scientist,  journalist or consumer of data.

“The Truthful Art” encourages us to use data truthfully and fearlessly, and provides processes and principles to do so.

This week I noted a new study published by the Center for Immigration Studies (CIS). A recent Presidential Executive Order asserts that the US is in special danger from travelers from seven particular countries. The order is controversial and is currently being challenged in the courts.

The CIS study found that 72 individuals from the “seven terror-associated countries”  were convicted in terror cases since 9/11/2001.  The study offers this number as evidence of the exceptional danger posed by immigration from the seven countries.

It seems like there may be more of story here than “72 terrorists from seven countries”. The study provided a link to the raw data used. I undertook an evaluation of the data and conclusions using some of the techniques I had just been reading about.

The date used to select cases in the study was “Conviction Date”.   A more meaningful date would be “Offense Date” Offense Date was not given,  but a “Charge Date” was available. I saw this  as a better proxy for when the offense occurred.  As shown in the table below, the number of days between Charge and Conviction can be quite substantial.  Using Conviction Date skews the offense into the wrong time period.

Days from Charge to Conviction

Median 75% Maximum
381 840 2407

 

Now instead of looking at “72”, I “broadened” my view of the data as Cairo would suggest.  What about the other countries?  Are there slices of the data that provide insight?

When I plotted two country groups — banned and others — over time, an interesting story emerged.  There are no defendants from the banned countries in the last three years of the study. This suggests that travelers from those countries may actually pose less risk than travelers from other countries.

After 9/11, US domestic counter-terrorism efforts were greatly expanded and overhauled.   The decline shown in the chart suggests to me that the current screening procedures are effective and continually improving.

I’m going to continue my journey through “The Truthful Art“.

-Rob

Share

Guest Speakers for Munich Masters Summit

We’ve got a couple of special  guest presenters lined up for the Munich Masters Summit for Qlik , 5-7 April.

Ralf Becher of TIQ Solutions GmbH, Qlik Luminary and  well known in the Qlik Community, will give a talk titled Spice your Qlik Sense app with Extensions and Widgets.  Ralf will present  use cases for extensions and showcase some of the incredible extensions he has created.

Nick Webster of Websy LTD will present  The Search for Sensey McSenseface.  Imagine building a dashboard simply by talking. Nick will demonstrate a natural language interface combined with Qlik APIs that allow a user to generate visualizations on the fly by asking questions like “show me sales for Germany”.  It’s pretty intriguing stuff.

Those special speakers are in addition to our already packed three day agenda.

I hope to see you in Munich. Can’t make Munich? Maybe you will join us  September 2017 in Boston.

Learn more about the Summit or register for Munich.

 

-Rob

Share

Web Development for Qlik Developers

I just finished the four day “Web Dev for Qlik Dev” course with Nick Webster of Websy.io.  I rate the course Excellent!

The course focus is to equip  Qlik Developers with a basic understanding of how to use Web Technologies with the Qlik Sense APIs.  The week starts with an introduction to web technologies — HTML, CSS & Javascript.  And while I have some older experience as a web developer, I appreciated the brief review of current standards and practices.

We then moved on to using the Capability APIs for visualization in a mashup. We spent the last two days focusing on the lower level Engine API and the associated enabling technologies such as JSON and Enigma.js.

Lot’s of hands on work through well constructed exercises. Nick offered a lot of practical direction and tips.

I highly recommend the course to anyone who is considering or exploring integrating  Qlik Sense content into existing web apps or other mashup forms.

-Rob

Share

Document Open Processing

Summary: QlikView OnOpen Document triggers fire after the saved opening sheet is calculated.  That sheet may have some heavy calculations that slow the user’s opening experience.

Maybe you already know this, maybe not.

I commonly use a Document OnOpen trigger to make sure that my Document opens on the correct sheet.  I do this because I can be lazy or sloppy during development and save the document with the wrong sheet open.  What ever I saved with becomes the opening sheet.  I hate to reinstall an app just because I saved on the wrong sheet.

Are there any “gotchas” with doing things this way?

I just noticed (after all these years) that even if my OnOpen trigger opens a trivial or empty sheet, my document can take a long time to open.  Why? Because the data requires a lot of decompression? No.

It’s because objects on the saved sheet are getting calculated and then the OnOpen Activate Sheet is being applied!  If my saved sheet has some heavy objects, I may wait a while for those calculations to complete before the Activate Sheet runs and only then do I see output.

This is also true if I use a “Select in Field” OnOpen trigger to reduce my set to something like the current month.  The trigger fires after opening sheet charts are calculated with the saved state — which is generally all data.

Of course I can leverage this behavior by making sure I save my document with selections that represent a small set, and then refine using an OnOpen trigger.

-Rob

Share

The Document Analyzer Compare Tool

In my consulting practice, I’m frequently engaged to improve the performance of one or more QlikView applications.   In addition to an improved application,  I typically deliver a report of measured improvement and what was changed to achieve the improvements.

I use the Document Analyzer tool as my tuning workbench.  I’ve developed, and made available as a free download,  a companion  “Document Analyzer Compare Tool” that automates the comparison  and reporting of different Document Analyzer runs.  You can download both tools from the QlikView Cookbook Tools section.  Let’s look at the details.

Document Analyzer v3.0 introduced an option to save analysis results to a QVD.

 

When set to “Yes” a reload of DA will create a results QVD named
DocumentAnalyzerResults_appname_YYYYMMSS_hhmmss.QVD

The results QVD qill be created in the same directory as Document Analyzer.qvw.

The “Optional Analysis Title” will be included in the QVD.  Use this label to identify the phase of your tuning or developement work such as “baseline” or “final”.

The DA Compare Tool can consume and report on these result QVDs.   To load results, open DACompareTool.qvw and enter a directory path on the “Main” sheet. Press the “Load Result QVDs” button and the directory will be scanned for result QVDs.

A summary table will show what result QVDs were loaded.  Note that each version of the DACompareTool requires results from a minumum version of Document Analyzer.  DaCompareTool v1 requires Document Analyzer v3.3 or later.  Results from versions earlier than the minumum will not be loaded but will be reported in log messages.

The “Trending” sheet will display gross performance metrics for all results loaded. You can if course filter which results appears on these charts using listboxes.

The remaining sheets are the “Compare” sheets, which compare two result sets.  The compare sheets use a naming common to “diff” programs — one result is selected as the “Left” side and another as the “Right”.  Left/Right reflects their position in the comparison charts.

On any compare sheet use, the “Select Analysis” button to select two analyses.

Close the Select Analysis dialog by clicking the X in the upper right of the dialog. This will lock your Left/Right selections.

Throughout the UI,  data associcated with “Left” will be light green and “Right” will be light blue.

Now that two results have been selected, various comparisons are available.  Items associated with only one side will be highlighted in the associated color. You can compare and filter

Object Calculation times:

Expression contents:

Script Tabs and Lines:

Table Names and Sizes:

The orange color indicates that both side contain the named table, but with a different row count.

I’ve also included a “Server RAM” sheet that can be helpful in computing the RAM required for a set of QVWs.  You will still need to add in estimated cache requirements, but it’s a good starting point.

“Concurrent User Sessions” is an input field allowing you to model an estimated number of concurrent users for each document.

DACompareTool looks best if the Open Sans font is installed.  If Open Sans is not installed, you’ll see a warning on the “Introduction Sheet”.   It will still work without Open Sans, just not as pretty You can download Open Sans from google fonts.

If you do  performance tuning, I hope you find the DACompareTool useful.  Let me know in comments here or on this site’s contact form if you have suggestions or questions.

I’ll be diving deeper into Document Analyzer and the Compare Tool at my Feb 2 “Document Performance Tuning” class and in an expanded Performance Tuning session with Oleg Troyansky at the 2017 Masters Summit for Qlik.  I hope to meet you at either or both for a deep dive into the principles and practice of Qlik performance.

-Rob

 

Share