All posts by Rob Wunderlich

Evaluating a Data Story

I’m midway through Alberto Cairo’s new book “The Truthful Art” and finding it very stimulating.  It’s an interesting time to be a data scientist,  journalist or consumer of data.

“The Truthful Art” encourages us to use data truthfully and fearlessly, and provides processes and principles to do so.

This week I noted a new study published by the Center for Immigration Studies (CIS). A recent Presidential Executive Order asserts that the US is in special danger from travelers from seven particular countries. The order is controversial and is currently being challenged in the courts.

The CIS study found that 72 individuals from the “seven terror-associated countries”  were convicted in terror cases since 9/11/2001.  The study offers this number as evidence of the exceptional danger posed by immigration from the seven countries.

It seems like there may be more of story here than “72 terrorists from seven countries”. The study provided a link to the raw data used. I undertook an evaluation of the data and conclusions using some of the techniques I had just been reading about.

The date used to select cases in the study was “Conviction Date”.   A more meaningful date would be “Offense Date” Offense Date was not given,  but a “Charge Date” was available. I saw this  as a better proxy for when the offense occurred.  As shown in the table below, the number of days between Charge and Conviction can be quite substantial.  Using Conviction Date skews the offense into the wrong time period.

Days from Charge to Conviction

Median 75% Maximum
381 840 2407

 

Now instead of looking at “72”, I “broadened” my view of the data as Cairo would suggest.  What about the other countries?  Are there slices of the data that provide insight?

When I plotted two country groups — banned and others — over time, an interesting story emerged.  There are no defendants from the banned countries in the last three years of the study. This suggests that travelers from those countries may actually pose less risk than travelers from other countries.

After 9/11, US domestic counter-terrorism efforts were greatly expanded and overhauled.   The decline shown in the chart suggests to me that the current screening procedures are effective and continually improving.

I’m going to continue my journey through “The Truthful Art“.

-Rob

Guest Speakers for Munich Masters Summit

We’ve got a couple of special  guest presenters lined up for the Munich Masters Summit for Qlik , 5-7 April.

Ralf Becher of TIQ Solutions GmbH, Qlik Luminary and  well known in the Qlik Community, will give a talk titled Spice your Qlik Sense app with Extensions and Widgets.  Ralf will present  use cases for extensions and showcase some of the incredible extensions he has created.

Nick Webster of Websy LTD will present  The Search for Sensey McSenseface.  Imagine building a dashboard simply by talking. Nick will demonstrate a natural language interface combined with Qlik APIs that allow a user to generate visualizations on the fly by asking questions like “show me sales for Germany”.  It’s pretty intriguing stuff.

Those special speakers are in addition to our already packed three day agenda.

I hope to see you in Munich. Can’t make Munich? Maybe you will join us  September 2017 in Boston.

Learn more about the Summit or register for Munich.

 

-Rob

Web Development for Qlik Developers

I just finished the four day “Web Dev for Qlik Dev” course with Nick Webster of Websy.io.  I rate the course Excellent!

The course focus is to equip  Qlik Developers with a basic understanding of how to use Web Technologies with the Qlik Sense APIs.  The week starts with an introduction to web technologies — HTML, CSS & Javascript.  And while I have some older experience as a web developer, I appreciated the brief review of current standards and practices.

We then moved on to using the Capability APIs for visualization in a mashup. We spent the last two days focusing on the lower level Engine API and the associated enabling technologies such as JSON and Enigma.js.

Lot’s of hands on work through well constructed exercises. Nick offered a lot of practical direction and tips.

I highly recommend the course to anyone who is considering or exploring integrating  Qlik Sense content into existing web apps or other mashup forms.

-Rob

Document Open Processing

Summary: QlikView OnOpen Document triggers fire after the saved opening sheet is calculated.  That sheet may have some heavy calculations that slow the user’s opening experience.

Maybe you already know this, maybe not.

I commonly use a Document OnOpen trigger to make sure that my Document opens on the correct sheet.  I do this because I can be lazy or sloppy during development and save the document with the wrong sheet open.  What ever I saved with becomes the opening sheet.  I hate to reinstall an app just because I saved on the wrong sheet.

Are there any “gotchas” with doing things this way?

I just noticed (after all these years) that even if my OnOpen trigger opens a trivial or empty sheet, my document can take a long time to open.  Why? Because the data requires a lot of decompression? No.

It’s because objects on the saved sheet are getting calculated and then the OnOpen Activate Sheet is being applied!  If my saved sheet has some heavy objects, I may wait a while for those calculations to complete before the Activate Sheet runs and only then do I see output.

This is also true if I use a “Select in Field” OnOpen trigger to reduce my set to something like the current month.  The trigger fires after opening sheet charts are calculated with the saved state — which is generally all data.

Of course I can leverage this behavior by making sure I save my document with selections that represent a small set, and then refine using an OnOpen trigger.

-Rob

The Document Analyzer Compare Tool

In my consulting practice, I’m frequently engaged to improve the performance of one or more QlikView applications.   In addition to an improved application,  I typically deliver a report of measured improvement and what was changed to achieve the improvements.

I use the Document Analyzer tool as my tuning workbench.  I’ve developed, and made available as a free download,  a companion  “Document Analyzer Compare Tool” that automates the comparison  and reporting of different Document Analyzer runs.  You can download both tools from the QlikView Cookbook Tools section.  Let’s look at the details.

Document Analyzer v3.0 introduced an option to save analysis results to a QVD.

 

When set to “Yes” a reload of DA will create a results QVD named
DocumentAnalyzerResults_appname_YYYYMMSS_hhmmss.QVD

The results QVD qill be created in the same directory as Document Analyzer.qvw.

The “Optional Analysis Title” will be included in the QVD.  Use this label to identify the phase of your tuning or developement work such as “baseline” or “final”.

The DA Compare Tool can consume and report on these result QVDs.   To load results, open DACompareTool.qvw and enter a directory path on the “Main” sheet. Press the “Load Result QVDs” button and the directory will be scanned for result QVDs.

A summary table will show what result QVDs were loaded.  Note that each version of the DACompareTool requires results from a minumum version of Document Analyzer.  DaCompareTool v1 requires Document Analyzer v3.3 or later.  Results from versions earlier than the minumum will not be loaded but will be reported in log messages.

The “Trending” sheet will display gross performance metrics for all results loaded. You can if course filter which results appears on these charts using listboxes.

The remaining sheets are the “Compare” sheets, which compare two result sets.  The compare sheets use a naming common to “diff” programs — one result is selected as the “Left” side and another as the “Right”.  Left/Right reflects their position in the comparison charts.

On any compare sheet use, the “Select Analysis” button to select two analyses.

Close the Select Analysis dialog by clicking the X in the upper right of the dialog. This will lock your Left/Right selections.

Throughout the UI,  data associcated with “Left” will be light green and “Right” will be light blue.

Now that two results have been selected, various comparisons are available.  Items associated with only one side will be highlighted in the associated color. You can compare and filter

Object Calculation times:

Expression contents:

Script Tabs and Lines:

Table Names and Sizes:

The orange color indicates that both side contain the named table, but with a different row count.

I’ve also included a “Server RAM” sheet that can be helpful in computing the RAM required for a set of QVWs.  You will still need to add in estimated cache requirements, but it’s a good starting point.

“Concurrent User Sessions” is an input field allowing you to model an estimated number of concurrent users for each document.

DACompareTool looks best if the Open Sans font is installed.  If Open Sans is not installed, you’ll see a warning on the “Introduction Sheet”.   It will still work without Open Sans, just not as pretty You can download Open Sans from google fonts.

If you do  performance tuning, I hope you find the DACompareTool useful.  Let me know in comments here or on this site’s contact form if you have suggestions or questions.

I’ll be diving deeper into Document Analyzer and the Compare Tool at my Feb 2 “Document Performance Tuning” class and in an expanded Performance Tuning session with Oleg Troyansky at the 2017 Masters Summit for Qlik.  I hope to meet you at either or both for a deep dive into the principles and practice of Qlik performance.

-Rob

 

Munich Masters Summit

I’m excited to announce our first 2017 Masters Summit for Qlik event in Munich Germany on 5-7 April 2017.

munich-email-banner

 

Designed for Qlik Developers who have basic skills and experience,  the Summit presents three days of intense hands-on sessions in topics such as Advanced Scripting, Data Modeling, Advanced Aggregation and Set Analysis, and Visualization Techniques, applicable to both QlikView and Qlik Sense.

You have some experience with Qlik, have taken the beginning courses. How do you ramp up to create more success with your Qlik program? Learn from seasoned experts, authors and world class presenters Rob Wunderlich, Barry Harmsen, Oleg Troyansky and Bill Lay.

In addition to the hands-on exercises, you’ll come away with many valuable sample files and documents. You’ll also get a chance to meet and network with Qlik Developers from around the world.

Our 2017 program features an expanded Performance Tuning section and additional content specific to Qlik Sense.

In four years over 800 Qlik Developers have attended eleven Summits around the world. Their feedback is overwhelming positive. Read about their experience here.

I hope you can join us in Munich to take your Qlik skills to the next level! Read about the details of registration here.

-Rob

Is Data Beautiful? The Art of Adrien Segel

This past spring I enjoyed a glorious week of whitewater kayaking with Noah Weinstein at the incredible Otterbar Lodge on Northern California’s Salmon river.

Noah runs the Artists In Residence program at Autodesk’s Pier 9 Workshop in San Francisco.  Autodesk is a long time big QlikView customer and the Pier 9 Workshop is a working lab that demonstrates and tests practical applications of Autodesk products.

During downtime at Otterbar Lodge, Noah introduced me to the work of sculptor Adrien Segel.  Adrien sources data sets of observations from the natural world and transforms those data into beautiful, fascinating and challenging sculptures.

Adrien’s “Wind at Ravenswood Slough” project visualizes wind speed and direction over a 48 hour period at a single location. The Y-axis (vertical) represents time, the X-axis (length of the bars) represent speed — and here’s the advantage of a physical 3-D rendition — the Z-axis indicates wind direction.

Mount wind-at-ravenswood-pic1the finished sculpture  at the site where the data was collected and you have a deep understanding and delightfully personal relationship with data.  Brilliant!

 

 

 

 

Check out some of Adrien’s other projects like the  Snow Water Equivalent Cabinet  I found this dataset personally  interesting because of my love of rivers and river seasons.  Adrien takes a direct approach in mapping the data and the  result is a functional, fascinating  and intimate piece of furniture.

Can data be beautiful?  I think so.

-Rob

 

 

 

 

QV 12.10 “Can Execute External Programs” is back

In an earlier post I wrote about how to authorize the script Execute statement and the differences between QlikView versions 11 and 12.  I’ve just tested the new QlikView v12.10 Initial Release and the rules have changed again.

To run an Execute statement in QV12 Desktop requires that “User Preference, Security,  Script (Allow Database Write and Execute Statements)” be checked on.  This is also true in QV12.10.

An additional Script Setting “Can Execute External Programs” is present in QV12 but had no effect.  In QV12.10, this setting behaves as it does in QV11. If off, you will be asked to authorize when the Execute statement is encountered in the script.

For an Execute to run in QV12.10 Desktop without prompting, both “Script (Allow Database Write and Execute Statements)” and “Can Execute External Programs” must be on.

What about QV12.10 Server? Like QV12 Server, Execute is authorized by the setting:

AllowExecuteCommand=1

The value is set in “C:\Windows\System32\config\systemprofile\AppData\Roaming\QlikTech\QlikViewBatch\settings.ini”, and it is off by default after install.

Does QV12.10 Server need the  “Can Execute External Programs” setting? No. As a matter of fact, if it’s off, the reload sets it on. I don’t mean temporarily. I mean it sets and saves it in the qvw.

Updating the matrix from my previous post:execute-rules

 

What is the intent behind this QV12.10 change?  I’m not sure.

-Rob

QVC Documentation for Qlik Sense

QlikView Components (QVC — should we rename it to “Qlik Components”?) documentation is now available in a Qlik Sense QVF format.   You can download it from github here:

After downloading the “QVC Documentation.qvf” file, you’ll need to copy the file to your Qlik Sense apps folder or drop it on Qlik Sense desktop window to view.

The “Using QVC” sheet provides general instructions for including the QVC library from either the file system or the web.

2016-11-09_19-31-39

The “Details” sheet provides descriptions and parameters for all of the QVC routines.

2016-11-09_19-30-50

 

Examples are not yet available in qvf format, as they are in qvw format. I’m working on that.

Have fun using QVC in Qlik Sense. If you have questions or need help in using QVC use the QVC Forum here.  I also teach online classes on QVC if you want a deeper education on the capabilities and applications of QVC.

-Rob

Storing a Data Model in a Single QVD

Have you ever thought it might be interesting to store a  Qlik data model into a single QVD?  This can be useful in a number of cases such as:

  • Archiving (and retrieving) data models.
  • Overcoming the “single binary load” restriction.

QlikView Components (QVC) Version 11 introduced two new routines to do just that:

Qvc.ExportModel — Exports all tables of the current model into a single QVD.

Qvc.ImportModel — Import a data model created by Qvc.ExportModel.

Even if you don’t have QVC V11 installed, you can try Qvc.ExportModel right now using  http include.  Add these lines to any QlikView script (instructions for Qlik Sense further on down in this post).

$(Must_Include=https://github.com/RobWunderlich/Qlikview-Components/releases/download/v11.1/Qvc.qvs);
 CALL Qvc.ExportModel
 CALL Qvc.Cleanup

Mind the wrap. The Must_Include should be on one line. Using QVC requires the Qvc.qvs library be included (usually at the beginning of script), CALLing Qvc routines, and CALLing a Cleanup routine at the end of your script.

Assuming this script is included in “Sales Dash.qvw”,  the default exported model QVD will be named “Sales Dash.qvd” in the same directory.

 

Now, to import this QVD model into another qvw, replace the CALL to ExportModel in the above sample with:

CALL Qvc.ImportModel('Sales Dash.qvd')

The original model will be reconstructed as individual tables.

Qvc.ExportModel has three optional parameters:

CALL Qvc.ExportModel(['qvddir'],['qvdname'],['addTimestamp']);
Parameter Number Parameter Description
1 String. Optional. Relative or absolute directory where the model QVD will be stored. If relative, it follows the same rules as the STORE script statement for relative directory.
2 String, Optional. Name for the model QVD. If omitted, the name of the QVW will be used. For example, if QVW is “Sales.qvw”, then QVD will be “Sales.qvd”.
3 String, Optional. 1/0 True/False. If True, a timestamp of the form _YYYYMMDDhhmmss will be appended to the QVD name. Default if omitted is False.

 

Qlik Sense has no default path  so parameter #1, a lib:// for the QVD should be specified.  Alternatively, if a lib has been established with a DIRECTORY statement, parameter 1 can be omitted.

Qlik Sense will require a web file Connection for the http Must_Include.

webfile-connection

After defining the web connection and having an appropriate folder connection to store the QVD in,  Qlik Sense script would look like this:

$(Must_Include=lib://QvcWeb);
 Call Qvc.ExportModel('lib://QVData')
 CALL Qvc.Cleanup;

 

That’s all there is to it!  If you are already using QVC, I hope you’ll find these routines a welcome addition to the library.  If you are new to QC, explore more at QlikviewComponents.org.

-Rob

Thanks to Jörgen Peerik for raising the single-QVD export idea during a QVC class.