Monthly Archives: April 2018

General

Qonnections 2018 Reflections

9 Comments

I just wrapped up the week at Qonnections 2018.  I found it to be an inspiring conference and the most positive I’ve attended in years.

What was inspiring?  The cognitive insights demo was fantastic.  This is the future of BI.  We have well understood principles and algorithms recently formalized in the discipline of machine learning. Let’s incorporate them in the tool.   If the goal of data visualization is to amplify cognition of data, then I would say cognitive insights is to visualization as visualization is to data.

On the technical front, the previews and demos of elastic cloud and Qlik Core provided me the “aha” moment to understand how the Qlik Big Data Index will be possible.  I can see how many distributed indexing and aggregation engines can provide massively scalable access to a data lake.

The most exciting announcement to me wasn’t technical, but on the commercial side.  For a “small uptick in maintenance”,  QlikView  users will be entitled to use the same license for Qlik Sense.  To me this signals a new strategy for dealing with the two products going forward.  There was also a roundtable discussion on feature gap between the two products and the announcement that some level of pixel-perfect control will be available in Qlik Sense.

Taken together with main stage comments from CEO Mike Capone and CTO Mike Potter,  I’m seeing a needed correction to the “two product strategy” which so heavily favored Qlik Sense.  No one expects QlikView to live forever,  but loyal QlikView customers want to adopt a new product only when they see that it clearly benefits them.   Ideally we will see a converging product that blends the best features and lessons learned from both products.

-Rob

Share
Productivity, Tools

Exploring Data Lineage with NodeGraph

6 Comments

In the Qlik world, we are frequently faced with questions like:

  • Where did this field come from?
  • What applications (if any) use this QVD?
  • If I change this database table, what applications will be affected?
  • Am I creating QVDs that are not being consumed?

I’ve maintained a QlikView lineage application for my customers over the years.  I was never completely happy with it as Qlik lineage metadata is inconsistent and the critically important field level lineage was never available.

All that changed when I discovered NodeGraph.  Now I’m a NodeGraph fan and partner.

NodeGraph is an add-on for your QlikView and Qlik Sense solutions that allows you to explore, visualize and trace where your data comes from, whether it’s up-to-date, how it’s calculated, and much more.

NodeGraph scans both QlikView and Qlik Sense files and produces a graph of applications and data in a beautiful easy to navigate interface.  Select any node, for example a dashboard QVW, and lines trace the data lineage through transformed QVDs all the way back to the source database.

 

You can search, filter and drill in any direction;  from applications,  QVDs, source tables, fields, charts or SQL queries.

Where did this field come from?  Right-Click any field and select the Field Explorer to get a report of the complete lineage for this field, including all script transformations!

 

I’m looking at a chart that shows a count of Customers.  How was that count calculated? Where did the data come from?  In the Content view I can review the expression and see the lineage for fields used.

 

One of my favorite features is the one-click application documentation.  The tool generates a pretty printable/PDF document suitable for Auditors that describes all data sources, transformations, and expressions used in the selected application.

 

There are more uses and features in NodeGraph than I can cover in a blog post, including governance reports, application usage, GDPR compliance and testing.

If you are coming to Qonnections,  see NodeGraph in action at the TechLab10 booth (#603) in the Discovery Expo.

Learn more about NodeGraph at nodegraph.se. If you have questions or would like to schedule a free demo,  reach out to the NodeGraph team on the website or info@nodegraph.se.  If you are in North America you can contact me directly for questions or to scheduling a demo.

-Rob

Share
Performance, Scripting

AutoNumber vs AutoNumberHash128

16 Comments

Summary:  AutoNumberHash128(A, B) runs about 30% faster than AutoNumber(A &’-‘ & B).

It’s a common practice to use the script AutoNumber() function to reduce the storage required for large compound keys in a Qlik data model. For example:

AutoNumber(A & '-' & B) as %KeyField

As a standard practice, we generally include a separator like ‘-‘ to ensure ‘1’ & ’11’ does not get confused with ’11’ & ‘1’.

The AutoNumber process can add significant run time to a script with many rows.

I’ve always wondered what the AutoNumberHash128() function was good for.

AutoNumberHash128(A,B) as %KeyField

This function first hashes A & B and then autonumbers the result. The end result is the same as the first example given using AutoNumber().  I find the AutoNumberHash128 syntax a bit simpler as a separator is not required.

What surprised me is that the AutoNumberHash128() function runs faster.  Typically about 30% faster than a plain AutoNumber with a concatenated string parameter.

Why is it faster?  The difference is in the function used to create the single value to be autonumbered.  Hash128 is considerably faster than string concatenation (&).

AutoNumberHash128() can take any number of fields, but it does not have an “AutoId” parameter.  The “AutoId” (second parameter) in AutoNumber() is recommended to ensure we get sequential integers when autonumbering more than one key field.  Sequential integers are the most memory efficient storage for keys.

Don’t despair.  AutoNumberHash128() will use the “default” AutoId.  That is fine if you are autonumbering only one key.  If you are doing more than one key, use AutoNumberHash128() for your largest — most rows — key and use AutoNumber() with AutoId for the rest.  You will improve the script run time of one key.

Another possible tradeoff when you have many large keys is to use AutoNumberHash128 for all keys and forgo the sequential integer optimization.  You will use only 8 bytes per key value which could be significantly less than the original string keys.

-Rob

Update 20 Sept 2022

Things have changed somewhat with the addition of the AutoNumber statement, which is the recommended method to autonumber keys. AutoId is no longer a problem.  In my recent testing  creating compound key fields, I still find that Hash128() is somewhat faster than the & operator.  Here’s the results. Option 4 is creating the key with the & operator and AutoNumber statement. Option 5 is creating the key with Hash128() and AutoNumber statement.

 

Share
General

Qonnections 2018

4 Comments

Are you going to Qonnections in Orlando April 23?   I’ll be there and presenting three breakout sessions.  In addition to checking in with colleagues and hearing about what’s new, here are some things I’m looking forward to:

  • Hearing from the new Leadership team
  • Qlik Core!
  • Learning more about subscription pricing
  • Freakonomics!

I’ll be presenting three technical breakout sessions.

148255 – Options and Levels for Qlik Sense Mashup Integration: Tuesday 10:30am  –  This session aims to introduce you to the possibilities for re-using Qlik Sense content and data in other applications such as web pages.  We’ll cover the very simple “codeless”  embedding using URLs all the way through data fetching and custom visualization using enigma.js.   This session is suitable for technical and management staff who want to understand the potential in mashups and get a measure of the effort involved.  My colleague Nick Webster will be joining me in this presentation.

148870 – Automated Testing of Qlik Applications:  Tuesday 3:00pm – A favorite of mine, I’ll be discussing and demonstrating the free regression testing tools from Qlik that allow you to completely automate the QA testing of your Qlik — both Qlik Sense and QlikView — applications.  I’ll also touch on some current trends in unit testing and operational monitoring.

148256 – QlikView Document Performance Tuning Using Document Analyzer:  Thursday 9:30am – Document Analyzer (DA) is a popular free tool that can be used to examine and improve the performance of a QlikView document.  In this session I’ll be showing the tool and my typical approach to improving the response time of a specific document.  Admittedly,  Document Analyzer has little documentation so consider this your DA training session. If you’ve never seen DA,  Dalton Ruer, aka @QlikDork recently produced a  DA intro video.

Other sessions I hope I can make it to:  Nick Webster’s “Putting Qlik in a Mirror” – just for the wow factor;  Göran Sander’s sessions on Butler SOS and SenseOps;  Sessions on Qlik Core and View/Sense coexistence.  Sadly  I’m going to miss Speros Kokenes’ sessions because he’s scheduled at the same time as me.

I’ll also be hanging around in the Discovery Expo — look for the hat at the Masters Summit booth or elsewhere.

Hope to see you there!

-Rob

Share