Category Archives: QSDA

QSDA Pro 2.5 Brings Extended Expression Validation

Summary: Set Modifier field names and Set Identifiers are not validated by Qlik syntax check. QSDA Pro 2.5 validates everything and raises an “Unrecognized Name” flag for invalid names.

QSDA Pro syntax checks all expressions in a Qlik App using the Qlik syntax checker. You’re familiar with the Qlik syntax checker. It’s the message at the bottom of the expression editor that tells you your expression is “OK”.

The syntax checker is also good at telling you when the expression has an error, although it’s not always specific about the problem.

QSDA Pro, using the Qlik API version of syntax checker, tells you what is wrong with the expression:

The Qlik syntax checker has a significant limitation. It does not validate field names used in sets. Whether used in the API or the editor, the syntax checker will not raise an error if the field “EmpPurchase” does not exist in the data model.

This lack of validation can be a serious problem if “EmpPurchase” is removed or renamed in the data model. The expression will still return a number but the exclusion of employee purchases will no longer be applied. That could be a very subtle but important fail.

Recognizing this limitation and it’s potential impact, QSDA Pro 2.5 validates all names used in sets and will raise a new Quality flag, “Unrecognized Name”, if the name is invalid.

Another place this validation is extremely useful is Chart Filters in Qlik SaaS. The chart will cheerily display that filter “Expression2 > 1” is being applied. Even though field “Expression2” no longer exists in the data model.

But QSDA knows.

Ever use a bookmark as a set identifier? And then the bookmark got deleted or renamed?

I’ve used simple examples to demonstrate, but where this validation really shines is in longer expressions where it may be easier to overlook that something has gone wrong.

Yes, you need QSDA Pro. We all need QSDA Pro.

Learn more about QSDA Pro and download your own copy at https://easyqlik.com/qsda/

-Rob

What does QSDA stand for? Qlik Sense Document Analyzer. The combination of data model and sheets we know as an “App” in Qlik Sense was called a “Document” in QlikView. When I first created this tool in 2009 to help maintain QlikView, I called it “Document Analyzer”. When it came time to create a similar tool for Qlik Sense, I stuck with the “DA” name as the function and usefulness of “Document Analyzer” was well established in the Qlik community.

Share

Mind the Concat() sort-weight

Summary: While looking into long expressions I noticed that the optional sort-weight argument has an impact on the distinctness of Concat(distinct…). Incorrect use of sort-weight can generate bloated expressions containing redundant code.

In my work tuning Qlik Apps I sometimes encounter very long expressions. An expression many thousands of characters long can be difficult to debug or comprehend the expression goal. To help in working with long expressions I’ve added an Expression histogram and an Expression Decoder feature to my QSDA Pro product. (These features are currently in beta, generally available in early Feb).

I’ve noted expressions of length greater than 50k across apps from different customers. What did these expressions have in common that made them so large?

  • They used the Concat() function in $() to dynamically generate a part of the expression.
  • They used the optional sort-weight argument of Concat() incorrectly.
  • They were much bigger than necessary — sometimes 100x — but the expanded expression worked as intended.

In the process of reviewing the expressions I learned something surprising. As a reminder here’s the syntax of the Concat function:

Concat({[SetExpression] [DISTINCT] [TOTAL []]} string[, delimiter[, sort_weight]])

We use the DISTINCT keyword to return the unique set of values from the string argument (usually a field). The Qlik documentation for DISTINCT says:

If the word DISTINCT occurs before the function arguments, duplicates resulting from the evaluation of the function arguments are disregarded.

https://help.qlik.com/en-US/sense/November2022/Subsystems/Hub/Content/Sense_Hub/ChartFunctions/StringAggregationFunctions/concat.htm

This means that the set of distinct values is the combinations of string and sort_weight (if used). Let me demonstrate with an example. Here’s a sample data table.

For the expression: Concat(Dim, ',') we receive output "a,b,c,c,c“.

Adding the DISTINCT keyword: Concat(DISTINCT Dim, ',') we now get “a,b,c“.

Adding a non-distinct sort-weight argument: Concat(DISTINCT Dim, ',', RecId) we now get "a,b,c,c,c” again. More output than I expected. It’s a distinct list of the combinations of Dim and RecId.

Adding a distinct sort-weight argument: Concat(DISTINCT Dim, ',', Weight) we now get "a,b,c“.

How about if we used an unlinked data island field for sort-weight? The Island field has two values.

Concat(DISTINCT Dim, ',', IslandField) returns "a,b,c,a,b,c“. Item count is the product of Dim * IslandField values. Remember this for later.

Ok, this is all very interesting but the behavior is super obvious and I would notice it if it came up in my App. What’s this got to do with ginormous expressions?

Developers sometimes use Concat along with Dollar Sign Expansion (DSE) to generate dynamic expression fragments. For example to ignore all fields from several tables in a set modifier:

Sum ({<
$(='[' & concat({<$Table={'Table1', 'Table2', 'Table3'}>}$Field,']=,[') & ']=')
>} Value)

Sometimes $(=Concat(...)) is used to build the list inside a Pick() or Match(). These type of expressions frequently have awkward syntax including lots of “& chr(39) &” type stuff. Inevitably the expression gets built by copying and modifying an expression from elsewhere in the App. An expression that contains a sort-weight. A sort-weight that doesn’t get removed. It may be an island field or a field that has a many to one relationship. The result is an expanded expression that works but is larger than it needs to be. No one notices (unless they use QSDA Pro) because it’s the expanded expression.

As a simple example, suppose the “ignore filter” expression above was supposed to generate something like "Quarter=,Month=,Year=“. If I inadvertently use a sort-weight field that has 100 distinct values the result will be repeated 100 times. The expression would still work but it would be 100x larger than necessary.

I recently found an example where Concat was used to generate an If() function from data (very clever) that should have had 15 branches. But an unrelated sort-weight field of 95 values resulted in 1425 branches! It “worked” but did a lot of unnecessary calculation.

If you are a solo developer or working in a small team you may never encounter this issue. But if you are a consultant or maintaining legacy Apps you may stumble across it. I’ve been playing with ways to flag this condition in QSDA Pro. QSDA already flags data island expressions . I’m testing creating a new flag specifically for Concat().

My colleague Oleg Troyansky uses QSDA in his Performance Tuning session at the Masters Summit for Qlik. Live events will return in Fall 2023 but in meantime you can attend individual workshops on-line during February through April. More information and schedule here.

Happy Qliking
-Rob

Share

QSDA Pro 1.7 is Released, What’s New?

QSDA Pro 1.7 is Released. So What’s New?

The QLIKLOGIN connection method allows your QSDA users to connect to Qlik Sense using the same login process they use for the Hub — Windows, SAML, whatever. After logging in, user “Sally” will see only her apps — the same list Sally would see in the Qlik Sense Hub.

The “Filter using this flag” button filters all your resources to everything associated with this flag, giving you fast focus on what needs to be examined or remediated.

Hyperlinks in resource lists provide one click to open the subject Viz in the Qlik Sense sheet editor.

Still my favorite QSDA feature, Viz Deconstruction itemizes the calculation time of each individual expression in a chart so you can focus in on what the heck is taking so long in this chart.

Stuff Administrators Will Love

  • Password protect Admin pages
  • Offline license keys for air gapped servers
  • Possible to install QSDA without Windows Admin rights
  • Https support
  • Optional API security

What Can QSDA Pro Do For Your Team?

  • Expose undetected errors — before your users do!
  • Gives actionable advice
  • insight and focus for developing and debugging
  • Save time on tasks like “how/where are we using field ‘abc’?”

Join a Webinar

“Introducing QSDA Pro” will show you the features and benefits offered in QSDA Pro, as well as an opportunity to ask questions.

Share

QSDA Pro as a Quality Tool

I my last two posts I introduced QSDA Pro and the flag workflow.  In this post I’m going to show how you can use QSDA Pro to quickly uncover quality issues in your Qlik Sense App.

QSDA groups flags by category, and the “Quality” category is where I usually begin my app review.  Quality flags are used to indicate where something appears to be seriously broken such as:

  • An expression syntax error.
  • A reference to a Master Measure that no longer exists in the library.
  • A missing extension.
  • A bookmarked field that is no longer exists in the data model.

These are the kinds of things you want to discover before your users do!

It’s relatively easy to create new errors in an existing Qlik app.  Qlik does not warn or block you from deleting a resource like a Field or Measure that is in use.

A properly done field rename may update names in expressions.  But it will not update variables or bookmarks.  And I’ve seen plenty of field renames go wrong and break expressions as well.

It can be difficult to detect all app errors by visually checking your app.  The broken piece may be a color expression or a calculation condition that isn’t readily visible.

So yes, easy to break, sometimes hard to detect and heros find  problems before their users do.  Enter QSDA Pro.

In QSDA menu,  Help -> Flag List will display a list of all potential flags in the installed version. Here’s the current list for Quality (The “{}” bits are values that will get filled in when the flag is created.)

Some of these are fairly straightforward, some a bit more subtle.

Why would you have a missing Master Dimension or Measure?  A fairly common scenario is not understanding that Master Items are referenced by internal Id, not by Name.  So you create something called “Sales” and use it in several charts.  Later you are doing some exploration and create something called “Sales2” which you like better. So you think if you delete “Sales” and rename “Sales2” to “Sales” everything will be using the new “Sales”.  Wrong.

Because the master item is now gone, I can’t tell you what used to be in this master item.  You will have to rely on an app backup. Or…if you have a previous QSDA Analysis for this app you can look up the master item definition there!

All Expressions and Dimensions are validated using the Qlik syntax checker. A typical error might be a bad fieldname (perhaps because the field was removed from the model).

Flag Details will give us a detailed error message, location of this expression  (Master Library) and importantly the Use Count. This use count is zero so we know deleting the measure is a viable option.

QSDA uses the Qlik syntax checker so the test is only as good as what I  get from Qlik.  There are some limitations, for example when column labels are used in an expression.  You’ll note that the Qlik expression editor flags this as invalid so QSDA will pass this message on.

The “Parse Error” flag is raised when Qlik Syntax says the expression is ok but QSDA finds the expression does not conform to expected syntax rules. This could be a problem with the QSDA parser  (a work in progress) or it could be Qlik tolerating something unusual.  For example, do you think this is a valid expression?

sum({<Sales={">10 between < 90"}>}Sales)

The Qlik syntax checker will declare it valid. And it will return an official looking result — which may or may not be correct.  But what is that “between” keyword? (if you’re curious why “>10 के बीच < 90” also seems to work, see https://masterssummit.com/something-to-get-confused-about/).

Nervous about refactoring or changing your apps?  Can you relate to “Change Paralysis”  or Fear Driven Development (FDD)?  We all need a tool like QSDA Pro to proactively plan changes as well as audit after the fact for unintended errors.

Download QSDA Pro now and analyze some of your own apps.  QSDA Pro is free to use during the beta period.

You can analyze published and unpublished apps, so go ahead and take a look at some of those production apps.   Let me know in the comments if you find something interesting!

-Rob

 

 

 

Share

QSDA Pro Flags

In my first post introducing QSDA  Pro, I told how Document Analyzer was a tool I developed for use in my own consulting work and shared with the community of Qlik developers.  The Qlik Sense QVF version got a big boost in utility when Axis Group joined the project and contributed their many many years of Qlik experience.

In this post I will  introduce the  Flag workflow of QSDA.

QSDA communicates it’s advice through “Flags”. Flags are issues that warrant your attention.

The Summary page gives an overview of Flags for this analysis. Clicking any of the Flag boxes or Flags in the Nav Ribbon will take you the Flags page.

The Flags page provides a collapsible list of all flags and buttons for more detail and flag status.

 

Flag text identifies the error and in most cases provides some detail, such as a fieldname or number of bytes that could be saved.

The dimension failed validation. The error is: Bad field name(s) "[PostalCode]" .

Click  to  show the Flag Details panel.

The details panel will provide more context for the flag to help you evaluate your response.  Depending on the flag, there may be additional hyperlinks.  Click the “?” icon to link to the Help article for this flag type.

To support a workflow of “checking off” the flags,  buttons and flag status indicators are available in the both the Flag Details and Flag List views.

 Mark this flag instance status  as “Resolved”.

 Mark this flag instance status  as “Hidden”.

Resolved and Hidden flags are not shown in the list unless the filter is activated on the filter bar. Flags with no status are “Unresolved” and are always shown.

Buttons will automatically toggle to allow you undo a status. You can set/unset statuses on individual flags or at the flag type level.  Hover over a button to discover its function.

From the Flag List you can view all flags.  In the resource pages, you will also see the same flags but attached to the resource.  For example, on the Dimension page,  any associated flag will be shown with the specific Dimension.

These workflow features allow you to punch through the list of issues in your app, hiding those you don’t care about and checking off issues as you resolve them.

Learn more about QSDA Pro and download the latest version.

-Rob

 

 

 

Share

Introducing QSDA Pro

QSDA Pro is the next generation of my Document Analyzer tool for Qlik Sense.  I published the first Document Analyzer for QlikView in 2009 followed by the Qlik Sense version in 2017.  I think it’s fair to say both are the most widely used Qlik developer add-on tool.

Qlik Sense Document Analyzer (QSDA) has been extremely useful. But it’s had some limitations, especially in the server environment. I’ve wanted to address those shortcomings,  and now I bring you my response :

QSDA Pro is a completely rewritten standalone app that provides  features of the current QVF based QSDA and more.  QSDA Pro can analyze and provide advice on apps in QS on Windows, Qlik Cloud, and QS Desktop.

QSDA can help you produce cleaner, smaller and faster apps by identifying:

  • Unused resources, such as fields, that can be removed.
  • Errors in your app such as missing fields, syntax errors or missing key values.
  • Bad practices and recommendations on how to remediate.
  • Highlight performance hotspots.

QSDA Pro can do all of the above and more for public sheets and private sheets.  “Is anyone using this field in a personal chart or bookmark?” is no longer an unknowable thing.

QSDA is also your goto companion for those ad hoc tasks like modifying use of a field. Maybe a renaming or dividing by 1000.

Where am I using the field “Company Name”?  Easy.  Filter on the field name and I can instantly see the field is not referenced by Variables, but it is used in 22 Charts, 23 Dimensions and 5 Expressions. Oh and 1 Bookmark.

What Charts where?  Click Viz on the ribbon to get a list.  I can explore the chart Dimensions and Expressions there or click a link to open the associated sheet in the Qlik Sense Hub.

QSDA is smart! It can determine that all four of these measures calculate the same thing, even with differences of variables, casing and whitespace.  And it recommends you should settle on one Measure while very helpfully showing how many times each variation is used to help you in your decision.

In the coming weeks I’ll be posting more about the capabilities of QSDA Pro.  I’ll also be responding to feedback on the QSDA Pro Community site.

QSDA Pro is currently in public beta testing and is free to use during the testing period, which is expected to last for several months.

You can learn more about QSDA Pro and download your own copy.

Licensing terms for the supported production version will be announced later. There will be a free tier which may have  limits on volume and  features.

-Rob

 

Share