QSDA Pro 3.4 was recently released and here’s a roundup of some cool new features.
QSDA Pro is a quality tool for Qlik Sense Apps. QSDA uncovers errors, advises on best practices and provides powerful tools to optimize your Qlik Apps and minimize your development time.
Apply Recommendation
Now QSDA not only makes recommendations to improve your App, but in many cases offers to make the changes for you! Pressing the “Apply Flag Recommendation” button will update your Qlik App with the recommended change — in this example replacing a free form expression with a Master Measure.
As always, QSDA is smart enough to make matches using the logical meaning of the expression rather than a limited text match.
Learn more and see “Apply Recommendation” in action in this video here.
Load Script
As requested by several customers, we are now collecting the Load Script for reference and viewing within QSDA. It’s presented in it’s multi-tab format and a search across tabs function is provided.
Deprecated Visualizations
A new flag “Deprecated Visualization” is raised for any visualization that Qlik has listed in a deprecation notice. For example, the Multi KPI. This will give your team a heads up to consult the Qlik doc for replacement options.
HTML Report
A new HTML Report option in Tools generates a formatted report from your analysis. You can use this generated report to share results outside of QSDA or serve as an artifact in your change management process.
Large App List Performance
For large App Lists we’ve instituted streamed results and paging to make the list more manageable. We still recommend you use the connection filters to focus the list to specific streams or spaces but we understand that doesn’t work for everyone.
Read the entire list of changes and usability improvements here
Summary: A look at techniques to reduce the memory footprint of your Qlik Data Models.
There are good reasons to reduce the memory footprint of your Qlik Apps. Smaller Apps tend to perform better and require fewer server resources. Qlik Cloud imposes size quotas and there is a financial cost to exceeding the quota.
In this post we’ll look at techniques to reduce the memory requirements of your data model.
First, lets review how Qlik Sense stores App data in memory.
The Qlik Load script transforms your data into two types of internal data structures.
A Symbol Table is created for each field. The Symbol table contains a list of the distinct values of this field. The overall size of the symbol table is therefore a function of how many values and the width of the values.
Logical Tables are the tables we see created in the data model, i.e. fact and dimension tables. The Logical tables contain one row for each row of data. The tables do not contain the field values directly. Rather they contain a pointer to a value in the associated Symbol table. This pointer is sometime called an Index.
The width of a pointer is the number of bits required to index the number of values in the Symbol table. For example, in the image above we have 5 values for “Year”. This would require 3 bits as 2^3 is 8, which would cover the required 5 values. 2^2 would be too small as that would only cover 4 values.
MyQSDA Pro tool shows the memory required for each field, including a breakdown by Symbol and Index.
I’m going to use QSDA Pro for my demonstrations in this article.
Using less memory for fields involves reducing the size of the symbol table, the index, or both. There are four techniques we can employ to reduce field memory:
Eliminate the field
Reduce the number of field values
Optimize field values to reduce width, including AutoNumber
Reduce the number of rows
Let’s look at each of these options.
Eliminate the field
Removing the field from the data model is the easiest way to reduce memory :). Of course this is only viable if the field is not used by your application. To determine if a field is used, you must consider all the places a field may be referenced — master items, variables, charts, bookmarks, etc — for both public and private content. Fortunately, QSDA scans all public and private content and makes the inuse determination for you.
In some cases you will not want to drop a field as you anticipate it being used in the future. The benefit from removing fields is relative to the size of the field. When considering whether you want to drop a field, look to the largest fields first where you will gain the most benefit.
QSDA Drop Unused Fields Dialog
Reduce the number of field values
Reducing the number of distinct values will reduce both symbol and index space. How might we do this?
A common scenario involves timestamps. For example let’s look at a field named “ShipTime” which displays in our app as hh:mm:ss, reporting the time an order was shipped. The analysis requirement is that occasionally an analyst will need to see this value in an order detail chart. This field has 459,918 distinct values. Something seems off. There are only 86,400 possible hh:mm:ss values per day and we only ship during one shift so I would expect to see no more than 28,800 values.
When I examine the values in the app I discover the field contains an entire Datetime value and the display format hh:mm:ss is being used in the chart. I don’t need the date portion. I’ll use the Frac() function in the script to extract just the time.
Reload followed by a new QSDA Analysis and here’s the result below. A massive reduction in distinct values and associated savings in both symbol and index. This reduces the total memory for this field from 5200KB to 345KB.
Another potential scenario for this field is that the original developer required both the date and time. In this case our approach is to split the datetime into separate Date and Time fields like this. Remember that formatting functions like Date() and Time() do not change values, we must use numeric functions like Floor() and Frac().
If I need to display the “Ship timestamp” in a chart, I’ll do it like this in the chart:
ShipDate & ' ' & ShipTime
A QSDA analysis now shows a total of 863KB for both fields. A big reduction from 5200KB!
Other potential ways to reduce the number of values:
Drop seconds or fractions from Time fields.
Some accounting systems return fractional amounts, or you may have unneeded fractions due to a calculation. Round the final amount to the penny or appropriate magnitude.
Optimize field values to reduce width
Reducing the width of the field values decreases the Symbol space but not the Index space. How can we reduce the width of a value without changing the business meaning of the value? Isn’t a value a value?
We’ll start with the easy way — AutoNumber. The AutoNumber statement (and AutoNumber function) translates values into a set of sequential integers. This is useful for compacting key values. We typically don’t need to see the key values, we just need to them to link our tables together.
AutoNumbering a field is done by adding this statement at the end of your script:
Here’s the before and after sizes for SalesOrderDetailID, showing a total reduction of 143MB.
Before AutoNumber
After AutoNumber
Note that the symbol space has been completely eliminated! When the symbol set consists of sequential integers, Qlik does not store the symbols. Instead, the index value is used as a proxy for the symbol value.
Now we need to dive a bit deeper into the actual format of a symbol table. The symbol table diagram at the top of this post is a bit of a simplification.
Generally speaking, strictly numeric values can be stored in 8 byte cells. For a field that contains only numeric values the symbol table is an array of 8 byte values.
We can index to the nth value simply by multiplying by 8. This is a very efficient and compact storage format. In QSDA these fields will show an Average Symbol Width of 8.00.
QSDA Fields
For fields that contain strings, the symbol table layout is a bit different. As strings can be of varying length, an array of fixed cells won’t do. If there are any strings in the field, the entire field is considered “mixed”.
The symbol table for a mixed field is an array of 4 byte pointers that point to symbol values elsewhere in memory. The values consists of:
1 byte flag
The varying length string
1 byte null terminator
The total symbol space for a string is 6 bytes plus the length of the string. The storage for string “a” would be 7 bytes.
Looking at the “LastName” field in the image above, we know that each symbols value carries a 6 byte overhead. We can infer that the average length of the LastName strings is 6.54. That is, 12.54 – 6.
When the value is a Dual() value, the symbol value picks up an extra 4 or 8 bytes to hold the numeric value. 4 bytes is used for integers, 8 for decimal values.
The total symbol space for a dual value is 6 bytes plus the length of the string, plus 4 or 8 bytes.
A field may contain both string and dual types. Dual is a value attribute, not a field attribute. For more on that topic see “Dual Storage vs Dual Behavior“.
Ok, we’ve just gone down a deep rabbit hole. Maybe interesting, but is this important to reducing the overall memory footprint of your app? Generally no, sometimes yes. Let’s look at some examples.
Sometimes fields that are strictly numeric get inadvertently loaded as Dual() values. Certain operations including where clauses in loads can cause this behavior. Frankly, I don’t have comprehensive understanding of all the ways this can happen. But when it does happen, we can see numeric fields having a symbol size of more than 8.
The typical way to return these fields to 8 bytes (if you can’t fix the root cause of the issue) is to wrap the field with Num() or +0 when loading.
Num(UnitPrice) as UnitPrice UnitPrice+0 as UnitPrice
I prefer neither approach. I typically use a TempFormat table instead.
Is it worth the trouble to fix these? At first glance it may look like a big deal, but the memory savings is typically small.
The UnitCost field occupies 40MB and the symbol width should be 8 instead of 13.45. Almost all of the storage is in the Index Bytes. The field has a relatively small amount of values. Making those 2,011 values 8 bytes each would save me a total of 10967 Symbol Bytes. (27055 – (2011 * 8))
All that said there are occasions when you have a larger number of distinct values and this can make a significant difference. I almost always “fix” these fields without giving it too much thought as it’s easy to do with a tool like QSDA.
Reduce the number of rows
Reducing the number of rows that reference a field decreases the Index space, but not the Symbol space.
Wait! Are you suggesting I load less data? No, I’m talking about data model changes that will not change the business value.
A typical opportunity to reduces rows is when you have very sparse data on a large fact table. Consider an OrderDetail table that includes return information on each Order Line. Only 0.5% of Order Lines get returned, so we end up with a lot of Index space pointing to nothing. This is indicated by a low Information Density for the table field.
QSDA Table Fields
These four Return fields in the SalesOrderDetail table require 23.28MB.
By moving these fields to a new “Returns” table, linked by SalesOrderDetailID, the Return fields and the linking key now require 227KB — a savings of 23MB, a relatively significant amount.
I hope you found some useful tips in this post. If you want to learn more about how QSDA Pro can help you optimize your Qlik Apps, join me on Nov 19 for the Click Less, Qlik More – QSDA Pro webinar or reach out to us.
Longish post today where I show off the power of QSDA Pro to reduce Qlik App resources, saving you money and creating a more responsive experience for your users.
As a follow on to my last post “Help! My Qlik Cloud Reload Exceeds Quota!” I’m going to dive deeper into the available QSDA features to reduce memory requirements of a Qlik App. As a bonus we’ll use QSDA to improve the performance of a chart.
There are many good reasons to optimize your Qlik Apps to a leaner size. Smaller apps tend to perform better and place less demand on your server resources. If you are a Qlik Cloud customer, or planning a move to Qlik Cloud, App memory size impacts your license tier — and therefore your direct monthly costs.
In this post our starting point is I that have identified the App “BikeShop Sales” as having a large (597 MB) footprint. I got this information from Qlik App Metadata Analyzer as shown below.
Qlik App Metadata Analyzer
I could also have identified this memory footprint in QSDA Pro when we select this App from the list.
QSDA Applist
Let’s see how QSDA Pro can help. I’ll launch a new Analysis and give it the Description “Baseline”.
When the analysis completes press “View” to see the results.
QSDA Analysis Summary page
In the Resources card I see that unused fields occupy 383 MB — almost 2/3 of the base data model! Unused fields are fields that are not used anywhere in the front end. They can be removed without impacting the contents of charts or the user experience.
I also see some other suggested savings in dropping unused tables (“All fields unused but keys — 439 KB”) and a 530 KB savings by AutoNumbering keys.
A fast and simple way to remove unused fields is to drop them at the end of the script with a “DROP Fields …” statement. AutoNumbering is also best done at the end of the script. Let’s implement the QSDA recommendations. Open the Qlik Data Load Editor for “BikeShop Sales” in another browser tab.
The QSDA Tools menu provides a handy Script Generator to generate recommended script statements. Select the script generator from the Tools menu.
QSDA Tools menu
In the Script Generator > AutoNumber tab select the top checkbox to select all recommended fields. Press the “Copy” button to copy the generated AutoNumber statement to your clipboard. Paste the statements at the end of your Qlik Script. Repeat for the Drop Fields and Drop Tables tabs. (I won’t use the Load Statement tab just yet).
Now that I have updated the script with the AutoNumber and Drop statements, Reload.
Reload with AutoNumber, Drop Fields, Drop Tables
Returning to the QSDA App Card, I can see that Base App Memory is now 213 MB. Quite the savings from 507 MB!
App card after reload
At this stage, I recommend running a new analysis to confirm that the script changes have not introduced any problems or new issues.
Note that Reload Peak Memory is unchanged, because we loaded the unused data before we dropped it. Reload memory has an impact on the performance of on-prem servers and for Qlik Cloud, reload memory is a direct cost.
Can we use less reload memory by not loading the unused fields and tables? It may be that a field is used the load script but is not used in the front end, so you have to examine your script to determine. This very clean script loads directly from QVDs so the answer is clear.
I’ll start by commenting (or removing) the tables we don’t need instead of dropping them at script end. To identify the unused tables I can refer to the previously generated script, the Flags, or return to the Script Generator tool in the Baseline analysis.
What about Tables with both used and unused fields? Can I easily load just the used fields and omit the unused fields? QSDA Script Generator to the rescue again. Open the Script Generator > Load Statement in the Baseline analysis and sort on the “Unused Bytes” column. Select the SalesOrderDetail table.
A Load statement fieldlist for [SalesOrderDetail] will be created with unused fields commented out. You can exclude the unused fields entirely by unchecking the “Include Unused Fields” box. I’ll start with just the SalesOrderDetail table for now. Note that the other tables with big Unused Fields savings, Notes and Customers, are already being excluded completely.
Before reloading, I’ll remove the AutoNumber and Drop statements I added earlier. Many of those drops are no longer valid as I am not loading the fields at all. Here is the updated load script.
Updated script to exclude unused fields
After reloading, I return to the QSDA App Card. Reload Mem is reduced from 693 MB to 268 MB! I’ll run another analysis to see where things stand.
I can address the remaining unused fields with DROP Fields or using the Load Statement method. I will add the AutoNumber statements back in as well. I hope the process is clear enough now so I won’t show these remaining steps in detail here.
Let’s turn our attention to another great value of QSDA — Performance improvement. The “Calc Time Distribution” chart plots the calculation time of each chart in the App. I see I have an outlier that takes about 4 seconds to calculate.
What is this chart and why is is taking to long to calculate? Flip over to the Viz page and sort by the Calc Time column. The viz in question is a Table named “Salesperson Scorecard” on sheet “Dashboard” and it takes 4.142 seconds to calculate. Expanding the row shows the chart Dimensions and Measures and also exposes the “Viz Deconstruction” icon
Viz Details
Clicking the Viz Deconstruct icon will calculate each chart Measure in turn and tell us what is taking so long to calc in this chart. We can see the total calc time is over four seconds and measures three and four take considerably longer than the first two. But why?
Let’s focus on this chart to get some insight into the cause and solution of the poor performance. Back on the Viz page, Right-click the Viz of interest and select “Filter to Selected”. This will filter our pages to only those objects — Dimensions, Expressions, etc — associated with this Viz.
Click over to the Expressions page and we’ll see that our long running expressions have flags. Flags are the advice system of QSDA, identifying errors, bad practices and performance issues. Our two second expression “Longest Deal” has a “Key Field used in Expression” flag.
Clicking the “i” icon next to the flag will open the Flag Details panel which contains a link to the help article for this flag.
In the help article we’re informed that using a key field in an expression can sometimes cause poor performance. The article goes on to advise using a non-key field instead, creating a new field if necessary. I’ll take this advice and create a new field in the SalesOrderHeader table:
Reload and change the chart Measure to use the new field.
Max(Aggr(Sum(LineSales), SalesOrderRecord))
Run a new analysis to check the calc time of the chart (and check we didn’t impact anything else!). In the summary we’ll see a significant decrease in total app calc time and we can drill into our chart to see what’s changed.
Whoa! That measure has gone from 2 seconds to 0.008 seconds.
But Rob, didn’t I just blow up my RAM by making a copy of this big field? Don’t guess, use QSDA to see exactly what the cost is.
Additional RAM of SalesOrderRecord field
The cost is 50 KB of RAM and no significant increase in Reload Mem. Was it worth it? I think so.
If you chose to put the new field on the SalesOrderDetail table instead of SalesOrderHeader, the QSDA results will quickly show you that this was a bad idea. RAM is much increased and performance not so good.
QSDA Pro gives you the tools to:
Reduce your App memory and compute requirements.
Focused advice and insight for improving your App.
Compare different solutions and measure the impact of change.
Track your progress and document the impact of your work.
I hope this post makes it clear why I say that QSDA Pro is “the tool that every Qlik developer needs”.
Do you have questions about QSDA Pro or want to schedule a demo for your team? Reach out to us.
If you’re attending Qlik Connect in Orlando, stop by the Motio booth on the show floor. I’ll be there (guy in the hat) to answer your questions and show you even more features of QSDA Pro that can help you create leaner, cleaner and faster Qlik Apps.
If you are a Qlik Cloud (SaaS) customer you may have seen this dreaded message during script reload. Ack! Ack!
Your quota may be different than 25Mb, but the problem remains the same. How do I modify this application to fit within my subscribed quota?
QSDA Pro V3.1 to the the rescue! QSDA analyzes your app and provides recommendations and easy to use tooling to drop unused data and optimize the remaining data, potentially greatly reducing the size of your Qlik App.
QSDA needs a successfully loaded app for analysis, so our first step is to reload the app using the “Limited load” feature of the Qlik script debugger.
Click the Debug button in the Qlik script editor, check the Limited load option, accept the default of 100 rows and click “Run” to initiate the reload. The reload will come in under your quota. When the reload completes, key Ctrl-s to persist the data.
You don’t have a business-ready app, but this is enough for QSDA to scan all charts, expressions and master items to determine what data is actually required.
In QSDA Pro we now select our Cloud connection and select the app of interest, in this case “BikeShop Sales2”. Click the “Open” button to prepare the app for analysis. I will also uncheck the “Collect Calctime” option because chart calculation times are not meaningful with our limited data. Description is optional, in this case I’ve entered “Limited Load”.
Click the “New Analysis” button to launch the QSDA Pro analysis. The analysis may take a minute or two to complete depending on the capacity of your tenant and the current Qlik Cloud load. When the analysis is complete press the “View” button to see the results.
In the Summary view we see there are 89 unused fields. Unused fields are fields that are not used in any charts, master items or expressions. These are fields that can be dropped at the end of script without impacting the user experience in the app. The actual storage numbers (KiB) are not meaningful because we only loaded 100 rows. The bar chart is useful in that is indicates about 2/3 of our data is unused. Hope!
QSDA provides a Script Generator in the Tools menu to generate a “DROP Fields …” script statement for unused fields. Select the script generator from the Tools menu.
In the Script Generator > Drop Fields tab select the top checkbox to select all recommended fields. Press the “Copy” button to copy the generated Drop Fields statement to your clipboard.
Paste the Drop Fields statement at the end of your Qlik Script and reload.
Reload successful! Victory!
A subsequent QSDA analysis shows the app size has been reduced from the problematic 37Mb to 15MB!
QSDA Pro can quickly and efficiently get your Qlik Apps below your Qlik Cloud quota by pruning unnecessary data. Don’t pay for more than you need.
In a follow up post I’ll walk through some additional QSDA features to help you reduce Qlik app footprint even further and improve chart response time to boot.
Download QSDA Pro and try it for yourself. Or reach out to us to learn more about license options or arrange a demo or POC for your team.
-Rob
Are you going to Qlik Connect? I’ll be at the Motio booth on the show floor ready to demo QSDA Pro or answer any questions you may have.Or just to say Hej 🙂
QSDA Pro 2.6 is available now and a new feature is Flag Levels. Each flag type is assigned a level of Info, Warning or Error with a matching color. Color helps you focus on what’s important.
Flag Levels may be customized and refined for your requirements using the new Admin > Project Config page.
“Missing Theme“. The default level for this flag is Error. A missing theme can occur when moving an App from one Qlik environment to another.
Want to learn more about QSDA Pro? Are you QSDA Pro curious or want a refresher on the capabilities and value of QSDA Pro to your Qlik development team? Join me on 24 January, 2024 for “Accelerate Your Qlik Development Through Automation” where I’ll walk you through the features of QSDA Pro and show you how QSDA can help you produce faster, leaner and cleaner Qlik Apps.
Summary: Set Modifier field names and Set Identifiers are not validated by Qlik syntax check. QSDA Pro 2.5 validates everything and raises an “Unrecognized Name” flag for invalid names.
QSDA Pro syntax checks all expressions in a Qlik App using the Qlik syntax checker. You’re familiar with the Qlik syntax checker. It’s the message at the bottom of the expression editor that tells you your expression is “OK”.
The syntax checker is also good at telling you when the expression has an error, although it’s not always specific about the problem.
QSDA Pro, using the Qlik API version of syntax checker, tells you what is wrong with the expression:
The Qlik syntax checker has a significant limitation. It does not validate field names used in sets. Whether used in the API or the editor, the syntax checker will not raise an error if the field “EmpPurchase” does not exist in the data model.
This lack of validation can be a serious problem if “EmpPurchase” is removed or renamed in the data model. The expression will still return a number but the exclusion of employee purchases will no longer be applied. That could be a very subtle but important fail.
Recognizing this limitation and it’s potential impact, QSDA Pro 2.5 validates all names used in sets and will raise a new Quality flag, “Unrecognized Name”, if the name is invalid.
Another place this validation is extremely useful is Chart Filters in Qlik SaaS. The chart will cheerily display that filter “Expression2 > 1” is being applied. Even though field “Expression2” no longer exists in the data model.
But QSDA knows.
Ever use a bookmark as a set identifier? And then the bookmark got deleted or renamed?
I’ve used simple examples to demonstrate, but where this validation really shines is in longer expressions where it may be easier to overlook that something has gone wrong.
What does QSDA stand for? Qlik Sense Document Analyzer. The combination of data model and sheets we know as an “App” in Qlik Sense was called a “Document” in QlikView. When I first created this tool in 2009 to help maintain QlikView, I called it “Document Analyzer”. When it came time to create a similar tool for Qlik Sense, I stuck with the “DA” name as the function and usefulness of “Document Analyzer” was well established in the Qlik community.
Summary: While looking into long expressions I noticed that the optional sort-weight argument has an impact on the distinctness of Concat(distinct…). Incorrect use of sort-weight can generate bloated expressions containing redundant code.
In my work tuning Qlik Apps I sometimes encounter very long expressions. An expression many thousands of characters long can be difficult to debug or comprehend the expression goal. To help in working with long expressions I’ve added an Expression histogram and an Expression Decoder feature to my QSDA Pro product. (These features are currently in beta, generally available in early Feb).
I’ve noted expressions of length greater than 50k across apps from different customers. What did these expressions have in common that made them so large?
They used the Concat() function in $() to dynamically generate a part of the expression.
They used the optional sort-weight argument of Concat() incorrectly.
They were much bigger than necessary — sometimes 100x — but the expanded expression worked as intended.
In the process of reviewing the expressions I learned something surprising. As a reminder here’s the syntax of the Concat function:
This means that the set of distinct values is the combinations of string and sort_weight (if used). Let me demonstrate with an example. Here’s a sample data table.
For the expression: Concat(Dim, ',') we receive output "a,b,c,c,c“.
Adding the DISTINCT keyword: Concat(DISTINCT Dim, ',') we now get “a,b,c“.
Adding a non-distinct sort-weight argument: Concat(DISTINCT Dim, ',', RecId) we now get "a,b,c,c,c” again. More output than I expected. It’s a distinct list of the combinations of Dim and RecId.
Adding a distinct sort-weight argument: Concat(DISTINCT Dim, ',', Weight) we now get "a,b,c“.
How about if we used an unlinked data island field for sort-weight? The Island field has two values.
Concat(DISTINCT Dim, ',', IslandField) returns "a,b,c,a,b,c“. Item count is the product of Dim * IslandField values. Remember this for later.
Ok, this is all very interesting but the behavior is super obvious and I would notice it if it came up in my App. What’s this got to do with ginormous expressions?
Developers sometimes use Concat along with Dollar Sign Expansion (DSE) to generate dynamic expression fragments. For example to ignore all fields from several tables in a set modifier:
Sometimes $(=Concat(...)) is used to build the list inside a Pick() or Match(). These type of expressions frequently have awkward syntax including lots of “& chr(39) &” type stuff. Inevitably the expression gets built by copying and modifying an expression from elsewhere in the App. An expression that contains a sort-weight. A sort-weight that doesn’t get removed. It may be an island field or a field that has a many to one relationship. The result is an expanded expression that works but is larger than it needs to be. No one notices (unless they use QSDA Pro) because it’s the expanded expression.
As a simple example, suppose the “ignore filter” expression above was supposed to generate something like "Quarter=,Month=,Year=“. If I inadvertently use a sort-weight field that has 100 distinct values the result will be repeated 100 times. The expression would still work but it would be 100x larger than necessary.
I recently found an example where Concat was used to generate an If() function from data (very clever) that should have had 15 branches. But an unrelated sort-weight field of 95 values resulted in 1425 branches! It “worked” but did a lot of unnecessary calculation.
If you are a solo developer or working in a small team you may never encounter this issue. But if you are a consultant or maintaining legacy Apps you may stumble across it. I’ve been playing with ways to flag this condition in QSDA Pro. QSDA already flags data island expressions . I’m testing creating a new flag specifically for Concat().
The QLIKLOGIN connection method allows your QSDA users to connect to Qlik Sense using the same login process they use for the Hub — Windows, SAML, whatever. After logging in, user “Sally” will see only her apps — the same list Sally would see in the Qlik Sense Hub.
The “Filter using this flag” button filters all your resources to everything associated with this flag, giving you fast focus on what needs to be examined or remediated.
Hyperlinks in resource lists provide one click to open the subject Viz in the Qlik Sense sheet editor.
Still my favorite QSDA feature, Viz Deconstruction itemizes the calculation time of each individual expression in a chart so you can focus in on what the heck is taking so long in this chart.
Stuff Administrators Will Love
Password protect Admin pages
Offline license keys for air gapped servers
Possible to install QSDA without Windows Admin rights
Https support
Optional API security
What Can QSDA Pro Do For Your Team?
Expose undetected errors — before your users do!
Gives actionable advice
insight and focus for developing and debugging
Save time on tasks like “how/where are we using field ‘abc’?”
Join a Webinar
“Introducing QSDA Pro” will show you the features and benefits offered in QSDA Pro, as well as an opportunity to ask questions.
I my last two posts I introduced QSDA Pro and the flag workflow. In this post I’m going to show how you can use QSDA Pro to quickly uncover quality issues in your Qlik Sense App.
QSDA groups flags by category, and the “Quality” category is where I usually begin my app review. Quality flags are used to indicate where something appears to be seriously broken such as:
An expression syntax error.
A reference to a Master Measure that no longer exists in the library.
A missing extension.
A bookmarked field that is no longer exists in the data model.
These are the kinds of things you want to discover before your users do!
It’s relatively easy to create new errors in an existing Qlik app. Qlik does not warn or block you from deleting a resource like a Field or Measure that is in use.
A properly done field rename may update names in expressions. But it will not update variables or bookmarks. And I’ve seen plenty of field renames go wrong and break expressions as well.
It can be difficult to detect all app errors by visually checking your app. The broken piece may be a color expression or a calculation condition that isn’t readily visible.
So yes, easy to break, sometimes hard to detect and heros find problems before their users do. Enter QSDA Pro.
In QSDA menu, Help -> Flag List will display a list of all potential flags in the installed version. Here’s the current list for Quality (The “{}” bits are values that will get filled in when the flag is created.)
Some of these are fairly straightforward, some a bit more subtle.
Why would you have a missing Master Dimension or Measure? A fairly common scenario is not understanding that Master Items are referenced by internal Id, not by Name. So you create something called “Sales” and use it in several charts. Later you are doing some exploration and create something called “Sales2” which you like better. So you think if you delete “Sales” and rename “Sales2” to “Sales” everything will be using the new “Sales”. Wrong.
Because the master item is now gone, I can’t tell you what used to be in this master item. You will have to rely on an app backup. Or…if you have a previous QSDA Analysis for this app you can look up the master item definition there!
All Expressions and Dimensions are validated using the Qlik syntax checker. A typical error might be a bad fieldname (perhaps because the field was removed from the model).
Flag Details will give us a detailed error message, location of this expression (Master Library) and importantly the Use Count. This use count is zero so we know deleting the measure is a viable option.
QSDA uses the Qlik syntax checker so the test is only as good as what I get from Qlik. There are some limitations, for example when column labels are used in an expression. You’ll note that the Qlik expression editor flags this as invalid so QSDA will pass this message on.
The “Parse Error” flag is raised when Qlik Syntax says the expression is ok but QSDA finds the expression does not conform to expected syntax rules. This could be a problem with the QSDA parser (a work in progress) or it could be Qlik tolerating something unusual. For example, do you think this is a valid expression?
sum({<Sales={">10 between < 90"}>}Sales)
The Qlik syntax checker will declare it valid. And it will return an official looking result — which may or may not be correct. But what is that “between” keyword? (if you’re curious why “>10 के बीच < 90” also seems to work, see https://masterssummit.com/something-to-get-confused-about/).
Nervous about refactoring or changing your apps? Can you relate to “Change Paralysis” or Fear Driven Development (FDD)? We all need a tool like QSDA Pro to proactively plan changes as well as audit after the fact for unintended errors.
Download QSDA Pro now and analyze some of your own apps. QSDA Pro is free to use during the beta period.
You can analyze published and unpublished apps, so go ahead and take a look at some of those production apps. Let me know in the comments if you find something interesting!
In my first post introducing QSDA Pro, I told how Document Analyzer was a tool I developed for use in my own consulting work and shared with the community of Qlik developers. The Qlik Sense QVF version got a big boost in utility when Axis Group joined the project and contributed their many many years of Qlik experience.
In this post I will introduce the Flag workflow of QSDA.
QSDA communicates it’s advice through “Flags”. Flags are issues that warrant your attention.
The Summary page gives an overview of Flags for this analysis. Clicking any of the Flag boxes or Flags in the Nav Ribbon will take you the Flags page.
The Flags page provides a collapsible list of all flags and buttons for more detail and flag status.
Flag text identifies the error and in most cases provides some detail, such as a fieldname or number of bytes that could be saved.
The dimension failed validation. The error is: Bad field name(s) "[PostalCode]" .
Click to show the Flag Details panel.
The details panel will provide more context for the flag to help you evaluate your response. Depending on the flag, there may be additional hyperlinks. Click the “?” icon to link to the Help article for this flag type.
To support a workflow of “checking off” the flags, buttons and flag status indicators are available in the both the Flag Details and Flag List views.
Mark this flag instance status as “Resolved”.
Mark this flag instance status as “Hidden”.
Resolved and Hidden flags are not shown in the list unless the filter is activated on the filter bar. Flags with no status are “Unresolved” and are always shown.
Buttons will automatically toggle to allow you undo a status. You can set/unset statuses on individual flags or at the flag type level. Hover over a button to discover its function.
From the Flag List you can view all flags. In the resource pages, you will also see the same flags but attached to the resource. For example, on the Dimension page, any associated flag will be shown with the specific Dimension.
These workflow features allow you to punch through the list of issues in your app, hiding those you don’t care about and checking off issues as you resolve them.
