“Missing Theme“. The default level for this flag is Error. A missing theme can occur when moving an App from one Qlik environment to another.
Want to learn more about QSDA Pro? Are you QSDA Pro curious or want a refresher on the capabilities and value of QSDA Pro to your Qlik development team? Join me on 24 January, 2024 for “Accelerate Your Qlik Development Through Automation” where I’ll walk you through the features of QSDA Pro and show you how QSDA can help you produce faster, leaner and cleaner Qlik Apps.
Summary: For the data viz nerds among us, I share my fascination with a groundbreaking data viz project at the 1900 Paris Exhibition.
In Tim Harford’s excellent book “The Data Detective: Ten Easy Rules to Make Sense of Statistics” the author mentions a set of infographics produced by W.E.B. Du Bois and assistants for the 1900 Paris Exhibition. The “Exhibition of American Negroes” is a powerful collection of photographs and data visualization that provide a strong counter to racist propaganda and ignorance of the time.
I encourage you to explore the entire contents of the exhibit, but in this post I’m going to highlight some of the data visualizations. You can browse the collection here, the charts are on the first two 100 entry grids.
Du Bois and his Atlanta University students made 63 hand-drawn diagrams. The data visualizations were created on presentation cardboard and conceived for a temporary installation at the fair. The original diagrams still exist and have been displayed in various museums.
The hand drawn nature of the charts makes them feel very personal and they tell their stories very well. I expect that the team was pushing the boundaries of data viz knowledge and technique of the time.
This multivariate diagram is one of my favorites
There are charts covering many aspects of African American life — professions, geography, income, property — and importantly showing change over time. This is a chart comparing marital status of African Americans to Germans. Why Germans? I’ll let you research that.
Measured against our current standards for data viz, we could make some criticisms about choices of shape and scale, but I don’t see any dishonesty. One of the design choices I found amusing was what to do when the bar size exceeded the available space. Wrap it!
I hope you find this bit of data viz history as fascinating as I do. I’ve provided some additional reading links below. Again, you can browse the collection at the Library of Congress. I’m curious to hear which visualizations you find most interesting and why.
If you or one of colleagues would be interested in attending one of these Summits, and believe you can get approval for the expense we would ask you to kindly pre-register now.
We will not be taking any payment at this time and you should not make any travel arrangements.
If you pre-register on or before January 15, 2024, you can attend the event for a specially-discounted priceof $1800 (that’s a 25% discount off our usual early-bird price)
This is your opportunity to help shape the future of Masters Summit events and attend with a deeply-discounted price. If we determine enough of you are interested in returning, based on pre-registration, we will move forward with our 2024 plans
See the new Sessions below.
Managing your tenant
Setting up development and promotion flows
Approaches for managing data.
Monitoring your tenant
Migrating to Qlik cloud
Planning your migration
Available tools, and how to use them.
Performance Tuning and Capacity Management
Best practices for performance and data scalability with the Qlik capacity licensing in mind.
Review opportunities in data modelling and visualizations to improve performance.
Review available tools for monitoring performance.
Real-world AutoML Applications
Understand machine learning predictive models and what business problems they can tackle.
Profiling data for ML, basic statistics, feature engineering, avoiding common pitfalls.
Evaluating and explaining your models using AutoML results
Deploying and monitoring your model
Building reports from exiting sheets or objects
Output presentation quality reports in various formats including annotations.
Automating the distribution of reports to a variety of delivery channels.
Review existing report connector blocks.
Discover the self-serve options for subscription and alert-based reporting.
Automating Analytics Operations and Automation
Dynamic Actions and Business Process Integration
Going beyond: Qlik-CLI and the Qlik APIs
Qlik Cloud Integration
Understand what APIs are available, the necessary skill sets required to develop with them and how to correctly structure your project architecture.
See the art of the possible with a mix of real-world solutions and innovative demos.
Discover useful tools to help speed up development and reduce costs.
Get tips and tricks for the dark art of authorization and authentication in both client and server implementations.
As I was updating my incremental load materials for the upcoming October/November Masters Summit for Qlik, I ran across an old note about creating “Attributes” in QVDs using Tags. Due to a bug in QVD generation I couldn’t implement the idea at the time. But the bug is long since fixed so here I present the general idea of “QVD Attributes” (my name, not an official Qlik thing).
We can assign Tags to Qlik Tables using the script TAG statement:
Tag TABLE TranData with 'MaxId=$(vMaxId)'; Tag TABLE TranData with 'Source=Salesforce';
If the table “TranData” is subsequently written to a QVD using the STORE statement, the tag will be included in the QVD file.
We can read tags directly from a QVD with script:
LOAD String%Table as Tag FROM [lib://mylib/TranData.qvd] (XmlSimple, table is [QvdTableHeader/TableTags/String]);
Note that we are reading only the XML header from the beginning of the file. This is very fast.
If we use a structured tag format like “key=value” we can create a generalized “Attribute” store in a QVD that can tell us something about the contents of that QVD. That could be useful when we want to use the QVD in a subsequent script.
How do we retrieve these tags in a useful and consistent way? I think loading them as variables would be a good approach. We can create a reusable subroutine to load all “key=value” tags from a QVD and generate variables named “Tag:key”. We then call the Sub, using the path to any QVD:
We might utilize the new variables in something like an incremental Load.
SQL SELECT * From Trans Where Id > $(Tag:MaxId);
Or as a control to conditionally execute a script block.
If '$(Tag:Source)' = 'Salesforce' Then // Use the Salesforce Load statement EndIf
At the bottom of this post is a complete sample including the subroutine. You only need to change the “QVDFiles” to your lib to run this on your system.
// Subroutine to Load Tagname=Value pairs from a QVD as "Tag:Tagname" Variables.
SubField(Tag, '=', 1) as TagName,
SubField(Tag, '=', 2) as TagValue
LOAD String%Table as Tag
(XmlSimple, table is [QvdTableHeader/TableTags/String])
Where String%Table like '*=*';
For _i = 0 to NoOfRows('_TempTags') - 1
Let _vname = 'Tag:' & Peek('TagName', $(_i), '_TempTags');
Let [$(_vname)] = Peek('TagValue', $(_i), '_TempTags');
Drop Table _TempTags;
// END of Subroutine
// Generate some sample data
RecNo()*2 as Id,
Rand() as Amount,
'A' as Customer
// Something we may want to record from the data is the max Id value, for later incremental load.
LOAD Max(Id) as MaxId Resident TranData;
Let vMaxId = Peek('MaxId');
Drop Table TempId;
// We can tag with $(variable) values or literal values.
Tag TABLE TranData with 'MaxId=$(vMaxId)';
Tag TABLE TranData with 'Source=Salesforce';
// Some other tag that is not part of our name=value scheme.
Tag TABLE TranData with "ABC";
// STORE and DROP the QVD
Store TranData Into [lib://QVDFiles/TranData.qvd] (qvd);
Drop Table TranData;
// Call our subroutine to load tags as variables, from the disk QVD.
// Demonstrate how we might use these tag values.
// In a SQL Where clause.
SET select = SQL SELECT * From Trans Where Id > $(Tag:MaxId);
// In a conditional to select a block of script.
If '$(Tag:Source)' = 'Salesforce' Then
Trace Going to use the Salesforce load statement;
Brian will share some inside tips on using Qlik Application Automation and I expect a few interesting tricks. I’m hoping he shows off the capabilities of his WhatsApp for Qlik integration!
Masters Summit for Qlik is the premier Qlik Sense & QlikView advanced education event. In three packed days of lectures and hands on exercises, you’ll learn from experts and take your Qlik Dev skills to the next level in data modeling, scripting, expressions and visualization. We’ll also cover performance tuning and Qlik API Integration for beginners. You’ll come away with a wealth of new knowledge, code samples and documentation.
Find out what over 1500 of your colleagues have learned already.
I frequently create Date groups in my Qlik apps to facilitate selections like “Yesterday” or “Last Week”. Some years ago I published a pattern for creating Date groups in the Recipes section of QlikCookbook.com and it’s been a very popular download. I’ve responded to a few requests on Qlik Community looking to do the same thing for Time groups, so I decided to publish a recipe for Time Grouping as well.
The recipe demonstrates selecting ranges such as “Day Shift”.
How do we create ranges that map to our data? In the script, create a Range table that defines range Names, Start and End times.
Ranges: LOAD Range as Range, Time#(RangeStart, 'hh:mm:ss') as RangeStart, Time#(RangeEnd, 'hh:mm:ss') as RangeEnd INLINE [ Range, RangeStart, RangeEnd Midnight Shift, 00:00:00, 05:00:00 Early Morning, 05:00:01, 09:00:00 Daylight, 06:00:00, 17:59:59 Day Shift, 09:00:01, 17:00:00 Early Evening, 17:00:01, 20:00:00 Evening, 20:00:01, 23:59:59 12am-6am, 00:00:00, 05:59:59 6am-12pm, 06:00:00, 11:59:59 12pm-6pm, 12:00:00, 17:59:59 6pm-12am, 18:00:00, 23:59:59 ];
Then use IntervalMatch to link the time field in our data (EventTime in this example) with the Range field.
JOIN (Ranges) IntervalMatch (EventTime) LOAD RangeStart, RangeEnd RESIDENT Ranges; // No longer need RangeStart, RangeEnd, but may keep them for documentation or debugging. DROP FIELDS RangeStart, RangeEnd;
There are additional notes provided in the downloadable example, but that’s it for the basic pattern. You can download the example from here Time Grouping
You may have noticed I used Time#() to read the text time values. That makes it easy to write the range values as readable hh:mm:ss.
If you are generating the ranges using another technique like Interval#() or division, be mindful of rounding. A Qlik time value is represented numerically by the fraction of 1, a whole day. 0.5 is 12:00:00 PM, halfway through the day.
There are four methods I’m aware of to generate a Qlik time. For example to generate the numeric time value corresponding to 12:00:05 AM:
All four will generate a floating point number to represent the time. Will they produce exactly the same result? Results that would pass an equality test?
Time#() = Interval#() = MakeTime() = x/y?
For some inputs they are equivalent. For a significant number of inputs the odd man out is Interval#(). Given the 86,400 second values in a day, Interval#() will not match the others 36% of the time. Try sharing that interesting fact with your spouse at the breakfast table.
Join me at the Masters Summit for Qlik in Dublin or Orlando this fall to learn advanced scripting techniques for Qlik as well as inside tips. Our team of expert consultants will cover data modeling, set analysis, visualization techniques and Qlik integration as well as answer your “how do I” questions. Register now or get more information at the Masters Summit for Qlik website.
Summary: Set Modifier field names and Set Identifiers are not validated by Qlik syntax check. QSDA Pro 2.5 validates everything and raises an “Unrecognized Name” flag for invalid names.
QSDA Pro syntax checks all expressions in a Qlik App using the Qlik syntax checker. You’re familiar with the Qlik syntax checker. It’s the message at the bottom of the expression editor that tells you your expression is “OK”.
The syntax checker is also good at telling you when the expression has an error, although it’s not always specific about the problem.
QSDA Pro, using the Qlik API version of syntax checker, tells you what is wrong with the expression:
The Qlik syntax checker has a significant limitation. It does not validate field names used in sets. Whether used in the API or the editor, the syntax checker will not raise an error if the field “EmpPurchase” does not exist in the data model.
This lack of validation can be a serious problem if “EmpPurchase” is removed or renamed in the data model. The expression will still return a number but the exclusion of employee purchases will no longer be applied. That could be a very subtle but important fail.
Recognizing this limitation and it’s potential impact, QSDA Pro 2.5 validates all names used in sets and will raise a new Quality flag, “Unrecognized Name”, if the name is invalid.
Another place this validation is extremely useful is Chart Filters in Qlik SaaS. The chart will cheerily display that filter “Expression2 > 1” is being applied. Even though field “Expression2” no longer exists in the data model.
But QSDA knows.
Ever use a bookmark as a set identifier? And then the bookmark got deleted or renamed?
I’ve used simple examples to demonstrate, but where this validation really shines is in longer expressions where it may be easier to overlook that something has gone wrong.
What does QSDA stand for? Qlik Sense Document Analyzer. The combination of data model and sheets we know as an “App” in Qlik Sense was called a “Document” in QlikView. When I first created this tool in 2009 to help maintain QlikView, I called it “Document Analyzer”. When it came time to create a similar tool for Qlik Sense, I stuck with the “DA” name as the function and usefulness of “Document Analyzer” was well established in the Qlik community.
I’m pleased to announce that I’ve joined forces with Motio, Inc, producers of the great DevOps tools Soterre and Gitoqlok. Motio has acquired my QSDA Pro product and I’m excited to combine our tools to provide a comprehensive platform for professional Qlik App development in Qlik Sense and Qlik SaaS. You can read the announcement here.
I will continue to lead the development of QSDA Pro, enhancing the product and bringing new superpowers to my customers in the integration with Soterre and Gitoqlok.
The QSDA Pro customer base is growing fast! To date hundreds of organizations have purchased QSDA Pro and are realizing the full benefits of the tool. More are joining every day and I’m excited to add Motio’s dedicated product support and admin teams to handle this growth.
Gitoqlokis a free chrome plugin that allows you to use a git repository such as GitHub, GitLab, Azure DevOps, etc, to manage all the pieces of Qlik Sense app — Measures, Sheets, Charts, Load script and more.
The plugin operates seamlessly within the Qlik Sense authoring experience to provide the power of fine grained version control you expect with git. You’ll see a list of changes and commit those changes as you would expect in any software development project. Of course there is collision detection, branching and merging and all the goodness you expect from git.
Because the Gitoqlok team understands the visual nature of Qlik development, you can view diffs in a visual format. A slider lets you pick a commit for compare.
You can also use the same slider in a non-diff “time machine” mode to see how the application sheets looked at any point in time.
Gitoqlok comes with a lot of very useful Qlik dev features, such as the ability to import a chart or script snippet from app to another. Or deploy an entire app from one server to another!
Most Gitoqlok features are free to use. Premium features and support are available in a reasonably priced subscription.
If you are attending QlikWorld next week Drop by the Motio booth (#315) to see more. Motio is also presenting two breakout sessions showcasing Gitoqlok, including their integration with my QSDA Pro tool.
My Masters Summit for Qlik colleague Oleg Troyansky will be presenting his always valuable “Performance Tuning” course on March 29 as part of our “Masters Summit at Home” online series.
Oleg will do a deep dive into the causes and corrections for Qlik App performance issues. He’ll explain the why of good and bad performance while explaining important Qlik Engine concepts and demonstrating best practices. I highly recommend this course.
Oleg will demonstrate available performance tools including my favorite — QSDA Pro. As part of the class each student will receive a trial license key for QSDA and I’ll be available before and after class to help you get QSDA installed and configured.
Also coming up in the “Masters Summit at Home” series:
“Effective Visualizations” with Bill Lay on March 15. Bill will teach you how to effectively “tell the story” using Qlik visualizations in a series of interesting and challenging scenarios. Bill is an engaging and thoughtful presenter. I always come away from Bill’s lectures with fresh ideas and useful tools.