If you’ve ever seen my presentations on automated testing of Qlik apps, you know I’m a big fan of the free Qlik Scalability Tools — both QV and QS.
I use the Regression Test feature to automatically validate Qlik app changes before promoting to production. Regression testing allows us to validate that charts continue to produce correct results after updates are made to the application or platform software.
This graphic provides a brief overview of the regression testing concept.
The Qlik Sense Scalability Tool (QSST) provides a GUI workbench to run and analyze regression tests. The workbench is a great tool for a developer or QA Analyst to validate an app on an adhoc basis.
What if I want to run the analysis unattended on a schedule? For example, after every reload to validate that my app is still working correctly and my expected data was loaded.
In this post I’ll demonstrate using a system monitor to run the regression test as an ongoing validation and send an alert if the validation fails.
I won’t go into the details of setting up the Scalability Tool. You can get that from the product documentation. If you would like me to do a presentation on automated testing for your team contact me here.
We will need a system monitor that is capable of running commands on a schedule, monitoring the command output and sending alerts or status via something like email or Slack. You may already have a capable system monitor installed at your site.
NodeGraph is my current favorite Qlik-centric testing tool for ongoing quality monitoring (I’m a partner). NodeGraph’s Test module allows for testing expression values, presence of fields and update times. At the moment NodeGraph does not support running QSST Regression tests, but I expect it will be part of the product by 2018Q4.
For the time being, I’ll demonstrate batch regression testing using Woodstone’s Servers Alive, a low cost but powerful system monitor I’ve used in the past. You can try Servers Alive for free.
Let’s assume we have used the QSST workbench to author the regression test scenario and have executed the scenario to create a baseline log. When using the workbench, we would re-execute the scenario at a later time to create a compare log and then analyze the differences between the logs using the workbench Analysis tool.
QSST provides command line versions of both the scenario executor (“SDKExerciserConsole.exe”) and the regression log analysis (“RegressionAnalyzerConsole.exe”) . We will need to bit of script to stitch these two operations together and process the output.
I’ve posted a powershell script here that wraps all the necessary operations and exits with an ERRORLEVEL of “1” if validation fails (differences found between compare and baseline logs), or “0” if no differences are found.
- Download the RunRegressionTest.ps1 script and place it in a directory named “Regression”.
- Create a subdirectory under Regression named “baseline”.
- Copy your baseline log file for each app of interest to the baseline directory. Do not rename the file.
- Following the comments in the file, update the first three variables in RunRegressionTest.ps1 to identify the location of your QSST install and your scenario json file(s).
We can check that the script runs correctly from a powershell command prompt in the “Regression” directory.
.\RunRegressionTest.ps1 "ABC Sales Demo"
where “ABC Sales Demo” is the name of a json file in our scenario directory. If all goes well, we should see output messages like:
PS C:\QlikSense-Projects\Regression> .\RunRegressionTest.ps1 "ABC Sales Demo"
Comparing C:\QlikSense-Projects\Regression\temp\results\ABC Sales Demo_localhost_[1-0-1--1]_18072311263304_regression.log
to baseline: C:\QlikSense-Projects\Regression\baseline\ABC Sales Demo_localhost_[1-0-1--1]_18072216105188_regression.log
Differences found
Now to implement this in Servers Alive. In Servers Alive we define things to test as “checks”. In this case I’ll use the External(errorlevel) check type and provide the command string to run my powershell script.
The full command string is: (no line wrap)
powershell -NoProfile -ExecutionPolicy bypass -File "C:\QlikSense-Projects\Regression\RunRegressionTest.ps1" "ABC Sales Demo" "p2"
where “ABC Sales Demo” is the name of my scenario json file. “p2” is a dummy parameter that works around a windows bug. Without that extra parm, a blank space is added to the previous parm causing a failure.
On the Alert tab I’ll specify to send a message to Slack on failure. On the schedule tab I’ll specify when to run this check.
All set up! When run either manually or on schedule, a failed check — that is, the regression test found unexpected differences — the check will display red on the SA console and I’ll also receive a Slack message.
I’ve just introduced a number of moving parts. If this is all new to you what I hope you’ll take away is that formal, structured and automated monitoring of your Qlik Applications is possible with relatively low cost and effort.
Want some assistance with planning your testing strategy and implementation? I offer consulting in planning and implementation of automated testing and monitoring for your Qlik environment. Contact me if you want to chat about your needs and how I can help.
I also typically do an optional lunchtime session on testing at the Masters Summit for Qlik. Come along to Philadelphia or Johannesburg to talk testing and many many more Qlik topics.
-Rob
