Category Archives: Scripting

Mind The Space

Here’s a heads up on a QlikView script syntax issue.  The problem came up several times in a beginner class last week and I’ve also noted beginners on the QlikCommunity Forum struggling with this as well.

There should be a space after a script keyword, separating it from the next word, right?

Resident  MyTab    // correct
ResidentMyTab     // incorrect

If we omit the space  the syntax coloring will alert us and we may get a red error underline as well.

What if a bracket is used, such as the case when a name contains a space?  If a bracket appears immediately after the keyword, the syntax highlighter will make look it correct. But it is invalid syntax and will fail  when reloaded.

There are a number of places in script where this problem can happen. Here are a few examples.

- LOAD * INLINE[
- [F1] as[F2]
- LOAD[F1]

Good news! QlikView Version 12 corrects the problem. In QV12, the missing space will be flagged in the editor.

 

While the requirement for a space may seem obvious to an experienced developer or programmer type, it can be a problem for beginners who are copying an example from a book where the space is not so clear, and who rely on highlighting to tell them they have it right.

It can be a difficult error to debug. The resulting script error message can be something  indirect like “missing/misplaced FROM..” when the actual error is a missing space after an “AS” keyword.

I’m going to add the following slide to all my beginner trainings:

Words in expressions or script must be separated by a delimiter. Depending on context, the delimiter will be one of:

Space  Comma  (  {  <  = +  -  *   / Semicolon

[ Single-quote and Double-quote are not delimiters.

I know this incomplete, leaving off tabs, newline, =>. Just keeping it simple.

-Rob

 

 

Share

Product Improvements For Variables?

On 2 Oct, I was pleased to join a group of eight Qlik Luminaries on a field trip (we had a bus!) to Qlik R&D in Lund, Sweden.  Ralf Becher has written a nice overview of the trip here.

One outcome of our meeting is a list of enhancements we believe would be valuable additions to QlikView and Qlik Sense.  You can view the entire working list here.

The list includes quite a few topics.  Today I want to focus on improvements to Variables.

A common challenge is”script only” variables getting inadvertently promoted to the UI.  There is a desire to better control and identify variables in the script.

What follows are a number of specific feature improvements for variables raised during the meeting.  The general theme is to extend the current management functions for Fields to Variables as well.

  •  A HidePrefix and $hidden Tag for Variables.  Hidden variables would not appear in the Variable overview or Expression Editor dropdown unless “Show System Variables” was checked.
  • A “DROP VARIABLES vg*” statement.  The DROP VARIABLE statement would delete a list of variables from both the script and UI. Script deletion of a UI variable is not possible today. Importantly, the list can include wildcards.
  • Script functions that provide access to  existing variables.  Similar to functions currently available for Fields, the functions would be:
    • NoOfVariables()
    • VariableName(nbr)
  • COMMENT VARIABLE varname WITH ‘comment’
    COMMENT VARIABLES USING mapname
    Currently the only way to add comments to variables is through the Variable Overview dialog or the VBScript API.  Comments are a useful feature and it would be great if comments could be set using script statements.

What do you think?  Would you use these additions?  Are there additional improvements you would suggest for Variables?

-Rob

 

Share

Better Calendar Scripts

TLDR: A new Qlik Sense training video uses a tired old-school Master Calendar script. I propose that updated training materials should use an updated script.

I just watched  a new  video by Michael Tarallo of Qlik titled “Understanding the Master Calendar – Qlik Sense and QlikView“.  Before I  pick on the contents of this particular video, I want to  credit  Michael for  producing many excellent training videos that are worth watching and learning from. I  highly recommend them.

The video does a great job of explaining the need for and function of a Master Calendar in your data model. It then goes on to show an actual script.

I can’t discuss Master Calendar without expressing disappointment that Calendar generation is not yet a builtin function in Sense. Something like QlikView Components (QVC) does with the single script line:

CALL Qvc.CalendarFromField('OrderDate');

On to the script used in this new video. I’ll reproduce the entire script below and then comment on the techniques used and suggest some more “modern” approaches.

The video script is  similar to the script used in the current QlikView Developer course . I acknowledge that this script works as is and produces correct results. But I don’t think it should be taught to newbies as good scripting. Here’s the script from the video:

Obsolete Code

1. Why is this field created  and where is it used?

It’s not used. It’s left over from a very old version of the exercise and it doesn’t serve any purpose.

2. Why are we sorting the table? Is this statement useful?

Even if I could think of a good reason why  the Calendar table should be in order, it already is in this order because the TempCalendar was generated in a loop. Statement unnecessary.

Inefficient Code

Loading a Resident table can be very slow for a large table.

 

Experienced scripters use the FieldValues array instead.

What’s the difference? FieldValues only reads the distinct values of a field — maybe a thousand or so for several years of dates. Resident reads every row of the table. For a 10M row fact table, that’s 10M reads and the time scales up linearly. The difference can be dramatic.

Error Prone and Extra Work

Peek(), used on lines 13 & 14,  is one of those functions that fails silently. That is, if you misspell a field or table,  you won’t get a script error. Misspelling a variable will also not generate  a script error. Maybe. Or maybe not. Or maybe you will get a Calendar that starts in year 1899.  Depends on which line you make the spelling error on. If your misspelling does result in a script syntax error, it will be downstream from where you created the problem. There are multiple ways to mess this one up and some very curious potential results.

Don’t forget to DROP those temp tables on lines 15 and 35.

And those varMinDate and varMaxDate variables really should be cleared as well.

You can avoid all the syntax traps and extra cleanup by coding this a as a Preceding Load.  Here’s the same script written as a Preceding Load:

Nothing to remember (or forget) to clean up.  If you misspell a fieldname, you will get an understandable error at the right place. This is the calendar script I wish we would provide to newcomers.

Of course if you’ve attended the Masters Summit for QlikView, you’ve learned all about FieldValues, Preceding Loads and more. If not, attend a summit this Fall in NYC or Copenhagen.

QVC users don’t even get out of bed to generate Calendars. But they know that all that best practice stuff is happening under the covers. If you want to learn more about QVC, join my online class June 4 or a future class at Q-on.bi.

-Rob

Entire script suitable for copying:

MasterCalendar: 
Load 
 TempDate AS OrderDate, 
 week(TempDate) As Week, 
 Year(TempDate) As Year, 
 Month(TempDate) As Month, 
 Day(TempDate) As Day, 
 'Q' & ceil(month(TempDate) / 3) AS Quarter, 
 Week(weekstart(TempDate)) & '-' & WeekYear(TempDate) as WeekYear, 
 WeekDay(TempDate) as WeekDay 
;

//=== Generate a temp table of dates === 
LOAD 
 date(mindate + IterNo()) AS TempDate
 ,maxdate // Used in InYearToDate() above, but not kept 
WHILE mindate + IterNo() <= maxdate;

//=== Get min/max dates from Field ===/
LOAD
 min(FieldValue('OrderDate', recno()))-1 as mindate,
 max(FieldValue('OrderDate', recno())) as maxdate
AUTOGENERATE FieldValueCount('OrderDate');

Share

Qlikview Components (QVC) Training June 4

I’ll be leading a 3.5  hour training June 4 on using the free open source “QlikView Components” scripting library.   The training is held on-line and you can find out more and register at this link on the Qlik-On website.

QV Developers around the world use QVC in their scripting to:

  • Save time.
  • Improve Quality.
  • Implement Advanced Functions.
  • Get it Right the First Time.

Create a master calendar? No problem — one line of script:

CALL Qvc.CalendarFromField('OrderDate');

Fiscal calendar that begins on Month 4? Again no problem:

CALL Qvc.CalendarFromField('OrderDate','','',4);

What was that Set Analysis syntax for month-to-date in the previous year? The Calendar generates a series of variables for common period to period analysis.

=Sum($(vSetPreviousYearMTD) Sales)

That’s all there is to it. Even if you add fields to the Calendar. Even if you use Italian or Norwegian for the calendar field names.

Quick! Write the script to create an AsOf table for field Year-Month.  I’m waiting…   I can’t do it either without rooting around for an old project to copy from. But I can write:

CALL Qvc.AsOfTable('Year-Month');

Here’s are a few more things QVC can do:

  • Load variables from an external file.
  • Load custom icons from a folder.
  • Logging, including rolling external files.
  • Pause mid-script and let you inspect the contents of a table.
  • Incremental Reloads!
  • NVL!

The QVC project was founded in 2012 with the goal of simplifying common scripting tasks and implementing best practices.  The focus is on what we call the “big middle” of scripting tasks. 

The library is continually refined and enhanced with input from QV developers around the world and we expect some exciting new testing routines soon!

I hope you can join me June 4 to learn the details of installing and using QVC. You will leave the class with the confidence and skills to use all the QVC routines and begin supercharging your script!

-Rob

 

 

 

Share

Those Uncaught STORE Errors

Every software product has it’s bits of tribal lore, those unintuitive quirks that when revealed make you say “aha!” and feel empowered to mentor others.

A QlikView factoid of this variety is that: “General Script Error” from  a script STORE  statement usually means that the target directory does not exist.  For example, a “General Script Error” from the following statement  if “somedir” does not exist:

STORE mytab INTO somedir\mytab.qvd;

it could actually be any type of output file error. Could be missing directory, could be  locked file or an illegal file name, You know this. You’ve been using QlikView for some time.

I include this tidbit in my beginner classes, and the students are usually grateful to be receiving these “inside tips” .  But occasionally, I get a Programmer type in the class who slowly raises one eyebrow Spock fashion and asks “Why doesn’t it throw a ‘Directory not found’ message'”?  I’ve never been able to give a satisfactory answer to that question,  (This is usually when I announce lunch).

Someone pointed out to me recently that:

SET ErrorMode=0;

Which is supposed to allow Script to continue after errors, does not affect  STORE output file errors. That is, the script still fails with “General Script Error”. This is because the STORE file error is Uncaught/Unhandled.  You’ll recognize this construct if you’ve done any programming.

It’s not like Script can’t catch IO errors. For example, input errors are handled just as you would expect. These statements won’t cause the script to stop even though “foo.bar” does not exist:

SET ErrorMode=0;
 LOAD X FROM foo.bar;

Most of QlikView — both script & UI — fails and recovers extremely gracefully. I don’t know why STORE seems to have been left out in the cold.

-Rob

P.S. I’ll be at Qonnections next week. Let’s see how many of us can ask “Why doesn’t STORE catch output file errors?” at the R&D Q-Bar. J

Share

Using Variables as Metadata

A customer asked me today if there was a QVW property for an application description that could be pulled as metadata by his scanner app.

I’ve found that using Variables is a very simple way to define additional metadata.

[qlikview]SET vAppDescription = This is the main Sales app. blah blah;[/qlikview]

A different script can load that variable with this bit of script:

[qlikview]Meta:
LOAD [RawValue] as AppDescription
FROM [some qvw path.qvw]
(XmlSimple, Table is [DocumentSummary/VariableDescription]) WHERE Name = ‘vAppDescription’ ;[/qlikview]

-Rob

Join me at the San Francisco Masters Summit for QikView for more scripting tips.

Share

Qvc Gets Colorful

Qlikview allows customization of the basic eighteen color palette at the Document or User level through property dialogs. For color control beyond the palette,  many developers utilize variables.

The set of colors and variables I use for a document or project I refer to as a “color theme”.  I’ve built up several themes over the years as well as generously borrowed theme ideas from others.

Wait! Isn’t there an single ideal theme we should all use? No. I find a regular need to update or craft new themes in response to client requirements, the latest understanding of best practices, and frankly, fashion. 

In V9 of the Qlikview Components (Qvc) scripting library I’ve added a color theming routine.  The SUB Qvc.ColorTheme(themeFile) uses an external file to define color variables. The variables are then referenced by chart objects to implement the theme.

Implementing a color theme with Qvc requires two actions; assigning colors to the variables and assigning the variables to sheet objects.

Qvv.ColorTheme takes it’s input from an xlsx or xlsm file that contains at least these two columns:

ColorVariable — A variable name. Any variable name matching the pattern Qvc.Color.v.* will be kept by the routine. Other variable names may be used as intermediate names to compute final values.

ColorValue — a valid Qlikview script color function such as rgb(0,0,0) or white().

A sample theme is provided with the Qvc distribution in etc\Colors\ColorSample1.xlsm. This sample file also offers the option to use the Excel color picker to assign a color.

In the sample file you’ll see:

Specific colors defined early in the file that are referenced later:

“Logical” color variables that define “good” and “bad” things. 

“Object Attributes” that define items like the chart 18 color palette.

After including the qvc.qvs runtime we can add the script statement

CALL Qvc.ColorTheme('ColorSample1.xlsm')

and then reload. Great! Now our document contains a bunch of Qvc.Color.v.* variables with proper color values. How do we tell our object color properties to use those variables? Do we have to type them in each object? No.

Also included with the Qvc distribution is etc\QvcColor.qvt, a QV theme file that assigns the Qvc.Color.* variable names to object definition. You apply this qvt theme to your Document as you would any other qvt theme, using the “Layout, Apply Theme” button. Apply Theme is available at the Document level (Settings, Document Properties) or at the individual object level on the layout tab.

After pressing “Apply Theme”, select and apply the etc\QvcColors.qvt file. After Apply, you’ll see that color properties in the object (or all objects if done in Document Settings) reference the Qvc.Color.* properties.

Changes to the theme file will be reflected in the object colors after reload.

You may also assign the QvcColor.qvt theme as the default document theme for new objects in “Settings, Document Properties, Presentation”. This is particularly useful if you are starting a new document.

If you develop a color theme you find useful, please let me know on the QVC User Forum and I’ll include your theme as a sample in a future release.

-Rob

I’m now teaching a 3 hour on-line course on using the QVC library at http://q-on.bi/. The next course is scheduled for 7 Nov. See http://q-on.bi/courses/qlikview-components-scripting-library/ for details.

Share

Preceding Load is Elegant

I love the preceding load feature of Qlikview scripting. It can make complex things simple.

Consider for example, having to parse an arbitrary number of key:value pairs from input like:

Name:Shoes, Size:L, Color:Blue Suede, Stock: 200
Name:Socks, Model:Mens Casual, Stock:0, Color:Black
Name:Pants, Error: No attributes found

We want each key to become a Field populated with it’s matching value. Here’s the entire script to do just that.  A preceding load is a series of chained LOADs that execute from the bottom up — so read the script from the bottom.

And now we can build a chart like this.

Simple. Elegant.

I’ll be showing some other cool examples of Preceding Load in my “Advanced Scripting” session at the Masters Summit for Qlikview Oct 1.

-Rob

Script suitable for copy/paste

data:
// 4. Generic Load to transpose Key to Field
Generic LOAD RecId, Key, Value
;
// 3. Separate key & value
LOAD
RecId,
subfield(Pair,':',1) as Key,
subfield(Pair,':',2) as Value
;
// 2. Break out each key:value pair
LOAD
RecId,
subfield(Input,',') as Pair
;
// 1. Load the raw Input
LOAD *, RecNo() as RecId INLINE [
Input
Name:Shoes, Size:L, Color:Blue Suede, Stock: 200
Name:Socks, Model:Mens Casual, Stock:0, Color:Black
Name:Pants, Error: No attributes found
] (delimiter is '|')
;

Share

Becoming a Scripting Ninja

As a QV Consultant, I split my time between delivering training and QV Practice.  I like both sides of the business, but I particularly enjoy contributing to the excitement of using QV that results from training.

During a lunch break today, I mentioned the Masters Summit for Qlikview to one of my students. He asked “What does it take to become a Master?”. Of course I answered “Attend the summit! “.  But it reminded me of a list I present when delivering  Qlikview Developer (Scripting & Data Modeling) training.

Most of us experienced instructors augment the standard curriculum with our own real-world experiences. When teaching scripting, in addition to surveying the range of scripting statements and modeling solutions, I emphasize three specific scripting features as “Ninja skills” — script features whose mastery makes you deadly effective. My “Scripting Ninja” list is:

1. MAPPING LOAD. Understanding how to create a Mapping table. How to use that table in all it’s forms — ApplyMap(), MAP USING, RENAME USING and the other metadata USINGs — TAG and COMMENT.

2. IntervalMatch — has many uses  including Slowly Changing dimensions, Currency Rates, Dimension Bucketing to name a few.

3. Preceding Load — can be used to create powerful parsing scripts and simplify the maintenance and accuracy of script. Here’s a primer on Preceding Load.

Those are the three special QV scripting features that stand out for me as exceptionally powerful and well worth mastering. Are there others you would add to the Ninja list?

-Rob

 

Share

Speed up Script Development with BUFFER

A BUFFER prefix on a LOAD or SQL statement creates and maintains an automatic QVD for that statement. Subsequent executions of the LOAD/SELECT statement will read from the QVD, avoiding another (slower) trip to the database. A read from QVD is generally 5-10 times faster than fetching from database.

TranTab:
BUFFER LOAD 
TranID,
Amount,
CustomerID,
etc…
;
SQL SELECT * FROM TRANSACTIONS
;

On first execution, the SELECT will fetch rows from the database and the resulting TranTab will be stored in a specially named QVD on the local machine. On subsequent reloads, TranTab will automatically be loaded from the local QVD.

If you make a change to the TranTab LOAD/SQL statement, QV Reload will detect the change and fetch from the database again and update the local QVD.

During script development it’s not uncommon to perform a reload several times. You can greatly reduce the duration of a script run by adding BUFFER to your statements. Any script changes/adds you make will automatically invalidate that buffer and re-fetch from the database.

Don’t forget to remove the BUFFER keyword before moving to production!

You can read more about BUFFER and some optional parameters in the Qlikview Help.

-Rob

Share