Category Archives: Masters Summit

Roll Your Own Qlik “Connector”

Summary: I demonstrate how to use a simple Python REST server to wrap any data source for loading by the Qlik REST connector.

In my years of Qlik development I have many times needed to load data from a source not supported by available connectors. Sometimes the data resides on a factory floor controller and can only be accessed using something like telnet. One approach is to create a batch process that extracts the data to a flat file accessible to Qlik. This works, but you have to deal with issues of timing and failure monitoring.

If you want real time direct loading from Qlik script, you can write a custom connector for your data source. Writing a custom connector is a programming chore of medium complexity and there are limited languages you can work in.

As an alternative to a custom connector or batch process, I’m going to show how you can use the out-of-the-box Qlik REST connector to access your special data, wherever it lives!

REST is a popular way of exchanging data between systems. Data is requested from a REST Endpoint using a URL like:

http://localhost:3000/customers

Data is returned in JSON format like:

[{"Name": "Abc corp", "City": "New York"},
{"Name": "GeoFarm": "City": "Boston"}]

The Qlik REST connector translates JSON data into the rows and fields of a Qlik table. This URL request + JSON output makes it very easy to test REST endpoints using a browser.

Due to REST’s popularity, the tools we need to create REST endpoints are available in just about all programming languages, platforms and lowcode services. Use whatever language you are familiar with or that runs in your environment. I’m going to show a quick example using Python. If you want to create this example yourself, make sure you have Python 3+ installed along with the packages Flask and Pandas.

For my “special data source”, I’m going to use output from the Windows “Tasklist” command. The output looks like below.

Our REST Endpoint must provide three components:

  1. A web server that will handle the incoming URL.
  2. Data from our special/custom data source.
  3. A conversion of data from our native format to JSON.

Here is a complete Python program that satisfies all three requirements.

from flask import Flask, json
from io import StringIO
import pandas as pd
api = Flask(__name__)

@api.route('/tasklist', methods=['GET'])
def get_tasklist():
    import subprocess
    subprocess = subprocess.Popen("tasklist /FO CSV", shell=True, stdout=subprocess.PIPE, text=True)
    csvStringIO = StringIO(subprocess.stdout.read())
    df = pd.read_csv(csvStringIO);
    return Response(df.to_json(orient='records'), content_type='application/json')
if __name__ == '__main__':
    api.run(port=3000, host='0.0.0.0') 

Save this code in file “SimpleRest.py” and start the python task with the command:

python SimpleRest.py

In a Qlik App, create a new REST connection. The URL will be our new REST endpoint. If your Python task is running on other than “localhost”, change the host in the URL.

Give it a Name. Test and Save.

Use the “Select data” button on the connection and select the “root” table. The Qlik wizard will build the required script.

Reload and you should have data!

Let’s break down the pieces of the python program to understand what you would change and what you would re-use in a new scenario.

The first four lines import required libraries and initialize Flask, the REST library we will use. Here is the remaining code.

@api.route('/tasklist', methods=['GET'])
def get_tasklist():
    import subprocess
    subprocess = subprocess.Popen("tasklist /FO CSV", shell=True, stdout=subprocess.PIPE, text=True)
    csvStringIO = StringIO(subprocess.stdout.read())
    df = pd.read_csv(csvStringIO);
    return Response(df.to_json(orient='records'), content_type='application/json')
if __name__ == '__main__':
    api.run(port=3000, host='0.0.0.0') 

The bolded lines above start a webserver listening on port 3000 and declare what code will run when the URL path is “/tasklist”.

@api.route('/tasklist', methods=['GET'])
def get_tasklist():
    import subprocess
    subprocess = subprocess.Popen("tasklist /FO CSV", shell=True, stdout=subprocess.PIPE, text=True)
    csvStringIO = StringIO(subprocess.stdout.read())
    df = pd.read_csv(csvStringIO);
    return Response(df.to_json(orient='records'), content_type='application/json')
if __name__ == '__main__':
    api.run(port=3000, host='0.0.0.0') 

These two lines above run the tasklist /FO CSV command and capture the output. To make wrangling easier I added tasklist options “/FO CSV” which generates output in CSV format.

@api.route('/tasklist', methods=['GET'])
def get_tasklist():
    import subprocess
    subprocess = subprocess.Popen("tasklist /FO CSV", shell=True, stdout=subprocess.PIPE, text=True)
    csvStringIO = StringIO(subprocess.stdout.read())
    df = pd.read_csv(csvStringIO);
    return Response(df.to_json(orient='records'), content_type='application/json')
if __name__ == '__main__':
    api.run(port=3000, host='0.0.0.0') 

Lastly we wrangle the output from CSV to JSON format and send the data to the requestor.

I hope you can see from this example how easy it can be to expose your custom data to Qlik and have real time loading. I used Python for this example, but you could implement this same example in many languages with little effort.

if you’ve created my example in your own environment, you should now be able to create a treemap chart of Memory Usage. Tip: parse the [Mem Usage] field in the script with

Num#(SubField([Mem Usage], ' ', 1), '#,##0') * 1024 as [Mem Usage]

Want to talk more data loading techniques? Join me at the Masters Summit for Qlik in Fall 2022 where in addition to heads down learning, we’ll be kicking the tires on things like Server Side Extensions and Qlik Application Automation.

-Rob

Share

Masters Summits 2022

We’re baaaack!  After a two year break from live events, I’m pleased to announce that the Masters Summit for Qlik starts up again in Fall 2022. 

Masters Summit 2022 Madrid and New Orleans

Madrid – September 19-21

New Orleans – November 14-16

Our goal in this three day hands on education event is to “Take your Qlik skills to the next level”  to make you more productive and increase the business value of your Qlik Sense or QlikView applications.

Through lecture,  hands on activities and takeaway code samples,  the Summit will expand your knowledge with advanced techniques and a deeper understanding of the core skills required in all Qlik application development:

  • Data Modeling
  • Advanced Scripting
  • Advanced Aggregation & Set Analysis
  • Visualization.

In addition to core topics, we’ll have 1/2 day workshops on performance tuning and an introduction to creating Qlik Sense mashups using APIs.  See the complete agenda here.

Our evening guest speakers, networking events and optional lunchtime lectures fill out the program with additional content and lively discussion.

Our panel of five presenters are well known as authors, educators,  Qlik experts and members of the Qlik Luminary and MVP programs.

Have you taken basic Qlik training and/or worked with the product for a while?  Do you find yourself struggling with data modeling questions such as slowly changing dimensions and rolling time analysis?  Syntax for aggregation questions like “what products do top salesreps in each region sell?”  When do I need “$()” and when do I not?  

I hope you can join us in Madrid or New Orleans to take your Qlik skills to the next level.  The early bird registration discount is available until August 19.  Event details and online registration.

See you there!

-Rob

Share

Load Master Items in Script?

At the Masters Summit for Qlik, I always try to show something new and cool.  At past events I’ve shown things like Alexa selections in Qlik Sense and the new Document Analyzer for Qlik Sense.

So what do I have lined up for the upcoming Amsterdam and Washington DC events?

In QlikView we use variables in a similar way to Master Measures in Qlik Sense. That is, as a single version of a chart expression.  In QlikView, we typically use script to load variables from an external source to facilitate reuse and portability.

Qlik Sense does not have the ability to create Master Measures using script. But wouldn’t that be useful if it could…

I’ve been working on “loading Master Measures in script”  and believe I have a working solution.  I’ll be demonstrating at the Summit!

I hope you can join us at the upcoming summit. Our goal at this three day  hands on education event is to “Take your Qlik skills to the next level”  to make you more productive and increase the business value of your QlikView or Qlik Sense applications.

More information and registration here.

-Rob

 

Share

A Common SSE Plugin Project

Summary: I introduce qcb-qlik-sse, a community Server Side Extension to share custom Qlik functions. 

At the Masters Summit for Qlik, I dive into several different methods of creating reusable script and custom functions.  In QlikView we have the ability to write custom functions using VbScript/Jscript in the qvw Module.

Custom functions have been useful for things like regular expressions, geo calculations, url encoding, encryption and others.  I’ll call them “edge functions” — some of us need them, some of us don’t.

Qlik Sense does not have the module facility. How can we satisfy the requirement for custom function in Qlik Sense?  The Server Side Extension (SSE) facility can fill the need and is available to both Qlik Sense and QlikView.

An SSE Plugin runs as a separate task and provides communication with a Qlik Script or chart Expression via a TCP port. The same SSE Plugin can serve both QS and QV.

Anyone can write an SSE. The Qlik team provides the SSE base and you write a plugin that wires your new functions to Qlik. The new functions can be used in both Script and Charts.  A number of plugins have already been produced.

SSE seems to be the ideal place to provide a collection of edge functions.  Rather than a bunch of one-offs, I’m thinking a good idea would be to pool resources into a single effort that could be shared, much like QlikView Components did for Script.

I’ve implemented this idea as qcb-qlik-sse.  This server uses as it’s base the qlik-sse package created by Miralem Drek.

qcb-qlik-sse is written in javascript and runs in node.js.  At startup, the server scans it’s “/functions” directory and discovers what functions are available.  The general idea is that you can add new function by creating a new js file.  You can remove function you don’t want available by deleting the corresponding js file.

See what functions I’ve already implemented in the doc here. I’ve also provided an example qvf and qvw that exercise the functions.

If you want to try it out,  download the project and define the plugin to QS or QV as documented here.  You will also need node.js 8 or later installed.

Defined functions will show up in the suggestion list in both the script and expression editors.

 

If you want to add functions, some javascript skills are required.  Follow the directions in the readme and submit a PR.

I’ve labeled the project as “experimental” at this stage because I anticipate there could be some significant restructuring as I get feedback.

Let me know your ideas and if you find this useful!

-Rob

Want to kick the tires on reusable code and make your Qlik team more efficient? Come to the Masters Summit for Qlik, a three day advanced training event for Qlik Developers. 

 

 

Share

Masters Summit Fall 2019

Fall 2019 dates and location for the Masters Summit for Qlik have been announced .  Our goal at this  hands on education event is to “Take your Qlik skills to the next level”  to make you more productive and increase the business value of your QlikView or Qlik Sense applications.

When registering for either location, choose one of two tracks:

Our traditional “Qlik Sense and QlikView” track is designed for Qlik Developers who create applications using the out-of-the box clients: Qlik Sense Hub or the QlikView Desktop. Through lecture,  hands on activities and takeaway code samples,  the Summit will expand your knowledge with advanced techniques and a deeper understanding of the core skills required in all Qlik application development:

  • Data Modeling
  • Advanced Scripting
  • Advanced Aggregation & Set Analysis
  • Visualization.
  • Performance

See the complete agenda here.

Our Qlik API track is designed for Web Developers who create Qlik applications and mashups using the Qlik Engine API, Qlik Core, QAP and custom visualizations.

  • Qlik Core
  • Generic Objects
  • Loading & Visualizing Data
  • Architecture and Performance

The API track is led by Nick Webster and Speros Kokenes. You can see the agenda for the API track here.

Our evening guest speakers, networking events and optional lunchtime lectures fill out the program with additional content and lively discussion.

Our panel of six presenters are well known as authors, educators,  Qlik experts and members of the Qlik Luminary and MVP programs.

Learn more about determining if the Summit is right for you and choosing a track in our Frequently Asked Questions.  If you have any unanswered questions or want to learn more, reach out to registration@masterssummit.com .

I hope you can join us in Amsterdam or Washington D.C. to take your Qlik skills to the next level.  Event details and online registration.

See you there!

-Rob

Share

Masters Summit Stockholm

The 16th edition of the Masters Summit for Qlik will take place 1-5 April 2019  at the Stockholm City Conference Centre .  Our goal at this  hands on education event is to “Take your Qlik skills to the next level”  to make you more productive and increase the business value of your QlikView or Qlik Sense applications.

 

New for this summit, you can choose from two tracks:

Our traditional track, 1-3 April, is designed for Qlik Developers who create applications using the out-of-the box clients: Qlik Sense Hub or the QlikView Desktop. Through lecture,  hands on activities and takeaway code samples,  the Summit will expand your knowledge with advanced techniques and a deeper understanding of the core skills required in all Qlik application development:

  • Data Modeling
  • Advanced Scripting
  • Advanced Aggregation & Set Analysis
  • Visualization.
  • Performance

See the complete agenda here.

Our new API track, 3-5 April, is designed for Developers who will create Qlik applications and mashups using Qlik APIs, Qlik Core, QAP and custom visualizations.

  • Qlik Core
  • Generic Objects
  • Loading & Visualizing Data
  • Architecture and Performance

The API track is led by Nick Webster and Speros Kokenes. You can see the agenda for the API track here.

Our evening guest speakers, networking events and optional lunchtime lectures fill out the program with additional content and lively discussion.

Our panel of six presenters are well known as authors, educators,  Qlik experts and members of the Qlik Luminary and MVP programs.

Learn more about determining if the Summit is right for you and choosing a track in our Frequently Asked Questions.  If you have any unanswered questions or want to learn more, reach out to registration@masterssummit.com .

I hope you can join us in Stockholm to take your Qlik skills to the next level.  The early bird registration discount is available until 28 February.  Event details and online registration.

See you there!

-Rob

Share

Masters Summit Philadelphia

The 15th edition of the Masters Summit for Qlik will take place October 3-5 2018 at the Bellevue Hotel in Philadelphia PA .  Our goal in this three day hands on education event is to “Take your Qlik skills to the next level”  to make you more productive and increase the business value of your QlikView or Qlik Sense applications.

Through lecture,  hands on activities and takeaway code samples,  the Summit will expand your knowledge with advanced techniques and a deeper understanding of the core skills required in all Qlik application development:

  • Data Modeling
  • Advanced Scripting
  • Advanced Aggregation & Set Analysis
  • Visualization.

In addition to core topics, we’ll have 1/2 day workshops on performance tuning and an introduction to creating Qlik Sense mashups using APIs.  See the complete agenda here.

Our evening guest speakers, networking events and optional lunchtime lectures fill out the program with additional content and lively discussion.

Our panel of five presenters are well known as authors, educators,  Qlik experts and members of the Qlik Luminary and MVP programs.

In addition to our regular presenters, we’ve got some exciting evening guest speakers in Speros Kokenes and Paul Van Siclen.

Have you taken basic Qlik training and/or worked with the product for a while?  Do you find yourself struggling with data modeling questions such as slowly changing dimensions and rolling time analysis?  Syntax for aggregation questions like “what products do top salesreps in each region sell?”  When do I need “$()” and when do I not?  More self assessment to determine if the summit is right for you can be found here.

I hope you can join us in Philadelphia to take your Qlik skills to the next level.  The early bird registration discount is available until August 17.  Event details and online registration.

See you there!

-Rob

Share

Masters Summit Prague

The 14th edition of the Masters Summit for Qlik will take place in Prague on 3-5 April 2018.  In this three day hands on education event our goal is to “Take your Qlik skills to the next level”  making you more productive and increasing the business value of your QlikView or Qlik Sense applications.

Through lecture,  hands on activities and takeaway code samples,  the Summit will expand your knowledge with advanced techniques and understanding of the core skills required in all Qlik application creation:

  • Data Modeling
  • Scripting
  • Advanced Aggregation & Set Analysis
  • Visualization.

In addition to core topics, we’ll have 1/2 day workshops on performance tuning and an introduction to creating Qlik Sense mashups using APIs.  See the complete agenda here.

Our evening guest speakers, networking events and optional lunchtime lectures fill out the program with additional content and lively discussion.

Our panel of five presenters are well known as authors, educators,  Qlik experts and members of the Qlik Luminary and MVP programs.

Have you taken basic Qlik training and/or worked with the product for a while?  Do you find yourself struggling with data modeling questions such as slowly changing dimensions and rolling time analysis?  Syntax for aggregation questions like “what products do top salesreps in each region sell?”  When do I need “$()” and when do I not?  More self assessment to determine if the summit is right for you can be found here.

I hope you can join us in Prague to take your Qlik skills to the next level.  The early bird registration discount is available until 2 March.  Event details and online registration.

See you there!

-Rob

Share

Get the Numbers Right

I travel a lot for business and personal reasons. I’ve used TripIt (owned by Concur) for several years to plan and manage my trips.

It’s a  great tool, can’t image living without it.

Today TriptIt sent me a lovely graphic email summarizing my 2017 travel.  The design was  a very fashionable  card layout with share-to-social links.

Pretty cool, but for one problem.  It’s not my data.

Yes, those are definitely not my travel stats.

When I teach Qlik application development, a common  question raised by students or other newcomers is: “What’s the most important skill to master?  Scripting?  Expression writing?  SQL? Pre-attentive attributes? Color rules?”

My answer is always the same: “First, get the numbers right.  Second, keep the numbers right”.

It doesn’t matter how it looks or how fast it calculates if the results are wrong.

Join me at the Masters Summit for Qlik in Prague 3-5 April where in addition to a full range of Qlik Dev topics, we’ll be discussing automating quality.  I’ll be demonstrating tools to automate dashboard validation and continuous quality monitoring.

-Rob

BTW I emailed the TripIt folks about the error, they’ve promised to look into it.  

Share

Guest Speakers for Boston Masters Summit

 

I’m pleased to announce our guest speakers for the upcoming  Boston Masters Summit for Qlik , October 23-25

Anthony Deighton, Qlik CTO, will be our evening speaker.  Since 2005 Anthony has been responsible for guiding product strategy and leading all aspects of the company’s R&D efforts.  Anthony’s talk is sure to be interesting, stimulating and informative.

Nick Webster’s evening talk at the Munich summit was such a hit that we’re bringing him back in an expanded role.  Nick will present a half day, hands-on session titled “Qlik Sense Integration”.

Want to understand the possibilities of the web and the Qlik platform?  Nick’s  session will help you understand the basics of HTML and CSS and show you how to construct a basic web page that displays data from Qlik Sense.  After completing the session you’ll have a fully functional, interactive web application and the inspiration to continue your journey into the world of web development with Sense.  No prior experience with Web Development or Qlik Sense is required.

Those special speakers are in addition to our already packed three day agenda.

I hope to see you in Boston!

Learn more about the Summit or register for Boston.

 

-Rob

Share