Summary: Preceding load used to slow down your script. but no more. Beginning with QV Nov 2017, preceding load has no performance penalty.
I’ve posted several times about the elegance of preceding load. I’ve also written about how preceding load can make your script run significantly slower. Good news! Beginning with QV release Nov 2017 (12.20) the performance penalty has been eliminated.
Test 0, the first bar, indicates the time in seconds to perform an optimized load of a 20 million row QVD. Test 1, which follows, is loading the same QVD but with the addition of two new calculated fields in the same LOAD statement. The calculations are trivial, so the increase in elapsed time is mostly due to the loss of the optimized load.
Test 2 creates the same calculated fields using preceding load and you can see the dramatic increase in elapsed time. Test 5 adds a “LOAD *” to the preceding load stack and again shows a large increase in duration.
Tests 3, 4 & 6 repeat the same tests using Resident as the source instead of QVD. Once again, a significant increase in duration when preceding is used.
I’ve been running this same test suite for several years across multiple QV releases, different machines and varying datasets. The results are generally the same.
The problem, as explained to me by Henric Cronström and confirmed by my own observations, is that the preceding load code uses only a single processing thread. So while tests 1 & 3 above will use multiple threads, tests 2,4,5,6 will use only a single thread. One way to think of this is not that preceding load runs slower, but that non-preceding load runs faster.
I never did understand why Preceding-Resident ran slower than Preceding-QVD, but I no longer care!
You can see optimized QVD (test 0) is about the same. Adding calculated fields (test 1) is slightly better between releases.
What is really significant is there is no longer any increase when using preceding load. Further, Resident performs faster than QVD as I would expect. (Note both tests used an SSD drive).
This is all great news as there are many cases where preceding load can help make your code more maintainable and understandable. I hated to choose between clarity and performance.
What about Qlik Sense? I’ve confirmed that Feb 2018 Desktop exhibits the new “no-penalty” performance. I don’t know about previous releases.
No reason to fear preceding load!