Welcome Guest, you are in: Login

QVSource: The QlikView & Qlik Sense API Connector (Docs)


QVSource Twitter Demo - A New Approach

Since we released the first beta of QVSource we have been listening to your comments and have been developing a new, improved range of Twitter Applications that will make it even easier for you to get started with and help you get the most value out of your Twitter data.

In doing this we have taken a new approach to the applications and their set up.

Overview

There is now a collection of Twitter QlikView applications, with a 'Data' QlikView application that has optimised load script and is dedicated to getting the data via QVSource (for example Twitter and Klout) with some simple dashboards.

Then there are the 'UI' QlikView applications which build on the data application's output and have a much richer set of dashboards, each targeting a different kind of usage.

We currently have 3 UI QlikView applications (although they may not all currently be in the download)
  • Conference Tracker (this is actually currently also the Load Script and Data Model app)
  • Sample Charts & KPIs (see below)
  • Single Column Dashboard - iPad friendly, (coming soon)

Due to this new approach more UI apps can be added in the future.

Image

Download

IMPORTANT: You will need QVSource version 1.4.2.8 or later.

We are starting to move our QlikView applications over to GitHub.

To understand why we are doing this and what it means to you, we have written this page.

You can download the latest zipped package from here or you can visit the QVSource repository on GitHub here where you can track changes and download it.

When you unzip the download you will see the following files.

Please also note that some of the screen shots of file names and directories may not currently match the download.

Image

On this page we will now look how to use each application in turn and the relationship between the Data and UI applications. There are also links, for detailed explanations of the data models and load scripts for each application.

Load Script and Data Model

The load script and data model QlikView application allows you to create the following schema to base your applications on:

Image

Overview

This section refers to the following QlikView application and text file in the download pack.

Image

The purpose of this application is to take the Twitter search terms and accounts defined in the config file, then on reloading gets the data via QVSource and saves them to a series of QVDs.

The Twitter data is also enriched by using the following connectors
  • Klout
  • Sentiment and Text Analysis

The load script has been optimised to make as few calls as possible to Twitter, Klout and the Sentiment APIs.

!IMPORTANT!

This application sets up the data model as well as creating and managing the QVDs, which the other UI applications reply on.

As such this application must be reloaded to get the latest data and only then can the subsequent UI applications be reloaded - they're explained in more detail below.

Setting it up for Your Needs

All configuration for the application is done via the config text file. The most important sections of this are exaplained here:

Variables

The following variables are set in the config file.

IMPORTANT: Note that by default vUseRandomSentimentForAllEngines and vUseRandomKloutScore are set to 1 meaning random values will be returned - you should set these to 0 when you are ready to start loading your app with real data.


// --------------------------------------------------------------------------------------------------------
// THE FOLLOWING SHOULD BE REVIEWED BEFORE RELOADING THIS APPLICATION
// --------------------------------------------------------------------------------------------------------

//
// See end of this config file for valid values.
//
let vLocalTimeZone = 'London';
let vLocalReloadTime = date(ConvertToLocalTime(gmt(), '$(vLocalTimeZone)'), 'DD/MM/YYYY hh:mm');

let vQVSourceAppId = 'Twitter_Demo'; 	// Used for example in error emails and logging.
let vWorkingFolder = 'c:\QVSource\Temp\';

let vTitle = 'Twitter Tracker Demo';

//
// If you simply want to collect Tweets you can turn off sentiment analysis 
// and Klout score lookups.
//
let vGetSentiment = 1;
let vGetKloutScores = 1;

//
// Useful during development work to speed up load times. If you want to change this to 1 you should 
// also delete any sentiment QVD files so that they will be re-generated with real data.
//
let vUseRandomSentimentForAllEngines = 0;

//
// Useful during development work to speed up load times. If you want to change this to 1 you should 
// delete the Klout.qvd file so that it will be re-generated with real data.
//

// NOTE: This applies only to the large number of users associated with the searched for Tweets.
// The Klout score for the users in the TwitterAccounts table is currently 'hard coded' to be looked up.
//
let vUseRandomKloutScore = 1;		

//
// IMPORTANT: This uses AlchemyAPI so only turn this on if you have entered an API key into QVSource 
// with sufficient quota to score all the data you are likely to load.
//
let vIncludeKeywordLookup = 0;

You should find the notes in the config file explain the variables - however a couple of additional points:

vWorkingFolder
The QlikView application saves some intermediate CSV files here which are read and used by QVSource. See this page for more information.

vLocalTimeZone
The time zone where the Tweets you are tracking are based. This is particularly useful for example if you are tracking a conference as all the users are located in the same place and you would like the time based charts to reflect their local time.

vQVSourceAppId
This will be passed to connectors and used for error logging.

vIncludeKeywordLookup
You should only use this if you have an account with AlchemyAPI which has enough calls to analyse your anticipated Tweet volume. AlchemyAPI do offer a free package which allows 1000 calls per day. The part of the script which uses this to extract keywords from the Tweets is also not yet optimised to minimise the number of calls so this will likely be the slowest part of the load script. In the first instance we would recommend you leave this turned off (set to 0).

vUseRandomSentimentForAllEngines and vUseRandomKloutScore
We would recommend in the first instance you leave these turned on until you are ready to start testing for production. Leaving these on means QVSource will return random values rather than accessing the APIs and will make the scripts load much more quickly.

Search Terms

The following table allows you to specify the search terms you would like to find matching Tweets for:


TwitterSearchTerms:
LOAD * INLINE [
Grouping, SearchTerm
    SAP, #SAP,
    Business Objects, #businessobjects, 
    Spotfire, #TibcoSpotfire,
    Spotfire, #SpotfireSupport,
    Cognos, #IBMCognos,
    LogiXML, #LogiXML,
    PowerPivot, #powerpivot,
    Tableau, #tableau,
    MicroStrategy, #microstrategy,
];

Note that the app is configured to run each search term with both " RT" (retweets only) and " -RT" (tweets only) on the end. This means if you have 5 search terms you will actually be running 10 different searches. You can of course adjust the load script easily to change this.

You can leave this table empty if you only want to track the details of certain Twitter accounts (see below).

Twitter Accounts to Track

The following table can be used to specify a list of Twitter accounts to track. This means that each time the load script runs, various information about these account (number of followers, klout score etc.) will the stored into a table against a time stamp allowing you to chart the change in these variables against time.


TwitterAccountsToTrack:
LOAD * INLINE [
    Grouping, TwitterAccount
    QlikView, QlikView,
    SAP, SAP,
    Business Objects, businessobjects,
    Spotfire, TibcoSpotfire,
    Cognos, IBMCognosCTO,
    LogiXML, LogiXML,
    PowerPivot, powerpivot,
    Tableau, tableau,
    MicroStrategy, microstrategy
];

You can leave this table empty if you only want to track Tweets matching certain search criteria (see above).

Sentiment Engines

This table allows you to configure the sentiment engines you wish to analyse all the Tweets with.


SentimentEngines:
LOAD * INLINE [
Engine
Random
Sentiment140
];

The values here must be selected from the following list:

  • Repustate
  • Sentiment140 (if you are using the TextAnalytics Connector V2)
  • AlchemyAPI (if you are using the TextAnalytics Connector V2)
  • Random

Note that the Sentiment Analysis & Text Analytics Connector does support other APIs (additional to those in the list above) but the default load script is only designed to work with the above without further modification. It would be a relatively simple task to amend the load script to work with other sentiment APIs.

Note also that if the vUseRandomSentimentForAllEngines variable above is set to 1 (recommended during initial testing) then random sentiment will always be returned.

With the exception of the TwitterSentiment and Random options you should also be aware that you will need to have entered a valid API key in to QVSource for the other options and this API key must also have a sufficient quota to be able to score all the Tweets which your load script retrieves.

Change Log

As we make changes to this application they will be recorded in the first tab as shown here:

If changes are made which actually change the data model/schema it is usually recommended that you delete your QVDs and start reloading fresh. The alternative is to write script to upgrade your previous QVDs (although it is unlikely more than one table would change at a time) into the new schema.

Image

UI App - Sample Charts & KPIs

This application uses the QVDs that were generated by the 'Load Script & Data Model' application above to create a new data model that has an advanced set of metrics and KPIs - indeed there is no QVSource-based script in this application.

You can see a screen shot of one of the tabs below.

Image

You can view this application online here

Reloading Data

You must reload the 'Load Script & Data Model' application before this one.

You can set up your QlikView Server so that it reloads when the first application successfully loads.

To do this, go to the Documents tab of your QlikView Management Console and set up a scheduled reload for the 'Load Script & Data Model' application.

Next, select the 'Sample Charts & KPIs' application and enable scheduled reloads, however, this time select "On Event from Another Task" and from the corresponding dropdown pick the 'Load Script & Data Model' application.

Image

Setting up the application this way means that if the initial 'Load Script & Data Model' application takes longer than expected or it fails for some reason, then this application will not automatically reload.


(QVSource works with Qlik Sense as well as QlikView - See this page for notes.)
QVSource - The QlikView & Qlik Sense API Connector | © Copyright 2011 - 2016 Industrial CodeBox Ltd