Welcome Guest, you are in: Login

QVSource: The QlikView & Qlik Sense API Connector (Docs)


Overview

The Twitter Connector loads data from the Twitter API into your QlikView application allowing you to extract data from Twitter such as Tweets based on a search term or hashtag and information about users. If you have never used Twitter before you might find this page useful.

You can also use this data in conjunction with our other connectors such as the Klout Connector or Sentiment and Text Analysis Connector to further enrich it.

To fully use the connector and all it's tables you need to have a Twitter account.

The Connector

You can find the Twitter Connector V2 in QVSource's connectors tab:

Image

On opening the connector you should see the following:

Image

This shows the new, more intuitive UI for QVSource.

Authentication

To Authenticate, pressing the "Authenticate" button begins authentication process.

IMPORTANT: You must ensure that your system's clock is accurately set. Deviations can cause an error when authenticating with the Twitter API.

Image

Enter Twitter account user/email and password and click on "Authorize app" button. 

Image

If authorization is successful Twitter will return a PIN number. When the PIN appears press type the PIN in the textbox at the bottom left and press "Close button.

Image

With this the authentication process is complete and the generated token and tokensecret keys will be filled in "Connector Settings" tab.

Image

''These keys will be filled for the last authenticated account only! Previous one will be overwritten.''

Available Tables

By selecting 'All Tables' from the drop down list, you'll see that the following tables are available

Image

Performing a Twitter Search

The most simple request to make is a Twitter Search, this is where you search for a term on Twitter.

Some important notes on the Twitter Search API
It is important to understand the following points about the Twitter Search REST API (which QVSource uses in the background).

  • The search API is limited to making a maximum of 180 API calls per 15 minutes per authentication token.
  • Each of these API calls will return a maximum of 100 results.
  • As noted here, the Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets.

In the background, the QVSource Search table will make the number of requests specified in the 'Max Number Of Search Pages' parameter (15 by default) until it either: - Reaches the limit of the search index (6 - 9 days as noted above). - Runs out of search API calls (currently 180 per 15 minute window). - Runs out of search results (i.e. it has retrieved all the Tweets matching the search criteria).

As an example, if you had have 12 search terms that are each generating 1500 Tweets per 15 minutes QVSource should be able to gather these Tweets. The theoretical maximum would be to retrieve 180 x 100 = 18,000 Tweets per 15 minutes or 72,000 Tweets per hour.

In most of the QlikView/Qlik Sense + QVSource use cases we have come across users are working with search terms generating far fewer Tweets meaning QVSource can handle even more queries.

You should always make use of the 'Since ID' parameter to tell QVSource the id of the most recent Tweet you have already retrieved (and preferably saved into a QVD file or other data store) for a given search query to ensure that it makes the minimum number of API calls necessary.

Typically want to run regular reloads and accumulate new data into, for example, a QVD file. Our demo starter application is designed to run in this way.

So, in this example we'll search for qlikview.

From the list of tables on the left, select 'Search (Simple)'. On doing so the input boxes to the right will change to reflect what parameters are available.

Enter 'qlikview'.

Image

Then click the 'Run Table/Generate Script' button.

Image

QVSource will now request a search on Twitter for that search term. And the results will be shown in the main Data section

Image

Going to the Script tab will then give you the script that you can copy into your QlikView applications.

Image

Twitter Search Notes

If you are getting back results you which do not appear correct it might be because:
  • The result Tweet contains a url which contains your search query but as it is currently in it's shortened state (e.g. t.co/...) this phrase is not visible.
  • The result is from a user who has the search query mentioned somewhere in the profile information.

We have also had reports from a user experiencing apparently incorrect behaviour of the since_id parameter whereby if it is for a Tweet which is more than a few hours old it is ignored. We have currently not been able to reproduce this but are investigating.

Search API Query Operators

Below are official query operators from the Twitter API documentation:

Note that it also appears that you can add ' RT' (e.g. 'qlikview RT') to search for only retweets or ' -RT' (e.g. 'qlikview -RT') to exclude retweets although this is not currently listed in the docs.

OperatorFinds tweets…
watching nowcontaining both “watching” and “now”. This is the default operator.
“happy hour”containing the exact phrase “happy hour”.
love OR hatecontaining either “love” or “hate” (or both).
beer -rootcontaining “beer” but not “root”.
#haikucontaining the hashtag “haiku”.
from:alexiskoldsent from person “alexiskold”.
to:techcrunchsent to person “techcrunch”.
@mashablereferencing person “mashable”.
superhero since:2010-12-27containing “superhero” and sent since date “2010-12-27” (year-month-day).
ftw until:2010-12-27containing “ftw” and sent before the date “2010-12-27”.
movie -scary :)containing “movie”, but not “scary”, and with a positive attitude.
flight :(containing “flight” and with a negative attitude.
traffic ?containing “traffic” and asking a question.
hilarious filter:linkscontaining “hilarious” and linking to URL.
news source:twitterfeedcontaining “news” and entered via TwitterFeed

There is also a useful article entitled How to Become a Twitter Search Ninja.

Working with multiple accounts

As mentioned before the authentication process stores token/tokensecret keys only for the last authenticated user. If there is need more than user to be used to call the API (for example to have two QlikView apps that analyze the data for two accounts) token/tokensecret keys can be provided as a QlikView script parameter described in "Using parameters in QlikView script" above.

Advanced Features

Handling quota limits and paging through follower ids

We are working on a new feature for working more cleanly and effectively with quota limits. This page has a detailed explanation of this with regards to the FollowerIds table.

Demo App

Twitter Tracker Demo Application

A new Twitter demo application which has a more highly optimised load script and a richer data model on top of which two example front ends are shown.

You can view one of the front ends here

API Status

You can check the Twitter API Status here.

This page might also be useful as it gives a calendar of recent and upcoming changes to the API.

Please note that we are working on a new version of this Connector for the new Twitter API version 1.1.

Known Issues

Currently the favorite_count and retweet_count fields always seem to come back as 0 from the API (we have posted about this here).

This has now been resolved by Twitter.

Twitter Action URLs (Web Intents)

It's possible to use Twitter URLs to respond to Tweets or follow users. These can be constructed in your QlikView application, which when clicking launch Twitter on the appropriate page. The following table shows the URL, with token to replace, for the popular actions

ActionURL
Reply to a Tweethttps://twitter.com/intent/tweet?in_reply_to=[TWEET_ID]
Retweet a Tweethttps://twitter.com/intent/retweet?tweet_id=[TWEET_ID]
Favourite a Tweethttps://twitter.com/intent/favorite?tweet_id=[TWEET_ID]
Short User Profile, with option to Followhttps://twitter.com/intent/user?screen_name=[SCREEN_NAME]

You can read more about Twitter Web Intents here.

Change Log

1.1.2 - 15/02/2016
  • Fixed bug in Followers table where it made unnecessary API requests and used up all of current quota unnecessarily.

1.1.1 - 08/02/2016
  • Fixed issue in RateLimit table where _x002F_ was shown in data instead of / character.

1.1.1 - 08/02/2016
  • Fixed issue in RateLimit table where _x002F_ was shown in data instead of / character.

1.1.0 - 01/12/2015
  • Minor internal refactoring.
  • Fixed issue where Post_XXXX tables would error in web edition.

1.0.9 - 11/11/2015
  • Just a note that Twitter are now referring to favourites as likes (https://blog.twitter.com/2015/hearts-for-developers), however they are not changing the name in the API and so for now QVSource will continue to use the API naming.

1.0.8 - 13/10/2015
  • DirectMessages, Favorites, HomeTimeline, UserTimeline and Mentions tables now 'stream' data in tables to client.
  • Fixed bug where Favorites table ignored username input.

1.0.7 - 22/09/2015
  • Updated table descriptions on Search tables.

1.0.6 - 26/08/2015
  • Minor tweak to CustomRequest table internals.

1.0.5 - 10/08/2015
  • For UserTimeline table:
    • Added a max number of pages input parameter to control how many API calls will be made and allow finer control over the number of results.
    • Added a second retry in the case of 404 error from API (which seems to be a known issue at present for several API end points - https://twittercommunity.com/t/intermittent-404-responses-from-rest-api/46712/23).

1.0.4.0 - 10/06/2015
  • Fixed mistake in TweetLookup table description which previously stated 60 calls can be made per rate limit window whereas the actual value is 180.
  • Added links to corresponding Twitter API docs for all tables where this is appropriate.

1.0.3.9 - 01/06/2015
  • Fixed bug where search query for Twitter search table(s) was not being URL encoded.

1.0.3.8 - 20/04/2015
  • profile_image_url_https now used instead of profile_image_url for profile_image_url column.
  • Added first_hash_tag, first_expanded_url, first_user_mention, first_symbol, first_media_expanded_url and first_media_photo_url columns.
  • https version of media photos now extracted in preference to http version.
  • Renderers added for links and images (for QVSource Web Edition).

1.0.3.7 - 09/03/2015
  • Fixed bug causing authentication issue when certain languages were used as input parameters to connectors (e.g. search in Arabic).

1.0.3.6 - 24/10/2014
  • Fixed bug where Post_UserLookupById_Info was shown as a child/'follow on' table for UserLookup rather than Post_UserLookup_Info.

1.0.3.5 - 22/10/2014
  • Improved descriptions of timeline and mentions tables to more clearly illustrate the maximum number of results they can return and number of API calls used.
  • Fixed bug in the cursoring of the FollowerIds and FriendIds tables.

1.0.3.4 - 15/10/2014
  • Fix now Twitter have dropped support for SSL3.

1.0.3.3 - 06/10/2014
  • Minor edits to some parameter labels.
  • Updated description for UserSearch table.
  • Followers/Following tables can now return up to 15 x 200 = 3000 results.
  • Added descriptions to Followers and Following tables.
  • since_id and max_id parameters added to Favorites table.
  • max_id parameter added to Mentions, UserTimeline HomeTimeline, DirectMessagesReceived and DirectMessagesSent tables.

1.0.3.2 - 05/05/2014
  • The Tweet Lookup table will now accept 100 comma separated IDs at a time and 60 of these requests can be made per 15 minutes.
  • Added the following trend related tables: TrendsNearLocation, TrendsForPlace, LocationsWithTrends.
  • Fixed bug in UserSearch table with duplicate results coming back.

1.0.3.1 - 20/03/2014
  • Added API logging for UserLookup and CustomRequest tables.

1.0.3.0 - 21/02/2014
  • Fixed error on table note for UserLookup table - the API used for these allows 180 calls per 15 minutes (not 15 as previously stated).
  • Added Post_UserLookup_Info and Post_UserLookupById_Info tables.

1.0.2.0 - 15/01/2014
  • IMPORTANT: Fixed bug where Search and UserSearch tables were not using https (which is now required). If you are using either of these tables you will need to upgrade to this version.

1.0.1.2 - 04/12/2013
  • UserLookup table no longer fails completely if one or more of the usernames do not exists. An error will now be logged to the log file.

1.0.1.1 - 27/11/2013
  • Fixed CustomRequest table.

1.0.1.0 - 08/11/2013
  • Now forcing result_type=mixed on search requests. This is documented to be the API default however in practice it seems to not be.
  • Added Favorites table.
  • Fixed bug where retweet_status media_photo_count and media_photo_urls were picking up the values from the main tweet.

1.0.0.9 - 27/08/2013
  • BREAKING CHANGE: Removed cursor column (please use the new Post_FollowerIds_Info table).

1.0.0.8 - 17/06/2013
  • Added more columns to TweetLookup table (to match columns from All Columns search table).

1.0.0.7 - 14/06/2013
  • Fixed info labels on UserLookup tables.
  • Added new FollowerId tables.
  • Better handling when quota limit is reached on Follower/Following tables.

1.0.0.6 - 29/05/2013
  • Added TweetLookup table.

1.0.0.5 - 16/05/2013
  • Added favorite_count column for Tweets (this seems to have been recently added to the Twitter API). Note that despite this favorite_count and retweet_count always seem to come back as 0 from the API (we have posted about this - https://dev.twitter.com/discussions/17653).
  • Added symbols columns (e.g. to pull out count and CSV listing of things like %QLIK).
  • UserTimeline now returns up to 3200 Tweets.
  • HomeTimeline now returns up to 800 Tweets.

1.0.0.4 - 30/03/2013
  • Removed maximum 15 page limit on Search. It is now possible in theory to get 180x100 = 18,000 Tweets per 15 minute window.

1.0.0.3 - 08/03/2013
  • Added missing comman after ???_timestamp columns.

1.0.0.2 - 07/03/2013
  • Should display balloon message if not authenticated.

1.0.0.1 - 26/02/2013
  • Added lang column for Tweet objects (although at time of writing this has not been added to the API -> https://dev.twitter.com/blog/introducing-new-metadata-for-tweets).
  • Added proper timestamp, date, and time columns for created_at fields.
  • Added profile_image_url to users.

1.0.0.0 - 25/02/2013
  • No longer beta.
  • New tables added.
  • Bug fixes and new fields added to search tables.

0.8.0 - 18/02/2013
  • First public release.

0.7.0 - 11/11/2012
  • Initial release.


(QVSource works with Qlik Sense as well as QlikView - See this page for notes.)
QVSource - The QlikView & Qlik Sense API Connector | © Copyright 2011 - 2016 Industrial CodeBox Ltd