Yo, Tableau Web Data Connector!
Whether your data's stored in flat files, relational databases, big data clusters, or the cloud, Tableau supports an ever-growing ensemble of native connectors that allow you to connect painlessly and focus on the real task at hand: asking and answering questions with your data.
In prior versions, Tableau enabled developers to provide access to non-native data through ODBC drivers and a data engine API, but with the release of v9.1 and the introduction of Web Data Connectors (WDCs), Tableau has dramatically reduced the barrier to entry for developers to create custom connections to new data sources. Anyone with a little HTML and JavaScript know-how can open up a world of data to Tableau, and connectors are being written and shared by the community all the time.
Overwhelming opportunity
While my marketing colleagues have always had the privilege of building beautiful visualizations on top of their more traditional "business" data, the data I and my developer colleagues care about (bugs/features, their LOEs, and priority, or build run statuses and times, or application performance metrics) sits relatively dormant, just out of reach of the Tableau client.
With the introduction of WDCs, we now have a unified platform atop which we can enable self-service analytics for all of the data in our various cloud systems. Compared to the relatively unfavorable costs we would have had to shoulder in building interfaces to this data in the past, WDCs make the task significantly more palatable:
- The "web" in web data connector means any developer conversant in JavaScript can get their hands dirty. No ODBC or C/Java/TDE API knowledge required. No messy data dumps into intermediary text files.
- Support for incremental and full refreshes via Tableau Online and Server means the onus of automation is on Tableau. One less thing to worry about building.
- A unified platform means any work we do can be open sourced, easily leveraged by others, improved upon, and shared back again, reducing development and maintenance costs in the long run.
Because so much data from so many sources is so close at hand so suddenly, the opportunity is almost overwhelming. Rather than making a mad sprint to build out connectors for all of our data needs as quickly as possible, I decided to take a step back and try and build a system to make developing high quality connectors very simple and very repeatable.
Introducing the Web Data Connector Generator
The Web Data Connector Generator aims to reduce the time it takes to write a connector from hours to minutes. Built on yeoman, the self-proclaimed "scaffolding tool for modern web apps," usage is as simple as firing a command on the terminal (yo web-data-connector) and answering a few questions.
Before you get started, you'll want to think about the underlying data source or API you're connecting to:
- Does it require some form of authentication? If so, how do you authenticate (e.g. basic authentication, OAuth, etc)?
- Does it support or depend on end-user configuration? If so, what configurations do you want to expose and how should you represent those options (e.g. a pick list, a text field, a checkbox, etc)?
- Does the API support cross-origin calls from a browser/client, or do you need to bypass CORS restrictions using a server-side proxy?
- Where do you plan to host your connector? Do you want to deploy to a cloud provider, or do you have your own strategy?
Based on your answers, the Generator stubs out a directory structure, adds heavily annotated application source files, fills in a few details, and downloads several utility libraries needed to help you get started quickly.
A first class web-application development workflow
Once everything's initialized, development couldn't be simpler! Just run grunt on your terminal and start hacking away in your favorite code editor.
For the uninitiated, grunt is a JavaScript task runner that automates tedious tasks and can be used to unify development or deployment processes across projects or teams. Based on your answers, the Generator defines and configures the set of tasks to be run when you run grunt in your Gruntfile.js.
Out of the box, it performs the following:
- Validates your JavaScript syntax and style,
- Compiles and minifies all JavaScript source code into a single file for high performance,
- Starts a persistent local development server that serves your connector
- Launches a daemon process that listens for changes you make to your source code, re-running all of the above steps on save.
As you're making changes, you can see and test your work in the SDK simulator, a browser, or Tableau itself at http://localhost:9001. Once you're done making changes, you can kill the grunt process by typing ctrl+c in your terminal.
If you're a practitioner of test-driven development, you're in luck! The Generator stubs out a number of basic unit tests. To run them, run npm test in your connector's root directory. You can add and edit tests in the test directory. And for those who've gone all-in on continuous integration and continuous deployment, a .travis.yml skeleton is also included at the root, for use with Travis CI.
Simplifying the WDC SDK
In addition to providing a basic structure and workflow for your connector, the Generator comes packaged with a wrapper around the connector API whose purpose is to allow you to focus on application logic without having to dig through SDK documentation.
The vast majority of the code you write will be in just two predefined methods: one to inform Tableau what fields and datatypes to expect from your connector, the other to retrieve and shape the data into the format expected by Tableau.
The wrapper takes care of the rest of the nitty gritty details, including:
- Naming and registering your connector with Tableau,
- Stepping through WDC workflow steps on your behalf,
- Boilerplate initialization during the user interaction phase,
- Providing helper methods for things like getting/setting persistent connection and authentication details from Tableau
- Providing a generic error handler for jQuery AJAX functions.
Examples in the wild
While developing the generator, we've spun up and published several connectors based on it, including a GitHub web data connector (a copy of which is deployed to Heroku) and a drupal.org data connector (hosted on GitHub Pages).
I hope this generator saves you time in sketching out connectors for all of your web data needs, and I'm very excited to see all of the connectors this scaffolding tool enables! Send me a link to yours and I'll add it here and to the Generator docs.
The source code for the Generator itself is also open and available for tinkering and I encourage feedback, ideas, and code to improve it.