Interview with Toby Potter from DataSift - Social Data and Business Intelligence
With the BI Forum fast approaching, Toby Potter from DataSift agreed to an interview for the blog, to discuss social data, it's value and how it can be combined with other business intelligence. Toby will be one of the guest speakers at the BI Forum in Brighton sharing more details and experiences in this growing area. Over to the interview...
[James Knight] "For the readers of the blog that may not have heard of DataSift, tell us a bit about the company and what they can provide."
[Toby Potter] "DataSift provides the leading Social Data Platform for enterprise. DataSift collects, structures and enriches unstructured social, news and blog data allowing our customers to filter and collect the data of interest to their business for further, more detailed analysis.
We provide real-time access to the full Twitter firehose, bit.ly (click) data, Facebook, YouTube, millions of blogs, news and others, complemented by our historical archive of data."
[James Knight] "Big data is a hot topic and lots of our customers want to do some 'big data stuff', but what does the term mean to you?"
[Toby Potter] "Whilst we consider ourselves to be a Big Data Platform - our core platform is into the Petabytes and we’re growing at several Terabytes per day - our focus is really on the Right Data. Using the power of our filtering capabilities, we’re able to take those huge volumes of data and provide the Right Data to our customers based upon the rules they define.
Even given that we are processing 1 Trillion (1,000,000,000,000) items of social data a week in our historics platform - and doubling every 2 months."
[James Knight] "There's a ton of social data out there, which may make people wary of the effort, timescales and costs involved. How can people start their journey in the use of social analytics and gain benefit quickly?"
[Toby Potter] "Social analytics is actually pretty straightforward to get started on - for example, you can simply create a Pay As You Go account on the DataSift platform and be collecting data within a few miutes - but the challenge is often realising the insight that’s available within social data. A typical tweet for example, might have anywhere between 50 and 100 individual data points.
As with any analytics project, be clear on your objectives and you should see whether this is going to be of benefit very quickly. Think about how you would capture the data, what SLA (if any) do you need if this was to become mainstream, how would you analyse it and do you want to merge it with existing, internal sources.
The bottom line however, is to understand the action you would take from any insight gained. No matter how interesting the results, if there’s no action taken from the insight, you have to question the value of any analytics project."
[James Knight] "So far we've been talking about social media in relation to big data, but how can it also be used to enhance regular BI reports, such as those produced using Oracle BI EE?"
[Toby Potter] "Typically, BI reports focus on reporting on data that’s available internally; sales data, customer data and effectiveness of a marketing campaign for example.
Social data in and of itself provides insight into the world outside; bringing this into your organisation’s Business Intelligence environment provides the business with a whole new perspective and delivers a new view of how your business is performing.
Both uses provide tremendous value, but it’s the combination of the two that really drives insight. Being able to link social discussions, or a particular news story, drove product interest and traffic to your web-site which both increased online sales, but also drove footfall to your physical stores (which also saw an uplift in sales) would provide the justification in running more similar campaigns. Understanding the social profiles of the customer base may also help you tune your offline messaging too."
[James Knight] "Out of those that you are allowed to talk about publicly, what's the cleverest use of social data that you have seen so far?"
[Toby Potter] "This is what I love most about my job; there are so many uses for the data that almost every discussion is different!
In the media business, I would point towards SecondSync as a great example of using social data to disrupt and add value to the fairly traditional business of measuring TV audiences. They capture social discussions around particular TV programmes and provide both a dashboard view and deeper analytics services to allow broadcasters to better understand their audience and how to engage with them.
News is an obvious area for analytics on social data and many of our customers are able to understand how stories are shared socially, enabling them to better target news stories, drive up audience and therefore drive increased advertising revenues.
Finance is another interesting area; the ability to bring together all of the discussion around particular investments, measure the sentiment and identify breaking news and the market reaction to it quickly, gives our customers an edge.
Customer services is another area and not just being able to react to complaints. Understanding what is working and what isn’t for your customers, collecting feedback and relating that to churn figures for example, allows you to understand how better to adapt your business. As an example, when the broadband goes down in a particular area, people tend to tweet about it very quickly; whilst your engineers may well be alerted, customer services are often left to pick up a sudden, increased load, without having any ideas as to why or what the problem might be. Integrating social data helps overcome this.
More broadly, by looking into the data itself, I’ve seen our customers collecting all of the geo-located tweets to understand population density to better inform telecoms infrastructure; retailers analyse data to understand future fashions to better inform stock ordering/promotions; competitor benchmarking to understand how your business compares to your peers.
The most innovative I’m seeing at the moment is the ability to build out profiles, much richer than simple demographics, of social customers to better understand their interests and so provide them with more useful promotions.
One of our customers, Local Response, have integrated a real-time social feed into their online advertising platform to drive “Intent Targetting”, the ability to target more appropriate and context sensitive advertising to web-site viewers."
[James Knight] "Social data can be quite messy. How can organisations uncover value?"
[Toby Potter] "As a gross over simplification, there are two fundamental elements required: the ability to capture, process and cleanse the data in the first place (this is where DataSift comes in), and the ability to analyse it, which is where the Oracle platforms and Endeca in particular come into play.
Getting the data into a format where it can be analysed initially - for more traditional types of data this would typically be some kind of ETL tool - is key to being able to then store it ready for further analysis.
Endeca is fantastic at then taking this data and providing a flexible “data playground” allowing users to discover what insight might be hidden inside. For more structured analytics the more traditional BI tools provide a great environment to distribute reports around the business to a broader Business As Usual user base."
[James Knight] "Technically, how are companies dealing with social data, and do you see a standard approach or a variety of different approaches?"
[Toby Potter] "As is to be expected in a relatively young space, there are many approaches. The majority of businesses are dipping their toes in and pulling data directly from the various provider APIs out there, but this provides a very distorted view of the potential; most APIs provide very restricted data sets and have limits on the number of requests you can make, so you’re potentially missing out on a lot.
Increasingly, as organisations recognise the importance and value of the data available, they are turning to specialist providers such as DataSift to provide a reliable, enterprise quality feed of the data they need.
We see the same journey often - initial experiment seems to yield value; build infrastructure to handle and incorporate data; deliver valuable insight ... and then an API changes, or a major event changes volume levels, or the business interest grows and so on. The result is almost always frustration with the amount of expense, lack of reliability and general hard work that is required to get this working, but the insight is so valuable the need for a working solution over-rides everything else.
It is all of this work that DataSift aims to replace with a single platform to take away this maintenance and reliability nightmare.
If you’re looking at social data coming in to your business, I’d highly recommend looking at the tools that are already out there to save you a lot of time, money and frustration!"
[James Knight] "At Rittman Mead, we've been working with Oracle Endeca Information Discovery (OEID), allowing us to combine unstructured, semi-structured and structure data. What additional value do you see OEID providing in the analysis of social data?"
[Toby Potter] "This is exactly where the maximum value from the social data can be gained. Combining the data with existing sources, enhances the potential from the data, but having the right tools to extract this value is fundamental. Endeca provides exactly that kind of creative analytical environment where you can explore the data and determine where the most value lies.
Using Endeca, once the data is better understood, it’s then straightforward to productionise the areas of maximum benefit and focus energy on delivering that insight to where it can have most impact in your business."Many thanks to Toby for taking the time out for this interview, which we hope has provided some useful insight. There's still time to register for the BI Forum and see Toby's presentation at the Brighton event.