Category Archives: Big Data

Handling Big Data: How to balance fast response times and large data volumes

data-balanceDuring this series, we’ve looked at some key requirements of a modern information delivery platform. This series wouldn’t be complete without writing a word or two on the subject of Big Data.

With the growing popularity of Big Data projects, it is important to remember that the full value of Big Data is only realized when you transform it into business insight and get it into the hands of the people who need it, across your organization.

This requires an interface onto Big Data that is both intuitive and responsive, so business users can quickly navigate these large data sets to reach important information that help drive their daily activities.

Continue reading

Are you missing most of the value of your Big Data?

Big DataIf there is a data lake in your organization, what percentage of your organization see data from it on a daily basis?

I suspect the answer in most organizations, even the most Big Data savvy ones, is very low – possibly much lower than 10%.

Continue reading

Big Data – is it as big as it seems?

Big Data204,000,000 emails, 4,000,000 Google searches, 2,460,000 pieces of Facebook content, 277,000 tweets.

This is just a small piece of all the information that’s shared online in one single minute. 2.4 billion people are now online. According to IBM, 2.5 exabytes – that’s 2.5 billion gigabytes! – of data was generated every day in 2012, and with each like, share and click the world’s data pool is expanding. But what does this mean for our organizations?

Continue reading

Big Data: is IT delivering?

IT at workIn a recent article, Paul Gittins at Capgemini wrote about a Big Data survey which asked 1,000 C-level executives and senior decision makers how data is changing businesses.

Among the results were the expected 64% of companies who believe Big Data is changing traditional business boundaries and the 24% who report disruption from new competitors moving into their industry. However, as Gittins points out, one unexpected stat stood out from the crowd of forseeables: Over a third (36%) of non-IT decision-makers surveyed said their business unit has circumvented IT in order to implement the data analytics it needs.

This begs an interesting question: Is IT risking irrelevancy?

Continue reading

Sharing Big Data insights with business users

Big DataAs we approach the end of February it seems like 2015 could be the year where the backlash starts against Big Data. I’ve seen numerous articles this year. including this one from InformationWeek, this one from ZDNet and this one from Quartz – ouch!, which paint a less rosy picture of the value of Big Data and, in fact, Big Data in general.

Is it all it is hyped up to be?

Continue reading

Dashboards, BBC Weather and Big Data

A couple of days ago, I wrote a post outlining why I think the BBC Weather site points the way to the future of dashboards. Interestingly, not only does the BBC site provide good lessons for dashboards in general, but I also believe it has a lot to say about how to get the best out of Big Data.

Big Data is generating a huge buzz in the Business Intelligence community at the moment, so much so that I have heard people suggest that if you are not currently taking Big Data to heart then your organization is bound to fail. The truth (as always) is a little different from the hype; Big Data analysis is a powerful tool, but one which should be used carefully, and this is where I think that the lessons from the BBC Weather site can help.

As I said in the previous post, I think the BBC Weather site is an excellent example of a dashboard and if more dashboards were built using its “ENTICE” principles then we would see greater use of and greater value from BI in our organizations.

However, not only is the BBC Weather site an excellent example of a dashboard, it is an example of a dashboard written on top of a truly vast set of data. I have no doubt that the underpinning weather data would more than hold its own when compared to all of the “big” data sets being considered in BI organizations around the world today.

The important thing to note is that although the dashboard is underpinned by a huge volume of data (and very sophisticated analysis) the final interface for the end-user is incredibly simple.

As an end user of a weather dashboard I don’t need to see individual data points, or understand the analysis which has gone into computing the forecast, I simply need high level answers which will make a difference to me.

I use the end results of deep analysis performed by highly skilled people served up to me in such a way that I don’t even really know I am using it.

The same is true in business. The business of analyzing big data lies with data analysts (or “data scientists” as they seem to be becoming known) that is their job; it is what they are paid to do. The rest of us have another job to do, running our organizations, something we can do better with insight gained from big data analysis … but, the quicker we can get into the numbers, get what we need, get out and get on with our  job, the more efficient we  will be. For me, the obvious way for that to happen is to have that insight packaged up and accessible, simply and intuitively, through dashboards.

A couple of years ago it was reported that someone said “HANA will make exec dashboards obsolete”. Nothing could be further from the truth; interactive dashboards are the perfect interface for the vast majority of us to consume the big data analysis performed by the comparatively tiny number of data analysts/scientists who understand it all the way down.

In fact, the more I think about it, the more I see interactive dashboards as the ONLY required access mechanism to data in organizations for everyone who does not have “analyst” (or scientist) on their business card … but, that is an idea for another day.

If you would like to find out about the quickest, easiest and lowest cost way of creating interactive dashboards on datasets large and small then take a look at DecisionPoint. For more details about DecisionPoint, watch this short two-minute video and see for yourself how it is the perfect tool to replace Xcelsius.

This post also appeared on the SAP Communist Network (SCN) here.

The BBC Weather page and the future of dashboards

Drawing a comparison between a business dashboard and the BBC weather page might seem odd, but I am convinced it is valid and more than that points a way to the future of dashboards as we use them in business.

Over the past few years the world of dashboards has moved on from “at-a-glance dashboards”, which just give a summary overview, to “interactive dashboards” which allow the end user to click (or more recently tap) on the screen and navigate their data in a simple, intuitive manner.

So, modern dashboards give both a summary of the situation and the ability to easily get more detailed information through simple, intuitive navigation. Which brings us to the BBC weather page, as it also exhibits exactly these properties, as this 25 second video shows:

People say that the “consumerization of IT” means users want a Google-like interface to their data. I disagree, consumerization really means that people will want a BBC-Weather-like interface to their data. To my mind this means six things:

  1. Easy-to-use
  2. No training required
  3. Task focused
  4. Interactive
  5. Connected to data
  6. Engaging

And it is no coincidence that the first letters of these spell ENTICE, because the best dashboards have the ability to entice users into using them like no BI technology has ever done before.

The fascinating thing is that the weather app is letting me do multi-dimensional (or OLAP) analysis on the data. E.g.:

  • More detailed forecast = drill down on time
  • Different day = drill across on time
  • Different Location = drill across on location
  • Map = slice-and-dice to view data by geography rather than by time
  • Hour-by-hour = drill across geographic view by time

So it is no surprise that this is exactly the type of navigation required in an interactive dashboard. For users of SAP BusinessObjects Dashboards / Xcelsius this is a double edged sword, because although Xcelsius was one of the first dashboarding tools to provide the flexibility required to create this type of interactive dashboard … with Xcelsius it can become complex to implement this cube-based, OLAP style navigation in the underlying two-dimensional Excel spreadsheet.

Fortunately, there is an answer to this in the form  of DecisionPoint from Antivia. For more details about DecisionPoint, watch this short two-minute video and see for yourself how it is the perfect tool to replace Xcelsius.

As a footnote, it is interesting to note that the BBC Weather “dashboard” is written in HTML and JavaScript and it also has something important to say about “big data” but those are stories for another day.

This post also appeared on SAP SCN here.

Personal Analytics – A Cautionary Tale for Big Data Projects ?

Big DataIn a recent blog post Stephen Wolfram, creator of Mathematica, author of “A New Kind of Science”, creator of Wolfram|Alpha, and founder and CEO of Wolfram Research, wrote how, over a long period of time, he amassed “probably one of the world’s largest collections of personal data“. In the post, he walks through various analyses he recently performed on this data.

In my opinion the results of these analyses hold an interesting, cautionary tale for people working in the new world of Big Data, where there is a risk of analysing data just because it is there.

My assessment of the various results of Stephen’s analysis would be:

  • 95% obvious – (e.g. “there’s been a progressive increase in my email volume over the years”, “peaks [in email] are often associated with intense early-stage projects, where I am directly interacting with lots of people” )
  • 5% interesting but not useful – (e.g. “7% of all keystrokes are backspaces”, “a large volume of Stephen’s work has been done between midnight and 6am”)

The working-at-night observation might be more interesting if data were analysed across many people to see if there was a correlation between not requiring much sleep and success. Alas, even if this were true it would not be that useful (unless you believe that you can train yourself to require less sleep).

There is one area which might be useful, that is the analysis of the amount of time Stephen worked on his book “A New Kind of Science“; the data here may help him, in the future, estimate more accurately how long another book would take to write but, ironically, it is unlikely this would help him write another book any faster.

One would have hoped that with all this data, collected over so many years, and studied by someone with such an analytical mind, that there would be some usable insights, ones which could be acted on to make a change for the better … but, alas it would seem not.

And, that is the cautionary tale for the BI world

Be wary of “analysis for its own sake” or those suggesting that expensive Big Data projects “don’t need business requirementsbecause they are “finding insight that wasn’t known about before”. All too often, these types of projects produce interesting results but no useful information (and by useful I mean actionable, i.e. information which can be used to drive action and therefore hope to change something).

Don’t get me wrong, I am as convinced as everyone else that the analysis of “Big Data” will provide many valuable insights over the coming months and years. I am just concerned that if we plunge headlong into it, without thinking, the amount of time, money and effort wasted will far outweigh the benefits.

As I have said before, with any business intelligence / analytics project, it should always come down to business requirements. Always have an idea of what you are looking for and why. A project to “analyse our web site data to understand if there is a better way of laying out our site to keep people on it for longer” is many, many more times more likely to produce a tangible result than a project to “analyse clickstream data to see if there is anything interesting we didn’t know”.

Perhaps I am being too cynical, if I am I would love to hear any stories about a data analysis project which produced something truly unexpected that was used to make a significant business impact (apocryphal stories about beer and diapers/nappies need not apply).

For more thoughts on big data, particularly as it affects dashboards, watch my on-demand webinar here.

What is Big Data ? – another sterile BI debate ?

There is a lot of talk about “Big Data” at the moment, but, as always, it is difficult to really get to the bottom of what it is all about. This was demonstrated on Twitter this morning during a conversation between Steve Lucas and Timo Elliott from SAP.

Steve Lucas: Wondering if people agree on the definition of the term “big data”. If anyone is willing can you direct message me your definition?

Timo Elliot: @nstevenlucas reality is that #bigdata term was created to talk about #Hadoop, etc. But everybody has interpreted largely, no “true” def.

In my mind, the issue is that the use of the term, “Big Data” is most wide-spread (and most abused) in the marketing of products and services. So, as Timo hints, every “definition” is slanted towards the particular marketing whims of a particular vendor. A good example of this is the recent press release from Reardon Commerce, which talks about analysis of punctuality data from 67M US airline flights as “Big Data”.

I am not sure that this data really counts as “Big Data”. I have not thought about this deeply, but I am relatively sure that with a copy of SQL Server Express and a moderately powered laptop you could get to all the same headline conclusions highlighted in the press release (e.g. this is historically the worst day to travel, this day is not as bad as you would expect, …) in a matter of hours or days.

That is not to say that this analysis is not useful. Significant insight can come from analysis of small amounts of data, the point I am making is that labeling this as “Big Data”, is probably not that helpful in terms of understanding what the term means.

In fact, with “Big Data”, I think we have quickly reached the point we often get to with technology definitions (as Timo and I have discussed here and here), where the term actually does more harm than good, because the sterile discussion of what it means wastes time and gets in the way of focusing on what really matters.

As always, the key is to approach things from a business, rather than a technology angle. All you need to be aware of is that there are new, maturing technologies which allow you to analyse unparalleled volumes of data. If you have a lot of raw data and a business problem you think it will solve then these technologies might be for you. But, ALWAYS work from the business requirements down and NEVER ask (or worry) if a particular problem is an example of “Big Data” or not.