Business Technology

Published

The big data conundrum

4 Mins

Having just edited a supplement for The Times on Big Data I got really excited by the possibilities. New York City’s new data-guy increased the hit rate of illegal-housing conversion detections from 13 per cent of properties inspected to 70 per cent purely by using Big Data analytics. When Big Data delivers it really does live up to the hype.

But! A few firms I talked to really struggled to come up with any demonstrable gains, or weren’t doing Big Data at all. And then I spoke to Clive Dunn, founder of Dunnhumby and the inventor of the Tesco Clubcard, who told me Big Data is often over-hyped. “Most firms would be far better off looking at the simple things they can do,” said Dunn. He reckons that many firms will probably never find anything worth much in their data, and would be better off spending their time and resources elsewhere. This, from the king of Big Data.

Why is Big Data problematic? Five reasons:

1. Dunn’s right about the dangers of neglecting small data

 Often, firms would be far better off looking at their basic offering, rather than looking for esoteric trends in data.

2. It is expensive

Those Big Data scientists usually have a PhD in maths. And it takes a team of Big Data specialists to run a project. Which is great if you are Vodafone or Aldi. But small firms will struggle to justify the budget.

3. You may not have enough data

One message which really came loud and clear from the industry is that there is a difference between Big Statistics and Big Data. The former uses ordinary functions on a calculator: plus, minus, multiplication and percentage buttons. Taking limited datasets and doing routine maths on them is fine, but ain’t Big Data. Nor need it be – applying algorithms to small data is overkill, and there’s no shame in saying so.

4. You’ll know what, but not why

A curiosity of Big Data is that you may discover amazing correlations, but never grasp why the pattern exists. For example, supermarket shoppers may buy aubergines whenever they buy garlic, but you will never discover the full recipe. In fact, Big Data scientists often don’t want to know. Not their domain. So you get the “what”, but not the “why”. Which may limit the value of Big Data.

5. It’s hard to know what questions to ask

Big Data consultants make this point again and again. You never know what the data will tell you, but need to ask a question in order to delve into the data. It is chicken and egg. The solution is to play with the data, find leads, then ask questions, and then play some more, then refine the question again. Each Big Data firm will have their own methodology, but this is the basic way it works. You need to accept the fact that your data may contain gems, but you may never find them. Which is annoying.

Conclusion? Big Data is a wonderful thing for firms with the right data pools, and the brains to know what to do with it. But it is becoming obvious that it is not right for every firm. If you are too small, too data-light, or still focusing on more basic strategy, there is no shame in bowing out and saying you’ll stick to easier stuff.

Charles Orton-Jones is a business journalist.

Share this story

Government pumps more funds into Start-Up Loans scheme
Top tips on how to brainstorm
Send this to a friend