Mining the gold seam of structured data

By Andrew Buss

Much of the talk of the last year has been about Big Data and how to get timely, actionable intelligence from the whole range of information assets on which to base better business decisions. With relational databases for many being a fairly mature quantity, much of the focus has been on developing new ways to mine the morass of unstructured data that is being created through collaboration applications such as email and IM, as well as new applications and services such as social media. But are many companies missing a trick by not further developing and utilising their structured business information assets?

Taking a step back and looking at data overall, both structured and unstructured data are growing, and at similar –usually high – rates for most companies. Just dealing with this growth is a challenge.
Despite the growth in data volumes overall, a recent survey on Big Data highlighted that the most valuable business information is predominantly held in a structured format. Not only that, but there is no massive trend away from traditional databases and similar tools (Figure 1)

Figure 1

The point here is that while it may well be that there are new information sources becoming available through all the unstructured data being accumulated, it is often easy to forget there is a valuable pool of structured information that already exists to tap into.

Many would argue that being structured, this data is already being used to drive better-informed business decisions, and to some extent, they’re right. But even so, many companies are far from perfect when it comes to this, with less than half of respondents feeling that they are performing well here (Figure 2).

Figure 2

When it comes to exploiting unstructured data, there is undoubted potential, but less than 1 in 5 respondents thought their company was above average here, with the result that a lot of time and learning will be needed to really make progress.

So without ignoring the long term potential of mining unstructured data in the longer term, it is clear that there is still a lot of value to be reaped from improving access to and insight from both existing, and potentially new, structured data stores. The question is then how to go about doing this.

Bearing in mind that this was an online survey, there is likely to be some skew towards respondents with an interest in the subject. However, we can still look at the differences in behaviour between those who feel they are exploiting their structured data well (those who answered either 4 or 5 to exploiting structured data in Figure 2) and compare their approach with those who feel they could do with further optimisation (others) (Figure 3).

Figure 3

What comes through in the survey is that those companies that exploit their structured information well tend to invest more in a number of the cutting edge data management solutions such as in-memory and WORM databases; scale out storage architectures; and distributed data analytics, indexing and search.

Yet it is also apparent that these companies also make much more extensive use of mature, bread and butter relational database technologies. The use of general purpose RDBMSs is still noticeably higher in this group despite the commoditisation and extensive use of these products across all companies.

But where these companies really stand out is in their use of optimised RDBMSs – particularly high performance systems that can work on finer grained data or return results in real-time or at least close to it – and OLAP systems that enable them to make more effective use of the structured data that they do have in place already.

With this in mind, when a vendor comes calling looking to implement a Big Data solution to unlock the potential of your business information, the best way forward in getting better business outcomes may not be to rush off and implement a brand new Hadoop cluster but instead to embrace and extend the good old database with new performance and analytical capabilities.

CLICK HERE TO VIEW ORIGINAL PUBLISHED ON