In recent years the importance of analytics has grown dramatically, indeed an increasing number of organizations now want to use real-time analytics to make operational business decisions. The challenge is the amount of work – often repetitive work – that analysts and data scientists must carry out to make their data just perfect for the workloads they seek to apply against it.
This drudgery was bad enough when we built data warehouses and ran BI (business intelligence) queries against them to look for historical trends we could predict from. That kind of backward-looking reporting was yesterday, though.
Today, the value of information has never been higher, hence the demand for real-time data analytics – but that risks making the drudgery far broader in scope. Real-time analytics means that the data scientists and analysts doing the work cannot wait days for data to be extracted, cleansed, transformed into suitable formats, collated, and finally made available to be worked on. In short, they need an IT infrastructure built to enable them to work as analysts, not as data cleansers. They need a more responsive data ecosystem.
As a consequence, IT systems need to support five key requirements via a combination of hardware, software and processes.
There is another factor of which it is important to be aware, but it is one that is easy to overlook, namely data governance. There have always been legal limits on how an organization is allowed to use the data it holds. but today legislation such as GDPR and other privacy requirements make it essential that data be used responsibly. Add in the expectations of customers and shareholders, and data security and governance cannot be ignored.
Getting points one to five right can take much of the drudgery out of analytics, but some challenges are likely to remain. But improving the ability of the organization to use data is essential to business success.