Steps to managing the data explosion

By Tony Lock

Most organisations are experiencing considerable growth in the volumes of data they hold. As a consequence storage management is now becoming a visible challenge in many organisations, especially in terms of the costs associated with acquiring and operating storage platforms. It is not, however, only the growth in the quantity of data being created and stored that is challenging IT departments. External regulatory and legislative requirements governing where and how data should be held and secured are also raising concerns. As importantly, users now expect all data to be available to them at all times, even though it may have been many months or even years since that data was last accessed.

With all of the marketing around cloud solutions, a frequently raised question is: what role might cloud storage play in data storage and information management going forwards? In particular could cloud systems be utilised to store ‘archive’ data, namely information that is infrequently, if ever accessed, but which the company, or the individual user, does not wish to be permanently erased.

Figure 1

Over half of respondents to the survey stated that their policies governing how long data should be kept, or indeed what data should be stored long term, are patchy or ad hoc; indeed only 20% of these enterprises indicated they have comprehensive policies in place. It will be interesting to see how long the large minority who do not consider it necessary to have any policies concerning data classification and storage can maintain this approach, especially as legislative and industry specific regulatory constraints become felt in organisations of all sizes.

What also becomes clear from this study, and others Freeform Dynamics has performed in recent years, is that many organisations recognise they could better exploit the corporate information sources they hold — if only they could find the data easily and make it securely available to the correct users at the right time. Thus, as well as well as the growing cost of storage acquisition and management being a concern, there is also a widely perceived potential upside improving the exploitation of corporate information better.

One of the areas identified by those organisations with the more comprehensive information policies in place as offering benefits, is to focus on consolidation and simplification. This, they suggest, should apply wherever possible across the storage infrastructure considered as a whole. Such a concept is easy to understand following other IT infrastructure consolidation initiatives, especially those involving server virtualisation. Storage consolidation projects can also have clear business case / cost advantages and often provides a simple starting point for other storage projects to move forwards.

While many storage consolidation projects are based on cost savings, similar claims have also been made for the potential opportunities for organisations to exploit external ‘Cloud Storage’ offerings. However, for a wide variety of reasons encompassing concerns over security, where data is stored and basics such as simple guaranteed network access and responsiveness, few organisations of any size are yet willing to move ‘live’ data to the ‘cloud’.

However, one possible area of best practice identified by organisations in the research, indicated that cloud based solutions might be a potential answer for data that could be “archived”. By its very nature archive data is rarely, if ever accessed by users, but represents information which the user or the organisation is not prepared or able to delete permanently from its systems. That said, few organisations have comprehensive archiving policies in place and it is often the case that only specific data sets are subject to such policies where they exist. Quite often the data sets subject to archiving are those covered by legislative retention requirements.

So, how can organisations move forwards to manage data more cost effectively, and more actively utilise archiving solutions as well decide whether this is best performed in the cloud or in house?

Figure 2

The best practices most often identified by those organisations with effective information management procedures are summarised in Figure 2.

The key stages identified involve basics such as establishing precisely what data the company holds and then asking senior business managers to establish its value to the organisation. At this stage any constraints on data storage, resulting from regulatory or legislative requirements or simply internal ‘governance’ and usage expectations, should be matched to data classes. This is especially important in terms of identifying security requirements and whether any inhibitors or restrictions on where certain classes of information may be stored apply.

There is an additional point to note. Such issues may have a profound impact on which categories of corporate data should or should not be held in external archiving systems (whether in-house or in an external storage clouds in general). The result should be the implementation of straightforward storage management policies to categorise data and know where to store information at different stages in its life, including archiving off to low cost long term platforms (where instant access and performance is not a requirement).

In this context it is important that all users, especially senior managers understand not only the policies themselves but why they exist if the organisation is to expect such guidelines to be followed. Users who understand why data storage policies exist are far more likely to follow them and are usually less inclined to find ways around the policies if they have a solid understanding of potential business consequences. This is significant, particularly where end users may have started to utilise consumer cloud storage solutions, such as Dropbox etc., that are targeting end consumers rather than enterprises to hold sensitive corporate information.

The organisation can then assess which options it has available to store data over long time frames, including which information could be migrated to an archive. It could well be that initial steps might consider utilising cloud or off-site based archiving solutions to help reduce the volume of information held on ‘live’ systems and the effort of managing the information in the data centre.

From conversations with many data centre managers it is clear that the potential use of cloud repositories to hold data that is actively accessed and modified is likely to remain relatively low for the foreseeable future. Few It professionals and businesses are today comfortable with moving such primary business information to the cloud. But cloud archiving solutions have been around for some time and could be an attractive option for some enterprises.

ORIGINALLY PUBLISHED BY

Tony Lock

Click here for more posts from this author

Tony is an IT operations guru. As an ex-IT manager with an insatiable thirst for knowledge, his extensive vendor briefing agenda makes him one of the most well informed analysts in the industry, particularly on the diversity of solutions and approaches available to tackle key operational requirements. If you are a vendor talking about a new offering, be very careful about describing it to Tony as ‘unique’, because if it isn’t, he’ll probably know.