Is it possible to measure IT Security?

It is a commonly held principle in many areas of business that if you can’t measure something “quantitatively”, it will be difficult to raise the quality objectively. The applicability of this statement to the world of IT security is clear. Without having some form of metrics in place, it is tough, if not impossible, to judge whether security is getting better over time. Indeed, it is probably fair to say that many organisations have only one way to assess security – namely, “did anything go wrong” – but this is hardly a metric for the forward-looking.

Meanwhile, of course, the drivers for proactively monitoring ’security’ and the associated effectiveness of security solutions are becoming increasingly high-profile. Regulatory pressures on organisations to secure their operations are more explicit than in the past, while customers and shareholders are less prepared to tolerate IT security breaches. The continuing spread of legislation dictating that organisations actively notify affected parties when data is potentially lost or at risk is certain to add to the pressure to demonstrate that security measures are properly established.

Another potential driver for security monitoring and measurement which is very easy to overlook is cost-justifying the role played by individual IT security measures. If it were possible to evaluate the effectiveness of security in simple terms, for example through verifying numbers of attacks and threats which have been prevented from executing, such measurements could help justify existing spend. They could even validate requests for additional resources as either the threat landscape changes or as new business requirements come into play.

And all the while the nature of security threats are changing, both in the technical vectors used to breach systems and in the “philosophy” of those organising system attacks. The days of hackers attacking systems for the kudos of breaking into systems are over. Today the vast majority of security breaches are commercially driven with the goal of making money. With threats becoming more sophisticated, how can an organisation test the effectiveness of even the basic elements of security tooling, such as anti-virus solutions, firewalls and web page checks?

Some organisations do attempt to line up a range of tools, say in the anti-virus space, and compare how well they detect threats. Most such tests rely on using ’known’ sources of malicious code and this approach is fine if an organisation is certain that it is only ever going to be subjected to the threats of the day before yesterday.

But as has already been stated, threats change all the time with new challenges being pushed into the arena almost every hour of every day. IT today has to be ready, in the words of Douglas Adams, “to expect the unexpected”. Even more importantly, and this is something that needs recognition by everyone working in an organisation, security is not ’somebody else’s problem’. As illustrated by the results of our last poll, security is a challenge to be addressed by everyone.

So if measuring the effectiveness of solutions against known threats is at best only part of the answer, the question arises of how security tools can be measured accurately against unknown threats or the real world at large, particularly against so-called “zero-day attacks” – that is, exploits on as-yet-undiscovered security holes. Is it really possible to test against the unknown? There are some moves afoot for example, to test security products using wild sources of infection rather than running tests on predefined, ’canned’ threats. It will be interesting to see how these new tests develop, and how much attention they get amongst IT professionals.

It is likely that a growing number of organisations will look at measuring security and extend this to attempting to qualitatively gauge the effectiveness of the security tools on offer. But given just how difficult it is to apply any measure to security, it is important to look beyond security tools and processes, and look at where measurements can be applied in reducing risks across the board. For example, we know that the education of users concerning their responsibilities in protecting systems in data has significant benefits in raising security effectiveness – and it is perfectly possible to measure the level and effectiveness of awareness across an organisation.

It should also be within most organisations’ ken to run auditing tools on a regular basis, and log the outputs as part of an ongoing security improvement programme. It is unlikely that any organisation will come out unscathed, but this uncomfortable truth should not be a reason – as we have heard from a number of organisations – for any such checks to be turned off, for fear of what they might turn up.

In IT security there are few absolutes, but a good start is at least to identify a baseline which can be built upon. If you have found any good ways to measure security tools and IT operational security effectiveness, we’d be very interested in hearing your secrets of success.

Tony Lock

Click here for more posts from this author

Tony is an IT operations guru. As an ex-IT manager with an insatiable thirst for knowledge, his extensive vendor briefing agenda makes him one of the most well informed analysts in the industry, particularly on the diversity of solutions and approaches available to tackle key operational requirements. If you are a vendor talking about a new offering, be very careful about describing it to Tony as ‘unique’, because if it isn’t, he’ll probably know.