Want Budget; Use Metrics

We have all heard the business adage that you cannot manage what you don’t measure. For those in Information Security or Information Technology, this can have far-reaching implications. Without concrete data to query and present, business unit leaders are left wanting. It is difficult to grasp the importance of security or its necessity if there is nothing to back it up. A sound Metrics Program can help.

There are three main components to a Metrics Program: measuring, benchmarking, and metrics. Each work in concert with one another. In order for a Metrics Program to be effective, you must first determine what needs to be measured. Since it is easy to become inundated with mountains of information, it would be prudent to take it slow in the beginning. There is a lot of data out there to gather, but not all of it is valuable. So, first determine what is meaningful. For example, what if you want to measure the effectiveness of your email security solution? Below is the data that should be measured:
• Number of messages per day
• Number of spam detected
• Number of spam not detected
• Number of spam false positives
• Number of viruses and spyware detected in email
• Number of viruses and spyware not detected in email

Above are a number of items to measure, and while beneficial in many regards, they don’t achieve optimal performance until benchmarking occurs and metrics are defined.
Benchmarking is critical not only in a metrics program but in many aspects of security, IT, and business. For example, suppose you start benchmarking the response time to the company website. You will probably be measuring the number of sessions per hour and bandwidth utilization. In the process of measuring this information, you notice that bandwidth utilization was at 90% and you immediately determine that the company needs more bandwidth. If you baselined over a period of time and used some rationalization to realize that the bandwidth spike was due to “Black Thursday” and not indicative of normal bandwidth trends, you probably would have concluded that you don’t need to give more of your IT budget to your ISP. Benchmarking puts it into context.

Now after you ascertain what to measure, you will need to determine your metrics. While the list above outlines data to measure, metrics provide the basic statistical analysis. It provides meaning. Now if I told you that last year we had 1000 computers infected with viruses, that would startle many people. However, if I worked at an organization that was a global firm with 140,000+ employees/computers, that would be .006% or 1 in 144, and it does not seem so scary anymore.

Following are some metrics for the spam filter/email virus blocker example from earlier:
• Percentage of detected spam to total messages per day
• Percentage of undetected spam to total messages per day
• Percentage of false positive spam to total messages per day
• Percentage of detected email viruses to total messages per day
• Percentage of undetected email viruses to total messages per day

After using the email metrics above and some additional ones, you determine that email viruses constitute a large portion of your virus outbreaks. At this point, you want to know if your email security solution is effective or if you should go shopping. You are now in a good position to make a determination since you have the data behind you and a solid Metrics Program. By analyzing all the data, you determined that 100 out of 714 infected emails bypassed your email security solution’s detection, which equates to an 86% detection rate. The product you are considering has a 96% detection rate, which means that the number of viruses that bypass detection drops to 28. Your company policy dictates that an infected machine must be reimaged. It takes roughly 2 hours to reimage each machine, which costs the company $200 per hour in employee and opportunity cost. Based on 72 less machines becoming infected, the maximum solution price should not exceed $28,800 more than the existing solution in order to remain economical. Armed with this analysis, it is increasingly easier to acquire budget and far more effective than building a case on feeling.

While the example illustrated in this blog is a mere hypothetical scenario, the benefits of a Metrics Program are substantial. It can illustrate the effectiveness and performance of your security program, diagnose trends and issues, determine resource allocation, and secure security budget. If you would like more information or help in establishing a Metrics Program, let us know; we can help.

By Jason Suplita, Senior Risk Management Consultant