CyberSecurity: The Business of Cybersecurity Capstone - Security Programs - Measuring Performances

This blog includes how metrics are used to manage cybersecurity program performance and a little about the design of such a metrics program.

Organizations strive to deliver the most value with a given level of investment. We call this the value proposition. The development and use of sound and repeatable cyber security management practices brings organizations closer to meeting this objective. Management wants to know that the security team is working to bring value to the organization. Measuring management practices and cybersecurity activities are an important part of that process. Executives often ask the chief information security officer that what we call the CISO, questions like,

• what will the security control cost me? Or 
• is your security system working? Or 
• even the most harmonious, why is this control system not working? 

That last question often comes right after something bad has happened. While CISO's might claim that the cost and benefits and performance of cybersecurity are impossible to measure, in fact they are measurable. It's just that doing that requires an effort to design and use a performance management program based on effective performance metrics. There's an old saying that you get more of what you measure. Applying that little bit of wisdom can lead us to the concept of cyber security performance management.



Cybersecurity Performance Management
Cybersecurity performance management is the process of designing, implementing and managing a means to use collected data elements called measurement or metrics to determine the effectiveness of specific part of the cybersecurity program. And hopefully the overall security program.
Performance measures are the measurements that may indicate the effectiveness of security countermeasures or controls, technical and managerial that are in use in the organization. Organizations typically use three types of measurements.

• Those that determine the effectiveness of the execution of policy most commonly issue specific security policies. 
• Secondly, those that determine the effectiveness and/or the efficiency of the delivery of cybersecurity services, whether they are managerial services such as security training or technical services such as the installation of antivirus software. 
• And finally those that assess the impact of an incident or other security event on the organization or its mission. 

Performance measurements are increasingly required in today's regulated environments. It's no longer sufficient simply to claim effective cyber security and organization must document that it is taking effective steps to control risk in order to support a claim of due diligence. The following factors must be considered during the development and implementation of a cybersecurity performance management program.

• First, measurement must yield quantifiable information. These would be percentages, averages or other number. This data supports the measurement needs and must be readily obtainable. 
• Second, only repeatable cybersecurity processes should be considered for measurement. This makes sure that the data being collected is reliable over time. 
• And finally, measurement must be useful for tracking performance and directing resources to where they're needed. 

Four factors are critical to the success of any cyber security performance management program.

• First, strong upper level management support. This is critical not only for the success of the program but for the program's implementation. 
• Second, practical cybersecurity policies and procedures. These should specify the cybersecurity management structure, identify key responsibilities and lay the foundation to reliably measure progress and compliance. 
• Third, quantifiable performance measurements. They should be designed to capture and provide meaningful performance data. Based on cybersecurity performance, goals, and objectives the performance measurement should be easily obtainable and feasible to implement. 
• Fourth, results oriented measurement analysis. This should be used to apply lessons learned, improve of effectiveness of existing security controls and plan for the implementation of future security controls to meet new cybersecurity requirements as they occur.

Managing the use of cybersecurity performance measurements or metrics requires a commitment from the cybersecurity management team. This effort will consume resources including

• people's time, 
• hardware cycles, 
• perhaps an investment in some sort of specialty software. 

The results of the effort must be periodically and consistently reviewed to make sure that they remain relevant and useful. Before beginning the process of designing, collecting and using measurements, the CISO should be prepared to answer the following questions.

• Why should the measurements you be selected be collected? 
• What's specific measurement will be collected? 
• How will this measurements be collected? 
• When will this measurements be collected? 
• Who will collect this measurements? And 
• where and what point in the functional process of the organization, will these measurements be collected? 

The cybersecurity measurement development process is divided into major activities.

• First, the identification and definition of the current program. 
• And the second the development and selection of specific measurements to gauge the implementation, effectiveness, efficiency and impact of the security controls that are in place. 

Phase one of the performance measurement development process identifies relevant stakeholders and their interest in cybersecurity measurement. The primary stakeholders are those with key cyber security responsibilities or data ownership. Secondary stakeholders such as training and human resources personnel may not be primarily responsible for cybersecurity but have a relevant task in some aspect of their job.
Phase two of the performance measurement development process is to identify and document the cybersecurity performance, goals and objectives that would guide security control implementation for the cybersecurity program of a specific information system.
Phase three focuses on organization specifics cybersecurity practices. Details of how security controls should be implemented are usually specified in organizations specific policies and procedures that define a baseline of cybersecurity practices for the information system.
In phase four, any existing measurement data repositories that could be used to derive the measurement data are are reviewed. Following that review, applicable information is extracted and used to identify appropriate implementation evidence to support measurement development and data collection.



Phases five, six, and seven involve developing measurements that track process implementation, efficiency, effectiveness and mission impact. One of the critical tasks in the measurement process is to assess and quantify what will be measured. While cybersecurity planning and organizing activities may only require time estimates, you must obtain more detailed measurements when assessing the effort spent to complete production and project tasks. This usually means that we have a form of time reporting system or paper-based or automated time accounting mechanism. Measurements collected from production statistics depend greatly on the number systems and the number of users of those systems. As the number of systems changes and as the number of users of those systems changes the effort to maintain the same level of service will vary. Some organizations track these two values to measure the service being delivered. Other organizations need a more detailed measurement.

• Perhaps, including a number of new users added. 
• The number of access control changes. 
• The number of users removed or deauthorized. 
• The number of access control violations. 
• The number of awareness briefings. 
• The number of systems by type. 
• Numbers of incidents by category. 
• Maybe the virus and worm outbreaks need to be listed. This number of malicious code instances blocked or many, many, many other possible measurements might be what we're looking for.