Enable breadcrumbs token at /includes/pageheader.html.twig

Experts Explore Mystery of Security Metrics

Cyber specialists attempt to make sense out of a modern-day Tower of Babel.

System and information security can be measured in a number of ways, including how it affects the bottom line. Credit: Shutterstock

The AFCEA Cyber Committee shares how organizations struggle to measure the strength of their systems security.

Security metrics are part of determining the amount of risk an organization is willing to take. Credit: Shutterstock

Senior executives are increasingly interested in objective measurements to determine the robustness of their organizations’ cybersecurity protections. However, measuring the adequacy of network and data security can be likened to verifying the amount of air in a room: A formula can ascertain how much air the room contains in theory, but does it take into account the leaky windows?

The AFCEA Cyber Committee examined the security metrics topic for two years, during which time it sent two surveys to association member organizations requesting input about the security metrics they use. The results were surprisingly poor and yielded no useful data, which led committee members to explore if many organizations were struggling to define appropriate measures to assess their security posture.

To try to get a better handle on what organizations were doing in the area of cyber metrics, the committee decided to reach out to a handful of organizations respected for their cybersecurity programs. From these interviews, it became apparent that the term security metrics has different meanings to different organizations.

After analyzing the interview results, it also became clear that one of the fundamental problems in identifying security metrics is the lack of a broadly accepted definition of what they comprise. In addition, even organizations with relatively mature cybersecurity programs and robust security metrics were struggling to find the right way to communicate the organizations’ state of security to their boards of directors or senior executives, committee members agreed.

The interview results also showed that organizations often have very different security metrics programs and were more a Tower of Babel than had been initially apparent. The committee found that, regardless of the taxonomy adopted, organizations were all ultimately striving to be in a position to assess the risk of accomplishing the overall mission of the organization. It also became apparent that there was a logical maturing process for organizations as they strove to define security metrics that could accurately portray an organization’s overall security posture.

To determine if security metrics guidance existed that could help these organizations, committee members reviewed several publicly available compendiums, including NIST’s Special Performance Measurement Guide for Information Security and the Center for Internet Security’s The CIS Security Metrics. They concurred that while these documents might not solve the conundrum of designing security metrics, they do offer ideas about the best way to go about improving information security.

Among the commonalities in these guides was the need for security metrics to be tied to specific objectives. For example, an organization can assess the robustness of its security wellness based on the financial risk to the company or agency if systems are compromised.

Committee members also agreed that, for better or worse, what gets measured gets attention and can improve; security metrics should inform decisions; and security metrics by themselves do not provide a good overall measure of the security of an organization at a point in time.

Several organizations interviewed described their security metrics effort as aligning their security metrics with mission risks. These organizations typically had implemented a set of technical compliance security measures and, in some cases the technical measures were quite extensive. However, senior management could not conclude from the technical metrics if their organizations’ security posture was sufficiently robust to meet the organizations’ overall objectives, or if the return on investment of additional resources in cybersecurity was appropriate.

The goal for these organizations then became to identify those security metrics that were most important to understanding and evaluating risks to the ability to perform their missions or their strategic objectives. Once defined, these risk-based metrics were tracked and regularly reported to senior management.

While the U.S. Defense Department’s eventual goal is to define a set of security metrics for measuring the risk to executing warfighting and humanitarian missions in an environment of increasing cyber attacks, for other organizations, the potential for significant financial loss, an amount that differs for each company; the inability to meet customer expectations; or the consequence of reputational damage from a major cyber incident were used to help identify the acceptable amount of risk to their organizations.

More of the AFCEA Cyber Committee's conclusions as well as recommendations for designing security metrics are available in the SIGNAL Resource Library online.