Agencies Review Effect Of Government Oversight
Survey reveals that standards have made a difference, but officials want changes. 
A report titled Is FISMA Making the Grade? demonstrates that large and small agencies believe their information technology security has improved since the Federal Information Security Management Act of 2002 (FISMA) was instated. However, large-agency chief information security officers (CISOs) believe the FISMA report cards reflect their security more accurately than do officials from small organizations.
A new report grades the graders as computer security officers share how federal guidance and reporting are affecting their organizations. Many believe the regulations have improved their organizations’ safety measures, but there is a disparity of opinion about effectiveness, and some contend that funding should be tied to the process.
In a survey commissioned by the Merlin International Federal Research Consortium (MFRC) and conducted by the MFRC and O’Keeffe and Company Incorporated, chief information security officers (CISOs) gave their opinions of the Federal Information Security Management Act of 2002 (FISMA) and the effect it has had on their agencies. The act establishes stronger lines of management accountability for information security and allows the legislative branch substantial oversight. FISMA mandates that organizations report to the Office of Management and Budget (OMB) and to Congress on their compliance with the act’s requirements. Agencies in the federal government have been graded annually since 2003 on their compliance with FISMA regulations, receiving a Federal Computer Security Report Card.
MFRC’s objective for the Is FISMA Making the Grade? report was to learn federal CISOs’ perspectives on the FISMA report cards and to assess the value of the grading. The survey also sought to determine whether large and small organizations had the same experiences and opinions. O’Keeffe personnel interviewed 30 CISOs out of a total of 117.
CISOs shared that their report cards are improving, with 75 percent reporting higher overall grades in 2007 than in previous years. Only 4 percent reported lower grades, and 21 percent related that they did not know if theirs had improved or worsened. Seventy-five percent of respondents said that security had improved or had significantly improved since last year, with small organizations—those with fewer than 10,000 employees—especially noting this change. No CISO said security had worsened, but 21 percent responded that security had not changed, and 4 percent said they did not know whether it had changed.
Adjustments to certification and accreditation (C&A) efforts led the way to the higher rankings. Two adjustments, C&A streamlining processes and devoting resources to C&A, combined were cited by 36 percent of respondents as factors increasing FISMA grades. Establishing enforceable internal information technology security policies to reflect FISMA compliance goals was mentioned by 16 percent of those surveyed as a reason for improved grades.
However, large and small agencies felt differently about which aspect of C&A improved their scores. Streamlining C&A processes was mentioned by 26 percent of large-agency CISOs, the highest percentage for any response in that category. Only 16 percent of small-agency CISOs cited the same option. The highest percentage of small-agency respondents—19 percent—credited devoting resources to C&A as the reason for their score increases. In large agencies, 11 percent gave that response.
Despite the generally positive opinions about security improvements and regulation compliance, some dissatisfaction and discrepancies come to light in the responses, and respondents believe the FISMA process could be improved. CISOs still face challenges, including understanding the language used in guidelines. Ambiguity in FISMA language requirements was mentioned by 16 percent of the CISOs at large agencies and 16 percent at small agencies as a reason for lower scores, and 46 percent of all respondents recommended clarifying FISMA language guidelines as a way to improve FISMA’s value. Forty-two percent of the CISOs also suggested providing better guidance for the yearly agency security control tests.
Mark Zalubas, chief technology officer of Merlin International, explains that ambiguity exists in the language delineating the differences between compliance and noncompliance. “There’s some level of interpretation that you have to apply to what you’re being asked to do,” he says. As a result, two agencies may have systems with the same level of protection, but one perceives itself as compliant and the other does not. “It’s hard to compare apples to apples,” Zalubas asserts.
CISOs also were split as to whether the report cards provide real insight into an organization’s information security. Although 54 percent of those interviewed said the grades did reflect their security level, the percentage of respondents who chose that option was higher among CISOs in large agencies than among those in small ones. Seventy-eight percent of small-agency CISOs reported that security in their organizations had improved or significantly improved since the last FISMA report card compared to 70 percent of large-agency CISOs. However, only 36 percent of small agencies said FISMA reporting provided genuine insight into their information technology security versus 60 percent of large agencies. “I think the fact that large agencies get more out of FISMA than the small agencies was a bit of a surprise,” Zalubas shares.
He also notes that the higher grades might not guarantee more security because agencies improved what they would be graded on. In other words, they “teach to the test.” While the survey results show that CISOs do believe their organizations are more secure since FISMA, no questions determined the ratio between more security and higher FISMA grades.
CISOs brought up the lack of relationship between grades and funding. Report card scores have little bearing on either agency information technology security funding or overall agency information technology funding. Three-quarters of the survey takers said they had seen no inverse relationship between their FISMA grades and their information technology security funding, and 79 percent had found no link between their FISMA grades and their overall information technology budget.
Zalubas says CISOs believe funding should be tied to the score in some way. The CISOs were not pressed for ideas on how the funding should be related to FISMA, but Zalubas offers two possible options.
One school of thought says that agencies that score well should be rewarded. However, these agencies may not need the money because their security already is good. The other camp asserts that those who receive poor grades should receive the money to make improvements. But those agencies could have other problems besides funding, and some organizations might seek a low score to obtain additional funding.
Regardless of the scores’ connection to funding, CISOs across the government are working to improve their FISMA grades. “Once you find out where your holes are, you want to plug them,” Zalubas explains. A key trend emerged during the survey interviews as 83 percent of CISOs stated they plan to increase information technology audit trails and authorization efforts over the next year.
At the end of Is FISMA Making the Grade?, three recommendations are offered to the federal government to make the report cards more valuable. The first suggestion advises clarifying definitions in FISMA language; the second, establishing separate rules for small and large agencies; and the third, creating a better link between FISMA compliance and business implications.
An official at the OMB commented on the report, saying, “We appreciate the attention given to the FISMA performance measures and implementation of FISMA within the agencies. Since the passage of FISMA, agencies have improved information security. FISMA provides an effective framework to prioritize information security by basing the requirements on risk. By implementing a risk-based framework for information protection, we are able to have a process that works for both large and small agencies. In our future activities we will be working with both small and large agencies to further improve and strengthen FISMA implementation.”
Zalubas believes one of this report’s greatest benefits is showing that grades in general are improving, and CISOs believe that FISMA compliance equates to better protection of information technology assets. By reporting grades over time, officials have a better idea of the trends than if they looked at a single grade in a vacuum. “In general, there’s a wave moving in a positive direction,” Zalubas says.
He states that the two audiences that benefit from reading this report are federal agencies and organizations that sell to federal agencies. Government employees can learn what their peers think of the process, and sellers can determine whether their offerings hit the market sweet spot or whether they should adjust their business plan.
Timothy M. Clark, director of business development at F5 Networks Incorporated, one of the members of the MFRC, explains that he received a good general education on FISMA from the MFRC survey and that he was surprised that more funding is not linked to the report cards. He shares that his company’s interest in the findings is centered on how to serve the federal client better. “I think some of the findings really focus the efforts in some of our functionalities and products,” he states.
Adam Vincent, senior federal solutions architect at Layer 7 Technologies Incorporated, another member of the MFRC, agrees that the survey is helpful for industry, especially for companies determining what services to offer to government programs.
Both Clark and Vincent believe the breakdown of large and small agency responses is important and beneficial.
The men also were pleased to see grades improving across the board, although Vincent states that he was surprised at how poor security used to be.
Officials at O’Keeffe and the MFRC provided the report to public and private parties they believed would have interest in the findings. The report also is available online at Merlin International’s Web site. Some elected officials and leaders at certain government agencies were alerted to the survey’s results before its public release.
Web Resources
Is FISMA Making the Grade?: www.merlin-intl.com/IAStudy.asp
Merlin International: www.merlin-intl.com
O’Keeffe and Company: www.okeeffeco.com
FISMA Implementation Project: http://csrc.nist.gov/sec-cert
