AFCEA Home   AFCEA
Defense Foundation Europe Homeland Security Industry Intelligence SIGNAL Magazine    


     Member Login
Search
 
 
Member ID Password
Forgot: Password | Member ID             CAC/TWIC/PIV/FiXs Logon

   JOIN AFCEA NOW!
Join today to connect with the premier network of thought leaders in the global security community.

Member Benefits:
   Are You Already a Member?
  Click here to log in!

Shortcuts

NightWatch
Intelligence Newsletter
Mazz-INT Blog
Awards Program
Writing Contest
Courses
Scholarships
Intelligence Directory


EVALUATING INTELLIGENCE
Bill Nolte

At a recent event sponsored by the Bipartisan Policy Center on domestic intelligence reform, the keynote speaker was Director of National Intelligence James Clapper.  At the conclusion of his remarks, the floor was opened to the audience and the first question suggested that – the 9/11 Commission, the Intelligence Reform and Terrorism Prevention Act, and the passage of nine years notwithstanding –American intelligence remained imperfect.  That is my characterization of the question, but you get the point. (As an aside, AFCEANs who are not familiar with the Bipartisan Policy Center and its activities should take a look at its website.)

Director Clapper acknowledged that shortcomings still occur and that the Community makes an effort to learn from these, but that the process remains difficult and complex.  Later in the day, Michael Leiter, Director of the National Counter Terrorism Center, made much the same point, and noted the need for resilience on the part of the public in dealing with issues such as terrorism.

This is the same Michael Leiter of course whose resignation was being demanded in some quarters when “the underwear bomber” made his ill-fated landing in Detroit.  After that incident, I calculated for my students the number of flights that had entered the United States since September 2011.  The number, as you would guess, includes a fair number of zeros and a commensurate number of commas. The number of flights that had ended in a successful terrorist attack and civilian deaths was much smaller: zero to be exact.   But in response to the Christmas incident, many in public office determined (before any analysis, as it happened) that “the system had failed.”  So, let’s let a few heads roll.  And while we’re at it, let’s spend a few hundred million dollars to see to it that no one ever again tries the old explosives in the boxers trick. 

If we succeed, even to perfection, it will remain far simpler (and far less expensive) for terrorists to come up with some new trick.  Perhaps, taking a clue from narcotics traffickers, they can swallow the explosives and attach a receiver to one of the bags.  Or maybe they just move to a softer target, such as urban mass transit.  Any situation in which the counter-counter measures are cheaper and simpler than the counter measures is not a winning game.

My real point here is that it is time to get serious about measuring intelligence.  How good should it be? How good is it?  In the absence of such metrics, however inexact, intelligence will always be subject to the “but you’re not perfect” accusation.  This is true even though each of us lives every day with systems and structures we know not to be perfect.  We do not expect police departments to achieve a zero crime rate in any jurisdiction in the country.  We know that, actuarially speaking, every physician can expect a patient/mortality ratio of 1:1.

If anyone expects that at the end of this note I’m going to provide answers either to how good intelligence should be or how good it is, you’re reading the wrong column.  The late Peter Drucker noted many years ago that one of the factors holding back the correct evaluation of information was that we had no metrics for the value of information.  He did not have one either, but he noted, wisely as it turned out, that at the turn of the 20th century no one knew how to measure industrial productivity. 

Over time, however, we developed productivity metrics, all of which, as anyone can tell us, are imperfect.  Do we accurately count services in calculating gross domestic product?  Probably not, but we almost certainly do a better job on this than we did twenty or thirty years ago.  We know from personal experience that mileage estimates on cars are less than fully accurate, but we rely on them to provide a frame of reference, something they have proven useful in doing.  In recent years, we have done much to address Drucker’s challenge.  As we have recognized information as more and more valuable, the incentive to measure its value has grown.   As one example, University of Maryland colleagues from the Robert H. Smith School of Business have done exceptional work in evaluating security, for private and public sector purposes alike.  How much of its revenue should a small business spend to secure its information?   Certainly less than 100 per cent of its revenue, but this work is very, very important. (1) It also begins to close an important gap in allowing organizations to factor security as a cost of doing business. 

It may be that some phenomena in life or in nature cannot be measured.  More often, however, we reach a point where something we never measured needs to be measured.  Over time, as the need grows, those things tend to get measured, not by happenstance but rather by effort at designing a measure and then reaching consensus on its use.

Imperfect or not, even as a “first crack at a hard problem,” the intelligence profession needs to make an effort at answering the questions of how one evaluates intelligence.  This needs to result in a metric (or set of metrics) that Congress can endorse and that the public can understand and accept.  Perhaps as importantly, it needs to produce a form of measurement that can be examined on any routine day, not one that gets invoked only in the immediate aftermath of yet another “failure.”  Finally, it needs to measure performance in a field in which secrecy (and therefore restricted access to data and information) remains central to success.  Look at it this way: if it had been easy, it would have been accomplished decades ago.

               

William Nolte is chairman of the AFCEA Intelligence Committee and a research professor at the University of Maryland School of Public Policy



(1) See works by Lawrence Gordon, Martin Loeb, and others, on this important subject.

View and post comments here

Go Back

Connect with AFCEA Intelligence


Like us on Facebook!
Follow us on Twitter
Join Us on Linkedin
See our Flickr Photos
Videos from our events
Event Presentations


SIGNAL Magazine



SIGNAL Online
SIGNAL Digital Edition
SIGNAL Resource Library

 Latest Intelligence from
 SIGNAL Media

     Read More ...

About Us
Who We Are
Our Vision
Our Goals
What We Offer

Committees
Intelligence Committee
EPIC Committee

Events
Event List
Exhibitor Support


Resource Center
Blogs
Writing Contest
Articles
Courses
Scholarships

Awards
Overview
Recent Winners
Nominations

Corporate
Corporate Showcase

Contact
Contact AFCEA Intelligence
The Armed Forces Communications
and Electronics Association
  © AFCEA International  
Tarpit