Search:  

 Blog     e-Newsletter       Resource Library      Directories      Webinars     Apps
AFCEA logo
 

EVALUATING INTELLIGENCE

Tuesday, October 19, 2010
Bill Nolte

At a recent event sponsored by the Bipartisan Policy Center on domestic intelligence reform, the keynote speaker was Director of National Intelligence James Clapper.  At the conclusion of his remarks, the floor was opened to the audience and the first question suggested that – the 9/11 Commission, the Intelligence Reform and Terrorism Prevention Act, and the passage of nine years notwithstanding –American intelligence remained imperfect.  That is my characterization of the question, but you get the point. (As an aside, AFCEANs who are not familiar with the Bipartisan Policy Center and its activities should take a look at its website.)

Director Clapper acknowledged that shortcomings still occur and that the Community makes an effort to learn from these, but that the process remains difficult and complex.  Later in the day, Michael Leiter, Director of the National Counter Terrorism Center, made much the same point, and noted the need for resilience on the part of the public in dealing with issues such as terrorism.

This is the same Michael Leiter of course whose resignation was being demanded in some quarters when “the underwear bomber” made his ill-fated landing in Detroit.  After that incident, I calculated for my students the number of flights that had entered the United States since September 2011.  The number, as you would guess, includes a fair number of zeros and a commensurate number of commas. The number of flights that had ended in a successful terrorist attack and civilian deaths was much smaller: zero to be exact.   But in response to the Christmas incident, many in public office determined (before any analysis, as it happened) that “the system had failed.”  So, let’s let a few heads roll.  And while we’re at it, let’s spend a few hundred million dollars to see to it that no one ever again tries the old explosives in the boxers trick. 

If we succeed, even to perfection, it will remain far simpler (and far less expensive) for terrorists to come up with some new trick.  Perhaps, taking a clue from narcotics traffickers, they can swallow the explosives and attach a receiver to one of the bags.  Or maybe they just move to a softer target, such as urban mass transit.  Any situation in which the counter-counter measures are cheaper and simpler than the counter measures is not a winning game.

My real point here is that it is time to get serious about measuring intelligence.  How good should it be? How good is it?  In the absence of such metrics, however inexact, intelligence will always be subject to the “but you’re not perfect” accusation.  This is true even though each of us lives every day with systems and structures we know not to be perfect.  We do not expect police departments to achieve a zero crime rate in any jurisdiction in the country.  We know that, actuarially speaking, every physician can expect a patient/mortality ratio of 1:1.

If anyone expects that at the end of this note I’m going to provide answers either to how good intelligence should be or how good it is, you’re reading the wrong column.  The late Peter Drucker noted many years ago that one of the factors holding back the correct evaluation of information was that we had no metrics for the value of information.  He did not have one either, but he noted, wisely as it turned out, that at the turn of the 20th century no one knew how to measure industrial productivity. 

Over time, however, we developed productivity metrics, all of which, as anyone can tell us, are imperfect.  Do we accurately count services in calculating gross domestic product?  Probably not, but we almost certainly do a better job on this than we did twenty or thirty years ago.  We know from personal experience that mileage estimates on cars are less than fully accurate, but we rely on them to provide a frame of reference, something they have proven useful in doing.  In recent years, we have done much to address Drucker’s challenge.  As we have recognized information as more and more valuable, the incentive to measure its value has grown.   As one example, University of Maryland colleagues from the Robert H. Smith School of Business have done exceptional work in evaluating security, for private and public sector purposes alike.  How much of its revenue should a small business spend to secure its information?   Certainly less than 100 per cent of its revenue, but this work is very, very important. (1) It also begins to close an important gap in allowing organizations to factor security as a cost of doing business. 

It may be that some phenomena in life or in nature cannot be measured.  More often, however, we reach a point where something we never measured needs to be measured.  Over time, as the need grows, those things tend to get measured, not by happenstance but rather by effort at designing a measure and then reaching consensus on its use.

Imperfect or not, even as a “first crack at a hard problem,” the intelligence profession needs to make an effort at answering the questions of how one evaluates intelligence.  This needs to result in a metric (or set of metrics) that Congress can endorse and that the public can understand and accept.  Perhaps as importantly, it needs to produce a form of measurement that can be examined on any routine day, not one that gets invoked only in the immediate aftermath of yet another “failure.”  Finally, it needs to measure performance in a field in which secrecy (and therefore restricted access to data and information) remains central to success.  Look at it this way: if it had been easy, it would have been accomplished decades ago.

               

William Nolte is chairman of the AFCEA Intelligence Committee and a research professor at the University of Maryland School of Public Policy



(1) See works by Lawrence Gordon, Martin Loeb, and others, on this important subject.

Comments

I recently heard Mike Hayden (former CIA Director) speak, and he told of a question he once got asking him, on a scale of one to ten, to rate how well the Community was doing since 9-11. His response was telling ... something like "eight to ten are not even on our scale." Perfect intelligence is impossible. Even our adversaries can't accurately predict! There are just too many variables and dependencies in a very human endeavor. All we can do is strive for perfection. That's all God asks! It ought to be good enough for politics too! As an old friend was wont to say: There are only "Intelligence Failures" and "Operational Successes." It will ever be the case!

By John Casciano