Industry Off and Running in ODNI Xpress Challenge

More than 245 companies, organizations and academic institutions are vying to develop machine analytics tools for the intelligence community in an open competition. The starting gun for this challenge went off in April, and a total prize purse of $500,000 awaits, including $270,000 for the top five performing teams. Another $230,000 will be awarded in additional categories.

This challenge is part of the Office of the Director of National Intelligence (ODNI) In-Vest program, which seeks to push the limits of machine-generated finished intelligence. Known as Xpress, the ongoing challenge focuses on developing machine-generated analysis products that would speed delivery of vital information while also improving the quality and homogeneity of traditional human-generated analytic products.

David Isaacson, program manager at ODNI, relates that the development of these intelligence analysis products, including the review process, currently is time consuming. The ODNI is looking to artificial intelligence to reduce that process time to minutes. Staffers also would be able to provide more sophisticated feedback, which would be another factor in improving the timeliness of intelligence products.

The competitors range from large multinational companies to even high school teams. Isaacson notes that the challenge is offering a total of $30,000—added to the prize purse by the Office of the Undersecretary of Defense for Intelligence—to the top three high school submissions. The competitors come from 24 countries across six continents, and Xpress will continue to accept registrants up to its July 5 closing date.

Full disclosure: SIGNAL Magazine is playing a role in the challenge, albeit a passive one. InnoCentive, the contractor hosting the Xpress competition, asked SIGNAL for permission to use the magazine’s digital content as source material for the organizations developing the new technologies. This SIGNAL data is serving as a level playing field for the contest, as competitors will be demonstrating their technologies using the same wellspring of information. ODNI planners extracted a sample of material from among the 15,000 SIGNAL articles, columns and blog entries going back several years, and they assembled it into the format they prefer for competitors to submit their entries. This also gives participants an idea of what the ODNI is looking for in a successful product.

“There is a lot of high-quality wheat, but also a lot of high-quality chaff to sort through,” Isaacson says of the data, which in some cases must be distinguished from similar topics that overlap.

The choice to open up the competition to the entire Internet posed a host of problems, Isaacson adds. These include copyright issues, lack of control over source material, code validation and irrelevant websites. Limiting the competition to specific source material that comprised relevant issues eliminated those problems, he points out.

The supplemental $230,000 will be apportioned for awards such as literal, inferential and evaluative categories. Each of those three areas represent a consolidation of intelligence analysis criteria. Evaluations will be done in-house by analytic integrity standard experts within the ODNI, Isaacson says.

For the literal category, the ODNI is seeking the ability to craft sound written material in response to the posed intelligence question. The inferential category looks for the ability to discern and categorize how the offered reasons support analytic judgments and conclusions. The evaluative category examines the ability to make reasoned assertions and incorporate alternative analyses, Isaacson explains.

This tiered approach to scoring aims at increasing sophistication in the products, he says. Other prize money will go to the submission with the most creative approach to conveying information. The contestants know all the criteria for judging in a fully transparent process, Isaacson notes.

The Xpress challenge has helped the ODNI clarify what its next steps will be in machine analytics, he continues. For example, the intelligence community’s analytic toolset lacks an objective means for quality control during product development. This capability will be essential if the competitions generate machine-generated intelligence products that would be incorporated into the community’s kit.

The next challenge, Xtend, will explore opportunities for machine evaluation of intelligence products. It will examine automated methodologies that complement the automated intelligence production capabilities demonstrated in Xpress. Xtend will be launched late summer to avoid overlapping with Xpress, and it will run for 60 days, he says.

Isaacson explains that planners are modulating these challenges to avoid overtaxing participants. Where Xpress is product-intensive, Xtend is more of a theoretical challenge—almost a white paper, he says. It represents a natural evolution of the capabilities explored in Xpress.

The ODNI also has launched a new effort, the Intelligence Formulation Of Risk Management (In-FoRM) program to explore more basic research opportunities in machine analytics. It creates unclassified symposia and workshops for select topics.

After the July 5 closing date, Xpress enters the evaluation phrase. Isaacson cannot predict how many of the 245 registrants actually will submit entries. If more than 60 submissions come in, ODNI and Defense Department senior editors for intelligence products will help down-select the entries to a maximum of 60. These more promising entries will be turned over to the ODNI’s analytic integrity and standards experts for full evaluation. The goal is to finish this scoring by early September, he explains.

The organizations with products that score the highest and appear most promising will asked to submit their source codes so judges can validate the submissions. This will ensure the submission is genuine, Isaacson allows, emphasizing that the ODNI is not obtaining the source code for any purpose other than verification.

Organizations wishing to register for the ongoing competition or learn more about it can visit the website.