WASHINGTON \u2014 It\u2019s not Skynet, yet, but in an Air Force experiment artificial intelligence tools managed to out-perform human professionals in a key piece of planning military operations, service officials recently revealed.<\/p>\n
In the service\u2019s latest \u201cDASH\u201d experiment<\/a> this past fall, the Air Force pitted AI tools from half a dozen companies against military personnel from the US, Canada and UK and asked each to solve hypothetical \u201cbattle management\u201d problems, from standard Air Force tasks like planning an airstrike or rerouting aircraft whose home base had been damaged, to more obscure scenarios like gathering intelligence on an anomalous electromagnetic signal or protecting a disabled and drifting Navy vessel.<\/p>\n When the Air Force evaluated the resulting proposed Courses of Action (COAs), it found that at least one of the AI algorithms not only generated more COAs in less time than the humans, but it actually made fewer errors than the humans did as well.<\/p>\n \u201cThese machine-generated recommendations were up to 90 percent faster than traditional methods, with the best in machine-class solutions showing 97 percent viability and tactical validity,\u201d said Col. John Ohlund, director of the Air Force\u2019s Advanced Battle Management System<\/a> Cross-Functional Team (ABMS CFT), according to an official article<\/a> published by the Air Force on Monday about DASH-3 (Decision Advantage Sprint for Human-Machine Teaming).<\/p>\n The article said that human-generated courses of action took about 19 minutes, with 48 percent of the options \u201cbeing considered viable and tactically valid.\u201d<\/p>\n \u201cAnd our team didn\u2019t observe hallucinations during the experiment,\u201d Ohlund said.<\/p>\n That\u2019s a stark contrast both to other recent Air Force tests<\/a>, where experimental planning algorithms made subtle errors, and to broader concerns about generative AI being prone to output grossly inaccurate \u201challucinations.\u201d<\/a><\/p>\n Now, there are plenty of caveats to that human-machine comparison, the colonel cautioned in a follow-up interview with Breaking Defense. The first was sheer complexity and unfamiliarity of the scenario, deliberately designed to stress and stretch the human participants.<\/p>\n \u201cWe challenged the operators outside of their comfort zone for their normal training,\u201d Ohlund explained. \u201cWe gave them multi-domain problems to solve [in] one hour.\u201d<\/p>\n The reason for this approach? The overall project Ohlund is working on, ABMS, is meant to be a multi-service and \u201cmulti-domain\u201d command system \u2014 that is, one for all the services to use for operations involving all the officially recognized \u201cdomains\u201d of air, land, sea, space, and cyberspace<\/a>.<\/p>\n But the personnel participating in the experiment, unsurprisingly, were mostly trained and experienced in Air Force operations in the air. One relatively junior participant, for instance, was an enlisted airman with two years\u2019 training, who simply didn\u2019t have experience with electronic warfare<\/a> capabilities.<\/p>\n What\u2019s more, the participants were used to working on actual command-and-control networks with real-life performance data. But those networks and that data are classified, meaning it might take months to get novel software certified<\/a> as secure to use with them. So the experiment used unclassified approximations of the real thing. That allowed six different civilian companies to provide AI tools, but also added another layer of unfamiliarity for the military participants.<\/p>\n On the other hand, the AI tools obviously hadn\u2019t completed Joint Professional Military Education<\/a> courses or spent years working in joint operations centers, either. So the scenario and the data were unfamiliar to both the organic and the digital participants.<\/p>\n In fact, both humans and AI received the same pre-wargame briefing with \u201cprobably about 20 pages of documents to read before the experiment,\u201d Ohlund said. That ranged from the broad guidance known as \u201ccommander\u2019s intent\u201d to spreadsheets full of stats on different missiles, jammers, and so on, including their probabilities of successfully defeating a given target. (Telling a generative AI to base its outputs on a clearly defined, human-verified dataset in this way is a well-attested technique for combating hallucinations<\/a>.)<\/p>\n The difference is that the humans often got overwhelmed by all the new information and forgot or misremembered it when crunch time came. The best AIs retained absolutely everything.<\/p>\n \u201cThe computer does not forget,\u201d Ohlund said.<\/p>\n The computer also did best with a lot of human prep work. For the specific AI tool whose proposed Courses Of Action were 97 percent \u201cviable and tactically valid,\u201d the software development team had made sure their algorithm could actually digest the briefing package. Ohlund said, \u201cthat particular company had normalized all the data, had adjusted every bit of spreadsheets, had translated all the narratives.\u201d In its report, the Air Force did not identify the companies involved in the experiment.<\/p>\n The scenario was also designed to stress the humans out \u2014 a factor that didn\u2019t affect the emotionless algorithms at all.<\/p>\n \u201cWe wanted the operators to have a realistic operating environment, meaning they\u2019re not just sitting there waiting to solve a problem \u2026 . They have to be stressed,\u201d Ohlund told Breaking Defense. \u201cTime critical, you\u2019re under pressure, you\u2019ve got to make a decision right now. Formulate your plan, give me your plan, go go go.\u201d<\/p>\n In other words, human beings do best when they\u2019re doing familiar things in familiar environments, but AIs can ingest a firehose of novel information unfazed.<\/p>\n So what comes next? None of the six AI tools in the experiment is ready for operational use by planning staffs today, Ohlund said. None of this is a standalone replacement for human planners, either. Instead, he explained, they\u2019re intended to evolve into \u201cmicroservices<\/a>\u201d that plug into a larger military command-and-control system.<\/p>\n In fact, in the formalized, machine-friendly framework used by the ABMS team\u2019s AI architecture, known the Transformational Model<\/a>, generating Courses Of Action \u2014 which is what AI did in this experiment \u2014 is just one step out of 13 to develop an executable plan. Of course, more algorithmic microservices can be developed to handle those other 12 steps as well.<\/p>\n