Security<\/p>\n
You\u2019ll need a lot of detailed prompts to get solid output – and even then it may have errors and typos<\/p>\n
Cisco tested AI\u2019s ability to write an accurate report on a tabletop security incident response exercise, and found that while the tech can save time, many risks remain.<\/p>\n
The networking giant revealed its results in a Thursday blog post https:\/\/blogs.cisco.com\/security\/ai-generated-reporting-lessons-learned-from-talos-incident-response by Nate Pors, a senior incident commander in the Cisco Talos Incident Response team.<\/p>\n
Pors opened by observing that when to used generate long-form technical content, large language models can deliver \u201csignificant inaccuracies, unusual conclusions, and inconsistent writing styles.\u201d<\/p>\n
LLMs make those mistakes because they\u2019re essentially a fancy autocomplete system that makes educated guesses. Pors wrote that the nature of LLMs therefore sees them mess up in four ways:<\/p>\n
One involves giving an LLM \u201cgranular, single-task instructions\u201d that focus on \u201ca specific, small portion of the report.\u201d Doing so means \u201crisk of hallucination or cross-contamination between sections is significantly reduced.\u201d Telling an LLM which sources to use also helps. So does setting rules about the style and format of output.<\/p>\n
Using those techniques, Cisco says the time required to draft an incident report based on a tabletop exercise fell by 50 percent.<\/p>\n
“A blind test of the sample report in our quality assurance process showed no noticeable drop in overall writing quality,” Pors wrote. “The peer reviewer, professional editor, and management reviewer all made complimentary comments about the report while unaware that it was AI-generated. The peer reviewer commented that the incidence of typos and grammatical errors was far lower than in the average report.”<\/p>\n
But the Talos team also found \u201cediting multiple sample reports within a single session resulted in cross-contamination of content from one report\u2019s source material to another, even if the notes used to generate the first report were deleted from the project\u2019s reference documents.\u201d<\/p>\n
The researchers therefore recommend starting a new session, and re-entering prompts, for each new incident report.<\/p>\n
They also developed a spelling-and-grammar-checking prompt that \u201challucinated numerous grammar issues \u2026 failed to identify actual issues,\u201d had a success rate below 50 percent and \u201cwould behave inconsistently, sometimes catching issues and sometimes overlooking them.<\/p>\n
\u201cIt is currently unsuitable for production use,\u201d Pors concluded.<\/p>\n
Pors said Cisco concluded that its approach \u201ccould be adapted to any cybersecurity reporting use case with standardized inputs and predictable outputs,” but also warned authors must “take ownership of every word of the final report.”<\/p>\n
“While testing, we found that the LLMs generated recommendations that were duplicative, irrelevant, or not actionable.\u00a0If this were used in a production environment without manual checks, it could result in\u00a0poor-quality\u00a0recommendations in a final report.”<\/p>\n
Those problems arose when considering a tabletop exercise, a far simpler affair than analysis of an incident that involves analyzing log files from multiple systems. \u00ae<\/p>\n","protected":false},"excerpt":{"rendered":"Security You\u2019ll need a lot of detailed prompts to get solid output – and even then it may…\n","protected":false},"author":2,"featured_media":597689,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[554,733,4308,86,56,54,55],"class_list":["post-597688","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-technology","tag-uk","tag-united-kingdom","tag-unitedkingdom"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/597688","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/comments?post=597688"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/posts\/597688\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media\/597689"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/media?parent=597688"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/categories?post=597688"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/uk\/wp-json\/wp\/v2\/tags?post=597688"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}