In another well-publicised case, an OpenAI agent undergoing a security evaluation broke out of its testing environment and breached systems at AI platform Hugging Face.
The episode heightened concerns about the AI industry’s increased use of AI agents, systems programmed to take multi-step actions without human supervision.
Anthropic did not immediately respond to a request from AFP for comment on the Philadelphia police incident.
Police said the tip, dated July 18, was flagged as spam and never reached the department’s Real-Time Crime Centre for vetting.
They added that there was no sign that police systems had been breached or department data compromised.
Anthropic discovered the incident on September 28, shut down the automated testing process responsible and added a new validation step for future tests, police said.
The company alerted the department on October 7, and the two sides met the following day.
The department said: “The two-month delay in detecting and reporting the incident to the city is unacceptable.”
Police said their safeguards had limited the impact, but that these “do not diminish the seriousness of an AI system presenting fabricated information as though it came from a person with knowledge of a homicide”.
“Unsolved cases involve real victims, grieving families and investigators working to secure answers.”
– AFP