Fake citations generated by AI, known as “hallucinations,” can rack up punishments and fines for lawyers.
John Blanchard/S.F. Chronicle
A lawsuit against Bay Area Rapid Transit by an officer who was fired after refusing to be vaccinated against COVID-19 faced possible dismissal this spring because the officer’s lawyer had fallen ill and missed numerous deadlines. When the lawyer resurfaced, she filed arguments explaining her absence and why it shouldn’t be used against her client — but three of the four cases she cited as precedent, a federal magistrate found, were nonexistent.
The fictitious cases, like many others uncovered by courts in recent years, were evidently the product of artificial intelligence, U.S. Magistrate Judge Thomas Hixson said in a ruling this month. He fined attorney Jessica Barsotti $1,000 and ordered her to take an hour of training in “the ethical use of artificial intelligence in the practice of law.”
“The Court does not prohibit or oppose the use of artificial intelligence in legal advocacy, so long as counsel independently verifies the accuracy of AI-generated content,” Hixson wrote.
Article continues below this ad
He did not identify the nonexistent cases, saying he didn’t want to imply that they were real. But one of Barsotti’s filings last month, arguing that a lawyer’s inadvertent errors or neglect shouldn’t be used to punish her client, relied on three purported rulings by federal courts between 1964 and 2003 that could not be found in official records of published decisions.
Barsotti, who missed nearly four months of law practice because of illness and other problems, could not be reached for comment. If her case filings were the product of AI misinformation, they were far from unique — a recent report cited the case as one of 957 in the United States, and almost 1,400 worldwide, in which courts have found AI-generated errors known as “hallucinations,” cases that were either misrepresented or fictitious.
San Francisco Chronicle Logo
Make us a Preferred Source to get more of our news when you search.
Add Preferred Source
The case is also one of a relatively small but increasing number in which attorneys have faced fines and other penalties.
In California, for example:
Article continues below this ad
Two years after Joan Torres and his partner Leslie Munoz broke up in 2022, Torres sued for access to their pet dog, Kyra. A family court judge in San Diego refused, relying on two cases cited by Munoz’s lawyer. The cases appeared to say courts must consider the emotional well-being of both sides in such decisions and should maintain “stability” as much as possible.
But the state’s 4th District Court of Appeal found this March that one of those cases did not exist, and the other one was a case from another year that had nothing to do with pets. The cause of the errors, the court said, was the lawyer’s use of artificial intelligence for research and the apparent failure, by both the lawyer and the judge, to verify the results.
“Courts and attorneys alike have a responsibility to protect the legal system against distortion by fabricated law, particularly in this new era of hallucinated citations generated by artificial intelligence (AI) tools,” Justice Martin Buchanan wrote in a 3-0 decision. The court declined to grant Torres immediate access to Kyra, saying he had not promptly challenged the false case filings, but imposed a $5,000 fine on his former partner’s lawyer.
In Nevada County, 57-year-old Kyle Kjoller was held without bail last year on charges of illegal gun possession. When he sought release, prosecutors filed arguments that, Kjoller’s lawyers said, cited three nonexistent cases and three others that had nothing to do with his case. District Attorney Jesse Wilson acknowledged using AI but denied relying on it for the filings; nevertheless, the California Supreme Court has ordered lower courts to consider penalties against Wilson’s office. Kjoller remains in jail.
Training courses like the one ordered for the lawyer in the San Francisco case are available at law schools and regulatory agencies, which advise participants that they are not foolproof.
Article continues below this ad
In a guidance issued to California’s attorneys in November 2023, the State Bar’s Committee on Professional Responsibility and Conduct said a lawyer “must critically review, validate, and correct both the input and the output of generative AI to ensure the content accurately reflects and supports the interest and priorities of the client.” The American Bar Association declared in 2024 that lawyers have “an ethical duty to verify citations and filings.”
But such ethical duties require enforcement, and there have been few reported cases of attorneys or law firms disciplined by regulatory agencies for failing to verify the accuracy of AI-generated legal filings.
“No one has been disbarred to my knowledge yet, but I think it’s only a matter of time,” Sean Fitzpatrick, an industry leader as CEO of LexisNexis Legal, told the Chronicle. “Our vision is that every lawyer will have their own personal AI assistant in the future, will understand their practice, do things for them they used to do manually.”
But “while it is driving efficiency,” he added, “it’s also causing significant problems.”
Or as the California Supreme Court observed in August in an advisory memo to the state’s 300,000 licensed attorneys and judges, “AI is not a lawyer. … It may give wrong answers, even if it claims to be accurate. It may give advice that does not fit your case.”
Article continues below this ad
Mariano-Florentino Cuéllar served six years as a justice on the court before leaving in 2021 to become president of the Carnegie Endowment for International Peace. One central question about AI and the law, he told the Chronicle, is, “What do we want to see as the role of humans? Who takes accountability for the decision?”
It’s a question best addressed by judges in civil lawsuits, Cuéllar said — as for “sending someone to prison, it will be a long time before humans accept having a machine do that.”
Civil cases appear to represent most of those cited in a recent worldwide study by legal researcher Damien Charlotin that found nearly 1,400 cases, including 957 in the United States, in which courts found that AI had produced hallucinated content. In more than half those cases, the report said, individuals were representing themselves, without a lawyer.
“These are just the tip of the iceberg,” and there have probably been at least 10 times that many cases with AI fabrications in the last few years, said Eugene Volokh, a retired UCLA law professor who is now a senior researcher at Stanford University’s Hoover Institution.
The primary impact, he said, is not on the reputation of lawyers or their profession — “people who are already cynical about the legal system will stay cynical” — but on the law and the public’s respect for it.
Article continues below this ad
Or as Joe Patrice, an attorney and senior editor at Above the Law, told the Chronicle, “The technology does a lot of very powerful things, but its most fierce advocates want it to do many more things that it isn’t particularly good at.” Technology can save time and money, he said, but “slow thinking is where a lot of good ideas come from.”
David Beavans, attorney for Joan Torres in the San Diego dispute over the couple’s pet dog, said he and other lawyers “have seen opposing counsel cite cases for the wrong purpose, misunderstand them innocently, or even just conclude differently than we do” in past cases that did not involve AI. “But never before have we seen citations that are entirely fictitious.”
How can a computerized system, programmed by experts to decipher court rulings and extract their principles for future cases, misstate the law or turn it upside down?
One reason is the way many rulings are written, with subtleties that machines are not equipped to detect. For example, courts often summarize one side’s legal arguments in a paragraph, then explain in the next paragraph why the author of the opinion disagrees. AI equipment has been known to interpret the first paragraph, inaccurately, as the court’s statement of the law.
That’s what may have happened last November, when San Francisco Superior Court Judge Joseph Quinn dismissed a defamation and fraud suit against multibillionaire Elon Musk by Aaron Greenspan, a former investor in Musk’s Tesla electric vehicles and operator of PlainSite, a website promoting access to legal information.
Quinn’s ruling cited a statement of the law in a 2020 state appeals court decision, but that statement was actually the opposite of the court’s conclusion in the following paragraph.
The judge later acknowledged the error that he had failed to check, but issued a revised decision that deleted the case citation and still dismissed the suit, a ruling that Greenspan is appealing. Quinn hasn’t publicly attributed the mixup to AI, but Patrice of Above the Law told the Chronicle it has all the signs of a misreading by artificial intelligence.
“I think AI has tremendous upside as long as lawyers and judges use it for the tasks it’s good at,” Patrice said. “It can format documents, it can summarize long reports, it can even produce decent first draft copy. It can’t — reliably — pick up on nuances like the one in that Elon case.”
Big names in the legal world are not immune. Last month Sullivan & Cromwell, a major Wall Street law firm, apologized to a federal bankruptcy court judge in New York for a case filing with AI-generated errors that were discovered by another prominent firm, Boies Schiller Flexner.
But AI saves law firms time and money in research that can be essential to their work, and its use has been steadily increasing in the legal community since 2022, when technology companies unveiled research tools such as OpenAI’s ChatGPT chatbot.
Los Angeles attorney Lauren Teukolsky, who has represented employees in workplace cases for 26 years, said she recently used an AI tool, Deep Research, to examine the law for a client, whose employer sought to fire him because he asked to be transferred to the day shift in order to spend nights with his ailing wife. The device quickly found the cases she needed, Teukolsky said, and she was able to verify its findings in 10 to 15 minutes, leading to a settlement of the case.
“An attorney would take at least a week” to complete that task, Teukolsky told the Chronicle. She said the results were “as good as a very competent attorney would produce.”
Or, said Volokh, the former UCLA law professor, suppose a lawyer’s client was a husband or wife of modest means who was getting divorced and needed to divide the couple’s property equally. “In court, attorney fees could eat up the value of the property,” he said, but the lawyer could accomplish the task affordably by using AI and double-checking the results.
Last year, a worldwide survey by the International Legal Technology Association of 560 law firms, with a total of 157,000 lawyers, found that 80% of the firms — and all 57 that had 700 or more attorneys — said they were using or exploring the use of AI technology.
The survey said 69% of legal professionals were using AI tools like ChatGPT in their work, more than double the 31% reported in 2024.
Oversight agencies say the tools can be used safely, and efficiently, as long as someone carefully reviews their findings to make sure they accurately reflect the cases and legal principles they cite.
Former San Francisco District Attorney Chesa Boudin, who now heads the Criminal Law & Justice Center at UC Berkeley, filed arguments supporting Kjoller, the Nevada County gun posession defendant, and thinks AI can be beneficial — but only if regulated more strictly than it is now.
The technology constitutes “a tremendous opportunity as well as risk for the legal profession,” Boudin told the Chronicle. “It depends on regulation, transparency, human oversight, things that right now unfortunately are largely missing.”
And regulation, he added, should be aimed at lawyers and judges who cite hallucinated cases, not the AI companies that produce them. “It is the responsibility of the lawyer/judge authoring the brief/order to get it right,” Boudin said by email.
Another drawback is AI’s potential impact on the lower-paid legal staff who help attorneys prepare cases for court — first- and second-year lawyers, law students and recent graduates hired as law clerks, and legally trained assistants classified as paralegals. Some law firms have decided that AI can perform those tasks adequately at far less expense.
“It will cost jobs, paralegals, clerks,” said Teukolsky, the Los Angeles employment lawyer.
“Loss of jobs: That’s the fear,” said Natalie Knowlton, associate director for legal innovation at the Stanford Center on the Legal Profession. “Some firms say they can do more with fewer associates,” but others have maintained or even increased their staffs to take on more cases, she said.
And Fitzpatrick of LexisNexis Legal acknowledged that “if the amount of legal work does not change and can be done more efficiently, we’ll need less people.”
Another potential problem, as illustrated by the confusion over the law in the Musk case, is that computers that quickly and efficiently provide logical answers to legal questions can overlook more subtle issues, like questions of legal ethics — conflicts of interest, confidentiality and the duties of lawyers to their clients.
Jennifer Ellis, a Pennsylvania attorney who specializes in ethical issues, says she posed an ethical question to an AI device and “it just made up” the answer.
“Human beings respond to confident writing by believing it,” Ellis said. “The more confident AI sounds, the more suspicious you should be.”
It might also depend on what questions are posed to the electronic device, and whether they are based on actual facts. In a 2024 report, “AI on Trial,” Stanford researchers said they asked an AI tool provided by Practical Law, owned by Thomson Reuters, “Why did Justice (Ruth Bader) Ginsburg dissent in Obergefell?” without mentioning that this was the 2015 case in which the Supreme Court declared a constitutional right to same-sex marriage.
The machine replied that Ginsburg dissented “because she believed that the court’s interpretation undermined Congress’s intent under Section 601 (a) (1) to protect copyright owners against unauthorized importation of low-priced, foreign-made copies.”
The case actually had nothing to do with copyrights, and the late Ginsburg did not dissent but was part of the court’s 5-4 majority.