In July last year, Louis Flores typed a question into the AI chatbot Grok: \u201cWhat kind of case law can help us in a lawsuit to stop the privatization of NYCHA?\u201d<\/p>\n
Flores, a long-time community organizer, lives a block from the Fulton & Elliott-Chelsea Houses, an expansive public housing development in Manhattan that\u2019s run by the New York City Housing Authority. For the last six years, he and a group of other activists have been fighting a legal battle, largely without an attorney, to stop a plan to demolish the existing buildings and build new ones.<\/p>\n
Within seconds, Grok gave him a detailed list of cases that he could cite as precedent. Flores packaged them into a 42-page legal brief that he submitted to the court, arguing why the city should postpone the demolition of the buildings until the project goes through a more rigorous public review process.<\/p>\n
\u201cI felt confident in the case overall, but we had no help,\u201d he told Gothamist. \u201cI had no idea how it would go.\u201d<\/p>\n
It did not go well.<\/p>\n
About a week later, NYCHA\u2019s attorney sent a letter to Judge James D\u2019Auguste, pointing out that four of the cases that Flores had cited either don\u2019t exist or don\u2019t say what Flores claimed they did.<\/p>\n
Grok had hallucinated them \u2014 a fancy, AI-speak way of saying that the chatbot had made them up. Once the judge learned of what had happened, he didn\u2019t just reprimand Flores and the other plaintiffs or ask them to refile the brief. He threw out the entire lawsuit.<\/p>\n
\u201cWe worked so hard for six years to get to the point where a judge would look at our case,\u201d Flores said, adding that he and his fellow organizers are in grief over the judge\u2019s decision. \u201cI felt like we were denied justice.\u201d<\/p>\n
It\u2019s the most extreme example so far among dozens of legal cases in New York that have already tested the limits of how artificial intelligence is being used in the courts, and the willingness of judges to accept legal filings that haven\u2019t been produced by humans.<\/p>\n
\u201cWe\u2019re getting blamed for being poor and not having attorneys,\u201d Flores said.<\/p>\n
The state\u2019s court system acknowledges that AI\u2019s use in legal proceedings has become \u201cincreasingly common\u201d and that \u201cjudges are seeking guidance\u201d on how to deal with it in their courtrooms, according to an October memorandum issued by an advisory committee established in 2024 to ensure the technology is being used responsibly. Committee members have concluded that AI chatbots don\u2019t require a \u201cnovel rule\u201d because their hallucinations are just an extension of an old problem: Litigants and lawyers have always made mistakes in their filings.<\/p>\n
Al Baker, a court spokesperson, said the Administrative Board of the Courts is currently reviewing a proposed policy<\/a> that updates the current rule requiring all signed filings, including those written with the help of AI, to contain only accurate information. The proposed policy does not go so far as banning the technology or forcing litigants and lawyers to disclose their use of AI when they submit court documents. Errors created by AI would simply be subject to the same fines and penalties as any other inaccurate information in case filings, according to Baker.<\/p>\n \u201cThe purpose of this policy statement is to promote uniformity and consistency and avoid a hodgepodge of conflicting part rules,\u201d the advisory committee wrote in its memo last year.<\/p>\n But the guidelines don\u2019t say anything about what penalties judges should hand down for false or fabricated information generated by AI. And at the federal level, which is not subject to the state courts\u2019 oversight, at least one judge has created his own rules around the use of chatbots.<\/p>\n Bruce Green, a law professor at Fordham University and an expert on legal ethics, said he expects that both sides will eventually adapt: the courts will create more sophisticated rules around the technology, and new AI tools will be better at producing more accurate citations. But in the current, largely unregulated environment, he said it\u2019s worth requiring transparency from litigants.<\/p>\n \u201cIt\u2019s fair to ask them whether they\u2019ve used AI,\u201d Green said. \u201cBut for the moment, I don\u2019t think judges should forbid anybody, unrepresented people or lawyers, from using AI tools.\u201d<\/p>\n A free tool<\/p>\n The rapid advancement of AI has provided a valuable tool for those who typically have less access to the legal system.<\/p>\n \u201cMuch of the population cannot afford a lawyer at all,\u201d said Stephen Gillers, a professor emeritus and expert on legal ethics at NYU Law School. \u201cIt\u2019s getting worse: Legal fees are going up. People\u2019s ability to pay them is going down.\u201d<\/p>\n Millions of people<\/a> across the United States file lawsuits in state courts without an attorney, according to a 2015 study on low-income litigants. In U.S. federal courts, national data<\/a> shows that the lawyer-less make up about a quarter of cases each year.<\/p>\n Green said AI \u201cholds a lot of promise for people who can\u2019t afford a lawyer.\u201d But he said its unfettered use could bog down the legal system if the courts can\u2019t adapt quickly and develop a clear set of rules around the technology.<\/p>\n Judges now have to be more vigilant about identifying fictitious case law, and faulty legal documents can lead to delays.<\/p>\n \u201cThere\u2019s a risk that people will be using it in ways that are burdening the court,\u201d Green said.<\/p>\n Green said it\u2019s unreasonable to expect courts and judges who are already overwhelmed to verify every citation in every document. But he said he sympathizes with those who are using AI to advance their cases because they can\u2019t afford attorneys, and that litigants without lawyers have always been more likely to submit incorrect information because of the very fact that they lack legal representation.<\/p>\n The broad availability of AI chatbots may now be amplifying that tendency.<\/p>\n \u201cThey\u2019re trying to do the best they can,\u201d Green said. \u201cIt\u2019s a lot to expect that they\u2019re going to file legal briefs at the same quality as what lawyers would file.\u201d<\/p>\n Flores said he used Grok exclusively for research, wrote the filings himself and was embarrassed by the errors he submitted to the court. But he never expected them to be fatal to his case. The judge, he said, had overreacted in his decision<\/p>\n \u201cHe wasn\u2019t looking at the case on its merits,\u201d Flores said.<\/p>\n His lawsuit aside, those without attorneys typically fare better than lawyers when AI makes mistakes.<\/p>\n Because there are no uniform rules governing AI use in court, there are no official tallies of how often AI tools have been used in legal filings or how often judges have disciplined litigants and lawyers for AI-produced errors. But Damien Charlotin, a researcher at the French business school HEC Paris, has assembled a database<\/a> of more than 900 legal decisions across 31 different countries where judges have sometimes sanctioned plaintiffs with financial and legal penalties for filing erroneous, AI-generated legal documents.<\/p>\n In roughly 40% of those cases, lawyers, paralegals or other legal professionals were responsible for including the erroneous AI-generated information in the filings. Lawyers included the hallucinated errors in 24 out of the 54 New York City cases. Some received fines of up to $10,000 or were referred to the bar association for possible disciplinary action, while others got off with just a warning.<\/p>\n Green said lawyers have \u201cno excuse\u201d for filing faulty information to the courts, but that unrepresented litigants deserve more leniency.<\/p>\n AI v. Nicki Minaj<\/p>\n Tameer Peak turned to ChatGPT and Gemini when he sued the rapper Nicki Minaj for defamation in 2024. Peak, a once devout fan of Minaj, accused the rapper of making comments during a social media livestream implying that he was mentally unstable.<\/p>\n Peak said he thought his case was straightforward. But after lawyers quoted him five-figure fees to bring a lawsuit on his behalf, he decided to file it himself \u2014 with a little help.<\/p>\n \u201cThese tools provide access to people who would not have access because of a retainer fee or just simply a lawyer not believing in their case,\u201d Peak said. \u201cAI has allowed me to understand the legal system a bit more, like certain jargon or procedural steps.\u201d<\/p>\n Peak said he used ChatGPT and Gemini to format his complaint and a variety of other motions. He also used it to find previous cases that could support his own.<\/p>\n Unlike in Flores\u2019 petition, those cases did exist. But one letter Peak wrote to the judge asking the court to move the case forward included quotes from past lawsuits that were fabricated. In Judge Vernon Broderick\u2019s response, he said he was \u201cconcerned\u201d about Peak\u2019s use of artificial intelligence \u201cduring the course of his litigation.\u201d<\/p>\n Broderick, however, did not throw out Peak\u2019s lawsuit. He instead issued a warning and wrote in his order that he was \u201csympathetic\u201d to Peak\u2019s status as an unrepresented plaintiff.<\/p>\n \u201cI think that the judge was as fair as he could be,\u201d Peak said, adding that the judge saw it as a relatively minor issue. \u201cThey want to decide a case on merit.\u201d<\/p>\n