A recent high-profile case of AI hallucination serves as a stark warning
By Jaclyn Diaz, NPR - July 10, 2025
https://www.npr.org/2025/07/10/nx-s1-5463512/ai-courts-lawyers-mypillow-fines
Trust and verify
Charlotin has found three main issues when lawyers or others use AI to file court documents: The first are the fake cases created, or hallucinated, by AI chatbots.
The second is that AI creates a fake quote from a real case.
The third is harder to spot, he said. That’s when the citation and case name are correct, but the legal argument being cited is not actually supported by the case that is sourced, Charlotin said.
:
The American Bar Association, the national representative of the legal profession, issued its first ethical guidance on the use of AI last year. The organization warned that because these tools “are subject to mistakes, lawyers’ uncritical reliance on content created by a [generative artificial intelligence] tool can result in inaccurate legal advice to clients or misleading representations to courts and third parties.”
It continued, “Therefore, a lawyer’s reliance on, or submission of, a GAI tool’s output—without an appropriate degree of independent verification or review of its output—could violate the duty to provide competent representation.”
The Advisory Committee on Evidence Rules, the group responsible for studying and recommending changes to the national rules of evidence for federal courts, has been slow to act and is still working on amendments for the use of AI for evidence.
In the meantime, Grossman has this suggestion for anyone who uses AI: “Trust nothing — verify everything.”