Major law firm Cozen O’Connor is implementing AI hallucination detection software after two of its lawyers were sanctioned for filing legal documents containing fake case citations generated by ChatGPT. The firm has adopted technology from startup Clearbrief that scans legal briefs for fabricated facts and citations—functioning like spell-check but for AI-generated hallucinations.
The incident that prompted this move occurred in September when two Cozen O’Connor defense lawyers admitted to filing a document riddled with fake cases after one used ChatGPT to draft it, violating firm policy. A Nevada district court judge imposed sanctions, requiring each lawyer to pay $2,500 and write to their former law school deans explaining the fiasco. Cozen O’Connor subsequently fired the lawyer who used ChatGPT.
The problem is accelerating across the legal profession. Legal data analyst Damien Charlotin has tracked 660 cases involving hallucinated content in legal filings between April 2023 and December 2024, with the rate reaching four to five new cases per day. While most involve self-represented litigants or small firms, large firms aren’t immune—hallucinations often slip through via junior staff, paralegals, or consultants.
AI hallucinations stem from how large language models work: they predict the most likely next word rather than verifying factual accuracy. Companies like Thomson Reuters and LexisNexis are positioning their legal research services as safer alternatives by constraining AI models to vetted databases of case law rather than the open internet. Legal tech startup Harvey, valued at $8 billion, has partnered with LexisNexis to pipe verified legal databases into its generative tools.
Clearbrief’s solution works as a Microsoft Word plug-in that detects citations using natural language processing, creates links to relevant case law, and flags fabricated or unsupported claims. The new cite-check feature, launched Friday, generates reports before documents reach partners or courts. For Cozen O’Connor, this creates a paper trail showing who ran checks and when—crucial documentation if judges question citation accuracy.
The issue extends beyond law: consulting firm Deloitte agreed to partially refund $290,000 to the Australian government after a report contained allegedly AI-generated errors. Experts acknowledge hallucinations will persist, requiring lawyers to treat chatbot output as starting points requiring verification.
Key Quotes
You have to be pragmatic. Lawyers will play around with chatbots whether the tools are authorized or not.
Kristina Bakardjiev, Cozen O’Connor partner responsible for technology implementation, acknowledges the reality that policies alone won’t prevent AI use. This pragmatic stance explains why firms are shifting from prohibition to risk mitigation strategies.
AI hallucinations are hard to eliminate because they’re baked into the way chatbots work. Large language models are trained to predict the word that is most likely to come next, given the words before it.
This explanation from the article clarifies why hallucinations aren’t simply bugs to be fixed—they’re fundamental to how LLMs function, making them an inherent risk that requires management rather than elimination.
The model makers can’t get hallucinations to zero for answering open-ended questions about the world. However, companies can dramatically reduce their risk by forcing a large language model to cite from a specific data set.
Michael Dahn, senior vice president at Thomson Reuters, articulates the industry’s solution: constraining AI models to verified datasets rather than the open internet, which forms the basis for specialized legal AI tools.
Our Take
The legal profession’s AI hallucination crisis reveals a broader truth about generative AI adoption: general-purpose tools aren’t suitable for specialized professional work without significant guardrails. The fact that Cozen O’Connor fired an employee yet still implements detection tools shows firms recognize prohibition is futile—the productivity allure is too strong.
What’s particularly telling is the emergence of an entire verification ecosystem. We’re seeing AI companies, legal research giants, and startups all racing to build safety layers, creating a multi-billion dollar market around making AI trustworthy. Harvey’s $8 billion valuation partly reflects this premium on verified, domain-specific AI.
The paper trail aspect is underappreciated but crucial. As AI becomes embedded in professional workflows, documentation proving due diligence will be essential for liability protection. This suggests a future where every AI-assisted document carries metadata showing what checks were performed—a new standard of professional responsibility for the AI age.
Why This Matters
This story highlights a critical challenge facing professional services as AI adoption accelerates: the tension between productivity gains and accuracy risks. The legal profession’s experience serves as a cautionary tale for other industries integrating generative AI into high-stakes workflows.
The 660 documented cases of AI hallucinations in legal filings represent just the tip of the iceberg, as many errors likely go undetected. The accelerating rate—from 120 cases to 660 in months—suggests the problem is growing faster than solutions. This creates significant liability exposure for firms and professionals who sign off on AI-generated content.
The emergence of AI detection tools to catch AI errors illustrates how the technology ecosystem is evolving to address its own shortcomings. Companies like Thomson Reuters, LexisNexis, and Harvey are building competitive moats around verified data, while startups like Clearbrief create verification layers. This spawns entirely new market categories around AI safety and verification.
For businesses across sectors, the lesson is clear: unrestricted use of general-purpose chatbots in professional contexts carries unacceptable risks. Organizations need structured approaches combining policy, specialized tools, verification systems, and documentation trails.
Related Stories
- How to Comply with Evolving AI Regulations
- PwC Hosts ‘Prompting Parties’ to Train Employees on AI Usage
- Business Leaders Share Top 3 AI Workforce Predictions for 2025
- CEOs Express Insecurity About AI Strategy and Implementation
Source: https://www.businessinsider.com/lawyers-legal-tech-companies-fight-ai-chatgpt-hallucinations-2025-12