Short answerThe main risk of using AI in finance is a hallucination: a fluent, confident, wrong number or citation. You prevent it with controls, not trust. Ground the model in your own data, keep AI output as a draft, and put a human sign-off on anything posted, paid or sent outside. Governance stops the mistake, not the model.
In 2025, one of the Big Four accounting firms had to partially refund a government client after a report it delivered cited academic papers that did not exist and quoted a court judgment that was never written, as Fortune reported in October. The firm had used a general-purpose AI model to help draft it. Nobody checked the citations. That is an AI hallucination, and it is the sharpest risk of putting AI near your finances: the output is fluent, confident, and sometimes completely invented. This guide names where AI hallucinations hit finance and gives you a sign-off framework so one never reaches your investors, your auditor or your bank.
What are the risks of using AI in finance, and how do you prevent hallucinations?
A hallucination is when an AI model generates something plausible and wrong and presents it with the same confidence as the truth. It is not a bug you can patch out. It is how the technology works: a language model predicts likely text, not verified facts.
The scale is documented. In a 2024 Stanford study, leading general-purpose language models gave hallucinated answers to between 58% and 88% of specific legal questions, and invented the core holding of a case at least 75% of the time. These were factual, checkable questions with real answers. The models made things up anyway.
"But we connected it to our own data," founders say. That helps; it does not cure. A second 2024 Stanford study tested purpose-built research tools that ground answers in real documents and market themselves as hallucination-free. They still hallucinated 17% to 33% of the time. Grounding lowers the rate. It does not reach zero.
Finance is unusually exposed for three reasons. The outputs are numbers, so a wrong one looks exactly like a right one. The work is read by people who act on it. And the cost of a confident error is counted in money and trust, not typos.
Where AI lies in finance: five failure points
Hallucinations are not random noise. They cluster in predictable places, and knowing them tells you where to look before you sign.
| Failure | What it looks like | Example in finance |
|---|---|---|
| Fabricated figures | A number with no source in your data | An invented ARR or churn rate in a board deck |
| Invented citations | A "study" or benchmark that does not exist | A made-up industry average in an investor memo |
| Confident math errors | Arithmetic or logic that is subtly wrong | A miscalculated runway or LTV:CAC ratio |
| Stale or mismatched data | Right method, wrong or outdated inputs | Last quarter's numbers presented as current |
| Plausible but wrong story | A tidy narrative that misreads the data | "Churn fell because of the new plan," when it didn't |
A sign-off control framework
The fix is not a cleverer prompt. It is a control layer between the model and anyone who acts on its output. Map every AI-assisted finance task to its risk and the human checkpoint it must pass before it counts.
| Task | Hallucination risk | Control | Who signs off |
|---|---|---|---|
| Transaction coding, reconciliation | Miscategorised or invented entries | AI proposes, system logs, human clears exceptions | Bookkeeper / controller |
| First-draft forecast or budget | Invented growth or assumption | Show every source; a human owns each assumption | Founder / finance lead |
| Board pack, investor update | Fabricated figure or benchmark | Line-by-line check against source data | Founder / CFO |
| Payment run, accounts payable | Wrong payee or amount | AI prepares, a person releases; no auto-pay | Authorised signer |
| External filing (tax, lender, regulator) | Any error becomes a compliance breach | Full human review before submission | Founder / adviser |
This is the sign-off layer we describe in the AI CFO stack and its sign-off layer: the human checkpoint that sits above every automated task. The same discipline keeps an automated bookkeeping process auditable, and it matters more, not less, as agentic AI starts to close the books with fewer prompts.
The "never let AI touch this unchecked" checklist
Before anything an AI helped produce leaves your finance function, a named human confirms:
- Any number going outside: to an investor, board, lender, auditor or tax authority.
- Any figure the model "remembered" rather than pulled from your source data.
- Any citation, benchmark or industry average: confirm it exists and matches the source.
- Any journal entry posted to the ledger, with who approved it and when.
- Any payment, new payee or bank detail before money moves.
- Any assumption in a forecast: growth, churn, hiring, pricing.
- Any explanation of why a number moved: check the cause is real, not a neat guess.
If a task isn't on this list, automating it is low-risk. If it is, the model drafts and a person signs. The line does not move because a tool got more fluent.
The regulators already assume you checked
Regulators are not waiting for AI to mature. In June 2024, FINRA reminded firms that its rules are technology-neutral: using a language model does not shift responsibility off you, and a misleading number in a customer communication breaches the rules whether a human or a machine wrote it (FINRA Regulatory Notice 24-09). Three months earlier, the SEC brought its first "AI-washing" cases, and two investment advisers paid $400,000 combined to settle charges of false statements about their use of AI (SEC, March 2024). The message for founders is plain: AI is your tool, and its mistakes are your liability.
What this means for you
Treat every AI output as a confident intern's first draft: fast, useful, and never trusted without a check. Ground the model in your own numbers so it quotes your reality instead of inventing one. Keep an audit trail of what it changed and who approved it. And put a named person on anything that leaves the building. The goal is not to slow down. It is to make sure the one number that is wrong never reaches the one reader who acts on it.
Frequently asked questions
What is an AI hallucination in a finance context?
It is when an AI model generates a plausible but false output and presents it with full confidence: an invented figure, a citation to a study that does not exist, or a runway calculation that is subtly wrong. In finance the danger is that a made-up number looks identical to a real one, so nobody questions it until it has already been acted on.
Which finance tasks are most exposed to AI hallucinations?
Anything that produces a number a person acts on: board decks, investor updates, forecasts, benchmarks, and tax or lender filings. Free-text analysis is riskier than structured coding. The rule of thumb: the closer a task is to a decision or an outside reader, the tighter the human check needs to be, and the less you should trust a first draft.
Does connecting AI to my own data stop hallucinations?
It lowers the rate, not to zero. In a 2024 Stanford study, purpose-built tools that ground answers in real documents still hallucinated 17% to 33% of the time. Grounding helps the model quote your reality instead of inventing one, but you still need a human to confirm the output before it counts. Retrieval is a safety belt, not autopilot.
Are we liable if an AI tool produces a wrong number in an investor report?
Yes. Regulators treat AI as your tool and its errors as your responsibility. FINRA reminded firms in 2024 that its rules are technology-neutral, so a misleading figure breaches the rules whether a human or a model wrote it. Investors and auditors hold the founder accountable for what is signed, not the software that drafted it.
Sources
- Stanford RegLab & Institute for Human-Centered AI — Hallucinating Law: Legal Mistakes with Large Language Models Are Pervasive (Large Legal Fictions), 2024. https://law.stanford.edu/2024/01/11/hallucinating-law-legal-mistakes-with-large-language-models-are-pervasive/
- Stanford RegLab — Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, 2024. https://reglab.stanford.edu/publications/hallucination-free-assessing-the-reliability-of-leading-ai-legal-research-tools/
- FINRA — Regulatory Notice 24-09: FINRA Reminds Members of Their Obligations When Using Generative AI and Large Language Models, 2024. https://www.finra.org/rules-guidance/notices/24-09
- U.S. Securities and Exchange Commission — SEC Charges Two Investment Advisers with Making False and Misleading Statements About Their Use of Artificial Intelligence, 2024. https://www.sec.gov/newsroom/press-releases/2024-36
- Fortune — Deloitte was caught using AI in a government report after a researcher flagged hallucinations, 2025. https://fortune.com/2025/10/07/deloitte-ai-australia-government-report-hallucinations-technology-290000-refund/
