What "making things up" actually means
The technical word is hallucination. It means the model produces text that reads as a fact but is not one. It is not a bug you can patch. It is how the tool works.
A language model predicts the next word. It does not look things up unless you connect it to something to look up. When it has no source, it produces the most plausible-sounding words instead of the true ones.
Researchers at OpenAI and Georgia Tech argued in September 2025 that the cause is the scoring system. Their paper "Why Language Models Hallucinate" says standard training and evaluation reward confident guessing over saying "I don't know" (openai.com). A model that guesses scores better on tests than a model that abstains. So models learn to guess.
That matters for you in one practical way. Telling the AI "do not make anything up" does not change the incentive it was trained on. Only changing the task does.
How often it happens, with real numbers
The Vectara Hallucination Leaderboard measures the easiest possible version of the task. It hands a model a document and asks for a short, factual summary of that document only, using nothing outside it (github.com/vectara). No research. No recall. Just "restate what is in front of you."
Even at that, models invent things. Here is a sample from the 11 May 2026 update (github.com/vectara):
| Model | Hallucination rate | Answer rate |
|---|---|---|
| Finix S1 32B | 1.8% | 99.5% |
| GPT-5.4-nano | 3.1% | 100.0% |
| Gemini 2.5 Flash Lite | 3.3% | 99.5% |
| Llama 3.3 70B Instruct Turbo | 4.1% | 99.5% |
| GPT-4.1 | 5.6% | 99.9% |
| Grok 3 | 5.8% | 93.0% |
| GPT-5.4 | 7.0% | 99.9% |
| Gemini 2.5 Pro | 7.0% | 99.1% |
| GPT-5.4-pro | 8.3% | 100.0% |
| GPT-5.5 | 9.3% | 100.0% |
Read that number carefully. It is a floor, not a ceiling. Ask for something the AI has no document for and the rate climbs well above these figures. A statistic, a case reference, a competitor's pricing. There is nothing for it to be faithful to.
Now look at the ordering, because it is the most useful thing on this page. On this benchmark GPT-5.4-nano scored 3.1% while the larger GPT-5.4 scored 7.0%, GPT-5.4-pro scored 8.3% and GPT-5.5 scored 9.3% (github.com/vectara). The bigger, newer, more expensive model scored three times worse at staying faithful to a supplied document.
Bigger models are better at reasoning and worse at shutting up. That is why "just upgrade to the flagship model" is not a document-accuracy strategy.
The practical rule is short. Every unsourced fact in an AI draft is a guess until you check it.
Five things that genuinely reduce it
These are ordered by how much difference they make for a small business.
- Give it the source. Paste the contract, the invoice, the policy, the transcript. Do not ask from memory. A model summarizing your document is doing the 1.8%-error task. A model answering from nothing is doing something much worse.
- Demand quotes with locations. Ask for the exact sentence and the page or section number for every claim. Fabricated quotes are much easier to spot than fabricated paraphrases, because you can search for them.
- Give it permission to fail. Add: "If the answer is not in the documents I gave you, write NOT IN SOURCE and stop." Models guess partly because the request implies an answer must exist.
- Work in small pieces. One section at a time, with only the sources for that section. Long documents with long context invite the model to fill gaps.
- Separate drafting from checking. Draft in one session. Check in a fresh one, with the sources open beside you. Do not ask the same model that wrote the claim whether the claim is true.
A ten-minute checking routine
Run this on any document going to a customer, a lender, an insurer or a court.
- Highlight every number. Prices, dates, percentages, quantities, deadlines. Trace each one to a document you own.
- Highlight every proper noun. Company names, people, product names, statute names, case names. Confirm each one exists and is spelled correctly.
- Open every citation. If the AI cited a study, a law or an article, click it. A citation that does not resolve is the single strongest hallucination signal.
- Recompute every total. Do not eyeball arithmetic. AI addition looks right and is often wrong.
- Read the document aloud once. Invented sentences frequently sound slightly too smooth and slightly too general.
Ten minutes of this is cheaper than any of the outcomes in the last section.
Five red flags in a document somebody else handed you
You will often be checking work you did not write — a contractor's report, a staff member's proposal, a supplier's summary. These are the tells, taken from documented cases.
- A citation that will not resolve. A broken DOI, a dead link, a journal article that no search engine can find. In the MAHA report case, reporters found fake paper titles sitting alongside broken DOI links (politifact.com).
- A real expert attached to a paper they never wrote. The MAHA report named a Columbia University epidemiologist as author of a study she never wrote (politifact.com). The name is real. That is what makes it convincing.
- Leftover tool markers in the text. Reporters found "oaicite" markers, an artefact of AI citation output, left in the MAHA report (politifact.com). Search any document for stray tokens like that.
- Statistics with no year, no sample size and no publisher. "Studies show 73% of customers…" with nothing attached. Real research comes with a source you can name.
- Uniform confidence. Human experts hedge on the hard parts. Text that is equally certain about everything, including the parts that should be uncertain, is a signal worth checking.
None of these prove AI was used. All of them mean the document needs a source check before you sign it.
What most articles get wrong about this
Myth 1: a better prompt fixes it. Prompts help at the margins. They do not remove the underlying behavior. The OpenAI paper's argument is that the incentive sits in training and evaluation, not in your wording (openai.com).
Myth 2: the newest, biggest model has solved it. The table above is the counter-evidence. At the 11 May 2026 update, GPT-5.4-nano beat GPT-5.5 by 6.2 percentage points on faithfulness to a supplied document, 3.1% against 9.3% (github.com/vectara). Paying more does not buy accuracy on this specific task.
Myth 3: turning the "temperature" down to zero stops it. Lower randomness makes output more repeatable. A model can repeat the same wrong fact very consistently. Consistency is not accuracy.
Myth 4: it only affects complicated work. The documents that get people in trouble are ordinary. A letter citing a policy. A proposal quoting a statistic. A report listing its sources.
Where it is safe enough, and where it is not
| Document type | Risk if a fact is invented | Reasonable use |
|---|---|---|
| Internal first drafts, brainstorms | Low | Draft freely, check before sharing |
| Marketing copy with no claims or figures | Low | Draft freely, check names and spelling |
| Marketing copy with statistics or comparisons | High | Only use figures you sourced yourself |
| Customer emails quoting your own policy | High | Paste the policy in, require quotes |
| Quotes, invoices, anything with money | Very high | AI drafts wording only, never the numbers |
| Legal, medical, tax, insurance, regulatory filings | Very high | Human expert reviews every claim |
The pattern is simple. AI is good at shaping words you supply. It is unreliable at supplying facts you did not.
What it costs when the check is skipped
The AI Hallucination Cases database, maintained by Damien Charlotin at HEC Paris, tracks court decisions where a tribunal found that a filing contained AI-generated fabrications. It listed 2,008 cases worldwide as of its 2 September 2026 update, with 1,378 in the United States, 214 in Canada and 62 in the UK (damiencharlotin.com). The database is free to browse and download as a CSV.
It happens to large organizations too. It happens to large organizations too. In May 2025, NOTUS reported that the White House "Make America Healthy Again" report cited studies that do not exist (politifact.com). At least 18 citations were changed or replaced within days (science.org).
The cost to a small business is rarely a fine. It is the client who finds the error first.
Where these figures come from
- OpenAI — "Why language models hallucinate" (September 2025), the research explaining why training rewards guessing over abstaining. https://openai.com/index/why-language-models-hallucinate/
- Kalai, Nachum, Vempala & Zhang — "Why Language Models Hallucinate", full paper (arXiv 2509.04664). https://arxiv.org/pdf/2509.04664
- Vectara Hallucination Leaderboard (HHEM), updated 11 May 2026 — hallucination rates when models summarize a supplied document. https://github.com/vectara/hallucination-leaderboard
- Damien Charlotin, AI Hallucination Cases Database, updated 2 September 2026 — 2,008 cases, 1,378 in the USA. https://www.damiencharlotin.com/hallucinations/
- PolitiFact — "How fake citations appeared in RFK Jr.'s MAHA report" (30 May 2025). https://politifact.com/article/2025/may/30/MAHA-report-AI-fake-citations/
- Science — "Trump officials downplay fake citations in high-profile report on children's health" (2025). https://www.science.org/content/article/trump-officials-downplay-fake-citations-high-profile-report-children-s-health
People also ask
- Can AI answer my customer inquiries for me?
- Do I have to tell customers when a reply is written by AI?
- What should I never let AI write without checking it first?
- How do I give AI my own documents to work from?
