Why AI assistants make things up — and what citations actually fix

A language model predicts likely text. That is the whole mechanism, and it explains the behaviour completely. Asked what your refund window is, it produces the words that most plausibly follow that question — and “30 days” is an extremely plausible continuation whether or not it is true of you.

The trap is that there is no internal difference between a fact it has seen and a fact it has constructed. Both come out in the same confident register, with the same fluency. A person who doesn’t know something usually sounds like it. A model doesn’t, and that mismatch is why people are caught out.

So “making things up” isn’t a bug that will be patched. It’s the same capability that lets the model write a fluent sentence at all, pointed at a question it has no grounds to answer. The engineering question is not how to remove it, but how to give the model grounds — and how to make it obvious when it hasn’t got any.

Grounding means supplying the model with the relevant source material at the moment the question is asked, and instructing it to answer from that and nothing else. The practical effect is large: instead of reaching into a general impression of the world, it is summarising a specific passage sitting in front of it.

This is what an assistant built over your own content does. Someone asks about your returns policy, the passage of your returns policy is retrieved, and the answer is written from it. The model is doing reading comprehension, which it is genuinely good at, rather than recall, which it is not.

What grounding does not do is make the model incapable of error. It can still misread, over-generalise from a partial passage, or blend two passages that shouldn’t be combined. Grounding takes the failure rate from “regularly” to “occasionally”, which is a real improvement and not a guarantee. Anyone selling you a guarantee here is overselling. The mechanics of how it works are in fine-tuning vs retrieval.

A citation attached to an answer — this document, this page; this video, this timestamp — does one thing precisely: it makes the answer checkable.

What that buys you:

• Anyone who doubts an answer can verify it in seconds instead of taking it on trust or abandoning the question.
• When something is wrong, you can see immediately whether the assistant misread the source or the source itself was wrong — which are different problems with different fixes.
• The reader ends up in your material, which is usually where you wanted them.

What it does not buy you:

• Correctness. A perfectly faithful answer from an out-of-date document is a wrong answer with a receipt. Citations relocate the problem to your content, where you can actually fix it.
• Certainty that anyone will check. Most people won’t. The citation’s value is that it’s available and that it disciplines the system, not that every reader audits it.

It’s worth being clear-eyed about this, because “it cites its sources” is often sold as though it eliminated the risk. It doesn’t. It makes the risk visible and correctable, which is the most anyone can honestly offer.

The single most valuable behaviour in a business-facing assistant is the willingness to decline. Not the range of what it can answer — the reliability with which it refuses to answer what it can’t.

Consider the cost asymmetry. An assistant that says “that isn’t covered in our material — support can help” has cost you one unanswered question. An assistant that invents a 30-day return window you don’t offer has created a commitment a customer will reasonably hold you to, a complaint, and a support conversation that starts from a worse place than silence. The second failure is many times more expensive than the first, and it’s the one people don’t test for.

It also has a knock-on benefit: the gaps become visible. Every honest “I don’t have that” is a question your documentation doesn’t answer, which is exactly the list you want when deciding what to write next.

Twenty minutes of adversarial testing tells you more than any vendor page, ours included. Do this before you put anything in front of an audience.

• Ask something you know isn’t documented. The only acceptable answer is that it doesn’t have it. Anything confident is a fail, and one fail here should end the evaluation.
• Ask a false-premise question. “Why does your enterprise plan include on-site training?” when it doesn’t. Good behaviour is correcting the premise; bad behaviour is answering the question as asked.
• Ask for a number. Prices, deadlines, notice periods, dosages. Numbers are where invented answers do the most damage, and where you should check the citation every time.
• Ask the same thing three ways. Formal, casual, misspelt. Answers that contradict each other mean retrieval is unstable, not that the model is creative.
• Ask about a competitor or something off-topic. It should stay inside your material rather than free-associating about the wider industry.

This is the standard we build assistants over your documents to meet: answers confined to your uploaded material, a document and page on every one, and an honest refusal when your content doesn’t cover the question. Test ours the same way — that’s the point of publishing the test.