Hallucinations: how much can you trust a model, and how do you make it safe
3 min read
A language model does not “know” things: it generates the most plausible text given the context. Almost always, plausible and true coincide. When they do not, the model produces a hallucination: invented information, stated with the same confidence as correct information. This is not a growing pain that models will leave behind: it is a property of the mechanism. And for a company it is an operational and legal risk, not a cosmetic one.
The risk is already case law
Two cases are enough to frame the problem. A Canadian tribunal found Air Canada liable for honouring a discount its chatbot had invented: the company argued the chatbot was “a separate entity responsible for its own actions”, and the judge replied that no, what your system tells customers binds you. In the United States (and by now in Europe too) a series of lawyers have been sanctioned for filing submissions with non-existent rulings, generated by the model and never checked. The judicial line is clear: the hallucination is not an excuse — it is yours.
Why “the model will improve” is not a strategy
Frontier models hallucinate less than their predecessors, it is true. But “less” across millions of interactions still means “every day”, and paradoxically the better models produce errors that are more credible, ones that slip past the quick read of a distracted operator. Safety cannot depend on hoping the error will not happen: it must depend on an architecture that contains it when it does.
The four defences that actually work
- Anchor answers to your data (grounding). The model must not answer “from memory”: it must read the documents, the databases, the company systems — and answer only from those. If the information is not there, the right answer is “I don’t know”, not a fluent invention. Technically, this is done by connecting the model to an operational model of your data: the ontology says what exists, the language model narrates it.
- Verifiable citations, always. Every relevant claim must point to its source: the paragraph in the contract, the line in the management system, the production batch. An answer without a source is the model’s opinion; an answer with a source is a fact checkable in ten seconds.
- Constrain actions, not just words. When the AI doesn’t just write but acts — opening a ticket, amending an order, stopping a line — the hallucination becomes an act. Here the defence is structural: the agent can only execute typed actions on objects that exist in the ontology, and critical actions require human approval. An agent cannot invent a customer if the system only lets it act on real customers.
- Measure, per use case. “How much does it hallucinate?” has no answer in the abstract: it has an answer for your task, with your documents. Before going into production, a test set with known answers is built and measured against. Afterwards, sampling continues. What you don’t measure, you discover in court.
The practical rule
Always distinguish between two uses. AI that assists an expert person (summarises, searches, drafts proposals): manageable risk, the person is the control. AI that answers or acts externally without review: full risk, all four defences are needed — or better, don’t do it at all until they are in place. The difference between the two cases is exactly what a serious supplier helps you map out before selling you anything.
Do you have a use case where an error would be costly? Half an hour with one of our experts to work out which defences you actually need.