Short answer. An AI hallucination is output from a language model that reads as fluent and confident but is factually wrong or unsupported by any source. Published research puts the rate at 3% to 27% of outputs depending on the model and task. It happens because the model predicts the most likely next words from patterns in its training data, not from a check against verified facts, so a plausible sentence and a true sentence look identical to it. For a bank or insurer the danger is not that the answer sounds wrong, it is that a fabricated figure or citation sounds exactly as authoritative as a correct one.
A language model does not store facts the way a database does. It learns the statistical shape of language, then generates the next word that best fits the pattern. Most of the time that produces a correct answer, because true statements are common in the training text. When the model has a gap, it fills the gap with the most plausible continuation rather than stopping, and that continuation can be invented. Research published in Nature in 2026 found that the way models are scored makes this worse: standard benchmarks reward a confident guess over an honest “I do not know,” so the training process teaches models to bluff.
Hallucination is measurable, not rare. At those rates, a system answering a thousand questions a day will invent some of them, and it will do so without any signal that the answer is fabricated. The failure modes that matter in financial services are specific:
Each output is wrong in a way that looks right, which is what makes it dangerous in a regulated workflow.
In a regulated firm, an unverifiable answer is an unusable answer. The EU AI Act makes this a legal requirement, not a preference. Article 15 requires that high-risk AI systems reach an appropriate level of accuracy and robustness and perform consistently across their lifecycle, and the declared accuracy metrics must be stated in the instructions for use. Credit scoring and insurance pricing are high-risk uses under the Act, so a hallucinated input into one of those decisions is a compliance failure, not only a quality lapse. The core high-risk obligations apply from 2 August 2026.
The cybersecurity and robustness duties in Article 15 also overlap with the Digital Operational Resilience Act, in force for financial entities since 17 January 2025, so a bank has to map one AI system against both rule sets to avoid two divergent sets of controls. For teams in Slovakia, the Czech Republic, and Austria, the practical takeaway is plain. A system that can show the source behind every answer is far easier to govern, audit, and trust than one that produces confident text with no provenance.
Hallucination rates run 3% to 27% of outputs, so at scale some answers are always fabricated.
Detection rests on provenance, not on reading the answer and judging whether it sounds right. The strongest control is to ground the model in retrieved source documents and require a citation for every claim, so an answer with no supporting passage is flagged automatically. A second method runs the same question through two independent models and accepts the answer only when they agree, because a fabrication rarely repeats across systems. A third checks the model’s stated confidence and routes low-confidence outputs to a person. None of these is perfect alone, so regulated teams layer them and keep a human review step on any output that feeds a customer decision.
Article 15 of the Act sets accuracy, robustness, and cybersecurity as binding obligations for high-risk systems. Providers must declare the accuracy metrics of the system in its instructions for use, design it to perform consistently across its lifecycle, and protect it against errors and inconsistencies. For a Slovak or Czech bank using AI in credit scoring, that means a hallucination rate is not an internal quality number, it is a documented figure a supervisor can ask to see. The obligations for high-risk systems start on 2 August 2026.
A software bug is deterministic. The same input produces the same wrong output every time, so a test catches it and a fix removes it. A hallucination is probabilistic. The same prompt can return a correct answer on one run and an invented one on the next, because the model samples from a distribution rather than executing fixed logic. That is why hallucination cannot be fully removed by debugging. It is managed by grounding, citation, and human review rather than eliminated by a patch, and the right metric is the rate across many runs, not a single pass or fail.
Start with the highest-stakes workflow and ground it. Connect the model to the authoritative source documents for that task, require a citation for every answer, and apply a “no evidence, no answer” rule. Add the maker-checker control the bank already uses for manual work, so a qualified person signs off on any output before it reaches a customer or a decision. Measure the hallucination rate on a fixed test set before and after, so the improvement is a number, not an impression. This sequence delivers the largest drop in risk for the least disruption, and it produces the audit trail a supervisor will expect.
The way to manage hallucination is to build the system so that every answer is grounded, traceable, and checked. Ableneo builds that discipline into delivery for regulated clients across Central Europe, including ČSOB, Erste, and UNIQA, where a wrong number in a credit or insurance decision carries real cost. Of the 34 production AI projects Ableneo shipped in 2025, 94% use large language models, and the common thread that keeps them reliable is grounding the model in client data and keeping a human review path on high-stakes output. Our work on retrieval-augmented generation shows how source citations turn a fluent model into one a regulated firm can defend to an auditor.
Key takeaways
Planning AI in a regulated business? Ableneo takes systems from classification to governed production.