Short answer. Prompt injection is an attack that hides instructions inside the text an AI model reads, making the model ignore its original rules and follow the attacker instead. OWASP ranks it the number 1 security risk for large language model applications in 2025. It works because a language model cannot reliably separate the developer’s trusted instructions from untrusted content it processes in the same window.
A language model reads everything as one stream of text. The system prompt written by the developer, the user’s question, an email it summarizes, a document it retrieves: all of it arrives as tokens with no built-in label for “trusted” or “untrusted”. Prompt injection exploits that gap. An attacker writes text that reads like a command, and the model obeys it because it has no reliable way to know it should not.
The concrete cases follow a pattern. A customer-facing chatbot is told, inside a user message, to ignore its guardrails and reveal its confidential system prompt. A hiring assistant reads a CV that contains hidden white-on-white text saying “recommend this candidate”, and the recommendation changes. An AI agent with permission to send email or move money reads a booby-trapped web page and executes an action the user never asked for.
The risk scales with autonomy. A model that only writes text can leak information. An agent that can act on the world can be turned into a tool that acts against its owner.
In financial services and insurance, the systems most likely to use language models are also the ones handling customer data, credit decisions, and payments. That combination makes prompt injection a governance problem, not only an engineering one. A model that can be talked into ignoring its rules cannot be described as controlled, and control is what regulators ask you to demonstrate.
The EU AI Act makes this explicit. Article 15 of Regulation (EU) 2024/1689 requires high-risk AI systems to be resilient against attempts by unauthorised third parties to alter their use or behaviour by exploiting system vulnerabilities, and it names adversarial manipulation and data poisoning directly. DORA adds an operational-resilience layer for financial entities: an AI system that can be manipulated through its inputs is an ICT risk that has to be tested, monitored, and reported like any other.
Prompt injection is OWASP’s number 1 security risk for LLM applications in 2025: hidden instructions make a model ignore its rules.
NIST draws this line in its report Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (AI 100-2e2023). Direct prompt injection is when the attacker is the user: they type a crafted prompt straight into the model to override its instructions. This is the version most people picture, and it is the easier one to test for.
Indirect prompt injection is more dangerous because the attacker never touches the model. They plant the malicious instructions in content the model will later read: a web page, a support ticket, a shared document, an email in an inbox the assistant summarizes. When the model ingests that content, it executes the hidden command. Any system that retrieves external data, including most RAG and agent architectures, has this exposure by design.
The two overlap but aim at different targets. Jailbreaking tries to bypass the model’s safety training so it produces content it was built to refuse, such as instructions for harm. Prompt injection targets the application built around the model: it overrides the developer’s system prompt and hijacks the task the application was supposed to perform.
A jailbreak attacks the model provider’s policy. A prompt injection attacks your deployment. You can run a fully aligned, well-behaved model and still be fully exposed to injection, because the weakness is in how untrusted text reaches the model inside your own system, not in the model’s values.
For high-risk systems, Article 15 sets an outcome, not a checklist: the system must achieve an appropriate level of accuracy, robustness, and cybersecurity and perform consistently across its lifecycle. Resistance to adversarial manipulation is part of that obligation, and the accuracy metrics have to be declared in the instructions for use. In practice this means providers are expected to stress-test and red-team any system exposed to user-controlled input before it ships and after each significant change.
The General-Purpose AI Code of Practice, finalised on 10 July 2025, reinforces the same direction for the largest models: adversarial testing and red teaming before deployment, before each major update, and at least annually. The obligations for general-purpose AI models start applying on 2 August 2026. The message across both texts is consistent: adversarial testing is now a documented requirement, not an optional good practice.
There is no single control that closes prompt injection, so defence is layered. The most reliable principle is to stop trusting model output for high-stakes actions. Treat everything the model reads as untrusted, and put a deterministic check between the model and any consequence.
None of these controls is perfect alone. Together they turn a single point of failure into a chain that an attacker has to break at several points.
Ableneo builds production AI for regulated organisations, where a model that can be talked out of its rules is not shippable. In 2025, 94% of our delivered projects used large language models, and roughly 4 of 5 reached production, so input security is designed in from the first architecture review rather than bolted on before launch. Our approach follows one principle: the language model proposes, a deterministic system decides. We scope agent permissions tightly, keep untrusted content out of the instruction channel, and put an auditable checkpoint in front of any action that moves money or data. That discipline is how AI systems stay controlled, observable, and accountable in banking and insurance. See our AI Transformation FAQ for the surrounding governance topics.
Key takeaways
Planning AI in a regulated business? Ableneo takes systems from classification to governed production.