How Do You Set Success Criteria for an AI Pilot in a Bank Before You Start?

7 min readAbleneo AI transformation team

Short answer. Fix three things in writing before the pilot starts: the production bar, the single metric and threshold that promotes the pilot to production; one kill criterion that stops it; and a named production owner who holds the budget. Ableneo moves 4 of 5 projects to production, and the reason is that the bar is agreed on day one, not argued over after the demo.

1What This Means in Practice

A pilot without a written bar cannot pass or fail. It can only produce a demo that impresses a room and then waits for a decision nobody is positioned to make. Setting the criteria first turns the pilot into a test with a clear yes or no at the end, and gives the board the go or no-go basis it will ask for.

In practice the criteria are a short contract between the business sponsor and the delivery team, agreed before any model is chosen. It states what the pilot has to prove, against which baseline, and who decides. Keep it to one page.

Ableneo shipped 34 production AI engineering projects in a single year, 94% of them using large language models, across banking, telecom, insurance and other regulated sectors. The projects that reach production share one habit: the production threshold is written before the build, so the pilot has something concrete to clear.

2Why This Matters for Regulated Industries

In a bank, an AI pilot that reaches production without defined criteria becomes a compliance gap, not just a project risk. A supervisor asks for the metric the system was tested against and the threshold it had to clear. A pilot that was promoted on the strength of a good demo has no answer, and the gap surfaces during an examination rather than during the project, where it is far more expensive to fix.

DORA adds a second reason. The moment an AI system goes live it is ICT in scope, so its rollback, monitoring and incident thresholds belong in the criteria before production, not after an auditor asks. Banks in Slovakia and the Czech Republic operate under the same EU framework, supervised by the NBS and the CNB, so the discipline is identical across the CEE region: define the numbers first, then build to them.

Write the production bar, one kill criterion, and a named owner before the pilot starts, not after the demo.

3What Should Go in the Success Criteria?

Three numbers and one name. The first number is a target business metric with its current baseline and the threshold that counts as success, so improvement is measured against where the bank starts, not against zero. The second is a quality floor, the minimum accuracy or pass rate the system must clear on the bank’s own cases, tested on data it has not seen. The third is a cost ceiling per unit of work, so the business case survives contact with production volumes. The name is the person who will own the system in production.

Vague goals like “make document search better” are not criteria because they cannot be scored. A criterion is written so that two people reading the pilot results would reach the same go or no-go answer.

4How Do You Write a Kill Criterion?

One sentence, signed by whoever holds the budget, written on day one. It names the condition under which the pilot stops. For example: if accuracy on held-out cases stays below 90% after eight weeks, we stop. A kill criterion protects the budget and the team from a pilot that drifts for months without a decision.

Industry analysts expect a large share of agentic AI projects to be cancelled before they reach production. A written stop condition is how a bank cancels early and cheaply, on evidence, instead of late and expensively, on politics. It also makes the go decision stronger, because a pilot that clears a bar it could have failed carries real weight with the board.

5Who Should Own the Pilot and the Production System?

The production owner should not be the pilot champion or the vendor contact. The champion is motivated to keep the pilot alive. The owner is accountable for whether it runs in production and appears in next year’s budget. Those are different jobs, and naming the owner before the pilot starts closes the gap where a successful pilot stalls because no team is positioned to operate it.

In a bank the owner usually sits in the business line that will use the system, with model risk and ICT risk as named sign-offs. That way the promotion decision has one accountable name and two control gates, agreed before anyone writes code.

6How Is This Different From a Proof of Concept?

A proof of concept answers one question: can this work technically. A pilot with production criteria answers a different one: will we run this, and when. The two get confused, which is why many proofs of concept produce a working demo and no decision. The difference is the bar. A proof of concept can succeed on a data scientist’s laptop. A pilot only succeeds if it clears a production threshold the business agreed to in advance, on the bank’s own data, at the bank’s own cost and risk tolerance.

7What Does the EU AI Act Require Before a High-Risk AI Pilot Goes Live?

For a high-risk system, Regulation (EU) 2024/1689 requires a documented risk management system (Article 9), defined levels of accuracy, robustness and cybersecurity (Article 15), and a quality management system (Article 17) before the system is put into service. Article 9 requires testing against preliminarily defined metrics and thresholds, which is the legal form of setting a production bar first. Credit scoring and creditworthiness assessment are listed as high-risk in Annex III. For a bank piloting that kind of use case, measurable criteria set before production are a condition of going live, not a nice-to-have, and defining them on day one is how the pilot stays on the right side of the deadline.

8The Ableneo Perspective

Ableneo’s production record, 4 of 5 projects reaching production across banking, telecom and insurance in four countries, comes from one discipline: the production bar is written before the build, not discovered after it. The work with T-Mobile Czech Republic cut document validation from 7 hours to 30 minutes, 14 times faster, at over 90% accuracy, because the target was fixed before the first model ran. That is how a pilot becomes a system a bank can sign off and keep running. More on how Ableneo approaches AI in production is at ableneo.com.

Key takeaways

Sources

Planning AI in a regulated business? Ableneo takes systems from classification to governed production.

Talk to Ableneo