Short answer. A small language model (SLM) is a language model with roughly 1 to 14 billion parameters, small enough to run on a single GPU or an on-premise server. SLMs cost 5 to 20 times less to operate than large model APIs, keep data inside your own infrastructure, and match or beat much larger models on narrow, domain-specific tasks. For a bank or insurer that cannot send customer data to an external API, a self-hosted SLM is often the only production-viable option.
A large language model like the ones behind public chat assistants can carry 200 billion or more parameters and runs in a vendor’s cloud. A small language model does the same core job, predicting and generating text, at a fraction of that size. Microsoft’s Phi-4-mini, released in February 2025, holds 3.8 billion parameters and runs on a single consumer GPU or an edge device. Its larger sibling, Phi-4, uses 14 billion parameters and scores competitively with models above 70 billion on mathematical reasoning and code generation.
The practical consequence is cost and control. A private SLM endpoint serving 10,000 queries a day typically runs 500 to 2,000 dollars a month. The equivalent traffic on a frontier LLM API runs 5,000 to 50,000 dollars a month. When GPU utilisation stays above 60 to 70 percent, running an SLM on-premise saves 30 to 50 percent over three years compared with cloud LLM calls.
The workloads that suit an SLM are narrow and high-volume:
Banks and insurers in Central Europe operate under GDPR, DORA, and the EU AI Act at the same time. Each one raises the cost of sending customer data to a third-party model hosted outside the institution’s control. An SLM removes that exposure because the model runs where the data already sits. No customer record leaves the network, no external processor is added to the DORA register of information, and EU data residency becomes a deployment decision rather than a contract negotiation.
The regulatory calculus also changes the buy decision. Under DORA, every material ICT third party is an outsourcing arrangement that has to be assessed, contracted, and monitored. A self-hosted SLM keeps that dependency in-house. For teams already mapping their AI systems against the EU AI Act, a model they train and operate themselves is easier to document, test, and explain than a closed API they cannot inspect.
An SLM carries roughly 1 to 14 billion parameters and runs on a single GPU or on-premise.
A recent survey of 70 open small models (arXiv, 2025) defines the class as transformer decoder models roughly in the 100 million to 5 billion parameter range, though many production teams stretch the working definition up to 14 billion. Large language models aim for broad general capability across every task. SLMs trade some of that breadth for speed, lower cost, and the ability to run locally.
An LLM answers almost anything adequately. An SLM, fine-tuned on your domain, answers your specific questions better and cheaper, and returns a response in tens of milliseconds instead of seconds. The difference is not intelligence in the abstract, it is fit to a defined job at a defined cost.
Use an SLM when the task is narrow and repeats at volume, when the data is sensitive and cannot leave the perimeter, and when latency or cost per call matters. Use a large model when the task needs broad reasoning across many domains or open-ended generation the bank cannot script in advance.
Many production systems use both. An SLM handles the high-volume, sensitive, well-defined work, and a larger model handles the rare, complex query. The deciding question is which model clears the task at the required accuracy for the lowest cost and risk.
Yes. Models under roughly 14 billion parameters fit on a single enterprise GPU and can be deployed inside a private data centre or an EU-based sovereign cloud. Because there are no external API calls, sensitive records never leave the institution’s infrastructure.
This is the property that makes SLMs attractive to regulated firms. 62 percent of European organisations report seeking sovereign AI options, and on-premise deployment is the most direct way to guarantee that customer data stays under EU jurisdiction. The data-residency question moves from a legal review of a vendor contract to an infrastructure choice the institution controls.
The EU AI Act classifies a general-purpose AI model as carrying systemic risk when its training compute exceeds 10^25 floating-point operations. SLMs sit far below that line, so they do not trigger the systemic-risk obligations that apply to the largest frontier models.
Two points still hold. The Act’s transparency obligations for general-purpose models applied from 2 August 2025 and cover models of any size. And if an SLM is embedded in a high-risk use case, such as credit scoring or insurance pricing, the high-risk obligations attach to that system regardless of the model’s parameter count. Small models still carry the transparency and high-risk duties that fit their use, and only the systemic-risk tier drops away.
On narrow tasks, a fine-tuned SLM often matches or beats a general LLM. Financial language is precise and repetitive, which suits a smaller model trained on domain data. Phi-4, at 14 billion parameters, already scores competitively with 70-billion-parameter models on reasoning and code benchmarks. When you add fine-tuning on your own documents, the gap on your specific task usually closes or reverses.
The trade-off is scope. An SLM tuned for claims processing will not answer general legal questions well. Accuracy on the task you care about comes from focus, not size.
Ableneo shipped 34 production AI projects in 2025, and 94 percent of them use language models. Roughly 4 of every 5 reach production, which is where the SLM-versus-LLM choice stops being academic. For a regulated client we start from the data and the obligation, not the model catalogue: what has to stay inside the perimeter, what latency the workflow needs, and what accuracy the task demands. The answer is often a small model, fine-tuned and self-hosted, wrapped in governance a bank can audit. Our work across financial services in Central Europe shows that the model which clears the task at the lowest risk usually wins, and that model is frequently small. See our AI transformation FAQ for the wider picture.
Key takeaways
Planning AI in a regulated business? Ableneo takes systems from classification to governed production.