Small Language Models Are Winning the Enterprise Cost Argument
Fine-tuned models a fraction the size of frontier LLMs are handling most internal workloads at a tenth of the inference bill.
For much of the past two years, enterprise AI conversations assumed bigger was better. The largest frontier models set the benchmarks, and most pilots were built on them. That assumption is now being tested by an unglamorous metric: the monthly inference bill.
The cost argument
Fine-tuned small language models — a fraction of the size of frontier systems — are handling a growing share of internal workloads. Classification, extraction, summarisation of internal documents and routing of support requests rarely need the reasoning depth of the largest models.
Several technology leaders described moving these high-volume tasks to smaller models and cutting inference costs by as much as ninety percent, with little measurable drop in quality.
“Most of our requests are simple. We were paying frontier prices for filing-clerk work.”
Beyond cost
Price is the headline, but it is not the only advantage.
- Smaller models can run inside the company's own cloud tenancy or data centre
- Latency drops, which matters for customer-facing and real-time uses
- Fine-tuning on domain data often improves accuracy for narrow tasks
A portfolio, not a single model
The emerging pattern is a routing layer that sends each request to the cheapest model capable of handling it, escalating to a larger model only when needed. That approach turns model choice into an ongoing optimisation rather than a one-time platform bet.
For CIOs, the lesson is to measure cost per task, not just accuracy per benchmark. The best model for a job is increasingly the smallest one that does it well.