The most expensive architectural decision in most AI systems is one nobody made deliberately: everything goes to the frontier model, because that is what the first prototype used and nothing forced a revisit.
For a pilot that is correct. For production traffic at volume, it is the single largest source of waste in a typical AI bill.
The number that makes this worth reading
Processing 100 million tokens per day on a small model using rented hardware costs around $50 a day. The same volume through a frontier model API costs roughly $1,560. That is a 32x gap.
A more concrete example: a system producing 10,000 summaries a day runs about $7.20 on a small model against roughly $58 on a small frontier-tier model, with output quality that human evaluators could not reliably distinguish.
The critical qualifier is on tasks where the small model performs equally well. That is not all tasks. It is a specific and identifiable subset, and the entire engineering problem is knowing which subset.
Which tasks does a small model actually handle?
From what I have deployed and measured, the line falls fairly consistently.
Small model territory:
- Classification. Routing, triage, sentiment, intent detection.
- Structured extraction. Pulling fields from an invoice, a form, a transcript.
- Format conversion and normalisation.
- Short summarisation of bounded input.
- Tool argument construction inside an agent loop.
- Retrieval query rewriting and reranking.
Frontier model territory:
- Multi-step planning where the steps are not known in advance.
- Open-ended reasoning over ambiguous instructions.
- Tasks requiring broad world knowledge outside your domain.
- Anything where an undetectable wrong answer is expensive.
- Code generation of any complexity.
Look at that first list against a typical agent loop and something becomes obvious. A large share of what an agent does step to step is not reasoning. It is parsing, formatting, calling and normalising. A 3 to 9 billion parameter model running locally handles most of that loop faster, cheaper and more privately than a frontier model over an API, and the latency saving alone often matters more to users than the cost saving does.
The routing architecture
The pattern multiple enterprise deployments now confirm: route roughly 80% of predictable requests to a small model, escalate the genuinely complex 20%.
Build it in three layers, in this order.
Layer one, deterministic rules. Most production traffic is classifiable before the model sees it. Task type, input length, source system, data sensitivity. A request that is “extract these six fields from this document” goes to the small model, always, because you already know what it is. This layer alone typically captures 60 to 70% of the achievable saving and it carries no quality risk, because you decided in advance.
Layer two, confidence-based escalation. The small model handles the request and can defer. Implementation options range from an explicit “I am not confident” output token to a validity check on the structured output to a cheap secondary check. When it defers, the frontier model takes over. This catches the cases your rules got wrong.
Layer three, measurement. Log the routing decision, the model used, the outcome and the cost, on every request. Without this you cannot answer the only two questions that matter: what is your escalation rate, and is quality on the small-model path holding? Teams that skip this layer either over-route to the small model and quietly degrade quality, or panic at the first bad output and route everything back to the frontier model.
| Frontier only | Rules-based routing | Rules plus confidence escalation | |
|---|---|---|---|
| Cost relative to baseline | 100% | 35 to 50% | 20 to 35% |
| Engineering effort | None | Low, days | Moderate, weeks |
| Quality risk | None | Low if rules are conservative | Low, but needs monitoring |
| Latency | Baseline | Better on routed traffic | Worse on escalated traffic |
That last row is the honest tradeoff. An escalated request pays the small model’s latency and then the frontier model’s. If tail latency is your constraint, keep the confidence layer narrow.
Where teams get this wrong
Optimising too early. Do not route before the workflow is stable. While you are still discovering what a task needs, the frontier model is the right tool, and premature routing means you are debugging your router and your task at the same time. Optimise after the task is well defined, and expect to redo it when the task changes.
Evaluating on vibes. “The small model seemed worse” is not a finding. Build a test set of 100 to 200 real examples from your own traffic with known good outputs, and measure. In my experience teams are wrong about which model is better on a specific narrow task about a third of the time, in both directions.
Ignoring fine-tuning. Off the shelf, a small model is competent. Fine-tuned on your data for one job, a small model routinely beats frontier models on accuracy for that job, because it has learned your domain vocabulary and your output format. If a task runs at high volume and has stable requirements, fine-tuning is usually a better investment than prompt engineering.
Forgetting the operational cost. Self-hosting a small model means you own uptime, scaling, GPU capacity and version management. At low volume that overhead exceeds the API saving comfortably. The crossover for most workloads sits somewhere around a few million tokens a day, and it moves with your engineering cost. Run the calculation for your own situation rather than trusting anyone’s benchmark, mine included.
The privacy angle nobody costs properly
There is a second reason to care about small models in Australia, and it is not on the invoice.
A model running on your own infrastructure processes data that never leaves it. For workloads touching personal information, health records or anything with a residency constraint, that removes an entire category of compliance work: no cross-border disclosure analysis, no vendor data processing terms, no argument about whether inference logs constitute retention. Given where Australian AI regulation is heading, that simplification will get more valuable, not less.
I have seen organisations spend more on the legal and assurance work for a cloud AI deployment than the compute would have cost on-premises for two years. That cost never appears in a model comparison spreadsheet.
Where to start
Pick your highest-volume AI workflow. Sample 200 real requests. Classify them by task type. If more than half are classification, extraction, formatting or short structured generation, you have a routing opportunity that will pay for itself inside a quarter.
Then run the boring experiment: those requests through a small model, scored against your existing outputs. If quality holds, route them. If it does not, you have learned something specific about your workload rather than a general opinion about model size.
This sits alongside prompt caching as the highest-return cost work available, and I have covered the rest of that ground in why AI bills grow while token prices fall. Between them, a 50 to 80% reduction on a mature workload is normal, without touching quality. Very few optimisations in enterprise software have that profile.
Sources
- Small language models outperform frontier AI on cost, speed and accuracy, Forbes
- Small language models eat the edge: the 32x cost disruption reshaping enterprise AI, Practical Logix
- Small language models for on-device agents in 2026, Digital Applied
- Small language models vs LLMs: how to cut AI inference costs, Innoflexion