THE VERDICT
● API for almost everything. Cheaper than people assume, and no hardware to own.
● Local when data cannot leave, latency must be predictable, or you need it to work offline.
● The misconception: local is not usually cheaper. At most volumes an API wins on cost.
The three real reasons to run locally
| Reason | When it applies |
|---|
| Data cannot leave | Regulated sectors, client confidentiality, anything you would not paste into a browser |
| Predictable latency | No network variance, no provider outage, no rate limit mid-task |
| Offline operation | Field work, air-gapped environments, unreliable connectivity |
COST IS NOT ON THAT LIST, AND SHOULD NOT BE
A cheap model tier runs around $0.20 per million input tokens. A million tokens is a large amount of text.
Against that, local costs you hardware, electricity, setup time and ongoing maintenance. Unless you are running enormous volume, the API is cheaper. People choose local for control and then justify it on cost, which gets the reasoning backwards.
Where the quality gap actually bites
A small local model handles classification, extraction, summarising and routing perfectly well. It struggles with long multi-step reasoning, nuanced writing, and anything where being subtly wrong matters.
The practical pattern is a split: run cheap high-volume work locally, route the hard cases to an API. That gets most of the privacy benefit on the bulk of your data while keeping quality where it counts.
Which one
| If you are... | Pick |
|---|
| Handling data that cannot leave | Local, and check the licence permits your use |
| Trying to save money | API. Price a month before assuming otherwise |
| Doing bulk classification or extraction | Either works. Try local first, it is a fair test |
| Needing an audit trail on the model itself | Local, on a fully open family where the training data is published |
FAQ
Is running a model locally cheaper?
Usually not, once hardware, electricity and time are counted. Local wins on control rather than cost at most volumes.
What size do I need?
The smallest that does your job. For classification and extraction that is often far smaller than people assume.
Can I mix both?
Yes, and it is the common pattern — bulk work local, hard cases routed to an API.