This isn't an argument against cloud AI — for most day-to-day use, a cloud service is faster to start with, requires no hardware, and is kept current by people whose job is exactly that. The point of this article is narrower and more practical: there's a specific category of business use where running a model locally is the more sensible choice, and it's worth understanding that category clearly rather than either dismissing local AI as a hobbyist novelty or assuming every business needs to run its own.
What "local LLM" actually means
A local (or self-hosted) LLM is a language model that runs entirely on hardware you control — a workstation, a server in your own premises, or a VPS you manage — rather than sending each request to a provider's cloud infrastructure. Tools like Ollama have made this genuinely accessible: point it at a model, and it handles running that model locally with a straightforward interface, no different in spirit from installing any other piece of business software.
The short version: with a cloud API, your prompt and any data in it travels to someone else's servers to be processed. With a local LLM, it never leaves your hardware. That's the entire distinction, and it's also the entire reason the privacy conversation exists.
The privacy and data-sovereignty case, honestly stated
When you send a prompt to a cloud AI service, that data leaves your business and becomes subject to that provider's terms — how long it's retained, whether it's used to improve future models, who can access it internally, and which jurisdiction it's legally subject to. Reputable providers publish clear policies on this, and many offer business tiers with stronger guarantees than free consumer access. That's genuinely worth knowing and factoring in.
But a policy is still a policy, not a physical boundary. It can change, it can be misconfigured, and in some arrangements enterprise-grade guarantees cost more or require particular plan tiers. A local LLM sidesteps that conversation entirely for the specific data you route through it: there's no provider terms to read, because there's no provider in the loop. The data physically doesn't leave premises you control.
- No third party in the data path. A prompt containing a client's contract, a patient note, or unreleased financial figures never transits a network you don't control.
- No dependency on a provider's retention or training policy. Whether or not a cloud provider uses conversation data to improve future models is a real, currently active question for a lot of businesses. It's simply not a question when there's no cloud provider involved.
- Jurisdiction stops being a factor. Data residency and cross-border transfer concerns, which can be genuinely complex with cloud services, are a non-issue when processing never leaves the country — or the building.
- Consistent, auditable behaviour. The model, its configuration, and what it does with input are all under your control and don't change without your involvement.
Being honest about the other side: local doesn't automatically mean secure. A poorly secured server running a local LLM is still a poorly secured server — the privacy benefit comes from controlling where the data goes, not from any inherent security property of running AI locally. The usual hygiene (access control, patching, network segmentation from anything internet-facing) still applies in full.
Realistic hardware expectations
This is where the conversation usually goes one of two unhelpful directions — either "you need a data centre" or "any old laptop will do." Neither is accurate. The honest middle ground:
| VRAM | What it realistically supports |
|---|---|
| 8GB | Smaller models comfortably — genuinely useful for well-scoped tasks like drafting, summarising, and simple Q&A against a known document set |
| 16GB | A noticeably more capable range of models, with better reasoning and longer context handling — the point where local AI starts feeling closer to a general assistant rather than a narrow tool |
| 24GB+ | Larger models still, approaching (though not matching) the largest cloud-hosted models on many everyday tasks |
The realistic entry point is a single workstation or a modest dedicated server with a consumer or prosumer GPU — not a rack of enterprise accelerators. This is achievable on a budget that a small business can genuinely justify against the cost of a cloud AI subscription over a couple of years, and it doesn't require a background in machine learning to get running; tools like Ollama were built specifically to make this approachable.
The realistic capability gap, stated plainly: the largest cloud-hosted models are still ahead of what's practical to run locally, particularly on the hardest, most open-ended tasks. But most day-to-day business use isn't the hardest, most open-ended task — it's drafting from a known template, summarising a document, or answering questions against material you already have. For that category, a well-chosen local model is very often genuinely sufficient, not a compromised substitute.
Where this actually makes sense for a business
- Anything involving client or patient data that would otherwise need to be de-identified, redacted, or specifically cleared before it could go anywhere near a cloud service.
- Internal document summarisation and search over material a business would never want indexed or transmitted externally — contracts, HR records, financial reports.
- Repetitive, well-scoped drafting tasks where the same kind of output is generated repeatedly and a mid-sized model handles the pattern reliably.
- Regulatory or contractual obligations that specifically require data to stay within a jurisdiction, or off third-party infrastructure entirely.
It's rarely an all-or-nothing decision. Plenty of businesses land on a mixed approach: local for anything sensitive, cloud for everything else where the larger model's extra capability is worth the tradeoff and the data genuinely isn't sensitive. That's not a compromise position — it's matching the tool to what each specific task actually needs.
If you're curious what running local models on realistic, non-enterprise hardware actually looks like in practice — including the parts that don't go smoothly — we've documented a full hands-on hardware build and benchmarking process separately on the blog, starting with LLM on a Budget, Part I. That series is the build log and the numbers; this article is the business case for when doing something like it is actually the right call.
Weighing up whether a local LLM makes sense for how your business actually handles data? Get in touch and we can talk through what's actually sensitive in your workflow and where local AI genuinely helps.
Get in touch →Frequently asked questions
Why would a business run a local LLM instead of using a cloud AI API?
Mainly data control. A prompt sent to a cloud API leaves your premises and is subject to the provider's data handling terms. A local LLM processes everything on hardware you control, so sensitive data never leaves the business regardless of any provider's current policy.
Is a self-hosted LLM as capable as a cloud AI service like GPT or Claude?
Not at the very top end — the largest cloud models remain ahead on the hardest tasks. For well-scoped, repetitive tasks like drafting or summarising against known material, a mid-sized local model is often genuinely sufficient.
What hardware do I need to run a local LLM for business use?
A GPU with meaningful VRAM is the main requirement — roughly 8GB for smaller models, 16GB or more for a noticeably more capable range. A single well-specced workstation or modest server is enough; no data centre required.
Does running a local LLM mean I don't need cloud AI at all?
Not necessarily. Many businesses use a mixed approach — local for sensitive data, cloud where data isn't sensitive and the larger model's extra capability is worth it. It's about matching the tool to what each task actually requires.