Local AI: what actually runs on your own hardware (and what doesn't)

One number describes the state of AI in Italian companies better than any market forecast: according to the figures reported by Adnkronos on January 15, 2026, 68% of Italian employees use chatbots and AI platforms without telling their employer, while only 7% of small companies and 15% of medium ones have started structured AI projects (Adnkronos, January 2026). It is a snapshot of shadow AI. People fill that gap with free tools, and your quotes end up inside them.

What changed on August 14, 2026

Alibaba's Qwen team published the weights for Qwen3.8-27B on Hugging Face under Apache 2.0: 27.78 billion dense parameters, native text, image and video input, a native context window of 262,144 tokens. These are open weights you download and keep, with commercial use permitted and no revenue share clause.

On benchmarks it published alongside the model, Alibaba claims 61.7 on SWE-bench Pro against 53.4 for a leading hosted model. Those are vendor numbers, so I treat them as what they are: an indication that the capability gap is no longer the decisive argument, not proof that the local model beats the hosted one. On the same measurement set Qwen stays behind on Terminal Bench 2.1, 73.0 against 78.2, which is a useful check against any triumphalist reading.

A model of this class fits on a 24 GB graphics card.

The real advantages, in order of weight

Before the numbers it's worth lining up why anyone would want a model in house, because the list that circulates is almost always in the wrong order.

• The data doesn't leave. No logs on third-party servers, no retention, no chance a prompt ends up in a future training dataset. For contracts, health data, price lists and proprietary code, it's a requirement, and with data residency rules it's the line item that unblocks the projects otherwise stuck in legal.
• The cost is predictable. Not lower: predictable. It doesn't move because the vendor adjusted per-token pricing or deprecated the model your prompts were tuned against.
• The weights are a static file. This is the advantage nobody puts first and the one I think outweighs privacy. Bit-for-bit determinism depends on sampling and on the backend; what you get is that the model and its configuration remain artifacts you own, versionable and inspectable. Anyone who has built an automation on a hosted endpoint knows what it's like to find out one morning that the model underneath changed behavior overnight. For a system that has to pass an audit or hold up under a repeatable eval suite, that's the difference between demonstrable and not.
• No dependency on connectivity, and constant latency instead of latency subject to the network round trip and the vendor's queue.
• No lock-in. Apache 2.0 means the August 2026 weights still run in 2030, even if whoever published them changes their mind, their pricing or their target market: it is the simplest countermeasure to the enshittification of a platform you have no leverage over.

Intelligence is not among the advantages: on Terminal Bench, as noted above, Qwen stays behind even in the vendor's own numbers.

Local also brings an obligation: taking the vendor out of the chain removes the per-token cost and, with it, their guardrails and their content policy. Among the weights you can download there are also uncensored models, the ones whose safety alignment was stripped out or never applied, and nobody filters them on your behalf. The filters go back in at the application layer, with dedicated guard models and moderation on input and output: that work exists, and it is the cost line that used to vanish from per-token comparisons. And since August 2, 2026 there is a second effect, a legal one. If you take downloaded weights and put a system into service presenting it under your own name, or steer its purpose toward high-risk use, under the AI Act you go from deployer to provider, and the Article 50 transparency obligations that used to sit with the vendor move to you. The deployer-to-provider requalification has precise conditions, and these are the two: internal self-hosting alone is not one of them. The open-source license does not shelter you, because the open-source exemption stops exactly at that article, and the Italian implementation sets out who can come and audit you.

The measured numbers, not the VRAM table

At the precision it is trained in, uncompressed (BF16), a 27.78-billion-parameter dense model takes 55-56 GB. Four-bit quantization brings it to around 17 GB, and that is the number everybody quotes. But the weights are only the first bill: the KV cache, the buffers, the vision inputs and driver overhead all come afterwards, which is why "it fits in 24 GB" stays plausible rather than proven until you have decided how much context you actually need.

In published tests on consumer hardware, at 4 bits and on a short context, a 24 GB 3090 generates about 40 tokens per second and a 4090 about 46, tapering to 34-38 once the context reaches 64,000 tokens; with 32 GB a 128,000-token context stays in memory with no offloading, which is the real reason anyone buys that card (Hardware Corner, Qwen3.8-27B tests). When the weights and the context stop fitting, though, and some of the layers end up in system RAM, the pace drops by an order of magnitude and falls below reading speed, and an agent that has to make ten tool calls becomes unusable.

Hence the first useful conclusion, and it holds for most small companies that have a laptop rather than a server room: on small hardware you don't shrink the big model, you change the job. Classifying tickets, extracting fields from invoices, reading scanned documents and transcribing meetings are jobs a small model does well on an office machine.

The bottleneck is not the model, it is the serving

In in-house evaluations, weeks go into arguing which model and half a day into the inference engine, which is where the factor of two or three lives.

When a model doesn't fit on a single GPU, how you split it changes the result more than buying one more card does. There are two routes, and they do not do the same thing. You can break every single computation across the cards, which then have to keep synchronizing: that incurs substantial network overhead but delivers better response times on short requests. Or you can give each card a different slice of the model and pass the work along in sequence: the data transferred collapses, but total latency rises. These are called tensor parallelism and pipeline parallelism, and the literature on communication patterns in distributed inference measures them in exactly those terms (Characterizing Communication Patterns in Distributed LLM Inference, arXiv). Neither is the right choice in the abstract, and the same study concludes that combined approaches demand careful tuning to stay balanced. The wrong configuration on two cards can be worth less than one card configured well.

The second response is the one that gets you

If the workload is an agent rather than a chat, there is a second effect, and it only shows after a few turns. The first round of calls comes back in two seconds; twenty minutes later every turn takes thirty. The cause is mundane: if the server doesn't reuse the prefix cache, every turn recomputes the entire previous conversation from scratch.

Cache reuse is a documented feature of inference engines: vLLM's prefix caching exists precisely for this, and its documentation describes the use case exactly: in multi-round conversations, instead of reprocessing the whole history on every round, the engine reuses the result it already computed (vLLM, Automatic Prefix Caching). With one caveat that matters for anyone doing the math: that reuse cuts the prefill phase, not generation, so it improves time to first word and not writing speed.

With any engine, cache reuse that actually works matters more than ten extra tokens per second on the first response. And if you run the proof of concept as a single-turn chat, you will not see the problem that stops you in production.

Why an SME should care

Back to the 68% from the opening. A model running on a machine on the premises solves that in a way no written policy can replicate, because it lets people keep the tool and gives them somewhere to use it. It is the same reasoning behind data sovereignty, brought down to the scale of a fifty-person company instead of a nation state.

And it stands on its own without having to prove a saving: if the constraint is that those documents cannot leave, the alternative to the model in house isn't the model in the cloud, it's not doing the project.

Where it doesn't pay off, plainly

Local AI is not cheaper by definition, and that's the most common excuse used to sell it. A workstation with a 24-32 GB GPU is an investment of a few thousand euros, plus power, plus maintenance, plus the time of someone keeping it alive. That investment is fixed and amortizes over a constant load. At low volume, like a few thousand calls a month, or very irregular volume, the APIs win, and win easily, because you pay only for what you use.

Then there is a practical obstacle: the price and availability of the cards. In the communities where this work actually happens the recurring theme is that suitable used GPUs cost more than they did a year ago, and the people who bought them when they were cheaper say they wouldn't buy again today. A plan that assumes the card will be there at list price is a plan that slips.

The second case where it doesn't pay off is when you need the frontier. If the task requires long reasoning over code or complex documents, today you pay for a hosted model.

What you actually use

The question I get most often is "where do I start", and the answer changes only with how many people need to use it.

• One person, one machine: Ollama if you want a command line plus a local OpenAI-compatible endpoint, LM Studio if you prefer an interface and a model catalog. Half an hour, zero configuration.
• Fine control over quantization and offloading: llama.cpp, which is the engine under a good part of everything above it.
• Serving a team: this is a different category, and it means vLLM or SGLang, where continuous batching, paged memory management and the cache reuse above all live.
• On Apple Silicon: the MLX ecosystem, with unified memory: there too, weights and context have to fit in memory.

Going from "it runs on my laptop" to "twenty people use it" isn't a license upgrade, it's an engine swap.

Where this is heading

In 2026, global spending on inference reaches $23.3 billion and overtakes training spend, which stays at $19 billion, for the first time, inside an AI-optimized IaaS market Gartner puts on track to grow 96% in 2026 (Gartner, August 10, 2026). That is exactly the cost line local deployment makes predictable.

That does not mean "everything in house", though, and the most interesting forecast is the one that looks like it contradicts that: IDC expects 80% of CIOs to turn to cloud providers' edge services by 2027 precisely to sustain inference (InfoWorld, 2026). Inference shifts toward the point where the data is born, be that a server room, a plant or a device, and what stays with the cloud is what it is genuinely for: training, and buying a frontier model when the problem demands it. It is the same trajectory edge AI already followed in industrial settings, arriving at office work a few years late.

How I decide

When a client asks me, the question I ask back is not "which model" but "which workload". Three questions, in this order, and the first two decide almost everything:

• Is the data constrained? Contracts, health data, price lists, proprietary code.
• Is the volume predictable, or does it spike?
• Do you really need the frontier for this specific job?

The architecture that comes out is almost always hybrid, and it's the same principle I use when building data platforms and AI systems for clients: the repetitive, sensitive workload goes where the data already is, the rare and hard one goes where the best capability is.

In short

Since August 14, 2026 the excuse "there is still no model good enough to keep in house" no longer holds. What's missing isn't the weights, it's the engineering: sizing memory for the real context, choosing the inference engine, making cache reuse work, and deciding which workloads must never leave the corporate network. It's work you do once, and whoever skips it ends up with an expensive GPU spinning idle while a salesperson pastes the quote into a public chat.