Local AI: what actually runs on your own hardware (and what doesn't)

One number describes the state of AI in Italian companies better than any market forecast: according to the figures reported by Adnkronos on January 15, 2026, 68% of Italian employees use chatbots and AI platforms without telling their employer, while only 7% of small companies and 15% of medium ones have started structured AI projects (Adnkronos, January 2026). It is a snapshot of shadow AI. That gap does not stay empty. People fill it, with free tools, and with your quotes inside them.

That's the context for reading what happened on August 14. And it should be read without enthusiasm, because the interesting part is not the model.

What changed on August 14, 2026

Alibaba's Qwen team published the weights for Qwen3.8-27B on Hugging Face under Apache 2.0: 27.78 billion dense parameters, native text, image and video input, a native context window of 262,144 tokens. These are open weights you download and keep, with commercial use permitted and no revenue share clause.

On benchmarks it published alongside the model, Alibaba claims 61.7 on SWE-bench Pro against 53.4 for a leading hosted model. Those are vendor numbers, so I treat them as what they are: an indication that the capability gap is no longer the decisive argument, not proof that the local model beats the hosted one. On the same measurement set Qwen stays behind on Terminal Bench 2.1, 73.0 against 78.2, which is a useful check against any triumphalist reading.

The practical point is elsewhere: a model of this class fits on a 24 GB graphics card. And that is where the part no model card tells you begins.

The real advantages, in order of weight

Before the numbers it's worth lining up why anyone would want a model in house, because the list that circulates is almost always in the wrong order.

• The data doesn't leave. No logs on third-party servers, no retention, no chance a prompt ends up in a future training dataset. For contracts, health data, price lists and proprietary code, this isn't a commercial advantage; it's a requirement. And with data residency rules it's the line item that unblocks the projects otherwise stuck in legal.
• The cost is predictable. Not lower: predictable. It doesn't move because the vendor adjusted per-token pricing or deprecated the model your prompts were tuned against.
• The weights are a static file. This is the advantage nobody puts first and the one I think outweighs privacy. It is not bit-for-bit determinism, which depends on sampling and on the backend: it is that the model and its configuration remain artifacts you own, versionable and inspectable. Anyone who has built an automation on a hosted endpoint knows what it's like to find out one morning that the model underneath changed behavior overnight. For a system that has to pass an audit or hold up under a repeatable eval suite, that's the difference between demonstrable and not.
• No dependency on connectivity, and constant latency instead of latency subject to the network round trip and the vendor's queue.
• No lock-in. Apache 2.0 means the August 2026 weights still run in 2030, even if whoever published them changes their mind, their pricing or their target market.

Note what isn't on that list: "it's smarter". It isn't, and anyone selling it that way is selling something else.

And note what the list is missing: an obligation. Taking the vendor out of the chain removes more than the per-token cost: it removes their guardrails and their content policy too. Among the weights you can download there are also uncensored models, the ones whose safety alignment was stripped out or never applied, and nobody filters them on your behalf. The filters go back in at the application layer, with dedicated guard models and moderation on input and output: that work exists, and it is the cost line that used to vanish from per-token comparisons. And there is a second effect that matters more than it looks, and since August 2, 2026 it is a legal one. If you take downloaded weights and put a system into service presenting it under your own name, or steer its purpose toward high-risk use, the AI Act does not leave you as a mere deployer: you become the provider, and the Article 50 transparency obligations that used to sit with the vendor move to you. The deployer-to-provider requalification has precise conditions, and these are the two: internal self-hosting alone is not one of them. The open-source license does not shelter you, because the open-source exemption stops exactly at that article, and the Italian implementation sets out who can come and audit you. This is not an argument against local: it is the reason local is the governed option only if you actually govern it.

The measured numbers, not the VRAM table

At the precision it is trained in, uncompressed (BF16), a 27.78-billion-parameter dense model takes 55-56 GB. Four-bit quantization brings it to around 17 GB, and that is the number everybody quotes. But the weights are only the first bill: the KV cache, the buffers, the vision inputs and driver overhead all come afterwards, which is why "it fits in 24 GB" stays plausible rather than proven until you have decided how much context you actually need.

In published tests on consumer hardware, at 4 bits and on a short context, a 24 GB 3090 generates about 40 tokens per second and a 4090 about 46, tapering to 34-38 once the context reaches 64,000 tokens; with 32 GB a 128,000-token context stays in memory with no offloading, which is the real reason anyone buys that card (Hardware Corner, Qwen3.8-27B tests). When the weights and the context stop fitting, though, and some of the layers end up in system RAM, the pace drops by an order of magnitude and falls below reading speed: there the nature of the problem changes. The model still answers, but it doesn't work. An agent that has to make ten tool calls isn't slow, it's unusable.

Hence the first useful conclusion, and it holds for most small companies that have a laptop rather than a server room: on small hardware you don't shrink the big model, you change the job. Classifying tickets, extracting fields from invoices, reading scanned documents and transcribing meetings are jobs a small model does well on an office machine. A big model, choked, does all of them badly.

The bottleneck is not the model, it is the serving

This is the point that always gets skipped in in-house evaluations. Weeks go into arguing which model, and half a day into the inference engine, which is where the factor of two or three lives.

When a model doesn't fit on a single GPU, how you split it changes the result more than buying one more card does. There are two routes, and they do not do the same thing. You can break every single computation across the cards, which then have to keep synchronizing: that incurs substantial network overhead but delivers better response times on short requests. Or you can give each card a different slice of the model and pass the work along in sequence: the data transferred collapses, but total latency rises. These are called tensor parallelism and pipeline parallelism, and the literature on communication patterns in distributed inference measures them in exactly those terms (Characterizing Communication Patterns in Distributed LLM Inference, arXiv). Neither is the right choice in the abstract, and the same study concludes that combined approaches demand careful tuning to stay balanced. Translated: the wrong configuration on two cards can be worth less than one card configured well, and no model benchmark table will tell you that.

The second response is the one that gets you

If the workload is an agent rather than a chat, there is a second effect nobody measures during the trial. The first round of calls comes back in two seconds; twenty minutes later every turn takes thirty. The cause is mundane: if the server doesn't reuse the prefix cache, every turn recomputes the entire previous conversation from scratch.

This isn't an esoteric detail, it's a documented feature of serious inference engines. vLLM's prefix caching exists precisely for this, and its documentation describes the use case exactly: in multi-round conversations, instead of reprocessing the whole history on every round, the engine reuses the result it already computed (vLLM, Automatic Prefix Caching). With one caveat that matters for anyone doing the math: that reuse cuts the prefill phase, not generation, so it improves time to first word and not writing speed.

The lesson is tool-independent: a cache reuse that actually works matters more than ten extra tokens per second on the first response. And if you run the proof of concept as a single-turn chat, you will not see the problem that stops you in production.

Why an Italian SME should care

Back to the 68% from the opening. A model running on a machine on the premises solves that in a way no written policy can replicate, because it doesn't ask anyone to give up the tool: it gives them somewhere to use it. It is the same reasoning behind data sovereignty, brought down to the scale of a fifty-person company instead of a nation state.

And it stands on its own without having to prove a saving: if the constraint is that those documents cannot leave, the alternative to the model in house isn't the model in the cloud, it's not doing the project.

Where it doesn't pay off, plainly

Local AI is no longer cheaper by definition, and that's the most common excuse used to sell it. A workstation with a 24-32 GB GPU is an investment of a few thousand euros, plus power, plus maintenance, plus the time of someone keeping it alive. That investment is fixed: it amortizes over a constant load, not over a few thousand calls a month. At low or very irregular volume the APIs win, and win easily, because you pay only for what you use.

Then there is an obstacle no market analysis records and anyone who has tried knows: the price and availability of the cards. In the communities where this work actually happens the recurring theme isn't which model to download, it's that suitable used GPUs cost more than they did a year ago, and the people who bought them when they were cheaper say they wouldn't buy again today. A plan that assumes the card will be there at list price is a plan that slips.

The second case where it doesn't pay off is when you need the frontier. If the task requires long reasoning over code or complex documents, today you pay for a hosted model and don't tell yourself stories.

What you actually use

The question I get most often is "where do I start", and the answer changes only with how many people need to use it.

• One person, one machine: Ollama if you want a command line plus a local OpenAI-compatible endpoint, LM Studio if you prefer an interface and a model catalog. Half an hour, zero configuration.
• Fine control over quantization and offloading: llama.cpp, which is the engine under a good part of everything above it.
• Serving a team: this is a different category, and it means vLLM or SGLang, where continuous batching, paged memory management and the cache reuse above all live.
• On Apple Silicon: the MLX ecosystem, where unified memory changes the arithmetic but not the threshold.

Going from "it runs on my laptop" to "twenty people use it" isn't a license upgrade, it's an engine swap. Discovering that in production means discovering it at the worst possible moment.

Where this is heading

In 2026, global spending on inference reaches $23.3 billion and overtakes training spend, which stays at $19 billion, for the first time, inside an AI-optimized IaaS market Gartner puts on track to grow 96% in 2026 (Gartner, August 10, 2026). That is exactly the cost line local deployment makes predictable, and it's why "where does inference run" moved from technical teams to boardrooms.

That does not mean "everything in house", though, and the most interesting forecast is the one that looks like it contradicts that: IDC expects 80% of CIOs to turn to cloud providers' edge services by 2027 precisely to sustain inference (InfoWorld, 2026). The cloud is not losing; it is moving closer to where the data already sits. Inference shifts toward the point where the data is born, be that a server room, a plant or a device, and what stays with the cloud is what it is genuinely for: training, and buying a frontier model when the problem demands it. This isn't a tinkerer's niche: it's the same trajectory edge AI already followed in industrial settings, arriving at office work a few years late.

How I decide

When a client asks me, the question I ask back is not "which model" but "which workload". Three questions, in this order, and the first two decide almost everything:

• Is the data constrained? Contracts, health data, price lists, proprietary code.
• Is the volume predictable, or does it spike?
• Do you really need the frontier for this specific job?

The architecture that comes out is almost always hybrid, and it's the same principle I use when building data platforms and AI systems for clients: the repetitive, sensitive workload goes where the data already is, the rare and hard one goes where the best capability is.

In short

Since August 14, 2026 the excuse "there is still no model good enough to keep in house" no longer holds. What's missing isn't the weights, it's the engineering: sizing memory for the real context, choosing the inference engine, making cache reuse work, and deciding which workloads must never leave the corporate network. It's work you do once, and whoever skips it ends up with an expensive GPU spinning idle while a salesperson pastes the quote into a public chat.