The short version: pick the workload first, then the box. A single-user assistant, a team chat box and a million-token research rig are three different purchases, and the number that decides each one is different. Below is what our hardware measurably did for each workload — and, just as importantly, the workloads where we have no data and refuse to invent it.
Why hardware guides for local AI are so bad
Search any variant of “best hardware for local LLM” and the results share a structure: a table of GPUs by VRAM, a paragraph of memory arithmetic, a recommendation. Look for the measurement behind the recommendation and there usually isn’t one. Vendors rank for their own product categories — the top results for mini PCs are frequently the mini-PC sellers reviewing themselves. Affiliate roundups quote tokens-per-second figures that trace back, through several hops, to two or three original sources, stripped of the model, quant, context length and runtime that produced them. That stripping is the whole problem: a benchmark number is a config in disguise, and a config-free number cannot inform a purchase.
This page is narrower and, we think, more useful: what the two classes of hardware we own actually did, under frozen workloads, with every figure linked to its raw runs.
Start from the workload, not the box
Four questions decide almost every local-AI hardware choice. Answer them before looking at a single spec sheet.
How many people? One person’s assistant and a five-person team are different machines — not because of raw speed but because of how the wait grows under concurrency.
How long are the prompts? A chat workload and a “paste a 50-page document” workload stress different parts of the machine. The second is dominated by prefill, which behaves nothing like generation.
How big is the model, really? Not the parameter count — the class. A 35B-class mixture-of-experts model and a 284B one live on opposite sides of a boundary that no amount of unified memory in one box crosses.
Is it on all day? A box that is fast for ten minutes and a box that holds its pace for eight hours are different products, and the difference appears in no spec sheet.
What a 128 GB unified-memory box does
Our workhorse class: a Ryzen AI Max+ 395 machine with 128 GB of unified memory, llama.cpp pinned by image digest, serving a 35B-class MoE model. Everything here is measured on frozen workloads with failures kept in the denominator.
Solo assistant. First answer token in 210msunit_replicated, decoding at 15.9ms/tokenrepeated — under the threshold where a person registers waiting, and ahead of reading pace. For one user on everyday tasks, this class of box stopped being the bottleneck.
A small team. At four simultaneous chats the median wait for the first word is 338msrepeated; at eight it is 865msrepeated, with completion holding at 100.0% of requestsrepeated. The full ladder is how to size this honestly: pick the wait your users tolerate and read it backwards.
Long documents. The first question over a 32k-token document costs 33965msrepeated — that is the prefill bill, and it is the number document-workflow buyers should look at instead of decode speed. Stay in the session and the prompt cache brings the next question back in 860msrepeated; the mechanics, and the one condition it needs, matter more than hardware here.
All day. Three continuous hours at concurrency four moved the decode pace by 1.6%single_run, on two commercially identical units. The sustained-load report answers “will a mini PC throttle if I leave it serving”.
Automation. Strict-JSON tasks completed at 100.0% of requestsrepeated — and the reliability variable turned out to be the model, not the settings.
What this memory class runs in general is its own page; the complete measured tables live in the benchmark repository.
What a pair of DGX Sparks does
The other class we own: two GB10 units joined over a 200G link, serving a 284B-parameter MoE through vLLM. This answers a specific question — running the frontier open-weights class under your own roof — and it comes with operational homework: a kernel-selection flag the automatic mode skips, stock GPU monitoring that half-works, and a context curve to a million tokens whose first fill is an event you schedule rather than a request you fire. The measured buyer answer, including who should not buy one, is its own page; the two-column comparison with the Strix box is here.
Which box for which workload
Reading our own measurements back as a decision:
- One user, chat and code assistance — the 128 GB unified class does this with room to spare. The choices that follow matter more than the box: quantization and backend both moved our numbers more than most upgrade plans would.
- A small team on chat-shaped work — same class; size it from the concurrency ladder rather than a seat count.
- Document-heavy work — the same class again, but plan around prefill, not decode, and treat session reuse as a hardware-grade optimization.
- The frontier open-weights class — one box does not do it. A pair did, with seams, and we published both the recipe and the traps.
- Latency-critical single-stream work on heavy prompts — read the prefill figures above before buying anything; this is where local hardware disappoints people most often.
What we have not measured
This is the section every other hardware guide omits, and it is why ours is short where theirs are confident.
Discrete-GPU builds. Single and dual RTX-class cards, the used-3090 question, the 5090 question — these dominate the search results and we have no primary data on them. Our testbed declares a CUDA lane; it has no published runs, so we make no claims. When it runs, it will land with raw runs like everything else.
Apple Silicon. Declared lane, zero published runs, zero claims.
CPU-only and small-VRAM offload. The “model doesn’t fit” regime is the most-asked and least-measured question below this hardware class. We have not run it.
Power and money. We publish no prices — hardware prices move and a lab that quotes them ages badly. Watts per token is a real gap we intend to close once the metering methodology is settled; until then we have no electricity figures either.
If a guide covers all of these confidently, check whether it measured any of them.
How to buy without trusting anyone, including us
Decide the workload, then demand of any number you read: what device and firmware, what runtime build, what model artifact and quant, what prompt and output length, what concurrency, which side of the wire, how many repeats. The protocol is written up here and the harness is public — the frozen workloads we run are the ones you can run on a box before or after buying it.
Common questions with one-line measured answers are indexed on the answers page; every figure above has a permanent page with its scope, limitations and raw evidence in the claim registry.