How I Benchmark a Local LLM

Chatting with a model in Open WebUI tells you it loaded. It does not tell you how fast inference is, how long the first token took, or how much memory the weights held, under conditions you could publish twice. I wanted a number the benchmark database would accept. This page is the protocol. The Qwen row is the worked example: the kit, the commands, and the figures copied from the run file.

If you only want to talk to a model, stop at run your first local LLM with Ollama. This page is the next job.

The kit this run used

Three boxes sit in this room. Only one of them is in the table.

The machine under test is an NVIDIA DGX Spark. NVIDIA’s hardware page lists 128 GB of LPDDR5x unified memory at 273 GB/s of memory bandwidth. Those are specifications, not measurements. The runtime on it for this row is ollama 0.32.15.

The weights came from the Ollama library tag huihui_ai/Qwen3.8-abliterated:27b, a community fine-tune of Qwen3.8 27B. The model card lives on Hugging Face; the GGUF build is next to it if you are not using Ollama. Abliterated is the publisher’s name for a refusal-removal edit. This page measures speed and memory, not alignment.

The NAS in the same room is a Minisforum N5 Pro. It holds the lab’s files. It was not the box under test, and I have not published an inference row for it. Can you run AI on a NAS is still the honest page for that hardware class: usually yes, usually slowly, and I will not pretend the Spark figure is what an N5 would print.

If you do not have a Spark and you still want to feel a 27B, renting a GPU by the hour on RunPod or Vast.ai is the other path. You will not get 23.285 tokens per second unless you also have this hardware. The arithmetic for when renting beats buying is cloud GPU vs owning hardware.

Pull the model

On the Spark, with Ollama already installed:

ollama pull huihui_ai/Qwen3.8-abliterated:27b
ollama ps

ollama ps is the check that the weights actually landed in memory rather than spilling to the CPU.

The default tag is a k-quant, Q4_K_M.

That is four-bit class quantization that is never quite four bits per weight, which is its own subject. The card reports 27.3B parameters. I pinned context length at 8192 for the timed runs, which is a choice, not the model’s maximum.

How much VRAM you need is still the arithmetic for whether a model fits. Put your own pool through the model fit calculator if you are not on a Spark. This page is what happened when one model that already fitted was actually run.

Why a chat window is the wrong clock

Open WebUI is a good way to use a model. It is a poor way to time one.

The UI streams tokens through a browser, a reverse proxy, and a chat template. A stopwatch on the page mixes network, thinking text, and decode. The next session uses a different prompt length, a different stop, and maybe a model that was already in memory. None of that is wrong for chatting. All of it is wrong for a table.

A measurement here has to survive a skeptic. Same model name, same quant, same context, same runtime version, same prompt, enough repeats to be a mean, and a file nobody typed numbers into.

What I pin before I hit run

Four figures, the ones the run schema requires:

• Tokens per second: decode only. Ollama already returns eval_count and eval_duration. Tokens per second is that ratio. I do not run a second wall clock that can disagree with it.
• Time to first token: wall clock from the request starting to the first response token, on a model already in memory. Not the first thinking chunk, and not a cold load dressed up as latency.
• Load time: Ollama’s load_duration after an explicit unload. A warm cache reporting 0.0006 s is not a load time, and the validator’s floor is there to refuse it.
• Memory: /api/ps size_vram, stored in GiB (1024³). On this machine that is unified memory. The KV cache is why the same weights cost more as context grows; 8192 is part of the identity of this row, not a footnote. Ollama still reports the resident size as size_vram. It is not a discrete VRAM card.

harness/protocol/a-speed.json pins the rest so it is not in my head: decode cap 128 (a cap, not a length; the model may stop sooner), temperature 0, 8192 context, three timed runs, think: false. The model tag comes from harness/queue/models.json. Every model on the speed table eats that same protocol. A tag that cannot do that is skipped, not given a private quiz. harness/config.json is a leftover single-file snapshot. Do not use it as an output.

Prompts live in harness/prompts/. The warm-up and the timed prompt are the same sentence: “Reply with one sentence about what a kilowatt hour is.” The warm-up is discarded. The timed prompt runs three times so the mean is one configuration rather than three anecdotes.

I did check thinking on during characterization. The first visible token arrived later because the model thought first, and Ollama’s eval_count then mixed thinking with the answer. That is a different study, and it is not this row.

Chassis temperature is not logged in v1. That rule is still owed.

Run the harness

The code lives in harness/ of the QuietWatts repository. It talks to Ollama over HTTP on the lab LAN. There is no SSH step in v1. OLLAMA_HOST overrides the host in protocol/a-speed.json if you are pointing at a different daemon.

From the repo root, against a Spark that already has the model. --run-id must be new. Reusing one would overwrite history.

1. Confirm the model is the tag you think it is: ollama list.

2. Dry-run the harness so it prints JSON and writes nothing:

   python3 harness/run.py --model-id qwen38-abliterated-27b --run-id 2026-08-spark-qwen38

3. Write a run file and let the validator see it. Pick a new id. The published file is already 2026-08-spark-qwen38.json; reusing that id is refused.

   RUN_ID=2026-08-spark-qwen38-r2 bash pipeline/jobs/measure.sh

   measure.sh calls harness/run.py with --output, then scripts/validate-benchmarks.mjs. If the validator refuses the file, the job deletes it. A bad number never reaches a commit that way.

What harness/run.py enforces, in order:

1. Unload the model, then wait until it is gone from /api/ps. If a chat UI still holds the tag, the harness exits and writes nothing.
2. One generate. Record load_duration as loadTimeS. Discard the rest.
3. Three timed generates while it stays resident. Mean of tokens per second and time to first token.
4. Unload again.
5. Write JSON. Do not edit it.

If you change the model tag, the context, or the runtime, that is a new row, not an edit to this one.

What came back

Copied from site/src/data/benchmarks/runs/2026-08-spark-qwen38.json. If a later sentence here disagrees with that file, the file is right.

After the cold load and the discarded warm-up, the three warm timed runs decoded at 23.285 tokens per second. Time to first token was 208.6 ms. The cold load itself was 6.236 s. Resident memory was 16.238 GiB. Date of the run: 2026-08-26.

Twenty-three tokens per second is a number to feel, not a slogan. Put it in the tokens per second simulator and watch a sentence stream at that rate before you decide whether you would sit in front of it. Adult reading is roughly five tokens per second; this is several times that, on this hardware, at this quant and this context.

The receipt:

Field	Value
Hardware	NVIDIA DGX Spark
Runtime	ollama 0.32.15
Model	huihui_ai/Qwen3.8-abliterated:27b
Params	27.3B
Quant	Q4_K_M
Context	8192
Tokens/sec	23.285
TTFT	208.6 ms
Cold load	6.236 s
Memory	16.238 GiB
Timed runs	3
Date	2026-08-26

The live table, and the changelog, are on the benchmark database. The conditions are on the methodology page.

What the validator refuses

The site will not build a non-sample run that fails scripts/validate-benchmarks.mjs. The failures that matter:

• Zero on tokens per second, time to first token, load, or memory. A zero means the harness did not measure it.
• Unknown hardware. The name has to match machines.json, which is what /methodology renders.
• No sourced bounds for that machine. An unbounded Spark would have waved garbage through. Bounds live in expected-ranges.json with a source on each metric. They are plausibility bands, not targets.
• Out of range. CPU fallback, a missed warm-up (TTFT in seconds instead of milliseconds), or a warm cache recorded as load time.

I do not widen a bound to make a file pass. If a genuine result falls outside, the bound was wrong: change it in the same PR, with a better source, and say so. A later attempt on this same tag, while Open WebUI still held a 128k session, landed below the Spark tokens/sec floor. The measure job deleted that file. It is not this row, and I did not move the floor to fit it.

What this does not mean

It is not a ranking of models. It is one model, one quant, one context, one runtime, one machine. Which local model should you run is still the page that refuses to rank what I have not measured.

It is not your PC, and it is not the N5 Pro. A Spark has 128 GB of unified memory at 273 GB/s, per NVIDIA’s hardware page. Your card’s bandwidth and capacity are a different machine.

It is not Q8, not 32k context, not thinking-on, not a quality score, and not tokens per watt. Those need their own runs. Leveled text tasks, VLM questions, and ComfyUI jobs are in the queue as drafts. They are not this table, and they will not share a tokens/sec column with it when they land. The tools that wait on measured watts are still waiting, because this row is speed and memory.

It is not advice to buy a Spark. Cloud GPU vs owning hardware is still the page for that arithmetic, and it still says buying often never pays back. Put your own hours into the break-even calculator before you treat a desk-side box as cheaper than renting.

Where to go next

• Read the row on the live page: benchmark database
• Read the conditions: methodology
• Feel 23 tokens per second: tokens per second simulator
• If you have not run a model at all yet: Ollama, step by step
• If you are choosing a size, not a clock: which local model should you run

FAQ

Can I reproduce this on my own Ollama host?

Yes, if you have the same model tag and you point harness/run.py at your daemon. You will not get 23.285 tokens per second unless you also have this hardware. You should get a JSON file the validator understands, or a clear failure. Pass --model-id and a new --run-id. Set OLLAMA_HOST, or edit the host in harness/protocol/a-speed.json. Do not reuse harness/config.json as the output.

What if Open WebUI still has the model loaded?

Close that session, or pick another model in the UI, then re-run. Suite A unloads and waits until /api/ps is empty of this tag. A chat UI that keeps it resident at a long context is not this protocol, and the harness will refuse to time that slot rather than publish it.

Why not time the Open WebUI stream?

Because the browser is not the runtime. Time to first token on a page includes the UI, thinking, and whether the weights were already loaded. The harness asks Ollama, records the daemon’s own eval counters, and keeps thinking off so those counters are decode.

Why three runs?

Fewer than three is an anecdote. The schema will not accept it. The mean is the three warm timed generates, not the cold load mixed in.

Did thinking-on make it faster?

No. Apparent tokens per second went up because Ollama counted thinking tokens with the answer, while the first visible token arrived later. That mix is why thinking-on is not this row.

Is 16.238 GiB VRAM?

On this machine it is unified memory, reported by Ollama as size_vram. The method page stores it in GiB. It is not a 16 GB discrete card.

Did the N5 Pro run this model?

No. The N5 Pro is the NAS in the same room. This row is the Spark. An N5 inference figure will land as its own run file, or it will not land at all.

Sources

• The four figures in the table are copied from site/src/data/benchmarks/runs/2026-08-spark-qwen38.json, written by harness/run.py on 2026-08-26. They were not typed into this article.
• Ollama 0.32.15 /api/generate and /api/ps are the measurement surface: load_duration, eval_count, eval_duration, size_vram.
• Model tag: huihui_ai/Qwen3.8-abliterated:27b. Weights home: huihui-ai/Huihui-Qwen3.8-27B-abliterated on Hugging Face.
• Spark memory capacity and bandwidth cited above are from NVIDIA’s DGX Spark hardware documentation, not from this run.
• Expected-range sources for the Spark are recorded in site/src/data/benchmarks/expected-ranges.json.