Method

A number without its conditions is not a number.

The commitment is narrow and it is the whole product: a number gets published here only if it came off hardware I own, under a configuration documented on this page before the run. Nothing is estimated, extrapolated, or taken from a vendor claim. Where a figure comes from somewhere else, it carries a citation to its source and is labeled research-based, not measured.

Test hardware

• NVIDIA DGX Spark, GPU inference. Specification from NVIDIA's DGX Spark hardware page. Inference numbers live on /benchmarks; nothing here is a measured tokens/sec figure.
• Minisforum N5 Pro, storage and CPU inference. Full specification goes up alongside the first run.

Runtimes and versions

Each run records the runtime and the version it was measured on, and that pairing is shown in the results table. Published speed rows used Ollama, with the version string taken from the daemon at measurement time and written into the run file. A later runtime is a new run, not an edit. ComfyUI image and video jobs are queued in the same repo and are not a published runtime yet.

The queue

Models to test live in harness/queue/models.json. The shared speed settings live in harness/protocol/a-speed.json. Every model on the speed table eats the same prompt, the same context, the same run count, and think: false. A tag that cannot do that is skipped, not given a private quiz.

Only suite A (text speed) is published on /benchmarks today. Leveled text tasks, VLM questions, and ComfyUI workflows are in the queue as drafts. They will not share a tokens/sec column with suite A when they land.

What gets measured

• Tokens/sec, decode throughput from Ollama's eval_count / eval_duration, averaged over the timed runs
• Time to first token (TTFT), wall clock from request start to the first response token, on a model already in memory
• Load time, Ollama's load_duration on a cold load after an explicit unload. A warm cache is not a load time
• Memory, Ollama /api/ps size_vram during generation, stored in GiB (1024³)

Test procedure

These are the rules the published runs were held to. The walkthrough is how I benchmark a local LLM. The harness lives under harness/. Speed prompts are in harness/prompts/.

• Thinking is off. A generate that emits thinking tokens fails the run. Thinking-on is a different configuration and is not mixed into this mean.
• The harness unloads the model and waits until it is gone from /api/ps before the cold load. A chat UI that keeps the same tag resident at a long context is not suite A. The run exits rather than time that slot.
• Then a cold load (loadTimeS). One warm-up generate is discarded. Three timed generates of the same prompt follow, still resident. Tokens/sec and TTFT are the arithmetic mean of those three.
• Context length, temperature, and the decode cap are recorded in harness/protocol/a-speed.json, not assumed. The model tag comes from the queue.
• num_predict is a cap. The model may stop earlier. The mean still uses whatever decode length actually ran.
• Chassis temperature is not logged in v1. The thermal rule is still owed; v1's guarantee is an unloaded model, not a stated idle temperature.

Validation before publication

The harness writes a JSON file that must match the schema in site/src/data/benchmarks/. scripts/validate-benchmarks.mjs then checks every metric against sourced expected ranges for that hardware. An out-of-range number fails the run rather than reaching a draft. A failed or suspicious run gets investigated and re-run, never patched by hand. A later attempt that lands outside the band is deleted by the measure job the same way. It is not edited into range.

Published results are append-only. A re-test on a newer runtime is a new entry, and the changelog on the database page preserves the history. No run is published without me reading it first; the automation opens a pull request and a person merges it.

What I don't do

• No vendor-supplied numbers, ever
• No hand-edited results. Data flows from harness to page unmodified
• No quality verdicts from vibes. Leveled tasks with checkable graders are queued; they are not in the table yet
• No typed method version. The header label is derived from whether a non-sample run file exists