Same questions. Every lane. Losses included.

Reflex · modelless — no neural weights: decisions come from a per-label corpus plus a fitted sigmoid gate, so it answers in micro-seconds and is bit-deterministic on a given build. laya (rust) — our Rust port running the open-weights laya checkpoints on the local GPU (device noted per run: Metal, or the Apple Neural Engine where the lane serves it). laya (python) — the original torch reference, driven as a subprocess oracle on the same cases. clm — the external Contrastive-LM reference (Apache-2.0, not affiliated): Qwen3-8B pooling + trained heads, served on its own stack and measured on ours — a comparison lane (full detail in the clm entry below). gliner — the external GLiNER2.5-Decide zero-shot classifier (Apache-2.0, not affiliated, 340M DeBERTa-v3 encoder): their gliner2 package answers the same candidate-label questions — a comparison lane, never a product lane. agentjev — the external AgentJev-0.6B typed-decision specialist (Apache-2.0, not affiliated, Qwen3-0.6B backbone): their service answers over HTTP, measured gold-label on our split — a comparison lane, never a product lane. Instinct — the trained-specialist hybrid (open source: riir-instinct): a distilled specialist (pinned by a BLAKE3 digest checked at load) composed with Reflex — the modelless lane's fused gate decides when to consult it; the cell is the registered arm's single frozen test read over the same seat and protocol as every other lane (per suite, only where a hybrid arm won pre-registration). Rethink — the same family's encoder arm, served by the hosted Rethink product (record-only: the encoder class is refused at serve, so it is measured on a subset of suites so far and its row discloses the partial coverage). Where the encoder's own read sits below the seated arm's, the row shows the served answer with the ↩ fallback mark and the encoder's read in the note (best-of-family); where it reads ahead, the violet record cell stands on its own measurement. openthai — the external OpenThai-SystemOne decision specialist (Apache-2.0, not affiliated, Qwen3.5-0.8B backbone): their FastAPI service answers the same decision questions, measured gold-label on our splits — a comparison lane, and the board's Thai-capability row. paw — the external ProgramAsWeights compile-a-classifier lane (MIT SDK, not affiliated): one neural program compiled per question shape from our committed spec, answered over their hosted REST — or through their local llama.cpp runtime as the paw (local) posture — a comparison lane, never a product lane (free-text answers under the exact-match law, so it carries no confidence columns). clef-flash (9B) · local — Cloudflare ships two Clef decision models (not affiliated): Clef (27B) and Clef-flash (9B). This lane is Clef-flash (9B), measured as their open weights rather than their hosted API: the community 4-bit quant (mlx-community/clef-flash-4bit) served on our box behind a loopback shim — a comparison lane, never a product lane. Clef (27B) is not on the board: its hosted posture is a future owner-gated cell.

CLM (Contrastive Language Models) is an external, unaffiliated research project — an Apache-2.0 "System One" decision model (Kwok, Kang, Suresh, Saad-Falcon, Pavone, Ré, Mirhoseini — github.com/Contrastive-LM/CLM · project blog). Architecture: a frozen Qwen3-8B encoder with last-token pooling plus ~20M-parameter state/action heads trained with a contrastive (InfoNCE) objective, so a softmax over state–action scores is the answer distribution. These rows use their reference head CLM-v0.1-8B (75 MB, on Hugging Face).

How we measure it: their stack serves, ours measures. The lane runs on the 4090 box — one vLLM Qwen3-8B pooling server plus clm-serve (the heads) behind their TypeSafe-compatible /v1/systemone endpoint — and our harness drives it with the same byte-identical questions as every other lane. Their prompt/prose rendering is byte-pinned against the reference implementation (goldens generated from their own Python at the pinned commit cca045ff), so the comparison is their rendering law, not our paraphrase of it. Latency figures include the HTTP round-trip, and their published "9× vs Jev" speedup — measured against Jev's remote endpoint, network included — deliberately does not carry into these tables. CLM is a comparison lane only: it is not part of Reflex and ships in no release; accuracy columns compare it exactly as they compare everyone else.

Some laya (rust) runs serve from the Apple Neural Engine — the encoder executes as one compiled whole-graph Core ML model per length bucket (BC1S, FP16), while the gather and the decision head stay host-side. FP16 math fails a bit-level drift bar by nature, so the ANE rows are gated the way every device row should be: decision-level — top-1 agreement against the frozen reference captures (gate ≥ 99.9%), and every argmax flip must sit inside the near-tie band (the reference's own top-2 margin < 0.04). Every flip, near-tie or not, is listed — never hidden. Probability drift is published as observation, never as a gate. Sequences longer than the largest bucket are served by the other devices and named as skips on the ANE row.

It is corpus-bound by design: out-of-domain text lands near chance, and those rows are published rather than trimmed. Its axes are latency and selective accuracy — how right it is on the questions it chooses to answer (acc@50cov, abstain / sel-acc) — not raw accuracy against a neural model.

Where the gap was the scorer rather than the domain, it closes. Per-label count tables (naive-Bayes log-odds over the train rows, plus a sentence-pair view for premise/hypothesis tasks) are cal-selected per suite, and the test split is read once. When count tables landed this took several suites to or past the laya lanes at the same sub-millisecond latency. 2026-10-01 correction: the modelless cells published 2026-09-28..30 were measured on a silently truncated 4,000-row-per-suite corpus pull while every posture they serve was selected on the full pools — the board read up to 42 points low on five suites through no engine change. Both hosts were re-run on the rebuilt full pools (every cell now carries a corpus digest; the board change log keeps the record), and the earlier numbers are restored — banking77 now reads above its count-table-era cell, a real engine gain. The test split never enters a corpus. A separate transductive line under each table shows what happens if the engine may also read the test text (self-labelled, gold never read): it adds nothing, because the train rows already carry the vocabulary.

They should be: the rust lane is a port of the same checkpoints and is checked against the python oracle's own outputs per suite. What differs is latency — and the python figure includes the subprocess round-trip (IPC).

The openthai model reads every option as prompt text — the input template inlines each option (<|ts_opt_i|> option: description) and one forward scores all 256 slots at the answer position, so per-question cost scales with option count. massive carries 59 options (~203 tokens/question) vs xnli's 3, and on MPS fp32 that prices at — per option (—/question, p50). The 4090 prices the identical prompts at — — same design, same accuracy (— on both hosts), — cheaper tokens. Every figure here is read from the massive_intent_en openthai cells in the table above. Verified as real, not box contamination: a preflight-passed exclusive-GPU re-read reproduced the number within 5% and made every @m3-max-metal openthai cell quotable from its own run.

The lanes are orders of magnitude apart. On a linear axis every modelless bar would be invisible; on the log axis each gridline is 10×, so bar length compares ratios, not differences. When a lane lands far past the readable range the axis breaks — the same slanted break sign the sizes chart uses: the log axis runs to 500 ms over most of the track, and a bar past the sign continues on a compressed scale, its exact value on the cell and in the tooltip.

acc@50cov is accuracy on the half of the questions the lane is most confident about. abstain / sel-acc is the fraction the calibrated gate declined to answer, and the accuracy on the rest.

G1 gates calibration honesty only: does the fitted confidence beat both the raw readout and the conformal-naive floor? It is not an accuracy gate. FAIL means the calibrator had nothing to improve; NO CLAIM means it never fitted (the calibration window was too thin) — neither means the answers got worse.

That lane did not run on that suite (for example, a suite that only the LLM lane can answer). A missing bar reads "not run", never zero.

No. Charts and tables are both drawn in your browser from data/bench.json, which is generated from the harness output. Re-publishing that file re-draws everything.

Retired (owner call, 2026-10-02): on the honest wide-eval populations (96–100 cases, template-disjoint) the modelless engine read at chance on them, so the suites were deleted from the reflex harness and the board rows went with them — the old small-suite rows were template-shared and read high (the verdict record: HISTORY.md). The quarantined caveat page that used to carry the lanes' record was removed in the same call; its data never entered this board's areas/cc/index math.