The Neutrino Engine is the runtime behind every Neutrino model: one path from a ternary-family artifact to a token stream, on datacenter GPUs, Apple silicon, and desktop x86. Two constraints govern it. Output must be token-identical to a full-precision reference, and no part of the speed the format buys is lost between the artifact and the silicon. This post covers the budget a token spends, the certification system that gates every build, the drafted path that takes an H100 from 396 to 763 tokens per second, and the record on other stacks’ artifacts.
1. Where a token’s time goes
Single-stream decode has one dominant invariant. Producing a token requires reading essentially every weight in the model once: each of the 36 decoder layers multiplies the current token’s activations against its attention and MLP matrices, and at batch size one each weight participates in a single multiply-accumulate before it is next needed a full token later. Bytes per token divided by memory bandwidth is a hard floor on the time a token can take, and the floor is the one part of the problem no kernel negotiates with. Shrinking the artifact lowers that floor on every backend at once. Neutrino-1 8B ships as one file of 3.88 GB against roughly 16 GB for the same weights at fp16, so its floor sits four times lower before a line of kernel code runs.
Above that floor, batch-one decode is a latency problem, and we measured exactly where the latency lives. Undrafted decode at 396 tokens per second on an H100 80 GB moves about 1.5 TB of weight traffic per second, under what the part can supply. A study of the same runtime lane on a 24-layer model, on an NVIDIA A100, resolved the step: the layers’ pure DRAM floor was 104 microseconds per token, and the layers cost 590 microseconds beyond it, spent across about 120 kernel boundaries at about a microsecond of launch each plus each small kernel’s internal dependency chain. Collapsing the whole step into a single megakernel returns none of that 590: a software barrier inside a fused kernel costs what the graph launch it replaces costs, so the time belongs to the dependency chains and not to the dispatch. That is a property of the part, and it is why the engine spends its effort on the byte floor instead of on kernel count. One stage behaves differently. The unembedding, which projects the final hidden state onto the full vocabulary, runs at the DRAM roofline and accounts for 43 percent of all DRAM traffic in the step.
Those two facts assign the engine two jobs. On CPUs the byte floor is the whole fight: the M5 path at 24.9 tokens per second moves about 97 GB/s through a laptop memory system, which is what the machine has, so the CPU backend sits on the floor and the format’s four-times reduction converts to rate directly. On GPUs the fight is the distance between the floor and the measured rate, and lowering the floor is what makes that distance worth closing. Either way the serving path protects the floor first: no floating-point shadow copy of the weights, no conversion at decode time, no tensor read twice for one token.
24-layer model, NVIDIA A100, same runtime lane, batch one. The vertical rules inside the second segment are the kernel boundaries.
2. The artifact as a memory object
Bytes on disk and bytes per token are different numbers, and the artifact’s composition is where they part. Of the 3.88 GB file, 2.60 GB, 67.2 percent, is the ternary weight lane: the q, k, v, o, gate, up and down projections of all 36 layers, and this lane streams in full on every token. Another 1.24 GB, 32.1 percent, is two untied int8 embedding tables of 151,936 by 4,096, touched a row at a time. Metadata, row scales and norm vectors take 26 MB between them. The weight lane divides into 36 identical slabs of 72,351,744 bytes, each split 21.7 percent attention and 78.3 percent MLP, so at short context the MLP projections carry most of a token’s traffic and attention is the minority term. The balance moves with scale: embeddings are 32.1 percent of the 8B file and 47.5 percent of the 0.6B’s, where the vocabulary dominates the download.
3. The cache in the budget
The weights are the static half of the budget. Attention reads the KV cache on every step, so each token of context adds bytes that every later token moves again. At the 8B’s geometry, 36 layers of grouped-query attention with 8 KV heads of dimension 128, a cached position costs 147,456 bytes at 16 bits per element: 144 KiB of keys and values that join the per-token traffic for the rest of the session. Grouped-query attention is doing budget work here, since caching all 32 query heads would cost four times as much.
Cache growth is linear and indifferent to the weight format: 0.60 GB at 4,096 tokens of context, 2.42 GB at 16,384, 4.83 GB at 32,768, 6.04 GB at the model’s full 40,960 window. Past roughly 26,000 tokens the cache outweighs the 3.88 GB artifact, and the object the engine streams each token stops being mostly the model. A format four times smaller than fp16 moves that crossover earlier, which is why cache traffic is a first-class term in the engine’s budget and gets scheduled like one.
4. One artifact, every backend
The same file serves four backends: H100-class GPUs through CUDA, the Apple M5’s GPU, the M5’s CPU cores, and desktop x86. The bytes are the same bytes on all four. There is no per-platform re-export, no quality tier, no variant with different numerics. The artifact that ships is the artifact that was evaluated, so a rate measured on any backend describes the same model the benchmark battery scored.
What stays identical is everything that defines the model: the stored weight values, the computation they imply, and the output token stream under greedy decode. What differs is how each machine is fed. Dispatch differs, since work is issued to a datacenter GPU, to an integrated GPU, and to a pool of CPU threads through three mechanisms with three launch costs. Parallelism differs, since the traversal is divided among thousands of GPU threads in one case and a handful of performance cores in another. The memory hierarchy differs most: HBM on the datacenter part, unified memory on Apple silicon, DDR behind deep cache hierarchies on desktop x86, each rewarding a different access pattern. The loader resolves those differences once, at load time, arranging the same weights for the device in 1.9 seconds for the 8B on a datacenter GPU and 0.3 seconds for the 0.6B, without changing a single value. The kernels, their tiling and their instruction selection are the sealed part of the stack.
The rates span a factor of thirty. Neutrino-1 8B decodes at 763 tokens per second drafted and 396 plain on an H100 80 GB, at 33.7on the optimized base-M5 path and 24.9 on its CPU cores, and at 30.7 through our llama.cpp fork on an NVIDIA L4 inside 4.68 GiB of VRAM. Neutrino-1 0.6B clears 200 tokens per second on that laptop’s CPU alone.
One file across all five rows. The drafted rate is the same artifact with the 0.6B running ahead of it.
5. Token identity as a release gate
The engine’s release gate is a comparison, not a benchmark. A candidate build decodes a fixed prompt battery under greedy decoding, and its token stream is checked position by position against a stored reference trajectory from the full-precision implementation. The check is binary per position and the gate is absolute: one divergent token anywhere in the battery fails the build, whatever the speed gain that produced it. A stack that emits different tokens is serving a different model, and every quality number published for the original stops applying to it. Speed that changes the answer is not speed.
The gate runs at every boundary the model crosses. Engine builds are compared against the full-precision reference. The drafted path is compared against the undrafted path on the same artifact. Converted packs are compared against the engine’s own trajectory before release, and imported foreign artifacts are compared against their reference implementations before any race is scored. A change that cannot hold identity does not ship, and its speed is never recorded.
6. Drafted decode, certified
Speculative decoding is the engine’s most invasive optimization and the reason the gate exists in its strictest form. The mechanics are a three-phase loop. Neutrino-1 0.6B runs ahead as a draft and commits k candidate tokens, and k is not fixed: the controller sets it cycle by cycle from recent acceptance. The 8B then scores all k positions in a single verify pass costing one decode step, one traversal of the weights and one trip through the launch chain, whichever bound is binding. The engine keeps the longest prefix on which the draft’s tokens exactly match what the 8B would have chosen, replaces the first disagreement with the 8B’s own token, and resumes drafting after it. Every emitted token is the 8B’s choice by construction, so the draft changes where the bytes move and never what comes out.
- 01The 0.6B draftsOne to seven candidate tokens, length set from recent acceptance.
- 02The 8B verifies onceEvery drafted position scored in a single batched forward pass.
- 03The agreed prefix is keptThe longest run where the draft chose what the 8B would have chosen.
- 04The 8B closes the cycleThe first disagreement is replaced by the 8B's own token.
The economics come from the size ratio. The draft artifact is a twelfth of the 8B’s bytes, so six draft steps and one verify traversal move a fifth of the bytes seven full steps would, and the cycle wins whenever the accepted prefix amortizes the draft run. The cycle’s floor is one full step plus a small draft overhead, which is why every measured prompt class lands above the undrafted rate. Construction is not certification: the drafted path was certified against the undrafted path by direct comparison on the shipping artifact, 27,648 consecutive tokens across the prompt battery, zero divergences, before any drafted rate was recorded.
Acceptance length sets the rate, and acceptance length is a property of the text rather than of the engine: it is the average number of next tokens a 596M-parameter model gets exactly right before the 8B would have chosen differently. Counting and enumeration are the most determined continuations in the battery, and the shipping configuration accepted 100 percent of drafted tokens there in a fixed six-token probe, 74 verify passes for 512 tokens, about seven tokens emitted per 8B forward pass, for 763 tok/s against 395.9 undrafted. Factual recall holds 96.5 percent of drafted tokens, 5.8 of six per pass, at 613. Prose runs at 532. Open-ended chat and code are the least determined continuations in the battery, where each token carries more of the writer’s choice, and both still clear the undrafted rate on the same machine.
Rings mark 200, 400, 600 and 800 tok/s; the dashed ring is 395.9 tok/s of undrafted decode on the same H100 80 GB. Clockwise from counting at the top.
The controller reads that property per class and sizes the draft to it. Short factual completions hold 97 to 99 percent acceptance however far the draft runs, so the controller rides long drafts there, committing five and a half tokens per verify pass and keeping 5.49 of them. Code and adversarial counting resolve their exact prefix in one or two tokens, so it commits one or two and turns the pass over: 181 and 200 verify passes per 512 tokens, each one still cheaper than a full traversal per token. The policy is a single rule applied to nine different texts, and every class it produces sits above the undrafted rate.
7. Drafting on a laptop
The drafted path runs on Apple silicon, and the first thing it holds there is identity. On an Apple M5 MacBook with 16 GB, both models resident in one process under a hard 6 GiB cap, the 8B and the 0.6B draft produced token-identical output to plain decode on 6 prompts of 6. Drafting on a Mac changes the speed and never the text. On factual prompts it also changes the speed: 22.00 tokens per second plain becomes 25.71 drafted, a gain of 1.168 times at 74.4 percent acceptance, medians of three 512-token greedy runs at k = 4 with peak memory of 4.3 GiB. The cadence is what the mechanism predicts. Three of four drafted tokens survive each pass, the 8B contributes its own, and 3.98 tokens land per verify pass instead of one per traversal.
Each step is one verify pass. The dashed line is the same machine without a draft model, 512-token greedy runs under a 6 GiB cap.
8. The cross-stack record
The engine is also raced on other stacks’ public artifacts, and the protocol is fixed. Both stacks run on the same machine in the same session with rounds interleaved, so thermal state and clock drift land on both sides equally, and a token-identity check against the reference implementation’s output runs before any rate counts. Import is gated the same way: a foreign artifact is translated into our container and every tensor is verified bit-exact against the source file before the model runs a single round. What follows is measured under that protocol, on weights we did not train.
On a public 27B ternary model, the engine decodes at 105.15 tokens per second against the reference stack’s 97.80 on the same machine, faster in six rounds out of six, our slowest round above their fastest. It does so while reading the model from a file 19 percent smaller than the published artifact, and the saving is the container rather than the weights, the same information stored closer to its true size. A 2B ternary model from a major lab runs at 102.4 tokens per second in our engine on an Apple M5 against 89.0 in its own reference runtime, three same-moment rounds on one laptop with the throttling shared, and nothing faster has been published for that model on Apple silicon by anyone. Their weights, their machine class, our runtime. On a datacenter part the same model runs 1.77 times faster in our kernels than in the lab’s own GPU kernels on one A100. On desktop x86, against a published lookup-table stack on its home instruction set, 2.12 times. At the single-token level, the engine’s GEMV runs 12 to 13 percent ahead of an int4 GEMV kernel at the batch size where decode actually lives.
Their fastest round is 98.12 and our slowest is 102.65, so the two sets of six never overlap, and ours reads the model from a file 19 percent smaller than the published artifact.
Faster in six rounds of six, our slowest round above their fastest, read from a file 19 percent smaller than the published artifact.
No faster single-stream rate has been published for that model on Apple silicon by anyone, including its authors.
The weights are theirs and unchanged; the traversal is ours.
Measured on the instruction set that stack was written for.
The batch size decode actually lives at, on the operation that dominates it.
9. Distribution surfaces
The engine installs from pip and serves the released artifacts directly; it is the stack every number in this post was measured on. Two further surfaces carry the same artifact into the two ecosystems with their own runtimes. A GGUF pack targets the llama.cpp ecosystem, together with our CUDA-enabled fork of llama.cpp, which decodes the 8B at 30.7 tokens per second on an NVIDIA L4 with full GPU offload, in 4.68 GiB of VRAM at 4,096 context, inside the budget of an 8 GB card, and whose output is token-identity gated like everything else carrying the model’s name. An MLX pack covers the Apple GPU path and ships with the same gate.
