Documentation.Documentation.

$ pip install fermion-research
$ fermion chat

The package is fermion on PyPI and the default artifact is fermionresearch/Neutrino-8B. The first fermion chat runs a fixed sequence before the prompt appears. Every later run skips straight to the load.

1. 1

   Fetch

   The downloader pulls the 2.56 GB coded transport for Neutrino-1 8B from the release repository into the standard Hugging Face cache.

2. 2

   Expand

   The transport decodes losslessly to the 3.88 GB serving artifact. One file; there is nothing to assemble.

3. 3

   Verify

   The engine computes the artifact's SHA-256 and compares it against the digest published with the release. A mismatch aborts the load with both digests printed.

4. 4

   Bind a backend

   The engine probes the host and binds the prebuilt native runtime published for it, compiled CPU binaries for macOS arm64 and Linux x86-64, with a bit-exact torch reference path underneath. fermion info names the backend in use, and --backend pins it.

Then the prompt opens and tokens stream. At the end of each session the engine prints its steady-state decode rate for that machine.

What the engine maps is almost entirely two lanes: the ternary weight lane holding every projection of the 36 decoder layers, uniform at 72.4 MB per layer, and the int8 token-embedding table. Scales, norms, and the header account for the remaining 0.7 percent. The 2.56 GB transport is this same file, coded tighter for the wire.

Upstream llama.cpp runs the 8B GGUF pack on CPU. Our fork adds the CUDA kernels for this weight format, so the whole model offloads to the GPU. The pack itself ships in the 8B repository under gguf/. Build the fork with CUDA enabled:

$ git clone https://github.com/fermionresearch/llama.cpp && cd llama.cpp
$ git checkout fermion-fv5
$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DGGML_CUDA=ON
$ cmake --build build -j --target llama-completion llama-bench

Then run the pack with every layer offloaded:

$ ./build/bin/llama-completion -m neutrino-8b-fv5.gguf -ngl 99 -c 4096 -p "Explain why the sky is blue." -n 256 --temp 0 -no-cnv

-ngl 99 offloads all layers; -c 4096 is the context length the published number is measured at. In that configuration the 8B pack decodes at 30.7 tokens per second on an NVIDIA L4 and holds 4.68 GiB of VRAM, comfortably inside an 8 GB card.

Decode rates are measured single stream: one prompt, one reply, batch of one, greedy decoding, steady state over at least 512 generated tokens. The engine prints exactly this number at the end of every session, so a session on your machine is already a run of the protocol.

On the GGUF path, the fork carries llama.cpp’s own benchmark tool, which separates prefill from decode:

$ ./build/bin/llama-bench -m neutrino-8b-fv5.gguf -ngl 99 -p 512 -n 128

The published L4 number is the tg128 row, the generation row. Prefill rates run far higher and are not comparable; quoting one as the other is the most common way speed claims go wrong.

For the accuracy battery, each entry on the model card states the suite version, prompt format, shot count, and answer extraction rule. Those four parameters define the number; run any public evaluation suite with the same four against the shipping artifact.