Rivet-1B: A From-Scratch 1.09B Reasoning Model

Josh Angel · LinkedIn · Hugging Face August 2026

Released: josh-a/rivet-1b-pt (pretrained base, training logs, per-question evals, and the paper PDF this article is based on). Not released: the post-trained model (rivet-1b-it) and training datasets (rivet-sft, rivet-cpt).

---

Abstract

Rivet-1B is a 1.09B parameter reasoning language model trained entirely from scratch: custom tokenizer, custom pretraining corpus, custom post-training. No weights, embeddings, or tokenizer state were inherited from any existing model. It uses the Qwen3 architecture purely for serving compatibility and shares nothing with Qwen beyond the config schema.

The model was built as the reasoning core of a personal agent: it thinks in dedicated <think> blocks, drafts, rewrites, summarises, triages, and writes short code in a specific voice. Out-of-pocket spend was limited to a short burst of rented H100 compute for context extension and teacher-model API calls for distillation. The bulk of training, approximately 130B tokens of pretraining and continued pretraining, ran on owned 2x RTX 5090 hardware at zero marginal cost.

Against a 30-task held-out release bar, the released checkpoint (v10) scores 22/30: strong on directed task execution, identity, and short reasoning; weak on factual recall and multi-step arithmetic, which I show is a parameter-count ceiling rather than a data-mix problem. Needle-in-a-haystack retrieval is 100% at the native 16K context and at 64K via YaRN extrapolation.

This report documents the full pipeline, the data recipes, the eval methodology, and the engineering failures that shaped the result, including a silently 58%-empty SFT corpus, a KV-cache bug that invalidated an entire round of eval conclusions, and a multi-node experiment that halved throughput.

1. Why build a 1B from scratch

The practical question: how much useful capability can one person put into a model they fully own, on hardware they already have, for pocket change?

Fine-tuning an existing open model would have been cheaper and immediately stronger. It also would have meant inheriting someone else's tokenizer, data decisions, license posture, and unknown pretraining mixture. The goals here were different:

1. Full provenance. Every token the model saw is known and documented. For a personal agent trained partly on private communications, that matters.
2. A reasoning-native format. <think> and </think> are dedicated single tokens in the vocabulary (IDs 4 and 5), not strings learned post-hoc. Reasoning is part of the model's native grammar.
3. A specific voice and persona, baked into weights rather than simulated with ever-longer system prompts.
4. The learning. Running pretraining end-to-end surfaces failure modes you never see when fine-tuning: data-mix design, warmup dynamics, loss-plateau interpretation, context extension, and the long tail of infrastructure bugs documented in Section 8.

The model is named Rivet, after the agent it powers.

2. Architecture

Rivet-1B is a decoder-only transformer in the Qwen3 family: RMSNorm pre-norm, QK-Norm, grouped-query attention, SwiGLU, RoPE, tied embeddings.

Parameter split: 65.5M embedding (6%), 1,021M across 22 transformer layers (94%).

Design notes:

• QK-Norm (RMSNorm on queries and keys per head, before RoPE) for training stability, following OLMo 2 and Qwen 3. Cost: 5,632 parameters total.
• Tied embeddings as regularizer and parameter saver; standard at this scale (SmolLM2, Gemma).
• GQA 2:1 halves KV-cache memory at inference with negligible quality cost.
• Qwen3-family config so the trained weights export mechanically to Qwen3ForCausalLM and serve on vLLM, llama.cpp, Ollama, and transformers with zero custom code. The export is a ~250-line key-remap; the architecture match is exact (RMSNorm pre-norm + QK-Norm + GQA + SwiGLU + RoPE + tied embeddings).

3. Tokenizer

Custom SentencePiece BPE, 32,000 vocab, trained on 10M sentences sampled from the pretraining corpus, byte fallback enabled.

Special tokens are first-class vocabulary entries:

Two consequences worth noting:

• Because <think> is a single dedicated token, the model cannot "almost" start a reasoning block. Reasoning delimiters are atomic, which made SFT format compliance effectively 100%.
• Serving requires eos_token_id=[7, 2] (end-of-turn, then EOS) and care with skip_special_tokens, which strips the think tags by default. I serve bare ChatML with no reasoning parser and split on </think> client-side.

4. Pretraining

4.1 Corpus

The assembled pool is 387.5B unique tokens (722 GB tokenized, packed, uint16), ~99% public data plus a small slice of proprietary English data I built (<0.2%, not redistributed).

v7 pretraining corpus, 283.4B tokens:

v8 continued-pretraining increment, 104.1B tokens: multi-language code (69B, StarCoderData), books (15B, Project Gutenberg), reasoning (10B, OpenThoughts / OpenR1), conversations (9B, StackExchange + GitHub issues), Proprietary English v3 (0.38B).

Mix design followed published small-model recipes (SmolLM2, Phi, Hermes), validated against ablations rather than guessed. The full composition is documented below; raw bins are not re-uploaded since the sources are public.

4.2 Runs

Training used a custom RivetForCausalLM wrapper under the HuggingFace Trainer with a WSD schedule (warmup 2% / stable 78% / decay 20%), AdamW, bf16, gradient checkpointing, sequence packing, DDP.

Narrative arc:

• v6 was deliberately treated as a pilot: 6.3B tokens (~5.8x params) is far below Chinchilla-optimal, and the model was predictably undertrained. Its value was as a warm-start init: v7's first eval at step 5K was 1.898, where a from-zero run sits at 6-7.
• v7 was the main event: 1,220,703 steps at ~24K tokens/sec aggregate (3.4 s/step, both 5090s pinned ~97%), seven weeks of continuous uptime. Eval loss descended through a series of plateaus (1.92 at 148K steps, 1.85 at 295K) with the LR decay phase delivering the final drop to 1.598.
• v8 CPT extended context from 2048 to 16K via RoPE ABF (theta 100K to 1e6) on a long-document mix. This was the one rented-compute run: 2x H100 on Brev. (Its ~0.97 eval loss is measured on the long-doc eval set and is not comparable to the v7/v9 figures.)
• v9 warm-started v8 and sampled +20B tokens from the pool reweighted toward reasoning/math/code. This measurably raised the knowledge floor: base-completion probes went from garbage ("capital of New Zealand" completed to "£100,000,000") to correct ("Wellington"), and stayed correct through SFT.

Throughput anchor: 24K tokens/sec aggregate (~2B tokens/day) for the 1B on 2x 5090.

4.3 What pretraining at this scale actually teaches

A consistent pattern across checkpoint A/B probes (seeds pinned, greedy and sampled): the model learns to say something English long before it learns to say something correct. Broken-output rates dropped steadily from step 148K to 324K while factual accuracy stayed flat, then jumped only after tens of billions more tokens. Eval loss was the only monotonic signal; greedy spot-checks of individual facts were anecdote-grade noise on the ±5K-step scale.

5. Context extension

Native context is 16,384 tokens, trained via RoPE ABF (theta raised 100K to 1e6) during the v8 CPT run on long documents.

Validation: needle-in-a-haystack retrieval is 100% at 16K native and at 64K via YaRN (factor 4 extrapolation at inference, no further training).

One operational gotcha worth recording: checkpoints repeatedly saved a stale max_position_embeddings=2048 from an early config bug. The fingerprint of a correct checkpoint is rope_theta=1e6 with max_position_embeddings=16384; I now verify both on every export.

6. Post-training

6.1 Format and masking

SFT data is ShareGPT-format conversations. Every assistant turn is a <think> block (real planning, capped short) followed by the finished artifact. Loss is computed on assistant tokens only (prompt tokens masked to -100), multi-turn supported with each assistant turn unmasked individually. Chat rendering is bare ChatML through the HF tokenizer's apply_chat_template, so SFT tokenization is bit-identical to what vLLM and llama.cpp produce at inference.

6.2 The v10 corpus

The released model's SFT corpus is 42,066 weight-expanded rows from 33,477 unique:

A Rivet system prompt is baked into ~70% of rows so the persona lives in the weights; 30% are system-free so the model still behaves bare. Task signal is ~21% of the final mix.

The distilled task-completion set was purpose-built: ~4,650 prompts across seven task clusters, answered by Claude Sonnet under an instruction enforcing direct voice, no customer-service filler, finished artifacts only, and short substantive think traces. Earlier distillation rounds produced the "apex" reasoning tier: 3,155 examples across 27 categories with real extended-thinking traces from Claude Opus 4.6 via the Batch API.

6.3 What the data iterations taught

The path from SFT v8 to v10 was driven by eval failures, and each fix was a data composition fix, not a hyperparameter fix:

• v8 scored eval_loss 0.251 but felt flat. Root cause found later: the corpus it trained on was 58% empty rows (124,955 of 214,351), written as structural placeholders by a reformatting bug. They passed the loader's len(convs) < 2 check because empty rows still had two structural turns. Validation must check non-empty content, not structure.
• v9-SFT scored 10/30 on the release bar. Failure clusters: task-execution ramble (meta-commentary instead of artifacts), verbose think traces eating the answer budget, word-problem math collapse, hallucinated context, and a hard safety fail (complied with a phishing-email request).
• v10 cut generic instruction filler (alpaca, helpdesk-register data) that taught deflection, added the distilled task-completion set, capped think length in the data, and baked the persona into 70% of rows. Result: 22/30, with task-execution, identity, and short reasoning all passing.
• v11 attempted to recover factual recall by rebalancing the mix toward knowledge data. It scored 19/30 and facts got worse. Conclusion: factual weakness at 1B is a parameter ceiling, not an SFT-mix problem. v11 was discarded; v10 stands.

The refusal question was decided explicitly: the v10 corpus is unfiltered. A 40-example refusal set was built and then deliberately excluded. Rivet is a personal model that attempts any request; the deployer owns safety. This is called out on the model card in the first screen of text.

7. Evaluation

7.1 Release bar

The primary gate is a 30-task held-out battery (release_bar.jsonl) spanning identity, drafting, rewriting, summarising, triage, code, reasoning, facts, multi-turn recall, format, voice, and refusal behavior. Scored greedy, temperature 0, with the production system prompt.

v10 passes: identity (3/3), all drafting/rewriting/summarising tasks, triage, short code (fib, bash), single-step arithmetic, decisions, format, multi-turn recall, voice.

v10 fails: factual recall (NZ capital answered "Canterbury"), multi-step word problems (arrival-time, work-rate), one tool-explanation task, and two cutoff-adjacent tasks.

7.2 Base-model knowledge floor

Continued pretraining's effect was verified directly on base completions (no chat template): "The capital of New Zealand is" went from numeric garbage (v8) to "Wellington" (v9). This matters because it isolates pretraining gains from SFT presentation gains.

7.3 GSM8K

On the full GSM8K test set (1,319 problems, greedy decoding, chat format with the production system prompt, answers extracted after </think>, with any unparseable generation re-run individually to rule out batching artifacts), the released checkpoint achieves 12.7% (168/1319). Per-question results ship with the model (evals/gsm8k_v10_results.jsonl in the repo). This sits in the expected band for a 1B model trained on 130B tokens, and is consistent with the release bar's weakest cluster (multi-step word problems). GSM8K measures precisely the capability this model is not built for; the release bar in 7.1 measures the one it is.

Set against published 1B-class baselines, the number reads differently:

Every model run by me under one uniform harness on a fixed 300-problem test subset (seed 42), 0-shot, no few-shot examples. Greedy: single deterministic generation. pass@4: correct within four temp-0.7 samples. Per-question results for every model in evals/. Published baseline figures (Pythia ~2%, TinyLlama 1.4% 5-shot) are consistent with these runs. Qwen2.5-1.5B (math-saturated 18T-token corpus) sits well above the field and is shown for completeness; one third-party report of Qwen2.5-1.5B base at 2.1% zero-shot is a harness artifact and did not reproduce. Rivet-1B additionally scores 12.7% greedy on the full 1,319-problem test set.

Contamination disclosure. Rivet-1B's SFT corpus contains 2,635 rows drawn from the GSM8K train split (verified by exact and prefix match over all 41,696 SFT rows), about 6% of the mix. Training on the train split is standard practice; the test split was verified absent (zero matches), so the scores above measure generalization from train to test in the conventional way. I state this so nobody has to catch it.

Variance study (same 300-problem subset; per-condition results in evals/gsm8k_variance_results.jsonl): a single greedy number understates the model. Four samples at temp 0.7 give avg@1 11.2% but pass@4 25.7%; removing the system prompt changes nothing (13.3% greedy); removing the #### format instruction nearly doubles the score to 21.0%, because unconstrained generations run their think trace to completion and end with a natural-language answer, while the format instruction truncates or derails a third of generations. Reading: the point estimate is stable, the ceiling is higher than any single greedy run shows, and the format instruction is a liability on math, worth fixing in the next SFT round.

7.4 Long context

NIAH: 100% retrieval at 16K native, 100% at 64K under YaRN (factor 4).

7.4 Honest summary

Rivet-1B is sharp at doing: given a task, it produces a clean artifact in the right voice with a visible reasoning trace. It is modest at knowing: at 1.09B parameters it is not a knowledge base and will confidently produce plausible-wrong facts. The eval program's job was to make that tradeoff explicit and measured rather than vibes.

8. Engineering failures that shaped the result

The most useful part of this report for anyone replicating the path:

1. The 58%-empty SFT corpus. A reformat step wrote {"conversations": []} placeholders for rows it couldn't convert. The loader's structural check passed them. An entire SFT round trained on it. Fix: validate content, not structure; count empty rows in the build script and fail loudly.
2. The KV-cache bug that invalidated a month of conclusions. The custom model's forward() returned CausalLMOutput instead of CausalLMOutputWithPast, breaking HF generate()'s cache contract. Output looked like garbage, and "SFT is broken" was the working theory until manual greedy decoding produced fluent reasoning. The model was fine; the decode path was not. Rather than retrofit KV cache through every custom attention layer, I exported to standard Qwen3 format and deleted the problem.
3. The tied-weights safetensors crash. transformers v5 requires _tied_weights_keys declared (as a dict; v4 wanted a list) plus post_init() and config mirroring of tie_word_embeddings. First checkpoint save crashed at step 2000 on rented H100s. A lesson paid for in rented compute.
4. Multi-node is not free. 4x 5090 across two machines over 10 GbE ran at half the throughput of single-node 2x 5090: bf16 gradient all-reduce of ~2 GB/step over TCP dominated. At 1B params, compute does not outweigh comms until ~100 GbE interconnects. Killed the experiment, stayed single-node.
5. Infinite eval. An IterableDataset with while True: yield was reused for eval; the Trainer iterated faithfully forever. First eval ran 4.5 hours before being killed. Fix: materialize a fixed eval subset.
6. Loss logging inflation. transformers v5 logs train loss multiplied by grad_accum x world_size (20x in my config). A run that looked broken (loss "187") was healthy (~9.3). Fixed with a Trainer.log() override.
7. Stale config on save. max_position_embeddings=2048 persisted into 16K-trained checkpoints. Verify config fingerprints on every checkpoint, not just weights.

None of these are exotic. All of them cost real time, and none appear in fine-tuning-a-Llama tutorials because that path never exercises them.

9. Cost

I deliberately don't quote a headline dollar figure. Any such number is arbitrary: it depends entirely on how you account for owned hardware, power, and time, and every convention produces a different answer. What I can state precisely is the shape of the spend. The dominant cost was patience, not money: roughly 130B tokens of pretraining and continued pretraining, plus every SFT round, ran on owned GPUs (2x RTX 5090, already paid for) at zero marginal cost over several months. Cash spend was limited to the two things owned hardware couldn't provide: a few days of rented H100 capacity for the 16K context-extension run, and teacher-model API calls for the distillation sets. Both were small relative to the compute they unlocked.

10. Limitations

• Facts are unreliable. This is the 1B parameter ceiling, demonstrated experimentally (Section 6.3, v11). Do not use Rivet-1B as a knowledge source.
• Multi-step arithmetic and word problems are unreliable.
• Unfiltered. No refusal or safety training, by design. The deployer owns that layer.
• Personal-data provenance. SFT included the author's own writing (voice/persona data). That persona lives only in the post-trained model, which is not released. The released base model was never exposed to it.
• English-only, by corpus design.

11. What's next

The 1B is released and frozen. Current work is a ~3B growth model: block expansion (LLaMA-Pro-style, function-preserving) carrying the trained 1B forward, then continued pretraining on the 387.5B-token pool already on disk, then re-SFT. The motivation is exactly the v11 lesson: facts and multi-step reasoning are parameter-limited, and block expansion inherits ~130B tokens of training instead of starting over.

That model is unreleased and not covered by this report.

12. Reproduce

• Pretrained base weights + model card + raw TensorBoard logs (tensorboard/) + GSM8K results (evals/): josh-a/rivet-1b-pt
• The post-trained model (rivet-1b-it) and training datasets (rivet-sft, rivet-cpt) are not released; their composition is documented in this report.

Serving quickstart is on the model card. The two non-obvious requirements: eos_token_id=[7, 2], and handle <think> as special tokens (no reasoning parser; split on </think> yourself).

---

Trained in Auckland, New Zealand.