🏗️ Building on HF

witcheer

15 4 1

https://x.com/witcheer

AI & ML interests

Local AI Maxxing.

Recent Activity

updated a dataset 1 day ago

witcheer/rtx-5090-benchmarks

liked a model 1 day ago

Qwen/Qwen3.8-27B

updated a model 5 days ago

witcheer/gemma-3-27b-eagle3-drafter

View all activity

Organizations

posted an update 3 months ago

Post

274

new dataset: which local LLM best *drives an agent*? benchmarked 4 models for pairing with Hermes Agent (@NousResearch ) - a CodeAct agent that writes python to call its tools. RTX 5090, llama.cpp. two phases, hybrid:

>>> phase A (synthetic): scored 4 axes — code-as-action, long-context, instruction-following under Hermes' real ~3.5K-token prompt, multi-step loops. top was a near-tie (within noise): an 18B frankenmerge (Qwopus) edged Qwen3.6-27B, and Hermes' own 36B came LAST.

>>> phase B (real harness): installed Hermes, ran the top 3 through 14 multi-step tasks x3 repeats. the tie broke — and an efficiency gap appeared:

Qwen3.6-27B    100%   | 3.0 turns | 364 tok
Qwopus-18B     85.7%  | 3.6 turns | 870 tok
Nemotron-30B   85.7%  | 4.4 turns | 1334 tok

Qwen is perfect AND 2.4-3.7x more token-efficient — something a synthetic test can't see (only the real agent loop can). verdict: Qwen3.6-27B for local Hermes.

dataset: witcheer/hermes-pairing-bench
collection: witcheer/rtx-5090-benchmark-rig-6a17e365b534abb474250e11

posted an update 3 months ago

Post

333

new dataset: turboquant KV cache benchmarks for qwen3.6-35B-A3B on RTX 4060 Ti 8GB.

>>> 18 structured runs covering turboquant turbo2/turbo3/turbo4 vs standard q4_0 V cache, context depths 3.5K to 50K, checkpoint modes, and two agent harnesses (hermes vs pi).

>>> novel finding: llama.cpp default context checkpoints (every 8192 tokens, ~63 MiB each) silently accumulate in VRAM and trigger the 8GB cliff at ~26K context. disabling with --checkpoint-every-n-tokens -1 gives smooth degradation instead.

turbo3, 64K ctx: 35.17 tok/s (62 graph splits)
turbo2, 64K ctx: 35.13 tok/s (62 graph splits)
turbo4, 64K ctx: 13.93 tok/s (decompression cliff)
std q4_0, 64K ctx: 31.22 tok/s (82 splits, broken in agents)

third dataset in the collection. now 3 datasets covering dense, MoE offload, and turboquant benchmarks.

[dataset]( witcheer/rtx-4060ti-8gb-turboquant-bench-2026-05) | [collection]( witcheer/8gb-vram-local-llms-practitioner-tested-69fa0e855c51e3c15a9d95d4)

posted an update 3 months ago

Post

164

updated my MoE offload bench dataset + collection.

>>> previous finding: Qwen3.6-35B-A3B via full expert offload on RTX 4060 Ti 8GB + 32GB RAM → 7.4 tok/sec. RAM-ceilinged, disk-bound.

>>> new finding: built llama.cpp from source inside WSL2, swept -ncmoe values for partial offload.

ncmoe 32, 16K ctx → 29.7 tok/sec
ncmoe 30, 16K ctx → 32.0 tok/sec
ncmoe 30, 32K ctx → 35.4 tok/sec
ncmoe 28, 16K ctx → 16.3 tok/sec (VRAM cliff)
ncmoe 30, 65K ctx → 17.4 tok/sec (VRAM cliff)

4.8x faster than full offload. 8GB VRAM cliff is sharp - crossing ~7 GB halves throughput instantly.

the hybrid SSM+attention architecture means 32K context is nearly free (KV cache only scales for 10/40 layers).

dataset: witcheer/windows-rtx-4060ti-8gb-moe-offload-bench-2026-05

collection: https://hf.co/collections/witcheer/8gb-vram-local-llms-practitioner-tested

1 reply

reacted to danielhanchen's post with 🔥 3 months ago

Post

8948

We made a guide on how to run open LLMs in Claude Code, Codex and OpenClaw.

Use Gemma 4 and Qwen3.6 GGUFs for local agentic coding on 24GB RAM

Run with self-healing tool calls, code execution, web search via the Unsloth API endpoint and llama.cpp

Guide: https://unsloth.ai/docs/basics/api

witcheer

AI & ML interests

Recent Activity

Organizations

witcheer's activity