thetom-ai commited on
Commit
5167fff
·
verified ·
1 Parent(s): 4827165

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +5 -14
README.md CHANGED
@@ -16,12 +16,6 @@ inference: false
16
 
17
  # DeepSeek-V4-Flash — Config-I (GGUF)
18
 
19
- > 🔧 **KNOWN ISSUE, FIX IN PROGRESS (2026-08-03).** These files were quantized on a
20
- > branch where `Q2_0` carries ggml type ID **47**; the canonical TurboQuant fork uses
21
- > **42**. As a result the current files fail to load with an error like
22
- > `tensor 'blk.0.ffn_gate_inp.weight' has offset X, expected Y`. The tensor data is
23
- > fine, only the type field is wrong. Corrected files are being re-uploaded now.
24
- > **Please hold off downloading until this notice is removed.**
25
 
26
 
27
 
@@ -34,19 +28,16 @@ Config-I quantization of [deepseek-ai/DeepSeek-V4-Flash-0731](https://huggingfac
34
 
35
  ## ⚠️ Runtime requirement
36
 
37
- This GGUF uses fork-specific ggml types and **requires this exact branch**:
38
 
39
  ```bash
40
  git clone https://github.com/TheTom/llama-cpp-turboquant
41
- cd llama-cpp-turboquant
42
- git checkout tom/merge-upstream-dsv4
 
43
  ```
44
 
45
- Stock llama.cpp and earlier fork tips **cannot load it**. They lack both the `deepseek4` architecture and the type table this file was written against: the expert tensors are ggml type 47 (`Q2_0` in this branch), which older builds read as out of range, producing shifted tensor offsets and a load failure like `tensor 'blk.0.ffn_gate_inp.weight' has offset X, expected Y`. If you see that error, you are on the wrong build, not a corrupt download. Tracking PR: https://github.com/TheTom/llama-cpp-turboquant/pull/256
46
-
47
- **Metal (Mac) users:** run with `TQ_NO_ROTATE=1` in the environment. The fused rotate-act Metal kernel currently produces incorrect batched-prefill results on this architecture (decode is unaffected); the env var routes through the safe dequant path at a modest prefill-speed cost.
48
-
49
- **CUDA users:** use the `tom/merge-upstream-dsv4` branch above — earlier fork builds route TQ3_1S through a fused kernel that produces incorrect output on this architecture (fixed by routing through the verified dequant path; costs some decode speed, correctness first).
50
 
51
  ## Recipe
52
 
 
16
 
17
  # DeepSeek-V4-Flash — Config-I (GGUF)
18
 
 
 
 
 
 
 
19
 
20
 
21
 
 
28
 
29
  ## ⚠️ Runtime requirement
30
 
31
+ This GGUF uses the TurboQuant `TQ3_1S` weight type, so it needs the [TurboQuant llama.cpp fork](https://github.com/TheTom/llama-cpp-turboquant) rather than stock llama.cpp:
32
 
33
  ```bash
34
  git clone https://github.com/TheTom/llama-cpp-turboquant
35
+ cd llama-cpp-turboquant # default branch is what you want
36
+ cmake -B build -DGGML_CUDA=ON # or -DGGML_METAL=ON on a Mac
37
+ cmake --build build -j
38
  ```
39
 
40
+ Verified against the fork's default branch (`feature/turboquant-kv-cache`): loads and generates correctly on both Metal and CUDA.
 
 
 
 
41
 
42
  ## Recipe
43