llama.cpp TurboQuant: Is It Supported? What Works Today (2026)

Last updated: November 9, 2026. By Veer. Status checked on November 9, 2026 against the llama.cpp GitHub repository (source at commit 7d7f0287, November 8), Google Research’s TurboQuant post, the vLLM v0.20.0 release notes and Ollama’s FAQ. This is a researched explainer built from public sources, not a hands-on test.

If you searched “llama.cpp turboquant,” you probably saw a Reddit thread claiming a huge memory saving and a flag you can’t find in your build. That’s expected. TurboQuant is a Google Research method for compressing the key-value (KV) cache, and llama.cpp’s maintainers have declined to merge it so far. This guide covers what TurboQuant is, what the llama.cpp project actually decided, where it does run, and what to use instead.

Does llama.cpp support TurboQuant?

No. Official llama.cpp builds have no TurboQuant cache type. The current common/arg.cpp defines the allowed KV cache types as f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1. Pass anything else, such as turbo3, and llama.cpp stops with “Unsupported cache type.”

Here is the paper trail on GitHub, with dates.

Maintainer Johannes Gaessler closed the CPU pull request with this explanation: “so far no one has presented evidence that would justify the opportunity cost of adding TurboQuant support.” A day later, maintainer pwilkin wrote that anyone with “a feature showing demonstrable gains with some quants compared to equivalent quantization types already in the main repo” should open a new PR “with full benchmarks as indicated in the contributing guidelines.”

So the door isn’t welded shut. It just needs evidence that TurboQuant beats the existing types at the same bit width. Nobody has cleared that bar in mainline yet.

What is TurboQuant?

TurboQuant is a vector-quantization algorithm from Google Research that compresses the KV cache, the memory an LLM uses to remember your conversation. It does not shrink the model weights. Google announced it on March 24, 2026, with the paper (arXiv 2504.19874) presented at ICLR 2026.

Google’s post says TurboQuant first randomly rotates the vectors, then uses one extra bit for an error-correction step called QJL. Its reported results, tested on Gemma and Mistral:

• KV cache quantized to 3 bits without training or fine-tuning, with no accuracy loss in Google’s tests.
• KV memory reduced by “a factor of at least 6x” on needle-in-a-haystack tasks.
• Up to 8x faster attention-logit computation with 4-bit TurboQuant versus 32-bit unquantized keys on H100 GPUs.

Those are Google’s numbers on Google’s setup, not a measurement of llama.cpp on your PC. The gap between “works in the paper” and “worth merging in llama.cpp” is exactly what the maintainers are arguing about.

Where does TurboQuant work today?

TurboQuant works in some community llama.cpp forks and in vLLM, but not in mainline llama.cpp or Ollama. Here is the current picture.

On the forks: TheTom’s repository has about 2,445 stars, with its last push on October 22, 2026. The buun fork has about 906 stars and was pushed on November 8, 2026, and describes itself as an “Experimental llama.cpp fork for inference research and development.” We haven’t reviewed either codebase. Forks are maintained by one or two people, aren’t reviewed by the upstream team, and can fall behind llama.cpp’s fast-moving releases, so treat them as experiments.

If you only use mainline llama.cpp or Ollama, there is nothing to switch on. For related setup choices, see our llama.cpp vs Ollama vs LM Studio guide.

Is TurboQuant better than q4_0 or q8_0 in llama.cpp?

Not at 4 bits, according to the most detailed independent benchmark we found. It only pulls ahead at 2 to 3 bits, where the built-in types don’t go. That’s also why the llama.cpp maintainers wanted same-bit-width comparisons.

The test is from developer Anbeeld, linked in the June 2026 GitHub thread, where a maintainer called it the kind of analysis he expects. It ran Qwen 3.6 27B (Q5_K_S weights, 64k context) on one RTX 3090, using the author’s own fork called BeeLlama. These are the author’s figures. The “99.9% precision” column measures the worst 0.1% of token positions against a bf16 cache, where higher is better.

Source: Anbeeld, KV Cache Quantization Benchmarks for Long Context. Speed is prefill throughput on one GPU, useful only for comparing rows.

Three takeaways from the author’s own write-up. First, turbo4 saves only a little memory over q4_0 (25.8% versus 28.1% of bf16) while running about 17% slower and not beating it on quality. Second, the author notes that llama.cpp’s regular q4 and q5 types already apply a rotation, “the same trick TurboQuant uses,” which narrows the gap. Third, the 3-bit turbo3_tcq mode is the one worth keeping, “a real precision/VRAM/speed compromise for users who need the context to fit.” His rule of thumb: “If q5 or q4 fits, use q5 or q4.”

A separate data point comes from the closed CPU pull request. Its author reported on Qwen3.5-4B that tbq4_0 reached 3.94x compression against q4_0’s 3.56x with similar quality, but CPU generation dropped from 15.72 to about 8.3 tokens per second. That is one small model on one CPU, reported by the PR’s author.

Caveat: community reports differ. In the same GitHub thread, one user said a fork’s turbo4 ran at 36 tokens per second where q4_0 and q5_0 ran at 18 to 20 on their setup. Results depend heavily on your GPU, backend and context length.

What should you use instead of TurboQuant in llama.cpp?

Quantize the KV cache with the built-in -ctk and -ctv flags, starting with q8_0 and moving to q5_0 only if memory is tight. Both flags exist in current llama.cpp, with the environment variables LLAMA_ARG_CACHE_TYPE_K and LLAMA_ARG_CACHE_TYPE_V. A quantized V cache generally needs Flash Attention, set with -fa on.

llama-server -m your-model.gguf -c 65536 -fa on -ctk q8_0 -ctv q8_0

Use this ladder as a starting point, based on the benchmark above:

1. q8_0 / q8_0 is the safe, high-quality choice at about 53% of bf16 size.
2. q5_0 / q5_0 is the benchmark’s “good precision/size, relatively safe” option, at about 34%.
3. q5_0 K with q4_1 V is what the author calls the “best default if VRAM-constrained.”
4. q4_0 / q4_0 saves the most among the standard types but shows visible precision loss, especially for tool calls and code.

Keys are more sensitive than values, so spend your bits on K first. Ollama users can do the same with OLLAMA_KV_CACHE_TYPE set to q8_0 or q4_0, which Ollama says needs Flash Attention. If memory is the real problem, the cheapest fix may be hardware. Our mini PC for a home server guide covers small, low-cost machines for home use.

What about llama.cpp MTP, which people search alongside TurboQuant?

MTP (multi-token prediction) is a different feature: it speeds up generation, while TurboQuant shrinks memory. llama.cpp’s speculative decoding docs list --spec-type draft-mtp, described as “Use Multi Token Prediction (MTP) heads from the main model.” It only helps models that ship with MTP heads.

MTP is in mainline and under active work. GitHub shows MTP-related pull requests merged on November 3 and November 5, 2026, and open bug reports, such as draft-mtp crashing on an AMD RADV setup. It is not a way to get TurboQuant, and the two aren’t tied together. Some forks bundle both, which is probably why the searches overlap.

Should you use a TurboQuant fork?

Only if you’re comfortable with an unreviewed build and you truly can’t fit your context with q5_0 or q4_0. The extra savings appear at 3 bits and below, where the benchmark shows real quality loss in the tail.

• You probably don’t need it if your model and context fit with q8_0, q5_0 or q4_0.
• You might try it if you’re stuck on a small GPU, need a very long context, and can accept lossy output for summarizing or chat.
• Avoid 2-bit modes for code, JSON or tool calls. The benchmark author calls the 2-bit tail results “not a tool-call, JSON, code-edit, or math mode.”

If you try a fork, check its README for the exact flags, which can differ from mainline. Pin a version, compare output against q8_0 on your own prompts, and don’t assume a Reddit screenshot matches your hardware.

What would change the answer?

A new mainline pull request that shows measurable gains over q4_0 and q5_0 across several models would reopen the question. The maintainers said exactly that on June 3. Watch discussion #20969 and the pull request list. Until a merge happens, no official binary will accept TurboQuant types. For other recent local-model coverage, see our post on Reflection AI Beam.

FAQ

Does llama.cpp support TurboQuant?

No. Mainline llama.cpp accepts only f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1 for its KV cache as of November 9, 2026. Community forks add turbo3 and turbo4 types.

Why was TurboQuant rejected from llama.cpp?

Maintainers said no one had shown evidence justifying the cost of adding it. The CPU pull request was closed on June 2, 2026, and they asked for benchmarks showing gains over existing quant types at the same size.

What is the llama.cpp flag for TurboQuant?

There is no flag in mainline llama.cpp. In community forks the cache types are set with -ctk and -ctv using names such as turbo3 or turbo4, but those names only work in the fork that defines them.

Does Ollama support TurboQuant?

No. Ollama’s TurboQuant pull request (#15505) was closed without merging on May 30, 2026, and its FAQ lists only f16, q8_0 and q4_0 for KV cache quantization.

Does vLLM support TurboQuant?

Partly. vLLM’s v0.20.0 release notes, dated April 27, 2026, list a TurboQuant 2-bit KV cache backend. Check the current vLLM documentation for the flags that apply to your version.

Is llama.cpp MTP the same as TurboQuant?

No. MTP is a speculative-decoding feature that uses a model’s own multi-token-prediction heads to generate faster. TurboQuant compresses the KV cache to save memory. They solve different problems.

Sources

• Google Research, TurboQuant: Redefining AI efficiency with extreme compression, March 24, 2026
• Zandieh et al., TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
• llama.cpp, PR #21089, issue #20977, PR #21010, discussion #20969, checked November 9, 2026
• llama.cpp, common/arg.cpp and speculative decoding docs
• vLLM, v0.20.0 release notes
• Ollama, PR #15505 and FAQ
• Anbeeld, KV Cache Quantization Benchmarks for Long Context
• Community forks: TheTom/llama-cpp-turboquant, spiritbuun/buun-llama-cpp