I highly recommend bit better project for most people the https://github.com/poisonxa16/pxa it has much more advanced features and ported lot of the optimizations that are here.
This repository is a fork of llama.cpp, a C/C++ project for running language models locally from GGUF model files. It includes the normal llama.cpp command-line tools and OpenAI-compatible HTTP server; this is a complete llama.cpp source tree, not an extension that must be installed on top of another copy.
The v100-optimized branch focuses on NVIDIA compute capability 7.0 (SM70 / Volta) and 7.5 (SM75 / Turing). The main tested cards are a Tesla V100 32 GB and an RTX 2080 Ti with 22 GB, separately and together. Other SM70/SM75 GPUs can use the same architecture builds, but model/context presets depend on VRAM and were not measured on every card.
The fork primarily targets faster long-prompt processing, lower time to the first generated token, better use of older CUDA hardware, and mixed SM70+SM75 systems. Most optimizations are automatic once the correct CUDA architecture is built; a few model-specific or experimental paths are documented separately.
Start here. The first build choice is the GPU architecture, not the model.
| Hardware in the machine | Examples | Build guide |
|---|---|---|
| SM70 only | Tesla V100, Titan V | Build for SM70 |
| SM75 only | RTX 20-series, T4, Titan RTX | Build for SM75 |
| SM70 + SM75 together | V100 + RTX 2080 Ti | Build for both architectures |
SM70 and SM75 are CUDA architecture targets. They do not describe VRAM capacity. On recent NVIDIA drivers you can check both the GPU name and compute capability directly:
nvidia-smi --query-gpu=name,compute_cap,memory.total --format=csv,noheaderChoose the SM70 guide when it reports 7.0, the SM75 guide when it reports 7.5, and the mixed guide when both appear. For example, the RTX benchmarks in this repository use a 22 GB RTX 2080 Ti; their largest memory presets do not automatically fit an ordinary 11 GB card.
If terms such as GGUF, context/KV cache, batch/ubatch, MoE, MMQ, or MTP are unfamiliar, read Terms used in this fork. It is intentionally short.
After building for the hardware, use the guide for the model family. This is where model-specific memory layouts and tuning belong.
| Model family | What matters here | Guide |
|---|---|---|
| Qwen3.8-27B | Dense model; simple single-GPU presets and an advanced mixed-GPU long-context preset | Qwen3.8-27B |
| Qwen3.8-Flash-Next | Very large MoE; experimental expert streaming from system RAM (tested on a 256 GB host); model-defined sparse attention | Qwen3.8-Flash-Next |
| Ornith 1.5 35B-A3B | MoE; selective MMQ, mixed-GPU placement, optional MTP and multi-slot serving | Ornith |
| Gemma 4 | Dense and routed-expert variants; generic MoE router fusion where eligible | Gemma 4 |
| Other GGUF models | Start with normal llama.cpp options; compatible fork kernels/fusions are selected automatically | Upstream model documentation |
A flag used for one row in the benchmark graph is not necessarily a good global setting. In particular, globally forcing MMQ helps some measured MoE configurations but hurts dense Qwen3.8.
A model first reads the new input and then writes the answer. Prompt processing (PP) measures the reading phase; token generation (TG) measures the answer-generation phase. Higher tokens per second is faster.
The Qwen3.8 generation comparison uses a single V100, Q8_0 K/V, 100k cached tokens and TG512. MTP3 acceptance was 94.97% on both engines. The validated Volta MTP verifier attention and Q5_K/Q6_K four-token kernels used here are enabled automatically; set GGML_CUDA_VOLTA_Q8_FATTN_TC=0, GGML_CUDA_VOLTA_Q5_X4=0, or GGML_CUDA_VOLTA_Q6_W4R4=0 to roll back an individual path.
Most rows add about 1,000 new tokens to an already-computed long history. A label such as 100k cached + 1k input does not mean the benchmark rereads 101,000 tokens from zero. See the short terminology guide if that distinction is unclear.
The September 12 rows compare against upstream 3057bb66. The Qwen3.8-Flash-Next row is a later September 16 matched run against upstream 83078fec0. Exact commands, revisions, model files, and caveats are kept with the benchmark evidence rather than on this landing page: September 12 comparison · Flash-Next September 16 comparison.
The normal build already contains the SM70/SM75 kernel work. You should not have to discover a long list of hidden environment variables before the fork is useful.
| Feature | Default | Scope |
|---|---|---|
| CUDA attention/quantization/kernel selection for supported shapes | automatic | General SM70/SM75 paths |
--moe-router-fusion |
on | Compatible MoE routing graphs; falls back when ineligible |
--selected-attn |
on | Currently qualified for Qwen3.8-Flash-Next sparse-history attention; ineligible graphs fall back |
--moe-layer-first |
off | Experimental request-wide host-expert scheduling; currently qualified for Flash-Next |
--exact-set-top-k |
off | Experimental Flash-Next exact-set selector optimization; intended to generalize to compatible selectors |
Selected-entry attention does not make an ordinary dense-attention model approximately sparse. It only changes how an already-selected sparse history is calculated. The Flash-Next numerical/quality caveats are documented in its model guide.
At a high level, the branch contains:
- SM70/SM75 FlashAttention and other CUDA kernel work for long-context prompt processing;
- faster quantized-weight conversion and matrix paths on the tested older GPUs;
- mixed V100/Turing execution and internal GPU-to-GPU reduction work;
- long-context cache/checkpoint and multi-slot serving improvements;
- generic MoE routing optimizations that can help models beyond Qwen;
- an experimental request-wide MoE path for models whose expert weights live partly in system RAM.
Not every item applies to every model. The model guides say which special settings were actually used.
Whichever hardware guide you use, first confirm the devices llama.cpp sees:
./build/bin/llama-server --list-devicesThen follow the relevant model guide rather than copying a command for a different architecture/model from a benchmark report.
The server exposes an OpenAI-compatible API. For general server endpoints, authentication, networking, and options, use the llama.cpp server documentation.
The graph is a summary, not the source of truth for every setting. Detailed evidence is split by topic:
- Published upstream comparison and exact settings
- Qwen3.8-Flash-Next current-upstream comparison
- Flash-Next hardware/placement checks
- Flash-Next numerical audit
- MoE router-fusion cross-model check
- Quantization-kernel checks
- Saved-state correctness checks
The branch is benchmarked on Linux with CUDA 12.9. Performance depends on model quantization, prompt shape, context length, GPU memory, and device placement.
The base project is maintained by ggml-org/llama.cpp. General llama.cpp build/platform documentation remains under docs/. Use this fork's hardware/model guides for settings that differ from upstream.
For a fork-specific issue, include the commit, GPU model, VRAM, CUDA version, GGUF filename/quantization, and launch command. Remove credentials and private prompts from logs before sharing them.
See LICENSE and CONTRIBUTING.md.