Activity

1. obitodaitu commented on Sep 1, 2026

   

   obitodaitu

   on Sep 1, 2026

   More actions

   merge quickly
2. AI-Anarkan commented on Sep 1, 2026

   

   AI-Anarkan

   on Sep 1, 2026

   Author

   More actions

   🚀 Full Stack Breakthrough: 524K Context, Radix GDN Caching & The Dual RTX 5090 Hybrid MoE Architecture

   Hey @FlashML-org team! 👋

   We wanted to drop a major technical update on our 24/7 production AI workstation.

   Over the past few days, we cross-pollinated optimizations from the speculative drafting ecosystem (specifically adapting concepts from syv-ai/qwen38-27b-rtx3090 to Blackwell SM120) and combined them with FreeToken's cutting-edge MoE runtime.

   The results blew our expectations away: we achieved 524,288 tokens (524K full context) on FreeToken 176.9B, sub-millisecond multi-turn prefill via Radix GDN Caching, and built an ultra-high-throughput dual-GPU pipeline reaching 215–350+ tok/s decode on our companion coding engine.

   Here is the complete architectural deep-dive, hardware benchmarks, and production config.

   ---

   🏗️ 1. Workstation Hardware & Dual-GPU Topology

   We run on Proxmox VE 9.2 (IOMMU PCIe 4.0/5.0 Passthrough) inside an Ubuntu 26.04 LTS guest with strict kernel isolation:

                     ┌─────────────────────────────────────────────────────────────┐
                     │   AI Creative Workstation — 28 vCPUs | 136 GB RAM | 39 GB Swap│
                     └──────────────────────────────┬──────────────────────────────┘
                                                    │
                      ┌─────────────────────────────┴─────────────────────────────┐
                      │                                                           │
        ┌─────────────▼──────────────┐                              ┌─────────────▼──────────────┐
        │   NVIDIA RTX 5090 #1 (GPU 0)│                              │   NVIDIA RTX 5090 #2 (GPU 1)│
        │   32 GB VRAM | Blackwell   │                              │   32 GB VRAM | Blackwell   │
        └─────────────┬──────────────┘                              └─────────────┬──────────────┘
                      │                                                           │
           ⚡ FreeToken Runtime                                        ⚡ vLLM + DFlash2 Runtime
      • Model: Qwen3.8-Flash-Next-NVFP4 (176.9B)                 • Model: Qwen3.8-27B-W4A16 AutoRound Fast
      • Context: 524,288 tokens (524K Full Pool)                 • Context: 262,144 tokens (256K Full Pool)
      • 63.3 GB Pinned Host RAM SHM Expert Tables                • Speculative Block Drafter (k=7, 1.19 GB)
      • Radix GDN-Aware Prefix Caching (90%+ hit)                • Multi-Image Vision Active (VISION=1)
      • MoE Prefill Hit D2D (cudaMemcpyBatchAsync)               • Speed: 216.7 – 350+ tok/s decode
      • Port: :1919 (Dedicated to Autonomous Agents & MoE)       • Port: :8001 (Dedicated to OpenCode & IDEs)
   

   Both engines co-exist in total memory isolation without PCIe lane starvation or host RAM exhaustion (~32 GB safety margin under full concurrent load).

   ---

   ⚡ 2. FreeToken 176.9B MoE: Breakthrough Optimizations

   Production Service Configuration (:1919)

   ft serve \
     --model /models/Qwen3.8-Flash-Next-NVFP4 \
     --gpu 0 \
     --host 0.0.0.0 \
     --port 1919 \
     --served-model-name RadixArk/Qwen3.8-Flash-Next-NVFP4 \
     --memory-ratio 0.90 \
     --expert-load parallel \
     --moe-cache-auto \
     --kv-reserve-tokens 524288 \
     --max-seq-len-override 524288 \
     --max-output-tokens 65536 \
     --max-prefill-length 8192 \
     --cache-type radix \
     --enable-cache-report \
     --enable-special-token-ckpt \
     --moe-prefill-hit-d2d \
     --reasoning-parser qwen3 \
     --tool-call-parser qwen3_coder

   The 4 Levers That Multiplied Our Performance:

   1. Radix Prefix Caching GDN-Aware (--cache-type radix + --enable-cache-report)
      Autonomous coding agents prepend 1.5k–4k tokens of system prompt, skill instructions, and tool catalogs on every turn.

      • Turn 1 (Cold Prefill): 1,626 tokens prefilled at 5.46s.
      • Turn 2 (Radix Hit): 1,600 out of 1,780 prompt tokens (90%+) served directly from cache $\to$ TTFT dropped to virtually 0 ms!
      • Turn 3 (Multi-turn Tool Loop): 1,728 / 2,058 tokens cached, maintaining sub-second turn transitions.

   2. MoE Prefill Hit D2D (--moe-prefill-hit-d2d)
      Uses cudaMemcpyBatchAsync on CUDA 13.2+ for GPU resident experts. When prompt tokens hit cached experts, the transfer bypasses the host PCIe bus entirely, keeping PCIe bandwidth free for miss streaming.

   3. Tool-Call Decode Checkpointing (--enable-special-token-ckpt)
      Preserves the recurrent decode state right at the <tool_call> boundary. When an agent executes bash/python/MCP tools and returns the output, the context resumes cleanly without invalidating preceding tokens.

   4. DMA & Host RAM Calibration (ft bench bw --dtype nvfp4 --gpu 0)

      • Host RAM STREAM Read: 53.0 GB/s
      • PCIe Linear H2D / D2H: 28.9 / 28.7 GB/s
      • CPU MoE Kernel: 47.2 GB/s

   ---

   🏎️ 3. The Companion Engine: Qwen3.8-27B DFlash2 ($k=7$) on GPU 1

   Inspired by the non-autoregressive speculative decoding architecture in syv-ai/qwen38-27b-rtx3090, we built a dedicated 256K instance on RTX 5090 #2 (GPU 1):

   • Base Model: Qwen3.8-27B-W4A16-AutoRound-fast (Marlin INT4 GEMMs)
   • Speculative Drafter: DFlash2-W4A16 ($k=7$ block verification, 1.19 GB)
   • KV Cache: 13.5 GB allocated via dynamic int8_per_token_head scales (262,144 tokens full context).
   • Multimodal Vision: Active (VISION=1, MAX_IMAGES=8) with CPU offload (VLLM_VISION_CPU_OFFLOAD_GB=1).

   Live Real-World Benchmarks:

   • Standard Complex Python/TS Generation (1024 tokens): 216.7 tok/s stable decode (4.73s total generation time).
   • Lookup-Augmented Drafting (LOOKUP=1): Blazes up to 350+ tok/s on refactoring and repetitive codebase tasks.

   ---

   📊 4. Full Production Cluster Summary

   Host / GPU	Engine & Architecture	Real Decode Speed	Context Window	Target Use Case
   RTX 5090 #1 (GPU 0)	FreeToken 176.9B MoE (Radix Cache + D2D)	42–65 tok/s (0ms TTFT)	524,288 (524K)	Autonomous Agents, Deep MoE Reasoning
   RTX 5090 #2 (GPU 1)	Qwen3.8-27B DFlash2 ($k=7$) (vLLM)	216.7 – 350+ tok/s	262,144 (256K)	High-Speed Interactive Coding & IDE Autocomplete
   RTX 3090 (Remote Node)	Qwen3.8-27B DFlash2 ($k=7$) (WSL2)	120 – 160+ tok/s	131,072 (128K)	Distributed Dev Workloads & Offloading

   ---

   🎯 Final Thoughts

   What the FreeToken team achieved with ZeroTable host SHM mapping and parallel offloaded MoE execution is unprecedented. Combining it with Radix Prefix Caching, D2D batch copy, and Tool-Call Checkpointing turns a single consumer RTX 5090 into an industrial-grade frontier model server with a massive 524K context window.

   Kudos to the maintainers! We hope these parameters and benchmark insights help the community push their hardware to the absolute maximum. 🔥

   👍React with 👍1Reacted by KeenZ