Skip to content

FreeToken suggesting models that do not fit into available RAM #419

Open
Open
FreeToken suggesting models that do not fit into available RAM#419
Labels
Desktopproblem related to FreeToken DesktopbugSomething isn't workinglinux

Description

@azazar
azazar
opened on Sep 9, 2026

Before you start

  • I have read the FAQ and my problem is not answered there.
  • I have read the Roadmap and this is not already planned there.
  • I have searched existing issues and found no duplicate.
  • I have restarted the Desktop app to pick up the latest update and the problem still happens.

What happened

FreeToken suggesting models that do not fit into available RAM.

Image

Desktop app version

v0.2.0-beta.18/freetoken version 0.1.2+gaf71ba432

OS

Debian GNU/Linux 13 (trixie), Linux 6.12.107+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.107-1 (2026-08-29) x86_64 GNU/Linux

GPU and driver

RTX 5060 Ti, 16GB, driver 595.91.07

CPU and system RAM

AMD Ryzen 5 9600X, 96GB

Activity

  1. azazar
    added
    bugSomething isn't working
    Desktopproblem related to FreeToken Desktop
    on Sep 9, 2026
  2. github-actions
    added
    linux
    on Sep 9, 2026

  3. andy-yang-1 commented on Sep 9, 2026

    @andy-yang-1
    andy-yang-1
    on Sep 9, 2026
    Collaborator

    @azazar This model works on your device because it will offload 50GB to disk

  4. azazar commented on Sep 9, 2026

    @azazar
    azazar
    on Sep 9, 2026
    Author

    It doesn't work.

    Image

    Log

    [13:20:45] cmd/INFO  ft serve --model /home/user/.freetoken/models/Qwen3.8-Flash-Next-NVFP4 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.95 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420 --gpu GPU-00efa0cb-183f-8e14-e0c4-4c35cd459f48
[13:20:47] stdout/INFO  [2026-09-09|13:20:47] INFO     Parsed arguments:
[13:20:47] stdout/INFO  ServerArgs(model_path='/home/user/.freetoken/models/Qwen3.8-Flash-Next-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', ple_backend='disk', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.95, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=132870', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Qwen3.8-Flash-Next-NVFP4', tool_call_parser='qwen3_coder', reasoning_parser='qwen3', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420', gpu=('GPU-00efa0cb-183f-8e14-e0c4-4c35cd459f48',), gpu_assigned=None)
[13:20:47] stdout/INFO  [2026-09-09|13:20:47|initializer] INFO     --gpu GPU-00efa0cb-183f-8e14-e0c4-4c35cd459f48 -> GPU-00efa0cb-183f-8e14-e0c4-4c35cd459f48
[13:20:47] stdout/INFO  [2026-09-09|13:20:47|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=20, top_p=0.95
[13:20:47] stderr/INFO  INFO:     Started server process [132870]
[13:20:47] stderr/INFO  INFO:     Waiting for application startup.
[13:20:47] stderr/INFO  INFO:     Application startup complete.
[13:20:47] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[13:20:49] stderr/INFO  /home/user/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1034: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[13:20:49] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[13:20:49] stdout/INFO  [2026-09-09|13:20:49|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[13:20:50] stdout/INFO  [2026-09-09|13:20:50|core|rank=0] INFO     Auto-selected attention backend: qsa_sparse
[13:20:50] stdout/WARN  [2026-09-09|13:20:50|core|rank=0] WARNING  Page size is overridden to 64 for the qsa_sparse backend
[13:20:50] stdout/INFO  [2026-09-09|13:20:50|core|rank=0] INFO     benchbw profile recommends hybrid for 'nvfp4' experts on this GPU
[13:20:50] stdout/INFO  [2026-09-09|13:20:50|core|rank=0] INFO     Auto-selected MoE backend: hybrid
[13:20:50] stdout/INFO  [2026-09-09|13:20:50|core|rank=0] INFO     No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected backend 'hybrid'
[13:20:50] stdout/INFO  [2026-09-09|13:20:50|core|rank=0] INFO     Resolved config: moe_backend='hybrid', attention_backend='qsa_sparse', cache_type='hybrid_radix', page_size=64
[13:20:50] stdout/INFO  [2026-09-09|13:20:50|core|rank=0] INFO     Free memory before loading model: 15.27 GiB
[13:20:58] stderr/INFO  Loading weights: 100%|██████████| 206/206 [00:07<00:00, 27.24it/s]
[13:20:58] stdout/INFO  [2026-09-09|13:20:58|core|rank=0] INFO     PLE disk backend: io_uring, O_DIRECT, wait-sync
[13:20:59] stdout/WARN  [2026-09-09|13:20:59|core|rank=0] WARNING  expert banks: low free RAM -> serial build (avoids parallel-reader OOM; override with --expert-load parallel)
[13:20:59] stdout/INFO  [2026-09-09|13:20:59|core|rank=0] INFO     expert banks: slow path (serial build)
[13:22:10] stderr/INFO  Loading Qwen3.8-Flash-Next NVFP4 experts: 100%|██████████| 192/192 [01:08<00:00,  2.81it/s]
[13:22:10] stdout/ERROR  [2026-09-09|13:22:10|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=1024 slots, kv=128 pages) needs 3046637568 B > budget 2557061534 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[13:22:10] stderr/INFO  Process freetoken-TP0-scheduler:
[13:22:10] stderr/INFO  Traceback (most recent call last):
[13:22:10] stderr/INFO    File "/home/user/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[13:22:10] stderr/INFO      self.run()
[13:22:10] stderr/INFO    File "/home/user/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[13:22:10] stderr/INFO      self._target(*self._args, **self._kwargs)
[13:22:10] stderr/INFO    File "/home/user/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 81, in _run_scheduler
[13:22:10] stderr/INFO      scheduler = Scheduler(args)
[13:22:10] stderr/INFO                  ^^^^^^^^^^^^^^^
[13:22:10] stderr/INFO    File "/home/user/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 65, in __init__
[13:22:10] stderr/INFO      self.engine = Engine(config)
[13:22:10] stderr/INFO                    ^^^^^^^^^^^^^^
[13:22:10] stderr/INFO    File "/home/user/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 343, in __init__
[13:22:10] stderr/INFO      self._init_offload_moe_cache(config)
[13:22:10] stderr/INFO    File "/home/user/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 595, in _init_offload_moe_cache
[13:22:10] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[13:22:10] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[13:22:10] stderr/INFO    File "/home/user/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 490, in _resolve_auto_moe_cache_size
[13:22:10] stderr/INFO      return resolve_moe_cache_auto(
[13:22:10] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[13:22:10] stderr/INFO    File "/home/user/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[13:22:10] stderr/INFO      return plan_cache_budget(
[13:22:10] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[13:22:10] stderr/INFO    File "/home/user/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[13:22:10] stderr/INFO      assert total <= budget_bytes, (
[13:22:10] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[13:22:10] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=1024 slots, kv=128 pages) needs 3046637568 B > budget 2557061534 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[13:22:11] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=1024 slots, kv=128 pages) needs 3046637568 B > budget 2557061534 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[13:22:11] stderr/INFO  INFO:     Shutting down
[13:22:11] stderr/INFO  INFO:     Waiting for application shutdown.
[13:22:11] stderr/INFO  INFO:     Application shutdown complete.
[13:22:11] stderr/INFO  INFO:     Finished server process [132870]
[13:22:11] stderr/INFO  /home/user/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[13:22:11] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '

  5. azazar commented on Sep 9, 2026

    @azazar
    azazar
    on Sep 9, 2026
    Author

    Attempted to set MoE compute mode to CPU. Can't fit.

    [13:32:43] stderr/INFO  Loading weights: 100%|██████████| 206/206 [00:07<00:00, 27.94it/s]
[13:32:43] stdout/INFO  [2026-09-09|13:32:43|core|rank=0] INFO     PLE disk backend: io_uring, O_DIRECT, wait-sync
[13:32:43] stdout/WARN  [2026-09-09|13:32:43|core|rank=0] WARNING  expert banks: low free RAM -> serial build (avoids parallel-reader OOM; override with --expert-load parallel)
[13:32:43] stdout/INFO  [2026-09-09|13:32:43|core|rank=0] INFO     expert banks: slow path (serial build)
[13:33:57] stderr/INFO  Loading Qwen3.8-Flash-Next NVFP4 experts: 100%|██████████| 192/192 [01:10<00:00,  2.72it/s]
[13:33:58] stdout/INFO  [2026-09-09|13:33:58|core|rank=0] INFO     torch intra-op threads: 6 -> 1 (cores reserved for the pinned CPU MoE pool)
[13:33:58] stdout/INFO  [2026-09-09|13:33:58|core|rank=0] INFO     CPU MoE executor ready: threads=5 (pinned to cores 0..4) isa=avx512bf16+avx512vnni(nvfp4-w4a8) fmt=nvfp4 H=2560 I=640 experts=512 layers=48 top_k=10 act=silu max_tokens=4
[13:33:58] stderr/INFO  Process freetoken-TP0-scheduler:
[13:33:58] stdout/ERROR  [2026-09-09|13:33:58|FrontendAPI] ERROR    Backend supervisor: AssertionError: Not enough memory for KV cache, try reducing --num-pages
[13:33:58] stderr/INFO  Traceback (most recent call last):
[13:33:58] stderr/INFO    File "/home/m/Private/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[13:33:58] stderr/INFO      self.run()
[13:33:58] stderr/INFO    File "/home/m/Private/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[13:33:58] stderr/INFO      self._target(*self._args, **self._kwargs)
[13:33:58] stderr/INFO    File "/home/m/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 81, in _run_scheduler
[13:33:58] stderr/INFO      scheduler = Scheduler(args)
[13:33:58] stderr/INFO                  ^^^^^^^^^^^^^^^
[13:33:58] stderr/INFO    File "/home/m/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 65, in __init__
[13:33:58] stderr/INFO      self.engine = Engine(config)
[13:33:58] stderr/INFO                    ^^^^^^^^^^^^^^
[13:33:58] stderr/INFO    File "/home/m/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 353, in __init__
[13:33:58] stderr/INFO      self.num_pages = self._pool_cls.solve_num_pages(config, available_memory)
[13:33:58] stderr/INFO                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[13:33:58] stderr/INFO    File "/home/m/.freetoken/venv/lib/python3.12/site-packages/freetoken/kvcache/base.py", line 73, in solve_num_pages
[13:33:58] stderr/INFO      assert num_pages > 1, "Not enough memory for KV cache, try reducing --num-pages"
[13:33:58] stderr/INFO             ^^^^^^^^^^^^^
[13:33:58] stderr/INFO  AssertionError: Not enough memory for KV cache, try reducing --num-pages
[13:33:58] health/ERROR  AssertionError: Not enough memory for KV cache, try reducing --num-pages
[13:33:58] stderr/INFO  INFO:     Shutting down
[13:33:58] stderr/INFO  INFO:     Waiting for application shutdown.
[13:33:58] stderr/INFO  INFO:     Application shutdown complete.
[13:33:58] stderr/INFO  INFO:     Finished server process [139776]
[13:33:59] stderr/INFO  /home/m/Private/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[13:33:59] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '

  6. andy-yang-1 commented on Sep 9, 2026

    @andy-yang-1
    andy-yang-1
    on Sep 9, 2026
    Collaborator

    @azazar I took a closer look and found that your vram is a little bit limited. I think you can try to improve memory_ratio (maybe 0.95).

  7. azazar commented on Sep 10, 2026

    @azazar
    azazar
    on Sep 10, 2026
    Author

    @azazar I took a closer look and found that your vram is a little bit limited. I think you can try to improve memory_ratio (maybe 0.95).

    I've tried different values already (0.95, 0.9, and 0.85). That GPU isn't used for graphics, so it's memory is fully dedicated to FreeToken. All of that doesn't work.

  8. jason-fxz commented on Sep 12, 2026

    @jason-fxz
    jason-fxz
    on Sep 12, 2026
    Collaborator

    16GiB VRAM is tight for running Qwen3.8 Next Flash. We will consider KV cache budget when suggest model in Desktop.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Desktopproblem related to FreeToken DesktopbugSomething isn't workinglinux

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions