Skip to content

AssertionError: cache budget too small #150

Open
Open
AssertionError: cache budget too small#150

Description

@lolmc0587
lolmc0587
opened on Aug 24, 2026
[23:48:51] cmd/INFO  ft serve --model /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.85 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420
[23:48:58] stdout/INFO  [2026-08-24|23:48:58] INFO     Parsed arguments:
[23:48:58] stdout/INFO  ServerArgs(model_path='/home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.85, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=18650', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Gemma-4-26B-A4B-NVFP4', tool_call_parser='gemma4', reasoning_parser='gemma4', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420')
[23:48:58] stdout/INFO  [2026-08-24|23:48:58|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=64, top_p=0.95
[23:48:58] stderr/INFO  INFO:     Started server process [18650]
[23:48:58] stderr/INFO  INFO:     Waiting for application startup.
[23:48:58] stderr/INFO  INFO:     Application startup complete.
[23:48:58] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[23:48:59] stderr/INFO  /home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1015: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[23:48:59] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[23:48:59] stdout/INFO  [2026-08-24|23:48:59|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[23:49:00] stdout/INFO  [2026-08-24|23:49:00|core|rank=0] INFO     Auto-selected attention backend: triton
[23:49:00] stdout/INFO  [2026-08-24|23:49:00|core|rank=0] INFO     benchbw profile recommends hybrid for 'nvfp4' experts on this GPU
[23:49:00] stdout/INFO  [2026-08-24|23:49:00|core|rank=0] INFO     Auto-selected MoE backend: hybrid
[23:49:00] stdout/INFO  [2026-08-24|23:49:00|core|rank=0] INFO     No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected backend 'hybrid'
[23:49:00] stdout/INFO  [2026-08-24|23:49:00|core|rank=0] INFO     Resolved config: moe_backend='hybrid', attention_backend='triton', cache_type='swa_radix', page_size=1
[23:49:00] stdout/INFO  [2026-08-24|23:49:00|core|rank=0] INFO     Free memory before loading model: 7.11 GiB
[23:49:09] stdout/INFO  [2026-08-24|23:49:09|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4)
[23:49:09] stderr/INFO  Process freetoken-TP0-scheduler:
[23:49:09] stdout/ERROR  [2026-08-24|23:49:09|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -965292852 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:49:09] stderr/INFO  Traceback (most recent call last):
[23:49:09] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[23:49:09] stderr/INFO      self.run()
[23:49:09] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[23:49:09] stderr/INFO      self._target(*self._args, **self._kwargs)
[23:49:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 74, in _run_scheduler
[23:49:09] stderr/INFO      scheduler = Scheduler(args)
[23:49:09] stderr/INFO                  ^^^^^^^^^^^^^^^
[23:49:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 64, in __init__
[23:49:09] stderr/INFO      self.engine = Engine(config)
[23:49:09] stderr/INFO                    ^^^^^^^^^^^^^^
[23:49:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 335, in __init__
[23:49:09] stderr/INFO      self._init_offload_moe_cache(config)
[23:49:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 584, in _init_offload_moe_cache
[23:49:09] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[23:49:09] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[23:49:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 481, in _resolve_auto_moe_cache_size
[23:49:09] stderr/INFO      return resolve_moe_cache_auto(
[23:49:09] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[23:49:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[23:49:09] stderr/INFO      return plan_cache_budget(
[23:49:09] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[23:49:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[23:49:09] stderr/INFO      assert total <= budget_bytes, (
[23:49:09] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[23:49:09] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -965292852 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:49:10] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -965292852 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:49:10] stderr/INFO  INFO:     Shutting down
[23:49:10] stderr/INFO  INFO:     Waiting for application shutdown.
[23:49:10] stderr/INFO  INFO:     Application shutdown complete.
[23:49:10] stderr/INFO  INFO:     Finished server process [18650]
[23:49:10] stderr/INFO  /home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[23:49:10] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[23:49:15] cmd/INFO  ft serve --model /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.85 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420
[23:49:18] stdout/INFO  [2026-08-24|23:49:18] INFO     Parsed arguments:
[23:49:18] stdout/INFO  ServerArgs(model_path='/home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.85, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=19942', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Gemma-4-26B-A4B-NVFP4', tool_call_parser='gemma4', reasoning_parser='gemma4', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420')
[23:49:18] stdout/INFO  [2026-08-24|23:49:18|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=64, top_p=0.95
[23:49:18] stderr/INFO  INFO:     Started server process [19942]
[23:49:18] stderr/INFO  INFO:     Waiting for application startup.
[23:49:18] stderr/INFO  INFO:     Application startup complete.
[23:49:18] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[23:49:20] stderr/INFO  /home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1015: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[23:49:20] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[23:49:20] stdout/INFO  [2026-08-24|23:49:20|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[23:49:20] stdout/INFO  [2026-08-24|23:49:20|core|rank=0] INFO     Auto-selected attention backend: triton
[23:49:20] stdout/INFO  [2026-08-24|23:49:20|core|rank=0] INFO     benchbw profile recommends hybrid for 'nvfp4' experts on this GPU
[23:49:20] stdout/INFO  [2026-08-24|23:49:20|core|rank=0] INFO     Auto-selected MoE backend: hybrid
[23:49:20] stdout/INFO  [2026-08-24|23:49:20|core|rank=0] INFO     No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected backend 'hybrid'
[23:49:20] stdout/INFO  [2026-08-24|23:49:20|core|rank=0] INFO     Resolved config: moe_backend='hybrid', attention_backend='triton', cache_type='swa_radix', page_size=1
[23:49:20] stdout/INFO  [2026-08-24|23:49:20|core|rank=0] INFO     Free memory before loading model: 7.11 GiB
[23:49:28] stdout/INFO  [2026-08-24|23:49:28|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4)
[23:49:28] stderr/INFO  Process freetoken-TP0-scheduler:
[23:49:28] stdout/ERROR  [2026-08-24|23:49:28|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -965292852 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:49:28] stderr/INFO  Traceback (most recent call last):
[23:49:28] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[23:49:28] stderr/INFO      self.run()
[23:49:28] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[23:49:28] stderr/INFO      self._target(*self._args, **self._kwargs)
[23:49:28] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 74, in _run_scheduler
[23:49:28] stderr/INFO      scheduler = Scheduler(args)
[23:49:28] stderr/INFO                  ^^^^^^^^^^^^^^^
[23:49:28] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 64, in __init__
[23:49:28] stderr/INFO      self.engine = Engine(config)
[23:49:28] stderr/INFO                    ^^^^^^^^^^^^^^
[23:49:28] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 335, in __init__
[23:49:28] stderr/INFO      self._init_offload_moe_cache(config)
[23:49:28] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 584, in _init_offload_moe_cache
[23:49:28] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[23:49:28] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[23:49:28] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 481, in _resolve_auto_moe_cache_size
[23:49:28] stderr/INFO      return resolve_moe_cache_auto(
[23:49:28] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[23:49:28] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[23:49:28] stderr/INFO      return plan_cache_budget(
[23:49:28] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[23:49:28] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[23:49:28] stderr/INFO      assert total <= budget_bytes, (
[23:49:28] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[23:49:28] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -965292852 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:49:30] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -965292852 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:49:30] stderr/INFO  INFO:     Shutting down
[23:49:30] stderr/INFO  INFO:     Waiting for application shutdown.
[23:49:30] stderr/INFO  INFO:     Application shutdown complete.
[23:49:30] stderr/INFO  INFO:     Finished server process [19942]
[23:49:30] stderr/INFO  /home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[23:49:30] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[23:49:32] cmd/INFO  ft serve --model /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.8 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420
[23:49:34] stdout/INFO  [2026-08-24|23:49:34] INFO     Parsed arguments:
[23:49:34] stdout/INFO  ServerArgs(model_path='/home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.8, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=21197', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Gemma-4-26B-A4B-NVFP4', tool_call_parser='gemma4', reasoning_parser='gemma4', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420')
[23:49:34] stdout/INFO  [2026-08-24|23:49:34|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=64, top_p=0.95
[23:49:34] stderr/INFO  INFO:     Started server process [21197]
[23:49:34] stderr/INFO  INFO:     Waiting for application startup.
[23:49:34] stderr/INFO  INFO:     Application startup complete.
[23:49:34] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[23:49:36] stderr/INFO  /home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1015: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[23:49:36] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[23:49:36] stdout/INFO  [2026-08-24|23:49:36|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[23:49:36] stdout/INFO  [2026-08-24|23:49:36|core|rank=0] INFO     Auto-selected attention backend: triton
[23:49:36] stdout/INFO  [2026-08-24|23:49:36|core|rank=0] INFO     benchbw profile recommends hybrid for 'nvfp4' experts on this GPU
[23:49:36] stdout/INFO  [2026-08-24|23:49:36|core|rank=0] INFO     Auto-selected MoE backend: hybrid
[23:49:36] stdout/INFO  [2026-08-24|23:49:36|core|rank=0] INFO     No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected backend 'hybrid'
[23:49:36] stdout/INFO  [2026-08-24|23:49:36|core|rank=0] INFO     Resolved config: moe_backend='hybrid', attention_backend='triton', cache_type='swa_radix', page_size=1
[23:49:36] stdout/INFO  [2026-08-24|23:49:36|core|rank=0] INFO     Free memory before loading model: 7.11 GiB
[23:49:44] stdout/INFO  [2026-08-24|23:49:44|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4)
[23:49:44] stdout/ERROR  [2026-08-24|23:49:44|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1347066266 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:49:44] stderr/INFO  Process freetoken-TP0-scheduler:
[23:49:44] stderr/INFO  Traceback (most recent call last):
[23:49:44] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[23:49:44] stderr/INFO      self.run()
[23:49:44] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[23:49:44] stderr/INFO      self._target(*self._args, **self._kwargs)
[23:49:44] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 74, in _run_scheduler
[23:49:44] stderr/INFO      scheduler = Scheduler(args)
[23:49:44] stderr/INFO                  ^^^^^^^^^^^^^^^
[23:49:44] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 64, in __init__
[23:49:44] stderr/INFO      self.engine = Engine(config)
[23:49:44] stderr/INFO                    ^^^^^^^^^^^^^^
[23:49:44] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 335, in __init__
[23:49:44] stderr/INFO      self._init_offload_moe_cache(config)
[23:49:44] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 584, in _init_offload_moe_cache
[23:49:44] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[23:49:44] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[23:49:44] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 481, in _resolve_auto_moe_cache_size
[23:49:44] stderr/INFO      return resolve_moe_cache_auto(
[23:49:44] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[23:49:44] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[23:49:44] stderr/INFO      return plan_cache_budget(
[23:49:44] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[23:49:44] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[23:49:44] stderr/INFO      assert total <= budget_bytes, (
[23:49:44] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[23:49:44] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1347066266 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:49:46] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1347066266 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:49:46] stderr/INFO  INFO:     Shutting down
[23:49:46] stderr/INFO  INFO:     Waiting for application shutdown.
[23:49:46] stderr/INFO  INFO:     Application shutdown complete.
[23:49:46] stderr/INFO  INFO:     Finished server process [21197]
[23:49:46] stderr/INFO  /home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[23:49:46] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[23:50:02] cmd/INFO  ft serve --model /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4 --port 1919 --moe-backend hybrid --max-running-requests 4 --memory-ratio 0.8 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420
[23:50:04] stdout/INFO  [2026-08-24|23:50:04] INFO     Parsed arguments:
[23:50:04] stdout/INFO  ServerArgs(model_path='/home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='hybrid', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.8, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=22489', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Gemma-4-26B-A4B-NVFP4', tool_call_parser='gemma4', reasoning_parser='gemma4', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420')
[23:50:04] stdout/INFO  [2026-08-24|23:50:04|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=64, top_p=0.95
[23:50:04] stderr/INFO  INFO:     Started server process [22489]
[23:50:04] stderr/INFO  INFO:     Waiting for application startup.
[23:50:04] stderr/INFO  INFO:     Application startup complete.
[23:50:04] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[23:50:06] stderr/INFO  /home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1015: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[23:50:06] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[23:50:06] stdout/INFO  [2026-08-24|23:50:06|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[23:50:06] stdout/INFO  [2026-08-24|23:50:06|core|rank=0] INFO     Auto-selected attention backend: triton
[23:50:06] stdout/INFO  [2026-08-24|23:50:06|core|rank=0] INFO     Resolved config: moe_backend='hybrid', attention_backend='triton', cache_type='swa_radix', page_size=1
[23:50:07] stdout/INFO  [2026-08-24|23:50:07|core|rank=0] INFO     Free memory before loading model: 7.11 GiB
[23:50:14] stdout/INFO  [2026-08-24|23:50:14|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4)
[23:50:14] stderr/INFO  Process freetoken-TP0-scheduler:
[23:50:14] stdout/ERROR  [2026-08-24|23:50:14|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1347066266 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:50:14] stderr/INFO  Traceback (most recent call last):
[23:50:14] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[23:50:14] stderr/INFO      self.run()
[23:50:14] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[23:50:14] stderr/INFO      self._target(*self._args, **self._kwargs)
[23:50:14] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 74, in _run_scheduler
[23:50:14] stderr/INFO      scheduler = Scheduler(args)
[23:50:14] stderr/INFO                  ^^^^^^^^^^^^^^^
[23:50:14] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 64, in __init__
[23:50:14] stderr/INFO      self.engine = Engine(config)
[23:50:14] stderr/INFO                    ^^^^^^^^^^^^^^
[23:50:14] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 335, in __init__
[23:50:14] stderr/INFO      self._init_offload_moe_cache(config)
[23:50:14] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 584, in _init_offload_moe_cache
[23:50:14] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[23:50:14] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[23:50:14] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 481, in _resolve_auto_moe_cache_size
[23:50:15] stderr/INFO      return resolve_moe_cache_auto(
[23:50:15] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[23:50:15] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[23:50:15] stderr/INFO      return plan_cache_budget(
[23:50:15] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[23:50:15] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[23:50:15] stderr/INFO      assert total <= budget_bytes, (
[23:50:15] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[23:50:15] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1347066266 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:50:16] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1347066266 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:50:16] stderr/INFO  INFO:     Shutting down
[23:50:16] stderr/INFO  INFO:     Waiting for application shutdown.
[23:50:16] stderr/INFO  INFO:     Application shutdown complete.
[23:50:16] stderr/INFO  INFO:     Finished server process [22489]
[23:50:16] stderr/INFO  /home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[23:50:16] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[23:51:57] cmd/INFO  ft serve --model /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.8 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420
[23:51:59] stdout/INFO  [2026-08-24|23:51:59] INFO     Parsed arguments:
[23:51:59] stdout/INFO  ServerArgs(model_path='/home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.8, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=25099', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Gemma-4-26B-A4B-NVFP4', tool_call_parser='gemma4', reasoning_parser='gemma4', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420')
[23:51:59] stdout/INFO  [2026-08-24|23:51:59|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=64, top_p=0.95
[23:51:59] stderr/INFO  INFO:     Started server process [25099]
[23:51:59] stderr/INFO  INFO:     Waiting for application startup.
[23:51:59] stderr/INFO  INFO:     Application startup complete.
[23:51:59] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[23:52:01] stderr/INFO  /home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1015: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[23:52:01] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[23:52:01] stdout/INFO  [2026-08-24|23:52:01|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[23:52:01] stdout/INFO  [2026-08-24|23:52:01|core|rank=0] INFO     Auto-selected attention backend: triton
[23:52:01] stdout/INFO  [2026-08-24|23:52:01|core|rank=0] INFO     benchbw profile recommends hybrid for 'nvfp4' experts on this GPU
[23:52:01] stdout/INFO  [2026-08-24|23:52:01|core|rank=0] INFO     Auto-selected MoE backend: hybrid
[23:52:01] stdout/INFO  [2026-08-24|23:52:01|core|rank=0] INFO     No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected backend 'hybrid'
[23:52:01] stdout/INFO  [2026-08-24|23:52:01|core|rank=0] INFO     Resolved config: moe_backend='hybrid', attention_backend='triton', cache_type='swa_radix', page_size=1
[23:52:01] stdout/INFO  [2026-08-24|23:52:01|core|rank=0] INFO     Free memory before loading model: 6.94 GiB
[23:52:09] stdout/INFO  [2026-08-24|23:52:09|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4)
[23:52:09] stderr/INFO  Process freetoken-TP0-scheduler:
[23:52:09] stdout/ERROR  [2026-08-24|23:52:09|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1491927040 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:52:09] stderr/INFO  Traceback (most recent call last):
[23:52:09] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[23:52:09] stderr/INFO      self.run()
[23:52:09] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[23:52:09] stderr/INFO      self._target(*self._args, **self._kwargs)
[23:52:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 74, in _run_scheduler
[23:52:09] stderr/INFO      scheduler = Scheduler(args)
[23:52:09] stderr/INFO                  ^^^^^^^^^^^^^^^
[23:52:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 64, in __init__
[23:52:09] stderr/INFO      self.engine = Engine(config)
[23:52:09] stderr/INFO                    ^^^^^^^^^^^^^^
[23:52:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 335, in __init__
[23:52:09] stderr/INFO      self._init_offload_moe_cache(config)
[23:52:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 584, in _init_offload_moe_cache
[23:52:09] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[23:52:09] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[23:52:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 481, in _resolve_auto_moe_cache_size
[23:52:09] stderr/INFO      return resolve_moe_cache_auto(
[23:52:09] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[23:52:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[23:52:09] stderr/INFO      return plan_cache_budget(
[23:52:09] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[23:52:09] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[23:52:09] stderr/INFO      assert total <= budget_bytes, (
[23:52:09] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[23:52:09] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1491927040 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:52:10] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1491927040 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:52:10] stderr/INFO  INFO:     Shutting down
[23:52:10] stderr/INFO  INFO:     Waiting for application shutdown.
[23:52:10] stderr/INFO  INFO:     Application shutdown complete.
[23:52:10] stderr/INFO  INFO:     Finished server process [25099]
[23:52:10] stderr/INFO  /home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[23:52:10] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[23:52:34] cmd/INFO  ft serve --model /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.9 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420
[23:52:37] stdout/INFO  [2026-08-24|23:52:37] INFO     Parsed arguments:
[23:52:37] stdout/INFO  ServerArgs(model_path='/home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.9, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=26422', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Gemma-4-26B-A4B-NVFP4', tool_call_parser='gemma4', reasoning_parser='gemma4', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420')
[23:52:37] stdout/INFO  [2026-08-24|23:52:37|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=64, top_p=0.95
[23:52:37] stderr/INFO  INFO:     Started server process [26422]
[23:52:37] stderr/INFO  INFO:     Waiting for application startup.
[23:52:37] stderr/INFO  INFO:     Application startup complete.
[23:52:37] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[23:52:38] stderr/INFO  /home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1015: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[23:52:38] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[23:52:38] stdout/INFO  [2026-08-24|23:52:38|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[23:52:38] stdout/INFO  [2026-08-24|23:52:38|core|rank=0] INFO     Auto-selected attention backend: triton
[23:52:38] stdout/INFO  [2026-08-24|23:52:38|core|rank=0] INFO     benchbw profile recommends hybrid for 'nvfp4' experts on this GPU
[23:52:38] stdout/INFO  [2026-08-24|23:52:38|core|rank=0] INFO     Auto-selected MoE backend: hybrid
[23:52:38] stdout/INFO  [2026-08-24|23:52:38|core|rank=0] INFO     No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected backend 'hybrid'
[23:52:38] stdout/INFO  [2026-08-24|23:52:38|core|rank=0] INFO     Resolved config: moe_backend='hybrid', attention_backend='triton', cache_type='swa_radix', page_size=1
[23:52:39] stdout/INFO  [2026-08-24|23:52:39|core|rank=0] INFO     Free memory before loading model: 6.91 GiB
[23:52:47] stdout/INFO  [2026-08-24|23:52:47|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4)
[23:52:47] stderr/INFO  Process freetoken-TP0-scheduler:
[23:52:47] stdout/ERROR  [2026-08-24|23:52:47|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -773383783 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:52:47] stderr/INFO  Traceback (most recent call last):
[23:52:47] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[23:52:47] stderr/INFO      self.run()
[23:52:47] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[23:52:47] stderr/INFO      self._target(*self._args, **self._kwargs)
[23:52:47] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 74, in _run_scheduler
[23:52:47] stderr/INFO      scheduler = Scheduler(args)
[23:52:47] stderr/INFO                  ^^^^^^^^^^^^^^^
[23:52:47] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 64, in __init__
[23:52:47] stderr/INFO      self.engine = Engine(config)
[23:52:47] stderr/INFO                    ^^^^^^^^^^^^^^
[23:52:47] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 335, in __init__
[23:52:47] stderr/INFO      self._init_offload_moe_cache(config)
[23:52:47] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 584, in _init_offload_moe_cache
[23:52:47] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[23:52:47] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[23:52:47] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 481, in _resolve_auto_moe_cache_size
[23:52:47] stderr/INFO      return resolve_moe_cache_auto(
[23:52:47] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[23:52:47] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[23:52:47] stderr/INFO      return plan_cache_budget(
[23:52:47] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[23:52:47] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[23:52:47] stderr/INFO      assert total <= budget_bytes, (
[23:52:47] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[23:52:47] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -773383783 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:52:48] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -773383783 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:52:48] stderr/INFO  INFO:     Shutting down
[23:52:48] stderr/INFO  INFO:     Waiting for application shutdown.
[23:52:48] stderr/INFO  INFO:     Application shutdown complete.
[23:52:48] stderr/INFO  INFO:     Finished server process [26422]
[23:52:48] stderr/INFO  /home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[23:52:48] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[23:53:34] cmd/INFO  ft serve --model /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.9 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420
[23:53:36] stdout/INFO  [2026-08-24|23:53:36] INFO     Parsed arguments:
[23:53:36] stdout/INFO  ServerArgs(model_path='/home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='auto', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=False, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.9, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=28078', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Gemma-4-26B-A4B-NVFP4', tool_call_parser='gemma4', reasoning_parser='gemma4', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420')
[23:53:36] stdout/INFO  [2026-08-24|23:53:36|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=64, top_p=0.95
[23:53:36] stderr/INFO  INFO:     Started server process [28078]
[23:53:36] stderr/INFO  INFO:     Waiting for application startup.
[23:53:36] stderr/INFO  INFO:     Application startup complete.
[23:53:36] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[23:53:38] stderr/INFO  /home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1015: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[23:53:38] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[23:53:38] stdout/INFO  [2026-08-24|23:53:38|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[23:53:38] stdout/INFO  [2026-08-24|23:53:38|core|rank=0] INFO     Auto-selected attention backend: triton
[23:53:38] stdout/INFO  [2026-08-24|23:53:38|core|rank=0] INFO     benchbw profile recommends hybrid for 'nvfp4' experts on this GPU
[23:53:38] stdout/INFO  [2026-08-24|23:53:38|core|rank=0] INFO     Auto-selected MoE backend: hybrid
[23:53:38] stdout/INFO  [2026-08-24|23:53:38|core|rank=0] INFO     No MoE cache sizing flag given; defaulting to --moe-cache-auto for auto-selected backend 'hybrid'
[23:53:38] stdout/INFO  [2026-08-24|23:53:38|core|rank=0] INFO     Resolved config: moe_backend='hybrid', attention_backend='triton', cache_type='swa_radix', page_size=1
[23:53:38] stdout/INFO  [2026-08-24|23:53:38|core|rank=0] INFO     Free memory before loading model: 6.94 GiB
[23:53:46] stdout/INFO  [2026-08-24|23:53:46|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4)
[23:53:46] stderr/INFO  Process freetoken-TP0-scheduler:
[23:53:46] stdout/ERROR  [2026-08-24|23:53:46|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -751206400 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:53:46] stderr/INFO  Traceback (most recent call last):
[23:53:46] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[23:53:46] stderr/INFO      self.run()
[23:53:46] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[23:53:46] stderr/INFO      self._target(*self._args, **self._kwargs)
[23:53:46] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 74, in _run_scheduler
[23:53:46] stderr/INFO      scheduler = Scheduler(args)
[23:53:46] stderr/INFO                  ^^^^^^^^^^^^^^^
[23:53:46] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 64, in __init__
[23:53:46] stderr/INFO      self.engine = Engine(config)
[23:53:46] stderr/INFO                    ^^^^^^^^^^^^^^
[23:53:46] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 335, in __init__
[23:53:46] stderr/INFO      self._init_offload_moe_cache(config)
[23:53:46] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 584, in _init_offload_moe_cache
[23:53:46] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[23:53:46] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[23:53:46] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 481, in _resolve_auto_moe_cache_size
[23:53:46] stderr/INFO      return resolve_moe_cache_auto(
[23:53:46] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[23:53:46] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[23:53:46] stderr/INFO      return plan_cache_budget(
[23:53:46] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[23:53:46] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[23:53:46] stderr/INFO      assert total <= budget_bytes, (
[23:53:46] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[23:53:46] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -751206400 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:53:48] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -751206400 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:53:48] stderr/INFO  INFO:     Shutting down
[23:53:48] stderr/INFO  INFO:     Waiting for application shutdown.
[23:53:48] stderr/INFO  INFO:     Application shutdown complete.
[23:53:48] stderr/INFO  INFO:     Finished server process [28078]
[23:53:48] stderr/INFO  /home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[23:53:48] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[23:54:13] cmd/INFO  ft serve --model /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4 --port 1919 --moe-backend offload --max-running-requests 4 --memory-ratio 0.9 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420
[23:54:16] stdout/INFO  [2026-08-24|23:54:16] INFO     Parsed arguments:
[23:54:16] stdout/INFO  ServerArgs(model_path='/home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='offload', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.9, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=29404', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Gemma-4-26B-A4B-NVFP4', tool_call_parser='gemma4', reasoning_parser='gemma4', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420')
[23:54:16] stdout/INFO  [2026-08-24|23:54:16|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=64, top_p=0.95
[23:54:16] stderr/INFO  INFO:     Started server process [29404]
[23:54:16] stderr/INFO  INFO:     Waiting for application startup.
[23:54:16] stderr/INFO  INFO:     Application startup complete.
[23:54:16] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[23:54:17] stderr/INFO  /home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1015: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[23:54:17] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[23:54:17] stdout/INFO  [2026-08-24|23:54:17|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[23:54:17] stdout/INFO  [2026-08-24|23:54:17|core|rank=0] INFO     Auto-selected attention backend: triton
[23:54:17] stdout/INFO  [2026-08-24|23:54:17|core|rank=0] INFO     Resolved config: moe_backend='offload', attention_backend='triton', cache_type='swa_radix', page_size=1
[23:54:18] stdout/INFO  [2026-08-24|23:54:18|core|rank=0] INFO     Free memory before loading model: 6.94 GiB
[23:54:26] stdout/INFO  [2026-08-24|23:54:26|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4)
[23:54:26] stdout/ERROR  [2026-08-24|23:54:26|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -751206400 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:54:26] stderr/INFO  Process freetoken-TP0-scheduler:
[23:54:26] stderr/INFO  Traceback (most recent call last):
[23:54:26] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[23:54:26] stderr/INFO      self.run()
[23:54:26] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[23:54:26] stderr/INFO      self._target(*self._args, **self._kwargs)
[23:54:26] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 74, in _run_scheduler
[23:54:26] stderr/INFO      scheduler = Scheduler(args)
[23:54:26] stderr/INFO                  ^^^^^^^^^^^^^^^
[23:54:26] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 64, in __init__
[23:54:26] stderr/INFO      self.engine = Engine(config)
[23:54:26] stderr/INFO                    ^^^^^^^^^^^^^^
[23:54:26] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 335, in __init__
[23:54:26] stderr/INFO      self._init_offload_moe_cache(config)
[23:54:26] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 584, in _init_offload_moe_cache
[23:54:26] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[23:54:26] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[23:54:26] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 481, in _resolve_auto_moe_cache_size
[23:54:26] stderr/INFO      return resolve_moe_cache_auto(
[23:54:26] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[23:54:26] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[23:54:26] stderr/INFO      return plan_cache_budget(
[23:54:26] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[23:54:26] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[23:54:26] stderr/INFO      assert total <= budget_bytes, (
[23:54:26] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[23:54:26] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -751206400 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:54:26] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -751206400 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:54:26] stderr/INFO  INFO:     Shutting down
[23:54:26] stderr/INFO  INFO:     Waiting for application shutdown.
[23:54:26] stderr/INFO  INFO:     Application shutdown complete.
[23:54:26] stderr/INFO  INFO:     Finished server process [29404]
[23:54:26] stderr/INFO  /home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[23:54:26] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '
[23:55:09] cmd/INFO  ft serve --model /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4 --port 1919 --moe-backend hybrid --max-running-requests 4 --memory-ratio 0.85 --host 127.0.0.1 --cors-origins tauri://localhost,http://tauri.localhost,http://localhost:1420
[23:55:13] stdout/INFO  [2026-08-24|23:55:13] INFO     Parsed arguments:
[23:55:13] stdout/INFO  ServerArgs(model_path='/home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4', tp_info=DistributedInfo(rank=0, size=1), dtype=torch.bfloat16, max_running_req=4, attention_backend='auto', moe_backend='hybrid', nvfp4_backend='triton', expert_load='auto', moe_cache_size=0, moe_cache_rate=None, moe_cache_auto=True, kv_reserve_tokens=8192, moe_cache_policy='lru', moe_prefill_overlap=True, moe_prefill_hit_d2d=False, moe_collect_stats=False, moe_cpu_threads=0, moe_cpu_layers=None, moe_hybrid_max_fetch=-1, cuda_graph_bs=None, cuda_graph_max_bs=None, page_size=1, memory_ratio=0.85, linear_state_cache_ratio=2.0, swa_full_tokens_ratio=0.2, swa_num_pages_override=None, distributed_timeout=60.0, use_dummy_weight=False, use_pynccl=True, max_seq_len_override=None, num_page_override=None, num_token_override=None, max_extend_tokens=8192, cache_type='radix', offline_mode=False, decode_log_interval=40, special_token_ckpt=False, _unique_suffix='.pid=30960', server_host='127.0.0.1', server_port=1919, num_tokenizer=0, silent_output=False, shell_mode=False, served_model_name='Gemma-4-26B-A4B-NVFP4', tool_call_parser='gemma4', reasoning_parser='gemma4', sampling_defaults='model', max_output_tokens=None, enable_cache_report=False, cors_origins='tauri://localhost,http://tauri.localhost,http://localhost:1420')
[23:55:13] stdout/INFO  [2026-08-24|23:55:13|FrontendAPI] INFO     Default sampling config (source=model): temperature=1.0, top_k=64, top_p=0.95
[23:55:13] stderr/INFO  INFO:     Started server process [30960]
[23:55:13] stderr/INFO  INFO:     Waiting for application startup.
[23:55:13] stderr/INFO  INFO:     Application startup complete.
[23:55:13] stderr/INFO  INFO:     Uvicorn running on http://127.0.0.1:1919 (Press CTRL+C to quit)
[23:55:14] stderr/INFO  /home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py:1015: FutureWarning: torch.cuda._set_allocator_settings is deprecated. Use torch._C._accelerator_setAllocatorSettings instead.
[23:55:14] stderr/INFO    torch.cuda.memory._set_allocator_settings("expandable_segments:True")
[23:55:14] stdout/INFO  [2026-08-24|23:55:14|core|rank=0] INFO     Enabled expandable_segments (override via PYTORCH_ALLOC_CONF)
[23:55:15] stdout/INFO  [2026-08-24|23:55:15|core|rank=0] INFO     Auto-selected attention backend: triton
[23:55:15] stdout/INFO  [2026-08-24|23:55:15|core|rank=0] INFO     Resolved config: moe_backend='hybrid', attention_backend='triton', cache_type='swa_radix', page_size=1
[23:55:15] stdout/INFO  [2026-08-24|23:55:15|core|rank=0] INFO     Free memory before loading model: 6.93 GiB
[23:55:23] stdout/INFO  [2026-08-24|23:55:23|core|rank=0] INFO     expert banks: FTW fast path (FTW checkpoint /home/lolmc/.freetoken/models/Gemma-4-26B-A4B-NVFP4)
[23:55:23] stderr/INFO  Process freetoken-TP0-scheduler:
[23:55:23] stdout/ERROR  [2026-08-24|23:55:23|FrontendAPI] ERROR    Backend supervisor: AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1128120320 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:55:23] stderr/INFO  Traceback (most recent call last):
[23:55:23] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
[23:55:23] stderr/INFO      self.run()
[23:55:23] stderr/INFO    File "/home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
[23:55:23] stderr/INFO      self._target(*self._args, **self._kwargs)
[23:55:23] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/server/launch.py", line 74, in _run_scheduler
[23:55:23] stderr/INFO      scheduler = Scheduler(args)
[23:55:23] stderr/INFO                  ^^^^^^^^^^^^^^^
[23:55:23] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/scheduler/scheduler.py", line 64, in __init__
[23:55:23] stderr/INFO      self.engine = Engine(config)
[23:55:23] stderr/INFO                    ^^^^^^^^^^^^^^
[23:55:23] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 335, in __init__
[23:55:23] stderr/INFO      self._init_offload_moe_cache(config)
[23:55:23] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 584, in _init_offload_moe_cache
[23:55:23] stderr/INFO      size, pages, overlap = self._resolve_auto_moe_cache_size(config, banks)
[23:55:23] stderr/INFO                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[23:55:23] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/engine.py", line 481, in _resolve_auto_moe_cache_size
[23:55:23] stderr/INFO      return resolve_moe_cache_auto(
[23:55:23] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^^^
[23:55:23] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 122, in resolve_moe_cache_auto
[23:55:23] stderr/INFO      return plan_cache_budget(
[23:55:23] stderr/INFO             ^^^^^^^^^^^^^^^^^^
[23:55:23] stderr/INFO    File "/home/lolmc/.freetoken/venv/lib/python3.12/site-packages/freetoken/engine/cache_budget.py", line 89, in plan_cache_budget
[23:55:23] stderr/INFO      assert total <= budget_bytes, (
[23:55:23] stderr/INFO             ^^^^^^^^^^^^^^^^^^^^^
[23:55:23] stderr/INFO  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1128120320 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:55:24] health/ERROR  AssertionError: cache budget too small: minimum plan (moe=256 slots, kv=8192 pages) needs 1361969152 B > budget -1128120320 B (raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)
[23:55:24] stderr/INFO  INFO:     Shutting down
[23:55:25] stderr/INFO  INFO:     Waiting for application shutdown.
[23:55:25] stderr/INFO  INFO:     Application shutdown complete.
[23:55:25] stderr/INFO  INFO:     Finished server process [30960]
[23:55:25] stderr/INFO  /home/lolmc/.local/share/uv/python/cpython-3.12.14-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
[23:55:25] stderr/INFO    warnings.warn('resource_tracker: There appear to be %d '

Activity

  1. MT-z commented on Sep 3, 2026

    @MT-z
    MT-z
    on Sep 3, 2026

    Ran into the same assertion from a different direction (Qwen3.8-Flash-Next-NVFP4 on a 4090 ballasted down to 10.22 GiB free, --memory-ratio 0.9 --kv-reserve-tokens 8192), and the numbers show why the advice at the end cannot help in this case:

    AssertionError: cache budget too small: minimum plan (moe=512 slots, kv=129 pages)
needs 1628749824 B > budget -3178334613 B
(raise memory_ratio, lower kv_reserve_tokens, or free GPU memory)

    The budget is negative. net_cache_budget_bytes is memory_ratio * baseline_free - weights - fixed_cache, so for this run:

    term bytes
    free before load 10.22 GiB
    x --memory-ratio 0.9 9.20 GiB
    - weights + fixed cache ~12.16 GiB
    = budget -2.96 GiB
    minimum plan needs 1.52 GiB one expert layer + the KV reserve
    shortfall 4.48 GiB

    Once the budget is negative the three suggestions are unreachable: raising memory_ratio to 1.0 adds 1.02 GiB against a 4.48 GiB gap and takes the graph headroom to zero, and kv_reserve_tokens only moves the 1.52 GiB minimum plan, not the negative term. The real answer is that the model's non-expert weights do not fit on the card at all.

    That case is worth separating in the message, because it is a different user action. Something like:

    weights + fixed caches need 12.16 GiB but --memory-ratio 0.9 of 10.22 GiB free leaves 9.20 GiB:
this GPU cannot hold the model's non-expert weights (4.48 GiB short of the minimum cache plan).
Try a smaller checkpoint or a larger GPU; memory_ratio and kv_reserve_tokens cannot close a gap this size.

    and keep the current wording for the case the assertion was written for, where the budget is positive but too small to hold the minimum plan -- there the three suggestions do apply.

    For calibration, the same box with a model whose dense part does fit (Ornith-1.5-35B-A3B-NVFP4, same 10.22 GiB free, same flags) boots normally at 2950 expert slots, so the failure really is specific to the weights not fitting rather than to the small card.

    Written with AI assistance; every number above was measured on my hardware and I can reproduce it.

  2. lolmc0587 commented on Sep 9, 2026

    @lolmc0587
    lolmc0587
    on Sep 9, 2026
    Author

    I have lower max concurrency to 1, vram budget to 0.95 and change --kv-reserve-tokens 1024 and it works
    I think it just calculate wrong vram budget

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions