CUDA OoM with larger --kv-reserve-tokens (workaround: empiric lower --memory-ratio) #168

Hello, and thanks for open sourcing this project.

I have a similar issue as #44 , the default size of KV-cache is very small.

When running as

ft serve --model nvidia/Qwen3.6-35B-A3B-NVFP4 --kv-reserve-tokens 131072

it breaks citing error allocating a 64 or 128 MiB GPU memory block:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB. GPU 0 has a total capacity of 11.63 GiB of which 21.56 MiB is free. Including non-PyTorch memory, this process has 11.07 GiB memory in use. Of the allocated memory 10.85 GiB is allocated by PyTorch, with 22.00 MiB allocated in private pools (e.g., CUDA Graphs), and 36.30 MiB is reserved by PyTorch but unallocated.

It seems I can work around the problem by passing a lower --memory-ratio (like 0.8), but why cannot it reserve the memory it needs upfront?

Also, less than 1 GiB VRAM is used by other applications, but --memory-ratio of 0.9 breaks the same way.

E.g. now when it works with 0.8:
10502MiB is used by Freetoken, 11039MiB used in total (out of 12 GiB on my aging 3060).