cli.md

CLI reference

ft <command> [args]

Command	Purpose
ft serve	Start the API server (OpenAI /v1/*, Anthropic /v1/messages, Responses)
ft shell	Chat with a server in the terminal
ft ctl	Query and manage a running server over HTTP
ft launch	Configure and launch a coding agent against a server
ft checkpoint	Convert an HF checkpoint to the FTW fast-load format
ft bench bw	Benchmark CPU vs PCIe bandwidth to calibrate hybrid MoE decode

ft --version prints the installed version (torch-free; nightly wheels carry a +g<sha> build stamp, tagged releases a bare version). Every command supports --help.

ft serve

ft serve --model <path-or-hf-id> [options]

--model is the only required flag — dtype, attention backend, MoE backend, MoE cache size, KV capacity, CUDA-graph sizes and the tool-call/reasoning parsers all resolve automatically from the checkpoint and the GPU.

Model

Flag	Default	Meaning
--model-path, --model	required	Local dir, HF repo id, or an FTW dir (auto-detected)
--served-model-name	basename of --model	Model id reported by /v1/models
--hf-overrides	—	JSON object applied to the checkpoint's config as vLLM's --hf-overrides: a nested config section updates key by key, any other value is replaced whole. A YaRN rope_parameters override serves original_max_position_embeddings * factor positions

Server & runtime

Flag	Default	Meaning
--host	127.0.0.1	Bind address
--port	1919	Bind port
--api-key	$FREETOKEN_API_KEY, else off	Require the key on every route except /health, as Authorization: Bearer <key> or x-api-key: <key>; other requests get a 401
--gpu	GPU 0	GPU to run on: a UUID from nvidia-smi -L or an nvidia-smi index; see below
--max-running-requests	4	Max concurrently running requests
--max-output-tokens	32768	Default output budget for requests that omit one
--max-seq-len-override	from checkpoint	Max sequence length
--max-prefill-length	8192	Chunked-prefill chunk size in tokens; with offloaded experts see performance/prefill.md
--cuda-graph-max-bs, --graph	= max running requests	Max batch size captured as CUDA graphs
--decode-log-interval	40	Scheduler status line every N decode steps

Choosing a GPU

For example, a machine with an RTX 5090 and an RTX 3060 Ti:

$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 3060 Ti (UUID: GPU-2f3a9b1c-8d7e-4a05-b6c1-0e5f9a3d7b42)
GPU 1: NVIDIA GeForce RTX 5090 (UUID: GPU-9e8d7c6b-5a49-4f13-8207-c1b0a4e6d3f5)

ft serve --model ... --gpu 1             # by nvidia-smi index -- the 5090
ft serve --model ... --gpu GPU-9e8d7c6b  # the same card by UUID (a unique prefix is enough)

KV cache & memory

Flag	Default	Meaning
--memory-ratio	0.9	Fraction of free VRAM the engine may use (weights + MoE cache + KV)
--num-pages / --num-tokens	auto	KV capacity override in pages / tokens (mutually exclusive; auto sizes from VRAM left after weights and MoE cache)
--page-size	1	KV page size; DSV4 forces 128, the TRTLLM backend needs 16/32/64, SWA models require 1
--cache-type	radix	radix (prefix reuse; SWA/GDN-aware variants picked automatically) or naive
--attention-backend, --attn	auto	trtllm/fi/fa/triton/dsv4_sparse/dsa; prefill,decode pair allowed; auto picks per model + GPU

MoE offload

See models.md for what each strategy does, and performance/prefill.md for prefill speed.

Flag	Default	Meaning
--moe-strategy	auto	fused/offload/cpu/hybrid; auto → offload; fused on unified-memory GPUs (GB10), see models.md. --moe-backend is the deprecated old spelling
--quant-backend	auto	Kernel per quantized layer type, layer[.kind]=name entries: linear=marlin,moe=b12x or moe.nvfp4=triton. A layer-level entry applies to every kind whose table lists the name
--nvfp4-backend	—	Deprecated: stands in for --quant-backend moe.nvfp4=<marlin|b12x|triton> (flashinfer means b12x); cannot be combined with --quant-backend
--moe-cache-size / --moe-cache-rate / --moe-cache-mem / --moe-cache-auto	auto	GPU expert-cache size as slots / fraction of all experts / memory such as 12GiB / sized from free VRAM (mutually exclusive; auto is enabled by default for offload-family strategies)
--moe-cache-scope	unified	unified: one LRU pool of slots per expert quant (quant type and bank layout), so all MoE layers share one pool when they share a quant format; per-layer: each layer has its own pool. per-layer does not combine with --moe-strategy cpu or --moe-cpu-layers
--moe-cache-split	equal	How the pools share the slots: equal (in proportion to their layer counts), or profile:<path>: every pool keeps top_k slots and the rest go to the most used experts in the profile's per-layer expert counts. With offload, a pool smaller than the expert count lowers --max-running-requests to its slots / top_k
--kv-reserve-tokens	8192	KV token floor reserved before --moe-cache-auto fills experts
--moe-cpu-threads	physical cores	CPU worker threads for the cpu/hybrid executor
--moe-cpu-layers	all on GPU	With offload/hybrid: which MoE layers decode on CPU (3,7,11, a count, a fraction, or auto). auto is for hosts that cap CUDA pinned memory (Windows/WSL): it pins layers from the middle out until CUDA refuses one and OS-locks the rest (FREETOKEN_PIN_BUDGET_GB and FTW checkpoints lock from a size estimate instead), see models.md; every value needs an expert format the CPU executor serves (bf16, fp8_block, nvfp4, mxfp4, GGUF on x86 CPUs)
--moe-hybrid-max-fetch	auto	With hybrid: max experts fetched over PCIe per layer per step; rest computed on CPU
--moe-locked-fetch	auto	With hybrid and locked layers: share of a locked layer's misses fetched per step into its GPU slots through pinned staging. Auto takes p / (2p + c) from the ft bench bw row of the layer's kind (at most 1 per step without a row); 0 = locked layers decode on CPU only
--moe-locked-max-fetch	8	Most experts a locked layer fetches per step; sizes the staging (that many rows of the widest expert, per bank)
--moe-collect-stats	off	Log decode MoE stats with each decode log line: hit rate, and per layer-step the misses, experts fetched and experts computed on CPU, for pinned, locked and CPU-only layers
--moe-prefill-hit-d2d	off	Prefill: copy cache-hit experts device-side, stream only misses (CUDA >= 13)
--disable-moe-prefill-overlap	overlap on	Disable the two-buffer prefill copy overlap

API behaviour

Flag	Default	Meaning
--sampling-defaults	model	Fill unspecified sampling params from the checkpoint's generation_config.json (none = framework defaults)
--tool-call-parser	auto	Tool-call format; auto-inferred from the model family
--reasoning-parser	auto	Splits chain-of-thought into reasoning_content; auto-inferred; off disables
--enable-cache-report	off	Report prefix-cache hits in each response's usage block
--anthropic-inline-system	auto	Placement of late Anthropic system instructions: auto, preserve, or fold

Image input

Experimental. Needs a checkpoint whose family registers a vision encoder (models.md lists them and how each one maps the flags below); a request carrying images is rejected otherwise. Images are accepted on all three protocols (OpenAI image_url, Anthropic image blocks, Responses input_image) as an http(s) URL or base64. Images inside a tool result (an Anthropic tool_result block from Claude Code's Read, a Responses function_call_output from Codex's view_image) are moved to the user turn that follows the tool message, as vLLM does, because chat templates render tool messages as plain text. GET /v1/stats reports what the server accepts as model.input_modalities (["text"] or ["text", "image"]), so a client can gate its attachment controls without reading the checkpoint config.

Flag	Default	Meaning
--text-model-only	off	Serve a multimodal checkpoint text-only: no encoder tower is built (its VRAM goes to the KV/expert pools) and every multimodal input is rejected. Same as --mm-disable with every encoder kind
--mm-disable	none	Encoder towers to leave unbuilt (vision, audio); every input they would serve is rejected
--mmproj	found next to the model	GGUF models only: the llama.cpp vision tower file (mmproj-*.gguf), see Image input. Without it, FreeToken takes a *mmproj*.gguf that fits the model (output width, tower type, image tokens) from the model file's own folder, BF16 before F32, F16 and quantized files. With none, or with files of several models it cannot tell apart, the model serves text only; a repo that keeps each quant in a subfolder and the mmproj at the top needs --mmproj. The log names the file it loads
--mm-encoder-weights	host	Where the encoder tower's block weights live. host streams them from pinned host banks two blocks at a time behind the compute, so the GPU holds two blocks instead of the whole tower; small images pay the copy time, large ones hide it behind the compute. gpu keeps them resident. An encoder without a block stack stays resident either way
--image-min-tokens, --image-max-tokens	processor defaults	Per-image token budget: the image processor resizes every image to take between these many tokens, converted to the family's own units by its processor. A family with fixed budgets honors the maximum only and refuses one below its smallest budget at start-up
--mm-processor-kwargs	none	JSON object of extra keyword arguments for the checkpoint's image processor call, for knobs the token budget does not cover; applied after the budget, so an explicit key wins
--mm-embed-cache-device	cpu	Where encoded image embeddings live between prefill chunks. cpu keeps them out of the VRAM budget; cuda skips the copy back
--allowed-media-domains	any	Comma-separated hostname allowlist for image URLs; requests for other domains are rejected with a 400. Empty allows any domain
--allowed-local-media-path	off	Directory file:// image refs may be read from; unset rejects local files

ft shell

ft shell                                    # attach to a running server
ft shell --model ~/models/Qwen3.6-35B-A3B   # serve + chat in one process

• Attach mode talks to --server URL (default http://127.0.0.1:1919)
• --api-key KEY (default $FREETOKEN_API_KEY) attaches to a server started with --api-key; with --model the shell uses that server's own key
• /help inside the shell lists the commands (/think, /cache, /reset).

ft ctl

ft ctl [--base-url http://127.0.0.1:1919] [--api-key KEY] [--timeout 10] [--json] <subcommand>

--api-key defaults to $FREETOKEN_API_KEY.

Subcommand	Endpoint	Purpose
health	GET /health	Server status, model, load progress
stats	GET /v1/stats	Throughput, latency, VRAM, pool occupancy, accepted input modalities
generate [prompt] [--max-tokens N] [--ignore-eos]	POST /generate	Raw completion smoke test (no chat template)
cache	GET /v1/cache/status	Cache pool table
cache --moe N | --kv N | --mamba N | --swa N [--wait 300]	POST /v1/cache/rebuild	Live pool resizing without a restart (k/m suffixes; --kv/--swa in tokens)
requests [--since N] [--limit N]	GET /v1/requests	Recent request ring

ft launch

ft launch {claude,codex,dsh,hermes,openclaw,opencode} [options] [-- <agent args>]

Discovers the served model via /v1/models, writes the agent's provider config, installs the agent CLI if missing, then launches it. Cloud API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, …) are cleared from the child environment so the agent cannot silently fall back to a paid endpoint. When /v1/stats reports image among model.input_modalities, the written config declares the model image-capable, which Codex, OpenCode, OpenClaw and dsh require before their image tools and attachments send anything; Claude Code and Hermes need no declaration.

Flag	Meaning
--server URL	Server to point the agent at (default http://127.0.0.1:1919)
--api-key KEY	Key of a server started with --api-key (default $FREETOKEN_API_KEY). The agent gets it in its environment; Hermes and OpenClaw keep it in their config files
--dry-run	Print the planned config changes and command, touch nothing
-y, --yes	Approve install/config prompts
--config	Configure without launching
--install-only	Just install the agent CLI (needs no server)
--force-reinstall	Re-run the agent installer
-- <args>	Forwarded verbatim to the agent

ft checkpoint

ft checkpoint --model <hf_dir> --out <ftw_dir> [--dtype bfloat16] [--moe-backend offload] [--quant-backend moe.nvfp4=b12x] [--shard-gib 8] [--gpu <uuid-or-index>]

Converts an HF safetensors checkpoint to FTW, FreeToken's self-contained fast-load format; point ft serve --model at the output dir. MoE experts are always packed into expert banks, which serve the fused and the offload strategies alike (DeepSeek-V4 experts serve offload only); --moe-backend is kept for compatibility. See the FTW caveats in models.md; FTW files from older builds can be repaired with scripts/ftw_hotfix.py instead of reconverting.

ft bench bw

ft bench bw --model /path/to/model.gguf   # every MoE layer kind of a checkpoint (GGUF or HF)
ft bench bw                               # canonical geometry per expert format
ft bench bw --gpu 1                       # a specific GPU (UUID or nvidia-smi index, as for ft serve)

Measures host-RAM vs PCIe bandwidth with the real cpu/offload MoE kernels and writes one row per MoE layer kind, which ft serve --moe-strategy hybrid --moe-hybrid-max-fetch -1 reads to split each layer's misses.

• One profile per GPU, at ~/.cache/freetoken/benchbw/<gpu-uuid>.json; a profile from other hardware or another profile version is ignored.
• A row's key is the layer's CPU format and options, expert shape, top_k and CPU worker threads; a layer without an exact row keeps a fetch cap of 1. Bench with the same --cpu-threads as the server's --moe-cpu-threads.
• Each run replaces the rows it benched and keeps the others.
• What to measure: --model (checkpoint paths or presets), --dtype, --formats, --isa.
• --threshold (default 2.0) sets each row's recommended: hybrid when CPU bandwidth beats PCIe by that factor.

On NVLink-C2C hosts, see models.md for FREETOKEN_H2D_BLOCKS_PER_BANK if PCIe-gather sits well below the linear H2D ceiling.