feat(server): expose /v1/tokenize endpoint for client-side token validation #473

Problem

FreeToken has a TokenizeManager (python/freetoken/tokenizer/tokenize.py) used internally via ZMQ, but it's not exposed as an HTTP endpoint. OpenAI-compatible clients (like Understory, llama-swap, LiteLLM) cannot pre-validate prompt token counts before sending requests.

This causes silent failures: when a prompt exceeds max_seq_len, FreeToken drops the request (Input sequence length exceeds max_seq_len) and returns 400. The client has no way to know in advance.

llama.cpp exposes /tokenize which clients use to count tokens and chunk/split prompts before submission.

Proposed solution

Add a POST /v1/tokenize endpoint that accepts {input: str} and returns {tokens: int, token_ids: list[int]}, mirroring llama.cpp's API for compatibility.

The TokenizeManager.tokenize() method already exists — just needs an HTTP route in server/openai_api.py or a new server/tokenize_api.py.

Context

• Testing Understory (knowledge graph agent) with 72K-token system prompt against 65K ctx → request dropped silently
• Had to manually probe token counts via curl + error messages
• llama.cpp's /tokenize endpoint is used by Understory's agent to validate before sending

Environment

ft version: 0.1.2
model: nvidia/Qwen3.6-35B-A3B-NVFP4
gpu: RTX 3080 Ti Laptop (16GB, sm_86)

Happy to implement this.