[Feature] Expose per-request inference performance metrics in /v1/chat/completions #503

Before you start

• I have read the Roadmap and this is not already planned there.
• I have read the FAQ.
• I have searched existing issues and found no duplicate.

Applies to

Engine

What problem does this solve

Summary

I would like to request support for exposing per-request inference performance metrics in the OpenAI-compatible /v1/chat/completions API.

Currently, clients can obtain token usage information such as prompt_tokens and completion_tokens, but it would also be very useful to obtain runtime performance metrics for each request.

Ideally, FreeToken could optionally return metrics such as:

• Time to First Token (TTFT)

• Prefill time

• Prefill throughput (tokens/s)

• Decode/generation time

• Decode throughput (tokens/s)

• Total request latency

• Number of prompt tokens actually computed during prefill

• Number of cached/reused prompt tokens when prefix caching is enabled

Motivation

FreeToken is focused on high-performance LLM inference, so exposing these metrics through the API would make it much easier to benchmark and compare different:

• Models

• Quantization formats

• Hardware configurations

• Attention backends

• MoE backends

• Runtime configurations

For example:

TTFT:          350 ms
Prefill:       2450 tokens/s
Decode:        52.3 tokens/s
Total latency: 6.2 s

Without server-side metrics, clients have to estimate these values using HTTP/SSE timestamps.

This is especially inaccurate for prefill performance because the client cannot precisely determine when the internal prefill stage starts and finishes.

Having these metrics reported directly by the inference engine would provide more accurate and consistent measurements.

Proposed API Response

For a non-streaming /v1/chat/completions request, FreeToken could optionally include a metrics or performance object.

Example:

{
  "id": "chatcmpl-xxx",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 4096,
    "completion_tokens": 256,
    "total_tokens": 4352
  },
  "metrics": {
    "ttft_ms": 320.5,
    "prefill_tokens": 4096,
    "prefill_time_ms": 820.4,
    "prefill_tokens_per_second": 4992.7,
    "decode_tokens": 256,
    "decode_time_ms": 5120.8,
    "decode_tokens_per_second": 49.99,
    "total_time_ms": 5480.2
  }
}

The exact field names are only suggestions.

Prefix Cache Considerations

It would be helpful to distinguish between total prompt tokens and tokens that were actually computed during prefill.

For example:

{
  "usage": {
    "prompt_tokens": 8192,
    "completion_tokens": 256,
    "total_tokens": 8448
  },
  "metrics": {
    "cached_prompt_tokens": 6144,
    "prefill_tokens": 2048,
    "prefill_time_ms": 410.0,
    "prefill_tokens_per_second": 4995.1
  }
}

In this case:

prompt_tokens = cached_prompt_tokens + prefill_tokens
8192 = 6144 + 2048

This distinction would make prefill_tokens_per_second more meaningful when prefix caching is enabled.

Otherwise, calculating prefill throughput using all prompt tokens could produce misleading results.

Streaming Requests

It would also be useful to support performance metrics when:

{

"stream": true

}

For streaming responses, the metrics could be included in the final SSE chunk, similar to usage information.

Example:

{

"choices": [],

"usage": {

"prompt_tokens": 4096,

"completion_tokens": 256,

"total_tokens": 4352

},

"metrics": {

"ttft_ms": 320.5,

"prefill_tokens_per_second": 4992.7,

"decode_tokens_per_second": 49.99,

"total_time_ms": 5480.2

}

}

Optional / Opt-In Behavior

To preserve OpenAI API compatibility, this feature could be optional.

For example, it could be enabled using a server-side flag:

ft serve 

--model <model> 

--enable-performance-metrics

Alternatively, it could be enabled per request:

{

"model": "...",

"messages": [...],

"performance_metrics": true

}

Another possibility would be to extend the existing cache-reporting mechanism so that diagnostic and performance information can be enabled together.

Suggested Metrics

The following metrics would be useful:

Minimum Useful Set

If implementing all of these metrics is too much initially, I think the following four metrics would already cover most benchmarking and monitoring use cases:

ttft_ms
prefill_tokens_per_second
decode_tokens_per_second
total_time_ms

It would also be useful to expose the underlying token counts and durations:

prefill_tokens
prefill_time_ms
decode_tokens
decode_time_ms

This would allow clients to calculate additional metrics themselves.

Benefits

This feature would make it much easier to:

• Benchmark FreeToken inference performance

• Compare different models and quantization formats

• Compare different hardware configurations

• Build inference monitoring dashboards

• Diagnose prefill vs. decode bottlenecks

• Measure the impact of prefix caching

• Compare FreeToken with other inference engines

Thanks for considering this feature.

Proposed solution

Add optional per-request inference performance metrics to the OpenAI-compatible /v1/chat/completions API.

For a non-streaming request, FreeToken could include a metrics or performance object in the response.

For example:

{
  "id": "chatcmpl-xxx",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 4096,
    "completion_tokens": 256,
    "total_tokens": 4352
  },
  "metrics": {
    "ttft_ms": 320.5,
    "prefill_tokens": 4096,
    "prefill_time_ms": 820.4,
    "prefill_tokens_per_second": 4992.7,
    "decode_tokens": 256,
    "decode_time_ms": 5120.8,
    "decode_tokens_per_second": 49.99,
    "total_time_ms": 5480.2
  }
}

The exact field names are only suggestions.

It would also be useful to distinguish between total prompt tokens and tokens that were actually computed during prefill, especially when prefix caching is enabled.

For example:

{
  "usage": {
    "prompt_tokens": 8192,
    "completion_tokens": 256,
    "total_tokens": 8448
  },
  "metrics": {
    "cached_prompt_tokens": 6144,
    "prefill_tokens": 2048,
    "prefill_time_ms": 410.0,
    "prefill_tokens_per_second": 4995.1
  }
}

This would avoid misleading prefill throughput results when a large part of the prompt is reused from the prefix cache.

For streaming requests ("stream": true), the performance metrics could be included in the final SSE chunk, similar to usage information.

For example:

{
  "choices": [],
  "usage": {
    "prompt_tokens": 4096,
    "completion_tokens": 256,
    "total_tokens": 4352
  },
  "metrics": {
    "ttft_ms": 320.5,
    "prefill_tokens_per_second": 4992.7,
    "decode_tokens_per_second": 49.99,
    "total_time_ms": 5480.2
  }
}

To preserve OpenAI API compatibility, this feature could be opt-in.

For example, using a server-side option:

ft serve \
  --model <model> \
  --enable-performance-metrics

or a request-level option:

{
  "model": "...",
  "messages": [...],
  "performance_metrics": true
}

If exposing all metrics initially is too complex, a minimal useful set could be:

ttft_ms
prefill_tokens_per_second
decode_tokens_per_second
total_time_ms

Optionally, exposing the underlying token counts and durations would allow clients to calculate additional metrics themselves:

prefill_tokens
prefill_time_ms
decode_tokens
decode_time_ms

Alternatives considered

Alternatives Considered

1. Measure Metrics on the Client Side

Clients can estimate metrics such as TTFT and decode throughput by recording timestamps while consuming the HTTP or SSE response.

For example:

• TTFT can be estimated from request start time to the first streamed token.
• Decode throughput can be estimated from the number of generated tokens divided by the elapsed generation time.

However, this approach has several limitations:

• Network latency is included in the measurement.
• Client-side scheduling and buffering may affect timing.
• The client cannot accurately determine when the internal prefill stage starts or finishes.
• Prefill throughput cannot be measured precisely from the client side.
• Results may vary between HTTP clients and environments.

Because of this, client-side measurements are useful for end-to-end latency, but they are less suitable for accurate inference-engine benchmarking.

2. Expose Metrics Only Through Server Logs

Another option would be to log prefill and decode performance information on the server side.

For example:

Prefill: 4992.7 tokens/s
Decode: 49.99 tokens/s
TTFT: 320.5 ms

This is useful for debugging, but it is less convenient for API consumers because:

• Clients cannot directly associate metrics with individual requests.
• External monitoring systems would need to parse server logs.
• It is difficult to build dashboards or automated benchmarks on top of logs.
• Logs may be unavailable to remote API consumers.

3. Expose Metrics Only Through a Monitoring Endpoint

Performance statistics could also be exposed through a Prometheus-compatible /metrics endpoint.

This would be useful for system-level monitoring and aggregate statistics, but it would not fully solve the per-request use case.

For example, aggregate metrics are useful for observing:

• Overall token throughput
• Average latency
• GPU utilization
• Request queueing
• Cache hit rates

However, they do not allow a client to directly determine the performance characteristics of a specific /v1/chat/completions request.

4. Return Only a Single TPS Value

Another simpler option would be to expose only one tokens_per_second value.

However, a single TPS metric can be ambiguous because prefill and decode have very different performance characteristics.

For example:

Prefill: 5000 tokens/s
Decode: 50 tokens/s

Combining these into one number makes it difficult to understand whether a performance issue comes from prompt processing or token generation.

Therefore, exposing separate prefill and decode metrics would provide much more useful information.

Preferred Approach

The preferred approach is to expose optional per-request performance metrics directly in the /v1/chat/completions response while keeping server logs and monitoring endpoints available for aggregate observability.

This would provide accurate engine-side measurements while also making the metrics easy to consume for benchmarking, monitoring, and external tooling.