[Feature] Add relay model diagnostics for connectivity, latency, and identity consistency #2834

Area

Multiple areas

What are you trying to accomplish?

This request is for a safe, unified diagnostic workflow for custom relay providers and the models exposed through them. A relay being reachable does not necessarily mean that a configured model can be called successfully, that the request is routed to the requested model, or that the endpoint supports the model capabilities OpenCodex expects.

The desired outcome is to let users quickly answer: “Can this provider/model be used right now, how fast is it, and is the observed response consistent with my configuration?” This is especially valuable for OpenAI-compatible relays and third-party gateways where model names, routing, fallbacks, and supported features may differ from the upstream service.

A related request is #1765, but this proposal expands the scope from a simple availability check to a provider-focused diagnostic view with latency, response metadata, capability checks, and routing-risk signals.

What prevents this today?

Today users have to select and invoke providers/models one at a time, use ad-hoc curl commands, or inspect raw logs. This makes it difficult to compare custom relays and to distinguish:

• network/DNS/TLS failures from authentication failures;
• an unavailable model from rate limiting or quota exhaustion;
• a slow provider from a slow first token or slow generation;
• a configured model ID from the model ID actually returned by the endpoint;
• a genuine capability mismatch from a silent fallback or model substitution.

Provider catalog information alone cannot expose these cases. A manual test prompt can consume credits or accidentally send user content. Users currently lack a consistent, low-risk way to verify custom relay behaviour before relying on it for real work.

What should OpenCodex do?

The diagnostic workflow should be available from the UI, with an optional CLI/JSON interface, and should support testing one model, all models for one provider, or all configured custom providers.

1. Lightweight connectivity check

When possible, provide a check that does not run a real completion. Report endpoint reachability, DNS/TLS/HTTP status, authentication validity, API compatibility, and whether the configured model can be discovered from the provider catalog or endpoint.

2. Controlled minimal request test

When the user explicitly starts an active test, send a fixed, very short, non-private request. Clearly distinguish this from the lightweight check and warn that it may consume quota or incur a small charge. Use bounded concurrency, limited retries, cancellation, and no conversation history, files, secrets, or user prompts. Record connection time, time to first token (TTFT), total response time, HTTP status, and a redacted error category. Preserve the last result and timestamp.

3. Identity, routing, and capability consistency

Use observable evidence to flag discrepancies between configuration and behaviour, including:

• configured provider/model ID versus the response model field;
• resolved provider/model route and fallback information;
• provider metadata and declared capabilities;
• streaming, tool-calling, vision, reasoning, and context-window behaviour;
• usage fields, including prompt/completion tokens and cache-read/cache-write tokens when supplied;
• response metadata across repeated or cross-model checks.

Suggested states: Healthy / Consistent; Reachable but identity unconfirmed; Model ID mismatch; Possible fallback or model substitution; Capability mismatch; Authentication failure; Model unavailable; Rate limited; Quota exhausted; Endpoint incompatible; Timeout.

Without provider-signed attestation, OpenCodex cannot mathematically prove a model’s internal identity. The goal is to identify observable inconsistencies and substitution risks, not to make an absolute “real/fake model” claim.

Suggested UI and CLI

Provide “Test this model”, “Test all models for this provider”, and “Test all custom providers”. Show configured model, status, returned model, connect time, TTFT, total time, and risk/notes. A history view should show last check time and recent latency trends.

For automation, consider:

ocx providers diagnose --provider example-relay --model claude-opus-5
ocx providers diagnose --provider example-relay --all-models
ocx providers diagnose --all --json

JSON output would allow scripts or routing decisions to consume the results.

Example usage or interface

The exact interface is up to the maintainers, but an example workflow could be:

1. Open Providers and choose a custom relay.
2. Click “Test all models”.
3. Run lightweight checks first.
4. Ask for confirmation before active completion tests because they may consume quota.
5. Show per-model status, returned model ID, connection latency, TTFT, total latency, token usage, cache fields when available, and a redacted explanation for failures.
6. Allow rerunning one model and exporting results as JSON.

Example output:

Provider: example-relay
Configured model       Status                    Returned model       Connect   TTFT     Total
claude-opus-5          Possible fallback        claude-sonnet-4      240 ms    820 ms   3.2 s
gpt-5.6-sol            Healthy / Consistent     gpt-5.6-sol          180 ms    410 ms   1.8 s
gemini-2.5-pro         Unavailable              —                    190 ms    —        —

The diagnostic result should never automatically delete, disable, rewrite, or remap a provider/model because of one transient failure. It should report evidence and let the user decide.

Alternatives or workarounds

Current workarounds are manual model calls, curl/Postman requests, provider dashboards, and custom scripts. These can confirm that something responds, but they do not provide a consistent comparison across providers, do not always expose TTFT or returned model metadata, and can accidentally send private content or consume more quota than expected.

A single OpenCodex workflow could reuse the provider catalog, routing resolver, request/usage logging, and capability metadata, giving users one place to understand whether a custom relay is reachable, usable, slow, incompatible, or behaving inconsistently.

Additional context

Related: #1765

Security and privacy expectations:

• never include API keys, account identifiers, conversation content, files, or raw sensitive headers;
• redact provider error messages before displaying or persisting them;
• make active tests manual by default;
• bound concurrency, retries, request size, and test frequency;
• clearly label estimates and provider-reported fields;
• treat missing model identity metadata as “unconfirmed”, not automatically as fraud;
• support providers that do not expose a model catalog or cache fields.

This could start as a read-only diagnostics panel and grow incrementally: connectivity and latency first, then capability checks and evidence-based routing warnings. It would provide immediate value to custom-relay users without requiring a large all-at-once redesign.

Checks

• I searched existing issues and documentation.
• This request describes a concrete OpenCodex workflow rather than merely naming a desired technology.
• I removed secrets and personal data.