common, server : report model input/output modalities in GET /models by angt · Pull Request #29987 · ggml-org/llama.cpp

angt · 2026-10-05T10:07:41Z

Overview

Proposal to add an architecture object to GET /v1/models and GET /models.
This object gives the model input and output modalities.

Additional information

Supersedes #29944

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

angt · 2026-10-05T12:33:38Z

@ngxson This PR may do a bit more than needed but I think the changes are still reasonable.
But I can make it smaller by keeping the existing input-modality handling and only updating output_modalities, if you prefer.

ngxson

I feel like the impl is a bit convoluted: same notion is spread out to multiple files and ti's unclear about the data flow

Instead, I might be better to:

Have a struct server_model_architecture and pass the struct around, instead of passing json
server_model_architecture::to_json() renders it into json, same idea as server_task_result
server_model_architecture constructor can take either const llama_model * (server_context case) or string model_path (router case)

Also, additions to docs should be minimal, since it's an user-facing docs

ngxson · 2026-10-05T14:18:14Z

+Check for membership. Tolerate values that you do not know:
+
+```js
+const useSystemOne =
+    model.architecture?.output_modalities?.includes("decisions") === true;
+```
+
+Without decision metadata, `output_modalities` is `["text"]`. This default is for compatibility only. It does not mean that the model can generate text. Values can change. New combinations such as `["text", "decisions"]` use the same shape.
+
+The router returns the same `architecture` object in [`GET /models`](#get-models-list-available-models). You can find a native decision model without a probe or a model load. This works for unloaded and sleeping models too. Older servers can omit `architecture`. If it is absent, use the legacy behavior of your client.


maybe simplify this part to only specify core technical specs, for example "Tolerate values" and "Older servers can omit" are trivial to know and don't need to be in docs

ngxson · 2026-10-05T14:19:28Z

+
+The values work like this:
+
+- A loaded model reports both arrays. Its values replace the cached values in full.
+- The cache keeps the values across sleep and unload. A known decision model stays advertised.
+- Before the first report, the values come from the offline computation.
+- Offline computation cannot see video. Only a loaded model reports `video` in `input_modalities`.
+- If the metadata or the model file is not available, both arrays are `["text"]`.
+- A source or preset refresh computes both arrays again. A replaced model does not keep old values.


remove redundant impl details in docs, they will add maintenance costs

ngxson · 2026-10-05T14:20:11Z

+// model output modalities
+//
+
+std::vector<std::string> server_model_output_modalities(common_decision_type decision_type) {


this should be part of server-decision ?

ngxson · 2026-10-05T14:27:58Z

+        /* model_aliases           */ impl->model_aliases,
+        /* model_tags              */ impl->model_tags,
+        /* model_path              */ impl->params_base.model.path,
+        /* model_output_modalities */ server_model_output_modalities(common_get_decision_type(impl->model_tgt)),


the whole architecture object should be stored here, not just the output part

ngxson · 2026-10-05T14:29:27Z

+            // the child replaces both arrays in full; a bad or missing value changes nothing
+            if (args.loaded_info.contains("architecture") && args.loaded_info.at("architecture").is_object()) {
+                const json & child_arch = args.loaded_info.at("architecture");
+                for (const char * key : { "input_modalities", "output_modalities" }) {
+                    if (!child_arch.contains(key) || !child_arch.at(key).is_array()) {
+                        continue;
+                    }
+                    std::vector<std::string> modalities;
+                    bool valid = true;
+                    for (const auto & m : child_arch.at(key)) {
+                        if (!m.is_string()) {
+                            valid = false;
+                            break;
+                        }
+                        modalities.push_back(m.get<std::string>());
+                    }
+                    if (valid) {
+                        meta.architecture[key] = std::move(modalities);
+                    }
+                }
+            }
        }


not sure if we really need this part? looks quite overkill, ~20 LOC for an edge case that may never happen in reality

ngxson

let's merge this version to have it in v0.6 release, I will push a follow-up to clean up the internal code

cc @ggerganov

angt · 2026-10-05T15:14:36Z

The cleanup was arriving, sorry 😅

angt · 2026-10-05T15:16:06Z

I was in agreement with all the points but testing takes time: https://github.com/angt/llama.cpp/tree/common-server-report-model-input-output-modalities-in-get-models-minimal

…alities llama.cpp 0.6.0 reports decisions in architecture.output_modalities of GET /models (ggml-org/llama.cpp#29987), including for unloaded and sleeping models. Use it instead of probing /v1/systemone and remembering probe results across sessions.

ggml-org#29903 also enables embedding mode for the laya/kev/clef decision models, but that half calls common_get_decision_type() and references the COMMON_DECISION_TYPE_* enums, which arrived in the post-0.5.0 commits a4cb4c6 (ggml-org#29818) and 4d60b4d (ggml-org#29987) and are absent here: common.cpp:1345: error: 'common_get_decision_type' was not declared in this scope Keep the n_batch <= n_ubatch clamp, which is self-contained and the part the commit is named for. The decision-model block can return once those upstream commits are merged. Assisted-by: Hermes Agent

…gml-org#29987) Signed-off-by: Adrien Gallouët <angt@huggingface.co>

common, server : report model input/output modalities in GET /models

02779b5

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

angt requested review from a team as code owners October 5, 2026 10:07

github-actions Bot added documentation Improvements or additions to documentation server labels Oct 5, 2026

ngxson reviewed Oct 5, 2026

View reviewed changes

ngxson approved these changes Oct 5, 2026

View reviewed changes

ngxson merged commit 4d60b4d into ggml-org:master Oct 5, 2026
12 checks passed

ggerganov mentioned this pull request Oct 5, 2026

llama.cpp : bump version to 0.6.0 #29997

Merged

This was referenced Oct 5, 2026

common, server : add model type to GET /models #29944

Closed

server : group the model modalities in one struct #30011

Closed

ngxson mentioned this pull request Oct 5, 2026

server: refactor modalities handling #30015

Open

angt deleted the common-server-report-model-input-output-modalities-in-get-models branch October 6, 2026 06:28

Nihmar mentioned this pull request Oct 8, 2026

Align the embedded web UI and its server API with llama.cpp v0.6.0 Nihmar/omphalos#395

Closed

6 tasks

bappisinsane mentioned this pull request Oct 10, 2026

changelog : llama-server REST API #9291

Open

thelittlefireman pushed a commit to thelittlefireman/llama.cpp that referenced this pull request Oct 12, 2026

common, server : report model input/output modalities in GET /models (g…

57e2c88

…gml-org#29987) Signed-off-by: Adrien Gallouët <angt@huggingface.co>

Navigation Menu

Repository navigation

common, server : report model input/output modalities in GET /models - #29987

common, server : report model input/output modalities in GET /models#29987
ngxson merged 1 commit into
ggml-org:masterfrom
angt:common-server-report-model-input-output-modalities-in-get-models

angt commented Oct 5, 2026

Uh oh!

angt commented Oct 5, 2026

Uh oh!

ngxson left a comment

Uh oh!

ngxson Oct 5, 2026

Uh oh!

ngxson Oct 5, 2026

Uh oh!

ngxson Oct 5, 2026

Uh oh!

ngxson Oct 5, 2026

Uh oh!

ngxson Oct 5, 2026

Uh oh!

ngxson left a comment

Uh oh!

Uh oh!

angt commented Oct 5, 2026

Uh oh!

angt commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

angt commented Oct 5, 2026

Overview

Additional information

Requirements

Uh oh!

angt commented Oct 5, 2026

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

ngxson Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

ngxson Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

ngxson Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

ngxson Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

ngxson Oct 5, 2026

Choose a reason for hiding this comment

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

angt commented Oct 5, 2026

Uh oh!

angt commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants