Skip to content

/v1/embeddings ignores prompt_name / task_type, so every input is embedded as a query #758

Open

Description

@noamsto

What happens

/v1/embeddings and /api/embeddings always embed with task_query, whatever the request asks for. For embed-gemma this silently selects the wrong prefix. A retrieval index built through the server stores every document with the query prefix (task: search result | query: ) instead of title: none | text: .

flm serve llama3.2:1b --embed 1
for t in query document; do
  curl -s http://127.0.0.1:52625/v1/embeddings -H 'Content-Type: application/json' \
    -d "{\"model\":\"embed-gemma:300m\",\"input\":\"The capital of France is Paris.\",\"prompt_name\":\"$t\"}" \
    | jq -c '.data[0].embedding[:4]'
done
# identical vectors

On embed-gemma:300m, cosine(query, document) for the same text is 1.000000. It should differ, because the prefixes differ. With the task passed through it is 0.957706.

An unknown name ("prompt_name":"not_a_task") or a non-string ("prompt_name":7) is also accepted, and the server answers 200 with a query embedding.

Cause

The engine already supports every task. embedding_task_type_t has ten values (auto_embedding_model.hpp:33-44), and Gemma_Embedding::_get_task_prefix maps each one to its prefix (modeling_gemma_embedding.hpp:13-54). The REST handler never reads the request's task and hardcodes embedding_task_type_t::task_query (rest_handler.cpp:900).

Expected

Read prompt_name, with task_type accepted as an alias, and pass the matching task to embed(). Refuse an unknown or non-string value with a 400 instead of embedding with a different task. A request with no task keeps today's task_query default.

Fix we are carrying

https://github.com/noamsto/nix-amd-ai/blob/46d7a73c1044435cec0ff08410e1a3bc87cbc544/pkgs/fastflowlm/patches/embed-task-prompt.patch

It maps the REST names (query/search_query, document/search_document, clustering, classification, STS, and so on) onto embedding_task_type_t and passes the result to embed(). It refuses unknown, non-string, or conflicting prompt_name/task_type values with 400 invalid_value. One resolved task applies to every element of an input array. The mapping and validation are ported from OpenFlowLM-Next's SERVER-EMBED-TASK-PROMPT (openai_compat::task_names/resolve_task/task_policy, Vegard Berget). The helpers need a FASTFLOWLM_LINUX_LIMITED_MODELS guard, as the patch shows. I can send this as a PR against main if that's useful.

Probe logs, before and after: https://github.com/noamsto/nix-amd-ai/blob/46d7a73c1044435cec0ff08410e1a3bc87cbc544/bench-logs/oflm-api-conformance-2026-09-28-after-174/README.md

Environment

FLM v1.0.6 built from the tag, with our downstream request-handling patches, none of which touched this call before the fix. Model: embed-gemma:300m via --embed 1.
Ryzen AI MAX+ 395 (Strix Halo), XDNA2 NPU, 8 columns. NixOS, Linux 7.2.8, amdxdna 0.10, NPU FW 1.1.2.65.

This is separate from the embedding-quality reports in #661, #647 and #732. It is about which prefix gets applied, not about the kernel output.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions