[TODO] Re-introduce semantic cache (deferred)

[moonming]

Context

Stage 4b (pgvector semantic cache) shipped on AISIX-Cloud, but the
DP-side client + chat-dispatch integration were never written.
Customers selecting "Semantic" in the dashboard got a non-functional
option. This issue tracks the work needed to bring it back, after we
remove the half-shipped surface.

The removal landed in:

• AISIX-Cloud#198 — cp-api side (/dp/cache/{lookup,put},
  cache_entries_semantic table, dashboard form fields,
  pgvector / redis_semantic / qdrant enum values).
• ai-gateway#130 — DP side (CacheBackend enum trimmed to
  Memory + Redis, QdrantCacheConfig, schema, stale comments).

PR #126 (the abandoned Draft of a SemanticBackend HTTP client) was
closed when these landed — its wire shape was wrong (designed against
the issue sketch, not the real cp-api endpoints).

What needs to be decided before re-introduction

1. Where does the embedding ProviderKey live?

Cloud's existing schema didn't have an embedding_provider_key_id
on cache_policies. Three options:

• A: New schema column on cache_policies —
  embedding_provider_key_id UUID REFERENCES env_provider_keys(id).
  Dashboard form gains a dropdown of provider keys whose provider
  supports embeddings (OpenAI / Cohere / Bedrock). Cleanest, most
  explicit, but biggest schema + UI surface.
• B: DP reuses the chat request's own ProviderKey. Implicit;
  works for OpenAI but breaks for Anthropic (no embeddings endpoint).
• C: Global env var on the DP (AISIX_EMBEDDING_API_KEY).
  Single-tenant only; not viable for multi-org SaaS.

Recommend A. B is too magical (the same ProviderKey doing chat

• embeddings is a coincidence, not an invariant); C breaks the
  multi-tenant story.

2. Hardcode 1536 dim, or per-policy vector(N)?

Cloud's cache_entries_semantic.embedding was vector(1536),
implicitly tying every customer to text-embedding-3-small. To
support text-embedding-3-large (3072) or Cohere embed-english-v3.0
(1024) we'd need either:

• Multiple vector(N) columns on the same table (one per dim).
• Sharded tables, one per dim.
• One table per (env, dim) tuple (overkill).

Recommend the multi-column approach — pgvector allows null on
unused dims; the schema stays single-table; the lookup path picks
the column by the policy's embedding_model name.

3. Multi-tenant scope on /dp/cache/lookup

Cloud's old endpoint scoped lookups by policy_id (the cache_policy
UUID). That's correct: the policy already pins the env via its FK.
The DP just sends policy_id + embedding. Keep this — no scope
field on the wire.

4. Embedding-provider failure semantics

If the DP's embedding call to OpenAI/Cohere/Bedrock errors or times
out:

• A: Treat as cache miss, dispatch upstream chat normally. Add
  a aisix_semantic_embedding_failures_total{cause} Prometheus
  counter for ops. The cache is an optimization layer; failures
  must not propagate to the user.
• B: Return 5xx to the user.

A.

5. Dashboard UX for the embedding dropdown

Once #1 is decided, the cache-policies form needs:

• A new "Embedding provider key" dropdown when backend is semantic.
• Filtering: only show provider keys whose provider supports
  embeddings (OpenAI / Cohere / Bedrock).
• A guard that prevents selecting a provider key that's been deleted
  out from under the policy.

Re-introduction work breakdown (estimate)

• CP: re-add cache_entries_semantic (multi-dim) + the two
  mTLS endpoints. ~2 days.
• CP: embedding_provider_key_id column + validation +
  dashboard dropdown. ~2 days.
• DP: EmbeddingClient against the chosen ProviderKey. Needs
  OpenAI / Cohere / Bedrock variants. ~3 days.
• DP: SemanticBackend HTTP client against the real
  /dp/cache/{lookup,put} wire. ~1 day.
• DP: chat::dispatch integration — recognise
  policy.backend == Semantic, compute embedding, look up,
  fall through on miss, store on success. ~2 days.
• E2E: Playwright + live DP that creates a Semantic policy,
  sends two semantically-similar prompts, asserts the second
  hits cache. ~1 day.

Total: ~11 dev-days for an MVP that supports OpenAI embeddings
end-to-end. Multi-provider embedding support extends the DP work.

Why deferred now

The Stage 4b ship was technically complete on cp-api but customer-
facing nonfunctional. We'd rather have a clear "not implemented"
absence than a "configured but doesn't work" silent feature. When
product / customers prioritize this, we re-open this issue and pick
up from the breakdown above.

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• [P1] aisix-cache has no semantic backend (Cloud Stage 4b shipped, DP missing) #116

🔗 Related PRs

#41 - feat(proxy): move embed api to new provider [merged]
#84 - feat(observability): trace span for embeddings api [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[jarvis9443]

Shipped: #918 (in-process semantic layer) + #921 (shared Redis vector store), with the CP surface in api7/AISIX-Cloud#1283 and docs in api7/docs#2060. See api7/AISIX-Cloud#558 for the delivery summary.