[API Proposal]: Add and remove tools mid-conversation without invalidating the prompt cache #7822

Background and motivation

[API Proposal]: Add and remove tools mid-conversation without invalidating the prompt cache

Background and motivation

Tool definitions sit at the start of every request, ahead of the conversation, so changing ChatOptions.Tools partway through a conversation changes the prompt prefix and invalidates the prompt cache. With cached input priced at around a tenth (in some cases, even less) of uncached input on current models, a conversation that gains a tool after a long history pays for that whole history again.

Providers now accept a tool change at a position in the conversation, leaving everything before it, the tools array included, unchanged:

There's no way to express this through IChatClient today. It is a different thing from deferral (HostedToolSearchTool.DeferredTools, #7371): a deferred tool is declared up front and loaded later, while a positional change introduces or withdraws a tool at a point in the history.

Some places where tools appear partway through a conversation: an agent gains a reply_to_user tool once the user messages it mid-run; a skill or plugin is loaded on demand; an MCP server connects or reconnects; a capability unlocks after a login or approval.

API Proposal

namespace Microsoft.Extensions.AI;

[Experimental(DiagnosticIds.Experiments.AIToolChanges)]
public sealed class ToolAdditionContent : AIContent
{
    public ToolAdditionContent(AIFunctionDeclaration tool);

    // Serialized as name, description and JSON schemas; deserializes to a declaration.
    public AIFunctionDeclaration Tool { get; }
}

[Experimental(DiagnosticIds.Experiments.AIToolChanges)]
public sealed class ToolRemovalContent : AIContent
{
    public ToolRemovalContent(string toolName);

    public string ToolName { get; }
}

Semantics:

• The content says when the model was told about a tool. ChatOptions.Tools stays the set of tools that can be invoked. FunctionInvokingChatClient doesn't change: it already resolves calls from ChatOptions.Tools on every iteration.
• Adapters with a positional form leave out of the top-level tool definitions any tool in ChatOptions.Tools whose first change in the history is a ToolAdditionContent. They send the content's definition at its position instead. Every other tool is declared up front as today.
• Adapters without one apply the changes to the tool definitions they send: they drop tools whose last change is a removal and add tools whose last change is an addition. They then remove the content from the history. This costs a cache miss but keeps behavior correct.
• The definition sent at the position comes from the content, not from ChatOptions.Tools. That keeps the earlier part of the conversation identical from request to request, including after the tool is taken out of ChatOptions.Tools again.

API Usage

IChatClient client = inner.AsBuilder().UseFunctionInvocation().Build();
AIFunction replyToUser = AIFunctionFactory.Create(ReplyToUser);

// A message arrives mid-run: make the tool invocable, and tell the model about it here.
options.Tools = [.. options.Tools, replyToUser];
history.Add(new(ChatRole.System,
[
    new TextContent("The user sent a message: \"Also, say hi!\" Answer it with ReplyToUser."),
    new ToolAdditionContent(replyToUser),
]));

ChatResponse response = await client.GetResponseAsync(history, options);

The draft PR (#7823) implements this for the OpenAI Responses client (additional_tools) and the Chat Completions client (the apply-to-tools fallback). A matching prototype of the Anthropic adapter, which lives in anthropics/anthropic-sdk-csharp, maps the content to tool_addition / tool_removal. Both were checked against the live APIs using the same scenario: a long prefix, then a tool introduced by a mid-conversation message, compared with adding the same tool to ChatOptions.Tools. The table shows cached input tokens on the first request after the change:

In every run, FunctionInvokingChatClient invoked the tool the history introduced.

Alternative Designs

• FunctionInvokingChatClient invokes functions found in the history. This makes the conversation carry the ability to run code, and it breaks for persisted histories, because an AIFunction can't be deserialized. A second registry would be needed anyway.
• A per-request list of tools that can be invoked but aren't declared, such as a ChatOptions counterpart to FunctionInvokingChatClient.AdditionalTools. That's a second tool list with subtle semantics. With the proposal, ChatOptions.Tools already serves that purpose, and the history decides where each tool is declared.
• Content that carries only the tool name, with the definition taken from ChatOptions.Tools. Once the tool is removed from ChatOptions.Tools, the earlier addition can't be replayed identically, which breaks the cache from that point.
• A tool-change list on ChatOptions, keyed by message index. Positions in a separate list drift when callers trim or compact the history. Content moves with its message.

Risks

• Silent drop. Adapters ignore AIContent they don't know, so an adapter that doesn't implement this would hide an added tool from the model. A shared helper that applies the fallback would make that one call per adapter. Should it be public, so adapters outside this repository (Anthropic, Gemini) can use it? The draft PR keeps it internal to the OpenAI library.
• Removal on the Responses API. It has no removal item. The draft doesn't send ToolRemovalContent there, and FunctionInvokingChatClient reports a call to a tool no longer in ChatOptions.Tools as not found. Narrowing with tool_choice.allowed_tools is an option, but it interacts with ChatOptions.ToolMode.
• Function declarations only. Hosted tools (for example HostedWebSearchTool) can't be serialized, so they're left for later.
• Placement rules are the provider's. For example, Anthropic accepts tool changes only directly after a user turn. Adapters map the content where the caller put it and let the API reject an invalid placement, rather than moving it.
• Experimental registration. The types are registered the way other experimental AIContent types were before they were stabilized: in AIJsonUtilities.DefaultOptions, not as [JsonDerivedType] on AIContent.

Generated with Claude Opus 5.5 (Claude Code)

API Proposal

namespace System.Collections.Generic;

public class MyFancyCollection<T> : IEnumerable<T>
{
    public void Fancy(T item);
}

API Usage

// Fancy the value
var c = new MyFancyCollection<int>();
c.Fancy(42);

// Getting the values out
foreach (var v in c)
    Console.WriteLine(v);

Alternative Designs

No response

Risks

No response