Repository navigation
Tool Calls and Non-Text Generated Data #81
Description
Hi,
Many of the tools I'm creating will return a text response but will ultimately be used to generate various large data or custom object models that would be visualized to the user in chat..
On platforms like Open-WebUI, this generally works in a way where there is a "sources" or "attachments" key that comes in the final output for the streamed response. This allows showing companion data to the streamed text response.
I may be missing something, but I'm not seeing a way for this type of data to be carried through after Tool calls.
As I see it right now, I'd have to store this data as the tool call is made and look for it separate form this library.
Can you please confirm if there is any built in support for this? I think this would be a where the Tool Output should support some kind of sidecar passthrough data that wouldn't be converted to a key/value JSON text response.
Thanks
Activity
tattn commentedon Oct 27, 2025
I hadn’t considered a mechanism for retaining data as context that wouldn’t be passed to the LLM.
As you mentioned, it’s possible to manage such data on the app side, which allows for more flexible implementation.
james-333i commentedon Oct 27, 2025
I think there are a number of valid use cases including:
a) Files that tools access to get contextual data from could be presented as "references"
b) Generating / editing files
c) Sidecar JSON objects that would be used to present something visually to the user but not used in text responses.
Examples:
A tool that reads analytics data (text) could be paired with a tool that then creates an excel file from that data with graphs.
A tool that finds files that match a specific criteria. The LLM would just be prompted to respond with something like "Here are the results" and then the sidecar data would be presented to the user.
Along these same lines, currently for input only image attachments are supported and not other file attachments and it assumes otherwise any attachment is meant as true context for the LLM and not for tool use. It is common to have input data that is specifically for tools and not necessarily needed to be processed by the LLM.
Certainly, the developer could make each tool look for global input data and store global output data but in ideal scenarios, tools are designed to be chained so they would want to pass data from one tool to another and then back to the LLM and passthrough as well.