Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/architecture/model-capabilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,9 @@ A chat is Basic (Q&A), the chat engine's retrieval answers with Studio beside it
- `capability_level` on each row of `GET /llm/connections/{id}/models` and `GET /llm/providers/llamacpp/models` that can fill the chat slot.
- `POST /workspaces/{id}/chat/threads` takes `mode` ([chat](chat.md#agent-threads)).

The model picker labels each chat model by how it did in the Agentic tests: "Agentic", "Agentic, may need nudges", "Low Agentic score" or "Not tested". Settings › Models shows the chat model's level, its evidence line and its note, or for an assumed row only "Not run: SurfSense expects this expensive flagship model to pass."; then the mode new chats start in, and either that the mode is chosen per chat in the composer or why Agentic is not available.
In dev builds only, the model picker tags each chat model by how it did in the Agentic tests: "Agentic", "Agentic, may need nudges", "Low Agentic score" or "Not tested"; production shows no tag. Settings › Models shows the chat model's level, its evidence line and its note, or for an assumed row only "Not run: SurfSense expects this expensive flagship model to pass."; then the mode new chats start in, and either that the mode is chosen per chat in the composer or why Agentic is not available.

The composer's switch ([`mode-picker.tsx`](../../surfsense_local/frontend/src/features/chat/modes/mode-picker.tsx)) is a menu of "Basic (Q&A)" and "Agentic", each with one line on what it does. On a new chat it starts at the model's `default_mode` and follows the model when it changes; a pick holds for that model for the session, in a context above the open workspace. The new thread's `mode` is always sent, with `remember: true` only when the user picked it, so only a pick becomes the model's default. Beside Agentic it says the `reason`: "Passed 2 of 8 Agentic tests, so it may stop early or make mistakes.", "Not tested with this model yet, so it may stop early or make mistakes.", "Passed on its full-size version. A local copy may do worse.", or that a flagship is assumed; once Agentic is picked, the same line shows under the composer, except for a model that passed. A gated Agentic is disabled with its reason, such as "This model can’t use tools." In an open chat the switch shows the chat's mode, "This chat runs in Agentic mode", and offers only "Start a new chat in Basic (Q&A) mode" (or Agentic), which picks that mode and opens a new chat. A refused Agentic chat is worded by its code in a toast, and the question and images go back into the composer. A refusal by a gate the selection could not say, such as a local model's tool calls, disables Agentic with its reason for that model for the session ([`new-chat-mode.ts`](../../surfsense_local/frontend/src/features/chat/modes/new-chat-mode.ts)), so the next chat starts in Basic (Q&A) rather than asking again. The agent reads no attached image yet, so an Agentic chat, new or open, takes none ([chat](chat.md)).
The composer's switch ([`mode-picker.tsx`](../../surfsense_local/frontend/src/features/chat/modes/mode-picker.tsx)) is a menu of "Basic (Q&A)" and "Agentic", each with one line on what it does. On a new chat it starts at the model's `default_mode` and follows the model when it changes; a pick holds for that model for the session, in a context above the open workspace. The new thread's `mode` is always sent, with `remember: true` only when the user picked it, so only a pick becomes the model's default. On a new chat each mode gets a short line, and an Agentic that did not pass carries a tag ("Not tested", "Low score", "May need nudges", "Local copy", "Expected to pass"); the `reason`, or a gate, shows in a tooltip beside the Agentic row on hover or focus and is read with the row, in both menus: "Passed 2 of 8 Agentic tests, so it may stop early or make mistakes.", "Not tested with this model yet, so it may stop early or make mistakes.", "Passed on its full-size version. A local copy may do worse.", or that a flagship is assumed. Nothing is said under the composer. A gated Agentic is disabled with its reason, such as "This model can’t use tools." In an open chat, under "This chat’s mode", the switch lists the same two rows with the chat's mode checked; the other is marked "New chat", and picking it picks that mode and opens a new chat. A refused Agentic chat is worded by its code in a toast, and the question and images go back into the composer. A refusal by a gate the selection could not say, such as a local model's tool calls, disables Agentic with its reason for that model for the session ([`new-chat-mode.ts`](../../surfsense_local/frontend/src/features/chat/modes/new-chat-mode.ts)), so the next chat starts in Basic (Q&A) rather than asking again. The agent reads no attached image yet, so an Agentic chat, new or open, takes none ([chat](chat.md)).

## Known gaps

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ import { intl } from "@/i18n/intl"

import { ComposerImage } from "./attached-image"
import { ComposerAddMenu } from "./composer-add-menu"
import { AgenticNote } from "./modes/agentic-note"
import { ModePicker } from "./modes/mode-picker"
import { useNewChatMode } from "./modes/new-chat-mode"
import { ModelPicker, modelControlButtonClassName } from "./model-picker"
Expand Down Expand Up @@ -288,9 +287,6 @@ export function ChatComposer({
{limitNotice}
</p>
) : null}
{placement === "center" ? (
<AgenticNote model={model} mode={newChatMode} />
) : null}
{placement === "bottom" ? (
<div className="mt-1 flex min-h-7 items-center justify-between gap-3 px-2">
{/* Keeps its line and leaves the mode and the model name what is
Expand Down
11 changes: 8 additions & 3 deletions surfsense_local/frontend/src/features/chat/model-picker.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { useRef, useState } from "react"
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"

import { Badge } from "@/components/ui/badge"
import {
DropdownMenu,
DropdownMenuContent,
Expand Down Expand Up @@ -231,10 +232,14 @@ export function ModelPicker({
</span>
) : null}
</span>
{candidate.capability_level ? (
<span className="shrink-0 text-[11px] text-muted-foreground">
{/* Dev builds only; production never shows this badge. */}
{import.meta.env.DEV && candidate.capability_level ? (
<Badge
variant="secondary"
className="h-4 px-1.5 text-[10px]"
>
{capabilityLabel(candidate.capability_level)}
</span>
</Badge>
) : null}
</DropdownMenuRadioItem>
)
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { afterEach, describe, expect, it, vi } from "vitest"
import { cleanup, screen, within } from "@testing-library/react"
import { cleanup, screen, waitFor, within } from "@testing-library/react"
import userEvent from "@testing-library/user-event"
import { AssistantRuntimeProvider, useLocalRuntime } from "@assistant-ui/react"

Expand Down Expand Up @@ -130,7 +130,7 @@ describe("the composer's mode switch", () => {
expect(composerNote()).toEqual([])
})

it("starts an untested model in Basic, and warns once Agentic is picked", async () => {
it("starts an untested model in Basic, and warns only in the menu", async () => {
const user = userEvent.setup()
renderComposer({ model: SONNET_99 })

Expand All @@ -145,9 +145,41 @@ describe("the composer's mode switch", () => {

expect(trigger().textContent).toContain("Agentic")
expect(nextChat()).toBe("agentic")
expect(composerNote()).toEqual([
"Not tested with this model yet, so it may stop early or make mistakes.",
])
expect(composerNote()).toEqual([])
})

it("tags an untested Agentic, and says the full warning beside the row", async () => {
const user = userEvent.setup()
renderComposer({ model: SONNET_99 })

await user.click(trigger())
const agentic = await screen.findByRole("menuitemradio", {
name: /^Agentic/,
})
await user.hover(agentic)

expect(within(agentic).getByText("Not tested")).toBeTruthy()
expect(agentic.textContent).toContain("Does tasks across your files")
// Once read with the row, once in the tooltip.
await waitFor(() =>
expect(
screen.getAllByText(
"Not tested with this model yet, so it may stop early or make mistakes."
)
).toHaveLength(2)
)
})

it("leaves a passing model's Agentic untagged", async () => {
const user = userEvent.setup()
renderComposer({ model: KIMI })

await user.click(trigger())
const agentic = await screen.findByRole("menuitemradio", {
name: /^Agentic/,
})

expect(within(agentic).queryByText("Not tested")).toBeNull()
})

it("offers a low scorer Agentic with its score", async () => {
Expand Down Expand Up @@ -178,13 +210,15 @@ describe("the composer's mode switch", () => {
})

await user.click(trigger())
await user.click(
await screen.findByRole("menuitemradio", { name: /^Agentic/ })
const agentic = await screen.findByRole("menuitemradio", {
name: /^Agentic/,
})
expect(agentic.textContent).toContain(
"Passed on its full-size version. A local copy may do worse."
)
await user.click(agentic)

expect(composerNote()).toEqual([
"Passed on its full-size version. A local copy may do worse.",
])
expect(composerNote()).toEqual([])
})

it("holds Agentic out with the reason when a gate blocks it", async () => {
Expand Down Expand Up @@ -245,28 +279,39 @@ describe("the composer's mode switch", () => {
expect(nextChat()).toBe("agentic")
})

it("shows an open chat's mode and offers the other only as a new chat", async () => {
it("lists both modes for an open chat, and opens a new chat for the other", async () => {
const user = userEvent.setup()
const onNewChat = vi.fn()
renderComposer({ model: KIMI, threadMode: "agentic", onNewChat })

expect(trigger().textContent).toContain("Agentic")
await user.click(trigger())
expect(
await screen.findByText("This chat runs in Agentic mode")
).toBeTruthy()
expect(screen.queryByRole("menuitemradio")).toBeNull()
await user.click(
screen.getByRole("menuitem", {
name: /^Start a new chat in Basic \(Q&A\) mode/,
})
)
expect(await screen.findByText("This chat’s mode")).toBeTruthy()
const agentic = screen.getByRole("menuitemradio", { name: /^Agentic/ })
const basic = screen.getByRole("menuitemradio", { name: /^Basic/ })
expect(agentic.getAttribute("aria-checked")).toBe("true")
expect(within(basic).getByText("New chat")).toBeTruthy()
expect(within(agentic).queryByText("New chat")).toBeNull()
await user.click(basic)

expect(onNewChat).toHaveBeenCalledOnce()
expect(nextChat()).toBe("basic")
})

it("does not offer an open chat's other mode when a gate blocks Agentic", async () => {
it("keeps an open chat when its own mode is picked again", async () => {
const user = userEvent.setup()
const onNewChat = vi.fn()
renderComposer({ model: KIMI, threadMode: "agentic", onNewChat })

await user.click(trigger())
await user.click(
await screen.findByRole("menuitemradio", { name: /^Agentic/ })
)

expect(onNewChat).not.toHaveBeenCalled()
})

it("holds an open chat's other mode when a gate blocks Agentic", async () => {
const user = userEvent.setup()
const onNewChat = vi.fn()
renderComposer({
Expand All @@ -280,13 +325,14 @@ describe("the composer's mode switch", () => {
})

await user.click(trigger())
const offer = await screen.findByRole("menuitem", {
name: /^Start a new chat in Agentic mode/,
const agentic = await screen.findByRole("menuitemradio", {
name: /^Agentic/,
})
expect(offer.textContent).toContain(
expect(agentic.getAttribute("aria-disabled")).toBe("true")
expect(agentic.textContent).toContain(
"This install doesn’t include the agent."
)
await user.click(offer)
await user.click(agentic)

expect(onNewChat).not.toHaveBeenCalled()
})
Expand Down
Loading
Loading