fix(import): resume missing chat threads

docs/adr/0022-markdown-only-cloud-import.md

@@ -12,13 +12,22 @@ Moving the hosted service's current users onto the desktop app is the point of t
 
 - The export is one ZIP for the whole account, markdown only ([contract 3](../contracts/03-export-bundle.md)): every ready document's extracted content with its folder structure and title, plus the chat threads. Original uploads and old generated artifacts stay behind. Tool calls and agent steps are dropped, and citations are reduced to document titles.
 - Import is a bulk upload on the API side, and nothing touches the network. It creates one local workspace per hosted workspace, writes each markdown file through the upload path so it gets a `dedup_key`, keeps `folder_path`, `source` and the hosted ids in `document_metadata`, and enqueues the existing `ingest_document` ([`modules/migration/`](../../surfsense_local/backend/modules/migration/)).
+- An imported chat thread keeps its hosted id in `cloud_id`. Re-import skips a
+  thread with that id without changing it and creates one that is missing.
 - The app re-chunks and re-embeds everything locally. Markdown is in `TEXT_SUFFIXES` ([`worker/ingestion/parsing.py`](../../surfsense_local/backend/worker/ingestion/parsing.py)), so Docling is skipped and import is chunk and embed only.
 - An imported assistant message keeps its citations as a "Sources: A, B" line in its text, because local citations are chunk-backed and imported ones cannot be.
 
 ## Consequences
 
 - The migration is lossy by design. The export step on the `/sunset` page says so, and the info tooltip beside Import from SurfSense cloud in the app's Settings says original files and generated artifacts stay behind.
-- Re-running a bundle is safe for documents. `dedup_key` skips each one already imported, and each workspace is found again by its `cloud_id`. A re-export after edits lands as a second copy, not an update.
-- Chat threads carry no dedup key, so they import only with a workspace's first import. The `ponytail:` in [`modules/migration/service.py`](../../surfsense_local/backend/modules/migration/service.py) names the upgrade: a hosted thread id column on `chat_threads`.
+- Re-running a bundle is safe for documents and threads. `dedup_key` skips each
+  document already imported, each workspace is found again by its `cloud_id`,
+  and each thread is found by its own `cloud_id`. A re-export after document
+  edits lands as a second copy, not an update; a locally edited thread is
+  retained.
+- A workspace with threads imported before hosted ids were retained is marked
+  during upgrade. Re-import matches an exported thread by its preserved creation
+  time, writes its hosted id into the one matching unkeyed row, and keeps that
+  row's local title and messages. Ambiguous or unmatched rows stay untouched.
 - A large account is embed-bound on a laptop: minutes to an hour for thousands of documents, in the background.
 - Migrating original files would be a `format` bump on contract 3, if users ask for it.

docs/architecture/data-model.md

@@ -46,6 +46,7 @@ The conventions every table follows are in [`shared/db.py`](../../surfsense_loca
 |---|---|
 | `id`, `name`, `created_at`, `updated_at` | a name is 1 to 200 characters after trimming |
 | `cloud_id` | nullable, unique: the hosted workspace an import came from, so re-importing the same bundle reuses the row ([`import.md`](import.md)) |
+| `has_unkeyed_imported_threads` | true only when a message signature identifies chat history imported before hosted thread ids were retained; re-import claims a uniquely matching legacy thread and otherwise leaves that history untouched |
 
 The API seeds one workspace, "My Workspace", at startup when none exists. Deleting a workspace first stops its plugin runs ([`stop_workspace_runs.py`](../../surfsense_local/backend/modules/plugins/bundles/stop_workspace_runs.py)), then cascades its documents, threads, artifacts and runs.
 
@@ -114,7 +115,7 @@ One `active` row, enforced in code rather than by a singleton constraint, becaus
 
 | Table | Columns | Notes |
 |---|---|---|
-| `chat_threads` | `id`, `workspace_id`, `title`, `opencode_session_id`, `source_scope`, `history_start_message_id`, `created_at`, `updated_at` | `title` is nullable; the API defaults it to "New chat". `opencode_session_id`, from `0021`, is set on a thread the agent answers, whose turns live in that opencode session rather than in `chat_messages` ([`chat.md`](chat.md#agent-threads)). `source_scope`, JSON from `0024`, holds the sources ticked for the thread; null means every source ([`chat.md`](chat.md#source-scope)). `history_start_message_id`, from `0026`, is the oldest message the model is still sent once history was trimmed, and `NULL` until it first was ([`chat.md`](chat.md#message-assembly)) |
+| `chat_threads` | `id`, `workspace_id`, `cloud_id`, `title`, `opencode_session_id`, `source_scope`, `history_start_message_id`, `created_at`, `updated_at` | `cloud_id` is the nullable, unique hosted thread id. `title` is nullable; the API defaults it to "New chat". `opencode_session_id`, from `0021`, is set on a thread the agent answers, whose turns live in that opencode session rather than in `chat_messages` ([`chat.md`](chat.md#agent-threads)). `source_scope`, JSON from `0024`, holds the sources ticked for the thread; null means every source ([`chat.md`](chat.md#source-scope)). `history_start_message_id`, from `0026`, is the oldest message the model is still sent once history was trimmed, and `NULL` until it first was ([`chat.md`](chat.md#message-assembly)) |
 | `chat_messages` | `id`, `chat_thread_id`, `role`, `content`, `created_at`, `completed_at` | `role` is `user`, `assistant` or `system`; `completed_at` arrived in `0002` |
 
 `content` is JSON: `{"text"}` for a user turn, plus `images: [{"key", "mime", "size_bytes", "sha256"}]` when it carried any, and `source_scope` with `resolved: {"count", "ids_sha256"}` when it was grounded on a scope, and `{"text", "citations"}` for an assistant turn, whose text carries `[citation:<chunk_id>]` markers. The server reads `text` to build the model's history, and a user turn's `images` for the newest turn that carried some; the citations are for the UI. Imported user turns also carry an empty `citations` list. Messages are indexed on `(chat_thread_id, created_at)` and cascade with their thread. Visibility, authorship, cloning, turn ids, token usage and LangGraph checkpoints are left out. See [`chat.md`](chat.md).
@@ -378,12 +379,13 @@ erDiagram
 | `0025` | `0025_source_roots_and_folders.py` | `source_roots` and `folders`; a Library root and root folder per workspace; `documents.folder_id` by a plain `ALTER TABLE`; every `FILE` and `NOTE` filed, an imported `folder_path` becoming a chain of folders (case and normalization variants merged, levels past 8 joined with " / " into the eighth); `content_hash` filled from `dedup_key`; the dedup index swapped to per folder by `DROP` and `CREATE INDEX` |
 | `0026` | `0026_chat_history_start.py` | `chat_threads.history_start_message_id`, by a plain `ALTER TABLE`; no row is backfilled, so a thread sends its whole history until it first outgrows its budget |
 | `0027` | `0027_retire_pre_host_egress_grants.py` | egress rows under `model_download`, `model_search` and `image_model_pull` removed; `host:huggingface.co` allowed only where search and a download were both allowed and the host had no row; downgrading changes nothing |
+| `0029` | `0029_chat_thread_cloud_id.py` | `chat_threads.cloud_id`, nullable and unique; marks workspaces with older unkeyed imported threads |
 
 - Before the API applies a pending revision to an existing database, `upgrade_to_head` writes a `VACUUM INTO` copy to `<data>/backups/<from>-<to>.db` (via `.partial`, renamed once complete) and keeps the newest two ([`migration_snapshot.py`](../../surfsense_local/backend/shared/migration_snapshot.py)). A new database or one at head writes none. If the copy fails, nothing migrates and the API start fails with the reason. To restore: quit SurfSense, replace `surfsense.db` with the snapshot, and start the app.
 - Migrations run on every API start and are idempotent. Autogenerate is off: it renders a rename as a drop plus an add, which deletes a column's data silently, and `env.py` carries no `target_metadata`, so it cannot be used by accident.
 - SQLite cannot alter a CHECK constraint in place, so `0004`, `0009`, `0012` and `0013` copy `selected_models` into a new table.
 - A revision that touches a table already holding rows should read the live schema first (`op.get_bind()`, `sa.inspect`) rather than assume its shape.
-- [`tests/integration/test_migrations.py`](../../surfsense_local/backend/tests/integration/test_migrations.py) fails when the models and the migration history disagree, when a second upgrade is not a no-op, and when a failed migration leaves anything behind. `test_migration_0012.py`, `test_migration_0013.py`, `test_migration_0014.py` and `test_migration_0027.py` beside it test what those revisions change.
+- [`tests/integration/test_migrations.py`](../../surfsense_local/backend/tests/integration/test_migrations.py) fails when the models and the migration history disagree, when a second upgrade is not a no-op, and when a failed migration leaves anything behind. `test_migration_0012.py`, `test_migration_0013.py`, `test_migration_0014.py`, `test_migration_0027.py` and `test_migration_0029.py` beside it test what those revisions change.
 
 ## Known gaps
 

docs/architecture/import.md

@@ -26,12 +26,31 @@ The rest runs as a FastAPI `BackgroundTasks` job in the API process, with its ow
 The local schema has no folder table, so the hierarchy is kept as data. Markdown is in `TEXT_SUFFIXES`, so ingest reads the file directly and never starts Docling: an import is chunking and embedding only. The staged bundle is deleted when the job ends.
 
 Threads from `chats.json` become ordinary local threads. Each message is stored as `{"text": ..., "citations": []}`, and a message that carried citation titles gains `\n\nSources: A, B`. Local citations point at chunks and imported ones have none, so they become text rather than dead chips.
+The hosted thread id is kept as `cloud_id`. A thread already carrying that id
+is left exactly as it is; a missing one and all of its messages are created.
 
 ## Running it again
 
-Re-importing a bundle is safe for documents. The workspaces are found by `cloud_id` and every document whose `dedup_key` is already there is skipped, so an interrupted import is finished by running it again, and documents already enqueued wait on the persistent queue. A re-export after edits lands each edited document as a second copy, not an update.
-
-Threads carry no dedup key, so only the run that creates a workspace imports its threads. The `ponytail:` comment in `service.py` names the cost, a later export's new threads never arrive, and the upgrade, a cloud thread id column on `chat_threads`.
+Re-importing a bundle is safe for documents and threads. The workspaces are
+found by `cloud_id`, every document whose `dedup_key` is already there is
+skipped, and every thread whose `cloud_id` is already there is skipped. Running
+an interrupted import again therefore fills the missing documents and threads;
+documents already enqueued wait on the persistent queue. A thread renamed
+locally keeps that name, while a thread newly present in a later export is
+added. A re-export after document edits lands each edited document as a second
+copy, not an update.
+
+App versions before this database migration imported threads without retaining
+their hosted ids. On upgrade, a workspace is marked when a user or assistant
+message has the old importer's signature: the `citations` key and identical
+`created_at` and `completed_at`. A later local turn does not hide that history,
+while a workspace containing only local turns is not marked. On re-import, an
+exported thread claims the one unkeyed row with its preserved creation time,
+fills that row's `cloud_id`, and leaves its local title and messages untouched.
+An ambiguous or unmatched row stays untouched rather than guessing identity
+from titles or messages. To import newer hosted threads into one of those
+marked workspaces, delete the workspace and import its bundle afresh, or
+reconcile the histories manually.
 
 ## Entry points
 
@@ -42,5 +61,4 @@ After the 202 the dashboard reloads its workspaces and opens the first imported
 
 ## Known gaps
 
-- An import interrupted before its threads were written never imports them: the re-run finds the workspace and skips threads.
 - There is no summary or progress endpoint: the button reads "Importing…" only while the upload is in flight, and after the 202 each document's own ingest status is the only progress.

docs/contracts/03-export-bundle.md

@@ -76,6 +76,8 @@ Import keeps the frontmatter in the stored file. It is a few lines and carries u
 ]
 ```
 
+- `id` is the globally unique primary key of the hosted chat thread. Import
+  stores it as `chat_threads.cloud_id` and uses it to deduplicate re-imports.
 - `role` is `user` or `assistant`. System, tool, and step messages are dropped.
 - `text` is plain markdown with every hosted citation marker **removed** by the exporter.
 - `citations` is `[{"title": ...}]`, distinct titles in first-seen order. It carries no chunk or document ids.
@@ -96,7 +98,11 @@ Import stores each message as `{"text": text, "citations": []}`, appending `\n\n
 - Cap ZIP entries and unpacked size with import-specific limits; the upload limits in `documents/storage.py` (10,000 entries) are too small for a large account once OKF index/log files are counted.
 - Write each document through the existing upload path so it gets a content-hash `dedup_key`; store `folder_path` (directory part of `path` under `documents/`), `source`, and `cloud: {workspace_id, document_id}` in `document_metadata`.
 - Re-importing the same bundle is a no-op per document via `dedup_key`. A re-export after edits lands as a second copy, not an update; acceptable for a one-shot migration.
-- Chat threads carry no dedup key, so they import only with a workspace's first import. A re-import adds none, and an import interrupted before its threads were written never adds them.
+- Re-importing skips each chat thread already present by its hosted `id` and
+  creates any missing thread. Threads imported by app versions that did not
+  retain this id claim the one unkeyed row with the same creation time, writing
+  its hosted id while keeping the local title and messages. Ambiguous or
+  unmatched rows stay untouched.
 
 ## Fixture
 

surfsense_local/backend/alembic/versions/0029_chat_thread_cloud_id.py

@@ -0,0 +1,58 @@
+"""Remember the hosted identity of an imported chat thread.
+
+Both columns use plain ALTER TABLE: rebuilding either parent table with foreign
+keys enabled would cascade-delete its child rows.
+
+Revision ID: 0029
+Revises: 0028
+"""
+
+from collections.abc import Sequence
+
+import sqlalchemy as sa
+
+from alembic import op
+
+revision: str = "0029"
+down_revision: str | Sequence[str] | None = "0028"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    op.add_column("chat_threads", sa.Column("cloud_id", sa.Integer(), nullable=True))
+    op.create_index("chat_threads_cloud_id", "chat_threads", ["cloud_id"], unique=True)
+    op.add_column(
+        "workspaces",
+        sa.Column(
+            "has_unkeyed_imported_threads",
+            sa.Boolean(),
+            nullable=False,
+            server_default=sa.false(),
+        ),
+    )
+    op.execute(
+        """
+        UPDATE workspaces
+        SET has_unkeyed_imported_threads = 1
+        WHERE cloud_id IS NOT NULL
+          AND EXISTS (
+              SELECT 1
+              FROM chat_threads
+              JOIN chat_messages
+                ON chat_messages.chat_thread_id = chat_threads.id
+              WHERE chat_threads.workspace_id = workspaces.id
+                AND chat_messages.role IN ('user', 'assistant')
+                AND json_type(
+                    chat_messages.content, '$.citations'
+                ) IS NOT NULL
+                AND chat_messages.completed_at = chat_messages.created_at
+          )
+        """
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("workspaces", "has_unkeyed_imported_threads")
+    op.drop_index("chat_threads_cloud_id", table_name="chat_threads")
+    op.drop_column("chat_threads", "cloud_id")

surfsense_local/backend/modules/chat/models.py

@@ -17,12 +17,17 @@ class MessageRole(enum.StrEnum):
 
 class ChatThread(Base):
     __tablename__ = "chat_threads"
-    __table_args__ = (Index("chat_threads_workspace", "workspace_id"),)
+    __table_args__ = (
+        Index("chat_threads_workspace", "workspace_id"),
+        Index("chat_threads_cloud_id", "cloud_id", unique=True),
+    )
 
     id: Mapped[int] = mapped_column(primary_key=True)
     workspace_id: Mapped[int] = mapped_column(
         ForeignKey("workspaces.id", ondelete="CASCADE")
     )
+    # The hosted thread this one was imported from, so a re-import finds it.
+    cloud_id: Mapped[int | None]
     title: Mapped[str | None]
     # Set when the agent answers this thread: its turns live in that opencode
     # session, not in chat_messages. Chosen when the thread is opened, and kept.

surfsense_local/backend/modules/migration/router.py

@@ -70,16 +70,13 @@ def start_import(
         request.app.state.session_factory,
         staged.path,
         manifest,
-        {
-            workspace.cloud_id: (workspace.id, created)
-            for workspace, created in workspaces
-        },
+        {workspace.cloud_id: workspace.id for workspace in workspaces},
     )
     return ImportAccepted(
         workspaces=[
             ImportedWorkspace(
                 id=workspace.id, cloud_id=workspace.cloud_id, name=workspace.name
             )
-            for workspace, _ in workspaces
+            for workspace in workspaces
         ]
     )