Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ What the maintainers are working on, grouped by when. This page names the initia

- **Release and CI health.** v2.0.3 was the first release to carry the llama.cpp, sd.cpp and audio.cpp runtimes, built and packaged on all three runners, and v2.2.0 the first whose installers carry opencode, signed on Windows and notarized on macOS. What is left is CI that catches a break before a release, and the installers' third-party notices ([about](architecture/about.md)). See [packaging](architecture/packaging.md#known-gaps) and [updates](architecture/updates.md).
- **Egress gaps.** The follow-up fetch for a remote image URL leaves without a consent decision, and some stored grants no longer match what Settings › Network shows. See [egress](architecture/egress.md).
- **Import from cloud.** The hosted export window closes on 18 Oct 2026. Imports need a progress summary, and a re-run must bring back the chat threads an interrupted import missed. See [import](architecture/import.md).
- **Import from cloud.** The hosted export window closes on 18 Oct 2026. Imports need a progress summary. See [import](architecture/import.md).
- **Scraper API and MCP license mode.** PATs are purged on 18 Oct 2026; after that, license mode is how scraper API and MCP users keep access. See [contract 2](contracts/02-scraper-api-auth.md).
- **Model catalog.** One catalog for local and remote models, each classified offline from a reviewed, packaged manifest and keyed by model type. See the [proposal](proposals/model-catalog.md).
- **Hosted wind-down to 18 Oct 2026.** `/sunset` lacks the deletion date, the refund offer and the MCP change; background jobs keep running through the export-only tail; and the purge is still ahead. See [sunset](architecture/sunset.md), the [license portal](architecture/license/portal.md) and the page plan in [`plans/community-local/portal/02-pages.md`](../plans/community-local/portal/02-pages.md).
Expand Down
4 changes: 3 additions & 1 deletion docs/adr/0022-markdown-only-cloud-import.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@ Moving the hosted service's current users onto the desktop app is the point of t
- The export is one ZIP for the whole account, markdown only ([contract 3](../contracts/03-export-bundle.md)): every ready document's extracted content with its folder structure and title, plus the chat threads. Original uploads and old generated artifacts stay behind. Tool calls and agent steps are dropped, and citations are reduced to document titles.
- Import is a bulk upload on the API side, and nothing touches the network. It creates one local workspace per hosted workspace, writes each markdown file through the upload path so it gets a `dedup_key`, keeps `folder_path`, `source` and the hosted ids in `document_metadata`, and enqueues the existing `ingest_document` ([`modules/migration/`](../../surfsense_local/backend/modules/migration/)).
- An imported chat thread keeps its hosted id in `cloud_id`. Re-import skips a
thread with that id without changing it and creates one that is missing.
thread with that id without changing it. In an ordinary workspace it creates
a missing thread; in a workspace marked as holding legacy unkeyed imports, it
only assigns that id to the unique unkeyed row with the same creation time.
- The app re-chunks and re-embeds everything locally. Markdown is in `TEXT_SUFFIXES` ([`worker/ingestion/parsing.py`](../../surfsense_local/backend/worker/ingestion/parsing.py)), so Docling is skipped and import is chunk and embed only.
- An imported assistant message keeps its citations as a "Sources: A, B" line in its text, because local citations are chunk-backed and imported ones cannot be.

Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/data-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,7 @@ erDiagram
| `0025` | `0025_source_roots_and_folders.py` | `source_roots` and `folders`; a Library root and root folder per workspace; `documents.folder_id` by a plain `ALTER TABLE`; every `FILE` and `NOTE` filed, an imported `folder_path` becoming a chain of folders (case and normalization variants merged, levels past 8 joined with " / " into the eighth); `content_hash` filled from `dedup_key`; the dedup index swapped to per folder by `DROP` and `CREATE INDEX` |
| `0026` | `0026_chat_history_start.py` | `chat_threads.history_start_message_id`, by a plain `ALTER TABLE`; no row is backfilled, so a thread sends its whole history until it first outgrows its budget |
| `0027` | `0027_retire_pre_host_egress_grants.py` | egress rows under `model_download`, `model_search` and `image_model_pull` removed; `host:huggingface.co` allowed only where search and a download were both allowed and the host had no row; downgrading changes nothing |
| `0029` | `0029_chat_thread_cloud_id.py` | `chat_threads.cloud_id`, nullable and unique; marks workspaces with older unkeyed imported threads |
| `0029` | `0029_chat_thread_cloud_id.py` | `chat_threads.cloud_id`, nullable and unique; `workspaces.has_unkeyed_imported_threads`, marking workspaces with older unkeyed imported threads |
| `0030` | `0030_speech_to_text_slot.py` | `selected_models` rebuilt so `speech_to_text` is a type and `audiocpp` may hold it beside `audio_gen`, carrying every column, `0019`'s `settings` included, which `0018`'s rebuild predates; downgrading drops a `speech_to_text` selection |

- Before the API applies a pending revision to an existing database, `upgrade_to_head` writes a `VACUUM INTO` copy to `<data>/backups/<from>-<to>.db` (via `.partial`, renamed once complete) and keeps the newest two ([`migration_snapshot.py`](../../surfsense_local/backend/shared/migration_snapshot.py)). A new database or one at head writes none. If the copy fails, nothing migrates and the API start fails with the reason. To restore: quit SurfSense, replace `surfsense.db` with the snapshot, and start the app.
Expand Down
11 changes: 6 additions & 5 deletions docs/contracts/03-export-bundle.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,11 +98,12 @@ Import stores each message as `{"text": text, "citations": []}`, appending `\n\n
- Cap ZIP entries and unpacked size with import-specific limits; the upload limits in `documents/storage.py` (10,000 entries) are too small for a large account once OKF index/log files are counted.
- Write each document through the existing upload path so it gets a content-hash `dedup_key`; store `folder_path` (directory part of `path` under `documents/`), `source`, and `cloud: {workspace_id, document_id}` in `document_metadata`.
- Re-importing the same bundle is a no-op per document via `dedup_key`. A re-export after edits lands as a second copy, not an update; acceptable for a one-shot migration.
- Re-importing skips each chat thread already present by its hosted `id` and
creates any missing thread. Threads imported by app versions that did not
retain this id claim the one unkeyed row with the same creation time, writing
its hosted id while keeping the local title and messages. Ambiguous or
unmatched rows stay untouched.
- Re-importing skips each chat thread already present by its hosted `id`. In an
ordinary workspace it creates a missing thread. In a workspace marked as
holding threads imported before hosted ids were retained, it instead claims
the one unkeyed row with the same creation time, writing its hosted id while
keeping the local title and messages; ambiguous or unmatched rows stay
untouched and no new thread is created.

## Fixture

Expand Down
Loading