feat(mcp): finish persistent — remove all v1 limitations + per-database catalog

hyperdb-mcp/CHANGELOG.md

@@ -68,20 +68,45 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
   both `database` and `persist` are set, `database` wins. Combining
   `persist: true` with `--ephemeral-only` returns a clear
   `InvalidArgument` error.
-- **Catalog updates are database-aware.** `_table_catalog` rows are
-  only written when ingesting to the primary or persistent database.
-  User-attached databases manage their own metadata if any.
-
-### Limitations (deferred to a follow-up)
-
-- `load_files` and `watch_directory` reject `database` / `persist`
-  with a clear error — their connection pool is bound to the primary
-  database and can't reach attached databases. Use `load_file` with
-  `persist: true` for one-off persistent ingests until pool routing
-  ships.
-- `load_file` with `mode: "merge"` rejects non-primary `database`.
-  The merge implementation uses a temp table and cross-database DML
-  isn't yet verified across all hyperd versions.
+- **Cross-database merge.** `load_file` with `mode: "merge"` now
+  accepts any writable `database`. The merge path keeps the temp
+  table inside the target database so DELETE-USING and INSERT-SELECT
+  stay single-DB — no cross-database DML is required. Engine
+  helpers `table_exists_in`, `column_metadata_in`, and
+  `alter_table_add_columns_in` carry the routing.
+- **`export(format="hyper", database=...)`.** The hyper-format
+  export now snapshots whichever database the caller named (via a
+  new `ExportOptions.source_db` field plumbed into
+  `populate_export_target`). Default behavior (snapshot primary)
+  unchanged.
+- **`load_files` and `watch_directory` accept `database` / `persist`.**
+  Their connection pool now opens the resolved target's `.hyper`
+  file directly as its workspace, so unqualified ingest SQL routes
+  into the right database without further plumbing. The watcher's
+  reconnect-recovery path re-resolves the target so a hyperd
+  restart picks the right file. `WatcherHandle` records its
+  `target_db`; `detach_database` rejects with `InvalidArgument` if
+  any active watcher targets the alias (call `unwatch_directory`
+  first to release it).
+- **Per-database `_table_catalog`.** Every writable database
+  receives its own catalog, lazily seeded on first ingest. The
+  catalog CRUD API gains `*_in(target_db)` siblings
+  (`ensure_exists_in`, `upsert_stub_in`, `set_metadata_in`,
+  `get_in`, `list_in`, `delete_for_in`, `reconcile_in`). The
+  per-engine catalog-presence cache is now keyed by canonical
+  alias (`Mutex<HashMap<String, bool>>`); detach clears the entry.
+- **`set_table_metadata.database` parameter.** Routes the catalog
+  write to the named database's `_table_catalog`. Read-only
+  attachments are rejected up front with a clear "re-attach with
+  writable:true" message.
+
+### Changed (breaking — pre-1.0)
+
+- `Engine::catalog_present_in_persistent` → `Engine::catalog_present_in(alias, prober)`.
+- `Engine::mark_catalog_present` → `Engine::mark_catalog_present_for(alias)`.
+- `Engine::catalog_present_cache` field shape: `Mutex<Option<bool>>` → `Mutex<HashMap<String, bool>>` keyed by lowercased alias.
+- New `Engine::clear_catalog_cache_for(alias)` paired with `detach_database`.
+- `table_catalog::ensure_exists_in_database(engine, alias)` is now a deprecated wrapper over `ensure_exists_in(engine, Some(alias))`.
 
 ### Removed
 
@@ -92,10 +117,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 ### Performance
 
-- **Per-engine `_table_catalog` presence cache.** Catalog reads/writes
+- **Per-database `_table_catalog` presence cache.** Catalog reads/writes
   used to round-trip a `pg_catalog.pg_tables` probe on every call;
-  now the existence check is cached for the engine's lifetime and
-  primed immediately after `CREATE TABLE IF NOT EXISTS`.
+  now the existence check is cached per (engine, alias) and primed
+  immediately after `CREATE TABLE IF NOT EXISTS`. `detach_database`
+  clears the alias's entry so a re-attach to a different file isn't
+  served stale.
 
 ### Fixed
 

hyperdb-mcp/README.md

@@ -203,7 +203,7 @@ describe({ database: "persistent" })
 sample({ table: "customers", database: "persistent" })
 ```
 
-The `database` parameter is available on `query`, `execute`, `load_data`, `load_file`, `describe`, `sample`, `chart`, and `export`. The shorthand `persist: true` (sugar for `database: "persistent"`) is available on `load_data` and `load_file`. Pass any user-attached writable alias (created via `attach_database`) to target a custom database.
+The `database` parameter is available on `query`, `execute`, `load_data`, `load_file`, `load_files`, `watch_directory`, `describe`, `sample`, `chart`, `export`, and `set_table_metadata`. The shorthand `persist: true` (sugar for `database: "persistent"`) is available on `load_data`, `load_file`, `load_files`, and `watch_directory`. Pass any user-attached writable alias (created via `attach_database`) to target a custom database.
 
 (`query_data` and `query_file` are one-shot tools that materialize the inline data into their own temp table and query it — they do not accept a `database` parameter because the data isn't in a persisted database to begin with.)
 
@@ -218,37 +218,9 @@ CREATE TABLE "persistent"."public"."revenue_2026" AS
   SELECT region, SUM(amount) FROM scratch_orders GROUP BY region;
 ```
 
-The `_table_catalog` (which tracks MCP-managed metadata for your tables) lives in the persistent attachment automatically — there's nothing to manage. If you want a pristine `.hyper` file for export with no MCP bookkeeping, run `DROP TABLE "persistent"."public"."_table_catalog"` once and subsequent sessions opening that file will leave it dropped.
+**Per-database `_table_catalog`:** every writable database — persistent and any user-attached writable file — gets its own `_table_catalog` lazily seeded on first ingest. MCP-managed metadata (load tool, params, timestamps, prose fields set via `set_table_metadata`) lives alongside the data file, so opening a `.hyper` file later as a primary workspace finds the catalog ready. If you want a pristine `.hyper` file for export with no MCP bookkeeping, run `DROP TABLE "<alias>"."public"."_table_catalog"` once and subsequent sessions opening that file will leave it dropped.
 
-**v1 limitations:** `load_files` and `watch_directory` use a connection pool bound to the primary database and don't yet accept `database`/`persist` — use `load_file` with `persist: true` for one-off persistent ingests. Merge mode (`load_file` with `mode: "merge"`) only works against the primary database in v1.
-
-### Daemon management
-
-The daemon is normally invisible — it auto-spawns and idle-times-out on its own. For diagnostics:
-
-```bash
-hyperdb-mcp daemon status   # Show running daemon (PID, endpoint, started_at, version)
-hyperdb-mcp daemon stop     # Gracefully shut down the daemon
-hyperdb-mcp daemon          # Run as a daemon explicitly (rarely needed)
-```
-
-State files live at `~/.hyperdb/` by default (override with `HYPERDB_STATE_DIR`).
-
-### Recovery from hyperd crashes
-
-The daemon polls `hyperd` every 5 seconds. If the process has exited (crashed, OOM, killed), the daemon spawns a replacement, atomically updates `~/.hyperdb/daemon.json` with the new endpoint, and continues serving clients. Clients see one failed tool call (the request that was in flight when hyperd died); the next tool call transparently reconnects to the new hyperd via the same recovery path used for normal connection drops.
-
-If a client itself notices hyperd is unreachable before the next polling tick, it sends a fast-path `REPORT_HYPERD_ERROR` signal to the daemon so the restart kicks off without waiting for the timer.
-
-If hyperd repeatedly fails to start (3 attempts within 60 seconds — e.g., misconfigured `HYPERD_PATH`, port exhaustion, broken binary), the daemon shuts itself down and removes the discovery file. The next MCP client to start up will then spawn a fresh daemon, surfacing any persistent failure clearly to the user rather than spinning silently.
-
-**Known limitation:** if hyperd hangs (alive at the OS level but unresponsive to queries), the daemon's polling can't detect it and your tool call may stall indefinitely. The recovery path is `hyperdb-mcp daemon stop` followed by reconnecting from your MCP client.
-
-### Other behavioral flags
-
-| Flag | Behavior |
-|---|---|
-| `--read-only` | Disables `execute`, `load_data`, `load_file`, `watch_directory`, `save_query`, `delete_query`, and Hyper-format export. See [Read-Only Mode](#read-only-mode). |
+**Detach safety:** `detach_database` rejects with `InvalidArgument` if any active watcher targets the alias — call `unwatch_directory` first. This prevents the watcher's pool from silently writing into a now-detached file (or worse, the wrong file if the alias is later re-attached to a different path).
 
 ### Daemon management
 

hyperdb-mcp/src/attach.rs

@@ -366,7 +366,7 @@ impl AttachRegistry {
         // DB" which implicitly promises a catalog; leaving an
         // attached-but-unseeded file would silently violate that.
         if file_was_created {
-            if let Err(e) = crate::table_catalog::ensure_exists_in_database(engine, &req.alias) {
+            if let Err(e) = crate::table_catalog::ensure_exists_in(engine, Some(&req.alias)) {
                 let detach_sql = format!("DETACH DATABASE \"{}\"", req.alias.replace('"', "\"\""));
                 if let Err(de) = engine.execute_command(&detach_sql) {
                     tracing::warn!(

hyperdb-mcp/src/engine.rs

@@ -195,17 +195,19 @@ pub struct Engine {
     /// to `false` after the server consumes it via
     /// [`Self::take_persistent_was_created`].
     persistent_was_created: bool,
-    /// Cached "_table_catalog exists in persistent" probe. Populated on
-    /// first call to [`Self::catalog_present_in_persistent`] and held
-    /// for the lifetime of the connection. Avoids repeated
-    /// `pg_catalog.pg_tables` round-trips on every catalog read/write.
+    /// Cached "_table_catalog exists in `<alias>`" probes, keyed by
+    /// canonical alias (lowercase). Populated on first call to
+    /// [`Self::catalog_present_in`] for each `(engine, alias)` pair.
+    ///
     /// Lives on the Engine because the catalog is per-engine-lifetime
     /// (a `ConnectionLost` reconnect creates a fresh Engine, so the
-    /// cache resets at the right boundary). `None` means "not yet
-    /// probed"; `Some(false)` is cacheable too — once the catalog is
-    /// confirmed absent on a read-only or `--ephemeral-only` flow it
-    /// stays absent for the whole engine lifetime.
-    catalog_present_cache: std::sync::Mutex<Option<bool>>,
+    /// cache resets at the right boundary). Detaching an alias clears
+    /// its entry via [`Self::clear_catalog_cache_for`] so a re-attach
+    /// to a different file/writability doesn't reuse a stale value.
+    /// `Some(false)` is cacheable too — once the catalog is confirmed
+    /// absent it stays absent for the rest of the engine's lifetime
+    /// unless explicitly cleared.
+    catalog_present_cache: std::sync::Mutex<std::collections::HashMap<String, bool>>,
     log_dir: PathBuf,
 }
 
@@ -339,7 +341,7 @@ impl Engine {
             ephemeral_path,
             persistent_path,
             persistent_was_created,
-            catalog_present_cache: std::sync::Mutex::new(None),
+            catalog_present_cache: std::sync::Mutex::new(std::collections::HashMap::new()),
             log_dir,
         })
     }
@@ -414,7 +416,7 @@ impl Engine {
             ephemeral_path: ephemeral_path.to_path_buf(),
             persistent_path,
             persistent_was_created,
-            catalog_present_cache: std::sync::Mutex::new(None),
+            catalog_present_cache: std::sync::Mutex::new(std::collections::HashMap::new()),
             log_dir: log_dir.to_path_buf(),
         }))
     }
@@ -596,48 +598,56 @@ impl Engine {
         self.persistent_was_created
     }
 
-    /// Returns whether `_table_catalog` exists in the persistent
-    /// attachment, caching the result on first call so subsequent
-    /// catalog read/write paths skip the `pg_catalog.pg_tables` probe.
+    /// Returns whether `_table_catalog` exists in `alias`, caching
+    /// the per-DB result on first call so subsequent catalog read/
+    /// write paths skip the `pg_catalog.pg_tables` probe.
     ///
     /// `prober` is the SQL-side existence check; the cache layer here
     /// is intentionally generic so the catalog module can keep its
-    /// existing probe SQL in one place.
-    ///
-    /// Returns `Ok(false)` immediately when the engine has no
-    /// persistent attachment (no probe needed).
+    /// probe SQL in one place.
     ///
     /// # Errors
     /// Propagates whatever error `prober` returns on the first call.
     /// On subsequent calls, the cached value is returned without
     /// re-running the probe.
-    pub fn catalog_present_in_persistent<F>(&self, prober: F) -> Result<bool, McpError>
+    pub fn catalog_present_in<F>(&self, alias: &str, prober: F) -> Result<bool, McpError>
     where
         F: Fn(&Engine) -> Result<bool, McpError>,
     {
-        if !self.has_persistent() {
-            return Ok(false);
-        }
+        let key = alias.to_ascii_lowercase();
         // Fast path: cache already populated.
         if let Ok(guard) = self.catalog_present_cache.lock() {
-            if let Some(present) = *guard {
+            if let Some(&present) = guard.get(&key) {
                 return Ok(present);
             }
         }
         // Slow path: run the probe and cache its result.
         let present = prober(self)?;
         if let Ok(mut guard) = self.catalog_present_cache.lock() {
-            *guard = Some(present);
+            guard.insert(key, present);
         }
         Ok(present)
     }
 
-    /// Synchronously set the catalog-presence cache to `true` — used by
-    /// `table_catalog::ensure_exists` after a `CREATE TABLE IF NOT
-    /// EXISTS` so subsequent reads/writes skip the existence probe.
-    pub fn mark_catalog_present(&self) {
+    /// Synchronously set the catalog-presence cache to `true` for
+    /// `alias` — used by `table_catalog::ensure_exists_in` after a
+    /// successful `CREATE TABLE IF NOT EXISTS` so subsequent reads/
+    /// writes against that DB skip the existence probe.
+    pub fn mark_catalog_present_for(&self, alias: &str) {
+        let key = alias.to_ascii_lowercase();
         if let Ok(mut guard) = self.catalog_present_cache.lock() {
-            *guard = Some(true);
+            guard.insert(key, true);
+        }
+    }
+
+    /// Drop the cached probe result for `alias`. Called by
+    /// `detach_database` so that re-attaching the same alias to a
+    /// different file (or with different writability) doesn't reuse a
+    /// stale entry.
+    pub fn clear_catalog_cache_for(&self, alias: &str) {
+        let key = alias.to_ascii_lowercase();
+        if let Ok(mut guard) = self.catalog_present_cache.lock() {
+            guard.remove(&key);
         }
     }
 
@@ -926,6 +936,52 @@ impl Engine {
             .collect())
     }
 
+    /// Like [`Self::column_metadata`] but for a table in `target_db`.
+    /// `None` falls back to `column_metadata` (primary). `Some(alias)`
+    /// reads via the qualified `pg_catalog.pg_attribute` join used by
+    /// `describe_columns_via_pg_catalog` — the connection-bound
+    /// `Catalog` API can't see attached databases.
+    ///
+    /// # Errors
+    ///
+    /// Returns [`ErrorCode::TableNotFound`] when no rows come back from
+    /// the qualified probe. Propagates connection errors.
+    pub fn column_metadata_in(
+        &self,
+        target_db: Option<&str>,
+        table: &str,
+    ) -> Result<Vec<ColumnSchema>, McpError> {
+        let Some(db) = target_db else {
+            return self.column_metadata(table);
+        };
+        let rows = describe_columns_via_pg_catalog(self, db, table)?;
+        if rows.is_empty() {
+            return Err(McpError::new(
+                ErrorCode::TableNotFound,
+                format!("Table '{table}' does not exist in database '{db}'"),
+            ));
+        }
+        Ok(rows
+            .into_iter()
+            .map(|r| ColumnSchema {
+                name: r
+                    .get("name")
+                    .and_then(|v| v.as_str())
+                    .unwrap_or_default()
+                    .to_string(),
+                hyper_type: r
+                    .get("type")
+                    .and_then(|v| v.as_str())
+                    .unwrap_or_default()
+                    .to_string(),
+                nullable: r
+                    .get("nullable")
+                    .and_then(serde_json::Value::as_bool)
+                    .unwrap_or(true),
+            })
+            .collect())
+    }
+
     /// Returns true if `table` exists in the `public` schema. Avoids
     /// the per-version error-string ambiguity of
     /// [`Catalog::get_table_definition`] by listing names instead.
@@ -940,6 +996,29 @@ impl Engine {
         Ok(names.iter().any(|n| n.as_str() == table))
     }
 
+    /// Like [`Self::table_exists`] but for a table in `target_db`.
+    /// `None` falls back to `table_exists` (primary). `Some(alias)`
+    /// probes the qualified `pg_catalog.pg_tables` of the attached
+    /// database — the connection-bound `Catalog` API can't see
+    /// attached databases.
+    ///
+    /// # Errors
+    ///
+    /// Propagates connection errors from the probe query.
+    pub fn table_exists_in(&self, target_db: Option<&str>, table: &str) -> Result<bool, McpError> {
+        let Some(db) = target_db else {
+            return self.table_exists(table);
+        };
+        let esc_db = db.replace('"', "\"\"");
+        let esc_tbl = table.replace('\'', "''");
+        let sql = format!(
+            "SELECT 1 AS one FROM \"{esc_db}\".pg_catalog.pg_tables \
+             WHERE schemaname = 'public' AND tablename = '{esc_tbl}'"
+        );
+        let rows = self.execute_query_to_json(&sql)?;
+        Ok(!rows.is_empty())
+    }
+
     /// Issue a single `ALTER TABLE "<table>" ADD COLUMN "<n1>" <t1>,
     /// ADD COLUMN "<n2>" <t2>, …` statement that adds all columns
     /// atomically. Hyper supports the multi-column form (verified
@@ -968,6 +1047,23 @@ impl Engine {
         &self,
         table: &str,
         cols: &[ColumnSchema],
+    ) -> Result<(), McpError> {
+        self.alter_table_add_columns_in(None, table, cols)
+    }
+
+    /// Like [`Self::alter_table_add_columns`] but for a table in
+    /// `target_db`. `None` keeps the unqualified identifier; `Some(alias)`
+    /// emits `"db"."public"."table"` so the ALTER lands in the attached
+    /// database.
+    ///
+    /// # Errors
+    ///
+    /// Same as [`Self::alter_table_add_columns`].
+    pub fn alter_table_add_columns_in(
+        &self,
+        target_db: Option<&str>,
+        table: &str,
+        cols: &[ColumnSchema],
     ) -> Result<(), McpError> {
         if cols.is_empty() {
             return Ok(());
@@ -983,7 +1079,14 @@ impl Engine {
                 ));
             }
         }
-        let quoted_table = format!("\"{}\"", table.replace('"', "\"\""));
+        let quoted_table = match target_db {
+            Some(db) => {
+                let esc_db = db.replace('"', "\"\"");
+                let esc_tbl = table.replace('"', "\"\"");
+                format!("\"{esc_db}\".\"public\".\"{esc_tbl}\"")
+            }
+            None => format!("\"{}\"", table.replace('"', "\"\"")),
+        };
         let add_clauses = cols
             .iter()
             .map(|c| {