Skip to content

nightly 2026-10-05 (v0.3.38-beta.47): exit status on record, three stuck-card fixes, daemon RSS under load cut - #412

Merged
physics515 merged 9 commits into
masterfrom
nanna-nightly-2026-10-05
Oct 5, 2026
Merged

physics515 merged 9 commits into
masterfrom
nanna-nightly-2026-10-05

Conversation

@physics515

@physics515 physics515 commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Nightly run 2026-10-05, which also prepares v0.3.38-beta.47. Ten commits: one dependency increment, four fix or performance increments, a benchmark baseline, release prep, and two doc corrections. Every code commit's gate was green.

Dependencies

  • Toolchain: nightly-2026-10-02 → 2026-10-03, the newest that builds the workspace. cargo build --release -p nanna-daemon takes 7m51s from cold. The CI mirrors moved with it.
  • Rust lockfile: powerfmt 0.2.1, signal-hook 0.4.5, twox-hash 2.1.5.
  • GUI: postcss 8.5.29.
  • Still blocked: rten 0.27 (ocrs 0.13.1 wants ^0.26, so dependabot chore(deps): bump rten from 0.26.0 to 0.27.0 #411 cannot build), and TypeScript 7 (vue-tsc 3.3.12). The braces and node-forge advisories still have no fixed release.

What shipped

  1. An unhookable daemon death now leaves its exit status on record (nanna-core::exit_record, GUI daemon_manager, doctor).
    • Problem: the exit record said "died through a path no hook could see" for exactly the deaths worth explaining: the AppImage SIGBUS, and the unexplained 2026-09-28 death. The GUI saw the exit status but only logged it.
    • Change: the GUI now adds observed_exit (code or signal, and whether it sent the kill itself) to the record. It does so only when the record is still that process's running marker: same PID, and written after the spawn, so a reused PID never matches.
    • The next boot names the signal, and the new nanna doctor check daemon.last_exit reports the same verdict.
  2. Three P25 card-run defects, found by a review of the Stage 3 code:
    • Stop kept meaning stop only for 64 edits. The pause marker was searched in the newest 64 activity rows. After 64 renames or reorders, the sweep took the paused card back. New TaskRepository::newest_activity_of reads markers by action. On the old read, the extended stall test released the paused card.
    • A stopped card reassigned to another agent stayed in_progress forever. A pause now holds only for the member it stopped, and the new member's run starts.
    • hand_backs_in_a_row counted across completions. It now stops at completed. It deliberately does not stop at reopened, which the harness writes mid-cycle.
  3. list_all, stats and the dream timescale stopped cloning every vector. They went through all_entries(), which clones every embedding and model bucket in order to read text, state or a timestamp. They now use map_entries.
  4. The daemon trims the heap after a large reply. It runs malloc_trim(0) once a reply of ≥ 1 MiB has been sent, at most once per 10 s, at ~4.5 ms each. The boot trim moved into the same module (crate::heap).

Measurements (items 3 and 4)

Setup: release daemons, isolated, each on a copy of the operator's store (3 730 memories). Every 2 min a stdlib-Python WebSocket client sent memory.list (a 14.4 MB reply), a keyword search and stats.

RSS after 9 cycles ~15 cycles later
master ~425 MB ~439 MB 608 MB at ~40, still climbing
master + MALLOC_ARENA_MAX=2 ~352 MB ~352 MB 366 MB
clone fix ~300 MB ~306 MB 377 MB, climbing
clone fix + trim ~178 MB ~213 MB (one 267 MB peak) still drifting slowly
  • Idle, all builds were flat at 142.5 MB.
  • The trimmed build is not flat. It drifts at about a third of the old rate. The table is in bench/BASELINE.md Suite 5.
  • Filed next: serialize memory.list without the intermediate serde_json::Value.

Verification

  • Workspace: cargo clippy --workspace --all-targets --exclude nanna-gui 0 warnings. cargo test --workspace --exclude nanna-gui 2828 passed / 0 failed (2811 on master).
  • Scripted-model daemon e2e: cargo test -p nanna-client --test e2e_daemon 56/56.
  • GUI crate: clippy 0 warnings, tests 103 passed.
  • Frontend: vitest 437/437, typecheck 0 errors, pnpm generate green. tauri.conf.json validates against the Tauri 2 schema.
  • Release guard simulated: conf true | published true.
  • Real binaries, isolated (scratch HOME, config, data dir, port 51990):
    • The real GUI spawned the real daemon, and I sent the daemon SIGKILL. The record gained "observed_exit": {"signal": 9, …}.
    • The GUI's health monitor restarted the daemon, which logged Previous exit was UNCLEAN: … the app saw it end by signal 9 (SIGKILL) at ….
    • nanna doctor reported [ok] daemon.last_exit a daemon is running (PID …) while it ran, and [warn] … the app saw it end by signal 9 (SIGKILL) with the log path after it was killed.
  • Not verified: a live model run (this host has none), and GUI-over-WebDriver (the GUI change is backend-only and was driven directly).
  • Side finding: kill -BUS sent to a running daemon is swallowed by Rust std's stack-overflow handler, which resets the handler and returns. A real fault still kills the process (the AppImage cores show that). Benign.

Not done, and why

Research folded into ROADMAP

  • The nightly abort rename and its upstream fix (above).
  • glibc arena retention: the 2026 fixes converge on M_ARENA_MAX plus trim-on-idle (ctox#248, ivygrep#409, scrypted#2027). Measured here instead of assumed.
  • Re-checked: Tauri 3 is still alpha.4. turso 0.8.2 is still pre-release. onyums 0.5.0 and arti-client 0.47 unchanged; rmcp 3.5.0 is out (not yet evaluated). fsrs 6.6.2, and burn 0.22 still pre-release.
  • Closed a stale box: boa_engine 0.22 was already taken on 2026-09-08.

Shipped

  • Merged as 6dbb5dae once every check passed (build-release 41m, compile-tests 27m, deterministic-budgets 11m, daemon e2e, Playwright, Vitest, both typechecks, aarch64 under qemu, cargo audit, pnpm audit).
  • Released: v0.3.38-beta.47, a pre-release with a Windows NSIS installer plus a Linux AppImage and deb. Built by run 37316936881.
  • Updater manifest: chore(release): publish v0.3.38-beta.47 to the updater — Windows and Linux #413, merged as 3e32ad7f. Both signatures are byte-identical to the release's .sig files, and both download URLs answer 200. The basic-automation and legacy physics515 raw URLs both serve 0.3.38.

🤖 Generated with Claude Code

Nanna and others added 9 commits October 5, 2026 05:49
…file bumps

- Toolchain 2026-10-02 -> 2026-10-03, the newest nightly that builds the
  workspace: 2026-10-04 removed `core::intrinsics::abort` (rust#163574),
  which `branches 0.4.6` calls, and `branches` arrives through
  `turso_core` (^0.4.3 even in 0.8.2-pre.2). Release build of
  nanna-daemon green from cold in 7m51s. CI mirrors moved with it.
- Lockfile: powerfmt 0.2.1, signal-hook 0.4.5, twox-hash 2.1.5.
- GUI: postcss 8.5.29 (with nanoid 3.3.20, @oxc-project/types 0.153).
- Still blocked: rten 0.27 (ocrs 0.13.1 wants ^0.26), TypeScript 7
  (vue-tsc 3.3.12).
- Roadmap: the branches blocker filed with its upstream fix; the stale
  boa_engine 0.22 box closed (done in 69a68aa); glibc arena research
  folded into the idle-RSS item.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tatus on record

The exit record said only "it died through a path no hook could see"
for exactly the deaths worth explaining: the AppImage SIGBUS and the
unexplained 2026-09-28 15:31Z death (no core, no OOM, no panic). The
GUI waits on its sidecar and gets the status, but only logged it.

- The record type moves to `nanna_core::exit_record` (the GUI links
  nanna-core, not nanna-daemon); `nanna_daemon::exit_reason` re-exports.
- On a sidecar exit the GUI adds `observed_exit` (code or signal, and
  whether it sent the kill) when the record is still that process's
  `running` marker: same PID AND written after the spawn, so a reused
  PID never matches. Written before the exit is flagged, so no respawn
  races it. The record still reads as unclean.
- The next boot logs "... the app saw it end by signal 9 (SIGKILL) at".
- `nanna doctor` gains `daemon.last_exit`: a live daemon (Linux checks
  /proc/<pid>/comm), a clean stop, or a WARN for an unclean death, a
  panic or an IPC-server failure, with the log path and coredumpctl.

Verified live: isolated GUI + daemon, SIGKILL the daemon, the record
gains the signal, the auto-restarted daemon's warning names SIGKILL,
and doctor reports ok while it runs and warn after it is killed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ts, hand-backs reset on completion

Found by reviewing the Stage 3 card-run code:

- The pause marker was looked for in a card's newest 64 activity rows;
  64 renames or reorders after a Stop pushed it out and the stall sweep
  took the paused card back for the router. New
  `TaskRepository::newest_activity_of(task_id, actions)` reads markers
  by action (run start/end, and the rate-limit wait).
- A stopped card reassigned to another member stayed `in_progress` with
  no run forever. A pause now holds only for the member it stopped (the
  `run_ended` row names who held the card); on the reassignment wake the
  card goes back to `pending` (actor `handover`) and the new member's run
  starts. An unnamed marker is never un-paused.
- `hand_backs_in_a_row` now stops at a `completed` row, so hand-backs in
  different recurrence rounds no longer add up to the bound.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…vector

`MemoryService::list_all`, `stats` and `with_store_timescale` read the
store through `all_entries()`, which clones every entry whole: the
current embedding and every model bucket (6 KiB per 1536-dim vector per
model). They need content, FSRS state, or a timestamp. They now project
under the read lock with `map_entries`, as the export already does.

Measured on the release daemon, isolated, on a copy of the operator's
3 730-memory store, `memory.list` + search + stats every 2 min: RSS after
nine cycles ~425 MB before, ~292-306 MB after (MALLOC_ARENA_MAX=2 on the
old code: ~352 MB). Idle it is flat at 142.5 MB either way.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
glibc keeps freed pages at the high water. Each 14 MB `memory.list`
reply (a Value tree, then a string, then a frame) stepped a long-lived
daemon's RSS up and it never came back: on a copy of the operator's
store, idle 142.5 MB -> 608 MB after ~40 light load cycles, still
climbing.

`crate::heap::after_large_reply` runs `malloc_trim(0)` on the blocking
pool after the IPC writer has sent a reply of >= 1 MiB (so its string is
dropped), at most once per 10 s (~4.5 ms each). The boot trim moves into
the same module. Linux glibc only; elsewhere a no-op.

Same load, nine cycles: old ~425 MB, MALLOC_ARENA_MAX=2 ~352 MB, clone
fix ~300 MB, clone fix + trim ~178 MB, and it holds (192 MB later).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ore and after

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Version 0.3.37 -> 0.3.38 in the workspace, root package, GUI and Tauri
config (the guard: base matches tauri.conf.json and differs from the
published 0.3.37). Release notes for this run; README names doctor's
new daemon.last_exit check.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ld flat

A longer sample of the trimmed daemon (15 cycles) reads ~213 MB with one
267 MB peak, not the ~178-192 MB it showed at 9. Say so: still about a
third of the untrimmed rate (~439 MB at the same point).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@physics515
physics515 merged commit 6dbb5da into master Oct 5, 2026
13 checks passed
@physics515
physics515 deleted the nanna-nightly-2026-10-05 branch October 5, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant