Skip to content

[Bug]: archive-completed of a Todo with a released lease record drifts the runtime-shadow qualification (orphaned lease stays in the candidate head) #4315

Closed
#4358
Closed
[Bug]: archive-completed of a Todo with a released lease record drifts the runtime-shadow qualification (orphaned lease stays in the candidate head)#4315
#4358
Labels
bugSomething isn't workingcontrol-planeQuota, todo, scheduler, registry, state, goal, or runtime control-plane change.direction/shared-coordinationShared-goal authority and cross-host coordination contracts.

Description

@wchwawa
wchwawa
opened on Sep 13, 2026

Summary

Under active coordination.runtime_shadow capture, todo archive-completed on a Todo that still holds a released task-lease record makes the bounded qualification drift: coordination-shadow inspect reports drifted / shadow_projection_drift, and qualify / read-candidate reject from then on. The same archive of a Todo that never had a lease stays matched. Restoring bytes does not requalify; the only recovery is rollback --provider-revision … --execute plus a fresh bootstrap --execute, which discards the captured lineage.

This is the ordinary hard_lease flow (task-lease acquire → todo complete with the lease → todo archive-completed), so it blocks the D3 card's "audit sustained mixed-writer coverage against the final command matrix". #4167 declared it as the pending ladder row s2c2.archive_after_leased_completion_parity and made s2c2.parity_equal archive nothing rather than hide it. Nothing on main has changed the path since; #4286 isolates the canonical (post-promotion) archive transaction in todo_archive.ts and does not touch this legacy capture path.

Issue origin: observed and reproduced in a disposable environment (real CLI, production FileAuthorityStore, no live Goal).
Reproduced on: f38847b1c (first seen) and 58dbaeaec (today).

Reproduction (public CLI only)

Registry: one hard_lease Goal, registered_agents: [agent-a, agent-b], coordination.runtime_shadow = {schema_version: loopx_coordination_runtime_shadow_config_v0, enabled: true, provider: file_v0}; disposable --runtime-root.

loopx --registry R --runtime-root RT --format json coordination-shadow bootstrap --goal-id G --execute
loopx … todo add --goal-id G --role agent --text "Task to complete." --task-class advancement_task        # todo_id = T
loopx … task-lease acquire --goal-id G --todo-id T --owner agent-a --idempotency-key k --ttl-seconds 120   # version 1
loopx … todo complete --goal-id G --todo-id T --agent-id agent-a --task-lease-idempotency-key k \
       --task-lease-expected-version 1 --evidence validation://x --no-follow-up
loopx … coordination-shadow inspect --goal-id G          # status=matched
loopx … todo archive-completed --goal-id G --max-active-done 0 --execute
       # coordination_runtime_shadow.outcome=delivered, drain.last_cursor=6
loopx … coordination-shadow inspect --goal-id G          # status=drifted, reason_code=shadow_projection_drift

Control: the same sequence without the lease (soft_claim, todo complete without lease flags) ends with inspect matched after the archive.

Observed difference between the two projections after the archive (todos partition equal, handoff_mode equal):

  • source projection (build_runtime_shadow_source_snapshot): leases: []
  • candidate head (last transaction of the runtime-shadow lineage): leases: [{todo_id: T, status: "released", owner: agent-a, version: 1, …}]

Root cause

  • The source snapshot projects only leases whose Todo is still in the active Todo projection; a lease file whose Todo is gone stays in lease_inventory but is not projected. test_source_snapshot_preserves_inventory_without_projecting_orphan_leases pins that rule.
  • todo archive-completed under capture prepares and commits one todos-partition entry. The candidate's leases partition still holds the released lease committed by the earlier fence-close transaction (task_lease_fence_close), and no writer ever retires it.
  • After the archive the source therefore has zero projected leases while the head keeps one, so qualifySnapshot compares different head digests.

Candidate fixes (owner decision)

  1. Archive writer retires orphaned leases. When archive-completed moves a Todo that has a lease record, it also prepares/commits a leases-partition entry whose projection drops the archived Todos' leases. Keeps the orphan rule and the existing test; one command then produces two partition entries, so their ordering and a crash between them need the same settle-from-receipts treatment as today's single entry.
  2. Source snapshot keeps released leases of archived Todos. Changes the orphan rule and test_source_snapshot_preserves_inventory_without_projecting_orphan_leases; a bootstrap baseline would then import every historical released lease, which grows the live head.
  3. Compose-time retirement in the TypeScript head. composeLocalAuthorityShadowHead drops leases whose Todo left the todos partition when a todos entry commits. No new writer entry, but the head is then derived rather than captured, which weakens "one receipt per mutation" and replay determinism.

If the roadmap retires the legacy archive writer before D3 (T4), closing by retirement is also an answer; the ladder row would then need a canonical-archive twin instead.

Option 1 looks most consistent with "one receipt per mutation", but it is your call. Once decided, I can turn the pending ladder row into an executable regression with a matching mutant within the qualification scope.

Evidence

  • Pending declaration and reproduction notes: examples/shared-goal-authority-e2e/README.md (pending rows paragraph), loopx/control_plane/testing/authority_e2e_ladder.py (PENDING_ROWS), RFC §11.2 in both languages.
  • Related rows that pass today: s2c2.parity_equal (mixed writers without archive), s2c2.parity_divergent_detects_foreign_edit (drift detection and rollback recovery).

Activity

  1. wchwawa
    added
    bugSomething isn't working
    control-planeQuota, todo, scheduler, registry, state, goal, or runtime control-plane change.
    direction/shared-coordinationShared-goal authority and cross-host coordination contracts.
    on Sep 13, 2026

  2. huangruiteng commented on Sep 13, 2026

    @huangruiteng
    huangruiteng
    on Sep 13, 2026
    Collaborator

    Thanks for the precise reproduction and for keeping the ladder gap explicit. Confirmed on current main 6b337bc with the real public CLI and production FileAuthorityStore, using only disposable synthetic goals:

    • Leased completion: inspect=matched at cursor 5; archive delivers the Todos entry at cursor 6, then inspect/qualify/read-candidate report drift and refuse qualification. Source leases = 0, candidate leases = 1 (released); the physical lease inventory still contains 1 record.
    • Unleased soft_claim control: archive remains matched, and bounded qualification succeeds. Reading the archived Todo correctly returns todo_missing with a qualified lineage.
    • Restoring the pre-archive Markdown bytes does not repair the candidate history. delivered is transport/receipt evidence; the writer explicitly reports parity_verdict=not_evaluated.

    The root cause is confirmed: archive captures only {handoff_mode, todos}, while the source's lease membership depends on the active Todo set. The two capture partitions are physically separate but not semantically independent. #4286 is now merged, but its canonical archive transaction does not change this pre-promotion capture path. This is a D3 capture-qualification blocker; it does not establish a File/SQLite storage-engine failure or failure of the canonical archive command.

    There is also an adjacent failure the fix needs to cover. I reproduced this sequence on the same revision:

    1. Complete and archive a leased Todo before bootstrap, retaining its historical lease file.
    2. Bootstrap: inspect=matched because the baseline correctly excludes that orphan.
    3. Add another Todo and acquire its lease.
    4. The lease capture stalls with source_partition_continuity_unproved; inspection then refuses qualification with outbox_pending.

    beginLeaseOutboxEntry computes both its before-image digest and after-image from the full physical lease directory (current code), while the source snapshot excludes archived/orphaned lease records. A one-time archive-side removal alone would therefore leave continuity inconsistent for subsequent lease writers.

    Direction: prefer option 1's explicit, source-correlated retirement, including that subsequent-writer boundary. Keep historical lease files/receipts for audit, and preserve the existing source rule. Retirement here means removing an edge from the current capture projection, not deleting history or releasing an already released lease again.

    Please make the repair and executable ladder row/mutant one cohesive package, with these acceptance conditions:

    • Bootstrap, archive capture, and subsequent lease capture use the same current-lease membership rule and compatible before/after digests. Keep the semantic rule in the typed coordination owner; Python should adapt the Markdown source rather than grow another rule implementation.
    • If archive produces two partition entries, durably prepare their source correlation before the primary replacement. A crash or bounded drain between entries must remain visibly pending/unqualified until settlement, with deterministic replay and no duplicate effects. Do not manufacture a physical task_lease_record mutation merely because the current schema associates that source kind with the leases partition; represent the archive-caused projection change honestly and validate its source proof/lock ordering.
    • Cover leased archive, unleased control, preview/no-op, retained historical leases at bootstrap, and a subsequent lease acquisition after archive/recovery. Preserve unrelated active leases and archived Todo dependencies. Exercise crashes around preparation, primary replacement, markers and drain, plus interleaving with a lease writer.
    • Activate s2c2.archive_after_leased_completion_parity, add archive to the mixed-writer coverage, and make a matching mutant fail when retirement or membership continuity is omitted. Use the real CLI/File provider; include other real providers if their shared store path changes. Keep the >=10-day soak and promotion holds explicit.

    I would not choose option 2: importing all historical lease files changes the live-state contract and growth characteristics. A deterministic derived head (option 3) is not inherently non-replayable, but a filter added only to composeLocalAuthorityShadowHead would not reconcile source digests, partition receipts/cursors and retained-history replay. That would be a larger protocol change, not a local shortcut.

    Nor should this wait for T4: the current roadmap requires qualified capture for D3 and puts full legacy-writer retirement after approved cutover. Removing the test now would leave that migration window unproved.

    One recovery wording clarification: rollback archives the old lineage/outbox (archive_retained=true); rebootstrap starts a new qualification lineage. Historical audit evidence is retained, but qualification continuity is reset. The additional reproduction above shows why a green bootstrap alone is not sufficient recovery evidence.

    No runtime fix is included in this comment; keep this issue and the promotion hold open until the repair and regression evidence land.

  3. songoow
    added 2 commits that reference this issue on Sep 14, 2026

    fix(coordination): drop orphaned leases from the candidate head

    5a622f3

    fix(coordination): drop orphaned leases from the candidate head

    31be2b5
  4. huangruiteng
    closed this as completedin #4358on Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcontrol-planeQuota, todo, scheduler, registry, state, goal, or runtime control-plane change.direction/shared-coordinationShared-goal authority and cross-host coordination contracts.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions