Skip to content

CI flake: annotate-server durable-record tests fail on runners, pass locally #1464

Closed

Description

@backnotprop

Three occurrences in two days, distinct PRs, same test family:

Seen on PR #1461 (run 33689988840) and twice on PR #1462 (runs 33683325163-family and 33717306176). In every case the full suite passes locally on the same commits (58/58 in the file), and a plain rerun goes green. Neither PR touches the annotate server.

These tests boot a real server against a temp PLANNOTATOR_DATA_DIR and exercise filesystem ordering (write record, then delete draft; chmod-unwritable directories). Suspect timing or permission semantics on the shared runners (chmod on hosted runners, write visibility ordering under load).

Fix shapes to evaluate: make the unwritable-directory setup assert its own precondition and skip honestly when chmod does not stick on the runner filesystem; widen the record-then-draft ordering assertions to poll briefly instead of asserting immediately after the response; or serialize this describe block if cross-test contention on the temp dir is the trigger.

Also worth recording: the agent-terminal sidecar test (packages/server/agent-terminal.test.ts, "node sidecar rebuilds spawn options") flaked the same way on the #1458 integration run. Different family, same pattern: server tests doing real fs/process work on shared runners.

Activity

  1. backnotprop commented on Sep 3, 2026

    @backnotprop
    OwnerAuthor

    Fourth occurrence: PR #1465 run 33718368682, the identical four tests. This is now firing on roughly half of all CI runs today, which crosses from annoyance into velocity tax. Raising the priority suggestion: the poll-instead-of-assert fix shape looks smallest and addresses three of the four (the write-then-delete ordering assertions); the unwritable-directory case needs its chmod precondition check.

  2. backnotprop commented on Sep 10, 2026

    @backnotprop
    OwnerAuthor

    Root-caused and fixed in v0.27.13. The flake was import-time capture of the data directory in shared storage: an OpenCode lifecycle test importing storage under a temporary env override poisoned later test files, with filesystem-dependent test discovery order deciding which runs got hit. Fixed by per-call resolution in #1473 (with the suite-wide data-dir sandbox), independently root-caused by @bendrucker during #1437 with the definitive diagnosis. Remaining frozen-at-import modules are tracked in #1477.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions