Skip to content

Adopt an own start job that settled before the first birth observation - #254

Merged
HemSoft merged 2 commits into
mainfrom
fix/issue-174-start-settled-before-observe
Oct 23, 2026
Merged

HemSoft merged 2 commits into
mainfrom
fix/issue-174-start-settled-before-observe

Conversation

@HemSoft

@HemSoft HemSoft commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Fixes an intermittent start failure in the source VM owner and the host Git owner (#174).

Problem

captureBirth correlated a unit's first invocation with our start only if that invocation was observed while our accepted start job was still attached (git_vm_owner_linux.go and git_early_stop_linux.go: if invocation == "" && before.Job.ID == 0 { refuse }). The units are Type=exec, so systemd completes the start job as soon as the stub's exec succeeds: across 158 source starts in the journal, Starting → Started took a median of 16 ms and a minimum of 4.9 ms. In that window the owner (nice 10, GOMAXPROCS=2) writes the durable StartJob record and then observes the unit. Under CPU contention its first observation came after the job had finished; it refused, the start stayed unsettled, stop refused at before-capture, and Close refused with "source VM ownership requires recovery".

Seen once in the aggregate E2E qualification (#251's close-provider at 3133646: start not settled, invocation empty, the stub exiting after its 10 s grant wait) while another agent's race builds competed for CPU; an immediate rerun passed. The rule came in with #244 (source VM owner) and #226 (host Git owner).

Fix

A job-free first sighting is accepted only when the job tracker proves that our own StartJob was removed with result done and that no other job settled on that unit since our StartTransientUnit; otherwise the start still fails closed. This rests on ordering checked in the sources: systemd 259 job_uninstall() emits JobRemoved before detaching the job from the unit, and godbus v5.1.0 delivers signals synchronously ahead of any later reply, so a later job-free observation follows every relevant JobRemoved. A replacement start by anyone else needs its own job, which is either still attached (path mismatch, refused) or recorded as foreign. The tracker gains a foreign mark and a non-consuming settled check; WaitJob is unchanged. Both owners use the same proof.

Evidence

  • TestGitVMOwnerStartAdoptsOwnJobSettledBeforeFirstObservation, TestGitVMOwnerUnbornStartedAttemptBlocksClose, TestSystemdJobSettledRequiresOwnDoneJobAndNoForeignJob, TestGitBirthStartAdoptsOwnJobSettledBeforeFirstObservation, and the reworked TestGitBirthRejectsSettledJobBeforeAnyInvocationWasCorrelated (unproven, another job, foreign and failed jobs rejected; own done job accepted). All six accepting cases fail on the base; every rejecting case passes on both.
  • Real host, under the shared VM lock, with a temporary 200 ms delay after StartTransientUnit: the base failed at stage unit-start with the production signature; this branch passed and stopped the unit.
  • go vet ./... and go test -race ./... pass (reviewer package 236 s); new tests pass -race -count=20.
  • GIT-UNIT.md, GIT-VM-SOURCE.md and SYSTEMD-SUPERVISOR.md describe the rule.

🤖 Generated with Claude Code

HemSoft and others added 2 commits October 23, 2026 07:07
The source VM and host Git units are Type=exec, so systemd completes the
start job as soon as the stub's exec succeeds (median 16 ms, min 4.9 ms
on the host). A starved owner could make its first observation after
that, see the new invocation with no job attached and refuse birth. The
start then stayed unsettled, stop refused before capture and Close
required recovery forever.

The job tracker now proves the missed correlation without waiting or
consuming the result: our accepted job was removed with result done and
no other job was removed on the unit since StartTransientUnit. systemd
emits JobRemoved before it detaches the job and godbus delivers that
signal ahead of any later reply, so a job-free observation follows every
relevant signal. Without that proof, including a failed, unknown or
foreign job, both owners refuse as before.

Refs #174

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #174

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@HemSoft

HemSoft commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T11:22:59.370042Z 5760b05 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 5760b059dd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant