Skip to content

emrg: the scheduler asks its subject's kind before every open - #2096

Merged
argszero merged 3 commits into
masterfrom
fix/scheduler-asks-the-kind-before-opening
Oct 11, 2026
Merged

argszero merged 3 commits into
masterfrom
fix/scheduler-asks-the-kind-before-opening

Conversation

@argszero

Copy link
Copy Markdown
Owner

Closes #2095

emrg/server/scheduler.py was the one daemon module outside the kind-before-open family: grep -n "non_regular_kind\|NotARegularFile\|special_file_kind" emrg/server/scheduler.py answered nothing, and its readers had the family's shape — an exists() gate (true for a FIFO) and then a bare read_text inside try/except OSError. An open is not a read: on a FIFO the open waits for a peer and raises nothing, so no handler fires and the call never returns. The scheduler runs on the daemon's event loop, so the block is every connected client, the scheduler and the evolution cycle.

What changes

Three module-level helpers, and every site routed through them:

  • _refuse_non_regular(path, opened) — asks emrg.tools.base.non_regular_kind (the shared predicate, imported rather than restated) and raises emrg.memory.NotARegularFile, which is an OSError naming the kind. A path that does not exist yet is not a refusal: the stat's FileNotFoundError returns, so read_table still seeds a missing tasks.yml and _append_task_run still creates its JSONL.
  • _read_text(path) — path.read_text() behind that guard. Eleven call sites: read_table, _read_custom_template, _resolve_project_path, _load_project_config, _load_saturation_state, _report_interrupted_cycle, _load_next_run_state, _load_task_runs, _build_evolution_prompt, _ensure_emrg_project_entry, list_templates.
  • _open_text(path, mode) — open(...) behind the same guard, for the two write sites (_write_recovery_receipt, _append_task_run): opening a FIFO for writing waits for a reader, so the writer blocks in the same place.

No reader's contract changes. Because NotARegularFile is an OSError, every call site's existing handler keeps the answer it already gives a file it cannot read — TableUnreadable for tasks.yml, None / {} for projects.yml, False for the saturation state, [] for the task runs. The only new behaviour is that the answer arrives instead of never arriving. The one site with no handler of its own — _build_evolution_prompt, reading a task's configured template_path — now names the kind rather than hanging.

emrg/server/daemon.py and the tool layer are untouched: this is the module that never joined, not a second spelling of the rule.

Verification

  • tests/test_scheduler.py — 142 passed.
  • Full suite — 4734 passed, 16 skipped (308 s).
  • python -c "from emrg.client.app import run_client" and python -m emrg --help: rc 0.
  • Local guards rc 0: check-undefined-names, check-doc-count, check-citation-resolves, check_nonlocal, check_unbound_reads, check-extension-load.

Both directions, tests/test_scheduler.py::TestSchedulerReadersAskTheKindBeforeOpening (12 tests; every call goes through tests/bounded_read.within, so a regression fails on the deadline instead of wedging the suite):

  • with this change: 12 passed;
  • with master's emrg/server/scheduler.py under this same test file: 11 failed, 1 passed in 55 s, every failure on tests/bounded_read.py:62 — AssertionError: the read did not return within 5.0s. The one that passes is the regular-file control, which is what tells this guard from a reader that answers "absent" to everything — each value asserted above is also what a missing subject answers.

The three persistence tests in the module (test_task_handler_task_runs_persist_across_restart, test_task_handler_task_runs_are_never_truncated, test_evolution_cycle_log_work_not_truncated) caught a real defect in an earlier draft of the helper: non_regular_kind stats its subject, so an append to a file that did not exist yet raised FileNotFoundError and the writer silently declined to write. The FileNotFoundError branch above is that fix, and those tests are why it is there.

CI: pending.

@argszero

Copy link
Copy Markdown
Owner Author

test-windows cause and fix — a36eee38

The failing leg was test_the_control_with_regular_files:

AttributeError: module 'os' has no attribute 'mkfifo'

os.mkfifo does not exist on Windows. The other 11 tests in the class carry
@_needs_mkfifo (pytest.mark.skipif(not hasattr(os, "mkfifo"), reason="this platform has no FIFOs"));
the control did not — and it did not need one. It called self._handler(...), whose only job is
to plant a FIFO, and then replaced that path with a regular file one line later. The control was
platform-dependent for a file it threw away.

Fixed by building the handler without the FIFO helper. The control now runs on Windows instead
of being skipped there: a control that only runs where its subjects run is not a control, and the
platform least able to run the subjects above it is the one where the contrast matters most.

Measured in both directions, using a plugin that deletes os.mkfifo at configure time to reproduce
the Windows condition inside a Linux process:

  • with the fix: 1 passed, 11 skipped — the eleven FIFO subjects skip, the control runs
  • without it: 1 failed — AttributeError: module 'os' has no attribute 'mkfifo', the CI error

Full suite 4734 passed, 16 skipped — the same totals as before the change, so nothing moved into
skipped on Linux either.

CI on this push is pending; this comment records a cause, not a verdict.

`Path.write_text` is `open(path, "w")`, so it blocks on a FIFO exactly where
`read_text` does: it waits at the open for a reader, raises nothing, so no handler
can fire and the call never returns — and the scheduler runs on the daemon's event
loop.

The previous commit routed the module through `_read_text` and `_open_text` and its
body says "every site routed through them". Five sites reach their subject through
`Path`'s own method instead, and none of them asks:

    _write_custom_template    <templates>/<name>.md.tmp
    _save_saturation_state    <saturation>/<task>.json
    _write_heartbeat          <task-runs>/<task>.heartbeat.json
    _save_next_run_state      <next-run>/<task>.json
    _write_final_summary      <logs>/summary.json

Measured on that commit (`a36eee38`), one child process per arm under an 8 s cap,
driving each real writer with a `mkfifo` at its subject: all five **did not return**.

`_write_text` is the third of the three shapes an open takes, next to the two the
module already has. Three of the five callers swallow `OSError` by contract
("best-effort, never raises"), so their refusal is a return rather than a message —
the same reading the read side gives.

The census is what kept missing it, so it is now asked of the source rather than of a
reviewer's memory: `test_no_open_shape_reaches_its_subject_unasked` parses
`emrg/server/scheduler.py` and requires every `read_text`, `write_text` and `open(` to
be inside the one helper written for it. On the parent commit that leg fails naming
exactly the five functions above.

Also names the event loop's reads **and writes** in the refusal: the sentence is
asserted as the reason, and for a writer the read-only wording named the wrong
operation.

Verified: the six new legs fail on `a36eee38` (naming the five functions and blocking
on the deadline) and pass here; 4 mutation arms KILLED — the helper stops asking, one
site bypasses it, a blanket refusal that writes nothing, a raw `write_text`
reintroduced — plus 2 controls SURVIVED; full suite 4728 passed, 29 skipped.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20261011-123924

Reading, not the PR page. scripts/check-merge-landing-diff.py 2096 against base 99477516:
landing tree e47ec54b9623, changing 2 paths (emrg/server/scheduler.py +104 −21,
tests/test_scheduler.py +358 −0). The PR page renders 7 paths; 5 of them are the base's own
later commits
(daemon.py, skills/registry.py, test_session.py, test_skills_registry.py,
test_daemon.py), shown as reversals this PR does not make — reviewed from the merge base, not
from the page.

scripts/check-merge-plan-suite.py 2096 → that tree: 4749 passed, 17 skipped (304 s).

Both directions (head a87999f0, one arm at a time):

  • as written: tests/test_scheduler.py -k AskTheKind → 19 passed.
  • guard inert (_refuse_non_regular returning immediately): 16 failed, 3 passed — every FIFO
    test failing on the deadline at tests/bounded_read.py:62 ("did not return within 5.0s"), and the
    three survivors are exactly the controls (the two pipe-free ones and the source-shape census).
    So the suite's signal is the guard, not the fixture.
  • the census leg separately: reintroducing Path.write_text at one site (the exact shape the
    docstring says was missed three times) → write_text is called outside _write_text() by ['_write_final_summary']. It names the offender, which is what a structural guard has to do.

What I checked on the code, since a raise is a behaviour change and not only a de-hang:

  • _build_evolution_prompt now raises NotARegularFile where it used to hang. That path is called
    outside any try in run(), so the raise ends the handler task — but a directory at the same
    path already did exactly that (IsADirectoryError is an OSError), so this introduces no failure
    class the site did not have; it converts an unbounded block into an existing one.
  • _write_custom_template propagating is likewise the directory case that was already there, and
    template_create / template_update return (ok, error) to the daemon frame that owns them.
  • _refuse_non_regular catching only FileNotFoundError keeps absence meaning "the writer is about
    to create it" — the #2095 shape (a kind guard that breaks an appending writer on a fresh
    install) is not reintroduced, and test_the_control_writes_every_subject_when_it_is_absent pins
    it. A stat failing for another reason still reaches each caller's existing except OSError.
  • the two read helpers route through _read_text/_open_text/_write_text rather than three
    copies of the guard, and the census keeps a fourth site from appearing.

One non-blocking nit (comment only, no code change asked for): _refuse_non_regular's comment
says the check "is one syscall with no gap for the path to change in". The kind check is one stat,
but a stat-then-open gap exists here exactly as it does in every guard of this family
(#2062's three tools, #2074's daemon readers, #2076's memory reader) — the sentence reads as a
stronger claim than the code makes. Worth softening if the file is touched again; it changes nothing
about the fix, whose subject is a subject that was already of another kind, not one swapped in
under it.

@pm25coder pm25coder left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20261011-123215

Landing tree e47ec54b9623 (merge of master 99477516 with head a87999f0); plan-suite 4438 passed / 327 skipped; both CI legs green.

Local verification on that landing tree, forward and reverse. The 12 FIFO arms are all @_needs_mkfifo and skip on Windows (16 skipped), so the refusal behaviour itself is measured only on Linux — I do not read those skips as a pass. What I could measure:

  • forward: TestSchedulerWritersAskTheKindBeforeOpening + TestSchedulerReadersAskTheKindBeforeOpening → 3 passed, i.e. the ast census leg test_no_open_shape_reaches_its_subject_unasked plus both pipe-free controls.
  • reverse: restoring master's emrg/server/scheduler.py makes the census leg fail, naming 11 callers (read_text is called outside _read_text() by [...]) — the leg discriminates the fix from the state it replaces.

The three wrappers (_read_text / _write_text / _open_text) route every read_text / write_text / open( in the module, and both controls pin the half the guard could have broken: absence still answers [] / creates the file exactly as before.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20261011-130932

Reviewed on the landing tree e47ec54b9623 (base 99477516), because the head is stale:
check-merge-freshness.py 2096 reports STALE (head a87999f0, behind_by=3), so the green CI run
38109659676 was about a tree that can no longer be merged and refreshing it is the shape that voids
the two standing votes.

  • check-merge-plan-suite.py 2096 → final tree e47ec54b9623, suite OK: 4749 passed, 17 skipped
    (318.55 s).
  • check-merge-landing-diff.py 2096 → the merge changes 2 paths: emrg/server/scheduler.py
    (+104 −21) and tests/test_scheduler.py (+358 −0). The five other paths in diff(base, head) are
    the base's own later commits, shown as reversals this PR does not make (the documented reading
    hazard, not a defect).
  • Code read: _refuse_non_regular branches on FileNotFoundError before the kind check — the right
    asymmetry (a reader refuses, a writer creates; read_table seeds a missing tasks.yml and
    _append_task_run creates the JSONL it appends to), and NotARegularFile is an OSError, so
    every call site's existing handler keeps the answer it already gave. Routed through _read_text /
    _open_text / _write_text rather than repeated per site; the third shape (Path.write_text) is
    the one the census leg in tests/test_scheduler.py now guards.

Both directions were measured on this head: with the kind guard removed the FIFO tests fail at
tests/bounded_read.py:62 (deadline), and the census mutation reports
write_text is called outside _write_text() by ['_write_final_summary'].

@argszero
argszero merged commit 3f18f6a into master Oct 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants