Repository navigation
Decode the loader status instead of printing a raw Win32 number (#2186) - #2194
Conversation
A managed-Postgres bootstrap that could not start reported "initdb failed (exit code -1073741515) for ... Output:" and nothing else. -1073741515 is 0xC0000135, STATUS_DLL_NOT_FOUND: Windows killed the process in the loader before a line of its own code ran, which is also why Output: was blank and always would be for this class of failure. DarlingToolExitCode decodes an NTSTATUS exit code into its name, separates loader failures from crashes, names the two causes that account for nearly all of them (the bundled MSVC runtime absent from pg-runtime\pgsql\bin, or a service account that cannot read a user-profile install tree) and the two checks that tell them apart. An empty capture now says it is empty BECAUSE the process was killed before it could write. Wired into every bundled-binary failure: initdb (bootstrap and upgrade), pg_ctl start/status/stop/reload, pg_upgrade and its --check, and the post-upgrade analyze. Two messages also stopped blaming the wrong thing -- pg_ctl status no longer calls the data directory unusable when the verdict came from Windows rather than from pg_ctl, and pg_upgrade --check no longer reports incompatible clusters when pg_upgrade never ran to form an opinion. The #1738 refusal now quotes and decodes the probe exit code it already had instead of asserting that the binaries did not run without saying how it knew. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A crash can be transient (bad memory, a one-off), so 'restarting will not clear it' over-claimed. The useful guidance - do not just restart and hope, capture the WER entry - survives the softening. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Reviewed. This is a self-contained Darling-only C# change ( Went through the decode/diagnose/format logic in detail:
No correctness, security, or performance issues found. One very minor, non-blocking nit: Also noted: the CHANGELOG.md hunk in this PR shows as a near-total file rewrite (2676 insertions / 2674 deletions) in |
…or-surfacing # Conflicts: # CHANGELOG.md
|
Heads up from the other work stream: this PR's The Dockerfile built from the floating Your PR is the control that proved it: an unrelated change failing the identical step the same way. dev's own Linux build passed at 18:22 and would fail on its next run. #2199 pins the build stage to |
| _logger.LogWarning( | ||
| "Managed Postgres stop reported exit code {ExitCode}: {Output}", | ||
| DarlingToolExitCode.Describe(exitCode), | ||
| DarlingToolExitCode.FormatOutput(output, exitCode)); |
There was a problem hiding this comment.
Minor: this changes the {ExitCode} structured-logging field from an int to a decorated string (e.g. "3" vs "-1073741515 = 0xC0000135 STATUS_DLL_NOT_FOUND"). Same pattern at the pg_ctl reload failed log (line ~2075) and the Post-upgrade analyze reported log in DarlingStoreUpgrade.cs (line ~2252).
For the exception-message call sites this is clearly the point of the PR, but these three are ILogger message templates, which typically flow into structured log sinks (Seq/App Insights/etc.) where ExitCode may be queried/alerted on as a number (e.g. ExitCode != 0, aggregations, dashboards). Collapsing it to a mixed-format string could silently break those queries downstream.
If that's a real consumer, consider keeping the raw int bound to {ExitCode} and adding the decoded text as a separate {ExitCodeDescription}/{Diagnosis}-style parameter instead of replacing the numeric field.
ReviewScoped entirely to the Darling service ( Lite/Darling parity: No drift to flag. Lite has no managed-Postgres bootstrap at all (embedded DuckDB, no bundled Postgres runtime, no Correctness: Traced the decoder ( Security: No injection surface — Performance: Negligible — string formatting only on already-slow failure/log paths, nothing in a hot path. One inline note on a structured-logging field-type change (int → decorated string) in three |
|
#2199 is merged, so the SDK pin is on dev now. Your Confirmed the fix works before merging it rather than after: |
Status: green on everything this PR can affect, blocked by a repo-wide CI break it did not causeFinal sha
The Linux failure is #2200, filed separatelyIt fails before compiling anything:
#2200 has the diagnosis and the suggested one-line fix (pin the Dockerfile tag to the band On the CHANGELOG conflictIt came back as a single whole-file conflict rather than the usual both-added-an-entry kind: dev's copy is CRLF, and my original edit had silently normalized the file to LF, so both sides looked entirely rewritten. Resolved by taking dev's file wholesale and re-inserting the entry and link-ref with CRLF preserved. Also filed#2197 - the follow-on missing-credential CLI message still reads as a first run even after a bootstrap has already failed. That is the half of #2185's misdirection that lives in Verified locally on the merged treeFull |
| if (!OperatingSystem.IsWindows()) | ||
| { | ||
| return; | ||
| } |
There was a problem hiding this comment.
Nit: every other Windows-only gate in this test suite uses Assert.SkipUnless(OperatingSystem.IsWindows(), "...") (see DarlingConfigTests, DarlingFileSecurityTests, DarlingPgRuntimeVersionPinTests, QueryStoreSliceRepairLiveTests). This test instead early-returns, so on non-Windows CI it reports as passed with zero assertions run instead of skipped. That's a quiet deviation from the "236 skipped (all gated)" accounting the PR description relies on, and it means a future regression here wouldn't show up as a gap in the test report on Linux CI — it would just look like a passing test. Worth switching to the established Assert.SkipUnless pattern for consistency:
Assert.SkipUnless(OperatingSystem.IsWindows(), "requires the real Windows loader to reproduce a loader-kill exit status.");
Review summaryReviewed the diff (CHANGELOG.md, Scope check: this is Darling-only (managed-Postgres bootstrap/upgrade). Lite has no equivalent managed-Postgres code path ( Correctness spot-checks, all held up:
Left one inline nit on a test-convention deviation (early-return instead of No missing-index DMV suggestions, no security concerns (no new user input reaches these code paths — |
Fully greenMerged the current
#2200 is closed as a duplicate of #2199. CHANGELOG byte checkFlagged by a sibling lane:
The earlier whole-file conflict was this hazard, caught and resolved by taking dev's copy and re-inserting with a script that splits and joins on CRLF (which preserves the bare CRs, since they sit inside the split line strings rather than at the boundaries) rather than saving the file from an editor. Not merging, per instructions. |
…or-surfacing # Conflicts: # CHANGELOG.md
|
Reviewed the diff (CHANGELOG.md, Correctness — Traced the NTSTATUS-range logic in Lite/Darling parity — No parity gap. Lite has no managed-Postgres component (it embeds DuckDB, not a bundled PostgreSQL binary), and its only Security — All the paths fed into Performance — No concerns; this is exception/log-message construction on failure paths only. One very minor (non-blocking) nit: in Nothing else stood out. The test coverage here is unusually thorough (the red-then-green methodology, the live loader-failure repro, and the source-text wiring pin that guards against a repeat of #1738's "correct code nothing calls" failure mode). |
Green again on top of #2198Merged
Full The CHANGELOG conflict was one line this time, and that is the pointLast round it came back as a ~5,350-line whole-file conflict. This round it was a single 5-line hunk at the link-ref block - Verified on both measures, as before:
Resolution was scripted, not editor-saved: split and join on CRLF, which leaves the 6 bare CRs untouched inside the line strings. Not merging, per instructions. |
… pass (#2186) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Reviewed this against correctness, Lite/Darling parity, security, and performance. Scope check: all changes are confined to Correctness:
Security: no new process/file/network surface - No blocking issues found. The test coverage (decoder pins, shipped-message pins, and a source-wiring pin) is notably thorough for this kind of change. |
…or-surfacing # Conflicts: # CHANGELOG.md
ReviewReviewed the diff for correctness, Lite/Darling parity, security, and performance. Summary: no blocking issues found. Scope check (parity): This PR only touches Correctness:
Minor/non-blocking style nit: In Security: No new external input handling — exit codes and exe paths originate from the service's own spawned processes, not untrusted input. No secrets or SQL touched. Performance: All changes are on error/failure paths (dictionary lookups + string building), no hot-path impact. |
Green on top of #2202Merged
Full Carrying
|
Every container build started failing tonight, on dev and on both open PRs, without any commit causing it. The Dockerfile built from the floating mcr.microsoft.com/dotnet/sdk:10.0 tag; that tag advanced to an image shipping SDK 10.0.400, and global.json requests 10.0.302 with rollForward "latestPatch", which rolls only inside the 3xx band. So the publish inside the container died with: A compatible .NET SDK was not found. Requested SDK version: 10.0.302 global.json file: /src/global.json Installed SDKs: 10.0.400 surfacing as a bare "exit code: 155" with no compiler diagnostics at all, which is what made it read like a code failure rather than an image drift. The runner-side publish in the same job succeeded seconds earlier, because setup-dotnet resolves its SDK FROM global.json (global-json-file: global.json). Only the container held an independent opinion about which SDK to use, and that opinion was supplied by whatever the upstream tag happened to point at that hour. Pinning to 10.0.302 removes the second opinion: the container now uses exactly what the repo asks for, and this line and global.json get bumped together, deliberately, rather than one of them being bumped by a tag move at Microsoft. The aspnet runtime stage stays on the floating :10.0 tag deliberately. global.json gates the SDK only, runtime roll-forward is permissive by design, and pinning it would mean tracking security patches by hand in a place nobody would remember to look. Confirmed repo-wide rather than assumed mine: PR erikdarlingdata#2194, an unrelated change from a different work stream, fails the identical step the same way, and dev's own Linux build passed at 18:22 and would fail on its next run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes #2186.
The report
A managed-Postgres bootstrap failed with this, and nothing else:
-1073741515is0xC0000135=STATUS_DLL_NOT_FOUND. Windows killedinitdb.exein the loader, before a line of its own code ran, which is also whyOutput:is empty and always will be for this class of failure: the one field an operator reads is guaranteed blank exactly when the failure is a load failure. It reads as "no information available" rather than "this is a loader failure", and attention went to the follow-on missing-credential message anddarling.json, neither of which was the fault.Verifying the issue's claims first
Both held.
InitializeClusterAsyncinterpolated the raw signed code (DarlingManagedPostgres.cs:891on dev), and the #1738 sibling at line 1103 namesSTATUS_DLL_NOT_FOUNDin a comment only - there was no decoder anywhere in the file to reuse, so this adds one rather than moving one.One claim was worth testing rather than assuming: that the empty
Output:really is guaranteed. A test drives a genuine loader failure through the realRunToolAsyncand pins it - exit code-1073741515, captured output length0.Before / after, verbatim
Produced by a real Windows loader failure driven through the real
EnsureRunningAsync, not a hand-written string: a scratchpg-runtimewhoseinitdb.exeis a binary whose app-local DLL cannot be resolved (a doctoredPATH), which Windows kills with0xC0000135and zero bytes on both streams. Nothing touched DARLING01 or the repo'spg-runtime.zip.Before (dev):
After (this branch):
The leading clause is deliberately unchanged: existing field reports and the issue tracker are searchable by
initdb failed (exit code, so the fix must not rename what operators paste into search.The restricted-token caveat in check (a) is in there because the field thread's most confusing evidence was
initdb --versionsucceeding by hand while the service run failed. PostgreSQL's frontend utilities re-execute themselves viaget_restricted_token()/CreateRestrictedProcesswith the Administrators and Power Users groups removed, so a tree readable only via Administrators fails the real run while passing the by-hand check (paquier.xyz, PG source). Verified before shipping the claim, and it is attached only to tools that actually do this (initdb, pg_ctl, pg_upgrade) - never tovacuumdb.What changed
DarlingToolExitCodedecodes any exit code at or above0xC0000000- the NTSTATUS error range, which no real program exits with deliberately - so an unlisted status is not left as opaque as-1073741515was. It separates loader statuses (0xC0000135,0xC0000139,0xC000007B,0xC0000142,0xC0000022) from crashes (0xC0000005,0xC0000374,0xC0000409); a crash must not send anyone hunting for missing DLLs, which would be the same wrong-direction error pointed somewhere new.Wired into every bundled-binary failure, per the issue's "or any bundled Postgres binary": initdb (bootstrap and upgrade), pg_ctl start/status/stop/reload, pg_upgrade and its
--check, and the post-upgrade analyze.Two messages also stopped blaming the wrong thing, which the decode made visible:
pg_ctl statussaid "the data directory is not usable" for every unexpected code. That is pg_ctl's own exit 4 talking; on a Windows status it pointed an operator at deleting a healthy store to fix a missing DLL. The verdict is now conditional.pg_upgrade --checkreported "the clusters are not compatible" on any non-zero exit. Only pg_upgrade can reach that verdict, so on a Windows status it was invented.NOTHING has been changedstill holds either way.And the #1738 refusal asserted "its binaries did not run" while discarding the one piece of evidence it held - the probe's own exit code died in a local.
ReadRuntimeMajorAsyncnow carries it out, and the refusal quotes and decodes it.Tests
Watched red first against an inert seam (the new API returning today's behavior): 16 failed, 9 passed, then green. The 9 that passed under the inert seam are the ones asserting today's correct behavior is preserved - ordinary exit codes stay bare, real output passes through untouched - so the red was the decode, not the harness.
DarlingToolExitCodeTests- the decode, both diagnosis families, the empty-output note, that ordinary codes (initdb's 1, pg_ctl status's 3) are left completely alone, and the liveRunToolAsyncpin above.DarlingManagedPostgresTests- the shipped messages: the field report rebuilt from its own numbers, the conditional data-directory verdict, and that the start failure diagnoses before it points at a server log that a loader failure guarantees is absent. Plus a wiring pin at the source, because three correct builders that no throw site calls is precisely what Runtime advance has no direction check: a package with an OLDER PostgreSQL major replaces a working newer runtime and the store cannot start (hit on DARLING01) #1738 already was, and behavioral coverage cannot reach it - reproducing it needs a bundled Postgres that dies in the Windows loader, which is not something a CI runner can be asked to arrange.Full
Darling.Tests: 4256 passed, 0 failed, 236 skipped (all gated). Service project builds 0 Warning(s). The two warnings in a full rebuild (AbandonableStep.csCA1068,NpgsqlRootCertificateValidationTests.csCA2022) are in files this branch does not touch.Not in scope
The installer half is #2187.
🤖 Generated with Claude Code