Repository navigation
Store host profile: shared_buffers no longer reads stale on a 4 GB host (#5716) - #5720
Conversation
…ding derivation (#5716)
| /// <summary>#5716: both verdict producers (--check-settings / get_store_host, and the startup profile that | ||
| /// feeds the self-alert) must route their derived target through the tolerance before ClassifyVerdict.</summary> |
There was a problem hiding this comment.
The parenthetical maps the two producers backwards, and the self-alert half is not true of this verdict at all.
GatherSettingProfilesAsyncis the one that builds the startup profile:GatherStartupProfileAsynccalls it atDarlingStoreHostProfile.cs:1105, andDarlingWorker.LogStoreHostProfileAsynclogs oneLogWarningperStaleAfterHardwareChangerow off that result (DarlingWorker.cs:2569,2575-2583). It also serves--check-settings' first table andget_store_hostviaGatherAsync(:1063).ComputeAndStoreManagedConfVerdictsAsyncis the one that writes the storedcollect.managed_conf_verdictsrows, read back by--check-settings' second table (DarlingCliCommands.cs:963), the MCP self-store reader (Mcp/DarlingStoreMetricsReader.cs:2314) and the alert read.- Nothing feeds the self-alert from a stale verdict.
DarlingWorker.ReadRejectedManagedConfSettingNamesAsyncfilters the stored rows toVerdict == HostSettingVerdict.RejectedValueonly (DarlingWorker.cs:9463-9466), andApplyStoreSettingsAsync's own contract says so in its words: "PendingRestartandStaleAfterHardwareChangedo NOT fire this" (DarlingSelfAlertEvaluator.cs:5450-5452).
The test itself is right and the pin is the useful one here — both producers do need the tolerance, and both have it. It is the comment (and the same sentence in the PR body and the changelog entry) that names a consumer this verdict never reaches.
| return derivedValueMb; | ||
| } | ||
|
|
||
| return DarlingManagedPostgres.DeriveMemorySettings(ramBytesForDerivation).SharedBuffersMb == current |
There was a problem hiding this comment.
Non-blocking, but worth recording next to the ApplyWalDriftTolerance comparison this is modelled on: the sibling chose a band (IsWalSizeWithinDriftOf), this one is exact equality against the raw reading as read right now.
That is tight enough for the stores that re-render darling-managed.conf every start — WriteManagedConfFile re-derives shared_buffers from GatherManagedConfRenderInputs' fresh GetTotalPhysicalMemoryBytes() (DarlingManagedPostgres.cs:4209, ManagedConfFile.cs:188), so the file and this check read the same number within one start. It is not tight enough for a store still on the marker-gated postgresql.conf path, where the v5 block was written once in the cluster's life (DarlingManagedPostgres.cs:3864): there the stored figure is frozen at whatever ullTotalPhys said then, and QuantizeRam's own remarks are explicit that the reading is not stable — "ullTotalPhys is not guaranteed bit-identical across reboots — a balloon/Dynamic-Memory guest can report a different current total with no operator resize — and these ARE cloud VMs" (:1378-1382).
shared_buffers is ram / 4, so a 4 MB wobble in the reported total moves the derivation by 1 MB. On the issue's own host class that is 4092 → 4088 MB, derivation 1023 → 1022, in force still 1023, and the row goes back to reading stale-after-hardware-change forever — the same complaint one wobble step over. Only reachable below 4 GB (at or above it the 1 GB cap makes both derivations 1024), which is exactly the host #5716 was filed about.
If you want the band, the ingredients are already here: forgive when current equals the derivation for any raw reading that quantizes to the same GB, i.e. compare DeriveMemorySettings over the quantized reading and over the raw reading and accept either, or accept any current within 1 MB of the raw derivation. Fine to leave as-is and let a second report decide.
The changelog entry names a surface this verdict never reachedThe
The first three are right. The self-alert is not, and the code says so in its own words rather than by inference:
So an operator on a ~4 GB store was seeing this on the startup profile line, in Same sentence appears twice more, with the producers additionally swapped — in the PR body's "What changes" bullet list, and in the new test's doc comment (commented inline at
Dropping "and the self-alert" from the entry (and from the body bullet) is the whole fix. The code change itself is correct and complete — both producers genuinely needed the tolerance and both got it, before |
There was a problem hiding this comment.
Verdict: CHANGES REQUESTED
The code change is correct and complete. What I am requesting is a prose fix: the changelog entry, a shipped artifact, names a surface this verdict never reached. Detail in the two inline comments and the top-level note.
Claims
- VERIFIED — v5 writes
shared_buffersfrom the raw reading and v8 never re-states it —DarlingManagedPostgres.cs:3864-3868→:1020-1027(unquantized);BuildHardwareSizingConfAppend:1565-1580emits the other four keys only, its bullet at:1538saying "EXCLUDED STRUCTURALLY". Same on thedarling-managed.confpath:ManagedConfFile.cs:188raw to v5,:204quantized only for v8, overGatherManagedConfRenderInputs:4209. - VERIFIED — the profile derives its target from the quantized reading —
DarlingStoreHostProfile.cs:932, the one source of all eight targets. - VERIFIED — a ~4092 MB host has 1023 MB in force against 1024 —
DeriveMemorySettings:1119-1148ismin(ram / 4, 1 GB),QuantizeRam:1387-1392rounds 4092 to 4096 MB; the new test asserts both as preconditions. - VERIFIED — same shape as
ApplyWalDriftTolerance, no formula re-implemented —:914-925vs:894-902, comparison routed throughDeriveMemorySettings. - VERIFIED — applied in both verdict builders right before
ClassifyVerdict—:1043/:1045,:1508/:1510;ClassifyVerdicthas exactly those two callers (ClassifyValueVerdict:795is the string twin, no numeric target). - VERIFIED — the three verdicts (1023 → Matches derived 1023; 1024 → Matches; 512 on 4 GB → stale derived 1024) — walked through
:932→:922→:777. Cannot forgive falsely: 1023 MB pins the raw reading to 4092-4095 MB, so a forgiven value is a legitimate value for this host. - VERIFIED — other settings and an unreadable current value untouched —
:917; the other three memory keys do not need it because v8 re-states them from the quantized reading (:1572-1574). - VERIFIED — the startup log, MCP tool and CLI table print the profile's own derived figure —
DarlingWorker.cs:2580,Mcp/DarlingMcpStoreHostTools.cs:171,:1186/1270/1303/1315. Nothing uses the derived figure to WRITE a conf value, so this changes reporting only. - VERIFIED — the revert check's shape, deducible from the pure functions: a no-op body fails case 1 only; cases 2 and 3 land identically either way; the source pin still finds the call sites, which the claim correctly does not list.
- UNVERIFIED — 78 passed; 1146 / 0 failed; 1584
Stage=Guard— need a test run, no .NET SDK here. I did read that no existing test asserts the old stale reading, andManagedConfVerdictsLiveTests:167-173' 256 MB plant is unaffected (thepg_file_settingsarm at:1512-1520overrides outright). - UNVERIFIED — build: 0 warnings, 0 errors.
CSharpSourceWalkerandSerialLoopMemberBodyboth live inDarling.Tests, already called from siblings, so the references resolve on inspection. - UNVERIFIED — "Closes #5716": the issue text is not readable from the tree, so the 4092 MB host class is taken from the diff's own arithmetic rather than the report.
- REFUTED —
ComputeAndStoreManagedConfVerdictsAsyncis "the startup profile … andDarlingSelfAlertEvaluator" — the startup profile is the OTHER producer (GatherStartupProfileAsync→GatherSettingProfilesAsyncat:1105; warning logged atDarlingWorker.cs:2575-2583), and no self-alert consumes a stale verdict (DarlingWorker.cs:9463-9466filters toRejectedValue;DarlingSelfAlertEvaluator.cs:5450-5452says stale does NOT fire). Same two errors in the test doc comment atDarlingStoreHostProfileTests.cs:406-407. - REFUTED — the changelog's "and the self-alert flagged it on every start" — same two citations.
Riskiest lines
Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs:922— the comparison deciding whether a mismatch is forgiven — pinned bySharedBuffersVerdict_RawReadingGave1023_QuantizedGives1024_1023InForceMatchesandSharedBuffersVerdict_FourGbHostWith512FromATwoGbHost_IsStaleDarling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs:1508— the call site feeding the persisted rows the mcp/viewer roles cannot recompute — pinned bySharedBuffersTolerance_IsAppliedBeforeClassifyVerdictByBothVerdictProducersDarling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs:917— the name/null guard; a slip forgives every setting on every host, in the Matches direction — pinned bySharedBuffersTolerance_LeavesEveryOtherSettingAndAnUnreadableValueAlone
Traps near this diff
- "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply —
:922callsDeriveMemorySettingsand the tests derive expectations the same way (DarlingStoreHostProfileTests.cs:352-353). - "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding, not a nit" — does not apply —
grep -rln "HostSettingVerdict\|shared_buffers" Lite/ Lite.Tests/returns nothing; this subsystem is Darling-only, behind[SupportedOSPlatform("windows")]. - "A pin that encodes the lie … Ask what a pin would fail ON." (#3699) — does not apply — the behavioural tests re-compose the pipeline at
:349-360and would pass without the call sites, which is why the source pin exists; together, dropping the tolerance from either producer fails a test. - "An expression-bodied member holding a
{ }property pattern is read SHORT by the T-SQL convention guard's member walk (#3667; the #3607 precedent)" — does not apply — the new member is block-bodied and so are both members the source scan walks; a short read returns -1 and failstolerance >= 0loudly rather than passing vacuously.
Divergence (blind read vs PR body)
Two places, one sentence, and the code is not where they disagree. The body inverts the producers (the startup profile is GatherSettingProfilesAsync, :1105), and both the body and the changelog list the self-alert among the surfaces that flagged this — it never did, by deliberate design (DarlingSelfAlertEvaluator.cs:5450-5452). Nothing in the diff goes unmentioned by the body.
Non-blocking, raised by neither account (inline at :922): the tolerance is exact equality against the raw reading as read right now, where #5459's sibling chose a band. On the marker-gated postgresql.conf path v5 was written once in the cluster's life (DarlingManagedPostgres.cs:3864), and QuantizeRam:1378-1382 says the reading drifts on balloon guests "with no operator resize — and these ARE cloud VMs". A 4 MB drift moves ram / 4 by 1 MB and the row reads stale again, below 4 GB only — #5716's own host class.
Reviewer checklist
- Run the reported suites and the 0-warning build; none is checkable from a read-only tree without an SDK.
- On a real ~4 GB managed store:
--check-settingsshould showshared_buffersasmatcheswith the derived column at the value in force, and the startup profile log should stop warning on it. - Confirm the persisted
collect.managed_conf_verdictsrow is rewritten tomatcheson the next service start — the mcp/viewer roles read that table and cannot recompute it. - Decide whether the tolerance should become a band; that needs the fleet's real
ullTotalPhysstability on sub-4 GB hosts.
…and, fix producer comment (#5716)
Follow-up review of commit 1fd53d0 (band widening + producer-comment fix)This commit does exactly what round 1 asked for and nothing more. Prose fix. The changelog Band widening. I worked through the arithmetic by hand (no SDK available to run the suite in this environment):
No new bugs found. Full verdict below. |
There was a problem hiding this comment.
Verdict: LGTM
Claims
- VERIFIED — the changelog entry no longer credits the self-alert, listing only the startup profile log,
--check-settings,get_store_hostand the stored settings verdicts — currentgh pr view 5720body'sENTRY:line; cross-checked againstDarlingWorker.cs'sRejectedValue-only alert filter andDarlingSelfAlertEvaluator's documentedStaleAfterHardwareChange-never-fires contract (established last round, unchanged this round). - VERIFIED — "For a 4 GB reading the band is 896 to 1024 MB" —
DarlingStoreHostProfileTests.cs'sSharedBuffersVerdict_BucketEdges_AreInsideTheBandAndOneBytePastIsNotassertslowestMb == 896,highestMb == 1024. - VERIFIED — the bucket rule lives in one place and both ends go through
DeriveMemorySettings, no formula re-implemented —DarlingManagedPostgres.cs:1401(QuantizedRamBucketcallsQuantizeRamdirectly) andDarlingStoreHostProfile.cs:927-931(both ends throughDeriveMemorySettings, then a plain range check). - VERIFIED — a real resize (2 GB or 3 GB host moved to 4 GB) still reads stale —
SharedBuffersVerdict_FourGbHostWith512FromATwoGbHost_IsStaleand the newSharedBuffersVerdict_FourGbHostWith768FromAThreeGbHost_IsStale. - VERIFIED — every other setting and an unreadable current value are untouched — the name/null guard at
DarlingStoreHostProfile.cs:921-924is unchanged by this commit; only the body after it was widened from exact equality to the band. - UNVERIFIED — the reported run counts (81 passed, 560 Census/Ratchet 0 failed, 1584
Stage=Guard0 failed, 0-error/1-warning build) and the revert check (reverting the band makes the drift test fail 1023≠1022, "the other 80 pass either way") — no .NET SDK/build available in this read-only environment; I verified the drift scenario's arithmetic by hand instead (4088 MB raw derives to 1022 MB, but 4088 MB's bucket is[3584 MB, 4607 MB]whoseDeriveMemorySettingsends are 896/1024, so 1023 falls inside the band under the new code), which is consistent with the claim but not a substitute for actually running the suite.
Riskiest lines
Darling/PerformanceMonitor.Darling.Service/DarlingStoreHostProfile.cs:931— the widened range check deciding whether a mismatch is forgiven — pinned bySharedBuffersVerdict_BucketEdges_AreInsideTheBandAndOneBytePastIsNotDarling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs:1403-1406—QuantizedRamBucket's half-GB offset and theMath.Max(..., 1)clamp at the bottom of the RAM scale — pinned by the same bucket-edges test (asserts both ends and the one-byte-past-either-end flip)Darling/PerformanceMonitor.Darling.Service/DarlingManagedPostgres.cs:1126—Math.Min(ram / 4, oneGb), the cap the band's monotonicity assumption rests on (both bucket ends must derive through the same non-decreasing function for the range check to be sound) — pinned by the bucket-edges test's highest-end assertion (1024, capped)
Traps near this diff
- "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — does not apply — the new tests assert
QuantizedRamBucket/DeriveMemorySettingsoutput against independently-written golden arithmetic (4 * gb - gb / 2, etc.), not against a second production copy of the formula, andDarlingStoreHostProfile.cs:927is the only production consumer ofQuantizedRamBucket. - "Lite and Darling ship in parity (CONTRIBUTING's two-store rule): a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding, not a nit" — does not apply —
shared_bufferstolerance andHostSettingVerdictare managed-PostgreSQL concepts; Lite has no managed PostgreSQL store to size (grep -rln "HostSettingVerdict\|QuantizedRamBucket" Lite/ Lite.Tests/returns nothing). - "An expression-bodied member holding a
{ }property pattern is read SHORT by the T-SQL convention guard's member walk (#3667)" — does not apply —QuantizedRamBucket(DarlingManagedPostgres.cs:1401) is block-bodied.
Divergence (blind read vs PR body)
None — the body's account matches the diff on every claim graded above. Both issues round 1 raised (the mislabeled producer/self-alert prose, and the exact-match-vs-band request) are resolved exactly as the body describes, and the new tests (drift, 3 GB→4 GB resize, bucket edges) pin the behavior the body claims.
Reviewer checklist
- Run
DarlingStoreHostProfileTestsplus the*Census*/*Ratchet*andStage=Guardsuites and confirm the counts in the test plan (81 passed; 560 total, 0 failed, 1 skipped; 1584 total, 0 failed) and the 0-error/1-pre-existing-warning build. - On a real ~4 GB managed store with a raw reading that has drifted a few MB since its
shared_bufferswas written: confirm--check-settingsandget_store_hostshowmatchesrather than flipping back to stale.
Addressed in 1fd53d0; the newest automated verdict on that head is LGTM.
Claude posting for Erik Darling.
Closes #5716.
Why
The v5 block writes
shared_buffersonce, from the raw RAM reading (min of 25% of RAM and 1 GB), and the v8 block never re-states it. The host profile derives its target from the quantized reading, so a 4 GB host that the OS reports as about 4092 MB has 1023 MB in force against a 1024 MB target and readsstale_after_hardware_changeon every start.What changes
DarlingStoreHostProfile.ApplySharedBuffersRawReadingTolerance(same shape asApplyWalDriftTolerance, but a band, as the review asked): forshared_buffersonly, the value in force reads Matches when some raw reading that quantizes to the same GB as the current reading derives it. That is the definition of "no hardware change" the v8 fingerprint already uses (same quantized RAM). In practice: the value lies betweenDeriveMemorySettingsof the bucket's lowest and highest raw readings, inclusive. When it is forgiven, the derived figure reported is the value in force. Otherwise the quantized target stands.DarlingManagedPostgres.QuantizedRamBucket, next toQuantizeRam, returns the lowest and highest raw reading (half a GB either side of the quantized value, inclusive). The tolerance computes both ends throughDeriveMemorySettings; no formula is re-implemented.QuantizeRam's notes). A host whose v5 block wrote 1023 MB from 4092 MB and now reads 4088 MB (derivation 1022) is the same host and now reads Matches. For a 4 GB reading the band is 896 to 1024 MB.It is applied in both places that build verdicts, right before
ClassifyVerdict:GatherSettingProfilesAsync: builds the startup profile (GatherStartupProfileAsynccalls it, andDarlingWorkerlogs one warning per stale row from that result), and serves the first table of--check-settingsandget_store_host.ComputeAndStoreManagedConfVerdictsAsync: writes the storedcollect.managed_conf_verdictsrows. The viewer and MCP roles read those rows and cannot recompute them, and the second table of--check-settingsreads them.No self-alert consumes a stale verdict: the alert read keeps only
RejectedValuerows, andApplyStoreSettingsAsyncdocuments thatStaleAfterHardwareChangedoes not fire it. So this change affects reporting only, never an alert.The startup log line, the MCP tool and the CLI table print the profile's own derived figure, so they show the one the verdict matched (never "1023 in force, 1024 derived, Matches").
Verdicts now, on a 4 GB host (quantized target 1024 MB):
Every other setting is untouched: the method returns the derived value unchanged for any name other than
shared_buffers, and for an unreadable current value.Test plan
DarlingStoreHostProfileTests: the original three cases and the "other settings and unreadable value untouched" check, plus the review round's drift case (4088 MB, 1023 in force), the 3 GB to 4 GB case (768 stale), and the bucket edges (the 3.5 GB and 4.5 GB minus one byte readings match their own derivation; one byte past either end is a neighbouring bucket and the figure is stale). The source pin that both verdict producers call the tolerance beforeClassifyVerdictis kept.SharedBuffersVerdict_RawReadingDriftedWithNoResize_1023InForceStillMatchesfails (1023 against 1022); the other 80 pass either way. Restored afterwards. (Round 1's check, a no-op body, failed the 4092 MB case.)DarlingStoreHostProfileTests: 81 passed.*Census*and*Ratchet*: 560 total, 0 failed, 1 skipped.-trait Stage=Guard: 1584 total, 0 failed.DarlingOperatorGroupScriptTests.cs, a file this PR does not touch).Installer.Testsnot run on this laptop.Review round 1
ComputeAndStoreManagedConfVerdictsAsyncfeeds the startup profile and that the self-alert flagged this. Fixed in this body, the changelog entry below, and the test doc comment on the source pin (commit 1fd53d0). Each producer-to-surface claim above was checked against the code:GatherStartupProfileAsynccallsGatherSettingProfilesAsync(DarlingStoreHostProfile.cs), andDarlingWorkercallsGatherStartupProfileAsyncfor the startup log andComputeAndStoreManagedConfVerdictsAsyncfor the stored rows.QuantizedRamBucketas the one home of the bucket rule (commit 1fd53d0).Changelog
SECTION: Fixed
ENTRY: - A store host with about 4 GB of RAM no longer reports
shared_buffersas stale when its hardware has not changed ([#5720]) - The setting is written once from the raw RAM reading (1023 MB on a host the OS reports as about 4092 MB), but the host profile compared it with a target derived from the rounded reading (1024 MB), so the startup profile log,--check-settings,get_store_hostand the stored settings verdicts showed it as stale on every start. A value that any raw reading rounding to the same whole GB would have produced now reads Matches, so a reading that drifts without a resize does not bring the warning back, and the derived figure shown is the value in force. A real resize, such as 512 MB or 768 MB left over from a 2 GB or 3 GB host on a 4 GB host, still reads stale.REF:
[#5720]: #5720