Emit dimension-batch digests in a deterministic order (#1801)

[erikdarlingdata]

Closes #1801.

A production field instance logged nine self-healing 40P01s in its first night on the v38 dedup write path, and zero before it. Its own pg.log captured two full lock graphs: two sessions on the same statement shape, each waiting on a tuple the other held in collect.query_text_dim.

Mechanism

The dim upsert binds a batch through unnest, and the arrays came out of a Dictionary in enumeration order — per-session arrival order, which is not a total order across sessions. Two collection cycles whose batches share two digests in opposite relative order each take the row lock the other needs next, and the cycle closes.

The within-batch digest dedup does not help here, and was never meant to: it makes a batch conflict-free with itself (the SQLSTATE 21000 guard), not ordered against its siblings. Two different properties that happen to concern the same key — which is why the dedup being correct and the batch still deadlocking are not in tension.

Mechanism chosen: client-side sort in ToArrays

One deterministic total order, imposed at the one point every caller shares — so query_plan_dim is fixed by the same edit rather than waiting for its name to appear in a log.

Client-side rather than ORDER BY u.digest in the statement, for two reasons:

• The lock order becomes a property of the values bound, owing nothing to how the planner chooses to execute the insert. An ORDER BY feeding INSERT … ON CONFLICT is a strong hint about row processing order, not a guarantee about lock acquisition — and this is a correctness property, not a performance one.
• It stays checkable without a database, which matters because the property is invisible to behavioural tests (see below).

The statement's doc now records the requirement and points at where it is imposed, so the SQL is still self-documenting about what it depends on.

Ordinal sort on the hex key is byte-wise digest order: every digest is the same length, and hex digits ascend in ASCII in the same sequence as the nibbles they encode. The test asserts the raw bytes anyway, so the claim does not rest on the representation the implementation happens to key by.

The hour-throttled conditional DO UPDATE is untouched.

collect.module_map — checked, deliberately unchanged

DarlingModuleMap.RefreshSql carries ORDER BY server_name, sql_handle, collection_time DESC because its DISTINCT ON requires it — and that is already a deterministic ascending order on its conflict key, so concurrent refreshes take the locks in the same relative order and this cycle cannot form there. Added as a comment cross-reference only, because an ORDER BY that reads as purely about picking the latest row is exactly the kind a later edit removes as redundant.

Pins — the ordering is invisible to behavioural tests

An unordered batch upserts byte-identical data; only the lock-acquisition order differs. Every round-trip, dedup and live test passes either way. So it is pinned twice:

1. Behavioural — the batch is fed in deliberately reversed digest order (the arrangement a dictionary would faithfully hand back) and the output must come out ascending, byte-wise, with index alignment intact.
2. Source — the sort must be present in ToArrays at all, and entries.Values must be gone.

Mutation	Result
Restore entries.Values enumeration (build verified clean first)	RED, both pins — source: Assert.Contains() Failure: Not found: "OrderBy"; behavioural: Assert.Equal() Failure: Collections differ at index 0 / Expected: "04A9E731…" / Actual: "D2F45524…"

The build was confirmed to succeed before reading the mutation's verdict — a mutation that fails to compile runs the previous binary and is indistinguishable from a working guard.

No live deadlock repro is claimed. Reproducing a two-session lock cycle on demand is timing-dependent, and a test that passes because the race did not happen is worse than no test. The property under guard is the total order, which is exactly what these two pin.

Verification

• PerformanceMonitor.sln -t:Rebuild -c Debug: 0 Warning(s) 0 Error(s).
• Darling.Tests against a live PostgreSQL 18.4 + TimescaleDB 2.28.1 rig (SearchPath=collect,config), so the gated legs ran rather than skipped: 3,742 passed / 0 failed / 8 skipped.
• Lite is unaffected — the diff touches only Darling/ and no shared collector code.
• CHANGELOG diff is additions-only.

Generated with Claude Code