[rbuergi]

Fixes #971.

Verdict up front

On a fresh replica's first write to a path, "the store never moves backward" was enforced by nothing at all. Not by the in-process guard — its high-water table is empty on a new process, so the write took the unverified fast path. Not by the database — the Postgres upsert carried no version condition, and the in-memory store was a plain _nodes[path] = node. #968 closed the in-process window; this closes the cross-process one it explicitly recorded as residual.

The failure is silent by construction: the stale write is acked, and the next write is minted one above the rolled-back row and accepted for a perfectly good reason — so the wrong lineage persists with no error anywhere.

The fix, in two halves

1. The invariant moves into the store

MeshNode.Version is the node's forward-only revision counter, so a correctly-produced write is always at or above what is stored. Every backend that can express that now refuses a strictly-regressing write at the row and reports it by emitting the STORED node instead of the one it was handed:

backend	condition
Postgres	INSERT … AS target … ON CONFLICT (namespace, id) DO UPDATE SET … WHERE target.version <= EXCLUDED.version — row count 0 ⇒ refused
in-memory	one AddOrUpdate keeping the higher Version — atomic against every concurrent writer, no read-then-write window
Snowflake	WHEN MATCHED AND t."version" <= s."version" THEN UPDATE SET …; the no-MERGE emulator fallback gets a conditional DELETE … AND "version" <= :version plus a WHERE NOT EXISTS guard on the INSERT

Equal versions still apply — re-persisting an unchanged node is a legitimate, common shape. A refused write publishes no change notification (nothing changed, and a notification carrying the loser would hand every subscriber the state the store just rejected), and on Snowflake none of the trigger-replacement steps run either, since they all derive from a row that was not written. WriteMany reads per-command NpgsqlBatchCommand.Rows, so a refused node in a batch is neither announced nor reported as committed, and the caller gets durable truth back in its original order.

The contract is written into IStorageAdapter.Write's doc, including the explicit licence for backends that cannot express it to behave as before.

2. The conflict resolves by MERGING, never by refusing

Per the maintainer's policy on the issue. Callers of stream.Update do not write conflict-retry loops and must not have to, so MonotonicWriteGuardStorageAdapter now funnels both conflict detectors — its own verification read, and a store that refused the write — into one resolution: re-read, merge the loser into durable truth, write once. Deliberately one attempt, never a retry loop: if the row moved again during the merge, the second refusal is logged and the newer row is kept.

The merged node carries the latest row's version, specifically so re-persisting it cannot itself read as a rollback. That also preserves the saved.Version > written.Version signal RefusedWriteOwnerRebaseTest pins, so the owner-rebase path is unchanged.

How each policy rule maps onto the shared merge

You asked me not to write a second merge. I did not — but one rule genuinely could not be expressed through the existing entry point, and I want to be plain about which.

MeshNodePatchMerge.Apply (three-way) cannot serve this at all. Its whole classification turns on live == base, and handed baseValues: null it takes the "no conflict signal" branch and applies the patch wholesale — i.e. the stale snapshot overwrites the newer row, which is the defect. There is no argument shape that makes it behave two-way. So the addition is a second entry point on the same class (ApplyTwoWay), sharing the recursion, StringDelta, and MergeArray, rather than a second merge implementation that could drift. MeshNodeConflictMerge on top of it owns only the MeshNode member mapping (which members are user data, which are the store's own bookkeeping, and how Content is projected to JSON and materialised back into its CLR type — the node as a whole is never serialized, because HubConfiguration/GlobalServiceConfigurations are delegates).

rule	how it lands	shared with the three-way merge?
non-strings → latest wins	every non-string, non-object, non-array leaf keeps live and reports the member	identical — Apply already refuses conflicting scalars and keeps live
strings → try to merge	StringDelta.Compute(latest, stale) reduces the pair to one splice; a pure insertion means stale ⊃ latest (take the superset), a pure removal means latest ⊃ stale (keep latest); a splice that both removes and inserts is a genuine divergence ⇒ not auto-resolvable	new rule, shared primitive. Apply's string merge rebases the writer's splice over the intervening edit — that needs a base. With none, only the superset case is decidable.
arrays → element identity, not RFC 7396 replace	calls the same MergeArray with an empty base, which is exactly "neither side can be shown to have removed anything, so keep every element" — union by whole-element value equality	same code path
unresolvable → latest wins + a warning in the activity log	onOverwritten → OverwrittenMembers → a LogWarning and a durable ActivityLog MeshNode	new — Apply's onRefuse has no durable sink today

Which MeshNode members take the string merge: only Name and Description — the prose. NodeType, Category, Icon and DesiredId are wire identifiers, not text; the superset rule would happily splice "Code" and "CodeCell" into one of them and silently retype the node, which drives routing and per-node hub activation. PreRenderedHtml is derived from Content, so it follows the merged content rather than being spliced into broken markup. All five take latest-wins-and-report, so the policy's fallback still covers them and nothing is dropped silently. The user data that actually benefits from merging lives in Content, which is merged leaf by leaf.

Two ambiguities the base-less merge cannot resolve, decided deliberately and documented at the call site rather than hidden:

• A superset string, and a union array, can resurrect content the newer writer deleted. Without a common ancestor, "the stale writer inserted this" and "the newer writer deleted it" are literally the same observation. The alternative silently destroys content an acked write added, which is the failure mode of record — so the policy resolves toward keeping content. Resurrection is visible in the content and reported; destruction is not.
• A key absent from the latest object is never adopted from the stale one. The serializer suppresses defaults, so absence is ambiguous between "removed" and "is the default" — and under both readings the newer state is the one to keep. Adopting there would let an older snapshot re-assert a field the newer write had just defaulted.

The warning is the load-bearing half

Latest-wins is acceptable; latest-wins invisibly is the #968 defect — an acked write that rolled a row back with no error anywhere. So every member the merge could not resolve produces both:

• a structured LogWarning naming the path, both versions, and the merged / dropped member lists; and
• a durable ActivityLog MeshNode at {path}/_Activity/write-conflict-{latestVersion}, category WriteConflict, ActivityStatus.Warning, with one LogLevel.Warning message per dropped member, naming it.

The id is keyed on the durable version alone, never the loser's. That is the right granularity ("this revision had a conflict") and it is what bounds the litter: a wedged owner mints an increasing version on every retry — the #725/#872 fork shape, 834, 835, 836 … against a stuck 2423 — so keying on the losing version would accumulate one satellite per attempt, forever. Re-recording overwrites the same node at Version = 1, which the store's equal-version rule accepts, so the record can never conflict with itself. It is best-effort by contract (a failed trace must not fail the resolution it describes) and skipped for paths already inside an _Activity namespace, so a conflict on a trace can never write a trace about a trace.

Snowflake

Fixed, not deferred — but unverified against a live endpoint. SnowflakePartitionStorageProvider has no write path of its own (grep for INSERT/MERGE/UPDATE/Write( in it returns nothing): it exposes SnowflakePathRoutingAdapter, whose Write delegates straight to the per-schema SnowflakeStorageAdapter.Write. That is the path now made version-conditional, so the provider's gap closes with it.

Its other per-process dictionaries (_registeredPartitions, _schemasInitialized, _provisioned) are not the same defect and are deliberately untouched: they are idempotent provisioning promise-caches, not an authority on durable state. A second replica starting with an empty _provisioned just re-runs an idempotent CREATE SCHEMA IF NOT EXISTS; nothing is lost.

I could not execute this against Snowflake or its emulator — that backend has no CI coverage here. The change is compile-clean and shape-identical to the Postgres one, and that is the whole of my evidence for it.

Not covered, and why

FileSystem, Caching, Sqlite, Cosmos and AzureBlob adapters keep their previous behaviour, which the interface doc now explicitly sanctions. FileSystem/Caching/Sqlite are single-process dev and test hosts — there is no second replica, and a file-level read-then-write "CAS" would be racy anyway. Cosmos has ETag optimistic concurrency but no version <= predicate: it would need a read plus IfMatch on every write, i.e. an extra round-trip on the hot path, against a backend that is emulator-unverified with a known contract gap. In every one of these the in-process guard remains the protection it always was, which is sound precisely because they are not multi-replica.

Verification

Fail-without, run against current main (the two behaviour-bearing files reverted to origin/main, rebuilt, re-run) — both new tests fail, and the output names the defect exactly:

• AWriterWithNoHighWaterMark_CannotRollTheDurableRowBack → [replica write] emitted=v1/stale-snapshot, Expected 1 to be greater than or equal to 7000. On main the fresh replica's write landed: the v1 snapshot went straight over the v7000 row. That is the issue's silent rollback, reproduced.
• AConflictIsMerged_NotRefused_AndTheDroppedValueReachesTheActivityLog → [durable] v9000 name=durable-advance description=alpha beta, Expected "alpha beta gamma" … but found "alpha beta". On main the conflict was refused, so the losing write's text was destroyed rather than merged, and no activity-log record existed.

Pass-with: 2/2. The property is constructed, not raced — "a second writer with no prior mark" is a fresh MonotonicWriteGuardStorageAdapter instance over the same store, which is exactly a newly started replica's state. No load, no sleeps, no timing budget.

Regression, biased toward everything that writes an explicit MeshNode.Version (found by grepping the test tree for Version = alongside a .Write():

scope	result
MeshWeaver.Hosting.Test (whole project)	129 / 129
write-integrity pins: MonotonicWriteGuardWindowTest, RefusedWriteOwnerRebaseTest, StaleActivationSeedRollbackTest, StaleActivationDurableFirstSeedTest + the 2 new	8 / 8
Monolith `Version	Storage
PartitionRootBootstrapTest, OwnerlessPartitionRepairTest	12 / 12
GhostPartitionRootRecoveryTest, BulkSaveInstallTest	6 / 6

One existing test needed a setup change, and it is worth reading rather than skimming. MonotonicWriteGuardTests.StaleHighWaterMark_DoesNotRefuse_WhenTheStoreMovedOutOfBand simulated "another replica deleted + recreated the node" by writing v1 straight over v50 on the leaf, with no delete. The leaf is now version-conditional, so the store refuses that in-place rewind — which is precisely the behaviour this PR exists to add, so the setup, not the guard, was what broke. It now does what its own comment says: Delete then recreate at v1, which is how a legitimate rewind has always been expressed (the RecreateAfterDelete* cases beside it, and the guard's "Legitimate rewinds" note — the row is gone, so the recreate faces nothing to regress against). The property it pins is untouched and still fails if the guard ever trusts its mark over a verification read. An in-place rewind through IStorageAdapter is no longer possible on a conditional backend — that is a deliberate, load-bearing consequence, not a side effect: an out-of-band restore is done with database tooling, not through this interface.

The Postgres SQL was executed against real Postgres, not merely compiled — the aliased-target + conditional-ON CONFLICT shape on a scratch temp table: create INSERT 0 1, advance to 7000 INSERT 0 1, regressing write at v1 INSERT 0 0 with the row untouched, equal-version rewrite INSERT 0 1. That zero row count is exactly what WriteAsyncCore now reads to report the refusal.

dotnet build -c Release -warnaserror clean on MeshWeaver.Hosting, MeshWeaver.Hosting.PostgreSql, MeshWeaver.Hosting.Snowflake and MeshWeaver.Hosting.Monolith.Test, after merging current main.

What I could not verify locally: the Postgres and Snowflake test suites. The PG suite needs its container fixture and this machine was already carrying heavy contention; the Snowflake one has no CI coverage at all. The SQL probe above is the strongest evidence I could get for the PG statement short of that suite — CI is the check for the rest, and the PG change touches every write, so it is the thing to watch on this PR.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR closes the cross-process gap in monotonic mesh-node persistence by moving the “version never regresses” invariant into the storage backends (where possible) and teaching the write-integrity guard to resolve refused/regressing writes by merging into durable truth and recording any dropped fields.

Changes:

• Add version-conditional upserts to Postgres, Snowflake, and the in-memory adapter; refused writes return the stored (winning) row and do not emit change notifications.
• Introduce a store-refusal conflict resolution path in MonotonicWriteGuardStorageAdapter that merges the losing write into the latest durable row and writes an ActivityLog satellite when values are dropped.
• Add deterministic tests pinning store-level refusal + merge behavior and a “What’s New” entry documenting the user-visible outcome.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/MeshWeaver.Hosting.Test/MonotonicWriteGuardTests.cs	Updates test commentary to reflect the new store-level version gate and the “delete + recreate” legitimate rewind shape.
test/MeshWeaver.Hosting.Monolith.Test/StoreLevelWriteConflictTest.cs	Adds monolith integration tests pinning store-level refusal and merge-with-activity-log behavior.
src/MeshWeaver.Mesh.Contract/Services/IStorageAdapter.cs	Documents the new conditional-write contract and “refused write emits stored row / publishes no Changes” behavior.
src/MeshWeaver.Mesh.Contract/MeshNodeConflictMerge.cs	Adds MeshNode-level conflict merge mapping over the new two-way JSON merge policy.
src/MeshWeaver.Hosting/Persistence/MonotonicWriteGuardStorageAdapter.cs	Routes both verification-based and store-refusal conflicts through a single merge path and writes ActivityLog satellites for dropped members.
src/MeshWeaver.Hosting/Persistence/InMemoryStorageAdapter.cs	Implements version-conditional “upsert” semantics via ConcurrentDictionary.AddOrUpdate and avoids publishing changes on refused writes.
src/MeshWeaver.Hosting.Snowflake/SnowflakeStorageAdapter.cs	Adds version-conditional upsert semantics (MERGE and emulator fallback) and refusal reporting.
src/MeshWeaver.Hosting.PostgreSql/PostgreSqlStorageAdapter.cs	Adds a version predicate to the upsert and plumbs refusal detection through Write and WriteMany.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-09-write-conflicts-merge-not-vanish.md	Documents the new end-user behavior: stale saves no longer silently overwrite newer content; dropped fields are recorded.
src/MeshWeaver.Data/Serialization/MeshNodePatchMerge.cs	Adds ApplyTwoWay + TryMergeTwoWay for base-less conflict resolution (store-refusal merge policy).
src/MeshWeaver.Data.Contract/ActivityCategory.cs	Introduces WriteConflict activity category for durable conflict-trace records.

Suppressed comments (1)

src/MeshWeaver.Mesh.Contract/MeshNodeConflictMerge.cs:196

• JsonNode.Deserialize(type, options) can throw NotSupportedException as well as JsonException. Catching only JsonException means conflict resolution can still throw for unsupported content types; it should instead report Content as overwritten and keep latest.

        catch (JsonException)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rbuergi]

Copilot's review caught a real one, now fixed in 7750ac8.

JsonSerializer.SerializeToNode / Deserialize throw NotSupportedException when no compatible converter exists for the type or one of its members — not only JsonException. The merge caught only the latter, so content this serializer cannot project would have escaped conflict resolution and faulted the whole write chain — precisely what the guard promises never to do, and worse than the refusal it replaced.

Both documented serialization failures now report Content as an overwritten member (so it lands in the activity-log entry and the warning) and keep the newer row.

Deliberately not widened to a blanket catch: an arbitrary exception from a custom converter is a genuine fault, not an unmergeable input, and it must still propagate. Swallowing it would hide a defect behind a plausible-looking latest-wins — the exact shape this change exists to eliminate. A loud failure is acceptable; a silent wrong one is not.

Write-integrity pins re-run after the change: 8/8.

[github-actions]

Test Results (shard 0)

685 tests   685 ✅  5m 15s ⏱️
 10 suites    0 💤
 10 files      0 ❌

Results for commit 7750ac8.

[github-actions]

Test Results (shard 3)

   11 files     11 suites   5m 6s ⏱️
1 882 tests 1 776 ✅ 106 💤 0 ❌
2 007 runs  1 901 ✅ 106 💤 0 ❌

Results for commit 7750ac8.

[github-actions]

Test Results (shard 4)

1 219 tests   1 212 ✅  6m 6s ⏱️
   11 suites      7 💤
   11 files        0 ❌

Results for commit 7750ac8.

[github-actions]

Test Results (shard 5)

1 713 tests   1 522 ✅  6m 8s ⏱️
   11 suites    191 💤
   11 files        0 ❌

Results for commit 7750ac8.

[github-actions]

Test Results (shard 2)

1 885 tests   1 883 ✅  6m 32s ⏱️
   11 suites      2 💤
   11 files        0 ❌

Results for commit 7750ac8.

[github-actions]

Test Results (shard 1)

1 188 tests   1 188 ✅  7m 22s ⏱️
   11 suites      0 💤
   11 files        0 ❌

Results for commit 7750ac8.

[github-actions]

Test Results

   65 files     65 suites   36m 32s ⏱️
8 572 tests 8 266 ✅ 306 💤 0 ❌
8 697 runs  8 391 ✅ 306 💤 0 ❌

Results for commit 7750ac8.