Release candidate 10.0.21-rc.1 by branarakic-agent · Pull Request #2984 · OriginTrail/dkg

branarakic-agent · 2026-10-03T15:07:41Z

10.0.21 release candidate

This branch is testnet-canary at a58fbf36f plus the [10.0.21] changelog section and the version bump. Every code change in it is already on testnet-canary and was reviewed in its own PR. It is meant to merge last: after it, testnet-canary calls itself 10.0.21, and anything merged later needs its own changelog edit.

Open before it merges:

The devnet and testnet gates have not run on this head (see "Validation"). The full cold recovery passed on this head's code; a second run, on the pushed head 41cfe62bf itself, passed as well.

What 10.0.21 contains

72 PRs merged into testnet-canary since v10.0.20 (abfd785d3), 19 of them on 4 October.

Area	PR	Change
Public-graph recovery	#2939	Bounded whole-asset exports, compressed exact responses and an opt-in duplex batch stream for recovering the Verifiable Memory of a registered public graph. Every asset is still verified, authenticated and stored atomically before it is acknowledged.
	#2946	Recovery no longer reads the block header of each publish receipt, which it never used.
	#2947, #2954	Opt-in preparation of the next recovery batch (advance sizing, one registered-public read per pass, a stable wire choice), and per-phase timing lines for every recovery batch and pass.
	#2956	The two switches get their lasting names: `DKG_EXACT_BATCH_STREAM_ENABLED` (the first name is still read) and `DKG_VM_RECOVERY_PREFETCH_ENABLED`.
	#2979	A stream that breaks is opened once more with the same peer for the assets not yet stored; every exchange that stops early logs why.
	#2978	While a stream-capable Core is connected, an attempt on a peer without the stream is bounded to two minutes.
	#2987	A serving Core that cannot admit a stream request answers `BUSY`; the requester pauses and asks the same Core again, and a busy or broken attempt no longer costs the Core its turn in the peer rotation.
	#2996	Background reads of seven contract views leave together in one Multicall3 call, where the canonical Multicall3 is deployed. A full cold recovery of a 564-asset graph went from 8,509 requests and 54.5 minutes to 4,350 and 31.1. Off with `DKG_DISABLE_RPC_READ_BATCHING=1`.
	#3020	A read stops starting at an RPC endpoint that refused it with HTTP 401 or 403. With the default Base mainnet endpoints, receipt lookups are no longer sent first to the one that refuses them.
	#3021	The receipt of a first publish is requested together with the three authentication views, where it was read after them.
	#3024	The first reconcile after a cold start is asked again when its read-authority check gets no answer from the chain, where the graph waited for the periodic sweep.
Async publishing	#2941	A store rejection before the publish transaction was recorded is retried as the same job instead of being stranded as `tx_submit_timeout`.
	#2944	A transient RPC failure while the transaction is prepared is retried the same way; `retryState.blocker` says why a job is not moving.
	#2948	Configured RPC URLs no longer reach error messages or persisted failures on the write path.
	#2950, #2617	The failure decision is one pure function, and its precedence an ordered rule table. No behaviour change.
	#2952	A job that waits for a chain re-check reports the latest outcome (`blocker.lastCheck`).
	#2961	A draft of a published asset is numbered from its confirmed version; a version gap is refused with `409 PUBLISH_INTENT_STALE`.
	#2959	A failed `swmCurrentAssertion` stamp after the commit is retried instead of reported as success.
	#3009	A damaged working-draft record is reported as a server error (`KA_WM_LIFECYCLE_CORRUPT`) instead of the 409 that tells a caller to reopen the draft.
	#3004	A share job that waits for that repair is marked with `lastError.diagnosticCode`, cannot be cleared while it has retry budget, and a synchronous share answers a retryable 503.
	#2906, #2930	Opt-in cleanup of finalized snapshot files after a grace period, and its fix-forward.
	#2581, #2977	`wm/discard` and `wm/write` on a shared or published asset answer 409 instead of 500; #2977 adds the tests for `wm/write`.
Store pressure	#2919	The promotion audit checks 64 signed copies per query.
	#2923	The periodic sweep admits at most eight bound graphs per tick.
	#2922	A store queue timeout names up to three operations that held the slots.
	#2933	A caller that stops waiting no longer gets a healthy managed Oxigraph restarted.
	#2908	The dashboard's store-wide counts are cached for 30 s and shared by concurrent callers.
	#2994	A graph whose recovery fetch the node's own sync admission refuses waits for capacity, where its reconcile pass ran again every 2.5 s with all of its chain reads.
	#2988	Chain reconcile no longer searches every old workspace operation of a graph for each Knowledge Asset the node lacks; a missing asset is fetched from peers. On one Core this search had kept the triple store CPU-bound.
Root-entity publishes	#2991	A node ignores finalization messages that are not graph-scoped, and `publishFromSharedMemory` refuses a publish without `contentScopeVersion` (`LEGACY_KA_READ_ONLY`). About 1,150 lines of root-entity promotion code are removed.
Authority and private graphs	#2955	An approved member can subscribe to a private graph that was never registered on chain. Fixes the 10.0.20 known issue #2871.
	#2967	A catalog authority refresh discarded by #2955's facts fence is composed again instead of dropped.
	#2966	Readiness of an authorized unregistered graph no longer depends on Verifiable Memory it cannot have; a resolved registration lookup no longer waits for unrelated index work.
	#2992	A share no longer fails when the graph's own metadata changes while its authority is read; legacy shared-memory markers are retired once Verifiable Memory holds the asset; a join-approved member of an unregistered private graph takes part in the graph's catalog; a query scoped to a graph the node does not hold returns an empty result.
Sync and shared memory	#2611	An accepted sync page is decoded once instead of twice.
	#2624	Shared Working Memory materialization skips an unchanged assertion graph on the embedded Oxigraph backends.
	#2615	A failed network admission probe is warned once instead of twice.
	#3012	A serving node admits one sync request per peer for each of Shared Working Memory and Verifiable Memory; byte-budget pages for Shared Working Memory sync; metadata with both compatible confirmation labels is accepted as a transaction.
Agent lifecycle	#2934, #3005	An agent restarted in place re-subscribes gossip; the gossip manager and its wiring registries are one session that a restart replaces as a whole.
	#2935	`stop()` drains Context Graph subscription writes.
Daemon security	#2764	The dashboard embeds the node-operator token only for trusted local requests; node-wide operations require a node-admin token.
Storage internals	#2763, #2797	Adapters format SPARQL terms through one validated serializer, in observe mode, with a new counter that also covers relative and RFC 3987-invalid IRIs on every adapter write.
	#2909, #2910	Faster literal escaping and trusted JSON result decoding.
	#2926	The former `dkg-core` deep-import path of the RFC 3987 validator resolves again.
Packaging	#2927	Protocol persistence stores move to a new package, `@origintrail-official/dkg-node-store`. Same database, no migration.
Repository, CI, tests	#2924, #2957	Root tidy: files moved to `misc/` and next to what they configure.
	#2918, #2920, #2983	Trusted CI controller rotation; repository scripts routed by who runs them; the moved mutation-test configs routed to the core lane.
	#2938	Real-chain and devnet coverage for subscribing by name hash. Tests only.
	#2986	The governed HTTP pacing test no longer depends on wall-clock timing. Tests only.
	#2998, #3023	One promotion-agent harness for the two pointer recovery suites; the sender-key test accepts either send order. Tests only.
Chat memory	#3010	A legacy chat turn linked to two sessions no longer supplies its completion to both.
Documentation	#2997	The large-content guide that the literal size error links to is restored.
Refactors without a behaviour change	#2613, #2598, #2631, #2629	Name-registry scan policies per scan mode; the catch-up proof model in its own module; the author-catalog path verifier normalizes path nodes once; the async-promote worker normalizes its log sink once.
	#3000, #3001, #3002, #3016	The Shared Working Memory materialization memo in its own module; one required capacity read for the reconcile admission wait; recovery transport candidates ordered by fixed tiers; receipt decoding, recovery timing, experiment policy and preparation each in a module of its own.
	#3008	Chain reconcile's unused inline fetch mode, workspace fingerprint and negative cache are removed, about 2,600 lines.

Commits on this branch

Changelog (3139984c4, e1db8d88f, a675ec4ee, 0086a4f82, 41cfe62bf): [Unreleased] becomes the dated [10.0.21] section. It has a theme paragraph, an "Upgrading from 10.0.20" table with 24 rows, ten known issues, and Fixed, Changed and Added entries for the 62 PRs that had none. The two existing entries (fix(agent): number a draft of a published KA from its confirmed version; refuse a version gap with a typed terminal error (#2958) #2961, fix(core): keep the former dist path of the RFC 3987 IRI validator #2926) are kept as written. ci: rotate trusted CI controller to dfb3460 #2918, ci(planner): route repository scripts by who runs them #2920, test: cover subscribe-by-name-hash on a real chain and a real devnet #2938, test(cli): cover wm/write on a shared asset after the WM lifecycle 409 mapping #2977, fix(ci): route the moved mutation-pilot configs to the core job #2983, test(chain): stabilize governed HTTP pacing assertion #2986, refactor: share promotion-agent recovery test harness #2998 and test(agent): accept either send order from the sender-key fan-out #3023 have no entry (CI and tests only).
Version bump (91c5797ee): 25 package.json files move from 10.0.20 to 10.0.21, version only; the lockfile is unchanged. release-packages.mjs verify-versions --version 10.0.21 passes.
Five merges of testnet-canary (d517ff872, b61b94090, 065225d0c, cff35e1db, ca81a78c8), without conflicts. The branch differs from testnet-canary only in CHANGELOG.md and the 25 version fields.

What operators and the release need to do

Cores: the batch stream is off by default. Set DKG_EXACT_BATCH_STREAM_ENABLED=1 on the Cores that should serve recovery, on more than one Core per graph: a recovery that cannot finish an exchange falls back to the ordinary protocol on the other holders. A Core that already runs with DKG_EXPERIMENTAL_EXACT_BATCH_STREAM=1 keeps serving without a change.
API clients with agent-scoped tokens: node-wide routes now answer 403 (fix(daemon): scope the dashboard token and node-wide operations to node admins #2764). The upgrade table lists them.
SDK embedders: publishFromSharedMemory without contentScopeVersion is refused (fix(agent): retire root-entity publishes (ignore old-format finalization, refuse unscoped publish) #2991); publish through publishFromFinalizedAssertion. The daemon's publish routes are unaffected.
Core operators, after the upgrade: a Core that was held up by the old search (fix(agent): stop searching historical workspace operations in chain reconcile #2988) starts recovering what it lacks from peers. On the one Core measured, reconcile passes went from 67 to 575 an hour, chain RPC from about 4,100 to 8,200 calls an hour, the store stayed healthy with no rejected background work, and the node's own sync admission was refused about 500 times in the hour, so /api/status read saturated at times.
RPC dashboards: background contract reads on Base, Gnosis and Base Sepolia now arrive as eth_call under the consumer multicall3.aggregate3 (perf(chain): send background contract views together through Multicall3 #2996), with fewer requests under the seven read labels they replace. The usage snapshot's consumerVocabularyVersion is 3.
npm: @origintrail-official/dkg-node-store is new and public, and dkg-node-ui and dkg depend on it. The release helper discovers it; it has to be published with the set.
CI controller: ci(planner): route repository scripts by who runs them #2920, refactor(node-store): move protocol persistence stores out of node-ui (Phase 1) #2927 and fix(ci): route the moved mutation-pilot configs to the core job #2983 edit scripts/lib/ci-routing.mjs, which takes effect only after the trusted controller pin is rotated (as ci: rotate trusted CI controller to dfb3460 #2918 did) to a commit that contains them.

Not in this candidate

PR	State
#2999	Source file-size guard. Against `a58fbf36f` it conflicts in one test file and its baseline is 18 files behind; it needs a rebase and a fresh baseline (comments on the PR). For after the release.
#2921	Bounded complete Shared Working Memory reads. Left out of 10.0.21 by decision on 4 October and scheduled for 10.1.0: #2988 and #2991 removed the two readers it bounded in the agent, which gives this release most of its effect, and what is left (storage-layer bounds for the remaining complete reads) needs a rebase and a new one-Core pilot. Whether it comes back depends on store-pressure figures from Cores on 10.0.21; see the comment on the PR.

testnet-canary was red from 7b04cd785 to 62851c708: #2957 and #2920 landed together and failed the routing guard in "Build packages". #2983 fixed it. Nine PRs that the merge queue dropped during that time, and #2629, were merged directly on 3 October after the local check below.

Validation

Done:

release-packages.mjs verify-versions --version 10.0.21: 25 files.
Local check of the ten PRs merged on 3 October, as one tree. The tree tested is identical to testnet-canary 384f75258 (same git tree object). pnpm install --frozen-lockfile, build:packages (26 of 26), lint, test:scripts (435 pass, including the routing guard that had failed) and test:inventory pass. Unit suites on that tree: rdf-utils 236, core 2,261, storage 1,259 (34 skipped: live Blazegraph), publisher 1,800, chain 2,362 (4 skipped), the CLI suites these PRs touch 258, and the agent's sync, shared-memory and recovery suites 2,015 (110 files). No failure. The whole agent lane and the Hardhat-backed suites were not run locally.
Two 30-minute cold recoveries of a 564-asset registered public graph (one cold node against the live network, the stream and preparation on, the node's default settings otherwise), on earlier heads:

`testnet-canary` head	Assets after 30 minutes	Stream exchanges	Error-level log lines
`5ca8ca7e9` (after #2979)	260	28 of 28 completed	0
`7b04cd785` (after #2938, #2934, #2764, #2935, #2930, #2920, #2957)	310	34 of 34 completed	0

A full cold recovery of that graph on 6c14e9f1b (testnet-canary 8dfa2d6aa with fix(sync): a busy stream Core answers BUSY, is asked again and keeps its turn in recovery #2987, plus fix(agent): stop searching historical workspace operations in chain reconcile #2988), with the same build on the test node and on the serving Core:

Build	Result	Time	Stream exchanges	Per asset	Failures
`384f75258` on both sides	513 of 564 at the two-hour limit, not passed	7,190 s	64 of 80 completed	6.4 s	16, with three stalls of 13 to 22 minutes
`6c14e9f1b` on both sides	564 of 564, passed	3,270 s (54.5 min)	62 of 62 completed	3.5 s	none

In the passing run about 2.9 s of each asset's 3.5 s is the test node's own verified store step and about 0.4 s is waiting for the Core. No busy answer, retry or error-level line on either side. The first run's Core spent its store on the search that #2988 removes.
The final head adds #2991 and three test-only commits to that build. #2991 does not touch the recovery path, and no run includes it.
The same-Core retry of #2979 fired twelve times in the 384f75258 run; the BUSY answer of #2987 has not occurred on live data.

Full cold recoveries of that graph on 4 October, with the test node on the build named and the serving Core on a build with fix(agent): wait for sync capacity instead of re-running a refused reconcile pass #2994. Same machine, 564 assets in every run, every run passed with all stream exchanges completed:

Test node build	Time to the gate	Per asset while assets arrive	Requests	Failovers	Error-level log lines
`6c14e9f1b` (3 October, for comparison)	3,270 s	3.5 s	8,509	not counted	0
#2996	1,931 s	3.20 s	4,390	168	0
#2996, #3020	1,922 s	3.07 s	4,202	51	0
#2996, #3020, #3021	1,830 s	2.86 s	4,251	52	1 (a known start-up line)
`8097794bd` plus #2992 at `5cda7d2c4`	1,841 s	2.95 s	4,198	54	0
`0086a4f82` plus #2992, #3008, #3009, #3012, #3010 and #3024, as 10.0.21: the same git tree as this head outside the changelog	1,715 s	2.79 s	4,099	45	1 (a known start-up line)
this head, `41cfe62bf`, while a second test harness ran on the same machine	2,417 s	3.91 s	4,666	58	0

The time before the first asset arrives varied between 81 s and 139 s across these runs for a reason unrelated to them (#3024 addresses it), so the per-asset column is the one to compare. The run on 8097794bd plus #2992 was made on a busier machine; its processor-only stages were 6% slower. In the run on the candidate tree the first recovery started 76 s after the cold start: with #3024 the node asked for its read authority again eleven times in the first 66 s and did not wait for the periodic sweep. In the run on 41cfe62bf it started at 82 s.
The last two rows are the same code. The run on 41cfe62bf shared its machine with another test harness: its processor-only stages were 45% to 125% slower, while its chain-side figures per asset (authority read, batch sizing, wait for the request budget) were unchanged, so the difference is the machine. In that run the serving Core dropped the connection once in the middle of an exchange. The ten assets of that batch were already stored, the node was connected again within a second and went on; 59 of 60 exchanges completed.

Local check of the eight PRs merged at 09:48Z on 4 October, as one tree on 9fa40f9f9: build, lint, test inventory, 435 script tests, and 1,087 agent, 157 chain, 96 publisher and 31 CLI tests in the suites they touch.
Local check of the six PRs merged at 15:06Z on 4 October, as one tree on 2a7964c4d (with chore(ci): enforce the source size ratchet on current canary #2999 for its scripts): pnpm install --frozen-lockfile, build:packages (26 of 26), lint, test:scripts (453 pass) and test:inventory pass. The chain unit suite (135 files, 2,455 tests) and the publisher unit suite (100 files, 1,806 tests) pass, and so do 151 agent test files (every changed one and the reconcile, recovery and sync suites: 3,224 tests, 2 performance files skipped), the changed node-ui tests (163) and the changed CLI tests (125 in six files, and the 69 live-daemon route tests of knowledge-assets-route.test.ts; in one of five runs of that file a share right after a graph registration answered A promote prerequisite is temporarily unavailable while the machine was busy, which has also been seen in CI on an unrelated PR). The source-size guard of chore(ci): enforce the source size ratchet on current canary #2999 fails on that tree.
Post-merge CI of 2a7964c4d: CI and EVM Integration Tests passed. This PR's CI on 0086a4f82 passed.
The merged branch was compared with the tested candidate: git diff between the two is empty outside CHANGELOG.md.
The post-merge CI of a58fbf36f: EVM Integration Tests passed; in CI one CLI shard failed because its runner could not reach the download host for a test binary (a connect timeout, no test ran), and the failed jobs were re-run. This PR's CI on 41cfe62bf, the same code, passed.

Not done, and needed before the cut:

The comprehensive devnet suite on the release commit (RELEASE_PROCESS.md section 4), and the testnet soak.

Decisions for the release lead

Known issues. Four are carried over from 10.0.20 (Peers that connect while both are starting never see each other's sync support #2854, A member added to a private Context Graph only by wallet address does not receive its shared memory until it joins #2862, A restarted member Edge does not reconnect to its private graph's curator Edge #2865, VM refresh after KA updates: durability, batching and bounds follow-ups #2866) with the catalog replay loop. Private unregistered Context Graph join cannot activate after approval #2871 is removed as fixed; the issue is still open on GitHub and can be closed at the cut. Five are new: the stream's dependence on the Cores that serve it, Same-number re-share of an unpublished KA draft is not superseded by the replication lanes (follow-up of #2958) #2964, Authority decisions about one graph are discarded by unrelated store writes (node-wide authority-facts revision) #2968, Lifecycle pointer stamps still swallow store failures after commit (VM stamps after a confirmed publish, publishAsync swm stamp) #2962 and Follow-up to #2992: deferred review items #2993 (the catalog of an unregistered private graph on a member).
VM refresh after KA updates: durability, batching and bounds follow-ups #2866. The 10.0.20 notes said a rate-limited backfill was planned for 10.0.21. There is no PR for it, so the known issue now says it is not in this release.
The changelog date is 4 October and is set again at the cut.
Whether Cores enable the stream as part of this rollout. Without it the release changes nothing about recovery speed.

After the merge

The steps of RELEASE_PROCESS.md: the canary to main promotion PR, the devnet gate on the resulting main commit, the signed tag, npm (canary, then testnet, then latest and mainnet), the beacons and Cores, and the GitHub release assembled from the [10.0.21] section.

🤖 Generated with Claude Code

Turn [Unreleased] into the dated [10.0.21] section and add entries for the 33 pull requests merged since 10.0.20 that had none. The two existing entries (#2961, #2926) are kept as written. The section has a theme paragraph, an "Upgrading from 10.0.20" table, the known issues (four carried over from 10.0.20, #2871 removed as fixed), and Fixed, Changed and Added entries. CI-only and test-only changes (#2918, #2920, #2938) have no entry. The date is the day of this draft and is set again at the cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Version-only bump of the root package and every packages/* manifest from 10.0.20 to 10.0.21 (25 files, including the new dkg-node-store). The lockfile is unchanged. `release-packages.mjs verify-versions --version 10.0.21` passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

….0.21-rc.1

otReviewAgent