Repository navigation
evodb curDBTransaction.IsClean() abort from out-of-transaction credit-pool snapshot write at DISK_SNAPSHOT_PERIOD boundaries #7549
Description
Summary
dashd/dash-qt aborts at flush time with
dashd: evo/evodb.cpp:59: bool CEvoDB::CommitRootTransaction():
Assertion `curDBTransaction.IsClean()' failed.
Posix Signal: Aborted
when a credit-pool lookup on the chain tip happens outside a dbTx while the
tip is on a DISK_SNAPSHOT_PERIOD (576) boundary. The interrupted evodb state
then fails VerifyDB on restart (*** Found EvoDB inconsistency, you must reindex to continue) even though the chain is at tip, so the node needs a full
-reindex to start again.
Reproduced deterministically on v23.1.8 (release). The relevant code
(CommitRootTransaction, CCreditPoolManager::AddToCache) is unchanged on
master at time of writing, so master is affected too.
Root cause
CCreditPoolManager::AddToCache() writes a snapshot to evodb unconditionally at
every 576-boundary, with no guarantee of being inside a committed transaction:
// evo/creditpool.cpp
void CCreditPoolManager::AddToCache(const uint256& block_hash, int height,
const CCreditPool &pool) {
{ LOCK(cache_mutex); creditPoolCache.insert(block_hash, pool); }
if (height % DISK_SNAPSHOT_PERIOD == 0) {
evoDb.Write(std::make_pair(DB_CREDITPOOL_SNAPSHOT, block_hash), pool);
}
}evoDb.Write targets the cur layer, which is only committed/rolled back by the
CEvoDBScopedCommitter obtained from BeginTransaction(). AddToCache is
reached from ConstructCreditPool ← GetCreditPool, which has four call sites —
only one is inside a dbTx:
| Call site | In a dbTx? |
|---|---|
evo/specialtxman.cpp:573 GetCreditPool(pindex->pprev) (ConnectBlock) |
yes (safe) |
rpc/rawtransaction.cpp:693 GetCreditPool(pTipBlockIndex) (getassetunlockstatuses, no-height branch) |
no |
node/miner.cpp:471 GetCreditPool(pindexPrev) (block assembly) |
no — but see note |
node/interfaces.cpp:433 getCreditPoolCounts (Qt CreditPoolFeed, timer-driven) |
no |
A block's own credit-pool snapshot is written when its successor connects
(ConnectBlock uses pprev). So while the tip is a 576-boundary block, that
block's snapshot is not yet in evodb. An out-of-dbTx GetCreditPool(tip) then
misses both the in-memory cache and the snapshot, runs ConstructCreditPool(tip)
→ AddToCache(tip) → the bare evoDb.Write lands in the uncommitted cur
layer. Nothing commits or rolls it back, so the next FlushStateToDisk →
CommitRootTransaction() fails assert(curDBTransaction.IsClean()). If the next
block connects first, its ConnectBlock dbTx commits cur and the stray write is
harmless — hence this is intermittent rather than a guaranteed crash.
Reproduction (regtest, v23.1.8)
dashd -regtest -datadir=DIR -daemon=1 -fallbackfee=0.0001
dash-cli -regtest generatetoaddress 575 <addr> # tip 575 (575 % 576 = 575)
dash-cli -regtest getassetunlockstatuses '["1"]' # no height -> queries tip
dash-cli -regtest stop # clean shutdown
dash-cli -regtest generatetoaddress 576 <addr> # tip 576 (576 % 576 = 0)
dash-cli -regtest getassetunlockstatuses '["1"]' # no height -> queries tip
dash-cli -regtest stop # -> Assertion `IsClean()' failed
Observed matrix (V20 active): 575+call → clean; 576+call → abort;
576+no call → clean; tip 580 + getassetunlockstatuses '["1"]' 576 (explicit
historical boundary height) → clean; getblocktemplate at tip 576 → clean.
Notes on the other out-of-dbTx doors:
getblocktemplate(miner.cpp:471) reaches the same bare write but did not
abort in testing — its template-validity check appears to open and roll back a
dbTx, whoseRollbackCurTransaction()→Clear()incidentally wipes the stray
write. This masking looks accidental.getCreditPoolCounts(interfaces.cpp:433) is polled by the Qt GUI's
CreditPoolFeedon a timer, so a running dash-qt wallet is expected to hit
this on its own at 576-boundaries with no user action. (Reasoned from the call
path; not separately GUI-tested.)
Exploitability
We believe this is not remotely triggerable: the RPC door is credential-gated
and (in the configs we checked) bound to loopback/private ranges; Drive/Platform's
own get_asset_unlock_statuses always passes a chainlocked height, so the DAPI
path takes the safe historical-block branch; and there is no mempool/p2p path
into GetCreditPool. The GUI door is self-inflicted. It reads as a reliability
bug, not a remote DoS. If you consider node aborts security-sensitive per your
SECURITY.md, we're happy to move this to security@dash.org instead.
Possible fixes (for discussion, not prescriptive)
- Make the snapshot write in
AddToCacheconditional on being inside a committed
transaction — read paths compute-and-cache only, durable writes go through the
same committerConnectBlockuses. (deterministicmns.cpp's snapshot writer
only runs fromProcessBlock, always in a dbTx — a useful contrast.) - Or wrap the out-of-dbTx
GetCreditPoolcallers in a rolled-back
BeginTransaction(read-only semantics) so any stray write is cleared — which
is effectively whatgetblocktemplatedoes by accident today.
The assert itself is correct (it prevents writing an inconsistent evodb); the bug
is on the write side. Happy to test any candidate fix against the regtest repro.
Activity
DashBot-0001 commentedon Aug 6, 2026
A few additions from following this through to recovery, in case they help triage.
1. Field data point. On the node where we hit this, the last block written to
evodb before the abort was height
2517120, and2517120 % 576 == 0— i.e. thetip was sitting exactly on a snapshot boundary, matching the mechanism above.
The abort came ~3 minutes after that block, while the tip was still on the
boundary (the next block had not yet connected).
2. Recovery is a full
-reindex, and the lighter option is blocked.-reindex-chainstatelooks like the natural fix (block files and block index areintact; only chainstate/evodb need rebuilding), but on a node with txindex it
errors out — and txindex is sticky from the presence of
indexes/txindex/even when it is not set in
dash.conforsettings.json, so a node can betxindex-enabled with no flag anywhere. Either
-reindex-chainstate -txindex=0(two-step) or a full
-reindexworks; we used the full reindex.Worth noting because the error message says "you must reindex to continue"
without indicating which reindex, and the obvious lighter choice fails.
3. Amplification on containerised setups. With a
restart: unless-stoppedpolicy, the single abort becomes a restart loop: each start re-runs
VerifyDB,fails the same way, and exits — in our case ~1400 times over several hours,
during which the node serves nothing and RPC returns
-28. For a masternodethat ends in a PoSe ban, which then requires a separate ProUpServTx to clear
(resyncing does not lift the ban). So the blast radius of one stray write is
considerably larger than a single crash.
4. Re the Qt path (
getCreditPoolCountsviaCreditPoolFeed): in theoriginal report I flagged that as reasoned-from-call-path rather than tested.
Adding a supporting observation — we previously had a
dash-qtwallet node anda headless masternode hit this same assertion on the same day, with no
operator action on the wallet node. That is consistent with the GUI feed's timer
polling
GetCreditPool(tip)on its own and self-triggering at a boundary,though I still have not tested the GUI path in isolation.
5. Detection, for anyone else hitting this. The dangerous window is
predictable: it is the ~1 block interval whenever
tip % 576 == 0, so the nextone is at
((tip / 576) + 1) * 576. That makes it straightforward to correlatean unexplained evodb abort against the boundary schedule.
Happy to run any candidate patch against the regtest reproducer.
sibTamias commentedon Aug 27, 2026
Field reports: same abort on evonodes at every 576-boundary (v23.1.8)
Confirming #7549 on dashmate evonodes (headless, no Qt). Several incidents, all on
height % 576 == 0, same sequence as the issue:CommitRootTransactionIsClean()abort while tip sits on the boundary → Dockerrestart: unless-stopped→VerifyDBFound EvoDB inconsistency, you must reindex to continueloop → PoSe ban.Environment
/Dash Core:23.1.8/, version230108), imagedashpay/dashd:23mainnet-core+ Drive/tenderdash)dash-qt). Corerpcwhitelistfordrive_consensusincludesgetassetunlockstatuses.1) 2026-06-24 — one evonode
2493504 % 576 == 0)CEvoDB::CommitRootTransaction(): Assertion curDBTransaction.IsClean() failedEvoDB inconsistencyat 2493504-reindexwas started by mistake and aborted; recovered later from a consistent peer snapshot.2) 2026-08-24 — several evonodes at once
Same second, same height — not disk corruption and not an operator stop/update.
2526912 % 576 == 0), DKGInitNewQuorumCEvoDB::CommitRootTransaction(): Assertion curDBTransaction.IsClean() failed-28)Other evonodes on the same Core version connected 2526912 without aborting — matches the issue’s “if the next block connects first, the stray
curwrite is committed/harmless” race.PoSe bans on the crashed set. Recovered by rsync of Core+Platform datadir from a healthy node after
dashmate stop --safe(not-reindex; txindex makes-reindex-chainstateunusable, as you noted).A dashmate 4.1.1 package update happened after this crash and was not the trigger.
3) 2026-08-27 — one evonode again (after a successful 08-25 restore)
That node had been at tip for ~2 days after restore. Then:
2517120Masternode State = UNKNOWN; on-chain POSE_BANNEDPeers restored at the same time stayed up through this boundary — again consistent with an intermittent stray write, not a bad block.
Trigger on evonodes (vs Qt)
These nodes have no
CreditPoolFeed/dash-qt. The remaining out-of-dbTx door that is actually wired on dashmate evonodes is Drive → Core RPCgetassetunlockstatuses.The issue text says Drive always passes a chainlocked height (safe historical branch). If that is true for every call, something else on the evonode path is still hitting
GetCreditPool(tip)in this ~3 minute window (DKG / flush / helper). If Drive sometimes omitsheight, that would explain several evonodes in the same LLMQ dying together at 00:22:39 UTC on 08-24.Happy to capture a
debug.logaround the next boundary (2529216 =2528640 + 576) with RPC tracing if that helps.Recovery note for operators
On Docker masternodes the abort is not “one crash”: VerifyDB on every start makes it a permanent loop and a PoSe ban.
-reindexfrom genesis is not operationally acceptable (~2.5M blocks). A consistent copy ofchainstate+evodbfrom a peer that did get the next block (so it never aborted) brings the node back; ProUpServTx is still required to clear the ban.The
assert(IsClean())is doing its job; the durable snapshot write inCCreditPoolManager::AddToCache()atDISK_SNAPSHOT_PERIODoutside a dbTx is the bug. A fix in 23.1.x would stop this class of fleet outage.sibTamias commentedon Aug 27, 2026
Addendum — synchronous upgrade/restart and a shared race window
A group of evonodes was upgraded to Core 23.1.8 and restarted together:
dashpay/dashd:23core containers were created in the same ~40-second window, so they almost certainly came up on the same starting height (shared snapshot) and stayed in lockstep.They later hit the same
DISK_SNAPSHOT_PERIODboundary at the same time, aborted together, and were PoSe-banned almost simultaneously.Other evonodes on the same Core version that had been started a few minutes earlier or later survived that same block.
So this is still #7549. A synchronous upgrade/restart does not corrupt EvoDB by itself; it only puts that group into the same race window (same tip, likely the same Drive/helper timer phase) when
height % 576 == 0.sibTamias commentedon Sep 15, 2026
Field report: 2026-09-15 — dashmate evonode at the next 576-boundary (v23.1.8)
Same abort as the reports above. Headless dashmate evonode, no Qt. Core v23.1.8 (
/Dash Core:23.1.8/, protocol70240), dashmate 4.1.1, Drive/tenderdash. No Core patch in this release.Sequence (UTC)
UpdateTip2539008 (2539008 % 576 == 0), hash00000000000000162e2bf675350d85f7efa4e2fdc7138f964d77ca9a5016668d,progress=1.000000FlushStateToDisk: write evodb cache to disk startedthenCEvoDB::CommitRootTransaction(): Assertion curDBTransaction.IsClean() failed→Posix Signal: Aborted/Segmentation faultLoaded best chain: … height=2539008VerifyDB(): *** irrecoverable inconsistency in block data at 2539008→ Dockerrestart: unless-stoppedcrash-loop; Drive sees Core RPC-28 Loading block index…Abort was ~5.5 minutes after the boundary block, while tip was still
2539008. Next network block was ~10.5 minutes after the boundary. Same race as2517120/2528640: flush while still onheight % 576 == 0.debug.logfragment (dash-7549-2026-09-15-evodb-isclean.log):One assert became a multi-hour outage (VerifyDB fails on every start while best chain is still
2539008). During that window the masternode missed DKG: on-chainPoSePenaltyjumped by ~66% of the registered-MN maximum (observed 1862, then decaying 1/block).PoSeBanHeight = -1. Platform epoch 82 started later the same morning while Core was already looping — not the trigger.No RPC trace of
getassetunlockstatuseswithoutheighton this incident. Same open question as 2026-08-27: Driverpcwhitelistincludes that call; this node has no QtCreditPoolFeed.Next boundary: 2539584 =
2539008 + 576. Watch forIsClean()in the minutes whiletip % 576 == 0.PastaPastaPasta commentedon Sep 22, 2026
Fixed on
developby two commits that landed as part of #7639 (merged in d1e64da). #7639's description didn't reference this issue, so GitHub never linked the two:fix(evo): only write credit pool disk snapshots inside a block-scoped EvoDB transaction.CCreditPoolManagernow writes the boundary snapshot only whenevoDb.HasActiveTransaction()is true (src/evo/creditpool.cpp:147). AGetCreditPool(tip)from outside a block connection (RPC, miner, Qt feed, and after feat: version 2 asset unlocks with stable txids and InstantSend locks (DIP-0027 amendment, v24) #7639 also mempool acceptance and IS signing) can no longer leave a stray write in the uncommittedcurlayer.fix(evo): persist the credit pool snapshot once a block connection hits a cached pool. Without this, skipping the write outside a transaction would mean the snapshot for a boundary block that was first built outside a transaction never got written, because the next block connection hits the in-memory cache. This commit writes it on that hit. Unit test:credit_pool_snapshot_persisted_after_transactionless_constructioninsrc/test/evo_assetlocks_tests.cpp.We tested the reproduction from the issue body (tip 575 →
getassetunlockstatusesat the boundary → flush): it passes on a develop build and aborts, then failsVerifyDB, on a build from before the fix.A correction to the trigger analysis for evonodes: Drive always passes an explicit
core_chain_locked_heighttogetassetunlockstatuses, so it doesn't go through the no-heightbranch. The explicit-height branch also callsGetCreditPool(chain[height]). If the boundary block is chainlocked before its successor connects, Drive's call makes the same stray write. Drive only calls it while broadcast withdrawals exist, which is why validators in one quorum abort in the same second.v23.1.8 is still affected. Neither commit is on
master, and v23.1.x has no active-transaction tracking inCEvoDB, so it needs an adapted backport rather than a cherry-pick. That will go in a v23.1.x point release. Until then, evonodes on 23.1.8 will keep hitting this atheight % 576 == 0boundaries.The recovery-side problem is tracked separately in #7703: a node interrupted between the coins flush and the EvoDB commit can only recover with a full reindex. That remains open.
Closing as fixed for v24.
🤖 Posted autonomously by Claude on behalf of pasta.