[jeremydmiller]

Closes #5414

Round three of exploratory crash testing with the durability chaos rig (src/Testing/DurabilityChaosRig, outside both solutions and CI). Every leg now has a recorded result, and every non-clean result is either fixed in 6.63 or explained below.

What the rig gained

• Stores: efpg (EF Core on PostgreSQL), plus mysql, oracle and sqlite (plain ADO.NET handlers with an enlisted outbox and an in-transaction mark-as-handled).
• Transports: mysqlqueue, oraclequeue, sqlitequeue, and pgshard (global partitioning over 4 sharded PostgreSQL queues).
• Knobs:
  • CHAOS_HANDLER_DELAY, CHAOS_RETRY_COOLDOWN and CHAOS_DURABILITY=solo.
  • CHAOS_DB_CONTAINER (any container for dbdown).
  • CHAOS_TOXIPROXY (a net fault: latency, reset_peer, black hole).
  • CHAOS_TENANTS (a database per tenant, the second tenant on its own container).
• Ledger: processed.envelope_id, so a duplicate can be followed through the worker logs.
• Logs: timestamped worker logs, and each worker prints its assigned node number.

Results (3 workers, 120 s of chaos unless noted)

Leg	Seeds	Result
1 MySQL queue	51, 52	Found #5428 (cleanup deadlock) and #5438 (outbox move not atomic). Both fixed: 3 reruns on main, 0 duplicates, 2 fully CLEAN. Left: deadlock dead letters, see below.
1 Oracle queue	53, 54	Found #5429 (362 / 1182 duplicates). Fixed: 0 duplicates on rerun. Left: connection failures during outages become dead letters (lower priority, not filed).
2 SQLite queue	55, 56, solo	Found #5430 (Balanced mode node numbers, UTC health checks). Fixed: seed 55 CLEAN.
3 Dead letter replay racing kills, SQL Server	57, 58	57 CLEAN. 58 had 16 duplicates = #5426, fixed: CLEAN on rerun.
4 SIGTERM-heavy, 0.5-3 s handlers, 5 s retry cooldown	59, 60	CLEAN (slow to settle because of the handler backlog).
5 Toxiproxy network faults	61, 62	Lost work equals dead letters exactly. Transient network errors on the commit with no retry policy in the rig: expected, see below.
6 EF Core on PostgreSQL, outages + 5 s timeouts	63, 64	CLEAN.
7 Global partitioning, sharded PostgreSQL queues	65 (×3), 66, 67	The first seed 65 run had 8 duplicates = #5426. On main: 5/5 CLEAN.
8 Database per tenant, tenant database outages	80, 81, 82, 83	Found #5435 (two tenant databases shared one agent Uri). Fixed: 4/4 CLEAN.

Issues fixed in 6.63 from this round: #5426, #5428, #5429, #5430, #5435, #5438.

Expected, not bugs

Transient database errors on the handler's commit dead-letter the message, because the rig configures no retry for them:

• MySQL deadlocks inside MarkIncomingEnvelopeAsHandledInTransactionAsync
• Oracle ORA-50201 / ORA-12516 while the database is paused
• Toxiproxy resets and black holes

The commit rolled back in each case, so nothing is half-done and the message is in the dead letter table. An application covers this with a retry rule such as opts.OnException<MySqlException>().RetryWithCooldown(...).

🤖 Generated with Claude Code

https://claude.ai/code/session_013nKJrqsJw3j23Xd46q37ob