RdsLogSource advances the resume marker inside the fetch, so any failure before the write loses that chunk permanently #3008

Darling/PerformanceMonitor.Darling.Service/Targets/RdsLogSource.cs records the RDS resume marker as a side effect of the fetch. On the current dev copy that is line 121:

if (!string.IsNullOrEmpty(response.Marker))
{
    _markers[key] = response.Marker;
}

The caller has not seen the chunk yet, let alone stored it. RdsDeadlockIngestor.IngestAsync and RdsPlanIngestor.IngestAsync then parse the text and COPY the rows. If anything between the fetch and a committed COPY fails, the marker has already moved — and on this transport that is permanent. DownloadDBLogFilePortion is consume-once, the marker is per-process and in memory, and the next call resumes past bytes that were never stored. Every readable deadlock (or plan) in that chunk is gone for the life of that marker.

The pg_read_file route does not have this problem. It re-reads an overlapping byte-window tail every cycle, so a cycle that drops a slab gets it again next time. The RDS route has no overlap to fall back on.

What can reach it today

• A store-write failure. WriteAsync opens a connection and runs a binary COPY under importer.Timeout = ServiceCommandDeadlines.CollectionSweepSeconds. A COPY that trips its deadline, a connection that drops, or a store that refuses the write all throw after the marker moved.
• Cancellation. Service shutdown or a cycle deadline landing mid-COPY, same shape.
• A parse throw — not yet. PgDeadlockLogParser.Extract and FromBlock cannot throw on dev: every failure path is a TryParse or a null return, and a block that will not parse is skipped. The throw that makes this acute arrives with Refuse deadlock reports from a non-UTC PostgreSQL log (#2993) #3001 (PgLogTimezoneUnsupportedException, for PgDeadlockLogParser captures the log's timezone abbreviation and never reads it — occurred_at assumes UTC no matter what log_timezone says #2993), which is open and unmerged — there are zero references to that type on dev.

Both ingestors are affected identically. Each holds its own RdsLogSource precisely because the marker is consumed per read, so this is a per-transport bug rather than shared-cursor contention.

Correcting the earlier framing of why this is quiet

An earlier writeup said this was latent because pg_deadlocks RDS log reads were denied fleet-wide by IAM. That is not the case, and it was retracted with store evidence: pg_deadlocks PERMISSIONS rows stop at 2026-08-30 15:09:37, and SUCCESS runs from 15:10:52 onward across all managed targets. The route works and has worked for six days. No IAM denial is current on that fleet.

So the real reason nothing has been lost yet is considerably weaker: no target is known to run a non-UTC log_timezone, so the #3001 throw would not fire even once it lands. Three of the fifty managed targets were sampled directly as UTC, via a shared configuration-file setting — forty-seven were never checked, and nothing in the codebase pins that GUC. Under the IAM framing this was safe until someone deliberately restored a grant. In reality it is safe until one parameter-group change on one cluster: no deliberate act required, and no alarm when it happens.

A latency claim that points at the wrong guardian is worse than no claim, which is why this is recorded here rather than left in a commit message.

Fix direction

Do not persist the marker until the caller has successfully stored the chunk's rows: hand the candidate marker back with the chunk and have the caller commit it after the write. That protects any failure between fetch and durable write — parse, store, cancellation — rather than one exception type, and it needs nothing from #3001.

It also sidesteps the representation problem the asymmetric alternative runs into. DarlingWorker.RunOneAsync classifies a run as either a row count or a thrown status, so "stored 8 rows AND refused some" has no clean representation today. A chunk that is committed whole or re-fetched whole never needs one.

Note that this converts the transport from at-most-once (silent loss) to at-least-once (a chunk can be re-stored if the process dies between a committed COPY and the marker moving). That is not a new property: the marker is in memory, so a restart already re-reads a bounded tail, and RdsLogSource's own doc comment records re-reading as the accepted trade. The fix makes the in-process behaviour match the across-restart behaviour that was already chosen.

A sibling defect this does NOT close

Filed separately, because it survives this one closing: same category — the marker outruns durability — different mechanism, and unlike the above it is active now with no exception required.

#3009 — a deadlock report cut at a chunk boundary is skipped or stored with participants missing, and the marker moves past it. The parse succeeds, so withholding the commit cannot see it.