Skip to content

test(streams) astubbs#255: the evidence suite - a seam-on divergence lane, the broker-backed laws, and a benchmark with a noise floor - #398

Draft
astubbs wants to merge 12 commits into
feats/ks-streams-task-lifecyclefrom
test/ks-streams-evidence-suite
Draft

astubbs wants to merge 12 commits into
feats/ks-streams-task-lifecyclefrom
test/ks-streams-evidence-suite

Conversation

@astubbs

@astubbs astubbs commented Aug 31, 2026 •

Copy link
Copy Markdown
Owner

Serves #255.

depends on #394

Description

The claim this PR makes: this module's three outstanding evidence gaps are closed, and each is
closed with a gate that has been shown to go red - Apache Kafka's own suite now runs with the seam
ON and every divergence is classified by mechanism, the properties stock Kafka Streams cannot express
are asserted against a real broker with their controls, and the realistic keyed benchmark reports its
own noise floor beside every ratio.

Nothing here touches the seam. No main Java in parallel-consumer-streams changed, and the patch is
untouched - deliberately, because two sibling rungs are changing dispatch semantics on this same base.

1. The seam-on evidence lane

The seam-off oracle has always been the behaviour-preservation gate: the patch changes nothing
while the switch is off. This adds its opposite - what Kafka's own suite says about the path this
module actually ships - and, more usefully, a verdict on it.

Seam-on failures are expected, which is why the branch forest's version of this run recorded them
and gated on nothing. The refusal envelope throws where stock would have run a construct; a committed
offset is Parallel Consumer's frontier rather than Streams' consumedOffsets; a record handed to a
worker has not been processed by the time process() returns. A gate that goes red every build is a
gate people stop reading.

So the run and the verdict are separate surefire executions. kafka-upstream-tests-seam-on
carries testFailureIgnore and never reddens a build; seam-on-evidence-classification reads what it
wrote and fails when a divergence matches no named mechanism. That split is the only arrangement in
which "these failures are expected" and "a new divergence fails the build" are both true.

The seam-off run is the control arm, and the two are differenced case by case. A divergence is a
case that passes with the switch off and fails with it on; nothing else is attributable to the
dispatch path. A case failing in both arms is failing for a reason the seam had nothing to do with,
and counting it would credit a change that did not cause it.

Three mechanisms are derived from the code at run time, so a sibling rung changing the envelope
needs no edit here at all:

Mechanism How it is recognised
refused-construct the marker is the longest common prefix of every message PcUnsupportedConstruct can produce, and the attribution is whichever constant the message names. No refusal string is copied
commit-frontier-encoding the assertion renders an OffsetAndMetadata, so it is an assertion about the commit encoding whatever the case is called
ledgered-flake a flaky-case: marker in any docs/inflight/ note

What cannot be derived is attributed in the ledger, not in the classifier. "This assertion fails
because Kafka's test drives the task synchronously" is a judgement about a test, not a fact
recoverable from expected: <5> but was: <0>. Those live in
docs/inflight/test-streams-seam-on-divergence-triage.md, keyed by method rather than by
parameterisation, under four named classes - asynchronous-dispatch, stream-time-never-advances,
exception-type-lost-in-the-worker, split-poll-wait. Three consequences, all deliberate: the
reasoning is reviewed as prose by whoever changes the semantics under it; the PR that closes a
mechanism deletes its entries in the same change; and an entry that stops matching is reported as
stale rather than rotting silently.

The flake is absorbed by mechanism rather than by a retry. The invalid-timestamps flake has been
failing this module's oracle intermittently for weeks. Because the oracle fails fast, one sighting
used to abort the lane before the measured arm ran at all - Kafka's suite twice, re-run, for a failure
already diagnosed. The lane now relaxes that one execution and makes the verdict itself: every
control-arm failure must carry a flaky-case: marker, or the lane stops and says there is nothing
trustworthy to difference against. The ordinary build's oracle is untouched and still fatal.

One inherited finding is corrected. The forest's triage counted that flake's seam-on failure as
the flake firing. It is attributed here to asynchronous-dispatch instead, on evidence the earlier
triage did not have: a sibling case fails seam-on in exactly the same parameterisation with exactly
the same assertion shape, in every seam-on run taken here, while the control arm was clean in the same
runs. A coin-flip flake does not land on two sibling tests at the same parameter in consecutive runs.
The flake registry is therefore consulted last among the divergence mechanisms and first on
the control arm - a flake explains a dirty control; letting it explain a divergence is how a real one
disappears.

2. The laws Kafka's own suite cannot state

Kafka's suite is the oracle for behaviour preservation. It cannot state the properties this module
exists for, because stock Kafka Streams has no arm in which two records of one partition run at once.
These do, and each carries its control:

  • PcDrivenStreamsProofTest - PC-path output is identical, record for record, to what stock Kafka
    Streams produced. Two controls answering different questions: a same-harness seam-off arm, which
    separates "the patch changed the output" from "concurrency changed the output"; and the stock
    baseline itself, generated outside this module.
  • PcDrivenStatefulProofTest - the same for a stateful topology, where Kafka's own store wrappers
    read the per-task record context ambiently. The module's existing envelope test already said this
    proof was the reason non-windowed aggregation is not refused - and the proof was not on this stack,
    so that sentence was pointing at nothing.
  • CommitFrontierCrashRestartTest - nothing lost or double-counted across a crash. The committed
    frontier is a point below which every record has been processed, not the offset of the last one
    handed out, and under concurrent dispatch those are different records.
  • ShadowedStreamsControlTest - stock, single-threaded Kafka Streams running our generated
    classes. It is a control: a concurrency change that quietly breaks the serial path is not a fix.

The baseline is generated in another module, and that is the whole point. This module compiles
patched Kafka classes into its own build output, which precedes the kafka-streams jar on the
classpath, so every KafkaStreams instance in this JVM runs the patch. A "stock arm" written here
would share every defect the patch introduced. The baselines come from
parallel-consumer-example-streams, which does not depend on this module and asserts its own
classpath is stock before recording a row. Both fixture tests were re-run on this branch and the
tracked fixtures still match what genuinely-stock Kafka Streams produces.

The lane above defers to one of these, and says so. commit-frontier-encoding cannot tell a
deliberate encoding divergence from an offset regression - both render an OffsetAndMetadata and
nothing in the assertion text separates them. Its javadoc names CommitFrontierCrashRestartTest as
what actually defends offset correctness, because that arm asserts on records that arrived rather than
on a base64 metadata string. An acknowledged blind spot with a named owner beats a predicate
pretending to cover it.

3. The realistic-domain benchmark, with a noise floor

A card issuer screens every authorisation against a fraud-scoring service; the screener has been down
and the queue has to clear. The canonical Kafka Streams enrichment shape, keyed the way a real
deployment would have to key it, with a skewed keyspace and a heavy-tailed cost - the two things that
cost this seam most.

Three arms, not two, and the third is what this rung adds. It is a second stock arm on the same
records: whatever it differs from the first by is this machine in this state, and that bounds how far
a ratio can move with nothing changed at all. The headline assertion no longer asks whether the ratio
beat one - a direction check is a coin toss at parity and is satisfied by a run in which the mechanism
did nothing. It asks whether the ratio beat the floor by a margin.

No figure from an earlier run survives in this diff. The forest's numbers were measured on another
machine; the headline multipliers, the wake-on-work ablation percentage and the CPU-bound refutation
are removed or attributed to the branch they were taken on. DEMO.md shows the report's shape with
the numbers elided and says why. The run prints its own, with the floor beside them.

Proven able to fail - every gate here, with the prediction stated first

The lane, three ways.

  • An unexplained divergence. Removing the close-time commit-state publication from
    PcTaskDispatcher.close() was predicted to make exactly one case,
    shouldClearCommitStatusesInCloseDirty, fail seam-on while staying green seam-off, with no
    mechanism recognising a bare commitNeeded assertion. Exactly that: one unexplained divergence,
    named, with its recorded failure and stack; every other divergence still explained by the same six
    mechanisms; the script exited 1. Reverted and verified byte-identical by checksum. That run also
    demonstrated the control-arm path unplanned - the invalid-timestamps flake fired seam-off, and the
    lane named it from its ledger note and carried on instead of aborting.

  • An expected-class divergence. The clean run classifies every divergence under six named
    mechanisms, with the refused constructs named individually, and exits 0.

  • A missing or stale report directory. Running the classification execution alone against deleted
    directories failed with "That is an ERROR and not an empty result"; against directories from an
    earlier run it failed on the freshness check, naming the file, when it was written, and when the
    build started.

  • A control arm nobody has diagnosed - and this one arrived on its own. On a later CI run
    StreamThreadTest.shouldChangeStateAtStartClose[1] failed seam-off, in Kafka's own test, on a
    10-second shutdown bound. The lane named the ledgered invalid-timestamps flake beside it, refused to
    produce a verdict on the divergence set while an undiagnosed control-arm failure stood, and said
    so in a verdict that outranked the classification - which was itself clean, no unexplained
    divergences at all. That is the mode this design exists for, and it had only ever been exercised on
    synthetic reports before.

    It is recorded in docs/inflight/test-streamthreadtest-start-close-shutdown-bound.md with the
    argument that carries it: the same case passed in the seam-ON arm of the same run, minutes
    apart on the same runner with the same patched classes, and a defect the patch introduced cannot be
    repaired by turning the seam on. One sighting, and the note says which observation would void it.

SeamOnDivergenceClassifierTest pins the first three at unit level on synthetic reports in milliseconds,
plus what the end-to-end runs cannot cheaply reach: an arm whose pin did not arrive (both arms
recording dispatch=false differences the control against itself and reports a flawless zero), arms
covering different classes, an attribution to an undeclared class, and a triaged attribution beating a
flake marker on the same case.

The benchmark, through a documented flag rather than a code edit. Predicted: with a worker pool of
one, PC dispatch has no concurrency to exploit, so the seam-on arm lands inside the noise floor and
the headline assertion fails while the run stays otherwise healthy. Exactly that - --pool 1 failed
one case, the headline, and its message carried the whole diagnosis including both stock arms and the
PC arm; the steady-state case, which asserts on sample size rather than a ratio, stayed green.

The margin rule itself is pinned by NoiseFloorTest, which runs in the ordinary build because it is a
pure function of two numbers. Its first case is the one that matters, and it is the one --pool 1
cannot reach: a 1.01x observation does not beat even the tightest possible floor. That is the
shape of the sabotaged run on #391 that printed its own headline claim, and a
direction check passes it.

Two defects found in the machinery being ported, both silent

bin/streams-benchmark.sh ran nothing and exited 0. Carried across unchanged it drove failsafe
with -Dit.test and -DskipUTs; these benchmarks live in src/test/java and this module has no
src/test-integration, so that selects nothing, runs nothing, and succeeds. It now drives surefire -
and because failIfNoSpecifiedTests has to be off for the Kafka executions' sake, the script asserts
a surefire report exists for every class it selected and exits 2 when one does not.

--pool N was documented and dead. The harness hardcoded the pool, so a reader could ask for a
configuration, be told in every log line that they had got it, and be measuring something else.

And one defect this PR introduced and caught before merge: naming the classifier in the
default-test execution silently dropped the root pom's two surefire excludes, promoting every
broker-backed arm out of failsafe and into the unit lane, where they ran, needed Docker, and went
green. Caught by reading the run rather than the tick. Both directions are now verified.

The one thing red here that is not expected, and it is NOT being greened

PcDrivenStatefulProofTest.pcDrivenAggregationMatchesTheStockBaseline timed out on repetition 2 of 3
in the Integration Tests lane - the run that first put that class on CI. Repetitions 1 and 3 passed
in the same JVM.

The obvious reading is wrong, and the run says so itself. Its StreamThread logged
polled 1207 times against Processed 14 total records in the two minutes it was stuck. A thread
short of CPU polls less; this one was going round its loop briskly and finishing almost nothing,
which is the shape of work dispatched and not completed rather than of a busy runner. The ambient
probe was clean - no rebalance dwell, no lag stagnation, no frozen partitions - so the group was
progressing at the broker while the task was not progressing at the processor.

A control arm was run rather than a verdict guessed. The whole streams integration suite,
locally, at the lane's own forkCount=4 -DreuseForks=true alongside core's integration suite: green,
including this class. Uncontended at forkCount=1: green in three separate runs. The instrumentation
is confirmed rather than assumed - failsafe pins no <forkCount> of its own, so the property does
arrive, unlike the surefire trap the root pom documents.

So contention is not established, and it is not being claimed. The one term the local arm cannot
vary is core count: four JVMs and four brokers on a developer machine is not four JVMs and four
brokers on two cores. docs/inflight/test-streams-stateful-proof-stalled-on-ci-while-still-polling.md
records the sighting, both arms, the two unattributed exception lines from the interleaved four-fork
log, and the experiment that would settle it - the same runner at forkCount=1, which holds the
machine fixed and varies only the load.

Nothing was loosened, shortened or serialised to make this pass. The lane re-runs on every push, so
the rate is being collected meanwhile.

The duplication reports, triaged

dups: clones was right once and the clone is gone. Both engines agreed on consumeByKey plus
flatten being copied between the seam-on dispatch arm and the stock control arm, byte-identical
apart from the deadline. That is worth fixing here for a reason specific to this module - those two
are a matched experiment, and a copy of the reading logic in each is a second varying term waiting to
appear the first time somebody tunes one. It now lives on BrokerStreamsIntegrationTest, which both
extend and which already holds the same broker, the same serdes and the same starting offset; the
deadline stays a parameter at each call site. Whole streams integration suite re-run green afterwards.
PMD CPD now passes both its rules.

dups: similarity and jscpd's threshold stay red, and the reason is what they measure. Every
remaining hit is an import block or a javadoc header between sibling classes - the offsets give it
away, they are lines 1 to 22 of the files. jscpd is lexical and counts comments; PMD CPD is
token-based and reports none of them. A rung that adds this much test code with similar imports and
long javadoc moves a lexical percentage, and +0.11% against a +0.1% limit is that, not new logic.

One real clone is deliberate and stays: the fixture format, written by
StockBaselineFixtureSupport and read by BaselineFixture. Those two live in modules that must not
share code - if the example module depended on the streams module, the baseline it generates would
run the patched classes and stop being a baseline. Extracting it would need a third module whose only
job is to reconnect the two things this design keeps apart.

Where this came from, and your siblings will move these results

Copied out of #271, the Kafka Streams feasibility study, which remains
the source god PR and keeps the semantic work, the measurements and the open review threads. The
seam-on run machinery and the triage method come from feats/ks-streams-seam-on-upstream-gate; the
benchmark suite and DEMO.md come from test/ks-streams-realistic-domain-benchmark. PC-side classes
are renamed io.confluent to bz.stub.

Two sibling rungs are in flight on this same base - stream time and punctuation, and error
surfacing and backpressure - and both change dispatch semantics. Their results will move this
lane's, and the lane absorbs that by design
: it classifies by the failure's own evidence rather than
by test name wherever it can, and where it cannot the attribution is a ledger entry those rungs edit
in the PR that closes it. Two of the four attributed classes are theirs by name -
stream-time-never-advances and exception-type-lost-in-the-worker - and the lane reports an entry
that has stopped matching, so a rung that fixes one gets told rather than leaving a stale expectation.

Not here, named so a reader does not expect them: HeadOfLineBlockingBenchmarkTest, deliberately
left on the forest because it isolates head-of-line blocking on a fixture whose own construction bounds
its multiplier and the settled decision is to lead with backlog catch-up; and the example module,
which is #391's.

The dependency gate will be red until #394 merges. That is expected.

Checklist

  • Docs updated - the module README (the seam-on lane and what its failures mean, the table of laws
    and their controls, and the reinstatement rule now naming the command that answers it), DEMO.md
    rewritten with its figures elided and the negative control documented, two docs/inflight/
    notes (the divergence triage, and the flake note gaining a machine-readable marker plus what the
    lane now does with it), and two method write-ups carried with the benchmark whose citations
    pointed at them
  • User-facing feature documentation data added under docs/features/ - N/A - the module is still unpublished, so there is no artifact to depend on and nothing for the ladder to point a user at. The roadmap stage is unchanged, for the same reason the four rungs below gave
  • Tests added/updated - the whole PR is tests. SeamOnDivergenceClassifierTest and
    NoiseFloorTest at unit level; SeamOnDivergenceLaneTest as the lane's gate; four broker-backed
    law arms with their controls; and the benchmark suite with its noise-floor arm
  • Title & body reflect the final content of this PR
  • Ran ce-simplify and ce-code-review locally - N/A - asked for review on the PR instead. Most of the diff is a copy from branches already reviewed on astubbs/parallel-consumer#271; what is new here - the classifier and its ledger-driven mechanisms, the control-arm integrity check, the noise floor and the margin rule, and the two silent defects found in the ported machinery - is what a reviewer should spend attention on

astubbs and others added 5 commits September 1, 2026 09:39
…rgence nobody has named

The seam-off execution has always been the behaviour-preservation gate: the patch changes nothing
while the switch is off. This adds its opposite - what Apache Kafka's own suite says about the path
this module actually ships - and, more usefully, a verdict on it.

THE PROBLEM WITH A SEAM-ON RUN, AND THE SHAPE THAT SOLVES IT

Seam-on failures are expected. The refusal envelope throws where stock would have run a construct;
the committed offset is Parallel Consumer's frontier rather than Streams' consumedOffsets; a record
handed to a worker has not been processed by the time process() returns. A gate that goes red every
build for those reasons is a gate people stop reading, which is why the branch forest's version of
this run recorded its failures and gated on nothing.

So the run and the verdict are SEPARATE surefire executions. kafka-upstream-tests-seam-on carries
testFailureIgnore and never reddens a build; seam-on-evidence-classification reads what it wrote and
fails when a divergence matches no named mechanism. That split is the only arrangement in which
"these failures are expected" and "a new divergence fails the build" are both true.

THE SEAM-OFF RUN IS THE CONTROL ARM, AND IT IS DIFFERENCED CASE BY CASE

A divergence is a case that PASSES with the switch off and FAILS with it on. Nothing else is
attributable to the dispatch path: a case failing in both arms is failing for a reason the seam had
nothing to do with, and counting it would credit a change that did not cause it. The classifier
therefore reports four disjoint sets - divergences, failing-in-both, passing-only-seam-on, and the
control arm's own failures - rather than one number.

MECHANISMS ARE DERIVED FROM THE CODE WHERE THAT IS POSSIBLE

RefusedConstruct takes its marker as the longest common prefix of every message PcUnsupportedConstruct
can produce, and attributes a failure to whichever constant the message names. No refusal string is
copied, so the refusal envelope - which belongs to another rung and moves independently - can gain or
lose a construct with no edit in this lane at all.

CommitFrontierEncoding recognises an assertion that renders an OffsetAndMetadata, which is an
assertion about the commit encoding whatever the case is called. Its limitation is stated rather than
hidden: an offset REGRESSION inside a Kafka unit test renders identically and would be explained by
this mechanism too. Nothing in the assertion text separates the two, so the lane does not pretend to -
what defends offset correctness is a broker-backed law, not this.

WHAT CANNOT BE DERIVED IS ATTRIBUTED IN THE LEDGER, NOT HERE

"This assertion fails because Kafka's test drives the task synchronously" is a judgement about a test,
not a fact recoverable from `expected: <5> but was: <0>`. Those attributions live in
docs/inflight/test-streams-seam-on-divergence-triage.md, keyed by method rather than by
parameterisation, and the lane reads them. Three consequences, all deliberate: the reasoning is
reviewed as prose by whoever changes the semantics under it; the PR that closes a mechanism deletes
its entries in the same change; and an entry that stops matching is reported as stale rather than
rotting silently.

THE FLAKE IS ABSORBED BY MECHANISM RATHER THAN BY A RETRY

StreamThreadTest.shouldLogAndRecordSkippedRecordsForInvalidTimestamps has been failing this module's
oracle intermittently for weeks, and its note says so. Because the oracle fails fast, one sighting used
to abort the lane before the measured arm ran at all - a full re-run of Kafka's suite twice, for a
failure already diagnosed. The lane now relaxes THAT ONE EXECUTION and makes the verdict itself: every
control-arm failure must carry a flaky-case marker in docs/inflight/, or the lane stops and says there
is nothing trustworthy to difference against. The ordinary build's oracle is untouched and still fatal.

The flake registry is consulted FIRST on the control arm and LAST among the divergence mechanisms, and
that ordering is load-bearing - see the triage note's own finding.

PROVEN ABLE TO FAIL, THREE WAYS, WITH THE PREDICTION STATED FIRST

Sabotage, end to end. Removing the close-time commit-state publication from PcTaskDispatcher.close()
was predicted to make exactly one case, shouldClearCommitStatusesInCloseDirty, fail seam-on while
staying green seam-off, with no mechanism recognising a bare commitNeeded assertion. Exactly that: one
unexplained divergence, named, with its recorded failure and stack; every other divergence still
explained by the same six mechanisms; the script exited 1. Reverted and verified byte-identical by
checksum. That run also demonstrated the control-arm path unplanned - the invalid-timestamps flake
fired seam-off, and the lane named it from its ledger note and carried on instead of aborting.

A stale report directory. Running the classification execution alone against report directories from
an earlier run failed on the freshness check, naming the file, when it was written and when the build
started. maven.build.timestamp is passed in for exactly this, and is required rather than optional: a
check that turns itself off when its input is missing is the failure mode this lane exists to avoid.

A missing report directory. The same execution against deleted directories failed with "That is an
ERROR and not an empty result", rather than reading an empty glob as zero failures.

SeamOnDivergenceClassifierTest pins all three at unit level on synthetic reports, plus the cases the
end-to-end runs cannot cheaply reach: an arm whose pin did not arrive (both arms recording
dispatch=false differences the control against itself and reports a flawless zero), arms covering
different classes, an attribution to an undeclared class, and a triaged attribution beating a flake
marker on the same case.

NO COUNT APPEARS IN THE MACHINERY OR IN THE LEDGER. The per-arm totals are printed for a reader and
the verdict does not consult them; which cases diverge moves with the branch, the Kafka version and
the seam, and the command that re-derives them is in the note and in the README.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot a local-only claim

A lane that only ever runs on the machine of whoever wrote it is half a deliverable: the first person
to change the dispatch path's semantics learns nothing, and the ledger of attributions rots with
nobody to notice.

RED HERE IS NOT EXPECTED, WHICH IS WHAT MAKES IT DIFFERENT FROM THE QUARANTINE LANE

That lane runs tests known to be failing, so its verdict lives in the step summary and its run step
carries continue-on-error. This one asserts that every seam-on divergence is EXPLAINED, and green is
the ordinary state. Red means one of exactly two things, both wanting a human rather than a re-run: a
divergence nobody has attributed - something changed the dispatch path's semantics - or an attribution
in docs/inflight/ that went stale because a rung fixed the thing it described. So the job gates.

PATH-SCOPED, AND THE LEDGER IS IN THE TRIGGER SET

The run costs Kafka's whole suite twice, and nothing outside the streams module, the lane script and
docs/inflight/ can change its answer. The ledger belongs in that set on purpose: an edit to an
attribution changes the verdict exactly as much as an edit to the code does, and a triage note edited
without re-running is how a lane starts lying.

The whole measurement goes to the step summary and is kept as an artifact with both arms' surefire
reports, so a red run can be read without re-running it.

A red on the dependency fetch is a download failure rather than a divergence, and the step says so:
this module pulls Kafka's sources and test-sources CLASSIFIER artifacts by explicit artifactItems,
which dependency:go-offline does not resolve, so the cache-warming job does not cover them. That gap
is already recorded in docs/inflight/, and this workflow points at it rather than restating it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… way to state

Apache Kafka's own suite is the oracle for behaviour PRESERVATION. It cannot state the properties this
module exists for, because stock Kafka Streams has no arm in which two records of one partition run at
once - so no stock test asserts what happens when they do. These are the arms that do.

WHAT IS NOW ASSERTED AGAINST A BROKER, AND WITH WHAT CONTROL

PcDrivenStreamsProofTest - the PC path's output is identical, record for record, to what stock Kafka
Streams produced. Two controls, and they answer different questions: a same-harness seam-off arm,
which separates "the patch changed the output" from "concurrency changed the output"; and the stock
baseline itself, which is generated OUTSIDE this module.

PcDrivenStatefulProofTest - the same for a stateful topology, which is where Kafka's own store
wrappers read the per-task record context ambiently. The stateless arm's only ambient reader is the
probe this module wrote; this one makes Kafka's own run. The module's existing envelope test already
said this proof was the reason non-windowed aggregation is not refused - and the proof was not on this
stack, so that sentence was pointing at nothing.

CommitFrontierCrashRestartTest - nothing is lost or double-counted across a crash. The committed
frontier is a point below which every record has been processed, not the offset of the last one handed
out, and under concurrent dispatch those are different records.

ShadowedStreamsControlTest - stock, single-threaded Kafka Streams running our generated classes. It is
itself a control: a concurrency change that quietly breaks the serial path is not a fix.

THE BASELINE IS GENERATED IN ANOTHER MODULE, AND THAT IS THE WHOLE POINT

This module compiles patched copies of Kafka's classes into its own build output, which precedes the
kafka-streams jar on the classpath, so EVERY KafkaStreams instance in this JVM runs the patch. A "stock
arm" written here would share every defect the patch introduced and the comparison would prove nothing.
The baselines come from parallel-consumer-example-streams, which does not depend on this module and
asserts its own classpath is stock before recording a row.

The fixture carries the INPUTS as well as the outputs, and the arms here replay those inputs rather
than rebuilding them from a second copy of the generation rules - the two modules cannot share code in
this reactor order, so replaying is what stops the arms drifting apart in what they were fed.

Re-verified here rather than trusted: both fixture tests were run on this branch and the tracked
fixtures still match what genuinely-stock Kafka Streams produces, so the baselines describe this Kafka
version and not the branch they were recorded on.

THE LANE ABOVE DEFERS TO ONE OF THESE, AND SAYS SO

The seam-on evidence lane's commit-frontier-encoding mechanism cannot tell a deliberate encoding
divergence from an offset regression: both render an OffsetAndMetadata in a Kafka unit test's assertion
and nothing in that text separates them. Its javadoc names CommitFrontierCrashRestartTest as what
actually defends offset correctness, because that arm asserts on records that arrived rather than on a
base64 metadata string. An acknowledged blind spot with a named owner is worth more than a predicate
pretending to cover it.

WHERE THIS CAME FROM, AND WHAT WAS DELIBERATELY LEFT

Copied out of #271's branch forest, package-renamed io.confluent to bz.stub,
compiled and run unchanged otherwise. HeadOfLineBlockingBenchmarkTest is NOT carried across: it
isolates head-of-line blocking on a fixture whose own construction bounds its multiplier, and the
settled decision is to lead with backlog catch-up instead. The two files that referenced it now say so
rather than pointing at a class that is not here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… that says when to ignore it

A card issuer screens every authorisation against a fraud-scoring service. The screener has been down,
authorisations have queued, and it has just come back. That is the workload: the canonical Kafka
Streams enrichment shape, keyed the way a real deployment would have to key it, with a skewed keyspace
and a heavy-tailed service cost - the two things that cost this seam the most.

Three arms, not two, and the third is the point of this change.

THE NOISE FLOOR, MEASURED INSIDE THE RUN THAT QUOTES IT

The extra arm is a second STOCK arm, given the same records. Whatever it differs from the first by is
this machine in this state, and that is an upper bound on how far a ratio can move with nothing
changed at all. Without it on screen a run taken on a busy machine looks exactly like a finding - and
that is not hypothetical, it is what #391 found on the example rung, where a
run reading 1.19x on its control had two stock arms differing by 1.20x. The control had measured
nothing and nothing said so.

So the headline assertion no longer asks whether the ratio beat one. It asks whether the ratio beat
the floor by a margin. A direction check is a sound test of an arm that loses and a coin toss for an
arm at parity, and - the reason this rule exists - it is satisfied by a run in which the mechanism did
nothing at all.

PROVEN ABLE TO FAIL, AND WITH A DOCUMENTED FLAG RATHER THAN A CODE EDIT

Prediction, stated first: with a worker pool of one, PC dispatch has no concurrency to exploit, so the
seam-on arm should land inside the noise floor and the headline assertion should fail while the run
stays otherwise healthy.

Exactly that. `bin/streams-benchmark.sh --pool 1` failed one case - the headline - and its message
carried the whole diagnosis: measured 0.99x against a floor of 1.00x, threshold 1.51x, with both stock
arms and the PC arm printed beside it. The steady-state case, which asserts on sample size rather than
on a ratio, stayed green. That asymmetry is what says the right thing broke.

`--pool 1` reaching the harness at all is a fix in this commit. The script has always documented
`--pool N`, and the harness hardcoded four - so a reader could ask for a configuration, be told in
every log line that they had got it, and be measuring something else. The same class of failure the
script's own "an unknown option is an error" rule was written against, arriving by the other route.

The margin rule itself is pinned by NoiseFloorTest, which runs in the ordinary build because it is a
pure function of two numbers and should not need a broker or a `performance` tag to be exercised. Its
first case is the one that matters: a 1.01x observation does not beat even the tightest possible
floor. That is the shape of a sabotaged run, and a direction check passes it.

THE SCRIPT RAN NOTHING ON THIS LAYOUT, AND WOULD HAVE EXITED 0

Carried across unchanged, `bin/streams-benchmark.sh` drove FAILSAFE with `-Dit.test` and `-DskipUTs`.
These benchmarks live in `src/test/java` and this module has no `src/test-integration`, so that
selects nothing, runs nothing, and succeeds. It now drives surefire - and because
`failIfNoSpecifiedTests` has to be off for the Kafka upstream executions' sake, maven can no longer
report an empty selection, so the script asserts a surefire report exists for every class it selected
and exits 2 when one does not. A skip is never a pass.

NO FIGURE FROM AN EARLIER RUN SURVIVES IN THIS DIFF

The forest's numbers were measured on another machine and describe it, so the headline multipliers,
the wake-on-work ablation percentage and the CPU-bound refutation are all removed from the javadoc or
attributed to the branch they were taken on. `DEMO.md` shows the report's SHAPE with the numbers
elided, and says why: a figure written into documentation quietly stops describing the reader's
machine. The run prints its own, and prints the floor beside them.

Two method write-ups come with the suite because its citations were pointing at them - choosing the
statistic that states the claim, and control arms varying exactly one term. Both are records of work
done on the branch forest, so their citations into that forest's plan and benchmark carry paragraph
markers rather than being repointed at files that would make the record describe a run that never
happened.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… it pulled Docker into the unit lane

Naming the seam-on classifier in this module's `default-test` execution silently dropped the two
excludes the root pom sets on surefire - `**/integrationTest*/**/*.java` and `**/*IT.java` - because
an `<excludes>` list REPLACES the inherited one rather than adding to it.

The effect was invisible in the only way that matters: everything passed. Every broker-backed arm in
this module lives in an `integrationTests` package, so they were quietly promoted out of failsafe and
into the surefire `test` phase, where they ran, needed Docker, and went green. A unit lane that has
started requiring Docker does not announce it; it just takes longer until the day it runs somewhere
without a daemon.

Caught by reading the run rather than the tick: the module's `test` phase reported far more cases than
its unit suite has, and the class names in the list were the integration ones. Fixed by repeating the
root's two entries beside the new one - `combine.children="append"` would express the intent, but a
literal list is what a reader can check against the root pom without knowing that attribute exists.

Verified both directions: the `test` phase now runs the unit suite alone with the seam-off oracle
still at its usual figures, and `verify` runs every broker-backed arm including the ones this stack
adds.

The generated stock-baseline fixtures are classified in the copyright scanner as regenerated-by-a-
generator, which is the first of the four reasons that table accepts and the same one `*.patch`
carries. They are rewritten wholesale by `StockBaselineFixtureSupport`, so a header would be
destroyed on the next run. Two explicit paths rather than a blanket `*.tsv`: the next `.tsv` to
arrive should be classified on its own merits, not inherit an exemption written for these.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

Dependency Review

The following issues were found:
  • ✅ 0 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ⚠️ 4 package(s) with unknown licenses.
See the Details below.

License Issues

.github/workflows/streams-seam-on-evidence.yml

PackageVersionLicenseIssue Type
actions/cache/restore4.*.*NullUnknown License
actions/checkout6.*.*NullUnknown License
actions/setup-java5.*.*NullUnknown License
actions/upload-artifact7.*.*NullUnknown License
Excluded from license check: pkg:githubactions/actions/github-script

OpenSSF Scorecard

PackageVersionScoreDetails
actions/actions/cache/restore 4.*.* 🟢 6.4
Details
CheckScoreReason
Maintained🟢 1013 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10
Code-Review🟢 10all changesets reviewed
Binary-Artifacts🟢 10no binaries found in the repo
Packaging⚠️ -1packaging workflow not detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies🟢 3dependency not pinned by hash detected -- score normalized to 3
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Security-Policy🟢 9security policy file detected
Branch-Protection⚠️ 1branch protection is not maximal on development and all release branches
SAST🟢 10SAST tool is run on all commits
actions/actions/checkout 6.*.* 🟢 7
Details
CheckScoreReason
Code-Review🟢 10all changesets reviewed
Maintained🟢 1024 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10
Binary-Artifacts🟢 10no binaries found in the repo
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Packaging⚠️ -1packaging workflow not detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Pinned-Dependencies🟢 3dependency not pinned by hash detected -- score normalized to 3
Security-Policy🟢 9security policy file detected
SAST🟢 10SAST tool is run on all commits
Branch-Protection🟢 6branch protection is not maximal on development and all release branches
actions/actions/setup-java 5.*.* 🟢 5.4
Details
CheckScoreReason
Maintained🟢 1030 commit(s) and 2 issue activity found in the last 90 days -- score normalized to 10
Code-Review🟢 3Found 9/26 approved changesets -- score normalized to 3
Binary-Artifacts🟢 10no binaries found in the repo
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Packaging⚠️ -1packaging workflow not detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
License🟢 10license file detected
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
Signed-Releases⚠️ -1no releases found
Fuzzing⚠️ 0project is not fuzzed
Security-Policy🟢 9security policy file detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
SAST🟢 10SAST tool is run on all commits
actions/actions/upload-artifact 7.*.* 🟢 4.5
Details
CheckScoreReason
Code-Review🟢 10all changesets reviewed
Pinned-Dependencies⚠️ -1no dependencies found
Maintained⚠️ 00 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0
Binary-Artifacts🟢 10no binaries found in the repo
Token-Permissions⚠️ -1No tokens found
Dangerous-Workflow⚠️ -1no workflows found
Packaging⚠️ -1packaging workflow not detected
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
Signed-Releases⚠️ -1no releases found
Security-Policy⚠️ 0security policy file not detected
SAST🟢 10SAST tool is run on all commits
Branch-Protection⚠️ 0branch protection not enabled on development/release branches

Scanned Files

  • .github/workflows/streams-seam-on-evidence.yml

@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

❌ Duplicate Code Report

Two engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check.

✅ PMD CPD

PR Base Change
Clones 31 30 🫤 +1
Duplicated lines 1015 963 :face_with_raised_eyebrow: +52
Duplication 0.13% 0.12% 🫤 +0.01%
Rule Limit Status
Max duplication 0.5% ✅ Pass (0.13%)
Max increase vs base +0.1% ✅ Pass (+0.01%)
⚠️ 1 new clones introduced
  • 26 lines (109 tokens): parallel-consumer-examples/parallel-consumer-example-streams/src/test/java/bz/stub/parallelconsumer/examples/streams/integrationTests/StockBaselineFixtureSupport.java:335 <-> parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/BaselineFixture.java:141

❌ jscpd (language-agnostic)

PR Base Change
Clones 103 91 :face_with_monocle: +12
Duplicated lines 1538 1294 :face_with_raised_eyebrow: +244
Duplication 1.10% 0.99% :face_with_monocle: +0.11%
Rule Limit Status
Max duplication 2% ✅ Pass (1.10%)
Max increase vs base +0.1% ❌ FAIL (+0.11%)
⚠️ 12 new clones introduced
  • 16 lines: parallel-consumer-examples/parallel-consumer-example-streams/src/test/java/bz/stub/parallelconsumer/examples/streams/integrationTests/StockBaselineFixtureTest.java:69 <-> parallel-consumer-examples/parallel-consumer-example-streams/src/test/java/bz/stub/parallelconsumer/examples/streams/integrationTests/StockStatefulBaselineFixtureTest.java:95
  • 10 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/RebalanceUnderPcDispatchTest.java:14 <-> parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/StreamsBenchmarkHarness.java:17
  • 11 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenStreamsDispatchTest.java:7 <-> parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/ShadowedStreamsControlTest.java:6
  • 14 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenStreamsDispatchTest.java:17 <-> parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/ShadowedStreamsControlTest.java:15
  • 9 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenStatefulProofTest.java:10 <-> parallel-consumer-examples/parallel-consumer-example-streams/src/test/java/bz/stub/parallelconsumer/examples/streams/integrationTests/StockStatefulBaselineFixtureTest.java:11
  • 11 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenStatefulProofTest.java:22 <-> parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenStreamsProofTest.java:13
  • 8 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenProofSupport.java:9 <-> parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/ShadowedStreamsControlTest.java:7
  • 70 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PaymentAuthorisationBenchmarkTest.java:10 <-> parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/WorkloadMatrixBenchmarkTest.java:9
  • 56 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PartitionScalingBenchmarkTest.java:1 <-> parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/WorkloadMatrixBenchmarkTest.java:1
  • 14 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/CommitFrontierCrashRestartTest.java:15 <-> parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/StreamsBenchmarkHarness.java:17
  • 26 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/BaselineFixture.java:141 <-> parallel-consumer-examples/parallel-consumer-example-streams/src/test/java/bz/stub/parallelconsumer/examples/streams/integrationTests/StockBaselineFixtureSupport.java:335
  • 11 lines: parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/BacklogCatchUpBenchmarkTest.java:7 <-> parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PaymentAuthorisationBenchmarkTest.java:6

Powered by astubbs/duplicate-code-cross-check

Row(final String key, final String value, final long timestamp) {
this.key = key;
this.value = value;
this.timestamp = timestamp;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Duplicate code detected - 26 lines duplicated with parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/BaselineFixture.java:141

@github-actions

Copy link
Copy Markdown

🧪🔒 Quarantine Lane Report

Quarantined test Outcome Owner Meaning
ProducerManagerTest.producedRecordsCantBeInTransactionWithoutItsOffsetDirect 🔴 failing (expected) #262 quarantine holding

🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its @Quarantined annotation + docs/quarantined-tests.md entry (a merge-blocking review thread has been opened). Lane: non-gating; rules: see the Quarantine Audit check.

…a developer's local repo

`-pl .,parallel-consumer-streams` builds neither `parallel-consumer-core` nor its tests jar, and this
module needs both. On a developer box they are already installed, so the lane ran; the first CI run
failed in three seconds on an unresolved `0.6.0.0-SNAPSHOT`, which is the useful direction for that
mistake to fail in - it is the same class as a check that passes because of something only the
author's machine has.

`-am` instead, which is what `bin/chaos-test.sh` already does for the same reason, and which also
brings the parent in so the enforcer's reactor-convergence rule is satisfied without naming the root.
Core's own tests are filtered out by the `-Dincluded.groups` the lane already passes, so the extra
module costs a compile rather than a suite.

Re-run locally after the change: both arms produced, every divergence explained, exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 31, 2026 •

Copy link
Copy Markdown

⚠️ SpotBugs Report

445 bug(s) found (rule-level exclusions only - see docs/inflight/static-spotbugs-rule-registry.md). See the annotations on the Files Changed tab for details.

astubbs and others added 6 commits September 1, 2026 10:54
…f-master list

The workstream signpost lists what master lacks so an agent listing this directory can see the shape
of the work. This rung removes its own entry rather than leaving a reader to infer it from a PR
number, which is the same discipline the rungs below used.

What is left on that list is now all semantics - refusal is already down, lifecycle is the parent,
and what remains is stream time, error surfacing and the example module. The distinction is worth
stating rather than implying: master will lack the behaviour, not the means of measuring it, so the
next rung arrives with a lane that can already tell it whether it changed anything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…m that did not reproduce it

`PcDrivenStatefulProofTest.pcDrivenAggregationMatchesTheStockBaseline` timed out on repetition 2 of 3
in the Integration Tests lane - the first run that put that class on CI. One sighting, written down
rather than re-run past.

WHAT RULES OUT THE COMFORTABLE READING

The StreamThread's own progress line, logged as the drain gave up, reports 1207 polls against 14
completions in two minutes. A thread short of CPU polls LESS; this one went round its loop briskly and
finished almost nothing, which is the shape of work dispatched and not completed rather than of a
starved machine. The ambient probe was clean - no rebalance dwell, no lag stagnation, no frozen
partitions - so the group progressed at the broker while the task did not progress at the processor.
That is why "the runner was busy" is not available here, however busy it was, and why nothing has been
loosened, shortened or serialised.

THE CONTROL ARM, AND THE TERM IT COULD NOT VARY

Run rather than guessed: the whole streams integration suite locally at the lane's own settings,
forkCount=4 with reuseForks=true, alongside core's integration suite - green, including this class.
Uncontended at forkCount=1, green in three separate runs.

The instrumentation is confirmed rather than assumed. Failsafe pins no <forkCount> of its own, unlike
surefire where the root pom's <forkCount>${surefire.forkCount}</forkCount> makes a bare -DforkCount a
silent no-op, so the property does arrive - and the autopsy dump from the CI failure shows the lane
running with the same two values.

So contention is NOT established and is not being claimed. The one term the local arm cannot vary is
core count: four JVMs and four brokers on a developer machine is not four JVMs and four brokers on
two. The note names the experiment that would settle it - the same runner at forkCount=1, which holds
the machine fixed and varies only the load - and records the two exception lines that would explain
the rest immediately if they belong to this fork, unattributed because four forks share one
interleaved log and a class that interrupts workers by design was running in the same window.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…PR has landed

`bin/check-branch-self-reference.sh` caught the new note citing the pull request that is carrying it.
The citation is right and stays - a merged PR is a permanent link, and it is the shortest way to say
which change first ran that class on CI - so the fix is to word it as it will read afterwards and
confirm that with the `post-merge: checked` markers, the same treatment the invalid-timestamps flake
note already carries.

One sentence did need rewriting rather than confirming: "the lane re-runs on every push to that
branch" describes something that stops existing at the merge. It now names the triggers that outlive
it - every push and every merge to master - and says why a second sighting is the cheap next step: one
occurrence cannot tell a rare race from a one-off.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t read differently are not comparable

The duplicate-code check flagged a genuine clone on this PR - `consumeByKey` plus `flatten`, shared
between the seam-on dispatch arm and the stock control arm, byte-identical apart from the deadline.
Both engines agreed on it, which is the tell that it is real rather than jscpd counting a shared
import block.

It is worth fixing rather than explaining away for a reason specific to this module: these two are a
matched experiment. The whole method here is comparing arms that differ in exactly one term, and a
copy of the reading logic in each is a second term waiting to appear the first time somebody tunes
one of them. Drifted topic-reading logic is already a named source of flaky CI in this repository.

Now on `BrokerStreamsIntegrationTest`, which both extend and which already holds precisely this kind
of thing - the same broker, the same serdes, the same starting offset. The DEADLINE stays a parameter
at each call site, because how long an arm waits is a property of the workload it runs and that class
deliberately carries only what the arms must agree on. Its javadoc says which two arms produced the
clone, so the next reader knows why it is shared rather than inlined.

Verified rather than assumed: the whole streams integration suite green afterwards, both sides of the
clone included.

WHAT WAS LOOKED AT AND DELIBERATELY LEFT

The other clone both engines agree on is the fixture FORMAT - the writer in
`StockBaselineFixtureSupport` against the reader in `BaselineFixture`. Those two live in modules that
cannot share code, and that is the point rather than an oversight: the example module must not depend
on the streams module, or the baseline it generates would run the patched classes and stop being a
baseline. Extracting it would need a third module whose only purpose is to reconnect the two things
this design keeps apart.

The remaining jscpd hits are import blocks and javadoc between sibling benchmark classes - jscpd is
lexical and counts comments, which is why PMD CPD, which is token-based, reports neither.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… and this is what it caught

`StreamThreadTest.shouldChangeStateAtStartClose[1]` failed in the SEAM-OFF arm on a GitHub-hosted
runner - "Condition not met within timeout 10000. Thread never shut down." No prior art anywhere in
this repository: not in the ledger, not in the solutions, not on the tracker, open or closed.

THE LANE DID THE RIGHT THING AND IT HAD NEVER DONE IT BEFORE

The control-arm integrity check had only ever been exercised on synthetic reports. On this run it
named the ledgered invalid-timestamps flake beside this failure, refused to produce a verdict on the
divergence set while an UNdiagnosed control-arm failure stood, and said so in a verdict that outranks
the classification - which was itself clean, no unexplained divergences at all. That is the fourth
way this lane has now been shown to fail, and the only one that arrived by itself.

THE DIAGNOSIS, AND IT IS ONE SIGHTING

The same case PASSED in the seam-ON arm of the same run. That is close to a control arm handed over
for free: two arms, same patched classes, same runner, minutes apart, differing only in the dispatch
switch. A defect the patch introduced cannot be REPAIRED by turning the seam on - seam-on is strictly
more machinery on top of the same patched code. What varies with load, and only with load, is whether
a thread finishes shutting down inside a fixed ten seconds.

Recorded with a flaky-case marker, and the note is explicit that this rests on one sighting plus that
asymmetry, that a failure in BOTH arms of one run would void it, and that quarantine is not the lever
here - it is one sighting on one PR, and the case lives inside Apache Kafka's own compiled test class
where the annotation has nowhere to go.

WHY THE MARKER IS NOT A SILENCER, SINCE IT DOES GREEN THIS LANE

The lane still prints the failure, still names the note beside it, and still refuses to attribute it
to the seam. What the marker buys is that one timing bound in Kafka's own test stops withholding the
verdict on the divergence set - otherwise the lane is permanently red for a reason that is not a
divergence, and a permanently red lane is one nobody reads, which is the failure this whole design
was built against.

ONE MORE THING THE RUN TAUGHT THE TRIAGE NOTE

The lane reported a divergence attribution as having matched nothing, which normally means a rung
fixed it. There is a benign second cause, now written down: a case failing in the CONTROL arm falls
out of the divergence set entirely, because a divergence is green-off-and-red-on. The
invalid-timestamps case does exactly that whenever its flake fires seam-off. So check which list a
case is in before deleting an entry the lane called stale.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…so it is intermittent

The same lane, the same code, a later push on the same PR: the whole streams integration suite passed
including the class that stalled. Recorded because a green re-run is a data point about the rate and
nothing at all about the cause, and this note exists so the next green one is not mistaken for a
diagnosis.

What it does change is the priority of the experiment the note already names. A standing break would
be reproducible on demand; an intermittent one is exactly the case where re-running until green is
tempting and useless, so the core-count arm - the same runner at forkCount=1, holding the machine
fixed and varying only the load - is now the only thing that will settle it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
astubbs added a commit that referenced this pull request Sep 1, 2026
Merges `feats/ks-streams-reconciled` - the reconciliation of #395,
#396, #398 and #391 on top
of the refreshed #394 - into this branch, so that
#271 can be retargeted onto that rung and its displayed diff collapse to the
residue. This is the plan's "retarget move",
`docs/plans/2026-08-31-001-process-god-branch-decomposition-plan.md`, and it is an ordinary merge: no
history was rewritten and nothing was force-pushed. The pre-merge tip is preserved as
`origin/backup/pre-stack-merge-271`.

The merge also brings `master` forward, because the stack is current with it and this branch was
not. That is most of the volume and all but a handful of the conflicts.

## Which side won, and the check that made it safe to say so

The rule was: the rungs' refined versions win wherever both sides have a file; the spike's residue is
kept. **The second half of that is only safe if the spike contributed nothing to the shared files,
and that was measured rather than assumed.** Both trees were exported with package paths normalised
(`io.confluent.parallelconsumer` and `bz.stub.parallelconsumer` folded to one token, and
`io.confluent.csid.utils` to its new home) and compared file by file against the branch's own merge
base. The answer is clean: outside `parallel-consumer-streams` and the streams example, **every
difference this branch carries is the package rename itself plus a copyright-header line** - the same
rename `master` performed independently, which is why taking the rungs' side loses nothing. The one
substantive exception is the streams example's stock-baseline fixtures, which are already on the
stack because #398 depends on them.

- **Files on both sides: the rungs' version, 93 of them.** They carry current master, the rename, and
  every fix the five rungs landed.
- **Files only on the spike: kept, 32 of them.** The plan and result documents, the branch handover,
  the ranked next-work notes, the eight `docs/solutions/` write-ups, and two streams tests the stack
  deliberately left behind - `HeadOfLineBlockingBenchmarkTest`, which
  #398 says it is leaving on the forest, and
  `ProcessorContextConfinementTest`. That residue is what this PR still exists to review.
- **Files master deleted: they stay deleted, 12 of them** - `.semaphore/`, `.travis-archived.yml`,
  `service.yml`, two `.idea` run configurations, the uppercase `docs/` names master lowercased, and
  the `InternalRuntimeException` pair master renamed to `PCInternalRuntimeException`.
- **Two shared ledgers merged as unions, not resolved to a side**:
  `docs/inflight/pr-strategy-doc-merge-triggers.md` and `docs/inflight/release-0.6.0.0.md` keep this
  branch's paragraphs alongside master's.

**The strongest check is the one that makes a mis-paired rename impossible to hide.** Rather than
counting recorded renames, the merged tree was differenced against `feats/ks-streams-reconciled`
directly: the only differences are the 32 additions and the two union-merged ledgers. Nothing else
survived on the wrong side, in the wrong module, or under an `io/confluent` path -
`git ls-files | grep io/confluent` is empty, and the prescribed `grep -rnE 'io[\./]*conflu'` sweep
returns only prose in dated documents.

**Both sides were already renamed, which is not what the plan predicted.** It expected the spike to be
pre-rename; `origin/feats/ks-on-pc-spike` had in fact moved five commits past the worktree's copy and
carries the rename. That is the good case AGENTS.md describes - the mis-pairing surfaces as
rename/rename conflicts on the right files instead of silently applying one module's edit to another -
and it did: three `TestConventionsArchTest.java` files were offered paired across modules, and the
right file was taken into each module rather than accepting git's pairing.

## One collision the merge produced, and the reason it is a good one

`HeadOfLineBlockingBenchmarkTest` carried a private `sleep(Duration)` helper. Since
#395 hoisted the identical helper onto `BrokerStreamsIntegrationTest` -
because two arms that simulate cost differently are not comparable - the two now collide, and the
compiler said so rather than either winning silently. The local copy is deleted and the shared one
inherited, which is what the hoist was for.

## The gates, and the one that is a scoping artefact

`bin/check-all.sh` was run in this worktree. Four gates failed on first pass and **three of the four
did not exist on this branch before the merge** - they arrived with master, and every finding was on
content that predates them.

- **`check-inflight-tags`** flagged sixteen notes with no `inflight-type`, all of them this branch's
  own and all older than the tag scheme. Tagged - a first pass a reviewer should correct where it
  reads wrong. `bug-core-tests-jar-junit-parallelism-leak.md` additionally carries a `closed` state,
  because #265 removed the file it is about.
- **`check-file-refs`** flagged citations broken by the rename - the exact class
  [`docs/citations.md`](docs/citations.md) says must be repaired rather than left. Forty-seven paths
  were re-pointed to their successors, **each verified to exist before the edit was written**; the
  claims around them are untouched, which is the line that document draws. What genuinely has no repo
  path - Apache Kafka's own sources inside the published jar, a placeholder in a shell recipe, a class
  that was only ever proposed - carries a line-scoped `file-refs: N/A` with its reason. Two targets
  that moved out of the tree point at the history holding them instead.
- **`check-issue-refs`** flagged two bare `#NNN` in the handover; qualified.
- **`check-branch-self-reference`** is the artefact, and the gate's own header names this exact
  situation: mid-merge the merge base is still the old one, so master's sentences read as this
  branch's. Twenty-five of the twenty-eight files it flagged are on `feats/ks-streams-reconciled` too
  and twenty-four are byte-identical to it, so marking them would be attesting to somebody else's
  notes - which that header explicitly says not to do. Committing the merge is the documented fix.

## Verified

JDK 17, macOS. The whole reactor compiles, main and test. The reconciled branch it merges was verified
in its own right before this: seam-off oracle unchanged, module unit suite green, and all 35
broker-backed integration arms green against a real broker.
@astubbs astubbs added the 0.6.0.0 Targeted at the 0.6.0.0 release label Sep 1, 2026
@astubbs astubbs added this to the 0.6.0.0 milestone Sep 1, 2026
@astubbs
astubbs marked this pull request as draft September 2, 2026 14:48
@astubbs astubbs removed the 0.6.0.0 Targeted at the 0.6.0.0 release label Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant