Repository navigation
test(streams) astubbs#255: the evidence suite - a seam-on divergence lane, the broker-backed laws, and a benchmark with a noise floor - #398
Conversation
…rgence nobody has named The seam-off execution has always been the behaviour-preservation gate: the patch changes nothing while the switch is off. This adds its opposite - what Apache Kafka's own suite says about the path this module actually ships - and, more usefully, a verdict on it. THE PROBLEM WITH A SEAM-ON RUN, AND THE SHAPE THAT SOLVES IT Seam-on failures are expected. The refusal envelope throws where stock would have run a construct; the committed offset is Parallel Consumer's frontier rather than Streams' consumedOffsets; a record handed to a worker has not been processed by the time process() returns. A gate that goes red every build for those reasons is a gate people stop reading, which is why the branch forest's version of this run recorded its failures and gated on nothing. So the run and the verdict are SEPARATE surefire executions. kafka-upstream-tests-seam-on carries testFailureIgnore and never reddens a build; seam-on-evidence-classification reads what it wrote and fails when a divergence matches no named mechanism. That split is the only arrangement in which "these failures are expected" and "a new divergence fails the build" are both true. THE SEAM-OFF RUN IS THE CONTROL ARM, AND IT IS DIFFERENCED CASE BY CASE A divergence is a case that PASSES with the switch off and FAILS with it on. Nothing else is attributable to the dispatch path: a case failing in both arms is failing for a reason the seam had nothing to do with, and counting it would credit a change that did not cause it. The classifier therefore reports four disjoint sets - divergences, failing-in-both, passing-only-seam-on, and the control arm's own failures - rather than one number. MECHANISMS ARE DERIVED FROM THE CODE WHERE THAT IS POSSIBLE RefusedConstruct takes its marker as the longest common prefix of every message PcUnsupportedConstruct can produce, and attributes a failure to whichever constant the message names. No refusal string is copied, so the refusal envelope - which belongs to another rung and moves independently - can gain or lose a construct with no edit in this lane at all. CommitFrontierEncoding recognises an assertion that renders an OffsetAndMetadata, which is an assertion about the commit encoding whatever the case is called. Its limitation is stated rather than hidden: an offset REGRESSION inside a Kafka unit test renders identically and would be explained by this mechanism too. Nothing in the assertion text separates the two, so the lane does not pretend to - what defends offset correctness is a broker-backed law, not this. WHAT CANNOT BE DERIVED IS ATTRIBUTED IN THE LEDGER, NOT HERE "This assertion fails because Kafka's test drives the task synchronously" is a judgement about a test, not a fact recoverable from `expected: <5> but was: <0>`. Those attributions live in docs/inflight/test-streams-seam-on-divergence-triage.md, keyed by method rather than by parameterisation, and the lane reads them. Three consequences, all deliberate: the reasoning is reviewed as prose by whoever changes the semantics under it; the PR that closes a mechanism deletes its entries in the same change; and an entry that stops matching is reported as stale rather than rotting silently. THE FLAKE IS ABSORBED BY MECHANISM RATHER THAN BY A RETRY StreamThreadTest.shouldLogAndRecordSkippedRecordsForInvalidTimestamps has been failing this module's oracle intermittently for weeks, and its note says so. Because the oracle fails fast, one sighting used to abort the lane before the measured arm ran at all - a full re-run of Kafka's suite twice, for a failure already diagnosed. The lane now relaxes THAT ONE EXECUTION and makes the verdict itself: every control-arm failure must carry a flaky-case marker in docs/inflight/, or the lane stops and says there is nothing trustworthy to difference against. The ordinary build's oracle is untouched and still fatal. The flake registry is consulted FIRST on the control arm and LAST among the divergence mechanisms, and that ordering is load-bearing - see the triage note's own finding. PROVEN ABLE TO FAIL, THREE WAYS, WITH THE PREDICTION STATED FIRST Sabotage, end to end. Removing the close-time commit-state publication from PcTaskDispatcher.close() was predicted to make exactly one case, shouldClearCommitStatusesInCloseDirty, fail seam-on while staying green seam-off, with no mechanism recognising a bare commitNeeded assertion. Exactly that: one unexplained divergence, named, with its recorded failure and stack; every other divergence still explained by the same six mechanisms; the script exited 1. Reverted and verified byte-identical by checksum. That run also demonstrated the control-arm path unplanned - the invalid-timestamps flake fired seam-off, and the lane named it from its ledger note and carried on instead of aborting. A stale report directory. Running the classification execution alone against report directories from an earlier run failed on the freshness check, naming the file, when it was written and when the build started. maven.build.timestamp is passed in for exactly this, and is required rather than optional: a check that turns itself off when its input is missing is the failure mode this lane exists to avoid. A missing report directory. The same execution against deleted directories failed with "That is an ERROR and not an empty result", rather than reading an empty glob as zero failures. SeamOnDivergenceClassifierTest pins all three at unit level on synthetic reports, plus the cases the end-to-end runs cannot cheaply reach: an arm whose pin did not arrive (both arms recording dispatch=false differences the control against itself and reports a flawless zero), arms covering different classes, an attribution to an undeclared class, and a triaged attribution beating a flake marker on the same case. NO COUNT APPEARS IN THE MACHINERY OR IN THE LEDGER. The per-arm totals are printed for a reader and the verdict does not consult them; which cases diverge moves with the branch, the Kafka version and the seam, and the command that re-derives them is in the note and in the README. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot a local-only claim A lane that only ever runs on the machine of whoever wrote it is half a deliverable: the first person to change the dispatch path's semantics learns nothing, and the ledger of attributions rots with nobody to notice. RED HERE IS NOT EXPECTED, WHICH IS WHAT MAKES IT DIFFERENT FROM THE QUARANTINE LANE That lane runs tests known to be failing, so its verdict lives in the step summary and its run step carries continue-on-error. This one asserts that every seam-on divergence is EXPLAINED, and green is the ordinary state. Red means one of exactly two things, both wanting a human rather than a re-run: a divergence nobody has attributed - something changed the dispatch path's semantics - or an attribution in docs/inflight/ that went stale because a rung fixed the thing it described. So the job gates. PATH-SCOPED, AND THE LEDGER IS IN THE TRIGGER SET The run costs Kafka's whole suite twice, and nothing outside the streams module, the lane script and docs/inflight/ can change its answer. The ledger belongs in that set on purpose: an edit to an attribution changes the verdict exactly as much as an edit to the code does, and a triage note edited without re-running is how a lane starts lying. The whole measurement goes to the step summary and is kept as an artifact with both arms' surefire reports, so a red run can be read without re-running it. A red on the dependency fetch is a download failure rather than a divergence, and the step says so: this module pulls Kafka's sources and test-sources CLASSIFIER artifacts by explicit artifactItems, which dependency:go-offline does not resolve, so the cache-warming job does not cover them. That gap is already recorded in docs/inflight/, and this workflow points at it rather than restating it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… way to state Apache Kafka's own suite is the oracle for behaviour PRESERVATION. It cannot state the properties this module exists for, because stock Kafka Streams has no arm in which two records of one partition run at once - so no stock test asserts what happens when they do. These are the arms that do. WHAT IS NOW ASSERTED AGAINST A BROKER, AND WITH WHAT CONTROL PcDrivenStreamsProofTest - the PC path's output is identical, record for record, to what stock Kafka Streams produced. Two controls, and they answer different questions: a same-harness seam-off arm, which separates "the patch changed the output" from "concurrency changed the output"; and the stock baseline itself, which is generated OUTSIDE this module. PcDrivenStatefulProofTest - the same for a stateful topology, which is where Kafka's own store wrappers read the per-task record context ambiently. The stateless arm's only ambient reader is the probe this module wrote; this one makes Kafka's own run. The module's existing envelope test already said this proof was the reason non-windowed aggregation is not refused - and the proof was not on this stack, so that sentence was pointing at nothing. CommitFrontierCrashRestartTest - nothing is lost or double-counted across a crash. The committed frontier is a point below which every record has been processed, not the offset of the last one handed out, and under concurrent dispatch those are different records. ShadowedStreamsControlTest - stock, single-threaded Kafka Streams running our generated classes. It is itself a control: a concurrency change that quietly breaks the serial path is not a fix. THE BASELINE IS GENERATED IN ANOTHER MODULE, AND THAT IS THE WHOLE POINT This module compiles patched copies of Kafka's classes into its own build output, which precedes the kafka-streams jar on the classpath, so EVERY KafkaStreams instance in this JVM runs the patch. A "stock arm" written here would share every defect the patch introduced and the comparison would prove nothing. The baselines come from parallel-consumer-example-streams, which does not depend on this module and asserts its own classpath is stock before recording a row. The fixture carries the INPUTS as well as the outputs, and the arms here replay those inputs rather than rebuilding them from a second copy of the generation rules - the two modules cannot share code in this reactor order, so replaying is what stops the arms drifting apart in what they were fed. Re-verified here rather than trusted: both fixture tests were run on this branch and the tracked fixtures still match what genuinely-stock Kafka Streams produces, so the baselines describe this Kafka version and not the branch they were recorded on. THE LANE ABOVE DEFERS TO ONE OF THESE, AND SAYS SO The seam-on evidence lane's commit-frontier-encoding mechanism cannot tell a deliberate encoding divergence from an offset regression: both render an OffsetAndMetadata in a Kafka unit test's assertion and nothing in that text separates them. Its javadoc names CommitFrontierCrashRestartTest as what actually defends offset correctness, because that arm asserts on records that arrived rather than on a base64 metadata string. An acknowledged blind spot with a named owner is worth more than a predicate pretending to cover it. WHERE THIS CAME FROM, AND WHAT WAS DELIBERATELY LEFT Copied out of #271's branch forest, package-renamed io.confluent to bz.stub, compiled and run unchanged otherwise. HeadOfLineBlockingBenchmarkTest is NOT carried across: it isolates head-of-line blocking on a fixture whose own construction bounds its multiplier, and the settled decision is to lead with backlog catch-up instead. The two files that referenced it now say so rather than pointing at a class that is not here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… that says when to ignore it A card issuer screens every authorisation against a fraud-scoring service. The screener has been down, authorisations have queued, and it has just come back. That is the workload: the canonical Kafka Streams enrichment shape, keyed the way a real deployment would have to key it, with a skewed keyspace and a heavy-tailed service cost - the two things that cost this seam the most. Three arms, not two, and the third is the point of this change. THE NOISE FLOOR, MEASURED INSIDE THE RUN THAT QUOTES IT The extra arm is a second STOCK arm, given the same records. Whatever it differs from the first by is this machine in this state, and that is an upper bound on how far a ratio can move with nothing changed at all. Without it on screen a run taken on a busy machine looks exactly like a finding - and that is not hypothetical, it is what #391 found on the example rung, where a run reading 1.19x on its control had two stock arms differing by 1.20x. The control had measured nothing and nothing said so. So the headline assertion no longer asks whether the ratio beat one. It asks whether the ratio beat the floor by a margin. A direction check is a sound test of an arm that loses and a coin toss for an arm at parity, and - the reason this rule exists - it is satisfied by a run in which the mechanism did nothing at all. PROVEN ABLE TO FAIL, AND WITH A DOCUMENTED FLAG RATHER THAN A CODE EDIT Prediction, stated first: with a worker pool of one, PC dispatch has no concurrency to exploit, so the seam-on arm should land inside the noise floor and the headline assertion should fail while the run stays otherwise healthy. Exactly that. `bin/streams-benchmark.sh --pool 1` failed one case - the headline - and its message carried the whole diagnosis: measured 0.99x against a floor of 1.00x, threshold 1.51x, with both stock arms and the PC arm printed beside it. The steady-state case, which asserts on sample size rather than on a ratio, stayed green. That asymmetry is what says the right thing broke. `--pool 1` reaching the harness at all is a fix in this commit. The script has always documented `--pool N`, and the harness hardcoded four - so a reader could ask for a configuration, be told in every log line that they had got it, and be measuring something else. The same class of failure the script's own "an unknown option is an error" rule was written against, arriving by the other route. The margin rule itself is pinned by NoiseFloorTest, which runs in the ordinary build because it is a pure function of two numbers and should not need a broker or a `performance` tag to be exercised. Its first case is the one that matters: a 1.01x observation does not beat even the tightest possible floor. That is the shape of a sabotaged run, and a direction check passes it. THE SCRIPT RAN NOTHING ON THIS LAYOUT, AND WOULD HAVE EXITED 0 Carried across unchanged, `bin/streams-benchmark.sh` drove FAILSAFE with `-Dit.test` and `-DskipUTs`. These benchmarks live in `src/test/java` and this module has no `src/test-integration`, so that selects nothing, runs nothing, and succeeds. It now drives surefire - and because `failIfNoSpecifiedTests` has to be off for the Kafka upstream executions' sake, maven can no longer report an empty selection, so the script asserts a surefire report exists for every class it selected and exits 2 when one does not. A skip is never a pass. NO FIGURE FROM AN EARLIER RUN SURVIVES IN THIS DIFF The forest's numbers were measured on another machine and describe it, so the headline multipliers, the wake-on-work ablation percentage and the CPU-bound refutation are all removed from the javadoc or attributed to the branch they were taken on. `DEMO.md` shows the report's SHAPE with the numbers elided, and says why: a figure written into documentation quietly stops describing the reader's machine. The run prints its own, and prints the floor beside them. Two method write-ups come with the suite because its citations were pointing at them - choosing the statistic that states the claim, and control arms varying exactly one term. Both are records of work done on the branch forest, so their citations into that forest's plan and benchmark carry paragraph markers rather than being repointed at files that would make the record describe a run that never happened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… it pulled Docker into the unit lane Naming the seam-on classifier in this module's `default-test` execution silently dropped the two excludes the root pom sets on surefire - `**/integrationTest*/**/*.java` and `**/*IT.java` - because an `<excludes>` list REPLACES the inherited one rather than adding to it. The effect was invisible in the only way that matters: everything passed. Every broker-backed arm in this module lives in an `integrationTests` package, so they were quietly promoted out of failsafe and into the surefire `test` phase, where they ran, needed Docker, and went green. A unit lane that has started requiring Docker does not announce it; it just takes longer until the day it runs somewhere without a daemon. Caught by reading the run rather than the tick: the module's `test` phase reported far more cases than its unit suite has, and the class names in the list were the integration ones. Fixed by repeating the root's two entries beside the new one - `combine.children="append"` would express the intent, but a literal list is what a reader can check against the root pom without knowing that attribute exists. Verified both directions: the `test` phase now runs the unit suite alone with the seam-off oracle still at its usual figures, and `verify` runs every broker-backed arm including the ones this stack adds. The generated stock-baseline fixtures are classified in the copyright scanner as regenerated-by-a- generator, which is the first of the four reasons that table accepts and the same one `*.patch` carries. They are rewritten wholesale by `StockBaselineFixtureSupport`, so a header would be destroyed on the next run. Two explicit paths rather than a blanket `*.tsv`: the next `.tsv` to arrive should be classified on its own merits, not inherit an exemption written for these. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dependency ReviewThe following issues were found:
License Issues.github/workflows/streams-seam-on-evidence.yml
OpenSSF Scorecard
Scanned Files
|
❌ Duplicate Code ReportTwo engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check. ✅ PMD CPD
|
| PR | Base | Change | |
|---|---|---|---|
| Clones | 103 | 91 | :face_with_monocle: +12 |
| Duplicated lines | 1538 | 1294 | :face_with_raised_eyebrow: +244 |
| Duplication | 1.10% | 0.99% | :face_with_monocle: +0.11% |
| Rule | Limit | Status |
|---|---|---|
| Max duplication | 2% | ✅ Pass (1.10%) |
| Max increase vs base | +0.1% | ❌ FAIL (+0.11%) |
⚠️ 12 new clones introduced
- 16 lines:
parallel-consumer-examples/parallel-consumer-example-streams/src/test/java/bz/stub/parallelconsumer/examples/streams/integrationTests/StockBaselineFixtureTest.java:69<->parallel-consumer-examples/parallel-consumer-example-streams/src/test/java/bz/stub/parallelconsumer/examples/streams/integrationTests/StockStatefulBaselineFixtureTest.java:95 - 10 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/RebalanceUnderPcDispatchTest.java:14<->parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/StreamsBenchmarkHarness.java:17 - 11 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenStreamsDispatchTest.java:7<->parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/ShadowedStreamsControlTest.java:6 - 14 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenStreamsDispatchTest.java:17<->parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/ShadowedStreamsControlTest.java:15 - 9 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenStatefulProofTest.java:10<->parallel-consumer-examples/parallel-consumer-example-streams/src/test/java/bz/stub/parallelconsumer/examples/streams/integrationTests/StockStatefulBaselineFixtureTest.java:11 - 11 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenStatefulProofTest.java:22<->parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenStreamsProofTest.java:13 - 8 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PcDrivenProofSupport.java:9<->parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/ShadowedStreamsControlTest.java:7 - 70 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PaymentAuthorisationBenchmarkTest.java:10<->parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/WorkloadMatrixBenchmarkTest.java:9 - 56 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PartitionScalingBenchmarkTest.java:1<->parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/WorkloadMatrixBenchmarkTest.java:1 - 14 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/CommitFrontierCrashRestartTest.java:15<->parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/StreamsBenchmarkHarness.java:17 - 26 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/BaselineFixture.java:141<->parallel-consumer-examples/parallel-consumer-example-streams/src/test/java/bz/stub/parallelconsumer/examples/streams/integrationTests/StockBaselineFixtureSupport.java:335 - 11 lines:
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/BacklogCatchUpBenchmarkTest.java:7<->parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/PaymentAuthorisationBenchmarkTest.java:6
Powered by astubbs/duplicate-code-cross-check
| Row(final String key, final String value, final long timestamp) { | ||
| this.key = key; | ||
| this.value = value; | ||
| this.timestamp = timestamp; |
There was a problem hiding this comment.
parallel-consumer-streams/src/test/java/bz/stub/parallelconsumer/streams/integrationTests/BaselineFixture.java:141
🧪🔒 Quarantine Lane Report
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its |
…a developer's local repo `-pl .,parallel-consumer-streams` builds neither `parallel-consumer-core` nor its tests jar, and this module needs both. On a developer box they are already installed, so the lane ran; the first CI run failed in three seconds on an unresolved `0.6.0.0-SNAPSHOT`, which is the useful direction for that mistake to fail in - it is the same class as a check that passes because of something only the author's machine has. `-am` instead, which is what `bin/chaos-test.sh` already does for the same reason, and which also brings the parent in so the enforcer's reactor-convergence rule is satisfied without naming the root. Core's own tests are filtered out by the `-Dincluded.groups` the lane already passes, so the extra module costs a compile rather than a suite. Re-run locally after the change: both arms produced, every divergence explained, exit 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
…f-master list The workstream signpost lists what master lacks so an agent listing this directory can see the shape of the work. This rung removes its own entry rather than leaving a reader to infer it from a PR number, which is the same discipline the rungs below used. What is left on that list is now all semantics - refusal is already down, lifecycle is the parent, and what remains is stream time, error surfacing and the example module. The distinction is worth stating rather than implying: master will lack the behaviour, not the means of measuring it, so the next rung arrives with a lane that can already tell it whether it changed anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…m that did not reproduce it
`PcDrivenStatefulProofTest.pcDrivenAggregationMatchesTheStockBaseline` timed out on repetition 2 of 3
in the Integration Tests lane - the first run that put that class on CI. One sighting, written down
rather than re-run past.
WHAT RULES OUT THE COMFORTABLE READING
The StreamThread's own progress line, logged as the drain gave up, reports 1207 polls against 14
completions in two minutes. A thread short of CPU polls LESS; this one went round its loop briskly and
finished almost nothing, which is the shape of work dispatched and not completed rather than of a
starved machine. The ambient probe was clean - no rebalance dwell, no lag stagnation, no frozen
partitions - so the group progressed at the broker while the task did not progress at the processor.
That is why "the runner was busy" is not available here, however busy it was, and why nothing has been
loosened, shortened or serialised.
THE CONTROL ARM, AND THE TERM IT COULD NOT VARY
Run rather than guessed: the whole streams integration suite locally at the lane's own settings,
forkCount=4 with reuseForks=true, alongside core's integration suite - green, including this class.
Uncontended at forkCount=1, green in three separate runs.
The instrumentation is confirmed rather than assumed. Failsafe pins no <forkCount> of its own, unlike
surefire where the root pom's <forkCount>${surefire.forkCount}</forkCount> makes a bare -DforkCount a
silent no-op, so the property does arrive - and the autopsy dump from the CI failure shows the lane
running with the same two values.
So contention is NOT established and is not being claimed. The one term the local arm cannot vary is
core count: four JVMs and four brokers on a developer machine is not four JVMs and four brokers on
two. The note names the experiment that would settle it - the same runner at forkCount=1, which holds
the machine fixed and varies only the load - and records the two exception lines that would explain
the rest immediately if they belong to this fork, unattributed because four forks share one
interleaved log and a class that interrupts workers by design was running in the same window.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…PR has landed `bin/check-branch-self-reference.sh` caught the new note citing the pull request that is carrying it. The citation is right and stays - a merged PR is a permanent link, and it is the shortest way to say which change first ran that class on CI - so the fix is to word it as it will read afterwards and confirm that with the `post-merge: checked` markers, the same treatment the invalid-timestamps flake note already carries. One sentence did need rewriting rather than confirming: "the lane re-runs on every push to that branch" describes something that stops existing at the merge. It now names the triggers that outlive it - every push and every merge to master - and says why a second sighting is the cheap next step: one occurrence cannot tell a rare race from a one-off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t read differently are not comparable The duplicate-code check flagged a genuine clone on this PR - `consumeByKey` plus `flatten`, shared between the seam-on dispatch arm and the stock control arm, byte-identical apart from the deadline. Both engines agreed on it, which is the tell that it is real rather than jscpd counting a shared import block. It is worth fixing rather than explaining away for a reason specific to this module: these two are a matched experiment. The whole method here is comparing arms that differ in exactly one term, and a copy of the reading logic in each is a second term waiting to appear the first time somebody tunes one of them. Drifted topic-reading logic is already a named source of flaky CI in this repository. Now on `BrokerStreamsIntegrationTest`, which both extend and which already holds precisely this kind of thing - the same broker, the same serdes, the same starting offset. The DEADLINE stays a parameter at each call site, because how long an arm waits is a property of the workload it runs and that class deliberately carries only what the arms must agree on. Its javadoc says which two arms produced the clone, so the next reader knows why it is shared rather than inlined. Verified rather than assumed: the whole streams integration suite green afterwards, both sides of the clone included. WHAT WAS LOOKED AT AND DELIBERATELY LEFT The other clone both engines agree on is the fixture FORMAT - the writer in `StockBaselineFixtureSupport` against the reader in `BaselineFixture`. Those two live in modules that cannot share code, and that is the point rather than an oversight: the example module must not depend on the streams module, or the baseline it generates would run the patched classes and stop being a baseline. Extracting it would need a third module whose only purpose is to reconnect the two things this design keeps apart. The remaining jscpd hits are import blocks and javadoc between sibling benchmark classes - jscpd is lexical and counts comments, which is why PMD CPD, which is token-based, reports neither. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… and this is what it caught `StreamThreadTest.shouldChangeStateAtStartClose[1]` failed in the SEAM-OFF arm on a GitHub-hosted runner - "Condition not met within timeout 10000. Thread never shut down." No prior art anywhere in this repository: not in the ledger, not in the solutions, not on the tracker, open or closed. THE LANE DID THE RIGHT THING AND IT HAD NEVER DONE IT BEFORE The control-arm integrity check had only ever been exercised on synthetic reports. On this run it named the ledgered invalid-timestamps flake beside this failure, refused to produce a verdict on the divergence set while an UNdiagnosed control-arm failure stood, and said so in a verdict that outranks the classification - which was itself clean, no unexplained divergences at all. That is the fourth way this lane has now been shown to fail, and the only one that arrived by itself. THE DIAGNOSIS, AND IT IS ONE SIGHTING The same case PASSED in the seam-ON arm of the same run. That is close to a control arm handed over for free: two arms, same patched classes, same runner, minutes apart, differing only in the dispatch switch. A defect the patch introduced cannot be REPAIRED by turning the seam on - seam-on is strictly more machinery on top of the same patched code. What varies with load, and only with load, is whether a thread finishes shutting down inside a fixed ten seconds. Recorded with a flaky-case marker, and the note is explicit that this rests on one sighting plus that asymmetry, that a failure in BOTH arms of one run would void it, and that quarantine is not the lever here - it is one sighting on one PR, and the case lives inside Apache Kafka's own compiled test class where the annotation has nowhere to go. WHY THE MARKER IS NOT A SILENCER, SINCE IT DOES GREEN THIS LANE The lane still prints the failure, still names the note beside it, and still refuses to attribute it to the seam. What the marker buys is that one timing bound in Kafka's own test stops withholding the verdict on the divergence set - otherwise the lane is permanently red for a reason that is not a divergence, and a permanently red lane is one nobody reads, which is the failure this whole design was built against. ONE MORE THING THE RUN TAUGHT THE TRIAGE NOTE The lane reported a divergence attribution as having matched nothing, which normally means a rung fixed it. There is a benign second cause, now written down: a case failing in the CONTROL arm falls out of the divergence set entirely, because a divergence is green-off-and-red-on. The invalid-timestamps case does exactly that whenever its flake fires seam-off. So check which list a case is in before deleting an entry the lane called stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…so it is intermittent The same lane, the same code, a later push on the same PR: the whole streams integration suite passed including the class that stalled. Recorded because a green re-run is a data point about the rate and nothing at all about the cause, and this note exists so the next green one is not mistaken for a diagnosis. What it does change is the priority of the experiment the note already names. A standing break would be reproducible on demand; an intermittent one is exactly the case where re-running until green is tempting and useless, so the core-count arm - the same runner at forkCount=1, holding the machine fixed and varying only the load - is now the only thing that will settle it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merges `feats/ks-streams-reconciled` - the reconciliation of #395, #396, #398 and #391 on top of the refreshed #394 - into this branch, so that #271 can be retargeted onto that rung and its displayed diff collapse to the residue. This is the plan's "retarget move", `docs/plans/2026-08-31-001-process-god-branch-decomposition-plan.md`, and it is an ordinary merge: no history was rewritten and nothing was force-pushed. The pre-merge tip is preserved as `origin/backup/pre-stack-merge-271`. The merge also brings `master` forward, because the stack is current with it and this branch was not. That is most of the volume and all but a handful of the conflicts. ## Which side won, and the check that made it safe to say so The rule was: the rungs' refined versions win wherever both sides have a file; the spike's residue is kept. **The second half of that is only safe if the spike contributed nothing to the shared files, and that was measured rather than assumed.** Both trees were exported with package paths normalised (`io.confluent.parallelconsumer` and `bz.stub.parallelconsumer` folded to one token, and `io.confluent.csid.utils` to its new home) and compared file by file against the branch's own merge base. The answer is clean: outside `parallel-consumer-streams` and the streams example, **every difference this branch carries is the package rename itself plus a copyright-header line** - the same rename `master` performed independently, which is why taking the rungs' side loses nothing. The one substantive exception is the streams example's stock-baseline fixtures, which are already on the stack because #398 depends on them. - **Files on both sides: the rungs' version, 93 of them.** They carry current master, the rename, and every fix the five rungs landed. - **Files only on the spike: kept, 32 of them.** The plan and result documents, the branch handover, the ranked next-work notes, the eight `docs/solutions/` write-ups, and two streams tests the stack deliberately left behind - `HeadOfLineBlockingBenchmarkTest`, which #398 says it is leaving on the forest, and `ProcessorContextConfinementTest`. That residue is what this PR still exists to review. - **Files master deleted: they stay deleted, 12 of them** - `.semaphore/`, `.travis-archived.yml`, `service.yml`, two `.idea` run configurations, the uppercase `docs/` names master lowercased, and the `InternalRuntimeException` pair master renamed to `PCInternalRuntimeException`. - **Two shared ledgers merged as unions, not resolved to a side**: `docs/inflight/pr-strategy-doc-merge-triggers.md` and `docs/inflight/release-0.6.0.0.md` keep this branch's paragraphs alongside master's. **The strongest check is the one that makes a mis-paired rename impossible to hide.** Rather than counting recorded renames, the merged tree was differenced against `feats/ks-streams-reconciled` directly: the only differences are the 32 additions and the two union-merged ledgers. Nothing else survived on the wrong side, in the wrong module, or under an `io/confluent` path - `git ls-files | grep io/confluent` is empty, and the prescribed `grep -rnE 'io[\./]*conflu'` sweep returns only prose in dated documents. **Both sides were already renamed, which is not what the plan predicted.** It expected the spike to be pre-rename; `origin/feats/ks-on-pc-spike` had in fact moved five commits past the worktree's copy and carries the rename. That is the good case AGENTS.md describes - the mis-pairing surfaces as rename/rename conflicts on the right files instead of silently applying one module's edit to another - and it did: three `TestConventionsArchTest.java` files were offered paired across modules, and the right file was taken into each module rather than accepting git's pairing. ## One collision the merge produced, and the reason it is a good one `HeadOfLineBlockingBenchmarkTest` carried a private `sleep(Duration)` helper. Since #395 hoisted the identical helper onto `BrokerStreamsIntegrationTest` - because two arms that simulate cost differently are not comparable - the two now collide, and the compiler said so rather than either winning silently. The local copy is deleted and the shared one inherited, which is what the hoist was for. ## The gates, and the one that is a scoping artefact `bin/check-all.sh` was run in this worktree. Four gates failed on first pass and **three of the four did not exist on this branch before the merge** - they arrived with master, and every finding was on content that predates them. - **`check-inflight-tags`** flagged sixteen notes with no `inflight-type`, all of them this branch's own and all older than the tag scheme. Tagged - a first pass a reviewer should correct where it reads wrong. `bug-core-tests-jar-junit-parallelism-leak.md` additionally carries a `closed` state, because #265 removed the file it is about. - **`check-file-refs`** flagged citations broken by the rename - the exact class [`docs/citations.md`](docs/citations.md) says must be repaired rather than left. Forty-seven paths were re-pointed to their successors, **each verified to exist before the edit was written**; the claims around them are untouched, which is the line that document draws. What genuinely has no repo path - Apache Kafka's own sources inside the published jar, a placeholder in a shell recipe, a class that was only ever proposed - carries a line-scoped `file-refs: N/A` with its reason. Two targets that moved out of the tree point at the history holding them instead. - **`check-issue-refs`** flagged two bare `#NNN` in the handover; qualified. - **`check-branch-self-reference`** is the artefact, and the gate's own header names this exact situation: mid-merge the merge base is still the old one, so master's sentences read as this branch's. Twenty-five of the twenty-eight files it flagged are on `feats/ks-streams-reconciled` too and twenty-four are byte-identical to it, so marking them would be attesting to somebody else's notes - which that header explicitly says not to do. Committing the merge is the documented fix. ## Verified JDK 17, macOS. The whole reactor compiles, main and test. The reconciled branch it merges was verified in its own right before this: seam-off oracle unchanged, module unit suite green, and all 35 broker-backed integration arms green against a real broker.
Serves #255.
depends on #394
Description
The claim this PR makes: this module's three outstanding evidence gaps are closed, and each is
closed with a gate that has been shown to go red - Apache Kafka's own suite now runs with the seam
ON and every divergence is classified by mechanism, the properties stock Kafka Streams cannot express
are asserted against a real broker with their controls, and the realistic keyed benchmark reports its
own noise floor beside every ratio.
Nothing here touches the seam. No main Java in
parallel-consumer-streamschanged, and the patch isuntouched - deliberately, because two sibling rungs are changing dispatch semantics on this same base.
1. The seam-on evidence lane
The seam-off oracle has always been the behaviour-preservation gate: the patch changes nothing
while the switch is off. This adds its opposite - what Kafka's own suite says about the path this
module actually ships - and, more usefully, a verdict on it.
Seam-on failures are expected, which is why the branch forest's version of this run recorded them
and gated on nothing. The refusal envelope throws where stock would have run a construct; a committed
offset is Parallel Consumer's frontier rather than Streams'
consumedOffsets; a record handed to aworker has not been processed by the time
process()returns. A gate that goes red every build is agate people stop reading.
So the run and the verdict are separate surefire executions.
kafka-upstream-tests-seam-oncarries
testFailureIgnoreand never reddens a build;seam-on-evidence-classificationreads what itwrote and fails when a divergence matches no named mechanism. That split is the only arrangement in
which "these failures are expected" and "a new divergence fails the build" are both true.
The seam-off run is the control arm, and the two are differenced case by case. A divergence is a
case that passes with the switch off and fails with it on; nothing else is attributable to the
dispatch path. A case failing in both arms is failing for a reason the seam had nothing to do with,
and counting it would credit a change that did not cause it.
Three mechanisms are derived from the code at run time, so a sibling rung changing the envelope
needs no edit here at all:
refused-constructPcUnsupportedConstructcan produce, and the attribution is whichever constant the message names. No refusal string is copiedcommit-frontier-encodingOffsetAndMetadata, so it is an assertion about the commit encoding whatever the case is calledledgered-flakeflaky-case:marker in anydocs/inflight/noteWhat cannot be derived is attributed in the ledger, not in the classifier. "This assertion fails
because Kafka's test drives the task synchronously" is a judgement about a test, not a fact
recoverable from
expected: <5> but was: <0>. Those live indocs/inflight/test-streams-seam-on-divergence-triage.md, keyed by method rather than byparameterisation, under four named classes -
asynchronous-dispatch,stream-time-never-advances,exception-type-lost-in-the-worker,split-poll-wait. Three consequences, all deliberate: thereasoning is reviewed as prose by whoever changes the semantics under it; the PR that closes a
mechanism deletes its entries in the same change; and an entry that stops matching is reported as
stale rather than rotting silently.
The flake is absorbed by mechanism rather than by a retry. The invalid-timestamps flake has been
failing this module's oracle intermittently for weeks. Because the oracle fails fast, one sighting
used to abort the lane before the measured arm ran at all - Kafka's suite twice, re-run, for a failure
already diagnosed. The lane now relaxes that one execution and makes the verdict itself: every
control-arm failure must carry a
flaky-case:marker, or the lane stops and says there is nothingtrustworthy to difference against. The ordinary build's oracle is untouched and still fatal.
One inherited finding is corrected. The forest's triage counted that flake's seam-on failure as
the flake firing. It is attributed here to
asynchronous-dispatchinstead, on evidence the earliertriage did not have: a sibling case fails seam-on in exactly the same parameterisation with exactly
the same assertion shape, in every seam-on run taken here, while the control arm was clean in the same
runs. A coin-flip flake does not land on two sibling tests at the same parameter in consecutive runs.
The flake registry is therefore consulted last among the divergence mechanisms and first on
the control arm - a flake explains a dirty control; letting it explain a divergence is how a real one
disappears.
2. The laws Kafka's own suite cannot state
Kafka's suite is the oracle for behaviour preservation. It cannot state the properties this module
exists for, because stock Kafka Streams has no arm in which two records of one partition run at once.
These do, and each carries its control:
PcDrivenStreamsProofTest- PC-path output is identical, record for record, to what stock KafkaStreams produced. Two controls answering different questions: a same-harness seam-off arm, which
separates "the patch changed the output" from "concurrency changed the output"; and the stock
baseline itself, generated outside this module.
PcDrivenStatefulProofTest- the same for a stateful topology, where Kafka's own store wrappersread the per-task record context ambiently. The module's existing envelope test already said this
proof was the reason non-windowed aggregation is not refused - and the proof was not on this stack,
so that sentence was pointing at nothing.
CommitFrontierCrashRestartTest- nothing lost or double-counted across a crash. The committedfrontier is a point below which every record has been processed, not the offset of the last one
handed out, and under concurrent dispatch those are different records.
ShadowedStreamsControlTest- stock, single-threaded Kafka Streams running our generatedclasses. It is a control: a concurrency change that quietly breaks the serial path is not a fix.
The baseline is generated in another module, and that is the whole point. This module compiles
patched Kafka classes into its own build output, which precedes the
kafka-streamsjar on theclasspath, so every
KafkaStreamsinstance in this JVM runs the patch. A "stock arm" written herewould share every defect the patch introduced. The baselines come from
parallel-consumer-example-streams, which does not depend on this module and asserts its ownclasspath is stock before recording a row. Both fixture tests were re-run on this branch and the
tracked fixtures still match what genuinely-stock Kafka Streams produces.
The lane above defers to one of these, and says so.
commit-frontier-encodingcannot tell adeliberate encoding divergence from an offset regression - both render an
OffsetAndMetadataandnothing in the assertion text separates them. Its javadoc names
CommitFrontierCrashRestartTestaswhat actually defends offset correctness, because that arm asserts on records that arrived rather than
on a base64 metadata string. An acknowledged blind spot with a named owner beats a predicate
pretending to cover it.
3. The realistic-domain benchmark, with a noise floor
A card issuer screens every authorisation against a fraud-scoring service; the screener has been down
and the queue has to clear. The canonical Kafka Streams enrichment shape, keyed the way a real
deployment would have to key it, with a skewed keyspace and a heavy-tailed cost - the two things that
cost this seam most.
Three arms, not two, and the third is what this rung adds. It is a second stock arm on the same
records: whatever it differs from the first by is this machine in this state, and that bounds how far
a ratio can move with nothing changed at all. The headline assertion no longer asks whether the ratio
beat one - a direction check is a coin toss at parity and is satisfied by a run in which the mechanism
did nothing. It asks whether the ratio beat the floor by a margin.
No figure from an earlier run survives in this diff. The forest's numbers were measured on another
machine; the headline multipliers, the wake-on-work ablation percentage and the CPU-bound refutation
are removed or attributed to the branch they were taken on.
DEMO.mdshows the report's shape withthe numbers elided and says why. The run prints its own, with the floor beside them.
Proven able to fail - every gate here, with the prediction stated first
The lane, three ways.
An unexplained divergence. Removing the close-time commit-state publication from
PcTaskDispatcher.close()was predicted to make exactly one case,shouldClearCommitStatusesInCloseDirty, fail seam-on while staying green seam-off, with nomechanism recognising a bare
commitNeededassertion. Exactly that: one unexplained divergence,named, with its recorded failure and stack; every other divergence still explained by the same six
mechanisms; the script exited 1. Reverted and verified byte-identical by checksum. That run also
demonstrated the control-arm path unplanned - the invalid-timestamps flake fired seam-off, and the
lane named it from its ledger note and carried on instead of aborting.
An expected-class divergence. The clean run classifies every divergence under six named
mechanisms, with the refused constructs named individually, and exits 0.
A missing or stale report directory. Running the classification execution alone against deleted
directories failed with "That is an ERROR and not an empty result"; against directories from an
earlier run it failed on the freshness check, naming the file, when it was written, and when the
build started.
A control arm nobody has diagnosed - and this one arrived on its own. On a later CI run
StreamThreadTest.shouldChangeStateAtStartClose[1]failed seam-off, in Kafka's own test, on a10-second shutdown bound. The lane named the ledgered invalid-timestamps flake beside it, refused to
produce a verdict on the divergence set while an undiagnosed control-arm failure stood, and said
so in a verdict that outranked the classification - which was itself clean, no unexplained
divergences at all. That is the mode this design exists for, and it had only ever been exercised on
synthetic reports before.
It is recorded in
docs/inflight/test-streamthreadtest-start-close-shutdown-bound.mdwith theargument that carries it: the same case passed in the seam-ON arm of the same run, minutes
apart on the same runner with the same patched classes, and a defect the patch introduced cannot be
repaired by turning the seam on. One sighting, and the note says which observation would void it.
SeamOnDivergenceClassifierTestpins the first three at unit level on synthetic reports in milliseconds,plus what the end-to-end runs cannot cheaply reach: an arm whose pin did not arrive (both arms
recording
dispatch=falsedifferences the control against itself and reports a flawless zero), armscovering different classes, an attribution to an undeclared class, and a triaged attribution beating a
flake marker on the same case.
The benchmark, through a documented flag rather than a code edit. Predicted: with a worker pool of
one, PC dispatch has no concurrency to exploit, so the seam-on arm lands inside the noise floor and
the headline assertion fails while the run stays otherwise healthy. Exactly that -
--pool 1failedone case, the headline, and its message carried the whole diagnosis including both stock arms and the
PC arm; the steady-state case, which asserts on sample size rather than a ratio, stayed green.
The margin rule itself is pinned by
NoiseFloorTest, which runs in the ordinary build because it is apure function of two numbers. Its first case is the one that matters, and it is the one
--pool 1cannot reach: a 1.01x observation does not beat even the tightest possible floor. That is the
shape of the sabotaged run on #391 that printed its own headline claim, and a
direction check passes it.
Two defects found in the machinery being ported, both silent
bin/streams-benchmark.shran nothing and exited 0. Carried across unchanged it drove failsafewith
-Dit.testand-DskipUTs; these benchmarks live insrc/test/javaand this module has nosrc/test-integration, so that selects nothing, runs nothing, and succeeds. It now drives surefire -and because
failIfNoSpecifiedTestshas to be off for the Kafka executions' sake, the script assertsa surefire report exists for every class it selected and exits 2 when one does not.
--pool Nwas documented and dead. The harness hardcoded the pool, so a reader could ask for aconfiguration, be told in every log line that they had got it, and be measuring something else.
And one defect this PR introduced and caught before merge: naming the classifier in the
default-testexecution silently dropped the root pom's two surefire excludes, promoting everybroker-backed arm out of failsafe and into the unit lane, where they ran, needed Docker, and went
green. Caught by reading the run rather than the tick. Both directions are now verified.
The one thing red here that is not expected, and it is NOT being greened
PcDrivenStatefulProofTest.pcDrivenAggregationMatchesTheStockBaselinetimed out on repetition 2 of 3in the
Integration Testslane - the run that first put that class on CI. Repetitions 1 and 3 passedin the same JVM.
The obvious reading is wrong, and the run says so itself. Its StreamThread logged
polled 1207 timesagainstProcessed 14 total recordsin the two minutes it was stuck. A threadshort of CPU polls less; this one was going round its loop briskly and finishing almost nothing,
which is the shape of work dispatched and not completed rather than of a busy runner. The ambient
probe was clean - no rebalance dwell, no lag stagnation, no frozen partitions - so the group was
progressing at the broker while the task was not progressing at the processor.
A control arm was run rather than a verdict guessed. The whole streams integration suite,
locally, at the lane's own
forkCount=4 -DreuseForks=truealongside core's integration suite: green,including this class. Uncontended at
forkCount=1: green in three separate runs. The instrumentationis confirmed rather than assumed - failsafe pins no
<forkCount>of its own, so the property doesarrive, unlike the surefire trap the root pom documents.
So contention is not established, and it is not being claimed. The one term the local arm cannot
vary is core count: four JVMs and four brokers on a developer machine is not four JVMs and four
brokers on two cores.
docs/inflight/test-streams-stateful-proof-stalled-on-ci-while-still-polling.mdrecords the sighting, both arms, the two unattributed exception lines from the interleaved four-fork
log, and the experiment that would settle it - the same runner at
forkCount=1, which holds themachine fixed and varies only the load.
Nothing was loosened, shortened or serialised to make this pass. The lane re-runs on every push, so
the rate is being collected meanwhile.
The duplication reports, triaged
dups: cloneswas right once and the clone is gone. Both engines agreed onconsumeByKeyplusflattenbeing copied between the seam-on dispatch arm and the stock control arm, byte-identicalapart from the deadline. That is worth fixing here for a reason specific to this module - those two
are a matched experiment, and a copy of the reading logic in each is a second varying term waiting to
appear the first time somebody tunes one. It now lives on
BrokerStreamsIntegrationTest, which bothextend and which already holds the same broker, the same serdes and the same starting offset; the
deadline stays a parameter at each call site. Whole streams integration suite re-run green afterwards.
PMD CPD now passes both its rules.
dups: similarityand jscpd's threshold stay red, and the reason is what they measure. Everyremaining hit is an import block or a javadoc header between sibling classes - the offsets give it
away, they are lines 1 to 22 of the files. jscpd is lexical and counts comments; PMD CPD is
token-based and reports none of them. A rung that adds this much test code with similar imports and
long javadoc moves a lexical percentage, and +0.11% against a +0.1% limit is that, not new logic.
One real clone is deliberate and stays: the fixture format, written by
StockBaselineFixtureSupportand read byBaselineFixture. Those two live in modules that must notshare code - if the example module depended on the streams module, the baseline it generates would
run the patched classes and stop being a baseline. Extracting it would need a third module whose only
job is to reconnect the two things this design keeps apart.
Where this came from, and your siblings will move these results
Copied out of #271, the Kafka Streams feasibility study, which remains
the source god PR and keeps the semantic work, the measurements and the open review threads. The
seam-on run machinery and the triage method come from
feats/ks-streams-seam-on-upstream-gate; thebenchmark suite and
DEMO.mdcome fromtest/ks-streams-realistic-domain-benchmark. PC-side classesare renamed
io.confluenttobz.stub.Two sibling rungs are in flight on this same base - stream time and punctuation, and error
surfacing and backpressure - and both change dispatch semantics. Their results will move this
lane's, and the lane absorbs that by design: it classifies by the failure's own evidence rather than
by test name wherever it can, and where it cannot the attribution is a ledger entry those rungs edit
in the PR that closes it. Two of the four attributed classes are theirs by name -
stream-time-never-advancesandexception-type-lost-in-the-worker- and the lane reports an entrythat has stopped matching, so a rung that fixes one gets told rather than leaving a stale expectation.
Not here, named so a reader does not expect them:
HeadOfLineBlockingBenchmarkTest, deliberatelyleft on the forest because it isolates head-of-line blocking on a fixture whose own construction bounds
its multiplier and the settled decision is to lead with backlog catch-up; and the example module,
which is #391's.
The dependency gate will be red until #394 merges. That is expected.
Checklist
and their controls, and the reinstatement rule now naming the command that answers it),
DEMO.mdrewritten with its figures elided and the negative control documented, two
docs/inflight/notes (the divergence triage, and the flake note gaining a machine-readable marker plus what the
lane now does with it), and two method write-ups carried with the benchmark whose citations
pointed at them
docs/features/-N/A - the module is still unpublished, so there is no artifact to depend on and nothing for the ladder to point a user at. The roadmap stage is unchanged, for the same reason the four rungs below gaveSeamOnDivergenceClassifierTestandNoiseFloorTestat unit level;SeamOnDivergenceLaneTestas the lane's gate; four broker-backedlaw arms with their controls; and the benchmark suite with its noise-floor arm
ce-simplifyandce-code-reviewlocally -N/A - asked for review on the PR instead. Most of the diff is a copy from branches already reviewed on astubbs/parallel-consumer#271; what is new here - the classifier and its ledger-driven mechanisms, the control-arm integrity check, the noise floor and the margin rule, and the two silent defects found in the ported machinery - is what a reviewer should spend attention on