Repository navigation
Conversation
…very control loop pass
checkPipelinePressure() runs on every control loop pass and logged
"isPoolQueueLow(): Max loading factor steps reached: {}/{}" at WARN with no rate
limiting whenever DynamicLoadFactor#isMaxReached() held. Two configurations reach
that state, and both spam the log for as long as the queue sits below target:
- a dynamic factor that has stepped up to its cap (the reported 100/100) - the
condition is then permanent, so the line repeats for the life of the process;
- a FIXED factor - PCModule#initDynamicLoadFactor() builds DynamicLoadFactor(n, n)
when messageBufferSize is set, so isMaxReached() is true from construction and the
WARN fires from the very first pass. Following the README's own PARTITION-ordering
tuning advice therefore earns permanent log noise saying nothing is wrong. Nobody
had reported that half; it fell out of reading the code for the reported one.
The reporting changes; the buffering does not. The factor, the queue target and the
step-up rules are untouched.
DynamicLoadFactor now knows whether it is fixed (isStaticFactor(): it starts at its
own ceiling, via messageBufferSize or initialLoadFactor == maximumLoadFactor). Such a
factor cannot step, so maybeStepUp() short-circuits rather than running the
warm-up/cool-down checks, and the ceiling is reported at debug - the user asked for a
fixed buffer and got one, which is not a warning. PCModule says so through a new
DynamicLoadFactor.fixedAt(n) factory.
A DYNAMIC factor at its cap still warns: it means the in-flight target will not grow
any further, which a user may want to act on. Demoting it would fix the volume by
throwing away the signal. Instead it is rate limited to once per 30s with the
existing RateLimiter (as BrokerPollSystem and ProcessingShard already do) and reworded
to read as saturation - naming maximumLoadFactor/messageBufferSize as the thing to
change - rather than as a failure.
LoadFactorCeilingReportingTest drives the real pressure-check pass 500 times through
TestParallelEoSStreamProcessor with the load factor wired through PCModule, and
asserts on captured log output. Verified red before the fix: 500 warnings in the fixed
case (where zero is now correct) and 500 in the dynamic case (where 1 is now correct).
The stall in the original report is a different defect, fixed upstream (confluentinc#547/confluentinc#606) and
further in the confluentinc#857 family here (#119). This changes only what is logged.
Upstream-Issue: confluentinc#402
Forwarded: not-needed
Applied-Upstream: no
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RtNUsxokE9g2pSEjBHZqNA
…ise entry The manifest is the source of truth for the fork side too, so the entry advances at the transition that causes it - opening the PR - not at merge time. Upstream-Issue: confluentinc#402 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RtNUsxokE9g2pSEjBHZqNA
…ared Review found the new ceiling messages printing getQueueTargetLoaded() (in-flight target x loading factor) as the target the executor pool queue was "below", when the branch is entered by isPoolQueueLow() comparing the queue against the un-multiplied getPoolLoadTarget(). In the fixed-factor case that reported "0 queued vs 1008" for a check that actually tested "0 vs 16" - 63x out, in the exact messages this PR exists to make trustworthy. Both numbers matter, so both are now reported and each is labelled: the pool queue target is what the queue was measured against, the loaded target is the in-flight buffer that raising the factor would grow. This also matches the sibling "stepped up loading factor" debug line. LoadFactorCeilingReportingTest now asserts on the values in both the debug and warn messages - not asserting on them is what let this through. Also makes lastWorkRequestWasFulfilled volatile. Adding the test-visible setter widened writes past the control thread that owns the field, and SpotBugs was right to flag it (AT_STALE_THREAD_WRITE_OF_PRIMITIVE, new vs base). The field is touched once per control loop pass, so the barrier is free; verified the finding is gone from spotbugsXml.xml locally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RtNUsxokE9g2pSEjBHZqNA
Brings in the repo-hygiene workflow whose two new required checks (shell: sigpipe, workflows: action versions) this branch predated, so the PR was blocked waiting on contexts it could never produce. Conflict: src/docs/development/upstream-map.yaml. #211 slimmed the schema (dropped `forwarded`, `todo`, `backlink`, the long-tail DEFERRED block and scripts/upstream-backlink.sh) and mirrored every open confluentinc issue into this fork. Resolved by taking master's file whole and re-adding the issue-402-max-load-factor-log-noise entry in the new shape: `forwarded`/`todo` dropped (no tooling reads them now), the backlink action folded into notes, `fork_issue: 155` recorded now that confluentinc#402's mirror exists, and every issue reference qualified by repo per the new house rule. scripts/upstream-map.py validate passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RtNUsxokE9g2pSEjBHZqNA
#211 made a bare `#NN` below #1000 a CI failure on added lines: the fork's numbers sit entirely inside confluentinc's range, so an unqualified reference is a coin flip. Three added lines in docs/inflight/pr-155-load-factor-noise.md tripped the gate; the javadoc in AbstractParallelEoSStreamProcessor and LoadFactorCeilingReportingTest passed only via the "upstream #402" escape hatch, which the same commit deprecates in favour of naming the owner. Both now say #155 / confluentinc#402. Also moves the "answer the original reporter when this merges" note out of upstream-map.yaml and into the inflight doc. AGENTS.md is explicit that the manifest has no todo: field and loose ends belong in docs/inflight/ - the field was already dropped when the manifest was slimmed, so leaving the note there would have been an entry no tooling reads. Verified with .github/scripts/issue-ref-gate.js over this branch's diff against master: clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RtNUsxokE9g2pSEjBHZqNA
… of the index Two things #211/#212 surfaced in this PR's own files. #212's lesson - await/assert the thing itself, never a proxy - has an adjacent shape here worth naming. fixedMessageBufferSizeDoesNotWarnOnEveryPass asserts NO warnings, which would pass vacuously if the appender were ever attached to the wrong logger. It is not vacuous today, because the debug assertion below it can only hold when the capture is live and pointed at the code under test - but nothing said so, and the two are separable by a well-meaning edit. Now they are commented as a pair, pointing at the write-up. The test needed no other change: it is fully synchronous, driving 500 checkPipelinePressure() passes on the test thread and reading the appender after, so there is no await to get wrong. Also rewords the upstream-map note that said "this manifest has no todo: field". bin/todo-index.sh read the literal marker and wanted to index it, which is the gate working - the fix is to stop writing a marker in prose, not to regenerate the index around a false entry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RtNUsxokE9g2pSEjBHZqNA
…hanges) Directory move only, so every path is 100% similar and git's exact-rename detection cannot fail on it. The content edits follow in the next commit. Generated by bin/rename-packages.sh.
Text edits only. No file moves in this commit, so it cannot dilute the rename detection in its parent. Generated by bin/rename-packages.sh.
ProcessingShard#getWorkIfAvailable selected work in two steps: isAvailableToTakeAsWork() evaluated three terms - not in flight, no success verdict, retry delay passed - and onQueueingForExecution() then acted, re-validating none of them. Nothing makes that decision atomic. The shipped engine is safe today only by architecture: the control loop is the only selector, so no second decision can interleave with a completion. The direct-pull engine in development gives every worker its own selector, and there the gap is real: a claim decided before another worker's completion could act on an already-completed record - the user function ran a second time and the offset was committed a second time. Observed at 4 occurrences in 14,400,000 record completions on the development branch; with assertions on it surfaces as an AssertionError from PartitionState#onSuccess, in production it is silent. The boolean inFlight and the Optional<Boolean> maybeUserFunctionSucceeded were the two questions selection asked, held apart. They are now one AtomicReference<ExecutionState> with six states - AVAILABLE, IN_FLIGHT, IN_FLIGHT_SUCCEEDED, IN_FLIGHT_FAILED, SUCCEEDED, FAILED - and a claim reads the state once, decides against that value, and compare-and-sets from that exact value: the check IS the act. The retry delay is deliberately not a state - nothing fires a transition when a clock passes a point - so FAILED covers both waiting and due, and isDelayPassed() separates them at the moment of asking, ordered after the volatile state read so the previous holder's retry deadline is visible. WorkClaimStateMachineTest pins the machine: the losing interleaving played out by hand with no threads (the concurrent reproduction needed millions of completions per occurrence; this is exact and runs in milliseconds), every state crossed on one record, and the properties a refused claim must keep - the verdict stands and the delivery count does not move. Same defect class, other instances: the registration order in PartitionState#maybeRegisterNewPollBatchAsWork makes a record selectable before its offset is registered - latent for the same single-selector reason, recorded in docs/inflight/bug-a-record-is-selectable-before-its-offset-is-registered.md rather than fixed here. The follow-up this fix leaves open - a lost claim is silent, and means opposite things on the two engines - is docs/inflight/core-a-lost-claim-means-two-different-things.md. Adapted from 2e83185 on perf/engine-concurrency: the delivery- abandonment mechanism that version interacts with stays on that branch (its WorkManager side is not part of this change), and the javadoc here describes this tree, where the second selector does not exist yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
…lete the clamp ProcessingShard.availableWorkContainerCnt was corrected after every decrement by a clamp to zero, commented "in case of possible race condition". That number was summed across shards and added to numberRecordsOutForProcessing to decide, in WorkManager.isSufficientlyLoaded(), whether the broker poller pauses or resumes - so a wrong value did not misreport, it mis-gated record intake. And the clamp only caught the low side. The comment was wrong about the mechanism. There was no race. There were two conditional decrements whose conditions did not match the condition the counter was incremented on, both reproducible on a single thread: - ProcessingShard.remove(long), reached from the revocation sweep, deducted only for a record that was isAvailableToTakeAsWork() - which additionally requires the retry delay to have passed. A record parked in retry back-off had already been counted in by markAvailableAgain(), so revoking it left the increment behind, permanently, high, in the direction nothing corrected. The retry-queue entry that normally nets that increment out is removed on the same path, so nothing was left to cancel it. revokingARecordParkedInRetryBackoffLeavesNoPhantomAwaitingSelection is that sequence; it fails on the old code with the counter reading 1 against an empty shard. - Both stale-container sweeps deducted unconditionally, so sweeping a record that was out at a worker - already deducted at selection - deducted it a second time. That is the low-side drift the clamp existed to hide. The gate only ever consumed the SUM of "awaiting selection" and "out for processing", never the split, and that sum is just "records inside the system". RecordPopulation derives it by conservation - admitted - retired, two monotonic LongAdders - where a record is admitted when its WorkContainer enters a shard's entry map and retired when it leaves by any path. The conservation walk test drives every departure path in one sequence and holds the O(1) figure against an O(n) scan of the shards. The high-side drift is rebalance-shaped - revocation of a parked retry is what leaves the phantom - so it is recorded as a CANDIDATE fourth mechanism in the paused-consumption family (#119, confluentinc#857), flagged in docs/inflight/bug-857-family.md as a hypothesis rather than a diagnosis: enough phantoms keep intake paused until a restart clears the in-memory counter. Adapted from fa4d1cf on perf/engine-concurrency: the verdict-free abandonment departure stays on that branch with its WorkManager side (the conservation walk notes where the branch extends it), and ProcessingShardStaleReplacement909Test - which postdates the original commit - is reconciled the way that branch's merge reconciled it, through the accounted addWorkContainer path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
…-noise # Conflicts: # .idea/runConfigurations/All_examples.xml # .idea/runConfigurations/_Tag__transactions__.xml # README.adoc # bin/check-copyright-headers.sh # bin/rename-packages.sh # docs/inflight/pr-blockers-and-collisions.md # docs/refactoring.md # docs/todo-index.md # parallel-consumer-core/src/main/java/bz/stub/parallelconsumer/internal/ConsumerManager.java # parallel-consumer-core/src/main/java/bz/stub/parallelconsumer/internal/Documentation.java # parallel-consumer-core/src/test-integration/java/bz/stub/parallelconsumer/integrationTests/AmbientProbeExtension.java # parallel-consumer-core/src/test-integration/java/bz/stub/parallelconsumer/integrationTests/SampleTestingFailsafePluginInclusionCore.java # parallel-consumer-core/src/test-integration/java/bz/stub/parallelconsumer/integrationTests/chaostests/AbstractRevokeUnderWorkScenario.java # parallel-consumer-core/src/test-integration/java/bz/stub/parallelconsumer/integrationTests/chaostests/ChaosScenarioBase.java # parallel-consumer-core/src/test-integration/java/bz/stub/parallelconsumer/integrationTests/chaostests/ProgressProbe.java # parallel-consumer-core/src/test-integration/java/bz/stub/parallelconsumer/integrationTests/utils/ManagedPCInstance.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/AbstractParallelEoSStreamProcessorConfigurationTest.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/AmbientProbeExtensionTest.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/CommitRejectionTestBase.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/MockConsumerCommitTimeoutTest.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/MockConsumerEarlyCloseTest.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/MockConsumerSaslAuthenticationTest.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/MockConsumerTest.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/TestConventionRules.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/internal/ProducerManagerTest.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/internal/utils/BlockedThreadAsserter.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/internal/utils/JavaEnvTest.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/internal/utils/StringTestUtils.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/internal/utils/ThreadUtils.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/offsets/OffsetEncodingBackPressureTest.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/offsets/OffsetEncodingTests.java # parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardManagerTest.java # parallel-consumer-core/src/test/resources/junit-platform.properties # parallel-consumer-core/src/test/resources/logback-test.xml # parallel-consumer-examples/parallel-consumer-example-core/src/test/resources/logback-temp-test.xml # parallel-consumer-examples/parallel-consumer-example-reactor/src/test/resources/logback-test.xml # parallel-consumer-examples/parallel-consumer-example-streams/src/test/resources/logback-test.xml # parallel-consumer-examples/parallel-consumer-example-vertx/src/test/resources/logback-test.xml # parallel-consumer-mutiny/src/test/resources/logback-test.xml # parallel-consumer-reactor/src/test/resources/logback-test.xml # parallel-consumer-vertx/src/test/java/bz/stub/parallelconsumer/vertx/VertxTest.java # parallel-consumer-vertx/src/test/resources/logback-test.xml # src/docs/README_TEMPLATE.adoc # src/docs/development/upstream-map.yaml
…e kill on #201's CI CLASS2_STALL/LAG_STAGNATION on three partitions at once, committed offsets stagnant 154s against the 150s bound with the group STABLE and heartbeats flowing - the second sighting's exact signature. Seed captured before the log expires. The cooperative control passed in the same run: third time the red draws eager and cooperative holds. Also deletes this PR's own tracking note per docs/inflight rules - work a PR resolves is tracked by that PR; its content lives in the commit bodies and the PR description. That closes the 'inflight: tags' check this note was failing (it predates the tag scheme). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
…ot timing UNORDERED dispatch re-walks the whole in-flight prefix on every pass while KEY skips an occupied shard in one step, so their scan costs diverge as in-flight depth grows. Nothing pinned that, and a change that made one shape quadratic would have shown up only as a slower benchmark on someone's laptop. The first version of this guard timed the two modes and compared durations. That is not a test, it is a race with the rest of the suite: it ran inside a thread-parallel run, and its failures were the suite's load rather than the code's behaviour. Four attempts to stabilise it - fastest-of-three, a ratio instead of absolutes, CPU time instead of wall clock, @isolated - each moved the flake without removing it, and three of them were made after a bisect had already shown the dispatch code unchanged. So it counts instead. DispatchScanMeter is a LongAdder shared across every shard of one ShardManager, incremented once per entry the scan examines. The test asserts exact totals - RECORDS examinations for KEY, the triangular sum for UNORDERED - which are properties of the algorithm, not of the machine: deterministic, load-immune, and they say which shape changed rather than that something got slower. The meter is main-code rather than test-code deliberately: it counts inside the scan loop, where a test double cannot reach without changing the thing being measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
…sured Every figure this project has published is throughput. Nothing measures how long a record actually spends inside Parallel Consumer, and no existing meter can be made to answer it: `user.function.processing.time` starts when the handler starts, and `WorkContainer#timeTakenAsWorkMs` starts when a delivery is claimed. Both begin AFTER the interval that matters - the time a record sits in the shards, fetched but not yet picked up. That interval is the one operational blind spot the metric set has. An instance holding 543 records against a configured maxConcurrency of 5,000 is ambiguous today between "the poller has not fetched more" and "they are fetched and queued behind other work", and the two call for opposite responses. Under a backlog, residence time is Little's law applied to PC's own buffers - buffered depth over throughput - so it is exactly the quantity that separates them, and it is what makes the cost of `DynamicLoadFactor`'s over-buffering visible as latency rather than as an argument. WHAT IS MEASURED. From the record leaving the consumer - the instant its `WorkContainer` is constructed from a poll batch - to the control loop finishing with that delivery. Produce time and time waiting on the broker are excluded: they are the environment, not the engine, and `ConsumerRecord#timestamp()` is there for anyone who wants them in. The end instant is the controller's rather than the worker's `succeededAt`, because the hop from the worker's mailbox back to the control loop is PC's own queueing and belongs inside the number. FAILURES AND ABANDONMENTS ARE SAMPLED TOO, which is the surprising choice and the one that makes the number honest. Arrival is stamped once, at construction, so a record that fails twice before succeeding carries both attempts and both retry delays in its final sample. Sampling only successes would make the distribution look best exactly when the engine is doing worst - which is the case this measurement exists to expose. It also means the sample count exceeds the record count whenever anything is retrying, so the denominator for a rate is `pc.processed.records`, never this timer's own count. A record whose partition is revoked mid-flight records nothing: its result is discarded and it will be redelivered elsewhere, so counting it would measure a rebalance rather than the engine. ONE TIMER, NOT ONE PER PARTITION, unlike the `processed.records` and `failed.records` counters beside it. A percentile histogram is a much heavier meter than a counter, and percentiles cannot be merged after the fact - so per-partition timers could never be combined into a consumer-wide p99, while an operator narrowing a bad p99 to a partition still has the per-partition counters and offset gauges. A per-partition breakdown, if wanted, has to be an additional meter rather than a re-tagging of this one. Flagged as a judgement call, not a settled one. THE TEST WAS BROKEN BEFORE IT WAS TRUSTED. Its sample count is asserted exactly (records + 1), not "greater than zero", because the plausible wrong implementation - sampling only successes - produces exactly the record count and sails through a loose assertion. Sabotaged by removing the failure-path sample: the test fails, `expected 1001L`. Its other assertion is that the maximum clears the one-second default retry delay, which is the single observable that a time-in-flight measure could not produce. Release-Note: New Micrometer timer `pc.record.residence.time` reports how long each record spends inside Parallel Consumer - from leaving the consumer to Parallel Consumer finishing with it, including client-side queueing and every retry, and excluding produce time and broker wait. Percentiles are published alongside the mean, and failures and abandonments are sampled as well as successes, so a retrying workload is visible rather than hidden. It is the only meter that distinguishes records not yet fetched from records fetched and queued.
Stamping the arrival instant in WorkContainer's constructor made it the first thing that constructor reads off the module, and WorkManagerTest built its containers from `mock(PCModuleTestEnv.class)` - a mock of a class that is already a test double, whose working MutableClock the mock replaced with null. The test errored where it had previously passed, which is a real regression in the test's double rather than in the test's subject: TreeMap ordering was never in question. Uses the class's own PCModuleTestEnv field instead, which needs no stubbing at all. Every sibling that mocks PCModule for this purpose stubs clock() by hand - ShardManagerTest#retryQueueOrdering and its two neighbours - so the unstubbed one was the outlier. DEFECT-CLASS SWEEP: four sites across every module's sources construct a WorkContainer from a mocked PCModule. Three already stub clock(); this was the only one that did not, and it is fixed. None outside parallel-consumer-core. Also files the PARTITION starvation note under `bug-` rather than `perf-`, since the prefix names the AREA and defects in product code belong under bug- where a reader scanning for them will look; and states, up front, the symptom rather than the mechanism - you configure maxConcurrency 24 and get 2 to 6 - plus that messageBufferSize is a workaround and not the fix, because a user cannot be expected to derive `max.poll.records x partitions` and the real fix (a prefetch target expressed in shard coverage) is not built. Full unit suite: every Java module passes. The one red module is the cross-language conformance prebuild, which fails on this machine's toolchains - go.mod uses a `tool` block the installed Go does not parse, and the Rust runner cannot execute the cached protoc (permission denied). Neither is reachable from anything on this branch.
Both add one field to ProcessingShard and ShardManager; the union is what the engine branch always had. ShardOccupancy stacks on this because its retirement helpers are written against the conservation plumbing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
# Conflicts: # parallel-consumer-core/src/main/java/bz/stub/parallelconsumer/state/ProcessingShard.java
… skipped rather than walked Antony's design. ShardManager entered EVERY shard on every dispatch pass with no guard at all - not even isEmpty() - paying an iterator, a HashSet, an ArrayList and a walk to the head purely to discover that an ordered shard already had a record out at a worker, and then breaking. That is one bit of information, and ProcessingShard#getCountOfWorkInFlight() now answers it with one comparison. IT ALSO REPLACES AN ESTIMATE. ShardManager#getUpperBoundOnSelectableWork(), which decides how many parked workers the direct-pull engine wakes, could previously do no better than min(awaitingSelection, shardCount) - counting an occupied shard as offering one record when it offers none. It now counts the shards that can actually yield. In the steady state of an ordered workload most shards are occupied, so the old figure over-reported by roughly the worker count on every pass and every one of those wake-ups found nothing. WHERE THE CHARGE LIVES, AND WHY IT CANNOT LEAK. The count is taken and released by the two halves of one transition on the record itself - claimed, then landed - rather than by the sites that add and remove shard entries. Each container is handed its shard's meter at admission, before it is published into the entry map, so the write is safe and there is no removal site whose condition can be got wrong. That is what makes the awkward path need no special case: a result returning for a revoked partition takes handleFutureResult's stale branch, which ends the flight and tells no shard anything, and the charge still comes back. A record whose shard was discarded first releases into a meter nobody reads. Held against an independent O(n) scan of the containers in every assertion, the same arrangement RecordPopulation has. IT IS AN OPTIMISATION, NEVER THE GUARANTEE. The ordering invariant is still enforced where it always was - by the per-record claim and by the isOrderRestricted() break inside the scan - so a reading that is transiently low costs a wasted scan, never a second record out of an ordered shard. The one behaviour genuinely given up is the scan's opportunistic stale-container sweep on a blocked shard, which is picked up on the next pass once the shard is free, or by the poller's own sweep. UNORDERED IGNORES IT, deliberately. Shards there are never blocked, so the check costs one comparison that is never true, and nothing reads the count in that mode. Making it help UNORDERED is a separate question - see docs/inflight/next-direct-pull-unordered-selection.md. LongAdder rather than AtomicLong: written twice per delivery from whichever worker took the record, read at most once per shard per pass, and never read at all under UNORDERED - so writes should scale and reads are rare. Under an ordered mode, where the count IS read, writes cannot contend at all, because at most one record of the shard is out at a time. WHAT THIS DID NOT MOVE, WHICH IS WORTH RECORDING. OrderingModeDispatchParityTest's KEY examination count was expected to fall and did not, and the constant is unchanged. That workload never returns a record, and LoopingResumingIterator resumes each pass where the previous one stopped, so every shard is entered exactly once and none is ever revisited while occupied - the guard never gets the chance to fire. Its javadoc now says so, so the next reader does not conclude the guard is inert. That the guard does fire, and does suppress examinations, is asserted against the same meter by ShardInFlightCountTest. Every assertion in ShardInFlightCountTest was proven red by sabotage before being trusted: disabling the guard (both ordered modes go red on the examination count), never releasing the charge (five assertions across four tests), and reverting the upper bound to min(awaitingSelection, shardCount).
…prefix, in UNORDERED dispatch
The direct-pull engine collapses above about a hundred workers - 1,357 msg/s at
concurrency 5,000 against the shipped engine's 18,376, with runs that never
finish - and there were two candidate mechanisms with completely different
fixes: workers losing the claim CAS to each other, or the shard scan itself
being paid N times. This settles it and removes the one that turned out to
matter.
The measurement is a controlled experiment rather than another benchmark.
DirectPullScanCostMeasurementTest decouples the number of scanning threads
from the number of records in flight - in the engine they are the same number,
so no observation of the engine can separate them - and reads the cost off
DispatchScanMeter, which counts and is therefore unaffected by machine load.
With a SINGLE scanner, where claim contention cannot exist at all, entries
examined per record dispatched:
10 in flight -> 1.00
100 in flight -> 9.99
1,000 in flight -> 98.02
5,000 in flight -> 440.13
That is the whole shape of the collapse, present with zero contention. Adding
scanners at a fixed depth changes almost nothing (1,000 in flight: 97.71 with
one scanner, 102.87 with ten, 106.56 with a hundred). Contention is real at the
top end and secondary: 5,000 x 5,000 costs 1,621.89 against the 440.13 the walk
alone accounts for.
So the mechanism is the scan, not the claim, and no amount of lock tuning could
have helped. Under UNORDERED a shard holds every record of its topic-partition
including the in-flight ones - a record leaves on success, not on being taken -
and the scan walked that prefix to reach the first claimable entry, once per
record on every worker.
ShardOccupancy adds an index of the offsets no worker is holding, and the
unordered path walks that instead. It is maintained by the two halves of one
state transition on the record itself, claimed and landed, which is the
arrangement that already keeps the per-shard in-flight count from drifting -
so there is no removal site whose condition can be got wrong. After:
1 x 5,000 -> 1.00 (was 440.13)
5,000 x 5,000 -> 1.60 (was 1,621.89)
Flat at every concurrency measured.
The ordered modes are deliberately untouched and still walk the entry map. In
KEY and PARTITION the in-flight record at the head is what makes a shard refuse
a second taker, so removing it from that walk's view breaks ordering silently -
which is what cost perf/split-shard-inflight ten failing tests. They also have
nothing to gain: the break fires after one entry either way.
OrderingModeDispatchParityTest's UNORDERED constant goes from 410,000 to 20,000
for 20,000 records, which is the outcome its own comment asked for by name.
ShardSelectableIndexTest holds the index against the shard on every transition,
because the way an index like this fails is by omission - a record left out is
never offered again, and nothing else goes red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
…he pressure system reading the executor A handler that blocks holds an OS thread for as long as it blocks, and a machine can only bring platform threads out of a park at a fixed rate - so reachable concurrency is min(maxConcurrency, activationRate x handlerLatency), not maxConcurrency. Measured on a 12-core laptop at maxConcurrency 5,000 with a 100ms handler: platform threads held 2,673 records in flight, virtual threads held all 5,000. Raising maxConcurrency past that ceiling does nothing; making the work not hold a thread removes the term. useVirtualThreads (default from the pc.virtualThreads system property) runs the user function on Executors#newVirtualThreadPerTaskExecutor, reached reflectively because the published artifact is still Java 8 bytecode. Opt-in, JDK 21+ at runtime, and refused loudly rather than silently falling back. WHAT THE POOL CHANGE FORCED. setupWorkerPool now returns ExecutorService rather than ThreadPoolExecutor, because a virtual-thread executor is neither. Two subsystems read the concrete type and had to stop: - requireRejectionIsVisible inspects a rejection handler that only a ThreadPoolExecutor has. It now applies to that case and passes any other pool through - a virtual-thread executor starts a thread per task, so it has no bounded queue to reject from. - The pressure system read the executor's queue depth to decide whether to step the load factor. There is no queue under virtual threads, so UserFunctionTaskAccounting derives the same figure from submissions minus completions - counted before the submit, because a virtual thread can be running the task before submit() returns, and an increment placed after would make the derived depth transiently negative. The load-factor ceiling report is silent under a virtual-thread pool: "the queue is low" is that pool's permanent resting state rather than a symptom, and the advice the line gives would be actively wrong - raising the load factor buys a deeper record buffer, not more concurrency. External engines (Vert.x, Reactor, Mutiny) opt out via supportsVirtualThreads(): their worker "pool" is one dispatch thread by design and their concurrency lives in the external runtime, so an unbounded executor there would silently make dispatch concurrent. WorkerPoolAccountingAgreementTest pins the derived queue depth against the executor's own for the platform pool, which is what stops the two accountings drifting. Adapted from b6083f5 on perf/engine-concurrency. Two things that commit predates were kept rather than reverted: master's shutdown guards in submitWorkToPool (the closing-state return, and the shut-down-pool rejection that drops the batch and stops the poll rather than escaping to the control thread - SubmitWorkToPoolShutdownRaceTest covers them), and the load-factor reporting, which is #201's and is left in master's form. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
… and refuse a green run that skipped it
Adds the execution-mode axis docs/inflight/test-opt-in-engine-paths-are-unexercised.md
asks for, with virtual threads as its first non-default entry.
WHY AN AXIS AND NOT A VIRTUAL-THREADS TEST SUITE. These paths are meant to be
behaviourally equivalent to the default engine. What needs asserting is not that
virtual threads work but that virtual threads and the default AGREE - on ordering,
commits, retries, rebalance, shutdown. That is the suite we already have, run again
with a different selector. Adding direct pull later is a matrix entry plus a row in
the guard, not a second job; the shape is written into the workflow comment and
deliberately left unpopulated.
THE SELECTOR HAD TO REACH THE FORK. Surefire runs tests in a separate JVM and a -D on
the Maven command line does not reach it, so `mvn test -Dpc.virtualThreads=true` would
have run the whole suite on the DEFAULT engine and reported green. The pom now
forwards pc.virtualThreads and pc.directPull through systemPropertyVariables.
BUILD ON 17, TEST ON 21. Virtual threads are a runtime capability and PC reaches them
reflectively, so nothing compiles against JDK 21 - and nothing can: Jabel 1.0.0, which
is what emits Java 8 bytecode from Java 17 source here, does not run on JDK 21 at all.
It rewrites javac internals with a Byte Buddy that predates 21, and having got past
that it strips Source$Feature.LAMBDA and takes the compiler down. Measured on Temurin
21.0.9; three stacked blockers, recorded in the inflight note. Surefire's fork was
already parameterised as ${jvm.location}, so the lane needs no build change and tests
the SHIPPED Java 8 bytecode on a JDK 21 runtime - what a user with the option on does.
bin/check-execution-mode.sh IS THE ANTI-SILENT-GREEN DEVICE. The upstream PR's
virtual-thread tests used JUnit Assumptions and CI runs a JDK without virtual threads,
so they skipped and the job passed - its author said so himself, and this repository
has shipped that shape before. Two halves now prevent it: the Java tests FAIL rather
than skip when the mode was selected and the runtime cannot serve it, and the guard
reads the surefire XML and refuses a green verdict when the mode-proving tests did not
execute. Two distinct reds, as check-ossindex-audit.sh has: exit 1 the lane is broken,
exit 2 the tree has a real disagreement, exit 1 wins when both.
Its self-test earned its place immediately - it caught the guard reading every surefire
attribute as zero on macOS, because BSD sed rejects `s/…/…/;t;d` on stderr while still
exiting 0. And pointing the finished guard at a real JDK 17 report caught the guard
committing its own defect: two JDK-agnostic tests in the marker class counted as proof
that the mode had run. Those moved to VirtualThreadOptionTest, so every test in the
marker class now requires the mode, and a JDK 17 report is correctly called a broken
lane.
The job is advisory and its name is new. Required checks are matched by name in the
master ruleset and a required context no run has produced blocks every PR whose base
predates it, so making it required is a separate deliberate act. Re-adding a JDK axis
also re-opens the decision that removed the last one for cost (#128), so this is
scoped to the unit suite only.
Release-Note: CI now runs the core unit suite a second time with virtual threads enabled on a JDK 21 runtime, and fails the lane if that run skipped the tests instead of running them.
…iced in The two changes meet in the same method and neither is a superset: #201 splits the report into a debug line for a pinned factor and a rate-limited warn for a saturated dynamic one; #360 needs the whole report silenced under a virtual-thread pool, where 'the queue is low' is the resting state and both branches would mislead. Resolved by keeping #201's reporter and returning early for the virtual pool. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
Direct pull hands work selection to the workers themselves: instead of the control loop reading a target depth off the executor's queue and pushing records into it, every worker blocks on the shards and takes its own next record. The 2022 attempt was abandoned because its workers spun on an empty shard set; DirectPullWorkerPool gives them the blocking wait it never had. Measured, it was 3x FASTER at ten workers and 20x SLOWER at five thousand. The collapse was not lock contention on the shards, which was the obvious suspect and the wrong one - it was the SCAN. UNORDERED dispatch re-walked the whole in-flight prefix on every pass, so with N workers scanning concurrently the cost went quadratic in flight depth. At 5,000 in flight it examined 440 entries per record dispatched. Two changes fix it, and they help the shipped engine too, not only direct pull: - A shard counts its own in-flight records, so an ORDERED shard that already has one out is skipped in a single comparison rather than walked. - ShardOccupancy keeps an index of the offsets a shard is not holding in flight, so UNORDERED dispatch walks unheld offsets instead of the whole prefix. 440 examinations per record becomes 1.00. Opt-in and preview-grade: pc.directPull selects it, and ExternalEngine refuses it (its worker pool is one dispatch thread, so there are no workers to hand the shards to). STACKED, and the parents are real rather than stylistic: #335's ExecutionState is what makes a claim safe when two workers select concurrently; #336's conservation counters are what the retirement paths update; #358's meter is what the scan-cost tests measure with; #360's pool widening is what lets pullers run on virtual threads (the fastest configuration measured is both together); and #201's load-factor reporter meets the direct-pull replacement for checkPipelinePressure in the same method, reconciled here. Full core suite: 457 tests, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
…turned The harness that measured everything in this stack, plus the data and the conclusions - including the ones that went against us. WHAT IT IS. bench/run-bisect.sh runs one Java source file against every released PC version and against several engines, so a number can be attributed to a version, an engine, an ordering mode, a key distribution, a concurrency, an arrival rate or a work model rather than to "the benchmark". Eleven arms: core, core-vt, core-dpvt, the four async engines, the Kafka Streams threading model, a plain consumer, a thread-pool floor, and KIP-932 share groups. Three latency families - residence (poll to completion), drain (engine start to completion) and end-to-end (intended send instant to completion), the last being the one coordinated omission cannot fool. WHAT IT OVERTURNED, and this is why the results ship with it: - All-distinct keys make KEY ordering free - every record is its own shard - so 369 of 376 historical result rows measured UNORDERED wearing KEY's name. Under a Zipf distribution the same engine on the same records runs at a third of the throughput and sustains ONE record in flight of a configured 24. - A ~1.75x claim against the last release had no measurement behind it. Measured properly, core against core across fourteen cells is 0.96-1.04x: the shipped default did not get faster. What did is the opt-in engines, and only at high concurrency with short handlers. - A share-groups comparison did not reproduce at its own operating point - 66,524 then 29,709 then 11,225 msg/s across three brokers in one day, while every PC arm held within 3% - so no share-group figure is quotable in either direction. - Head-of-line blocking, finally measured: on a tailed handler with distinct keys, PC's KEY ordering holds end-to-end p99 at the handler's own tail (510/509/508ms across 50/70/90% utilisation) while the Kafka Streams threading model amplifies to ~940ms. The instrument defects are documented as prominently as the results, because six of the last eight produced plausible numbers rather than errors - a work model wired to three of five call sites, percentiles rendered after teardown, a per-thread RNG that made virtual threads draw once each, a sustained-in-flight column that read zero on short runs. docs/solutions/best-practices/benchmark-a-rival-on-the-semantics-it- actually-offers.md is the method that survived all of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
Dependency ReviewThe following issues were found:
Vulnerabilitiesbench/llingr/go.modOnly included vulnerabilities with severity high or higher. OpenSSF ScorecardScorecard details
Scanned Files
|
✅ Duplicate Code ReportTwo engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check. ✅ PMD CPD
|
| PR | Base | Change | |
|---|---|---|---|
| Clones | 87 | 78 | :face_with_monocle: +9 |
| Duplicated lines | 1224 | 1110 | :face_with_raised_eyebrow: +114 |
| Duplication | 1.14% | 1.17% | 🙂 -0.03% |
| Rule | Limit | Status |
|---|---|---|
| Max duplication | 2% | ✅ Pass (1.14%) |
| Max increase vs base | +0.1% | ✅ Pass (-0.03%) |
⚠️ 10 new clones introduced
- 12 lines:
parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardSelectableIndexTest.java:58<->parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/WorkClaimStateMachineTest.java:67 - 17 lines:
parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardInFlightCountTest.java:57<->parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardSelectableIndexTest.java:52 - 8 lines:
parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardInFlightCountTest.java:77<->parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardSelectableIndexTest.java:69 - 7 lines:
parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ProcessingShardStaleReplacement909Test.java:7<->parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardInFlightCountTest.java:10 - 18 lines:
parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/OrderingModeDispatchParityTest.java:1<->parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardSelectableIndexTest.java:1 - 15 lines:
parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/DirectPullScanCostMeasurementTest.java:1<->parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardSelectableIndexTest.java:1 - 15 lines:
parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/DirectPullConcurrentSelectionTest.java:7<->parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardInFlightCountTest.java:8 - 12 lines:
parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/DirectPullConcurrentSelectionTest.java:207<->parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/state/ShardSelectableIndexTest.java:199 - 16 lines:
parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/internal/WorkerPoolAccountingAgreementTest.java:145<->parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/internal/WorkerPoolAccountingAgreementTest.java:65 - 13 lines:
parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/internal/VirtualThreadExecutionModeTest.java:1<->parallel-consumer-core/src/test/java/bz/stub/parallelconsumer/internal/VirtualThreadOptionTest.java:1
Powered by astubbs/duplicate-code-cross-check
astubbs
added a commit
that referenced
this pull request
Aug 25, 2026
…sure the same thing twice Owner's call: the benchmarking work (#362) and the demonstration work in the language-proxy descendants (#328, #331, #332) independently built workload generators, engine arms and result rendering. Two work models will disagree, and the bench's took four defect-fix rounds to become honest - a second implementation will rediscover those silently, in front of an audience. Records why it matters and the candidate directions, without picking one: post-merge reconciliation, not a blocker on either side, but due before the demos are used for anything public. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
Draft
2 of 4 tasks
🧪🔒 Quarantine Lane Report
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its |
|
astubbs
added a commit
that referenced
this pull request
Aug 25, 2026
The bench harness stacks on #361 because an arm that silently runs platform threads is worse than one that fails, and the integration branch now has its own PR rather than staying invisible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MedgsxqrM8vjSt5ncuAo8g
4 tasks done
astubbs
marked this pull request as draft
August 25, 2026 13:58
astubbs
added a commit
that referenced
this pull request
Aug 31, 2026
…eatures, filed and bounded Fifth bit of the follow-up Codex strategy conversation (2026-08-29/30): the deliberate move away from adaptive-concurrency territory, mining what PC uniquely knows where Kafka semantics meet application execution. Eight new notes, each kept to the idea, its prior-art hook, and the caveat that decides whether it is honest: - core-record-semantic-tracing.md - the per-record "why did it wait" timeline; #359 stamps only the endpoints, and the per-gate stamps are the opportunity model's own instrumentation. - core-ordering-profiler.md - ordering tax, ordering-scope discovery, per-record critical path: one architectural profiler in three stages, near-free on the shard state #361 added. - core-retry-economics.md - blast-radius quarantine, per-function retry amplification, and storm detection (#333's failure-fraction inhibitor promoted to an operator warning). - core-capacity-fingerprinting.md - persist what the controller learns: warm starts, empirical workload models, semantic regression detection; open question is where the fingerprint lives. - perf-workload-replay-simulator.md - sanitised trace capture feeding the #362 harness as a capacity-planning simulator over real workloads. - release-certified-execution-semantics.md - publish #293's conformance matrix as a certification claim; a table that cannot show a cross certifies nothing. - core-function-manifest.md - split the idea on the positioning line: adopt the language-neutral manifest, leave "pc deploy" to platforms (embedded-not-cluster). - core-scheduler-canarying.md - A/B a scheduler over 1% of ordering domains; stratify or the comparison reports the sampling. Amendments: web-control-plane gains true lag as the fifth instrument (broker lag vs effective lag); the facades note gains the migration advisor with its honesty bound (observation mode sees keys and poll cadence, not handler time); the research program gains broker portability as question 6 with the reproduce-at-own-operating-point trap named; and the opportunity model gains the standing task the conversation closed on - inventory the boundary knowledge on purpose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PFA3CXkGc2m32cBJMAWTK
astubbs
added a commit
that referenced
this pull request
Aug 31, 2026
This PR ships the capability; three places in the repository still say it is deferred. A claims document that contradicts the code in the same release is worse than one that is merely out of date, because a reader has no way to tell which side is stale. Back-filled from 3456e38 on `perf/engine-concurrency` (#363), the commit that measured virtual threads against Parallel Consumer itself and corrected the four places the repository called them deferred. Only the claims travel here; the measurement notes and the feature record do not, and that is deliberate - see below. - `docs/data/roadmap.yaml` moves the `virtual-threads` horizon from `next-0x` to `0.6.0.0`. - `docs/data/module-maturity.yaml` drops `Virtual threads` from `explicit_non_defect_deferrals`, leaving micro-batching and the dead-letter queue, which are still genuinely deferred. - The README's release-posture section now says the processing function can run on virtual threads via the opt-in `useVirtualThreads` option on a JDK 21 runtime, and that the published artifacts are unchanged Java 8 bytecode so only the runtime has to move. README.adoc IS GENERATED and was not hand-edited. The source is `src/docs/README_TEMPLATE.adoc`, rendered by `io.whelk.asciidoc:asciidoc-template-maven-plugin`, bound to `process-sources` on the root pom with `inherited=false` - so `./mvnw -N process-sources` regenerates it. The template was edited and the plugin run; the resulting README.adoc diff is byte-identical to the one in 3456e38, which had edited both files by hand. Worth recording because the campaign branch's hand-edit is exactly the thing that silently drifts the two apart, and it happened to agree here. WHAT IS DELIBERATELY NOT HERE. `docs/features/virtual-threads.yaml` and the two measurement notes it references go to #362 instead, and this is the routing the gates forced rather than a preference. The feature record's `references.path` entries name `docs/inflight/perf-virtual-threads-measured.md` and `docs/inflight/perf-platform-threads-are-the-ceiling.md`, so adding it alone takes `bin/check-docs-data.sh` from clean to two failures. Adding those notes then takes `bin/check-file-refs.sh` from clean to six, because they cite `bench/run-bisect.sh`, `bench/threads/ThreadCeiling.java` and `bench/threads/AsyncCeiling.java` - all of which live on the bench-harness branch. The whole package therefore lands where the files it cites already are. `docs/data/testing-evidence.yaml`'s matching deferral line goes with it, because that line cites the feature record. That also honours what this PR's own description already said: the feature record lands with the engine-comparison documentation, not here. Gates: check-docs-data, check-file-refs, check-inflight-tags, check-issue-refs, check-copyright-headers and check-branch-self-reference all clean, matching the pre-change baseline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…PR that measured them STRATEGY.md is a claims document nothing tests, so its contract is that work which falsifies a claim carries the update. This PR's measurements falsified two, and the updates were stranded on `perf/engine-concurrency` (#363) when the reviewable children were cut out of it. They belong here: this is the branch the numbers came from, and it is the only child that carries the notes they cite. WHAT THE DELTA CHANGES. The Share Groups position is rewritten around what survived. The throughput comparison is WITHDRAWN IN BOTH DIRECTIONS - not softened, withdrawn: the 2.5x against us did not reproduce at its own operating point (66,524 / 29,709 / 11,225 msg/s across three brokers in one day, a 5.9x range, while every PC arm held within 3%), the obvious explanation was tested with a negative control and refuted, and the 14.5% figure that was its counterweight goes with it because it is the same arm and the same variance. What replaces them is structural and was always the real argument: a share consumer cannot poll while records are unacknowledged, so an honest share processor is batch-synchronous, while PC holds records from many polls outstanding - which is what the offset encoding buys. Per-key ordering and retry semantics stay capability differences rather than benchmark results. The end-to-end latency success metric stops saying "not measured today" and names `pc.record.residence.time`, with the boundary it draws (client-side queueing and retries in, produce time and broker wait out) and the gap this harness cannot close: it drains a pre-produced backlog, so at 100% utilisation the number measures the backlog rather than the engine. Recording the gap beside the metric is the point - a success metric that quietly cannot be read at saturation is worse than one that admits it. The two CSVs are the async arms' arrival-capacity and tail-skew data, produced by the barrier-wiring fix this PR carries and stranded with everything else. RECONCILIATION STILL PENDING, and deliberately not attempted here. The `## Other runtimes` section this delta edits does not exist on master - it arrives from the proxy chain (`5ac6b6dc5` on `feats/proxy-requirements`, #293), which is extending the same section in parallel. Applying only this branch's delta means whichever lands second will have a real conflict in that section to resolve by hand. That is the correct outcome and not a mistake to tidy up: reconciling now would mean this PR silently editing the proxy chain's claims, or the proxy chain's, ours. Both sides own their own claims; the merge owns the reconciliation. Gates: check-docs-data and check-issue-refs clean. check-file-refs and check-inflight-tags stay at this branch's pre-existing 25 and 8 respectively - nothing added here appears in either failure list, verified by name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…bulary the gate enforces Port fix, not a content change: not one word of either note's body is touched. Both notes arrived in the previous commit from `perf/engine-concurrency` (#363), carrying `inflight-type: perf` and `inflight-impact: performance`. Neither value exists any more - `bin/lib/inflight-tags.sh` enforces `bug feature task register` for the type, and an impact set in which the performance axis is spelled `throughput`. The campaign branch predates that tightening, so its notes are tagged in a vocabulary master retired. - `perf-platform-threads-are-the-ceiling.md` becomes `bug` + `throughput`. It is the diagnosis of a throughput limitation - reachable concurrency is capped by platform-thread activation rate, proven by a forty-line control with no Kafka and no Parallel Consumer in it - and the impact tag answers what it costs you not to know, which is throughput. - `perf-virtual-threads-measured.md` becomes `feature` + `throughput`. It is the measurement of the capability #360 ships, against Parallel Consumer itself rather than against a control. `docs/inflight/AGENTS.md` says a feature should carry an impact whenever it addresses one, and this one addresses the ceiling above. Sequencing, recorded because the log otherwise misleads: these two files and `docs/features/virtual-threads.yaml` were meant to be their own commit, separate from the STRATEGY claims delta. `git checkout <ref> -- <path>` stages as it writes, so they were swept into the previous commit, whose message describes only the STRATEGY change - and the retags, made in the working tree afterwards, were left behind by the same mechanism. Fixed forward rather than by resetting: the repository's history guard refuses a reset-backwards without an explicit operator decision, and a rewrite is a merge-prep action, not a mid-flight one. Re-cutting the two into their intended commits is a merge-prep offer, not something to take unilaterally now. WHY THE VIRTUAL-THREAD DOCUMENTATION PACKAGE IS ON THIS PR at all, since #360 is the one that ships the option: the gates decided it, not preference. `docs/features/virtual-threads.yaml` names both of these notes in its `references.path` entries, so on #360 it took `bin/check-docs-data.sh` from clean to two failures. Adding the notes there then took `bin/check-file-refs.sh` from clean to six, because they cite `bench/run-bisect.sh`, `bench/threads/ThreadCeiling.java` and `bench/threads/AsyncCeiling.java` - files that exist only here. `docs/data/testing-evidence.yaml`'s deferral line cites the feature record, so it travels with them. #360 keeps the three claims updates that stand alone (roadmap horizon, module-maturity deferral, README) and its own description already said the feature record lands with the engine-comparison documentation. NOT brought across: `docs/plans/2026-08-22-001-feat-virtual-threads-plan.md`. Tested here rather than assumed - moving it to this branch does resolve the bench citations, cutting its dangling paths from 30 to 15, but 15 remain across nine paths, eight of them inflight notes that live only on the campaign branch. Repairing them would mean importing eight more notes with their own cascades, or writing fifteen opt-out markers into a dated plan document. Its citation web points at the campaign branch's evidence base, so it stays with it on #363. Gates: check-docs-data clean (43 files valid, up from 42 - the new feature record and its references both resolve). check-inflight-tags holds at this branch's pre-existing 8 problems and check-file-refs at its pre-existing 25 - neither ported note appears in either failure list, verified by name rather than by count. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… never had Fork-original files committed with no header at all, which bin/check-copyright-headers.sh refuses and which was blocking this PR's CI. They get the plain fork notice per docs/copyright.md - written entirely for the fork, no upstream derivation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
5 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
depends on #361 - direct pull finished, scan made O(1)
Description
The harness that measured everything in this stack, the data it produced, and the conclusions — including the ones that went against us.
bench/run-bisect.shruns one Java source file against every released PC version and against several engines, so a number can be attributed to a version, an engine, an ordering mode, a key distribution, a concurrency, an arrival rate or a work model — rather than to "the benchmark". Eleven arms:core,core-vt,core-dpvt, the four async engines, the Kafka Streams threading model, a plain consumer, a thread-pool floor, and KIP-932 share groups. Three latency families: residence (poll→completion), drain (engine start→completion), and end-to-end (intended send instant→completion) — the last being the only one coordinated omission cannot fool.Why the results ship with the harness rather than as a summary: every headline below is reproducible from the committed CSVs and the command that produced them, and two of them contradict things this project previously said.
KEYordering free — every record becomes its own shard — so 369 of 376 historical result rows measuredUNORDEREDwearingKEY's name. Under a Zipf distribution the same engine on the same records runs at a third of the throughput and sustains one record in flight of a configured 24.coreagainstcoreacross fourteen cells is 0.96×–1.04×: the shipped default did not get faster. What did is the opt-in engines, and only at high concurrency with short handlers.KEYordering holds end-to-end p99 at the handler's own tail (510/509/508ms at 50/70/90% utilisation) while the Kafka Streams threading model amplifies to ~940ms.The instrument defects are documented as prominently as the results, because six of the last eight produced plausible numbers rather than errors: a work model wired to three of five call sites, percentiles rendered after teardown, a per-thread RNG that made virtual threads draw once each, a sustained-in-flight column that read zero on short runs.
docs/solutions/best-practices/benchmark-a-rival-on-the-semantics-it-actually-offers.mdis the method that survived all of it.Dependency, and why it is a dependency at all. The harness compiles against any PC version by design — new options are reached through system properties and reflection, which is what keeps the version bisect working. It stacks on #361 anyway because its arms must be real: on a master without the virtual-thread and direct-pull options,
MODES=core-vtwould silently run platform threads and report the result as if it had not. An arm that lies is worse than an arm that fails, and this harness exists specifically because of that class of error.Known open items, tracked rather than hidden (
docs/inflight/perf-open-work-register.md): the async engines could not run under controlled arrival until a barrier-wiring fix that ships here; Vert.x still has no end-to-end measure (no record context on its completion hook); the share-broker variance is unexplained; and theproxyarm's collapse under key skew is unattributed.Checklist
docs/solutions/docs/features/- N/A - developer tooling, not a product surface