Repository navigation
WIP: Fixes #21 - Dynamic concurrency control - #22
Closed
Antony Stubbs (astubbs) wants to merge 3 commits into
Closed
Antony Stubbs (astubbs) wants to merge 3 commits into
Antony Stubbs (astubbs) wants to merge 3 commits into
Conversation
…y-limits project Uses concepts from tcp congestion control to set the thread pool size by measuring performance of execution. Originally planned to do this by hand using concepts from Engineering Control Theory, but stumbled upon this library, which uses very similar theory from TCP congestion control theory.
…deps for core) - More pluggable execution model
Antony Stubbs (astubbs)
force-pushed
the
features/dynamic-concurrency-control
branch
from
November 5, 2020 17:05
e5ca55d to
a861249
Compare
Antony Stubbs (astubbs)
force-pushed
the
features/dynamic-concurrency-control
branch
from
November 5, 2020 17:23
a861249 to
ba6b71f
Compare
Antony Stubbs (astubbs)
force-pushed
the
master
branch
6 times, most recently
from
November 10, 2020 14:32
42029e9 to
8fe9766
Compare
Antony Stubbs (astubbs)
force-pushed
the
master
branch
5 times, most recently
from
March 30, 2021 10:25
5c841c3 to
bc85ba3
Compare
Antony Stubbs (astubbs)
force-pushed
the
master
branch
5 times, most recently
from
September 9, 2021 09:40
6312a34 to
3ad49c2
Compare
Antony Stubbs (astubbs)
force-pushed
the
master
branch
from
October 13, 2021 16:56
bdc2932 to
84cf11a
Compare
Antony Stubbs (astubbs)
force-pushed
the
master
branch
2 times, most recently
from
February 16, 2022 17:41
4c62a9f to
b5b166f
Compare
Antony Stubbs (astubbs)
force-pushed
the
master
branch
from
March 4, 2022 13:26
5be0137 to
84c7522
Compare
Antony Stubbs (astubbs)
force-pushed
the
master
branch
from
March 23, 2022 13:49
3905c7c to
3dcb9d7
Compare
Antony Stubbs (astubbs)
changed the title
Antony Stubbs (astubbs)
added a commit
to astubbs/parallel-consumer
that referenced
this pull request
Aug 14, 2026
A mirror records what a closed PR said; it does not keep the code. The 35 PRs closed in the 2023-06-15 sweep were reachable through refs/pull/<n>/head in the upstream repository - not a copy we control. If that repo is deleted, or a contributor's fork disappears, the commits go with it and the mirrors describe work whose diff no longer exists. Checked each of the 35 against this fork's refs. 29 were already safe, raised from branches that still exist here. Six were reachable only from upstream, and were not even present in a local clone: confluentinc#22, confluentinc#204, confluentinc#270, confluentinc#405, confluentinc#443 and confluentinc#506. All six are now pinned as annotated archive/upstream-pr-<n> tags carrying the upstream title, author, head branch and closure date, so provenance survives without the upstream thread. confluentinc#443 was the sharpest case - a third-party fork (Robbie-Palmer) can vanish independently of Confluent's repository and nothing would have flagged it. Tags rather than branches: they are not swept by branch-cleanup tooling and read as archival rather than live work. Recorded as a recurring check, since a head safe today can be orphaned tomorrow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2 of 4 tasks
Antony Stubbs (astubbs)
added a commit
to astubbs/parallel-consumer
that referenced
this pull request
Aug 14, 2026
…able Review of this PR found the preservation work sound but the write-up wrong in two places, and unbacked in a third. Corrects the confluentinc#443 rationale, which was fabricated. Its head label is confluentinc:pyallel-consumer and that branch is still live upstream; the contributor's own fork is already gone and made no difference. All 35 swept heads carry confluentinc: labels, so the exposure is upstream branch or repository loss, not a contributor's fork vanishing. The original text inferred the risk from the PR author field without checking the head repo. Corrects "29 were already safe, because their PRs were raised from branches that still exist on this fork". Raised-from is not the criterion that was used or that holds: confluentinc#271's own branch is gone and its head survives only because an unrelated branch contains it, while confluentinc#22, confluentinc#270 and confluentinc#405 have same-named fork branches that do NOT contain their heads. The wording is now reachable-from, and the method records that stale remote-tracking refs were reconciled against a live ls-remote. Records the six SHAs in upstream-map.yaml under sweep-2023-admin-closure with the tag names and a re-verify command, so the containment check can be redone without re-querying upstream - PR numbers alone did not allow that. The cohort comment there said all heads "were verified reachable", which this work disproved for six of them; it now says what was actually found. Stops claiming a recurring check that nothing performs. --audit covers tracking and mirroring, not reachability, and would report clean with every archive tag deleted. Named as a manual step until a containment check is wired in. Also drops the fork-network durability implication (deleting a public parent re-roots the network rather than destroying objects, so the tags do discharge the stated threat), trims the refactoring entry to the file's one-line prior-art footnote style since fork-upstream mapping is explicitly out of scope there, and fixes which method holds the user-function try/catch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Antony Stubbs (astubbs)
added a commit
to astubbs/parallel-consumer
that referenced
this pull request
Aug 14, 2026
Restores the six-row table removed in the previous commit. The review finding was that the tag/PR/SHA tuples had two owners and a corrected SHA could drift - not that a table was the wrong shape. Deleting it was an over-correction; six rows are more scannable than a prose list. The table now carries only what does not change - upstream PR, author, and what the PR was. The tag name, target SHA and check date stay single-owner in sweep-2023-admin-closure.preserved_heads. The descriptions are the real upstream titles, fetched rather than recalled. The first draft of this table invented them and got three wrong, including pasting confluentinc#45's title onto confluentinc#22 - the same fabrication-from-memory this PR exists to correct. Embedded upstream refs are dropped from the titles so the issue-reference gate stays green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Antony Stubbs (astubbs)
added a commit
to astubbs/parallel-consumer
that referenced
this pull request
Aug 17, 2026
…nk branch recovery Extends the throttling ideation with idea 8: one adaptive controller (#227 / confluentinc#21, TCP-congestion-theory concurrency) fed by latency gradient, failures, and a structured rate-limit exception the user function throws to teach the engine per-service ceilings at runtime. Capacity limits are discoverable; contractual quotas are not - explicit ceilings and adaptive discovery compose by min(), not compete. Also: idea 5 gains the curated strategy-menu framing (each strategy a distinct situation, min-composition, enforcement not user-selectable); docs/refactoring.md idea bank gains the three missing flow-control branches (dynamic-concurrency-control @6f85eac41, auto-tuning-pressure @f4aa09788, rate-limiting @e9f49d321) plus upstream draft PR confluentinc#22; the inflight entry now records the standalone-vs- self-scaling gating decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fq6AcSoq6nMqbRHQxw25wx
4 tasks done
Antony Stubbs (astubbs)
added a commit
to astubbs/parallel-consumer
that referenced
this pull request
Aug 20, 2026
Three records held the same commits under different framings. `preserved_branch_tips` listed upstream branch tips pinned as tags. `sweep-2023-admin-closure.preserved_heads` listed swept PR heads pinned as tags. `branch_accounting` listed the branches. They were not three views of related things - they were the same SHAs. Every one of the six preserved PR heads is the tip of an upstream branch: confluentinc#443 and the `pyallel-consumer` tip are two names for 4533f6d, confluentinc#22 and `features/dynamic-concurrency-control` for ba6b71f, and so on for all six. Ten more overlapped between the branch-tip record and the mirror. That is the drift docs/upstream.md already warns about in its own words - a corrected SHA fixed in one copy while another still reads as authoritative - present in the file whose job is to stop mappings being re-derived. So the tag and the PR become FIELDS on the branch entry, and the other two records are gone. The tags themselves are kept: a branch can be deleted or force-moved, an annotated tag is the record that a specific commit was deliberately preserved. What changed is that the fact is written once. The reasoning from both deleted sections is absorbed into the section header rather than lost - why tags outlive branches, and how to verify them without fetching. Every reference in docs/upstream.md and the two inflight notes is repointed, and the claim that the bot branches "were deliberately not preserved" is corrected: they were not tagged, but they are mirrored now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SFRbSkHwaRw6YYFHHBWZ7p
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #21
Prevent user from having to choose a level of concurrency with either the core engine or the vertx engine.
Uses concepts from tcp congestion control to set the thread pool size by measuring performance of execution.
Originally planned to do this by hand using concepts from Engineering Control Theory, but stumbled upon Netflix's concurrency-limits library, which uses very similar theory from TCP congestion control theory.