diff --git a/AGENTS.md b/AGENTS.md index da4f9ad738..3eddfb7bee 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -84,8 +84,9 @@ is untracked (a whole triage doc was once written duplicating `docs/refactoring. | [`docs/self-hosted-runner.md`](docs/self-hosted-runner.md) | Setting up or operating the self-hosted highcpu runner | | [`docs/agent-harness.md`](docs/agent-harness.md) | Adding a rule you need agents to follow *reliably* - which layers fire on their own, and which are merely available | | [`docs/merge-checklist.md`](docs/merge-checklist.md) | Getting a PR ready to merge - what to offer the author, including the squash message and reorganising the commits | -| [`bin/AGENTS.md`](bin/AGENTS.md) | Writing or changing a script in `bin/` - the shell conventions, including the ones no check enforces | +| [`bin/AGENTS.md`](bin/AGENTS.md) | Writing or changing **any shell script** - `bin/`, `.githooks/`, `.claude/hooks/` - the shell conventions, including the cross-platform rule and the others no check enforces | | [`docs/inflight/AGENTS.md`](docs/inflight/AGENTS.md) | Adding, editing or retiring a note in `docs/inflight/` - what may live there, the tag vocabulary, and where a note's content goes when its work lands | +| [`parallel-consumer-proxy-clients/AGENTS.md`](parallel-consumer-proxy-clients/AGENTS.md) | Changing a client or its demo - the two test suites over them, which one your feature needs, and why neither substitutes for the other | | [`parallel-consumer-core/src/main/java/bz/stub/parallelconsumer/AGENTS.md`](parallel-consumer-core/src/main/java/bz/stub/parallelconsumer/AGENTS.md) | Changing a field in the engine - the `@GuardedBy` rule, the known shared state and its ledgers, and the shard-map pin | **A directory with its own `AGENTS.md` owns the rules for what goes in it - read it before you write @@ -292,9 +293,28 @@ counts. This is a community-maintained fork of the no-longer-maintained ## Build Requirements -- **JDK 17** (the project uses Jabel to compile Java 17 source to Java 8 bytecode) +- **JDK 17** (the project uses Jabel to compile Java 17 source to Java 8 bytecode). On JDK 21 the + build dies as `Unable to delombok: InvocationTargetException` in `parallel-consumer-core` - a + module your change never touched, so it reads as a real breakage and is not one. Check + `java -version` first. Set it **per command** (`JAVA_HOME=... ./mvnw`), never by repointing a + shared default that every other session inherits. - **Docker** (integration tests - TestContainers spins up Kafka brokers) - **Maven via wrapper** (`./mvnw`) - do not use system Maven +- **[mise](https://mise.jdx.dev)**, for the cross-language conformance module and the client + builds. `mise install` reads **`mise.toml`**, which is the single declaration of every toolchain + version; the CI matrix mirrors it and `bin/check-toolchain-versions.sh` fails if the two ever + disagree. It provides every foreign toolchain except the two that build in containers + (`bin/build-client.sh` names them: C++ and Swift). **Agents may install it and run `mise use -g` + themselves** - that call is recorded in + [`docs/inflight/parked-containerised-toolchains-and-runtime.md`](docs/inflight/parked-containerised-toolchains-and-runtime.md), + along with the rule that follows from it: *do not build an image fleet for problems mise has + already solved*. + + Without mise the build falls through to whatever the host has, and the conformance module then + **fails** - correctly, because it refuses to let a runner nobody could build look like a runner + that passed. The two failure modes read as project breakage and are not: Go reports + `go.mod: unknown block type: tool` when `go env GOTOOLCHAIN` is `local` rather than its default + `auto`, and Ruby reports a missing `bundler` when macOS's system 2.6 is on `PATH`. ## How to Build diff --git a/bin/AGENTS.md b/bin/AGENTS.md index 6c29f5a4b8..2d1c82a01a 100644 --- a/bin/AGENTS.md +++ b/bin/AGENTS.md @@ -1,8 +1,9 @@ # `bin/` - how these scripts relate to CI and to the reviewer Repo scripts. This doc owns the conventions for writing a script in `bin/`; the root AGENTS.md -routes here and keeps only what binds every session. Two conventions live here because nothing else -enforces them. +routes here and keeps only what binds every session. These conventions live here because nothing +else enforces them. The portability rule below is the exception to the directory in the title: it +binds **every shell script in the repo**, including `.githooks/` and `.claude/hooks/`. ## Naming a script here can grant it to the PR reviewer @@ -61,6 +62,33 @@ Two structural guards exist and are worth copying into any new checker's self-te buried mid-body and a match at the very end, and neither can trigger the bug - the first is small, the second has nothing following it. A case that reaches it is added in astubbs#210. +## Every script must be cross-platform, and degrade rather than guess + +Scripts here run on macOS dev machines, Linux CI runners, and self-hosted runners, so **a script that +only works on the author's machine is a broken script**. Two rules, and the second is the one that +actually bites: + +- **Never let a missing tool become a confident wrong answer.** Probe for what you need and skip what + is absent, saying so. `docker`, `gdate`, `realpath`, `timeout` and `sha256sum` are all absent + somewhere this repo runs. A check that cannot run must report that it did not run - the failure + mode to design against is the one where "I could not measure it" is indistinguishable from "it is + fine". + +- **`stat -f` is NOT a portable idiom, and must never be used as a try-then-fallback.** On macOS + `stat -f %b` is a *file's* allocated blocks; on GNU/Linux `-f` switches stat into *filesystem* + mode, where `%b` is the total blocks in the filesystem. It exits **0** and returns a number three + orders of magnitude too large, so `stat -f ... || stat -c ...` silently produces garbage on Linux + rather than falling through. Resolve the platform once from `uname -s` and branch. The same + divergence applies to `date -r` vs `date -d`, `sed -i ''` vs `sed -i`, and `readlink -f`. + + `.claude/hooks/warn-low-disk.sh` is the worked example, and it also shows the seam that makes such + a branch testable: which `stat` **syntax** to speak is read from the real `uname` and is never + injectable, while which platform **layout** to look for is injectable - so a self-test can exercise + the macOS branch on a Linux runner without pretending Linux has BSD stat. + +- **`df` needs `-P`.** Without POSIX output a long device name wraps onto its own line on Linux and a + column-indexing `awk` silently reads the wrong field. + ## Workflows **One version per action.** `bin/check-action-versions.sh` fails if any action appears at two @@ -78,6 +106,20 @@ caught it in review, and the same mistake had been seen before. The github-actions ecosystem was re-enabled in `.github/dependabot.yml` at the same time; it had been off since 2022, when the repo genuinely had no workflows. +**One toolchain declaration.** `bin/check-toolchain-versions.sh` fails when `mise.toml` and the +`.github/workflows/clients.yml` matrix disagree about a language's version, and runs in Repo Hygiene +behind its own self-test. The two *install* differently on purpose - mise locally, `setup-*` actions +on the runner, which carry the client matrix's caching and a SHA-pinned `ruby/setup-ruby` - but they +must never *declare* differently. Same class as the rule above: **nothing else could catch it**, and +they had already drifted by whole major versions (dotnet 8 against 9, node 22 against 25) before the +gate existed. The four languages with no host toolchain (swift and cpp build in containers, kotlin +and scala on the Maven reactor) are asserted absent from both, so "nobody declared it" cannot be +mistaken for "it deliberately has none". + +`bin/build-client.sh` asserts the *installed* version separately, at build time, on major.minor +only - the declarations are exact, but a developer one patch along from the pin is not a problem +worth blocking, and every failure the assertion exists for was a major or minor gap. + **Temporary CVE exclusions expire.** `bin/check-cve-exclusions.sh` parses the root pom's `excludeVulnerabilityIds` and fails once an entry marked `TEMPORARY-SINCE: YYYY-MM-DD` is more than 90 days old - also on an undated, unparseable or future-dated marker, and on an id with no rationale diff --git a/bin/build-client.sh b/bin/build-client.sh index 7f09e0dcdf..f0d3d56840 100755 --- a/bin/build-client.sh +++ b/bin/build-client.sh @@ -39,6 +39,8 @@ set -euo pipefail +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +readonly REPO_ROOT readonly CLIENTS_DIR="parallel-consumer-proxy-clients" readonly MODULE_PREFIX="parallel-consumer-proxy-client-" readonly PROTO_CONTEXT="parallel-consumer-proxy-protocol/src/main/proto" @@ -170,12 +172,97 @@ maven_module_list() { # -> comma-separated :artifactId list printf '%s' "$list" } +# ───────────────────────────────────────────────────────────────────────────────────────────────── +# TOOLCHAIN VERSION ASSERTION +# +# Maven shells out to whatever `go`, `ruby`, `node`, `python3`, `rustc` or `dotnet` is on PATH, and +# a wrong one does not announce itself - it produces an error deep inside a module nobody touched. +# Both real cases cost an hour each before anyone suspected their PATH: Go 1.23 against a 1.25 +# module reports `go.mod: unknown block type: tool`, and macOS's system Ruby 2.6 reports a missing +# `bundler`. Neither says "your toolchain is the wrong version", which is the only useful sentence. +# +# MAJOR.MINOR, NOT EXACT. mise.toml and CI pin an exact patch and `bin/check-toolchain-versions.sh` +# holds those two declarations identical - but asserting the patch HERE would fail every developer +# whose mise resolved one release along (this was written on a box with go 1.25.14 against a pinned +# 1.25.13) while catching nothing: every failure this guard exists for was a major or minor gap. +# +# A version it cannot read is a FAILURE, not a pass. "I could not check" and "it is fine" must never +# look the same - the rule bin/AGENTS.md states for every script here. +# ───────────────────────────────────────────────────────────────────────────────────────────────── + +# The runtime each language builds with. `typescript` is the one place the language and the tool have +# different names, matching the mapping in bin/check-toolchain-versions.sh. +tool_for_language() { # -> tool name, or empty when the language has no host toolchain + case "$1" in + typescript) printf 'node' ;; + go | ruby | rust | python | dotnet) printf '%s' "$1" ;; + *) printf '' ;; + esac +} + +declared_version() { # -> the version mise.toml pins, or empty + local mise="$REPO_ROOT/mise.toml" + [ -f "$mise" ] || return 0 + sed -n "s/^[[:space:]]*$1[[:space:]]*=[[:space:]]*\"\([^\"]*\)\".*/\1/p" "$mise" | head -1 +} + +active_version() { # -> the version actually on PATH, or empty when it cannot be determined + case "$1" in + go) go version 2>/dev/null | sed -n 's/.*go\([0-9][0-9.]*\).*/\1/p' ;; + ruby) ruby --version 2>/dev/null | sed -n 's/^ruby \([0-9][0-9.]*\).*/\1/p' ;; + rust) rustc --version 2>/dev/null | sed -n 's/^rustc \([0-9][0-9.]*\).*/\1/p' ;; + python) python3 --version 2>/dev/null | sed -n 's/^Python \([0-9][0-9.]*\).*/\1/p' ;; + node) node --version 2>/dev/null | sed -n 's/^v\([0-9][0-9.]*\).*/\1/p' ;; + dotnet) dotnet --version 2>/dev/null | sed -n 's/^\([0-9][0-9.]*\).*/\1/p' ;; + *) printf '' ;; + esac +} + +major_minor() { # -> first two components + printf '%s' "$1" | cut -d. -f1,2 +} + +assert_toolchain_version() { # + local language="$1" tool declared active + tool="$(tool_for_language "$language")" + [ -n "$tool" ] || return 0 + + declared="$(declared_version "$tool")" + if [ -z "$declared" ]; then + die 1 "$language: mise.toml pins no version for $tool, so the toolchain cannot be checked. + Add it, or remove $language from tool_for_language() and say why it has no host toolchain." + fi + + active="$(active_version "$tool")" + if [ -z "$active" ]; then + die 1 "$language: could not determine the installed $tool version - is it on PATH? + mise.toml pins $tool $declared; install it with 'mise install' (or 'mise use -g $tool@$declared'). + Refusing to build rather than guess: an unreadable toolchain is not a working one." + fi + + if [ "$(major_minor "$active")" != "$(major_minor "$declared")" ]; then + die 1 "$language: wrong $tool version. + on PATH : $active + mise.toml : $declared + Install the pinned one with 'mise install', or 'mise use -g $tool@$declared'. + Building anyway produces an error inside a module you did not touch - Go reports + 'unknown block type: tool', Ruby reports a missing bundler - which reads as project breakage." + fi + + if [ "$active" != "$declared" ]; then + printf '==> %s: %s %s on PATH, %s pinned - same minor, continuing\n' \ + "$language" "$tool" "$active" "$declared" + fi +} + build_natively() { # local phase="$1" local module_dir="$CLIENTS_DIR/$MODULE_PREFIX$LANGUAGE" local projects projects="$(maven_module_list "$module_dir")" + assert_toolchain_version "$LANGUAGE" + printf '==> %s: native toolchain via Maven (%s), projects %s\n' "$LANGUAGE" "$phase" "$projects" # No `clean`, deliberately: several agents and worktrees share this tree, and a clean here would # delete output nobody asked this script to rebuild. diff --git a/bin/check-toolchain-versions.sh b/bin/check-toolchain-versions.sh new file mode 100755 index 0000000000..8d8f1cf1cd --- /dev/null +++ b/bin/check-toolchain-versions.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash +# +# Copyright (C) 2026 Antony Stubbs and contributors +# + +# Asserts that `mise.toml` and the `.github/workflows/clients.yml` matrix declare the SAME toolchain +# versions. +# +# WHY IT EXISTS. The two install differently on purpose - mise locally, `setup-*` actions in CI, +# because those actions carry the client matrix's caching and `ruby/setup-ruby` is SHA-pinned. What +# is not on purpose is that they were two sources of truth for the same numbers with nothing +# comparing them, and they had already drifted by WHOLE MAJOR VERSIONS before anyone looked: dotnet +# 8.0.404 in CI against 9.0.101 on a developer machine, node 22.17.0 against 25.9.0. That is the +# classic shape of "passes locally, fails in CI" - and the nastier reverse, where CI runs the older +# runtime and the developer never sees the failure their change causes. +# +# WHAT IT DOES NOT DO. It does not check what is INSTALLED - that is +# `assert_toolchain_version` in bin/build-client.sh, which runs at build time against the machine. +# This one compares two declarations, so it is exact: a patch difference between two files that are +# both meant to say the same thing is a mistake, not tolerable drift. +# +# THE FOUR ABSENT LANGUAGES ARE CHECKED TOO, because "nobody declared it" and "it deliberately has no +# host toolchain" look identical otherwise: swift and cpp build in a container, kotlin and scala on +# the Maven reactor, so all four must be absent from mise.toml AND carry an empty matrix toolchain. +# +# Read-only: no network, no writes, nothing outside this repository. Safe under the `check-*` grant +# described in bin/AGENTS.md. +# +# Run: bin/check-toolchain-versions.sh + +set -uo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +# The two overrides exist only for bin/test-check-toolchain-versions.sh, which has to drive this +# gate against crafted fixtures to prove it can go red. bin/AGENTS.md's rule is that a regression +# test which has never failed proves nothing, and the only other way to make this one fail is to +# break the real repository. +MISE="${PC_MISE_TOML:-$REPO_ROOT/mise.toml}" +WORKFLOW="${PC_CLIENTS_WORKFLOW:-$REPO_ROOT/.github/workflows/clients.yml}" + +for f in "$MISE" "$WORKFLOW"; do + [ -f "$f" ] || { + printf 'check-toolchain-versions: MISSING %s - cannot compare, so this is a FAILURE rather than a pass\n' "$f" >&2 + exit 1 + } +done + +# python3 rather than a yaml/toml parser: it is already a hard dependency of the agent hooks and the +# conformance harness, and adding a parser dependency to a gate that guards the build is backwards. +python3 - "$MISE" "$WORKFLOW" <<'PY' +import re +import sys + +mise_path, workflow_path = sys.argv[1], sys.argv[2] + +# The matrix names the LANGUAGE; mise names the RUNTIME. They differ in exactly one place, and +# hard-coding that one mapping is clearer than inferring it. +LANGUAGE_TO_TOOL = {"typescript": "node"} + +# Languages with no host toolchain by design. Named here so that a language quietly losing its +# version is a failure rather than joining this list by accident. +NO_HOST_TOOLCHAIN = {"swift", "cpp", "kotlin", "scala"} + +mise_text = open(mise_path).read() +tools_block = re.search(r"^\[tools\]\s*$(.*?)(?=^\[|\Z)", mise_text, re.M | re.S) +if not tools_block: + print("check-toolchain-versions: mise.toml has no [tools] section", file=sys.stderr) + raise SystemExit(1) +declared = dict(re.findall(r'^\s*([A-Za-z0-9_-]+)\s*=\s*"([^"]+)"', tools_block.group(1), re.M)) + +workflow_text = open(workflow_path).read() +entries = re.findall(r"- language:\s*(\S+)(.*?)(?=\n\s*- language:|\Z)", workflow_text, re.S) +matrix = {} +for language, body in entries: + found = re.search(r"toolchain:\s*'([^']*)'", body) + if found is not None: + matrix[language] = found.group(1) + +if not matrix: + print("check-toolchain-versions: parsed NO languages out of the workflow matrix - the format " + "changed, and a gate that silently checks nothing is worse than no gate", file=sys.stderr) + raise SystemExit(1) + +problems = [] + +for language, ci_version in sorted(matrix.items()): + tool = LANGUAGE_TO_TOOL.get(language, language) + if language in NO_HOST_TOOLCHAIN: + if ci_version: + problems.append(f"{language}: builds without a host toolchain, but the CI matrix pins " + f"'{ci_version}'. Either it now needs one (add it to mise.toml) or the " + f"pin is stale.") + if tool in declared: + problems.append(f"{language}: builds without a host toolchain, but mise.toml declares " + f"{tool} = \"{declared[tool]}\".") + continue + if not ci_version: + problems.append(f"{language}: the CI matrix declares no toolchain version. If that is " + f"deliberate, add it to NO_HOST_TOOLCHAIN in this script and say why.") + continue + if tool not in declared: + problems.append(f"{language}: CI pins '{ci_version}' but mise.toml declares no {tool}.") + continue + if declared[tool] != ci_version: + problems.append(f"{language}: mise.toml says {tool} = \"{declared[tool]}\", the CI matrix " + f"says '{ci_version}'. These must be identical.") + +matrix_tools = {LANGUAGE_TO_TOOL.get(lang, lang) for lang in matrix} +for tool in sorted(set(declared) - matrix_tools): + problems.append(f"mise.toml declares {tool} = \"{declared[tool]}\", which no language in the CI " + f"matrix uses. Remove it, or add the language that needs it.") + +if problems: + print("check-toolchain-versions: the two declarations disagree.\n", file=sys.stderr) + for problem in problems: + print(f" - {problem}", file=sys.stderr) + print("\nmise.toml is the single declaration; the workflow matrix must mirror it. They install\n" + "differently on purpose - see the header of mise.toml.", file=sys.stderr) + raise SystemExit(1) + +checked = len(matrix) +print(f"check-toolchain-versions: {checked} language(s) agree between mise.toml and the CI matrix " + f"({len(declared)} pinned, {len(NO_HOST_TOOLCHAIN & set(matrix))} deliberately without a host toolchain).") +PY diff --git a/bin/ci-demo-conformance.sh b/bin/ci-demo-conformance.sh new file mode 100755 index 0000000000..95dc955591 --- /dev/null +++ b/bin/ci-demo-conformance.sh @@ -0,0 +1,208 @@ +#!/usr/bin/env bash +# +# Copyright (C) 2026 Antony Stubbs and contributors +# +# DOES EVERY LANGUAGE'S DEMO STILL BEHAVE THE SAME? (astubbs#242, plan unit U35.) +# +# The demo contract - parallel-consumer-proxy/demo/README.md - exists so that "a visitor moving +# between two languages' demos sees the same behaviour". Read that contract and notice what almost +# every clause is about: the flags, the defaults, the fingerprint printed first, the bootstrap +# address never printed, two tables with the same columns in the same order, the big replay dropping +# the serial arm, no latency anywhere. **It is a contract about standard output.** So it can be +# tested from standard output, in one place, without knowing a single thing about any of the eleven +# languages. +# +# THE TRICK THAT MAKES THIS CHEAP: THE LANGUAGES ARE EACH OTHER'S ORACLE. +# Rather than eleven expected-output files that rot, this reduces each demo's stdout to a SKELETON - +# the dials it echoed, the table headers, and the arm names in order, with every volatile figure +# discarded - and then requires the skeletons to be IDENTICAL to each other. Nothing to maintain, +# and it gets stronger with each language added rather than weaker. Ten agreeing and one differing +# points straight at the line that drifted. +# +# A skeleton diff is clean if EVERYONE drifts the same way, so it is paired with absolute assertions +# that hold regardless of agreement: the fingerprint echoes the input it was given, the broker +# address appears nowhere, and no latency is reported. +# +# WHAT THIS DELIBERATELY CANNOT SEE: whether a demo's sidecar arm truly goes through its client +# library rather than hand-rolled gRPC. Stdout cannot show that; code review and the conformance +# suite own it. A green run here is a drift check, not a correctness proof - do not read it as one. +# +# COLUMN WIDTH IS NOT CONTRACT, on purpose. `typescript-grpc` is longer than `java-grpc`, so a +# language with a long arm name would be in permanent violation of an alignment rule. Column +# IDENTITY and ORDER are contract; the skeleton keeps those and discards padding. +set -euo pipefail + +cd "$(git rev-parse --show-toplevel)" + +CLIENTS="parallel-consumer-proxy-clients" +WORK="$(mktemp -d)" + +# Identical input for every language, small enough to prove behaviour rather than measure it. +COMMON=(--records 20 --concurrency 4 --partitions 2 --replay-factor 2) +# --delay-ms is deliberately ABSENT from the flags above and supplied only here, so a passing run +# proves the environment layer rather than merely exercising it. That is the layer the Java seed +# silently dropped on its container path until a language agent reading the compose file caught it, +# and it is invisible from the native path - so nothing else would notice it breaking again. +export PC_DEMO_DELAY_MS=3 + +fail() { echo "ci-demo-conformance: $1" >&2; exit 1; } +note() { echo "ci-demo-conformance: $1"; } + +# The whole language-agnostic part. Turns one demo's stdout into a canonical skeleton: +# DIAL = - the effective-configuration fingerprint +# TITLE - a replay heading, volume masked +# HEADER - a results table's column row +# ROW - one arm's row, every figure discarded +# Everything else is noise (log lines, broker chatter, compose prefixes) and is dropped. +skeleton() { + sed -E -e 's/\x1b\[[0-9;]*m//g' -e 's/^[a-z0-9_-]*-[0-9]* *\| *//' "$1" \ + | sed -E -n \ + -e 's|^[[:space:]]*topic = .*|DIAL topic = |p' \ + -e 's|^[[:space:]]*([A-Za-z]+ = .*)$|DIAL \1|p' \ + -e 's|^[[:space:]]*Small replay.*|TITLE Small|p' \ + -e 's|^[[:space:]]*Big replay.*|TITLE Big|p' \ + -e 's|^[[:space:]]*arm[[:space:]]+records[[:space:]]+keys[[:space:]]+elapsed[[:space:]]+msg/s[[:space:]]+vs AK core.*|HEADER arm records keys elapsed msg/s vs-AK-core|p' \ + -e 's|^[[:space:]]*(.*[A-Za-z)])[[:space:]]+([0-9,]+)[[:space:]]+([0-9,]+)[[:space:]]+[0-9.]+s[[:space:]]+[0-9,-]+[[:space:]]+[0-9.]+x[[:space:]]*$|ROW \1 records=\2 keys=\3|p' +} + +normalise_arms() { + # The arm names differ by design - each names its own client - so they are reduced to their ROLE + # before comparison. The records and keys figures are NOT masked: they are deterministic, so + # they are exactly the part worth comparing across languages, and masking them was the mistake. + sed -E -e 's/^ROW AK core \(.*\) /ROW AK-CORE /' \ + -e 's/^ROW (pc-)?[a-z0-9-]+-(grpc|sidecar) \(.*\) /ROW SIDECAR /' \ + -e 's/^ROW (pc-)?[a-z0-9-]+-(grpc|sidecar) /ROW SIDECAR /' \ + -e 's/^ROW AK core /ROW AK-CORE /' "$1" +} + +# PC_DEMO_LANGUAGES restricts the set - a comma-separated list. Every language by default; a subset +# lets CI shard this across jobs and lets a developer prove the mechanism without eleven image +# builds. A restricted run reports what it left out, because a narrowed run that reads like a full +# one is the failure mode this whole script exists to prevent. +languages=() +for dir in "$CLIENTS"/parallel-consumer-proxy-client-*/demo/run.sh; do + [ -f "$dir" ] || continue + lang="$(basename "$(dirname "$(dirname "$dir")")")" + lang="${lang#parallel-consumer-proxy-client-}" + if [ -n "${PC_DEMO_LANGUAGES:-}" ]; then + case ",${PC_DEMO_LANGUAGES}," in *",$lang,"*) ;; *) continue ;; esac + fi + languages+=("$lang") +done +[ ${#languages[@]} -gt 0 ] || fail "found no /demo/run.sh at all" +note "found ${#languages[@]} demo(s): ${languages[*]}" +[ -z "${PC_DEMO_LANGUAGES:-}" ] || note "RESTRICTED to PC_DEMO_LANGUAGES=${PC_DEMO_LANGUAGES} - this run does not cover the rest" + +ran=(); skipped=() +for lang in "${languages[@]}"; do + run_sh="$CLIENTS/parallel-consumer-proxy-client-$lang/demo/run.sh" + log="$WORK/$lang.log" + note "=== $lang ===" + # The CONTAINER path for every language, deliberately: it is the one runtime every demo has, so + # this needs Docker and no language toolchain at all - which is the same promise the demos make + # to a reader. + if ! "$run_sh" --docker "${COMMON[@]}" > "$log" 2>&1; then + skipped+=("$lang(run failed)") + note "$lang SKIPPED - its container run exited non-zero; see $log" + continue + fi + # Skeleton from the DEMO's lines only. A compose capture interleaves the broker's log, and a + # broker prints `listeners = ...` which looks exactly like a fingerprint dial - so an unscoped + # skeleton reported drift between two demos that were identical, and the difference was their + # brokers' listener config. Same mistake as the credential check below, made twice. + if grep -q "^demo-1 *|" "$log"; then + sed -n '/^demo-1 *|/p' "$log" > "$log.demo" + else + cp "$log" "$log.demo" + fi + skeleton "$log.demo" > "$WORK/$lang.skel" + # A NON-EMPTY SKELETON IS NOT ENOUGH, and this exact hole was found by two language agents + # running this function over their own output: when the table format changed, the fingerprint + # still parsed, so the skeleton was non-empty, the skip never fired, the absolute assertions + # passed - and the drift check silently narrowed to comparing two headings. A green run then + # meant the tables were no longer checked at all. A demo without a parsed table now FAILS. + if [ ! -s "$WORK/$lang.skel" ]; then + fail "$lang: produced no recognisable output at all; see $log" + fi + if ! grep -q "^HEADER " "$WORK/$lang.skel"; then + fail "$lang: no results-table HEADER was parsed. Either the demo stopped printing one, or + its columns changed and this script's pattern did not - and the second is the dangerous + one, because it makes every language look identical. See $WORK/$lang.skel" + fi + if ! grep -q "^ROW " "$WORK/$lang.skel"; then + fail "$lang: a table header was parsed but no arm rows were; see $WORK/$lang.skel" + fi + ran+=("$lang") +done + +[ ${#ran[@]} -gt 0 ] || fail "no demo produced usable output; nothing was compared" + +# --- absolute assertions, which hold whether or not the languages agree ----------------------- +for lang in "${ran[@]}"; do + skel="$WORK/$lang.skel" + grep -q "^DIAL records = 20$" "$skel" \ + || fail "$lang: the fingerprint did not echo the records it was given - see $skel" + grep -q "^DIAL delayMs = 3$" "$skel" \ + || fail "$lang: PC_DEMO_DELAY_MS=3 did not reach the demo, and no flag supplied it - the + environment layer is broken; see $skel" + grep -q "^TITLE Small$" "$skel" || fail "$lang: printed no small-replay table" + grep -q "^TITLE Big$" "$skel" || fail "$lang: printed no big-replay table at --replay-factor 2" + # THE DEMO'S OWN LINES ONLY, and that scoping is the whole correctness of this check. A + # compose run interleaves the broker's log with the demo's, and a Kafka broker legitimately + # prints its own listener configuration - which is not the demo leaking anything. Checking the + # raw capture called Go a liar on this harness's first run; the offending line was broker-1's. + cp "$WORK/$lang.log.demo" "$WORK/$lang.demo-only" + + # the address is credential-grade (R48); it must appear nowhere the DEMO prints, not merely be + # omitted from the fingerprint block + if grep -qiE "bootstrap[ ._]servers?[ =:]" "$WORK/$lang.demo-only"; then + fail "$lang: the broker address appears in the demo's own output - the credential rule binds + the whole run, not only the fingerprint; see $WORK/$lang.demo-only" + fi + if grep -qiE "\b(latency|p99|p95|percentile)\b" "$WORK/$lang.demo-only"; then + fail "$lang: reported latency, which the contract forbids - the workload is closed-loop" + fi + note "$lang: absolute assertions pass" +done + +# --- the drift check: every language's skeleton must match every other's ------------------------ +# Java is compared separately: it is documented as carrying extra diagnostic arms no other language +# has or needs, so requiring it to match would enforce the opposite of the contract. +# EXTRA ARMS ARE LEGITIMATE, and requiring identical skeletons made them impossible - which put +# this script in direct conflict with the contract it enforces. The contract invites a language with +# more than one serious Kafka client to run both, and Go declined to add a sarama arm for exactly +# this reason: a third row in one language would have read as permanent drift. So the comparison is +# over the arms the contract REQUIRES - the serial one and the sidecar one - and any additional row +# is allowed and REPORTED rather than silently tolerated, so nobody can hide a drifted arm by +# calling it an extra. +required_only() { grep -E "^(DIAL|TITLE|HEADER|ROW (AK-CORE|SIDECAR))" "$1"; } +extra_arms() { grep -E "^ROW " "$1" | grep -vE "^ROW (AK-CORE|SIDECAR)" || true; } + +# Java is NOT exempt any more. It used to be, because it carries diagnostic arms no other language +# has - but that is exactly what the required-arms comparison above now handles: its AK core and +# sidecar arms are compared like everyone else's, and pc-core, java-direct, java-grpc-uds and +# java-raw-grpc are reported as extras. The seed being outside its own contract's check was never +# defensible; it is the one demo most likely to drift, since it is where changes start. +reference=""; drifted=0 +for lang in "${ran[@]}"; do + normalise_arms "$WORK/$lang.skel" > "$WORK/$lang.norm.all" + required_only "$WORK/$lang.norm.all" > "$WORK/$lang.norm" + if [ -s "$WORK/$lang.extra" ] || extra_arms "$WORK/$lang.norm.all" > "$WORK/$lang.extra"; then + [ -s "$WORK/$lang.extra" ] && note "$lang also runs extra arm(s): $(sed 's/ records=.*//; s/^ROW //' "$WORK/$lang.extra" | tr '\n' ' ')" + fi + if [ -z "$reference" ]; then reference="$lang"; continue; fi + if ! diff -u "$WORK/$reference.norm" "$WORK/$lang.norm" > "$WORK/$lang.diff"; then + echo "ci-demo-conformance: DRIFT between $reference and $lang:" >&2 + sed 's/^/ /' "$WORK/$lang.diff" >&2 + drifted=1 + fi +done +[ "$drifted" -eq 0 ] || fail "the demos no longer behave the same; see the diffs above" + +note "ran: ${ran[*]}" +[ ${#skipped[@]} -eq 0 ] || note "SKIPPED (not proven, not passed): ${skipped[*]}" +if [ -n "$reference" ]; then + note "every compared language matches '$reference' after masking" +else + note "only one comparable demo ran; drift could not be checked" +fi diff --git a/bin/run-conformance-in-container.sh b/bin/run-conformance-in-container.sh new file mode 100755 index 0000000000..3b0abc20d4 --- /dev/null +++ b/bin/run-conformance-in-container.sh @@ -0,0 +1,129 @@ +#!/usr/bin/env bash +# +# Copyright (C) 2026 Antony Stubbs and contributors +# + +# Runs the cross-language conformance suite for one container-toolchain language INSIDE that +# language's own image. +# +# WHY THIS EXISTS. `bin/build-client.sh` builds C++ and Swift in a container and extracts the +# artifacts to the host. That is sound when the host is Linux, and impossible when it is not: the +# artifact is a Linux binary, and on macOS the shell answers **126** - "found, but cannot execute" - +# for every scenario, twenty times, with an empty stdout and nothing saying "wrong operating system". +# So those two languages could not be driven from a developer's laptop at all. +# +# WHY THE WHOLE SUITE MOVES, NOT JUST THE RUNNER. The client spawns the sidecar as its own child +# process and reaches it over loopback (KTD41), and the suite's SidecarShim announces a bare `port:` +# with no host - deliberately, so the client exercises its real spawn-and-reap path instead of gaining +# a connect-to-an-existing-port option that would bind ten languages. Client, shim and engine +# therefore have to share ONE loopback. Measured on Docker Desktop 27.4.0: a container cannot reach +# the host's 127.0.0.1 even with `--network host`, while `host.docker.internal` works - an address no +# client can be told to use without changing the protocol and R29's authority allowlist. Moving the +# whole run inside one container makes that shared loopback the container's own, and needs no product +# change whatsoever. +# +# WHAT RUNS: this suite, the engine it hosts in its own JVM, the shim it writes, and the runner it +# spawns - all in the image, all on one loopback. +# +# WHY IT IS NOT `--network none`, which would have been a tidy proof that nothing crosses the +# boundary: the mounted ~/.m2 is a macOS repository, and Maven needs PLATFORM-CLASSIFIED artifacts +# for the Linux container - `com.google.protobuf:protoc:exe:linux-aarch_64`, which a host cache has +# never had a reason to hold. A cold run therefore has to fetch, and offline fails before the suite +# is even reached. Set PC_CONFORMANCE_OFFLINE=1 once the cache is warm to get that proof back; the +# classifiers coexist, so warming it costs the host nothing. +# +# CI DOES NOT USE THIS AND NEVER WILL, because its runners are Linux and the extracted artifact is +# native there. That is the rot risk to design against: this path is exercised only by developers on +# macOS, so a scheduled run is what would keep it honest. Recorded in +# docs/inflight/parked-containerised-toolchains-and-runtime.md. +# +# ONLY the conformance suite runs, not every test in every module `-am` builds. `-am` is still +# required - the suite depends on those modules' jars and test-jars - but running parallel-consumer-core's +# whole suite inside a resource-constrained container adds minutes and imports its contention-sensitive +# tests into a result that is supposed to be about C++. The first run here failed on three of them, +# including `processInKeyOrder`, which the flake ledger already records as contention. +# +# Run: bin/run-conformance-in-container.sh [extra maven args...] + +set -uo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +readonly REPO_ROOT +readonly CONFORMANCE_MODULE=":parallel-consumer-proxy-conformance" + +die() { # + local code="$1" + shift + printf 'bin/run-conformance-in-container.sh: %s\n' "$*" >&2 + exit "$code" +} + +LANGUAGE="${1:-}" +case "$LANGUAGE" in + cpp | swift) shift ;; + "") die 2 "usage: bin/run-conformance-in-container.sh [maven args...]" ;; + *) die 2 "$LANGUAGE does not build in a container - only cpp and swift do, and every other + language's runner already executes on this host. Run the suite the ordinary way: + ./mvnw test -pl $CONFORMANCE_MODULE -am -Dpc.conformance.language=$LANGUAGE" ;; +esac + +command -v docker >/dev/null 2>&1 \ + || die 2 "docker is not installed, and this path exists precisely because $LANGUAGE cannot run + natively here. Nothing was run - this is NOT a pass." +docker info >/dev/null 2>&1 \ + || die 2 "the docker daemon is not reachable. Nothing was run - this is NOT a pass." + +MODULE_DIR="$REPO_ROOT/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-$LANGUAGE" +[ -f "$MODULE_DIR/Dockerfile" ] || die 3 "no Dockerfile in $MODULE_DIR" + +IMAGE="pc-conformance-$LANGUAGE" + +printf '==> %s: building the conformance image (%s)\n' "$LANGUAGE" "$IMAGE" +# The same build context and named proto context bin/build-client.sh uses, so this image and the +# artifacts CI extracts come from one Dockerfile and one `build` stage. +docker buildx build \ + --target conformance \ + --build-context "proto=$REPO_ROOT/parallel-consumer-proxy-protocol/src/main/proto" \ + --tag "$IMAGE" \ + --load \ + "$MODULE_DIR" \ + || die 1 "$LANGUAGE: conformance image build failed" + +# The Maven repository is mounted rather than baked: the image would otherwise carry a copy of every +# dependency and go stale the moment one moved. +M2_DIR="${HOME}/.m2" +mkdir -p "$M2_DIR" + +printf '==> %s: running the suite inside the container\n' "$LANGUAGE" +# Network and offline travel together: with PC_CONFORMANCE_OFFLINE=1 the run proves nothing crossed +# the boundary, and without it Maven may fetch the Linux-classified artifacts a macOS ~/.m2 lacks. +network_args=() +maven_offline=() +if [ "${PC_CONFORMANCE_OFFLINE:-0}" = "1" ]; then + network_args=(--network none) + maven_offline=(--offline) + printf '==> %s: offline - no network at all, so a pass proves the run is self-contained\n' "$LANGUAGE" +fi + +docker run --rm \ + "${network_args[@]+"${network_args[@]}"}" \ + --volume "$REPO_ROOT:/repo" \ + --volume "$M2_DIR:/root/.m2" \ + --workdir /repo \ + "$IMAGE" \ + ./mvnw --batch-mode "${maven_offline[@]+"${maven_offline[@]}"}" test \ + -pl "$CONFORMANCE_MODULE" -am \ + -Dpc.conformance.language="$LANGUAGE" \ + -Dtest=ConformanceSuiteTest \ + -Dsurefire.failIfNoSpecifiedTests=false \ + -Dcopyright.skip=true \ + "$@" +status=$? + +if [ "$status" -ne 0 ]; then + die "$status" "$LANGUAGE: the conformance suite failed inside the container. + That is a real result - the runner and the engine were on the same loopback, so this is the + language's behaviour rather than the host's incompatibility." +fi + +printf '==> %s: conformance passed inside the container\n' "$LANGUAGE" diff --git a/bin/test-check-toolchain-versions.sh b/bin/test-check-toolchain-versions.sh new file mode 100755 index 0000000000..e68e6339ab --- /dev/null +++ b/bin/test-check-toolchain-versions.sh @@ -0,0 +1,136 @@ +#!/usr/bin/env bash +# +# Copyright (C) 2026 Antony Stubbs and contributors +# + +# Self-test for bin/check-toolchain-versions.sh. +# +# The gate's whole job is to notice a difference between two files, so the case that matters is not +# "does it pass on a healthy repo" - it is "can it be made to go red, for each way the two can +# disagree". bin/AGENTS.md: a regression test that has never failed proves nothing. +# +# Every case drives the gate against crafted fixtures through PC_MISE_TOML / PC_CLIENTS_WORKFLOW, so +# none of it depends on the state of the real repository - a self-test for a drift checker must not +# itself drift when someone legitimately bumps a version. +# +# Run: bin/test-check-toolchain-versions.sh + +set -uo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +GATE="$REPO_ROOT/bin/check-toolchain-versions.sh" +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +failures=0 + +assert() { # + if [ "$2" = "$3" ]; then + echo "ok: $1" + else + echo "FAIL: $1 (expected '$2', got '$3')" + failures=$((failures + 1)) + fi +} + +# A healthy pair, which every case below mutates in exactly one way. +write_mise() { # [extra lines...] + { + echo '[tools]' + echo 'go = "1.25.13"' + echo 'ruby = "3.4.4"' + echo 'node = "22.17.0"' + shift + for line in "$@"; do echo "$line"; done + } >"$1" +} + +write_workflow() { # + cat >"$1" < -> PASS | FAIL + if PC_MISE_TOML="$1" PC_CLIENTS_WORKFLOW="$2" "$GATE" >/dev/null 2>&1; then + echo PASS + else + echo FAIL + fi +} + +# The baseline. Without this the red cases below could all be red for some unrelated reason. +write_mise "$TMP/ok.toml" +write_workflow "$TMP/ok.yml" "1.25.13" "3.4.4" "22.17.0" "" +assert "matching declarations pass" PASS "$(verdict "$TMP/ok.toml" "$TMP/ok.yml")" + +# The failure the gate was written for, at the size it actually occurred (a whole major version). +write_workflow "$TMP/major.yml" "1.25.13" "3.4.4" "25.9.0" "" +assert "a major-version disagreement is caught" FAIL "$(verdict "$TMP/ok.toml" "$TMP/major.yml")" + +# ...and at the smallest size, because these are two declarations of one number: a patch difference +# between them is a mistake, not tolerable drift. +write_workflow "$TMP/patch.yml" "1.25.14" "3.4.4" "22.17.0" "" +assert "a single patch disagreement is caught" FAIL "$(verdict "$TMP/ok.toml" "$TMP/patch.yml")" + +# The typescript -> node mapping is the one place the two files use different names for one thing. +# If it broke, the baseline case above would fail; this asserts the mapping is real rather than the +# gate ignoring anything it does not recognise. +write_mise "$TMP/nonode.toml" +grep -v '^node' "$TMP/nonode.toml" >"$TMP/nonode2.toml" && mv "$TMP/nonode2.toml" "$TMP/nonode.toml" +assert "a language CI pins but mise omits is caught" FAIL "$(verdict "$TMP/nonode.toml" "$TMP/ok.yml")" + +# A tool nobody builds with is dead weight that will rot; it is also how a rename hides. +write_mise "$TMP/extra.toml" 'elixir = "1.18.0"' +assert "a mise tool no language uses is caught" FAIL "$(verdict "$TMP/extra.toml" "$TMP/ok.yml")" + +# Swift builds in a container. If it ever gains a host pin, that is either a real change or a stale +# copy-paste, and both need a human. +write_workflow "$TMP/swiftpin.yml" "1.25.13" "3.4.4" "22.17.0" "6.1.0" +assert "a container language gaining a CI pin is caught" FAIL "$(verdict "$TMP/ok.toml" "$TMP/swiftpin.yml")" + +write_mise "$TMP/swiftmise.toml" 'swift = "6.1.0"' +assert "a container language appearing in mise.toml is caught" FAIL "$(verdict "$TMP/swiftmise.toml" "$TMP/ok.yml")" + +# A pinned language silently losing its version reads as "no toolchain needed", which is exactly the +# state the four container/JVM languages are in - so it must not be inferred. +write_workflow "$TMP/lost.yml" "" "3.4.4" "22.17.0" "" +assert "a pinned language losing its CI version is caught" FAIL "$(verdict "$TMP/ok.toml" "$TMP/lost.yml")" + +# THE MOST IMPORTANT CASE. If the workflow format changes under it, a regex-based gate parses zero +# languages and every comparison trivially succeeds - a green run that checked nothing, which is +# worse than no gate because it is indistinguishable from a real pass. +echo 'jobs: {}' >"$TMP/empty.yml" +assert "a workflow it cannot parse FAILS rather than passing vacuously" FAIL "$(verdict "$TMP/ok.toml" "$TMP/empty.yml")" + +# Same argument for the inputs themselves. +assert "a missing mise.toml fails" FAIL "$(verdict "$TMP/does-not-exist.toml" "$TMP/ok.yml")" +assert "a missing workflow fails" FAIL "$(verdict "$TMP/ok.toml" "$TMP/does-not-exist.yml")" + +# The real repository must be consistent, or the gate is about to fail CI for everyone. +if "$GATE" >/dev/null 2>&1; then real=PASS; else real=FAIL; fi +assert "this repository's own declarations agree" PASS "$real" + +echo +if [ "$failures" -eq 0 ]; then + echo "All check-toolchain-versions self-tests passed" + exit 0 +fi +echo "$failures self-test(s) FAILED" +exit 1 diff --git a/docs/agent-harness.md b/docs/agent-harness.md index 413001c7e6..5e65484e4f 100644 --- a/docs/agent-harness.md +++ b/docs/agent-harness.md @@ -460,6 +460,20 @@ shipped its first version without it: a review then found six defects in one 25- letting the exact mistake it was named after through and two hard-blocking legitimate merges. Every one is a case in that file, and the suite goes red against the old parser. +The disk hook's cases are worth reading before adding a hook of your own, because it has the failure +mode every warn-only hook shares: **its correct behaviour on a healthy machine is to print nothing, +which is byte-identical to it being broken, unregistered, or not running at all.** So its silent case +is pinned to thresholds of zero rather than to a healthy disk, and pairs with forced cases proving the +same call path can be made to speak. An earlier version left the thresholds at their defaults, which +made the suite a function of how much free space the machine happened to have - three cases flipped +to failing mid-session when the host dropped below the default warn line. A self-test for a disk +warner must not itself depend on the disk. + +**Still unverified for this hook: whether the harness reaches it at all.** The `claude -p` reachability +check that settled that question for the other three could not be run when it was added. That is the +separate question flagged under *`if` matches a PREFIX* - a self-test can only prove what a script +does - and it stays open until someone runs it. + ## Adding to it 1. **Decide what the rule needs.** To be *known* -> a `CLAUDE.md`. To be *enforced* -> a hook or a diff --git a/docs/inflight/branch-polyglot-demos.md b/docs/inflight/branch-polyglot-demos.md index 58bb6d4cb0..dcccdb9fa1 100644 --- a/docs/inflight/branch-polyglot-demos.md +++ b/docs/inflight/branch-polyglot-demos.md @@ -1,8 +1,9 @@ # The ten per-language demos: ten branches, no PR, and what the fan-out found -Session handoff, 2026-08-21. Ten agents built the per-language demos in parallel, one worktree and -one branch each. **Nothing is merged and only `feats/java-sidecar-demo` is pushed.** This file exists -because the work is real, on disk, and invisible to `gh`. +Ten agents built the per-language demos in parallel, one worktree and one branch each; an eleventh +wave polished all of them. **All of it is now merged into `feats/polyglot-demos`**, which has commits +that are not pushed. This file exists because the work is real, on disk, and largely invisible to +`gh`. ## Where the code is @@ -70,8 +71,10 @@ recorded objections in their own notes, by instruction. These are unresolved: `bootstrap.servers` at INFO several times a run. It should bind the whole run. 6. **`bin/ci-demo-test.sh` is Java-only** - confirmed independently by every agent. Ten demos can ship while the contract claims both entry points are tested per language. -7. The big-replay title's `total * delayMs / 1000` is integer arithmetic and prints "would take 0s+" - at small volumes. A seed wart, faithfully mirrored everywhere. +7. ~~The big-replay title prints "would take 0s+" at small volumes.~~ **Fixed in all eleven**: the + unit is now chosen so the figure is never zero (`80s` at the demo's own defaults, `80ms` at CI's). + Two languages had already fixed it independently and differently - Ruby omitted the clause, Python + changed the unit - and Python's answer won because it keeps the information at every volume. 8. **Every non-JVM demo container is a two-toolchain image** and the contract does not say so - the sidecar is a child of the *running* demo, so it cannot be a discarded build stage. @@ -127,3 +130,105 @@ U35's second half - the *reading* demo (three modes, TTY prompt, serde marker, s the reconciliation it needs: **KTD40 in the plan still describes every demo as having three modes**, while the seed has none and says why. Plan and seed disagree on paper until someone amends KTD40 or records that it governs the reading demo only. That reconciliation, not the code, is the open item. + + +## What is next, in order (2026-08-22) + +1. **`close()` reports success and leaves non-daemon threads running** - + [`bug-engine-close-leaves-non-daemon-threads.md`](bug-engine-close-leaves-non-daemon-threads.md). + Surfaced as the Java demo hanging forever on the container path, but the defect is in + `parallel-consumer-core` and a user's application would hang the same way. It blocks CI rather + than failing it: nothing in either harness wraps a run in a timeout, so a hang burns the job. + Highest priority of anything here. +2. **Run the conformance harness across more than two languages.** Its patterns were repaired and + verified against real output from two languages, but the drift check still has not proved + anything at the width it exists for. This is the first thing the merged branch makes possible. +3. **Push, then decide the PR shape.** Current recommendation is to fold `feats/polyglot-demos` into + astubbs#328 rather than open a second PR, because that PR still ships the **old, disproven** + divergence rule and the correction's evidence is the fan-out itself - reviewing one without the + other reviews a conclusion with no argument. Reverse that if astubbs#293 merges first. +4. **The idle-machine measurement pass**, blocked twice: it needs a quiet box, and it needs the + unexplained baseline shift in [`next-demo-seed-followups.md`](next-demo-seed-followups.md) item 1 + resolved, because every ratio divides by a figure that moved 15% between sessions for reasons a + control arm refuted. +5. **The sidecar logging defect**, in the order its own record specifies - see + [`bug-sidecar-runtime-logging-and-address-leak.md`](bug-sidecar-runtime-logging-and-address-leak.md). + Each step alone regresses something, which is why it is one job. +6. **Owner decisions, not work**: the `enable.auto.commit` divergence between the two transports; the + KTD40 reconciliation (the plan still says every demo has three modes, the seed has none and says + why); and whether the merged `demos/*` and `polish/*` branches and their worktrees are cleaned up. + +### The broker is quiet now - measured, and the assertion is still missing + +`KAFKA_LOG4J_ROOT_LOGLEVEL` does not work; the cp-kafka image writes its own per-logger map +(`kafka=INFO`, `kafka.controller=TRACE`, `state.change.logger=TRACE`) which outranks the root level. +Four agents measured it independently - 1465, 927, 883 and 584 broker lines against 35-42 of the +demo's own. `KAFKA_LOG4J_LOGGERS` is the variable that merges into that map, and a container run of +the Java demo now gives: + +``` +broker lines: 12 (1 INFO, 1 WARN; the rest are the image's own "===>" banner, not log4j) +demo lines: 216 +``` + +**Still open: nothing asserts it.** The native path never sees broker output at all - Testcontainers +does not attach the broker's stdout - so this was invisible to every check that exists, and would +regress silently. A broker-line-count assertion in `bin/ci-demo-test.sh` is what closes it. + +## Reconciliation items found during the polish wave + +- **The column order is settled** - all eleven print + `arm | records | keys | elapsed | msg/s | vs AK core`, the contract states it, and Java now carries + a test for it. **The lesson is what is worth keeping**: the contract was silent on position and the + harness regex was not, so two independent implementers let a broken test define the spec. That is + what an under-specified contract plus a visible check will always produce. + +- **Ruby and Rust both publish host port 29092**, so their demos cannot run back to back without + `Bind for 0.0.0.0:29092 failed`. Python already parameterises its port + (`${PC_DEMO_BROKER_PORT:-19095}`), which is the shape that fixes it. Patching one language alone + only moves the collision. + + **This is also an argument for the shared-broker test harness** already agreed: one broker and one + reused topic for the whole suite removes per-language ports entirely, rather than allocating eleven + non-colliding ones. + +- **Eleven concurrent demos exhaust the Docker VM's disk**, and it does not present as a disk + problem. Three agents hit it within minutes of each other; the broker exits 1 having logged + `Formatting metadata directory ... No space left on device`, and the demo reports only "starting + the broker failed". Worth a friendlier message, and worth knowing before running the full eleven + again on one machine. + + +## A third wave is owed: every serious client, as its own arm + +Owner correction (2026-08-21): **excluding sarama from the Go demo was wrong, and so is the +equivalent exclusion in every other language.** The contract said "consider running both"; it now +says run them all. Each of these is somebody's production choice, and answering "is this fast in my +language" for one client out of three leaves two thirds of readers guessing. + +Known to be missing, from the agents' own READMEs: + +| language | runs | also serious, not yet an arm | +|---|---|---| +| Go | franz-go | **sarama**; confluent-kafka-go (blocked by `CGO_ENABLED=0` in its image - a real constraint, and the kind worth writing down) | +| Python | confluent-kafka | kafka-python, aiokafka | +| TypeScript | kafkajs | confluentinc-kafka-javascript (librdkafka binding) | +| Ruby | rdkafka | ruby-kafka is archived - a legitimate exclusion, and it belongs in the README with its reason | +| others | one each | unaudited; each demo's README must now name what it found and what it skipped, with reasons | + +Two things had to change before this was even possible, and both are done: the harness compares the +**required** arms and permits extras, and Java is no longer exempt from that comparison. Go had +already costed a sarama arm, found it cheap, and declined **because the harness would have failed it +for obeying the contract** - so the rule and the check were in direct conflict. + +## And the internals are NOT the seed's to dictate + +Owner correction, same conversation: **the Java demo is a seed, not a template.** Uniformity is owed +on everything the contract specifies - flags, banner, tables, arms, container rules, exit codes - and +on nothing below that line. Each demo should be written the way someone fluent in that language would +write it: its idioms, its concurrency primitives, its layout, its test framework. + +This is now stated in the contract under "Idiomatic inside, identical outside", because the risk was +live rather than theoretical: eleven agents were handed a Java reference and told to mirror a +contract, and the natural failure mode is transliteration. A Java program rewritten in Ruby teaches a +Ruby reader nothing about using this client in Ruby. diff --git a/docs/inflight/branch-uber-demo-eleven-lane-grid.md b/docs/inflight/branch-uber-demo-eleven-lane-grid.md new file mode 100644 index 0000000000..615e34bd42 --- /dev/null +++ b/docs/inflight/branch-uber-demo-eleven-lane-grid.md @@ -0,0 +1,121 @@ +# The uber demo: eleven language clients, one workload, one view + +Carried by `feats/polyglot-demos` (astubbs/parallel-consumer#331), which is the work this designs +the successor to. It had a branch and a draft PR of its own - astubbs/parallel-consumer#332, closed +as superseded - and that was over-engineering: a design nobody needs to review separately from the +work it follows does not need a rung in the stack, and this one had acquired two PRs stacked above +it. **Design only - nothing implemented, and three decisions are open below. Do not start until +they are settled; each one changes what gets built.** + +## Prior art, which reshaped this before any code + +[`next-polyglot-demo-app.md`](next-polyglot-demo-app.md) points at ideas 21-27 in +[`docs/ideation/2026-08-14-language-proxy-interaction-model-ideation.html`](../ideation/2026-08-14-language-proxy-interaction-model-ideation.html) +- seven ranked directions for "a demo that runs all eleven language bindings at once and proves in +one view that they read the same records". It says the eleven-language grid belongs to whichever +idea wins, and that the per-language demos on astubbs/parallel-consumer#331 are a deliberate narrow +cut of idea 23, not the grid. It also says to read the ideation before designing any demo, +orchestration, aggregation or perf-display work, because it records what was rejected and why. + +**Idea 21, "the skateboard stack"** (ranked first, 85% confidence, medium complexity) is the shape +the owner described: run the consumption work and report one cohesive view, *not* eleven existing +demos in sequence. Its four pillars: each language's demo is its conformance runner in costume; each +worker's Success report carries a receipt `{record_id, language, offset, attempt, payload_checksum}` +onto a `demo-observations` topic; one TUI aggregator consumes that topic, renders eleven lanes and +machine-asserts sameness, ending "11/11 languages converged at offset N"; CI records the run as an +asciinema cast. + +**Idea 22, "one consumer group across eleven runtimes"**, is a different demo that this design keeps +confusing with 21 - see decision 2. + +## Verified, not assumed + +**The receipts route exists.** `proxy.proto` carries `repeated ProduceRecord produce` in the worker's +report, commented as "the sanctioned route for worker output", at-least-once. So receipts ride the +protocol's own output path: Kafka becomes the aggregation bus, sameness is *computed* rather than +eyeballed, and nothing parses eleven demos' stdout - which is the orchestration shape the owner +ruled out. + +## What already exists, that did not when idea 21 was written + +| pillar | state | +|---|---| +| conformance runners for all eleven, over production gRPC | exists | +| **demo binaries that already spawn a real sidecar against a real broker** | exists, from astubbs/parallel-consumer#331 - see decision 1 | +| cpp and swift runnable off-Linux | exists - `bin/run-conformance-in-container.sh` | +| a shared broker | exists | +| fixed, parseable table shape, with `records`/`keys` deterministic across languages | exists | +| arms that name the product (`pc--grpc`), so a row identifies itself | exists | +| receipts, aggregator, cast | the new work | + +## Decision 1: which binary is the base + +An earlier version of this note said the expensive gap was "give the eleven conformance runners a +real-sidecar flag". **That is stale.** Idea 21 was written before astubbs/parallel-consumer#331, and +every language now has a demo binary that already spawns a real sidecar against a real broker. The +costume exists; it is just called a demo rather than a runner. + +- **Demo binaries as the base.** The hard per-language part - real sidecar, real broker, container + that works - is done. The work is adding receipts. The baggage: they seed their own backlog and + print their own tables, neither of which a grid wants. +- **Conformance runners as the base.** Frozen, observable, already machine-checked. The work is a + real-sidecar flag *and* receipts, in eleven languages. + +Leaning: **demos**, because the expensive half is already paid for. + +## Decision 2: the consumer-group topology, which decides what the demo CLAIMS + +The two are not variants of one demo. Only the first supports the convergence line. + +- **One topic, eleven separate consumer groups (idea 21).** Every language reads *every* record, so + per-record checksums are comparable and "11/11 converged at offset N" is a real assertion. The + claim is **"eleven runtimes read the same records identically"**. +- **One topic, one shared consumer group (idea 22).** The eleven *split* the records and rebalance + across languages. No per-record comparison exists. The claim is **"eleven runtimes cooperate in one + group"** - which is arguably the more surprising thing to show, and is a different demo. + +**Consequence either way:** the demos currently seed their own backlog, so eleven against one topic +means eleven backlogs unless exactly one seeds. A "who seeds" role is needed. Small, but it is a +contract question rather than wiring. + +## Decision 3: does the grid show a rate at all + +**Eleven runtimes on one host, at once, cannot produce quotable throughput.** This project has +already discarded an entire fan-out's figures for exactly that - the box ran at load 20-113 and every +agent was instructed to refuse to report rates. + +- **Deterministic columns only** - records, keys, offsets, checksum agreement, convergence. Immune to + contention, and it is idea 21's actual claim. +- **Rates too**, which forces sequential runs on a quiet host and stops it being one live view. + +[`next-perf-comparison-matrix.md`](next-perf-comparison-matrix.md) owns measurement semantics and the +blessed-numbers pipeline by prior agreement recorded in both docs; this track owns app, UI and +narrative. So the speed table is that track's artifact. Leaning: **no rates in the live grid.** + +## Smallest first step, once the decisions land + +Not the TUI. Take **two** languages - one that runs natively here and one that needs the container, +so the awkward case is in from the start - have them emit receipts to one topic, and get an +aggregator to say "2/2 converged at offset N". If two lanes converge, the remaining nine are +transcription. Building the UI first produces a beautiful view of one language. + +## What must not be done + +- **Do not build the grid by parsing eleven demos' stdout.** Ruled out by the owner, and it inherits + every per-language rendering difference as a parsing problem. Receipts make sameness computable. +- **Do not print a rate from a parallel run**, however good the wide table looks. +- **Do not take audience input as workload.** The ideation records a keynote mode rejected for + violating the no-visitor-input security posture; that rejection stands. + +## Cross-references + +- [`next-polyglot-demo-app.md`](next-polyglot-demo-app.md) - the ideation pointer, the + perf-track split, and the instruction to choose an idea and take it through brainstorm to a plan +- [`parked-demo-gallery.md`](parked-demo-gallery.md) and + [`branch-polyglot-demo-ideation.md`](branch-polyglot-demo-ideation.md) - adjacent demo thinking +- [`branch-polyglot-demos.md`](branch-polyglot-demos.md) - the eleven demos this stacks on, and what + the fan-out found +- [`next-demo-testing-infrastructure.md`](next-demo-testing-infrastructure.md) - the ranked ambition + for the demo harnesses +- `parallel-consumer-proxy/demo/README.md` - the contract the eleven demos keep +- `proxy.proto`, `ProduceRecord` - the sanctioned worker-output route the receipts would ride diff --git a/docs/inflight/bug-engine-close-leaves-non-daemon-threads.md b/docs/inflight/bug-engine-close-leaves-non-daemon-threads.md new file mode 100644 index 0000000000..eb4f204c6c --- /dev/null +++ b/docs/inflight/bug-engine-close-leaves-non-daemon-threads.md @@ -0,0 +1,100 @@ +# `close()` reports "Close complete" and leaves non-daemon threads running + +Found 2026-08-22 on `feats/polyglot-demos`. The symptom is the Java demo hanging forever on the +container path; the defect is in `parallel-consumer-core`, and a user's application would hang the +same way. **The mechanism below is established; why it is intermittent is not.** + +## The defect + +In a hung run, `AbstractParallelEoSStreamProcessor` logs, for **all four** of the demo's in-process +engines: + +``` +Signaling to close... x4 +Close complete. x4 +Control loop ending clean (state:CLOSED)... x4 +``` + +and yet the JVM will not exit, because four **non-daemon** threads are still alive with +`DestroyJavaVM` blocked behind them: + + + +``` +"pc-broker-poll" #98 prio=5 ... elapsed=173.04s waiting on condition +"pc-control" #99 prio=5 ... elapsed=173.04s waiting on condition +"pc-broker-poll" #118 prio=5 ... elapsed=171.99s waiting on condition +"pc-control" #119 prio=5 ... elapsed=171.99s waiting on condition +"DestroyJavaVM" #188 prio=5 ... elapsed=166.96s waiting on condition +``` + + + +**So `close()` returning is not evidence that the engine's threads are gone**, and neither is +`Control loop ending clean`. Two of the four engines shut down completely; two logged the same +success and left a `pc-broker-poll` / `pc-control` pair behind. Their `elapsed` values place them +5-6s before `DestroyJavaVM` appeared, which is the **big** replay's `pc-core` and `java-direct` arms. + +That a shutdown path can report success while leaving non-daemon threads is the finding. Anything +built on "close completed, therefore we are done" is built on nothing. + +## Do not start from the demo + +The demo's arms are correct: `pcCore` closes in a `finally`, `javaDirect` uses try-with-resources, +and the log confirms `close()` was called on every engine and returned. An earlier version of this +note said "two engine instances outlived their arms" - that was wrong, and it is the hypothesis a +reader will reach for first, so it is written down here as refuted. + +One counting trap, because it cost time: the demo pumps each spawned **sidecar's** stdout into its +own log, so a naive `grep -c "Confluent Parallel Consumer initialise"` returns **10**, not 4. Six of +those are engines in *other processes*. Filter on `[main]`, or the arithmetic says six engines were +never closed and sends you after a leak that does not exist. + +## What was measured + +Same image, same compose broker, same dials (`--records 20 --concurrency 4 --partitions 2 +--replay-factor 2`); a temporary `pc.demo.experiment.skipUds` flag removed only the demo's own arm +and left epoll **available**, so exactly one term moves. + +| run | UDS arm | epoll | outcome | +|---|---|---|---| +| `compose up` (original) | present | available | **hung** - killed at >4min | +| `-Dio.netty.transport.noNative=true` | absent | disabled | exited, 19s | +| `skipUds=true` | absent | **available** | exited, 23s | +| `skipUds=false` (control) | present | available | **hung** - still running at 240s | +| native, macOS | absent (no epoll on macOS) | n/a | exited, 50s | + +The third and fourth rows are the pair that matters: one term, outcome flips. The second row alone +could not convict the arm, because `noNative` also moves every other gRPC arm off epoll. + +**Do not read this as "the UDS arm is the bug".** It is the term that makes the defect appear, and +the defect is that a completed `close()` can leave threads running. The likeliest reading is that +the extra arm changes timing or load enough to lose a race that is always present - which is exactly +the kind of cause that a 2-versus-2 run count cannot distinguish from a hard dependency. Establish +the rate before believing either. + +## Open, in order + +1. **Find what those two threads are waiting on.** The dump says `waiting on condition` with no + stack captured in what was collected; a full `jstack` with stacks is one command and constrains + this more than any further black-box runs. +2. **Get the reproduction rate**, at fixed dials, with and without the arm. Everything above is + n=2 per configuration, and an intermittent race and a deterministic dependency look identical at + that sample size. +3. **Name the arm in the engine's thread name.** Two of four engines leak and the dump cannot say + which; this settles it in one run instead of by inference from timestamps. +4. **Then decide whether the demo works around it**, and only then. A workaround written before the + library defect is understood will hide it. + +## Blast radius, and what must not be done + +`demo/run.sh --docker` ends in the same `compose up --abort-on-container-exit` that hung, and +**neither `bin/ci-demo-test.sh` nor `bin/ci-demo-conformance.sh` wraps a run in a timeout** - so a +hang is not a red check, it is a job that burns to the CI limit. Note also that the clean native run +was on macOS, where there is no UDS arm at all; **CI is Linux, where the native path carries it**, so +that result is not evidence CI is safe. + +**Do not "fix" this with a harness timeout or a `System.exit`.** Both turn a process that does not +terminate into one that appears to, and the second would hide the same defect in the library the +demo exists to show off. A timeout belongs in the harness *as well*, so a future hang fails loudly +instead of burning a job - but it is not the fix. diff --git a/docs/inflight/bug-sidecar-runtime-logging-and-address-leak.md b/docs/inflight/bug-sidecar-runtime-logging-and-address-leak.md index aacaea430e..a5fcd0ef49 100644 --- a/docs/inflight/bug-sidecar-runtime-logging-and-address-leak.md +++ b/docs/inflight/bug-sidecar-runtime-logging-and-address-leak.md @@ -84,3 +84,39 @@ So the two halves fit: the Rust client **discards stdout** because that is the l drains for the port line, and the C++ client **inherits stderr** precisely so the sidecar can be diagnosed. Moving the sidecar's own logging to stderr is therefore not a proposal needing a design - at least one client already expects it there, and one has demonstrated the chain end to end. + + +## Confirmed live: the fix's own trap, hit by the C++ demo + +The C++ agent gave its demo image an SLF4J binding - step 1 of the fix ordering above - and got both +of the problems steps 2 and 4 exist to prevent, in one run: + +- **the flood**: 568 lines of demo output, roughly 500 of them routine sidecar chatter burying the + two tables; +- **the leak**: a full `ConsumerConfig values` dump per arm, printing `bootstrap.servers = [...]` + into lines the DEMO owns - not the broker's. `bin/ci-demo-conformance.sh` greps for exactly that + and would have failed the credential rule. + +It fixed its own image with `-Dorg.slf4j.simpleLogger.defaultLogLevel=warn`, overridable through +`PC_SIDECAR_LOG_LEVEL`. + +**This is not a C++ finding.** Every language whose demo image gains that binding inherits both +problems, and adding the binding is precisely what the first step of the fix asks for. It is the +clearest evidence yet that these four steps are one job: shipping step 1 alone would hand every +language a noisier demo and a credential leak at the same time. + + +## A sixth face, on the demo side: the banner cannot be first + +The contract requires the product's name to be the first thing printed. **In Java it is not**, and +the Java agent established why rather than working around it: `parallel-consumer-core`'s test jar is +on the demo's classpath - the sidecar spawn needs it - and carries a `logback-test.xml` with +`scan="true"`. Logback cannot watch a file inside a jar, warns, and the warning triggers a full +status dump of roughly thirty lines **before** anything the demo prints. + +**A `logback.xml` in the demo module cannot fix it**, because a `logback-test.xml` anywhere on the +classpath outranks it. Both paths were checked rather than assumed. + +So this belongs with the rest: the fix is wherever that test jar's logging configuration lives, which +is the same place steps 1 and 2 above have to land. It is one more reason the demos cannot be made +right one at a time from the demo side. diff --git a/docs/inflight/ci-toolchain-versions-declared-twice.md b/docs/inflight/ci-toolchain-versions-declared-twice.md new file mode 100644 index 0000000000..c00cc2567b --- /dev/null +++ b/docs/inflight/ci-toolchain-versions-declared-twice.md @@ -0,0 +1,67 @@ +# Toolchain versions are declared twice: mise locally, the CI matrix on the runner + +Found 2026-08-22 while getting the full suite to run locally for the first time. **Open - no fix +proposed yet, because collapsing the two needs an owner's call.** + +## The two mechanisms + +**Locally**, `bin/build-client.sh` expects **mise** to provide every foreign toolchain except the two +that build in containers. That decision, including the permission for agents to run `mise use -g` +themselves, is in +[`parked-containerised-toolchains-and-runtime.md`](parked-containerised-toolchains-and-runtime.md). +mise reads `~/.config/mise/config.toml` - a machine-global file, outside the repo. + +**On the runner**, `.github/workflows/clients.yml` uses per-language setup actions +(`actions/setup-go`, `ruby/setup-ruby` pinned by SHA, `actions/setup-dotnet`, `actions/setup-node`, +`actions/setup-python`) and reads the version from its own matrix. **`mise` appears nowhere in +`.github/`** - a grep that seems to find it is matching the word "misused" in a comment. + +Both are reasonable in isolation. Together they are two sources of truth for the same numbers, and +nothing compares them. + +## The drift is real today, not hypothetical + +CI matrix versus one developer machine an hour after `mise use -g`: + +| | CI matrix (`clients.yml`) | this machine | +|---|---|---| +| go | `1.25.13` | 1.25.14 | +| ruby | `3.4.4` | 3.4.10 | +| dotnet | `8.0.404` | **9.0.101** | +| node | `22.17.0` | **25.9.0** | +| rust | `1.88.0` | 1.94.1 | + +The go and ruby gaps are patch-level and unlikely to matter. **The dotnet and node gaps are whole +major versions**, which is the classic shape of "passed locally, failed in CI" - or worse, the +reverse, where CI is the one running the older runtime and a developer never sees the failure their +change causes. + +Note the empty rows are not a gap: `swift`, `kotlin`, `scala` and `cpp` declare `toolchain: ''` +because kotlin and scala ride the Maven reactor and swift and cpp build in containers. The split in +`bin/build-client.sh` and the CI matrix already agree about *which* languages need a toolchain at +all. It is only the versions that are stated twice. + +## The obvious collapse, and why it is not just done + +A repo-level `mise.toml` as the single declaration, installed in CI by `jdx/mise-action`, would make +one file authoritative for both. Two things to weigh before doing it: + +- **The setup actions are not merely version installers.** They carry per-language caching that the + workflow's own header calls out (`cache: maven`, setup-go's `cache: true`, the ruby row's + `cache-path`/`lockfile`). Replacing them with mise means either losing that caching or rebuilding + it by hand, and a slower client matrix is a real cost paid on every pull request. +- **`ruby/setup-ruby` is pinned by commit SHA**, which is a supply-chain posture the repo chose + deliberately. `mise` installing from a registry at build time is a different trust model, not + obviously a worse one, but it is a change of posture rather than a refactor. + +A cheaper middle option, if the full collapse is not wanted: keep both mechanisms and add a check +that the versions agree - the failure this file describes is silent drift, and a five-line gate +comparing `mise.toml` against the matrix would end it without touching how either installs anything. + +## What made this visible + +The suite could not run locally at all until mise was installed, because Maven fell through to +whatever was on `PATH`: a Go pinned to `GOTOOLCHAIN=local` and macOS's system Ruby 2.6. Nothing in +`AGENTS.md` said mise was required - that is fixed - but the deeper point is that **a developer can +satisfy every documented requirement and still be running different toolchains than CI**, with no +signal either way. diff --git a/docs/inflight/clients/cpp.md b/docs/inflight/clients/cpp.md index af31e68505..369b035ffe 100644 --- a/docs/inflight/clients/cpp.md +++ b/docs/inflight/clients/cpp.md @@ -98,6 +98,143 @@ workflow's `cpp` row now gates for real. `Exit value: 2` in the exec failure message, while Maven's own exit stays 1 - so read the message, not just the code, when a CI row goes red. +## The demo (plan unit U35) + +`demo/` beside the library: `run.sh`, a `Dockerfile`, a `docker-compose.yml`, `src/` and a +`tests/`. Two arms, per the contract - `AK core` on librdkafka, and `cpp-grpc` through **this +module's client library**, which spawns the sidecar itself. No hand-written gRPC anywhere: the Java +seed did that first and had to be rewritten, because it proved the engine worked and said nothing +about the client library. + +### What the C++ demo does differently, and why + +- **Container-only, with no native mode**, which is the one real divergence. `run.sh --native` + answers with the reason rather than failing as an unknown flag. It follows from the same fact that + makes `bin/build-client.sh` build this language in an image: gRPC, protobuf and librdkafka arrive + as system dev packages, so there is no host toolchain to run natively against. +- **The broker is always supplied from outside.** Java starts one with Testcontainers when + `--bootstrap` is absent; C++ has no Testcontainers, and the demo container is never granted the + host Docker socket, so a missing address is an error with an explanation rather than a silently + different run. +- **The sidecar is a launcher script the image installs**, at `/app/sidecar/sidecar`, named by + `PC_DEMO_SIDECAR`. A foreign client library spawns a binary by absolute path and the sidecar is a + JVM program, so something has to bridge that; `exec` in the launcher is load-bearing, because a + forking wrapper would hold the lifecycle pipe the proxy watches for parent death. + **`PC_DEMO_SIDECAR` is deliberately not an eighth flag** - the contract fixes the list at seven, + and where the binary lives is a property of the image. +- **The demo image is a RUNTIME image built from its own toolchain stage**, the opposite of + `../Dockerfile`, which exports statically linked artifacts and runs nothing. Nothing is extracted + here, so there is no portability claim to prove and no reason to pay for the static link's absl + archive group; the binary is dynamically linked and runs where it was built. +- **`demo/CMakeLists.txt` is a project of its own**, not a target in the module's. The demo needs + librdkafka and the library does not, and adding it to the module's project would put a Kafka + client into the image whose whole job is a portability claim about gRPC. +- **The Maven local repository IS a BuildKit cache mount here**, which the Java demo's Dockerfile + says it could not have: that image computes a classpath file pointing into `/root/.m2`, so a cache + mount would name jars the running container does not have. This one copies the jars out + (`dependency:copy-dependencies`) and uses a wildcard classpath, which has no such coupling. + `sharing=locked`, because a Maven local repository is not safe for concurrent writers and ten + sibling demos may build at once. + +### What was actually run, and what was not + +Both entry paths were exercised by hand on this branch: `demo/run.sh` at the volume +`bin/ci-demo-test.sh` uses for the Java demo (`--records 20 --delay-ms 1 --concurrency 4 +--partitions 2`), once with `--replay-factor 1` and once with `--replay-factor 2` so that the big +replay's second seed, second table and footnote ran too. Both arms processed every record and the +run exited 0. The no-argument path was exercised three ways: `run.sh` under **bash 3.2**, which is +what macOS ships and where the reference demo's empty-array expansion once aborted; the binary with +no arguments and no broker, which explains itself and exits 2; and `--help`, which the binary +answers itself because `docker compose run demo --help` reaches it directly. + +**No measurement was taken and none should be read into those runs.** At twenty records the figures +are start-up and rebalance, which is exactly why the CI volume asserts arms rather than numbers. A +default-scale run is deferred to an unloaded machine - this one was running ten demo waves at once. + +### The reader-experience pass, and the one thing it could not honour + +The demo contract's "The output a reader actually sees" section was rewritten after someone watched +a demo and found it unimpressive. Three of its four rules are applied here; the fourth was already +true. + +- **The banner is the first thing printed**, before the usage text and before a refused flag, not + only before the fingerprint. C++'s copy names the language `C++` rather than the module + directory's `cpp`. +- **Both arms name their library**: `AK core (librdkafka)` and `cpp-grpc (this client)`. The demo + README now also answers the contract's "where a language has more than one serious client, say so" + clause, and the answer for C++ is that it does not have one - `cppkafka` and `modern-cpp-kafka` + are wrappers over librdkafka, so a second arm would price a header-only binding. +- **`records` and `keys` are new columns**, counted per arm from that arm's own delivery path + (`rd_kafka_message_t` for AK core, `InboundRecord` for the sidecar arm) rather than restated from + the seeding loop. The sidecar arm's key set is guarded by the same mutex as its counter, because + every executor thread writes it; an unsynchronised `std::set` there is a race that would present + as a slightly wrong key count rather than as a crash. +- **Broker log levels were already `WARN`** in `demo/docker-compose.yml` and were not touched. +- **The SIDECAR was not quiet, and that turned out to be two contract violations rather than + noise.** This image gives the sidecar an SLF4J binding it would otherwise lack (see + `demo/Dockerfile` for why). At slf4j-simple's default level it emitted the overwhelming majority + of the demo's own output on a twenty-record run - including a full `ConsumerConfig values` dump + per arm, which prints `bootstrap.servers`. The sidecar's stderr is inherited by the demo's + container, so that address lands in the demo's own lines: `bin/ci-demo-conformance.sh` greps for + exactly that and would have failed C++ on the credential rule. `demo/sidecar-launcher.sh` now + passes `-Dorg.slf4j.simpleLogger.defaultLogLevel=warn`, overridable with `PC_SIDECAR_LOG_LEVEL`. + **Every language whose demo image gains that binding inherits both problems**, so this is worth + carrying into the repo-wide defect note rather than leaving as a C++ detail. + +#### `bin/ci-demo-conformance.sh` no longer recognises this table, and that is silent + +**Owed back to whoever owns `bin/`** - out of this note's ownership, so it is recorded rather than +fixed. That script reduces each demo's stdout to a skeleton and requires the skeletons to match. Two +of its `awk` patterns are now stale against the contract they enforce: + +- the header pattern is `arm ... elapsed ... msg/s ... vs AK core`, which the two new columns break; +- the row pattern accepts `[A-Za-z0-9 _-]*` for an arm name and then `s `, + which both the parentheses in `AK core (librdkafka)` and the two extra figures break. + +**Neither failure is visible as a failure.** With both patterns missing, a skeleton degrades to its +`DIAL` and `TITLE` lines, every language degrades identically, the diff is clean and the script +reports agreement - having stopped checking the columns and the arm order it exists to check. The +absolute assertions (the dials echoed, the address absent, no latency) still hold, so the run is not +worthless; it is just weaker than it reads. Fixing it means widening both patterns and normalising +the new `(library)` suffix the way `normalise_arms` already normalises the sidecar arm's name. + +#### What this pass was verified against + +`demo/run.sh --records 20 --concurrency 4 --partitions 2 --replay-factor 2` with +`PC_DEMO_DELAY_MS=3` - the conformance harness's own input - in the container, which is C++'s only +mode. It exited 0. Observed: the banner first, then the fingerprint; both arms labelled with their +library; both tables carrying `records` and `keys`; `20/20` in the small replay and `40/40` in the +big one, which is what the seeding makes deterministic at this volume (the key space is larger than +the record count, so keys equals records here and would equal the key space at default scale). With +the sidecar quietened the whole run is a few dozen lines, and `bootstrap.servers` appears nowhere in +them. + +**No throughput figure from that run means anything and none is recorded here.** Ten language demos +were building and running on the box at the time; the elapsed columns are contention and start-up. +Output shape is what was being proven. + +#### Rendering moved out of `demo.cpp` so the contract could be tested + +`demo/src/demo_report.{h,cpp}` now owns `ArmResult`, the banner and the table; `demo.cpp` prints +what it returns. The split exists for one reason: the banner's wording, the column set and their +order are contract, and the arms that produce the figures need a broker that the image build does +not have - but rendering does not. `demo/tests/demo_report_test.cpp` holds all of it, and runs at +image-build time beside the option tests. The renderers return strings rather than writing to +`std::cout`, because the thing worth asserting is the text. + +### What is NOT wired up + +- **Neither entry point is in `bin/ci-demo-test.sh`.** That script and `.github/` were outside this + wave's ownership, so the C++ demo is not run on every pull request the way the Java one is. The + contract is explicit that "a demo with one tested entry point has an untested entry point", so + this is a real gap rather than a deferral of polish. What it would need: a `cpp` row that runs + `demo/run.sh --records --replay-factor 1`, and nothing else - there is no native path to + test separately, which makes C++ the cheapest language to add there. +- The demo's own tests cover the option surface and the output shape. The ARMS still need a broker, + which the image build has none of - so nothing at build time proves that the `records` and `keys` + columns are counted correctly, only that the table reports them. Running the demo is what proves + the figures. + ## Not implemented, and therefore not declared Heartbeats and the liveness lease, the manifest reconnect and `Drop`, `WorkerDied`, terminal diff --git a/docs/inflight/clients/dotnet.md b/docs/inflight/clients/dotnet.md index 9e4ddf30a5..5cc2b23a55 100644 --- a/docs/inflight/clients/dotnet.md +++ b/docs/inflight/clients/dotnet.md @@ -11,8 +11,12 @@ report, and a clean client-initiated shutdown, proven by one end-to-end test aga sidecar. The module is at `parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/`; its maturity and testing-evidence deferrals are lifted. Later waves: leases and heartbeats, the manifest reconnect, -worker death, terminal outcomes, the shutdown drain, the demo and its container, NuGet packaging, -and the rest of the conformance suite. +worker death, terminal outcomes, the shutdown drain, NuGet packaging, and the rest of the +conformance suite. + +**The demo has landed** at `parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/` +- two arms, both entry points, container included. What it found, what it diverges on, and what it +left open are in [The demo](#the-demo) below. ## The falsification result @@ -150,6 +154,173 @@ Formatting is checked by `dotnet format`, correctness by the build. green here. `dotnet format --verify-no-changes` - whitespace, style and analyzers together - is green too, and is the stricter command a local check should use. +## The demo + +`demo/run.sh` in this module, mirroring the contract in `parallel-consumer-proxy/demo/README.md`: +the same flags and defaults, the same `PC_DEMO_*` variables with the same precedence, the banner +first, then the fingerprint without the bootstrap address, then the same two tables, no latency +anywhere. Two arms - `AK core (Confluent.Kafka)`, serial, and `dotnet-grpc (this client)`, this +module's client library over a sidecar it spawns. The demo project joined the module's one +solution, so the ordinary `dotnet build` and the clients CI row keep it compiling under the same +analyzers-as-errors lint as the library. + +### The divergence that stopped being one: an awaited timer, not a blocking sleep + +The contract used to list C# among the languages where a blocking sleep is fine, naming Python and +TypeScript as the only exceptions. **For this client that list was wrong, and the demo diverged.** +The library's executors are `Task`s on the thread pool, so `Thread.Sleep` in the user's function +occupies a pool thread; at the contract's default `--concurrency 100` the pool would be injecting +replacement threads at roughly one a second and the sidecar arm would report the injection rate +rather than the engine's throughput. Both arms use `await Task.Delay(...)`, because the serial arm +is the denominator of every ratio and must not differ from its numerator by the wait primitive as +well as by the transport. + +**Recorded here rather than edited into the contract, and the contract has since absorbed it.** The +argument this note made - that the predicate is the client's concurrency *shape* and not the +language's name - is now the rule as written: *is the client thread-per-record?* Six languages fail +it, C# among them for exactly this reason, and the note's two guesses at the neighbours (Kotlin's +coroutines, Swift's cooperative pool) are both on that list. So this module no longer diverges from +anything; the demo's code and README say so, and the reasoning is kept here because it is the +evidence behind a rule eleven languages now follow. Rust supplied the measurement that made it +undeniable, not this module - 10,341 msg/s through its blocking adapter against 3,518 with a raw +thread sleep. + +### The reader-experience pass (the contract's "output a reader actually sees") + +Applied here after the contract gained the section. Three changes, all in the demo, none of them +touching what is measured: + +- **The banner is the first thing the demo prints**, before the broker is resolved and before the + fingerprint, in the contract's exact shape with `.NET` as the language. It is printed before the + argument parsing too, so a reader who mistypes a flag is still told what they were trying to run. + **What still precedes it on the native path is `run.sh`'s own scaffolding** - the mode line, the + Maven sidecar build, the `dotnet build`. That is the script, not the demo, and the demo is what + the contract binds and what the conformance harness captures from the container; a script that + has to build a JVM sidecar and a .NET project cannot make the banner byte one. The `dotnet build` + summary block was silenced (`-consoleLoggerParameters:NoSummary`) so the banner is at least the + line after the build, and build errors are unaffected by that flag. +- **Both arms are labelled with the role and the client**: `AK core (Confluent.Kafka)` and + `dotnet-grpc (this client)`. The bare `dotnet-grpc` survives as a separate constant for the + consumer group id - a label with spaces and brackets has no business in a Kafka group name or the + metric names derived from it. +- **Two deterministic columns, `records` and `keys`**, appended after `vs AK core`. Appended rather + than inserted for a concrete reason, below. Keys are counted over the key BYTES as base64 (a + Kafka key is not text) with a `` sentinel, in a `ConcurrentDictionary` in both arms - the + serial arm does not need concurrency but must not reach a compared figure by a different route + than the arm it is the denominator of. Measured at `--records 20 --partitions 2`: both arms + report 20 records and 20 keys in the small replay, and the sidecar arm 40 and 40 in the big one, + which is what the seeding (`key-{ordinal % 1000}`) predicts. + +### The new output silently blinds `bin/ci-demo-conformance.sh`, and `bin/**` was not mine to fix + +**Reproduced, not inferred**: the script's own `skeleton()` awk, run verbatim over this demo's real +capture, emits `DIAL`s, both `TITLE`s and both `HEADER`s - and **not one `ROW` line**. Before this +change the same capture produced three. + +Two independent breakages, and only one of them is about the labels: + +- **The `ROW` matcher is end-anchored on the ratio**: `... [0-9,-]+ [0-9.x-]+[[:space:]]*$`. Two more + columns after `vs AK core` leave nothing for `$` to match, so the row is dropped **whatever the + arm is called**. Every language adding the contract's `records` and `keys` hits this, including + one that keeps `AK core` bare. +- **The arm-name class is `[A-Za-z][A-Za-z0-9 _-]*`**, which admits the space in `AK core` but not + the brackets or the dot in `AK core (Confluent.Kafka)`. And `normalise_arms` maps + `^ROW [a-z0-9]+-grpc$`, anchored, so `dotnet-grpc (this client)` would not normalise either. + +**The failure mode is the dangerous one: silent.** Unmatched lines are dropped rather than flagged, +so every language emits a skeleton with no `ROW` lines, the skeletons still agree with each other, +and the drift check passes green while having stopped comparing the arms entirely - which is most of +what it was there to compare. + +The `HEADER` matcher is *not* anchored at its end, which is why the two new columns went **after** +`vs AK core` rather than before: that ordering is the one that keeps the header recognised, and +every language should use it. Whoever owns `bin/**` needs to extend the `ROW` pattern by two columns, +widen its arm-name class, and widen `normalise_arms`, in the change that lands these labels. The +absolute assertions - dials echoed, no bootstrap address, no latency - are unaffected and still pass. + +### Two findings against the product, both from running it + +- **The sidecar's own log is unreachable from the client, in every language.** The proxy's `Main` + prints `port: ` on stdout and that channel belongs to the client library, which drains and + discards everything else on it. `logback-classic` is on the sidecar's runtime classpath with no + configuration file anywhere in `parallel-consumer-proxy/src/main`, so its default + `ConsoleAppender` writes to **stdout** - into the channel that is discarded. `ClientOptions` + offers `SidecarErrorLog`, and it carries stderr, which is empty. Measured: a failing sidecar arm + with `PC_DEMO_SIDECAR_LOG` set produced not one line. Either the sidecar should log to stderr, or + the lifecycle channel's non-port lines should be offered to the client; today a sidecar that + refuses a session can only be debugged by rebuilding it. +- **R48's reason-free rejection is correct and still costs an hour.** `Configure` failed with + "constructing the session's Kafka clients and engine failed with + `org.apache.kafka.common.KafkaException`" and nothing else, deliberately, because a Kafka + `ConfigException` embeds property values and those may be credentials. The actual cause was the + bootstrap address (below). **Naming the offending property KEY without its value would have + settled it immediately** and gives nothing away - the keys are the client's own and travel in + `Configure`; it is the values R48 protects. Worth putting to the protocol documents. + +### The trap the next .NET reader will hit + +Testcontainers for .NET returns its address as a URI: measured here, +`plaintext://127.0.0.1:62347/` - lower-cased scheme and **trailing slash**. librdkafka accepts that +string verbatim, so the AK core arm worked with it untouched; the Java client behind the sidecar +rejects it, and the trailing slash survives naive scheme-stripping. `DemoBroker.NormaliseBootstrap` +reduces each comma-separated entry to `host:port`. Two clients, two parsers, one address that +travels between them - the same shape will bite any language whose Kafka client is librdkafka-based +(Python, Ruby, C++, Go's confluent-kafka-go) if it starts its broker with that language's +Testcontainers. + +### The container could not be built on an arm64 host, and that was a module bug + +`Grpc.Tools` **2.71.0** ships a bundled `linux_arm64` `protoc` that segfaults - MSB6006, "exited +with code 139" - when MSBuild spawns it inside a container on an Apple Silicon machine. The whole +module, not just the demo, was unbuildable there. + +Established with a control arm rather than by upgrading and hoping. In one container, one four-line +`.proto`, one project, **only the Grpc.Tools version changed**: 2.71.0 died at MSB6006, 2.83.0 +generated the stubs. Two facts make it an environment bug rather than a bad argument: the same +`protoc`, run **by hand with MSBuild's exact command line** (captured with +`-consoleLoggerParameters:ShowCommandLine`), succeeded at *both* versions; and the same minimal +project built under `--platform linux/amd64` at 2.71.0. So it is specific to protoc-spawned-by- +MSBuild on linux/arm64 - which means **CI's amd64 runners never saw it**, and only a developer on +Apple Silicon would. + +`GrpcVersion` is therefore 2.83.0 with the control written beside it in `Directory.Build.props`. +The bump is otherwise inert: `ProtobufVersion` stayed at 3.31.1, the library builds warning-free +under the same analyzers-as-errors lint, `dotnet format --verify-no-changes` is clean, and the +module's conformance test still passes against the real wire. **If it is ever lowered, re-run the +control.** + +### What is open + +- **No CI runs this demo.** `bin/ci-demo-test.sh` drives the Java demo through both entry points and + is Java-only; extending it - or adding a sibling - was out of this session's ownership + (`bin/**` was off limits). Until then the .NET demo has exactly the untested entry points that + script exists to prevent, and the contract says both entry points being tested is part of it. +- **`bin/ci-demo-conformance.sh` needs its `ROW` and `normalise_arms` patterns widened** for the new + arm labels, or its drift check silently stops comparing arms - see the section above for the two + exact regexes. Same ownership boundary: `bin/**` is not this branch's. +- **No default-scale run has ever happened.** Every run was at `--records 20`, chosen to prove the + machinery on a machine running ten agents at once. At that volume both arms are dominated by + consumer-group join time and the ratios in the tables are meaningless; nothing in this branch + should be read as a measurement. A run at the contract's defaults on an unloaded machine is + outstanding. +- **The image is large and rebuilt from scratch on any source change.** It carries a .NET SDK, a + JDK, a populated `/root/.m2` and the built reactor, because the sidecar classpath baked in at + build time has to point at files the running container actually has - the same trade the Java + demo's image documents. Nothing here has been optimised; a reader on a cold cache waits several + minutes. Measured while verifying the reader-experience pass: the `COPY . .` layer is invalidated + by **any** change in the repository, including one that touches no .NET file at all - so making a + commit mid-build costs the whole Maven layer again, which was 480-520s on its own here, plus + around 400s to export and load a 2.3GB image. **Do not edit the tree while `--docker` is + building**, and budget well past a ten-minute timeout for a cold container run. +- **A killed `docker compose up` strands the network and the NEXT run fails, not the killed one.** + Observed here: `network pc-dotnet-demo_default was found but has incorrect label + com.docker.compose.network set to "default" (expected: "")`. It reads like a compose-file defect + and is not one; `docker compose -f /docker-compose.yml down --remove-orphans` clears it. + Worth knowing before anyone "fixes" the compose file in response. +- **`shared/JvmToolchain.cs` is compiled into two projects by `Compile Include` link** - the test + harness and the demo both have to find a JVM, and neither is a library anyone references. If a + third consumer appears, that is the moment to reconsider an assembly. + ## Plan defects (same as Go's, restated because they are still open) - **The unit's verification command cannot work as written.** diff --git a/docs/inflight/clients/go.md b/docs/inflight/clients/go.md index a40b2137c2..ed1512eb20 100644 --- a/docs/inflight/clients/go.md +++ b/docs/inflight/clients/go.md @@ -11,8 +11,12 @@ report, and a clean client-initiated shutdown, proven by one end-to-end test aga sidecar. The module is at `parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/`; its maturity and testing-evidence deferrals are lifted. Later waves: leases and heartbeats, the manifest reconnect, -worker death, terminal outcomes, the shutdown drain, the demo and its container, packaging, and the -rest of the conformance suite. +worker death, terminal outcomes, the shutdown drain, packaging, and the rest of the conformance +suite. + +**The demo wave has landed too** - `parallel-consumer-proxy-client-go/demo/`, both entry points run. +Its divergences from the contract, and what it found in the contract, are the last two sections of +this file. ## The falsification result @@ -214,3 +218,238 @@ it cost a negative control: naming what is implemented, so it cannot fall out of step by omission. - **The token is echoed as the received message**, never rebuilt from parsed fields, so "opaque" is structural rather than a rule someone has to remember. + +## The demo wave: what it diverges on, and why + +The demo is `parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/`, keeping +`parallel-consumer-proxy/demo/README.md`'s contract. Flags, environment variables, precedence, +defaults, the fingerprint-first rule, the never-print-the-bootstrap rule, the two tables and their +columns, and no latency anywhere: all mirrored exactly, and `options_test.go` derives the seven +environment-variable names from the flag list rather than trusting seven hand-written bindings. + +Three divergences, all recorded in the demo's own README where a reader meets them: + +- **The demo binary never starts a broker; `run.sh` does.** Java's `DemoBroker` falls back to + Testcontainers when no `--bootstrap` arrives. Go has no comparable dependency in this repo, and + adding one would put a Docker client library and its tree into a demo whose point is that the + application needs no infrastructure. So natively `run.sh` starts a `cp-kafka` container on a + random free port and removes it on exit, in the container the compose sibling is the broker, and + the binary always receives an address. The *user-facing* contract is unchanged - omit + `--bootstrap` and a broker appears. +- **`PC_DEMO_SIDECAR_CLASSPATH`**, which is plumbing rather than an eighth flag: no `--flag`, no + default a user would set, and it disappears the day the sidecar ships as a binary. +- **The demo is a nested Go module.** Go's module graph propagates requirements to every consumer, + so franz-go in the library's `go.mod` would hand a Kafka client library to applications whose + whole reason for using the proxy is not needing one. The consequence: `go build ./...` and + `go test ./...` in the parent do NOT descend into `demo/`, so the module's Maven-driven build + never compiles it - `run.sh` and the Dockerfile do. + +### Open follow-up: staticcheck does not cover the demo module + +`scripts/analyse.sh` runs `go vet` over the demo and stops there. `go tool staticcheck` builds the +version pinned in the module it runs in, and `demo/go.mod` does not pin it; adding the `tool` +directive puts staticcheck's dependency tree into the module graph that `demo/Dockerfile` resolves, +and that image was verified without it. Small, and worth doing on an unloaded machine: add the +directive, `go mod tidy`, extend the script, then rebuild the demo image once to confirm the +container path still builds. + +### Two things worth knowing before the next Go wave + +- **`GOTOOLCHAIN=local` makes the client module unbuildable on an older Go.** The library declares + `go 1.25.0` and uses `tool` directives, which Go 1.23 cannot even *parse* (`unknown block type: + tool`) - so on a machine pinned to `local` with 1.23 installed, `go build` fails on the go.mod + rather than on the code. `run.sh` probes the module with `go list -m`, retries once with + `GOTOOLCHAIN=auto`, says out loud that it is doing so, and only then falls back to the container. + Measured on the development machine, which is pinned that way. +- **Kafka refuses `0.0.0.0` as a KRaft controller listener's bind address**, not only as an + advertised one: the format step aborts with "advertised.listeners cannot use the nonroutable + meta-address" before the broker starts. `run.sh`'s native broker binds `CONTROLLER://localhost` + for that reason - measured, not reasoned about. The compose sibling was already correct because + it binds the service name. + +## Contract defects the demo wave found (report only - do not edit the contract) + +Recorded rather than fixed, per KTD23: the shared contract at `parallel-consumer-proxy/demo/README.md` +is the integrator's to change. + +1. **"Omit `--bootstrap` to start one" assumes every language can start a broker, and most cannot + cheaply.** The contract inherits Java's Testcontainers fallback without saying *who* starts the + broker, and nine of the ten remaining languages will each have to decide it alone - some with a + heavyweight dependency, some by moving it into `run.sh` as Go did, some perhaps not at all. The + contract should say which layer owns it, or say explicitly that it is the language's choice as + long as the user-facing promise holds. + +2. **Every non-JVM demo container is a two-toolchain image, and the contract does not mention it.** + "A reader with only Docker can run it" is satisfied, but only by shipping a JDK alongside the + language's own runtime, because the sidecar is a JVM application spawned as a child of the + running demo - it cannot be a build-stage artifact the way the demo binary can. Go's image is a + `golang` build stage discarded into an `eclipse-temurin:17-jdk` runtime. Ten languages will each + rediscover this; it belongs in the contract's container section beside the socket rule. + +3. **The flag/environment table has no room for the sidecar's location, which every non-JVM demo + needs.** Go added `PC_DEMO_SIDECAR_CLASSPATH` under the `PC_DEMO_` prefix the contract reserves + for flags. An integrator diffing environment variables across languages will see an extra one + and cannot tell from the contract whether it is a divergence or a shared necessity. It is the + latter, in some spelling, for every language whose demo spawns the JVM sidecar. + +4. **`bin/ci-demo-test.sh` runs the Java demo only.** The contract says both entry points are tested + "on every pull request" and that "a per-language demo inherits this", but the script hard-codes + the Java `run.sh` path and the Java arm list. Nothing yet runs a per-language demo's two entry + points in CI, so ten demos can ship untested while the contract says otherwise. Out of this + wave's ownership; named here because it is the gap the contract's own closing paragraph warns + about. + +5. **The big replay's table can be one row, and the `vs AK core*` column then compares the only arm + against an arm that is not in the table.** That is what the contract asks for and it is not + wrong, but it reads oddly, and it will read identically in every language except Java. Worth a + sentence in the contract saying the single-row case is expected rather than a bug. + +## The reader-experience wave: what the Go demo now prints + +The contract's "The output a reader actually sees" section was rewritten from someone watching a +demo and finding it unimpressive. Applied here as four changes, all inside +`parallel-consumer-proxy-client-go/demo/`: + +- **A banner, printed before anything else the demo says.** `main.go` holds it as a `const` copied + verbatim from the contract rather than composed from parts, and `report_test.go` asserts its four + lines, the 64-character rules, the words "Parallel Consumer", the fixed second line, and that it + names *Go* - the one thing that differs per language and the one thing a copy-paste would leave + pointing at the wrong demo. +- **Both arms name the client they ran**: `AK core (franz-go)` and `go-grpc (this client)`. The + labels are still the row labels and the `=== arm starting ===` lines, so a reader sees the client + named at every point the arm speaks. +- **`records` and `keys` columns**, from a small `keySet` both arms share. In the sidecar arm the + key is recorded *before* the count is incremented, which is what makes the set safe to read the + instant the count reaches the target - a counted record has already contributed its key. Covered + by `go test -race`. +- **The broker log levels were already `WARN`** in `docker-compose.yml`, from the contract's own + commit; nothing here touched them. + +### The column ORDER is a judgement, and it is where eleven demos will drift + +The contract fixes that the two columns exist and what they mean; it does not say where they go, and +eleven agents chose independently at the same moment. Go chose: + +``` +arm | records | keys | elapsed | msg/s | vs AK core +``` + +Reasoning, for whoever reconciles: the contract's own heading is "reports what it did, **not just** +how fast", so what it did precedes how fast; and `vs AK core` is derived from `msg/s`, so the three +speed columns stay adjacent rather than being split by two counters. **If the other ten chose +differently, Go is the one to move** - nothing here depends on the order but the format strings and +one test. + +### `bin/ci-demo-conformance.sh` cannot parse this output yet + +Not ours to fix (`bin/**` is outside this wave's ownership), and it is not a Go-specific problem - +it will be true of all eleven demos at once. Three of its regexes are written against the old shape: + +- the `HEADER` line matches `arm ... elapsed ... msg/s ... vs AK core`, and `records` and `keys` + now sit between `arm` and `elapsed`; +- the `ROW` line matches an arm name of `[A-Za-z][A-Za-z0-9 _-]*` followed directly by an elapsed + figure - the parentheses in `AK core (franz-go)` fail the character class, and the two new + integer columns fail the "elapsed comes next" anchor; +- `normalise_arms` maps `^ROW [a-z0-9]+-grpc$` to `ROW SIDECAR`, which no longer matches + `go-grpc (this client)`. + +The consequence is a *silent* one and worth naming: a demo whose skeleton comes out empty is +**SKIPPED, not failed** ("produced no recognisable fingerprint or table"), and a run in which every +language is skipped fails only on "no demo produced usable output". Whoever updates the script +should also make the new deterministic columns assertions in their own right - `records = 20` and +`keys = 20` for the standard invocation - since those two are the only figures in the table the +harness can hold every language to. + +### Go has three serious Kafka clients: named in the README, one run + +franz-go (pure Go, the one that runs), confluent-kafka-go (a cgo binding to librdkafka) and sarama +(pure Go). The contract asks for the disclosure and invites a second arm; the demo's README carries +the disclosure and does not add the arm. + +- **confluent-kafka-go is ruled out on the container path**, not on preference. `demo/Dockerfile` + builds with `CGO_ENABLED=0` precisely so the binary can be produced in a `golang` stage and run + in the `eclipse-temurin` stage the spawned JVM sidecar needs. A cgo client makes that two + runtimes or two images. +- **sarama is cheap and still not added**, and this is the part worth overturning later: the + blocker is `bin/ci-demo-conformance.sh`, which requires every language's skeleton to be identical + and exempts only Java for its extra arms. A third row in Go alone is permanent drift, so the + harness would fail a demo for doing what the contract invited. **The right sequence is: give the + harness a way to say "this language legitimately carries an extra arm", then add sarama.** Filed + here rather than acted on because the harness is outside this wave's ownership. + +### Two smaller things + +- **The banner is the first line of the DEMO's output, not of the terminal, natively.** `run.sh` + prints its mode line and its build progress first. That is build chatter rather than the demo + introducing itself as something else, and duplicating the banner in the script would have meant + printing it twice in one run. Recorded in case the integrator wants it in the script instead. +- **`keys` is `min(records, 1000)` in every language**, because all eleven seed `key-{n % 1000}` - + checked across the other ten demos' broker files rather than assumed. That is what makes the + column comparable, and it also means the column stops distinguishing anything above 1000 records: + at the default 2000 both arms report 1,000, which is correct and says less than it looks like. + +### What was actually run, and what it is worth + +`demo/run.sh --records 20 --concurrency 4 --partitions 2 --replay-factor 2`, natively, exit 0. The +shape is right: banner, then the fingerprint, then two tables; both arms named with their client; +`records` 20 and `keys` 20 on the small replay and 40/40 on the big one, which is what the seeding +predicts; no bootstrap address and no latency anywhere. + +**No throughput figure from this run means anything and none is quoted.** Eleven language agents +were building and running demos on this machine at once at load averages between 40 and 100, and +the serial arm took 10.3s for twenty records with a 2ms delay - about eighty times the work implied +by its own dial. The deterministic columns are exactly why that run is still evidence: they are the +half of the table that a loaded machine cannot distort. + +**The first two attempts died on the environment, not on the demo**, and it is worth knowing the +signature. The broker container exited immediately with +`java.nio.file.FileSystemException: /var/lib/kafka/data/__cluster_metadata-0: No space left on +device` - the Docker VM's disk, exhausted by eleven agents' images and broker volumes at once. What +the demo *shows* is `the broker did not become ready within 60s`, followed by the broker's own log, +which is `run.sh` doing the right thing; the space message is only visible because it prints that +log. Several other languages' brokers (`pc-cpp-demo-broker-1`, `pc-rust-demo-broker-1`) were dying +the same way in the same minutes. It cleared on its own as other agents' runs finished. Nothing was +pruned: reclaiming 30GB of shared images would have cost the other ten agents a full rebuild. + +**The conformance harness's blindness to the new shape was verified, not inferred.** The +`skeleton()` function was lifted verbatim out of `bin/ci-demo-conformance.sh` and run against a +sample of the new output: it emits every `DIAL` and the `TITLE`, and **no `HEADER` and no `ROW` at +all**. A skeleton with dials but no rows is non-empty, so the script does not even take its +"produced no recognisable fingerprint or table" skip - it compares dial lists between languages and +reports agreement, having silently stopped checking the arms. That is worse than the skip, and it is +the reason this is written down rather than left for CI to find. + +### The container path ran too, and the same two columns came out identical + +`demo/run.sh --docker` at the same dials, exit 0. Same banner, same arm labels, and - the point of +the exercise - **`records` 20 / `keys` 20 and 40/40, exactly as in the native run**, while the +speed columns disagreed wildly: the sidecar arm was 8.1x the serial arm natively and 0.5x it in the +container, on the same machine an hour apart. That is not a result about containers; it is eleven +agents' load, and it is precisely the reason the deterministic pair was added. + +### VERIFIED: "The broker is quiet" does not hold, and it is not the compose file's fault + +The container run's own log: **883 levelled lines from the broker against 42 from the demo** - 826 +INFO, 56 TRACE, 1 WARN. The tables are buried exactly as the contract's commit says they must not +be, with `KAFKA_LOG4J_ROOT_LOGLEVEL: WARN` and `KAFKA_TOOLS_LOG4J_LOGLEVEL: WARN` correctly set. + +The cause is in the image, read out of `confluentinc/cp-kafka:7.9.0` rather than guessed. +`/etc/confluent/docker/log4j.properties.template` renders the root logger from the environment +variable - and then unconditionally emits a hardcoded per-package block that **overrides it**: + +``` +{% set loggers = { 'kafka': 'INFO', 'kafka.controller': 'TRACE', + 'state.change.logger': 'TRACE', 'kafka.log.LogCleaner': 'INFO', ... } %} +``` + +A per-logger level beats the root level in log4j, so `KAFKA_LOG4J_ROOT_LOGLEVEL` cannot silence any +package under `kafka`. The top emitters in the captured run line up exactly: `kafka.coordinator.group` +341 lines, `state.change.logger` 233, `kafka.log.UnifiedLog$` 92, `kafka.log.LogManager` 63, +`kafka.cluster.Partition` 60. + +**The lever that would work is `KAFKA_LOG4J_LOGGERS`**, which the same template parses over those +defaults - something like +`KAFKA_LOG4J_LOGGERS: "kafka=WARN,kafka.controller=WARN,state.change.logger=WARN,kafka.log.LogCleaner=WARN"`. +Not applied here: this wave was told explicitly not to touch broker log levels, and the change +belongs in all eleven compose files at once rather than in Go's alone. `KAFKA_TOOLS_LOG4J_LOGLEVEL` +is harmless but redundant - the tools template already defaults to WARN. diff --git a/docs/inflight/clients/java.md b/docs/inflight/clients/java.md index c9efb3fde7..564fb118f9 100644 --- a/docs/inflight/clients/java.md +++ b/docs/inflight/clients/java.md @@ -71,3 +71,81 @@ reasoning**; what a later Java wave needs to know is the shape and what it const payload, FIFO hand-out through one executor, the asynchronous processor answering off-thread, the session-end stage, and the records-out-for-processing leak check. Redundancy between two suites is cheaper than a scenario deleted by whoever knew least about what it was protecting. + +## The reader-experience pass on the demo output (astubbs#242) + +The demo contract (`parallel-consumer-proxy/demo/README.md`) grew three output rules after someone +watched a run and found it unimpressive. Java, as the reference the other ten transcribe, took them +first. What changed here, and what the pass could not close. + +### What the Java demo now prints + +- **A banner, before anything the demo controls.** `ReferenceDemo.BANNER`, printed from `announce` + in `main` **before** `DemoBroker.resolve` - so it precedes the "starting a broker in a container" + paragraph rather than following it. The effective-configuration block moved with it, for the same + reason. `runFor` no longer prints the fingerprint, which means `ReferenceDemoIT` no longer sees it; + that is deliberate - the contract binds the demo's output, and the test asserts on returned + results. +- **Every arm names its client**, as `arm (client)`. `ArmResult` now carries the two separately and + renders `label()`; everything that has to *match* - `baselineOf`, `ReferenceDemoIT`'s expected + arms, `bin/ci-demo-test.sh`'s `REQUIRED_ARMS` - still keys off `arm()` alone, which is what kept + the change from rippling. +- **Two evidence columns, `records` and `keys`.** `ArmTally` accumulates both in one call, so an arm + cannot count a record without noting its key. The arm column is sized to the widest label in the + table rather than to a constant, because `pc-core (ParallelEoSStreamProcessor)` does not fit one. + +### THE TWO `bin/` SCRIPTS THAT NOW NEED UPDATING - one fails loudly, one fails silently + +Both read the tables by regex, and both were written against the old row shape. **Neither was +touched by this pass** (the fan-out that produced it owns `bin/`), and every language changing its +labels breaks them identically, so this is a shared item rather than a Java one. + +- **`bin/ci-demo-test.sh` FAILS.** Its per-arm assertion is + `grep -qE "^[[:space:]]*${arm}[[:space:]]+[0-9]"` - the arm name followed by its elapsed figure. + A row now reads `AK core (KafkaConsumer) 1.9s ...`, so what follows the name is `(`, not a + digit, and every required arm reports "no row". The `java-grpc-uds` assertion further down has the + same shape and the same fate. +- **`bin/ci-demo-conformance.sh` PASSES, having stopped checking the arms.** Its `HEADER` pattern + survives (it is anchored only at the start, and the two new columns were appended *after* + `vs AK core` partly for that reason). Its `ROW` pattern does not: it forbids `(` and `,` in the + name and requires the line to *end* at the ratio. So no `ROW` lines reach the skeleton, the + skeletons still diff clean on `DIAL`/`TITLE`/`HEADER` alone, and the drift check silently covers + less than it did. `normalise_arms` is moot for the same reason. **A green run of this script after + the label change is not evidence the arms agree.** + +### Column order, and why `records`/`keys` went last + +The contract names the two columns but not their position. They are appended after `vs AK core` +rather than inserted after `arm`, which keeps `ci-demo-conformance.sh`'s `HEADER` pattern matching +and reads as "the old table, plus its evidence". **The other ten demos have to match this**, and +nothing enforces the order - the conformance script compares column identity through a pattern that +would accept either arrangement. + +### The contract clause this could not honour: the banner is not the first line + +Roughly thirty lines of logback's own configuration status print before the banner, and **both** +entry points have it - checked, not assumed. `parallel-consumer-core`'s test jar is on the demo +classpath (the sidecar spawn needs it) and carries a `logback-test.xml` with `scan="true"`; logback +cannot watch a file inside a jar, warns, and a warning makes it dump its whole status. **A +`logback.xml` in the demo module would not fix it** - a `logback-test.xml` anywhere on the classpath +outranks it - so the fix belongs wherever that test jar's logging config does, not here. Recorded +rather than worked around; the demo's own README says so too. + +Everything the demo itself controls is in contract order: banner, then the effective configuration, +then the run - the broker paragraph now comes after both rather than between them. + +### What was actually run + +`demo/run.sh --records 20 --concurrency 4 --partitions 2 --replay-factor 2`, natively and with +`--docker`; both exited 0 and printed both tables. Natively, five arms (`java-grpc-uds` correctly +absent on macOS, with the message naming the container); in the container, all six. `records` and +`keys` read 20/20 in the small replay and 40/40 in the big one on both paths - the big replay's arms +re-read from the beginning, so they see the whole seeded backlog, which is why 40 is right there. +`DemoOptionsTest`, `ConfigureParityTest` and `ArmCompletionTest` pass (29 tests), and +`ReferenceDemoIT` passes in the integration lane - so the new `uniqueKeys` equality assertion has +been executed against a real broker and all five arms that platform runs, not merely compiled. + +**No throughput figure from these runs is worth anything and none is recorded here.** Eleven agents +were doing this simultaneously on one machine - load average was over 80 on twelve cores, the first +container attempt had its broker killed with exit 137, and the image build took a quarter of an +hour. The runs prove the output SHAPE. diff --git a/docs/inflight/clients/kotlin.md b/docs/inflight/clients/kotlin.md index 1550ca8263..ce0584bd0d 100644 --- a/docs/inflight/clients/kotlin.md +++ b/docs/inflight/clients/kotlin.md @@ -10,10 +10,10 @@ shared note. `Configure`, one `Dispatch` wave, the user's function, the report with the token echoed verbatim, and a clean client-initiated shutdown - proven by one end-to-end test against the real test-mode sidecar. The module is at `parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/`; -its maturity and testing-evidence deferrals are lifted. Later waves: leases and heartbeats, the -manifest reconnect, worker death, terminal outcomes, the `Shutdown` drain, the demo and its -container, publishing, and the rest of the conformance suite - **most of which this client now -inherits rather than implements.** +its maturity and testing-evidence deferrals are lifted. **The demo and its container have since +landed** - see "The demo" below for what is open about them. Later waves: leases and heartbeats, the +manifest reconnect, worker death, terminal outcomes, the `Shutdown` drain, publishing, and the rest +of the conformance suite - **most of which this client now inherits rather than implements.** ## It wraps `java-grpc`, and the two reasons it did not are fixed at source @@ -171,3 +171,174 @@ implementation - and the registry did want a resolved classpath rather than a tw objects by `JvmClientBindings` in the conformance module, whose README section *A JVM client is a binding, not a subprocess* owns the reasoning. Kotlin is the JVM client that keeps the spawn path covered, because `Sidecar.kt` is the only JVM spawn there is. + +## The demo, and the three things still open about it + +`parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/` - `run.sh`, +`Dockerfile`, `docker-compose.yml`, `README.md` and Kotlin sources under `demo/src`. Two arms, per +the shared contract: AK core, and Kotlin over the sidecar through this module's own client. It +keeps the contract's flags, `PC_DEMO_*` variables, precedence, defaults, fingerprint and two tables. + +**It is not a Maven module, and that was forced.** A new module needs a line in the clients +aggregator pom, which the parallel language waves all share. So the demo lives inside this module, +compiled by a `kotlin-demo` profile activated by `-Dpc.kotlinDemo`, into `target/test-classes` - +the module's published surface is guarded by `-Xexplicit-api=strict` and a demo is not part of it. +**Scala should copy this arrangement rather than re-deciding it**: it is one profile, one extra +Kotlin/Scala compile execution, and one `build-classpath` execution. + +- **No CI entry-point test.** `bin/ci-demo-test.sh` runs the *Java* demo through both entry points + on every pull request; nothing does that for Kotlin, so the contract's "both entry points are + tested" clause is unmet here. That script is `bin/**` and shared, so extending it belongs to + whoever owns the fan-out's CI, not to this module. **This is the largest open item.** +- **The engine reactor edge is now reachable, on purpose.** `-Dpc.kotlinDemo` puts + `parallel-consumer-proxy` in this module's reactor, because the demo hands the spawned sidecar + its own classpath. The module's standing invariant - `./mvnw -pl + :parallel-consumer-proxy-client-kotlin -am validate` must not print `parallel-consumer-proxy` - + still holds for the DEFAULT lane, and was re-measured after the profile was added. Any future + change here must re-measure it rather than assume it. +- **The demo profile also widens `conformance-classpath.txt`**, because that execution takes test + scope and the profile adds the harness to it. Harmless - the conformance runner never names the + engine artifacts, exactly as in the harness lane - but worth knowing before wondering why the + file changes size under `-Dpc.kotlinDemo`. + +### The divergence that stopped being one + +**The simulated work is `delay`, not `Thread.sleep`** - and as of the contract's reader-experience +rewrite this is what the contract *asks for* rather than an exception to it. The rule used to name +nine languages where a blocking sleep was fine, Kotlin among them, and this demo took the rule over +the list. The contract has since replaced the list with a predicate - **is the client +thread-per-record?** - and names Kotlin's bounded coroutine dispatcher as one of the six execution +models that fail it. **The prediction recorded here held**: this note said the language list was +probably a contract defect rather than a Kotlin exception, and that any coroutine-, fiber- or +async-native client was in the same position; the rewrite found four of the nine safe-listed +languages were not safe, and Swift and C# - both named as suspects here - are two of them. + +The reasoning, kept because it is the evidence rather than the ruling: this client runs each record +as a coroutine on `Dispatchers.IO`, whose default parallelism is 64. A blocking sleep there caps +in-flight records at 64 however high `--concurrency` is set, while the fingerprint keeps printing +the number the reader asked for - a throughput figure reported against settings that did not apply, +which is the exact failure the fingerprint exists to prevent. + +Measured, on a heavily loaded machine (load average 23 on 12 cores, ten agents): at +`--records 4000 --delay-ms 50 --concurrency 200 --replay-factor 1` the sidecar arm finished in +**2.7s**, which is 200 record-seconds of work in 2.7s, so about **74 records in flight** - above the +64 ceiling a blocking sleep could ever reach. It is an inference from one arm rather than a two-arm +controlled experiment, and the control arm was not run; contention can only push the in-flight +figure *down*, never above the thread ceiling, so the direction survives the load even though the +absolute rate does not. + +This was recorded here rather than edited into `parallel-consumer-proxy/demo/README.md`, because +that is the fan-out's shared contract file and was not this wave's to change. It was changed by the +contract's own owner afterwards, which is the arrangement working as intended. + +### The reader-experience rules, applied - and the one thing they break that this module cannot fix + +The contract's "The output a reader actually sees" section was rewritten after someone watched a +demo and found it unimpressive. Three of its clauses landed in `KotlinDemo.kt`: + +- **The banner is the first thing the demo prints**, ahead of the broker line, the effective + configuration and every log line. It is printed by the *program*, not by `run.sh`, and that is + the load-bearing choice: on the `--docker` path `run.sh` execs `docker compose` and prints nothing + further, so a banner in the script would be absent from exactly the entry point + `bin/ci-demo-conformance.sh` exercises. The cost is that in native mode `run.sh`'s + `Mode: native - ...` line still precedes it. Judged the lesser evil - the alternative is either a + banner printed twice natively, or no banner at all in a container - but it does mean the literal + first line of a native run is still not the product's name. +- **Both arms name their client.** `AK core (KafkaConsumer)` and `kotlin-sidecar (this client)`. + Kotlin is the awkward case the contract's own examples do not cover: `franz-go` and `rdkafka` are + the language's own clients, and Kotlin has none - **there is no Kotlin-native implementation of + the Kafka protocol**, only wrappers over the same `kafka-clients` jar. So the label names Apache + Kafka's `KafkaConsumer` rather than inventing a Kotlin-sounding name for a client that is not + Kotlin's, and `demo/README.md` answers the contract's "is there a second serious client?" question + with why a third arm would compare two wrappers over one client. +- **`records` and `keys` are now columns.** Both are deterministic here for a reason worth writing + down: the backlog seeded for a replay is *exactly* the arm's target, so no arm can overshoot, and + `records` therefore equals the target rather than approximately equalling it. `keys` is + `min(records, 1000)` - `DemoBroker.KEY_SPACE` is 1000 and the seeder writes `key-{i % KEY_SPACE}` + - so any language that seeds the same way reports the same figure. **A language whose key space + differs will report a different `keys` and look like drift**; that is a seeding question, not an + output question, and it is the one new column that can disagree for a legitimate reason. + +**Column order chosen: `arm | records | keys | elapsed | msg/s | vs AK core`.** The contract fixes +the columns' identity but not where the two new ones go. Records and keys sit beside the arm because +they are what it *did*, ahead of how fast it did it, and because the ratio column carries a footnote +marker and reads worst mid-table. The three pre-existing columns keep their relative order. **If the +other ten languages appended instead, this is the line that will differ** - and the fix is one +`String.format` here, not a redesign. + +**`bin/ci-demo-conformance.sh` cannot parse this output any more, and that is `bin/**` to fix, not +this module.** Two of its `awk` rules stop matching, independently: + +- the `HEADER` rule matches `arm[[:space:]]+elapsed`, and the header now reads + `arm records keys elapsed`; +- the `ROW` rule's arm-name class is `[A-Za-z][A-Za-z0-9 _-]*`, which excludes the parentheses every + language's arm label now carries, and its `normalise_arms` sed matches `^ROW [a-z0-9]+-sidecar$`, + which `kotlin-sidecar (this client)` no longer is. + +Both break for **every** language simultaneously, so the script fails closed - "produced no +recognisable fingerprint or table" - rather than silently passing a demo it can no longer read. That +is the right failure, but it means the script must be updated in the same change that lands these +labels, or the conformance gate reports nothing at all. + +### What was actually run, and under what load + +All on the loaded box described above. **Absolute throughput here is contended and should be +re-measured on an idle machine before it is quoted anywhere.** + +- Native, `--records 20 --delay-ms 1 --concurrency 4 --partitions 2 --replay-factor 1`: both arms + completed, tables printed, exit 0. +- Native, **no arguments at all** - the double-click case, and the one that has broken before: both + replays ran and exit was 0. AK core 2000 records in 6.7s; kotlin-sidecar 2000 in 1.3s, then 40000 + in 3.5s. +- Container, `demo/run.sh --docker --records 20 ...`: image built from the repository context + (9.17MB uploaded, so `.dockerignore` is doing its job), broker came up as a compose sibling, the + sidecar was spawned inside the demo container with no host Docker socket anywhere, both arms + completed, exit 0. +- Container, **`docker compose up` with no arguments and no environment** - the second documented + entry point, and the one a reader who has never seen this repository types: both replays ran, + demo container exited 0. AK core 2000 in 7.6s; kotlin-sidecar 2000 in 1.1s, then 40000 in 5.0s. +- Argument handling, without a broker: an unknown flag exits **2** with the usage text rather than + reporting numbers for settings nobody asked for; and `PC_DEMO_RECORDS=7 PC_DEMO_DELAY_MS=9 + --records 11` printed `records = 11, delayMs = 9`, which is the contract's precedence - flags beat + the environment beats the defaults - and no bootstrap address anywhere in the fingerprint. + +**After the reader-experience rewrite**, on a box under **load average 43-80 across 12 cores** with +ten other language agents on it. Shape only - **no throughput figure from this run is worth +quoting**, and none is recorded here for that reason. + +- Native, `--records 20 --concurrency 4 --partitions 2 --replay-factor 2`: exit 0. The banner is the + first thing the JVM prints; both arms reported `20 records / 20 keys`, the big replay `40 / 40`. +- Container, the same flags via `--docker`: exit 0, `demo-1 exited with code 0`. The banner is the + **first line of the demo container's own stream** - which is the property that matters, because + compose interleaves the broker's log with it and the conformance script reads `demo-1` lines only. + Same figures: `20 / 20`, `20 / 20`, `40 / 40`. +- An unknown flag still exits **2**, with the banner, the message and the usage text - re-checked + because the banner now prints before argument parsing. +- **One aborted container attempt is worth knowing about**: the in-container Maven compile took + **500s** under this load and a 10-minute cap killed it mid image-import. It is a machine-load + observation, not a demo defect - the same command completed on the next attempt - but anything + timing this entry point in CI should budget for a cold image build, not a warm one. + +### A sighting that is not the Kotlin client's, recorded here only because of file ownership + +Running this module's default lane (`./mvnw -pl :parallel-consumer-proxy-client-kotlin -am test`) +during the demo wave, on a machine at **load average 83 of 12 cores**, failed one test in +`parallel-consumer-core`: + +``` +BlockedThreadAsserterTest.functionThatReturnsOnItsOwnScheduleIsRejected + BlockedThreadAsserter accepted a function that returns on a timer rather than on the unblocker, + but it must have rejected it +``` + +**Contention, established by control rather than asserted**: the same class re-run on its own +passed 7/7 immediately afterwards, and the Kotlin module's own tests are green (20/20). This is the +helper's *self-test* - it feeds the asserter a function that returns on its own timer and expects +rejection - so under enough load the timer-returning function is late enough to look like a genuine +unblock. + +`docs/inflight/test-untracked-ci-flakes.md` already carries `BlockedThreadAsserter` prior art, but +for a **different** test and a different signature (`assertUnblocksAfter` measuring a window two +milliseconds short, owned by astubbs#262). This one is a new signature on the same helper. +**It belongs in that file, not this one** - it is recorded here because the Kotlin demo wave owns +only `clients/kotlin.md`, and the fan-out's integrator should move it. diff --git a/docs/inflight/clients/python.md b/docs/inflight/clients/python.md index 94ce91697c..96e0ba1254 100644 --- a/docs/inflight/clients/python.md +++ b/docs/inflight/clients/python.md @@ -1,8 +1,150 @@ # Python client (astubbs#242) Wave one landed: connect, configure, one dispatched record through a worker process, report, clean -drain. Leases, heartbeats, reconnect, worker death, terminal outcomes, the demo and PyPI packaging -are all deferred and named in the module's testing-evidence `limitation`. +drain. **The demo landed on `demos/python`** (`parallel-consumer-proxy-client-python/demo/`) - two +arms, both entry points, container included. Leases, heartbeats, reconnect, worker death, terminal +outcomes and PyPI packaging are all still deferred and named in the module's testing-evidence +`limitation`. + +## The demo's divergences from the shared contract, and which of them is a contract defect + +The contract is `parallel-consumer-proxy/demo/README.md`, which the demo wave may not edit. These +are recorded here for the integrator to resolve. + +### `--concurrency` defaults to 16, not the seed's 100 - forced, not chosen + +The proxy's executor-count function is `IntUnaryOperator.identity()` +(`docs/inflight/blocker-executor-count-formula.md`), so `max_concurrency` **is** the worker count, +and in Python a worker is a process. The seed's default of 100 means a hundred interpreters. Sixteen +is what the demo asks for instead; `--concurrency` still does exactly what it says, and the +fingerprint prints the effective value, so nothing is hidden. + +Fixed rather than derived from `os.cpu_count()` on purpose: a default that changes with the machine +cannot be compared between two readers' runs, and the fingerprint would be the only place the +difference showed. + +**This does not unblock the blocker.** It is a demo picking a survivable number; the formula is +still an open owner decision that every one of the ten client authors inherits. + +### The contract's two new columns stop `bin/ci-demo-conformance.sh` seeing any table at all + +**Measured, not predicted**: the six-column table this demo now prints, fed through that script's +own `skeleton()` awk, produces **no `HEADER` line and no `ROW` lines** - only the `DIAL` and `TITLE` +ones. Two independent causes, and either alone is enough: + +- the header pattern requires `arm` to be followed immediately by `elapsed`, and `records` and + `keys` now sit between them; +- the row pattern allows `[A-Za-z0-9 _-]` in an arm name, and every arm label now carries its + client's name in **brackets** - `AK core (confluent-kafka)`. + +The script still exits 0. Its absolute assertions read `DIAL` and `TITLE` lines, which survive, so +the run stays green while the drift check silently narrows to *the fingerprint and the two +headings* - it can no longer see a table's columns, its arms, or their order, which was most of +what it was for. That is the check-that-passes-without-having-run class again +(`docs/solutions/workflow-issues/a-check-that-reports-success-without-having-run.md`). + +`bin/**` is outside the language waves' file scope, so **the integrator owns the repair**, and it +is worth more than a regex fix: `records` and `keys` are *deterministic*, which the contract says +is exactly what the harness relies on - so the skeleton should carry their **values** rather than +masking them like elapsed and msg/s. Today it masks every figure, so eleven languages could each +report a different record count and the drift check would still be clean. + +### `KAFKA_LOG4J_ROOT_LOGLEVEL: WARN` does not quieten the broker, in any of the eleven files + +The contract's "the broker is quiet" clause is not being met, and the compose setting that was +supposed to meet it is not enough. Observed on this demo's container path against +`confluentinc/cp-kafka:7.9.0`: the env var is set on the container, the generated config does say +`log4j.rootLogger=WARN, stdout` - and the broker still printed **927 lines** of controller +elections and log-segment chatter, interleaved by compose with the **35** the demo itself produced, +which is exactly the burial the clause was written to prevent. The cause is +that the image also writes **per-logger levels that override the root**: + +``` +log4j.logger.kafka=INFO +log4j.logger.kafka.controller=TRACE +log4j.logger.state.change.logger=TRACE +log4j.logger.kafka.log.LogCleaner=INFO +``` + +Those come from the image's default `KAFKA_LOG4J_LOGGERS`, so the fix is to set that variable too +rather than to change the root level - roughly +`KAFKA_LOG4J_LOGGERS: "kafka=WARN,kafka.controller=WARN,state.change.logger=WARN,kafka.log.LogCleaner=WARN"`. + +**Not applied here on purpose.** It is one line in eleven identical compose files and the demo wave +was told to leave broker log levels alone; fixing it in one language would leave ten quiet in the +compose file and loud on screen. It belongs in the same pass that set the root level. + +### Where the two new columns go is a coordination question the contract does not answer + +The contract names `records` and `keys` and says the tables keep "same columns, same order"; it +does not say where in the row they sit. This demo prints +`arm | records | keys | elapsed | msg/s | vs AK core` - what ran, what it did, then how fast, so a +reader meets the evidence before the number it justifies. + +Eleven language agents chose independently and simultaneously. Until the harness above can see a +header again, **nothing will detect a disagreement**, so this is a merge-time reconciliation rather +than something CI will catch. + +### The banner has to be printed by `run.sh` too, or it is not the first thing a reader sees + +`reference_demo.py` prints it, which covers `docker compose up` and running the module by hand. It +is not sufficient on its own: a native run builds the sidecar's classpath with Maven and starts a +broker first, so the product's name would arrive a minute or so into a screen of build output. +`run.sh` therefore prints the same banner before its own first line and sets +`PC_DEMO_BANNER_PRINTED=1` - a statement of fact, not a dial - which the demo honours and the +compose file forwards, so no path prints it twice. + +Any language whose `run.sh` builds or installs anything before starting the demo has the same +problem, and a language that put the banner only in its demo program will not have noticed it. + +**One consequence to know before writing a check for it**: on the container path launched through +`run.sh`, the banner is on the host's stdout and *not* among the `demo-1 |` lines a compose capture +scopes to. A conformance assertion that looks for the banner in the demo's own output would fail +this language and pass the ones that print it twice, which is the wrong way round. Assert it on the +whole capture, or settle the wrapper-versus-demo question first. + +### Both arms are timed from the first record, not from before consumption + +The seed's clock starts just before consumption, with client construction and the sidecar spawn +already outside the window - "no other arm charges itself for client construction or teardown". +`ParallelConsumerClient.poll()` forks the worker pool, spawns the sidecar, completes the handshake +and starts consumption in **one call**, so that line cannot be drawn in the same place here. + +The demo keeps the seed's rule instead of its line number: start-up is outside the window for both +arms, at the first record either sees. Measured, on the run that settled it: including the JVM boot +would have reported roughly a quarter of the sidecar arm's rate. Any language whose client library +bundles spawn and start into one call - Ruby and C++ next - inherits this question. + +### The demo never starts a broker; `run.sh` does + +The seed uses Testcontainers. The Python equivalent would put a Docker client library into the demo +of a Kafka client library, so `run.sh` brings up the same compose broker its container path uses and +hands the demo an address. The reader-facing promise is unchanged (omit `--bootstrap`, a broker +appears) and the non-negotiable rule is untouched: the demo container is never given the host Docker +socket. `reference_demo.py` invoked directly with no bootstrap says so and exits 2. + +Same question for every non-JVM language, and the compose broker is the cheaper answer in all of +them. + +## Not done, and owed to whoever picks this up + +- **`bin/ci-demo-test.sh` runs the Java demo only** - the script hard-codes the Java module's paths + and arm names, and it is outside this wave's file scope. Until an integrator generalises it, the + contract's "both entry points are tested" clause is met by the Java demo alone and the Python + demo's two entry points are untested in CI. This is the demo defect class the script's own header + is about, so it matters more than it looks. +- **The native path needs a JDK**, because the sidecar is a JVM binary. `run.sh` builds + `parallel-consumer-proxy`'s classpath and exports `PC_DEMO_SIDECAR_CLASSPATH`. The demo already + reads `PC_DEMO_SIDECAR` for an absolute binary, so the day a native sidecar exists the JDK half + drops out of `run.sh` and out of the Dockerfile with no other change. +- **`dependency:build-classpath` needs `-DincludeScope=runtime`**, and the failure is quiet: the + default scope pulls in `parallel-consumer-core`'s test jar, whose `logback-test.xml` configures + the sidecar's logging and prints logback's own status report to **stdout**, ahead of the + `port: ` line the client library scans for. It survives only because that scan tolerates + preceding lines. Any language whose demo spawns the JVM sidecar from a Maven-built classpath hits + this. +- **`src/docs/development/upstream-map.yaml` still has no entry for this work** - outside the wave's + file scope, and unchanged from wave one. ## Two spec divergences, both confirmed independently by the Go wave @@ -35,6 +177,9 @@ stdout/stderr drain threads would otherwise be inherited by the fork, which Pyth Worth reading before the same question is answered differently in another language with real processes (Ruby, C++). -## Not done, and owed to whoever picks the module up - -`src/docs/development/upstream-map.yaml` has no entry for this work — outside the wave's file scope. +The demo inherits it as a constraint on its own bookkeeping: the counters both replays wait on are +`multiprocessing` primitives created before the client exists, because a closure over an ordinary +`int` would be duplicated by the fork and every worker would count only its own records. The +contract's **unique keys** column lands on the same constraint - a `set` cannot cross a fork either +- and `KeyTally` in `demo/reference_demo.py` answers it with a shared byte per key slot rather than +with a `Manager` dictionary, which would put an IPC round trip on the timed path. diff --git a/docs/inflight/clients/ruby.md b/docs/inflight/clients/ruby.md index c1d0fd482d..70a8422bdf 100644 --- a/docs/inflight/clients/ruby.md +++ b/docs/inflight/clients/ruby.md @@ -7,8 +7,9 @@ anything the Ruby wave learns that a later session needs go HERE - never appende shared note. **Wave one landed:** connect, configure, one dispatched record through a worker thread, report, -clean drain. Leases, heartbeats, reconnect, worker death, terminal outcomes, the demo and RubyGems -publishing are all deferred and named in the module's testing-evidence `limitation`. +clean drain. Leases, heartbeats, reconnect, worker death, terminal outcomes and RubyGems publishing +are all deferred and named in the module's testing-evidence `limitation`. The demo has since landed +- see below. ## Two decisions the specification does not make, taken here @@ -137,6 +138,200 @@ undrained pipe either. waiting thread leaves a window in which a wave arrives and finds no queue. - **`send` is `Object#send`.** The method that puts a message on the stream is called `emit`. +## The demo, and the five decisions it took (astubbs#242, plan unit U35) + +`demo/` in this module, keeping the contract in `parallel-consumer-proxy/demo/README.md`. Two arms - +`AK core` (the `rdkafka` gem, serial) and `ruby-grpc` (this module's client library over a sidecar it +spawns). The decisions below are the ones a later session would otherwise re-litigate; the ones that +are purely Ruby-facing live in `demo/README.md`, beside the code they explain. + +- **`rdkafka`, not `ruby-kafka`, for the serial arm.** librdkafka behind FFI is what a Ruby + application actually consumes Kafka with (Karafka is built on it); `ruby-kafka` has been archived + by its authors since 2023. A comparison whose serial arm is an unmaintained gem flatters the + sidecar for a reason that has nothing to do with Parallel Consumer. It ships **precompiled for + linux-gnu**, so the demo image installs it without a C toolchain - the reason the container path + is not slow to build for a native extension. +- **The blocking sleep was checked rather than inherited, and the contract's rule has since changed + to match.** It is fine here for a reason confirmed against *this* client's own design: the + executors are **threads**, and MRI releases the GVL around `sleep`. At the time the contract named + nine languages as safe; four of them turned out not to be, and it now states a predicate instead - + *is the client thread-per-record?* - citing Ruby as the one language that was checked rather than + assumed. The lesson is not about Ruby: **a wave that inherits a per-language verdict inherits + whatever the author of the list guessed**, and only re-deriving it against the client in front of + you can tell the two apart. +- **The demo does not start a broker; `demo/run.sh` does.** The Java seed uses Testcontainers. Ruby + has no equivalent this demo would rather depend on, so the entry point starts the *compose* broker + - the same service the container path uses, one definition rather than two - and hands the address + in. Natively that broker publishes a **host listener on 29092**, which is the one thing in the + compose file the Java seed's does not have. +- **Two `PC_DEMO_` variables with no flag**, `PC_DEMO_SIDECAR_CLASSPATH` and `PC_DEMO_SIDECAR_JAVA`. + The sidecar is a JVM program and Ruby cannot build one, so the launcher and classpath are computed + by the entry point (Maven natively, baked in at image build) and handed over. They are plumbing, + deliberately not dials: a flag would invite pointing the demo at an arbitrary binary, which is the + decision `SidecarCommand` refuses to make on the library's side. +- **A native run needs two toolchains** - Ruby 3.2+ *and* a JDK - so `run.sh`'s auto-detection tests + for both and says which one is missing. Only one of the ten languages avoids this; it is worth + knowing before reading the mode line as a Ruby problem. + +### The contract's environment precedence does not survive the seed's container path + +Recorded rather than edited, since `parallel-consumer-proxy/demo/README.md` is the shared contract: +it promises that every flag has a `PC_DEMO_` variable and that **the environment beats the +defaults**. Compose forwards nothing it is not told to, and the Java seed's `docker-compose.yml` +declares only `PC_DEMO_BOOTSTRAP` and `PC_DEMO_ARGS` - so on the container path every other +`PC_DEMO_` variable is silently dropped, and that is the path a reader without the language's +toolchain always takes. This module's compose file forwards all of them explicitly and reads a blank +one as "not supplied". **Either the seed should do the same or the contract should say the +environment is a native-path feature**; an owner decision, and the same gap will be in whichever of +the ten demos transcribed the seed's compose file literally. + +**SETTLED, the first way, in 9d3ee9390.** The seed's compose file now forwards every `PC_DEMO_` +variable explicitly, as do the other nine. The prediction held exactly: the gap was in **seven** of +the ten demos, all of them transcribed from the seed's compose file before it was fixed, and it was +`bin/ci-demo-conformance.sh` that found them on its first run rather than any of the seven agents. +Kept here rather than deleted because the shape is the durable lesson - a promise made by the +contract and kept only by the *native* entry point is invisible from the path a reader without the +toolchain always takes, and no per-language review would have caught it, because every one of the +seven was individually consistent with the seed it copied. + +### `blocker-executor-count-formula.md` does not block Ruby + +The identity executor-count function means `--concurrency 100` becomes 100 executors. In Python that +is 100 worker *processes*, which is why it blocks that demo; here they are threads, and the demo +observed the proxy granting exactly the number asked for (`--concurrency 4` -> 4 executor threads, +ceiling 4). Ruby therefore needs no cap invented in a demo. It does mean **the demo prints the +granted executor count**, which is the one place a reader can see the formula's effect. + +## The reader-experience pass, and the one thing it breaks outside this module (astubbs#242) + +The project owner ran **this** demo and was unimpressed by three things, all of which are now +clauses in `parallel-consumer-proxy/demo/README.md` under "The output a reader actually sees". This +module was the one they watched, so what follows is the fix rather than a transcription of someone +else's. + +- **The first line named a dial, not the product.** It was `ruby-grpc: the proxy granted 100 + executor threads, ceiling 100`, which does not contain the words Parallel Consumer. The banner is + now `Demo::BANNER` in `demo/demo.rb`, printed by `Demo.run` before anything else. **It is in the + Ruby program and not in `demo/run.sh`**, because `docker compose up` in `demo/` is a documented + entry point with no script in front of it - a banner in the wrapper would be missing from the way + a reader with only Docker starts the demo. The consequence, stated because it is a real gap: on + the **native** path `run.sh` still speaks first, with its mode line and "Building the sidecar...". + The banner is the first thing the *demo* prints, not the first thing on the terminal. +- **The arms named a category, not a client.** `AK core` -> `AK core (rdkafka)` and `ruby-grpc` -> + `ruby-grpc (this client)`. The labels are no longer reused as identifiers: `AK_CORE_GROUP` and + `SIDECAR_GROUP` carry the consumer-group names, because a group id built from a label with a space + and brackets in it is a thing to explain rather than to read. +- **Two new columns, `records` and `keys`**, appended after `vs AK core` rather than inserted before + it - see the harness note below for why that position was not free. The AK core arm collects keys + from `Rdkafka::Consumer::Message#key`; the sidecar arm collects them inside `Completion`, under + the mutex that already guards the counter, since N executor threads reach it at once. + +Observed at `--records 20 --concurrency 4 --partitions 2 --replay-factor 2`: `20 / 20` on both +small-replay arms and `40 / 40` on the big-replay sidecar arm, which is what the seeding predicts - +`key-#{index % 1000}` over 40 records is 40 distinct keys, and the two arms agreeing on both figures +is the point of having them. + +### The contract's new labels are invisible to `bin/ci-demo-conformance.sh`, and it will not go red + +**Reported, not fixed - `bin/**` is outside every language agent's file scope.** That script reduces +each demo's stdout to a skeleton and requires the skeletons to match. Two of its patterns cannot see +an arm row any more, and neither failure announces itself: + +- its row matcher accepts an arm name of `[A-Za-z][A-Za-z0-9 _-]*` and every language's arm name now + contains **parentheses**, so no `ROW` line is emitted at all; +- `normalise_arms` rewrites `ROW -grpc` to `ROW SIDECAR`, and the sidecar arm is now + `ruby-grpc (this client)`. + +The dangerous half is that the drift check still **passes**: all ten languages lose their `ROW` +lines identically, so the skeletons still match while no longer comparing the one thing the rows +were there for. The header assertion survives only because that pattern is unanchored - +`arm elapsed msg/s vs AK core` is still a prefix of the new header, which is the reason the two new +columns were appended rather than inserted, and the reason `vs AK core` kept its wording after the +arm it names was relabelled. + +Whoever owns `bin/` needs to widen the row pattern to allow `(`, `)` and `.`, re-point +`normalise_arms` at the new sidecar label, and ideally assert the deterministic `records`/`keys` +pair the contract added them for - that is the assertion those columns exist to make possible, and +nothing uses it yet. + +### `KAFKA_LOG4J_ROOT_LOGLEVEL: WARN` DOES NOT QUIETEN THIS BROKER - measured + +**Reported, not fixed: broker log levels were explicitly placed out of scope for this pass.** It is +the first thing to fix before anyone runs a demo to be watched again, and the fix is one line in +each of the eleven compose files, so it wants doing once across the fan-out rather than eleven +times. + +The measurement, from the verification run below (`--records 20 --concurrency 4 --partitions 2 +--replay-factor 2`, `docker compose up`, the setting present in the compose file and the broker +container recreated from it): **1520 lines of combined output, of which 1465 were the broker, 1042 +of those at INFO, and 41 were the demo's own.** The thing a reader came to watch is about one line +in thirty-seven. + +The cause is in the image rather than in the compose file. The log4j template under +`/etc/confluent/docker/` in `confluentinc/cp-kafka:7.9.0` renders `KAFKA_LOG4J_ROOT_LOGLEVEL` into +`log4j.rootLogger` **and then unconditionally writes a block of explicit per-logger levels over the +top of it**: + +``` +log4j.rootLogger={{ env["KAFKA_LOG4J_ROOT_LOGLEVEL"] | default('INFO') }}, stdout +... +{% set loggers = { 'kafka': 'INFO', 'kafka.controller': 'TRACE', + 'state.change.logger': 'TRACE', ... } -%} +``` + +A named logger beats the root logger in log4j, so `kafka.*` stays at INFO and the controller and +state-change loggers stay at **TRACE** whatever the root is set to. The root level only ever +governed the loggers *not* in that map, which is why the setting looks applied, changes the +rendered `log4j.properties`, and quietens almost nothing. + +The variable that reaches those loggers is `KAFKA_LOG4J_LOGGERS`, which the same template merges +over the defaults (`parse_log4j_loggers`) - something of the shape +`KAFKA_LOG4J_LOGGERS: "kafka=WARN,kafka.controller=WARN,state.change.logger=WARN,org.apache.kafka=WARN"` +beside the existing root setting. Whoever applies it should **count the broker's INFO lines before +and after** rather than reading the rendered properties file: the current setting demonstrates that +a config change can be visible in the file and absent from the output. + +### What was checked for noise in the demo's own output, and what was found + +Separately from the broker, everything the demo process itself emits was read. **It is clean**: the +sidecar's stderr is inherited by the demo process and the JVM printed nothing, `rdkafka` printed +nothing, `grpc` printed nothing. The `puts` calls in `demo/lib/` are the fingerprint, the topic and +seeding lines, the two per-arm markers and the tables - nothing removable. One thing was wrong and +is fixed: + +- **`(AK core is serial and would take 0s+)`.** The big-replay heading priced the arm it drops with + integer-divided seconds, so at any volume under 500 record-milliseconds it told the reader the arm + had been dropped to save no time at all - and the contract's own conformance volume is exactly + such a volume. `Comparison#serial_cost` now prints the figure only when there is one worth + quoting; "AK core is serial" is true at every volume. Verified both ways: at `--delay-ms 2` the + clause is absent, and the control at `--delay-ms 100` - one term changed - prints "and would take + 4s+". + +Two things a watcher might still notice, neither of them the demo's doing and so neither changed: + +- **`docker compose up --build` shows the whole image build first**, including a four-minute Maven + step and a screenful of `cp: warning: behavior of -n is non-portable`. That belongs to the + Dockerfile's classpath copy, and it is only ever seen on a cold build. +- **The AK core arm prints its "starting" line and then nothing for as long as it runs** - about ten + seconds at the demo's own defaults. That is silence rather than noise, and no clause of the + contract asks for progress inside an arm; if one ever does, that arm is where it goes. + +### Eleven demos, one host port: 29092 is not per-language + +Every language's `docker-compose.yml` publishes the broker's host listener on **29092**, so two +demos running at once on one machine cannot both start. This pass hit it as +`Bind for 0.0.0.0:29092 failed: port is already allocated`, with the **Rust** demo's broker holding +it. The verification runs were completed by remapping this module's published port temporarily; the +compose file is unchanged. + +It costs a reader nothing - they run one demo - so it is not obviously a defect to fix. It costs a +**fan-out** a great deal, and `bin/ci-demo-conformance.sh` runs the languages sequentially for what +may be this reason. Worth deciding once, for all eleven, rather than discovering per language. + +The same run also exhausted Docker's disk (`no space left on device` while exporting a 1.38GB +image), with 33GB reclaimable in dangling images from the parallel builds. `docker image prune -f` +recovered 15GB without touching a tagged image, a running container or a build cache. + ## Local verification, and what it is pinned to From `parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/`: @@ -173,6 +368,50 @@ as the ledger states it: **`git commit -- `, never `git add` then commit. ## Owed to whoever picks this module up +### The demo's native path has never been executed + +**The container path was run end to end and the native path was not.** What ran, on a machine shared +with nine other demo builds: `run.sh --docker` at `--records 20 --delay-ms 1 --concurrency 4 +--partitions 2` with `--replay-factor 1` and again with `--replay-factor 2` (so the big-replay branch +and its footnote ran too); the **no-argument** invocation with the same values supplied through +`PC_DEMO_*`, which is what proved compose's environment forwarding; `--help` and an unknown flag +through both `run.sh` and the image's own entrypoint (exit 0 and exit 2). Both arms completed every +time and the process exited 0. **No throughput figure from those runs means anything** - twenty +records at 1ms is group-join cost, and the machine was carrying nine other agents. + +The reader-experience pass added to that, and still not to the native path: the container path at +`--records 20 --concurrency 4 --partitions 2 --replay-factor 2` (the conformance harness's own +volume), through `docker compose up` on the image `run.sh --docker` built, exit 0 with both tables +and both new columns; then a second run at `--delay-ms 100` as the control for the big-replay +heading. RuboCop 1.89.0 was run in a `ruby:3.4-bookworm` container over the whole module and is +clean at 25 files. **No throughput figure from any of it means anything** - twenty records is +group-join cost, and ten other language agents were building on the same machine throughout. + +The native path is unrun because **this machine ships Ruby 2.6.10 and has no other**, and the +library's floor is 3.2. Three things in it are therefore unexercised: the Maven build that computes +`PC_DEMO_SIDECAR_CLASSPATH`, the compose broker's **HOST listener on 29092**, and `bundle install` +building `rdkafka` from source on a platform with no precompiled gem (macOS). The first reader with +a modern Ruby should run `demo/run.sh --native` before trusting any of it. + +Also never run: **the demo at its own defaults** (2000 records, replay factor 20), deliberately - +a full-scale measurement belongs on an unloaded machine. + +### The demo has no CI coverage, and the contract says it should + +`bin/ci-demo-test.sh` runs the Java demo through both entry points on every pull request, and the +contract is explicit that a per-language demo inherits that: "mirroring the flags and the tables is +not enough if nobody ever runs the container you shipped". That script is Java-only and outside this +wave's file scope, so **nothing runs this demo automatically**. Whoever owns `bin/` should decide +whether it grows a per-language loop or each language gets its own script - a decision that wants +making once for all ten rather than ten times. + +### The demo image is 1.38GB + +`ruby:3.4-bookworm` (1.0GB) plus a JRE plus the sidecar's jars. `-slim` would take most of it back +but removes the C toolchain that `rdkafka` needs where no precompiled gem exists, so it is a trade +rather than a saving. `docs/inflight/next-container-image-cache-and-size.md` is where that belongs +if it is worth doing across the fan-out. + - **The CI matrix row pins Ruby 3.4.4; every run behind this wave happened on 4.0.6.** The library targets 3.2 (`Thread::Queue#pop(timeout:)` is the binding floor) and nothing used is newer, but 3.4 has not actually been exercised. The workflow has one writer and no wave edits it; whoever diff --git a/docs/inflight/clients/rust.md b/docs/inflight/clients/rust.md index e8ba6cf379..19036a9e01 100644 --- a/docs/inflight/clients/rust.md +++ b/docs/inflight/clients/rust.md @@ -11,8 +11,12 @@ report, and a clean client-initiated shutdown, proven by one end-to-end test aga sidecar. The module is at `parallel-consumer-proxy-clients/parallel-consumer-proxy-client-rust/`; its maturity and testing-evidence deferrals are lifted. Later waves: leases and heartbeats, the manifest reconnect, -worker death, terminal outcomes, the shutdown drain, the demo and its container, crates.io -publishing, and the rest of the conformance suite. +worker death, terminal outcomes, the shutdown drain, crates.io publishing, and the rest of the +conformance suite. + +**The demo landed too**, at `.../parallel-consumer-proxy-client-rust/demo/` - two arms, both +replays, both entry points run. What it found is in "What the demo wave found" below; what is still +open from it is in "Open from the demo wave". ## The falsification result @@ -120,6 +124,144 @@ inventing a third shape. reported the same collision; both clients resolved it the same way.) - **The port line is scanned for, not read as the first line**, per the guide's harness carve-out. +## What the demo wave found + +The demo is a crate of its own at `.../parallel-consumer-proxy-client-rust/demo/`, beside the +client rather than inside it: it needs `rdkafka` for its AK core arm and for seeding, and the +client library must never need a Kafka client at all. A `src/bin/` target would have shared the +library's dependency list and quietly contradicted the claim the demo exists to make. + +### The blocking-sleep carve-out needs a Rust footnote, and every async language should be asked + +The contract's per-language rule exempts Python and TypeScript from a blocking sleep and names Rust +among the languages where one is fine. It is fine - and *where* it blocks decides whether the +engine's ceiling means anything, because this client's executors are tasks on an async runtime. + +Measured with one term changed and everything else identical (12 cores, 40,000 records, +`--delay-ms 2`, `--concurrency 100`, same broker, same run of the demo): + +| the user's function | msg/s | +|---|---| +| `blocking(\|r\| thread::sleep(..))` - the library's blocking-pool entry point | **10,341** | +| `async move { thread::sleep(..) }` - the same sleep, inside the executor task | **3,518** | + +2.9x, and the low figure is about the core count divided by the delay - the predicted ceiling for a +runtime whose worker threads are all asleep. The prediction was stated before the run and held. + +**This is not a Rust-only question.** C#, TypeScript and Python's asyncio all have a user function +running on a scheduler with a bounded worker set. The wave sync should decide whether the contract +says "a blocking sleep, through whatever the client library offers for blocking user code" rather +than "a blocking sleep is fine". + +### The demo cannot see the sidecar's logs at all + +The proxy module carries no logback configuration, so logback's default console appender writes to +**stdout** - and stdout is the lifecycle channel `sidecar.rs` drains and discards after the port +line. Setting `sidecar_stderr` does not bring them back; it governs the other stream. Observed, not +inferred: the sidecar prints several dozen start-up lines before its `port:` line when run by hand, +and none of them appear in any of the demo's runs. + +So a Rust application whose sidecar fails to configure sees "the session failed" and no reason, +which is exactly the case `SidecarStderr::Inherit`'s doc comment says the default exists to +prevent. Two possible fixes, and the choice is not the demo's: give the proxy a logback +configuration that logs to stderr, or have the client forward drained stdout lines somewhere the +application can reach. + +### The container needs two things the reference's does not, and both failed first + +The demo's image carries **two** toolchains, because the application is Rust and the sidecar it +spawns is a Java program. Both extras were found by running it, not by reading: + +- **`libprotobuf-dev`, not just `protobuf-compiler`.** Debian's `protobuf-compiler` ships the protoc + binary only; the well-known types the frozen schema imports (`duration.proto`, `timestamp.proto`) + live in `libprotobuf-dev`. Without it the build fails with "File not found" for those imports + while a perfectly good protoc is installed - which reads as a broken schema. **Every non-JVM + client that containerises its codegen will meet this**, and it is a third data point for the + build-time-toolchain question already open above. +- **Nothing may look for `.git` inside the image.** The root `.dockerignore` excludes `.git`, so a + walk up the tree to find the repository - which is how the demo locates its own compose file - + cannot succeed there. The first version asked for it eagerly, which would have failed the + container before its first Kafka call; only the branch that starts a broker natively needs it, so + the resolution is now lazy. Caught by reading the `.dockerignore` against the code path, not by + the container failing. Any language whose demo derives paths from the repository root has the + same trap. + +### `build.rs` picked a `protoc` it could not execute, in preference to a working one + +Fixed on this branch. `protoc_from_maven_repository()` selected the Maven-downloaded protoc on +`is_file()` alone, and Maven stores that artifact `0644` - `protobuf-maven-plugin` chmods the copy +it extracts, never the one in `~/.m2`. Because that fallback is consulted *before* `PATH`, a +machine with a perfectly good protoc installed still failed, with a `Permission denied` naming a +path under `~/.m2` - which reads as a corrupt download. The candidate filter now requires the +executable bit, so an unexecutable copy falls through to `PATH` instead of being preferred over it. + +**Where else the class could live, checked:** `parallel-consumer-proxy-client-go`'s +`scripts/generate-proto.sh` resolves the same `~/.m2/com/google/protobuf/protoc` directory the same +way - and it already `chmod +x`es what it finds, so it does not have the defect. It fixes it by +*mutating the user's local Maven repository*, which is the trade the Rust fix declines to make; the +two are worth reconciling at the wave sync, not unilaterally. No other client resolves that +directory (`grep -rn 'com/google/protobuf/protoc' parallel-consumer-proxy-clients/` finds only +those two). + +## Open from the demo wave + +- **No CI row runs this demo.** `bin/ci-demo-test.sh` runs the Java demo through both entry points + on every pull request, and the contract says a per-language demo inherits that. Wiring it is + outside this branch's ownership (it may not touch `bin/` or `.github/`), so it is open. Until it + is wired, the Rust demo's container is exactly the kind of thing the contract warns about: + shipped, and only ever run by the person who wrote it. +- **The demo crate is not in any Maven or cargo gate.** The module's pom runs + `cargo clippy --all-targets -- -D warnings` and `cargo test` in the *client* crate's directory, + which does not reach `demo/`. Both commands pass in `demo/` today, run by hand. The pom is not + this branch's to edit either. + +## The reader-experience polish, and the one thing it breaks + +The demo now keeps the contract's rewritten output rules: the banner is the first thing printed, +each arm names the client that produced its row (`AK core (rdkafka)`, `rust-grpc (this client)`), +and both tables carry `records` and `keys` beside `msg/s`. Broker log levels were already set and +were not touched. + +**`bin/ci-demo-conformance.sh` can no longer see a single arm row, and this is not Rust-specific.** +Observed, not predicted - its own `skeleton()` awk was run by hand over this demo's captured +output, and the result was `DIAL`s, two `TITLE`s, two `HEADER`s and **zero `ROW` lines**. Two +independent causes, either of which is sufficient: + +- its arm-name character class is `[A-Za-z0-9 _-]*`, which excludes the parentheses the contract + now requires in every arm label; +- its row pattern is anchored with `$` immediately after the ratio column, so any additional + column - which is exactly what `records` and `keys` are - fails the match. + +`normalise_arms` has the same problem one step later: `s/^ROW [a-z0-9]+-grpc$/ROW SIDECAR/` cannot +match `rust-grpc (this client)`. The failure is quiet rather than red: the skeleton is still +non-empty, so no language is *skipped*; the drift check simply stops comparing arm identity and +order, which is one of the two things it exists to compare. `bin/` is outside this branch's +ownership, so it is open here rather than fixed. + +**The column order was chosen to keep the one regex that still works.** `records` and `keys` go +*after* `vs AK core`, because the `HEADER` pattern is not end-anchored and therefore still matches; +putting them anywhere else would have broken the header check as well as the row check. Eleven +languages picked their own order simultaneously, so **whoever reconciles this must confirm all +eleven agree** - the drift check cannot tell them apart until its row pattern is repaired, and a +demo whose columns are in a different order from its neighbours' would pass today. + +## Two shared-machine hazards seen while verifying, neither of them the demo's fault + +Both were hit on a box running eleven language agents at once, and both are worth knowing before +someone reads a failed demo run as a defect. + +- **Ruby's and Rust's compose files publish the same host port, 29092.** Two native demos cannot run + at once, and the second reports + `Bind for 0.0.0.0:29092 failed: port is already allocated` from `docker compose up`. Python's file + already parameterises its port (`${PC_DEMO_BROKER_PORT:-19095}`), which is the shape that fixes + this; changing Rust's alone would only move the collision, so it is recorded rather than patched. +- **The Docker VM ran out of disk**, and the symptom names Kafka rather than the disk: the broker + container exits 1 having logged + `Formatting metadata directory /var/lib/kafka/data ... No space left on device`, and the demo + reports only "starting the broker failed". The C++ demo's broker was in the same state at the same + time. The host volume was at 99% with eleven image builds in flight. Nothing was pruned - that is + shared state - beyond this demo's own compose project. + ## Local gates, and the proof each one fires Run from the module directory. **`cargo clippy --all-targets -- -D warnings` is the lint gate** - diff --git a/docs/inflight/clients/scala.md b/docs/inflight/clients/scala.md index 89185e5143..2f935f0ea8 100644 --- a/docs/inflight/clients/scala.md +++ b/docs/inflight/clients/scala.md @@ -10,9 +10,10 @@ shared note. user's function, the report with the token echoed verbatim, and a clean client-initiated shutdown - proven by two end-to-end tests against the real test-mode sidecar. The module is at `parallel-consumer-proxy-clients/parallel-consumer-proxy-client-scala/`; its maturity and -testing-evidence deferrals are lifted and it has a CI row. Later waves: leases and heartbeats, the -manifest reconnect, worker death, terminal outcomes, the `Shutdown` drain, the demo and its -container, and publishing - **most of which this client inherits rather than implements.** +testing-evidence deferrals are lifted and it has a CI row. **The demo and its container landed on +`demos/scala`** - see "The demo" below. Later waves: leases and heartbeats, the manifest reconnect, +worker death, terminal outcomes, the `Shutdown` drain, and publishing - **most of which this client +inherits rather than implements.** ## The decisions this wave was left to make @@ -154,3 +155,254 @@ row to un-defer. Two facts the row's owner may want: `src/docs/README_TEMPLATE.adoc` against the *process's* working directory, so a build started from a module directory fails on a file that never moved. The row runs `scanner-cmd` with the module as its working directory, so any JVM client whose recipe shells out to Maven hits this. + +## The demo + +`parallel-consumer-proxy-clients/parallel-consumer-proxy-client-scala/demo/` - `run.sh`, `Dockerfile`, +`docker-compose.yml`, a `README.md` recording what is Scala's own, a `logback.xml`, and the sources +under `demo/src/main/scala`. It keeps the contract in `parallel-consumer-proxy/demo/README.md` +exactly: the seven flags with the same defaults, the same `PC_DEMO_*` variables with flags beating +environment beating defaults, the banner first and the effective-configuration fingerprint straight +after it, never carrying the bootstrap address, the two tables in the same order with the same +columns, and no latency anywhere. + +**Two arms - `AK core (KafkaConsumer)` and `scala-grpc (this client)` - and the four extra ones the +Java seed carries are absent on purpose.** Scala is a JVM language, so it *could* run `pc-core`, `java-direct` and the rest against +the same broker in the same process, which is exactly why the temptation had to be named and +declined: the contract's value is that a reader who has run one language's demo has run them all, and +a six-row Scala table beside a two-row Ruby table would not be that. `scala-grpc` goes through +`ParallelConsumerClient`, not the wire - nothing in the demo's sources names a protobuf message, a +channel or a token. + +### The demo is behind a Maven profile, for the reason the harness lane is + +The sidecar arm has to hand its child process a classpath carrying `parallel-consumer-proxy`. +Declared unconditionally that is a permanent reactor edge to the engine, and `bin/build.sh` opens +with `clean` - which would delete the sidecar jar every other language's conformance test spawns. So +the demo's sources are a **test source root added only by the `scala-demo` profile** +(`-Dpc.scalaDemo`, passed by `run.sh` and by the Dockerfile), and its dependencies live there too. + +**Verified with a control arm rather than asserted**, because "the edge is only in the profile" is +exactly the kind of claim that is comfortable and wrong. One term changed, everything else identical, +and the reactor listing is what flips: + +| command | reactor | `parallel-consumer-proxy` | +|---|---|---| +| `./mvnw -pl :parallel-consumer-proxy-client-scala -am validate` | 8 modules | **absent** | +| the same, plus `-Dpc.scalaDemo` | 9 modules | present, as "Language Proxy" | + +Read the reactor listing, not the build result: **that command currently fails on this machine for an +unrelated reason**, and a reader who greps for `BUILD SUCCESS` will think the check is broken. +`validate` produces no artifacts, so in a reactor whose snapshots are not installed locally +`parallel-consumer-proxy-client-java-grpc` cannot resolve `parallel-consumer-proxy-client-java-api` +and Maven has a cached Central miss for it. Confirmed to have nothing to do with Scala: the identical +failure reproduces on `./mvnw -pl :parallel-consumer-proxy-client-java-grpc -am validate`, which +never mentions this module. `-U`, or any lane that reaches `package`, does not have it. + +This is the third thing in this module arranged that way, after `scala-e2e-harness` and the +conformance runner's classpath. Any JVM client whose demo spawns a real sidecar will need the same +shape, and it is worth stating once rather than each wave rediscovering it. + +### Three findings that are not Scala's, recorded rather than fixed + +The first two apply to **every JVM demo, the Java seed included**, and neither is this branch's to +edit. + +- **With no logback configuration on the classpath, logback's fallback is root at `DEBUG`.** A + fifty-record run of this demo emitted over four thousand lines of Netty frame dumps, docker-java + HTTP headers and Kafka client configuration, with the two tables buried in the middle. Measured on + the first run of this demo, before `demo/logback.xml` existed. The Java seed's demo module carries + no logging configuration either, so it will do the same; this branch fixed only its own, by pointing + `-Dlogback.configurationFile` at a file in `demo/` rather than shipping a resource, because + `target/test-classes` is also what `scripts/conformance-runner` runs from and the demo's logging + preferences must not silently become the shared suite's. +- **Every Kafka client logs its full effective configuration at `INFO` when constructed, and + `bootstrap.servers` is in it.** The contract's rule that a demo never prints the bootstrap address + exists because own-cluster mode puts a user's real broker there - and the demo's own fingerprint + honours it, while the client's dump prints it anyway, several times a run. `org.apache.kafka` is at + `WARN` in this demo's logging configuration for that reason and not for noise. **Whoever owns the + Java seed should check the same thing.** *Settled since:* the contract now says the rule binds the + whole run rather than only the fingerprint block, and `bin/ci-demo-conformance.sh` enforces it by + grepping every line the demo prints. So this is closed as a contract question and stays open only + as something each language's logging configuration has to keep honouring. +- **`SidecarCommand` requires an absolute path to an executable, and a JVM sidecar is a jar**, so + every JVM caller writes "this JVM's `java` plus a `-cp` argument" again. That is now in three places + in this module's orbit: `OneRecordThroughTheSidecarTest`, the Java seed's `SidecarProcess`, and this + demo. It reinforces the note above that spawning belongs to the Java lifecycle unit rather than to + each client. + +### The reader-experience wave: banner, arm labels, and the two deterministic columns + +Applied from the rewritten contract, after someone watched a demo run and found it unimpressive. +Four things changed in this module, and two of them cost a decision worth recording. + +- **The banner is printed by the demo, not by `run.sh`.** It has to appear on the container path too, + where `run.sh` is not in the picture, so the application owns it - and printing it in both places + would show it twice on the native path. `println` rather than the logger: a timestamp and a logger + name in front of the rule line would put a configuration line first again, which is the exact + failure the banner exists to fix. +- **Which forced the launcher's own output to standard error.** `Mode: native - a JDK toolchain is + present` is a configuration line, and it was the first thing on standard output. So was the Maven + build's: even at `-q` it prints the copyright check's summary and one `Jabel: initialized` per + compiled module, and on the first run of this change **four `Jabel` lines and two copyright lines + sat above the banner**. Both now go to standard error. Nothing is suppressed - a failing build still + explains itself and `set -e` still stops the script - and standard output now opens with the banner + on both paths. Any JVM demo whose `run.sh` shells out to Maven has this, because the build's + chatter is not the demo's output. +- **The fingerprint moved ahead of the broker.** It used to be the first thing `runFor` did, which put + it *after* `DemoBroker.resolve` had pulled and started a Kafka image - so a reader waited the better + part of a minute before learning what settings the numbers were produced with. `main` now prints it + before resolving, which is also the contract's stated order: banner, configuration, run. +- **Arm labels name the client**, so `AK core (KafkaConsumer)` and `scala-grpc (this client)`. The + group ids are unaffected: they were always built from separate literals rather than from these + constants, which is lucky rather than designed - a label carrying spaces and parentheses into a + `group.id` would have been an unpleasant way to find that out. + +**Scala's answer to "does this language have more than one serious client" is unusual and is recorded +in the demo's README**, as the contract asks. Alpakka/Pekko Kafka, FS2 Kafka, ZIO Kafka and +`kafka-streams-scala` are what a Scala team actually uses, and every one of them **wraps +`org.apache.kafka.clients.consumer.KafkaConsumer`** rather than speaking the protocol. So unlike Go or +Ruby there is no second independent client to run as a second arm - a second row would measure a +streaming façade's own scheduling on top of the same consumer, which is a different question. + +#### The column order was chosen without coordination, and the integrator has to reconcile it + +Eleven agents applied this contract at once, in eleven worktrees, and the contract fixes *which* +columns exist without fixing where the two new ones go. This module chose: + +``` +arm | records | keys | elapsed | msg/s | vs AK core +``` + +`records` and `keys` come immediately after the arm because they are what shows the work happened, +and the contract's own argument for them is that the table should demonstrate the run rather than +assert it. **If another language put them at the end, one of the two has to move** - the contract's +"same columns, same order" is the binding clause and neither choice is more correct. + +**`bin/ci-demo-conformance.sh` cannot read the new table, and it does not fail - it goes quiet.** +That file is not this branch's to edit, and this was measured rather than reasoned about: its +`skeleton` function was run verbatim over this demo's captured output, and what came back was six +`DIAL` lines and two `TITLE` lines and **nothing else**. No `HEADER`, no `ROW`. + +That is the dangerous shape. The skeleton is not empty, so the script's own +`[ ! -s "$lang.skel" ]` guard does not skip the language; the absolute assertions all still pass; +and the drift check happily compares eleven skeletons that no longer say anything about the tables. +**A green run would mean the table columns and the arm order had stopped being checked at all** - +which is precisely the "narrowed run that reads like a full one" the script's own header says it +exists to prevent. + +Two of its `awk` patterns were written against the old shape: + +- its `HEADER` pattern matches `arm elapsed msg/s vs AK core` as a prefix, which no longer describes + the header row under this column order. Note that a language which *appended* the two columns + instead would still match it - so the header line alone cannot be relied on to reveal the drift; +- its `ROW` pattern is `^ s $` with the name constrained to + `[A-Za-z][A-Za-z0-9 _-]*`. **The arm labels alone break it** - `AK core (KafkaConsumer)` contains + parentheses - and it is anchored at the end, so trailing columns break it too. Its + `normalise_arms` sed, `^ROW [a-z0-9]+-grpc$`, does not match `scala-grpc (this client)` either. + +None of that is fixable from a language branch without eleven conflicting edits to one shared file. +Whoever integrates the fan-out should update those three patterns once, against whatever column order +is settled on, and then re-run the drift check - which is the point at which any disagreement between +the eleven becomes visible rather than theoretical. **Worth adding while they are in there: a +`ROW`-count or `HEADER`-present assertion**, so that a skeleton which has quietly stopped recognising +the tables fails instead of comparing clean. + +#### One blemish observed and deliberately not fixed here + +The big replay's heading computes the serial arm's cost as `total * delayMs / 1000` in integer +seconds, so at the conformance harness's own volume it reads **"AK core is serial and would take +0s+"** - which is both untrue and the sort of thing that makes a demo look unfinished. It is +pre-existing, it is in the shared table title rather than in anything this wave touched, and the +wording is almost certainly identical in the other ten languages. Changing it unilaterally would +introduce exactly the cross-language prose drift the contract exists to prevent, so it is reported +rather than fixed. + +### What was run, and what was not + +Run natively on macOS, under heavy concurrent load from the parallel client fan-out - load average +above 100 on twelve cores at the time of the second run - so **no throughput figure from these runs +means anything and none is recorded here**. What they prove is that the machinery works: + +- `demo/run.sh --records 20 --replay-factor 2 --partitions 2 --concurrency 8` - both arms completed, + both tables rendered, exit 0, 59 lines of output end to end. The `--replay-factor 2` was chosen over + 1 deliberately: at 1 the big replay is skipped, and the second table's rendering path would never + have executed at all. +- `demo/run.sh` with **no arguments at all**, configured entirely through `PC_DEMO_RECORDS`, + `PC_DEMO_REPLAY_FACTOR`, `PC_DEMO_PARTITIONS` and `PC_DEMO_CONCURRENCY`. This is the case that has + broken before - `bash 3.2` under `set -u` treats an empty array expansion as an unbound variable - + and it proves the environment layer at the same time. +- `--help` and a misspelled flag both reach the usage text; the misspelled flag exits 2. + +Re-run twice after the reader-experience wave, as +`demo/run.sh --records 20 --concurrency 4 --partitions 2 --replay-factor 2` - the flags +`bin/ci-demo-conformance.sh` uses. Exit 0 both times, both tables rendered, and: + +- standard output opens with the banner, with nothing above it; +- both arms report **20 records over 20 keys** in the small replay and the sidecar arm **40 over 40** + in the big one, identical across the two runs while every timing moved - which is the determinism + claim actually observed rather than asserted; +- the two absolute assertions the conformance harness makes hold on the captured output: no + `bootstrap.servers` anywhere in the demo's own lines, and no latency, percentile, `p95` or `p99`. + +**The container path has now been run, and it passes.** This was the gap two earlier sessions left +open. `demo/run.sh --docker --records 20 --concurrency 4 --partitions 2 --replay-factor 2`, image +built from scratch, **exit 0**. What it proves, beyond "it runs": + +- the banner is the first line the demo container prints, with nothing above it - the container + entry point is the `Dockerfile`'s, so this is a genuinely separate proof from the native one; +- **the sidecar spawns as a child process inside the demo's own container** (`Sidecar listening on + loopback port 43325`, then again on a different port for the big replay). That is the rule the + compose file declines to break by adding a sidecar service, and until now nothing had shown it + actually works in a container; +- the broker was reached as a compose sibling, with no host Docker socket anywhere; +- **the same deterministic figures as the two native runs**: 20 records over 20 keys in both arms of + the small replay, 40 over 40 in the big one. Three runs, two entry points, one machine state that + changed wildly in between - and those four numbers did not move while every timing did. That is + the determinism claim tested rather than restated; +- the two absolute assertions hold on the demo's own 50 lines: no `bootstrap.servers`, no latency. + +The image build took **310s of Maven inside the container** plus about two minutes to export and +load a 288MB layer, on a box carrying eleven agents. That is why earlier sessions skipped it; it is +worth knowing the cost is real rather than a one-off. + +#### The broker is NOT quiet, and the compose fix that was supposed to do it cannot + +Measured on this run: the broker emitted **584 `INFO` lines and 1 `WARN`** while +`KAFKA_LOG4J_ROOT_LOGLEVEL: WARN` and `KAFKA_TOOLS_LOG4J_LOGLEVEL: WARN` were both set and both +confirmed present in the container's environment. **This is not a Scala problem** - the same two +lines were added to all eleven compose files by the wave that wrote the contract's "the broker is +quiet" clause, so all eleven are equally affected, and nothing here is this branch's to change. + +Root-caused rather than guessed at, by reading the image's own template +(`/etc/confluent/docker/log4j.properties.template` in `confluentinc/cp-kafka:7.9.0`): + +``` +log4j.rootLogger={{ env["KAFKA_LOG4J_ROOT_LOGLEVEL"] | default('INFO') }}, stdout +{% set loggers = { 'kafka': 'INFO', 'kafka.controller': 'TRACE', + 'kafka.log.LogCleaner': 'INFO', 'state.change.logger': 'TRACE', ... } -%} +{% if env['KAFKA_LOG4J_LOGGERS'] %} ... merge ... {% endif %} +``` + +`KAFKA_LOG4J_ROOT_LOGLEVEL` **worked** - the generated file really does say `log4j.rootLogger=WARN`. +It is simply outranked: the template then writes a fixed per-logger map, and a named logger beats the +root in log4j. So `kafka` stays at `INFO`, and `kafka.controller` and `state.change.logger` stay at +**`TRACE`** - which are, precisely, the "controller elections and log-segment chatter" the contract +names as the noise it was trying to remove. The one variable that can lower them is +`KAFKA_LOG4J_LOGGERS`, which merges into that map: + +```yaml +KAFKA_LOG4J_LOGGERS: "kafka=WARN,kafka.controller=WARN,state.change.logger=WARN,kafka.log.LogCleaner=WARN" +``` + +Whoever owns the compose files should make that change in all eleven at once and re-measure, rather +than eleven branches each editing the file they were told not to. **The clause is currently a no-op +on the noisiest categories**, and nothing would have caught it, because no check counts broker lines +and the native path never sees them at all - Testcontainers does not attach the broker's stdout to +the demo's. + +**It is still not wired into `bin/ci-demo-test.sh`**, which runs the Java demo through both entry +points on every pull request. That file is shared across the fan-out and this branch does not own it; +whoever integrates the eleven demos should add the Scala row. The container path has now been proven +once, by hand, on one machine - which is not the same as being *kept* working, and that is what the +row is for. diff --git a/docs/inflight/clients/swift.md b/docs/inflight/clients/swift.md index b6946dc74f..2a6daf5625 100644 --- a/docs/inflight/clients/swift.md +++ b/docs/inflight/clients/swift.md @@ -124,3 +124,138 @@ being rediscovered. Full detail in [`cpp.md`](cpp.md). - **A missing Docker is exit 2, not a pass** (`bin/build-client.sh`). Through Maven that surfaces as `Exit value: 2` in the exec failure message, while Maven's own exit stays 1 - so read the message, not just the code, when a CI row goes red. + +## The demo (astubbs#242, plan unit U35, R72) + +**Status: written, in `parallel-consumer-proxy-clients/parallel-consumer-proxy-client-swift/demo/`.** +Two arms - `AK core (swift-kafka-client)`, and `swift-grpc (this client)` over the client library in +this module - plus the contract's flags, environment variables, fingerprint and two tables. It is +its own SwiftPM package with a `.package(path: "..")` dependency on the client, its own `Dockerfile` +and its own `docker-compose.yml`; there is **no Maven module** and it is not in the reactor. + +### The reader-experience pass (the contract's "output a reader actually sees") + +Applied here after the contract gained that section: the demo opens with the standard banner +(printed **before** the option parse, so a run that dies for want of a sidecar has still said what +it is), both arms name the client that produced their row, and both tables carry `records` and +`keys` beside `elapsed`, `msg/s` and the ratio. Broker log levels were already `WARN` in this +module's `docker-compose.yml` and were not touched. + +- **Unique keys are counted from what the arm processed, never asked of the broker**, and in Swift + there was no choice: `swift-kafka-client` has no admin client and no public metadata API. It is + also the better figure - a broker-side answer describes the topic, not the run. Each arm counts + the keys of the records that reached its own user function, decoded from bytes the same lossy way + on both arms, so a non-UTF-8 key counts as one key on both rather than vanishing from one. +- **The AK core arm now needs `NIOCore` declared.** `KafkaConsumerMessage.key` is a NIO + `ByteBuffer`, so `demo/Package.swift` gained `swift-nio` at swift-kafka-client's own lower bound - + a name added to the graph, not a version added to the solve. +- **Column ORDER was a guess, and eleven agents guessed independently.** This demo prints + `arm | records | keys | elapsed | msg/s | vs AK core`, reading the contract's own sentence - + "every arm reports what it did, not just how fast" - as putting the work before the speed. The + contract does not fix the order, and the drift check will find out; if the other ten went the + other way, this is the file to change. +- **`bin/ci-demo-conformance.sh` cannot see these rows any more, and it will not go red saying so.** + Its `skeleton()` awk matches an arm row as ` s ` with the name in + `[A-Za-z0-9 _-]*`, so it matches neither the new column order nor the parenthesised client name - + and its `normalise_arms` reduces `ROW -grpc` to `ROW SIDECAR`, which `swift-grpc (this + client)` no longer matches either. The effect is that **no `ROW` lines reach the skeleton at all**, + so arm identity and order stop being compared while the run still passes. The `HEADER` line goes + the same way in this demo's order (`arm` is no longer followed by `elapsed`). Both regexes need + widening once the eleven agree on an order; `bin/` was not this task's to edit. + +### What was actually run, and what it does NOT establish + +Both arms were run end to end in the container, twice, on a machine shared with nine other agent +sessions: + +- `--records 30 --replay-factor 1 --partitions 3` - fingerprint, both arms, small table, exit 0. +- `--records 20 --replay-factor 3 --partitions 2` - both replays, both tables and the across-replays + footnote, exit 0. +- The image's own entry point with **no arguments** (parses the defaults and fails on the absent + broker, exit 1), with `--help` (exit 0) and with a misspelled flag (exit 2). +- `run.sh` with no arguments, with flags, with an inherited `PC_DEMO_PARTITIONS`, `--help`, + `--native`, an unknown flag and a flag missing its value - against a stubbed `docker`. +- No line the demo printed contained the bootstrap address; the only occurrences in the whole + transcript are the broker's own logs. + +**These runs prove the machinery, not any number.** Volumes that small are dominated by start-up: +the sidecar arm came out at 0.9x of AK core over 20 records and 2.7x over 60, which is the same +demo saying two different things about the same code. Nothing here should be quoted as a +measurement, and a default-scale run on an unloaded machine has not been done. + +**One observation that a real measurement will have to account for**, seen in both runs and not yet +explained: the AK core arm's wall clock is far larger than its simulated work (30 records at 2 ms +took 1.79 s). `swift-kafka-client` feeds its message `AsyncSequence` from a poll loop whose +`pollInterval` defaults to 100 ms, so at these volumes the arm is timing that loop rather than the +work. Whether it amortises at contract scale is unknown - it was not tested, and the demo does not +change the default. + +### What is owed + +- **`bin/ci-demo-test.sh` runs the Java demo only.** The contract says a per-language demo inherits + "both entry points are tested", and this one is tested by neither. It is a shared script, so + wiring Swift in is a separate change to a file this wave did not own. +- **The demo package has no committed `Package.resolved`.** Generating one needs a toolchain that + does not exist on this box outside the image. The image copies the resolved file it produced to + `/app/Package.resolved` so a reader can at least see what was built; committing one properly means + extracting it from a build, which is a `bin/build-client.sh`-shaped job rather than a demo one. +- **The blocking-sleep divergence is reasoned, not measured.** See below. +- **The big replay's title says "would take 0s+" at tiny volumes.** It is the Java seed's own + expression, `total * delayMs / 1000` in integer arithmetic, mirrored faithfully; at the contract's + defaults it reads correctly. Left as-is rather than diverging for a cosmetic case, but it is the + seed's wart rather than Swift's. + +### The three divergences + +- **`Task.sleep`, not a blocking sleep - and this is no longer a divergence, because the contract + adopted the finding.** Its rule used to name languages and listed Swift as safe for a blocking + sleep; it now asks whether the CLIENT is thread-per-record, and names Swift's cooperative pool + among those that are not. Rust measured the same class of error at 10,341 msg/s through its + blocking adapter against 3,518 with a raw thread sleep. The mechanism here is unchanged: `poll` + starts `executorCount` Swift concurrency **tasks** on the + cooperative pool, whose width is the core count. A blocking sleep in the user function occupies a + pool thread for its whole duration, so an arm asking for 100 in-flight records could only ever + have core-count many running - the table would report the pool's ceiling while appearing to report + the engine's. This is the same mechanism this module already recorded for the conformance + runner's ceiling barrier ("a waiter that blocked its thread would take one of those threads out of + the pool"), so it is an established property of this client rather than a new guess. + **It has not been measured here** - no blocking-sleep control arm has been run, and no figure for + the gap should be quoted until one has. +- **`--partitions` reaches the broker's `num.partitions`, not a `CreateTopics` call.** + `swift-kafka-client` has no admin client and no public metadata API, so the demo cannot create a + topic with a partition count, cannot describe an existing one, and cannot do what the Java seed + does when a topic already exists with the wrong count. The topic is auto-created by the first + produce and `docker-compose.yml` carries the count. Consequence: with `--bootstrap` pointing at + someone else's cluster the flag has no effect - pass `--topic` for a topic you made yourself. +- **No native mode.** There is no Swift toolchain on a developer box here, so `run.sh --native` + refuses with the reason, and the demo can never start its own broker (a demo container gets no + host Docker socket). The compose sibling is the only broker it meets. + +### Things the next demo wave should not rediscover + +- **The demo has to be its OWN SwiftPM package.** Its AK core arm needs `swift-kafka-client`, which + vendors librdkafka and drags in zstd, OpenSSL and SwiftNIO. Putting that in the client library's + manifest would add all of it to every consumer's graph, and to the committed `Package.resolved` + the clients workflow keys its cache on, to serve a demo none of them build. A path dependency is + also what makes it *legal*: the library's targets use `unsafeFlags`, which bars a package from + being depended on by version - local paths are exempt. +- **`swift-kafka-client` publishes only `1.0.0-alpha.N` tags, and `main` is unusable here.** `main` + declares `swift-tools-version:6.2.3`, which the pinned Swift 6.1 toolchain cannot parse, so the + failure arrives at manifest load and reads as a resolution error rather than a version conflict. + The demo pins `exact: "1.0.0-alpha.9"` - the newest tag whose tools version (5.9) the image can + read. A base-image bump is what unblocks a move, exactly as it is for swift-log 1.10.1. +- **A Dockerfile cannot inherit a stage from another Dockerfile**, and `docker compose up` - the + path a reader with only Docker takes - cannot build one image before another. So the demo image + repeats the client module's `toolchain`, `plugins` and `codegen` stages **byte-identically**, which + makes BuildKit's cache key match and pays for the protoc-plugin compile once per machine instead + of once per Dockerfile. Only the codegen stage's schema COPY differs, because this build's context + is already the repository. **Edit those stages in one file and you must edit the other.** +- **`maven:` as a base is usable after all, if you clear `MAVEN_CONFIG`.** The Java demo's Dockerfile + rejects that image because it sets `MAVEN_CONFIG=/root/.m2` and the wrapper appends it to its own + command line, so `./mvnw package` dies with `Unknown lifecycle phase "/root/.m2"`. `ENV MAVEN_CONFIG=` + removes the conflict and keeps the wrapper, which saves pulling a second JDK base - worth knowing + on a machine where ten agents share a disk. +- **The sidecar's jars are COPIED OUT rather than the Maven repository baked in.** The Java demo + bakes `~/.m2` into its image because it computes a classpath pointing there; naming an output + directory instead (`dependency:build-classpath`, then copying each entry) lets the download cache + stay a BuildKit cache mount, which is not part of any image. diff --git a/docs/inflight/clients/typescript.md b/docs/inflight/clients/typescript.md index 7b03e398e2..1425096c19 100644 --- a/docs/inflight/clients/typescript.md +++ b/docs/inflight/clients/typescript.md @@ -11,8 +11,12 @@ report with the token echoed verbatim, and a clean client-initiated shutdown, pr end-to-end test against the real test-mode sidecar. The module is at `parallel-consumer-proxy-clients/parallel-consumer-proxy-client-typescript/`; its maturity and testing-evidence deferrals are lifted, so its CI row now runs. Later waves: leases and heartbeats, -the manifest reconnect, worker death, terminal outcomes, the shutdown drain, the demo and its -container, npm publishing, and the rest of the conformance suite. +the manifest reconnect, worker death, terminal outcomes, the shutdown drain, npm publishing, and +the rest of the conformance suite. + +**The demo landed in the demo wave** and lives at +`parallel-consumer-proxy-clients/parallel-consumer-proxy-client-typescript/demo/`. What it decided, +what it had to diverge on, and what it is still missing is the "The demo" section below. ## Decisions this wave took that a sync should confirm or overturn @@ -148,3 +152,244 @@ wave reported. Actual wave-one effort, for whatever an uncalibrated point is wor session, roughly two hours wall clock, ~1,100 lines of hand-written TypeScript across nine source files, two test files and a generation script, on top of ~3,900 lines of generated stubs. The end-to-end test passed on its first run against the real wire. + +## The demo + +`parallel-consumer-proxy-clients/parallel-consumer-proxy-client-typescript/demo/` - `run.sh`, +`Dockerfile`, `docker-compose.yml`, its own `README.md`, and five TypeScript sources under `src/`. +It keeps the contract in `parallel-consumer-proxy/demo/README.md`: the same seven flags with the +same defaults, one `PC_DEMO_` variable per flag with flags beating environment beating defaults, +the effective-configuration fingerprint printed before the run and **never** the bootstrap address, +the two tables in the same order, and no latency anywhere. + +Two arms, which is the whole contract outside Java: **`AK core`** (kafkajs, one record at a time) +and **`typescript-grpc`** (this module's client library, which spawns the sidecar as a child +process). The big replay runs the sidecar arm only. + +### The sanctioned divergence: the work is an awaited timer, and what that implies + +The plan singles TypeScript out because a blocking sleep on a single event loop stops the +transport, the executors and the timers at once - the "parallel" arm would run exactly as serially +as the serial one. The work is therefore +`await new Promise(resolve => setTimeout(resolve, delayMs))`. + +**What that implies about the sidecar arm, stated rather than left to be inferred:** the +parallelism it shows is *promise concurrency on one event loop*. `Configured.executor_count` +becomes that many concurrent `await`s, not that many threads, which is the wave-one concurrency +decision recorded above being exercised end to end for the first time. The demo therefore +demonstrates exactly the case this client is good at (overlapping waits) and says in its README +that a **synchronously** CPU-bound processor would block the loop and collapse the arm to serial. +Nothing in the demo tests that failure mode - it remains untested here as it was in wave one. + +### Divergences that were NOT sanctioned, and why they were taken anyway + +- **`run.sh` starts the broker; the demo program refuses to run without an address.** The Java demo + starts one with Testcontainers when `--bootstrap` is absent. Two reasons: the containerised path + is *always* "an address was supplied" anyway, because a demo container is never granted the host + Docker socket - so broker-starting is a property of the launcher, not the demo; and the + alternative was a **47 MB** `@testcontainers/kafka` dependency (measured on a clean install) in a + package tree whose ordinary `npm ci` sits on the CI matrix's critical path, bought for one code + path the container never takes. The flags, precedence, fingerprint and tables are unchanged, so + this is invisible from outside - but it means `--bootstrap`'s documented "omit to start one" is + the *script's* promise here, not the program's. +- **The demo is a separate npm package** (`demo/package.json`) that reaches the library through + `file:..`. npm resolves that to a symlink (measured: `added 1 package in 294ms`, and it does not + pack or reinstall the parent's dependencies), so the demo loads the library's built `dist/` - the + artifact a user would install - and **kafkajs is the demo's dependency and never the library's**. + A client library that pulled in a Kafka client would contradict the arm it exists to demonstrate. +- **`eslint.config.mjs` ignores `demo/**`, after trying not to.** Adding the demo's tsconfig to the + typed-lint projects is the version that *should* be right - `no-floating-promises` is exactly what + an arm racing a countdown against a session's end needs, and while it was wired up it immediately + caught an unused processor parameter. But the demo is a separate npm package, so the typed rules + can only resolve kafkajs when `demo/node_modules` exists, and the CI matrix row installs the + library's dependencies only. **Measured both ways:** with the demo's project listed and its + node_modules present, `npx --no-install eslint .` exits 0; with it absent, 62 `no-unsafe-*` errors + and exit 1. Green locally and red in CI is worse than not running, and a config that includes the + project only when the directory exists is a gate that silently disappears - so the directory is + ignored and the reasoning is written into the config. **If an integrator wants the demo linted, + the lever is the CI row installing `demo/`'s dependencies**, not the eslint config. + +### Measured, and what these numbers are NOT + +Ten agents were running on this machine, so the brief for this wave was reduced to proving the +thing runs rather than measuring anything. **These are not throughput figures and must not be +quoted as any**: at 60-100 records the group join dominates every arm, and the honest reading is +"both arms completed and the tables printed". + +| run | arm | records | elapsed | msg/s | +|---|---|---|---|---| +| native, `--records 100 --replay-factor 1 --partitions 4` | AK core | 100 | 1.4s | 70 | +| same | typescript-grpc | 100 | 0.7s | 140 | +| native, **no arguments**, `PC_DEMO_RECORDS=60 PC_DEMO_REPLAY_FACTOR=2 PC_DEMO_PARTITIONS=4` | AK core | 60 | 1.3s | 45 | +| same, small replay | typescript-grpc | 60 | 0.8s | 76 | +| same, big replay | typescript-grpc | 120 | 0.7s | 164 | +| native, `--records 80 --replay-factor 2 --partitions 4` | AK core | 80 | 3.7s | 21 | +| same, small replay | typescript-grpc | 80 | 1.2s | 68 | +| same, big replay | typescript-grpc | 160 | 1.1s | 148 | +| **container**, `--docker --records 60 --replay-factor 2 --partitions 4` | AK core | 60 | 4.4s | 13 | +| same, small replay | typescript-grpc | 60 | 1.5s | 40 | +| same, big replay | typescript-grpc | 120 | 3.1s | 38 | + +Every one exited 0. The spread between two native runs at the same settings (AK core at 70 then 21 +msg/s) is the load on the box, and it is the clearest possible evidence that none of these are +measurements. A real one wants an unloaded machine and the defaults. + +### Three things the runs found, which no amount of reading would have + +- **A KRaft broker started with `KAFKA_LISTENERS=...://0.0.0.0:...` exits 1 during preflight**, with + `advertised.listeners cannot use the nonroutable meta-address 0.0.0.0`, because the CONTROLLER + listener has no entry in `advertised.listeners` and is therefore taken from `listeners`. The + compose file next door already binds to its service name for this reason and says so; the + standalone `docker run` in `run.sh` had to learn it. Every listener now binds to + `$BROKER_HOSTNAME`, and Docker still forwards the published port to a listener bound to the + container's own address. +- **kafkajs 2.2.4 prints `TimeoutNegativeWarning` on Node 25.** Cosmetic - the run is unaffected - + but kafkajs's last release is 2022 and this is the shape of problem an unmaintained client will + keep producing. CI pins Node 22, where it was not seen. If it becomes more than noise, the + replacement candidate is `@confluentinc/kafka-javascript`, which costs native prebuilds in the + image. kafkajs also logs one `The group coordinator is not available` ERROR per run while + `__consumer_offsets` is being created; it retries and succeeds, and it is left visible rather + than filtered, because suppressing a broker error class to tidy a demo's output is how a real one + gets hidden. +- **The sidecar the product SHIPS has no SLF4J provider**, and the demo is how that surfaced. + `logback-classic` is `test` scope repository-wide, so a runtime-scoped classpath for + `bz.stub.parallelconsumer.proxy.Main` prints `No SLF4J providers were found` and every `log.info` + in the sidecar goes nowhere. This is a property of the proxy module, not of the demo, and fixing + it is not a demo wave's call - but the demo was about to hide it: `build-classpath` with no + `includeScope` writes *every* scope, so the native path was quietly flattered by test-scope + logback while the container was not. `run.sh` now passes `-DincludeScope=runtime`, so both entry + points run the sidecar on the classpath a user would get, and both print the warning. **For the + proxy's owners:** a shipped sidecar that cannot log is a diagnosability problem well beyond this + demo. + +### Open, and NOT done by this wave + +- **Nothing runs this demo in CI.** The contract's "both entry points are tested" section says a + per-language demo inherits `bin/ci-demo-test.sh`, and `bin/**` was not this agent's to edit. The + integrator has to wire the TypeScript entry points in, or the demo joins the ones that were + shipped and never executed again. +- **There is no `ReferenceDemoIT` equivalent.** The Java demo has a test that calls its entry method + and asserts what the arms did. Nothing here does; the evidence is the runs recorded above. +- **CHANGELOG and any cross-language index.** Not this agent's files. + +The container path is **not** on this list: `demo/run.sh --docker` built the image and ran both +arms and both replays to exit 0, on a repository-context two-stage build (a JDK stage that +materialises a flat `/sidecar/lib`, then a Node image with a Temurin JRE copied in beside it). The +broker is a compose sibling and the demo container is given an address, never the Docker socket; +the sidecar is spawned as a child process inside the demo container and is not a compose service. + +### Things in the shared contract that look wrong from here (recorded, not edited) + +1. **`--bootstrap ADDR - an existing broker; omit to start one` is a promise only a language with a + Testcontainers binding can keep in the program itself.** Java, Go, .NET and a few others have + one; TypeScript's costs 47 MB and Rust's and C++'s are worse. The contract would be more honest + phrased as "the entry point starts one", leaving where that happens to the language. +2. **The big replay's heading interpolates `total * delayMs / 1000` seconds**, which prints + `would take 0s+` at any small volume. Faithfully mirrored, and it reads as a bug in every + language that mirrors it. A floor, or a different phrasing under a second, would fix it once. +3. **CLOSED by the polish wave below.** ~~The contract does not say what an arm is called.~~ Java's + arms are `java-grpc` and friends and its table column was 14 characters wide; `typescript-grpc` + is 15, so this demo widened the column and wondered whether it had broken something. The + rewritten contract says **identity and order only**, and `bin/ci-demo-conformance.sh`'s header + says the same outright - so width is explicitly free, and the arm labels the polish wave added + widened it again. What replaced this concern is a narrower one: the contract still does not fix + the column *order* for the two columns it just added. See "The contract gap this wave opened". + +## The reader-experience polish wave + +A later pass over `parallel-consumer-proxy/demo/README.md` added three reader-facing clauses after +someone watched a demo and found it unimpressive. All three are now kept here, and one contract gap +opened by them is recorded below rather than edited into the contract. + +### What changed + +- **The demo opens by naming the product.** `BANNER` in `demo/src/report.ts`, printed by + `main.ts` as the first thing the demo writes, before the effective configuration. It lives in the + *program* and not in `run.sh` deliberately: both entry points - `run.sh` and a bare + `docker compose up` - then open with it, and it is printed exactly once on either. `run.sh` still + prints its mode line and its build output first, because that is the *launcher* getting ready, not + the demo; a reader on the native path sees the banner after the Maven build. +- **Both arms name their client.** `AK core (kafkajs)` and `typescript-grpc (this client)`. The arm + *name* was split from its *label*: `SIDECAR_ARM_NAME` (`typescript-grpc`) still supplies the + consumer group id and the instance tag, because those are identifiers and + `typescript-grpc (this client)` is a caption. +- **Two new columns, `records` and `keys`.** `Countdown` became `Tally`, which counts and also + collects the distinct keys the arm observed. Both arms now read the record's key - the AK core + arm through `eachMessage: async ({ message }) => ...`, the sidecar arm through the + `InboundRecord` its processor is already handed - and neither deserializes a value, so neither is + charged for work the other avoids. Column order chosen here is + `arm | records | keys | elapsed | msg/s | vs AK core`. +- **Broker log levels were NOT touched**; `WARN` was already set in `docker-compose.yml` and in + `run.sh`'s standalone `docker run`. + +### The judgement the contract asked for: TypeScript's second Kafka client + +The contract asks a language with more than one serious client to say so in its own README, and to +*consider* running both as separate arms. TypeScript has kafkajs and +`@confluentinc/kafka-javascript` (a librdkafka binding, the successor to `node-rdkafka`). **One arm, +and it stays kafkajs** - the reasoning is in `demo/README.md` under "TypeScript has two serious +Kafka clients": no native toolchain on the demo's critical path, and a third arm would be a second +*serial* row that compares two Kafka clients with each other rather than pricing the sidecar hop, +while putting this demo's table out of step with the other ten. The cost is stated in that README as +reasoning and explicitly not as a measurement: kafkajs is the slower of the two, so this baseline is +conservative and the `vs AK core` ratio correspondingly generous. **Nothing here measured that.** + +### The contract gap this wave opened, recorded and not edited + +**The contract fixes the new columns' identity but not their POSITION.** "output | the two tables +above, same columns, same order" is a cross-language requirement, and the section that adds +`records` and `keys` says only that each arm "also reports" them - so eleven demos polished in +parallel can each pick a different position and every one of them will have kept the contract as +written. This module put them directly after `arm`, on the argument that the deterministic +"what it did" figures belong before the volatile "how fast" ones. **If the fan-out disagrees, this +is the file to change, and one sentence in the contract would settle it for good.** + +Related, and now answered: the earlier note here asked for a sentence saying column *width* is not +contract. The rewritten contract says identity and order only, and +`bin/ci-demo-conformance.sh`'s header says it outright, so **that item is closed** - this table is +wider again and that is allowed. + +### `bin/ci-demo-conformance.sh` needs a matching change, and it was not this agent's file + +`bin/**` was explicitly out of scope for this wave. Its `skeleton()` awk reduces a results row with + +``` +/^[[:space:]]*[A-Za-z][A-Za-z0-9 _-]*[[:space:]]+[0-9.]+s[[:space:]]+[0-9,-]+[[:space:]]+[0-9.x-]+[[:space:]]*$/ +``` + +which cannot match the new rows for **two independent reasons**: the arm label now contains +parentheses, which the name character class excludes, and `records`/`keys` now sit between the arm +and its elapsed figure. `normalise_arms()` has the same problem - it maps `ROW -grpc` to +`ROW SIDECAR`, and the row is now `typescript-grpc (this client)`. The `HEADER` pattern also stops +matching, because `elapsed` no longer follows `arm`. **Every language is changing the same way at +the same time**, so this is one edit to one file by whoever owns `bin/**`, not eleven - but until it +lands the conformance script will report "no skeleton" and *skip* rather than fail, which is the +quiet outcome, so it is written here where it can be found. + +### What was run, and what it is not + +Ten other agents were on this machine, so this wave proved the thing runs and **measured nothing**. +The msg/s figures below are not throughput and must not be quoted as any; the `records` and `keys` +columns are the point, and those are deterministic. + +| run | outcome | +|---|---| +| native, `--records 20 --concurrency 4 --partitions 2 --replay-factor 2` | exit 0. Banner, fingerprint, both tables. Small replay: both arms 20 records / 20 keys. Big replay: 40 records / 40 keys | +| container, `--docker` with the same flags | exit 0. Identical `records` and `keys` on every row - the same 20/20 and 40/40 - which is the cross-entry-point half of the determinism the contract claims | + +The container run's AK core arm took 31s for 20 records because kafkajs hit +`KafkaJSNumberOfRetriesExceeded: This is not the correct coordinator for this group` against a +just-started broker, restarted its consumer, and then finished. It exited 0 and processed all 20 +records; the *elapsed* and *msg/s* on that row are the retry, not the client. Now noted in +`demo/README.md` under the noise section, where the milder form of the same error already was. + +One operational trap worth recording: a `--docker` run killed part-way leaves +`pc-typescript-demo-broker-1` behind, and the next run fails with a container-name conflict rather +than reusing or replacing it. `docker compose -f /docker-compose.yml down` clears it. Not +fixed here - it is compose's normal behaviour and the compose file already tells a reader to run +`down` - but it will bite anyone who interrupts a run. + +### Still not done, and unchanged by this wave + +Everything in "Open, and NOT done by this wave" above still stands: nothing runs this demo in CI, +there is no `ReferenceDemoIT` equivalent, and the CHANGELOG is not this agent's file. diff --git a/docs/inflight/next-demo-testing-infrastructure.md b/docs/inflight/next-demo-testing-infrastructure.md new file mode 100644 index 0000000000..4f60054aad --- /dev/null +++ b/docs/inflight/next-demo-testing-infrastructure.md @@ -0,0 +1,98 @@ +# Next: testing infrastructure for the multi-language demos + +Deferred by the owner (2026-08-21) - captured as ambition, not scheduled. Ranked by how much drift +each item makes **impossible** rather than merely detectable, because eleven implementations of a +prose contract will drift and the only question is whether anything notices. + +The two coverages this builds on, and the console/ledger split it assumes, are **owned by** +[`parallel-consumer-proxy-clients/AGENTS.md`](../../parallel-consumer-proxy-clients/AGENTS.md) - +read that first; this file is the candidate work, not the rules. + +## 1. Generate the option surface instead of transcribing it + +The seven flags, their defaults, their `PC_DEMO_*` names and the help text are **pure data**, and are +currently reproduced by eleven people reading prose carefully. Put them in one machine-readable file +beside the contract and generate each language's parser - or at minimum its help text and defaults +table. + +**The precedent is already in this repo and it works**: `proxy.proto` is one schema producing eleven +languages' wire code, and nobody transcribes a protocol field by hand. The demo surface is the same +shape of problem. This is the biggest single win available, because it **retires** a class of +assertion rather than adding one. + +## 2. Golden `Configure` messages, per client + +Record the exact protocol message each client sends for identical `ClientOptions`, and diff the +eleven against each other. + +**This defect class has already bitten twice, in the same file, weeks apart**: the Java demo's +hand-written arm left `ordering` unset - so it silently ran KEY-ordered against four unordered arms - +and then left `capabilities` empty, negotiating a different session. Both produced a working arm and +a plausible number. `ConfigureParityTest` in the Java demo module is the shape, at one-client scale; +generalising it is the work. + +The protocol message is the **real interface** between eleven clients and one engine, it is invisible +to the output suite, and conformance does not compare clients to each other. + +## 3. Output assertions that imply coverage underneath + +Cheap to add, and each is a strong proxy for machinery the demo otherwise only exercises silently. + +- **Peak in-flight against the configured ceiling.** Two assertions in one number: parallelism + genuinely happened (peak > 1; the serial arm must peak at exactly 1) and the engine's ceiling held + (peak <= configured). **This would have caught a real bug**: Kotlin's blocking sleep capped + in-flight at 64 while the fingerprint printed 200, and it took arithmetic to expose. Owner approved + putting this in the table tail. +- **Second pass consumes nothing.** Re-open the same consumer group after the run and poll once; + print `second pass: 0 records`. If offsets committed correctly that is zero, and if the custom + run-length/bitset encoding lost or mis-encoded anything it is not. **The most intricate code in the + product, proved end to end by one line of output and about five lines of demo code.** +- **Distinct `(partition, offset)` equals records processed** - nothing was redelivered, which + exercises epoch fencing and the retry path. +- **Per-partition spread** - proves assignment worked and no partition starved. +- **A failure-percentage knob**, reporting that every record eventually succeeded and the maximum + attempts seen. That exercises retry scheduling and terminal resolution, an entire subsystem the + demo currently never touches. Decision 9 in the comparison demo's own record listed this as an + intended knob from the start. + +## 4. The per-record ledger, and assertions on the file rather than the screen + +The demo writes a machine-readable ledger - identifier, key, partition, offset, attempt, received and +completed timestamps - **off by default, behind a flag**, with assertions run against the file. The +rule that keeps this from spoiling the demo is in the AGENTS.md linked above. + +What a ledger makes checkable that a console cannot: + +- **overlap** - records whose processing intervals overlap prove concurrency *directly*, rather than + being inferred from a rate; +- **order** - with a known publish order and key format, per-key sequence is checked outright. Pacing + the producer deliberately makes the interleaving unambiguous in the record. + +## 5. Assert the public API shape + +The client-authoring guide describes the surface every client should expose and **nothing enforces +it**, so one language can end up with `open/poll/close` and another with `connect/consume/shutdown`. +A canonical list of names and argument order, checked per language. Cheap, and it is the thing a user +actually meets. + +## 6. One shared seed dataset, defined as data + +Each demo currently re-implements "records over a key space". Define it once - key format, value +format, order - and have every demo consume that definition, so `unique keys = 1000` becomes an +assertion about shared data rather than eleven implementations that happen to agree. + +## 7. Make the contract executable + +A checker that parses the flag table out of `parallel-consumer-proxy/demo/README.md` and asserts +every language's `--help` lists exactly those flags. Then the document cannot drift from the code in +either direction - today the doc is authoritative but unenforced. + +## 8. Per-language performance baselines - wide bands only + +Worth having to catch a change that makes something ten times slower. **Not** to police percentages. + +**The trap is already documented**: measurements here have not been shown to reproduce across +sessions - the Java seed's serial arm moved from ~345 to ~300 msg/s on one machine, the obvious cause +was refuted by a control arm, and it remains unexplained +([`next-demo-seed-followups.md`](next-demo-seed-followups.md), item 1). Until that is understood a +tight band is a flake generator, and a band wide enough to be honest is still worth having. diff --git a/docs/inflight/next-name-every-serious-client-per-language.md b/docs/inflight/next-name-every-serious-client-per-language.md new file mode 100644 index 0000000000..358532fa48 --- /dev/null +++ b/docs/inflight/next-name-every-serious-client-per-language.md @@ -0,0 +1,56 @@ +# The "is there a second serious client?" answer is missing for most languages + +Open, 2026-08-22. The contract asks every demo to **run every serious client in its ecosystem as its +own arm**, or say why not. One language does this properly, three name libraries without answering +the question, and three say nothing at all. + +## Where each language stands + +Measured by whether `/demo/README.md` carries a table naming the ecosystem's Kafka clients +*and* a reason for each one the demo does not run: + +| state | languages | +|---|---| +| **Reasoned table** - names the alternatives and why each is or is not an arm | ruby | +| **Names libraries, no reasoned table** | python (`aiokafka`, `confluent-kafka`), rust (`rdkafka`), swift (`swift-kafka-client`) | +| **Has some prose, no library named** | java, kotlin, typescript, dotnet | +| **Nothing** | scala, go, cpp | + +Go is a known case with the reasoning already written down elsewhere rather than in its README: its +agent named franz-go, confluent-kafka-go and sarama, ruled out confluent-kafka-go because cgo would +break the `CGO_ENABLED=0` build that lets the binary run in the JVM stage, and **declined sarama only +because the conformance harness then rejected extra arms**. That constraint has since been lifted - +the harness now permits and reports extra arms - so **sarama is unblocked and nothing has picked it +up.** + +## The concrete miss: Karafka + +Ruby's table is the good one, and it still has a hole. It names `rdkafka` (the serial arm) and +`ruby-kafka` (**not run** - archived by its authors in 2023, which is a sound reason). It does not +name **Karafka**, which appears exactly once in the whole module, as an aside in the `Gemfile`: "it +is what Karafka is built on". + +That is the wrong one to leave out. Karafka is Ruby's *concurrent consumption framework* - the same +category as Parallel Consumer itself - so it is the most informative comparison available in that +ecosystem, not the least. `ruby-kafka` has a reason to be skipped; Karafka has none recorded. + +## What to do, and what NOT to do + +**Do the survey per language before adding any arm.** The value is the reasoning, not the row: a +reader asking "is this fast in my language" is asking about the client they already use, and an +omission with no reason reads as an oversight - which, in Karafka's case, it was. + +**Do not assume the answer from outside the ecosystem.** Each of these needs someone to check what is +actually current and maintained: an archived project is a good reason to skip, a cgo dependency that +breaks a static build is a good reason to skip, and "I had not heard of it" is not. + +**Beware the two traps already paid for here.** A client that needs cgo can break the container +build's static-linking assumption (Go's case), and an extra arm changes the demo's runtime and disk +cost in CI - the eleven-language matrix is already the slowest thing in the build. + +## Related + +- The arm-naming pass of 2026-08-22 prefixed every Parallel Consumer arm with `pc-`, so any new arm + should follow: `pc--` for a PC arm, and the plain client name for an `AK core` arm. +- [`next-demo-testing-infrastructure.md`](next-demo-testing-infrastructure.md) - the ranked ambition + for the demo harnesses; more arms is more surface for the drift check to compare. diff --git a/docs/inflight/next-soak-tests-per-language.md b/docs/inflight/next-soak-tests-per-language.md new file mode 100644 index 0000000000..140b991f02 --- /dev/null +++ b/docs/inflight/next-soak-tests-per-language.md @@ -0,0 +1,58 @@ +# Soak tests for every client library: leaks, and performance that holds + +Candidate work, recorded 2026-08-22. **Not started.** One soak lane per client library, running long +enough to expose what a short run cannot: resources that grow, and throughput that decays. + +## Why this is not speculative + +The demo fan-out already produced a defect of exactly this shape, from a run of **twenty records**: +the Java demo prints its tables and then never exits, because non-daemon engine threads outlive the +arms that made them ([`bug-java-demo-hangs-on-exit.md`](bug-java-demo-hangs-on-exit.md)). That is a +resource leak - it only looks like a hang because the leaked resource happens to be a thread that +blocks JVM shutdown. A leak of anything the JVM does *not* wait on (a buffer, a consumer, a socket, +an offset map) produces no symptom at all in a short run, which is precisely the case for everything +the current suites exercise. + +So the question this lane answers is not "might there be leaks" - one is already open. It is **how +many of that class are currently invisible**, in eleven libraries, none of which is run for longer +than a few seconds by anything that exists today. + +## What it has to measure, per language + +- **Resources that should return to baseline**: live threads, open file descriptors and sockets, + heap after a forced collection, and the sidecar child process count. Sampled across many + create/use/close cycles of the client, not once at the end - the shape of the curve is the finding, + and a single end-of-run reading cannot tell a leak from a high-water mark. +- **Throughput that should not decay.** A rate that starts at X and finishes at 0.6X over an hour is + a defect the comparison demo structurally cannot see, because it reports one number for one short + replay. Report first-decile against last-decile, not an average, which hides exactly this. +- **Offsets and correctness under duration**: records processed equals records produced, keys stay + distinct, and nothing is committed twice. The deterministic `records`/`keys` pair the demos already + compute is the cheapest available oracle and should be reused rather than reinvented. + +## Constraints worth settling before starting + +- **It cannot live in the default suite.** A lane that takes an hour cannot gate a pull request. The + repo already has the machinery for this decision - `bin/performance-test.sh` and the self-hosted + highcpu runner ([`docs/self-hosted-runner.md`](../self-hosted-runner.md)) - so the question is + which of those it joins, not whether to build a third thing. +- **A soak result on a loaded box is worthless**, and this project has already paid for that lesson: + every throughput figure from the eleven-agent fan-out was discarded because the machine ran at load + 20-113. A soak lane that does not assert an idle host will produce numbers nobody may cite, which + is worse than producing none. +- **Leak thresholds must be stated per language, not shared.** A JVM's heap after GC, a Go runtime's + goroutine count and a Ruby process's RSS are not comparable quantities, and a single cross-language + threshold would be either meaningless or permanently red somewhere. +- **This is the third coverage**, alongside the two in + [`parallel-consumer-proxy-clients/AGENTS.md`](../../parallel-consumer-proxy-clients/AGENTS.md) - + the API-level conformance suite and the demo output-conformance check. That doc should name it + once it exists, and say what it catches that neither of the others can. + +## Related + +- [`next-demo-testing-infrastructure.md`](next-demo-testing-infrastructure.md) - the ranked ambition + for the demo harnesses; this is the durability half of it. +- [`next-demo-seed-followups.md`](next-demo-seed-followups.md) item 1 - the unexplained AK core + baseline shift (~345 to ~300 msg/s between sessions, a control arm refuted the obvious cause). A + soak lane that reports sustained throughput is one of the few things that could turn that from an + anecdote into a measurement. diff --git a/docs/inflight/parked-containerised-toolchains-and-runtime.md b/docs/inflight/parked-containerised-toolchains-and-runtime.md index 87ec503666..ac6a33780e 100644 --- a/docs/inflight/parked-containerised-toolchains-and-runtime.md +++ b/docs/inflight/parked-containerised-toolchains-and-runtime.md @@ -53,6 +53,77 @@ and running a single test still needs the interpreter on the host. Node and Pyth someone else's containers. This is a local-and-agent reproducibility question, which lowers its stakes. +### MEASURED 2026-08-22: extraction does not work on macOS at all, and the caveat above understates why + +The first full local run of the conformance suite got all ten runners built and then failed **20 of +20 scenarios for exactly the two container-route languages**, cpp and swift, each with **exit 126** +- "found, but cannot execute": + +``` +runner : swift +exit : 126 +file : ELF 64-bit LSB pie executable, ARM aarch64, dynamically linked +host : macOS, arm64 +``` + +The caveat above frames this as **linking discipline** - "static by default" so a glibc mismatch does +not bite on a different host. That is the right rule for Linux-to-Linux, and **it does not help +here**: no amount of static linking makes a Linux ELF execute on Darwin. Build-and-extract is sound +only when the host OS matches the image OS. On macOS the artifact is unrunnable however it is linked. + +**The consequence for the fallback idea is the sharp one.** If a developer without mise is served by +"build it in a container instead", that fallback is broken precisely on the machines that need it: +macOS developers are the ones who cannot get Swift and C++ toolchains from mise, and they are the +ones for whom an extracted Linux binary cannot run. A container **build** fallback therefore has to +be a container **run** fallback too - which is section 2's question, not section 1's, and is a much +larger change than pointing an exec at Docker. + +**CI is unaffected and will stay green through this**, because its runners are Linux, so an extracted +Linux binary is native there. That is the trap: the gap is invisible to every check that exists and +appears only on a developer machine, which is the same shape as the toolchain drift recorded in +[`ci-toolchain-versions-declared-twice.md`](ci-toolchain-versions-declared-twice.md). + +Nothing here argues against the container route for *building* - it built both languages cleanly on a +host with neither toolchain, which is what it is for. It argues that "built" and "runnable here" are +different claims, and only the first one is currently true off Linux. + +### SOLVED for the conformance suite, 2026-08-22: run the whole thing inside + +`bin/run-conformance-in-container.sh ` now drives either language on macOS: + +``` +cpp 5 scenarios driven 10 tests 0 failures BUILD SUCCESS +swift 5 scenarios driven 10 tests 0 failures BUILD SUCCESS +``` + +**It needed no product change at all** - no client API, no protocol change, nothing touching R29's +authority allowlist. The insight was the owner's: containerising only the *runner* is what created +every boundary problem. The client spawns its sidecar as a child and dials `127.0.0.1` (KTD41), and +`SidecarShim` announces a bare port with no host deliberately, so the client exercises its real +spawn-and-reap path. Move the whole run inside - suite, in-JVM engine, shim, runner - and that +loopback is simply the container's own. + +A `conformance` stage in each module's existing Dockerfile is `FROM toolchain` plus a JDK, with the +runner **copied from the same `build` stage CI extracts from**, so this path and CI drive one binary +rather than two builds that could differ. `LanguageRunners` takes it via +`PC_CONFORMANCE_PREBUILT_RUNNERS` and carries an empty build command - the shape Kotlin and Scala +already use for "the reactor built it" - which also avoids running the container route *inside* a +container that has no Docker. + +**Two things are still open.** + +1. **CI will never exercise this**, because its runners are Linux and the extracted artifact is + native there. It is a macOS-developer path only, so it rots silently - the exact risk the owner + named before it was built. A scheduled run is what would keep it honest, and nothing schedules + one yet. +2. **It is not `--network none`**, which would have proved nothing crosses the boundary. A macOS + `~/.m2` holds no `com.google.protobuf:protoc:exe:linux-aarch_64`, so a cold offline run dies in + the protocol module before the suite is reached. `PC_CONFORMANCE_OFFLINE=1` restores the proof + once the cache is warm; the classifiers coexist, so warming costs the host nothing. + +**The DEMO runtime is untouched and needs nothing** - each demo already runs in its own container, +where binary and OS match by construction. This section was only ever about the conformance suite. + **Cheaper middle path to weigh first:** this repo already manages Java, Go and Node through mise, and mise also covers Rust, Ruby and .NET. Extending what exists costs far less than an image fleet; containers earn their place for the genuinely awkward two (Swift on Linux, C++ with gRPC dev libs). diff --git a/docs/inflight/test-processinkeyorder-sanity-check-races-the-first-poll.md b/docs/inflight/test-processinkeyorder-sanity-check-races-the-first-poll.md index 6678e5e2b5..756aa2d9e1 100644 --- a/docs/inflight/test-processinkeyorder-sanity-check-races-the-first-poll.md +++ b/docs/inflight/test-processinkeyorder-sanity-check-races-the-first-poll.md @@ -60,6 +60,7 @@ wants. | 2026-09-01 | astubbs/parallel-consumer#390 (the ten foreign dispatch clients), full reactor `test` with `-Dpc.foreignClients`, macOS | `processInKeyOrder(CommitMode)[1]` red on the **sanity check**, `actual size is 0 while expected size is 9` - the same assertion and the same shape as the original sighting, a different parameterisation again. Same load condition: a second agent was running `parallel-consumer-core` tests in another worktree on the same box at the time. That PR's diff touches no file under `parallel-consumer-core`. | | 2026-09-01 | the same, second run, on a box with nothing else running | red again, and wider: `processInKeyOrder` failed on **all three** parameterisations plus `inFlightMessagesCommittedIfProcessedDuringShutdown[3]`, same assertion each time. **A control arm was run and it does not support "load".** The identical command on the base branch, same box, same window: green. The method alone on the branch: green 3 of 3. `parallel-consumer-core` is byte-identical between the two branches, so nothing in the diff can explain the split - and CI's `tests` job, which runs the same suite on a Linux runner, was green on that PR. Reported as an unexplained rate rather than a verdict: red 2 of 2 locally on one branch, green 1 of 1 on its base, green on CI. | | 2026-09-01 | astubbs/parallel-consumer#293 (the language-proxy engine), while merging the Wagon A stack into it; macOS, JDK 17 | **The control arm here reddens `master` itself, which no previous sighting had done, and it is the first evidence that this is master-state rather than branch-state.** On the merged branch, full default reactor `test` with another agent's `parallel-consumer-streams` suite running in a second worktree: `processInKeyOrder(CommitMode)[1]` red on the sanity check, 1 failure in 543. The control - unmodified `master` at `b2e6c190d`, `-pl parallel-consumer-core -am test`, same box, **nothing else running** - was **redder**: `processInKeyOrder` failed on **all three** parameterisations, 3 failures in 533, same assertion at the same line. The method alone on the branch: green 3 of 3. So the branch arm is not the worse arm, and the quieter box is not the greener one - which retires "load is the variable that moves it" as the whole story, and specifically refutes the row above's "the identical command on the base branch was green". Whatever this is, it is in `master` and it does not need a proxy branch to appear. | +| 2026-09-01 | astubbs/parallel-consumer#331 (the ten per-language demos), while forward-merging astubbs/parallel-consumer#328 into it; macOS, JDK 17 | `processInKeyOrder(CommitMode)[3]` red on the sanity check, 1 failure in 543, same assertion and same line. **The strongest same-day control arm so far, and it points the same way as the row above.** `parallel-consumer-core` is byte-identical between that merge's result and the one a rung below it - `git diff --cached feats/java-sidecar-demo -- parallel-consumer-core` is empty - and the rung below ran the identical full-reactor command **five times on the same box within the same few hours: 543 tests, 0 failures, every time**. So one tree red once and a byte-identical tree green five times, with no code difference to appeal to. That is a rate, not a cause: it rules out both branches' diffs and leaves the test itself. | The fix, when somebody takes it, is the same rule the solutions write-up already states: await the diff --git a/docs/inflight/test-reactor-concurrency-exceeded-under-load.md b/docs/inflight/test-reactor-concurrency-exceeded-under-load.md new file mode 100644 index 0000000000..421b14d081 --- /dev/null +++ b/docs/inflight/test-reactor-concurrency-exceeded-under-load.md @@ -0,0 +1,74 @@ +# `ReactorPCTest.concurrencyTest` exceeds max concurrency on a saturated machine + +Sighting recorded 2026-08-22, on `feats/polyglot-demos`, running locally. **Diagnosed as contention, +with a control arm - not quarantined, and not a known flake before this.** Recorded because it had +no entry anywhere, and the next person to hit it would repeat the whole investigation. + +## The failure + +``` +[ERROR] bz.stub.parallelconsumer.reactor.ReactorPCTest.concurrencyTest +Max concurrency should never be exceeded +expected to be less than: 1200 +but was : 1266 +``` + +The threshold is `maxConcurrency * MAX_CONCURRENCY_OVERFLOW_ALLOWANCE`, i.e. an allowance that +already exists to absorb scheduling slop. The failing run overshot what the allowance absorbs. + +**Conditions:** the full Maven reactor was running while **two Docker image builds** (C++ and Swift +demo containers) compiled on the same 12-core box. That is well past the load anything here is meant +to be measured under. + +## The control arm, and why it was not skipped + +Both arms uncontended, **run sequentially** - running them together would reintroduce the variable +under test - five runs each, same host, same JDK 17: + +| arm | result | observed max concurrency | +|---|---|---| +| `feats/polyglot-demos` | **5 pass / 0 fail** | 1000, 1073, 1000, 1062, 1000 | +| `master` (control) | **5 pass / 0 fail** | 1000, 1000, 1076, 1000, 1000 | + +Indistinguishable. The branch does not run hotter than master, and neither approaches 1200 without +load. **Contention, not a regression** - and 0/5 uncontended against 1/1 under saturation is the +distinction, not a verdict from a single run either way. + +## Why a control was warranted rather than an assumption + +This branch's stack **does** modify the classes that bound concurrency - `ShardManager`, +`WorkManager`, `ProcessingShard`, `WorkContainer` - so "it was just load" was a hypothesis, not an +observation. The change adds the **abandonment path** the language proxy needs: work returned with +no verdict becomes selectable again without earning a retry delay it never earned. + +There is even a plausible mechanism from that change to this symptom, which is why it had to be ruled +out rather than waved away: `WorkManager.onAbandonedResult` decrements +`numberRecordsOutForProcessing`, and a **double** decrement would under-count in-flight work, let +extra records out, and surface as observed concurrency exceeding the maximum. The author guarded +exactly that - an early return for `isReturnForSupersededDelivery()`, placed before a branch that +decrements unconditionally, with a comment saying why. + +**It is unreachable from this test**, which is what closes the question: nothing in +`parallel-consumer-core` or `parallel-consumer-reactor` calls the abandon path, and +`isReturnForSupersededDelivery()` requires `abandonedAtDelivery >= 0`, which only a proxy return +sets. The new branches are inert here. The matching master baseline is the empirical half of the +same answer. + +## Prior art: none, and that is the point of this file + +Checked before forming a hypothesis, and all five returned nothing for this test: +`docs/inflight/`, `docs/quarantined-tests.md`, `docs/solutions/`, `docs/plans/`, +`docs/test-hardening/`. The single grep hit was a Streams benchmarking document that happens to +contain the phrase "max concurrency" - unrelated. `git log` shows the test itself last changed in +the package rename (content-only) and before that in upstream work from 2022. + +## What NOT to do + +**Do not raise `MAX_CONCURRENCY_OVERFLOW_ALLOWANCE`, and do not quarantine this.** Quarantine +requires a diagnosis and is master-state, not PR-state; this is neither failing on master nor +failing uncontended. Loosening the allowance would remove the only signal that the concurrency bound +is real, in the library whose entire purpose is bounded concurrency. + +**If it is seen again, capture the load average with it.** The one thing this sighting cannot +supply is the threshold at which it flips, and that is the number that would decide whether the +suite needs a contention guard rather than the test needing a change. diff --git a/docs/inflight/test-untracked-ci-flakes.md b/docs/inflight/test-untracked-ci-flakes.md index 203fc22613..f18dad00b6 100644 --- a/docs/inflight/test-untracked-ci-flakes.md +++ b/docs/inflight/test-untracked-ci-flakes.md @@ -83,7 +83,12 @@ defect: - `CheckQuarantineOwnersScriptTest` - two different methods, on different runs - `ProxyProcessorLivenessTest.aSlowWorkerKeepsItsRecordWhileHeartbeatsContinueAndLosesItWhenTheyStop` - `JStreamParallelEoSStreamProcessorTest.testConsumeAndProduce` - added 2026-08-17, seen with ~60 - worktrees live on the box; passed in isolation and in the same session's full post-change run + worktrees live on the box; passed in isolation and in the same session's full post-change run. + **Seen again 2026-08-22** on `feats/polyglot-demos`, failing on `Expected size: 1 but was: 0`, + while a Go test suite ran in the same command on a box with many worktrees live. The branch that + met it had changed **zero** files under `parallel-consumer-core` (checked, not assumed), and the + same module's tests had passed in three earlier full runs that session. Same classification: + contention, not a regression - `ParallelEoSStreamProcessorTest.processInKeyOrder(CommitMode)[3]` - added 2026-08-21, seen once while building U10 in the proxy module (`-am` again). **A different assertion from the commit-frontier symptom this file records for the same test further down**: this one failed on diff --git a/mise.toml b/mise.toml new file mode 100644 index 0000000000..bcccedecf3 --- /dev/null +++ b/mise.toml @@ -0,0 +1,29 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# The toolchain versions this repository builds against. +# +# THIS FILE IS THE SINGLE DECLARATION. `.github/workflows/clients.yml` installs the same versions +# with per-language `setup-*` actions rather than with mise, because those actions also carry the +# caching the client matrix depends on and `ruby/setup-ruby` is pinned by commit SHA - a supply-chain +# posture worth keeping. So the two INSTALL differently on purpose, and `bin/check-toolchain-versions.sh` +# asserts they never DECLARE differently. Change a version here and that gate will tell you to change +# the matrix to match; nothing else compares them, and they had already drifted by whole major +# versions (dotnet 8 against 9, node 22 against 25) before the gate existed. +# +# Four languages are deliberately absent, and their absence is checked too: +# swift, cpp - built in a container by `bin/build-client.sh`, so no host toolchain is pinned +# kotlin, scala - built by the Maven reactor, so the JDK is their toolchain +# +# The JDK itself is not here. It is installed per developer (sdkman, brew, a distribution package) +# and by `actions/setup-java` in CI; AGENTS.md states the requirement and the failure mode when it is +# wrong. Folding it in would be more DRY and is a bigger change than this file. +# +# Install everything with `mise install`, or one with `mise use -g go@1.25.13`. + +[tools] +go = "1.25.13" +python = "3.13.5" +rust = "1.88.0" +dotnet = "8.0.404" +node = "22.17.0" +ruby = "3.4.4" diff --git a/parallel-consumer-proxy-clients/AGENTS.md b/parallel-consumer-proxy-clients/AGENTS.md new file mode 100644 index 0000000000..fe2c1090ab --- /dev/null +++ b/parallel-consumer-proxy-clients/AGENTS.md @@ -0,0 +1,81 @@ +# `parallel-consumer-proxy-clients/` - two kinds of coverage, and which one your change needs + +Rules for working inside the client collection. The root `AGENTS.md` routes here; this file owns the +part that binds anyone adding to a client or its demo. + +## There are TWO test suites over this code, and they do not overlap + +Knowing which one a change needs is the point of this file, because the easy mistake is to assume +the other one already covers you. + +| | **Cross-language conformance** | **Demo output conformance** | +|---|---|---| +| lives in | `parallel-consumer-proxy-conformance` | `bin/ci-demo-conformance.sh` | +| binds to | the client library's **API**, called directly | the demo's **standard output** | +| runs against | the in-JVM `ProxyHarness` with mock Kafka clients | a real broker, a real spawned sidecar, a real container | +| proves | that every client means the same thing by the protocol | that every demo still *behaves* the same to a reader | +| cannot see | whether a demo is readable, or runs at all | whether the sidecar arm truly uses the client library | + +**Neither is a substitute for the other, and each has already caught what the other structurally +cannot.** Conformance could not see that seven demos silently dropped six of seven configuration +dials, because it never runs a demo. The output suite could not see that the direct and gRPC +transports disagree about `enable.auto.commit`, because that transport's conformance test injects +its own consumer and never reaches the code path that builds one. + +## The rule: a feature that reaches the demo needs an assertion in the output suite + +**When you add a feature to a client or the proxy that a user can observe, add the assertion that +proves it - in the suite that can see it.** + +- The feature changes the client's **API or protocol behaviour** -> a conformance scenario. +- The feature changes **what a demo prints, accepts, or exits with** -> an assertion in + `bin/ci-demo-conformance.sh`, and the contract entry it enforces in + `parallel-consumer-proxy/demo/README.md`. +- It does both -> both. That is common and not a sign you have over-tested. + +This is a rule because **nothing enforces it**. A demo can gain a flag, a column or a mode that no +check ever looks at, and the first sign of trouble is a reader finding it broken. Every defect the +output suite has found so far was of exactly that shape. + +### What the output suite is cheap at, and why that matters + +It asserts on stdout, so it needs no knowledge of any language. Adding a language costs nothing, and +adding an assertion covers eleven implementations at once. It also uses the languages as **each +other's oracle**: outputs are reduced to a skeleton with the volatile figures discarded, and the +skeletons must match. There is no expected-output file to maintain, and the check gets *stronger* +with each language rather than more expensive. + +So the bar for adding an assertion there is low. Take it. + +## Keep the console and the assertions apart + +**Two audiences, two channels, and mixing them makes both worse.** + +- **The console is for a person.** The first line names the product; the tables are the point; the + broker is quietened to `WARN` so it does not bury them. Do not put machine-readable diagnostics + there because a test wants them. +- **The technical record goes to a file.** A demo may write a per-record ledger - identifier, key, + partition, offset, attempt, received and completed timestamps - and assertions run against *that*. + It is off by default and switched on by a flag. + +That split is what lets the assertions get more detailed over time without the demo getting worse to +watch. Things worth asserting from a ledger that would be noise on screen: + +- **overlap**: records whose processing intervals overlap prove concurrency actually happened, + rather than being inferred from a rate; +- **order**: with a known publish order and key format, per-key sequence can be checked directly; +- **exactly once through the demo**: distinct `(partition, offset)` equals the processed count, so + nothing was redelivered; +- **attempts**: with a failure percentage configured, that every record eventually succeeded. + +## Performance baselines: wide bands, and only for catastrophe + +A per-language baseline is worth having, with **a great deal of headroom** - it is there to catch a +change that makes something ten times slower, not to police percentages. + +**Be careful what you promise here.** Measurements on this project have not yet been shown to +reproduce *across sessions*: the Java seed's serial arm moved from ~345 to ~300 msg/s on one machine +with the cause unexplained and the obvious candidate refuted by a control arm +([`docs/inflight/next-demo-seed-followups.md`](../docs/inflight/next-demo-seed-followups.md), item 1). +Until that is understood, a tight band would be a flake generator. A band wide enough to be honest +is still worth having; a tight one is worse than none. diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/Dockerfile b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/Dockerfile index 9498adef99..354e5abf10 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/Dockerfile +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/Dockerfile @@ -87,3 +87,35 @@ COPY --from=build /work/build/pc-cpp-selftest / COPY --from=build /work/build/pc-cpp-selftest-dynamic / COPY --from=build /work/build/pc-cpp-conformance-runner / COPY --from=build /work/build/link-report.txt / + +# ───────────────────────────────────────────────────────────────────────────────────────────────── +# The stage the CONFORMANCE SUITE runs inside, for hosts that cannot execute what `artifact` exports. +# +# `artifact` above is a Linux binary. On a Linux host that runs natively and this stage is unused; on +# macOS it cannot be executed at all - the shell answers 126, "found but cannot execute" - so C++ and +# Swift simply could not be driven from a developer's laptop. +# +# Containerising only the runner does not fix it. The client spawns the sidecar as its own child and +# talks to it over loopback, and the suite's shim announces a bare port with no host, deliberately, so +# the client exercises its real spawn-and-reap path. Client, shim and engine must therefore share ONE +# loopback - and measured on Docker Desktop, a container cannot reach the host's 127.0.0.1 even with +# --network host. So the whole run comes in here instead, where that loopback is this container's own. +# +# The runner is COPIED from the same `build` stage CI extracts from, so this stage and CI drive the +# identical binary rather than two builds that could differ. Nothing is compiled here. +FROM toolchain AS conformance + +# A JDK, not a JRE: the suite is built and run by Maven inside this container. +COPY --from=eclipse-temurin:17-jdk /opt/java/openjdk /opt/java/openjdk +ENV JAVA_HOME=/opt/java/openjdk +ENV PATH="/opt/java/openjdk/bin:${PATH}" + +COPY --from=build /work/build/pc-cpp-conformance-runner /opt/pc-runners/pc-cpp-conformance-runner + +# Read by LanguageRunners.prebuiltRunner: with this set, the C++ entry carries NO build command, +# exactly as the JVM languages do, because its binary is already here. +ENV PC_CONFORMANCE_PREBUILT_RUNNERS=/opt/pc-runners + +# The repository is mounted here by bin/run-conformance-in-container.sh rather than copied: the suite +# is driven from the working tree the developer is actually editing. +WORKDIR /repo diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/README.md b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/README.md index e492ea1d50..3fe3cd57ea 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/README.md +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/README.md @@ -86,6 +86,21 @@ lifecycle never reaching `validate` where the enforcer is bound. - **The extracted artifacts carry the timestamps of the layer that made them**, not of your build, so `ls -l` is no evidence a compile ran. Read `link-report.txt` for what was built, and with what. +## The demo + +```bash +parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/run.sh +``` + +Needs Docker and nothing else - no C++ toolchain, no gRPC dev packages, no JDK. It replays the same +records through two arms, **librdkafka one record at a time** and **C++ over the sidecar through +this client library**, and prints two throughput tables. The broker is a compose sibling; the +sidecar is a child process the library spawns, not a service you deploy. + +[`demo/README.md`](demo/README.md) records what is specific to C++ - chiefly that there is no native +mode, because the container *is* the toolchain here. The contract every language's demo keeps is +[`parallel-consumer-proxy/demo/README.md`](../../parallel-consumer-proxy/demo/README.md). + ## Using it ```cpp diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/CMakeLists.txt b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/CMakeLists.txt new file mode 100644 index 0000000000..570fff69fd --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/CMakeLists.txt @@ -0,0 +1,134 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# The C++ demo (astubbs#242, plan unit U35). Like the module's own CMake project beside it, this is +# only ever configured INSIDE an image - here demo/Dockerfile - because gRPC, protobuf and +# librdkafka arrive as system DEV PACKAGES rather than as a versioned toolchain. +# +# A PROJECT OF ITS OWN RATHER THAN A TARGET IN ../CMakeLists.txt, and the reason is the build +# environment rather than taste: the demo needs librdkafka, which the client library does not, and +# adding it to the module's project would put a Kafka client into the image that `bin/build-client.sh +# cpp` builds and into the artifact-portability claim that image makes. The demo is a RUNTIME image; +# that one is a BUILD environment that exports artifacts and runs nothing. Keeping the projects +# separate keeps that distinction true. +# +# TWO TARGETS: +# pc-cpp-demo the demo itself - both arms, the two tables +# pc-cpp-demo-tests the entry path and the OUTPUT SHAPE, run by ctest inside the image build +# (see tests/) - the two parts of the demo that need no broker + +cmake_minimum_required(VERSION 3.20) + +project(parallel-consumer-proxy-client-cpp-demo LANGUAGES CXX) + +set(CMAKE_CXX_STANDARD 17) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +find_package(Threads REQUIRED) + +enable_testing() + +set(PC_MODULE_ROOT "${CMAKE_CURRENT_SOURCE_DIR}/..") + +# ── The FROZEN schema, mounted in as a BuildKit named context ──────────────────────────────────── +# Identical in shape to the module's own: never copied into this module, and the placement comes +# from the .proto's own options rather than from this command line. +set(PC_PROTO_ROOT "" CACHE PATH "Root of the proto import path (contains parallelconsumer/proxy/v1)") +if(NOT PC_PROTO_ROOT) + message(FATAL_ERROR "PC_PROTO_ROOT must name the directory containing parallelconsumer/proxy/v1/proxy.proto") +endif() +set(PC_PROTO_FILE "${PC_PROTO_ROOT}/parallelconsumer/proxy/v1/proxy.proto") +if(NOT EXISTS "${PC_PROTO_FILE}") + message(FATAL_ERROR "No proto at ${PC_PROTO_FILE}") +endif() + +find_program(PC_PROTOC protoc REQUIRED) +find_program(PC_GRPC_CPP_PLUGIN grpc_cpp_plugin REQUIRED) + +set(PC_GEN_DIR "${CMAKE_BINARY_DIR}/gen") +set(PC_GEN_SOURCES + "${PC_GEN_DIR}/parallelconsumer/proxy/v1/proxy.pb.cc" + "${PC_GEN_DIR}/parallelconsumer/proxy/v1/proxy.grpc.pb.cc") + +add_custom_command( + OUTPUT ${PC_GEN_SOURCES} + COMMAND "${CMAKE_COMMAND}" -E make_directory "${PC_GEN_DIR}" + COMMAND "${PC_PROTOC}" + "--proto_path=${PC_PROTO_ROOT}" + "--cpp_out=${PC_GEN_DIR}" + "--grpc_out=${PC_GEN_DIR}" + "--plugin=protoc-gen-grpc=${PC_GRPC_CPP_PLUGIN}" + "${PC_PROTO_FILE}" + DEPENDS "${PC_PROTO_FILE}" + COMMENT "protoc: generating C++ and gRPC stubs from the frozen proxy.proto" + VERBATIM) + +# ── Dependencies ───────────────────────────────────────────────────────────────────────────────── +# DYNAMICALLY LINKED, unlike the module's static artifacts, and that is the runtime image's whole +# point: nothing here is extracted to a host, so there is no portability claim to prove and no +# reason to pay for the static link's absl archive group. The demo runs where it was built. +find_package(PkgConfig REQUIRED) +pkg_check_modules(PC_GRPCPP REQUIRED grpc++) +pkg_check_modules(PC_PROTOBUF REQUIRED protobuf) +# librdkafka: C++'s own Kafka client, and the AK core arm's whole reason to exist. The C API, not +# rdkafka++ - see demo/src/demo_broker.h. +pkg_check_modules(PC_RDKAFKA REQUIRED rdkafka) + +# ── The client library, compiled from the module beside this one ───────────────────────────────── +# GLOBBED rather than listed, so the demo cannot fall out of step with a library that grows a source +# file: a stale list would fail at LINK time with an undefined symbol, days later and far from the +# cause. The two `main`s next door are removed by name; a third would announce itself immediately as +# a duplicate main rather than silently. +file(GLOB PC_CLIENT_SOURCES "${PC_MODULE_ROOT}/src/*.cpp") +list(REMOVE_ITEM PC_CLIENT_SOURCES + "${PC_MODULE_ROOT}/src/selftest.cpp" + "${PC_MODULE_ROOT}/src/conformance_runner.cpp") +list(LENGTH PC_CLIENT_SOURCES PC_CLIENT_SOURCE_COUNT) +if(PC_CLIENT_SOURCE_COUNT LESS 1) + message(FATAL_ERROR "No client library sources found under ${PC_MODULE_ROOT}/src") +endif() + +set(PC_INCLUDE_DIRS + "${PC_MODULE_ROOT}/src" + "${CMAKE_CURRENT_SOURCE_DIR}/src" + "${PC_GEN_DIR}" + ${PC_GRPCPP_INCLUDE_DIRS} + ${PC_PROTOBUF_INCLUDE_DIRS} + ${PC_RDKAFKA_INCLUDE_DIRS}) + +add_executable(pc-cpp-demo + "${CMAKE_CURRENT_SOURCE_DIR}/src/demo.cpp" + "${CMAKE_CURRENT_SOURCE_DIR}/src/demo_broker.cpp" + "${CMAKE_CURRENT_SOURCE_DIR}/src/demo_options.cpp" + "${CMAKE_CURRENT_SOURCE_DIR}/src/demo_report.cpp" + ${PC_CLIENT_SOURCES} + ${PC_GEN_SOURCES}) +target_include_directories(pc-cpp-demo PRIVATE ${PC_INCLUDE_DIRS}) +target_compile_options(pc-cpp-demo PRIVATE -Wall -Wextra -Werror) +# Generated code is not this module's to keep warning-clean, and -Werror over it would make a Debian +# protoc bump break the build for a reason nobody here can fix. +set_source_files_properties(${PC_GEN_SOURCES} PROPERTIES COMPILE_OPTIONS "-w") +target_link_libraries(pc-cpp-demo PRIVATE + Threads::Threads + ${PC_GRPCPP_LDFLAGS} + ${PC_PROTOBUF_LDFLAGS} + ${PC_RDKAFKA_LDFLAGS}) + +# ── The demo's own tests ───────────────────────────────────────────────────────────────────────── +# The module's harness, not a second one: tests/test_main.cpp next door is the runner, and +# test_support.h is the sixty lines of macros. Only demo_options.cpp and demo_report.cpp are linked +# in - the two units with no broker in them - so these tests need neither gRPC nor Kafka and run in +# a second inside the image build. demo.cpp and demo_broker.cpp are deliberately absent: an arm +# cannot be tested without a broker, whereas the contract's rules about the OUTPUT can be. +add_executable(pc-cpp-demo-tests + "${PC_MODULE_ROOT}/tests/test_main.cpp" + "${CMAKE_CURRENT_SOURCE_DIR}/tests/demo_options_test.cpp" + "${CMAKE_CURRENT_SOURCE_DIR}/tests/demo_report_test.cpp" + "${CMAKE_CURRENT_SOURCE_DIR}/src/demo_options.cpp" + "${CMAKE_CURRENT_SOURCE_DIR}/src/demo_report.cpp") +target_include_directories(pc-cpp-demo-tests PRIVATE + "${CMAKE_CURRENT_SOURCE_DIR}/src" + "${PC_MODULE_ROOT}/tests") +target_compile_options(pc-cpp-demo-tests PRIVATE -Wall -Wextra -Werror) +target_link_libraries(pc-cpp-demo-tests PRIVATE Threads::Threads) + +add_test(NAME demo-entry-and-output COMMAND pc-cpp-demo-tests) diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/Dockerfile b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/Dockerfile new file mode 100644 index 0000000000..5c564354e8 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/Dockerfile @@ -0,0 +1,149 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE DEMO CONTAINER for the C++ client (astubbs#242, plan unit U35, governed by KTD40). +# +# THIS IS A RUNTIME IMAGE, AND THE OTHER C++ DOCKERFILE IS NOT. ../Dockerfile is a containerised +# BUILD environment: it exports static artifacts to a host directory with `--output type=local` and +# runs nothing. This one is something a person runs. Read that file first anyway - it explains why +# C++ builds in a container at all (gRPC and protobuf arrive as system dev packages rather than as +# a versioned toolchain, so mise cannot serve this language), and its Debian trixie base is the +# reason this one is Debian trixie too. +# +# FOR C++ THE CONTAINER IS THE ONLY WAY IN, which is the one place this demo diverges from the +# reference. Java's run.sh picks native or container; there is no native path here, because there is +# no host C++ toolchain and a demo that told a reader to install gRPC dev packages first would not +# be a one-command demo. demo/README.md records it. +# +# THE BUILD CONTEXT IS THE REPOSITORY, and that is the same deliberate departure the Java demo's +# Dockerfile makes. Two things here come from outside this module and cannot come any other way: the +# FROZEN proxy.proto, which is never copied into a client module, and the SIDECAR, which is a Maven +# reactor module that cannot be built without its parents and siblings. The hazard the +# module-directory rule protects against - uploading other agents' target/ output - is addressed by +# the repository-root .dockerignore instead. Those two files must be read together. +# +# THE SIDECAR IS BUILT IN, NOT DEPLOYED. It is a child process the client library spawns and +# supervises, so the reader installs, deploys and operates nothing (KTD41). It is emphatically not a +# compose service; see docker-compose.yml. + +# ── The C++ toolchain, shared by the build stage and the runtime stage ─────────────────────────── +# The runtime stage is built FROM this rather than from a slimmer base, so the demo's dynamically +# linked binary is guaranteed the exact libraries it was linked against. The image is large as a +# result, which is the trade a runtime image makes and a build image does not - the Java demo's +# Dockerfile records the same trade for the same reason. +FROM debian:trixie-slim AS cpp-toolchain + +# apt's caches are what the cache mounts below preserve, so the image must stop deleting them. +RUN rm -f /etc/apt/apt.conf.d/docker-clean \ + && printf 'Binary::apt::APT::Keep-Downloaded-Packages "true";\n' > /etc/apt/apt.conf.d/keep-cache + +# librdkafka-dev is the one package this image needs that ../Dockerfile does not: it is C++'s own +# Kafka client, and the AK core arm is the whole reason the demo has a comparison to make. +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ + --mount=type=cache,target=/var/lib/apt/lists,sharing=locked \ + apt-get update \ + && apt-get install --yes --no-install-recommends \ + build-essential \ + cmake \ + pkg-config \ + libgrpc++-dev \ + libprotobuf-dev \ + protobuf-compiler \ + protobuf-compiler-grpc \ + librdkafka-dev \ + libssl-dev \ + zlib1g-dev \ + libzstd-dev \ + ca-certificates + +# ── The demo binary ────────────────────────────────────────────────────────────────────────────── +FROM cpp-toolchain AS cpp-build + +WORKDIR /work +# The frozen schema, from the module that owns it. Never copied into a client module - there is +# exactly one proxy.proto in this repository. +COPY parallel-consumer-proxy-protocol/src/main/proto /work/proto +# The client library, its test harness, and the demo. The demo links the library from source rather +# than from an artifact, because that is what proves the demo exercises THIS library. +COPY parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/src /work/module/src +COPY parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/tests /work/module/tests +COPY parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo /work/module/demo + +RUN cmake -S /work/module/demo -B /work/build \ + -DCMAKE_BUILD_TYPE=Release \ + -DPC_PROTO_ROOT=/work/proto \ + && cmake --build /work/build --parallel "$(nproc)" + +# THE DEMO'S OWN TESTS, IN HERE, for the same reason the client library's run inside its image: this +# host has no C++ toolchain, so a check that only ran on the host would only ever run in CI. They +# cover the entry path - the no-argument case, the flag/environment precedence, the fingerprint's +# silence about the broker - which is where every failure this fan-out's demos have actually had +# lived. A red test fails the IMAGE BUILD, which fails the demo. +RUN ctest --test-dir /work/build --output-on-failure + +# ── The sidecar ────────────────────────────────────────────────────────────────────────────────── +# A JDK image builds it; only its jars and a JRE reach the final stage. +FROM eclipse-temurin:17-jdk AS sidecar-build + +WORKDIR /src +COPY . . + +# THE JARS ARE COPIED OUT, NOT LEFT IN A MAVEN REPOSITORY, and that is what lets the local +# repository be a BuildKit cache mount here where the Java demo's Dockerfile could not have one. Its +# classpath file points into /root/.m2, so a cache mount - which is not part of the resulting image - +# would leave it naming jars the running container does not have. Real copies plus a wildcard +# classpath have no such coupling, and the Maven downloads stay out of the image. +# +# `sharing=locked` because ten sibling demos may build at once and a Maven local repository is not +# safe for concurrent writers. +# +# -Dcopyright.skip=true because the header check needs the repository's git history to find its fork +# point, and .dockerignore excludes .git from the context. It is a source-tree gate that already runs +# in every ordinary build and in CI's own copyright lane. +# +# AND THEN AN SLF4J BINDING, WHICH THE RUNTIME SCOPE DOES NOT CARRY. The proxy logs through the SLF4J +# API and every binding in this reactor is test-scoped, so the sidecar spawned by this image would +# print `No SLF4J providers were found` and then nothing at all - measured, not assumed. That +# silences exactly the diagnostics the client library goes out of its way to preserve: it inherits +# the sidecar's stderr by default so that a misconfigured broker explains itself instead of becoming +# an unexplained hang. slf4j-simple writes INFO to stderr with no configuration file, which is what a +# demo wants. Its version is READ OFF the api jar already copied rather than pinned here, so it +# cannot drift from the root pom's. +RUN --mount=type=cache,target=/root/.m2,sharing=locked \ + ./mvnw --batch-mode -q -DskipTests -Dcopyright.skip=true \ + -pl :parallel-consumer-proxy -am \ + package dependency:copy-dependencies \ + -DincludeScope=runtime -DoutputDirectory=/app/sidecar/lib \ + && cp -r parallel-consumer-proxy/target/classes /app/sidecar/classes \ + && if [ -z "$(ls -A /app/sidecar/lib)" ]; then \ + echo 'the sidecar build produced no dependencies - the demo would fail at spawn' >&2; \ + exit 1; \ + fi \ + && SLF4J_VERSION="$(ls /app/sidecar/lib/slf4j-api-*.jar | sed 's|.*/slf4j-api-\(.*\)\.jar|\1|')" \ + && ./mvnw --batch-mode -q --non-recursive -Dcopyright.skip=true \ + dependency:copy "-Dartifact=org.slf4j:slf4j-simple:${SLF4J_VERSION}" \ + -DoutputDirectory=/app/sidecar/lib + +# ── The demo image ─────────────────────────────────────────────────────────────────────────────── +FROM cpp-toolchain AS demo + +# A JRE, not the JDK that built the sidecar: the demo runs the proxy, it does not compile it. +COPY --from=eclipse-temurin:17-jre /opt/java/openjdk /opt/java/openjdk +ENV JAVA_HOME=/opt/java/openjdk +ENV PATH="/opt/java/openjdk/bin:${PATH}" + +COPY --from=sidecar-build /app/sidecar /app/sidecar +COPY parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/sidecar-launcher.sh \ + /app/sidecar/sidecar +RUN chmod +x /app/sidecar/sidecar + +COPY --from=cpp-build /work/build/pc-cpp-demo /app/pc-cpp-demo + +# The absolute path the client library is handed. It is an environment variable rather than a flag +# because the demo's flag list is fixed at seven by the contract, and this is a property of the image +# rather than of the run. +ENV PC_DEMO_SIDECAR=/app/sidecar/sidecar + +# PC_DEMO_ARGS is unquoted on purpose: it carries the caller's flags through compose as a single +# string and must word-split into separate arguments. `exec` so the demo is pid 1's own process and +# a `docker compose down` reaches it. +ENTRYPOINT ["/bin/sh", "-c", "exec /app/pc-cpp-demo $PC_DEMO_ARGS"] diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/README.md b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/README.md new file mode 100644 index 0000000000..22ca540cf8 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/README.md @@ -0,0 +1,131 @@ +# The C++ demo + +```bash +# from anywhere in the repo +parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/run.sh + +# or, from this directory, the plain container path with nothing else needed +docker compose up +``` + +Needs Docker and nothing else - no C++ toolchain, no gRPC development packages, no JDK. Everything +is built in the image: the demo, the client library it links, and the sidecar it spawns. + +**The contract this keeps - and that every other language's demo keeps - is +[`parallel-consumer-proxy/demo/README.md`](../../../parallel-consumer-proxy/demo/README.md).** +Read that first. This file only records what is specific to C++. + +## The two arms + +| arm | what it is | +|---|---| +| `AK core (librdkafka)` | librdkafka, one record at a time, with a blocking sleep as the simulated work. | +| `pc-cpp-grpc (this client)` | this application as a **foreign client**: the client library in the module above spawns the sidecar, receives records over a socket, runs the same sleep on them and reports outcomes back. On this path the application does no Kafka I/O at all. | + +**Each label names the library that produced the row, because "AK core" is a category rather than a +client** - it means "that language's own Kafka client", and the answer is different in every +language. A reader cannot judge a comparison without knowing what produced it. + +### C++ has one serious Kafka client, and that is why only one runs + +The contract asks a language with more than one serious client to say so here, and to consider +running both as separate arms - because the choice materially changes the number, and a reader +asking "is this fast in my language" is really asking about the client they already use. + +**C++ does not have that choice.** librdkafka is the implementation; the other names a C++ developer +will find - `cppkafka`, `modern-cpp-kafka` - are *wrappers over librdkafka* rather than independent +clients, so a second arm would price a header-only binding and nothing else. The AK core arm uses +librdkafka directly, which is the floor any of those wrappers sits on. + +The arm goes through **this module's client library**, never through hand-written gRPC. The Java +seed was written the other way first and had to be rewritten: speaking the protocol by hand proves +the *engine* works and says nothing about the *client library*, which is the artifact users actually +touch. Java keeps a hand-written arm as a control because one JVM can hold every arm at once and the +pair prices the library. C++ has nothing to compare one against, so two arms is the whole demo - +which is the contract everywhere except Java. + +The demo also creates the topic and pre-produces the backlog with librdkafka. That is not a hole in +the sidecar arm's claim: "the application does no Kafka I/O" is a statement about *that path*, and a +comparison needs both sides. + +## What the tables report + +Both replays print the same six columns in the same order, which every language mirrors: + +``` + arm elapsed msg/s records keys vs AK core + AK core (librdkafka) 4.1s 487 2,000 1,000 1.0x + pc-cpp-grpc (this client) 0.4s 4,761 2,000 1,000 9.8x +``` + +**`records` and `keys` are the deterministic pair**, and they are what makes the demos comparable +across languages at all. Elapsed and msg/s belong to the machine that ran them, so no two languages +can be compared on those; records must equal the target - a short arm is a failed arm, not a fast +one - and keys is the count of distinct record keys the arm actually observed, which shows the +backlog was spread rather than one key repeated. `bin/ci-demo-conformance.sh` relies on exactly that. + +Each arm counts its own keys from its own delivery path: the AK core arm from the `rd_kafka_message_t` +librdkafka handed it, and the sidecar arm from the `InboundRecord` the client library handed it. They +are not read off the seeding loop, which would make both columns a restatement of the input. + +**No latency, ever.** The backlog is pre-produced, so the workload is closed-loop and a per-record +timing would be flattered by however far an arm had fallen behind. + +## What is specific to C++ + +### There is no native mode + +The reference demo picks native or container and says which it chose. **C++ is container-only**, and +`run.sh --native` says so rather than failing as an unknown flag. The reason is the same one that +makes `bin/build-client.sh` build this language in an image: gRPC and protobuf arrive as system +*development packages* rather than as a versioned toolchain, so there is no C++ toolchain on a +developer's machine to run natively against. A demo that told a reader to install gRPC dev packages +first would not be a one-command demo. + +Two consequences follow, and both are visible in the output: + +- **The broker is always a compose sibling.** Java starts one with Testcontainers when no + `--bootstrap` was supplied; C++ has no Testcontainers, and a demo container is never granted the + host Docker socket (plan unit U35), so it could not start one anyway. Running the binary with no + broker address prints an explanation and exits 2 rather than running something different. +- **The image carries its own toolchain.** The runtime stage is built from the same Debian trixie + toolchain stage the binary was compiled in, so the dynamically linked demo is guaranteed the + libraries it was linked against. That is the opposite of `../Dockerfile`, whose whole point is + exporting *statically* linked artifacts that run off-image. + +### The sidecar arrives as a launcher script + +A foreign client library spawns a binary by absolute path; the sidecar is a JVM program. The image +installs a four-line launcher at `/app/sidecar/sidecar` that `exec`s the JVM - `exec` rather than a +fork, because the client holds the write end of that process's stdin and EOF there is the sidecar's +parent-death signal (KTD19). `PC_DEMO_SIDECAR` names it. + +**That is an environment variable and not an eighth flag**: the contract fixes the demo's flag list +at seven, and where the sidecar binary lives is a property of the image rather than of the run. + +The launcher also runs the sidecar at **WARN**. This image gives it an SLF4J binding it would +otherwise lack - without one the proxy prints `No SLF4J providers were found` and then nothing at +all - and at the binding's default level the sidecar's routine INFO buries the tables and prints +`bootstrap.servers` into the demo's own output, which the contract forbids. Warnings and errors +still come through, which was the point of adding the binding; `PC_SIDECAR_LOG_LEVEL=info` turns it +back up for a debugging run. + +### librdkafka's C API, not its C++ one + +The C++ binding has no admin client, so creating the topic would drop to the C API anyway. One API +in one file beats two. + +## Where the code is + +`src/` beside this file - `demo.cpp` (the arms), `demo_report.{h,cpp}` (the banner and the two +tables), `demo_options.{h,cpp}` (the flags, the environment and their precedence) and +`demo_broker.{h,cpp}` (librdkafka: the AK core consumer, the topic, the backlog). `CMakeLists.txt` is a project of its own rather than a target in +the module's, so the client library's build image never grows a Kafka client it does not use. + +`tests/` runs under the module's own sixty-line harness and **inside the image build**: a red test +fails the image, which fails the demo. `demo_options_test.cpp` covers the entry path - the +no-argument case, flags beating the environment, the fingerprint's silence about the broker address - +because that is where every failure this fan-out's demos have actually had lived. +`demo_report_test.cpp` covers the output, because the output *is* the contract: the banner's wording, +the column set and their order, an arm naming its client, and no latency anywhere. Neither needs a +broker, which is why both can run at image-build time. diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/docker-compose.yml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/docker-compose.yml new file mode 100644 index 0000000000..509b3e2787 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/docker-compose.yml @@ -0,0 +1,106 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE ONE-COMMAND PATH for the C++ demo (astubbs#242, plan unit U35, R20). +# +# THE BROKER IS A SIBLING, AND THAT IS A RULE RATHER THAN A LAYOUT CHOICE. A demo container is never +# granted the host Docker socket - a documented socket mount is root-equivalent host access taught as +# the normal way to run the product - so the demo is given an ADDRESS instead of a socket and reaches +# a broker on its own network. In C++ this is the only mode there is: the reference demo starts a +# broker with Testcontainers on its native path, and C++ has no native path and no Testcontainers. +# +# THE SIDECAR IS NOT A SERVICE HERE, deliberately. It would be easy to add one and it would teach the +# wrong thing: KTD41's product model is that the client library spawns and supervises the sidecar +# itself, so the user never installs, deploys or operates a process. A compose service would show a +# deployment the product does not ask for. The demo spawns it as a child process inside its own +# container, through the launcher the image installs. +# +# TWO WAYS IN, AND BOTH ARE SUPPORTED: +# +# cd && docker compose up # the plain one, defaults, nothing else needed +# demo/run.sh [--records N ...] # the same thing with the demo's flags +# +# `docker compose up` on its own is the path to keep working: it is what a reader who has never seen +# this repository will type. It needs no arguments, no environment and no wrapper script - the demo +# service runs with the demo's own defaults, and PC_DEMO_ARGS below is empty unless run.sh sets it. +# Add --build after changing any source, and `docker compose down` to remove the broker afterwards. +# +# The demo container exits when the run finishes; the broker keeps running until you stop it. run.sh +# passes --abort-on-container-exit --exit-code-from demo so that a scripted run stops on its own and +# reports the demo's exit code rather than compose's. + +name: pc-cpp-demo + +services: + broker: + # Pinned as a literal because a compose file cannot derive anything. It must track + # in the root pom - the mapping is CP major = AK major + 4, so AK 3.9.x means + # CP 7.9 - exactly as the Java demo's compose file does. + image: confluentinc/cp-kafka:7.9.0 + environment: + CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk + # QUIET. A demo is a thing someone WATCHES, and a KRaft broker at INFO emits hundreds of + # lines of its own configuration, controller elections and log-segment chatter - which buries + # the two tables the demo exists to show. Kafka's own warnings and errors still come through. + KAFKA_LOG4J_ROOT_LOGLEVEL: WARN + KAFKA_TOOLS_LOG4J_LOGLEVEL: WARN + # ROOT ALONE IS NOT ENOUGH, and this was measured rather than assumed: the cp-kafka image + # writes kafka=INFO, kafka.controller=TRACE and state.change.logger=TRACE into its own + # log4j config, and a named logger beats the root. With only the root set, one demo run + # interleaved 927 broker lines with 35 of the demo's own. These are the named loggers that + # actually produce the noise. + KAFKA_LOG4J_LOGGERS: "kafka=WARN,kafka.controller=WARN,kafka.server=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.network=WARN,kafka.request.logger=WARN,state.change.logger=WARN,org.apache.kafka=WARN" + KAFKA_NODE_ID: 1 + KAFKA_PROCESS_ROLES: broker,controller + # Bound to the service name, not 0.0.0.0: this image formats its KRaft storage with the same + # config it boots with, and the format step refuses a non-routable advertised address. + KAFKA_LISTENERS: PLAINTEXT://broker:9092,CONTROLLER://broker:9093 + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092 + KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT + KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093 + KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + # The demo forms one consumer group per arm, one after another, and the default three-second + # settling delay would be charged to every one of them. + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 500 + healthcheck: + # The demo must not start before the broker can serve an API request. Listing topics is the + # cheapest call that proves that, and it proves more than a port being open does. + test: ["CMD", "kafka-topics", "--bootstrap-server", "broker:9092", "--list"] + interval: 5s + timeout: 10s + retries: 30 + start_period: 10s + + demo: + build: + # The repository, because the demo needs the frozen proxy.proto from one module and the + # sidecar from a Maven reactor that cannot build without its parents and siblings. The root + # .dockerignore is what keeps this from uploading every target/ directory in the tree; the + # Dockerfile explains why that pair had to exist. + context: ../../.. + dockerfile: parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/Dockerfile + # A demo that has printed its tables is done. Without this, compose's default would restart a + # container that exited cleanly and the reader would watch the whole run happen again. + restart: "no" + depends_on: + broker: + condition: service_healthy + environment: + # Supplying this is what stops the demo asking for a broker it cannot start. + PC_DEMO_BOOTSTRAP: broker:9092 + # The flags demo/run.sh was called with, forwarded as one string. + PC_DEMO_ARGS: ${PC_DEMO_ARGS:-} + # EVERY OTHER DIAL, forwarded explicitly. Compose passes nothing it is not told to, so a + # missing line here is a dial the container silently ignores while the run still prints a + # fingerprint claiming otherwise. Caught across seven languages at once by + # bin/ci-demo-conformance.sh, which is why that check exists. Adding a flag means adding it + # here too. + PC_DEMO_RECORDS: ${PC_DEMO_RECORDS:-} + PC_DEMO_DELAY_MS: ${PC_DEMO_DELAY_MS:-} + PC_DEMO_CONCURRENCY: ${PC_DEMO_CONCURRENCY:-} + PC_DEMO_PARTITIONS: ${PC_DEMO_PARTITIONS:-} + PC_DEMO_REPLAY_FACTOR: ${PC_DEMO_REPLAY_FACTOR:-} + PC_DEMO_TOPIC: ${PC_DEMO_TOPIC:-} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/run.sh b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/run.sh new file mode 100755 index 0000000000..f3ae71528c --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/run.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# +# Copyright (C) 2026 Antony Stubbs and contributors +# +# The C++ demo: the same records through C++'s own Kafka client, and through C++ over the sidecar. +# One command, no setup. +# +# It mirrors the reference demo's interface - same flags, same defaults, same two tables out. The +# contract is parallel-consumer-proxy/demo/README.md; what is specific to C++ is demo/README.md +# beside this file. +set -euo pipefail + +cd "$(git rev-parse --show-toplevel)" + +DEMO_DIR="parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo" + +usage() { + cat <<'USAGE' +usage: demo/run.sh [options] + + --records N records in the comparison replay (default 2000) + --delay-ms N simulated work per record, ms (default 2) + --concurrency N max in-flight records (default 100) + --partitions N partitions on the demo topic (default 10) + --replay-factor N big replay = records x N; 1 skips (default 20) + --bootstrap ADDR an existing broker; omit to use the compose sibling + --topic NAME an existing topic; omit to create one + + --docker run in a container - the only mode C++ has, and the default + -h, --help this + +Needs Docker and nothing else: no C++ toolchain, no gRPC dev packages, no JDK. Everything is +built in the image, including the sidecar, which the client library spawns as a child process. + +Every flag also has an environment variable: --delay-ms is PC_DEMO_DELAY_MS. +USAGE +} + +# ${args[@]+...} rather than "${args[@]}": under `set -u`, bash 3.2 - which is what macOS ships, and +# what a first-time user runs this with - treats an EMPTY array expansion as an unbound variable and +# aborts. That fails exactly the no-argument case, which is the double-click case. +args=() +while [ $# -gt 0 ]; do + case "$1" in + --records|--delay-ms|--concurrency|--partitions|--replay-factor|--bootstrap|--topic) + if [ $# -lt 2 ]; then + echo "$1 needs a value" >&2 + exit 2 + fi + args+=("$1" "$2"); shift 2 ;; + --docker) shift ;; + --native) + # Answered rather than rejected as unknown, because a reader arriving from the Java + # demo will type it. + cat >&2 <<'NATIVE' +There is no native mode for the C++ demo, and that is the one place it diverges from the +reference. C++ needs gRPC, protobuf and librdkafka as system development packages rather than +as a versioned toolchain, so the container IS the toolchain here - the same reason +bin/build-client.sh builds this language in an image. Run it without --native. +NATIVE + exit 2 ;; + -h|--help) usage; exit 0 ;; + *) echo "unknown option: $1" >&2; usage >&2; exit 2 ;; + esac +done + +echo "Mode: container - C++ has no host toolchain path, so the demo brings its whole toolchain." + +if ! docker info >/dev/null 2>&1; then + echo "Docker is not running - the demo needs it for the broker and for its own image." >&2 + exit 1 +fi + +# The broker is a COMPOSE SIBLING and the demo container is never given the host Docker socket: a +# documented socket mount is root-equivalent host access taught as the normal way to run the product +# (plan unit U35). PC_DEMO_ARGS carries the flags through to the container, where the same parser +# reads them. +PC_DEMO_ARGS="${args[*]+${args[*]}}" \ + exec docker compose -f "$DEMO_DIR/docker-compose.yml" up \ + --build --abort-on-container-exit --exit-code-from demo diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/sidecar-launcher.sh b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/sidecar-launcher.sh new file mode 100755 index 0000000000..1d8b99f66e --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/sidecar-launcher.sh @@ -0,0 +1,48 @@ +#!/bin/sh +# +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE SIDECAR, AS AN EXECUTABLE THE CLIENT LIBRARY CAN SPAWN. The sidecar is a JVM program, and a +# foreign client library spawns a binary by absolute path - it knows nothing about classpaths. This +# launcher is the whole of the adaptation, and the demo image installs it at +# /app/sidecar/sidecar. Nobody runs it by hand: the client library spawns and supervises it (KTD41), +# which is why it is not a compose service either. +# +# `exec` IS LOAD-BEARING, NOT STYLE. The client holds the write end of this process's stdin and +# never writes to it, so EOF there is the sidecar's proof that its parent is gone (KTD19). A wrapper +# that FORKED the JVM and waited would keep that pipe open in a process the proxy is not watching, +# and a dead application would leave a live JVM holding group membership. Replacing this shell with +# the JVM keeps the pipe attached to the process that reads it. +# +# It takes no arguments of its own and passes on whatever it is given: the proxy REFUSES arguments, +# because every setting - bootstrap servers, credentials, ordering, concurrency, subscription - +# arrives connect-time over the protocol (R39). "$@" is here so that a caller who passes something +# gets the proxy's own refusal rather than a silent discard. +# +# AND THE SIDECAR IS QUIET, FOR THE SAME TWO REASONS THE BROKER IS. +# +# This image gives the sidecar an SLF4J binding, because without one the proxy prints +# `No SLF4J providers were found` and then nothing at all - see demo/Dockerfile. At slf4j-simple's +# default level that binding then emits several hundred lines per run, and two of them matter: +# +# 1. IT BURIES THE TABLES. Measured on a twenty-record run: the overwhelming majority of the +# demo's own output was the sidecar's routine INFO, including a full `ConsumerConfig values` +# dump per arm. A demo is a thing someone WATCHES, which is why every compose file here already +# pins the broker to WARN. +# 2. IT PRINTS THE BROKER ADDRESS. `bootstrap.servers = [...]` appears in those config dumps, and +# the sidecar's stderr is inherited by the demo's container - so it lands in the demo's own +# output, which the contract says must never carry that address (own-cluster mode puts a user's +# real broker there). `bin/ci-demo-conformance.sh` asserts exactly this and would fail on it. +# +# WARN, NOT OFF: the whole point of giving this image a binding was that a misconfigured broker +# should explain itself rather than become an unexplained hang. Warnings and errors still come +# through; only the routine goes. `PC_SIDECAR_LOG_LEVEL=info` turns it back up for a debugging run +# without editing anything. +set -eu + +PC_SIDECAR_HOME="${PC_SIDECAR_HOME:-/app/sidecar}" +PC_SIDECAR_LOG_LEVEL="${PC_SIDECAR_LOG_LEVEL:-warn}" + +exec java -cp "${PC_SIDECAR_HOME}/lib/*:${PC_SIDECAR_HOME}/classes" \ + "-Dorg.slf4j.simpleLogger.defaultLogLevel=${PC_SIDECAR_LOG_LEVEL}" \ + bz.stub.parallelconsumer.proxy.Main "$@" diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo.cpp b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo.cpp new file mode 100644 index 0000000000..5b780f8b79 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo.cpp @@ -0,0 +1,381 @@ +// Copyright (C) 2026 Antony Stubbs and contributors +// +// THE C++ DEMO (astubbs#242, plan unit U35). The same records through two arms: +// +// - AK core (librdkafka) - C++'s own Kafka client, one record at a time. "AK core" is always +// spelled out, because bare "core" reads as parallel-consumer-core (CONCEPTS.md) - +// and never left bare EITHER, because "AK core" is a category rather than a client +// and a reader cannot judge a comparison without knowing what produced it. +// - pc-cpp-grpc (this client) - this application as a FOREIGN CLIENT. The client library in the +// module above spawns the sidecar, receives records over a socket, runs the same +// sleep on them and reports outcomes back. On this path the application does no +// Kafka I/O at all: the sidecar owns the consumer, the producer, the group +// membership and the offsets. +// +// THE ARM GOES THROUGH THE CLIENT LIBRARY, NEVER THROUGH HAND-WRITTEN gRPC. The Java seed was +// written the other way first and had to be rewritten: speaking the protocol by hand proves the +// ENGINE works and says nothing about the client library, which is the artifact users actually +// touch. Java keeps a hand-written arm as a CONTROL because one JVM can hold every arm at once and +// the pair prices the library; C++ has nothing to compare it against, so two arms is the whole +// demo here - which is the contract everywhere except Java. +// +// NO LATENCY IS REPORTED, and that is a rule rather than an omission. The backlog is pre-produced, +// so the workload is closed-loop and per-record timings are flattered by however far an arm fell +// behind. Throughput is the only honest number this shape can produce. +// +// The contract this file keeps is parallel-consumer-proxy/demo/README.md. Read that first. + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "demo_broker.h" +#include "demo_options.h" +#include "demo_report.h" +#include "parallel_consumer_proxy_client.h" + +namespace pcp = parallelconsumer::proxy; + +namespace parallelconsumer::proxy::demo { +namespace { + +using Clock = std::chrono::steady_clock; + +/// No arm may take longer than this before the demo calls it stalled rather than slow. +constexpr std::chrono::minutes kArmBudget{10}; + +/// How often the sidecar arm looks up from waiting to ask whether the session is still alive. +constexpr std::chrono::milliseconds kProgressCheck{200}; + +/// THE ARM LABELS, AND WHY EACH CARRIES A LIBRARY NAME. +/// +/// "AK core" is a CATEGORY, not a client: it means "that language's own Kafka client", and the +/// answer differs in every language. A reader cannot judge a comparison without knowing what +/// produced it, so the role and the library are both said. In C++ the library is librdkafka - see +/// demo/README.md for why there is no second candidate worth running as its own arm. +/// +/// The sidecar arm is labelled with what DRIVES it, which is this module's client library rather +/// than hand-written gRPC. That distinction is the whole point of the arm, and until it was in the +/// label the output did not carry it. +constexpr const char* kAkCore = "AK core (librdkafka)"; +constexpr const char* kSidecarArm = "pc-cpp-grpc (this client)"; + +/// Where the sidecar binary is when nothing said otherwise - the path the demo image installs it +/// at. `PC_DEMO_SIDECAR` overrides it, which is what a reader running the binary outside that image +/// would set. +/// +/// IT IS NOT A FLAG, deliberately: the contract fixes the demo's flag list at seven, and this is a +/// property of the image rather than of the run. See demo/README.md. +constexpr const char* kDefaultSidecarPath = "/app/sidecar/sidecar"; + +/// A fresh group per arm per replay, so every arm reads the same records from the beginning. +std::string group_id(const std::string& arm) { + const auto now = std::chrono::system_clock::now().time_since_epoch(); + return "pc-demo-" + arm + "-" + + std::to_string(std::chrono::duration_cast(now).count()); +} + +ArmResult finished(const std::string& arm, Clock::time_point started, int processed, std::size_t unique_keys) { + const auto elapsed = Clock::now() - started; + std::cout << "=== " << arm << " finished: " << processed << " records over " << unique_keys + << " keys in " << std::chrono::duration_cast(elapsed).count() + << "ms ===" << std::endl; + return ArmResult{arm, elapsed, processed, static_cast(unique_keys)}; +} + +/// The simulated work, identical in both arms so they differ by transport and nothing else. +/// +/// A BLOCKING SLEEP IS THE RIGHT ANSWER IN C++, and the contract says so: the divergence it calls +/// out - a non-occupying wait - binds Python, whose client runs worker processes, and TypeScript, +/// whose single event loop a blocking sleep would stop. A C++ executor is a thread, and a sleeping +/// thread occupies nothing. +void simulated_work(int delay_ms) { + if (delay_ms > 0) { + std::this_thread::sleep_for(std::chrono::milliseconds(delay_ms)); + } +} + +/// The two tables, same columns and same order in every language. Rendered next door in +/// demo_report.cpp so the shape can be tested without a broker; printed here. +void report(const std::string& title, const std::vector& results, const ArmResult* baseline, + bool across_replays) { + std::cout << render(title, results, baseline, across_replays) << std::flush; +} + +/// The demo itself. +class ReferenceDemo { +public: + ReferenceDemo(DemoOptions options, std::string topic, std::string sidecar) + : options_(std::move(options)), + broker_(options_.bootstrap), + topic_(std::move(topic)), + sidecar_(std::move(sidecar)) {} + + void run() { + // Then the fingerprint, before anything runs: a number without its settings is not + // reproducible. The bootstrap address is deliberately not in it. + std::cout << "\nEffective configuration:\n " << options_.describe() << "\n topic = " << topic_ + << std::endl; + + broker_.ensure_topic(topic_, options_.partitions); + broker_.seed(topic_, 0, options_.records); + + std::vector small; + small.push_back(ak_core(options_.records)); + small.push_back(cpp_grpc(options_.records)); + report("Small replay - every arm over the same " + std::to_string(options_.records) + + " records (the comparison)", + small, &small.front(), false); + + if (!options_.big_replay_wanted()) { + std::cout << "\nBig replay skipped (--replay-factor " << options_.replay_factor << ")." + << std::endl; + return; + } + + const int total = options_.big_replay_records(); + broker_.seed(topic_, options_.records, total); + + // AK core is excluded here because it does not go parallel: it would need total * delayMs + // milliseconds to finish a backlog the sidecar arm clears in seconds, and a demo that makes + // a reader wait that long to learn nothing new is not worth the wall clock. + std::vector big; + big.push_back(cpp_grpc(total)); + // the unit is chosen so the figure is never zero - see the demo contract. + const long serial_millis = static_cast(total) * options_.delay_ms; + const std::string serial_cost = serial_millis >= 1000 + ? std::to_string(serial_millis / 1000) + "s" + : std::to_string(serial_millis) + "ms"; + report("Big replay - " + std::to_string(total) + " records, parallel arms only (AK core is serial" + + " and would take " + serial_cost + "+)", + big, &small.front(), true); + } + +private: + /// The serial arm: C++'s own Kafka client, one record at a time, the same sleep. + ArmResult ak_core(int target) { + std::cout << "\n=== " << kAkCore << " starting over " << target << " records ===" << std::endl; + KafkaHandle consumer = broker_.subscribed_consumer(topic_, group_id("ak-core")); + + int processed = 0; + // THE KEYS THIS ARM ACTUALLY SAW, which is half of what turns the table from an assertion + // that work happened into a demonstration of it. A set rather than a counter because + // "unique" is the claim; a counter would report the same number as `processed` and prove + // nothing about the backlog being spread. + std::set keys; + // The clock starts AFTER the consumer is built and stops before it closes, because this arm + // is the denominator of every ratio in both tables and no other arm charges itself for + // client construction or teardown. + const auto started = Clock::now(); + const auto deadline = started + kArmBudget; + while (processed < target) { + if (Clock::now() > deadline) { + throw std::runtime_error(std::string(kAkCore) + " stalled at " + std::to_string(processed) + + " of " + std::to_string(target)); + } + rd_kafka_message_t* message = rd_kafka_consumer_poll(consumer.get(), 500); + if (message == nullptr) { + continue; + } + if (message->err != RD_KAFKA_RESP_ERR_NO_ERROR) { + // Partition EOF and the periodic informational errors are not records and not + // failures; anything else is worth saying out loud rather than silently retrying + // until the budget runs out. + if (message->err != RD_KAFKA_RESP_ERR__PARTITION_EOF) { + std::cerr << "[AK core] " << rd_kafka_message_errstr(message) << '\n'; + } + rd_kafka_message_destroy(message); + continue; + } + simulated_work(options_.delay_ms); + ++processed; + // The key as bytes and by length, because a Kafka key is neither a C string nor + // guaranteed to be text. A null key is legal and is not a key, so it is not counted - + // this demo seeds `key-` on every record, which makes the + // count `min(records, key space)` and the same in every language. + if (message->key != nullptr) { + keys.emplace(static_cast(message->key), message->key_len); + } + rd_kafka_message_destroy(message); + } + ArmResult result = finished(kAkCore, started, processed, keys.size()); + rd_kafka_consumer_close(consumer.get()); + return result; + } + + /// The sidecar arm: the same work, reached through THIS MODULE'S CLIENT LIBRARY. + /// + /// The library spawns the sidecar as a child process and supervises it, so the reader installs, + /// deploys and operates nothing (KTD41). That is why the sidecar is not a compose service in + /// the file beside this one. + ArmResult cpp_grpc(int target) { + std::cout << "\n=== " << kSidecarArm << " starting over " << target << " records ===" + << std::endl; + + pcp::ClientOptions client_options; + client_options.sidecar_path = sidecar_; + client_options.topics = {topic_}; + client_options.max_concurrency = options_.max_concurrency; + client_options.ordering = pcp::ProcessingOrder::Unordered; + client_options.kafka_properties = broker_.consumer_properties(group_id("pc-cpp-grpc")); + client_options.instance_tag = "pc-cpp-demo"; + // The library says nothing until an application asks it to, so the demo asks - on stderr, + // leaving stdout to the tables. Debug is per-record and stays off. + client_options.logger = [](pcp::LogLevel level, const std::string& line) { + if (level != pcp::LogLevel::Debug) { + std::cerr << "[client " << pcp::to_string(level) << "] " << line << '\n'; + } + }; + + std::mutex mutex; + std::condition_variable changed; + int processed = 0; + // GUARDED BY THE SAME MUTEX AS `processed`, because unlike the AK core arm's set this one is + // written by every executor thread at once. An unsynchronised std::set here is a data race + // that would usually look like a slightly wrong key count rather than like a crash. + std::set keys; + + std::unique_ptr client = pcp::Client::connect(client_options); + // The clock starts once the session is open, for the same reason the AK core arm's starts + // once its consumer is built: neither arm charges itself for start-up. + const auto started = Clock::now(); + client->poll([&](const pcp::InboundRecord& record) { + simulated_work(options_.delay_ms); + { + const std::lock_guard lock(mutex); + ++processed; + if (record.key.has_value()) { + keys.insert(*record.key); + } + } + changed.notify_all(); + return pcp::Outcome::success(); + }); + + std::shared_future session_end = client->session_end(); + const auto deadline = started + kArmBudget; + for (;;) { + std::unique_lock lock(mutex); + if (changed.wait_for(lock, kProgressCheck, [&] { return processed >= target; })) { + break; + } + const int so_far = processed; + lock.unlock(); + + // Reaching the target is not the only thing that can end this wait. A session that + // failed or completed early must be reported as exactly that: without this check a + // broken run would sit here until the budget expired and then blame a stall. + if (session_end.wait_for(std::chrono::seconds(0)) == std::future_status::ready) { + session_end.get(); // rethrows the fault, when the session died of one + throw std::runtime_error(std::string(kSidecarArm) + " ended early at " + + std::to_string(so_far) + " of " + std::to_string(target) + + " - the session ended before the backlog did"); + } + if (Clock::now() > deadline) { + throw std::runtime_error(std::string(kSidecarArm) + " stalled at " + std::to_string(so_far) + + " of " + std::to_string(target)); + } + } + + int completed = 0; + std::size_t distinct_keys = 0; + { + const std::lock_guard lock(mutex); + completed = processed; + distinct_keys = keys.size(); + } + ArmResult result = finished(kSidecarArm, started, completed, distinct_keys); + // After the clock stops: the drain and the sidecar's reaping are teardown, and no other arm + // charges itself for that either. + client->shutdown(); + return result; + } + + DemoOptions options_; + DemoBroker broker_; + std::string topic_; + std::string sidecar_; +}; + +/// The sidecar binary this demo spawns, and proof that it is there before anything is timed. +std::string resolve_sidecar(const Environment& environment) { + const std::optional configured = environment("PC_DEMO_SIDECAR"); + const std::string path = configured.value_or(kDefaultSidecarPath); + if (path.empty() || path.front() != '/') { + throw std::runtime_error("PC_DEMO_SIDECAR must be an absolute path, got '" + path + "'"); + } + if (::access(path.c_str(), X_OK) != 0) { + throw std::runtime_error( + "no executable sidecar at " + path + + " - the demo image installs one there; outside it, set PC_DEMO_SIDECAR to the " + "sidecar launcher. The client library spawns it; you never start it yourself."); + } + return path; +} + +} // namespace +} // namespace parallelconsumer::proxy::demo + +int main(int argc, char** argv) { + namespace demo = parallelconsumer::proxy::demo; + + const std::vector args(argv + 1, argv + argc); + + // THE VERY FIRST THING PRINTED, WHATEVER HAPPENS NEXT - before the usage text, before a refused + // flag, before the fingerprint. A reader who runs this must be told what they are looking at: + // the demo used to open on a configuration dial, which names neither the product nor the point. + std::cout << demo::banner(); + + if (demo::DemoOptions::help_requested(args)) { + std::cout << demo::usage(); + return 0; + } + + const demo::Environment environment = demo::process_environment(); + demo::DemoOptions options; + try { + options = demo::DemoOptions::parse(args, environment); + } catch (const std::exception& refused) { + // A misspelled flag must not be reported as a result for settings nobody asked for. + std::cerr << refused.what() << '\n' << demo::usage(); + return 2; + } + + if (options.bootstrap.empty()) { + std::cerr << "\nNo broker address. The C++ demo never starts one of its own - it runs inside " + "its\nown container, which is never granted the host Docker socket, so the broker " + "is a\ncompose sibling. Run demo/run.sh, which starts both, or pass --bootstrap / " + "set\nPC_DEMO_BOOTSTRAP to reach a cluster you already have.\n"; + return 2; + } + + try { + const std::string sidecar = demo::resolve_sidecar(environment); + std::string topic = options.topic; + if (topic.empty()) { + topic = "pc-demo-" + std::to_string(std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()) + .count()); + } + demo::ReferenceDemo(options, topic, sidecar).run(); + } catch (const std::exception& failed) { + // Loudly, and with a non-zero status: a demo whose numbers ten other languages copy must + // never exit 0 having printed a plausible table for a run that did not happen. + std::cerr << "\nThe demo failed: " << failed.what() << '\n'; + return 1; + } + return 0; +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_broker.cpp b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_broker.cpp new file mode 100644 index 0000000000..db56d3111d --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_broker.cpp @@ -0,0 +1,252 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +#include "demo_broker.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace parallelconsumer::proxy::demo { +namespace { + +/// The key space the seeded records spread over. Ordering is UNORDERED in every arm, so this +/// changes nothing today; it exists so that a KEY-ordered lane added later has more than one key to +/// shard across, rather than needing the seeding rewritten first. The same number as the reference +/// demo's, so the two are seeding the same shape of backlog. +constexpr int kKeySpace = 1000; + +constexpr int kAdminTimeoutMs = 30000; +constexpr int kMetadataTimeoutMs = 10000; +constexpr int kFlushTimeoutMs = 120000; + +/// What the delivery reports said, collected off librdkafka's own poll thread. +struct DeliveryOutcome { + std::mutex mutex; + std::string first_failure; + std::atomic failed{0}; + + void record(const std::string& reason) { + const std::lock_guard lock(mutex); + ++failed; + if (first_failure.empty()) { + first_failure = reason; + } + } +}; + +void delivery_report(rd_kafka_t* /*handle*/, const rd_kafka_message_t* message, void* opaque) { + if (message->err == RD_KAFKA_RESP_ERR_NO_ERROR) { + return; + } + static_cast(opaque)->record(rd_kafka_err2str(message->err)); +} + +/// Builds a configuration from a property map, refusing anything librdkafka will not take. +/// +/// The returned configuration is OWNED BY THE CALLER until `rd_kafka_new` consumes it - which it +/// does only on success, hence the explicit destroy on the failure path in `handle()`. +rd_kafka_conf_t* configuration(const std::map& properties) { + rd_kafka_conf_t* conf = rd_kafka_conf_new(); + char problem[512]; + for (const auto& property : properties) { + if (rd_kafka_conf_set(conf, property.first.c_str(), property.second.c_str(), problem, + sizeof(problem)) != RD_KAFKA_CONF_OK) { + rd_kafka_conf_destroy(conf); + // The KEY only. A property map is where credentials live, and the natural rendering of + // a configuration error is exactly how one reaches a log file. + throw std::runtime_error("librdkafka rejected the property '" + property.first + "': " + problem); + } + } + return conf; +} + +KafkaHandle handle(rd_kafka_type_t type, rd_kafka_conf_t* conf) { + char problem[512]; + rd_kafka_t* created = rd_kafka_new(type, conf, problem, sizeof(problem)); + if (created == nullptr) { + rd_kafka_conf_destroy(conf); + throw std::runtime_error(std::string("could not create the Kafka client: ") + problem); + } + return KafkaHandle(created); +} + +int partitions_of(rd_kafka_t* client, const std::string& topic) { + rd_kafka_topic_t* subject = rd_kafka_topic_new(client, topic.c_str(), nullptr); + if (subject == nullptr) { + throw std::runtime_error("could not describe the existing topic " + topic); + } + const struct rd_kafka_metadata* metadata = nullptr; + const rd_kafka_resp_err_t error = rd_kafka_metadata(client, 0, subject, &metadata, kMetadataTimeoutMs); + rd_kafka_topic_destroy(subject); + if (error != RD_KAFKA_RESP_ERR_NO_ERROR) { + throw std::runtime_error("could not describe the existing topic " + topic + ": " + + rd_kafka_err2str(error)); + } + int found = 0; + for (int index = 0; index < metadata->topic_cnt; ++index) { + if (topic == metadata->topics[index].topic) { + found = metadata->topics[index].partition_cnt; + } + } + rd_kafka_metadata_destroy(metadata); + return found; +} + +} // namespace + +DemoBroker::DemoBroker(std::string bootstrap) : bootstrap_(std::move(bootstrap)) { + if (bootstrap_.empty()) { + throw std::runtime_error("no broker address"); + } +} + +std::map DemoBroker::consumer_properties(const std::string& group_id) const { + return {{"bootstrap.servers", bootstrap_}, + {"group.id", group_id}, + {"auto.offset.reset", "earliest"}, + {"enable.auto.commit", "false"}}; +} + +void DemoBroker::ensure_topic(const std::string& topic, int partitions) const { + // A producer handle, because librdkafka's admin operations hang off an ordinary client and a + // producer is the one that needs no group. + KafkaHandle admin = handle(RD_KAFKA_PRODUCER, configuration({{"bootstrap.servers", bootstrap_}})); + + char problem[512]; + rd_kafka_NewTopic_t* requested = + rd_kafka_NewTopic_new(topic.c_str(), partitions, 1, problem, sizeof(problem)); + if (requested == nullptr) { + throw std::runtime_error("could not describe the topic to create: " + std::string(problem)); + } + + rd_kafka_AdminOptions_t* options = + rd_kafka_AdminOptions_new(admin.get(), RD_KAFKA_ADMIN_OP_CREATETOPICS); + rd_kafka_AdminOptions_set_request_timeout(options, kAdminTimeoutMs, problem, sizeof(problem)); + rd_kafka_queue_t* replies = rd_kafka_queue_new(admin.get()); + rd_kafka_CreateTopics(admin.get(), &requested, 1, options, replies); + + rd_kafka_event_t* reply = rd_kafka_queue_poll(replies, kAdminTimeoutMs + 5000); + + // Everything below reads out of `reply`, so the strings are copied before any of these are + // destroyed - a librdkafka error string does not outlive its event. + std::string failure; + bool already_exists = false; + if (reply == nullptr) { + failure = "the broker did not answer the create-topic request within " + + std::to_string(kAdminTimeoutMs / 1000) + "s"; + } else if (rd_kafka_event_error(reply) != RD_KAFKA_RESP_ERR_NO_ERROR) { + failure = rd_kafka_event_error_string(reply); + } else { + std::size_t count = 0; + const rd_kafka_topic_result_t** results = + rd_kafka_CreateTopics_result_topics(rd_kafka_event_CreateTopics_result(reply), &count); + for (std::size_t index = 0; index < count; ++index) { + const rd_kafka_resp_err_t error = rd_kafka_topic_result_error(results[index]); + if (error == RD_KAFKA_RESP_ERR_TOPIC_ALREADY_EXISTS) { + already_exists = true; + } else if (error != RD_KAFKA_RESP_ERR_NO_ERROR) { + failure = rd_kafka_topic_result_error_string(results[index]); + } + } + } + + if (reply != nullptr) { + rd_kafka_event_destroy(reply); + } + rd_kafka_queue_destroy(replies); + rd_kafka_AdminOptions_destroy(options); + rd_kafka_NewTopic_destroy(requested); + + if (!failure.empty()) { + throw std::runtime_error("could not create the demo topic " + topic + ": " + failure); + } + if (!already_exists) { + std::cout << "Created topic " << topic << " with " << partitions << " partitions" << std::endl; + return; + } + + // Reusing a topic silently is fine; reusing one with a DIFFERENT partition count is not, + // because the effective-configuration block would print a --partitions value that never + // applied - and that block is the demo's whole reproducibility promise. + const int existing = partitions_of(admin.get(), topic); + if (existing != partitions) { + throw std::runtime_error("topic " + topic + " already exists with " + std::to_string(existing) + + " partitions, but this run asked for " + std::to_string(partitions) + + " - pass --topic to name a fresh one, or --partitions " + + std::to_string(existing)); + } + std::cout << "Topic " << topic << " already exists with the requested " << partitions + << " partitions, reusing it" << std::endl; +} + +void DemoBroker::seed(const std::string& topic, int from, int to) const { + if (to <= from) { + return; + } + + DeliveryOutcome outcome; + rd_kafka_conf_t* conf = configuration({{"bootstrap.servers", bootstrap_}, {"linger.ms", "20"}}); + rd_kafka_conf_set_opaque(conf, &outcome); + rd_kafka_conf_set_dr_msg_cb(conf, delivery_report); + KafkaHandle producer = handle(RD_KAFKA_PRODUCER, conf); + + std::cout << "Producing records " << from << " to " << to << "..." << std::endl; + for (int index = from; index < to; ++index) { + const std::string key = "key-" + std::to_string(index % kKeySpace); + const std::string value = "record-" + std::to_string(index); + for (;;) { + const rd_kafka_resp_err_t error = rd_kafka_producev( + producer.get(), RD_KAFKA_V_TOPIC(topic.c_str()), + RD_KAFKA_V_KEY(key.data(), key.size()), + RD_KAFKA_V_VALUE(const_cast(value.data()), value.size()), + RD_KAFKA_V_MSGFLAGS(RD_KAFKA_MSG_F_COPY), RD_KAFKA_V_END); + if (error == RD_KAFKA_RESP_ERR_NO_ERROR) { + break; + } + if (error != RD_KAFKA_RESP_ERR__QUEUE_FULL) { + throw std::runtime_error("the demo could not seed its backlog: " + + std::string(rd_kafka_err2str(error))); + } + // A full local queue is back-pressure, not a failure: serve the delivery reports that + // will drain it and try the same record again. + rd_kafka_poll(producer.get(), 100); + } + } + + const rd_kafka_resp_err_t flushed = rd_kafka_flush(producer.get(), kFlushTimeoutMs); + if (flushed != RD_KAFKA_RESP_ERR_NO_ERROR) { + throw std::runtime_error("the demo could not seed its backlog: " + + std::string(rd_kafka_err2str(flushed)) + ", " + + std::to_string(rd_kafka_outq_len(producer.get())) + + " records still unsent"); + } + if (outcome.failed.load() > 0) { + throw std::runtime_error("the demo could not seed its backlog: " + std::to_string(outcome.failed.load()) + + " records were not delivered, the first because " + + outcome.first_failure); + } + std::cout << "Produced " << (to - from) << " records" << std::endl; +} + +KafkaHandle DemoBroker::subscribed_consumer(const std::string& topic, const std::string& group_id) const { + KafkaHandle consumer = handle(RD_KAFKA_CONSUMER, configuration(consumer_properties(group_id))); + // Routes the main queue onto the consumer queue, so rd_kafka_consumer_poll sees everything. + rd_kafka_poll_set_consumer(consumer.get()); + + rd_kafka_topic_partition_list_t* subscription = rd_kafka_topic_partition_list_new(1); + rd_kafka_topic_partition_list_add(subscription, topic.c_str(), RD_KAFKA_PARTITION_UA); + const rd_kafka_resp_err_t error = rd_kafka_subscribe(consumer.get(), subscription); + rd_kafka_topic_partition_list_destroy(subscription); + if (error != RD_KAFKA_RESP_ERR_NO_ERROR) { + throw std::runtime_error("could not subscribe to " + topic + ": " + rd_kafka_err2str(error)); + } + return consumer; +} + +} // namespace parallelconsumer::proxy::demo diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_broker.h b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_broker.h new file mode 100644 index 0000000000..0836c9f3a4 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_broker.h @@ -0,0 +1,115 @@ +// Copyright (C) 2026 Antony Stubbs and contributors +// +// C++'S OWN KAFKA CLIENT, which in C++ means librdkafka - there is no other. It is here for two +// jobs, and only one of them is an arm of the comparison: +// +// 1. the AK core arm - a consumer, one record at a time, the same sleep every other arm runs; +// 2. the fixture - creating the topic and pre-producing the backlog every arm then replays. +// +// The second is not a hole in the sidecar's claim. "The application does no Kafka I/O on the +// sidecar path" is a statement about THAT PATH, not about this process: a comparison needs both +// sides, so the same binary that runs the foreign-client arm also seeds the topic with an ordinary +// Kafka client. A genuinely foreign application carries no Kafka client library at all, which is +// the property the sidecar arm stands in for. +// +// THE C API RATHER THAN librdkafka's C++ ONE, deliberately: the C++ binding has no admin client, so +// creating the topic would drop to the C API anyway, and one API in one file beats two. + +#ifndef PARALLELCONSUMER_PROXY_DEMO_BROKER_H +#define PARALLELCONSUMER_PROXY_DEMO_BROKER_H + +#include + +#include +#include +#include + +namespace parallelconsumer::proxy::demo { + +/// An owned `rd_kafka_t`. librdkafka's handles are C, and every early return in this demo would +/// otherwise leak one. +class KafkaHandle { +public: + KafkaHandle() = default; + explicit KafkaHandle(rd_kafka_t* handle) : handle_(handle) {} + KafkaHandle(const KafkaHandle&) = delete; + KafkaHandle& operator=(const KafkaHandle&) = delete; + KafkaHandle(KafkaHandle&& other) noexcept : handle_(other.handle_) { other.handle_ = nullptr; } + KafkaHandle& operator=(KafkaHandle&& other) noexcept { + if (this != &other) { + reset(); + handle_ = other.handle_; + other.handle_ = nullptr; + } + return *this; + } + ~KafkaHandle() { reset(); } + + [[nodiscard]] rd_kafka_t* get() const { return handle_; } + + void reset() { + if (handle_ != nullptr) { + rd_kafka_destroy(handle_); + handle_ = nullptr; + } + } + +private: + rd_kafka_t* handle_ = nullptr; +}; + +/// The broker this demo reads from, however the reader got here. +/// +/// ONE WAY IN, AND THAT IS THE DIVERGENCE THIS LANGUAGE HAS. The Java reference starts a broker +/// with Testcontainers when none was supplied; C++ has no Testcontainers, and the demo runs only +/// inside its own container, where a demo container is never granted the host Docker socket +/// (plan unit U35) and so could not start one anyway. The address always arrives from outside - +/// from the compose sibling, or from `--bootstrap` - and a missing one is an error with an +/// explanation rather than a silently different run. See demo/README.md. +/// +/// The same door serves own-cluster mode, where the address is the user's real cluster, so nothing +/// here logs or echoes it. +class DemoBroker { +public: + explicit DemoBroker(std::string bootstrap); + + /// Creates the demo's topic, tolerating one a previous run left behind - but never one with a + /// DIFFERENT partition count, because the fingerprint would then print a `--partitions` value + /// that never applied, and that block is the demo's whole reproducibility promise. + /// + /// @throws std::runtime_error + void ensure_topic(const std::string& topic, int partitions) const; + + /// Produces the backlog every arm then replays. + /// + /// PRE-PRODUCED RATHER THAN PRODUCED ALONGSIDE THE ARMS, which is what makes the workload + /// closed-loop - and in turn why no arm reports latency. A per-record timing here would be + /// flattered by however far an arm had fallen behind, so throughput is the only honest number + /// this shape can produce. + /// + /// @throws std::runtime_error if any record failed to reach the broker. A discarded delivery + /// report would let the demo report a full backlog, run every arm against a short one, + /// and print numbers for a workload that never existed. + void seed(const std::string& topic, int from, int to) const; + + /// The Kafka properties every arm needs to reach this broker. + /// + /// `enable.auto.commit=false` is set for the same reason the Java reference sets it: Parallel + /// Consumer owns offset commits and refuses a consumer that also commits for itself. The + /// sidecar forces the setting whatever the map says, so on the sidecar arm this line is + /// belt-and-braces; the AK core arm below genuinely needs it, since it commits nothing and a + /// half-auto-committed run would replay a different backlog next time. + [[nodiscard]] std::map consumer_properties(const std::string& group_id) const; + + /// A subscribed consumer for the AK core arm, built from consumer_properties(). + /// + /// @throws std::runtime_error + [[nodiscard]] KafkaHandle subscribed_consumer(const std::string& topic, const std::string& group_id) const; + +private: + std::string bootstrap_; +}; + +} // namespace parallelconsumer::proxy::demo + +#endif // PARALLELCONSUMER_PROXY_DEMO_BROKER_H diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_options.cpp b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_options.cpp new file mode 100644 index 0000000000..d9c5b48529 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_options.cpp @@ -0,0 +1,177 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +#include "demo_options.h" + +#include +#include +#include +#include +#include +#include + +namespace parallelconsumer::proxy::demo { +namespace { + +/// Trims the whitespace an environment variable set from a compose file routinely carries. An +/// empty value after trimming is treated as unset, so `PC_DEMO_TOPIC=` in a compose file means +/// "the demo names its own" rather than "the empty topic". +std::string trim(const std::string& raw) { + const std::size_t first = raw.find_first_not_of(" \t\r\n"); + if (first == std::string::npos) { + return {}; + } + const std::size_t last = raw.find_last_not_of(" \t\r\n"); + return raw.substr(first, last - first + 1); +} + +int number(const std::string& source, const std::string& raw) { + const std::string text = trim(raw); + try { + std::size_t consumed = 0; + const long parsed = std::stol(text, &consumed); + if (consumed != text.size() || parsed > std::numeric_limits::max() + || parsed < std::numeric_limits::min()) { + throw std::invalid_argument("out of range"); + } + return static_cast(parsed); + } catch (const std::exception&) { + throw std::invalid_argument(source + " needs a whole number, got '" + raw + "'"); + } +} + +int positive(const std::string& source, const std::string& raw) { + const int parsed = number(source, raw); + if (parsed < 1) { + throw std::invalid_argument(source + " must be at least 1, got " + std::to_string(parsed)); + } + return parsed; +} + +int non_negative(const std::string& source, const std::string& raw) { + const int parsed = number(source, raw); + if (parsed < 0) { + throw std::invalid_argument(source + " must not be negative, got " + std::to_string(parsed)); + } + return parsed; +} + +std::optional from_environment(const Environment& environment, const std::string& suffix) { + const std::optional raw = environment(std::string(kEnvironmentPrefix) + suffix); + if (!raw.has_value()) { + return std::nullopt; + } + const std::string trimmed = trim(*raw); + return trimmed.empty() ? std::nullopt : std::optional(trimmed); +} + +const std::string& value(const std::vector& args, std::size_t index, const std::string& flag) { + if (index >= args.size()) { + throw std::invalid_argument(flag + " needs a value"); + } + return args[index]; +} + +} // namespace + +Environment process_environment() { + return [](const std::string& name) -> std::optional { + const char* raw = std::getenv(name.c_str()); + return raw == nullptr ? std::nullopt : std::optional(raw); + }; +} + +bool DemoOptions::help_requested(const std::vector& args) { + for (const std::string& argument : args) { + if (argument == "-h" || argument == "--help") { + return true; + } + } + return false; +} + +DemoOptions DemoOptions::parse(const std::vector& args, const Environment& environment) { + DemoOptions options; + + // The environment first, so a flag can override it below. + if (const auto raw = from_environment(environment, "RECORDS")) { + options.records = positive("PC_DEMO_RECORDS", *raw); + } + if (const auto raw = from_environment(environment, "DELAY_MS")) { + options.delay_ms = non_negative("PC_DEMO_DELAY_MS", *raw); + } + if (const auto raw = from_environment(environment, "CONCURRENCY")) { + options.max_concurrency = positive("PC_DEMO_CONCURRENCY", *raw); + } + if (const auto raw = from_environment(environment, "PARTITIONS")) { + options.partitions = positive("PC_DEMO_PARTITIONS", *raw); + } + if (const auto raw = from_environment(environment, "REPLAY_FACTOR")) { + options.replay_factor = non_negative("PC_DEMO_REPLAY_FACTOR", *raw); + } + if (const auto raw = from_environment(environment, "BOOTSTRAP")) { + options.bootstrap = *raw; + } + if (const auto raw = from_environment(environment, "TOPIC")) { + options.topic = *raw; + } + + for (std::size_t index = 0; index < args.size(); ++index) { + const std::string& flag = args[index]; + if (flag == "--records") { + options.records = positive(flag, value(args, ++index, flag)); + } else if (flag == "--delay-ms") { + options.delay_ms = non_negative(flag, value(args, ++index, flag)); + } else if (flag == "--concurrency") { + options.max_concurrency = positive(flag, value(args, ++index, flag)); + } else if (flag == "--partitions") { + options.partitions = positive(flag, value(args, ++index, flag)); + } else if (flag == "--replay-factor") { + // 1 or less skips the big replay, so this one is allowed to be zero + options.replay_factor = non_negative(flag, value(args, ++index, flag)); + } else if (flag == "--bootstrap") { + options.bootstrap = value(args, ++index, flag); + } else if (flag == "--topic") { + options.topic = value(args, ++index, flag); + } else { + throw std::invalid_argument("unknown option: " + flag); + } + } + + // Checked as a long here rather than trusted as an int later: records * replay_factor overflows + // silently, and a wrapped value turns the big replay into a tiny one that still prints a + // confident throughput figure. + const long big = static_cast(options.records) * (options.replay_factor < 1 ? 1 : options.replay_factor); + if (big > std::numeric_limits::max()) { + throw std::invalid_argument("--records times --replay-factor is " + std::to_string(big) + + ", which is more records than the demo can count; lower one of them"); + } + return options; +} + +int DemoOptions::big_replay_records() const { + return records * (replay_factor < 1 ? 1 : replay_factor); +} + +bool DemoOptions::big_replay_wanted() const { return replay_factor > 1; } + +std::string DemoOptions::describe() const { + std::ostringstream out; + out << "records = " << records << "\n delayMs = " << delay_ms << "\n maxConcurrency = " << max_concurrency + << "\n partitions = " << partitions << "\n replayFactor = " << replay_factor; + return out.str(); +} + +std::string usage() { + return "\nusage: demo/run.sh [options]\n" + " --records N records in the comparison replay (default 2000)\n" + " --delay-ms N simulated work per record, ms (default 2)\n" + " --concurrency N max in-flight records (default 100)\n" + " --partitions N partitions on the demo topic (default 10)\n" + " --replay-factor N big replay = records x N; 1 skips (default 20)\n" + " --bootstrap ADDR an existing broker; omit to use the compose sibling\n" + " --topic NAME an existing topic; omit to create one\n" + "\nEvery flag has an environment variable: --delay-ms is PC_DEMO_DELAY_MS.\n" + "Flags beat the environment beats the defaults.\n"; +} + +} // namespace parallelconsumer::proxy::demo diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_options.h b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_options.h new file mode 100644 index 0000000000..05a48c5d39 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_options.h @@ -0,0 +1,83 @@ +// Copyright (C) 2026 Antony Stubbs and contributors +// +// The demo's dials, and C++'s copy of the interface every per-language demo mirrors +// (parallel-consumer-proxy/demo/README.md). +// +// FLAGS BEAT THE ENVIRONMENT BEATS THE DEFAULTS - the ordinary convention, and the one the Java +// reference states rather than implies: a container passes configuration by environment while a +// person at a terminal passes flags, and both must be able to override the other's layer. +// +// R39 DOES NOT GOVERN A DEMO. R39 constrains how configuration reaches the PROXY - connect-time, +// over the protocol, never by argv or environment. A demo is an application, so its own flags are +// not a violation of it. Without this note somebody reads `--records` as breaking the plan's rule +// and deletes it. + +#ifndef PARALLELCONSUMER_PROXY_DEMO_OPTIONS_H +#define PARALLELCONSUMER_PROXY_DEMO_OPTIONS_H + +#include +#include +#include +#include + +namespace parallelconsumer::proxy::demo { + +/// Where environment values come from. Injected rather than read from `getenv` at the point of use, +/// so the precedence rule can be tested without mutating this process's own environment. +using Environment = std::function(const std::string&)>; + +/// This process's environment, for the one caller that should use it. +Environment process_environment(); + +/// The prefix every variable this demo reads carries, so a reader can grep one string. +inline constexpr const char* kEnvironmentPrefix = "PC_DEMO_"; + +/// The demo's effective configuration. +struct DemoOptions { + int records = 2000; + int delay_ms = 2; + int max_concurrency = 100; + int partitions = 10; + int replay_factor = 20; + + /// The broker to use. Empty means none was supplied - which this demo cannot recover from, and + /// says so; see the README's divergence note. Never printed, never logged. + std::string bootstrap; + + /// The topic to use. Empty means the demo names its own. + std::string topic; + + /// Whether the caller asked for the usage text rather than a run. + /// + /// Answered here rather than only in run.sh because the script is not the only way in: + /// `docker compose run demo --help` reaches this binary directly. + static bool help_requested(const std::vector& args); + + /// Parses the command line over the environment over the defaults. + /// + /// @throws std::invalid_argument on an unknown flag, a missing value, or a value that is not a + /// number in range. A demo that silently ignored a misspelled flag would report numbers + /// for settings the user did not ask for. + static DemoOptions parse(const std::vector& args, const Environment& environment); + + /// The records the big replay consumes in total, including the small replay's own. + [[nodiscard]] int big_replay_records() const; + + /// Whether the big replay is worth running at all; a factor of 1 or less skips it. + [[nodiscard]] bool big_replay_wanted() const; + + /// The effective configuration, printed before the run. + /// + /// A number without its settings is not reproducible, so this is part of the contract every + /// language's demo keeps rather than a debugging aid. THE BOOTSTRAP ADDRESS IS DELIBERATELY + /// ABSENT: own-cluster mode puts a user's real broker there, and the credential-hygiene rule + /// that binds the client library binds a demo too. + [[nodiscard]] std::string describe() const; +}; + +/// The usage text, identical in wording to the reference demo's. +std::string usage(); + +} // namespace parallelconsumer::proxy::demo + +#endif // PARALLELCONSUMER_PROXY_DEMO_OPTIONS_H diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_report.cpp b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_report.cpp new file mode 100644 index 0000000000..0de48ca326 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_report.cpp @@ -0,0 +1,79 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +#include "demo_report.h" + +#include +#include +#include + +namespace parallelconsumer::proxy::demo { +namespace { + +/// The banner's rule, the width the contract prints it at. +constexpr int kRuleWidth = 64; + +/// The arm column. Wide enough for the longest label this language has - a label now carries the +/// library that produced the row, so "pc-cpp-grpc (this client)" is the size to fit. +/// +/// COLUMN WIDTH IS DELIBERATELY NOT CONTRACT, and bin/ci-demo-conformance.sh says so: a language +/// with a longer arm name would otherwise be in permanent violation of an alignment rule. Column +/// IDENTITY and ORDER are what every language must match. +constexpr const char* kHeaderFormat = " %-27s %10s %8s %10s %12s %12s\n"; +constexpr const char* kRowFormat = " %-27s %10s %8s %9.1fs %12s %12s\n"; + +} // namespace + +double ArmResult::rate_per_second() const { + const double seconds = std::chrono::duration(elapsed).count(); + return seconds > 0 ? processed / seconds : 0; +} + +std::string with_thousands(long value) { + std::string digits = std::to_string(value); + // A negative number would otherwise have a separator inserted after its sign; nothing here + // produces one, but a renderer that mangles input it was not expecting is a bug waiting for a + // caller. + const std::size_t floor = digits.empty() || digits.front() != '-' ? 0 : 1; + for (std::size_t at = digits.size(); at > floor + 3;) { + at -= 3; + digits.insert(at, ","); + } + return digits; +} + +std::string banner() { + const std::string rule(kRuleWidth, '='); + return "\n" + rule + "\n PARALLEL CONSUMER - " + kLanguage + " demo\n" + + " The same records, twice: one at a time, then all at once.\n" + rule + "\n"; +} + +std::string render(const std::string& title, const std::vector& results, const ArmResult* baseline, + bool across_replays) { + char line[512]; + std::string table = "\n\n" + title + "\n"; + std::snprintf(line, sizeof(line), kHeaderFormat, "arm", "records", "keys", "elapsed", "msg/s", + across_replays ? "vs AK core*" : "vs AK core"); + table += line; + for (const ArmResult& result : results) { + std::string ratio = "-"; + if (baseline != nullptr && baseline->rate_per_second() > 0) { + char rendered[32]; + std::snprintf(rendered, sizeof(rendered), "%.1fx", + result.rate_per_second() / baseline->rate_per_second()); + ratio = rendered; + } + std::snprintf(line, sizeof(line), kRowFormat, result.arm.c_str(), + with_thousands(result.processed).c_str(), + with_thousands(result.unique_keys).c_str(), + std::chrono::duration(result.elapsed).count(), + with_thousands(static_cast(result.rate_per_second())).c_str(), + ratio.c_str()); + table += line; + } + if (across_replays) { + table += "\n * against the SMALL replay's AK core arm. Across replays, so not like-for-like.\n"; + } + return table; +} + +} // namespace parallelconsumer::proxy::demo diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_report.h b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_report.h new file mode 100644 index 0000000000..b1e537cff3 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/src/demo_report.h @@ -0,0 +1,76 @@ +// Copyright (C) 2026 Antony Stubbs and contributors +// +// WHAT A READER ACTUALLY SEES, which the contract treats as the demo's product rather than its +// packaging: "a demo that is correct and unreadable has failed at the only thing it does" +// (parallel-consumer-proxy/demo/README.md, "The output a reader actually sees"). +// +// SEPARATED FROM demo.cpp SO IT CAN BE TESTED, and for no other reason. The banner's wording, the +// column set and their order are contract - eleven languages must print the same shape - and the +// arms that produce the figures need a broker, which the image build has none of. Rendering does +// not, so it is a pure function of an ArmResult here and tests/demo_report_test.cpp holds it to the +// contract inside the image build. That is the same split demo_options made, for the same reason. +// +// THESE FUNCTIONS RETURN STRINGS RATHER THAN PRINTING. A renderer that writes to std::cout can only +// be tested by capturing a stream, and the thing worth asserting is the text. + +#ifndef PARALLELCONSUMER_PROXY_DEMO_REPORT_H +#define PARALLELCONSUMER_PROXY_DEMO_REPORT_H + +#include +#include +#include + +namespace parallelconsumer::proxy::demo { + +/// What one arm achieved: how long it took, over how many records, and across how many keys. +/// +/// THE LAST TWO ARE DETERMINISTIC AND THE FIRST IS NOT, which is the whole point of carrying them. +/// Elapsed and msg/s depend on the machine, so no two languages can ever be compared on them; +/// records and keys are a property of the backlog, so every language processing the same records +/// must report the same pair. That is what makes `bin/ci-demo-conformance.sh` able to compare +/// languages at all, and what turns the table from an assertion that work happened into a +/// demonstration of it. +struct ArmResult { + std::string arm; + std::chrono::nanoseconds elapsed{0}; + + /// Records this arm's user function ran on. Must equal the target: a short arm is a failed arm, + /// not a fast one. + int processed = 0; + + /// Distinct record keys this arm observed, which is what shows the backlog was really spread + /// rather than one key repeated. + int unique_keys = 0; + + /// Throughput, which is the only *timing* figure this demo reports - no latency, ever, because + /// the backlog is pre-produced and the workload is therefore closed-loop. + [[nodiscard]] double rate_per_second() const; +}; + +/// The language this demo is for, as it appears in the banner. C++, not "cpp": the banner is for a +/// human, and the module directory's name is not what the language is called. +inline constexpr const char* kLanguage = "C++"; + +/// The first thing the demo prints, and the same in every language but its name. +/// +/// A READER WHO RUNS THIS MUST BE TOLD WHAT THEY ARE LOOKING AT. Before this existed the first line +/// out of the demo was a configuration dial, which names neither the product nor the point - the +/// contract calls that out by name as having told the reader nothing. +[[nodiscard]] std::string banner(); + +/// One results table: the title, then a row per arm. +/// +/// @param baseline the arm every ratio is measured against, or nullptr for no ratio column content +/// @param across_replays whether the baseline came from the OTHER replay, which makes the ratio not +/// like-for-like and must be said rather than left for the reader to notice +[[nodiscard]] std::string render(const std::string& title, const std::vector& results, + const ArmResult* baseline, bool across_replays); + +/// A whole number with thousands separators, hand-rolled rather than taken from a locale: the demo +/// image is a slim one and its locale is C, so `std::locale("en_US.UTF-8")` would throw there and +/// nowhere else. +[[nodiscard]] std::string with_thousands(long value); + +} // namespace parallelconsumer::proxy::demo + +#endif // PARALLELCONSUMER_PROXY_DEMO_REPORT_H diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/tests/demo_options_test.cpp b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/tests/demo_options_test.cpp new file mode 100644 index 0000000000..4e04db74c0 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/tests/demo_options_test.cpp @@ -0,0 +1,162 @@ +// Copyright (C) 2026 Antony Stubbs and contributors +// +// The demo's option surface, which is the part of it that has actually broken before. +// +// EVERY FAILURE THIS FAN-OUT'S DEMOS HAVE HAD LIVED IN THE ENTRY PATH, not in the arms: the +// no-argument case, the precedence between a flag and an environment variable, a value that +// silently wrapped. They are checked here because they are the only part of the demo that can be +// checked without a broker - the arms need one, and the image build has none. +// +// They run under the module's own sixty-line harness (../../tests/test_main.cpp) rather than a +// second one, and they run INSIDE THE DEMO IMAGE at build time, for the same reason the client +// library's tests do: this host has no C++ toolchain, so a check that only ran on the host would +// only ever run in CI. + +#include +#include +#include +#include +#include + +#include "demo_options.h" +#include "test_support.h" + +namespace { + +namespace demo = parallelconsumer::proxy::demo; + +/// An environment that is exactly what the test says it is - no more, and nothing inherited from +/// the process running the test. +demo::Environment environment_of(std::map values) { + return [values = std::move(values)](const std::string& name) -> std::optional { + const auto found = values.find(name); + return found == values.end() ? std::nullopt : std::optional(found->second); + }; +} + +demo::Environment empty_environment() { return environment_of({}); } + +std::string refusal(const std::vector& args, const demo::Environment& environment) { + try { + demo::DemoOptions::parse(args, environment); + } catch (const std::exception& refused) { + return refused.what(); + } + return {}; +} + +PCP_TEST(no_arguments_is_the_documented_defaults, + "the no-argument run - the double-click case - is the contract's defaults") { + const demo::DemoOptions options = demo::DemoOptions::parse({}, empty_environment()); + + PCP_CHECK_EQ(options.records, 2000); + PCP_CHECK_EQ(options.delay_ms, 2); + PCP_CHECK_EQ(options.max_concurrency, 100); + PCP_CHECK_EQ(options.partitions, 10); + PCP_CHECK_EQ(options.replay_factor, 20); + PCP_CHECK(options.bootstrap.empty()); + PCP_CHECK(options.topic.empty()); +} + +PCP_TEST(flags_beat_the_environment_beats_the_defaults, + "a flag overrides an environment variable, which overrides the default") { + const demo::Environment environment = environment_of({{"PC_DEMO_RECORDS", "50"}, + {"PC_DEMO_DELAY_MS", "7"}, + {"PC_DEMO_CONCURRENCY", "9"}, + {"PC_DEMO_PARTITIONS", "3"}, + {"PC_DEMO_REPLAY_FACTOR", "4"}, + {"PC_DEMO_BOOTSTRAP", "broker:9092"}, + {"PC_DEMO_TOPIC", "from-env"}}); + + const demo::DemoOptions options = demo::DemoOptions::parse({"--records", "11", "--topic", "from-flag"}, + environment); + + PCP_CHECK_EQ(options.records, 11); // the flag won + PCP_CHECK_EQ(options.topic, std::string("from-flag")); + PCP_CHECK_EQ(options.delay_ms, 7); // the environment won over the default + PCP_CHECK_EQ(options.max_concurrency, 9); + PCP_CHECK_EQ(options.partitions, 3); + PCP_CHECK_EQ(options.replay_factor, 4); + PCP_CHECK_EQ(options.bootstrap, std::string("broker:9092")); +} + +PCP_TEST(every_flag_has_an_environment_variable, + "the seven flags and the seven PC_DEMO_ variables are the same seven settings") { + const demo::DemoOptions options = demo::DemoOptions::parse( + {"--records", "1", "--delay-ms", "2", "--concurrency", "3", "--partitions", "4", + "--replay-factor", "5", "--bootstrap", "flagged:9092", "--topic", "flagged"}, + empty_environment()); + + PCP_CHECK_EQ(options.records, 1); + PCP_CHECK_EQ(options.delay_ms, 2); + PCP_CHECK_EQ(options.max_concurrency, 3); + PCP_CHECK_EQ(options.partitions, 4); + PCP_CHECK_EQ(options.replay_factor, 5); + PCP_CHECK_EQ(options.bootstrap, std::string("flagged:9092")); + PCP_CHECK_EQ(options.topic, std::string("flagged")); +} + +PCP_TEST(an_empty_environment_value_is_unset, + "a compose file's empty PC_DEMO_TOPIC means 'name your own', not 'the empty topic'") { + const demo::DemoOptions options = + demo::DemoOptions::parse({}, environment_of({{"PC_DEMO_TOPIC", ""}, {"PC_DEMO_BOOTSTRAP", " "}})); + + PCP_CHECK(options.topic.empty()); + PCP_CHECK(options.bootstrap.empty()); +} + +PCP_TEST(a_misspelled_flag_is_refused, + "an unknown flag stops the run rather than reporting numbers for settings nobody asked for") { + PCP_CHECK_CONTAINS(refusal({"--record", "10"}, empty_environment()), "unknown option: --record"); + PCP_CHECK_CONTAINS(refusal({"--records"}, empty_environment()), "--records needs a value"); + PCP_CHECK_CONTAINS(refusal({"--records", "ten"}, empty_environment()), "needs a whole number"); + PCP_CHECK_CONTAINS(refusal({"--records", "0"}, empty_environment()), "must be at least 1"); + PCP_CHECK_CONTAINS(refusal({"--delay-ms", "-1"}, empty_environment()), "must not be negative"); + PCP_CHECK_CONTAINS(refusal({}, environment_of({{"PC_DEMO_RECORDS", "0"}})), + "PC_DEMO_RECORDS must be at least 1"); +} + +PCP_TEST(the_big_replay_cannot_overflow, + "records times replay-factor is refused rather than wrapped into a confident small number") { + PCP_CHECK_CONTAINS(refusal({"--records", "2000000", "--replay-factor", "2000"}, empty_environment()), + "more records than the demo can count"); +} + +PCP_TEST(a_replay_factor_of_one_skips_the_big_replay, + "the big replay runs only when it is bigger than the small one") { + PCP_CHECK(!demo::DemoOptions::parse({"--replay-factor", "1"}, empty_environment()).big_replay_wanted()); + PCP_CHECK(!demo::DemoOptions::parse({"--replay-factor", "0"}, empty_environment()).big_replay_wanted()); + PCP_CHECK(demo::DemoOptions::parse({"--replay-factor", "2"}, empty_environment()).big_replay_wanted()); + + // A skipped big replay still consumes the small replay's own records, never zero. + PCP_CHECK_EQ(demo::DemoOptions::parse({"--records", "10", "--replay-factor", "0"}, empty_environment()) + .big_replay_records(), + 10); +} + +PCP_TEST(the_fingerprint_never_prints_the_broker, + "the effective configuration carries every dial and no address") { + const demo::DemoOptions options = + demo::DemoOptions::parse({"--bootstrap", "secret-broker.internal:9092"}, empty_environment()); + + const std::string fingerprint = options.describe(); + + // Own-cluster mode puts a user's real broker in there, and a demo is bound by the same + // credential-hygiene rule as the client library. + PCP_CHECK_ABSENT(fingerprint, "secret-broker.internal"); + PCP_CHECK_CONTAINS(fingerprint, "records = 2000"); + PCP_CHECK_CONTAINS(fingerprint, "delayMs = 2"); + PCP_CHECK_CONTAINS(fingerprint, "maxConcurrency = 100"); + PCP_CHECK_CONTAINS(fingerprint, "partitions = 10"); + PCP_CHECK_CONTAINS(fingerprint, "replayFactor = 20"); +} + +PCP_TEST(help_is_answered_wherever_it_arrives, + "--help reaches this binary directly under `docker compose run demo --help`") { + PCP_CHECK(demo::DemoOptions::help_requested({"--help"})); + PCP_CHECK(demo::DemoOptions::help_requested({"--records", "10", "-h"})); + PCP_CHECK(!demo::DemoOptions::help_requested({"--records", "10"})); + PCP_CHECK_CONTAINS(demo::usage(), "PC_DEMO_DELAY_MS"); +} + +} // namespace diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/tests/demo_report_test.cpp b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/tests/demo_report_test.cpp new file mode 100644 index 0000000000..054529a727 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-cpp/demo/tests/demo_report_test.cpp @@ -0,0 +1,150 @@ +// Copyright (C) 2026 Antony Stubbs and contributors +// +// THE OUTPUT IS THE CONTRACT, so it is the thing tested. +// +// parallel-consumer-proxy/demo/README.md fixes the banner's wording, the column set, the column +// order, and the fact that an arm names the client that produced its row - eleven languages must +// print the same shape or `bin/ci-demo-conformance.sh` has nothing to compare. None of that needs a +// broker, an image or a running arm: it is a pure function of an ArmResult, which is why +// demo_report.{h,cpp} exists as a unit at all. +// +// WHAT THIS CANNOT SEE, deliberately: whether the FIGURES are right. That the AK core arm counts +// the keys librdkafka handed it, and the sidecar arm counts the keys the client library handed it, +// is proven by running the demo - these tests prove the table says what the contract requires once +// the arms have reported. +// +// They run under the module's own sixty-line harness and INSIDE THE IMAGE BUILD, like every other +// C++ test here: this host has no C++ toolchain. + +#include +#include +#include + +#include "demo_report.h" +#include "test_support.h" + +namespace { + +namespace demo = parallelconsumer::proxy::demo; + +demo::ArmResult arm(const std::string& name, double seconds, int processed, int unique_keys) { + demo::ArmResult result; + result.arm = name; + result.elapsed = std::chrono::duration_cast( + std::chrono::duration(seconds)); + result.processed = processed; + result.unique_keys = unique_keys; + return result; +} + +PCP_TEST(the_banner_names_the_product_first, + "a reader who runs this is told what they are looking at, not which dial was set") { + const std::string printed = demo::banner(); + + PCP_CHECK_CONTAINS(printed, "PARALLEL CONSUMER"); + PCP_CHECK_CONTAINS(printed, "C++ demo"); + PCP_CHECK_CONTAINS(printed, "The same records, twice: one at a time, then all at once."); + // The rule the contract draws it with, at the width the contract draws it. + PCP_CHECK_CONTAINS(printed, std::string(64, '=')); + // The banner is a heading, not a fingerprint: a dial here would be the mistake it replaced. + PCP_CHECK_ABSENT(printed, "records = "); +} + +PCP_TEST(the_banner_says_the_language_rather_than_the_module_directory, + "the module is parallel-consumer-proxy-client-cpp; the language is C++") { + PCP_CHECK_ABSENT(demo::banner(), "cpp demo"); +} + +PCP_TEST(the_table_carries_the_contract_columns_in_order, + "arm, records, keys, elapsed, msg/s, vs AK core - identity and order are contract") { + const std::vector results{arm("AK core (librdkafka)", 2.0, 20, 20)}; + + const std::string header = demo::render("Small replay", results, nullptr, false); + const std::size_t at_arm = header.find("arm"); + const std::size_t at_elapsed = header.find("elapsed"); + const std::size_t at_rate = header.find("msg/s"); + const std::size_t at_records = header.find("records"); + const std::size_t at_keys = header.find("keys"); + const std::size_t at_ratio = header.find("vs AK core"); + + PCP_CHECK(at_arm != std::string::npos); + PCP_CHECK(at_arm < at_records); + PCP_CHECK(at_records < at_keys); + PCP_CHECK(at_keys < at_elapsed); + PCP_CHECK(at_elapsed < at_rate); + PCP_CHECK(at_rate < at_ratio); +} + +PCP_TEST(every_arm_reports_what_it_did_and_not_only_how_fast, + "records and keys are what demonstrate the run rather than assert it") { + const std::vector results{arm("AK core (librdkafka)", 2.0, 2000, 1000), + arm("pc-cpp-grpc (this client)", 0.5, 2000, 1000)}; + + const std::string table = demo::render("Small replay", results, &results.front(), false); + + // Both figures, on both rows, with the separators the rate column already used. + PCP_CHECK_CONTAINS(table, "2,000"); + PCP_CHECK_CONTAINS(table, "1,000"); + // The serial arm is its own baseline, and the parallel arm is four times it at a quarter of the + // wall clock - the ratio the reader came for. + PCP_CHECK_CONTAINS(table, "1.0x"); + PCP_CHECK_CONTAINS(table, "4.0x"); +} + +PCP_TEST(an_arm_names_the_client_that_produced_its_row, + "'AK core' is a category; a reader cannot judge a comparison without the library") { + const std::vector results{arm("AK core (librdkafka)", 2.0, 20, 20), + arm("pc-cpp-grpc (this client)", 1.0, 20, 20)}; + + const std::string table = demo::render("Small replay", results, &results.front(), false); + + PCP_CHECK_CONTAINS(table, "AK core (librdkafka)"); + PCP_CHECK_CONTAINS(table, "pc-cpp-grpc (this client)"); +} + +PCP_TEST(a_cross_replay_ratio_says_so, + "the big replay's baseline came from the small replay, which is not like-for-like") { + const std::vector baseline{arm("AK core (librdkafka)", 2.0, 20, 20)}; + const std::vector big{arm("pc-cpp-grpc (this client)", 1.0, 40, 40)}; + + const std::string same_replay = demo::render("Small replay", baseline, &baseline.front(), false); + const std::string across = demo::render("Big replay", big, &baseline.front(), true); + + PCP_CHECK_ABSENT(same_replay, "vs AK core*"); + PCP_CHECK_CONTAINS(across, "vs AK core*"); + PCP_CHECK_CONTAINS(across, "not like-for-like"); +} + +PCP_TEST(no_arm_reports_latency, + "the backlog is pre-produced, so the workload is closed-loop and a per-record timing lies") { + const std::vector results{arm("AK core (librdkafka)", 2.0, 20, 20)}; + + const std::string table = demo::render("Small replay", results, &results.front(), false); + + // The same words bin/ci-demo-conformance.sh refuses across every language. + PCP_CHECK_ABSENT(table, "latency"); + PCP_CHECK_ABSENT(table, "p99"); + PCP_CHECK_ABSENT(table, "percentile"); +} + +PCP_TEST(a_zero_length_arm_reports_no_throughput_rather_than_a_division, + "an arm that finished inside the clock's resolution must not print an infinity") { + const std::vector results{arm("AK core (librdkafka)", 0.0, 20, 20)}; + + PCP_CHECK_EQ(results.front().rate_per_second(), 0.0); + // A nullptr-rate baseline yields the placeholder rather than a nan. + const std::string table = demo::render("Small replay", results, &results.front(), false); + PCP_CHECK_ABSENT(table, "inf"); + PCP_CHECK_ABSENT(table, "nan"); +} + +PCP_TEST(thousands_separators_are_hand_rolled_because_the_image_locale_is_C, + "std::locale(\"en_US.UTF-8\") throws in the demo image and nowhere else") { + PCP_CHECK_EQ(demo::with_thousands(0), std::string("0")); + PCP_CHECK_EQ(demo::with_thousands(999), std::string("999")); + PCP_CHECK_EQ(demo::with_thousands(1000), std::string("1,000")); + PCP_CHECK_EQ(demo::with_thousands(1234567), std::string("1,234,567")); + PCP_CHECK_EQ(demo::with_thousands(-1234), std::string("-1,234")); +} + +} // namespace diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/Bz.Stub.ParallelConsumer.Proxy.Client.sln b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/Bz.Stub.ParallelConsumer.Proxy.Client.sln index 97a5651667..84dcf44dd1 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/Bz.Stub.ParallelConsumer.Proxy.Client.sln +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/Bz.Stub.ParallelConsumer.Proxy.Client.sln @@ -13,6 +13,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bz.Stub.ParallelConsumer.Pr EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "ConformanceRunner", "tests\ConformanceRunner\ConformanceRunner.csproj", "{41E68D9C-D0A3-4C11-A78C-94DA79AF3C72}" EndProject +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "demo", "demo", "{E0EA0813-DB4A-4469-91D7-4A4DF30FCCE2}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Bz.Stub.ParallelConsumer.Proxy.Client.Demo", "demo\Bz.Stub.ParallelConsumer.Proxy.Client.Demo\Bz.Stub.ParallelConsumer.Proxy.Client.Demo.csproj", "{7681D2EC-9632-4421-BECD-8C0BE4B09F96}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -59,6 +63,18 @@ Global {41E68D9C-D0A3-4C11-A78C-94DA79AF3C72}.Release|x64.Build.0 = Release|Any CPU {41E68D9C-D0A3-4C11-A78C-94DA79AF3C72}.Release|x86.ActiveCfg = Release|Any CPU {41E68D9C-D0A3-4C11-A78C-94DA79AF3C72}.Release|x86.Build.0 = Release|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Debug|Any CPU.Build.0 = Debug|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Debug|x64.ActiveCfg = Debug|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Debug|x64.Build.0 = Debug|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Debug|x86.ActiveCfg = Debug|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Debug|x86.Build.0 = Debug|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Release|Any CPU.ActiveCfg = Release|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Release|Any CPU.Build.0 = Release|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Release|x64.ActiveCfg = Release|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Release|x64.Build.0 = Release|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Release|x86.ActiveCfg = Release|Any CPU + {7681D2EC-9632-4421-BECD-8C0BE4B09F96}.Release|x86.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE @@ -67,5 +83,6 @@ Global {E36656E9-135C-4D43-993C-CB07F8155752} = {827E0CD3-B72D-47B6-A68D-7590B98EB39B} {57E44FC0-18AD-484B-8DB2-C987E086353C} = {0AB3BF05-4346-4AA6-1389-037BE0695223} {41E68D9C-D0A3-4C11-A78C-94DA79AF3C72} = {0AB3BF05-4346-4AA6-1389-037BE0695223} + {7681D2EC-9632-4421-BECD-8C0BE4B09F96} = {E0EA0813-DB4A-4469-91D7-4A4DF30FCCE2} EndGlobalSection EndGlobal diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/Directory.Build.props b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/Directory.Build.props index a4b982d58c..024c542166 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/Directory.Build.props +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/Directory.Build.props @@ -52,8 +52,18 @@ - 2.71.0 + 2.83.0 3.31.1 + + + + Exe + pc-dotnet-demo + Bz.Stub.ParallelConsumer.Proxy.Client.Demo + false + + false + Major + + + + + + + + + + + + + + + + + + + + diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/DemoBroker.cs b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/DemoBroker.cs new file mode 100644 index 0000000000..a616e784e4 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/DemoBroker.cs @@ -0,0 +1,261 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +using System.Globalization; +using System.Text; +using Confluent.Kafka; +using Confluent.Kafka.Admin; +using Testcontainers.Kafka; + +namespace Bz.Stub.ParallelConsumer.Proxy.Client.Demo; + +/// +/// The broker the demo reads from, however the reader got here. +/// +/// +/// Two ways in, and the second is a rule rather than a convenience: +/// +/// NOTHING SUPPLIED - the demo starts a real broker in a container, because that is what a +/// user actually runs. +/// AN ADDRESS SUPPLIED - the demo uses it and starts nothing. This is how the demo runs +/// INSIDE its own container, and it is not optional there: a demo container is never granted +/// the host Docker socket (plan unit U35), so it could not start a broker even if it wanted to. +/// It reaches a compose sibling on the demo's own network instead. +/// +/// The same door serves own-cluster mode, where the address is the user's real cluster - so nothing +/// here logs or echoes it. +/// +internal sealed class DemoBroker : IAsyncDisposable +{ + /// + /// The broker image, pinned as a literal. + /// + /// + /// The Java demo DERIVES this from the Kafka client on its own classpath (CP major = AK major + + /// 4). .NET cannot: Confluent.Kafka's version tracks librdkafka, which has no such mapping to a + /// broker release. So this is pinned, and it tracks the same two places the compose file beside + /// it does - the root pom's kafka.version, through that mapping. + /// + private const string BrokerImage = "confluentinc/cp-kafka:7.9.0"; + + /// + /// The key space the seeded records spread over. Ordering is UNORDERED in both arms, so this + /// changes nothing today; it exists so that a key-ordered lane added later has more than one key + /// to shard across, rather than needing the seeding rewritten first. + /// + private const int KeySpace = 1000; + + private readonly KafkaContainer? _container; + + private DemoBroker(string bootstrap, KafkaContainer? container) + { + Bootstrap = bootstrap; + _container = container; + } + + /// + /// Where the broker is. NEVER PRINTED - see for why. + /// + public string Bootstrap { get; } + + /// Uses the supplied broker, or starts one when none was supplied. + /// The address from --bootstrap or the environment, or null. + /// Bounds the container start. + /// The broker. + public static async Task ResolveAsync(string? supplied, CancellationToken cancellationToken) + { + if (!string.IsNullOrWhiteSpace(supplied)) + { + // deliberately not logged: own-cluster mode puts a real address here + Console.WriteLine("Using the broker supplied by the caller."); + return new DemoBroker(NormaliseBootstrap(supplied.Trim()), null); + } + + Console.WriteLine($"No broker supplied, starting one in a container: {BrokerImage}"); + var container = new KafkaBuilder(BrokerImage) + // The demo forms one consumer group per arm, one after another, and the default + // three-second settling delay would be charged to every one of them. The compose file + // beside this module sets the same value for the same reason. + .WithEnvironment("KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS", "500") + .Build(); + await container.StartAsync(cancellationToken).ConfigureAwait(false); + return new DemoBroker(NormaliseBootstrap(container.GetBootstrapAddress()), container); + } + + /// Creates the demo's topic, tolerating one a previous run already left behind. + /// The topic to create. + /// How many partitions it must have. + /// A task that completes once the topic exists with the requested partition count. + public async Task EnsureTopicAsync(string topic, int partitions) + { + using var admin = new AdminClientBuilder(new AdminClientConfig { BootstrapServers = Bootstrap }) + .Build(); + try + { + await admin.CreateTopicsAsync( + [new TopicSpecification { Name = topic, NumPartitions = partitions, ReplicationFactor = 1 }]) + .ConfigureAwait(false); + Console.WriteLine(string.Create( + CultureInfo.InvariantCulture, $"Created topic {topic} with {partitions} partitions")); + } + catch (CreateTopicsException failure) + when (failure.Results.All(result => result.Error.Code == ErrorCode.TopicAlreadyExists)) + { + // Reusing a topic silently is fine; reusing one with a DIFFERENT partition count is not, + // because the effective-configuration block would print a --partitions value that never + // applied - and that block is the demo's whole reproducibility promise. + var existing = admin.GetMetadata(topic, TimeSpan.FromSeconds(30)).Topics.Single().Partitions.Count; + if (existing != partitions) + { + throw new InvalidOperationException(string.Create( + CultureInfo.InvariantCulture, + $"topic {topic} already exists with {existing} partitions, but this run asked for {partitions} - pass --topic to name a fresh one, or --partitions {existing}")); + } + + Console.WriteLine(string.Create( + CultureInfo.InvariantCulture, + $"Topic {topic} already exists with the requested {partitions} partitions, reusing it")); + } + } + + /// + /// Produces the backlog both arms then replay. + /// + /// + /// Pre-produced rather than produced alongside the arms, and that is what makes the workload + /// closed-loop - which is in turn why no arm reports latency. + /// + /// The topic to seed. + /// The first record ordinal to produce. + /// One past the last record ordinal to produce. + public void Seed(string topic, int from, int to) + { + if (to <= from) + { + return; + } + + Console.WriteLine(string.Create(CultureInfo.InvariantCulture, $"Producing records {from} to {to}...")); + + // A delivery report that FAILED is not an exception at the call site, and a demo that + // reported a full backlog while running its arms against a short one would print numbers + // for a workload that never existed. So the first failure is captured and raised below. + Error? firstFailure = null; + using (var producer = new ProducerBuilder( + new ProducerConfig { BootstrapServers = Bootstrap, LingerMs = 20 }).Build()) + { + for (var ordinal = from; ordinal < to; ordinal++) + { + var message = new Message + { + Key = Encoding.UTF8.GetBytes( + string.Create(CultureInfo.InvariantCulture, $"key-{ordinal % KeySpace}")), + Value = Encoding.UTF8.GetBytes( + string.Create(CultureInfo.InvariantCulture, $"record-{ordinal}")), + }; + Produce(producer, topic, message, report => + { + if (report.Error.IsError) + { + Interlocked.CompareExchange(ref firstFailure, report.Error, null); + } + }); + } + + producer.Flush(TimeSpan.FromMinutes(2)); + } + + if (firstFailure is not null) + { + throw new InvalidOperationException( + $"the demo could not seed its backlog: {firstFailure.Reason}"); + } + + Console.WriteLine(string.Create(CultureInfo.InvariantCulture, $"Produced {to - from} records")); + } + + /// + /// The Kafka properties each arm's consumer needs to reach this broker - the AK core arm builds + /// its own consumer from them, and the sidecar arm sends them to the proxy, which builds one. + /// + /// + /// enable.auto.commit is here because Parallel Consumer owns offset commits and refuses a + /// consumer with auto-commit on. The sidecar forces the setting itself whatever the map says, so + /// on that path this line is redundant; the AK core arm needs it because Kafka's own default is + /// true, and an auto-committing serial arm would be doing measurably less work than the arm it + /// is the denominator of. + /// + /// The consumer group for this arm and replay. + /// The properties, in the wire's own string-to-string shape. + public Dictionary ConsumerProperties(string groupId) => new(StringComparer.Ordinal) + { + ["bootstrap.servers"] = Bootstrap, + ["group.id"] = groupId, + ["auto.offset.reset"] = "earliest", + ["enable.auto.commit"] = "false", + }; + + /// Stops the container, if this demo started one. + /// A task that completes once the broker is gone. + public async ValueTask DisposeAsync() + { + if (_container is not null) + { + await _container.DisposeAsync().ConfigureAwait(false); + } + } + + /// + /// Produces one record, waiting out a full local queue rather than failing on it. + /// + /// + /// librdkafka's produce is non-blocking and throws Local_QueueFull when its send buffer is + /// full, where the Java producer would simply block. The big replay seeds tens of thousands of + /// records in a tight loop, so this is reached in an ordinary run rather than under stress; the + /// wait lets the background sender drain and retries the same record. + /// + private static void Produce( + IProducer producer, + string topic, + Message message, + Action> onDelivered) + { + while (true) + { + try + { + producer.Produce(topic, message, onDelivered); + return; + } + catch (ProduceException full) when (full.Error.Code == ErrorCode.Local_QueueFull) + { + producer.Poll(TimeSpan.FromMilliseconds(100)); + } + } + } + + /// + /// Reduces a bootstrap address to the host:port list Kafka's own clients expect, + /// whatever URL decoration it arrived with. + /// + /// + /// NOT COSMETIC, AND FINDING IT COST THE SIDECAR ARM A WHOLE RUN. Testcontainers for .NET hands + /// its address back as a URI - measured, on this machine: plaintext://127.0.0.1:62347/, + /// lower-cased scheme and TRAILING SLASH included. librdkafka accepts that string, so the AK + /// core arm works with it untouched. The sidecar's consumer is a JAVA client, and Java's + /// bootstrap.servers parser rejects it - the trailing slash above all, which no amount of + /// scheme-stripping removes. The address travels to the proxy in Configure, so the arm + /// died at session construction with a deliberately reason-free error (R48 withholds it, because + /// a Kafka ConfigException embeds property values and those may be credentials). + /// + /// Each comma-separated entry is normalised independently, so a caller's own multi-broker + /// --bootstrap survives, and an entry that is already host:port is left exactly as + /// it was rather than round-tripped through a parser that could rewrite it. + /// + /// + private static string NormaliseBootstrap(string address) => string.Join( + ',', + address.Split(',', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries) + .Select(entry => entry.Contains("://", StringComparison.Ordinal) + ? string.Create(CultureInfo.InvariantCulture, $"{new Uri(entry).Host}:{new Uri(entry).Port}") + : entry)); +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/DemoOptions.cs b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/DemoOptions.cs new file mode 100644 index 0000000000..9f18218251 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/DemoOptions.cs @@ -0,0 +1,241 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +using System.Globalization; + +namespace Bz.Stub.ParallelConsumer.Proxy.Client.Demo; + +/// +/// The demo's dials, and the interface EVERY language's demo mirrors - the flags, their defaults, +/// their environment variables and the precedence between them. +/// +/// +/// The contract is parallel-consumer-proxy/demo/README.md; the reference implementation of +/// this type is the Java demo's DemoOptions, and this is that surface in C#. Flags beat the +/// environment beats the defaults - a container passes configuration by environment while a person +/// at a terminal passes flags, and either must be able to override the other. +/// +/// R39 constrains how configuration reaches the PROXY. A demo is an application, so its flags are +/// not a violation of it. Without this note somebody reads --records as breaking the plan's +/// own rule and deletes it. +/// +/// +internal sealed record DemoOptions +{ + /// Prefix for every environment variable this demo reads, so a reader greps one string. + public const string EnvironmentPrefix = "PC_DEMO_"; + + public int Records { get; private init; } = 2000; + + public int DelayMs { get; private init; } = 2; + + public int MaxConcurrency { get; private init; } = 100; + + public int Partitions { get; private init; } = 10; + + public int ReplayFactor { get; private init; } = 20; + + /// + /// The broker to use, when the caller supplied one. Null means "start one" - the Testcontainers + /// default. Inside the demo's own container this is always present, because a demo container is + /// never granted the host Docker socket and so cannot start a broker of its own; it reaches a + /// compose sibling instead. + /// + public string? Bootstrap { get; private init; } + + /// The topic to use, when the caller supplied one. Null means the demo names its own. + public string? Topic { get; private init; } + + /// The records the big replay consumes in total, including the small replay's own. + public int BigReplayRecords => Records * Math.Max(1, ReplayFactor); + + /// True when the big replay is worth running at all; a factor of 1 or less skips it. + public bool BigReplayWanted => ReplayFactor > 1; + + /// Whether the caller asked for the usage text rather than a run. + /// + /// Handled here rather than only in run.sh, because the script is not the only way in: + /// docker compose run demo --help reaches this program directly, and answering that with + /// "unknown option: --help" would be a poor first impression. + /// + /// The process arguments. + /// Whether help was asked for. + public static bool IsHelpRequested(IEnumerable arguments) => + arguments.Any(argument => + string.Equals(argument, "-h", StringComparison.Ordinal) + || string.Equals(argument, "--help", StringComparison.Ordinal)); + + /// + /// Parses the demo's command line, falling back to the environment and then to the defaults. + /// + /// + /// The process arguments, which may legitimately be empty - that is the double-click case, and + /// it must work. + /// + /// + /// The environment to read, passed in rather than read from so this + /// is testable without mutating the process's own environment. + /// + /// The effective options. + /// + /// On an unknown flag, a missing value, or a value that is not a number in range. A demo that + /// silently ignored a misspelled flag would report numbers for settings nobody asked for. + /// + public static DemoOptions Parse( + IReadOnlyList arguments, IReadOnlyDictionary environment) + { + var options = FromEnvironment(environment); + + for (var index = 0; index < arguments.Count; index++) + { + var flag = arguments[index]; + switch (flag) + { + case "--records": + options = options with { Records = Positive(flag, Value(arguments, ++index, flag)) }; + break; + case "--delay-ms": + options = options with { DelayMs = NonNegative(flag, Value(arguments, ++index, flag)) }; + break; + case "--concurrency": + options = options with + { + MaxConcurrency = Positive(flag, Value(arguments, ++index, flag)), + }; + break; + case "--partitions": + options = options with { Partitions = Positive(flag, Value(arguments, ++index, flag)) }; + break; + case "--replay-factor": + // 1 or less skips the big replay, so this one is allowed to be zero + options = options with + { + ReplayFactor = NonNegative(flag, Value(arguments, ++index, flag)), + }; + break; + case "--bootstrap": + options = options with { Bootstrap = Value(arguments, ++index, flag) }; + break; + case "--topic": + options = options with { Topic = Value(arguments, ++index, flag) }; + break; + default: + throw new ArgumentException($"unknown option: {flag}", nameof(arguments)); + } + } + + return options.Validated(); + } + + /// + /// The effective configuration, for printing before the run. + /// + /// + /// A number without its settings is not reproducible, so this is part of the contract every + /// language's demo keeps rather than a debugging aid. THE BOOTSTRAP ADDRESS IS DELIBERATELY + /// ABSENT: own-cluster mode puts a user's real broker address there, and the credential-hygiene + /// rule that binds the proxy binds a demo too - nothing logged, nothing echoed. + /// + /// The fingerprint block, one setting per line. + public override string ToString() => string.Create( + CultureInfo.InvariantCulture, + $"records = {Records}\n delayMs = {DelayMs}\n maxConcurrency = {MaxConcurrency}\n partitions = {Partitions}\n replayFactor = {ReplayFactor}"); + + /// + /// Every flag's environment variable, applied in one place so the two spellings of a setting + /// cannot drift apart: the flag's name, upper-snake-cased, under the PC_DEMO_ prefix. + /// + private static DemoOptions FromEnvironment(IReadOnlyDictionary environment) + { + var options = new DemoOptions(); + + if (Named(environment, "RECORDS") is { } records) + { + options = options with { Records = Positive(EnvironmentPrefix + "RECORDS", records) }; + } + + if (Named(environment, "DELAY_MS") is { } delayMs) + { + options = options with { DelayMs = NonNegative(EnvironmentPrefix + "DELAY_MS", delayMs) }; + } + + if (Named(environment, "CONCURRENCY") is { } concurrency) + { + options = options with + { + MaxConcurrency = Positive(EnvironmentPrefix + "CONCURRENCY", concurrency), + }; + } + + if (Named(environment, "PARTITIONS") is { } partitions) + { + options = options with { Partitions = Positive(EnvironmentPrefix + "PARTITIONS", partitions) }; + } + + if (Named(environment, "REPLAY_FACTOR") is { } replayFactor) + { + options = options with + { + ReplayFactor = NonNegative(EnvironmentPrefix + "REPLAY_FACTOR", replayFactor), + }; + } + + if (Named(environment, "BOOTSTRAP") is { } bootstrap) + { + options = options with { Bootstrap = bootstrap }; + } + + if (Named(environment, "TOPIC") is { } topic) + { + options = options with { Topic = topic }; + } + + return options; + } + + /// An environment value, trimmed, or null when it is absent or blank. + private static string? Named(IReadOnlyDictionary environment, string suffix) => + environment.TryGetValue(EnvironmentPrefix + suffix, out var raw) && !string.IsNullOrWhiteSpace(raw) + ? raw.Trim() + : null; + + private static string Value(IReadOnlyList arguments, int index, string flag) => + index >= arguments.Count + ? throw new ArgumentException($"{flag} needs a value", nameof(arguments)) + : arguments[index]; + + private static int Positive(string flag, string raw) + { + var parsed = Number(flag, raw); + return parsed < 1 + ? throw new ArgumentException($"{flag} must be at least 1, got {parsed}", nameof(flag)) + : parsed; + } + + private static int NonNegative(string flag, string raw) + { + var parsed = Number(flag, raw); + return parsed < 0 + ? throw new ArgumentException($"{flag} must not be negative, got {parsed}", nameof(flag)) + : parsed; + } + + private static int Number(string flag, string raw) => + int.TryParse(raw.Trim(), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed) + ? parsed + : throw new ArgumentException($"{flag} needs a whole number, got '{raw}'", nameof(flag)); + + private DemoOptions Validated() + { + // Checked as a long here rather than trusted as an int later: Records * ReplayFactor + // overflows silently, and a wrapped value turns the big replay into a tiny one that still + // prints a confident throughput figure. + var bigReplay = (long)Records * Math.Max(1, ReplayFactor); + return bigReplay > int.MaxValue + ? throw new ArgumentException( + string.Create( + CultureInfo.InvariantCulture, + $"--records times --replay-factor is {bigReplay}, which is more records than the demo can count; lower one of them"), + nameof(Records)) + : this; + } +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/Program.cs b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/Program.cs new file mode 100644 index 0000000000..40653fd492 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/Program.cs @@ -0,0 +1,470 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +using System.Collections.Concurrent; +using System.Diagnostics; +using System.Globalization; +using System.Text; +using Confluent.Kafka; + +namespace Bz.Stub.ParallelConsumer.Proxy.Client.Demo; + +/// +/// The .NET demo. The same records through .NET's own Kafka client, one at a time, and +/// through this module's client library over a real sidecar. +/// +/// +/// THE CONTRACT IS parallel-consumer-proxy/demo/README.md, and this keeps it: the same +/// flags with the same defaults, the same PC_DEMO_ environment variables with the same +/// precedence, the effective configuration printed before anything runs, the same two tables in the +/// same order, and no latency anywhere. A reader who has run one language's demo has run them all. +/// +/// TWO ARMS, WHICH IS THE WHOLE CONTRACT OUTSIDE JAVA: +/// +/// +/// AK core (Confluent.Kafka) - a plain Confluent.Kafka consumer, one record at +/// a time. Always spelled "AK core", never bare "core", which reads as +/// parallel-consumer-core - and always carrying the client's own name, because "AK core" +/// is a CATEGORY and a reader cannot judge a comparison without knowing what produced it. +/// pc-dotnet-grpc (this client) - this module's client library over a sidecar it spawns +/// itself, which is what the second half of the label names. ON +/// THIS PATH THE APPLICATION DOES NO KAFKA I/O: the sidecar owns the consumer, the producer, +/// the group membership and the offsets. In a genuinely foreign application that is the whole +/// story - it needs no Kafka client library at all. In this demo it is a statement about the +/// PATH and not about the process: the same process creates the topic, produces the backlog and +/// runs the AK core arm with an ordinary Kafka client, because a comparison needs both +/// sides. +/// +/// +/// JAVA CARRIES FOUR MORE ARMS AND THIS DELIBERATELY DOES NOT. Java is the only place where the +/// sidecar hop can be priced honestly, because every arm there runs in one JVM against one broker +/// with one workload. Here the two arms are two different client libraries as well as two different +/// engines, so the gap between them is not a wire cost and is not read as one. +/// +/// +internal static class Program +{ + /// No arm may take longer than this before the demo calls it stalled rather than slow. + private static readonly TimeSpan ArmBudget = TimeSpan.FromMinutes(10); + + /// + /// THE FIRST THING THIS DEMO PRINTS, AND IT NAMES THE PRODUCT. + /// + /// + /// Contract, not decoration. A reader who runs this and is met with + /// pc-dotnet-grpc: the proxy granted 100 executor threads has been told nothing about what + /// they are looking at - not the product's name, not what is about to happen. Every language + /// prints this same banner, differing only in its own name, so a visitor moving between two + /// demos recognises the second one instantly. + /// + private const string Banner = """ + ================================================================ + PARALLEL CONSUMER - .NET demo + The same records, twice: one at a time, then all at once. + ================================================================ + """; + + /// + /// The serial arm's label: the ROLE and the CLIENT, because "AK core" alone is a category. + /// + /// + /// The answer differs in every language - rdkafka in Ruby, franz-go in Go, + /// kafkajs in TypeScript - and a reader evaluating "is this fast in my language" is + /// really asking about the client they already use. In .NET the answer is + /// Confluent.Kafka, and there is no second serious answer to choose between; the demo's + /// own README says so rather than leaving the reader to wonder. + /// + private const string AkCore = "AK core (Confluent.Kafka)"; + + /// The sidecar arm's label, naming what drives it: this module's own client library. + private const string SidecarArm = "pc-dotnet-grpc (this client)"; + + /// + /// The sidecar arm's identifier, for places a label cannot go. + /// + /// + /// Consumer group names travel to the broker and end up in metric names, so the label's spaces + /// and brackets have no business there. The label is for the reader; this is for Kafka. + /// + private const string SidecarArmId = "pc-dotnet-grpc"; + + /// + /// Stands in for a null key when counting distinct keys. + /// + /// + /// Every other key is counted by its base64 text, which cannot contain <, so this can + /// never collide with a real one. A null key is one distinct key, not an absent record. + /// + private const string NullKeyToken = ""; + + private static async Task Main(string[] arguments) + { + // BEFORE EVERYTHING, including the argument parsing: a reader who mistypes a flag should + // still be told what they were trying to run. + Console.WriteLine(Banner); + + if (DemoOptions.IsHelpRequested(arguments)) + { + Usage(); + return 0; + } + + DemoOptions options; + try + { + options = DemoOptions.Parse(arguments, EnvironmentVariables()); + } + catch (ArgumentException badArgument) + { + await Console.Error.WriteLineAsync(badArgument.Message).ConfigureAwait(false); + Usage(); + + // A misspelled flag must not be reported as a result for settings nobody asked for. + return 2; + } + + try + { + await using var broker = await DemoBroker + .ResolveAsync(options.Bootstrap, CancellationToken.None).ConfigureAwait(false); + var topic = options.Topic ?? string.Create( + CultureInfo.InvariantCulture, $"pc-demo-{Stopwatch.GetTimestamp()}"); + await RunAsync(options, broker, topic).ConfigureAwait(false); + return 0; + } +#pragma warning disable CA1031 // The demo's one top-level handler: a stack trace is not a result. + catch (Exception failure) + { + await Console.Error.WriteLineAsync($"The demo did not finish: {failure}").ConfigureAwait(false); + return 1; + } +#pragma warning restore CA1031 + } + + private static void Usage() => Console.WriteLine( + """ + + usage: demo/run.sh [options] + + --records N records in the comparison replay (default 2000) + --delay-ms N simulated work per record, ms (default 2) + --concurrency N max in-flight records (default 100) + --partitions N partitions on the demo topic (default 10) + --replay-factor N big replay = records x N; 1 skips (default 20) + --bootstrap ADDR an existing broker; omit to start one + --topic NAME an existing topic; omit to create one + + Every flag has an environment variable: --delay-ms is PC_DEMO_DELAY_MS. + Flags beat the environment beats the defaults. + """); + + private static Dictionary EnvironmentVariables() + { + var environment = new Dictionary(StringComparer.Ordinal); + foreach (System.Collections.DictionaryEntry entry in Environment.GetEnvironmentVariables()) + { + if (entry.Key is string name && entry.Value is string value) + { + environment[name] = value; + } + } + + return environment; + } + + private static async Task RunAsync(DemoOptions options, DemoBroker broker, string topic) + { + // THE FINGERPRINT, FIRST AND ALWAYS. A number without its settings is not reproducible. + // The bootstrap address is not in it: own-cluster mode puts a user's real broker there. + Console.WriteLine($"\nEffective configuration:\n {options}\n topic = {topic}"); + + await broker.EnsureTopicAsync(topic, options.Partitions).ConfigureAwait(false); + broker.Seed(topic, 0, options.Records); + + var small = new List + { + await AkCoreArmAsync(options, broker, topic, options.Records).ConfigureAwait(false), + await SidecarArmAsync(options, broker, topic, options.Records).ConfigureAwait(false), + }; + var baseline = small.Find(result => string.Equals(result.Arm, AkCore, StringComparison.Ordinal)); + Report( + string.Create( + CultureInfo.InvariantCulture, + $"Small replay - both arms over the same {options.Records} records (the comparison)"), + small, + baseline, + acrossReplays: false); + + if (!options.BigReplayWanted) + { + Console.WriteLine(string.Create( + CultureInfo.InvariantCulture, $"\nBig replay skipped (--replay-factor {options.ReplayFactor}).")); + return; + } + + var total = options.BigReplayRecords; + broker.Seed(topic, options.Records, total); + + // AK CORE IS EXCLUDED HERE BECAUSE IT DOES NOT GO PARALLEL. It would need total * delayMs + // milliseconds to finish a backlog the sidecar arm clears in seconds, and a demo that makes + // a reader wait that long to learn nothing new is not worth the wall clock. + var big = new List + { + await SidecarArmAsync(options, broker, topic, total).ConfigureAwait(false), + }; + // the unit is chosen so the figure is never zero - see the demo contract. + var serialMillis = (long)total * options.DelayMs; + var serialCost = serialMillis >= 1000 ? $"{serialMillis / 1000}s" : $"{serialMillis}ms"; + Report( + string.Create( + CultureInfo.InvariantCulture, + $"Big replay - {total} records, parallel arms only (AK core is serial and would take {serialCost}+)"), + big, + baseline, + acrossReplays: true); + } + + /// The serial arm: .NET's own Kafka client, one record at a time, the same wait. + private static async Task AkCoreArmAsync( + DemoOptions options, DemoBroker broker, string topic, int target) + { + Console.WriteLine(string.Create( + CultureInfo.InvariantCulture, $"\n=== {AkCore} starting over {target} records ===")); + + using var consumer = new ConsumerBuilder( + new ConsumerConfig(broker.ConsumerProperties(GroupId("ak-core")))).Build(); + consumer.Subscribe(topic); + + // The clock starts AFTER the consumer is built and stops before it closes, because this arm + // is the denominator of every ratio in both tables and the other arm charges itself for + // neither client construction nor teardown. + var startedAt = Stopwatch.GetTimestamp(); + var processed = 0; + var keys = NewKeySet(); + while (processed < target) + { + // The one arm that does not wait on a completion still needs the budget, or a backlog + // shorter than the target spins here forever with no output. + if (Stopwatch.GetElapsedTime(startedAt) > ArmBudget) + { + throw new InvalidOperationException(string.Create( + CultureInfo.InvariantCulture, $"{AkCore} stalled at {processed} of {target}")); + } + + var delivered = consumer.Consume(TimeSpan.FromMilliseconds(500)); + if (delivered?.Message is null) + { + continue; + } + + keys.TryAdd(KeyToken(delivered.Message.Key), 0); + await SimulatedWorkAsync(options.DelayMs, CancellationToken.None).ConfigureAwait(false); + processed++; + } + + return Finished(AkCore, Stopwatch.GetElapsedTime(startedAt), processed, keys.Count); + } + + /// + /// The client library over a real sidecar - the arm the whole design exists for. + /// + /// + /// IT GOES THROUGH THE CLIENT LIBRARY, NOT THE PROTOCOL. An earlier version of the Java demo + /// spoke gRPC by hand; it proved the engine worked and said nothing about the client library, + /// which is the artifact users actually touch. This arm spawns the sidecar the way an + /// application does - by asking the library to - and runs an ordinary delegate on the records. + /// + private static async Task SidecarArmAsync( + DemoOptions options, DemoBroker broker, string topic, int target) + { + Console.WriteLine(string.Create( + CultureInfo.InvariantCulture, $"\n=== {SidecarArm} starting over {target} records ===")); + + var sidecar = SidecarCommand.Resolve(); + var processed = 0; + var keys = NewKeySet(); + var reachedTarget = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + + await using var client = await ParallelConsumerClient.ConnectAsync(new ClientOptions + { + SidecarPath = sidecar.Path, + SidecarArguments = sidecar.Arguments, + SidecarErrorLog = SidecarErrorLog(), + Topics = [topic], + MaxConcurrency = options.MaxConcurrency, + Ordering = ProcessingOrder.Unordered, + KafkaProperties = broker.ConsumerProperties(GroupId(SidecarArmId)), + InstanceTag = "pc-dotnet-demo", + }).ConfigureAwait(false); + + var startedAt = Stopwatch.GetTimestamp(); + var session = client.PollAsync(async (record, cancellationToken) => + { + // On arrival, matching the serial arm: the column is "unique keys SEEN", which is a + // property of what the engine handed this function, not of the work afterwards. + keys.TryAdd(KeyToken(record.Key), 0); + await SimulatedWorkAsync(options.DelayMs, cancellationToken).ConfigureAwait(false); + if (Interlocked.Increment(ref processed) >= target) + { + reachedTarget.TrySetResult(); + } + + return Outcome.Succeed(); + }); + + return await AwaitedAsync( + SidecarArm, startedAt, reachedTarget.Task, session, () => processed, () => keys.Count, target) + .ConfigureAwait(false); + } + + /// + /// THE SIMULATED WORK, AND THE ONE PLACE THIS DEMO DIVERGES FROM THE OTHERS. + /// + /// + /// The contract says a blocking sleep is fine in C#, and it names Python and TypeScript as the + /// exceptions. IT IS NOT FINE HERE, and the reason is this client's own shape rather than the + /// language's: the library's executors are s on the thread pool, so a hundred + /// of them blocking in Thread.Sleep is a hundred pool threads occupied, and the pool + /// injects new ones at roughly one per second once its core count is used up. The sidecar arm + /// would then report the thread pool's injection rate rather than the engine's throughput - a + /// number that looks like a measurement and is not one. + /// + /// So the wait is an awaited timer, which is what "non-occupying" means in a language whose + /// concurrency is tasks rather than threads - the contract's own rule for Python and TypeScript, + /// applied for the same reason one language over. BOTH ARMS USE IT: the AK core arm holds one + /// record at a time, so it cannot starve anything, but an arm that is the denominator of every + /// ratio must not differ from its numerator by the wait primitive as well as by the transport. + /// + /// + private static Task SimulatedWorkAsync(int delayMs, CancellationToken cancellationToken) => + delayMs <= 0 ? Task.CompletedTask : Task.Delay(delayMs, cancellationToken); + + /// + /// Where the sidecar's own logging goes. Discarded by default, because the sidecar is a JVM with + /// a logging framework attached and its output is not what this demo is showing; set + /// PC_DEMO_SIDECAR_LOG to any value to see it, which is the first thing to do when the + /// sidecar arm will not start. + /// + private static TextWriter? SidecarErrorLog() => + string.IsNullOrEmpty(Environment.GetEnvironmentVariable("PC_DEMO_SIDECAR_LOG")) + ? null + : Console.Error; + + /// + /// Waits for the arm to reach its target, for its session to end, or for the budget - whichever + /// comes first. + /// + /// + /// REACHING THE TARGET IS NOT THE ONLY THING THAT ENDS A SESSION: a failed or completed stream + /// ends it too. Without the count check below, a broken run prints a plausible row at a + /// plausible rate and exits 0, which is the worst thing a demo whose shape ten other languages + /// copy can do. + /// + private static async Task AwaitedAsync( + string arm, + long startedAt, + Task reachedTarget, + Task session, + Func processed, + Func uniqueKeys, + int target) + { + var ended = await Task.WhenAny(reachedTarget, session).WaitAsync(ArmBudget).ConfigureAwait(false); + if (ended == session) + { + // Faulted sessions rethrow here, with the session's first fatal error. + await session.ConfigureAwait(false); + } + + var count = processed(); + return count < target + ? throw new InvalidOperationException(string.Create( + CultureInfo.InvariantCulture, $"{arm} ended early at {count} of {target}")) + : Finished(arm, Stopwatch.GetElapsedTime(startedAt), count, uniqueKeys()); + } + + private static ArmResult Finished(string arm, TimeSpan elapsed, int processed, int uniqueKeys) + { + Console.WriteLine(string.Create( + CultureInfo.InvariantCulture, + $"=== {arm} finished: {processed} records over {uniqueKeys} keys in {(long)elapsed.TotalMilliseconds}ms ===")); + return new ArmResult(arm, elapsed, processed, uniqueKeys); + } + + /// + /// The set an arm counts its distinct keys in. + /// + /// + /// Concurrent in both arms even though only one of them needs it to be: the serial arm is the + /// denominator of every ratio in both tables, and it must not differ from its numerator in how + /// it arrives at a figure the two are then compared on. + /// + private static ConcurrentDictionary NewKeySet() => new(StringComparer.Ordinal); + + /// + /// One record key, as the text the distinct-key count is taken over. + /// + /// + /// Base64 rather than the key's own characters, because keys are BYTES on this wire and an + /// arbitrary byte string is not text. A null key is one distinct key of its own - it is not the + /// same as an empty one, and it is not an absent record. + /// + private static string KeyToken(byte[]? key) => + key is null ? NullKeyToken : Convert.ToBase64String(key); + + /// A fresh group per arm per replay, so every arm reads the same records from the beginning. + private static string GroupId(string arm) => + string.Create(CultureInfo.InvariantCulture, $"pc-demo-{arm}-{Stopwatch.GetTimestamp()}"); + + /// + /// One of the demo's two tables. Same columns, same order, in every language. + /// + /// + /// FOUR OF THE FIVE FIGURES SAY SOMETHING DIFFERENT. Elapsed, msg/s and the ratio are this + /// machine on this run and mean nothing anywhere else; RECORDS and KEYS are deterministic, so + /// every language over the same backlog must print the same two numbers and any language that + /// does not has a real defect. That is what makes the table demonstrate the run rather than + /// assert it - a short arm is a FAILED arm, not a fast one - and it is what + /// bin/ci-demo-conformance.sh compares languages on. + /// + /// No latency, in either table. The backlog is pre-produced, so the workload is closed-loop and + /// a per-record timing would be flattered by however far an arm had fallen behind. + /// + /// + private static void Report( + string title, IReadOnlyList results, ArmResult? baseline, bool acrossReplays) + { + var table = new StringBuilder("\n\n").Append(title).Append('\n'); + table.AppendLine(string.Format( + CultureInfo.InvariantCulture, + " {0,-30} {1,10} {2,8} {3,10} {4,14} {5,14}", + "arm", + "records", + "keys", + "elapsed", + "msg/s", + acrossReplays ? "vs AK core*" : "vs AK core")); + foreach (var result in results) + { + var ratio = baseline is null || baseline.RatePerSecond == 0 + ? "-" + : string.Create( + CultureInfo.InvariantCulture, $"{result.RatePerSecond / baseline.RatePerSecond:F1}x"); + table.AppendLine(string.Format( + CultureInfo.InvariantCulture, + " {0,-30} {1,10} {2,8} {3,9:F1}s {4,14} {5,14}", + result.Arm, + result.Processed.ToString("N0", CultureInfo.InvariantCulture), + result.UniqueKeys.ToString("N0", CultureInfo.InvariantCulture), + (long)result.Elapsed.TotalMilliseconds / 1000d, + ((long)result.RatePerSecond).ToString("N0", CultureInfo.InvariantCulture), + ratio)); + } + + if (acrossReplays) + { + table.Append("\n * against the SMALL replay's AK core arm. Across replays, so not like-for-like.\n"); + } + + Console.WriteLine(table.ToString()); + } +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/SidecarCommand.cs b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/SidecarCommand.cs new file mode 100644 index 0000000000..1cffb1031b --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/SidecarCommand.cs @@ -0,0 +1,111 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +using Bz.Stub.ParallelConsumer.Proxy.Client.Jvm; + +namespace Bz.Stub.ParallelConsumer.Proxy.Client.Demo; + +/// +/// The sidecar binary this demo hands to the client library, and the arguments it is launched with. +/// +/// +/// THE SIDECAR IS A JVM PROGRAM IN THIS REPOSITORY, not a shipped executable, so "the binary" is a +/// java launcher and the proxy is a classpath argument. That is scaffolding, not the product +/// model: the library still launches it directly and never through a shell, because a wrapper +/// process would inherit the write end of the lifecycle pipe and defeat the sidecar's parent-death +/// signal. +/// +/// IT IS ALSO NOT A COMPOSE SERVICE, in the container or out of it. KTD41's product model is that +/// the client library spawns and supervises the sidecar itself, so the user never installs, deploys +/// or operates a process - and a demo that showed one would be teaching a deployment the product +/// does not ask for. +/// +/// +/// The absolute path of the binary to launch. +/// Its arguments, passed verbatim. +internal sealed record SidecarCommand(string Path, IReadOnlyList Arguments) +{ + /// The sidecar's real entry point - not TestModeMain, which serves fixtures. + private const string MainClass = "bz.stub.parallelconsumer.proxy.Main"; + + /// + /// The whole classpath, when the caller already knows it. The demo's container sets this, + /// because its image holds the built reactor and its Maven repository at paths this program + /// would otherwise have to rediscover. + /// + private const string ClasspathVariable = "PC_DEMO_SIDECAR_CLASSPATH"; + + /// The JVM to launch it with, when the caller has an opinion. + private const string JavaVariable = "PC_DEMO_JAVA"; + + /// + /// What run.sh runs to produce the two things below. Named in the failure message rather than + /// only in a README, because the reader who hits it is not reading a README. + /// + private const string BuildCommand = + "./mvnw --batch-mode -pl :parallel-consumer-proxy -am -DskipTests package " + + "dependency:build-classpath -Dmdep.includeScope=runtime " + + "'-Dmdep.outputFile=${project.build.directory}/sidecar-classpath.txt'"; + + private const string ProxyModule = "parallel-consumer-proxy"; + + /// Resolves the command, from the environment when it says, by discovery otherwise. + /// The launcher and its arguments. + public static SidecarCommand Resolve() => new( + JvmToolchain.JavaLauncher(Environment.GetEnvironmentVariable(JavaVariable)), + ["-cp", Classpath(), MainClass]); + + /// + /// The sidecar's classpath: the proxy's own compiled output plus its runtime dependencies. + /// + /// + /// target/classes rather than the packaged jar, deliberately - it is what a + /// package build always leaves behind, it carries the proxy's resources (its logging + /// configuration among them), and it needs no jar-name matching that a classifier could fool. + /// + private static string Classpath() + { + var supplied = Environment.GetEnvironmentVariable(ClasspathVariable); + if (!string.IsNullOrWhiteSpace(supplied)) + { + return supplied; + } + + var proxy = System.IO.Path.Combine(RepositoryRoot(), ProxyModule, "target"); + var classes = System.IO.Path.Combine(proxy, "classes"); + var dependencies = System.IO.Path.Combine(proxy, "sidecar-classpath.txt"); + if (!Directory.Exists(classes) || !File.Exists(dependencies)) + { + throw new InvalidOperationException( + $"the sidecar is not built - run '{BuildCommand}', or set {ClasspathVariable}"); + } + + return classes + System.IO.Path.PathSeparator + File.ReadAllText(dependencies).Trim(); + } + + /// + /// Walks up from this assembly to the repository root. + /// + /// + /// THE MARKER IS THE PROXY MODULE'S POM, NOT .git, and that is not a stylistic choice: + /// the demo's image is built from a context whose .dockerignore excludes .git, so + /// a .git walk finds nothing there and the container path would fail for a reason that + /// has nothing to do with the demo. The pom is present wherever the sidecar could be built. + /// + private static string RepositoryRoot() + { + var directory = new DirectoryInfo(AppContext.BaseDirectory); + while (directory is not null) + { + if (File.Exists(System.IO.Path.Combine(directory.FullName, ProxyModule, "pom.xml"))) + { + return directory.FullName; + } + + directory = directory.Parent; + } + + throw new InvalidOperationException( + $"no checkout above {AppContext.BaseDirectory} holds {ProxyModule}/pom.xml - " + + $"set {ClasspathVariable} to name the sidecar's classpath directly"); + } +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Dockerfile b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Dockerfile new file mode 100644 index 0000000000..caa4304a4d --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Dockerfile @@ -0,0 +1,73 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE DEMO CONTAINER for the .NET client (astubbs#242, plan unit U35, governed by KTD40), so that a +# reader with only Docker can run the demo - no .NET SDK, no JDK, nothing to install. +# +# IT CARRIES TWO TOOLCHAINS, WHICH NO OTHER CLIENT'S IMAGE HAS TO. The demo is .NET, and the sidecar +# it spawns is a JVM program in this repository rather than a shipped binary - so this image needs a +# .NET SDK to build the demo AND a JDK to build and run the proxy. That is scaffolding rather than +# the product model: the client library still spawns the sidecar itself, as a child process. +# +# THE BUILD CONTEXT IS THE REPOSITORY, AND THAT IS A DELIBERATE DEPARTURE from the rule the C++ +# client's Dockerfile states - the module directory is the only ordinary context, so a build never +# uploads other agents' target/ output. A Maven reactor module cannot be built without its parents +# and siblings, so this build cannot keep it. The hazard is addressed directly instead, by the +# repository-root .dockerignore that excludes every target/ directory, .git and .claude; the Java +# demo's Dockerfile makes the same argument, and those two files must be read together. +# +# .dockerignore DOES NOT COVER .NET's BUILD OUTPUT, and the RUN below is why that is not a hole: +# MSBuild writes bin/ and obj/ beside each .csproj rather than under target/, and obj/ holds a +# NuGet asset file full of absolute host paths that do not exist in this image. Left in place they +# make `dotnet build` here fail with a restore error about the developer's own home directory. The +# root .dockerignore is not edited to add them because it is shared by every client module and by +# ten concurrent worktrees; a rule this image can enforce for itself belongs in this image. +# +# WHY THE DEPENDENCIES ARE BAKED IN RATHER THAN CACHE-MOUNTED. A BuildKit cache mount on /root/.m2 +# is the obvious way to keep the Maven downloads out of the image, and it is wrong here: a cache +# mount is not part of the resulting image, so the sidecar classpath computed below would point at +# jars the running container does not have. The image is large as a result. That is the trade a +# runtime image makes and a build image does not. +# +# A .NET SDK BASE WITH A JDK ADDED, rather than a JDK base with .NET added. The .NET SDK has no +# apt package on Debian that tracks a pinned version cleanly, and the SDK image is the one that also +# pins the runtime the demo is built against. openjdk-17 is what the repository's own build +# requires, and it is available for both architectures this image is built on. +FROM mcr.microsoft.com/dotnet/sdk:8.0 + +RUN apt-get update \ + && apt-get install -y --no-install-recommends openjdk-17-jdk-headless \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /src +COPY . . + +# See the header: .dockerignore covers target/, not bin/ and obj/. +RUN find /src/parallel-consumer-proxy-clients -type d \( -name bin -o -name obj \) -prune -exec rm -rf {} + + +# THE SIDECAR. -Dcopyright.skip=true because the header check needs the repository's git history to +# find its fork point and .dockerignore excludes .git from the context; skipping it here loses +# nothing, since it is a source-tree gate that already runs in every ordinary build and in CI's own +# copyright lane. +# +# The ${project.build.directory} expression is SINGLE-QUOTED so the shell leaves it for Maven, which +# evaluates it per module - without that, -am has every module in the reactor overwrite one file. +RUN ./mvnw --batch-mode -q -DskipTests -Dcopyright.skip=true \ + -pl :parallel-consumer-proxy -am package \ + dependency:build-classpath -Dmdep.includeScope=runtime \ + '-Dmdep.outputFile=${project.build.directory}/sidecar-classpath.txt' + +# THE DEMO, and its sidecar classpath resolved ONCE at build time. Computing it at run time would +# put Maven in the startup path of a demo whose whole promise is one command. The entries point at +# /root/.m2 and at the reactor's own target/classes, both of which are in this image - which is the +# reason the Maven downloads are baked in above rather than cache-mounted. +RUN dotnet publish --nologo -c Release -o /app \ + parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo.csproj \ + && printf '%s:%s' /src/parallel-consumer-proxy/target/classes \ + "$(cat /src/parallel-consumer-proxy/target/sidecar-classpath.txt)" > /app/sidecar-cp.txt + +# PC_DEMO_SIDECAR_CLASSPATH is read from the file rather than baked into an ENV because a Dockerfile +# cannot read a file into ENV, and the demo published under /app has no repository above it to +# discover one from. PC_DEMO_ARGS is unquoted on purpose: it carries the caller's flags through +# compose as a single string and must word-split into separate arguments. +ENTRYPOINT ["/bin/sh", "-c", \ + "PC_DEMO_SIDECAR_CLASSPATH=\"$(cat /app/sidecar-cp.txt)\" exec dotnet /app/pc-dotnet-demo.dll $PC_DEMO_ARGS"] diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/README.md b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/README.md new file mode 100644 index 0000000000..512fcd39c3 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/README.md @@ -0,0 +1,156 @@ + + +# The .NET demo + +```bash +# from anywhere in the repo - picks native or container for you +parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/run.sh + +# or, from this directory, the plain container path with nothing else needed +docker compose up +``` + +Needs Docker. A .NET SDK and a JDK are optional: with both, the demo runs natively and starts its +broker in a container; without either, the demo runs in a container too and the broker is a compose +sibling. `run.sh` announces which it chose, and why, before it builds anything. + +The demo itself opens with the banner every language's demo opens with - the product's name and +what is about to happen - then the effective configuration, then the run: + +``` +================================================================ + PARALLEL CONSUMER - .NET demo + The same records, twice: one at a time, then all at once. +================================================================ +``` + +**The contract this keeps - and that every other language's demo keeps - is +[`parallel-consumer-proxy/demo/README.md`](../../../parallel-consumer-proxy/demo/README.md).** Read +that first. This file only records what is specific to .NET. + +## The two arms + +| arm, as the tables label it | what it is | +|---|---| +| `AK core (Confluent.Kafka)` | `Confluent.Kafka`, one record at a time, in this process | +| `pc-dotnet-grpc (this client)` | this module's client library, over a sidecar it spawns itself | + +**Each label carries the role and the client**, because "AK core" is a category rather than a +client and a reader cannot judge a comparison without knowing what produced it. In .NET the answer +is `Confluent.Kafka`, the `librdkafka` binding Confluent publishes, and the contract asks a +language with a second serious client to say so here and consider running both. **.NET does not +appear to have one.** The packages a reader is likely to meet - KafkaFlow, Streamiz - are +frameworks built **on top of** `Confluent.Kafka` rather than protocol implementations of their own, +so a second arm would price a wrapper over the same client and not a different client. If that is +wrong, the fix is a third arm here rather than a footnote. + +On the `pc-dotnet-grpc` path **the application does no Kafka I/O**: the sidecar owns the consumer, the +producer, the group membership and the offsets. That is a statement about the *path*, not about the +process - the same process creates the topic, produces the backlog and runs the `AK core` arm with +an ordinary Kafka client, because a comparison needs both sides. A genuinely foreign application +carries no Kafka client library at all, which is the property this arm stands in for. + +Two arms is the whole contract outside Java. Java carries four more (`pc-core`, `java-direct`, +`java-grpc-uds`, `java-raw-grpc`) because one JVM can hold every one of them at once and each pair +changes exactly one term; here the two arms are two different client libraries as well as two +different engines, so a difference between them is not a wire cost and must not be read as one. + +## What the tables print + +Five columns, the same five in both tables and in the same order: + +``` + arm elapsed msg/s vs AK core records keys + AK core (Confluent.Kafka) 4.9s 4 1.0x 20 20 + pc-dotnet-grpc (this client) 1.7s 11 2.9x 20 20 +``` + +`records` and `keys` are the **deterministic** pair. Elapsed, msg/s and the ratio describe one +machine on one run and travel nowhere; every language over the same backlog must print the same +record count and the same distinct-key count, so those two are what `bin/ci-demo-conformance.sh` +can compare across languages. They also make the table demonstrate the run rather than assert it - +a throughput figure cannot show the work happened, and an arm that is **short** is a failed arm +rather than a fast one. + +Keys are counted over the record key's **bytes** - base64, because a Kafka key is not text - and a +null key counts as one distinct key of its own. There is no latency column, in either table, and +that is contract: the backlog is pre-produced, so the workload is closed-loop and a per-record +timing would be flattered by however far an arm had fallen behind. + +## What is specific to .NET + +### The simulated work is an awaited timer, not a blocking sleep - and it is no longer a divergence + +This module's demo diverged from the contract here, and **the contract has since moved to meet it**. +The rule used to name languages and listed C# as safe for a blocking sleep; it now asks a question +about the *client* instead - is it thread-per-record? - and C# is one of the six languages the +question rules out. + +It rules this one out because the library's executors are `Task`s on the thread pool: a hundred of +them sitting in `Thread.Sleep` is a hundred pool threads occupied, and the pool injects +replacements at roughly one per second once its core count is used up. The sidecar arm would then +report the thread pool's injection rate rather than the engine's throughput - a number that looks +like a measurement and is not one. + +So the wait is `await Task.Delay(...)`, which is what "non-occupying" means in a language whose +concurrency is tasks rather than threads. **Both arms use it.** The `AK core` arm holds one record +at a time and so could not starve anything, but it is the denominator of every ratio in both tables, +and an arm must not differ from its numerator by the wait primitive as well as by the transport. + +### The sidecar is a JVM, so "the binary" is `java` and the proxy is a classpath + +The client library takes an absolute path to a binary and launches it directly, never through a +shell - a wrapper process would inherit the write end of the lifecycle pipe and defeat the sidecar's +parent-death signal. In this repository the sidecar is not a shipped executable but +`parallel-consumer-proxy`'s `Main`, so the demo hands the library a `java` launcher with `-cp` and +the main class. That is scaffolding for a repository that builds its sidecar from source; the +product model is unchanged, and the sidecar is **not** a compose service in either mode. + +`run.sh` builds that classpath with Maven before it starts the demo; the container bakes it in at +image build time and passes it as `PC_DEMO_SIDECAR_CLASSPATH`. + +### The broker address is normalised before it travels + +Testcontainers for .NET returns its address as a URI - measured here as +`plaintext://127.0.0.1:62347/`, trailing slash included. librdkafka accepts that string; the Java +client behind the sidecar rejects it, so the address is reduced to `host:port` before it goes +anywhere. It cost the sidecar arm a whole run to find, because R48 deliberately withholds the reason +from the proxy's error - a Kafka `ConfigException` embeds property values, and those may be +credentials. + +### Three extra environment variables, and none is part of the contract + +| variable | what it does | +|---|---| +| `PC_DEMO_SIDECAR_CLASSPATH` | the sidecar's classpath, instead of discovering it from the checkout. The container sets it | +| `PC_DEMO_SIDECAR_LOG` | any value sends the sidecar's standard error to this demo's, which is the first thing to try when the sidecar arm will not start | +| `PC_DEMO_JAVA` | the JVM to launch the sidecar with, ahead of `JAVA_HOME` and `PATH` | + +The contract's own variables - `PC_DEMO_RECORDS`, `PC_DEMO_DELAY_MS`, `PC_DEMO_CONCURRENCY`, +`PC_DEMO_PARTITIONS`, `PC_DEMO_REPLAY_FACTOR`, `PC_DEMO_BOOTSTRAP`, `PC_DEMO_TOPIC` - are exactly +the flags, and flags beat the environment beats the defaults. + +### The demo project is in the module's one solution + +`dotnet build`, `dotnet test` and the CI row's `dotnet format analyzers` all run at the module root +with no project argument, and error out if they find more than one candidate there - so the demo +joined `Bz.Stub.ParallelConsumer.Proxy.Client.sln` rather than adding a second solution. It also +means the module's ordinary build keeps the demo compiling, under the same analyzers-as-errors lint +as the library. + +## What has been run, and what has not + +Both entry points, by hand, at `--records 20` - the native one with no arguments at all (the case +that has broken before, scaled down by `PC_DEMO_*` variables) and with explicit flags and a big +replay; the container one through `run.sh --docker`. Both arms completed in every run and both +exited 0, and the deterministic columns agree with what the seeding predicts: 20 records over 20 +keys in the small replay, 40 over 40 in the big one. + +**No run at the contract's defaults has ever happened**, and nothing here should be read as a +measurement: at twenty records both arms are dominated by consumer-group join time, and the runs +above were taken on a machine running ten agents at once. The `msg/s` and `vs AK core` columns in +this file's example are shape, not a figure anybody should quote. + +`bin/ci-demo-test.sh` runs the **Java** demo through both of its entry points on every pull request. +Nothing runs this one in CI yet - which is exactly the gap that script exists to close. Open work, +with the rest, in [`docs/inflight/clients/dotnet.md`](../../../docs/inflight/clients/dotnet.md). diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/docker-compose.yml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/docker-compose.yml new file mode 100644 index 0000000000..546e46b2f8 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/docker-compose.yml @@ -0,0 +1,108 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE ONE-COMMAND PATH for the .NET demo (astubbs#242, plan unit U35, R20). +# +# THE BROKER IS A SIBLING, AND THAT IS A RULE RATHER THAN A LAYOUT CHOICE. The demo could start its +# own broker with Testcontainers - it does exactly that on the native path - but doing so from +# inside a container requires mounting the host Docker socket, and U35 forbids it: a documented +# socket mount is root-equivalent host access taught as the normal way to run the product. So +# broker mode inside a container reaches a compose sibling on the demo's own network, and the demo +# container is given an address instead of a socket. +# +# THE SIDECAR IS NOT A SERVICE HERE, deliberately. It would be easy to add one, and it would teach +# the wrong thing: KTD41's product model is that the client library spawns and supervises the +# sidecar itself, so the user never installs, deploys or operates a process. A compose service +# would show a deployment the product does not ask for. The demo spawns it as a child process +# inside its own container, exactly as it does natively. +# +# TWO WAYS IN, AND BOTH ARE SUPPORTED: +# +# cd && docker compose up # the plain one, defaults, nothing else needed +# demo/run.sh --docker [--records N ...] # the same thing with the demo's flags +# +# `docker compose up` on its own is the path to keep working: it is what a reader who has never seen +# this repository will type. It needs no arguments, no environment and no wrapper script - the demo +# service runs with the demo's own defaults, and PC_DEMO_ARGS below is empty unless run.sh sets it. +# Add --build after changing any source, and `docker compose down` to remove the broker afterwards. +# +# The demo container exits when the run finishes; the broker keeps running until you stop it. +# run.sh passes --abort-on-container-exit --exit-code-from demo so that a scripted run stops on its +# own and reports the demo's exit code rather than compose's. + +name: pc-dotnet-demo + +services: + broker: + # Pinned as a literal because a compose file cannot derive anything. It must track + # in the root pom - the mapping is CP major = AK major + 4, so AK 3.9.x means + # CP 7.9 - and the same literal appears in DemoBroker.cs, which cannot derive it either: the + # .NET Kafka client's version tracks librdkafka, not a broker release. + image: confluentinc/cp-kafka:7.9.0 + environment: + CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk + # QUIET. A demo is a thing someone WATCHES, and a KRaft broker at INFO emits hundreds of + # lines of its own configuration, controller elections and log-segment chatter - which buries + # the two tables the demo exists to show. Kafka's own warnings and errors still come through. + KAFKA_LOG4J_ROOT_LOGLEVEL: WARN + KAFKA_TOOLS_LOG4J_LOGLEVEL: WARN + # ROOT ALONE IS NOT ENOUGH, and this was measured rather than assumed: the cp-kafka image + # writes kafka=INFO, kafka.controller=TRACE and state.change.logger=TRACE into its own + # log4j config, and a named logger beats the root. With only the root set, one demo run + # interleaved 927 broker lines with 35 of the demo's own. These are the named loggers that + # actually produce the noise. + KAFKA_LOG4J_LOGGERS: "kafka=WARN,kafka.controller=WARN,kafka.server=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.network=WARN,kafka.request.logger=WARN,state.change.logger=WARN,org.apache.kafka=WARN" + KAFKA_NODE_ID: 1 + KAFKA_PROCESS_ROLES: broker,controller + # Bound to the service name, not 0.0.0.0: this image formats its KRaft storage with the + # same config it boots with, and the format step refuses a non-routable advertised address. + KAFKA_LISTENERS: PLAINTEXT://broker:9092,CONTROLLER://broker:9093 + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092 + KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT + KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093 + KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + # The demo forms one consumer group per arm, one after another, and the default three-second + # settling delay would be charged to every one of them. The native path sets the same value. + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 500 + healthcheck: + # The demo must not start before the broker can serve an API request. Listing topics is the + # cheapest call that proves that, and it proves more than a port being open does. + test: ["CMD", "kafka-topics", "--bootstrap-server", "broker:9092", "--list"] + interval: 5s + timeout: 10s + retries: 30 + start_period: 10s + + demo: + build: + # The repository, because a Maven reactor module cannot build without its parents and + # siblings, and the demo's sidecar is one of those modules. The root .dockerignore is what + # keeps this from uploading every target/ directory in the tree; the Dockerfile explains why + # that pair had to exist, and what .dockerignore does not cover for .NET. + context: ../../.. + dockerfile: parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/Dockerfile + # A demo that has printed its tables is done. Without this, compose's default would restart a + # container that exited cleanly and the reader would watch the whole run happen again. + restart: "no" + depends_on: + broker: + condition: service_healthy + environment: + # Supplying this is what stops the demo trying to start a broker of its own. + PC_DEMO_BOOTSTRAP: broker:9092 + # The flags demo/run.sh was called with, forwarded as one string. + PC_DEMO_ARGS: ${PC_DEMO_ARGS:-} + # EVERY OTHER DIAL, forwarded explicitly. Compose passes nothing it is not told to, so a + # missing line here is a dial the container silently ignores while the run still prints a + # fingerprint claiming otherwise. Caught across seven languages at once by + # bin/ci-demo-conformance.sh, which is why that check exists. Adding a flag means adding it + # here too. + PC_DEMO_RECORDS: ${PC_DEMO_RECORDS:-} + PC_DEMO_DELAY_MS: ${PC_DEMO_DELAY_MS:-} + PC_DEMO_CONCURRENCY: ${PC_DEMO_CONCURRENCY:-} + PC_DEMO_PARTITIONS: ${PC_DEMO_PARTITIONS:-} + PC_DEMO_REPLAY_FACTOR: ${PC_DEMO_REPLAY_FACTOR:-} + PC_DEMO_TOPIC: ${PC_DEMO_TOPIC:-} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/run.sh b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/run.sh new file mode 100755 index 0000000000..67658a5f28 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/demo/run.sh @@ -0,0 +1,129 @@ +#!/usr/bin/env bash +# +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE .NET DEMO: the same records through .NET's own Kafka client, one at a time, and through this +# module's client library over a sidecar it spawns itself. One command, no setup. +# +# This mirrors the reference demo's interface EXACTLY - same flags, same defaults, same environment +# variables, same two tables out. The contract is parallel-consumer-proxy/demo/README.md; the +# reference implementation of it is the Java demo's run.sh, one directory family over. +set -euo pipefail + +cd "$(git rev-parse --show-toplevel)" + +MODULE_DIR="parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet" +DEMO_DIR="$MODULE_DIR/demo" +PROJECT="$DEMO_DIR/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/Bz.Stub.ParallelConsumer.Proxy.Client.Demo.csproj" +DEMO_BIN="$DEMO_DIR/Bz.Stub.ParallelConsumer.Proxy.Client.Demo/bin/Debug/net8.0/pc-dotnet-demo" + +usage() { + cat <<'USAGE' +usage: demo/run.sh [options] + + --records N records in the comparison replay (default 2000) + --delay-ms N simulated work per record, ms (default 2) + --concurrency N max in-flight records (default 100) + --partitions N partitions on the demo topic (default 10) + --replay-factor N big replay = records x N; 1 skips (default 20) + --bootstrap ADDR an existing broker; omit to start one + --topic NAME an existing topic; omit to create one + + --docker run in a container, broker as a compose sibling + --native run on this machine's own toolchains + -h, --help this + +With neither --docker nor --native, the demo picks: native when a .NET SDK and a JDK are both +present, a container otherwise. It says which it chose, and why, on the first line of output. + +Every flag also has an environment variable: --delay-ms is PC_DEMO_DELAY_MS. +USAGE +} + +# ${args[@]+...} rather than "${args[@]}": under `set -u`, bash 3.2 - which is what macOS ships, +# and what a first-time user runs this with - treats an EMPTY array expansion as an unbound +# variable and aborts. That fails exactly the no-argument case, which is the double-click case. +args=() +mode="auto" +while [ $# -gt 0 ]; do + case "$1" in + --records|--delay-ms|--concurrency|--partitions|--replay-factor|--bootstrap|--topic) + if [ $# -lt 2 ]; then + echo "$1 needs a value" >&2 + exit 2 + fi + args+=("$1" "$2"); shift 2 ;; + --docker) mode="docker"; shift ;; + --native) mode="native"; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "unknown option: $1" >&2; usage >&2; exit 2 ;; + esac +done + +have_dotnet_sdk() { + # `dotnet --list-sdks` rather than `command -v dotnet`: a machine can carry the RUNTIME alone, + # which runs an app but cannot build one, and that fails three minutes in rather than here. + command -v dotnet >/dev/null 2>&1 && [ -n "$(dotnet --list-sdks 2>/dev/null)" ] +} + +have_jdk_toolchain() { + # The sidecar is a JVM program in this repository, so the native path needs a JDK as well as a + # .NET SDK - `mvnw -v` proves the wrapper has one it can actually run. + [ -x ./mvnw ] && ./mvnw -v >/dev/null 2>&1 +} + +if [ "$mode" = "auto" ]; then + if have_dotnet_sdk && have_jdk_toolchain; then + mode="native" + echo "Mode: native - a .NET SDK and a JDK toolchain are both present on this machine." + else + mode="docker" + echo "Mode: container - this machine is missing a .NET SDK or a JDK, so the demo brings both." + fi +else + echo "Mode: $mode - asked for explicitly." +fi + +if ! docker info >/dev/null 2>&1; then + if [ "$mode" = "docker" ]; then + echo "Docker is not running - the demo needs it for the broker and for its own image." >&2 + else + echo "Docker is not running - the demo starts its broker in a container." >&2 + fi + exit 1 +fi + +if [ "$mode" = "docker" ]; then + # The broker is a COMPOSE SIBLING and the demo container is never given the host Docker socket: + # a documented socket mount is root-equivalent host access taught as the normal way to run the + # product (plan unit U35). PC_DEMO_ARGS carries the flags through to the container, where the + # same parser reads them. + PC_DEMO_ARGS="${args[*]+${args[*]}}" \ + exec docker compose -f "$DEMO_DIR/docker-compose.yml" up \ + --build --abort-on-container-exit --exit-code-from demo +fi + +# THE SIDECAR, BUILT BEFORE THE DEMO THAT SPAWNS IT. The client library launches a binary; in this +# repository that binary is a JVM and the proxy is a classpath, so the classpath has to exist before +# the demo runs. It is written into the proxy module's own target/ by ${project.build.directory}, +# which is evaluated PER MODULE - without that, `-am` has every module in the reactor write the same +# file and the last one to build wins, which is whichever module Maven happened to finish with. +# +# -am is not optional: this module's parent is not in a single-module reactor and the enforcer's +# ReactorModuleConvergence rule fails the build before anything runs without it. +echo "Building the sidecar (parallel-consumer-proxy) and its classpath..." +./mvnw --batch-mode -q -pl :parallel-consumer-proxy -am -DskipTests package \ + dependency:build-classpath -Dmdep.includeScope=runtime \ + '-Dmdep.outputFile=${project.build.directory}/sidecar-classpath.txt' + +# NoSummary as well as quiet: the "Build succeeded / 0 Warning(s) / Time Elapsed" block is four +# lines of scaffolding immediately before the demo's own banner, and the banner is the first thing a +# reader is meant to see. Build ERRORS are not part of that summary and still print. +echo "Building the demo..." +dotnet build --nologo --verbosity quiet -consoleLoggerParameters:NoSummary "$PROJECT" + +# The built executable, not `dotnet run`, and not because it is faster: `dotnet run` interposes a +# process between this script and the demo, and the demo is the process whose lifecycle pipe keeps +# the spawned sidecar alive. Launching the apphost directly keeps the parent-death signal a +# straight line, and gives a scripted caller the demo's own exit code rather than a wrapper's. +exec "$DEMO_BIN" ${args[@]+"${args[@]}"} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/shared/JvmToolchain.cs b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/shared/JvmToolchain.cs new file mode 100644 index 0000000000..3e551d1ea1 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/shared/JvmToolchain.cs @@ -0,0 +1,68 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +namespace Bz.Stub.ParallelConsumer.Proxy.Client.Jvm; + +/// +/// Finding a JVM to launch the sidecar with - the one thing this module's test harness and its demo +/// genuinely do the same way. +/// +/// +/// IT IS COMPILED INTO BOTH PROJECTS BY LINK, not shared through an assembly, because neither the +/// test project nor the demo is a library anybody references, and inventing a third assembly to +/// hold twenty-five lines would cost more than it saves. A Compile Include item in each +/// .csproj points here; there is no project reference to follow. +/// +/// WHAT IS DELIBERATELY NOT HERE: the classpath. The harness wants the proxy's TEST classpath, to +/// reach TestModeMain; the demo wants its RUNTIME classpath, to reach the real Main - +/// and the harness resolves it by shelling out to Maven while the demo is handed one by its +/// entry-point script. Two different answers to two different questions, so they stay apart. +/// +/// +internal static class JvmToolchain +{ + /// + /// Resolves the JVM launcher: the caller's explicit choice, then JAVA_HOME, then + /// PATH. + /// + /// + /// A PATH lookup is acceptable HERE and nowhere else. This is scaffolding choosing a JVM to run + /// the sidecar with, not a client library choosing which binary receives a user's Kafka + /// credentials - the library still refuses a sidecar path that is not absolute, which is why + /// this returns one. + /// + /// + /// The launcher the caller was told to use, if any - each caller reads its own environment + /// variable, because the demo's is not the test suite's. + /// + /// An absolute path to a java launcher. + /// When no JVM can be found. + public static string JavaLauncher(string? explicitLauncher) + { + if (!string.IsNullOrEmpty(explicitLauncher)) + { + return explicitLauncher; + } + + var javaHome = Environment.GetEnvironmentVariable("JAVA_HOME"); + if (!string.IsNullOrEmpty(javaHome)) + { + var candidate = Path.Combine(javaHome, "bin", "java"); + if (File.Exists(candidate)) + { + return candidate; + } + } + + foreach (var entry in (Environment.GetEnvironmentVariable("PATH") ?? string.Empty) + .Split(Path.PathSeparator, StringSplitOptions.RemoveEmptyEntries)) + { + var candidate = Path.Combine(entry, "java"); + if (File.Exists(candidate)) + { + return Path.GetFullPath(candidate); + } + } + + throw new InvalidOperationException("no JVM found - set JAVA_HOME or name one explicitly"); + } +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests.csproj b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests.csproj index 69b8641e1c..fdd8a5f64a 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests.csproj +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests.csproj @@ -32,4 +32,9 @@ + + + + + diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests/ConformanceHarness.cs b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests/ConformanceHarness.cs index 258569f16d..ae3f86d4c5 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests/ConformanceHarness.cs +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-dotnet/tests/Bz.Stub.ParallelConsumer.Proxy.Client.Tests/ConformanceHarness.cs @@ -1,5 +1,9 @@ // Copyright (C) 2026 Antony Stubbs and contributors +using System.Diagnostics; +using Bz.Stub.ParallelConsumer.Proxy.Client.Jvm; + + namespace Bz.Stub.ParallelConsumer.Proxy.Client.Tests; /// @@ -104,41 +108,14 @@ private static string RepositoryRoot() } /// - /// Resolves the JVM launcher. A PATH lookup is acceptable HERE and nowhere else: this is test - /// scaffolding choosing a JVM, not a client library choosing which sidecar receives the user's - /// Kafka credentials. + /// Resolves the JVM launcher, naming this suite's own override variable. /// - private static string JavaLauncher() - { - var explicitJava = Environment.GetEnvironmentVariable("PC_PROXY_TEST_JAVA"); - if (!string.IsNullOrEmpty(explicitJava)) - { - return explicitJava; - } - - var javaHome = Environment.GetEnvironmentVariable("JAVA_HOME"); - if (!string.IsNullOrEmpty(javaHome)) - { - var candidate = System.IO.Path.Combine(javaHome, "bin", "java"); - if (File.Exists(candidate)) - { - return candidate; - } - } - - foreach (var entry in (Environment.GetEnvironmentVariable("PATH") ?? string.Empty) - .Split(System.IO.Path.PathSeparator, StringSplitOptions.RemoveEmptyEntries)) - { - var candidate = System.IO.Path.Combine(entry, "java"); - if (File.Exists(candidate)) - { - return System.IO.Path.GetFullPath(candidate); - } - } - - throw new InvalidOperationException( - "no JVM found - set JAVA_HOME or PC_PROXY_TEST_JAVA"); - } + /// + /// The lookup itself is shared with the demo, which spawns the same sidecar the same way - + /// see shared/JvmToolchain.cs, compiled into both projects by link. + /// + private static string JavaLauncher() => + JvmToolchain.JavaLauncher(Environment.GetEnvironmentVariable("PC_PROXY_TEST_JAVA")); /// The sidecar's classpath, as Maven resolved it. /// diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/README.md b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/README.md index 39ba46cd7a..147e12ef06 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/README.md +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/README.md @@ -90,8 +90,17 @@ so a session negotiates nothing else and the proxy sends nothing else. Not implemented, and not silently half-implemented - the capability is simply not declared: heartbeats and the liveness lease, the `Manifest` reconnect and `Drop`, `WorkerDied`, `Terminal` -outcomes, the proxy-initiated `Shutdown` drain and the `Released` outcome. Also absent: the demo -and its container, packaging, and the rest of the conformance suite. +outcomes, the proxy-initiated `Shutdown` drain and the `Released` outcome. Also absent: packaging, +and the rest of the conformance suite. + +**The demo has landed** - [`demo/`](demo/), with its container. It is the one artefact here that a +reader runs rather than reads, and it drives *this* library over a real sidecar against a real +broker, alongside an arm using Go's own Kafka client for comparison: + +```bash +demo/run.sh # native or container, whichever this machine can do +cd demo && docker compose up # the plain container path, nothing else needed +``` The current findings, including the ones the frozen documents could not answer, are in [`docs/inflight/clients/go.md`](../../docs/inflight/clients/go.md). diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/.gitignore b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/.gitignore new file mode 100644 index 0000000000..4c40deb674 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/.gitignore @@ -0,0 +1,6 @@ +# `go build ./...` in the PARENT module writes nothing, because it matches several packages and the +# toolchain then compiles as a check and discards the objects - the module's pom says so at length. +# THIS directory is one `main` package, so the same command DOES write a binary here, named after +# the directory. run.sh and the Dockerfile both use `go build -o` and put theirs in target/; this +# catches the one a developer produces by hand. +/demo diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/Dockerfile b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/Dockerfile new file mode 100644 index 0000000000..86fe087b0e --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/Dockerfile @@ -0,0 +1,87 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE DEMO CONTAINER for the Go client (astubbs#242, plan unit U35, governed by KTD40). +# +# TWO TOOLCHAINS, WHICH IS WHY THIS IS A MULTI-STAGE BUILD AND THE JAVA ONE IS NOT. The demo itself +# is a Go binary, and it needs no Go at run time - Go links statically. The SIDECAR it spawns is a +# JVM application, and that toolchain has to be in the final image because the sidecar is a child +# process of the running demo, not a build artifact. So: Go builds and is thrown away, the JDK +# stays. +# +# THE BUILD CONTEXT IS THE REPOSITORY, AND THAT IS A DELIBERATE DEPARTURE. The C++ Dockerfile states +# the rule - "the module directory is the only ordinary context, so the build never uploads other +# agents' target/ output" - and it is a good rule that this build cannot keep, for the same reason +# the Java demo cannot: a Maven reactor module cannot be built without its parents and its sibling +# modules, and the sidecar is one. The hazard the rule protects against is addressed directly +# instead, by the repository-root .dockerignore that excludes every target/ directory, .git and +# .claude/worktrees. Those two files must be read together; if you add a build output directory to +# this repo, add it there too. +# +# WHY THE MAVEN DEPENDENCIES ARE BAKED IN RATHER THAN CACHE-MOUNTED. A BuildKit cache mount on +# /root/.m2 is the obvious way to keep the downloads out of the image, and it is wrong here: a cache +# mount is not part of the resulting image, so the sidecar classpath computed below would point at +# jars the running container does not have. The image is large as a result. That is the trade a +# runtime image makes and a build image does not. + +# --------------------------------------------------------------------------------------------- +# Stage 1: the demo binary. +# +# ONLY THE GO CLIENT MODULE IS COPIED IN, even though the context is the whole repository - the +# demo module's `replace` directive points at ../, so the client library beside it is the entire +# rest of what this stage needs. Copying less also means a change anywhere else in the repo does not +# invalidate this layer. +# +# CGO_ENABLED=0 because the binary is copied into a different base image below; a dynamically linked +# one would be looking for the golang image's libc. Nothing here needs cgo - franz-go and the client +# library are pure Go. +# --------------------------------------------------------------------------------------------- +FROM golang:1.25-bookworm AS demo-build + +WORKDIR /src +COPY parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go ./parallel-consumer-proxy-client-go +RUN cd parallel-consumer-proxy-client-go/demo \ + && CGO_ENABLED=0 go build -o /out/pc-go-demo . + +# --------------------------------------------------------------------------------------------- +# Stage 2: the image that runs, carrying the JVM the spawned sidecar needs. +# +# A PLAIN JDK, NOT THE `maven:` IMAGE, and that is a bug fix rather than a preference. The official +# Maven image sets MAVEN_CONFIG=/root/.m2, and the Maven WRAPPER appends $MAVEN_CONFIG to its own +# command line - so `./mvnw ` inside that image dies with `Unknown lifecycle phase +# "/root/.m2"`. Using the image's own `mvn` would dodge it and break the repo's rule that builds go +# through the wrapper (AGENTS.md, "Build Requirements"). A JDK base keeps the wrapper and removes +# the conflict: .mvn/wrapper/maven-wrapper.jar is committed, so the wrapper needs nothing but Java. +# --------------------------------------------------------------------------------------------- +FROM eclipse-temurin:17-jdk + +WORKDIR /src +COPY . . + +# ONE Maven invocation, and the sidecar's classpath computed at BUILD time into a file. Computing it +# at run time would put Maven in the startup path of a demo whose whole promise is one command. +# +# -Dcopyright.skip=true because the header check needs the repository's git history to find its fork +# point, and .dockerignore excludes .git from the context. Skipping it here loses nothing: it is a +# source-tree gate that already runs in every ordinary build and in CI's own copyright lane, and +# re-running it inside a demo image would prove nothing new about the sources it just copied. +# +# target/classes is appended rather than the built jar, so nothing has to guess the jar's name. +RUN ./mvnw --batch-mode -q -DskipTests -Dcopyright.skip=true \ + -pl :parallel-consumer-proxy -am \ + package dependency:build-classpath \ + '-Dmdep.outputFile=${project.build.directory}/sidecar-classpath.txt' \ + && mkdir -p /app \ + && cp parallel-consumer-proxy/target/sidecar-classpath.txt /app/sidecar-classpath.txt \ + && printf ':/src/parallel-consumer-proxy/target/classes' >> /app/sidecar-classpath.txt + +COPY --from=demo-build /out/pc-go-demo /app/pc-go-demo + +# Where the demo reads the sidecar's classpath from. It is plumbing rather than an eighth flag - +# there is no --sidecar-classpath and no default a user would set - and it exists only because the +# sidecar ships as a JVM application rather than a binary. run.sh sets the same variable natively. +ENV PC_DEMO_SIDECAR_CLASSPATH=/app/sidecar-classpath.txt + +# PC_DEMO_ARGS is unquoted on purpose: it carries the caller's flags through compose as a single +# string and must word-split into separate arguments. Empty is the ordinary case - `docker compose +# up` with no arguments must run the demo with its own defaults. +ENTRYPOINT ["/bin/sh", "-c", "exec /app/pc-go-demo $PC_DEMO_ARGS"] diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/README.md b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/README.md new file mode 100644 index 0000000000..bd2794abba --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/README.md @@ -0,0 +1,175 @@ +# The Go demo + +```bash +# from anywhere in the repo - picks native or container for you +parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/run.sh + +# or, from this directory, the plain container path with nothing else needed +docker compose up +``` + +Needs Docker. A Go toolchain and a JDK are optional: with both, the demo runs natively and starts +its broker in a container; without either, the demo runs in a container too and the broker is a +compose sibling. `run.sh` announces which it chose, and why, before it starts building. + +**The contract this keeps - and that every other language's demo keeps - is +[`parallel-consumer-proxy/demo/README.md`](../../../parallel-consumer-proxy/demo/README.md).** +Read that first. This file only records what is specific to Go. + +## The two arms + +| arm | what runs | +|---|---| +| **AK core (franz-go)** | [franz-go](https://github.com/twmb/franz-go), one record at a time, in this process | +| **pc-go-grpc (this client)** | this process as a **foreign client**, through the Go client library, over a real sidecar the library spawns as a child process | + +Both rows name the client that produced them, because **"AK core" is a category and not a client**: +the answer is franz-go here, `rdkafka` in Ruby, `kafkajs` in TypeScript, and a reader cannot judge +the comparison without knowing which one ran. + +On the `pc-go-grpc` path the application does no Kafka I/O: the sidecar owns the consumer, the +producer, the group membership and the offsets. That is a claim about the *path*, not about the +process - the same binary creates the topic, seeds the backlog and runs the AK core arm with +franz-go, because a comparison needs both sides. + +Two arms is the whole contract outside Java, which is the seed precisely because one JVM can hold +every engine at once and so price the sidecar hop with everything else held identical. Here the two +arms are different client libraries as well as different engines, so the gap between them is not +attributable to the wire. + +**The arm goes through the client library, never through hand-written gRPC.** An earlier version of +the Java seed spoke the protocol by hand; it proved the *engine* worked and said nothing about the +*client library*, which is the artifact users actually touch. + +## What is specific to Go + +### Go has three serious Kafka clients, and this demo runs one of them + +The contract asks a language with more than one serious client to say so, and to consider running +more than one as separate arms - because the choice materially changes the number, and a reader +asking "is this fast in my language" is really asking about the client they already use. Go has +three: + +| client | what it is | why it is not a second arm here | +|---|---|---| +| [franz-go](https://github.com/twmb/franz-go) | pure Go, the one this demo runs | - | +| [confluent-kafka-go](https://github.com/confluentinc/confluent-kafka-go) | a cgo binding to librdkafka | needs cgo. `demo/Dockerfile` builds with `CGO_ENABLED=0` precisely so the binary produced in the `golang` stage can *run* in the `eclipse-temurin` stage the spawned JVM sidecar needs; a dynamically linked one would be looking for the build image's libc | +| [sarama](https://github.com/IBM/sarama) | pure Go, the long-established one | cheap to add - one more serial arm costs `records x delay-ms` - but see below | + +**The judgement: name them here, run one.** The blocker is not cost, it is the drift check. +[`bin/ci-demo-conformance.sh`](../../../bin/ci-demo-conformance.sh) proves the eleven demos still +behave alike by requiring their output skeletons to be *identical*, arm rows included, and it +exempts exactly one language - Java, which is documented as carrying extra diagnostic arms. A third +row in Go and nowhere else is permanent drift, so the harness would fail on a demo that is doing +what the contract invited. + +A reader who wants the sarama number is one arm away, and adding it is the right change once the +harness can express "this language legitimately carries an extra arm". Until then a divergence that +breaks the check for the other ten languages is the wrong trade, and this table is the honest +version of the same information: **the AK core row is franz-go's number, not Go's**. + +### The demo binary never starts a broker - `run.sh` does + +This is the one place the Go demo's *shape* differs from the Java reference, and the integrator +should know it is deliberate. + +Java's `DemoBroker` starts a Testcontainers broker when no `--bootstrap` is supplied. Go has no +comparable dependency here, and adding one would put a Docker client library and its transitive +tree into a demo whose point is that the application needs no infrastructure. So the responsibility +moves one level out: + +- **natively**, `run.sh` starts a `cp-kafka` container on a free port and removes it on exit; +- **in the container**, the compose sibling is the broker and `PC_DEMO_BOOTSTRAP` names it; +- **the binary always receives an address**, and says so plainly if it does not. + +The user-facing contract is unchanged - omit `--bootstrap` and a broker appears - and the rule that +produced the container half is the same either way: **a demo container is never granted the host +Docker socket**, so a containerised demo reaches a broker it did not start. + +### The sidecar is a JVM application, so its "binary" is the JVM launcher + +The client library asks for an absolute path to a binary and spawns it directly, with no shell in +between, because the pipe it holds is the sidecar's parent-death signal. The proxy has no native +launcher yet, so the demo passes the JVM launcher as that binary and the proxy as a `-cp` argument. +`run.sh` computes that classpath once with Maven and hands it over in a file named by +`PC_DEMO_SIDECAR_CLASSPATH`; the Dockerfile bakes both in at image build time. + +**`PC_DEMO_SIDECAR_CLASSPATH` is plumbing, not an eighth flag.** It has no `--flag`, no default a +user would set, and it disappears the day the sidecar ships as a binary. Every variable the contract +*does* specify - one per flag, `PC_DEMO_` plus the flag in upper snake case - is implemented exactly +as written, and `options_test.go` derives the names from the flag list rather than trusting seven +hand-written bindings. + +### A blocking sleep is the right simulated work here + +The contract singles out Python (worker *processes*) and TypeScript (one event loop) as the two +languages that must not use a blocking sleep. Go is not one of them: a sleeping goroutine is as +cheap as a sleeping thread, so both arms use `time.Sleep` and differ by transport and engine only. + +### The demo is a nested Go module + +`demo/go.mod` is separate from the client library's, and the library is reached through a `replace` +directive. Go's module graph propagates requirements to every consumer, so franz-go in the +library's `go.mod` would hand a Kafka client library to applications whose whole reason for using +the proxy is not needing one. + +The consequence worth knowing: **`go build ./...` and `go test ./...` in the parent module do not +descend into this directory**, so the module's Maven-driven build never compiles the demo. `run.sh` +and the Dockerfile do, and both entry points are meant to be exercised - a demo with one tested +entry point has an untested entry point. + +Run the demo's own tests directly: + +```bash +cd parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo && go test ./... +``` + +## What it prints + +**A banner first**, naming the product before anything else does - identical in all eleven +languages bar the language's own name. Then the effective configuration, because a number without +its settings is not reproducible, and then one table per replay: + +``` +================================================================ + PARALLEL CONSUMER - Go demo + The same records, twice: one at a time, then all at once. +================================================================ + +Small replay - every arm over the same 2000 records (the comparison) + arm records keys elapsed msg/s vs AK core + AK core (franz-go) 2,000 1,000 4.1s 487 1.0x + pc-go-grpc (this client) 2,000 1,000 0.3s 6,250 12.8x +``` + +*(the shape of the output, not a measurement: elapsed and msg/s depend entirely on the machine.)* + +The banner is printed by the demo **binary**, so it is the first line of the demo's own output in +both modes. Natively, `run.sh` has already said which mode it chose and built the sidecar above it; +that is build progress rather than the demo introducing itself as something else. + +**`records` and `keys` come before the speed columns**, and are the only two figures here that are +deterministic. Throughput alone cannot show the work happened: a short arm reads as a fast one +rather than a failed one, and a backlog that collapsed onto a single key would report the same rate +as one properly spread. Every language over the same backlog reports the same pair - this demo +seeds `key-{n % 1000}`, so `keys` is `min(records, 1000)` everywhere - which is what lets +`bin/ci-demo-conformance.sh` compare languages that can never be compared on elapsed or msg/s. + +**The bootstrap address is never printed**: own-cluster mode puts a user's real broker there. + +**No latency is reported, in any arm.** The backlog is pre-produced, so the workload is closed-loop +and per-record timings would be flattered by however far an arm had fallen behind. Throughput is the +only honest *speed* number this shape can produce - which is exactly why `records` and `keys` sit +beside it. + +The big replay runs **only the arms that go parallel**, which here is `pc-go-grpc` alone: AK core would +need `records x replay-factor x delay-ms` milliseconds to finish a backlog the sidecar arm clears in +seconds, and waiting that long to learn nothing new is not worth the wall clock. + +## Depth + +The client library itself is [`../README.md`](../README.md); the protocol is owned by +[`client-authoring-guide.md`](../../../parallel-consumer-proxy/docs/client-authoring-guide.md) and +[`protocol-specification.md`](../../../parallel-consumer-proxy/docs/protocol-specification.md). +Findings from this wave, including anything the contract left open, are in +[`docs/inflight/clients/go.md`](../../../docs/inflight/clients/go.md). diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/arms.go b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/arms.go new file mode 100644 index 0000000000..ad1990c42a --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/arms.go @@ -0,0 +1,245 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +package main + +import ( + "context" + "errors" + "fmt" + "os" + "sync" + "sync/atomic" + "time" + + "github.com/astubbs/parallel-consumer/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/parallelconsumer" + "github.com/twmb/franz-go/pkg/kgo" +) + +// The arm names, which are also the row labels in both tables. +// +// EACH NAMES THE CLIENT IT ACTUALLY RAN, because "AK core" is a CATEGORY and not a client: the +// answer is franz-go here, rdkafka in Ruby, kafkajs in TypeScript, and a reader cannot judge a +// comparison without knowing which one produced it. "AK core" is still spelled out in full - bare +// "core" reads as parallel-consumer-core (CONCEPTS.md) - and the sidecar row names what drives it, +// which on that path is this repository's own Go client library rather than any Kafka client. +const ( + armAKCore = "AK core (franz-go)" + armGoSidecar = "pc-go-grpc (this client)" +) + +// armBudget is how long an arm may take before the demo calls it stalled rather than slow. +const armBudget = 10 * time.Minute + +// armResult is what one arm achieved: over how many records, spread over how many keys, in how +// long. +// +// PROCESSED AND KEYS ARE THE DETERMINISTIC PAIR, and that is why they are reported rather than +// throughput alone. A rate cannot show the work happened - a short arm looks like a fast one - and +// elapsed and msg/s can never be compared between languages or between machines. These two can: +// every language seeding the same backlog reports the same records and the same distinct keys, so +// bin/ci-demo-conformance.sh has something to hold every demo to. +type armResult struct { + arm string + elapsed time.Duration + processed int + keys int +} + +// keySet counts the DISTINCT keys an arm observed, which is what shows the backlog was really +// spread rather than one key repeated. +// +// Guarded because the sidecar arm's function runs on many executors at once; the serial arm shares +// the type rather than keeping a bare map of its own, so both arms count the same thing the same +// way. A nil key and an empty one both land on "", which is the honest answer - this demo seeds +// neither, and a language that did would have to say which it meant. +type keySet struct { + mu sync.Mutex + seen map[string]struct{} +} + +func newKeySet() *keySet { return &keySet{seen: make(map[string]struct{})} } + +func (k *keySet) add(key []byte) { + k.mu.Lock() + defer k.mu.Unlock() + // string(key) on a []byte map index does not allocate - the compiler special-cases it - so + // counting keys costs nothing measurable next to the simulated work. + k.seen[string(key)] = struct{}{} +} + +func (k *keySet) size() int { + k.mu.Lock() + defer k.mu.Unlock() + return len(k.seen) +} + +// ratePerSecond is throughput, which is the only SPEED figure this demo reports - see broker.seed +// for why latency is not one of them. The records and keys beside it are not speed at all; they are +// the evidence that the work happened. +func (r armResult) ratePerSecond() float64 { + seconds := r.elapsed.Seconds() + if seconds <= 0 { + return 0 + } + return float64(r.processed) / seconds +} + +// akCore is the serial arm: Go's own Kafka client, one record at a time, the same sleep. +// +// A BLOCKING SLEEP IS THE RIGHT SIMULATED WORK IN GO and is what both arms use. The contract's +// non-occupying-wait rule singles out Python (worker processes) and TypeScript (one event loop); +// Go's goroutines make a sleeping worker as cheap as a sleeping thread, so time.Sleep is the +// idiomatic and honest choice here. +func (d demo) akCore(ctx context.Context, target int) (armResult, error) { + logf("\n=== %s starting over %d records ===", armAKCore, target) + client, err := kgo.NewClient( + kgo.SeedBrokers(d.broker.bootstrap), + kgo.ClientID("pc-go-demo-ak-core"), + kgo.ConsumerGroup(groupID("ak-core")), + kgo.ConsumeTopics(d.topic), + kgo.ConsumeResetOffset(kgo.NewOffset().AtStart()), + // Parallel Consumer owns commits on the other arm; this arm owns nothing but its own + // progress, and committing would only add broker round trips to the thing being timed. + kgo.DisableAutoCommit(), + ) + if err != nil { + return armResult{}, fmt.Errorf("%s: connecting: %w", armAKCore, err) + } + defer client.Close() + + // The clock starts AFTER the client is built and stops before it closes, because this arm is + // the denominator of every ratio in both tables and the other arm does not charge itself for + // client construction or teardown either. + started := time.Now() + deadline := started.Add(armBudget) + processed := 0 + keys := newKeySet() + for processed < target { + // The arm that waits on nothing still needs the budget armBudget promises, or a backlog + // shorter than the target spins here forever with no output. + if time.Now().After(deadline) { + return armResult{}, fmt.Errorf("%s stalled at %d of %d", armAKCore, processed, target) + } + pollCtx, cancel := context.WithTimeout(ctx, 500*time.Millisecond) + fetches := client.PollFetches(pollCtx) + cancel() + if errs := fetches.Errors(); len(errs) > 0 { + for _, e := range errs { + // A poll that timed out is this loop's own deadline expiring on an empty backlog, + // not a fault: the arm re-polls until the target or armBudget decides. + if errors.Is(e.Err, context.DeadlineExceeded) { + continue + } + if errors.Is(e.Err, context.Canceled) { + return armResult{}, fmt.Errorf("%s: cancelled at %d of %d", armAKCore, processed, target) + } + return armResult{}, fmt.Errorf("%s: fetching: %w", armAKCore, e.Err) + } + } + fetches.EachRecord(func(r *kgo.Record) { + keys.add(r.Key) + time.Sleep(time.Duration(d.options.delayMs) * time.Millisecond) + processed++ + }) + } + return finished(armAKCore, started, processed, keys.size()), nil +} + +// goSidecar is the arm the whole design exists for: this application as a FOREIGN CLIENT. +// +// THROUGH THE CLIENT LIBRARY, NEVER BY HAND. An earlier version of the Java seed spoke the protocol +// directly; it proved the engine worked and said nothing about the client library, which is the +// artifact users actually touch. So this arm calls parallelconsumer.Open/Poll/Close exactly as a +// user's program would, and the only Go code between the user's function and the engine is the +// library itself. +// +// ON THIS PATH THE APPLICATION DOES NO KAFKA I/O. The library spawns the sidecar, records arrive +// over a socket, this process's own function runs on them and reports outcomes back; the sidecar +// owns the consumer, the producer, the group membership and the offsets. In a genuinely foreign +// language that is the whole story - the application needs no Kafka client library at all. Here it +// is a statement about the PATH and not about the process: the same binary creates the topic, +// produces the backlog and runs the AK core arm with franz-go, because a comparison needs both +// sides. +func (d demo) goSidecar(ctx context.Context, target int) (armResult, error) { + logf("\n=== %s starting over %d records ===", armGoSidecar, target) + + client, err := parallelconsumer.Open(ctx, parallelconsumer.Options{ + SidecarPath: d.sidecar.path, + SidecarArgs: d.sidecar.args, + SidecarStderr: os.Stderr, + Topics: []string{d.topic}, + Ordering: parallelconsumer.OrderUnordered, + MaxConcurrency: int32(d.options.maxConcurrency), + KafkaProperties: d.broker.consumerProperties(groupID("pc-go-grpc")), + InstanceTag: "pc-go-demo", + }) + if err != nil { + return armResult{}, fmt.Errorf("%s: opening the session: %w", armGoSidecar, err) + } + defer func() { + // Close is what reaps the sidecar, so it runs even on the failure paths below. + if err := client.Close(); err != nil { + logf("%s: closing the session: %v", armGoSidecar, err) + } + }() + + var processed atomic.Int64 + keys := newKeySet() + // done is closed by whichever executor goroutine counts the target record. Every executor can + // reach it, and closing a channel twice panics, so the close is behind a sync.Once. + done := make(chan struct{}) + var reachedTarget sync.Once + + started := time.Now() + pollErr := client.Poll(ctx, func(_ context.Context, rec parallelconsumer.InboundRecord) (parallelconsumer.Outcome, error) { + // PLACE SERDE SETUP IN YOUR LANGUAGE HERE - keys and values are bytes, and this demo needs + // only the key, and only to count how many distinct ones the backlog really spread over. + // The sleep is the same simulated work the AK core arm runs, so the two arms differ by + // transport and engine and by nothing else. + // + // The key is recorded BEFORE the count is incremented, which is what makes the set safe to + // read the moment the count reaches the target: a record that has been counted has already + // contributed its key, so nothing can still be arriving when the wait below ends. + keys.add(rec.Key) + time.Sleep(time.Duration(d.options.delayMs) * time.Millisecond) + if processed.Add(1) >= int64(target) { + reachedTarget.Do(func() { close(done) }) + } + return parallelconsumer.Succeed(), nil + }) + if pollErr != nil { + return armResult{}, fmt.Errorf("%s: starting the poll: %w", armGoSidecar, pollErr) + } + + select { + case <-done: + case <-client.Done(): + // Reaching the target is not the only thing that ends the wait: a session that faulted or + // completed ends it too. Without this a broken run would print a plausible row at a + // plausible rate and exit 0, which is the worst thing a demo whose shape ten other + // languages copy can do. + count := int(processed.Load()) + if err := client.Err(); err != nil { + return armResult{}, fmt.Errorf("%s: the session ended at %d of %d: %w", + armGoSidecar, count, target, err) + } + return armResult{}, fmt.Errorf("%s: the session ended cleanly at %d of %d", armGoSidecar, count, target) + case <-time.After(armBudget): + return armResult{}, fmt.Errorf("%s stalled at %d of %d", armGoSidecar, processed.Load(), target) + case <-ctx.Done(): + return armResult{}, fmt.Errorf("%s: cancelled at %d of %d", armGoSidecar, processed.Load(), target) + } + return finished(armGoSidecar, started, int(processed.Load()), keys.size()), nil +} + +func finished(arm string, started time.Time, processed, keys int) armResult { + elapsed := time.Since(started) + logf("=== %s finished: %d records over %d keys in %dms ===", arm, processed, keys, elapsed.Milliseconds()) + return armResult{arm: arm, elapsed: elapsed, processed: processed, keys: keys} +} + +// groupID names a fresh consumer group per arm per replay, so every arm reads the same records from +// the beginning. +func groupID(arm string) string { + return fmt.Sprintf("pc-demo-%s-%d", arm, time.Now().UnixNano()) +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/broker.go b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/broker.go new file mode 100644 index 0000000000..f6ab2d565e --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/broker.go @@ -0,0 +1,136 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +package main + +import ( + "context" + "errors" + "fmt" + "sync" + "time" + + "github.com/twmb/franz-go/pkg/kadm" + "github.com/twmb/franz-go/pkg/kerr" + "github.com/twmb/franz-go/pkg/kgo" +) + +// keySpace is the key space the seeded records spread over. Ordering is unordered in both arms, so +// it changes nothing today; it exists so that a key-ordered lane added later has more than one key +// to shard across rather than needing the seeding rewritten first. +const keySpace = 1000 + +// broker is the cluster both arms read from. +// +// UNLIKE THE JAVA REFERENCE, IT NEVER STARTS ONE. Java's DemoBroker falls back to Testcontainers +// when no address was supplied; here run.sh starts the broker - as a container natively, as a +// compose sibling in the container - and this type is always handed an address. demo/README.md +// records that divergence and why it was taken. The rule that produced it is the same one either +// way: a demo container is never granted the host Docker socket, so a containerised demo reaches a +// broker it did not start. +// +// The address is never logged or echoed: the same door serves own-cluster mode, where it is the +// user's real cluster. +type broker struct { + bootstrap string +} + +// ensureTopic creates the demo's topic, tolerating one a previous run left behind. +// +// A topic that already exists with a DIFFERENT partition count is refused rather than reused: the +// effective-configuration block would otherwise print a --partitions value that never applied, and +// that block is the demo's whole reproducibility promise. +func (b broker) ensureTopic(ctx context.Context, topic string, partitions int) error { + client, err := kgo.NewClient(kgo.SeedBrokers(b.bootstrap), kgo.ClientID("pc-go-demo-admin")) + if err != nil { + return fmt.Errorf("connecting to the broker: %w", err) + } + defer client.Close() + + admin := kadm.NewClient(client) + created, err := admin.CreateTopics(ctx, int32(partitions), 1, nil, topic) + if err != nil { + return fmt.Errorf("creating the demo topic %s: %w", topic, err) + } + switch createErr := created[topic].Err; { + case createErr == nil: + logf("Created topic %s with %d partitions", topic, partitions) + return nil + case errors.Is(createErr, kerr.TopicAlreadyExists): + details, err := admin.ListTopics(ctx, topic) + if err != nil { + return fmt.Errorf("describing the existing topic %s: %w", topic, err) + } + existing := len(details[topic].Partitions) + if existing != partitions { + return fmt.Errorf("topic %s already exists with %d partitions, but this run asked for "+ + "%d - pass --topic to name a fresh one, or --partitions %d", + topic, existing, partitions, existing) + } + logf("Topic %s already exists with the requested %d partitions, reusing it", topic, partitions) + return nil + default: + return fmt.Errorf("creating the demo topic %s: %w", topic, createErr) + } +} + +// seed produces the backlog both arms then replay. +// +// PRE-PRODUCED RATHER THAN PRODUCED ALONGSIDE THE ARMS, which is what makes the workload +// closed-loop - and in turn why no arm reports latency. A per-record timing here would be flattered +// by however far an arm had fallen behind, so throughput is the only honest number this shape can +// produce. +func (b broker) seed(ctx context.Context, topic string, from, to int) error { + if to <= from { + return nil + } + client, err := kgo.NewClient( + kgo.SeedBrokers(b.bootstrap), + kgo.ClientID("pc-go-demo-seed"), + kgo.DefaultProduceTopic(topic), + kgo.ProducerLinger(20*time.Millisecond), + ) + if err != nil { + return fmt.Errorf("connecting to the broker to seed: %w", err) + } + defer client.Close() + + logf("Producing records %d to %d...", from, to) + // A discarded promise swallows the reason, and Flush does not report a send that failed - so + // without this the demo would report a full backlog, run both arms against a short one, and + // print numbers for a workload that never existed. + var once sync.Once + var firstFailure error + for i := from; i < to; i++ { + client.Produce(ctx, &kgo.Record{ + Key: []byte(fmt.Sprintf("key-%d", i%keySpace)), + Value: []byte(fmt.Sprintf("record-%d", i)), + }, func(_ *kgo.Record, err error) { + if err != nil { + once.Do(func() { firstFailure = err }) + } + }) + } + if err := client.Flush(ctx); err != nil { + return fmt.Errorf("the demo could not seed its backlog: %w", err) + } + if firstFailure != nil { + return fmt.Errorf("the demo could not seed its backlog: %w", firstFailure) + } + logf("Produced %d records", to-from) + return nil +} + +// consumerProperties is what an arm's consumer needs to reach this broker, as the string map the +// protocol's Configure carries. +// +// enable.auto.commit is false because Parallel Consumer owns offset commits and refuses a consumer +// with auto-commit on. The sidecar forces it itself, so this is belt and braces on that path; it is +// set here so the two arms are configured from one place and read as the same cluster. +func (b broker) consumerProperties(groupID string) map[string]string { + return map[string]string{ + "bootstrap.servers": b.bootstrap, + "group.id": groupID, + "auto.offset.reset": "earliest", + "enable.auto.commit": "false", + } +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/docker-compose.yml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/docker-compose.yml new file mode 100644 index 0000000000..a74f3657dc --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/docker-compose.yml @@ -0,0 +1,109 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE ONE-COMMAND PATH for the Go demo (astubbs#242, plan unit U35, R20). It mirrors the Java +# seed's compose file beside the Java client, and the two rules below are the reason both exist in +# this shape rather than a simpler one. +# +# THE BROKER IS A SIBLING, AND THAT IS A RULE RATHER THAN A LAYOUT CHOICE. Starting a broker from +# inside the demo container would require mounting the host Docker socket, and U35 forbids it: a +# documented socket mount is root-equivalent host access taught as the normal way to run the +# product. So the containerised demo is handed an address and starts nothing. Natively, run.sh +# starts the broker on the host instead - the demo binary itself never starts one in either mode, +# which is where Go diverges from Java's Testcontainers fallback (see README.md). +# +# THE SIDECAR IS NOT A SERVICE HERE, deliberately. It would be easy to add one and it would teach +# the wrong thing: KTD41's product model is that the client library spawns and supervises the +# sidecar itself, so a user never installs, deploys or operates a process. A compose service would +# show a deployment the product does not ask for. The Go client library spawns it as a child +# process inside this container, exactly as it does natively. +# +# TWO WAYS IN, AND BOTH ARE SUPPORTED: +# +# cd && docker compose up # the plain one, defaults, nothing else needed +# demo/run.sh --docker [--records N ...] # the same thing with the demo's flags +# +# `docker compose up` on its own is the path to keep working: it is what a reader who has never seen +# this repository will type. It needs no arguments, no environment and no wrapper script - the demo +# service runs with the demo's own defaults, and PC_DEMO_ARGS below is empty unless run.sh sets it. +# Add --build after changing any source, and `docker compose down` to remove the broker afterwards. +# +# The demo container exits when the run finishes; the broker keeps running until you stop it. run.sh +# passes --abort-on-container-exit --exit-code-from demo so that a scripted run stops on its own and +# reports the demo's exit code rather than compose's. + +name: pc-go-demo + +services: + broker: + # Pinned as a literal because a compose file cannot derive anything. run.sh READS THIS LINE for + # the broker it starts natively, so the pin lives here once rather than in both files; keep it + # as `image: ` on one line or that read fails loudly. It must track in the + # root pom - the mapping is CP major = AK major + 4, so AK 3.9.x means CP 7.9. + image: confluentinc/cp-kafka:7.9.0 + environment: + CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk + # QUIET. A demo is a thing someone WATCHES, and a KRaft broker at INFO emits hundreds of + # lines of its own configuration, controller elections and log-segment chatter - which buries + # the two tables the demo exists to show. Kafka's own warnings and errors still come through. + KAFKA_LOG4J_ROOT_LOGLEVEL: WARN + KAFKA_TOOLS_LOG4J_LOGLEVEL: WARN + # ROOT ALONE IS NOT ENOUGH, and this was measured rather than assumed: the cp-kafka image + # writes kafka=INFO, kafka.controller=TRACE and state.change.logger=TRACE into its own + # log4j config, and a named logger beats the root. With only the root set, one demo run + # interleaved 927 broker lines with 35 of the demo's own. These are the named loggers that + # actually produce the noise. + KAFKA_LOG4J_LOGGERS: "kafka=WARN,kafka.controller=WARN,kafka.server=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.network=WARN,kafka.request.logger=WARN,state.change.logger=WARN,org.apache.kafka=WARN" + KAFKA_NODE_ID: 1 + KAFKA_PROCESS_ROLES: broker,controller + # Bound to the service name, not 0.0.0.0: this image formats its KRaft storage with the same + # config it boots with, and the format step refuses a non-routable advertised address. + KAFKA_LISTENERS: PLAINTEXT://broker:9092,CONTROLLER://broker:9093 + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092 + KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT + KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093 + KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + # The demo forms one consumer group per arm per replay, and the default three-second settling + # delay would be charged to every one of them. run.sh sets the same value natively. + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 500 + healthcheck: + # The demo must not start before the broker can serve an API request. Listing topics is the + # cheapest call that proves that, and it proves more than an open port does. + test: ["CMD", "kafka-topics", "--bootstrap-server", "broker:9092", "--list"] + interval: 5s + timeout: 10s + retries: 30 + start_period: 10s + + demo: + build: + # The repository, because the sidecar is a JVM application built by a Maven reactor module + # that cannot build without its parents and siblings. The root .dockerignore is what keeps + # this from uploading every target/ directory in the tree; the Dockerfile explains the pair. + context: ../../.. + dockerfile: parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/Dockerfile + # A demo that has printed its tables is done. Without this, compose's default would restart a + # container that exited cleanly and the reader would watch the whole run happen again. + restart: "no" + depends_on: + broker: + condition: service_healthy + environment: + # Supplying this is what stops the demo asking for a broker it cannot start. + PC_DEMO_BOOTSTRAP: broker:9092 + # The flags demo/run.sh was called with, forwarded as one string. + PC_DEMO_ARGS: ${PC_DEMO_ARGS:-} + # EVERY OTHER DIAL, forwarded explicitly. Compose passes nothing it is not told to, so a + # missing line here is a dial the container silently ignores while the run still prints a + # fingerprint claiming otherwise. Caught across seven languages at once by + # bin/ci-demo-conformance.sh, which is why that check exists. Adding a flag means adding it + # here too. + PC_DEMO_RECORDS: ${PC_DEMO_RECORDS:-} + PC_DEMO_DELAY_MS: ${PC_DEMO_DELAY_MS:-} + PC_DEMO_CONCURRENCY: ${PC_DEMO_CONCURRENCY:-} + PC_DEMO_PARTITIONS: ${PC_DEMO_PARTITIONS:-} + PC_DEMO_REPLAY_FACTOR: ${PC_DEMO_REPLAY_FACTOR:-} + PC_DEMO_TOPIC: ${PC_DEMO_TOPIC:-} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/go.mod b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/go.mod new file mode 100644 index 0000000000..aa96acf2c6 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/go.mod @@ -0,0 +1,38 @@ +// Copyright (C) 2026 Antony Stubbs and contributors +// +// A MODULE OF ITS OWN, AND THAT IS THE POINT. The demo needs a Kafka client for its AK core arm; +// the client LIBRARY must not. Go's module graph propagates requirements to every consumer, so +// putting franz-go in the library's go.mod would hand a Kafka client library to applications whose +// whole reason for using the proxy is not needing one. A nested module keeps that dependency here. +// +// The side effect is worth knowing: `go build ./...` and `go test ./...` in the parent module do +// NOT descend into a directory that has its own go.mod, so the parent's Maven-driven build never +// compiles this. demo/run.sh and the Dockerfile build it, and both entry points are exercised. +// +// The replace directive is what makes the demo drive THIS checkout's client library rather than a +// published version - there is none, and the demo must exercise the code beside it. + +module github.com/astubbs/parallel-consumer/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo + +go 1.25.0 + +require ( + github.com/astubbs/parallel-consumer/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go v0.0.0 + github.com/twmb/franz-go v1.21.5 + github.com/twmb/franz-go/pkg/kadm v1.18.0 +) + +require ( + github.com/klauspost/compress v1.18.6 // indirect + github.com/pierrec/lz4/v4 v4.1.26 // indirect + github.com/twmb/franz-go/pkg/kmsg v1.13.1 // indirect + golang.org/x/crypto v0.55.0 // indirect + golang.org/x/net v0.58.0 // indirect + golang.org/x/sys v0.47.0 // indirect + golang.org/x/text v0.41.0 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260414002931-afd174a4e478 // indirect + google.golang.org/grpc v1.82.1 // indirect + google.golang.org/protobuf v1.36.11 // indirect +) + +replace github.com/astubbs/parallel-consumer/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go => ../ diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/go.sum b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/go.sum new file mode 100644 index 0000000000..160ef4b728 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/go.sum @@ -0,0 +1,50 @@ +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao= +github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= +github.com/pierrec/lz4/v4 v4.1.26 h1:GrpZw1gZttORinvzBdXPUXATeqlJjqUG/D87TKMnhjY= +github.com/pierrec/lz4/v4 v4.1.26/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4= +github.com/twmb/franz-go v1.21.5 h1:cVYI2+JTTKSvohhy8bCOleYrS7G79ZBrLVFIJsoHm8M= +github.com/twmb/franz-go v1.21.5/go.mod h1:rfoMTnVk7107fhTGxfEKIHP/e7tPe6oyij/ywzO0czk= +github.com/twmb/franz-go/pkg/kadm v1.18.0 h1:WRf/LZmDdcDXwX7WMbtDU++v+b3NzYh2bCGoPMmzirw= +github.com/twmb/franz-go/pkg/kadm v1.18.0/go.mod h1:XeLhGoLXLFzK8/ryv5FfpxPxGwj4oFEGpPJMB/x6KDE= +github.com/twmb/franz-go/pkg/kmsg v1.13.1 h1:fG5kItwysTk5UXqVwb64EpQEy3TydF3vYYK21nUQ+bI= +github.com/twmb/franz-go/pkg/kmsg v1.13.1/go.mod h1:+DPt4NC8RmI6hqb8G09+3giKObE6uD2Eya6CfqBpeJY= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/otel v1.43.0 h1:mYIM03dnh5zfN7HautFE4ieIig9amkNANT+xcVxAj9I= +go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0= +go.opentelemetry.io/otel/metric v1.43.0 h1:d7638QeInOnuwOONPp4JAOGfbCEpYb+K6DVWvdxGzgM= +go.opentelemetry.io/otel/metric v1.43.0/go.mod h1:RDnPtIxvqlgO8GRW18W6Z/4P462ldprJtfxHxyKd2PY= +go.opentelemetry.io/otel/sdk v1.43.0 h1:pi5mE86i5rTeLXqoF/hhiBtUNcrAGHLKQdhg4h4V9Dg= +go.opentelemetry.io/otel/sdk v1.43.0/go.mod h1:P+IkVU3iWukmiit/Yf9AWvpyRDlUeBaRg6Y+C58QHzg= +go.opentelemetry.io/otel/sdk/metric v1.43.0 h1:S88dyqXjJkuBNLeMcVPRFXpRw2fuwdvfCGLEo89fDkw= +go.opentelemetry.io/otel/sdk/metric v1.43.0/go.mod h1:C/RJtwSEJ5hzTiUz5pXF1kILHStzb9zFlIEe85bhj6A= +go.opentelemetry.io/otel/trace v1.43.0 h1:BkNrHpup+4k4w+ZZ86CZoHHEkohws8AY+WTX09nk+3A= +go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0= +golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M= +golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis= +golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= +golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= +golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= +golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= +golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= +gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= +gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260414002931-afd174a4e478 h1:RmoJA1ujG+/lRGNfUnOMfhCy5EipVMyvUE+KNbPbTlw= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260414002931-afd174a4e478/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/grpc v1.82.1 h1:NnAxzGRA0677vCa4BUkOAnO5+FfQqVl9iUXeD0IqcGE= +google.golang.org/grpc v1.82.1/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA= +google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= +google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/main.go b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/main.go new file mode 100644 index 0000000000..94b1208744 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/main.go @@ -0,0 +1,163 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +// Command pc-go-demo is the Go demo of the Parallel Consumer language proxy: the same records +// through Go's own Kafka client and through Go over the sidecar (astubbs#242, plan unit U35). +// +// THE CONTRACT IT KEEPS IS parallel-consumer-proxy/demo/README.md, and the reference +// implementation is the Java seed beside the Java client. Same flags, same environment variables, +// same precedence, same defaults, same two tables in the same order, the effective configuration +// printed first and the bootstrap address never printed at all. What is specific to Go - and the +// short list of places it diverges - is in demo/README.md beside this file. +// +// TWO ARMS, WHICH IS THE WHOLE CONTRACT OUTSIDE JAVA, AND EACH NAMES THE CLIENT IT ACTUALLY RAN: +// +// - AK core (franz-go) - Go's own Kafka client, one record at a time. "AK core" is a +// CATEGORY, not a client, so the library is named beside it - a reader cannot judge a +// comparison without knowing what produced it. Always spelled "AK core", never bare "core", +// which reads as parallel-consumer-core (CONCEPTS.md). +// - pc-go-grpc (this client) - this application as a FOREIGN CLIENT, through the Go client library, +// over a real sidecar the library spawns as a child process. The application does no Kafka I/O +// on that path: the sidecar owns the consumer, the producer, the group membership and the +// offsets. +// +// Java carries four more arms because one JVM can hold every engine at once and price the hop +// exactly. Go cannot - its two arms are different client libraries as well as different engines - +// so two arms is the whole comparison here, by design rather than by omission. +package main + +import ( + "context" + "fmt" + "os" + "os/signal" + "syscall" + "time" +) + +// banner is the demo's opening lines, fixed by the shared contract and identical in all eleven +// languages except for the language's own name. Copied verbatim rather than composed, including +// the sixty-four rule characters and the two spaces either side of the dash, because eleven demos +// drifting apart on their own banner would be a poor advertisement for a contract about output. +const banner = ` +================================================================ + PARALLEL CONSUMER - Go demo + The same records, twice: one at a time, then all at once. +================================================================` + +func main() { + os.Exit(run(os.Args[1:])) +} + +// demo is one run: the settings, the cluster, the topic and the sidecar command, resolved once. +type demo struct { + options options + broker broker + topic string + sidecar sidecarCommand +} + +func run(args []string) int { + if helpRequested(args) { + logf("%s", usageText) + return 0 + } + + // THE BANNER IS THE FIRST THING PRINTED, before the settings and before any arm says anything + // about itself. A reader who runs this and meets `pc-go-grpc: the proxy granted 100 executor + // threads` has been told nothing about what they are looking at; the words "Parallel Consumer" + // have to appear before anything else does. Identical in every language bar the language's own + // name - see the contract's "It opens by saying what it is". + logf("%s", banner) + + opts, err := parseOptions(args, environment) + if err != nil { + // A misspelled flag must never be reported as a result for settings nobody asked for. + errorf("%v", err) + errorf("%s", usageText) + return 2 + } + + if opts.bootstrap == "" { + errorf("No broker address was supplied. Unlike the Java seed this demo never starts one " + + "itself - demo/run.sh does, and then passes it here. Run demo/run.sh, or pass " + + "--bootstrap ADDR / PC_DEMO_BOOTSTRAP for a cluster you already have.") + return 2 + } + + sidecar, err := resolveSidecar() + if err != nil { + errorf("%v", err) + return 2 + } + + topic := opts.topic + if topic == "" { + topic = fmt.Sprintf("pc-demo-%d", time.Now().UnixNano()) + } + + // Ctrl-C ends the run rather than orphaning a sidecar: the client library reaps its child on + // Close, and cancelling the context is what gets us there. + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + d := demo{options: opts, broker: broker{bootstrap: opts.bootstrap}, topic: topic, sidecar: sidecar} + if err := d.run(ctx); err != nil { + errorf("%v", err) + return 1 + } + return 0 +} + +func (d demo) run(ctx context.Context) error { + // THE FINGERPRINT COMES FIRST, and it never carries the bootstrap address: own-cluster mode + // puts a user's real broker there. A number without its settings is not reproducible. + logf("\nEffective configuration:\n %s\n topic = %s", d.options, d.topic) + + if err := d.broker.ensureTopic(ctx, d.topic, d.options.partitions); err != nil { + return err + } + if err := d.broker.seed(ctx, d.topic, 0, d.options.records); err != nil { + return err + } + + akCore, err := d.akCore(ctx, d.options.records) + if err != nil { + return err + } + sidecarArm, err := d.goSidecar(ctx, d.options.records) + if err != nil { + return err + } + small := []armResult{akCore, sidecarArm} + report(fmt.Sprintf("Small replay - every arm over the same %d records (the comparison)", + d.options.records), small, &akCore, false) + + if !d.options.bigReplayWanted() { + logf("\nBig replay skipped (--replay-factor %d).", d.options.replayFactor) + return nil + } + + total := d.options.bigReplayRecords() + if err := d.broker.seed(ctx, d.topic, d.options.records, total); err != nil { + return err + } + + // AK CORE IS EXCLUDED HERE BECAUSE IT DOES NOT GO PARALLEL. It would need total * delayMs + // milliseconds to finish a backlog the sidecar arm clears in seconds, and a demo that makes a + // reader wait that long to learn nothing new is not worth the wall clock. That leaves one row, + // which is what the contract's "every arm that goes parallel" means in a language whose only + // other arm is serial. + big, err := d.goSidecar(ctx, total) + if err != nil { + return err + } + // the unit is chosen so the figure is never zero - see the demo contract. + serialMillis := total * d.options.delayMs + serialCost := fmt.Sprintf("%dms", serialMillis) + if serialMillis >= 1000 { + serialCost = fmt.Sprintf("%ds", serialMillis/1000) + } + report(fmt.Sprintf("Big replay - %d records, parallel arms only (AK core is serial and would take %s+)", + total, serialCost), []armResult{big}, &akCore, true) + return nil +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/options.go b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/options.go new file mode 100644 index 0000000000..009eb9d193 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/options.go @@ -0,0 +1,238 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +package main + +import ( + "fmt" + "os" + "strconv" + "strings" +) + +// envPrefix is the prefix of every environment variable this demo reads, so a reader can grep one +// string. It is part of the shared contract, not a Go choice. +const envPrefix = "PC_DEMO_" + +// options are the demo's dials, and the surface every language's demo mirrors: +// parallel-consumer-proxy/demo/README.md is the contract, and the Java reference implementation of +// this exact struct is DemoOptions in parallel-consumer-proxy-client-java-demo. +// +// PRECEDENCE IS FLAGS BEAT ENVIRONMENT BEATS DEFAULTS, and it is stated rather than implied +// because eleven demos have to agree on it: a container passes configuration by environment while +// a person at a terminal passes flags, and both must be able to override the other's layer. +// +// R39 constrains how configuration reaches the PROXY. A demo is an application, so these flags are +// not a breach of it - the note lives here because without it someone reads --records as breaking +// the plan's own rule and deletes it. +type options struct { + records int + delayMs int + maxConcurrency int + partitions int + replayFactor int + + // bootstrap is empty when the caller supplied no broker. See run.sh: on this client the broker + // is started by the entry point rather than by the demo binary, which is the one place Go + // diverges from the Java reference - demo/README.md says why. + bootstrap string + + // topic is empty when the caller supplied none, in which case the demo names its own. + topic string +} + +func defaults() options { + return options{ + records: 2000, + delayMs: 2, + maxConcurrency: 100, + partitions: 10, + replayFactor: 20, + } +} + +// helpRequested reports whether the caller asked for the usage text rather than a run. +// +// Handled in the binary and not only in run.sh, because the script is not the only way in: +// `docker compose run demo --help` reaches this binary directly, and answering that with +// "unknown option: --help" would be a poor first impression of a demo ten other languages copy. +func helpRequested(args []string) bool { + for _, a := range args { + if a == "-h" || a == "--help" { + return true + } + } + return false +} + +// parseOptions applies the environment over the defaults and then the flags over both. +// +// It REFUSES an unknown flag, a missing value and a value that is not a number in range. A demo +// that silently ignored a misspelled flag would report numbers for settings the user did not ask +// for, which is worse than not running. +func parseOptions(args []string, env func(string) string) (options, error) { + o := defaults() + if err := o.applyEnvironment(env); err != nil { + return o, err + } + + for i := 0; i < len(args); i++ { + flag := args[i] + if !takesAValue(flag) { + return o, fmt.Errorf("unknown option: %s", flag) + } + // Every flag this demo has takes a value, so the unknown flag is rejected FIRST and a + // missing value is always the remaining error. Doing it the other way round consumes the + // next argument on behalf of a flag that does not exist, and the message then names the + // wrong thing. + if i+1 >= len(args) { + return o, fmt.Errorf("%s needs a value", flag) + } + i++ + raw := args[i] + + var err error + switch flag { + case "--records": + o.records, err = positive(flag, raw) + case "--delay-ms": + o.delayMs, err = nonNegative(flag, raw) + case "--concurrency": + o.maxConcurrency, err = positive(flag, raw) + case "--partitions": + o.partitions, err = positive(flag, raw) + case "--replay-factor": + // 1 or less skips the big replay, so this one is allowed to be zero + o.replayFactor, err = nonNegative(flag, raw) + case "--bootstrap": + o.bootstrap = raw + case "--topic": + o.topic = raw + } + if err != nil { + return o, err + } + } + return o, o.validate() +} + +// flags is the demo's whole command line, in the order the contract lists it. One list, so the +// parser and the rejection of an unknown flag cannot disagree about what exists. +var flags = []string{ + "--records", "--delay-ms", "--concurrency", "--partitions", "--replay-factor", + "--bootstrap", "--topic", +} + +func takesAValue(flag string) bool { + for _, known := range flags { + if known == flag { + return true + } + } + return false +} + +func (o *options) applyEnvironment(env func(string) string) error { + type binding struct { + suffix string + apply func(string, string) error + } + bindings := []binding{ + {"RECORDS", func(name, raw string) (err error) { o.records, err = positive(name, raw); return }}, + {"DELAY_MS", func(name, raw string) (err error) { o.delayMs, err = nonNegative(name, raw); return }}, + {"CONCURRENCY", func(name, raw string) (err error) { o.maxConcurrency, err = positive(name, raw); return }}, + {"PARTITIONS", func(name, raw string) (err error) { o.partitions, err = positive(name, raw); return }}, + {"REPLAY_FACTOR", func(name, raw string) (err error) { o.replayFactor, err = nonNegative(name, raw); return }}, + {"BOOTSTRAP", func(_, raw string) error { o.bootstrap = raw; return nil }}, + {"TOPIC", func(_, raw string) error { o.topic = raw; return nil }}, + } + for _, b := range bindings { + name := envPrefix + b.suffix + raw := strings.TrimSpace(env(name)) + if raw == "" { + // An EMPTY variable is not a setting. Compose substitutes an unset variable as the + // empty string - PC_DEMO_ARGS: ${PC_DEMO_ARGS:-} is in the compose file beside this - + // so treating "" as a value would make an unset variable override a default with zero. + continue + } + if err := b.apply(name, raw); err != nil { + return err + } + } + return nil +} + +func (o options) validate() error { + // Checked as a 64-bit product rather than trusted as an int later: on a 32-bit build + // records * replayFactor overflows silently, and a wrapped value turns the big replay into a + // tiny one that still prints a confident throughput figure. + big := int64(o.records) * int64(max(1, o.replayFactor)) + if big > int64(^uint32(0)>>1) { + return fmt.Errorf("--records times --replay-factor is %d, which is more records than the "+ + "demo can count; lower one of them", big) + } + return nil +} + +// bigReplayRecords is the records the big replay consumes in total, including the small replay's. +func (o options) bigReplayRecords() int { return o.records * max(1, o.replayFactor) } + +// bigReplayWanted reports whether the big replay is worth running; a factor of 1 or less skips it. +func (o options) bigReplayWanted() bool { return o.replayFactor > 1 } + +// String is the effective-configuration fingerprint, printed before the run. +// +// A number without its settings is not reproducible, so this is part of the contract every +// language's demo keeps rather than a debugging aid. THE BOOTSTRAP ADDRESS IS DELIBERATELY ABSENT: +// own-cluster mode puts a user's real broker there, and the credential-hygiene rule that binds the +// proxy binds a demo too - nothing logged, nothing echoed. +func (o options) String() string { + return fmt.Sprintf("records = %d\n delayMs = %d\n maxConcurrency = %d\n partitions = %d\n replayFactor = %d", + o.records, o.delayMs, o.maxConcurrency, o.partitions, o.replayFactor) +} + +func positive(name, raw string) (int, error) { + n, err := number(name, raw) + if err != nil { + return 0, err + } + if n < 1 { + return 0, fmt.Errorf("%s must be at least 1, got %d", name, n) + } + return n, nil +} + +func nonNegative(name, raw string) (int, error) { + n, err := number(name, raw) + if err != nil { + return 0, err + } + if n < 0 { + return 0, fmt.Errorf("%s must not be negative, got %d", name, n) + } + return n, nil +} + +func number(name, raw string) (int, error) { + n, err := strconv.Atoi(strings.TrimSpace(raw)) + if err != nil { + return 0, fmt.Errorf("%s needs a whole number, got %q", name, raw) + } + return n, nil +} + +// environment is os.Getenv as the func the parser takes, so the parser is testable without +// mutating this process's environment. +func environment(name string) string { return os.Getenv(name) } + +const usageText = `usage: demo/run.sh [options] + + --records N records in the comparison replay (default 2000) + --delay-ms N simulated work per record, ms (default 2) + --concurrency N max in-flight records (default 100) + --partitions N partitions on the demo topic (default 10) + --replay-factor N big replay = records x N; 1 skips (default 20) + --bootstrap ADDR an existing broker; omit and run.sh starts one + --topic NAME an existing topic; omit to create one + +Every flag has an environment variable: --delay-ms is PC_DEMO_DELAY_MS. +Flags beat the environment beats the defaults.` diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/options_test.go b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/options_test.go new file mode 100644 index 0000000000..35dca3461d --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/options_test.go @@ -0,0 +1,207 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +package main + +import ( + "strings" + "testing" +) + +// noEnvironment is an empty environment, passed rather than read from the process so these tests +// never depend on what the shell running them happens to export. +func noEnvironment(string) string { return "" } + +func environmentOf(pairs map[string]string) func(string) string { + return func(name string) string { return pairs[name] } +} + +// TestNoArgumentsIsTheDocumentedDefault guards THE CASE THAT HAS ACTUALLY BROKEN, twice in this +// family: the demo started with nothing at all. The defaults are also the numbers the contract +// publishes, so this doubles as the check that they have not drifted from +// parallel-consumer-proxy/demo/README.md. +func TestNoArgumentsIsTheDocumentedDefault(t *testing.T) { + got, err := parseOptions(nil, noEnvironment) + if err != nil { + t.Fatalf("no arguments must parse, got %v", err) + } + want := options{records: 2000, delayMs: 2, maxConcurrency: 100, partitions: 10, replayFactor: 20} + if got != want { + t.Errorf("no arguments gave %+v, want %+v", got, want) + } +} + +// TestFlagsBeatEnvironmentBeatsDefaults is the precedence rule stated in the contract, and it is +// the one thing every language's demo has to agree on that no single language can check alone. +func TestFlagsBeatEnvironmentBeatsDefaults(t *testing.T) { + env := environmentOf(map[string]string{ + "PC_DEMO_RECORDS": "50", + "PC_DEMO_DELAY_MS": "7", + "PC_DEMO_TOPIC": "from-the-environment", + }) + got, err := parseOptions([]string{"--records", "60"}, env) + if err != nil { + t.Fatalf("parse: %v", err) + } + if got.records != 60 { + t.Errorf("a flag must beat the environment: records = %d, want 60", got.records) + } + if got.delayMs != 7 { + t.Errorf("the environment must beat the default: delayMs = %d, want 7", got.delayMs) + } + if got.topic != "from-the-environment" { + t.Errorf("topic = %q, want from-the-environment", got.topic) + } + if got.partitions != 10 { + t.Errorf("an unmentioned setting keeps its default: partitions = %d, want 10", got.partitions) + } +} + +// TestEveryFlagHasAnEnvironmentVariable checks the mapping the contract states as a rule - +// PC_DEMO_ plus the flag in upper snake case - rather than trusting seven hand-written bindings to +// have been written the same way. +func TestEveryFlagHasAnEnvironmentVariable(t *testing.T) { + values := map[string]string{ + "--records": "11", "--delay-ms": "12", "--concurrency": "13", "--partitions": "14", + "--replay-factor": "15", "--bootstrap": "somewhere:9092", "--topic": "a-topic", + } + for _, flag := range flags { + name := envPrefix + strings.ToUpper(strings.ReplaceAll(strings.TrimPrefix(flag, "--"), "-", "_")) + viaEnv, err := parseOptions(nil, environmentOf(map[string]string{name: values[flag]})) + if err != nil { + t.Fatalf("%s: %v", name, err) + } + viaFlag, err := parseOptions([]string{flag, values[flag]}, noEnvironment) + if err != nil { + t.Fatalf("%s: %v", flag, err) + } + if viaEnv != viaFlag { + t.Errorf("%s and %s must set the same thing: %+v vs %+v", flag, name, viaFlag, viaEnv) + } + } +} + +// TestAnEmptyEnvironmentVariableIsNotASetting matters because compose substitutes an unset variable +// as the empty string. Treating "" as a value would let an unset PC_DEMO_RECORDS override the +// default with zero, and the demo would report a confident throughput for no records at all. +func TestAnEmptyEnvironmentVariableIsNotASetting(t *testing.T) { + got, err := parseOptions(nil, environmentOf(map[string]string{"PC_DEMO_RECORDS": " "})) + if err != nil { + t.Fatalf("parse: %v", err) + } + if got.records != 2000 { + t.Errorf("records = %d, want the default 2000", got.records) + } +} + +func TestBadInputIsRefusedRatherThanIgnored(t *testing.T) { + cases := []struct { + name string + args []string + env map[string]string + want string + }{ + {"an unknown flag", []string{"--recrods", "5"}, nil, "unknown option: --recrods"}, + {"a missing value", []string{"--records"}, nil, "--records needs a value"}, + {"a value that is not a number", []string{"--records", "lots"}, nil, "needs a whole number"}, + {"a records count below one", []string{"--records", "0"}, nil, "must be at least 1"}, + {"a negative delay", []string{"--delay-ms", "-1"}, nil, "must not be negative"}, + {"a bad environment value", nil, map[string]string{"PC_DEMO_CONCURRENCY": "0"}, "PC_DEMO_CONCURRENCY must be at least 1"}, + {"a big replay the demo cannot count", []string{"--records", "2000000", "--replay-factor", "2000"}, nil, "more records than the demo can count"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + env := noEnvironment + if c.env != nil { + env = environmentOf(c.env) + } + _, err := parseOptions(c.args, env) + if err == nil { + t.Fatalf("expected a refusal mentioning %q, got none", c.want) + } + if !strings.Contains(err.Error(), c.want) { + t.Errorf("error was %q, want it to mention %q", err, c.want) + } + }) + } +} + +// TestAnUnknownFlagDoesNotSwallowTheNextArgument is the reason the parser rejects the flag before +// it reads a value: doing it the other way round consumes the following argument on behalf of a +// flag that does not exist, and the message then names the wrong thing. +func TestAnUnknownFlagDoesNotSwallowTheNextArgument(t *testing.T) { + _, err := parseOptions([]string{"--nonsense"}, noEnvironment) + if err == nil || !strings.Contains(err.Error(), "unknown option: --nonsense") { + t.Errorf("error was %v, want it to name the unknown flag rather than a missing value", err) + } +} + +// TestTheFingerprintNeverCarriesTheBootstrapAddress is a CONTRACT RULE, not a style preference: +// own-cluster mode puts a user's real broker address in that field, and the demo prints the +// fingerprint on every run. +func TestTheFingerprintNeverCarriesTheBootstrapAddress(t *testing.T) { + o, err := parseOptions([]string{"--bootstrap", "secret-broker.internal:9093", "--topic", "orders"}, noEnvironment) + if err != nil { + t.Fatalf("parse: %v", err) + } + if strings.Contains(o.String(), "secret-broker") { + t.Errorf("the fingerprint printed the bootstrap address:\n%s", o.String()) + } + // The five settings the reference implementation prints, in its order. + for _, want := range []string{"records = ", "delayMs = ", "maxConcurrency = ", "partitions = ", "replayFactor = "} { + if !strings.Contains(o.String(), want) { + t.Errorf("the fingerprint is missing %q:\n%s", want, o.String()) + } + } +} + +func TestBigReplay(t *testing.T) { + o, err := parseOptions([]string{"--records", "100", "--replay-factor", "3"}, noEnvironment) + if err != nil { + t.Fatalf("parse: %v", err) + } + if !o.bigReplayWanted() || o.bigReplayRecords() != 300 { + t.Errorf("wanted = %v, records = %d; want true and 300", o.bigReplayWanted(), o.bigReplayRecords()) + } + + // A factor of 1 or 0 skips the big replay, and the total then stays the small replay's own so + // that nothing downstream has to special-case it. + for _, factor := range []string{"1", "0"} { + o, err := parseOptions([]string{"--records", "100", "--replay-factor", factor}, noEnvironment) + if err != nil { + t.Fatalf("parse: %v", err) + } + if o.bigReplayWanted() { + t.Errorf("--replay-factor %s must skip the big replay", factor) + } + if o.bigReplayRecords() != 100 { + t.Errorf("--replay-factor %s: total = %d, want 100", factor, o.bigReplayRecords()) + } + } +} + +func TestHelpIsRecognisedByTheBinaryItself(t *testing.T) { + // Not only by run.sh: `docker compose run demo --help` reaches the binary directly. + for _, args := range [][]string{{"-h"}, {"--help"}, {"--records", "5", "--help"}} { + if !helpRequested(args) { + t.Errorf("%v must be recognised as a help request", args) + } + } + if helpRequested([]string{"--records", "5"}) { + t.Error("a plain run must not be treated as a help request") + } +} + +// TestTheTableIsTheContractsTable pins the columns, their order and their widths, because a reader +// who has run one language's demo is supposed to have run them all. The reference is the Java +// seed's ReferenceDemo#report. +func TestTheTableIsTheContractsTable(t *testing.T) { + if got, want := thousands(1234567), "1,234,567"; got != want { + t.Errorf("thousands(1234567) = %q, want %q", got, want) + } + if got, want := thousands(999), "999"; got != want { + t.Errorf("thousands(999) = %q, want %q", got, want) + } + if got, want := thousands(0), "0"; got != want { + t.Errorf("thousands(0) = %q, want %q", got, want) + } +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/report.go b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/report.go new file mode 100644 index 0000000000..ab07f1e61c --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/report.go @@ -0,0 +1,95 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +package main + +import ( + "fmt" + "os" + "strings" +) + +// logf is the demo's whole output channel: plain lines on stdout, no logging framework. +// +// The tables are the product here, and the shared contract fixes their columns and their order, so +// a framework's prefixes and levels would only get between the reader and them. Everything the +// sidecar itself emits goes to stderr, which keeps the two separable by a scripted caller. +func logf(format string, args ...any) { + fmt.Fprintf(os.Stdout, format+"\n", args...) +} + +func errorf(format string, args ...any) { + fmt.Fprintf(os.Stderr, format+"\n", args...) +} + +// armColumnWidth is the arm column, wide enough for the longest label an arm can carry now that +// each names the client it ran - `pc-go-grpc (this client)` is twenty-one characters against the +// seven `pc-go-grpc` used to need. Column WIDTH is deliberately not contract, unlike column identity +// and order: a language with a longer client name would otherwise be in permanent violation of an +// alignment rule it cannot keep. +const armColumnWidth = 26 + +// report prints one replay's table. +// +// SAME COLUMNS, IN THE SAME ORDER, AS EVERY OTHER LANGUAGE'S DEMO, so a reader who has run one has +// run them all - that is the whole point of the shared contract. Width is the one thing that is +// not shared, and armColumnWidth says why. THROUGHPUT ONLY as far as +// speed goes: the backlog is pre-produced, so the workload is closed-loop and per-record timings +// would be flattered by however far an arm fell behind. +// +// WHAT IT DID COMES BEFORE HOW FAST IT DID IT. `records` and `keys` are first because a rate on its +// own cannot show the work happened - a short arm reads as a fast one - and because they are the +// only two figures here that are DETERMINISTIC: every language over the same backlog reports the +// same pair, while elapsed and msg/s can never be compared across languages or machines. The three +// speed columns stay adjacent at the right, since `vs AK core` is derived from msg/s. +// +// baseline is the AK core arm the ratios are against, or nil when there is none to compare with. +// acrossReplays marks the big replay, whose ratios are against the SMALL replay's AK core - which +// is not like-for-like, and says so under the table rather than in a footnote nobody reads. +func report(title string, results []armResult, baseline *armResult, acrossReplays bool) { + logf("%s", renderReport(title, results, baseline, acrossReplays)) +} + +// renderReport is the table as text, split out from report so a test can assert the columns the +// contract fixes without capturing this process's stdout. +func renderReport(title string, results []armResult, baseline *armResult, acrossReplays bool) string { + var b strings.Builder + fmt.Fprintf(&b, "\n\n%s\n", title) + + comparison := "vs AK core" + if acrossReplays { + comparison = "vs AK core*" + } + fmt.Fprintf(&b, " %-*s %9s %8s %10s %12s %12s\n", + armColumnWidth, "arm", "records", "keys", "elapsed", "msg/s", comparison) + + for _, r := range results { + ratio := "-" + if baseline != nil && baseline.ratePerSecond() != 0 { + ratio = fmt.Sprintf("%.1fx", r.ratePerSecond()/baseline.ratePerSecond()) + } + fmt.Fprintf(&b, " %-*s %9s %8s %9.1fs %12s %12s\n", + armColumnWidth, r.arm, thousands(r.processed), thousands(r.keys), + r.elapsed.Seconds(), thousands(int(r.ratePerSecond())), ratio) + } + if acrossReplays { + b.WriteString("\n * against the SMALL replay's AK core arm. Across replays, so not like-for-like.\n") + } + return b.String() +} + +// thousands renders a count with comma separators, which Go's fmt has no verb for. +func thousands(n int) string { + sign := "" + if n < 0 { + sign, n = "-", -n + } + digits := fmt.Sprintf("%d", n) + var out strings.Builder + for i, d := range digits { + if i > 0 && (len(digits)-i)%3 == 0 { + out.WriteByte(',') + } + out.WriteRune(d) + } + return sign + out.String() +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/report_test.go b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/report_test.go new file mode 100644 index 0000000000..c57fff5d72 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/report_test.go @@ -0,0 +1,154 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +package main + +import ( + "strings" + "testing" + "time" +) + +// The output rules the shared contract - parallel-consumer-proxy/demo/README.md - states for every +// language's demo. They are checked here because the contract's own words are about STANDARD +// OUTPUT, and standard output is the one thing a unit test can hold a demo to without a broker. +// +// bin/ci-demo-conformance.sh checks the same shape ACROSS languages by requiring their skeletons to +// match. That is a drift check and needs eleven images; these are the same rules asserted for this +// one demo in a second, so a change that breaks them is caught before a container is ever built. + +// TestBannerOpensBySayingWhatItIs is the rule that came from someone watching a demo and being +// unimpressed: the first thing printed said `-grpc: the proxy granted 100 executor threads`, +// which does not contain the words Parallel Consumer. +func TestBannerOpensBySayingWhatItIs(t *testing.T) { + lines := strings.Split(strings.Trim(banner, "\n"), "\n") + if len(lines) != 4 { + t.Fatalf("the banner must be a rule, two lines and a rule; got %d lines:\n%s", len(lines), banner) + } + if lines[0] != lines[3] { + t.Errorf("the banner's two rules must match: %q vs %q", lines[0], lines[3]) + } + if got := len(lines[0]); got != 64 { + t.Errorf("the contract's rule is 64 characters wide, got %d", got) + } + if strings.Trim(lines[0], "=") != "" { + t.Errorf("the rule must be made of '=' alone, got %q", lines[0]) + } + if !strings.Contains(lines[1], "PARALLEL CONSUMER") { + t.Errorf("the banner must name the product, got %q", lines[1]) + } + // The one thing that differs per language. A banner that forgot it would still pass every + // check above and leave a reader unable to tell which of eleven demos they had run. + if !strings.Contains(lines[1], "Go demo") { + t.Errorf("the banner must name this language's demo, got %q", lines[1]) + } + if want := " The same records, twice: one at a time, then all at once."; lines[2] != want { + t.Errorf("the banner's second line is fixed by the contract:\n got %q\nwant %q", lines[2], want) + } +} + +// TestEveryArmNamesTheClientItRan guards the contract's "AK core is a category, not a client". A +// reader cannot judge a comparison without knowing what produced it, and the answer differs in +// every language. +func TestEveryArmNamesTheClientItRan(t *testing.T) { + if !strings.HasPrefix(armAKCore, "AK core") { + t.Errorf("the serial arm keeps the role name in full: %q", armAKCore) + } + if !strings.Contains(armAKCore, "franz-go") { + t.Errorf("the serial arm must name the Kafka client it actually ran: %q", armAKCore) + } + if !strings.Contains(armGoSidecar, "pc-go-grpc") { + t.Errorf("the sidecar arm keeps its language-and-transport name: %q", armGoSidecar) + } + if !strings.Contains(armGoSidecar, "(") { + t.Errorf("the sidecar arm must say what drives it: %q", armGoSidecar) + } + for _, arm := range []string{armAKCore, armGoSidecar} { + if len(arm) > armColumnWidth { + t.Errorf("arm %q is %d characters and the arm column is %d - the table would be ragged", + arm, len(arm), armColumnWidth) + } + } +} + +// TestTableReportsWhatEachArmDid is the pair of columns the contract added so that the table +// DEMONSTRATES the run rather than asserting it: throughput alone cannot show the work happened. +func TestTableReportsWhatEachArmDid(t *testing.T) { + results := []armResult{ + {arm: armAKCore, elapsed: 2 * time.Second, processed: 2000, keys: 1000}, + {arm: armGoSidecar, elapsed: time.Second, processed: 2000, keys: 1000}, + } + table := renderReport("Small replay", results, &results[0], false) + + header := lineContaining(t, table, "arm") + // Column IDENTITY and ORDER are contract; padding is not, so the check is on the sequence. + wantOrder := []string{"arm", "records", "keys", "elapsed", "msg/s", "vs AK core"} + if got := strings.Fields(strings.ReplaceAll(header, "vs AK core", "vs-AK-core")); len(got) != 6 { + t.Fatalf("the header must carry six columns, got %d in %q", len(got), header) + } + at := 0 + for _, column := range wantOrder { + next := strings.Index(header[at:], column) + if next < 0 { + t.Fatalf("column %q is missing or out of order in %q", column, header) + } + at += next + len(column) + } + + row := lineContaining(t, table, armGoSidecar) + for _, want := range []string{"2,000", "1,000"} { + if !strings.Contains(row, want) { + t.Errorf("the arm's row must report %s, got %q", want, row) + } + } + // Records and keys are the DETERMINISTIC pair; a rate is not, so a run that processed nothing + // must not be able to hide behind a plausible-looking elapsed. + empty := renderReport("Small replay", []armResult{{arm: armAKCore, elapsed: time.Second}}, nil, false) + if !strings.Contains(lineContaining(t, empty, armAKCore), " 0 ") { + t.Errorf("an arm that processed nothing must say 0, got:\n%s", empty) + } +} + +// TestNoLatencyAnywhere is an absolute the contract states and bin/ci-demo-conformance.sh also +// enforces: the backlog is pre-produced, so per-record timings are flattered by however far an arm +// fell behind, and reporting them would be dishonest rather than merely extra. +func TestNoLatencyAnywhere(t *testing.T) { + table := renderReport("Big replay", + []armResult{{arm: armGoSidecar, elapsed: time.Second, processed: 10, keys: 10}}, nil, true) + for _, forbidden := range []string{"latency", "p99", "p95", "percentile"} { + if strings.Contains(strings.ToLower(table), forbidden) { + t.Errorf("the table reported %q, which the contract forbids:\n%s", forbidden, table) + } + } +} + +// TestKeySetCountsDistinctKeys covers the counter both arms share, including the concurrent use the +// sidecar arm makes of it - `go test -race` is where that half earns its keep. +func TestKeySetCountsDistinctKeys(t *testing.T) { + keys := newKeySet() + done := make(chan struct{}) + for worker := 0; worker < 4; worker++ { + go func() { + defer func() { done <- struct{}{} }() + for i := 0; i < 250; i++ { + keys.add([]byte("key-" + string(rune('a'+i%7)))) + } + }() + } + for worker := 0; worker < 4; worker++ { + <-done + } + if got := keys.size(); got != 7 { + t.Errorf("1000 records over 7 distinct keys must count 7, got %d", got) + } +} + +func lineContaining(t *testing.T, text, want string) string { + t.Helper() + for _, line := range strings.Split(text, "\n") { + if strings.Contains(line, want) { + return line + } + } + t.Fatalf("no line containing %q in:\n%s", want, text) + return "" +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/run.sh b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/run.sh new file mode 100755 index 0000000000..01135dd2e0 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/run.sh @@ -0,0 +1,249 @@ +#!/usr/bin/env bash +# +# Copyright (C) 2026 Antony Stubbs and contributors +# +# The Go demo: the same records through Go's own Kafka client and through Go over the sidecar. One +# command, no setup. +# +# THE CONTRACT IS parallel-consumer-proxy/demo/README.md and the reference implementation is the +# Java seed's run.sh beside the Java client - same flags, same defaults, same two tables out. What +# is specific to Go is in README.md beside this file; the largest of it is here, in that this +# script starts the broker rather than the demo binary doing it. +set -euo pipefail + +cd "$(git rev-parse --show-toplevel)" + +DEMO_DIR="parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo" +COMPOSE_FILE="$DEMO_DIR/docker-compose.yml" + +usage() { + cat <<'USAGE' +usage: demo/run.sh [options] + + --records N records in the comparison replay (default 2000) + --delay-ms N simulated work per record, ms (default 2) + --concurrency N max in-flight records (default 100) + --partitions N partitions on the demo topic (default 10) + --replay-factor N big replay = records x N; 1 skips (default 20) + --bootstrap ADDR an existing broker; omit to start one + --topic NAME an existing topic; omit to create one + + --docker run in a container, broker as a compose sibling + --native run on this machine's own Go and JDK toolchains + -h, --help this + +With neither --docker nor --native, the demo picks: native when both toolchains are present, +a container otherwise. It says which it chose, and why, on the first line of output. + +Every flag also has an environment variable: --delay-ms is PC_DEMO_DELAY_MS. +USAGE +} + +# ${args[@]+...} rather than "${args[@]}": under `set -u`, bash 3.2 - which is what macOS ships, +# and what a first-time user runs this with - treats an EMPTY array expansion as an unbound +# variable and aborts. That fails exactly the no-argument case, which is the double-click case. +args=() +mode="auto" +bootstrap_supplied="${PC_DEMO_BOOTSTRAP:-}" +while [ $# -gt 0 ]; do + case "$1" in + --records|--delay-ms|--concurrency|--partitions|--replay-factor|--topic) + if [ $# -lt 2 ]; then + echo "$1 needs a value" >&2 + exit 2 + fi + args+=("$1" "$2"); shift 2 ;; + --bootstrap) + if [ $# -lt 2 ]; then + echo "$1 needs a value" >&2 + exit 2 + fi + # Remembered as well as forwarded: it is what decides whether this script starts a + # broker at all. It is never echoed - own-cluster mode puts a real address here. + bootstrap_supplied="$2" + args+=("$1" "$2"); shift 2 ;; + --docker) mode="docker"; shift ;; + --native) mode="native"; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "unknown option: $1" >&2; usage >&2; exit 2 ;; + esac +done + +# `./mvnw -v` rather than `command -v java`: the wrapper needs a JDK it can actually run, and a +# JRE-only or absent JAVA_HOME fails here rather than three minutes into a build. +have_jdk_toolchain() { + [ -x ./mvnw ] && ./mvnw -v >/dev/null 2>&1 +} + +# Asked of the module rather than of `go version`, because the answer depends on the module's own +# `go` directive and on GOTOOLCHAIN, and a version comparison in shell would have to know both. +have_go_toolchain() { + command -v go >/dev/null 2>&1 || return 1 + if (cd "$DEMO_DIR" && go list -m >/dev/null 2>&1); then + return 0 + fi + # This machine's Go is older than the module needs. Fetching a matching toolchain is Go's OWN + # DEFAULT behaviour, disabled only where GOTOOLCHAIN has been pinned to local - so the demo + # asks for it explicitly for this run rather than telling the reader to install something. + if (cd "$DEMO_DIR" && GOTOOLCHAIN=auto go list -m >/dev/null 2>&1); then + export GOTOOLCHAIN=auto + echo "This machine's Go is older than the demo needs; using Go's own toolchain download" \ + "(GOTOOLCHAIN=auto) for this run." + return 0 + fi + return 1 +} + +if [ "$mode" = "auto" ]; then + if have_go_toolchain && have_jdk_toolchain; then + mode="native" + echo "Mode: native - a Go toolchain and a JDK toolchain are both present on this machine." + else + mode="docker" + echo "Mode: container - this machine is missing a Go or JDK toolchain, so the demo brings its own." + fi +else + echo "Mode: $mode - asked for explicitly." + # An explicit --native still has to CHECK, and for a reason beyond a nicer error: the Go probe + # is also what turns GOTOOLCHAIN on when this machine's Go is too old, so skipping it here would + # let the run reach `go build` and fail there on a machine the auto path would have handled. + if [ "$mode" = "native" ]; then + if ! have_go_toolchain; then + echo "--native needs a Go toolchain this module can build with, and this machine has none." >&2 + exit 1 + fi + if ! have_jdk_toolchain; then + echo "--native needs a JDK toolchain to build the sidecar, and this machine has none." >&2 + exit 1 + fi + fi +fi + +if ! docker info >/dev/null 2>&1; then + if [ "$mode" = "docker" ]; then + echo "Docker is not running - the demo needs it for the broker and for its own image." >&2 + else + echo "Docker is not running - the demo starts its broker in a container." >&2 + fi + exit 1 +fi + +if [ "$mode" = "docker" ]; then + # The broker is a COMPOSE SIBLING and the demo container is never given the host Docker + # socket: a documented socket mount is root-equivalent host access taught as the normal way to + # run the product (plan unit U35). PC_DEMO_ARGS carries the flags through to the container, + # where the same parser reads them. + PC_DEMO_ARGS="${args[*]+${args[*]}}" \ + exec docker compose -f "$COMPOSE_FILE" up \ + --build --abort-on-container-exit --exit-code-from demo +fi + +# --------------------------------------------------------------------------------------------- +# Native. Three things have to exist before the demo binary runs: the sidecar, the broker, and the +# binary itself. +# --------------------------------------------------------------------------------------------- + +# THE SIDECAR IS A JVM APPLICATION, so "spawn the sidecar binary" means "spawn the JVM launcher +# with the proxy on its classpath". Computed here, once, and handed to the demo in a file - the +# alternative is Maven in the startup path of a demo whose whole promise is one command. +# +# -am is not optional: this module's parent is not in a single-module reactor and the enforcer's +# ReactorModuleConvergence rule fails the build before anything runs without it. +echo "Building the sidecar..." +./mvnw --batch-mode -q -pl :parallel-consumer-proxy -am -DskipTests \ + package dependency:build-classpath \ + '-Dmdep.outputFile=${project.build.directory}/sidecar-classpath.txt' + +SIDECAR_CLASSPATH_FILE="$PWD/$DEMO_DIR/../target/demo-sidecar-classpath.txt" +mkdir -p "$(dirname "$SIDECAR_CLASSPATH_FILE")" +{ + tr -d '\n' < parallel-consumer-proxy/target/sidecar-classpath.txt + printf ':%s/parallel-consumer-proxy/target/classes' "$PWD" +} > "$SIDECAR_CLASSPATH_FILE" +export PC_DEMO_SIDECAR_CLASSPATH="$SIDECAR_CLASSPATH_FILE" + +# THE BROKER, when the caller did not bring one. The demo binary never starts a broker in either +# mode - see README.md - so this is the only place it happens natively. +# +# The image is read out of the compose file rather than pinned twice: a compose file cannot derive +# anything, so the literal lives there and this reads it. Exactly one `image:` line is expected; +# anything else is a change to the compose file that has to be looked at rather than guessed past. +BROKER_CONTAINER="" +cleanup() { + if [ -n "$BROKER_CONTAINER" ]; then + docker rm -f "$BROKER_CONTAINER" >/dev/null 2>&1 || true + fi +} +trap cleanup EXIT + +if [ -z "$bootstrap_supplied" ]; then + BROKER_IMAGE="$(grep -E '^[[:space:]]*image:[[:space:]]' "$COMPOSE_FILE" | awk '{print $2}')" + if [ "$(printf '%s\n' "$BROKER_IMAGE" | grep -c .)" -ne 1 ]; then + echo "expected exactly one image: line in $COMPOSE_FILE, found: $BROKER_IMAGE" >&2 + exit 1 + fi + echo "No broker supplied, starting one in a container: $BROKER_IMAGE" + + # A random high port, retried on collision, because five demos may be running at once on one + # machine and a fixed port makes the second one fail. Docker itself is the free-port test: if + # the port is taken the run fails and the next attempt picks another. + started="" + for _ in 1 2 3 4 5; do + BROKER_PORT=$(( 20000 + (RANDOM % 20000) )) + # CONTROLLER binds a ROUTABLE name rather than 0.0.0.0: Kafka treats an unadvertised + # listener's bind address as its advertised one when it formats KRaft storage, and refuses + # the meta-address there. Measured - 0.0.0.0 fails at the format step, before the broker + # starts, with "advertised.listeners cannot use the nonroutable meta-address". + if BROKER_CONTAINER=$(docker run -d \ + -e CLUSTER_ID=MkU3OEVBNTcwNTJENDM2Qk \ + -e KAFKA_NODE_ID=1 \ + -e KAFKA_PROCESS_ROLES=broker,controller \ + -e KAFKA_LISTENERS="PLAINTEXT://0.0.0.0:${BROKER_PORT},CONTROLLER://localhost:9093" \ + -e KAFKA_ADVERTISED_LISTENERS="PLAINTEXT://localhost:${BROKER_PORT}" \ + -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \ + -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \ + -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@localhost:9093 \ + -e KAFKA_INTER_BROKER_LISTENER_NAME=PLAINTEXT \ + -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ + -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 \ + -e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \ + -e KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=500 \ + -p "127.0.0.1:${BROKER_PORT}:${BROKER_PORT}" \ + "$BROKER_IMAGE" 2>/dev/null); then + started="yes" + break + fi + BROKER_CONTAINER="" + done + if [ -z "$started" ]; then + echo "could not start a broker container on any of five candidate ports" >&2 + exit 1 + fi + + # Ready means "can serve an API request", not "the port is open" - the same thing the compose + # sibling's healthcheck asserts, for the same reason. + ready="" + for _ in $(seq 1 60); do + if docker exec "$BROKER_CONTAINER" \ + kafka-topics --bootstrap-server "localhost:${BROKER_PORT}" --list >/dev/null 2>&1; then + ready="yes" + break + fi + sleep 1 + done + if [ -z "$ready" ]; then + echo "the broker did not become ready within 60s:" >&2 + docker logs "$BROKER_CONTAINER" 2>&1 | tail -20 >&2 + exit 1 + fi + args+=(--bootstrap "localhost:${BROKER_PORT}") +fi + +# Built rather than `go run`: `go run` puts the compiler's output in a temp directory and adds a +# process between the shell and the demo, and the demo spawns a child whose parent-death pipe must +# come from the demo itself. +DEMO_BINARY="$PWD/$DEMO_DIR/../target/pc-go-demo" +echo "Building the demo..." +(cd "$DEMO_DIR" && go build -o "$DEMO_BINARY" .) + +"$DEMO_BINARY" ${args[@]+"${args[@]}"} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/sidecar.go b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/sidecar.go new file mode 100644 index 0000000000..4ebf6db727 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/demo/sidecar.go @@ -0,0 +1,93 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +package main + +import ( + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" +) + +// sidecarClasspathEnv names the file holding the sidecar's classpath. +// +// IT IS PLUMBING, NOT AN EIGHTH FLAG. The contract's environment rule is that every FLAG has a +// PC_DEMO_ variable; this one has no flag and no default a user would set, because it exists only +// because the shipped sidecar is not yet a binary. run.sh writes the file and sets this; the +// Dockerfile bakes both in. demo/README.md records it as a Go divergence for the integrator. +const sidecarClasspathEnv = "PC_DEMO_SIDECAR_CLASSPATH" + +// mainClass is the sidecar's entry point: bz.stub.parallelconsumer.proxy.Main, which binds an +// ephemeral loopback port, prints "port: ", and serves until its parent dies. +const mainClass = "bz.stub.parallelconsumer.proxy.Main" + +// sidecarCommand is what the client library spawns: an ABSOLUTE binary path plus its arguments. +// +// TODAY THAT BINARY IS THE JVM LAUNCHER AND THE PROXY IS A CLASSPATH ARGUMENT, which is a fact +// about this repository rather than about the product: the sidecar ships as a JVM application and +// has no native launcher yet. The library's contract is unaffected - it spawns one process +// directly, with no shell between it and the child, so the pipe this process holds is the one the +// sidecar watches for parent death. +type sidecarCommand struct { + path string + args []string +} + +// resolveSidecar assembles that command, or explains what is missing. +func resolveSidecar() (sidecarCommand, error) { + java, err := javaLauncher() + if err != nil { + return sidecarCommand{}, err + } + classpath, err := sidecarClasspath() + if err != nil { + return sidecarCommand{}, err + } + return sidecarCommand{path: java, args: []string{"-cp", classpath, mainClass}}, nil +} + +// javaLauncher resolves the JVM launcher to an absolute path, preferring JAVA_HOME. +// +// A PATH LOOKUP IS ACCEPTABLE HERE AND NOWHERE IN THE LIBRARY. The library refuses a relative or +// PATH-resolved SidecarPath because it hands that binary the user's Kafka credentials; this demo is +// choosing a JVM to run a proxy it also built, against a broker it also seeded, and the same +// reasoning that lets internal/harness look java up applies. What the library receives is still +// absolute, which is the property that matters to it. +func javaLauncher() (string, error) { + if home := os.Getenv("JAVA_HOME"); home != "" { + candidate := filepath.Join(home, "bin", "java") + if _, err := os.Stat(candidate); err == nil { + return candidate, nil + } + } + found, err := exec.LookPath("java") + if err != nil { + return "", fmt.Errorf("the demo found no JVM to run the sidecar with - set JAVA_HOME, or "+ + "run the demo in its container (demo/run.sh --docker): %w", err) + } + return filepath.Abs(found) +} + +// sidecarClasspath reads the classpath file the entry point wrote. +// +// It FAILS rather than guessing. A demo that silently ran a differently-built sidecar would report +// throughput for an engine nobody asked for, and the message names the command that produces the +// file rather than leaving the reader to find it. +func sidecarClasspath() (string, error) { + path := strings.TrimSpace(os.Getenv(sidecarClasspathEnv)) + if path == "" { + return "", errors.New(sidecarClasspathEnv + " is not set, so the demo does not know where " + + "the sidecar is - start the demo through demo/run.sh, which builds it and sets this") + } + raw, err := os.ReadFile(path) + if err != nil { + return "", fmt.Errorf("reading the sidecar classpath from %s: %w", path, err) + } + classpath := strings.TrimSpace(string(raw)) + if classpath == "" { + return "", fmt.Errorf("the sidecar classpath at %s is empty", path) + } + return classpath, nil +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/pom.xml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/pom.xml index 2409f8a01c..6ac976c4ce 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/pom.xml +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/pom.xml @@ -148,11 +148,19 @@ repo forbids, not the one it wants. `go vet` and `go tool staticcheck` are the same story. - THE ONE BUILT ARTEFACT IS ALREADY COVERED. The shared conformance suite prebuilds + THE BUILT ARTEFACTS ARE ALREADY COVERED, because every one of them lands in target/. + The shared conformance suite prebuilds `go build -o target/conformance-runner ./cmd/conformance-runner` - grep - `target/conformance-runner` in the conformance module's LanguageRunners - and Maven's - default clean takes target/ whole, no configuration needed. Proven both ways: the - binary is there after the suite's prebuild and gone after `mvn clean`. + `target/conformance-runner` in the conformance module's LanguageRunners - and the + demo's run.sh writes `target/pc-go-demo` and `target/demo-sidecar-classpath.txt`. + Maven's default clean takes target/ whole, no configuration needed. Proven both ways: + the conformance binary is there after the suite's prebuild and gone after + `mvn clean`. + + THE DEMO IS A NESTED GO MODULE (demo/go.mod), so `go build ./...` above does not + descend into it and this module's Maven build never compiles it - demo/run.sh and + demo/Dockerfile do. demo/README.md says why the module is nested: franz-go must not + reach an application that uses the client library. gen/ IS SOURCE, NOT OUTPUT. The protobuf stubs under gen/ are committed on purpose, because Go has no codegen step at `go get` time - scripts/generate-proto.sh says so diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/scripts/analyse.sh b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/scripts/analyse.sh index a4db4c522d..9023af9287 100755 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/scripts/analyse.sh +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-go/scripts/analyse.sh @@ -40,4 +40,15 @@ go vet ./... echo "==> staticcheck ./... (pinned in go.mod)" go tool staticcheck ./... +# THE DEMO IS A NESTED MODULE (demo/go.mod), so neither `./...` above reaches it - Go's package +# patterns stop at a directory that has its own go.mod. It is analysed here explicitly, or it is +# analysed nowhere. +# +# `go vet` ONLY, AND THAT IS A GAP RATHER THAN A JUDGEMENT. `go tool staticcheck` builds the version +# pinned in the module it runs in, and the demo module does not pin it - adding the directive would +# put staticcheck's dependency tree into the module graph that the demo's Dockerfile resolves, and +# that image was verified without it. Recorded as an open follow-up in docs/inflight/clients/go.md. +echo "==> go vet ./... (demo, a nested module)" +(cd demo && go vet ./...) + echo "==> static analysis clean" diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/Dockerfile b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/Dockerfile new file mode 100644 index 0000000000..898c8bb81b --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/Dockerfile @@ -0,0 +1,67 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE DEMO CONTAINER for the Kotlin client (astubbs#242, plan unit U35, governed by KTD40). +# +# The contract requires one per language so that a reader with only Docker can run the demo - and +# R72 is explicit that a JVM language is included, not exempted: a visitor evaluating "does this +# work in my language" reads an exemption as an admission that the other languages are the hard +# case. +# +# THIS IS A RUNTIME IMAGE. It keeps its toolchain rather than exporting past it, because it is +# something a person runs rather than a containerised build that hands artifacts back to the host. +# +# THE BUILD CONTEXT IS THE REPOSITORY, which departs from the ordinary rule that a module directory +# is the only context. A Maven reactor module cannot be built without its parents and the modules +# it depends on. The hazard that rule protects against - uploading other agents' target/ output - +# is addressed directly by the repository-root .dockerignore, which excludes every target/ +# directory, .git and .claude/worktrees. Read the two together; if you add a build output directory +# to this repo, add it there too. +# +# WHY THE DEPENDENCIES ARE BAKED IN RATHER THAN CACHE-MOUNTED. A BuildKit cache mount on /root/.m2 +# is the obvious way to keep the Maven downloads out of the image, and it is wrong here: a cache +# mount is not part of the resulting image, so the classpath computed below would point at jars the +# running container does not have. The image is large as a result - that is the trade a runtime +# image makes and a build image does not. +# +# A PLAIN JDK, NOT THE `maven:` IMAGE. That image sets MAVEN_CONFIG=/root/.m2 and the Maven WRAPPER +# appends $MAVEN_CONFIG to its own command line, so `./mvnw ` inside it dies with +# `Unknown lifecycle phase "/root/.m2"`. Using the image's own `mvn` would dodge that and break the +# repo's rule that builds go through the wrapper (AGENTS.md, "Build Requirements"). +FROM eclipse-temurin:17-jdk + +WORKDIR /src +COPY . . + +# ONE Maven invocation, with the classpath computed at BUILD time into a file. Computing it at run +# time would put Maven in the startup path of a demo whose whole promise is one command. +# +# -Dpc.kotlinDemo activates the kotlin-demo profile: it compiles demo/src and puts the engine on +# the classpath, which the demo hands to the sidecar it spawns. test-compile rather than package, +# because the demo compiles into this module's test output tree - it is an application that uses +# the client, not part of the published library surface. -DskipTests rather than +# -Dmaven.test.skip, because the latter would skip the very compilation the demo needs. +# +# -Dcopyright.skip=true because the header check needs the repository's git history to find its +# fork point, and .dockerignore excludes .git from the context. Skipping it here loses nothing: it +# is a source-tree gate that already runs in every ordinary build and in CI's own copyright lane. +RUN ./mvnw --batch-mode -q -DskipTests -Dcopyright.skip=true -Dpc.kotlinDemo \ + -pl :parallel-consumer-proxy-client-kotlin -am test-compile \ + && mkdir -p /app \ + && MODULE_DIR=parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin \ + && cp "$MODULE_DIR/target/demo-classpath.txt" /app/cp.txt \ + && printf ':/src/%s/target/classes:/src/%s/target/test-classes' "$MODULE_DIR" "$MODULE_DIR" >> /app/cp.txt + +# A REAL FORKED JVM, not `mvn exec:java`, and this is load-bearing rather than stylistic. The client +# library spawns the sidecar as a child process and hands it this JVM's java.class.path. Under +# exec:java that property would be Maven's own classpath and the child would come up without the +# engine on it. The native path in run.sh forks for exactly the same reason. +# +# -Dlogback.configurationFile is not decoration: parallel-consumer-core's test output is on this +# classpath and carries a logback-test.xml, which logback prefers over any logback.xml whatever the +# classpath order - so the demo's own configuration is ignored unless it is named. run.sh passes +# the same flag. +# +# PC_DEMO_ARGS is unquoted on purpose: it carries the caller's flags through compose as a single +# string and must word-split into separate arguments. +ENTRYPOINT ["/bin/sh", "-c", \ + "exec java -cp \"$(cat /app/cp.txt)\" -Dlogback.configurationFile=pc-kotlin-demo-logback.xml bz.stub.parallelconsumer.client.coroutines.demo.KotlinDemoKt $PC_DEMO_ARGS"] diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/README.md b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/README.md new file mode 100644 index 0000000000..f941c14cf7 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/README.md @@ -0,0 +1,119 @@ +# The Kotlin demo + +```bash +# from anywhere in the repo - picks native or container for you +parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/run.sh + +# or, from this directory, the plain container path with nothing else needed +docker compose up +``` + +Needs Docker. A JDK is optional: with one, the demo runs natively and starts its broker in a +container; without one, the demo runs in a container too and the broker is a compose sibling. +`run.sh` announces which it chose, and why, before it starts anything; the demo itself then opens +with the shared banner naming the product, ahead of its effective configuration and every log line. + +**The contract this keeps - and that every other language's demo keeps - is +[`parallel-consumer-proxy/demo/README.md`](../../../parallel-consumer-proxy/demo/README.md).** Read +that first. This file records only what is specific to Kotlin. + +## The two arms + +| arm, as the table labels it | what it is | +|---|---| +| **`AK core (KafkaConsumer)`** | Apache Kafka's own `KafkaConsumer`, driven from Kotlin, one record at a time | +| **`pc-kotlin-grpc (this client)`** | this module's `ParallelConsumerClient`: it spawns the sidecar as a child process, receives records over a socket, runs a **suspending** function on them and reports outcomes back | + +The labels carry the client and not only the role, because "AK core" is a *category*: it is +`franz-go` in Go and `rdkafka` in Ruby, and a reader cannot judge a comparison without knowing which +one produced it. + +**Kotlin has no second serious Kafka client to run as a third arm.** There is no Kotlin-native +implementation of the Kafka protocol: the Kotlin-facing libraries are wrappers over this same +`kafka-clients` jar, so a second serial arm would be two wrappers over one client rather than two +clients - which is why this arm names the underlying client instead of inventing a Kotlin-sounding +label for it. + +Both arms report `records` and `keys` beside their timings, and both figures are **deterministic**: +the backlog seeded for a replay is exactly the arm's target, so `records` equals it rather than +approximately equalling it, and the seeder writes `key-{i % 1000}`, so `keys` is +`min(records, 1000)`. They are what a reader - and `bin/ci-demo-conformance.sh` - can hold two +languages to; elapsed and msg/s never can be. + +Two arms is the whole contract everywhere except Java. Java can hold the engine, the client library +in process and the raw wire in one JVM, so each of its extra pairs changes exactly one term; Kotlin +has no second Kafka client to compare a wrapper against, so a third arm here would compare two +things at once and mean nothing. + +**The sidecar arm goes through the client library, not the protocol.** An earlier version of the +Java demo spoke gRPC by hand: it proved the engine worked and said nothing about the client library, +which is the artifact users actually touch. Everything the sidecar arm does here - the spawn, the +handshake, the dispatch, the report, the reap - is code in `../src/main/kotlin`, called the way an +application calls it. + +## What is specific to Kotlin + +### The simulated work is `delay`, not `Thread.sleep` + +**No longer a divergence.** The shared contract used to name nine languages where a blocking sleep +was fine, Kotlin among them, and this demo took the rule over the list. The contract has since +replaced the list with the predicate - *is the client thread-per-record?* - and by that predicate +this client is not, so `delay` is now what the contract asks for rather than an exception to it. +The measurement below is what settled it, and is kept because it is the evidence. + +The user's function here is `suspend (InboundRecord) -> Outcome`, and each record runs as a +coroutine on `Dispatchers.IO`. `Dispatchers.IO` has a default parallelism of 64. So a blocking sleep +inside that coroutine occupies one of 64 threads, and **caps in-flight records at 64 however high +`--concurrency` is set** - while the effective-configuration block goes on printing the number the +reader asked for. That is precisely the failure the fingerprint exists to prevent: a throughput +figure reported against settings that did not apply. + +Observed rather than assumed. At `--records 4000 --delay-ms 50 --concurrency 200 --replay-factor 1`, +the sidecar arm finished in **2.7s**. 4000 records at 50ms each is 200 record-seconds of work, so +the mean number of records in flight was about **74** - above the 64 a blocking sleep could ever +have reached. The suspending wait demonstrably ran wider than a blocking one can. (That is an +inference from one arm rather than a two-arm controlled experiment; a loaded machine can only push +the in-flight figure *down*, never above the thread ceiling, so the direction of the conclusion is +safe even though the absolute rate is not.) + +The AK core arm still uses `Thread.sleep`, and that is not an inconsistency: it is a serial loop +with nothing to interleave, so blocking and suspending cost it the same wall clock. The distinction +only bites where records run concurrently. + +### The demo is not a Maven module + +The Java demo is. This one lives inside the client module, under `demo/`, compiled by the +`kotlin-demo` profile in [`../pom.xml`](../pom.xml). A new module needs a line in the clients +aggregator pom, and that file is shared by every language wave at once. + +Two consequences worth knowing before you edit anything here: + +- **The demo compiles into the module's test output tree** (`target/test-classes`), because the + module's published surface is guarded by `-Xexplicit-api=strict` and a demo is not part of it. + Its sources live under `demo/src`, not `src/test`, so nothing reads it as a test. The module + already puts a program-that-uses-the-client there - `scripts/conformance-runner` launches one. +- **The engine dependency is behind `-Dpc.kotlinDemo`.** The sidecar arm hands the spawned child + this JVM's classpath, so the demo needs `parallel-consumer-proxy` on it - but declaring that + unconditionally would give the module a permanent reactor edge to the engine, and `bin/build.sh` + opens with `clean`, so an ordinary build of this module would delete the sidecar jar every other + language's conformance test spawns. The invariant still holds and is measurable: + + ```bash + ./mvnw -pl :parallel-consumer-proxy-client-kotlin -am validate # must not print parallel-consumer-proxy + ``` + +### The logging configuration has a deliberately strange name + +`pc-kotlin-demo-logback.xml`, named explicitly in `-Dlogback.configurationFile` by both `run.sh` and +the `Dockerfile`. `parallel-consumer-core`'s test output is on this demo's classpath and carries a +`logback-test.xml`, which logback prefers over any `logback.xml` whatever the classpath order - so +named the obvious thing, the demo's own configuration was silently ignored and the first run printed +core's INFO levels over the tables. The demo's own output goes through `println`, not the logger, so +it survives any level a reader or a container imposes. + +## What has not been done + +- **No CI entry-point test.** `bin/ci-demo-test.sh` runs the *Java* demo through both entry points + on every pull request. Nothing yet does that for this one, and the contract's "both entry points + are tested" clause is therefore unmet here. Tracked in + [`docs/inflight/clients/kotlin.md`](../../../docs/inflight/clients/kotlin.md). diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/docker-compose.yml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/docker-compose.yml new file mode 100644 index 0000000000..fab1db43b9 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/docker-compose.yml @@ -0,0 +1,107 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE ONE-COMMAND PATH for the Kotlin demo (astubbs#242, plan unit U35, R20). +# +# THE BROKER IS A SIBLING, AND THAT IS A RULE RATHER THAN A LAYOUT CHOICE. The demo could start its +# own broker with Testcontainers - it does exactly that on the native path - but doing so from +# inside a container requires mounting the host Docker socket, and U35 forbids it: a documented +# socket mount is root-equivalent host access taught as the normal way to run the product. So +# broker mode inside a container reaches a compose sibling on the demo's own network, and the demo +# container is given an address instead of a socket. +# +# THE SIDECAR IS NOT A SERVICE HERE, deliberately. It would be easy to add one and it would teach +# the wrong thing: KTD41's product model is that the client library spawns and supervises the +# sidecar itself, so the user never installs, deploys or operates a process. A compose service +# would show a deployment the product does not ask for. The Kotlin client spawns it as a child +# process inside its own container, exactly as it does natively - Sidecar.kt is the code that does +# it, and it is the same code either way. +# +# TWO WAYS IN, AND BOTH ARE SUPPORTED: +# +# cd && docker compose up # the plain one, defaults, nothing else needed +# demo/run.sh --docker [--records N ...] # the same thing with the demo's flags +# +# `docker compose up` on its own is the path to keep working: it is what a reader who has never +# seen this repository will type. It needs no arguments, no environment and no wrapper script - the +# demo service runs with the demo's own defaults, and PC_DEMO_ARGS below is empty unless run.sh +# sets it. Add --build after changing any source, and `docker compose down` to remove the broker. +# +# The demo container exits when the run finishes; the broker keeps running until you stop it. +# run.sh passes --abort-on-container-exit --exit-code-from demo so a scripted run stops on its own +# and reports the demo's exit code rather than compose's. + +name: pc-kotlin-demo + +services: + broker: + # Pinned as a literal because a compose file cannot derive anything. It must track + # in the root pom the same way DemoBroker's brokerImage does at run time - the + # mapping is CP major = AK major + 4, so AK 3.9.x means CP 7.9. + image: confluentinc/cp-kafka:7.9.0 + environment: + CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk + # QUIET. A demo is a thing someone WATCHES, and a KRaft broker at INFO emits hundreds of + # lines of its own configuration, controller elections and log-segment chatter - which buries + # the two tables the demo exists to show. Kafka's own warnings and errors still come through. + KAFKA_LOG4J_ROOT_LOGLEVEL: WARN + KAFKA_TOOLS_LOG4J_LOGLEVEL: WARN + # ROOT ALONE IS NOT ENOUGH, and this was measured rather than assumed: the cp-kafka image + # writes kafka=INFO, kafka.controller=TRACE and state.change.logger=TRACE into its own + # log4j config, and a named logger beats the root. With only the root set, one demo run + # interleaved 927 broker lines with 35 of the demo's own. These are the named loggers that + # actually produce the noise. + KAFKA_LOG4J_LOGGERS: "kafka=WARN,kafka.controller=WARN,kafka.server=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.network=WARN,kafka.request.logger=WARN,state.change.logger=WARN,org.apache.kafka=WARN" + KAFKA_NODE_ID: 1 + KAFKA_PROCESS_ROLES: broker,controller + # Bound to the service name, not 0.0.0.0: this image formats its KRaft storage with the same + # config it boots with, and the format step refuses a non-routable advertised address. + KAFKA_LISTENERS: PLAINTEXT://broker:9092,CONTROLLER://broker:9093 + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092 + KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT + KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093 + KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + # The demo forms one consumer group per arm per replay, and the default three-second settling + # delay would be charged to every one of them. The native path sets the same value. + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 500 + healthcheck: + # The demo must not start before the broker can serve an API request. Listing topics is the + # cheapest call that proves that, and it proves more than a port being open does. + test: ["CMD", "kafka-topics", "--bootstrap-server", "broker:9092", "--list"] + interval: 5s + timeout: 10s + retries: 30 + start_period: 10s + + demo: + build: + # The repository, because a Maven reactor module cannot build without its parents and the + # modules it depends on. The root .dockerignore is what keeps this from uploading every + # target/ directory in the tree; the Dockerfile explains why that pair had to exist. + context: ../../.. + dockerfile: parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/Dockerfile + # A demo that has printed its tables is done. Without this, compose's default would restart a + # container that exited cleanly and the reader would watch the whole run happen again. + restart: "no" + depends_on: + broker: + condition: service_healthy + environment: + # Supplying this is what stops the demo trying to start a broker of its own. + PC_DEMO_BOOTSTRAP: broker:9092 + # The flags demo/run.sh was called with, forwarded as one string. + PC_DEMO_ARGS: ${PC_DEMO_ARGS:-} + # EVERY OTHER DIAL, forwarded explicitly. Compose passes nothing it is not told to, so a + # missing line here is a dial the container silently ignores while the run still prints a + # fingerprint claiming otherwise. Caught across seven languages at once by + # bin/ci-demo-conformance.sh, which is why that check exists. Adding a flag means adding it + # here too. + PC_DEMO_RECORDS: ${PC_DEMO_RECORDS:-} + PC_DEMO_DELAY_MS: ${PC_DEMO_DELAY_MS:-} + PC_DEMO_CONCURRENCY: ${PC_DEMO_CONCURRENCY:-} + PC_DEMO_PARTITIONS: ${PC_DEMO_PARTITIONS:-} + PC_DEMO_REPLAY_FACTOR: ${PC_DEMO_REPLAY_FACTOR:-} + PC_DEMO_TOPIC: ${PC_DEMO_TOPIC:-} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/run.sh b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/run.sh new file mode 100755 index 0000000000..a6f46ff85f --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/run.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +# +# Copyright (C) 2026 Antony Stubbs and contributors +# +# The Kotlin demo: the same records through Kotlin's own Kafka client and through Kotlin over the +# sidecar. One command, no setup. +# +# It mirrors the reference demo's interface exactly - same flags, same defaults, same two tables. +# The contract is parallel-consumer-proxy/demo/README.md; what is specific to Kotlin is in the +# README beside this file. +set -euo pipefail + +cd "$(git rev-parse --show-toplevel)" + +MODULE_DIR="parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin" +DEMO_DIR="$MODULE_DIR/demo" +MODULE=":parallel-consumer-proxy-client-kotlin" +MAIN_CLASS="bz.stub.parallelconsumer.client.coroutines.demo.KotlinDemoKt" + +usage() { + cat <<'USAGE' +usage: demo/run.sh [options] + + --records N records in the comparison replay (default 2000) + --delay-ms N simulated work per record, ms (default 2) + --concurrency N max in-flight records (default 100) + --partitions N partitions on the demo topic (default 10) + --replay-factor N big replay = records x N; 1 skips (default 20) + --bootstrap ADDR an existing broker; omit to start one + --topic NAME an existing topic; omit to create one + + --docker run in a container, broker as a compose sibling + --native run on this machine's own JDK toolchain + -h, --help this + +With neither --docker nor --native, the demo picks: native when a JDK toolchain is present, +a container otherwise. It says which it chose, and why, on the first line of output. + +Every flag also has an environment variable: --delay-ms is PC_DEMO_DELAY_MS. +USAGE +} + +# ${args[@]+...} rather than "${args[@]}": under `set -u`, bash 3.2 - which is what macOS ships, +# and what a first-time user runs this with - treats an EMPTY array expansion as an unbound +# variable and aborts. That fails exactly the no-argument case, which is the double-click case. +args=() +mode="auto" +while [ $# -gt 0 ]; do + case "$1" in + --records|--delay-ms|--concurrency|--partitions|--replay-factor|--bootstrap|--topic) + if [ $# -lt 2 ]; then + echo "$1 needs a value" >&2 + exit 2 + fi + args+=("$1" "$2"); shift 2 ;; + --docker) mode="docker"; shift ;; + --native) mode="native"; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "unknown option: $1" >&2; usage >&2; exit 2 ;; + esac +done + +have_jdk_toolchain() { + # `mvnw -v` rather than `command -v java`: the wrapper needs a JDK it can actually run, and a + # JRE-only or absent JAVA_HOME fails here rather than three minutes into a build. + [ -x ./mvnw ] && ./mvnw -v >/dev/null 2>&1 +} + +if [ "$mode" = "auto" ]; then + if have_jdk_toolchain; then + mode="native" + echo "Mode: native - a JDK toolchain is present on this machine." + else + mode="docker" + echo "Mode: container - no JDK toolchain here, so the demo brings its own." + fi +else + echo "Mode: $mode - asked for explicitly." +fi + +if ! docker info >/dev/null 2>&1; then + if [ "$mode" = "docker" ]; then + echo "Docker is not running - the demo needs it for the broker and for its own image." >&2 + else + echo "Docker is not running - the demo starts its broker in a container." >&2 + fi + exit 1 +fi + +if [ "$mode" = "docker" ]; then + # The broker is a COMPOSE SIBLING and the demo container is never given the host Docker socket: + # a documented socket mount is root-equivalent host access taught as the normal way to run the + # product (plan unit U35). PC_DEMO_ARGS carries the flags through to the container, where the + # same parser reads them. + PC_DEMO_ARGS="${args[*]+${args[*]}}" \ + exec docker compose -f "$DEMO_DIR/docker-compose.yml" up \ + --build --abort-on-container-exit --exit-code-from demo +fi + +# -Dpc.kotlinDemo activates the kotlin-demo profile, which is what compiles demo/src and puts the +# ENGINE on the classpath. It is a profile rather than the default build because an unconditional +# dependency here would give this module a permanent reactor edge to parallel-consumer-proxy, and +# `bin/build.sh` opens with `clean` - see the module's pom. +# +# -am is not optional: this module's parent is not in a single-module reactor and the enforcer's +# ReactorModuleConvergence rule fails the build before anything runs without it. +# +# test-compile, not package: the demo compiles into the module's test output tree, because it is an +# application that uses the client rather than part of the published library surface. +# +# -DskipTests, not -Dmaven.test.skip: the latter would skip test COMPILATION, which is where the +# demo is built. +./mvnw --batch-mode -q -pl "$MODULE" -am -DskipTests -Dpc.kotlinDemo test-compile + +CLASSPATH_FILE="$MODULE_DIR/target/demo-classpath.txt" +if [ ! -f "$CLASSPATH_FILE" ]; then + echo "$CLASSPATH_FILE was not written - the kotlin-demo profile did not run." >&2 + exit 1 +fi + +# A REAL FORKED JVM, and this is load-bearing rather than stylistic. The client library spawns the +# sidecar as a child process and hands it this JVM's java.class.path. Under `mvn exec:java` the main +# would run inside MAVEN's JVM, so that property would be Maven's own classpath and the child +# sidecar would come up without the engine on it. The container path forks for the same reason. +# +# target/classes is the client library; target/test-classes is where the demo's own classes and its +# logging configuration land. +# +# -Dlogback.configurationFile is NOT decoration. parallel-consumer-core's test output is on this +# classpath and carries a logback-test.xml, which logback prefers over any logback.xml whatever the +# classpath order - so the demo's own configuration was silently ignored until it was named +# explicitly. The Dockerfile passes the same flag. +exec java -cp "$(cat "$CLASSPATH_FILE"):$MODULE_DIR/target/classes:$MODULE_DIR/target/test-classes" \ + -Dlogback.configurationFile=pc-kotlin-demo-logback.xml \ + "$MAIN_CLASS" ${args[@]+"${args[@]}"} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/kotlin/bz/stub/parallelconsumer/client/coroutines/demo/DemoBroker.kt b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/kotlin/bz/stub/parallelconsumer/client/coroutines/demo/DemoBroker.kt new file mode 100644 index 0000000000..c82420a497 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/kotlin/bz/stub/parallelconsumer/client/coroutines/demo/DemoBroker.kt @@ -0,0 +1,186 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +package bz.stub.parallelconsumer.client.coroutines.demo + +import org.apache.kafka.clients.admin.Admin +import org.apache.kafka.clients.admin.AdminClientConfig +import org.apache.kafka.clients.admin.NewTopic +import org.apache.kafka.clients.producer.KafkaProducer +import org.apache.kafka.clients.producer.ProducerConfig +import org.apache.kafka.clients.producer.ProducerRecord +import org.apache.kafka.common.errors.TopicExistsException +import org.apache.kafka.common.serialization.ByteArraySerializer +import org.apache.kafka.common.utils.AppInfoParser +import org.testcontainers.containers.KafkaContainer +import org.testcontainers.utility.DockerImageName +import java.util.Properties +import java.util.concurrent.ExecutionException +import java.util.concurrent.atomic.AtomicReference + +/** + * The broker the demo reads from, however the reader got here. + * + * Two ways in, and the second one is a rule rather than a convenience: + * + * - **Nothing supplied** - the demo starts a real broker in a container, which is KTD40's honest + * default because it is what a user actually runs. + * - **An address supplied** - the demo uses it and starts nothing. This is how the demo runs + * *inside* its own container, and it is not optional there: **a demo container is never granted + * the host Docker socket** (plan unit U35), so it could not start a broker even if it wanted to. + * It reaches a compose sibling on the demo's own network instead. + * + * The same door serves own-cluster mode, where the address is the user's real cluster - so nothing + * here logs or echoes it. + */ +internal class DemoBroker private constructor( + val bootstrap: String, + private val container: KafkaContainer?, +) : AutoCloseable { + + /** + * Creates the demo's topic, tolerating one a previous run left behind - but not one with a + * different partition count, because the fingerprint would then print a `--partitions` value + * that never applied, and that block is the demo's whole reproducibility promise. + */ + fun ensureTopic(topic: String, partitions: Int) { + adminProperties().let { config -> + Admin.create(config).use { admin -> + try { + admin.createTopics(listOf(NewTopic(topic, partitions, 1.toShort()))).all().get() + Console.say("Created topic $topic with $partitions partitions") + } catch (e: ExecutionException) { + if (e.cause !is TopicExistsException) { + throw IllegalStateException("could not create the demo topic $topic", e.cause) + } + val existing = partitionsOf(admin, topic) + check(existing == partitions) { + "topic $topic already exists with $existing partitions, but this run asked for " + + "$partitions - pass --topic to name a fresh one, or --partitions $existing" + } + Console.say("Topic $topic already exists with the requested $partitions partitions, reusing it") + } + } + } + } + + /** + * Produces the backlog every arm then replays. + * + * Pre-produced rather than produced alongside the arms, which is what makes the workload + * closed-loop - and in turn why no arm reports latency. A per-record timing here would be + * flattered by however far an arm had fallen behind, so throughput is the only honest number + * this shape can produce. + */ + fun seed(topic: String, from: Int, to: Int) { + if (to <= from) { + return + } + val config = Properties().apply { + put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrap) + put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, ByteArraySerializer::class.java.name) + put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer::class.java.name) + put(ProducerConfig.LINGER_MS_CONFIG, "20") + } + + Console.say("Producing records $from to $to...") + // flush() does not throw for a send that failed and a discarded Future swallows the reason, + // so without this the demo would report a full backlog, run every arm against a short one, + // and print numbers for a workload that never existed. + val firstFailure = AtomicReference() + KafkaProducer(config).use { producer -> + for (i in from until to) { + val key = "key-${i % KEY_SPACE}".toByteArray() + val value = "record-$i".toByteArray() + producer.send(ProducerRecord(topic, key, value)) { _, exception -> + if (exception != null) { + firstFailure.compareAndSet(null, exception) + } + } + } + producer.flush() + } + firstFailure.get()?.let { throw IllegalStateException("the demo could not seed its backlog", it) } + Console.say("Produced ${to - from} records") + } + + /** + * The Kafka properties every arm needs to reach this broker. + * + * `enable.auto.commit` is false because Parallel Consumer owns offset commits. The sidecar + * forces it itself whatever this map says, so on the sidecar arm the line is belt and braces; + * it is the AK core arm - an ordinary consumer with Kafka's own default of true - that would + * otherwise commit behind the demo's back and make a re-run read a partial backlog. + */ + fun consumerProperties(groupId: String): Map = linkedMapOf( + "bootstrap.servers" to bootstrap, + "group.id" to groupId, + "auto.offset.reset" to "earliest", + "enable.auto.commit" to "false", + ) + + override fun close() { + container?.stop() + } + + private fun adminProperties() = Properties().apply { + put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrap) + } + + private fun partitionsOf(admin: Admin, topic: String): Int = + try { + admin.describeTopics(listOf(topic)).allTopicNames().get()[topic]!!.partitions().size + } catch (e: ExecutionException) { + throw IllegalStateException("could not describe the existing topic $topic", e.cause) + } + + companion object { + + /** + * The key space the seeded records spread over. Every arm runs UNORDERED, so this changes + * nothing today; it exists so a KEY-ordered lane added later has more than one key to shard + * across rather than needing the seeding rewritten first. + */ + private const val KEY_SPACE = 1_000 + + private const val FALLBACK_IMAGE = "confluentinc/cp-kafka:7.9.0" + + /** Uses the supplied broker, or starts one when none was supplied. */ + fun resolve(supplied: String?): DemoBroker { + val address = supplied?.trim()?.takeIf { it.isNotEmpty() } + if (address != null) { + // deliberately not echoed: own-cluster mode puts a real address here + Console.say("Using the broker supplied by the caller.") + return DemoBroker(address, null) + } + val image = brokerImage() + Console.say("No broker supplied, starting one in a container: $image") + val started = KafkaContainer(DockerImageName.parse(image)) + .withEnv("KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR", "1") + .withEnv("KAFKA_TRANSACTION_STATE_LOG_MIN_ISR", "1") + .withEnv("KAFKA_TRANSACTION_STATE_LOG_NUM_PARTITIONS", "1") + // the demo forms one consumer group per arm per replay, and the default + // three-second settling delay would be charged to every one of them + .withEnv("KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS", "500") + started.start() + return DemoBroker(started.bootstrapServers, started) + } + + /** + * The broker image, tracking the Kafka client this build carries: CP major = AK major + 4 + * (AK 3.9 -> CP 7.9), CP minor tracks AK minor. The compose file beside this demo pins the + * same image as a literal, because a compose file cannot derive anything - keep them in + * step. + */ + private fun brokerImage(): String = + try { + val parts = AppInfoParser.getVersion().split("-")[0].split(".") + "confluentinc/cp-kafka:${parts[0].toInt() + 4}.${parts[1]}.0" + } catch (e: RuntimeException) { + Console.say( + "Could not read the Kafka version from '${AppInfoParser.getVersion()}' ($e), " + + "falling back to $FALLBACK_IMAGE" + ) + FALLBACK_IMAGE + } + } +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/kotlin/bz/stub/parallelconsumer/client/coroutines/demo/DemoOptions.kt b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/kotlin/bz/stub/parallelconsumer/client/coroutines/demo/DemoOptions.kt new file mode 100644 index 0000000000..97af873c9d --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/kotlin/bz/stub/parallelconsumer/client/coroutines/demo/DemoOptions.kt @@ -0,0 +1,137 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +package bz.stub.parallelconsumer.client.coroutines.demo + +/** + * The demo's dials, and the interface every per-language demo mirrors. + * + * The surface - the seven flags, their defaults, their `PC_DEMO_*` environment variables and the + * precedence between them - is the shared contract in `parallel-consumer-proxy/demo/README.md`. + * Nothing here is a Kotlin choice: flags beat the environment beats the defaults, because a + * container passes configuration by environment and a person at a terminal passes flags, and each + * has to be able to override the other. + * + * **R39 does not govern a demo.** R39 constrains how configuration reaches the *proxy*; a demo is + * an application, so `--records` is not a breach of it. Said here because without it the flags read + * as a violation of the plan's own rule. + */ +internal class DemoOptions private constructor( + val records: Int, + val delayMs: Int, + val maxConcurrency: Int, + val partitions: Int, + val replayFactor: Int, + /** An address the caller supplied; `null` means "start a broker". */ + val bootstrap: String?, + /** A topic the caller supplied; `null` means the demo names its own. */ + val topic: String?, +) { + + /** The records the big replay consumes in total, the small replay's own included. */ + val bigReplayRecords: Int get() = records * maxOf(1, replayFactor) + + /** Whether the big replay is worth running; a factor of 1 or less skips it. */ + val bigReplayWanted: Boolean get() = replayFactor > 1 + + /** + * The effective configuration, printed before the run. + * + * A number without its settings is not reproducible, so this is part of the contract rather + * than a debugging aid. **The bootstrap address is deliberately absent**: own-cluster mode puts + * a user's real broker there, and nothing in this demo logs or echoes it. + */ + override fun toString(): String = + "records = $records" + + "\n delayMs = $delayMs" + + "\n maxConcurrency = $maxConcurrency" + + "\n partitions = $partitions" + + "\n replayFactor = $replayFactor" + + companion object { + + /** The prefix on every environment variable this demo reads, so a reader greps one string. */ + const val ENV_PREFIX: String = "PC_DEMO_" + + /** + * Whether the caller asked for the usage text rather than a run. + * + * Handled here and not only in `run.sh`, because the script is not the only way in: + * `docker compose run demo --help` reaches this main directly, and answering that with + * "unknown option: --help" would be a poor first impression. + */ + fun isHelpRequested(args: Array): Boolean = args.any { it == "-h" || it == "--help" } + + /** + * Parses the command line, falling back to the environment and then to the defaults. + * + * @param args the process arguments, which may legitimately be empty - that is the + * double-click case, and it must work + * @param env the environment to read, passed in rather than read from [System] so this is + * testable without mutating the process environment + * @throws IllegalArgumentException on an unknown flag, a missing value, or a value that is + * not a number in range - a demo that silently ignores a misspelled flag reports numbers + * for settings nobody asked for + */ + @Suppress("CyclomaticComplexMethod") // one branch per flag; splitting it would only hide the table + fun parse(args: Array, env: Map): DemoOptions { + var records = envInt(env, "RECORDS", DEFAULT_RECORDS, atLeast = 1) + var delayMs = envInt(env, "DELAY_MS", DEFAULT_DELAY_MS, atLeast = 0) + var maxConcurrency = envInt(env, "CONCURRENCY", DEFAULT_CONCURRENCY, atLeast = 1) + var partitions = envInt(env, "PARTITIONS", DEFAULT_PARTITIONS, atLeast = 1) + // 0 and 1 both mean "skip the big replay", so this one is allowed to be zero + var replayFactor = envInt(env, "REPLAY_FACTOR", DEFAULT_REPLAY_FACTOR, atLeast = 0) + var bootstrap = envText(env, "BOOTSTRAP") + var topic = envText(env, "TOPIC") + + var i = 0 + while (i < args.size) { + val flag = args[i] + when (flag) { + "--records" -> records = number(flag, value(args, ++i, flag), atLeast = 1) + "--delay-ms" -> delayMs = number(flag, value(args, ++i, flag), atLeast = 0) + "--concurrency" -> maxConcurrency = number(flag, value(args, ++i, flag), atLeast = 1) + "--partitions" -> partitions = number(flag, value(args, ++i, flag), atLeast = 1) + "--replay-factor" -> replayFactor = number(flag, value(args, ++i, flag), atLeast = 0) + "--bootstrap" -> bootstrap = value(args, ++i, flag) + "--topic" -> topic = value(args, ++i, flag) + else -> throw IllegalArgumentException("unknown option: $flag") + } + i++ + } + + // Checked as a Long rather than trusted as an Int later: records * replayFactor + // overflows silently, and a wrapped value turns the big replay into a tiny one that + // still prints a confident throughput figure. + val bigReplay = records.toLong() * maxOf(1, replayFactor) + require(bigReplay <= Int.MAX_VALUE) { + "--records times --replay-factor is $bigReplay, which is more records than the demo can " + + "count; lower one of them" + } + return DemoOptions(records, delayMs, maxConcurrency, partitions, replayFactor, bootstrap, topic) + } + + private const val DEFAULT_RECORDS = 2_000 + private const val DEFAULT_DELAY_MS = 2 + private const val DEFAULT_CONCURRENCY = 100 + private const val DEFAULT_PARTITIONS = 10 + private const val DEFAULT_REPLAY_FACTOR = 20 + + private fun value(args: Array, index: Int, flag: String): String { + require(index < args.size) { "$flag needs a value" } + return args[index] + } + + private fun envText(env: Map, suffix: String): String? = + env[ENV_PREFIX + suffix]?.trim()?.takeIf { it.isNotEmpty() } + + private fun envInt(env: Map, suffix: String, fallback: Int, atLeast: Int): Int = + envText(env, suffix)?.let { number(ENV_PREFIX + suffix, it, atLeast) } ?: fallback + + private fun number(name: String, raw: String, atLeast: Int): Int { + val parsed = raw.trim().toIntOrNull() + ?: throw IllegalArgumentException("$name needs a whole number, got '$raw'") + require(parsed >= atLeast) { "$name must be at least $atLeast, got $parsed" } + return parsed + } + } +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/kotlin/bz/stub/parallelconsumer/client/coroutines/demo/KotlinDemo.kt b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/kotlin/bz/stub/parallelconsumer/client/coroutines/demo/KotlinDemo.kt new file mode 100644 index 0000000000..e06fb810c9 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/kotlin/bz/stub/parallelconsumer/client/coroutines/demo/KotlinDemo.kt @@ -0,0 +1,392 @@ +// Copyright (C) 2026 Antony Stubbs and contributors + +package bz.stub.parallelconsumer.client.coroutines.demo + +import bz.stub.parallelconsumer.client.coroutines.ClientOptions +import bz.stub.parallelconsumer.client.coroutines.Outcome +import bz.stub.parallelconsumer.client.coroutines.ParallelConsumerClient +import bz.stub.parallelconsumer.client.coroutines.ProcessingOrder +import bz.stub.parallelconsumer.client.coroutines.SidecarCommand +import kotlinx.coroutines.CompletableDeferred +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.coroutineScope +import kotlinx.coroutines.delay +import kotlinx.coroutines.launch +import kotlinx.coroutines.runBlocking +import kotlinx.coroutines.withTimeout +import kotlinx.coroutines.withTimeoutOrNull +import org.apache.kafka.clients.consumer.ConsumerConfig +import org.apache.kafka.clients.consumer.KafkaConsumer +import org.apache.kafka.common.serialization.ByteArrayDeserializer +import java.nio.file.Path +import java.time.Duration +import java.util.Locale +import java.util.Properties +import java.util.concurrent.ConcurrentHashMap +import java.util.concurrent.atomic.AtomicInteger +import kotlin.system.exitProcess +import kotlin.time.Duration.Companion.minutes +import kotlin.time.Duration.Companion.seconds + +/** No arm may take longer than this before the demo calls it stalled rather than slow. */ +private val ARM_BUDGET = 10.minutes + +/** + * The serial arm's label: the role, and the client that actually produced the number. + * + * "AK core" alone is a CATEGORY, and the shared contract forbids leaving it at that - a reader + * cannot judge a comparison without knowing what ran it, and the answer differs in every language + * (`franz-go` in Go, `rdkafka` in Ruby). Kotlin has no Kafka client of its own: this arm is Apache + * Kafka's own `KafkaConsumer`, driven from Kotlin, which is why the label names it rather than + * inventing a Kotlin-sounding one. `demo/README.md` carries the "is there a second serious client?" + * answer the contract also asks for. + */ +private const val AK_CORE = "AK core (KafkaConsumer)" + +/** The sidecar arm's label: the role, and what drives it - this module's own client library. */ +private const val PC_KOTLIN_GRPC = "pc-kotlin-grpc (this client)" + +/** The real sidecar's entry point, launched as an ordinary child process by the client library. */ +private const val SIDECAR_MAIN = "bz.stub.parallelconsumer.proxy.Main" + +/** Stands in for a keyless record, so one still counts as a distinct key rather than vanishing. */ +private const val NO_KEY = "" + +/** + * The banner every language prints, differing only in its own name. + * + * Contract, not decoration: the demo's job is to be watched, and a first line reading + * `pc-kotlin-grpc: the proxy granted 100 executor threads` tells a reader nothing about what they + * are looking at. The product is named before anything else, including the effective configuration. + */ +private val BANNER = """ + ================================================================ + PARALLEL CONSUMER - Kotlin demo + The same records, twice: one at a time, then all at once. + ================================================================ +""".trimIndent() + +/** + * What one arm achieved. + * + * [processed] and [keys] are the contract's two DETERMINISTIC figures: every language running the + * same records reports the same pair, which is what lets `bin/ci-demo-conformance.sh` compare + * languages at all - elapsed and msg/s never can be. [processed] must equal the target, because a + * short arm is a failed arm rather than a fast one; [keys] shows the backlog was really spread + * across keys rather than one key repeated. + */ +internal class ArmResult(val arm: String, val elapsed: Duration, val processed: Int, val keys: Int) { + /** Throughput, which is the only rate figure this demo reports. */ + val ratePerSecond: Double + get() = elapsed.toNanos().let { if (it > 0) processed * NANOS_PER_SECOND / it else 0.0 } + + private companion object { + const val NANOS_PER_SECOND = 1_000_000_000.0 + } +} + +/** The key as counted: the bytes Kafka held, or the keyless sentinel. This demo never deserializes. */ +private fun keyOf(key: ByteArray?): String = key?.toString(Charsets.UTF_8) ?: NO_KEY + +/** + * The demo's own output, on stdout and unconditional. + * + * Deliberately not the logger: the tables and the fingerprint are the demo's product, and a reader + * who changed a log level, or whose container inherited one, must still see them. Everything the + * libraries have to say goes through slf4j and is turned down by `logback.xml` beside this source. + */ +internal object Console { + fun say(line: String) { + println(line) + } +} + +public fun main(args: Array) { + // FIRST, before the mode, the broker, the configuration or a single log line: what is this? + Console.say(BANNER) + if (DemoOptions.isHelpRequested(args)) { + usage() + return + } + val options = try { + DemoOptions.parse(args, System.getenv()) + } catch (e: IllegalArgumentException) { + Console.say(e.message ?: "bad arguments") + usage() + // a misspelled flag must not be reported as a result for settings nobody asked for + exitProcess(2) + } + + DemoBroker.resolve(options.bootstrap).use { broker -> + val topic = options.topic ?: "pc-demo-${System.nanoTime()}" + runBlocking { KotlinDemo(options, broker, topic).run() } + } +} + +private fun usage() { + Console.say( + """ + usage: demo/run.sh [options] + --records N records in the comparison replay (default 2000) + --delay-ms N simulated work per record, ms (default 2) + --concurrency N max in-flight records (default 100) + --partitions N partitions on the demo topic (default 10) + --replay-factor N big replay = records x N; 1 skips (default 20) + --bootstrap ADDR an existing broker; omit to start one + --topic NAME an existing topic; omit to create one + + Every flag has an environment variable: --delay-ms is PC_DEMO_DELAY_MS. + Flags beat the environment beats the defaults. + """.trimIndent() + ) +} + +/** + * **The Kotlin demo.** The same records through Kotlin's own Kafka client and through Kotlin over + * the sidecar, one command, no setup. + * + * The contract it keeps - the flags, the environment variables, the defaults, the fingerprint, the + * two tables, no latency - is `parallel-consumer-proxy/demo/README.md`, and it is the same contract + * in every language. What is specific to Kotlin is in `demo/README.md` beside this file. + * + * ## Two arms, and only two + * + * - **AK core (KafkaConsumer)** - Apache Kafka's own `KafkaConsumer` driven from Kotlin, one record + * at a time. Always spelled "AK core", never bare "core", which reads as `parallel-consumer-core` + * (`CONCEPTS.md`) - and never "AK core" alone in the table, because that is a category rather than + * a client and a reader cannot judge a comparison without knowing what ran it. + * - **pc-kotlin-grpc (this client)** - this module's own `ParallelConsumerClient`: it spawns the sidecar as a + * child process, receives records over a socket, runs a suspending function on them and reports + * outcomes back. **On this path the application does no Kafka I/O** - the sidecar owns the + * consumer, the producer, the group membership and the offsets. That is a claim about the *path*, + * not about this process: the same JVM creates the topic, seeds the backlog and runs the AK core + * arm with ordinary Kafka clients, because a comparison needs both sides. + * + * The Java demo carries four more arms. They are Java's alone: one JVM can hold the engine, the + * client library in process and the raw wire at once, so each *pair* changes exactly one term. + * Kotlin has no second Kafka client to compare a wrapper against, so two arms is the whole demo - + * and the sidecar arm here goes through the client library rather than the protocol, because an + * earlier Java demo spoke the wire by hand and proved the engine worked while saying nothing about + * the artifact users actually touch. + * + * ## The user function suspends; it does not block + * + * The simulated work is `delay`, not `Thread.sleep`, and that is the one place this demo departs + * from the letter of the shared contract - `demo/README.md` states the reasoning and the + * measurement that settled it. + * + * [run] returns the results rather than only printing them, so that a caller - a test, a future CI + * entry-point check - can assert what the arms actually did, driving the same code a reader runs + * rather than a parallel path that could pass while the real one is broken. + */ +internal class KotlinDemo( + private val options: DemoOptions, + private val broker: DemoBroker, + private val topic: String, +) { + + suspend fun run(): List { + Console.say("\nEffective configuration:\n $options\n topic = $topic") + + broker.ensureTopic(topic, options.partitions) + broker.seed(topic, 0, options.records) + + val small = listOf(akCore(options.records), kotlinSidecar(options.records)) + report( + "Small replay - every arm over the same ${options.records} records (the comparison)", + small, baselineOf(small), acrossReplays = false, + ) + + if (!options.bigReplayWanted) { + Console.say("\nBig replay skipped (--replay-factor ${options.replayFactor}).") + return small + } + + val total = options.bigReplayRecords + broker.seed(topic, options.records, total) + + // AK core is excluded: it does not go parallel, so it would need total * delayMs + // milliseconds to finish a backlog the sidecar arm clears in seconds, and making a reader + // wait that long to learn nothing new is not worth the wall clock. + val big = listOf(kotlinSidecar(total)) + // the unit is chosen so the figure is never zero - see the demo contract. + val serialMillis = total.toLong() * options.delayMs + val serialCost = + if (serialMillis >= MILLIS_PER_SECOND) "${serialMillis / MILLIS_PER_SECOND}s" + else "${serialMillis}ms" + report( + "Big replay - $total records, parallel arms only (AK core is serial and would take " + + "$serialCost+)", + big, baselineOf(small), acrossReplays = true, + ) + + return small + big + } + + /** + * The serial arm: Kotlin's own Kafka client, one record at a time, the same wait. + * + * `Thread.sleep` here rather than `delay`, and it is not an inconsistency with the other arm: + * this loop has no other work to interleave, so blocking and suspending cost the same wall + * clock. The distinction only bites where records run concurrently, which is exactly where the + * sidecar arm uses `delay`. + */ + private fun akCore(target: Int): ArmResult { + Console.say("\n=== $AK_CORE starting over $target records ===") + val config = Properties().apply { + putAll(broker.consumerProperties(groupId("ak-core"))) + put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer::class.java.name) + put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ByteArrayDeserializer::class.java.name) + } + + var processed = 0 + // A plain HashSet: this arm is single-threaded by definition, and the sidecar arm - which + // is not - uses a concurrent one. + val keys = HashSet() + KafkaConsumer(config).use { consumer -> + consumer.subscribe(listOf(topic)) + // The clock starts AFTER the consumer is built and stops before it closes, because this + // arm is the denominator of every ratio in both tables and the other arm charges itself + // for neither construction nor teardown. + val startedAt = System.nanoTime() + val deadline = startedAt + ARM_BUDGET.inWholeNanoseconds + while (processed < target) { + // The arm that waits on no latch still needs the budget, or a backlog shorter than + // the target spins here forever with no output. + check(System.nanoTime() <= deadline) { "$AK_CORE stalled at $processed of $target" } + for (record in consumer.poll(Duration.ofMillis(POLL_MILLIS))) { + Thread.sleep(options.delayMs.toLong()) + keys.add(keyOf(record.key())) + processed++ + } + } + return finished(AK_CORE, startedAt, processed, keys.size) + } + } + + /** + * The client library over a real sidecar - the arm the whole design exists for, and the one an + * application would actually write. + * + * The library spawns the sidecar itself ([SidecarCommand]); nothing here installs, deploys or + * operates a process. The clock starts after the spawn and the handshake, for the same reason + * the AK core arm's starts after its consumer is built: neither arm charges itself for start-up. + */ + private suspend fun kotlinSidecar(target: Int): ArmResult { + Console.say("\n=== $PC_KOTLIN_GRPC starting over $target records ===") + val processed = AtomicInteger() + // Concurrent, because this arm runs maxConcurrency records at once; a HashSet here would + // silently lose keys and under-report the very figure that proves the backlog was spread. + val keys = ConcurrentHashMap.newKeySet() + val done = CompletableDeferred() + + val client = ParallelConsumerClient.open( + options = ClientOptions( + topics = listOf(topic), + kafkaProperties = broker.consumerProperties(groupId("pc-kotlin-grpc")), + maxConcurrency = options.maxConcurrency, + ordering = ProcessingOrder.UNORDERED, + ), + sidecar = sidecarCommand(), + ) + + return client.use { + val startedAt = System.nanoTime() + coroutineScope { + val poller = launch(Dispatchers.IO) { + client.poll { record -> + // The simulated work: a NON-OCCUPYING wait, because this arm runs + // maxConcurrency records at once and a blocking sleep would cap it at the + // dispatcher's thread count while the fingerprint still printed the number + // the reader asked for. demo/README.md carries the measurement. + delay(options.delayMs.toLong()) + keys.add(keyOf(record.key)) + if (processed.incrementAndGet() >= target) { + done.complete(Unit) + } + Outcome.Success() + } + } + withTimeoutOrNull(ARM_BUDGET) { done.await() } + ?: error("$PC_KOTLIN_GRPC stalled at ${processed.get()} of $target") + val result = finished(PC_KOTLIN_GRPC, startedAt, processed.get(), keys.size) + // The clock has already stopped, so the teardown - drain, close, reap the child - + // is charged to no arm. `use` closes again afterwards; close is idempotent. + client.close() + withTimeout(SHUTDOWN_BUDGET) { poller.join() } + result + } + } + } + + /** + * The sidecar the client library is told to spawn. + * + * The "binary" for a JVM sidecar is a JVM plus a classpath, and the classpath handed to the + * child is **this process's own** - which is why `demo/run.sh` forks a real JVM rather than + * running the demo inside Maven's. Under `mvn exec:java` this property would be Maven's + * classpath and the child would come up without the engine on it. + */ + private fun sidecarCommand(): SidecarCommand { + val java = Path.of(System.getProperty("java.home"), "bin", "java") + return SidecarCommand(java, listOf("-cp", System.getProperty("java.class.path"), SIDECAR_MAIN)) + } + + private fun finished(arm: String, startedAt: Long, processed: Int, keys: Int): ArmResult { + val elapsed = Duration.ofNanos(System.nanoTime() - startedAt) + Console.say("=== $arm finished: $processed records over $keys keys in ${elapsed.toMillis()}ms ===") + return ArmResult(arm, elapsed, processed, keys) + } + + /** A fresh group per arm per replay, so every arm reads the same records from the beginning. */ + private fun groupId(arm: String) = "pc-demo-$arm-${System.nanoTime()}" + + private companion object { + const val POLL_MILLIS = 500L + const val MILLIS_PER_SECOND = 1000 + val SHUTDOWN_BUDGET = 60.seconds + } +} + +private fun baselineOf(results: List): ArmResult? = results.firstOrNull { it.arm == AK_CORE } + +/** + * The results table. + * + * `records` and `keys` sit beside the arm - what it DID - before the figures that say how fast it + * did it. They are there because throughput alone cannot show the work happened: a short arm is a + * failed arm rather than a fast one, and a single key repeated is not a spread backlog. They are + * also the only two figures in this table that another language can be held to, which is what + * `bin/ci-demo-conformance.sh` compares. + */ +private fun report(title: String, results: List, baseline: ArmResult?, acrossReplays: Boolean) { + val table = StringBuilder("\n\n").append(title).append('\n') + table.append( + String.format( + Locale.ROOT, " %-30s %9s %8s %10s %14s %14s%n", + "arm", "records", "keys", "elapsed", "msg/s", + if (acrossReplays) "vs AK core*" else "vs AK core", + ) + ) + for (result in results) { + val ratio = if (baseline == null || baseline.ratePerSecond == 0.0) { + "-" + } else { + String.format(Locale.ROOT, "%.1fx", result.ratePerSecond / baseline.ratePerSecond) + } + table.append( + String.format( + Locale.ROOT, " %-30s %9s %8s %9.1fs %14s %14s%n", + result.arm, + String.format(Locale.ROOT, "%,d", result.processed), + String.format(Locale.ROOT, "%,d", result.keys), + result.elapsed.toMillis() / 1000.0, + String.format(Locale.ROOT, "%,d", result.ratePerSecond.toInt()), ratio, + ) + ) + } + if (acrossReplays) { + table.append("\n * against the SMALL replay's AK core arm. Across replays, so not like-for-like.\n") + } + Console.say(table.toString()) +} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/resources/pc-kotlin-demo-logback.xml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/resources/pc-kotlin-demo-logback.xml new file mode 100644 index 0000000000..81a140de37 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/demo/src/main/resources/pc-kotlin-demo-logback.xml @@ -0,0 +1,43 @@ + + + + + + %d{HH:mm:ss.SSS} %-5level %logger{28} - %msg%n + + + + + + + + + + diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/pom.xml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/pom.xml index eaa8aebe9b..684efb824c 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/pom.xml +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-kotlin/pom.xml @@ -240,6 +240,158 @@ + + + + kotlin-demo + + + pc.kotlinDemo + + + + + + bz.stub.parallelconsumer + parallel-consumer-proxy-client-java-harness + ${project.version} + test + + + + org.apache.kafka + kafka-clients + ${kafka.version} + test + + + + org.testcontainers + testcontainers + test + + + org.testcontainers + kafka + test + + + + + + + ${project.basedir}/src/test/resources + + + ${project.basedir}/demo/src/main/resources + + + + + org.jetbrains.kotlin + kotlin-maven-plugin + ${kotlin.version} + + + + demo-compile + test-compile + + test-compile + + + + ${project.basedir}/demo/src/main/kotlin + + + + + + + + + + org.apache.maven.plugins + maven-dependency-plugin + + + + + demo-classpath + generate-test-resources + + build-classpath + + + ${project.build.directory}/demo-classpath.txt + test + + + + + + + diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/Makefile b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/Makefile index 179d3b19b5..37f1f859fb 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/Makefile +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/Makefile @@ -10,18 +10,19 @@ BIN := $(VENV)/bin REPO_ROOT := ../.. CLASSPATH_FILE := target/sidecar-classpath.txt -.PHONY: all build compile deps test lint ruff typecheck proto proto-check install clean distclean help +.PHONY: all build compile deps demo-build test lint ruff typecheck proto proto-check install clean distclean help all: build test help: @echo "build deps + compile - what Maven's compile phase runs, and the whole of it" @echo "deps install the package and its dev tools into $(VENV)" + @echo "demo-build build, plus the demo/ extra (confluent-kafka) - what demo/run.sh runs" @echo "compile parse every source file - the syntax check Maven's compile phase must fail on" @echo "test run the suite (spawns the real sidecar, and the test-mode one end to end)" @echo "lint ruff + mypy - the whole static analysis, and exactly what the CI row runs" @echo "ruff the linter alone (pyflakes, bugbear, bandit)" - @echo "typecheck mypy alone: --strict over src, --check-untyped-defs over tests, tools and scripts" + @echo "typecheck mypy alone: --strict over src, --check-untyped-defs over tests, tools, scripts, demo" @echo "proto regenerate the committed stubs from the frozen proxy.proto" @echo "proto-check regenerate and fail if the committed stubs have drifted" @echo "clean remove the build tree, the caches and the cached sidecar classpath - NOT $(VENV)" @@ -57,14 +58,23 @@ deps: $(BIN)/python # interpreter accepts but the declared floor (requires-python = ">=3.10") does not, since it parses # with whatever $(PYTHON) built $(VENV). Both belong to `make lint`. # -# THE SAME FOUR DIRECTORIES AS `typecheck`, for the reason given there, and poc/ is excluded for the +# THE SAME FIVE DIRECTORIES AS `typecheck`, for the reason given there, and poc/ is excluded for the # reason given in [tool.ruff] and [tool.mypy] in pyproject.toml: it is preserved evidence, not # maintained source. Keep the three lists in step. # # Output lands in __pycache__ beside each source, which is already on both clean lists - the `clean` # target below and the maven-clean-plugin filesets in pom.xml. compile: deps - $(BIN)/python -m compileall -q src tests tools scripts + $(BIN)/python -m compileall -q src tests tools scripts demo + +# THE DEMO'S OWN INSTALL, and the only target that installs anything `make build` does not. +# confluent-kafka is the AK core arm's Kafka client and what seeds the backlog; it is not a +# dependency of the library, of its tests, or of the conformance runner, so it lives in its own +# extra and off the CI row's critical path. demo/run.sh calls THIS rather than repeating the pip +# line, so a developer and the demo install identically - the same rule that makes `make` the one +# recipe everywhere else here. +demo-build: build + $(BIN)/pip install --quiet --editable '.[demo]' test: build $(CLASSPATH_FILE) $(BIN)/python -m pytest -q @@ -85,7 +95,7 @@ ruff: build # so a runner that stopped compiling must redden this lane rather than the Java module's. typecheck: build $(BIN)/mypy --strict src - $(BIN)/mypy --check-untyped-defs tests tools scripts + $(BIN)/mypy --check-untyped-defs tests tools scripts demo proto: build $(BIN)/python tools/generate_proto.py @@ -121,7 +131,7 @@ $(CLASSPATH_FILE): # one, change the other. clean: rm -rf target build dist src/*.egg-info .pytest_cache .ruff_cache .mypy_cache - find src tests tools scripts -type d -name '__pycache__' -prune -exec rm -rf {} + + find src tests tools scripts demo -type d -name '__pycache__' -prune -exec rm -rf {} + # The deeper clean, kept because the old `clean` did it and it is genuinely wanted when a # dependency pin changes under an existing venv. Not what Maven runs. diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/README.md b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/README.md index 928c52ff01..b470456469 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/README.md +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/README.md @@ -86,7 +86,7 @@ by argv, environment variable or file. ## Not yet - and what that means Later waves add: the liveness lease and heartbeats, reconnect with a manifest, worker-death -reporting, terminal outcomes to a dead-letter topic, the demo container, PyPI packaging, and the +reporting, terminal outcomes to a dead-letter topic, PyPI packaging, and the conformance scenarios beyond the four the harness serves today. **These are un-negotiated capabilities, not half-built features.** The client declares exactly the one capability it implements (`dispatch`) in its handshake, so the proxy holds it to that rather than to promises; @@ -96,6 +96,7 @@ each wave adds its token alongside its duty. ```bash make build # install into .venv, then parse every source file - what Maven's compile runs +make demo-build # build, plus the demo's extra (confluent-kafka) - what demo/run.sh runs make test # the suite: the handshake against the real sidecar, and the end-to-end test make lint # ruff - the same check CI runs make proto # regenerate the stubs from the frozen proxy.proto @@ -190,6 +191,20 @@ edits: it, the queue is discarded at shutdown and the proxy reclaims those records when the stream ends - sending a message outside the negotiated set would be this client's own violation. +## The demo + +```bash +demo/run.sh # picks native or container for you; needs Docker either way +``` + +The same records through `confluent_kafka.Consumer` one at a time, and through this library over a +real sidecar - two arms, and two tables giving each arm's records, unique keys and throughput, with +no setup. It keeps the cross-language contract in +[`../../parallel-consumer-proxy/demo/README.md`](../../parallel-consumer-proxy/demo/README.md); +[`demo/README.md`](demo/README.md) records what is specific to Python, of which the load-bearing +item is that `--concurrency` defaults to 16 rather than the seed's 100 - here an in-flight record is +a worker **process**. + ## Also here `poc/` is the preserved specification probe: a working client written from the protocol documents diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/Dockerfile b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/Dockerfile new file mode 100644 index 0000000000..b3882f8436 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/Dockerfile @@ -0,0 +1,81 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE DEMO CONTAINER for the Python client (astubbs#242, plan unit U35, governed by KTD40). +# +# It exists so a reader needs Docker and nothing else - no Python, no JDK, no Maven. That is the +# same reason the Java demo's container exists, and R72's "Java is included, not exempted" is the +# statement that no language gets to skip this. +# +# IT CARRIES TWO TOOLCHAINS, WHICH THE SEED'S DOES NOT. The application is Python; the sidecar it +# spawns is a JVM binary built out of this repository's Maven reactor. So the image is a JDK base +# with a Python interpreter added, rather than either alone. When a native sidecar binary exists, +# the JDK half of this file is what disappears - and the demo already reads PC_DEMO_SIDECAR for +# exactly that case (see reference_demo.resolve_sidecar). +# +# THE BUILD CONTEXT IS THE REPOSITORY, AND THAT IS A DELIBERATE DEPARTURE. The C++ Dockerfile +# states the rule - "the module directory is the only ordinary context, so the build never uploads +# other agents' target/ output" - and it is a good rule that this build cannot keep: it builds a +# Maven reactor module, which needs its parents and its siblings. The hazard the rule protects +# against is addressed directly instead, by the repository-root .dockerignore that excludes every +# target/ directory, every .venv, .git and .claude/worktrees. Those two files must be read +# together; if you add a build output directory to this repo, add it there too. +# +# WHY THE DEPENDENCIES ARE BAKED IN RATHER THAN CACHE-MOUNTED, on both sides. A BuildKit cache +# mount on /root/.m2 (or on pip's cache) is the obvious way to keep the downloads out of the image, +# and it is wrong here: a cache mount is not part of the resulting image, so the classpath computed +# below would point at jars the running container does not have, and the venv would be missing the +# wheels it installed. The image is large as a result. That is the trade a RUNTIME image makes and +# a build image does not. +# +# A PLAIN JDK, NOT THE `maven:` IMAGE. The official Maven image sets MAVEN_CONFIG=/root/.m2, and +# the Maven WRAPPER appends $MAVEN_CONFIG to its own command line - so `./mvnw ` inside that +# image dies with `Unknown lifecycle phase "/root/.m2"`. Using the image's own `mvn` would dodge it +# and break the repo's rule that builds go through the wrapper (AGENTS.md, "Build Requirements"). +FROM eclipse-temurin:17-jdk + +# python3-venv is separate from python3 on Debian and Ubuntu, and `python3 -m venv` fails with a +# message about ensurepip without it. The venv brings its own pip, so python3-pip is not needed. +RUN apt-get update \ + && apt-get install --yes --no-install-recommends python3 python3-venv \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /src +COPY . . + +# THE SIDECAR, and its classpath computed at BUILD time into a file. Computing it at run time would +# put Maven in the startup path of a demo whose whole promise is one command. +# +# -DincludeScope=runtime is a bug fix rather than tidiness: the default scope drags in +# parallel-consumer-core's TEST jar, whose logback-test.xml then configures the sidecar's logging +# and prints logback's own status report to stdout, ahead of the `port: ` line the client +# library scans for. +# +# -Dcopyright.skip=true because the header check needs the repository's git history to find its +# fork point, and .dockerignore excludes .git from the context. Skipping it here loses nothing: it +# is a source-tree gate that already runs in every ordinary build and in CI's own copyright lane. +RUN ./mvnw --batch-mode -q -DskipTests -Dcopyright.skip=true -DincludeScope=runtime \ + -pl :parallel-consumer-proxy -am \ + package dependency:build-classpath \ + '-Dmdep.outputFile=${project.build.directory}/proxy-classpath.txt' \ + && mkdir -p /app \ + && cp parallel-consumer-proxy/target/proxy-classpath.txt /app/cp.txt \ + && printf ':/src/parallel-consumer-proxy/target/classes' >> /app/cp.txt + +# THE APPLICATION. A venv rather than the system interpreter, matching what `make build` does on a +# developer's machine, so the demo runs against the same installed package either way. The `demo` +# extra is what adds confluent-kafka: the AK core arm's Kafka client, and what seeds the backlog. +WORKDIR /src/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python +RUN python3 -m venv .venv \ + && .venv/bin/pip install --quiet --upgrade pip \ + && .venv/bin/pip install --quiet --editable '.[demo]' + +# PC_DEMO_SIDECAR_CLASSPATH IS READ FROM THE FILE HERE, not baked in with ENV: a Dockerfile's ENV +# cannot read a file, and the classpath is one. The demo then launches `java` directly with it - +# never through a shell, because a shell wrapper holds the write end of the sidecar's stdin and +# defeats the parent-death watch that stops a leaked JVM holding Kafka group membership. This +# entrypoint shell exec's away before that matters; it is the demo's parent, not the sidecar's. +# +# PC_DEMO_ARGS is unquoted on purpose: it carries the caller's flags through compose as a single +# string and must word-split into separate arguments. +ENTRYPOINT ["/bin/sh", "-c", \ + "export PC_DEMO_SIDECAR_CLASSPATH=\"$(cat /app/cp.txt)\"; exec .venv/bin/python demo/reference_demo.py $PC_DEMO_ARGS"] diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/README.md b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/README.md new file mode 100644 index 0000000000..dfbd40bdf7 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/README.md @@ -0,0 +1,174 @@ + + +# The Python demo + +```bash +# from anywhere in the repo - picks native or container for you +parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/run.sh + +# or, from this directory, the plain container path with nothing else needed +docker compose up +``` + +Needs Docker. Python 3.10+ and a JDK are optional: with both, the demo runs natively and starts its +broker in a container; without either, the demo runs in a container too and the broker is a compose +sibling. It prints the product's banner first, then announces which it chose and why. + +**The contract this keeps - and that every other language's demo keeps - is +[`parallel-consumer-proxy/demo/README.md`](../../../parallel-consumer-proxy/demo/README.md).** +Read that first. This file only records what is specific to Python. + +## The two arms + +| arm | what it is | +|---|---| +| **AK core (confluent-kafka)** | `confluent_kafka.Consumer`, one record at a time, in this process. | +| **pc-python-grpc (this client)** | This module's client library. It spawns the sidecar as a child process, receives records over a socket, runs the user's function in **worker processes**, and reports outcomes back. The application does no Kafka I/O on this path. | + +Both labels name the library that actually ran, which is contract: **"AK core" is a category, not a +client**, and a reader cannot judge a comparison without knowing what produced it. + +Two arms is the whole contract outside Java. The seed carries four more because one JVM can hold +every engine at once; Python has one way to reach the engine, so a wrapper arm and a raw-wire arm +would have nothing to be compared against. In particular **nothing here speaks +the protocol by hand** - the seed's first version did, and it proved the engine worked while saying +nothing about the client library, which is the artifact users actually touch. + +## What is specific to Python + +### Python has more than one serious Kafka client, and this demo runs `confluent-kafka` + +The contract asks a language with several to say so here, because "the choice materially changes +the number, and a reader evaluating *is this fast in my language* is really asking about the client +they already use". Python's three: + +| client | what it is | +|---|---| +| [`confluent-kafka`](https://github.com/confluentinc/confluent-kafka-python) | A binding to **librdkafka**, the C client. What this demo's AK core arm runs, and what seeds the backlog. | +| [`kafka-python`](https://github.com/dpkp/kafka-python) | A pure-Python implementation of the protocol - no C library underneath it. | +| [`aiokafka`](https://github.com/aio-libs/aiokafka) | `asyncio`-native, built on `kafka-python`'s protocol code. | + +Only `confluent-kafka` runs here. **Nothing in this repository has measured the other two**, so +this is a choice on grounds rather than on numbers: a binding to a C client is the denominator +least likely to flatter the sidecar arm, which makes it the conservative one to divide by. It is +also the client the sidecar itself effectively competes with, since the sidecar's consumer is the +Java client rather than a Python one. + +**Running a second AK core arm is a live option** the contract explicitly allows, and this demo has +not taken it. `aiokafka` would be the interesting one - it is the shape an asynchronous Python +application already has - and it would need the demo's serial arm to grow an event loop, which is +why it is recorded as an option rather than done. + +### `--concurrency` defaults to 16, not the seed's 100 + +**This is the only default that differs, and it is not a preference.** The proxy's executor-count +function is `IntUnaryOperator.identity()` today, so a client asking for 100 in-flight records gets +**100 executors** - and in Python an executor is a worker *process*, not a thread. A hundred +interpreters on a laptop measures the machine's memory pressure rather than the engine. + +Sixteen is a number a laptop can hold. It is fixed rather than derived from `os.cpu_count()`, so +that two readers' fingerprints are comparable. `--concurrency 100` still does exactly what it says. + +The formula itself is an open owner decision, not this demo's to make - +[`docs/inflight/blocker-executor-count-formula.md`](../../../docs/inflight/blocker-executor-count-formula.md). + +### The simulated work is `time.sleep`, and that satisfies the contract's predicate + +The contract's rule is now a property of the **client**, not of the language: *is it +thread-per-record?* This one is not - it hands the user's function to a worker **process** - so +Python is one of the six that needs its own non-occupying wait. + +`time.sleep` **is** that wait. It releases the GIL and parks the thread on the kernel's timer, so +it costs no CPU and no lock; the thing the rule exists to rule out is a busy loop +(`while time.monotonic() < deadline: pass`), which would pin a core per in-flight record. No other +primitive is available that would occupy less: this client hands a worker one record and takes one +outcome back, so an event loop inside the worker cannot overlap a second record, and +`asyncio.run(asyncio.sleep(d))` per record would hold the process for exactly as long plus a loop +set-up. + +**What a Python wait occupies is a whole worker process** - and here that is not a +misreport, which is the distinction the contract's predicate is drawing. The rule's hazard is a +table "reporting the runtime's ceiling while appearing to report the engine's". Python's ceiling +*is* the concurrency the fingerprint printed, because the proxy's executor count is the worker +count, so a sleeping worker per in-flight record is exactly the number on the page. The cost lands +on how expensive that number is to buy - which is why this demo's divergence is the default +`--concurrency` above, and not the wait primitive. + +`simulate_work` in [`reference_demo.py`](reference_demo.py) carries the same reasoning at the point +of use. An earlier version of the contract named languages rather than stating this predicate, and +listed Python for a reason that did not survive; that argument is settled and no longer tracked in +[`docs/inflight/clients/python.md`](../../../docs/inflight/clients/python.md). + +### The `keys` column is counted in shared memory, because a `set` cannot cross a fork + +Every language's table reports **records processed** and **unique keys seen**, and both are +contract because they are deterministic - the same records give the same two figures in any +language, which is what makes them comparable when elapsed and msg/s never are. + +In Python the second one is awkward for the same reason everything else here is: the user's +function runs in a worker process, so a `set` closed over by the processor would be duplicated by +the fork and each worker would report only what it saw. `KeyTally` in +[`reference_demo.py`](reference_demo.py) uses a shared **byte per key slot** instead - one +unsynchronised write, since the only value ever written is 1 - and both arms use it, so the two +rows are counted the same way. A `multiprocessing.Manager` dictionary would have counted exactly +and for any key at all; it was rejected because it puts an IPC round trip on the critical path of +the arm the demo exists to time. + +The price is that a key this demo did not seed has no slot. That only arises if `--topic` names a +topic already holding somebody else's records, and it is never silent: those records are counted +separately and the arm prints that its keys figure is an undercount. + +### The clock starts at the first record, in both arms + +The seed starts its clock just before consumption, having already built its client and spawned its +sidecar outside the window - "no other arm charges itself for client construction or teardown". +Python cannot draw the line in the same place: `ParallelConsumerClient.poll()` forks the worker +pool, spawns the sidecar, completes the handshake and starts consumption in one call. + +So this demo keeps the seed's **rule** rather than its line number: start-up is outside the window +for both arms, at the first record either of them sees. Including a JVM boot in a throughput figure +would report roughly a quarter of the arm's real rate. + +### The demo never starts a broker itself + +The seed uses Testcontainers. The equivalent here would put a Docker client library into the demo of +a Kafka client library, so `run.sh` brings up the same compose broker its container path uses and +hands the demo an address. From the reader's side the contract's promise is unchanged - omit +`--bootstrap` and a broker appears - and the rule that matters is untouched: **the demo container is +never granted the host Docker socket**, so inside a container the address is a compose sibling +either way. + +`reference_demo.py` run by hand with no `--bootstrap` says so and exits 2 rather than guessing. + +For the native path the broker publishes a host listener on `127.0.0.1:19095` +(`PC_DEMO_BROKER_PORT` moves it, which matters if you have another language's demo up). `run.sh` +stops the broker on the way out; `PC_DEMO_KEEP_BROKER=1` leaves it running, which is worth about ten +seconds a run while developing. + +### The sidecar is a JVM, so the native path needs a JDK + +The client library wants an absolute path to a binary. Until a native sidecar exists that binary is +`java`, and the classpath is an argument about the *binary* rather than configuration - bootstrap +servers, credentials, ordering and concurrency still travel only in the connect-time handshake +(R39). `run.sh` builds the classpath and exports `PC_DEMO_SIDECAR_CLASSPATH`; the container bakes it +in at image build time. `PC_DEMO_SIDECAR` names a binary directly, and is the shape this takes the +day a native sidecar lands. + +The sidecar is **not a compose service**, deliberately: the client library spawns and supervises it, +so the user never installs, deploys or operates a process (KTD41). A compose service would show a +deployment the product does not ask for. + +## The files + +| file | what it is | +|---|---| +| [`run.sh`](run.sh) | The entry point. Same flags as the seed, plus `--docker` / `--native`. | +| [`reference_demo.py`](reference_demo.py) | The arms, the timing and the two tables. | +| [`demo_options.py`](demo_options.py) | The flags, the environment variables and the precedence between them. | +| [`demo_kafka.py`](demo_kafka.py) | The topic, the seeded backlog, and the properties both arms' consumers use. | +| [`Dockerfile`](Dockerfile) | A JDK base with a Python interpreter added - two toolchains, because the application is Python and the sidecar is a JVM. | +| [`docker-compose.yml`](docker-compose.yml) | The demo and its broker sibling. Also the broker the native path uses. | + +`make demo-build` is what installs the demo's one extra dependency, `confluent-kafka`. It is +deliberately not in the `dev` extra: the library never imports it, and neither the test suite nor +the conformance runner needs it. diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/demo_kafka.py b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/demo_kafka.py new file mode 100644 index 0000000000..e25eb7d9b1 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/demo_kafka.py @@ -0,0 +1,191 @@ +# Copyright (C) 2026 Antony Stubbs and contributors + +"""The broker the demo talks to, and the backlog every arm replays. + +This is Python's counterpart to the Java demo's ``DemoBroker``, minus one thing it deliberately +does not do: **it never starts a broker.** The seed uses Testcontainers, which has no equivalent +here that would not put Docker's API into a client library's demo, so ``demo/run.sh`` starts the +compose broker and hands this an address. From the reader's side the promise is unchanged - run +``demo/run.sh`` with no arguments and a broker appears - and it keeps the rule that matters: a +demo container is never given the host Docker socket, so inside the container the address is a +compose sibling either way. + +Everything here uses `confluent-kafka` (librdkafka), which is Python's own Apache Kafka client. +It is the AK core arm's engine, and it is also what creates the topic and produces the backlog - +exactly as the seed's single JVM does, and for the same reason: a comparison needs both sides. +The sidecar arm's *path* still carries no Kafka I/O in this process, which is the claim the arm +actually makes. +""" + +from __future__ import annotations + +import logging +from typing import Any + +from confluent_kafka import Producer +from confluent_kafka.admin import AdminClient, NewTopic + +__all__ = ["KEY_SPACE", "consumer_properties", "ensure_topic", "key_of", "key_slot", "seed"] + +log = logging.getLogger(__name__) + +KEY_SPACE = 1_000 +"""How many distinct keys the backlog uses, matching the seed so the workloads are comparable. + +It is load-bearing for a key-ordered engine and merely descriptive for an unordered one; this +demo runs unordered, so it is here for parity rather than for effect. +""" + +_KEY_PREFIX = "key-" +"""The backlog's key format, written once because two places now depend on it. + +:func:`seed` produces it and :func:`key_slot` reads it back, and the reason the second exists is +the contract's **unique keys** column: the sidecar arm's user function runs in worker *processes*, +so "the set of keys this arm saw" cannot be a Python ``set`` in any one of them. A key that maps +back to an ordinal maps to a slot in a shared array instead - one byte, no lock, no IPC. +""" + + +_TOPIC_CREATION_TIMEOUT = 30.0 +"""Long enough for a broker that has just booted to elect leaders for every partition.""" + + +def key_of(ordinal: int) -> bytes: + """The key the backlog's ``ordinal``-th record carries.""" + return f"{_KEY_PREFIX}{ordinal % KEY_SPACE}".encode() + + +def key_slot(key: bytes | None) -> int | None: + """Which of :data:`KEY_SPACE` slots ``key`` occupies, or ``None`` if it is not ours. + + ``None`` means a record this demo did not seed - a pre-existing topic named with ``--topic`` + holding somebody else's records. The demo counts those separately and says so rather than + guessing at a slot, because a hashed slot would collide and quietly *under*-report the one + figure the contract calls deterministic. + """ + if key is None: + return None + try: + text = key.decode() + except UnicodeDecodeError: + return None + if not text.startswith(_KEY_PREFIX): + return None + try: + ordinal = int(text[len(_KEY_PREFIX):]) + except ValueError: + return None + return ordinal if 0 <= ordinal < KEY_SPACE else None + + +def ensure_topic(bootstrap: str, topic: str, partitions: int) -> None: + """Creates the demo's topic, tolerating one that a previous run already left behind. + + Reusing a topic silently is fine; reusing one with a **different partition count** is not, + because the effective-configuration block would print a ``--partitions`` value that never + applied - and that block is the demo's whole reproducibility promise. + """ + admin = AdminClient({"bootstrap.servers": bootstrap}) + wanted = NewTopic(topic, num_partitions=partitions, replication_factor=1) + created = admin.create_topics([wanted]) + try: + created[topic].result(timeout=_TOPIC_CREATION_TIMEOUT) + log.info("Created topic %s with %d partitions", topic, partitions) + except Exception as failure: + # librdkafka reports this as a KafkaException wrapping TOPIC_ALREADY_EXISTS. Matching on + # the error CODE rather than the message text, because the text is librdkafka's to change. + if not _is_topic_exists(failure): + raise RuntimeError(f"could not create the demo topic {topic}: {failure}") from failure + existing = _partitions_of(admin, topic) + if existing != partitions: + raise RuntimeError( + f"topic {topic} already exists with {existing} partitions, but this run asked " + f"for {partitions} - pass --topic to name a fresh one, or --partitions {existing}" + ) from failure + log.info("Topic %s already exists with the requested %d partitions, reusing it", + topic, partitions) + + +def seed(bootstrap: str, topic: str, first: int, last: int) -> None: + """Produces the backlog every arm then replays, for record ordinals ``[first, last)``. + + **Pre-produced rather than produced alongside the arms**, and that is what makes the workload + closed-loop - which is in turn why no arm reports latency. A per-record timing here would be + flattered by however far an arm had fallen behind, so throughput is the only honest number + this shape can produce. + + A failed send is fatal. ``flush()`` does not raise for a send that failed, so without the + delivery callback below the demo would report a full backlog, run every arm against a short + one, and print numbers for a workload that never existed. + """ + if last <= first: + return + + failures: list[Any] = [] + + def delivered(error: Any, _message: Any) -> None: + if error is not None: + failures.append(error) + + log.info("Producing records %d to %d...", first, last) + producer = Producer({ + "bootstrap.servers": bootstrap, + "linger.ms": 20, + # librdkafka buffers in the producer's own memory; the default 100k message ceiling is + # below the big replay's backlog, and hitting it raises BufferError per send rather than + # blocking. Polling below keeps the queue draining, and this keeps the headroom honest. + "queue.buffering.max.messages": 1_000_000, + }) + for ordinal in range(first, last): + producer.produce( + topic, + key=key_of(ordinal), + value=f"record-{ordinal}".encode(), + on_delivery=delivered, + ) + # Serve delivery callbacks as we go: librdkafka queues them, and a callback backlog is + # memory this process holds until the flush at the end. + if ordinal % 10_000 == 0: + producer.poll(0) + producer.flush() + if failures: + raise RuntimeError(f"the demo could not seed its backlog: {failures[0]}") + log.info("Produced %d records", last - first) + + +def consumer_properties(bootstrap: str, group_id: str) -> dict[str, str]: + """The Kafka properties every arm's consumer needs to reach this broker. + + ``enable.auto.commit`` is set false explicitly. The sidecar forces it false itself + (``KafkaClientFactory`` sets it "whatever the map says"), because Parallel Consumer owns offset + commits and core refuses a consumer that commits underneath it - but stating it here keeps the + two arms configured from one place, and means the AK core arm below is not quietly running + with auto-commit while the sidecar arm is not. + + These are **Java** Kafka client property names, because the sidecar hands them to a + ``KafkaConsumer``. That they are also librdkafka's spellings for the four settings used here is + a convenience, not a rule: `confluent-kafka` and the Java client do not agree on every key. + """ + return { + "bootstrap.servers": bootstrap, + "group.id": group_id, + "auto.offset.reset": "earliest", + "enable.auto.commit": "false", + } + + +def _is_topic_exists(failure: Exception) -> bool: + args = getattr(failure, "args", ()) + code = getattr(args[0], "code", None) if args else None + if code is None: + return False + from confluent_kafka import KafkaError + return bool(code() == KafkaError.TOPIC_ALREADY_EXISTS) + + +def _partitions_of(admin: AdminClient, topic: str) -> int: + metadata = admin.list_topics(topic=topic, timeout=_TOPIC_CREATION_TIMEOUT) + described = metadata.topics.get(topic) + if described is None: + raise RuntimeError(f"could not describe the existing topic {topic}") + return len(described.partitions) diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/demo_options.py b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/demo_options.py new file mode 100644 index 0000000000..766f71d6f2 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/demo_options.py @@ -0,0 +1,223 @@ +# Copyright (C) 2026 Antony Stubbs and contributors + +"""The demo's dials, and Python's copy of the interface every per-language demo mirrors. + +The contract is `parallel-consumer-proxy/demo/README.md`; the seed that defines the spellings is +the Java demo's ``DemoOptions``. Flags beat environment variables beat defaults - the ordinary +convention, chosen because a container passes configuration by environment while a person at a +terminal passes flags, and both must be able to override the other's layer. + +**R39 does not govern a demo.** R39 constrains how configuration reaches the *proxy*; a demo is an +application, so its flags are not a violation of it. Nothing here reaches the sidecar by argv or +environment - the client library still carries every setting in the connect-time handshake. + +**One default diverges from the seed, deliberately: ``--concurrency``.** In Java 100 in-flight +records are 100 sleeping threads. Here they are 100 worker *processes*, because the proxy's +executor-count function is `IntUnaryOperator.identity()` today +(`docs/inflight/blocker-executor-count-formula.md` - an open owner decision, not this demo's to +make). A demo that spawned 100 interpreters on a laptop would be measuring the machine's swap +behaviour, so this one asks for :data:`DEFAULT_CONCURRENCY`. Every other default is the seed's. +""" + +from __future__ import annotations + +import argparse +import dataclasses +import os +from collections.abc import Mapping + +__all__ = ["ENV_PREFIX", "USAGE", "DemoOptions", "parse"] + +ENV_PREFIX = "PC_DEMO_" +"""Prefix for every environment variable this demo reads, so a reader can grep one string.""" + +DEFAULT_RECORDS = 2000 +DEFAULT_DELAY_MS = 2 +DEFAULT_CONCURRENCY = 16 +"""The one default that is not the seed's. See this module's docstring for why. + +Sixteen rather than a hundred, and rather than a count derived from ``os.cpu_count()``: a demo +whose headline setting changes with the machine cannot be compared between two readers' runs, and +the fingerprint would be the only place the difference showed. A fixed number is reproducible, and +the flag is there for a reader with a bigger machine. +""" +DEFAULT_PARTITIONS = 10 +DEFAULT_REPLAY_FACTOR = 20 + +USAGE = f"""usage: demo/run.sh [options] + + --records N records in the comparison replay (default {DEFAULT_RECORDS}) + --delay-ms N simulated work per record, ms (default {DEFAULT_DELAY_MS}) + --concurrency N max in-flight records (default {DEFAULT_CONCURRENCY}, not 100: + in Python that many in-flight records is that many worker PROCESSES) + --partitions N partitions on the demo topic (default {DEFAULT_PARTITIONS}) + --replay-factor N big replay = records x N; 1 skips (default {DEFAULT_REPLAY_FACTOR}) + --bootstrap ADDR an existing broker; omit and run.sh starts one + --topic NAME an existing topic; omit to create one + +Every flag has an environment variable: --delay-ms is PC_DEMO_DELAY_MS. +Flags beat the environment beats the defaults.""" + + +@dataclasses.dataclass(frozen=True) +class DemoOptions: + """The effective configuration, after flags, environment and defaults have been resolved.""" + + records: int = DEFAULT_RECORDS + delay_ms: int = DEFAULT_DELAY_MS + max_concurrency: int = DEFAULT_CONCURRENCY + partitions: int = DEFAULT_PARTITIONS + replay_factor: int = DEFAULT_REPLAY_FACTOR + bootstrap: str | None = None + topic: str | None = None + + @property + def big_replay_records(self) -> int: + """The records the big replay consumes in total, including the small replay's own.""" + return self.records * max(1, self.replay_factor) + + @property + def big_replay_wanted(self) -> bool: + """True when the big replay is worth running at all; a factor of 1 or less skips it.""" + return self.replay_factor > 1 + + @property + def delay_seconds(self) -> float: + return self.delay_ms / 1000.0 + + def fingerprint(self, topic: str) -> str: + """The effective configuration, for printing before the run. + + A number without its settings is not reproducible, so this is part of the contract every + language's demo keeps rather than a debugging aid. **The bootstrap address is deliberately + absent**: own-cluster mode puts a user's real broker address here, and the + credential-hygiene rule that binds the proxy binds a demo too. The keys are the seed's + spellings, camel case included, so that two languages' fingerprints diff cleanly. + """ + return ( + f"records = {self.records}" + f"\n delayMs = {self.delay_ms}" + f"\n maxConcurrency = {self.max_concurrency}" + f"\n partitions = {self.partitions}" + f"\n replayFactor = {self.replay_factor}" + f"\n topic = {topic}" + ) + + +class OptionError(Exception): + """A flag, or an environment variable, that this demo cannot act on. + + Raised rather than tolerated: a demo that silently ignores a misspelled flag reports numbers + for settings the user did not ask for. + """ + + +def is_help_requested(argv: list[str]) -> bool: + """Whether the caller asked for the usage text rather than a run. + + Handled here rather than only in ``run.sh``, because the script is not the only way in: + ``docker compose run demo --help`` reaches ``reference_demo`` directly, and answering that + with "unrecognized arguments" would be a poor first impression of the demo ten languages copy. + """ + return any(argument in ("-h", "--help") for argument in argv) + + +def parse(argv: list[str], env: Mapping[str, str] | None = None) -> DemoOptions: + """Parses the demo's command line, falling back to the environment and then to the defaults. + + :param argv: the process arguments, which may legitimately be empty - that is the double-click + case, and it must work. + :param env: the environment to read, passed in rather than read from :mod:`os` so this is + testable without mutating the interpreter's own environment. + :raises OptionError: on an unknown flag, a missing value, or a value that is not a number in + range. + """ + environment = os.environ if env is None else env + + # From the environment first, so a flag of the same name overwrites it below. argparse's own + # defaults are therefore the environment's values rather than the constants. + defaults = DemoOptions( + records=_positive("PC_DEMO_RECORDS", _from_env(environment, "RECORDS"), DEFAULT_RECORDS), + delay_ms=_non_negative( + "PC_DEMO_DELAY_MS", _from_env(environment, "DELAY_MS"), DEFAULT_DELAY_MS), + max_concurrency=_positive( + "PC_DEMO_CONCURRENCY", _from_env(environment, "CONCURRENCY"), DEFAULT_CONCURRENCY), + partitions=_positive( + "PC_DEMO_PARTITIONS", _from_env(environment, "PARTITIONS"), DEFAULT_PARTITIONS), + # 1 or less skips the big replay, so this one is allowed to be zero. + replay_factor=_non_negative( + "PC_DEMO_REPLAY_FACTOR", _from_env(environment, "REPLAY_FACTOR"), + DEFAULT_REPLAY_FACTOR), + bootstrap=_from_env(environment, "BOOTSTRAP"), + topic=_from_env(environment, "TOPIC"), + ) + + # add_help=False and exit_on_error are both deliberate: argparse's own --help text would not be + # the contract's, and its default behaviour on a bad flag is to print to stderr and call + # sys.exit - which a caller that wants to report the error itself cannot intercept. + parser = argparse.ArgumentParser(prog="demo/run.sh", add_help=False, exit_on_error=False) + parser.add_argument("--records", type=str) + parser.add_argument("--delay-ms", dest="delay_ms", type=str) + parser.add_argument("--concurrency", dest="max_concurrency", type=str) + parser.add_argument("--partitions", type=str) + parser.add_argument("--replay-factor", dest="replay_factor", type=str) + parser.add_argument("--bootstrap", type=str) + parser.add_argument("--topic", type=str) + + try: + parsed, extra = parser.parse_known_args(argv) + except argparse.ArgumentError as bad: + raise OptionError(str(bad)) from bad + if extra: + raise OptionError(f"unknown option: {extra[0]}") + + options = DemoOptions( + records=_positive("--records", parsed.records, defaults.records), + delay_ms=_non_negative("--delay-ms", parsed.delay_ms, defaults.delay_ms), + max_concurrency=_positive( + "--concurrency", parsed.max_concurrency, defaults.max_concurrency), + partitions=_positive("--partitions", parsed.partitions, defaults.partitions), + replay_factor=_non_negative( + "--replay-factor", parsed.replay_factor, defaults.replay_factor), + bootstrap=parsed.bootstrap if parsed.bootstrap else defaults.bootstrap, + topic=parsed.topic if parsed.topic else defaults.topic, + ) + + # Checked here rather than trusted later. Python integers do not overflow, so the hazard the + # seed guards against cannot arise - but a big replay of a hundred million records is still a + # demo that never finishes, and saying so now beats saying nothing for an hour. + if options.big_replay_records > 100_000_000: + raise OptionError( + f"--records times --replay-factor is {options.big_replay_records}, which is more " + "records than this demo will finish in a sane wall clock; lower one of them") + return options + + +def _from_env(env: Mapping[str, str], suffix: str) -> str | None: + raw = env.get(ENV_PREFIX + suffix) + return None if raw is None or not raw.strip() else raw.strip() + + +def _positive(name: str, raw: str | None, fallback: int) -> int: + if raw is None: + return fallback + parsed = _number(name, raw) + if parsed < 1: + raise OptionError(f"{name} must be at least 1, got {parsed}") + return parsed + + +def _non_negative(name: str, raw: str | None, fallback: int) -> int: + if raw is None: + return fallback + parsed = _number(name, raw) + if parsed < 0: + raise OptionError(f"{name} must not be negative, got {parsed}") + return parsed + + +def _number(name: str, raw: str) -> int: + try: + return int(raw.strip()) + except ValueError as bad: + raise OptionError(f"{name} needs a whole number, got {raw!r}") from bad diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/docker-compose.yml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/docker-compose.yml new file mode 100644 index 0000000000..be565d5de1 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/docker-compose.yml @@ -0,0 +1,124 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE ONE-COMMAND PATH for the Python demo (astubbs#242, plan unit U35, R20). +# +# THE BROKER IS A SIBLING, AND THAT IS A RULE RATHER THAN A LAYOUT CHOICE. Starting a broker from +# inside a container requires mounting the host Docker socket, and U35 forbids it: a documented +# socket mount is root-equivalent host access taught as the normal way to run the product. So the +# demo container is given an ADDRESS instead of a socket, and reaches a compose sibling on the +# demo's own network. +# +# THE SIDECAR IS NOT A SERVICE HERE, deliberately. It would be easy to add one, and it would teach +# the wrong thing: KTD41's product model is that the client library spawns and supervises the +# sidecar itself, so the user never installs, deploys or operates a process. A compose service +# would show a deployment the product does not ask for. The Python client spawns it as a child +# process inside its own container, exactly as it does natively. +# +# THIS FILE SERVES BOTH PATHS, WHICH THE SEED'S DOES NOT. demo/run.sh --native also brings up the +# `broker` service here rather than starting a Testcontainers broker of its own, because the +# alternative is putting a Docker client library into the demo of a Kafka client library. That is +# why the broker publishes a HOST listener as well as the internal one - the native demo runs +# outside this network. +# +# TWO WAYS IN, AND BOTH ARE SUPPORTED: +# +# cd && docker compose up # the plain one, defaults, nothing else needed +# demo/run.sh --docker [--records N ...] # the same thing with the demo's flags +# +# `docker compose up` on its own is the path to keep working: it is what a reader who has never seen +# this repository will type. It needs no arguments, no environment and no wrapper script - the demo +# service runs with the demo's own defaults, and PC_DEMO_ARGS below is empty unless run.sh sets it. +# Add --build after changing any source, and `docker compose down` to remove the broker afterwards. +# +# The demo container exits when the run finishes; the broker keeps running until you stop it. run.sh +# passes --abort-on-container-exit --exit-code-from demo so that a scripted run stops on its own and +# reports the demo's exit code rather than compose's. + +name: pc-python-demo + +services: + broker: + # Pinned as a literal because a compose file cannot derive anything. It must track + # in the root pom the same way the Java demo's DemoBroker#brokerImage does at + # run time - the mapping is CP major = AK major + 4, so AK 3.9.x means CP 7.9. + image: confluentinc/cp-kafka:7.9.0 + ports: + # PUBLISHED AND INTERNAL MUST BE THE SAME NUMBER. An advertised listener carries a port as + # well as a host, so a remapped published port would advertise one the host cannot reach and + # the native demo's first metadata refresh would send it somewhere with nothing listening. + - "${PC_DEMO_BROKER_PORT:-19095}:${PC_DEMO_BROKER_PORT:-19095}" + environment: + CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk + # QUIET. A demo is a thing someone WATCHES, and a KRaft broker at INFO emits hundreds of + # lines of its own configuration, controller elections and log-segment chatter - which buries + # the two tables the demo exists to show. Kafka's own warnings and errors still come through. + KAFKA_LOG4J_ROOT_LOGLEVEL: WARN + KAFKA_TOOLS_LOG4J_LOGLEVEL: WARN + # ROOT ALONE IS NOT ENOUGH, and this was measured rather than assumed: the cp-kafka image + # writes kafka=INFO, kafka.controller=TRACE and state.change.logger=TRACE into its own + # log4j config, and a named logger beats the root. With only the root set, one demo run + # interleaved 927 broker lines with 35 of the demo's own. These are the named loggers that + # actually produce the noise. + KAFKA_LOG4J_LOGGERS: "kafka=WARN,kafka.controller=WARN,kafka.server=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.network=WARN,kafka.request.logger=WARN,state.change.logger=WARN,org.apache.kafka=WARN" + KAFKA_NODE_ID: 1 + KAFKA_PROCESS_ROLES: broker,controller + # THREE LISTENERS, and the third is what the native path needs. PLAINTEXT is bound to the + # service name rather than 0.0.0.0 because this image formats its KRaft storage with the same + # config it boots with, and the format step refuses a non-routable advertised address. HOST is + # bound to 0.0.0.0 - it must accept a connection arriving through Docker's published port - + # and ADVERTISED as 127.0.0.1, which is where a process on the host actually reaches it. + KAFKA_LISTENERS: PLAINTEXT://broker:9092,HOST://0.0.0.0:${PC_DEMO_BROKER_PORT:-19095},CONTROLLER://broker:9093 + KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,HOST://127.0.0.1:${PC_DEMO_BROKER_PORT:-19095} + KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,HOST:PLAINTEXT + KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093 + KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 + KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 + # The demo forms one consumer group per arm, one after another, and the default three-second + # settling delay would be charged to every one of them. + KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 500 + healthcheck: + # The demo must not start before the broker can serve an API request. Listing topics is the + # cheapest call that proves that, and it proves more than a port being open does. + test: ["CMD", "kafka-topics", "--bootstrap-server", "broker:9092", "--list"] + interval: 5s + timeout: 10s + retries: 30 + start_period: 10s + + demo: + build: + # The repository, because the sidecar is built from the Maven reactor and a reactor module + # cannot build without its parents and siblings. The root .dockerignore is what keeps this + # from uploading every target/ directory in the tree; the Dockerfile explains the pair. + context: ../../.. + dockerfile: parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/Dockerfile + # A demo that has printed its tables is done. Without this, compose's default would restart a + # container that exited cleanly and the reader would watch the whole run happen again. + restart: "no" + depends_on: + broker: + condition: service_healthy + environment: + # Supplying this is what stops the demo asking to be run through run.sh: unlike the seed, this + # demo never starts a broker itself, in a container or out of one. + PC_DEMO_BOOTSTRAP: broker:9092 + # The flags demo/run.sh was called with, forwarded as one string. + PC_DEMO_ARGS: ${PC_DEMO_ARGS:-} + # EVERY OTHER DIAL, forwarded explicitly. Compose passes nothing it is not told to, so a + # missing line here is a dial the container silently ignores while the run still prints a + # fingerprint claiming otherwise. Caught across seven languages at once by + # bin/ci-demo-conformance.sh, which is why that check exists. Adding a flag means adding it + # here too. + PC_DEMO_RECORDS: ${PC_DEMO_RECORDS:-} + PC_DEMO_DELAY_MS: ${PC_DEMO_DELAY_MS:-} + PC_DEMO_CONCURRENCY: ${PC_DEMO_CONCURRENCY:-} + PC_DEMO_PARTITIONS: ${PC_DEMO_PARTITIONS:-} + PC_DEMO_REPLAY_FACTOR: ${PC_DEMO_REPLAY_FACTOR:-} + PC_DEMO_TOPIC: ${PC_DEMO_TOPIC:-} + # NOT A DIAL - a statement that run.sh already printed the banner, so the demo does not print + # a second one. Empty when you run `docker compose up` yourself, which is when the demo's own + # banner is the only one there is. + PC_DEMO_BANNER_PRINTED: ${PC_DEMO_BANNER_PRINTED:-} diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/reference_demo.py b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/reference_demo.py new file mode 100644 index 0000000000..8053d9dda3 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/reference_demo.py @@ -0,0 +1,519 @@ +# Copyright (C) 2026 Antony Stubbs and contributors + +"""**The Python demo**: the same records through Python's own Kafka client, and through Parallel +Consumer reached over the sidecar. + +The contract it keeps is `parallel-consumer-proxy/demo/README.md` - same flags, same environment +variables, same defaults but one, same two tables, the effective configuration printed first, and +no latency. What is specific to Python is in `demo/README.md` beside this file. + +## The two arms, and why there are exactly two + +* **AK core (confluent-kafka)** - `confluent_kafka.Consumer`, one record at a time, in this + process. Always spelled "AK core", never bare "core", which reads as `parallel-consumer-core` + (`CONCEPTS.md`) - and always with the client that actually ran, because "AK core" is a category + and Python has more than one client in it. +* **pc-python-grpc (this client)** - this repository's Python client library, which spawns the sidecar + as a child process, receives records over a socket, runs the user's function in **worker + processes**, and reports outcomes back. **The application does no Kafka I/O on this path**: the + sidecar owns the consumer, the producer, the group membership and the offsets. + +The seed carries four more arms because one JVM can hold every engine at once. Python cannot, and +a language whose only Kafka client is its own has nothing to compare a wrapper or a raw wire +against - so **two arms is the whole contract here**, and adding a hand-rolled gRPC arm would only +re-measure the engine while saying nothing about the client library, which is the artifact users +actually touch. + +## What the clock covers, and why it is not the seed's window exactly + +Both arms are timed **from the first record reaching the user's function to the last one being +counted**. The seed starts its clock a line earlier, just before consumption begins, having already +built its client and spawned its sidecar outside the window - "no other arm charges itself for +client construction or teardown", as its AK core arm puts it. + +Python cannot draw the line in the same place: :meth:`ParallelConsumerClient.poll` forks the worker +pool, spawns the sidecar, completes the handshake and starts consumption, and it is one call. So +this demo keeps the seed's *rule* rather than its line number - start-up is outside the window for +both arms, measured identically, at the first record either of them sees. Including a JVM boot in a +throughput figure would have reported roughly a quarter of the arm's real rate, which is a worse +answer than a documented window. + +## Run it + + parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/run.sh +""" + +from __future__ import annotations + +import dataclasses +import logging +import multiprocessing +import os +import pathlib +import re +import shutil +import sys +import time +from multiprocessing.context import ForkContext, SpawnContext +from typing import Any + +# The demo's own modules sit beside this file rather than on the installed package's path: the demo +# is not part of the shipped library, and a reader should be able to see all of it in one directory. +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) + +import demo_kafka +import demo_options +from confluent_kafka import Consumer, KafkaError, KafkaException + +from parallel_consumer import ( + ClientOptions, + ParallelConsumerClient, + ProcessingOrder, + SidecarCommand, +) + +log = logging.getLogger("demo") + +BANNER = """\ +================================================================ + PARALLEL CONSUMER - Python demo + The same records, twice: one at a time, then all at once. +================================================================""" +"""The first thing this demo prints, and the same in every language bar its own name. + +Contract, not decoration. The first line used to be an arm announcing its executor count, which +tells a reader nothing about what they are watching - so the shape is fixed in +``parallel-consumer-proxy/demo/README.md`` and every language prints it identically. + +``PC_DEMO_BANNER_PRINTED`` suppresses it, and is a **statement of fact rather than a preference**: +``demo/run.sh`` prints the banner before its own setup lines, so that the product's name is the +first thing on screen even when a Maven build and a broker start-up come between that and the run. +It then sets this, because the same banner twice in one screen is worse than either alone. Nothing +else sets it, so ``docker compose up`` and ``python demo/reference_demo.py`` both still print it. +""" + +AK_CORE = "AK core (confluent-kafka)" +"""**"AK core" is a category; ``confluent-kafka`` is the client that actually ran.** + +A reader cannot judge a comparison without knowing what produced it, and the answer differs in +every language - `rdkafka` in Ruby, `franz-go` in Go, `kafkajs` in TypeScript. Python has more +than one serious Kafka client, which is exactly why naming this one matters; ``demo/README.md`` +says which and why this demo runs ``confluent-kafka``. +""" + +PYTHON_GRPC = "pc-python-grpc (this client)" +"""The sidecar arm, labelled with what drives it: this repository's own Python client library.""" + +SIDECAR_MAIN = "bz.stub.parallelconsumer.proxy.Main" +"""The sidecar's entry point. It is a JVM today; :func:`resolve_sidecar` is where that stops +mattering the moment a native binary exists.""" + +ARM_BUDGET_SECONDS = 600.0 +"""No arm may take longer than this before the demo calls it stalled rather than slow.""" + + +@dataclasses.dataclass(frozen=True) +class ArmResult: + """One arm's finished run: what it was, what it did, and how fast it did it. + + ``processed`` and ``keys`` are the contract's **deterministic** pair - every language running + the same records reports the same two figures, which is what makes them comparable when + elapsed and msg/s never can be. They also demonstrate the run rather than asserting it: a + short arm is a failed arm, not a fast one, and a single key repeated would mean the backlog + was never spread at all. + """ + + arm: str + elapsed_seconds: float + processed: int + keys: int + + @property + def rate_per_second(self) -> float: + return 0.0 if self.elapsed_seconds <= 0 else self.processed / self.elapsed_seconds + + +class KeyTally: + """The distinct keys an arm saw, countable from **worker processes**. + + This is Python's problem alone, and it is the fork that causes it: the sidecar arm's user + function runs in a worker process, so an ordinary ``set`` closed over by the processor would be + duplicated by the fork and every worker would report only its own keys. A + :class:`multiprocessing.Manager` dictionary would count exactly and generally, and was + rejected: it puts an IPC round trip on the critical path of the very arm the demo exists to + time, and almost every record in the small replay carries a key no worker has seen. + + So the tally is a shared **byte per key slot** - one unsynchronised write, because the only + value ever written is 1 and two workers writing it to the same slot cannot disagree. Both arms + use it, so the two rows are counted the same way rather than one exactly and one approximately. + + ``foreign`` counts records this demo did not seed. It should always be zero; if it is not, the + keys column is an undercount and the demo says so rather than printing a number it cannot + stand behind. + """ + + def __init__(self, context: ForkContext | SpawnContext) -> None: + self._seen = context.Array("b", demo_kafka.KEY_SPACE, lock=False) + self._foreign = context.Value("i", 0) + + def observe(self, key: bytes | None) -> None: + slot = demo_kafka.key_slot(key) + if slot is None: + with self._foreign.get_lock(): + self._foreign.value += 1 + return + self._seen[slot] = 1 + + @property + def distinct(self) -> int: + return sum(self._seen) + + @property + def foreign(self) -> int: + return int(self._foreign.value) + + +def main(argv: list[str]) -> int: + logging.basicConfig(level=logging.INFO, format="%(message)s", stream=sys.stdout) + # BEFORE ANYTHING ELSE, including the usage text and any complaint about a bad flag: a reader + # must be told what they are looking at before they are told anything else about it. + if not os.environ.get("PC_DEMO_BANNER_PRINTED"): + print(BANNER) + + if demo_options.is_help_requested(argv): + print(demo_options.USAGE) + return 0 + try: + options = demo_options.parse(argv) + except demo_options.OptionError as bad: + # A misspelled flag must not be reported as a result for settings nobody asked for. + print(f"{bad}\n\n{demo_options.USAGE}", file=sys.stderr) + return 2 + + if not options.bootstrap: + print( + "This demo needs a broker address and cannot start one itself.\n" + "Run it through demo/run.sh, which starts a broker for you, or pass --bootstrap " + "ADDR / set PC_DEMO_BOOTSTRAP.", + file=sys.stderr) + return 2 + + try: + sidecar = resolve_sidecar() + except FileNotFoundError as missing: + print(str(missing), file=sys.stderr) + return 2 + + topic = options.topic or f"pc-demo-{time.time_ns()}" + run(options, sidecar, topic) + return 0 + + +def run(options: demo_options.DemoOptions, sidecar: SidecarCommand, topic: str) -> list[ArmResult]: + """Runs the whole demo and hands back every arm's result. + + Returns the results rather than only printing them, so a test can assert what the arms + actually did against the same code path a reader runs. + """ + assert options.bootstrap is not None # noqa: S101 - main() refuses to get here without one + bootstrap = options.bootstrap + + print(f"\nEffective configuration:\n {options.fingerprint(topic)}") + + demo_kafka.ensure_topic(bootstrap, topic, options.partitions) + demo_kafka.seed(bootstrap, topic, 0, options.records) + + small = [ + ak_core_arm(options, topic, options.records), + sidecar_arm(options, sidecar, topic, options.records), + ] + report(f"Small replay - every arm over the same {options.records} records (the comparison)", + small, baseline_of(small), across_replays=False) + + if not options.big_replay_wanted: + print(f"\nBig replay skipped (--replay-factor {options.replay_factor}).") + return small + + total = options.big_replay_records + demo_kafka.seed(bootstrap, topic, options.records, total) + + # AK core is excluded here because it does not go parallel: it would need + # total * delay_ms milliseconds to finish a backlog the sidecar arm clears in seconds, and a + # demo that makes a reader wait that long to learn nothing new is not worth the wall clock. + serial_estimate = duration(total * options.delay_ms) + big = [sidecar_arm(options, sidecar, topic, total)] + report(f"Big replay - {total} records, parallel arms only (AK core is serial and would take " + f"{serial_estimate}+)", big, baseline_of(small), across_replays=True) + + return small + big + + +def ak_core_arm(options: demo_options.DemoOptions, topic: str, target: int) -> ArmResult: + """Python's own Kafka client, one record at a time. The arm every language has. + + Serial by construction: `confluent_kafka.Consumer` hands back one message per poll and the + user's work happens on this thread before the next one is asked for. That is the shape a + Python application without Parallel Consumer actually has, which is what makes it the + denominator worth measuring against. + """ + print(f"\n=== {AK_CORE} starting over {target} records ===") + config: dict[str, Any] = dict( + demo_kafka.consumer_properties(options.bootstrap or "", group_id(AK_CORE))) + consumer = Consumer(config) + # The same tally the sidecar arm uses, in a process that could have used a plain set. Two + # counting methods would make the two rows of the "keys" column mean subtly different things. + keys = KeyTally(worker_context()) + processed = 0 + started_at: float | None = None + try: + consumer.subscribe([topic]) + deadline = time.monotonic() + ARM_BUDGET_SECONDS + while processed < target: + # The one arm that does not wait on an event still needs the budget, or a backlog + # shorter than the target spins here forever with no output. + if time.monotonic() > deadline: + raise RuntimeError(f"{AK_CORE} stalled at {processed} of {target}") + message = consumer.poll(0.5) + if message is None: + continue + error = message.error() + if error is not None: + if error.code() == KafkaError._PARTITION_EOF: + continue + raise KafkaException(error) + if started_at is None: + # The clock starts at the FIRST RECORD, not at subscribe: the group join and the + # first rebalance are start-up, and the sidecar arm does not charge itself for + # theirs either. See this module's docstring. + started_at = time.monotonic() + keys.observe(message.key()) + simulate_work(options.delay_seconds) + processed += 1 + elapsed = time.monotonic() - (started_at or time.monotonic()) + finally: + consumer.close() + return finished(AK_CORE, elapsed, processed, keys) + + +def sidecar_arm(options: demo_options.DemoOptions, sidecar: SidecarCommand, topic: str, + target: int) -> ArmResult: + """The client library over a real sidecar - the arm the whole design exists for. + + On this path the application does no Kafka I/O: it spawns a binary, receives records over a + socket, runs its own function on them in **worker processes**, and reports outcomes back, + while the sidecar owns the consumer, the producer, the group membership and the offsets. That + is a claim about the *path*, not about this process - the same interpreter seeded the topic and + runs the AK core arm with an ordinary Kafka client, because a comparison needs both sides. A + genuinely foreign application carries no Kafka client library at all, which is the property + this arm stands in for. + + **Nothing here speaks the protocol by hand**, deliberately. An earlier version of the seed did + exactly that, and it proved the engine worked while saying nothing about the client library. + """ + print(f"\n=== {PYTHON_GRPC} starting over {target} records " + f"({options.max_concurrency} worker processes) ===") + + # THE COUNTERS ARE multiprocessing PRIMITIVES, AND THAT IS NOT A STYLE CHOICE. The user's + # function runs in worker PROCESSES, so a closure over an ordinary int would be duplicated by + # the fork and every worker would count only its own records - the demo would then wait forever + # for a total that no single process ever reaches. They are created here, before the client + # exists, so every worker inherits the same shared memory. + context = worker_context() + counted = context.Value("i", 0) + started_at = context.Value("d", 0.0) + ended_at = context.Value("d", 0.0) + finished_event = context.Event() + keys = KeyTally(context) + delay = options.delay_seconds + + def process(record: Any) -> None: + if started_at.value == 0.0: + with started_at.get_lock(): + if started_at.value == 0.0: + started_at.value = time.monotonic() + keys.observe(record.key) + simulate_work(delay) + with counted.get_lock(): + counted.value += 1 + reached = counted.value >= target + if reached: + ended_at.value = time.monotonic() + finished_event.set() + # Returning None is how an ordinary Python function says "this record succeeded". No + # Outcome is needed unless the function also produces records or fails deliberately. + return None + + client_options = ClientOptions( + topics=[topic], + max_concurrency=options.max_concurrency, + ordering=ProcessingOrder.UNORDERED, + kafka_properties=demo_kafka.consumer_properties( + options.bootstrap or "", group_id(PYTHON_GRPC)), + ) + with ParallelConsumerClient(client_options, sidecar=sidecar) as client: + client.poll(process) + if not finished_event.wait(ARM_BUDGET_SECONDS): + raise RuntimeError(f"{PYTHON_GRPC} stalled at {counted.value} of {target}") + elapsed = ended_at.value - started_at.value + processed = counted.value + + # Reaching the target is not the only thing that can end the wait - a failed session ends it + # too. Without this a broken run prints a plausible row at a plausible rate and exits 0, which + # is the worst thing a demo whose shape ten languages copy can do. + if processed < target: + raise RuntimeError(f"{PYTHON_GRPC} ended early at {processed} of {target}") + return finished(PYTHON_GRPC, elapsed, processed, keys) + + +def simulate_work(seconds: float) -> None: + """The user's function, in both arms: a wait that occupies nothing while it waits. + + **The contract's predicate is whether the client is thread-per-record. This one is not** - it + hands the user's function to a worker process - so Python is one of the six languages that owes + its own non-occupying wait, and :func:`time.sleep` is it: it releases the GIL and parks the + calling thread on the kernel's timer, so the wait costs no CPU and no lock. The busy loop it + rules out (``while time.monotonic() < deadline: pass``) would pin a core per in-flight record. + + What a Python wait *does* occupy is a whole **worker process**, and no wait primitive changes + that. ``asyncio.sleep`` was considered and rejected: the client hands a worker one record and + takes one outcome back, so an event loop inside the worker cannot overlap a second record, and + ``asyncio.run`` per record would hold the process for exactly as long while adding a loop + set-up to every one. + + That occupancy is **not** the misreport the rule is aimed at, which is a table showing the + runtime's ceiling while appearing to show the engine's. Here the ceiling *is* the number the + fingerprint printed, because the proxy's executor count is the worker count - one sleeping + worker per in-flight record, exactly as many as asked for. TypeScript's case is the other one: + one event loop, where a blocking sleep caps in-flight work at one whatever the fingerprint + says. + + So the divergence lands where the cost actually is: ``--concurrency`` defaults to + :data:`demo_options.DEFAULT_CONCURRENCY` rather than the seed's 100, because in this language + that number is a process count. ``docs/inflight/clients/python.md`` carries the reasoning. + """ + if seconds > 0: + time.sleep(seconds) + + +def worker_context() -> ForkContext | SpawnContext: + """The same start method the client's own pool uses, so the counters above are inheritable. + + ``fork`` where the platform has it - that is what lets :func:`sidecar_arm`'s processor be a + closure over the shared counters. Where only ``spawn`` exists the client library says so + itself, in an error naming the picklability requirement, rather than failing obscurely. + """ + if "fork" in multiprocessing.get_all_start_methods(): + return multiprocessing.get_context("fork") + return multiprocessing.get_context("spawn") + + +def resolve_sidecar() -> SidecarCommand: + """Where the sidecar binary is, as an **absolute** path - never a ``PATH`` lookup. + + The client library refuses anything else, and it is right to: this process hands the sidecar + its Kafka credentials, so which binary runs is a security decision that belongs to the + application. ``demo/run.sh`` and the demo container both set these variables; a reader running + ``reference_demo.py`` by hand gets told what to set. + + ``PC_DEMO_SIDECAR`` names a binary directly and is the shape this will take once a native + sidecar exists. Until then ``PC_DEMO_SIDECAR_CLASSPATH`` names the JVM classpath, and the + "binary" is ``java`` plus that classpath - which is arguments about the *binary*, not + configuration: bootstrap servers, credentials, ordering and concurrency still travel only in + the connect-time handshake. + """ + binary = os.environ.get("PC_DEMO_SIDECAR") + if binary: + return SidecarCommand.coerce(binary) + + classpath = os.environ.get("PC_DEMO_SIDECAR_CLASSPATH") + if not classpath: + raise FileNotFoundError( + "no sidecar: set PC_DEMO_SIDECAR to an absolute binary, or " + "PC_DEMO_SIDECAR_CLASSPATH to the proxy's JVM classpath. demo/run.sh builds the " + "second one for you - run that instead.") + return SidecarCommand(executable=pathlib.Path(java_binary()), + args=("-cp", classpath, SIDECAR_MAIN)) + + +def java_binary() -> str: + """The JVM to run the sidecar with. ``JAVA_HOME`` wins, as it does everywhere else here.""" + explicit = os.environ.get("PC_DEMO_JAVA") + if explicit: + return explicit + java_home = os.environ.get("JAVA_HOME") + if java_home: + candidate = pathlib.Path(java_home) / "bin" / "java" + if candidate.is_file(): + return str(candidate) + found = shutil.which("java") + if found is None: + raise FileNotFoundError( + "no java found for the sidecar: set JAVA_HOME, PC_DEMO_JAVA, or put a JDK 17 java on " + "PATH") + return found + + +def duration(milliseconds: int) -> str: + """A wall clock a reader can read, in whichever unit says something. + + Integer seconds are right for the default backlog - 40,000 records at 2ms is ``80s`` - and + wrong for the small volumes CI and the conformance harness run, where the same arithmetic + printed ``0s`` and told a reader that a serial arm would take no time at all. + """ + return f"{milliseconds // 1000}s" if milliseconds >= 1000 else f"{milliseconds}ms" + + +def group_id(arm: str) -> str: + """A fresh group per arm per replay, so every arm reads the same records from the beginning. + + The arm's own label now carries the client's name in brackets, and a group id is not the place + for punctuation - so everything that is not a letter or a digit becomes a dash. + """ + slug = re.sub(r"[^A-Za-z0-9]+", "-", arm).strip("-") + return f"pc-demo-{slug}-{time.time_ns()}" + + +def finished(arm: str, elapsed: float, processed: int, keys: KeyTally) -> ArmResult: + print(f"=== {arm} finished: {processed} records, {keys.distinct} keys " + f"in {int(elapsed * 1000)}ms ===") + if keys.foreign: + # Never silently. The keys column is contract precisely because it is deterministic, and a + # topic that already held somebody else's records makes it an undercount. + print(f" NOTE: {keys.foreign} record(s) carried a key this demo did not seed, so the " + "keys figure below is an undercount. Pass --topic with a fresh name.") + return ArmResult(arm, elapsed, processed, keys.distinct) + + +def baseline_of(results: list[ArmResult]) -> ArmResult | None: + return next((result for result in results if result.arm == AK_CORE), None) + + +def report(title: str, results: list[ArmResult], baseline: ArmResult | None, *, + across_replays: bool) -> None: + """The two tables, in the contract's columns and order, so two languages diff cleanly. + + Six columns: what ran, **what it did**, then how fast. ``records`` and ``keys`` come before the + timings deliberately - throughput alone cannot show the work happened, and a reader scanning + left to right should meet the evidence before the number it justifies. Column *identity and + order* are contract; the widths are not, and this demo's are wider than the seed's because an + arm label now carries its client's name. + """ + heading = "vs AK core*" if across_replays else "vs AK core" + lines = ["", "", title, + f" {'arm':<30} {'records':>9} {'keys':>7} {'elapsed':>10} {'msg/s':>14} " + f"{heading:>14}"] + for result in results: + ratio = ("-" if baseline is None or baseline.rate_per_second == 0 + else f"{result.rate_per_second / baseline.rate_per_second:.1f}x") + lines.append(f" {result.arm:<30} {result.processed:>9,} {result.keys:>7,} " + f"{result.elapsed_seconds:>9.1f}s {int(result.rate_per_second):>14,} " + f"{ratio:>14}") + if across_replays: + lines.append("") + lines.append(" * against the SMALL replay's AK core arm. Across replays, so not " + "like-for-like.") + print("\n".join(lines)) + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/run.sh b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/run.sh new file mode 100755 index 0000000000..6f147cd859 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/demo/run.sh @@ -0,0 +1,195 @@ +#!/usr/bin/env bash +# +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE PYTHON DEMO: the same records through Python's own Kafka client, and through Parallel +# Consumer reached over the sidecar. One command, no setup. +# +# This mirrors the Java seed's run.sh - same flags, same defaults but one, same two tables out. +# The contract is parallel-consumer-proxy/demo/README.md; what is specific to Python is in +# README.md beside this file. +set -euo pipefail + +cd "$(git rev-parse --show-toplevel)" + +MODULE_DIR="parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python" +DEMO_DIR="$MODULE_DIR/demo" +COMPOSE_FILE="$DEMO_DIR/docker-compose.yml" + +# The host port the compose broker publishes for the NATIVE path. The container path never uses it - +# there the demo reaches the broker as a compose sibling on the demo's own network. It is a variable +# because eleven per-language demos will eventually exist and a reader may have two brokers up. +BROKER_PORT="${PC_DEMO_BROKER_PORT:-19095}" + +# THE FIRST THING ON SCREEN NAMES THE PRODUCT. That is contract +# (parallel-consumer-proxy/demo/README.md), and it has to be printed HERE rather than only by the +# demo: a native run builds a classpath and starts a broker first, so a reader would otherwise watch +# a minute of Maven output without being told what any of it is for. reference_demo.py prints the +# same banner when nothing else has - `docker compose up` and running it by hand - and +# PC_DEMO_BANNER_PRINTED below is how this script tells it not to print a second one. +banner() { + cat <<'BANNER' +================================================================ + PARALLEL CONSUMER - Python demo + The same records, twice: one at a time, then all at once. +================================================================ +BANNER +} + +usage() { + cat <<'USAGE' +usage: demo/run.sh [options] + + --records N records in the comparison replay (default 2000) + --delay-ms N simulated work per record, ms (default 2) + --concurrency N max in-flight records (default 16) + --partitions N partitions on the demo topic (default 10) + --replay-factor N big replay = records x N; 1 skips (default 20) + --bootstrap ADDR an existing broker; omit to start one + --topic NAME an existing topic; omit to create one + + --docker run in a container, broker as a compose sibling + --native run on this machine's own Python and JDK toolchains + -h, --help this + +With neither --docker nor --native, the demo picks: native when a Python 3.10+ interpreter and a +JDK toolchain are both present, a container otherwise. It says which it chose, and why, on the +first line of output. + +--concurrency defaults to 16 rather than the seed's 100, and that is the one number this demo +does not inherit: in Python, in-flight records are worker PROCESSES. See demo/README.md. + +Every flag also has an environment variable: --delay-ms is PC_DEMO_DELAY_MS. +USAGE +} + +banner +# Every path below has now printed it exactly once, so the demo itself must not print a second. +export PC_DEMO_BANNER_PRINTED=1 + +# ${args[@]+...} rather than "${args[@]}": under `set -u`, bash 3.2 - which is what macOS ships, +# and what a first-time user runs this with - treats an EMPTY array expansion as an unbound +# variable and aborts. That fails exactly the no-argument case, which is the double-click case. +args=() +mode="auto" +bootstrap_given="${PC_DEMO_BOOTSTRAP:-}" +while [ $# -gt 0 ]; do + case "$1" in + --bootstrap) + if [ $# -lt 2 ]; then echo "$1 needs a value" >&2; exit 2; fi + bootstrap_given="$2"; args+=("$1" "$2"); shift 2 ;; + --records|--delay-ms|--concurrency|--partitions|--replay-factor|--topic) + if [ $# -lt 2 ]; then echo "$1 needs a value" >&2; exit 2; fi + args+=("$1" "$2"); shift 2 ;; + --docker) mode="docker"; shift ;; + --native) mode="native"; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "unknown option: $1" >&2; usage >&2; exit 2 ;; + esac +done + +have_jdk_toolchain() { + # `mvnw -v` rather than `command -v java`: the wrapper needs a JDK it can actually run, and a + # JRE-only or absent JAVA_HOME fails here rather than three minutes into a build. The demo + # needs one because the sidecar is a JVM binary today. + [ -x ./mvnw ] && ./mvnw -v >/dev/null 2>&1 +} + +have_python() { + # The floor is the package's own requires-python. Asked of the interpreter rather than parsed + # out of `python3 --version`, so a 3.10.0rc or a vendor build answers correctly. + command -v python3 >/dev/null 2>&1 \ + && python3 -c 'import sys; sys.exit(0 if sys.version_info >= (3, 10) else 1)' 2>/dev/null +} + +if [ "$mode" = "auto" ]; then + if have_python && have_jdk_toolchain; then + mode="native" + echo "Mode: native - a Python 3.10+ interpreter and a JDK toolchain are both present." + else + mode="docker" + echo "Mode: container - this machine is missing Python 3.10+ or a JDK toolchain, so the" + echo " demo brings both." + fi +else + echo "Mode: $mode - asked for explicitly." +fi + +if ! docker info >/dev/null 2>&1; then + if [ "$mode" = "docker" ]; then + echo "Docker is not running - the demo needs it for the broker and for its own image." >&2 + else + echo "Docker is not running - the demo starts its broker in a container." >&2 + fi + exit 1 +fi + +if [ "$mode" = "docker" ]; then + # The broker is a COMPOSE SIBLING and the demo container is never given the host Docker + # socket: a documented socket mount is root-equivalent host access taught as the normal way to + # run the product (plan unit U35). The SIDECAR is not a service either - the client library + # spawns it as a child process inside the demo's own container, exactly as it does natively. + # PC_DEMO_ARGS carries the flags through to the container, where the same parser reads them. + PC_DEMO_ARGS="${args[*]+${args[*]}}" \ + exec docker compose -f "$COMPOSE_FILE" up \ + --build --abort-on-container-exit --exit-code-from demo +fi + +# --------------------------------------------------------------------------------------------- +# The native path. +# --------------------------------------------------------------------------------------------- + +# THE SIDECAR'S CLASSPATH, BUILT ONCE AND HANDED OVER AS A PATH. The sidecar is a JVM binary today, +# so the "absolute executable" the client library demands is `java`, and the classpath is an +# argument about the BINARY rather than configuration - bootstrap servers, credentials, ordering +# and concurrency still travel only in the connect-time handshake (R39). +# +# -am is not optional: this module's parent is not in a single-module reactor and the enforcer's +# ReactorModuleConvergence rule fails the build before anything runs without it. +# +# -DincludeScope=runtime, and it is a bug fix rather than tidiness: the default scope drags in +# parallel-consumer-core's TEST jar, whose logback-test.xml then configures logging for the sidecar +# and prints logback's own status report to STDOUT - ahead of the `port: ` line the client library +# scans for. It survives only because that scan tolerates preceding lines. +echo "Building the sidecar's classpath..." +./mvnw --batch-mode -q -pl :parallel-consumer-proxy -am -DskipTests \ + -DincludeScope=runtime package dependency:build-classpath \ + '-Dmdep.outputFile=${project.build.directory}/proxy-classpath.txt' + +PC_DEMO_SIDECAR_CLASSPATH="$(cat parallel-consumer-proxy/target/proxy-classpath.txt):$PWD/parallel-consumer-proxy/target/classes" +export PC_DEMO_SIDECAR_CLASSPATH + +# The venv, with the demo's own extra on top of the package's. `make` is the module's one recipe, +# so a developer and this script install identically. +echo "Preparing the Python environment..." +make -C "$MODULE_DIR" demo-build + +# THE BROKER, WHEN THE CALLER DID NOT NAME ONE. The seed starts its own with Testcontainers; this +# demo starts the same compose broker its container path uses, because the alternative is putting +# a Docker client library into a demo of a Kafka client library. From the reader's side the promise +# is the contract's: omit --bootstrap and a broker appears. +if [ -z "$bootstrap_given" ]; then + echo "Starting the broker (compose), publishing 127.0.0.1:$BROKER_PORT..." + PC_DEMO_BROKER_PORT="$BROKER_PORT" docker compose -f "$COMPOSE_FILE" up -d --wait broker + export PC_DEMO_BOOTSTRAP="127.0.0.1:$BROKER_PORT" + + # Stopped on the way out, because Testcontainers stops the seed's broker and a demo should not + # leave a container running that the reader never started. PC_DEMO_KEEP_BROKER=1 keeps it, which + # is worth about ten seconds on every repeat run while developing. + if [ -z "${PC_DEMO_KEEP_BROKER:-}" ]; then + # shellcheck disable=SC2317,SC2329 # invoked by the trap below, not by any call site + stop_broker() { + PC_DEMO_BROKER_PORT="$BROKER_PORT" docker compose -f "$COMPOSE_FILE" down \ + >/dev/null 2>&1 || true + } + trap stop_broker EXIT + fi +fi + +# NOT `exec`, unlike the seed: the trap above has to run, and exec would replace this shell before +# it could. The demo's exit code is carried out by hand instead - `|| status=$?` rather than a bare +# call, so `set -e` does not exit before the trap has stopped the broker - and a scripted caller +# sees exactly what the demo returned. +status=0 +"$MODULE_DIR/.venv/bin/python" "$DEMO_DIR/reference_demo.py" ${args[@]+"${args[@]}"} || status=$? +exit "$status" diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/pom.xml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/pom.xml index 18d2b864c1..773f711a80 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/pom.xml +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/pom.xml @@ -192,6 +192,13 @@ **/__pycache__/** + + ${project.basedir}/demo + + **/__pycache__ + **/__pycache__/** + + diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/pyproject.toml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/pyproject.toml index 9dbd007086..41f96f89ec 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/pyproject.toml +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-python/pyproject.toml @@ -51,6 +51,15 @@ dev = [ "pip-audit>=2.10,<3", ] +# THE DEMO'S OWN DEPENDENCY, and it is deliberately not in `dev`. confluent-kafka (librdkafka) is +# Python's own Apache Kafka client: it IS the demo's AK core arm, and it is what creates the topic +# and produces the backlog. The library never imports it - nothing in src/ holds a broker +# connection, which is the whole point of the sidecar - and neither the test suite nor the +# conformance runner needs it, so putting it in `dev` would add a compiled wheel to the CI row's +# critical path for a directory that row does not run. `make demo-build` installs this extra; +# demo/run.sh and the demo container both go through that. +demo = ["confluent-kafka>=2.3,<3"] + [project.urls] Homepage = "https://github.com/astubbs/parallel-consumer" Issues = "https://github.com/astubbs/parallel-consumer/issues" @@ -126,3 +135,11 @@ follow_imports = "skip" [[tool.mypy.overrides]] module = ["grpc_tools.*"] ignore_missing_imports = true + +# confluent-kafka publishes no type information: it is a C extension with a .py shim and no +# py.typed marker, so mypy cannot see past the import. The DEMO code that calls it is still +# checked - only the library's own shape is unknown - and the demo is not part of the shipped +# surface, which is why this sits under --check-untyped-defs rather than --strict. +[[tool.mypy.overrides]] +module = ["confluent_kafka.*"] +ignore_missing_imports = true diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/.gitignore b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/.gitignore index 4d5a8cb06d..1360235ac6 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/.gitignore +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/.gitignore @@ -12,6 +12,13 @@ # vendor/bundle, which is what the clients CI matrix caches for this row. Without it a CI run # installs to the system gem directory and caches an empty directory. +# THE DEMO'S LOCKFILE IS NOT COMMITTED, and the module's is - which is a difference worth stating. +# The demo's Gemfile pins every version it adds exactly and inherits the module's pins wholesale, so +# a lockfile here would add nothing but a list of the platforms it was last resolved on. Committing +# one would make a native run on any other platform rewrite it, so `git status` would be dirty after +# running the demo - which is a worse thing than resolving the same pinned versions again. +demo/Gemfile.lock + vendor/ .yardoc/ doc/ diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/.rubocop.yml b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/.rubocop.yml index c3d95d7647..83997663c2 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/.rubocop.yml +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/.rubocop.yml @@ -30,6 +30,10 @@ AllCops: # Generated by protoc from the FROZEN schema - never hand-edited, so never linted. - "lib/parallelconsumer/**/*" - "vendor/**/*" + # The demo has a bundle of its own, and its own vendor/ is as much third-party source as the + # module's. The demo's OWN Ruby is deliberately not excluded: it ships in this module, so the + # bug finder covers it. + - "demo/vendor/**/*" # ── The bug-finding departments, at full strength ──────────────────────────── # Lint/ and Security/ are enabled by default and are deliberately left untouched: an exclusion diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/README.md b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/README.md index de8fa00b9f..d4fcabeda8 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/README.md +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/README.md @@ -15,10 +15,22 @@ your block on executor threads, and reports each record's outcome. **Wave one.** Connect, configure, receive dispatch waves, run the block, report per-record outcomes, produce records back on success, shut down cleanly. Leases, heartbeats, the manifest -reconnect, worker-death reporting, terminal outcomes, the demo and RubyGems packaging are all -absent - **un-negotiated capabilities rather than half-built features**, since the client declares +reconnect, worker-death reporting, terminal outcomes and RubyGems packaging are all absent - +**un-negotiated capabilities rather than half-built features**, since the client declares `["dispatch"]` and nothing else. The module's testing-evidence record names them. +## Seeing it work + +```bash +demo/run.sh # needs Docker; Ruby and a JDK are optional +``` + +The same records through Ruby's own Kafka client (`rdkafka`) and through this library over a +sidecar, reporting each arm's throughput along with the records it processed and the distinct keys +it saw. [`demo/README.md`](demo/README.md) is its own documentation; the contract it +keeps, which every language's demo keeps, is +[`parallel-consumer-proxy/demo/README.md`](../../parallel-consumer-proxy/demo/README.md). + ## Using it ```ruby diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/Rakefile b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/Rakefile index dfb6266823..3d0857e44f 100644 --- a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/Rakefile +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/Rakefile @@ -40,8 +40,13 @@ end # NOT ADDED TO `default`: the parse belongs to the compile phase, and the test lane's RuboCop # already refuses to lint a file it cannot parse. desc "Parse every Ruby source file - the check Maven's compile phase must fail on" +# THE DEMO IS LISTED EXPLICITLY, AND NOT AS `demo/**/*.rb`. Its bundle installs into +# demo/vendor/bundle, and FileList would happily parse every gem in it - minutes of `ruby -c` over +# third-party source that this module does not ship. Two narrow patterns cover everything the demo +# actually is; a new demo directory needs a line here. task :syntax do - files = FileList["lib/**/*.rb", "spec/**/*.rb"].to_a + ["scripts/conformance-runner"] + files = FileList["lib/**/*.rb", "spec/**/*.rb", "demo/*.rb", "demo/lib/**/*.rb"].to_a + + ["scripts/conformance-runner"] broken = files.reject { |file| system(RbConfig.ruby, "-c", file, out: File::NULL) } abort "syntax check failed: #{broken.join(', ')}" unless broken.empty? puts "syntax OK: #{files.size} files" diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/demo/.bundle/config b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/demo/.bundle/config new file mode 100644 index 0000000000..209d05f323 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/demo/.bundle/config @@ -0,0 +1,3 @@ +--- +BUNDLE_PATH: "vendor/bundle" +BUNDLE_WITHOUT: "development:test" diff --git a/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/demo/Dockerfile b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/demo/Dockerfile new file mode 100644 index 0000000000..10fdd78d49 --- /dev/null +++ b/parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/demo/Dockerfile @@ -0,0 +1,103 @@ +# Copyright (C) 2026 Antony Stubbs and contributors +# +# THE DEMO CONTAINER for the Ruby client (astubbs#242, plan unit U35, governed by KTD40), so that a +# reader with only Docker can run the demo - no Ruby, no JDK, no Maven. +# +# TWO TOOLCHAINS, WHICH IS WHAT MAKES THIS IMAGE DIFFERENT FROM THE JAVA SEED'S. The application is +# Ruby and the sidecar it spawns is a JVM program, so the image is built in two stages: a JDK stage +# that builds the sidecar out of the repository, and a Ruby stage that carries a JRE to run it. Only +# the jars cross the boundary. +# +# THE BUILD CONTEXT IS THE REPOSITORY, AND THAT IS A DELIBERATE DEPARTURE from the rule the C++ +# client's Dockerfile states - "the module directory is the only ordinary context, so the build never +# uploads other agents' target/ output". This build cannot keep it: a Maven reactor module cannot be +# built without its parents and its siblings. The hazard is addressed directly instead, by the +# repository-root .dockerignore that excludes every target/ directory, .git and .claude. Those two +# files must be read together; if you add a build output directory to this repo, add it there too. + +# ── Stage 1: the sidecar ──────────────────────────────────────────────────────────────────────── +# +# A PLAIN JDK, NOT THE `maven:` IMAGE, and that is a bug fix rather than a preference: the official +# Maven image sets MAVEN_CONFIG=/root/.m2 and the Maven WRAPPER appends $MAVEN_CONFIG to its own +# command line, so `./mvnw ` inside that image dies with `Unknown lifecycle phase +# "/root/.m2"`. A JDK base keeps the wrapper, which is what this repo's AGENTS.md requires builds to +# go through. +FROM eclipse-temurin:17-jdk AS sidecar + +WORKDIR /src +COPY . . + +# THE MAVEN REPOSITORY IS A CACHE MOUNT HERE, WHICH THE JAVA SEED'S IMAGE COULD NOT DO. That image +# runs Maven's classpath at run time, so its dependencies have to be IN the image; this one copies +# the jars it needs out of the cache in the same RUN and never looks at /root/.m2 again. The id is +# this module's own, so ten concurrent client builds do not write one local repository at once. +# +# -Dcopyright.skip=true because the header check needs the repository's git history to find its fork +# point, and .dockerignore excludes .git from the context. It is a source-tree gate that already runs +# in every ordinary build and in CI's own copyright lane; re-running it here would prove nothing new +# about the sources it just copied. +# +# EVERY CLASSPATH ENTRY BECOMES A JAR IN /sidecar/lib, including the reactor's own target/classes +# directories - Maven resolves a sibling module to a directory or to a jar depending on how far the +# lifecycle got, and a directory cannot be copied into an image that does not have the tree it came +# from. Jarring it up removes the difference rather than depending on which one happened. +RUN --mount=type=cache,id=pc-ruby-demo-m2,target=/root/.m2/repository \ + ./mvnw --batch-mode -q -DskipTests -Dcopyright.skip=true \ + -pl :parallel-consumer-proxy -am \ + package dependency:build-classpath \ + '-Dmdep.outputFile=${project.build.directory}/sidecar-classpath.txt' \ + && mkdir -p /sidecar/lib \ + && n=0 \ + && for entry in $(tr ':' '\n' < parallel-consumer-proxy/target/sidecar-classpath.txt); do \ + if [ -f "$entry" ]; then cp -n "$entry" /sidecar/lib/ ; \ + elif [ -d "$entry" ]; then (cd "$entry" && jar cf "/sidecar/lib/reactor-$n.jar" .) ; fi ; \ + n=$((n + 1)) ; \ + done \ + && find parallel-consumer-proxy/target -maxdepth 1 -name '*.jar' \ + -not -name '*-tests.jar' -not -name '*-sources.jar' -not -name '*-javadoc.jar' \ + -exec cp {} /sidecar/lib/ \; + +# ── Stage 2: the demo ─────────────────────────────────────────────────────────────────────────── +# +# The Ruby the CI matrix pins for this module. `-bookworm` rather than `-slim`: `rdkafka` ships +# precompiled for linux-gnu, but a platform without a precompiled gem builds librdkafka from source, +# and a base image with no compiler turns that into a build failure a reader cannot act on. +FROM ruby:3.4-bookworm + +# A JRE, NOT A JDK: this image runs the sidecar, it does not build one. Copied from the Temurin +# image rather than installed with apt, so the JVM here and the JVM the sidecar was built with are +# the same distribution and major version. +COPY --from=eclipse-temurin:17-jre /opt/java/openjdk /opt/java/openjdk +ENV JAVA_HOME=/opt/java/openjdk + +COPY --from=sidecar /sidecar/lib /opt/pc-sidecar/lib + +# THE SIDECAR, AS THE CLIENT LIBRARY WANTS IT: an absolute path to a binary, and `java` itself +# rather than a wrapper script. Parent death is detected by EOF on the inherited pipe (KTD19), and a +# shell wrapper would inherit the write end and hold it open - which defeats the signal and leaks a +# JVM that still holds Kafka group membership. The `/*` is expanded by the JVM, not by a shell. +ENV PC_DEMO_SIDECAR_JAVA=/opt/java/openjdk/bin/java +ENV PC_DEMO_SIDECAR_CLASSPATH="/opt/pc-sidecar/lib/*" + +# The Gemfile first and on its own, so that changing the demo's Ruby does not reinstall its gems. +# BUNDLE_WITHOUT repeats what demo/.bundle/config says, because the official Ruby image points +# BUNDLE_APP_CONFIG at /usr/local/bundle and that file is therefore not read in here. +ENV BUNDLE_WITHOUT="development:test" +WORKDIR /app/demo +COPY parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/Gemfile /app/Gemfile +COPY parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/demo/Gemfile /app/demo/Gemfile +RUN bundle install + +# The client library the demo exercises, and the demo itself. Nothing is published to RubyGems in +# this wave, so the library is the one in this checkout or it is nothing. +COPY parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/lib /app/lib +COPY parallel-consumer-proxy-clients/parallel-consumer-proxy-client-ruby/demo /app/demo + +# PC_DEMO_ARGS is unquoted on purpose: it carries the caller's flags through compose as a single +# string and must word-split into separate arguments. +# +# `"$@"` AND THE TRAILING "demo" ARE WHAT MAKE `docker compose run demo --help` WORK. `sh -c +#