chore(fidelity): single upstream pin, SHA check, it.each extraction, two-tier mapping (#185)

.github/workflows/lint.yml

@@ -8,6 +8,9 @@ on:
       - 'tests/**'
       - 'scripts/**'
       - 'pyproject.toml'
+      - 'scripts/upstream_pin.json'
+      - 'CLAUDE.md'
+      - 'docs/UPSTREAM_SYNC.md'
       - '.github/workflows/lint.yml'
   pull_request:
     branches: [main]
@@ -17,6 +20,9 @@ on:
       - 'tests/**'
       - 'scripts/**'
       - 'pyproject.toml'
+      - 'scripts/upstream_pin.json'
+      - 'CLAUDE.md'
+      - 'docs/UPSTREAM_SYNC.md'
       - '.github/workflows/lint.yml'
   workflow_dispatch:
 
@@ -57,11 +63,27 @@ jobs:
         continue-on-error: true
         run: uv run python scripts/audit_test_quality.py
 
+      - name: Pin / doc drift check
+        id: pin_docs
+        continue-on-error: true
+        run: uv run python scripts/verify_test_fidelity.py --check-docs
+
+      # Tag and commit SHA come from scripts/upstream_pin.json (the single
+      # source of truth). No continue-on-error: a failed clone or a tag that
+      # no longer resolves to the pinned SHA aborts the job loudly.
       - name: Clone upstream vercel/chat at pinned parity tag
         id: clone_upstream
         run: |
-          git clone --depth 1 --branch chat@4.31.0 \
+          PIN_TAG="$(jq -r .pin.tag scripts/upstream_pin.json)"
+          PIN_SHA="$(jq -r .pin.sha scripts/upstream_pin.json)"
+          git clone --depth 1 --branch "$PIN_TAG" \
             https://github.com/vercel/chat.git /tmp/vercel-chat
+          HEAD_SHA="$(git -C /tmp/vercel-chat rev-parse HEAD)"
+          echo "Upstream vercel/chat: \`$PIN_TAG\` @ \`$HEAD_SHA\` (pinned \`$PIN_SHA\`)" >> "$GITHUB_STEP_SUMMARY"
+          if [ "$HEAD_SHA" != "$PIN_SHA" ]; then
+            echo "::error::$PIN_TAG resolved to $HEAD_SHA, but scripts/upstream_pin.json pins $PIN_SHA"
+            exit 1
+          fi
 
       - name: Test fidelity check (strict — zero missing in mapped core files)
         id: fidelity
@@ -88,6 +110,7 @@ jobs:
           echo "| Ruff check | ${{ steps.ruff_check.outcome }} |" >> $GITHUB_STEP_SUMMARY
           echo "| Ruff format | ${{ steps.ruff_format.outcome }} |" >> $GITHUB_STEP_SUMMARY
           echo "| Test audit | ${{ steps.audit.outcome }} |" >> $GITHUB_STEP_SUMMARY
+          echo "| Pin / doc drift | ${{ steps.pin_docs.outcome }} |" >> $GITHUB_STEP_SUMMARY
           echo "| Test fidelity | ${{ steps.fidelity.outcome }} |" >> $GITHUB_STEP_SUMMARY
           echo "| Pyrefly | ${{ steps.pyrefly.outcome }} |" >> $GITHUB_STEP_SUMMARY
           echo "" >> $GITHUB_STEP_SUMMARY
@@ -103,11 +126,12 @@ jobs:
           RUFF_CHECK: ${{ steps.ruff_check.outcome }}
           RUFF_FORMAT: ${{ steps.ruff_format.outcome }}
           AUDIT: ${{ steps.audit.outcome }}
+          PIN_DOCS: ${{ steps.pin_docs.outcome }}
           FIDELITY: ${{ steps.fidelity.outcome }}
           PYREFLY: ${{ steps.pyrefly.outcome }}
         run: |
           failures=0
-          for var in RUFF_CHECK RUFF_FORMAT AUDIT FIDELITY PYREFLY; do
+          for var in RUFF_CHECK RUFF_FORMAT AUDIT PIN_DOCS FIDELITY PYREFLY; do
             outcome="${!var}"
             if [ "$outcome" != "success" ]; then
               echo "$var failed (outcome: $outcome)"

CHANGELOG.md

@@ -2,8 +2,22 @@
 
 ## Unreleased (4.41 wave)
 
-Part of the upstream `4.31.0` → `4.41.1` sync wave (#184). `UPSTREAM_PARITY` stays `4.31.0` until the wave's final pin bump.
-
+Sync wave from `chat@4.31.0` to `chat@4.41.1` (tracking #184). `UPSTREAM_PARITY` stays `4.31.0` until #203.
+
+- **Tooling: fidelity tooling for the 4.41 wave** (#185, closes #79, advances #78). Tooling only; no consumer-visible behavior change.
+  - `scripts/upstream_pin.json` is now the single source for the upstream checkout: `pin` (strict CI tag `chat@4.31.0` plus its commit SHA) and `target` (`chat@4.41.1` plus its SHA). `verify_test_fidelity.py`, `lint.yml` (via `jq`) and the `CLAUDE.md` clone snippet all read it.
+  - CI fails if the cloned tag's HEAD differs from `pin.sha`, and writes the SHA to the job summary. The script does the same check when `TS_ROOT` is a git checkout, and only warns for a plain export.
+  - New `--check-docs` step in CI: `CLAUDE.md` and `docs/UPSTREAM_SYNC.md` can no longer name a different pin. This fixed the stale `4.30.0` line. The pin's major.minor must also match `UPSTREAM_PARITY`.
+  - `it.each` / `test.each` templates now count as one logical test each, with placeholders stripped. `test("…")` also counts, and `describe.each` titles are used for reporting. The strict count at the pin is now **733/733** (was 732): `thread.test.ts`'s existing `$label` template is now checked.
+  - Scope is two-tier. `MAPPING` stays strict. New `TARGET_MAPPING` rows are checked only by the new `--report-target` mode. New `UNMAPPED` lists deliberate skips with reasons, and every core `*.test.ts(x)` must be classified in one of the three.
+  - Per-file output now shows exact vs fuzzy match counts.
+  - Fuzzy-match ties now resolve the same way in every run. They used to depend on the per-process string hash seed, so the target total could flip between 282 and 283 missing.
+  - The fuzzy pass matches plain tests before `.each` templates. A template's placeholder-stripped title is short enough to claim a later plain test's translation, which reported `chat.test.ts` "should not match GitHub-style logins as Slack ids (case sensitivity)" as missing while hiding the real `isMention` template gap.
+  - Fails closed instead of silently undercounting: `--strict` and `--report-target` fail when an upstream test cannot be extracted (baseline mode only warns). The extractor now also reads modifier chains (`it.skip`, `it.concurrent.each`, …), `.for`, `<T>` type arguments, `.skipIf(c)` / `.runIf(c)` and single-quoted or wrapped titles, and reports any other call form (`it.todo`, `it.extend`, …). The checkout check also fails on local edits under `packages/chat/src`, and when `TS_ROOT/.git` exists but git cannot read it. `--report-target` also refuses a sparse or partial checkout (a core test file in the commit but not on disk) and a plain export missing a mapped file, instead of reporting that file as absent at the target.
+  - Test calls are discovered only in code: a `'test("x")'` fixture string or a commented-out `// it.each(…)` is neither counted nor reported as unextractable, and a `describe.only("s", () => { it("t") })` one-liner no longer drops its test. A template literal or block comment left open at end of file is an extraction error instead of silently hiding the tests after it, and a regex right after `if (…)` / `while (…)` / `for (…)` is lexed as a regex. A computed title (`"rejects " + kind`) is an extraction error instead of being matched by its literal prefix. Extraction over every upstream `*.test.ts(x)` at both tags is unchanged.
+  - `--report-target` prints its delta against the report committed at `HEAD` (via `git show`), so a second run before committing no longer compares against the first run's output and prints `+0`.
+  - `--check-docs` is case-insensitive and also catches `--branch=`, `-b`, line-continued and markdown-decorated pins.
+  - `scripts/fidelity_target.json` is committed as the authoritative wave-wide missing list at `chat@4.41.1`: 282 missing of 1036 (130 in strict-tier files, 152 in target-tier files), including 16 `.each` templates.
 - **BREAKING (security) — Telegram: webhook verification is required by default; repeated updates are deduplicated** (#224; upstream vercel/chat#858, #799, #813).
   - Telegram webhook deployments without `TELEGRAM_WEBHOOK_SECRET_TOKEN` / `secret_token` now **fail to start or return 401**: `mode="webhook"` raises `ValidationError` in the constructor, `mode="auto"` raises from `initialize()` when it resolves to webhook mode (so `Chat` initialization fails and is retried on every webhook until the config is fixed), and `handle_webhook` returns 401 `"Webhook verification required"` before reading the body. Previously the adapter logged a warning and dispatched every update, including `callback_query` button actions.
   - **Escape hatch:** `TelegramAdapterConfig(allow_unverified_webhooks=True)` or `TELEGRAM_ALLOW_UNVERIFIED_WEBHOOKS=true` (only the exact string `"true"` counts; an explicit `allow_unverified_webhooks=False` wins over the env var; a non-`bool` value such as `"false"` raises `ValidationError` rather than being treated as truthy). Polling mode needs neither.

CLAUDE.md

@@ -113,23 +113,38 @@ async mock bugs, and cross-file duplicates. PRs that introduce hard failures
 will not pass CI.
 
 **Fidelity check** (`scripts/verify_test_fidelity.py`) verifies every TS
-`it("...")` in the mapped core files has a matching Python `def test_*()`,
-pinned to `chat@4.31.0` (matches `UPSTREAM_PARITY`; upstream never tagged
-`chat@4.27.0`/`chat@4.28.0`). The `MAPPING` dict in that script is the
-authoritative scope list — extending it to the remaining unmapped
-`packages/chat/src/*.test.ts` files is tracked as issue #78.
+`it("...")` / `test("...")` in the mapped core files — plus one logical
+test per `it.each` template — has a matching Python `def test_*()`,
+pinned to `chat@4.31.0` (major.minor must match `UPSTREAM_PARITY`).
+The pin lives in **`scripts/upstream_pin.json`** (`pin` = strict CI tag +
+commit SHA, `target` = the sync wave's tag); nothing else hard-codes it.
+Scope is two-tier: `MAPPING` is the strict set; `TARGET_MAPPING` rows are
+only checked by `--report-target`; `UNMAPPED` lists deliberate skips with
+reasons. Every `packages/chat/src/**/*.test.ts(x)` must be in one of the
+three (remaining unmapped work: issue #78).
 **CI runs `--strict`** (see `.github/workflows/lint.yml`):
-any missing translation in a mapped file fails the build, and a missing
-upstream checkout also fails (the script exits non-zero when any mapped
-TS file isn't found). Baseline mode (the default without `--strict`) is
-retained for local workflows where a few ports land in flight —
-regenerate via `--update-baseline` after documenting intentional
-divergence in `docs/UPSTREAM_SYNC.md`.
+any missing translation in a mapped file fails the build, a missing
+upstream checkout fails, and a checkout whose HEAD differs from the pinned
+SHA fails (a plain export without `.git` only warns). CI also runs
+`--check-docs`, which fails if a `--branch chat@X` / "pinned to chat@X"
+phrase in this file or `docs/UPSTREAM_SYNC.md` disagrees with the pin.
+Baseline mode (the default without `--strict`) is retained for local
+workflows where a few ports land in flight — regenerate via
+`--update-baseline` after documenting intentional divergence in
+`docs/UPSTREAM_SYNC.md`.
 
 Before the fidelity check can run locally, clone the pinned upstream
 checkout (same command CI uses in `lint.yml`):
 ```bash
-git clone --depth 1 --branch chat@4.31.0 \
+git clone --depth 1 --branch "$(jq -r .pin.tag scripts/upstream_pin.json)" \
   https://github.com/vercel/chat.git /tmp/vercel-chat
 ```
 Then `TS_ROOT=/tmp/vercel-chat uv run python scripts/verify_test_fidelity.py --strict`.
+
+**Sync-wave target report:** with a checkout of the target tag
+(`jq -r .target.tag scripts/upstream_pin.json`),
+`TS_ROOT=<that checkout> uv run python scripts/verify_test_fidelity.py --report-target`
+checks `MAPPING` + `TARGET_MAPPING`, never fails on missing tests, and
+rewrites `scripts/fidelity_target.json` (the committed wave-wide missing
+list, with exact vs fuzzy match counts). Wave PRs regenerate it and quote
+the printed delta.

CONTRIBUTING.md

@@ -81,10 +81,11 @@ Vercel Chat version. See [UPSTREAM_SYNC.md](docs/UPSTREAM_SYNC.md#version-mappin
    uv run ruff check src/ tests/ scripts/
    uv run ruff format --check src/ tests/ scripts/
    uv run python scripts/audit_test_quality.py
-   # verify_test_fidelity.py needs the upstream TS repo at $TS_ROOT (default
-   # /tmp/vercel-chat). Without it, the script silently skips checks and exits
-   # 0, so releases can ship unverified. Clone once:
-   #   git clone https://github.com/vercel/chat.git /tmp/vercel-chat
+   # verify_test_fidelity.py needs a checkout of the pinned upstream tag at
+   # $TS_ROOT (default /tmp/vercel-chat). It fails if the checkout is missing
+   # or its HEAD differs from the SHA in scripts/upstream_pin.json. Clone once:
+   #   git clone --depth 1 --branch "$(jq -r .pin.tag scripts/upstream_pin.json)" \
+   #     https://github.com/vercel/chat.git /tmp/vercel-chat
    uv run python scripts/verify_test_fidelity.py
    uv run pytest tests/ --tb=short -q
    ```