Skip to content

fix(ci): content watchdog waits out the grace period while the mirror catches up - #3813

Open
Hugo0 wants to merge 1 commit into
devfrom
fix/content-watchdog-grace
Open

Hugo0 wants to merge 1 commit into
devfrom
fix/content-watchdog-grace

Conversation

@Hugo0

@Hugo0 Hugo0 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Why

On 2026-10-23 the content watchdog paged "Content is not reaching production … stuck for 2525m". Nothing was stuck. The check ran at 12:19:52 UTC, one minute after Konrad pushed content (mono 312e1fa). The mirror published it at 12:20, and production was current minutes later.

Two defects in the alert:

  • A mirror that is behind skipped the 45-minute grace period, as if it had failed. Behind is also what a healthy mirror looks like for the minute after every push.
  • The age was measured from the previous publish (21 Oct), so a one-minute delay read as 42 hours.

Change

  • Behind mono now gets the grace period, aged from the expected mono commit.
  • A degraded watchdog or a mirror that published a commit off main still alerts at once.
  • Production behind peanut-content keeps its age from the newest mirror commit, as before.

Verified

Ran the step's script with a fake gh:

  • expected commit 5 min old, mirror behind → no alert, exit 0;
  • expected commit 60 min old, mirror behind → alert, exit 1;
  • mirror current → "production content is current", exit 0.

The watchdog runs on a schedule, so it uses the copy on main: this takes effect with the next dev→main release.

… catches up

A check one minute after a content push paged as "stuck 2525m": a mirror that is
behind skipped the grace period, and the age was measured from the previous publish.
@vercel

vercel Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
peanut-wallet Ready Ready Preview Oct 23, 2026 2:55pm UTC

Request Review

@coderabbitai

coderabbitai Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ This PR changes CI workflows or the backdoor scan itself

The scan in tests.yml runs this PR's own workflow, so a change here could switch it off. This check stays red until the change is merged. Someone other than the author must read these files before anyone runs the branch:

  • .github/workflows/content-pipeline-watchdog.yml

@github-actions

Copy link
Copy Markdown
Contributor

Code-analysis diff

Painscore total: 10484.57 → 10484.57 (0)
Findings: 0 net (+0 new, -0 resolved)

@chip-peanut-bot chip-peanut-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chip review — no blocking findings — this is not an approval

One major watchdog gap: repeated content commits can indefinitely defer a stuck-mirror alert.

Findings

  • MAJOR · .github/workflows/content-pipeline-watchdog.yml:183 · New content commits can indefinitely postpone a stuck-mirror alert
    If the mirror stops publishing and mono receives publishable content commits every 30 minutes, EXPECTED is always the newest commit and EXPECTED_AT is always under 45 minutes old. This branch therefore exits within grace on every scheduled run, even though the oldest unpublished content has been stuck for hours. Start the grace clock at the oldest still-unmirrored publishable commit (or preserve when the behind state first began), so later pushes cannot reset the deadline.

Checked clean

  • Pinned head and base match; reviewed the watchdog workflow and mono mirror workflow.
  • The failed guard is the workflow-change gate; its deterministic backdoor scan reported clean. Other head checks were partly still running at review time.

Security review by moonshotai/kimi-k3: 0 finding(s), marked with the model name. It reads the diff only and answers only security, privacy and money, so treat its findings as advice.

Third opinion: did not run — claude-api_error. This review is one reviewer short.

Exact head: d6342455f7f3 · Context: repo, mono · Took 4m

# reporting "stuck 2525m" measured from the previous publish.
STALE="the mirror is behind mono (expected \`${EXPECTED:0:7}\`, published \`${MIRRORED:0:7}\`, compare: ${FRESH})"
MIRROR_BROKEN=true
WAITING_SINCE=${EXPECTED_AT:-}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAJOR: New content commits can indefinitely postpone a stuck-mirror alert

If the mirror stops publishing and mono receives publishable content commits every 30 minutes, EXPECTED is always the newest commit and EXPECTED_AT is always under 45 minutes old. This branch therefore exits within grace on every scheduled run, even though the oldest unpublished content has been stuck for hours. Start the grace clock at the oldest still-unmirrored publishable commit (or preserve when the behind state first began), so later pushes cannot reset the deadline.

@github-actions

Copy link
Copy Markdown
Contributor

🧪 UI test report — ✅ all green

Suites

  • ✅ unit: 13779 ran, 0 failed, 0 skipped, 5.7m

📊 Coverage (unit)

metric %
statements 85.7%
branches 77.1%
functions 80.8%
lines 87.0%
⏱ 10 slowest test cases
time test
🐢 9.0s src/app/(mobile-ui)/qr-pay/__tests__/qr-pay-states.test.tsx › Network failure keeps loading while retries remain, then shows the generic error
🐢 7.7s scripts/__tests__/indexable-urls.test.ts › excludes missing locale files, drafts, redirects, and retired pages from crawl and submission lists
🐢 7.4s scripts/__tests__/signup-attribution-paths.test.ts › includes newly published articles and locale fallbacks without editing the manifest
4.8s src/utils/__tests__/deferred-link.test.ts › preserves campaigns on old apps and content on new apps for every public android landing
4.7s scripts/__tests__/indexable-urls.test.ts › gives crawls and IndexNow exactly the canonical inventory
4.0s src/features/setup/__tests__/useSignupPasskey.test.tsx › opens the help drawer for NotReadableError and retries from it with one more ceremony
4.0s src/app/(mobile-ui)/qr-pay/__tests__/qr-pay-states.test.tsx › MANTECA_USER_NOT_PROVISIONED fails fast with copy that names the real cause
4.0s src/app/(mobile-ui)/qr-pay/__tests__/qr-pay-states.test.tsx › MANTECA_MERCHANT_RECENT_REFUND fails fast with copy that names the real cause
4.0s src/app/(mobile-ui)/qr-pay/__tests__/qr-pay-states.test.tsx › a refused idempotency key tells the user to scan again, not to contact support
4.0s src/app/(mobile-ui)/qr-pay/__tests__/qr-pay-states.test.tsx › User KYC not approved fails fast with copy that names the real cause
📍 Inline annotations are in the **Unit test report** check above. Coverage artifact: `coverage-unit`. Generated by `.github/workflows/tests.yml`.

This branch was successfully deployed

1 active deployment
Preview — d6342455 Deployed Oct 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant