Repository navigation
Description
Summary
The Gerrit Required Verify workflow in this repository fails routinely, and
each failure emails the change author and the releng/collab-it lists. This is
not a recent regression — it has been steady for at least the last 35 days.
Sample: the last 200 failed runs (2026-06-30 → 2026-08-04), against 200
consecutive runs over 2026-07-27 → 2026-08-04 for the rate.
| Measure | Value |
|---|---|
| Failure rate (last 200 runs) | 44 failed, 6 cancelled, 150 succeeded (22%) |
| Failure notifications | ~5.7 per day |
| Attributable to cause A below | 182 of 200 failures (91%) |
| Attributable to cause B below | 16 of 200 failures (8%) |
| Genuine INFO.yaml validation failures | 1 of 200 (working as intended) |
Neither dominant cause is a fault in this repository's own content. Both have
upstream fixes, linked below.
Cause A — verify is dispatched for private Gerrit changes (91%)
lfit/releng-gerrit_to_platform dispatches on patchset-created regardless of
whether the change is private. The Gerrit hook runs server-side with full
visibility; the fdio.required.gha account that has to clear and set the vote
does not have it. Both the prepare and vote jobs then die with:
error: fatal: Not found: 46421
fatal: one or more reviews failed; review output above
Worked example — change 46421:
| Gerrit event | Time (UTC) | Run | Result |
|---|---|---|---|
| Uploaded patch set 1 (private) | 07:36:49 | 30888530294 | ❌ Not found: 46421 |
| Uploaded patch set 2 (private) | 07:52:40 | 30889582764 | ❌ Not found: 46421 |
| Unset private | 07:55:34 | 30889787320 | ✅ SUCCESS |
Uploading private and publishing later is a normal VPP developer workflow — 9 of
the last 60 vpp master changes did it — and every patchset pushed while
private produces a failure email.
Granting the service account visibility of private changes would not fix this:
private change refs are not replicated to GitHub, so the verify job could not
fetch the change anyway.
Importantly, unsetting private fires a fresh dispatch on its own, so suppressing
the earlier dispatches costs no verification coverage.
Upstream fix: lfit/releng-gerrit_to_platform#116
Cause B — shallow checkout when the change has already merged (8%)
gerrit-compose-required-info-yaml-verify.yaml checks isolation with
git diff --name-only HEAD~1 over a fetch-depth: 1 checkout. For CSIT
oper-* cherry-picks, which are submitted pre-approved and merge within seconds
of the dispatch, the branch tip is the change commit, so HEAD~1 lies beyond
the shallow boundary:
fatal: ambiguous argument 'HEAD~1': unknown revision or path not in the working tree.
##[error]Process completed with exit code 128.
Worked example: run 30809172607
for change 46399, merged at 11:21:02,
dispatched at 11:21:06.
Upstream fixes: lfreleng-actions/checkout-gerrit-change-action#129 and
lfit/releng-reusable-workflows#843
Contributing factors in this repository
Independent of the two causes above, both Gerrit workflows here reference
retired sources:
lfit/gerrit-review-action— the action now lives underlfreleng-actionstechnote-space/workflow-conclusion-action— archived November 2023, targets
Node.js 20, and is the source of the deprecation warnings that appear as
annotations on every runlfit/releng-reusable-workflows/...@main— unpinned; behaviour can change
without a corresponding change here
Proposed actions
- Short term, here. Add a visibility gate to
gerrit-required-verify-non-voting.yaml: query Gerrit as the voting account
before doing anything, and skip the remaining jobs when the change is not
readable. This removes ~91% of the failure emails immediately and is safe to
revert once the dispatcher is fixed. A change implementing this is ready and
can be pushed for review on request. - Short term, here. Refresh the action references above and pin the
reusable workflow call to a release SHA. - Upstream. Land Bug: dispatch fires for changes the platform cannot read lfit/releng-gerrit_to_platform#116 so no dispatch happens
while a change is unreadable — this is the correct fix and benefits every LF
project, not only FD.io. - Upstream. Land the checkout/isolation fixes for cause B.
- Longer term. Move the Gerrit required-verify reusable workflows to an
actively maintained home: Feat: host the Gerrit required-verify reusable workflows lfreleng-actions/generic-workflows#23
Happy to supply the full run classification or raw logs behind any of the
figures above.
Activity
Feat: Add the Gerrit change visibility check
ModeSevenIndustrialSolutions commentedon Nov 3, 2026
Status update (2026-11-02)
The failure rate in this issue's title is no longer current. The implementation work is now tracked in #5.
Measured now
The one remaining failure (run 41057651379, 2026-10-30) was
Unable to resolve action actions/checkout@3d3c42e5…: GitHub could not download an action. It is not a verification failure, and nothing in this workflow's content caused it.The drop comes from the inline visibility gate added in
0f8daf8("skip required-verify for unreadable changes", 2026-10-20), which is Cause A's short-term fix. It is a hand-written script, though, so the debt below remains.The "Proposed actions", one by one
gerrit-required-verify-non-voting.yaml0f8daf8). Superseded by the maintained action in #5, which fails closed.lfit/releng-gerrit_to_platform#116(no dispatch for unreadable changes)checkout-gerrit-change-actionfixed it from v1.1.1 (a--depthon the refspec fetch, so the parent stays reachable) and the replacement reusable checks out at depth 2. The released reusable pins v1.2.1.lfreleng-actions/generic-workflowsv0.7.0 (https://github.com/lfreleng-actions/generic-workflows/releases/tag/v0.7.0), tracked in lfreleng-actions/generic-workflows#23.What #5 delivers
Replaces the inline gate,
lfit/gerrit-review-action(retired),technote-space/workflow-conclusion-action(archived), andlfit/releng-reusable-workflows/…@main(unpinned) with released, SHA-pinned, hardened equivalents, in the same four-job shape. It also addsstep-security/harden-runnerin block mode, which the file has never had. The checked pins, the five FD.io-specific differences from the generic example (the file name the dispatcher matches on, thecomment-only: truethat keeps it non-voting, the fixed dispatcher inputs, the concurrency group, thechecksvalue), the prerequisites that could not be read from outside the organisation, and a validation list are all in #5.Two things found while preparing #5
INFO.yamlmore strictly than the legacy check (it now enforcesissue_tracking.typeand.url, checksuriformats, runs yamllint, and matches the declared repository to the Gerrit project).csit,vppandgovppall pass it as-is, using the validator version that will run. Details and the reproduction command are in Feat: Move Gerrit Required Verify to generic-workflows #5.gerrit-verify.yamlis a separate matter, now tracked. It has the same retired references, but it only runs for the Gerrit project that owns this repository (GERRIT_PROJECT: .github), not forvpporcsit, and it is not part of the required-verify path in this issue. Its replacement,linting-verify.yaml, is tracked in Feat: Add the linting-verify reusable workflow lfreleng-actions/generic-workflows#124; this repository adopts it in a separate change once released. Do not move it tolinting.yaml: for a per-patchset Gerrit dispatch that workflow lints the branch tip, not the change (detail in Feat: Move Gerrit Required Verify to generic-workflows #5).Still open and out of scope for now
gerrit_to_platformv0.4.0 on this Gerrit.checkout-gerrit-change-actionnow names the mirror as the cause (Feat: Name the mirror when a change is missing lfreleng-actions/checkout-gerrit-change-action#176), and that reaches this workflow whengeneric-workflowsnext bumps the action.lfit/releng-reusable-workflowsis NOT fully retired by Feat: Move Gerrit Required Verify to generic-workflows #5. Feat: Move Gerrit Required Verify to generic-workflows #5 removes the reference ingerrit-required-verify-non-voting.yaml(gerrit-compose-required-info-yaml-verify.yaml@main).gerrit-verify.yamlstill callscompose-repo-linting.yaml@mainfrom the same repository, and keeps doing so until its own replacement exists (see above). Until then the dependency stays, unpinned, for that one workflow.FDio/archived-ci-managementalso references the legacy workflows; it is archived and can be left.chore(deps): Bump lfit/gerrit-review-action from 0.8 to 0.9