Skip to content

Bug: Gerrit Required Verify fails on ~22% of dispatches #4

Open
0 of 1 issue completed
Open

Description

Summary

The Gerrit Required Verify workflow in this repository fails routinely, and
each failure emails the change author and the releng/collab-it lists. This is
not a recent regression — it has been steady for at least the last 35 days.

Sample: the last 200 failed runs (2026-06-30 → 2026-08-04), against 200
consecutive runs over 2026-07-27 → 2026-08-04 for the rate.

Measure Value
Failure rate (last 200 runs) 44 failed, 6 cancelled, 150 succeeded (22%)
Failure notifications ~5.7 per day
Attributable to cause A below 182 of 200 failures (91%)
Attributable to cause B below 16 of 200 failures (8%)
Genuine INFO.yaml validation failures 1 of 200 (working as intended)

Neither dominant cause is a fault in this repository's own content. Both have
upstream fixes, linked below.

Cause A — verify is dispatched for private Gerrit changes (91%)

lfit/releng-gerrit_to_platform dispatches on patchset-created regardless of
whether the change is private. The Gerrit hook runs server-side with full
visibility; the fdio.required.gha account that has to clear and set the vote
does not have it. Both the prepare and vote jobs then die with:

error: fatal: Not found: 46421
fatal: one or more reviews failed; review output above

Worked example — change 46421:

Gerrit event Time (UTC) Run Result
Uploaded patch set 1 (private) 07:36:49 30888530294 ❌ Not found: 46421
Uploaded patch set 2 (private) 07:52:40 30889582764 ❌ Not found: 46421
Unset private 07:55:34 30889787320 ✅ SUCCESS

Uploading private and publishing later is a normal VPP developer workflow — 9 of
the last 60 vpp master changes did it — and every patchset pushed while
private produces a failure email.

Granting the service account visibility of private changes would not fix this:
private change refs are not replicated to GitHub, so the verify job could not
fetch the change anyway.

Importantly, unsetting private fires a fresh dispatch on its own, so suppressing
the earlier dispatches costs no verification coverage.

Upstream fix: lfit/releng-gerrit_to_platform#116

Cause B — shallow checkout when the change has already merged (8%)

gerrit-compose-required-info-yaml-verify.yaml checks isolation with
git diff --name-only HEAD~1 over a fetch-depth: 1 checkout. For CSIT
oper-* cherry-picks, which are submitted pre-approved and merge within seconds
of the dispatch, the branch tip is the change commit, so HEAD~1 lies beyond
the shallow boundary:

fatal: ambiguous argument 'HEAD~1': unknown revision or path not in the working tree.
##[error]Process completed with exit code 128.

Worked example: run 30809172607
for change 46399, merged at 11:21:02,
dispatched at 11:21:06.

Upstream fixes: lfreleng-actions/checkout-gerrit-change-action#129 and
lfit/releng-reusable-workflows#843

Contributing factors in this repository

Independent of the two causes above, both Gerrit workflows here reference
retired sources:

  • lfit/gerrit-review-action — the action now lives under lfreleng-actions
  • technote-space/workflow-conclusion-action — archived November 2023, targets
    Node.js 20, and is the source of the deprecation warnings that appear as
    annotations on every run
  • lfit/releng-reusable-workflows/...@main — unpinned; behaviour can change
    without a corresponding change here

Proposed actions

  1. Short term, here. Add a visibility gate to
    gerrit-required-verify-non-voting.yaml: query Gerrit as the voting account
    before doing anything, and skip the remaining jobs when the change is not
    readable. This removes ~91% of the failure emails immediately and is safe to
    revert once the dispatcher is fixed. A change implementing this is ready and
    can be pushed for review on request.
  2. Short term, here. Refresh the action references above and pin the
    reusable workflow call to a release SHA.
  3. Upstream. Land Bug: dispatch fires for changes the platform cannot read lfit/releng-gerrit_to_platform#116 so no dispatch happens
    while a change is unreadable — this is the correct fix and benefits every LF
    project, not only FD.io.
  4. Upstream. Land the checkout/isolation fixes for cause B.
  5. Longer term. Move the Gerrit required-verify reusable workflows to an
    actively maintained home: Feat: host the Gerrit required-verify reusable workflows lfreleng-actions/generic-workflows#23

Happy to supply the full run classification or raw logs behind any of the
figures above.

Activity

  1. ModeSevenIndustrialSolutions commented on Nov 3, 2026

    @ModeSevenIndustrialSolutions
    ContributorAuthor

    Status update (2026-11-02)

    The failure rate in this issue's title is no longer current. The implementation work is now tracked in #5.

    Measured now

    When filed (2026-06-30 to 2026-08-04, 200 runs) Now (2026-10-29 to 2026-11-01, last 100 runs)
    Success 150 (75%) 98
    Failure 44 1
    Cancelled 6 1

    The one remaining failure (run 41057651379, 2026-10-30) was Unable to resolve action actions/checkout@3d3c42e5…: GitHub could not download an action. It is not a verification failure, and nothing in this workflow's content caused it.

    The drop comes from the inline visibility gate added in 0f8daf8 ("skip required-verify for unreadable changes", 2026-10-20), which is Cause A's short-term fix. It is a hand-written script, though, so the debt below remains.

    The "Proposed actions", one by one

    # Proposed action Status
    1 Visibility gate in gerrit-required-verify-non-voting.yaml Done (0f8daf8). Superseded by the maintained action in #5, which fails closed.
    2 Refresh the retired action references and pin the reusable call Not done: tracked in #5, with exact pins.
    3 Upstream: lfit/releng-gerrit_to_platform#116 (no dispatch for unreadable changes) Merged and released as v0.4.0, but not deployed on this Gerrit. Deploying it is a server-side change and is out of scope for now. Once it is deployed, the gate in #5 can be removed.
    4 Upstream: Cause B (shallow checkout when the change has already merged) Done. checkout-gerrit-change-action fixed it from v1.1.1 (a --depth on the refspec fetch, so the parent stays reachable) and the replacement reusable checks out at depth 2. The released reusable pins v1.2.1.
    5 Longer term: move the required-verify reusables to an actively maintained home Done and released: lfreleng-actions/generic-workflows v0.7.0 (https://github.com/lfreleng-actions/generic-workflows/releases/tag/v0.7.0), tracked in lfreleng-actions/generic-workflows#23.

    What #5 delivers

    Replaces the inline gate, lfit/gerrit-review-action (retired), technote-space/workflow-conclusion-action (archived), and lfit/releng-reusable-workflows/…@main (unpinned) with released, SHA-pinned, hardened equivalents, in the same four-job shape. It also adds step-security/harden-runner in block mode, which the file has never had. The checked pins, the five FD.io-specific differences from the generic example (the file name the dispatcher matches on, the comment-only: true that keeps it non-voting, the fixed dispatcher inputs, the concurrency group, the checks value), the prerequisites that could not be read from outside the organisation, and a validation list are all in #5.

    Two things found while preparing #5

    • INFO.yaml strictness is not a risk for FD.io. The replacement validates INFO.yaml more strictly than the legacy check (it now enforces issue_tracking.type and .url, checks uri formats, runs yamllint, and matches the declared repository to the Gerrit project). csit, vpp and govpp all pass it as-is, using the validator version that will run. Details and the reproduction command are in Feat: Move Gerrit Required Verify to generic-workflows #5.
    • gerrit-verify.yaml is a separate matter, now tracked. It has the same retired references, but it only runs for the Gerrit project that owns this repository (GERRIT_PROJECT: .github), not for vpp or csit, and it is not part of the required-verify path in this issue. Its replacement, linting-verify.yaml, is tracked in Feat: Add the linting-verify reusable workflow lfreleng-actions/generic-workflows#124; this repository adopts it in a separate change once released. Do not move it to linting.yaml: for a per-patchset Gerrit dispatch that workflow lints the branch tip, not the change (detail in Feat: Move Gerrit Required Verify to generic-workflows #5).

    Still open and out of scope for now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions