Skip to content

Feat: Move Gerrit Required Verify to generic-workflows #5

Open

Description

Summary

Replace the hand-written visibility gate and the retired or unpinned references in .github/workflows/gerrit-required-verify-non-voting.yaml with the maintained, hardened equivalents that are now released. This is the implementation work for #4 (Cause A, Cause B and the "contributing factors"); #4 records the evidence.

The urgent symptom is already fixed. Over the last 100 runs of this workflow (2026-10-29 to 2026-11-01): 98 success, 1 failure, 1 cancelled. The one failure was Unable to resolve action actions/checkout@…, a GitHub-side download error, not a verification failure. The inline gate from 0f8daf8 ("skip required-verify for unreadable changes") did that. What remains is technical debt and security posture, not a failure rate.

This issue contains everything needed to author the change. Nothing here needs to be re-researched; each fact below was checked on 2026-11-02, and the sources are given so a reviewer can re-check them.

What is wrong in the file today

Line numbers are for main at 0f8daf8.

Line Today Problem
56-89 visibility job: hand-written ssh gerrit query, key written to ~/.ssh/id_rsa No harden-runner, no permissions:; an SSH failure leaves rows empty, which reads as "not visible" and silently skips verification
98, 141 lfit/gerrit-review-action@3a1b2789… # v1.0.0 Retired source; the action now lives in lfreleng-actions
115 lfit/releng-reusable-workflows/.github/workflows/gerrit-compose-required-info-yaml-verify.yaml@main Unpinned: behaviour changes without a change here. Superseded (see below)
138 technote-space/workflow-conclusion-action@45ce8e0e… # v3.0.3 Archived 2023-11-16, Node 20, source of the deprecation annotations on every run
all no permissions: block The workflow runs with the repository default token scope
all no hardening No step-security/harden-runner anywhere

The replacement

The same job graph, three of the four jobs rebuilt on released components:

visibility -> prepare -> change-isolation -> vote
Job Today After
visibility inline ssh script lfreleng-actions/gerrit-change-visible-action v0.0.2
prepare lfit/gerrit-review-action v1.0.0 lfreleng-actions/gerrit-review-action v1.3.0
info-yaml-verify legacy reusable @main lfreleng-actions/generic-workflows/.github/workflows/change-isolation.yaml v0.7.0
vote technote-space/workflow-conclusion-action + lfit/gerrit-review-action im-open/workflow-conclusion v3.0.0 + lfreleng-actions/gerrit-review-action v1.3.0

Start from the released example, not from scratch:
examples/change-isolation/gerrit-info-yaml-only.yaml in generic-workflows v0.7.0
(https://github.com/lfreleng-actions/generic-workflows/blob/2442f8a20bb9f023e57d9396f9de4278b5e84785/examples/change-isolation/gerrit-info-yaml-only.yaml).
It is the FD.io shape (INFO.yaml isolation only) and is already pinned correctly. Its header comment documents every case below. Adapt it with the five FD.io-specific differences in the next section.

The five FD.io-specific differences from the example

These are the places a straight copy goes wrong.

1. Keep the file name. This is the one that breaks everything if wrong.

lfit/releng-gerrit_to_platform finds required workflows by path, not by name:. From src/gerrit_to_platform/github.py (filter_workflows, called with search_required=True from helpers.py): a workflow in the org's .github repository is dispatched on a patchset when its path contains all three, case-insensitively: verify (the event's filter), gerrit, and required.

gerrit-required-verify-non-voting.yaml satisfies all three. Do not rename it. A file that loses any of the three is silently never dispatched, with no error anywhere. Keep name: Gerrit Required Verify as it is too.

2. The workflow is non-voting: keep comment-only: true

Today both gerrit-review-action steps pass comment-only: true (lines 107 and 150), so this workflow comments on the change and does not vote. The example does not have this, because it votes. Add comment-only: true to both Clear votes and Set vote, exactly as today. Dropping it would make this workflow start casting real Verified votes on every FD.io change.

3. The dispatcher's inputs are a fixed contract

Keep the workflow_dispatch inputs exactly as they are, names and types. gerrit_to_platform sends GERRIT_BRANCH, GERRIT_CHANGE_ID, GERRIT_CHANGE_NUMBER, GERRIT_CHANGE_URL, GERRIT_EVENT_TYPE, GERRIT_PATCHSET_NUMBER, GERRIT_PATCHSET_REVISION, GERRIT_PROJECT, GERRIT_REFSPEC and TARGET_REPO (it sets TARGET_REPO itself, to owner/repo). TARGET_REPO is required: true here and required: false, default: '' in the example; keep required: true. Pass it on as repository: ${{ inputs.TARGET_REPO }}.

4. Keep the file's existing concurrency shape, with the project added

The current group is required-verify-${{ github.workflow }}-${{ github.event.inputs.GERRIT_BRANCH }}-${{ github.event.inputs.GERRIT_CHANGE_ID || github.run_id }}. The example's group also includes inputs.GERRIT_PROJECT, because Gerrit allows the same Change-Id in more than one project and this central repository dispatches for all of them. Take the example's group (with the project) and keep cancel-in-progress: true. Do not put a concurrency: block in the reusable call; the reusable has none by design.

5. Replace TARGET_REPO-less assumptions: the checks input

The example passes exactly the FD.io policy already:

    with:
      checks: |
        [
          {"name": "INFO.yaml", "paths": "/INFO.yaml"}
        ]
      repository: ${{ inputs.TARGET_REPO }}
      gerrit_refspec: ${{ inputs.GERRIT_REFSPEC }}
      gerrit_project: ${{ inputs.GERRIT_PROJECT }}
      gerrit_branch: ${{ inputs.GERRIT_BRANCH }}

Do not add a .github/** check. FD.io wants INFO.yaml isolation only; that policy belongs to ONAP. Do not pass gerrit_url: it falls back to the GERRIT_URL repository variable (see Prerequisites).

Pins

Every pin is a commit SHA. A uses: line cannot take an annotated tag object, and several of these tags are annotated.

Action Version Commit to pin
lfreleng-actions/generic-workflows/.github/workflows/change-isolation.yaml v0.7.0 2442f8a20bb9f023e57d9396f9de4278b5e84785 (not the tag object fb9813dd…)
lfreleng-actions/gerrit-change-visible-action v0.0.2 fa327eebc2b217aa23a42e3da4096109a009a4ed
lfreleng-actions/gerrit-review-action v1.3.0 264fe205361cbb2ee8f26280998a75680ee33b53
step-security/harden-runner v2.22.3 46b06cea07f9e64621fdf066ba77ac6915163d43
lfreleng-actions/harden-runner-block-action v0.15.0 2fed66c9c36c4687c31ca349b925b81ec629673f
im-open/workflow-conclusion v3.0.0 8eac7f17381a6917bc04fec3e2c92e70ebc37526
allow-list: lfreleng-actions/.github file harden-runner/lf-gerrit-servers/gerrit_servers.txt main 6d4a65c5601afd6c2fbe52bda4b0024b132f265d (content is byte-identical to the current HEAD 17856b93…; it carries gerrit.fd.io:29418 and :443)

harden-runner v2.22.3 is mandatory in every job. It fixes a DNS bypass on westus2 hosted runners (GitHub's egress proxy experiment installs a systemd-resolved drop-in) that makes block-mode jobs fail or hang until their timeout. Nothing older may remain.

gerrit-review-action v1.3.0 only adds an optional closed-change input (default warn) to v1.2.0; the example already uses it, with no other change. im-open/workflow-conclusion v3.0.0 is the version the released example pins.

The example's change-isolation call carries the all-zero placeholder SHA 0000000000000000000000000000000000000000 # REPLACE: vX.Y.Z. Replace it with the v0.7.0 commit above. Leaving the placeholder makes the job fail loudly, by design.

Hardening (already in the example; do not weaken)

  • visibility, prepare and vote run block mode with the Gerrit-servers list, loaded by harden-runner-block-action. That is all they talk to beyond GitHub, so add no other hosts. Do not add api.github.com: harden-runner always allows GitHub's own endpoints.
  • permissions: {} at the top of the file, and permissions: {} on visibility and prepare. vote needs actions: read, because workflow-conclusion reads job results through the API. change-isolation needs contents: read.
  • disable-telemetry: true with egress-policy: block.
  • The change-isolation reusable hardens its own jobs (per-job host lists). It needs no harden-runner step in this file.

The example explains the order of the two steps (loader, then harden-runner) and why the allow-list file has a non-standard name; keep its comments.

Behaviour to expect (state it in the PR description)

Case visibility Later jobs
Readable change succeeds, visible=true run as before
Private (unreadable) change succeeds, visible=false skipped: no comment, no failure email
Gerrit unreachable, refused credential or unreadable answer fails, no output skipped; the failed visibility job is the signal

Fail-closed change from today: the inline gate treats an SSH failure as "not visible" and skips quietly. The action fails the job instead, so a Gerrit outage now shows up as a failed run rather than hiding. That is intended, but expect the occasional failure email where there was silence. The dispatcher fix that makes the gate unnecessary (lfit/releng-gerrit_to_platform#116, v0.4.0) is not deployed on this Gerrit, so the gate stays. The example's comments say when it can go.

Prerequisites to confirm before merging

These could not be read from outside the organisation:

  1. Variables and secret. The current file already reads vars.GERRIT_SERVER, vars.GERRIT_SSH_REQUIRED_USER, vars.GERRIT_KNOWN_HOSTS and secrets.GERRIT_SSH_REQUIRED_PRIVKEY, and it runs, so they exist. The new workflow additionally reads vars.GERRIT_URL, through the reusable's fallback. The legacy reusable it replaces read the same variable and the runs succeed, so it is almost certainly set; confirm it (it is the Gerrit base URL with any path prefix, for example https://gerrit.fd.io/r). Confirm vars.GERRIT_SERVER is exactly gerrit.fd.io, since the allow-list names that host.
  2. vars.GERRIT_KNOWN_HOSTS must be an ssh-keyscan -p 29418 gerrit.fd.io line, which names the port: [gerrit.fd.io]:29418 ssh-…. gerrit-change-visible-action verifies the host key strictly, so a bare gerrit.fd.io ssh-… line fails with a host-key error. The existing gerrit-review-action steps use the same value, so it probably already has the right form; the action's failure message says "check the key, user name and known_hosts".
  3. The org's Actions policy. At repository level FDio/.github allows all actions and does not require SHA pinning (checked via the API). The organisation-level policy needs admin rights to read. step-security/harden-runner and lfreleng-actions/harden-runner-block-action are new to this repository; if the organisation restricts actions to an allow-list, those two, plus lfreleng-actions/gerrit-change-visible-action, lfreleng-actions/gerrit-review-action and im-open/workflow-conclusion, need adding. lfreleng-actions/checkout-gerrit-change-action and shimataro/ssh-key-action already run here.

INFO.yaml strictness: assessed, low risk

change-isolation.yaml validates INFO.yaml with info-yaml-validate-action v0.1.0, which is stricter than the legacy check: the bundled schema now requires issue_tracking.type and .url (the old schema declared them but never enforced them), checks uri formats, runs yamllint first, and checks that the declared repositories match the Gerrit project. The job runs only when a change touches the root INFO.yaml, so no other change is affected.

All of FD.io's INFO.yaml files were run through the exact validator version on 2026-11-02, with the project name the dispatcher sends as GERRIT_PROJECT:

Repository Schema + yamllint With repository match
csit valid valid (--project csit)
vpp valid valid (--project vpp)
govpp valid valid (--project govpp)
.github valid not a Gerrit project, never dispatched

The validator was checked to reject what it should (a missing issue_tracking.type, an unquoted date, trailing spaces, a wrong project), so these passes are not vacuous. To reproduce:

uvx --from "git+https://github.com/lfreleng-actions/info-yaml-validate-action@d3e4e7369a8ea5f6c19900c92c266a1e1c5b8172" \
  info-yaml-validate --project vpp INFO.yaml

Deleting INFO.yaml, or renaming it away, fails the check by design: the file is mandatory in every Gerrit project.

What this issue does not cover

  • gerrit-verify.yaml (the lint workflow, same directory) has the same retired references (lfit/gerrit-review-action v0.8, technote-space/workflow-conclusion-action, lfit/releng-reusable-workflows/.github/workflows/compose-repo-linting.yaml@main). Do not touch it in this issue. Its replacement is being built and is tracked in Feat: Add the linting-verify reusable workflow lfreleng-actions/generic-workflows#124 (linting-verify.yaml); this repository adopts it in a separate change once that is released.

    Do not migrate it to linting.yaml, even though the older generic-workflows docs suggest a Gerrit-mirrored project "migrates wholesale" to it. That workflow takes no Gerrit checkout inputs, and gerrit_to_platform dispatches on refs/heads/<GERRIT_BRANCH>, so it would lint the branch tip, not the change and could report green. That was checked with real git and against the dispatcher's source, and is why #124 builds a sibling workflow and corrects those docs.

    Its blast radius for FD.io is small. It has no required in its filename, so gerrit_to_platform does not treat it as a required workflow: it only runs for the Gerrit project that owns this repository, the Gerrit project named .github. Its recent runs all carry GERRIT_PROJECT: .github with repository: FDio/.github (109 runs in total, the latest on 2026-10-28, all workflow_dispatch). It lints changes to this repository itself, not changes to vpp, csit or the other projects, and it is not part of the required-verify path that Bug: Gerrit Required Verify fails on ~22% of dispatches #4 describes. Its retired references are real debt but are not causing the failures in Bug: Gerrit Required Verify fails on ~22% of dispatches #4.

    It is not FD.io-specific, though: the same gerrit-verify.yaml calling compose-repo-linting.yaml runs in 30 repositories (18 opendaylight, 8 lfit, and the .github repositories of onap, o-ran-sc and FDio). When the time comes, one migration step applies here too: the legacy workflow ran actionlint as a separate job, while the replacement runs it only if the repository's own .pre-commit-config.yaml defines the hook. For this repository the answer is yes: FDio/.github's .pre-commit-config.yaml already defines an actionlint hook (checked 2026-11-03), so migrating loses nothing on that front.

  • FDio/archived-ci-management also references the legacy workflows. It is archived; leave it.

  • The dispatcher fix (lfit/releng-gerrit_to_platform#116) being deployed on this Gerrit. A separate, server-side change; out of scope for now. Once it is, remove the visibility job as the example describes.

  • The second dispatcher precondition: a project whose GitHub mirror is absent, empty or archived is still dispatched and can never verify. checkout-gerrit-change-action now reports the mirror as the cause when that happens (Feat: Name the mirror when a change is missing lfreleng-actions/checkout-gerrit-change-action#176), but generic-workflows v0.7.0 still pins the release before that change, so it will not appear here until generic-workflows bumps the action. The FD.io GitHub mirrors are healthy today (the four with an INFO.yaml are populated and unarchived), so this is not an FD.io problem now.

Validation (state results in the pull request)

  • grep -n "uses:" .github/workflows/gerrit-required-verify-non-voting.yaml shows no lfit/, technote-space/ or @main, and no all-zero SHA.
  • grep -n "harden-runner@" … shows only 46b06cea… # v2.22.3, on every harden-runner step.
  • grep -n "comment-only" … shows comment-only: true on both gerrit-review-action steps.
  • ls .github/workflows/ | grep -i required still lists gerrit-required-verify-non-voting.yaml (the name is unchanged).
  • actionlint and zizmor --persona auditor report nothing on the file. The released example passes zizmor with line-scoped ignores for two audits (impostor-commit/stale-action-refs on the all-zero placeholder, which no longer applies once replaced, and secrets-outside-env on the SSH key, which does); check whether the key finding appears here and handle it in this repository's own zizmor configuration, not with an inline suppression.
  • Test it for real. Merging is the first time any of this runs against Gerrit, because the workflow only runs when dispatched. Before merging, trigger it with workflow_dispatch from the branch against a real, public, open change in an FD.io Gerrit project, supplying every input (take them from a recent run of the current workflow: the run's log shows GERRIT_BRANCH, GERRIT_CHANGE_NUMBER, GERRIT_REFSPEC and the rest, and TARGET_REPO is FDio/<project>). Use a change you are happy to have a comment posted on, since this workflow comments (it is non-voting). Confirm visibility reports visible=true, change-isolation passes, and the comment arrives on the change. Then try a private change (upload one with git push origin HEAD:refs/for/master%private, and unset it afterwards, which also dispatches a fresh run) and confirm the later jobs are skipped and nothing is posted. State plainly in the PR which of these you did and which you could not; a step you could not run must not be reported as passed.
  • After merge, watch the first dispatched runs (a patchset upload on any FD.io project). Expect Change visibility, Clear vote, Change Isolation (with Plan and Report) and Set vote in the run.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions