Repository navigation
Conversation
…step EnsureTraceSigningKeys plans the trace signing key of each selected GitHub repository before repos.yaml is written: a fresh install, recovery when FULLSEND_TRACE_SIGNING_KEY is missing or FULLSEND_TRACE_SIGNING_KEY_ID is absent or names a retained key, or a rotation on request. It records the installation ID and public keys in the repository's repos.yaml entry, carving a glob-covered repository into its own entry. Before any edit it refuses an entry that contradicts the committed trace_context or the key ID variable, a key ID variable that is not 16 lowercase hex characters, and a read-only manifest that must record a key. GitLab repositories are skipped without any read. convergeTraceSigningKey writes a plan before the scaffold or config commit. It reads FULLSEND_TRACE_SIGNING_KEY_ID again and stops unless its presence and value are unchanged, then deletes it, writes the seed and sets it to the new key ID, each step only after the one before succeeds. Nothing calls EnsureTraceSigningKeys yet. Signed-off-by: Dharit Shah <dhshah@redhat.com>
…ys in repos install repos install now plans trace signing keys after it records the install defaults and before it writes repos.yaml. It persists the trace_context edits with its other manifest edits, under the compare-before-write guard and the size limit, and convergence writes FULLSEND_TRACE_SIGNING_KEY and FULLSEND_TRACE_SIGNING_KEY_ID before the scaffold or config commit. - A fresh install from a writable repos.yaml gets a new installation ID and key pair. A later run writes a new pair, keeping the installation ID and retaining the previous public key, when the secret is missing or the key ID variable is missing or names a retained key. - --rotate-trace-signing-key writes a new pair for a healthy install and gives an existing install without trace_context its first keys. - Install stops before it writes repos.yaml or any key when an entry contradicts the committed trace_context or the key ID variable, when that variable is not a key ID, and when a read-only repos.yaml must record a key. From a read-only repos.yaml, a fresh install without the flag goes ahead without keys and says so. - The token-scope check and the sign-off identity lookup now run before install plans any repos.yaml edit, so a failure there leaves repos.yaml unchanged instead of recording a key that is never written. - Output shows installation and key IDs, never the seed. The progress line of the repos.yaml edit that copies a glob-covered repository into its own entry, whether the tracker default or the trace_context, names the new entry and the glob. When writing the seed or setting the key ID variable fails after the variable was deleted, the error says to keep and commit the repos.yaml the run wrote and to retry with it, since the seed may have been written. Repositories skipped for adoption get no key. GitLab repositories get no key yet, and install says so. - On GitHub, repos uninstall now deletes FULLSEND_TRACE_SIGNING_KEY_ID, so its Removed: line counts one more variable, and the orphan check counts the variable as managed. - repos uninstall --help says that only GitLab uninstall deletes .fullsend/config.yaml. BREAKING CHANGE: repos install now records each GitHub repository's trace signing keys as config.trace_context in its repos.yaml entry, on a fresh install from a writable repos.yaml and on the first --rotate-trace-signing-key run of an existing install. A repository that only a glob entry covers gets its own entry, copied from the glob, when that happens, so later edits to the glob entry do not reach it; edit its own entry instead. Keep and commit the repos.yaml edits install makes: a later install from a copy without them stops the run, except after a run that failed writing the seed or the key ID variable, when it replaces the key that run may have written without retaining it. If the copy with those edits is lost, rebuild the entry's trace_context from the block in the .fullsend/config.yaml that the last install committed. An entry that already declares trace_context by hand gets a new key pair, as a recovery, on its first install with this release; from a read-only repos.yaml that run stops. fullsend v0.37.0 through v0.55.0 refuse a repos.yaml that contains trace_context, and earlier releases cannot read this manifest format. v0.56.0 accepts it but does not manage FULLSEND_TRACE_SIGNING_KEY_ID: its repos status reports the variable as orphan drift and its repos uninstall leaves it, after which this release refuses the repository when it is declared again without keys, until the variable is deleted or the entry's trace_context is restored. Use a release with this change for every repos command on that manifest. Runs are unaffected: releases before v0.56.0 ignore the block in .fullsend/config.yaml, and v0.56.0 validates it. Signed-off-by: Dharit Shah <dhshah@redhat.com>
Document the key lifecycle in the trace context reference, the trace signing keys section and flag in the repos CLI reference, the trace_context config reference, the variable and secret inventories, the glob carve-out and repos.yaml write cases in the guides, and the parity paragraph in the Go contributor guide. GitLab keys, removing a retained key, keys from install flags and delivery to runs are marked planned. Signed-off-by: Dharit Shah <dhshah@redhat.com>
The ADR 0154 status note now says that repos install provisions the signing keys of GitHub repositories; the rest stays pending. Signed-off-by: Dharit Shah <dhshah@redhat.com>
PR Summary by QodoProvision and rotate GitHub trace signing keys during repos install
AI Description
Diagram
High-Level Assessment
Files changed (35)
|
|
🤖 Finished Review · ✅ Success · Started 1:17 PM UTC · Completed 1:28 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $4.20 |
Site previewPreview: https://2631e7af-site.fullsend-ai.workers.dev Commit: |
Code Review by Qodo
1. Overlapping installs can mismatch keys
|
| if exists != plan.KeyIDVarExists || value != plan.KeyIDVar { | ||
| return notSet(fmt.Sprintf("%s changed after this run read it, as when another install or uninstall of %s runs at the same time; "+ | ||
| "once that run has finished, run repos install again (after another install, with the repos.yaml that install wrote)", forge.VarTraceSigningKeyID, repoFullName)) | ||
| } | ||
| if err := client.DeleteRepoVariable(ctx, owner, repo, forge.VarTraceSigningKeyID); err != nil { |
There was a problem hiding this comment.
1. Overlapping installs can mismatch keys 🐞 Bug ☼ Reliability
convergeTraceSigningKey checks the key ID variable, then deletes and replaces the variable and seed through separate, unlocked API calls before the configuration commit. When two installs of the same repository use separate manifest copies, both can pass the check and interleave their writes and commits, leaving the committed public key different from the seed and key ID on GitHub.
Agent Prompt
## Issue description
Concurrent installs can pass the key-ID reread and interleave seed, variable, and configuration writes, leaving the committed verification key out of sync with the seed.
## Fix Focus Areas
- internal/repos/trace_keys.go[557-592]
- internal/repos/converge.go[1959-1966]
## Recommended Fix
Coordinate installs per repository across the key transition and configuration commit, and revalidate the current state while holding that coordination. Do not rely on the variable reread alone as a compare-and-swap.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| func EnsureTraceSigningKeys(ctx context.Context, cfg TraceKeysConfig, clients ForgeClientFactory, progress ProgressFunc) (map[string]*TraceKeyPlan, error) { | ||
| m := cfg.Manifest | ||
| selected, err := selectManifestRepos(ctx, m, clients, cfg.RepoFilter) |
There was a problem hiding this comment.
2. New glob-matched repos miss their keys 🐞 Bug ≡ Correctness
EnsureTraceSigningKeys builds key plans from the repositories returned by its preflight glob expansion, but convergence expands the glob again through the live client. If a repository appears in the organization listing between those expansions, convergence installs it without a key plan; later plain installs see its workflow and leave it keyless.
Agent Prompt
## Issue description
A glob-matched repository first discovered during convergence is installed without trace signing keys, and later plain installs do not provision them.
## Fix Focus Areas
- internal/repos/trace_keys.go[105-115]
- internal/repos/converge.go[800-817]
- internal/cli/repos.go[860-875]
## Recommended Fix
Use one stable selected repository set for key planning and convergence, or detect newly discovered GitHub repositories before installing them and plan and persist their keys under the same manifest-write safeguards. Do not install a newly discovered repository without its required key plan.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| if readOnly && !cfg.Rotate && t.reason == reasonFreshInstall { | ||
| t.reason, t.readOnlySkip = "", true | ||
| } |
There was a problem hiding this comment.
3. Fleet installs remain keyless 🔗 Cross-repo conflict ≡ Correctness
EnsureTraceSigningKeys skips fresh-install key provisioning when repos.yaml is read-only, and later plain installs do not provision an existing keyless repository. The .fullsend repository’s Sync Scaffold workflow always makes that manifest read-only and invokes repos install --direct without the rotation flag, so a GitHub repository it installs without keys stays keyless on subsequent syncs.
Agent Prompt
## Issue description
Sync Scaffold uses a read-only fleet manifest, so it cannot retain the new trace-key writeback; later plain syncs cannot provision a keyless install.
## Fix Focus Areas
- internal/repos/trace_keys.go[138-149]
- internal/repos/trace_keys.go[396-415]
- /cross_repos/.fullsend/.github/workflows/sync-scaffold.yml[82-91]
## Recommended Fix
Coordinate a durable manifest-update path with the `.fullsend` repository: record and commit generated trace_context entries before applying installs, and provide a one-time provisioning path for existing keyless repositories without rotating healthy keys on every scheduled sync.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
Risk Assessment: moderate (2/5) DetailsLarge diff across 35 files with moderate git churn and breaking manifest changes is balanced by strong test coverage, zero protected or security-sensitive paths, and close alignment with a planned slice of the linked issue. |
ReviewFindingsMedium
Low
Labels: PR changes the repos install CLI flow and Go code. Next steps:
|
| // Persisted rather than Would persist. Repositories that fail | ||
| // RequireInferenceAuth are skipped. Progress names installation and key IDs, | ||
| // and the glob a new entry was copied from, never the seed. | ||
| func EnsureTraceSigningKeys(ctx context.Context, cfg TraceKeysConfig, clients ForgeClientFactory, progress ProgressFunc) (map[string]*TraceKeyPlan, error) { |
There was a problem hiding this comment.
[low] pattern-inconsistency
EnsureTraceSigningKeys does not replace a nil progress callback with a no-op when it starts. Its siblings in internal/repos do: EnsureManagedConfigDefaults, RemoveFromManifest, Uninstall and Converge. If a future caller passes nil, the function panics at its first progress call. The only production caller (internal/cli/repos.go) passes a non-nil closure, so this is latent.
Suggested fix: Add 'if progress == nil { progress = func(_, _, _ string) {} }' at the top of EnsureTraceSigningKeys.
| // read returns the same. KeyIDVar is a key ID, never key material. | ||
| type TraceKeyPlan struct { | ||
| Key tracecontext.SigningKey | ||
| // Reason is why the key is planned: a fresh install, a recovery or a |
There was a problem hiding this comment.
[low] api-shape
The exported struct TraceKeyPlan has an exported field Reason, but its type traceKeyReason and the constants reasonFreshInstall, reasonRecovery and reasonRotation are unexported. Code outside package repos cannot name the type or compare against the constants.
Suggested fix: Either export the type and constants, or declare Reason as string.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Summary
Third slice of the trace signing key work in #9060 (ADR 0154), after #9321, #9322 and #9391.
fullsend repos installnow provisions and rotates the signing keys of GitHub repositories: the installation ID and public keys go in the repo'srepos.yamlentry asconfig.trace_context, the private seed inFULLSEND_TRACE_SIGNING_KEY, and its key ID in theFULLSEND_TRACE_SIGNING_KEY_IDvariable from the design note on #9060. Nothing reads the key at run time yet; GitLab keys, keys from install flags, removing a retained key and delivery to runs come later.Related Issue
Part of #9060
Changes
internal/repos/trace_keys.go:EnsureTraceSigningKeysplans each selected GitHub repo's key beforerepos.yamlis written (a fresh install; a recovery when the secret is missing or the key ID variable is absent or names a retained key; a rotation) and records the block in the repo's entry. Before any edit it refuses an entry that conflicts with the committedtrace_context, the variable or another entry's installation ID, a variable that is not a key ID, and a read-only manifest that must record a key.internal/repos/trace_keys.goandinternal/repos/converge.go: before the scaffold or config commit,convergeTraceSigningKeyreads the variable again and stops unless it is unchanged, then deletes the variable, writes the seed and sets the variable, each step only after the one before succeeds. A failed step stops that repo before its commit. When the seed or variable write fails, the error says to keep and commit therepos.yamlthe run wrote and to retry with it, since the secret may already hold the new seed.internal/repos/manifest_edit.goandinternal/repos/managed_config_writeback.go:carveOutForForgereturns the name of the glob it copied, in place of the forge name no caller used. The progress line of the edit that copies a glob-covered repo into its own entry, the first-installtrackerdefault ortrace_context, names the new entry and that glob.internal/cli/repos.go: install runs the key preflight before therepos.yamlwrite and persists its edits under the compare-before-write guard and the size limit, and adds--rotate-trace-signing-key. The token-scope check and the sign-off identity lookup move before anyrepos.yamledit, so a failure there leaves the file unchanged.repos uninstall --helpnow says only GitLab uninstall deletes.fullsend/config.yaml.internal/forge/forge.go,internal/repos/uninstall.goandinternal/repos/drift.go:forge.VarTraceSigningKeyIDnames the variable. On GitHub, uninstall deletes it and the orphan check counts it as managed.docs/reference/trace-context.md; indocs/cli/repos.md, the Trace signing keys section, the flag, the uninstall notes and which checks run before and after therepos.yamlwrite;docs/reference/config-reference.md; the variable and secret inventories; the glob carve-out notes in the getting-started guides;cli-internals.md; and the parity paragraph indocs/contributing/go-code.md. What telemetry: implement ADR 0154 trace continuity #9060 still owes is marked planned.docs/ADRs/0154-correlate-work-item-and-behaviour-scenario-traces.md) now saysrepos installprovisions the signing keys of GitHub repositories; the rest stays pending. Context, Decision and Consequences are unchanged.Behaviour notes
feat(#9060)!): install now writesconfig.trace_contextintorepos.yamlon a fresh GitHub install from a writable file and on the first--rotate-trace-signing-keyrun of an existing install. Keep and commit those edits: a later install from a copy without them stops the run, except after a run that failed writing the seed or the key ID variable, when it replaces the key that run may have written without retaining it. A lost entry can be rebuilt from the block in the committed.fullsend/config.yaml. An entry that declarestrace_contextby hand gets a new key pair, as a recovery, on its first install with this release; from a read-onlyrepos.yamlthat run stops.repos.yamlthat containstrace_context, and earlier releases cannot read this manifest format. v0.56.0 accepts it but does not manageFULLSEND_TRACE_SIGNING_KEY_ID: itsrepos statusreports the variable as orphan drift and itsrepos uninstallleaves it, after which this release refuses the repo when it is declared again without keys, until the variable is deleted or the entry'strace_contextis restored. Use a release with this change for everyreposcommand on such a manifest. Runs are unaffected: releases before v0.56.0 ignore the block in.fullsend/config.yaml, and v0.56.0 validates it.trackerdefault copied a glob-covered repo into its own entry. Now every GitHub repo that only a glob covers gets one when install first records its keys, and the output names the new entry and the glob. Later glob edits, such as afullsend_refpin,kill_switchor role and resource restrictions, no longer reach that repo. Fleet-wide controls belong ingithub.config,defaults.configorgithub.fullsend_ref, but a copied entry keeps the values it copied and they still apply, so a control moved off the glob also has to be deleted from each copied entry.Removed:line ofrepos uninstallcounts one more variable, since deleting a missing variable returns no error.FULLSEND_TRACE_SIGNING_KEY_IDonce per selected GitHub repo in the preflight and once more before each key write. It also checks thatFULLSEND_TRACE_SIGNING_KEYexists when the entry has keys, or when that decides whether a keyless repo was fully uninstalled.--rotate-trace-signing-keygives it keys.repos.yamlwrite: whether a selected repo uses each--openai-api-key,--vertex-project,--vertex-wif-providerand--jira-*value given, and the GCP project-number lookup that derives a repo's WIF provider. A failure there stops the run afterrepos.yamlrecorded any new key, so that key's seed is never written, and the next run keeps it as a retained key.Review questions
trace_context? Here only--rotate-trace-signing-keydoes. ADR 0154 says a standard install generates the pair, and the design note does not cover existing installs.Testing
make lintpasses:pre-commit run --from-ref b2945ba14 --to-ref HEADruns its hooks, semgrep and gitleaks included, over every changed file.internal/repos/trace_keys_test.go,internal/repos/trace_keys_converge_test.goandinternal/cli/repos_trace_keys_test.goare new. Of 41 mutants of the new code, 39 make a test fail. One survivor drops earlier failures from a failed key step's error; the other makes the same change on the fresh path, where the key step's error has no earlier failures to keep, so no test can catch it.go test -race -count=1over every package (internal/runtimein a separate run,internal/mintcorewith-tags github): 0 data races. On macOS, five tests fail, as they do on v0.56.0 (CA pool, temp path, two gateway-refresh timings, the arm64 install script). The twoTestRunAgenttests that call the live GitHub API (TestRunAgent_URLRefsNoOrgConfig calls api.github.com and hangs when the call is rate limited #9372) were skipped.golangci-lint run --new-from-rev b2945ba14 ./...reports 0 issues.pre-commitand the race tests ofinternal/repos,internal/forge/...,internal/scaffoldandinternal/cli, apart from the same five macOS failures.carveOutGlobEntry's return for a name no glob covers.Checklist
!for breaking changes)