Skip to content

feat(#9060)!: provision and rotate GitHub trace signing keys in repos install - #9477

Open
dhshah13 wants to merge 4 commits into
fullsend-ai:mainfrom
dhshah13:feat/github-trace-key-provisioning
Open

dhshah13 wants to merge 4 commits into
fullsend-ai:mainfrom
dhshah13:feat/github-trace-key-provisioning

Conversation

@dhshah13

Copy link
Copy Markdown
Contributor

Summary

Third slice of the trace signing key work in #9060 (ADR 0154), after #9321, #9322 and #9391. fullsend repos install now provisions and rotates the signing keys of GitHub repositories: the installation ID and public keys go in the repo's repos.yaml entry as config.trace_context, the private seed in FULLSEND_TRACE_SIGNING_KEY, and its key ID in the FULLSEND_TRACE_SIGNING_KEY_ID variable from the design note on #9060. Nothing reads the key at run time yet; GitLab keys, keys from install flags, removing a retained key and delivery to runs come later.

Related Issue

Part of #9060

Changes

  • internal/repos/trace_keys.go: EnsureTraceSigningKeys plans each selected GitHub repo's key before repos.yaml is written (a fresh install; a recovery when the secret is missing or the key ID variable is absent or names a retained key; a rotation) and records the block in the repo's entry. Before any edit it refuses an entry that conflicts with the committed trace_context, the variable or another entry's installation ID, a variable that is not a key ID, and a read-only manifest that must record a key.
  • internal/repos/trace_keys.go and internal/repos/converge.go: before the scaffold or config commit, convergeTraceSigningKey reads the variable again and stops unless it is unchanged, then deletes the variable, writes the seed and sets the variable, each step only after the one before succeeds. A failed step stops that repo before its commit. When the seed or variable write fails, the error says to keep and commit the repos.yaml the run wrote and to retry with it, since the secret may already hold the new seed.
  • internal/repos/manifest_edit.go and internal/repos/managed_config_writeback.go: carveOutForForge returns the name of the glob it copied, in place of the forge name no caller used. The progress line of the edit that copies a glob-covered repo into its own entry, the first-install tracker default or trace_context, names the new entry and that glob.
  • internal/cli/repos.go: install runs the key preflight before the repos.yaml write and persists its edits under the compare-before-write guard and the size limit, and adds --rotate-trace-signing-key. The token-scope check and the sign-off identity lookup move before any repos.yaml edit, so a failure there leaves the file unchanged. repos uninstall --help now says only GitLab uninstall deletes .fullsend/config.yaml.
  • internal/forge/forge.go, internal/repos/uninstall.go and internal/repos/drift.go: forge.VarTraceSigningKeyID names the variable. On GitHub, uninstall deletes it and the orphan check counts it as managed.
  • Docs: the key lifecycle in docs/reference/trace-context.md; in docs/cli/repos.md, the Trace signing keys section, the flag, the uninstall notes and which checks run before and after the repos.yaml write; docs/reference/config-reference.md; the variable and secret inventories; the glob carve-out notes in the getting-started guides; cli-internals.md; and the parity paragraph in docs/contributing/go-code.md. What telemetry: implement ADR 0154 trace continuity #9060 still owes is marked planned.
  • Accepted ADR edit: the status note of ADR 0154 (docs/ADRs/0154-correlate-work-item-and-behaviour-scenario-traces.md) now says repos install provisions the signing keys of GitHub repositories; the rest stays pending. Context, Decision and Consequences are unchanged.

Behaviour notes

  • Breaking (commit feat(#9060)!): install now writes config.trace_context into repos.yaml on a fresh GitHub install from a writable file and on the first --rotate-trace-signing-key run of an existing install. Keep and commit those edits: a later install from a copy without them stops the run, except after a run that failed writing the seed or the key ID variable, when it replaces the key that run may have written without retaining it. A lost entry can be rebuilt from the block in the committed .fullsend/config.yaml. An entry that declares trace_context by hand gets a new key pair, as a recovery, on its first install with this release; from a read-only repos.yaml that run stops.
  • Releases: v0.37.0 through v0.55.0 refuse a repos.yaml that contains trace_context, and earlier releases cannot read this manifest format. v0.56.0 accepts it but does not manage FULLSEND_TRACE_SIGNING_KEY_ID: its repos status reports the variable as orphan drift and its repos uninstall leaves it, after which this release refuses the repo when it is declared again without keys, until the variable is deleted or the entry's trace_context is restored. Use a release with this change for every repos command on such a manifest. Runs are unaffected: releases before v0.56.0 ignore the block in .fullsend/config.yaml, and v0.56.0 validates it.
  • Glob carve-outs: before, only the first-install tracker default copied a glob-covered repo into its own entry. Now every GitHub repo that only a glob covers gets one when install first records its keys, and the output names the new entry and the glob. Later glob edits, such as a fullsend_ref pin, kill_switch or role and resource restrictions, no longer reach that repo. Fleet-wide controls belong in github.config, defaults.config or github.fullsend_ref, but a copied entry keeps the values it copied and they still apply, so a control moved off the glob also has to be deleted from each copied entry.
  • On GitHub, the Removed: line of repos uninstall counts one more variable, since deleting a missing variable returns no error.
  • Install reads FULLSEND_TRACE_SIGNING_KEY_ID once per selected GitHub repo in the preflight and once more before each key write. It also checks that FULLSEND_TRACE_SIGNING_KEY exists when the entry has keys, or when that decides whether a keyless repo was fully uninstalled.
  • Install and dry run print one line per selected GitLab repo saying keys are not provisioned there yet, and write no key or variable there, even with the flag.
  • A dry run discards the keys it generates, so its new key IDs, and a fresh install's installation ID, differ from the live run's.
  • Known limit: a glob-covered GitHub repo created during a run, after the preflight listed the org, is installed without a key. Later plain runs leave it keyless; only --rotate-trace-signing-key gives it keys.
  • Known limit: some convergence input checks run after the repos.yaml write: whether a selected repo uses each --openai-api-key, --vertex-project, --vertex-wif-provider and --jira-* value given, and the GCP project-number lookup that derives a repo's WIF provider. A failure there stops the run after repos.yaml recorded any new key, so that key's seed is never written, and the next run keeps it as a retained key.

Review questions

  • Should a plain install give keys to an existing install whose entry has no trace_context? Here only --rotate-trace-signing-key does. ADR 0154 says a standard install generates the pair, and the design note does not cover existing installs.
  • If a flag stays, should that first provisioning use a flag named for provisioning rather than rotation? Renaming it or changing its meaning after a release is itself a breaking change.

Testing

  • make lint passes: pre-commit run --from-ref b2945ba14 --to-ref HEAD runs its hooks, semgrep and gitleaks included, over every changed file.
  • Tests added/updated for new or modified logic: internal/repos/trace_keys_test.go, internal/repos/trace_keys_converge_test.go and internal/cli/repos_trace_keys_test.go are new. Of 41 mutants of the new code, 39 make a test fail. One survivor drops earlier failures from a failed key step's error; the other makes the same change on the fresh path, where the key step's error has no earlier failures to keep, so no test can catch it.
  • go test -race -count=1 over every package (internal/runtime in a separate run, internal/mintcore with -tags github): 0 data races. On macOS, five tests fail, as they do on v0.56.0 (CA pool, temp path, two gateway-refresh timings, the arm64 install script). The two TestRunAgent tests that call the live GitHub API (TestRunAgent_URLRefsNoOrgConfig calls api.github.com and hangs when the call is rate limited #9372) were skipped. golangci-lint run --new-from-rev b2945ba14 ./... reports 0 issues.
  • Each of the 4 commits, checked out alone, builds, vets, and passes pre-commit and the race tests of internal/repos, internal/forge/..., internal/scaffold and internal/cli, apart from the same five macOS failures.
  • Patch coverage: locally, 373 of the 403 added executable Go lines run in the tests (92.6%). The 30 others are error-return lines and carveOutGlobEntry's return for a name no glob covers.

Checklist

  • PR title follows Conventional Commits (correct type, ! for breaking changes)
  • Commits are signed off (DCO) — human and human-directed agent sessions only
  • I wrote this contribution myself and can explain all changes in it

…step

EnsureTraceSigningKeys plans the trace signing key of each selected
GitHub repository before repos.yaml is written: a fresh install,
recovery when FULLSEND_TRACE_SIGNING_KEY is missing or
FULLSEND_TRACE_SIGNING_KEY_ID is absent or names a retained key, or a
rotation on request. It records the installation ID and public keys in
the repository's repos.yaml entry, carving a glob-covered repository
into its own entry. Before any edit it refuses an entry that
contradicts the committed trace_context or the key ID variable, a key
ID variable that is not 16 lowercase hex characters, and a read-only
manifest that must record a key. GitLab repositories are skipped
without any read.

convergeTraceSigningKey writes a plan before the scaffold or config
commit. It reads FULLSEND_TRACE_SIGNING_KEY_ID again and stops unless
its presence and value are unchanged, then deletes it, writes the seed
and sets it to the new key ID, each step only after the one before
succeeds. Nothing calls EnsureTraceSigningKeys yet.

Signed-off-by: Dharit Shah <dhshah@redhat.com>
…ys in repos install

repos install now plans trace signing keys after it records the
install defaults and before it writes repos.yaml. It persists the
trace_context edits with its other manifest edits, under the
compare-before-write guard and the size limit, and convergence writes
FULLSEND_TRACE_SIGNING_KEY and FULLSEND_TRACE_SIGNING_KEY_ID before the
scaffold or config commit.

- A fresh install from a writable repos.yaml gets a new installation ID
  and key pair. A later run writes a new pair, keeping the installation
  ID and retaining the previous public key, when the secret is missing
  or the key ID variable is missing or names a retained key.
- --rotate-trace-signing-key writes a new pair for a healthy install
  and gives an existing install without trace_context its first keys.
- Install stops before it writes repos.yaml or any key when an entry
  contradicts the committed trace_context or the key ID variable, when
  that variable is not a key ID, and when a read-only repos.yaml must
  record a key. From a read-only repos.yaml, a fresh install without
  the flag goes ahead without keys and says so.
- The token-scope check and the sign-off identity lookup now run before
  install plans any repos.yaml edit, so a failure there leaves
  repos.yaml unchanged instead of recording a key that is never
  written.
- Output shows installation and key IDs, never the seed. The progress
  line of the repos.yaml edit that copies a glob-covered repository
  into its own entry, whether the tracker default or the trace_context,
  names the new entry and the glob. When writing the seed or setting
  the key ID variable fails after the variable was deleted, the error
  says to keep and commit the repos.yaml the run wrote and to retry
  with it, since the seed may have been written. Repositories skipped
  for adoption get no key. GitLab repositories get no key yet, and
  install says so.
- On GitHub, repos uninstall now deletes FULLSEND_TRACE_SIGNING_KEY_ID,
  so its Removed: line counts one more variable, and the orphan check
  counts the variable as managed.
- repos uninstall --help says that only GitLab uninstall deletes
  .fullsend/config.yaml.

BREAKING CHANGE: repos install now records each GitHub repository's
trace signing keys as config.trace_context in its repos.yaml entry, on
a fresh install from a writable repos.yaml and on the first
--rotate-trace-signing-key run of an existing install. A repository
that only a glob entry covers gets its own entry, copied from the glob,
when that happens, so later edits to the glob entry do not reach it;
edit its own entry instead. Keep and commit the repos.yaml edits
install makes: a later install from a copy without them stops the run,
except after a run that failed writing the seed or the key ID
variable, when it replaces the key that run may have written without
retaining it. If the copy with those edits is lost, rebuild the
entry's trace_context from the block in the .fullsend/config.yaml that
the last install committed. An entry that already declares
trace_context by hand gets a new key pair, as a recovery, on its first
install with this release; from a read-only repos.yaml that run stops.
fullsend v0.37.0 through v0.55.0 refuse a repos.yaml that contains
trace_context, and earlier releases cannot read this manifest format.
v0.56.0 accepts it but does not manage FULLSEND_TRACE_SIGNING_KEY_ID:
its repos status reports the variable as orphan drift and its repos
uninstall leaves it, after which this release refuses the repository
when it is declared again without keys, until the variable is deleted
or the entry's trace_context is restored. Use a release with this
change for every repos command on that manifest. Runs are unaffected:
releases before v0.56.0 ignore the block in .fullsend/config.yaml, and
v0.56.0 validates it.

Signed-off-by: Dharit Shah <dhshah@redhat.com>
Document the key lifecycle in the trace context reference, the trace
signing keys section and flag in the repos CLI reference, the
trace_context config reference, the variable and secret inventories,
the glob carve-out and repos.yaml write cases in the guides, and the
parity paragraph in the Go contributor guide. GitLab keys, removing a
retained key, keys from install flags and delivery to runs are marked
planned.

Signed-off-by: Dharit Shah <dhshah@redhat.com>
The ADR 0154 status note now says that repos install provisions the
signing keys of GitHub repositories; the rest stays pending.

Signed-off-by: Dharit Shah <dhshah@redhat.com>
@dhshah13
dhshah13 requested a review from a team as a code owner November 14, 2026 13:14
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Provision and rotate GitHub trace signing keys during repos install

✨ Enhancement 🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Provision GitHub trace signing keys during install and retain prior public keys during rotation or
 recovery.
• Refuse conflicting key state before writes, and stop repository commits when key writes fail.
• Document the key lifecycle and cover installation, retry, uninstall, and drift behavior.
Diagram

graph TD
  CLI["Repos install"] --> PLAN["Key preflight"] --> MANIFEST["Manifest entry"] --> CONVERGE["Repo convergence"] --> SECRET["Signing secret"] --> VARIABLE["Key ID variable"] --> COMMIT["Config commit"]
  PLAN --> VARIABLE
  PLAN --> COMMIT
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Serialize installs with a repository lease
  • ➕ Could prevent overlapping installs from changing the seed after the key-ID recheck.
  • ➖ Requires a reliable lease and recovery protocol that this GitHub path does not currently have.
  • ➖ Adds operational complexity across install and uninstall.

Recommendation: Use the PR's preflight-and-convergence approach for this slice: it preserves the manifest as the public-key source of truth and limits partial-write damage without introducing a new coordination system. Its variable recheck is not a lock, so the documented one-install-at-a-time requirement remains important; consider a lease if concurrent installs must become safe.

Files changed (35) +3376 / -153

Enhancement (7) +746 / -37
repos.goWire trace-key planning and rotation into repos install +84/-30

Wire trace-key planning and rotation into repos install

• Adds the rotation flag, runs scope and sign-off checks before manifest edits, and persists key plans through the existing guarded write. Passes plans to convergence and corrects uninstall help text.

internal/cli/repos.go

forge.goName the trace signing key-ID variable +8/-0

Name the trace signing key-ID variable

• Adds a shared constant for the GitHub variable that identifies the last confirmed signing seed.

internal/forge/forge.go

converge.goWrite planned trace keys before repository commits +27/-0

Write planned trace keys before repository commits

• Carries preflight plans into both convergence paths and stops a repository before its scaffold or configuration commit if key convergence fails.

internal/repos/converge.go

drift.goTreat GitHub trace key IDs as managed variables +6/-0

Treat GitHub trace key IDs as managed variables

• Excludes the GitHub key-ID variable from orphan-variable reports without making it required.

internal/repos/drift.go

managed_config_writeback.goIdentify glob origins in manifest writeback progress +9/-6

Identify glob origins in manifest writeback progress

• Tracks the glob copied for a new repository entry and names both entries in tracker-default progress output.

internal/repos/managed_config_writeback.go

trace_keys.goPlan, validate, and converge GitHub trace signing keys +608/-0

Plan, validate, and converge GitHub trace signing keys

• Implements fresh provisioning, recovery, rotation, stale-state refusals, read-only handling, and glob carve-outs. Rechecks the key-ID variable before ordered variable deletion, seed write, and key-ID write, with failures preventing the repository commit.

internal/repos/trace_keys.go

uninstall.goDelete GitHub trace key IDs on uninstall +4/-1

Delete GitHub trace key IDs on uninstall

• Adds the key-ID variable to the GitHub uninstall cleanup list alongside the existing signing-secret cleanup.

internal/repos/uninstall.go

Refactor (1) +20 / -9
manifest_edit.goReturn the source glob when carving out an entry +20/-9

Return the source glob when carving out an entry

• Changes carve-out helpers to return the copied glob's name and adds a shared formatter for manifest-edit progress.

internal/repos/manifest_edit.go

Tests (16) +2127 / -55
repos_inference_auth_test.goAvoid printing secret records in auth assertions +1/-1

Avoid printing secret records in auth assertions

• Checks the count of secret writes without allowing their values into assertion output.

internal/cli/repos_inference_auth_test.go

repos_install_read_counts_test.goAccount for trace-key reads in install tests +19/-7

Account for trace-key reads in install tests

• Verifies the cached preflight read and live pre-write reread of the key-ID variable, including dry-run behavior.

internal/cli/repos_install_read_counts_test.go

repos_managed_config_test.goExpect trace_context in fresh managed overlays +15/-2

Expect trace_context in fresh managed overlays

• Checks that a fresh install records generated trace values and renders them beside the tracker default.

internal/cli/repos_managed_config_test.go

repos_manifest_write_test.goTest guarded manifest writes with generated keys +23/-7

Test guarded manifest writes with generated keys

• Checks that a competing manifest write prevents key writes, while retry provisions the selected repository and retains the other repository's edits.

internal/cli/repos_manifest_write_test.go

repos_safety_order_test.goGuard trace-key edits behind safety checks +21/-9

Guard trace-key edits behind safety checks

• Verifies rejected installs neither report persisted manifest edits nor write forge values, with secret-safe assertions.

internal/cli/repos_safety_order_test.go

repos_trace_keys_test.goExercise trace-key behavior through the install CLI +381/-0

Exercise trace-key behavior through the install CLI

• Covers flag wiring, dry runs, provisioning and rotation, failed-write retries, pre-write refusals, read-only manifests, adoption skips, and pending re-enrollment.

internal/cli/repos_trace_keys_test.go

drift_test.goTest forge-specific trace-key drift handling +49/-0

Test forge-specific trace-key drift handling

• Checks GitHub management versus GitLab orphan reporting and confirms neither trace credential is required.

internal/repos/drift_test.go

install_uninstall_parity_test.goInclude trace credentials in cleanup parity checks +15/-9

Include trace credentials in cleanup parity checks

• Adds the out-of-band GitHub seed and key-ID writes to install/uninstall parity fixtures.

internal/repos/install_uninstall_parity_test.go

managed_config_blocked_install_test.goKeep blocked-install secret assertions value-safe +1/-1

Keep blocked-install secret assertions value-safe

• Tests the absence of secret writes by count rather than printing records containing values.

internal/repos/managed_config_blocked_install_test.go

managed_config_writeback_test.goCheck glob carve-out progress +7/-1

Check glob carve-out progress

• Verifies tracker writeback identifies the new explicit entry and its source glob.

internal/repos/managed_config_writeback_test.go

manifest_signoff_test.goAdapt sign-off carve-out test to the new return value +1/-1

Adapt sign-off carve-out test to the new return value

• Keeps the sign-off copy test aligned with the helper's source-glob return value.

internal/repos/manifest_signoff_test.go

trace_context_helpers_test.goExtend deterministic trace-key test fixtures +29/-10

Extend deterministic trace-key test fixtures

• Adds seed and key-ID helpers and checks that errors never expose signing seeds or verification keys.

internal/repos/trace_context_helpers_test.go

trace_keys_converge_test.goTest ordered key writes and retry outcomes +716/-0

Test ordered key writes and retry outcomes

• Exercises dry runs, write-step failures, variable changes between preflight and convergence, commit failures, uninstall-only recovery, and full re-enrollment.

internal/repos/trace_keys_converge_test.go

trace_keys_test.goTest trace-key planning and preflight refusals +827/-0

Test trace-key planning and preflight refusals

• Covers fresh and healthy installs, rotation and recovery, glob entries, GitLab skips, read-only manifests, malformed variables, committed-key conflicts, and installation-ID collisions.

internal/repos/trace_keys_test.go

uninstall_test.goVerify trace credential cleanup on uninstall +21/-7

Verify trace credential cleanup on uninstall

• Checks GitHub key-ID deletion, preserves GitLab's distinct variable list, and updates expected deletion counts.

internal/repos/uninstall_test.go

scaffold_env_consistency_test.goRecognize the trace key-ID variable in environment checks +1/-0

Recognize the trace key-ID variable in environment checks

• Adds the new forge variable to the canonical names used by scaffold consistency tests.

internal/scaffold/scaffold_env_consistency_test.go

Documentation (11) +483 / -52
0154-correlate-work-item-and-behaviour-scenario-traces.mdMark GitHub key provisioning as implemented +6/-2

Mark GitHub key provisioning as implemented

• Updates ADR 0154's implementation status and identifies GitLab provisioning, key delivery, and runtime integration as pending.

docs/ADRs/0154-correlate-work-item-and-behaviour-scenario-traces.md

repos.mdDocument trace-key install and uninstall behavior +168/-17

Document trace-key install and uninstall behavior

• Explains provisioning, rotation, recovery, refusals, read-only manifests, concurrency limits, compatibility, and the new flag. Clarifies glob carve-outs and GitHub uninstall behavior.

docs/cli/repos.md

go-code.mdRecord trace-key install/uninstall parity obligations +8/-3

Record trace-key install/uninstall parity obligations

• Identifies the new out-of-band GitHub secret and variable writes and their required uninstall cleanup.

docs/contributing/go-code.md

cli-internals.mdList the trace-key rotation flag +1/-0

List the trace-key rotation flag

• Adds the one-run GitHub rotation control to the CLI command overview.

docs/guides/dev/cli-internals.md

migrating-from-github-setup.mdExplain migration-facing manifest edits +14/-5

Explain migration-facing manifest edits

• Lists the rotation control and explains when install creates explicit entries from globs or records trace_context.

docs/guides/getting-started/migrating-from-github-setup.md

operations.mdDescribe the GitHub trace secret and key-ID variable +5/-1

Describe the GitHub trace secret and key-ID variable

• Documents their formats, ownership, cleanup, and the fact that the seed is not yet delivered to runs.

docs/guides/getting-started/operations.md

repo-management.mdClarify read-only and glob-covered installs +13/-2

Clarify read-only and glob-covered installs

• Explains the trace-key exception for fresh installs from read-only manifests and when glob-covered repositories acquire explicit entries.

docs/guides/getting-started/repo-management.md

infrastructure-reference.mdAdd provisioned trace credentials to infrastructure reference +14/-4

Add provisioned trace credentials to infrastructure reference

• Describes the generated seed and key-ID variable while distinguishing provisioning from future runtime delivery.

docs/guides/infrastructure/infrastructure-reference.md

config-reference.mdUpdate trace_context configuration semantics +16/-10

Update trace_context configuration semantics

• Explains GitHub-generated keys, retained-key behavior, installation identity, and persistence in concrete manifest entries.

docs/reference/config-reference.md

trace-context.mdSpecify the GitHub signing-key lifecycle +234/-4

Specify the GitHub signing-key lifecycle

• Adds the provisioning and ordered-write contract, recovery guidance, stale-state refusals, rotation, re-enrollment, and concurrency limitations.

docs/reference/trace-context.md

preflight_read_cache.goDocument trace reads in the preflight cache +4/-4

Document trace reads in the preflight cache

• Clarifies that trace-key preflight shares cached read-only forge state before convergence resumes direct reads.

internal/repos/preflight_read_cache.go

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 1:17 PM UTC · Completed 1:28 PM UTC

Commit: 2fbfafc · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $4.20

@github-actions

Copy link
Copy Markdown

Site preview

Preview: https://2631e7af-site.fullsend-ai.workers.dev

Commit: 2fbfafc5f37b29a3007c2f43e967e395ae2245fc

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📎 Requirement gaps (0) 🔗 Cross-repo conflicts (1) 📜 Skill insights (1)

Grey Divider


Action required

1. Overlapping installs can mismatch keys 🐞 Bug ☼ Reliability
Description
convergeTraceSigningKey checks the key ID variable, then deletes and replaces the variable and
seed through separate, unlocked API calls before the configuration commit. When two installs of the
same repository use separate manifest copies, both can pass the check and interleave their writes
and commits, leaving the committed public key different from the seed and key ID on GitHub.
Code

internal/repos/trace_keys.go[R561-565]

+	if exists != plan.KeyIDVarExists || value != plan.KeyIDVar {
+		return notSet(fmt.Sprintf("%s changed after this run read it, as when another install or uninstall of %s runs at the same time; "+
+			"once that run has finished, run repos install again (after another install, with the repos.yaml that install wrote)", forge.VarTraceSigningKeyID, repoFullName))
+	}
+	if err := client.DeleteRepoVariable(ctx, owner, repo, forge.VarTraceSigningKeyID); err != nil {
Relevance

●● Moderate

Unlocked check-then-act across forge writes is architectural; no close precedent on locking or
concurrency in this PR's pattern.

PR-#8258
PR-#8015

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The reread and comparison precede three independent forge writes, with no synchronization between
them. Both convergence branches perform those writes before their scaffold or configuration commit,
permitting another run to overwrite the forge values before the first run commits its different
public key.

internal/repos/trace_keys.go[515-527]
internal/repos/trace_keys.go[557-592]
internal/repos/converge.go[1810-1819]
internal/repos/converge.go[1959-1966]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Concurrent installs can pass the key-ID reread and interleave seed, variable, and configuration writes, leaving the committed verification key out of sync with the seed.
## Fix Focus Areas
- internal/repos/trace_keys.go[557-592]
- internal/repos/converge.go[1959-1966]
## Recommended Fix
Coordinate installs per repository across the key transition and configuration commit, and revalidate the current state while holding that coordination. Do not rely on the variable reread alone as a compare-and-swap.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



Remediation recommended

2. New glob-matched repos miss their keys 🐞 Bug ≡ Correctness
Description
EnsureTraceSigningKeys builds key plans from the repositories returned by its preflight glob
expansion, but convergence expands the glob again through the live client. If a repository appears
in the organization listing between those expansions, convergence installs it without a key plan;
later plain installs see its workflow and leave it keyless.
Code

internal/repos/trace_keys.go[R105-107]

+func EnsureTraceSigningKeys(ctx context.Context, cfg TraceKeysConfig, clients ForgeClientFactory, progress ProgressFunc) (map[string]*TraceKeyPlan, error) {
+	m := cfg.Manifest
+	selected, err := selectManifestRepos(ctx, m, clients, cfg.RepoFilter)
Relevance

●● Moderate

Race between glob expansions is a semantic concurrency edge case; PR explicitly documents it as a
known limit, so no close precedent decides it.

PR-#8015
PR-#9322

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Preflight glob expansion lists organization repositories when building plans, while convergence
performs its own expansion. A repository absent from the first list has no entry in TraceKeyPlans,
so the key writer returns without action; once its workflow exists, the keyless-entry branch does
not plan a key on a plain retry.

internal/repos/trace_keys.go[105-125]
internal/repos/manifest.go[1155-1170]
internal/repos/converge.go[800-817]
internal/repos/trace_keys.go[530-536]
internal/repos/trace_keys.go[396-405]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A glob-matched repository first discovered during convergence is installed without trace signing keys, and later plain installs do not provision them.
## Fix Focus Areas
- internal/repos/trace_keys.go[105-115]
- internal/repos/converge.go[800-817]
- internal/cli/repos.go[860-875]
## Recommended Fix
Use one stable selected repository set for key planning and convergence, or detect newly discovered GitHub repositories before installing them and plan and persist their keys under the same manifest-write safeguards. Do not install a newly discovered repository without its required key plan.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


3. Fleet installs remain keyless 🔗 Cross-repo conflict ≡ Correctness
Description
EnsureTraceSigningKeys skips fresh-install key provisioning when repos.yaml is read-only, and
later plain installs do not provision an existing keyless repository. The .fullsend repository’s
Sync Scaffold workflow always makes that manifest read-only and invokes repos install --direct
without the rotation flag, so a GitHub repository it installs without keys stays keyless on
subsequent syncs.
Code

internal/repos/trace_keys.go[R140-142]

+		if readOnly && !cfg.Rotate && t.reason == reasonFreshInstall {
+			t.reason, t.readOnlySkip = "", true
+		}
Relevance

●● Moderate

Read-only manifest and fresh-install key gap is a semantic behaviour question with no close accepted
or rejected precedent.

PR-#8832
PR-#8331

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The workflow explicitly prevents manifest edits from surviving its run. The new read-only exception
discards a fresh key plan, while the keyless existing-install branch requires rotation, which this
workflow never requests.

fullsend -> .fullsend
internal/repos/trace_keys.go[138-149]
internal/repos/trace_keys.go[396-415]
External repo: fullsend-ai/.fullsend, .github/workflows/sync-scaffold.yml [68-91]
External repo: fullsend-ai/.fullsend, repos.yaml [1-16]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Sync Scaffold uses a read-only fleet manifest, so it cannot retain the new trace-key writeback; later plain syncs cannot provision a keyless install.
## Fix Focus Areas
- internal/repos/trace_keys.go[138-149]
- internal/repos/trace_keys.go[396-415]
- /cross_repos/.fullsend/.github/workflows/sync-scaffold.yml[82-91]
## Recommended Fix
Coordinate a durable manifest-update path with the `.fullsend` repository: record and commit generated trace_context entries before applying installs, and provide a one-time provisioning path for existing keyless repositories without rotating healthy keys on every scheduled sync.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗



Informational

4. ADR status claim has no pinned evidence 📜 Skill insight ≡ Correctness
Description
The ADR 0154 status note says repos install provisions GitHub signing keys but links only to the
trace-context reference, while the PR description names internal/repos/trace_keys.go and
EnsureTraceSigningKeys without an implementation commit SHA. Because the PR’s only SHA is the base
ref b2945ba14, readers cannot trace the claim to #9060 or this change, or verify it against a
fixed revision if the linked documentation or repos install later changes.
Code

docs/ADRs/0154-correlate-work-item-and-behaviour-scenario-traces.md[R25-28]

+defines its contract. `fullsend repos install` provisions the signing keys of
+GitHub repositories (see the
+[key lifecycle](../reference/trace-context.md#key-lifecycle)). Carrier
+publication, key provisioning on GitLab, the flags through which the
Relevance

● Weak

Recent rejections (#9182, #9186) of 'link/pin the implementing change beside a current-behavior
claim' in ADR docs match this finding closely.

PR-#9186
PR-#9182

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The added status note makes a current-behaviour claim about repos install and links to a
documentation anchor rather than the issue or PR that introduced the behaviour. The PR description
identifies the implementing file and function but provides no SHA for that implementation; its only
SHA is the base ref b2945ba14, which does not fix the claim to an implementing revision.

docs/ADRs/0154-correlate-work-item-and-behaviour-scenario-traces.md[25-28]
docs/ADRs/0154-correlate-work-item-and-behaviour-scenario-traces.md[25-27]
Skill: writing-adrs

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The ADR 0154 status note says that `repos install` provisions GitHub signing keys, but it does not link to the introducing issue or PR. The PR description identifies the implementing file and function without giving an implementation commit SHA.

## Fix Focus Areas
- docs/ADRs/0154-correlate-work-item-and-behaviour-scenario-traces.md[25-28]

## Recommended Fix
Add a link in the status sentence to the introducing work, such as https://github.com/fullsend-ai/fullsend/issues/9060 or this PR, alongside the existing key-lifecycle link. In the PR description, add evidence identifying `internal/repos/trace_keys.go`, `EnsureTraceSigningKeys`, and the full SHA of the commit containing the implementation; do not use `main` or the base ref.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


Grey Divider

Context sources
✅ Linked PRs:
  🌐 PR #9321
  🌐 PR #9322
  🌐 PR #9391
✅ Compliance rules (platform): 96 rules
✅ Skills: 5 invoked
  writing-user-docs
  writing-adrs
  check-patch-coverage
  cutting-releases
  adr-corner
✅ Cross-repo context — repo relationships
  Explored: repo: fullsend-ai/experiments (sha: d4292120) — View relationship
  Explored: repo: fullsend-ai/agents (sha: 5fa2e3a3) — View relationship
  Explored: repo: fullsend-ai/test-creator (sha: a5082d20) — View relationship
  Explored: repo: fullsend-ai/pi-anthropic-vertex (sha: d2ecd039) — View relationship
  Explored: repo: fullsend-ai/pi-xai-vertex (sha: a86139bf) — View relationship
  Explored: repo: fullsend-ai/.fullsend (sha: e6bfe5b1) — View relationship
  Explored: repo: fullsend-ai/metrics (sha: 56061ee1) — View relationship
✅ Context preparation
  Commit messages: summarized; full text retained for this review
Review mode: Auto: 🧠 Deep: Many independent stateful security-sensitive paths make missed defects plausible.

Grey Divider

Tip of the day
💡 Did you know, you can ask Qodo to dismiss a finding you disagree with, with your reason on record

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment on lines +561 to +565
if exists != plan.KeyIDVarExists || value != plan.KeyIDVar {
return notSet(fmt.Sprintf("%s changed after this run read it, as when another install or uninstall of %s runs at the same time; "+
"once that run has finished, run repos install again (after another install, with the repos.yaml that install wrote)", forge.VarTraceSigningKeyID, repoFullName))
}
if err := client.DeleteRepoVariable(ctx, owner, repo, forge.VarTraceSigningKeyID); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Overlapping installs can mismatch keys 🐞 Bug ☼ Reliability

convergeTraceSigningKey checks the key ID variable, then deletes and replaces the variable and
seed through separate, unlocked API calls before the configuration commit. When two installs of the
same repository use separate manifest copies, both can pass the check and interleave their writes
and commits, leaving the committed public key different from the seed and key ID on GitHub.
Agent Prompt
## Issue description
Concurrent installs can pass the key-ID reread and interleave seed, variable, and configuration writes, leaving the committed verification key out of sync with the seed.
## Fix Focus Areas
- internal/repos/trace_keys.go[557-592]
- internal/repos/converge.go[1959-1966]
## Recommended Fix
Coordinate installs per repository across the key transition and configuration commit, and revalidate the current state while holding that coordination. Do not rely on the variable reread alone as a compare-and-swap.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

Comment on lines +105 to +107
func EnsureTraceSigningKeys(ctx context.Context, cfg TraceKeysConfig, clients ForgeClientFactory, progress ProgressFunc) (map[string]*TraceKeyPlan, error) {
m := cfg.Manifest
selected, err := selectManifestRepos(ctx, m, clients, cfg.RepoFilter)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. New glob-matched repos miss their keys 🐞 Bug ≡ Correctness

EnsureTraceSigningKeys builds key plans from the repositories returned by its preflight glob
expansion, but convergence expands the glob again through the live client. If a repository appears
in the organization listing between those expansions, convergence installs it without a key plan;
later plain installs see its workflow and leave it keyless.
Agent Prompt
## Issue description
A glob-matched repository first discovered during convergence is installed without trace signing keys, and later plain installs do not provision them.
## Fix Focus Areas
- internal/repos/trace_keys.go[105-115]
- internal/repos/converge.go[800-817]
- internal/cli/repos.go[860-875]
## Recommended Fix
Use one stable selected repository set for key planning and convergence, or detect newly discovered GitHub repositories before installing them and plan and persist their keys under the same manifest-write safeguards. Do not install a newly discovered repository without its required key plan.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

Comment on lines +140 to +142
if readOnly && !cfg.Rotate && t.reason == reasonFreshInstall {
t.reason, t.readOnlySkip = "", true
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

3. Fleet installs remain keyless 🔗 Cross-repo conflict ≡ Correctness

EnsureTraceSigningKeys skips fresh-install key provisioning when repos.yaml is read-only, and
later plain installs do not provision an existing keyless repository. The .fullsend repository’s
Sync Scaffold workflow always makes that manifest read-only and invokes repos install --direct
without the rotation flag, so a GitHub repository it installs without keys stays keyless on
subsequent syncs.
Agent Prompt
## Issue description
Sync Scaffold uses a read-only fleet manifest, so it cannot retain the new trace-key writeback; later plain syncs cannot provision a keyless install.
## Fix Focus Areas
- internal/repos/trace_keys.go[138-149]
- internal/repos/trace_keys.go[396-415]
- /cross_repos/.fullsend/.github/workflows/sync-scaffold.yml[82-91]
## Recommended Fix
Coordinate a durable manifest-update path with the `.fullsend` repository: record and commit generated trace_context entries before applying installs, and provide a one-time provisioning path for existing keyless repositories without rotating healthy keys on every scheduled sync.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Nov 14, 2026
@fullsend-ai-review

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Large diff across 35 files with moderate git churn and breaking manifest changes is balanced by strong test coverage, zero protected or security-sensitive paths, and close alignment with a planned slice of the linked issue.

@fullsend-ai-review

Copy link
Copy Markdown

Review

Findings

Medium

  • [backward-incompatible] internal/repos/manifest_edit.go — Any install that records a trace key now copies a glob-only repository into its own explicit repos.yaml entry (via carveOutForForge). Later edits to the glob (fullsend_ref pin, kill_switch, roles, allowed_remote_resources) then stop applying to that repository. The PR discloses this and names the glob in the progress line, but a copied entry silently stops following a fleet-wide kill_switch.
    Remediation: Consider copying only the trace_context block, or warning at install time when the glob entry carries fleet-control fields. This is a maintainer design decision.

Low

  • [pattern-inconsistency] internal/repos/trace_keys.go:105 — EnsureTraceSigningKeys does not replace a nil progress callback with a no-op; its siblings (EnsureManagedConfigDefaults, RemoveFromManifest, Uninstall, Converge) do. The only production caller passes a non-nil closure, so this is latent.
    Remediation: Add if progress == nil { progress = func(_, _, _ string) {} } at the top of the function.
  • [api-shape] internal/repos/trace_keys.go:50 — Exported TraceKeyPlan.Reason has the unexported type traceKeyReason (constants also unexported), so code outside repos cannot name or compare it.
    Remediation: Export the type and constants, or declare Reason as string.
  • [backward-incompatible] internal/cli/repos.go — A plain repos install now adds config.trace_context to repos.yaml. v0.37.0–v0.55.0 refuse such a manifest; v0.56.0 accepts it but reports FULLSEND_TRACE_SIGNING_KEY_ID as orphan drift and leaves it on uninstall. Declared via ! and the PR body.
    Remediation: Keep the ! and call it out in the release notes.
  • [backward-incompatible] internal/repos/trace_keys.go — An entry that already declares trace_context by hand gets a new key pair (recovery) on its first install when the secret is missing or the key ID variable is missing or names a retained key; the declared key moves to retained, and a read-only manifest stops the run. Disclosed in the PR body.
    Remediation: Call this out in the release notes.
  • [breaking-cli] internal/cli/repos.go — --rotate-trace-signing-key also provisions the first keys for an existing install while a plain install does not; renaming it after release would itself be breaking. The PR lists this as an open review question.
    Remediation: Settle the flag name and provisioning semantics before release.

Labels: PR changes the repos install CLI flow and Go code.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

@fullsend-ai-review fullsend-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the review comment for full details.

// Persisted rather than Would persist. Repositories that fail
// RequireInferenceAuth are skipped. Progress names installation and key IDs,
// and the glob a new entry was copied from, never the seed.
func EnsureTraceSigningKeys(ctx context.Context, cfg TraceKeysConfig, clients ForgeClientFactory, progress ProgressFunc) (map[string]*TraceKeyPlan, error) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] pattern-inconsistency

EnsureTraceSigningKeys does not replace a nil progress callback with a no-op when it starts. Its siblings in internal/repos do: EnsureManagedConfigDefaults, RemoveFromManifest, Uninstall and Converge. If a future caller passes nil, the function panics at its first progress call. The only production caller (internal/cli/repos.go) passes a non-nil closure, so this is latent.

Suggested fix: Add 'if progress == nil { progress = func(_, _, _ string) {} }' at the top of EnsureTraceSigningKeys.

// read returns the same. KeyIDVar is a key ID, never key material.
type TraceKeyPlan struct {
Key tracecontext.SigningKey
// Reason is why the key is planned: a fresh install, a recovery or a

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] api-shape

The exported struct TraceKeyPlan has an exported field Reason, but its type traceKeyReason and the constants reasonFreshInstall, reasonRecovery and reasonRotation are unexported. Code outside package repos cannot name the type or compare against the constants.

Suggested fix: Either export the type and constants, or declare Reason as string.

@fullsend-ai-review fullsend-ai-review Bot added component/install CLI install and app setup go Pull requests that update go code labels Nov 14, 2026
@codecov

codecov Bot commented Nov 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.47727% with 30 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/repos/trace_keys.go 90.84% 13 Missing and 13 partials ⚠️
internal/cli/repos.go 95.00% 1 Missing and 1 partial ⚠️
internal/repos/manifest_edit.go 80.00% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

This branch was successfully deployed

1 active deployment
site-preview — 2fbfafc5 Deployed Nov 14, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/install CLI install and app setup go Pull requests that update go code risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant