[eipasteur]

In short

Stacked on #614 (feat/agentcore-engine-data).

Moves the runtime's remaining credential and provider access behind the control-plane grant, retires grants as soon as their use ends, and lets customer MCP servers run as Gateway targets so their credentials never enter the agent's microVM. Active with enable_engine_data; strict enforcement is a separate flag (enforce_runtime_grants, default false); keeping custom MCP out of the runtime is a third, opt-in profile (runtime_mcp_credentials = "gateway_only").

Grants and the broker

• Git credentials are issued by the credential broker only against a valid Gateway grant for an execution that includes the repository.
• Invocation accounting. The broker records each agent invocation before returning credential material. Begins are capped per credential grant, and invocations whose ticket has expired no longer count in the credential inventory.
• Custom MCP secrets are resolved by the broker from the execution record (or a short-lived verification grant for verify-mcp) instead of the runtime reading SSM.
• Source-control runtime alias. Provider review operations (default branch, review status, set to draft) go to an alias of the source-control Lambda that requires a grant, accepts only those operations, and acts only on review numbers the execution's own unit review records list. Control-plane callers keep the unqualified function.
• Transition flag. While enforce_runtime_grants is off, requests that carry any grant field are verified exactly as under enforcement, and requests in the old shape (runtimes started before the flag) are served the old way and logged as LEGACY_RUNTIME_REQUEST. Turn it on once that count is zero; the old shapes are then denied.
• Grant retirement. Each grant is retired when its use ends: stage grants when the stage settles, synchronous engine-command grants when the command returns, quorum-edit grants after their callbacks. Cancel, rewind, recompose, repair, delete and the durable watchdog retire the whole execution, conditioned on the run they observed so a stale request cannot revoke a newer run. Every consumer rejects a retired grant and fails closed if the retirement table cannot be read: the Gateway interceptor (feat(agentcore): serve the built-in MCP through AgentCore Gateway (opt-in) #613) checks retirement on every request before the target is invoked, and engine-data, the broker and the alias check it on their direct calls. Retirement writes are retried; a grant that cannot be retired fails its stage or run and blocks any relaunch until it is. The runtime role has no access to the retirement table.
• The interceptor serves only MCP grant modes (stage, discussion, conflict). An engine-data execution grant is refused at the Gateway.
• Compose jobs run in the background, so their grant is bounded to 15 minutes and retired at once if dispatch fails.
• Bedrock IAM renewal tokens are bound to the grant that authorized the first handoff: a renewal is refused once that grant is retired or the execution is cancelled. A grant whose retirement could not be written yet (recorded on the execution) is refused by every consumer.

Two small fixes ride along, found while testing (each its own commit): a Bedrock IAM renewal no longer requests an STS session past the signed renewal deadline, and a cancel that races the durable watchdog now completes instead of returning 500.

The HTTP retirement helpers live in lambda/intents/grant-retirement.js; the recovery contract (failure, persisted state, response, retry) is documented in docs/development/recovery-ownership.md.

Customer MCP servers as Gateway targets

• mcp_gateway_custom_targets (default {}, requires enable_mcp_gateway) declares one remote MCP target per entry: the HTTPS endpoint, an AgentCore Identity credential provider (OAuth client-credentials or API key) created out of band and referenced by ARN together with its secret ARN, an optional VPC Lattice private endpoint, and an allow rule by space, stage and tool. No secret enters Terraform state. The Gateway role gets GetWorkloadAccessToken, GetResourceOauth2Token or GetResourceApiKey, and secretsmanager:GetSecretValue on exactly the configured providers. The interceptor receives the allow rules only (no endpoint, no credential, bounded to 2 KB); the runtime learns only that customer targets exist.
• The target name is the map key and is immutable: agents and allow rules refer to tools through it, so renaming creates a different target. Widening an allow rule or rebinding a target also applies to running stages, against the rule that a running execution keeps the configuration it started with; let running intents finish or restart them before applying such a change. Narrowing is a revocation: once the apply has updated the interceptor, the next call of a running stage is denied and the tools leave the next tools/list. Rotation happens in AgentCore Identity.
• The interceptor authorizes customer targets from the signed grant only. Only stage grants reach a customer target; the grant's space, stage and the tool must match the allow rule. Role, persona and agentRef are never read. An allowed call carries x-aidlc-caller, a versioned envelope of grant-derived fields (space, intent, execution, stage, unit, grant id, expiry), and never Authorization: the provider's credential is the one the server receives. On every other request the interceptor allows (initialize, ping, tools/list, notifications, and calls to the aidlc target) the header is set to a fixed {"v": 1, "none": true} value, so a caller-supplied header never reaches a server. tools/list filters customer tools with the same predicate and keeps pagination. A malformed registry denies every request.
• Passthrough proxy. When customer targets exist, stage and persona sessions get one runtime-owned aidlc-gateway-tools entry for Kiro, Claude, Codex and OpenCode. It lists the Gateway's tools page by page without the aidlc___ ones and forwards calls and results unchanged under their qualified names (<target>___<tool>), with the same grant and session as the built-in proxy. Discussion and conflict sessions get no entry. The entry carries no endpoint, grant or credential.
• The contract a customer server must follow (authenticate the Gateway's credential, read x-aidlc-caller for scoping, trust it only on an authenticated request, treat the none value as no caller) is in docs/using-the-platform/customer-mcp-servers.md.

runtime_mcp_credentials = "gateway_only"

Opt-in profile (default "allow", unchanged). It requires enable_mcp_gateway, enable_engine_data and enforce_runtime_grants, and sets RUNTIME_MCP_CREDENTIALS on the broker, projects, intents, orchestrator, agents and runtime (managed environments inherit it). One helper, lambda/shared/runtime-mcp-policy.js; an unknown value fails loudly at cold start.

• Direct custom MCP servers are refused wherever they are written or used: the space and platform MCP editors reject stdio and remote entries and secret values before any setting is written (reads and clears keep working, the frontend explains the profile); verify-mcp refuses before a verification grant is issued; intent launch re-reads the pinned snapshot and refuses to start while direct servers are configured; the orchestrator fails such a stage inside the existing run-<attempt> dispatch step (no step renamed or added), so it never reaches the runtime; the broker refuses MCP secret resolution for both authorities before any read; run-stage fails the stage before secret resolution or launch as defense in depth.
• engine-data strips the stored direct MCP configuration from every runtime-facing META response (getExecution, getExecutionRecords, and the META createExecution and updateExecution return); the record itself is untouched.
• The CLIs load only the platform's MCP config: Claude --strict-mcp-config, Kiro includeMcpJson: false, Codex with the workspace and its ancestors marked untrusted, OpenCode with every server another layer defines written as a disabled stand-in. A sentinel secret is shown never to reach dispatch payloads, the child env or the process store.
• The runtime role loses its MCP secret read and gets explicit Denies on the MCP secret parameters, AgentCore Identity tokens and API keys, and the customer providers' secrets.

Scope and known limits

• Git credentials returned to the runtime are still provider tokens: an OAuth token is not narrowed to the repository, and a GitHub App token carries repository-wide write. Per-intent Git isolation needs provider operations outside the agent VM; branch protection limits the impact meanwhile.
• Still reaching the VM under gateway_only: Git provider tokens during the engine's push and fetch windows, and each CLI's model credentials. The profile isolates the agent's MCP tools; it does not replace network egress control, since code the agent runs (a shell command, curl, a script in the repository) can still open connections from the microVM. Restricting the runtime's egress is the next hardening step.
• Background compose and discussion grants are not observed to completion; they expire with their TTL or are retired by a stop.
• Sessions already running when the first customer target is added do not get the aidlc-gateway-tools entry; it is written when a stage session starts on the updated runtime.
• The runtime role's direct permissions are removed in the next PR of the stack (feat(terraform): remove the runtime role's direct data access under enforced runtime grants #616).

Validation

• Rebased on main. On the stack tip: npm test 7349 passed, 2 todo; test:release 172 passed (including the runtime_mcp_credentials wiring test); terraform test 49/49 (mcp-gateway-custom-targets.tftest.hcl covers the targets, the provider permissions, the registry bound, the profile's prerequisites and the runtime role's Denies); lint, format:check, dep:check, secretlint and sdk:check pass. The suite includes a real durable-runner test of retirement retries and replay, broker handler tests in transition and strict modes, concurrency cases (stale rewind, cancel/watchdog race, same-nonce begins at the cap), static IAM assertions for the retirement table, the interceptor's customer-target and retirement suites, and the CLI isolation tests against the generated configs.
• Not yet verified on a deployed Gateway: that x-aidlc-caller and the Identity-injected credential reach the target and that a caller-supplied header is overwritten; tool naming and sync (<target>___<tool> in tools/list, filtered out for another space); the credential provider ARN service name the Identity API returns (acps versus bedrock-agentcore, both accepted by validation); revocation latency after narrowing an allow rule; and the CLI isolation flags against the real Claude, Kiro, Codex and OpenCode binaries.

Upgrade notes

Requires enable_engine_data (and therefore enable_mcp_gateway). Creates the grant-retirement table (TTL, not deletion-protected: entries are short-lived), the source-control runtime alias, and broker/consumer IAM. Enable enforce_runtime_grants only after LEGACY_RUNTIME_REQUEST has drained.

Customer targets and gateway_only are off by default. Adding a target creates it and the Gateway role's provider permissions; the interceptor picks up the allow rules on the same apply. Turning on gateway_only adds the runtime role Denies and refuses existing direct custom MCP entries at the next intent launch (remove them in the space or platform settings first; a snapshot from which they were removed is repaired on launch).

[jeromevdl]

I examined the size and coupling of #615 and found no material defect requiring changes. Although broker authorization and grant retirement could be separate PRs, splitting now would mainly reorganize already-reviewed work. I would keep the PR together and clarify the recovery contract, rather than require stack surgery without a demonstrated engineering benefit.

Two improvements for maintainability and doc:

1. Document the grant-retirement recovery contract
The PR adds recovery behavior that operators currently need to reconstruct from implementation comments. Please extend
docs/development/recovery-ownership.md to explain:

• How failed retirement writes produce pendingGrantRetirements, and how consumers treat those grants.
• What must succeed before start, rewind, recompose, or repair can relaunch an execution.
• What the repairRelaunch cleanup/relaunch phases mean and how retries resume them.
• Which retirement failures return 503 before an operation changes state, versus after a synchronous command has already acted.
• The supported retry procedure and relevant failure log codes.

A compact table covering failure → persisted state → response → retry behavior would be sufficient.

2. Give HTTP retirement policy a clearer owner
In lambda/intents/index.js, the new helpers now handle retirement retries, pending-grant cleanup, relaunch guards, and HTTP refusal responses alongside the endpoint implementations.

Consider moving the existing helpers—particularly retireExecutionGrantsOrRefuse, retirePendingGrantsOrRefuse, relaunchWithoutPendingGrants, and issueSyncGatewayGrant—into a small module within lambda/intents/.

Keep endpoint authorization and route-specific state transitions in the handlers. Preserve the existing conditional writes, retry behavior, and response semantics. Keep the durable orchestrator adapter separate because it has different execution and retry requirements; no generic lifecycle framework is needed.

[eipasteur]

Both done. docs/development/recovery-ownership.md gains the retirement contract and a failure → persisted state → response → retry table, plus the repairRelaunch phases. The HTTP helpers moved to lambda/intents/grant-retirement.js (a factory over ddb/store/logger/issuer; the pure response builders at module scope). Conditional writes, retries and responses are unchanged, and the orchestrator adapter is untouched.

[github-actions]

Complexity and test coverage

Automated quality report using code complexity and test coverage to highlight potential hot spots in this PR.
Review 🟣 and 🔴 items first: consider simplifying the code, improving test coverage, or both. This report is advisory.

	File / function	Complexity	Δ	Branch coverage	Δ	Uncovered branches	Depending files
🔴	lambda/v2-orchestrator/quorum-edit.js:61 — runQuorumEdit	21 → 32	+11	60.2% → 57.9%	-2.3	35 → 48	1
🔴	lambda/credential-broker/mcp-secrets.js:165 — authorizeMcpSecretRequest	— → 85	—	— → 96.3%	—	— → 2	1
🔴	lambda/agentcore/auth-resolver.js:130 — resolveInvocationAgentAuth	16 → 29	+13	95.2% → 96.4%	+1.2	2 → 2	1
🟠	lambda/intents/index.js:1785 — handler	1904 → 1994	+90	77.1% → 77.2%	+0.1	405 → 419	0
🟠	lambda/v2-orchestrator/section.js:1715 — runParallelSection / runLaneBody	51 → 56	+5	79.6% → 79.4%	-0.2	20 → 21	1
🟠	lambda/agentcore/commands/run-stage.js:1739 — runStageAttempt	871 → 872	+1	85.5% → 85.5%	0	132 → 132	1
🟠	lambda/agents/index.js:469 — handler	301 → 311	+10	56.1% → 64.8%	+8.7	130 → 107	0
🟠	lambda/source-control/index.js:443 — handler	69 → 75	+6	0.0% → 16.9%	+16.9	79 → 69	0
🟠	lambda/v2-orchestrator/index.js:418 — handler	362 → 371	+9	86.7% → 87.2%	+0.5	45 → 45	0
🟠	lambda/shared/v2-process-store.js:532 — createProcessStore / updateExecution	131 → 134	+3	91.6% → 91.9%	+0.3	19 → 19	10

Show 7 more functions

	File / function	Complexity	Δ	Branch coverage	Δ	Uncovered branches	Depending files
🟠	lambda/shared/gateway-grant.js:229 — verifyGatewayGrant	17 → 22	+5	83.0% → 86.2%	+3.2	9 → 8	14
🟠	lambda/shared/invocation-accounting.js:33 — beginCredentialInvocation	— → 19	—	— → 84.2%	—	— → 6	2
🟠	lambda/v2-orchestrator/index.js:2913 — runStage / <anonymous>	— → 18	—	— → 86.7%	—	— → 6	0
🟠	lambda/v2-orchestrator/index.js:1013 — handler / executeStage	42 → 44	+2	91.7% → 92.1%	+0.4	6 → 6	0
🟠	lambda/credential-broker/invocation-accounting.js:50 — beginBrokerInvocation	— → 19	—	— → 94.7%	—	— → 2	2
🟠	lambda/shared/gateway-grant-retirement.js:185 — retireGatewayExecution	— → 17	—	— → 90.0%	—	— → 2	9
🟠	lambda/credential-broker/agent-authentication.js:35 — authorizeAgentCredentialRequest	8 → 16	+8	79.2% → 94.3%	+15.1	5 → 2	1

🟢 Improving · 🟠 Needs attention · 🔴 High risk · 🟣 Highest risk

— = unavailable · n/a = not applicable · * = incomplete tests

CI run and full report artifacts

[JWThewes]

Please see the 2 inline finding(s) below.

[JWThewes]

[P1] Fence issuance before the final retirement snapshot

Rewind and recompose retire existing grants before replacing the old run ID, with artifact archival between those operations. A concurrent gate answer can still pass its run-ID CAS, resume the old durable execution, and obtain a new grant issued after retiredAt. Broker and source-control authorization do not compare that grant's run ID with the current execution, so the old runtime can continue obtaining credentials or performing allowed review operations after relaunch.

Suggested fix: After fencing and stopping the old run, perform another execution retirement conditioned on the retired run ID before relaunch, as lane repair does, or atomically combine the run fence and retirement. Add a concurrent gate-resume and grant-issuance regression.

Evidence: WAITING remains rewindable and recomposable, and the answer route conditions on the still-current run ID. The run ID is not replaced until lines 5289 and 5640, after the initial retirement and archival. Resumed runStage legs receive a new jti. isGatewayGrantRetired accepts a grant issued after retiredAt, while assertGatewayExecution and assertEngineGrantFor do not reject an old orchestratorRunId. This is distinct from the supplied recovery-documentation and helper-extraction feedback.

[JWThewes]

[P2] Expire abandoned invocation accounting records

Active invocation rows are created without agentAuthTtl, so their inventory references also have no TTL. If a runtime dies or completion cannot be recorded, the invocation and references remain permanently even though material inventory merely filters them after the 24-hour ticket lifetime.

Suggested fix: Set agentAuthTtl when creating an invocation to a deadline beyond the accounting ticket lifetime so the invocation and generated references eventually expire. Add coverage for an abandoned active invocation and its inventory references.

Evidence: The invocation object still contains state, startedAt, and heartbeatAt but no agentAuthTtl. finishCredentialInvocation adds that attribute only on successful completion. The process table enables TTL on agentAuthTtl, while isStaleInvocation only filters expired active rows and loadInventory still queries every reference and reads its target. The supplied complexity report names this function but does not report this retention defect.