Skip to content

fix: give acquiring runners the config the manager loop runs with - #1012

Merged
alongubkin merged 7 commits into
mainfrom
alon/alien-1463-cli-finished-updates-drop-manager-injected-env-vars
Oct 14, 2026
Merged

alongubkin merged 7 commits into
mainfrom
alon/alien-1463-cli-finished-updates-drop-manager-injected-env-vars

Conversation

@alongubkin

@alongubkin alongubkin commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Problem

When the manager starts an update and alien deploy takes the lease and finishes it, every container is rolled twice. The second rollout uses a different spec: the containers lose ALIEN_DEPLOYMENT_ID and ALIEN_DEPLOYMENT_NAME, and command-enabled containers and daemons also lose ALIEN_COMMANDS_*. They run without these variables until the manager runs another update.

For a single-replica database, that means two restarts back to back instead of one.

Root cause

Two runners built two different configs for the same operation:

  • Manager loop (loops/deployment.rs): builds an environment snapshot with the deployment id and name, the command receiver variables, and the deployment's own variables. It puts that snapshot in the config before it runs a step.
  • POST /v1/sync/acquire (routes/sync.rs): returned the record's raw deploymentConfig, so the control plane's copy of the variables went out without the manager's snapshot. In standalone mode it returned no config at all.
  • alien deploy and alien-deploy-cli up: both use the acquired config as-is (since fix(cli): hydrate acquired deployment configuration #380).

The executor sees a different desired config after the first rollout is ready, so it runs a second update.

The variables were not left out on purpose. Acquire already hands out the secret-bearing config: the record field is skip_serializing precisely so that acquire is the one route that returns it, and the acquire helper's doc says it exists for config fields redacted from GET /v1/deployments. Every value in the snapshot comes from data the caller already has, or is entitled to:

  • the deployment id and name;
  • the commands token, which is the deployment token the CLI is already authenticated with;
  • the deployment's variables, which are already on the acquired record.

A runner has to write all of these into the workloads anyway.

Fix

One shared builder, RunnerConfigSources::deployment_config, used by both the loop and the acquire route. It holds the code the loop already ran, moved out unchanged: the environment snapshot, the management config, monitoring, the native image host, and how a control plane's config gets merged. Target-stack resolution (supplied stack or release stack) moves into target_release_stack, which both paths also share.

  • Acquire now returns the config the loop would run with, in both platform and standalone mode. Neither CLI needed a change.
  • The BYO Horizon env-var bridge stays loop-only. Its management token belongs to the manager process, and handing it out to every acquirer would be a new exposure.
  • If the config can't be built (for example, a command-enabled resource with no deployment token), acquire releases the leases it just took and returns the error. Otherwise the caller would hold a lease it doesn't know about, and the manager would be blocked until it expired.
  • Small cleanup: a missing stack_settings is now an error rather than a panic. The tautological with_environment_snapshot test is replaced with one that runs the real builder.

Proof

  • tests/acquire_runner_config.rs: runs the CLI's real acquire helper (acquire_deployment_with_payload) against the real /v1/sync/acquire route over TCP, using sqlite stores. The deployment has an update queued to a release with a command-enabled container. The test decodes deploymentConfig the way alien deploy does and asserts the exact variable list: id, name, the four command variables scoped to the container, and the user variable.
    • Fails on main (no deploymentConfig in the response) and passes with the fix.
    • A second test checks that a config that can't be built fails the acquire with COMMAND_TOKEN_MISSING and leaves the deployment unlocked.
  • routes::sync::tests::acquired_config_is_the_config_the_manager_loop_runs_with: the reported case, where a control plane supplies the config and stacks. The acquired deploymentConfig equals the loop's config field for field; only the snapshot's createdAt may differ.
  • loops::deployment::tests::runner_config_replaces_control_plane_snapshot_with_manager_variables: the builder replaces the control plane's snapshot with the manager's variables.

Run in a remote sandbox (nightly-2026-10-03):

  • cargo nextest run -p alien-manager --lib --test acquire_runner_config: 334 passed, including the 4 new tests.
  • Same integration test with crates/alien-manager/src from main: both tests fail (acquire returns the config to run with panics because deploymentConfig is missing; the lease test fails because the acquire succeeds).
  • cargo clippy -p alien-manager --tests: no warnings in the changed files.

🤖 Generated with Claude Code

The manager loop builds the deployment's environment snapshot (deployment
id and name, command receiver variables, the deployment's own variables)
before it runs a step. The acquire route returned the raw record config
instead, so when the manager started an update and `alien deploy` took
the lease and finished it, the CLI applied a second, different container
spec: every workload rolled again and lost those variables.

Both paths now build the config with one helper, and acquire releases
the leases it took if that config cannot be built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@enclave-ai

enclave-ai Bot commented Oct 13, 2026

Copy link
Copy Markdown

Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin
alongubkin marked this pull request as ready for review October 13, 2026 16:27
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[High impact] The PR appears safe to merge; no blocking issue remains.

Summary

The manager loop and acquiring runners now share RunnerConfigSources::deployment_config and target-stack lookup.

  • Acquiring runners get the manager's deployment config.
  • Compute-planner test fixtures now set the encryption key.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Record[Deployment record] --> Stack[Shared target-stack lookup]
  Stack --> Builder[Shared runner-config builder]
  Builder --> Loop[Manager loop]
  Builder --> Acquire[Acquire response]
  Acquire --> Runner[CLI runner]
  Builder -->|Config build fails during acquire| Rollback[Return leases and restore retry requests]
Loading

Reviews (4) · Last reviewed commit: "test: set encryption_key in the compute ..." · Reviewed by Greptile

Comment thread crates/alien-manager/src/routes/sync.rs Outdated
Comment thread crates/alien-manager/tests/acquire_runner_config.rs Outdated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

alongubkin and others added 2 commits October 14, 2026 04:14
The acquire route gave back an unreturned lease by calling
set_retry_requested and then release. Two problems:

- a failed retry restore returned early, so the lease stayed held until
  it expired;
- against the platform store, set_retry_requested is the user-facing
  retry endpoint, while the platform derives the runner's retry request
  from the claimed operation and never consumes a stored flag. A plain
  release is the correct undo there.

DeploymentStore::release_unused_acquisition now owns the undo. The
default only releases. The SQLite store, whose acquire clears
retry_requested, restores it in the same UPDATE that clears the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inished-updates-drop-manager-injected-env-vars
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

Main's #1000 tests build PersistentStorage without the encryption_key
field #1005 added, so alien-core tests do not compile on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

alongubkin added a commit that referenced this pull request Oct 14, 2026
…key (#1019)

ALIEN-1477. alien-core tests don't compile on main (`d07764433`): #1000
added compute planner tests that build `PersistentStorage` without the
`encryption_key` field #1005 added. The two PRs merged one after the
other, so each passed CI alone. `cargo test -p alien-core` fails with
E0063 at `compute_planner.rs` 1312, 1462 and 1518. Main's Tests workflow
is skipped on push, so it only shows on PRs whose test selection
includes alien-core.

This sets `encryption_key: None` in the three fixtures. Test-only, no
behavior change.

Checked in a Linux sandbox on main (via #1012, which merges main) + this
commit: `cargo check --workspace --all-targets` is clean, `cargo nextest
run -p alien-core` 660/660 pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin
alongubkin merged commit 51d3f1c into main Oct 14, 2026
31 checks passed
@alongubkin
alongubkin deleted the alon/alien-1463-cli-finished-updates-drop-manager-injected-env-vars branch October 14, 2026 03:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant