Repository navigation
fix: give acquiring runners the config the manager loop runs with - #1012
Merged
alongubkin merged 7 commits intoOct 14, 2026
Merged
alongubkin merged 7 commits into
alongubkin merged 7 commits into
Conversation
The manager loop builds the deployment's environment snapshot (deployment id and name, command receiver variables, the deployment's own variables) before it runs a step. The acquire route returned the raw record config instead, so when the manager started an update and `alien deploy` took the lease and finished it, the CLI applied a second, different container spec: every workload rolled again and lost those variables. Both paths now build the config with one helper, and acquire releases the leases it took if that config cannot be built. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alongubkin
marked this pull request as ready for review
October 13, 2026 16:27
Member
Author
|
@greptileai review |
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
@greptileai review |
The acquire route gave back an unreturned lease by calling set_retry_requested and then release. Two problems: - a failed retry restore returned early, so the lease stayed held until it expired; - against the platform store, set_retry_requested is the user-facing retry endpoint, while the platform derives the runner's retry request from the claimed operation and never consumes a stored flag. A plain release is the correct undo there. DeploymentStore::release_unused_acquisition now owns the undo. The default only releases. The SQLite store, whose acquire clears retry_requested, restores it in the same UPDATE that clears the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…inished-updates-drop-manager-injected-env-vars
Member
Author
|
@greptileai review |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
@greptileai review |
Member
Author
|
@greptileai review |
alongubkin
added a commit
that referenced
this pull request
Oct 14, 2026
…key (#1019) ALIEN-1477. alien-core tests don't compile on main (`d07764433`): #1000 added compute planner tests that build `PersistentStorage` without the `encryption_key` field #1005 added. The two PRs merged one after the other, so each passed CI alone. `cargo test -p alien-core` fails with E0063 at `compute_planner.rs` 1312, 1462 and 1518. Main's Tests workflow is skipped on push, so it only shows on PRs whose test selection includes alien-core. This sets `encryption_key: None` in the three fixtures. Test-only, no behavior change. Checked in a Linux sandbox on main (via #1012, which merges main) + this commit: `cargo check --workspace --all-targets` is clean, `cargo nextest run -p alien-core` 660/660 pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
alongubkin
deleted the
alon/alien-1463-cli-finished-updates-drop-manager-injected-env-vars
branch
October 14, 2026 03:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the manager starts an update and
alien deploytakes the lease and finishes it, every container is rolled twice. The second rollout uses a different spec: the containers loseALIEN_DEPLOYMENT_IDandALIEN_DEPLOYMENT_NAME, and command-enabled containers and daemons also loseALIEN_COMMANDS_*. They run without these variables until the manager runs another update.For a single-replica database, that means two restarts back to back instead of one.
Root cause
Two runners built two different configs for the same operation:
loops/deployment.rs): builds an environment snapshot with the deployment id and name, the command receiver variables, and the deployment's own variables. It puts that snapshot in the config before it runs a step.POST /v1/sync/acquire(routes/sync.rs): returned the record's rawdeploymentConfig, so the control plane's copy of the variables went out without the manager's snapshot. In standalone mode it returned no config at all.alien deployandalien-deploy-cli up: both use the acquired config as-is (since fix(cli): hydrate acquired deployment configuration #380).The executor sees a different desired config after the first rollout is ready, so it runs a second update.
The variables were not left out on purpose. Acquire already hands out the secret-bearing config: the record field is
skip_serializingprecisely so that acquire is the one route that returns it, and the acquire helper's doc says it exists for config fields redacted fromGET /v1/deployments. Every value in the snapshot comes from data the caller already has, or is entitled to:A runner has to write all of these into the workloads anyway.
Fix
One shared builder,
RunnerConfigSources::deployment_config, used by both the loop and the acquire route. It holds the code the loop already ran, moved out unchanged: the environment snapshot, the management config, monitoring, the native image host, and how a control plane's config gets merged. Target-stack resolution (supplied stack or release stack) moves intotarget_release_stack, which both paths also share.stack_settingsis now an error rather than a panic. The tautologicalwith_environment_snapshottest is replaced with one that runs the real builder.Proof
tests/acquire_runner_config.rs: runs the CLI's real acquire helper (acquire_deployment_with_payload) against the real/v1/sync/acquireroute over TCP, using sqlite stores. The deployment has an update queued to a release with a command-enabled container. The test decodesdeploymentConfigthe wayalien deploydoes and asserts the exact variable list: id, name, the four command variables scoped to the container, and the user variable.deploymentConfigin the response) and passes with the fix.COMMAND_TOKEN_MISSINGand leaves the deployment unlocked.routes::sync::tests::acquired_config_is_the_config_the_manager_loop_runs_with: the reported case, where a control plane supplies the config and stacks. The acquireddeploymentConfigequals the loop's config field for field; only the snapshot'screatedAtmay differ.loops::deployment::tests::runner_config_replaces_control_plane_snapshot_with_manager_variables: the builder replaces the control plane's snapshot with the manager's variables.Run in a remote sandbox (nightly-2026-10-03):
cargo nextest run -p alien-manager --lib --test acquire_runner_config: 334 passed, including the 4 new tests.crates/alien-manager/srcfrommain: both tests fail (acquire returns the config to run withpanics becausedeploymentConfigis missing; the lease test fails because the acquire succeeds).cargo clippy -p alien-manager --tests: no warnings in the changed files.🤖 Generated with Claude Code