Skip to content

test: record controller polling delays instead of sleeping them - #886

Merged
alongubkin merged 2 commits into
mainfrom
alon/alien-1216-controller-test-delays
Oct 5, 2026
Merged

alongubkin merged 2 commits into
mainfrom
alon/alien-1216-controller-test-delays

Conversation

@alongubkin

@alongubkin alongubkin commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Problem

SingleControllerExecutor (crates/alien-infra/src/core/controller_test.rs) slept for each controller's suggested_delay between steps. Those are 2–30 s, and a create+delete flow has 10–20 polls. The mocked providers answer at once, so this was pure idle time. GCP worker flows took 37–62 s each, and alien-infra spent ~960 test-seconds per CI run.

Nothing checked the delays either. A controller that regressed to polling a cloud API in a hot loop would still pass.

Change

  • DelayMode::Record is the default. The executor records each suggested delay (suggested_delays()) and continues.
  • .real_delays() keeps real waits for the three #[ignore] live-cloud tests (azure_sandbox_image_live, azure_sandbox_image_refusals_live, kubernetes_sandbox_live).
  • A suggested delay over 10 minutes (MAX_SUGGESTED_DELAY) fails the run. That catches a seconds/milliseconds mix-up or a runaway backoff.
  • The GCP and AWS worker create+delete flows check each phase separately (take_suggested_delays, assert_polling_delays).
    • Every wait must be at least 1 s.
    • GCP create and delete, and AWS create, must poll at least once.
    • AWS Lambda deletion completes in the call, so its delete phase may not poll.
  • Two unit tests for the executor: delays are recorded without sleeping, and an over-limit delay is rejected.

Safe to stop sleeping?

  • Steps, mock calls and assertions are unchanged. Only the idle time between steps goes away.
  • The Azure wall-clock waits (RBAC, managed identity, Container Apps) are already 0 under #[cfg(test)]. The seeded-timestamp tests (sandbox build timeout, retire window) seed their times explicitly.
  • Paused tokio time isn't used: these tests talk to a real local httpmock server, and paused time would let reqwest timeouts fire early.

Verified

  • cargo test -p alien-infra --features all-platforms,kubernetes,local,test --lib --tests: 813 lib tests pass in 11 s, plus the integration tests.
  • CI benchmark (same harness change, 2 runs): alien-infra 961 → 199/209 test-seconds, all tests pass.

🤖 Generated with Claude Code

SingleControllerExecutor slept each controller's suggested delay (2-30 s)
between steps, although every mocked provider answers at once. alien-infra's
tests spent ~960 test-seconds idle.

- The executor records suggested delays by default and continues;
  `.real_delays()` keeps real waits for the three live cloud tests.
- A suggested delay over 10 minutes fails the run.
- The GCP and AWS worker flows assert that every poll waits at least a
  second, so a hot loop against a cloud API fails a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium risk] Test infrastructure records polling delays instead of sleeping.

The PR appears safe to merge; no outstanding finding or new actionable issue remains.

Summary

The PR replaces sleeps with recorded polling delays in mocked controller tests, retains real waits for three live tests, and checks worker polling delays separately for create and delete.

  • The follow-up change separates phase assertions using take_suggested_delays() and adds a shared minimum-delay assertion.

Reviews (2) · Last reviewed commit: "test: check polling delays per phase"

Comment thread crates/alien-infra/src/worker/aws.rs Outdated
Assert each phase separately (take_suggested_delays), so a create-phase wait
can't cover a delete phase that hot-loops. GCP create and delete, and AWS
create, must poll; AWS Lambda deletion completes in the call, so its delete
only has to respect the minimum wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@alongubkin
alongubkin merged commit 8a2eee5 into main Oct 5, 2026
22 checks passed
@alongubkin
alongubkin deleted the alon/alien-1216-controller-test-delays branch October 5, 2026 13:43
alongubkin added a commit that referenced this pull request Oct 5, 2026
…896)

## Problem
After the recent test fixes, `Rust crates` is ~10.5 min, and the cold
build (~5.3 min) is most of it. Every PR builds and tests all 57
packages, even a change to one crate. Replaying the last 50 merged PRs,
about half affect 11 or fewer packages.

## Change
- **`scripts/affected-rust-packages.py`** reads the PR diff (base..merge
commit) and `cargo metadata`.
- It selects the changed packages plus every package that depends on
them, through normal, dev or build dependencies.
- **Any changed file outside the packages selects the full workspace,**
unless it is documentation (`docs/`, `*.md`). That covers manifests, the
lockfile, toolchain, `.cargo/`, nextest config, workflows, Dockerfiles,
scripts, `examples/` and `packages/`. Unknown inputs always mean "test
everything".
- **Packages under `examples/` and `tests/` are fixtures** that other
packages' tests read (the CLI checks every example's `template.toml`),
so changes there select the full workspace too.
- **A docs-only change selects no packages** (mode `none`), and the Rust
job and check are skipped.
- **`changes` runs it on pull requests** (stable cargo, so the pinned
nightly isn't installed there). Pushes, merge groups and manual runs are
always `full`.
- **`Rust crates` builds and tests that selection** (`$RUST_SELECTION`),
with the CI features limited to the selected packages.
- The privileged sandbox-agent tests run when `alien-sandbox-agent` is
selected.
  - The image contract test runs when `alien-core` is selected.
- The smoke harness runs in full mode, since its inputs are scripts and
Dockerfiles, which always select full.
- **New `Rust check` job,** in parallel and only in affected mode:
`cargo check --workspace --all-targets` with the CI features, so no
package can stop compiling unnoticed. It's added to `All tests passed`.

## Measured (CI benchmark, 2 runs each, `depot-ubuntu-24.04-arm-8`)

| PR shape | Full run today | Affected build + tests | `Rust check`
(parallel) |
|---|---|---|---|
| `alien-cli` only (1 package) | ~310 s build + ~214 s tests | 142 s + 2
s | ~193 s |
| `alien-infra` (8 packages) | same | 194 s + 44 s | ~187 s |
| `alien-core` or `Cargo.lock` (42+) | same | ≈ full | — |

The replays select as expected: #819 (CLI only) gets 1 package, #886
(alien-infra) 8, #865 (core) 42. #880 and #891 changed CI files, so they
get the full run.

This PR changes `ci-fast.yml` and a script, so its own CI takes the full
run.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant