Skip to content

test: run the worker integration tests in their own job, as two real tests - #891

Merged
alongubkin merged 2 commits into
mainfrom
alon/alien-1216-worker-integration-job
Oct 5, 2026
Merged

alongubkin merged 2 commits into
mainfrom
alon/alien-1216-worker-integration-job

Conversation

@alongubkin

@alongubkin alongubkin commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Problem

alien-local::worker_integration set the Rust crates job's wall clock. On main today it takes 607 test-seconds of a 607 s test run.

  • The first test built the bindings addon in release mode (napi build --release, ~5 min), and the other tests waited on its lock.
  • Then 8 tests each ran alien build of examples/basic-worker-ts again, one at a time.

In 60 days its failures were all readiness timeouts, install errors or build races it caused itself. None was a product bug.

Change

  • New path-filtered Worker integration job in ci-fast.yml, in parallel with Rust crates.
    • It builds the addon (cargo build -p alien-bindings-node --lib, debug, staged as alien-bindings-node.linux-arm64-gnu.node) and the AI gateway once.
    • Then it runs cargo nextest run -p alien-local --test worker_integration.
    • It runs when the local runtime, build toolchain, bindings, worker runtime, packages, the example app or the root/examples pnpm files change, and on manual runs.
    • It's added to All tests passed.
  • The main Rust job filters the binary out, so it still compiles there.
  • Its tests share a nextest test group with max-threads = 1: two alien build runs in the same example directory could delete each other's generated files.
  • The 8 build-heavy tests become 2:
    • worker_lifecycle_on_one_image checks:
      • a never-started worker is unhealthy
      • start, then a second start returns the same URL
      • the Local binding URL equals the worker URL
      • /health returns ok with a timestamp, and the health check passes
      • after stop, the worker isn't running and isn't healthy
      • it restarts and serves again
      • delete removes the extracted dir and the URL
    • two_workers_from_one_image_run_side_by_side: two workers from one image start at the same time, get distinct ports, and both serve /health. Both run the same Bun-compiled binary, so this also covers processes sharing its embedded addon.
  • Removed:
    • test_delete_prevents_port_reuse: it asserted two random ports differ, so it can fail on correct code.
    • test_port_allocation_fallback: it only printed which port it got.
  • A missing bun now fails the tests instead of skipping them.

Verified locally (macOS, addon prebuilt)

cargo nextest run -p alien-local --test worker_integration: 5 passed in 61 s.

🤖 Generated with Claude Code

…tests

The 11 alien-local worker tests set the Rust job's wall clock: the first one
built the bindings addon in release mode while the rest waited, then each of
8 built the example app again, one at a time (~600 test-seconds per run).

- A path-filtered `Worker integration` job builds the addon and AI gateway
  once (debug) and runs the suite; the main Rust job filters it out.
- The 8 build-heavy tests become 2: one worker through its whole lifecycle,
  and two workers from one image started side by side.
- Dropped `test_delete_prevents_port_reuse` (could fail on correct code)
  and `test_port_allocation_fallback` (asserted nothing beyond restart).
- A missing Bun now fails the tests instead of skipping them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium risk] Reorganizes test execution and CI filtering for worker tests.

The PR appears safe to merge; no outstanding finding or new actionable issue remains.

Summary

The PR moves worker integration tests into a path-filtered CI job and consolidates the build-heavy tests.

  • The new nextest group serializes the tests that build from the shared example directory.
  • Both previous findings are addressed: the builds are serialized, and the worker-job filter includes the previously missing dependency files.

Reviews (2) · Last reviewed commit: "test: serialize the worker builds and ru..."

Comment thread .github/workflows/ci-fast.yml
Comment thread .github/workflows/ci-fast.yml
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@alongubkin
alongubkin merged commit 8ced1f2 into main Oct 5, 2026
23 checks passed
@alongubkin
alongubkin deleted the alon/alien-1216-worker-integration-job branch October 5, 2026 15:01
alongubkin added a commit that referenced this pull request Oct 5, 2026
…894)

## Problem
The `alien-worker-runtime::container_app` tests (5 tests, ~236–341
test-seconds in the Rust job) each start a real `alien-test-app`
process.
- `test_utils::get_test_app_path` builds it with `cargo build -p
alien-test-app` and caches the path in a `Once`. But nextest runs each
test in its own process, so every test runs that build.
- `-p alien-test-app` resolves features differently from the workspace
test build. So the first build recompiles dependencies, and the others
wait on the target-dir lock. Each test takes ~47 s, nearly all of it
waiting.

## Change
- **`get_test_app_path` uses `ALIEN_TEST_APP_BIN` when it's set,** and
fails if that path isn't a file. Without it, the tests build the binary
themselves as before, so local `cargo test` is unchanged.
- **The `Worker integration` job (#891) builds `alien-test-app` once,**
in the same `cargo build` as the addon and gateway. It exports
`ALIEN_TEST_APP_BIN` and runs `container_app` together with
`worker_integration`.
- **The main Rust job filters `container_app` out.** The test binary
still compiles there.
- **The job now runs on any change matching the `rust` filter,** plus
the JS inputs in `worker_integration`. The tested packages,
alien-test-app, the addon and the gateway depend on 31 workspace crate
directories, so a hand-kept path list would drift. It runs in parallel,
so it adds no wall-clock time.
- **It runs only the two test targets** (`--test worker_integration
--test container_app`).

## Verified locally
With `ALIEN_TEST_APP_BIN` set: 5 passed, 3–6 s each.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
alongubkin added a commit that referenced this pull request Oct 5, 2026
…896)

## Problem
After the recent test fixes, `Rust crates` is ~10.5 min, and the cold
build (~5.3 min) is most of it. Every PR builds and tests all 57
packages, even a change to one crate. Replaying the last 50 merged PRs,
about half affect 11 or fewer packages.

## Change
- **`scripts/affected-rust-packages.py`** reads the PR diff (base..merge
commit) and `cargo metadata`.
- It selects the changed packages plus every package that depends on
them, through normal, dev or build dependencies.
- **Any changed file outside the packages selects the full workspace,**
unless it is documentation (`docs/`, `*.md`). That covers manifests, the
lockfile, toolchain, `.cargo/`, nextest config, workflows, Dockerfiles,
scripts, `examples/` and `packages/`. Unknown inputs always mean "test
everything".
- **Packages under `examples/` and `tests/` are fixtures** that other
packages' tests read (the CLI checks every example's `template.toml`),
so changes there select the full workspace too.
- **A docs-only change selects no packages** (mode `none`), and the Rust
job and check are skipped.
- **`changes` runs it on pull requests** (stable cargo, so the pinned
nightly isn't installed there). Pushes, merge groups and manual runs are
always `full`.
- **`Rust crates` builds and tests that selection** (`$RUST_SELECTION`),
with the CI features limited to the selected packages.
- The privileged sandbox-agent tests run when `alien-sandbox-agent` is
selected.
  - The image contract test runs when `alien-core` is selected.
- The smoke harness runs in full mode, since its inputs are scripts and
Dockerfiles, which always select full.
- **New `Rust check` job,** in parallel and only in affected mode:
`cargo check --workspace --all-targets` with the CI features, so no
package can stop compiling unnoticed. It's added to `All tests passed`.

## Measured (CI benchmark, 2 runs each, `depot-ubuntu-24.04-arm-8`)

| PR shape | Full run today | Affected build + tests | `Rust check`
(parallel) |
|---|---|---|---|
| `alien-cli` only (1 package) | ~310 s build + ~214 s tests | 142 s + 2
s | ~193 s |
| `alien-infra` (8 packages) | same | 194 s + 44 s | ~187 s |
| `alien-core` or `Cargo.lock` (42+) | same | ≈ full | — |

The replays select as expected: #819 (CLI only) gets 1 package, #886
(alien-infra) 8, #865 (core) 42. #880 and #891 changed CI files, so they
get the full run.

This PR changes `ci-fast.yml` and a script, so its own CI takes the full
run.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant