Skip to content

ci(backend): Start devservices and per-worker Snuba bootstrap early - #127397

Merged
joshuarli merged 5 commits into
masterfrom
ref/ci-early-devservices
Oct 14, 2026
Merged

joshuarli merged 5 commits into
masterfrom
ref/ci-early-devservices

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Backend test shards spend ~2m10s before pytest starts, and most of it is serial: Python setup (~16s), then devservices up (image pulls ~38s, then ~30s waiting for snuba-snuba-1 to bootstrap its own default database), then bootstrap-snuba.py (~30s), which only then starts the per-worker ClickHouse bootstrap. This PR overlaps those. Only backend test opts in.

1. Start devservices before Python setup (setup-sentry, new early-devservices input, default off)

  • Right after setup-uv, .github/actions/setup-sentry/devservices-bg.py start (stdlib only) reads the devservices version pinned in uv.lock with tomllib and runs, in the background:
    uv run --no-project --python python3 --default-index <lock's index> --with devservices==<version> -- devservices up --mode backend-ci
    • No venv to manage: uv builds and caches the environment.
    • --python python3 uses the runner's system Python, so uv doesn't download .python-version's 3.13.
  • The existing "Start new devservices" step runs devservices-bg.py wait instead, which prints the log and exits with devservices up's status. Without early-devservices, that step runs exactly the same commands as on master.
  • The state is a log file and an exit-status file under $RUNNER_TEMP/devservices-ci/. A timeout (the existing devservices-timeout-minutes) kills the whole process group.
  • The early path skips the global git config gc.auto workaround. devservices 1.5.1 downloads its dependencies as GitHub zipballs and never runs git clone, so there's nothing for it to protect.

2. Run the per-worker Snuba bootstrap in the background (bootstrap-snuba.py --start / --wait)

  • Started right after checkout, so it's already waiting when containers come up.
  • With DEVSERVICES_STATUS_FILE set (the exit-status file above):
    • Phase 1 (create default_gw{i} and run snuba bootstrap --force) starts once ClickHouse answers, snuba-snuba-1 exists (for its image and network), and Docker reports snuba-clickhouse-1, kafka-kafka-1 and redis-redis-1 healthy, the same check devservices uses. It no longer waits for snuba-snuba-1's own bootstrap. If container names ever change, a successful devservices up also satisfies this, so the worst case is today's timing.
    • Phase 2 (stop snuba-snuba-1, start snuba-gw{i}) waits for devservices up to exit 0, so snuba-snuba-1 is never stopped while devservices is still health-checking it.
    • If devservices up fails, the bootstrap fails fast instead of waiting out its timeout.
  • Without the env var (acceptance and other callers), behavior is unchanged.
  • "Bootstrap per-worker Snuba instances" now runs --wait, which prints the background log and exits with its status.

Flakiness notes

  • Phase 1 now runs alongside snuba-snuba-1's own bootstrap. They target different databases. Kafka topic creation is shared, but snuba ignores TOPIC_ALREADY_EXISTS, and today's 3 per-worker bootstraps already race each other the same way. The existing 3x retry is kept.
  • Both background processes detach into their own session, so they survive across steps. Both waits have hard timeouts, and the exit-status files are written atomically.

Local verification (Docker sandbox, 4 CPUs, real devservices up --mode backend-ci, real snuba/ClickHouse images)

  • Old flow (15s simulated Python setup, then devservices up, then bootstrap), images cached: 103s until snuba-gw* were healthy.
  • This version, images cached: 68s until all 3 snuba-gw* returned 200, with phase 1 overlapping devservices' wait. The getsentry version (default mode) took 64s.
  • Fake-uv tests: success, failure (exit code passed through), timeout (including a child that ignores SIGTERM, killed via the process group) and uv missing all report correctly in wait. A bad lockfile fails the start step.
  • ruff and mypy --strict pass on the helper. The action schema check passes, and actionlint and zizmor show no new findings.

The real test is this PR's backend test shards: compare "Setup sentry env" + "Bootstrap per-worker Snuba instances" against master's ~84s + ~35s median.

Companion getsentry PR: getsentry/getsentry#22506, same change for setup-getsentry / start-devservices.

via @joshuarli.

--

View Junior Session [Sentry]

Overlap the backend shard setup with itself:

- setup-sentry can now start devservices in the background right after
  setup-uv (early-devservices), installing the devservices version pinned
  in uv.lock into a throwaway venv with the system python. Image pulls and
  health checks then overlap the venv restore. The existing devservices
  step waits for it, or runs devservices up in the foreground like before
  if the early start didn't happen.
- bootstrap-snuba.py can run in the background (--start/--wait). With
  DEVSERVICES_STATUS_FILE set, phase 1 (per-worker ClickHouse bootstrap)
  starts once ClickHouse, Kafka and Redis are healthy instead of waiting
  for snuba-snuba-1, and phase 2 (stopping snuba-snuba-1, starting the
  per-worker APIs) waits for devservices up to succeed. It fails fast if
  devservices fails. Without the env var it behaves as before.

Only the backend test shards opt in.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Oct 14, 2026
@sentry-junior
sentry-junior Bot requested a review from joshuarli October 14, 2026 17:03
@sentry-junior sentry-junior Bot added the Trigger: Override Selective Testing Run the full test suite; necessary in cases where selected tests are flaky due to reshuffling label Oct 14, 2026
sentry-junior Bot and others added 2 commits October 14, 2026 17:22
Same behavior as the shell helper: parse the pinned devservices version
from the lockfile with tomllib, install it into a throwaway venv, run
devservices up in the background and record its exit status for
bootstrap-snuba.py. Timeouts kill the whole process group, and the git gc
workaround is undone on every path.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Use `uv run --with devservices==<locked version>` instead of managing a
throwaway venv, and drop the foreground fallback, the pid/started files
and the git gc workaround on the early path (devservices 1.5.1 fetches
dependencies as GitHub zipballs, it doesn't git clone). The non-early
path is unchanged from master.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
@joshuarli
joshuarli marked this pull request as ready for review October 14, 2026 17:44
@joshuarli
joshuarli requested a review from a team as a code owner October 14, 2026 17:44
…helpers

Replace fork/setsid/dup2 and the broad BaseException handling with a
detached re-exec of the helper that records the child's exit status.
Drop the in-helper timeout/process-group kill; the wait step enforces
the timeout and the runner reaps leftover processes. Trim comments.

Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3530dc7. Configure here.

subprocess.run(timeout=...) kills it and records exit 124, so the wait
step and bootstrap-snuba.py both fail fast instead of polling.
@joshuarli
joshuarli merged commit 3945c1e into master Oct 14, 2026
90 checks passed
@joshuarli
joshuarli deleted the ref/ci-early-devservices branch October 14, 2026 18:45
adrianviquez pushed a commit that referenced this pull request Oct 14, 2026
…127397)

Backend test shards spend ~2m10s before pytest starts, and most of it is
serial: Python setup (~16s), then `devservices up` (image pulls ~38s,
then ~30s waiting for `snuba-snuba-1` to bootstrap its own `default`
database), then `bootstrap-snuba.py` (~30s), which only then starts the
per-worker ClickHouse bootstrap. This PR overlaps those. Only `backend
test` opts in.

**1. Start devservices before Python setup** (`setup-sentry`, new
`early-devservices` input, default off)
- Right after `setup-uv`,
`.github/actions/setup-sentry/devservices-bg.py start` (stdlib only)
reads the `devservices` version pinned in `uv.lock` with `tomllib` and
runs, in the background:
`uv run --no-project --python python3 --default-index <lock's index>
--with devservices==<version> -- devservices up --mode backend-ci`
  - No venv to manage: uv builds and caches the environment.
- `--python python3` uses the runner's system Python, so uv doesn't
download `.python-version`'s 3.13.
- The existing "Start new devservices" step runs `devservices-bg.py
wait` instead, which prints the log and exits with `devservices up`'s
status. Without `early-devservices`, that step runs exactly the same
commands as on master.
- The state is a log file and an exit-status file under
`$RUNNER_TEMP/devservices-ci/`. A timeout (the existing
`devservices-timeout-minutes`) kills the whole process group.
- The early path skips the global `git config gc.auto` workaround.
devservices 1.5.1 downloads its dependencies as GitHub zipballs and
never runs `git clone`, so there's nothing for it to protect.

**2. Run the per-worker Snuba bootstrap in the background**
(`bootstrap-snuba.py --start` / `--wait`)
- Started right after checkout, so it's already waiting when containers
come up.
- With `DEVSERVICES_STATUS_FILE` set (the exit-status file above):
- **Phase 1** (create `default_gw{i}` and run `snuba bootstrap --force`)
starts once ClickHouse answers, `snuba-snuba-1` exists (for its image
and network), and Docker reports `snuba-clickhouse-1`, `kafka-kafka-1`
and `redis-redis-1` healthy, the same check devservices uses. It no
longer waits for `snuba-snuba-1`'s own bootstrap. If container names
ever change, a successful `devservices up` also satisfies this, so the
worst case is today's timing.
- **Phase 2** (stop `snuba-snuba-1`, start `snuba-gw{i}`) waits for
`devservices up` to exit 0, so `snuba-snuba-1` is never stopped while
devservices is still health-checking it.
- If `devservices up` fails, the bootstrap fails fast instead of waiting
out its timeout.
- Without the env var (acceptance and other callers), behavior is
unchanged.
- "Bootstrap per-worker Snuba instances" now runs `--wait`, which prints
the background log and exits with its status.

**Flakiness notes**
- Phase 1 now runs alongside `snuba-snuba-1`'s own bootstrap. They
target different databases. Kafka topic creation is shared, but snuba
ignores `TOPIC_ALREADY_EXISTS`, and today's 3 per-worker bootstraps
already race each other the same way. The existing 3x retry is kept.
- Both background processes detach into their own session, so they
survive across steps. Both waits have hard timeouts, and the exit-status
files are written atomically.

**Local verification** (Docker sandbox, 4 CPUs, real `devservices up
--mode backend-ci`, real snuba/ClickHouse images)
- Old flow (15s simulated Python setup, then `devservices up`, then
bootstrap), images cached: **103s** until `snuba-gw*` were healthy.
- This version, images cached: **68s** until all 3 `snuba-gw*` returned
200, with phase 1 overlapping devservices' wait. The getsentry version
(`default` mode) took 64s.
- Fake-`uv` tests: success, failure (exit code passed through), timeout
(including a child that ignores SIGTERM, killed via the process group)
and `uv` missing all report correctly in `wait`. A bad lockfile fails
the start step.
- `ruff` and `mypy --strict` pass on the helper. The action schema check
passes, and `actionlint` and `zizmor` show no new findings.

The real test is this PR's `backend test` shards: compare "Setup sentry
env" + "Bootstrap per-worker Snuba instances" against master's ~84s +
~35s median.

Companion getsentry PR: getsentry/getsentry#22506, same change for
`setup-getsentry` / `start-devservices`.

<!-- junior-request-attribution:start -->
via @joshuarli.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3AC8V02RHC7%3A1791911755.006269 -->

--

[View Junior
Session](https://junior-prod.sentry.dev/conversations/slack%3AC8V02RHC7%3A1791911755.006269)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC8V02RHC7%3A1791911755.006269/?project=4510944073809921)

<!-- junior-session-footer:end -->

---------

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Joshua Li <14209781+joshuarli@users.noreply.github.com>
joshuarli added a commit that referenced this pull request Oct 21, 2026
#128313)

Acceptance shards spend ~130s before pytest starts, and the end of that
is serial. On a recent master run, `devservices up` started only after
the venv restore and editable install. It then waited ~20s for
`snuba-snuba-1` to finish its own bootstrap after ClickHouse, Kafka and
Redis were healthy. Only after that did `bootstrap-snuba.py` start,
taking another ~30s. "Wait for sentry env" (30s) and "Bootstrap
per-worker Snuba instances" (30s) ran back to back.

This gives acceptance the overlap backend got in #127397, with no script
changes:

- `setup-sentry` runs with `early-devservices: 'true'`, so `devservices
up` starts right after `setup-uv`, alongside the venv setup.
- `bootstrap-snuba.py --start` runs before setup-sentry, with
`DEVSERVICES_STATUS_FILE` set. Phase 1 (per-worker ClickHouse bootstrap)
starts as soon as its dependencies are healthy. Phase 2 still waits for
`devservices up` to succeed before stopping `snuba-snuba-1`.
- The later step runs `--wait` instead of the foreground bootstrap.

The `--start` step sits before the background "Setup sentry env" step,
so it doesn't start mid-way through that step (see #127566 on event-file
truncation).

The estimate is ~30–40s off every shard, measured from the end of setup
to the start of pytest. Comparing this PR's "Wait for sentry env" and
"Bootstrap per-worker Snuba instances" against master's ~30s each will
show it. This is separate from #128312 so each change can be measured on
its own.

<!-- junior-request-attribution:start -->
via @joshuarli.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:slack%3AC1B4LB39D%3A1792538208.590529 -->

--

[View Junior
Session](https://junior-prod.sentry.dev/conversations/slack%3AC1B4LB39D%3A1792538208.590529)
[[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC1B4LB39D%3A1792538208.590529/?project=4510944073809921)

<!-- junior-session-footer:end -->

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components Trigger: Override Selective Testing Run the full test suite; necessary in cases where selected tests are flaky due to reshuffling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant