Repository navigation
ci(backend): Start devservices and per-worker Snuba bootstrap early - #127397
Merged
Merged
Conversation
Overlap the backend shard setup with itself: - setup-sentry can now start devservices in the background right after setup-uv (early-devservices), installing the devservices version pinned in uv.lock into a throwaway venv with the system python. Image pulls and health checks then overlap the venv restore. The existing devservices step waits for it, or runs devservices up in the foreground like before if the early start didn't happen. - bootstrap-snuba.py can run in the background (--start/--wait). With DEVSERVICES_STATUS_FILE set, phase 1 (per-worker ClickHouse bootstrap) starts once ClickHouse, Kafka and Redis are healthy instead of waiting for snuba-snuba-1, and phase 2 (stopping snuba-snuba-1, starting the per-worker APIs) waits for devservices up to succeed. It fails fast if devservices fails. Without the env var it behaves as before. Only the backend test shards opt in. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Same behavior as the shell helper: parse the pinned devservices version from the lockfile with tomllib, install it into a throwaway venv, run devservices up in the background and record its exit status for bootstrap-snuba.py. Timeouts kill the whole process group, and the git gc workaround is undone on every path. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Use `uv run --with devservices==<locked version>` instead of managing a throwaway venv, and drop the foreground fallback, the pid/started files and the git gc workaround on the early path (devservices 1.5.1 fetches dependencies as GitHub zipballs, it doesn't git clone). The non-early path is unchanged from master. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
joshuarli
marked this pull request as ready for review
October 14, 2026 17:44
…helpers Replace fork/setsid/dup2 and the broad BaseException handling with a detached re-exec of the helper that records the child's exit status. Drop the in-helper timeout/process-group kill; the wait step enforces the timeout and the runner reaps leftover processes. Trim comments. Co-Authored-By: Joshua Li <14209781+joshuarli@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3530dc7. Configure here.
subprocess.run(timeout=...) kills it and records exit 124, so the wait step and bootstrap-snuba.py both fail fast instead of polling.
joshuarli
approved these changes
Oct 14, 2026
adrianviquez
pushed a commit
that referenced
this pull request
Oct 14, 2026
…127397) Backend test shards spend ~2m10s before pytest starts, and most of it is serial: Python setup (~16s), then `devservices up` (image pulls ~38s, then ~30s waiting for `snuba-snuba-1` to bootstrap its own `default` database), then `bootstrap-snuba.py` (~30s), which only then starts the per-worker ClickHouse bootstrap. This PR overlaps those. Only `backend test` opts in. **1. Start devservices before Python setup** (`setup-sentry`, new `early-devservices` input, default off) - Right after `setup-uv`, `.github/actions/setup-sentry/devservices-bg.py start` (stdlib only) reads the `devservices` version pinned in `uv.lock` with `tomllib` and runs, in the background: `uv run --no-project --python python3 --default-index <lock's index> --with devservices==<version> -- devservices up --mode backend-ci` - No venv to manage: uv builds and caches the environment. - `--python python3` uses the runner's system Python, so uv doesn't download `.python-version`'s 3.13. - The existing "Start new devservices" step runs `devservices-bg.py wait` instead, which prints the log and exits with `devservices up`'s status. Without `early-devservices`, that step runs exactly the same commands as on master. - The state is a log file and an exit-status file under `$RUNNER_TEMP/devservices-ci/`. A timeout (the existing `devservices-timeout-minutes`) kills the whole process group. - The early path skips the global `git config gc.auto` workaround. devservices 1.5.1 downloads its dependencies as GitHub zipballs and never runs `git clone`, so there's nothing for it to protect. **2. Run the per-worker Snuba bootstrap in the background** (`bootstrap-snuba.py --start` / `--wait`) - Started right after checkout, so it's already waiting when containers come up. - With `DEVSERVICES_STATUS_FILE` set (the exit-status file above): - **Phase 1** (create `default_gw{i}` and run `snuba bootstrap --force`) starts once ClickHouse answers, `snuba-snuba-1` exists (for its image and network), and Docker reports `snuba-clickhouse-1`, `kafka-kafka-1` and `redis-redis-1` healthy, the same check devservices uses. It no longer waits for `snuba-snuba-1`'s own bootstrap. If container names ever change, a successful `devservices up` also satisfies this, so the worst case is today's timing. - **Phase 2** (stop `snuba-snuba-1`, start `snuba-gw{i}`) waits for `devservices up` to exit 0, so `snuba-snuba-1` is never stopped while devservices is still health-checking it. - If `devservices up` fails, the bootstrap fails fast instead of waiting out its timeout. - Without the env var (acceptance and other callers), behavior is unchanged. - "Bootstrap per-worker Snuba instances" now runs `--wait`, which prints the background log and exits with its status. **Flakiness notes** - Phase 1 now runs alongside `snuba-snuba-1`'s own bootstrap. They target different databases. Kafka topic creation is shared, but snuba ignores `TOPIC_ALREADY_EXISTS`, and today's 3 per-worker bootstraps already race each other the same way. The existing 3x retry is kept. - Both background processes detach into their own session, so they survive across steps. Both waits have hard timeouts, and the exit-status files are written atomically. **Local verification** (Docker sandbox, 4 CPUs, real `devservices up --mode backend-ci`, real snuba/ClickHouse images) - Old flow (15s simulated Python setup, then `devservices up`, then bootstrap), images cached: **103s** until `snuba-gw*` were healthy. - This version, images cached: **68s** until all 3 `snuba-gw*` returned 200, with phase 1 overlapping devservices' wait. The getsentry version (`default` mode) took 64s. - Fake-`uv` tests: success, failure (exit code passed through), timeout (including a child that ignores SIGTERM, killed via the process group) and `uv` missing all report correctly in `wait`. A bad lockfile fails the start step. - `ruff` and `mypy --strict` pass on the helper. The action schema check passes, and `actionlint` and `zizmor` show no new findings. The real test is this PR's `backend test` shards: compare "Setup sentry env" + "Bootstrap per-worker Snuba instances" against master's ~84s + ~35s median. Companion getsentry PR: getsentry/getsentry#22506, same change for `setup-getsentry` / `start-devservices`. <!-- junior-request-attribution:start --> via @joshuarli. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC8V02RHC7%3A1791911755.006269 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/slack%3AC8V02RHC7%3A1791911755.006269) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC8V02RHC7%3A1791911755.006269/?project=4510944073809921) <!-- junior-session-footer:end --> --------- Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Joshua Li <14209781+joshuarli@users.noreply.github.com>
3 tasks done
joshuarli
added a commit
that referenced
this pull request
Oct 21, 2026
#128313) Acceptance shards spend ~130s before pytest starts, and the end of that is serial. On a recent master run, `devservices up` started only after the venv restore and editable install. It then waited ~20s for `snuba-snuba-1` to finish its own bootstrap after ClickHouse, Kafka and Redis were healthy. Only after that did `bootstrap-snuba.py` start, taking another ~30s. "Wait for sentry env" (30s) and "Bootstrap per-worker Snuba instances" (30s) ran back to back. This gives acceptance the overlap backend got in #127397, with no script changes: - `setup-sentry` runs with `early-devservices: 'true'`, so `devservices up` starts right after `setup-uv`, alongside the venv setup. - `bootstrap-snuba.py --start` runs before setup-sentry, with `DEVSERVICES_STATUS_FILE` set. Phase 1 (per-worker ClickHouse bootstrap) starts as soon as its dependencies are healthy. Phase 2 still waits for `devservices up` to succeed before stopping `snuba-snuba-1`. - The later step runs `--wait` instead of the foreground bootstrap. The `--start` step sits before the background "Setup sentry env" step, so it doesn't start mid-way through that step (see #127566 on event-file truncation). The estimate is ~30–40s off every shard, measured from the end of setup to the start of pytest. Comparing this PR's "Wait for sentry env" and "Bootstrap per-worker Snuba instances" against master's ~30s each will show it. This is separate from #128312 so each change can be measured on its own. <!-- junior-request-attribution:start --> via @joshuarli. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC1B4LB39D%3A1792538208.590529 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/slack%3AC1B4LB39D%3A1792538208.590529) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC1B4LB39D%3A1792538208.590529/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Joshua Li <14209781+joshuarli@users.noreply.github.com>
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Backend test shards spend ~2m10s before pytest starts, and most of it is serial: Python setup (~16s), then
devservices up(image pulls ~38s, then ~30s waiting forsnuba-snuba-1to bootstrap its owndefaultdatabase), thenbootstrap-snuba.py(~30s), which only then starts the per-worker ClickHouse bootstrap. This PR overlaps those. Onlybackend testopts in.1. Start devservices before Python setup (
setup-sentry, newearly-devservicesinput, default off)setup-uv,.github/actions/setup-sentry/devservices-bg.py start(stdlib only) reads thedevservicesversion pinned inuv.lockwithtomlliband runs, in the background:uv run --no-project --python python3 --default-index <lock's index> --with devservices==<version> -- devservices up --mode backend-ci--python python3uses the runner's system Python, so uv doesn't download.python-version's 3.13.devservices-bg.py waitinstead, which prints the log and exits withdevservices up's status. Withoutearly-devservices, that step runs exactly the same commands as on master.$RUNNER_TEMP/devservices-ci/. A timeout (the existingdevservices-timeout-minutes) kills the whole process group.git config gc.autoworkaround. devservices 1.5.1 downloads its dependencies as GitHub zipballs and never runsgit clone, so there's nothing for it to protect.2. Run the per-worker Snuba bootstrap in the background (
bootstrap-snuba.py --start/--wait)DEVSERVICES_STATUS_FILEset (the exit-status file above):default_gw{i}and runsnuba bootstrap --force) starts once ClickHouse answers,snuba-snuba-1exists (for its image and network), and Docker reportssnuba-clickhouse-1,kafka-kafka-1andredis-redis-1healthy, the same check devservices uses. It no longer waits forsnuba-snuba-1's own bootstrap. If container names ever change, a successfuldevservices upalso satisfies this, so the worst case is today's timing.snuba-snuba-1, startsnuba-gw{i}) waits fordevservices upto exit 0, sosnuba-snuba-1is never stopped while devservices is still health-checking it.devservices upfails, the bootstrap fails fast instead of waiting out its timeout.--wait, which prints the background log and exits with its status.Flakiness notes
snuba-snuba-1's own bootstrap. They target different databases. Kafka topic creation is shared, but snuba ignoresTOPIC_ALREADY_EXISTS, and today's 3 per-worker bootstraps already race each other the same way. The existing 3x retry is kept.Local verification (Docker sandbox, 4 CPUs, real
devservices up --mode backend-ci, real snuba/ClickHouse images)devservices up, then bootstrap), images cached: 103s untilsnuba-gw*were healthy.snuba-gw*returned 200, with phase 1 overlapping devservices' wait. The getsentry version (defaultmode) took 64s.uvtests: success, failure (exit code passed through), timeout (including a child that ignores SIGTERM, killed via the process group) anduvmissing all report correctly inwait. A bad lockfile fails the start step.ruffandmypy --strictpass on the helper. The action schema check passes, andactionlintandzizmorshow no new findings.The real test is this PR's
backend testshards: compare "Setup sentry env" + "Bootstrap per-worker Snuba instances" against master's ~84s + ~35s median.Companion getsentry PR: getsentry/getsentry#22506, same change for
setup-getsentry/start-devservices.via @joshuarli.
--
View Junior Session [Sentry]