Skip to content

ci: retry failed XCTest cases on the iOS simulator - #882

Merged
turnipdabeets merged 1 commit into
mainfrom
ci/retry-failed-xctest-cases
Oct 1, 2026
Merged

turnipdabeets merged 1 commit into
mainfrom
ci/retry-failed-xctest-cases

Conversation

@turnipdabeets

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

test-ios-simulator has been red on most runs since #854 removed -retry-tests-on-failure. That includes 4 of the last 5 runs on main, plus #861, #872 and #881.

The failing tests vary from run to run (PostHogQueueTest, PostHogSDKTest, PostHogSDKPersonProfilesTest, PostHogAutocaptureIntegrationSpec). They all fail the same way: a stubbed request reaches MockPostHogServer after the 30s wait, so the test reports "The expected requests never arrived". When a test force-unwraps the result, the empty response crashes it, as in PostHogSDKPersonProfilesTest.swift:188.

  • The stalls also happen in green runs. On the last green main run (b3b42e28f), PostHogQueueTest took 84s, with single tests at 21s and 34s. In healthy runs it takes about 3s.
  • The requests are late, not lost. In one add_item_to_queue failure, the wait expired, and then the queue drained about 13s later.
  • They only happen on CI. Locally, 760 PostHogQueueTest runs (half under heavy CPU load) and a full-suite run showed no stalls.

#854 dropped retries because rerunning Swift Testing suites reinstalls irreversible swizzles. This PR adds retries back for XCTest only:

  • The suite runs once, as it does today.
  • If the only failures are XCTest cases (failed, or running when the test process crashed), the script reruns just those cases with test-without-building -only-testing:, in a fresh process, up to twice. Only the selected cases run, so no Swift Testing suite runs again.
  • Swift Testing failures, build failures, or more than 10 failed cases still fail the job immediately.
  • Retry output is appended to xcodebuild-ios.log, so the existing "Report flaky (retried) tests" step lists every test that only passed on retry.

This keeps the job usable while the stall itself is investigated. It doesn't fix the stall.

💚 How did you test it?

  • Fed the parser the raw xcodebuild-ios.log artifacts from 8 recent red runs. It picked the right cases in each, including the crashed one in feat: add disableGeoip config option #872's attempt 4. The two test: isolate deep link event fixtures #881 runs, which also had Swift Testing failures, were correctly not retried.
  • Ran the retry step locally against real CI logs from fix: keep nested dates and URLs in event properties #861 (two PostHogSDKTest failures) and feat: add disableGeoip config option #872 (a crash). It reran only those cases on an iOS 26 simulator, they passed, and the script exited 0. A log with Swift Testing failures exited 1 without retrying.
  • Confirmed -only-testing:PostHogTests/<Class>/<method>: selects exactly one Quick case and runs no Swift Testing suites.
  • Checked that the script runs under macOS's /bin/bash 3.2 and passes shellcheck.
  • The full simulator job still needs to run on this PR's CI.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Investigated and drafted with Claude Code (Claude Opus 5.5).

  • Ruled out cross-framework interference: in the CI logs, Swift Testing starts only after XCTest finishes. Also ruled out mock servers created mid-test, and the one blocking stub (gate.wait(), which is in a Swift Testing suite).
  • Considered raising the 30s wait instead. Stalls reach about 40s and would still slow every affected run, so retrying in a fresh process is more reliable.
  • The root cause of the CI-only stall is still unknown. A follow-up with timestamped queue and mock-server logging on CI should show whether the time is lost before the send, inside URLSession, or in the stub.

🤖 Generated with Claude Code

Reruns only failed or crashed XCTest cases in a fresh process; Swift Testing failures still fail the job.

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

worth rolling this back once we root cause this

@turnipdabeets
turnipdabeets merged commit a040982 into main Oct 1, 2026
47 checks passed
@turnipdabeets
turnipdabeets deleted the ci/retry-failed-xctest-cases branch October 1, 2026 15:41
@turnipdabeets

Copy link
Copy Markdown
Contributor Author

worth rolling this back once we root cause this

@dustinbyrne I have this debugging right now #883

@turnipdabeets turnipdabeets mentioned this pull request Oct 1, 2026
2 of 4 tasks
@dustinbyrne

Copy link
Copy Markdown
Contributor

i saw that, thanks for doing that!

@turnipdabeets

Copy link
Copy Markdown
Contributor Author

i saw that, thanks for doing that!

@dustinbyrne I think #885 and #886 should fix this finally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants