Skip to content

test: isolate GeoIP and replay event-trigger tests from other SDK instances - #899

Merged
turnipdabeets merged 2 commits into
mainfrom
test/fix-geoip-replay-trigger-flakes
Oct 5, 2026
Merged

turnipdabeets merged 2 commits into
mainfrom
test/fix-geoip-replay-trigger-flakes

Conversation

@turnipdabeets

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

Two tests fail intermittently on test-ios-simulator, for example on #897 (run 1, run 2):

✘ "flags requests have geoip_disable when disableGeoIp is enabled"
  Expectation failed: (body["geoip_disable"] as? Bool → false) == true

✘ "isActive returns false when waiting for event trigger"
  Expectation failed: (integration?.isActive() → true) == false

Both have the same cause as #885: another test's SDK instance is still running.

  • GeoIP: every SDK in the test process sends requests to localhost:9001, and OHHTTPStubs routes each /flags request to whichever MockPostHogServer is current. The test checks server.flagsRequests.last, which can be another instance's request sent with GeoIP enabled.
  • Replay event triggers: the suite saves the recording config (with event triggers) to storage under the shared test_project_token, then starts the SDK. Storage is kept per token, so any other suite saving remote config under that same token in between replaces the event triggers, and replay starts as active.

Changes

  • MockPostHogServer: a new flagsProjectToken, the same filter as batchProjectToken but for /flags.
  • GeoIP suite:
    • Each test uses its own token, and sets batchProjectToken and flagsProjectToken to it.
    • init() no longer deletes the whole Application Support folder. That folder holds every suite's storage, and the suite doesn't need it cleared because its tokens are unique.
  • Replay event-trigger suite: each test uses its own token.

Other suites are unchanged.

This is test-only. SDK behavior and public API are unchanged.

💚 How did you test it?

I reproduced both failures deterministically with a temporary change that simulates the other SDK instance:

Repro main This branch
A second SDK with a different token sends /flags after the GeoIP test's own request ✘ geoip_disable → false ✔
Another suite saves remote config (no event triggers) under test_project_token during replay test setup ✘ isActive() → true in 6 tests ✔ 12/12

Without the repros:

  • make test passes (935 Swift Testing + 174 XCTest tests).
  • make lint reports no new warnings.
  • The GeoIP and replay event-trigger suites pass in the iOS simulator (Xcode 26.4.1), 3 of 3 runs.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran pnpm changeset to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

  • Tooling: written with Claude Code, following the pattern from test: ignore other SDK instances' batches in session event tests #885.
  • Out of scope: about 10 other suites also delete the whole Application Support folder in setup, which can disturb any suite that runs at the same time. I left them alone and only removed the GeoIP one, which the fix made unnecessary.
  • Other flakes: "survey sent reports each question in the language it was answered in" also fails often on other branches. It's a separate issue and isn't addressed here.

🤖 Generated with Claude Code

…tances

- GeoIP tests use a token per test and record only their own /flags and
  /batch requests, and no longer wipe the shared Application Support folder.
- Replay event-trigger tests use a token per test, so another suite saving
  remote config under the shared test token can't replace the seeded
  recording config.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@turnipdabeets
turnipdabeets requested a review from a team as a code owner October 2, 2026 23:27
@turnipdabeets turnipdabeets self-assigned this Oct 2, 2026
@greptile-apps

greptile-apps Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Retrigger

[Medium risk] Test isolation refactored to use unique tokens per test.

Remove the shared-token REPRO write before merging; it can make another test fail intermittently.

Reviews (1) · Last reviewed commit: "test: isolate GeoIP and replay event-tri..."

Comment on lines +54 to +56
// REPRO: another suite's SDK saves its remote config (no event triggers) under the shared test token
PostHogStorage(PostHogConfig(projectToken: "test_project_token"))
.setDictionary(forKey: .remoteConfig, contents: ["sessionRecording": ["endpoint": "/s/"]])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Replay tests overwrite shared config

getSut() now overwrites .remoteConfig under the shared test_project_token in every replay test. If this lands between the seed and read in loadsCachedRemoteConfig(), that test reads the replay dictionary instead of ["foo": "bar"] and fails. .serialized does not serialize unrelated suites.

Remove this temporary REPRO write, or give both SDK instances in a dedicated regression test their own tokens.

Suggested change
// REPRO: another suite's SDK saves its remote config (no event triggers) under the shared test token
PostHogStorage(PostHogConfig(projectToken: "test_project_token"))
.setDictionary(forKey: .remoteConfig, contents: ["sessionRecording": ["endpoint": "/s/"]])

Knowledge Base Used: Restore iOS test coverage and harden test assertions

Prompt To Fix With AI
This is a comment left during a code review.
Path: PostHogTests/PostHogSessionReplayEventTriggersTest.swift
Line: 54-56

Comment:
**Replay tests overwrite shared config**

`getSut()` now overwrites `.remoteConfig` under the shared `test_project_token` in every replay test. If this lands between the seed and read in `loadsCachedRemoteConfig()`, that test reads the replay dictionary instead of `["foo": "bar"]` and fails. `.serialized` does not serialize unrelated suites.

Remove this temporary `REPRO` write, or give both SDK instances in a dedicated regression test their own tokens.

```suggestion

```

**Knowledge Base Used:** [Restore iOS test coverage and harden test assertions](https://app.greptile.com/posthog-org-19734/-/custom-context/knowledge-base/posthog/posthog-ios/-/reverts/rollback_854-20260929-ios-test-coverage-and-flaky-assertions-a8c1692.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, that was the temporary repro write and shouldn't have been committed. Removed in 38e3fde.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@turnipdabeets
turnipdabeets enabled auto-merge (squash) October 5, 2026 15:35
@turnipdabeets
turnipdabeets merged commit 1f550d2 into main Oct 5, 2026
46 checks passed
@turnipdabeets
turnipdabeets deleted the test/fix-geoip-replay-trigger-flakes branch October 5, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants