investigate: snapshot propagation latency degrades non-linearly under concurrent watchers (5th flake at maxForks=2 + 10s budget)

[moonming]

Summary

The harness-level fixes shipped in #169 (`waitConfigPropagation` 5s→10s + `maxForks` 4→2) did not eliminate the recurring flake on `guardrail-keyword-e2e.test.ts`. 5th occurrence observed on PR #172's CI (post-#169 rebase): `waitConfigPropagation: condition not met within 10s`.

Run	maxForks	budget	Outcome
1 (original #157)	4	5s	flake
2 (#165)	4	5s	flake
3 (#167)	4	5s	flake
4 (#169 own first attempt)	4	10s	flake (motivated maxForks→2 escalation)
5 (#172, post-#169 rebase)	2	10s	flake

Pattern

• Failure is concentrated on `guardrail-keyword-e2e.test.ts` — the test that gates readiness on the last resource in a 4-resource setup batch (Model + ApiKey + ProviderKey + Guardrail).
• Infra knobs (lower parallelism, longer budget) reduce frequency but don't eliminate it.
• Even at maxForks=2 with 10s budget, propagation latency for the trailing Guardrail resource exceeds the budget.

This pattern strongly supports the product-side hypothesis that #157 noted as deferred:

Real product slowness — the gateway's snapshot propagation queue under N concurrent watch streams from N gateway instances could have a contention bug. Worth investigating whether propagation latency degrades non-linearly with concurrent watcher count.

Possible product-side root causes (need investigation)

1. Lock contention in the snapshot reload path. When the gateway receives an etcd watch event, does the snapshot reload acquire a write-lock that blocks readers? Under N concurrent gateway instances → N watch streams → N times the lock-pressure on the shared etcd's revision dispatch loop.

2. etcd revision-stream serialization across concurrent `Watch()` RPCs. etcd processes watch events in a single goroutine per stream; multiple streams against the same etcd may share contention on the underlying boltdb. Worth measuring per-revision dispatch latency under load.

3. Per-resource-type sequential application. If the gateway applies a multi-resource batch one resource at a time and re-builds the entire snapshot between each resource (rather than batching), the LAST resource (Guardrail) lands after Model + ApiKey + ProviderKey have each triggered N rebuilds. With 4 resources and N=2 gateways, that's 8 rebuilds before Guardrail is visible.

4. Watch-event coalescing dropped. etcd allows server-side coalescing of watch events; if the gateway's client doesn't coalesce, every revision triggers a full snapshot reload. Under load that's O(N × revisions) reloads.

Suggested investigation path

1. Add OTLP / Prom counters for "snapshot propagation latency by resource type" — measure 95th/99th percentile under `maxForks=2` and `maxForks=4` runs.
2. Add a stress test that spawns N=8 `aisix` instances against one etcd, each writing a 4-resource batch and timing snapshot-visibility for the LAST resource. Compare N=1 vs N=2 vs N=4 vs N=8 propagation latency.
3. If latency degrades super-linearly with N, that's the product bug; if linear-only, the harness budget needs further bump but no product change.

Workaround if root-cause takes time

If a product-side fix is too involved to ship quickly, the harness could:

• Bump `waitConfigPropagation` deadline further (10s→20s or 30s).
• Force `singleFork: true` in vitest config (sequential e2e files, eliminates concurrency entirely — slower but deterministic).
• Refactor `guardrail-keyword-e2e.test.ts` to register the Guardrail resource FIRST (before Model + ApiKey + ProviderKey), so propagation completion of the last resource is no longer the load-bearing readiness signal.

Severity

MEDIUM as a product concern (only affects observed CI flakiness, no production data loss); HIGH as a CI-velocity concern (5 reruns burned on this same flake).

Audit trail

Surfaced cumulatively across #156, #165, #167, #169 (own first run), #172. The hypothesis was flagged as a LOW audit finding on #169 ("symptomatic fix; product-side hypothesis still open"). With 5 occurrences across 5 PRs, the hypothesis warrants product-side investigation rather than further harness band-aids.

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• test(e2e): guardrail-keyword propagation race under heavy CI parallelism (transient flake) #157

🔗 Related PRs

api7/api7-ee-3-gateway#1277 - fix: resolve 13 flaky test cases causing CI instability [merged]
api7/api7ee-3-control-plane#2389 - fix(e2e): stabilize flaky tests and optimize kine poll interval [merged]
api7/api7ee-3-control-plane#2394 - perf(e2e): optimize test suite performance by 27% [merged]
api7/api7-ee-3-gateway#1481 - test(ci): fix flaky tests [merged]
api7/api7-ee-3-gateway#1530 - test(ci): fix flaky tests (follow-up to #1481) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[moonming]

Sixth occurrence on PR #184. Pattern unchanged (guardrail-keyword propagation budget). Reran.

[jarvis9443]

Re-investigated — closing as harness flake, not a product bug, already mitigated.

The premise (concurrent watchers contend non-linearly on a shared snapshot) doesn't hold: each e2e gateway process watches a unique random etcd prefix (tests/e2e/src/harness/app.ts:55, aisix-e2e-${randomUUID()}), so no two processes share a snapshot or key-space. Each process's snapshot RCU (supervisor.rs:344) only ever rebuilds its own tiny config. The one genuinely shared resource is host CPU — under maxForks parallel aisix processes a starved CI runner just delays the whole watch→apply→visible pipeline (wall-clock, roughly linear in fork count), which is why guardrail-keyword (the longest 4-resource propagation chain) is the canary.

It was already mitigated by #251 (waitConfigPropagation 10s→30s, see admin.ts:94) and there's been no recurrence since. If we ever want full determinism the only lever is harness-side (singleFork: true), not a product change.