Skip to content

[Due for payment 2026-06-12] [Due for payment 2026-06-09] [$50] Flaky test: MiddlewareTest › HandleUnusedOptimisticID - fetch call count assertion fails intermittently #90660

Closed

Description

@mountiny

Description

Two tests in `tests/unit/MiddlewareTest.ts` are flaky and fail intermittently in CI:

  • `Middleware › HandleUnusedOptimisticID › Request with preexistingReportID` (line 149)
  • `Middleware › HandleUnusedOptimisticID › OpenReport to a chat with preexistingReportID and clean up optimistic participant data` (line 343)

Both fail on the same assertion:

```
expect(global.fetch).toHaveBeenCalledTimes(2)
```

Evidence

Observed in CI run #33684 (attempt 1 failed, attempt 2 passed — confirming this is a flaky/timing issue).

The logs also report a worker teardown leak:

A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --detectOpenHandles to find leaks. Active timers can also cause this.

Likely cause

The tests use `waitForNetworkPromises()` / `waitForBatchedUpdates()` before asserting on `global.fetch` call count. A race condition between the `SequentialQueue.unpause()` call and the fetch mock resolution can cause the second fetch to fire after the assertion runs. The worker teardown leak suggests an open handle (timer or promise) that affects subsequent test isolation.

Upwork Automation - Do Not Edit
  • Upwork Job URL: https://www.upwork.com/jobs/~022054904870462893459
  • Upwork Job ID: 2054904870462893459
  • Last Price Increase: 2026-05-14
Issue OwnerCurrent Issue Owner: @mallenexpensify

Activity

  1. added
    ExternalAdded to denote the issue can be worked on by a contributor
    BugSomething is broken. Auto assigns a BugZero manager.
    on May 14, 2026
  2. added
    Help WantedApply this label when an issue is open to proposals by contributors
    on May 14, 2026
  3. melvin-bot commented on May 14, 2026

    @melvin-bot

    Triggered auto assignment to Contributor-plus team member for initial proposal review - @dmkt9 (External)

  4. changed the title [-]Flaky test: MiddlewareTest › HandleUnusedOptimisticID - fetch call count assertion fails intermittently[/-] [+][$250] Flaky test: MiddlewareTest › HandleUnusedOptimisticID - fetch call count assertion fails intermittently[/+] on May 14, 2026
  5. melvin-bot commented on May 14, 2026

    @melvin-bot
  6. changed the title [-][$250] Flaky test: MiddlewareTest › HandleUnusedOptimisticID - fetch call count assertion fails intermittently[/-] [+][$50] Flaky test: MiddlewareTest › HandleUnusedOptimisticID - fetch call count assertion fails intermittently[/+] on May 14, 2026
  7. melvin-bot commented on May 14, 2026

    @melvin-bot

    Upwork job price has been updated to $50

  8. mountiny commented on May 14, 2026

    @mountiny
    ContributorAuthor

    Making this $50 as its flaky test

  9. yusufdeveloper2903 commented on May 14, 2026

    @yusufdeveloper2903
    Contributor

    🚨 Edited by proposal-police: This proposal was edited at 2026-05-14 12:46:58 UTC.

    Proposal

    What is the root cause of that problem?

    Both flaky tests push two requests onto the SequentialQueue, then assert expect(global.fetch).toHaveBeenCalledTimes(2) after only a fixed number of microtask/macrotask flushes — waitForNetworkPromises() (two waitForBatchedUpdates rounds) in the first test, and a single waitForBatchedUpdates() in the second:

    But the queue does not process both requests in a fixed number of cycles. process() handles one request at a time through a recursive chain — it resolves the fetch mock (two async hops for the async mock + its async json()), runs the middleware chain (which performs async Onyx writes), applies the response via Onyx.update, and only then does .then(() => process()) pick up the next request and fire its fetch:

    currentRequestPromise = processWithMiddleware(requestToProcess, true)
    .then((response) => {
    Log.info('[SequentialQueue] Request processed successfully', false, {
    command: requestToProcess.command,
    shouldPauseQueue: response?.shouldPauseQueue ?? false,
    hasQueueFlushedData: !!requestToProcess.queueFlushedData,
    });
    // A response might indicate that the queue should be paused. This happens when a gap in onyx updates is detected between the client and the server and
    // that gap needs resolved before the queue can continue.
    if (response?.shouldPauseQueue) {
    Log.info("[SequentialQueue] Handled 'shouldPauseQueue' in response. Pausing the queue.");
    pause();
    }
    Log.info('[SequentialQueue] Removing persisted request because it was processed successfully.', false, {
    command: requestToProcess.command,
    remainingRequests: getAllPersistedRequests().length,
    });
    endPersistedRequestAndRemoveFromQueue(requestToProcess);
    if (requestToProcess.queueFlushedData) {
    Log.info('[SequentialQueue] Will store queueFlushedData.', false, {
    command: requestToProcess.command,
    queueFlushedDataLength: requestToProcess.queueFlushedData.length,
    });
    saveQueueFlushedData(...requestToProcess.queueFlushedData);
    }
    sequentialQueueRequestThrottle.clear();
    Log.info('[SequentialQueue] Continuing to process next request');
    return process();

    The async depth required for the second fetch to fire is variable. Under CI scheduling pressure it exceeds the fixed number of flushes, so the assertion runs while only the first fetch has been called → toHaveBeenCalledTimes(2) fails. The same unbounded process() chain still running after the test returns is also the open handle behind the "worker process failed to exit gracefully" teardown leak.

    Note: commit a7ede572585 improved this file's test isolation (beforeEach resets), but it did not address this assertion-timing race.

    What changes do you think we should make in order to solve the problem?

    Stop waiting a fixed number of cycles and instead wait for the queue to actually drain, using the existing SequentialQueue.waitForIdle() idiom (already used in ReportAttachments, WorkspaceUpgradeTest, PaginationTest). push() of the first request synchronously calls flush(true), which resets isReadyPromise before the test reaches its wait; that promise resolves in process().finally() only after the whole recursive chain has drained both requests:

    /**
    * Returns a promise that resolves when the sequential queue is done processing all persisted write requests.
    */
    function waitForIdle(): Promise<unknown> {
    return isReadyPromise;
    }

    Middleware › HandleUnusedOptimisticID › Request with preexistingReportID (L147):

                SequentialQueue.unpause();
    -           await waitForNetworkPromises();
    +           await SequentialQueue.waitForIdle();

    Middleware › HandleUnusedOptimisticID › OpenReport to a chat with preexistingReportID and clean up optimistic participant data (L341):

                SequentialQueue.unpause();
    -           await waitForBatchedUpdates();
    +           await SequentialQueue.waitForIdle();
    +           await waitForBatchedUpdates();

    (SequentialQueue is already imported in this file; the trailing waitForBatchedUpdates() in the second test keeps the follow-up Onyx reads consistent.) This makes the fetch count deterministic and guarantees the queue is fully drained before teardown, which also removes the worker-exit leak.

    The identical fixed-cycle pattern exists at the other SequentialQueue.unpause() sites in this file (L93-L94, L198-L199, L253-L254, L438-L439); applying the same waitForIdle() wait there prevents the same flake from resurfacing in those tests.

    What alternative solutions did you explore? (Optional)

    Adding more waitForBatchedUpdates() cycles (or widening waitForNetworkPromises) — rejected: it only lowers the probability of the race, it does not eliminate it, since the queue's async depth is unbounded relative to any fixed cycle count. await SequentialQueue.getCurrentRequest() — rejected: currentRequestPromise is null synchronously after unpause() (the chain starts inside an async Onyx.connectWithoutView callback), so awaiting it immediately resolves and waits for nothing.

  10. MelvinBot commented on May 14, 2026

    @MelvinBot
    Contributor

    Proposal

    What is the root cause of that problem?

    The two failing tests wait a fixed number of async cycles before asserting on global.fetch call count, but SequentialQueue requires a variable number of async hops between processing request #1 and issuing fetch #2.

    Specifically:

    Between the two fetches, the queue chains through: fetch() → response.json() → middleware processing (HandleUnusedOptimisticID + SaveResponseInOnyx with Onyx writes) → endPersistedRequestAndRemoveFromQueue (another Onyx write) → recursive process() → fetch #2. That's 5-6+ async hops.

    Test 1 (MiddlewareTest.ts:147) uses waitForNetworkPromises() (2 cycles) — barely enough, fails when Onyx operations schedule extra macrotasks.

    Test 2 (MiddlewareTest.ts:341) uses only waitForBatchedUpdates() (1 cycle) — clearly insufficient for 2 sequential requests with middleware processing.

    Note: SequentialQueue.waitForIdle() can't help here because unpause() calls flush(false) which skips resetting isReadyPromise, so it's already resolved from resetQueue().

    What changes do you think we should make in order to solve the problem?

    Add sufficient async cycle flushing in both tests to cover the full SequentialQueue processing chain for 2 sequential requests:

    • Test 1 (line 147): After SequentialQueue.unpause(), change await waitForNetworkPromises() → await waitForNetworkPromises() followed by an additional await waitForBatchedUpdates() to cover the extra async hops from middleware Onyx operations between the two fetches.

    • Test 2 (line 341): After SequentialQueue.unpause(), change await waitForBatchedUpdates() → await waitForNetworkPromises() followed by await waitForBatchedUpdates() (same as test 1, since it has the same two-request pattern).

    This directly addresses the insufficient wait budget without introducing new synchronization primitives.

    What alternative solutions did you explore?

    1. Using SequentialQueue.waitForIdle(): Won't work because unpause() → flush(false) preserves the already-resolved isReadyPromise from resetQueue(), so waitForIdle() resolves immediately before processing starts.
    2. Modifying flush() to always reset the promise: Would fix the test but changes production semantics — the shouldResetPromise=false path exists to prevent hanging READ requests during unpause (documented at line 271-274). Too risky for a test fix.

    Next Steps for Contributor+ team: Reply with @MelvinBot implement this to create a draft PR, @MelvinBot <your feedback> to refine this analysis, or explain why you are rejecting Melvin's proposal.

  11. 71 remaining items

  12. melvin-bot commented on Jun 11, 2026

    @melvin-bot

    Payment Summary

    Upwork Job

    • ROLE: @dmkt9 paid $(AMOUNT) via Upwork (LINK)

    BugZero Checklist (@mallenexpensify)

    • I have verified the correct assignees and roles are listed above and updated the necessary manual offers
    • I have verified that there are no duplicate or incorrect contracts on Upwork for this job (https://www.upwork.com/ab/applicants/2054904870462893459/hired)
    • I have verified the PR was not reverted
    • I have applied any discounts due to bugs/regressions introduced by this PR
    • I have paid out the Upwork contracts or cancelled the ones that are incorrect
    • I have verified the payment summary above is correct
  13. melvin-bot commented on Jun 11, 2026

    @melvin-bot

    @mallenexpensify Whoops! This issue is 2 days overdue. Let's get this updated quick!

  14. mallenexpensify commented on Jun 15, 2026

    @mallenexpensify
    Contributor

    Payment Summary

    Contributor+: @dmkt9 paid $125 via Upwork

    $125 cuz tests weren't needed.

  15. dmkt9 commented on Jun 16, 2026

    @dmkt9
    Contributor

    Contributor+: @dmkt9 due $125 via NewDot

    Hi @mallenexpensify, I'm still receiving payment through Upwork.

  16. mallenexpensify commented on Jun 16, 2026

    @mallenexpensify
    Contributor

    @dmkt9 can you please accept the job below? Please reply here and tag me once you have.
    If you're not sent an offer in Upwork when you're assigned to an issue, please tag me. If an Upwork job exists, please submit a proposal.
    https://www.upwork.com/jobs/~022066902892631038155

  17. dmkt9 commented on Jun 16, 2026

    @dmkt9
    Contributor

    @dmkt9 can you please accept the job below? Please reply here and tag me once you have. If you're not sent an offer in Upwork when you're assigned to an issue, please tag me. If an Upwork job exists, please submit a proposal. upwork.com/jobs/~022066902892631038155

    @mallenexpensify Thanks. I've accepted the offer

  18. mallenexpensify commented on Jun 16, 2026

    @mallenexpensify
    Contributor

    @dmkt9 paid, summary updated above. Thx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Awaiting PaymentAuto-added when associated PR is deployed to productionBugSomething is broken. Auto assigns a BugZero manager.DailyKSv2ExternalAdded to denote the issue can be worked on by a contributor

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions