Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
This repository was archived by the owner on Nov 21, 2026. It is now read-only.

test_async_executor_bulk_command_is_exact_at_parallel_four stored 8742 of 9742 once on a darwin/arm64 runner #99

Closed

Description

@tae898

On 2026-10-02 the Test Python Bindings job for darwin/arm64, Python 3.11, on c4fdf9b failed with assert 8742 == 9742: one batch of 1,000 missing. The test is the regression test for ArcadeData#7615 (records dropped silently by the async executor at parallel level 4, fixed by ArcadeData#7625 for 26.10.1). The failure stopped at the count assertion, so the log does not say whether the batch was dropped with no callback (ArcadeData#7615 back) or rolled back and reported through on_error after its retries ran out, which ArcadeData#7625 allows under heavy contention.

Not reproduced locally on the current engine: 0 of 140 runs of the same load (all 16 cores, 3 cores, and 2 cores shared with two busy loops). The failed job was re-run. The fix here makes the count assertion print the on_error calls, so a recurrence says which of the two it is; if it is the first, it goes upstream with a Java reproduction.

Activity

  1. tae898 commented on Oct 10, 2026

    @tae898
    Author

    Cause found: this is the upstream behaviour tracked in ArcadeData#9694. With the default retry settings a periodic batch commit on a one-bucket type at parallel level 4 can exhaust its retries and roll back the batch, about 0.4% of runs of this scenario even after upstream's ArcadeData#9535. Keeping this open until ArcadeData#9694 is fixed and verified; the test is right to fail on it.

  2. tae898 commented on Oct 11, 2026

    @tae898
    Author

    ArcadeData#9694 is fixed by ArcadeData#9703 (merged into upstream main as 49b4fe2, in 26.11.1) and verified there: 0 lost batches in 13,000 runs of the Java reproduction across JDK 21, JDK 25, no per-command callback and four buckets (details on ArcadeData#9694). Keeping this open until the bindings pick up an engine build that contains it; then this test should stop failing on that runner and this issue closes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions