Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
This repository was archived by the owner on Nov 21, 2026. It is now read-only.

Windows CI: a vector-index call occasionally hangs until the step timeout, and the log has no Java stack #10

Closed

Description

@tae898

Twice in three days one Windows job of Test Python Bindings hung inside a vector-index call until the step timeout, while the other 19 platform jobs passed and the rerun, or the next commit, passed:

date commit job test where
2026-09-27 39553a0294 windows/amd64, Python 3.12 test_vector.py::test_per_index_cache_params (creates an LSM_VECTOR index on an empty type, reads its metadata) 25 minutes, no stack (before faulthandler_timeout)
2026-09-29 5ce049482c (run) windows/amd64, Python 3.14.7 test_vector_delta_visibility.py::test_deleted_vectors_are_filtered_from_delta_results faulthandler_timeout fired at 10 minutes

The 10-minute dump of the second one shows the Python side only: the test thread waiting inside the Java call behind find_nearest (vector.py, _find_neighbor_pairs). What the JVM was doing, a lock, a wait on a pool thread, or a loop, is not in the log, and without it there is nothing to take upstream.

Proposed: when a test is still running shortly before faulthandler_timeout, the test session prints every Java thread's name, state, and stack to stderr (Thread.getAllStackTraces() through JPype), so the next occurrence carries the evidence. Engine builds: the 26.10.1-SNAPSHOT image CI pulled at each run.

Activity

  1. tae898 commented on Sep 29, 2026

    @tae898
    Author

    The Java thread dump is merged in #11 (ec5f99c1b3): a test still running after 540 s now writes every Java thread's stack to the log, past pytest's capture. Keeping this open until a recurrence shows where the vector call waits.

  2. tae898 commented on Sep 29, 2026

    @tae898
    Author

    A third occurrence, on 10b79be96e (run), Windows/amd64 Python 3.12.10: test_vector.py::TestLSMVectorIndex::test_lsm_cosine_distance_3d_orthogonal_vectors waited 10 minutes at the same line as the second, vector.py line 420, idx.findNeighborsFromVector(java_vector, k): the engine's own call. Three in two days, all Windows, all inside vector search. Main now carries the Java thread dump (#11); I am dispatching extra runs on it to catch one.

  3. tae898 commented on Sep 29, 2026

    @tae898
    Author

    The Java dump from #11 fired on the third occurrence: run 36625374176, main at 4bb3ab4073, windows/amd64 Python 3.13, test_vector_sql.py::TestVectorSQL::test_create_index_with_native_int8_encoding_sql, dump at 540 s.

    The Java main thread is not in the index. It is blocked writing a log line to stdout:

    "main" daemon=False state=RUNNABLE
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write
        at java.lang.System$Out.write
        ...
        at java.util.logging.ConsoleHandler.publish
        at com.arcadedb.log.DefaultLogger.log(DefaultLogger.java:455)
        at com.arcadedb.log.LogManager.log(LogManager.java:254)
        at com.arcadedb.index.vector.LSMVectorIndex.build(LSMVectorIndex.java:10070)
        ...
        at com.arcadedb.query.sql.parser.CreateIndexStatement.executeDDL
    

    A write to a file does not block like that; a write to a pipe that nobody drains does, once its buffer is full. At the same moment the Python side printed this dump to the real stdout without trouble, so the handle Java was writing to was no longer the step's stdout. None of the Java log lines appear anywhere in the job log, so pytest's fd capture had been receiving them.

    Most likely mechanism: pytest's default fd capture swaps fds 1 and 2 with dup2 around every test. On Windows that closes the OS handle the JVM cached for System.out/System.err when it started, and Windows reuses handle values, so a later pipe or socket (the server tests leave dozens of HttpClient selectors behind) can own the value Java still writes to. That fits everything seen so far: Windows only, intermittent, late in the run, and always in the vector tests, which log the most (index build progress). The two earlier hangs were probably the same write, reached from find_nearest.

    Change: run the Windows job with --capture=sys, so pytest never swaps the fds and the handles the JVM holds stay the runner's own pipes, which it always drains. No test uses capfd. The cost is Java's log lines appearing in the Windows job log. Keeping this open until the Windows jobs have run clean for a while, since one green run proves nothing for a hang this intermittent.

  4. tae898 commented on Sep 29, 2026

    @tae898
    Author

    Merged in PR #25 (01647768fa). The Windows jobs of that PR's run passed with the flag in force: Java's own log lines (INFO [LSMTreeIndex] Building index ...) now appear in the Windows job logs, which never happened under fd capture. Leaving this open until the Windows jobs have run clean for a while.

  5. tae898 commented on Oct 1, 2026

    @tae898
    Author

    Clean since PR #25 (01647768fa, --capture=sys on Windows): 9 runs of the bindings tests from 2026-09-29 22:35Z to 2026-10-01 06:28Z (4 on main, 5 on PRs), each with all 5 Windows Python jobs passing, 45 of 45 Windows jobs and no hang. The cause and fix are recorded as BUGS F142.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions