You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Windows CI: a vector-index call occasionally hangs until the step timeout, and the log has no Java stack #10
Twice in three days one Windows job of Test Python Bindings hung inside a vector-index call until the step timeout, while the other 19 platform jobs passed and the rerun, or the next commit, passed:
date
commit
job
test
where
2026-09-27
39553a0294
windows/amd64, Python 3.12
test_vector.py::test_per_index_cache_params (creates an LSM_VECTOR index on an empty type, reads its metadata)
25 minutes, no stack (before faulthandler_timeout)
The 10-minute dump of the second one shows the Python side only: the test thread waiting inside the Java call behind find_nearest (vector.py, _find_neighbor_pairs). What the JVM was doing, a lock, a wait on a pool thread, or a loop, is not in the log, and without it there is nothing to take upstream.
Proposed: when a test is still running shortly before faulthandler_timeout, the test session prints every Java thread's name, state, and stack to stderr (Thread.getAllStackTraces() through JPype), so the next occurrence carries the evidence. Engine builds: the 26.10.1-SNAPSHOT image CI pulled at each run.
The Java thread dump is merged in #11 (ec5f99c1b3): a test still running after 540 s now writes every Java thread's stack to the log, past pytest's capture. Keeping this open until a recurrence shows where the vector call waits.
A third occurrence, on 10b79be96e (run), Windows/amd64 Python 3.12.10: test_vector.py::TestLSMVectorIndex::test_lsm_cosine_distance_3d_orthogonal_vectors waited 10 minutes at the same line as the second, vector.py line 420, idx.findNeighborsFromVector(java_vector, k): the engine's own call. Three in two days, all Windows, all inside vector search. Main now carries the Java thread dump (#11); I am dispatching extra runs on it to catch one.
The Java dump from #11 fired on the third occurrence: run 36625374176, main at 4bb3ab4073, windows/amd64 Python 3.13, test_vector_sql.py::TestVectorSQL::test_create_index_with_native_int8_encoding_sql, dump at 540 s.
The Java main thread is not in the index. It is blocked writing a log line to stdout:
"main" daemon=False state=RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write
at java.lang.System$Out.write
...
at java.util.logging.ConsoleHandler.publish
at com.arcadedb.log.DefaultLogger.log(DefaultLogger.java:455)
at com.arcadedb.log.LogManager.log(LogManager.java:254)
at com.arcadedb.index.vector.LSMVectorIndex.build(LSMVectorIndex.java:10070)
...
at com.arcadedb.query.sql.parser.CreateIndexStatement.executeDDL
A write to a file does not block like that; a write to a pipe that nobody drains does, once its buffer is full. At the same moment the Python side printed this dump to the real stdout without trouble, so the handle Java was writing to was no longer the step's stdout. None of the Java log lines appear anywhere in the job log, so pytest's fd capture had been receiving them.
Most likely mechanism: pytest's default fd capture swaps fds 1 and 2 with dup2 around every test. On Windows that closes the OS handle the JVM cached for System.out/System.err when it started, and Windows reuses handle values, so a later pipe or socket (the server tests leave dozens of HttpClient selectors behind) can own the value Java still writes to. That fits everything seen so far: Windows only, intermittent, late in the run, and always in the vector tests, which log the most (index build progress). The two earlier hangs were probably the same write, reached from find_nearest.
Change: run the Windows job with --capture=sys, so pytest never swaps the fds and the handles the JVM holds stay the runner's own pipes, which it always drains. No test uses capfd. The cost is Java's log lines appearing in the Windows job log. Keeping this open until the Windows jobs have run clean for a while, since one green run proves nothing for a hang this intermittent.
Merged in PR #25 (01647768fa). The Windows jobs of that PR's run passed with the flag in force: Java's own log lines (INFO [LSMTreeIndex] Building index ...) now appear in the Windows job logs, which never happened under fd capture. Leaving this open until the Windows jobs have run clean for a while.
Clean since PR #25 (01647768fa, --capture=sys on Windows): 9 runs of the bindings tests from 2026-09-29 22:35Z to 2026-10-01 06:28Z (4 on main, 5 on PRs), each with all 5 Windows Python jobs passing, 45 of 45 Windows jobs and no hang. The cause and fix are recorded as BUGS F142.
Activity
tests: dump every Java thread's stack for a test still running after …
tae898 commentedon Sep 29, 2026
The Java thread dump is merged in #11 (
ec5f99c1b3): a test still running after 540 s now writes every Java thread's stack to the log, past pytest's capture. Keeping this open until a recurrence shows where the vector call waits.tae898 commentedon Sep 29, 2026
A third occurrence, on
10b79be96e(run), Windows/amd64 Python 3.12.10:test_vector.py::TestLSMVectorIndex::test_lsm_cosine_distance_3d_orthogonal_vectorswaited 10 minutes at the same line as the second,vector.pyline 420,idx.findNeighborsFromVector(java_vector, k): the engine's own call. Three in two days, all Windows, all inside vector search. Main now carries the Java thread dump (#11); I am dispatching extra runs on it to catch one.tae898 commentedon Sep 29, 2026
The Java dump from #11 fired on the third occurrence: run 36625374176, main at
4bb3ab4073, windows/amd64 Python 3.13,test_vector_sql.py::TestVectorSQL::test_create_index_with_native_int8_encoding_sql, dump at 540 s.The Java main thread is not in the index. It is blocked writing a log line to stdout:
A write to a file does not block like that; a write to a pipe that nobody drains does, once its buffer is full. At the same moment the Python side printed this dump to the real stdout without trouble, so the handle Java was writing to was no longer the step's stdout. None of the Java log lines appear anywhere in the job log, so pytest's fd capture had been receiving them.
Most likely mechanism: pytest's default fd capture swaps fds 1 and 2 with
dup2around every test. On Windows that closes the OS handle the JVM cached forSystem.out/System.errwhen it started, and Windows reuses handle values, so a later pipe or socket (the server tests leave dozens ofHttpClientselectors behind) can own the value Java still writes to. That fits everything seen so far: Windows only, intermittent, late in the run, and always in the vector tests, which log the most (index build progress). The two earlier hangs were probably the same write, reached fromfind_nearest.Change: run the Windows job with
--capture=sys, so pytest never swaps the fds and the handles the JVM holds stay the runner's own pipes, which it always drains. No test usescapfd. The cost is Java's log lines appearing in the Windows job log. Keeping this open until the Windows jobs have run clean for a while, since one green run proves nothing for a hang this intermittent.ci: run the Windows pytest step with --capture=sys, so the JVM keeps …
tae898 commentedon Sep 29, 2026
Merged in PR #25 (
01647768fa). The Windows jobs of that PR's run passed with the flag in force: Java's own log lines (INFO [LSMTreeIndex] Building index ...) now appear in the Windows job logs, which never happened under fd capture. Leaving this open until the Windows jobs have run clean for a while.tae898 commentedon Oct 1, 2026
Clean since PR #25 (
01647768fa,--capture=syson Windows): 9 runs of the bindings tests from 2026-09-29 22:35Z to 2026-10-01 06:28Z (4 on main, 5 on PRs), each with all 5 Windows Python jobs passing, 45 of 45 Windows jobs and no hang. The cause and fix are recorded as BUGS F142.