Skip to content

Windows Python 3.14 parity: test_vector_read_your_writes missed 1 of 200 committed vectors once on main (intermittent) #409

Open

Description

@tae898

The scheduled run of test-native.yml on main of 2026-11-16 (run 43799876573, commit 4c95ed1) failed one job, "Parity against the installed wheel (windows-x64, Python 3.14)", with one failed test out of 3,447: tests/test_vector_read_your_writes.py::test_through_the_facade_a_task_finds_the_vector_it_committed_in_the_call_before.

The assertion: 1 of 200 committed vectors were not found by the next search: [(396, [162, 169, 427, 165, 454])]. It is not a crash (the existing --retry-crash of the Windows legs did not apply: crash 0, flaky_crash 0) and not a timeout. The same job passed on a re-run of the failed job (attempt 2), and the two previous scheduled runs (2026-11-15 and 2026-11-14) were green, so it is intermittent.

Why it is worth an issue and not only a re-run: the test checks the property that a vector committed in one call is found by the next search, the one an agent loop depends on (#306). One miss in 200 can be an approximate-search recall miss for a vector queried by itself, or a real visibility race between the commit and the index on this platform. The artifact native-parity-windows-x64-py3.14 of the run holds the per-file log and XML.

Proposal: keep rerunning a failed Windows leg once, as we do, and treat a second failure as real; do not auto-retry assertion failures (only crashes are retried today). For this test: run it in a loop on a Windows runner with the number of misses logged, then decide between a bug in the commit-to-search path and a test that should query with a slack (the vector must be among the top k, not first). If it fails again in the next two weeks it is quarantined with a marker and a tracking line here.

The policy for rerunning Windows legs is also not written in CONTRIBUTING.md or the CI docs; I will add it in the docs pull request of 26.11.1.post4.

Activity

  1. self-assigned this
    on Nov 16, 2026
  2. added
    bugSomething isn't working
    pythonPull requests that update python code
    vectorVector indexes and find_nearest
    concurrencyThreads, transactions and conflicts
    nativeThe arcadedb package (native library build)
    on Nov 16, 2026
  3. added this to the 26.12.1 milestone on Nov 16, 2026
  4. tae898 commented on Nov 16, 2026

    @tae898
    CollaboratorAuthor

    A second hit, and on Linux this time: PR #410 (the post4 release line into main), run 43863995512, job "Test arcadedb (linux/amd64, Python 3.12)", 3,596 passed and 1 failed: the same test, 1 of 200 committed vectors were not found by the next search: [(447, [271, 253, 169, 401, 386])]. So it is not Windows-specific and not a Python 3.14 matter. Hit 1 was the scheduled run on main at 4c95ed1 (before post4), so the cause is older than the post4 changes; the failing job is re-run to see whether it repeats on the same bytes.

    Two hits in two full runs is too few to give a rate, so my next step is a Java repro on the engine API (a committed vector missing from the very next search), in a loop on JDK 21 and 25, before anyone treats it as a CI flake.

    (Edited: the first version of this comment gave a rate of "two hits in about 7,000 test runs of this file"; that number counted the tests of two full suites, not runs of this test, so I removed it.)

  5. tae898 commented on Nov 16, 2026

    @tae898
    CollaboratorAuthor

    Correction to my comment above: I removed a rate that I could not support (it counted the tests of two full suites, not runs of this test). The failing job on #410 is re-run; the loop repro comes next.

  6. tae898 commented on Nov 17, 2026

    @tae898
    CollaboratorAuthor

    The failure reproduces locally, on Linux, and not only through the facade. The loop (600 planned iterations, one run of tests/test_vector_read_your_writes.py each, 4 tests per iteration) runs on the mini (i9-12900HK) in a systemd scope at nice 19 (I meant it to run on the 2 efficiency cores 12 and 13, but a user scope with AllowedCPUs does not pin a process on this host, so it ran on all 20 cores), with the native wheel that CI built from main (the artifact native-wheel-linux-x86_64 of run 44011962939, 26.12.1.dev0, commit 259e33639b), without conftest.py. The mini was loaded at the same time (load 10 to 24 from other jobs), which is closer to a shared CI runner than to a quiet machine.

    Result so far: 194 iterations, 193 pass and 1 failure, at iteration 82 (2026-11-17 06:07:07Z): test_a_task_finds_the_vector_it_committed_in_the_call_before (the plain database path, not the AsyncDatabase facade of the CI failure) with 1 of 200 committed vectors were not found by the next search: [(349, [144, 164, 326, 253, 95])]. The same assertion as the CI hits (the corpus is searched with ef_search of 400 and every vector is the nearest neighbour of itself at distance 0, so a miss is a write that the next search did not see, not approximate recall).

    So the race is not Windows-specific, not Python 3.14 specific and not a facade matter: it is in the commit-to-search path of the native build, at a rate of about one iteration in 190 under load. The failing output is saved with the loop's log (~/stage409/keep/ on the mini).

    Next: let the loop finish its 600 iterations for a rate, run the JVM build and the published 26.11.1.post4 native wheel (without the two facade tests, which need the module that only main has) for comparison, and then reduce it to a Java loop on the engine API, to see whether the engine or the binding is the place.

    (Edited 2026-11-17: the first version said the loop ran on 2 efficiency cores. It did not; it ran unpinned at nice 19 on a loaded machine. The failure counts are as read.)

  7. tae898 commented on Nov 17, 2026

    @tae898
    CollaboratorAuthor

    More data, 2026-11-17 06:39Z, from the loop of the comment above. At 06:31Z it stood at 194 iterations with 1 failure (that count was right at that time); at 06:39Z it stands at 218 iterations with 9, that is 8 failures in the last 24 iterations, after I started four more loops on the same mini (the load average went from about 10 to 40). I re-read every saved output and counted a run as a failure only when pytest itself printed failed (a JVM run that prints 2 passed and then exits nonzero is the separate JVM exit crash, and is not counted here). All runs: nice 19 and NOT pinned to any cores (my pin did not work, see the correction below), the mini loaded by other jobs:

    build iterations clean real test failures what failed
    native from main (CI wheel, 26.12.1.dev0) 218 209 9 1 of 200 committed vectors were not found (8), 2 of 200 ... (1)
    native, the published 26.11.1.post4 (the two tests without the facade, which post4 does not have) 30 24 6 1 of 200 ... (6)
    JVM from main 13 0 4 1 of 200 ... (3) and RecordNotFoundException: Record #1:446 not found raised from the vector search (1); the other 9 runs passed and then exited nonzero
    JVM, the published 26.11.1.post4 (two tests, no facade) 9 0 1 1 of 200 ... (1); the other 8 passed and exited nonzero

    So the published release is affected, on the native and on the JVM build, not only main. The rate depends on the load of the machine: 1 failure in the first 194 iterations of the native loop, 8 in the next 24 while the load was four times higher. That points at a timing race in the engine's commit-to-search path and not at the Python binding. The failing vector is not always the same (349, 492, 447, 456 and others), and (447, [271, 253, 169, 401, 386]) is exactly the signature of the CI failure on #410: the JVM run reproduced it.

    Next: a Java program on the engine API that replays the test (4 writers, 50 inserts each, a conflict retry, the search right after each commit) in a loop, against the released 26.11.1 jars and the main jars, so that it can be filed against the engine.

    (Edited 2026-11-17: the first version of this comment said that my earlier count of 1 failure in 194 was wrong and called the mini quiet during the first 100 iterations. Neither was right: the count was correct at 06:31Z and the load during those iterations was not measured.)

    (Second edit, 2026-11-17: the table's runs were unpinned, not on 2 efficiency cores each, as the first version of this line said; a user-level systemd scope with AllowedCPUs does not restrict cores on this host.)

  8. tae898 commented on Nov 17, 2026

    @tae898
    CollaboratorAuthor

    Correction to my comment of 06:39Z: it said that my count of 1 failure in 194 was wrong. It was right at the time (06:31Z); the failures came later, when the load of the mini rose. I edited the comment in place.

  9. tae898 commented on Nov 17, 2026

    @tae898
    CollaboratorAuthor

    Correction: my two comments above said the loop ran on 2 efficiency cores. It was not pinned: a user-level systemd scope with AllowedCPUs does not restrict cores on this host (no cpuset controller, Cpus_allowed_list stays 0-19), so it ran at nice 19 on all cores of a loaded machine. I edited both comments in place. The failure counts are as read.

  10. tae898 commented on Nov 17, 2026

    @tae898
    CollaboratorAuthor

    The race is in the engine, and it needs more than one writer. I wrote a Java repro against the engine API (no Python, no bindings, Repro.java, about 470 lines): writer threads each commit an insert (its own transaction, retried on NeedRetryException, as run_in_transaction(retries=60) does) and then, in the same thread, search k=5, efSearch=400 for the vector they just committed; a miss is that vector absent from the hits. A scenario is a fresh database with 300 base vectors of dimension 8 and an index whose graph is built first, then 4 writers with 50 inserts each (or 1 writer with 200). The JVM sees 2 cpus (affinity read back: 12 and 13, or 14 and 15, nice 19, on the mini). Every number below is read from the TOTALS block of its run.

    engine JDK writers scenarios committed inserts scenarios with a miss inserts missed search exceptions found by the immediate re-searches still missing on them missing after all writers ended
    26.11.1 (a3a5d1245) 25 4 200 40,000 47 (24%) 152 (0.38%) 6 124 34 1
    26.11.1 (a3a5d1245) 21 4 100 20,000 16 (16%) 18 (0.09%) 0 15 3 0
    main 515aff93c6 25 4 200 40,000 63 (32%) 154 (0.39%) 5 126 33 2
    main 515aff93c6 21 4 100 20,000 26 (26%) 28 (0.14%) 0 22 6 0
    26.11.1 (a3a5d1245) 25 1 100 20,000 0 (0%) 0 (0.00%) 0 0 0 0
    main 515aff93c6 25 1 100 20,000 0 (0%) 0 (0.00%) 0 0 0 0
    main 515aff93c6, rebuild threshold raised to 1,000,000 mutations 25 4 100 20,000 16 (16%) 21 (0.10%) 1 17 5 0

    The two re-search columns count misses and search exceptions together (RecordNotFoundException on loading a RID the index returned: 6 and 5 in the 200-scenario runs on JDK 25, none on JDK 21).

    What it shows:

    • One writer never misses (100 scenarios, 20,000 inserts, on both engines). With four writers 0.1 to 0.4 percent of the committed inserts are missed, on JDK 21 and on JDK 25, on the released 26.11.1 and on main. So it is not the Python bindings and not one JDK.
    • Most misses are found by the three immediate re-searches straight after; a few are still missing on them, and of those some appear within a second and a few are not found within 3 seconds (JDK 21, main: 2 of 28). A vector that was still missing after every writer had ended was seen once and twice in the two 200-scenario runs on JDK 25 and never on JDK 21. That last one could mean a lost index entry; I have not looked into it yet.
    • The graph rebuild contributes but is not all of it: raising the rebuild threshold to 1,000,000 mutations lowers the rate (63 of 200 scenarios on main became 16 of 100) but not to zero.
    • The engine logs one WARNING per recovery in these runs: LSMVectorIndex Recovered 1 vectors from in-memory index (page-parsed: N, in-memory: N+1). 1,212 and 1,222 lines in the two 200-scenario runs on JDK 25, none in the one-writer runs. I do not know yet whether it is a cause or a symptom.

    What I did not control: the runs shared the mini with other jobs at nice 19 and the load was not the same in each, so compare the rows by the yes or no of a miss, not as rates between JDKs (the rate rises with load, see the comment of 06:39Z).

    Next: upstream has an open PR on the same subject, ArcadeData/arcadedb#12087 ("a commit replay holds its vectors back from searches until its commit concludes"). I am building its head and running the same repro on both JDKs, and then I will file the engine issue with Repro.java and these numbers.

  11. tae898 commented on Nov 17, 2026

    @tae898
    CollaboratorAuthor

    Result of the check I announced in my last comment: the engine fix is in an open upstream pull request. I built the head of ArcadeData/arcadedb#12087 (ba7eabaa00, five commits over its merge-base 2eb615cd26: the three of #11969 and the two of that PR) and ran the same Java repro on JDK 21 and JDK 25, 100 scenarios each. With the PR head none of 40,000 committed inserts was missed (0 of 100 scenarios on each JDK). Its own merge-base, built the same way and run on the same cores, missed in 18 of 100 scenarios on JDK 21 (22 inserts) and 14 of 100 on JDK 25 (15 inserts), and main 515aff93c6 in 26 of 100 on JDK 21 and 63 of 200 on JDK 25. The table, the repro and the commands are in my comment on the PR: ArcadeData/arcadedb#12087 (comment)

    So I will not file a separate engine issue; the PR already addresses it (a commit replay holds its vectors back from searches until its commit concludes), and it conflicts with main at the moment. I will run the repro again on its rebased head and on the first main build that contains it.

    What this means for the bindings: until an engine release contains that change, tests/test_vector_read_your_writes.py can fail on any CI run when the load is high, on the released 26.11.1 engine that both published wheels carry, and so can an application that searches for a vector right after committing it from several threads. I have not changed any test or wheel for it; whether to mark the test as a known flake until then is open.

  12. tae898 commented on Nov 19, 2026

    @tae898
    CollaboratorAuthor

    Update on the race: I ran the same Java repro on the rebased head of ArcadeData/arcadedb#12087 (de3fd1ac10, with main merged in), 100 scenarios on each JDK, and no scenario missed a vector: 0 of 100 on Temurin 25.0.5 and 0 of 100 on Temurin 21.0.13. So the engine fix still holds after its rebase. It is still an open pull request and not in a release, so the published 26.11.1 wheels still carry the race, and I have not changed any test or wheel. I will run it again on the first main build that contains the fix.

  13. tae898 commented on Nov 21, 2026

    @tae898
    CollaboratorAuthor

    I traced the race to the engine's commit-to-search path in 26.11.1 (a3a5d12457, the engine the failing runs link), and it is the one that ArcadeData/arcadedb#12087 fixes. A vector's commit has two phases. In phase 1 the index replay hands out the vector id and publishes its location and its delta-buffer entry (LSMVectorIndex.put, persistVectorWithLocation); the pages become readable only in phase 2 (TransactionContext.commit2ndPhase). A graph rebuild takes its id snapshot (deltaSnapshotId = nextId.get() in buildGraphFromScratchExclusively), reads only published pages without a file lock, and at the end trims the delta buffer to vectorId >= deltaSnapshotId. A commit that sits between its two phases when the snapshot is taken is trimmed as already in the graph without having been read, and the next search misses it until a later rebuild reads the pages. As I read the test and rebuildGraphBeforeSearch, it hits this because its 300 to 500 vector index (below the 1,000 vector limit for background rebuilds) rebuilds synchronously on the searching thread once 100 changes are pending, while three other writers commit.

    The failing runs did not contain the fix: the scheduled run 43799876573 (attempt 1; the re-run passed) links engine 26.11.1, #11969 merged on 2026-11-17 and #12087 is still open (head de3fd1ac10). A second miss on the same day, in run 43863995512 on PR #410 (attempt 1; the re-run passed), came from the Linux Python 3.12 leg of the JVM wheel, a different wheel and platform, which fits the engine as the cause. Which exact sub-window loses the vector (the Recovered N vectors from in-memory index path of the rebuild, or the trim before it) is not established. On an idle laptop with 4 writers I got 0 misses in 8,600 committed inserts over 43 scenarios, although the Recovered line appeared 65 times. So a miss seems to need a commit thread that is slow between its two phases, which fits the dependence on load in the earlier loops; that is an inference, not traced to a line. Next: a deterministic Java repro that splits one commit between commit1stPhase and commit2ndPhase with a rebuild in between (on 26.11.1 and on the #12087 head), and a CI run on an engine snapshot that contains the fix. I am not changing the test: it checks the property an agent loop needs.

  14. tae898 commented on Nov 21, 2026

    @tae898
    CollaboratorAuthor

    Closing the loop on the trace in my last comment, with a repro that has no race to win: it splits one commit by hand between TransactionContext.commit1stPhase and commit2ndPhase (both public, the Raft path drives them the same way), runs a vector graph rebuild on another thread in between, and then searches for the vector.

    Engine Rounds with the committed vector missing from the search after the commit
    26.11.1 (a3a5d12457) 5 of 5 (rebuildBlockedByInFlightCommit=true each time; a later buildVectorGraphNow() finds it every time)
    head of ArcadeData/arcadedb#12087 (de3fd1ac10) 0 of 5

    So the miss follows from the order of the two phases alone, and the pull request closes it. This is one JDK (25.0.4.1 for the run; the engine of the second row was built with GraalVM 25.0.2's javac), and the earlier comments here have the 100-scenario runs on Temurin 21 and 25. I ran it on the mini in the JDK 25 image of the 26.11.1 build, with the engine jars of that image for the first row and ./mvnw -pl engine -am package of the pull request's head for the second.

    PhaseSplitRepro.java
    // Deterministic repro for ArcadeData/arcadedb-python#409 (the order is fixed by hand, so there is no race to win).
    // One commit is split by hand between its two phases (TransactionContext.commit1stPhase then commit2ndPhase, both public: the Raft path drives them
    // the same way), a vector-graph rebuild runs in between on another thread, then the vector is searched for. No race to win: the order is fixed.
    //
    // Expected on 26.11.1 (a3a5d12457): the search after commit2ndPhase does not return the vector (hypothesis, from the code: the rebuild takes its id
    // snapshot after the replay handed the id out, reads only published pages, trims the delta buffer below the snapshot). A later buildVectorGraphNow()
    // finds it again. With the head of ArcadeData/arcadedb#12087 (de3fd1ac10) the search must return it.
    //
    // Run (classpath: the engine jars of the release, e.g. ~/build-scratch/repro409/lib-rel/*; JDK 21 or 25):
    //   java -cp '/home/tk/build-scratch/repro409/lib-rel/*' PhaseSplitRepro.java /home/tk/build-scratch/pr409/tmp
    import com.arcadedb.database.Database;
    import com.arcadedb.database.DatabaseFactory;
    import com.arcadedb.database.DatabaseInternal;
    import com.arcadedb.database.Document;
    import com.arcadedb.database.RID;
    import com.arcadedb.database.Record;
    import com.arcadedb.database.TransactionContext;
    import com.arcadedb.index.IndexInternal;
    import com.arcadedb.index.TypeIndex;
    import com.arcadedb.index.vector.LSMVectorIndex;
    import com.arcadedb.schema.Schema;
    import com.arcadedb.schema.TypeLSMVectorIndexBuilder;
    import com.arcadedb.serializer.json.JSONObject;
    import com.arcadedb.utility.Pair;
    
    import java.nio.file.Files;
    import java.nio.file.Path;
    import java.util.ArrayList;
    import java.util.List;
    import java.util.Random;
    
    public class PhaseSplitRepro {
      static final int DIM = 8, BASE = 300, K = 5, EF = 400;
    
      static float[] vec(final int i) {
        final Random rnd = new Random(i);
        final float[] v = new float[DIM];
        double n = 0;
        for (int d = 0; d < DIM; d++) {
          v[d] = (float) (rnd.nextDouble() * 2 - 1);
          n += v[d] * v[d];
        }
        n = Math.sqrt(n);
        for (int d = 0; d < DIM; d++)
          v[d] /= n;
        return v;
      }
    
      /** The vids the search returns, nearest first (the binding's search: findNeighborsFromVector, hits loaded by RID). */
      static List<Integer> search(final Database db, final List<LSMVectorIndex> subs, final int vid) {
        final List<Integer> out = new ArrayList<>();
        for (final LSMVectorIndex lsm : subs) {
          try {
            for (final Pair<RID, Float> p : lsm.findNeighborsFromVector(vec(vid), K, EF)) {
              final Record r = db.lookupByRID(p.getFirst(), true);
              out.add(((Number) ((Document) r.asDocument()).get("vid")).intValue());
            }
          } catch (final Exception e) {
            out.add(-1); // a search that throws (RecordNotFoundException on an RID whose record is not readable yet) is a miss too
            System.out.println("  search threw " + e);
          }
        }
        return out;
      }
    
      public static void main(final String[] args) throws Exception {
        final Path dir = Files.createTempDirectory(Path.of(args[0]), "phasesplit-");
        final Database db = new DatabaseFactory(dir.resolve("db").toString()).create();
        try {
          db.command("sql", "CREATE VERTEX TYPE Doc");
          db.command("sql", "CREATE PROPERTY Doc.embedding ARRAY_OF_FLOATS");
          db.begin();
          for (int i = 0; i < BASE; i++)
            db.command("sql", "INSERT INTO Doc SET vid = ?, embedding = ?", i, vec(i));
          db.commit();
    
          final TypeLSMVectorIndexBuilder b = (TypeLSMVectorIndexBuilder) db.getSchema().buildTypeIndex("Doc", new String[] { "embedding" })
              .withType(Schema.INDEX_TYPE.LSM_VECTOR);
          b.withDimensions(DIM);
          b.withSimilarity("cosine");
          b.withMaxConnections(32);
          b.withBeamWidth(100);
          b.withQuantization("INT8");
          final JSONObject meta = new JSONObject();
          meta.put("addHierarchy", true);
          b.withMetadata(meta);
          final TypeIndex index = b.create();
          final List<LSMVectorIndex> subs = new ArrayList<>();
          for (final IndexInternal s : index.getSubIndexes())
            if (s instanceof LSMVectorIndex l)
              subs.add(l);
          for (final LSMVectorIndex l : subs)
            l.buildVectorGraphNow();
    
          int misses = 0;
          for (int round = 0; round < 5; round++) {
            final int vid = BASE + round;
            db.begin();
            db.command("sql", "INSERT INTO Doc SET vid = ?, embedding = ?", vid, vec(vid));
            final TransactionContext tx = ((DatabaseInternal) db).getTransaction();
            final TransactionContext.TransactionPhase1 phase1 = tx.commit1stPhase(true); // replay done: id, location and delta entry published; pages not
            final Thread rebuild = new Thread(() -> {
              for (final LSMVectorIndex l : subs)
                l.buildVectorGraphNow(); // reads the published pages only, between the two phases
            }, "rebuild-between-phases");
            rebuild.start();
            rebuild.join(30_000);
            final boolean rebuildBlocked = rebuild.isAlive();
            tx.commit2ndPhase(phase1); // pages published, the commit has returned
            if (rebuildBlocked)
              rebuild.join(30_000);
            final List<Integer> after = search(db, subs, vid);
            final boolean found = !after.isEmpty() && after.get(0) == vid;
            System.out.println("round " + round + " vid " + vid + ": rebuildBlockedByInFlightCommit=" + rebuildBlocked + " nearestAfterCommit=" + after
                + (found ? " FOUND" : " MISSED"));
            if (!found) {
              misses++;
              for (final LSMVectorIndex l : subs)
                l.buildVectorGraphNow();
              final List<Integer> healed = search(db, subs, vid);
              System.out.println("   after one more buildVectorGraphNow: " + healed + (!healed.isEmpty() && healed.get(0) == vid ? " FOUND" : " MISSED"));
            }
          }
          System.out.println("TOTALS rounds=5 missed=" + misses);
          System.exit(misses == 0 ? 0 : 1);
        } finally {
          db.drop();
        }
      }
    }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingconcurrencyThreads, transactions and conflictsgithub_actionsPull requests that update GitHub Actions codenativeThe arcadedb package (native library build)pythonPull requests that update python codevectorVector indexes and find_nearest

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions