Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.

JVM: opt-in JDK 25 AOT cache (ARCADEDB_AOT=1) with fault and concurrency probes - #490

Merged
tae898 merged 4 commits into
mainfrom
jvm/aot-cache-opt-in
Oct 24, 2026
Merged

tae898 merged 4 commits into
mainfrom
jvm/aot-cache-opt-in

Conversation

@tae898

@tae898 tae898 commented Oct 23, 2026

Copy link
Copy Markdown

What

An opt-in Project Leyden AOT cache (JDK 25, JEP 483/514/515) for the JVM wheel. ARCADEDB_AOT=1 or start_jvm(aot=True) (also jvm_kwargs={"aot": True}) lets the JVM keep a per-user cache of the classes it loads and links at start. It is off by default, and nothing changes for anyone who does not set it. New module arcadedb_embedded/aot.py, a small hook in start_jvm, tests in tests/test_aot.py, a section in docs/api/jvm.md and a line in docs/development/architecture.md.

  • Cache directory: $XDG_CACHE_HOME/arcadedb/aot or ~/.cache/arcadedb/aot (macOS ~/Library/Caches/arcadedb/aot, Windows %LOCALAPPDATA%\arcadedb\Cache\aot); ARCADEDB_AOT_CACHE=<file> overrides it. File name = hash of package version, bundled JRE version, jars (jar_fingerprint sha256 plus each jar's size and mtime), the exact class path (including JPype's org.jpype.jar, which the cache records too) and platform.
  • First process trains with -XX:AOTCacheOutput=<unique tmp> (the JVM writes it when the process exits normally); one trainer at a time via an exclusive lock file holding its pid, other processes start normally. The next start recovers a finished or dead trainer: the bundled JVM must accept the file (-XX:AOTMode=on -version), then an atomic rename and a sidecar with size and sha256. Use is -XX:AOTCache=<file> -XX:AOTMode=auto; a file whose size or sha256 differs from the sidecar is deleted and rebuilt, so a damaged cache is never mapped.
  • Every failure (unwritable or blocked directory, full disk, JRE below 25 or unreadable, missing JPype jar, lock error) is a normal start with a debug line on the arcadedb_embedded.aot logger; start_jvm never raises because of it. User-given -XX:AOT*, -XX:SharedArchiveFile or -Xshare flags win and the bindings add nothing.

Why

Measured on the laptop with ab.py (study jvm-aot-ab-laptop-20261022 in the benchmark notes, 12 clean pairs per figure): cold total -33 to -36 percent, JVM start -25 to -30 percent, database open -37 to -42 percent, and no detectable difference in steady-state inserts, lookups or scans. A tiny training run was as good as a broad one. A cache records absolute jar paths and jar mtime and size, so it cannot ship inside the wheel and has to be created on the user's machine.

Robustness matrix

Generated from the probe output (.notes/bench/repros/jvm-aot-robust-20261024: driver.py, child.py, job.sh, job.log, matrix.md). Every repetition is fresh processes and must end with a working database (CHECK ok); "mapped" means the JVM's own -Xlog:aot log shows it opened and mapped the cache. The probes ran this branch's aot.py and jvm.py on top of the merged-main wheel 26.11.1.dev0 (Corretto 25.0.4.1 jre), cores 0-5, through the laptop heavy-job wrapper, with the JVM output kept unfiltered per run.

probe variant repetitions passed checks per repetition
1-train-then-use - 10 10 no_leftovers, one_cache, run1_ok, run1_trained, run2_mapped, run2_ok, run3_mapped, run3_ok
2-race-8 - 20 20 after1_ok, after2_ok, all8_ok, cache_mapped_by_after2, no_leftovers, one_cache, trainers<
3-corrupt truncate-half 3 3 healed_by_r3, no_leftovers, one_cache, r1_not_using_bad, r1_ok, r2_ok, r3_ok
3-corrupt zero-length 3 3 healed_by_r3, no_leftovers, one_cache, r1_not_using_bad, r1_ok, r2_ok, r3_ok
3-corrupt random-middle-4k 3 3 healed_by_r3, no_leftovers, one_cache, r1_not_using_bad, r1_ok, r2_ok, r3_ok
3-corrupt random-whole 3 3 healed_by_r3, no_leftovers, one_cache, r1_not_using_bad, r1_ok, r2_ok, r3_ok
3-corrupt header-zeroed 3 3 healed_by_r3, no_leftovers, one_cache, r1_not_using_bad, r1_ok, r2_ok, r3_ok
3-corrupt meta-missing 3 3 healed_by_r3, no_leftovers, one_cache, r1_not_using_bad, r1_ok, r2_ok, r3_ok
3-corrupt meta-garbage 3 3 healed_by_r3, no_leftovers, one_cache, r1_not_using_bad, r1_ok, r2_ok, r3_ok
4-stale jar-touched 3 3 good_cache_mapped, good_runs_ok, no_leftovers, r1_not_using_old, r1_ok, r2_ok, r3_ok, retrained_and_used_by_r3
4-stale jar-replaced 3 3 good_cache_mapped, good_runs_ok, no_leftovers, r1_not_using_old, r1_ok, r2_ok, r3_ok, retrained_and_used_by_r3
4-stale version-changed 3 3 good_cache_mapped, good_runs_ok, no_leftovers, r1_not_using_old, r1_ok, r2_ok, r3_ok, retrained_and_used_by_r3
4-stale jar-touched-fixed-path 3 3 good_cache_mapped, good_runs_ok, no_leftovers, r1_not_using_old, r1_ok, r2_ok, r3_ok, retrained_and_used_by_r3
4-stale jar-replaced-fixed-path 3 3 good_cache_mapped, good_runs_ok, no_leftovers, r1_not_using_old, r1_ok, r2_ok, r3_ok, retrained_and_used_by_r3
5-readonly-dir - 3 3 all_ok, no_cache_written
5-path-blocked-by-file - 3 3 all_ok, not_trained
5-full-disk-tmpfs-1m - 3 3 3_of_3_checks_ok, trained_at_least_once
5-full-disk-tmpfs-20m - 3 3 3_of_3_checks_ok, trained_at_least_once
6-jdk24-fake - 3 3 all_ok, no_aot_flags, nothing_written
6-jre-release-unreadable - 3 3 all_ok, no_aot_flags
6-jvm-rejects-aot-flag-(documented-limit) - 3 3 ArcadeDBError_reported, exits_nonzero_cleanly, jvm_message_unfiltered, lock_released
6-opt-in-off - 3 3 no_aot_flags, no_dir, ok
7-trainer-killed - 5 5 all_ok, healed, no_leftovers

Total: 95 of 95 repetitions passed; SUMMARY runs=514 failed_probe_variants=0

Notes on the cases: probe 2 starts 8 processes at once with no cache, 20 rounds, and then needs two more runs to reach a mapped cache with exactly one cache file and no leftover temp or lock files. Probe 3 damages a good cache six ways plus a missing or garbage sidecar; the damaged file is never mapped and the cache is rebuilt by the third run. Probe 5 used a 1 MB and a 20 MB tmpfs mounted in a user namespace (unshare -rm, no root) as the cache location, a read-only directory, and a path blocked by a file. Probe 6 has no second JDK on this machine, so an older JRE is simulated by patching jre_identity (JDK 24, and an unreadable release file). Probe 7 SIGKILLs a training process of the probe itself.

Known limit (probe 6, "JVM rejects AOT flag"): if the JVM itself rejects a flag at creation, the start fails with the usual ArcadeDBError and the lock is freed; there is no in-process retry because JPype cannot start a second JVM after a failed one. The flags are only added for a bundled JRE of version 25 or newer, so this is not reachable today, and the probe pins the behaviour.

Tests

tests/test_aot.py, 27 tests: key derivation, per-platform cache directory, JRE identity, promotion, damaged caches, stale locks, fallbacks (mocked), override path, off by default. Run with uv run --no-project --with JPype1 --with pytest and PYTHONPATH=src pytest tests/test_aot.py --noconftest (27 passed). The pre-existing test_log_dir.py and test_jvm_args.py subprocess tests fail in a source tree without jars both on main and on this branch; they pass the same way against the wheel overlay.

Still open

  • Making it the default needs a new ab.py run on the release wheel (the study ran on the merged-main wheel).
  • A process that never exits normally (os._exit, SIGKILL) trains nothing, so the next start trains again; a long-lived first process holds the lock and others start without a cache for as long as it lives (or 30 minutes, then the lock is stale).
  • Windows and macOS code paths (cache directory, pid check) are covered by unit tests with mocks only, not run.

🤖 Generated with Claude Code

tae898 and others added 4 commits October 24, 2026 10:29
Per-user cache of the classes the JVM loads and links at start, keyed by package version, bundled JRE, jars and class path. One process trains at a time (lock file, unique temp file), the next start validates it with the bundled JVM and renames it into place; a cache whose size or sha256 differs is rebuilt. Every failure is a normal start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… training lock and re-raise

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The only subprocess runs the wheel's own java with an argv list, fixed flags
and our cache path; no shell, no user input. CI's Bandit command
(bandit 1.9.4, -c pyproject.toml, low/low) reports no issues.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
undo() restored the real sys.platform, so on macOS and Windows cache_dir()
left XDG_CACHE_HOME for ~/Library/Caches or %LOCALAPPDATA% and the damaged-cache
test looked at a directory the plan never used (5 failures per job). Reproduced
on Linux by forcing sys.platform=darwin: 5 failed before, 27 passed after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 force-pushed the jvm/aot-cache-opt-in branch from 5de6e08 to 3cbf5e0 Compare October 24, 2026 01:29
@tae898
tae898 merged commit 43d723c into main Oct 24, 2026
42 of 44 checks passed
@tae898
tae898 deleted the jvm/aot-cache-opt-in branch October 24, 2026 02:09
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant