chore!: build and test the labs components from the submodule by ludamad · Pull Request #25318 · AztecProtocol/aztec-packages

ludamad · 2026-08-25T21:14:28Z

Builds and tests the labs components from the labs/ submodule against this tree, instead of the in-tree copies.

make labs-use-local (run by fast-labs/full-labs/bench-labs): points the labs manifests at the packages built here (labs-deps), records the providers' content hashes in labs/labs-aztec-toolchain/fnd-hashes, refreshes the lockfiles and commits the result as a marker commit. Every labs build and test cache key follows this tree: a change to a foundation package rebuilds and retests labs, an unrelated change does not, a dirty provider disables caching.
labs-fast covers what a foundation change can break: labs compiled against the portals with its unit/e2e tests, and the contracts against this tree's nargo/bb. labs-full adds docs, spartan, playground, the claude tooling tests and the benches; make bench-labs runs the labs benches on demand.
scripts/labs_env.sh defines how labs is driven from here (inherited ci3 root cleared, TEST_CMD_PREFIX so the foundation test engine can run labs test lines, TEST_CMD_SKIP from labs-patches/test_cmd_skip for the tests that mount or package only the labs checkout). The TXE starts from labs/yarn-project; noir-projects-txe-tests delegates to the submodule.
Labs patches: 0001 portals the protocol artifacts packages, 0002 the test-command prefix/skip hook, 0003 the toolchain hash from fnd-hashes in foundation mode (one code path for both modes). All queued for aztec-node.
Removes the fuzzing container builds (container-builds/avm-fuzzing-container, container-builds/fuzzing-container) and their image workflows.
CLAUDE.md documents the make targets, scripts and cache identity.

fcarreiro · 2026-08-26T13:34:45Z

I confirm that the network deploys/benches/scenario testing/devnet are not expected to run from this repo (neither for v5 nor v6). I think I deleted their calls in #25296 .

fcarreiro · 2026-08-27T09:27:40Z

+# bootstrap-side collectors (scripts/labs_test_cmds.sh).
+LABS_MAKE := $(ROOT)/scripts/labs_env.sh $(MAKE)
+
+labs-fast: labs-use-local


I was wondering if for fast mode, the fnd would just want to build and run the e2e tests, and all the rest could be in full. IDK if you'd really ever want to build the docs, spartan, playground. For labs, the more you run, the less chances of surprises when we bump a version, but there might be a reasonable tradeoff.

(for this PR in any case, it's fine, you/someone can remove them afterwards)

Done in #25331: labs-fast keeps yarn-project (+tests, e2e), aztec-nr, noir-contracts and the contract snapshots — what a foundation change can break; docs, spartan, playground and claude-tests move to labs-full with the benches. The pin-bump PR should run ci-full so bumps still exercise everything.

fcarreiro · 2026-08-27T09:28:53Z

+	  && (cd yarn-project && YARN_ENABLE_IMMUTABLE_INSTALLS=false yarn install --mode=update-lockfile) \
+	  && (cd docs && YARN_ENABLE_IMMUTABLE_INSTALLS=false yarn install --mode=update-lockfile) \


Could this update/bump the versions of other packages that are not just the changed ones?

Only what use-local rewrote, in normal conditions: yarn keeps existing resolutions for unchanged descriptors and re-resolves the portal ones (and prunes their now-unused npm entries). The exception is tag ranges such as latest, which it re-resolves whenever it touches the lock. The rewrite lives in the marker commit and is never exported, so nothing can leak upstream; the cost of a re-resolution would be moved labs cache keys. #25331 makes commit-use-local list any lock entries that changed outside the portal set, so it shows up in the build log.

fcarreiro · 2026-08-27T09:34:27Z

Should this be in /scripts? (since it's not expected to go upstream)

Kept here: the whole labs-patches/ directory is foundation-only tooling (bootstrap.sh, the tests, this list) — only the contents of the .patch files ever go upstream, as aztec-node PRs. The README in #25330 says so explicitly. Since the compose flavours are not wanted back, the list is permanent and documented in CLAUDE.md (#25331).

fcarreiro · 2026-08-27T09:45:58Z

+# hashes the same line the engine records.
+ define test
+ 	$(call run_command,$(1),$(ROOT)/$(2),\
+-	  export MAKEFILE_TARGET=$(1) && ./bootstrap.sh test_cmds $(3) | $(ROOT)/ci3/filter_test_cmds | $(ROOT)/ci3/atomic_append /tmp/test_cmds)


Instead of this patch, consider locally patching labs/.test_patterns.yml (or filter_test_cmds) to just add the patterns that you want to skip

(this is optional)

The patch exists for the cd labs prefix, which a test-pattern file cannot express: the foundation engine runs from its own root, so every labs line has to enter the submodule and clear the inherited ci3 root. .test_patterns.yml also has different semantics (flake handling, and it would skip in labs' own CI too). The skip list itself is a plain foundation file. The way this patch retires is upstreaming it as a generic hook in the test macro, which is what it already is.

fcarreiro · 2026-08-27T09:51:16Z

+-    # Composing the upstream content hashes keys rebuilds on exactly the same inputs that decide
+-    # whether the binaries themselves rebuild.
+-    #
+    # The identity is this directory's committed content, as in pinned mode: the driving


Looks like the two branches of the IF could be unified now? (modulo some smaller conditional inside)

IIUC the provided hash includes bb/nargo/noir_profiler so the checks, optional additions etc are moot

(this I would like to see changed if you agree that some of the code now wouldn't make sense)

Agreed, and went one step further in #25331: foundation mode no longer probes bin/ at all. The foundation records the optional binaries it built as an optional= line in fnd-hashes, so both modes are the same code — directory content hash plus the optional binaries expected on this machine (from the pins, or from that record). The hash is computable before provisioning in both modes, and the "binaries not found, build first" error is gone. Checked in the devbox: identical hash with and without bin/, different when the record changes.

fcarreiro · 2026-08-27T09:54:34Z

 }

+# The labs benches come from the submodule's own bench_cmds, run from labs/ with its ci3.
+function labs_bench_cmds {


Our benches run quickly (<5m), but I wouldn't mind if fnd doesnt run them regularly. We would realize when we bump versions.

Agreed. The labs benches are only in the default bench_cmds on #25321 (which wires labs_bench_cmds in); I'd drop them from the default there so the nightly benches stay foundation-only, keeping make bench-labs for on-demand runs — Adam's call whether to touch that approved PR or do it as a follow-up.

fcarreiro

Approved modulo small changes.

…ile drift note (#25331) Review follow-ups for #25318, as a separate PR into `ad/delete-in-tree-labs`. - **Patch 0003 — one `hash` path.** Foundation mode no longer probes `bin/` (and no longer refuses to run before a build): both modes hash the directory's committed content plus the optional binaries expected on this machine — predicted from the pins in pinned mode, and in foundation mode read from a new `optional=` line that `scripts/labs_fnd_hashes.sh` records with the provider hashes. Verified in the devbox: identical hash with and without `bin/` provisioned, a different hash when the `optional=` record changes, pinned mode unchanged. - **`labs-fast` scope.** Keeps `yarn-project` (+tests, e2e), `aztec-nr`, `noir-contracts`, `contract-snapshots-tests` — what a foundation change can break. `docs`, `spartan`, `playground`, `claude-tests` move to `labs-full` with the benches (the pin-bump PR should run `ci-full`). Saves the ~3 min of upstream-uncached work (`nargo check` in aztec-nr is kept; the docs example compile goes). - **Lockfile refresh.** `commit-use-local` now lists lock entries that changed outside the portal rewrite. Yarn keeps existing resolutions for unchanged descriptors; the exception is tag ranges (`latest`), which it re-resolves — that is now visible in the build log rather than silently moving every labs cache key. The rewrite lives in the marker commit and is never exported. - **CLAUDE.md** `<labs_build_tooling>`: the make targets, `scripts/labs_env.sh` / `labs_test_cmds.sh` / `labs_fnd_hashes.sh`, `labs-patches/test_cmd_skip` and the cache identity. Not changed, with reasons in the threads: the skip list stays in `labs-patches/` (the whole directory is foundation-only tooling; only the `.patch` contents go upstream), and patch 0002 stays (it exists for the `cd labs` prefix, which a test-pattern file cannot express). Co-authored-by: Adam Domurad <adomurad@Adams-MBP.ht.home>

- Submodules in labs `aztec-labs-eng/aztec-node` repo and builds from it instead of the in-tree copy - Adds a `labs-patches` series AI Summary: ``` **Stack** (merge in order): #25306 labs submodule + patch-series tooling → #25318 build and test labs from the submodule → #25321 delete the in-tree labs copies Adds `aztec-labs-eng/aztec-node` as the shallow `labs/` submodule (tracking `main`, `update = none`) and `labs-patches/`, the foundation's patch series on top of it. **Nothing builds from `labs/` yet**: CI clones and checks it out, applies an (empty) series, and runs the tooling's lifecycle test. Experiments on labs code land here as patches first; upstream later. `labs-patches/bootstrap.sh`: - `apply` (root bootstrap, `post-merge`/`post-checkout` hooks, `make labs-patched`) checks the submodule out at the gitlink and `git am`s the series; fixed committer identity and author dates make the applied SHAs identical on every machine. The series is recognised by *content* (patch ids, plus the commits the state records as applied), so cold applies, bumps across shallow fetches and edited series still tell it apart from work in progress — which `apply` and `bump` refuse to discard (`LABS_PATCHES_FORCE=1` is the last resort); uncommitted edits are stashed, never reset. - `export` regenerates the series from commits in `labs/`; `bump <ref>` moves the pin and re-applies; `check` verifies the series in a temporary worktree; `commit-use-local` records a build's manifest rewrite as a marker commit the series ignores (used by #25318). - The pre-commit hook refuses staging a patched or marker head as the gitlink (they exist only in your clone); hooks are guarded so older branches keep committing. - `labs-patches/tests/lifecycle_test` (12 checks) runs with `check` under `make labs-patches-tests`, in `fast-foundation`. Also: CLAUDE.md rules for the submodule (never `git add labs` by hand; `git add -u`/`commit -a` stage the gitlink), CODEOWNERS for the new files, and the `.claude` layout test exempts `labs/` like `noir/noir-repo`. ```

The labs components are built from the labs/ submodule against this tree and tested under this repo's test engine; make fast no longer builds the in-tree copies (their removal is the next change). labs-use-local runs the submodule toolchain's use-local against this checkout once the packages it portals into are built (labs-deps, including the protocol artifacts packages' dist/ from fnd-artifacts-stage), records the content hashes of every provider in labs/labs-aztec-toolchain/fnd-hashes, refreshes the lockfiles and commits the result as the marker commit, so the labs tree is clean for CI and every labs cache key follows this tree: a change to bb.js, wsdb, a circuit or an L1 contract rebuilds and retests labs, an unrelated change does not. The series carries the three labs-side pieces this needs (portals for the artifacts packages, the toolchain identity from fnd-hashes, and a TEST_CMD_PREFIX/TEST_CMD_SKIP hook on the labs test macro), all candidates to go upstream. scripts/labs_env.sh is the single definition of how the foundation drives labs (inherited ci3 root cleared, prefix into labs/, the compose-flavour skip list in labs-patches/test_cmd_skip); the Makefile's LABS_MAKE executes it and scripts/labs_test_cmds.sh sources it to collect the labs benches and the release compat tests. The test engine consults labs' quarantine list as well as this repo's. The TXE is started from labs/yarn-project and the txe tests are delegated to the submodule. aztec-up, release-image and the docker-compose based tests stay with labs CI: they package or mount the labs tree on its own, where the portals do not resolve. The chonk and ultrahonk bench-input scripts and bench_ivc build just the labs yarn-project (labs-yarn-project).

…mode The three *-artifacts packages stayed pinned to the published release in use-local mode, so a local circuit or protocol-contract change in the foundation tree never reached this repo's consumers without a release. The foundation tree stages their installable dist/ directories from the built circuits (noir-projects/fnd/bootstrap.sh stage_packages) before driving this repo, so use-local can portal them like every other package. Carried in the foundation patch series as `labs-patches/0001-feat-toolchain-portal-the-protocol-artifacts-package.patch`, added by AztecProtocol/aztec-packages#25318.

…repo A repo that carries this one as a subdirectory runs the shared test engine from its own root, where the root-relative commands emitted here do not resolve. The prefix is inserted after the hash so such a driver can cd into this checkout. Carried in the foundation patch series as `labs-patches/0002-feat-make-TEST_CMD_PREFIX-for-test-commands-collecte.patch`, added by AztecProtocol/aztec-packages#25318.

In foundation mode the toolchain hash called the provider bootstraps in the foundation checkout, which inherited this repo's exported ci3 root and broke, and only covered bb and noir: the packages the manifests portal into were not part of any cache key here. The driving checkout now records the content hashes of everything it provides in labs-aztec-toolchain/fnd-hashes, committed with the use-local rewrite, and this directory is hashed the same way as in pinned mode. A dirty provider disables caching here as it would there. Carried in the foundation patch series as `labs-patches/0003-feat-toolchain-foundation-mode-takes-its-identity-fr.patch`, added by AztecProtocol/aztec-packages#25318 and revised by AztecProtocol/aztec-packages#25333.

…#25363) ## Problem `bootstrap.sh build_and_test` runs the test engine **concurrently** with `make`, and each target streams its test commands to `/tmp/test_cmds` as soon as it builds. Those tests outlive the target that emitted them, so any later target that rewrites the same tree yanks files out from under a running `node`: - `Cannot find module '@aztec/foundation/config'`, `Tests: 0 total`, `Time: 0.104 s` — a `dest/` was deleted mid-resolution - `Bus error (core dumped)` with no jest output — a mapped file was replaced underneath the process Neither names a cause, and both land on whichever test happened to be running, so they read as unrelated flakes. The run halts on first failure, so one of these ends a 10,000+ test run. Two targets violated the ordering. ### 1. `yarn-project` built twice (labs) `labs-full` chained onto `labs-fast`, making two sub-make invocations. `make` only de-duplicates targets **within a process**, so the second invocation rebuilt `yarn-project` — a prerequisite of `playground`, `docs` and `yarn-project-benches`. Nothing stopped it: `yarn-project` is an existing directory that is not `.PHONY`, its prerequisite `labs-aztec-toolchain` *is* phony (so always "newer"), and `$(call build,...)` writes no stamp. That rebuild re-enters `yarn-project/bootstrap.sh`, which starts with `clean-lite` (`rm -rf` of every gitignored path, including all of `dest/`) and `npm_install_deps` (unconditional `node_modules` re-extraction) — while `labs-fast`'s tests were still reading that tree. Observed: `labs-fast` left at 23:53:12, `labs-full` entered at 23:53:12 and re-ran `yarn-project` at 23:55:02; a test container started 23:55:04 and died at 23:55:06. Both goal sets now run in a single sub-make, so `yarn-project` builds once. `docs`, `playground` and `yarn-project` each have their own `yarn.lock`, so no full goal writes the `yarn-project` workspace. Introduced by #25318, which split the single Makefile into root + submodule sub-makes. Pre-split this was one process with `yarn-project: bb-ts ... bb-avm-sim bb-cdb`, which gave both properties for free. ### 2. `bb-avm-sim` unordered against `bb-ts` Both install into the same `barretenberg/ts` `node_modules` — the hazard the existing `bb-cdb` comment already describes. `bb-ts` was never added to the chain, and it is the one that emits the bb.js test commands, so its tests run out of that `node_modules` while `bb-avm-sim`'s `npm_install_deps` re-extracts it. Observed: `build_bb_avm_sim` started 00:06:04; `native_socket.test.js` and `crs/net_crs.test.js` both died `Bus error` at 00:06:17, ~1s after launch. `net_crs.test.js` only downloads CRS, so no test-level bug explains both. Chain is now `bb-ts -> bb-avm-sim -> bb-cdb -> bb-avm-sim-cross-copy`. ## Verification `make -p` confirms the resolved graph: ``` labs-full: labs-use-local # no longer chains through labs-fast bb-avm-sim: ipc-codegen ipc-runtime bb-cpp-native bb-ts bb-cdb: ipc-codegen ipc-runtime bb-avm-sim ``` `make full` now makes one labs sub-make call covering both goal sets. dmesg from the failing runs shows no OOM and no filesystem error, consistent with SIGBUS from a file replaced under an mmap rather than resource exhaustion. ## Related, not fixed here - `labs/Makefile` omits `yarn-project` from `.PHONY` — the only one of its 8 directory-named targets missing, against the file's own stated rule. Separate PR: aztec-labs-eng/aztec-node#111. - `ci3/npm_install_deps` re-extracts `node_modules` on every call with no "already installed" check. Making it a no-op when the tree already matches `nm_hash` would remove this whole class rather than individual instances. ## Disabled: `labs-patches/0005` Renamed to `*.patch.disabled`. `apply` globs `*.patch`, so it stays in the queue but is not applied. **It is not deleted and nothing in it is lost.** The docs it adds carry `references:` frontmatter pointing at `l1-contracts/`, `noir-projects/fnd/`, `barretenberg/cpp/` and `avm-transpiler/src/`. Those live in this repo, but `docs/scripts/check_doc_references.sh` resolves references against `git rev-parse --show-toplevel`, which inside the submodule is `labs/`. So the docs build fails on paths that are perfectly valid one directory up. Verified against the pinned labs tree: the original patch reproduces the CI failure, and without it the checker passes (43 references, all resolve). **The references are correct — the checker is what's wrong.** It has no notion of the parent repo. Stripping the references would delete roughly a third of the patch's 157 reference lines (I tried it: removing just `l1-contracts` emptied 16 lines and left 18 failures across the other three trees). The proper fix is in the labs repo: resolve against the outer repo when one exists, or skip references whose first path segment is absent. Re-enable the patch then — by fixing the checker, not by deleting the metadata. This is unrelated to the races above; it surfaced only because fixing them let full runs get far enough to reach `labs/docs`.

ludamad requested review from a team and charlielye as code owners August 25, 2026 21:14

ludamad added the ci-no-squash label Aug 25, 2026

ludamad force-pushed the ad/delete-in-tree-labs branch 2 times, most recently from 31f4b80 to 490d17e Compare August 25, 2026 22:48

ludamad force-pushed the ad/labs-submodule branch from 310000e to 27182a3 Compare August 25, 2026 22:48

This was referenced Aug 25, 2026

chore!: delete the in-tree labs components #25321

Merged

chore: labs submodule with a foundation patch series #25306

Merged

ludamad force-pushed the ad/delete-in-tree-labs branch from 490d17e to e821508 Compare August 26, 2026 16:10

ludamad changed the title ~~chore!: delete the in-tree labs components~~ chore!: build and test the labs components from the submodule Aug 26, 2026

ludamad removed the ci-no-squash label Aug 26, 2026

ludamad force-pushed the ad/labs-submodule branch from 27182a3 to e751696 Compare August 26, 2026 18:12

ludamad force-pushed the ad/delete-in-tree-labs branch from e821508 to 1b65d99 Compare August 26, 2026 18:12

ludamad force-pushed the ad/labs-submodule branch from e751696 to 6ee4cc7 Compare August 26, 2026 18:57

ludamad force-pushed the ad/delete-in-tree-labs branch 2 times, most recently from a8931b4 to 5b56dff Compare August 26, 2026 19:14

ludamad force-pushed the ad/labs-submodule branch from 6ee4cc7 to e56cecc Compare August 26, 2026 19:14

ludamad force-pushed the ad/delete-in-tree-labs branch from 5b56dff to 9308726 Compare August 26, 2026 19:42

fcarreiro reviewed Aug 27, 2026

View reviewed changes

fcarreiro approved these changes Aug 27, 2026

View reviewed changes

ludamad mentioned this pull request Aug 27, 2026

chore(labs): one toolchain hash path, a leaner labs-fast, and a lockfile drift note #25331

Merged

ludamad force-pushed the ad/delete-in-tree-labs branch from e0613be to 42b2b5f Compare August 27, 2026 14:36

ludamad force-pushed the ad/labs-submodule branch from 294337a to 7235d44 Compare August 27, 2026 14:36

Base automatically changed from ad/labs-submodule to next August 27, 2026 16:03

ludamad force-pushed the ad/delete-in-tree-labs branch from 42b2b5f to 0389487 Compare August 27, 2026 17:20

ludamad merged commit 38fd5fc into next Aug 27, 2026
16 checks passed

ludamad deleted the ad/delete-in-tree-labs branch August 27, 2026 17:36

fcarreiro mentioned this pull request Aug 28, 2026

chore: sync the foundation labs-patches series aztec-labs-eng/aztec-node#109

Merged

charlielye mentioned this pull request Aug 30, 2026

fix(ci): build labs in one sub-make, serialize bb-avm-sim after bb-ts #25363

Merged

		&& (cd yarn-project && YARN_ENABLE_IMMUTABLE_INSTALLS=false yarn install --mode=update-lockfile) \
		&& (cd docs && YARN_ENABLE_IMMUTABLE_INSTALLS=false yarn install --mode=update-lockfile) \

Conversation

ludamad commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fcarreiro commented Aug 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fcarreiro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ludamad commented Aug 25, 2026 •

edited

Loading