Repository navigation
test(#4888): deepen e2e-js to the full Bolt conformance spec - #4990
Conversation
Mirror the merged e2e-python/e2e-go/e2e-csharp siblings: all 39 spec.yaml scenarios in a new js-bolt-conformance.test.js, beer + type-matrix fixtures seeded over HTTP, TLS via a keytool-generated derived image, it.failing for the 6 known gaps (#4890), 2 infra skips. Existing VLP regression untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
11 tasks: fixture harness + connection, auth, transactions (w/ two-writer race helper), causal+multi-db, result-handling, type round-trip A/B, errors, protocol, TLS derived-image, README+coverage proof. TDD against a live container; it.failing for 6 gaps, it.skip for 2 infra scenarios. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add js-bolt-conformance.test.js exercising every bolt/conformance/spec.yaml scenario against the official neo4j-driver, mirroring the merged e2e-python/ e2e-go/e2e-csharp suites: - beer default DB + type-matrix.cypher seeded over HTTP (never over Bolt) - TLS CONN-002/CONN-005 via a keytool self-signed cert baked into a derived image (testcontainers fromDockerfile); self-skip if keytool is absent - 6 known gaps asserted with Jest it.failing (the xfail-strict equivalent), all tracking #4890: RESULT-004, TYPE-003, TYPE-011, TYPE-012, ERR-002, PROTO-002 - 2 infra scenarios skipped: CONN-004 (3-node HA), ERR-003 (raw socket) - honors ARCADEDB_DOCKER_IMAGE; existing js-bolt-e2e.test.js left untouched Local run: 36 pass / 2 skip; CONN-003 (neo4j:// routing) passes on Linux CI but times out on macOS/Docker Desktop where bridge IPs are not host-routable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Tick the box to add this pull request to the merge queue (same as
|
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Coverage variation | ✅ +24.13% coverage variation |
| Diff coverage | ✅ ∅ diff coverage |
Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (cdb4573) 134228 57647 42.95% Head commit (2b919d8) 166035 (+31807) 111375 (+53728) 67.08% (+24.13%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>
Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#4990) 0 0 ∅ (not applicable) Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
There was a problem hiding this comment.
Code Review
This pull request implements the e2e-js Bolt conformance suite covering all 39 scenarios in bolt/conformance/spec.yaml against the official neo4j-driver, along with its design document, implementation plan, and README. The review feedback highlights several improvement opportunities: wrapping keytool executions in a try-catch block to prevent test suite crashes, cleaning up the temporary certificate directory in afterAll, instantiating new GenericContainer instances from the built image tag to avoid mutating a shared instance, and pulling a record before calling result.summary() to fully conform to the RESULT-003 spec scenario.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| function generateTlsCerts() { | ||
| let keytool; | ||
| try { | ||
| keytool = execFileSync("which", ["keytool"]).toString().trim(); | ||
| } catch (_) { | ||
| return null; | ||
| } | ||
| const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bolt-tls-")); | ||
| const keystore = path.join(dir, "keystore.p12"); | ||
| const truststore = path.join(dir, "truststore.jks"); | ||
| const cert = path.join(dir, "bolt.cer"); | ||
| execFileSync(keytool, [ | ||
| "-genkeypair", "-alias", "bolt", "-keyalg", "RSA", "-keysize", "2048", | ||
| "-validity", "3650", "-keystore", keystore, "-storetype", "PKCS12", | ||
| "-storepass", TLS_STORE_PASSWORD, "-keypass", TLS_STORE_PASSWORD, | ||
| "-dname", "CN=localhost, OU=ArcadeDB, O=ArcadeDB, L=Test, ST=Test, C=US", | ||
| ]); | ||
| execFileSync(keytool, [ | ||
| "-exportcert", "-alias", "bolt", "-keystore", keystore, | ||
| "-storetype", "PKCS12", "-storepass", TLS_STORE_PASSWORD, "-file", cert, | ||
| ]); | ||
| execFileSync(keytool, [ | ||
| "-importcert", "-alias", "bolt", "-keystore", truststore, | ||
| "-storetype", "JKS", "-storepass", TLS_STORE_PASSWORD, "-file", cert, | ||
| "-noprompt", | ||
| ]); | ||
| return dir; | ||
| } |
There was a problem hiding this comment.
If keytool is present on the system but fails to execute (e.g., due to permission issues, environment-specific security policies, or JDK version differences), the unhandled exception will crash the entire test suite. Wrapping the keytool executions in a try-catch block ensures that any failure is caught gracefully, the temporary directory is cleaned up, and the function returns null to safely skip the TLS tests.
function generateTlsCerts() {
let keytool;
try {
keytool = execFileSync("which", ["keytool"]).toString().trim();
} catch (_) {
return null;
}
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bolt-tls-"));
const keystore = path.join(dir, "keystore.p12");
const truststore = path.join(dir, "truststore.jks");
const cert = path.join(dir, "bolt.cer");
try {
execFileSync(keytool, [
"-genkeypair", "-alias", "bolt", "-keyalg", "RSA", "-keysize", "2048",
"-validity", "3650", "-keystore", keystore, "-storetype", "PKCS12",
"-storepass", TLS_STORE_PASSWORD, "-keypass", TLS_STORE_PASSWORD,
"-dname", "CN=localhost, OU=ArcadeDB, O=ArcadeDB, L=Test, ST=Test, C=US",
]);
execFileSync(keytool, [
"-exportcert", "-alias", "bolt", "-keystore", keystore,
"-storetype", "PKCS12", "-storepass", TLS_STORE_PASSWORD, "-file", cert,
]);
execFileSync(keytool, [
"-importcert", "-alias", "bolt", "-keystore", truststore,
"-storetype", "JKS", "-storepass", TLS_STORE_PASSWORD, "-file", cert,
"-noprompt",
]);
return dir;
} catch (_) {
try {
fs.rmSync(dir, { recursive: true, force: true });
} catch (__) {}
return null;
}
}There was a problem hiding this comment.
Done in 1ac9148. Cert generation is wrapped in try-catch: a present-but-failing keytool now removes the temp dir and returns null so the TLS scenarios skip instead of crashing the suite.
| afterAll(async () => { | ||
| if (tlsRequired) await tlsRequired.stop(); | ||
| if (tlsOptional) await tlsOptional.stop(); | ||
| }); |
There was a problem hiding this comment.
The temporary directory certDir created in beforeAll is never cleaned up, leaving leftover files (keystore.p12, truststore.jks, bolt.cer, Dockerfile) in the OS temp directory after every test run. We should clean it up in afterAll using fs.rmSync.
afterAll(async () => {
if (tlsRequired) await tlsRequired.stop();
if (tlsOptional) await tlsOptional.stop();
if (certDir) {
try {
fs.rmSync(certDir, { recursive: true, force: true });
} catch (_) {
// ignore cleanup errors
}
}
});There was a problem hiding this comment.
Done in 1ac9148. The TLS afterAll now fs.rmSync(certDir, {recursive:true, force:true}) inside a try-catch.
| tlsRequired = await builtImage | ||
| .withExposedPorts(2480, 7687) | ||
| .withEnvironment({ JAVA_OPTS: tlsJavaOpts("REQUIRED") }) | ||
| .withStartupTimeout(120000) | ||
| .withWaitStrategy( | ||
| Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching( | ||
| (sc) => sc === 204 | ||
| ) | ||
| ) | ||
| .start(); |
There was a problem hiding this comment.
Reusing and mutating the shared builtImage instance across multiple tests is risky because tlsRequired is still running when CONN-005 starts. Mutating the shared GenericContainer instance's environment or configuration can lead to unexpected side effects or test pollution. Instead, instantiate a new GenericContainer using the built image tag "arcadedb-bolt-tls-test:latest".
| tlsRequired = await builtImage | |
| .withExposedPorts(2480, 7687) | |
| .withEnvironment({ JAVA_OPTS: tlsJavaOpts("REQUIRED") }) | |
| .withStartupTimeout(120000) | |
| .withWaitStrategy( | |
| Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching( | |
| (sc) => sc === 204 | |
| ) | |
| ) | |
| .start(); | |
| tlsRequired = await new GenericContainer("arcadedb-bolt-tls-test:latest") | |
| .withExposedPorts(2480, 7687) | |
| .withEnvironment({ JAVA_OPTS: tlsJavaOpts("REQUIRED") }) | |
| .withStartupTimeout(120000) | |
| .withWaitStrategy( | |
| Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching( | |
| (sc) => sc === 204 | |
| ) | |
| ) | |
| .start(); |
There was a problem hiding this comment.
Done in 1ac9148. buildTlsImage now returns the image tag and CONN-002 starts a fresh new GenericContainer(tag), so no shared builder instance is mutated.
| tlsOptional = await builtImage | ||
| .withExposedPorts(2480, 7687) | ||
| .withEnvironment({ JAVA_OPTS: tlsJavaOpts("OPTIONAL") }) | ||
| .withStartupTimeout(120000) | ||
| .withWaitStrategy( | ||
| Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching( | ||
| (sc) => sc === 204 | ||
| ) | ||
| ) | ||
| .start(); |
There was a problem hiding this comment.
Instantiate a new GenericContainer using the built image tag "arcadedb-bolt-tls-test:latest" to avoid mutating the shared builtImage instance.
| tlsOptional = await builtImage | |
| .withExposedPorts(2480, 7687) | |
| .withEnvironment({ JAVA_OPTS: tlsJavaOpts("OPTIONAL") }) | |
| .withStartupTimeout(120000) | |
| .withWaitStrategy( | |
| Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching( | |
| (sc) => sc === 204 | |
| ) | |
| ) | |
| .start(); | |
| tlsOptional = await new GenericContainer("arcadedb-bolt-tls-test:latest") | |
| .withExposedPorts(2480, 7687) | |
| .withEnvironment({ JAVA_OPTS: tlsJavaOpts("OPTIONAL") }) | |
| .withStartupTimeout(120000) | |
| .withWaitStrategy( | |
| Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching( | |
| (sc) => sc === 204 | |
| ) | |
| ) | |
| .start(); |
There was a problem hiding this comment.
Done in 1ac9148. CONN-005 likewise starts a fresh new GenericContainer(tag).
| // In neo4j-driver v6, result.summary() is the consume/DISCARD path: it | ||
| // ends the stream and returns a ResultSummary WITHOUT materializing the | ||
| // records into a user-facing array (unlike `await result`). Driving the | ||
| // async iterator first and then calling summary() deadlocks, so this | ||
| // consumes via summary() directly. | ||
| const result = s.run("MATCH (b:Beer) RETURN b.name AS name LIMIT 5"); | ||
| const summary = await result.summary(); | ||
| expect(summary).toBeDefined(); | ||
| expect(summary.query).toBeDefined(); |
There was a problem hiding this comment.
The test currently calls result.summary() directly without pulling any records first, which does not fully conform to the spec scenario RESULT-003 ("When the driver PULLs 1 row of a 5-row result then calls result.consume() (DISCARD)"). We can pull exactly 1 row using for await...break (which properly closes the async iterator and avoids any deadlock) before calling result.summary().
| // In neo4j-driver v6, result.summary() is the consume/DISCARD path: it | |
| // ends the stream and returns a ResultSummary WITHOUT materializing the | |
| // records into a user-facing array (unlike `await result`). Driving the | |
| // async iterator first and then calling summary() deadlocks, so this | |
| // consumes via summary() directly. | |
| const result = s.run("MATCH (b:Beer) RETURN b.name AS name LIMIT 5"); | |
| const summary = await result.summary(); | |
| expect(summary).toBeDefined(); | |
| expect(summary.query).toBeDefined(); | |
| const result = s.run("MATCH (b:Beer) RETURN b.name AS name LIMIT 5"); | |
| for await (const _record of result) { | |
| break; // pull 1 and discard remainder | |
| } | |
| const summary = await result.summary(); | |
| expect(summary).toBeDefined(); | |
| expect(summary.query).toBeDefined(); |
There was a problem hiding this comment.
Done in 1ac9148. RESULT-003 now pulls one row via 'for await ... break' (which closes the iterator cleanly) before result.summary(). Verified locally it does not deadlock and passes.
Address Gemini review on PR #4990: - wrap keytool cert generation in try-catch: keytool present-but-failing now cleans up the temp dir and skips TLS instead of crashing the suite - clean up the temp cert dir in the TLS afterAll (was leaked) - start each TLS container from the built image tag via a fresh GenericContainer instead of reusing/mutating one shared builder instance - RESULT-003 pulls one row (for await...break, which closes the iterator cleanly) before summary(), matching the spec's 'PULL 1 then DISCARD' step Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Code review: e2e-js Bolt conformance suite (#4888)Reviewed the full diff (test-only + docs). Overall this is a high-quality, carefully-documented conformance suite that faithfully mirrors the merged Python/Go/C# siblings. The A few observations, all minor: 1. TLS derived image is never cleaned up (parity gap). 2. TX-005 / ERR-004 assert less than the Python sibling. The JS race tests assert only that a 3. keytool-absent path reports CONN-002/005 as PASSED, not SKIPPED. 4. Race-based tests are inherently timing-sensitive. TX-005/ERR-004 depend on a real write-write conflict materializing within the 500ms window. The code and comments already acknowledge this and keep the unconditional diagnostic 5. CONN-003 not skipped on macOS. Documented clearly in the README (Docker-bridge address not host-routable on Docker Desktop), and it passes on the Linux CI runners, so this is an accepted local-only failure rather than a defect. No action needed - confirming I read it as intentional. Things done well
None of the above is blocking. Items 1 and 2 are the ones I would most encourage tightening for true parity with the Python suite. |
Address Claude review on PR #4990: - TX-005/ERR-004 now also assert no racing error is a non-retryable ClientError/DatabaseError (noNonRetryableErrors) - the part that actually proves the conflict is retryable, matching the Python sibling - CONN-002/CONN-005 use a load-time keytool gate (it.skip when absent) so the keytool-less path reports as SKIPPED, not a misleading PASS - remove the throwaway derived TLS image in afterAll (docker rmi), parity with the Python suite's teardown Verified locally: TX-005/ERR-004 and CONN-002/005 pass; image removed on teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Addressed in 28c21e0 (verified locally against the branch image): 1. TLS derived image cleanup. 2. TX-005/ERR-004 negative assertion. Added 3. keytool-absent honest skip. Replaced the in-body 4. Race timing-sensitivity. Kept as-is with the unconditional diagnostic 5. CONN-003 on macOS. Correct - it is an intentional local-only failure (Docker-bridge address not host-routable on Docker Desktop) that passes on the Linux CI runners, same as the merged siblings. No change. |
PR Review:
|
Reword the PROTO-002 comment so it no longer reads as contradicting the literal assertion: the body asserts the Neo4j-correct '>= 5' and it.failing inverts it, staying green while 4.x is negotiated and flipping red on 5.x. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Thanks for the second pass. Status against this review:
|
PR Review:
|
Review:
|
|
Disposition of this pass (all 6 are non-blocking; keeping current behavior, with reasoning):
No code change for this pass - items 1-6 are all accepted-as-parity or inherited. The two earlier substantive findings (negative race assertion, TLS image cleanup) were fixed in 28c21e0, and the PROTO-002 comment nit in 2b919d8. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4990 +/- ##
============================================
- Coverage 65.32% 65.31% -0.02%
+ Complexity 817 815 -2
============================================
Files 1682 1682
Lines 134228 134232 +4
Branches 28712 28713 +1
============================================
- Hits 87685 87674 -11
- Misses 34526 34536 +10
- Partials 12017 12022 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Summary
Closes #4888 (epic #4882, Group B). Deepens the
e2e-jssuite from connect + WHERE-filter reads to the full 39-scenario Bolt conformance spec (bolt/conformance/spec.yaml, #4883), at parity with the mergede2e-python(#4885),e2e-csharp(#4886), ande2e-go(#4887) suites.A new
e2e-js/src/js-bolt-conformance.test.jsimplements every scenario against the officialneo4j-driver, mirroring the established sibling pattern. The existing VLP regression (js-bolt-e2e.test.js, #4452/#4271) is left untouched.What's covered
All 39 scenarios, named
[AREA-NNN]for grep-able traceability:bolt://,neo4j://routing, and TLSbolt+ssc://(CONN-002/005 via a keytool self-signed cert baked into a derived image)nonerejected), transactions (autocommit, explicit, managedexecuteWrite, transient-retry), bookmarks, multi-database isolationFixture strategy
Mirrors the Python fixture exactly: beer default DB +
type-matrix.cypherseeded over HTTP only (never over Bolt, so seeding never exercises the serialization path under test). HonorsARCADEDB_DOCKER_IMAGE.Known gaps (not regressions)
The 6 server-side gaps the matrix surfaces are asserted with Jest's native
it.failing()(thexfail(strict=True)equivalent) - each asserts the Neo4j-correct behavior, stays green while the gap reproduces, and turns red (XPASS) the moment the server is fixed, forcing aspec.yamlupdate in the same PR. All track #4890:Skipped (infra): CONN-004 (needs a 3-node HA cluster), ERR-003 (needs a raw Bolt socket the managed driver can't produce).
Verification
Local run against a branch-built image: 36 pass / 2 skip.
CONN-003(neo4j://routing) passes on the Linux CI runners but times out on macOS/Docker Desktop, where the Docker-bridge address advertised in the ROUTE reply isn't host-routable - identical behavior to the merged Python/Go/C# suites. No new CI job needed: the existingjs-e2e-testsjob runsnpm test(Jest auto-discovers the new file) and already passesARCADEDB_DOCKER_IMAGE.jest-junitoutput is already wired.Out of scope (deferred)
Multi-version driver-band matrix + nightly run (#4891), published compatibility matrix/badge (#4892), server-side gap fixes (#4890).
Design + plan:
docs/superpowers/specs/2026-07-04-e2e-js-bolt-conformance-design.md,docs/superpowers/plans/2026-07-04-e2e-js-bolt-conformance.md.🤖 Generated with Claude Code