Skip to content

test(#4888): deepen e2e-js to the full Bolt conformance spec - #4990

Merged
robfrank merged 7 commits into
mainfrom
feat/4888-e2e-js-bolt-conformance
Jul 4, 2026
Merged

robfrank merged 7 commits into
mainfrom
feat/4888-e2e-js-bolt-conformance

Conversation

@robfrank

@robfrank robfrank commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Closes #4888 (epic #4882, Group B). Deepens the e2e-js suite from connect + WHERE-filter reads to the full 39-scenario Bolt conformance spec (bolt/conformance/spec.yaml, #4883), at parity with the merged e2e-python (#4885), e2e-csharp (#4886), and e2e-go (#4887) suites.

A new e2e-js/src/js-bolt-conformance.test.js implements every scenario against the official neo4j-driver, mirroring the established sibling pattern. The existing VLP regression (js-bolt-e2e.test.js, #4452/#4271) is left untouched.

What's covered

All 39 scenarios, named [AREA-NNN] for grep-able traceability:

  • connection incl. bolt://, neo4j:// routing, and TLS bolt+ssc:// (CONN-002/005 via a keytool self-signed cert baked into a derived image)
  • auth (valid / invalid / none rejected), transactions (autocommit, explicit, managed executeWrite, transient-retry), bookmarks, multi-database isolation
  • result-handling (streaming, partial pull, discard, summary counters)
  • type round-trip (Node, Relationship, Path, ByteArray, nested list/map, null, temporal, Duration, Point)
  • errors (syntax, semantic, transient) and protocol (negotiation, 5.x, RESET)

Fixture strategy

Mirrors the Python fixture exactly: beer default DB + type-matrix.cypher seeded over HTTP only (never over Bolt, so seeding never exercises the serialization path under test). Honors ARCADEDB_DOCKER_IMAGE.

Known gaps (not regressions)

The 6 server-side gaps the matrix surfaces are asserted with Jest's native it.failing() (the xfail(strict=True) equivalent) - each asserts the Neo4j-correct behavior, stays green while the gap reproduces, and turns red (XPASS) the moment the server is fixed, forcing a spec.yaml update in the same PR. All track #4890:

Scenario Gap
RESULT-004 write counters never populated
TYPE-003 native Path never produced
TYPE-011 Duration unimplemented
TYPE-012 Point unimplemented
ERR-002 semantic error mapped as SyntaxError
PROTO-002 Bolt 5.x never advertised

Skipped (infra): CONN-004 (needs a 3-node HA cluster), ERR-003 (needs a raw Bolt socket the managed driver can't produce).

Verification

Local run against a branch-built image: 36 pass / 2 skip. CONN-003 (neo4j:// routing) passes on the Linux CI runners but times out on macOS/Docker Desktop, where the Docker-bridge address advertised in the ROUTE reply isn't host-routable - identical behavior to the merged Python/Go/C# suites. No new CI job needed: the existing js-e2e-tests job runs npm test (Jest auto-discovers the new file) and already passes ARCADEDB_DOCKER_IMAGE. jest-junit output is already wired.

Out of scope (deferred)

Multi-version driver-band matrix + nightly run (#4891), published compatibility matrix/badge (#4892), server-side gap fixes (#4890).

Design + plan: docs/superpowers/specs/2026-07-04-e2e-js-bolt-conformance-design.md, docs/superpowers/plans/2026-07-04-e2e-js-bolt-conformance.md.

🤖 Generated with Claude Code

robfrank and others added 4 commits July 4, 2026 17:33
Mirror the merged e2e-python/e2e-go/e2e-csharp siblings: all 39 spec.yaml
scenarios in a new js-bolt-conformance.test.js, beer + type-matrix fixtures
seeded over HTTP, TLS via a keytool-generated derived image, it.failing for
the 6 known gaps (#4890), 2 infra skips. Existing VLP regression untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
11 tasks: fixture harness + connection, auth, transactions (w/ two-writer
race helper), causal+multi-db, result-handling, type round-trip A/B, errors,
protocol, TLS derived-image, README+coverage proof. TDD against a live
container; it.failing for 6 gaps, it.skip for 2 infra scenarios.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add js-bolt-conformance.test.js exercising every bolt/conformance/spec.yaml
scenario against the official neo4j-driver, mirroring the merged e2e-python/
e2e-go/e2e-csharp suites:

- beer default DB + type-matrix.cypher seeded over HTTP (never over Bolt)
- TLS CONN-002/CONN-005 via a keytool self-signed cert baked into a derived
  image (testcontainers fromDockerfile); self-skip if keytool is absent
- 6 known gaps asserted with Jest it.failing (the xfail-strict equivalent),
  all tracking #4890: RESULT-004, TYPE-003, TYPE-011, TYPE-012, ERR-002,
  PROTO-002
- 2 infra scenarios skipped: CONN-004 (3-node HA), ERR-003 (raw socket)
- honors ARCADEDB_DOCKER_IMAGE; existing js-bolt-e2e.test.js left untouched

Local run: 36 pass / 2 skip; CONN-003 (neo4j:// routing) passes on Linux CI
but times out on macOS/Docker Desktop where bridge IPs are not host-routable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mergify

mergify Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@codacy-production

codacy-production Bot commented Jul 4, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Coverage ∅ diff coverage · +24.13% coverage variation

Metric Results
Coverage variation ✅ +24.13% coverage variation
Diff coverage ✅ ∅ diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (cdb4573) 134228 57647 42.95%
Head commit (2b919d8) 166035 (+31807) 111375 (+53728) 67.08% (+24.13%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#4990) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements the e2e-js Bolt conformance suite covering all 39 scenarios in bolt/conformance/spec.yaml against the official neo4j-driver, along with its design document, implementation plan, and README. The review feedback highlights several improvement opportunities: wrapping keytool executions in a try-catch block to prevent test suite crashes, cleaning up the temporary certificate directory in afterAll, instantiating new GenericContainer instances from the built image tag to avoid mutating a shared instance, and pulling a record before calling result.summary() to fully conform to the RESULT-003 spec scenario.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +102 to +129
function generateTlsCerts() {
let keytool;
try {
keytool = execFileSync("which", ["keytool"]).toString().trim();
} catch (_) {
return null;
}
const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bolt-tls-"));
const keystore = path.join(dir, "keystore.p12");
const truststore = path.join(dir, "truststore.jks");
const cert = path.join(dir, "bolt.cer");
execFileSync(keytool, [
"-genkeypair", "-alias", "bolt", "-keyalg", "RSA", "-keysize", "2048",
"-validity", "3650", "-keystore", keystore, "-storetype", "PKCS12",
"-storepass", TLS_STORE_PASSWORD, "-keypass", TLS_STORE_PASSWORD,
"-dname", "CN=localhost, OU=ArcadeDB, O=ArcadeDB, L=Test, ST=Test, C=US",
]);
execFileSync(keytool, [
"-exportcert", "-alias", "bolt", "-keystore", keystore,
"-storetype", "PKCS12", "-storepass", TLS_STORE_PASSWORD, "-file", cert,
]);
execFileSync(keytool, [
"-importcert", "-alias", "bolt", "-keystore", truststore,
"-storetype", "JKS", "-storepass", TLS_STORE_PASSWORD, "-file", cert,
"-noprompt",
]);
return dir;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If keytool is present on the system but fails to execute (e.g., due to permission issues, environment-specific security policies, or JDK version differences), the unhandled exception will crash the entire test suite. Wrapping the keytool executions in a try-catch block ensures that any failure is caught gracefully, the temporary directory is cleaned up, and the function returns null to safely skip the TLS tests.

function generateTlsCerts() {
  let keytool;
  try {
    keytool = execFileSync("which", ["keytool"]).toString().trim();
  } catch (_) {
    return null;
  }
  const dir = fs.mkdtempSync(path.join(os.tmpdir(), "bolt-tls-"));
  const keystore = path.join(dir, "keystore.p12");
  const truststore = path.join(dir, "truststore.jks");
  const cert = path.join(dir, "bolt.cer");
  try {
    execFileSync(keytool, [
      "-genkeypair", "-alias", "bolt", "-keyalg", "RSA", "-keysize", "2048",
      "-validity", "3650", "-keystore", keystore, "-storetype", "PKCS12",
      "-storepass", TLS_STORE_PASSWORD, "-keypass", TLS_STORE_PASSWORD,
      "-dname", "CN=localhost, OU=ArcadeDB, O=ArcadeDB, L=Test, ST=Test, C=US",
    ]);
    execFileSync(keytool, [
      "-exportcert", "-alias", "bolt", "-keystore", keystore,
      "-storetype", "PKCS12", "-storepass", TLS_STORE_PASSWORD, "-file", cert,
    ]);
    execFileSync(keytool, [
      "-importcert", "-alias", "bolt", "-keystore", truststore,
      "-storetype", "JKS", "-storepass", TLS_STORE_PASSWORD, "-file", cert,
      "-noprompt",
    ]);
    return dir;
  } catch (_) {
    try {
      fs.rmSync(dir, { recursive: true, force: true });
    } catch (__) {}
    return null;
  }
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1ac9148. Cert generation is wrapped in try-catch: a present-but-failing keytool now removes the temp dir and returns null so the TLS scenarios skip instead of crashing the suite.

Comment on lines +266 to +269
afterAll(async () => {
if (tlsRequired) await tlsRequired.stop();
if (tlsOptional) await tlsOptional.stop();
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The temporary directory certDir created in beforeAll is never cleaned up, leaving leftover files (keystore.p12, truststore.jks, bolt.cer, Dockerfile) in the OS temp directory after every test run. We should clean it up in afterAll using fs.rmSync.

    afterAll(async () => {
      if (tlsRequired) await tlsRequired.stop();
      if (tlsOptional) await tlsOptional.stop();
      if (certDir) {
        try {
          fs.rmSync(certDir, { recursive: true, force: true });
        } catch (_) {
          // ignore cleanup errors
        }
      }
    });

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1ac9148. The TLS afterAll now fs.rmSync(certDir, {recursive:true, force:true}) inside a try-catch.

Comment thread e2e-js/src/js-bolt-conformance.test.js Outdated
Comment on lines +273 to +282
tlsRequired = await builtImage
.withExposedPorts(2480, 7687)
.withEnvironment({ JAVA_OPTS: tlsJavaOpts("REQUIRED") })
.withStartupTimeout(120000)
.withWaitStrategy(
Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching(
(sc) => sc === 204
)
)
.start();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Reusing and mutating the shared builtImage instance across multiple tests is risky because tlsRequired is still running when CONN-005 starts. Mutating the shared GenericContainer instance's environment or configuration can lead to unexpected side effects or test pollution. Instead, instantiate a new GenericContainer using the built image tag "arcadedb-bolt-tls-test:latest".

Suggested change
tlsRequired = await builtImage
.withExposedPorts(2480, 7687)
.withEnvironment({ JAVA_OPTS: tlsJavaOpts("REQUIRED") })
.withStartupTimeout(120000)
.withWaitStrategy(
Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching(
(sc) => sc === 204
)
)
.start();
tlsRequired = await new GenericContainer("arcadedb-bolt-tls-test:latest")
.withExposedPorts(2480, 7687)
.withEnvironment({ JAVA_OPTS: tlsJavaOpts("REQUIRED") })
.withStartupTimeout(120000)
.withWaitStrategy(
Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching(
(sc) => sc === 204
)
)
.start();

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1ac9148. buildTlsImage now returns the image tag and CONN-002 starts a fresh new GenericContainer(tag), so no shared builder instance is mutated.

Comment thread e2e-js/src/js-bolt-conformance.test.js Outdated
Comment on lines +303 to +312
tlsOptional = await builtImage
.withExposedPorts(2480, 7687)
.withEnvironment({ JAVA_OPTS: tlsJavaOpts("OPTIONAL") })
.withStartupTimeout(120000)
.withWaitStrategy(
Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching(
(sc) => sc === 204
)
)
.start();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Instantiate a new GenericContainer using the built image tag "arcadedb-bolt-tls-test:latest" to avoid mutating the shared builtImage instance.

Suggested change
tlsOptional = await builtImage
.withExposedPorts(2480, 7687)
.withEnvironment({ JAVA_OPTS: tlsJavaOpts("OPTIONAL") })
.withStartupTimeout(120000)
.withWaitStrategy(
Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching(
(sc) => sc === 204
)
)
.start();
tlsOptional = await new GenericContainer("arcadedb-bolt-tls-test:latest")
.withExposedPorts(2480, 7687)
.withEnvironment({ JAVA_OPTS: tlsJavaOpts("OPTIONAL") })
.withStartupTimeout(120000)
.withWaitStrategy(
Wait.forHttp("/api/v1/ready", 2480).forStatusCodeMatching(
(sc) => sc === 204
)
)
.start();

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1ac9148. CONN-005 likewise starts a fresh new GenericContainer(tag).

Comment thread e2e-js/src/js-bolt-conformance.test.js Outdated
Comment on lines +535 to +543
// In neo4j-driver v6, result.summary() is the consume/DISCARD path: it
// ends the stream and returns a ResultSummary WITHOUT materializing the
// records into a user-facing array (unlike `await result`). Driving the
// async iterator first and then calling summary() deadlocks, so this
// consumes via summary() directly.
const result = s.run("MATCH (b:Beer) RETURN b.name AS name LIMIT 5");
const summary = await result.summary();
expect(summary).toBeDefined();
expect(summary.query).toBeDefined();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The test currently calls result.summary() directly without pulling any records first, which does not fully conform to the spec scenario RESULT-003 ("When the driver PULLs 1 row of a 5-row result then calls result.consume() (DISCARD)"). We can pull exactly 1 row using for await...break (which properly closes the async iterator and avoids any deadlock) before calling result.summary().

Suggested change
// In neo4j-driver v6, result.summary() is the consume/DISCARD path: it
// ends the stream and returns a ResultSummary WITHOUT materializing the
// records into a user-facing array (unlike `await result`). Driving the
// async iterator first and then calling summary() deadlocks, so this
// consumes via summary() directly.
const result = s.run("MATCH (b:Beer) RETURN b.name AS name LIMIT 5");
const summary = await result.summary();
expect(summary).toBeDefined();
expect(summary.query).toBeDefined();
const result = s.run("MATCH (b:Beer) RETURN b.name AS name LIMIT 5");
for await (const _record of result) {
break; // pull 1 and discard remainder
}
const summary = await result.summary();
expect(summary).toBeDefined();
expect(summary.query).toBeDefined();

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 1ac9148. RESULT-003 now pulls one row via 'for await ... break' (which closes the iterator cleanly) before result.summary(). Verified locally it does not deadlock and passes.

Address Gemini review on PR #4990:
- wrap keytool cert generation in try-catch: keytool present-but-failing now
  cleans up the temp dir and skips TLS instead of crashing the suite
- clean up the temp cert dir in the TLS afterAll (was leaked)
- start each TLS container from the built image tag via a fresh
  GenericContainer instead of reusing/mutating one shared builder instance
- RESULT-003 pulls one row (for await...break, which closes the iterator
  cleanly) before summary(), matching the spec's 'PULL 1 then DISCARD' step

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Jul 4, 2026

Copy link
Copy Markdown

Code review: e2e-js Bolt conformance suite (#4888)

Reviewed the full diff (test-only + docs). Overall this is a high-quality, carefully-documented conformance suite that faithfully mirrors the merged Python/Go/C# siblings. The [AREA-NNN] naming, the it.failing() gap-tracking pattern (green while the gap reproduces, XPASS when fixed), and the HTTP-only seeding strategy are all exactly right. No new dependency is introduced (neo4j-driver/testcontainers were already used by js-bolt-e2e.test.js), and Jest auto-discovers the file so no CI change is needed. Nice work.

A few observations, all minor:

1. TLS derived image is never cleaned up (parity gap). buildTlsImage() builds and tags arcadedb-bolt-tls-test:latest, but afterAll only stops the two containers (tlsRequired/tlsOptional). The Python sibling explicitly removes the built image in teardown (docker.from_env().images.remove(image_tag, force=True)). Here the derived image lingers in the local daemon after every run. Not a CI problem (ephemeral runners), but it accumulates on developer machines and is a small parity gap worth closing.

2. TX-005 / ERR-004 assert less than the Python sibling. The JS race tests assert only that a Neo.TransientError.* is present. Python additionally asserts the absence of a non-retryable error: assert not any(isinstance(e, (ClientError, DatabaseError)) for e in errors). That negative assertion is the part that actually proves the conflict is retryable (the scenario intent). Consider adding the equivalent so a regression that surfaces the conflict as a non-retryable ClientError/DatabaseError would be caught.

3. keytool-absent path reports CONN-002/005 as PASSED, not SKIPPED. if (!certDir) return console.warn(...) inside it() makes the test report green even though nothing was exercised, which is slightly misleading in a report that otherwise distinguishes pass vs. skip (CONN-004/ERR-003 use real it.skip). CI always has a JDK so this only affects local runs without keytool, but a runtime-conditional skip would be more honest. Minor.

4. Race-based tests are inherently timing-sensitive. TX-005/ERR-004 depend on a real write-write conflict materializing within the 500ms window. The code and comments already acknowledge this and keep the unconditional diagnostic console.log, a reasonable deliberate choice here (mirrors Python print) despite the general remove-debug-output guideline. Flagging the flakiness surface; if these prove flaky in CI, a retry/loop around the race would be the fix.

5. CONN-003 not skipped on macOS. Documented clearly in the README (Docker-bridge address not host-routable on Docker Desktop), and it passes on the Linux CI runners, so this is an accepted local-only failure rather than a defect. No action needed - confirming I read it as intentional.

Things done well

  • HTTP-only fixture seeding keeps the serialization-under-test path out of setup - correct and important.
  • it.failing() assertions each encode the Neo4j-correct behavior and link Bolt: tracking issue for protocol/type-fidelity gaps surfaced by certification #4890, so fixing the server forces a spec.yaml update in the same PR. Good forcing function.
  • Thorough try/finally session/driver cleanup throughout; no obvious leaked sessions.
  • The RESULT-003 comment about result.summary() being the v6 DISCARD path (and the iterator-then-summary deadlock) is exactly the kind of note that saves the next maintainer an hour.

None of the above is blocking. Items 1 and 2 are the ones I would most encourage tightening for true parity with the Python suite.

Address Claude review on PR #4990:
- TX-005/ERR-004 now also assert no racing error is a non-retryable
  ClientError/DatabaseError (noNonRetryableErrors) - the part that actually
  proves the conflict is retryable, matching the Python sibling
- CONN-002/CONN-005 use a load-time keytool gate (it.skip when absent) so the
  keytool-less path reports as SKIPPED, not a misleading PASS
- remove the throwaway derived TLS image in afterAll (docker rmi), parity with
  the Python suite's teardown

Verified locally: TX-005/ERR-004 and CONN-002/005 pass; image removed on teardown.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@robfrank

robfrank commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed in 28c21e0 (verified locally against the branch image):

1. TLS derived image cleanup. afterAll now runs docker rmi -f on the derived tag (guarded, ignores in-use/already-gone), closing the parity gap with the Python teardown. Confirmed the image is gone after the run.

2. TX-005/ERR-004 negative assertion. Added noNonRetryableErrors(errors) to both: the racing errors must contain no Neo.ClientError.*/Neo.DatabaseError.* code, not just a TransientError somewhere. Verified the race satisfies it (only a TransientError surfaces, no non-retryable code).

3. keytool-absent honest skip. Replaced the in-body return console.warn(...) with a load-time KEYTOOL_AVAILABLE gate and const tlsIt = KEYTOOL_AVAILABLE ? it : it.skip, so CONN-002/CONN-005 report as SKIPPED (not PASSED) on hosts without a JDK. The remaining in-body guard only covers the rare keytool-present-but-generation-failed edge.

4. Race timing-sensitivity. Kept as-is with the unconditional diagnostic console.log (deliberate, mirrors the Python sibling). If CI shows flakiness, a bounded retry loop around the race is the follow-up rather than a structural change now.

5. CONN-003 on macOS. Correct - it is an intentional local-only failure (Docker-bridge address not host-routable on Docker Desktop) that passes on the Linux CI runners, same as the merged siblings. No change.

@claude

claude Bot commented Jul 4, 2026

Copy link
Copy Markdown

PR Review: test(#4888): deepen e2e-js to the full Bolt conformance spec

Thorough, well-structured test-only PR. It faithfully mirrors the merged Python/Go/C# suites (same raceTwoWriters helper, boltscratch DB, fetchSize/fetch_size, neo4j 6.x pin, HTTP-only seeding), keeps the existing VLP regression untouched, and makes excellent use of it.failing() so a server-side fix forces an XPASS and a matching spec.yaml update. The [AREA-NNN] naming, self-skipping TLS via keytool, and cleanup in afterAll are all solid. Below are findings, most minor.

1. Parity gap: TX-005 / ERR-004 are weaker than the Python sibling (medium)

The Python test_TX_005 and test_ERR_004 assert both directions:
```python
assert any(isinstance(e, TransientError) for e in errors)
assert not any(isinstance(e, (ClientError, DatabaseError)) for e in errors) # <-- missing in JS
```
The JS versions (lines 445-449, 807-810) only assert the positive errors.some(... startsWith("Neo.TransientError")). They drop the negative check. The scenario's documented intent is "the conflict must be retryable, not merely present" - a run where one writer got a TransientError and the other a non-retryable ClientError/DatabaseError would pass in JS but fail in Python. Recommend adding the symmetric assertion so the JS suite doesn't silently miss a regression the sibling suites catch, e.g.:
```js
expect(errors.some(e => /^Neo.(ClientError|DatabaseError)/.test(String(e.code)))).toBe(false);
```

2. Dangling derived TLS image is never removed (low)

buildTlsImage() builds arcadedb-bolt-tls-test:latest via GenericContainer.fromDockerfile(...).build(TLS_IMAGE_TAG). afterAll stops the containers and removes certDir, but the built image itself is a tagged image that testcontainers does not auto-remove, so it lingers on the CI runner / dev host across runs. Consider removing it in afterAll to keep runs hermetic.

3. Race-based tests are inherently timing-sensitive (low, acknowledged)

TX-005/ERR-004 depend on the 500ms-held transaction producing a genuine write conflict surfaced as Neo.TransientError.*. The comments already call this out and it matches the sibling suites, so this is acceptable - just flagging that if the server's conflict-error mapping ever changes, these are the first tests to go flaky. No retry/tolerance is built in, which is the right call for a conformance assertion.

4. PROTO-002 comment vs. literal assertion reads as contradictory (nit)

The inline comment (lines 828-833) says "this asserts ... the negotiated protocol is < 5", but the literal code is expect(Number(info.protocolVersion)).toBeGreaterThanOrEqual(5) wrapped in it.failing. The behavior is correct (the it.failing inverts it), but the phrasing contradicts the code at a glance. Consider rewording to "asserts the Neo4j-correct >= 5 under it.failing, so it flips red when the server starts advertising 5.x."

5. CONN-003 runs unconditionally and hangs ~180s on macOS (info)

CONN-003 is a real it(), and per the README it times out against the full jest.setTimeout(180000) on Docker Desktop hosts. Documented and matches the siblings, so optional - but gating it behind an env flag (rather than a 3-minute local hang) would improve local DX. Your call given the cross-suite consistency.

Minor

  • package.json has empty author/description/keywords and "main": "index.js" with no index.js present - cosmetic and pre-existing to the module, not introduced here.
  • The unconditional console.log diagnostic in raceTwoWriters is intentional (documented) and appropriate for a race diagnostic - fine to keep.

Verdict

Coverage is comprehensive (39 scenarios, gaps honestly encoded as it.failing/skip), the pattern is consistent with the merged suites, and no product code changes. Addressing finding #1 (the missing negative assertion) would bring it to full parity with the Python suite; the rest are optional polish. Nice work.

Reviewed by Claude Code.

Reword the PROTO-002 comment so it no longer reads as contradicting the
literal assertion: the body asserts the Neo4j-correct '>= 5' and it.failing
inverts it, staying green while 4.x is negotiated and flipping red on 5.x.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@robfrank

robfrank commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the second pass. Status against this review:

  • build support #1 (TX-005/ERR-004 negative assertion) and Bump gremlin.version from 3.4.10 to 3.5.1 #2 (dangling TLS image) were already addressed in 28c21e0 before this review posted: both race tests now assert noNonRetryableErrors(errors) (no Neo.ClientError.*/Neo.DatabaseError.* among the racing errors), and afterAll runs docker rmi -f on the derived tag. Verified locally.
  • Bump mongo-java-driver from 3.12.4 to 3.12.10 #4 (PROTO-002 comment vs assertion) - reworded in the latest commit to state that the body asserts the Neo4j-correct >= 5 and it.failing inverts it (green while 4.x is negotiated, XPASS on 5.x), so it no longer reads as contradictory.
  • Bump slf4j-simple.version from 1.7.31 to 1.7.32 #3 (race timing), Bump jedis from 3.6.1 to 3.6.3 #5 (CONN-003 local hang), and the minor package.json/console.log notes - keeping as-is for cross-suite parity. CONN-003 fails on the ~30s driver connection timeout locally (not the full 180s) and passes on the Linux runners, matching the merged Python/Go/C# suites; I'd rather not gate it behind a JS-only env flag and diverge from them.

@claude

claude Bot commented Jul 4, 2026

Copy link
Copy Markdown

PR Review: test(#4888): deepen e2e-js to the full Bolt conformance spec

Reviewed the full diff. This is a clean, test-only PR that brings e2e-js to parity with the merged Python/Go/C# conformance suites. The implementation is careful and the intent is well-documented in-code. Overall: looks good to merge, with a few minor observations below (none blocking).

Strengths

  • Faithful 1:1 mirror of the Python suite (raceTwoWriters, HTTP-only seeding, it.failing for the 6 tracked server gaps, honest skips for CONN-004/ERR-003). The parity makes the whole family easy to reason about.
  • Correct use of it.failing for known gaps: each asserts the Neo4j-correct behavior so it stays green while the gap reproduces and flips to XPASS (red) the moment the server is fixed - a genuinely good regression tripwire, and the in-code comments say exactly what to do when that happens (convert to it() + flip spec.yaml).
  • Solid resource hygiene: every session/driver/container is closed in finally, the derived TLS image and temp cert dir are torn down in afterAll, and cleanup failures are swallowed so they can't mask the real result.
  • Honest skips vs. misleading passes: KEYTOOL_AVAILABLE is resolved at load time so TLS scenarios report as real skips on hosts without a JDK, rather than silently "passing".
  • --runInBand + per-file jest.setTimeout(180000) correctly overrides the 10s default testTimeout for the container-bound work.

Observations (minor / non-blocking)

  1. Race-based tests are inherently timing-sensitive (TX-005, ERR-004). Both assert errors.length > 0 and require at least one Neo.TransientError.*. If the engine ever lets both 500ms-overlapping writers commit (or serializes them without conflict), the suite goes red on a non-deterministic condition. This exactly matches the Python suite's assertions, so it's accepted parity rather than a new risk - but it's the most likely source of future flakiness, worth watching if these start intermittently failing in CI.

  2. Container startup depends on a live GitHub fetch. defaultDatabases=beer[...]{import:https://github.com/.../OpenBeer.gz} downloads the dataset at boot; a GitHub outage or slow network fails beforeAll (bounded by the 120s startup timeout). Inherited from the existing pattern, not introduced here, but it's a real external dependency for the whole file.

  3. Diagnostic console.log left in raceTwoWriters. Intentional and documented ("printed unconditionally so it shows even on an unexpected pass"), mirroring the Python print. Fine as-is; flagging only because the repo convention is to strip debug output - here it's deliberate diagnostics, not leftover debugging.

  4. Static TLS image tag arcadedb-bolt-tls-test:latest. A fixed tag on a shared Docker host could collide if two builds of this suite ever run concurrently on the same daemon. Not a concern under --runInBand on isolated CI runners; only matters if the CI topology ever shares a Docker host.

  5. Unix-only host assumptions. execFileSync("which", ["keytool"]) and the docker rmi teardown assume a Unix host with those binaries on PATH. Correct for the Linux CI runners (and the README documents the macOS CONN-003 caveat), so fine for the intended environment - just not portable to a Windows runner.

  6. Doc bloat. ~1550 lines of the diff are design/plan markdown under docs/superpowers/. No issue, just noting the code under review is really the single 893-line test file.

Test coverage

Coverage is the point of the PR and it's thorough: all 39 spec scenarios accounted for, with the 6 known gaps and 2 infra-blocked scenarios explicitly and traceably marked. No production code changes, so no regression surface beyond CI time.

Nice work - the cross-language consistency here is the standout.

Reviewed by Claude Code.

@claude

claude Bot commented Jul 4, 2026

Copy link
Copy Markdown

Review: test(#4888): deepen e2e-js to the full Bolt conformance spec

Reviewed the full diff. This is a well-executed, test-only PR that faithfully mirrors the merged Python/Go/C# conformance siblings. Coverage is complete (verified 39 unique [AREA-NNN] ids against spec.yaml's 39 scenarios), the it.failing/it.skip handling of known gaps is clean and self-documenting, and the review-feedback commits already hardened the TLS fixture and race assertions. The existing VLP regression is untouched as claimed. Nice work.

A few points worth considering before merge:

1. Timing-based race tests are the main CI-stability risk (raceTwoWriters, TX-005 + ERR-004)

raceTwoWriters relies on a fixed setTimeout(500) overlap to force a write-write conflict, then asserts expect(errors.length).toBeGreaterThan(0). If the server ever serializes the two explicit transactions (second RUN blocks until the first commits) instead of raising a conflict, errors is empty and the test hard-fails rather than skips. This runs twice (TX-005 and ERR-004 are near-identical), doubling the flake surface, and --runInBand means no isolation between them. It passes in the Python sibling so the behavior is presumably stable, but this is the highest-probability source of an intermittent red on CI. Worth watching; if it flakes, consider a bounded retry on the "no conflict observed" path before asserting.

2. TLS coverage silently depends on a JDK the js-e2e-tests job never installs

KEYTOOL_AVAILABLE gates CONN-002/CONN-005, but the js-e2e-tests job (mvn-test.yml:346) only runs setup-node - no setup-java. So keytool presence relies entirely on ubuntu-latest's preinstalled Temurin. That works today, but the moment GitHub changes the base image, both TLS scenarios degrade to a silent it.skip with no signal that a whole conformance area stopped running. Adding a setup-java step to that job (as the Maven jobs already do) would make TLS coverage deterministic rather than incidental.

3. Residual misleading-pass path in the TLS tests

The load-time tlsIt gate correctly turns keytool-absent into a real SKIP. But if keytool is present at load time yet generateTlsCerts() fails at beforeAll time (returns null), the test body hits if (!certDir) return console.warn(...) and reports PASS, not skip - the exact misleading-green the earlier "honest TLS skip" commit set out to eliminate, just on the runtime-failure branch instead of the absent branch. Since buildTlsImage is skipped too, nothing is actually exercised. Consider failing (or at least a Jest-visible skip) rather than an early return there.

4. neo4j-driver ^6.0.1 vs. the "pinned" claim + documented fragility

The file header warns the whole suite depends on the 6.x driver silently downgrading to Bolt 4.4, "a future driver major dropping legacy negotiation would break the whole suite." Given that fragility, ^6.0.1 (caret = any 6.x) is looser than the README's "single pinned" wording implies. package-lock.json protects CI reproducibility, but a local npm install (or a lockfile refresh) can drift the minor. An exact pin, or at least a ~6.0 tilde, would match the stated intent more closely.

Minor

  • raceTwoWriters prints a console.log per collected error unconditionally (even on pass). The rationale comment is fair, but it adds noise to every green run; gating it behind a debug env var would keep the signal.
  • TX-005 and ERR-004 are functionally identical assertions over the same helper - intentional per the two distinct spec ids and consistent with the siblings, just noting the duplication is deliberate.

None of these are blockers; (1) and (2) are the two I'd most want addressed since they affect CI signal reliability.

🤖 Generated with Claude Code

@robfrank

robfrank commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator Author

Disposition of this pass (all 6 are non-blocking; keeping current behavior, with reasoning):

  1. Race timing-sensitivity (TX-005/ERR-004) - kept. Identical to the Python sibling's assertions; a bounded retry would weaken the conformance guarantee. Will revisit only if CI shows intermittent failures.
  2. Beer import fetches from GitHub at boot - inherited from the existing js-bolt-e2e.test.js / Python fixture pattern, bounded by the 120s startup timeout. Not introducing a new dependency here, so leaving it consistent with the siblings.
  3. raceTwoWriters console.log - intentional race diagnostic (documented), matching the Python print. Kept.
  4. Static TLS image tag - kept fixed for parity with the Python fixture (same arcadedb-bolt-tls-test:latest). No collision under --runInBand on the ephemeral GitHub-hosted runners; a per-run tag would only matter on a shared self-hosted daemon, which isn't the CI topology.
  5. Unix-only which/docker rmi - correct for the Linux CI runners; the README already documents the macOS CONN-003 caveat. Windows runners aren't in scope for this suite.
  6. Doc size - the docs/superpowers/ design+plan are the standard artifacts the sibling PRs also carried; the reviewable code is the single test file.

No code change for this pass - items 1-6 are all accepted-as-parity or inherited. The two earlier substantive findings (negative race assertion, TLS image cleanup) were fixed in 28c21e0, and the PROTO-002 comment nit in 2b919d8.

@codecov

codecov Bot commented Jul 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.31%. Comparing base (c0dbdc8) to head (2b919d8).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #4990      +/-   ##
============================================
- Coverage     65.32%   65.31%   -0.02%     
+ Complexity      817      815       -2     
============================================
  Files          1682     1682              
  Lines        134228   134232       +4     
  Branches      28712    28713       +1     
============================================
- Hits          87685    87674      -11     
- Misses        34526    34536      +10     
- Partials      12017    12022       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@robfrank
robfrank merged commit 44e4604 into main Jul 4, 2026
26 of 31 checks passed
@robfrank
robfrank deleted the feat/4888-e2e-js-bolt-conformance branch July 4, 2026 22:07
robfrank added a commit that referenced this pull request Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bolt/e2e-js: deepen js-bolt-e2e to the full conformance spec

1 participant