fix(harness): close the remaining known-open harness failures - #34
Merged
Merged
Conversation
Incident 2026-09-17: a test that recorded telemetry without isolating HEX_DIR wrote a stray row into the live instance store, because hex sessions export HEX_DIR. Two tests fail until .cargo/config.toml forces the sandbox; a third proves the built binary still honors its own HEX_DIR outside cargo. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
.cargo/config.toml forces HEX_DIR to /tmp/hex-test-hex-dir for unit tests, integration binaries, nextest, and cargo run, even when the shell exports a live HEX_DIR. A test that forgets to isolate HEX_DIR now writes into the sandbox, never the live telemetry store. The path is absolute so the container lane (root, bind mount) leaves no root-owned files in the tree. The built binary run outside cargo still honors its own HEX_DIR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
…_live tests Fails on the current tree: lane_argv passes only --run-ignored all, and no .config/nextest.toml exists. Incident: hex-nightly-tests::nightly exited 100 three nights in a row at test 80 of 1547 because one host-bound #[ignore] test failed under fail-fast and hid the rest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
…ritten entry Fixture 3: pre_tool_use, matcher Write|Edit|MultiEdit|NotebookEdit, command hex hook worktree-guard, trusted_hash written by Codex CLI 0.154.0 on 2026-09-17. Runs in every lane; the live comparison test stays host-only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
…_live tests The lane argv adds --profile nightly --no-fail-fast, and .config/nextest.toml gives the nightly profile default-filter 'not test(/_live$/)'. Host-bound tests (live iii engine, live Codex config) carry the _live suffix and a reason string; everything else, model tests included, runs in the container and the full tally is reported every night. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
…orts bails by progress Eleven model-free tests for FIX-016 and the bail contract. Incident: hex-memory-maintenance::index has 243 error rows since 2026-07-09; the 2026-09-17 16:58 UTC run spent 827 s on one 176-chunk file under the background throttle because the budget is checked only between files, and every resumable bail lands as an error. Fails until the within-file predicate is enforced and the bail decision is implemented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
… when progress was made embed_and_store checks an injected over_budget predicate before each batch after the first; remaining chunks stay FTS5-only and backfill re-embeds them on later ticks (FIX-016 closed). A bail that completed a file or stored a vector exits 0 and records a memory/index::budget-bail warn row with the counts; a bail with no progress, or the fourth consecutive bail, still exits 1. A clean run resets the counter in memory.db metadata. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
… cap is 4096 Incident 2026-09-16: three distill::judge-error rows, one with finish_reason=length and no content, two with JSON cut mid-string, because memory_judge runs at max_tokens 256 on a model whose hidden reasoning tokens count against the cap. parse_chat_response fixture tests fail until finish_reason is read; the builtin test fails until the cap is raised. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
…ken cap parse_chat_response returns ProviderError::Truncated when the provider reports finish_reason=length, so a cut-off judge decision lands as a named truncation in the distill::judge-error row instead of a JSON parse error. memory_judge max_tokens goes from 256 to 4096: one small JSON object, with headroom for the model's hidden reasoning tokens. The extract path treats a truncated response like an upstream failure (same strike and poison-slice path); hex doctor reports it with its own wording. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
Residual from the v0.53.3 review: against an engine that accepts TCP and never finishes the websocket handshake, every ops::call_builtin inside hex harness serve still leaked one detached connection thread plus socket after the bounded shutdown. New integration binary ops_shared_client fails until call_builtin uses the process-wide client that serve installs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
worker::runtime::serve connects through connect_engine_client, which installs its long-lived client into a process-wide OnceLock in ops. When that client is present, ops::call_builtin runs trigger on a current-thread runtime against it: no per-call register_worker, thread, socket, or shutdown. A stalled engine can no longer accumulate detached connection threads inside serve. CLI subprocesses keep the per-call client with its bounded shutdown; the SDK handshake timeout stays a fork-level follow-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
…onal build Review finding on this branch: .cargo/config.toml forces HEX_DIR for every cargo-launched process, build scripts included, so hex upgrade's cargo build --features personal would resolve the overlay roots under the sandbox and silently ship a binary with every personal worker missing. Fails until harness_build_env also sets HEX_OVERLAY_DIR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
…grade build.rs resolves the overlay root from HEX_OVERLAY_DIR first, then HEX_DIR, then HOME/hex, and hex upgrade passes HEX_OVERLAY_DIR beside HEX_DIR, so the forced cargo [env] sandbox cannot redirect the personal-module discovery. A --features personal build that resolves no overlay dir now fails loudly instead of shipping a binary without the instance's workers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
Shared test helpers move to tests/support/mod.rs (fd_limits and ops_shared_client stay separate binaries); the Codex handler normalization is one helper used by the fixture and the live test; run_index_body handles index results once; parse_chat_response is private; stale red-phase comments and a duplicated decision comment are gone; contributes_progress names the shared rule; model-dependent #[ignore] tests carry reason strings; docs point at the renamed live test and the real nextest config. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
- telemetry_store: with_hex_dir restores HEX_DIR through the EnvVar guard, so the sandbox tests pass under plain cargo test in any order - llm_provider doctor check: classify() is pure and tested for the Truncated, Upstream, Deferred, and Ok arms - embed.rs: the model-dependent #[ignore] test carries a reason string Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Third slice of the 2026-09-17 harness incident work. v0.53.2 closed the per-call iii client fd leak and v0.53.3 (#33) bounded its teardown. This PR clears the five failures that were still open on the instance afterwards: the nightly container lane now reports a full tally every night,
hex memory indexno longer raises an alert for a budget bail it recovers from on the next tick, the distill judge stops cutting off its own JSON, tests can no longer write telemetry into a live store, andhex harness servereuses one iii client instead of opening one per call. The SDK-side handshake timeout in thehex-iiifork stays as follow-on work.What changed for operators
#[ignore]test failed under fail-fast. The lane now runs--profile nightly --no-fail-fast;.config/nextest.tomlexcludes tests named*_live(live iii engine, live~/.codex/config.toml). Verified in the container: 1569 passed, 2_liveskipped, model tests included.memory/index::budget-bailwarn row. Zero progress or a fourth consecutive bail still exits 1.memory_judgeran at 256 output tokens on a model whose hidden reasoning counts against the cap; threejudge-errorrows on 09-16 were truncations misreported as parse failures. Cap is 4096 andfinish_reason=lengthis a namedProviderError::Truncated..cargo/config.tomlforcesHEX_DIR=/tmp/hex-test-hex-dirfor every cargo-launched process (hex sessions export a liveHEX_DIR, and a test that forgot to isolate wrote a stray row today). The built binary run outside cargo still honors its ownHEX_DIR.serveinstalls its long-lived client into a process-wide slot andops::call_builtinreuses it, so a stalled engine can no longer accumulate detached connection threads insideserve. CLI subprocesses keep the per-call client with its bounded shutdown.Design decisions to check
_livenaming plus a nextest profiledefault-filteris the exclusion mechanism. A skip inside the test body was rejected as a silent pass; a source-scanning guard test was rejected because the testing standard forbids tests that readsrc/.memory.dbmetadata.HEX_DIRsandbox reaches build scripts. Review caught thathex upgrade'scargo build --features personalwould have resolved the personal overlay under the sandbox and silently shipped a binary with no instance workers.build.rsnow readsHEX_OVERLAY_DIRfirst (upgrade passes it besideHEX_DIR) and fails loudly when--features personalresolves no overlay dir. Empirically checked: 6 overlay entries generated with the var, a named panic without it.block_onon a single current-thread runtime from parallel handler threads changes scheduling; the per-call runtime is dropped on return.Tests
Each bug fix lands as a failing-test commit followed by its fix commit (
git logorder). New coverage:tests/ops_shared_client.rs(refused port and stalled-handshake peer, fd count returns to baseline, no per-call shutdown row),tests/telemetry_store.rssandbox proofs, eleven model-free index budget tests through injected seams, providerfinish_reasonfixtures, a Codex hook-hash parity fixture captured from a Codex-writtentrusted_hash. Workspace: 1582 tests, clippy and fmt clean.Residual (documented in
docs/residual-review-findings/): the SDK's connect has no handshake timeout; CLI subprocesses rely on process exit. A secondinstall_shared_clientcall drops the rejected client without shutdown; no production path installs twice.Plan:
docs/plans/2026-09-17-1412-fix-known-open-harness-failures-plan.md🤖 Generated with Claude Code
https://claude.ai/code/session_0189QfSdS5G3cvZiY2Uf51cb