diff --git a/.claude/rules/env-macro-gating.md b/.claude/rules/env-macro-gating.md new file mode 100644 index 0000000000..8e1f9e1861 --- /dev/null +++ b/.claude/rules/env-macro-gating.md @@ -0,0 +1,40 @@ +# Environment-Variable and Macro Gating + +Environment variables and preprocessor macros that gate *behavior* (feature +flags, opt-in/opt-out switches, conditional code paths) are configuration +surface. They are easy to add and hard to remove: each one becomes a +permutation that must be documented, wired through CI, and reasoned about +forever. Keep them to a minimum. + +## 1. Adding a behavior gate requires explicit user permission + +**Before adding any new environment variable or macro that isolates or gates +behavior, you MUST explicitly ask the user for permission first.** State what +the flag would be, why you think it's needed, and what the default is — then +wait for an explicit yes. + +This covers: + +- New `std::getenv(...)` / `os.environ[...]` reads that change what the code + does. +- New `#ifdef` / `#if defined(...)` blocks (or `-D` compile flags) that select + between behaviors. +- New pytest CLI flags / env vars that toggle a code path. + +It does **not** cover reading an *already-agreed* flag, or env/macros that are +purely diagnostic (a log-verbosity knob that changes no behavior) — though when +in doubt, ask. + +Prefer alternatives that need no gate: do the thing unconditionally when it is +always correct; derive the decision from existing state; or rely on an existing +project invariant (e.g. onboard work always holds an exclusive `task-submit` +lock, so a device force-reset on the error path needs no opt-in flag). + +## 2. Every env/macro touch is a chance to delete one + +Whenever you modify code that reads an environment variable or branches on a +macro, check whether it can be **removed** — the gate may be dead, redundant, +or replaceable by unconditional behavior now that the surrounding code has +changed. If so, delete it (and its CI wiring, docs, and any include added only +for it). Leaving a stale gate is worse than never adding it: it implies a +choice that no longer exists. diff --git a/CLAUDE.md b/CLAUDE.md index e655ec66ae..9ade23f03c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -32,7 +32,7 @@ clang-format -i ## Important Rules -1. **Consult `.claude/rules/` for coding conventions** (architecture, codestyle, terminology, [discipline](.claude/rules/discipline.md), [doc-consistency](.claude/rules/doc-consistency.md)) — these are always-loaded guidelines. **Consult `.claude/skills/` for task-specific workflows** (e.g., `git-commit/` when committing, `testing/` when running tests) +1. **Consult `.claude/rules/` for coding conventions** (architecture, codestyle, terminology, [discipline](.claude/rules/discipline.md), [doc-consistency](.claude/rules/doc-consistency.md), [env-macro-gating](.claude/rules/env-macro-gating.md)) — these are always-loaded guidelines. **Consult `.claude/skills/` for task-specific workflows** (e.g., `git-commit/` when committing, `testing/` when running tests) 2. **Do not modify directories outside your assigned area** unless the user explicitly requests it 3. Create new subdirectories under your assigned directory as needed 4. When in doubt, ask the user before making changes to other areas diff --git a/docs/investigations/2026-06-a5-aicore-op-timeout-cascade.md b/docs/investigations/2026-06-a5-aicore-op-timeout-cascade.md index cf072e449e..60ec69116b 100644 --- a/docs/investigations/2026-06-a5-aicore-op-timeout-cascade.md +++ b/docs/investigations/2026-06-a5-aicore-op-timeout-cascade.md @@ -1,9 +1,11 @@ # a5 AICore op-timeout poisons the shared L2 worker for the rest of an xdist session -**Date**: 2026-06-08 -**Verdict**: contained (two independent guards landed — driver fail-fast + -fixture rebuild-then-skip). The poison itself is unrecoverable in-process on -a5; containment stops it from cascading. +**Date**: 2026-06-08 (force-reset recovery added 2026-06-09) +**Verdict**: recovered. #1005 *contained* the cascade (driver fail-fast + +fixture rebuild-then-skip). A follow-up found the poison **is** clearable +in-process with a *force* device reset (`aclrtResetDeviceForce`), so the +worker now recovers and the would-be-skipped tests actually run — see +"Force-reset recovery" below. ## Question @@ -70,7 +72,8 @@ a process exit + device reset (cascade variant 0/10). The amplifier is the so the poison spreads to every later test in the worker's process — exactly the CI cascade. -**Recovery is not possible in-process.** Two reset paths were measured: +**Soft-reset recovery is not possible in-process** (a *force* reset is — see +"Force-reset recovery" below). Two soft-reset paths were measured: - `aclrtSynchronizeDeviceWithTimeout` (the bounded device drain in fix #1) returns the *same* 507046 — it does not clear the sticky error. @@ -80,13 +83,25 @@ the CI cascade. (`simpler_init failed with code 507899`). The op-timeout poison survives a device reset within the process. -The only reset that recovers is **process exit** — which is exactly why the -separate-process cascade variant is 0/10. `aclrtResetDeviceForce` *would* -reset the physical device, but this box is shared (see -`.claude/rules/running-onboard.md`) and force-resetting kills other users' -work on the same die, so it is not an option. Net: after an AICore -op-timeout an a5 worker process cannot get a usable device back; only a new -process can. +A *soft* reset does not recover, but a **force** reset does. Follow-up +measurement (2026-06-09): after poisoning a card, `aclrtResetDeviceForce` +(rc=0) lets a fresh `Worker.init()` on the **same card in the same process** +succeed — a trivial noop then runs clean. The earlier belief that "only a new +process recovers" was specific to the *soft* `rtDeviceReset`; the *force* +reset clears the op-timeout sticky-error in place. Two safety facts were also +verified: + +- **Per-card scope.** With process A force-resetting device 2 while process B + ran a noop loop on device 3, B's 14/14 iterations passed straight through + A's reset — `aclrtResetDeviceForce(2)` does not disturb device 3. So under + xdist (gw0→dev4, gw1→dev5) one worker force-resetting its card cannot break + another's. +- **Exclusive ownership.** Force reset is destructive to anything else on the + *physical* card, but onboard work always holds an exclusive `task-submit` + lock (`.claude/rules/running-onboard.md`), so the only thing on the card is + this job. + +This is what the **force-reset recovery** fix below builds on. ## Fix @@ -136,40 +151,77 @@ failed with code 507899); the device context is not recoverable in-process after an earlier AICore error — skipping remaining 'tensormap_and_ringbuffer' L2 tests (a fresh worker process recovers)."* -## Why this shape +## Force-reset recovery (2026-06-09) + +The original containment left two gaps: the triggering test stays red, and the +"skipped" tests are never validated (real CI run 27182822054 went from 8 red to +`1 failed, 13 skipped` — contained, but the job is still red and 13 cases are +unverified). The follow-up measurement above showed `aclrtResetDeviceForce` +clears the poison in-process and is per-card safe under an exclusive lock, so +recovery is possible after all: + +- `DeviceRunner::finalize()` calls `force_reset_device()` + (`aclInit` + `aclrtSetDevice` + `aclrtResetDeviceForce`) whenever + `device_unusable_` is set. The card is reset before the runner is torn down. +- The conftest heal then closes + drops the poisoned `Worker` (unchanged from + #1005); the **next** test's `Worker.init()` lands on the now-clean card and + **succeeds**, so the previously-skipped tests actually run. No CLI/env flag — + it fires only on the rare device-poison path and onboard always holds an + exclusive `task-submit` lock (see `.claude/rules/env-macro-gating.md` for why + this is unconditional rather than gated). + +Verified on a5: a pooled `hang → noop` pair, hang FAILED (trigger, not retried), +noop **PASSED** 3/3 (force-reset recovered the card and the victim ran). The +`fixture rebuild-then-skip` path remains as the fallback when a force reset ever +fails (and on a2a3, which has no force-reset). + +This is the recovery layer; the triggering test itself is still red (retrying it +on a fresh card is a separate, later step). -The cascade cannot be *recovered* on a5 — the device is dead in-process and -force-reset is unsafe on shared hardware — so the goal is **containment**, -not recovery: +## Why this shape - fix #1 (driver) turns the confusing `halResMap rc=62` / `rtMalloc 507899` - cascade into an immediate, self-explanatory "DeviceRunner marked unusable; - rebuild the Worker" — for *any* caller, not just pytest. -- fix #2 (fixture) turns the rest of an xdist worker's L2 tests from a storm - of red 507899 errors into **one real failure + clean skips** with a - reason. A fresh worker process (the next CI shard, or the standalone phase) - starts clean. - -Both are cheap, independent, and additive. + cascade into an immediate, self-explanatory fail-fast — for *any* caller, + not just pytest. +- force-reset recovery clears the card in-process so the worker re-inits and + the rest of the L2 tests **run** instead of cascading or being skipped. +- fixture rebuild-then-skip is the fallback when a force reset itself fails. + +The same three guards are mirrored into `src/a2a3/` (the a2a3 device_runner +had none of them; #1005 was a5-only). a2a3 has the identical +`device_unusable_` / `recover_device_or_mark_unusable` / `force_reset_device` +chain wired into its `run()` / `finalize()`. The mechanism is CANN-level +(`aclrtResetDeviceForce`), so it is expected to behave as on a5, but a2a3 +**was not hardware-verified locally** — this dev box is Ascend950PR (a5) only; +the `st-onboard-a2a3` CI job is the a2a3 verification channel. ## When to reconsider -- If a future CANN release makes `aclrtSynchronizeDeviceWithTimeout` (or a - scoped reset API) actually clear an op-timeout sticky error in-place, - fix #1 could be upgraded from fail-fast to true in-place recovery, and - the worker would not need rebuilding at all. -- a2a3 shows the same `finalize_common` cascade note (507018/507899/507901) - but the trigger there has historically been blamed on runner contention; - the a2a3 side was intentionally **not** touched (could not reproduce the - cascade deterministically on a2a3 hardware here). If an a2a3 repro lands, - port the same two guards to `src/a2a3/`. +- The triggering test is still reported failed (its `207001` is likely + transient). Adding a bounded retry (≤3 attempts) of just that test on a + freshly-reset card would turn the last red into green — a follow-up step. +- a2a3 now carries the mirrored guards (see "Why this shape") but they are + CI-verified only — no a2a3 silicon on this box. If the `st-onboard-a2a3` + job ever shows the force reset failing or not recovering on real a2a3 + hardware, revisit `force_reset_device()` there (the CANN force-reset + semantics may differ from a5). ## References -- CI run 27090242388, job `st-onboard-a5` (`gh run view --job 79952403308`). -- Fix: `src/a5/platform/onboard/host/device_runner.{h,cpp}` +- CI run 27090242388, job `st-onboard-a5` (`gh run view --job 79952403308`) — + original 8-failure cascade. CI run 27182822054 — post-#1005 contained run + (`1 failed, 13 skipped`). +- Containment (#1005): `src/a5/platform/onboard/host/device_runner.{h,cpp}` (`device_unusable_`, `recover_device_or_mark_unusable`), `conftest.py` (`pytest_runtest_makereport`, `_register_l2_pool_heal`). +- Force-reset recovery: `src/a5/platform/onboard/host/device_runner.{h,cpp}` + (`force_reset_device`, called from `finalize()` on the `device_unusable_` + path). +- a2a3 mirror (CI-verified only): `src/a2a3/platform/onboard/host/` + `device_runner.{h,cpp}` — identical `device_unusable_` / + `recover_device_or_mark_unusable` / `force_reset_device` chain. - `DeviceRunnerBase::finalize_common` — prior note on why finalize skips a pre-destroy stream sync on the error-state stream. - `.claude/rules/running-onboard.md` — `task-submit` device isolation. +- `.claude/rules/env-macro-gating.md` — why force reset is unconditional + (no opt-in flag) rather than gated. diff --git a/src/a2a3/platform/onboard/host/device_runner.cpp b/src/a2a3/platform/onboard/host/device_runner.cpp index 4034ba21c1..e0dc7742e0 100644 --- a/src/a2a3/platform/onboard/host/device_runner.cpp +++ b/src/a2a3/platform/onboard/host/device_runner.cpp @@ -184,6 +184,23 @@ int DeviceRunner::destroy_comm_stream(void *stream) { // `src/common/platform/onboard/host/device_runner_base.cpp`. int DeviceRunner::run(Runtime &runtime, int block_dim, int launch_aicpu_num) { + // A prior AICore launch/sync error poisoned the device context and the + // in-place drain could not clear it. Refuse to run rather than cascade into + // rtMalloc 507899 / halResMap rc=62 (init_aicore_register_addresses). A soft + // close()+reset does NOT clear the poison, but finalize() force-resets the + // card on this path so the next Worker re-inits clean in the same process + // (see force_reset_device()). Failing fast here turns the rest of an xdist + // worker session's tests from a slow, confusing failure cascade into a + // single fast, self-explanatory error; the runner is then recovered at + // finalize. + if (device_unusable_) { + LOG_ERROR( + "DeviceRunner marked unusable by a prior AICore failure; refusing to run. " + "A soft reset does not clear the poison; finalize() will force-reset " + "the card so the next Worker on it inits clean." + ); + return -1; + } if (validate_launch_aicpu_num(launch_aicpu_num) != 0) return -1; int rc = ensure_device_initialized(); @@ -371,11 +388,20 @@ int DeviceRunner::run(Runtime &runtime, int block_dim, int launch_aicpu_num) { rc = launch_aicore_kernel(stream_aicore_, kernel_args_.device_k_args_); if (rc != 0) { LOG_ERROR("launch_aicore_kernel failed: %d", rc); + recover_device_or_mark_unusable(rc); return rc; } rc = sync_run_streams(); - if (rc != 0) return rc; + if (rc != 0) { + // sync_run_streams surfaces the AICore op-timeout (STARS-reaped op -> + // 507000/507018/507046 at AICPU/AICore stream sync). The op-timeout + // leaves the device context poisoned for the SAME DeviceRunner's next + // run, so attempt recovery / mark-unusable here too, not only on the + // launch-error path above. + recover_device_or_mark_unusable(rc); + return rc; + } read_device_wall_ns(); @@ -407,6 +433,127 @@ int DeviceRunner::run(Runtime &runtime, int block_dim, int launch_aicpu_num) { // `bind_callable_to_runtime`, and `upload_chip_callable_buffer` live on // `DeviceRunnerBase`. +void DeviceRunner::recover_device_or_mark_unusable(int aicore_rc) { + // An AICore launch failure (207001) or an op-timeout reaped by STARS + // (surfaced as 507000/507018/507046 at stream sync) leaves the device + // context in a sticky-error state: the streams stay poisoned and the + // SAME DeviceRunner's next run() fails early — observed on a2a3 as + // `rtMalloc failed: 507899`, and on a5 as `halResMap failed (rc=62)` in + // init_aicore_register_addresses. Reused across a session (the L2 + // st_worker pool hands one ChipWorker to every test class on a device), + // that one error poisons every later test in the xdist worker process. + // + // Try to drain the device so a subsequent run can recover in place. Use + // the BOUNDED aclrtSynchronizeDeviceWithTimeout, NOT an unbounded + // aclrtSynchronizeStream* on the error-state stream — the latter wedges + // subsequent tests (see DeviceRunnerBase::finalize_common's comment on + // why finalize deliberately skips a pre-destroy sync). If the bounded + // device drain itself errors, the context is unrecoverable without a full + // device reset, so mark the runner unusable: run() then fails fast with a + // clear message instead of cascading, and finalize() force-resets the card. + int sync_rc = aclrtSynchronizeDeviceWithTimeout(PLATFORM_STREAM_SYNC_TIMEOUT_MS); + if (sync_rc != ACL_SUCCESS) { + LOG_ERROR( + "Device unrecoverable after AICore error %d: aclrtSynchronizeDeviceWithTimeout failed: %d. " + "Marking DeviceRunner unusable; run() will fail fast and finalize() will force-reset the " + "card (a soft in-process reset does not clear the poison).", + aicore_rc, sync_rc + ); + device_unusable_ = true; + } else { + LOG_WARN("AICore error %d: device drained via aclrtSynchronizeDeviceWithTimeout", aicore_rc); + } +} + +namespace { + +// RAII: bring ACL up if needed, and finalize on scope exit ONLY if this guard +// is the one that initialized it. On the L2 poison path the rt-layer runner +// never brought ACL up (acl_ready_ is false, so finalize() did a bare +// rtDeviceReset and no aclFinalize), so aclInit here genuinely initializes ACL +// and we own its teardown. If some other owner already init'd ACL, aclInit +// returns 100002 and we leave it alone — finalizing it would tear ACL down for +// the rest of the process. +class AclInitGuard { +public: + AclInitGuard() { + aclError rc = aclInit(nullptr); + if (rc == ACL_SUCCESS) { + owns_ = true; + ok_ = true; + } else if (static_cast(rc) == ACL_ERROR_REPEAT_INITIALIZE) { + ok_ = true; + } else { + LOG_ERROR("force_reset_device: aclInit failed: %d", static_cast(rc)); + } + } + ~AclInitGuard() { + if (owns_) { + (void)aclFinalize(); + } + } + AclInitGuard(const AclInitGuard &) = delete; + AclInitGuard &operator=(const AclInitGuard &) = delete; + bool ok() const { return ok_; } + +private: + bool owns_{false}; + bool ok_{false}; +}; + +// RAII: bind the device to this thread for the force reset, and unbind it on +// scope exit so the per-thread device reference does not leak into the next +// Worker on this card. +class DeviceBindGuard { +public: + explicit DeviceBindGuard(int device_id) : + device_id_(device_id) { + aclError rc = aclrtSetDevice(device_id_); + if (rc == ACL_SUCCESS) { + bound_ = true; + } else { + LOG_ERROR("force_reset_device: aclrtSetDevice(%d) failed: %d", device_id_, static_cast(rc)); + } + } + ~DeviceBindGuard() { + if (bound_) { + (void)aclrtResetDevice(device_id_); + } + } + DeviceBindGuard(const DeviceBindGuard &) = delete; + DeviceBindGuard &operator=(const DeviceBindGuard &) = delete; + bool bound() const { return bound_; } + +private: + int device_id_; + bool bound_{false}; +}; + +} // namespace + +void DeviceRunner::force_reset_device() { + if (device_id_ < 0) { + return; + } + // aclrtResetDeviceForce is an ACL API; bring ACL up and bind the device, + // both released on scope exit (LIFO) so a repeated poison-then-reset cycle + // in a long-lived process leaks neither ACL state nor a device reference. + AclInitGuard acl_guard; + if (!acl_guard.ok()) { + return; + } + DeviceBindGuard bind_guard(device_id_); + if (!bind_guard.bound()) { + return; + } + aclError rc = aclrtResetDeviceForce(device_id_); + if (rc != ACL_SUCCESS) { + LOG_ERROR("force_reset_device: aclrtResetDeviceForce(%d) failed: %d", device_id_, static_cast(rc)); + } else { + LOG_WARN("force_reset_device: aclrtResetDeviceForce(%d) cleared the poisoned card", device_id_); + } +} + int DeviceRunner::finalize() { if (device_id_ == -1) { return 0; @@ -465,7 +612,22 @@ int DeviceRunner::finalize() { } } + // On the poison path the soft reset above does NOT clear the op-timeout + // sticky-error — a fresh in-process Worker.init then fails at rtStreamCreate + // 507899. A FORCE reset clears it, so the next Worker on this card inits + // clean in the SAME process and the remaining tests run instead of cascading + // / being skipped. Only reached on the (rare) device-poison path; onboard + // work always holds an exclusive task-submit lock on the card (enforced by + // .claude/rules/running-onboard.md), and the reset scopes to this card alone, + // so it cannot disturb other devices/users. + if (device_unusable_) { + force_reset_device(); + } + device_id_ = -1; + // A finalized runner has reset the device; clear the poison flag so a + // reused DeviceRunner object (re-init after close) starts clean. + device_unusable_ = false; LOG_INFO_V0("DeviceRunner finalized"); return rc; } diff --git a/src/a2a3/platform/onboard/host/device_runner.h b/src/a2a3/platform/onboard/host/device_runner.h index 795bee313a..501c76b25f 100644 --- a/src/a2a3/platform/onboard/host/device_runner.h +++ b/src/a2a3/platform/onboard/host/device_runner.h @@ -200,6 +200,36 @@ class DeviceRunner : public DeviceRunnerBase { // acl_ready_, so runtimes that never ask for ACL (e.g. pure rt-layer) stay unaffected. bool acl_ready_{false}; + // Set true when an AICore launch/sync error (e.g. an op-timeout reaped by + // STARS, surfaced as 507000/507018 at stream sync, or a 207001 launch + // failure) left the device context in a sticky-error state that an + // in-place drain could not clear. Once set, run() fails fast instead of + // cascading into the confusing downstream failures (rtMalloc 507899, or + // halResMap rc=62 at init_aicore_register_addresses) that a poisoned + // context produces. The poison survives a close()+soft-reset for the life + // of the process (an in-process re-init fails with rtStreamCreate 507899), + // but a *force* reset clears it: finalize() calls force_reset_device() on + // this path so the next Worker re-inits clean in the same process (see + // force_reset_device()). This flag fails run() fast and drives that + // recovery. See run() and recover_device_or_mark_unusable(). + bool device_unusable_{false}; + + // On an AICore launch/sync error, best-effort drain the device so a later + // run() on the same DeviceRunner can recover in place; if the drain itself + // errors the context is unrecoverable without a full reset, so flip + // device_unusable_ and let run() fail fast. + void recover_device_or_mark_unusable(int aicore_rc); + + // Force-reset the card via aclrtResetDeviceForce to clear an op-timeout + // sticky-error that the soft rtDeviceReset cannot (a soft reset + fresh + // in-process Worker.init still fails at rtStreamCreate 507899, whereas a + // force reset lets the next init succeed in the same process). Called from + // finalize() only on the device-poison path (device_unusable_). Safe + // because onboard work always holds an exclusive task-submit lock on the + // card (.claude/rules/running-onboard.md) and the reset scopes to this card + // only (does not disturb other devices). + void force_reset_device(); + // Shared collectors (`l2_swimlane_collector_`, `dump_collector_`, // `pmu_collector_`, `scope_stats_collector_`) live on `DeviceRunnerBase`. // diff --git a/src/a5/platform/onboard/host/device_runner.cpp b/src/a5/platform/onboard/host/device_runner.cpp index f65bac1ba2..7e19b9f2b6 100644 --- a/src/a5/platform/onboard/host/device_runner.cpp +++ b/src/a5/platform/onboard/host/device_runner.cpp @@ -146,16 +146,17 @@ int DeviceRunner::run(Runtime &runtime, int block_dim, int launch_aicpu_num) { // A prior AICore launch/sync error poisoned the device context and the // in-place drain could not clear it. Refuse to run rather than cascade // into halResMap rc=62 (init_aicore_register_addresses) or rtMalloc - // 507899. On a5 the poison is unrecoverable in-process (even close()+reset - // + a fresh Worker.init fails with rtStreamCreate 507899); only a fresh - // process clears it. Failing fast here turns the rest of an xdist worker - // session's tests from a slow, confusing failure cascade into a single - // fast, self-explanatory error (the test fixture then skips the remainder). + // 507899. A soft close()+reset does NOT clear the poison on a5, but + // finalize() force-resets the card on this path so the next Worker re-inits + // clean in the same process (see force_reset_device()). Failing fast here + // turns the rest of an xdist worker session's tests from a slow, confusing + // failure cascade into a single fast, self-explanatory error; the runner is + // then recovered at finalize. if (device_unusable_) { LOG_ERROR( "DeviceRunner marked unusable by a prior AICore failure; refusing to run. " - "The device context is unrecoverable in-process on a5 (close + re-create does " - "not reset it); a fresh worker process is required to recover." + "A soft reset does not clear the poison on a5; finalize() will force-reset " + "the card so the next Worker on it inits clean." ); return -1; } @@ -438,6 +439,96 @@ void DeviceRunner::recover_device_or_mark_unusable(int aicore_rc) { } } +namespace { + +// RAII: bring ACL up if needed, and finalize on scope exit ONLY if this guard +// is the one that initialized it. On the L2 poison path the rt-layer runner +// never brought ACL up (acl_ready_ is false, so finalize() did a bare +// rtDeviceReset and no aclFinalize), so aclInit here genuinely initializes ACL +// and we own its teardown. If some other owner already init'd ACL, aclInit +// returns 100002 and we leave it alone — finalizing it would tear ACL down for +// the rest of the process. +class AclInitGuard { +public: + AclInitGuard() { + constexpr int kAclRepeatInit = 100002; + aclError rc = aclInit(nullptr); + if (rc == ACL_SUCCESS) { + owns_ = true; + ok_ = true; + } else if (static_cast(rc) == kAclRepeatInit) { + ok_ = true; + } else { + LOG_ERROR("force_reset_device: aclInit failed: %d", static_cast(rc)); + } + } + ~AclInitGuard() { + if (owns_) { + (void)aclFinalize(); + } + } + AclInitGuard(const AclInitGuard &) = delete; + AclInitGuard &operator=(const AclInitGuard &) = delete; + bool ok() const { return ok_; } + +private: + bool owns_{false}; + bool ok_{false}; +}; + +// RAII: bind the device to this thread for the force reset, and unbind it on +// scope exit so the per-thread device reference does not leak into the next +// Worker on this card. +class DeviceBindGuard { +public: + explicit DeviceBindGuard(int device_id) : + device_id_(device_id) { + aclError rc = aclrtSetDevice(device_id_); + if (rc == ACL_SUCCESS) { + bound_ = true; + } else { + LOG_ERROR("force_reset_device: aclrtSetDevice(%d) failed: %d", device_id_, static_cast(rc)); + } + } + ~DeviceBindGuard() { + if (bound_) { + (void)aclrtResetDevice(device_id_); + } + } + DeviceBindGuard(const DeviceBindGuard &) = delete; + DeviceBindGuard &operator=(const DeviceBindGuard &) = delete; + bool bound() const { return bound_; } + +private: + int device_id_; + bool bound_{false}; +}; + +} // namespace + +void DeviceRunner::force_reset_device() { + if (device_id_ < 0) { + return; + } + // aclrtResetDeviceForce is an ACL API; bring ACL up and bind the device, + // both released on scope exit (LIFO) so a repeated poison-then-reset cycle + // in a long-lived process leaks neither ACL state nor a device reference. + AclInitGuard acl_guard; + if (!acl_guard.ok()) { + return; + } + DeviceBindGuard bind_guard(device_id_); + if (!bind_guard.bound()) { + return; + } + aclError rc = aclrtResetDeviceForce(device_id_); + if (rc != ACL_SUCCESS) { + LOG_ERROR("force_reset_device: aclrtResetDeviceForce(%d) failed: %d", device_id_, static_cast(rc)); + } else { + LOG_WARN("force_reset_device: aclrtResetDeviceForce(%d) cleared the poisoned card", device_id_); + } +} + // `print_handshake_results`, `prepare_orch_so`, `register_callable`, // `register_callable_host_orch`, `unregister_callable`, `has_callable`, // `bind_callable_to_runtime`, and `upload_chip_callable_buffer` live on @@ -506,6 +597,18 @@ int DeviceRunner::finalize() { } } + // On the poison path the soft reset above does NOT clear the op-timeout + // sticky-error — a fresh in-process Worker.init then fails at rtStreamCreate + // 507899. A FORCE reset clears it, so the next Worker on this card inits + // clean in the SAME process and the remaining tests run instead of cascading + // / being skipped. Only reached on the (rare) device-poison path; onboard + // work always holds an exclusive task-submit lock on the card (enforced by + // .claude/rules/running-onboard.md), and the reset is verified to scope to + // this card alone, so it cannot disturb other devices/users. + if (device_unusable_) { + force_reset_device(); + } + device_id_ = -1; // A finalized runner has reset the device; clear the poison flag so a // reused DeviceRunner object (re-init after close) starts clean. diff --git a/src/a5/platform/onboard/host/device_runner.h b/src/a5/platform/onboard/host/device_runner.h index 7f0ca87943..2084c45ccd 100644 --- a/src/a5/platform/onboard/host/device_runner.h +++ b/src/a5/platform/onboard/host/device_runner.h @@ -206,11 +206,12 @@ class DeviceRunner : public DeviceRunnerBase { // in-place drain could not clear. Once set, run() fails fast instead of // cascading into the confusing downstream failures (halResMap rc=62 at // init_aicore_register_addresses, or rtMalloc 507899) that a poisoned - // context produces. On a5 the poison survives close()+device-reset for the + // context produces. On a5 the poison survives a close()+soft-reset for the // life of the process (an in-process re-init fails with rtStreamCreate - // 507899) and a force-reset is unsafe on shared silicon, so the only real - // recovery is a fresh process — this flag just contains the blast radius. - // See run() and recover_device_or_mark_unusable(). + // 507899), but a *force* reset clears it: finalize() calls + // force_reset_device() on this path so the next Worker re-inits clean in the + // same process (see force_reset_device()). This flag fails run() fast and + // drives that recovery. See run() and recover_device_or_mark_unusable(). bool device_unusable_{false}; // On an AICore launch/sync error, best-effort drain the device so a later @@ -219,6 +220,16 @@ class DeviceRunner : public DeviceRunnerBase { // device_unusable_ and let run() fail fast. void recover_device_or_mark_unusable(int aicore_rc); + // Force-reset the card via aclrtResetDeviceForce to clear an op-timeout + // sticky-error that the soft rtDeviceReset cannot (verified: a soft reset + // + fresh in-process Worker.init still fails at rtStreamCreate 507899, + // whereas a force reset lets the next init succeed in the same process). + // Called from finalize() only on the device-poison path (device_unusable_). + // Safe because onboard work always holds an exclusive task-submit lock on + // the card (.claude/rules/running-onboard.md) and the reset is verified to + // scope to this card only (does not disturb other devices). + void force_reset_device(); + /** * Initialize performance profiling device buffers *