Fix: per-device staged-SO naming to stop paired-die AICPU device fault (507899 cascade)

docs/ci.md

@@ -224,3 +224,4 @@ No `--platform` means "run all sims" — tests with no sim in their `platforms`
 
 - **macOS libomp collision**: on macOS, the root `conftest.py` sets `KMP_DUPLICATE_LIB_OK=TRUE` before `import pytest` to work around a duplicate-libomp abort triggered by homebrew numpy and pip torch coexisting in one Python process (see [troubleshooting/macos-libomp-collision.md](troubleshooting/macos-libomp-collision.md)). Standalone `python test_*.py` bypasses conftest — rely on the env var being exported by the shell or `tools/verify_packaging.sh`.
 - **sim hangs / `rc=-1` under CPU oversubscription**: on a few-vCPU runner, high `--max-parallel` (or many concurrent sim cases) oversubscribes the host CPUs, where sim's busy-spin handshake can livelock (hang → `rc=124`) or the deinit timeout can false-trip (`run_prepared failed with code -1`). Mitigate with `--max-parallel 2`; onboard is unaffected (see [troubleshooting/sim-oversubscription-hang.md](troubleshooting/sim-oversubscription-hang.md)).
+- **`st-onboard-a2a3` mass 507899 is not OOM**: a whole-suite collapse of `507899`/`507018`/`prepare_callable -1` is an AICPU device-fault cascade (`simpler_aicpu_exec` exception), not memory exhaustion. Diagnosis recipe and the per-device preinstall-name fix are in [troubleshooting/a2a3-507899-aicpu-shared-so-fault.md](troubleshooting/a2a3-507899-aicpu-shared-so-fault.md).

docs/troubleshooting/a2a3-507899-aicpu-shared-so-fault.md

@@ -0,0 +1,111 @@
+# st-onboard-a2a3 mass 507899/507018 — AICPU shared-SO device fault
+
+## Symptom
+
+`st-onboard-a2a3` fails intermittently (observed ~19% of runs after #870, vs
+~10% before) with a whole-suite collapse: the `L2 tensormap_and_ringbuffer`
+phase reports ~10 failed + ~23 errors at once. The surfaced Python errors are
+
+```text
+RuntimeError: simpler_init failed with code 507899
+RuntimeError: prepare_callable failed with code -1
+RuntimeError: run_prepared failed with code 507018 / 507046 / 507901
+[ERROR] rtMalloc failed: 507899 (size=...)
+[ERROR] Failed to allocate device GM for ChipCallable buffer
+```
+
+It looks like an out-of-memory. **It is not.** `npu-smi info` taken right after
+the failure shows every chip `Health=OK` with HBM ~3 GB / 64 GB used.
+
+## Root cause
+
+`507899` decodes (in the CANN runtime slog) as `[driver error:internal error]`
+and `507901` as `[hdc disconnect]`. They are a **cascade after a device fault**,
+not the cause. The earliest real error is an **AICPU exception**:
+
+```text
+ProcLogicCqReport: Task run failed, sqe_type=1(aicpu), errType=0x1(task exception)
+ProcessStarsAicpuErrorInfo: error from device(chipId:N, dieId:0/1),
+                            an exception occurred during AICPU execution
+PrintAicpuErrorInfo: Aicpu kernel execute failed,
+                     soName=simpler_inner_<fp>.so, funcName=simpler_aicpu_exec,
+                     errorCode=0x2a                       # 507018 ACL_ERROR_RT_AICPU_EXCEPTION
+```
+
+`simpler_aicpu_exec` faults the **whole chip**; afterwards every
+`rtStreamCreate` / `rtMalloc` on that chip returns 507899/507901, so the next
+test's `simpler_init` (which eagerly creates streams since #870) fails and the
+rest of the suite collapses.
+
+Two facts pin the mechanism:
+
+1. On a2a3 the `npu-smi` Phy-IDs pair as **die0/die1 of one Ascend910**
+   (devices 8/9, 4/5, …). The exception fires on **both dies of one chip at the
+   same instant** — a chip-shared resource was corrupted.
+2. The runtime stages SOs under the **shared preinstall directory**
+   `/usr/lib64/aicpu_kernels/0/aicpu_kernels_device/`. The dispatcher (#870)
+   wrote the AICPU runtime SO there under a **content-fingerprint-only** name
+   `simpler_inner_<fp>.so` — identical across both dies. Paired dies share that
+   filesystem, so both wrote/renamed/executed the same file; concurrent
+   bootstrap corrupted the mmap'd image and trapped `simpler_aicpu_exec`.
+
+A single-die 50× solo loop of the unregister/re-prepare/dedup tests **never**
+reproduced; only the parallel multi-die suite did — consistent with a cross-die
+shared-file race, not an intra-process use-after-free.
+
+## How this was diagnosed (repeatable recipe)
+
+The host-side simpler log shows only the cascade. To see the originating AICPU
+exception you must surface the CANN device slog, which is otherwise hidden:
+
+1. Run the job with the Ascend slog redirected to stdout (the test harness's
+   `parallel_scheduler` captures each case's stdout into a per-case `::group::`):
+
+   ```bash
+   ASCEND_SLOG_PRINT_TO_STDOUT=1 ASCEND_GLOBAL_LOG_LEVEL=1 \
+   python -m pytest examples tests/st --platform a2a3 --device <range> -v \
+     --pto-session-timeout 600
+   ```
+
+   (`pto_runtime_c_api.cpp` skips its own `dlog_setlevel` when
+   `ASCEND_GLOBAL_LOG_LEVEL` is set, so the env value wins.)
+
+2. In the captured log, find the **first** `PrintAicpuErrorInfo` /
+   `ProcessStarsAicpuErrorInfo` — note `soName`, `funcName`, `errorCode`, and the
+   `chipId/dieId`. Everything after it that says `507899 [driver error...]` is
+   noise.
+
+3. `npu-smi info` confirms whether it is really OOM (it is not here).
+
+4. Device-side AICPU dumps under `/var/log/npu/...` and `msnpureport` are
+   **not readable by the CI runner user** (`Not have permission to export
+   files`), so the stdout slog above is the only channel from CI. The exception
+   *type* (0x2a) plus the faulting `soName/funcName` is enough to localize.
+
+## Fix
+
+Make the staged SO names **per-device** so paired dies never touch the same
+file:
+
+- Dispatcher inner SO: `simpler_inner_<fp>_<device_id>.so`
+  (`src/common/aicpu_dispatcher/aicpu_dispatcher.cpp` `MakeInnerSoPath`,
+  `src/common/host/load_aicpu_op.cpp` `MakeInnerSoBasename`). The real
+  `device_id` (was hardcoded `0`) is threaded from `DeviceRunner` through
+  `BootstrapDispatcher` into both the host JSON reader name and the device-side
+  writer (`DeviceArgs.device_id`). Bootstrap cache keyed by `(fp, device_id)`.
+- AICPU executor orchestration SO: `libdevice_orch_<pid>_<cid>_<device_id>.so`
+  (`create_orch_so_file`). `device_id` reaches the executor via a new
+  `KernelArgs.device_id` field → `set_orch_device_id()` in `kernel.cpp` →
+  `get_orch_device_id()` in `aicpu_executor.cpp`. This one was already pid-named
+  (mostly safe); the suffix is defense-in-depth for the case where two dies'
+  device-side pids collide.
+
+Applied symmetrically to a2a3 and a5.
+
+## Confirming the fix
+
+Rerun `st-onboard-a2a3` several times (it is flaky, so one green run is weak
+evidence). The fix landed with 5/5 consecutive passes plus the root-cause
+match. If the AICPU 0x2a fault ever returns, re-run the slog recipe above and
+check whether any *other* SO staged under the shared `aicpu_kernels/0/`
+directory still uses a non-per-device name.

src/a2a3/platform/include/aicpu/orch_so_file.h

@@ -44,10 +44,14 @@
  *                       resident orch SOs (one per cid) do not collide on
  *                       the same on-disk file. Pass -1 for the legacy
  *                       single-slot dispatch path.
+ * @param device_id      ACL device ordinal, suffixed onto the onboard name so
+ *                       the paired dies of one chip (which share this preinstall
+ *                       filesystem) never stage/execute the same on-disk file.
  * @param out_path       Buffer that receives the full file path on success
  * @param out_path_size  Size of `out_path` in bytes
  * @return Open writable fd on success, -1 on failure
  */
-int32_t create_orch_so_file(const char *dir, int32_t callable_id, char *out_path, size_t out_path_size);
+int32_t
+create_orch_so_file(const char *dir, int32_t callable_id, int32_t device_id, char *out_path, size_t out_path_size);
 
 #endif  // PLATFORM_AICPU_ORCH_SO_FILE_H_

src/a2a3/platform/include/aicpu/platform_regs.h

@@ -66,6 +66,17 @@ void set_platform_pmu_reg_addrs(uint64_t pmu_regs);
  */
 uint64_t get_platform_pmu_reg_addrs();
 
+/**
+ * Set the ACL device ordinal for the current run. Pushed by the platform layer
+ * (kernel.cpp) before aicpu_execute() from KernelArgs.device_id; the executor
+ * reads it to make the staged orchestration SO filename unique per device so
+ * paired dies sharing the preinstall filesystem never collide.
+ */
+void set_orch_device_id(int device_id);
+
+/** Get the ACL device ordinal set for the current run (0 if unset). */
+int get_orch_device_id();
+
 #ifdef __cplusplus
 }
 #endif

src/a2a3/platform/include/common/kernel_args.h

@@ -108,6 +108,13 @@ struct KernelArgs {
     // (no special-casing — write-through works because sim AICPU and host
     // share memory). Zero when the buffer was not allocated.
     uint64_t device_wall_data_base{0};
+    // ACL device ordinal. Pushed to the AICPU so the executor can suffix the
+    // staged orchestration SO name (libdevice_orch_<pid>_<cid>_<device_id>.so):
+    // paired a2a3 dies share the preinstall filesystem, and a content/pid-only
+    // name risks a cross-die write/execute collision (see simpler_inner fix).
+    // Trailing field — keeps the CANN-fixed front offsets and AICore-read
+    // fields in place.
+    uint32_t device_id{0};
 };
 
 #ifdef __cplusplus

src/a2a3/platform/onboard/aicpu/CMakeLists.txt

@@ -91,8 +91,8 @@ set_target_properties(aicpu_kernel PROPERTIES OUTPUT_NAME aicpu_kernel)
 # rtAicpuKernelLaunchExWithArgs(KERNEL_TYPE_AICPU_KFC), invokes
 # DynTileFwkBackendKernelServerInit, which writes the bundled inner SO bytes
 # (passed via the extended DeviceArgs at offsets 120/128) to
-# /usr/lib64/aicpu_kernels/0/aicpu_kernels_device/simpler_inner_<fp>.so under
-# a content-fingerprint basename. After bootstrap the host registers the
+# /usr/lib64/aicpu_kernels/0/aicpu_kernels_device/simpler_inner_<fp>_<device_id>.so
+# under a content-fingerprint + device-id basename. After bootstrap the host registers the
 # preinstall file via rtsBinaryLoadFromFile + rtsFuncGetByName and launches
 # per-task through rtsLaunchCpuKernel; the dispatcher SO itself is never
 # referenced again.

src/a2a3/platform/onboard/aicpu/kernel.cpp

@@ -104,6 +104,8 @@ extern "C" __attribute__((visibility("default"))) int simpler_aicpu_exec(void *a
     // Dump enable is an execution control flag propagated via handshake.
     // The dump base address is only the backing storage location.
     set_platform_regs(k_args->regs);
+    // Device ordinal for per-device orchestration-SO naming in the executor.
+    set_orch_device_id(static_cast<int>(k_args->device_id));
     set_platform_dump_base(k_args->dump_data_base);
     set_dump_tensor_enabled(GET_PROFILING_FLAG(k_args->enable_profiling_flag, PROFILING_FLAG_DUMP_TENSOR));
     set_platform_l2_perf_base(k_args->l2_perf_data_base);

src/a2a3/platform/onboard/aicpu/orch_so_file.cpp

@@ -15,19 +15,24 @@
 
 #include <cstdio>
 
-int32_t create_orch_so_file(const char *dir, int32_t callable_id, char *out_path, size_t out_path_size) {
-    // Pid + callable_id naming: AICPU device libc may lack mkstemps. With
-    // per-callable_id dispatch, multiple orch SOs can be resident in the
-    // same device process at once (one per cid in `orch_so_table_`), so
-    // the on-disk file name must be unique per cid — otherwise the
-    // second cid's `O_TRUNC` would silently shred the first cid's already
-    // dlopen'd file image and the next launch on cid=0 would SIGBUS.
-    // callable_id < 0 is the legacy single-slot path: pid alone is fine.
+int32_t
+create_orch_so_file(const char *dir, int32_t callable_id, int32_t device_id, char *out_path, size_t out_path_size) {
+    // Pid + callable_id + device_id naming: AICPU device libc may lack mkstemps.
+    // With per-callable_id dispatch, multiple orch SOs can be resident in the
+    // same device process at once (one per cid in `orch_so_table_`), so the
+    // on-disk file name must be unique per cid — otherwise the second cid's
+    // `O_TRUNC` would silently shred the first cid's already dlopen'd file
+    // image and the next launch on cid=0 would SIGBUS. The device_id suffix
+    // additionally isolates the paired dies of one chip, which share this
+    // preinstall filesystem and would otherwise race on one shared file if
+    // their device-side pids collided (mirrors the simpler_inner fix).
+    // callable_id < 0 is the legacy single-slot path.
     int32_t written;
     if (callable_id >= 0) {
-        written = snprintf(out_path, out_path_size, "%s/libdevice_orch_%d_%d.so", dir, getpid(), callable_id);
+        written =
+            snprintf(out_path, out_path_size, "%s/libdevice_orch_%d_%d_%d.so", dir, getpid(), callable_id, device_id);
     } else {
-        written = snprintf(out_path, out_path_size, "%s/libdevice_orch_%d.so", dir, getpid());
+        written = snprintf(out_path, out_path_size, "%s/libdevice_orch_%d_dev%d.so", dir, getpid(), device_id);
     }
     if (written < 0 || static_cast<size_t>(written) >= out_path_size) {
         return -1;

src/a2a3/platform/onboard/host/device_runner.cpp

@@ -352,15 +352,15 @@ int DeviceRunner::ensure_binaries_loaded() {
 
     // One-shot bootstrap: libaicpu_extend_kernels invokes our dispatcher,
     // which writes the runtime AICPU SO bytes to
-    // /usr/lib64/aicpu_kernels/0/aicpu_kernels_device/simpler_inner_<fp>.so
+    // /usr/lib64/aicpu_kernels/0/aicpu_kernels_device/simpler_inner_<fp>_<device_id>.so
     // using sched-thread (HwHiAiUser) write permission. The dispatcher SO
     // itself is never persisted to disk — only the transient
     // libaicpu_extend_kernels dlopen. Subsequent per-task AICPU launches
     // resolve symbols via rtsBinaryLoadFromFile + rtsFuncGetByName +
     // rtsLaunchCpuKernel directly against the preinstall file.
     int rc = load_aicpu_op_.BootstrapDispatcher(
         dispatcher_so_binary_.data(), dispatcher_so_binary_.size(), aicpu_so_binary_.data(), aicpu_so_binary_.size(),
-        stream_aicpu_
+        stream_aicpu_, device_id_
     );
     if (rc != 0) {
         LOG_ERROR("LoadAicpuOp::BootstrapDispatcher failed: %d", rc);
@@ -667,6 +667,8 @@ int DeviceRunner::run(Runtime &runtime, int block_dim, int launch_aicpu_num) {
     // dlopen'd). Read it directly when populating KernelArgs.
     kernel_args_.args.log_level = static_cast<uint32_t>(HostLogger::get_instance().level());
     kernel_args_.args.log_info_v = static_cast<uint32_t>(HostLogger::get_instance().info_v());
+    // Device ordinal for the AICPU executor's per-device orchestration-SO name.
+    kernel_args_.args.device_id = static_cast<uint32_t>(device_id_);
 
     rc = kernel_args_init_ffts_base_addr(kernel_args_);
     if (rc != 0) {

src/a2a3/platform/sim/aicpu/orch_so_file.cpp

@@ -24,16 +24,19 @@
 
 #include <cstdio>
 
-int32_t create_orch_so_file(const char *dir, int32_t callable_id, char *out_path, size_t out_path_size) {
+int32_t
+create_orch_so_file(const char *dir, int32_t callable_id, int32_t device_id, char *out_path, size_t out_path_size) {
     // mkstemps: multiple sim workers can share a process, so names must be
     // unique per call.  The "XXXXXX" template is replaced in-place.
-    // callable_id is embedded purely for log readability (mkstemps already
-    // guarantees uniqueness regardless).
+    // callable_id / device_id are embedded purely for log readability
+    // (mkstemps already guarantees uniqueness; sim has no shared preinstall
+    // filesystem, so the onboard cross-die concern does not apply here).
     int32_t written;
     if (callable_id >= 0) {
-        written = snprintf(out_path, out_path_size, "%s/libdevice_orch_cid%d_XXXXXX.so", dir, callable_id);
+        written =
+            snprintf(out_path, out_path_size, "%s/libdevice_orch_cid%d_dev%d_XXXXXX.so", dir, callable_id, device_id);
     } else {
-        written = snprintf(out_path, out_path_size, "%s/libdevice_orch_XXXXXX.so", dir);
+        written = snprintf(out_path, out_path_size, "%s/libdevice_orch_dev%d_XXXXXX.so", dir, device_id);
     }
     if (written < 0 || static_cast<size_t>(written) >= out_path_size) {
         return -1;

src/a2a3/platform/src/aicpu/platform_regs.cpp

@@ -35,6 +35,7 @@
 
 static uint64_t g_platform_regs = 0;
 static uint64_t g_platform_pmu_reg_addrs = 0;
+static int g_orch_device_id = 0;
 
 void set_platform_regs(uint64_t regs) { g_platform_regs = regs; }
 
@@ -44,6 +45,10 @@ void set_platform_pmu_reg_addrs(uint64_t pmu_regs) { g_platform_pmu_reg_addrs =
 
 uint64_t get_platform_pmu_reg_addrs() { return g_platform_pmu_reg_addrs; }
 
+void set_orch_device_id(int device_id) { g_orch_device_id = device_id; }
+
+int get_orch_device_id() { return g_orch_device_id; }
+
 volatile uint32_t *get_reg_ptr(uint64_t reg_base_addr, RegId reg) {
     return reinterpret_cast<volatile uint32_t *>(reg_base_addr + reg_offset(reg));
 }

src/a2a3/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp

@@ -275,7 +275,9 @@ int32_t AicpuExecutor::run(Runtime *runtime) {
                 const int32_t num_candidates = sizeof(candidate_dirs) / sizeof(candidate_dirs[0]);
 
                 for (int32_t i = 0; i < num_candidates && !file_created; i++) {
-                    int32_t fd = create_orch_so_file(candidate_dirs[i], callable_id, so_path, sizeof(so_path));
+                    int32_t fd = create_orch_so_file(
+                        candidate_dirs[i], callable_id, get_orch_device_id(), so_path, sizeof(so_path)
+                    );
                     if (fd < 0) {
                         LOG_INFO_V0(
                             "Thread %d: Cannot create SO at %s (errno=%d), trying next path", thread_idx, so_path, errno

src/a5/platform/include/aicpu/orch_so_file.h

@@ -44,10 +44,14 @@
  *                       resident orch SOs (one per cid) do not collide on
  *                       the same on-disk file. Pass -1 for the legacy
  *                       single-slot dispatch path.
+ * @param device_id      ACL device ordinal, suffixed onto the onboard name so
+ *                       the paired dies of one chip (which share this preinstall
+ *                       filesystem) never stage/execute the same on-disk file.
  * @param out_path       Buffer that receives the full file path on success
  * @param out_path_size  Size of `out_path` in bytes
  * @return Open writable fd on success, -1 on failure
  */
-int32_t create_orch_so_file(const char *dir, int32_t callable_id, char *out_path, size_t out_path_size);
+int32_t
+create_orch_so_file(const char *dir, int32_t callable_id, int32_t device_id, char *out_path, size_t out_path_size);
 
 #endif  // PLATFORM_AICPU_ORCH_SO_FILE_H_

src/a5/platform/include/aicpu/platform_regs.h

@@ -57,6 +57,17 @@ void set_platform_regs(uint64_t regs);
  */
 uint64_t get_platform_regs();
 
+/**
+ * Set the ACL device ordinal for the current run. Pushed by the platform layer
+ * (kernel.cpp) before aicpu_execute() from KernelArgs.device_id; the executor
+ * reads it to make the staged orchestration SO filename unique per device so
+ * paired dies sharing the preinstall filesystem never collide.
+ */
+void set_orch_device_id(int device_id);
+
+/** Get the ACL device ordinal set for the current run (0 if unset). */
+int get_orch_device_id();
+
 #ifdef __cplusplus
 }
 #endif

src/a5/platform/include/common/kernel_args.h

@@ -87,6 +87,11 @@ struct KernelArgs {
     // See the a2a3 kernel_args.h for the full design rationale (CANN's
     // AICPU args copy makes inline fields write-only).
     uint64_t device_wall_data_base{0};
+    // ACL device ordinal. Pushed to the AICPU so the executor can suffix the
+    // staged orchestration SO name (libdevice_orch_<pid>_<cid>_<device_id>.so),
+    // mirroring the per-device simpler_inner preinstall fix. Trailing field —
+    // keeps the CANN-fixed front offsets and AICore-read fields in place.
+    uint32_t device_id{0};
 };
 
 #ifdef __cplusplus

src/a5/platform/onboard/aicpu/kernel.cpp

@@ -100,6 +100,8 @@ extern "C" __attribute__((visibility("default"))) int simpler_aicpu_exec(void *a
     // mirrored into Handshake), so decode the umbrella bitmask once and
     // hand it to the existing platform-state setters.
     set_platform_regs(k_args->regs);
+    // Device ordinal for per-device orchestration-SO naming in the executor.
+    set_orch_device_id(static_cast<int>(k_args->device_id));
     set_platform_dump_base(k_args->dump_data_base);
     set_dump_tensor_enabled(GET_PROFILING_FLAG(k_args->enable_profiling_flag, PROFILING_FLAG_DUMP_TENSOR));
     set_platform_l2_perf_base(k_args->l2_perf_data_base);