From fb6eb0c235f322d2fffa0c01ed51b1dc39e1cd5d Mon Sep 17 00:00:00 2001 From: Chao Wang <26245345+ChaoWao@users.noreply.github.com> Date: Wed, 8 Jul 2026 00:35:45 -0700 Subject: [PATCH] Fix: ACK-gate L2 swimlane rotation after moving AICore FIN early Move the tensormap_and_ringbuffer AICore FIN write ahead of the swimlane record (order: ACK -> pmu_begin -> start -> execute -> FIN -> end -> pmu_end -> dump -> record) so completion is signalled before the ~0.5us record write-back, and gate the three swimlane timestamps on l2_swimlane_enabled. FIN now precedes the record's dcci+dsb, so the completion-before-dispatch invariant no longer proves the old buffer's tail record has drained at a BUFFER_SIZE rotation. AICPU therefore stops enqueueing the just-filled buffer at rotation: it stashes the buffer and releases it to the host only when AICore ACKs the new buffer's first task (that ACK is emitted after the previous task's record dcci+dsb, proving it drained). host_build_graph writes the record before FIN and has no such window; it shares the rotate but relies on the next-rotation / run-end backstop. - reorder + timestamp gating: a2a3 tmr aicore_executor.cpp - ACK-gate: shared l2_swimlane_collector_aicpu.{cpp,h}, new l2_swimlane_aicpu_on_aicore_ack wired into the a2a3 tmr scheduler_completion path; the deferred-enqueue stash is reset in l2_swimlane_aicpu_init so it cannot leak a dangling buffer across launches - on_aicore_dispatch gains the gate reg_task_id; all callers updated (a2a3 + a5, tmr scheduler_dispatch + hbg aicpu_executor) and the a5 aicpu collector header brought in line with the shared signature - docs/investigations: switch-overhead decomposition (the gap is mostly profiling self-cost - get_sys_cnt reads + record write-back) Verified: a2a3 L2 swimlane ST smoke + qwen3_14b_decode level 4 pass on silicon; a2a3sim + a5 sim/plain swimlane smokes pass; a2a3/a5 runtimes and sims rebuild clean. The boundary path (>1024 tasks/core) is not exercised by these workloads. --- ...e-swimlane-switch-overhead-and-ack-gate.md | 123 +++++++++++++++ docs/investigations/README.md | 1 + .../aicore/l2_swimlane_collector_aicore.h | 15 +- .../aicpu/l2_swimlane_collector_aicpu.h | 43 ++++-- .../host_build_graph/aicpu/aicpu_executor.cpp | 11 +- .../aicore/aicore_executor.cpp | 63 +++----- .../scheduler/scheduler_completion.cpp | 10 ++ .../runtime/scheduler/scheduler_dispatch.cpp | 11 +- .../aicpu/l2_swimlane_collector_aicpu.h | 43 ++++-- .../host_build_graph/aicpu/aicpu_executor.cpp | 6 +- .../runtime/scheduler/scheduler_dispatch.cpp | 6 +- .../aicpu/l2_swimlane_collector_aicpu.cpp | 142 +++++++++++++----- 12 files changed, 355 insertions(+), 119 deletions(-) create mode 100644 docs/investigations/2026-07-aicore-swimlane-switch-overhead-and-ack-gate.md diff --git a/docs/investigations/2026-07-aicore-swimlane-switch-overhead-and-ack-gate.md b/docs/investigations/2026-07-aicore-swimlane-switch-overhead-and-ack-gate.md new file mode 100644 index 0000000000..f7e2d639df --- /dev/null +++ b/docs/investigations/2026-07-aicore-swimlane-switch-overhead-and-ack-gate.md @@ -0,0 +1,123 @@ +# L2 swimlane AICore: switch-overhead source + FIN-early reorder & ACK-gate + +**Date**: 2026-07-08 +**Verdict**: measured — the ~0.8 µs inter-task "switch" is the per-task +record write-back (`dcci(record, CACHELINE_OUT) + dsb`) on AICore's own +timeline; it is **inherent, not reducible by reordering**. Two changes +shipped alongside the measurement: (1) sample the swimlane `end_time` +after an early FIN so the record write no longer sits on the +completion-signal path; (2) an AICPU-side **ACK-gate** on buffer +rotation, required because that early FIN otherwise races the record +flush at BUFFER_SIZE boundaries. + +## Question + +Switching between two back-to-back AICore tasks shows a ~0.7–1.0 µs gap +on the L2 swimlane in normal dispatch (not in speculative early +dispatch). Where does it come from, and can moving the FIN write earlier +(release the core to AICPU sooner) shrink it? + +## What was tried + +Ran `qwen3_14b_decode` (`StressBatch16Seq3500`, block_dim auto → 72 +cores, 1088 AICore tasks) with `--enable-l2-swimlane 4` on a2a3 silicon +via `task-submit`. Decomposed each core's inter-task gap from the +records (`aicore_tasks = [core, token, reg_task_id, start, end, +receive_to_start_cycles]`, 50 MHz → 20 ns/tick), classifying a gap as a +**switch** when the next task was already dispatched before the previous +ended (`dispatch(K+1) < end(K)`, i.e. work was waiting) vs a **WAIT** +gap (core idle, scheduler/dep-bound). Switches split at `receive_time` +(`= start − receive_to_start_cycles`): + +- `prev_tail = receive(K+1) − end(K)` — AICore's own post-task work: + `record_task` (the `dcci(record, OUT) + dsb`) + loop-around pickup. +- `setup = receive_to_start_cycles(K+1)` — payload `dcci(ENTIRE)` + ACK. + +## Result + +| metric (steady state) | mean | p50 | p90 | p99 | +| --------------------- | ---- | --- | --- | --- | +| STEADY switch (`disp < end`) | 0.81 | 0.80 | 1.00 | 1.22 | +| ↳ prev_tail (record + loop) | 0.52 | 0.50 | 0.66 | 0.88 | +| ↳ setup (payload dcci + ack) | 0.29 | 0.28 | 0.40 | 0.60 | +| first switch / core (cold) | 1.39 | 1.48 | 1.86 | 1.98 | +| WAIT gap (core idle) | 126 | 54 | 381 | 698 | + +(µs; n = 581 steady switches, 403 wait gaps, 32 cores.) + +The switch's dominant half is **prev_tail ≈ 0.5 µs = the record +write-back `dcci(record, CACHELINE_OUT) + dsb`**. `setup` (~0.28 µs) is +the payload invalidate + ACK. The **WAIT gap (p99 ~700 µs)** dwarfs the +whole switch — decode latency is a scheduler/dep-graph question, not a +DFX one. + +**The switch gap is largely profiling self-cost, not a real hardware +task switch.** With the swimlane on, every task issues three +`get_sys_cnt_aicore()` SPR reads (receive / start / end), each ~100–200 +ns, and one record write-back. The `receive` read lands entirely inside +the switch gap; the `end`/`start` reads contribute their boundary +fractions. So the ~0.8 µs decomposes roughly as **get_sys_cnt reads +(~0.2–0.4 µs) + record write-back `dcci+dsb` (~0.4–0.5 µs)** — i.e. the +gap is dominated by the instrumentation itself. This is an *observer +effect*: with the swimlane off none of these reads or the record write +happen, so the gap vanishes — which is exactly why the overhead "only +appears on the swimlane" and not in a non-profiled / early-dispatch run. +`prev_tail`'s "record + loop" therefore includes the `receive` read, and +`setup` includes part of the `start` read; neither is pure dcci/ack. + +**Moving FIN earlier does not shrink the AICore switch.** `record_task` +still runs between `end(K)` and `receive(K+1)`, so prev_tail is +unchanged. What the early FIN *does* change: AICPU observes completion +~0.5 µs sooner (before the record write), decoupling the record flush +from the completion signal. That is a pipelining nicety on the AICPU +side, not a switch-gap reduction. + +## What shipped + +1. **`end_time` after an early FIN** (`aicore_executor.cpp`, tmr): + op order is now `ACK → pmu_begin → start → execute → FIN → end → + pmu_end → dump → record`. FIN fires right after `execute_task`; + `record_task` runs last, off the completion path. The three swimlane + timestamps are gated on `l2_swimlane_enabled`. + +2. **AICPU ACK-gate on rotation** (`l2_swimlane_collector_aicpu.cpp`, + shared a2a3). Because tmr now writes FIN *before* the record, the + completion-before-dispatch invariant no longer proves the old + buffer's tail record has drained at a BUFFER_SIZE rotation — AICPU + could enqueue a buffer whose last record's `dcci+dsb` is still in + flight (host then discards it as `start_time == 0`). Fix: at + rotation, publish the new buffer to AICore but **stash** the + just-filled buffer; release it to the host only when AICPU observes + AICore ACK the new buffer's first task (`reg_task_id == gate`). By + AICore's single-threaded program order that ACK is emitted only + after the previous task's `record_task + dsb`, so the tail record is + proven drained. `host_build_graph` writes the record *before* FIN, + so it has no such window — it does not wire the ACK hook and relies + on the next-rotation / run-end backstop. + +## Why not chase the switch further + +The 0.5 µs is a cache-line clean-out + `dsb` on a DFX-only path; it is +sub-noise against the ~700 µs WAIT tail and self-correcting (the host +tolerates a dropped tail record). The per-task `dcci(head)` poll was +separately measured as effectively free, and deferring the record +`wmb`/`dsb` already measured as sub-noise +(`2026-06-l2-swimlane-defer-wmb.md`). The accuracy-over-performance rule +for profiling paths applies. + +## When to reconsider + +- If a workload runs **> 1024 AICore tasks on a single core** the + ACK-gate's boundary path becomes live (qwen decode's ~15 tasks/core + never crosses it, so the ST smoke and this run exercise only the + common path). Re-measure buffer-tail record loss there to confirm the + gate eliminates the `start_time == 0` discards. +- If the record write-back moves off HBM (e.g. an on-chip swimlane + staging buffer), the 0.5 µs prev_tail would shrink and the switch + decomposition should be re-run. + +## References + +- [2026-06 — L2 swimlane: defer per-task wmb to rotation](2026-06-l2-swimlane-defer-wmb.md) +- `.claude/rules/codestyle.md` (no logging on AICPU hot paths), + `docs/hardware/mmio-performance.md` (dcci/dsb costs). diff --git a/docs/investigations/README.md b/docs/investigations/README.md index 20eac6fb26..3c145c618b 100644 --- a/docs/investigations/README.md +++ b/docs/investigations/README.md @@ -84,6 +84,7 @@ that ...". Newest first. +- [2026-07 — L2 swimlane AICore: switch-overhead source + FIN-early reorder & ACK-gate](2026-07-aicore-swimlane-switch-overhead-and-ack-gate.md) — measured: the ~0.8 µs inter-task switch is the record write-back `dcci(record,OUT)+dsb` (~0.5 µs) + payload setup (~0.28 µs), inherent and not reducible by moving FIN; the WAIT gap (p99 ~700 µs) dominates decode. Shipped: sample `end_time` after an early FIN, and an AICPU ACK-gate on buffer rotation (release the old buffer only when AICore ACKs the new buffer's first task) to close the FIN-before-record boundary race the reorder introduced - [2026-07 — Removing PTO2LocalReadyBuffer exposed a missing dcci in EP dispatch](2026-07-local-buffer-removal-ep-combine-regression.md) — RESOLVED in #1245: local-buffer removal changed dispatch timing and unmasked a latent kernel bug (dispatch never dcci'd `recv_count_out` to HBM → local_expert read count=0 → all-zero output); fixed with a one-line dcci in the example kernel - [2026-06 — Gating the two residual profiling enable() calls on the orch/scheduler hot path](2026-06-orch-profiling-enable-gates-hot-path.md) — gated under existing `PTO2_PROFILING`; magnitude unmeasured, no new macro - [2026-06 — Replacing COND with GM+dcci for AICore→AICPU notification](2026-06-cond-vs-gm-notification.md) diff --git a/src/a2a3/platform/include/aicore/l2_swimlane_collector_aicore.h b/src/a2a3/platform/include/aicore/l2_swimlane_collector_aicore.h index 79c9252f03..59cc3ba94d 100644 --- a/src/a2a3/platform/include/aicore/l2_swimlane_collector_aicore.h +++ b/src/a2a3/platform/include/aicore/l2_swimlane_collector_aicore.h @@ -66,12 +66,15 @@ struct L2SwimlaneAicoreLocalState { * * Race avoidance: AICPU rotates strictly before `write_reg(DATA_MAIN_BASE)` * for the first task of a new BUFFER_SIZE batch — driven by AICPU's own - * per-core dispatch count (no AICore-side signal). The runtime's - * completion-before-dispatch invariant (AICore per core is single-threaded - * and AICPU does not dispatch task K+1 until K FIN'd) guarantees all prior - * tasks have FIN'd at rotation time, so AICore has already finished writing - * their records and dcci'd them out before AICPU enqueues the old buffer to - * the ready queue. + * per-core dispatch count (no AICore-side signal). At rotation AICPU only + * publishes the new buffer; it does NOT hand the old buffer to the host there. + * The completion-before-dispatch invariant proves all prior tasks FIN'd, but + * this runtime writes FIN before the record below, so a FIN-gated release could + * publish a buffer whose tail record's `dcci(record, CACHELINE_OUT) + dsb` has + * not landed. AICPU instead releases the old buffer once it observes AICore ACK + * the new buffer's first task (l2_swimlane_aicpu_on_aicore_ack): by this core's + * single-threaded program order that ACK is emitted only after the previous + * task's record dcci+dsb, so the old buffer's last record is guaranteed drained. * * @param head Per-core L2SwimlaneActiveHead channel. The executor * resolves it right after Phase 1 handshake exit diff --git a/src/a2a3/platform/include/aicpu/l2_swimlane_collector_aicpu.h b/src/a2a3/platform/include/aicpu/l2_swimlane_collector_aicpu.h index 8c8b56d75b..4564dedf12 100644 --- a/src/a2a3/platform/include/aicpu/l2_swimlane_collector_aicpu.h +++ b/src/a2a3/platform/include/aicpu/l2_swimlane_collector_aicpu.h @@ -85,24 +85,47 @@ void l2_swimlane_aicpu_init(int worker_count); * * 1. Maintain the per-core AICPU-side dispatch count. * 2. Rotate the AICore buffer when the count is about to cross a - * PLATFORM_AICORE_BUFFER_SIZE boundary — enqueue the just-filled buffer - * to the ready queue and pop the next one from free_queue. + * PLATFORM_AICORE_BUFFER_SIZE boundary — publish the next buffer from + * free_queue to AICore and STASH the just-filled buffer for deferred, + * ACK-gated release (see l2_swimlane_aicpu_on_aicore_ack). * 3. Bump the AICore pool's `total_record_count` so host reconcile * (total == collected + dropped) stays accurate at all levels — * including AICORE_TIMING (level=1), where `complete_task` is bypassed. * - * Race safety: rotation runs BEFORE the dispatch register write. The runtime's - * completion-before-dispatch invariant (AICore per core is single-threaded - * and AICPU does not dispatch task K+1 until K FIN'd) guarantees AICore has - * FIN'd — and dcci'd out — every record in the old buffer by then. No - * AICore-side signal is needed; AICPU has full dispatch visibility itself. + * Race safety: rotation runs BEFORE the dispatch register write. The + * completion-before-dispatch invariant proves prior tasks FIN'd, but + * tensormap_and_ringbuffer writes FIN before the swimlane record, so the old + * buffer's tail record may not have drained at rotation time. The old buffer is + * therefore not enqueued here; it is released once AICore ACKs the boundary + * dispatch (whose token is `reg_task_id`). host_build_graph writes the record + * before FIN so it has no such window and need not wire the ACK hook. * * No-op if l2_swimlane is disabled or `core_id` is out of range. * - * @param core_id Core index this dispatch targets - * @param thread_idx Owning AICPU thread (target ready-queue for rotation) + * @param core_id Core index this dispatch targets + * @param thread_idx Owning AICPU thread (target ready-queue for rotation) + * @param reg_task_id Per-core dispatch token of this dispatch; when it is the + * boundary dispatch it becomes the ACK gate for releasing + * the just-filled buffer */ -void l2_swimlane_aicpu_on_aicore_dispatch(int core_id, int thread_idx); +void l2_swimlane_aicpu_on_aicore_dispatch(int core_id, int thread_idx, uint32_t reg_task_id); + +/** + * Post-completion hook: release an ACK-gated AICore buffer. + * + * Called from the completion path on every matched AICore COND event (ACK or + * FIN). When a rotation has stashed the previous buffer with a gate equal to + * `reg_task_id`, that event proves AICore reached at least the new buffer's + * first-task ACK — hence passed the old buffer's last record dcci+dsb — so the + * stashed buffer is enqueued to the ready queue. No-op when nothing is stashed + * or the token does not match the gate. Only tensormap_and_ringbuffer wires + * this; host_build_graph relies on the next-rotation / run-end backstop. + * + * @param core_id Core index the COND event was observed on + * @param thread_idx Owning AICPU thread (target ready-queue) + * @param reg_task_id Task token extracted from the observed COND value + */ +void l2_swimlane_aicpu_on_aicore_ack(int core_id, int thread_idx, uint32_t reg_task_id); /** * Commit an AICPU-side timing record for one completed task. diff --git a/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp b/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp index 6ab91ecf6b..911346eca7 100644 --- a/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp +++ b/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp @@ -281,11 +281,14 @@ inline bool AicpuExecutor::try_dispatch_task( pending_task_ids_[core_id] = task_id; // AICore buffer rotation: count this dispatch and rotate before write_reg - // when crossing a BUFFER_SIZE boundary. The completion-before-dispatch - // invariant makes this race-free (all prior tasks on this core have FIN'd, - // so AICore has dcci'd their records out of the old buffer). + // when crossing a BUFFER_SIZE boundary. This runtime writes the swimlane + // record before FIN, so the completion-before-dispatch invariant already + // guarantees the old buffer's records are drained at rotation; the shared + // rotate still stashes the buffer for ACK-gated release, drained here by the + // next-rotation / run-end backstop (no ACK hook wired). `task_id` is passed + // as the (unused-for-hbg) gate token. if (l2_swimlane_enabled) { - l2_swimlane_aicpu_on_aicore_dispatch(core_id, thread_idx); + l2_swimlane_aicpu_on_aicore_dispatch(core_id, thread_idx, static_cast(task_id)); } // Publish task data before AICore can observe the dispatched task_id. diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp index 84ff7211e7..f9ec9c97c6 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp @@ -133,28 +133,20 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in } { - // receive_time marks the moment AICPU's full "task is ready to - // execute" signal landed on this core. Paired with start_time - // (captured after the per-task dcci + ack pair) it lets DFX split - // head_OH into the AICPU→AICore-ready propagation (dispatch_ts → - // receive_time, hardware + scheduling-bound) and the AICore-local - // critical-path prep (receive_time → start_time, software-tunable). - // Stored in the record as a 32-bit delta `start_time - receive_time`. + // receive_time = task pickup: DATA_MAIN_BASE returned a new task_id. + // Paired with start_time (captured after the per-task dcci + ack) it + // lets DFX split head_OH into the AICPU→AICore-ready propagation + // (dispatch_ts → receive_time) and the AICore-local prep + // (receive_time → start_time). Stored as a 32-bit delta + // `start_time - receive_time`. // - // For the common path (not_ready == 0) the new task_id on - // DATA_MAIN_BASE is itself the ready signal, so receive_time is - // stamped immediately and local_setup covers dcci + ack. - // - // For the speculative early-dispatch path (not_ready == 1) the - // dcci ran BEFORE the dependency-wait spin, so its cost is hidden - // behind the doorbell-wait — not on the critical path between - // "task genuinely ready" and "kernel begins". receive_time is - // re-stamped after the doorbell arrives, so propagation absorbs - // both the original NoC delivery AND any speculation overshoot, - // while local_setup stays the pure ack-on-critical-path cost. This - // makes local_setup the clean "AICore prep we can't hide" figure - // for both paths. - uint64_t receive_time = get_sys_cnt_aicore(); + // Common path (not_ready == 0): the new task_id is itself the ready + // signal, so receive_time is the true ready moment. Speculative path + // (not_ready == 1): receive_time stays at pickup — before the + // doorbell wait — so it precedes the producer's end_time; the host + // folds it to start_time for those tasks (detected when receive + // precedes the producer task's end_time). + uint64_t receive_time = l2_swimlane_enabled ? get_sys_cnt_aicore() : 0; uint32_t task_id = reg_val; // Decode: register holds task_id directly @@ -195,27 +187,24 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in write_reg(RegId::COND, AICORE_EXITED_VALUE); break; } - // Re-stamp receive_time at the moment the doorbell landed: the - // dcci above ran during the speculative-staging window - // (overlapped with the dependency wait, off the critical path). - // Propagation now absorbs the speculation overshoot; local_setup - // = start - receive stays the pure ack-on-critical-path cost. - receive_time = get_sys_cnt_aicore(); } write_reg(RegId::COND, MAKE_ACK_VALUE(task_id)); - // Performance profiling: record start time - uint64_t start_time = get_sys_cnt_aicore(); - - // PMU: start counting window around kernel execution + // PMU window brackets kernel execution. if (pmu_enabled) { pmu_aicore_begin(); } - // Execute the task + uint64_t start_time = l2_swimlane_enabled ? get_sys_cnt_aicore() : 0; + execute_task(exec_payload); + last_reg_val = reg_val; + write_reg(RegId::COND, MAKE_FIN_VALUE(task_id)); + + uint64_t end_time = l2_swimlane_enabled ? get_sys_cnt_aicore() : 0; + if (pmu_enabled) { pmu_aicore_end(); } @@ -224,7 +213,6 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in pipe_barrier(PIPE_ALL); } - // Performance profiling: record task execution. // Two identity fields go into the record (different roles): // - task_token_raw (PTO2 ring/local) is pulled from the dispatch // payload's LocalContext.async_ctx — already in AICore cache @@ -238,16 +226,7 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in // under SPMD (block_num > num_cores) and MIX cluster spread, // where multiple dispatches of the same task share the same // task_token_raw. - last_reg_val = reg_val; - write_reg(RegId::COND, MAKE_FIN_VALUE(task_id)); - - // Sample end_time AFTER the FIN write so the op-event end marks the - // moment the AICPU can first observe completion — any compute-end -> - // FIN gap (epilogue / write-back) shows directly on the bar instead - // of being inferred. The record write itself stays off the critical - // path (it runs after FIN, so it no longer delays completion). if (l2_swimlane_enabled) { - uint64_t end_time = get_sys_cnt_aicore(); uint64_t task_token_raw = exec_payload->local_context.async_ctx.task_token.raw; l2_swimlane_aicore_record_task( l2_swimlane_head, &l2_swimlane_local, task_token_raw, task_id, receive_time, start_time, end_time diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_completion.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_completion.cpp index 4f0f40ff21..44563c7461 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_completion.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_completion.cpp @@ -351,6 +351,16 @@ void SchedulerContext::check_running_cores_for_completion( ); if (!t.matched) continue; +#if PTO2_PROFILING + // Release an ACK-gated AICore swimlane buffer if this matched ACK/FIN is + // the one a rotation is waiting on: it proves the core advanced past the + // just-rotated buffer's tail record (FIN precedes the record write on + // this runtime, so rotation could not release the buffer itself). + if (l2_swimlane_level_ != L2SwimlaneLevel::DISABLED) { + l2_swimlane_aicpu_on_aicore_ack(core_id, thread_idx, static_cast(reg_task_id)); + } +#endif + #if PTO2_SCHED_PROFILING if (l2_swimlane.l2_swimlane_enabled && (t.running_done || t.pending_done)) { l2_swimlane.complete_hit_count++; diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp index 3bff0ffcf4..15ba623e90 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp @@ -189,13 +189,14 @@ SchedulerContext::PublishHandle SchedulerContext::prepare_subtask_to_core( // AICore buffer rotation lives on the dispatch path: count this dispatch // and rotate before write_reg when we're about to cross a BUFFER_SIZE - // boundary. The completion-before-dispatch invariant makes this race-free - // (all prior tasks on this core have FIN'd, so AICore has dcci'd their - // records out of the old buffer). Gated on the same enable bit as flush - // so level=1 (AICORE_TIMING-only) participates without needing complete_task. + // boundary. The just-filled buffer is not handed to the host here — it is + // released once AICore ACKs this boundary dispatch (see the ACK hook in + // check_running_cores_for_completion), because FIN precedes the swimlane + // record on this runtime. `reg_task_id` is passed as that ACK gate. Gated on + // the same enable bit as flush so level=1 (AICORE_TIMING-only) participates. #if PTO2_PROFILING if (l2_swimlane_level_ != L2SwimlaneLevel::DISABLED) { - l2_swimlane_aicpu_on_aicore_dispatch(core_id, thread_idx); + l2_swimlane_aicpu_on_aicore_dispatch(core_id, thread_idx, reg_task_id); } #endif diff --git a/src/a5/platform/include/aicpu/l2_swimlane_collector_aicpu.h b/src/a5/platform/include/aicpu/l2_swimlane_collector_aicpu.h index 8c8b56d75b..79fdeae39f 100644 --- a/src/a5/platform/include/aicpu/l2_swimlane_collector_aicpu.h +++ b/src/a5/platform/include/aicpu/l2_swimlane_collector_aicpu.h @@ -85,24 +85,47 @@ void l2_swimlane_aicpu_init(int worker_count); * * 1. Maintain the per-core AICPU-side dispatch count. * 2. Rotate the AICore buffer when the count is about to cross a - * PLATFORM_AICORE_BUFFER_SIZE boundary — enqueue the just-filled buffer - * to the ready queue and pop the next one from free_queue. + * PLATFORM_AICORE_BUFFER_SIZE boundary — publish the next buffer from + * free_queue to AICore and STASH the just-filled buffer for deferred, + * ACK-gated release (see l2_swimlane_aicpu_on_aicore_ack). * 3. Bump the AICore pool's `total_record_count` so host reconcile * (total == collected + dropped) stays accurate at all levels — * including AICORE_TIMING (level=1), where `complete_task` is bypassed. * - * Race safety: rotation runs BEFORE the dispatch register write. The runtime's - * completion-before-dispatch invariant (AICore per core is single-threaded - * and AICPU does not dispatch task K+1 until K FIN'd) guarantees AICore has - * FIN'd — and dcci'd out — every record in the old buffer by then. No - * AICore-side signal is needed; AICPU has full dispatch visibility itself. + * Race safety: rotation runs BEFORE the dispatch register write. The + * completion-before-dispatch invariant proves prior tasks FIN'd, but + * tensormap_and_ringbuffer may write FIN before the swimlane record, so the old + * buffer's tail record may not have drained at rotation time. The old buffer is + * therefore not enqueued here; it is released once AICore ACKs the boundary + * dispatch (whose token is `reg_task_id`) or, when a runtime does not wire the + * ACK hook, by the next-rotation / run-end backstop. * * No-op if l2_swimlane is disabled or `core_id` is out of range. * - * @param core_id Core index this dispatch targets - * @param thread_idx Owning AICPU thread (target ready-queue for rotation) + * @param core_id Core index this dispatch targets + * @param thread_idx Owning AICPU thread (target ready-queue for rotation) + * @param reg_task_id Per-core dispatch token of this dispatch; when it is the + * boundary dispatch it becomes the ACK gate for releasing + * the just-filled buffer */ -void l2_swimlane_aicpu_on_aicore_dispatch(int core_id, int thread_idx); +void l2_swimlane_aicpu_on_aicore_dispatch(int core_id, int thread_idx, uint32_t reg_task_id); + +/** + * Post-completion hook: release an ACK-gated AICore buffer. + * + * Called from the completion path on every matched AICore COND event (ACK or + * FIN). When a rotation has stashed the previous buffer with a gate equal to + * `reg_task_id`, that event proves AICore reached at least the new buffer's + * first-task ACK — hence passed the old buffer's last record dcci+dsb — so the + * stashed buffer is enqueued to the ready queue. No-op when nothing is stashed + * or the token does not match the gate. Runtimes that write the record before + * FIN need not wire this; the next-rotation / run-end backstop covers them. + * + * @param core_id Core index the COND event was observed on + * @param thread_idx Owning AICPU thread (target ready-queue) + * @param reg_task_id Task token extracted from the observed COND value + */ +void l2_swimlane_aicpu_on_aicore_ack(int core_id, int thread_idx, uint32_t reg_task_id); /** * Commit an AICPU-side timing record for one completed task. diff --git a/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp b/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp index 8a36db238e..42560ab6bf 100644 --- a/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp +++ b/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp @@ -285,9 +285,11 @@ inline bool AicpuExecutor::try_dispatch_task( pending_task_ids_[core_id] = task_id; // AICore buffer rotation: count this dispatch and rotate before write_reg - // when crossing a BUFFER_SIZE boundary. + // when crossing a BUFFER_SIZE boundary. The just-filled buffer is stashed + // for ACK-gated release and drained via the next-rotation / run-end backstop + // (no ACK hook wired). `task_id` is passed as the gate token. if (l2_swimlane_enabled) { - l2_swimlane_aicpu_on_aicore_dispatch(core_id, thread_idx); + l2_swimlane_aicpu_on_aicore_dispatch(core_id, thread_idx, static_cast(task_id)); } // Publish task data before AICore can observe the dispatched task_id. diff --git a/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp b/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp index 1b6ad25d33..f1acafef85 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp +++ b/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp @@ -172,10 +172,12 @@ void SchedulerContext::dispatch_subtask_to_core( // AICore buffer rotation lives on the dispatch path: count this dispatch // and rotate before write_reg when we're about to cross a BUFFER_SIZE - // boundary. The completion-before-dispatch invariant makes this race-free. + // boundary. The just-filled buffer is stashed for ACK-gated release; a5 does + // not wire the ACK hook, so it drains via the next-rotation / run-end + // backstop. `reg_task_id` is passed as the gate token. #if PTO2_PROFILING if (l2_swimlane_level_ != L2SwimlaneLevel::DISABLED) { - l2_swimlane_aicpu_on_aicore_dispatch(core_id, thread_idx); + l2_swimlane_aicpu_on_aicore_dispatch(core_id, thread_idx, reg_task_id); } #endif diff --git a/src/common/platform/shared/aicpu/l2_swimlane_collector_aicpu.cpp b/src/common/platform/shared/aicpu/l2_swimlane_collector_aicpu.cpp index aad43cd5cf..83c19335bf 100644 --- a/src/common/platform/shared/aicpu/l2_swimlane_collector_aicpu.cpp +++ b/src/common/platform/shared/aicpu/l2_swimlane_collector_aicpu.cpp @@ -60,6 +60,24 @@ static L2SwimlaneAicoreTaskPool *s_aicore_task_pools[PLATFORM_MAX_CORES] = {}; // Single-writer per cell (the scheduler thread that owns the core). static uint32_t s_aicore_dispatched_count[PLATFORM_MAX_CORES] = {}; +// Per-core deferred AICore old-buffer enqueue (ACK-gate for the FIN-before-record +// window). The tensormap_and_ringbuffer AICore executor writes FIN before the +// swimlane record, so releasing a just-filled buffer on the FIN that gated the +// boundary dispatch could publish a buffer whose tail record has not drained +// yet. Instead the rotation stashes the old buffer here and it is released only +// once AICore ACKs the first task of the NEW buffer (reg_task_id == gate): by +// AICore's single-threaded program order that ACK proves the old buffer's last +// record dcci+dsb has landed. host_build_graph writes the record before FIN, so +// it has no such window — it never wires the ACK hook and its stashed buffer is +// released by the next-rotation / run-end backstop instead. Single-writer per +// cell (the scheduler thread that owns the core). +struct AicorePendingEnqueue { + uint64_t buf_ptr = 0; // 0 == nothing pending + uint32_t buf_seq = 0; + uint32_t gate_reg_task_id = 0; +}; +static AicorePendingEnqueue s_aicore_pending_enqueue[PLATFORM_MAX_CORES] = {}; + // Per-core cached current-records-buffer pointer. Written by AICPU when // rotating buffers from inside `complete_record`. AICore writes to its own // per-core L2SwimlaneAicoreTaskBuffer (host-allocated, AICPU rotates) and AICPU @@ -232,9 +250,14 @@ void l2_swimlane_aicpu_init(int worker_count) { // launch must start counting from 0 so the rotation boundary check // (count % BUFFER_SIZE == 0) lands on the right dispatches. Stale values // from a prior launch would skip the first rotation (count already past a - // boundary) or trigger one prematurely. + // boundary) or trigger one prematurely. The deferred-enqueue stash must be + // cleared too: a prior launch that ended with a buffer still stashed (gate + // ACK never observed, or run-end flush hit a full queue) would otherwise + // leave a dangling buf_ptr that a matching reg_task_id in this launch could + // flush — freeing a buffer that no longer belongs to us. for (int i = 0; i < PLATFORM_MAX_CORES; i++) { s_aicore_dispatched_count[i] = 0; + s_aicore_pending_enqueue[i] = AicorePendingEnqueue{}; } void *l2_swimlane_base = reinterpret_cast(g_platform_l2_swimlane_base); @@ -362,13 +385,42 @@ static void switch_records_buffer(int core_id, int thread_idx) { ); } -// Try to rotate the AICore buffer for `core_id`. Called from the completion -// path after a successful L2SwimlaneAicpuTaskRecord commit so the just-FIN'd task's -// AICore record is guaranteed to be in the old buffer before we enqueue it. -// On success bumps `ac_state->head.current_buf_seq`; on failure (empty free queue -// or full ready queue) the old buffer is abandoned in place, AICore overflows -// it from now on, and the drop count grows. -static void aicore_rotate(int core_id, int thread_idx) { +// Release a deferred AICore buffer (stashed by aicore_rotate) to the ready +// queue. Shared by the ACK hook (prompt release), the next-rotation backstop, +// and run-end flush. No-op when nothing is pending. On ready-queue-full the +// buffer stays stashed for a later retry; the same no-double-count rationale as +// the rotation queue-full branch applies (reconcile reports silent_loss). +static void flush_aicore_pending_enqueue(int core_id, int thread_idx) { + AicorePendingEnqueue &pe = s_aicore_pending_enqueue[core_id]; + if (pe.buf_ptr == 0) { + return; + } + L2SwimlaneAicoreTaskBuffer *old_buf = reinterpret_cast(pe.buf_ptr); + old_buf->count = static_cast(PLATFORM_AICORE_BUFFER_SIZE); + wmb(); + int rc = enqueue_ready_buffer( + s_l2_swimlane_header, thread_idx, core_id, pe.buf_ptr, pe.buf_seq, L2SwimlaneBufferKind::AicoreTask + ); + if (rc != 0) { + LOG_ERROR( + "Thread %d: Core %d failed to enqueue deferred AICore buffer (queue full); will retry at flush", thread_idx, + core_id + ); + return; + } + pe.buf_ptr = 0; +} + +// Rotate the AICore buffer for `core_id` at a PLATFORM_AICORE_BUFFER_SIZE +// dispatch boundary: pop a fresh buffer from free_queue, publish it via the head +// channel, and DEFER the just-filled buffer's enqueue via +// s_aicore_pending_enqueue (see that declaration for the FIN-before-record +// rationale). `new_buf_first_reg_task_id` is the reg_task_id of the boundary +// dispatch — the first task written into the new buffer — used as the ACK gate +// that later releases the stashed buffer. On success bumps +// `ac_state->head.current_buf_seq`; on empty free queue the old buffer is kept +// active in place (AICore overflows it, drop count grows). +static void aicore_rotate(int core_id, int thread_idx, uint32_t new_buf_first_reg_task_id) { L2SwimlaneAicoreTaskPool *ac_state = s_aicore_task_pools[core_id]; if (ac_state == nullptr) { return; @@ -410,30 +462,19 @@ static void aicore_rotate(int core_id, int thread_idx) { return; } - // Enqueue the just-filled AICore buffer with count = BUFFER_SIZE. + // Defer the just-filled buffer's enqueue behind the ACK gate instead of + // handing it to the host now (see s_aicore_pending_enqueue). First drain any + // buffer still stashed from a prior rotation whose gate ACK was never + // observed: its gate task completed a full BUFFER_SIZE batch ago, so it is + // long safe to release. If that drain fails (ready queue full) the older + // buffer is overwritten below and shows up as silent_loss at reconcile — + // the same lossy regime the pre-defer code hit on a queue-full enqueue. if (old_buf_ptr != 0) { - L2SwimlaneAicoreTaskBuffer *old_buf = reinterpret_cast(old_buf_ptr); - old_buf->count = static_cast(PLATFORM_AICORE_BUFFER_SIZE); - wmb(); - int rc = enqueue_ready_buffer( - s_l2_swimlane_header, thread_idx, core_id, old_buf_ptr, seq, L2SwimlaneBufferKind::AicoreTask - ); - if (rc != 0) { - // Ready queue full — we leave current_buf_ptr pointing at the - // old buffer so the run-end flush path retries the enqueue (the - // host is draining concurrently; the queue may have space by - // then). We deliberately do NOT bump dropped here for the same - // reason as the empty-free-queue branch: counting a drop now - // would double-count if the flush succeeds in delivering the - // buffer to the host. Reconcile reports the actual loss as - // silent_loss when neither this rotation nor the flush - // delivers the records. - LOG_ERROR( - "Thread %d: Core %d failed to enqueue AICore buffer at rotation (queue full); will retry at flush", - thread_idx, core_id - ); - return; - } + flush_aicore_pending_enqueue(core_id, thread_idx); + AicorePendingEnqueue &pe = s_aicore_pending_enqueue[core_id]; + pe.buf_ptr = old_buf_ptr; + pe.buf_seq = seq; + pe.gate_reg_task_id = new_buf_first_reg_task_id; } // Pop next buffer from free_queue and publish via the head channel. @@ -458,17 +499,20 @@ static void aicore_rotate(int core_id, int thread_idx) { // per-core dispatch count and rotates the AICore buffer when the count is // about to cross a PLATFORM_AICORE_BUFFER_SIZE boundary. // -// Race safety: rotation runs before the dispatch register write. The -// completion-before-dispatch invariant (AICore per core is single-threaded -// and AICPU does not dispatch task K+1 until K FIN'd) guarantees AICore has -// already finished writing — and dcci'd out — every record in the old buffer -// by then. AICPU can safely enqueue the old buffer to the ready queue. +// Race safety: rotation runs before the dispatch register write and publishes +// the new buffer to AICore, but does NOT hand the old buffer to the host here. +// The old buffer is stashed and released only once AICore ACKs the first task +// of the new buffer (l2_swimlane_aicpu_on_aicore_ack) — the completion-before- +// dispatch invariant proves prior tasks FIN'd, but tensormap_and_ringbuffer +// writes FIN before the swimlane record, so FIN alone does not prove the tail +// record has drained. `reg_task_id` is that boundary dispatch's token, stored +// as the ACK gate. // // total_record_count accounting also lives here: one AICore record == one // dispatch, so the dispatch count IS the AICore-side total. Bumping here // (instead of inside complete_task) means level=1 (AICORE_TIMING-only) gets // accurate reconcile counts even when complete_task is bypassed. -void l2_swimlane_aicpu_on_aicore_dispatch(int core_id, int thread_idx) { +void l2_swimlane_aicpu_on_aicore_dispatch(int core_id, int thread_idx, uint32_t reg_task_id) { if (!g_enable_l2_swimlane) { return; } @@ -484,12 +528,28 @@ void l2_swimlane_aicpu_on_aicore_dispatch(int core_id, int thread_idx) { // batch (prev = BUFFER_SIZE, 2*BUFFER_SIZE, ...). PLATFORM_AICORE_BUFFER_SIZE // is asserted power-of-two so the mod lowers to a bitwise AND. if (prev > 0 && (prev & (PLATFORM_AICORE_BUFFER_SIZE - 1)) == 0) { - aicore_rotate(core_id, thread_idx); + aicore_rotate(core_id, thread_idx, reg_task_id); } s_aicore_dispatched_count[core_id] = prev + 1; ac_state->head.total_record_count += 1; } +void l2_swimlane_aicpu_on_aicore_ack(int core_id, int thread_idx, uint32_t reg_task_id) { + if (core_id < 0 || core_id >= PLATFORM_MAX_CORES) { + return; + } + AicorePendingEnqueue &pe = s_aicore_pending_enqueue[core_id]; + // Fast path: nothing stashed, or this is not the gating ACK/FIN. gate_reg_task_id + // is the boundary dispatch's token; the first COND event carrying it means + // AICore reached at least ACK of the new buffer's first task, hence passed + // record_task + dsb of the old buffer's last record. reg_task_id is unique + // per BUFFER_SIZE window, so this cannot fire early. + if (pe.buf_ptr == 0 || pe.gate_reg_task_id != reg_task_id) { + return; + } + flush_aicore_pending_enqueue(core_id, thread_idx); +} + int l2_swimlane_aicpu_complete_task( int core_id, int thread_idx, uint32_t reg_task_id, uint64_t dispatch_time, uint64_t finish_time ) { @@ -616,6 +676,12 @@ void l2_swimlane_aicpu_flush(int thread_idx, const int *cur_thread_cores, int co L2SwimlaneAicoreTaskPool *ac_state = s_aicore_task_pools[core_id]; if (ac_state == nullptr) continue; + // Drain any ACK-gated buffer still stashed from a rotation whose gate + // ACK was never observed before teardown. By run end every AICore task + // has completed, so the buffer's records are fully drained. Enqueue it + // before the current buffer so ready-queue order follows buffer_seq. + flush_aicore_pending_enqueue(core_id, thread_idx); + rmb(); uint64_t ac_buf_ptr = ac_state->head.current_buf_ptr; if (ac_buf_ptr == 0) continue;