[sim] dcci lacks cache-line atomicity in simulation mode — handshake store reordering can cause deadlock #266

Problem

On hardware, dcci(addr, SINGLE_CACHE_LINE, CACHELINE_OUT) atomically flushes an entire 64-byte cache line to HBM. Other processors (AICPU) reading from HBM see a complete cache-line snapshot — no partial visibility.

In simulation mode, dcci is implemented as std::atomic_thread_fence(std::memory_order_seq_cst) (dmb ish on aarch64). This provides fence semantics but no cache-line atomicity. Individual stores pass through the store buffer independently and can be reordered by the aarch64 weak memory model.

Reproduced Failures

multi-round-paged-attention and batch_paged_attention hung on a2a3sim + aarch64. Root cause: during the AICore-AICPU handshake, physical_core_id/core_type stores were reordered after the aicore_done signal store. AICPU observed aicore_done != 0 but read a stale physical_core_id (0), causing two cores to share the same register block — deadlock.

Core  0: AIC, physical_id=0, reg_addr=0xaaab24c20d40  <- correct
Core 27: AIV, physical_id=0, reg_addr=0xaaab24c20d40  <- wrong, should be 27

Not affected: hardware (a2a3) — dcci atomically flushes the cache line; x86 simulation — TSO memory model prevents store-store reordering.

Fix Status

Writer side (AICore) — applied to HBG and TMR

Insert dcci barrier between data fields and signal field:

my_hank->physical_core_id = get_physical_core_id();
my_hank->core_type = core_type;
dcci(my_hank, SINGLE_CACHE_LINE, CACHELINE_OUT);  // <- added: ensure data visible first
my_hank->aicore_done = block_idx + 1;
dcci(my_hank, SINGLE_CACHE_LINE, CACHELINE_OUT);

Modified files:

• src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp
• src/a2a3/runtime/host_build_graph/aicore/aicore_executor.cpp

Reader side (AICPU) — applied to HBG and ABG, missing from TMR

Insert __sync_synchronize() after aicore_done spin-wait exit:

while (hank->aicore_done == 0) {}
__sync_synchronize();  // <- HBG/ABG: applied, TMR: missing
CoreType type = hank->core_type;
uint32_t physical_core_id = hank->physical_core_id;

Applied:

• src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp
• src/a2a3/runtime/aicpu_build_graph/aicpu/aicpu_executor.cpp

Missing:

• src/a2a3/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp

Note on TMR Reader Side

The missing __sync_synchronize() in TMR is unlikely to cause issues in practice:

1. The writer-side dcci (dmb ish) already guarantees global store ordering
2. All fields reside in the same 64-byte aligned Handshake struct (single cache line) — when the reader fetches aicore_done, the same cache-line fetch also brings the updated physical_core_id

However, adding it would ensure formal correctness and consistency across all three runtimes.

Root Cause

The simulation dcci only provides fence semantics without modeling the hardware's cache-line atomic flush. For multi-field release-acquire protocols within the same cache line, developers must manually insert dcci/fence at the correct position, whereas the hardware's cache-line atomicity provides this guarantee implicitly.