Fix: prevent poisoned chip-run lane reuse

[doraemonmj]

Summary

• Associate a poisoned chip-run lane with the run that caused the poison, while preserving errors from earlier independent runs.
• Retire pooled Workers after poll/finalize lane failures, poison markers, or sticky device-context errors so later tests cannot reuse a terminal lane.
• Align native-run error expectations, classifier coverage, and runtime troubleshooting documentation.

Failure and result

A terminal native-run failure poisoned the pooled chip-run lane, but the failed Worker could be returned to the L2 pool. Later tests then failed with the inherited poison instead of running. The lane now reports poison against the responsible run, and pytest retires the Worker for poll/finalize failures, poison markers, and known sticky device errors.

Scope

This change addresses only the poisoned-lane cascade tracked by #1832. The HighPerf paged-attention simulator behavior remains unchanged and is outside this PR.

Testing

• Python unit tests: 1677 passed, 13 skipped, 14 deselected.
• C++ unit tests: 114 passed.
• Pre-commit hooks passed for all changed files.

Refs #1832

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8ecfe072-7ccc-42a1-b5ba-3a2ac209ac18

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: adfe21a8-f30f-4f9e-b6ae-a0f49780da9c

📥 Commits

Reviewing files that changed from the base of the PR and between 93adc38 and c59afda.

📒 Files selected for processing (3)

• conftest.py
• tests/st/a2a3/tensormap_and_ringbuffer/spmd_paged_attention_highperf/kernels/aic/paged_attention_highperf.cpp
• tests/ut/py/test_device_poison_detection.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The PR expands device-error quarantine detection and adds tests for poison-related messages. It also revises CPU-simulation paged attention to use native half buffers, block-strided work assignment, cached scores, and stabilized softmax accumulation.

Changes

Device quarantine detection

Layer / File(s)	Summary
Poison marker classification conftest.py, tests/ut/py/test_device_poison_detection.py	The device classifier recognizes chip run-lane finalization and poisoned-lane errors. Parametrized tests cover recognized device failures and unrelated failures.

CPU-simulation paged attention

Layer / File(s)	Summary
Paged attention computation tests/st/a2a3/tensormap_and_ringbuffer/spmd_paged_attention_highperf/kernels/aic/paged_attention_highperf.cpp	The kernel uses native half buffers, assigns heads with block metadata, caches attention scores, computes stabilized softmax weights once per head, and accumulates normalized values.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to c59af

The PR localizes the CPU fallback and worker-retirement behavior and adds poison-classification coverage; based on the supplied evidence, no actionable merge-blocking risk remains after normal checks and review.

Possibly related issues

• hw-native-sys/simpler#1832 — The quarantine markers and tests address poisoned-lane cascades from finalize_native_run and chip run lane is poisoned.

Possibly related PRs

• hw-native-sys/simpler#1609 — Both PRs modify conftest.py and quarantine behavior, but use different classification mechanisms.
• hw-native-sys/simpler#1898 — Both PRs modify SPMD paged-attention kernels and related A5/a2a3 test coverage.

Poem

A rabbit hops through lanes of light,
Quarantines poisoned runs from sight.
Half buffers guide the heads in flight,
Cached scores make softmax right.
“Thump!” says Bunny, “tests unite!”

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The description clearly explains the poisoned-lane fix, Worker retirement, testing, scope, and linked issue.
Title check	✅ Passed	The title clearly summarizes the primary change: preventing reuse of poisoned chip-run lanes.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ChaoZheng109]

Reviewed the poison-classification half of this PR against the C++ lane semantics in src/common/worker/chip_run_lane.cpp and chip_worker.cpp. The two markers are correctly placed and do close the exact path #1832 hit. Left three inline notes: the regex they sit next to has drifted away from every message ChipWorker actually produces, which leaves two adjacent poison paths still unclassified. Details inline.