Skip to content

test(a5): classify the deadlock head without racing the publication window - #1914

Merged
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
ChaoWao:fix-preemption-sensitive-deadlock-classification-uts
Aug 20, 2026
Merged

ChaoWao merged 1 commit into
hw-native-sys:mainfrom
ChaoWao:fix-preemption-sensitive-deadlock-classification-uts

Conversation

@ChaoWao

@ChaoWao ChaoWao commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Why

Two cases in tests/ut/cpp/a5/test_orchestrator_fanin.cpp asserted a classification that is only reachable inside a timing window, and raced a thread to get there:

std::thread scheduler = service_reclaim_publication_once(PTO2_MAX_RING_DEPTH - 1);
testing::internal::CaptureStderr();
TaskOutputTensors child = orch.submit_dummy_task(child_args);   // blocks in alloc()
...
EXPECT_NE(log.find("oldest task owned by an open scope on this ring"), std::string::npos);

A5 reports a provable head-of-line deadlock only with an acknowledged exact watermark. The reclaim spin asks for one after 10 ms without progress and abandons at its 500 ms backstop, so the assertion held only if the servicing thread was scheduled inside that ~490 ms window. On a 3-core macos-latest runner under ctest -j it is not: the allocator falls to the timeout classification — exactly what it is specified to do without an acknowledgment — and the message assertion fails while orch_error_code in the same case still reads PTO2_ERROR_HEAP_RING_DEADLOCK.

Observed on an unrelated PR (red, then green on re-run of the same commit).

What changes

Neither case is about the handshake — both are about which classification a head gets once the watermark is exact. ReclaimPublicationRequest already answers that an unwired request counts as synchronized: with no publisher to ask, the watermark in shared memory is the only one there is. So detaching the orchestrator's reclaim consumers from the scheduler's masks arms the structural check at the spin's first classification pass:

void unwire_reclaim_publication() {
    for (int r = 0; r < PTO2_MAX_RING_DEPTH; r++) {
        orch.rings[r].task_allocator.set_reclaim_publication_request(nullptr, nullptr);
        orch.rings[r].fanin_pool.set_reclaim_publication_request(nullptr, nullptr, static_cast<uint8_t>(r));
    }
}

No thread, no window, no preemption sensitivity. This is also the shape the a2a3 siblings of both cases already have — that allocator has no batched publication to wire, which is why only a5 ever flaked here. The change restores parity rather than inventing a test idiom.

What still covers the wired path

ClosedChildHeadUsesTimeoutWithOpenParentOnSharedRing, untouched: it stays wired, is never serviced, and asserts that the structural claim is withheld and the report falls to the timeout. Preemption cannot invalidate it — the deciding condition is an absolute-time comparison that becomes true regardless of scheduling, and no other exit can reach it first. It costs 500 ms, which is what asserting the absence of the other exit costs.

Testing

  • test_a5_orchestrator_fanin — 9/9. Both cases drop from a 10–526 ms race to 22 ms, which is fixture setup; only the timeout case still spends 500 ms
  • ctest -LE requires_hardware — 107/107
  • Negative control — making reclaim_head_matches_open_task always return false reddens exactly the two cases, each after falling through to the 500 ms timeout, and leaves ClosedChildHeadUsesTimeoutWithOpenParentOnSharedRing green, since it wants that branch

Relation to #1913

Same defect class — a unit test whose verdict depends on thread scheduling rather than on the code being correct — found in the same triage, fixed by the same move: assert the correctness property, not the duration. Independent commit on main; the two do not overlap in files and can land in either order.

@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@ChaoWao, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 46 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: af4d3175-750c-467e-b2b6-9a6d5c553af1

📥 Commits

Reviewing files that changed from the base of the PR and between 93adc38 and 6e4727a.

📒 Files selected for processing (1)
  • tests/ut/cpp/a5/test_orchestrator_fanin.cpp

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…indow

`SubmitPathHeapDeadlockLogReportsRingAndRealHeapState` and
`StructuralCheckRejectsOpenAncestorWhenNestedScopesShareRing` spawned a thread
that waited for `publication_request_mask` and drained it once, then asserted
that the blocked allocator reported a *provable* head-of-line deadlock. A5
reaches that classification only with an acknowledged exact watermark, which
the reclaim spin asks for after 10 ms without progress and abandons at its
500 ms backstop. So both cases asserted that the servicing thread got scheduled
inside a 490 ms window. On a loaded `macos-latest` runner it does not: the
allocator then falls to the timeout classification — which is what it is
specified to do without an acknowledgment — and the message assertion fails
while `orch_error_code` in the same case still reads
`PTO2_ERROR_HEAP_RING_DEADLOCK`.

Neither case is about the handshake. Both are about which classification a head
gets once the watermark is exact, and `ReclaimPublicationRequest` already
treats an unwired request as synchronized: with no publisher to ask, the
watermark in shared memory is the only one there is. Detaching the two
orchestrators from the scheduler's masks therefore arms the structural check at
the spin's first classification pass, with no thread and no window — the shape
the a2a3 siblings of both cases already have, since that allocator has no
batched publication to wire.

The wired-but-unacknowledged path keeps its own case:
`ClosedChildHeadUsesTimeoutWithOpenParentOnSharedRing` asserts that without an
acknowledgment the structural claim is withheld and the report falls to the
timeout. Preemption cannot invalidate it — the deciding condition is an
absolute-time comparison that becomes true regardless of scheduling, and no
other exit can reach it first. It costs 500 ms, which is what asserting the
absence of the other exit costs.

Verification:

- `test_a5_orchestrator_fanin` — 9/9. Both cases drop from a 10-526 ms race to
  22 ms, which is fixture setup; only the timeout case still spends 500 ms.
- `ctest -LE requires_hardware` — 107/107.
- Negative control: making `reclaim_head_matches_open_task` always return false
  reddens exactly the two cases, each after falling through to the 500 ms
  timeout, and leaves `ClosedChildHeadUsesTimeoutWithOpenParentOnSharedRing`
  green — it wants that branch.
@ChaoWao
ChaoWao merged commit 72a5fa7 into hw-native-sys:main Aug 20, 2026
16 checks passed
@ChaoWao
ChaoWao deleted the fix-preemption-sensitive-deadlock-classification-uts branch August 20, 2026 08:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant