Skip to content

Fix first-kernel-launch crash on Windows (ROCm/Adrenalin WDDM startup race) - #873

Closed
unimo-chan wants to merge 1 commit into
Niko1221:mainfrom
unimo-chan:fix-windows-wddm-crash
Closed

unimo-chan wants to merge 1 commit into
Niko1221:mainfrom
unimo-chan:fix-windows-wddm-crash

Conversation

@unimo-chan

Copy link
Copy Markdown
Contributor

Problem
On Windows with an AMD card (gfx1100, Adrenalin driver), the ROCm runtime can race the WDDM driver at startup. When it loses the race, hipMemGetInfo fails and the engine crashed at the first kernel launch (0xC0000005 in amdhip64_7.dll).

Fix

  • device.cu: mem_get_info() now retries hipMemGetInfo a few times (giving the driver time to settle); if it still fails it returns the error cleanly instead of crashing. The Windows/HIP path in device_info() uses this retrying mem_get_info instead of cudaMemGetInfo.
  • device_main.cpp: device init + selftest are wrapped in a retry loop (5 attempts) so a transient startup failure recovers.

Verified
selftest 5/5, server + chat OK, no first-kernel-launch crash on a RX 7900 XT / Adrenalin 32.0.32015.

… race)

On Windows with an AMD card (gfx1100, Adrenalin driver), the ROCm runtime can
race the WDDM driver at startup. When it loses the race, hipMemGetInfo fails
and the engine crashed at the first kernel launch (0xC0000005 in amdhip64_7.dll).

- device.cu: mem_get_info() now retries hipMemGetInfo a few times (giving the
  driver time to settle); if it still fails it returns the error cleanly instead
  of crashing. The Windows/HIP path in device_info() uses this retrying
  mem_get_info instead of cudaMemGetInfo.
- device_main.cpp: device init + selftest are wrapped in a retry loop (5
  attempts) so a transient startup failure recovers.

Verified: selftest 5/5, server + chat OK, no first-kernel-launch crash on a
RX 7900 XT / Adrenalin 32.0.32015.
Niko1221 added a commit that referenced this pull request Oct 6, 2026
…try per process, before the DXGI budget (#873, #654)

Reworked from #873: no 8 x 250 ms retry at every failing call site, no #if in device_info (cudaMemGetInfo already maps to this function), no retry of real failures (only hipErrorInvalidValue, once), before/after lines in the log. The strata-device 5x retry loop is dropped.

Co-authored-by: unimo-chan <yugo.miyachi@gmail.com>

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Niko1221

Niko1221 commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Taken for 0.1.40 in a reduced form: one hipFree(0) warm-up and one hipMemGetInfo retry per process, only for hipErrorInvalidValue, before the DXGI budget, with before/after lines in the log. The 8 x 250 ms retries at every call site and the strata-device loop are dropped. Thank you.

@Niko1221 Niko1221 closed this Oct 6, 2026
Niko1221 added a commit that referenced this pull request Oct 6, 2026
…try per process, before the DXGI budget (#873, #654)

Reworked from #873: no 8 x 250 ms retry at every failing call site, no #if in device_info (cudaMemGetInfo already maps to this function), no retry of real failures (only hipErrorInvalidValue, once), before/after lines in the log. The strata-device 5x retry loop is dropped.

Co-authored-by: unimo-chan <yugo.miyachi@gmail.com>
anon761 pushed a commit to anon761/Strata-upstream that referenced this pull request Oct 6, 2026
…try per process, before the DXGI budget (Niko1221#873, Niko1221#654)

Reworked from Niko1221#873: no 8 x 250 ms retry at every failing call site, no #if in device_info (cudaMemGetInfo already maps to this function), no retry of real failures (only hipErrorInvalidValue, once), before/after lines in the log. The strata-device 5x retry loop is dropped.

Co-authored-by: unimo-chan <yugo.miyachi@gmail.com>
sskver pushed a commit to sskver/Strata that referenced this pull request Oct 7, 2026
…try per process, before the DXGI budget (Niko1221#873, Niko1221#654)

Reworked from Niko1221#873: no 8 x 250 ms retry at every failing call site, no #if in device_info (cudaMemGetInfo already maps to this function), no retry of real failures (only hipErrorInvalidValue, once), before/after lines in the log. The strata-device 5x retry loop is dropped.

Co-authored-by: unimo-chan <yugo.miyachi@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants