Skip to content

Fix Windows ROCm builds failing to find mt.exe + stop one ROCm version cancelling the others - #37

Merged
sreeram-11 merged 2 commits into
lemonadefrom
sreeram/fix-windows-rocm-mt-exe
Nov 3, 2026
Merged

sreeram-11 merged 2 commits into
lemonadefrom
sreeram/fix-windows-rocm-mt-exe

Conversation

@sreeram-11

@sreeram-11 sreeram-11 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Windows ROCm builds have failed on every run since b12013 (Oct 30), and because of that no ROCm build has been released since, Linux included.

  • Upstream started embedding a manifest in every Windows .exe (ggml-org/llama.cpp#31329). To do that, ROCm's linker needs Microsoft's mt.exe.
  • mt.exe is installed on the runner (it comes with the Windows SDK), but our windows-rocm job never puts it on PATH, so every .exe fails to link with lld-link: error: unable to find mt.exe in PATH.
    • Upstream doesn't hit this because their Windows HIP job only builds the ggml-hip DLL, never an .exe.
  • A ROCm version is only released when both its Windows and Linux builds exist, so the Linux builds were dropped too. b12013 to b12062 have no ROCm archives.

Fix

  • windows-rocm Build step: find the newest Windows SDK folder that has mt.exe and add it to the end of PATH for that step only, then run mt.exe /? so a broken mt.exe fails with a clear message instead of a link error.
  • windows-rocm and ubuntu-24-rocm: add fail-fast: false.
    • Until now, when one ROCm version failed it cancelled the other two, which hid that 7.14 and 10.0 were failing as well. Every other matrix in release.yml already does this.

Notes

@superm1 superm1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have concerns with this direction of changes but I would suggest straight cherry picks from upstream.

@sawansri sawansri left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

@sreeram-11

Copy link
Copy Markdown
Collaborator Author

I don't have concerns with this direction of changes but I would suggest straight cherry picks from upstream.

Thanks @superm1! Agreed that cherry-picking from upstream is the better default.

There's nothing upstream to pick for this one though.

  • Upstream's Windows ROCm release job only builds the ggml-hip DLL (--target ggml-hip), so it never links an .exe and never needs mt.exe. This fork builds every executable with ROCm's clang, which is why only we hit it after cmake : embed an app manifest in all Windows exes ggml-org/llama.cpp#31329.
  • Same for fail-fast: upstream's ROCm jobs build a single ROCm version each, so there's nothing for one to cancel.

Both changes only make sense in this fork.

@sreeram-11
sreeram-11 merged commit e53542f into lemonade Nov 3, 2026
25 of 26 checks passed
@sreeram-11
sreeram-11 deleted the sreeram/fix-windows-rocm-mt-exe branch November 3, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants