Skip to content

HIP (gfx1200) RuntimeError Exception Code: 0xC0000005 #654

Closed as not planned
Closed as not planned
HIP (gfx1200) RuntimeError Exception Code: 0xC0000005#654

Description

@remielowik
remielowik
opened on Oct 3, 2026

Ran the start-here.bat and it downloaded & tried to start but it simply fails with a stacktrace:

Strata - Qwen3.8-Flash-Next on a normal PC (a GPU + system RAM + CPU)
[ok] GPU: AMD Radeon RX 9060 XT (16 GB, AMD)

Starting qwen3.8-flash-next-iq3_xxs: it loads about 47 GB into RAM and locks part of it for the GPU.
While it does, YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES (longer the first time after a
restart). That is normal: please wait and don't close this window - the browser opens when it is ready.
Later, closing this window stops the model.

Settings (strata-iq3_xxs.json): --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5
--max-context 65536 --kv int8 --resident-experts --control-vector-layer-range 4 --cvec-mode project
--cvec-dir per-layer; server 127.0.0.1:8080
loading the model (the first start takes a minute or two) ...
[strata] starting the engine: reading the model's weights ...
Traceback (most recent call last):
File "C:\Users\Remie\Desktop\llama\strata\serve\server.py", line 3060, in
sys.exit(main())
^^^^^^
File "C:\Users\Remie\Desktop\llama\strata\serve\server.py", line 2925, in main
engine = StrataEngine(exe, engine_args(cfg), cwd=cfg.get("cwd"), log=cfg.get("log"), env=env, lazy=lazy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Remie\Desktop\llama\strata\serve\server.py", line 362, in init
raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else "") +
RuntimeError: the engine exited before it was ready (see C:\Users\Remie\Desktop\llama\strata\strata-iq3_xxs.log)
the engine log's last lines:
Exception Code: 0xC0000005
0x00007FF8FE603DF0, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x453DF0 byte(s), hipProfilerRegisterChunkCallbackExt() + 0xAD570 byte(s)
0x00007FF8FE5F4D7D, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x444D7D byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9E4FD byte(s)
0x00007FF8FE5F0E43, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x440E43 byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9A5C3 byte(s)
0x00007FF8FE5F03FA, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x4403FA byte(s), hipProfilerRegisterChunkCallbackExt() + 0x99B7A byte(s)
0x00007FF8FE5F4B7C, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x444B7C byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9E2FC byte(s)
0x00007FF8FE5F48C7, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x4448C7 byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9E047 byte(s)
0x00007FF8FE5F81A1, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x4481A1 byte(s), hipProfilerRegisterChunkCallbackExt() + 0xA1921 byte(s)
0x00007FF8FE600DCD, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x450DCD byte(s), hipProfilerRegisterChunkCallbackExt() + 0xAA54D byte(s)
0x00007FF8FE5CDFFD, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x41DFFD byte(s), hipProfilerRegisterChunkCallbackExt() + 0x7777D byte(s)
0x00007FF8FE58D02D, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x3DD02D byte(s), hipProfilerRegisterChunkCallbackExt() + 0x367AD byte(s)
0x00007FF8FE59ECB9, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x3EECB9 byte(s), hipProfilerRegisterChunkCallbackExt() + 0x48439 byte(s)
0x00007FF8FE5699AF, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE1B0000) + 0x3B99AF byte(s), hipProfilerRegisterChunkCallbackExt() + 0x1312F byte(s)
0x00007FF9C459259D, C:\WINDOWS\System32\KERNEL32.DLL(0x00007FF9C4580000) + 0x1259D byte(s), BaseThreadInitThunk() + 0x1D byte(s)
0x00007FF9C652AF78, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FF9C64D0000) + 0x5AF78 byte(s), RtlUserThreadStart() + 0x28 byte(s)
Press any key to continue . . .

Activity

  1. Niko1221 commented on Oct 3, 2026

    @Niko1221
    Niko1221
    on Oct 3, 2026
    Owner

    Thanks for the report. The crash is inside AMD's HIP runtime (amdhip64_7.dll) while the engine reads the model. Your config uses the low-RAM mode's resident variant (--resident-experts), which page-locks the experts the GPU doesn't hold. That variant hasn't run on an AMD card on Windows yet, so it's our first suspect.

    Could you run SETUP.bat --low-ram mmap (the experts are then read through the file cache instead) and tell us whether it starts? Please also post:

    • how much RAM the PC has, and your AMD driver version (AMD Software > System)
    • the first ~60 lines of strata-iq3_xxs.log
    • the output of engine\strata-device.exe --selftest (from the Strata folder, after set PATH=%CD%\engine\rocm\bin;%PATH%)

  2. remielowik commented on Oct 3, 2026

    @remielowik
    remielowik
    on Oct 3, 2026
    Author

    Strata - Qwen3.8-Flash-Next on a normal PC (a GPU + system RAM + CPU)

    === Step 1: checking your PC ===
    Your AMD GPUs:
    GPU 0: AMD Radeon RX 9060 XT, 16 GB VRAM - can be used
    [ok] GPU: AMD Radeon RX 9060 XT, 15.9 GB VRAM, gfx1200 (AMD: docs/AMD_HIP.md)
    [ok] RAM: 48 GB
    [!] Windows' page file is 3.0 GB: the graphics card's memory needs room there too (issue #60), so the model may not start or may use less VRAM. Set it to "System managed": System > About > Advanced system settings > Performance > Advanced > Virtual memory
    [ok] CPU: AMD64 Family 25 Model 97 Stepping 2, AuthenticAMD (AVX-512)

    === Step 2: your choices ===

    1. Qwen3.8-Flash-Next Qwen; GSQ-RCO quants by ISTA-DASLab - the original model

    2. Swift 1.5 UkisAI's fine-tune of Qwen3.8-Flash-Next - thinks much shorter (-63% thinking tokens, 1.8x sooner answers by its authors' numbers)

    3. Qwen3.8-Flash-Next Coder ISTA-DASLab's coding version - half the experts (code, tools, images kept): needs ~32 GB of RAM, faster; weaker outside coding

    4. Qwen3.8-Flash-Next (Unsloth) Unsloth's 4-bit quantization (EXPERIMENTAL) - 4-bit, 111 GB download, most experts read from the SSD: slow (7-8.5 tokens/s on a 64 GB PC) [experimental]
      Which model? [1]: 1
      [ok] model: Qwen3.8-Flash-Next

    5. Q2_0 2-bit, the fastest; download 66 GB, uses ~34 GB of RAM <- needs 48 GB RAM, you have 48

    6. IQ2_XS 2-bit i-quant, a little better quality, close in speed; download 68 GB, uses ~36 GB of RAM <- needs 48 GB RAM, you have 48

    7. IQ3_XXS 3-bit i-quant, better quality, slower (more CPU work per token); download 76 GB, uses ~43 GB of RAM <- fits in the low-RAM mode (the GPU holds ~25%, the rest in RAM)

    8. IQ3_S 3.5-bit i-quant, the best quality (matches the full model), the slowest; needs a 64 GB PC with little else running; download 84 GB, uses ~50 GB of RAM <- fits in the low-RAM mode (the GPU holds ~22%, the rest from the SSD)
      Which size? [1]: 3
      [ok] size: IQ3_XXS

    Context length = how much text the model can see at once (your chat, files, tool output).
    Longer needs more VRAM for it, so fewer experts fit on the GPU:

    1. 8K tokens
    2. 32K tokens
    3. 64K tokens (recommended for your GPU)
    4. 128K tokens
    5. 256K tokens
    6. 384K tokens (experimental: setup adds rope scaling)
    7. 512K tokens (experimental: setup adds rope scaling)
      Context? [3]: 3
      [ok] context: 65536 tokens

    KV cache precision (the model's memory of the conversation):

    1. 8-bit (recommended: what every published number was measured with)
    2. 4-bit half the memory (about 4% faster at 128K), but measurably less precise on long
      documents; long-context lookups (needle tests) still pass
      KV cache? [1]: 1
      [ok] KV cache: 8-bit
      [ok] images: off
      [ok] low-RAM mode: IQ3_XXS's experts (43 GB) are read from the model folder through the OS file cache instead of a copy in RAM (48 GB); the GPU holds ~24% of them
      [!] most of the experts are read from the SSD while it answers: expect it to be much slower than with enough RAM (a faster SSD and a smaller size help)

    EXPERIMENTAL - speed projection: a small control vector applied while the model runs (layers 4-44).
    It changes how the model answers: its package describes it as a refusal-direction projection (the
    model declines far fewer requests). Off unless you choose it; when on, the web app can switch it off
    per chat. Details: data/experimental-speed-projection/README.md
    Turn on the experimental speed projection? [n]: n
    [ok] experimental speed projection: off

    === Step 3: Python packages ===
    [ok] numpy, jinja2, regex, pyyaml, tqdm, requests, cmake, ninja, pillow, psutil already installed

    === Step 4: the Strata engine ===
    [ok] llama.cpp 3cf0325 (gguf-py, ggml, mtmd)
    [ok] ready-made AMD engine already installed
    [ok] engine: C:\Users\Remie\Desktop\llama\strata\engine\strata.exe

    === Step 5: downloading Qwen3.8-Flash-Next IQ3_XXS ===
    [ok] Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf already downloaded
    [ok] Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf already downloaded
    [ok] model files present

    === Step 6: preparing the model for Strata ===
    [ok] model prepared: C:\Users\Remie\Desktop\llama\Strata-data\packs\iq3_xxs
    [ok] MTP draft layer: C:\Users\Remie\Desktop\llama\Strata-data\mtp\rt

    === Step 7: writing the start script ===
    [!] no hipBLASLt tuning table for gfx1200 with hipBLASLt 100500 (have: gfx1200-hipblaslt-100202.txt): the prompt's dense matrix products use plain hipBLAS (tools/hip/tune_hipblaslt makes a table: docs/AMD_HIP.md, Tuning table)
    [ok] start script: run-iq3_xxs.bat

    All set.
    API (OpenAI): http://127.0.0.1:8080/v1 (any API key; model name: anything)
    API (Anthropic): http://127.0.0.1:8080/v1/messages
    Next time: just run START-HERE.bat (or run-iq3_xxs.bat) - it starts right away
    [ok] GPU: AMD Radeon RX 9060 XT (16 GB, AMD)

    Starting qwen3.8-flash-next-iq3_xxs: it maps about 47 GB of experts from the model files (the OS file cache reads them).
    While it does, YOUR PC CAN BE SLOW OR STOP RESPONDING FOR 1-3 MINUTES (longer the first time after a
    restart). That is normal: please wait and don't close this window - the browser opens when it is ready.
    Later, closing this window stops the model.

    Settings (strata-iq3_xxs.json): --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5
    --max-context 65536 --kv int8 --mmap-experts; server 127.0.0.1:8080
    loading the model (the first start takes a minute or two) ...
    [strata] starting the engine: reading the model's weights ...
    Traceback (most recent call last):
    File "C:\Users\Remie\Desktop\llama\strata\serve\server.py", line 3060, in
    sys.exit(main())
    ^^^^^^
    File "C:\Users\Remie\Desktop\llama\strata\serve\server.py", line 2925, in main
    engine = StrataEngine(exe, engine_args(cfg), cwd=cfg.get("cwd"), log=cfg.get("log"), env=env, lazy=lazy)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "C:\Users\Remie\Desktop\llama\strata\serve\server.py", line 362, in init
    raise RuntimeError("the engine exited before it was ready" + (f" (see {log})" if log else "") +
    RuntimeError: the engine exited before it was ready (see C:\Users\Remie\Desktop\llama\strata\strata-iq3_xxs.log)
    the engine log's last lines:
    Exception Code: 0xC0000005
    0x00007FF8FE553DF0, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x453DF0 byte(s), hipProfilerRegisterChunkCallbackExt() + 0xAD570 byte(s)
    0x00007FF8FE544D7D, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x444D7D byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9E4FD byte(s)
    0x00007FF8FE540E43, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x440E43 byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9A5C3 byte(s)
    0x00007FF8FE5403FA, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x4403FA byte(s), hipProfilerRegisterChunkCallbackExt() + 0x99B7A byte(s)
    0x00007FF8FE544B7C, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x444B7C byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9E2FC byte(s)
    0x00007FF8FE5448C7, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x4448C7 byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9E047 byte(s)
    0x00007FF8FE5481A1, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x4481A1 byte(s), hipProfilerRegisterChunkCallbackExt() + 0xA1921 byte(s)
    0x00007FF8FE550DCD, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x450DCD byte(s), hipProfilerRegisterChunkCallbackExt() + 0xAA54D byte(s)
    0x00007FF8FE51DFFD, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x41DFFD byte(s), hipProfilerRegisterChunkCallbackExt() + 0x7777D byte(s)
    0x00007FF8FE4DD02D, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x3DD02D byte(s), hipProfilerRegisterChunkCallbackExt() + 0x367AD byte(s)
    0x00007FF8FE4EECB9, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x3EECB9 byte(s), hipProfilerRegisterChunkCallbackExt() + 0x48439 byte(s)
    0x00007FF8FE4B99AF, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x3B99AF byte(s), hipProfilerRegisterChunkCallbackExt() + 0x1312F byte(s)
    0x00007FF9C459259D, C:\WINDOWS\System32\KERNEL32.DLL(0x00007FF9C4580000) + 0x1259D byte(s), BaseThreadInitThunk() + 0x1D byte(s)
    0x00007FF9C652AF78, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FF9C64D0000) + 0x5AF78 byte(s), RtlUserThreadStart() + 0x28 byte(s)
    Press any key to continue . . .

  3. remielowik commented on Oct 3, 2026

    @remielowik
    remielowik
    on Oct 3, 2026
    Author

    log file:
    Exception Code: 0xC0000005
    0x00007FF8FE553DF0, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x453DF0 byte(s), hipProfilerRegisterChunkCallbackExt() + 0xAD570 byte(s)
    0x00007FF8FE544D7D, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x444D7D byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9E4FD byte(s)
    0x00007FF8FE540E43, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x440E43 byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9A5C3 byte(s)
    0x00007FF8FE5403FA, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x4403FA byte(s), hipProfilerRegisterChunkCallbackExt() + 0x99B7A byte(s)
    0x00007FF8FE544B7C, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x444B7C byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9E2FC byte(s)
    0x00007FF8FE5448C7, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x4448C7 byte(s), hipProfilerRegisterChunkCallbackExt() + 0x9E047 byte(s)
    0x00007FF8FE5481A1, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x4481A1 byte(s), hipProfilerRegisterChunkCallbackExt() + 0xA1921 byte(s)
    0x00007FF8FE550DCD, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x450DCD byte(s), hipProfilerRegisterChunkCallbackExt() + 0xAA54D byte(s)
    0x00007FF8FE51DFFD, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x41DFFD byte(s), hipProfilerRegisterChunkCallbackExt() + 0x7777D byte(s)
    0x00007FF8FE4DD02D, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x3DD02D byte(s), hipProfilerRegisterChunkCallbackExt() + 0x367AD byte(s)
    0x00007FF8FE4EECB9, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x3EECB9 byte(s), hipProfilerRegisterChunkCallbackExt() + 0x48439 byte(s)
    0x00007FF8FE4B99AF, C:\Users\Remie\Desktop\llama\strata\engine\amdhip64_7.dll(0x00007FF8FE100000) + 0x3B99AF byte(s), hipProfilerRegisterChunkCallbackExt() + 0x1312F byte(s)
    0x00007FF9C459259D, C:\WINDOWS\System32\KERNEL32.DLL(0x00007FF9C4580000) + 0x1259D byte(s), BaseThreadInitThunk() + 0x1D byte(s)
    0x00007FF9C652AF78, C:\WINDOWS\SYSTEM32\ntdll.dll(0x00007FF9C64D0000) + 0x5AF78 byte(s), RtlUserThreadStart() + 0x28 byte(s)

    this repeats for every run i tried

  4. remielowik commented on Oct 3, 2026

    @remielowik
    remielowik
    on Oct 3, 2026
    Author

    how much RAM the PC has, and your AMD driver version (AMD Software > System):
    48 GB ram, 26.8.1 (latest)

    the output of engine\strata-device.exe --selftest (from the Strata folder, after set PATH=%CD%\engine\rocm\bin;%PATH%):
    strata-device: cudaMemGetInfo: invalid argument (this is odd?)

  5. remielowik
    changed the title [-]HIP (gfx1200)[/-] [+]HIP (gfx1200) RuntimeError Exception Code: 0xC0000005[/+] on Oct 3, 2026

  6. remielowik commented on Oct 3, 2026

    @remielowik
    remielowik
    on Oct 3, 2026
    Author

    Claude found the issue:
    I found the real cause, and it isn't RAM or the model size. Machine-wide environment variables point to a ROCm build at C:\TheRock\build that no longer exists:
    HIP_PATH = C:\TheRock\build
    HIP_DEVICE_LIB_PATH = C:\TheRock\build\lib\llvm\amdgcn\bitcode
    LLVM_PATH = C:\TheRock\build\lib\llvm
    And suggests the following:
    Strata could protect itself: when it uses its bundled runtime, the server could remove HIP_PATH, HIP_DEVICE_LIB_PATH and LLVM_PATH from the engine's environment. Then a leftover ROCm install couldn't cause this. I can add that if you want.

  7. remielowik commented on Oct 3, 2026

    @remielowik
    remielowik
    on Oct 3, 2026
    Author

    And for the selftest:
    Found the cause: hipMemGetInfo fails with "invalid argument" until the process has done a hipMalloc, after which it reports memory correctly (16,235 of 16,304 MiB free). Now checking the bundled runtime using dependencies from rocm\bin.

  8. caizca-cloud commented on Oct 4, 2026

    @caizca-cloud
    caizca-cloud
    on Oct 4, 2026

    @Niko1221 Hello, Same crash here — amdhip64_7.dll + 0x453DF0 ... but not from the TheRock variables.

    strata-crash-gfx1201.dmp
    gfx1201-crash-report.md

  9. dingyiyuan commented on Oct 5, 2026

    @dingyiyuan
    dingyiyuan
    on Oct 5, 2026

    Solved on the gfx1100 machine: the 0xC0000005 before any output was a temporary-file failure in the comgr JIT, not a driver fault

    Environment where it happened and where it now runs: RX 7900 XTX / gfx1100, Windows 11 25H2
    (26200.9457), driver 32.0.31041.1004 (Adrenalin 26.8.1), Strata 0.1.39 ready-made
    strata-windows-x64-hip.zip, bundled ROCm 10.2.0a20260930.

    Symptom. strata.exe --serve exited 0xC0000005 inside the bundled amdhip64_7.dll
    (offset 0x453df0) before writing one byte to its log, reproducibly, on all of 0.1.35 /
    0.1.38 / 0.1.39.

    Cause. HIP device initialisation has PAL build its blit kernels, and that JIT goes
    through comgr, which creates temporary files while compiling. On this machine temporary-file
    creation was failing, which is exactly the cascade

    :1:devprogram.cpp :1391: Error: Creating the executable from ISA assembly text failed.
:1:paldevice.cpp  :2777: Couldn't create blit kernels!
:1:palvirtual.cpp :1060: Could not create BlitManager!
:1:paldevice.cpp  :1189: Couldn't create the device transfer manager!

    Direct evidence, same machine, bundled compiler, compiling the engine's own source:

    $ clang++.exe ... -x hip -c src/core/device.cu -o device.cu.obj      # inherits the default TEMP
clang++: error: unable to make temporary file: permission denied

$ # same command, TEMP/TMP/TMPDIR pointed at an ordinary writable directory
exit 0, object produced in 5 s

    Fix, and it works. Start the engine (or the server that spawns it) with TEMP, TMP and
    TMPDIR set to a normal writable directory. Then, on the same machine:

    [ok] experts loaded: 49.80 GiB at 3.72 GiB/s
[ok] filling the GPU's expert cache (8740 experts, 17.71 GiB of VRAM)
[ok] ready: http://127.0.0.1:8095/v1   (context 32768 tokens)

GET /health    -> {"status":"ok","loaded":true,"service":"strata"}
POST /v1/chat/completions -> "STRATA IS RUNNING"
strata serve: prompt 64 tokens = 0 reused + 64 read in 931 ms (68.7 tok/s),
              64 generated in 1366 ms (46.8 tok/s), drafts accepted 36 of 48
strata serve: decode expert cache hit rate: 82.6% (26903 hits / 32560 lookups)

    Two suggestions, both cheap and both about diagnosis rather than function.

    1. The failure is worth a guard. When this happens the engine prints nothing at all, so
      the symptom is indistinguishable from a driver or runtime fault, and the PAL/comgr lines
      above only appear in other contexts. Having the engine (or setup.py/the launcher) check
      that it can create a temporary file before initialising HIP — or report the comgr failure
      explicitly — would turn a multi-day driver investigation into a one-line diagnosis.
    2. strata-device --selftest is not a health check for HIP. It links only amdhip64_7.dll
      and no BLAS library, so it always sees the runtime's pre-initialisation state and prints
      cudaMemGetInfo: invalid argument even on a machine where the engine runs perfectly. I
      measured this with a small probe: on this same card, hipMemGetInfo returns
      hipErrorInvalidValue for the first call in a process and then the correct
      23.97 GiB free / 23.98 GiB total once hipblas / libtensilelite-host have been
      loaded. AMD's own hipInfo.exe fails at hipInfo.cpp:164 for the same reason (no BLAS).
      If --selftest is kept, loading a BLAS library first would make it meaningful.

    Thanks — the logging in strata generate: is what made this findable: the presence of
    "session is up (engine 0.1.39)" and "438 MiB of VRAM free with everything loaded" is how the
    fix was confirmed.

  10. Niko1221
    added a commit that references this issue on Oct 6, 2026

    device (Windows HIP): one hipFree(0) warm-up and one hipMemGetInfo re…

    949dcbe

  11. Niko1221 commented on Oct 6, 2026

    @Niko1221
    Niko1221
    on Oct 6, 2026
    Owner

    0.1.40 adds a one-time hipFree(0) warm-up before the first hipMemGetInfo on Windows HIP (from #873). Please try it, together with a clean HIP_PATH and a writable TEMP.

  12. Niko1221
    added a commit that references this issue on Oct 6, 2026

    device (Windows HIP): one hipFree(0) warm-up and one hipMemGetInfo re…

    fa228b0
  13. anon761
    added a commit that references this issue on Oct 6, 2026

    device (Windows HIP): one hipFree(0) warm-up and one hipMemGetInfo re…

    4ed0841
  14. Niko1221
    added 2 commits that reference this issue on Oct 7, 2026

    serve: the AMD engine's environment: a HIP_PATH that points nowhere i…

    0e773c8

    docs: troubleshooting for #781 #831, #649, #961 and #654

    5628e28

  15. Niko1221 commented on Oct 7, 2026

    @Niko1221
    Niko1221
    on Oct 7, 2026
    Owner

    0.1.40.2: when the server starts the AMD engine it drops HIP_PATH, HIP_DEVICE_LIB_PATH and LLVM_PATH if they point at a folder that does not exist, and replaces a TEMP / TMP / TMPDIR the engine cannot create files in with a writable folder next to the log (it says so in the log). Thanks to remielowik and dingyiyuan for finding both causes. If you start strata.exe by hand, set HIP_PATH empty and TEMP to a normal folder. If the crash still happens with a clean environment, please post the first lines of the engine log.

  16. sskver
    added a commit that references this issue on Oct 7, 2026

    device (Windows HIP): one hipFree(0) warm-up and one hipMemGetInfo re…

    d5cf658
  17. 0xrjman
    added 2 commits that reference this issue on Oct 7, 2026

    serve: the AMD engine's environment: a HIP_PATH that points nowhere i…

    59dd271

    docs: troubleshooting for Niko1221#781 Niko1221#831, Niko1221#649, Ni…

    83b2d1e

  18. Niko1221 commented on Oct 13, 2026

    @Niko1221
    Niko1221
    on Oct 13, 2026
    Owner

    Thanks for the report. To keep the tracker current we are closing issues older than three days: 0.1.44 is out with a long list of fixes, and many older reports are covered by it or by 0.1.43.1.

    If this still happens on 0.1.44, please leave a comment here with your engine log (and the version line it prints), and we will reopen it. A new issue that links this one works too.

  19. Niko1221
    closed this as not plannedon Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions