Skip to content

IOCTL regression using ASR and embedding models simultaneously (Krackan Point) #655

Open
Open
IOCTL regression using ASR and embedding models simultaneously (Krackan Point)#655

Description

@danimalquackers
danimalquackers
opened on Aug 6, 2026

Hardware: AMD Ryzen AI 350 (Krackan Point) chip with XDNA2 support on a Framework 13 mainboard. Running NixOS with the nix-amd-ai flake to install FastFlowLM, Lemonade, and other backends.

You can see noamsto/nix-amd-ai#79 for earlier discussion and bisecting of this issue. TL;DR: after FastFlowLM 0.9.44, FLM stopped being able to load and run any models on my Krackan Point system, with an IOCTL-related error. The downstream repository author helped me bisect the issue and ruled out it being a difference in kernel version, since downgrading to 0.9.44 on my up-to-date (kernel 7.1.3) system allowed models to run again. Note that the same error appeared regardless of the model chosen, and regardless of whether it was run with sudo or not:

❯ flm run gemma4-it:e4b
[FLM]  Loading model: /home/example/.config/flm/models/Gemma4-E4B-IT-NPU2
[ERROR]  Failed to load model: DRM_IOCTL_AMDXDNA_CREATE_HWCTX IOCTL failed (err=-22): Invalid argument

The kernel logs also included several related lines:

[84319.443131] amdxdna 0000:c2:00.1: [drm] *ERROR* aie2_send_mgmt_msg_wait: command opcode 0x2 failed, status 0x2000003
[84319.443143] amdxdna 0000:c2:00.1: [drm] *ERROR* aie2_hwctx_init: Alloc hw resource failed, ret -22
[84319.443198] amdxdna 0000:c2:00.1: [drm] *ERROR* amdxdna_drm_create_hwctx_ioctl: Init hwctx failed, ret -22

Other relevant output:

❯ flm validate
[Linux]  Kernel: 7.1.3
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.8
[Linux]  Memlock Limit: infinity

c2:00.1 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Strix/Krackan/Strix Halo Neural Processing Unit [1022:17f0] (rev 20)
	Subsystem: Framework Computer Inc. Device [f111:000b]
	Kernel driver in use: amdxdna
	Kernel modules: amdxdna

Plus some troubleshooting steps from an older kernel and FLM version:

❯ journalctl -k -b | grep -iE 'amdxdna|pasid|carveout|sva'
Aug 05 14:01:09 pike kernel: amdxdna 0000:c2:00.1: [drm] Load firmware amdnpu/17f0_10/npu_7.sbin
Aug 05 14:01:09 pike kernel: amdxdna 0000:c2:00.1: enabling device (0000 -> 0002)
Aug 05 14:01:09 pike kernel: [drm] Initialized amdxdna_accel_driver 0.6.0 for 0000:c2:00.1 on minor 0

❯ cat /sys/bus/pci/devices/0000:c2:00.1/iommu_group/type
─────┬───────────────────────────────────────────────────────────────────────────────
     │ File: /sys/bus/pci/devices/0000:c2:00.1/iommu_group/type
─────┼───────────────────────────────────────────────────────────────────────────────
   1 │ identity
─────┴──────────────────────────────────────────────────────────────────────────────

❯ sudo grep -H . /sys/module/amdxdna/parameters/*
/sys/module/amdxdna/parameters/aie2_max_col:128
/sys/module/amdxdna/parameters/force_cmdlist:Y

Activity

  1. Atomic-Germ commented on Aug 7, 2026

    @Atomic-Germ
    Atomic-Germ
    on Aug 7, 2026

    What are your iommu settings? I have similar hardware and it's working with this version.
    Kernel: 7.2.0-0.rc6.260806g0d8395707651.51.fc45.x86_64
    NPU FW Version: 1.1.2.64
    amdxdna version: 0.10
    Memlock Limit: infinity

    I don't know if any recent changes would have brokens something with amdxnda 0.80 but I'd look at that.

    journalctl -k -b | grep -iE 'amdxdna|pasid|carveout|sva'
Aug 07 08:09:36 fedora kernel: amdxdna 0000:c2:00.1: [drm] Load firmware amdnpu/17f0_10/npu_7.sbin
Aug 07 08:09:36 fedora kernel: amdxdna 0000:c2:00.1: enabling device (0000 -> 0002)
Aug 07 08:09:36 fedora kernel: [drm] Initialized amdxdna_accel_driver 0.10.0 for 0000:c2:00.1 on minor 0

    and I have this:

    /sys/module/amdxdna/parameters/force_iova:N
/sys/module/amdxdna/parameters/tdr_timeout_ms:2000

    EDIT: Are you trying to run two flm run sessions? That will reliably give that exact error.

  2. simongoudie commented on Aug 8, 2026

    @simongoudie
    simongoudie
    on Aug 8, 2026

    I'm on: Lenovo Yoga 7 Pro, Ryzen AI 350 / Krackan Point, 1022:17f0 rev 20). Kernel 7.1.6, amdxdna 0.8, NPU FW 1.1.2.64., FLM 0.9.46. NixOS/Hyprland

    Just adding that I can reproduce the error and log with two running sessions as well (flm run x2, or lemonade+flm run), but I'm running fine when using just the one.

  3. danimalquackers commented on Aug 9, 2026

    @danimalquackers
    danimalquackers
    on Aug 9, 2026
    Author

    Thanks, that tipped me off to look for other processes. Strangely, this seems to pop up only when using concurrent server mode. I updated everything back to the latest versions and now flm run works.

    It looks like I might've had flm serve running in a service, but it wasn't responding so I assumed it had failed. However, what I discovered is that flm serve was running but doesn't seem to be able to load the primary model if both --asr 1 and --embed 1 are set. If either --asr or --embed are set but not both, then the model loads fine, but otherwise it starts up but without the main model:

    ❯ flm serve "qwen3.5:2b" --embed 1 --asr 1
[FLM]  ASR mode enabled: reserving additional 1GB of memory
[FLM]  Embedding mode enabled: reserving additional 300MB of memory
[FLM]  Using environment-specified port: 52625
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Whisper-V3-Turbo-NPU2
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Embedding-Gemma-300M-NPU2
[FLM]  Qwen3.5 pre-resize image height to 720 pixels if larger than that
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Qwen3.5-2B-NPU2
[ERROR]  Failed to load model: DRM_IOCTL_AMDXDNA_CREATE_HWCTX IOCTL failed (err=-22): Invalid argument
[Error]  Failed to load default model: qwen3.5:2b
[FLM]  Starting server on port 52625...
[FLM]  WebServer started on port 52625 with 10 I/O threads
[FLM]  Press Ctrl+C to stop.

❯ flm serve "qwen3.5:2b" --embed 1
[FLM]  Embedding mode enabled: reserving additional 300MB of memory
[FLM]  Using environment-specified port: 52625
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Embedding-Gemma-300M-NPU2
[FLM]  Qwen3.5 pre-resize image height to 720 pixels if larger than that
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Qwen3.5-2B-NPU2
[FLM]  Starting server on port 52625...
[FLM]  WebServer started on port 52625 with 10 I/O threads
[FLM]  Press Ctrl+C to stop.

❯ flm serve "qwen3.5:2b" --asr 1
[FLM]  ASR mode enabled: reserving additional 1GB of memory
[FLM]  Using environment-specified port: 52625
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Whisper-V3-Turbo-NPU2
[FLM]  Qwen3.5 pre-resize image height to 720 pixels if larger than that
[FLM]  Loading model: /home/dhellstern/.config/flm/models/Qwen3.5-2B-NPU2
[FLM]  Starting server on port 52625...
[FLM]  WebServer started on port 52625 with 10 I/O threads
[FLM]  Press Ctrl+C to stop.

    The first fails with the error I was seeing:

    ug 08 18:04:34 pike kernel: amdxdna 0000:c2:00.1: [drm] *ERROR* aie2_send_mgmt_msg_wait: command opcode 0x2 failed, status 0x2000003
Aug 08 18:04:34 pike kernel: amdxdna 0000:c2:00.1: [drm] *ERROR* aie2_hwctx_init: Alloc hw resource failed, ret -22
Aug 08 18:04:34 pike kernel: amdxdna 0000:c2:00.1: [drm] *ERROR* amdxdna_drm_create_hwctx_ioctl: Init hwctx failed, ret -22

    The other two succeed. For now I'll run with just --asr 1 and no embedding model, but this still seems to be a regression since it used to be possible to use both sidecars.

    Other info:

    Aug 05 16:50:22 pike kernel: amdxdna 0000:c2:00.1: [drm] Load firmware amdnpu/17f0_10/npu_7.sbin
Aug 05 16:50:22 pike kernel: amdxdna 0000:c2:00.1: enabling device (0000 -> 0002)
Aug 05 16:50:22 pike kernel: [drm] Initialized amdxdna_accel_driver 0.8.0 for 0000:c2:00.1 on minor 0

    /sys/module/amdxdna/parameters/aie2_max_col:128
/sys/module/amdxdna/parameters/force_cmdlist:Y
/sys/module/amdxdna/parameters/force_iova:N

  4. Atomic-Germ commented on Aug 9, 2026

    @Atomic-Germ
    Atomic-Germ
    on Aug 9, 2026

    @danimalquackers I just reproduced the same with flm serve qwen3.6-moe:35b-a3b --embed 1 --asr 1 -- works with one or the other but not both. I'm guessing this and the original two-processes issue are one and the same.
    I bet asr and embed try to use the same process; or rather try not to.

  5. danimalquackers
    changed the title [-]IOCTL regression on Krackan Point chips[/-] [+]IOCTL regression using ASR and embedding on Krackan Point chips[/+] on Aug 9, 2026

  6. danimalquackers commented on Aug 29, 2026

    @danimalquackers
    danimalquackers
    on Aug 29, 2026 · Hidden as outdated
    Author
  7. danimalquackers
    changed the title [-]IOCTL regression using ASR and embedding on Krackan Point chips[/-] [+]IOCTL regression using ASR and embedding models simultaneously [/+] on Aug 29, 2026
  8. danimalquackers
    changed the title [-]IOCTL regression using ASR and embedding models simultaneously [/-] [+]IOCTL regression using ASR and embedding models simultaneously (Krackan Point)[/+] on Aug 29, 2026

  9. expelledboy commented on Aug 29, 2026

    @expelledboy
    expelledboy
    on Aug 29, 2026

    The column-budget hypothesis is refuted — three contexts load fine on two different hosts

    I have a Strix Halo (rev 0x11, npu5) box and ran your exact failing command. It works:

    $ flm serve llama3.2:1b --asr 1 --embed 1 -p 11441
[FLM]  ASR mode enabled: reserving additional 1GB of memory
[FLM]  Embedding mode enabled: reserving additional 300MB of memory
[FLM]  Loading model: .../Whisper-V3-Turbo-NPU2
[FLM]  Loading model: .../Embedding-Gemma-300M-NPU2
[FLM]  Loading model: .../Llama-3.2-1B-NPU2
[FLM]  WebServer started on port 11441 with 10 I/O threads

    All three loaded — same three-model shape as your failing paste — no CREATE_HWCTX error, nothing in journalctl -k, and the LLM then served inference normally. @noamsto independently ran the same command on Strix Point (rev 0x10, npu4) with the same result.

    So "one context consumes the whole 8-column array, leaving nowhere for a second to land" cannot be the mechanism: two hosts hold three simultaneous hardware contexts. Separately, on this box I also ran three independent flm serve processes at once (llama3.2:1b, qwen3:0.6b, translategemma:4b, three ports) and all three answered concurrent requests.

    One correction to the quoted text, since it's about my data: "the Halo concurrency numbers in #94 pair an NPU model with an iGPU model — one NPU context — and run fine" describes an older result. The runs above are three NPU contexts, no iGPU involved. That quote is also stale in another way — its author retracted the column-budget hypothesis on noamsto/nix-amd-ai#94 after running the test himself, so it shouldn't be weighed as a current position.

    What the reports actually line up on

    host rev kernel amdxdna FW FLM --asr 1 --embed 1
    @danimalquackers 0x20 7.1.x 0.8.0 1.1.2.64 0.9.46 fails
    @simongoudie 0x20 7.1.6 0.8 1.1.2.64 0.9.46 fails (2 sessions)
    @Atomic-Germ ? 7.2.0-rc6 0.10.0 1.1.2.64 ? fails
    @noamsto 0x10 7.1.8 0.8 1.1.2.64 1.0.3 works
    this box 0x11 7.1.0 0.8 1.1.2.65 1.0.3 works

    Reading down the columns, most of the obvious candidates are already excluded:

    • Not the device profile. 0x10 and 0x11 work, so it isn't Halo-specific — but nothing here shows 0x20 is the cause either, because no rev-0x20 host has been tested on the same FLM version as the two that work.
    • Not the firmware. I'm the only host on 1.1.2.65 and it works, but @noamsto works on 1.1.2.64, which is what all three failing hosts run. Firmware is out. (I'd wondered about this and it's the thing this table kills.)
    • Not the amdxdna version by itself. 0.8 appears on both working hosts and on two failing ones.

    The one variable that separates the two groups cleanly is the FLM version. Both working reports are 1.0.3; both Krackan failures are 0.9.46. And @danimalquackers already noted on 2026-08-09 that updating to latest fixed plain flm run — the --asr 1 --embed 1 case just wasn't re-tested afterwards.

    The missing cell

    Has anyone run FLM 1.0.3 with --asr 1 --embed 1 on a rev-0x20 host? As far as I can tell from this thread, no — and it's the only cell that distinguishes "fixed between 0.9.46 and 1.0.3" from "genuinely npu6-specific".

    @danimalquackers — if you're still on 0.9.46, that's a one-command test and it would settle this. If it works, the issue closes as already-fixed. If it still fails on 1.0.3, then rev 0x20 really is the differentiator and it's worth chasing the dev_npu6_info profile, with two known-good comparison hosts to diff against.

    @Atomic-Germ — your FLM version isn't recorded for the qwen3.6-moe reproduction, and your amdxdna is 0.10 on a 7.2.0-rc6 kernel while everyone else is on 0.8/7.1.x. Both would be useful to pin down; if you're on 1.0.3 and still failing, that changes the picture considerably.

    Environment, for the diff

    1022:17f0 rev 0x11, kernel 7.1.0, [drm] Initialized amdxdna_accel_driver 0.8.0 for 0000:c5:00.1, FW 1.1.2.65, Memlock Limit: infinity, NixOS 26.11. IOMMU group type identity, booted with iommu.passthrough=0 (i.e. not amd_iommu=off). FLM 1.0.3 from the upstream fastflowlm_1.0.3_linux.tar.gz release.

    Note the PCI address differs per host — c5:00.1 here and on @noamsto's Strix Point, c2:00.1 on the Framework — so anything reading a hard-coded PCI path from someone else's writeup will look at the wrong device; /sys/class/accel/accel0 is the stable route.

    Happy to run further tests here on rev 0x11 if a specific configuration would help narrow it.

  10. simongoudie commented on Aug 30, 2026

    @simongoudie
    simongoudie
    on Aug 30, 2026

    This might help? v1.0.3 on rev20

    ❯ flm validate
    [Linux] Kernel: 7.2.0
    [Linux] NPU: /dev/accel/accel0 with 8 columns
    [Linux] NPU FW Version: 1.1.2.64
    [Linux] amdxdna version: 0.10
    [Linux] Memlock Limit: infinity

    ❯ flm --version
    FLM v1.0.3

    ❯ lspci -nn -d 1022:17f0 -v
    c2:00.1 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Strix/Krackan/Strix Halo Neural Processing Unit [1022:17f0] (rev 20)
    Subsystem: Lenovo Device [17aa:382d]
    Flags: bus master, fast devsel, latency 0, IRQ 98, IOMMU group 23
    Memory at b0900000 (32-bit, non-prefetchable) [size=1M]
    Memory at b0a00000 (32-bit, non-prefetchable) [size=8K]
    Memory at 5020800000 (64-bit, prefetchable) [size=512K]
    Memory at b0a03000 (32-bit, non-prefetchable) [size=4K]
    Memory at b0a02000 (32-bit, non-prefetchable) [size=4K]
    Capabilities:
    Kernel driver in use: amdxdna
    Kernel modules: amdxdna

    ❯ flm serve llama3.2:1b --asr 1 --embed 1 -p 11441
    [FLM] ASR mode enabled: reserving additional 1GB of memory
    [FLM] Embedding mode enabled: reserving additional 300MB of memory
    [FLM] Using user-specified port: 11441
    [FLM] Loading model: /home/simon/.config/flm/models/Whisper-V3-Turbo-NPU2
    [FLM] Loading model: /home/simon/.config/flm/models/Embedding-Gemma-300M-NPU2
    [FLM] Loading model: /home/simon/.config/flm/models/Llama-3.2-1B-NPU2
    [FLM] Starting server on port 11441...
    [FLM] WebServer started on port 11441 with 10 I/O threads
    [FLM] Press Ctrl+C to stop.

  11. BearGrabs commented on Oct 1, 2026

    @BearGrabs
    BearGrabs
    on Oct 1, 2026

    Can reproduce the issue when loading both embedding and audio; works fine when loading only embedding or audio. This issue only occurs with Qwen models for me.

    Other Info:
    FLM Version: 1.0.7
    [Linux] Kernel: 7.2.6+deb14-amd64
    [Linux] NPU: /dev/accel/accel0 with 8 columns
    [Linux] NPU FW Version: 1.1.2.64
    [Linux] amdxdna version: 0.10
    [Linux] Memlock Limit: infinity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions