Skip to content

Critical bug + Performance report RTX 4090 #29

Closed
Closed
Critical bug + Performance report RTX 4090#29

Description

@Lufen34
Lufen34
opened on Sep 27, 2026

Hello ! Thank you for your work. I am really impressed by the result so far 💪

Here is my system :

OS : Cachyos (linux)

Hardware : RTX 4090 + 192 GB of RAM + AMD R9 7950X3D

version running : 0.1.9

command I run :

./setup.sh \
  --family qwen \
  --model IQ3_S \
  --context 262144 \
  --kv int8 \
  --vision none \
  --experimental-speed-projection off \
  --port 1919 \
  --host 0.0.0.0 \
  --yes%

Performance start at 75 token/s degrades slowly around 55 tok/s when it reaches a big amount of token generation which is expected and very impressive.

I do get a bug tho, everytime It reaches a certain amount of token, my cpu suddenly stop working and my gpu stays indefinitively running/processing at 100% but no more token is generated as show here :

[strata] thinking: 56869 of max 128000 tokens, 58.1 tok/s, 998 s
[strata] thinking: 56912 of max 128000 tokens, 58.1 tok/s, 999 s
[strata] thinking: 56956 of max 128000 tokens, 58.0 tok/s, 1000 s
[strata] thinking: 56999 of max 128000 tokens, 58.0 tok/s, 1001 s
[strata] thinking: 57042 of max 128000 tokens, 58.0 tok/s, 1002 s
[strata] thinking: 57086 of max 128000 tokens, 58.0 tok/s, 1003 s
[strata] thinking: 57130 of max 128000 tokens, 58.0 tok/s, 1004 s
[strata] thinking: 57172 of max 128000 tokens, 58.0 tok/s, 1005 s
[strata] thinking: 57215 of max 128000 tokens, 57.9 tok/s, 1006 s
[strata] thinking: 57257 of max 128000 tokens, 57.9 tok/s, 1007 s
[strata] thinking: 57300 of max 128000 tokens, 57.9 tok/s, 1008 s
[strata] thinking: 57344 of max 128000 tokens, 57.9 tok/s, 1009 s
[strata] thinking: 57387 of max 128000 tokens, 57.9 tok/s, 1010 s
[strata] thinking: 57426 of max 128000 tokens, 57.9 tok/s, 1011 s
[strata] thinking: 57468 of max 128000 tokens, 57.8 tok/s, 1012 s
[strata] thinking: 57510 of max 128000 tokens, 57.8 tok/s, 1013 s
[strata] thinking: 57552 of max 128000 tokens, 57.8 tok/s, 1014 s
[strata] thinking: 57613 of max 128000 tokens, 57.8 tok/s, 1015 s
[strata] thinking: 57720 of max 128000 tokens, 57.9 tok/s, 1016 s
[strata] thinking: 57828 of max 128000 tokens, 57.9 tok/s, 1017 s
[strata] thinking: 57936 of max 128000 tokens, 58.0 tok/s, 1018 s
[strata] thinking: 58045 of max 128000 tokens, 58.0 tok/s, 1019 s
[strata] thinking: 58156 of max 128000 tokens, 58.1 tok/s, 1020 s
[strata] thinking: 58265 of max 128000 tokens, 58.1 tok/s, 1021 s
[strata] thinking: **58303** of max 128000 tokens, 57.6 tok/s, 1032 s
[strata] thinking: **58303** of max 128000 tokens, 57.0 tok/s, 1042 s
[strata] thinking: **58303** of max 128000 tokens, 56.4 tok/s, 1052 s
[strata] thinking: **58303** of max 128000 tokens, 55.9 tok/s, 1062 s
[strata] thinking: **58303** of max 128000 tokens, 55.4 tok/s, 1072 s

The prompt I used to make him spill a lot of tokens :

prompt.md

Edit : just for clarity I still have enough VRAM room and plenty of RAM left

Activity

  1. Lufen34
    changed the title [-]Critical bug + Performance report[/-] [+]Critical bug + Performance report RTX 4090[/+] on Sep 27, 2026

  2. Lufen34 commented on Sep 27, 2026

    @Lufen34
    Lufen34
    on Sep 27, 2026
    Author

    this might have been fixed and the Strata v0.1.10 so I give it a try again

  3. Lufen34 commented on Sep 27, 2026

    @Lufen34
    Lufen34
    on Sep 27, 2026
    Author

    Issue still present, this time way earlier :(

    [strata] thinking: 6767 of max 128000 tokens, 71.7 tok/s, 113 s
    [strata] thinking: 6819 of max 128000 tokens, 71.5 tok/s, 114 s
    [strata] thinking: 6879 of max 128000 tokens, 71.4 tok/s, 115 s
    [strata] thinking: 6961 of max 128000 tokens, 71.5 tok/s, 116 s
    [strata] thinking: 7014 of max 128000 tokens, 71.3 tok/s, 117 s
    [strata] thinking: 7081 of max 128000 tokens, 71.2 tok/s, 118 s
    [strata] thinking: 7162 of max 128000 tokens, 71.3 tok/s, 119 s
    [strata] thinking: 7220 of max 128000 tokens, 71.2 tok/s, 120 s
    [strata] thinking: 7268 of max 128000 tokens, 70.9 tok/s, 121 s
    [strata] thinking: 7329 of max 128000 tokens, 70.8 tok/s, 122 s
    [strata] thinking: 7417 of max 128000 tokens, 71.0 tok/s, 123 s
    [strata] thinking: 7487 of max 128000 tokens, 71.0 tok/s, 124 s
    [strata] thinking: 7556 of max 128000 tokens, 71.0 tok/s, 125 s
    [strata] thinking: 7624 of max 128000 tokens, 70.9 tok/s, 126 s
    [strata] thinking: 7687 of max 128000 tokens, 65.0 tok/s, 137 s
    [strata] thinking: 7687 of max 128000 tokens, 59.9 tok/s, 147 s
    [strata] thinking: 7687 of max 128000 tokens, 55.6 tok/s, 157 s
    [strata] thinking: 7687 of max 128000 tokens, 51.8 tok/s, 167 s
    [strata] thinking: 7687 of max 128000 tokens, 48.6 tok/s, 177 s
    [strata] thinking: 7687 of max 128000 tokens, 45.7 tok/s, 187 s
    , [strata] thinking: 7687 of max 128000 tokens, 43.1 tok/s, 197 s

    ❯ git describe --tags --always
v0.1.11

  4. Niko1221 commented on Sep 27, 2026

    @Niko1221
    Niko1221
    on Sep 27, 2026
    Owner

    I know the reason, will be patched withing 1-2hrs from now!

  5. Niko1221 commented on Sep 27, 2026

    @Niko1221
    Niko1221
    on Sep 27, 2026
    Owner

    Could you please change kv-cache by not using int? Tell me if it works.

  6. Lufen34 commented on Sep 27, 2026

    @Lufen34
    Lufen34
    on Sep 27, 2026
    Author

    Testing now :)

  7. Lufen34 commented on Sep 27, 2026

    @Lufen34
    Lufen34
    on Sep 27, 2026
    Author

    Same issue with Q4_0 :

    ./setup.sh \
  --family qwen \
  --model IQ3_S \
  --context 262144 \
  --kv q4_0 \
  --vision none \
  --experimental-speed-projection off \
  --port 1919 \
  --host 0.0.0.0 \
  --yes

    [strata] thinking: 8611 of max 128000 tokens, 69.7 tok/s, 142 s
    [strata] thinking: 8675 of max 128000 tokens, 69.6 tok/s, 143 s
    [strata] thinking: 8741 of max 128000 tokens, 69.5 tok/s, 144 s
    [strata] thinking: 8799 of max 128000 tokens, 69.5 tok/s, 145 s
    [strata] thinking: 8859 of max 128000 tokens, 69.4 tok/s, 146 s
    [strata] thinking: 8926 of max 128000 tokens, 69.4 tok/s, 147 s
    [strata] thinking: 8993 of max 128000 tokens, 69.3 tok/s, 148 s
    [strata] thinking: 9068 of max 128000 tokens, 69.4 tok/s, 149 s
    [strata] thinking: 9132 of max 128000 tokens, 64.5 tok/s, 160 s
    [strata] thinking: 9132 of max 128000 tokens, 60.2 tok/s, 170 s
    [strata] thinking: 9132 of max 128000 tokens, 56.5 tok/s, 180 s
    [strata] thinking: 9132 of max 128000 tokens, 53.2 tok/s, 190 s

    Image Image
  8. Niko1221
    added a commit that references this issue on Sep 27, 2026

    engine: fix a race in the CPU expert pool, and never hang silently ag…

    06438fd

  9. Niko1221 commented on Sep 27, 2026

    @Niko1221
    Niko1221
    on Sep 27, 2026
    Owner

    Thanks a lot for the detailed report and the screenshots. The GPU at 100% but only 91 W, with exactly one CPU core busy, pinned it down: the GPU and the engine's host thread were each waiting for the other.

    What we found: a race in the CPU expert pool.

    • The pool's workers go to sleep after 20 ms without work (the fix for Cpu 50% even when not procesesing anything, doing nothing... #4).
    • On your 4090 most experts sit in VRAM, so many layers give the CPU nothing to do, and the workers fell asleep in the middle of an answer too.
    • A worker that woke late was still counted as idle, so it could take a job from the next batch while that batch was still being set up. That could make the engine lose count of finished jobs and wait forever, which is why it hit at random points (7.7K, 9.1K, 58K).
    • It didn't depend on the KV format, which is why switching to Q4_0 didn't help. Sorry for the wrong guess there.

    Fixed in v0.1.12:

    • Every job is now tied to its batch, so a late worker can't touch the next one. Speed is unchanged.
    • The engine can no longer hang silently. If a request makes no progress for 2 minutes, it stops with a log line saying where it was stuck. The server answers with an error and starts the engine again on the next request.

    I couldn't make it hang on my 12 GB card, even with a stress build that makes the workers sleep constantly: 37K tokens on the old engine, and 25K on the new one at 262K with KV streaming, with no stall. So your run is the real test. Please update (git pull, then run setup again to get engine 0.1.12) and try your prompt. If it ever stops again, the log will contain no progress for 120 s ... (issue #29) with the stage; please reopen this and paste that line. Closing for now.

  10. Niko1221
    closed this as completedon Sep 27, 2026

  11. Lufen34 commented on Sep 27, 2026

    @Lufen34
    Lufen34
    on Sep 27, 2026
    Author

    Haha I just tested but this time Instead of hanging it just stop abruptly.

    [strata] thinking: 6450 of max 128000 tokens, 78.5 tok/s, 101 s
[strata] thinking: 6596 of max 128000 tokens, 79.3 tok/s, 102 s
[strata] thinking: 6751 of max 128000 tokens, 80.2 tok/s, 103 s
[strata] thinking: 6906 of max 128000 tokens, 81.0 tok/s, 104 s
[strata] thinking: 7064 of max 128000 tokens, 81.9 tok/s, 105 s
[strata] thinking: 7219 of max 128000 tokens, 82.7 tok/s, 106 s
[strata] thinking: 7372 of max 128000 tokens, 83.5 tok/s, 107 s
[strata] thinking: 7517 of max 128000 tokens, 84.2 tok/s, 108 s
[strata] thinking: 7651 of max 128000 tokens, 84.8 tok/s, 109 s
[strata] done: 7678 tokens in 109 s (84.9 tok/s) (stop, cancel=False)
[strata] thinking: 9 of max 262042 tokens, 35.7 tok/s, 1 s
[strata] thinking: 61 of max 262042 tokens, 48.7 tok/s, 2 s
[strata] done: 89 tokens in 3 s (51.4 tok/s) (stop, cancel=False)

    CPU and gpu are not stuck or processing anymore however I still get the issue at the same range the following issue :

    Image

    Also My peak token generation increased drastically, which is good news haha.

    If I can help in anyway please, let me know :)

  12. Lufen34 commented on Sep 27, 2026

    @Lufen34
    Lufen34
    on Sep 27, 2026
    Author

    "please reopen this and paste that line. Closing for now."

    @Niko1221 Sorry I don't have the permission to reopen it :)

    edit: Good news is that now performance isn't droping over time, I am under constant 70tok/s 🥳

    I am trying to generate multiple time see if this bug happens a lot or not

  13. Lufen34 commented on Sep 27, 2026

    @Lufen34
    Lufen34
    on Sep 27, 2026
    Author

    It's okay problem is solved, got just an unlucky run I guess. Thank you for your work 💪

  14. Lufen34 commented on Sep 27, 2026

    @Lufen34
    Lufen34
    on Sep 27, 2026
    Author

    @Niko1221 Sorry to ping you, just wanted to let you know the same issue is back again :

    Image Image

    took me a while tho this time

  15. Niko1221 commented on Sep 27, 2026

    @Niko1221
    Niko1221
    on Sep 27, 2026
    Owner

    Did you update the engine when commited for #31 or is this on one prior?

  16. 25 remaining items

  17. Hugua700
    added a commit that references this issue on Oct 6, 2026

    serve: the prompt progress line says how far the engine is (N of M to…

    235cf4b
  18. anon761
    added 4 commits that reference this issue on Oct 6, 2026

    engine: fix a race in the CPU expert pool, and never hang silently ag…

    d17fe0c

    engine 0.1.13: about 2x faster long prompts; stall reports for issue N…

    4a87048

    The PCIe share by the copy kernel for every pack (Niko1221#31, Niko12…

    06ac2a0

    serve: the prompt progress line says how far the engine is (N of M to…

    d4741fd
  19. Niko1221
    added a commit that references this issue on Oct 8, 2026

    serve: the #29 watchdog waits (up to 10x, STRATA_WATCHDOG_IO_S) while…

    f2e07af
  20. MistVVK
    added a commit that references this issue on Oct 11, 2026

    serve: ファイルの段を OS が読んでいる間は、#29 の見張りが最大 10 倍まで待つ（STRATA_WATCHDOG_IO_S、N…

    c3824de
  21. lask3802
    added 2 commits that reference this issue on Oct 12, 2026

    Remote stage: a stuck worker ends as a stuck main process does; parki…

    1cf20d1

    Remote stage: a stuck worker ends as a stuck main process does; parki…

    758cbec
  22. giridhart
    added a commit that references this issue on Oct 15, 2026

    serve: the Niko1221#29 watchdog waits (up to 10x, STRATA_WATCHDOG_IO_…

    7419ba7
  23. svgop
    added a commit that references this issue on Oct 17, 2026

    F-225: planner formal 3v3 REVERSES the call - onepass arm M1 189.3 vs…

    f615286
  24. Rarebigshot17
    added a commit that references this issue on Oct 18, 2026

    WIP server: port 8101 in the examples; no run-<model>.bat or issue Ni…

    8775d95
  25. lask3802
    added a commit that references this issue on Oct 20, 2026

    Remote stage: a stuck worker ends as a stuck main process does; parki…

    f31c390
  26. nuke1984
    added a commit that references this issue on Oct 22, 2026

    bench+worklog: Option A closed - the E1/persist no-wait protocol re-t…

    6aff4a8
  27. kgbrah
    added a commit that references this issue on Oct 24, 2026

    Remote stage: a stuck worker ends as a stuck main process does; parki…

    ef046e0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions