Repository navigation
[Issue]: Subpar Gemma 4 E4B output quality on FLM #641
Description
Required system details:
PS C:\WINDOWS\system32> (Get-WmiObject Win32_OperatingSystem).Version
10.0.26200
PS C:\WINDOWS\system32> (Get-WmiObject win32_Processor).Name
AMD Ryzen AI 9 365 w/ Radeon 880M
PS C:\WINDOWS\system32> (Get-WmiObject win32_VideoController).Name
AMD Radeon(TM) 880M Graphics
Hi, I wanted to report that Gemma 4 output quality with the E4B model is pretty clearly subpar compared to other quants of similar size (in this case Unsloth's gemma-4-E4B-it-qat-UD-Q4_K_XL). This may or may not be due to the outdated template or something to do with quantization, but in any ase it's a bit of a shame since normally it would be one of the preferred models to use for small tool-use applications in my case. Since it's a bit hard to just showcase what I'm talking about in one go, I made a small repository that can be used to easily reproduce the issue:
https://github.com/BoneHorror/flm_g4e4b_quality_test
But the tl;dr of what one could expect is:
Scenario:
The model, with previous conversation history, is asked to find balance changes made to "Marines". It does not have earlier knowledge of Marines or other specifics, but Gemma 4 E4B can typically infer most of what's needed to complete this query.
Answers -
When FLM is generating: (G4 E4B)
<tool>Model is listing docs in: balance_changes
I've listed the files in the `balance_changes` directory. It seems to be organized by race, with files for Terran (`terran.txt`), UED (`ued.txt`), Zerg (`zerg.txt`), and Protoss (`protoss.txt`).
To find the specific balance changes made to Marines, I would need you to tell me which race group (Terran, UED, Zerg, or Protoss) Marines belong to in this documentation structure so I can read the correct file for you!
When llama-server is generating with G4 E4B Q4_K_XL:
<tool>Model is listing docs in: balance_changes
<tool>Model is reading: balance_changes/terran.txt
Reading /home/user/Repositories/discord_bot/docs/balance_changes/terran.txt
Based on the `terran.txt` documentation, here are the balance changes made to Marines:
* **Cost:** Decreased to **40** (from 50).
* **Train Time:** Decreased to **320 frames** (from 360).
* **Note:** Terran units no longer include Medics; these roles were moved to the UED faction.
This is a typical scenario, you can find more examples in the repo linked above. You can see that via FLM the model stopped short of completing the query properly, even though as seen in the other case it could absolutely find the information.
I have previously tried to report it on the project's Discord but ended up not having enough time to write it up properly. I hope someone can take a look here, as well.
Activity
Atomic-Germ commentedon Aug 14, 2026
Can you compare it to a q4_1? That's where the FLM one is from I'm pretty sure.
BoneHorror commentedon Sep 2, 2026
Sorry I took a while.
I re-tested against Q4_1, though I would be worried more about model size comparisons (since ultimately if I can have seemingly much higher quality at a lower size, doesn't it feel like there's something mathematically wrong with quant/format?) still. Also it seems to me that you guys figured out non Q4_0/Q4_1 formats as seen in d8b9b44 docs update
Here are example results for the retest:
(I used Unsloth's Q4_1 https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/blob/main/gemma-4-E4B-it-Q4_1.gguf )
Evidently FLM version is still 'lazier'. I re-ran 5 times only but each time results were similar to the original test, FLM nearly always only makes 1 tool call and gives up but the Q4_1 model on llama does two and gives a better/more accurate answer.
I hope I don't come off as being too obnoxious, I genuinely want to use FLM but it just seems that there's something amiss with models that I know can do a bit more normally. I'm trying to write up an issue for weird artifacts during generation with new Gemma 4 12B too.
[-]Subpar Gemma 4 E4B output quality on FLM[/-][+][Issue]: Subpar Gemma 4 E4B output quality on FLM[/+]