[iamanishx]

Part of #671 (the "NVFP4 GGUF, still refused" item, token_embd half).

An NVFP4 GGUF whose token_embd is itself NVFP4 was refused: the GGUF embedding method looks rows up through the vendored ggml get_rows, which has a dequant case for every ggml type except NVFP4, so GgmlEmbeddingKernel.unusable_reason turned such tables away. Dense linears, experts and their .scale sidecars already work since #732; only the vocab table was missing.

• ggml-cuda/dequantize.cuh, ggml-cuda/getrows.cu: a pair-style dequantize_nvfp4 for k_get_rows and the GGML_TYPE_NVFP4 dispatch. The 16-value sub-block is the kernel's block, so each byte's nibbles land at j and j+8 as NVFP4 packs them, and any row length that is a multiple of 64 works (the k-quant get_rows path needs a multiple of 256).
• freetoken.patch, VERSION: the hunks for both files, in the patch's file order. The patch applies cleanly to the upstream commit (8c2134a) and reproduces the vendored tree byte for byte.
• layers/quantization/embedding/gguf.py: NVFP4 tables load. A table with a .scale is still refused: llama.cpp reads no tok_embd scale and asserts that a tied NVFP4 output has no sidecar scales, so applying one would diverge from the reference. An untied NVFP4 output.weight with a scale goes through the linear path from feat(gguf): apply the weight scales of nvfp4 gguf tensors #732 as before.
• tests/kernels/test_ggml.py: NVFP4 dequantize and get_rows against gguf-py's CPU dequantization, exact in fp32 / bf16 / fp16 over every fp4 code and UE4M3 scale code; embed and tied-head logits through GgmlEmbeddingKernel; kernel selection for scaled vs unscaled tables.

Tested with:

RTX PRO 6000 Blackwell Server Edition (sm_120) and A10 (sm_86, software UE4M3 decode), driver 610.57.04, torch 2.14.1+cu130, Triton 3.8.0, nvcc 13.0, ggml JIT-built on each GPU:

PYTHONPATH=python python -m pytest -q -p no:cacheprovider tests/kernels/test_ggml.py

24 passed on both. With main's getrows.cu, dequantize.cuh and embedding/gguf.py swapped back in, test_nvfp4_table_kernel_selection fails with "NVFP4 GGUF tables are not supported" and test_nvfp4_dequantize_and_rows with "unsupported src0 type: nvfp4".

[jason-fxz]

Thanks, the kernel and the loader change look right. Checked on an RTX PRO 6000 (sm_120), torch 2.14.1+cu130, Triton 3.8.0:

• dequantize_nvfp4 matches ggml's own CPU and CUDA NVFP4 dequantization (nibbles at j / j+8, ue4m3_to_fp32 already halved), and freetoken.patch applied to 8c2134a reproduces the vendored tree byte for byte.
• tests/kernels/test_ggml.py: 24 passed; with main's getrows.cu, dequantize.cuh and embedding/gguf.py swapped back, the 5 new cases fail with the messages in the description.
• Real checkpoints: there are public GGUFs of served architectures with an NVFP4 token_embd, so the last paragraph of the description can go. I ran three, one per layout:

file	token_embd	output	FreeToken serves	NVFP4 table vs exact F16 table
FreedomAISVR/Qwen3.5-9B-NVFP4-GGUF	NVFP4	Q6_K	yes	6/6 prompts token-identical
jan1k/Qwen3.5-9B-Uncensored-Genesis-NVFP4-GGUF (244 scaled linears)	NVFP4	NVFP4, no scale	yes	6/6 token-identical
WhiskyAKM/Gemma-4-12B-it-qat-NVFP4-GGUF (fast)	NVFP4	tied	yes	6/6 token-identical

The control is the same file with token_embd rewritten as F16 (every NVFP4 value is exact in f16; for the tied Gemma the original NVFP4 bytes are kept as output.weight, so the head runs the same kernel). Greedy, 200 tokens, thinking off, --cache-type naive.

One request, on the tests (rules in tests/README.md, "What earns a place here", merged after this branch's base): please keep only test_nvfp4_dequantize_and_rows with its _nvfp4_weights helper, and drop the other two tests along with the imports only they use. That test is the one that guards the new code: it checks against an independent reference, and a wrong nibble or scale index would otherwise go unnoticed. test_nvfp4_table_kernel_selection restates the two branches of unusable_reason. In test_nvfp4_table_embed_and_tied_logits, the embed half runs the same get_rows, and the logits half tests the NVFP4 GEMM from #732. Folding the kept test into test_q4_dequantize_and_rows as a second ggml type would also be fine.

[iamanishx]

@jason-fxz i have updated the PR . Thanks for the review