[Maphist0]

Overview

Add support for DiffuSpace's Clover-35B-A3B-Preview, in AR mode only.

• Self-speculative generation for Clover is split into a separate PR.

Additional information

Validated with BF16 CPU and H200 CUDA inference.

Validation:

CPU compile:

cmake -S . -B build-clover-cpu \
    -DGGML_CUDA=OFF \
    -DGGML_BLAS=OFF \
    -DLLAMA_CURL=OFF \
    -DLLAMA_BUILD_TESTS=OFF \
    -DLLAMA_BUILD_SERVER=OFF \
    -DCMAKE_BUILD_TYPE=Release

CUDA compile:

cmake -S . -B build-clover-cuda \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES=90a \
    -DLLAMA_CURL=OFF \
    -DLLAMA_BUILD_TESTS=OFF \
    -DLLAMA_BUILD_SERVER=OFF \
    -DCMAKE_BUILD_TYPE=Release

Ckpt convert:

python source/convert_hf_to_gguf.py \
    <clover_ckpt_path> \
    --outtype bf16 \
    --outfile clover-bf16.gguf

Launch ordinary AR inference:

build-clover-cuda/bin/llama-server \
    -m clover-bf16.gguf \
    -ngl 99 -c 8192 -fa on \
    --host 127.0.0.1 --port 8080

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, I've used GPT-6.1 Sol for majority of code generation under my review. Code generated by AI was reviewed before adding into the codebase.

[tsurumitsu]

I tested draft-self with Clover-35B-A3B-Preview (Q4_K_M GGUF) on an NVIDIA GB10 using the implementation from this PR.

I found that llama-server fails to initialize self-speculative decoding because params.speculative.draft.ctx_tgt is null.

The error is:

failed to initialize speculative decoding context:
draft-self requires a causal target context with memory,
a mask token and a positive draft capacity

It appears that has_spec only covers has_draft || spec_mtp, so the target context assignment inside that branch is skipped for draft-self.

I was able to fix this locally by adding the following assignment in tools/server/server-context.cpp, immediately before speculative decoding initialization:

params_base.speculative.draft.ctx_tgt = ctx_tgt;

After rebuilding, self-speculative decoding initialized successfully, and the server reported:

• Draft tokens generated: 72
• Draft tokens accepted: 50
• Acceptance rate: 69.4%

For reference, generation throughput was 13.93 tok/s without speculation versus 9.63 tok/s with draft-self and --spec-draft-n-max 2 in this short test. Thus, the initialization fix works, but I did not observe a performance improvement.

The local fix is committed as d57ef183c. I can provide the patch if useful.

[Maphist0]

I was able to fix this locally by adding the following assignment in tools/server/server-context.cpp, immediately before speculative decoding initialization:

params_base.speculative.draft.ctx_tgt = ctx_tgt;

Thank you @tsurumitsu very much for the help! This single-line fix has been committed.

[tsurumitsu]

Thanks for the fix! I retested the latest PR (cb09d4355) on my NVIDIA GB10.

1. Bug fix confirmed

The previous ctx_tgt initialization issue no longer reproduces. The server starts successfully with --spec-type draft-self, and token generation runs without the previous initialization error.

2. Performance results

Test environment:

• Hardware: NVIDIA GB10 (single device)
• Model: Clover-35B-A3B-Preview-Q4_K_M.gguf
• Build: CUDA (SM 121), Release
• Commit: cb09d4355
• Server settings: -ngl 99 -c 4096 -np 1
• Speculative settings: --spec-type draft-self --spec-draft-n-max 2
• Request: Identical Japanese prompt, temperature 0, max_tokens 128, non-streaming

Mode	Run 1	Run 2	Average
Normal decoding	13.54 tok/s	13.40 tok/s	13.47 tok/s
draft-self	8.08 tok/s	8.12 tok/s	8.10 tok/s

With draft-self enabled, measured decoding throughput was approximately 40% lower than normal decoding on GB10.

Additional observations:

• Draft acceptance rate: 82.56% (71/86 tokens, first run)
• Prompt caching was observed on repeated requests.
• All four non-streaming runs reached the 128-token limit (finish_reason: length).
• In the speculative runs, the token limit was reached during reasoning, before a final answer was produced.

These are preliminary measurements (two runs per mode), not a comprehensive benchmark.

3. Summary

The initialization fix is confirmed working on GB10.

However, in this configuration, draft-self remains significantly slower than normal decoding despite the relatively high draft acceptance rate observed in the first run.

Thanks again for the fix and your work on Clover support!

[ngxson]

this is nowhere near what AGENTS.md specified. ask your agent to read that before coding anything

[0wwafa]

Clover is still unsupported?

Loading model... |0.00.075.552 E llama_model_load: error loading model: error loading model hyperparameters: key not found in model: clover_moe.expert_shared_count
0.00.075.567 E llama_model_load_from_file_impl: failed to load model
0.00.075.610 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
-0.00.124.010 E llama_model_load: error loading model: error loading model hyperparameters: key not found in model: clover_moe.expert_shared_count
0.00.124.038 E llama_model_load_from_file_impl: failed to load model
0.00.124.049 E cmn  common_init_: failed to load model 'Clover-35B-A3B-Preview-IQ3_XXS.gguf'
0.00.124.058 E srv    load_model: failed to load model, 'Clover-35B-A3B-Preview-IQ3_XXS.gguf'
0.00.124.380 E srv  server_run_i: exiting due to model loading error
llama_server exited with code 1
Error: the server exited before becoming ready

[CISC]

Clover is still unsupported?

This PR implementation requires expert_shared_count metadata, but for some reason the official GGUF is missing that, it needs to be updated. @Maphist0