Skip to content

4TS fixed 80ts #1

Closed
Closed
4TS fixed 80ts#1

Description

@HsYn35
HsYn35
opened on Sep 25, 2026

model shards: Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf, Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf
index.txt: 1079 tensors, 302 served natively, 0 in extra.bin, arena 1.43 GiB
tokenizer/: vocab 248320, merges 247587, pre qwen35, specials {'tokenizer.ggml.eos_token_id': 248046, 'tokenizer.ggml.padding_token_id': 248044, 'tokenizer.ggml.bos_token_id': 248044}
[ok] model prepared: D:\LLAMA\Strata-0.1.2\packs\swift-iq3_xxs
The MTP draft layer (speculative decoding, ~2x faster output) comes from the original Qwen checkpoint:
only its ~5 GB of MTP tensors are downloaded.

D:\LLAMA\Strata-0.1.2.venv\Scripts\python.exe D:\LLAMA\Strata-0.1.2\tools\mtp_fetch.py fetch --out D:\LLAMA\Strata-0.1.2\mtp

MTP block in the BF16 checkpoint

31 tensors, 5.214 GB, in 28 shards.

tensor dtype shape MB
mtp.fc_embedding.weight BF16 2560x2560 13.1
mtp.fc_hidden.weight BF16 2560x2560 13.1
mtp.hyper_connection_mixer.hc_norm.weight BF16 10240 0.0
mtp.hyper_connection_mixer.input_mix_weight_down.weight BF16 320x10240 6.6
mtp.hyper_connection_mixer.input_mix_weight_up.weight BF16 10240x320 6.6
mtp.layers.0.attn_hyper_connection.block_inject_weight.weight BF16 4x10240 0.1
mtp.layers.0.attn_hyper_connection.hc_norm.weight BF16 10240 0.0
mtp.layers.0.attn_hyper_connection.input_mix_weight_down.weight BF16 320x10240 6.6
mtp.layers.0.attn_hyper_connection.input_mix_weight_up.weight BF16 10240x320 6.6
mtp.layers.0.mlp.experts.down_proj BF16 512x2560x640 1677.7
mtp.layers.0.mlp.experts.gate_up_proj BF16 512x1280x2560 3355.4
mtp.layers.0.mlp.gate.weight BF16 512x2560 2.6
mtp.layers.0.mlp.shared_expert.down_proj.weight BF16 2560x640 3.3
mtp.layers.0.mlp.shared_expert.gate_proj.weight BF16 640x2560 3.3
mtp.layers.0.mlp.shared_expert.up_proj.weight BF16 640x2560 3.3
mtp.layers.0.mlp.shared_expert_gate.weight BF16 1x2560 0.0
mtp.layers.0.mlp_hyper_connection.block_inject_weight.weight BF16 4x10240 0.1
mtp.layers.0.mlp_hyper_connection.hc_norm.weight BF16 10240 0.0
mtp.layers.0.mlp_hyper_connection.input_mix_weight_down.weight BF16 320x10240 6.6
mtp.layers.0.mlp_hyper_connection.input_mix_weight_up.weight BF16 10240x320 6.6
mtp.layers.0.self_attn.indexer.index_qk_proj.weight BF16 640x2560 3.3
mtp.layers.0.self_attn.indexer.k_layernorm.weight BF16 128 0.0
mtp.layers.0.self_attn.indexer.q_layernorm.weight BF16 128 0.0
mtp.layers.0.self_attn.k_norm.weight BF16 256 0.0
mtp.layers.0.self_attn.k_proj.weight BF16 512x2560 2.6
mtp.layers.0.self_attn.o_proj.weight BF16 2560x6144 31.5
mtp.layers.0.self_attn.q_norm.weight BF16 256 0.0
mtp.layers.0.self_attn.q_proj.weight BF16 12288x2560 62.9
mtp.layers.0.self_attn.v_proj.weight BF16 512x2560 2.6
mtp.pre_fc_norm_embedding.weight BF16 2560 0.0
mtp.pre_fc_norm_hidden.weight BF16 10240 0.0

mtp.fc_embedding.weight 100%
mtp.fc_hidden.weight 100%
mtp.hyper_connection_mixer.hc_norm.weight 100%
mtp.hyper_connection_mixer.input_mix_weight_down.weight 100%
mtp.hyper_connection_mixer.input_mix_weight_up.weight 100%
mtp.layers.0.attn_hyper_connection.block_inject_weight.weight 100%
mtp.layers.0.attn_hyper_connection.hc_norm.weight 100%
mtp.layers.0.attn_hyper_connection.input_mix_weight_down.weight 100%
mtp.layers.0.attn_hyper_connection.input_mix_weight_up.weight 100%
mtp.layers.0.mlp.experts.down_proj 4%
mtp.layers.0.mlp.experts.down_proj 8%
mtp.layers.0.mlp.experts.down_proj 12%
mtp.layers.0.mlp.experts.down_proj 16%
mtp.layers.0.mlp.experts.down_proj 20%
mtp.layers.0.mlp.experts.down_proj 24%
mtp.layers.0.mlp.experts.down_proj 28%
mtp.layers.0.mlp.experts.down_proj 32%
mtp.layers.0.mlp.experts.down_proj 36%
mtp.layers.0.mlp.experts.down_proj 40%
mtp.layers.0.mlp.experts.down_proj 44%
mtp.layers.0.mlp.experts.down_proj 48%
mtp.layers.0.mlp.experts.down_proj 52%
mtp.layers.0.mlp.experts.down_proj 56%
mtp.layers.0.mlp.experts.down_proj 60%
mtp.layers.0.mlp.experts.down_proj 64%
mtp.layers.0.mlp.experts.down_proj 68%
mtp.layers.0.mlp.experts.down_proj 72%
mtp.layers.0.mlp.experts.down_proj 76%
mtp.layers.0.mlp.experts.down_proj 80%
mtp.layers.0.mlp.experts.down_proj 84%
mtp.layers.0.mlp.experts.down_proj 88%
mtp.layers.0.mlp.experts.down_proj 92%
mtp.layers.0.mlp.experts.down_proj 96%
mtp.layers.0.mlp.experts.down_proj 100%
mtp.layers.0.mlp.experts.gate_up_proj 2%
mtp.layers.0.mlp.experts.gate_up_proj 4%
mtp.layers.0.mlp.experts.gate_up_proj 6%
mtp.layers.0.mlp.experts.gate_up_proj 8%
mtp.layers.0.mlp.experts.gate_up_proj 10%
mtp.layers.0.mlp.experts.gate_up_proj 12%
mtp.layers.0.mlp.experts.gate_up_proj 14%
mtp.layers.0.mlp.experts.gate_up_proj 16%
mtp.layers.0.mlp.experts.gate_up_proj 18%
mtp.layers.0.mlp.experts.gate_up_proj 20%
mtp.layers.0.mlp.experts.gate_up_proj 22%
mtp.layers.0.mlp.experts.gate_up_proj 24%
mtp.layers.0.mlp.experts.gate_up_proj 26%
mtp.layers.0.mlp.experts.gate_up_proj 28%
mtp.layers.0.mlp.experts.gate_up_proj 30%
mtp.layers.0.mlp.experts.gate_up_proj 32%
mtp.layers.0.mlp.experts.gate_up_proj 34%
mtp.layers.0.mlp.experts.gate_up_proj 36%
mtp.layers.0.mlp.experts.gate_up_proj 38%
mtp.layers.0.mlp.experts.gate_up_proj 40%
mtp.layers.0.mlp.experts.gate_up_proj 42%
mtp.layers.0.mlp.experts.gate_up_proj 44%
mtp.layers.0.mlp.experts.gate_up_proj 46%
mtp.layers.0.mlp.experts.gate_up_proj 48%
mtp.layers.0.mlp.experts.gate_up_proj 50%
mtp.layers.0.mlp.experts.gate_up_proj 52%
mtp.layers.0.mlp.experts.gate_up_proj 54%
mtp.layers.0.mlp.experts.gate_up_proj 56%
mtp.layers.0.mlp.experts.gate_up_proj 58%
mtp.layers.0.mlp.experts.gate_up_proj 60%
mtp.layers.0.mlp.experts.gate_up_proj 62%
mtp.layers.0.mlp.experts.gate_up_proj 64%
mtp.layers.0.mlp.experts.gate_up_proj 66%
mtp.layers.0.mlp.experts.gate_up_proj 68%
mtp.layers.0.mlp.experts.gate_up_proj 70%
mtp.layers.0.mlp.experts.gate_up_proj 72%
mtp.layers.0.mlp.experts.gate_up_proj 74%
mtp.layers.0.mlp.experts.gate_up_proj 76%
mtp.layers.0.mlp.experts.gate_up_proj 78%
mtp.layers.0.mlp.experts.gate_up_proj 80%
mtp.layers.0.mlp.experts.gate_up_proj 82%
mtp.layers.0.mlp.experts.gate_up_proj 84%
mtp.layers.0.mlp.experts.gate_up_proj 86%
mtp.layers.0.mlp.experts.gate_up_proj 88%
mtp.layers.0.mlp.experts.gate_up_proj 90%
mtp.layers.0.mlp.experts.gate_up_proj 92%
mtp.layers.0.mlp.experts.gate_up_proj 94%
mtp.layers.0.mlp.experts.gate_up_proj 96%
mtp.layers.0.mlp.experts.gate_up_proj 98%
mtp.layers.0.mlp.experts.gate_up_proj 100%
mtp.layers.0.mlp.gate.weight 100%
mtp.layers.0.mlp.shared_expert.down_proj.weight 100%
mtp.layers.0.mlp.shared_expert.gate_proj.weight 100%
mtp.layers.0.mlp.shared_expert.up_proj.weight 100%
mtp.layers.0.mlp.shared_expert_gate.weight 100%
mtp.layers.0.mlp_hyper_connection.block_inject_weight.weight 100%
mtp.layers.0.mlp_hyper_connection.hc_norm.weight 100%
mtp.layers.0.mlp_hyper_connection.input_mix_weight_down.weight 100%
mtp.layers.0.mlp_hyper_connection.input_mix_weight_up.weight 100%
mtp.layers.0.self_attn.indexer.index_qk_proj.weight 100%
mtp.layers.0.self_attn.indexer.k_layernorm.weight 100%
mtp.layers.0.self_attn.indexer.q_layernorm.weight 100%
mtp.layers.0.self_attn.k_norm.weight 100%
mtp.layers.0.self_attn.k_proj.weight 100%
mtp.layers.0.self_attn.o_proj.weight 100%
mtp.layers.0.self_attn.q_norm.weight 100%
mtp.layers.0.self_attn.q_proj.weight 100%
mtp.layers.0.self_attn.v_proj.weight 100%
mtp.pre_fc_norm_embedding.weight 100%
mtp.pre_fc_norm_hidden.weight 100%

D:\LLAMA\Strata-0.1.2.venv\Scripts\python.exe D:\LLAMA\Strata-0.1.2\tools\mtp_pack.py --src D:\LLAMA\Strata-0.1.2\mtp --experts q2_0 --out D:\LLAMA\Strata-0.1.2\mtp\mtp-q2_0.gguf
mtp.fc_embedding.weight: BF16 13.1 MB
mtp.fc_hidden.weight: BF16 13.1 MB
mtp.hyper_connection_mixer.hc_norm.weight: F32 0.0 MB
mtp.hyper_connection_mixer.input_mix_weight_down.weight: BF16 6.6 MB
mtp.hyper_connection_mixer.input_mix_weight_up.weight: BF16 6.6 MB
mtp.layers.0.attn_hyper_connection.block_inject_weight.weight: BF16 0.1 MB
mtp.layers.0.attn_hyper_connection.hc_norm.weight: F32 0.0 MB
mtp.layers.0.attn_hyper_connection.input_mix_weight_down.weight: BF16 6.6 MB
mtp.layers.0.attn_hyper_connection.input_mix_weight_up.weight: BF16 6.6 MB
mtp.layers.0.mlp.experts.down_proj: q2_0 235.9 MB
mtp.layers.0.mlp.experts.gate_up_proj: q2_0 471.9 MB
mtp.layers.0.mlp.gate.weight: BF16 2.6 MB
mtp.layers.0.mlp.shared_expert.down_proj.weight: BF16 3.3 MB
mtp.layers.0.mlp.shared_expert.gate_proj.weight: BF16 3.3 MB
mtp.layers.0.mlp.shared_expert.up_proj.weight: BF16 3.3 MB
mtp.layers.0.mlp.shared_expert_gate.weight: BF16 0.0 MB
mtp.layers.0.mlp_hyper_connection.block_inject_weight.weight: BF16 0.1 MB
mtp.layers.0.mlp_hyper_connection.hc_norm.weight: F32 0.0 MB
mtp.layers.0.mlp_hyper_connection.input_mix_weight_down.weight: BF16 6.6 MB
mtp.layers.0.mlp_hyper_connection.input_mix_weight_up.weight: BF16 6.6 MB
mtp.layers.0.self_attn.indexer.index_qk_proj.weight: BF16 3.3 MB
mtp.layers.0.self_attn.indexer.k_layernorm.weight: F32 0.0 MB
mtp.layers.0.self_attn.indexer.q_layernorm.weight: F32 0.0 MB
mtp.layers.0.self_attn.k_norm.weight: F32 0.0 MB
mtp.layers.0.self_attn.k_proj.weight: BF16 2.6 MB
mtp.layers.0.self_attn.o_proj.weight: BF16 31.5 MB
mtp.layers.0.self_attn.q_norm.weight: F32 0.0 MB
mtp.layers.0.self_attn.q_proj.weight: BF16 62.9 MB
mtp.layers.0.self_attn.v_proj.weight: BF16 2.6 MB
mtp.pre_fc_norm_embedding.weight: F32 0.0 MB
mtp.pre_fc_norm_hidden.weight: F32 0.0 MB
wrote D:\LLAMA\Strata-0.1.2\mtp\mtp-q2_0.gguf: 0.889 GB of tensor data
D:\LLAMA\Strata-0.1.2.venv\Scripts\python.exe D:\LLAMA\Strata-0.1.2\tools\mtp_rt.py --gguf D:\LLAMA\Strata-0.1.2\mtp\mtp-q2_0.gguf --out D:\LLAMA\Strata-0.1.2\mtp\rt
experts.bin 707788800 B, dense.bin 116099072 B, 29 tensors -> D:\LLAMA\Strata-0.1.2\mtp\rt
fc_embedding.weight q8_0 2560 2560 0 6963200
fc_hidden.weight q8_0 2560 2560 6963200 6963200
hyper_connection_mixer.hc_norm.weight f32 1 10240 13926400 40960
hyper_connection_mixer.input_mix_weight_down.weight bf16 320 10240 13967360 6553600
hyper_connection_mixer.input_mix_weight_up.weight bf16 10240 320 20520960 6553600
attn_hyper_connection.block_inject_weight.weight bf16 4 10240 27074560 81920
attn_hyper_connection.hc_norm.weight f32 1 10240 27156480 40960
attn_hyper_connection.input_mix_weight_down.weight bf16 320 10240 27197440 6553600
attn_hyper_connection.input_mix_weight_up.weight bf16 10240 320 33751040 6553600
mlp.gate.weight bf16 512 2560 40304640 2621440
mlp.shared_expert.down_proj.weight q8_0 2560 640 42926080 1740800
mlp.shared_expert.gate_proj.weight q8_0 640 2560 44666880 1740800
mlp.shared_expert.up_proj.weight q8_0 640 2560 46407680 1740800
mlp.shared_expert_gate.weight bf16 1 2560 48148480 5120
mlp_hyper_connection.block_inject_weight.weight bf16 4 10240 48153600 81920
mlp_hyper_connection.hc_norm.weight f32 1 10240 48235520 40960
mlp_hyper_connection.input_mix_weight_down.weight bf16 320 10240 48276480 6553600
mlp_hyper_connection.input_mix_weight_up.weight bf16 10240 320 54830080 6553600
self_attn.indexer.index_qk_proj.weight q8_0 640 2560 61383680 1740800
self_attn.indexer.k_layernorm.weight f32 1 128 63124480 512
self_attn.indexer.q_layernorm.weight f32 1 128 63124992 512
self_attn.k_norm.weight f32 1 256 63125504 1024
self_attn.k_proj.weight q8_0 512 2560 63126528 1392640
self_attn.o_proj.weight q8_0 2560 6144 64519168 16711680
self_attn.q_norm.weight f32 1 256 81230848 1024
self_attn.q_proj.weight q8_0 12288 2560 81231872 33423360
self_attn.v_proj.weight q8_0 512 2560 114655232 1392640
pre_fc_norm_embedding.weight f32 1 2560 116047872 10240
pre_fc_norm_hidden.weight f32 1 10240 116058112 40960
[ok] MTP draft layer: D:\LLAMA\Strata-0.1.2\mtp\rt

=== Step 7: writing the start script ===
[ok] start script: run-swift-iq3_xxs.bat

All set.
API (OpenAI): http://127.0.0.1:8080/v1 (any API key; model name: anything)
API (Anthropic): http://127.0.0.1:8080/v1/messages
Next time: just run START-HERE.bat (or run-swift-iq3_xxs.bat) - it starts right away
Images: send them in the chat page, in chat.py (/image ) or over the API

Starting swift-1.5-iq3_xxs (loads 34-43 GB into RAM: 30-90 s). Close this window to stop it.
loading the vision encoder ...
loading the model (the first start takes a minute or two) ...
ready: http://127.0.0.1:8080/v1 (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, context 65536 tokens, images on)
open http://127.0.0.1:8080/ in a browser to chat; close this window to stop the model
[strata] thinking: 17 of max 8192 tokens, 4.1 tok/s, 15 s
[strata] done: 55 tokens in 25 s (stop)
[strata] answering: 63 of max 8192 tokens, 5.1 tok/s, 15 s
[strata] answering: 120 of max 8192 tokens, 4.3 tok/s, 31 s
[strata] answering: 187 of max 8192 tokens, 4.3 tok/s, 46 s
[strata] answering: 188 of max 8192 tokens, 2.6 tok/s, 75 s
[strata] answering: 396 of max 8192 tokens, 4.5 tok/s, 90 s
[strata] answering: 456 of max 8192 tokens, 4.4 tok/s, 106 s
[strata] done: 473 tokens in 109 s (stop)

5080 128gb 9950x3d what problem

fixed problem hdd change
RTX5080

Strata - Qwen3.8-Flash-Next on a normal PC (NVIDIA GPU + system RAM + CPU)

Starting swift-1.5-iq3_xxs (loads 34-43 GB into RAM: 30-90 s). Close this window to stop it.
loading the vision encoder ...
loading the model (the first start takes a minute or two) ...
ready: http://127.0.0.1:8080/v1 (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, context 65536 tokens, images on)
open http://127.0.0.1:8080/ in a browser to chat; close this window to stop the model
[strata] done: 80 tokens in 2 s (stop)
[strata] done: 578 tokens in 9 s (stop)
[strata] answering: 797 of max 8192 tokens, 60.5 tok/s, 15 s
[strata] done: 1674 tokens in 28 s (stop)
[strata] answering: 758 of max 8192 tokens, 70.3 tok/s, 15 s
[strata] answering: 2174 of max 8192 tokens, 84.2 tok/s, 30 s
[strata] answering: 3365 of max 8192 tokens, 82.4 tok/s, 45 s
[strata] answering: 4577 of max 8192 tokens, 81.9 tok/s, 60 s
[strata] done: 4690 tokens in 62 s (stop)
[strata] thinking: 134 of max 8192 tokens, 52.6 tok/s, 15 s
[strata] answering: 1151 of max 8192 tokens, 65.5 tok/s, 30 s
[strata] answering: 2530 of max 8192 tokens, 77.7 tok/s, 45 s
[strata] answering: 3683 of max 8192 tokens, 77.4 tok/s, 60 s
[strata] answering: 4878 of max 8192 tokens, 77.9 tok/s, 75 s
[strata] answering: 6072 of max 8192 tokens, 78.3 tok/s, 90 s
[strata] done: 6431 tokens in 96 s (stop)
[strata] reading the prompt: 12599 tokens, 20 s so far
[strata] thinking: 770 of max 8192 tokens, 60.9 tok/s, 35 s

Activity

  1. HsYn35
    changed the title [-]4TS[/-] [+]4TS fixed 80ts[/+] on Sep 25, 2026
  2. HsYn35
    closed this as completedon Sep 25, 2026
  3. xyzzing
    added a commit that references this issue on Sep 29, 2026

    strata-rocm tune: paired interleaved tuning controller + the CUDA->HI…

    242788c
  4. maxious
    added a commit that references this issue on Oct 4, 2026

    sycl: measure the oneDNN avenue (exp 24) and link it opt-in - the fus…

    9461fbe
  5. AdrianBM96
    added 2 commits that reference this issue on Oct 4, 2026

    docs/fork: documentar el port del fork y el estado de despliegue

    2192f22

    docs/fork: resultados del port Niko1221#1 y Niko1221#3 (ambos SIN gan…

    bd53206
  6. rwkeyes
    added 2 commits that reference this issue on Oct 5, 2026

    ports/vulkan: GDN class-A Niko1221#1-3 (conv, l2_norm, beta_gate) und…

    692690e

    ports/vulkan: THE PER-FAMILY TRIVIAL-WORK ABLATION, MEASURED ON THE A…

    9cd885e
  7. TheHubUser4860938
    added a commit that references this issue on Oct 13, 2026

    audit fix: the user's real name was in git - the classifier now takes…

    8e7e20c
  8. FlareP1
    added 4 commits that reference this issue on Oct 13, 2026

    Prompt cache: checkpoints where agents' prompts part - where a prompt…

    2e26dd0

    serve: a request that declares no tools gets the model's text verbati…

    101fff6

    POST /v1/choose: a choice among single-token options in one forward p…

    fc79347

    Merge pull request Niko1221#1 from IvanAdriazola, its content in 2e26dd0

    e3727cb
  9. sumomomomomo
    added a commit that references this issue on Oct 14, 2026

    conversation tests: a use-after-free in the report check, and a stagi…

    6b15565
  10. svgop
    added 3 commits that reference this issue on Oct 17, 2026

    T8: goal-round ledger opened — F-214..F-218 banked, F-218-informed pr…

    0cc9747

    F-220: smoothness knob Niko1221#1 — prewarm pass wired into all batte…

    a5ef33f

    T15C overlooked-wins dossier: full 417-knob/549-site env inventory wi…

    4745f11
  11. DamirLukina88
    added a commit that references this issue on Oct 17, 2026

    D.L. AI: audit round 30 - message uids and job ids must be real numbe…

    549b70a
  12. svgop
    added a commit that references this issue on Oct 17, 2026

    T17F integrated: pack re-quant is the new Niko1221#1 lever (+6-10% M1…

    d2fa79c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions