Skip to content

Store FSE statePtr and force-inline ZSTD_initFseState - #4938

Open
conNULL wants to merge 1 commit into
meta:devfrom
conNULL:fse-stateptr-opt
Open

conNULL wants to merge 1 commit into
meta:devfrom
conNULL:fse-stateptr-opt

Conversation

@conNULL

@conNULL conNULL commented Nov 12, 2026

Copy link
Copy Markdown

What and why

In ZSTD_decodeSequence, table + state is recomputed across all three FSE state tables (stateLL, stateML, stateOffb) on every sequence while ZSTD_initFseState is non-forced-inline (unlike ZSTD_updateFseStateWithDInfo).

Replace ZSTD_fseState::state with a precomputed entry pointer (statePtr = table + nextState + lowBits) and mark ZSTD_initFseState FORCE_INLINE_TEMPLATE so &seqState does not address-escape to an out-of-line call on older GCC (which prevented SRA and spilled bitContainer). (Part 1 of a 3-part ZSTD_decodeSequence series.)

Benchmark results

  1. Upstream make (-O3) across bare-metal x86-64 and aarch64 (decompression MB/s vs dev eb1c858, pinned single core, median of 7 interleaved -i2s runs across built-in 10 MB synthetic -b1/-b3 and 12-file Silesia 212 MB zstd -d -b at -1/-3/-9):

Results: AMD EPYC 7B13 64-Core Processor (x86_64, plain make -O3, 7 interleaved rounds, -i2s/round)

Compiler Variant Lorem 10MB -1 Lorem 10MB -3 Silesia -1 Silesia -3 Silesia -9
gcc (Debian 15.2.0-14) 15.2.0 dev 1435.4 MB/s 1246.4 MB/s 1288.6 MB/s 1188.2 MB/s 1278.6 MB/s
gcc (Debian 15.2.0-14) 15.2.0 pr1 1478.6 (+3.01%) 1305.1 (+4.71%) 1348.6 (+4.66%) 1254.0 (+5.54%) 1314.4 (+2.80%)
Debian clang version 19.1.7 (10.1+build1) dev 1170.2 MB/s 1060.8 MB/s 1189.1 MB/s 1078.3 MB/s 1178.8 MB/s
Debian clang version 19.1.7 (10.1+build1) pr1 1257.8 (+7.49%) 1116.9 (+5.29%) 1230.0 (+3.44%) 1120.0 (+3.87%) 1214.3 (+3.01%)

Results: Neoverse-V2 (aarch64, plain make -O3, 7 interleaved rounds, -i2s/round)

Compiler Variant Lorem 10MB -1 Lorem 10MB -3 Silesia -1 Silesia -3 Silesia -9
aarch64-linux-gnu-gcc-15 (Debian 15.2.0-4) 15.2.0 dev 1137.2 MB/s 1047.9 MB/s 1394.6 MB/s 1212.8 MB/s 1260.7 MB/s
aarch64-linux-gnu-gcc-15 (Debian 15.2.0-4) 15.2.0 pr1 1144.1 (+0.61%) 1053.6 (+0.54%) 1398.9 (+0.31%) 1217.8 (+0.41%) 1262.3 (+0.13%)
Debian clang version 19.1.7 (10.1+build1) dev 1412.7 MB/s 1326.5 MB/s 1435.2 MB/s 1324.6 MB/s 1363.3 MB/s
Debian clang version 19.1.7 (10.1+build1) pr1 1642.0 (+16.23%) 1547.4 (+16.65%) 1519.8 (+5.89%) 1441.5 (+8.83%) 1447.4 (+6.17%)

Results: Intel(R) Xeon(R) Platinum 8173M CPU @ 2.00GHz (x86_64, plain make -O3, 7 interleaved rounds, -i2s/round)

Compiler Variant Lorem 10MB -1 Lorem 10MB -3 Silesia -1 Silesia -3 Silesia -9
gcc (Debian 15.2.0-14) 15.2.0 dev 1100.0 MB/s 956.6 MB/s 1079.0 MB/s 915.8 MB/s 924.8 MB/s
gcc (Debian 15.2.0-14) 15.2.0 new 1229.3 (+11.75%) 1062.7 (+11.09%) 1153.5 (+6.90%) 997.1 (+8.88%) 1007.6 (+8.95%)
Debian clang version 19.1.7 (10.1+build1) dev 994.0 MB/s 867.9 MB/s 1021.7 MB/s 850.8 MB/s 845.1 MB/s
Debian clang version 19.1.7 (10.1+build1) new 992.0 (-0.20%) 869.2 (+0.15%) 1027.8 (+0.60%) 855.0 (+0.49%) 854.3 (+1.09%)
  1. Fleetbench Zstd decompression suite (https://github.com/google/fleetbench 11 production corpus workloads, Clang + FDO, 20 runs on dedicated benchmark hosts):
    • AMD EPYC Milan (x86-64): -3.90% geomean cycles (-2.76% Fleet)
    • ARM Neoverse V2 (AArch64): -3.55% geomean cycles (-3.58% Fleet)
    • Intel Skylake (x86-64): +0.75% geomean cycles (+0.27% time, neutral)

Checklist

  • I searched open pull requests and issues: this problem is not already being addressed.
  • Bug fix: it comes with a reproducer or a test that fails without the fix.
  • Performance change: numbers from repeated runs are included, with CPU, compiler, flags and data.
  • I stated below whether AI or other automated tools were used.

Tools used

An AI coding assistant (Gemini) was used to help prototype code variants, inspect compiler disassembly, and collect benchmark measurements; all code changes and benchmark results were reviewed and verified by the author.

Replace `ZSTD_fseState::state` (`table + state` recomputed across all three FSE
tables on every sequence) with a precomputed entry pointer (`statePtr`), and
mark `ZSTD_initFseState` `FORCE_INLINE_TEMPLATE` (matching
`ZSTD_updateFseStateWithDInfo`) so `&seqState` does not address-escape to an
out-of-line call on older GCC.
@meta-cla meta-cla Bot added the CLA Signed label Nov 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant