Skip to content

Streamline sample_grammar - #1606

Merged
LostRuins merged 1 commit into
LostRuins:concedo_experimentalfrom
Reithan:improve-grammar-sampler-efficiency
Jun 17, 2025
Merged

LostRuins merged 1 commit into
LostRuins:concedo_experimentalfrom
Reithan:improve-grammar-sampler-efficiency

Conversation

@Reithan

@Reithan Reithan commented Jun 17, 2025

Copy link
Copy Markdown
Part Old code Patched Cost difference (per candidate)
Helper-vector construction vector(count) ⇒ construct n default elements in each of the two vectors no elements constructed, only reserve −2 trivial ctor/destructor runs
First push_back on each vector triggers one full reallocation + move of the n default objects none (capacity was pre-reserved) −1 mem-copy of n objects
Grammar-reject pass iterates 2 n items (junk + real) iterates n items −n visits
Per-reject write write 8-byte logit in main array write 1-byte flag in rejects bandwidth / cache-line traffic drops
Final remove_if reads logit that’s already in the same cache line as tk reads rejects[i] (extra byte fetch) +1 cheap read

Estimated speed up should be 20-40% with bigger/heavier grammars, and should never be slower than current implementation.

@LostRuins LostRuins left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@LostRuins
LostRuins merged commit f07434f into LostRuins:concedo_experimental Jun 17, 2025
@Reithan
Reithan deleted the improve-grammar-sampler-efficiency branch June 17, 2025 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants