Repository navigation
Base the tokenizer API on source offsets #153569
Description
The tokenizer currently stores many source positions as pointers into buffers that can move. Reallocating the input means rebasing all these pointers, and missing one is very easy, especially with incremental input and f-strings.
I think tokenizer positions should be offsets into the decoded source instead:
typedef Py_ssize_t TokenizerOffset;
typedef struct {
TokenizerOffset start;
TokenizerOffset end;
} TokenizerSpan;The main ideas would be:
- One source object owns the decoded text.
- The cursor only stores its current offset and line boundaries.
- Tokens, errors and f-string state use offset spans instead of pointers.
- pegen and
_tokenizeask the source for a view or copy of a span. - Sequential tokenization keeps the current line in the cursor, while uncommon line lookups can use a small sparse index.
- Normal parsing can keep one contiguous buffer, while
tokenize(readline)can eventually use reclaimable chunks.
For incremental tokenization, new decoded input would be appended only when the cursor needs more data. Existing offsets would remain valid even if the underlying storage moves. Once tokenize has returned copied token and line strings, old input could be discarded when no active token, cursor, f-string frame or error still refers to it.
This is very nice because it separates source storage from tokenizer state, removes pointer rebasing, and means consumers no longer need to access tokenizer internals.
Linked PRs
- gh-153569: Split tokenizer lexer into focused files #153585
- gh-153569: Add tokenizer source and cursor primitives #153587
- gh-153569: Replace tokenizer input readers #156472
- gh-153569: return token spans and unify decoded source storage #156482
- gh-153569: centralize formatted-string state and source spans #156484
- gh-153569: hide tokenizer state behind token and source operations #156654
- gh-153569: consolidate tokenizer input, layout, and diagnostics #157055
- gh-153569: Simplify tokenizer state #158124
- gh-153569: Simplify tokenizer source views and reader ownership #158849
Activity
pythongh-153569: fix tokenizer includes in external builds
pythongh-153569: fix cross-platform tokenizer builds
pythongh-153569: avoid retaining duplicate reader input
Merge branch 'pythongh-153569-tokenizer-reader-cutover' into pythongh…
pythongh-153569: tighten tokenizer iterator cleanup
pythongh-153569: preserve tolerant f-string delimiter state
Merge branch 'pythongh-153569-tokenizer-offset-state' into pythongh-1…
Merge branch 'pythongh-153569-tokenizer-offset-state' into pythongh-1…
pythongh-153569: strengthen tokenizer validation
pythongh-153569: preserve malformed f-string delimiter state
pythongh-153569: speed up UTF-8 readline decoding
Merge branch 'pythongh-153569-tokenizer-reader-cutover' into pythongh…
Merge branch 'pythongh-153569-tokenizer-offset-state' into pythongh-1…
pythongh-153569: complete tokenizer validation coverage
pythongh-153569: Add tokenizer source and cursor primitives (python#1…
pythongh-153569: streamline tokenizer hot paths
gh-153569: Unify tokenizer input readers (#156472)
pythongh-153569: move tokenizer state to source offsets
pythongh-153569: track reader lines with the source cursor
pythongh-153569: retain decoded reader input in source text
pythongh-153569: cover reader source relocation
pythongh-153569: bound tokenizer input storage
pythongh-153569: return tokenizer tokens as source spans
pythongh-153569: remove unused tokenizer cursor support
pythongh-153569: store f-string state as source spans
pythongh-153569: store f-string state as source spans
pythongh-153569: track f-string comments as source spans
pythongh-153569: remove unused tokenizer cursor support
pythongh-153569: store f-string state as source spans
pythongh-153569: track f-string comments as source spans
pythongh-153569: simplify f-string span handling
pythongh-153569: streamline tokenizer buffer hot paths
pythongh-153569: remove unused tokenizer cursor support
pythongh-153569: store f-string state as source spans
pythongh-153569: track f-string comments as source spans
pythongh-153569: simplify f-string span handling
pythongh-153569: keep token span invariants explicit
pythongh-153569: preserve comments after not-equal expressions
pythongh-153569: remove duplicated tokenizer state
pythongh-153569: simplify tokenizer mode state
pythongh-153569: simplify tokenizer source storage
pythongh-153569: preserve debug text in format specs
pythongh-153569: keep f-string comments with their mode
pythongh-153569: fix multiline continuation error offsets
pythongh-153569: simplify formatted-string state
pythongh-153569: handle carriage returns after comments
pythongh-153569: test nesting without parser recursion
pythongh-153569: finish nested formatted-string expressions
pythongh-153569: simplify tokenizer input state
pythongh-153569: store f-string state as source spans
pythongh-153569: remove unused tokenizer cursor support
pythongh-153569: track f-string comments as source spans
pythongh-153569: simplify f-string span handling
pythongh-153569: keep token span invariants explicit
pythongh-153569: preserve comments after not-equal expressions
pythongh-153569: remove duplicated tokenizer state
pythongh-153569: simplify tokenizer mode state
pythongh-153569: simplify tokenizer source storage
pythongh-153569: preserve debug text in format specs
pythongh-153569: keep f-string comments with their mode
pythongh-153569: fix multiline continuation error offsets
pythongh-153569: simplify formatted-string state
pythongh-153569: handle carriage returns after comments
pythongh-153569: test nesting without parser recursion
pythongh-153569: finish nested formatted-string expressions
pythongh-153569: simplify tokenizer input state
pythongh-153569: store formatted-string text as source spans
pythongh-153569: make the reader own tokenizer input state
pythongh-153569: remove redundant reader counters
pythongh-153569: share string prefix and quote scanning
pythongh-153569: keep only active formatted-string state
pythongh-153569: normalize prepared tokenizer input once
pythongh-153569: bound tokenizer input storage
pythongh-153569: return tokenizer tokens as source spans
pythongh-153569: remove the unused tokenizer cursor and line index
pythongh-153569: store formatted-string text as source spans
pythongh-153569: derive token locations from the live buffer
pythongh-153569: derive tokenizer failures from done state
pythongh-153569: keep buffer relocation inside the reader
pythongh-153569: use tokenizer source for parser diagnostics
pythongh-153569: make the reader own interactive state
pythongh-153569: initialize file encoding without tokenizing
pythongh-153569: remove redundant reader counters
pythongh-153569: share string prefix and quote scanning
pythongh-153569: let the reader construct tokenizer state
pythongh-153569: keep only active formatted-string state
pythongh-153569: normalize prepared tokenizer input once
pythongh-153569: store formatted-string text as source spans
pythongh-153569: derive token locations from the live buffer
pythongh-153569: derive tokenizer failures from done state
pythongh-153569: keep buffer relocation inside the reader
pythongh-153569: use tokenizer source for parser diagnostics
pythongh-153569: make the reader own interactive state
pythongh-153569: initialize file encoding without tokenizing
pythongh-153569: remove redundant reader counters
pythongh-153569: let the reader construct tokenizer state
pythongh-153569: keep only active formatted-string state
pythongh-153569: return tokenizer tokens as source spans
pythongh-153569: remove the unused tokenizer cursor and line index
pythongh-153569: store formatted-string text as source spans
pythongh-153569: derive token locations from the live buffer
pythongh-153569: derive tokenizer failures from done state
pythongh-153569: keep buffer relocation inside the reader
pythongh-153569: use tokenizer source for parser diagnostics
pythongh-153569: make the reader own interactive state
pythongh-153569: initialize file encoding without tokenizing
pythongh-153569: remove redundant reader counters
pythongh-153569: let the reader construct tokenizer state
pythongh-153569: keep only active formatted-string state
pythongh-153569: remove the unused tokenizer cursor and line index
pythongh-153569: bound tokenizer input storage
pythongh-153569: return tokenizer tokens as source spans
pythongh-153569: store formatted-string text as source spans
pythongh-153569: derive token locations from the live buffer
pythongh-153569: derive tokenizer failures from done state
pythongh-153569: keep buffer relocation inside the reader
pythongh-153569: use tokenizer source for parser diagnostics
pythongh-153569: make the reader own interactive state
pythongh-153569: initialize file encoding without tokenizing
pythongh-153569: remove redundant reader counters
pythongh-153569: let the reader construct tokenizer state
pythongh-153569: keep only active formatted-string state
pythongh-153569: assert tokenizer state invariants
pythongh-153569: store formatted-string text as source spans
pythongh-153569: derive token locations from the live buffer
pythongh-153569: derive tokenizer failures from done state
pythongh-153569: keep buffer relocation inside the reader
pythongh-153569: use tokenizer source for parser diagnostics
pythongh-153569: make the reader own interactive state
pythongh-153569: initialize file encoding without tokenizing
pythongh-153569: remove redundant reader counters
pythongh-153569: let the reader construct tokenizer state
pythongh-153569: keep only active formatted-string state
pythongh-153569: assert tokenizer state invariants
pythongh-153569: read diagnostic lines through the source API
pythongh-153569: finish tokenizer offsets, diagnostics, and opaque API
pythongh-153569: finish tokenizer offsets, diagnostics, and opaque API
pythongh-153569: remove obsolete tokenizer cursor checks
pythongh-153569: record final review against current main
pythongh-153569: record published review stack and CI status
gh-153569: return token spans and unify decoded source storage (#156482)
pythongh-153569: rebase remaining stack after token spans landed
pythongh-153569: finish tokenizer offsets, diagnostics, and opaque API
pythongh-153569: remove obsolete tokenizer cursor checks
pythongh-153569: record published rebases and fresh CI
pythongh-153569: split state consolidation from the opaque API cutover
pythongh-153569: finish tokenizer offsets and the opaque API cutover
pythongh-153569: remove unused tokenizer cursor and source lookup APIs
pythongh-153569: report tokenizer diagnostics without rewinding the s…
pythongh-153569: hide tokenizer state behind the consumer API