Skip to content

Formatting logic rewrite in Rust - #80

Open
ChHecker wants to merge 36 commits into
mainfrom
rust
Open

ChHecker wants to merge 36 commits into
mainfrom
rust

Conversation

@ChHecker

@ChHecker ChHecker commented Sep 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR completely rewrites the core execution logic in Rust compiled to a Wasm plugin (wasm32-unknown-unknown).

The previous Typst-native implementation became increasingly difficult to maintain and extend, particularly due to complex, fragile regex patterns used for number parsing. The new architecture splits the processing pipeline into clean, explicit phases: Lexer → Parser → Typst Generator.

This also enables the solution to many long-standing issues with this package.


Key Improvements & Architecture

Performance & Memory

  • Compile-Time Static Units: Unit dictionaries are now compiled directly into the Wasm binary using nested phf (Perfect Hash Function) maps generated at build time (build.rs). This eliminates runtime file I/O and dynamic parsing overhead, drastically speeding up unit evaluation.

Modular Parsing Pipeline

  • Replaced monolithic regex matching with a dedicated tokenization and parsing engine.
  • Cleanly isolates number parsing, symbol resolution, and configuration application.

New Features & API Changes

  • Flexible Range Formatting: Manual control over exponent placement and unit positioning in ranges via exponent-mode and unit-mode (e.g., controlling behavior for (2 to 4) °C vs 2 °C to 4 °C).
  • Uncertainty Notation Support: Full support for compact/parenthetical uncertainty notation (e.g., "2.123(45)") (credit to @catlop, who already introduced it in the Typst version in Add parentheses notation for symmetric uncertainties #74).
  • Global Configuration Management: Added global configuration helpers (#update-num-config, #update-qty-config, etc.) to set workspace-wide defaults without repeating arguments across individual function calls.
  • Custom Decimal Separators: Explicit decimal separator overrides, available both per-call and globally.
  • Custom Postfixes: Added foundation for custom unit postfixes, currently only supporting custom exponents for the long-form notation.
  • Square Root Units (sqrt):
    • Added support for square root expressions on units (e.g., sqrt(m)).
    • Current Limitation: Nested evaluation currently resolves inner exponents first before applying the root. Expression structures like sqrt(m^3) work as expected, but combinations like sqrt(m)^3 are not yet supported and will fail parsing.

Work in Progress & Known Limitations

  • Error Reporting: Panic handling and user-facing error messages have been significantly improved using structured Result returns in Wasm, but error diagnostic formatting is not yet fully optimal.

Testing Strategy

Since this is a major architectural rewrite, testing is critical before merging into main:

  1. This PR acts as the primary draft/tracking branch for community testing.
  2. Please test edge cases (unusual unit combinations, range formatting, global config interactions) against your documents and report any parsing anomalies or performance bottlenecks!

heisenbergpxh and others added 28 commits July 13, 2026 14:24
Numbers and units were always typeset via a math.equation, so they
inherited the math font even in documents set in a different body
font, creating a visible mismatch with surrounding prose (siunitx has
supported this via `mode = text` in LaTeX for a long time).
Add a `mode: "math" | "text"` parameter to num, unit, qty, numrange,
and qtyrange (default "math", fully backward compatible). In "text"
mode, a `show math.text: set text(font: text.font, weight: text.weight)`
rule restyles only the literal glyphs to the current document font and
weight, while the math font still handles layout (superscripts,
fractions, stretchy parentheses), so positioning stays correct and
bold/emph contexts are inherited automatically. Unit symbols
deliberately stay upright even in italic text, matching SI convention.
Adds a _display-math helper in format.typ shared by all five public
functions, and documents the feature in the README and example file.
@ChHecker ChHecker self-assigned this Sep 16, 2026
@catlop

catlop commented Sep 16, 2026

Copy link
Copy Markdown

Here are some backward compatibility issues I’ve found updating a document to this version:

  1. ± in number doesn’t work anymore so I had to convert all ± into +-
  2. Fractional powers for units don’t work anymore so I had to convert all unit^0.5 to sqrt(unit) or unit^(1/2)

I’d also like to suggest being able to use √unit and √(unit) in unit expressions. Being able to use √ instead of sqrt is a feature of typst so it’d be nice to have it here too.

@ChHecker

ChHecker commented Sep 16, 2026 •

Copy link
Copy Markdown
Owner Author

Here are some backward compatibility issues I’ve found updating a document to this version:

1. `±` in number doesn’t work anymore so I had to convert all `±` into `+-`

2. Fractional powers for units don’t work anymore so I had to convert all `unit^0.5` to `sqrt(unit)` or `unit^(1/2)`

I’d also like to suggest being able to use √unit and √(unit) in unit expressions. Being able to use √ instead of sqrt is a feature of typst so it’d be nice to have it here too.

Thank you for testing the new rewrite!

  1. Including unicode for +- and sqrt is fairly simple in the lexer. I am not sure whether I like allowing sqrt unit instead of sqrt(unit), which would be necessary to allow √unit.
  2. I'm also not too sure about decimals as exponents, because then one needs to pass the num configuration to unit, for example for the decimal separator. Unfortunately, that's apparently a breaking change... Could you comment on whether your use of 0.5 was because you think it's better or because 1/2 didn't work before?

@catlop

catlop commented Sep 16, 2026

Copy link
Copy Markdown

I only used 0.5 because 1/2 and sqrt weren’t possible before (thanks for this PR making both possible btw!). I definitely prefer 1/2 to 0.5. Apparently there are situations where units can have irrational exponents (https://physics.stackexchange.com/questions/13245/dimensional-analysis-restricted-to-rational-exponents) so a use case could be approximating that irrational exponent with decimals (the top answer in the link gives an example of a quantity with units mᵝ / s but implementing something like that seems difficult).

As for the sqrt unit issue, you could have sqrt and √ be different tokens that are rendered the same way. That way the parser accepts √unit and denies sqrt unit and they are rendered the same. This would make writing some units faster and take up less line space in the code, and is also the way typst handles unicode √ behaviorwise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants