Repository navigation
String NFC/NFD normalization and lone surrogate handling #18
Description
Description
Three related Unicode string gaps (all in test_gap_string_methods):
-
NFC/NFD normalization:
"é".normalize("NFC").lengthreturns 2 instead of 1;"é".normalize("NFD").lengthreturns 3 instead of 2. Perry stores strings as UTF-8 bytes and.lengthreturns byte length, not codepoint/grapheme count. -
isWellFormed()/toWellFormed(): Lone surrogate detection returns wrong results."\uD800".isWellFormed()returnstrue(should befalse). -
UTF-16 length semantics: JS
.lengthcounts UTF-16 code units, Perry counts bytes. Diverges on any string with characters outside BMP.
These are all manifestations of the same root issue: Perry uses UTF-8 byte-oriented string representation while JS uses UTF-16.
Activity
proggeramlug commentedon Apr 14, 2026
Partially resolved in v0.5.20.
.lengthnow returns UTF-16 code units instead of UTF-8 byte count. All position-based string APIs (charAt,slice,substring,indexOf,lastIndexOf,padStart,padEnd) converted to UTF-16 indexing.Results:
"café".length→ 4 (was 5)"日本語".length→ 3 (was 9)"😀".length→ 2 (was 4).normalize().lengthnow matches Nodetest_gap_string_methodsdiffs: 4 → 2. Remaining 2 are lone surrogate handling (isWellFormed/toWellFormed) — needs WTF-8 string support, tracked separately.proggeramlug commentedon Apr 14, 2026
Two of three sub-issues resolved in v0.5.20:
✅ NFC/NFD normalization length —
.normalize("NFC").lengthnow returns correct codepoint count✅ UTF-16 length semantics —
"café".length→ 4,"日本語".length→ 3,"😀".length→ 2Remaining issue (lone surrogate handling) tracked separately in a new ticket.