Adopt GB18030-2022 - #336
Merged
Merged
Conversation
This implements the Unicode Technical Committee recommendation around GB18030-2022 in a matter suitable for this standard, taking into account existing practice and the closeness between GBK and gb18030. In particular, using the text file attached to https://www.unicode.org/L2/L2023/23003r-gb18030-recommendations.pdf this does the following: 1. Merges the first set of 18 mappings, which are bidirectional, directly into index gb18030, replacing existing PUA entries. This ends up impacting GBK and gb18030. 2. The second set of 18 mappings (from PUA to bytes) are encoded as an encoder only table, for both GBK and gb18030. 3. The third set of 18 mappings (from bytes to code points) are ignored, as they are already covered by index gb18030 ranges. (Presumably they are included because the recommendation covers the transition from "Previous Mappings" to "Current Mappings" to "Recommended Mappings", whereas we are going directly from "Previous Mappings" to "Recommended Mappings".) The reason for changing GBK as well is because Chromium and WebKit have already code in the wild that impacts GBK to some degree (although the encoder only table is excluded for GBK only at the moment, including that would make the most sense compatibility-wise) and no fallout has been recorded. Additionally GBK is already positioned as a rough subset of gb18030 in this standard, with the decoder being shared completely. Tests: encoding/legacy-mb-schinese has some GB18030-2022 coverage already. The aim is to complete that with web-platform-tests/wpt#48239 and web-platform-tests/wpt#48240. This supersedes #335. This fixes #27 and fixes #312.
This was referenced Sep 18, 2024
Closed
This was referenced Sep 18, 2024
hsivonen
reviewed
Sep 18, 2024
hsivonen
approved these changes
Sep 18, 2024
hsivonen
left a comment
hsivonen
left a comment
Member
There was a problem hiding this comment.
Technically LGTM, but please 1) revise the informative description of index-gb18030-ranges.txt and 2) please re-run the visualization generator script.
Thanks.
xfq
added
the
i18n-tracker
hsivonen
approved these changes
Sep 19, 2024
webkit-commit-queue
pushed a commit
to annevk/WebKit
that referenced
this pull request
Sep 20, 2024
https://bugs.webkit.org/show_bug.cgi?id=279903 Reviewed by Alex Christensen. For GBK and gb18030 we have used the same backing table for quite a while now. This backing table was updated to account for GB18030-2022 at some point and this impacted GBK as well. However, the encoder side table was kept disabled for GBK, despite it actually allowing GBK to be more compatible with its former self. whatwg/encoding#336 now standardizes the behavior that GBK and gb18030 are to remain aligned in these matters and this change implements that. The corresponding tests are from this PR: web-platform-tests/wpt#48240 * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-decoder.any.js: * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-encoder-expected.txt: * LayoutTests/imported/w3c/web-platform-tests/encoding/legacy-mb-schinese/gbk/gbk-encoder.html: * Source/WebCore/PAL/pal/text/TextCodecCJK.cpp: (PAL::gb18030AsymmetricEncode): (PAL::gbEncodeShared): Canonical link: https://commits.webkit.org/283987@main
annevk
added a commit
to web-platform-tests/wpt
that referenced
this pull request
Sep 21, 2024
See whatwg/encoding#336 for details.
moz-v2v-gh
pushed a commit
to mozilla/gecko-dev
that referenced
this pull request
Sep 25, 2024
… a=testonly Automatic update from web-platform-tests Encoding: impact of GB18030-2022 on GBK See whatwg/encoding#336 for details. -- wpt-commits: 1ac8deee082ecfb5d3b6f9c56cf9d1688a2fc218 wpt-pr: 48240
gecko-dev-updater
pushed a commit
to marco-c/gecko-dev-wordified-and-comments-removed
that referenced
this pull request
Sep 26, 2024
… a=testonly Automatic update from web-platform-tests Encoding: impact of GB18030-2022 on GBK See whatwg/encoding#336 for details. -- wpt-commits: 1ac8deee082ecfb5d3b6f9c56cf9d1688a2fc218 wpt-pr: 48240 UltraBlame original commit: 0917498f655f23ccc2b8f8a9bbb45759cc8ed8ff
gecko-dev-updater
pushed a commit
to marco-c/gecko-dev-wordified
that referenced
this pull request
Sep 26, 2024
… a=testonly Automatic update from web-platform-tests Encoding: impact of GB18030-2022 on GBK See whatwg/encoding#336 for details. -- wpt-commits: 1ac8deee082ecfb5d3b6f9c56cf9d1688a2fc218 wpt-pr: 48240 UltraBlame original commit: 0917498f655f23ccc2b8f8a9bbb45759cc8ed8ff
gecko-dev-updater
pushed a commit
to marco-c/gecko-dev-comments-removed
that referenced
this pull request
Sep 26, 2024
… a=testonly Automatic update from web-platform-tests Encoding: impact of GB18030-2022 on GBK See whatwg/encoding#336 for details. -- wpt-commits: 1ac8deee082ecfb5d3b6f9c56cf9d1688a2fc218 wpt-pr: 48240 UltraBlame original commit: 0917498f655f23ccc2b8f8a9bbb45759cc8ed8ff
jamienicol
pushed a commit
to jamienicol/gecko
that referenced
this pull request
Sep 26, 2024
… a=testonly Automatic update from web-platform-tests Encoding: impact of GB18030-2022 on GBK See whatwg/encoding#336 for details. -- wpt-commits: 1ac8deee082ecfb5d3b6f9c56cf9d1688a2fc218 wpt-pr: 48240
Member
Author
|
Given that there's been no further feedback I plan on merging this tomorrow. I'll update the commit message to note that this change also more clearly documents the -2000 to -2005 change. |
lexborisov
added a commit
to lexbor/lexbor
that referenced
this pull request
Oct 6, 2024
PR in WHATWG::Encoding whatwg/encoding#336.
Closed
Nopey
added a commit
to Nopey/encoding_rs
that referenced
this pull request
Aug 3, 2025
The previous commit hash, 1d519bf8e5555cef64cf3a712485f41cd1a6a990, was part of PR whatwg/encoding#336, and eventually got merged as commit 2c3853e461afd718be198772170d024e427aee21. There is only a mild change between these two commits, where wording in encoding.bs was changed.
jwidar
pushed a commit
to jwidar/LatencyZeroGithub
that referenced
this pull request
Sep 16, 2025
… a=testonly Automatic update from web-platform-tests Encoding: impact of GB18030-2022 on GBK See whatwg/encoding#336 for details. -- wpt-commits: 1ac8deee082ecfb5d3b6f9c56cf9d1688a2fc218 wpt-pr: 48240
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements the Unicode Technical Committee recommendation around GB18030-2022 in a matter suitable for this standard, taking into account existing practice and the closeness between GBK and gb18030.
In particular, using the text file attached to https://www.unicode.org/L2/L2023/23003r-gb18030-recommendations.pdf this does the following:
The reason for changing GBK as well is because Chromium and WebKit have already code in the wild that impacts GBK to some degree (although the encoder only table is excluded for GBK only at the moment, including that would make the most sense compatibility-wise) and no fallout has been recorded. Additionally GBK is already positioned as a rough subset of gb18030 in this standard, with the decoder being shared completely.
Tests: encoding/legacy-mb-schinese has some GB18030-2022 coverage already. The aim is to complete that with web-platform-tests/wpt#48239 and web-platform-tests/wpt#48240.
This supersedes #335. This fixes #27 and fixes #312.
(See WHATWG Working Mode: Changes for more details.)
Preview | Diff