Repository navigation
If gb18030 is revised, consider aligning the Encoding Standard #27
Description
Cause GB18030-2005 is already one-to-one mapping bettween Unicode & GBK18030 except
The 14 characters that still mapped into Unicode PUA that at 2005,
But nowadays, all the 14 characters have correlated mapping into Unicode,
So I suggest encoding standard mapping those characters to normal Unicode characters but PUA characters.
The following 80 characters are the GBK chracters that ever mapped to Unicode PUA, and
the corresponding Unicode non-PUA character
Han Character GBK Unicode PUA Unicode non-PUA
FE50 E815 2E81
FE51 E816 20087
FE52 E817 20089
FE53 E818 200CC
FE54 E819 2E84
FE55 E81A 3473
FE56 E81B 3447
FE57 E81C 2E88
FE58 E81D 2E8B
FE59 E81E 9FB4
FE5A E81F 359E
FE5B E820 361A
FE5C E821 360E
FE5D E822 2E8C
FE5E E823 2E97
FE5F E824 396E
FE60 E825 3918
FE61 E826 9FB5
FE62 E827 39CF
FE63 E828 39DF
FE64 E829 3A73
FE65 E82A 39D0
FE66 E82B 9FB6
FE67 E82C 9FB7
FE68 E82D 3B4E
FE69 E82E 3C6E
FE6A E82F 3CE0
FE6B E830 2EA7
FE6C E831 215D7
FE6D E832 9FB8
FE6E E833 2EAA
FE6F E834 4056
FE70 E835 415F
FE71 E836 2EAE
FE72 E837 4337
FE73 E838 2EB3
FE74 E839 2EB6
FE75 E83A 2EB7
FE76 E83B 2298F
FE77 E83C 43B1
FE78 E83D 43AC
FE79 E83E 2EBB
FE7A E83F 43DD
FE7B E840 44D6
FE7C E841 4661
FE7D E842 464C
FE7E E843 9FB9
FE80 E844 4723
FE81 E845 4729
FE82 E846 477C
FE83 E847 478D
FE84 E848 2ECA
FE85 E849 4947
FE86 E84A 497A
FE87 E84B 497D
FE88 E84C 4982
FE89 E84D 4983
FE8A E84E 4985
FE8B E84F 4986
FE8C E850 499F
FE8D E851 499B
FE8E E852 49B7
FE8F E853 49B6
FE90 E854 9FBA
FE91 E855 241FE
FE92 E856 4CA3
FE93 E857 4C9F
FE94 E858 4CA0
FE95 E859 4CA1
FE96 E85A 4C77
FE97 E85B 4CA2
FE98 E85C 4D13
FE99 E85D 4D14
FE9A E85E 4D15
FE9B E85F 4D16
FE9C E860 4D17
FE9D E861 4D18
FE9E E862 4D19
FE9F E863 4DAE
FEA0 E864 9FBB
The following 14 characters are the GB18030-2005 chracters that are still mapped to Unicode PUA, and
I suggest the encoding standard mapping those characters into Unicode non-PUA, cause we have no need
to waiting GB18030 to update it's spec just for those 14 chracters, and we could be sure those 14 chracters's
corresponding Unicode non-PUA characters are decided.
Han Character GBK Unicode PUA Unicode non-PUA
FE51 E816 20087
FE52 E817 20089
FE53 E818 200CC
FE59 E81E 9FB4
FE61 E826 9FB5
FE66 E82B 9FB6
FE67 E82C 9FB7
FE6C E831 215D7
FE6D E832 9FB8
FE76 E83B 2298F
FE7E E843 9FB9
FE90 E854 9FBA
FE91 E855 241FE
FEA0 E864 9FBB
And according to these, we can decode all GBK encoding family strings to non-PUA Unicode,
Besides these, we still have the need to convert all the historical Unicode PUA characters
to proper GBK(GB18030) characters.
Activity
vyv03354 commentedon Jan 17, 2016
I disagree. We shouldn't invent yet another new encoding anymore.
annevk commentedon Jan 17, 2016
I tend to agree with @vyv03354. Since no implementation does this and developers are asked to use utf-8, I don't really see an upside here. This only increases the chance that things break.
lygstate commentedon Jan 17, 2016
@vyv03354 @annevk We are not invent new encoding, just getting exist encoding works.
annevk commentedon Jan 18, 2016
Fair, changing an encoding is not inventing a new one. However, it is not clear why we should change it, since implementations mostly agree here.
lygstate commentedon Jan 19, 2016
@annevk @vyv03354 Please consider the following situation, suppose a have text with a Unicode character U20087, when I convert this character to GBK,
What I should to do? 0xFE51 or other invalid character?
So we are just refinement the exist convert table to the final state?
vyv03354 commentedon Jan 19, 2016
We don't convert any plane-2 characters in GBK encoder. It will be changed to a character reference (𠂇).
Japanese users were suffered from encoding "improvements" of JIS standards and industrial de-facto standards. Even one character change is considered as a new encoding in ISO coded character set standards. Such a change will have more harm than good even if it is out of good will.
lygstate commentedon Jan 19, 2016
@vyv03354 That's really different, cause JIS doesn't mapping any characters to PUA Unicode character, that's just because at that time, The Unicode is didn't have enough charset for GBK, but now it's has, that's totally different.
annevk commentedon Jan 20, 2016
@lygstate as with the other issue, I recommend using utf-8 instead. I agree with @vyv03354 that changing implementations at this point is more likely to lead to breakage than happy users.
Artoria2e5 commentedon Sep 5, 2016
Will you add something like a line of "note" to the description for gb18030 in the spec mentioning this issue? PUA really brings a lot of issues to users as using its codepoints without a common agreement is like inventing a nationwide Unicode dialect.
To be frank I would rather leave the dialect pollution in the legacy encoder/decoder bridge than let it spread in the new world, so please consider adding:
and as a basis for these changes,
See also:
@lygstate Could you please consider reopening this issue if you find my — um — attempt helpful?
gbk-gb18030-pua.txt
annevk commentedon Sep 6, 2016
@Artoria2e5 the "new world" should use utf-8 exclusively.
lygstate commentedon Sep 6, 2016
@annevk But we still need a way to migrade from the old world.
Artoria2e5 commentedon Sep 6, 2016
@annevk It's true that the modern world should use UTF-8 for information exchange, processing and storage. But given that character representations in UTF-8 relies on codepoints assigned in Unicode, it makes sense to use the formal, universal codepoint assignments in this universal encoding.
As stated previously, by emitting PUA codepoints in the decoder, you are speaking in a Unicode dialect codepoint-wise, resulting in a less interchangeable UTF-8 variant, thus contradicting the point of using UTF-8 everywhere. (The use of PUA here cannot be justified by a lack of definition as these ideographs do have formal assignments.) The encoder part is more about discouraging old PUA usage.
And we need to make sure that the way gives us actual "new world" stuff.
By the way, there should be 24 PUA codepoints in the 2005 standard instead of 14, according to the L2/06-394 "Update on GB 18030:2005" by Ken Lunde.
An interesting but sad example of this dialect split can be shown using the character U+20087 (𠂇), assigned to PUA codepoint U+E816 () in the mapping. Search engines like Google won't do normalization on PUA forms where several different sets of agreements exist, and you can see it from the search results.
annevk commentedon Sep 6, 2016
Given that no browser implements gb18030 like that I don't see why we should change this. We could easily break those relying on these bytes mapping to PUA. I'm also somewhat reluctant to add a note, since as far as I can tell this is just someone's opinion and those maintaining gb18030 have not decided to care.
aphillips commentedon Sep 6, 2016
The GB18030 mapping is naturally fungible wrt PUA characters, since Unicode continues to encode Chinese code points. I think this should be recognized by Encoding.
I agree that we should not remove mapping of Unicode PUA -> GB18030 (compatibility). But the problem here is round-tripping of real Unicode code points with GB18030.
If I have a U+20087, convert it to GB18030, and the later reserialize the GB data as UTF-8, I will get back U+E816 rather than the original (and correct) code point. That's undesirable and a loss of information. The fact that existing implementations haven't caught up with standardization doesn't mean that we shouldn't make this change.
@annevk Under what circumstances would we change? One of the problems with establishing a standard is that implementations are trying hard to be compliant with it...
24 remaining items
wswartzendruber commentedon Aug 29, 2021
Hello. I am an independent individual who is implementing a GB 18030 encoder purely as a hobby (it is quite challenging to do concisely). Was there ever an update on GB 18030-2018?
lygstate commentedon Sep 5, 2021
@kenlunde Hi, does the new 2018 standard released?
TimothyGu commentedon Sep 14, 2021
It appears that it's still under review. See IRGN2496 that was published on 2021-09-09:
(note, this is the same update as IRGN2453, dated 2021-03-05)
One thing that could have delayed the publication of the new revision could be related to the new regulations alluded to in IRGN2453:
TimothyGu commentedon Sep 14, 2021
@kenlunde just told me that during a meeting yesterday(!!), the delegates from China anticipate the revision to be published next year (2022). Here's some more information:
xfq commentedon Aug 17, 2022
The GB 18030-2022 Standard has been published in July 2022. The new version includes 196 Chinese characters that are in the Table of General Standard Chinese Characters but not in GB 18030-2005, includes 17,000+ other new Chinese characters, and has established three implementation levels.
This standard is mandatory in Mainland China. All software supporting Chinese information processing and exchange must support level 1, some software such as operating systems, databases, and middleware must support level 2, and systems for public services must support level 3. (The higher the level, the more Chinese characters are included.)
hsivonen commentedon Aug 17, 2022
More information in English
Support for the mandatory characters can be achieved vie UTF-8, which the Encoding Standard already supports (and, evidently, there are now UTF-8-only software and formats).
vyv03354 commentedon Aug 17, 2022
Do we implement 18 code point swaps, after all?
annevk commentedon Oct 27, 2022
https://unicode-org.atlassian.net/browse/ICU-22098 might have implications for what some implementations do, I presume. That raises the importance of resolving #57 somewhat.
hfhchan commentedon Oct 28, 2022
GB18030-2022 will take effect on 1 Aug 2023. Compliance criteria include, at a minimum, not emitting PUA characters for the 24 characters for input methods, and not using the 24 PUA codepoints for fonts.
However, most existing products sold on the Chinese market fail these tests and those old versions will still be expected to be used, even though they will no longer be allowed to be sold after the effective date. Also there's existing UTF-8 content which are using those PUA codepoints.
To be backwards compatible with older products based on the GB18030-2005 standards, both the PUA and the non-PUA codepoints should map to the correct GB18030-2022 2-byte sequences.
Whether or not the 4-byte sequences should map to the non-PUA codepoints is less of an issue -- it is not expected that there be data in GB18030 that are stored in the 4-byte form. However, if keeping the double mapping to U+3000 is deemed web compatible, then keeping the 4-byte sequences mapped to the non-PUA codepoints should also be web compatible in the same manner.
annevk commentedon Sep 17, 2024
I created #336 which I hope addresses this. Review most welcome!
Adopt GB18030-2022
Adopt GB18030-2022