Repository navigation
Bad symbols in code #436
Description
Activity
Yeah, its because whatever opened them treated the text as utf-8 and treats the values > 127 as unknown, when its really the windows code page 1252. VSCode is always doing this for me and its a pain to clean up if you don't notice before a commit. For comments like these its probably best to convert correctly to utf-8 these days, for print statements maybe just remove the none ascii as you suggest?
Let us strip such characters from all code please. I see no reason to use them. Everything can be Ascii.
For game data, I added the following .gitattributes file, which avoided troubles ever since, but never understood how this all works exactly:
https://github.com/TheSuperHackers/GeneralsGamePatch/blob/main/.gitattributes
Anyone wants to take care of massaging this?
Can someone take care of this?
Hmm. So you convert ascii files to utf8? Can you ask OmniBlade about that?
For Keyboard.cpp and Debug.cpp this is probably necessary, but all others look like they could remain ascii.
Any change but hex literal will alter what the char is in the binary
Potentially making it draw garbage when its drawnMost of these are comments and in general I'm okay with bumping the comments to UTF-8. I think we need to be careful with the literals in the code as that will change what is in the binary but I can only spot a handful of places where they were changed. Ideally as @tomsons26 said in the discord chat, those need replacing carefully, especially the keyboard related ones. The logging ones, well the risk is there that you get a garbage character in your output, not too serious.
Using a Python script I converted all files to utf-8, with 66 files having special characters.
https://github.com/TheSuperHackers/GeneralsGameCode/compare/main...DevGeniusCode:GeneralsGameCode:Hex-literals?expand=1
Most of them were in comments, only in 7 cases were they part of the code, I binary compared the files before and after the changeAny idea?
VS2022
\diff_binary.py -l \before_vs22.exe \after_vs22.exe Result of comparison: content File 1 length: 21053952 File 2 length: 21053952 Num. differences: 11 offset differs: 0x158 List of differences: offset 0x00000158: 0x9c != 0x72 ('�' != 'r') offset 0x00000159: 0x70 != 0x76 ('p' != 'v') offset 0x00f1e5e9: 0x35 != 0x36 ('5' != '6') offset 0x00f1e5eb: 0x35 != 0x32 ('5' != '2') offset 0x00f1e5ec: 0x38 != 0x33 ('8' != '3') offset 0x00f1e5ee: 0x34 != 0x33 ('4' != '3') offset 0x0117fbfc: 0x9c != 0x72 ('�' != 'r') offset 0x0117fbfd: 0x70 != 0x76 ('p' != 'v') offset 0x0117fc18: 0x9c != 0x72 ('�' != 'r') offset 0x0117fc19: 0x70 != 0x76 ('p' != 'v') offset 0x011b3f4c: 0x06 != 0x07 ( 6 != 7) Elapsed time: 00:00:01VS6
\diff_binary.py -l \before_vs6.exe \after_vs6.exe Result of comparison: content File 1 length: 15413360 File 2 length: 15413360 Num. differences: 9 offset differs: 0x158 List of differences: offset 0x00000158: 0x56 != 0x49 ('V' != 'I') offset 0x00000159: 0x72 != 0x79 ('r' != 'y') offset 0x00b24004: 0x56 != 0x49 ('V' != 'I') offset 0x00b24005: 0x72 != 0x79 ('r' != 'y') offset 0x00bd756b: 0x30 != 0x33 ('0' != '3') offset 0x00bd756c: 0x36 != 0x35 ('6' != '5') offset 0x00bd756e: 0x31 != 0x34 ('1' != '4') offset 0x00bd756f: 0x32 != 0x39 ('2' != '9') offset 0x00eb300c: 0x02 != 0x03 ( 2 != 3) Elapsed time: 00:00:01Right so there aren't bad symbols, just win1252, in actual code they should be replaced with hex literals since the game is coded with 1252 in mind
comments - depending on situationbytes from retail binaries
CPU string
4C 31 20 49 6E 73 74 72 75 63 74 69 6F 6E 20 54 72 61 63 65 20 43 61 63 68 65 3A 20 25 64 20 77 61 79 20 73 65 74 20 61 73 73 6F 63 69 61 74 69 76 65 2C 20 25 64 6B 20 B5 4F 50 73 0D 0A 00
messagebox string45 73 20 69 73 74 20 65 69 6E 20 67 72 61 76 69 65 72 65 6E 64 65 72 20 46 65 68 6C 65 72 20 61 75 66 67 65 74 72 65 74 65 6E 2E 20 53 6F 6C 63 68 65 20 46 65 68 6C 65 72 20 6B F6 6E 6E 65 6E 20 64 75 72 63 68 20 76 69 65 6C 65 20 76 65 72 73 63 68 69 65 64 65 6E 65 20 44 69 6E 67 65 20 77 69 65 20 56 69 72 65 6E 2C 20 FC 62 65 72 68 69 74 7A 74 65 20 48 61 72 64 77 61 72 65 20 75 6E 64 20 48 61 72 64 77 61 72 65 2C 20 64 69 65 20 64 65 6E 20 4D 69 6E 64 65 73 74 61 6E 66 6F 72 64 65 72 75 6E 67 65 6E 20 64 65 73 20 53 70 69 65 6C 73 20 6E 69 63 68 74 20 65 6E 74 73 70 72 69 63 68 74 2C 20 61 75 73 67 65 6C F6 73 74 20 77 65 72 64 65 6E 2E 20 54 69 70 70 73 20 7A 75 72 20 56 6F 72 67 65 68 65 6E 73 77 65 69 73 65 20 66 69 6E 64 65 73 74 20 64 75 20 69 6E 20 64 65 6E 20 46 6F 72 65 6E 20 75 6E 74 65 72 20 77 77 77 2E 67 65 6E 65 72 61 6C 73 2E 65 61 2E 63 6F 6D 2C 20 49 6E 66 6F 72 6D 61 74 69 6F 6E 65 6E 20 7A 75 6D 20 54 65 63 68 6E 69 73 63 68 65 6E 20 4B 75 6E 64 65 6E 64 69 65 6E 73 74 20 69 6D 20 48 61 6E 64 62 75 63 68 20 7A 75 6D 20 53 70 69 65 6C 2EL'$' line replacement is 0x20AC
all the strange - should go and be replaced with normal -
or the integerßin host byte ordertoor the integer in host byte orderimo the keyboard comments should describe the char instead of using it
copyright as (c) instead of the symbol
When it comes to UTF-8 in source files, I can say following:
In code:
As already pointed out, code should definitely be ASCII only, including strings (which should use escape sequences as necessary), to make code portable / produce consistent results in different build environments (system encoding / compiler / OS).In comments:
Could theoretically cause problems as well. Here are cases to consider based on system encoding (used by compiler?):UTF-8: OK
If system encoding is UTF-8, there is obviously no issue. This includes basically any relevant linux distro, which all switched to UTF-8 as default system encoding long time ago. (I think similar applies to other relevant non-windows OSes)Single byte encodings (such as WINDOWS-1252): OK
If system uses (strictly) single byte encoding (i.e. one that does not use multi byte sequences), it should be fine. I am talking about encodings like WINDOWS-1252, or other encodings used (mainly) for European languages (using extended latin, cyrilic etc.). These are used as default system encodings on Windows for these languages AFAIK. They use char values >= 128 for extended (non-ascii) characters and have convenient property, that any sequence of bytes produces valid string. UTF-8 on other hand has convenient property that all multibyte sequences have all their bytes >= 128, so none of these bytes can be accidentally interpreted as ascii char (e.g. multibyte sequence cannot end comment prematurely by interpreting as*/etc.). So even though multibyte sequences of UTF-8 will interpret as garbage in these encodings, that garbage is still valid (and safe) character sequence, and it does not matter, because it is in comment.Variable-width (Multi byte) encodings (other than UTF-8): PROBLEM?
Here I am talking about Windows legacy encodings, used mostly by Asian languages such as "Shift JIS" (or Windows-932; default for Japanese Windows?) or other encodings used for Asian languages. Here I think it could cause problems, in case, where UTF-8 multibyte sequence would not produce valid char sequence in given encoding (used as system one). I Don't know how compiler would behave in that case. or how much valid concern this is. Also this could probably be work-arounded, if necessary, by setting compiler encoding explicitly to UTF-8 or WINDOWS-1252.Anyway UTF-8 in comments is not something, which would concern me personally. Probably can be re-evalued, in case, it causes real-world issues.
Reacted by DevGeniusCode
There is this code in a file:
When I changed and saved this file it messed up the file diff on these lines with the bad characters.
We may want to