fix(plugin-mysql): set the session character set on connect and decode MySQL text by its declared charset (#2725)

[datlechin]

Fixes #2725

Root cause

The reported comment, メール・記事ç´^P付ã^A‘, is UTF-8 that was stored through a session the server treated as Latin 1, then read back through UTF-8. The ^P and ^A are the giveaway: MySQL's latin1 is cp1252 with five bytes passed through as C1 controls, and the stored bytes are C3A3C692C2A1…. Every correct UTF-8 client shows the same string. TablePro was reading what the server holds.

TablePro could write exactly those bytes itself, and could not read such data any other way:

1. Connect. The driver asked for utf8mb4 only through the handshake's collation byte (MYSQL_SET_CHARSET_NAME) and never checked the result. A server with init_connect='SET NAMES latin1' (any user without CONNECTION_ADMIN), MariaDB with skip-character-set-client-handshake, or a server without utf8mb4 leaves the session latin1 while libmariadb still reports utf8mb4. TablePro kept sending UTF-8, so every comment or value it saved there was stored double-encoded. The handshake's utf8mb4_general_ci also made WHERE col = @var fail with ERROR 1267 Illegal mix of collations on MySQL 8.
2. SQL export. A MySQL dump carried no SET NAMES. Restoring it with a client whose default is Latin 1 (a mysql CLI without a UTF-8 locale, the MySQL 5.7 client, docker-entrypoint-initdb.d) produces byte-identical Characters are incorrectly decoded in table comments #2725 mojibake.
3. Decode. Every text cell was decoded as UTF-8 with an ISO-8859-1 fallback, ignoring the field's declared charset. The block was copied three times in a file no test can compile. After SET NAMES latin1, ’ and € came back as invisible C1 controls.

Fix

• establishSessionCharacterSet calls mysql_set_character_set("utf8mb4") on every connect path (main connection, metadata pool, reconnect, idle-release reacquire). It falls back to a plain SET NAMES utf8, and if the server refuses both it keeps the server's own session rather than failing the connect, so an engine that rejects SET NAMES still connects. Sessions now get the server's default utf8mb4 collation, the same as the mysql 8.4 CLI.
• One decoder, MySQLColumnDecoding, keyed on each field's charsetnr through mariadb_get_charset_by_nr, serves the text, prepared and streaming paths, column names and error messages. latin1 reads UTF-8 first and then MySQL's real latin1 table (MySQLLatin1), which keeps the existing SET NAMES latin1 trick working. The ISO-8859-1 fallback is gone. The Foundation encodings in MySQLCharacterSet are limited to the 19 charsets that decode byte-for-byte like the server. scripts/check-mysql-charset-decoding.sh re-checks that against a live server. sjis, ujis, big5, euckr, greek, hebrew, koi8, cp866, latin7 and tis620 each disagree on some characters and stay out.
• Outgoing SQL stays the UTF-8 bytes of the string, as the mysql client sends them. Following a mid-session SET NAMES on the way out was considered and rejected: a legacy dump that declares SET NAMES latin1 over UTF-8 bytes restores byte-for-byte today only because nothing re-encodes it.
• Encoding option (Options pane, MySQL and MariaDB): UTF-8 or UTF-8 via Latin 1. The second matches Sequel Ace's mode, but the read side is lossless. It sets only character_set_client = latin1, so writes are stored the way a latin1 client stored them. Results stay utf8mb4 and each value is repaired client-side. So correctly stored text never comes back as ?, and the handshake stays utf8mb4, which keeps non-ASCII logins and database names working.
• SQL export writes mysqldump's SET NAMES utf8mb4 prologue and @OLD_* restore epilogue into every split part. Each DDL statement is now written in one write, so a part can never end mid-statement, and the epilogue counts toward the split size.
• MariaDBFieldMetadata moves the pure type mapping out of the CMariaDB file. That revives MariaDBFieldClassifierTests and MariaDBTypeNameTests (27 cases behind a canImport that was never true) and deletes the parser copy GeometryWKBParserTests carried.
• Also fixed while rewriting the decoder: GEOMETRY from a parameterized query was shown as raw WKB bytes. Separate commit, same loop: after one value over 64 KB, the prepared-statement refetch freed a buffer libmariadb still wrote into, so every later row showed an earlier row's text. The fix rebinds.

Measured

A swiftc harness ran the real plugin sources against a throwaway MySQL 8.4.11, as a user with init_connect='SET NAMES latin1':

	origin/main	this branch
session	latin1 / latin1 / latin1	utf8mb4 / utf8mb4 / utf8mb4
ALTER TABLE … COMMENT 'メール・記事紐付け' stored as	C3A3C692C2A1… (#2725)	E383A1E383BC…
it’s 5€ after SET NAMES latin1	its 5 (C1 controls)	it’s 5€
WHERE note = @v on MySQL 8	ERROR 1267	1 row
parameterized SELECT of GEOMETRY	raw WKB bytes	POINT(1.0 2.0)
parameterized rows over 64 KB (a…, b…, c…, short)	a… four times	correct
UTF-8 via Latin 1: dbl / good comments	n/a	メール・記事紐付け / メール・記事紐付け
UTF-8 via Latin 1: stored bytes of a new COMMENT '新規'	n/a	C3A6E28093C2B0… (what a latin1 client stores)

scripts/check-mysql-charset-decoding.sh passes against MySQL 8.4.11 and MariaDB 12.3.3 (gb18030 skipped there, the server lacks it).

Before / After

Sidebar of a database whose dbl table comment was written through a Latin 1 client. Before: the comment shows as stored. After: the same connection with Encoding set to UTF-8 via Latin 1, while good (stored correctly) is unchanged.

Before	After

The new Encoding row in the Options pane:

Tests

• New: MySQLLatin1Tests (all 256 bytes against the server's own conversion, the Characters are incorrectly decoded in table comments #2725 string repaired), MySQLCharacterSetTests, MySQLColumnDecodingTests (including Databend's boolean and hex-binary shapes, which moved into the decoder during the rebase onto feat(plugin-mysql): add TiDB and Databend connection types #2711), MySQLConnectionEncodingTests (the field matches the curated MySQL and MariaDB entries byte-for-byte as JSON, and the plugin declares it), SQLExportEncodingTests (every part is wrapped, the epilogue counts toward the cap).
• Revived: MariaDBFieldClassifierTests, MariaDBTypeNameTests; GeometryWKBParserTests now runs against the real parser.
• No UI test. The only UI is a declarative dropdown the metadata tests cover, and the flow needs a live MySQL server holding legacy data.

Review

Codex could not review: the account hit its usage limit (resets Sep 15). The code-review skill reviewed instead. A security review found that reading column names by name_length could run past a name containing a NUL, which libmariadb copies with strdup. It is now bounded with strnlen.

Follow-ups

The iOS MySQL driver has the same connect-time gap and ignores the synced Encoding field. It follows in a stacked PR. Separate PRs follow for the other encoding defects found during this investigation: PostgreSQL client_encoding, ClickHouse whole-body decoding, SQL Server non-N literals, SQL import encodings, and MySQL session replay.

https://claude.ai/code/session_01U7kkDGsR4M4v9NWiycc97U

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
TablePro	🟢 Ready	View Preview	Sep 11, 2026, 3:26 PM

💡 Tip: Enable Automations to automatically generate PRs for you.