fix(ios): set the MySQL session character set on iPhone and decode results by their declared charset (#2725)

CHANGELOG.md

@@ -38,6 +38,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- Garbled non-Latin text saved from iPhone and iPad to MySQL servers that force a Latin 1 session. (#2725)
+- **Encoding** ignored on iPhone and iPad by a MySQL connection synced from the Mac. (#2725)
+- Binary MySQL columns shown as text on iPhone and iPad, and searched with `LIKE`.
+- Crash opening a MySQL result whose column has no name on iPhone and iPad.
 - Wrong SQLSTATE code in PostgreSQL, Redshift, CockroachDB and PGlite error messages.
 - Read-only write explanation never shown on PostgreSQL servers.
 - Safe Mode minimum from a configuration profile missing from the toolbar, the Database menu and the connection form. (#2030)

CLAUDE.md

@@ -225,7 +225,7 @@ To ship one: add the record type or field in CloudKit Console (or `xcrun cktool
 
 **A pooled metadata read assumes a second connection reaches the same database, and an embedded engine breaks that assumption**: `MetadataConnectionPool` builds a whole new driver, so it is only correct when the database lives on a server the driver reconnects to. When the database lives *inside* the driver instance, the pool gets a different database: a second `duckdb_open(":memory:")` is a fresh empty database, and a second `duckdb_open` on the same *file* is a second independent read-write instance that the first never sees (DuckDB's file lock does not conflict within one process). The failure is silent, because an empty catalog is indistinguishable from "no tables", which is why #2108 survived a manual refresh. `supportsConnectionPooling` is the opt-out, and it is read only by `DatabaseManager.canPool`; DuckDB and PGlite set it `false`. SQLite-family engines keep pooling, because multi-connection access to one file is what they are built for. Two rules follow. First, every metadata read goes through `DatabaseManager.withMetadataDriver` so `metadataRoute` can apply the rule; reaching for `MetadataConnectionPool.shared.withDriver` directly bypasses it, which is how routines kept pooling after the sidebar stopped. Second, a capability with no `DriverPlugin` static is curated per type and `buildMetadataSnapshot` must carry it over from the built-in snapshot, or `register(snapshot:forTypeId:)` resets it to the struct default the moment the plugin loads. That is not hypothetical: it silently disabled MongoDB's `authenticationIsDatabaseScoped` (#1970) for every build that had the plugin installed. `registerVariant` already treats the curated entry as authoritative, which is the only reason PGlite's flag ever worked.
 
-**A MySQL session's character set is server state, and the driver sets it rather than asking for it**: `MYSQL_SET_CHARSET_NAME` only puts a collation byte in the handshake, and the server is free to ignore it. `init_connect='SET NAMES latin1'` for any user without `CONNECTION_ADMIN`, MariaDB's `skip-character-set-client-handshake`, and a server with no `utf8mb4` all leave the session latin1 while libmariadb still reports `utf8mb4`. The driver sends UTF-8 either way, so every comment and value it wrote there was stored double-encoded, which is the `メール` of #2725, and a user-variable comparison on MySQL 8 failed with `ERROR 1267` under the handshake's `utf8mb4_general_ci`. So `MariaDBPluginConnection.establishSessionCharacterSet` calls `mysql_set_character_set` on every connect path, falls back to a plain `SET NAMES utf8` (libmariadb itself rewrites `utf8` to `utf8mb3`, which a pre-5.5.3 server does not know), and keeps the server's own session when it refuses both rather than failing the connect: a MySQL-protocol engine that rejects `SET NAMES` must still connect. Two rules follow. Outgoing SQL stays the UTF-8 bytes of the Swift string and never follows a mid-session `SET NAMES`: a legacy dump that declares `SET NAMES latin1` over UTF-8 bytes restores byte-for-byte only because nothing re-encodes it, which is also what the `mysql` client does. And every result cell decodes through `MySQLColumnDecoding` by its field's own `charsetnr`, never as "UTF-8 and hope": a latin1 field reads UTF-8 first and then MySQL's latin1, which is cp1252 plus five C1 pass-through bytes and not Foundation's `.isoLatin1` or `.windowsCP1252`. The Foundation encodings in `MySQLCharacterSet` are a hand transcription of the server's tables; `scripts/check-mysql-charset-decoding.sh` diffs them against a live server, and a charset that disagrees stays out of the table. **UTF-8 via Latin 1** (`mysqlConnectionEncoding`) sets only `character_set_client` to latin1 and repairs the text client-side, so correctly stored text is never read back as `?`, which a latin1 results charset would do and a structure edit would then write back.
+**A MySQL session's character set is server state, and the driver sets it rather than asking for it**: `MYSQL_SET_CHARSET_NAME` only puts a collation byte in the handshake, and the server is free to ignore it. `init_connect='SET NAMES latin1'` for any user without `CONNECTION_ADMIN`, MariaDB's `skip-character-set-client-handshake`, and a server with no `utf8mb4` all leave the session latin1 while libmariadb still reports `utf8mb4`. The driver sends UTF-8 either way, so every comment and value it wrote there was stored double-encoded, which is the `メール` of #2725, and a user-variable comparison on MySQL 8 failed with `ERROR 1267` under the handshake's `utf8mb4_general_ci`. So `MariaDBCharacterSet.establishSession`, which the macOS plugin and the iOS driver share, calls `mysql_set_character_set` on every connect path, falls back to a plain `SET NAMES utf8` (libmariadb itself rewrites `utf8` to `utf8mb3`, which a pre-5.5.3 server does not know), and keeps the server's own session when it refuses both rather than failing the connect: a MySQL-protocol engine that rejects `SET NAMES` must still connect. Two rules follow. Outgoing SQL stays the UTF-8 bytes of the Swift string and never follows a mid-session `SET NAMES`: a legacy dump that declares `SET NAMES latin1` over UTF-8 bytes restores byte-for-byte only because nothing re-encodes it, which is also what the `mysql` client does. And every result cell decodes through `MySQLColumnDecoding` by its field's own `charsetnr`, never as "UTF-8 and hope": a latin1 field reads UTF-8 first and then MySQL's latin1, which is cp1252 plus five C1 pass-through bytes and not Foundation's `.isoLatin1` or `.windowsCP1252`. The Foundation encodings in `MySQLCharacterSet` are a hand transcription of the server's tables; `scripts/check-mysql-charset-decoding.sh` diffs them against a live server, and a charset that disagrees stays out of the table. **UTF-8 via Latin 1** (`mysqlConnectionEncoding`) sets only `character_set_client` to latin1 and repairs the text client-side, so correctly stored text is never read back as `?`, which a latin1 results charset would do and a structure edit would then write back.
 
 **A MongoDB update or delete is anchored on `_id` or it does not run**: `generateDelete` used to fall back to a filter built from the remaining columns, which silently dropped every value it could not stringify (all binary) and then `deleteOne`d the first partial match, so a document with a binary `_id` could delete a different document. Both paths now skip with a logged warning instead, matching what `generateUpdate` already did.
 

Plugins/MySQLDriverPlugin/DatabendResultShape.swift

@@ -6,7 +6,7 @@
 import Foundation
 import TableProPluginKit
 
-internal enum DatabendResultShape {
+nonisolated internal enum DatabendResultShape {
     static let booleanTypeName = "BOOLEAN"
 
     private static let shortFieldType: UInt32 = 2

Plugins/MySQLDriverPlugin/GeometryWKBParser.swift

@@ -8,7 +8,7 @@
 
 import Foundation
 
-enum GeometryWKBParser {
+nonisolated enum GeometryWKBParser {
     /// Parses MySQL's internal geometry binary format to WKT string.
     ///
     /// MySQL internal binary format:

Plugins/MySQLDriverPlugin/MariaDBCharacterSet.swift

@@ -0,0 +1,117 @@
+import CMariaDB
+import Foundation
+import OSLog
+
+nonisolated internal enum MariaDBCharacterSet {
+    private static let logger = Logger(subsystem: "com.TablePro", category: "MariaDBCharacterSet")
+
+    static func establishSession(on mysql: UnsafeMutablePointer<MYSQL>, encoding: MySQLConnectionEncoding) -> Bool {
+        if mysql_set_character_set(mysql, MySQLConnectionEncoding.sessionCharacterSetName) != 0 {
+            let refusal = errorSummary(of: mysql, encoding: encoding)
+            if run(MySQLConnectionEncoding.sessionFallbackStatement, on: mysql) {
+                logger.notice("Server refused utf8mb4 (\(refusal, privacy: .public)), so the session uses utf8")
+            } else {
+                logger.warning("Server refused a UTF-8 session (\(refusal, privacy: .public)); keeping its own")
+            }
+        }
+        for statement in encoding.sessionStatements where !run(statement, on: mysql) {
+            return false
+        }
+        return true
+    }
+
+    static func name(forCollation collation: UInt32) -> String? {
+        guard let info = mariadb_get_charset_by_nr(collation), let name = info.pointee.csname else { return nil }
+        return String(cString: name)
+    }
+
+    static func describeColumns(
+        of fields: UnsafeMutablePointer<MYSQL_FIELD>?,
+        count: Int,
+        encoding: MySQLConnectionEncoding,
+        flavor: MySQLServerFlavor = .mysql
+    ) -> MySQLResultColumns {
+        var columns = MySQLResultColumns()
+        guard let fields else { return columns }
+        for index in 0..<count {
+            let field = fields[index]
+            let flags = UInt(field.flags)
+            let decoding = MySQLColumnDecoding(
+                typeRaw: field.type.rawValue,
+                length: field.length,
+                charsetnr: field.charsetnr,
+                characterSetName: name(forCollation: field.charsetnr),
+                flavor: flavor
+            )
+            columns.append(
+                name: decodedName(of: field, encoding: encoding) ?? "column_\(index)",
+                typeCode: typeCode(of: field, flags: flags),
+                typeName: typeName(of: fields + index, decoding: decoding),
+                decoding: decoding,
+                flags: flags
+            )
+        }
+        return columns
+    }
+
+    private static func typeName(
+        of field: UnsafePointer<MYSQL_FIELD>,
+        decoding: MySQLColumnDecoding
+    ) -> String {
+        guard decoding != .databendBoolean else { return DatabendResultShape.booleanTypeName }
+        return mysqlTypeToString(field)
+    }
+
+    private static func typeCode(of field: MYSQL_FIELD, flags: UInt) -> UInt32 {
+        if (flags & mysqlSetFlag) != 0 { return 248 }
+        if (flags & mysqlEnumFlag) != 0 { return 247 }
+        return field.type.rawValue
+    }
+
+    static func decodedName(of field: MYSQL_FIELD, encoding: MySQLConnectionEncoding) -> String? {
+        guard let name = field.name else { return nil }
+        let bytes = UnsafeRawBufferPointer(start: name, count: strnlen(name, Int(field.name_length)))
+        return mysqlSessionText(bytes, encoding: encoding)
+    }
+
+    private static func run(_ statement: String, on mysql: UnsafeMutablePointer<MYSQL>) -> Bool {
+        let status = statement.withCString { mysql_real_query(mysql, $0, UInt(strlen($0))) }
+        if let discarded = mysql_store_result(mysql) {
+            mysql_free_result(discarded)
+        }
+        return status == 0
+    }
+
+    private static func errorSummary(of mysql: UnsafeMutablePointer<MYSQL>, encoding: MySQLConnectionEncoding) -> String {
+        let code = mysql_errno(mysql)
+        guard let message = mysql_error(mysql) else { return "error \(code)" }
+        return "\(code) \(mysqlSessionText(cString: message, encoding: encoding))"
+    }
+}
+
+nonisolated func mysqlTypeToString(_ fieldPtr: UnsafePointer<MYSQL_FIELD>) -> String {
+    let field = fieldPtr.pointee
+    let flags = UInt(field.flags)
+
+    // MariaDB extended metadata: detect JSON stored as LONGTEXT.
+    // `MARIADB_CONST_STRING` is length-prefixed (not null-terminated), so we must read
+    // exactly `attr.length` bytes. `String(cString:)` would scan past the buffer into
+    // adjacent memory and intermittently fail the comparison when that memory is non-zero.
+    var attr = MARIADB_CONST_STRING()
+    if mariadb_field_attr(&attr, fieldPtr, MARIADB_FIELD_ATTR_FORMAT_NAME) == 0,
+       let str = attr.str, attr.length > 0,
+       let value = String(data: Data(bytes: str, count: Int(attr.length)), encoding: .utf8),
+       value == "json" {
+        return "JSON"
+    }
+
+    if (flags & mysqlEnumFlag) != 0 { return "ENUM" }
+    if (flags & mysqlSetFlag) != 0 { return "SET" }
+
+    return mariaDBTypeName(
+        typeRaw: field.type.rawValue,
+        flags: flags,
+        charsetnr: field.charsetnr,
+        length: field.length
+    )
+}

Plugins/MySQLDriverPlugin/MariaDBFieldClassifier.swift

@@ -5,7 +5,7 @@
 
 import Foundation
 
-internal enum MariaDBFieldClassifier {
+nonisolated internal enum MariaDBFieldClassifier {
     private static let bitType: UInt32 = 16
     private static let binaryCharset: UInt32 = 63
     private static let blobOrStringTypes: Set<UInt32> = [249, 250, 251, 252, 253, 254]

Plugins/MySQLDriverPlugin/MariaDBFieldMetadata.swift

@@ -6,15 +6,15 @@
 import Foundation
 import TableProPluginKit
 
-internal let mysqlNotNullFlag: UInt = 0x0001
-internal let mysqlPriKeyFlag: UInt = 0x0002
-internal let mysqlBinaryFlag: UInt = 0x0080
-internal let mysqlEnumFlag: UInt = 0x0100
-internal let mysqlAutoIncrementFlag: UInt = 0x0200
-internal let mysqlSetFlag: UInt = 0x0800
-internal let mysqlBinaryCharset: UInt32 = 63
+nonisolated internal let mysqlNotNullFlag: UInt = 0x0001
+nonisolated internal let mysqlPriKeyFlag: UInt = 0x0002
+nonisolated internal let mysqlBinaryFlag: UInt = 0x0080
+nonisolated internal let mysqlEnumFlag: UInt = 0x0100
+nonisolated internal let mysqlAutoIncrementFlag: UInt = 0x0200
+nonisolated internal let mysqlSetFlag: UInt = 0x0800
+nonisolated internal let mysqlBinaryCharset: UInt32 = 63
 
-internal func makeColumnMeta(name: String, typeName: String, flags: UInt) -> PluginColumnInfo {
+nonisolated internal func makeColumnMeta(name: String, typeName: String, flags: UInt) -> PluginColumnInfo {
     PluginColumnInfo(
         name: name,
         dataType: typeName,
@@ -24,7 +24,7 @@ internal func makeColumnMeta(name: String, typeName: String, flags: UInt) -> Plu
     )
 }
 
-internal func mariaDBTypeName(
+nonisolated internal func mariaDBTypeName(
     typeRaw: UInt32,
     flags: UInt,
     charsetnr: UInt32,