[Bug]: New Codex runtime entries can disable existing shadow-home providers #5817

Activity

1. zenDevNomad

   changed the title [-][Bug]: Codex 0.147 queue DB disables shadow-home provider instances[/-] [+][Bug]: New Codex runtime entries can disable existing shadow-home providers[/+] on Aug 9, 2026
2. EwenGG commented on Aug 20, 2026

   

   EwenGG

   on Aug 20, 2026

   More actions

   I can confirm this issue on Windows, with an additional sidecar-only variant that may be relevant to the proposed fix in #7201.

   Environment

   • T3 Code Nightly: 0.0.34-nightly.20260820.1141
   • Codex CLI: 0.148.0
   • OS: Windows 11 Pro, version 10.0.26200, x64
   • Provider setup: two Codex provider instances using one shared home and a separate shadow home for the second account
   • Authentication: ChatGPT subscription authentication
   • The default Codex provider remains usable, but the shadow-home provider is disabled

   Sanitized configuration:

   {
     "driver": "codex",
     "enabled": true,
     "config": {
       "homePath": "C:\\Users\\<user>\\.codex",
       "shadowHomePath": "C:\\Users\\<user>\\.codex_p"
     }
   }

   Observed behavior

   This has recurred across several recent T3 Code Nightly and Codex updates.

   On startup or provider refresh, the secondary Codex provider is disabled during materializeCodexShadowHome. T3 encounters a regular SQLite runtime sidecar in the shadow home where it expects a symbolic link, then aborts provider creation.

   Exact error

   Driver 'codex' failed to create instance:
   Cannot create Codex shadow home entry 'logs_2.sqlite-shm'
   because 'C:\Users\<user>\.codex_p\logs_2.sqlite-shm'
   already exists and is not a symlink.
   

   The corresponding server trace reports:

   CodexShadowHomeEntryConflictError:
   Cannot create Codex shadow home entry 'logs_2.sqlite-shm'
   because 'C:\Users\<user>\.codex_p\logs_2.sqlite-shm'
   already exists and is not a symlink.
   
   at CodexHomeLayout.ensureSymlink
   at materializeCodexShadowHome
   

   Previous occurrences on the same machine failed on other members of the same class of runtime files:

   memories_1.sqlite-shm
   queue_1.sqlite
   logs_2.sqlite-shm
   

   Because materialization stops at the first conflict, the filename displayed by T3 does not represent the complete set of conflicting entries.

   Shadow-home state at failure time

   A complete inspection of the shadow-home root found eight regular SQLite sidecar files:

   Shadow-home entry	Type	Size
   logs_2.sqlite-shm	regular file	32,768 bytes
   logs_2.sqlite-wal	regular file	4,466,112 bytes
   memories_1.sqlite-shm	regular file	32,768 bytes
   memories_1.sqlite-wal	regular file	4,152 bytes
   queue_1.sqlite-shm	regular file	32,768 bytes
   queue_1.sqlite-wal	regular file	4,152 bytes
   state_5.sqlite-shm	regular file	32,768 bytes
   state_5.sqlite-wal	regular file	16,512 bytes

   The problem was therefore not limited to queue_1.sqlite.

   Importantly, the corresponding base database entries were still symbolic links to the shared Codex home:

   logs_2.sqlite       -> <shared-home>\logs_2.sqlite
   memories_1.sqlite   -> <shared-home>\memories_1.sqlite
   queue_1.sqlite      -> <shared-home>\queue_1.sqlite
   state_5.sqlite      -> <shared-home>\state_5.sqlite
   

   In this occurrence, the divergence was specifically:

   base .sqlite file: symbolic link
   -wal sidecar:       regular shadow-local file
   -shm sidecar:       regular shadow-local file
   

   This is slightly different from the original #5817 reproduction, where the shadow base database itself is described as becoming a regular file.

   Recurrence evidence

   Retained local logs show repeated failures on at least two consecutive days:

   • August 19: repeated provider-creation failures on memories_1.sqlite-shm
   • August 20: repeated provider-creation failures on logs_2.sqlite-shm
   • An earlier retained error also references queue_1.sqlite

   The problem has returned after multiple manual recoveries and subsequent updates or restarts. It is not an isolated stale file left by a single interrupted process.

   Different SQLite families become affected at different times. Some sidecars already existed from the previous day, while others were created or rewritten during the next T3/Codex run.

   Impact

   • The complete secondary Codex provider is marked disabled.
   • Provider creation fails before the account can be used.
   • The provider remains unavailable across refreshes and restarts.
   • The first conflicting filename changes between occurrences.
   • Removing only the filename shown in the UI is insufficient because additional regular sidecars may remain.
   • The default provider and shared Codex home continue to work.
   • The secondary account authentication file remains valid.
   • The failure occurs during shadow-home materialization, before normal provider use.

   Recovery confirmation

   For diagnostic purposes, I backed up and moved all eight regular *.sqlite-wal and *.sqlite-shm files out of the shadow home. I did not modify the private auth.json or models_cache.json files.

   I then disabled and re-enabled only the affected provider instance.

   After rematerialization:

   materializeCodexShadowHome: Success
   provider enabled: true
   provider status: ready
   authentication status: authenticated
   SQLite sidecar symlinks: 10
   regular conflicting sidecars: 0
   

   This immediately restored the provider, confirming that the regular SQLite sidecars directly caused provider creation to fail.

   However, the same class of conflict has returned after later updates or runs, so this procedure is only a temporary recovery.

   Relation to PR #7201

   PR #7201 appears to target the same underlying class of failure:

   #7201

   Its stated goal—keeping newly created Codex SQLite state shadow-local and treating a base database plus its WAL/SHM sidecars as one family—matches the recurring behavior observed here.

   However, my captured state includes a sidecar-only divergence: the shadow base .sqlite entry remained a symbolic link while its -wal and -shm entries were regular files.

   The PR description and diff appear to select shadow-local ownership when the shadow base database is already a regular file. I cannot confirm whether it also covers the observed case where only the WAL/SHM members have become regular files.

   This Windows reproduction may therefore be useful for validating #7201 against both forms:

   1. A regular shadow base database plus its sidecars.
   2. A symlinked shadow base database with regular shadow WAL/SHM sidecars.

   I am not proposing a separate implementation here. I am documenting the additional observed state so that the existing fix can be evaluated against the complete failure mode.
3. tgangso commented on Oct 7, 2026

   

   tgangso

   on Oct 7, 2026

   More actions

   Adding updated Windows findings here as requested when #17005 was closed as a duplicate.

   Environment

   • Windows, T3 Code desktop
   • T3 Code: 0.0.46-nightly.20261007.2787
   • Codex CLI: 0.161.0
   • Multiple Codex provider instances sharing a Codex home, with separate shadow homes for authentication

   Observed failure

   Three Codex providers failed to initialize with this sanitized error:

   Driver 'codex' failed to create instance:
   Cannot create Codex shadow home entry 'memories_1.sqlite-shm'
   because '<shadow-home>/memories_1.sqlite-shm'
   already exists and is not a symlink.
   

   Inspection confirmed the sidecar-only state already described in this issue: base SQLite databases were symlinks to the shared home, while some -shm and -wal files were regular files in the shadow homes. Some WAL files were nonempty.

   In the installed build, materializeCodexShadowHome attempts to link shared-home entries, and ensureSymlink rejects these existing regular files. The initial creation of the conflicting files was not captured; no fresh-install reproduction or repository regression test was run.

   Workaround and verification

   The conflicting files were backed up and moved out of the shadow homes, allowing T3 to recreate the links. Each affected provider's launch arguments were also configured to set sqlite_home explicitly to the shared Codex directory.

   After T3 reloaded the settings, all three provider connection probes succeeded. No regular SQLite sidecar conflicts remained at that check. Long-term recurrence has not yet been tested.

   This confirms the failure on the versions above and records the observed recovery. It is not a recommendation to delete SQLite sidecars or a claim that a permanent upstream fix has been verified. Related shared-directory approach: #9229.
4. dustenhubbard commented on Oct 8, 2026

   

   dustenhubbard

   on Oct 8, 2026

   More actions

   Another entry for this issue, from Codex 0.161.0: cloud-config-bundle-cache.json. This one should not be linked, either.

   Environment

   • T3 Code: 0.0.46-nightly.20261007.2774 (desktop)
   • Codex CLI: 0.161.0
   • macOS 27.0.1, Apple Silicon
   • Two Codex accounts: the default ~/.codex, and a second instance with shadowHomePath set to ~/.codex-t3/work

   What happened

   After a T3 restart, the second instance showed only "Disabled" in Settings → Providers, with its toggle still on. Its status cache had stopped updating at the restart. The settings were unchanged.

   cloud-config-bundle-cache.json existed as a regular file in both the shared home and the shadow home. In the installed build, PRIVATE_ENTRY_NAMES is auth.json and models_cache.json, so materializeCodexShadowHome tries to link this file and ensureSymlink rejects the regular copy. I did not find the error text in the server trace. I am going by the file state and the code.

   Why linking is the wrong fix for this file

   The file is per-account. It holds chatgpt_user_id, account_id, and the workspace's signed enterprise config_toml and requirements_toml, with a one-hour expiry. If it were shared, one account could pick up another account's workspace policy. It belongs with auth.json and models_cache.json.

   Linking also does not last. I deleted the shadow copy and turned the provider off and on. The provider came back, but within seconds there was a regular file in the shadow home again, and the shared copy had not changed. So the next restart would disable the provider again.

   Workaround

   A small job deletes the shadow copy every two minutes, so a restart does not find one. Codex fetches the bundle again when it needs it. This only matters until the file is treated as private.

   Suggested fix

   Add cloud-config-bundle-cache.json to PRIVATE_ENTRY_NAMES. This is a narrower case than the fix this issue asks for. The general fix should also not default to linking: a new per-account file is safer kept local than shared.
5. TimeAndLocation commented on Oct 13, 2026

   

   TimeAndLocation

   on Oct 13, 2026

   More actions

   Windows reproduction: split WAL files can lose committed writes

   Adding a data-loss reproduction for the sidecar-only variant already described here.

   Environment: Windows 11 x64 (10.0.26200), T3 Code Nightly 0.0.46-nightly.20261013.3040, Codex CLI 0.162.1, Node 24.14.0. Two Codex accounts were configured with a shared home and a shadow home.

   In addition to provider-creation conflicts (db-backups in this installation), archived Codex databases failed SQLite integrity checks. Codex's retained logs explicitly reported corruption in thread_history_1.sqlite. The precise historical corruption timing was not captured, so the fixture below establishes a reproducible unsafe write mechanism rather than proving the origin of every damaged page.

   Reproduction

   A fixture invoking T3's actual materializeCodexShadowHome implementation from the installed server reproduced this sequence:

   1. Create and close a WAL-mode SQLite database in the shared home. At materialization time its -wal and -shm files do not exist.
   2. Materialize the shadow. T3 links the base database, but there are no sidecars to link.
   3. Open the shared database and commit a row, keeping that connection open.
   4. Open the same physical database through the shadow path and commit a different row.
   5. Close both connections and reopen the shared database.

   Both inserts succeed, but only one of the two committed rows remains. The shadow connection uses its own regular WAL/SHM files for the linked base database. This does not require overlapping transactions, a crash, an authentication failure, or model inference.

   Here is a standalone Node reproduction of the resulting file layout. Run as an .mjs file on Windows with file-symlink support. It uses only a fresh temporary directory:

   import { DatabaseSync } from 'node:sqlite';
   import { mkdtempSync, mkdirSync, symlinkSync } from 'node:fs';
   import { tmpdir } from 'node:os';
   import { join } from 'node:path';
   import assert from 'node:assert/strict';
   
   const root = mkdtempSync(join(tmpdir(), 't3-shadow-wal-'));
   const shared = join(root, 'shared');
   const shadow = join(root, 'shadow');
   mkdirSync(shared);
   mkdirSync(shadow);
   const sharedDb = join(shared, 'state_5.sqlite');
   const shadowDb = join(shadow, 'state_5.sqlite');
   
   const seed = new DatabaseSync(sharedDb);
   seed.exec('PRAGMA journal_mode=WAL; CREATE TABLE writes (id INTEGER PRIMARY KEY)');
   seed.close();
   // This is the database layout materializeCodexShadowHome creates when sidecars are absent.
   symlinkSync(sharedDb, shadowDb, 'file');
   
   const a = new DatabaseSync(sharedDb);
   a.exec('INSERT INTO writes VALUES (1)');
   const b = new DatabaseSync(shadowDb);
   b.exec('INSERT INTO writes VALUES (2)');
   b.close();
   a.close();
   
   const check = new DatabaseSync(sharedDb);
   const rows = check.prepare('SELECT id FROM writes ORDER BY id').all();
   check.close();
   console.log(rows);
   assert.equal(rows.length, 2, 'A committed write was lost');

   Observed: the assertion fails with 1 !== 2. The same write check with two independent physical provider databases retains both writes. The actual-T3 fixture also retained 1/2 with the shadow layout and 2/2 with independent homes.

   The relevant main-branch code still enumerates only entries currently present in the shared home: CodexHomeLayout.ts. SQLite explains why multiple names and separate journals/WAL files are unsafe.

   An acceptance test should cover a linked base database whose sidecars were absent during materialization, then verify both committed writes survive. Handling an existing regular sidecar conflict alone would not establish database safety. Independent direct homes were used as the local workaround, preserving the account credentials and available rollouts; this does not claim to preserve the intended ongoing shared-history behavior of shadow homes.

   Reproduction and diagnosis assisted by GPT-6.1-Sol in Codex desktop.
6. tgangso commented on Oct 13, 2026

   

   tgangso

   on Oct 13, 2026

   Last edited by tgangso

   More actions

   Follow-up to my earlier report: the providers became unavailable again on Windows, but this time with a different shadow-home failure mode.

   Environment

   • T3 Code: 0.0.46-nightly.20261013.3066 (desktop, Nightly)
   • Codex CLI: 0.162.1 (verified with the CLI configured in T3)
   • Windows 11 Pro 25H2, build 26200.9457, x64
   • Three enabled Codex account instances sharing one homePath (<user-home>/.codex), each with a separate shadowHomePath (<t3-userdata>/providers/codex/<account-instance>/shadow). Paths and account identifiers are placeholders.
   • All three instances retained an explicit sqlite_home override pointing to the shared Codex directory, with cli_auth_credentials_store="file".

   New findings

   • The explicit sqlite_home override pointing to the shared Codex directory was still configured on all three affected providers.
   • The initial failure was a CodexShadowHomeFileSystemError wrapping EPERM: operation not permitted, symlink. T3 attempted to link a newly present, non-runtime configuration backup from the shared home into each shadow home. That single file blocked initialization of all three providers.
   • Unlike the earlier regular-file conflicts, inspection found no conflicting regular shadow entries at that point; the required links were missing.
   • After relocating the backup outside the mirrored directory and reloading the providers, two authenticated successfully. The remaining provider then failed to create links for memories_1.sqlite-shm and memories_1.sqlite-wal.
   • A direct attempt to create those missing links without administrator privileges returned Windows error 1314: a required privilege is not held by the client. Creating only those two links with administrator permission and reloading the remaining provider restored it.

   Verification and implication

   All three affected providers subsequently showed Authenticated in T3, and the provider connection probes succeeded. The backup was preserved with its contents verified unchanged.

   The sqlite_home override alone did not prevent T3's shadow-home initialization from requiring links for SQLite sidecars and other shared-home entries. This is a related Windows link-creation/permissions failure, distinct from the existing regular-file conflict reported earlier. It may warrant separate tracking at maintainers' discretion.

   This was a local recovery, not a verified permanent fix. No database-integrity testing was performed, and I am not claiming data loss or corruption in this installation. Personal paths, account details, and identifiers are omitted.