[IBKR] Dead Gateway connection reports health=healthy; orders accepted but never transmitted until manual reconnect (no keepalive) #294

Activity

1. luokerenx4

   added a commit that references this issue on Jun 12, 2026

   fix(ibkr,ui): community-issue sweep — currency-keyed account values (#…

   ...

   c277433
2. luokerenx4 commented on Jun 13, 2026

   

   luokerenx4

   on Jun 13, 2026

   Contributor

   More actions

   Implemented in #335 (merged) — needs verification on a real Gateway setup, and yours is the canonical repro environment.

   Root cause confirmed, plus one mechanism beyond your analysis: the IBKR account surface is cache-backed, so routine health queries never touch the socket at all — a dead-but-idle connection is invisible to failure counting even in principle. That's why health: healthy survived a dead socket indefinitely.

   What landed:

   • Active heartbeat: reqCurrentTime every 45s with a 5s timeout; a failed beat marks the connection dead.
   • connectionClosed() marks dead immediately (previously it only rejected in-flight requests).
   • While dead, everything refuses loudly: cache reads (getAccount/getPositions) AND all order paths (place/modify/cancel) return "TWS/Gateway connection lost — reconnect pending… orders will NOT transmit" instead of accepting orders into the void.
   • init() force-tears-down a half-open socket — isConnected() lies on half-open TCP, which previously made the recovery loop's re-init a silent no-op.

   Not yet done from your list: treating error 1102 as an explicit recovery trigger (recovery currently rides the existing re-init loop, which the dead flag now actually un-blocks). Specs cover the gate wiring; what we could NOT reproduce locally is the silent half-open drop itself (we run TWS, not Gateway with daily restarts). If you can run your repro — Gateway daily auto-restart while idle — against master and confirm (1) health leaves healthy, (2) order placement refuses during the window, (3) it recovers without manual reconnect, we'll close this.
3. rudyll commented on Jun 13, 2026

   

   rudyll

   on Jun 13, 2026

   Author

   More actions

   Verified on master (#335), IB Gateway 10.47 paper — reproduced the silent half-open deterministically (SIGSTOP the Gateway → socket stays ESTABLISHED, peer mute, same shape as the idle daily-restart drop → SIGCONT).

   Confirmed working:

   • The heartbeat catches the half-open: connectionDead flips ~50s in, getAccount() then refuses with the new 503 instead of serving stale cache forever.
   • The gate is firm — it rides connectionDead, which is cleared only by a real reconnect (markAlive in init()), so cached reads can't un-dead it. (Health status still flaps healthy↔offline because consecutiveFailures resets on any cached success — cosmetic, the gate doesn't follow it.)
   • Auto-recovers on reconnect, unattended. So ∞ → ~50s. 👍

   One residual gap, order path: _ensureAlive() rides connectionDead, which for a silent half-open (no FIN/RST → connectionClosed never fires) is only set by the 45s heartbeat. So there's a ≤50s window right after the drop where placeOrder/modify/cancel aren't refused and transmit into the dead socket — exactly the daily-restart-while-idle case, narrowed but not closed.

   Suggested fix: on the transmit path only, do a synchronous liveness probe — a reqCurrentTime round-trip with a short timeout — immediately before sending, and refuse on timeout. Reads can stay cache-backed; only the irreversible order path needs confirm-then-send rather than trusting a flag that can be ≤50s stale. Closes the window without raising heartbeat frequency. (For what it's worth, we gate our own order path with a forced reconnect-and-verify before every push — confirmed it refuses inside the blind window even while health still reads healthy — so we're covered, but anything placing orders straight through, e.g. the UI, isn't.)

   Repro recipe (deterministic, no nightly wait): kill -STOP <gateway-pid> → wait ~50s → heartbeat marks dead, reads 503 → kill -CONT → recovers. Faithful silent half-open without waiting for the real auto-restart.

   (error 1102 as an explicit recovery trigger is still open on our side — recovery rode the existing re-init loop fine without it here.)
4. luokerenx4 commented on Jul 17, 2026

   

   luokerenx4

   on Jul 17, 2026 – with ChatGPT Codex Connector

   Contributor

   More actions

   Implemented the remaining gap in #658 (merged to dev, merge commit fa4c2795).

   What was still wrong

   #335 correctly added a 45s heartbeat and a hard connectionDead gate, but the order path still trusted that flag. During a silent half-open, the flag can be up to heartbeat + timeout (~50s) behind reality. A place/modify/cancel inside that window could therefore write to a mute socket before the next heartbeat marked it dead.

   There was also a presentation/state issue above the adapter: the broker could know the transport was dead while UTA health still depended on passive failure counting. Because IBKR account reads are cache-backed, that is not a reliable health signal.

   What changed

   • placeOrder, modifyOrder, and cancelOrder now require a reqCurrentTime round-trip on the same ordered socket immediately before the client send.
   • The write probe has a 3s timeout. On failure the adapter marks the transport dead, returns an explicit "order was not transmitted" error, and never calls placeOrder/cancelOrder on the client.
   • Concurrent heartbeat/write probes share one in-flight current-time request, so the single callback slot cannot be overwritten.
   • connectionClosed, 502/504/1100, and heartbeat failure publish an authoritative dead signal to UTA. UTA goes directly to offline/down and starts recovery instead of waiting for six caller-driven cache failures.
   • 1101/1102 only nudge recovery immediately; they do not mark the account healthy. Reconnect plus the normal private account read must succeed before health returns to healthy/readable.
   • Synchronous socket-write failures and timeouts clear the probe slot so a later reconnect can retry normally.

   Verification performed

   Deterministic specs cover:

   • probe happens before every place/modify/cancel send;
   • a failed probe marks dead and the client send is never called;
   • concurrent current-time probes coalesce;
   • 1100 is immediately dead, while 1102 is only a recovery nudge;
   • UTA health moves directly offline and becomes green only after the recovery ladder passes.

   Live read-only acceptance used IB Gateway 10.37.1r / serverVersion 203 in a paper account:

   1. Started healthy/readable.
   2. Froze the Gateway container with docker pause (TCP remained half-open).
   3. At 46s UTA changed directly to offline/down/recovering.
   4. The account endpoint refused with HTTP 503 instead of serving cached data.
   5. Unpaused Gateway; UTA returned to healthy/readable in 2s and account/position reads returned 200.
   6. No order endpoint was called.

   Full repository tests passed (3161 tests); the targeted UTA/IBKR suite passed 151 tests.

   Reproduction invitation

   Could you please retry against dev or a build containing fa4c2795?

   Safe half-open check:

   1. Use a paper account (read-only API is sufficient for the health/recovery part).
   2. Freeze Gateway without closing TCP: docker pause <container> or SIGSTOP the Gateway process. Install a trap/cleanup so it is always resumed.
   3. Expect health to leave healthy within about 50s, cached account reads to refuse once dead is known, and automatic recovery after unpause/SIGCONT.
   4. For the residual write-window check, use paper only: immediately after freezing Gateway, attempt place/modify/cancel. It should fail after the short liveness probe with "order was not transmitted"; after resuming Gateway, no queued order should appear. Please do not run this part against a live-money account.

   If you can report the TWS/Gateway version, API serverVersion, transition timings, and sanitized error text, that will let us compare environments. Please omit account IDs, balances, orders, and raw logs.

   I am leaving the issue open for external confirmation; if the paper repro behaves differently, this comment should also provide the exact path to investigate next.
5. RainMona commented on Aug 17, 2026

   

   RainMona

   on Aug 17, 2026

   Contributor

   More actions

   Community triage (current dev, Aug 2026)

   Status: likely fixed in code — needs a final reporter retest

   Current dev includes both layers discussed in-thread:

   • fix(uta,cli,ibkr): live-testing rounds 6+7 — alpaca bracket legs + IBKR acceptance run #335 heartbeat / connectionDead gate
   • fix(ibkr): close liveness and option-mark gaps #658 write-path reqCurrentTime probe before place/modify/cancel, plus UTA moving offline on transport death

   That should close the silent half-open window where orders could be accepted into a mute socket.

   Ask @rudyll (or anyone with Gateway): please re-run the deterministic recipe on current dev (SIGSTOP / docker pause, then attempt place/modify/cancel during the freeze):

   • health should leave healthy within ~50s
   • reads should 503 once dead
   • orders should fail with “not transmitted” after the short probe
   • resume should auto-recover

   If that matches, we can close.

   — triage note
6. eutialia

   added 2 commits that reference this issue on Sep 5, 2026

   fix(uta): make IBKR liveness a coarse failure detector

   ...

   a7ff398

   fix(ibkr): make transport liveness a coarse failure detector

   ...

   8f5c654
7. eutialia

   added a commit that references this issue on Sep 29, 2026

   fix(ibkr): harden transport liveness and UTA recovery against gateway…

   ...

   a533a60
8. luokerenx4

   added

   status:needs-acceptance

   on Oct 2, 2026