Skip to content

[Bug]: T3 Connect doc says held webhooks are kept per environment, but the relay keeps one inbox per link #16995

Description

@coygeek

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

Docs

Steps to reproduce

Summary: The held-webhooks paragraphs of docs/internals/t3-connect.md describe the relay's offline webhook store as one Durable Object per environment, deleted "when no user has the environment linked", with "per-environment order, caps, and retry timing". The relay actually keeps one HookInboxObject per managed link (one account's link of one environment), named by that link's endpoint key. When two accounts link the same environment, each link gets its own inbox, caps and ordering, and unlinking deletes only the unlinking account's held requests. The doc also omits that turning hold_webhooks_while_offline off deletes what is already held. This is a documentation defect only; the relay's per-link design is the one the HookInbox.ts, HookInboxObject.ts and deploymentConfig.ts comments describe and the relay tests exercise. Related but separate: #16988 covers the wrong :environmentId path segment in lines 7-8 of the same doc; this report covers lines 13-30 only.

  1. On the main build, read docs/internals/t3-connect.md lines 13-30.
  2. Read the inbox naming: infra/relay/src/hooks/HookInbox.ts lines 17-21 and infra/relay/src/worker.ts lines 246-253, where hold, wake and clear all call hookInboxes.getByName(endpointKey).
  3. Read what an endpoint key identifies: infra/relay/src/deploymentConfig.ts lines 128-132, and infra/relay/src/environments/ManagedEndpointProvider.ts lines 998-1022, where the tunnel name's hash is a SHA-256 of managedEndpointDigestInput(namespace, userId, environmentId).
  4. Read when held requests are deleted: infra/relay/src/http/Api.ts lines 539-558 (unlink clears only the unlinking link's endpoint key) and infra/relay/src/hooks/HeldHooks.ts lines 95-126 (opting out clears only the caller's own links' endpoint keys).
  5. Run the relay tests that exercise this behavior; the command and its output (per-test durations removed) are under Logs or stack traces.

Expected behavior

docs/internals/ records architectural decisions and constraints a maintainer would otherwise get wrong (repository AGENTS.md, Documentation section). The held-webhooks paragraphs should therefore name the unit the relay actually stores, orders and caps held requests by, and when it deletes them: one inbox per managed link (endpoint key), deleted when that link is unlinked, when its environment opts out of holding, after 24 hours, or once delivered.

Actual behavior

The doc text on the main build:

16: raw request, including the hook token in the path, in a Durable Object for
17: that environment, with SQLite storage. ...
20: relay to deliver right away. Requests are deleted once the environment
21: answers, after 24 hours, or when no user has the environment linked. ...
28: read-once bodies of up to 1 MiB that need per-environment order, caps, and
29: retry timing. Queues cap messages at 128 KB and cannot hold one environment's
30: requests back while it is away.

The code on the same build:

  • infra/relay/src/hooks/HookInbox.ts lines 17-21: "Each managed endpoint's requests live in its own HookInboxObject, named by endpoint key". infra/relay/src/hooks/HookInboxObject.ts line 27: "One per managed endpoint, addressed by endpoint key."
  • infra/relay/src/deploymentConfig.ts lines 128-132: "The hash covers user and environment, so one key names exactly one link, unlike the environment id, which any account can claim."
  • infra/relay/src/hooks/HeldHooks.ts lines 95-98, on ownEndpointKeys: "The environment id alone would also match other accounts' links of it." setHoldWhileOffline (lines 118-126) clears each own endpoint key's inbox when holding is turned off; wake (lines 128-138) also works per endpoint key.
  • infra/relay/src/http/Api.ts lines 539-558, in unlink: "Requests held for this link's endpoint go with it." It calls inbox.clear({ endpointKey }) for the unlinking account's endpoint only.
  • infra/relay/src/hooks/HookInboxStore.ts lines 17-27: the 24-hour TTL and the 1,000-request, 50 MiB and 100-per-hook caps are enforced inside each HookInboxObject, so per endpoint key. The comments on these constants still say "one environment".

The step 5 test run passed; its output is under Logs or stack traces.

The HeldHooks fixture links one environment to two accounts with distinct endpoint keys (and distinct environment public keys), and opting out clears only the caller's endpoint key; the Api test shows unlink clearing only the unlinked link's endpoint key. Storage, caps and unlink deletion are therefore per link. Opting out clears every active link proven by the calling environment's key (infra/relay/src/environments/EnvironmentLinks.ts lines 374-404 and 432-445), which is still per link rather than every account's link of the environment id. The doc's per-environment description is wrong in three places: the storage unit (line 17), the deletion condition (lines 20-21), and the rationale (lines 28-30).

Evidence

  • Expected source: repository AGENTS.md, Documentation section (docs/internals/ records decisions and hard-to-discover constraints), together with the relay's own contract in the HookInbox.ts and deploymentConfig.ts comments cited above.
  • Failure source: docs/internals/t3-connect.md lines 13-30, compared with infra/relay/src/worker.ts lines 246-253, HeldHooks.ts lines 95-138 and Api.ts lines 539-558.
  • Evidence provenance: observed
  • Local verification: reproduced
  • Reproduction completeness: complete

The mismatch between the doc and the code was established by reading both on the main build. Per-link clearing was observed by running the existing relay unit tests, which use a mocked inbox. That HookInboxObject instances are named by endpoint key comes from reading worker.ts. A deployed relay with two accounts linking one environment was not exercised. Some relay code comments use the same per-environment wording as the doc: HookInboxStore.ts lines 11-21 and HookInbox.ts line 25 ("the environment's inbox"). git blame shows lines 13-30 unchanged since #15487 added them, and HookInbox.ts in that same change already named inboxes by endpoint key, so the doc has never matched the code.

Restoration check

Failing: docs/internals/t3-connect.md says held requests are stored in a Durable Object per environment, deleted when no user has the environment linked, with per-environment order and caps. Passing: the doc identifies the managed link (endpoint key) as the unit of storage, ordering and caps, and states deletion conditions that match Api.ts unlink and HeldHooks.setHoldWhileOffline. The step 5 tests still pass, showing the doc was changed to match the relay rather than the reverse.

Triage assessment

  • Impact level: P3
  • Assessment status: supported
  • Impact basis: Only internal maintainer documentation is wrong; relay behavior is unaffected. A reader would misjudge which held requests survive another account's unlink and how caps are shared, but nothing in the product follows this text.
  • Workaround status: available
  • Workaround basis: The comments in HookInbox.ts lines 17-21, HookInboxObject.ts, deploymentConfig.ts, HeldHooks.ts and Api.ts, the getByName(endpointKey) calls in worker.ts, and the relay tests show the per-link behavior today; a reader should not rely on the per-environment wording in the HookInboxStore.ts comments.

Impact

Cosmetic issue

Version or commit

main @ 3143335 (2026-10-07); doc and relay hook code identical in v0.0.46-nightly.20261007.2787

Environment

Logs or stack traces

pnpm install --frozen-lockfile
cd infra/relay
npx vp test run src/hooks/HeldHooks.test.ts src/http/Api.test.ts -t "own endpoints|unlinked endpoint|wakes the caller" --reporter=verbose
✓ src/hooks/HeldHooks.test.ts > HeldHooks > opting out clears only the caller's own endpoints
✓ src/hooks/HeldHooks.test.ts > HeldHooks > wakes the caller's ready endpoints and reports whether anything was waiting
✓ src/http/Api.test.ts > relay environment unlink > drops the unlinked endpoint's held webhooks, even if clearing fails
Test Files  2 passed (2)
     Tests  3 passed | 33 skipped (36)

Screenshots, recordings, or supporting files

No response

Workaround

The comments in HookInbox.ts lines 17-21, HookInboxObject.ts, deploymentConfig.ts, HeldHooks.ts and Api.ts, the getByName(endpointKey) calls in worker.ts, and the relay tests show the per-link behavior today; a reader should not rely on the per-environment wording in the HookInboxStore.ts comments.

Activity

  1. juliusmarminge commented on Oct 7, 2026

    @juliusmarminge
    Member

    Note

    Grok responding on behalf of Julius.

    Thanks, confirmed on main (3143335). This is a docs-only mismatch: the relay keeps held webhooks per managed link (endpoint key), not per environment.

    What the doc says (docs/internals/t3-connect.md#L13-L30):

    • L16-17: the relay stores the request "in a Durable Object for that environment".
    • L20-21: requests are deleted "once the environment answers, after 24 hours, or when no user has the environment linked".
    • L28-30: held requests "need per-environment order, caps, and retry timing".

    What the code does:

    • Inbox is keyed by endpoint key: worker.ts#L248-L252 opens every hold, wake and clear call with hookInboxes.getByName(endpointKey). HookInbox.ts#L17-L21 says "Each managed endpoint's requests live in its own HookInboxObject, named by endpoint key", and HookInboxObject.ts#L27 says "One per managed endpoint, addressed by endpoint key."
    • One endpoint key per link: the key is the hash suffix of the managed tunnel name. That hash is SHA-256 of ${stage}:${userId}:${environmentId} (ManagedEndpointProvider.ts#L998-L1022, deploymentConfig.ts#L88-L94). deploymentConfig.ts#L128-L132 says "one key names exactly one link, unlike the environment id, which any account can claim."
    • Unlink clears only that link's inbox: Api.ts#L539-L558 says "Requests held for this link's endpoint go with it", then calls inbox.clear({ endpointKey }) for the unlinked link's tunnel. The clear is best effort; on failure it logs a warning.
    • Turning holding off clears what's already held: in HeldHooks.ts#L118-L126, setHoldWhileOffline calls inbox.clear({ endpointKey }) for each of the caller's own endpoint keys when holdWebhooksWhileOffline is false. Per #L95-L98, those are the "endpoint keys of the managed links the calling environment key proved". The current doc doesn't mention this.

    Suggested doc changes (L16-30):

    • L16-17: store the request "in a Durable Object for that link's managed endpoint (named by its endpoint key)".
    • L20-21: requests are deleted "once the environment answers, after 24 hours, when that link is unlinked, or when the environment turns hold_webhooks_while_offline off".
    • L28-30: "per-endpoint (per-link) order, caps, and retry timing" and "hold one endpoint's requests back".

    Optional comment cleanup: some comments use the same per-environment wording: HookInboxStore.ts#L11-L21 ("one environment's held webhook requests", "the environment's HookInboxObject", and the per-environment caps) and HookInbox.ts#L25 ("the environment's inbox").

    Only docs and comments are affected; no runtime change is needed. #16988 covers the wrong :environmentId path segment on L8 of the same doc, so both fixes could land in one docs PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationvia-triageFiled through npx t3 triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions