[4.41/SH1] Shared guarded downloader (redirect policy, byte cap, timeout, credential host binding)

[patrick-chinchill]

Summary

Port upstream's shared guarded attachment downloader (@chat-adapter/shared download.ts) as chat_sdk/shared/download.py: HTTPS only, internal addresses refused (literal and after DNS), every redirect hop re-validated, per-hop header policy so credentials never follow a redirect to an untrusted host, decoded-body cap, overall deadline, injectable transport. Helper and tests only; adapters adopt it in their own issues.

Upstream changes

• bb926884 fix(teams): secure attachment downloads (#850) — chat@4.39.0 — introduces downloadAttachment, validateAttachmentUrl, createResolver and the transport option: literal HTTPS/internal check, DNS-pinned resolver rejecting any internal answer, manual redirects (max 5), 25 MB decoded cap, 30s deadline, gzip/deflate/br, NetworkError for every failure.
• 153bd964 fix(messenger): guard attachment downloads (#856) — chat@4.39.0 — hosts allowlist option (case-insensitive exact-or-subdomain match), enforced on every redirect hop.
• b6fa24c6 fix(adapters): guard attachment downloads across slack, discord, telegram, and whatsapp (#865) — chat@4.39.0 — per-hop headers (mapping or function of the URL) and the onResponse pre-body hook.
• 7c269653 fix(adapters): restrict attachment credentials to trusted hosts (#859) — chat@4.39.0 — Slack/WhatsApp send credentials only to trusted hosts (the policy per-hop headers expresses). No shared-code change.
• 6adca361 feat(slack): support egress proxies (#916) — chat@4.41.0 — shared slice: the downloader itself enforces the deadline (response wait and body read), so it holds for a caller-supplied transport that ignores cancellation.

Current Python behavior

• grep -rn 'validate_attachment_url\|ipaddress' src/chat_sdk → nothing; the only download_attachment hit is Messenger's private _download_attachment (adapters/messenger/adapter.py:890). No shared helper and no address-range checks.
• Per-adapter fetches have no size cap, deadline or internal-address check:
  • Slack: allowlist adapters/slack/adapter.py:3493; httpx fetch :3520-3545 (return resp.content).
  • Teams: allowlist adapters/teams/adapter.py:1269; :1325-1331 aiohttp session.get(url) (follows redirects by default), resp.read().
  • Messenger: adapters/messenger/adapter.py:890-898 session.get(url), no host check.
  • Telegram: adapters/telegram/adapter.py:2415-2421.
  • WhatsApp: adapters/whatsapp/adapter.py:680-749 (suffix allowlist :717-725).
• NetworkError(adapter, message, original_error) exists at shared/errors.py:72-81.
• aiohttp is declared in 9 platform extras (plus all/dev) but not in slack; Slack lazily imports httpx, which pyproject.toml does not declare.

Scope

• src/chat_sdk/shared/download.py: DEFAULT_LIMIT = 25 * 1024 * 1024, DEFAULT_REDIRECTS = 5, DEFAULT_TIMEOUT_MS = 30_000, REDIRECT_STATUSES = {301, 302, 303, 307, 308}; the IPv4/IPv6 blocked-range tables copied exactly from upstream download.ts; is_blocked_address(ip: str) -> bool.
• validate_attachment_url(url, adapter, hosts=None) -> str (normalized URL): internal literal → NetworkError(adapter, "Refusing to fetch an internal attachment URL"); non-HTTPS or off-allowlist → "Refusing to fetch an untrusted attachment URL".
• create_resolver(adapter, query=None): rejects when any resolved address is blocked; an empty answer → "Could not resolve the attachment host"; query injectable for tests.
• AttachmentTransport + AttachmentResponse Protocols (status, reason, headers, async chunk iterator, close()). Default transport: aiohttp (lazy import), pinned resolver, allow_redirects=False.
• async download_attachment(url, *, adapter, headers=None, hosts=None, limit=DEFAULT_LIMIT, on_response=None, redirects=DEFAULT_REDIRECTS, timeout_ms=DEFAULT_TIMEOUT_MS, transport=None) -> bytes:
  • headers is a mapping or Callable[[str], Mapping | None], evaluated per hop; defaults accept-encoding and user-agent: Vercel.ChatSDK;
  • relative Location resolved against the current URL and re-validated; missing Location → "Attachment redirect has no location"; past the limit → "Too many attachment redirects";
  • non-2xx → f"Failed to fetch file: {status} {reason}".strip();
  • on_response runs before the body is read; the response is closed if it raises.
• Body reader: unsupported content-encoding → "Unsupported attachment encoding: X"; Content-Length precheck only for identity encoding ("Attachment exceeds the download limit"); running cap on decoded bytes; body longer than declared → "Attachment body exceeds its declared length".
• Re-export from chat_sdk.shared. No adapter changes.

Out of scope

Adapter adoption and host lists: #213 (Slack, incl. HTML-login check via on_response, proxy config, Grid/GovSlack hosts), #218, #225, #239 (WhatsApp/Messenger), #229 (Discord). Minimal Messenger fix: #234.

Porting notes

• Security class. SSRF and resource exhaustion via attachment.fetch_data() on payload-supplied URLs. Port exactly upstream's checks and test cases; do not document bypass techniques in comments, docs or the PR.
• Resolve-then-connect (no TOCTOU). Recommended default: aiohttp TCPConnector(resolver=<AbstractResolver subclass>) that validates every answer and returns only vetted addresses, so the socket connects to exactly what was checked (SNI/cert still use the hostname). One connector per call (force_close=True) so pooled connections cannot bypass it.
• Literal-host normalization. urllib.parse does not canonicalize numeric IPv4 hosts like WHATWG URL. Normalize bracketed IPv6 and numeric IPv4 notations (e.g. ipaddress + socket.inet_aton) before the range check so every upstream "rejects internal file URL %s" case passes.
• Allowlist. host == allowed or host.endswith("." + allowed) on the lowercased hostname; keep upstream's trailing-dot rejection.
• Credentials. Upstream sends a static headers mapping on every hop. Recommended Python-only hardening (recorded in docs/UPSTREAM_SYNC.md): for a static mapping, drop authorization/cookie/proxy-authorization on hops whose origin differs from the first URL; the function form stays caller-controlled.
• Deadline. asyncio.timeout(timeout_ms / 1000) around redirects + body read; map TimeoutError to NetworkError(adapter, "Timed out fetching the attachment"). Never swallow an outer CancelledError; close responses/session in finally.
• Decoding. aiohttp decodes gzip/deflate, br only with Brotli; advertise br only when decodable. response.content.iter_chunked() yields decoded bytes, so the cap applies after decompression.
• Transport. A caller-supplied (proxy) transport skips the pinned resolver; literal/scheme/allowlist checks still run every hop. aiohttp is a lazy import inside the default transport. [4.41/SL8] Slack: guarded downloads, Enterprise Grid org-wide installs, egress proxy #213 chooses between aiohttp in the slack extra and an httpx transport.
• Error strings must match upstream exactly.

Tests

packages/adapter-shared/src/download.test.ts [guarded attachment downloads] is not fidelity-mapped. Port to new tests/test_shared_download.py; add a MAPPING row if #185's it.each expansion has landed.

• it.each groups (one test per case): "accepts public HTTPS host %s"; "rejects non-HTTPS URL %s"; "rejects internal file URL %s with a network error"; "accepts allowlisted host URL %s"; "rejects off-allowlist URL %s".
• it: "returns the validated DNS results to the socket"; "rejects mixed public and internal DNS results"; "reports an empty DNS result as a resolution failure, not a refusal"; "rejects hostnames that resolve to internal addresses"; "applies the host allowlist to redirect targets"; "follows redirects between allowlisted hosts"; "resolves headers per hop and drops credentials on redirects"; "rejects responses that fail the onResponse check"; "rejects redirects to internal addresses"; "follows redirects to other public HTTPS hosts"; "rejects redirect chains past the redirect limit"; "rejects redirects without a location header"; "rejects error statuses"; "times out slow downloads with a distinct error"; "times out a transport that never responds and ignores the signal"; "times out a body read that the transport does not tie to the signal"; "decodes gzip response bodies"; "applies the download limit to decompressed bytes"; "rejects unsupported content encodings"; "stops reading attachments at the download limit"; "rejects declared sizes over the limit before reading"; "reads declared-size bodies into a single buffer"; "rejects bodies that exceed their declared length".
• Fake in-memory transport + fake resolver query (no network); deadline tests await a never-resolved Future with a tiny timeout_ms, no asyncio.sleep.
• Python-specific: numeric-host normalization; static-mapping credential stripping on a cross-origin hop; import works without aiohttp (monkeypatch sys.modules).

Acceptance criteria

• Full validation command from CLAUDE.md passes.
• Every upstream download.test.ts case above is ported, with exact error strings.
• docs/UPSTREAM_SYNC.md records resolver pinning via aiohttp, the credential-stripping hardening, and that the allowlist row (:666) is revisited as adapters adopt the helper.
• CHANGELOG entry under "Unreleased (4.41 wave)" (security label).
• Consumer-visible changes: none here; note the 25 MB / 30s defaults apply on adoption.

Dependencies

Blocked by #185. Blocks #213, #218, #225, #239, #229.

Metadata

• Effort: L
• Consumer impact: none here; Slack/Teams downloads gain a 25 MB cap and 30s deadline when [4.41/SL8] Slack: guarded downloads, Enterprise Grid org-wide installs, egress proxy #213/[4.41/T3] Teams inbound: author email hydration, protected inline attachments, secure downloads #218 land.
• Suggested branch: sync/4.41-sh1

Part of #184.