Skip to content

fix: surface the manager's own error for every failed manager request - #976

Merged
alongubkin merged 3 commits into
mainfrom
alon/alien-1379-manager-error-bodies
Oct 10, 2026
Merged

alongubkin merged 3 commits into
mainfrom
alon/alien-1379-manager-error-bodies

Conversation

@alongubkin

@alongubkin alongubkin commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

Problem

alien deployments get printed only Unexpected response: 400 Bad Request when the manager answered with a structured error (code, message, source). Most other manager-backed commands had the same problem.

Root cause

The progenitor-generated manager client keeps only the status for most failures:

  • a status the route's OpenAPI does not list (400 on GET /v1/deployments/{id}) arrives as Error::UnexpectedResponse with the body unread, and the sync SdkResultExt::into_sdk_error never read it;
  • a status listed without a body schema (401/403/404 on most routes) becomes Error::ErrorResponse(ResponseValue::empty(..)), so the generated code throws the body away.

An async body-reading adapter (into_sdk_error_reading_body) existed but was opt-in per call site. Earlier fixes converted one site at a time; about 60 sites still used the sync path.

Alternatives considered

  • Declare 400 (and other statuses) on each route's utoipa annotation: fixes only the routes someone remembers, and a typed error schema turns a non-JSON error body (proxy HTML 502) into InvalidResponsePayload, which loses the status and breaks retry decisions.
  • Switch more call sites to the opt-in adapter: same per-site approach that left this gap, and it can't recover bodies of listed body-less statuses.

Fix

  • client-sdks/manager/scripts/fix-openapi.mjs removes every non-2xx response from the progenitor input spec (openapi-3.0.json, used only by the Rust SDK; the 3.1 spec and TS SDK are unchanged). Every error status now reaches the client as UnexpectedResponse with its body.
  • SdkResultExt::into_sdk_error is async and always reads the body: an Alien error JSON becomes that error (with requestId); anything else gives UNEXPECTED_RESPONSE with the status and retryability. SdkResultExtReadingBody, convert_sdk_error_reading_body and the typed Error<types::AlienError> path are removed. convert_sdk_error is now async.
  • Call sites add .await, so the compiler found every one of them.
  • Retry decisions: a classified manager error now carries the manager's own retryable flag on every route. The checkpoint call has worked this way since fix: retry a deployment checkpoint after a transient manager error #801: a 500 DEPLOYMENT_LEASE_LOST with retryable: false must not be retried. GENERIC_ERROR is the exception: it is always sent with retryable: false, including a server's sanitized "Internal server error" for a retryable cause such as a database error, so its flag is no decision and the HTTP status decides (5xx/429/408 retryable), as before this change. An unstructured 5xx, or a network error, stays retryable based on its status.
  • renew_lease wrapped every error in GenericError, which is never retryable, so any renewal failure, even a refused connection, lost the lease right away. It now uses ManagerRequestFailed, which inherits retryable (the checkpoint call already does), so the renewal loop's retry-until-deadline path can run.
  • Regenerated the spec with the generate:manager-rust-sdk steps after generate:manager-openapi (the manager's openapi.json is unchanged): only error responses are removed.

Proof

  • New deployment_resolver tests run the real alien deployments get dep_… lookup against an HTTP mock. On main both fail (left: "UNEXPECTED_RESPONSE", right: "DEPLOYMENT_CONTEXT_UNAVAILABLE" / "DEPLOYMENT_NOT_FOUND"). On this branch both pass: the 400's code, message and source error reach the rendered CLI output, and a 404 keeps the manager's error while still mapping to "was not found".
  • Renewal: an_unreachable_manager_fails_the_renewal_as_retryable and a_renewal_the_manager_rejects_is_not_retried pass; alien-deployment manager_api_transport tests 27/27.
  • SDK test: the generated get_deployment against a loopback server returns the server's error for both an unlisted 400 and a listed 404; a malformed HTML 500 still keeps status 500 and stays retryable.
  • cargo check --workspace --all-targets is clean. Lib tests pass for alien-manager-api (8), alien-cli (417), alien-deployment (174), alien-bindings (365) and alien-operator (92). The only failures were tests that need the sandbox environment (pnpm-built core, systemd), and they are unrelated to this change.
  • node --test client-sdks/manager/scripts/fix-openapi.test.mjs passes.
  • Retries, real CLI binaries (main vs this branch) against a scripted fake manager running alien deployments wait: 503 empty, 502 HTML, retryable 500, refused or dropped connection, truncated body, non-UTF-8 and non-Alien JSON bodies all retry and succeed on both. A sanitized GENERIC_ERROR 500 ended the wait on the first response before the GENERIC_ERROR rule; with it, the wait retries and succeeds like main. SDK tests an_unclassified_server_error_is_retryable_by_its_status and a_classified_server_error_keeps_its_own_retryable_flag cover the rule.
  • Against a real manager: a hostname-conflict 400 now prints the context error and its source instead of Unexpected response: 400 Bad Request; a deleted deployment's 404, an invalid token's 401 and a forbidden 403 print the manager's message; list/get/wait/create/delete output is unchanged.

Rollout

Client-only change. The GENERIC_ERROR rule also applies to the deployment runner's manager calls: a sanitized 500 on a checkpoint is now retried with backoff (like a network error since #801) instead of failing the deployment, so a deterministic unclassified 500 there shows as a stuck step with retry warnings rather than a failed deployment. Downstream Rust users of alien-manager-api need .into_sdk_error().await. Code that matched on the Unexpected response: <status> message text should use http_status_code instead, because the message is now the server's own.

🤖 Generated with Claude Code

@enclave-ai

enclave-ai Bot commented Oct 10, 2026

Copy link
Copy Markdown

Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review.

@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@greptile-apps

greptile-apps Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Critical impact] The PR appears safe to merge; no blocking issue remains.

Summary

Manager-backed Rust calls now read failed responses and preserve the manager’s error message, code, source, and request ID.

  • Manager-backed commands show the manager’s own error details.
  • Lease renewal retries temporary manager connection failures.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Manager request] --> B{Request succeeds?}
  B -->|Yes| C[Return response]
  B -->|No HTTP response| D[Convert connection error]
  B -->|Error HTTP response| E[Read body]
  E --> F{Alien error JSON?}
  F -->|No or read fails| G[Keep HTTP status and status-based retry decision]
  F -->|Yes| H[Keep manager details and request ID]
  H --> I{GENERIC_ERROR?}
  I -->|Yes| J[Use HTTP status for retries]
  I -->|No| K[Keep manager retryable flag]
Loading

Reviews (5) · Last reviewed commit: "fix: retry an unclassified manager serve..." · Reviewed by Greptile

Comment thread crates/alien-deployment/src/manager_api_transport.rs
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@alongubkin
alongubkin marked this pull request as ready for review October 10, 2026 05:16
@alongubkin
alongubkin force-pushed the alon/alien-1379-manager-error-bodies branch from efe613e to 106663e Compare October 10, 2026 05:59
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

alongubkin and others added 3 commits October 10, 2026 10:36
The generated manager client kept only the HTTP status for most failed
requests. A status the route's OpenAPI did not list arrived with its
body unread, and the sync into_sdk_error never read it. A status listed
without a body schema (401/403/404 on most routes) had its body dropped
by the generated code. The body-reading adapter was opt-in per call
site, so commands such as `alien deployments get` printed only
"Unexpected response: 400 Bad Request".

The progenitor input spec now declares no error responses, so every
error status reaches the client with its body, and into_sdk_error is
async and always reads it. The opt-in adapter and the typed error path
are gone; there is one conversion for every route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
renew_lease wrapped every manager error in GenericError, which is
never retryable, so the renewal loop treated even a network blip as a
lost lease and its retry-until-deadline branch could not run. Use
ManagerRequestFailed, which inherits retryable from the source, as the
checkpoint call already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GENERIC_ERROR always carries retryable: false, including a server's
sanitized "Internal server error" for a retryable internal cause.
Reading error bodies made that flag decide retries, so a transient 500
ended alien deployments wait at once. For GENERIC_ERROR the HTTP status
decides again; classified errors keep their own flag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin
alongubkin force-pushed the alon/alien-1379-manager-error-bodies branch from 106663e to 8bf968c Compare October 10, 2026 07:36
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

@alongubkin
alongubkin merged commit 82f6f46 into main Oct 10, 2026
31 checks passed
@alongubkin
alongubkin deleted the alon/alien-1379-manager-error-bodies branch October 10, 2026 07:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant