fix(sdk): retry sandbox file uploads on transient failures

[islo-labs]

What changed

• Retry async sandbox file POSTs after HTTP 429 and transient transport errors, up to six attempts.
• Use numeric Retry-After when present and bounded exponential delays otherwise. Reopen the file for every attempt so the complete multipart body is sent.
• Add regression tests for repeated 429s, read errors, and immediate failure on HTTP 400.
• Bump the SDK to 0.3.20 for release and export ASYNC_UPLOAD_FILE_RETRY_ATTEMPTS so Harbor integrations can avoid stacking retry loops.

Verification

• uv run pytest tests/ -q: 48 passed.
• Ruff check, format check, Fern version sync check, changed-file pre-commit hooks, and wheel build passed.

Harbor plugin PR islo-labs/harbor-env#3 delegates uploads once when this SDK capability is present and retains a bounded fallback for published SDK 0.3.19. The benchmark now uses modern bundled Harbor from islo-labs/harbor-fork#14 through islo-labs/e2e-tests#142; that environment also delegates retries to this SDK. The bundled Harbor concurrency-50 rerun 38242210245 passed; both named tasks reached verification and scored 1.0.

Refs: ISL-1671

[islo-labs]

Starting automated code review...

[islo-labs]

The 429 and transport retry behavior in this SDK change looks sound in isolation. There is one cross-repo issue to resolve before the paired feature lands:

Coordinate the upload retry budget with the Harbor PR. Harbor's _sdk_upload_file now retries this six-attempt SDK call four times. A sustained 429 can therefore produce 24 POSTs and 23 sleeps for one artifact; with a 60-second Retry-After, that is about 23 minutes waiting before failure, plus request time. This can hold many verifier trials open under the exact concurrency-50 load in ISL-1671. Please make one layer own the policy (for example, have Harbor call the SDK helper directly), or expose a configurable attempt/time budget in the SDK and give Harbor a single bounded budget. Coordinate the adjustment with harbor-env#3.

[islo-labs]

Starting automated code review...

[islo-labs]

The earlier 24-attempt concern is addressed for the paired harbor-env plugin: its wrapper now calls this SDK helper once. The SDK's 429/transport handling and file reopen behavior still look sound.

One integration gap remains for ISL-1671's actual benchmark: the cited e2e-tests bench workflow installs harbor[islo], and its test invokes harbor run --env islo. That selects Harbor's bundled Islo environment, whose current _sdk_upload_file still has a two-attempt Tenacity wrapper. The harbor-env PR is not loaded by this workflow, so the run still has nested retries (up to 12 POSTs) once this SDK change is released. Please either route the benchmark through the plugin, or update the bundled Harbor environment and its dependency on this SDK change before treating the named rerun as covered. The open e2e-tests#140 branch that produced the cited run retains the same install and --env islo path.

[islo-labs]

The cross-repo review points are addressed: the plugin calls the six-attempt SDK upload once and keeps a bounded fallback for published SDK 0.3.19. The benchmark now selects the modern bundled Islo environment from islo-labs/harbor-fork#14, which follows the same single retry policy. Concurrency-50 Terminal-Bench 4.0 run 38239298896 passed; both named trials reached verification without exceptions and scored 1.0. The benchmark route and result gate are in islo-labs/e2e-tests#142.

[islo-labs]

Starting automated code review...

[islo-labs]

Re-review: the earlier cross-repository retry-budget finding is addressed. The SDK owns six upload attempts and exposes the capability marker used by both Harbor integrations to avoid another retry loop. The file is reopened for each attempt, 429 follows numeric Retry-After, transient transport failures are retried, and other HTTP errors fail immediately. The paired benchmark ran against this exact SDK commit and the two named trials reached verification. No remaining code findings.

[islo-labs]

Starting E2E verification...

[islo-labs]

Verification Report — PASSED

Verified the four PR heads together for ISL-1671. The SDK owns a six-attempt upload retry budget; the Harbor plugin and bundled environment delegate to it when present. No UI changed, so screenshots and a frontend preview do not apply.

Scenario 1: HTTP 429 honors Retry-After and succeeds after three retries — PASSED

Ran PR SDK 0.3.20 against a local HTTP listener returning 429 three times with Retry-After: 0.2, then 200. The multipart POST succeeded on attempt 4. Measured gaps were 0.209, 0.204, and 0.203 seconds; all four requests contained the full 73,000-byte artifact.

Scenario 2: Dropped connection recovers with complete artifact — PASSED

The listener read each POST and closed the first two connections without a response, then returned 200. The SDK retried and sent the full 10,000,000-byte artifact on all three attempts. Recorded multipart request sizes were 10,000,188 bytes each; retry gaps were 1.021 and 2.023 seconds.

Scenario 3: HTTP 400 fails immediately — PASSED

The listener returned 400 to the verifier upload. The SDK raised HTTPStatusError for 400 after exactly one POST.

Scenario 4: Concurrency 50 target trials reach verification — PASSED

Inspected the immutable bench-results artifact from run 38242210245, which completed successfully on the e2e PR head fefad7de. Workflow logs show HARBOR_CONCURRENCY=50, Terminal-Bench 4.0, and installation of pinned bundled Harbor 13c8255d plus SDK b72aaa6d. The artifact has 64 trial results, 57 with reward 1.0. Both biped-contact-dynamics and telecom-entity-resolution have one result, exception_info: null, verifier_result.rewards.reward: 1.0, and verifier/reward.txt: 1.

Focused integration checks

• Harbor plugin PR head 90e3b7b: upload retry tests passed with published SDK 0.3.19 and again with PR SDK 0.3.20: 5 passed, 87 deselected each time.
• Benchmark PR head fefad7de: bundled Islo selection and missing-verifier-result gate tests passed: 2 passed, 1 deselected.
• Harbor fork Ubuntu CI has one failing Docker network policy case, dynamic/shared-allowlist (reward 0 instead of 1). The same case fails on main in run 38240987931; the upload-specific benchmark passed.

Local stack check

launch-platform completed successfully. None of the four PRs changes a launch-platform service, so its five services are on current main as recorded in /workspace/.platform-state.json. Health requests returned HTTP 200 from the web API, Bear Agent, and gateway. I read their local startup logs after the scenarios; the controlled faults ran against the SDK listener, and the concurrency run used its own GitHub runner.