Skip to content

fix(async): return *Job from Location header for v3 lifecycle deletes/scales/restarts - #2

Merged
wayneeseguin merged 7 commits into
fivetwenty-io:mainfrom
norman-abramovitz:feat-async-job-correctness
May 26, 2026
Merged

wayneeseguin merged 7 commits into
fivetwenty-io:mainfrom
norman-abramovitz:feat-async-job-correctness

Conversation

@norman-abramovitz

@norman-abramovitz norman-abramovitz commented May 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

CF v3 lifecycle operations (DELETE, scale, start/stop/restart) are async: CF responds 202 Accepted with an empty body and a Location header pointing at /v3/jobs/{jobGuid}. The current capi clients call HTTP and return only error, throwing the Location header (and therefore the job GUID) on the floor. Callers that need to poll for completion can't.

This PR retrofits 6+ resource clients to the existing (*Job, error) shape that OrganizationsClient.Delete and SpacesClient.Delete already use — no new pattern, just consistency.

The pattern (review once, applies to every commit)

The fix is identical at every callsite:

// Before
func (c *FooClient) Delete(ctx context.Context, guid string) error {
    _, err := c.httpClient.Delete(ctx, "/v3/foos/"+guid)
    if err != nil {
        return fmt.Errorf("deleting foo: %w", err)
    }
    return nil
}

// After
func (c *FooClient) Delete(ctx context.Context, guid string) (*capi.Job, error) {
    resp, err := c.httpClient.Delete(ctx, "/v3/foos/"+guid)
    if err != nil {
        return nil, fmt.Errorf("deleting foo: %w", err)
    }
    location := resp.Headers.Get("Location")
    if location == "" {
        return nil, fmt.Errorf("deleting foo: no Location header on async response")
    }
    jobGUID := location
    if idx := strings.LastIndex(location, "/"); idx >= 0 {
        jobGUID = location[idx+1:]
    }
    if jobGUID == "" {
        return nil, fmt.Errorf("deleting foo: malformed Location header %q", location)
    }
    return &capi.Job{Resource: capi.Resource{GUID: jobGUID}}, nil
}

Reviewing commit 1 (Apps.Delete) is sufficient context for commits 2–7 — the rest mechanically apply the same change to different resources. Each commit message confirms it.

Commits

# Resource Change
1 Apps.Delete error → (*Job, error) from Location
2 Routes.Delete + ServiceCredentialBindings.Delete Same. Previous impl json.Unmarshal'd the empty body into Job and silently produced Job{GUID:""} — callers polling the empty GUID got 404s. SCB is polymorphic: user-provided returns 204 (sync, nil job), managed returns 202+Location.
3 Apps.Start/Stop/Restart/Restage error → (*Job, error)
4 Processes.Scale error → (*Job, error)
5 Apps/Processes sync-complete fallback Some older CFs (< ~3.190) return 200/204 with no Location for some endpoints. Treat that as a sync-complete and return (nil, nil) instead of a hard error on missing Location.
6 Remove fabricated Apps.Restage The standalone Restage method didn't correspond to any CF v3 endpoint. CF v3 has no atomic restage — it's a composition of package lookup → build → set-droplet → stop → start. Removing the fabrication; callers should compose explicitly.
7 Roles.Delete error → (*Job, error)

Verification

  • CF v3 API spec — Location-header contract for delete action across /v3/apps/{guid}, /v3/routes/{guid}, /v3/service_credential_bindings/{guid}, /v3/roles/{guid}. All four documented to return 202 + Location → /v3/jobs/{jobGuid}.
  • cloud_controller_ng source — destroy actions emit head :accepted, 'Location' => … directly.
  • Validated end-to-end against a real CF 3.180 deployment.

Breaking changes

Yes — the public interfaces change signature (error → (*Job, error)). Existing callers that ignore async completion need to update their code:

// Before
err := client.Apps().Delete(ctx, guid)

// After (caller can ignore the job if they don't need it)
_, err := client.Apps().Delete(ctx, guid)

This is parallel to the way OrganizationsClient.Delete and SpacesClient.Delete already work — they return (*Job, error) today. Bringing the rest of v3 in line.

Tests

Each commit adds:

  • A "happy path" test asserting the returned Job.GUID matches the Location-header GUID
  • A "missing Location" regression that asserts an error rather than Job{GUID: ""}
  • For the sync-fallback commit, a test exercising 200/204 with no Location

Full suite green.

Out of scope (separate PR)

A second PR (#3, feat-link-meta) adds an optional Meta map[string]interface{} field to capi.Link. Independent change; either PR can land first.

CF v3 DELETE /v3/apps/{guid} is async: 202 Accepted with a Location
header pointing at /v3/jobs/{guid} and an empty body. The previous
signature discarded the header so callers couldn't poll for completion
- an anomaly versus OrganizationsClient / SpacesClient which already
return (*Job, error).

The async Location-header contract is defined in the CF v3 OpenAPI
spec under /v3/apps/{guid} delete. Impl parses the trailing GUID from
the header and returns a Job with GUID populated; consumers call
Jobs().Get (or Jobs().PollUntilComplete) for full state.

- Interface: AppCRUDClient.Delete now returns (*Job, error)
- Impl: parse Location header; guard missing/malformed values
- Test: simulate 202+Location; add missing-Location regression
- Batch + mock call sites updated for the new signature
- Example (examples/apps/main.go) prints the returned job guid
  parse Location header

DELETE /v3/routes/{guid} and DELETE /v3/service_credential_bindings/{guid}
return 202 Accepted with empty body and a Location header pointing at
/v3/jobs/{jobGuid}. The previous impls json.Unmarshal'd the empty body
into capi.Job, which silently produced a Job with GUID="" — callers that
polled against /v3/jobs/ got 404s or errors instead of the actual job.

Same pattern as AppsClient.Delete (fix in 05c428d). ServiceCredentialBindings
is polymorphic: user-provided bindings return 204 No Content (sync, nil job);
managed bindings return 202 + Location (async). Tests cover both shapes
plus the missing-Location protocol-violation case.

Verified against:
- cloud_controller_ng main branch destroy actions (head :accepted,
  'Location' => ...)
- CF v3 API docs 3.216/3.217 route + service_credential_binding deletes
CF v3 /v3/apps/{guid}/actions/{start|stop|restart|restage} are async:
CF responds 202 Accepted with a Location header pointing at
/v3/jobs/{jobGuid}. The prior implementation unmarshalled the empty
response body as an App (or Build, for restage) and discarded the
Location header, so callers got a zero-value resource and no way to
observe the action's completion.

Mirror the Delete fix: parse the Location header, extract the job
GUID, and return *Job. AppLifecycleClient interface, CLI callers, and
mocks updated in lockstep.

Restage switches from the client-side two-step workflow (list
packages + create build) back to the deprecated-but-still-supported
POST /actions/restage endpoint so all four lifecycle actions share
the same async-job shape. The two-step modern flow remains possible
via Packages().List + Builds().Create + Droplets().SetCurrent for
callers that need the build-state detail; restage() itself now lines
up with start/stop/restart for uniform callsite code.
POST /v3/processes/{guid}/actions/scale is async per the CF v3 API
spec: CF responds 202 Accepted with a Location header pointing at
/v3/jobs/{jobGuid}. Mirror the app-lifecycle actions (Start, Stop,
Restart, Restage) and return *Job so callers have a uniform async-
completion surface.

The prior implementation unmarshalled the response body as Process
and discarded the Location header, leaving callers with either a
stale Process or a zero value when CF sent no body.

AppLifecycleClient's Scale signature, tests, and the CLI + example
callers updated in lockstep. The CLI echoes the queued job GUID and
reports the user's requested scale targets — terminal state requires
polling Jobs().Get(job.GUID).
CF v3 transitioned lifecycle actions (start/stop/restart/restage) and
process scale from synchronous (200 + resource body) to asynchronous
(202 + Location → /v3/jobs/{jobGuid}) somewhere in the 3.18x–3.20x
range. The .3 tag assumed modern async only and errored with "no
Location header on async response" against older CF targets — we hit
this immediately running Stratos dev.61 against adepttech's 3.180.0.

AppsClient.postActionJob and ProcessesClient.Scale now treat missing
Location as synchronous completion and return (nil, nil), letting
callers branch on job == nil to synthesize a terminal-COMPLETE
response instead of polling a non-existent job. Matches the service-
binding delete pattern where user-provided bindings return 204 and
managed bindings return 202 + Location.
CF v3 has no /v3/apps/{guid}/actions/restage endpoint. It never did —
CF deliberately replaced the v2 atomic /v2/apps/{guid}/restage with the
builds resource, quoting v3.216 docs:

  "The specialized /v2/apps/:guid/restage endpoint is replaced by the
  builds resource. Builds allow finer-grained control and increased
  flexibility when staging packages into droplets. The V3 API avoids
  making assumptions about what users want to happen."

Our Apps().Restage() pointed at the non-existent v3 action URL
(inherited from fivetwenty-io/capi commit 255713a; see also:
https://github.com/cloudfoundry/cloud_controller_ng/wiki/How-to-restage-an-app-directly-using-V3-APIs-(with-examples)).
The method is unusable on every CF version. Removing it forces callers
to either use /v2/apps/{guid}/restage directly (still live per RFC-0032
until end of 2026) or compose primitives as cf-cli does.

CF CLI v8 restage reference sequence
------------------------------------
No-strategy (atomic downtime) path, as implemented in
command/v7/shared/app_stager.go + actor/v7action/:

  1. GET  /v3/apps?names=<n>&space_guids=<s>        (resolve app)
  2. GET  /v3/packages?app_guids=<a>&states=READY
          &order_by=-created_at&per_page=1          (newest READY pkg)
  3. GET  /v3/packages?guids=<p>&app_guids=<a>      (verify)
  4. POST /v3/builds  body {"package":{"guid":"<p>"}}
  5. GET  /v3/builds/<build_guid>                   (poll until
          state != STAGING; returns droplet_guid)
  6. POST /v3/apps/<a>/actions/stop                 (if already running)
  7. PATCH /v3/apps/<a>/relationships/current_droplet
          body {"data":{"guid":"<droplet_guid>"}}
  8. POST /v3/apps/<a>/actions/start
  9. GET  /v3/apps/<a>/processes
          + per-proc GET /v3/processes/<g>/stats    (poll ready)

Strategy=rolling or canary replaces steps 6-8 with:
  - POST /v3/deployments  body {droplet, relationships.app, strategy,…}
  - GET  /v3/deployments/<d>                        (poll until DEPLOYED)

The primitives on fw-capi already cover every call above:
PackagesClient.List, BuildsClient.Create/Get, AppsClient.Stop/Start,
AppsClient.SetCurrentDroplet. Composition belongs to the caller or to
a future orchestration layer analogous to cf-cli's shared.AppStager.
Until that exists, Stratos routes restage through the v2 endpoint.
@wayneeseguin

Copy link
Copy Markdown
Contributor

Great idea, thank you for this!

@wayneeseguin
wayneeseguin merged commit 24c57b2 into fivetwenty-io:main May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants