Skip to content

docs: reorganize the upgrade runbook for better UX - #1624

Open
Haven Xia (HavenXia) wants to merge 2 commits into
agent-substrate:mainfrom
HavenXia:doc-ux-update
Open

docs: reorganize the upgrade runbook for better UX#1624
Haven Xia (HavenXia) wants to merge 2 commits into
agent-substrate:mainfrom
HavenXia:doc-ux-update

Conversation

@HavenXia

@HavenXia Haven Xia (HavenXia) commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Part of #1197

  • Tests pass
  • Appropriate changes to documentation are included in the PR

@Raji14 Raji (Raji14) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few suggestions before merge:

  1. Warning 1 under "Three things that break an upgrade":
    The heading is "Three things that break an upgrade", but Warning 1 is phrased as the correct action: "Flip the node's version label before deleting its old worker pods."
    Consider rephrasing it to state the anti-pattern (like warnings 2 & 3 do):

    Do not delete old worker pods before flipping the node's version label. Otherwise the old pool reschedules replacements onto the same node next to the new atelet...

  2. Pre-flight check for multi-node clusters:
    In the previous version, line 199 noted: # At least two nodes; on a single node the roll is a full stop. This was omitted from the new checklist. We should add a checklist item confirming the cluster has at least 2 worker nodes, since a rolling zero-downtime upgrade cannot work on a single node.
  3. Rollback list completeness (Steps 3 & 4):
    In ## Something's wrong, or I need to undo, the list jumps from "Past step 5" to "Past step 2". For an operator who reached Step 3 (deployed new atelet) or Step 4 (cloned workerpool) but hasn't flipped nodes yet, it would be helpful to include:
    • - Past step 4: delete the cloned WorkerPool (kubectl -n $NS delete workerpool $NEW_WORKERPOOL)
    • - Past step 3: delete the new DaemonSet (kubectl delete daemonset -n ate-system -l app=atelet,ate.dev/substrate-version=$NEW_VERSION)
  4. "Check your progress" note phrasing:
    "Steps 1, 2 and 6 leave no mark you can read back from the cluster."
    Those steps do update cluster objects (e.g. ate-controller / api deployments, CRDs); it's just that the 3 quick-check commands above don't query them. Clarifying: "The three quick-check commands above do not report whether steps 1, 2, or 6 have run..." would avoid confusion.
  5. Step 2 readiness check:
    In Step 2: "wait for READY to equal DESIRED again on every serving pool... before step 5."
    Because Step 4 clones the serving pool, we should probably advise waiting before Step 4 so operators don't clone a pool whose Deployment is actively rolling.
  6. Doc consistency (60s vs 30m):
    Good catch updating the suspend window from 60s to 30 minutes to match workloadGracePeriod. As a follow-up (or in this PR), tools/setup-gcp/README.md#2-create-cluster still mentions "about 60 seconds" and should be updated as well.

Phrase the first warning as the mistake, like the other two. Add
rollback entries for the new DaemonSet and the cloned pools, so an
operator who stopped before flipping any node finds their step in the
list. Say what the progress commands do not show instead of claiming
that steps leave no trace. Wait for a controller-triggered roll to
settle before cloning pools. Note that a single-node cluster makes the
per-node step a full stop. The setup-gcp README's eviction warning
still said 60 seconds; it is 30 minutes.
@HavenXia

Copy link
Copy Markdown
Collaborator Author

Thanks, pushed second commit.

On point 2:
I remove "# At least two nodes; on a single node the roll is a full stop." because a single node does not stop the roll, it just means every actor is suspended at once during step 5, so I put that as a note at the top of step 5 rather than a go/no-go item in the checklist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants