docs: reorganize the upgrade runbook for better UX - #1624
Open
Haven Xia (HavenXia) wants to merge 2 commits into
Open
docs: reorganize the upgrade runbook for better UX#1624Haven Xia (HavenXia) wants to merge 2 commits into
Haven Xia (HavenXia) wants to merge 2 commits into
Conversation
Raji (Raji14)
left a comment
Collaborator
There was a problem hiding this comment.
A few suggestions before merge:
- Warning 1 under "Three things that break an upgrade":
The heading is "Three things that break an upgrade", but Warning 1 is phrased as the correct action: "Flip the node's version label before deleting its old worker pods."
Consider rephrasing it to state the anti-pattern (like warnings 2 & 3 do):Do not delete old worker pods before flipping the node's version label. Otherwise the old pool reschedules replacements onto the same node next to the new atelet...
- Pre-flight check for multi-node clusters:
In the previous version, line 199 noted:# At least two nodes; on a single node the roll is a full stop.This was omitted from the new checklist. We should add a checklist item confirming the cluster has at least 2 worker nodes, since a rolling zero-downtime upgrade cannot work on a single node. - Rollback list completeness (Steps 3 & 4):
In## Something's wrong, or I need to undo, the list jumps from "Past step 5" to "Past step 2". For an operator who reached Step 3 (deployed new atelet) or Step 4 (cloned workerpool) but hasn't flipped nodes yet, it would be helpful to include:- Past step 4: delete the cloned WorkerPool (kubectl -n $NS delete workerpool $NEW_WORKERPOOL)- Past step 3: delete the new DaemonSet (kubectl delete daemonset -n ate-system -l app=atelet,ate.dev/substrate-version=$NEW_VERSION)
- "Check your progress" note phrasing:
"Steps 1, 2 and 6 leave no mark you can read back from the cluster."
Those steps do update cluster objects (e.g.ate-controller/apideployments, CRDs); it's just that the 3 quick-check commands above don't query them. Clarifying: "The three quick-check commands above do not report whether steps 1, 2, or 6 have run..." would avoid confusion. - Step 2 readiness check:
In Step 2: "wait for READY to equal DESIRED again on every serving pool... before step 5."
Because Step 4 clones the serving pool, we should probably advise waiting before Step 4 so operators don't clone a pool whose Deployment is actively rolling. - Doc consistency (60s vs 30m):
Good catch updating the suspend window from 60s to 30 minutes to matchworkloadGracePeriod. As a follow-up (or in this PR),tools/setup-gcp/README.md#2-create-clusterstill mentions "about 60 seconds" and should be updated as well.
Phrase the first warning as the mistake, like the other two. Add rollback entries for the new DaemonSet and the cloned pools, so an operator who stopped before flipping any node finds their step in the list. Say what the progress commands do not show instead of claiming that steps leave no trace. Wait for a controller-triggered roll to settle before cloning pools. Note that a single-node cluster makes the per-node step a full stop. The setup-gcp README's eviction warning still said 60 seconds; it is 30 minutes.
Collaborator
Author
|
Thanks, pushed second commit. On point 2: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1197