Repository navigation
Need an operations runbook for Substrate operators #581
Description
Summary
There is no operational documentation for the people running Substrate.
An operator's questions are structurally different from an actor developer's. They are fleet-level and cross-component: is the platform healthy, which component owns this symptom, are we out of capacity, is one tenant hurting everyone, is a node degrading. Answering those means knowing which of seven-plus components to look at, and Substrate currently gives almost no help with that routing.
This issue asks for docs/operations.md: a symptom → component → signal → action runbook.
Scope boundary
This is the operator column of the audience split; the actor-developer guide is the other. They must stay separate documents.
| Substrate operator | Actor developer | |
|---|---|---|
| Unit of concern | fleet, pools, nodes, components | one actor |
| Placement (pod/node) | primary diagnostic lever | not exposed — see #569 |
| Cluster access | full | limited or none |
| Typical entry point | an alert, or a tenant complaint | their own app misbehaving |
The overlap — "a tenant says their actor is slow, is it us or them?" — is the escalation boundary, and deserves its own section rather than being split across both docs.
What an operator has today
| Capability | What exists | Coverage |
|---|---|---|
| Metrics | rpc.server.call.duration (ateapi, atelet), atenet.router.route.duration, atelet.snapshot.size |
three instruments, three components |
| Dashboards | ate-grpc, ate-e2e-latency, ate-snapshot (tools/setup-gcp/dashboards/) |
GCP only; no kind equivalent |
| Alerts | — | none ship. tools/setup-gcp has create dashboards; there is no create alerts |
| GKE scrape config | one PodMonitoring: atenet-router-monitoring.yaml |
needed for the Envoy round-trip lines |
| kind scrape annotations | prometheus.io/scrape on ateapi, atelet, atenet-router |
3 of 7+ workloads |
| Fleet CLI | kubectl ate get workers, get actors, get atespaces, top workers |
|
| Traces | --trace, Jaeger (kind) / Cloud Trace (GKE) |
control-plane operations |
| Bounded metric labels | ate.workerpool.name, ate.worker.state, ate.sandbox.class (internal/ateattr/ateattr.go:50-56) |
see the enum problem below |
Activity
HavenXia commentedon Jul 29, 2026
I think we has zero alerts today, so nobody enters through one. If the runbook assumes alert-driven operation, a minimal alert set is a prerequisite, should we create an issue to track that?
git286 commentedon Jul 29, 2026
For M2, I think alerts is a nice to have, not P0.