Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -714,7 +714,7 @@ def sort_key(item):
l3_items = [
i
for i in items
if _item_scene_level(i) == SceneTestLevel.HOST and not any(m.name == "skip" for m in i.iter_markers())
if _item_scene_level(i) == SceneTestLevel.NODE and not any(m.name == "skip" for m in i.iter_markers())
]
if l3_items:
sample = ", ".join(sorted({i.nodeid for i in l3_items})[:3])
Expand Down Expand Up @@ -1137,7 +1137,7 @@ def pytest_runtestloop(session):
# levels the dispatcher itself uses for children, it wants direct control.
if runtime_filter is not None:
return
if _normalize_cli_scene_level(level_filter) in (SceneTestLevel.CHIP, SceneTestLevel.HOST):
if _normalize_cli_scene_level(level_filter) in (SceneTestLevel.CHIP, SceneTestLevel.NODE):
return

# User explicitly asked for collect-only / scoped-run — don't orchestrate.
Expand Down
11 changes: 6 additions & 5 deletions docs/dfx/host-trace.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,13 +123,14 @@ including time the caller spends polling or doing other host work; blocking
Every hierarchical worker that drives next-level children emits these spans
through the logger compiled into `_task_interface`, since the orchestrator and
scheduler code they run is the same at every level above the chip. **The leading
word names the level that emitted them**, so `<level>` below is one of `host`
word names the level that emitted them**, so `<level>` below is one of `node`
(L3), `network1` (L4), `network2` (L5) or `network3` (L6) — see
[`python/simpler/worker_level.py`](../../python/simpler/worker_level.py) for the
ladder:
ladder. It is never `host`: that names the *processor* this span ABI belongs to,
which every one of those levels runs on:

| Span | Host decision point |
| ---- | ------------------- |
| Span | Decision point |
| ---- | -------------- |
| `<level>.graph_build` | serialized Python graph callback |
| `<level>.submit` | next-level task publication after slot allocation |
| `<level>.dispatch` | scheduler handoff to a worker thread |
Expand Down Expand Up @@ -268,7 +269,7 @@ tree, and the root span already carries the identity that tells two runs apart.

Only `claim_release` was added for this: it wraps `release_native_run` inside
finalize, the point a successor's launch becomes admissible, and no other span
marks that boundary. `host.post_fence_retirement` covers the L3 orchestrator's
marks that boundary. `node.post_fence_retirement` covers the L3 orchestrator's
`release_run` tail for the same reason.

The identity is `run_id / dispatch_id / run_epoch / slot_id / generation`. Each
Expand Down
32 changes: 23 additions & 9 deletions docs/hierarchical-level-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,23 @@ NPU clusters:
L6 network3 / CLOS2 / Cluster ── full cluster (N6 super-nodes)
L5 network2 / CLOS1 / SuperNode ── super-node (N5 pods)
L4 network1 / POD / Pod ── pod (4 hosts)
L3 host / HOST / Node ── single host machine (16 chips + M SubWorkers)
L3 node / HOST / Node ── single host machine (16 chips + M SubWorkers)
L2 chip / CHIP / Processor ── one NPU chip (shared device memory)
L1 — / DIE / L2Cache ── chip die (hardware-managed)
L0 core / CORE / AIV, AIC ── individual compute core (hardware-managed)
```

The first column is the level word the code uses — `WorkerLevel` in
The first column is the level word **this code** uses — `WorkerLevel` in
[`python/simpler/worker_level.py`](../python/simpler/worker_level.py), the
`[STRACE]` span prefix, and `SceneTestLevel`. L1 has no word because no Worker
sits there.
sits there. The second column is the **Ascend architecture documentation's** code
for the same level, and the third is the physical entity it denotes.

L3 is where the three disagree, and that is worth reading closely: we call it
`node`, the architecture docs call it `HOST`, and the entity is a Node. `host` is
the *processor* name in this repository — `HostLogger`, the `host_span` ABI,
`host_runtime.so` — so reusing it as a level word made one word mean both "which
processor" and "which level".

**L2 is the boundary** between two worlds:

Expand All @@ -62,20 +69,27 @@ sits there.

| Level | Workers it contains | Status |
| ----- | ------------------- | ------ |
| L3 (`host`) | `ChipWorker` ×N + `SubWorker` ×M | Implemented |
| L3 (`node`) | `ChipWorker` ×N + `SubWorker` ×M | Implemented |
| L4 (`network1`) | `Worker(level=3)` ×N + `SubWorker` ×M | Local implemented; remote sim implemented; HCOMM profiles pending |
| L5 (`network2`) | `Worker(level=4)` ×N | Local L4 code path, untested; remote proposed |
| L6 (`network3`) | `Worker(level=5)` ×N | Local L4 code path, untested; remote proposed |

**This table is the one place the level words are mapped to physical entities.**
A level above `host` is named for how many network hops separate it from the
host, not for the hardware that implements it: `network1` is the layer commonly
deployed as a **pod**, `network2` as a **supernode**, `network3` as a
Every word names a **topology position**, never a processor and never a
deployment fact. A level above `node` is named for how many network hops separate
it from the node, not for the hardware that implements it: `network1` is the layer
commonly deployed as a **pod**, `network2` as a **supernode**, `network3` as a
**cluster**. The words count hops because that correspondence is not fixed —
sometimes a host sits under a pod, sometimes directly under a supernode — and
sometimes a node sits under a pod, sometimes directly under a supernode — and
naming a level after an interconnect does not work either, since it is mostly SU
and sometimes RoCE. Note the digit counts hops rather than the level number:
`network1` is L4. The ladder is defined once, in
`network1` is L4.

L3 is `node` for the same reason, and **not** `host`: `host` names the *processor*
that `HostLogger`, the `host_span` ABI and `host_runtime.so` belong to, and L3
merely happens to run there. So does L4, and so does part of the chip runtime —
a level word taken from a deployment location cannot be exclusive. The ladder is
defined once, in
[`python/simpler/worker_level.py`](../python/simpler/worker_level.py); everything
else in the tree uses the level word alone.

Expand Down
2 changes: 1 addition & 1 deletion python/simpler/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -4399,7 +4399,7 @@ def __init__(
self.level = level
# Rebound from the level in `init()`; the default matches the C++ table's
# so a span emitted before init names L3 rather than nothing.
self._host_span_prefix = _span_prefix(WorkerLevel.host)
self._host_span_prefix = _span_prefix(WorkerLevel.node)
self._config = config
self._callable_registry: dict[int, Any] = {}
self._identity_registry: dict[bytes, _CallableIdentityState] = {}
Expand Down
25 changes: 16 additions & 9 deletions python/simpler/worker_level.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,26 @@
"""One word per level of the worker hierarchy, and the trace prefix derived from it.

`[STRACE]` span names lead with the level that emitted them, so a reader can tell
a chip-runtime span from a host-scheduler one without consulting the tree. The
a chip-runtime span from a node-scheduler one without consulting the tree. The
words live here because the emitting code cannot supply them: `Orchestrator` and
`WorkerThread` are level-agnostic — the same code runs at every level above the
chip — so the word is a property of the process, resolved once from its Worker's
level.

Above `host` the physical layer is not fixed: sometimes a host sits under a pod,
sometimes directly under a supernode. Naming those levels after a specific
interconnect does not work either (mostly SU, sometimes RoCE). So they are named
by how many network hops separate them from the host — `network1` is one hop up,
whatever hardware implements it. The digit counts hops, not the level number:
**Every word names a topology position, never a processor or a deployment fact.**
`node` is one machine's orchestration; above it the physical layer is not fixed —
sometimes a node sits under a pod, sometimes directly under a supernode — and
naming those levels after an interconnect does not work either (mostly SU,
sometimes RoCE). So they count network hops from the node: `network1` is one hop
up, whatever hardware implements it. The digit counts hops, not the level number:
`network1` is L4.

`host` is deliberately absent. It names a *processor* — the CPU that `HostLogger`,
`host_span` and `host_runtime.so` all belong to — and L3 merely happens to run
there. So does L4, and so does part of the chip runtime, which is why a level word
taken from a deployment location cannot be exclusive. That is the same reason L4
is not called `pod`.

`docs/hierarchical-level-runtime.md` is the one place that maps these words onto
physical entities; this module deliberately does not repeat that mapping, so the
two cannot drift.
Expand All @@ -49,14 +56,14 @@ class WorkerLevel(IntEnum):

core = 0
chip = 2
host = 3
node = 3
network1 = 4
network2 = 5
network3 = 6


def span_prefix(level: int) -> str:
"""Return the `[STRACE]` name prefix for `level`, e.g. 3 -> ``"host"``.
"""Return the `[STRACE]` name prefix for `level`, e.g. 3 -> ``"node"``.

Raises `ValueError` for a level the ladder does not name, rather than
inventing a word. L5 and L6 run what
Expand All @@ -67,7 +74,7 @@ def span_prefix(level: int) -> str:

Note `WorkerLevel(level).name`, not `str(WorkerLevel(level))`: this project
targets Python 3.10, where `enum.StrEnum` does not exist and `str()` on any
`Enum` yields ``"WorkerLevel.host"``.
`Enum` yields ``"WorkerLevel.node"``.
"""
try:
return WorkerLevel(level).name
Expand Down
2 changes: 1 addition & 1 deletion simpler_setup/scene_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ def _golden_thread_cap():

class SceneTestLevel(IntEnum):
CHIP = 2
HOST = 3
NODE = 3
NETWORK1 = 4


Expand Down
Loading
Loading