Skip to content

Update: share Graph Definitions across HBG submissions #1874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ChaoWao merged 1 commit into from
Aug 18, 2026
Merged

Update: share Graph Definitions across HBG submissions #1874

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions docs/investigations/2026-08-hbg-graph-definition-single-upload.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# 2026-08 — hbg: upload Graph Definitions once as shared device objects

## Question

Breaking down the host side of `examples/a2a3/host_build_graph/qwen3_14b_decode`
(40 Graph submissions of one 277-node, 130,192-byte Definition) showed two
dominant costs outside pure orchestration:

- the orchestration window rebuilt and zero-filled a 132,752-byte submission
image per layer — 98.1% of those bytes a byte-identical Definition copy
(KNOWN_ISSUES at the time);
- the upload stage shipped all 40 images: 5.31 MB across 40
`device_malloc` + `rtMemcpy` round trips, a stable 14.4–15.9 ms per run.

Both trace to one design choice: the Definition travels *inside* every
submission image.

## Change

`254f924e` (measured by `4d434174`/`b8095e39`, enabled by `f868ac52`):

- each distinct Definition uploads **once** as a
`[GraphDefinitionHeader][Definition image]` object retained by
`acquire_graph_definition_buffer`, keyed by content identity;
- `GraphSubmission` carries `definition_addr` + `definition_hash` instead of
the inline image; a submission is now 2,568 bytes;
- device localize validates the shared object through a one-time verify gate
(first localizer FNV-hashes, peers spin on the state word) and binds
topology against the shared image in place — the per-occurrence embedded
Definition copy in execution storage is gone.

## Result (qwen3, `SIMPLER_SKIP_DEVICE_RUN=1`, 5 serial runs each, median)

| Stage | Before | After | Δ |
| ----- | ------ | ----- | - |
| orch image build (incl. zero-fill) | 931 µs (826) | 23.6 µs (10.0) | −97.5% |
| submission bytes | 5,310,080 | 232,944 | −95.6% |
| orch window total | 1,834 µs | 836 µs | −54% |
| **H2D upload time** | **14.66 ms** | **12.10 ms** | **−17%** |

## Why the H2D time gain is far below the byte gain

Bytes fell 95.6% but the upload *time* fell only 17%: the stage's cost was
never bandwidth-dominated. The effective rate is absurd on both sides —
0.36 GB/s before, 0.02 GB/s after — which is the signature of fixed
per-call costs dominating data movement. The optimized stage still makes 41
allocation-and-copy pairs: one 130,192-byte shared Definition object and 40
2,568-byte reference submissions.

Each upload pays:

1. **`rtMalloc` per object** — `MemoryAllocator::alloc` calls CANN
`rtMalloc(RT_MEMORY_HBM)` (a driver round trip) plus a mutex-guarded map
insert; the one Definition object and 40 submissions make 41 allocations
per run, freed again at teardown.
2. **`rtMemcpy` (sync, `RT_MEMCPY_HOST_TO_DEVICE`) per object** — each
call is a blocking submit-and-wait on the copy stream: host builds the
descriptor, pushes to the driver, and blocks for completion. Each 2.5 KB
reference submission never occupies the link long enough for bandwidth to
matter. The 12.1 ms / 41 ≈ 295 µs figure is the average across one large
and 40 small calls, not a uniform per-submission measurement, but it bounds
the scale of the driver and synchronization overhead dominating the stage.

So the model is `time ≈ N × (malloc + memcpy latency) + bytes / BW`, and at
these sizes the first term dominates by two orders of magnitude. The byte
reduction could only remove the (already small) second term — consistent
with the measured 2.5 ms saving landing near 40 × 130 KB / 0.36 GB/s of
actual removed traffic.

**The follow-up this points at**: batch the 40 reference submissions into one
packed upload (they are POD and contiguous by construction; one malloc, one
memcpy, per-device offsets), which attacks the *N* in the dominant term.
Expected to take the stage from ~12 ms to well under 1 ms. Not done in this
change to keep the wire-v2 commit reviewable and independently revertible.

## Notes

- The first-cut device verify gate returned "busy-looking" nulls to peer
submissions and surfaced as `sched_error_code=5 INVALID_ARGS`; the fix is
the spin-wait on `verify_state` (dispatch-path legal: spin, no sleep).
- `graph record` (245–687 µs) remains per-run and untouched — the per-run
Definition cache discard is a separate, still-open item (KNOWN_ISSUES).
1 change: 1 addition & 0 deletions docs/investigations/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ that ...".

Newest first.

- [2026-08 — hbg: uploading Graph Definitions once as shared device objects](2026-08-hbg-graph-definition-single-upload.md) — cut the per-replay 130 KB Definition re-serialization (image build 931→24 µs, orch total −54%), and measured the follow-on truth that the H2D stage is latency-bound: bytes fell 95.6% but time only 17% because fixed allocation and synchronous-copy costs dominate 41 calls (one 130 KB Definition plus 40 × 2.5 KB references) — the next lever is batching the reference submissions into one packed upload
- [2026-07 — Why qwen3_14b_decode held a device for 406 s, and the four answers that were wrong](2026-07-qwen-scene-test-406s-decomposition.md) — root cause was torch thread oversubscription in goldens, not any of the big things: the reference walks 3584 tiny slice ops per layer and torch sizes its pool from the core count, so a 320-core host paid 6.35 s/layer against 1.05 s at 4. Capping to 8 (#1601) cut the golden 359 s → ~40 s and let the case rejoin the sweep. Records the measured decomposition (compile 59 s / fixture 13 s / golden 359 s / device tens of ms), how to split queue wait from card-held work via the `npu-lock` timestamps, and why golden caching, vectorisation, a nightly split and `skip_golden` were each dropped
- [2026-07 — `sync_start` drain retry ABA across reusable barrier state](2026-07-sync-start-drain-retry-aba.md) — reproduced #1455 deterministically by suspending an old-attempt scheduler between the ack barrier and election; retained a generation-tagged O(log N) tree with fixed thread-0 coordination after packed-atomic, no-root-broadcast, and rotating-coordinator experiments; removed the invasive runtime hook after hook-free UT/ST coverage was established
- [2026-07 — Host worker dispatch latency: where the remaining ~50 µs goes](2026-07-host-dispatch-latency-budget.md) — measured & dropped: after #1499 a `Worker.run()` costs **43.3 µs fixed + 8.08 µs per task**, so the "94% of latency is our runtime, not the IPC" headline is an artifact of benchmarking a *single* task — at 256 tasks/run the fixed cost is 2%. Also settles #1498 against itself for latency: a pipe wake measures 4.4 µs one way against a 3.5 µs bare fork+shm round trip, so blocking *adds* more than the whole IPC floor to every dispatch; only the CPU-while-idle argument survives
Expand Down
61 changes: 57 additions & 4 deletions src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,50 @@ bool upload_graph_submissions(
std::unordered_map<uint64_t, uint32_t> occurrences;
uploaded_bytes = 0;
const size_t count = graph_host_upload_count(graph_state);
// Pass 1: upload each distinct Definition once as a shared device object
// ([GraphDefinitionHeader][Definition image]) keyed by content identity.
// Submissions reference the object's GM address, so this pass completing
// before any submission is uploaded is what makes the reference safe —
// the device boots only after both passes.
GraphHostDefinitionList definitions = graph_host_definitions(graph_state);
struct UploadedDefinition {
void *device_object; // GM address; host must not dereference
const GraphDefinition *host_view; // the host-side image the object was built from
};
std::unordered_map<uint64_t, UploadedDefinition> definition_objects;
for (const GraphHostDefinition &entry : definitions.entries) {
if (entry.data == nullptr || entry.bytes < sizeof(GraphDefinition)) continue;
const auto *definition = reinterpret_cast<const GraphDefinition *>(entry.data);
if (definition->total_bytes != entry.bytes || definition->full_key != entry.full_key) continue;
const size_t object_bytes = sizeof(GraphDefinitionHeader) + entry.bytes;
void *object =
api->acquire_graph_definition_buffer(entry.full_key, object_bytes, alignof(GraphDefinitionHeader));
if (object == nullptr) {
LOG_ERROR(
"host-orch: failed to retain %zu bytes for Graph Definition key=%#llx", object_bytes,
static_cast<unsigned long long>(entry.full_key)
);
return false;
}
std::vector<std::byte> staging(object_bytes, std::byte{0});
auto *header = reinterpret_cast<GraphDefinitionHeader *>(staging.data());
header->magic = GRAPH_DEFINITION_OBJECT_MAGIC;
header->verify_state.store(
static_cast<uint32_t>(GraphDefinitionVerifyState::UPLOADED), std::memory_order_relaxed
);
header->definition_bytes = static_cast<uint32_t>(entry.bytes);
header->content_hash = definition->content_hash;
header->full_key = definition->full_key;
std::memcpy(staging.data() + sizeof(GraphDefinitionHeader), entry.data, entry.bytes);
if (api->copy_to_device(object, staging.data(), object_bytes) != 0) {
LOG_ERROR("host-orch: failed to upload Graph Definition object");
return false;
}
definition_objects.emplace(definition->content_hash, UploadedDefinition{object, definition});
uploaded_bytes += object_bytes;
}

// Pass 2: per-submission execution storage + the small reference image.
for (size_t index = 0; index < count; ++index) {
std::optional<GraphHostUpload> upload = graph_host_upload(graph_state, index);
if (!upload.has_value() || upload->outer_slot == nullptr || upload->data == nullptr ||
Expand All @@ -435,13 +479,21 @@ bool upload_graph_submissions(
LOG_ERROR("host-orch: Graph submission size does not match its POD image");
return false;
}
const GraphDefinition *definition = graph_submission_definition(*submission);
auto object_it = definition_objects.find(submission->definition_hash);
if (object_it == definition_objects.end() || object_it->second.device_object == nullptr) {
LOG_ERROR("host-orch: Graph submission has no uploaded Definition object");
return false;
}
// Capacities come from the host-side Definition image the device
// object was built from; the GM object itself is never dereferenced
// on the host.
const GraphDefinition *definition = object_it->second.host_view;
size_t execution_bytes = 0;
if (definition == nullptr || definition->full_key != submission->graph_key || definition->task_count == 0 ||
definition->task_count > GRAPH_MAX_NODES ||
if (definition->task_count == 0 || definition->task_count > GRAPH_MAX_NODES ||
definition->full_key != submission->graph_key ||
!graph_execution_storage_bytes(
static_cast<int32_t>(definition->task_count), definition->tensor_arg_count,
definition->scalar_arg_count, definition->total_bytes, &execution_bytes
definition->scalar_arg_count, &execution_bytes
)) {
LOG_ERROR("host-orch: invalid Graph execution storage request");
return false;
Expand All @@ -457,6 +509,7 @@ bool upload_graph_submissions(
);
return false;
}
submission->definition_addr = reinterpret_cast<uint64_t>(object_it->second.device_object);
submission->execution_storage = reinterpret_cast<uint64_t>(execution_storage);
submission->execution_storage_bytes = execution_bytes;
submission->local_execution = 0;
Expand Down
20 changes: 15 additions & 5 deletions src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/pto_orchestrator.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -686,6 +686,18 @@ std::optional<GraphHostUpload> graph_host_upload(GraphHostState &state, size_t i
return GraphHostUpload{upload.outer_slot, upload.image.data(), upload.image.size()};
}

GraphHostDefinitionList graph_host_definitions(GraphHostState &state) {
GraphHostDefinitionList list;
list.entries.reserve(state.definitions.size());
for (auto &[key, image] : state.definitions) {
const GraphDefinition *header = graph_definition(image);
if (header != nullptr && header->total_bytes == image.size()) {
list.entries.push_back(GraphHostDefinition{key, image.data(), image.size()});
}
}
return list;
}

static uint32_t next_fanin_seen_epoch(PTO2OrchestratorState *orch) {
uint32_t next = orch->fanin_seen_current_epoch + 1;
if (next == 0) {
Expand Down Expand Up @@ -1401,10 +1413,9 @@ bool graph_build_submission_image(
const std::vector<std::byte> &definition_image, const CoreTaskArgs &args, std::vector<std::byte> *submission_image
) {
if (submission_image == nullptr || graph_definition(definition_image) == nullptr) return false;
const size_t definition_offset = PTO2_ALIGN_UP(sizeof(GraphSubmission), alignof(GraphDefinition));
const size_t tensors_offset = PTO2_ALIGN_UP(definition_offset + definition_image.size(), alignof(GraphTensor));
const size_t tensors_offset = PTO2_ALIGN_UP(sizeof(GraphSubmission), alignof(GraphTensor));
const size_t tensor_bytes = static_cast<size_t>(args.tensor_count()) * sizeof(GraphTensor);
if (definition_offset > UINT32_MAX || tensors_offset > UINT32_MAX || tensors_offset > UINT32_MAX - tensor_bytes) {
if (tensors_offset > UINT32_MAX || tensors_offset > UINT32_MAX - tensor_bytes) {
return false;
}
const size_t tensors_end = tensors_offset + tensor_bytes;
Expand All @@ -1416,7 +1427,6 @@ bool graph_build_submission_image(
return false;
}
submission_image->assign(total_bytes, std::byte{0});
std::memcpy(submission_image->data() + definition_offset, definition_image.data(), definition_image.size());
auto *tensors = reinterpret_cast<GraphTensor *>(submission_image->data() + tensors_offset);
for (int32_t i = 0; i < args.tensor_count(); ++i)
tensors[i] = graph_tensor_pack(args.tensor(i).ref());
Expand All @@ -1430,8 +1440,8 @@ bool graph_build_submission_image(
const GraphDefinition &definition = *graph_definition(definition_image);
GraphSubmission submission{};
submission.graph_key = definition.full_key;
submission.definition_hash = definition.content_hash;
submission.total_bytes = static_cast<uint32_t>(submission_image->size());
submission.definition_offset = static_cast<uint32_t>(definition_offset);
submission.tensors_offset = static_cast<uint32_t>(tensors_offset);
submission.tensor_count = static_cast<uint32_t>(args.tensor_count());
submission.scalars_offset = static_cast<uint32_t>(scalars_offset);
Expand Down
61 changes: 57 additions & 4 deletions src/a5/runtime/host_build_graph/host/runtime_maker.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -467,6 +467,50 @@ bool upload_graph_submissions(
std::unordered_map<uint64_t, uint32_t> occurrences;
uploaded_bytes = 0;
const size_t count = graph_host_upload_count(graph_state);
// Pass 1: upload each distinct Definition once as a shared device object
// ([GraphDefinitionHeader][Definition image]) keyed by content identity.
// Submissions reference the object's GM address, so this pass completing
// before any submission is uploaded is what makes the reference safe —
// the device boots only after both passes.
GraphHostDefinitionList definitions = graph_host_definitions(graph_state);
struct UploadedDefinition {
void *device_object; // GM address; host must not dereference
const GraphDefinition *host_view; // the host-side image the object was built from
};
std::unordered_map<uint64_t, UploadedDefinition> definition_objects;
for (const GraphHostDefinition &entry : definitions.entries) {
if (entry.data == nullptr || entry.bytes < sizeof(GraphDefinition)) continue;
const auto *definition = reinterpret_cast<const GraphDefinition *>(entry.data);
if (definition->total_bytes != entry.bytes || definition->full_key != entry.full_key) continue;
const size_t object_bytes = sizeof(GraphDefinitionHeader) + entry.bytes;
void *object =
api->acquire_graph_definition_buffer(entry.full_key, object_bytes, alignof(GraphDefinitionHeader));
if (object == nullptr) {
LOG_ERROR(
"host-orch: failed to retain %zu bytes for Graph Definition key=%#llx", object_bytes,
static_cast<unsigned long long>(entry.full_key)
);
return false;
}
std::vector<std::byte> staging(object_bytes, std::byte{0});
auto *header = reinterpret_cast<GraphDefinitionHeader *>(staging.data());
header->magic = GRAPH_DEFINITION_OBJECT_MAGIC;
header->verify_state.store(
static_cast<uint32_t>(GraphDefinitionVerifyState::UPLOADED), std::memory_order_relaxed
);
header->definition_bytes = static_cast<uint32_t>(entry.bytes);
header->content_hash = definition->content_hash;
header->full_key = definition->full_key;
std::memcpy(staging.data() + sizeof(GraphDefinitionHeader), entry.data, entry.bytes);
if (api->copy_to_device(object, staging.data(), object_bytes) != 0) {
LOG_ERROR("host-orch: failed to upload Graph Definition object");
return false;
}
definition_objects.emplace(definition->content_hash, UploadedDefinition{object, definition});
uploaded_bytes += object_bytes;
}

// Pass 2: per-submission execution storage + the small reference image.
for (size_t index = 0; index < count; ++index) {
std::optional<GraphHostUpload> upload = graph_host_upload(graph_state, index);
if (!upload.has_value() || upload->outer_slot == nullptr || upload->data == nullptr ||
Expand All @@ -480,13 +524,21 @@ bool upload_graph_submissions(
LOG_ERROR("host-orch: Graph submission size does not match its POD image");
return false;
}
const GraphDefinition *definition = graph_submission_definition(*submission);
auto object_it = definition_objects.find(submission->definition_hash);
if (object_it == definition_objects.end() || object_it->second.device_object == nullptr) {
LOG_ERROR("host-orch: Graph submission has no uploaded Definition object");
return false;
}
// Capacities come from the host-side Definition image the device
// object was built from; the GM object itself is never dereferenced
// on the host.
const GraphDefinition *definition = object_it->second.host_view;
size_t execution_bytes = 0;
if (definition == nullptr || definition->full_key != submission->graph_key || definition->task_count == 0 ||
definition->task_count > GRAPH_MAX_NODES ||
if (definition->task_count == 0 || definition->task_count > GRAPH_MAX_NODES ||
definition->full_key != submission->graph_key ||
!graph_execution_storage_bytes(
static_cast<int32_t>(definition->task_count), definition->tensor_arg_count,
definition->scalar_arg_count, definition->total_bytes, &execution_bytes
definition->scalar_arg_count, &execution_bytes
)) {
LOG_ERROR("host-orch: invalid Graph execution storage request");
return false;
Expand All @@ -502,6 +554,7 @@ bool upload_graph_submissions(
);
return false;
}
submission->definition_addr = reinterpret_cast<uint64_t>(object_it->second.device_object);
submission->execution_storage = reinterpret_cast<uint64_t>(execution_storage);
submission->execution_storage_bytes = execution_bytes;
submission->local_execution = 0;
Expand Down
Loading
Loading