Repository navigation
POC: genuine cross-instance TXE dispatch across two TSISIM instances - #165
Draft
akapoor3518 wants to merge 10 commits into
Draft
akapoor3518 wants to merge 10 commits into
akapoor3518 wants to merge 10 commits into
Conversation
Adds a TSI_REMOTE_TXE_POC remote-worker mode to examples/simple/simple-backend-tsi.cpp: a client on one TSISIM instance can dispatch a real ADD op over plain TCP to a worker process on a second, fully independent TSISIM instance, which executes it on its own local Tsavorite/TXE runtime (ggml_backend_tsavorite_init -> ggml_add -> ggml_backend_graph_compute, the same path llama.cpp's own inference loop uses) and returns the numerically correct result. Verified across multiple runs (n=8, 32, 137), confirmed independently from both sides' logs. Zero changes to ggml-tsavorite.cpp itself. Also includes two small build-system fixes needed to get a from-scratch aarch64 build working on a bare host: onnxruntime-training is not actually required by tsi-pkg-build.sh's setup_python(), so its hard failure is downgraded to a warning; and ggml-tsavorite's target now explicitly requests C++20 via set_target_properties, since ggml/CMakeLists.txt's directory-scoped CMAKE_CXX_STANDARD 17 was silently winning over the backend's own --std=c++20 request (GCC's last-std-flag-wins rule), which breaks compilation against SDK headers that require C++20 (std::source_location, std::format). This is explicitly a POC: MAX_TXES_SUPPORTED is still 20 (not generalized to 40), acquire_device_blocking() does not route to remote devices, only ADD is dispatched remotely (not MAT_MUL yet), and none of this is wired into llama-cli's real inference graph. Full writeup, exact verified command output, and the 5 build/deployment bugs found along the way are documented separately. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…vorite.cpp Extends the round-2 ADD-only cross-instance POC to real MAT_MUL, wired into the actual multi-TXE row-tiling loop ggml_tsavorite_run_tmu_mul_mat() uses during real llama-cli inference (not just a standalone test harness): - New tsavorite-model-deployment.yaml control surface: multi_node_enable, remote_txe_host, remote_txe_port. txe_count keeps meaning TXE count on EACH node; multi_node_enable=false is byte-for-byte unchanged behavior. - Row-tile loop doubles to 2*txe_count logical lanes when multi_node_enable is true: [0,txe_count) dispatch locally exactly as before, [txe_count, 2*txe_count) serialize A row-tile + full B over TCP to a second TSISIM instance, which reconstructs a real ggml_mul_mat + graph_compute() call against its own local runtime (not a hand-rolled reimplementation of the Triton blob ABI). - Deliberately did NOT widen device_free[]/packed_args[]/MAX_TXES_SUPPORTED to a literal 2x range, since those are shared by every op type (ADD, RMS_NORM, etc.) and a remote-range deviceId reaching any of them would be undefined behavior; used an isolated remote-dispatch concurrency pool scoped only to MAT_MUL instead. - simple-backend-tsi.cpp's remote-worker mode extended to handle MAT_MUL requests alongside the existing ADD protocol, concurrently (one thread per connection). - Fixed a real transpose bug in the client-side result copyback (ggml mul_mat's result buffer is column-major flat, not row-major) caught via a synthetic test before the expensive real-model run. - Fixed a real concurrency deadlock: ggml-tsavorite.cpp's TXE dispatch state (workers vector, device_free[], their mutexes) is process-global and was never validated under concurrent top-level graph_compute() callers. Added a worker-side compute mutex plus capped client-side remote concurrency to 1 in-flight dispatch to guarantee the same repeated-sequential-call pattern round 2 already proved correct. Verified with a real TinyLlama 1.1B (Tiny-Llama-v0.3-FP32-1.1B-F32.gguf) end-to-end run: identical deterministic output vs a single-instance baseline, and genuine cross-instance MAT_MUL dispatch confirmed independently from the remote instance's own log (13M+ lines of real traffic). Currently a small (~4%) wall-clock regression vs baseline, not a speedup yet -- root cause and next steps documented in tsisim-multinode-poc-llamacpp.md ROUND 3. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root-causes and fixes the ~4.4% slowdown Round 3 measured, instead of just re-capping concurrency: - simple-backend-tsi.cpp: worker now uses a single persistent dispatch thread (spawned as the same OS thread that called ggml_backend_tsavorite_init()) fed by a job queue, instead of a fresh std::thread per connection calling ggml_backend_graph_compute() directly. Every previously-proven-correct case (local llama-cli inference, Round 2's ADD POC) always calls graph_compute() from the same OS thread that initialized the runtime; Round 3's per-connection thread design broke that invariant, which is the most likely real root cause of the residual hang Round 3's own mutex fix did not fully resolve. Verified with a 40-concurrent-connection stress test at production scale (M=56, K=2048) against the live worker: 40/40 correct, no hang. - ggml-tsavorite.cpp: rows_per_txe is now computed from num_of_txes only (never active_txes), so tile size always matches the non-multi-node baseline exactly instead of shrinking when multi_node_enable doubles the lane count. The remote lane-range of each wave is coalesced into ONE combined MAT_MUL request instead of up to num_of_txes tiny per-tile requests, letting the remote instance's own already-proven internal multi-TXE splitting handle it at full baseline tile size. This eliminates the ~2x kernel-run dispatch-count inflation Round 3 measured (1700 -> 3271 for the same 85 ops). Real-model result (TinyLlama 1.1B, same prompt/settings as Round 3): 41m34.626s multi-node vs Round 3's 57m24.062s single-instance baseline and 59m58.290s (broken) multi-node -- a genuine ~27.6% speedup over baseline, not a regression. Output byte-identical to both prior runs. Cross-instance dispatch independently verified via the remote instance's own worker log: 69 real coalesced MAT_MUL requests, all succeeded, production-scale M_valid values (56, 8, 2). See ROUND 4 section in tsisim-multinode-poc-llamacpp.md for full writeup. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…bility Replaces the per-wave fresh connect()/close() to the remote TXE worker (Round 3/4) with a single persistent TCP connection opened once at bring-up and reused for the life of the client process, with a reconnect-and-retry fallback and symmetric cleanup at every shutdown path. The worker side needed no changes -- its existing per-connection loop already serves multiple sequential requests before closing, verified via a 40/40-correct synthetic test over one connection and a clean-recovery test for an abrupt client disconnect. Also adds: (1) a documented, verified answer to where multi_node_enable needs to be set (client only -- the worker determines its role purely from its remote-worker CLI argument, and adding multi_node_enable to its own yaml would risk a spurious startup abort); (2) per-node MUL_MAT TXE-kernel-dispatch counts printed on both the client (local/remote breakdown) and worker (running total), traced to their real open-source origin in llama-context.cpp/ggml.c; (3) a documentation-only comment recording a newly-found, pre-existing property of the current rows_per_txe formula that makes the remote dispatch lane range mathematically unreachable for any M -- confirmed by a real TinyLlama 1.1B run showing zero remote dispatches -- left unfixed pending a design decision, since its outside this rounds scope. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…lly engages Round 5 proved that sizing rows_per_txe from num_of_txes alone (Round 4) makes the remote lane range mathematically unreachable for any M, since num_of_txes tiles at that size always cover M by construction. Fixed by computing the num_of_txes-only baseline first, and only recomputing from active_txes (spilling into the remote lane range) when that baseline already saturates every local lane. Small ops are byte-for-byte unaffected; only ops big enough to already need all num_of_txes local lanes get finer tiling and genuine remote parallelism. Verified: synthetic K=256/M=5000/N=64 test showed 20 local + 1 real remote dispatch, cross-confirmed from both sides worker.log. Real TinyLlama 1.1B comparison: single-instance baseline 40m3.231s vs multi-node 15m3.290s (62.4% faster), byte-identical output, node1 local=1700/node2 remote=85 dispatch-calls matching exactly on both sides own accounting.
akapoor requested that node1 know exactly how many TXEs/kernel-runs node2 used per remote dispatch, not the Round 5 placeholder (1 coalesced call = 1 unit). The worker already computes this (result->tsi_kernel_runs from its own ggml_mul_mat() run) -- now it sends that count back to the client right after the response magic, before the C data. The client accumulates the real count instead of a flat +1 per dispatch call, so the closing MUL_MAT OPU TSI_KERNEL-RUN breakdown line now shows the genuine per-node split directly, without needing to cross-reference instance 2 own worker.log to get its true count. Verified: synthetic K=256/M=5000/N=64 test now reports node1/local=20 + node2/remote=20 (previously node2/remote=1, the placeholder), matching what cross-referencing worker.log already showed.
Round 7b: node->tsi_kernel_runs (which feeds the standard GGML Perf Summary tables MUL_MAT row, shared by every backend) was still using the Round 5 placeholder (a flat +1 per coalesced remote wave, regardless of how many real kernel launches that wave actually caused on node 2). Fixed by tracking the real per-call remote count in a per-node atomic (remote_kernel_runs_this_node) and folding it into node->tsi_kernel_runs/ profile.kernel_calls/device stats at the end of the dispatch function, so the main table finally agrees with the separate Tsavorite-only breakdown line added in Round 7. Round 8: generalized multi_node_enable from a bool to an int representing total node count (1 = disabled, 2 = todays only implemented case, >2 accepted for a future real rack size like 18 but not yet functionally supported -- only one remote peer is ever contacted today). Renamed remote_txe_port -> remote_txe_port_start for future extension. Deployment yaml key multi_node_enable -> multi_node (int), remote_txe_port -> remote_txe_port_start.
tsi-pkg-build.sh generates ggml.sh (deployed to every TSISIM instance and invoked by run_llama_cli.sh before every inference run), which regenerates the deployment yaml on every invocation, preserving only fields its own template knows about. The multi_node_enable/remote_txe_host/remote_txe_port fields added in earlier rounds were only ever manually patched onto the already-deployed copy, never added to this tracked template -- meaning a fresh package build would silently regenerate ggml.sh without them, wiping multi-node config on every run_llama_cli.sh invocation. Added properly here, using the round 8 key names (multi_node int, remote_txe_port_start).
- ensure_tsi_runtime_initialized() now prints the deploy config ~2 fields/line instead of one long concatenated line, for demo legibility. - GGML Perf Summary TSI_KERNEL-RUN column split into N1_KRUN/N2_KRUN: ggml_tensor.tsi_kernel_runs and the perf-aggregation subtotal structs become 2-element arrays ([0]=node1/local, [1]=node2/remote), threaded through ggml-tsavorite.cpp -> ggml.c accumulator -> llama-context.cpp printer. Pure reporting change -- no dispatch/counting logic changed, verified against the same synthetic K=256/M=5000/N=64 split (20/20) Round 7/8 already confirmed.
Embeds the bring-up + verification steps (worker start, yaml config, connectivity check, run, 3-way dispatch verification, TXE utilization check) directly in the script header so they are discoverable without needing PR #165 open, which they still point to for full context.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOT READY FOR REVIEW. This is a draft POC/feasibility branch, kept open as a reference point for follow-up work — not intended to be reviewed or merged as-is.
Summary
Proof-of-concept for multi-node tensor-parallel inference: can two separate TSISIM instances (simulating two Helix-R boxes, 20 TXE each) participate in a single distributed computation over the network?
Adds a
TSI_REMOTE_TXE_POCremote-worker mode toexamples/simple/simple-backend-tsi.cpp: a client process on one TSISIM instance sends real MAT_MUL/ADD work over plain TCP to a worker process on a second, fully independent TSISIM instance. The worker executes it on its own local Tsavorite/TXE runtime (the sameggml_backend_tsavorite_init()→ggml_backend_graph_compute()call sequence llama.cpp's own inference loop uses) and returns the result. Zero changes toggml-tsavorite.cpp's own op-dispatch logic.Also includes two small build-system fixes needed for a from-scratch aarch64 build: a non-fatal fallback when
onnxruntime-traininghas no wheel for this platform, and an explicit C++20 standard override forggml-tsavorite(the parent build's directory-scoped C++17 default was silently winning under GCC's last-flag-wins rule).Final verified result
Same prompt, same model (TinyLlama-1.1B), same settings, single-instance vs. two-instance (40 TXE combined) run back-to-back:
is Luna.is Luna.(identical)Independently cross-checked three ways: node 1's own closing accounting line, node 2's own worker log (matching kernel-run totals), and
tsictlhardware execute counts on both instances showing a flat, even count across all 20 TXEs on each node. Verified genuinely cross-instance across three independent runs at different points in this PR's development (44–62% faster each time; wall-clock varies on a shared dev box, but the mechanism engaging and the output correctness never did).Full detail on Confluence
Architecture diagrams, the complete round-by-round investigation (including two real regressions found and fixed along the way), the full reproduction cheatsheet, and all raw logs live here — the persistent home for this, not this PR description:
https://tsavoritesi.atlassian.net/wiki/spaces/SD/pages/1944420365/
What changed in this PR
examples/simple/simple-backend-tsi.cpp— the remote-worker mode itself (client/worker TCP protocol, dispatch thread, MAT_MUL/ADD framing)ggml/src/ggml-tsavorite/ggml-tsavorite.cpp— remote-lane tile-sizing logic forggml_tsavorite_run_tmu_mul_mat()ggml/include/ggml.h,ggml/src/ggml.c— per-node (N1_KRUN/N2_KRUN) kernel-run accounting threaded through the standard perf-summary pipelinesrc/llama-context.cpp— perf table printer updated for the two-column breakdownggml/src/ggml-tsavorite/CMakeLists.txt,tsi-pkg-build.sh— the two build fixes described aboveExplicit scope — this is a POC, not close to production
Fixed two-node topology only, dotted-IPv4 addressing, validated on TSISIM only (not FPGA/silicon). Worker launch today is manual (start it by hand on node 2 before node 1 runs) — master-driven provisioning is tracked future work. Full scope notes on Confluence.
Why draft
This is a feasibility proof, opened as a reference point for follow-up work (generalizing the allocator, extending remote dispatch to more op types, wiring in master-driven worker provisioning) — not intended to merge as-is.
Test plan