[bong-water-water-bong]

Linux NPU route for 1bit serve --device onnx through AMD's Ryzen AI Software (RyzenAI execution provider), plus the ONNX pieces of the release package. AMD's Ryzen AI Software is not shipped and not copied into the repository: the engine builds against, and runs, an install the user made under AMD's own licence (RYZENAI_INSTALL_PATH at build time, ONEBIT_RYZENAI_ROOT at run time).

Owner decision 2026-10-20: ONNX goes in the engine release.

What changes

• app/serve.cpp (onnx device): reads the model's genai_config.json (RyzenAI provider, hybrid_opt_token_backend: npu). With the provider declared and ONEBIT_RYZENAI_ROOT=<venv> set, serve builds the child's environment: XILINX_XRT (ONEBIT_XRT_ROOT, else $XILINX_XRT, else /opt/xilinx/xrt), LD_LIBRARY_PATH with XRT's lib first (the venv's voe wheel bundles a stale libxrt_coreutil.so.2.19), and RYZENAI_EP_PATH. A server built against the venv (provider copy beside it) keeps its own directory before <venv>/deployment/lib; a server without one (the release package's, built against Microsoft's ONNX Runtime) gets <venv>/deployment/lib first so the venv's ONNX Runtime GenAI loads. Unset knob or a CPU model: unchanged. An NPU model's failed (5xx) non-streamed request is retried once after 1 s (one-off firmware TDR on the first dispatch after idle).
• Child lifecycle (serve.cpp): reap() bookkeeping shared by alive(), stop() and main's watchdog kill_now(); exit status on the "backend exited" line; ONEBIT_BACKEND_STOP_GRACE_S (default 30 s) before SIGKILL, with a warning line naming the amdxdna risk.
• Release package: scripts/build-onnx.sh puts the ryzenai-server, ONNX Runtime GenAI and ONNX Runtime LICENSE / ThirdPartyNotices.txt texts beside the binary; scripts/package-linux.sh packs them in onnx/. scripts/weekly.sh unsets RYZENAI_INSTALL_PATH for the release build, and package_checks fails when onnx/ is missing or holds anything beyond ryzenai-server, libonnxruntime-genai.so, libonnxruntime.so* and those texts (so no RyzenAI/VitisAI provider, AMD_LICENSE or AIE compiler library can ship). NOTICE says what onnx/ carries.
• scripts/build-onnx.sh: with RYZENAI_INSTALL_PATH, clears executable-stack requests on the copied libraries (patchelf --clear-execstack, copies only). Sets ryzenai-server's search path to $ORIGIN only: DT_RUNPATH for the Microsoft (release) build so LD_LIBRARY_PATH can put a venv's ONNX Runtime GenAI first (DT_RPATH kept the CPU-only copy: "Unknown provider name 'RyzenAI'", device check 2026-10-20), DT_RPATH for a build against the venv; no build-tree path.
• SIGTERM shim (scripts/onnx/signal_shim.c, linked with -Wl,--wrap=signal; ryzenai-server's sources unchanged). Root cause of the NPU shutdown hang: ryzenai-server's SIGTERM handler calls std::exit() from signal context; exit() destroys exit-time objects in reverse creation order, and g_server (holding the model) was created first, so ONNX Runtime GenAI's, the RyzenAI provider's and XRT's globals (created during the model load) are destroyed under a live session and hardware context. On the CPU this shows as OGA Error: 1 instances of N10Generators5ModelE were leaked on every exit; on the NPU it hung after [Server] Shutting down... (never reaching [InferenceEngine] Shutting down) until serve's SIGKILL. The shim blocks SIGTERM/SIGINT on every thread, takes them on its own thread, registers an exit handler that deletes the server (and model) first, and calls stop(): main() returns and exit() runs once, model before runtime globals.
• Docs: docs/onnx.md "Linux NPU" (non-invasive install: XRT debs extracted with dpkg-deb -x, not installed; traps; knobs; shutdown policy; measured tables), what the release package ships; docs/serve.md stop-grace knob; docs/npu.md pointer.
• Tests: tests/onnx_npu_smoke.sh (ctest onnx_npu_smoke under -DONEBIT_ONNX=ON, skip 77 without venv / NPU / model); smoke_serve case for a backend that exits on its own (fake_exit in tests/fake_backend.py).

Merge of main (47 commits)

Merged origin/main (no rebase). Conflicts:

• app/serve.cpp: Child takes main's on_line (stderr tail / watchdog) and this branch's env_override (..., ready_path, on_line, env_override); alive() keeps reap(), main's kill_now() now goes through reap() too; all_alive moved before the request handlers (the NPU retry uses it); the retry passes main's &watch; the exit path keeps main's lifecycle.count_backend_exit() and this branch's exit text.
• docs/windows.md: deleted, as on main (no Windows package). docs/onnx.md: main's Windows wording, this branch's Linux NPU section; Windows items dropped from "Next".
• docs/serve.md, tests/fake_backend.py: both sides kept.

Tests (ryzen, CPU only, no GPU, no NPU)

• CI configuration (gcc-16/g++-16, Ninja, Release, every CI target): build clean, ctest 34/34 passed.
• -DONEBIT_ONNX=ON build (Microsoft ONNX Runtime GenAI 0.11.2 / ORT 1.23.2): onnx_npu_smoke skipped (77, no NPU), smoke_serve and watchdog_serve passed.
• CPU model amd/Qwen2.5-0.5B-Instruct-quantized_int4-float16-cpu-onnx through 1bit serve (build tree and the unpacked package): "The capital of France is Paris.", 7 SSE chunks streamed, child gone after one SIGTERM.
• scripts/package-linux.sh on that build: onnx/ = ryzenai-server, the two ORT libraries, five licence texts; ldd resolves inside onnx/. The onnx/ part of package_checks passes on it and fails when AMD_LICENSE and libonnxruntime_providers_ryzenai.so are dropped in.
• After the RUNPATH/shim commit (d5d5736): ctest 34/34 again; the packaged ryzenai-server has RUNPATH $ORIGIN, no RPATH; the CPU model through the packaged 1bit serve answers "Paris", streams 7 chunks, and one SIGTERM ends serve in 0.14 s with exit 0, [InferenceEngine] Shutting down printed and no "leaked" lines (before: exit 15 with both leak lines). The onnx/ package check passes on the new package and fails on a DT_RPATH, shim-less server.
• Child environment with a fake venv/XRT/server: provider beside the server -> xrt/lib:<server>:<venv>/deployment/lib; none -> xrt/lib:<venv>/deployment/lib:<server>.

Device check (pending)

Not run: the NPU belongs to another agent right now. On strixhalo, on a quiet NPU, one job at a time under ~/npu.lock. Policy from 2026-10-11 stands: ryzenai-server does not always shut down cleanly; do not kill -9 anything holding /dev/accel/accel0.

# build this branch against the venv (route A) and package-style (route B)
git -C ~/engine fetch && git -C ~/engine worktree add ~/wt/onnx386-dev origin/1bit/onnx-npu-linux
cd ~/wt/onnx386-dev && git submodule update --init third_party/ryzenai-server
RYZENAI_INSTALL_PATH=~/ryzenai-1.8/venv cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=gcc-16 -DCMAKE_CXX_COMPILER=g++-16 -DONEBIT_ONNX=ON
RYZENAI_INSTALL_PATH=~/ryzenai-1.8/venv cmake --build build --target onebit
env -u RYZENAI_INSTALL_PATH cmake -B build-ms -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=gcc-16 -DCMAKE_CXX_COMPILER=g++-16 -DONEBIT_ONNX=ON
env -u RYZENAI_INSTALL_PATH cmake --build build-ms --target onebit

export ONEBIT_RYZENAI_ROOT=~/ryzenai-1.8/venv
export ONEBIT_XRT_ROOT=~/ryzenai-1.8/sysroot/opt/xilinx/xrt
M=~/models/oga/Qwen2.5-0.5B-Instruct_rai_1.8.0_npu_4K
T0=$(date '+%Y-%m-%d %H:%M:%S')

# expect: build-ms/onnx/ryzenai-server RUNPATH $ORIGIN (no RPATH); both binaries carry the shim
readelf -d build-ms/onnx/ryzenai-server | grep -E 'R(UN)?PATH'; readelf -d build/onnx/ryzenai-server | grep -E 'R(UN)?PATH'
strings -a build-ms/onnx/ryzenai-server | grep 1bit-onnx-signal-shim; strings -a build/onnx/ryzenai-server | grep 1bit-onnx-signal-shim

# route A: server built against the venv; the smoke test (health, models, chat, accel0 held, tok/s,
# SIGTERM reap; FAILS if serve had to SIGKILL)
flock ~/npu.lock tests/onnx_npu_smoke.sh build/1bit "$M"

# route B: the release-style server (Microsoft ORT beside it, venv libs first)
flock ~/npu.lock sh -c '
  build-ms/1bit serve --device onnx -m "$0" --port 18860 > /tmp/onnx386-b.log 2>&1 & S=$!
  until curl -sf localhost:18860/health >/dev/null; do kill -0 $S || exit 1; sleep 1; done
  curl -s localhost:18860/v1/chat/completions -H "content-type: application/json" \
    -d "{\"messages\":[{\"role\":\"user\",\"content\":\"What is the capital of France? Answer in one sentence.\"}],\"max_tokens\":32}"
  echo; ls -l /proc/$(pgrep -P $S ryzenai-server)/fd | grep -c accel
  kill -TERM $S; wait $S' "$M"
grep -E "Using NPU|execution provider|backend exited|SIGKILL|Shutting down|leaked" /tmp/onnx386-b.log

# faults since T0 (kernel log by timestamp)
journalctl -k --since "$T0" --no-pager | grep -iE 'amdxdna|aie2|xdna|tdr|timeout|BUG|panic|list_del'

Pass: both routes (route B is now expected to pass: the release-style server has RUNPATH, so the venv's ONNX Runtime GenAI loads) answer with "Paris" (known answer), the child holds accel0 (count >= 1), the log shows the NPU (VitisAI/RyzenAI) provider, and the kernel log shows no new amdxdna fault after T0. SIGTERM criterion: one SIGTERM to serve ends ryzenai-server on its own: the log shows [Server] Shutting down... then [InferenceEngine] Shutting down, no "leaked" lines, no ignored SIGTERM ... sending SIGKILL line, and serve's "backend exited" text (if any) is exit 0; route A's smoke test fails on the SIGKILL line. If ryzenai-server still hangs after [InferenceEngine] Shutting down, the hang is in the provider's own teardown of a released model (not the exit order) and the 30 s window plus the once-per-boot policy stay. A single TDR on the first dispatch is the known one-off (the retry covers it); any second fault stops the check.

Measured earlier (Strix Halo, 2026-10-11, AMD NPU-only 1.8.0 models through 1bit serve --device onnx, n=3, 128 generated)

Decode at 128 / 512 / 2048 prompt tokens: Qwen2.5-0.5B 4K 70.1 / 66.7 / 64.9 tok/s; Llama-3.2-1B 48.9 / 47.6 / 45.8; Llama-3.2-3B 21.1 / 20.4 / 18.8; Phi-3.5-mini 13.5 / 12.7 / 10.7. Table in docs/onnx.md.

Deferred

• Registry: --device auto sends every genai_config.json directory to onnx; the registry has no notion of an OGA directory's provider (NPU vs CPU).
• The NPU retry matches any 5xx from an NPU backend, not the TDR text specifically.

Licence

AMD's Ryzen AI Software (XRT, the RyzenAI execution provider, AMD's ONNX Runtime GenAI build) stays under AMD's EULA, installed by the user; the engine ships none of it and copies none of it into the repository, and the release package check enforces that. ryzenai-server (lemonade-sdk, MIT) is a pinned submodule; ryzenai-server, ONNX Runtime GenAI and ONNX Runtime (Microsoft, MIT) ship in onnx/ with their licence texts. tools/copyright.py --check passes.

🤖 Generated with Claude Code

[github-actions]

Preparing review...

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 598afb7)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Child Lifecycle Consistency The Child::stop() method now uses a grace period before escalating to SIGKILL, which is good for NPU contexts. However, the stop_grace_s() function does not validate that the value is within the expected range (0..86400) for the grace period, potentially allowing invalid values that could lead to unexpected behavior or denial of service. int stop_grace_s() { if (const char* e = std::getenv("ONEBIT_BACKEND_STOP_GRACE_S"); e && *e) { char* end = nullptr; const long v = std::strtol(e, &end, 10); if (end && *end == '\0' && v >= 0 && v <= 86400) return int(v); std::fprintf(stderr, "1bit serve: ONEBIT_BACKEND_STOP_GRACE_S=%s ignored (whole seconds, 0..86400); using 30\n", e); } return 30; } Environment Override Logic The merge_env() function handles environment variable overrides but does not ensure that the override variables are actually valid for the child process. If an invalid or dangerous environment variable is passed in env_override, it could lead to unexpected behavior or security issues. void merge_env(std::vector<std::string>& env_store, const std::vector<std::string>& env_extra, const std::vector<std::string>& env_override) { for (const auto& e : env_extra) { const std::string key = e.substr(0, e.find('=') + 1); bool present = false; for (const auto& have : env_store) present = present || have.rfind(key, 0) == 0; if (!present) env_store.push_back(e); } for (const auto& e : env_override) { const std::string key = e.substr(0, e.find('=') + 1); std::erase_if(env_store, [&](const std::string& have) { return have.rfind(key, 0) == 0; }); env_store.push_back(e); } } NPU Retry Logic The NPU retry logic in forward() only retries non-streamed requests that fail with a 5xx status. However, it does not check if the retry is actually needed or if the error is recoverable, which could lead to unnecessary retries or masking of actual errors. if (backends[pick].npu_retry && r.status >= 500 && !wants_stream(q.body) && all_alive()) { // The NPU's first dispatch after idle can hit a one-off firmware timeout (TDR); the // device recovers and the same request then runs. One retry, said aloud, for a reply // nothing of which reached the client (a stream is not retried: its bytes went out). std::fprintf(stderr, "1bit serve: %s answered %d on %s; retrying once (first NPU dispatch after idle can time out)\n", backends[pick].device.c_str(), r.status, q.path.c_str()); std::this_thread::sleep_for(std::chrono::seconds(1)); r.status = 200; r.body.clear(); forward(backends[pick].port, held, -1, id, drop_model, set_model, q, r, &watch); } Executable Stack Handling The script clears executable stacks on copied libraries, but it does not validate that the patchelf command is available before attempting to use it. If patchelf is not installed, the build will fail, which could be confusing for users. if [ -n "${RYZENAI_INSTALL_PATH:-}" ]; then rwe=() for so in "$prefix"/*.so "$prefix"/*.so.*; do [ -f "$so" ] && [ ! -L "$so" ] || continue readelf -lW "$so" 2>/dev/null | grep -q 'GNU_STACK.*RWE' && rwe+=("$so") done if [ ${#rwe[@]} -gt 0 ]; then command -v patchelf > /dev/null || { echo "build-onnx.sh: ${#rwe[@]} copied libraries request an executable stack (${rwe[*]##*/}) and patchelf is not installed to clear it: apt install patchelf"; exit 1; } for so in "${rwe[@]}"; do patchelf --clear-execstack "$so"; echo "cleared executable stack: ${so##*/}"; done else echo "executable-stack scan: none of the copied libraries request one" fi fi

[bong-water-water-bong]

Serve-path numbers (Linux NPU, strixhalo)

Build: this branch's working tree at ~/wt/onnx-npu-linux (base faea518 + the PR's changes, build/1bit + build/onnx/ryzenai-server, built 01:22 ADT), ONEBIT_RYZENAI_ROOT=~/ryzenai-1.8/venv, ONEBIT_XRT_ROOT=~/ryzenai-1.8/sysroot/opt/xilinx/xrt, ONEBIT_ONNX_SERVER=build/onnx/ryzenai-server, 1bit serve -m ~/models/oga/Qwen2.5-0.5B-Instruct_rai_1.8.0_npu_4K --device onnx. One server lifetime per model; all prompt lengths and repeats inside it. Client: streaming /v1/completions (stream_options.include_usage), 128 tokens, temperature 0; TTFT and decode by wall clock; prefill tok/s = prompt_tokens / TTFT (ryzenai-server sends no timings, so this includes the HTTP round trip and the server's tokenizing). Prompts cut to exactly N tokens with the model's own tokenizer.

Box: engine #315 GPU campaign running concurrently (HRX GLM servers, ~/box.lock held by it; those llama-server processes also keep /dev/accel/accel0 open through HRX's amdxdna backend, but never dispatch to it). NPU exclusive otherwise. Repeat 1 ran while the campaign was active (load 13-15, Tctl 86-90 C), repeats 2-3 while it was paused (Tctl 46-54 C). Cells start only with Tctl < 88 C.

Qwen2.5-0.5B-Instruct_rai_1.8.0_npu_4K (n=3, median +- half-range; per-repeat decode in brackets)

prompt tokens	TTFT s	prefill tok/s (from TTFT)	decode tok/s
128	0.317 +- 0.122	403 +- 100	70.1 +- 6.9 (64.3 / 70.1 / 78.2)
512	0.494 +- 0.082	1037 +- 184	66.7 +- 7.0 (61.3 / 66.7 / 75.3)
2048	0.970 +- 0.342	2111 +- 454	64.9 +- 6.3 (52.4 / 64.9 / 65.1)

The low end of each bracket is repeat 1 (campaign active). Server up (health 200) 3.0 s after exec; the child ryzenai-server (pid 227108) was the only holder of /dev/accel/accel0 in serve's process tree for the whole lifetime; no retry and no 5xx in the lifetime (warm-up + 9 requests).

Same model, the _npu_16K package, same protocol (quiet box): 128: 75.8 +- 0.9, 512: 72.4 +- 0.5, 2048: 63.6 +- 0.6 tok/s decode; TTFT 0.25 / 0.31 / 0.82 s. An 8192-token prompt was truncated to 4096 by ryzenai-server ([WARNING] Prompt exceeds maximum length (8192 > 4096): the model's rai_config.json lists 4096 under the 1.7.0 key that the server reads, so the _16K package gets the same 4096 cap as _4K on this path): TTFT 2.43 s on the 4096 kept, decode 36.2 tok/s at that context. Worth a line in docs/onnx.md next to the RYZENAI_VERSION note.

The other AMD NPU-only models through the same path (decode tok/s at 128 / 512 / 2048, n=3, quiet box): Llama-3.2-1B 48.9 / 47.6 / 45.8; Llama-3.2-3B 21.1 / 20.4 / 18.8; Phi-3.5-mini 13.5 / 12.7 / 10.7.

Stopping: what happened in the five lifetimes

1. SIGTERM to 1bit serve (three lifetimes: 0.5B 4K, Llama 1B, Llama 3B). ryzenai-server printed [Server] Shutting down... and went no further; serve reported backend pid N ignored SIGTERM for 30 s; killing it each time, so the backend died by SIGKILL with its context open. Each of those three closes left the same block in the kernel log at the moment of the close:

   amdxdna: *ERROR* aie_send_mgmt_msg_wait: command opcode 0x106 failed, status 0x4000003
   amdxdna: *ERROR* aie2_send_host_buf_msgs: fw ctx 15 addr 0xffffffffffffffff size 0x4000000
   amdxdna: *ERROR* aie2_hwctx_restart: Map host buf failed, ret -22
   amdxdna: *ERROR* amdxdna_pm_resume_get: Resume failed: -22
   amdxdna: amdxdna_drm_close: Resume failed (-22), pid N restore data not reclaimed
   amdxdna: device is already started
   

   No TDR, no IOMMU fault, no panic; but these are not clean closes, and amdxdna_drm_close is the path that panicked this morning.
2. SIGTERM to ryzenai-server itself, with serve left running (two lifetimes, to avoid serve's SIGKILL). Phi-3.5: the backend reached [InferenceEngine] Shutting down and died in 1.7 s with double free or corruption (!prev) (an abort in its teardown; kernel log clean). serve then printed a onnx backend exited, yet its Child::stop() still waited 30 s and printed the "ignored SIGTERM ... killing it" line: alive() reaps the child with waitpid(WNOHANG) but does not record it, so stop() never sees the exit, waits the full 30 s, and sends SIGKILL to a pid that is gone. Qwen2.5-0.5B 16K: the backend stopped after [Server] Shutting down... and is still there now (pid 272606, 58 threads all in futex_wait, 0 CPU, /dev/accel/accel0 open), with serve 272601 waiting on it; both left alone (no SIGKILL on an NPU holder), which ended the run.
3. tests/onnx_npu_smoke.sh passes its "child exited after SIGTERM" check with two non-streamed chat requests; after this workload (streamed completions, 9 per lifetime) the backend did not shut down cleanly in any of five lifetimes.

Things the PR could take from this (owner's call): never escalate to SIGKILL for a backend that holds the NPU (report and leave it instead); make alive()/stop() agree on a reaped child; note the shutdown hang and the 4096 cap as known issues in docs/onnx.md.

Raw logs: strixhalo ~/w6-measured/npu-matrix-r2/ (cells.log, per-cell JSON, serve logs, rapl.csv); harness: npu-re tools/npu-matrix/run_matrix_servers.sh + report_servers.py.

[github-actions]

Persistent review updated to latest commit 948017b

[bong-water-water-bong]

Follow-up: child reap + stop grace

Commit 948017b, from the two findings in the serve-path numbers comment above.

Reap bookkeeping (app/serve.cpp, POSIX Child). Every waitpid on a backend child now goes through Child::reap(), which records the exit status and sets pid_ = -1. alive() and stop() share it: a child that already exited (the Phi-3.5 double-free abort case) makes stop() a no-op, no SIGTERM to a gone pid, no 30 s wait, no "ignored SIGTERM ... killing it". The a <device> backend exited line now says how (exit 3 / signal 6).

Stop policy. SIGTERM goes to the child itself (it shares serve's process group, so there is no group signal: that would reach serve and every other backend). The grace before SIGKILL is ONEBIT_BACKEND_STOP_GRACE_S (default 30; a bad value is reported and 30 used), and one warning line precedes the escalation, naming the amdxdna risk. The escalation stays. docs/onnx.md (Linux NPU) now lists the ryzenai-server shutdown hang / double-free as a known issue with the operating policy until the driver fix is installed (one NPU server per session, no model cycling, leave a hung stop until reboot, ONEBIT_BACKEND_STOP_GRACE_S=86400 for a long-running server), and the 4096 prompt cap the _16K packages get through ryzenai-server as pinned (rai_config 1.7.0 key). docs/serve.md has the knob.

Test. tests/smoke_serve.sh gained a case: the fake backend exits on its own ("fake_exit": 3 in a request) and serve must end at once with backend exited: exit 3 and no SIGTERM wait (grace set to 20 s in the test so a regression shows as a slow exit).

Run on strixhalo (CPU-only, onebit/fast relink under ~/lb/thermal-run.sh, GPU campaign untouched, build/onnx/ryzenai-server untouched):

• ctest -R smoke_serve: Passed (5.3 s); the new case: ok a backend that exits on its own ends serve at once (rc 1, 1s).
• manual: SIGTERM to serve with a live fake backend, grace 2 s: stop took 0.18 s, rc 0. A backend that ignores SIGTERM (trap '' TERM wrapper), grace 1 s: warning line printed after 1.1 s, SIGKILL, child gone, rc 0.
• onnx_npu_smoke: not run. /dev/accel/accel0 is held by the hung ryzenai-server from the measurement run (pid 272606, child of serve 272601), left alone per the no-SIGKILL-on-an-NPU-holder rule; the serve-path shutdown behaviour with the real backend is unchanged by this commit except for the logging and the knob.

🤖 Generated with Claude Code

[github-actions]

Persistent review updated to latest commit 47e6a4d

[github-actions]

Persistent review updated to latest commit d5d5736

[github-actions]

Persistent review updated to latest commit 598afb7