[bong-water-water-bong]

The engine ships no ONNX Runtime of its own any more (owner, 2026-10-23). Lemonade carries ONNX wherever it exists: its ryzenai-llm backend on Windows and its onnxruntime backend on the CPU. The engine's NPU route is its own engine (--device npu), and the hybrid mode for Linux is being built on it (Gantt phyb1-3).

The Linux ONNX route (#386) had done its job: the data it was built to collect is recorded, and nothing in Lemonade's onebit backend references the engine's onnx/ directory.

Removed

• --device onnx in serve; the flag now fails with a removed-device message that names Lemonade's backends, like vulkan, zinc and mlx. A model directory with genai_config.json is no longer recognised.
• ONEBIT_ONNX, the onnx_npu_smoke test, scripts/build-onnx.sh, the SIGTERM shim, the third_party/ryzenai-server submodule, docs/onnx.md.
• The onnx/ directory of the Linux package, its weekly package checks, the ryzenai-server pin in the weekly changes, and the ryzenai-server / ONNX Runtime entries in NOTICE.

Untouched: Lemonade's own onnxruntime:cpu backend in the 1bit OS image, and the ONNX-export vocabulary of the model census.

Checked on ryzen: configure + build of onebit, smoke_serve, watchdog_serve and metrics_serve pass; 1bit serve --device onnx prints the removed-device message; --help mentions no ONNX.

🤖 Generated with Claude Code

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 386 - Partially compliant Compliant requirements: Linux NPU route for 1bit serve --device onnx through AMD's Ryzen AI Software (RyzenAI execution provider) ONNX pieces of the release package AMD's Ryzen AI Software is not shipped and not copied into the repository ONNX goes in the engine release Changes to app/serve.cpp for onnx device handling Child lifecycle improvements in serve.cpp Release package changes in scripts/build-onnx.sh and scripts/package-linux.sh SIGTERM shim implementation in scripts/onnx/signal_shim.c Documentation updates in docs/onnx.md, docs/serve.md, docs/npu.md Tests including tests/onnx_npu_smoke.sh Non-compliant requirements: None Requires further human verification: Device check (NPU testing) was not run as per PR description

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Removed Device Handling The code removes the onnx device handling from the serve command, including the logic for detecting ONNX model directories and setting up the environment for the RyzenAI provider. This change is correct for the PR's goal of removing the ONNX route, but the removal of the is_onnx_model_dir function and related logic should be confirmed to not affect other parts of the system that might rely on it. bool onebp_file(const std::string& model) { std::ifstream f(model, std::ios::binary); char magic[4] = {}; return f.read(magic, 4) && std::memcmp(magic, "1BP\0", 4) == 0; } // DwarfStar's ds4-server (third_party/ds4, MIT), built by scripts/build-ds4.sh std::string default_ds4() { if (const char* e = std::getenv("ONEBIT_DS4"); e && *e) return e; #ifdef ONEBIT_DS4_SERVER return built_path(ONEBIT_DS4_SERVER); #else return "ds4-server"; #endif } // SIGTERM / SIGINT: stop serving and take the backend down with us. std::atomic<bool> g_stop{false}; extern "C" void on_stop_signal(int) { g_stop = true; } int free_port() { #ifdef _WIN32 // Winsock is started by httplib.h (its WSInit) const SOCKET s = ::socket(AF_INET, SOCK_STREAM, 0); const bool bad_socket = s == INVALID_SOCKET; auto close_socket = [&] { ::closesocket(s); }; int len = sizeof(sockaddr_in); #else const int s = ::socket(AF_INET, SOCK_STREAM, 0); const bool bad_socket = s < 0; auto close_socket = [&] { ::close(s); }; socklen_t len = sizeof(sockaddr_in); #endif sockaddr_in a{}; a.sin_family = AF_INET; a.sin_addr.s_addr = htonl(INADDR_LOOPBACK); a.sin_port = 0; if (bad_socket || ::bind(s, reinterpret_cast<sockaddr*>(&a), sizeof(a)) != 0 || ::getsockname(s, reinterpret_cast<sockaddr*>(&a), &len) != 0) { if (!bad_socket) close_socket(); throw std::runtime_error("cannot find a free loopback port"); } const int port = ntohs(a.sin_port); close_socket(); return port; } #if defined(__linux__) // Who holds a loopback port: the backend serve started (or a process it started), or someone // else. free_port() releases its port before the child binds it, so another local process could // bind it first and answer the readiness probe; serve must not send it client requests. enum class PortOwner { kOurs, kOther, kUnknown }; PortOwner port_owner(pid_t child, int port) { // the LISTEN sockets on the port (any address), by inode std::vector<std::string> inodes; bool readable = false; for (const char* table : {"/proc/net/tcp", "/proc/net/tcp6"}) { std::ifstream f(table); if (!f) continue; readable = true; std::string line; std::getline(f, line); // header while (std::getline(f, line)) { // sl local_address rem_address st tx:rx tr:when retrnsmt uid timeout inode std::istringstream in(line); std::string sl, local, remote, st, txrx, trwhen, retr, uid, timeout, inode; if (!(in >> sl >> local >> remote >> st >> txrx >> trwhen >> retr >> uid >> timeout >> inode)) continue; const size_t colon = local.rfind(':'); if (st != "0A" || colon == std::string::npos) continue; // 0A: LISTEN if (std::strtol(local.c_str() + colon + 1, nullptr, 16) == port) inodes.push_back("socket:[" + inode + "]"); } } if (!readable || inodes.empty()) return PortOwner::kUnknown; // the child and its descendants (a backend may be a wrapper that starts the real server) std::unordered_map<pid_t, pid_t> parent; std::error_code ec; for (const auto& ent : fs::directory_iterator("/proc", ec)) { const std::string name = ent.path().filename().string(); if (name.empty() || name.find_first_not_of("0123456789") != std::string::npos) continue; std::ifstream st(ent.path() / "stat"); std::string stat; std::getline(st, stat); const size_t close = stat.rfind(')'); // the command name may hold spaces and parentheses if (close == std::string::npos) continue; std::istringstream rest(stat.substr(close + 1)); std::string state; long ppid = 0; if (rest >> state >> ppid) parent[pid_t(std::stol(name))] = pid_t(ppid); } for (const auto& [pid, _] : parent) { bool ours = false; for (pid_t p = pid, hops = 0; p > 1 && hops < 64; p = parent.count(p) ? parent[p] : 0, ++hops) if (p == child) { ours = true; break; } if (!ours) continue; const fs::path fds = fs::path("/proc") / std::to_string(pid) / "fd"; for (const auto& fd : fs::directory_iterator(fds, ec)) { std::error_code lec; const std::string target = fs::read_symlink(fd.path(), lec).string(); if (!lec && std::find(inodes.begin(), inodes.end(), target) != inodes.end()) return PortOwner::kOurs; } } return PortOwner::kOther; } NPU Retry Logic Removed The NPU retry logic for ONNX models has been removed. This logic was specifically designed to handle the one-off firmware timeout that occurs on the first NPU dispatch after idle. The removal of this logic means that if such a timeout occurs, the request will not be retried, potentially leading to a 5xx error for the client. } #endif { std::lock_guard<std::mutex> lock(route_mu); held = std::make_shared<InFlight>(&inflight[pick]); } ++routed[pick]; forward(backends[pick].port, held, -1, id, drop_model, set_model, o.precise, q, r, &watch); }; srv.Post("/v1/chat/completions", post); srv.Post("/v1/completions", post); // The rest of llama-server's API, for the devices that run llama-server (Lemonade's // llamacpp backend, which the onebit recipe inherits, forwards these to us). They go to // the first backend; the NPU and DwarfStar answer 501. const bool llama = device == "hrx" || device == "cpu" || (laya && (backends[0].device == "hrx" || backends[0].device == "cpu")); ONNX Package Removal The package-linux.sh script no longer includes the onnx/ directory in the release package. This is consistent with the PR's goal of removing the ONNX route, but it's important to ensure that all references to this directory are properly removed from the packaging process. put() { local rel=$1; shift [ -d "$build/$rel" ] || return 0 mkdir -p "$stage/$rel" local f for f in "$@"; do [ -f "$build/$rel/$f" ] && cp -a "$build/$rel/$f" "$stage/$rel/"; done find "$build/$rel" -maxdepth 1 \( -name '*.so' -o -name '*.so.*' \) -exec cp -a {} "$stage/$rel/" \; echo " $rel: $(ls "$stage/$rel" | wc -l) files" } cp -a "$build/1bit" "$stage/" put hrx/llama/bin llama-server llama-bench put ds4/hip ds4-server ds4 ds4-bench done fi Weekly Build Configuration The weekly.sh script no longer builds with ONNX support and removes the RYZENAI_INSTALL_PATH variable. This is correct for the PR's goal, but it's important to ensure that all build configurations and checks are updated to reflect this change. JOBS=${WEEKLY_JOBS:-4} export CMAKE_BUILD_PARALLEL_LEVEL=$JOBS # the submodules the packages are built from (laya only on its bump) # (not xdna-driver: it pins the amdxdna kernel driver, which the release does not build) SUBMODULES=(hrx-system llama.cpp lemonade ds4 tokenizers) mkdir -p "$LOGS" # pack_missing_libs (package_checks); this script's own copy, so it works before build checks out $SRC # shellcheck source=scripts/pack-elf.sh . "$(cd "$(dirname "$0")" && pwd)/pack-elf.sh" say() { echo "[$(date +%H:%M:%S)] $*"; } trap 'say "FAILED at line $LINENO: $BASH_COMMAND (logs: $LOGS)"' ERR # Heavy steps wait for room, and stop before the box runs out of memory (unified memory: GPU # allocations count too); one the guard stopped is retried, and the builds pick up where they were. MIN_GB=${WEEKLY_MIN_FREE_GB:-16} room() { local i for i in $(seq 360); do [ "$(awk '/MemAvailable/ {print int($2 / 1048576)}' /proc/meminfo)" -ge "$MIN_GB" ] && return 0 [ "$i" = 1 ] && say "waiting for ${MIN_GB} GB of free memory" sleep 60 done say "no ${MIN_GB} GB free in 6 hours"; return 1 } guard() { local try local rc for try in $(seq 10); do room || return 1 rc=0; "$SRC/scripts/mem-guard.sh" 4 "$@" || rc=$? [ "$rc" = 0 ] && return 0 # mem-guard kills with SIGKILL (exit 137); any other status is the command's own failure, # which a retry cannot fix if [ "$rc" != 137 ]; then say "FAILED (exit $rc): $*"; return "$rc"; fi say "memory guard stopped it (try $try): $*" done return 1 } # --- the tree ----------------------------------------------------------------------------- checkout() { # checkout <ref> [pr number to merge on top] [ -d "$SRC/.git" ] || git clone -q "https://github.com/$REPO.git" "$SRC" git -C "$SRC" fetch -q origin "$1" git -C "$SRC" checkout -q -f --detach FETCH_HEAD # the workspace is ours: a stray edit never blocks a run if [ -n "${2:-}" ]; then git -C "$SRC" fetch -q origin "pull/$2/head" git -C "$SRC" -c user.name=weekly -c user.email=weekly@1bit.gg merge -q --no-edit FETCH_HEAD fi local mods=("${SUBMODULES[@]/#/third_party/}") m for m in laya; do # fetched only when that pin is what moves if [ -n "${2:-}" ] && ! git -C "$SRC" diff --quiet HEAD^1 HEAD -- "third_party/$m"; then mods+=("third_party/$m") fi done git -C "$SRC" submodule sync -q git -C "$SRC" submodule update -q --init --recursive --depth 1 -- "${mods[@]}" 2>/dev/null || git -C "$SRC" submodule update -q --init --recursive -- "${mods[@]}" } pin() { git -C "$SRC" ls-tree HEAD "third_party/$1" | awk '{print $3}'; } # the machine a held bump failed on, for the release notes: its name and its AMD GPU(s), from # sysfs (scripts/gpu-targets.sh), e.g. "strixhalo (gfx1151)" or "ryzen (gfx1036, gfx1201)" host_desc() { local g; g=$("$(cd "$(dirname "$0")" && pwd)/gpu-targets.sh" 2>/dev/null | paste -sd, - | sed 's/,/, /g') echo "$(hostname -s) (${g:-no AMD GPU})" } # --- build / test ------------------------------------------------------------------------- build() { # the private components compiled in (docs/releases.md, "Private components"): the NPU runtime # (into 1bit) and the HIP kernels (into the HRX llama.cpp), each a checkout at the commit its # config/*.config pins; the build stops here when one cannot be fetched at that commit local npurt npurt=$("$SRC/scripts/fetch-private.sh" "$SRC/config/npu-runtime.config" "$W/npu-runtime" 2> "$LOGS/private.log") || { cat "$LOGS/private.log"; return 1; } "$SRC/scripts/fetch-private.sh" "$SRC/config/gpu-kernels.config" "$W/gpu-kernels" > /dev/null 2>> "$LOGS/private.log" || { cat "$LOGS/private.log"; return 1; } say "engine $(git -C "$SRC" rev-parse --short HEAD), NPU runtime $(git -C "$W/npu-runtime" rev-parse --short HEAD), HIP kernels $(git -C "$W/gpu-kernels" rev-parse --short HEAD)" local extra=() [ -d "$HOME/models/laya-pinned" ] && extra+=(-DONEBIT_LAYA_ROOT="$HOME/models/laya-pinned") # shellcheck disable=SC2206 extra+=(${WEEKLY_CMAKE_ARGS:-}) # The NPU kernels (docs/releases.md, "The NPU engine"): sealed, packed beside 1bit in # npu/kernels. On by default (the npu gate phase passed on Strix Halo; WEEKLY_NPU_KERNELS=0 # builds a package without them). The private checkout does the work through its one entry # point (key install, sealing, the per-commit kernel cache); this script names nothing in it. # ONEBIT_NPU_RELEASE makes the built 1bit load sealed kernels only. if [ "${WEEKLY_NPU_KERNELS:-1}" = 1 ]; then "$SRC/scripts/fetch-private.sh" "$SRC/config/npu-kernels.config" "$W/npu-kernels" > /dev/null 2>> "$LOGS/private.log" || { cat "$LOGS/private.log"; return 1; } guard "$W/npu-kernels/addons/release-kernels.sh" "$BUILD/npu/kernels" "${ONEBIT_NPU_KEY_DIR:-$HOME/.config/1bit-npu-key}" \ > "$LOGS/npu-kernels.log" 2>&1 || { tail -n 5 "$LOGS/npu-kernels.log"; return 1; } extra+=(-DONEBIT_NPU_PRIVATE="$W/npu-kernels" -DONEBIT_NPU_RELEASE=ON) say "NPU kernels $(git -C "$W/npu-kernels" rev-parse --short HEAD): $(ls "$BUILD/npu/kernels" | wc -l) sealed sets" fi cmake -S "$SRC" -B "$BUILD" -G Ninja -DCMAKE_BUILD_TYPE=Release \ -DONEBIT_HRX=ON -DONEBIT_GPU_PRIVATE="$W/gpu-kernels" \ -DONEBIT_DS4=ON -DONEBIT_HF_TOKENIZERS=ON \ -DONEBIT_NPU=ON -DONEBIT_NPU_RUNTIME=own -DONEBIT_NPU_RT_DIR="$npurt" \ -DONEBIT_SERVE_TEST_GGUF="$GGUF" "${extra[@]}" > "$LOGS/configure.log" 2>&1 Documentation Updates The documentation in docs/serve.md has been updated to remove references to the ONNX device. This is consistent with the PR's goal, but it's important to ensure that all documentation is properly updated to reflect the removal of the ONNX route. ```sh 1bit serve -m <model> [--port 8000] [--host 127.0.0.1] [--device auto|npu|hrx|cpu|ds4] [--ctx-size N] [--alias NAME] [--llama-server PATH] [--ds4 PATH] [--ssd-streaming] [--hrx-libhsa PATH] [--hrx-cwsr-placement=auto|on|off] [--mtp HEAD.gguf | --dflash DRAFT.gguf | --no-mtp] [--mtp-max N] [--mtp-p-min P] [--mmproj MMPROJ.gguf] [--laya | --laya-model DIR] [--route-policy FILE] [--pm [--pm-experts FILE] [--lemonade-url URL]] [--moe-slots N|auto] [--moe-subst R] [--moe-prefetch N] # refused in this build, below [--parallel N] [--embed MODEL.gguf] [--rerank MODEL.gguf] [--npu-opt KEY=VALUE ...] [--no-mem-check | --mem-check] [--watchdog SECS] [--watchdog-prompt-tps N] [--strict-device | --no-strict-device] [--placement-report] [--print-backend | --print-backend-env] One model per process: Endpoint GET /health, GET /v1/health 503 while the model loads, then 200 while every backend is up and answers its own health route; 503 again if one has exited, does not answer within 1.5 s, or the watchdog found it hung (below) GET /v1/models the one model (--alias, else the file or directory name) POST /v1/chat/completions streamed (SSE) or not; a thinking model's <think> block comes back as reasoning_content ("reasoning_format": "none" keeps it in content) POST /v1/completions POST /v1/embeddings with --embed (RAG) POST /v1/rerank with --rerank (RAG) POST /v1/embeddings or /v1/rerank with --embedding or --reranking: the model itself is an embedding or reranking model POST /v1/responses OpenAI Responses API, streamed or not (llama-server devices) POST /tokenize, /detokenize, /apply-template llama-server devices GET /slots, POST /slots/<id>?action=…, GET /props llama-server devices; slot save and restore answer 501 (llama-server runs without --slot-save-path) GET /metrics Prometheus text: the engine's own series (GTT, Tctl, CPU fallbacks, program-cache flushes, starts, watchdog kills) on every child-process device, then the backend's own (below) "llama-server devices" means hrx and cpu. These routes go to the llama-server behind the model, which is how Lemonade's llamacpp backend reaches them (the onebit recipe inherits it). On npu and ds4 they answer 501. 1bit serve has no authentication (SECURITY.md), so it refuses two kinds of request a web page could make with 403: on a loopback --host, any whose Host header is not a loopback name (localhost, 127.x.x.x, [::1]), and on any host, a POST whose Origin is neither a loopback origin nor the server's own. API clients send no Origin. A reverse proxy in front of a loopback serve must pass the upstream host (nginx's default proxy_set_header Host $proxy_host), not the public one. Will it fit? Before any llama-server starts (hrx, cpu, and the --embed / --rerank servers), serve estimates what the model will hold and refuses a load the box has no room for, in one line and in milliseconds, so Lemonade shows the reason instead of an allocation failure or an OOM minutes into the load. ggml-hrx reports free memory equal to total, so llama.cpp's own --fit cannot do this on HRX0. $ 1bit serve -m Qwen3.8-27B-UD-Q4_K_XL.gguf --device hrx --mem-check # strixhalo, 2026-10-12 Qwen3.8-27B-UD-Q4_K_XL: fits: need 33.00 GiB (Qwen3.8-27B-UD-Q4_K_XL.gguf: weights 16.35 + KV/state 16.15 + compute 0.50) at the model's own ctx 262144 (no --ctx-size), free 57.71 GiB (GTT 73.3 of 120.0 GiB free, MemAvailable 57.7 GiB) A load that does not fit prints the same line as 1bit serve: <model> does not fit: need …; free memory, lower --ctx-size, or pass --no-mem-check and exits 3 before any backend starts. Lemonade's loader logs that line with the exit code and fails the load ("1bit serve failed to start"). The estimate (app/mem_admit.h) reads only the GGUF header, the way tools/mem-budget's mem_budget.py model does: weights: the GGUF's size, every shard; KV: per attention layer, KV heads x (key + value length) x context x the cache type's bytes (-ctk / -ctv, f16 by default). MLA models (kv_lora_rank) cache the latent and the rope dimensions once. Qwen3.5 / 3.8 / Next attend in one layer of full_attention_interval. Per-layer head_count_kv arrays count only the layers with heads. Sliding-window layers count the window per sequence plus a ubatch, with llama.cpp's per-architecture pattern for which layers are windowed (gpt-oss and Gemma 2: every other layer; Gemma 3: 5 of 6). state: the gated delta-net recurrent state, per sequence; compute: 512 MiB (HRX measured 20-505 MiB); extras: a draft or MTP head (-md, with its KV), --mmproj. The context is --ctx-size (or a recipe's -c). Without one, llama-server takes the model's own on HRX0 (HRX reports its whole pool free, so --fit shrinks nothing), and serve judges that. On the CPU --fit shrinks an unset context, so serve judges llama.cpp's 4096 floor. Free is min(GTT total - GTT used, MemAvailable) when a backend runs on HRX0, since the GTT pool is the same DRAM, and MemAvailable on the CPU. A load is refused only when the need plus 1 GiB is more than that. Two things grow after load, so they only warn: the HRX graph-program cache (2-9 GiB as prompt shapes vary); landing at or above 95 GiB of GTT, where engine HRX: GLM-4.7-Flash intermittent HSA memory fault / NaN logits under varied-length requests (base fork tip too) #315's faults were seen. --no-mem-check (or ONEBIT_MEM_CHECK=0) skips the check. --mem-check prints the estimate and exits: 0 when it fits, 3 when it does not. A refused load exits with status 3 too. ctest mem_admit checks the KV and state sizes against HRX0 server logs: model ctx KV/state Qwen3-0.6B 4096 448 MiB GLM-4.7-Flash (MLA) 2048 105.75 MiB ZAYA1-8B 2048 80 MiB Qwen3.6-35B-A3B 2048 40 + 62.8 MiB ctest mem_admit_route checks that a sparse 4 TiB GGUF is refused in milliseconds, before any backend starts. On strixhalo (2026-10-12), estimates for 15 model and context pairs agree with mem_budget.py model (KV from llama-fit-params) within 1.1%. The pairs cover Qwen3 0.6B/8B, GLM-4.7-Flash, ZAYA1-8B, Qwen3.6-35B-A3B, Qwen3.8-27B, Qwen3-Coder-30B-A3B, gpt-oss-20b and MiniCPM4-8B, at 2K-262K context. When a backend hangs A hung GPU dispatch (a wedged HRX queue, a #315 fault) stops llama-server's main loop, while its HTTP thread keeps answering /health with 200. So 1bit serve checks two things itself, for the backends it runs as child processes (hrx, cpu, ds4; the in-process NPU route is not covered): </details> <details><summary><a href='https://github.com/1bit-MONSTER/engine/pull/529/files#diff-1e7de1ae2d059d21e1dd75d5812d5a34b0222cef273b7c3a2af62eb747f9d20aR227-R242'><strong>CMake Configuration</strong></a> The CMakeLists.txt file no longer includes the ONNX build configuration and related tests. This is correct for the PR's goal, but it's important to ensure that all build configurations are properly updated. </summary> ```text target_compile_definitions(onebit PRIVATE ONEBIT_DS4_SERVER="${_onebit_ds4_server}") endif() # ── Hugging Face tokenizers (docs/tokenizers.md) ─────────────────────────── # Off by default: it needs cargo. Any model's tokenizer.json, through our C ABI # over third_party/tokenizers (the same release as the tokenizers npm/PyPI # packages). tests/data/tokenizers/ holds golden cases for 18 models. option(ONEBIT_HF_TOKENIZERS "Build Hugging Face tokenizers (Rust) behind a C ABI" OFF) if(ONEBIT_HF_TOKENIZERS) find_package(Threads REQUIRED) include(cmake/hf_tokenizers.cmake) add_executable(hf_tokenizer_test tests/hf_tokenizer_test.cpp) target_link_libraries(hf_tokenizer_test PRIVATE onebit_hf_tokenizers nlohmann_json::nlohmann_json) # Golden cases whose tokenizer.json is under a model root (ONEBIT_TOKENIZER_ROOT, # e.g. ~/.config/flm/models: <root>/<model>/tokenizer.json). set(ONEBIT_TOKENIZER_ROOT "" CACHE PATH "Directory of model dirs holding tokenizer.json, for hf_tokenizer tests")