llamadart-native

Native build and release pipeline for llamadart binaries.

Purpose

This repository is responsible for:

• Building native llamadart binaries across platforms.
• Publishing release artifacts consumed by llamadart build hooks.
• Producing release metadata (assets.json + SHA256SUMS).

The Dart API/runtime stays in the main llamadart repository.

Workflow

• Native Build & Release (.github/workflows/native_release.yml)
  • Exact dispatch, manual or automatic for an upstream-aligned stable candidate, with an exact upstream ref/40-hex commit, exact native tag, smoke policy, and caller correlation identifier.
  • Builds one full backend set per platform/arch target.
  • Fails when any enabled backend in that target fails.
  • Publishes per-target native assets (Apple consolidated, others split core/backend libs).
  • Generates assets.json and SHA256SUMS.
  • Pins every build to one resolved llama.cpp commit and publishes a source tag whose submodule entry identifies that exact commit.
• Auto Stable Native Release (.github/workflows/auto_native_release.yml)
  • Manual detection run only; every upstream bump needs wrapper work first.
  • Resolves the latest stable upstream ggml-org/llama.cpp vMAJOR.MINOR.PATCH release tag and fails closed on any other shape.
  • Dispatches Native Build & Release only for an exact unbuilt stable upstream release, with the exact upstream ref/40-hex commit, the exact upstream-aligned native tag, smoke_policy=required, publish_release=true, and a deterministic auto-stable/<tag>/<digest> correlation identifier that is stable across retries.
  • Checks for queued or in-progress native release runs both while planning and immediately before dispatch, suppressing the dispatch when one is observable. It never publishes assets or mutates this repository itself.
  • Uploads native-discovery-report-<run-id> with a machine-readable candidate, noop, or incompatible status, the dispatch/skip/fail decision, and the exact upstream tag and commit before any dispatch.

Native Version Management

Stable upstream vMAJOR.MINOR.PATCH releases are the default distribution channel. Historical and new bNNNN nightlies remain available only through an explicit workflow input. The published tag is the native version contract consumed by downstream package hooks and Swift Package manifests. llama_cpp_tag selects the exact upstream llama.cpp release ref to build and llama_cpp_commit pins its expected full commit SHA; moving aliases such as latest and submodule are not dispatch inputs. native_release_tag selects the llamadart-native release tag and archive suffix and is always explicit.

When changing the upstream llama.cpp version:

1. Once main builds and qualifies the new stable upstream tag, run Auto Stable Native Release manually; it dispatches Native Build & Release for that tag. It has no schedule because every upstream bump so far has needed wrapper, patch or ISA-audit work before it could publish.
2. For a policy-compliant nightly bNNNN ref or wrapper-only rebuild tag, dispatch Native Build & Release manually with an independently resolved exact llama_cpp_tag/llama_cpp_commit, an exact native_release_tag, smoke_policy=required, and a stable correlation_id. Reuse or republication of an existing tag is rejected.
3. Verify the release contains per-platform native archives, llamadart-native-apple-xcframework-<tag>.zip, assets.json, and SHA256SUMS.

Native publication does not require dependency-pin PRs in companion repositories. Any downstream code, asset, or pin change remains an independent change with its own validation and approval boundary.

When rebuilding the wrapper without changing the upstream llama.cpp ref, dispatch Native Build & Release with the same llama_cpp_tag and a new native_release_tag. A rebuild of stable upstream v0.2.0 uses v0.2.0-1; a rebuild of nightly b9873 uses b9873-1. Historical b9873-llamadart.1 tags remain readable compatibility inputs but are never emitted for new releases. Do not republish different source under an existing tag; downstream caches are tag-keyed and the GitHub source tag should identify the native wrapper commit that produced the assets. Publish mode rejects tag collisions, stable rollbacks, and decreasing rebuild sequences.

Consumers determine stable versus nightly from the native tag grammar, not from GitHub's prerelease field alone. Newly emitted nightlies and wrapper rebuilds are prereleases, but immutable historical releases b10545 and b10356-llamadart.1 have prerelease=false and remain nightly inputs.

assets.json records native_release_tag, the requested llama_cpp_tag, its resolved llama_cpp_commit, and the native_commit targeted by the published source tag. It also records the caller correlation, smoke policy/conclusion, and exact workflow run/head. The historical tag field remains as a compatibility alias for the native tag. Consumers can therefore verify the built source independently of a moving ref or release label.

hook_contract_version is 2 from the release that split Windows x64 CUDA into its own asset. A consumer hook that supports only version 1 expects CUDA inside windows-x64 and must refuse the release.

Candidate builds and packaging are read-only. The final publication job alone can create the transaction-marked annotated tag and draft-first release; its assets are published only after exact digest and provenance checks. Retry a partial publication by rerunning the failed job in the same workflow run. Exact matching partial state—including a tag-only state with the same transaction marker—is resumed, while any unmarked tag or tag, release, correlation, or asset mismatch fails closed without force-moving tags or replacing assets.

To publish an already qualified nonpublishing preparation without rebuilding, use Publish Qualified Native Preparation (publish_prepared.yml) from main. Its explicit run/artifact digest and source inputs must match the original successful owner workflow, complete build/smoke gates, manifest, source bundle and same-run release result. Fresh complete remote history must also permit new publication or incomplete recovery; a fully published exact replay is a read-only no-op. publish_approved=false only validates; true publishes through the same immutable transaction code. Both native entrypoints share the non-cancelling native-release-main workflow lock, including preparations, so history cannot interleave with another native publication. The original build run/head/digest remain the release identity, with the publication executor recorded separately. See the version policy for the inputs and retry rules.

Every successful preparation or publication uploads native-release-result-<run-id>. Its JSON returns the correlation identifier, workflow run ID/attempt/URL/head SHA, native tag plus legacy tag, upstream ref/commit, native commit, publication artifact ID/URL/digest, manifest and checksum digests, exact checksum entries, required/actual bundle coverage, smoke conclusion, and exact GitHub Release and asset metadata when published. Publishing is rejected unless the required Linux x64 package-load smoke passes; smoke_policy=skip is explicit preparation-only behavior.

See Native Release Version Policy for ordering, prerelease classification, legacy compatibility, and the downstream contract.

Backend Policy (Worthy Sets)

Release CI builds backend lanes separately where needed and merges them into the following per-target bundles:

• Android: arm64 = Vulkan + OpenCL + CPU variants (Kleidi-enabled where safe); x86_64 = Vulkan + OpenCL + CPU
• iOS/macOS: Metal + CPU (consolidated into libllamadart, BLAS/Kleidi disabled)
• Linux x64: Vulkan + CUDA + BLAS + CPU (HIP/ROCm built in a separate release job)
• Linux arm64: Vulkan + BLAS + Kleidi + CPU variants (ARMv8.0 libggml-cpu.so plus runtime-selected DOTPROD+FP16 and I8MM modules, see docs/linux_arm64_cpu_variants.md)
• Windows x64: Vulkan + CUDA + BLAS + CPU
• Windows arm64: Vulkan + BLAS + Kleidi + CPU

Non-Apple targets use GGML_BACKEND_DL=ON, so backend libs are optional at package/runtime level.

Runtime Packaging Model

Release assets contain:

• Apple: consolidated libllamadart per target.
• Apple SPM: llamadart-native-apple-xcframework-<tag>.zip, a llamadart_native.xcframework built from the same Apple slices and wrapper code as the native-assets tarballs. Each framework slice embeds a PrivacyInfo.xcprivacy; the per-target Apple tarballs are bare dylibs and carry none. See docs/apple_privacy_manifest.md.
• Non-Apple core libs: llamadart, llama, llama-common, ggml, ggml-base (and mtmd where produced)
• Non-Apple backend libs: ggml-<backend> modules (ggml-vulkan, ggml-opencl, etc.)
• Windows backend runtime deps:
  • Windows x64 CUDA ships as its own asset, llamadart-native-windows-x64-cuda12-<tag>.tar.gz: ggml-cuda.dll plus the cudart64_12.dll, cublas64_12.dll and cublaslt64_12.dll only it imports. It is extracted over llamadart-native-windows-x64-<tag>.tar.gz from the same release, which loads without it. assets.json lists it with backend: "cuda" and module: "backend-cuda12".
  • Linux CUDA lanes include the CUDA runtime libraries required by ggml-cuda.
  • BLAS lanes include openblas*.dll required by ggml-blas.
  • NVIDIA driver DLLs (for example nvcuda.dll) are not bundled and are provided by GPU drivers.
  • The core lane ships the Visual C++ v14 runtime (msvcp140.dll, vcruntime140.dll, plus vcruntime140_1.dll on x64) and, on x64, the OpenMP runtime (vcomp140.dll) app-locally, so the bundle loads on Windows without the Visual C++ Redistributable. arm64 builds without OpenMP. See docs/windows_vc_runtime.md.
• Headers archive: llamadart-native-headers-<tag>.tar.gz with llama_cpp/... and libllamadart/... roots, including llama.cpp, ggml, mtmd, and llama_dart_wrapper.h.

Consumers can choose which backend libs to include in their package and load at runtime.

Release Asset Naming

Assets are suffixed with platform/arch, for example:

• libllamadart-linux-x64.so
• libllama-linux-x64.so
• libggml-vulkan-linux-x64.so
• libggml-opencl-android-arm64.so
• ggml-cuda-windows-x64.dll

Repository Layout

• .github/workflows/auto_native_release.yml: stable upstream candidate detector; dispatches the native release workflow for an exact unbuilt stable release and never publishes assets itself.
• .github/workflows/native_release.yml: build + package + release.
• .gitmodules: pinned native dependency submodules.
• CMakeLists.txt + CMakePresets.json: root-native build configuration.
• src/: llama_dart_wrapper.*.
• third_party/llama.cpp: upstream llama.cpp submodule.
• third_party/Vulkan-Headers: Vulkan API headers submodule for Android Vulkan builds.
• third_party/SPIRV-Headers: SPIR-V registry headers required by the llama.cpp Vulkan backend.
• third_party/OpenCL-Headers: OpenCL headers submodule (Android OpenCL builds).
• third_party/OpenCL-ICD-Loader: OpenCL loader submodule used to produce Android libOpenCL.so when NDK does not provide one.
• third_party/opencl-stubs: optional local fallback location for OpenCL headers/stubs.
• tools/build.py: cross-platform build entrypoint.
• tools/build_identity.py: stamps the string llama_dart_build_identity() returns on every build (docs/build_identity.md).
• tools/validate_exports.py: verifies required wrapper C exports, including exit-teardown, guarded mixed-batch processing, speculative-decoding and TTS symbols, in release artifacts.
• tools/validate_windows_bundle.py: checks a Windows bundle's import closure and its app-local Visual C++ runtime.
• tools/package_linux_artifact.py: preserves Linux ELF version files and SONAME symlinks while transporting split CI artifacts.
• tools/validate_linux_artifact.py: checks Linux archive members, symlinks, SONAMEs, and local DT_NEEDED dependencies, that no library built here imports __cxa_atexit, and that libllamadart.so cannot be unloaded.
• tools/package_apple_xcframework.py: packages Apple libllamadart slices as an SPM-compatible XCFramework zip.
• tools/apple_bundle_metadata.py: derives each framework Info.plist's CFBundleSupportedPlatforms and MinimumOSVersion (iOS) or LSMinimumSystemVersion (macOS) from the slice binary's LC_BUILD_VERSION.
• tools/apple/PrivacyInfo.xcprivacy: privacy manifest embedded in every XCFramework slice.
• tools/validate_apple_xcframework.py: checks each XCFramework slice's privacy manifest location, contents, and declared required-reason API categories against the slice binary's imports, and that its framework Info.plist declares the slice binary's platform and deployment minimum.
• scripts/generate_assets_manifest.sh: builds assets.json + checksums.
• scripts/auto_release_dispatch.py: decides whether a discovery run authorizes one exact native release dispatch and emits its immutable inputs.
• scripts/verify_release_provenance.py: verifies exact-source checkout, source tag, and manifest provenance contracts.
• .github/workflows/validate_linux_release_rows.yml: runs the release's Linux job bodies without publishing, weekly and on changes to their inputs.
• docs/platform_backend_strategy.md: platform/backend matrix.
• docs/release_version_policy.md: stable/nightly channels, wrapper rebuild ordering, provenance, and downstream coordination.
• docs/parity_tools.md: local build recipe and naming convention for upstream llama-cli parity checks against a released native tag.

Local Build (Preferred)

Builds are primarily driven by root CMakePresets.json via tools/build.py. Android arm64 CPU variants use isolated CMake build directories so per-variant ISA flags remain correct while packaging the full variant matrix. The raw android-arm64-v8a-full preset now represents the primary arm64 build, while tools/build.py assembles the additional CPU variant outputs.

Examples:

# macOS arm64 (Metal + CPU)
python3 tools/build.py apple --target macos-arm64

# Linux x64 (Vulkan + CUDA + BLAS + CPU)
python3 tools/build.py linux --arch x64

# Linux x64 CPU-only artifact validation build
python3 tools/build.py linux --arch x64 --backend cpu

# Linux x64 HIP/ROCm build
python3 tools/build.py linux --arch x64 --backend hip

# Android both ABIs (arm64: Vulkan + OpenCL + CPU variants; x86_64: Vulkan + OpenCL + CPU)
python3 tools/build.py android --abi all

# Windows x64 (Vulkan + CUDA + BLAS + CPU)
python3 tools/build.py windows --arch x64

# Windows arm64 (Vulkan + BLAS + Kleidi + CPU)
python3 tools/build.py windows --arch arm64

List supported combinations:

python3 tools/build.py list

Initialize submodules after clone:

git submodule update --init --recursive

This build produces the shipped runtime libraries only; upstream tools such as llama-cli are force-disabled here. See Parity Tool Builds for building them from a released tag's pinned llama.cpp commit in a separate clone.

The additive creation-time owner API prepares worker reclamation when a kill arrives during model, context or projector creation. It shares the exit registry and retains the existing ABI.

The generic speculative API replaces the fourteen MTP/n-gram family exports. The v0.8.0-3 wrapper rebuild also removes the constant-false llama_dart_speculative_need_embd query and retains llama_dart_speculative_need_embd_nextn for its MTP answer. Older consumers keep their pinned artifacts; incompatible library overrides require migration.

Generic MTP initialization requires empty target memory and checks rollback support before accepting the session. See the MTP rollback contract and local component checks.

Exit Teardown

ggml-metal aborts in its static destructor (GGML_ASSERT([rsets->data count] == 0)) when the process exits while a Metal buffer is still allocated. A Flutter macOS quit calls C exit without shutting the Dart isolates down, and a hot restart drops them without running their finalizers, so the caller cannot free its models in time.

libllamadart therefore tracks live objects in a registry and frees what is left of it at exit:

• llama_dart_model_load_from_file, llama_dart_init_from_model and llama_dart_mtmd_init_from_file create a tracked object. So do the TTS, speculative, MTP and n-gram init functions. Tracking happens inside the creating call. llama_dart_exit_track tracks any other object.
• llama_dart_exit_free frees a tracked object and removes it, so nothing is freed twice. It takes one pointer and can be a Dart NativeFinalizer callback.
• On Apple platforms teardown runs during exit, before the first static of libllamadart is destroyed. It frees the remaining objects in stage order: sessions, schedulers, contexts, model users such as mtmd contexts, backends, then models. It first waits up to 2 s (llama_dart_exit_set_wait_ms) for the calls in flight on other threads, and frees nothing if one is still running. A model load in flight is cancelled.
• llama_dart_decode, llama_dart_encode, llama_dart_process, the other llama_dart_ functions named after an upstream function, and the wrapper's own session functions are calls in flight. The decoding ones also wait for the backend, because Metal computes after llama_decode returns. Make every long native call on a tracked object through one of them: a Dart isolate can be killed inside any native call, and only a call that begins and ends in native code is then still accounted for.
• Once teardown has begun, a thread that reaches one of these functions outside a call in flight never returns from it: the objects it holds are gone. It stays blocked until the process is gone, so a static destructor of another library that joins such a thread hangs the exit.
• A native call on a tracked object that is not a call in flight is only safe while it is short: teardown allows 250 ms after a thread's last call in flight. A longer one is a use after free at exit.
• TTS, speculative and MTP state over a context that was created with the upstream function is tracked but not freed at exit, so a caller that has not switched to the tracked functions keeps its previous exit behavior.

On Linux, Android excepted, owned bundle statics are retained until the process is gone (src/llama_dart_static_destructors.c). C exit runs no wrapper teardown or forced termination: earlier host callbacks, dependency destructors and normal libc stream flushing proceed. -z nodelete keeps held native objects and finalizer callbacks valid across dlclose.

Before C exit, the host must stop and join workers and shut down its Dart isolates or Flutter engine. Await their exit and native finalizers; a kill request alone is insufficient. Direct C exit with live Dart isolates can abort the VM even without llama.cpp (#977). Owned static retention does not protect GPU-driver or BLAS destructors.

docs/v060_2_wrapper_rebuild.md describes the host protocol, preservation regression tests and remaining qualification.

On other platforms nothing runs at exit. llama_dart_exit_teardown runs teardown on request on every platform, frees included, and must be followed directly by exit. It is for native hosts: like llama_dart_exit_call_begin and llama_dart_exit_call_end, it is not meant to be bound from Dart. src/llama_dart_wrapper.h documents each function.

llamadart_exit_teardown_test covers the registry on every platform. On macOS and Linux it also writes a small decoder and a small encoder model and runs the scenarios that load one, so ctest needs no model file:

cmake -S . -B build/exit-teardown -G Ninja -DLLAMADART_BUILD_TESTS=ON
cmake --build build/exit-teardown --target llamadart_exit_teardown_test
ctest --test-dir build/exit-teardown -R exit_teardown --output-on-failure

-DLLAMADART_EXIT_TEARDOWN_TEST_MODEL=/path/to/model.gguf runs them against another model, and adding -DLLAMADART_EXIT_TEARDOWN_TEST_MMPROJ=/path/to/mmproj.gguf also runs the mtmd scenarios. Three scenarios need models of their own and run by hand:

build/exit-teardown/llamadart_exit_teardown_test model-mtp target.gguf mtp-draft.gguf
build/exit-teardown/llamadart_exit_teardown_test model-tts qwen3-tts.gguf mmproj-qwen3-tts.gguf
build/exit-teardown/llamadart_exit_teardown_test model-idle-untracked model.gguf

The last one exits with a model that is not tracked, and aborts on Metal.

Guarded mixed-batch DECODE and caller-owned extended-batch construction are documented in the mixed-batch native contract.

Exception Barrier

llama.cpp reports some failures by throwing a C++ exception, for example Unexpected empty grammar stack after accepting piece when a grammar sampler accepts a token that its grammar rejects. An exception that crosses the C ABI into a Dart FFI caller ends the process, so every llama_dart_ function that calls llama.cpp or allocates catches it, returns a failure value and keeps the message for the calling thread:

Return type	After a caught exception
pointer	NULL
bool	false
int32_t status or count	LLAMA_DART_STATUS_EXCEPTION (INT32_MIN)
llama_token (llama_dart_sampler_sample)	LLAMA_TOKEN_NULL
size_t	0
enum ggml_status	GGML_STATUS_FAILED
enum llama_dart_tts_status	LLAMA_DART_TTS_STATUS_UPSTREAM_ERROR, and the task fails
void	nothing; only the message is kept

llama_dart_last_error returns the message as valid UTF-8, or NULL when the thread's last such call caught nothing: each of them clears it on entry, which tells a caught exception from a failure that llama.cpp reports through the same value. The functions that free an object only set it, since a Dart finalizer may run one between a failed call and the read of its error. The upstream functions that can throw and that a caller would otherwise call directly have a wrapper too: llama_dart_sampler_accept, llama_dart_sampler_init_grammar_lazy_patterns, llama_dart_tokenize, llama_dart_token_to_piece, llama_dart_memory_clear, the three llama_dart_mtmd_bitmap_init_ functions and the llama_dart_ggml_backend_ functions that reach a backend. docs/v060_1_wrapper_rebuild.md classifies every upstream function that llamadart calls. A failed GGML_ASSERT still aborts. src/llama_dart_wrapper.h lists the functions without a barrier and what a caller may do with the objects of a call that failed.

llamadart_exit_teardown_test covers it with the barrier-free, barrier-free-idle, barrier-grammar and, on macOS, model-barrier and model-sample-accept scenarios. The grammar ones use a model with a vocabulary that the test writes. llamadart_barrier_test calls the helpers behind the barrier directly.

Cancel Byte

A caller cancels a request by setting a byte it owns to nonzero from another thread. Three exports let that byte stop native work in flight instead of waiting for it:

• llama_dart_cancel_flag_abort_callback, passed to llama_set_abort_callback with the byte as its data, stops a llama_decode or llama_encode after the node being computed; the call returns 2. Upstream installs it only on the CPU backend, so a graph on a GPU backend runs to its end. After a 2, a recurrent or hybrid model's sequence memory is undefined (upstream cannot roll recurrent state back): clear the sequence and prefill it again.
• llama_dart_mtmd_encode_chunk_cancellable is mtmd_encode_chunk with a cancel byte, for STT audio and vision encodes. A clip encode runs on clip's own backends, which upstream gives no abort callback, so the stop goes through mtmd_context_params.cb_eval: install llama_dart_mtmd_eval_callback there, which also does what llama_dart_tts_eval_callback does inside a TTS step. During the call it splits the encode after attention nodes and, once the byte is nonzero, ends the compute after the next node. The call then returns LLAMA_DART_MTMD_ENCODE_CANCELLED (2), and the context stays usable.

Chunking leaves CPU output unchanged. Metal fuses ops only within one command buffer, whose node ranges depend on the size of the graph it computes, so on Metal a chunked encode fuses other ops than an unchunked one and its output differs slightly.

llamadart_cancel_flag_test covers both. Its decode scenario uses the model that the exit teardown tests write; its mtmd-encode scenarios run when LLAMADART_EXIT_TEARDOWN_TEST_MODEL and LLAMADART_EXIT_TEARDOWN_TEST_MMPROJ name a real model and projector.

Vulkan Device Facts

llama_dart_vulkan_get_device_count and llama_dart_vulkan_get_device_info report, for each device that ggml-vulkan registers, the Vulkan API version of the loader and of the device, the subgroup size, the vendor and device ids, the device type, the driver version and the name. A caller needs them before it uses the Vulkan backend: ggml-vulkan requires Vulkan 1.2 but still registers a device whose driver stops at 1.1, and some of its shaders depend on the subgroup size.

They read the Vulkan loader directly (vulkan-1.dll, libvulkan.so.1, or libvulkan.so on Android), which libllamadart opens at run time and does not link. They create a Vulkan instance and no logical device, and do not load ggml-vulkan. Creating the instance loads the system's GPU drivers into the process, and a driver crash there cannot be caught, so call them only when the Vulkan backend is being considered. Device N is ggml's device VulkanN: the selection of ggml_vk_instance_init is mirrored, including GGML_VK_VISIBLE_DEVICES. Without a usable loader they return a negative llama_dart_vulkan_status, and on Apple platforms always LLAMA_DART_VULKAN_STATUS_UNSUPPORTED. src/llama_dart_wrapper.h documents the fields and the two cases where the order can differ from ggml's.

llamadart_vulkan_device_info_test runs the probe against a table-driven loader, and against the machine's own loader. To require a device there:

GGML_VK_VISIBLE_DEVICES=0 LLAMADART_TEST_VULKAN_DEVICES=1 \
  build/wrapper-contract/llamadart_vulkan_device_info_test

Experimental TTS Wrapper

libllamadart exposes a versioned, opaque C symbol contract around llama.cpp's experimental audio-generation helpers. The wrapper currently recognizes Qwen3-TTS projectors, exposes model capability metadata, and provides task start/step/cancel/reset plus caller-buffered float32 mono PCM reads after synthesis completes. The caller owns the llama_context and mtmd_context, must create the llama context with embeddings enabled, and must give the TTS task exclusive access to both contexts until the task reaches a terminal state or is reset.

The upstream API does not currently expose completed PCM incrementally, so the wrapper's step API is not a real-time audio stream. A step sees a cancel when it starts and when its frame generation or audio output returns. Qwen3-TTS decodes audio in 72-frame windows, inside the step that fills a window and the step that ends speech. Install llama_dart_tts_eval_callback as the mtmd context's cb_eval to stop that decode at its next chunk boundary, and attach a caller-owned cancel byte with llama_dart_tts_set_cancel_flag to cancel from a thread that must not touch the task. Public Dart support should remain experimental and capability-gated until artifact and platform validation is complete.

request.seed seeds all 16 codebooks: the talker sampler and the projector's code predictor. The code predictor's RNG belongs to the mtmd_context and upstream reseeds it only when the seed value changes, so llama_dart_tts_start runs one code-predictor call under a different seed first. Each task therefore starts the RNG from its own seed, and a repeated request and seed repeat the PCM. LLAMA_DEFAULT_SEED draws one random seed per task for both samplers.

The talker sampler gives the GGUF's tokenizer.ggml.suppress_tokens a -inf bias, as upstream common_sampler does. For Qwen3-TTS these are the codec's control tokens, which code2wav rejects as out-of-range codes, failing the synthesis. Earlier wrappers did not apply them. A fixed seed still gives the same PCM at default settings, top_k 1 and temp 0; at high temperature with top_k 0 (for example temp 2.5) PCM can differ even when no control token is drawn, because dist renormalizes without the suppressed mass.

The local smoke target is opt-in because it requires large model artifacts:

cmake -S . -B build/tts-smoke -G Ninja \
  -DGGML_CCACHE=OFF \
  -DGGML_METAL=OFF \
  -DLLAMADART_BUILD_TESTS=ON \
  -DLLAMADART_BUILD_TTS_SMOKE=ON
cmake --build build/tts-smoke --target \
  llamadart_speculative_api_test llamadart_tts_api_test \
  llamadart_tts_types_test llamadart_tts_eval_test llamadart_mtmd_compat_test \
  llamadart_tts_smoke llamadart_tts_lfm3_smoke llamadart_soprano_tts_smoke \
  llamadart_tts_voice_smoke
ctest --test-dir build/tts-smoke --output-on-failure
build/tts-smoke/llamadart_tts_smoke \
  /path/to/Qwen3-TTS-model.gguf \
  /path/to/mmproj-Qwen3-TTS.gguf \
  /tmp/tts-smoke.wav \
  "Hello from Llama Dart." en \
  --gpu

The smoke checks capability metadata, cancellation/reset, two consecutive syntheses, 24 kHz mono PCM metadata, finite/non-silent output, and WAV writing. The same seed must repeat the first synthesis's PCM after the cancel probes, and with a greedy talker, seeds 1 and 2 must give different PCM. It also checks the eval callback: uncancelled PCM stays byte-identical, frame steps stay whole, no chunk of the final decode exceeds a quarter of it, and a cancel issued a quarter of the way into a decode step, through llama_dart_tts_cancel or a cancel flag, makes that step return CANCELLED within a third of the decode's duration. That holds at the end-of-speech and frame-72 decodes even when the flag returns to zero right after the decode breaks, and no step returns OK after a break. Omit --gpu for a CPU-only run. An optional speaker-reference audio path may appear before the final --gpu flag.

LFM3-Audio

Builds whose llama.cpp has the LFM3-Audio generator (v0.8.0 and later) also drive LFM3-Audio and LFM3-Omni speech output. Callers opt in by model type: llama_dart_tts_get_info and llama_dart_tts_init stay Qwen3-only, and llama_dart_tts_get_info_ext / llama_dart_tts_init_ext take a mask that names LLAMA_DART_TTS_MODEL_TYPE_LFM3_AUDIO. llama_dart_tts_supported_model_types reports whether the build has it. The wrapper renders request.text through the model's tts chat template with request.language and the voice of llama_dart_tts_start_with_voice (female or male). Output is complete PCM; speaker references, sequence continuation, text/audio interleaving and streaming are not exposed. Scope and upstream gaps: docs/lfm3_support.md.

build/tts-smoke/llamadart_tts_lfm3_smoke \
  /path/to/LFM3-Audio-2B-Q4_K_M.gguf \
  /path/to/mmproj-LFM3-Audio-2B-Q8_0.gguf \
  /tmp/lfm3-smoke.wav --gpu

It checks that the Qwen3-only exports still refuse the model, the capability metadata, speaker-reference and voice rejection, cancel before and during generation, reuse after cancel, seed, voice and language effects, max_frames truncation, and that ASR through the generic mtmd input exports transcribes the speech back to its text.

Soprano

Builds whose llama.cpp has the Soprano generator (v0.8.0 and later) also drive Soprano, a Qwen3 backbone with a Vocos decoder projector, for callers whose _ext mask names LLAMA_DART_TTS_MODEL_TYPE_SOPRANO; the Qwen3-only exports and other masks report it as UNKNOWN. It has no capability bits: llama_dart_tts_start returns UNSUPPORTED for a speaker reference or a language, and llama_dart_tts_start_with_voice for a voice. Frame steps decode backbone tokens only; the step that ends generation decodes the whole utterance at 32 kHz, and the eval callback stops that decode at its next chunk boundary. A short utterance's decode is a single chunk. An utterance needs at least two frames. Upstream expects normalized English text and shows --temp 0; the wrapper's default temperature is 0.8, so set request.temperature for upstream-matching output.

No llama.cpp-format Soprano GGUF is published. Convert ekwek/Soprano-1.1-80M with upstream convert_hf_to_gguf.py, once for the model and once with --mmproj (upstream's tools/tts/README.md), and run the smoke from a Release build:

build/tts-smoke/llamadart_soprano_tts_smoke \
  /path/to/Soprano-1.1-80M.gguf \
  /path/to/mmproj-Soprano-1.1-80M.gguf \
  /tmp/soprano-smoke.wav "Hello from Llama Dart."

It checks that callers without the Soprano bit see UNKNOWN, the capability metadata, the speaker, language and voice rejections, cancel/reset, the one-frame failure, finite non-silent PCM, seeded repeats, a cancel issued inside the audio decode (it must stop at the decode's first chunk boundary) and max_frames truncation.

Chatterbox and Pocket TTS

Builds on llama.cpp v0.9.0 and later also drive Chatterbox (Turbo and Multilingual V3) and Pocket TTS for callers whose _ext mask names LLAMA_DART_TTS_MODEL_TYPE_CHATTERBOX or LLAMA_DART_TTS_MODEL_TYPE_POCKET_TTS. llama_dart_tts_get_voice_count and llama_dart_tts_get_voice_name list a projector's predefined voices, and llama_dart_tts_start_with_voice selects one by name; an unknown name is INVALID_ARGUMENT. Without a voice or a speaker reference the first voice speaks. Chatterbox Multilingual takes a language and a second sequence for classifier-free guidance. Rules, ABI delta and the qualified model files: docs/tts_voices.md.

build/tts-smoke/llamadart_tts_voice_smoke \
  /path/to/chatterbox-turbo-Q8_0.gguf \
  /path/to/mmproj-chatterbox-turbo-F16.gguf \
  /tmp/voice-smoke --speaker /path/to/reference.wav --gpu

Windows Build Notes

MSVC release builds keep interprocedural optimization enabled by default, but llama-common and mtmd are excluded from IPO/LTCG. Upstream llama-common is a large utility DLL, and current MSVC link.exe can access-violate while linking it with /LTCG. Upstream mtmd enables CMake's automatic Windows symbol export; CMake cannot generate the export definition from LTCG object files. These target-scoped overrides keep Windows release artifacts reproducible without changing the runtime packaging model.

To retest MSVC IPO after a compiler or upstream change, configure with:

cmake --preset windows-x64-full \
  -DLLAMADART_MSVC_LLAMA_COMMON_IPO=ON \
  -DLLAMADART_MSVC_MTMD_IPO=ON

Local Linux Build With Docker Cache

Use tools/docker_build_linux.sh to build Linux targets in a cached Docker image. The image is based on NVIDIA CUDA 12.8.1 and keeps heavy dependencies (CUDA, cross toolchains, Vulkan/BLAS dev packages) in reusable layers, so repeat builds are faster. This Docker flow is for local development only; CI Linux jobs run on native GitHub runners.

# Linux x64 full set
./tools/docker_build_linux.sh --arch x64 --jobs 8

# Linux arm64 full set (cross-compiled in container)
./tools/docker_build_linux.sh --arch arm64 --jobs 8

# Build both Linux targets
./tools/docker_build_linux.sh --arch all --jobs 8

Useful flags:

• --clean: clean preset build directories before build
• --rebuild-image: force image refresh
• --platform: override Docker platform (default linux/amd64)
• --image: custom image tag

Outputs are written to bin/linux/x64 and bin/linux/arm64. Note: Kleidi-enabled lanes require network access to fetch upstream Kleidi sources. Android arm64 CPU variants are built in isolated configurations so Kleidi can stay enabled without leaking newer ISA flags into lower-tier variant binaries.

Android OpenCL override env vars (optional):

• OPENCL_INCLUDE_DIR=/path/to/opencl/headers
• OPENCL_LIBRARY_ANDROID_ARM64_V8A=/path/to/arm64/libOpenCL.so
• OPENCL_LIBRARY_ANDROID_X86_64=/path/to/x86_64/libOpenCL.so

Maintainer Docs

• AGENTS.md: agent workflow and cross-repo handoff
• CONTRIBUTING.md: contributor setup/build/release steps