Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/build-audiocpp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ on:
pull_request:
paths:
- surfsense_local/electron/scripts/audiocpp/**
- surfsense_local/electron/scripts/not-staged/**
- surfsense_local/electron/scripts/msvc-runtime.mjs
- surfsense_local/electron/scripts/pinned-download.mjs
- surfsense_local/backend/modules/llm/providers/audiocpp/**
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/build-sdcpp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ on:
pull_request:
paths:
- surfsense_local/electron/scripts/sdcpp/**
- surfsense_local/electron/scripts/not-staged/**
- .github/workflows/build-sdcpp.yml

permissions:
Expand Down
10 changes: 5 additions & 5 deletions docs/architecture/packaging.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,19 +64,19 @@ The release workflow runs the three scripts directly. Without the parser pack, D
- `scripts/sdcpp/stage.mjs`, which `build:sdcpp` runs, stages stable-diffusion.cpp's `sd-server` into `electron/sdcpp/` with the libraries it loads from beside itself and its licences, leaving `sd-cli` behind. It runs `sd-server --help` from the staged folder before it swaps the folder into place.
- Windows downloads upstream's Vulkan archive, checked against its pinned SHA-256, and copies the MSVC and OpenMP runtimes beside it, which the archive needs and does not carry.
- Linux and macOS compile the pinned commit, because upstream's Linux archives need glibc 2.38 and its macOS archive needs macOS 26.0. The recipe, `scripts/sdcpp/recipe.mjs`, builds Vulkan on Linux, with one ggml library per micro-architecture, and Metal on macOS, for 13.3.
- Compiling needs CMake, a C++ compiler and the Vulkan SDK (`glslc` and headers) on Linux, or Xcode's command line tools on macOS; Windows needs Visual Studio 2022 or newer with the C++ tools, for the runtime it ships. Without them the script stages an empty folder and says why, and the app runs without local images. Release CI passes `--strict`, which fails instead.
- Compiling needs CMake, a C++ compiler and the Vulkan SDK (`glslc` and headers) on Linux, or Xcode's command line tools on macOS; Windows needs Visual Studio 2022 or newer with the C++ tools, for the runtime it ships. Without them the script stages an empty folder and says why, and the app runs without local images. Release CI and `pnpm dist` pass `--strict`, which fails instead, naming everything missing and the command that installs it with the machine's package manager.
- `scripts/audiocpp/stage.mjs`, which `build:audiocpp` runs, stages audio.cpp's `audiocpp_server` into `electron/audiocpp/` with the model specs of the three curated families. It adds eSpeak-ng 1.52.0 from the pinned `espeakng-loader` wheel, which Kokoro and Kitten phonemise through (Kokoro finds it through the server's environment, Kitten through `server.json`, [`local-models/catalog.md`](local-models/catalog.md)), and eSpeak-ng's GPL licence text, which the wheel does not carry. It runs `--list-devices` from the staged folder before it swaps the folder into place.
- macOS downloads upstream's archive, checked against its pinned SHA-256.
- Windows and Linux compile the pinned commit, because upstream's Linux archives need glibc 2.38 and its Windows archive compiles AVX-512 into the executable. The recipe, `scripts/audiocpp/recipe.mjs`, builds only the CPU backend, since the server runs with `--backend cpu`, with one ggml library per micro-architecture and only the curated model families.
- Compiling needs CMake and GCC 13 or newer on Linux, or Visual Studio 2022 or newer with the C++ tools on Windows. Without them the script stages an empty folder and says why, and the app runs without local audio. Release CI passes `--strict`, which fails instead.
- Compiling needs CMake and GCC 13 or newer on Linux, or Visual Studio 2022 or newer with the C++ tools on Windows. Without them the script stages an empty folder and says why, and the app runs without local audio. Release CI and `pnpm dist` pass `--strict`, which fails instead, naming everything missing and the command that installs it with the machine's package manager.

`pnpm dist` in `electron/` runs every staging step, every native runtime included, before `electron-builder`.
`pnpm dist` in `electron/` runs every staging step before `electron-builder`, the native runtimes first, so a missing toolchain stops it before the frontend and the Python binaries are built. It stages sd.cpp and audio.cpp with `--strict`, so an installer never ships without them; `predev` does not, so dev runs without a toolchain.

`frontend/` and `electron/` each pin their pnpm in `packageManager` (`pnpm@11.27.1`), which pnpm switches to locally and the desktop workflows read through `package_json_file`. The repository root's `packageManager` is the web app's, `pnpm@10.26.0`, and does not apply here.

## Building audio.cpp

[`build-audiocpp.yml`](../../.github/workflows/build-audiocpp.yml) compiles the Windows and Linux builds with `stage.mjs --strict`, checks them, and hands each staged folder on as an artifact. `release-local.yml` calls it, and its packaging jobs unpack the artifact before `stage.mjs`, which then finds the server already staged. A pull request that changes `scripts/audiocpp/`, the audio.cpp adapter or engine, the manifest or the voicing test runs it too. It caches the staged folder by the scripts' contents, so a release that does not change them reuses the last checked build, and voices with it again.
[`build-audiocpp.yml`](../../.github/workflows/build-audiocpp.yml) compiles the Windows and Linux builds with `stage.mjs --strict`, checks them, and hands each staged folder on as an artifact. `release-local.yml` calls it, and its packaging jobs unpack the artifact before `stage.mjs`, which then finds the server already staged. A pull request that changes `scripts/audiocpp/`, `scripts/not-staged/`, the audio.cpp adapter or engine, the manifest or the voicing test runs it too. It caches the staged folder by the scripts' contents, so a release that does not change them reuses the last checked build, and voices with it again.

Why it compiles rather than downloads, in plain words: upstream's ready-made programs for Linux and Windows do not run on many of the computers the app supports, for a different reason on each.

Expand All @@ -97,7 +97,7 @@ It is a stopgap. Once upstream publishes archives that meet both floors, the app

## Building sd.cpp

[`build-sdcpp.yml`](../../.github/workflows/build-sdcpp.yml) compiles the Linux and macOS builds with `stage.mjs --strict`, checks each against the app's floors, and hands each staged folder on as an artifact, as `build-audiocpp.yml` does. A pull request that changes `scripts/sdcpp/` runs it too, so a compile that a pin bump or a new runner image breaks fails in review rather than in a release. It caches the staged folder by the scripts' contents. Windows is a pinned download, so the release job stages it itself, as it does llama.cpp.
[`build-sdcpp.yml`](../../.github/workflows/build-sdcpp.yml) compiles the Linux and macOS builds with `stage.mjs --strict`, checks each against the app's floors, and hands each staged folder on as an artifact, as `build-audiocpp.yml` does. A pull request that changes `scripts/sdcpp/` or `scripts/not-staged/` runs it too, so a compile that a pin bump or a new runner image breaks fails in review rather than in a release. It caches the staged folder by the scripts' contents. Windows is a pinned download, so the release job stages it itself, as it does llama.cpp.

Why it compiles on Linux and macOS, in plain words: upstream's ready-made programs start only on the newest systems there, Ubuntu 24.04 (glibc 2.38) and macOS 26.0, while the app runs on Ubuntu 22.04, RHEL 9 and macOS 13.3, the floor its llama.cpp and audio.cpp builds already set. Upstream's Windows program runs on any x64 processor, since its AVX-512 code sits only in the per-processor ggml libraries ggml picks at start, so Windows takes it as built.

Expand Down
2 changes: 1 addition & 1 deletion surfsense_local/electron/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
"build:llamacpp": "node scripts/fetch-llamacpp.mjs",
"build:sdcpp": "node scripts/sdcpp/stage.mjs",
"build:audiocpp": "node scripts/audiocpp/stage.mjs",
"dist": "pnpm build:frontend && pnpm build:binaries && pnpm build:model && pnpm build:voice && pnpm build:parser && pnpm build:llamacpp && pnpm build:sdcpp && pnpm build:audiocpp && pnpm build && electron-builder",
"dist": "pnpm build:llamacpp && pnpm build:sdcpp --strict && pnpm build:audiocpp --strict && pnpm build:frontend && pnpm build:binaries && pnpm build:model && pnpm build:voice && pnpm build:parser && pnpm build && electron-builder",
"test": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON --test src/main/*.test.ts src/main/**/*.test.ts scripts/**/*.test.mjs",
"typecheck": "tsc --noEmit -p tsconfig.json",
"check:sidecars": "node scripts/check-sidecars.mjs"
Expand Down
16 changes: 10 additions & 6 deletions surfsense_local/electron/scripts/audiocpp/compile.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import { availableParallelism } from "node:os"
import { join } from "node:path"

import { visualStudio } from "../msvc-runtime.mjs"
import { CMAKE, GCC_13, VISUAL_STUDIO } from "../not-staged/tools.mjs"
import { COMMIT, SOURCE, TAG } from "./pins.mjs"
import { configureArgs } from "./recipe.mjs"

Expand All @@ -21,17 +22,20 @@ function output(cmd, args) {
}
}

/** What this machine lacks to compile it, or null. audio.cpp requires GCC 13. */
/** Everything this machine lacks to compile it, empty if nothing. audio.cpp requires GCC 13. */
export function missingToolchain() {
if (output("cmake", ["--version"]) == null) return "CMake"
const missing = []
if (output("cmake", ["--version"]) == null) missing.push(CMAKE)
if (process.platform === "win32") {
return visualStudio() ? null : "Visual Studio 2022 or newer with the C++ tools"
if (!visualStudio()) missing.push(VISUAL_STUDIO)
return missing
}
const cxx = process.env.CXX ?? "g++"
const version = output(cxx, ["-dumpversion"])
if (version == null) return "GCC 13 or newer"
if (Number(version.split(".")[0]) < 13) return `GCC 13 or newer (${cxx} is ${version})`
return null
if (version == null) missing.push(GCC_13)
// No package fixes an old one: the distribution's g++ is already installed.
else if (Number(version.split(".")[0]) < 13) missing.push({ name: `${GCC_13.name} (${cxx} is ${version})` })
return missing
}

/** Build the server in `work`; returns where its files and its source are. */
Expand Down
23 changes: 15 additions & 8 deletions surfsense_local/electron/scripts/audiocpp/stage.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
// electron/audiocpp, swapped in whole once the server lists its devices.
// macOS downloads upstream's archive; Windows and Linux compile the pinned
// source. Without a toolchain the app runs without local audio, unless
// --strict, which release CI passes.
// --strict, which release CI and `pnpm dist` pass.
import { execFileSync } from "node:child_process"
import { existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, renameSync, rmSync } from "node:fs"
import { tmpdir } from "node:os"
Expand All @@ -12,6 +12,8 @@ import { fileURLToPath } from "node:url"
import { compile, missingToolchain } from "./compile.mjs"
import { stageEspeak } from "./espeak.mjs"
import { copyMsvcRuntime } from "../msvc-runtime.mjs"
import { NotStaged, notStaged } from "../not-staged/message.mjs"
import { packageManager } from "../not-staged/package-manager.mjs"
import { TAG } from "./pins.mjs"
import { copyServerFiles, SERVER } from "./server-files.mjs"
import { unpackUpstream } from "./upstream-archive.mjs"
Expand Down Expand Up @@ -48,16 +50,21 @@ async function main() {
}

const host = `${process.platform}-${process.arch}`
const missing = !HOSTS.includes(host)
? `a supported host, not ${host}`
const reason = !HOSTS.includes(host)
? `audio.cpp is not staged: it has no build for ${host}.`
: process.platform === "darwin"
? null
: missingToolchain()
if (missing) {
if (strict) throw new Error(`staging audio.cpp needs ${missing}`)
: notStaged(
"audio.cpp",
`on ${process.platform === "win32" ? "Windows" : "Linux"} it is compiled from source`,
missingToolchain(),
packageManager()
)
if (reason) {
if (strict) throw new NotStaged(reason)
// Empty, so the packager still finds the folder; the sidecar never starts.
mkdirSync(OUT, { recursive: true })
console.warn(`audio.cpp not staged: it needs ${missing}. Local audio is unavailable.`)
console.warn(`${reason}\n\nThe app runs without local audio.`)
return
}

Expand Down Expand Up @@ -92,6 +99,6 @@ async function main() {
}

main().catch((error) => {
console.error(error)
console.error(error instanceof NotStaged ? error.message : error)
process.exitCode = 1
})
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
// The commands that install what a stage lacks, with this machine's package
// manager. A tool with no package there is named in the message, not here.

const INSTALL = {
apt: (packages) => [`sudo apt install ${packages.join(" ")}`],
dnf: (packages) => [`sudo dnf install ${packages.join(" ")}`],
pacman: (packages) => [`sudo pacman -S --needed ${packages.join(" ")}`],
brew: (packages) => [`brew install ${packages.join(" ")}`],
// winget takes one package per call.
winget: (packages) => packages.map((id) => `winget install ${id}`),
}

export function installCommands(missing, manager) {
const commands = missing.flatMap((tool) => tool.command ?? [])
const packages = [...new Set(missing.flatMap((tool) => tool.packages?.[manager] ?? []))]
if (packages.length) commands.push(...INSTALL[manager](packages))
return commands
}
20 changes: 20 additions & 0 deletions surfsense_local/electron/scripts/not-staged/message.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Why a native runtime was not staged on this machine. It is setup to fix,
// not a fault in the script, so it is printed alone, without a stack trace.

import { installCommands } from "./install-commands.mjs"

/** Thrown under --strict; a stage script prints its message and exits 1. */
export class NotStaged extends Error {}

/** Every missing tool and the command that installs them all; null if none is. */
export function notStaged(runtime, how, missing, manager) {
if (missing.length === 0) return null
const names = missing.map((tool) => tool.name)
const lacks =
names.length > 1 ? `${names.slice(0, -1).join(", ")} and ${names.at(-1)}` : names[0]
const found = `${runtime} is not staged: ${how}, and this machine lacks ${lacks}.`
const commands = installCommands(missing, manager)
return commands.length
? `${found} Install what is missing, then run the build again:\n\n${commands.map((command) => ` ${command}`).join("\n")}`
: `${found} Install what is missing and run the build again.`
}
69 changes: 69 additions & 0 deletions surfsense_local/electron/scripts/not-staged/message.test.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
import assert from "node:assert/strict"
import test from "node:test"

import { notStaged } from "./message.mjs"
import { CMAKE, CXX_COMPILER, GCC_13, VISUAL_STUDIO, VULKAN_SDK, XCODE_TOOLS } from "./tools.mjs"

const HOW = "on Linux it is compiled from source"

test("says nothing when nothing is missing", () => {
assert.equal(notStaged("audio.cpp", HOW, [], "apt"), null)
})

test("names the one tool missing and what to do", () => {
assert.equal(
notStaged("audio.cpp", HOW, [CMAKE], null),
"audio.cpp is not staged: on Linux it is compiled from source, and this machine lacks CMake. Install what is missing and run the build again."
)
})

test("names every missing tool at once, so one install fixes the build", () => {
assert.equal(
notStaged("audio.cpp", HOW, [CMAKE, GCC_13], null),
"audio.cpp is not staged: on Linux it is compiled from source, and this machine lacks CMake and GCC 13 or newer. Install what is missing and run the build again."
)
assert.equal(
notStaged("sd-server", HOW, [CMAKE, CXX_COMPILER, VULKAN_SDK], null),
"sd-server is not staged: on Linux it is compiled from source, and this machine lacks CMake, a C++ compiler and the Vulkan SDK (glslc and headers). Install what is missing and run the build again."
)
})

test("shows the one command that installs everything missing", () => {
assert.equal(
notStaged("audio.cpp", HOW, [CMAKE, GCC_13], "apt"),
`audio.cpp is not staged: on Linux it is compiled from source, and this machine lacks CMake and GCC 13 or newer. Install what is missing, then run the build again:

sudo apt install cmake g++`
)
})

test("on Windows, one winget call per package", () => {
assert.equal(
notStaged("audio.cpp", "on Windows it is compiled from source", [CMAKE, VISUAL_STUDIO], "winget"),
`audio.cpp is not staged: on Windows it is compiled from source, and this machine lacks CMake and Visual Studio 2022 or newer with the C++ tools. Install what is missing, then run the build again:

winget install Kitware.CMake
winget install Microsoft.VisualStudio.2022.BuildTools --override "--wait --passive --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended"`
)
})

test("on macOS, Xcode's tools come from xcode-select, not Homebrew", () => {
assert.equal(
notStaged("sd-server", "on macOS it is compiled from source", [CMAKE, XCODE_TOOLS], "brew"),
`sd-server is not staged: on macOS it is compiled from source, and this machine lacks CMake and Xcode's command line tools. Install what is missing, then run the build again:

xcode-select --install
brew install cmake`
)
})

test("a tool no package fixes is named but left out of the command", () => {
// The distribution's g++ is already installed, and too old.
const oldGcc = { name: "GCC 13 or newer (g++ is 11.4.0)" }
assert.equal(
notStaged("audio.cpp", HOW, [CMAKE, oldGcc], "apt"),
`audio.cpp is not staged: on Linux it is compiled from source, and this machine lacks CMake and GCC 13 or newer (g++ is 11.4.0). Install what is missing, then run the build again:

sudo apt install cmake`
)
})
22 changes: 22 additions & 0 deletions surfsense_local/electron/scripts/not-staged/package-manager.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
// The package manager that installs build tools here: the platform's, or on
// Linux the distribution family's, read from os-release. null where unknown.
import { readFileSync } from "node:fs"

const LINUX = [
["apt", ["debian", "ubuntu"]],
["dnf", ["fedora", "rhel"]],
["pacman", ["arch"]],
]

export function packageManager() {
if (process.platform === "darwin") return "brew"
if (process.platform === "win32") return "winget"
let release
try {
release = readFileSync("/etc/os-release", "utf8")
} catch {
return null
}
const ids = [...release.matchAll(/^ID(?:_LIKE)?="?([^"\n]*)"?$/gm)].flatMap((m) => m[1].split(" "))
return LINUX.find(([, family]) => family.some((id) => ids.includes(id)))?.[0] ?? null
}
48 changes: 48 additions & 0 deletions surfsense_local/electron/scripts/not-staged/tools.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
// What a stage can lack, and the packages that provide it per package manager.
// Vulkan's are llama.cpp's (docs/build.md): its loader does not always pull in
// SPIR-V's headers. GCC 13 is each distribution's default g++ only from
// Ubuntu 24.04 and Fedora 39.

export const CMAKE = {
name: "CMake",
packages: {
apt: ["cmake"],
dnf: ["cmake"],
pacman: ["cmake"],
brew: ["cmake"],
winget: ["Kitware.CMake"],
},
}

export const GCC_13 = {
name: "GCC 13 or newer",
packages: { apt: ["g++"], dnf: ["gcc-c++"], pacman: ["gcc"] },
}

export const CXX_COMPILER = {
name: "a C++ compiler",
packages: { apt: ["g++"], dnf: ["gcc-c++"], pacman: ["gcc"] },
}

export const VULKAN_SDK = {
name: "the Vulkan SDK (glslc and headers)",
packages: {
apt: ["glslc", "libvulkan-dev", "spirv-headers"],
dnf: ["glslc", "vulkan-loader-devel", "spirv-headers-devel"],
pacman: ["shaderc", "vulkan-icd-loader", "vulkan-headers", "spirv-headers"],
},
}

export const XCODE_TOOLS = {
name: "Xcode's command line tools",
command: "xcode-select --install",
}

export const VISUAL_STUDIO = {
name: "Visual Studio 2022 or newer with the C++ tools",
packages: {
winget: [
'Microsoft.VisualStudio.2022.BuildTools --override "--wait --passive --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended"',
],
},
}
Loading
Loading