Skip to content

docs: kernel readiness runbook, with the 23 Oct rehearsal (v7.3 + next-20261022) - #525

Merged
bong-water-water-bong merged 2 commits into
mainfrom
1bit/kernel-runbook
Oct 23, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
1bit/kernel-runbook

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Adds docs/kernel.md, the runbook the Gantt row "Kernel 7.4 readiness" is about: every source is pinned, so a release is a bump, not a scramble. Also records the 2026-10-23 rehearsal of that runbook against the newest tags that exist today.

What the page covers

build -> sign -> install -> grub entry -> reboot (owner) -> mokutil -> hw-gate -> weekly package + 1bit OS image, with the facts that go stale marked as facts and the commands that do not:

  • what runs today (v7.3-rc4 + the two IOMMU PerfOpt backports, why: prefill +21–25 %, decode unchanged, greedy output byte-identical), and the out-of-tree modules that must follow every kernel (amdxdna, the fan EC);
  • the two MOKs and which one signs what, the postinst.d hook, and the traps: a module-signing-only MOK is rejected for kernel images, an unsigned vmlinuz-*.unsigned in /boot becomes a boot entry, a firmware reset drops the MOKs;
  • GRUB_DEFAULT is left alone — the previous kernel stays the fallback and the owner chooses when the box moves;
  • the verify sequence after boot, then the hardware gate as the thing that makes a boot more than "it booted";
  • the weekly release preflight (VMLINUZ=) and os/mini/mkimage.sh (MOK_KEY/MOK_CERT, the signed UKI), which is where a new kernel lands in the packages.

The rehearsal (2026-10-23)

No v7.4* tag exists (v7.3 final, 2026-10-18; no v7.3.y; the 7.4 merge window is open — torvalds master is Merge tag 'net-next-7.4'). So it ran twice, on a separate 16-core machine, with the config of record (/boot/config-7.3.0-rc4-1bit-perfopt), LLVM=1 clang 21.1.8, nice -n 19 make -j12 bindeb-pkg:

tree PerfOpt olddefconfig delta bindeb-pkg release
v7.3 backports re-applied (343 insertions, 8 files, no rebase needed) 28 lines (clang + debug info only) pass, ~15 min 7.3.0-1bit-perfopt+
next-20261022 (7.4 merge-window content) already native, backports dropped 110 lines (new clang-21/7.4 symbols; drops ANDROID_BINDER*, ARCH_HAS_DEBUG_WX, …) pass, ~15 min 7.3.0-next-20261022-1bit-perfopt

The out-of-tree amdxdna builds against both trees (MAKE_RC=0, amdxdna.ko), so the most likely 7.4 break is not there yet — worth re-running the moment v7.4-rc1 is tagged, since next is a moving target.

Traps found and written down: libdw-dev is the packaging dependency that stops bindeb-pkg in dpkg-checkbuilddeps; LLVM=1 must be exported or the packaging stage re-enters make with the wrong compiler; the driver's own Makefile exports an empty ARCH, which makes scripts/Makefile.clang abort with add '--target=' option (pass ARCH=x86_64); a tree with commits past its tag gets + in the release string and in module vermagic.

Verification

  • python3 tools/copyright.py --check → rc 0 (the new page carries the Apache notice).
  • docs/kernel.md is linked from README.md and from the release run section of docs/releases.md.
  • Every number in the rehearsal section comes from the two builds above; the box was only read (kernel, config, MOKs, GRUB_DEFAULT, module signers), never installed on or rebooted.

Not done here (deliberately): install, sign, boot and gate on the box — that is the owner's reboot and step 2 to 8 of the runbook; the fan-EC module (its source lives only on the box); throughput re-measurement on the new kernel.

🤖 Generated with Claude Code

bong-water-water-bong and others added 2 commits October 23, 2026 07:13
The box runs a kernel we build (v7.3-rc4 plus the IOMMU PerfOpt backports, clang, debug info off,
two MOKs), and moving it to the next upstream tag was an investigation each time. This adds
`docs/kernel.md`: the step list from tag to a box that runs it - build on another machine, sign with
the two MOKs, install, leave GRUB_DEFAULT alone, the owner's reboot, the verify commands, the hardware
gate, and the weekly package / 1bit OS image that must pick the new kernel up - plus the traps that
have cost time (module-signing-only MOK rejected for images, an unsigned copy in /boot becoming a boot
entry, a firmware reset dropping the MOKs).

It also records the 2026-10-23 rehearsal:

- no `v7.4*` tag exists yet (v7.3 final, 2026-10-18; the 7.4 merge window is open, no `v7.3.y`), so
  the rehearsal ran against `v7.3` (backports re-applied, clean) *and* linux-next `next-20261022`
  (the 7.4 merge-window content, where the PerfOpt series is already native, so the backports are
  dropped rather than rebased);
- both built with the config of record at clang 21.1.8, `bindeb-pkg` passing in ~15 min each;
- the out-of-tree `amdxdna` builds against both trees - the most likely 7.4 break is not there yet;
- `olddefconfig` deltas (28 lines on v7.3, 110 on next) and the traps: `libdw-dev`, `export LLVM=1`,
  the driver's empty `ARCH` breaking `scripts/Makefile.clang`, and `+` in the release string for a tree
  past its tag.

Linked from README (the box's kernel is ours too) and from the release run, which already needs a
readable kernel image for 1bit OS.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@bong-water-water-bong
bong-water-water-bong merged commit 51e5ff1 into main Oct 23, 2026
8 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/kernel-runbook branch October 23, 2026 10:19
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Missing Hardware Context Details

The document describes the kernel readiness process but lacks specific details about NPU hardware-context and runlist lifetimes, which are critical for ensuring correctness when moving to a new kernel. These aspects are essential for the 1bit engine's operation on AMD Strix Halo's XDNA 2 NPU.

The Strix Halo box runs a kernel we build, not the distribution's: everything the engine and the NPU
need is pinned by hand, so the next kernel is a step list, not an investigation. This page is that
list, with the facts that go stale marked as facts and the commands that do not.
Incomplete MOK Handling

While the document mentions two MOKs and their respective purposes, it does not fully address potential issues with MOK enrollment or how to handle cases where MOKs may become invalid or need re-enrollment, especially after firmware resets or Secure Boot changes.

| key | path | signs |
|---|---|---|
| `strixhalo Secure Boot Kernel` (EKU codeSigning) | `/var/lib/shim-signed/mok/MOK-Kernel.{priv,pem,der}` | the kernel image |
| `strixhalo Secure Boot MOK` (module-signing OID) | `/var/lib/shim-signed/mok/MOK.{priv,pem,der}` | every `.ko` we build |

```sh
# kernel: automatic on install (this is what the hook does), or by hand
sudo sbsign --key /var/lib/shim-signed/mok/MOK-Kernel.priv \
            --cert /var/lib/shim-signed/mok/MOK-Kernel.pem \
            --output /boot/vmlinuz-<version> /boot/vmlinuz-<version>
# modules we built ourselves (in-tree modules carry the build-time key already)
sudo kmodsign sha512 /var/lib/shim-signed/mok/MOK.priv \
                       /var/lib/shim-signed/mok/MOK.der updates/amdxdna.ko
  • /etc/kernel/postinst.d/z0-mok-sign signs any kernel image the installer did not sign, and does
    nothing when sbverify --list already shows a signature.
  • Trap: a MOK whose EKU is module-signing only (Ubuntu's default style) is rejected by shim for
    kernel images
    . That is why there are two keys; signing a vmlinuz with MOK.priv produces a
    kernel that will not boot under Secure Boot.
  • Trap: an unsigned copy left in /boot as vmlinuz-*.unsigned becomes a GRUB menu entry.
    Keep originals outside /boot (/var/backups/kernels/).
  • A BIOS/firmware reset can drop the enrolled MOKs. The fix needs the owner at the console
    (MokManager), and can be primed over ssh beforehand:
    mokutil --import /var/lib/shim-signed/mok/MOK-Kernel.der, then the same for MOK.der, then the
    owner confirms both in MokManager on the next boot.

</details>

<details><summary><a href='https://github.com/1bit-MONSTER/engine/pull/525/files#diff-494189164462e19f5ede07003ce7b6850d55d7b4020fd4072f1fd9391c5ca989R112-R114'><strong>Unclear Module Signing Process</strong></a>

The document describes module signing but does not specify how to verify that the signing process was successful for all required modules, particularly the out-of-tree modules like `amdxdna` and `ec_su_axb35`, which must be rebuilt and signed against the new kernel.
</summary>

```markdown
sudo kmodsign sha512 /var/lib/shim-signed/mok/MOK.priv \
                       /var/lib/shim-signed/mok/MOK.der updates/amdxdna.ko

</details>

</td></tr>
</table>

bong-water-water-bong pushed a commit that referenced this pull request Oct 23, 2026
…follow-up)

Data for the owner's release today, no prose for its own sake:

- `docs/releases.md` gains a **v2026.43 (candidate, 23 Oct)** section: the release sha and every pin,
  the four artifacts and the six checks as tables (marked *pending* until the release box's
  `RELEASE-CANDIDATE.md` exists), and a grouped changelog of everything merged since the 18 Oct
  rehearsal - HRX speed (verify kernels +31-39 % on 27B MTP, split-K decode 23.7 -> 47.1 on
  Qwen3.5-4B 32K, Flash-Next MoE 5.1 -> 26 tok/s, prompt FA +45 % on Qwen3-4B, warm-up TTFT
  2.64 -> 0.80 s), the batched N=8 cells (50.7 / 138.1 / 194.3 aggregate tok/s), correctness
  (strict placement by default, 43/43 census, #315 resolved on a 37-arm soak, #433 closed on 120
  loads), the sealed NPU packaging, the Lemonade one-hop + catalog proofs (22 HRX entries), the OS
  image's signed UKI, and the tooling (gpu-hold queue, census, kernel runbook).
- `README.md` gets the same status as a table under the opening paragraph, with each number's
  standard (lemonade bench standard or ours) and the power state where it matters.

Every line names its PR; every number comes from the auto-opt database rows, the catalog proofs or a
closed issue, and carries the standard it was measured under. Nothing is claimed as shipped: the
candidate's artifacts and checks stay *pending* until the release run writes them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bong-water-water-bong added a commit that referenced this pull request Oct 23, 2026
* docs: the v2026.43 candidate section and the README status table (#525 follow-up)

Data for the owner's release today, no prose for its own sake:

- `docs/releases.md` gains a **v2026.43 (candidate, 23 Oct)** section: the release sha and every pin,
  the four artifacts and the six checks as tables (marked *pending* until the release box's
  `RELEASE-CANDIDATE.md` exists), and a grouped changelog of everything merged since the 18 Oct
  rehearsal - HRX speed (verify kernels +31-39 % on 27B MTP, split-K decode 23.7 -> 47.1 on
  Qwen3.5-4B 32K, Flash-Next MoE 5.1 -> 26 tok/s, prompt FA +45 % on Qwen3-4B, warm-up TTFT
  2.64 -> 0.80 s), the batched N=8 cells (50.7 / 138.1 / 194.3 aggregate tok/s), correctness
  (strict placement by default, 43/43 census, #315 resolved on a 37-arm soak, #433 closed on 120
  loads), the sealed NPU packaging, the Lemonade one-hop + catalog proofs (22 HRX entries), the OS
  image's signed UKI, and the tooling (gpu-hold queue, census, kernel runbook).
- `README.md` gets the same status as a table under the opening paragraph, with each number's
  standard (lemonade bench standard or ours) and the power state where it matters.

Every line names its PR; every number comes from the auto-opt database rows, the catalog proofs or a
closed issue, and carries the standard it was measured under. Nothing is claimed as shipped: the
candidate's artifacts and checks stay *pending* until the release run writes them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs: v2026.43 pins after #526, and no internal file references

Two review fixes on the candidate section:

- `gpu-kernels` pin `abaa6bc484d5` -> `744bd713d968` (#526, the top-1 batched window from 5
  tokens), with the batched cells noted as measured one pin earlier, and the engine row now names
  `7871c71d` — main when the last release-blocking PR merged. Adds the `npu-runtime` pin
  (`ce9fa59adf95`) so every row the release run records has a row here.
- the candidate paragraph pointed at a file that only exists on the release box; public docs name
  files in this repository only, so it says "the release run" and what that run records.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant