Fix flaky arm64 multi-arch build: register binfmt with current QEMU

[suyadav1]

Problem

The linux/arm64 leg of the multi-arch build fails ~50% of the time on ci_prod — 9 of the last 20 runs. Most recently build 123410, whose only code change was a one-line telegraf version bump.

The emulated gcc segfaults at random while compiling the ruby native gem extensions in kubernetes/linux/setup.sh. All three retries in that build died in a different place, which is what makes it clearly environmental rather than a code defect:

Attempt	Crash site
1	ossl_pkey_dh.o (openssl gem) — Segmentation fault (core dumped)
2	aarch64-unknown-linux-gnu-gcc: internal compiler error: Segmentation fault signal terminated program cc1
3	epoll.o (io-event gem) — Segmentation fault (core dumped)

Further evidence that this is not code-related:

• Commit 66d115cc2 both failed (123192) and succeeded (123209, 123210) with no changes in between.
• linux/amd64 compiles the exact same gem set in ~80s, every time.
• The failure always lands in emulated native compilation, never in ruby/gem logic.

Why it got worse recently. Two changes significantly increased the amount of C compiled under emulation:

• fluentd 1.19.x pulled in async-http → async → io-event, a native epoll extension.
• setup.sh deletes the bundled openssl default gem for CVE reasons, which forces a full openssl gem source build.

Root cause

The emulator was ancient.

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes pins the QEMU binary from that image into the kernel via the binfmt_misc F (fix-binary) flag. That image was last published 2023-01-17 and tops out at QEMU 7.2.

The apt-get install qemu binfmt-support qemu-user-static line above it doesn't help either — Ubuntu freezes upstream versions per release and only backports fixes, so jammy is pinned at QEMU 6.2 (1:6.2+dfsg-2ubuntu6.31, 31 Ubuntu revisions later and still 6.2). It was also redundant, since the --reset -p yes immediately overwrote whatever it registered.

Random cc1 segfaults under old qemu-user are a well-known problem, and the standard remedy is to register handlers from a current QEMU build.

Fix

.pipelines/azure_pipeline_mergedbranches.yaml

• Register binfmt handlers with tonistiigi/binfmt:qemu-v9.2.2-52 (QEMU 9.2.2) — the image docker/setup-qemu-action uses. Pulled from the MCR mirror, since this pipeline already documents working around Docker Hub rate limiting.
• Uninstall existing handlers first, so the new QEMU actually takes effect rather than losing to a stale F-flag registration.
• Drop the redundant apt QEMU install.
• Add an arm64 smoke test, so broken emulation fails the job in seconds instead of 30+ minutes into the build.
• Job timeout 120 → 180 min, so the added retries can't trade a segfault for a timeout.

kubernetes/linux/setup.sh

• Gem install retries 3 → 5 with backoff, as defense in depth. Retries are cheap because already-installed gems are skipped, so each attempt resumes where the previous one crashed (visible in 123410: 905s → 241s → 399s).
• Log the failing command in the retry warning, and switch the log/error messages from $@ to $*. Inside a quoted string $@ expands to one word per argument; echo re-joins them with spaces so the rendered output is unchanged, making this a readability fix rather than a bug fix. The invocation itself stays gem install "$@", which must keep argument boundaries intact.

scripts/build/linux/install-build-pre-requisites.sh

• Same QEMU change, so local dev builds match CI.

Validation

Queued as build 123605 on this branch. build_linux succeeded, including the ORAS push and ESRP signing steps that build 123410 never reached.

QEMU registration now reports current emulators:

Status: Downloaded newer image for mcr.microsoft.com/mirror/docker/tonistiigi/binfmt:qemu-v9.2.2-52
uninstalling: qemu-* not found
installing: arm64 OK
...
"supported": [ "linux/amd64", ..., "linux/arm64", ... ]
"emulators": [ "qemu-aarch64", ... ]

The arm64 smoke test passes (aarch64), and the previously fatal gem builds now complete on the first attempt:

#47 970.4  gem install attempt 1/5: gem install fluentd -v 1.19.3 --no-document
#47 2041.6 Successfully installed openssl-4.0.2
#47 2118.3 Successfully installed io-event-1.19.5
#47 2118.3 Successfully installed fluentd-1.19.3

Across the whole build log:

• 0 occurrences of Segmentation fault / internal compiler error / core dumped (123410 had 3)
• 0 gem retry warnings — every gem install succeeded on attempt 1
• Multi-arch manifest pushed: cidev:3.6.0-3-gd08216c60-20260814170628

Also verified before pushing: YAML parses, the rendered inline script and both shell scripts pass bash -n, the retry wrapper was unit-tested (succeeds on retry, exits 1 after exhaustion), and the binfmt flags were checked against the upstream README.

set -e semantics were audited in both scripts, since they differ: the ADO inline script has no set -e (proven by 123410's log continuing past the failed build), so the new critical steps use explicit || exit 1; the dev prereq script does (line 6), so the no-op --uninstall uses || true while --install is left to abort on genuine failure.

Note: Docker windows build for ltsc2019 hit an unrelated transient failure in 123605 and passed on the pipeline's automatic retry. This PR contains no Windows changes.

Follow-up (not in this PR)

The QEMU tag is pinned deliberately, for reproducibility. It's worth revisiting periodically, and longer term a native arm64 build agent would remove the emulation risk entirely rather than just making it much less likely.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.