Skip to content

perf(bridge): attach and configure a TAP in one netlink message - #179

Merged
CMGS merged 3 commits into
masterfrom
perf/bridge-add-one-setlink
Jul 30, 2026
Merged

perf(bridge): attach and configure a TAP in one netlink message#179
CMGS merged 3 commits into
masterfrom
perf/bridge-add-one-setlink

Conversation

@CMGS

@CMGS CMGS commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Bridge.Add issued eight netlink calls per NICLinkAdd, LinkByName, master, learning, MTU, txqlen, GRO, up. Seven of them take the kernel's node-wide rtnl lock, so on a dense warm-pool fill the lock, not the work, sets the ceiling.

This collapses the sequence to two ops: CreateTAP returns the index, and one RTM_SETLINK carries IFLA_MASTER + IFLA_MTU + IFLA_TXQLEN + IFLA_GRO_MAX_SIZE + IFF_UP together — the ip link set X up master br idiom that iproute2 has always used.

Why learning=false goes with it

Nothing in this repo or in sandbox ever installed a static FDB entry, so disabling learning left the bridge FDB permanently empty and every frame — unicast included — flooded to all ports. That bought no isolation and cost N-fold datapath amplification: on a bridge carrying a thousand taps, ordinary DHCP traffic was enough to drive the host into a broadcast storm.

Clones already get a fresh MAC from hotSwapNets before resume, so learning cannot flap a shared address between ports. Verified after the change: a non-permanent FDB entry appears per guest, and guest-bound unicast stops flooding.

One caveat: an FC clone cannot swap its guest MAC before resume, so an FC bridge clone shares the source's MAC until the operator applies the CLI's repair hints. With learning on that shared MAC flaps the FDB between ports; with learning off the clone simply received all of the source's traffic via flooding — the pair was never a functioning config either way (both answer ARP for the same MAC-keyed DHCP lease), so this PR changes how that window fails, not whether it works.

Measurements

A 384-core host, 512 MiB guests, one bridge, warm-pool fill driven by sandboxd.

what before after
netlink ops per NIC 8 2
bridge attach, isolated, W=64 32.9/s 244.7/s
end-to-end fill, 256 VMs, quiet console 26.2/s 28.0/s
end-to-end fill, 256 VMs, RC 192 38.2/s 40.6/s

The end-to-end win is ~7%, not the 7x the isolated op suggests: the fill is dominated by other rtnl consumers this PR cannot touch — TUNSETIFF device registration, the VMM's own tap open (another TUNSETIFF per queue fd), linkwatch carrier batches, IPv6 addrconf per port-up. Fitting X(W) = W/(P + W·S) across six concurrency points puts the serialized component at ~16.8 ms/VM for the bridge lane against ~5.4 ms/VM for the no-NIC lane; this PR removes a few ms of it. The change is worth landing for the op-count reduction and the flooding fix, not as a fix for the lane's throughput gap.

One caveat worth knowing before you land it

On a host that still logs to a console at default loglevel, this change is a regression, not a win: collapsing the ops drives more bridge-port state transitions per attach (2.2 → 4.0 entered … state lines), and each such line is written synchronously to the console while rtnl is held. On a host with two registered consoles that measured ~14 ms per line, which is more than the netlink calls this saves — a 256-VM fill went 23.3 s → 30.1 s. With the console quiet the same fill is 9.8 s → 9.1 s.

So the bridge lane wants console_loglevel < 6 in its deployment contract. That is a host setting, not something this PR can carry.

Behaviour and safety

  • do_setlink applies MTU before enslaving, so a jumbo bridge no longer dips the port's MTU on the way in — strictly better than the old enslave-then-set order.
  • Up-before-enslave is safe: a fresh TAP has no carrier until the VMM opens it, so the port cannot enter forwarding early.
  • A mid-message failure (e.g. EXFULL at a bridge's 1024-port limit) leaves an up-but-unattached TAP; Add's deferred tearDownTAPs is state-insensitive and already cleans that.
  • netlink resolves a tuntap index with a GETLINK after TUNSETIFF and drops the error, so CreateTAP could hand back index 0. Now guarded.
  • The CNI lane is untouched apart from the new CreateTAP signature.

Verification

go vet clean on GOOS=linux and GOOS=darwin; make fmt-check clean. On hardware: TAP comes up with the right master/state/MTU/txqlen/GRO, guest gets its DHCP address, reaches the internet and resolves DNS, and removing the VM leaves the bridge with zero ports. The numbers above come from full 256- and 1000-VM fills.

Not covered by unit tests — the touched paths are Linux-gated and the package has no netlink test harness.

CMGS and others added 3 commits July 30, 2026 10:03
Bridge.Add issued eight netlink calls per NIC — LinkAdd, LinkByName, master,
learning, MTU, txqlen, GRO, up. Seven take the kernel's node-wide rtnl lock, and
on a dense warm-pool fill that lock, not the work, sets the ceiling: a 384-core
host attached TAPs at ~19/s no matter how many workers pushed, and a 1000-VM
fill spent most of its wall clock queued behind it.

CreateTAP now returns the index and one RTM_SETLINK carries master, MTU, txqlen,
GRO and IFF_UP together — the `ip link set X up master br` idiom, two ops per
NIC. do_setlink applies MTU before enslaving, so a jumbo bridge no longer dips
the port's MTU on the way in. Measured on a 1000-VM bridge-lane fill: the netdev
ops fall out of the critical path, though the fill is dominated by other rtnl
consumers (TUNSETIFF registration, the VMM's own tap open, linkwatch), so the
end-to-end win is ~7%.

The learning=false tweak goes with it. Nothing ever installed static FDB
entries, so it left the bridge FDB empty and every frame — unicast included —
flooded to all ports: no isolation gained, N-fold datapath amplification at a
thousand taps. Clones already get a fresh MAC from hotSwapNets before resume, so
learning cannot flap a shared address.

netlink resolves a tuntap index with a GETLINK after TUNSETIFF and drops the
error, so guard against a zero index rather than passing one to the kernel.
…lint gate

attachBridgeUp is bridge-specific (IFLA_MASTER is its whole point) and the
network package carried no backend plumbing before it, so it moves unexported
into network/bridge; the two tuning constants TuneTAP shares are exported
instead. The G115 casts carry kernel-issued ifindexes and a guarded MTU, so
they take nolint with reasons. Two-line godocs tightened to one.
A zero-index return left a persistent TAP behind the error; on the bridge
recovery path the leftover carries exactly the name Verify checks, so the next
recover pass would boot the VM against an unattached TAP with no error
anywhere. Clearing TUNSETPERSIST before closing the fds makes the device die
with them, so every failure path converges by recreation.
@CMGS
CMGS merged commit cd129da into master Jul 30, 2026
4 checks passed
@CMGS
CMGS deleted the perf/bridge-add-one-setlink branch July 30, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant