perf(bridge): attach and configure a TAP in one netlink message - #179
Merged
Conversation
Bridge.Add issued eight netlink calls per NIC — LinkAdd, LinkByName, master, learning, MTU, txqlen, GRO, up. Seven take the kernel's node-wide rtnl lock, and on a dense warm-pool fill that lock, not the work, sets the ceiling: a 384-core host attached TAPs at ~19/s no matter how many workers pushed, and a 1000-VM fill spent most of its wall clock queued behind it. CreateTAP now returns the index and one RTM_SETLINK carries master, MTU, txqlen, GRO and IFF_UP together — the `ip link set X up master br` idiom, two ops per NIC. do_setlink applies MTU before enslaving, so a jumbo bridge no longer dips the port's MTU on the way in. Measured on a 1000-VM bridge-lane fill: the netdev ops fall out of the critical path, though the fill is dominated by other rtnl consumers (TUNSETIFF registration, the VMM's own tap open, linkwatch), so the end-to-end win is ~7%. The learning=false tweak goes with it. Nothing ever installed static FDB entries, so it left the bridge FDB empty and every frame — unicast included — flooded to all ports: no isolation gained, N-fold datapath amplification at a thousand taps. Clones already get a fresh MAC from hotSwapNets before resume, so learning cannot flap a shared address. netlink resolves a tuntap index with a GETLINK after TUNSETIFF and drops the error, so guard against a zero index rather than passing one to the kernel.
…lint gate attachBridgeUp is bridge-specific (IFLA_MASTER is its whole point) and the network package carried no backend plumbing before it, so it moves unexported into network/bridge; the two tuning constants TuneTAP shares are exported instead. The G115 casts carry kernel-issued ifindexes and a guarded MTU, so they take nolint with reasons. Two-line godocs tightened to one.
A zero-index return left a persistent TAP behind the error; on the bridge recovery path the leftover carries exactly the name Verify checks, so the next recover pass would boot the VM against an unattached TAP with no error anywhere. Clearing TUNSETPERSIST before closing the fds makes the device die with them, so every failure path converges by recreation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bridge.Addissued eight netlink calls per NIC —LinkAdd,LinkByName, master, learning, MTU, txqlen, GRO, up. Seven of them take the kernel's node-widertnllock, so on a dense warm-pool fill the lock, not the work, sets the ceiling.This collapses the sequence to two ops:
CreateTAPreturns the index, and oneRTM_SETLINKcarriesIFLA_MASTER+IFLA_MTU+IFLA_TXQLEN+IFLA_GRO_MAX_SIZE+IFF_UPtogether — theip link set X up master bridiom that iproute2 has always used.Why
learning=falsegoes with itNothing in this repo or in sandbox ever installed a static FDB entry, so disabling learning left the bridge FDB permanently empty and every frame — unicast included — flooded to all ports. That bought no isolation and cost N-fold datapath amplification: on a bridge carrying a thousand taps, ordinary DHCP traffic was enough to drive the host into a broadcast storm.
Clones already get a fresh MAC from
hotSwapNetsbefore resume, so learning cannot flap a shared address between ports. Verified after the change: a non-permanent FDB entry appears per guest, and guest-bound unicast stops flooding.One caveat: an FC clone cannot swap its guest MAC before resume, so an FC bridge clone shares the source's MAC until the operator applies the CLI's repair hints. With learning on that shared MAC flaps the FDB between ports; with learning off the clone simply received all of the source's traffic via flooding — the pair was never a functioning config either way (both answer ARP for the same MAC-keyed DHCP lease), so this PR changes how that window fails, not whether it works.
Measurements
A 384-core host, 512 MiB guests, one bridge, warm-pool fill driven by sandboxd.
The end-to-end win is ~7%, not the 7x the isolated op suggests: the fill is dominated by other
rtnlconsumers this PR cannot touch — TUNSETIFF device registration, the VMM's own tap open (another TUNSETIFF per queue fd), linkwatch carrier batches, IPv6 addrconf per port-up. FittingX(W) = W/(P + W·S)across six concurrency points puts the serialized component at ~16.8 ms/VM for the bridge lane against ~5.4 ms/VM for the no-NIC lane; this PR removes a few ms of it. The change is worth landing for the op-count reduction and the flooding fix, not as a fix for the lane's throughput gap.One caveat worth knowing before you land it
On a host that still logs to a console at default loglevel, this change is a regression, not a win: collapsing the ops drives more bridge-port state transitions per attach (2.2 → 4.0
entered … statelines), and each such line is written synchronously to the console whilertnlis held. On a host with two registered consoles that measured ~14 ms per line, which is more than the netlink calls this saves — a 256-VM fill went 23.3 s → 30.1 s. With the console quiet the same fill is 9.8 s → 9.1 s.So the bridge lane wants
console_loglevel < 6in its deployment contract. That is a host setting, not something this PR can carry.Behaviour and safety
do_setlinkapplies MTU before enslaving, so a jumbo bridge no longer dips the port's MTU on the way in — strictly better than the old enslave-then-set order.EXFULLat a bridge's 1024-port limit) leaves an up-but-unattached TAP;Add's deferredtearDownTAPsis state-insensitive and already cleans that.GETLINKafterTUNSETIFFand drops the error, soCreateTAPcould hand back index 0. Now guarded.CreateTAPsignature.Verification
go vetclean onGOOS=linuxandGOOS=darwin;make fmt-checkclean. On hardware: TAP comes up with the right master/state/MTU/txqlen/GRO, guest gets its DHCP address, reaches the internet and resolves DNS, and removing the VM leaves the bridge with zero ports. The numbers above come from full 256- and 1000-VM fills.Not covered by unit tests — the touched paths are Linux-gated and the package has no netlink test harness.