The batched kernel: three families, two kernels, one winner

This is a chapter of The UDP performance ladder: one UDP port, N destinations, every layer of the per-packet cost priced on real hardware. The ladder post carries the contract, the rig and the rules for reading the numbers; this chapter assumes them and goes deep on one rung: how to stop asking the kernel for one packet at a time. The three families, as the ladder names them: transition batching (many datagrams per syscall), transition elimination (shared request/completion rings), and stack batching (the kernel builds or merges the datagrams itself).

Registered I/O: the ring Windows had first

Windows never got the mmsg family (the closest relative is TransmitPackets, which can batch datagrams onto a connected socket from a kernel worker) and went straight to the ring model, which it has had since Server 2012 under the name Registered I/O (RIO): you register your buffers once up front (so the kernel pins and validates them once, not per operation), post receives and sends to a request queue, and harvest results from a completion queue, both living in memory shared with the kernel. Draining the completion queue is a user-mode read, no syscall at all; the kernel only needs an explicit poke (RIONotify) when the queue runs dry and you want to sleep until something arrives.

That Server 2012 date deserves a beat, because the timeline runs opposite to most people’s priors: the shared-ring model shipped on Windows seven years before io_uring brought it to Linux in 2019, and completion-based async I/O there is older still (I/O completion ports date back to Windows NT in the mid-nineties, an era when Linux was still on readiness polling). What io_uring added was not the idea but the generality: RIO is a Winsock extension and moves sockets only, while io_uring is one submission/completion interface for the whole I/O surface, files first (its original motivation, replacing the unloved aio), then sockets, timeouts, and by now a sizable slice of the syscall table. Windows got there first; Linux made it universal.

.NET exposes none of RIO. Its socket engine runs on overlapped I/O on Windows and epoll on Linux, so this rung means hand-written interop: fetching RIO’s function table via WSAIoctl, defining the native structs, and calling raw function pointers. The repository has the full ~400 lines; the shape of the hot loop is what matters here:

while (!ct.IsCancellationRequested) {
    // user-mode read of the shared ring: no syscall
    uint count = _rio.RIODequeueCompletion(_completionQueue, results, DequeueBatch);
    if (count == 0) {
        _rio.RIONotify(_completionQueue);          // arm the kernel wakeup
        count = _rio.RIODequeueCompletion(_completionQueue, results, DequeueBatch);
        if (count == 0) { Rio.WaitForSingleObject(_event, 100); continue; }
    }
    for (uint i = 0; i < count; i++) {
        // receives fan out into posted sends; completions recycle slots
    }
}

The design keeps everything else identical to the ladder’s rung 2 on purpose: one thread, the same serial receive-then-fan-out pipeline, zero steady-state allocation. The single changed variable is how work crosses into the kernel.

One RIO semantic dictates the engine’s whole buffer design, and it deserves its warning even in a measurement post: when a packet arrives and no receive is posted, RIO drops it, and no OS counter records the loss, not the NIC driver’s discards, not the UDP layer’s errors. The engine’s non-negotiable invariant is therefore a never-empty receive ring, and it keeps it with a pre-allocated pool of slots that rotate roles: each receive completion posts a fresh slot from the free pool as its replacement, the filled slot is sent from directly (zero copy) and returns to the pool when its sends complete, and if the pool is empty under transmit backpressure, the just-filled slot is reposted and that one datagram is dropped on a counter the forwarder owns. Posted receives are constant by construction, and when overload forces loss, it lands where you can see it, which is most of what separates a production forwarder from a demo. (The first version coupled slot recycling to send completion instead; the lies chapter has the resulting incident, five layers of counters all swearing nothing was dropped.)

That paragraph is dense, so here it is as something you can push on: both designs over the same six slots, with you playing the network. Let a burst arrive faster than sends complete, then check which counter noticed:

What the syscalls were costing

With the rotating pool in place, the ladder (receive loss against the forwarder’s counter, as in the main post):

The headline is the CPU column. At 200,000 packets per second, the same workload that costs the socket rungs 77 to 86% of a core costs the ring engine 55.5%, and at 150,000 it is well under half a core. The saturation point moves out too: sustained intake peaks around 362,000 packets per second before one core is spent, where the socket rungs give out by 250,000.

One more property the table understates: burst tolerance. Those 4,096 posted receives amount to megabytes of kernel-visible landing space, standing buffer the socket rungs had to be granted explicitly (the ladder’s rung 1 confession) but that this design carries by construction. A deep receive ring is a shock absorber as much as a throughput feature, and real traffic arrives in shocks.

So the syscall pair was worth about a quarter of the per-packet budget. Real, and measurably the largest single win so far, and also the first rung whose price is paid in kind: four hundred lines of unsafe interop, raw function pointers, and a bug class (the invisible drop) that managed sockets simply do not have.

The other two families, on the same socket

Rings are one family of three, and the other two fit the same Windows socket. For all its shared memory, the engine above still asks the kernel once per packet: every reposted receive and every send is a kick, even though completions are harvested for free. Transition batching inside RIO is spelled RIO_MSG_DEFER: a deferred request is a pure user-mode write into the request queue, and one RIO_MSG_COMMIT_ONLY call per dequeue batch kicks everything at once. Stack batching is spelled UDP_SEND_MSG_SIZE (USO, Windows 10 2004 and later): pack the received payloads back to back, hand the kernel one buffer plus the segment size, and the stack builds the datagrams itself, from a plain socket with no interop at all. A capable NIC can take over the actual split in hardware, but none is required: without one the kernel segments in software (Intel’s adapter guide documents the same fallback), and the win survives because the syscalls and stack traversals are amortized before the split ever happens.

The mechanism is easier to see than to say. One packed buffer goes down, the kernel splits it at the bottom; many packets come up, the kernel merges them on the way:

Measured on the same socket, in the same configuration as every other table in the series (the per-request column is the very engine from the table above):

Loss is within noise everywhere except the 400,000 row, where both survivors hold under 1.3%.

Deferred commits are worth 10 to 20 percent of the engine (55.5% to 49.8% at 200,000, 85.3% to 70.0% at 300,000) and push clean intake to roughly 400,000, for the price of one flag and two commit calls. USO halves the engine and keeps going: 28.0% of a core at 200,000, 35.5% against the ring engine’s 85.3% at 300,000 (2.4x cheaper, and flat where the others climb), and at 400,000 it forwards 98.7% of offered, past the ring engine’s entire ceiling, from about a hundred lines of ordinary C# and two setsockopt calls. On Windows, stack batching beat transition batching beat per-request rings, and the winning engine needed no interop at all.

The race also carried two stowaway findings, because it ended up running three times (Defender’s real-time protection and firewall on and off, and a NIC driver update in between; the repository records all three matrices). First, the Defender delta sorted itself by family: the per-packet engines gained ten to twelve points of a core from turning the security stack off, while USO barely moved. Windows’ filtering path is itself a per-packet cost, and batched sends dodge most of it. Second, the driver update moved only USO (nine points cheaper everywhere, with the socket and ring engines unchanged, and Get-NetAdapterUso confirms no hardware offload appeared): even inside one interface family, the driver’s handling of the batched path is its own variable. The family order survived every configuration.

Two honest edges. The USO engine’s receive side is still one syscall per datagram, and Windows’ receive twin (URO, UDP_RECV_MAX_COALESCED_SIZE, Windows 11 24H2 and later) turned out to be a promise this machine cannot keep. Every API said it was on and every probe said it was not: across hundreds of thousands of receives, under every condition that should coalesce, each receive carried exactly one datagram and zero coalescing metadata. Finding out why took a kernel trace and, in the end, an operating-system feature uninstall, and the answer reaches far enough past this benchmark that the whole investigation moved to a chapter of its own: Your Hyper-V checkbox turned off a UDP fast path.

Two facts from that investigation close this chapter’s account. First, even with the underlying cause fixed and URO demonstrably coalescing, it declines this workload: it coalesces 1200-byte datagrams, and 512, and 256, and even 64, but never 32, and this post’s payload is 32 bytes by design, the smallest and hardest case, sitting just under the size where Windows decides coalescing is worth doing. Linux does not draw that line in the same place, which is the one hard difference between the two receive twins in this chapter: the same 32-byte traffic that URO ignores is what UDP_GRO improved by twelve percent a few tables ago. Second, the dead opt-in is free: an interleaved A/B of USO against USO-plus-URO came back a null result, per-round deltas scattered both ways and smaller than the spread within either arm, so the setsockopt is safe to ship even where it does nothing. At QUIC-sized payloads, where URO does engage, it is worth about a sixth of the forwarder’s CPU; that chapter has the measurement too.

So the Windows receive column stays dark by the workload’s own choice, which is exactly what the USO engine’s 400,000 row shows: its loss arrives while nearly half the core still sits idle, a receive-side limit, not a CPU limit. And whether USO composes with RIO’s rings (packed sends posted to a request queue) is a measurement this ladder still owes.

That is one operating system’s verdict, from one socket. Time to run the same families where two of them were invented.

The Linux race: is io_uring even the right pick?

The Linux half deserved a race rather than an assumption, and it grew into the biggest grid of the series: the two plain-socket dispatch models, the three batching families alone and in combination, and the raw frames that the ladder’s rung 5 takes over. Everything runs over the same real link and generator as the Windows tables, from the WSL2 VM, whose mirrored networking mode gives it the host’s own view of the LAN. The io_uring engine is hand-rolled against the raw syscalls (no liburing) with the same slot design as the RIO forwarder, in its straightforward form (one io_uring_enter per completion batch, no multishot or SQPOLL); the combined lanes are the shape msquic’s Linux datapath uses, requests through the ring with UDP_SEGMENT packed sends and UDP_GRO opted in.

Annotations mark loss past the noise band at 300,000. On the unannotated rows the pattern is its own finding: everything running on rings or a receive offload holds 0.2% or better and the two winners hold 0.00 to 0.02%, while the engines whose receive path is still one datagram at a time (plain async, plain blocking, bare mmsg) shed 1 to 3% even at 200,000, because line-rate bursts against a per-datagram receive path always leak a little. The plain engines were measured at 200,000 only. (One accounting caveat rides with every Linux number: process CPU excludes softirq work, where the receive half of the stack runs; the lies chapter has the cross-OS details.)

Four results worth the trip. First, io_uring lost to plain batching, badly: 70.4% of a core against mmsg’s 43.6% for the same work. Both interfaces execute the same kernel UDP path once per datagram and both amortize the user-kernel transition, so the race reduces to per-operation wrapper cost, and io_uring’s ring bookkeeping, request parsing and task scheduling per datagram cost more than mmsg’s bare loop. Multishot receive with provided buffer rings would remove some of that; on this evidence it has a lot of ground to make up. io_uring’s real advantages (mixed I/O in one interface, operation chaining, zero-copy sends of large payloads) are all things a tiny-datagram fan-out never exercises. Pick the interface for the workload you have, not the benchmark crown it holds elsewhere.

The other half of the same finding is mmsg’s modest gap to plain blocking, 49.7% against 43.6%, and it prices the transition itself. A syscall’s entry and exit cost a couple hundred nanoseconds, so at 200,000 packets per second the transition bill this loop can shed is about six points of a core, and that is what mmsg collects; the batching works (bursts hand recvmmsg plenty of datagrams per call), it is what it removes that is cheap. The kernel’s per-datagram walk through the stack is untouched. That is the whole case for the third family: stack batching amortizes the walk as well, which is why it nearly halves mmsg again.

Second, the family that won on Windows wins bigger here, because Linux has both halves of it. UDP_SEGMENT, generic segmentation offload for UDP, lets one sendmsg carry a packed buffer of equal-size payloads plus a segment size; the kernel splits it into packets after traversing the stack once per batch rather than once per packet, and it fits a fan-out forwarder almost too well, because every send is the same bytes to a single destination. That alone does the ring engine’s work for 27.8% of a core. And unlike Windows, the receive twin actually exists down here: UDP_GRO is a software feature of the kernel’s receive path, no NIC support required. Opt in, and the kernel may hand back several same-flow datagrams as one blob whose segment size rides in a cmsg, which a forwarder can pass straight back out as a pre-packed GSO batch, zero copies. That takes another twelve percent off and cleans the loss column to zero: mmsg + GSO + GRO at 24.4% of a core is the cheapest socket engine on either operating system in this series. QUIC implementations ship on exactly this pair.

The last two rows of that table exist because the two operating systems’ offload engines were not structural twins, and the fix turned into the most surprising result of the race. The Windows engine receives one datagram per syscall; the Linux one was quietly also getting recvmmsg batching, so “GSO is worth 27.8%” was really “GSO plus receive batching”. Stripping the receive side back to one recvmsg per datagram prices each half honestly:

GRO alone replaces recvmmsg, and slightly beats it. Batching the receive syscall saves 20 points; letting the kernel coalesce saves 22, from a loop that still calls recvmsg once per receive, because a coalesced blob amortizes the stack traversal as well as the syscall. Stacking both buys another 1.4 points, which is to say they are nearly redundant: on the receive side you need one of the two, not both.

That is what makes Windows’ position worse than a missing feature. Linux has two independent remedies for per-packet receive cost and either one suffices; Windows has neither, because it never got an mmsg family and URO declines this workload (the Windows race above). Its receive path is pinned at one syscall per datagram with nothing to reach for.

Third, strapping the offloads onto io_uring did not rescue it, and one combination grew a secret thread pool. io_uring + GSO fixes the ring’s robustness (0.1% loss at 300,000 where the plain ring sheds 3.2%) but not its cost, because the ring’s per-operation overhead is precisely the part GSO cannot amortize. io_uring + GRO went wrong in a way a CPU column cannot convey: the first run burned 122 to 137% of a core on a single-threaded engine, and a mid-run thread census explained how: 12,739 kernel worker threads inside the process, against exactly one for the same engine without GRO. A recvmsg that carries control data (the GRO cmsg) is punted off io_uring’s polled fast path onto its io-wq worker pool, and for network operations that pool is unbounded by default. Cloudflare hit the same anatomy from another direction and wrote the missing manual for it; their remedy works here too. Capping the pool with IORING_REGISTER_IOWQ_MAX_WORKERS collapses the census to a single worker and the engine to the 63.1% the table shows, at which point GRO finally helps the ring, and it still loses to mmsg + GSO + GRO by 2.6x, because every operation keeps paying the worker detour. The ring’s fast path and the offloads’ cmsgs simply do not compose today, and nothing tells you except a thread count.

Fourth, two deployment notes. io_uring could not even start under a default container seccomp profile (io_uring_setup comes back blocked, a finding from an earlier containerized round of this race), so a containerized forwarder needs a security conversation before the ring is even an option, while mmsg and the offloads run everywhere. And the .NET lanes inverted versus Windows: on Linux, sync socket calls are emulated on top of the async engine, yet the async path burned 1.4 cores to forward what one blocking thread does for half of one. Per-OS ladders really are their own ladders, which is one reason the series never puts a Linux number next to a Windows one and calls a winner (the lies chapter has the others).

What each decision actually bought

The races above compare finished engines, which is how you pick one, but it hides what each individual decision is worth. Rebuilt as a waterfall, one decision per row, the same numbers answer that directly. Linux first, from the plain blocking loop to the winner:

Windows branches where Linux stacks, because USO is not built on the rings; both doors open from the same blocking baseline:

Read either table and the shape is the same: attacking the transition is worth single-digit points per decision, attacking the stack walk is worth tens, and that ratio, not any interface’s elegance, is the entire result of this chapter. Two footnotes keep the waterfall honest. Swapping mmsg’s loop for io_uring at the same job hands back 26.8 points, so a decision can also subtract. And the marginal values are order-dependent, only the endpoints are fixed: batching both syscall directions is worth 6 points when every send still walks the stack (the mmsg row), while batching just the receive syscall is worth 20 next to packed sends (the decomposition a few tables up), and GSO next to a one-datagram-per-receive loop measured 47.8%, within noise of no GSO at all. The credit moves between rows depending on which door you open first; the pair together always lands at 24.4%. The practical reading: take stack batching as one decision, both directions at once, and do not price its halves separately.

The verdict, and who else already knew

With both races on the table, the chapter closes on one sentence: the family order is identical on the two operating systems: stack batching beats transition batching beats per-request rings, by a factor of two or more, and on neither OS did the winning engine require a line of interop. The one asymmetry is the receive path: Linux offers two independent ways to stop paying per datagram there (mmsg batching and software GRO, either one sufficient), while Windows offers neither on this rig, since it never had an mmsg family and its URO never coalesced here (its own chapter has the why, and why this payload size would have been declined anyway).

None of this is a private discovery of one benchmark, which is worth saying because it is the strongest form of the claim: it is where the QUIC world already lives. Every serious QUIC stack ships on segmentation offload as its default socket path (msquic probes for USO and URO at startup on Windows and uses GSO/GRO on Linux; quic-go ships GSO with runtime fallback detection and lists Windows USO as planned work). io_uring is the addition msquic chose, with the moderate returns this race measures, and msquic’s next lever after the ring is an experimental XDP datapath for both operating systems, which is the ladder’s rung 5 by another name.

One more honesty note makes the family’s showing stronger, not weaker: every segmentation number in this chapter is software segmentation. No path measured here had hardware UDP-segmentation support (the Realtek advertises none on either OS, the Hyper-V virtual adapter does not forward it, and loopback has no NIC at all), so these figures are the family’s floor; a NIC that does the split in hardware only widens the gap.

Back in the ladder, rung 4 takes the ring engine to another language, and rung 5 goes below the stack entirely.