The UDP performance ladder: from UdpClient to XDP

Years ago I wrote UDPForwarder_rs, a small Rust tool that does one thing: listen on a UDP port and forward every incoming datagram to a list of destinations. The code has not aged well, but the idea has, because a UDP fan-out forwarder is close to the perfect networking benchmark: there is no protocol to parse and no business logic in the way, so every microsecond the process spends is spent moving packets. If you want to feel the difference between a garbage-collected socket loop and a kernel-bypass data path, this is the workload to feel it on.

This post climbs that difference one rung at a time. We start with the C# code everyone writes first, make it allocation-free, replace the socket APIs with the operating system’s batched I/O interfaces, rewrite the hot path in Rust, and finally go after the kernel’s network stack itself with raw Ethernet frames and XDP, a top rung that does not end the way I planned. This post prices each rung: how much complexity it costs to climb, and how much performance it actually pays back. Every step gets measured the same way, on real hardware, over a real network link, and a companion chapter collects the ways that setup can lie; several of them caught me.

This is a long post. The sections are self-contained on purpose: the table of contents above is the map, and each rung links back to the measurement setup it uses. All code lives in the udp-performance-ladder repository: the forwarder rungs, the load generator and measuring sink, and the micro-benchmarks, so every number in this post can be reproduced from source.

Three companion chapters carry the deep dives, and each stands on its own:

• The batched kernel: three families, two kernels, one winner: rung 3 at full depth. RIO, io_uring, mmsg and the segmentation offloads raced on both operating systems, including the io_uring worker-pool trap.
• The ways this benchmark lied to me: the harness. Every trap that nearly published a wrong number here, and the counting discipline that caught each one.
• Your Hyper-V checkbox turned off a UDP fast path: the detective story. A virtualization checkbox silently turns off a Windows networking fast path, and no API will tell you.

This post assumes you are comfortable reading C# and know what a socket is. Everything below that (syscalls, completion queues, RIO, io_uring, XDP) is introduced before it is used. The first three sections plus rung 1 and rung 2 are the on-ramp and form a complete story about managed-code networking; the later rungs descend into OS-specific APIs, and it is a fine place to stop when kernel bypass is not your problem.

One port in, N destinations out

The forwarder’s contract is deliberately tiny. It binds one UDP socket, and every datagram that arrives is sent, unchanged, to each of N configured destinations. No parsing, no filtering, no rewriting.

A note on naming, because the two terms get mixed up constantly: sending one input stream to many outputs is fan-out (or replication). Multiplexing is the opposite direction, many streams funneled into one. This post builds a fan-out forwarder.

Why would you run one? Because UDP has no built-in way to duplicate a stream. The textbook answer is IP multicast, and on a single switched LAN that you control, multicast is the right tool. But the moment one consumer sits behind a router that does not forward multicast, or a cloud network that does not support it at all, or the source only speaks unicast, you need a process in the middle that receives once and sends N times. Telemetry streams, game-state feeds, market data, log shipping: this box shows up everywhere.

For our purposes the forwarder has a second virtue: it is embarrassingly measurable. Packets in, packets times N out, and any gap between the two is loss we can count.

The ladder

The post reuses five implementations over and over, so let’s name them once and hang everything else off the names.

1. The naive loop (rung 1): UdpClient, one await per operation, a fresh buffer per packet. The code you write in ten minutes.
2. The frugal loop (rung 2): raw Socket, one pinned reusable buffer, the .NET 8 SocketAddress overloads that stop allocating per call. Same syscalls, near-zero garbage.
3. The batched kernel (rung 3): stop paying per-packet kernel costs, in three strengths: batch the syscall (sendmmsg/recvmmsg), eliminate it with shared rings (RIO, io_uring), or hand the stack whole batches to segment itself (GSO, USO). The kernel still forwards every packet; it just stops being asked one at a time.
4. The native rewrite (rung 4): the frugal and batched designs re-done in Rust, to isolate what the runtime costs once the allocations are already gone.
5. The stack bypass (rung 5): raw Ethernet frames through memory-mapped rings (AF_PACKET), and past that the full bypass, AF_XDP on Linux and Microsoft’s XDP-for-Windows, where packets are redirected at (or near) the network driver and never enter the kernel’s network stack at all.

Each climb removes exactly one layer of per-packet cost. That is the experiment design: each rung differs from the one below it by a single decision, so the measured gap between two rungs is attributable to that decision and nothing else.

How to read the numbers

Those gaps get measured on my Windows workstation (Ryzen 7 9800X3D, Windows 11, .NET 10, Realtek 2.5GbE linked at 1 Gbps), with a NAS on the same switch generating the traffic and hosting the measuring sink. One configuration covers every number in this post (the exact driver, Defender state and CPU statistic are in the lies chapter), and every forwarder self-reports CPU with identical process-time accounting, whatever its OS or language. The Linux rungs run on the same machine inside a WSL2 virtual machine (Hyper-V based) behind a virtual adapter, which fixes the one comparison rule this post never breaks: every comparison is a rung against the rung below it on the same OS; Linux and Windows numbers are never compared to each other.

Five facts are enough to read every table below:

• Datagrams are 32 bytes. Small packets are the hard case, and the only one with headroom: 1 GbE tops out near 81,000 full-size datagrams per second but about 1.49 million minimum-size frames per second.
• Loss is the sender’s count minus the forwarder’s own receive counter. UDP drops silently (nothing throws, nothing logs), so loss only exists if something counts at both ends; every datagram carries a sequence number, and a sink behind the forwarder double-checks real delivery where it can.
• CPU at a fixed rate is the headline metric. Two rungs that both sustain 200,000 packets per second are identical on a throughput chart; what separates them is that one does that work for 86% of a core and another for 28%. Every forwarder prints a once-per-second stats line with its own counters (packets in and out, allocation rate, process CPU), and the CPU and allocation numbers below come from there.
• The load is bursty and the runs are warmed. The generator emits line-rate bursts of up to 64 datagrams (the harder and more realistic arrival pattern), each data point is a 10 second run after a discarded 3 second warmup (a cold .NET forwarder pays JIT on its first packets), and every rung gets the same 1 MB socket buffers (rung 1 explains why that alignment matters).
• Loss within about 2% of zero is run-to-run noise. Read it as “clean”, not as exact.

Benchmarks of networking code are unusually easy to get wrong, and this one caught me more than once. Those stories (a sink slower than the forwarder it judged, Ethernet flow control quietly pacing the sender, why loopback numbers are fiction for the upper rungs) are collected in a chapter of their own, The ways this benchmark lied to me: worth your time, but not required to follow the ladder.

Rung 1: the naive loop

This is the forwarder almost everyone writes first, and there is nothing shameful about it:

public static class NaiveForwarder {
    public static async Task RunAsync(
        int listenPort,
        IReadOnlyList<IPEndPoint> destinations,
        CancellationToken ct) {
        using var client = new UdpClient(listenPort);
        client.Client.ReceiveBufferSize = 1 << 20;
        while (!ct.IsCancellationRequested) {
            UdpReceiveResult datagram = await client.ReceiveAsync(ct);
            foreach (IPEndPoint destination in destinations) {
                await client.SendAsync(datagram.Buffer, destination, ct);
            }
        }
    }
}

Fifteen lines, correct, and it will happily forward your home lab’s telemetry forever. Fourteen of them are the code everyone writes; the highlighted buffer line is not, and it earns its confession below. First, read the block the way the ladder reads it, as a per-datagram cost sheet:

• Allocations. ReceiveAsync hands back a UdpReceiveResult containing a freshly allocated byte[] and a freshly allocated IPEndPoint for the sender, per packet. At hundreds of thousands of packets per second that is tens of megabytes of garbage per second, all of it dead microseconds after birth. The GC is good at exactly this shape of garbage, but “good at” still means cycles, cache pollution, and periodic pauses that turn into loss spikes on a protocol with no retransmission.
• Syscalls. One receive plus N sends means 1 + N kernel transitions per datagram, and each send also re-resolves the destination into a native address structure.
• Serialization. The sends are awaited one after another, so a slow send path stalls the receive loop, and the whole forwarder runs on one logical thread regardless of how many cores the machine has.

None of this is visible at 1,000 packets per second. All of it is the wall you hit somewhere on the way to a million. Here is where the wall actually is (receive loss is the sender’s count against the forwarder’s own receive counter):

Now the confession the highlighted line owes. Left alone, UdpClient keeps the OS default receive buffer of roughly 64 KB, and with that default the same forwarder measured 2.2% loss at 200,000 and 9.5% at 250,000, purely from bursts overflowing the kernel buffer while the loop was busy. That is the quiet tax of convenience classes: the defaults you never chose are still choices, and they are exactly what bites when you use the friendly wrapper without thinking about the level underneath it. One line removes it, so every rung in this post runs with the same 1 MB buffer and the ladder measures one variable per rung. But remember that the naive forwarder you find in production will not have that line. It also would not survive long on a real Windows wire: the repo version needs two more bandages against an ICMP message that kills receive loops and a driver transmit path that kills sends, and the lies chapter prices both.

Two things in that table matter more than the headline rate.

First, the failure mode is not an error, it is arithmetic. Nothing throws, nothing logs, no counter in the application goes red. At 300,000 packets per second the forwarder is still cheerfully reporting that it forwarded everything it received. It just received a ninth less than was sent to it. The only reason we know is that something outside the process counts what was sent and does the subtraction. If your production forwarder does not have something playing that role, you do not have a monitoring gap, you have no idea whether you are dropping traffic at all. Call that discipline counting at both ends; it will save this post again at the top of the ladder, and the lies chapter shows what it caught behind the scenes.

Second, look at the CPU column next to the loss column. The cliff arrives as CPU saturates a single core, on a machine with sixteen logical cores. The naive loop is a strictly serial pipeline: receive, then send, then receive again. Fifteen cores sit idle while one core decides the throughput of the entire process.

So we have our first honest number: this forwarder is clean to 200,000 packets per second and breaks by 250,000, pinned to one core. Now let’s find out which of the three costs above is actually responsible.

Rung 2: the frugal loop

The obvious suspect is the garbage. Every datagram allocates a byte array and an IPEndPoint, and at 150,000 packets per second that is a lot of short-lived objects. So let’s remove them all.

Drop UdpClient for a raw Socket, allocate the receive buffer once (pinned, so the GC never relocates it under an in-flight I/O), resolve every destination to a SocketAddress at startup, and use the .NET 8 overloads that take a reusable SocketAddress for the sender rather than handing back a fresh IPEndPoint:

using var socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
socket.Bind(new IPEndPoint(IPAddress.Any, options.ListenPort));
socket.ReceiveBufferSize = 1 << 20;

SocketAddress[] destinations = options.Destinations.Select(e => e.Serialize()).ToArray();
byte[] buffer = GC.AllocateArray<byte>(65536, pinned: true);
Memory<byte> receiveMemory = buffer;
var sender = new SocketAddress(AddressFamily.InterNetwork);

while (!ct.IsCancellationRequested) {
    int received = await socket.ReceiveFromAsync(receiveMemory, SocketFlags.None, sender, ct);
    ReadOnlyMemory<byte> datagram = receiveMemory[..received];

    for (int i = 0; i < destinations.Length; i++) {
        await socket.SendToAsync(datagram, SocketFlags.None, destinations[i], ct);
    }
}

The syscall pattern is deliberately identical to rung 1: one receive, N sends, awaited in order. The only thing that changed is how much garbage the loop makes. That isolation is the point, because it turns the measurement into an answer to exactly one question.

It worked, and it did not matter

Measure the naive way with BenchmarkDotNet’s memory diagnoser, one benchmark method per datagram, and you get this:

Rung 2 removes nearly three quarters of the garbage, but where do its last 72 bytes come from? Splitting the benchmark answers that: SendToAsync allocates nothing, because it completes synchronously (there is buffer space, so there is nothing to wait for), while ReceiveFromAsync accounts for the entire 72 bytes, because it genuinely suspends and the compiler’s state machine gets boxed onto the heap.

That box is the last allocation, and it is not really per datagram. It is per async method invocation, and this benchmark invokes one per datagram while the forwarder runs its whole loop inside a single long-lived async method. BenchmarkDotNet models that directly with OperationsPerInvoke: put the loop inside the benchmark and let it divide the results by the batch size.

[Benchmark(OperationsPerInvoke = Batch)]
public async Task<int> Rung2_RawSocket_Loop() {
    int last = 0;
    for (int i = 0; i < Batch; i++) {
        await _rawSocket.SendToAsync(_payload, SocketFlags.None, _rawSocketSelf);
        last = await _rawSocket.ReceiveFromAsync(_receiveBuffer, SocketFlags.None, _sender);
    }
    return last;
}

Now the numbers describe the forwarder rather than the benchmark harness:

Rung 2 allocates nothing at all, and rung 1 drops from 272 to 200 bytes. The 72 byte difference is exactly the state machine box, amortized to nothing across a thousand datagrams: 272 minus 72 is 200, which is the real per-datagram garbage, the fresh receive buffer and the fresh IPEndPoint for the sender.

The running forwarders agree, which is the check that matters: their stats lines report allocation straight from the GC’s own counter, on the real traffic. At a sustained 150,000 packets per second:

rx     150,024 pps     38.4 Mbit/s | tx     150,024 pps     38.4 Mbit/s | alloc    36.8 MB/s gen0  14

rx     149,989 pps     38.4 Mbit/s | tx     149,989 pps     38.4 Mbit/s | alloc     0.0 MB/s gen0   0

Rung 1 churns 37 MB per second and 14 Gen0 collections per second; rung 2 allocates nothing measurable and never collects. Not “less garbage”: none.

(One overload trap is worth a sentence in passing: UdpClient’s parameterless ReceiveAsync()/SendAsync() are a different overload family that returns a fresh Task per call and costs about 160 bytes per datagram more. The benchmark passes a CancellationToken because the forwarder does; whatever you measure, make it the overloads you actually ship.)

Now the throughput on the wire, same ladder as rung 1:

That is the same forwarder within run noise: the CPU deltas against rung 1 bounce both directions by up to ten points across the rates with no consistent sign, both engines are clean to 200,000, and both break in the same band by 250,000. The percent or two the micro-benchmark shows is real, and it is far too small to survive wire-run variance.

Eliminating every allocation on the hot path bought nothing, because allocation was never the bottleneck. Look again at the micro-benchmark’s Mean column: about 4.4 microseconds per datagram, essentially unchanged. Whatever is eating those microseconds is not the GC, because we removed the GC entirely from this path and the clock did not notice. And the arithmetic closes: at 4.4 microseconds per datagram, a serial loop caps out at 1 / 4.4 us, about 230,000 packets per second, and the wire ladder indeed breaks between 200,000 and 250,000. A loopback micro-benchmark cannot see the NIC or the driver, so landing inside the real wall’s band is about as much agreement as this cross-check can give.

The remaining suspect is the one thing both rungs still do identically: cross into the kernel once per receive and once per send.

Was this rung a waste, then?

No, and it is worth being precise about why. Rung 2 is not faster, but it is more predictable. Rung 1 sustains 37 MB per second of garbage and 14 Gen0 collections per second at 150,000 packets per second. Those collections are individually cheap and they are still pauses, and a pause in a UDP forwarder is not a latency blip, it is packet loss with nothing behind it to retransmit. Averaged over a ten second run that disappears into the mean. In a p99.9 latency budget it does not.

So rung 2 is worth keeping: it costs about twenty lines of less idiomatic code, no portability, and no dependencies. Just do not expect it to raise your throughput ceiling, and do not let anyone tell you Span<T> made their network code fast without showing you the packets-per-second number next to the allocation number.

A word on System.IO.Pipelines

The usual answer to “high-performance I/O in .NET” is System.IO.Pipelines, and it is deliberately absent here. Pipelines solve a problem this workload does not have: parsing a contiguous byte stream whose message boundaries you must discover, while managing buffer lifetimes between a producer and a consumer. UDP hands you exactly one complete message per receive. There is no framing to do and no partial reads to stitch, so wrapping a pipe around it adds a copy and an abstraction in exchange for solving nothing.

Rung 3: the batched kernel

Rung 2 left exactly one suspect standing: the trip into the kernel. A syscall is a transition from your program into the kernel: the CPU switches privilege levels, speculative-execution mitigations flush state, and the kernel validates everything you handed it. None of that is free, and rungs 1 and 2 pay it twice per datagram, once to receive and once to send. At around 4 microseconds per datagram with the allocations already gone, that pair is the biggest cost still standing.

Attacking that pair comes in three strengths. Naming them once pays off for the whole chapter, because every engine from here on is one of these three, on one OS or the other:

1. Transition batching keeps the ordinary blocking calls but carries a whole batch of datagrams per syscall. Linux spells it sendmmsg/recvmmsg; Windows improvises it inside RIO with deferred commits. (This post’s traffic generator runs on the Linux spelling, at over a million packets per second.)
2. Transition elimination goes further: request and completion rings shared with the kernel, so a busy loop stops paying a syscall per operation entirely. Linux spells it io_uring; Windows spells it Registered I/O.
3. Stack batching attacks a different cost. Hand the kernel one packed buffer plus a segment size, and it builds (or merges) the datagrams itself, so even the per-packet walk through the network stack happens once per batch. Linux spells it UDP_SEGMENT (GSO); Windows spells it UDP_SEND_MSG_SIZE (USO); the receive-side twins are GRO and URO.

The first two change how often you ask the kernel; the third changes how much work each ask contains. Which family wins is a measurement, not an assumption, and this chapter runs the race on both OSes.

The full contest is its own chapter, because it grew into the biggest grid of the series: RIO’s rings and their deferred commits, io_uring alone and strapped to the offloads, mmsg, GSO, USO, GRO, a 12,739-thread kernel worker-pool explosion, and the reason Windows’ receive offload stays dark. Read it here: The batched kernel: three families, two kernels, one winner. The verdict travels light:

Stack batching beats transition batching beats per-request rings, in the same order on both kernels, and the winning engines are plain sockets with two setsockopt calls, no interop. The lopsided last row is not an oversight: Windows’ receive twin, URO, never coalesced a single datagram here, so the Windows winner batches sends only, while Linux has two receive-side remedies (mmsg and software GRO) and either one suffices. Why the receive twin stayed dark is the best story in the series, and it gets a chapter of its own. Stack batching is also where the QUIC world already lives: every serious QUIC stack ships on segmentation offload as its default socket path. The chapter has the mechanics, both races, the io_uring war story, and the honest edges. Hold on to RIO’s 55.5% figure here, because rung 4 is about to make the ring engine interesting for a different reason.

Rung 4: the native rewrite

Before this rung ran, I wrote down a prediction so it could not quietly become hindsight: once the C# is allocation-free, rewriting it in Rust buys less than the internet thinks. Here is what the measurement did to that sentence.

The port is deliberately boring: the frugal loop (rung 2), architecture kept identical. One thread, blocking recv_from, one send_to per destination, one reused buffer, destinations resolved once, the same aligned 1 MB receive buffer. About 120 lines of Rust against rung 2’s similar count of C#, and no design changes whatsoever, because the moment the rewrite also redesigns, the measurement stops isolating the runtime.

let mut buffer = [0u8; 65536];
loop {
    let (received, _source) = socket.recv_from(&mut buffer)?;
    stats.rx_packets.fetch_add(1, Ordering::Relaxed);

    for destination in &destinations {
        socket.send_to(&buffer[..received], destination)?;
    }
}

Same ladder, same harness, and at first glance a clear win for the rewrite:

Twenty-plus percent less CPU for the same work, and a cleaner loss column. I nearly published that as “what the runtime costs”, and it would have been wrong, because these two loops differ in one more way than the language: rung 2’s C# loop is async, and Rust’s is blocking. On the wire, an awaited receive genuinely suspends and resumes through the thread pool, per packet. That is a different I/O dispatch model, not a different language. So the comparison owed a control: the same C# loop with .NET 8’s synchronous SocketAddress overloads, blocking exactly like the Rust one.

The control redistributes the credit completely, and further than politeness would have predicted: essentially the whole apparent Rust win was the async machinery, removable in C# by changing four method calls. Blocking loop against blocking loop, C# and Rust sit within run noise of each other (63.1% versus 65.6% at 200,000, with the sign flipping across rates): on plain sockets, the language does not resolve as a cost at all on this rig. Notice also how the async penalty shrinks as load rises and the C# columns converge by 250,000: a saturated receive loop finds data already waiting, so its awaits complete synchronously and stop paying the suspension tax. Async costs the most exactly when you are not busy, which is the opposite of most people’s intuition about it.

Which raises the question the control demands: is that tax inherent to async, or is it .NET’s? Rust can answer, because the same forwarder runs on tokio’s single-threaded runtime with only the loop’s calls changed to .await:

Tokio does one better than zero: its async loop reads cheaper than the blocking one at moderate load. The async tax is an implementation property, not a concept property. In this program’s shape, Rust compiles the whole chain of awaits into one state machine that lives on block_on’s stack (a tokio::spawned task would cost one heap allocation), and on the current_thread runtime the same thread polls the future and runs the I/O driver, so a suspension never changes threads. In .NET, an await that actually has to wait is completed through an I/O completion port and resumed on a thread-pool I/O thread, a cross-thread trip per suspension, plus ExecutionContext bookkeeping on the way. The runtime is smart about the other case: operations that complete immediately skip the completion port entirely (FILE_SKIP_COMPLETION_PORT_ON_SUCCESS), and that is precisely why the tax fades at saturation, when data is always already there. What the 18 to 23 points bought at moderate load is the suspensions themselves. (In fairness: one task on one thread is tokio’s best case, and .NET’s dispatch model buys scale-out properties a one-socket benchmark never exercises.)

Now stack the ranking measured across this post. At 200,000 packets per second, one core’s worth of forwarding costs: C# async 86%, Rust blocking 66%, C# blocking 63%, Rust async 59%, C# on Registered I/O 55.5%, and USO does the same work for 28%. The kernel interface beat the dispatch model, and the language barely registers. My written-down prediction survives in the sharpest form yet: on plain sockets the rewrite bought approximately nothing that the dispatch model did not explain, and the cheapest-looking rung on a conference slide is the most expensive one per percent gained. Choose your kernel interface first, your dispatch model second, and your language last.

The obvious follow-up is Rust on a batched interface, and the RIO ring engine from the batched-kernel chapter is the natural port target: its slot-rotation design translates one to one, while the USO winner is mostly kernel work already, leaving a language little to save. If the effects are additive, a Rust port of the RIO engine should land at or a little under C# RIO’s 55.5%. The port (a 1:1 translation) measures 51.5%, about seven percent under, which makes RIO the one engine in this post where the language delta rises above run noise, and still the smallest lever on the board. It also stays clean to 350,000 (0.08% loss), the best intake of any ring engine here.

That closes every account above the kernel. Allocations: zero effect. Dispatch model: up to 23 points of a core in .NET, mildly negative in tokio. Language: zero to seven percent, depending on the engine. The kernel interface: the biggest lever on the board, and so far we have only pulled it from inside a socket. One layer is left on the bill, and it is the network stack itself.

Rung 5: the stack bypass

The socket path bottoms out around a quarter of a core at 200,000 packets per second (USO’s 28%, the offload pair’s 24% on Linux, the ring engines above them), and none of that budget goes to allocations, per-packet syscalls, or dispatch anymore. So what is it still paying for? Mostly the network stack. However cheaply the packets are handed over, every rung so far ends at a socket, which means the kernel still runs its full receive machinery per datagram: allocate a packet descriptor, walk the IP layer, find the owning socket, queue the payload, and do the mirror image N times on the way out. The last rung removes that, and it comes in two strengths.

The strong form is XDP (eXpress Data Path): you load a small program into the kernel, written for eBPF (a restricted bytecode the kernel verifies before it agrees to run it), and the network driver executes that program for every arriving packet at the earliest possible moment, before any stack processing. The program can drop the packet, bounce it back out the same NIC, or hand it to user space through AF_XDP, a socket type whose receive and send queues are descriptor rings over memory shared with the driver. Used that way, a packet travels from the NIC into your buffer and the kernel’s network stack never sees it. Microsoft’s XDP-for-Windows brings the same model to Windows.

The mild form is AF_PACKET with memory-mapped rings: raw Ethernet frames exchanged with the kernel through shared rings, one syscall kicking a whole transmit batch. The frames still ride the normal driver path, so it is a partial bypass, but it needs no eBPF and no special driver support, and that modesty is about to matter a great deal.

Down here, you are the network stack

Whichever strength you pick, the same bill arrives, because everything the stack was doing for free is now your code:

• Headers. Parse Ethernet, IPv4 and UDP on the way in; build all three per destination on the way out; recompute the IP checksum every time.
• Address discovery. Nobody resolves neighbours for you at this layer. The AF_PACKET engine reads its own MAC and IPv4 from the interface and learns the peer’s MAC from the neighbour table after an ARP probe, so deploying it means naming an interface and nothing else, but that convenience is code the engine had to contain.
• Ring discipline. Descriptor rings with backpressure on both sides, the same slot-rotation problem RIO taught in the batched-kernel chapter, except now a mistake drops raw frames instead of datagrams.
• Trust. The stack extends none to frames it did not build: an injected frame takes paths ordinary traffic skips, and fails in places no counter reports. The next section falls into exactly that hole.

That bill is the rung’s real price, and it is why the chapter’s question is not “how do I climb this” but “does my situation pay for the climb”.

AF_PACKET: the bypass you can actually run

The first outing looked perfect. In an early loopback round of the batched-kernel race, the engine parsed every frame at a fraction of a core, and its forwarding numbers read 0.00% loss at every rate. The sink received nothing. It was counting frames it had queued, not packets anyone got, and only the harness counting at both ends caught it: rung 1’s silent-drop trap, one layer lower and pointed at me this time.

Chasing it turned up the reason raw-frame injection is harder than the API suggests. The frames were fine (a second sniffer instance saw every one of them come back inbound) and they entered the IP receive path with no header, checksum or address errors, yet were never delivered. They die in the kernel’s input routing, which normal loopback traffic never reaches: a locally generated packet carries the route from its output lookup, so the receive path skips routing entirely. An injected frame has no such route, takes the full input path, and gets judged a martian (a packet whose source address cannot legitimately arrive where it did), a verdict no counter records.

The lesson generalizes past the bug, and it is the trust bullet above made concrete. Raw-frame forwarding is only honest toward another host, which is how the technique is deployed and, not coincidentally, how DPDK’s own AF_PACKET driver uses it: a DPDK application owns the port and transmits to the wire, never asking the local kernel to route an injected frame back to a local socket. Measured that way, over the real link with the sink seeing the traffic arrive (its residual shortfall sits against its own ~220,000 pps ceiling, not against the forwarder), it is the cheapest engine on the entire Linux board:

At 200,000 packets per second it forwards everything for an eighth of a core, half of even the mmsg+GSO+GRO winner. Then it falls off a cliff: by 300,000 it sheds a seventh of the arriving traffic while using barely more CPU, the signature of one thread doing every header itself with no kernel batching underneath to absorb the shock. The 12.5% also carries the sharpest form of the softirq caveat: the kernel fills these rings before the process ever runs, so process CPU understates AF_PACKET‘s total system cost more than it does the socket engines’.

Still, on commodity hardware this is the practical top of the Linux ladder, which raises the obvious question about the actual top.

The rung the hardware refused

I planned to end this post with AF_XDP racing XDP-for-Windows on the same rig as every other number. That plan died in a driver audit, and the way it died is worth more than the table it replaced.

XDP attaches in one of two modes. Native mode is the product: the driver itself runs your program before the stack exists for that packet. Generic mode is a compatibility fallback for drivers without the hook: the kernel runs your program after doing the per-packet stack work anyway, at which point the cost XDP exists to remove has already been paid. Generic-mode numbers look plausible and prove nothing. The trap has teeth because the fallback is silent: attach without pinning XDP_FLAGS_DRV_MODE, and an unsupported driver quietly serves you generic mode and a benchmark that measures the wrong thing convincingly, the same failure shape as the PAUSE frames in the appendix.

So, the audit, machine by machine:

• The workstation under Windows. XDP-for-Windows in native mode requires the NIC driver to implement Microsoft’s NDIS XDP extensions, which in practice means Intel, NVIDIA and Microsoft’s own adapters. A consumer Realtek 2.5GbE driver does not ship them. Generic only.
• The same machine under Linux. The same NIC binds the r8169 driver, which has no XDP hook. Generic only.
• The Linux VM. Its virtual NIC driver does implement the hook, so a program attaches in native mode and looks completely legitimate. But a packet reaches that virtual driver only after the physical Realtek driver, the Windows networking stack and the Hyper-V virtual switch have each done their work. XDP there bypasses the Linux stack after every cost it is meant to remove has already been paid one layer up. Valid as an A/B experiment inside the VM; invalid as a claim about what XDP buys on hardware.
• The NAS. r8168, Realtek’s out-of-tree driver. No hook either.

Every route to an XDP number in this environment is a generic-mode number, and I would rather publish the audit than the fiction.

And the audit is the finding. Rung 5’s entry fee is not code, it is hardware. Every climb until now was a software decision: rewrite a loop, learn an API, add a toolchain. This one starts with procurement: an Intel igb/igc-class NIC (inexpensive, but a physical purchase), or a cloud instance whose virtual NIC implements native XDP, which AWS’s ENA and GCP’s gVNIC both do. The rung itself is real and load-bearing at places that made the purchase; Meta’s Katran load balancer and Cloudflare’s DDoS mitigation both stand on XDP at packet rates no socket API reaches. But they arrived with the NICs, the drivers, and the teams the rung assumes. The deployment reality check this chapter promised turns out to be its headline: before you budget the code, check whether your driver will even take it.

What the ladder measured

The Windows ladder end to end, every rung against the same generator, the same bursty arrival profile, the same aligned buffers, one configuration:

The Linux board over the same link (from the virtualized adapter, so comparable within this table and to nothing above), at a sustained 200,000 packets per second:

And the hierarchy, which is the sentence this post exists for: the kernel interface is worth 2x and more (segmentation offload against plain sockets, on both OSes), the dispatch model up to a quarter of a core in .NET and roughly nothing in tokio (and 3x on Linux epoll), the language at most seven percent, and allocation removal zero. Climb in that order.

What the ladder still owes: a Windows ladder at QUIC-sized payloads, where URO does engage, an io_uring rematch with multishot receive and provided buffer rings (the real-link data makes parity with mmsg look optimistic), USO composed with RIO’s rings, and rung 5 itself, the day an XDP-capable NIC enters the building.

When to stop climbing

The decision aid, with the ladder now priced:

The rule of thumb

Climb one rung at a time, and only after measuring the rung you are on. Every level of this ladder buys throughput with a currency that gets more expensive as you go: first readability, then portability, then the entire managed ecosystem, and finally the kernel’s networking features themselves (at rung 5’s XDP end, even tcpdump no longer sees your packets). The naive loop is not the embarrassing version of the fast forwarder; it is the baseline that tells you whether you have a problem worth paying for.

Rung 2 is the cautionary tale in miniature. It is the optimization everyone reaches for first, it did exactly what it promised (allocation went to essentially zero), and it moved the throughput number by nothing at all, because the cost it removed was not the cost that mattered. Meanwhile the change that did move a number in this post was a socket buffer size: the least glamorous knob on the board, and a default nobody had consciously chosen. That is the normal outcome of an optimization chosen by reputation instead of by measurement. Most services never need rung 3. But when you do need it, you want to arrive with a profile, not a preference.

Two habits paid for this whole post, and they transfer. Count at both ends: every dramatic number here that later turned out wrong (the sink’s hidden ceiling, RIO’s invisible drop, AF_PACKET‘s phantom forwarding, the PAUSE frames’ flattery) was caught by the same discipline, an independent counter on the far side of the thing being measured. And read the top of the ladder before you budget the climb: rung 5 is real, and other people are standing on it, but no software decision gets you there without a NIC whose driver cooperates.

The ways this benchmark can lie

They are legion, they are instructive, and they have their own chapter: The ways this benchmark lied to me, from the sink that quietly judged its own ceiling to the two traps that each killed a Windows forwarder mid-run. Nothing there is needed to follow the ladder; all of it was needed to trust it.

One of those lies grew too large to keep here. Chasing a receive-side offload that reported success and did nothing ended in a kernel trace, an operating-system feature uninstall, and a finding that reaches well past this benchmark: Your Hyper-V checkbox turned off a UDP fast path.