# The ways this benchmark lied to me

> Sinks that judge their own ceiling, PAUSE frames that flatter, drops no counter records, and the two traps that kill Windows UDP forwarders.

- Author: Simon Wimmesberger
- Published: 2026-08-21
- Canonical: https://blog.wimmesberger.dev/posts/udp-ladder-benchmark-lies/
- Tags: networking, performance

---
*This is a chapter of [The UDP performance ladder](../udp-performance-ladder/). Nothing here is needed to follow the ladder; all of it was needed to trust it. Each entry either burned me during the project or is guarded against because it has burned others, and the discipline that caught every single one is the ladder's recurring motif: counting at both ends, an independent counter on the far side of the thing being measured.*

The rig in full: Ryzen 7 9800X3D (8 cores, 16 threads), Windows 11 build 26200, .NET 10.0.302, Realtek 2.5GbE controller linked at 1 Gbps; the NAS generator and sink run in a container and are driven over an HTTP control API, so a run is one request rather than one deployment.

**Loopback flatters everything.** Loopback traffic never touches the NIC, the driver, DMA, or interrupt handling. For the lower rungs that merely skews the numbers; for rung 5 it invalidates them, because XDP's entire value is attaching packet processing at or near the network driver, and on a virtual or loopback path you only get the *generic* fallback mode, which re-enters the very stack we are trying to bypass. Loopback is fine for development iteration and useless for publication. Hence the real switch, and a generator in a container with host networking (a bridge network would add its own forwarding layer and pollute the numbers).

**Cross-OS scoreboards are fiction here.** The Linux rungs run inside a Hyper-V based VM, so Linux packets traverse the virtual switch while Windows talks to the physical NIC. The shared overhead cancels within an OS, which is why every comparison is a rung against the rung below it on the same stack; cross-OS deltas are reported but explicitly not scored. The batched-kernel chapter adds the deeper reason: the two kernels do not even bill I/O work to a process the same way (Linux runs the receive half of its stack in softirq context, unbilled to the process).

**The generator must outrun the forwarder.** If the sender caps out first, every rung above that cap measures the sender. The generator is therefore built on the ladder's own upper-rung techniques and validated before any forwarder numbers count. Its first version was a plain send loop that topped out near 540,000 packets per second, below 1 GbE line rate for small packets; batching many datagrams per syscall lifted it to ~620,000 over the wire, which is its own small joke, because batched syscalls are exactly what rung 3 is about.

**The sink was the first thing counting at both ends caught.** The sink's first version was a single receive loop, and I validated it at 400,000 packets per second *over loopback*, then trusted it over the wire, where it turned out to cap near 150,000. Every "delivered" number above that was measuring the sink, and the first cliff I found for rungs 1 and 2 was the sink's cliff, not theirs. The tell that finally exposed it: the forwarders' own counters showed `tx == rx` exactly while the sink reported massive loss, so the packets were dying after the forwarder. Two consequences for every number in the series: the sink receives on four threads (more does not help; the NAS is CPU-bound) and validates end-to-end delivery only below its ~220,000 pps ceiling, and the primary cliff metric is the forwarder's own receive counter versus the sender's count, baselined after warmup, with the NIC's hardware counters as the tiebreaker when layers disagree.

**The scariest counter is one that stays at zero.** Rung 3's first RIO engine sent straight out of the receive buffer, so a receive slot could only be reposted after all its sends completed. Under load, send completions lag, slots pile up waiting, and the posted-receive ring drains to empty, where RIO discards arrivals without a trace. At 300,000 offered:

| Layer | Counter | Value |
| --- | --- | --- |
| Generator | packets sent | 2,999,658 |
| NIC hardware | unicast received | 2,999,923 |
| NIC driver | receive discards | 0 |
| Windows UDP layer | receive errors | 0 |
| Forwarder | packets seen | 2,624,376 |

The wire delivered everything (the NIC's counter reads slightly above the generator's because the harness's HTTP control traffic shares the link), every OS layer swore nothing was dropped, and 375,000 packets were simply gone; only the sink counting sequence numbers end to end caught it. If it had not been counting, this forwarder would have shipped. The slot-rotation design in [the batched-kernel chapter](../udp-ladder-batched-kernel/) exists because of this run.

**Fan-out multiplies transmit load.** With two destinations, every received packet leaves twice, so the forwarder's 1 GbE transmit side would saturate at roughly half the receive rate. That is why every published run fixes the destination count at one: transmit equals receive, and the whole link budget is spent on the machinery under test rather than on duplicate copies. The costs the ladder isolates are per send either way; N only multiplies them.

**Receive buffers hide the cliff.** A large `SO_RCVBUF` absorbs bursts and makes a too-slow forwarder look lossless in short runs. Runs are long enough to reach steady state, and buffer sizes are pinned and reported. This one nearly published a false win: before the buffers were aligned, an early rung 1 vs rung 2 comparison showed the frugal loop dramatically more burst-tolerant, and it would have been easy to hand that win to the allocation work. It belonged to `UdpClient`'s 64 KB default overflowing under bursts while rung 2's explicit 1 MB did not; aligned, the difference vanished entirely. Always check what *else* your optimized version changed. (On the RIO rungs the socket buffer is a no-op, posted receives are the buffering there, but they carry the same 1 MB setting so the alignment is uniform rather than argued.)

**Flow control flatters the device under test.** Ethernet flow control (802.3x PAUSE frames) lets an overwhelmed receiver throttle the sender at the MAC layer, invisibly. With it enabled, my forwarder measured 0.66% loss at 400,000 pps and looked heroic; it was actually being fed a paced stream because its NIC was quietly telling the generator to slow down. Disable PAUSE on the device under test, or you are measuring the receiver's mercy, not the code.

**The arrival pattern is part of the benchmark definition.** Same average rate, different shape: when the generator switched from a smooth stream to line-rate bursts of 64, the naive rung (still on its default socket buffer at the time) went from under 1% loss at 150,000 pps to almost 10%. Every number in the series is under the bursty profile, which is both the harder case and the closer one to real aggregated traffic. A benchmark that does not state its burst profile has not finished describing itself.

**The machine must hold still.** The NIC on the device under test runs with Energy-Efficient Ethernet, Green Ethernet, power-saving mode, and Gigabit Lite disabled, receive buffers at the driver maximum (4096), and interrupt moderation left on as the throughput-realistic default. The published configuration additionally has Defender real-time protection and the firewall disabled ([the batched-kernel chapter](../udp-ladder-batched-kernel/) prices what the filtering stack costs each engine family), runs Realtek driver 10.80.20.407 (a driver update mid-project silently reset three of the hygiene settings and measurably moved the segmented-send path, so the driver version is part of the configuration), and reports CPU as the median of a run's loaded seconds.

**A Windows UDP forwarder dies twice before it ships.** Two failures the firewall had been hiding surfaced the moment it was turned off. First, an inbound ICMP port-unreachable (a destination that is briefly unbound) surfaces as `ConnectionReset` on the forwarder's *next receive* and kills a naive loop; `SIO_UDP_CONNRESET` turns the behavior off, which is the honest semantic, since a UDP send has no delivery contract to break. Second, the current Realtek driver backpressures per-packet sends: at rates the previous driver absorbed, sends fail with `WSAENOBUFS`, fatal if unhandled and a counted drop if treated as what it is, transmit backpressure. Every engine in the repository now carries both bandages; the naive sample in the ladder's rung 1 shows neither, because the code everyone writes first has neither.

The moral of the whole catalogue fits in one line, and it is the ladder's closing rule too: **count at both ends, and treat any number you cannot cross-check as a rumor.**