two machines talking to each other
Copper. Glass. Sometimes radio. That is the entire physical inventory of networking. Everything else, Ethernet, IP, TCP, RDMA, NCCL, the whole 600-page tower of acronyms, is a way of dressing those three media so that a 1 you put in over here comes out as a 1 over there, with some guarantees about how often it doesn’t.
A computer stores everything as bits because transistors are switches and two states is the level count you can build billions of reliably. To move bits between machines you need the same three things humans needed for language: the data, a format both sides agree on, and a medium the signal can survive. Assign 0 volts to 0 and V volts to 1 and a wire becomes a sentence.
That paragraph is roughly where my understanding stopped being specific, and I work on a distributed-computing runtime that sits one layer above all of this. I keep catching myself saying things like “and then NCCL handles it” without being able to say what NCCL actually does on the wire. That’s a code smell. So this post is me following one byte the whole way, not the cartoon where chapter one is “the OSI model” and you nod along, but the version where I keep asking what is physically happening and refuse to move on until I can answer. If you know basic C and roughly what memory is, you have everything this post assumes; every other term gets defined when it first earns its keep.
Start at the bottom, with the dumbest question available.
What does it mean to send a 1?
I have a transistor on my side. The transistor’s output is either at supply voltage or at ground. If I run a wire from my output to your input, and you sample the wire’s voltage, you read my bit. Done.
That’s not done, is it. The wire has resistance. The wire has capacitance. The wire is sitting next to fifteen other wires that are all switching at the same time, and each one is radiating a tiny electromagnetic field that couples into mine. By the time my sharp edge reaches your end, the square wave I sent looks like a sad RC-charged exponential. Sometimes it looks like noise.
At low speeds, resistance is the main problem, and it’s boring: voltage drops over the wire, you measure a smaller voltage, you raise your threshold, life continues. At gigahertz rates the problem changes character. Current at high frequency doesn’t flow through the bulk of the wire, it pushes itself to the outer skin of the conductor. The skin effect. Effective cross-section collapses, effective resistance rises with √f, so at high rates copper gets more resistive, in a non-obvious way. This is why high-speed cables have weirdly exotic plating; the surface is doing all the work.
And then crosstalk. The wire next to me is a tiny antenna pointed at my wire. Two wires running parallel for a meter share mutual inductance, when that one toggles, it dumps a voltage spike onto mine. Multiply by everyone in the bundle. At 10Gbps you’d be eating noise off your neighbors constantly.
The fix is not to fight the noise. You design so the noise hits both sides of a pair equally and cancels. Twisted pair: two wires carrying the same signal but with opposite polarity, the receiver subtracts them. Any external noise couples into both wires roughly equally and disappears in the subtraction. The twist matters because over the length of one twist, each wire spends half the time on top and half on the bottom relative to whatever noise source is nearby, so even if the noise is non-uniform along the length, twisting averages it out. This trick is a century old, it’s hiding in the wall of every office, and it still carries most of the world’s traffic for the last few meters.
That gives me the cable category story. Cat5e: few twists per inch, good for 1Gbps. Cat6: tighter twists, internal separators between pairs to reduce crosstalk inside the cable, good for 10Gbps to ~55m. Cat6a: shielded foil, 10Gbps to 100m. Cat8: fully shielded, very tight twist rate, 25/40Gbps but only ~30m. Notice the pattern. Higher rate means tighter twists, more shielding, less distance. Physics is winning.
Which is why copper hits a wall around 10GbE for anything past a rack. The frequencies you need to push the data are high enough that the cable starts behaving like a low-pass filter, the skin effect dominates, your noise margin collapses, and you can either spend money on exotic copper or admit the photon won.
So I switch to photons. Glass doesn’t care about EMI (there’s no electromagnetic field around the cable next to mine to couple in) and it barely attenuates over kilometers if you pick the right wavelength. The bit becomes “is there light, or isn’t there light, at this wavelength right now.” A laser turns it on, a photodiode reads it on the other side.
Single-mode fiber versus multi-mode fiber comes down to core diameter. Multi-mode has a fat ~50µm core that lets light bounce down it at different angles. Each angle is a different “mode”, a different path the light can take through the core. The straight ray gets to the end first, the bouncy rays arrive later. Pulse spreads. This is modal dispersion. It limits how fast you can pulse, which limits distance. Single-mode is a tiny ~9µm core, small enough that only one mode propagates. No bouncing rays. No modal dispersion. You can run it tens of kilometers at 100Gbps with the right transceivers.
And then there’s the trick where you stuff multiple colors down the same fiber. DWDM, dense wavelength division multiplexing. Each color is its own bitstream. A receiver runs them through what is functionally a prism and separates them. A single fiber pair carrying 80 channels × 200Gbps = 16Tbps is not exotic anymore. The glass is dumb; the magic is the laser and the optical mux/demux at each end.
Which gets me to transceivers, the little modules with the laser, photodiode, and SerDes that you plug into the NIC. SFP at 1G, SFP+ at 10G, SFP28 at 25G, QSFP28 at 100G (four 25G lanes), QSFP-DD at 400G (eight 50G lanes), OSFP for 800G. The fiber doesn’t change between them. The transceiver does.
Wait. I jumped. I said “encode bits onto light” but I didn’t say how. How do you actually map bits to a waveform?
The simple version is NRZ, non-return-to-zero. Two voltage levels (or light intensity levels). 0V means 0, V means 1. The receiver samples once per bit period. This works up to a point. The point is when the bit rate gets close to the channel’s frequency limit. At 50Gbps NRZ you’d need ~25GHz of usable channel and copper just doesn’t have that past a few meters.
So you go to PAM4: four voltage levels instead of two, so each symbol now carries 2 bits. Same baud rate, double the bit rate. But the gap between adjacent levels is now V/3 instead of V, noise margin collapses by a factor of three. SNR drops by ~9.5dB. PAM4 is much more sensitive to noise.
This is the deal you make at 400G and 800G: you trade noise margin for bit rate because you’ve run out of frequency headroom. And to make the deal work, you bolt FEC on top.
Hold on, I should pin down “baud rate vs bit rate” because I always see engineers conflate them. Baud rate is symbols per second. Bit rate is bits per second. For NRZ they’re equal. For PAM4 the bit rate is 2× the baud rate. When someone says “100G QSFP28 NRZ” that’s 25 Gbaud × 4 lanes × 1 bit/symbol. When they say “400G QSFP-DD PAM4” that’s 50 Gbaud × 8 lanes × 2 bits/symbol. The wire is moving symbols at one rate, your bits are flowing at a different rate. Lose that distinction and your signal-integrity math will lie to you.
FEC, then. Because PAM4 eats noise margin, the receiver will get bits wrong sometimes. Forward Error Correction adds redundancy so the receiver can fix a bounded number of symbol errors per block without asking for a retransmit. Reed-Solomon RS(544, 514) is what 400G uses. The cost: latency. FEC blocks have to be received and decoded before they pass up. Maybe 100ns per hop. Small. In HPC every hundred nanoseconds matters and you can sometimes turn it off if your channel is clean enough.
Here’s a question that sounds dumb until you try to answer it. How does the receiver know where one bit ends and the next begins? There’s no clock wire on Ethernet. The transmitter has a clock, the receiver has a clock, and they are not the same clock. Their crystals drift. Without correction, the receiver samples at the wrong instant and reads garbage.
The fix: clock recovery from the data itself. The encoding has enough transitions that the receiver’s PLL can phase-lock onto the incoming signal. 8b/10b (used at 1G and 10G) maps every byte to a 10-bit symbol from a table that guarantees enough transitions and DC balance. Costs you 20% of your line rate as encoding tax. Painful but it works. At 25G and above the industry switched to 64b/66b, map 64 bits of payload to 66 bits on the wire, with a 2-bit header that always has a transition (either 01 or 10). Overhead drops to ~3%. The payload bits get scrambled (XORed with a known PRBS sequence) so even content like long runs of zeros end up with transitions on the wire. The receiver descrambles after recovery.
What happens when the PLL slips lock, bad cable, temperature drift, EMI burst? You get symbol errors. Symbol errors that flip a payload bit become CRC errors at the frame level. Symbol errors that fall outside the FEC budget become uncorrectable FEC errors. All of this is visible if you ask the NIC:
$ ethtool -S eth0 | grep -E 'error|drop|miss|fec|crc' | head -40
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
rx_crc_errors: 12
rx_missed_errors: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
fec_corrected_blocks_lane0: 4193827
fec_corrected_blocks_lane1: 4128612
fec_uncorrectable_blocks: 0(Exact field names depend on the NIC driver. mlx5 has its own naming, ice has another. Trust your driver’s man page, not mine.)
What to look for in that output: rx_crc_errors should be zero or near zero, every CRC error is a frame that arrived corrupted and got dropped. rx_missed_errors is the NIC saying it ran out of FIFO space, “I couldn’t even hand it to DMA.” Bad. fec_corrected_blocks will be huge and growing; that’s FEC doing its job silently. fec_uncorrectable_blocks should be zero, and if it isn’t, go look at the optics or the cable.
So there’s a chunk of metal+silicon called a NIC bridging the wire and the rest of the computer. What is it actually doing?
Roughly three things, separable on the die. There’s a PHY chip that handles the analog: SerDes, clock recovery, FEC, encoding. It eats voltages and emits bytes. There’s a MAC, Media Access Controller, that handles framing: it assembles bytes into frames, validates CRC, parses destination addresses, decides “this is mine, this isn’t.” Historically the MAC also did CSMA/CD for half-duplex collision avoidance; nobody runs half-duplex Ethernet for real work in 2026, so that part is vestigial. And then there’s the DMA engine plus ring-buffer logic, which is the part that’s going to matter enormously when I get to RDMA.
The ring buffer is the unit of conversation between the OS and the NIC. The driver (the piece of kernel code that speaks this NIC’s dialect) maintains circular queues of descriptors: small structures, 16-32 bytes typically, each pointing at a buffer in host memory. On receive, the NIC pulls the next descriptor, finds the buffer address, DMAs the packet bytes into it, and marks the descriptor as filled. (DMA, direct memory access: the device writes host RAM itself, over PCIe, the bus that connects expansion cards to the CPU; no CPU instruction touches the bytes.) On transmit, the driver writes a descriptor pointing at outbound data, kicks a doorbell register, and the NIC DMAs the data out and sends it.
Notice: the packet bytes don’t sit on the NIC. They flow through a small FIFO and go straight to host memory via PCIe DMA. The NIC is, mostly, a translator between wire format and bus format.
Now a packet arriving, the whole sequence, because the RDMA argument later stands on it.
Photon hits photodiode. Diode produces current. TIA amplifies. DSP equalizes. PHY recovers the clock, descrambles, decodes 64b/66b, runs FEC, produces bytes. MAC accumulates bytes, finds the frame boundary, validates the CRC, checks the destination MAC. If it’s ours (or broadcast, or a multicast we joined), keep. Otherwise drop. The classifier inspects fields (the 5-tuple usually) to decide which Rx ring this packet belongs in (this is RSS, I’ll come back). The NIC pops a descriptor from that ring, reads the host physical address, DMAs the packet into host memory, marks the descriptor filled, fires an interrupt, the hardware’s doorbell, forcing the CPU to suspend whatever it was doing and run driver code. The kernel (the part of the OS that owns all hardware and memory; your program lives outside it, in userspace) wakes a NAPI poll. NAPI walks the ring, allocates an sk_buff for each filled descriptor (which may or may not involve copying the bytes again, depending on driver). The protocol stack runs: L2 demux → L3 demux → L4 demux → socket queue. The application, blocked in recv() or selected via epoll, wakes up. recv() copies the data from the socket buffer to userspace.
Now count. The data touched memory at: NIC DMA into a host page (1 transfer over PCIe), maybe an internal sk_buff allocation copy, the socket-buffer-to-user copy. So one or two CPU-driven copies per packet, plus the DMA. And count the boundary crossings: an interrupt context, a softirq context, and a syscall, the controlled gate a program walks through to ask the kernel for anything, recv() included, each crossing costing a mode switch both ways.
At 100Gbps line rate with 1500-byte frames, about 8.3 million packets per second, every packet pays this. I’m going to come back to that number multiple times. It’s the entire reason RDMA exists.
The naive thing, one interrupt per packet, falls apart immediately. 8 million IRQs per second per core leaves no time for anything else. So Linux has NAPI: first packet on a quiet ring fires an interrupt, the interrupt handler schedules a soft-IRQ that polls the ring for up to a budget of packets (netdev_budget, default 300). While polling, interrupts on that ring are masked. Drain the ring, re-enable interrupts. Under load you effectively transition from interrupt-driven to poll-driven. Per-packet interrupt cost vanishes.
And there’s interrupt coalescing, configurable in the NIC. Don’t fire an IRQ until N packets have accumulated or T microseconds have elapsed:
$ ethtool -c eth0
rx-usecs: 50
rx-frames: 64
tx-usecs: 50
tx-frames: 64
adaptive-rx: on
adaptive-tx: onThis is the throughput-vs-latency dial. rx-usecs 100, rx-frames 256 says “wait up to 100µs or 256 packets, whichever comes first.” Great for batch throughput. Adds 100µs of receive latency on the floor. HPC turns this off, rx-usecs 0 rx-frames 1, and eats the CPU cost in exchange for microsecond-class latency. Or skips the whole question by going RDMA.
RSS, which I keep mentioning: Receive Side Scaling. A single Rx queue handled by a single CPU core caps out at whatever that core can do. So you split traffic across multiple queues and multiple cores. The NIC, in hardware, hashes the 5-tuple of each incoming packet and uses the hash to pick which Rx queue to deliver into. Each queue is bound to a CPU via interrupt affinity. The same flow always lands on the same queue, so a TCP connection’s packets get handled by one core in order, no out-of-order processing, no cross-core synchronization for that flow.
$ ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 63
TX: 63
Combined: 63
Current hardware settings:
RX: 0
TX: 0
Combined: 8Eight queues on this box. You’d set this with ethtool -L eth0 combined 8. Then /proc/interrupts should show eight separate rows for the NIC’s MSI-X vectors, each pinning to a different CPU column. There’s also RPS (the software version for dumb NICs) and RFS (steers a flow to the core where the application is actually reading). The reason any of this matters is cache locality and NUMA. (NUMA: big servers have two or more CPU sockets, each with its own directly-attached RAM; any core can reach any RAM, but reaching across sockets is slower.) A packet that lands on a core attached to NUMA node 0, copies into a socket buffer on node 0, and then gets recv’d by an application thread pinned to node 1, that data drags across the inter-socket interconnect. ~100ns extra latency, a fraction of local DRAM bandwidth. At 100Gbps it shows up. Aligning the NIC’s NUMA node, the IRQ affinity, and the application’s CPU pinning is the difference between a healthy and an unhealthy machine.
I’ve been saying frame this whole time like it’s obvious. It isn’t. An Ethernet frame, from front to back: 7 bytes of preamble (alternating 10101010 for sync), 1 byte of start-frame-delimiter (10101011 to mark “real bits start now”), 6 bytes of destination MAC, 6 bytes of source MAC, 2 bytes of EtherType (which says what’s inside: 0x0800 for IPv4, 0x86DD for IPv6, 0x0806 for ARP, 0x8100 for “VLAN tag follows”), then 46 to 1500 bytes of payload, then 4 bytes of FCS, a CRC-32 over the frame that the NIC computes on transmit and validates on receive. Mismatch, drop. After the FCS there’s a mandatory interframe gap, 96 bit-times of silence, ~1ns on 100GbE.
Why 46 minimum? Historical collision-detection physics required a minimum frame size. Why 1500 maximum? Historical Ethernet hardware limits. Both numbers are tradition; the wire underneath has no opinion about them.
So the destination MAC tells the next hop “this is for you.” But where did I get a MAC from? I have an IP. I have, say, 10.0.0.7. I need to put a 6-byte hardware address in the Ethernet header. There’s no IP-to-MAC field anywhere.
This is what ARP solves. Address Resolution Protocol. Beautifully simple. Host A wants to send to 10.0.0.7. Looks in its ARP cache. Not there. Host A broadcasts an ARP request: “Who has 10.0.0.7? Tell 10.0.0.5.” Sent to the broadcast MAC FF:FF:FF:FF:FF:FF. Every host on the segment receives it. The host that owns 10.0.0.7 replies, unicast, with its MAC. Host A caches the binding and now it can build the Ethernet header.
$ ip neigh show
10.0.0.1 dev eth0 lladdr a0:36:9f:00:11:22 REACHABLE
10.0.0.7 dev eth0 lladdr b8:ce:f6:01:02:03 STALESTALE means past TTL but still in the table; the next packet triggers a refresh. There’s no authentication in ARP, anyone on the broadcast domain can reply to anyone’s request. ARP spoofing exists exactly because of this. Everything at L2 trusts everything else at L2, which is exactly why VLANs and microsegmentation exist.
Sidebar: what’s a VLAN, then? The 802.1Q tag is 4 bytes wedged between the source MAC and the EtherType. 12 bits of VLAN ID gives you 4094 usable VLANs. 3 bits of PCP for priority (DSCP’s L2 cousin). A switch port is either access (untagged; the host doesn’t know about VLANs) or trunk (tagged, used between switches). Datacenters use VLANs for isolation, safety, and broadcast-domain reduction. 4094 isn’t enough for cloud scale, which is why VXLAN exists (VLAN inside UDP, 24 bits of VNI), and which I’ll come back to indirectly when I talk about RoCE.
How big can the frame be? The MTU (maximum transmission unit, the largest payload one frame may carry) is 1500 bytes by default, the same 1980s relic. At 100Gbps that costs real money: 100e9 / (1500 × 8) is ~8.3 million packets per second. Every packet pays per-packet costs: header processing, descriptor allocation, interrupt cost, sk_buff allocation, TCP segmentation, ACK generation, route cache lookups. Per-packet cost in modern Linux is on the order of a microsecond of CPU per packet, times 8 million per second, and you’re spending most of your CPU shoveling packets instead of doing work.
Jumbo frames raise the MTU to 9000 bytes. Now 100Gbps is ~1.39M pps. Six times fewer packets. Header overhead drops from ~3% to ~0.6%. You set it with sudo ip link set eth0 mtu 9000. The catch, and this is where it bites you, every hop on the path must agree. A switch in the middle still on 1500 will eat your jumbos or fragment them, depending on configuration. Path MTU Discovery is supposed to handle this automatically, but it’s been famously broken since the 90s because ICMP-filtering firewalls eat the back-message and the source just sees its packets disappear silently. The hack everyone uses is MSS clamping at the edges and just hoping.
One more thing about frames before I move up. Flow control. The link-layer can backpressure. The simplest form is 802.3x PAUSE: receiver buffer fills, receiver sends a PAUSE frame with “stop for N quanta,” sender stops on that link until the time runs out. Crude, it blocks all traffic on the link. Priority Flow Control (PFC, 802.1Qbb) refines this: pause per traffic class, not the whole link. Pause your RDMA class without pausing your management traffic. PFC is essential for RDMA over Ethernet, because RDMA needs lossless and TCP can tolerate loss but RDMA can’t. I’m getting ahead of myself; I’ll be back here.
Before I leave the frame: I don’t trust bytes I haven’t built. Raw socket in C, broadcast destination, custom EtherType so it doesn’t conflict with anything:
// raw_eth_send.c — build and send an Ethernet frame by hand.
// Compile: gcc raw_eth_send.c -o raw_eth_send
// Run: sudo ./raw_eth_send eth0
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <net/if.h>
#include <netinet/ether.h>
#include <linux/if_packet.h>
#include <arpa/inet.h>
int main(int argc, char **argv) {
if (argc < 2) { fprintf(stderr, "usage: %s <iface>\n", argv[0]); return 1; }
const char *ifname = argv[1];
int sock = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL));
if (sock < 0) { perror("socket"); return 1; }
struct ifreq ifr = {0};
strncpy(ifr.ifr_name, ifname, IFNAMSIZ - 1);
if (ioctl(sock, SIOCGIFINDEX, &ifr) < 0) { perror("SIOCGIFINDEX"); return 1; }
int ifindex = ifr.ifr_ifindex;
if (ioctl(sock, SIOCGIFHWADDR, &ifr) < 0) { perror("SIOCGIFHWADDR"); return 1; }
unsigned char src_mac[6];
memcpy(src_mac, ifr.ifr_hwaddr.sa_data, 6);
unsigned char dst_mac[6] = {0xff,0xff,0xff,0xff,0xff,0xff}; // broadcast
unsigned char frame[64] = {0};
memcpy(frame, dst_mac, 6);
memcpy(frame + 6, src_mac, 6);
frame[12] = 0x88; frame[13] = 0xb5; // EtherType: experimental local
memcpy(frame + 14, "HELLO_FROM_RAW_SOCKET", 21);
// FCS is appended automatically by the NIC.
struct sockaddr_ll dest = {0};
dest.sll_family = AF_PACKET;
dest.sll_protocol = htons(0x88b5);
dest.sll_ifindex = ifindex;
dest.sll_halen = 6;
memcpy(dest.sll_addr, dst_mac, 6);
ssize_t n = sendto(sock, frame, 60, 0, (struct sockaddr*)&dest, sizeof(dest));
if (n < 0) perror("sendto");
else printf("sent %zd bytes\n", n);
return 0;
}Open Wireshark on eth0, filter eth.type == 0x88b5, run the program with sudo, you’ll see one frame appear with the broadcast destination, your interface’s MAC, and the payload HELLO_FROM_RAW_SOCKET. FCS is added by the NIC; preamble/SFD is added by the PHY and never shows up in your buffer or in Wireshark. This is also useful for understanding what isn’t happening, no IP, no port, no checksum above L2, no ACK. You hand the NIC bytes; the NIC puts them on the wire. That’s it. You’re below TCP, below IP, below everything most people call “the network.”
A frame gets me across one room. To reach a server in Frankfurt I need a layer that knows about geography, because L2 doesn’t route, a MAC address names one specific NIC and has no notion of hops.
That’s IP. The IPv4 header is 20 bytes minimum, and most of its fields are bookkeeping you can look up when you need them. Four of them actually matter to this post.
TTL is misnamed, it’s a hop counter, not a time. Each router decrements it by 1; at 0, drop and send ICMP “TTL exceeded” back to the source. This is the entire trick behind traceroute: send packets with TTL=1, the first router drops and tells you who it is; then 2, then 3, and so on.
The protocol number lets the receiver demux into TCP (6), UDP (17), ICMP (1), whatever else, it says which parser gets the payload.
The header checksum is over the header only, not the payload. The payload’s integrity is the responsibility of L4 (TCP/UDP checksum) and L2 (FCS). Annoyingly the header checksum gets recomputed at every router because TTL changes. IPv6 dropped the header checksum entirely for exactly this reason, let L2 and L4 handle integrity, free the router from per-hop checksum work.
The ECN bits, two bits inside the old ToS byte, are how routers can signal congestion without dropping the packet. ECT(0), ECT(1), or CE. If both endpoints advertise ECN support, they send with ECT set. A router with a full queue can flip the bits to CE instead of dropping. The receiver echoes the mark back. The sender backs off. For TCP this is a minor optimization. For RDMA over Ethernet it’s essential, DCQCN lives or dies by ECN, and I’ll be back here.
Fragmentation. If a packet is bigger than the next link’s MTU and DF (Don’t Fragment) isn’t set, the router fragments, duplicates the header, splits the payload, the destination reassembles using the Identification field and Fragment Offset. This is, charitably, a 1980s design that has aged poorly. The reassembly buffer is per-IP-pair-per-ID, vulnerable to attackers crafting fragments to exhaust your memory. A dropped fragment makes every other fragment useless until timeout. Reassembly defeats per-flow ECMP because intermediate routers see only the first fragment’s L4 header. Modern networks set DF and rely on PMTUD, which as I said is broken whenever ICMP is filtered. Don’t break ICMP unless you mean it.
Subnetting and CIDR: the mask is a bitmask, the first N bits are network, the remaining bits are host. 10.0.0.0/24 is 256 addresses, 254 usable. 10.0.0.0/8 is 16M addresses. The routing table is consulted via longest-prefix match: if you have routes for 10.0.0.0/8 and 10.1.2.0/24, a packet for 10.1.2.42 matches both, /24 wins because it’s more specific. This is why route order in the config doesn’t matter at lookup time; the data structure (trie, or hardware TCAM) ranks by prefix length.
$ ip route get 8.8.8.8
8.8.8.8 via 10.0.0.1 dev eth0 src 10.0.0.42 uid 1000That ip route get is the command to remember. Tells you exactly which route, gateway, source IP, and interface the kernel would pick. Saves a lot of guessing.
So the destination IP determines the route, the route determines the next-hop gateway IP, ARP gives me the MAC for that gateway, and I send the frame. The gateway is a router. It decrements TTL, looks up the destination IP in its routing table, picks its next hop, rewrites the MAC, forwards. The IP packet survives end-to-end; the Ethernet header gets rewritten at every router.
I keep saying “router” and “switch.” What’s actually different? A switch operates within one broadcast domain: looks at the destination MAC, looks up its CAM table, forwards to the right port (or floods if it doesn’t know). Does not decrement TTL. Does not rewrite headers. A router operates between broadcast domains: looks at the destination IP, looks up its FIB, decrements TTL, rewrites the destination MAC (and the source MAC to its own egress port), forwards. In modern hardware the distinction blurs; most datacenter “switches” are L3 switches whose ASICs do both. The line is mostly about feature set (BGP, OSPF, deep ACLs).
There’s a vocabulary distinction between the RIB (Routing Information Base: everything the control plane has learned from BGP, OSPF, static config) and the FIB (Forwarding Information Base: the best route per prefix installed in hardware). The data path consults the FIB. Lookups happen at line rate, typically via TCAM, ternary content-addressable memory that can match wildcards. Longest-prefix match in TCAM is approximately O(1).
Okay, but what if I have two equally-good paths to a destination? ECMP. Hash some tuple, usually the L4 5-tuple, and use the hash to pick a path per flow. Same connection always takes the same path (preserves ordering). Great for aggregate bandwidth, bad for single-flow performance. If you’ve ever stared at iperf3 with a single TCP stream maxing at 25Gbps on a 100Gbps fabric, the answer is probably ECMP and you fix it by running parallel streams (-P 8) so the hash spreads them. Hash collisions (“flow polarization”) cause N flows to all land on one path while others sit idle. Cure: flowlet switching, burst-aware path selection that lets you switch paths within a flow during gaps too large to cause reordering. CONGA, LetFlow, HULA are the academic names.
Big enough datacenter, what does the network actually look like? Old enterprise networks were three-tier: access → aggregation → core. North-south traffic dominated, the core bottleneck was tolerable. ML training traffic is east-west, GPU 0 in rack A talking to GPU 0 in rack B, all-to-all gradient exchange across hundreds of nodes simultaneously. Three-tier breaks immediately. Aggregation becomes a hot spot, the core becomes a parking lot. Enter spine-leaf, a two-tier Clos topology: every leaf switch connects to every spine switch. Every server connects to a leaf (its top-of-rack). To get from server A on leaf L1 to server B on leaf L2: server A → L1 → some spine S → L2 → server B. Exactly three hops, always. ECMP spreads traffic across all spines. The fabric is “non-blocking” if leaf uplinks total at least as much bandwidth as leaf downlinks (1:1 oversubscription).
Oversubscription is the ratio of downlink to uplink bandwidth. 4:1 means servers can collectively offer 4× the bandwidth the fabric can carry, fine for general workloads where most servers are idle most of the time, terrible for HPC where everyone is at line rate simultaneously. ML training demands non-blocking 1:1, which is neither negotiable nor cheap: a 1024-GPU cluster with non-blocking 400G fabric needs a lot of spine.
Even a non-blocking fabric isn’t immune to incast: 100 senders all blasting one receiver at once, the offered load is 100×, the link can take 1×, buffer fills, packets drop, TCP backs off, retransmits, tail latency explodes. Switch buffer sizing is a religious war: shallow buffers (Tomahawk class, tens of MB) are low-latency but drop early under incast; deep buffers (Jericho class, gigabytes) absorb incast but add queueing latency. For training fabrics I think the shallow camp is simply right, incast there is not weather, it’s the workload, and the answer to “the workload overflows my buffer” has to be congestion control, not a bigger bucket that converts drops into queueing delay on every other flow. Deep buffers make sense where you can’t control the endpoints. (There’s a nastier failure lurking in this paragraph, PFC storms, but it needs machinery I haven’t built yet; it’s waiting in the RoCE section.)
Here’s a real one, run from the machine I’m writing this on, home connection, not a datacenter, and all the uglier and more instructive for it:
$ traceroute -n -q 3 -m 12 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 12 hops max, 40 byte packets
1 192.168.29.1 2.778 ms 3.589 ms 2.726 ms
2 10.247.56.1 5.079 ms 5.163 ms 6.061 ms
3 172.26.55.0 10.452 ms 11.691 ms 10.678 ms
4 192.168.143.238 10.462 ms
192.168.143.234 9.645 ms 10.056 ms
5 172.17.88.43 9.317 ms 9.303 ms 9.636 ms
6 172.27.249.66 10.173 ms
172.27.249.67 10.009 ms 9.899 ms
7 192.168.140.188 10.212 ms 11.028 ms
192.168.140.182 10.028 ms
8 * * *
9 * * *
10 * * *There’s half the post in that dump. Hops 1-7 are all private address space: my router, then my ISP’s carrier-grade NAT layers, RFC1918 addresses six hops deep into their network. Hops 4, 6, and 7 each show two responding addresses: that’s ECMP, live, successive probes hashed onto different parallel paths. And from hop 8 onward, * * *: some middlebox is eating the ICMP TTL-exceeded replies, exactly the filtering I complained about when I said PMTUD is broken. Google’s edge is in there somewhere, invisible, silently forwarding. For sustained path analysis use mtr --report -c 100 8.8.8.8, it runs this 100 times and reports loss / avg / best / worst / stdev per hop, which is how you catch “hop 4 drops 0.3% of packets,” the kind of thing that wrecks p99 latency without showing up in ping.
So I can ship a packet across the planet. Now: two processes on two machines need to talk. The IP layer gives me a path. It doesn’t give me ports, or reliability, or ordering. I need a layer above IP that says “this packet is for the process listening on port X.”
The minimum thing on top of IP is UDP, eight bytes of header: source port (2), destination port (2), length (2), checksum (2, optional in IPv4!). That’s it. No connection. No ordering. No retransmit. No congestion control. No flow control. You hand the kernel a buffer, it goes on the wire (or doesn’t, if the buffer’s full). If it gets dropped you don’t know. If it arrives twice you get it twice. If two packets arrive out of order, you get them out of order.
When does UDP make sense? When you’re sending a single packet that’s either small or self-contained. DNS: single packet most of the time, opening a TCP connection just to ask “what’s the IP for example.com” would be insane. DHCP and NTP, same shape. Video and gaming where a lost frame is less painful than a frozen stream; don’t wait for the retransmit of a frame already obsolete. UDP plus application-level FEC and packet pacing is the standard recipe. QUIC is the modern outlier: HTTP/3 is UDP underneath, with a fully reliable, ordered, congestion-controlled, multiplexed stream protocol reimplemented in userspace on top of UDP. The reason is largely to escape kernel TCP’s ossified head-of-line blocking and slow rollout of changes. The cost is reimplementing every wheel TCP has been refining for 40 years.
When does UDP not make sense? When you need reliability and you’re not willing to build it yourself. You can write a UDP echo server and blast it from a client faster than the server can drain, and watch the drops:
$ ss -u -n -m
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 524288 0 0.0.0.0:9999 0.0.0.0:*Recv-Q at the buffer size means the socket is full and packets are being dropped silently. There’s no backpressure to the sender, UDP. The dropped-packet counter is in /proc/net/snmp under Udp.InErrors, Udp.RcvbufErrors.
So if I want reliability without writing it myself, I use TCP. And TCP is what most people pick because TCP has spent four decades getting beaten on.
What TCP gives me: reliability, ordering, flow control, congestion control, connection state. Its header is 20 bytes minimum, ports, sequence number, acknowledgment number, flags, window, checksum. Two fields in there look small and aren’t. The sequence number is a byte-stream offset, not a packet number. The first byte of the connection has some initial sequence number (chosen randomly at SYN time to make spoofing harder); every subsequent byte gets the next number. A segment carrying 1000 bytes starting at seq 12345 covers bytes 12345 through 13344 of the stream. The acknowledgment number is “next byte I expect.” Cumulative ACK: if I ACK 13345 I’m saying “I have everything up to byte 13344.” That’s why a single ACK can acknowledge megabytes.
The window field is two bytes, so at most 65535. In 1981 that was generous. In 2026 on a 100Gbps link with 1ms RTT, the bandwidth-delay product is 12.5 MB and 64KB is nowhere near enough to fill the pipe. The fix is the Window Scale option negotiated in the SYN, a multiplier up to 2^14, giving an effective window of ~1GB. You should always have window scaling on. Modern Linux does.
The lifecycle is the three-way handshake (SYN, SYN-ACK, ACK) to open, and a four-way FIN dance to close. The side that sends the first FIN sits in TIME_WAIT for 2×MSL (60s on Linux by default) so that straggling packets from the connection don’t get misinterpreted by a future connection on the same 4-tuple. TIME_WAIT is the source of port exhaustion at scale: a reverse proxy making millions of short outbound connections per second runs out of ephemeral source ports because they’re all stuck in TIME_WAIT. Mitigations: increase the ephemeral port range, increase the source IP pool, use SO_REUSEADDR/SO_REUSEPORT, or keep connections long-lived.
How does TCP actually achieve reliability? The sender keeps a list of unACKed bytes. When it sends a segment, it starts a per-segment timer (RTO). If the ACK doesn’t arrive before RTO, retransmit. RTO is calculated from RTT measurements using Jacobson’s algorithm: track a smoothed RTT and the variance, set RTO = SRTT + 4×RTTVAR, clamped to a minimum (200ms on Linux by default, which is the famous “200ms tail latency cliff”). Fast retransmit: if the sender sees 3 duplicate ACKs in a row (the receiver kept ACKing the same byte because something past it was dropped), assume loss and retransmit immediately. SACK, selective ACK, lets the receiver say “I have non-contiguous block [a,b]” so the sender doesn’t have to retransmit a whole window past the loss.
Flow control vs congestion control. People conflate these. Different things. Flow control is between two endpoints: the receiver advertises a window in every segment (“I have N more bytes of room”), the sender won’t send beyond last_acked + window. Protects the receiver. Congestion control is between many endpoints sharing a path, the sender maintains its own cwnd that limits how much it has in flight, and adjusts it based on inferred network state. Protects the network.
Slow start: begin at one MSS (or 10 with IW10), every ACK increases cwnd by one MSS. cwnd doubles each RTT. This continues until cwnd reaches ssthresh or a loss is detected. Congestion avoidance past ssthresh: linear increase, one MSS per RTT. On loss: ssthresh = cwnd/2, cwnd = ssthresh (or 1, for RTO-detected loss). Restart growth from there. This is Reno. Linux’s default is CUBIC, which replaces linear growth with a cubic function of time-since-last-loss, ramps up quickly to where it was before, then settles. Better on high-BDP links where Reno’s linear ramp takes forever to refill a 1GB window. BBR is the model-based alternative, estimates the bottleneck bandwidth and min RTT, paces packets to fill the pipe without overflowing buffers. Doesn’t react to packet loss as a primary signal. Great when loss isn’t a congestion signal (wireless, FEC-protected fiber, RoCE). Controversial when competing with loss-based flows for the same buffer.
$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic
$ sysctl -w net.ipv4.tcp_congestion_control=bbrss -tin will show cwnd, ssthresh, rtt, rttvar, pacing_rate, algorithm name per connection. That’s where I go when a TCP connection is mysteriously slow.
Now the classic trap: Nagle plus delayed ACK. Each algorithm is individually reasonable. Combined they’re vicious. Nagle: if there’s already an unACKed small segment in flight, buffer subsequent small writes until either the ACK arrives or a full segment accumulates. Optimization for telnet-era workloads, coalesces a stream of single-byte writes. Delayed ACK: when receiving data, don’t ACK immediately, wait up to 40ms hoping to piggyback the ACK on outgoing data or coalesce ACKs for multiple segments.
What happens with both on, and an app that does small writes? Client writes byte 1; Nagle sends. Server receives; the delayed ACK timer starts (no outgoing data to piggyback on). Client wants to write byte 2; Nagle buffers it, because the previous segment hasn’t been ACKed. 40ms later the server’s delayed ACK finally fires, the ACK arrives, the client sends byte 2, repeat. 40ms per small write. RPC latency becomes 40ms × number of round trips. The cure is TCP_NODELAY: setsockopt(sock, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof(int)). One line. Always set it for HPC, ML, RPC. Default-on Nagle is right for most apps; the moment you care about latency, kill it.
One more block before I move up: TCP offload. A modern NIC can do the segmentation, the receive coalescing, the checksums. TSO (TCP Segmentation Offload): the kernel hands a 64KB buffer with a single TCP header template to the NIC, NIC segments into MTU-sized pieces and fixes up sequence numbers and checksums. CPU did one segment’s work instead of 40. GRO (Generic Receive Offload): on receive, the NIC or driver coalesces consecutive segments of one flow into one giant pseudo-packet before handing to the kernel. Kernel processes one packet instead of 40. Checksum offload: the NIC computes and validates checksums. Without these, 100Gbps would be torturing a CPU socket. With them, it’s manageable. One catch: tcpdump shows you the offloaded giant packets, which can be confusing, your single TCP segment might be 64KB long in the capture, larger than the MTU, because that’s what tcpdump sees before/after the offload step. ethtool -K eth0 gro off if you need to see wire-real packets.
Here’s the smallest TCP server worth measuring against, accept one connection, drain and discard:
// tcp_server.c — accept one connection, drain and discard.
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
int main(int argc, char **argv) {
int port = atoi(argv[1]);
int srv = socket(AF_INET, SOCK_STREAM, 0);
int yes = 1;
setsockopt(srv, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes));
struct sockaddr_in sa = { .sin_family = AF_INET,
.sin_port = htons(port),
.sin_addr.s_addr = INADDR_ANY };
bind(srv, (struct sockaddr*)&sa, sizeof(sa));
listen(srv, 16);
int c = accept(srv, NULL, NULL);
setsockopt(c, IPPROTO_TCP, TCP_NODELAY, &yes, sizeof(yes));
char buf[1 << 20];
ssize_t n;
while ((n = recv(c, buf, sizeof(buf), 0)) > 0) { /* discard */ }
return 0;
}Mid-transfer, ss -tin shows you the connection’s internal state. I don’t have a datacenter in my apartment, so this block is reconstructed from what a healthy jumbo-frame 10G transfer looks like on an mlx5 NIC, read it for the field names, not the digits:
$ ss -tin
ESTAB 0 0 10.0.0.42:51234 10.0.0.7:9000
cubic wscale:14,14 rto:204 rtt:0.221/0.05 mss:8948 pmtu:9000
rcvmss:8948 advmss:8948 cwnd:1486 ssthresh:1402
bytes_sent:8421384192 segs_out:941832 retrans:0/0
send 9414227197bps delivery_rate 9410112854bpscwnd 1486 × mss 8948 ≈ 13MB window. rtt 0.221ms, sub-millisecond intra-DC. delivery rate ~9.4Gbps, line rate on a 10G link. cubic. If retrans were nonzero I’d go look at the network before touching the application.
On a clean intra-DC 100G link with default everything, the shape you can expect: iperf3 with 8 streams aggregates to ~94 Gbps while a single stream stalls around ~25 Gbps, because ECMP hashes one flow onto one path. sockperf ping-pong with a 64-byte payload: median RTT in the ~25µs range, p99 near ~45µs. (Representative shape, not my bench output; your hardware will move these by tens of percent.)
But why is intra-DC TCP latency tens of microseconds when the wire between two machines five meters apart is, optically, ~25ns one way? Where does 30µs come from?
The accounting: syscall to send, ~200ns. Kernel protocol stack send, 1-2µs. Qdisc plus driver, ~500ns. NIC processing and serialization, 200-500ns. Wire: 25ns. Remote NIC ingress plus DMA plus interrupt, 1-5µs depending on coalescing. NAPI poll plus protocol stack, 1-2µs. Wakeup plus syscall return, 1-5µs. App reads, generates reply, sends back: similar costs in reverse. Adds up to ~30µs comfortably, and the wire’s share of it is 25 nanoseconds. Everything else is the operating system doing its job.
I can actually demonstrate the OS’s share with zero networking hardware, because loopback (127.0.0.1, the interface that never leaves the machine) is the kernel stack with the wire amputated. A 64-byte TCP ping-pong against localhost on the laptop I’m writing this on (M4 MacBook Pro, macOS, TCP_NODELAY on, 100,000 iterations):
64B TCP ping-pong over 127.0.0.1, 100000 iters:
median 12.0 us p99 22.0 us min 7.0 us max 93.0 usTwelve microseconds round-trip with no wire at all. Two socket stacks, four syscalls, two wakeups per iteration, that’s what they cost, before a single photon gets involved. (macOS numbers; Linux loopback is usually somewhat faster. The point survives.)
So how do I make it not be the kernel?
This is the question that makes the whole rest of the post inevitable. Let me think about what the kernel is actually doing on the send path, because if I want to remove the kernel I should know what I’m removing.
I call send(fd, buf, 4096, 0). Syscall entry, user mode to kernel mode, save registers, switch stacks, look up the syscall number. ~100-300ns on modern x86 with mitigations on. Look up the file descriptor: fd → struct file * → struct socket * → struct sock *. Cache hits, but still derefs. Allocate an sk_buff. Copy the user buffer into the sk_buff via copy_from_user. That’s the copy. TCP segments if needed, adds its header (seq, ack, flags, window, checksum). IP looks up the route, picks an interface, adds the IP header. L2 layer looks up the destination MAC in the ARP cache, adds the Ethernet header. The sk_buff is enqueued in the qdisc. The qdisc dequeues into the NIC’s Tx ring, writes a descriptor pointing at the sk_buff’s data, writes the NIC’s doorbell register. NIC DMAs the data out.
Data-copy count on the send path: 1, the copy_from_user. Syscall count: 1. The DMA isn’t a CPU copy, the bus does it.
On the receive path: NIC DMAs into a pre-posted Rx buffer, marks the descriptor filled. Interrupt fires, hardirq is short: it schedules the NAPI softirq and masks further interrupts on the ring. NAPI poll drains up to budget descriptors, allocates an sk_buff for each, hands to the protocol stack. L2 → L3 → L4 demux. TCP processes: order check, ACK generation, fills socket receive queue. Wakes the app if it’s epoll-waiting. App calls recv (a syscall); copy_to_user runs. 1 copy in the hot path on modern zero-copy NAPI drivers. Plus 1 more if the driver does a defensive copy. Maybe 2.
Where does the CPU actually burn? Memory copies. A copy at line rate: 100Gbps is 12.5 GB/s. A modern server has maybe 200 GB/s of DRAM bandwidth. One copy at 12.5 GB/s is 6% of memory bandwidth. Not great but okay. Now consider 400Gbps. 50 GB/s. One copy is 25% of memory bandwidth. Two copies (send plus receive on the same box, or send via a userspace proxy) is 50%. You’ve stolen half your server’s memory bandwidth to move data, which is doing no useful work whatsoever. And NUMA makes it worse: if the NIC is on socket 0 and your app is pinned to socket 1, every copy crosses the inter-socket interconnect, which is slower and contended with cache coherence traffic.
$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 ... 31
node 0 size: 256 GB
node 1 cpus: 32 33 34 35 ... 63
node 1 size: 256 GB
node distances:
node 0 1
0: 10 21
1: 21 10The “distance 21” vs “10” is roughly 2× latency. You want your app threads, the NIC IRQs, and the memory the NIC DMAs into all on the same node. numactl --cpunodebind=0 --membind=0 ./my_app, plus pin the NIC’s IRQs via /proc/irq/N/smp_affinity_list.
And the syscall itself isn’t free. On x86_64 with mitigations (KPTI for Meltdown, retpolines for Spectre v2, the various IBRS/eIBRS knobs), a bare syscall costs ~100-300ns. With KPTI specifically you pay for the page-table switch on every entry, partial TLB flush. For bulk transfers it doesn’t matter. For small-message workloads it dominates. If my message is 1µs of useful work and the syscall is 200ns, that’s 20% overhead from the syscall alone, before any kernel work.
perf record -F 99 -g -p $(pgrep my_tcp_app) -- sleep 30 on a saturated TCP server. Flamegraph. The functions eating cycles, roughly in order: __memcpy_avx_unaligned (that’s copy_to_user/copy_from_user), tcp_recvmsg_locked, tcp_transmit_skb, __alloc_skb, ip_finish_output2, __softirqd calls into NAPI, mlx5e_poll_rx_cq or whichever driver you have. The first one is the copy. You’re spending real CPU on moving data. The rest is protocol processing. The kernel is working. None of this is wasted, but all of it is spent on the path.
So: what if the application talked to the NIC directly? No syscall. No copy. No kernel.
There are two ways to do that. DPDK: a userspace process owns the NIC, polls the Rx ring in a busy loop, runs the protocol stack itself, huge pages, CPU pinning. Used heavily for packet processors, load balancers, DDoS scrubbers, things that don’t need a full TCP stack and benefit from bypassing the kernel entirely. The other is RDMA: kernel bypass plus hardware-enforced reliability. The transport state machine runs on the NIC itself. The application registers memory, posts work requests, polls for completions. Kernel is involved only in setup.
I’m going to take the RDMA path because that’s where ML training networks actually live.
So what does RDMA look like from the API up?
It’s so different from sockets that the first time I read the verbs API I kept scanning for the recv() call, certain I’d missed it. There isn’t one. There’s no send(). There’s no socket. There’s no port. The model is:
I register memory with the HCA (Host Channel Adapter, the RDMA NIC). I create Queue Pairs: pairs of work-request queues, one for sends and one for receives. I post Work Requests describing what I want the HCA to do. The HCA executes them and writes Completion Queue Entries to a CQ. I poll the CQ for completions.
The HCA is talked to via mapped memory and doorbell registers. The kernel is involved only when I set things up, ibv_reg_mr, ibv_create_qp, ibv_modify_qp go through the kernel because they touch hardware state. After that, posting WRs and polling CQs is all userspace.
Memory registration is the gate. Before the HCA can DMA my application memory, I have to register it. ibv_reg_mr(pd, buf, size, IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE | ...). The kernel pins the pages (no swapping, the HCA has the physical addresses; you can’t have the OS swap out a page from under it), walks the page tables to find physical addresses, uploads a translation table to the HCA’s MTT (Memory Translation Table), and returns two keys: an lkey for local QPs and an rkey to share with peers so they can RDMA Write/Read into your memory. Registration is expensive: TLB flushing, page-table walks, MTT updates. Do it once during setup, reuse for many transfers.
What can the HCA actually do? Three operation families.
RDMA Write: the initiator says “take these N bytes from (local_addr, lkey) and put them at (remote_addr, rkey) on QP X.” The local HCA DMAs the bytes out of local memory; the remote HCA DMAs them into remote memory. The remote CPU does not get an interrupt. Does not execute any code for this transfer. The remote application doesn’t know it happened unless it polls the destination memory or you send a separate notification.
RDMA Read: the initiator pulls bytes from remote memory. Same property; the remote CPU isn’t involved.
Send/Recv: two-sided. Sender posts a SEND WR; receiver must have pre-posted a RECV WR with a buffer. The HCAs match and deliver. Used for control messages.
Let me actually trace the bytes through a one-sided RDMA Write, because if I can trace them I’ll believe the claim.
App thread has buf[0..N] at virtual address va_src, in a registered region with lkey L. App builds an ibv_send_wr: opcode=WRITE, local SGE=(va_src, N, L), remote=(va_dst, R), QP=X. Calls ibv_post_send. That walks the WR list, formats descriptor entries into the QP’s send queue (memory-mapped to the application; the doorbell ring buffer is in process address space), and writes the doorbell register (a single MMIO write) to tell the HCA “new work on QP X.” HCA reads the descriptor via DMA from the QP context in host memory. Translates va_src to physical pages using the MTT (built during reg_mr). Walks the page list. DMAs the data out of host memory directly. No CPU copy, no copy_from_user, no sk_buff. HCA segments into IB-MTU chunks (typically 4096 bytes). Each chunk gets BTH (Base Transport Header) + RETH (RDMA Extended Transport Header) + payload + ICRC. Packets serialize onto the link. Remote HCA receives, validates ICRC, parses BTH, WRITE on QP X’, parses RETH for destination virtual address, length, rkey. Translates via its own MTT (entries previously installed when the remote app registered the same region with the matching rkey). DMAs the payload directly into the remote application’s memory. Last packet of the WRITE triggers ACK; local HCA posts a CQE.
Copy count: zero. The data lived in the application’s registered buffer the entire time. Syscall count: zero (after setup). Remote CPU cycles: zero.
This is the thing. This is what makes RDMA worth the architectural pain.
How do I know it completed? Two options: busy polling (spin on ibv_poll_cq until a completion appears) or event-driven (ibv_req_notify_cq arms the CQ to fire an event, ibv_get_cq_event blocks on an fd until then). Polling burns a CPU core, zero notification latency. Events save the CPU, add latency. HPC where every microsecond matters: spin polling, always. The CPU core is cheap compared to the work it’s gating. IBV_SEND_SIGNALED controls whether each WR generates a CQE, post N unsignaled WRs followed by 1 signaled, only the last produces a completion. Cuts CQ pressure when batching.
There’s an inline-data optimization for tiny payloads (≤64 bytes typically): the data is embedded in the WR descriptor itself, HCA reads from the descriptor and never DMAs your buffer. Saves a DMA round trip, drops latency floor for sub-64-byte messages to ~1µs even on a real network. And there are atomics, IBV_WR_ATOMIC_CMP_AND_SWP and FETCH_AND_ADD execute atomic operations on remote 64-bit values, with atomicity guaranteed on the remote HCA between concurrent RDMA atomics on the same QP. Used for distributed counters, distributed locks, the small-message coordination at the core of some AllReduce implementations.
I want a working example. Smallest demonstrably-real RDMA program that shows every step. About 300 lines, omitting some error handling for length:
// rdma_write.c — minimal RDMA Write demo.
// Build: gcc rdma_write.c -libverbs -o rdma_write
// Server: ./rdma_write
// Client: ./rdma_write <server-ip>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <inttypes.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <infiniband/verbs.h>
#define PORT 19875
#define BUF_SIZE (1 << 20) // 1 MiB
struct qp_info {
uint32_t qp_num;
uint16_t lid;
uint8_t gid[16];
uint64_t addr;
uint32_t rkey;
};
static void die(const char *m) { perror(m); exit(1); }
static int tcp_exchange(const char *peer, struct qp_info *local, struct qp_info *remote) {
int sock;
if (peer) {
struct sockaddr_in sa = { .sin_family = AF_INET, .sin_port = htons(PORT) };
inet_pton(AF_INET, peer, &sa.sin_addr);
sock = socket(AF_INET, SOCK_STREAM, 0);
if (connect(sock, (struct sockaddr*)&sa, sizeof(sa)) < 0) die("connect");
} else {
int srv = socket(AF_INET, SOCK_STREAM, 0);
int yes = 1; setsockopt(srv, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes));
struct sockaddr_in sa = { .sin_family = AF_INET, .sin_port = htons(PORT),
.sin_addr.s_addr = INADDR_ANY };
bind(srv, (struct sockaddr*)&sa, sizeof(sa));
listen(srv, 1);
sock = accept(srv, NULL, NULL);
close(srv);
}
if (write(sock, local, sizeof(*local)) != sizeof(*local)) die("write");
if (read(sock, remote, sizeof(*remote)) != sizeof(*remote)) die("read");
close(sock);
return 0;
}
int main(int argc, char **argv) {
const char *peer = (argc > 1) ? argv[1] : NULL;
int is_client = (peer != NULL);
int num_dev;
struct ibv_device **dev_list = ibv_get_device_list(&num_dev);
if (!dev_list || num_dev == 0) die("ibv_get_device_list");
struct ibv_context *ctx = ibv_open_device(dev_list[0]);
if (!ctx) die("ibv_open_device");
struct ibv_pd *pd = ibv_alloc_pd(ctx);
if (!pd) die("ibv_alloc_pd");
void *buf = aligned_alloc(4096, BUF_SIZE);
memset(buf, is_client ? 'X' : 0, BUF_SIZE);
struct ibv_mr *mr = ibv_reg_mr(pd, buf, BUF_SIZE,
IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ);
if (!mr) die("ibv_reg_mr");
struct ibv_cq *cq = ibv_create_cq(ctx, 64, NULL, NULL, 0);
if (!cq) die("ibv_create_cq");
struct ibv_qp_init_attr qp_attr = {
.send_cq = cq, .recv_cq = cq,
.cap = { .max_send_wr = 64, .max_recv_wr = 64,
.max_send_sge = 1, .max_recv_sge = 1, .max_inline_data = 64 },
.qp_type = IBV_QPT_RC,
};
struct ibv_qp *qp = ibv_create_qp(pd, &qp_attr);
if (!qp) die("ibv_create_qp");
struct ibv_qp_attr attr = {
.qp_state = IBV_QPS_INIT,
.pkey_index = 0, .port_num = 1,
.qp_access_flags = IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ,
};
ibv_modify_qp(qp, &attr,
IBV_QP_STATE | IBV_QP_PKEY_INDEX | IBV_QP_PORT | IBV_QP_ACCESS_FLAGS);
struct ibv_port_attr port_attr;
ibv_query_port(ctx, 1, &port_attr);
union ibv_gid gid;
ibv_query_gid(ctx, 1, 0, &gid);
struct qp_info local = {
.qp_num = qp->qp_num, .lid = port_attr.lid,
.addr = (uintptr_t)buf, .rkey = mr->rkey,
};
memcpy(local.gid, gid.raw, 16);
struct qp_info remote;
tcp_exchange(peer, &local, &remote);
memset(&attr, 0, sizeof(attr));
attr.qp_state = IBV_QPS_RTR;
attr.path_mtu = IBV_MTU_4096;
attr.dest_qp_num = remote.qp_num;
attr.rq_psn = 0;
attr.max_dest_rd_atomic = 1;
attr.min_rnr_timer = 12;
attr.ah_attr.is_global = 1;
memcpy(&attr.ah_attr.grh.dgid, remote.gid, 16);
attr.ah_attr.grh.sgid_index = 0;
attr.ah_attr.grh.hop_limit = 1;
attr.ah_attr.dlid = remote.lid;
attr.ah_attr.port_num = 1;
ibv_modify_qp(qp, &attr,
IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN |
IBV_QP_RQ_PSN | IBV_QP_MAX_DEST_RD_ATOMIC | IBV_QP_MIN_RNR_TIMER);
memset(&attr, 0, sizeof(attr));
attr.qp_state = IBV_QPS_RTS;
attr.timeout = 14; attr.retry_cnt = 7; attr.rnr_retry = 7;
attr.sq_psn = 0; attr.max_rd_atomic = 1;
ibv_modify_qp(qp, &attr,
IBV_QP_STATE | IBV_QP_TIMEOUT | IBV_QP_RETRY_CNT |
IBV_QP_RNR_RETRY | IBV_QP_SQ_PSN | IBV_QP_MAX_QP_RD_ATOMIC);
if (is_client) {
struct ibv_sge sge = { .addr = (uintptr_t)buf, .length = BUF_SIZE, .lkey = mr->lkey };
struct ibv_send_wr wr = {0}, *bad;
wr.wr_id = 1; wr.sg_list = &sge; wr.num_sge = 1;
wr.opcode = IBV_WR_RDMA_WRITE;
wr.send_flags = IBV_SEND_SIGNALED;
wr.wr.rdma.remote_addr = remote.addr;
wr.wr.rdma.rkey = remote.rkey;
if (ibv_post_send(qp, &wr, &bad)) die("ibv_post_send");
struct ibv_wc wc;
int n;
while ((n = ibv_poll_cq(cq, 1, &wc)) == 0) ;
if (n < 0 || wc.status != IBV_WC_SUCCESS) {
fprintf(stderr, "completion failed: status=%d\n", wc.status);
return 1;
}
printf("CLIENT: RDMA Write of %d bytes completed.\n", BUF_SIZE);
} else {
printf("SERVER: waiting. Buffer[0] = '%c' (expect 'X' after write)\n",
((char*)buf)[0]);
sleep(3);
printf("SERVER: after 3s, Buffer[0] = '%c' Buffer[BUF-1] = '%c'\n",
((char*)buf)[0], ((char*)buf)[BUF_SIZE - 1]);
}
return 0;
}Run server, then client: the server’s buffer goes from \0 to X everywhere, and the server’s CPU does nothing to make that happen. No interrupt handler. No copy. The HCA puts the bytes into the server’s memory directly via DMA, and the server only finds out because we sleep and re-check. (No RDMA hardware? rdma_rxe, Soft-RoCE, gives you this exact verbs API in software over any Ethernet NIC on Linux, slow but faithful. It’s how to poke at this stack for free.)
Now, I keep saying “RDMA” and “InfiniBand” and “RoCE” as if they’re the same thing. They’re not. Let me untangle.
InfiniBand is the original. Born in 1999 from the merger of two competing standards (Future I/O and Next Generation I/O). Designed from scratch for low-latency, high-bandwidth, lossless transport. Never won the universal-interconnect war (Ethernet did, TCP/IP, the public internet, economics). But it dominates HPC and ML training clusters. The vendor pool collapsed over time, Mellanox became the dominant supplier, NVIDIA bought Mellanox in 2020 explicitly because GPU training was about to need huge low-latency networks. Intel exited. Cornelis picked up the Omni-Path remnants. For most practical purposes in 2026, “InfiniBand” means “NVIDIA-Mellanox ConnectX-7/8 plus Quantum-2/3 switches.”
IB has its own physical layer (copper DAC for short, fiber for longer, lane speeds tracked Ethernet roughly: HDR=200G, NDR=400G, XDR=800G per 4-lane port). Its own link layer, credit-based flow control: sender literally cannot send without credits from the receiver, so no buffer overflow drops, so the link is natively lossless. Its own network layer: Local Identifiers within a subnet, Global Identifiers across subnets (128-bit, IPv6-format). Its own transport, fully in HCA hardware: RC (Reliable Connected), UC (Unreliable Connected), UD (Unreliable Datagram), XRC (eXtended Reliable, with shared receive queues for large clusters).
IB also requires an external Subnet Manager, a process running somewhere in the fabric that assigns LIDs, computes routing tables for switches, manages paths. Usually OpenSM. Without an SM, links come up at L1 but no L2 forwarding happens.
$ ibstat
CA 'mlx5_0'
CA type: MT4129
Number of ports: 1
Firmware version: 28.39.1200
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 12
Link layer: InfiniBandRoCE, RDMA over Converged Ethernet, is the answer to “what if I want RDMA semantics without buying a separate IB fabric?” Take the IB transport, run it over Ethernet. RoCEv1 wrapped IB directly in Ethernet (EtherType 0x8915). Not routable, same broadcast domain only. RoCEv2: UDP/IP encapsulation, Ethernet → IP → UDP → IB BTH → payload. Routable across normal L3 fabrics. UDP destination port fixed at 4791. This is what 99% of “RDMA over Ethernet” deployments mean today. A ConnectX-7 NIC can do both IB and RoCE (same hardware, different firmware mode). Most ML training clusters in 2026 that picked Ethernet (instead of IB) are running RoCEv2 on 400G or 800G fabric.
But, and this is the catch, IB is lossless by hardware design (credit-based flow control at link layer). Ethernet is lossy by default. RDMA’s RC transport, when it loses a packet, recovers with go-back-N: retransmit everything from the lost sequence number forward. With multi-megabyte windows in flight, even a single drop triggers a retransmit storm. Performance collapses.
So you have to make the Ethernet lossless. Two mechanisms cooperate.
PFC, which I mentioned in the frame chapter. Per-priority PAUSE. You assign RDMA traffic to a priority class (convention: class 3) using DSCP marking. The switch tracks queue occupancy per class. When the class-3 queue crosses Xoff, the switch sends a PFC PAUSE for class 3 upstream. Sender stops sending class 3. When queue drains below Xon, an unpause goes upstream.
$ sudo mlnx_qos -i ens5 --trust=dscp
$ sudo mlnx_qos -i ens5 --pfc 0,0,0,1,0,0,0,0PFC alone is too coarse. Pausing whole classes whenever a buffer fills creates head-of-line blocking inside the class. The fix is to react to congestion before PFC kicks in by reducing the offending flow’s send rate. That’s DCQCN, Data Center Quantized Congestion Notification. Three roles: a Congestion Point (switch experiencing congestion, marks IP ECN bits on packets passing through congested queues), a Notification Point (receiver, when it sees CE-marked packets sends a small Congestion Notification Packet back), a Reaction Point (sender, on receiving CNP multiplicatively reduces its rate using rate-limiting in the HCA, not in software). The sender then slowly probes back up. Tuning the increase/decrease constants is half of the art of running a RoCE fabric. NVIDIA ships defaults that work for most clusters; extreme workloads tune.
DCQCN + PFC together: ECN is the gentle signal that lets flows back off before queues fill. PFC is the hard backstop that prevents drops when ECN doesn’t work fast enough.
And here’s the nastier failure I promised back in the incast section: PFC storms. Pauses cascade in a loop: A pauses B, B pauses C, C ends up pausing A because of a shared queue. Class-3 queues never drain. Fabric deadlock. This is a documented, cluster-scale outage pattern, not a hypothetical, Microsoft’s SIGCOMM paper on running RoCE in production (“RDMA over Commodity Ethernet at Scale”) describes chasing exactly these. Mitigations: per-class isolation, careful topology, PFC watchdog (drops class-3 traffic after a pause that doesn’t lift within a timeout, “letting one fabric die so the others live”).
If I tcpdump -i ens5 -w roce.pcap udp port 4791 and open in Wireshark, I see: Ethernet header (with VLAN tag, with PCP=3 if I set that up), IPv4 (with DSCP typically 26, AF31, for RoCE traffic, ECN bits visible), UDP (dst port 4791), IB BTH (opcode, partition key, destination QP number, PSN), and for RDMA WRITE, RETH (virtual address, length, rkey). Then payload. Then ICRC. Wireshark dissects RoCEv2 natively if recent enough; filter infiniband shows just the IB fields. The OpCode in BTH tells the story, a large RDMA WRITE shows up as RC RDMA WRITE FIRST → RC RDMA WRITE MIDDLE (many) → RC RDMA WRITE LAST. Or RC RDMA WRITE ONLY if it fits in one packet.
So that’s the machinery. Whether it’s actually faster is a measurement question, and four numbers matter. Latency: single message, round-trip divided by 2, the lower bound for dependent transfers. Bandwidth: sustained data rate, large messages, many in flight, the upper bound for bulk. Message rate: small messages per second, which stresses the CPU/syscall/protocol path rather than the wire. CPU utilization: for a given throughput, how much CPU did it cost? Any one in isolation lies. You need all four.
For TCP I’d run iperf3 for bandwidth, sockperf for latency. iperf3 -c <server> -t 30 -P 4 gives me aggregate bandwidth with four parallel streams to defeat single-flow ECMP cap. sockperf ping-pong -i <server> --tcp -t 30 --msg-size 64 gives me round-trip latency distribution. Read [SUM] for aggregate, Retr for retransmits (should be near zero on a clean LAN, non-zero under load means buffer pressure or PFC issues), Cwnd per stream (if saturated at a value smaller than BDP, window-limited, tune tcp_rmem/tcp_wmem).
For RDMA the equivalent suite is perftest. ib_send_lat -s 64 -n 10000 <server> for latency on RDMA Send. ib_write_bw -s 65536 -q 4 --report_gbits <server> for bandwidth on RDMA Write. The -q 4 is “4 QPs in parallel” because single-QP throughput hits a per-QP ceiling related to HCA pipeline depth, multiple QPs use more pipeline. And a sweep across message sizes by looping ib_write_lat -s $size.
The shape to expect on ConnectX-6/7 class hardware at 100G, from published perftest results (same caveat as the TCP numbers earlier, not my bench output): 64-byte messages, TCP latency ~45µs, RDMA ~1.2µs, roughly 40× lower. 1MB messages, both near line rate, but TCP at ~78% sender CPU and ~82% receiver CPU against RDMA’s ~4% and ~3%. Small-message rate, TCP ~600K/s, RDMA ~20M/s.
Two things to absorb. RDMA latency is ~40× lower than TCP. RDMA CPU is ~20× lower at the same bandwidth. The wire is doing the same thing in all three cases. The difference is in how much CPU you spent to keep it fed.
When RDMA bandwidth is below line rate, the suspects in rough order: wrong PCIe gen (Gen3 x16 tops out around 128 Gbit/s theoretical, ~100 practical, at 200G+ you need Gen4 or Gen5), single QP and not enough parallelism in the HCA pipeline, small messages eating per-WR overhead, NUMA misalignment, buffer not aligned to page or hugepage boundary, driver/firmware mismatch. When RDMA latency is above expected, interrupt coalescing in the HCA (different from TCP-coalescing), DCQCN aggressive (rate limiter slowing you down), HCA in low-power state (disable PCIe ASPM in BIOS), ECN markings causing back-off where you didn’t expect.
But the whole point of this exercise is GPU-to-GPU networking, because that’s what training clusters actually do. What changes when GPUs enter?
The first thing that changes is where the data lives. A modern training GPU, the H100 SXM5, has 80GB of HBM3 (high-bandwidth memory: DRAM stacked physically next to the GPU die) at ~3.35 TB/s. Compare to a beefy server CPU: 16-channel DDR5 at ~600 GB/s. The GPU has about 5× the memory bandwidth of the CPU it’s plugged into. And the CPU cannot directly address GPU memory (in most architectures; CXL is changing this slowly).
To get data from GPU memory to a NIC, the naive path is: GPU memory → PCIe → CPU memory → NIC. PCIe Gen4 x16 moves ~32 GB/s per direction; Gen5 doubles that to ~64. (Per direction, throughout, spec sheets love to quote both directions summed, which is how the same link gets advertised as 64 and 128.) So the moment you start staging through CPU memory, the PCIe link becomes the bottleneck, even before the CPU cost of the copy. For inference workloads moving small token batches, this is fine. For training workloads shoveling 28GB of gradients across the fabric every step, it is not fine.
NVIDIA’s answer: GPUDirect RDMA. Let the NIC DMA directly from/to GPU memory. NIC gets the GPU’s physical addresses; the GPU’s BAR is mapped into the NIC’s view of PCIe address space; NIC’s DMA engine reads and writes GPU HBM directly. CPU stages nothing. Requirement: the GPU and NIC must be on the same PCIe root complex, both behind the same switch, or both rooted in the same CPU PCIe controller, so that peer-to-peer DMA can happen without bouncing through main memory.
$ nvidia-smi topo -m
GPU0 GPU1 GPU2 GPU3 NIC0 NIC1 CPU Affinity
GPU0 X NV4 NV4 NV4 PIX SYS 0-31
GPU1 NV4 X NV4 NV4 SYS PIX 0-31
GPU2 NV4 NV4 X NV4 PIX SYS 32-63
GPU3 NV4 NV4 NV4 X SYS PIX 32-63
NIC0 PIX SYS PIX SYS X PHB 0-31
NIC1 SYS PIX SYS PIX PHB X 32-63Read that as a map. GPU0 and NIC0 are PIX: same PCIe switch, peer-to-peer DMA available, fast. GPU0 and NIC1 are SYS: cross-socket, going over UPI, GPUDirect basically doesn’t help. Rule: each GPU should have a dedicated NIC on the same PCIe complex. The DGX H100 reference design has exactly this: 8 GPUs, 8 ConnectX-7 NICs, each pair on its own PCIe switch.
PCIe is also a bottleneck for inter-GPU communication inside a node: that ~32 GB/s per direction is shared across all GPUs talking through the same root complex. NVLink is the dedicated GPU-to-GPU interconnect. NVLink 4 on the H100 generation: 18 links per GPU at 50 GB/s of bidirectional bandwidth each, which NVIDIA advertises as 900 GB/s per GPU, both directions summed, so call it 450 GB/s each way. Roughly 14× what the GPU’s PCIe Gen4 connection can do. NVSwitch is the switch chip that knits the GPUs together; DGX H100 has 4 NVSwitch chips providing non-blocking all-to-all bandwidth across the 8 GPUs.
For an intra-node AllReduce, NVLink bypasses everything else. No NIC. No fabric. 8 GPUs talk all-to-all through NVSwitch at ~7 TB/s of aggregate bidirectional bandwidth. The whole RoCE/IB stack only matters once you cross a node.
(There’s a recent exception. NVIDIA’s NVLink Switch in the GH200/H200 SuperPOD architectures extends NVLink across nodes via dedicated optical NVLink Switch trays. 256 GPUs on a single NVLink fabric at ~900 GB/s per GPU. But these are still rare and expensive; most “big” clusters in 2026 still use IB or RoCE for inter-node.)
So for a 1024-GPU cluster (128 nodes), NVLink handles 8 GPUs inside each node, everything else traverses GPU HBM → PCIe → NIC → fabric → NIC → PCIe → remote GPU HBM. Exactly where GPUDirect RDMA + InfiniBand or RoCE re-enter.
Why do GPUs even need to talk to each other? I should answer that before I keep building.
Data-parallel training. N GPUs, each holding a copy of the model. Each step, each GPU processes a different mini-batch and computes gradients, the gradient being the vector of “nudge every weight this way” values that backpropagation produces, one number per parameter. Before the optimizer updates the model, all GPUs must end up with the same gradient, typically the average over all per-GPU gradients. Otherwise the model copies diverge and you’re not doing SGD anymore. For a 7B-parameter model in fp32, 7B × 4 bytes = 28 GB of gradients. Every step. Sum across 32 GPUs (4 nodes × 8 GPUs). Divide by 32. Broadcast back. Every step, every few hundred milliseconds.
This is where the network bandwidth shows up in your loss curve.
AllReduce. Input: each GPU i has tensor T_i. Output: every GPU has (Σ T_i) / N. Naïve implementation: GPU 0 collects everything, reduces, broadcasts. GPU 0 sees N-1 incoming flows and 1 outgoing, its link is the bottleneck, aggregate scales as O(1/N). Terrible.
Ring AllReduce. Arrange N GPUs in a logical ring. Split the gradient into N chunks. Two phases. ReduceScatter: in each of N-1 steps, every GPU sends one chunk to its right neighbor and receives one chunk from its left neighbor, adding the incoming chunk to its local copy. After N-1 steps, each GPU has the fully-reduced version of one specific chunk, the one it didn’t pass along. Then AllGather: N-1 more steps, each GPU sends its fully-reduced chunk around the ring. After N-1 steps, every GPU has every chunk fully reduced. AllReduce done. Total data each GPU sends: about 2M for large N. Bandwidth-optimal, every link busy every step. Latency: O(N) hops in time, which grows linearly with cluster size and hurts at huge scale.
For small messages where latency dominates, you use tree AllReduce: log₂(N) steps instead of N. Beats ring on latency, loses on bandwidth (root link is busier than leaves). Or recursive halving + doubling, hybrid that’s near-optimal for medium message sizes. NCCL picks the algorithm dynamically based on message size, topology, GPU count.
Which brings me to NCCL. NVIDIA Collective Communications Library. The C/C++ library every distributed PyTorch (DDP, FSDP), DeepSpeed, JAX, Megatron, etc. ultimately calls into for collective operations. AllReduce, AllGather, ReduceScatter, Broadcast, Reduce, plus point-to-point Send/Recv. Optimized for NVIDIA GPUs, NVLink, NVSwitch, GPUDirect RDMA, HCAs.
Before the first collective runs, NCCL discovers the topology: number of GPUs per process/node, NVLink topology, PCIe topology, available IB/RoCE devices and their PCIe affinity, bandwidth estimates per link. Then it computes one or more channels (rings or trees overlaid on the topology) that the actual collectives will execute on. For an 8-GPU all-NVLink-connected node, NCCL typically creates 2-4 rings (different orderings of the same 8 GPUs) and round-robins to use NVLink in both directions.
NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=GRAPH torchrun --nproc_per_node=8 train.py 2>&1 | grep -E 'Channel|tree|ring'You’ll see lines like Channel 00 : 0 1 2 3 4 5 6 7 (a ring) or tree-up lines.
NCCL picks the transport per link. P2P (NVLink) for intra-node GPUs on the same NVSwitch group. SHM (shared memory through CPU) for intra-node GPUs without NVLink between them (slow; it bounces through CPU memory via PCIe). NET (RDMA) for inter-node when the GPU-side NIC supports GPUDirect RDMA: the production path. NET (Socket/TCP) as fallback: slow, CUDA copies GPU→CPU, TCP socket, CPU→CUDA. If you see “NCCL INFO Channel … [via NET/Socket]” in logs, you have a misconfiguration and you wanted GPUDirect.
$ ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8Output has algbw (bytes/sec of the logical AllReduce) and busbw (algbw × 2(N-1)/N, the actual bandwidth used per ring link, accounting for the ring algorithm). busbw is what you compare to your NVLink or IB bandwidth to see how close to peak you got.
Concrete budget. 7B model, 4 nodes × 8 GPUs = 32 GPUs, inter-node NDR (400Gbps). Gradient tensor: 28 GB total. Per GPU after AllReduce every GPU needs the full 28 GB. Ring AllReduce moves about 2 × (N-1)/N × M per GPU = ~56 GB per GPU’s NIC bandwidth. NDR is 400Gbps = 50 GB/s. So per-GPU AllReduce time: 56 / 50 ≈ 1.12 seconds. LLaMA-7B forward + backward on H100 with reasonable batch: roughly 500ms per step. Communication time (~1.12s) is 2.2× the compute time. Total step is ~1.6s, of which 70% is waiting for AllReduce. The model is communication-bound. This is why network matters in training. This is why people pay for 400G fabrics. With 800G XDR the ratio inverts. With smarter algorithms, SHARP in-network aggregation, tree-based, it inverts further. The math is brutal but tractable.
Two big optimizations on top. Gradient compression: cast to fp16 or bf16 (half the bytes), or quantize to int8 (quarter). Less wire bandwidth, more numerical risk. Most production training uses fp16/bf16; int8 needs more care. And overlap: PyTorch DDP doesn’t wait for the backward pass to finish before starting AllReduce. As soon as each layer’s gradients are computed, a bucket starts its AllReduce in parallel with the next layer’s backward. Wired via NCCL’s async API and PyTorch’s autograd hooks (register_comm_hook). On a torch.profiler timeline, without overlap you see backward complete then AllReduce; with overlap, AllReduce starts while backward is still running. If AllReduce is shorter than backward, overlap is free. If it’s longer, the 7B case above, overlap hides only as much as backward provides; there’s still uncovered AllReduce at the end.
Then ZeRO / FSDP shards per-GPU state. Stage 1 shards optimizer state. Stage 2 also shards gradients (ReduceScatter on gradients; each rank ends up with its slice already reduced; AllGather parameters after the optimizer step). Stage 3 (FSDP) also shards parameters, AllGather parameters layer-by-layer during forward, AllGather again during backward, ReduceScatter gradients, the rank only ever holds its parameter slice between forward/backward. ZeRO-3 changes the communication profile substantially, more AllGathers of smaller per-layer messages instead of one giant AllReduce at the end. Good when AllReduce was the bottleneck and AllGathers can overlap with compute. Bad when per-AllGather latency dominates (small layers, high cluster count, latency-bound regime).
Okay. I have all the pieces. Let me try to do the actual thing, follow one byte of one gradient from the GPU register where it was born to the weight update on the other side of the cluster. Because if I can do that, I’ve understood it. And if I can’t, I haven’t.
Setup: 32-GPU cluster, 4 nodes × 8 H100 SXM5. Each node is a DGX H100 with 8 ConnectX-7 NICs at 400Gbps. Spine-leaf RoCEv2 fabric, non-blocking. PyTorch DDP, fp32 gradients, no compression. 7B-parameter model. We’re mid-step; backward is producing gradients. I’m following one specific 4-byte slice of one gradient, creation to weight update. (The timestamps are order-of-magnitude estimates assembled from the budgets earlier in this post, this is a walkthrough, not a profiler capture.)
T+0. The backward kernel running on GPU 0 of node 0 just wrote one fp32 value to an HBM address, call it 0x7f8200001234. One element of the gradient buffer for some weight matrix. The kernel has thousands more elements to write; this one happens to be ours.
T+~50µs. The whole layer’s gradient is ready. PyTorch’s DDP autograd hook fires when the bucket (a contiguous chunk of layers’ gradients) is complete and invokes dist.all_reduce on it; the call enqueues an NCCL operation and control returns immediately to autograd, which keeps running on the next layer. NCCL’s planner decided at init time that this AllReduce runs as a ring across all 32 GPUs, using two channels in parallel, each doing its own ReduceScatter then AllGather.
Ring phase 1, ReduceScatter on intra-node NVLink. GPU 0 sends chunk 0 to GPU 1, receives chunk 7 from GPU 7, adds it into its local copy of chunk 7. This is a CUDA kernel reading from one GPU’s HBM, writing to another’s through NVSwitch at hundreds of GB/s. After 7 steps every GPU holds exactly one chunk that is fully reduced across the 8 local GPUs, and with this send-ordering, the chunk that finishes on GPU i is chunk i+1 mod 8, not chunk i. I got this wrong in my first draft and had to trace a 4-GPU ring on paper to believe it; the off-by-one is maddening. Our slice lives in chunk 3, so it finishes summed-across-the-node on GPU 2.
T+~200µs. Intra-node phase done. NCCL extends the ring across nodes: GPU 2 of node 0 participates in the inter-node ReduceScatter with GPU 2 of nodes 1, 2, 3, NCCL pairs same-rank GPUs across nodes, the “rail-aligned” pattern that keeps each NIC pair on its own rail.
Inter-node ReduceScatter via GPUDirect RDMA. GPU 2 of node 0 has its locally-reduced chunk. NCCL posts an RDMA WRITE work request to a QP on the NIC paired with GPU 2, same PCIe switch. The WR points at GPU memory (registered with the HCA via GPUDirect RDMA at init). The NIC’s DMA engine reads directly from GPU HBM, across the PCIe switch, peer-to-peer, ~50 GB/s per direction of Gen5 x16 in practice. Bytes leave the NIC’s PHY as 4 lanes of NDR PAM4 at 100 Gbps each, encoded in RS-FEC, modulated into photons on 4 fibers in a QSFP-DD module.
T+~210µs. ToR switch. Photons land on a leaf port. Switch ASIC decodes them, parses Ethernet, parses IP, parses UDP (dst 4791 → RoCE), parses IB BTH (destination QP, opcode RC RDMA WRITE FIRST/MIDDLE/LAST). Hashes IP 5-tuple to pick uplink to a spine. Forwards. ECN bits get set if egress queue is congested. PFC PAUSE frames get generated if it’s really congested. Switch buffer is shallow (Tomahawk class); DCQCN should keep us from filling it.
T+~215µs. Spine switch. Same dance. Picks a downlink to the leaf above the destination rack.
T+~220µs. Destination ToR. Forwards to destination NIC.
T+~230µs. Remote NIC receives. Validates ICRC. Parses BTH (WRITE on QP X’, matching remote QP). Parses RETH (destination virtual address, length, rkey). Validates rkey is allowed for this QP. DMAs payload directly into destination GPU’s HBM via the PCIe switch the NIC shares with that GPU.
For the last packet of the WRITE, remote NIC sends ACK back, local NIC posts a CQE. NCCL polls the CQ, sees completion, knows that segment is done.
That transit took ~30µs for a fairly small message. A tiny message would be ~3µs; a large one is wire-bandwidth-limited. After 3 inter-node ReduceScatter steps, GPU 2 of node 0 holds the chunk fully reduced across all 32 GPUs, including our 4-byte slice, now equal to the sum across 32 GPUs (we’ll divide later).
T+~10ms (rough, for the full tensor). Phase 2, AllGather. Reverse the flow: GPU 2 of node 0 sends its now-final chunk around the inter-node ring, then around the intra-node ring. After 7+3 steps, every GPU has every chunk fully reduced.
Our 4-byte slice lands back on GPU 0 of node 0, at the same memory address it started at, 0x7f8200001234, but now containing the averaged gradient across all 32 GPUs.
T+~1.1s total for a 28GB gradient on this 32-GPU 400G fabric. Optimizer step. Adam reads the gradient, reads/updates its own state (momentum, variance), writes the new parameter value. CUDA kernel, intra-GPU, doesn’t touch the network. Model weight updated. One step done. Forward pass begins again.
Where’s the bottleneck? At 400G per NIC and 32 GPUs and 28GB gradients, the inter-node phase dominates and the bottleneck is per-NIC bandwidth. Until SHARP (in-network reduction at the switch) or 800G NIC speeds change the equation, you can’t go faster than your worst NIC’s bandwidth times the algorithmic factor of 2(N-1)/N.
Notice what’s absent from that path: TCP appears nowhere, and the kernel touched nothing after setup.
Which begs the question this whole post has been circling. Why can’t I just use TCP for this?
Latency. TCP intra-DC is ~30-50µs RTT. RDMA is ~1-3µs. For an 875MB per-GPU AllReduce shard with 64KB chunks (NCCL’s default-ish), that’s ~14,000 messages. At a 40µs penalty per message vs RDMA, you’ve eaten ~560ms of pure transport latency on top of wire time. Longer than a forward pass.
CPU. TCP at 400Gbps with the modern stack burns ~50-80% of a CPU socket per direction even with all the offloads. My DGX H100 has 2 CPU sockets driving 8 GPUs. The CPUs are also running data loading, the Python interpreter, the optimizer’s host-side bookkeeping, NCCL’s bootstrap, monitoring. I don’t have a CPU socket to throw at TCP. RDMA: ~3-5% CPU at the same bandwidth.
Copies. TCP needs at least one copy on send and one on receive. At 400Gbps = 50 GB/s each direction, that’s 100 GB/s of memory bandwidth on each end just for the copies. Server memory bandwidth budget is maybe 500 GB/s total. You’re consuming 20% of it on memory copies that did no useful work. And the other 80% is contended by GPU data movement, optimizer state in pinned host memory if you’re doing ZeRO offload, etc.
GPU awareness. TCP knows nothing about GPU memory. To send GPU data via TCP, you cudaMemcpy to CPU memory first, then TCP sends from there, an extra full traversal of the ~64 GB/s-per-direction Gen5 x16 link that GPUDirect avoids entirely, plus ~5-10µs of cudaMemcpy latency before the bytes even reach the kernel.
TCP physically cannot saturate 400G GPU-to-GPU at acceptable CPU cost. Not a matter of tuning. Architecture. Every layer of the TCP stack is doing work that RDMA’s design has eliminated. You cannot tune your way to RDMA performance on TCP because the tuning surface doesn’t include “make zero copies” or “stop entering the kernel.” This is why every serious distributed-training stack is RDMA + NCCL. There’s no TCP plan B that meets the spec.
Looking back at the whole trace, three of the causal links still surprise me. That the encoding on the fiber (PAM4) exists because copper ran out of frequency headroom, and FEC exists because PAM4 paid for its speed with noise margin, physics billing the protocol stack two layers up. That RDMA’s entire architecture follows from one accounting fact, the kernel costing a thousand times more than the wire. And that a lossless fabric is not a hardware property but a fragile agreement between PFC and DCQCN that can deadlock a datacenter when it breaks.
There’s plenty I didn’t pull apart: UCX, the transport layer under both MPI and NCCL. SHARP, where the switches themselves sum gradients mid-fabric, which halves AllReduce wire traffic and deserves its own post. The zoo of model-parallel communication patterns. CXL, which might eventually dissolve the CPU-memory/GPU-memory boundary this post spent so many words crossing.
What I want next is a real cluster for an afternoon: run nccl-tests at every message size from 1KB to 8GB and write down where measured busbw falls relative to 2(N-1)/N × line rate. The gap between those two curves is a map of everything in this post, switch buffers, DCQCN tuning, PCIe topology, some NUMA misalignment I haven’t noticed yet. Whichever it turns out to be, the trace I just walked tells me where to look.