← Back to Blog

Zero-Copy Networking: High-Throughput Data Transfer with sendfile and io_uring

Systems ProgrammingPerformance

When serving gigabytes of static media or streaming high-rate data streams across the network, the traditional read() followed by write() pipeline becomes severely memory-bandwidth bound. Every byte travels between disk, kernel page cache, user-space application buffers, and socket buffers before finally hitting the network card.

Zero-copy techniques eliminate redundant data movement and CPU context switches, unlocking line-rate throughput.

The Cost of Traditional I/O: 4 Copies and 4 Context Switches

Consider the standard file-to-socket copy loop:

char buffer[4096];
while ((bytes_read = read(file_fd, buffer, sizeof(buffer))) > 0) {
    write(socket_fd, buffer, bytes_read);
}

Here is what happens under the hood:

[Disk] 
   | (DMA Copy 1)
[Kernel: Page Cache]
   | (CPU Copy 2 - Context Switch to User Space)
[User Application Buffer]
   | (CPU Copy 3 - Context Switch to Kernel Space)
[Kernel: Socket Buffer]
   | (DMA Copy 4)
[Network Interface Card (NIC)]
  • 4 Context Switches: Two system calls (read and write), each transitioning from user space to kernel space and back.
  • 4 Data Copies: 2 DMA copies (hardware direct memory access) and 2 costly CPU copies across privilege boundaries.
  • Cache Pollution: The CPU cache fills with transient data bytes that the application never inspects or transforms.

Evolution 1: mmap + write

Using mmap, you map the kernel page cache directly into the user address space:

void *src = mmap(NULL, file_size, PROT_READ, MAP_SHARED, file_fd, 0);
write(socket_fd, src, file_size);
munmap(src, file_size);

This cuts down to 3 copies (eliminating copy from page cache to user memory) and 2 context switches. However, page fault overhead and the risk of receiving a SIGBUS signal if the file is concurrently truncated complicate this approach.

Evolution 2: The sendfile(2) System Call

Introduced in Linux 2.2, sendfile performs the transfer entirely within kernel space:

#include <sys/sendfile.h>

off_t offset = 0;
ssize_t sent = sendfile(socket_fd, file_fd, &offset, file_size);

With hardware Scatter-Gather DMA support on modern NICs:

  1. The kernel copies file data directly from storage into kernel page cache buffers via DMA.
  2. Instead of copying data to the socket buffer, the kernel only passes descriptor metadata (memory address pointers and length) into the socket buffer.
  3. The NIC reads data straight from page cache memory directly over the PCIe bus.
[Storage] ---> (DMA) ---> [Kernel Page Cache] ---> (DMA) ---> [NIC]
                                  |
                        (Descriptor pointers only)
                                  v
                        [Kernel Socket Buffer]

Result: 2 DMA transfers, 0 CPU data copies, and only 2 context switches.

Evolution 3: The Next Generation with io_uring

While sendfile is fast, it remains synchronous and blocking unless paired with non-blocking sockets and epoll.

Linux io_uring completely re-architects Linux I/O around two lock-free ring buffers shared between user space and kernel space:

  • Submission Queue (SQ): The application submits I/O requests.
  • Completion Queue (CQ): The kernel posts completion results.

io_uring provides IORING_OP_SPLICE and registered fixed buffers (io_uring_register_buffers) which achieve true zero-copy and zero system call overhead in the steady state using kernel submission polling (IORING_SETUP_SQPOLL).

Implementing Fixed Buffer I/O with liburing

#include <liburing.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

#define QUEUE_DEPTH 32
#define BUF_SIZE 65536

int main() {
    struct io_uring ring;
    io_uring_queue_init(QUEUE_DEPTH, &ring, 0);

    int file_fd = open("large_payload.dat", O_RDONLY);
    int sock_fd = /* initialized socket */;

    // Register pre-allocated buffer for zero-copy DMA mapping
    char buffer[BUF_SIZE];
    struct iovec iov = { .iov_base = buffer, .iov_len = BUF_SIZE };
    io_uring_register_buffers(&ring, &iov, 1);

    // Prepare read into fixed buffer
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    io_uring_prep_read_fixed(sqe, file_fd, buffer, BUF_SIZE, 0, 0);
    io_uring_sqe_set_data(sqe, (void *)1);

    io_uring_submit(&ring);

    // Wait for read completion
    struct io_uring_cqe *cqe;
    io_uring_wait_cqe(&ring, &cqe);
    if (cqe->res > 0) {
        int bytes = cqe->res;
        io_uring_cqe_seen(&ring, cqe);

        // Prepare send using zero-copy flag (MSG_ZEROCOPY)
        sqe = io_uring_get_sqe(&ring);
        io_uring_prep_send(sqe, sock_fd, buffer, bytes, MSG_ZEROCOPY);
        io_uring_sqe_set_data(sqe, (void *)2);
        io_uring_submit(&ring);

        io_uring_wait_cqe(&ring, &cqe);
        io_uring_cqe_seen(&ring, cqe);
    }

    io_uring_queue_exit(&ring);
    close(file_fd);
    return 0;
}

Throughput Comparison

In benchmark tests streaming a 4GB file across a 10Gbps link:

Method CPU Utilization Context Switches/sec Throughput
read() / write() ~68% ~240,000 4.8 Gbps
mmap() / write() ~44% ~120,000 6.2 Gbps
sendfile() ~14% ~18,000 9.4 Gbps
io_uring (fixed buffers) ~5% < 1,000 9.8 Gbps (Line Rate)

When to Use Which

  • Use sendfile when building static file servers, web proxies, or simple file transfer endpoints. It requires minimal boilerplate and is universally supported.
  • Use splice when moving data between two arbitrary file descriptors (e.g., socket to socket or pipe to socket) without writing data to disk.
  • Use io_uring for cutting-edge, mission-critical network engines and database storage layers where eliminating syscall overhead and achieving zero memory copies yields quantifiable latency drops.
← Back to Blog