03. TCP Server and Client

We’ll write a TCP server and client. The 2 programs are incomplete and incorrect in many ways, but are sufficient to demonstrate the syscalls from the last chapter.

3.1 Tips for Learning Socket Programming

Look Up Stuff in Man Pages

This is not a reference or beginner’s book, so we will not include every detail of using the socket API. You are expected to look things up.

man socket.2

The above command shows the man page for the socket() syscall. Man pages are divided into several sections, as specified by the numerical suffix. Examples:

• man read.2 returns the read() syscall (section 2 is for syscalls).
• man read returns the read shell command (in section 1; not what you want).
• man socket.7 returns the socket interface overview, not the syscall.

Find Online Resources

Man pages are for looking up things, not for learning; they are not tutorials, and they rarely explain anything. You may not be sure what is relevant, or the man page simplify does not have the answer. However, there are some good online resources that can fill the gap, such as Beej’s Guide.

And Googling is still effective in 2024 if you have a specific question.

3.2 Create a TCP Server

What the server will do: read data from the client, write a response, then close the connection.

Step 1: Obtain a Socket Handle

The socket() syscall takes 3 integer arguments.

int fd = socket(AF_INET, SOCK_STREAM, 0);

1. AF_INET is for IPv4. Use AF_INET6 for IPv6 or dual-stack sockets. This selects the IP level protocol. For simplicity, we’ll only consider IPv4.
2. SOCK_STREAM is for TCP. Use SOCK_DGRAM for UDP, which is not our concern.
3. The 3rd argument is 0 and useless for our purposes.

There are other types of sockets that are created with different arguments, such as Unix domain sockets for IPC. We’ll only use TCP, so you can forget about those arguments for now.

The socket.2 man page is not relevant at this point. How to create a TCP or UDP socket is documented in tcp.7 and udp.7 respectively.

Step 2: Configure the Socket

There are many options that change the behavior of a socket. Such as configuring TCP keepalive and Nagle’s algorithm (neither of which are our concern).

However, the socket() syscall does not have a way to pass these of options, so another syscall setsockopt() is used to configure socket options after the socket has been created.

int val = 1;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val));

• The 2nd and 3rd arguments specifies which option to set.
• The 4th argument is the option value.
• The option value is arbitrary bytes, so its length must be specified.

Socket options are scattered over several man pages. TCP-specific options are in tcp.7, while many generic options are in socket.7.

Most options are optional, except the SO_REUSEADDR option; SO_REUSEADDR is enabled (set to 1) for every listening socket. Without it, bind() will fail when you restart your server. This is related to delayed packets and TIME_WAIT, and you can easily find explanations by Googling.

Step 3: Bind to an Address

We’ll bind to the wildcard address 0.0.0.0:1234.

    // bind, this is the syntax that deals with IPv4 addresses
    struct sockaddr_in addr = {};
    addr.sin_family = AF_INET;
    addr.sin_port = ntohs(1234);
    addr.sin_addr.s_addr = ntohl(0);    // wildcard address 0.0.0.0
    int rv = bind(fd, (const sockaddr *)&addr, sizeof(addr));
    if (rv) {
        die("bind()");
    }

struct sockaddr_in holds an IPv4 address and port. You must initialize the structure as shown in the sample code. The ntohs() and ntohl() functions convert numbers to the required big endian format.

For IPv6, use struct sockaddr_in6 instead. That’s why the bind() syscall needs the size of the structure.

The socket API we’re using is derived from the BSD socket API, which is a simple API except for these socket address structures; these struct sockaddr_* are a big mess that just emulate a tagged union.

Step 4: Listen

The listen() syscall takes a backlog argument.

    // listen
    rv = listen(fd, SOMAXCONN);
    if (rv) {
        die("listen()");
    }

After the listen() syscall, the OS will automatically handle TCP handshakes and place established connections in a queue. The application can then retrieve them via accept(). The backlog argument is the size of the queue, which in our case is SOMAXCONN. SOMAXCONN is defined as 128 on Linux, which is sufficient for us.

Step 5: Accept Connections

The server enters a loop that accepts and processes each client connection.

    while (true) {
        // accept
        struct sockaddr_in client_addr = {};
        socklen_t addrlen = sizeof(client_addr);
        int connfd = accept(fd, (struct sockaddr *)&client_addr, &addrlen);
        if (connfd < 0) {
            continue;   // error
        }

        do_something(connfd);
        close(connfd);
    }

The accept() syscall also returns the peer’s address. The addrlen argument is both the input size and the output size.

Step 6: Read & Write

Our dummy processing is just 1 read() and 1 write().

static void do_something(int connfd) {
    char rbuf[64] = {};
    ssize_t n = read(connfd, rbuf, sizeof(rbuf) - 1);
    if (n < 0) {
        msg("read() error");
        return;
    }
    printf("client says: %s\n", rbuf);

    char wbuf[] = "world";
    write(connfd, wbuf, strlen(wbuf));
}

Both read() and write() return the number of bytes, which is ignored in our program. This chapter only demonstrates how to use the socket API, not how to actually do socket programming.

3.3 Create a TCP Client

What the client will do: write something, read back from the server, then close the connection.

Make a Connection

Nothing surprising for the connect() syscall.

    int fd = socket(AF_INET, SOCK_STREAM, 0);
    if (fd < 0) {
        die("socket()");
    }

    struct sockaddr_in addr = {};
    addr.sin_family = AF_INET;
    addr.sin_port = ntohs(1234);
    addr.sin_addr.s_addr = ntohl(INADDR_LOOPBACK);  // 127.0.0.1
    int rv = connect(fd, (const struct sockaddr *)&addr, sizeof(addr));
    if (rv) {
        die("connect");
    }

    char msg[] = "hello";
    write(fd, msg, strlen(msg));

    char rbuf[64] = {};
    ssize_t n = read(fd, rbuf, sizeof(rbuf) - 1);
    if (n < 0) {
        die("read");
    }
    printf("server says: %s\n", rbuf);
    close(fd);

Compile and Run

Compile our programs with the following command line:

g++ -Wall -Wextra -O2 -g 03_server.cpp -o server
g++ -Wall -Wextra -O2 -g 03_client.cpp -o client

Run ./server in one window and then run ./client in another window:

$ ./server
client says: hello

$ ./client
server says: world

3.4 More to Learn

Some important but not immediately relevant things.

Demystifying `struct sockaddr`

Let’s untangle the struct sockaddr mess.

struct sockaddr {
    unsigned short   sa_family;     // AF_INET, AF_INET6
    char             sa_data[14];   // useless
};

struct sockaddr_in {
    short           sin_family;     // AF_INET
    unsigned short  sin_port;       // port number, big endian
    struct in_addr  sin_addr;       // IPv4 address
    char            sin_zero[8];    // useless
};

struct sockaddr_in6 {
    uint16_t        sin6_family;    // AF_INET6
    uint16_t        sin6_port;      // port number, big endian
    uint32_t        sin6_flowinfo;
    struct in6_addr sin6_addr;      // IPv6 address
    uint32_t        sin6_scope_id;
};

struct sockaddr_storage {
    sa_family_t     ss_family;      // AF_INET, AF_INET6
    // enough space for both IPv4 and IPv6
    char    __ss_pad1[_SS_PAD1SIZE];
    int64_t __ss_align;
    char    __ss_pad2[_SS_PAD2SIZE];
};

All of these structures start with a 16-bit integer *_family (despite the different types), which distinguishes IPv4 from IPv6. They emulate a tagged union:

struct sane_sockaddr {
    uint16_t family;    // AF_INET, AF_INET6
    union {
        // whatever ...
    };
};

How to use these structures:

• struct sockaddr_storage is large enough for both IPv4 and IPv6. Use it in production instead.
• struct sockaddr_in and struct sockaddr_in6 are the concrete structures for IPv4 and IPv6.
• Cast a struct sockaddr_storage reference to struct sockaddr_in or struct sockaddr_in6 to initialize/read the structure.
• struct sockaddr * is the type used by the socket API, the structure itself is useless. Just cast any structure to this type.

Syscalls, APIs and Libraries

When you invoke any of the syscalls on Linux, you are actually invoking a thin wrapper in libc — a wrapper to the stable Linux syscall interface.

On Windows, the socket API follows the same BSD API, but with many different details. And the interface comes from an OS DLL rather than syscalls.

There are also many high-level libraries built on top of the socket API, but they are not good for learning, because you need the big picture first, without getting lost in small details.

More Socket: Get Addresses

You can get the address (IP + port) for both local and peer.

• getpeername(): Get the remote peer’s address (also returned by accept()).
• getsockname(): Get my exact address (after binding to a wildcard address).

More Socket: Configure My Local Address

bind() can also be used on the client socket before connect() to specify the source address. Without this, the OS will automatically select a source address. This is useful for selecting a particular interface from multiple interfaces.

If the port in bind() is zero, the OS will automatically choose a port.

The Linux-specific SO_BINDTODEVICE option selects the source interface by name.

More Socket: Domain Name Resolution

getaddrinfo() resolves a domain name into IP addresses. There is a sample program in its man page.

Unlike other socket APIs, this is not a syscall and is implemented in libc because name resolution is a complicated and high-level function. A name resolution on Linux typically involves reading a bunch of files in /etc and then sending and receiving UDP packets with a DNS server.

The Next Step: Protocol

That’s the bare minimum of the socket API you need to know. But knowing the API of something won’t get you very far. You need to create a protocol to actually communicate over a network.

Source code:

• 03_client.cpp
• 03_server.cpp