03. TCP Server and Client

Become familiar with the socket API. The code is incomplete and incorrect because network programming is more than calling APIs, which you’ll learn as you go.

3.1 Prerequisites

Get familiar with Linux

Although the principles of network programming are the same, there are many platform differences on Windows & MacOS. For beginners, it’s most convenient to just use Linux, even if you have no Linux experience. You don’t need to know much about Linux to program in it.

1. Get a Linux environment either via VirtualBox, WSL, or cloud providers (VPS).
2. Learn how to edit, move, copy, and delete files. You don’t have to write code in Linux. Learn how to copy files into Linux, or share files with a VM.
3. Compile code with g++. You don’t have to mess with build systems like makefiles.

$ g++ -Wall -Wextra -O2 -g foo.cpp bar.cpp -o prog
$ ./prog

Linux is the most relevant platform for server-side apps. You cannot avoid it.

Basic programming skills

1. C programming concepts: array, struct, memory, pointers.
2. Debugging skills:
   • Print stuff with printf(); verify conditions with assert().
   • Inspect syscalls with strace.
   • Inspect live programs or core dumps with gdb, show stack traces and etc.

C++ features are only used for minor conveniences like vector & string. These are very optional, you don’t need C++ knowledge.

Learn how to get documentation

This is not a reference book, we will not include every detail about the socket API. You are expected to look things up.

man socket.2

This command shows the man page for the socket() syscall. On Linux, all socket API methods are syscalls. Man pages are divided into several sections, as specified by the numerical suffix. Examples:

• man read.2 returns the read() syscall (section 2 is for syscalls).
• man read returns the read shell command (in section 1; not what you want).
• man socket.2 return the socket() syscall.
• man socket.7 returns the socket interface overview, not the syscall.

Man pages are great for looking up things you already know, but not for learning new things. There are great online resources for learning, such as Beej’s Guide.

3.2 Create a TCP Server

Let’s make the pseudo code real: Read data from the client, write a response, that’s it.

fd = socket()
bind(fd, address)
listen(fd)
while True:
    conn_fd = accept(fd)
    do_something_with(conn_fd)
    close(conn_fd)

Step 1: Obtain a socket handle

The socket() syscall takes 3 integer arguments.

int fd = socket(AF_INET, SOCK_STREAM, 0);

1. AF_INET is for IPv4. Use AF_INET6 for IPv6 or dual-stack sockets.
2. SOCK_STREAM is for TCP. Use SOCK_DGRAM for UDP.
3. The 3rd argument is 0 and useless for our purposes.

The combination of the 3 arguments determines the socket protocol:

man socket.2 lists all the flags, but only certain combinations are accepted. We’ll only be using TCP, so you can forget about those arguments for now. By the way, man ip.7 tells you how to create TCP/UDP sockets and the required #includes.

Step 2: Set socket options

There are many options that change the behavior of a socket, such as TCP no delay, IP QoS, etc. (none are our concern). These options are set via the socksockopt() API. Like the bind() API, this just passes a parameter to the OS as the actual socket has not been created yet.

int val = 1;
setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val));

• The combination of the 2nd & 3rd arguments specifies which option to set.
• The 4th argument is the option value.
• Different options use different types, so the size of the option value is also needed.

In this case, we set the SO_REUSEADDR option to an int value of 1, this option accepts a boolean value of 0 or 1. What does this do? This is related to delayed packets and TCP TIME_WAIT. Understanding this requires a non-trivial amount of TCP knowledge, you can read the explanations.

The effect of SO_REUSEADDR is important: if it’s not set to 1, a server program cannot bind to the same IP:port it was using after a restart. This is generally undesirable TCP behavior. You should enable SO_REUSEADDR for all listening sockets! Even if you don’t understand what exactly it is.

You can look up other socket options in man socket.7, man ip.7, man tcp.7, but don’t expect to understand them all.

Step 3: Bind to an address

We’ll bind to the wildcard address 0.0.0.0:1234. This is just a parameter for listen().

    struct sockaddr_in addr = {};
    addr.sin_family = AF_INET;
    addr.sin_port = ntohs(1234);        // port
    addr.sin_addr.s_addr = ntohl(0);    // wildcard IP 0.0.0.0
    int rv = bind(fd, (const sockaddr *)&addr, sizeof(addr));
    if (rv) { die("bind()"); }

struct sockaddr_in holds an IPv4:port pair stored as big endian numbers, converted by ntohs() and ntohl(). For example, 1.2.3.4 is represented by ntohl(0x01020304).

struct sockaddr_in {
    uint16_t       sin_family; // AF_INET
    uint16_t       sin_port;   // port in big endian
    struct in_addr sin_addr;   // IPv4
};
struct in_addr {
    uint32_t       s_addr;     // IPv4 in big endian
};

For IPv6, use struct sockaddr_in6 instead. The addr argument accepts both address types, so the method also needs the struct size because they are different.

struct sockaddr_in6 {
    uint16_t        sin6_family;   // AF_INET6
    uint16_t        sin6_port;     // port in big endian
    uint32_t        sin6_flowinfo; // ignore
    struct in6_addr sin6_addr;     // IPv6
    uint32_t        sin6_scope_id; // ignore
};
struct in6_addr {
    uint8_t         s6_addr[16];   // IPv6
};

struct sockaddr_in and struct sockaddr_in6 have different sizes, so the struct size (addrlen) is needed.

int bind(int sockfd, const struct sockaddr *addr, socklen_t addrlen);

struct sockaddr is not used anywhere, just type cast struct sockaddr_in or struct sockaddr_in6 to this pointer type to match the function prototype.

Step 4: Listen

All the previous steps are just passing parameters. The socket is actually created after listen(). The OS will automatically handle TCP handshakes and place established connections in a queue. The application can then retrieve them via accept().

    // listen
    rv = listen(fd, SOMAXCONN);
    if (rv) { die("listen()"); }

The 2nd argument is the size of the queue, which in our case is SOMAXCONN. SOMAXCONN is defined as 128 on Linux, which is sufficient for us.

Step 5: Accept connections

The server enters a loop that accepts and processes each client connection.

    while (true) {
        // accept
        struct sockaddr_in client_addr = {};
        socklen_t addrlen = sizeof(client_addr);
        int connfd = accept(fd, (struct sockaddr *)&client_addr, &addrlen);
        if (connfd < 0) {
            continue;   // error
        }

        do_something(connfd);
        close(connfd);
    }

The accept() syscall also returns the peer’s address. The addrlen argument is both the input and output size.

Step 6: Read & write

Our dummy processing is just 1 read() and 1 write().

static void do_something(int connfd) {
    char rbuf[64] = {};
    ssize_t n = read(connfd, rbuf, sizeof(rbuf) - 1);
    if (n < 0) {
        msg("read() error");
        return;
    }
    printf("client says: %s\n", rbuf);

    char wbuf[] = "world";
    write(connfd, wbuf, strlen(wbuf));
}

You can replace read/write with send/recv. The difference is that send/recv has an extra argument to pass some optional flags that we don’t need.

ssize_t write(int fd, const void *buf, size_t len);
ssize_t send(int fd, const void *buf, size_t len, int flags);

For now, we have ignored the return value of write() and there is no error handling. We’ll write real programs in the next chapter.

3.3 Create a TCP Client

Write something, read back from the server, then close the connection.

    int fd = socket(AF_INET, SOCK_STREAM, 0);
    if (fd < 0) {
        die("socket()");
    }

    struct sockaddr_in addr = {};
    addr.sin_family = AF_INET;
    addr.sin_port = ntohs(1234);
    addr.sin_addr.s_addr = ntohl(INADDR_LOOPBACK);  // 127.0.0.1
    int rv = connect(fd, (const struct sockaddr *)&addr, sizeof(addr));
    if (rv) {
        die("connect");
    }

    char msg[] = "hello";
    write(fd, msg, strlen(msg));

    char rbuf[64] = {};
    ssize_t n = read(fd, rbuf, sizeof(rbuf) - 1);
    if (n < 0) {
        die("read");
    }
    printf("server says: %s\n", rbuf);
    close(fd);

INADDR_LOOPBACK is defined as 0x7f000001, which is the address 127.0.0.1.

Compile our programs with the following command line:

g++ -Wall -Wextra -O2 -g 03_server.cpp -o server
g++ -Wall -Wextra -O2 -g 03_client.cpp -o client

Run ./server in one window and then run ./client in another window:

$ ./server
client says: hello

$ ./client
server says: world

3.4 More on socket API

Some important but not immediately relevant things.

Understand `struct sockaddr`

Let’s look at these function prototypes:

int accept(int sockfd, struct sockaddr *addr, socklen_t len);
int connect(int sockfd, const struct sockaddr *addr, socklen_t len);
int bind(int sockfd, const struct sockaddr *addr, socklen_t len);

We never used struct sockaddr, instead we forcibly cast either struct sockaddr_in or struct sockaddr_in6 to this pointer type. Here is how these structs are defined.

// pointless
struct sockaddr {
    unsigned short  sa_family;      // AF_INET, AF_INET6
    char            sa_data[14];    // useless
};
// IPv4:port
struct sockaddr_in {
    sa_family_t     sin_family;     // AF_INET
    uint16_t        sin_port;       // port number, big endian
    struct in_addr  sin_addr;       // IPv4 address
};
// IPv6:port
struct sockaddr_in6 {
    sa_family_t     sin6_family;    // AF_INET6
    uint16_t        sin6_port;      // port number, big endian
    uint32_t        sin6_flowinfo;
    struct in6_addr sin6_addr;      // IPv6 address
    uint32_t        sin6_scope_id;
};
// can store both sockaddr_in & sockaddr_in6
struct sockaddr_storage {
    sa_family_t     ss_family;      // AF_INET, AF_INET6
    char __some_padding[__BIG_ENOUGH_NUMBER];
};

The socket API is weird in that it defines many pointless types.

• struct sockaddr has no use at all; struct sockaddr * is practically just void *.
• struct sockaddr_storage is supposed to hold any address types. It’s trivially replaced by union { struct sockaddr_in v4; struct sockaddr_in6 v6 }.
• struct sockaddr_in & struct sockaddr_in6 are the only useful and concrete structs.
• sin_addr & sin6_addr are pointlessly nested structs with just a single field.
• *_family is practically a 16-bit integer, yet it has its own type.

What the API wants to achieve can be expressed by a simple tagged union:

struct fictional_sane_sockaddr {
    uint16_t family;    // tag: AF_INET, AF_INET6
    uint16_t port;
    union {
        struct { uint8_t ipv4[4]; };
        struct { uint8_t ipv6[16]; /* ... */ };
    };
};
// warning: not compatible with `struct sockaddr_*`

Syscalls, APIs, and libraries

On Linux, each socket function is a syscall wrapper in libc. The socket API is called BSD socket and is supported by all major platforms. On Windows, the API is mostly the same, with minor differences like different function names.

There are also socket libraries, but they are not as useful as you might think; the main complexity is not the API, but the rest of the things like protocols, event loops. So a library won’t do much. The socket API is simple and contains only a few methods. The only scary part is struct sockaddr_*.

Specifiy the local address before connect()

bind() can also be used on the client socket before connect() to specify the source address. Without this, the OS will automatically select a source address. This is useful for selecting a particular source address if multiple ones are available. If the port in bind() is zero, the OS will automatically pick a port.

Get the address of each side

If you are using wildcard IP or letting the OS pick the port, you don’t know the exact address. Use getsockname() to retrieve the local address of a TCP connection. Use getpeername() to retrieve the remote address (the same address returned from accept()).

int getsockname(int fd, struct sockaddr *addr, socklen_t *addrlen); // local
int getpeername(int fd, struct sockaddr *addr, socklen_t *addrlen); // remote

Domain name resolution

getaddrinfo() resolves a domain name into IP addresses. There is a sample program in its man page.

Unlike other socket APIs, this is not a Linux syscall and is implemented in libc because name resolution is a complicated and high-level function on Linux. It involves reading a bunch of files such as /etc/resolv.conf and /etc/hosts before querying a DNS server with UDP.

Socket and inter-process communication (IPC)

There are mechanisms that allow processes within the same machine to communicate such as Unix domain sockets, pipes, etc. They are just a computer network confined to a single machine, so the programming techniques are the same.

Unix domain sockets share the same API with network sockets. You can create either packet-based or byte-stream-based Unix domain sockets, like UDP or TCP. A Unix domain socket is created with different flags on the socket() method and uses struct sockaddr_un, but the rest is the same. Read man unix.7 for more info.

A pipe is a one-way byte stream. So you need a protocol like a TCP socket, which is not as trivial as you might think. You’ll learn about protocols in the next chapter.

Variants of read & write

We used read/write syscalls for sockets. They are the most generic IO interface also usable for disk files, pipes, etc. I list some variants of read/write just for your information.

Source code:

• 03_client.cpp
• 03_server.cpp