02. Socket Programming

Prerequisites: Basic networking knowledge.

2.1 What to learn: from black boxes to code

Computer networks are often simplified as boxes connected by lines, but the actual coding is left out. However, network programming is not trivial. Given an API with 2 methods: send data & receive data. What else do you need to know?

The TCP byte stream and protocols

People often visusalize computer network as peers exchanging “messages”. But the most common protocol, TCP, doesn’t actually produce messages. It produces a continuous stream of bytes, with no internal boundaries. Interpreting this byte stream is the job of an application protocol — rules for making sense of the byte stream, including how to split it into messages.

Splitting a byte stream into messages is trickier than you might think, especially within an event loop. It’s not like parsing some file formats.

Data serialization

The “message” you want to send via a network can be high-level objects like strings, structs, lists. However, computer networks only know 0s and 1s. Thus, it’s necessary to create a mapping between objects and bytes. This is called serialization (object to bytes) and deserialization (bytes to object).

While you can do this trivially via JSON or Protobuf libraries, rolling your own by manipulating bits and bytes is a good start to low-level programming.

Concurrent programming

With a protocol specification, you can easily build the client application. But the server side is more complicated because it deals with multiple connections. Dealing with many concurrent connections (even mostly idle) is historically difficult (the C10K problem). Although 10K is a tiny number on modern hardware, efficient software is required to make full use of the hardware.

The modern software solution is event-based concurrency with event loops, which drives modern scalable software like NGINX, Redis, Golang’s runtime, etc. Event-based concurrency may be a new programming paradigm for you, and it’s really complicated, so you should learn it by doing.

2.2 Network from a programmers’ perspective

Layers of protocols

Abstract idea: Network protocols are divided into layers. A lower layer can contain a higher layer as payload, and the higher layer adds new functions.

Reality: Ethernet contains IP, IP contains UDP or TCP, UDP or TCP contains application protocols.

Alternatively, we can also divide the layers by function:

The layer of small, discrete messages (IP)

When you download a large file, the hardware cannot possibly store the whole data before forwarding, it can only handle smaller units (IP packets). That’s why the lowest layer is packet-based. The ability to assemble packets into application data is provided by a higher layer, usually TCP.

The layer of multiplexing (Port number)

Multiple apps can share the same network on a single computer. How does the computer know which packet belongs to which app? This is called demultiplexing. The next layer of IP (UDP or TCP) adds a 16-bit port number to distinguish different apps. Each app must claim an unused local port number before it can send or receive data. The computer uses the 4-tuple to identify a “flow” of information:

(src_ip, src_port, dst_ip, dst_port)

The layer of reliable & ordered bytes (TCP)

Small messages are usually not what you want. The file transfer use case requires arbitrarily large data. Worse, the network is unreliable, IP packets can be dropped, reordered. TCP provides a layer of reliable & ordered bytes on top of IP packets, it handles retransmission, reordering automatically.

TCP/IP model

Network protocols layered by function:

3 layers represent 3 needs in networking. They are mapped nicely to TCP/IP concepts. There are other models, such as the TCP/IP model:

Application -> Transport layer (TCP/UDP) -> IP layer -> Link layer (below IP)

The TCP/IP model represents the protocol header structures, which puts TCP and UDP on the same level, but TCP has a higher function, while UDP is just IP+port.

There is also the OSI model, but the model is more complicated than the reality (TCP/IP) it tries to model, so just ignore it.

What’s actually relevant to us

Typical applications do not interact with the IP layer because multiplexing is a universal need. The only thing we care about from the IP layer is the source and destination address.

Ethernet is below IP, it’s also packet-based but uses a different type of address (MAC). MAC addresses are used by hardware that does not care about IP, such as some network switches. We do not care about this layer, it may not even exist in VPNs.

The layers above IP are what we care about. Applications use TCP or UDP, either directly by rolling their own protocol, or indirectly by using an implementation of a well-known protocol. We’ll do the former, like the real Redis.

Both TCP and UDP are contained by IP. IP can contain other protocols like SCTP, but in 2025, only TCP and UDP matter. Everything is built on top of TCP or UDP.

Conclusion: IP, port, TCP/UDP are the concepts we will deal with.

Request-response protocols

Redis, HTTP/1.1, and most RPC protocols are request-response protocols. Each request message is paired with a response message. If the messages are not both reliable & ordered, there will be troubles pairing a response with a request. Thus, most request-response protocols are based on TCP (DNS is an exception).

Packet vs. stream

TCP provides a byte stream, but typical apps expect messages; few apps use the byte stream without interpreting it. Thus, we either need to add a message layer to TCP, or add reliability & order to UDP. The former is far easier, so most apps use TCP, either by using a well-known protocol on top of TCP, or by rolling their own protocol.

TCP and UDP are not only functionally different, their semantics are incompatible. TCP or UDP is the first thing to decide for networked applications.

2.3 Socket primitives

Although we’re coding on Linux, these concepts are platform-agnostic.

What is a socket?

A socket is a handle to refer to a connection or something else. The API for networking is called the socket API, which is similar on different operating systems. The name “socket” has nothing to do with sockets on the wall.

A handle is an opague integer used to refer to things that cross an API boundary. In the same way that a Twitter handle refers to a Twitter user. On Linux, a handle is called a file descriptor (fd) and it’s local to the process. The name “file descriptor” is just a name; it has nothing to do with files, nor does it describe anything.

The socket() method allocates and returns a socket fd (handle), which is used later to actually create connections.

A handle must be closed when you’re done to free the associated resources on the OS side. This is the only thing in common between different types of handles.

Listening socket & connection socket

Listening is telling the OS that an app is ready to accept TCP connections from a given port. The OS then returns a socket handle for apps to refer to that port. From the listening socket, apps can retrieve (accept) incoming TCP connections, which is also represented as a socket handle. So there are 2 types of handles: listening socket & connection socket.

Creating a listening socket requires at least 3 API calls:

1. Obtain a socket handle via socket().
2. Set the listening IP:port via bind().
3. Create the listening socket via listen().

Then use the accept() API to wait for incoming TCP connections. Pseudo code:

fd = socket()
bind(fd, address)
listen(fd)
while True:
    conn_fd = accept(fd)
    do_something_with(conn_fd)
    close(conn_fd)

Connect from a client

A connection socket is created from the client side with 2 API calls:

1. Obtain a socket handle via socket().
2. Create the connection socket via connect().

Pseudo code:

fd = socket()
connect(fd, address)
do_something_with(fd)
close(fd)

socket() creates a typeless socket; the socket type (listening or connection) is determined after the listen() or connect() call. The bind() between socket() and listen() merely sets a parameter. The setsockopt() API can set other socket parameters that will be used later.

Read and write

Although TCP and UDP provide different types of services, they share the same socket API, including the send() and recv() methods. For message-based sockets (UDP), each send/recv corresponds to a single packet. For byte-stream-based sockets (TCP), each send/recv appends to/consumes from the byte stream.

On Linux, send/revc are just a variant of the more generic read/write syscalls used for both sockets, disk files, pipes, and etc. Different types of handle sharing the same read/write API is merely a coincidence; it’s unlikely to have a piece of code that works for both TCP and UDP, as they are semantically incompatible.

Summary: list of socket primitives

• Listening TCP socket:
  • bind() & listen()
  • accept()
  • close()
• Using a TCP socket:
  • read()
  • write()
  • close()
• Create a TCP connection: connect()

The next chapter will help you get started with real code.