The Nix daemon uses a custom binary protocol — the nix daemon protocol — to communicate with just about everything. When you run nix build on your machine, the Nix binary opens up a Unix socket to the Nix daemon and talks to it using the Nix protocol¹. When you administer a Nix server remotely using nix build --store ssh-ng://example.com [...], the Nix binary opens up an SSH connection to a remote machine and tunnels the Nix protocol over SSH. When you use remote builders to speed up your Nix builds, the local and remote Nix daemons speak the Nix protocol to one another.

Despite its importance in the Nix world, the Nix protocol has no specification or reference documentation. Besides the original implementation in the Nix project itself, the hnix-store project contains a re-implementation of the client end of the protocol. The gorgon project contains a partial re-implementation of the protocol in Rust, but we didn’t know about it when we started. We do not know of any other implementations. (The Tvix project created its own gRPC-based protocol instead of re-implementing a Nix-compatible one.)

So we re-implemented the Nix protocol, in Rust. We started it mainly as a learning exercise, but we’re hoping to do some useful things along the way:

• Document and demystify the protocol. (That’s why we wrote this blog post! 👋)
• Enable new kinds of debugging and observability. (We tested our implementation with a little Nix proxy that transparently forwards the Nix protocol while also writing a log.)
• Empower other third-party Nix clients and servers. (We wrote an experimental tool that acts as a Nix remote builder, but proxies the actual build over the Bazel Remote Execution protocol.)

Unlike the hnix-store re-implementation, we’ve implemented both ends of the protocol. This was really helpful for testing, because it allowed our debugging proxy to verify that a serialization/deserialization round-trip gave us something byte-for-byte identical to the original. And thanks to Rust’s procedural macros and the serde crate, our implementation is declarative, meaning that it also serves as concise documentation of the protocol.

Structure of the Nix protocol

A Nix communication starts with the exchange of a few magic bytes, followed by some version negotiation. Both the client and server maintain compatibility with older versions of the protocol, and they always agree to speak the newest version supported by both.

The main protocol loop is initiated by the client, which sends a “worker op” consisting of an opcode and some data. The server gets to work on carrying out the requested operation. While it does so, it enters a “stderr streaming” mode in which it sends a stream of logging or tracing messages back to the client (which is how Nix’s progress messages make their way to your terminal when you run a nix build). The stream of stderr messages is terminated by a special STDERR_LAST message. After that, the server sends the operation’s result back to the client (if there is one), and waits for the next worker op to come along.

The Nix wire format

Nix’s wire format starts out simple. It has two basic types:

• unsigned 64-bit integers, encoded in little-endian order; and
• byte buffers, written as a length (a 64-bit integer) followed by the bytes in the buffer. If the length of the buffer is not a multiple of 8, it is zero-padded to a multiple of 8 bytes. Strings on the wire are just byte buffers, with no specific encoding.

Compound types are built up in terms of these two pieces:

• Variable-length collections like lists, sets, or maps are represented by the number of elements they contain (as a 64-bit integer) followed by their contents.
• Product types (i.e. structs) are represented by listing out their fields one-by-one.
• Sum types (i.e. unions) are serialized with a tag followed by the contents.

For example, a “valid path info” consists of a deriver (a byte buffer), a hash (a byte buffer), a set of references (a sequence of byte buffers), a registration time (an integer), a nar size (an integer), a boolean (represented as an integer in the protocol), a set of signatures (a sequence of byte buffers), and finally a content address (a byte buffer). On the wire, it looks like:

3c 00 00 00 00 00 00 00 2f 6e 69 78 2f 73 74 6f 72 65 ... 2e 64 72 76 00 00 00 00  <- deriver
╰──── length (60) ────╯ ╰─── /nix/store/c3fh...-hello-2.12.1.drv ───╯ ╰ padding ╯

40 00 00 00 00 00 00 00 66 39 39 31 35 63 38 37 36 32 ... 30 33 38 32 39 30 38 66  <- hash
╰──── length (64) ────╯ ╰───────────────────── sha256 hash ─────────────────────╯

02 00 00 00 00 00 00 00                                                            ╮
╰── # elements (2) ───╯                                                            │
                                                                                   │
   39 00 00 00 00 00 00 00 2f 6e 69 78 ... 2d 32 2e 33 38 2d 32 37 00 00 .. 00 00  │
   ╰──── length (57) ────╯ ╰── /nix/store/9y8p...glibc-2.38-27 ──╯ ╰─ padding ──╯  │ references
                                                                                   │
   38 00 00 00 00 00 00 00 2f 6e 69 78 ... 2d 68 65 6c 6c 6f 2d 32 2e 31 32 2e 31  │
   ╰──── length (56) ────╯ ╰───────── /nix/store/zhl0...hello-2.12.1 ───────────╯  ╯

1c db e8 65 00 00 00 00 f8 74 03 00 00 00 00 00 00 00 00 00 00 00 00 00            <- numbers
╰ 2024-03-06 21:07:40 ╯ ╰─ 226552 (nar size) ─╯ ╰─────── false ───────╯

01 00 00 00 00 00 00 00                                                            ╮
╰── # elements (1) ───╯                                                            │
                                                                                   │ signatures
   6a 00 00 00 00 00 00 00 63 61 63 68 65 2e 6e 69 ... 51 3d 3d 00 00 00 00 00 00  │
   ╰──── length (106) ───╯ ╰─── cache.nixos.org-1:a7...oBQ== ────╯ ╰─ padding ──╯  ╯

00 00 00 00 00 00 00 00                                                            <- content address
╰──── length (0) ─────╯

This wire format is not self-describing: in order to read it, you need to know in advance which data-type you’re expecting. If you get confused or misaligned somehow, you’ll end up reading complete garbage. In my experience, this usually leads to reading a “length” field that isn’t actually a length, followed by an attempt to allocate exabytes of memory. For example, suppose we were trying to read the “valid path info” written above, but we were expecting it to be a “valid path info with path,” which is the same as a valid path info except that it has an extra path at the beginning. We’d misinterpret /nix/store/c3f-...-hello-2.12.1.drv as the path, we’d misinterpret the hash as the deriver, we’d misinterpret the number of references (2) as the number of bytes in the hash, and we’d misinterpret the length of the first reference as the hash’s data. Finally, we’d interpret /nix/sto as a 64-bit integer and promptly crash as we allocate space for more than $8 \times 10^{18}$ references.

There’s one important exception to the main wire format: “framed data”. Some worker ops need to transfer source trees or build artifacts that are too large to comfortably fit in memory; these large chunks of data need to be handled differently than the rest of the protocol. Specifically, they’re transmitted as a sequence of length-delimited byte buffers, the idea being that you can read one buffer at a time, and stream it back out or write it to disk before reading the next one. Two features make this framed data unusual: the sequence of buffers are terminated by an empty buffer instead of being length-delimited like most of the protocol, and the individual buffers are not padded out to a multiple of 8 bytes.

Serde

Serde is the de-facto standard for serialization and deserialization in Rust. It defines an interface between serialization formats (like JSON, or the Nix wire protocol) on the one hand and serializable data types on the other. This divides our work into two parts: first, we implement the serialization format, by specifying the correspondence between Serde’s data model and the Nix wire format we described above. Then we describe how the Nix protocol’s messages map to the Serde data model.

The best part about using Serde for this task is that the second step becomes straightforward and completely declarative. For example, the AddToStore worker op is implemented like

#[derive(serde::Deserialize, serde::Serialize)]
pub struct AddToStore {
    pub name: StorePath,
    pub cam_str: StorePath,
    pub refs: StorePathSet,
    pub repair: bool,
    pub data: FramedData,
}

These few lines handle both serialization and deserialization of the AddToStore worker op, while ensuring that they remain in-sync.

Mismatches with the Serde data model

While Serde gives us some useful tools and shortcuts, it isn’t a perfect fit for our case. For a start, we don’t benefit much from one of Serde’s most important benefits: the decoupling between serialization formats and serializable data types. We’re interested in a specific serialization format (the Nix wire format) and a specific collection of data types (the ones used in the Nix protocol); we don’t gain much by being able to, say, serialize the Nix protocol to JSON.

The main disadvantage of using Serde is that we need to match the Nix protocol to Serde’s data model. Most things match fairly well; Serde has native support for integers, byte buffers, sequences, and structs. But there were a few mismatches that we had to work around:

• Different kinds of sequences: Serde has native support for sequences, and it can support sequences that are either length-delimited or not. However, Serde does not make it easy to support length-delimited and non-length-delimited sequences in the same serialization format. And although most sequences in the Nix format are length-delimited, the sequence of chunks in a framed source are not. We hacked around this restriction by treating a framed source not as a sequence but as a tuple with $2^{64}$ elements, relying on the fact that Serde doesn’t care if you terminate a tuple early.
• The Serde data model is larger than the Nix protocol needs; for example, it supports floating point numbers, and integers of different sizes and signedness. Our Serde de/serializer raises an error at runtime if it encounters any of these data types. Our Nix protocol implementation avoids these forbidden data types, but the Serde abstraction between the serializer and the data types means that any mistakes will not be caught at compile time.
• Sum types tagged with integers: Serde has native support for tagged unions, but it assumes that they’re tagged with either the variant name (i.e. a string) or the variant’s index within a list of all possible variants. The Nix protocol uses numeric tags, but we can’t just use the variant’s index: we need to specify specific tags for specific variants, to match the ones used by Nix. We solved this by using our own derive macro for tagged unions. Instead of using Serde’s native unions, we map a union to a Serde tuple consisting of a tag followed by its payload.

But with these mismatches resolved, our final definition of the Nix protocol is fully declarative and pretty straightforward:

#[derive(TaggedSerde)]
//       ^^ our custom procedural macro for unions tagged with integers
pub enum WorkerOp {
    #[tagged_serde = 1]
    //              ^^ this op has opcode 1
    IsValidPath(StorePath, Resp<bool>),
    //             ^^            ^^ the op's response type
    //             || the op's payload
    #[tagged_serde = 6]
    QueryReferrers(StorePath, Resp<StorePathSet>),
    #[tagged_serde = 7]
    AddToStore(AddToStore, Resp<ValidPathInfoWithPath>),
    #[tagged_serde = 9]
    BuildPaths(BuildPaths, Resp<u64>),
    #[tagged_serde = 10]
    EnsurePath(StorePath, Resp<u64>),
    #[tagged_serde = 11]
    AddTempRoot(StorePath, Resp<u64>),
    #[tagged_serde = 14]
    FindRoots((), Resp<FindRootsResponse>),
    // ... another dozen or so ops
}

Next steps

Our implementation is still a work in progress; most notably the API needs a lot of polish. It also only supports protocol version 34, meaning it cannot interact with old Nix implementations (before 2.8.0, which was released in 2022) and will lack support for features introduced in newer versions of the protocol.

Since in its current state our Nix protocol implementation can already do some useful things, we’ve made the crate available on crates.io. If you have a use-case that isn’t supported yet, let us know! We’re still trying to figure out what can be done with this.

In the meantime, now that we can handle the Nix remote protocol itself we’ve shifted our experimental hacking over to integrating with Bazel remote execution. We’re writing a program that presents itself as a Nix remote builder, but instead of executing the builds itself it sends them via the Bazel Remote Execution API to some other build infrastructure. And then when the build is done, our program sends it back to the requester as though it were just a normal Nix remote builder.

But that’s just our plan, and we think there must be more applications of this. If you could speak the Nix remote protocol, what would you do with it?