tl;dr: We’re pleased to announce the beta release of Organist, a tool designed to ease the definition of reliable and low-friction development environments and workflows, building on the combined strengths of Nix and Nickel.

A mess of cables and knobs

I used to play piano as a kid. As a teenager, I became frustrated by the limitations of the instrument and started getting into synthesizers. That was a thrilling experience. Being able to turn a few knobs and get a totally different — and brand new — sound was an amazing feeling. But it was also frustrating. There are so many ways in which you can tune such an instrument (including so many ways of getting an awful sound out of it) that I ended up spending all of my time trying to come up with the right sound, and I didn’t have any left to actually practice the instrument. Besides, the sheer complexity and intricacy of the configuration meant that it was incredibly hard to remember how to get what I wanted out of it.

Then I started playing the organ. Organs might be impressive and daunting, but compared to a synthesizer, they are pretty boring. Modulo a few exceptions, the only controls you have in front of you are the stop knobs which allow you to turn a stop (sound) on or off. That still leaves quite a few possible combinations, but very far from the virtually infinite possibilities of an analog synthesizer. Now I had an expressive enough instrument, but one that knew how to get out of my way to let me play it rather than play with it.

Modern development projects are synthesizer-like these days. We combine a plethora of tools, each one coming with its own set of knobs. Like the old school modular synthesizers, they generally all come from different places and plugging them together requires a huge mess of ad-hoc cabling, like in the good old time!

Organist

It would be tempting to just ditch all of that complexity and only provide a hard-coded set of tools, with hard-coded settings and try to shoehorn everything into that setup. But we all know that such a thing is a fool’s errand. The variety of tools and knobs merely reflects the complexity of the real world, and we have to deal with it one way or another.

Instead, we can limit the scope of that complexity: provide the best possible defaults, and a framework to make it easy to override them in a tractable manner.

Organist is a tool meant to do just that: Provide you with an ergonomic way of managing the complexity of your development environment.

Let’s start with a few assumptions on the required qualities of a good development environment. As much as possible, it should:

1. Remain local. Asking contributors to apt get something just to contribute to your project is both a hindrance, and a source of failure and friction in the setup process.
2. Be seamless to instantiate. Every manual step that people need to perform before working on your project is a chance to mess up, give up, or waste endless amounts of time.
3. Be defined in the repository and kept in sync with the code. It must always be possible to go back to an old version and have everything ready to work on it.
4. Be tailored to your project. Every project has different needs and there’s no “one-size fits them all” as the US air force discovered more than half a century ago.

Organist tries to make it easy for you to address these points by centralizing all the necessary configuration in a single file (or set of files) that can be checked-in to your repository, with both sane defaults and a powerful configuration mechanism.

More concretely

Given a single configuration describing your project’s whole environment (project dependencies and their configuration, the services that are required to run, etc.), Organist will drop you into an environment in which all of these are available and properly configured.

Centralizing all the configuration into a single place might look like a dangerous idea, as centralizing the mess could make it even messier if done wrong. However, Organist provides the means to avoid this trap by leveraging Nickel as its configuration language. Nickel is a language first started by Tweag, with two fundamental features that help tremendously for managing the complexity:

• Contracts provide ahead-of-time sanity checking, as well as most of the niceties of a good programming language (completion, documentation, etc.)
• The merge system makes it possible to define a (possibly complex) default configuration on the library side, and only have the client override the relevant fields.

This design draws inspiration from the NixOS module system which has shown its value in projects like NixOS and home-manager.

Nix does the heavy lifting under the hood, provisioning the dependencies (thanks to the incredible Nixpkgs software distribution), and doing the grunt work of generating and combining all the configuration files. Indeed, Nix combines a few properties that make it an awesome fit for Organist’s needs:

1. It makes it easy to build fully local and reproducible environments. This means that it’s easy to ask Nix something like “make me this very specific version of gcc available only in my current shell session”, which is incredibly valuable.
2. It is totally agnostic to what it packages. This means that we can use it indifferently for building and pulling in a 2M lines toolchain or generating a simple text file.

Organist also currently reuses the Nix command-line interface as its main entry point to avoid reimplementing the wheel (and keep it compatible with plain Nix users).

Note that Organist is not the only tool in that space. devenv in particular follows some similar goals, although with a slightly different approach. The README highlights the difference with the most prominent alternatives.

Demo time (sort-of)

Screencast of Organist in action (click for higher-res)

The repository at https://github.com/tweag/organist-example shows an example of using Organist for a real-life(-ish) project. Let’s dissect the history a bit. All the steps link to the commit that implements them for completeness (although following them isn’t necessary to get the big picture).

It all starts with a simple Python script to upload files to S3 (step 1). The script works fine, but we now want to publish it as a standalone project.

This script requires the boto3 Python library. We can pull it in with Poetry (step 2). Nothing to do with Organist so far, just plain Python development.

However, poetry itself, as well as python need to be available. We can leverage Nix for that. We’ll do it through Organist to abstract things away.

Before anything, we need a slight bit of initial boilerplate. Nix’s template mechanism can handle that for us with nix flake init -t github:nickel-lang/organist (step 3). This being done, we can edit the Organist configuration to provide us with the right environment (step 4). This means:

• Changing the base shell type from Bash to Python310. This ensures that we have a python interpreter (3.10) available, as well as some development tools to make our life easier (python-lsp-server in particular by default);

  - shells = organist.shells.Bash,
  + shells = organist.shells.Python310,

• Adding poetry to the available packages in the shell (and removing hello as it’s not very useful beyond serving as example).

  - packages = {},
  + packages.poetry = organist.import_nix "nixpkgs#poetry",

We can now enter our new development shell nix develop, and run our program (🎉). However, it requires directly accessing the Amazon S3 servers, which isn’t practical for testing. Fortunately, great third-party implementations of the S3 API exist, and we can for instance leverage minio for our tests.

This requires:

• Having the minio executable available
• Being able to run it with the right parameters
• Passing the correct parameters and credentials to our program

This is a fair amount of work to put together, especially if this has to be replicated for every contributor to our project. Leveraging a bit of Nix, a command-runner like foreman or honcho, and some manual work, we could get this all running in a reasonably automated fashion. However, Organist provides a services extension to make that easier for us and allow defining everything in a single place (step 5).

What we have to do is set services.minio to the right command (using the Nix integration to transparently fetch the minio server), and run it:

services.minio = nix-s%"
    %{organist.import_nix "nixpkgs#minio"}/bin/minio server --address :9000 ./.minio-data
  "%,

$ nix run .#start-services start
17:18:57 system  | minio.1 started (pid=461162)
...
17:18:58 minio.1 | S3-API: http://127.0.0.1:9000
...

That’s not enough, because our program doesn’t know about this local minio instance and will still try to connect to S3. But we can just set the right environment (still from our Nickel configuration) to make it look there:

env.AWS_ENDPOINT_URL = "http://localhost:9000"

And we can now run our shiny service:

$ nix develop --command poetry run python3 src/main.py

Obviously, this is just set within the scope of our development environment. Doing so doesn’t alter production in any way, so we can still use Amazon S3 (or whatever else we prefer) there.

This still isn’t ideal because the AWS_ENDPOINT_URL has to hardcode the minio port number. But this is all code! So we can just factor it out:

let minio_port = std.to_string 9031 in
{
  env.AWS_ENDPOINT_URL = "http://localhost:%{minio_port}",
  services.minio = nix-s%"
    %{organist.import_nix "nixpkgs#minio"}/bin/minio server --address :%{minio_port} ./.minio-data
  "%,
}

Going further

This got us to a working environment. But we could go further into making it a nicer environment to work with.

For instance, we can leverage Organist to generate us a good EditorConfig configuration (84bf28) or even integrate with the awesome direnv to provide automatic reloading of the environment (84696f).

Going even further

The world of development tools is an exciting one, full of marvels. Organist interfaces with a number of them, but there’s so much more that could be done!

I’m planning for instance use to let Organist generate generate GitHub actions, or integrate with tf-ncl to make it easier to deploy our project.

Don’t hesitate to contribute you own Organist module to cover any need you have that’s not already handled.

Although Organist is still in its infancy, we hope that it will make your project setup easier. We’re eager to get some feedback on it, so don’t hesitate to try it out!

It’s all happening in the Organist repository on GitHub.