Today, I am very excited to announce the 1.0 release of Nickel.

A bit more than one year ago, we released the very first public version Nickel (0.1). Throughout various write-ups and public talks (1, 2, 3), we’ve been telling the story of our dissatisfaction with the state of configuration management.

The need for a New Deal

Configuration is everywhere. The manifest of a web app, the configuration of an Apache virtual host, an Infrastructure-as-Code (IaC) cloud deployment (Terraform, Kubernetes, etc.).

Configuration is more often than not considered a second-class engineering discipline, like a side activity of software engineering. That’s a dangerous mistake: with the advent of IaC for the cloud, configuration has become an important aspect of modern software systems, and a critical point of failure.

All the different but connected configurations composing a system are scattered across many languages, tools, and services (JSON, YAML, HCL, Puppet, Apache’s Tcl, and so on). Configuration management is indeed a cross-cutting concern, making failures harder to predict and often spectacular.

In the last decade, studies have shown that misconfigurations were the second largest cause of service-level disruptions in one of Google’s main production services ¹. Misconfigurations also contribute to 16% of production incidents at Facebook ², including the worst-ever outage of Facebook and Instagram that occurred in March 2019 ⁴. Fastly’s³ outage on the 8th June 2021, which basically broke a substantial part of the internet, was triggered by a configuration issue.

Modern configurations are complex. They require new tools to be dealt with, which is why we developed the Nickel configuration language.

Nickel 1.0

Nickel is a lightweight and generic configuration language. It can replace YAML as your new application’s configuration language, or it can generate static configuration files (YAML, JSON or TOML) to be fed to existing tools.

Unlike YAML, though, it anticipates large configurations by being programmable and modular. To minimize the risk of misconfigurations, Nickel features (opt-in) static typing and contracts, a powerful and extensible data validation framework.

Since the initial release (0.1), we’ve refined the semantics enough to be confident in a core design that is unlikely to radically change.

Since the previous stable version (0.3.1), efforts have been made on three principal fronts: tooling (in particular the language server), the core language semantics (contracts, metadata, and merging), and the surface language (the syntax and the stdlib). Please see the release notes for more details.

Tooling & set-up

Follow the getting started guide from Nickel’s website to get a working binary. Nickel also comes with:

• An LSP language server.
• A REPL nickel repl, a markdown documentation generator nickel doc and a nickel query command to retrieve metadata, types and contracts from code.
• Plugins for (Neo)Vim, VSCode, Emacs, and a tree-sitter grammar.
• A code formatter, thanks to Tweag’s tree-sitter-based Topiary.

Watch the companion video for a tour of these tools and features.

A primer on Nickel

Let me walk you through a small example showing what you can do in Nickel 1.0. You can try the code examples from this section in the online Nickel playground.

Just a fancy JSON

I’ll use a basic Kubernetes deployment of a MySQL service as a working example. The following is a direct conversion of a good chunk of mysql.yaml into Nickel syntax, omitting uninteresting values:

{
  apiVersion = "v1",
  kind = "Pod",
  metadata = {
    name = "mysql",
    labels.name = "mysql",
  },

  spec = {
    containers = [
      {
        resources = {
          image = "mysql",
          name = "mysql",
          ports = [
            {
              containerPort = 3306,
              name = "mysql",
            }
          ],
          volumeMounts = [
            {
              # name must match the volume name below
              name = "mysql-persistent-storage",
              # mount path within the container
              mountPath = "/var/lib/mysql",
            }
          ],
        }
      }
    ],
    volumes = [
      {
        name = "mysql-persistent-storage",
        cinder = {
          volumeID = "bd82f7e2-wece-4c01-a505-4acf60b07f4a",
          fsType = "ext4",
        },
      }
    ]
  }
}

This snippet looks like JSON, with a few minor syntax differences. Nickel has indeed the same primitive data types as JSON: numbers (arbitrary precision rationals), strings, arrays, and records (objects in JSON).

The previous example has a lot of repetition: the string "mysql", the name of the app, occurs several times.

Besides, a comment mentions that the name inside the volumeMounts field must match the name of the volume defined inside volumes. Developers are responsible for maintaining this invariant by hand. The absence of a single source of truth might lead to inconsistencies.

Finally, imagine that you now need to reuse the previous configuration several times with slight variations, where the app name and the MySQL port number may change.

This can’t be solved in pure YAML: all you can do is to copy and paste data, and try to manually ensure that copies always all agree. Unlike YAML though, Nickel is programmable.

Reusable configuration

Let’s upgrade our very first example:

# mysql-module.ncl
{
  config | not_exported = {
    port,
    app_name,
    volume_name = "mysql-persistent-storage",
  },

  apiVersion = "v1",
  kind = "Pod",
  metadata = {
    name = config.app_name,
    labels.name = config.app_name,
  },

  spec = {
    containers = [
      {
        resources = {
          limits.cpu = 0.5,
          image = "mysql",
          name = config.app_name,
          ports = [
            {
              containerPort = config.port,
              name = "mysql",
            }
          ],
          volumeMounts = [
            {
              name = config.volume_name,
              # mount path within the container
              mountPath = "/var/lib/mysql",
            }
          ],
        }
      }
    ],
    volumes = [
      {
        name = config.volume_name,
        cinder = {
          volumeID = "bd82f7e2-wece-4c01-a505-4acf60b07f4a",
          fsType = "ext4",
        },
      }
    ]
  }
}

The most important change is the new config field. The rest is almost the same as before, but I replaced the strings "mysql" with config.app_name, "mysql-persistent-storage" with config.volume_name and the hard-coded port number 3306 with config.port.

We can directly use the new fields config.xxx from within other fields. Indeed, in Nickel, you can refer to another part of a configuration you are defining from inside the very same configuration. It’s a natural way to describe data dependencies, when parts of the configuration are generated from other parts.

The | symbol attaches metadata to fields. Here, | not_exported indicates that config is an internal value that shouldn’t appear in the final exported YAML.

This explains most of the new machinery. But this configuration has still something off. Let us try to export it:

$ nickel export -f mysql-module.ncl --format yaml
error: missing definition for `port`
   ┌─ mysql.ncl:33:36
   │
 2 │     config | not_exported = {
   │ ╭───────────────────────────'
 3 │ │     port,
 4 │ │     app_name,
 5 │ │     volume_name = "mysql-persistent-storage",
 6 │ │   },
   │ ╰───' in this record
   · │
33 │               containerPort = config.port,
   │                               -------^^^^
   │                               │      │
   │                               │      required here
   │                               accessed here

Indeed, port and app_name don’t have a value! In fact, you should view this snippet as a partial configuration, a configuration with holes to be filled.

To recover a complete configuration, we need the merge operator &. Merging is a primitive operation that recursively combines records together. Merge is also able to provide a definite value to the fields app_name and port:

# file: mysql-final.ncl
(import "mysql-module.ncl") & {
  config = {
    app_name = "mysql-backend",
    port = 10500,
  }
}

Running nickel export --format yaml -f mysql-final.ncl will now produce a valid YAML configuration.

Partial configurations bear similarities with functions: both are a solution to the problem of how to make repetitive code reusable and composable. But the former seems more adapted to writing reusable configuration snippets.

It’s trivial to assemble several partial configurations together (as long as there’s no conflict): just merge them together.

Partial configurations are data, which can be queried, inspected, and transformed, as long as the missing fields aren’t required. For example, you can get the list of fields of our partial configuration:

$ nickel query -f mysql-module.ncl

Available fields
• apiVersion
• config
• kind
• metadata
• spec

Finally, partial configurations are naturally overridable. Recall that mysql-final.ncl contains the final complete configuration, and consider this example:

# file: override.ncl
(import "mysql-final.ncl") & {
  config.app_name | force = "mysql_overridden",
  metadata.labels.overridden = "true",
}

Exporting override.ncl produces a configuration where all the values depending on config.app_name (directly or indirectly) are updated to use the new value "mysql_overridden", and where metadata.labels has a new entry overridden set to "true".

Overriding is useful to tweak existing code that you don’t control, and wasn’t written to be customizable in the first place. It’s probably better to do without whenever possible, but for some application domains, it simply can’t be avoided.

Partial configurations are automatically extensible: you don’t have to even think about it.

On the other hand, functions are opaque values, which need to be fed with arguments before you can do anything with them. Assembling many of them, inspecting them before they are applied or making their result overridable range from technically possible but much more cumbersome to downright impossible.

Correct configurations

Nickel has two main mechanisms to ensure correctness and prevent misconfigurations as much as possible: static typing and contracts.

Static typing is particularly adapted for small reusable functions. The typechecker is rigorous and catches errors early, before the code path is even triggered.

For configuration data, we tend to use contracts. Contracts are a principled way of writing and applying runtime data validators. It feels like typing when used, relying on annotations and contract constructors (array contract, function contract, etc.), but the checks are performed at runtime. In return, you can easily define your own contracts and compose them with the existing ones.

Contracts are useful to validate the produced configuration (in that case, they will probably come from a library or be automatically generated from external schemas such as when writing Terraform configurations in Nickel). They can validate inputs as well:

# file: mysql-module-safe.ncl
let StartsWith = fun prefix =>
  std.contract.from_predicate (std.string.is_match "^%{prefix}")
in

{
  config | not_exported = {
    app_name
      | String
      | StartsWith "mysql"
      | doc m%"
          The name of the mysql application. The name must start with `"mysql"`,
          as enforced by the `StartsWith` contract.
        "%
      | default
      = "mysql",
    port
      | Number
      | default
      = 3306,
    volume_name = "mysql-persistent-storage",
  },

  # ...
}

The builtin String contract and the custom contract StartsWith "mysql" have been attached to app_name. The latter enforces that the value starts with "mysql", if we have this requirement for some reason. We won’t enter the details of custom contracts here, but it’s a pretty straightforward function.

We’ve used other metadata as well: doc is for documentation, while default indicates that the following definition is a default value. Metadata are leveraged by the Nickel tooling (the LSP, nickel query, nickel doc, etc.)

If we provide a faulty value for app_name, the evaluation will raise a proper error.

Going further

You can look at the main README for a general description of the project. The first blog post series explains the inception of Nickel, and the following posts focus on specific aspects of the language. The most complete source remains the user manual.

Configure your configuration

Zooming out from technical details, I would like to paint a broader picture here. My overall impression is that using bare YAML for e.g. Kubernetes is like programming in assembly. It’s certainly doable, but not desirable. It’s tedious, low-level, and lacks the minimal abstractions to make it scalable.

Some solutions do make configuration more flexible: YAML templating (Helm), tool-specific configuration languages (HCL), etc. But many of them feel like a band-aid over pure JSON or YAML which somehow accidentally grew up to become semi-programming languages.

Our final example is a snippet mapping a pair of high-level values — the only values we care configuring, app_name and port — to a “low-level” Kubernetes configuration. Once written, the only remaining thing to do is to… configure your configuration!

(import "mysql-module.ncl") & {
  config.app_name = "mysql_backup",
  config.port_number = 10400
}

Nickel allows to provide a well-defined interface for reusable parts of configuration, while still providing an escape hatch to override anything else when you need it. Such configuration snippets can be reused, composed, and validated thanks to types and contracts. Although Nickel undeniably brings in complexity, Nickel might paradoxically empower users to make configuration simple again.

If JSON is enough for your use-case, that’s perfect! There’s really no better place to be. But if you’ve ever felt underequipped to handle, write and evolve large configurations, Nickel might be for you.

Conclusion

We are happy to announce the 1.0 milestone for the Nickel configuration language. You can use it wherever you would normally use JSON, YAML, or TOML, but feel limited by using static text or ad-hoc templating languages. You can use it if your configuration is spread around many tools and ad-hoc configuration languages. Nickel could become one fit-them-all configuration language, enabling the sharing of the abstractions, code, schemas (contracts) and tooling across all your stack. Don’t hesitate to check the companion video to help you getting set up.

In our next blog post, we’ll show how to configure Terraform providers using Nickel, and thereby gain the ability to check the code against provider-specific contracts, ahead of performing the deployments. Stay tuned!

Your feedback, ideas, and opinions are invaluable: please use Nickel, break it, do cool things we haven’t even imagined, and most importantly, please let us know about it! Email us at [email protected] or go to Nickel’s open source GitHub repository.