Adopting Nickel in CodeCommons

22 October 2026 Simeon Carstens

The CodeCommons project had a complex architecture with multiple services and a plethora of configuration formats and files that are often filled in via ad-hoc templating. This made for complex maintenance and hard to discover bugs. In this blog post, I describe how the Nickel configuration language could be used for great good in this context and tell you about our experience adopting it.

• nickel
• machine-learning
• data-engineering

Complex architectures with multiple services require a lot of configuration. There are a lot of knobs to be tuned, and that configuration needs to be persisted in configuration files. These come in a large variety of formats - widely accepted standards such as YAML or TOML, ecosystem-specific formats such as key-value pairs in the Apache data engineering world, or ad-hoc custom configuration formats. At the same time, errors in configuration files can lead to hard to detect and frustrating bugs: invalid or missing field values can lead to late crashes after minutes of start-up, silent fallbacks to unwanted defaults due to incomplete or unknown configuration error handling in external packages, etc.

Nickel is a programmable configuration language developed by Tweag that aims to provide safe and validated configurations with ease. You’ll learn more about the most important Nickel features later, but if you’re impatient, you can read more about it on Nickel’s website or in our Nickel series on this blog.

This blog post is aimed at developers who have never used Nickel before. I show how we used Nickel to improve the difficult configuration management of a massive data ingestion pipeline, as part of the CodeCommons project. This ingestion pipeline relies on multiple industry-standard services, but also on components developed for the Software Heritage (SWH) project, and thus comes with a whole zoo of configuration files formats - all the characteristics of an ideal Nickel testing ground.

You’ll first learn the most important facts about CodeCommons, the ingestion part of it and Nickel, then I’ll describe how we tied Nickel into the existing architecture and what difference it makes, and finally I’ll summarize our observations.

CodeCommons ingestion pipeline architecture and configuration

In the CodeCommons project, Modus Create and French research institutions aim to build a reference source code database for training code-generation large language models (LLMs) and for research on source code. The project builds on top of the Software Heritage archive, which is the world’s largest archive of open-source code. In the following, I’ll discuss in some detail the configuration of a key part: the source code data ingestion pipeline Modus Create is working on.

Service architecture and configuration

The CodeCommons ingestion pipeline mostly relies on the existing Software Heritage ingestion pipeline when it comes to its components, but is deployed on a scientific high-performance computing (HPC) cluster. CodeCommons uses the same data structure as Software Heritage, namely a Merkle Directed Acyclic Graph (used, for example, by Git), to represent archived source code repositories. In this view, ingestion of source code is all about adding unique nodes to this graph. After scheduling ingestion of a project in a Redis queue, loaders read projects from that queue and map new data into the graph data structure - see the following figure for an overview. The leaves of this graph are source code files, and the loaders write them to custom object storage. Intermediate nodes represent things such as releases, revisions, or directories, and the loaders write them to distinct topics in a Kafka journal. From the Kafka journal, a custom-written “deduplicator” component reads topics and exports the graph data into two sets of Optimized Row Columnar (ORC) files: one appropriate for CodeCommons use, and one that will be used to update the existing Software Heritage archive.

Service-specific configuration options that a user might want to tune differently from run to run, and deployment-related parameters were all summarized in a simple Bash script with lines of the form export CONFIG_OPTION=<value> that served as our principal configuration file. This setup led to quite some difficulties, as we’ll see.

Configuration challenges

One of the particular difficulties with this setup was the fact that service-specific configuration files contained at least three different “classes” of parameters:

• a lot of parameters that rarely, if ever, needed to be adjusted (e.g., batch sizes, timeouts, etc.)
• settings that users of the pipeline realistically might want to change from one pipeline run to the other (e.g., deployment environment, list of projects to ingest, etc.),
• and finally, configuration options that were known only at pipeline runtime (e.g., hostnames and ports of upstream services).

These classes were non-negotiable - if we didn’t want to adapt the service’s code, we had to live with the above. At the time, we made it work with the following approach:

• each service used a template configuration file with rarely adjusted settings filled in. This file was in the format natively expected by the service (e.g. key-value pairs for Kafka, YAML for Software Heritage services, etc.). For Redis, for example, this file looked like so:

  save 60 100000
  appendonly yes
  rdbcompression yes
  
  protected-mode no
  
  dir ${REDIS_DATA_DIR}
  port ${REDIS_PORT}
  

• settings that might change from pipeline run to pipeline run were stored in a global “Bash config file” config.sh with lines export PARAMETER=<VALUE>. A couple example lines from this file were:

  export IS_IN_CLUSTER=[0|1]
  export PYTHON_VENV=${SHAREDWORKDIR}/tools/swh/venv
  export KAFKA_NUM_NETWORK_THREADS=16
  

• settings that were known only at runtime were passed from service to service via exchange files that, again, contained relevant export PARAMETER=<VALUE> lines, such as REDIS_SLURM_NODELIST (which was, essentially, the hostname of the compute node running Redis)

Finally, just before running a service, all the above were combined into a final, complete configuration file using tools such as envsubst. This environment variable-based approach also allowed us to use configuration settings in the orchestration Bash scripts directly, e.g. to send a command to the cluster’s job scheduler or run it locally: For example, the value of REDIS_DEPLOYMENT was checked in the orchestration script, and depending on its value, the Redis server was started either on the machine the script ran on or on a compute node.

But this approach had multiple weak points. On the operational side, when running the pipeline, values were not validated on even a basic level, e.g. encoding a Boolean value wrongly as 0 instead of false, and local copies of the reference config.sh file became outdated quickly: A developer maintained a config file config_local.sh, but after a git pull a new mandatory config option might have been added to the template file, which silently invalidated the local configuration. And there was config_cluster.sh over on the HPC cluster, too… These issues were very common and often led to crashes very late in the pipeline, after minutes-long start-up of preceding services had completed.

When extending the configuration, sensible / allowed values needed to be documented manually and this documentation needed to be maintained - what was sensible / allowed changed with time. When a new developer had to set up a configuration from scratch, no editor tooling was available that automatically gave you information such as type or allowed values or facilitated life via auto-completion.

All these made this system difficult to use and to maintain. How could we do better?

Reigning in the chaos: nickelizing the CodeCommons ingestion pipeline

Nickel

Nickel is, at its core, a bit like JSON with functions, not unlike the Nix language upon which it was designed to be an improvement. But on top, Nickel offers “configuration safety” features: It features contracts and optional static typing to validate data. While types are checked by Nickel before the program is run, contracts enforce arbitrary assertions lazily at runtime, thus covering different use cases. Nickel also aims to help its users write configurations that can be easily factorized into reusable parts. These can then be combined into more complex configurations. A configuration written in Nickel can then be easily exported into standard configuration file formats such as YAML, TOML or JSON.

Different shells of Nickel

In our specific project, we could intervene with Nickel on multiple levels, each entailing a different level of complexity:

• we could generate our Bash configuration file config.sh with Nickel from a nickelized version config.ncl of the existing file and leave everything else as is,
• we could additionally replace service config templates with Nickel configuration equivalents: translate the Kafka / Redis / etc.-specific configuration files to Nickel and merge them with the principal config.ncl at pipeline runtime, filling in values only known at runtime upon exporting the service config in the specific format it requires
• finally, in addition to both of the above, we could generate the actual Bash scripts that run the services with Nickel: each Bash script would be a templated string in Nickel, and completed and exported to a Bash file at runtime when needed. This would remove the need for a Bash config file in the first place, because we wouldn’t need to rely on environment variables that are read by Bash any more.

We chose to adopt the first, and partially the second, in the project, but here, I will describe only the first, most straightforward change.

Generating the main Bash configuration file with Nickel

The main Bash configuration file consisted of a flat list of around 120 exported Bash variables, most of which were prefixed with the name of the component they pertained to, for example, a configuration setting for the number of loader workers was called SWH_LOADER_CONCURRENCY. Variables were grouped by component / scope.

Step 1: Grouping configuration options

To bring order into this, we structured the Nickel equivalent of this configuration as a nested record, which we could feed into an “output” field in the correct format. A record is very similar to a JSON object, only that Nickel allows additional operations such as merging records. The very basic nickelized version of our configuration thus looked like so (remember, there were many more settings than shown below):

# schema.ncl
{
  config = {
    swh_loader = {
      SWH_LOADER_CONCURRENCY = 4
    },
    swh_scheduler = {
      SWH_SCHEDULER_DEPLOYMENT = "local"
    },
    deduplicator = {
      DEDUPLICATOR_USE_NIX = 0
    },
    monitoring = {
      GRAFANA_INSTALL_DIR = "/some/dir"
    }
  }
}

In schema.ncl, we declared all the configuration settings that would appear in the final Bash config file. But this made the configuration neither safer nor much more user-friendly.

Step 2: Simple contracts

A first step towards a safer configuration was then adding basic contracts. A contract serves to validate data at runtime, and is much more flexible than a static type. For example, we can have a very basic contract String that checks at runtime whether a variable is a string, or we can have a more complex contract that checks whether a number is a valid port - a positive natural number within a certain range. For most of our settings, very basic built-in contracts already did a good job in making sure we didn’t use values that didn’t make sense:

• the above number of workers defined by SWH_LOADER_CONCURRENCY needed to fulfill std.number.PosNat (be a positive natural number),
• switches such as SWH_WINERY_ENABLED needed to fulfill the Bool contract,
• GRAFANA_INSTALL_DIR needed to be a String,
• etc.

A slightly less trivial contract type we used were enums: In the above example, you’ll see the string “local”, and another valid value would be “slurm” (to run the scheduler component on a Slurm node). The contract [| 'local, 'slurm |] then allowed only the values 'local and 'slurm for this and similar fields. When output as a string, these would turn into local and slurm (no single quotes) - just what we needed.

Nickel also allows you to write more complex custom contracts. We used such a custom contract to check for correct Boolean value encoding:

let BooleanNumber =
  std.contract.from_validator (match {
    0 => 'Ok,
    1 => 'Ok,
    value => 'Error { message = "Expected 0 or 1, got %{value}" }
  })

which let a field take a Boolean value encoded as 0 or 1, and made sure that everything else would result in an invalid configuration.

Finally, we could encode the fact that we wanted some settings to be non-optional by not assigning them a default value.

In the end, our example looked as follows:

# schema.ncl
{
  config = {
    swh_loader = {
      SWH_LOADER_CONCURRENCY | std.number.PosNat | default = 10
    },
    swh_scheduler = {
      SWH_SCHEDULER_DEPLOYMENT | [| 'local, 'slurm |]
    },
    deduplicator = {
      DEDUPLICATOR_USE_NIX | BooleanNumber
    },
    monitoring = {
      GRAFANA_INSTALL_DIR | String
    }
  }
}

Once a field is annotated with a contract or a type, the Nickel LSP will immediately be aware and validate the configuration. This also holds for autocompletion: Nickel’s LSP will know right away and propose you any new element such as a custom contract or a field. Running nickel export on the final configuration will then fail if a field has an invalid value. But of course Nickel can do more than just hierarchical records with contracts!

Step 3: Modularizing the configuration

In Nickel, you can merge two records using the & operator, and you can import a Nickel file into another Nickel file with import <some_file.ncl>. These two useful features allowed us to make further improvements: composing the final configuration from multiple files. For this step, the schema.ncl discussed above stayed largely as it was, only that we removed some defaults for values that we would set later in “overrides” specific for local or cluster deployments. We added the following modules that we finally all merged with schema.ncl:

• components.ncl: contained enable / disable switches for pipeline components,
• run_dirs.ncl: contained paths for temporary directories used during ingestion,
• software.ncl: contained paths to binaries required by the pipeline,
• deployment.ncl: contained switches that determine where certain components are deployed (login or compute node).

That way, a developer wanting to change an option knew directly where to navigate, and these files also reused common paths, prefixes and the like, while schema.ncl really only gave the schema and default values of fields that were rarely touched.

Finally, for launching a simple, uncustomized run of the ingestion pipeline, we asked the user to complete two short files:

• user_statics_template.ncl: fields that were user-specific, but were otherwise static across pipeline runs (e.g. the parent directory of all binary dependencies),
• local_template.ncl / cluster_template.ncl: filled pipeline environment and run-specific variables such as the data output directory, and the file with the list of origins to ingest.

With this setup, for “just running” the pipeline with sensible default parameters, the user didn’t have to complete a file with ~120 options any more, many of which shared common data, but only had to set a handful of mandatory, intuitive settings. And of course, a user could customize the configuration to their liking by adding overrides to their copy of {local,cluster}_template.ncl, without touching any other file.

That wraps up the concrete implementation! If you are interested in looking at the full configuration, you’ll find it on the ingestion pipeline’s GitLab repository.

Lessons learned and observations

A couple of factors came together that led to introducing the above improvements to the configuration using Nickel: the team working on the ingestion pipeline was struggling a lot with the configuration management; my (now former) colleague Yann, who is mostly behind Nickel, was looking for a project that could be a test case for real-world Nickel adoption; and last but not least, I was simply curious and wanted to learn something new and less “mainstream”. This section is thus a mix of observations through these three different lenses.

Impact on development workflow

Running the pipeline became much easier; getting a working configuration was a matter of minutes. This made a difference to new joiners and to engineers like me who had been working on that project for a while already, but had their SSD self-brick (twice!) and needed to start from scratch.

A big improvement over the existing workflow was the availability of an LSP server for Nickel. If installed and correctly hooked up to your IDE, writing a configuration in Nickel feels just as developer-friendly as working on the actual pipeline code with a good Python LSP. As mentioned earlier, the LSP validates your configuration on-the-fly and yells at you as soon as you make a mistake such as assigning a value of the wrong type or that doesn’t fulfill a certain contract to a field. Note that this also applies to non-trivial, custom contracts, such as, for example, making sure a list of users had unique user IDs. (There is a blog post on Nickel LSP’s contract evaluator, if you are interested.) Auto-completion is likewise available instantaneously, and of course the Nickel LSP supports formatting your code, too. This way, you get immediate feedback about the validity of your configuration, and changes are fast and comfortable to implement.

Adoption by other developers

It is always hard to push engineers to learn a new tool with the promise of replacing something that works with something that works better. Are we sure it will work better? Will it be worth the effort? Can I just get my work done and learn this new skill later?

In our case, it was possible to introduce many of the benefits of Nickel without initially forcing anyone to use it - for a while, we maintained two copies of the main configuration: the Nickel version and the original, plain Bash version. This was obviously a maintenance burden, which we softened by the mechanism described in the next section. In the meantime, though, we removed the Bash config file from version control for easier maintenance, and developers were then using the Nickel config to generate their personal configuration files. Validation of using Nickel came then notably by an engineer who joined the project after the first version of the Nickel config was finished. This engineer did not encounter major configuration issues, and was instead very glad to be able to spend their time smoothing out other rough edges!

Short detour: keeping team members up to date about configuration file changes

As mentioned before, for a while, we kept both the old Bash template config file and the Nickel config existing in parallel. Now what happened if a team member, who didn’t use the Nickel config yet, updated the Bash template and pipeline code with a mandatory config field? This change would be easy to miss for developers not using the Bash template any more. To avoid team members using outdated Nickel configurations, we set up an additional step in our GitLab CI pipeline: Whenever a commit was merged on the main branch that changed the Bash config file, an issue telling developers to update the Nickel config and to adapt any local configuration files was opened. The issue text linked to the specific MR that changed the config file.

This would actually have been a good idea way earlier - we often encountered the issue of local configuration file copies being out of date: engineers would copy the template config file to, say, config.local_simeon.sh, but then the upstream config.sh would change, rendering the engineer’s copy invalid. Missing a single small MR was then all it needed to get possibly hard to debug crashes when running the pipeline the next time. We imagine that this is a common problem, and even if you decide after having read this blog post that Nickel is not for you, perhaps you can still make use of this completely configuration language-neutral trick.

Learning curve

This is more of a subjective topic, but it’s difficult to report about introducing a new tool to an existing project without mentioning the learning curve.

Admittedly, it was a long-time goal of mine to learn Nickel, so the basic motivation was high already. Nevertheless, it turned out that getting to a first working (and useful!) version of the configuration was rather easy. After having grasped the basic idea of writing a nested record and the trivial, but painful task of copying and splitting up the existing configuration into useful sections, the only slightly difficult part was writing the custom code to output the correctly formatted Bash file:

let config_string = config
  |> flatten_merge
  |> std.record.to_array
  |> std.array.map (fun { field, value } => "export %{field}=\"%{escape_single_quotes value}\"")
  |> std.string.join ("\n")
  in [...]

As an imperative programmer by training who’s been functionally curious for a little while, that whopper was the one part of “real” coding that required some help from the pros. I had a rough idea of what to do, but the required functional way of thinking turned this into an actual effort. Yann again stepped in to help me set up the above pipeline that flattens the hierarchical record (which is the most difficult part), converts data structures, and finally constructs the correctly formatted lines. It was then relatively straight-forward to add basic types and mark fields as required, which yielded a big improvement over editing a Bash config file template by hand.

When I became more independent after a little while, the Nickel Discord channel with very responsive and helpful members was of great help. But if your use case requires a common configuration format supported natively by Nickel, then there’s a good chance you can get pretty far without writing a single recursive function!

Conclusion

In this tutorial / use case / experience report / blog post hybrid, I showed how we were able to significantly improve our configuration management using basic Nickel. I hope this helps you think about what a simple Nickel use in your own project could look like, and what gains and efforts you might expect. As always, if you have any questions, insights, or are curious about the CodeCommons project or our Nickel adoption, don’t hesitate to reach out in the Discord channel or via the CodeCommons ingestion pipeline GitLab project.