One core value of Tweag is its dedication to the open-source community. Although our interests and expertise have become significantly broader over the years, our love for immutable, composable and typed architecture have made functional programming and programming languages in general an important part of our DNA. This long-standing activity was formalized last year as the Programming Languages & Compilers Group. The PL&C group has been busy in the second quarter of 2024, and this post is a summary of what we’ve been doing.

Our involvement varies depending on the availability of each team member and client engagements; if some projects might seem idle, this is usually just temporary. All projects appearing below are actively developed.

Rust

We have a bunch of Rust engineers at Tweag, and some of them have recently started to contribute to the Rust package manager cargo, which is a key part of the ecosystem. This is a choice motivated by our interest and expertise in build systems, our love for cargo and the need for contributions there.

Currently, in a cargo workspace spanning multiple crates, we can only publish one crate at a time, in dependency order. It’s been a long-standing issue as well as a focus area to be able to publish all the crates at once. To get there:

• Joe Neeman implemented local registry overlays; which make it possible to package a crate even if its dependencies aren’t published yet. (#13926)
• Joe Neeman and Tor Hovland added support to package all the crates in the workspace in a single command. Crates must still be published one at a time. (#13947, #14074)

Typically, cargo update is used to update dependencies to the latest versions that satisfy the version requirements defined in Cargo.toml. If you wanted to update the version requirements themselves to the latest available versions, you might use the 3rd party command cargo upgrade from cargo-edit. Another focus area for cargo is to bring this capability into cargo update.

• Tor Hovland implemented cargo update --breaking, which will upgrade the version requirements in Cargo.toml if there are breaking changes. (#13979, #14049, #14259)
• Tor Hovland implemented support for making breaking upgrades when doing a specific version update with cargo update --precise. At the time of writing, this isn’t merged yet. (#14140)

Furthermore, we have contributed with various smaller fixes: (#13874, #13886, #13960)

Haskell

GHC

GHC is the de facto standard Haskell compiler. Several of our GHC engineers are currently working on making it easier to seamlessly interface GHC with external build tools, such as Buck2. Most of our work on GHC is on behalf of Mercury.

• Torsten Schmits and Cheng Shao have been working on supporting bytecode linking for Template Haskell dependencies in single-file compilation mode, which allows external build systems to take advantage of the performance improvements for Template Haskell that Cabal builds have been enjoying for some time now. (GHC MR 13042)
• Torsten Schmits and Cheng Shao have implemented a way to print dependency metadata for a set of Haskell modules as JSON, for which Buck2 had to parse GHC’s generated Makefiles before. (GHC MR 11994)
• Sjoerd Visscher added single-file processing support to Haddock, allowing external build tools to incrementally (re-)build documentation for individual modules without compilation. (GHC MR 12707)
• Torsten Schmits has been working on performance improvements for dependency analysis. He wrote a patch (for -Wmissing-home-modules) that replaced a quadratic algorithm and reduces the startup time in a project with 10,000 modules by over a minute. He also wrote a WIP patch that introduces parallelism into the first phase of dependency graph computation, promising a reduction of the duration of this phase by a factor of 4 in some of our projects.
• Cheng Shao looked into GHC’s ARM64 Windows support (GHC issue) and made an MVP that can cross-compile simple Haskell programs to ARM64 Windows executable from Linux.
• Cheng Shao performed GHC housecleaning and removed legacy code paths related to 32-bit Darwin/Windows (GHC announcement).
• Cheng Shao has been working on Template Haskell support in GHC’s WASM backend (GHC issue). A WASM dynamic code loader based on LLVM’s WASM shared library ABI is being prototyped at the moment; once it’s finished, remaining Template Haskell support should be straightforward.
• Joseph Fourment joined us for an internship, during which he researched and implemented the initial steps towards more general and flexible let-bound types, which he wrote about on this blog. This effort introduces the capability for GHC to reuse in-memory data structures for type subexpressions that are shared by multiple larger types, promising substantial performance improvements when compiling programs with complex type-level computation.

Liquid Haskell

The Liquid Haskell contributions from Tweag are spearheaded by Facundo Domínguez. Liquid Haskell is a verification tool that allows you to write additional lightweight formal specifications for your Haskell programs. These specifications are then checked by the tool which discharges the proofs to SMT solvers, so that you don’t have to do it yourself.

• Facundo Domínguez released a new version of smtlib-backends with updates to documentation. smtlib-backends is a library to interface with SMT solvers via SMTLIB.
• We welcomed Jonathan Arnoult, a new intern who will work on reflecting functions away from their definitions.

Nickel

Nickel is a configuration programming language developed by Tweag aimed at infrastructure-as-code, build systems, Nix, or any complex system that needs to be configured and where YAML, JSON, TOML and the like aren’t sufficient.

• The Nickel team released versions 1.6 and 1.7.
• Yann Hamdaoui revived the previously stale nickel-kubernetes repository. In combination with updates to json-schema-to-nickel, we are able to auto-generate Nickel contracts (think “schemas”) for all Kubernetes resources at any given version.
• Yann Hamdaoui implemented pattern matching extensions (constants, wildcards, guards, arrays and or-patterns) following the addition of structural ADTs aka enum variants (#1897, #1904, #1910, #1912, #1916).
• Joe Neeman experimented with package management for Nickel and wrote an RFC to discuss and decide on various points in the design space.
• Yann Hamdaoui reworked the contract system quite a bit to better support missing boolean operation on contracts, such as JSON Schema’s any_of and not. Beside boolean operators, this rework also made it much more ergonomic to write and compose custom contracts with custom error reporting. (#1964, #1970, #1975, #1987, #1995)
• Yann Hamdaoui added span information for data imported from TOML to make validation errors more precise. (#1949)

Topiary

Topiary is a lightweight universal code formatter that relies on Tree-sitter grammars to handle a variety of languages. It has been developed by Tweag, and is used under the hood by the Nickel language to format code, but is also a standalone tool.

• The Topiary team released version 0.4.0, “Exquisite Elm”. Highlights include improved Nickel formatting support and new CSS formatting support from external contributor Eric Lavigne.
• Erin van der Veen moved all of Topiary’s dependencies to either published or vendored crates; that is, either those available on crates.io, or subsumed directly into our codebase. This prepares the ground for future releases of Topiary to crates.io, where projects with ad hoc dependencies (such as those direct from GitHub) are forbidden. (#672)
• Christopher Harrison feature gated language support, mainly to allow the development of experimental language formatters without impinging on the supported formatters. This ties in with his development (still in progress) of formatting rules for Pact, the smart contract language for the Kadena blockchain. (#711, #713)
• Erin van der Veen made a number of background, “quality of life” improvements, such as transitioning from TOML to Nickel for Topiary’s configuration. This allows for less complicated merging using Nickel’s record merging, especially in the future when Nickel implements custom merge functions. Another goal of this PR was to evaluate the use of Nickel as a library, which was a great success! (#703)

Closing words

The PL&C group will continue to contribute to the projects mentioned above in the near future. Stay tuned for the next quarterly report! In the meantime, you can find Tweag’s open source portfolio on Github and come chat with us on our Discord dedicated to our open-source activity, be it as a user, as a potential contributor, or simply to satisfy your own curiosity.