At Tweag, any employee can pitch proposals for internal projects. This is how we got content-addressed derivations in Nix, the formal verification tool Pirouette, or a yearly book budget to support continuous learning.
In May 2022 our Chief Architect Arnaud Spiwack (@aspiwack) accepted my pitch for “The Nix Book”, agreeing to fund work on improving Nix documentation and onboarding experience for three full months. This is a comprehensive report on what happened since and what I learned from it.
tl;dr
The goal of the project was to improve Nix onboarding and documentation experience to increase community growth by writing “The Nix Book”.
In short:
- Failing fast failed. The project took a different course than intended, and this is probably good.
- We are years away from “The Nix Book”.
- Writing is hard – many significant improvements are underway, but it needs time.
- Science: it works. Usability studies, surveys, and expert insights are leading the way. Also: Cognitive biases are lurking everywhere.
- The challenge is social, not technical. A documentation team was formed to tackle this.
- We should focus on enabling occasional contributors and help them grow into maintainers.
Since it is a very long text, each section is designed to be read on its own:
- Backstory: how this project came into being
- Key results: summary of notable contributions
- Additional results: complete list of concrete changes and ongoing discussions
- Measuring success: elaboration on what went well and how to do better in the future
- Next steps: my personal outlook on the future of Nix
The report in full detail will mostly be interesting for active contributors or those inclined to contributing to the Nix ecosystem. It may also be interesting for people who are working or want to work on a software project that is in a similar situation as Nix.1
Backstory
My work on Nix documentation started in March 2022 by participating in the regular Nix UX meeting. There I encountered Nix contributor and developer at Obsidian Systems John Ericson (@Ericson2314), who set out to document Nix’s architecture. While reviewing his pull request together, we quickly agreed that I should take the editorial lead. I learned a lot about Nix internals during our review sessions we had over multiple weeks, trying to sort out the facts and to present them in a consistent, readable manner.
The insights we found during these discussions helped refine the idea that got me excited about Nix in the first place: making software build and run is no different from writing the software to begin with – it’s all just programming. We can do it effectively or clumsily, depending on the (mental) tools we employ.
I was eager to write down what apparently was there all along, but somewhat hidden between the lines in Eelco Dolstra’s PhD thesis and Build Systems à la Carte. This led to my blog post Taming Unix with functional programming, which illustrates how we can think about building and deploying software in terms of programming language theory.
Most importantly, the principles underlying Nix and many of its mechanisms are amazingly simple – it’s just that most often they are not explained well. This discrepancy between contents and presentation in the Nix ecosystem always struck me as painful… and unnecessary.
During this time the desire to extend the scope of improving Nix documentation and learning material culminated in an internal pitch to compile and write what I would boldly but tentatively call The Nix Book.
The pitch had a fairly broad mission statement2:
Goals
- Improve the autodidactic Nix onboarding experience to increase community growth
Tasks
- Write a book actually explaining Nix and its surrounding ecosystem at a high level of abstraction
- Overhaul the Nix manual to make it a focused technical reference
- Improve discoverability of existing learning material
- Lead a Summer of Nix 2022 project to help achieve this
Success of the project hinged on the extent to which existing material already served its purpose, and whether attempts to improve it would be fruitful. This report shows what became of “The Nix Book”.
Key results
-
Taming Unix with functional programming
The article took a while to fully develop, but the result turned out to be highly successful with over 13,000 readers (making it the most viewed Tweag blog post so far) and a day on the Hacker News front page.
-
Detailed 2022 Nix community survey results
I needed more information than available in the original report, so I reached out to the Nix marketing team to get my hands on the raw data. From there, I compiled graphs for all quantitative questions. While this only represents a portion of what’s going on in the Nix community, it is evidence. This is a significant improvement over anecdata and intuition and can be used to more confidently reason through strategic decisions, such as prioritizing Nix over NixOS for onboarding.
-
Multiple discussions led to a conclusion that a team should be formed to serve as a sounding board for the effort.
- I had encounters with Nix author Eelco Dolstra (@edolstra), NixOS contributor and NixOS Wiki maintainer Jörg Thalheim (@Mic92), Cachix author Domen Kožar (@domenkozar), and my colleagues, Matthias Meschede (@MMesch) and Rok Garbas (@garbas).
- Encouraged by the simultaneous developments around reforming the NixOS Foundation, I had additional exchanges with my colleagues Théophane Hufschmitt (@thufschmitt), Silvan Mosberger (@infinisil), Tweag’s Founder Mathieu Boespflug (@mboes), Flox’ CEO Ron Efroni (@ron), and TVL developer Vincent Ambo (@tazjin).
Most importantly, the team provides a widely visible point of contact for potential contributors.
-
2022 Summer of Nix documentation stream
Led by Matthias (@MMesch), we organized a part of the Summer of Nix program dedicated to improving documentation. In addition, we drafted a line-up of presentations, one of which was also on Nix documentation.
-
How to contribute to documentation
Based on discussions, feedback, the practice of helping contributors, and the need to accommodate Summer of Nix participants’ work, I drafted a contribution guide to documentation in the Nix ecosystem to have as a reference.
-
10 sessions with (absolute or relative) Nix beginners of different software development backgrounds quickly produced some observations:
- People love control and reproducibility.
- Developers just want to get things done and do not care how it works.
- Engineers usually care most about one specific programming language or framework.
- People do not read, only skim.
- nixos.org navigation often does not meet user needs.
- Information about the Nix ecosystem is perceived as being highly dispersed and disorganized. Confusion and disorientation quickly kicks in and often results in “tab explosion”.
- The learning curve is perceived as extremely steep.
- The Nix language is easy for Haskell users, and obscure to almost everyone else without appropriate instructions.
-
Identified as the highest-value objective by Rok (@garbas), based on a comprehensive comparison of existing Nix language tutorials, and already partially validated by testing it with beginners, it should hopefully become the centerpiece of future efforts to learn and teach Nix.
-
No Nix Book this year
As originally envisioned and agreed upon by the Nix documentation team, work should continue based on nix.dev and nixos.wiki. Contrary to what was originally envisioned it should happen in a strictly incremental fashion, slowly migrating material towards more curated resources and as close to the source code as possible.
Additional results
Contributions to merged pull requests
mkDerivationlongDescription should be MarkdownContributions to unmerged pull requests
Ongoing discussions
nix develop output rendered on web sitenixos-common-styles repositorynix-shell installation by defaultmeta.longDescription from markdownnixpkgs manual contributing section are redundant.deb or .rpm
Measuring success
Some of the following insights appear almost trivial in retrospect. Yet, contributing to a major open source project is a set of skills on its own, one most people don’t learn at university or work.
As a high-level summary, one could say:
1. Do your homework first.
This presents a dilemma:
- Becoming competent at making improvements requires time which will not be available to actually making those improvements.
- Trying to make improvements without the necessary competence requires maintainers’ time, which was highly limited to begin with, or may backfire by making matters worse.
Therefore, what follows is an attempt to share experience, and, based on that experience, proposals to deal with the above dilemma.
Again, as a high-level summary:
2. Make it easier for others to do their homework.
Failing fast
Following best practices, the pitch contained abort criteria to avoid the sunk-cost fallacy:
- until 2022-05-31: Summer of Nix 2022 project proposal rejected by organization team (excluding Tweag staff)
- until 2022-05-31: not enough participants to cover planned tasks
- until 2022-07-15: preliminary questionnaires demonstrate satisfactory effectiveness of existing material for defined learning goals
- may leave room for conducting targeted incremental improvements
- until 2022-07-15: setting up surveys and collecting results shows that timeline is not realistic
- until: 2022-07-31: elaborating outline and surveying existing material shows that timeline is not realistic
- may leave room for cutting scope
While the Summer of Nix proposal was accepted (1) and user testing showed desperate need for improvements (3), it was not clear how many Summer of Nix participants would actually want to focus on documentation (2) until the end of July when the program had started. We estimated that of 20 participants, multiple would contribute to documentation in one way or another. It later turned out that only one would work on documentation specifically, and a few others would decide to write a blog post about their ongoing work.
At this point it was already quite evident that a more incremental approach would be inevitable. Both the evidence (4) and envisioned scope for the book (5) were unambiguous in that there was an order-of-magnitude divide between the possible and the desired.
By the beginning of July, the newly founded Nix documentation team decided to focus on more immediate problems and only briefly discussed “The Nix Book” as a long-term vision. Cutting scope occurred naturally: from now on, the focus would be on just the part up to teaching the Nix language.
Around that time I updated Mathieu (@mboes) on the current state and changed strategy, where we re-iterated on the cost-benefit estimate of spending internal budget and project goals – improving Nix onboarding and increasing Tweag’s visibility.
Note that I originally estimated the project to take six months, not three or four. While the changed schedule partially invalidated the relation between estimate and objectives, the time constraints forced me to keep even stricter focus on priorities. At the same time, clearly not being able to deliver on the vision due to the problem size removed most pressure with regard to producing specific artifacts and, thus, any temptation to cut corners.
Instead of spending three months, I spent four months. High time to evaluate.
What went well
Community building
Taking the time to listen and talk to people helped a lot with understanding the problem space and honoring Chesterton’s fence. Getting key people on board helped to build commitment and momentum as well as weed out bad ideas through critical discussion. It also meant that actual changes have to go through at least partial consensus, which requires each of those changes to be fairly small.
In the original pitch, I assumed that I would have to rely on Aaron Swartz’ Wikipedia authorship principle (which suggests that most open source contributors engage only occasionally and typically work on cosmetics, while the substance is provided by regulars). The assumption turned out to be true and Swartz’ findings were confirmed again.
Providing a central point of contact, naming directly responsible individuals, contacting potential contributors immediately, and actively setting examples appears to have resulted in a modest but noticeable increase of attention towards documentation issues, as well as many small and multiple significant pull requests.
User studies
Immediately starting with user studies quickly helped pinpoint concrete user needs and some obvious issues, and either validated or debunked some preconceptions that (at least as far as I perceived) had been discussed mostly based on intuition and oral history. It will still take more time to sort through them again and match the notes to GitHub issues and pull requests. This is to help (1) maintainers to keep track of what can be done and (2) whoever consults the session notes to keep up with what has already been resolved. These studies should continue if possible: at the least, to validate new material (as for example with the new Nix language tutorial) and measure the reduction in onboarding time after improvements have been implemented.
Increasing visibility
Publishing regular updates such as meeting notes, participating in ongoing discussions, and linking to relevant posts, issues, and pull requests seem to have increased awareness of the trajectory of the Nix ecosystem and of what Tweag is doing.
Getting involved consistently and staying active in a constructive manner helped a lot.
All feedback from within the community so far has been positive. Beginners and regular users found the changes in organization and the specific documentation work we got done helpful. Expert users and contributors are vocally happy about the efforts. (At least those who I did not annoy by nagging too much about phrasing and terminology.)
Note: there are also outside voices on the internet who doubt that this (or any) effort will lead to a significant improvement in terms of user experience.
Overview and visualization
Presenting high-level summaries and diagrams at the very beginning of introducing people to various topics was perceived as very helpful, both in the usability tests as well as in multiple informal interactions. It increases the readers’ confidence, and allows them to set realistic expectations before going into details. This is supported by scientific evidence.
I think there should be many more such overviews at the top of learning resources and reference materials.
Examples:
- Nix store architecture
- Nix concept map
- Modeling files and processes in a purely functional fashion with Nix
- Overview of using the Nix language in practice
- NixOS Wiki: Data flow of overlays
A particularly inspiring example of making complex problems accessible through visuals is Life cycle of a Poetry project by Attila Gulyas (@toraritte).
Brute-force analysis
Nix, Nixpkgs, and NixOS have a multitude of features and obscure corner cases which are barely, badly, or not at all documented. There are many resources of varying quality which have overlapping contents. The only way to get on top of things, apart from experimentation or diving into source code, often turns out to be research and an exhaustive analysis of prior art: to avoid the Dunning-Kruger effect (”I can easily do better.”), to account for Chesterton’s fence (”This is not good and can be removed/must be changed.”), and finally to simply get things right.
Incorrect documentation is often worse than no documentation.
— attributed to Bertrand Meyer
Improving over the current state is only reliably possible if the current state, and how it came about, is known.
Despite thorough initial overview it took me a while to even stumble upon relevant materials after sitting down again and again to dig through countless Discourse threads and NixOS Wiki pages:
- Nix documentation task force (NixCon 2019)
- Make Nix friendlier to beginners (NixCon 2015)
- Reading the Nix language (NixCon 2019)
Oftentimes, such research reveals underlying problems (as opposed to mere symptoms) or what caused those problems in the first place.
This kind of due diligence takes a lot of time and concentration, and can be very challenging work. There is also an enormous overhead of preserving insights for the future. However, I am convinced that if no one else has to repeat the effort, the results are worth it in long run. Each time I tried the brute-force approach, the quality of work turned out to be convincing as opposed to my other, less well-prepared proposals, which (rightfully) received substantial headwind.
Scientific method
Leveraging insights from scientific evidence (unsurprisingly) proved to be highly effective, and as a side effect removed most uncertainty about procedure.
The most important resources which shape my day-to-day documentation work:
-
Practical advice on effective teaching and learning, backed by broad and deep evidence.
The best-written and probably most important book I have ever read.
-
A framework for structuring software documentation around user needs.
-
A set of guidelines to write clearly in English.
This is meant quite literally – I refer to each of them every day, one way or another.
Many thanks to my colleague Andrea Bedini (@andreabedini) for recommending How Learning Works by Ambrose, et al. – it keeps changing my life to the better. I have heard multiple times that Visible Learning is the state of the art in learning science. For teaching in the context of software development, I also recommend the ideas behind Software Carpentry.
Lessons learned
Gather more context in the beginning
Looking beyond one’s own backyard by collecting testimonials from other software projects would have been helpful to more quickly see the big picture and, as a result, identify the most pressing, underlying issues. While I talked to many Nix experts and did much research on internal proceedings, I spent very little time on how other projects approached similar problems and which strategies were successful.
Recommendation:
Talk to people who solved similar problems in different contexts.
Example:
Ron (@ron) interviewed several leaders of open source foundations when preparing the NixOS Foundation reform, which surfaced very helpful, non-obvious insights – and also tales of caution.
Avoid planning fallacy
The idea one person could get even close to a complete book in less than half a year was, while not fully serious, quite presumptuous, and a pathological case of planning fallacy. It was not evident to me in the beginning, but the ecosystem is simply too large, the problems too numerous, and the high-level tasks too big to tackle at once.
Things are moving, but they are moving very, very slowly. Writing the architecture documentation chapter, which covers at most 60% of the topics that it would need to be considered comprehensive, took 8 weeks of wall-clock time. Writing a Nix language tutorial took 4 weeks of wall-clock time. This is extremely frustrating, but unavoidable due to lack of prior experience, some essential complexity, much accidental complexity, external factors considered in the other sections, and – of course – planning fallacy.
Recommendation:
Find out how much time it took to produce comparable results, and take it seriously.
Example:
The nix.dev version history goes back to 2016, and has been actively developed on the side since May 2020. In the time-span of about 2 years, 12 original articles were produced, i.e., one article per two months. After the fact, this matches my own experience very closely.
Identifying low-effort high-impact tasks
The Nix ecosystem is large and fragmented. There are many people involved, each with different – and sometimes diverging – interests. It is not enough to ask users what they need, because they will usually instead answer with what they want.
I spent some time dabbling at working on assorted issues before converging on a more systematic approach.
Recommendation:
Take enough time to identify issues (user studies) and sort them by effort-impact ratio (brute-force analysis) before delving into work.
Examples:
- The “Writing Nix Expressions” chapter in the Nix manual was a pretty bad introduction to the Nix language, throwing many people off early on (including myself, back in the day). Due to the sheer amount and length of other Nix language tutorials, it was not clear before actually working through all of them that this specific one really did not contain anything uniquely valuable. Removing the section was quick and painless.
- The Nix Pills cover advanced topics and have been reported to be confusing to beginners (including myself, back in the day) many times. The problem was that they were touted as beginner material in many places. Reordering recommendations and rewording the description appears to have helped substantially. (Although better guidance is still needed, see below.)
Make incremental changes
My colleague Clément Hurlin (@smelc) already wrote about this in Getting Things Merged. In an open source project’s community, where essentially everyone can be considered a volunteer, reviewers’ time is even more limited. There is no chance of getting a large pull request merged without having close allies among maintainers – people naturally won’t do more than take a glance, it’s too much work.
This imposes a significant additional cost on authoring pull requests, which has to be taken into account. One has to keep the big picture in mind while only presenting the next obvious step towards a vision. So far even merging a rendering of the vision appears to be too large a task.
On the other hand, small changes keep cognitive load manageable and allow for easier switching between tasks: simply because small tasks get finished quickly.
Recommendation:
Never stop asking the question, “What is the smallest possible change required for a tangible improvement?”
Examples:
- Unfortunately the pull request documenting Nix architecture still has not been merged properly. Therefore, it is not yet visible in the Nix manual. Although it is limited to the parts I felt confident publishing, it is a large addition. It will have to be split up into multiple parts to ease review.
- The Nix language tutorial is not finished yet. It takes 1-2 hours just to work through it – not to mention the time needed to make a review. Good progress so far was only possible due to Silvan Mosberger’s (@infinisil) persistent involvement and patient reviews.
Focus on the basics first
The usability study was particularly helpful in demonstrating the gap between what we may wish to have and what people actually need to succeed.
The problems people got stuck on were often trivial, such as not understanding a term or not finding a crucial bit of information to continue. This could be addressed with much less effort than required for creating full-blown tutorials or meticulously working out precise reference material.
Recommendations:
- Reorganize the nixos.org web site.
- Establish materials to help guide beginners across different problem domains in the ecosystem.
- Make improving documentation more appealing for contributors.
- Provide guidance to navigate each source repository.
Enable contributors
I was expecting to adjust the original goals and targets during the process, since observing actual users dealing with the material would unfold further requirements to guide my work.
However, the usability study results, as well as my own experience, showed that it is much more difficult to improve upon the overall situation than it originally appeared.
I believe coordinated incremental improvements will be more effective than having a few people attack large problems at full-steam. It’s not just that the sheer number of entangled issues is overwhelming, but also that Nix experts are subject to the curse of knowledge. Nix beginners, seeing our work with fresh eyes, have time and again proven invaluable allies by pointing out and often themselves addressing problems that tend to become invisible after getting used to them.
Systematically building momentum, setting examples, creating a culture, and enabling volunteers to contribute appears much more promising and is already bearing fruit.
Recommendation:
Focus future efforts on enabling contributors, by providing comprehensive guidance into the process of developing the Nix ecosystem.
This is not to say to stop improving the onboarding process for beginners. Becoming a contributor should instead be considered part of that process.
Examples:
More blog posts, tutorials, guides, and reference documentation are in the making.
The source code of beginner-oriented materials is opaque to outsiders.
Next steps
I agree with Tweag’s VP of Engineering Steve Purcell’s (@purcell) and Flox’ CEO Ron Efroni’s (@ron) assessment that Nix is at the inflection point towards a trajectory to mass adoption. At the same time I fear that we as a community are not ready for the corresponding influx of users and potential contributors – both in terms of documentation and organization.
Teaching in person is, of course, the most effective way of getting people into Nix, and Tweagers enjoy this rare privilege by default. But it does not scale. To handle more than 0.5%3 of the world’s 24 million software developers, we have to leverage that most of them are self-taught.
My current estimate to get Nix documentation and learning material into a shape that allows for growing the community to scale is on the order of multiple person-years. For comparison, this is how long it took to get flakes and the new CLI “almost ready”. Or how long it took to create “The Rust Book”, which was started end of 2015 and saw intense development activity for over two years.
Taming and helping to navigate the (mostly accidental) complexities of the Nix ecosystem continues to be a huge undertaking. It requires dedicated work and coordination, which is mostly about learning and teaching, communication, and social problems. Having been in both roles, I am more than ever convinced that volunteers are not sufficient to handle this on their own, and that we need more paid regular contributors.
With the new NixOS Foundation board and its corporate backing, in principle we have all the means to systematically grow the Nix pie for everyone. I am looking forward to the Foundation’s board delivering on their promise to develop a roadmap and to enable teams by providing organizational structure, the necessary permissions, and leadership’s attention. In my opinion, part of this endeavour should be a long-term funding scheme for ongoing development and maintenance targeting key objectives, including improving onboarding and documentation.
I would love to help with setting this up. And, while I don’t care who does the job as long as it’s being done, I thoroughly enjoyed doing what I did in the past four months, and would just as passionately continue if there was a possibility.
- Haskell may be a good example. Nix and Haskell, both as software projects and communities, share many features (reliability, expressive power, highly motivated contributors, pluralistic governance) and problems (learning curve, documentation, diverging feature sets). Maybe not coincidentally, there is also a significant overlap of users and contributors between the two.↩
- Most of the original pitch is reproduced in the Summer of Nix 2022 project proposal. The quote is slightly reworded from the tl;dr to make it consistent with the proposal’s contents.↩
- This number is based on informal estimates that at most 100,000 people have heard of or are using Nix.↩
About the author
After an artistic career in education, film making and graphics design, Valentin grew into informatics, where he quickly developed a mathematical and engineering mindset as well as a passion for functional programming, reproducibility, and effective communication.
If you enjoyed this article, you might be interested in joining the Tweag team.
