With Nix, Bitcoin Projects Can Accelerate Contributions And Deployment
This is an opinion editorial by niftynei, a developer educator who runs the Base58 Bitcoin protocol academy and an organizer of the upcoming, Nix-focused Bitcoin++ conference.
When you want to install a new app on your computer, where do you get it from?
Deciding which source to use is a problem that’s surprisingly difficult to solve, but it’s one that every computing platform struggles with.
Before the internet was widespread, new software “packages” were distributed on hard media, like CDs or USB sticks. If you wanted to get the latest version of your computer’s operating system (OS) or the newest version of Bitcoin Core, for example, you’d need to buy it physically at the store or have it mailed to you. You’d then load the disk or USB stick into your computer, and make a copy of the new software onto your local disk.
As a developer writing code, you’d have to physically hand out copies of your programs to friends and family. Or you’d work for a large corporation that had relationships with companies and computer stores, which would figure out how to get the software you wrote distributed to their clients and customers.
With the advent of always-connected computers, the landscape of software distribution forever changed. Now, instead of having to physically pick up a copy of the bits in a program, you can easily download a copy of it from the internet.
This made it much easier for new developers to write apps that anyone can download. The problem now, however, is answering the question: Where on the internet should users download the new software from?
The Hacker’s App Store
The way that mobile phone ecosystems solve the problem of new application distribution is through a program on your phone called the “App Store” (or the “Google Play store” on Android).
Developers upload the latest copy of their apps to either Apple’s or Google’s servers. Those companies then send out a silent notification to all of the phones that have the app installed, and the app gets updated in the background on users’ phones, without the users’ involvement.
If you’re a user and you want to add a new app to your phone, in most cases you’ll be able to find it in the App Store, and download it from there.
But not all software is available as a mobile phone app. Some software projects are desktop apps or tools that developers use to write new pieces of software. These applications need a distribution channel too. On desktop operating systems, we tend to call the “App Store” software by the term “package managers.” They manage the packages of software that you download onto your computer.
Modern package managers consist of a central server, sometimes called a repository, that can keep track of the latest version of the app, and a client-side application which runs on the package user’s computer.
If you’re a developer, you’ve probably run into “homebrew” or “apt-get install” somewhere in your life. “Homebrew” and “apt” are package managers.
In the way package managers work, the client-side application will check in with the central repository occasionally to see if there’s any new updates for any of the currently-installed packages. Or, if you have the right OS and the right permissions set, maybe your computer will automatically run these checks, download and install the latest version of an app.
(As an aside, you can add other sources of software packages to your local package-manager client. If a developer self hosts their apps instead of uploading them to the central repository, you’d need to add their self-hosted website to the list of places to look for updates in your own package-manager client app.)
What If We Could Reproduce Everything?
Nix is a package manager that aims to bring reproducible builds to all of the packages that it helps to distribute.
Nix solves the problems with software distribution by keeping detailed track of all of the inputs to a build, uniquely identifying every input and using the same ones as the original builder did. This ensures it’s able to build the same thing every time and on any machine.
Nix hashes every input and output of a build; this makes it easy to identify when an input has changed. When the inputs to a build change, the outputs are guaranteed to also change.
One big problem with reproducible builds is that different users will have different versions of the inputs that a program needs to be built with. Every new program that you build on your machine might need a different version of its input in order to be reproducible.
Nix fixes this by keeping track of builds by hash lists, and only using inputs that match the list of hashes it expects. The tradeoff is that Nix uses a lot more disc space and can often take a bit longer to download, as it’ll need to download the correct inputs to a binary build process if you don’t have them on your computer already.
I like to joke that Nix was only made possible thanks to Moore’s law, which means that we now have enough extra disk space to not need to worry about having multiple copies of the same program, just in slightly different versions.
So, Nix makes it possible to distribute a reproducible build of your software much much more easily, which means that a user can reliably rebuild your software on their machine and have it just work, no questions asked.
As a small aside, note that what “reproducible” means for a Nix build is that “if a package works on one machine, it will also work on another.” This isn’t the same as the reproducibility that Bitcoin Core developers try to guarantee for official Bitcoin binaries, which have a separate process to check that they’re bit-for-bit identical.
Nix And Bitcoin
Distributing open-source software in a way that’s easy for end users to get running, and is reproducible on any system, is a very useful and important feature. It makes it easier for anyone to start using the project and also easier for new developers to start contributing back to the project.
A few years ago, a set of intrepid developers launched a repository of the most common Bitcoin software projects that are packaged up as Nix packages and modules.
Packages make it much easier for anyone using the Nix package manager to install and start running a Bitcoin node, on almost any architecture. You don’t have to worry about installing the right dependencies before running the build commands, and the project will build from source on your own machine. It’s pretty ideal in terms of quickly being able to download, build and run a Bitcoin node from scratch.
The Nix-Bitcoin repository provides Nix “modules,” which are configurations for software applications that are meant to work on NixOS. The Nix-Bitcoin modules allow anyone who’s running NixOS to reproduce a build of Bitcoin Core and other Bitcoin open-source projects, and quickly get them running on their machines with minimal fuss.
In an interview for this article, Jonas Nick, one of the lead maintainers of the Nix-Bitcoin project, had the following to say about using the project:
“The main benefit that Nix-Bitcoin gets from Nix/NixOS is declarative config management. Configuring your system using the Nix programming language allows for abstraction, so you can modularize your logic to reuse and test individual components. This allows managing complexity, which inevitably occurs when building systems, and helps to build a correct and secure system. For example, a lot of the security features that Nix-Bitcoin provides would be very annoying to maintain if we didn’t have the module system. Reproducibility in NixOS also helps a lot with updates, because we know exactly what is running on the systems of our users. Finally, with NixOS can also get errors at evaluation time instead of at runtime.”
Bitcoin Core isn’t the only project that’s packaged up for distribution on Nix.
There’s also Core Lightning, BTCPay Server and a number of other projects. For the most part, these projects are added and bundled up for distribution on Nix by fans of the Nix package-management system, not the developers who are writing the software. For a more complete list of available packages, check out the “modules” list in the Nix-Bitcoin GitHub repository.
Why Not Nix Everywhere?
If Nix is such a great change to the way that Bitcoin projects are built and shipped, why doesn’t everyone use it? The truth is that Nix is a big project that’s a bit difficult to get into. The learning curve for using Nix is quite steep, and the paradigm of running Nix as a package manager can be quite different than most are used to.
NixOS started as a research project by Eelco Dolstra in 2003, and has slowly been building a following and user base since then.
Bitcoin++ Berlin: Expanding The Use Of Nix In Bitcoin
By organizing the upcoming Bitcoin++ conference in Berlin, we aim to change this. Bitcoin++ is a developer-focused conference series which focuses on different topics in the Bitcoin developer space. Originally started in Austin, Texas, in 2022, the most recent conference in late April 2023 focused on Layer 2 technologies in Bitcoin, such as Ark, the Lightning Network
and validity rollups.
The upcoming Berlin conference, planned for October 6 and October 7, 2023, aims to be a “Nix-pilling” event for the Bitcoin developer ecosystem. The core maintainers of the Nix-Bitcoin project on Fort Nix will be in attendance (including Nick, Pavol Rusnak and Eric Sirion, to
name a few). The two-day event will be full of hands-on workshops and lectures
designed to teach developers from across the ecosystem about how to use Nix to package
their projects and better bundle their development dependencies so that new
contributors can start coding and contributing faster, as well as demonstrate how fast and easy it is to deploy software using modules on NixOS.
Additional Article References
- Reproducibility with Nix
- Interview with “nixbitcoindev”
- Where to start
- Why Nix-Bitcoin, for Bitcoiners
This is a guest post by niftynei. Opinions expressed are entirely their own and do not necessarily reflect those of BTC Inc or Bitcoin Magazine.
