How We Sped Up Nix Package Installs in Devbox

For this week’s post, we’ll describe how we keep Devbox shells fast and performant when adding or updating new packages. This will dig into some of the Nix internals behind Devbox, but shouldn’t require a huge amount of Nix experience to follow along.

How Nix Finds and Installs Packages

Let's start by looking at a simple Nix flake. The example below (which is based on the default go-hello flake template) creates a development shell for a basic project with the go runtime and tools included:

{
  description = "A simple Go package";
  inputs.nixpkgs.url = "nixpkgs/nixos-21.11";

  outputs = { self, nixpkgs }:
    let
      supportedSystems = [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ];
      forAllSystems = nixpkgs.lib.genAttrs supportedSystems;
      nixpkgsFor = forAllSystems (system: import nixpkgs { inherit system; });
    in
    {
      devShells = forAllSystems (system:
        let
          pkgs = nixpkgsFor.${system};
        in
        {
          default = pkgs.mkShell {
            buildInputs = with pkgs; [ go gopls gotools go-tools ];
          };
        });
    };
}

If we start a devShell with this flake, Nix goes through the following high-level steps:

1. Clone Nixpkgs: Nix clones the commit of the Nixpkgs repo that matches our inputs.nixpkgs.url reference.
2. Evaluate the Build: Nix evaluates the specific commit of the Nixpkgs repo to determine the hash and store path of the packages we want to install in build inputs. This hash is a function of the package's inputs or contents and the specific platform on which we install the package.
3. Check the Cache: Once Nix has the store path, it checks our local Nix Store and remote binary caches (usually https://cache.nixos.org) to see if a pre-built package exists. If a package matches the store path of the target package, Nix downloads it to the Nix Store.
4. Build Cache Misses from Source: If the package doesn't exist in our local store or the remote cache, Nix uses the derivation to build the package from scratch.
5. Start the Shell: Once package installation is complete, Nix creates an isolated shell with all the packages and configurations included.

Previously, when generating a Devbox shell, we created a devShell like the one shown above and followed the same steps to fetch the packages the shell needed.

Unfortunately, the first two cloning and evaluation steps took 30 to 60 seconds when installing or updating packages. In the case of Devbox, this cost multiplied if your project pinned several packages since this increased the number of nixpkgs commits we had to evaluate. To keep Devbox fast and performant for all projects, we needed a way to skip the cloning and evaluation steps altogether.

Downloading Packages Directly from the Cache with fetchClosure

We found our solution in an experimental builtin Nix function called fetchClosure. A closure in Nix consists of the package you want to install, along with all of the dependencies needed to run the package on your platform. The fetchClosure function was added by the Nix team in version 2.8, and it does exactly what it says: it fetches a package's entire closure from a remote cache to your local machine.

For example, the invocation below will copy the Go 1.21.4 closure for a an Apple Silicon Mac into your local Nix store:

builtins.fetchClosure {
	fromStore = "<https://cache.nixos.org>";
	fromPath = /nix/store/vlj3bkikqcgk449psyadav12ii2swfpj-go-1.21.4;
}

Thanks to this awesome feature, we can bypass the cloning and evaluation steps of Nixpkgs. As long as we know the store path ahead of time, we can pull everything we need direct from the remote cache.

Ahead-of-time Store Paths in Nixhub

Of course, this leads to the next question — how do we get the store paths ahead of time?

Our answer was to use our Nixhub Search Service to evaluate the paths for the most popular platforms and store them in our search index. Whenever Devbox queries Nixhub for the packages it needs to install, it also receives the Nix store paths in the response.

For example, here's an excerpt of the Nixhub response when you search for Python 3.7:

{
	...
  "systems": {
    "aarch64-darwin": {
      "commit_hash": "80c24eeb9ff46aa99617844d0c4168659e35175f",
      "system": "aarch64-darwin",
      "last_updated": 1671268780,
      "version": "3.7.16",
      "store_hash": "a89sd5jwn01cdg97lkspl8cpf75y5142",
      "store_name": "python3",
      "store_version": "3.7.16",
      "meta_name": "python3-3.7.16",
      "meta_version": [
        ""
      ],
      "attr_paths": [
        "python37"
      ],
      "platforms": [
        "aarch64-darwin",
        "aarch64-linux",
        "x86_64-darwin",
        "x86_64-linux"
      ],
  ...
    },

Using the store_hash and store_namein this response, we can reconstruct the full store path for all of the platforms where package is supported. We use this path to query the official Nix Binary cache using fetchClosure, and copy the resulting closure to the user’s Nix Store. If we are unable to find the package in the cache, we can use the commit_hash to fall back to cloning and evaluating nixpkgs.

Our new generated flake thus looks something like this:

{
   description = "A devbox shell";
   inputs = {
     nixpkgs.url = "github:NixOS/nixpkgs/75a52265bda7fd25e06e3a67dee3f0354e73243c";
   };
   outputs = {
     self,
     nixpkgs,
   }:
      let
        pkgs = nixpkgs.legacyPackages.aarch64-darwin;
      in
      {
        devShells.aarch64-darwin.default = pkgs.mkShell {
          buildInputs = [
            (builtins.fetchClosure {
              fromStore = "<https://cache.nixos.org>";
              fromPath = "/nix/store/02cpvad60np366kmsqc3gnixbsw9jvg1-go-1.21.4";
              inputAddressed = true;
            }) 
          ];
        };
      };
 }

Using fetchClosure, we can save over 30 seconds off installation times for simple Devbox projects. For more complex projects, with N packages pinned to specific versions that resolve to different nixpkgs commit hashes, the savings are N * 30 seconds.

Less time waiting, less time for sword fights, and much happier users.

Keep in Touch with Devbox

If you're reading this, we'd love to hear from you about how you've been using Devbox for your projects, and what you'd like to see next in Devbox. You can follow us on Twitter, or chat with our developers live on our Discord Server. We also welcome issues and pull requests on our Github Repo.