Back

Dual-booting NixOS and Alpine Linux without root partitions

2023-04-02

Last week I read this post and I thought that it was a nice idea to try to setup a NixOS install which has no on-disk root partition. This means that the content of everything in /etc, /var, … is rebuilt from scratch at each boot. The only partitions that are actually required are /nix for the Nix store, and /boot that contains the kernel, initramfs and bootloader config to boot the system.

I also knew of Alpine Linux’s diskless mode, which allows it to run entirely from RAM. This mode allows for additionnal packages to be installed and configuration changes to be persisted using the Local Backup Utility (LBU).

Both of these methods aim for the same objective: to know exactly what composes a Linux install at any given time, by avoiding accumulating cruft in the form of unpredictable files all over the place (especially in /etc and /home). In NixOS, the system when it boots is exactly in the state defined by the configuration.nix file we used to build it, and it contains nothing else. In Alpine, we can easily inspect the files contained in the local backup, which is created manually in order to intentionnally save a chosen subset of files in /etc and other directories. When Alpine Linux boots, it starts from its base initramfs which is always the same, and adds only the files from the local backup (it can also install a set of APK packages).

The target setup

I wanted to try both methods, on a dedicated PC which I’m using for this occasion. This PC has a single SSD with 120GB of storage space, and I am using the rEFInd boot manager to provide a boot menu that allows me to boot into either system. NixOS uses Grub to boot, and Alpine is booted directly from rEFInd.

I also wanted to use extremely lightweight GUI programs, instead of a full-fledged desktop environment. I chose to work with the following programs as much as possible:

Creating the partition layout

Here is the partition layout I came up with for the dual-boot setup:

I just used cgdisk to create the partitions described above. The EFI system partition (ESP) has type ef00, and all others have type 8300 (Linux partition).

I formated the ESP with the following command to create a FAT32 partition:

mkfs.vfat -F 32 /dev/sda1

All the other partitions were formatted with mkfs.ext4.

Installing rEFInd

In a NixOS live system booted from USB, I used nix-shell -p refind to make the rEFInd binaries available in my shell.

Then, I used refind-install, which automatically detected that my ESP partition was /dev/sda1. It wrote all its essential files there, but unfortunately it did not detect that I would be needing the ext4 filesystem driver to boot Alpine (EFI/refind/drivers_x64/ext4_x64.efi on the ESP). I fixed this later, after I installed NixOS (see below).

Installing NixOS

From my NixOS live USB, I proceeded to install a NixOS system.

I first mounted a tmpfs on /mnt to be used as a root partition:

mount -t tmpfs tmpfs /mnt

This partition will be populated by NixOS at the time of installation with files that will all be erased upon reboot. That’s not an issue, as NixOS will just recreate everything it needs when booting.

Then, after creating the directories /mnt/{nix,boot,data}, I mounted all the corresponding partitions at their respective location. I also created the /mnt/boot/efi directory and mounted my ESP there.

I generated a base NixOS configuration with the standard nixos-generate-config invokation.

When customizing my configuration.nix, I had to be carefull with the following things:

Using nixos-install, I was able to build my NixOS system and populate the /boot partition with a kernel and a suitable GRUB config file.

I took care of moving all the relevant .nix files from /mnt/etc/nixos to my data partition, as they would have been lost on reboot if they stayed in the /etc directory of my tmpfs root. I also set system.copySystemConfiguration = true; to make sure that a copy of configuration.nix was kept in the Nix store, just in case.

GRUB itself still had to be installed, and the command was quite complex:

grub-install    --directory=/nix/store/xxxxxx-grub-xxx/lib/grub/x86_64-efi \
                --bootloader-id=NixOS \
                --efi-directory=/mnt/boot/efi \
                --boot-directory=/mnt/boot

The hardest part in this command is finding the correct value of the --directory argument. This argument has to point to the source files that grub-install will copy. For some reason, grub-install is not able to find them on its own when running from nix-shell -p grub. I had to use find on the Nix store to find the place where they were stored.

At this point, I was able to reboot and it worked. Or actually, the first try didn’t work because I forgot to set the hashed password, and I wasn’t able to log in… I fixed that and it worked!

Customizing NixOS

Once booted inside NixOS, the NixOS configuration was updated by editing my configuration.nix file stored on my data partition, and rebuilding using:

sudo nixos-rebuild switch -I nixos-config=path/to/configuration.nix

Classical NixOS configuration applies. In addition, I configured home-manager as a NixOS module directly in my system-wide configuration.nix. This allowed me to set some config files in my home directory to some static content defined by the NixOS configuration and stored in the NixOS store.

Unfortunately, Xfe didn’t run in NixOS, so I installed PCManFM instead. I also ended up installing qutebrowser to be able to access all websites more easily.

My final configuration.nix can be viewed here (link subject to change).

Installing Alpine Linux

I tried to read a bunch of pages on the Alpine Linux wiki to understand how things were supposed to work, and it seemed complicated at first, because I didn’t know how I would tell Alpine to restore its local backup and load additionnal packages from a specific on-disk partition. Would I need a custom kernel command line parameter? Where could I find it? It wasn’t described anywhere.

Actually, Alpine automatically detects the correct location for these files by itself, by detecting the partition from which the kernel and initramfs are loaded.

As a consequence, installing Alpine was extremely straightforward, I just had to copy files from the Alpine Linux ISO to my Alpine partition (/dev/sda4), and setup a refind_linux.conf file so that rEFInd would be able to load the Alpine kernel and its associated initramfs with the essential boot parameters.

I mounted the Alpine Linux ISO somewhere as a loop device, and mounted /dev/sda4 to /mnt to copy files there. The files I copied were the following: the entire apks and boot directories. Then I cleaned up the boot directory to remove everything related to EFI and Syslinux (the bootloader used for non-EFI systems).

The boot directory contains essentialy two files that are strictly necessary: vmlinuz-lts (the Linux kernel image) and initramfs-lts (the root filesystem image). rEFInd is able to boot them directly, simply by creating the refind_linux.conf file in the boot directory with the following content:

"Alpine Linux" "initrd=boot\initramfs-lts modules=loop,squashfs,sd-mod,usb-storage quiet"

I kept a few other files in boot/ just in case: modloop-lts, config-lts and System.map-lts.

Fixing rEFInd

Unfortunately, at this point rEFInd was not able to see files on my Alpine Linux partition because it was missing its ext4 filesystem driver, and therefore the Alpine Linux option was not displayed at boot.

I fixed this by booting inside my NixOS installation and doing refind-install again. This time it observed that /boot was an ext4 partition and it decided to install the ext4 filesystem driver to the EFI system partition (even though it actually didn’t need it for NixOS because NixOS is booted by GRUB, which is on the ESP and can read ext4 filesystems by itself).

Installing Alpine Linux, step 2

After I fixed rEFInd, I was able to boot into Alpine Linux. At this point, I had the exact same environment than if I had booted the Alpine ISO as a live USB drive. This environment was very generic and minimal and had a US keyboard layout which I didn’t like.

Configuring Alpine Linux was done using the regular setup-alpine script. When asked which target filesystem to use to install Alpine on, I specified none to do a diskless install. However, I was then given the opportunity to use a disk partition to store my configuration changes using the LBU (Local Backup Utility). In fact, the installer script automatically detected that /dev/sda4 should be used for this purpose, and had already mounted it at /media/sda4. I just had to accept the proposed setting. The installer also automatically proposed me to use this partition to store APK files downloaded when I installed additionnal software.

After doing the setup-alpine step, I just did a regular lbu_commit to save all of the updated configuration files to the Alpine partition. This includes the file /etc/apk/world that describes the set of additionnal packages I installed, so that they will be reinstalled automatically at the next boot.

Once the installation is completed, the /dev/sda4 partition contains the following:

Customizing Alpine Linux

To customize Alpine Linux, I did all the regular steps such as installing packages or editing configuration files, and then called lbu_commit to update the local backup to persist these changes to disk.

When I added a non-root user, I had to make sure that LBU would save my home directory as I would be adding some config files there as well. This was done with lbu_include /home. I took care of using lbu_exclude to avoid storing the .cache directory in my home as it has a tendency to become full of useless files.

The content of the local backup can be inspected with the following command:

tar tf /media/sda4/alpine.apkovl.tar.gz

This will list all of the files included in the local backup – these are the only files persisted when Alpine Linux reboots. When using lbu_include, lbu_exclude and lbu_commit, one should take care of keeping this set of files as minimal as possible, and to not store data files in there (only configuration files!). This gzipped backup file is tiny, around 43KB.

I did an lbu_commit after configuring IceWM and launching Xfe, so some of their config files are stored in there. For Sylpheed, it was a bit tricky to save only config files in there and not the IMAP cache, which is big and can contain sensitive data. I exited Sylpheed after setting up my account but before entering my password, so that the account settings were saved by LBU but not the IMAP cache.

Since all additionnal packages are loaded in RAM at boot time, this setup is more limited than the NixOS setup, and forces us to prioritize using extremely small and lightweight software. For instance, I did not bother installing Firefox or Qutebrowser, as they would have added 200-400MB of packages to decompress at each boot. Installing only a lightweight browser (Dillo), Alpine reports only 392 MiB in 234 packages installed.

Proof-of-concept

This blog post was written while booted on my Alpine Linux installation, using only an SSH client to access my web server and the Dillo web browser to look up references on the Internet.

I am globally very satisfied with this installation, I learned a lot and it is working like I wanted.

Comparing NixOS and Alpine

Advantages of the NixOS setup:

Advantages of the Alpine setup: