Cardless Raspberry Pi 4: How to use a Raspberry Pi 4 without an SD card ?
Introduction
Once upon a time, diskless clients were common. They were booting and loading the operating system from the network and keeping the user data on the network as well. It is possible to do this with a Raspberry Pi.
I have explained in a post before how Raspberry Pi 3B+ network boot works. The new Raspberry Pi 4 works a bit different. So I will explain it in this post, as well as running the operating system remotely without an SD card.
rpi4 in this article means Raspberry Pi 4 Model B.
The network in this post means wired ethernet network, not the wireless. Booting from wireless is not supported.
Specifically, at the end of this post, we will have an rpi4 without an SD card, connected to network through wired ethernet, that loads and boots the bootloaders and Linux kernel and mounts the Linux root file system from the network using NFS.
Prerequisites
You need a DHCP server, a TFTP server to host the bootcode and the Linux kernel and an NFS server to host the root file system. I am using (Ubuntu) Linux for TFTP and NFS.
It is useful to have a USB console cable which you can connect to the UART exposed in GPIO pins on rpi4 and see the boot messages. You need to use enable_uart=1 in config.txt to use it.
Although you will not need it later, you still need an SD card to update the bootcode and enable network boot on rpi4.
To start from a known state, I have used the recovery bootcode to install the latest critical bootloader image and a fresh Raspbian Buster Lite 2019-09-26 image for this post.
Changes in Raspberry Pi 4
The major change happened in rpi4 is that the bootloader (bootcode.bin) is moved into an (SPI) EEPROM (512KB). Even if the SD card has this file (it was in boot partition), it is ignored. So the boot sequence now is:
- Embedded (invisible, not readable, not writable, not deletable) firmware starts the boot.
- bootloader in EEPROM is loaded.
- start4[cd|db|x].elf (VideoCore) firmware and corresponding fixup4[cd|db|x].dat file is loaded.
- config.txt, cmdline.txt, Linux kernel, device tree files and overlays (dtbs) are loaded and operating system starts.
More and official information about this is here.
db means debug version. cd means cut-down version. x means the firmware includes camera drivers and codec.
rpi-eeprom package
It is possible to update the bootloader EEPROM using the rpi-eeprom package. It is pretty safe to apply the updates even the beta ones, because, in case of a problem/corruption it can be recovered easily by copying the recovery files into an SD card and booting the system.
rpi-eeprom-update manual page gives more information about how the update is installed. Actually rpi-eeprom-update does not install the update directly. It only copies recovery.bin and the EEPROM images (bootloader and USB controller VLI firmware) to the boot partition, and at the next reboot, recovery.bin installs the images and renames itself to recovery.000 to not run again in the following reboots. This basically mean you need a reboot to install any bootloader update.
There are actually two packages: (rpi-eeprom and rpi-eeprom-images)
$ dpkg -l | grep rpi-eeprom
ii rpi-eeprom 1.3-1 all Raspberry Pi 4 boot EEPROM updater
ii rpi-eeprom-images 1.3-1 all Raspberry Pi 4 boot EEPROM images
rpi-eeprom package installs rpi-eeprom-config and rpi-eeprom-update tools (these are scripts not binaries, you can look to them), and also a rpi-eeprom-update service. This service runs rpi-eeprom-update -a at startup, which installs the latest update. If you do not want this to happen automatically, you should stop and disable this service.
There are two branches of bootloader EEPROM images: critical (default) and beta. Which one you want to use is set with FIRMWARE_RELEASE_STATUS key in /etc/default/rpi-eeprom-update configuration file.
rpi-eeprom-update does not download the bootloader EEPROM images from the internet automatically, it only uses the files given as argument or the ones in /lib/firmware/raspberrypi/bootloader/{beta, critical} folder.
rpi-eeprom-images package provides the bootloader EEPROM images. If you upgrade this package, it will download all the bootloader EEPROM images and save them under /lib/firmware/raspberrypi/bootloader/{beta, critical}.
Here are the contents of these folders before issuing an upgrade (that comes with Raspbian image):
$ ls /lib/firmware/raspberrypi/bootloader/critical/
pieeprom-2019-05-10.bin pieeprom-2019-07-15.bin
pieeprom-2019-09-10.bin
$ ls /lib/firmware/raspberrypi/bootloader/beta/
pieeprom-2019-09-06.bin pieeprom-2019-09-23.bin
pieeprom-2019-09-10.bin pieeprom-2019-09-25.bin
Step 1: Enabling the network boot
Lets first check the bootloader version.
$ vcgencmd bootloader_version
Sep 10 2019 10:41:50
version f626c772b15ba1b7e0532a8d50a761b3ccbdf3bb (release)
timestamp 1568112110
At the time of writing this post, 2019-09-10 (f626c772) is the latest critical version of the bootcode.
Network boot support is added in a later (beta) version, 2019-09-23. You can see the release notes of the bootloader here.
That means, for network boot, I need a beta version. So I change the FIRMWARE_RELEASE_STATUS to beta.
$ cat /etc/default/rpi-eeprom-update
FIRMWARE_RELEASE_STATUS="beta"
Lets upgrade now.
$ sudo apt upgrade rpi-eeprom rpi-eeprom-images
....
$ ls /lib/firmware/raspberrypi/bootloader/critical/
pieeprom-2019-07-15.bin recovery.bin vl805-000137ab.bin
pieeprom-2019-09-10.bin vl805-00013701.bin vl805-000137ad.bin
$ ls /lib/firmware/raspberrypi/bootloader/beta/
pieeprom-2019-09-06.bin pieeprom-2019-10-16.bin recovery.bin
pieeprom-2019-09-10.bin pieeprom-2019-11-18.bin vl805-00013701.bin
pieeprom-2019-09-23.bin pieeprom-2019-12-03.bin vl805-000137ab.bin
pieeprom-2019-09-25.bin pieeprom-2020-01-09.bin vl805-000137ad.bin
pieeprom-2019-10-08.bin pieeprom-2020-01-17.bin
and check version now:
$ sudo rpi-eeprom-update
BCM2711 detected
*** UPDATE REQUIRED ***
BOOTLOADER: update required
CURRENT: Tue 10 Sep 10:41:50 UTC 2019 (1568112110)
LATEST: Fri 17 Jan 17:37:11 UTC 2020 (1579282631)
VL805: update required
CURRENT: 000137ab
LATEST: 000137ad
(VL805 is the USB controller)
so it picks up the EEPROM images. Now, either I need to reboot, and then rpi-eeprom-update service will run and install the updates (actually not install but copy them to the boot partition in order to be installed at next reboot, so I need another reboot) or run rpi-eeprom-update now and just do reboot once. So lets do the later:
$ sudo rpi-eeprom-update -a
BCM2711 detected
*** INSTALLING EEPROM UPDATES ***
BOOTLOADER: update required
CURRENT: Tue 10 Sep 10:41:50 UTC 2019 (1568112110)
LATEST: Fri 17 Jan 17:37:11 UTC 2020 (1579282631)
VL805: update required
CURRENT: 000137ab
LATEST: 000137ad
EEPROM updates pending. Please reboot to apply the update.
Now I reboot, and it takes longer than usual reboots, and then lets check the version again:
$ vcgencmd bootloader_version
Jan 17 2020 17:37:11
version 5e86aac5fad596d61caab70b1d43a987bab07de3 (release)
timestamp 1579282631
Now I need to enable network boot. This could be done before together with updating the bootloader. Lets first copy the bootloader image and extract the config:
$ cp /lib/firmware/raspberrypi/bootloader/beta/pieeprom-2020-01-17.bin pieeprom.bin
$ rpi-eeprom-config pieeprom.bin > config.txt
$ cat config.txt
[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
TFTP_IP=
TFTP_PREFIX=0
BOOT_ORDER=0x1
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=5
[none]
FREEZE_VERSION=0
Information about these options can be found here.
I will change BOOT_UART and BOOT_ORDER. I do not need to set TFTP_IP since my DHCP server is distributing the IP of TFTP server. I set the BOOT_ORDER to SD card first then network, so 0x21. Then apply this config to the bootloader image and then request installing this image and then reboot.
$ cat config.txt
[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
TFTP_IP=
TFTP_PREFIX=0
BOOT_ORDER=0x21
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=5
[none]
FREEZE_VERSION=0
$ rpi-eeprom-config --out pieeprom-out.bin --config config.txt pieeprom.bin
$ sudo rpi-eeprom-update -d -f ./pieeprom-out.bin
BCM2711 detected
*** INSTALLING ./pieeprom-out.bin ***
EEPROM update pending. Please reboot to apply the update.
$ sudo reboot
f you have connected the USB serial cable, you will see initial boot messages. We can check the actual config to be sure:
$ vcgencmd bootloader_config
[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
TFTP_IP=
TFTP_PREFIX=0
BOOT_ORDER=0x21
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=5
[none]
FREEZE_VERSION=0
All looks good.
Step 2: Loading the bootloader, firmware and Linux kernel from the network
I copied all the contents of the boot partition to TFTP server, and poweroff rpi4, remove the SD card and power on.
In the TFTP server logs, I can see many files are served, and on the boot messages I can see Linux boots. However, it stucks somewhere.
The reason is the root filesystem was in SD card and this is specified in cmdline passed to Linux kernel. Looking at the TFTP server:
$ cat cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=6c586e13-02
rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
We need to provide a root file system hosted somewhere else, in this case, using NFS.
Step 3: Mounting the root filesystem
Basically I am going to mount the root file system in the Raspbian image and serve this over NFS.
In order to find the location of the root file system in the image, lets look into the image with fdisk:
$ fdisk -l ./2019-09-26-raspbian-buster-lite.img
Disk ./2019-09-26-raspbian-buster-lite.img: 2.1 GiB, 2248146944 bytes, 4390912 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x6c586e13
Device Boot Start End Sectors Size Id Type
./2019-09-26-raspbian-buster-lite.img1 8192 532479 524288 256M c W95 FAT32 (LBA)
./2019-09-26-raspbian-buster-lite.img2 532480 4390911 3858432 1.9G 83 Linux
I need a byte offset, so the second (Linux) partition offset is 532480 and sector size is 512 bytes, so 532480*512=272629760.
Now lets mount the image:
$ sudo mount -o loop,offset=272629760 ./2019-09-26-raspbian-buster-lite.img /mnt/pi4
Now lets setup the NFS. I installed the nfs-kernel-server package and I am going to use /mnt/pi4 to share over NFS, and run exportfs after modifying the exports file.
$ cat /etc/exports
/mnt/pi4 192.168.1.1/24(rw,sync,no_subtree_check,no_root_squash)
$ sudo exportfs
Now we need to change the cmdline passed to Linux kernel for rpi4. Here is the one I am using:
$ cat cmdline.txt
console=serial0,115200 console=tty1 root=/dev/nfs
nfsroot=192.168.1.151:/mnt/pi4,nfsvers=3 ip=dhcp rw elevator=deadline
fsck.repair=yes rootwait
pay attention to root, nfsroot and ip.
You can find more information about NFS options here.
The last step is to modify /etc/fstab in the root file system. Because the boot and root partitions are specified also there as they are in the SD card. This is the fstab I am using:
$ cat /mnt/pi4/etc/fstab
proc /proc proc defaults 0 0
Now we can reboot rpi4.
Conclusion
If you check the boot messages, you will see all works fine and you get the login screen.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.