Bare Metal Raspberry Pi 3B+: Network Boot

March 22, 2019

Introduction

I decided to make a series posts, I think at least 3 posts, for bare metal Raspberry Pi usage. That is to use Raspberry Pi without an OS, so without a Linux, at least without the regular one.

Because deployment and debugging is much difficult on bare metal, I decided to divide this topic into (at least) three parts, thus posts:

Network Boot
JTAG
Bare Metal Programming

We need network boot, because it does not make any sense to write the SD card, plug it to rpi, reboot, and repeat this again, since we need to do this so many times.

We may need JTAG, that is the lowest possible debugging capability a platform may offer. It is an hardware debugging capability, supported also by ARM and by Raspberry Pi platform.

Then, we will write a simple program to run without an OS.

It is important to remember these posts apply to Raspberry Pi 3 Model B+. The previous models can and actually does function different, may require additional steps, or may have bugs preventing this operation.

When I say pi or rpi in this article, I specifically mean the Raspberry Pi 3 Model B+ board.

Network Boot

In this post, I will show how Raspberry Pi 3 Model B+ boots from network, without an SD card. I will do this step by step, so it will somehow show the boot sequence as well.

The network in this post means wired ethernet network, not the wireless. Booting from wireless is not supported.

Specifically, at the end of this post, we will have an rpi without an SD card, connected to network through wired ethernet, that boots directly from the network by:

getting its IP and information about the TFTP server by DHCP
connecting to TFTP to fetch required files for booting
using the files fetched from TFTP to complete the boot process

Prerequisites

Raspberry Pi 3 Model B+
USB Console Cable, which you connect to rpi GPIO pins on one end, and your computer’s USB port on the other.
A wired, preferably isolated (than the other devices in your network) network under your control. Both the computer running DHCP and TFTP and the rpi is wired to this network.
A linux computer, I am using my desktop computer running Ubuntu 18.04.
DHCP and TFTP server. I am using my linux computer for this purpose.

I also use tcpdump to observe the network traffic and screen to connect to rpi console over the serial port/USB.

Raspberry Pi Boot Sequence

Many documents explain the boot sequence of rpi. A very short summary is this.

Embedded firmware starts the boot. You cannot read/write/delete this code. Because it cannot be modified, this makes working low level on rpi safer than most other platforms. It is very difficult/impossible to brick rpi.

Then, bootcode.bin and then start.elf and fixup.dat completes the firmware boot process. There are different versions of start and fixup files. Normal version is without any suffix, other versions are _cd=cut down, _db=debug, _x=experimental.

bootcode.bin, start.elf and fixup.dat can be downloaded from the official Raspberry Pi repo.

Until this point, you have almost no control (other than changing UART logging, more on this later), and the mentioned binaries above are closed-source. At this point, another boot loader, kernel or OS starts to load. For a normal linux distribution this means:

kernel: which may have different file names, more on this later.
device tree blob: kernel needs this file to properly initialize the rpi.

Raspberry Pi has a separate kernel source tree here.

Of course, linux is not the only way to boot rpi, more on this also either in this post or in future posts.

Enabling Network Boot on Raspberry Pi

This is explained in this article.

IMPORTANT: What you set here is permanent (OTP=One Time Programmable), it cannot be reverted. More info here

First check if network boot is enabled, this seems to be the case for RPi 3B+:

$ vcgencmd otp_dump | grep 17:
17:3020000a

On Raspberry Pi 3B (not plus), I saw the value as 1020000a.

The value has to be 3020000a (actually bit 29 has to be set), if it is not, you have to add this to config.txt:

program_usb_boot_mode=1

and reboot rpi. Then, check the value again.

Preparing DHCP and TFTP

Network boot means the client computer will use TFTP to download the files required during boot from a remote computer. DHCP will be used both to configure the network interface (IP etc.) and to learn TFTP server address.

WARNING: If you have other devices (especially embedded devices like switches, routers) in the network, setting this DHCP option may cause problems. Be careful.

If you have a capable DHCP server at hand, you can use it for this purpose. However, many simple (home type) routers do not support setting custom DHCP options. We need to set DHCP Option 66, tftp-server-name. I will install a DHCP server, using the isc-dhcp-server package:

$ sudo apt install isc-dhcp-server

Then we need to configure /etc/dhcp/dhcpd.conf, for example I have this:

default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;
authoritative;
log-facility local7;
subnet 192.168.97.0 netmask 255.255.255.0 {
  range 192.168.97.100 192.168.97.200;
  option routers 192.168.97.1;
  option domain-name-servers 8.8.8.8, 8.8.4.4;
  option tftp-server-name "192.168.97.2";
}

Using this configuration, we can observe this DHCP Request from rpi:

0.0.0.0.bootpc > 255.255.255.255.bootps: [no cksum] BOOTP/DHCP, Request from b8:27:eb:99:a1:91 (oui Unknown), length 322, xid 0x26f30339, Flags [none] (0x0000)
	  Client-Ethernet-Address b8:27:eb:99:a1:91 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Parameter-Request Option 55, length 14: 
	      Vendor-Option, Vendor-Class, BF, Option 128
	      Option 129, Option 130, Option 131, Option 132
	      Option 133, Option 134, Option 135, TFTP
	      Subnet-Mask, Default-Gateway
	    ARCH Option 93, length 2: 0
	    NDI Option 94, length 3: 1.2.1
	    GUID Option 97, length 17: 0.145.161.153.243.145.161.153.243.145.161.153.243.145.161.153.243
	    Vendor-Class Option 60, length 32: "PXEClient:Arch:00000:UNDI:002001"

and the corresponding DHCP Response:

paula.bootps > 192.168.97.100.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x26f30339, Flags [none] (0x0000)
	  Your-IP 192.168.97.100
	  Server-IP paula
	  Client-Ethernet-Address b8:27:eb:99:a1:91 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Offer
	    Server-ID Option 54, length 4: paula
	    Lease-Time Option 51, length 4: 600
	    TFTP Option 66, length 12: "192.168.97.2"
	    Subnet-Mask Option 1, length 4: 255.255.255.0
	    Default-Gateway Option 3, length 4: _gateway

DHCP is ready.

Now lets setup TFTP. On Ubuntu 18.04, I am using tftpd package:

$ sudo apt install tftpd

Then, I configure a file named tftp under /etc/xinet.d with the following content:

service tftp
{
  protocol        = udp
  port            = 69
  socket_type     = dgram
  wait            = yes
  user            = nobody
  server          = /usr/sbin/in.tftpd
  server_args     = /home/mete/tftpboot
  disable         = no
}

You should set the server_args variable to where you want to keep files hosted by tftpd, and restart xinetd to start tftpd.

$ sudo service xinetd restart

TFTP is also ready now.

Configuration 1 - no files

After DHCP succeeds, rpi gets the IP address and knows the IP of the TFTP server. This is the first thing done.

From TFTP, it requests:

bootcode.bin
bootsig.bin

Since TFTP does not respond with anything, we are stuck here, no output on serial console.

We know bootcode.bin, but I do not know what bootsig.bin is. It sounds like a signature but there is no documentation.

I used tcpdump to debug this. I will not show it for other alternatives, but showing here once.

$ sudo tcpdump 'ether host b8:27:eb:99:a1:91' -i eno1 -vv

tcpdump: listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes

12:48:52.658401 IP (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 350)
    0.0.0.0.bootpc > 255.255.255.255.bootps: [no cksum] BOOTP/DHCP, Request from b8:27:eb:99:a1:91 (oui Unknown), length 322, xid 0x26f30339, Flags [none] (0x0000)
	  Client-Ethernet-Address b8:27:eb:99:a1:91 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    Parameter-Request Option 55, length 14:
	      Vendor-Option, Vendor-Class, BF, Option 128
	      Option 129, Option 130, Option 131, Option 132
	      Option 133, Option 134, Option 135, TFTP
	      Subnet-Mask, Default-Gateway
	    ARCH Option 93, length 2: 0
	    NDI Option 94, length 3: 1.2.1
	    GUID Option 97, length 17: 0.145.161.153.243.145.161.153.243.145.161.153.243.145.161.153.243
	    Vendor-Class Option 60, length 32: "PXEClient:Arch:00000:UNDI:002001"

12:48:52.658583 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    paula.bootps > 192.168.97.100.bootpc: [udp sum ok] BOOTP/DHCP, Reply, length 300, xid 0x26f30339, Flags [none] (0x0000)
	  Your-IP 192.168.97.100
	  Server-IP paula
	  Client-Ethernet-Address b8:27:eb:99:a1:91 (oui Unknown)
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Offer
	    Server-ID Option 54, length 4: paula
	    Lease-Time Option 51, length 4: 600
	    TFTP Option 66, length 12: "192.168.97.2"
	    Subnet-Mask Option 1, length 4: 255.255.255.0
	    Default-Gateway Option 3, length 4: _gateway

12:48:52.658873 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has paula tell 192.168.97.100, length 46

12:48:52.658896 ARP, Ethernet (len 6), IPv4 (len 4), Reply paula is-at a0:8c:fd:c3:a3:bb (oui Unknown), length 28

12:48:52.659077 IP (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 49)
    192.168.97.100.49153 > paula.tftp: [no cksum]  21 RRQ "bootcode.bin" octet

12:49:03.471670 IP (tos 0x0, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 48)
    192.168.97.100.49154 > paula.tftp: [no cksum]  20 RRQ "bootsig.bin" octet

You can also check syslog for files requested from TFTP.

Configuration 2 - bootcode.bin

Lets see what happens when we put bootcode.bin to TFTP. But before that, I would like to enable serial console for bootcode, so lets change it first:

$ strings bootcode.bin | grep BOOT_UART
BOOT_UART=0
$ sed -i -e "s/BOOT_UART=0/BOOT_UART=1/" bootcode.bin
$ strings bootcode.bin | grep BOOT_UART
BOOT_UART=1

Now restart and serial console shows:

Raspberry Pi Bootcode

USB ethernet boot
Done ARP for 192.168.97.2 got a0:8c:fd:c3:a3:bb

Raspberry Pi Bootcode

and that is all. When we look at the network, we see rpi requests the following files, in this order:

bootcode.bin
bootsig.bin
f399a191/start.elf
autoboot.txt
config.txt
recovery.elf
start.elf
fixup.dat

Something unexpected is it looks for start.elf under f399a191 directory. This is documented here

It looks for the files under a folder called the serial_number of the particular rpi. So f399a191 is the serial number of the rpi I am using.

It also looks for start.elf under the root directory as well, more on this later.

Configuration 3 - start.elf (and fixup.dat)

Now we have these in TFTP:

$ ls -R
.:
bootcode.bin  f399a191

./f399a191:
start.elf

the serial console shows this time a lot of output, mainly related to HDMI. What you see might be different, and I am going to omit the hdmi log lines after this one:

Raspberry Pi Bootcode

USB ethernet boot
Done ARP for 192.168.97.2 got a0:8c:fd:c3:a3:bb

Raspberry Pi Bootcode
Read File: start.elf, 2857060 (bytes)
MESS:00:00:51.124452:0: HDMI:EDID version 1.3, 1 extensions, screen size 53x30 cm
MESS:00:00:51.130277:0: HDMI:EDID features - videodef 0x80 standby suspend active off; colour encoding:RGB444|YCbCr422; sRGB is not default colourspace; preferred format is native; does not support GTF
MESS:00:00:51.147951:0: HDMI:EDID found unknown detail timing format: 3840x2160p hfp:48 hs:32 hbp:80 vfp:3 vs:5 vbp:23 pixel clock:262 MHz
MESS:00:00:51.160028:0: HDMI:EDID found DMT format: code 4, 640x480p @ 60 Hz in established timing I/II
MESS:00:00:51.169140:0: HDMI:EDID found DMT format: code 6, 640x480p @ 75 Hz in established timing I/II
MESS:00:00:51.178254:0: HDMI:EDID found DMT format: code 9, 800x600p @ 60 Hz in established timing I/II
MESS:00:00:51.187369:0: HDMI:EDID found DMT format: code 11, 800x600p @ 75 Hz in established timing I/II
MESS:00:00:51.196571:0: HDMI:EDID found DMT format: code 16, 1024x768p @ 60 Hz in established timing I/II
MESS:00:00:51.205859:0: HDMI:EDID found DMT format: code 18, 1024x768p @ 75 Hz in established timing I/II
MESS:00:00:51.215150:0: HDMI:EDID found DMT format: code 36, 1280x1024p @ 75 Hz in established timing I/II
MESS:00:00:51.224569:0: HDMI:EDID standard timings block x 8: 0x714F 8180 A9C0 A940 D1C0 E100 0101 0101
MESS:00:00:51.233735:0: HDMI:EDID found DMT format: code 21, 1152x864p @ 75 Hz (4:3) in standard timing 0
MESS:00:00:51.243024:0: HDMI:EDID found DMT format: code 35, 1280x1024p @ 60 Hz (5:4) in standard timing 1
MESS:00:00:51.252401:0: HDMI:EDID found DMT format: code 83, 1600x900p @ 60 Hz (16:9) in standard timing 2
MESS:00:00:51.261775:0: HDMI:EDID found DMT format: code 51, 1600x1200p @ 60 Hz (4:3) in standard timing 3
MESS:00:00:51.271152:0: HDMI:EDID found DMT format: code 82, 1920x1080p @ 60 Hz (16:9) in standard timing 4
MESS:00:00:51.280604:0: HDMI:EDID unknown standard timing 2048x1280 @ 60 Hz aspect ratio (16:10)
MESS:00:00:51.302075:0: HDMI:EDID parsing v3 CEA extension 0
MESS:00:00:51.306054:0: HDMI:EDID monitor support - underscan IT formats:yes, basic audio:yes, yuv444:yes, yuv422:yes, #native DTD:1
MESS:00:00:51.317694:0: HDMI:EDID found preferred CEA detail timing format: 1920x1080p @ 60 Hz (16)
MESS:00:00:51.326458:0: HDMI:EDID found CEA detail timing format: 1920x1080i @ 60 Hz (5)
MESS:00:00:51.334313:0: HDMI:EDID found unknown detail timing format: 3840x2160p hfp:176 hs:88 hbp:296 vfp:8 vs:10 vbp:72 pixel clock:297 MHz
MESS:00:00:51.346725:0: HDMI:EDID found unknown detail timing format: 2560x1440p hfp:48 hs:32 hbp:80 vfp:3 vs:5 vbp:33 pixel clock:241 MHz
MESS:00:00:51.358829:0: HDMI:EDID found CEA format: code 16, 1920x1080p @ 60Hz (native)
MESS:00:00:51.366553:0: HDMI:EDID found CEA format: code 5, 1920x1080i @ 60Hz
MESS:00:00:51.373497:0: HDMI:EDID found CEA format: code 4, 1280x720p @ 60Hz
MESS:00:00:51.380354:0: HDMI:EDID found CEA format: code 2, 720x480p @ 60Hz
MESS:00:00:51.387126:0: HDMI:EDID found CEA format: code 7, 1440x480i @ 60Hz
MESS:00:00:51.393983:0: HDMI:EDID found CEA format: code 22, 1440x576i @ 50Hz
MESS:00:00:51.400927:0: HDMI:EDID found CEA format: code 1, 640x480p @ 60Hz
MESS:00:00:51.407699:0: HDMI:EDID found CEA format: code 20, 1920x1080i @ 50Hz
MESS:00:00:51.414730:0: HDMI:EDID found CEA format: code 31, 1920x1080p @ 50Hz
MESS:00:00:51.421761:0: HDMI:EDID found CEA format: code 18, 720x576p @ 50Hz
MESS:00:00:51.428619:0: HDMI:EDID found CEA format: code 19, 1280x720p @ 50Hz
MESS:00:00:51.435563:0: HDMI:EDID found CEA format: code 39, 1920x1080i @ 50Hz
MESS:00:00:51.442595:0: HDMI:EDID found CEA format: code 32, 1920x1080p @ 24Hz
MESS:00:00:51.449626:0: HDMI:EDID found CEA format: code 33, 1920x1080p @ 25Hz
MESS:00:00:51.456657:0: HDMI:EDID found CEA format: code 34, 1920x1080p @ 30Hz
MESS:00:00:51.463688:0: HDMI:EDID found CEA format: code 3, 720x480p @ 60Hz
MESS:00:00:51.470459:0: HDMI:EDID found CEA format: code 6, 1440x480i @ 60Hz
MESS:00:00:51.477316:0: HDMI:EDID found CEA format: code 17, 720x576p @ 50Hz
MESS:00:00:51.484175:0: HDMI:EDID found CEA format: code 21, 1440x576i @ 50Hz
MESS:00:00:51.491146:0: HDMI:EDID found audio format 2 channels PCM, sample rate: 32|44|48|88|96 kHz, sample size: 16|20|24 bits
MESS:00:00:51.502389:0: HDMI:EDID found HDMI VSDB length 13
MESS:00:00:51.507692:0: HDMI:EDID HDMI VSDB has physical address 1.0.0.0
MESS:00:00:51.514110:0: HDMI:EDID HDMI VSDB supports AI:no, dual link DVI:no
MESS:00:00:51.520890:0: HDMI:EDID HDMI VSDB deep colour support - 48-bit:no 36-bit:no 30-bit:no DC_yuv444:no
MESS:00:00:51.530428:0: HDMI:EDID HDMI VSDB max TMDS clock 300 MHz
MESS:00:00:51.536329:0: HDMI:EDID HDMI VSDB does not support content type
MESS:00:00:51.542852:0: HDMI:EDID HDMI VSDB supports extended resolutions 3,2,1
MESS:00:00:51.549900:0: HDMI:EDID filtering formats with pixel clock > 162 MHz or h. blanking > 1023
MESS:00:00:51.558937:0: HDMI:EDID preferred mode remained as CEA (16) 1920x1080p @ 60 Hz with pixel clock 148 MHz
MESS:00:01:02.225913:0: HDMI:Setting property pixel encoding to Default
MESS:00:01:02.230834:0: HDMI:Setting property pixel clock type to PAL
MESS:00:01:02.236997:0: HDMI:Setting property content type flag to No data
MESS:00:01:02.243595:0: HDMI:Setting property fuzzy format match to enabled
MESS:00:01:02.468828:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:01:02.476302:0: hdmi: HDMI:>>>>>>>>>>>>>Rx sensed, reading EDID<<<<<<<<<<<<<
MESS:00:01:02.494553:0: hdmi: HDMI:EDID version 1.3, 1 extensions, screen size 53x30 cm
MESS:00:01:02.500903:0: hdmi: HDMI:EDID features - videodef 0x80 standby suspend active off; colour encoding:RGB444|YCbCr422; sRGB is not default colourspace; preferred format is native; does not support GTF
MESS:00:01:02.519102:0: hdmi: HDMI:EDID failed to find a matching detail format for 3840x2160p hfp:48 hs:32 hbp:80 vfp:3 vs:5 vbp:23 pixel clock:262 MHz
MESS:00:01:02.532368:0: hdmi: HDMI:EDID calculated refresh rate is 30 Hz
MESS:00:01:02.538801:0: hdmi: HDMI:EDID guessing the format to be 3840x2160p @30 Hz
MESS:00:01:02.546230:0: hdmi: HDMI:EDID found unknown detail timing format: 3840x2160p hfp:48 hs:32 hbp:80 vfp:3 vs:5 vbp:23 pixel clock:262 MHz
MESS:00:01:02.558865:0: hdmi: HDMI:EDID found DMT format: code 4, 640x480p @ 60 Hz in established timing I/II
MESS:00:01:02.568498:0: hdmi: HDMI:EDID found DMT format: code 6, 640x480p @ 75 Hz in established timing I/II
MESS:00:01:02.578133:0: hdmi: HDMI:EDID found DMT format: code 9, 800x600p @ 60 Hz in established timing I/II
MESS:00:01:02.587769:0: hdmi: HDMI:EDID found DMT format: code 11, 800x600p @ 75 Hz in established timing I/II
MESS:00:01:02.597492:0: hdmi: HDMI:EDID found DMT format: code 16, 1024x768p @ 60 Hz in established timing I/II
MESS:00:01:02.607301:0: hdmi: HDMI:EDID found DMT format: code 18, 1024x768p @ 75 Hz in established timing I/II
MESS:00:01:02.617112:0: hdmi: HDMI:EDID found DMT format: code 36, 1280x1024p @ 75 Hz in established timing I/II
MESS:00:01:02.627052:0: hdmi: HDMI:EDID standard timings block x 8: 0x714F 8180 A9C0 A940 D1C0 E100 0101 0101
MESS:00:01:02.636738:0: hdmi: HDMI:EDID found DMT format: code 21, 1152x864p @ 75 Hz (4:3) in standard timing 0
MESS:00:01:02.646549:0: hdmi: HDMI:EDID found DMT format: code 35, 1280x1024p @ 60 Hz (5:4) in standard timing 1
MESS:00:01:02.656446:0: hdmi: HDMI:EDID found DMT format: code 83, 1600x900p @ 60 Hz (16:9) in standard timing 2
MESS:00:01:02.666341:0: hdmi: HDMI:EDID found DMT format: code 51, 1600x1200p @ 60 Hz (4:3) in standard timing 3
MESS:00:01:02.676239:0: hdmi: HDMI:EDID found DMT format: code 82, 1920x1080p @ 60 Hz (16:9) in standard timing 4
MESS:00:01:02.686212:0: hdmi: HDMI:EDID unknown standard timing 2048x1280 @ 60 Hz aspect ratio (16:10)
MESS:00:01:02.708203:0: hdmi: HDMI:EDID parsing v3 CEA extension 0
MESS:00:01:02.712702:0: hdmi: HDMI:EDID monitor support - underscan IT formats:yes, basic audio:yes, yuv444:yes, yuv422:yes, #native DTD:1
MESS:00:01:02.724863:0: hdmi: HDMI:EDID found preferred CEA detail timing format: 1920x1080p @ 60 Hz (16)
MESS:00:01:02.734147:0: hdmi: HDMI:EDID found CEA detail timing format: 1920x1080i @ 60 Hz (5)
MESS:00:01:02.742526:0: hdmi: HDMI:EDID failed to find a matching detail format for 3840x2160p hfp:176 hs:88 hbp:296 vfp:8 vs:10 vbp:72 pixel clock:297 MHz
MESS:00:01:02.756085:0: hdmi: HDMI:EDID calculated refresh rate is 30 Hz
MESS:00:01:02.762519:0: hdmi: HDMI:EDID guessing the format to be 3840x2160p @30 Hz
MESS:00:01:02.769947:0: hdmi: HDMI:EDID found unknown detail timing format: 3840x2160p hfp:176 hs:88 hbp:296 vfp:8 vs:10 vbp:72 pixel clock:297 MHz
MESS:00:01:02.782888:0: hdmi: HDMI:EDID failed to find a matching detail format for 2560x1440p hfp:48 hs:32 hbp:80 vfp:3 vs:5 vbp:33 pixel clock:241 MHz
MESS:00:01:02.796189:0: hdmi: HDMI:EDID calculated refresh rate is 60 Hz
MESS:00:01:02.802624:0: hdmi: HDMI:EDID guessing the format to be 2560x1440p @60 Hz
MESS:00:01:02.810051:0: hdmi: HDMI:EDID found unknown detail timing format: 2560x1440p hfp:48 hs:32 hbp:80 vfp:3 vs:5 vbp:33 pixel clock:241 MHz
MESS:00:01:02.822683:0: hdmi: HDMI:EDID found CEA format: code 16, 1920x1080p @ 60Hz (native)
MESS:00:01:02.830926:0: hdmi: HDMI:EDID found CEA format: code 5, 1920x1080i @ 60Hz
MESS:00:01:02.838391:0: hdmi: HDMI:EDID found CEA format: code 4, 1280x720p @ 60Hz
MESS:00:01:02.845770:0: hdmi: HDMI:EDID found CEA format: code 2, 720x480p @ 60Hz
MESS:00:01:02.853062:0: hdmi: HDMI:EDID found CEA format: code 7, 1440x480i @ 60Hz
MESS:00:01:02.860450:0: hdmi: HDMI:EDID found CEA format: code 22, 1440x576i @ 50Hz
MESS:00:01:02.867904:0: hdmi: HDMI:EDID found CEA format: code 1, 640x480p @ 60Hz
MESS:00:01:02.875197:0: hdmi: HDMI:EDID found CEA format: code 20, 1920x1080i @ 50Hz
MESS:00:01:02.882750:0: hdmi: HDMI:EDID found CEA format: code 31, 1920x1080p @ 50Hz
MESS:00:01:02.890301:0: hdmi: HDMI:EDID found CEA format: code 18, 720x576p @ 50Hz
MESS:00:01:02.897679:0: hdmi: HDMI:EDID found CEA format: code 19, 1280x720p @ 50Hz
MESS:00:01:02.905146:0: hdmi: HDMI:EDID found CEA format: code 39, 1920x1080i @ 50Hz
MESS:00:01:02.912697:0: hdmi: HDMI:EDID found CEA format: code 32, 1920x1080p @ 24Hz
MESS:00:01:02.920249:0: hdmi: HDMI:EDID found CEA format: code 33, 1920x1080p @ 25Hz
MESS:00:01:02.927802:0: hdmi: HDMI:EDID found CEA format: code 34, 1920x1080p @ 30Hz
MESS:00:01:02.935353:0: hdmi: HDMI:EDID found CEA format: code 3, 720x480p @ 60Hz
MESS:00:01:02.942645:0: hdmi: HDMI:EDID found CEA format: code 6, 1440x480i @ 60Hz
MESS:00:01:02.950023:0: hdmi: HDMI:EDID found CEA format: code 17, 720x576p @ 50Hz
MESS:00:01:02.957403:0: hdmi: HDMI:EDID found CEA format: code 21, 1440x576i @ 50Hz
MESS:00:01:02.964896:0: hdmi: HDMI:EDID found audio format 2 channels PCM, sample rate: 32|44|48|88|96 kHz, sample size: 16|20|24 bits
MESS:00:01:02.976658:0: hdmi: HDMI:EDID found HDMI VSDB length 13
MESS:00:01:02.982482:0: hdmi: HDMI:EDID HDMI VSDB has physical address 1.0.0.0
MESS:00:01:02.989421:0: hdmi: HDMI:EDID HDMI VSDB supports AI:no, dual link DVI:no
MESS:00:01:02.996722:0: hdmi: HDMI:EDID HDMI VSDB deep colour support - 48-bit:no 36-bit:no 30-bit:no DC_yuv444:no
MESS:00:01:03.006781:0: hdmi: HDMI:EDID HDMI VSDB max TMDS clock 300 MHz
MESS:00:01:03.013203:0: hdmi: HDMI:EDID HDMI VSDB does not support content type
MESS:00:01:03.020245:0: hdmi: HDMI:EDID HDMI VSDB supports extended resolutions 3,2,1
MESS:00:01:03.027816:0: hdmi: HDMI:EDID filtering formats with pixel clock > 162 MHz or h. blanking > 1023
MESS:00:01:03.037370:0: hdmi: HDMI:EDID preferred mode remained as CEA (16) 1920x1080p @ 60 Hz with pixel clock 148 MHz
MESS:00:01:03.047666:0: hdmi: HDMI: hotplug attached with HDMI support
MESS:00:01:03.053932:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
MESS:00:01:03.065072:0: hdmi: HDMI: power_on to CEA mode 1080p60
MESS:00:01:03.072542:0: hdmi: HDMI: Action callback added to queue to happen at frame 2
MESS:00:01:03.078849:0: hdmi: HDMI: Action stop_3d_mode added to queue to happen at frame 2
MESS:00:01:03.086922:0: hdmi: HDMI: Action unmute added to queue to happen at frame 3
MESS:00:01:03.094478:0: hdmi: HDMI: Action cec_init added to queue to happen at frame 3
MESS:00:01:03.123558:0: *** Restart logging
MESS:00:01:13.126580:0: Failed to open command line file 'cmdline.txt'
MESS:00:02:13.247221:0: No kernel trailer - assuming DT-capable
MESS:00:02:23.251857:0: Failed to load Device Tree file 'bcm2710-rpi-3-b.dtb'

The important thing for us, it explicitly says it was looking for but failed to open cmdline.txt and bcm2710-rpi-3-b.dtb files.

The network traffic shows rpi requests for the following files, in this order:

bootcode.bin
bootsig.bin
f393a191/start.elf
f393a191/autoboot.txt
f393a191/config.txt
f393a191/recovery.elf
f393a191/start.elf
f393a191/fixup.dat
f393a191/recovery.elf
f393a191/config.txt
f393a191/dt-blob.bin
f393a191/recovery.elf
f393a191/config.txt
f393a191/bootcfg.txt
f393a191/cmdline.txt
f393a191/recovery8.img
f393a191/recovery8-32.img
f393a191/recovery7.img
f393a191/recovery.img
f393a191/kernel8.img
f393a191/kernel8-32.img
f393a191/kernel7.img
f393a191/kernel.img
f393a191/armstub8.bin
f393a191/armstub8-32.bin
f393a191/armstub7.bin
f393a191/armstub.bin
f393a191/bcm2710-rpi-3-b-plus.dtb
f393a191/bcm2710-rpi-3-b.dtb

This is a pretty amazing list, all these files are mentioned in various documents and posts related to rpi. I should say a few are new to me: autoboot, recovery and armstub files I have not heard them before in rpi context.

dt-blob.bin is a configuration which is embedded into start.elf, however, it can be provided externally also, more information is here.

As you might realize, some files have 8, 8-32, 7 or no suffix. This is related to Raspberry Pi board and what CPU that board has. Since this is a Raspberry Pi 3B+. it has an ARMv8 64-bit CPU. So, _8 files are 64-bit ARMv8, _8-32 are 32-bit ARMv8, _7 is 32-bit ARMv7 and no suffix is for previous generations.

It is a bit funny the logs do not mention fixup.dat. I think it is an absolute need because it is used to setup the memory, so lets add this to TFTP (under f399a191 folder) and reboot. Now the serial output shows:

Raspberry Pi Bootcode

USB ethernet boot
Done ARP for 192.168.97.2 got a0:8c:fd:c3:a3:bb

Raspberry Pi Bootcode
Read File: start.elf, 2857060 (bytes)
Read File: fixup.dat, 6666 (bytes)

<<< HDMI LOGS >>>

MESS:00:01:03.123558:0: *** Restart logging
MESS:00:01:13.126580:0: Failed to open command line file 'cmdline.txt'
MESS:00:02:13.247221:0: No kernel trailer - assuming DT-capable
MESS:00:02:23.251857:0: Failed to load Device Tree file 'bcm2710-rpi-3-b.dtb'

we see fixup.dat is used, and the last lines are still same.

You might wonder, as I showed in Configuration 2, rpi looks for start.elf also in the root TFTP folder (after looking for it under f399a191 directory, and only if it is not there), so what happens if we provide it at root instead. I tried this, and what happens is, if this file is provided at root, all other files are expected under root as well, so the serial number directory is ignored afterwards. So it is perfectly possible to do this without using the serial number directory, but I am using it.

Mini Summary

Until now, we successfully boot the rpi using the firmware files (bootcode.bin, start.elf and fixup.dat) remotely, over TFTP. After this, we need to provide a bootloader, OS or a bare metal program. I will show the first two in this post.

Configuration 4 - kernel7.img (and dtb, cmdline, config)

All the files mentioned here can be found in Raspberry Lite image.

Raspberry Lite image provides both kernel.img and kernel7.img, but no kernel8. I will use kernel7.img and put it to TFTP/f399a191, now lets boot and see what happens. I will also add config.txt and cmdline.txt at this point, because without them, we will not see anything on the console.

config.txt I am using in this post contains the following:

enable_uart=1

cmdline.txt I am using in this post contains the following:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1

By default, cmdline.txt contains more options than this, including the root file system, since I am not using the SD card in this post, I removed these options.

Now, serial console shows:

Raspberry Pi Bootcode

USB ethernet boot
Done ARP for 192.168.97.2 got a0:8c:fd:c3:a3:bb
Read File: config.txt, 14 (bytes)

Raspberry Pi Bootcode
Read File: config.txt, 14
Read File: start.elf, 2857060 (bytes)
Read File: fixup.dat, 6666 (bytes)
MESS:00:00:31.124665:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:31.128662:0: brfs: File read: 14 bytes

<<< HDMI LOGS >>>

MESS:00:00:42.515363:0: *** Restart logging
MESS:00:00:42.517852:0: brfs: File read: 14 bytes
MESS:00:00:47.147969:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:47.151943:0: Read command line from file 'cmdline.txt':
MESS:00:00:47.157828:0: 'dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1'
MESS:00:00:47.165354:0: brfs: File read: 57 bytes
MESS:00:01:39.032122:0: brfs: File read: /mfs/sd/kernel7.img
MESS:00:01:39.036081:0: Loading 'kernel7.img' to 0x8000 size 0x4b4d00
MESS:00:01:39.046574:0: No kernel trailer - assuming DT-capable
MESS:00:01:39.050822:0: brfs: File read: 4934912 bytes
MESS:00:01:49.747431:0: Failed to load Device Tree file 'bcm2710-rpi-3-b.dtb'
MESS:00:01:49.753419:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:01:49.761729:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:01:49.768017:0: uart: Baud rate change done...
MESS:00:01:49.771447:0: uart: Baud rate

OK that is interesting, even if cmdline and config is set accordingly we do not see anything from the kernel. The reason is the file mentioned before: bcm2710-rpi-3-b.dtb, but what is that ? dtb is device tree blob. Device Tree is a hardware description and it is specifically needed in embedded platforms, since linux kernel uses it to initialize drivers. Here, we need it, because kernel actually does not know where serial console/UART is located in the hardware.

Even if the file mentioned is bcm2710-rpi-3-b.dtb, if you look to the list of files it requests, rpi first asks for bcm2710-rpi-3-b-plus.dtb. This is normal since we are on a 3B+ board. Lets put this into TFTP also, and reboot:

Raspberry Pi Bootcode

USB ethernet boot
Done ARP for 192.168.97.2 got a0:8c:fd:c3:a3:bb
Read File: config.txt, 14 (bytes)

Raspberry Pi Bootcode
Read File: config.txt, 14
Read File: start.elf, 2857060 (bytes)
Read File: fixup.dat, 6666 (bytes)
MESS:00:00:31.201802:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:31.205798:0: brfs: File read: 14 bytes

<<< HDMI LOGS >>>

MESS:00:00:42.651059:0: *** Restart logging
MESS:00:00:42.653548:0: brfs: File read: 14 bytes
MESS:00:00:47.659539:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:47.663513:0: Read command line from file 'cmdline.txt':
MESS:00:00:47.669398:0: 'dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1'
MESS:00:00:47.676924:0: brfs: File read: 57 bytes
MESS:00:01:40.632465:0: brfs: File read: /mfs/sd/kernel7.img
MESS:00:01:40.636437:0: Loading 'kernel7.img' to 0x8000 size 0x4b4d00
MESS:00:01:40.646915:0: No kernel trailer - assuming DT-capable
MESS:00:01:40.651164:0: brfs: File read: 4934912 bytes
MESS:00:01:40.664232:0: brfs: File read: /mfs/sd/bcm2710-rpi-3-b-plus.dtb
MESS:00:01:40.669324:0: Loading 'bcm2710-rpi-3-b-plus.dtb' to 0x4bcd00 size 0x63e6
MESS:00:01:40.823648:0: brfs: File read: 25574 bytes
MESS:00:01:40.828507:0: brfs: File read: /mfs/sd/config.txt
MESS:00:01:41.855477:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:01:41.975887:0: Device tree loaded to 0x2eff9800 (size 0x67c1)
MESS:00:01:41.982350:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:01:41.988639:0: uart: Baud rate change done...
MESS:00:01:41.992071:0: uart: Baud rate[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.79-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1159 SMP Sun Nov 4 17:50:20 GMT 2018
[    0.000000] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7), cr=10c5383d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: Raspberry Pi 3 Model B Plus Rev 1.3
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] cma: Reserved 8 MiB at 0x3ac00000
[    0.000000] percpu: Embedded 17 pages/cpu @ba348000 s38720 r8192 d22720 u69632
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 240555
[    0.000000] Kernel command line: 8250.nr_uarts=1 bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000  dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Memory: 940232K/970752K available (7168K kernel code, 576K rwdata, 2076K rodata, 1024K init, 698K bss, 22328K reserved, 8192K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xbb800000 - 0xff800000   (1088 MB)
[    0.000000]     lowmem  : 0x80000000 - 0xbb400000   ( 948 MB)
[    0.000000]     modules : 0x7f000000 - 0x80000000   (  16 MB)
[    0.000000]       .text : 0x80008000 - 0x80800000   (8160 kB)
[    0.000000]       .init : 0x80b00000 - 0x80c00000   (1024 kB)
[    0.000000]       .data : 0x80c00000 - 0x80c9017c   ( 577 kB)
[    0.000000]        .bss : 0x80c97f04 - 0x80d468b0   ( 699 kB)

<<< MORE LINES OF KERNEL LOGS >>>

[    2.424099] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[    2.425389] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[    2.450929] Please append a correct "root=" boot option; here are the available partitions:
[    2.466368] 0100            4096 ram0
[    2.466373]  (driver?)
[    2.486195] 0101            4096 ram1
[    2.486200]  (driver?)
[    2.489333] mmc1: new high speed SDIO card at address 0001
[    2.518148] 0102            4096 ram2
[    2.518153]  (driver?)
[    2.537775] 0103            4096 ram3
[    2.537783]  (driver?)
[    2.557151] 0104            4096 ram4
[    2.557156]  (driver?)
[    2.576404] 0105            4096 ram5
[    2.576410]  (driver?)
[    2.595585] 0106            4096 ram6
[    2.595589]  (driver?)
[    2.614683] 0107            4096 ram7
[    2.614687]  (driver?)
[    2.633633] 0108            4096 ram8
[    2.633638]  (driver?)
[    2.652270] 0109            4096 ram9
[    2.652275]  (driver?)
[    2.670606] 010a            4096 ram10
[    2.670611]  (driver?)
[    2.688736] 010b            4096 ram11
[    2.688741]  (driver?)
[    2.706558] 010c            4096 ram12
[    2.706563]  (driver?)
[    2.724151] 010d            4096 ram13
[    2.724156]  (driver?)
[    2.741521] 010e            4096 ram14
[    2.741526]  (driver?)
[    2.758627] 010f            4096 ram15
[    2.758631]  (driver?)
[    2.775625] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    2.789563] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.14.79-v7+ #1159
[    2.801857] Hardware name: BCM2835
[    2.810875] [<8010ff30>] (unwind_backtrace) from [<8010c174>] (show_stack+0x20/0x24)
[    2.821260] usb 1-1.1.1: new high-speed USB device number 4 using dwc_otg
[    2.837004] [<8010c174>] (show_stack) from [<80788fa4>] (dump_stack+0xd4/0x118)
[    2.850138] [<80788fa4>] (dump_stack) from [<8011df30>] (panic+0xf0/0x274)
[    2.862731] [<8011df30>] (panic) from [<80b014f8>] (mount_block_root+0x1e8/0x2b8)
[    2.875878] [<80b014f8>] (mount_block_root) from [<80b017e8>] (mount_root+0x12c/0x134)
[    2.889477] [<80b017e8>] (mount_root) from [<80b01990>] (prepare_namespace+0x1a0/0x1e8)
[    2.903205] [<80b01990>] (prepare_namespace) from [<80b010ac>] (kernel_init_freeable+0x2a8/0x2bc)
[    2.917957] [<80b010ac>] (kernel_init_freeable) from [<8079e020>] (kernel_init+0x18/0x128)
[    2.932124] [<8079e020>] (kernel_init) from [<801080ac>] (ret_from_fork+0x14/0x28)

<<< MORE LINES OF KERNEL LOGS >>>

There are a few things to say about this output:

The serial console now works, because the dtb is also provided and used to initialize the system.

Loading 'kernel7.img' to 0x8000 size 0x4b4d00

kernel is loaded at address 0x8000.

[ 0.000000] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7), cr=10c5383d

it says it is a ARMv7 Processor, but it is actually ARMv8.

[ 0.000000] Kernel command line: 8250.nr_uarts=1 bcm2708_fb.fbwidth=1824 bcm2708_fb.fbheight=984 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1

this is the actual command line provided to kernel, what we set in cmdline.txt is at the end as you can see. The ones before are provided automatically by the firmware.

[    2.424099] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[    2.425389] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[    2.450929] Please append a correct "root=" boot option; here are the available partitions:

as expected, it failed to open the root file system, because we did not provide any, and because of this:

[ 2.775625] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

kernel panics, and stops there.

At this point, if you would like to start a normal linux system, you need to tell where the root filesystem is. Since we do not have the SD card, we need something on the network, and the easiest option is to start an NFS server and mount an NFS share as the root filesystem. There are tutorials for this that you can follow, however, I am not interested in this, it is not my aim to start linux.

Configuration 5 - kernel8.img

I want to mention this only to show a custom kernel and running 64-bit kernel on rpi. The official tutorial is here

Different than this tutorial, I am going to use aarch64-linux-gnu cross compiler, not the one provided in the Raspberry Pi repo, and I am using the bcmrpi3 64-bit (aarch64) config, not the 32-bit config. Remember, kernel7.img is an 32-bit kernel, whereas kernel8.img, being natively targeted to ARMv8, is a 64-bit kernel.

$ KERNEL=kernel8
$ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig
$ make -j4 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image

The result of the build, the kernel image, is located at arch/arm64/boot/Image. Lets put this to TFTP as kernel8.img, and reboot. We could also build the dtb (with dtbs target) and use it, but I am not sure if it is different than the official ones, I am using the one provided with the official Raspberry Pi image. The serial console then shows:

Raspberry Pi Bootcode

USB ethernet boot
Done ARP for 192.168.97.2 got a0:8c:fd:c3:a3:bb
Read File: config.txt, 14 (bytes)

Raspberry Pi Bootcode
Read File: config.txt, 14
Read File: start.elf, 2857060 (bytes)
Read File: fixup.dat, 6666 (bytes)
MESS:00:00:30.920139:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:30.924135:0: brfs: File read: 14 bytes

<<< HDMI LOGS >>>

MESS:00:00:42.310159:0: *** Restart logging
MESS:00:00:42.312648:0: brfs: File read: 14 bytes
MESS:00:00:47.318868:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:47.322842:0: Read command line from file 'cmdline.txt':
MESS:00:00:47.328727:0: 'dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1'
MESS:00:00:47.336264:0: brfs: File read: 57 bytes
MESS:00:01:32.487598:0: brfs: File read: /mfs/sd/kernel8.img
MESS:00:01:32.491558:0: Loading 'kernel8.img' to 0x80000 size 0xc4da00
MESS:00:01:32.502136:0: No kernel trailer - assuming DT-capable
MESS:00:01:32.506380:0: brfs: File read: 12900864 bytes
MESS:00:01:32.519126:0: brfs: File read: /mfs/sd/bcm2710-rpi-3-b-plus.dtb
MESS:00:01:32.524214:0: Loading 'bcm2710-rpi-3-b-plus.dtb' to 0xccda00 size 0x63e6
MESS:00:01:32.678529:0: brfs: File read: 25574 bytes
MESS:00:01:32.682775:0: brfs: File read: /mfs/sd/config.txt
MESS:00:01:33.691865:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:01:33.812315:0: Device tree loaded to 0x2eff9800 (size 0x67c1)
MESS:00:01:33.818781:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:01:33.825071:0: uart: Baud rate change done...
MESS:00:01:33.828504:0: uart: Baud rate[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.9.80-v8+ (mete@paula) (gcc version 7.3.0 (Ubuntu/Linaro 7.3.0-27ubuntu1~18.04) ) #1 SMP PREEMPT Fri Mar 22 15:26:14 CET 2019
[    0.000000] Boot CPU: AArch64 Processor [410fd034]

<<< MORE LINES OF KERNEL LOGS >>>

So the end result is same, it still needs the root filesystem, but this time we booted an 64-bit kernel.

A small detail: 32-bit kernel is loaded at 0x8000, whereas 64-bit kernel is loaded at 0x80000.

Configuration 6 - Das U-Boot

Other than Linux, what else can we do ? We can try u-boot, the universal boot loader. It is pretty simple to build it.

$ git clone git://git.denx.de/u-boot.git
$ make -j4 ARCH=arm CROSS_COMPILE=aarch64-linux-gnu- rpi_3_defconfig all

The result of the build, the u-boot image, is located at ./u-boot.bin. Lets put this to TFTP as kernel8.img. It does not matter the name of this file is called kernel, it has no connection with Linux, it is just the name firmware bootloader is looking for. However, because it is kernel8.img, 64-bit mode is used automatically.

Now if we reboot, the serial console shows:

Raspberry Pi Bootcode

USB ethernet boot
Done ARP for 192.168.97.2 got a0:8c:fd:c3:a3:bb
Read File: config.txt, 14 (bytes)

Raspberry Pi Bootcode
Read File: config.txt, 14
Read File: start.elf, 2857060 (bytes)
Read File: fixup.dat, 6666 (bytes)
MESS:00:00:30.190289:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:30.194286:0: brfs: File read: 14 bytes
MESS:00:00:41.580459:0: *** Restart logging
MESS:00:00:41.582949:0: brfs: File read: 14 bytes
MESS:00:00:46.588893:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:46.592867:0: Read command line from file 'cmdline.txt':
MESS:00:00:46.598752:0: 'dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1'
MESS:00:00:46.606291:0: brfs: File read: 57 bytes
MESS:00:01:28.741345:0: brfs: File read: /mfs/sd/kernel8.img
MESS:00:01:28.745303:0: Loading 'kernel8.img' to 0x80000 size 0x78490
MESS:00:01:28.755796:0: No kernel trailer - assuming DT-capable
MESS:00:01:28.760043:0: brfs: File read: 492688 bytes
MESS:00:01:28.773117:0: brfs: File read: /mfs/sd/bcm2710-rpi-3-b-plus.dtb
MESS:00:01:28.778211:0: Loading 'bcm2710-rpi-3-b-plus.dtb' to 0xf8490 size 0x63e6
MESS:00:01:28.932306:0: brfs: File read: 25574 bytes
MESS:00:01:28.936767:0: brfs: File read: /mfs/sd/config.txt
MESS:00:01:29.955361:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:01:30.076644:0: Device tree loaded to 0x2eff9800 (size 0x67c1)
MESS:00:01:30.083111:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:01:30.089401:0: uart: Baud rate change done...
MESS:00:01:30.092834:0: uart: Baud rate

U-Boot 2019.04-rc4-00018-ga00d15757d (Mar 22 2019 - 15:46:33 +0100)

DRAM:  948 MiB
RPI 3 Model B+ (0xa020d3)
MMC:   mmc@7e202000: 0, sdhci@7e300000: 1
Loading Environment from FAT... WARNING at drivers/mmc/bcm2835_sdhost.c:408/bcm2835_send_command()!
WARNING at drivers/mmc/bcm2835_sdhost.c:408/bcm2835_send_command()!
Card did not respond to voltage select!
In:    serial
Out:   vidconsole
Err:   vidconsole
Net:   No ethernet found.
starting USB...
USB0:   scanning bus 0 for devices... 4 USB Device(s) found
       scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot:  0 
U-Boot>

Das U-Boot is ready.

Although it says above that the bcm2710-rpi-3-b-plus.dtb is loaded, it is not used, since it is for the linux kernel. As u-boot is ready now, it can be used to boot anything actually.

A small note, although the name of the file (kernel8.img) is not that important, because the firmware is thinking this is a linux kernel, it still passes some information like it is passing to linux kernel.

Summary

In the second part, we first booted a normal Linux 32-bit kernel, then a custom build Linux 64-bit kernel, and finally the u-boot bootloader.

At this point, rpi is non-functional, but this was a step needed to proceed with bare metal programming.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.