Zero Client: Boot kernel and root filesystem from network with a Raspberry Pi2 or Pi3

Boot your Raspberry Pi from nothing but an Ethernet cable

A Zero Client is a computer that has nothing on its permanent storage but a bootloader. Rather, it loads everything from the network.

With the method presented in this article, you will be able to boot a Raspberry Pi into a full Debian OS with nothing more on the SD card other than the Raspberry firmware files and the u-boot bootloader on a FAT file system. The Linux kernel and the actual OS will be served over the local ethernet network.

We will only focus on the Raspberry Pi 3, but the instructions should work with minor adaptations also on a Pi 2.

The following instructions assume that you have already built…

  1. a full root file system for the Raspberry
  2. a u-boot binary, and
  3. a Linux kernel

… based on my previous blog post. Thus, you should already have the following directory structure:

We will do all the work inside of the ~/workspace directory.

Preparation of the SD card

You will only need a small SD card with a FAT filesystem on it. The actual storage of files in the running OS will be transparently done over the network. Mount the filesystem on /mnt/sdcard and do the following:

Copy firmware

Copy u-boot bootloader

Create config.txt

config.txt is the configuration file read by the Raspberry firmware blobs. Most importantly, it tells the firmware what kernel to load. “Kernel” is a misleading term here, since we will boot u-boot rather than the kernel.

Create /mnt/sdcard/config.txt with the following contents:

 

Make an universal boot script for the u-boot bootloader

To achieve maximum flexibility — to avoid the repetitive dance of manually removing the SD card, copying files to it, and re-inserting it — we will make an universal u-boot startup script that does nothing else than loading yet another u-boot script from the network. This way, there is nothing specific about the to-be-loaded Kernel or OS on the SD card at all.

Create a file boot.scr.mkimage  with the following contents:

Replace the server IP with the actual static IP of your server. Note that this script does nothing else other than loading yet another script called netboot-${serial#}.scr  from the server. serial# is the serial number which u-boot extracts from the Raspberry Pi hardware. This is usually the ethernet network device HW address. This way, you can have separate startup scripts for several Raspberry Pi’s if you have more than one. To keep the setup simple, set the file name to something predictable.

Compile the script into an u-boot readable image:

Copy boot.scr to the SD card:

The SD card preparation is complete at this point. We will now focus on the serving of the files necessary for boot.

Preparation of the file server

Do all of the following as ‘root’ user on a regular PC running Debian 9 (“Stretch”). This PC will act as the “server”.  This server will serve the files necessary to network-boot the Raspberry.

The directory /srv/tftp will hold …

  • an u-boot start script file
  • the kernel uImage file
  • and the binary device tree file.

… to be served by a TFTP server.

The directory /srv/rootfs_rpi3 will hold our entire root file system to be served by a NFS server:

You will find installation instructions of both TFTP and NFS servers further down.

 

Serve the root file system

Let’s copy the pre-built root file system into the directory from where it will be served by the NFS server:

(notice the slash at the end of the source directory)

 

Fix the root file system for network booting

Edit  /srv/rootfs_rpi3/etc/fstab  and comment out all lines. We don’t need to mount anything from the SD card.

When network-booting the Linux kernel, the kernel will configure the network device for us (either with a static IP or DHCP). Any userspace programs attempting to re-configure the network device will cause problems, i.e. a loss of conncection to the NFS server. Thus, we need to prevent systemd-networkd from managing the Ethernet device. Make the device unmanaged by removing the folowing ethernet configuration file:

If you don’t do that, you’ll get the following kernel message during boot:

That is because systemd has shut down and then re-started the ethernet device. Apparently NFS transfers are sensitive to that.

In case you want to log into the chroot to make additional changes that can only be done from within (e.g. running systemctl scripts etc.), you can do:

 

Serve Kernel uImage

In this step, we create a Linux kernel uImage that can be directly read by the u-boot bootloader. We read Image.gz directly from the Kernel source directory, and output it into the /srv/tftp directory where a TFTP server will serve it to the Raspberry:

 

Serve device tree binary

The u-boot bootloader will also need to load the device tree binary and pass it to the Linux kernel, so copy that too into the /srv/tftp directory.

 

Serve secondary u-boot script loading the kernel

Create a file netboot-rpi3.scr.mkimage with the following contents:

Replace the server IP with the static IP of your server PC. Then compile this script into an u-boot readable image and output it directly to the /srv/tftp directory:

Make sure that the filename of the .scr file matches with whatever file name you’ve set in the universal .scr script that we’ve prepared further above.

 

Install a NFS server

The NFS server will serve the root file system to the Raspberry and provide transparent storage.

Edit /etc/exports and add:

To apply the changed ‘exports’ configuration, run

Useful to know about the NFS server:

You can restart the NFS server by running service nfs-kernel-server restart

Configuration files are /etc/default/nfs-kernel-server  and /etc/default/nfs-common

 

Test NFS server

If you want to be sure that the NFS server works correctly, do the following on another PC:

Mount the root file system (fix the static IP for your server):

 

 

Install a TFTP server

To install:

After installation, check if the TFTP server is running:

This command will tell you the default serving directory (/srv/tftp):

Here is another command that tells you if the TFTP server is listening:

To get help about this server: man tftpd

Test TFTP

If you want to be sure that the TFTP server works correctly, do the following on another PC:

Then see if the server serves the Linux kernel we’ve installed before:

You now should have a local copy of the linux-rpi3.uImage file.

 

Complete

If you’ve done all of the above correctly, you can insert the prepared SD card into your Raspberry Pi and reboot it. The following will happen:

  1. The Raspberry Pi GPU will load the firmware blobs from the SD card.
  2. The firmware blobs will boot the image specified in config.txt. In our case, this is the u-boot binary on the SD card.
  3. The u-boot bootloader will boot.
  4. The u-boot bootloader loads and runs the universal boot.scr script from the SD card.
  5. The boot.scr downloads the specified secondary boot script from the network and runs it.
  6. The secondary boot script …
    • downloads the device tree binary from the network and loads it into memory.
    • downloads the Linux kernel from the network and loads it into memory
    • passes the device tree binary to the kernel, and boots the kernel
  7. the Linux kernel will bring up the ethernet device, connect to the NFS server, and load the regular OS from there.

Many things can go wrong in this rather long sequence, so if you run into trouble, check the Raspberry boot messages output on an attached screen or serial console, and the log files of the NFS and TFTP servers on your server PC.

 

Resources

https://www.raspberrypi.org/documentation/linux/kernel/building.md

http://www.whaleblubber.ca/boot-raspberry-pi-nfs/

https://cellux.github.io/articles/moving-to-nfs-root/

http://billauer.co.il/blog/2011/01/diskless-boot-nfs-cobbler/

https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt

http://wiki.linux-nfs.org/wiki/index.php/General_troubleshooting_recommendations

https://wiki.archlinux.org/index.php/NFS

Reading Raspberry Pi chip temperature with mainline Linux kernel

This tutorial is based on a previous article where we installed pure Debian 9 with a recent mainline/vanilla Linux kernel on a Raspberry Pi, and so differs from what would be done on a Raspbian Distribution with a Raspbian kernel. In this article, we will read the Raspberry Pi chip temperature. Here is my previous article:

Raspberry Pi 2 running pure Debian 9 (“Stretch”) and the Linux Mainline/Vanilla Kernel

As of mainline Linux Kernel 4.9.0-rc3, the sysfs entry /sys/class/thermal for the Broadcom BCM283x chip found on Raspberry Pi’s is empty. You can apply the following patch to Linux kernel 4.9.0-rc3, even though it will soon be superfluous because it seems that currently there is ongoing work by Linux Kernel developers to add in the missing functionalities.

The patch which will get you the standard sysfs temperature node which you can read like this:

This is the chip temperature in thousandths of degrees cenigrade, i.e. 40.084 degrees Celsius.

The following patch will give you a new entry in the mainline Kernel config, under Drivers -> SoC -> BCM -> Raspberry Pi thermal sysfs driver, which you have to enable, then recompile your kernel.

I have actually submitted this patch to a Linux kernel developer, but the process to get code into the Linux kernel is quite elaborate, and he said that they are already working on it, so I let it drop and decided to write this blog post instead.

 

Setting I2C bus speed on a Raspberry Pi via Device Tree

This tutorial is based on my previous article where we installed pure Debian 9 with a recent mainline/vanilla Linux kernel, and so differs from what would be done on a Raspbian Distribution with a Raspbian kernel. In this article, we will set the I2C bus speed on a Raspberry Pi. Here is my previous article:

Raspberry Pi 2 running pure Debian 9 (“Stretch”) and the Linux Mainline/Vanilla Kernel

Device Trees

The I2C bus on the Broadcom BCM283x chips found on Raspberry Pi’s is well and directly supported by the mainline/vanilla Linux kernel. Since with the Raspberry Pi we’re dealing with a System on a Chip (SoC), and not a regular PC, the hardware is configured with so-called device trees, which is a low-level description of the chip hardware compiled from text into binary format.

The rpi23-gen-image script mentioned in my previous tutorial installs the binary device tree into /boot/firmware/bcm2836-rpi-2-b.dtb. The U-Boot bootloader can read this file and pass it to the Linux kernel which interprets it and enables all the mentioned features in it.

The clock frequency for the I2C bus is configured in this .dtb file, and the default is 100kHz. There is a tool which allows you to inspect the .dtb file, outputting regular text. With this tool you also can make changes to the device configuration. Nowadays, this is the proper way to configure low-level devices on SoC’s!

 

Read the device tree

This will output the decoded device tree as text. Regarding I2C, you will find i2c@-entries like this:

Change the device tree

The clock-frequency value is what we want to change. The value is a raw binary unsigned 32-bit int stored big-endian, unreadable for humans.

But you can use another tool fdtget to read just this value decoded:

This will output 100000.

In my case, I wanted to set the I2C bus to the slowest frequency, to compensate for long cable lengths. I found that one of the lowest supported I2C clock frequencies is 4kHz. With fdtput you can set the clock-frequency property for each i2c device (there are 3 on the RPi):

And that’s it. You have to reboot for these settings to take effect.

I used an oscilloscope to verify that the SCL pin of the GPIO of the RPi indeed toggled with 4kHz, and it did!

Raspberry Pi2 and Pi3 running pure Debian 9 (“Stretch”) and the Linux Mainline/Vanilla Kernel

Update 2017 Feb 25: I have updated the step-by-step instructions based on the suggested fixes and improvements contained in the reader comments. I also have copied the step-by-step instructions from this blog post to the README.md file hosted on https://github.com/michaelfranzl/rpi23-gen-image. From now on I will update the instructions only on github, so expect that the instructions in this blog post will grow slightly out of date.

Update 2017 Mar 4: 64 bit kernel and Debian OS now works on the RPi3.

Gallium graphics drivers for Raspberry Pi’s VC4 chip are now fully supported by Linux Mainline

This has been a long way coming. In February 2014, Broadcom announced that they would release the formerly closed-source drivers for the VideoCore IV (VC4) GPU of their BCM283x family of System-on-a-chip (SoC), powering Raspberry Pi’s.

To make a long story short, Eric Anholt started porting the Open Source drivers, as documented by this presentation early 2015, and contributed code to the Linux (Mainline) Kernel, libdrm, Mesa, and X.org. I’m sure it was a long and painful work. But the results are worth it. A picture says more than 1000 words:

Debian 9 ("Stretch") running on a Raspberry Pi 2, powered by Linux 4.9.0-rc3 Mainline/Vanilla Kernel. Notable in this image: Graphics driver is "Gallium" running on the VC4 GPU of the Broadcom 2836 system-on-a-chip (SOC). Glxgears runs with 60 FPS and consumes very little CPU. I2C interface is recognized.
Debian 9 (“Stretch”) running on my Raspberry Pi 2 (and 3), powered by Linux 4.9.0-rc3 Mainline/Vanilla Kernel. Notable in this image: Graphics driver is recognized as “Gallium” running on the VC4 GPU of the Broadcom 2836 system-on-a-chip (SOC). The glxgears benchmark runs with 60 FPS (the vsync of the monitor) and consumes very little CPU. Even the Raspberry I2C interface is recognized by the Linux Mainline Kernel.

To emphasize the point: It is no longer necessary to run specialized distributions (like Raspbian) or Linux kernels (like the Raspbian flavor) in order to have the Raspberry Pi well supported. And this is good news. Debian is a well established and maintained standard Distribution. And even though the Raspberry Pi is not powerful enough for the professional desktop user, it is powerful enough for the casual desktop user, and it is very small and cheap, which opens up a whole lot of possibities for new real-world applications.

I ran an additional test: Gnome even runs on Wayland (modern replacement for the X Window System) on the Raspberry Pi 2 (and 3):

Gnome on Wayland on Raspberry Pi 2
Gnome on Wayland on Raspberry Pi 2

 

 

 

 

Remarks

It still is not a matter of a one-click installer to reproduce these results, you need some experience when you run into barriers. But it has gotten a whole lot easier. Github user drtyhlpr thankfully published the script rpi23-gen-image that can create a standard Debian distribution for the Raspberry Pi that can simply be copied to a SD card.

I have created a fork of this script to use the official Linux kernel instead of the Raspberry flavor one. Above screenshots are taken from a system that I’ve created with this script. My fork is developed into a slightly different direction:

  • Only official Debian releases 9 (“Stretch”) and newer are supported.
  • Only the official/mainline/vanilla Linux kernel is supported (not the raspberry flavor kernel).
  • The Linux kernel must be pre-cross-compiled on the PC running this script (instructions below).
  • Only U-Boot booting is supported.
  • The U-Boot sources must be pre-downloaded and pre-cross-compiled on the PC running this script (instructions below).
  • An apt caching proxy server must be installed to save bandwidth (instructions below).
  • The installation of the system to an SD card is done by simple copying or rsyncing, rather than creating, shrinking and expanding file system images.
  • The FBTURBO option is removed in favor or the working VC4 OpenGL drivers of the mainline Linux kernel.

All of these simplifications are aimed at higher bootstrapping speed and maintainability of the script. For example, we want to avoid testing of all of the following combinations:

RPi2 with u-boot, with official kernel
RPi2 without u-boot, with official kernel
RPi2 with u-boot, with raspberry kernel
RPi2 without u-boot, with raspberry kernel
RPi3 with u-boot, with official kernel
RPi3 without u-boot, with official kernel
RPi3 with u-boot, with raspberry kernel
RPi3 without u-boot, with raspberry kernel

Thus, the script only supports:

RPi2 with u-boot with official kernel
RPi3 with u-boot with official kernel

RPi2 (setting RPI_MODEL=2) is well supported. It will run the arm architecture of Debian, and a 32-bit kernel. You should get very good results, see my related blog posts:

https://michaelfranzl.com/2016/10/31/raspberry-pi-debian-stretch/

https://michaelfranzl.com/2016/11/10/setting-i2c-speed-raspberry-pi/

https://michaelfranzl.com/2016/11/10/reading-cpu-temperature-raspberry-pi-mainline-linux-kernel/

The newer RPi3 (setting RPI_MODEL=3) is supported too. It will run the arm64 architecture of Debian, and a 64-bit kernel. The support of this board by the Linux kernel will very likely improve over time.

In general, this script is EXPERIMENTAL. I do not provide ISO file system images. It is better to master the process rather than to rely on precompiled images. In this sense, use this project only for educational purposes.

How to do it

Basically, we will deboostrap a minimal Debian 9 (“Stretch”) system for the Raspberry on a regular PC running also Debian 9 (“Stretch”). Then we copy that system onto a SD card, then boot it on the Raspberry.

We will work with the following directories:

Set up your working directory:

Do the following steps as root user.

 

Set up caching for apt

This way, you won’t have to re-download hundreds of megabytes of Debian packages from the Debian server every time you run the rpi23-gen-image script.

Check its status page:

 

Install dependencies

The following list of Debian packages must be installed on the build system because they are essentially required for the bootstrapping process.

For a RPi2, you also need:

For a RPi3, you also need:

Kernel compilation

Get the latest Linux mainline kernel. This is a very large download, about 2GB. (For a smaller download of about 90 MB, consider downloading the latest stable kernel as .tar.xz from https://kernel.org.)

Confirmed working revision (approx. version 4.10, Feb 2017): 60e8d3e11645a1b9c4197d9786df3894332c1685

Working configuration files for this Linux kernel revision are included in this repository. (working-rpi2-linux-config.txt and working-rpi3-linux-config.txt).

If you want to generate the default .config file that is also working on the Raspberry, execute

For a RPi2:

For a RPi3:

Whichever .config file you have at this point, if you want to get more control as to what is enabled in the kernel, you can run the graphical configuration tool at this point:

For a RPi2:

For a RPi3:

Before compiling the kernel, back up your .config file so that you don’t lose it after the next make mrproper:

Compiling the kernel

Clean the sources:

Optionally, copy your previously backed up .config:

Find out how many CPU cores you have to speed up compilation:

Run the compilation on all CPU cores. This takes about 10 minutes on a modern PC:

For a RPi2:

For a RPi3:

Verify that you have the required kernel image.

For a RPi2 this is:

For a RPi3 this is:

 

U-Boot bootloader compilation

Confirmed working revision: b24cf8540a85a9bf97975aadd6a7542f166c78a3

Let’s increase the maximum kernel image size from the default (8 MB) to 64 MB. This way, u-boot will be able to boot even larger kernels. Edit ./u-boot/include/configs/rpi.h  and add above the very last line (directly above “#endif”):

Find out how many CPU cores you have to speed up compilation:

Compile for a RPi model 2 (32 bits):

Compile for a RPi model 3 (64 bits):

Verify that you have the required bootloader file:

Pre-download Raspberry firmware

The Raspberry Pi still needs some binary proprietary blobs for booting. Get them:

Confirmed working revision: bf5201e9682bf36370bc31d26b37fd4d84e1cfca

Build the system!

This is where you call the rpi23-gen-image.sh script.

For example:

You may want to modify the variables according to the section “Command-line parameters” below.

The file example.sh in my github repostory contains a working example.

Install the system on a SD card

Insert a SD card into the card reader of your host PC. You’ll need two partitions on it. I’ll leave as an exercise for the reader the creation of a partition table according to the following output of fdisk for a 64GB card:

The following commands will erase all contents of the SD card and install the system (copy via rsync) on the SD card:

Note about SD cards: Cheap (or sometimes even professional) SD cards can be weird at times. I’ve repeatedly noticed corrupt/truncated files even after proper rsync and proper umount on different brand new SD cards. TODO: Add a method to verify all file checksums after rsync.

Try booting the Raspberry

Insert the SD card into the Raspberry Pi, and if everything went well, you should see a console-based login prompt on the screen. Login with the login details you’ve passed into the script (USER_NAME and PASSWORD).

Alternatively, if you have included “avahi-daemon” in your APT_INCLUDES, you don’t need a screen and keyboard and can simply log in via SSH from another computer, even without knowing the Rasberry’s dynamic/DHCP IP address (replace “hostname” and “username” with what you have set as USER_NAME and HOSTNAME above):

Finishing touches directly on the Raspberry

Remember to change usernames, passwords, and SSH keys!

Check uber-low RAM usage

Running top shows that the freshly booted system uses only 23 MB out of the availabl 1GB RAM! Confirmed for both RPi2 and RPi3.

Network Time Synchronization

The Raspberry doesn’t have a real time clock. But the default systemd conveniently syncs time from the network. Check the output of timedatectl. Confirmed working for both RPi2 and RPi3.

Hardware Random Number Generator

The working device node is available at /dev/hwrng. Confirmed working for both RPi2 and RPi3.

I2C Bus

Also try I2C support:

Confirmed working for both RPi2 and RPi3.

Test onboard LEDs

As of the kernel revision referenced above, this only works on the RPi2. The RPi3 has only the red PWR LED on all the time, but otherwise is working fine.

By default, the green onboard LED of the RPi blinks in a heartbeat pattern according to the system load (this is done by kernel feature LEDS_TRIGGER_HEARTBEAT).

To use the green ACT LED as an indicator for disc access, execute:

To toggle the red PWR LED:

Or use the red PWR LED as heartbeat indicator (kernel option for this must be enabled):

Notes about systemd

systemd now replaces decades-old low-level system administration tools. Here is a quick cheat sheet:

Reboot machine:

Halt machine (this actually turns off the RPi):

Show all networking interfaces:

Show status of the Ethernet adapter:

Show status of the local DNS caching client:

Install GUI

Successfully tested on the RPi2 and RPI3.

If you want to install a graphical user interface, I would suggest the light-weight LXDE window manager. Gnome is still too massive to run even on a GPU-accelerated Raspberry.

Reboot, and you should be greeted by the LightDM greeter screen!

Test GPU acceleration via VC4 kernel driver

Successfully tested on the RPi2 and RPI3.

Glxinfo should output:

Kernel compilation directly on the Rasberry

Only successfully tested on the RPi2. Not yet tested on the RPI3.

In case you want to compile and deploy another Mainline Linux kernel directly on the Raspberry, proceed as described above, but you don’t need the ARCH and CROSS_COMPILE flags. Instead, you need the -fno-pic compiler flag for modules. The following is just the compilation step (configuration and installation omitted):

Follow-up articles

Setting I2C bus speed on a Raspberry Pi via Device Tree

Reading Raspberry Pi chip temperature with mainline Linux kernel

Zero Client: Boot kernel and root filesystem from network with a Raspberry Pi2 or Pi3

.