30.7. Diskless Operation

Updated by Jean-François Dockès.
Reorganized and enhanced by Alex Dupre.

A FreeBSD machine can boot over the network and operate without a local disk, using file systems mounted from an NFS server. No system modification is necessary, beyond standard configuration files. Such a system is relatively easy to set up because all the necessary elements are readily available:

The Intel® Preboot eXecution Environment (PXE) can be used to load the kernel over the network. It provides a form of smart boot ROM built into some networking cards or motherboards. See pxeboot(8) for more details.

A sample script (/usr/share/examples/diskless/clone_root) eases the creation and maintenance of the workstation's root file system on the server. The script will probably require a little customization.

Standard system startup files exist in /etc to detect and support a diskless system startup.

Swapping, if needed, can be done either to an NFS file or to a local disk.

There are many ways to set up diskless workstations. Many elements are involved, and most can be customized to suit local taste. The following will describe variations on the setup of a complete system, emphasizing simplicity and compatibility with the standard FreeBSD startup scripts. The system described has the following characteristics:

Caution:

As described, this system is insecure. It should live in a protected area of a network and be untrusted by other hosts.

30.7.1. Background Information

Setting up diskless workstations is both relatively straightforward and prone to errors. These are sometimes difficult to diagnose for a number of reasons. For example:

  • Compile time options may determine different behaviors at runtime.

  • Error messages are often cryptic or totally absent.

In this context, having some knowledge of the background mechanisms involved is useful to solve the problems that may arise.

Several operations need to be performed for a successful bootstrap:

  • The machine needs to obtain initial parameters such as its IP address, executable filename, server name, and root path. This is done using the DHCP or BOOTP protocols. DHCP is a compatible extension of BOOTP, and uses the same port numbers and basic packet format. It is possible to configure a system to use only BOOTP and bootpd(8) is included in the base FreeBSD system.

  • DHCP has a number of advantages over BOOTP such as nicer configuration files and support for PXE. This section describes mainly a DHCP configuration, with equivalent examples using bootpd(8) when possible. The sample configuration uses ISC DHCP which is available in the Ports Collection.

  • The machine needs to transfer one or several programs to local memory. Either TFTP or NFS are used. The choice between TFTP and NFS is a compile time option in several places. A common source of error is to specify filenames for the wrong protocol. TFTP typically transfers all files from a single directory on the server and expects filenames relative to this directory. NFS needs absolute file paths.

  • The possible intermediate bootstrap programs and the kernel need to be initialized and executed. PXE loads pxeboot(8), which is a modified version of the FreeBSD third stage loader, loader(8). The third stage loader will obtain most parameters necessary to system startup and leave them in the kernel environment before transferring control. It is possible to use a GENERIC kernel in this case.

  • Finally, the machine needs to access its file systems using NFS.

Refer to diskless(8) for more information.

30.7.2. Setup Instructions

30.7.2.1. Configuration Using ISC DHCP

The ISC DHCP server can answer both BOOTP and DHCP requests.

ISC DHCP is not part of the base system. Install the net/isc-dhcp42-server port or package.

Once ISC DHCP is installed, edit its configuration file, /usr/local/etc/dhcpd.conf. Here follows a commented example for PXE host corbieres:

default-lease-time 600;
max-lease-time 7200;
authoritative;

option domain-name "example.com";
option domain-name-servers 192.168.4.1;
option routers 192.168.4.1;

subnet 192.168.4.0 netmask 255.255.255.0 {
  use-host-decl-names on; 1
  option subnet-mask 255.255.255.0;
  option broadcast-address 192.168.4.255;

  host corbieres {
    hardware ethernet 00:02:b3:27:62:df;
    fixed-address corbieres.example.com;
    next-server 192.168.4.4; 2
    filename "pxeboot"; 3
    option root-path "192.168.4.4:/data/misc/diskless"; 4
  }
}

1

This option tells dhcpd to send the value in the host declarations as the hostname for the diskless host. An alternate way would be to add an option host-name corbieres inside the host declarations.

2

The next-server directive designates the TFTP or NFS server to use for loading loader(8) or the kernel file. The default is to use the same host as the DHCP server.

3

The filename directive defines the file that PXE will load for the next execution step. It must be specified according to the transfer method used. PXE uses TFTP, which is why a relative filename is used here. Also, PXE loads pxeboot, not the kernel. There are other interesting possibilities, like loading pxeboot from a FreeBSD CD-ROM /boot directory. Since pxeboot(8) can load a GENERIC kernel, it is possible to use PXE to boot from a remote CD-ROM.

4

The root-path option defines the path to the root file system, in usual NFS notation. When using PXE, it is possible to leave off the host's IP address as long as the BOOTP kernel option is not enabled. The NFS server will then be the same as the TFTP one.

30.7.2.2. Booting with PXE

By default, pxeboot(8) loads the kernel via NFS. It can be compiled to use TFTP instead by specifying the LOADER_TFTP_SUPPORT option in /etc/make.conf. See the comments in /usr/share/examples/etc/make.conf for instructions.

There are two other make.conf options which may be useful for setting up a serial console diskless machine: BOOT_PXELDR_PROBE_KEYBOARD, and BOOT_PXELDR_ALWAYS_SERIAL.

To use PXE when the machine starts, select the Boot from network option in the BIOS setup or type a function key during system initialization.

30.7.2.3. Configuring the TFTP and NFS Servers

If PXE is configured to use TFTP, enable tftpd(8) on the file server:

  1. Create a directory from which tftpd(8) will serve the files, such as /tftpboot.

  2. Add this line to /etc/inetd.conf:

    tftp	dgram	udp	wait	root	/usr/libexec/tftpd	tftpd -l -s /tftpboot

    Note:

    Some PXE versions require the TCP version of TFTP. In this case, add a second line, replacing dgram udp with stream tcp.

  3. Tell inetd(8) to reread its configuration file. Add inetd_enable="YES" to /etc/rc.conf in order for this command to execute correctly:

    # service inetd restart

Place tftpboot anywhere on the server. Make sure that the location is set in both /etc/inetd.conf and /usr/local/etc/dhcpd.conf.

Enable NFS and export the appropriate file system on the NFS server.

  1. Add this line to /etc/rc.conf:

    nfs_server_enable="YES"
  2. Export the file system where the diskless root directory is located by adding the following to /etc/exports. Adjust the mount point and replace corbieres with the names of the diskless workstations:

    /data/misc -alldirs -ro margaux corbieres
  3. Tell mountd(8) to reread its configuration file. If NFS is enabled in /etc/rc.conf, it is recommended to reboot instead.

    # service mountd restart

30.7.2.4. Building a Diskless Kernel

When using PXE, building a custom kernel with the following options is not strictly necessary. These options cause more DHCP requests to be issued during kernel startup, with a small risk of inconsistency between the new values and those retrieved by pxeboot(8) in some special cases. The advantage is that the host name will be set. Otherwise, set the host name in a client-specific /etc/rc.conf.

options     BOOTP          # Use BOOTP to obtain IP address/hostname
options     BOOTP_NFSROOT  # NFS mount root file system using BOOTP info

The custom kernel can also include BOOTP_NFSV3, BOOT_COMPAT and BOOTP_WIRED_TO. Refer to NOTES for descriptions of these options.

These option names are historical and slightly misleading as they actually enable indifferent use of DHCP and BOOTP inside the kernel.

Build the custom kernel, using the instructions in Chapter 9, Configuring the FreeBSD Kernel, and copy it to the place specified in /usr/local/etc/dhcpd.conf.

30.7.2.5. Preparing the Root File System

Create a root file system for the diskless workstations in the location listed as root-path in /usr/local/etc/dhcpd.conf.

30.7.2.5.1. Using make world to Populate Root

This method is quick and will install a complete virgin system, not just the root file system, into DESTDIR. Execute the following script:

#!/bin/sh
export DESTDIR=/data/misc/diskless
mkdir -p ${DESTDIR}
cd /usr/src; make buildworld && make buildkernel
make installworld && make installkernel
cd /usr/src/etc; make distribution

Once done, customize /etc/rc.conf and /etc/fstab placed into DESTDIR according to the system's requirements.

30.7.2.6. Configuring Swap

If needed, a swap file located on the server can be accessed via NFS.

30.7.2.6.1. NFS Swap

The kernel does not support enabling NFS swap at boot time. Swap must be enabled by the startup scripts, by mounting a writable file system and creating and enabling a swap file. To create a swap file:

# dd if=/dev/zero of=/path/to/swapfile bs=1k count=1 oseek=100000

To enable the swap file, add the following line to /etc/rc.conf:

swapfile=/path/to/swapfile

30.7.2.7. Miscellaneous Issues

30.7.2.7.1. Running with a Read-only /usr

If the diskless workstation is configured to run Xorg, adjust the XDM configuration file as it puts the error log on /usr by default.

30.7.2.7.2. Using a Non-FreeBSD Server

When the server for the root file system is not running FreeBSD, create the root file system on a FreeBSD machine, then copy it to its destination, using tar(1) or cpio(1).

In this situation, there are sometimes problems with the special files in /dev, due to differing major/minor integer sizes. A solution to this problem is to export a directory from the non-FreeBSD server, mount this directory onto a FreeBSD machine, and use devfs(5) to allocate device nodes transparently for the user.

All FreeBSD documents are available for download at http://ftp.FreeBSD.org/pub/FreeBSD/doc/

Questions that are not answered by the documentation may be sent to <freebsd-questions@FreeBSD.org>.
Send questions about this document to <freebsd-doc@FreeBSD.org>.