$FreeBSD: $
FreeBSD is a registered trademark of the FreeBSD Foundation.
Intel, Celeron, EtherExpress, i386, i486, Itanium, Pentium, and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this document, and the FreeBSD Project was aware of the trademark claim, the designations have been followed by the “™” or the “®” symbol.
I wrote this guide while setting up a "jumpstart" server to document what I had done so I could get back on the track if necessary. I used other guides as reference, but found that these were either outdated or lacked important details.
I started writing this guide using FreeBSD 5.3, but the project stopped because I had problems with diskless operation using the now obsolete version of dhclient(8). It turned out that using DHCP for configuration of diskless clients has not worked for a long time. Meanwhile 5.5 and 6.1 have been released. I am updating the document to match the latest stable release of FreeBSD, please excuse broken links and other errors.
Following the instructions in this guide may render your system(s) useless and cause data loss. Whatever you are up to, it is on your own risk.
I did screw up my own system quite seriously while trying to get things to do a jumpstart installation. I hope I have documented the path that does not screw up (too much) :-)
If you have problems with PXEBoot, please ask on the appropriate mailing list, don't ask me for help personally. This way everyone can help solve the problem and learn from it. If your problems are caused by errors in the document, or you have other comments on how to make this document better, feel free to contact me.
Please note: I have documented some advanced tricks that according to the documentation around should work, but I have not tested this yet. If you find examples that just won't work, please let me know. Before reporting this, make sure you are following the exact procedure described here, or else I can't know if the problem is caused by messed up configuration.
This section introduces the general concepts and ideas and the motivation for going into the next sections. You can skip this if you are in a hurry.
Intel's PXE (Preboot eXecution Environment) is a small boot code that lets your client query network configuration using DHCP and fetch a boot loader using TFTP. This has become standard in many computers and has replaced etherboot.
This guide sets out to explore two uses of network booting: Jumpstart installation of FreeBSD and diskless clients:
Jumpstart installation has roots at Sun (at least the name) and can be used to automate installation of a large number of identical machines or simply avoid using install cd's. A homogeneous server farm is easier to maintain, and automated installation lets you quickly setup a new server: plug it in, turn it on, drink coffee.
Diskless clients has a number of advantages: You can easily set up and manage a large number of identical diskless clients. No disk means less noise and longer lifetime of your hardware and a beter working environment for your users. No disk means that files must be fetched from a central server, hence only updating on the server.
In both cases the benefit is ease of maintenance when working with a large number of workstations or servers. If you're only doing it for one machine, then it's mostly for the sport and learning, but those too are valid reasons.
Setting up a server has been documented in numerous documents, however not every step is clear and various documents describe different paths to follow and some parts are obsolete. This document is probably no different.
The hardware must support PXE network booting. Many motherboards with on board Intel based network interface supports PXE. VIA produces mini-ITX format motherboards including some fanless ones, these are ideal for building cheap diskless and completely silent clients.
Any system can function as server as long as they support the services needed to serve the files to the booting client. However, it is easier to setup when using the same platform and version of FreeBSD on the server and client.
When a PXE client boots, the boot process goes through a number of stages, which are roughly as follows:
PXE sends a bootp query on the network and receives information for configuring the network interface: IP address, netmask, default router. Also it receives information about where to fetch a boot loader and the path name of the root device.
PXE fetches the boot loader with tftp (the boot loader is actually the normal FreeBSD loader with some extra pxe code and tftp or nfs support). Before any other file, the loader fetches boot/loader.rc and any included files using tftp. These files contains information about where to get a kernel and posibly a root device.
The kernel is loaded and the root file system mounted. The kernel will need to acquire information about the network configuration, either by using information passed from stage 2 or by sending a new bootp request.
To enter stage 1, PXE must be supported by the hardware. Alternatively one can install a flash boot image or boot on a floppy - this method is not covered in this document.
In stage 2, if pxeboot is compiled with nfs support (default) it will try to nfs mount the root device specified in the root-path option sent by the dhcp server. If pxeboot is compiled with tftp support, it will fetch the files using tftp. In both cases, it will first read the file boot/loader.rc where the path is relative to the nfs root directory or tftp directory.
What happens after stage 3 depends on how we populate the root file system.
The root device can either be a memory disk or an nfs share, both have their advantages and disadvantages:
A memory disk is loaded into RAM making file access fast once loaded, but storage space is limited.
A network mounted device, files are fetched as needed. Loading programs is slower but only programs actually needed are kept in memory and storage space is bigger.
Serving a memory disk is easy as it can be fetched with tftp, and tftp has to be setup anyway to serve the pxeboot loader. But, the memory disk is a bit more cumbersome to populate, customise and update.
The main disadvantage of using tftp to fetch a memory file system is that all clients will fetch the same files, load the same kernel and memory file system, regardless of hardware, version of FreeBSD or the purpose of the client.
With an nfs root device you can specify different root mounts for different clients. This means that it is posible to support multiple hardware platforms and configurations from the same server.
The main reason to avoid nfs is that it is difficult to set up in particular if there is a firewall in the way.
The jumpstart installation section will take the approach of using a memory disk device while for diskless clients we use nfs. So both methods are covered. Setting up pxeboot, you can choose the path you find best and ignore the other.
Keep in mind that all data will be transferred across the network, and at least initially without encryption making your traffic target for eavesdropping.
Do not boot diskless clients on an untrusted network! After boot, connections can be encrypted, but the encryption keys will have been sent in the clear at boot time.
The server configuration is similar for both diskless clients and jumpstart installation, allthough for diskless operation we will use NFS to serve files and for jumpstart we will use FTP.
This section describes the configuration of the required services. The population of the directories with files to be served is covered in the following sections.
We will assume that the server and pxebooting clients are located on the 172.16.0.0/16 subnet. All services may be identified by DNS lookup, such that nfs.example.com is the nfs server and tftp.example.com is the tftp server.
We will be using dhcpd v. 3.0 from the Internet Software Consortium (ISC) which can be installed from ports (net/isc-dhcp3-server). The configuration file, dhcpd.conf(5), should be something like this for the basic configuration:
# Configuration file for ISC's dhcpd v. 3.0
#
# Server is authoritative for the subnets.
authoritative;
# Disable dynamic dns update
ddns-update-style none;
subnet 172.16.0.0 netmask 255.255.0.0 {
range 172.16.128.0 172.16.255.254;
default-lease-time 3600;
max-lease-time 86400;
option domain-name "example.com";
option domain-name-servers ns.example.com;
option routers gateway.example.com;
}
This will assign unknown clients IP in the specified range. For jumpstart installation we add the following section within the subnet block:
group { # jumpstart clients
use-host-decl-names on;
next-server tftp.example.com;
filename "boot/pxeboot";
}
The line next-server tftp.example.com; tells the client where to fetch the bootloader, and filename "boot/pxeboot" tells which file to request. The path to the boot loader is relative to the root directory of the tftp server.
For diskless clients, the following section within the subnet block should replace the section above for jumpstart clients.
group { # diskless clients
use-host-decl-names on;
next-server tftp.example.com;
filename "boot/pxeboot";
option root-path "nfs.example.com:/var/diskless/FreeBSD";
host diskless-1 {
hardware ethernet 00:40:63:d4:89:73;
}
}
For diskless client setup, we also need to specify the root device: option root-path "nfs.example.com:/var/diskless/FreeBSD" tells the client where to get an NFS root mount, if not set it will default to <next-server>:/pxeroot. An alternative is to edit loader.conf(5) to specify a root device (this method is not covered for diskless clients, but is used for jumpstart).
For each client a host declaration is included, this may be omitted. but in particular if the network also hosts non-diskless clients it may be used to tighten control.
To enable dhcpd, add the following lines to rc.conf:
dhcpd_enable="YES" dhcpd_flags="-q" dhcpd_conf="/etc/dhcpd.conf" dhcpd_ifaces="" dhcpd_withumask="077" dhcpd_chuser_enable="YES" dhcpd_chroot_enable="NO" dhcpd_rootdir="/var/db/dhcpd"
Then start dhcpd:
# /usr/local/etc/rc.d/dhcpd.sh start
Note: If you make any changes to dhcpd.conf you need to restart the dhcpd daemon, you cannot just send a SIGHUP.
Unless you decide to serve all files with nfs you need to enable tftpd(8). tftp is supported in the FreeBSD base system, to enable tftp edit the file /etc/inetd.conf to enable the following line:
tftp dgram udp wait root /usr/libexec/tftpd tftpd -l -s /var/tftp
The -l option tells tftpd to log to syslogd(8), the -s /var/tftp instructs tftpd to change its root directory into /var/tftp.
If you have not enabled inetd(8) in your /etc/rc.conf, first add the line:
inetd_enable="YES"
Then start (or restart) inetd:
# /etc/rc.d/inetd start
For jumpstart installation we want to install using ftp rather than nfs. Using a local ftp server speeds up the installation as the local network is faster, saves bandwidth on the external connection and reduces load on the public servers.
You can use the ftpd distributed with FreeBSD and enable it in inetd.conf. I prefer vsftpd which can be installed from ports. To enable the vfstpd you need to add the following two lines to the vsftpd.conf file:
listen=YES background=YES
Then start the server with:
# /usr/local/etc/rc.d/vsftpd.sh start
You may want to toggle pasv_min_port and pasv_max_port to control which ports are used in pasive ftp-data. This is particularly important if there is a firewall between the client and the ftp server.
vsftpd will chroot into it's home directory as specified in the password file, usually /var/ftp. Now you need to fetch a release from a mirror and put it into some reasonable path, typically pub/FreeBSD/<RELEASE>.
NFS is a protocol for sharing file systems over the local network, it is an RPC service which makes it particularly difficult to handle, in particular across a firewall.
There are some limitations when exporting file systems. You can export an entire partition (disk label) or a specific directory. If you export a partition with the option -alldirs, then any sub-directory can be mounted, but with the permissions set for the partition. Alternatively you have to list each directory you wish to be exported and the permissions.
The latter should be preferred, although cumbersome, because it gives you more fine grained access control.
Enable nfs in /etc/rc.conf:
rpcbind_enable="YES" # Run the portmapper service (YES/NO). nfs_server_enable="YES" # This host is an NFS server (or NO). mountd_enable="YES" # Run mountd (or NO). mountd_flags="-r -p 59" # Force mountd to bind on port 59
As a minimum you need to enable rpcbind, nfsserver and mountd as shown above.
By default, when mountd starts it binds to some arbitrary port, and rpc is used to discover which, making it impossible to filter. With option '-p' mountd can be forced to bind to a specific port. Port 59 is assigned to "any private file service", so it sounds reasonable to use this.
You may optionally want to enable lockd and statd which provides file locking and status monitoring. The problem with these services is that they cannot be forced to bind to specific ports making filtering impossible.
Before starting up nfs, we must export the file systems the diskless clients need to mount. Users wants more than a bare base system, in particular they want access to their files in their home directory. The exports(5) file should be something like this:
/var/diskless/FreeBSD -ro -maproot=root:wheel -network 172.16.0.0 -mask 255.255.0.0 /home -alldirs -network 172.16.0.0 -mask 255.255.0.0 /var/diskless/<hostname>/var <hostname> /var/diskless/<hostname>/tmp <hostname>
Note: If /var/diskless is a directory residing in /var then you cannot use the option -alldirs and you cannot mount subdirectories of /var/diskless. Instead, you will have to list each directory which will be mounted as shown above. The positive effect is that this will enforce a more strict access control.
The -maproot option specifies which privileges the root user on the client will have for accessing files on the server via nfs. By default root:wheel is mapped to nobody:nobody.
We will need to create var and (optionally) /tmp directories and a swap file for each diskless client. This is described in the section diskless clients. In the above, it is assumed that the hostname of the diskless clients can be looked up and fixed ip address is assigned. If you control the network, you may choose not to specify the hostname or ip address.
Then start nfsd:
# /etc/rc.d/rpcbind start # /etc/rc.d/nfsd start # /etc/rc.d/mountd start
You should be able to see the exported shares with the command showmount(8). The directories exported should be created if they do not exist. If you update the exports file, you need to restart mountd for the changes to take effect.
If you are on a closed and controlled network you need not protect your server by a firewall, although it is recommended always to be cautious. If you have a firewall enabled however, you need to open such that the clients can fetch the files needed.
If you are on a closed and controlled network you need not protect your server by a firewall, although it is recommended always to be cautious. If you have a firewall enabled however, you need to open such that the clients can fetch the files needed.
To enable DHCP, you must allow packets with source port 68 and destination port 67 in, and the reverse out for any ip. The reason is that when the first dhcp request is sent, the client don't know it's ip and source ip port 67 in, and the reverse out for any ip. The reason is that when the first dhcp request is sent, the client don't know it's ip and source ip is set to 0.0.0.0, nor does it know the ip of the server so the destination ip is also set to 0.0.0.0:
pass in quick proto udp from 0/32 port = 68 to 0/32 port = 67 keep state pass in quick proto udp from 172.16.0.0/16 port = 68 to 172.16.0.2/32 port = 67 keep state
for inbound traffic and for outbound:
pass out quick proto udp from 172.16.0.2 port = 67 to 0/32 port = 68 keep state
TFTP receives requests on udp port 69, but files are transfered from ports in the dynamic port range (ports > 49151). To allow tftp you must add the following rules:
pass in quick proto udp from 172.16.0.0/16 to 172.16.0.2 port = 69 keep state
for inbound, and for outbound traffic:
pass out quick proto udp from 172.16.0.2 port > 49151 to 172.16.0.0/16 port > 49151 keep state
FTP is more complicated. The server listens on port 21 for incoming connections, as with any other service. FTP specifies two data transfer modes, active and passive. In active ftp-data, the server connects back to the client from port 20 to some unprivileged port chosen by the client. In passive ftp-data the client connects to some unprivileged port chosen by the server, typically in the dynamic port range.
# ftp pass in quick proto tcp from 172.16.0.0/16 to 172.16.0.2 port 21 flags S keep state # ftp-data passive pass in quick proto tcp from 172.16.0.0/16 to 172.16.0.2 port > 49151 flags S keep state
allows ftp-sessions and passive ftp-data, to allow active ftp-data, add the following line
# ftp-data active pass out quick proto tcp from 172.16.0.2 port 20 to 172.16.0.0/16 port > 1023 flags S keep state
We forced mountd to bind to a specific port, 59, this makes firewalling easy:
pass in quick proto tcp from 172.16.0.0/16 to 172.16.0.2 port 59 flags S keep state pass in quick proto tcp from 172.16.0.0/16 to 172.16.0.2 port 111 flags S keep state pass in quick proto tcp from 172.16.0.0/16 to 172.16.0.2 port 2049 flags S keep state pass in quick proto udp from 172.16.0.0/16 to 172.16.0.2 port 59 keep state pass in quick proto udp from 172.16.0.0/16 to 172.16.0.2 port 111 keep state pass in quick proto udp from 172.16.0.0/16 to 172.16.0.2 port 2049 keep state
Note: Originally, nfs was based on udp but FreeBSD's implementation supports tcp as well.
First we build the pxeboot loader with support for tftp, this means that the kernel will be fetched with tftp rather than mounting the nfs root filesystem first.
# cd /usr/src/sys/boot # make -DLOADER_TFTP_SUPPORT=YES
Note: If you have compiled world and /usr/obj exists this won't build, remove or rename this directory first.
Install it into the tftp directory, /var/tftp:
# cp i386/pxeldr/pxeboot /var/tftp/boot/ # cp i386/boot0/boot0 /var/tftp/boot/ # cp i386/boot2/boot1 /var/tftp/boot/ # cp i386/boot2/boot2 /var/tftp/boot/ # cp i386/mbr/mbr /var/tftp/boot/
For jumpstart installation we need the kernel to support tftp in order that it can fetch the root file system, this is a build option, the GENERIC kernel configuration file can be used without changes. For diskless systems, the GENERIC kernel supports NFS and NFS mount of the root file system, but additional BOOTP support will prove useful.
When configuring the kernel, make sure to include the network drivers for the client, or load them before boot. Also, make sure the following options are included in the kernel configuration file. For Jumpstart:
# Filesystems options MD_ROOT # MD is a potential root device options PSEUDOFS # Pseudo-filesystem framework # Memory pseudo devices device md # Memory "disks" needed for root file system
And for diskless operation:
options NFSCLIENT options NFS_ROOT options BOOTP options BOOTP_COMPAT
The options BOOTP and BOOTP_COMPAT allow the kernel to rerequest certain options sent with the dhcp response such as hostname. Strictly, these are not required to boot the diskless system, but having the hostname set makes it easier to set host specific configuration option.
Build the kernel:
# cd /usr/src # make -DLOADER_TFTP_SUPPORT=YES buildkernel
This builds a kernel with support for tftp. If you are compiling the kernel for a different version of FreeBSD than the server system, you need to "make buildworld" before building the kernel.
We will not overwrite our system kernel, instead we install directly into the directory /var/tftp. We first need to install a few files by hand:
# mkdir -p /var/tftp/boot/defaults # cp /usr/src/sys/i386/conf/GENERIC.hints /var/tftp/boot/device.hints # cp /usr/src/sys/boot/forth/loader.conf /var/tftp/boot/defaults/ # cp /usr/src/sys/boot/forth/loader.4th /var/tftp/boot/defaults/ # cp /usr/src/sys/boot/forth/support.4th /var/tftp/boot/defaults/
Note: Strictly, to install the kernel loader.4th and support.4th are not required, but we will include loader.4th in loader.rc and this includes support.4th.
Then install the kernel
# make DESTDIR=/var/tftp installkernel
In the tftp root directory, /var/tftp, create boot/loader.rc containing the following lines:
include /boot/loader.4th start
For Jumpstart installation create boot/loader.conf containing the following:
init_path="/stand/sysinstall" rootfs_load="YES" rootfs_name="/boot/jumpstart" rootfs_type="mfs_root" vfs.root.mountfrom="ufs:/dev/md0c" autoboot_delay=5
The loader will by default search first for init(8) and if not found then try to run sysinstall(8). Setting init_path allows us to toggle wether to run sysinstall or init at boot, provided both are present. You may add any kernel modules to be loaded with the usual syntax, in that case these are fetched with tftp and must be in the tftp directory.
The rootfs_name specifies that the loader should fetch the file /boot/jumpstart for use as root device, needed for jumpstart installation. Although the path appears absolute, on the server, this is relative to the tftp directory. In the next section we will create the root file system. You should not set any rootfs options, in fact no loader.conf is needed, the defaults will work fine.
If you try to boot at this stage, the pxeclient client should complete stage 3 and load the kernel.
In this section covers how to set up a jumpstart installation. By the end of this section, you should have set up your server such that (almost) all you need to do to install FreeBSD is to connect your new machine to your network.
The jumpstart installation will load a memory disk as root device and fetch the release using ftp. We could also do this providing both root device and release files with nfs, if you want to to this, skip to the next section.
For jumpstart installation we use a memory file system as root device. The easiest way to obtain such a memory disk is to grap the one found on the installation media, mfsroot.gz, and copy it to /var/tftp/boot/jumpstart.gz.
If you plan on interactive installation or only simple scripting, there is no reason to do the trouble of customizing a memory root file system.
You may want to create a custom root file system in order to enable special customization prior or posterior to the installation that need extra scripting support. This section explains how.
Essentially, the memory disk device distributed on the installation media contains just a minimal set of configuration files and a "bootcrunch" file. A bootcrunch file is a special compressed binary containing statically linked executables. This format is useful when storage is limited.
The memory disk is loaded into the RAM, which determines the upper size limit for a memory root device. You can create a custom memory disk in two ways:
If RAM is limited, you can create a custom bootcrunch file.
If you have plenty of RAM, then you can install the base system on a memory file system and load that. The base system takes up some 150 MB and with systems shipping with 1024MB RAM this may be the easiest option.
To create the bootcrunch file we can either customize the bootcrunch configuration file found in release/i386 - without any changes this will build the bootcrunch file found on installation media, or you can use the "rescue" make files as skeleton.
In the following we will use the makefiles found in /usr/src/resuce as skeleton to create a custom bootcrunch file. It turns out to be easier, because the make file also allows us to easily install the bootcrunch binary on the memory file system.
First create a copy of the rescue directory to leave original unmodified:
# cd /usr/src # cp -R rescue jumpstart # cd jumpstart # mv rescue jumpstart # mv librescue libjumpstart
In this directory there are three make-files, one in the top directory and one in the two sub-directories jumpstart and libjumpstart. Edit all these to replace all occurences of "rescue" with "jumpstart".
The make file in jumpstart defines which binaries and libraries to include, libjumpstart adds some specially linked libraries. Edit jumpstart/Makefile to suit your needs, you can use the bootcrunch configuration file found in release/i386 to see what you must include.
We need to create and mount the memory device to install the memory root file system. It can be any size as long as the jumpstart client has enough RAM. We create a 16MB memory disk:
# dd if=/dev/zero of=/var/tftp/boot/jumpstart bs=1k count=16k # mdconfig -a -t vnode -f jumpstart -u0 # disklabel -r -w md0 auto # newfs /dev/md0c
Next mount the file system:
# mount /dev/md0 /mnt # cd /mnt
We now have an empty memory disk which will be served the client. Now we need to populate the file system to start the system installation, first create the base directory layout:
# cd /usr/src # mtree -U -p /mnt -f /etc/mtree/BSD.root.mtree # make DESTDIR=/mnt distribution # mkdir /mnt/jumpstart
Now build and install the bootcrunch file:
# cd /usr/src/jumpstart # make NO_ATM=YES # make DESTDIR=/mnt install
Note: With the jumpstart makefiles I created, I got errors trying to compile with ATM, disabling this solved the problem.
And to finish up,
# cd /mnt # rmdir /mnt/sbin /mnt/bin # ln -s jumpstart bin # ln -s jumpstart sbin
dhclient uses a shell script which must be installed separately:
# cp /usr/src/sbin/dhclient/dhclient-script /mnt/jumpstart/dhclient-script # chmod +x /mnt/jumpstart/dhclient-script
Unmount the file system and run fsck:
# cd / # umount /mnt/mfs # fsck -t ufs /dev/md0 # mdconfig -d -u 0 # gzip /var/tftp/boot/jumpstart
At this point we have a fully equipped memory file system that can be used for installation. Booting up your jumpstart client, you should get the standard interactive installation menu.
Note: The release is set according to the version of your kernel, if your kernel is 6.1-RELEASE sysinstall will assume this is the version you want to install. If you built a kernel from updated sources you may have a 6-STABLE kernel, and installation will fail since there is no such release. You can set the release you want to install in the menus.
Booting with PXE takes more time as all the files needs to be fetched over the network. The pxeboot program will first look for compressed files, then the non-compressed, the only file you cannot compres is pxeboot.
So, to speed up the transfer, you can gzip the kernel, any kernel modules and the memory file system. We may need to edit and customize the configuration files, so we let these be uncompressed.
Before you reboot, make sure that your new server won't boot again using pxeboot, hence repeating the process. My experience with the VIA board was that I had to disable LAN boot completely in the bios, not just have it as secondary boot device.
If you are going to set up a large number or servers with the same base installation, you'll quickly get tired of navigating the sysinstall menus. sysinstall can be scripted, this section show you how. By the end of the section, all you need to do is to plug in your new machine and drink coffee.
sysinstall(8) will first look for the file /install.cfg on the root file system which sets installation parameters. This file can also be used to script sysinstall. When the execution exits, sysinstall will return to the interactive mode.
Hence, to automate the installation we need to create install.cfg and place it in the root of the memory filesystem we created. The syntax, variables and commands are described in sysinstall(8), however not all variables and commands are documented, for theese, we need to take a look at the source code also. Also a sample install.cfg is given in the source code directory, /usr/src/usr.sbin/sysinstall.
One important thing is that /install.cfg is read and executed strictly top down, variables must be set before the function using them.
If anything fails in the script, sysinstall will abort the rest of the script and go interactive.
################################ # install.cfg for jumpstart of FreeBSD # # See sysinstall(8) for details about how to script the process # This file has been edited from /usr/src/usr.sbin/sysinstall/install.cfg # Turn on extra debugging. debug=YES nonInteractive=YES noConfirm=YES noWarn=NO ################################
We don't want any interactive questioning, the whole point is to power up the machine and drink coffee. But we do like to have some debug information for when we get back and everything has failed misserably.
################################ # Disk partitioning # # WARNING: This will format the disk and dedicate the entire disk to # FreeBSD disk=ad0 partition=all bootManager=none diskPartitionEditor ################################
This section selects a disk and creates the partitions. Multiple disks can be partitioned but you must end each disk defintion with diskPartitionEditor.
Note: The sample install.cfg provided with sysinstall source sets partition to exclusive, this will set your disk in "dangerously dedicated" mode which is not recommended unless you really need it.
################################ # Disk labeling # # All sizes are expressed in 512 byte blocks! # # For example: # / 512MB, swap 512MB, /usr 8192MB, /var 8192MB, /home remaining ad0s1-1=ufs 1048576 / ad0s1-2=swap 1048576 none ad0s1-3=ufs 16777216 /usr ad0s1-4=ufs 16777216 /var ad0s1-5=ufs 1048576 /tmp ad0s1-6=ufs 0 /home 1 diskLabelEditor ################################
This section slices up the chosen diskpartitions. The line labeling /home is special, this will use any remaining space, so it must be last.
################################ # Host specific configuration: tryDHCP=YES netDev=vr0 # The following optional if using dhcp to configure the network hostname=jumpstart domainname=example.com ################################
Apparently sysinstall suffers from amnesia when it comes to the network configuration. Even though the network was configured on boot, sysinstall will need a network configuration as above unless you are installing from a disk.
This should be a general configuration file and all host configuration should be done with dhcp. For unknown reason, hostname is required even if passed with dhcp.
################################ # Select release to install # releaseName 6.1-RELEASE ################################
This section defines the release and which collection of packages and sources to install. releaseName is by default determined by the release of the kernel and sysinstall will then look for a directory of that name when fetching the relase. If you cvsup'ed your sources before building the custom kernel, releaseName may be x.x-STABLE or x.x-CURRENT, but there are no such releases, so we need to specify it here.
################################ # Set distribution to install, either use distSetCustom and choose # individual components or choose a collection, for example # distSetKernDeveloper which installs base and kernel sources distSetKernDeveloper ################################
FreeBSD has some predefined distributions you can choose: distSetMinimum, distSetUser, distSetXUser, distSetKernDeveloper, distSetXKernDeveloper, distSetDeveloper, distSetXDeveloper and distSetEverything. You can also customize defining the variable dists and then use distSetCustom.
Note: Some have reported problem using the custom distribution, that the kernel is not correctly installed. A work around is to install the minimal distribution and commit it, then select custom and commit that.
################################ # Select installation method # # We want an FTP install, so we also need to specify the ftp server to # fetch from netDev=vr0 tryDHCP=YES _ftpPath=ftp://ftp.example.com/pub/FreeBSD mediaSetFTP ################################
Again (!?) we need to specify the interface we want to use and configure it using dhcp. For ftp installation we must specify a server and path to the selected release. You can always check the release announcement for a list of available ftp servers, if you do not set up your own. With the above settings, sysinstall will fetch the release from this directory:
ftp://ftp.example.com/pub/FreeBSD/releases/i386/6.1-RELEASE
That is, sysinstall assumes the release files are found in the subdirectory releases/<architecture>/<releaseName> in the ftp-path.
At this point, your old disk is still alive and happy, the next section will commit everything to the disk, and we have set noConfirm!
################################ # # OK, everything is set. Do it! installCommit ################################
You can customize further after install commit setting the variable package to any package you'd like installed and followed by the command packageAdd. At this point, when installation is finnished it will return to the well known sysinstall menu to allow you to add any further customization. To avoid this end the install.cfg with the command shutdown(8). It may be better though, if you compiled your memory disk with halt(8) to terminate the installation with system halt -p.
If everything works well, simply power up your system, is should be busy installing installing and you busy drinking coffee :-)
It is a bit tiresome to change the install.cfg if it is located on the memory disk. If you created a custom memory file system with support for the ftp(1) command, then could create a minimal install.cfg and then fetch one to be included like this:
################################ # Host specific configuration: tryDHCP=YES netDev=vr0 # The following optional if using dhcp to configure the network hostname=jumpstart domainname=example.com ################################ system /bin/ftp ftp://ftp.example.com/pub/FreeBSD/jumpstart.cfg loadConfig jumpstart.cfg
This should configure the network interface and fetch the installation configuration file (altough I have not tested it yet). Furter scripting is posible, and you can also change loader.conf to run init(8) instead and script everything - the limits are given by the commands available on the root file system.
This section covers how to set up a diskless environment. By the end of the section you should be able to boot up your diskless client with access to the FreeBSD base system.
The diskless client will fetch kernel and kernel modules at boot using tftp. Then nfs mounting the root file system. Each client must have separate var and tmp partitions as processes on different clients may otherwise overwrite files.
Everything will be installed in /var/diskless, the base system and user applications in FreeBSD, see the following section. For each host we create a directory, <hostname> with subdirectories, var and tmp.
Without the host specific directories, var and tmp will be created as memory file systems. This has the advantage that cleaning up the "disk" is done on reboot, but at the price of precious RAM.
In the following we will create a separate root file system. This has many advantages, in particular it is easy to move or copy to other servers, and you don't risk exporting files containing confidential server information onto your diskless clients.
The following builds the base system and installs into /var/diskless/FreeBSD:
# cd /usr/src # make buildworld # mkdir /var/diskless/FreeBSD # make DESTDIR=/var/diskless/FreeBSD installworld # make DESTDIR=/var/diskless/FreeBSD distribution
Note: On FreeBSD 5.X you will have to cd into etc before making the distribution, make DESTDIR=/var/diskless/FreeBSD distribution.
You can make a more lean installation by disabling unneeded features in /etc/make.conf.
The configuration of the client is done on the server. The first to do is do edit /var/diskless/FreeBSD/.cshrc and set set promt = "FreeBSD diskless # ". Now chroot(8) into /var/diskless/FreeBSD. The prompt should now change,
FreeBSD diskless #
indicating that you are in the diskless file system, this helps you avoid messing up your server configuration. In the following, paths are relative to the root in the chrooted environment.
We need to modify the following files:
/etc/fstab
/etc/syslog.conf
/etc/resolv.conf
/etc/rc.conf
Of course, this is a minimal list, you may wish to modify other files as well, for example to give users access to cdrom or usb devices.
The root file system is mounted at boot, but this is mounted read only. We need to mount /var, /tmp and /home. We may also want to mount a swap partition.
Swap over NFS is slow, and should be avoided if posible. Also, since each diskless client needs it's own swap partition, you need to create a large number of swap files on the server wasting a lot of disk space. However, if you have limited RAM swap may be needed.
All diskless clients will share the same fstab for the common file systems:
### Common fstab for diskless clients # Device Mount FStype Options Dump Pass nfs.example.com:/var/diskless/FreeBSD / nfs ro 0 0 nfs.example.com:/home /home nfs rw, userquota 0 0 proc /proc procfs rw 0 0
The dump frequency and pass numbers are set to 0 to avoid running fsck(8). fsck should never be run on nfs mounted file systems, if the partition becomes corrupt, run fsck on the server.
The /var and /tmp partitions have been excluded. This is because these cannot be shared among the diskless clients. By default, at startup if the system detects that /var and /tmp are read only, it will create these as memory file systems. Although the size can be limited this wastes precious memory.
If you compiled the kernel with the BOOTP options, hostname should be set on boot, and we can then create a script that will mount host specific partitions given the hostname.
The diskless client should not log locally, even if the log directory is nfs mounted. This will spread out logs for all the diskless clients and make it more difficult to rotate logs and clean up. In stead the diskless clients should log to a remote syslog server.
To log to a remote syslog server, replace the entries in /etc/syslog.conf with the following:
*.* @syslog.example.com
It may be useful to log critical errors on the console also.
/etc/resolv.conf won't be updated on boot since using dhclient would cause the client to loose the root file system, and it's mounted read only anyway. So update resolv.conf manually to reflect the correct network configuration.
The diskless client should not run any services, with the posible only exception of syslog. Certain services are enabled by default, even if only intended to run as a local service. These are all controled in /etc/rc.conf. Defaults are set in /etc/defaults/rc.conf, we need only change a few. A sample rc.conf may be as follows:
# rc.conf for diskless clients # # System daemons: # sendmail_submit_enable="NO" # Do not enable local delivery or outbound sendmail_outbound_enable="NO" # sendmail daemon. This disables sendmail sendmail_msp_queue_enable="NO" # completely on the client. cron_enable="NO" # Cron should run on the server # # File systems # root_rw_mount="NO" # Root is exported read only background_fsck="NO" # NEVER run fsck on nfs mounted partitions nfs_client_enable="YES" # The diskless client is an NFS client tmpmfs="NO" # We mount /var and /tmp as NFS partitions varmfs="NO" # cleanvar_enable="YES" # Clean the /var directory (this is default) clear_tmp_enable="YES" # Clear /tmp at startup. clear_tmp_X="YES" # Clear and recreate X11-related directories # # Scripts run at startup only # newsyslog_enable="NO" # Logging to server update_motd="NO" # Root file system is read only
On a standalone machine, sendmail(8) is only used by cron to send status reports to root. Cron should run on the server, not on the client so sendmail can be disabled along with cron.
At boot, by default the root file system is first mounted read-only, then remounted read-write when switching into multiuser mode. The root file system is exported read-only, so we disable the remounting. You should never run fsck on an nfs mounted disk, if there are disk errors, run fsck on the server. Enabling the nfs client merely checks that nfs is supported by the kernel and if not loads the nfsclient kernel module. Strictly this line is not needed.
The default value for tmpmfs and varmfs is AUTO. The script then checks if these directories are writeable, if not memory file systems are created. To avoid this, we disable the script. By default /var is cleaned up on boot as it should. Then the server need not clean up after the clients. This is not the default for /tmp though, so we enable clearing /tmp.
Since syslog logs to the server, there is no need to run newsyslog on startup. This would only have the effect of rotating log files. Also, we cannot update the "message of the day" file, since the root is mounted read-only.
Of course, the above sample rc.conf only lists the settings required for diskless operation. You must add any other customization as needed, you posibly want to configure the console.
The previous section assumed the same configuration for all diskless clients. But there are some things that will differ, separate /var and /tmp partitions have been mentioned, but you may also be in the situation where there are differences in hardware or some clients are used by a single user with specific requirements. In this section we solve this problem.
At this point, you should be able to boot up the diskless client but there is still tons of stuff that remains to be tweaked to make the system useful. Among other things, we haven't considered how to manage users. The next section covers management and finetuning.
While we can boot now, there are still a lot of things that remains to be configured or tweaked to work properly. This section discusses some of these. This section will probably be incomplete forever, some things are not included because they are not any particular to diskless clients.
For a start, you might want to copy the password and group files from the server, however, managing users becomes significantly easier if you use LDAP or NIS to make user account information accessible from all hosts.
NIS is parcially supported in the FreeBSD base system, but LDAP is recommended as it easily integrates with other network services and other systems. This section will document an LDAP solution.
To support LDAP you need to install the following packages:
Obviously you also need to setup an LDAP server. You need to install net/openldap22-server. Setting up the LDAP server is beyond the scope of this document. Creating an LDAP directory and configuring clients to authenticate against this will not be covered at this moment.
Since we installed the full system in a separate directory, maintaining it is as simple as chroot'ing into that directory and update/install as on a normal system, both for installing third party applications from ports and for updating the base. This will not interfere with the server applications.
One thing to be aware of, when we set up the server, we installed kernel and modules in /var/tftp/boot on the server. This path is not within the diskless base directory. To solve this, you can change the configuration of the tftp server such that the kernel is loaded from /var/diskless/FreeBSD/boot or such that kernel is loaded via nfs.
For some ports to build correctly, you may need access to the device file system within the chroot'ed environment. To get this, before chrooting, run the command:
# mount -t devfs devfs /var/diskless/FreeBSD/dev
The first thing you want to install is cvsup and then update the ports tree.
FreeBSD Jumpstart Guide, Alfred Perlstein
Diskless X Server: a how to guide, Jerry Kendall
Pluggable Authentication Modules, Dag-Erling Smørgrav
MAN-pages: diskless(8) pxeboot(8)
This, and other documents, can be downloaded from ftp://ftp.FreeBSD.org/pub/FreeBSD/doc/.
For questions about FreeBSD, read the documentation before contacting <questions@FreeBSD.org>.
For questions about this documentation, e-mail <doc@FreeBSD.org>.