Difference between revisions of "Whole disk ZFS Install Guide"

(Installing the ZFS userspace tools)
(Add zfs parameters to grub config)
Line 317: Line 317:
==== Add zfs parameters to grub config ====
==== Add zfs parameters to grub config ====
Now we need to tell grub to use our whole disk zfs during boot time. That is done in two steps.  
Now we need to tell grub to use our whole disk zfs during boot time. That is done in two steps.  
First we edit <code>/etc/default/boot</code>
First we edit <code>/etc/default/grub</code>
{{file|name=/etc/default/boot|desc= |body=
{{file|name=/etc/default/grub|desc= |body=
GRUB_CMDLINE_LINUX="boot=zfs rpool=tank bootfs=tank/funtoo/ROOT"
GRUB_CMDLINE_LINUX="boot=zfs rpool=tank bootfs=tank/funtoo/ROOT"
Line 326: Line 326:
# ##i##grub-mkconfig -o /boot/grub/grub.cfg
# ##i##grub-mkconfig -o /boot/grub/grub.cfg
==== Finally install the grub bootloader ====
==== Finally install the grub bootloader ====
As last step, we need to install the grub bootloader properly:
As last step, we need to install the grub bootloader properly:

Revision as of 05:43, September 14, 2015

0. Introduction

ZFS seams to be one of the future filesystems. At the time of writing (31.08.2015) booting on ZFS is still a headache. Some linux distributions boot on ext2 partition and run zfs as rootfs like ZFS_Install_Guide and some are able to boot from zfs, some do support zfs only as a filesystem.

Funtoo linux supports ZFS as filesystem and as rootfs, however it does not support it as boot/grub filesystem. For easier search by search engines, it is referred to in this document as whole disk ZFS.

Funtoo linux uses the grub bootloader. So it has been not really understandable, why whole disk ZFS could not be supported by funtoo, as grub is able to do so. Or better, some linux distribution, using the grub bootloader run "whole disk ZFS". What they can, ... yes, we can! ... as well.

This guide is based on many trial and errors - caused by lack of knowledge - the actual ZFS_Install_Guide and a guide for whole disk ZFS for ubuntu. Ubuntu runs als grub bootloader, so some ideas are adapted from there.



This guide is a working pretty well on one computer - that is mine! If it does not run on your's, or if it breaks on your's, than you should try to sort the issue out and report it to this page.

So, you may expect that it MAY work! ... and you should be aware that it MAY break your installation. This guide is not developed enough to ENSURE a stable production environment.

... however, I use it for that! ;-) - crazzy as I am.


ZFS will run properly only on 64Bit machines. If you plan to run ZFS with 32Bit, you may also try russian roulette with six bullets. The outcome is clear, and sure not that what you want!

Not covered in this Guide are:

- obvious steps from the regular funtoo installation guide Install and the ZFS installation guide ZFS_Install_Guide

- other partition layouts than GPT

- other kernel sources than gentoo-sources as debian-sources do not work for sure

- usb bootable whole disk ZFS devices (as they need a proper set of udev rules)

- and may be many items more ... ;-)

1. Preparations

In this section, we will prepare everything, to be used during system and boot loader installation.

Create an installation envirnment

To be able to install funtoo on zfs, we need a suitable installation environment. The next steps describe the setup:

Downloading the ISO (With ZFS)

This is a copy by the date of 31.08.2015 from the ZFS_Install_Guide.

In order for us to install Funtoo on ZFS, you will need an environment that already provides the ZFS tools. Therefore we will download a customized version of System Rescue CD with ZFS included.

Name: sysresccd-4.2.0_zfs_0.6.2.iso  (545 MB)
Release Date: 2014-02-25
md5sum 01f4e6929247d54db77ab7be4d156d85
Download System Rescue CD with ZFS

Creating a bootable USB from ISO (From a Linux Environment)

This is a copy by the date of 31.08.2015 from the ZFS_Install_Guide.

After you download the iso, you can do the following steps to create a bootable USB:

Make a temporary directory
# mkdir /tmp/loop

Mount the iso
# mount -o ro,loop /root/sysresccd-4.2.0_zfs_0.6.2.iso /tmp/loop

Run the usb installer
# /tmp/loop/usb_inst.sh

That should be all you need to do to get your flash drive working.

Booting the ISO

While booting the above mentioned iso image - you should use exactly that one, unless you are writing another guide - you shoud use the option E) Alternative 64bit kernel (altker64) with more choice.... 64Bit, as mentioned above.

On the next page you should select 7. SystemRecsueCd with the default graphic environment. The system will come up with a small graphical environment and some tools. One of them is a graphical version of GParted, which will be used in this guide.

Preparing the harddrives

The preparation of the harddrives envolves the following steps:

Cleaning the disks

This guide was developed using GPT partition layout. Any other layout type is not included here.

All below this note is a copy by the date of 31.08.2015 from the respective section of the ZFS_Install_Guide.

First lets make sure that the disk is completely wiped from any previous disk labels and partitions. We will also assume that /dev/sda is the target drive.

# sgdisk -Z /dev/sda

This is a destructive operation and the program will not ask you for confirmation! Make sure you really don't want anything on this disk.


Do not forget to run this command for each drive participating the new pool. In this guide a mirror pool is created and the devices /dev/sda and /dev/sdb are used.

Create a pool

In this section, we create a whole disk ZFS / zpool. For the convenience of this guide, a mirror pool with the name tank will be created and not mounted (option -m none). Single disk ZFS or even zraid could be used respectively.

The option ashift=12 is set for hard drives with a 4096 sector size commonly used by SSD drives.

#   zpool create -f -o ashift=12 -m none -R /mnt/funtoo tank mirror /dev/sda /dev/sdb 

Here we use the devices /dev/sda and /dev/sdb, and not the partitions as described in the ZFS_Install_Guide. The devices will be partitioned with a GPT partition label and the respective pool including the ZFS filesystems will be created in one step.

The cachefile is omitted, as it only speeds booting and unfortunately creates destructive issues while manipulating the pool later on.


It is preferred to use /dev/disk/by-id/ ... Here it does not really matter, and at the important points we will use the reference to the drives by ID.

zpool status can be used to verify the pool.

Create the zfs datasets

This is a copy by the date of 31.08.2015 from the ZFS_Install_Guide, unless the root dataset has been changed to ROOT to avoid confusions with the normal linux root home directory /root.

We will now create some datasets. For this installation, we will create a small but future proof amount of datasets. We will have a dataset for the OS (/), and your swap. We will also show you how to create some optional datasets: /home, /usr/src, and /usr/portage. Notice, datasets are examples only and not strictly required.

Create some empty containers for organization purposes, and make the dataset that will hold /
#  zfs create -p tank/funtoo
#  zfs create -o mountpoint=/ tank/funtoo/ROOT

Optional, but recommended datasets: /home
#  zfs create -o mountpoint=/home tank/funtoo/home

Optional datasets: /usr/src, /usr/portage/{distfiles,packages}
#  zfs create -o mountpoint=/usr/src tank/funtoo/src
#  zfs create -o mountpoint=/usr/portage -o compression=off tank/funtoo/portage
#  zfs create -o mountpoint=/usr/portage/distfiles tank/funtoo/portage/distfiles
#  zfs create -o mountpoint=/usr/portage/packages tank/funtoo/portage/packages

df -k can be used to verify the mountpoints.

Create empty snapshot of the pool (purely optional)

Expecting some trial and error during the system installation, a snapshot of the empty disks makes it easy to return this point ... of the game ...

#   zfs snapshot -r tank@000-empty_pool  

zfs list -t snapshot allows to list the existing snapshots. The rollback will be descibed further down in the troubleshooting guide...

Flag the bios_grub partition

The freshly created GPT partition tables of the devices used in the pool contains two partitions. The first and larger one contains the zfs filesystem, the second (number 9), 8.0MiB in size, is reserved by Solaris (BF07).

We use now the graphical tool GParted selecting the partition 9 of each device and set the bios_grub flag. This action will add the flag and changes the partition type to BIOS boot partition (EF02).

This can be done by command line as well ...

2. Basic Funtoo installation

Now the installation environment has been set up, and the harddrives laid out. The next step will be the basic funtoo installation according to the following steps:

Import the pool

For installation purpose, the pool created will be imported under the mountpoint /mnt/funtoo. ashift=12 option which is used for a hard drives with a 4096 sector size.

#   zpool import -f -d /dev/disk/by-id -R /mnt/funtoo tank  

Using zpool status and / or df -k can be used to verify the pool and the mountpoint.

Change directory to the pool's root directory

Then we should change into the pool's mountpoint directory. At the same time this is the root directory of the intended funtoo installation. That is the directory where you will chroot into.

# cd /mnt/funtoo

Download and extract funtoo stage3

It is strongly recommended to use the generic_64 funtoo stage 3 tarball, to avoid a broken build. The download should be done using wget:

# wget http://build.funtoo.org/funtoo-current/x86-64bit/generic_64/stage3-latest.tar.xz

A snapshot is recommended to allow the rollback to this stage.

The tarball should be extracted using the following command:

# tar xpf stage3-latest.tar.xz

Omitting the option p will result in a broken system!


More information could be found under Now download and extract the Funtoo stage3 ...

Bind the kernel related directories

# mount -t proc none proc
# mount --rbind /dev dev
# mount --rbind /sys sys

Copy network settings

# cp -f /etc/resolv.conf etc


zpool.cache is omitted as described above.

Chroot into Funtoo

# env -i HOME=/root TERM=$TERM chroot . bash -l

Using the graphical installation environment as described above, allows now to open several windows and changing the root into the chroot environment usind the following steps:

# cd /mnt/funtoo
# env -i HOME=/root TERM=$TERM chroot . bash -l

This allows you to prepare allready the next steps, while the previous is still busy.

Setup portage

Now you should think a little about how to set up MAKEOPTS, LINGUAS and VIDEO_CARDS in your portage make.conf file.

Setup /etc/fstab

# <fs>                  <mountpoint>    <type>          <opts>          <dump/pass>

tank/funtoo/ROOT               /           zfs            noatime        0 0

Setup /etc/mtab

zfs and zpool will need it:

# grep -v rootfs /proc/mounts > /etc/mtab

Emerge portage tree

Before we install zfs and the bootchain, we should update our portage tree using:

# emerge --sync

Update the system

Now we will update the system using the following command:

# emerge --uaDN @world

Here consider to make a snapshot, so we could rollback to a clean and updated system... This must be done outside the chroot environment, because here we still have no zfs / zpool installation.

Emerge genkernel

genkernel is the tool used by gentoo and funtoo to build kernel and initramfs from sources. it supports zfs for the gentoo-sources as mentioned above.

# emerge genkernel

Generate initial (unbootable) kernel

We need to build one first kernel. The zfs needs to find a valid kernel to configure itself properly.

# genkernel kernel

No options are needed... And once the genkernel starts to build the kernel modules, the build may be interrupted with ctrl c. We have, what we need, and the system is still none-bootable.


As we have done now some cpu consuming tasks, consider to make a snapshot, so we could rollback to a clean and updated system... This must be done outside the chroot environment, because here we still have no zfs / zpool installation.

The debian-sources

From mid 2015 on, funtoo comes with a debian kernel and initramfs preinstalled. At the same time it is configured for debian-sources. Basically the debian distribution has a much better automatic hardware detection and configuration support. That is the advantage. The disadvantage is, we will have to recompile it, and it will take a long time! So first of all, we may delete the preinstalled kernel and initramfs:

# rm /boot*

3. Add ZFS support

Now we will add the ZFS support to the basic funtoo system:

The debian-sources

Using the debian-sources we have to ensure that the following kernel configuration is set:



Installing the ZFS userspace tools

This is a nearly indetical copy by the date of 31.08.2015 from the ZFS_Install_Guide. Emerge sys-fs/zfs (package not on wiki - please add). This package will bring in sys-kernel/spl (package not on wiki - please add), and sys-fs/zfs-kmod (package not on wiki - please add) as its dependencies:

# emerge zfs

Check to make sure that the zfs tools are working.

# zpool status
# zfs list
# zfs list -t snapshot

If /etc/mtab is missing, these two commands will complaine. In that case solve that with:

# grep -v rootfs /proc/mounts > /etc/mtab

Add the zfs tools to openrc.

# rc-update add zfs-import boot
# rc-update add zfs-mount boot
# rc-update add zfs-share default
# rc-update add zfs-zed default

A snapshot is recommended to allow the rollback to this stage. Now you can and should make the snapshot within the chroot environment.

Installing the ZFS initramfs, kernel and kernel modules

Build now kernel and initramfs and kernel modules:

# genkernel all --zfs --no-clean --callback="emerge @module-rebuild"
It looks like the following is working better module-rebuild is fetching debian-sources as well, and overwrites own configuration..
# genkernel all --zfs --no-clean --callback="emerge spl zfs-kmod zfs"

To force to get a specific kernek configuration, you may use:

# genkernel all --zfs --no-clean --kerneldir=/usr/src/linux -- kernel-config=/usr/src/<path_to_config> --callback="emerge spl zfs-kmod zfs"

The result should be the same.

Emerge, install und configure grub bootloader

This section now describes how to setup our boot configuration to boot from and into the whole disk ZFS.

Emerge grub

To emerge grub, we need to set some permanent USE flags first:

# echo "sys-boot/grub libzfs truetype" >> /etc/portage/package.use
# emerge grub

Be aware that it is not -truetype but truetype. We will need some fonts to display the grub menu...

Check grub zfs support

The zfs support of grub may be checked by invoking grub-probe:

# grub-probe .

it should return zfs for a directory on the zfs filesystem.

Install grub into /boot/grub directory

The command grub-install is basically a script taking care off all installation issues. Presently we tweak it a little to do what we want. In the first step, it will be invoked as usual with standard parameters for only one pool device.

# grub-install /dev/sda

That will create the /boot/grub directory and installs grub into it. It also installs the bootloader into the BIOS grub partition, which will be overwritten later on.

Add zfs parameters to grub config

Now we need to tell grub to use our whole disk zfs during boot time. That is done in two steps. First we edit /etc/default/grub

GRUB_CMDLINE_LINUX="boot=zfs rpool=tank bootfs=tank/funtoo/ROOT"

Thereafter we invoke grub-mkconfig to generate a new grub config file.

# grub-mkconfig -o /boot/grub/grub.cfg

Finally install the grub bootloader

As last step, we need to install the grub bootloader properly:

# grub-install $(readlink -f /dev/disk/by-id/...disk1)
# grub-install $(readlink -f /dev/disk/by-id/...disk2)

If all went through, without any error report, you have installed now a mirrored whole disk zpool, which may boot from any of the two disks. In any failure of one of the two disks, the system may be booted and recovered easily from the other one.

4. Final steps & cleanup

The final steps and the cleanup can be done according ZFS_Install_Guide. Further installation should be guided by Install and Funtoo_Linux_First_Steps.


enjoy it!