Difference between pages "Cloud Backup" and "Creating System Rescue CD ZFS Modules"

From Funtoo
(Difference between pages)
Jump to: navigation, search
 
(Build the kernel)
 
Line 1: Line 1:
[[Category:HOWTO]]
+
This HOWTO will show you how to create your own zfs srm or download the prebuilt zfs srm module from my website.
  
This howto will describe a method for automatically backing up your funtoo install to the internet, in this case dropbox, but any online storage will do. Gentoo describes a method of creating a stage 4 archive. The problem with a stage 4 is that it is large and it archives a lot of unnecessary files. Such as applications that can be reinstalled from an emerge world. Instead, this method will aim for more of a "stage 3.5."
+
== Compiling a compatible kernel ==
  
{{fancynote| This method does not attempt to backup everything. The intention is only to backup the system. Optionally you can also archive and copy your <tt>/home</tt> folder if you have enough online storage.}}
+
The first thing you need to do is decide for which version of System Rescue CD you will be building for. Each System Rescue CD version has a different set of kernels. Specifically each version has a Stable kernel and an Alternate Kernel. The stable kernels get minor version increments every release (Sometimes major ones but usually doesn't happen too quickly), the Alternate kernels tend to move up much faster. For example System Rescue CD had the following stable/alternate kernels:
  
== Use Case ==
+
{| class="wikitable"
A backup machine currently provides network drives on a home LAN to allow clients on the LAN to backup to, using apps such as Time Machine (Mac) and Genie Timeline (Windows). As this machine ''is'' the backup machine it doesn't have anywhere to backup to itself. In this situation a backup solution is provided by backing up to somewhere online - dropbox. If a restore from the backup is required, the client machine's backups would be trashed, and the backup machine restored.
+
|-
 +
! Version !! Stable !! Alternate
 +
|-
 +
| 3.7.0 || 3.4.47 || 3.9.4
 +
|-
 +
| 3.5.0 || 3.4.37 || 3.8.4
 +
|}
 +
 
 +
===Download kernel and patches===
 +
Once you decide which version you want to use, you can go to http://kernel.sysresccd.org/ and pick the version you want. After you do this, download all the stuff in that directory and put them in their own folder (a work directory). The files that are named std-sources are the patches for the regular stable kernel, the patches named alt-sources are the patches for the alternate kernel. Once you have the patches, go to kernel.org and also download the initial release of the kernel sources you want. For example, if you want to build an srm for 3.4.2 which uses the 3.4.35 kernel, just download the 3.4 (3.4.0) kernel. You just need to download the initial release because one of the patches you downloaded earlier is the patch to update the 3.4.0 to 3.4.35.
 +
 
 +
Once you download the kernel you want, let's extract and patch it.
  
== Automatic Backup Archives With Etckeeper ==
 
Etckeeper is a tool that is used to save versions of <tt>/etc</tt>, including meta-data in a version control repository such as git.
 
As etckeeper is not in the funtoo portage tree, layman is used to provide an overlay.
 
=== Install etckeeper via layman ===
 
Before you install layman it is worth mentioning that you probably want <tt>USE="git subversion"</tt> in <tt>/etc/portage/make.conf</tt>. After adjusting use flags, to install layman you run:
 
 
<console>
 
<console>
###i## emerge layman
+
# ##i##tar xf linux-3.4.tar.bz2
 +
# ##i##mv linux-3.4 linux-3.4.35-std342-amd64
 +
# ##i##bzcat std-sources-3.4-01-stable-3.4.35.patch.bz2 | patch -p1 -d linux-3.4.35-std342-amd64
 +
# ##i##bzcat std-sources-3.4-02-fc16.patch.bz2 | patch -p1 -d linux-3.4.35-std342-amd64
 +
# ##i##bzcat std-sources-3.4-03-aufs.patch.bz2 | patch -p1 -d linux-3.4.35-std342-amd64
 
</console>
 
</console>
In order to backup the layman configuration, but not the portage overlay trees, make the following modifications to the default install.
 
Tell Portage about layman-fetched repositories by adding the following line to <tt>/etc/portage/make.conf</tt>:
 
  
<pre>
+
===Retrieve kernel configuration===
source /etc/layman/make.conf
+
Once that is complete, you will need to get the kernel config specifically for the kernel and architecture you want. The easiest way I found to do this was to download the System Rescue CD you want, and then boot it inside a VM. Once you are inside the VM, go to the /root directory, and it will be a file named kernel-<version>.conf. Copy this file over SSH and put it in a safe location. Also copy it over into the linux kernel directory and rename it to .config.
</pre>
+
  
Modify the following lines in <tt>/etc/layman/layman.cfg</tt>:
+
===Build the kernel===
 +
Once you copy the kernel config over, do a make menuconfig to begin. You will only need to change two settings inside the config, add the proper kernel name, and remove the old initramfs source files directory. This directory doesn't exist locally. This is a remnant of upstream's build process.
  
<pre>
+
Follow the below tree structure, and make it equal to what is below:
storage  : /var/lib/layman
+
{{kernelop
installed : /etc/layman/installed.xml
+
|title=
make_conf : /etc/layman/make.conf
+
|desc=
</pre>
+
(-std342-amd64) Local version - append to kernel release
 +
      [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
 +
      ( ) Initramfs source file(s)
 +
}}
  
Add the bgo-overlay. As described on their web page, [http://bgo.zugaina.org/ bgo.zugaina.org].
+
The Local version above will end up making the kernel modules directory be 3.4.35-std342-amd64. The reason the kernel modules directory has to be equal to this is because our directory will be merged into the System Rescue CD later on, so the names need to be exactly the same so that the System Rescue CD system can find our kernel modules.
<console>
+
###i## layman -o http://gpo.zugaina.org/lst/gpo-repositories.xml -L
+
###i## layman -a bgo-overlay -o http://gpo.zugaina.org/lst/gpo-repositories.xml
+
</console>
+
  
More information about layman can be found here: http://www.gentoo.org/proj/en/overlays/userguide.xml
+
If you want to build an srm (which you do), you also need to change the following inside the kernel config so that squashfs-tools can work later on:
 +
{{kernelop
 +
|title=
 +
|desc=
 +
Miscellaneous Filesystems ->
 +
      <*>  SquashFS 4.0 - Squashed file system support
 +
      [*]    Squashfs XATTR support
 +
      [*]    Include support for ZLIB compressed file systems
 +
      [*]    Include support for LZO compressed file systems
 +
      [*]    Include support for XZ compressed file systems
 +
      [ ]    Use 4K device block size?
 +
      [ ]    Additional option for memory-constrained systems
 +
}}
  
Then unmask and install etckeeper.
+
Now save and exit, and continue to build your kernel.
<console>
+
###i## emerge etckeeper --autounmask-write
+
###i## emerge etckeeper
+
</console>
+
  
{{fancynote| To update layman overlays do:}}
 
 
<console>
 
<console>
###i## layman -S
+
# ##i##make bzImage modules
 +
# ##i##make modules_install
 
</console>
 
</console>
  
 +
{{fancynote|If you have multiple cores, do a make -j<# of processors+1> to build faster. Example: '''make -j9''' for an 8 core machine.}}
 +
 +
=== Using the portage tree ===
 +
Inside the portage tree, there are ebuilds already prepared that will download the kernel, patch it, and perform any required substitutions. Afterwards you literally just go into your /usr/src/<kernel> directory, and run make. To see the available versions, run the following:
  
If you see the following error -- apply this fix:
 
 
<console>
 
<console>
###i## emerge etckeeper
+
Versions available for standard kernel
Calculating dependencies... done!
+
# ##i##equery y std-sources
>>> Verifying ebuild manifests
+
!!! A file is not listed in the Manifest: '/var/lib/layman/bgo-overlay/sys-apps/etckeeper/files/etckeeper-gentoo-0.58.patch'
+
  
###i## cd /var/lib/layman/bgo-overlay/sys-apps/etckeeper
+
[I]3.4.37-r1 | o ~ o o o o o o o o o o o | o 3.4.37 | gentoo
###i## ebuild etckeeper-0.58-r2.ebuild manifest
+
 
###i## emerge etckeeper
+
Versions available for alternate kernel
 +
# ##i##equery y alt-sources
 +
 
 +
[I]3.8.4 | o ~ o o o o o o o o o o o | o 3.8.4 | gentoo
 
</console>
 
</console>
  
== Configure etckeeper ==
+
I haven't written the ebuilds in a way where it's easy to tell (at first glance) what System Rescue CD version you need. However, you can check the ebuild and you will see the version of the CD inside there. You can also check the chart above.
Move any config files that do not live in <tt>/etc</tt>. i.e.
+
Check <tt>/root</tt> for any files to be archive, such as iptables scripts and move them to <tt>/etc</tt>.
+
  
{{fancynote| because funtoo uses [[Boot-Update]], this means <tt>/boot/grub/grub.cfg</tt> does not need to be archived.}}
+
If you wanted to install the standard kernel sources, you just emerge it like any other application:
  
To ensure your portage world file is archived, make the following link:
 
 
<console>
 
<console>
###i## ln /var/lib/portage/world /etc/world
+
# ##i##emerge std-sources
 
</console>
 
</console>
  
Initialise the git repository.
+
Once the kernel and the kernel module are installed, you need to make the SRM.
<console>
+
###i## etckeeper init
+
Initialized empty Git repository in /etc/.git/
+
###i## etckeeper commit "Initial commit."
+
</console>
+
  
If you don't already have cron installed, emerge it now.
+
== Creating the SRM ==
<console>
+
It's time to gather the required files and pack them together in order for it to become an SRM. An SRM is nothing more than a directory that has been compressed with squashfs. You can emerge the "Bliss Initramfs Creator" which has a feature that will automagically build the srm for you.
###i## emerge vixie-cron
+
</console>
+
And write the cron job to save an hourly version of <tt>/etc</tt>.
+
  
{{fancynote| git will only create a new version (commit) if there are changes from the previous one.}}
 
Edit the file <tt>/etc/cron.hourly/etckeeper:
 
  
<pre>
+
You also need to have spl, zfs, and zfs-kmod installed on your system before you try to make the srm. Emerging '''"bliss-initramfs"''' should automatically pull those packages. If it doesn't, make sure you install them.
#! /bin/bash
+
etckeeper commit "Hourly auto-commit"
+
</pre>
+
  
== Encrypt and copy backups online ==
 
=== Copy To Dropbox ===
 
 
<console>
 
<console>
###i## emerge dropbox
+
# ##i##echo "sys-kernel/bliss-initramfs srm" >> /etc/portage/package.use
 +
# ##i##emerge bliss-initramfs
 
</console>
 
</console>
  
Add a dropbox user:
+
 
 +
You should now have the following directory layout inside the '''/opt/bliss-initramfs''' folder:
 +
 
 
<console>
 
<console>
###i## useradd dropbox
+
octopus bliss-initramfs # ls -l
 +
total 117
 +
-rw-r--r-- 1 root root 6718 May 23 18:05 CHANGES
 +
-rw-r--r-- 1 root root  176 May 23 18:05 CREDITS
 +
-rw-r--r-- 1 root root  431 May 23 18:05 HELP
 +
-rw-r--r-- 1 root root 1397 May 23 18:05 LICENSE
 +
-rw-r--r-- 1 root root 1852 May 23 18:05 README
 +
-rw-r--r-- 1 root root 3194 May 23 18:05 USAGE
 +
-rwxr-xr-x 1 root root 2891 May 23 18:05 createInit
 +
drwxr-xr-x 3 root root    4 May 23 18:11 files
 +
drwxr-xr-x 3 root root    6 May 23 18:11 hooks
 +
drwxr-xr-x 2 root root    5 May 23 22:01 resources
 
</console>
 
</console>
  
Write the dropbox init files in <tt>/etc/conf.d/dropbox</tt>:
 
  
<pre>
+
Then run the '''createInit''' script and follow the instructions for the kernel you want to make the srm for (In this case it's '''3.4.37-std350-amd64'''):
DROPBOX_USERS="dropbox"
+
</pre>
+
<br>
+
<pre>
+
#!/sbin/runscript
+
# Copyright 1999-2004 Gentoo Foundation
+
# Distributed under the terms of the GNU General Public License, v2 or later
+
# $Header: /var/cvsroot/gentoo-x86/sys-fs/dropbox/files/dropbox.init-1.0,v 1.4 2007/04/04 13:35:25 cardoe Exp $
+
  
NICENESS=5
+
<console>
 +
-----------------------------------
 +
| Bliss Initramfs Creator - v1.8.1
 +
| Author: Jonathan Vasquez <jvasquez1011@gmail.com>
 +
| Distributed under the Simplified BSD License
 +
-----------------------------------
  
depend() {
+
>>> Which initramfs would you like to generate:
    need localmount net
+
>> 1. ZFS
    after bootmisc
+
>> 2. Encrypted ZFS (LUKS + ZFS)
}
+
>> 3. More Options
 +
>> 4. Exit Program
  
start() {
+
>>> Current choice [1]: ##i##3 ↵
    ebegin "Starting dropbox..."
+
    for dbuser in $DROPBOX_USERS; do
+
        start-stop-daemon -S -b -m --pidfile /var/run/dropbox-$dbuser.pid  -N $NICENESS -u $dbuser -v -e HOME="/home/$dbuser" -x /opt/dropbox/dropboxd
+
    done
+
    eend $?
+
}
+
  
stop() {
+
>>> More Options:
    ebegin "Stopping dropbox..."
+
    for dbuser in $DROPBOX_USERS; do
+
        start-stop-daemon --stop --pidfile /var/run/dropbox-$dbuser.pid
+
    done
+
    eend $?
+
}
+
  
status() {
+
>> 1. ZFS - System Rescue Module
    for dbuser in $DROPBOX_USERS; do
+
>> 2. Back
        if [ -e /var/run/dropbox-$dbuser.pid ] ; then
+
>> 3. Exit Program
            echo "dropboxd for USER $dbuser: running."
+
        else
+
            echo "dropboxd for USER $dbuser: not running."
+
        fi
+
    done
+
    eend $?
+
}
+
</pre>
+
Start dropbox now and at boot time:
+
<console>
+
###i## chmod 0755 /etc/init.d/dropbox
+
###i## /etc/init.d/dropbox start
+
###i## rc-update add dropbox default
+
</console>
+
  
After starting the dropbox daemon, it will provide a http link. You will need to visit this site just once to associate your computer with your dropbox account.
+
>>> Current choice [1]: ##i##↵
  
Write the cron job to make the backup archive and move it online. Edit the file <tt>/etc/cron.daily/backup</tt>:
+
>>> Creating a ZFS System Rescue Module!
  
<pre>
+
>>> Do you want to use the current kernel: 3.8.13-ALL? [Y/n]: ##i##n ↵
#! /bin/bash
+
cd /etc
+
git bundle create /tmp/backup.bundle --all
+
cd /tmp
+
mv -v -f backup.bundle /home/dropbox/Dropbox/Private/
+
</pre>
+
  
Make the script executable:
+
>>> Please enter the kernel name: ##i##3.4.37-std350-amd64 ↵
<console>
+
 
###i## chmod +x /etc/cron.daily/backup
+
>>> Detected 64 bit platform
 +
>>> Checking to see if modules directory exists for 3.4.37-std350-amd64...
 +
>>> Creating SRMs for 3.4.37-std350-amd64...
 +
>>> Creating directory structure for initramfs...
 +
>>> Checking preliminary binaries...
 +
>>> Checking binaries...
 +
>>> Using ZFS
 +
>>> Checking modules...
 +
>>> Copying binaries...
 +
>>> Copying modules...
 +
>>> Copying documentation...
 +
>>> Copying udev rules...
 +
>>> Compressing kernel modules...
 +
>>> Getting dependencies...
 +
>>> Copying dependencies...
 +
>>> Configuring files...
 +
>>> Creating and Packing SRMs...
 +
>>> Complete :)
 
</console>
 
</console>
  
=== Encrypt Backups ===
+
 
It is a good idea to encrypt your backup before moving it online. This can be done with gpg, using a symmetric (password only) or public/private key encryption. Additionally you can chose to sign the backup to check its integrity before restoring.
+
Now you should have two new files:
 +
 
 
<console>
 
<console>
###i## emerge gpg
+
-rw-r--r-- 1 root root 2068480 May 24 01:05 zfs-core-3.4.37-std350-amd64.srm
 +
-rw-r--r-- 1 root root  483328 May 24 01:05 zfs-kmod-3.4.37-std350-amd64.srm
 
</console>
 
</console>
  
==== Symmetric Encryption ====
 
There is no preparation required to use a symmetric key as all that is required is simply a passphrase.  Just modify the cron job. Edit <tt>/etc/cron.daily/backup</tt>:
 
  
<pre>
+
Now all you need to do is put those two files in the root of your USB directory.
#! /bin/bash
+
cd /etc
+
git bundle create /tmp/backup.bundle --all
+
cd /tmp
+
echo 'encryption_password' | gpg -o backup.gpg --batch --homedir /root/.gnupg -vvv  --passphrase-fd 0 --yes -c backup.bundle
+
mv -v -f router.gpg /home/dropbox/Dropbox/Private/
+
</pre>
+
{{fancyimportant| Remember to change "encryption_password"}}
+
  
{{fancywarning| If you forget this password the backup will be unusable. Lose the password and you lose the backup.}}
+
If you are making srms for both the standard and alternate kernels, you will end up with two zfs-core files and two zfs-kmod files (1 set for each kernel). You don't need to put the zfs-core that it makes for one of the kernels. The zfs-core srm only has zfs program binaries, man pages, udev rules, and a few other things. The zfs-kmod is the srm that has the kernel modules (and only the kernel modules). So you can easily just put two zfs-kmods (one for each kernel version) and just re-use the same zfs-core for both of them.
  
As there is now sensitive information in this file, you might want to remove read permission:
+
== Using the prebuilt srm ==
<console>
+
 
###i## chmod og-r /etc/cron.daily/backup
+
=== Download the SRM ===
</console>
+
If you didn't build your own srm and want to use the prebuilt one, just emerge '''"zfs-srm"''' from the tree:
  
==== Private/Public key Encryption ====
 
Make a private/public encryption/decryptions key pair. The public key will be used to encrypt and the private key to decrypt.
 
 
<console>
 
<console>
###i## gpg --gen-key
+
# ##i##emerge zfs-srm
 
</console>
 
</console>
The public key is used to create the encrypted backup and needs to live on the computer being backed up. A copy of the private key needs to be made and stored securely in another place. If this machine becomes unbootable, and this is the only place the private key lives, the backup dies with it.
 
The private key should not be kept:
 
# In the same place as the back up
 
# On the machine being backed up
 
{{fancynote| The private key is the only key that will decrypt the backup. Lose this key and/or it's password and you lose the backup.}}
 
  
List the private keys:
+
You can check available versions just as you did above for the kernels:
<console>
+
###i## gpg -K
+
/root/.gnupg/secring.gpg
+
------------------------
+
sec  2048R/0EF13559 2012-01-21
+
uid                  my_key <noone@example.com>
+
ssb  2048R/67417FEB 2012-01-21
+
</console>
+
  
The private key can be exported using either the key name or key number. In this case "my_key" or "0EF13559".
 
To cut and paste the key. Ie, if logging in remotely.
 
 
<console>
 
<console>
###i## gpg -a --export-secret-key 0EF13559
+
Available versions of the zfs-srm
</console>
+
# ##i##equery y zfs-srm
  
To create a key file:
+
[I]3.5.0 | o ~ o o o o o o o o o o o | o 3.5.0 | gentoo
<console>
+
###i## gpg -o private_decryption.gpgkey --export-secret-key 0EF13559
+
 
</console>
 
</console>
  
Now store this key somewhere secure. The backup is only as secure as the private key.
+
(These versions match the version of the System Rescue CD).
  
Modify the cron job at <tt>/etc/cron.daily/backup</tt>:
+
Installing the zfs-srm will automatically pull the '''"bliss-isomaker"''' package which is just a script that assists you with rebuilding the System Rescue CD ISO so that the ISO includes the ZFS srms. The script lets you make a bootable ISO or a bootable USB.
  
<pre>
+
Once it's installed, switch to the /opt/bliss-isomaker folder
#! /bin/bash
+
cd /etc
+
git bundle create /tmp/backup.bundle --all
+
cd /tmp
+
gpg -o backup.gpg -r 'my-key' --batch --homedir /root/.gnupg -vvv  --passphrase-fd 0 --yes -e backup.bundle
+
mv -v -f backup.gpg /home/dropbox/Dropbox/Private/
+
</pre>
+
  
Replace "my-key" with the appropriate name from the key list.
 
Also note the change from -c for symmetric encryption to -e for private/public key encryption
 
 
==== Sign Backups ====
 
Create a 2nd private/public (signing) key pair. The private key is used to sign and the public key is used to check the authenticity/integrity.
 
 
<console>
 
<console>
###i## gpg --gen-key
+
# ##i##cd /opt/bliss-isomaker
 
</console>
 
</console>
  
{{fancynote| The password for this key will be required in the script below.}}
+
You should now see a directory layout that looks similar to this:
In this case the private key is required to sign the backup and the public key is used to check the integrity of the backup.
+
Follow a similar process as above to copy the public key to to another computer/storage media.
+
  
List the private keys:
 
 
<console>
 
<console>
###i## gpg -k
+
octopus bliss-isomaker # ls -l
 +
total 100
 +
drwxr-xr-x 2 root root    5 May 24 01:41 3.5.0
 +
-rw-r--r-- 1 root root 1397 May 24 01:31 LICENSE
 +
-rw-r--r-- 1 root root  312 May 24 01:31 README
 +
-rw-r--r-- 1 root root  576 May 24 01:31 USAGE
 +
-rwxr-xr-x 1 root root 3228 May 24 01:31 create
 +
drwxr-xr-x 2 root root    3 May 24 01:41 iso
 +
drwxr-xr-x 2 root root    4 May 24 01:41 resources
 +
drwxr-xr-x 2 root root    3 May 24 01:41 srm
 
</console>
 
</console>
{{fancynote| <tt>-K</tt> lists private keys while <tt>-k</tt> lists public keys.}}
 
  
Then export this public key via cut and paste:
+
The layout is as follows:
 +
* 3.5.0 - This folder contains the System Rescue CD 3.5.0 specific srms for both the standard and alternate kernel that were installed by emerge.
 +
* create - A script to automatically recreate the system rescue cd iso or usb with the zfs stuff included
 +
* iso - directory to place your system rescue cd iso in
 +
* srm - directory that has the srms
 +
* resources - the files in this folder contain function calls that the 'create' script uses. You don't need to worry about these.
 +
 
 +
=== Installing the SRM ===
 +
There are a few ways to do this, you can either use one of the scripts, or you can do it manually. Before anything, make sure to copy the SRMs from the <Version> folder to the srm folder if you are using a prebuilt one:
 +
 
 
<console>
 
<console>
###i## gpg -a --export <key name or number>
+
# ##i##cp 3.5.0/* srm
 
</console>
 
</console>
  
Or to create a key file:
+
==== Generating a new iso ====
 +
If you just want to remake a new iso so that you can burn onto a cd or use in a virtual machine, just copy your iso over to the iso directory, and run the 'create' script. The new iso will be located in the iso directory as well with a -zfs ending.
 +
 
 
<console>
 
<console>
###i## gpg -o public_signing.gpgkey --export <key name or number>
+
Running the script with 1 (or iso) chooses to make an iso. You can also pass the version as well.
</console>
+
# ##i##./create 1
  
Now store this key somewhere secure.
+
or
  
Modify the backup cron job at <tt>/etc/cron.daily/backup</tt>:
+
# ##i##./create iso 3.5.0
 +
</console>
 +
 
 +
==== Creating a fresh usb ====
 +
If you want to have zfs on a usb with system rescue cd, put the iso in your iso dir, and then run the usb script. It will ask you what usb you want to format (This will delete everything), and then install system rescue cd onto it. Once that is done it will copy the zfs stuff over.
  
<pre>
 
#! /bin/bash
 
cd /etc
 
git bundle create /tmp/backup.bundle --all
 
cd /tmp
 
echo 'signing_key_password' | gpg -s -o backup.gpg -r 'my-encryption-key' --batch --homedir /root/.gnupg -vvv  --passphrase-fd 0 --yes -e backup.bundle
 
mv -v -f backup.gpg /home/dropbox/Dropbox/Private/
 
</pre>
 
  
{{fancynote| the script will require the password for your private (signing) key to sign the backup. Replace "password" with the password for your signing private key.
 
And as there is sensitive information in this file don't forget to remove read permission.}}
 
 
<console>
 
<console>
###i## chmod og-r /etc/cron.daily/backup
+
Running the script with 2 (or usb) chooses to make an usb. You can also pass the version as well.
</console>
+
# ##i##./create 2
  
== To Restore From A Backup ==
+
or
This restore will assume your are starting with a new blank disk.
+
 
Start by performing a stage 3 install, upto and including section 5 "Chroot into your new system." http://www.funtoo.org/wiki/Funtoo_Linux_Installation
+
# ##i##./create usb 3.5.0
 +
</console>
  
Then the restore process is:
+
==== Manual installation onto a usb ====
# Download backup from dropbox
+
Assuming that your installing to a flash drive and that the flash drive is mounted in /mnt
# Decrypt
+
# Clone
+
# Link world file
+
# Emerge world
+
# Compile the kernel
+
# Restore grub bootloader
+
# Reboot
+
  
== Download backup from dropbox ==
+
1. Mount your usb drive
Log into your dropbox account and find your backup file. Move it to a public area if it isn't already in one. Then right click on it and click "copy public link."
+
Now on the computer to be restored, delete the contents of the /etc folder and download the backup file.
+
  
(Need to check if this needs done before chrooting into the new install).
 
 
<console>
 
<console>
###i## cd /etc
+
# ##i##mount /dev/sdX# /mnt/usbstick
###i## rm -rf *
+
###i## cd /tmp
+
###i## wget http://dl.dropbox.com/link-to-backup-file/backup.gpg
+
 
</console>
 
</console>
  
{{fancynote| if you have to copy the link from another computer and therefore can not cut and paste it, there is a "shorten link" option.}}
+
'''Where X# is the letter and number of your device. Immediately after you plug your usb in, type `dmesg | tail` in the console and you should see it.'''
  
== Decrypt ==
 
If you used a public/private key to encrypt, and optionally signed the backup, import the decryption and signing keys.
 
  
Note:
+
2. Copy the zfs .srms to /mnt/usbstick
# The decryption key is the private key of the encryption key pair - private_decryption.gpgkey
+
# The signing key is the public key of the signing key pair - public_signing.gpgkey
+
  
To import the keys by cut and paste:
 
 
<console>
 
<console>
###i## gpg --import <<EOF
+
# ##i##cp zfs-core-3.4.37-std350-amd64.srm zfs-kmod-3.4.37-std350-amd64.srm zfs-kmod-3.8.4-alt350-amd64.srm /mnt/usbstick
 
</console>
 
</console>
{{fancynote| The last line after pasting the key should be "EOF"}}
 
Repeat for both keys.
 
  
To import the keys by file:
+
3. List your /mnt/usbstick directory and you should see something similar to the following in your /mnt/usbstick
<console>
+
###i## gpg --import private_decryption.gpgkey
+
###i## gpg --import public_signing.gpgkey
+
</console>
+
  
Decrypt the backup:
 
 
<console>
 
<console>
###i## gpg -d backup.gpg > backup.bundle
+
octopus 3.5.0 # ls -l /mnt/usbstick
 +
total 305028
 +
drwxr-xr-x 3 root root      4096 May 23 20:51 boot
 +
drwxr-xr-x 2 root root      4096 May 23 20:51 bootdisk
 +
drwxr-xr-x 2 root root      4096 May 23 20:51 bootprog
 +
drwxr-xr-x 3 root root      4096 May 23 20:51 efi
 +
drwxr-xr-x 2 root root      4096 May 23 20:51 ntpasswd
 +
-rwxr-xr-x 1 root root      2349 May 23 20:51 readme.txt
 +
drwxr-xr-x 3 root root      4096 May 23 20:52 syslinux
 +
-rwxr-xr-x 1 root root 309252096 May 23 20:51 sysrcd.dat
 +
-rwxr-xr-x 1 root root        45 May 23 20:51 sysrcd.md5
 +
drwxr-xr-x 2 root root      4096 May 23 20:51 usb_inst
 +
-rwxr-xr-x 1 root root    15889 May 23 20:51 usb_inst.sh
 +
-rwxr-xr-x 1 root root      877 May 23 20:51 usbstick.htm
 +
-rwxr-xr-x 1 root root        6 May 23 20:51 version
 +
-rwxr-xr-x 1 root root  2068480 May 23 20:51 zfs-core-3.4.37-std350-amd64.srm
 +
-rwxr-xr-x 1 root root    483328 May 23 20:51 zfs-kmod-3.4.37-std350-amd64.srm
 +
-rwxr-xr-x 1 root root    483328 May 23 20:51 zfs-kmod-3.8.4-alt350-amd64.srm
 
</console>
 
</console>
  
If the backup was signed and you have correctly imported the signing public key you should see a message similar to:
+
Now un-mount your flash drive and boot it into the machine that you want to use ZFS on.
<console>
+
gpg: Good signature from "my_signing_key <noone@example.com>"
+
</console>
+
  
== Clone ==
 
 
<console>
 
<console>
###i## git clone /tmp/backup.bundle /etc/
+
# ##i##umount /mnt/usbstick
 
</console>
 
</console>
  
== Link world file ==
+
== Booting into the correct kernel ==
 +
'''If you are using the standard srm:'''
 
<console>
 
<console>
###i## ln /etc/world /var/lib/portage/world
+
C) Standard 64bit kernel (rescue64) with more choice... >
 +
1. SystemRescueCd with default options
 
</console>
 
</console>
  
== Emerge world ==
+
'''If you are using the alternative srm:'''
 
<console>
 
<console>
###i## emerge --sync
+
E) Alternative 64bit kernel (altker64) with more choice... >
###i## layman -S
+
1. SystemRescueCd with default options
###i## emerge -uDaNv world
+
 
</console>
 
</console>
  
== Compile the kernel (genkernel)==
+
2. Run 'depmod' so that the new modules can be seen by the kernel.
If you have genkernel set to save config files (the default):
+
<console>
+
###i## cp /etc/kernels/kernel-config-x86_64-<latest version>-gentoo /usr/src/linux/.config
+
</console>
+
  
Otherwise use the currently loaded kernel's config:
 
 
<console>
 
<console>
###i## zcat /proc/config.gz > /usr/src/linux/.config
+
# ##i##depmod
 
</console>
 
</console>
  
Then compile the kernel:
+
{{fancywarning|You must run '''depmod'''. If you don't, then you will get failed to load the ZFS stack error!}}
<console>
+
###i## genkernel --oldconfig --no-mrproper all
+
</console>
+
  
== Restore grub bootloader ==
+
3. Use ZFS as usual. If you type '''zpool status''' and then type '''dmesg | tail''',
<console>
+
###i## grub-install --no-floppy /dev/sda
+
###i## boot-update
+
</console>
+
  
Adjust the device as required if installing to another location.
+
you should see something that says:
  
== Reboot ==
 
 
<console>
 
<console>
###i## reboot
+
ZFS: Loaded module v0.6.1-1, ZFS pool version 5000, ZFS filesystem version 5
 
</console>
 
</console>
 +
 +
If you see the above, then the modules loaded successfully!
 +
 +
Enjoy System Rescue CD with ZFS :)
 +
 +
 +
== Using the premade iso ==
 +
 +
<big>If you don't want to do any of the above stuff but just want the ISO with the ZFS SRMs already included, simply [http://ftp.osuosl.org/pub/funtoo/distfiles/sysresccd/ Download the ISO].</big>
 +
 +
[[Category:HOWTO]]

Latest revision as of 17:32, 22 February 2014

This HOWTO will show you how to create your own zfs srm or download the prebuilt zfs srm module from my website.

Contents

[edit] Compiling a compatible kernel

The first thing you need to do is decide for which version of System Rescue CD you will be building for. Each System Rescue CD version has a different set of kernels. Specifically each version has a Stable kernel and an Alternate Kernel. The stable kernels get minor version increments every release (Sometimes major ones but usually doesn't happen too quickly), the Alternate kernels tend to move up much faster. For example System Rescue CD had the following stable/alternate kernels:

Version Stable Alternate
3.7.0 3.4.47 3.9.4
3.5.0 3.4.37 3.8.4

[edit] Download kernel and patches

Once you decide which version you want to use, you can go to http://kernel.sysresccd.org/ and pick the version you want. After you do this, download all the stuff in that directory and put them in their own folder (a work directory). The files that are named std-sources are the patches for the regular stable kernel, the patches named alt-sources are the patches for the alternate kernel. Once you have the patches, go to kernel.org and also download the initial release of the kernel sources you want. For example, if you want to build an srm for 3.4.2 which uses the 3.4.35 kernel, just download the 3.4 (3.4.0) kernel. You just need to download the initial release because one of the patches you downloaded earlier is the patch to update the 3.4.0 to 3.4.35.

Once you download the kernel you want, let's extract and patch it.

# tar xf linux-3.4.tar.bz2
# mv linux-3.4 linux-3.4.35-std342-amd64
# bzcat std-sources-3.4-01-stable-3.4.35.patch.bz2 | patch -p1 -d linux-3.4.35-std342-amd64
# bzcat std-sources-3.4-02-fc16.patch.bz2 | patch -p1 -d linux-3.4.35-std342-amd64
# bzcat std-sources-3.4-03-aufs.patch.bz2 | patch -p1 -d linux-3.4.35-std342-amd64

[edit] Retrieve kernel configuration

Once that is complete, you will need to get the kernel config specifically for the kernel and architecture you want. The easiest way I found to do this was to download the System Rescue CD you want, and then boot it inside a VM. Once you are inside the VM, go to the /root directory, and it will be a file named kernel-<version>.conf. Copy this file over SSH and put it in a safe location. Also copy it over into the linux kernel directory and rename it to .config.

[edit] Build the kernel

Once you copy the kernel config over, do a make menuconfig to begin. You will only need to change two settings inside the config, add the proper kernel name, and remove the old initramfs source files directory. This directory doesn't exist locally. This is a remnant of upstream's build process.

Follow the below tree structure, and make it equal to what is below:

(-std342-amd64) Local version - append to kernel release
      [*] Initial RAM filesystem and RAM disk (initramfs/initrd) support
      ( ) Initramfs source file(s)

The Local version above will end up making the kernel modules directory be 3.4.35-std342-amd64. The reason the kernel modules directory has to be equal to this is because our directory will be merged into the System Rescue CD later on, so the names need to be exactly the same so that the System Rescue CD system can find our kernel modules.

If you want to build an srm (which you do), you also need to change the following inside the kernel config so that squashfs-tools can work later on:

Miscellaneous Filesystems ->
      <*>   SquashFS 4.0 - Squashed file system support
       [*]     Squashfs XATTR support
       [*]     Include support for ZLIB compressed file systems
       [*]     Include support for LZO compressed file systems
       [*]     Include support for XZ compressed file systems
       [ ]     Use 4K device block size?
       [ ]     Additional option for memory-constrained systems

Now save and exit, and continue to build your kernel.

# make bzImage modules
# make modules_install
Note: If you have multiple cores, do a make -j<# of processors+1> to build faster. Example: make -j9 for an 8 core machine.

[edit] Using the portage tree

Inside the portage tree, there are ebuilds already prepared that will download the kernel, patch it, and perform any required substitutions. Afterwards you literally just go into your /usr/src/<kernel> directory, and run make. To see the available versions, run the following:

Versions available for standard kernel
# equery y std-sources

[I]3.4.37-r1 | o ~ o o o o o o o o o o o | o 3.4.37 | gentoo

Versions available for alternate kernel
# equery y alt-sources

[I]3.8.4 | o ~ o o o o o o o o o o o | o 3.8.4 | gentoo

I haven't written the ebuilds in a way where it's easy to tell (at first glance) what System Rescue CD version you need. However, you can check the ebuild and you will see the version of the CD inside there. You can also check the chart above.

If you wanted to install the standard kernel sources, you just emerge it like any other application:

# emerge std-sources

Once the kernel and the kernel module are installed, you need to make the SRM.

[edit] Creating the SRM

It's time to gather the required files and pack them together in order for it to become an SRM. An SRM is nothing more than a directory that has been compressed with squashfs. You can emerge the "Bliss Initramfs Creator" which has a feature that will automagically build the srm for you.


You also need to have spl, zfs, and zfs-kmod installed on your system before you try to make the srm. Emerging "bliss-initramfs" should automatically pull those packages. If it doesn't, make sure you install them.

# echo "sys-kernel/bliss-initramfs srm" >> /etc/portage/package.use
# emerge bliss-initramfs


You should now have the following directory layout inside the /opt/bliss-initramfs folder:

octopus bliss-initramfs # ls -l
total 117
-rw-r--r-- 1 root root 6718 May 23 18:05 CHANGES
-rw-r--r-- 1 root root  176 May 23 18:05 CREDITS
-rw-r--r-- 1 root root  431 May 23 18:05 HELP
-rw-r--r-- 1 root root 1397 May 23 18:05 LICENSE
-rw-r--r-- 1 root root 1852 May 23 18:05 README
-rw-r--r-- 1 root root 3194 May 23 18:05 USAGE
-rwxr-xr-x 1 root root 2891 May 23 18:05 createInit
drwxr-xr-x 3 root root    4 May 23 18:11 files
drwxr-xr-x 3 root root    6 May 23 18:11 hooks
drwxr-xr-x 2 root root    5 May 23 22:01 resources


Then run the createInit script and follow the instructions for the kernel you want to make the srm for (In this case it's 3.4.37-std350-amd64):

-----------------------------------
| Bliss Initramfs Creator - v1.8.1
| Author: Jonathan Vasquez <jvasquez1011@gmail.com>
| Distributed under the Simplified BSD License
-----------------------------------

>>> Which initramfs would you like to generate:
>> 1. ZFS
>> 2. Encrypted ZFS (LUKS + ZFS)
>> 3. More Options
>> 4. Exit Program

>>> Current choice [1]: 3 ↵

>>> More Options:

>> 1. ZFS - System Rescue Module
>> 2. Back
>> 3. Exit Program

>>> Current choice [1]: 

>>> Creating a ZFS System Rescue Module!

>>> Do you want to use the current kernel: 3.8.13-ALL? [Y/n]: n ↵

>>> Please enter the kernel name: 3.4.37-std350-amd64 ↵

>>> Detected 64 bit platform
>>> Checking to see if modules directory exists for 3.4.37-std350-amd64...
>>> Creating SRMs for 3.4.37-std350-amd64...
>>> Creating directory structure for initramfs...
>>> Checking preliminary binaries...
>>> Checking binaries...
>>> Using ZFS
>>> Checking modules...
>>> Copying binaries...
>>> Copying modules...
>>> Copying documentation...
>>> Copying udev rules...
>>> Compressing kernel modules...
>>> Getting dependencies...
>>> Copying dependencies...
>>> Configuring files...
>>> Creating and Packing SRMs...
>>> Complete :)


Now you should have two new files:

-rw-r--r-- 1 root root 2068480 May 24 01:05 zfs-core-3.4.37-std350-amd64.srm
-rw-r--r-- 1 root root  483328 May 24 01:05 zfs-kmod-3.4.37-std350-amd64.srm


Now all you need to do is put those two files in the root of your USB directory.

If you are making srms for both the standard and alternate kernels, you will end up with two zfs-core files and two zfs-kmod files (1 set for each kernel). You don't need to put the zfs-core that it makes for one of the kernels. The zfs-core srm only has zfs program binaries, man pages, udev rules, and a few other things. The zfs-kmod is the srm that has the kernel modules (and only the kernel modules). So you can easily just put two zfs-kmods (one for each kernel version) and just re-use the same zfs-core for both of them.

[edit] Using the prebuilt srm

[edit] Download the SRM

If you didn't build your own srm and want to use the prebuilt one, just emerge "zfs-srm" from the tree:

# emerge zfs-srm

You can check available versions just as you did above for the kernels:

Available versions of the zfs-srm
# equery y zfs-srm

[I]3.5.0 | o ~ o o o o o o o o o o o | o 3.5.0 | gentoo

(These versions match the version of the System Rescue CD).

Installing the zfs-srm will automatically pull the "bliss-isomaker" package which is just a script that assists you with rebuilding the System Rescue CD ISO so that the ISO includes the ZFS srms. The script lets you make a bootable ISO or a bootable USB.

Once it's installed, switch to the /opt/bliss-isomaker folder

# cd /opt/bliss-isomaker

You should now see a directory layout that looks similar to this:

octopus bliss-isomaker # ls -l
total 100
drwxr-xr-x 2 root root    5 May 24 01:41 3.5.0
-rw-r--r-- 1 root root 1397 May 24 01:31 LICENSE
-rw-r--r-- 1 root root  312 May 24 01:31 README
-rw-r--r-- 1 root root  576 May 24 01:31 USAGE
-rwxr-xr-x 1 root root 3228 May 24 01:31 create
drwxr-xr-x 2 root root    3 May 24 01:41 iso
drwxr-xr-x 2 root root    4 May 24 01:41 resources
drwxr-xr-x 2 root root    3 May 24 01:41 srm

The layout is as follows:

  • 3.5.0 - This folder contains the System Rescue CD 3.5.0 specific srms for both the standard and alternate kernel that were installed by emerge.
  • create - A script to automatically recreate the system rescue cd iso or usb with the zfs stuff included
  • iso - directory to place your system rescue cd iso in
  • srm - directory that has the srms
  • resources - the files in this folder contain function calls that the 'create' script uses. You don't need to worry about these.

[edit] Installing the SRM

There are a few ways to do this, you can either use one of the scripts, or you can do it manually. Before anything, make sure to copy the SRMs from the <Version> folder to the srm folder if you are using a prebuilt one:

# cp 3.5.0/* srm

[edit] Generating a new iso

If you just want to remake a new iso so that you can burn onto a cd or use in a virtual machine, just copy your iso over to the iso directory, and run the 'create' script. The new iso will be located in the iso directory as well with a -zfs ending.

Running the script with 1 (or iso) chooses to make an iso. You can also pass the version as well.
# ./create 1

or

# ./create iso 3.5.0

[edit] Creating a fresh usb

If you want to have zfs on a usb with system rescue cd, put the iso in your iso dir, and then run the usb script. It will ask you what usb you want to format (This will delete everything), and then install system rescue cd onto it. Once that is done it will copy the zfs stuff over.


Running the script with 2 (or usb) chooses to make an usb. You can also pass the version as well.
# ./create 2

or

# ./create usb 3.5.0

[edit] Manual installation onto a usb

Assuming that your installing to a flash drive and that the flash drive is mounted in /mnt

1. Mount your usb drive

# mount /dev/sdX# /mnt/usbstick

Where X# is the letter and number of your device. Immediately after you plug your usb in, type `dmesg | tail` in the console and you should see it.


2. Copy the zfs .srms to /mnt/usbstick

# cp zfs-core-3.4.37-std350-amd64.srm zfs-kmod-3.4.37-std350-amd64.srm zfs-kmod-3.8.4-alt350-amd64.srm /mnt/usbstick

3. List your /mnt/usbstick directory and you should see something similar to the following in your /mnt/usbstick

octopus 3.5.0 # ls -l /mnt/usbstick
total 305028
drwxr-xr-x 3 root root      4096 May 23 20:51 boot
drwxr-xr-x 2 root root      4096 May 23 20:51 bootdisk
drwxr-xr-x 2 root root      4096 May 23 20:51 bootprog
drwxr-xr-x 3 root root      4096 May 23 20:51 efi
drwxr-xr-x 2 root root      4096 May 23 20:51 ntpasswd
-rwxr-xr-x 1 root root      2349 May 23 20:51 readme.txt
drwxr-xr-x 3 root root      4096 May 23 20:52 syslinux
-rwxr-xr-x 1 root root 309252096 May 23 20:51 sysrcd.dat
-rwxr-xr-x 1 root root        45 May 23 20:51 sysrcd.md5
drwxr-xr-x 2 root root      4096 May 23 20:51 usb_inst
-rwxr-xr-x 1 root root     15889 May 23 20:51 usb_inst.sh
-rwxr-xr-x 1 root root       877 May 23 20:51 usbstick.htm
-rwxr-xr-x 1 root root         6 May 23 20:51 version
-rwxr-xr-x 1 root root   2068480 May 23 20:51 zfs-core-3.4.37-std350-amd64.srm
-rwxr-xr-x 1 root root    483328 May 23 20:51 zfs-kmod-3.4.37-std350-amd64.srm
-rwxr-xr-x 1 root root    483328 May 23 20:51 zfs-kmod-3.8.4-alt350-amd64.srm

Now un-mount your flash drive and boot it into the machine that you want to use ZFS on.

# umount /mnt/usbstick

[edit] Booting into the correct kernel

If you are using the standard srm:

C) Standard 64bit kernel (rescue64) with more choice... >
1. SystemRescueCd with default options

If you are using the alternative srm:

E) Alternative 64bit kernel (altker64) with more choice... >
1. SystemRescueCd with default options

2. Run 'depmod' so that the new modules can be seen by the kernel.

# depmod
Warning: You must run depmod. If you don't, then you will get failed to load the ZFS stack error!

3. Use ZFS as usual. If you type zpool status and then type dmesg | tail,

you should see something that says:

ZFS: Loaded module v0.6.1-1, ZFS pool version 5000, ZFS filesystem version 5

If you see the above, then the modules loaded successfully!

Enjoy System Rescue CD with ZFS :)


[edit] Using the premade iso

If you don't want to do any of the above stuff but just want the ISO with the ZFS SRMs already included, simply Download the ISO.