Difference between pages "Cloud Backup" and "Funtoo Resources on LAN"

From Funtoo
(Difference between pages)
Jump to: navigation, search
 
 
Line 1: Line 1:
[[Category:HOWTO]]
+
== Introduction ==
 +
This page will show you how to setup Funtoo resources on your LAN, so you can have a faster access to them via your local area network, for installing packages, updating your local trees, and keep everything in sync.
  
This howto will describe a method for automatically backing up your funtoo install to the internet, in this case dropbox, but any online storage will do. Gentoo describes a method of creating a stage 4 archive. The problem with a stage 4 is that it is large and it archives a lot of unnecessary files. Such as applications that can be reinstalled from an emerge world. Instead, this method will aim for more of a "stage 3.5."
+
One use case for this is, if you have more than one machine and you want to keep all machines up to date in such a way that all the machines update to the tree of one machine, and that one primary machines pulls from the outside world (funtoo @ github).
  
{{fancynote| This method does not attempt to backup everything. The intention is only to backup the system. Optionally you can also archive and copy your <tt>/home</tt> folder if you have enough online storage.}}
+
Example:
  
== Use Case ==
+
Machine A = Primary Server on LAN (Pulls from github, contains distfiles, contains binaries - built by FEATURES="buildpkg")
A backup machine currently provides network drives on a home LAN to allow clients on the LAN to backup to, using apps such as Time Machine (Mac) and Genie Timeline (Windows). As this machine ''is'' the backup machine it doesn't have anywhere to backup to itself. In this situation a backup solution is provided by backing up to somewhere online - dropbox. If a restore from the backup is required, the client machine's backups would be trashed, and the backup machine restored.
+
  
== Automatic Backup Archives With Etckeeper ==
+
Machine B = Another machine on your LAN. Could be a laptop. (Pulls from Machine A. Thus any distfiles that it can get will be pulled from the primary server - thus reducing network load and basically making your primary server a fast cache for future funtoo installs and upgrades. Also this means that you can compile packages on your primary server, and just pull them from Machine B. You can be confident that since Machine B can only be as up-to-date as Machine A, that it won't try to pull some unknown package that hasn't been compiled yet).
Etckeeper is a tool that is used to save versions of <tt>/etc</tt>, including meta-data in a version control repository such as git.
+
As etckeeper is not in the funtoo portage tree, layman is used to provide an overlay.
+
=== Install etckeeper via layman ===
+
Before you install layman it is worth mentioning that you probably want <tt>USE="git subversion"</tt> in <tt>/etc/portage/make.conf</tt>. After adjusting use flags, to install layman you run:
+
<console>
+
###i## emerge layman
+
</console>
+
In order to backup the layman configuration, but not the portage overlay trees, make the following modifications to the default install.
+
Tell Portage about layman-fetched repositories by adding the following line to <tt>/etc/portage/make.conf</tt>:
+
  
<pre>
+
== Setting up Machine A ==
source /etc/layman/make.conf
+
</pre>
+
  
Modify the following lines in <tt>/etc/layman/layman.cfg</tt>:
+
Machine A is your primary server and it's basically already complete. Just make sure that SSH is enabled and started (Which it is by default on Funtoo) and make sure that your distfiles and packages are being served out over http. For this setup, root over SSH will be enabled and this is the user that we will be using to sync. You can change this to whatever you want if you really want to. You will need to modify the settings on your own though.
  
<pre>
+
=== Setting up the binary server ===
storage  : /var/lib/layman
+
Follow the instructions on this page: [[How_to_set_up_a_binary_package_server#Setting_up_the_host_machine|Setting up the Host Machine]]
installed : /etc/layman/installed.xml
+
make_conf : /etc/layman/make.conf
+
</pre>
+
  
Add the bgo-overlay. As described on their web page, [http://bgo.zugaina.org/ bgo.zugaina.org].
+
=== Some security tips for SSH ===
<console>
+
In your /etc/ssh/sshd_config, you can add the following:
###i## layman -o http://gpo.zugaina.org/lst/gpo-repositories.xml -L
+
###i## layman -a bgo-overlay -o http://gpo.zugaina.org/lst/gpo-repositories.xml
+
</console>
+
  
More information about layman can be found here: http://www.gentoo.org/proj/en/overlays/userguide.xml
+
==== Change SSH default port ====
 +
You should definitely do this because if you don't and you open port 22 to the world, you will get attacked. I noticed that I was getting attacked multiple times every single day and the only way I was able to reduce it was to install '''fail2ban'''. However, changing the port to another port dropped the number of attacks from [Every Day * Multiple] to 0.
  
Then unmask and install etckeeper.
 
 
<console>
 
<console>
###i## emerge etckeeper --autounmask-write
+
# Change Port 22 to some other port
###i## emerge etckeeper
+
Port 8902
</console>
+
 
+
{{fancynote| To update layman overlays do:}}
+
<console>
+
###i## layman -S
+
 
</console>
 
</console>
  
 +
==== Limit users ====
 +
Attackers will try to brute force user names. You can set the '''"AllowUsers"''' value so that it only lets those users login.
  
If you see the following error -- apply this fix:
 
 
<console>
 
<console>
###i## emerge etckeeper
+
# Let's say that we only want root and roger to login
Calculating dependencies... done!
+
AllowUsers root roger
>>> Verifying ebuild manifests
+
!!! A file is not listed in the Manifest: '/var/lib/layman/bgo-overlay/sys-apps/etckeeper/files/etckeeper-gentoo-0.58.patch'
+
  
###i## cd /var/lib/layman/bgo-overlay/sys-apps/etckeeper
+
# PermitRootLogin should also be set to 'yes' if it isn't
###i## ebuild etckeeper-0.58-r2.ebuild manifest
+
PermitRootLogin yes
###i## emerge etckeeper
+
 
</console>
 
</console>
  
== Configure etckeeper ==
+
== Setting up secondary machine ==
Move any config files that do not live in <tt>/etc</tt>. i.e.
+
Check <tt>/root</tt> for any files to be archive, such as iptables scripts and move them to <tt>/etc</tt>.
+
  
{{fancynote| because funtoo uses [[Boot-Update]], this means <tt>/boot/grub/grub.cfg</tt> does not need to be archived.}}
+
This section will teach you how to pull from a remote tree, and set up your machine to use the distfiles and packages that your remote tree has (Could be a tree from a local network).
  
To ensure your portage world file is archived, make the following link:
+
We will assume that you are pulling the tree from your local network and that you are accessing the tree via ssh.
<console>
+
###i## ln /var/lib/portage/world /etc/world
+
</console>
+
  
Initialise the git repository.
+
We will assume that this is a fresh install and that you don't have any previous portage trees or setting tweaks.
<console>
+
###i## etckeeper init
+
Initialized empty Git repository in /etc/.git/
+
###i## etckeeper commit "Initial commit."
+
</console>
+
  
If you don't already have cron installed, emerge it now.
+
==== /etc/portage/make.conf ====
<console>
+
What we will first due is edit our make.conf so that it pulls resources from the correct locations.
###i## emerge vixie-cron
+
</console>
+
And write the cron job to save an hourly version of <tt>/etc</tt>.
+
  
{{fancynote| git will only create a new version (commit) if there are changes from the previous one.}}
+
We will assume that Machine A's IP is 192.168.1.100, SSH port is 8902, that we are logging in as the root user, and that the portage tree is in it's default location of /usr/portage on that machine.
Edit the file <tt>/etc/cron.hourly/etckeeper:
+
  
<pre>
+
Open up the make.conf file for Machine B and add the following:
#! /bin/bash
+
etckeeper commit "Hourly auto-commit"
+
</pre>
+
  
== Encrypt and copy backups online ==
 
=== Copy To Dropbox ===
 
 
<console>
 
<console>
###i## emerge dropbox
+
# This is where it will try to pull the tree the first time it syncs. After this it will just use w/e the git tree origin is set to
</console>
+
SYNC="ssh://192.168.1.100:8902/usr/portage"
  
Add a dropbox user:
+
# This sets where it will try to find source files (distfiles). We will try to pull from Machine A first, but if Machine A doesn't have the files we need, it will go to Gentoo to get them
<console>
+
GENTOO_MIRRORS="http://192.168.1.100/funtoo http://distfiles.gentoo.org"
###i## useradd dropbox
+
</console>
+
  
Write the dropbox init files in <tt>/etc/conf.d/dropbox</tt>:
+
# This sets where it will try to find binary packages (We are using an http server to make it more convenient for us because we don't need "security" to just download precompiled packages)
 
+
PORTAGE_BINHOST="http://192.168.1.100/funtoo/packages"
<pre>
+
DROPBOX_USERS="dropbox"
+
</pre>
+
<br>
+
<pre>
+
#!/sbin/runscript
+
# Copyright 1999-2004 Gentoo Foundation
+
# Distributed under the terms of the GNU General Public License, v2 or later
+
# $Header: /var/cvsroot/gentoo-x86/sys-fs/dropbox/files/dropbox.init-1.0,v 1.4 2007/04/04 13:35:25 cardoe Exp $
+
 
+
NICENESS=5
+
 
+
depend() {
+
    need localmount net
+
    after bootmisc
+
}
+
 
+
start() {
+
    ebegin "Starting dropbox..."  
+
    for dbuser in $DROPBOX_USERS; do
+
        start-stop-daemon -S -b -m --pidfile /var/run/dropbox-$dbuser.pid  -N $NICENESS -u $dbuser -v -e HOME="/home/$dbuser" -x /opt/dropbox/dropboxd
+
    done
+
    eend $?
+
}
+
 
+
stop() {
+
    ebegin "Stopping dropbox..."
+
    for dbuser in $DROPBOX_USERS; do
+
        start-stop-daemon --stop --pidfile /var/run/dropbox-$dbuser.pid
+
    done
+
    eend $?
+
}
+
 
+
status() {
+
    for dbuser in $DROPBOX_USERS; do
+
        if [ -e /var/run/dropbox-$dbuser.pid ] ; then
+
            echo "dropboxd for USER $dbuser: running."
+
        else
+
            echo "dropboxd for USER $dbuser: not running."
+
        fi
+
    done
+
    eend $?
+
}
+
</pre>
+
Start dropbox now and at boot time:
+
<console>
+
###i## chmod 0755 /etc/init.d/dropbox
+
###i## /etc/init.d/dropbox start
+
###i## rc-update add dropbox default
+
 
</console>
 
</console>
  
After starting the dropbox daemon, it will provide a http link. You will need to visit this site just once to associate your computer with your dropbox account.
+
=== If you already have a tree ===
 +
If you already have a tree and would like to delete it and start fresh, you can just delete the .git folder inside the /usr/portage directory and the next time you run '''emerge --sync''', portage will tell you that it isn't a git repository, and it will wipe the contents and sync the new tree.
  
Write the cron job to make the backup archive and move it online. Edit the file <tt>/etc/cron.daily/backup</tt>:
+
== Conclusion ==
  
<pre>
+
That is basically it.
#! /bin/bash
+
cd /etc
+
git bundle create /tmp/backup.bundle --all
+
cd /tmp
+
mv -v -f backup.bundle /home/dropbox/Dropbox/Private/
+
</pre>
+
 
+
Make the script executable:
+
<console>
+
###i## chmod +x /etc/cron.daily/backup
+
</console>
+
 
+
=== Encrypt Backups ===
+
It is a good idea to encrypt your backup before moving it online. This can be done with gpg, using a symmetric (password only) or public/private key encryption. Additionally you can chose to sign the backup to check its integrity before restoring.
+
<console>
+
###i## emerge gpg
+
</console>
+
 
+
==== Symmetric Encryption ====
+
There is no preparation required to use a symmetric key as all that is required is simply a passphrase.  Just modify the cron job. Edit <tt>/etc/cron.daily/backup</tt>:
+
 
+
<pre>
+
#! /bin/bash
+
cd /etc
+
git bundle create /tmp/backup.bundle --all
+
cd /tmp
+
echo 'encryption_password' | gpg -o backup.gpg --batch --homedir /root/.gnupg -vvv  --passphrase-fd 0 --yes -c backup.bundle
+
mv -v -f router.gpg /home/dropbox/Dropbox/Private/
+
</pre>
+
{{fancyimportant| Remember to change "encryption_password"}}
+
 
+
{{fancywarning| If you forget this password the backup will be unusable. Lose the password and you lose the backup.}}
+
 
+
As there is now sensitive information in this file, you might want to remove read permission:
+
<console>
+
###i## chmod og-r /etc/cron.daily/backup
+
</console>
+
 
+
==== Private/Public key Encryption ====
+
Make a private/public encryption/decryptions key pair. The public key will be used to encrypt and the private key to decrypt.
+
<console>
+
###i## gpg --gen-key
+
</console>
+
The public key is used to create the encrypted backup and needs to live on the computer being backed up. A copy of the private key needs to be made and stored securely in another place. If this machine becomes unbootable, and this is the only place the private key lives, the backup dies with it.
+
The private key should not be kept:
+
# In the same place as the back up
+
# On the machine being backed up
+
{{fancynote| The private key is the only key that will decrypt the backup. Lose this key and/or it's password and you lose the backup.}}
+
 
+
List the private keys:
+
<console>
+
###i## gpg -K
+
/root/.gnupg/secring.gpg
+
------------------------
+
sec  2048R/0EF13559 2012-01-21
+
uid                  my_key <noone@example.com>
+
ssb  2048R/67417FEB 2012-01-21
+
</console>
+
 
+
The private key can be exported using either the key name or key number. In this case "my_key" or "0EF13559".
+
To cut and paste the key. Ie, if logging in remotely.
+
<console>
+
###i## gpg -a --export-secret-key 0EF13559
+
</console>
+
 
+
To create a key file:
+
<console>
+
###i## gpg -o private_decryption.gpgkey --export-secret-key 0EF13559
+
</console>
+
 
+
Now store this key somewhere secure. The backup is only as secure as the private key.
+
 
+
Modify the cron job at <tt>/etc/cron.daily/backup</tt>:
+
 
+
<pre>
+
#! /bin/bash
+
cd /etc
+
git bundle create /tmp/backup.bundle --all
+
cd /tmp
+
gpg -o backup.gpg -r 'my-key' --batch --homedir /root/.gnupg -vvv  --passphrase-fd 0 --yes -e backup.bundle
+
mv -v -f backup.gpg /home/dropbox/Dropbox/Private/
+
</pre>
+
 
+
Replace "my-key" with the appropriate name from the key list.
+
Also note the change from -c for symmetric encryption to -e for private/public key encryption
+
 
+
==== Sign Backups ====
+
Create a 2nd private/public (signing) key pair. The private key is used to sign and the public key is used to check the authenticity/integrity.
+
<console>
+
###i## gpg --gen-key
+
</console>
+
 
+
{{fancynote| The password for this key will be required in the script below.}}
+
In this case the private key is required to sign the backup and the public key is used to check the integrity of the backup.
+
Follow a similar process as above to copy the public key to to another computer/storage media.
+
 
+
List the private keys:
+
<console>
+
###i## gpg -k
+
</console>
+
{{fancynote| <tt>-K</tt> lists private keys while <tt>-k</tt> lists public keys.}}
+
 
+
Then export this public key via cut and paste:
+
<console>
+
###i## gpg -a --export <key name or number>
+
</console>
+
 
+
Or to create a key file:
+
<console>
+
###i## gpg -o public_signing.gpgkey --export <key name or number>
+
</console>
+
 
+
Now store this key somewhere secure.
+
 
+
Modify the backup cron job at <tt>/etc/cron.daily/backup</tt>:
+
 
+
<pre>
+
#! /bin/bash
+
cd /etc
+
git bundle create /tmp/backup.bundle --all
+
cd /tmp
+
echo 'signing_key_password' | gpg -s -o backup.gpg -r 'my-encryption-key' --batch --homedir /root/.gnupg -vvv  --passphrase-fd 0 --yes -e backup.bundle
+
mv -v -f backup.gpg /home/dropbox/Dropbox/Private/
+
</pre>
+
 
+
{{fancynote| the script will require the password for your private (signing) key to sign the backup. Replace "password" with the password for your signing private key.
+
And as there is sensitive information in this file don't forget to remove read permission.}}
+
<console>
+
###i## chmod og-r /etc/cron.daily/backup
+
</console>
+
 
+
== To Restore From A Backup ==
+
This restore will assume your are starting with a new blank disk.
+
Start by performing a stage 3 install, upto and including section 5 "Chroot into your new system." http://www.funtoo.org/wiki/Funtoo_Linux_Installation
+
 
+
Then the restore process is:
+
# Download backup from dropbox
+
# Decrypt
+
# Clone
+
# Link world file
+
# Emerge world
+
# Compile the kernel
+
# Restore grub bootloader
+
# Reboot
+
 
+
== Download backup from dropbox ==
+
Log into your dropbox account and find your backup file. Move it to a public area if it isn't already in one. Then right click on it and click "copy public link."
+
Now on the computer to be restored, delete the contents of the /etc folder and download the backup file.
+
 
+
(Need to check if this needs done before chrooting into the new install).
+
<console>
+
###i## cd /etc
+
###i## rm -rf *
+
###i## cd /tmp
+
###i## wget http://dl.dropbox.com/link-to-backup-file/backup.gpg
+
</console>
+
 
+
{{fancynote| if you have to copy the link from another computer and therefore can not cut and paste it, there is a "shorten link" option.}}
+
 
+
== Decrypt ==
+
If you used a public/private key to encrypt, and optionally signed the backup, import the decryption and signing keys.
+
 
+
Note:
+
# The decryption key is the private key of the encryption key pair - private_decryption.gpgkey
+
# The signing key is the public key of the signing key pair - public_signing.gpgkey
+
 
+
To import the keys by cut and paste:
+
<console>
+
###i## gpg --import <<EOF
+
</console>
+
{{fancynote| The last line after pasting the key should be "EOF"}}
+
Repeat for both keys.
+
 
+
To import the keys by file:
+
<console>
+
###i## gpg --import private_decryption.gpgkey
+
###i## gpg --import public_signing.gpgkey
+
</console>
+
 
+
Decrypt the backup:
+
<console>
+
###i## gpg -d backup.gpg > backup.bundle
+
</console>
+
 
+
If the backup was signed and you have correctly imported the signing public key you should see a message similar to:
+
<console>
+
gpg: Good signature from "my_signing_key <noone@example.com>"
+
</console>
+
 
+
== Clone ==
+
<console>
+
###i## git clone /tmp/backup.bundle /etc/
+
</console>
+
 
+
== Link world file ==
+
<console>
+
###i## ln /etc/world /var/lib/portage/world
+
</console>
+
 
+
== Emerge world ==
+
<console>
+
###i## emerge --sync
+
###i## layman -S
+
###i## emerge -uDaNv world
+
</console>
+
 
+
== Compile the kernel (genkernel)==
+
If you have genkernel set to save config files (the default):
+
<console>
+
###i## cp /etc/kernels/kernel-config-x86_64-<latest version>-gentoo /usr/src/linux/.config
+
</console>
+
 
+
Otherwise use the currently loaded kernel's config:
+
<console>
+
###i## zcat /proc/config.gz > /usr/src/linux/.config
+
</console>
+
 
+
Then compile the kernel:
+
<console>
+
###i## genkernel --oldconfig --no-mrproper all
+
</console>
+
 
+
== Restore grub bootloader ==
+
<console>
+
###i## grub-install --no-floppy /dev/sda
+
###i## boot-update
+
</console>
+
 
+
Adjust the device as required if installing to another location.
+
 
+
== Reboot ==
+
<console>
+
###i## reboot
+
</console>
+
  
 +
[[Category:HOWTO]]
 
[[Category:First Steps]]
 
[[Category:First Steps]]

Revision as of 19:33, 4 March 2014

Contents

Introduction

This page will show you how to setup Funtoo resources on your LAN, so you can have a faster access to them via your local area network, for installing packages, updating your local trees, and keep everything in sync.

One use case for this is, if you have more than one machine and you want to keep all machines up to date in such a way that all the machines update to the tree of one machine, and that one primary machines pulls from the outside world (funtoo @ github).

Example:

Machine A = Primary Server on LAN (Pulls from github, contains distfiles, contains binaries - built by FEATURES="buildpkg")

Machine B = Another machine on your LAN. Could be a laptop. (Pulls from Machine A. Thus any distfiles that it can get will be pulled from the primary server - thus reducing network load and basically making your primary server a fast cache for future funtoo installs and upgrades. Also this means that you can compile packages on your primary server, and just pull them from Machine B. You can be confident that since Machine B can only be as up-to-date as Machine A, that it won't try to pull some unknown package that hasn't been compiled yet).

Setting up Machine A

Machine A is your primary server and it's basically already complete. Just make sure that SSH is enabled and started (Which it is by default on Funtoo) and make sure that your distfiles and packages are being served out over http. For this setup, root over SSH will be enabled and this is the user that we will be using to sync. You can change this to whatever you want if you really want to. You will need to modify the settings on your own though.

Setting up the binary server

Follow the instructions on this page: Setting up the Host Machine

Some security tips for SSH

In your /etc/ssh/sshd_config, you can add the following:

Change SSH default port

You should definitely do this because if you don't and you open port 22 to the world, you will get attacked. I noticed that I was getting attacked multiple times every single day and the only way I was able to reduce it was to install fail2ban. However, changing the port to another port dropped the number of attacks from [Every Day * Multiple] to 0.

# Change Port 22 to some other port
Port 8902

Limit users

Attackers will try to brute force user names. You can set the "AllowUsers" value so that it only lets those users login.

# Let's say that we only want root and roger to login
AllowUsers root roger

# PermitRootLogin should also be set to 'yes' if it isn't
PermitRootLogin yes

Setting up secondary machine

This section will teach you how to pull from a remote tree, and set up your machine to use the distfiles and packages that your remote tree has (Could be a tree from a local network).

We will assume that you are pulling the tree from your local network and that you are accessing the tree via ssh.

We will assume that this is a fresh install and that you don't have any previous portage trees or setting tweaks.

/etc/portage/make.conf

What we will first due is edit our make.conf so that it pulls resources from the correct locations.

We will assume that Machine A's IP is 192.168.1.100, SSH port is 8902, that we are logging in as the root user, and that the portage tree is in it's default location of /usr/portage on that machine.

Open up the make.conf file for Machine B and add the following:

# This is where it will try to pull the tree the first time it syncs. After this it will just use w/e the git tree origin is set to
SYNC="ssh://192.168.1.100:8902/usr/portage"

# This sets where it will try to find source files (distfiles). We will try to pull from Machine A first, but if Machine A doesn't have the files we need, it will go to Gentoo to get them
GENTOO_MIRRORS="http://192.168.1.100/funtoo http://distfiles.gentoo.org"

# This sets where it will try to find binary packages (We are using an http server to make it more convenient for us because we don't need "security" to just download precompiled packages)
PORTAGE_BINHOST="http://192.168.1.100/funtoo/packages"

If you already have a tree

If you already have a tree and would like to delete it and start fresh, you can just delete the .git folder inside the /usr/portage directory and the next time you run emerge --sync, portage will tell you that it isn't a git repository, and it will wipe the contents and sync the new tree.

Conclusion

That is basically it.