Difference between pages "Hostname" and "How to set up a binary package server"

(Difference between pages)
(Diving deeper)
 
m (Setting up the client machine)
 
Line 1: Line 1:
w.i.p
+
=== Why would you want a binary package server? ===
==Introduction==
+
 
A hostname is a unique name created to identify a machine on a network. In computer networking, a hostname  is a label that is assigned to a device connected to a computer network and that is used to identify the device in various forms of electronic communication such as the World Wide Web, e-mail or Usenet. Hostnames may be simple names consisting of a single word or phrase, or they may be structured.
+
Even though Funtoo is a source-based distribution, it does not mean that compiling from source every single time is beneficial. For instance, if you have multiple computers in your home network, and they all need upgrades, it wouldn't be convenient to compile the same package over and over again on each machine. Rather you could make your fastest computer compile the packages, and then just share those packages over the network or via USB drive (sneakernet).
==Configuration==
+
 
In Funtoo Linux <code>/etc/conf.d/hostname</code> is the only configuration file for setting a hostname. In OpenRC framework <code>/etc/conf.d/foo</code> is the configuration file for a corresponding Init script <code>/etc/init.d/foo</code>.  With the case of hostname, default value in <code>/etc/conf.d/hostname</code> is set to ''localhost'', means when system boots and OpenRC's <code>/etc/init.d/hostname</code> script started a hostname getting only ''localhost'' name. How it looks? In your shell promt this will look in following way, an example for root:
+
=== Recommendations ===
 +
 
 +
In order to make the most use of this, I recommend the following:
 +
 
 +
# Set all USE flags on all machines to be the same (Lowest Common Denominator)
 +
# Compile for a generic subarch. (For example, if all your machines are amd64, compile for amd64 not no specific ABIs like core2, etc.)
 +
 
 +
=== Configuring the make.conf for host machine ===
 +
 
 +
We first need to enable automatic binary packaging on the host. This means that when portage compiles the package,
 +
it will put it into a <tt>.tbz2</tt> file right after. To do this we will add <tt>buildpkg</tt> to the <tt>FEATURES var</tt> in <tt>/etc/make.conf</tt>.
 +
 
 +
My <tt>make.conf</tt> looks like this::
 +
 
 +
<pre>
 +
CFLAGS="-mtune=generic -O2 -pipe"
 +
CXXFLAGS="${CFLAGS}"
 +
USE="mmx sse sse2"
 +
MAKEOPTS="-j9"
 +
FEATURES="buildpkg userfetch"
 +
EMERGE_DEFAULT_OPTS="--quiet-build=y"
 +
VIDEO_CARDS="nvidia nouveau intel"
 +
INPUT_DEVICES="evdev synaptics"
 +
LINGUAS="en en_US"
 +
</pre>
 +
 
 +
Now that <tt>FEATURES="buildpkg"</tt> is set, from now on, every time you build a new package, it will automatically package it and you can find the <tt>.tbz2</tt> file in <tt>/usr/portage/packages</tt> by default. You can change the destination by using the <tt>PKGDIR</tt> setting in <tt>make.conf</tt>.
 +
 
 +
=== What will you need? ===
 +
 
 +
Besides the above advice, you will need one of the protocols that can be used for sharing (Like FTP, HTTP, or ssh).I recommend HTTP, so we will be using lighttpd server since it's small, quick, and efficient for these basic tasks.
 +
 
 +
=== Setting up the host machine ===
 +
 
 +
Install lighttpd
 +
 
 
<console>
 
<console>
localhost ~ # ##i## Hello :)
+
# ##i##emerge lighttpd
 
</console>
 
</console>
Let's play a bit with a configuration. Open <code>/etc/conf.d/hostname</code> with your favorite editor and set a hostname of your choice.  Below, I will use a real examples  from one of my working test boxes.
+
 
 +
After you install lighttpd, we will set a symlink in the htdocs folder to point to your packages folder (By default it's <tt>/usr/portage/packages</tt>):
 +
 
 
<console>
 
<console>
localhost ~ # ##i## nano /etc/conf.d/hostname
+
# ##i##cd /var/www/localhost/htdocs
 +
# ##i##mkdir funtoo && cd funtoo
 +
# ##i##ln -s /usr/portage portage
 +
# ##i##ln -s /usr/portage/distfiles distfiles
 +
# ##i##ln -s /usr/portage/packages packages
 
</console>
 
</console>
Let's set it to hostname="oleg-stable.host.funtoo.org". Save the file and restart  a hostname service:
+
 
 +
Now that that's set, the link to your packages will be <tt>http://ip-of-computer/funtoo/packages</tt>.
 +
 
 +
If the lighttpd server were on now and you navigated to the above address, you would get a 404 error because lighttpd by default doesn't show the directory listing. Add the following line to your /etc/lighttpd/lighttpd.conf so that if you request a page that doesn't contain an index page, it will just show the directory contents:
 +
 
 
<console>
 
<console>
localhost ~ # ##i## service hostname restart
+
# Activate directory listings globally
 +
dir-listing.activate = "enable"
 
</console>
 
</console>
Now, let's examine our changes, after a restarting a hostname
+
 
 +
Add the service to your start up process and start it up:
 +
 
 
<console>
 
<console>
oleg-stable ~ # ##i## Hello :)
+
# ##i##rc-config add lighttpd default
 +
# ##i##rc
 
</console>
 
</console>
== Diving deeper==
+
 
Notice, that in above output we seeing a shortened hostname and not a FQDN (Fully Qualified Domain Name). Don't be frustrated. This is  how  default bash promt <code>PS1</code> set. To get nice promts, please, follow http://www.funtoo.org/Prompt_Magic
+
Your host machine is now ready to serve out binaries.
Another way to test our settings is using a '''hostname''' command. Here we will show only  some of it's features. Let's try to execute '''hostname''' command:
+
 
<console>
+
=== Setting up the client machine ===
oleg-stable ~ # ##i## hostname
+
 
oleg-stable.host.funtoo.org
+
Basically the client machine will be receiving packages from a server that might not necessarily have the exact same USE flags that the client machine has. Even thought we set all the USE flags on all machines to the Lowest Common Denominator, sometimes things deviate. In order to solve this problem, we will make portage not respect USE flags. Meaning that if there are different USE flags on machines, it will still install those packages.
</console>
+
 
Now we see our fully qualified domain name hostname just how we configured it in <code>/etc/conf.d/hostname</code> in above paragraph. To get a short hostname we need to set '''-s ''' (short) argument to hostname command.  
+
To do this we need to add the <tt>--binpkg-respect-use=n</tt> to the end of emerge.
<console>
+
 
oleg-stable ~ # ##i## hostname -s
+
Before we use the above option, we also need to tell emerge to not only search for ebuilds, but also to search for packages as well. In order for it to search for packages, emerge will need to know where to look (the host) and set an option on portage to actually let it know to look for binaries.
oleg-stable
+
 
</console>
+
We do this by adding the <tt>--getbinpkg</tt> option to the end of emerge, and setting the <tt>PACKAGE_BINHOST</tt> variable in <tt>/etc/make.conf</tt>.
Good! Hostname offers more then just displaying a system host name but can also set one. Let's try:
+
 
 +
This is how my client's <tt>/etc/make.conf</tt> looks:
 +
 
 +
<pre>
 +
CFLAGS="-mtune=generic -O2 -pipe"
 +
CXXFLAGS="${CFLAGS}"
 +
USE="mmx sse sse2"
 +
MAKEOPTS="-j5"
 +
FEATURES="buildpkg userfetch getbinpkg"
 +
EMERGE_DEFAULT_OPTS="--quiet-build=y --binpkg-respect-use=n"
 +
VIDEO_CARDS="nvidia intel"
 +
INPUT_DEVICES="evdev synaptics"
 +
LINGUAS="en en_US"
 +
PORTAGE_BINHOST="http://ip-address-of-build-server/funtoo/packages"
 +
</pre>
 +
 
 +
After you set the package host (via <tt>PORTAGE_BINHOST</tt>), set the <tt>getbinpkg</tt> option in <tt>FEATURES</tt>, and add <tt>--binpkg-respect-use=n</tt> to <tt>EMERGE_DEFAULT_OPTS</tt>, you are then set. Just do a emerge <tt>--sync</tt>, and <tt>emerge -uDav world</tt> or whatever you want. You should now see all your
 +
packages being pulled from the host server.
 +
 
 +
Normally you could just do a
 
<console>
 
<console>
oleg-stable ~ # ##i## hostname foo.bar.baz
+
# ##i##emerge <package> --binpkg-respect-use=n
oleg-stable ~ # ##i## hostname
+
foo.bar.baz
+
 
</console>
 
</console>
As you can see, we changed a hostname on-the-fly. This is not recommended way.
 
{{fancywarning|Please, notice that using '''hostname''' command to configure will work temporary for a current session and will be reverted back to a value set in <code>/etc/conf.d/hostname</code> file with next system restart.}}
 
  
Now that we got a brief description of a hostname and basic configuration steps, its time to reflect another important case which is directly related to a Funtoo Linux hostname generation, a hosts.
+
But you will have to do that each time you want to install a binary with different use flags, so in order to automate this, we add it to the <tt>EMERGE_DEFAULT_OPTS</tt>.
  
==Hosts case==
+
[[Category:HOWTO]]
As per man page <code>hosts</code> stands for static table lookup for hostnames and it's configuration file is <code>/etc/hosts</code>. Here is how it looks
+
{{file|name=/etc/hosts|body=
+
# Auto-generated hostname. Please do not remove this comment.
+
127.0.0.1      oleg-stable.host.funtoo.org oleg-stable localhost localhost.localdomain
+
::1            oleg-stable.host.funtoo.org oleg-stable localhost localhost.localdomain
+
}}
+
As you can see it has entries from our <code>/etc/conf.d/hostname</code>. As you may have guessed, in Funtoo Linux <code>/etc/hosts</code> file entries are auto-generated, when OpenRC hostname service starts. Previously, it is used to edit <code>/etc/hosts</code> manually. In Funtoo Linux there is no such need.
+

Latest revision as of 11:08, February 23, 2015

Why would you want a binary package server?

Even though Funtoo is a source-based distribution, it does not mean that compiling from source every single time is beneficial. For instance, if you have multiple computers in your home network, and they all need upgrades, it wouldn't be convenient to compile the same package over and over again on each machine. Rather you could make your fastest computer compile the packages, and then just share those packages over the network or via USB drive (sneakernet).

Recommendations

In order to make the most use of this, I recommend the following:

  1. Set all USE flags on all machines to be the same (Lowest Common Denominator)
  2. Compile for a generic subarch. (For example, if all your machines are amd64, compile for amd64 not no specific ABIs like core2, etc.)

Configuring the make.conf for host machine

We first need to enable automatic binary packaging on the host. This means that when portage compiles the package, it will put it into a .tbz2 file right after. To do this we will add buildpkg to the FEATURES var in /etc/make.conf.

My make.conf looks like this::

CFLAGS="-mtune=generic -O2 -pipe"
CXXFLAGS="${CFLAGS}"
USE="mmx sse sse2"
MAKEOPTS="-j9"
FEATURES="buildpkg userfetch"
EMERGE_DEFAULT_OPTS="--quiet-build=y"
VIDEO_CARDS="nvidia nouveau intel"
INPUT_DEVICES="evdev synaptics"
LINGUAS="en en_US"

Now that FEATURES="buildpkg" is set, from now on, every time you build a new package, it will automatically package it and you can find the .tbz2 file in /usr/portage/packages by default. You can change the destination by using the PKGDIR setting in make.conf.

What will you need?

Besides the above advice, you will need one of the protocols that can be used for sharing (Like FTP, HTTP, or ssh).I recommend HTTP, so we will be using lighttpd server since it's small, quick, and efficient for these basic tasks.

Setting up the host machine

Install lighttpd

# emerge lighttpd

After you install lighttpd, we will set a symlink in the htdocs folder to point to your packages folder (By default it's /usr/portage/packages):

# cd /var/www/localhost/htdocs
# mkdir funtoo && cd funtoo
# ln -s /usr/portage portage
# ln -s /usr/portage/distfiles distfiles
# ln -s /usr/portage/packages packages

Now that that's set, the link to your packages will be http://ip-of-computer/funtoo/packages.

If the lighttpd server were on now and you navigated to the above address, you would get a 404 error because lighttpd by default doesn't show the directory listing. Add the following line to your /etc/lighttpd/lighttpd.conf so that if you request a page that doesn't contain an index page, it will just show the directory contents:

# Activate directory listings globally
dir-listing.activate = "enable"

Add the service to your start up process and start it up:

# rc-config add lighttpd default
# rc

Your host machine is now ready to serve out binaries.

Setting up the client machine

Basically the client machine will be receiving packages from a server that might not necessarily have the exact same USE flags that the client machine has. Even thought we set all the USE flags on all machines to the Lowest Common Denominator, sometimes things deviate. In order to solve this problem, we will make portage not respect USE flags. Meaning that if there are different USE flags on machines, it will still install those packages.

To do this we need to add the --binpkg-respect-use=n to the end of emerge.

Before we use the above option, we also need to tell emerge to not only search for ebuilds, but also to search for packages as well. In order for it to search for packages, emerge will need to know where to look (the host) and set an option on portage to actually let it know to look for binaries.

We do this by adding the --getbinpkg option to the end of emerge, and setting the PACKAGE_BINHOST variable in /etc/make.conf.

This is how my client's /etc/make.conf looks:

CFLAGS="-mtune=generic -O2 -pipe"
CXXFLAGS="${CFLAGS}"
USE="mmx sse sse2"
MAKEOPTS="-j5"
FEATURES="buildpkg userfetch getbinpkg"
EMERGE_DEFAULT_OPTS="--quiet-build=y --binpkg-respect-use=n"
VIDEO_CARDS="nvidia intel"
INPUT_DEVICES="evdev synaptics"
LINGUAS="en en_US"
PORTAGE_BINHOST="http://ip-address-of-build-server/funtoo/packages"

After you set the package host (via PORTAGE_BINHOST), set the getbinpkg option in FEATURES, and add --binpkg-respect-use=n to EMERGE_DEFAULT_OPTS, you are then set. Just do a emerge --sync, and emerge -uDav world or whatever you want. You should now see all your packages being pulled from the host server.

Normally you could just do a

# emerge <package> --binpkg-respect-use=n

But you will have to do that each time you want to install a binary with different use flags, so in order to automate this, we add it to the EMERGE_DEFAULT_OPTS.