Difference between pages "IPv4 calculations" and "How to set up a binary package server"

From Funtoo
(Difference between pages)
Jump to: navigation, search
(... transposed to computers)
 
 
Line 1: Line 1:
WARNING: Work in progress. Do not edit this article unless you are the original author.
+
=== Why would you want a binary package server? ===
  
 +
Even though Funtoo is a source-based distribution, it does not mean that compiling from source every single time is beneficial. For instance, if you have multiple computers in your home network, and they all need upgrades, it wouldn't be convenient to  compile the same package over and over again on each machine. Rather you could make your fastest computer compile the packages, and then just share those packages over the network or via USB drive (sneakernet).
  
= Refresh on TCP/IP model =  
+
=== Recommendations ===
  
When the ARPANet (a packet oriented network) was born in those good old seventies, engineers had to solve the problem of making computers being able to exchange packets of information over the network and they invented in 1974 something you are now using to view this page: TCP/IP! TCP/IP is a collection of various network protocols, being organized as a stack. Just like your boss does not do everything in the company and delegates at lower levels which in turn delegates at an even more lower level, no protocol in the TCP/IP suite takes all responsibilities, they are working together in a hierarchical and cooperative manner.  A level of the TCP/IP stack knows what its immediate lower subordinate can do for it and whatever it will do will be done the right way and will not worry about the manner the job will be done.  Also the only problem for a given level of the stack is to fulfill its own duties and deliver the service requested  by the upper layer, it does not have to worry about the ultimate goal of what upper levels do.
+
In order to make the most use of this, I recommend the following:
 
+
<illustration goes here TCP/IP model>
+
  
The above illustration sounds horribly familiar : yes, it is sounds like this good old OSI model. Indeed it is a tailored view of the original OSI model and it works the exact same way.
+
# Set all USE flags on all machines to be the same (Lowest Common Denominator)
 +
# Compile for a generic subarch. (For example, if all your machines are amd64, compile for amd64 not no specific ABIs like core2, etc.)
  
== The postal mail service ==
+
=== Configuring the make.conf for host machine ===
If you are not super familiar with OSI or TCP/IP models let's take an analogy with the national postal service because it works in a similar way. Suppose you want to send a parcel to your cousin  located in another in a town in the same country using the the national postal service. Both of you are using PO boxes located your respective postal outlets.  So basically you :
+
  
# take a box and pack what you want to send inside
+
We first need to enable automatic binary packaging on the host. This means that when portage compiles the package,  
# write the address of the destinee on the box like "to : Mr A, PO Box 123, 99 Street ABC, Oldcity.... from: Mr B, PO Box 456, 45 Street DEF, Newcity...."
+
it will put it into a <tt>.tbz2</tt> file right after. To do this we will add <tt>buildpkg</tt> to the <tt>FEATURES var</tt> in <tt>/etc/make.conf</tt>.
# give the box to the clerk at your post office outlet. The clerk will ask you for the level of service you want (standard or registered mail)
+
  
Once you drop the parcel to the postal service, you know they will route it to your cousin's PO box. You don't know exactly how it will be transported from one postal facility to another through the country, you only know that it will travel through the postal service networks then eventually reach the addressee. If you opted for the registered mail service additional care will be brought to ensure your parcel won't be lost in the middle.  
+
My <tt>make.conf</tt> looks like this::
 +
{{File
 +
|/etc/portage/make.conf|<pre>
 +
CFLAGS="-mtune=generic -O2 -pipe"
 +
CXXFLAGS="${CFLAGS}"
 +
USE="mmx sse sse2"
 +
MAKEOPTS="-j9"
 +
FEATURES="buildpkg userfetch"
 +
EMERGE_DEFAULT_OPTS="--quiet-build=y"
 +
VIDEO_CARDS="nvidia nouveau intel"
 +
INPUT_DEVICES="evdev synaptics"
 +
LINGUAS="en en_US"
 +
</pre>}}
  
A few days later you receive a greeting card from your cousin with an address like "to : Mr B, PO Box 456, 45 Street DEF, Newcity.... from: Mr A, PO Box 123, 99 Street ABC, Oldcity...."
+
Now that <tt>FEATURES="buildpkg"</tt> is set, from now on, every time you build a new package, it will automatically package it and you can find the <tt>.tbz2</tt> file in <tt>/usr/portage/packages</tt> by default. You can change the destination by using the <tt>PKGDIR</tt> setting in <tt>make.conf</tt>.
  
== ... transposed to computers ==
+
=== What will you need? ===
  
For a computer this is more or less the same. An application A1 residing on a computer C1 wants to send some data to another application A2 located on a remote computer C2. A1 knows the "PO Box" (port number in technical terms) of A2 which is P2, and of course the computer where the application A2 resides which is C2.  
+
Besides the above advice, you will need one of the protocols that can be used for sharing (Like FTP, HTTP, or ssh).I recommend HTTP, so we will be using lighttpd server since it's small, quick, and efficient for these basic tasks.
  
# A1 solicits the TCP/IP stack of C1  and talks to its ''Transport layer'' (the only one A1 can interact with) : "here is the '''data''' to send to computer C2, port P2 using a reliable transport".
+
=== Setting up the host machine ===
# The ''Transport layer'' adds ''Transport layer header'' in front of the data supplied by A1. This header includes several informations depending on whether a reliable transport (TCP) is asked or not (UDP). The result of concatenating the data and the transport layer's header is called an ''Layer 4 PDU'' shortened as '''L4PDU''' (PDU standing for ''Protocol Data Unit'').
+
# The ''Transport layer'' then gives the L4PDU to the ''Internet layer'' : "here is a data '''segment''' to send to computer C2".
+
# The ''Internet layer'' will add its own header (differs between IPv4 and IPv6 but basically it includes C1 and C2's network address) to the L4PDU thus giving an L3PDU.
+
# Before giving the packet (or L3PDU) to the lower layers, the C1's TCP/IP stack has to determine where the packet has to be sent. Two possibilities: either the packet is destined to a computer on a network on which C1 is directly connected either the packet is destined to a computer located on a network on which C1 is not directly connected. In that latter case the packet has to be forwarded to a relay that interconnects the network where C1 resides with one or more other networks. That relay (or '''gateway''' in technicals terms) will on its turn decide where the packet received from C1 has to be sent : either the gateway is connected to the same network where C2 resides and the gateway will directly forward the packet to C2, either it has to be forwarded to another gateway and so on.  No magic here! You apply the same principles when  you drive your car looking for routes to follow to reach your final destination : if a sign at at the crossroad explicitly points toward your final destination you follow that specific road else you follow the road indicated by "all destinations", hopping to encounter a more precise route at the next crossroad. For a computer the "all destinations" road is more : send the packet to a '''gateway''' (to another network).
+
# The C1's routing table is simply a set of rules like ''"All packets destined to computers located on network N1 have to be sent via the network interface NIC1 of C1"'' or ''"all packets destined to computers located on network N2 have to be sent via the network interface NIC2 of C2"'' or even ''"for all other destinations use the gateway GW1 reachable via the the network interface NIC1 of C1"''. How the C1's TCP/IP stacks knows what computer is located in what network? Shhh the explanation will be given in following paragraphs, this is ''precisely'' where netmasks intervenes.  for now just assume that C1 knows which one of its network interface it has to use to send the packet over the network.
+
  
= Classful and classless networks =
+
Install lighttpd
  
 +
<console>
 +
# ##i##emerge lighttpd
 +
</console>
  
Who would ''ever'' need millions of addresses afterall?  So in theory with those 32 bits we can have around 4 billions of computers within that network and arbitrarily retain that the very first connected computer must be given the number "0", the second one "1", the third one "2" and so on until we exhaust the address pool at number 4294967295 giving no more than 4294967296 (2^32) computers on that network because no number can be a duplicate.
+
After you install lighttpd, we will set a symlink in the htdocs folder to point to your packages folder (By default it's <tt>/usr/portage/packages</tt>):
  
 +
<console>
 +
# ##i##cd /var/www/localhost/htdocs
 +
# ##i##mkdir funtoo && cd funtoo
 +
# ##i##ln -s /usr/portage portage
 +
# ##i##ln -s /usr/portage/distfiles distfiles
 +
# ##i##ln -s /usr/portage/packages packages
 +
</console>
  
Those addresses follows the thereafter logic:
+
Now that that's set, the link to your packages will be <tt>http://ip-of-computer/funtoo/packages</tt>.
  
{| class="wikitable"
+
If the lighttpd server were on now and you navigated to the above address, you would get a 404 error because lighttpd by default doesn't show the directory listing. Add the following line to your /etc/lighttpd/lighttpd.conf so that if you request a page that doesn't contain an index page, it will just show the directory contents:
|-
+
| colspan="2" | '''32 bits (fixed length)'''
+
|-
+
| '''Network''' part (variable length of N bits ) || '''Host''' part (length : 32 - N bits)
+
|}
+
  
* The network address : this part is uniquely assigned amongst all of the organizations in the world (i.e. No one in the world can hold the same network part) 
+
<console>
* The host address :  unique within a given network part
+
# Activate directory listings globally
 +
dir-listing.activate = "enable"
 +
</console>
  
So in theory we can have something like this (remember the network nature is not to be unique, it hs to be be a collection of networks  :
+
Add the service to your start up process and start it up:
  
* Network 1 Host 1
+
<console>
*
+
# ##i##rc-config add lighttpd default
 +
# ##i##rc
 +
</console>
  
 +
Your host machine is now ready to serve out binaries.
  
Just like your birthday cake is divided in more or less smaller parts depending on how guests' appetite, the IPv4 address space has also been divided into more or less smaller parts just because organizations needs more or less computers on their networks. How to make this possible? Simply by dedicating a variable number of bits to the network part! Do you see the consequence? An IPv4 address being '''always''' 32 bits wide, the more bits you dedicate to the network part the lesser you have for the host part and vice-versa, this is a tradeoff, always. Basically, having more bits in :
+
=== Setting up the client machine ===
* the network part : means more networks possible at the cost of having less hosts per network 
+
* the host part : means less networks but more hosts per network
+
  
It might sounds a bit abstract but let's take an example : imagine we dedicate only 8 bits for the network part and the remaining 24 for the hosts part. What happens?  First if we only
+
Basically the client machine will be receiving packages from a server that might not necessarily have the exact same USE flags that the client machine has. Even thought we set all the USE flags on all machines to the Lowest Common Denominator, sometimes things deviate. In order to solve this problem, we will make portage not respect USE flags. Meaning that if there are different USE flags on machines, it will still install those packages.
  
+
To do this we need to add the <tt>--binpkg-respect-use=n</tt> to the end of emerge.
  
Is the network part assigned by each organization to itself? Of course not! Assignment are coordinated at the worldwide level by what we call Regional Internet Registries or RIRs which, in turn, can delegate assignments to third-parties located within their geographic jurisdiction. Those latter are called Local Internet Registries or LIRs (the system is detailed in RFC 7020). All of those RIRs are themselves put under the responsibility of now now well-known Internet Assigned Numbers Authority or [http://www.iana.org IANA]. As of 2014 five RIR exists :
+
Before we use the above option, we also need to tell emerge to not only search for ebuilds, but also to search for packages as well. In order for it to search for packages, emerge will need to know where to look (the host) and set an option on portage to actually let it know to look for binaries.
+
 
* ARIN (American Registry for Internet Numbers) : covers North America
+
We do this by adding the <tt>--getbinpkg</tt> option to the end of emerge, and setting the <tt>PACKAGE_BINHOST</tt> variable in <tt>/etc/make.conf</tt>.
* LACNIC (Latin America and Caribbean Network Information Centre): covers South America and the Caribbean
+
 
* RIPE-NCC (Réseaux IP Européens / or RIPE Network Coordination Centre): covers Europe, Russia and middle east
+
This is how my client's <tt>/etc/make.conf</tt> looks:
* Afrinic (Africa Network Information Center) : covers the whole Africa
+
{{File
* APNIC (Asian and Pacific Network Information Centre) : covers oceania and far east.
+
|/etc/portage/make.conf|<pre>
 +
CFLAGS="-mtune=generic -O2 -pipe"
 +
CXXFLAGS="${CFLAGS}"
 +
USE="mmx sse sse2"
 +
MAKEOPTS="-j5"
 +
FEATURES="buildpkg userfetch getbinpkg"
 +
EMERGE_DEFAULT_OPTS="--quiet-build=y --binpkg-respect-use=n"
 +
VIDEO_CARDS="nvidia intel"
 +
INPUT_DEVICES="evdev synaptics"
 +
LINGUAS="en en_US"
 +
PORTAGE_BINHOST="http://ip-address-of-build-server/gentoo/packages"
 +
</pre>}}
 +
 
 +
After you set the package host (via <tt>PORTAGE_BINHOST</tt>), set the <tt>getbinpkg</tt> option in <tt>FEATURES</tt>, and add <tt>--binpkg-respect-use=n</tt> to <tt>EMERGE_DEFAULT_OPTS</tt>, you are then set. Just do a emerge <tt>--sync</tt>, and <tt>emerge -uDav world</tt> or whatever you want. You should now see all your
 +
packages being pulled from the host server.
 +
 
 +
Normally you could just do a
 +
<console>
 +
# ##i##emerge <package> --binpkg-respect-use=n
 +
</console>
 +
 
 +
But you will have to do that each time you want to install a binary with different use flags, so in order to automate this, we add it to the <tt>EMERGE_DEFAULT_OPTS</tt>.
 +
 
 +
[[Category:HOWTO]]

Revision as of 04:25, 16 January 2014

Contents

Why would you want a binary package server?

Even though Funtoo is a source-based distribution, it does not mean that compiling from source every single time is beneficial. For instance, if you have multiple computers in your home network, and they all need upgrades, it wouldn't be convenient to compile the same package over and over again on each machine. Rather you could make your fastest computer compile the packages, and then just share those packages over the network or via USB drive (sneakernet).

Recommendations

In order to make the most use of this, I recommend the following:

  1. Set all USE flags on all machines to be the same (Lowest Common Denominator)
  2. Compile for a generic subarch. (For example, if all your machines are amd64, compile for amd64 not no specific ABIs like core2, etc.)

Configuring the make.conf for host machine

We first need to enable automatic binary packaging on the host. This means that when portage compiles the package, it will put it into a .tbz2 file right after. To do this we will add buildpkg to the FEATURES var in /etc/make.conf.

My make.conf looks like this::

CFLAGS="-mtune=generic -O2 -pipe"
CXXFLAGS="${CFLAGS}"
USE="mmx sse sse2"
MAKEOPTS="-j9"
FEATURES="buildpkg userfetch"
EMERGE_DEFAULT_OPTS="--quiet-build=y"
VIDEO_CARDS="nvidia nouveau intel"
INPUT_DEVICES="evdev synaptics"
LINGUAS="en en_US"

Now that FEATURES="buildpkg" is set, from now on, every time you build a new package, it will automatically package it and you can find the .tbz2 file in /usr/portage/packages by default. You can change the destination by using the PKGDIR setting in make.conf.

What will you need?

Besides the above advice, you will need one of the protocols that can be used for sharing (Like FTP, HTTP, or ssh).I recommend HTTP, so we will be using lighttpd server since it's small, quick, and efficient for these basic tasks.

Setting up the host machine

Install lighttpd

# emerge lighttpd

After you install lighttpd, we will set a symlink in the htdocs folder to point to your packages folder (By default it's /usr/portage/packages):

# cd /var/www/localhost/htdocs
# mkdir funtoo && cd funtoo
# ln -s /usr/portage portage
# ln -s /usr/portage/distfiles distfiles
# ln -s /usr/portage/packages packages

Now that that's set, the link to your packages will be http://ip-of-computer/funtoo/packages.

If the lighttpd server were on now and you navigated to the above address, you would get a 404 error because lighttpd by default doesn't show the directory listing. Add the following line to your /etc/lighttpd/lighttpd.conf so that if you request a page that doesn't contain an index page, it will just show the directory contents:

# Activate directory listings globally
dir-listing.activate = "enable"

Add the service to your start up process and start it up:

# rc-config add lighttpd default
# rc

Your host machine is now ready to serve out binaries.

Setting up the client machine

Basically the client machine will be receiving packages from a server that might not necessarily have the exact same USE flags that the client machine has. Even thought we set all the USE flags on all machines to the Lowest Common Denominator, sometimes things deviate. In order to solve this problem, we will make portage not respect USE flags. Meaning that if there are different USE flags on machines, it will still install those packages.

To do this we need to add the --binpkg-respect-use=n to the end of emerge.

Before we use the above option, we also need to tell emerge to not only search for ebuilds, but also to search for packages as well. In order for it to search for packages, emerge will need to know where to look (the host) and set an option on portage to actually let it know to look for binaries.

We do this by adding the --getbinpkg option to the end of emerge, and setting the PACKAGE_BINHOST variable in /etc/make.conf.

This is how my client's /etc/make.conf looks:

CFLAGS="-mtune=generic -O2 -pipe"
CXXFLAGS="${CFLAGS}"
USE="mmx sse sse2"
MAKEOPTS="-j5"
FEATURES="buildpkg userfetch getbinpkg"
EMERGE_DEFAULT_OPTS="--quiet-build=y --binpkg-respect-use=n"
VIDEO_CARDS="nvidia intel"
INPUT_DEVICES="evdev synaptics"
LINGUAS="en en_US"
PORTAGE_BINHOST="http://ip-address-of-build-server/gentoo/packages"

After you set the package host (via PORTAGE_BINHOST), set the getbinpkg option in FEATURES, and add --binpkg-respect-use=n to EMERGE_DEFAULT_OPTS, you are then set. Just do a emerge --sync, and emerge -uDav world or whatever you want. You should now see all your packages being pulled from the host server.

Normally you could just do a

# emerge <package> --binpkg-respect-use=n

But you will have to do that each time you want to install a binary with different use flags, so in order to automate this, we add it to the EMERGE_DEFAULT_OPTS.