Difference between pages "Broken Traffic Control" and "How Metro Works"

From Funtoo
(Difference between pages)
Jump to: navigation, search
 
(Introduction)
 
Line 1: Line 1:
The following traffic control examples '''should''' work, but are broken in older implementations of Linux traffic control, as tested by Daniel Robbins.
+
= Introduction =
  
{{fancyimportant|Many enterprise kernels do not have updated traffic control code. RHEL5 kernels are particularly bad in this area.}}
+
[[Metro]] is a somewhat complex set of shell (specifically bash) and python scripts and various files that automagically creates stages 1/2/3 tarballs used to emerge your [[Funtoo Linux]] system.
 +
Of course, [[Metro]] needs a seed to start its work and will recompile things like the so well known <tt>gcc</tt> and <tt>glibc</tt>.
 +
A 'seed' is either a pre-built stage3 archive available on any Funtoo Linux [http://docs.funtoo.org/wiki/Download mirror] or a snapshot of your current [[Funtoo Linux]] installation.
  
=== Broken in RHEL5: Prioritizing Outgoing UDP packets ===
+
= Arborescence description =  
  
These rules should be able to give outgoing UDP traffic a higher priority than other outgoing traffic, which could reduce the latency of outgoing UDP packets if your system (or ideally, your Linux router, which is the ideal point to use these shaping rules) is sending a lot of data all at once to a remote host.  
+
== Overview ==
 +
Don't search through your bin directories, everything lies in '''/usr/lib/metro'''. An unfolded view as of Metro revision 1.5.1 is this one :
  
However, currently, these rules do not work.
+
<pre>
 +
├── BUGS
 +
├── ChangeLog
 +
├── TODO
 +
├── etc
 +
│   ├── builds
 +
│   │   ├── funtoo
 +
│   │   │   ├── build.conf
 +
│   │   │   └── multi-targets
 +
│   │   │      ├── freshen
 +
│   │   │      ├── full
 +
│   │   │      ├── quick
 +
│   │   │      ├── stage1
 +
│   │   │      └── strategy
 +
│   │   │          └── seed
 +
│   │   │              ├── stage1
 +
│   │   │              └── stage3
 +
│   │   ├── gentoo
 +
│   │   │   ├── build.conf
 +
│   │   │   └── multi-targets
 +
│   │   │      ├── freshen
 +
│   │   │      ├── full
 +
│   │   │      ├── quick
 +
│   │   │      ├── stage1
 +
│   │   │      └── strategy
 +
│   │   │          └── seed
 +
│   │   │              ├── stage1
 +
│   │   │              └── stage3
 +
│   │   ├── hollow
 +
│   │   │   ├── build.conf
 +
│   │   │   └── multi-targets
 +
│   │   │      ├── freshen
 +
│   │   │      ├── full
 +
│   │   │      ├── quick
 +
│   │   │      ├── stage1
 +
│   │   │      └── strategy
 +
│   │   │          └── seed
 +
│   │   │              ├── stage1
 +
│   │   │              └── stage3
 +
│   │   └── ~funtoo
 +
│   │      ├── build.conf
 +
│   │      └── multi-targets
 +
│   │          ├── freshen
 +
│   │          ├── full
 +
│   │          ├── quick
 +
│   │          ├── stage1
 +
│   │          └── strategy
 +
│   │              └── seed
 +
│   │                  ├── stage1
 +
│   │                  └── stage3
 +
│   ├── fslayouts
 +
│   │   └── funtoo
 +
│   │      ├── layout.conf
 +
│   │      └── type
 +
│   │          ├── image
 +
│   │          └── snapshot
 +
│   └── metro.conf
 +
├── features
 +
│   └── probe.conf
 +
├── metro
 +
├── misc
 +
│   └── locale.gen
 +
├── modules
 +
│   ├── catalyst_support.py
 +
│   ├── catalyst_support.pyo
 +
│   ├── flexdata.py
 +
│   ├── flexdata.pyo
 +
│   ├── flexdata2.py
 +
│   ├── flexdata3.py
 +
│   ├── flexdata4.py
 +
│   ├── targets.py
 +
│   └── targets.pyo
 +
├── scripts
 +
│   ├── build.sh
 +
│   ├── cleaner.sh
 +
│   ├── dlstats.sh
 +
│   └── ezbuild.sh
 +
├── subarch
 +
│   ├── amd64
 +
│   │   ├── 2008.spec
 +
│   │   └── 2009.spec
 +
│   ├── amd64.spec
 +
│   ├── athlon-xp.spec
 +
│   ├── core2.spec
 +
│   ├── core2_32.spec
 +
│   ├── generic32.spec
 +
│   ├── generic64.spec
 +
│   ├── i686
 +
│   │   ├── 2008.spec
 +
│   │   └── 2009.spec
 +
│   ├── i686.spec
 +
│   ├── native32.spec
 +
│   ├── native64.spec
 +
│   ├── opteron-sse3.spec
 +
│   ├── opteron.spec
 +
│   ├── opteron_32.spec
 +
│   ├── pentium4.spec
 +
│   ├── phenom.spec
 +
│   ├── x86.spec
 +
│   ├── xen-nocona.spec
 +
│   └── xen-prescott.spec
 +
└── targets
 +
    └── gentoo
 +
        ├── openvz.spec
 +
        ├── snapshot
 +
        │   ├── common.spec
 +
        │   ├── global.spec
 +
        │   ├── source
 +
        │   │   ├── git
 +
        │   │   └── rsync
 +
        │   └── type
 +
        │      ├── dead
 +
        │      └── live
 +
        ├── snapshot.spec
 +
        ├── stage
 +
        │   ├── capture
 +
        │   │   ├── ami.spec
 +
        │   │   └── tar.spec
 +
        │   ├── common.spec
 +
        │   ├── files.spec
 +
        │   ├── stage3-derivative.spec
 +
        │   ├── stage3-generator.spec
 +
        │   └── steps.spec
 +
        ├── stage1
 +
        │   └── strategy
 +
        │      ├── local
 +
        │      └── remote
 +
        ├── stage1.spec
 +
        ├── stage2
 +
        │   └── strategy
 +
        │      ├── local
 +
        │      │   ├── stage1
 +
        │      │   └── stage3
 +
        │      └── remote
 +
        │          ├── stage1
 +
        │          └── stage3
 +
        ├── stage2.spec
 +
        ├── stage3-freshen.spec
 +
        ├── stage3-quick.spec
 +
        ├── stage3.spec
 +
        └── stage4.spec
 +
</pre>
  
<source lang="bash">
+
For the beginner, just retain that :
wanif=eth0
+
tc qdisc add dev $wanif root handle 1: prio bands 3
+
tc qdisc add dev $wanif parent 1:1 handle 10: sfq perturb 10
+
tc qdisc add dev $wanif parent 1:2 handle 20: sfq perturb 10
+
tc qdisc add dev $wanif parent 1:3 handle 30: sfq perturb 10
+
  
tc filter add dev $wanif protocol ip parent 1: prio 1 u32 match ip protocol 0x11 0xff flowid 1:1
+
# Python is the master of the house, everything (or at least near everything) in Funtoo tools is written Python, and as being a Funtoo tool, Metro in itself plus all of the stuff it uses are written in Python.
tc filter add dev $wanif protocol ip parent 1: prio 1 u32 match ip dst 0.0.0.0/0 flowid 1:2
+
# The first place to start with Metro tweaking are the .spec files located under '''targets/gentoo'''. Those files control what Metro does at each stage and are basically a melt of Metro directives and some Python embedded scripts and BASH embedded scripts. If something goes wrong.
</source>
+
# Another good place to look at is '''subarch''' : you will find here a couple of .spec files that governs how the build toolchain (libc, c/c++ compiler and related tools) will be generated, how they will be used (notice the CHOST variable) and at the end what will be put in the make.conf file in your stages archives.
  
This code above causes much data to be lost, since the code is not properly classifying all unmatched traffic using the lowest priority (leaf) class (1:3).
+
The [[Metro Data Model]] page explains the logic used in spec files.
  
If the following line is added, which is intended to match all other IP traffic and ensure it gets to 1:3, then behavior improves, but  all ARP packets are still discarded by the <tt>prio</tt> qdisc and the <tt>discarded</tt> counter of the qdisc is incremented accordingly:
+
TBC
  
<source lang="bash">
+
== The GNU toolchain ==
tc filter add dev $wanif protocol ip parent 1: prio 1 u32 match ip src 0.0.0.0/0 flowid 1:3
+
</source>
+
  
I have also experienced issues where the <tt>priomap</tt> does not work properly, and the majority of traffic flows into the wrong child class. It also appears that the first <tt>tc filter</tt> command, which is supposed to prioritize UDP traffic, has the effect of placing all traffic into class <tt>1:2</tt>.
+
''"How the build toolchain will be generated",'' what does it mean ? As you may know, the GNU toolchain is a '''multiform''' set of tools that translate a source program written in a "high" level language like C, C++ or Fortan in a binary form understandable by the microprocessor of your computer. Moreover this set of tools do various sorts of things like source code grammar and syntax checking, code optimization, dead code removal, code instrumentation, and so on.  
  
This code was tested on a physical interface that was part of a Linux bridge, as well as on the Linux bridge device ('brwan') itself. In both cases, <tt>prio</tt> exhibited strange and unusual behavior that deviated significantly from documented behavior. The kernel being used for this testing is a stable Red Hat Enterprise Linux 5.x kernel with OpenVZ patches (2.6.18-028stab079.2).
+
The '''multiform''' magic keyword  means that the GNU toolchain can be compiled for many processors archicture, this of course include the well-known IA-32 (x86 family processors) but also IA-64 (Itaninum), PA-RISC (HPPA), Motorola 68k series, Dec Alpha, SPARC series, ARM, and many others including the so well known DEC PDP/11 series (technically speaking, with the adequate packages and keywordings, Funtoo could bring the steam power on a DEC PDP/11 machine and it could be possible to build stage 3 images for that platform).
  
Based on this testing, <tt>prio</tt> appears to not work correctly at all, at least when used in this configuration, and should be avoided by Linux traffic control users.
+
Basically the GNU tool chains relies on three components :
  
More documentation on this bug:
+
* The so well-known '''GNU Compilers Collection''' (''sys-devel/gcc'' is the Funtoo package that deploys this suite). A common mistake is to call GNU Compiler, but set that aside for now.
 +
* '''Some related tools called behind the scene''' by the GNU Compiler Collection or required by the GNU autoconf/automake system upon which Funtoo relies on. Those behind the scene tools are for example the GNU linker (ld) or the the GNU Assembler (as). Under Funtoo, those tools are deployed by the package ''sys-devel/binutils''.
 +
* '''The GNU lib C''' : this one regroups a large set of system functions used by a large numbers of programs out there and contains functionalities like files manipulation, memory allocation, and so on. It can be barely seen as being a software interface (abstraction) between the system kernel and a program that lives in the userland world.
  
* http://lists.debian.org/debian-kernel/2007/08/msg00487.html
+
Note : the GNU libC is '''not''' the only one solution available in the Free Software world. Several alternatives like '''uLibC''' or '''newlib''' exist and can be used by the GNU Compiler Collection and the related tools it uses.
* Fix for this bug: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=dbaaa07a60edf71670b484a65e14400ab40c84f7
+
  
[[Category:Networking]]
+
The GNU toolchain can be built as being :
 +
 
 +
* '''a native toolchain''' : the computer that runs the toolchain is the same that executes the binaries produced the toolchain, this is the most common case. (e.g. an i686 processor machine running GNU/Linux with the GNU Lib C produces a toolchain that generates binaries for the exact same architecture an i686 processor machine running GNU/Linux with the GNU Lib C).
 +
 
 +
* '''a cross-compilation chain''' : the computer that runs the toolchain is NOT the same that executes the produced the toolchain (e.g. an i686 processor running GNU/Linux with the GNU Lib C produces a toolchain that generates binaries for a SPARC machine running Solaris). Moreover, you can do funny brain-dead things like : generate a toolchain running on a PowerPC GNU/Linux system that produces another toolchain targeting a SPARC machine under Solaris which, in its turn, produces binaries targeting an Alpha machine running under FreeBSD).
 +
 
 +
== Have you ever noticed ? ==
 +
 
 +
By '''CONVENTION''', the GNU toolchain internal mechanisms '''REQUIRES''' to prefix all the toolchain executables by a quadruplet as below :
 +
 
 +
<pre>CPU-machine-OS-libC</pre>
 +
 
 +
* '''CPU''' : i486, i686, sparc, sparc64, alpha...
 +
* '''machine''' :  pc, sun4u, unknown...
 +
* '''OS''' : linux, solaris, freebsd...
 +
* '''libC''' : gnu (for glibc), ulibc...
 +
 
 +
This is stricly conventional but powerful as it allows the toolchain to easily detect when it has to be build it as a native toolchain or as a cross-compilation toolchain. Also note that some architectures like SPARC version 9 (i.e. UltraSPARC) uses a 64 bits kernel and 32 bits userland binaries (although it is possible to have 64 bits binaries, most of the time producing 64 bits binaries does not bring any advantage and worse, can degrade the performance on some machines due to additional memory read/write cycles required to get/write 64 bits values). This is exactly the reason why you have '''sys-devel/gcc''' and '''sys-devel/kgcc64''', the later being only used to build the system kernel.
 +
 
 +
Now lets have a look on your machine at :
 +
 
 +
* the '''CHOST''' variable in your '''/etc/make.conf''' and what lies under your /usr/sbin.
 +
* where points commands like '''/usr/bin/ld''' or '''/usr/bin/as'''
 +
* what prefixes g++ or gcc in /usr/bin (take note that /usr/bin/g++ and /usr/bin/<CPU-machine-OS-libC>-g++ are exactly the same).
 +
 
 +
This is not a coincidence if the CHOST variable and the prefix used by the toolchain executables matches ;-)
 +
 
 +
= How things works ? =
 +
 
 +
 
 +
Playing with the cross-compilation capablities  of the GNU toolchain is out the scope of this article, just retain on this subject that it is mainly used with embedded systems which have too few resources to host a full compilation system or in build farms that generates binaries for several CPU architectures because it is now the time to dive in the main course of the action.
 +
 
 +
As you may have noticed in the [[Metro Quick Start Tutorial]] the beginning of the story lies in the script
 +
 
 +
TBC
 +
 
 +
= Adding a new sub architecture =
 +
 
 +
As of January 2011, Metro can handle the IA-32 architecture (x86/32 and x86/64 bits) as well as being able to handle several x86 flavous lying beneath the generic 'x86 and 'amd64' taxonomies. The example here is a try to add the necessary specification file to make it generate stage 1/2/3 tarballs for tiers architectures.
 +
 
 +
 
 +
 
 +
[[Category:Metro]]

Latest revision as of 03:36, 13 September 2012

Introduction

Metro is a somewhat complex set of shell (specifically bash) and python scripts and various files that automagically creates stages 1/2/3 tarballs used to emerge your Funtoo Linux system. Of course, Metro needs a seed to start its work and will recompile things like the so well known gcc and glibc. A 'seed' is either a pre-built stage3 archive available on any Funtoo Linux mirror or a snapshot of your current Funtoo Linux installation.

Arborescence description

Overview

Don't search through your bin directories, everything lies in /usr/lib/metro. An unfolded view as of Metro revision 1.5.1 is this one :

├── BUGS
├── ChangeLog
├── TODO
├── etc
│   ├── builds
│   │   ├── funtoo
│   │   │   ├── build.conf
│   │   │   └── multi-targets
│   │   │       ├── freshen
│   │   │       ├── full
│   │   │       ├── quick
│   │   │       ├── stage1
│   │   │       └── strategy
│   │   │           └── seed
│   │   │               ├── stage1
│   │   │               └── stage3
│   │   ├── gentoo
│   │   │   ├── build.conf
│   │   │   └── multi-targets
│   │   │       ├── freshen
│   │   │       ├── full
│   │   │       ├── quick
│   │   │       ├── stage1
│   │   │       └── strategy
│   │   │           └── seed
│   │   │               ├── stage1
│   │   │               └── stage3
│   │   ├── hollow
│   │   │   ├── build.conf
│   │   │   └── multi-targets
│   │   │       ├── freshen
│   │   │       ├── full
│   │   │       ├── quick
│   │   │       ├── stage1
│   │   │       └── strategy
│   │   │           └── seed
│   │   │               ├── stage1
│   │   │               └── stage3
│   │   └── ~funtoo
│   │       ├── build.conf
│   │       └── multi-targets
│   │           ├── freshen
│   │           ├── full
│   │           ├── quick
│   │           ├── stage1
│   │           └── strategy
│   │               └── seed
│   │                   ├── stage1
│   │                   └── stage3
│   ├── fslayouts
│   │   └── funtoo
│   │       ├── layout.conf
│   │       └── type
│   │           ├── image
│   │           └── snapshot
│   └── metro.conf
├── features
│   └── probe.conf
├── metro
├── misc
│   └── locale.gen
├── modules
│   ├── catalyst_support.py
│   ├── catalyst_support.pyo
│   ├── flexdata.py
│   ├── flexdata.pyo
│   ├── flexdata2.py
│   ├── flexdata3.py
│   ├── flexdata4.py
│   ├── targets.py
│   └── targets.pyo
├── scripts
│   ├── build.sh
│   ├── cleaner.sh
│   ├── dlstats.sh
│   └── ezbuild.sh
├── subarch
│   ├── amd64
│   │   ├── 2008.spec
│   │   └── 2009.spec
│   ├── amd64.spec
│   ├── athlon-xp.spec
│   ├── core2.spec
│   ├── core2_32.spec
│   ├── generic32.spec
│   ├── generic64.spec
│   ├── i686
│   │   ├── 2008.spec
│   │   └── 2009.spec
│   ├── i686.spec
│   ├── native32.spec
│   ├── native64.spec
│   ├── opteron-sse3.spec
│   ├── opteron.spec
│   ├── opteron_32.spec
│   ├── pentium4.spec
│   ├── phenom.spec
│   ├── x86.spec
│   ├── xen-nocona.spec
│   └── xen-prescott.spec
└── targets
    └── gentoo
        ├── openvz.spec
        ├── snapshot
        │   ├── common.spec
        │   ├── global.spec
        │   ├── source
        │   │   ├── git
        │   │   └── rsync
        │   └── type
        │       ├── dead
        │       └── live
        ├── snapshot.spec
        ├── stage
        │   ├── capture
        │   │   ├── ami.spec
        │   │   └── tar.spec
        │   ├── common.spec
        │   ├── files.spec
        │   ├── stage3-derivative.spec
        │   ├── stage3-generator.spec
        │   └── steps.spec
        ├── stage1
        │   └── strategy
        │       ├── local
        │       └── remote
        ├── stage1.spec
        ├── stage2
        │   └── strategy
        │       ├── local
        │       │   ├── stage1
        │       │   └── stage3
        │       └── remote
        │           ├── stage1
        │           └── stage3
        ├── stage2.spec
        ├── stage3-freshen.spec
        ├── stage3-quick.spec
        ├── stage3.spec
        └── stage4.spec

For the beginner, just retain that :

  1. Python is the master of the house, everything (or at least near everything) in Funtoo tools is written Python, and as being a Funtoo tool, Metro in itself plus all of the stuff it uses are written in Python.
  2. The first place to start with Metro tweaking are the .spec files located under targets/gentoo. Those files control what Metro does at each stage and are basically a melt of Metro directives and some Python embedded scripts and BASH embedded scripts. If something goes wrong.
  3. Another good place to look at is subarch : you will find here a couple of .spec files that governs how the build toolchain (libc, c/c++ compiler and related tools) will be generated, how they will be used (notice the CHOST variable) and at the end what will be put in the make.conf file in your stages archives.

The Metro Data Model page explains the logic used in spec files.

TBC

The GNU toolchain

"How the build toolchain will be generated", what does it mean ? As you may know, the GNU toolchain is a multiform set of tools that translate a source program written in a "high" level language like C, C++ or Fortan in a binary form understandable by the microprocessor of your computer. Moreover this set of tools do various sorts of things like source code grammar and syntax checking, code optimization, dead code removal, code instrumentation, and so on.

The multiform magic keyword means that the GNU toolchain can be compiled for many processors archicture, this of course include the well-known IA-32 (x86 family processors) but also IA-64 (Itaninum), PA-RISC (HPPA), Motorola 68k series, Dec Alpha, SPARC series, ARM, and many others including the so well known DEC PDP/11 series (technically speaking, with the adequate packages and keywordings, Funtoo could bring the steam power on a DEC PDP/11 machine and it could be possible to build stage 3 images for that platform).

Basically the GNU tool chains relies on three components :

  • The so well-known GNU Compilers Collection (sys-devel/gcc is the Funtoo package that deploys this suite). A common mistake is to call GNU Compiler, but set that aside for now.
  • Some related tools called behind the scene by the GNU Compiler Collection or required by the GNU autoconf/automake system upon which Funtoo relies on. Those behind the scene tools are for example the GNU linker (ld) or the the GNU Assembler (as). Under Funtoo, those tools are deployed by the package sys-devel/binutils.
  • The GNU lib C : this one regroups a large set of system functions used by a large numbers of programs out there and contains functionalities like files manipulation, memory allocation, and so on. It can be barely seen as being a software interface (abstraction) between the system kernel and a program that lives in the userland world.

Note : the GNU libC is not the only one solution available in the Free Software world. Several alternatives like uLibC or newlib exist and can be used by the GNU Compiler Collection and the related tools it uses.

The GNU toolchain can be built as being :

  • a native toolchain : the computer that runs the toolchain is the same that executes the binaries produced the toolchain, this is the most common case. (e.g. an i686 processor machine running GNU/Linux with the GNU Lib C produces a toolchain that generates binaries for the exact same architecture an i686 processor machine running GNU/Linux with the GNU Lib C).
  • a cross-compilation chain : the computer that runs the toolchain is NOT the same that executes the produced the toolchain (e.g. an i686 processor running GNU/Linux with the GNU Lib C produces a toolchain that generates binaries for a SPARC machine running Solaris). Moreover, you can do funny brain-dead things like : generate a toolchain running on a PowerPC GNU/Linux system that produces another toolchain targeting a SPARC machine under Solaris which, in its turn, produces binaries targeting an Alpha machine running under FreeBSD).

Have you ever noticed ?

By CONVENTION, the GNU toolchain internal mechanisms REQUIRES to prefix all the toolchain executables by a quadruplet as below :

CPU-machine-OS-libC
  • CPU : i486, i686, sparc, sparc64, alpha...
  • machine : pc, sun4u, unknown...
  • OS : linux, solaris, freebsd...
  • libC : gnu (for glibc), ulibc...

This is stricly conventional but powerful as it allows the toolchain to easily detect when it has to be build it as a native toolchain or as a cross-compilation toolchain. Also note that some architectures like SPARC version 9 (i.e. UltraSPARC) uses a 64 bits kernel and 32 bits userland binaries (although it is possible to have 64 bits binaries, most of the time producing 64 bits binaries does not bring any advantage and worse, can degrade the performance on some machines due to additional memory read/write cycles required to get/write 64 bits values). This is exactly the reason why you have sys-devel/gcc and sys-devel/kgcc64, the later being only used to build the system kernel.

Now lets have a look on your machine at :

  • the CHOST variable in your /etc/make.conf and what lies under your /usr/sbin.
  • where points commands like /usr/bin/ld or /usr/bin/as
  • what prefixes g++ or gcc in /usr/bin (take note that /usr/bin/g++ and /usr/bin/<CPU-machine-OS-libC>-g++ are exactly the same).

This is not a coincidence if the CHOST variable and the prefix used by the toolchain executables matches ;-)

How things works ?

Playing with the cross-compilation capablities of the GNU toolchain is out the scope of this article, just retain on this subject that it is mainly used with embedded systems which have too few resources to host a full compilation system or in build farms that generates binaries for several CPU architectures because it is now the time to dive in the main course of the action.

As you may have noticed in the Metro Quick Start Tutorial the beginning of the story lies in the script

TBC

Adding a new sub architecture

As of January 2011, Metro can handle the IA-32 architecture (x86/32 and x86/64 bits) as well as being able to handle several x86 flavous lying beneath the generic 'x86 and 'amd64' taxonomies. The example here is a try to add the necessary specification file to make it generate stage 1/2/3 tarballs for tiers architectures.