Difference between pages "How Metro Works" and "Talk:How Metro Works"

(Difference between pages)
(Introduction)
 
 
Line 1: Line 1:
= Introduction =
+
I recommend integrating this content into the [[Metro]] main page. We are missing a basic introduction to Metro there - a defintion of what it is, etc.
  
[[Metro]] is a somewhat complex set of shell (specifically bash) and python scripts and various files that automagically creates stages 1/2/3 tarballs used to emerge your [[Funtoo Linux]] system.
+
---
Of course, [[Metro]] needs a seed to start its work and will recompile things like the so well known <tt>gcc</tt> and <tt>glibc</tt>.
+
A 'seed' is either a pre-built stage3 archive available on any Funtoo Linux [http://docs.funtoo.org/wiki/Download mirror] or a snapshot of your current [[Funtoo Linux]] installation.
+
  
= Arborescence description =
+
Basic introduction is not needed, the goal of the article is to dive deeper in the subject and this would add redundancy, those who read the article know what metro is. At worst I will put cross a reference (best practice : do cross references, do not repeat the text between articles). Metro is not a single article, it is a set of several articles (this one, the quick start guide, the data modules, used and whatever else added in the future) grouped with the category "Metro"and this what the [[Metro]] page refers to :)
  
== Overview ==
+
However I agree with you, Metro need a bit more verbosity, I would make the [[Metro]] main page more verbose with a list of several sub pages dedicated to a particular subjects. This page is the common denominator of every other.
Don't search through your bin directories, everything lies in '''/usr/lib/metro'''. An unfolded view as of Metro revision 1.5.1 is this one :
+
 
+
<pre>
+
├── BUGS
+
├── ChangeLog
+
├── TODO
+
├── etc
+
│   ├── builds
+
│   │   ├── funtoo
+
│   │   │   ├── build.conf
+
│   │   │   └── multi-targets
+
│   │   │      ├── freshen
+
│   │   │      ├── full
+
│   │   │      ├── quick
+
│   │   │      ├── stage1
+
│   │   │      └── strategy
+
│   │   │          └── seed
+
│   │   │              ├── stage1
+
│   │   │              └── stage3
+
│   │   ├── gentoo
+
│   │   │   ├── build.conf
+
│   │   │   └── multi-targets
+
│   │   │      ├── freshen
+
│   │   │      ├── full
+
│   │   │      ├── quick
+
│   │   │      ├── stage1
+
│   │   │      └── strategy
+
│   │   │          └── seed
+
│   │   │              ├── stage1
+
│   │   │              └── stage3
+
│   │   ├── hollow
+
│   │   │   ├── build.conf
+
│   │   │   └── multi-targets
+
│   │   │      ├── freshen
+
│   │   │      ├── full
+
│   │   │      ├── quick
+
│   │   │      ├── stage1
+
│   │   │      └── strategy
+
│   │   │          └── seed
+
│   │   │              ├── stage1
+
│   │   │              └── stage3
+
│   │   └── ~funtoo
+
│   │      ├── build.conf
+
│   │      └── multi-targets
+
│   │          ├── freshen
+
│   │          ├── full
+
│   │          ├── quick
+
│   │          ├── stage1
+
│   │          └── strategy
+
│   │              └── seed
+
│   │                  ├── stage1
+
│   │                  └── stage3
+
│   ├── fslayouts
+
│   │   └── funtoo
+
│   │      ├── layout.conf
+
│   │      └── type
+
│   │          ├── image
+
│   │          └── snapshot
+
│   └── metro.conf
+
├── features
+
│   └── probe.conf
+
├── metro
+
├── misc
+
│   └── locale.gen
+
├── modules
+
│   ├── catalyst_support.py
+
│   ├── catalyst_support.pyo
+
│   ├── flexdata.py
+
│   ├── flexdata.pyo
+
│   ├── flexdata2.py
+
│   ├── flexdata3.py
+
│   ├── flexdata4.py
+
│   ├── targets.py
+
│   └── targets.pyo
+
├── scripts
+
│   ├── build.sh
+
│   ├── cleaner.sh
+
│   ├── dlstats.sh
+
│   └── ezbuild.sh
+
├── subarch
+
│   ├── amd64
+
│   │   ├── 2008.spec
+
│   │   └── 2009.spec
+
│   ├── amd64.spec
+
│   ├── athlon-xp.spec
+
│   ├── core2.spec
+
│   ├── core2_32.spec
+
│   ├── generic32.spec
+
│   ├── generic64.spec
+
│   ├── i686
+
│   │   ├── 2008.spec
+
│   │   └── 2009.spec
+
│   ├── i686.spec
+
│   ├── native32.spec
+
│   ├── native64.spec
+
│   ├── opteron-sse3.spec
+
│   ├── opteron.spec
+
│   ├── opteron_32.spec
+
│   ├── pentium4.spec
+
│   ├── phenom.spec
+
│   ├── x86.spec
+
│   ├── xen-nocona.spec
+
│   └── xen-prescott.spec
+
└── targets
+
    └── gentoo
+
        ├── openvz.spec
+
        ├── snapshot
+
        │   ├── common.spec
+
        │   ├── global.spec
+
        │   ├── source
+
        │   │   ├── git
+
        │   │   └── rsync
+
        │   └── type
+
        │      ├── dead
+
        │      └── live
+
        ├── snapshot.spec
+
        ├── stage
+
        │   ├── capture
+
        │   │   ├── ami.spec
+
        │   │   └── tar.spec
+
        │   ├── common.spec
+
        │   ├── files.spec
+
        │   ├── stage3-derivative.spec
+
        │   ├── stage3-generator.spec
+
        │   └── steps.spec
+
        ├── stage1
+
        │   └── strategy
+
        │      ├── local
+
        │      └── remote
+
        ├── stage1.spec
+
        ├── stage2
+
        │   └── strategy
+
        │      ├── local
+
        │      │   ├── stage1
+
        │      │   └── stage3
+
        │      └── remote
+
        │          ├── stage1
+
        │          └── stage3
+
        ├── stage2.spec
+
        ├── stage3-freshen.spec
+
        ├── stage3-quick.spec
+
        ├── stage3.spec
+
        └── stage4.spec
+
</pre>
+
 
+
For the beginner, just retain that :
+
 
+
# Python is the master of the house, everything (or at least near everything) in Funtoo tools is written Python, and as being a Funtoo tool, Metro in itself plus all of the stuff it uses are written in Python.
+
# The first place to start with Metro tweaking are the .spec files located under '''targets/gentoo'''. Those files control what Metro does at each stage and are basically a melt of Metro directives and some Python embedded scripts and BASH embedded scripts. If something goes wrong.
+
# Another good place to look at is '''subarch''' : you will find here a couple of .spec files that governs how the build toolchain (libc, c/c++ compiler and related tools) will be generated, how they will be used (notice the CHOST variable) and at the end what will be put in the make.conf file in your stages archives.
+
 
+
The [[Metro Data Model]] page explains the logic used in spec files.
+
 
+
TBC
+
 
+
== The GNU toolchain ==
+
 
+
''"How the build toolchain will be generated",'' what does it mean ? As you may know, the GNU toolchain is a '''multiform''' set of tools that translate a source program written in a "high" level language like C, C++ or Fortan in a binary form understandable by the microprocessor of your computer. Moreover this set of tools do various sorts of things like source code grammar and syntax checking, code optimization, dead code removal, code instrumentation, and so on.
+
 
+
The '''multiform''' magic keyword  means that the GNU toolchain can be compiled for many processors archicture, this of course include the well-known IA-32 (x86 family processors) but also IA-64 (Itaninum), PA-RISC (HPPA), Motorola 68k series, Dec Alpha, SPARC series, ARM, and many others including the so well known DEC PDP/11 series (technically speaking, with the adequate packages and keywordings, Funtoo could bring the steam power on a DEC PDP/11 machine and it could be possible to build stage 3 images for that platform).
+
 
+
Basically the GNU tool chains relies on three components :
+
 
+
* The so well-known '''GNU Compilers Collection''' (''sys-devel/gcc'' is the Funtoo package that deploys this suite). A common mistake is to call GNU Compiler, but set that aside for now.
+
* '''Some related tools called behind the scene''' by the GNU Compiler Collection or required by the GNU autoconf/automake system upon which Funtoo relies on. Those behind the scene tools are for example the GNU linker (ld) or the the GNU Assembler (as). Under Funtoo, those tools are deployed by the package ''sys-devel/binutils''.
+
* '''The GNU lib C''' : this one regroups a large set of system functions used by a large numbers of programs out there and contains functionalities like files manipulation, memory allocation, and so on. It can be barely seen as being a software interface (abstraction) between the system kernel and a program that lives in the userland world.
+
 
+
Note : the GNU libC is '''not''' the only one solution available in the Free Software world. Several alternatives like '''uLibC''' or '''newlib''' exist and can be used by the GNU Compiler Collection and the related tools it uses.
+
 
+
The GNU toolchain can be built as being :
+
 
+
* '''a native toolchain''' : the computer that runs the toolchain is the same that executes the binaries produced the toolchain, this is the most common case. (e.g. an i686 processor machine running GNU/Linux with the GNU Lib C produces a toolchain that generates binaries for the exact same architecture an i686 processor machine running GNU/Linux with the GNU Lib C).
+
 
+
* '''a cross-compilation chain''' : the computer that runs the toolchain is NOT the same that executes the produced the toolchain (e.g. an i686 processor running GNU/Linux with the GNU Lib C produces a toolchain that generates binaries for a SPARC machine running Solaris). Moreover, you can do funny brain-dead things like : generate a toolchain running on a PowerPC GNU/Linux system that produces another toolchain targeting a SPARC machine under Solaris which, in its turn, produces binaries targeting an Alpha machine running under FreeBSD).
+
 
+
== Have you ever noticed ? ==
+
 
+
By '''CONVENTION''', the GNU toolchain internal mechanisms '''REQUIRES''' to prefix all the toolchain executables by a quadruplet as below :
+
 
+
<pre>CPU-machine-OS-libC</pre>
+
 
+
* '''CPU''' : i486, i686, sparc, sparc64, alpha...
+
* '''machine''' :  pc, sun4u, unknown...
+
* '''OS''' : linux, solaris, freebsd...
+
* '''libC''' : gnu (for glibc), ulibc...
+
 
+
This is stricly conventional but powerful as it allows the toolchain to easily detect when it has to be build it as a native toolchain or as a cross-compilation toolchain. Also note that some architectures like SPARC version 9 (i.e. UltraSPARC) uses a 64 bits kernel and 32 bits userland binaries (although it is possible to have 64 bits binaries, most of the time producing 64 bits binaries does not bring any advantage and worse, can degrade the performance on some machines due to additional memory read/write cycles required to get/write 64 bits values). This is exactly the reason why you have '''sys-devel/gcc''' and '''sys-devel/kgcc64''', the later being only used to build the system kernel.
+
 
+
Now lets have a look on your machine at :
+
 
+
* the '''CHOST''' variable in your '''/etc/make.conf''' and what lies under your /usr/sbin.
+
* where points commands like '''/usr/bin/ld''' or '''/usr/bin/as'''
+
* what prefixes g++ or gcc in /usr/bin (take note that /usr/bin/g++ and /usr/bin/<CPU-machine-OS-libC>-g++ are exactly the same).
+
 
+
This is not a coincidence if the CHOST variable and the prefix used by the toolchain executables matches ;-)
+
 
+
= How things works ? =
+
 
+
 
+
Playing with the cross-compilation capablities  of the GNU toolchain is out the scope of this article, just retain on this subject that it is mainly used with embedded systems which have too few resources to host a full compilation system or in build farms that generates binaries for several CPU architectures because it is now the time to dive in the main course of the action.
+
 
+
As you may have noticed in the [[Metro Quick Start Tutorial]] the beginning of the story lies in the script
+
 
+
TBC
+
 
+
= Adding a new sub architecture =
+
 
+
As of January 2011, Metro can handle the IA-32 architecture (x86/32 and x86/64 bits) as well as being able to handle several x86 flavous lying beneath the generic 'x86 and 'amd64' taxonomies. The example here is a try to add the necessary specification file to make it generate stage 1/2/3 tarballs for tiers architectures.
+
 
+
 
+
 
+
[[Category:Metro]]
+

Latest revision as of 01:52, 20 February 2011

I recommend integrating this content into the Metro main page. We are missing a basic introduction to Metro there - a defintion of what it is, etc.

---

Basic introduction is not needed, the goal of the article is to dive deeper in the subject and this would add redundancy, those who read the article know what metro is. At worst I will put cross a reference (best practice : do cross references, do not repeat the text between articles). Metro is not a single article, it is a set of several articles (this one, the quick start guide, the data modules, used and whatever else added in the future) grouped with the category "Metro"and this what the Metro page refers to :)

However I agree with you, Metro need a bit more verbosity, I would make the Metro main page more verbose with a list of several sub pages dedicated to a particular subjects. This page is the common denominator of every other.