Difference between pages "ReBootstrap" and "Forking An Ebuild"

(Difference between pages)
(Status)
 
m
 
Line 1: Line 1:
This page documents the effort to bootstrap Funtoo Linux using Aboriginal Linux.
+
Often, a Funtoo developer needs to fork an upstream ebuild. This is necessary when we want to apply fixes to it. This page will explain the concepts of forking and how this works in the context of Funtoo.
  
= What is Aboriginal Linux? =
+
== Portage Tree Generation ==
  
Aboriginal Linux's motto is "we cross compile so you don't have to". It replaces cross compiling with native compiling under an emulator (generally QEMU). Funtoo can use this to natively create stage 1/2/3 root filesystems on an arbitrary target, which greatly simplifies adding support for new architectures and regression testing existing architectures.
+
Funtoo Linux generates its Portage tree using a special script that essentially takes a Gentoo tree as its starting point, and then applies various modifications to it. The modifications involve adding packages from various overlays, including our [https://github.com/funtoo/funtoo-overlay Funtoo-overlay]. Some packages added are brand new, while other packages are our special forked versions that replace existing packages.  
  
Aboriginal Linux creates a simple native development environment for each type of target hardware, building the smallest self-contained Linux system capable of rebuilding itself entirely from source code.  This requires seven packages: linux, uClibc, busybox, binutils, gcc, make, and bash.
+
In the vast majority of cases, when we fork a package, we take full responsibility for all ebuilds associated with that package, meaning that we have a full copy of the <tt>sys-foo/bar</tt> directory in one of our overlays.
  
The resulting system can then boot under QEMU (or on appropriate hardware, if available) to provide a native development environment, eliminating the need for any further cross compiling.  You can wget, configure, make, and install additional source packages inside the emulator.
+
If you're interested in seeing the actual script that does all these things, take a look at the following files:
  
Aboriginal Linux's web page is at http://landley.net/aboriginal and a long presentation about it is available at http://speakerdeck.com/u/mirell/p/developing-for-non-x86-targets-using-qemu
+
; http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/current-update.sh: cronned script that calls <tt>merge.py</tt>.
 +
;http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge.py: python script that does the heavy lifting of combining Gentoo tree with various overlays, including our flora and funtoo-overlay. When we want to change what overlays we merge, what packages we exclude as a matter of policy (such as stale packages in some overlays), we make changes to this file.
 +
; http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge_utils.py: python module that contains classes and methods that implement the merging functionality.
  
== How do I use Aboriginal Linux? ==
+
== Forking an Ebuild ==
  
Aboriginal Linux provides prebuilt native development environments for x86, x86-64, arm, mips, powerpc, and more, ready to run under the emulator. Install QEMU 1.0 or later, then go to http://landley.net/aboriginal/downloads/binaries", grab the appropriate system-image tarball, extract it, and run one of the following three shell scripts (included in the tarball):
+
In general, we fork ebuilds from Gentoo that we want to modify in some way. Before you fork an ebuild, it's important to understand that in general we fork entire packages, not just a single ebuild. This means that if you want to make some changes to <tt>sys-foo/bar</tt>, you are going to fork all <tt>sys-foo/bar</tt> ebuilds, and then Funtoo will be responsible for continuing to maintain these ebuilds until the package is unforked. Here are the steps we would use to fork <tt>sys-foo/bar</tt>:
  
* '''./run-emulator.sh''' - Boot to a shell prompt, with the simplest configuration: a read-only (squashfs) root filesystem with a tmpfs mounted on /home.
+
# Find <tt>sys-foo/bar</tt> in you regular Portage tree. Make sure you have run <tt>emerge --sync</tt> recently to ensure it is up-to-date. If you want to fork from very recent changes that are not yet in our tree, you may need to grab the most recent Gentoo Portage tree to serve as your source for <tt>sys-foo/bar</tt> (this typically isn't necessary.)
 +
<console>
 +
# alias to recursively grab latest from Gentoo Portage tree WITHOUT history
 +
# usage: getgen gentoo-x86/dev-db/mongodb
 +
alias getgen="cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot export -D$(date '+%Y-%m-%d')"
 +
</console>
 +
# Copy the <tt>sys-foo/bar</tt> directory in its entirety to <tt>funtoo-overlay/sys-foo/bar</tt>.
 +
# Make any necessary modifications to <tt>funtoo-overlay/sys-foo/bar</tt>.
 +
# Perform some funtoo-ification steps prior to commit.
 +
# Add and commit the changes to funtoo-overlay.
 +
# Push changes to funtoo-overlay.
  
* '''./dev-environment.sh''' - Boot to a shell prompt with a full development environment. This wrapper around run-emulator.sh adds a 2 gigabyte ext2 /dev/hdb image mounted on /home (persistent writeable space, in large amounts), and allocates 256 megabytes of physical memory for the emulator (enough to run gcc building complex packages). If distccd and the appropriate target's cross compiler are available in the host's $PATH, this script also configures the emulated environment to call out to the cross compiler via distcc, transparently moving the heavy lifting of compilation outside of the emulator.
+
At this point, the forked <tt>sys-foo/bar</tt> package will be part of funtoo-overlay. The next time our unified Portage tree is generated by <tt>merge.py</tt> (the one that users have in their <tt>/usr/portage</tt> and is updated via <tt>emerge --sync</tt>), your forked ebuild will be used in place of the Gentoo ebuild. Why is this? It is because our <tt>merge.py</tt> script has been defined with a policy that any ebuilds in funtoo-overlay will replace any existing Gentoo ebuilds if they exist. The mechanism of replacement is that our <tt>sys-foo/bar</tt> directory will be used in place of Gentoo's <tt>sys-foo/bar</tt> directory. So this is how the forking process works.
  
* '''./native-build.sh''' - Run an automated build.  This requires a "control image", which is a squashfs filesystem the emulated system sees as /dev/hdc mounted on /mnt.  The init script in the emulated root filesystem checks for /mnt/init and if that exists, it runs that instead of providing a shell prompt.  (Actually it asks the user to press a key if they want a shell prompt, with a three second timeout.)  See "http://landley.net/aboriginal/control-images" for details.
+
== Funtoo-ification ==
  
(You can build your own system images from source by following the instructions at "http://landley.net/aboriginal", but ReBootstrap does not require this.)
+
When we fork a package from Gentoo, we perform the following tweaks to the package directory before committing:
  
== What is an "LFS build"? ==
+
# Removal of <tt>ChangeLog</tt>.
 +
# Run <tt>ebuild foo-1.0.ebuild digest</tt> before committing. This will cause the <tt>Manifest</tt> file to be regenerated. Gentoo has a lot more entries in this file than we do, since we use mini-Manfiests that only include DIST listings (for distfiles only.) We want to commit our mini-Manifest (still called <tt>Manifest</tt>, just with less entries in it) rather than the one that came from Gentoo.
 +
# Edit the top of each ebuild, and remove all <tt>Copyright</tt> and <tt>$Header:</tt> lines at the top of the file. We have a LICENSE.txt and COPYRIGHT.txt file in the root of our Portage tree, which is easier to maintain than keeping all the years up-to-date in each ebuild. Also, the <tt>$Header:</tt> line is there for the CVS version control system in Gentoo which Funtoo does not use. ''The only comment that should remain on the top of the ebuild is the one stating that it is distributed under the GPLv2.''.
  
It's a [http://www.linuxfromscratch.org/lfs/view/6.8/ Linux From Scratch] root filesystem natively built on top of an Aboriginal Linux system image.
+
<console>
 +
# If you find yourself doing this often, place this function in your .bashrc, .zshrc, etc
 +
funtooize() {
 +
    if [ -z "$1" ]; then
 +
        search_path='.'
 +
    else
 +
        search_path=$1
 +
    fi
  
One of the example native-build.sh control-images Aboriginal Linux provides is "lfs-bootstrap.hdc", which natively compiles and installs (almost) all of the LFS chapter 6 packages under the emulated development environment, according to the LFS build instructions.
+
    find $search_path -type f -exec sed -i -e '/^# Copyright\|^# \$Header/d' {} +
 +
    find $search_path -type f -name "ChangeLog*" -delete
 +
    find $search_path -type f -name '*.ebuild' -exec ebuild {} manifest \;
 +
}
 +
</console>
  
The control image copies the read-only root filesystem into a writeable directory under /home, chroots into that, builds packages and installs them over the existing (busybox) versions, and then tars up the resulting filesystem.
+
Here are a few additional changes that you are allowed to make to any forked ebuilds:
  
This provides a much more capable native build environment for the target. Since Aboriginal Linux can already provide this for the supported targets, Funtoo uses this as its starting point instead of starting from the minimal build environment and building/installing all of its own prerequisites.
+
# Line length greater than 80 characters. Gentoo enforces an 80-character line length limit. We don't.
 +
# <tt>KEYWORDS</tt> of <tt>*</tt> and <tt>~*</tt>. Gentoo does not allow these shortcuts. We do. They allow you to say "all arches" and "all unstable arches" in a concise way. Gentoo doesn't allow these shortcuts because it's Gentoo's policy to have each arch team manually approve each package. We do not have this policy so we can use the shortcuts.
 +
# Use of <tt>4-python</tt> EAPI. We allow the use of this EAPI for enhanced python functionality.
  
= The plan of attack is as follows: =
+
[[Category:Development]]
 
+
# Start from an LFS build, plus necessary stuff for Portage to run.  (Git, python, portage tree, profile...)
+
## Use the i686 target first, so we can test quickly via chroot.
+
# Create a <tt>package.provided</tt> file for the local LFS system so that Portage doesn't complain of unsatisfied <tt>/var/db/pkg</tt> entries.
+
# Use Portage to build a stage1 tarball to /tmp/stage1root.
+
 
+
Once we have a stage 1 for the target, Funtoo has been bootstrapped and we can extend this approach to bootstrap Funtoo on any architecture.
+
 
+
== Details ==
+
 
+
* Setup LFS-i686 filesystem
+
 
+
:wget http://landley.net/aboriginal/downloads/binaries/extras/lfs-bootstrap.tar.gz-i686
+
:tar xvzf lfs-bootstrap.tar.gz-i686
+
:sudo env -i HOST=i686 $(which chroot) lfs-bootstrap /sbin/init.sh
+
 
+
* Install python
+
 
+
:wget http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2
+
:tar xf Python-2.7.2.tar.bz2
+
:cd Python-2.7.2
+
:./configure --prefix=/usr
+
:make -j $CPUS
+
:make install
+
 
+
* Install git
+
 
+
:wget http://git-core.googlecode.com/files/git-1.7.8.2.tar.gz
+
:tar xzvf git-1.7.8.2.tar.gz
+
:cd git-1.7.8.2
+
:./configure --without-tcltk --without-python --without-iconv
+
:make NO_PERL=1
+
:make NO_PERL=1 install
+
 
+
* Install portage tool
+
 
+
:cd /usr/lib
+
:git clone git://github.com/funtoo/portage-funtoo portage
+
:cd portage
+
:mkdir -p /usr/share/portage
+
:ln -s /usr/lib/portage/cnf /usr/share/portage/config
+
:for i in emerge ebuild; do ln -s /usr/lib/portage/bin/$i /usr/bin/$i; done
+
 
+
* Install portage tree, setup profiles
+
 
+
:cd /usr
+
:tar xvzf portage_tarball.tar.gz
+
:mv portage-system portage
+
:ln -s /usr/portage/profiles/default/linux/$ARCH/2008.0 /etc/make.profile
+
 
+
* setup package.provided
+
 
+
:git clone git://github.com/funtoo/funtoo-overlay /var/tmp/funtoo-overlay
+
:mkdir -p /etc/portage
+
:cp /var/tmp/funtoo-overlay/funtoo/scripts/stage3.provided /etc/portage/package.provided
+
 
+
* Try to emerge something.  Notice it doesn't work.  Frown at it.
+
** package.provided doesn't support USE flags.  What the...?
+
 
+
== Prerequisites ==
+
 
+
# A smaller Portage tree to not eat up so much filesystem space. This has now been implemented. The script <tt>/root/git/funtoo-overlay/funtoo/scripts/generate-system-tree.py</tt> can be used to generate a system-only Portage tree.
+
## The dependencies are still circular and strange.
+
### Libtool is an abomination.  It exists to make non-elf systems behave like ELF, and since Linux switched from a.out to ELF in 1996 (when it was 4 years old) there's nothing for Libtool to actually DO on Linux, yet it still manages not to do it properly.  Literally the ONLY thing installing Libtool on a Linux system does is introduce build breaks.  (Thank you, Free Software Foundation.)
+
# A <tt>package.provided</tt> list of a current Funtoo system. This can be found at <tt>/root/git/funtoo-overlay/funtoo/scripts/stage3-provided.txt</tt>.
+
#The profiles aren't target-agnostic.  There's no way to go
+
"build for this host, I don't really care what it is".
+
## You need an existing toolchain to build anything, and if it's gcc its "tuple" is available via "gcc -dumpmachine", so this is queryable at runtime.
+
## See [http://www.funtoo.org/wiki/Funtoo_1.0_Profile] for ongoing work to genericize this.
+
### the keywords=* stuff helps greatly, but make.conf moved into profiles and is still somewhat target-specific.  More cleanup needed here.  How do you build on an arbitrary host without human intervention (from a cron job)?
+
 
+
== Status ==
+
 
+
The approach above didn't seem to correctly provide needed dependencies. Options at this point:
+
 
+
# Bundle up what we have as a stage3 and see if Metro's steps are more successful at using it as a stage3 (we were trying simple emerge commands rather than the explicit metro steps for testing)
+
## When you cd into the profile directory and emerge an ebuild file directly, it generally builds.  It's dependency calculation that's screwed up: package.provided injects a blank package with no USE flags, and anything that depends on a package with a USE flag tries to rebuild that package.  I.E. package.provided is essentially useless.
+
### drobbins suggested something like "portage inject packagename USE="flag flag flag" [files...]" to retroactively notify portage of an installed package.  (Aboriginal Linux has a BINARY_PACKAGE_TARBALLS option using "touch timestamp; build package; find output -newer timestamp", and this could be inserted in the LFS build fairly easily.  Or just use a blank file list and not care that portage can't upgrade/uninstall the package, we just need the dependency generation to work.)
+
# If package.provided is not working, simply copy <tt>/var/db/pkg</tt> from an existing stage3 for a massive injection of fake package information.
+
# Or the option we are going to go with, which is stop trying to build a stage3, and build a minimal stage1 that we will feed to metro to take care of the rest.
+
 
+
[[Category:Labs]] [[Category:Projects]]
+

Latest revision as of 05:25, May 30, 2015

Often, a Funtoo developer needs to fork an upstream ebuild. This is necessary when we want to apply fixes to it. This page will explain the concepts of forking and how this works in the context of Funtoo.

Portage Tree Generation

Funtoo Linux generates its Portage tree using a special script that essentially takes a Gentoo tree as its starting point, and then applies various modifications to it. The modifications involve adding packages from various overlays, including our Funtoo-overlay. Some packages added are brand new, while other packages are our special forked versions that replace existing packages.

In the vast majority of cases, when we fork a package, we take full responsibility for all ebuilds associated with that package, meaning that we have a full copy of the sys-foo/bar directory in one of our overlays.

If you're interested in seeing the actual script that does all these things, take a look at the following files:

http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/current-update.sh
cronned script that calls merge.py.
http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge.py
python script that does the heavy lifting of combining Gentoo tree with various overlays, including our flora and funtoo-overlay. When we want to change what overlays we merge, what packages we exclude as a matter of policy (such as stale packages in some overlays), we make changes to this file.
http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge_utils.py
python module that contains classes and methods that implement the merging functionality.

Forking an Ebuild

In general, we fork ebuilds from Gentoo that we want to modify in some way. Before you fork an ebuild, it's important to understand that in general we fork entire packages, not just a single ebuild. This means that if you want to make some changes to sys-foo/bar, you are going to fork all sys-foo/bar ebuilds, and then Funtoo will be responsible for continuing to maintain these ebuilds until the package is unforked. Here are the steps we would use to fork sys-foo/bar:

  1. Find sys-foo/bar in you regular Portage tree. Make sure you have run emerge --sync recently to ensure it is up-to-date. If you want to fork from very recent changes that are not yet in our tree, you may need to grab the most recent Gentoo Portage tree to serve as your source for sys-foo/bar (this typically isn't necessary.)
# alias to recursively grab latest from Gentoo Portage tree WITHOUT history
# usage: getgen gentoo-x86/dev-db/mongodb
alias getgen="cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot export -D$(date '+%Y-%m-%d')"
  1. Copy the sys-foo/bar directory in its entirety to funtoo-overlay/sys-foo/bar.
  2. Make any necessary modifications to funtoo-overlay/sys-foo/bar.
  3. Perform some funtoo-ification steps prior to commit.
  4. Add and commit the changes to funtoo-overlay.
  5. Push changes to funtoo-overlay.

At this point, the forked sys-foo/bar package will be part of funtoo-overlay. The next time our unified Portage tree is generated by merge.py (the one that users have in their /usr/portage and is updated via emerge --sync), your forked ebuild will be used in place of the Gentoo ebuild. Why is this? It is because our merge.py script has been defined with a policy that any ebuilds in funtoo-overlay will replace any existing Gentoo ebuilds if they exist. The mechanism of replacement is that our sys-foo/bar directory will be used in place of Gentoo's sys-foo/bar directory. So this is how the forking process works.

Funtoo-ification

When we fork a package from Gentoo, we perform the following tweaks to the package directory before committing:

  1. Removal of ChangeLog.
  2. Run ebuild foo-1.0.ebuild digest before committing. This will cause the Manifest file to be regenerated. Gentoo has a lot more entries in this file than we do, since we use mini-Manfiests that only include DIST listings (for distfiles only.) We want to commit our mini-Manifest (still called Manifest, just with less entries in it) rather than the one that came from Gentoo.
  3. Edit the top of each ebuild, and remove all Copyright and $Header: lines at the top of the file. We have a LICENSE.txt and COPYRIGHT.txt file in the root of our Portage tree, which is easier to maintain than keeping all the years up-to-date in each ebuild. Also, the $Header: line is there for the CVS version control system in Gentoo which Funtoo does not use. The only comment that should remain on the top of the ebuild is the one stating that it is distributed under the GPLv2..
# If you find yourself doing this often, place this function in your .bashrc, .zshrc, etc
funtooize() {
    if [ -z "$1" ]; then
        search_path='.'
    else
        search_path=$1
    fi

    find $search_path -type f -exec sed -i -e '/^# Copyright\|^# \$Header/d' {} +
    find $search_path -type f -name "ChangeLog*" -delete
    find $search_path -type f -name '*.ebuild' -exec ebuild {} manifest \;
}

Here are a few additional changes that you are allowed to make to any forked ebuilds:

  1. Line length greater than 80 characters. Gentoo enforces an 80-character line length limit. We don't.
  2. KEYWORDS of * and ~*. Gentoo does not allow these shortcuts. We do. They allow you to say "all arches" and "all unstable arches" in a concise way. Gentoo doesn't allow these shortcuts because it's Gentoo's policy to have each arch team manually approve each package. We do not have this policy so we can use the shortcuts.
  3. Use of 4-python EAPI. We allow the use of this EAPI for enhanced python functionality.