Difference between pages "Git Merging Guide" and "ReBootstrap"

From Funtoo
(Difference between pages)
Jump to: navigation, search
(Package Replacement: Funtoo Overlay (branch to branch))
 
(Status)
 
Line 1: Line 1:
This page is here to show Funtoo Linux developers different techniques that can be used to merge various things.
+
This page documents the effort to bootstrap Funtoo Linux using Aboriginal Linux.
  
== Comparing Experimental and Master ==
+
= What is Aboriginal Linux? =
  
The best way to get a quick and dirty understanding of the differences between experimental and master is to do this:
+
Aboriginal Linux's motto is "we cross compile so you don't have to".  It replaces cross compiling with native compiling under an emulator (generally QEMU).  Funtoo can use this to natively create stage 1/2/3 root filesystems on an arbitrary target, which greatly simplifies adding support for new architectures and regression testing existing architectures.
  
<console>
+
Aboriginal Linux creates a simple native development environment for each type of target hardware, building the smallest self-contained Linux system capable of rebuilding itself entirely from source code.  This requires seven packages: linux, uClibc, busybox, binutils, gcc, make, and bash.
# ##i##cd /root/git/funtoo-overlay
+
# ##i##git diff --stat origin/master origin/experimental
+
</console>
+
  
This will show a summary of what modifications where made on a file-by-file basis.
+
The resulting system can then boot under QEMU (or on appropriate hardware, if available) to provide a native development environment, eliminating the need for any further cross compiling.  You can wget, configure, make, and install additional source packages inside the emulator.
  
== Package Replacement: Funtoo Overlay (branch to branch) ==
+
Aboriginal Linux's web page is at http://landley.net/aboriginal and a long presentation about it is available at http://speakerdeck.com/u/mirell/p/developing-for-non-x86-targets-using-qemu
  
When merging in funtoo-overlay, we might want to merge things from experimental to master. To do this, first pick a specific package to compare changes:
+
== How do I use Aboriginal Linux? ==
  
<console>
+
Aboriginal Linux provides prebuilt native development environments for x86, x86-64, arm, mips, powerpc, and more, ready to run under the emulatorInstall QEMU 1.0 or later, then go to http://landley.net/aboriginal/downloads/binaries", grab the appropriate system-image tarball, extract it, and run one of the following three shell scripts (included in the tarball):
# ##i##cd /root/git/funtoo-overlay
+
# ##i##git diff --stat origin/master origin/experimental app-shells/bash
+
app-shells/bash/bash-3.1_p17.ebuild  | 150 -------------------------
+
app-shells/bash/bash-3.2_p51.ebuild  |  199 ---------------------------------
+
app-shells/bash/bash-4.0_p37.ebuild  |  193 --------------------------------
+
app-shells/bash/bash-4.0_p38.ebuild  |  193 --------------------------------
+
app-shells/bash/bash-4.1_p10.ebuild  |  191 -------------------------------
+
app-shells/bash/bash-4.1_p7-r1.ebuild |  189 -------------------------------
+
app-shells/bash/bash-4.1_p9-r1.ebuild |  189 -------------------------------
+
app-shells/bash/bash-4.2_p10.ebuild  |    5 +-
+
8 files changed, 2 insertions(+), 1307 deletions(-)</console>
+
  
The "----" in the diff above shows that several ebuilds were removed ("----" means many lines were removed) in the experimental branch, and <tt>bash-4.2_p10.ebuild</tt> had slight modifications. This looks like a good candidate for grabbing from experimental to replace entirely what is in master. Here's an example of something that is ''not'' a good candidate for a wholesale replacement:
+
* '''./run-emulator.sh''' - Boot to a shell prompt, with the simplest configuration: a read-only (squashfs) root filesystem with a tmpfs mounted on /home.
  
<console>
+
* '''./dev-environment.sh''' - Boot to a shell prompt with a full development environmentThis wrapper around run-emulator.sh adds a 2 gigabyte ext2 /dev/hdb image mounted on /home (persistent writeable space, in large amounts), and allocates 256 megabytes of physical memory for the emulator (enough to run gcc building complex packages).  If distccd and the appropriate target's cross compiler are available in the host's $PATH, this script also configures the emulated environment to call out to the cross compiler via distcc, transparently moving the heavy lifting of compilation outside of the emulator.
# ##i##git diff --stat origin/master origin/experimental sys-apps/pciutils
+
sys-apps/pciutils/Manifest                        |    3 -
+
sys-apps/pciutils/files/conf.d-pciparm            |  28 -------
+
sys-apps/pciutils/files/init.d-pciparm            |  80 --------------------
+
.../files/pciutils-3.1.4-install-lib.patch        |  40 ----------
+
  sys-apps/pciutils/files/pciutils-3.1.7-fbsd.patch  |  11 ---
+
.../files/pciutils-3.1.7-install-lib.patch        |  41 ----------
+
.../pciutils-3.1.8-avoid-segfault-on-init.patch    |  16 ----
+
sys-apps/pciutils/files/pciutils.cron              |    2 -
+
sys-apps/pciutils/pciutils-3.1.8-r1.ebuild        |  76 -------------------
+
9 files changed, 0 insertions(+), 297 deletions(-)
+
</console>
+
  
In this example above, <tt>sys-apps/pciutils</tt> had a lot of cleanups in experimental, but the output above indicates that there is a new <tt>pciutils-3.1.8-1.ebuild</tt> in master that is not experimental. If we replace what is in master with that in experimental, we will lose the new ebuild! So we wouldn't want to do a wholesale replacement in this case. Old ebuilds that disappear are cleanups, but new ebuilds that disappear are not. Be sure to pay attention to whether the ebuilds that are being removed are old or new.
+
* '''./native-build.sh''' - Run an automated build. This requires a "control image", which is a squashfs filesystem the emulated system sees as /dev/hdc mounted on /mnt. The init script in the emulated root filesystem checks for /mnt/init and if that exists, it runs that instead of providing a shell prompt.  (Actually it asks the user to press a key if they want a shell prompt, with a three second timeout.)  See "http://landley.net/aboriginal/control-images" for details.
  
Back to our bash example. To inspect changes in more detail to make sure they are acceptable, specify the modified ebuild directly and drop the <tt>--stat</tt> option:
+
(You can build your own system images from source by following the instructions at "http://landley.net/aboriginal", but ReBootstrap does not require this.)
  
<console>
+
== What is an "LFS build"? ==
# ##i##git diff origin/master origin/experimental app-shells/bash/bash-4.2_p10.ebuild
+
diff --git a/app-shells/bash/bash-4.2_p10.ebuild b/app-shells/bash/bash-4.2_p10.ebuild
+
index 0c497ea..e603c15 100644
+
--- a/app-shells/bash/bash-4.2_p10.ebuild
+
+++ b/app-shells/bash/bash-4.2_p10.ebuild
+
@@ -37,7 +37,7 @@ SRC_URI="mirror://gnu/bash/${MY_P}.tar.gz $(patches)
+
+
LICENSE="GPL-3"
+
SLOT="0"
+
-KEYWORDS="~alpha ~amd64 ~arm ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~sparc-fbsd ~x86-fbsd"
+
+KEYWORDS="*"
+
IUSE="afs bashlogger examples mem-scramble +net nls plugins vanilla"
+
+
DEPEND=">=sys-libs/ncurses-5.2-r2
+
@@ -69,7 +69,6 @@ src_unpack() {
+
        cd lib/readline
+
        [[ ${READLINE_PLEVEL} -gt 0 ]] && epatch $(patches -s ${READLINE_PLEVEL} readline ${READLINE_VER})
+
        cd ../..
+
-     
+
        epatch "${FILESDIR}"/${PN}-4.1-document-system-bashrc.patch
+
}
+
+
@@ -104,7 +103,7 @@ src_compile() {
+
        myconf="${myconf} --with-curses"
+
+
        myconf="${myconf} --without-lispdir" #335896
+
-     
+
+
+
        use plugins && append-ldflags -Wl,-rpath,/usr/$(get_libdir)/bash
+
        econf \
+
                $(use_with afs) \
+
</console>
+
  
OK, these look like changes we want to merge into the master branch. Actually, we want to basically 'adopt' all these bash changes into master -- a wholesale import so that <tt>app-shells/bash</tt> in master looks exactly like that in experimental. To do this, we want to wipe out what is currently in the master branch related to <tt>app-shells/bash</tt>, and replace it entirely with the exact contents of <tt>app-shells/bash</tt> in the experimental branch.
+
It's a [http://www.linuxfromscratch.org/lfs/view/6.8/ Linux From Scratch] root filesystem natively built on top of an Aboriginal Linux system image.
  
This can be done as follows:
+
One of the example native-build.sh control-images Aboriginal Linux provides is "lfs-bootstrap.hdc", which natively compiles and installs (almost) all of the LFS chapter 6 packages under the emulated development environment, according to the LFS build instructions.
  
<console>
+
The control image copies the read-only root filesystem into a writeable directory under /home, chroots into that, builds packages and installs them over the existing (busybox) versions, and then tars up the resulting filesystem.
# ##i##git rm -rf app-shells/bash
+
# ##i##git checkout origin/experimental -- app-shells/bash
+
</console>
+
  
Now, let's review the changes git made. These are not yet committed:
+
This provides a much more capable native build environment for the targetSince Aboriginal Linux can already provide this for the supported targets, Funtoo uses this as its starting point instead of starting from the minimal build environment and building/installing all of its own prerequisites.
<console>
+
# ##i##git diff --cached --stat
+
  app-shells/bash/bash-3.1_p17.ebuild  |  150 -------------------------
+
app-shells/bash/bash-3.2_p51.ebuild  |  199 ---------------------------------
+
app-shells/bash/bash-4.0_p37.ebuild  |  193 --------------------------------
+
app-shells/bash/bash-4.0_p38.ebuild  |  193 --------------------------------
+
app-shells/bash/bash-4.1_p10.ebuild  |  191 -------------------------------
+
app-shells/bash/bash-4.1_p7-r1.ebuild |  189 -------------------------------
+
app-shells/bash/bash-4.1_p9-r1.ebuild |  189 -------------------------------
+
app-shells/bash/bash-4.2_p10.ebuild  |    5 +-
+
8 files changed, 2 insertions(+), 1307 deletions(-)
+
</console>
+
  
Looks good. These changes are already staged for commit -- notice the <tt>--cached</tt> option above. If you don't use <tt>--cached</tt>, you won't see any changes, because they're already cached for commit. Let's commit them:
+
= The plan of attack is as follows: =
  
<console>
+
# Start from an LFS build, plus necessary stuff for Portage to run.  (Git, python, portage tree, profile...)
# ##i##git commit -m "bash updates from experimental"
+
## Use the i686 target first, so we can test quickly via chroot.
# ##i##git push origin master
+
# Create a <tt>package.provided</tt> file for the local LFS system so that Portage doesn't complain of unsatisfied <tt>/var/db/pkg</tt> entries.
</console>
+
# Use Portage to build a stage1 tarball to /tmp/stage1root.
  
If we made any local changes to existing files that had not yet been added, and wanted to include those with the commit, we could use the <tt>-a</tt> option with <tt>git commit</tt>, above. Once the commit has been made, you should no longer see anything related to <tt>app-shells/bash</tt> listed when doing a diff of the branches.
+
Once we have a stage 1 for the target, Funtoo has been bootstrapped and we can extend this approach to bootstrap Funtoo on any architecture.
  
[[Category:Development]]
+
== Details ==
[[Category:Featured]]
+
 
[[Category:Tutorial]]
+
* Setup LFS-i686 filesystem
 +
 
 +
:wget http://landley.net/aboriginal/downloads/binaries/extras/lfs-bootstrap.tar.gz-i686
 +
:tar xvzf lfs-bootstrap.tar.gz-i686
 +
:sudo env -i HOST=i686 $(which chroot) lfs-bootstrap /sbin/init.sh
 +
 
 +
* Install python
 +
 
 +
:wget http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2
 +
:tar xf Python-2.7.2.tar.bz2
 +
:cd Python-2.7.2
 +
:./configure --prefix=/usr
 +
:make -j $CPUS
 +
:make install
 +
 
 +
* Install git
 +
 
 +
:wget http://git-core.googlecode.com/files/git-1.7.8.2.tar.gz
 +
:tar xzvf git-1.7.8.2.tar.gz
 +
:cd git-1.7.8.2
 +
:./configure --without-tcltk --without-python --without-iconv
 +
:make NO_PERL=1
 +
:make NO_PERL=1 install
 +
 
 +
* Install portage tool
 +
 
 +
:cd /usr/lib
 +
:git clone git://github.com/funtoo/portage-funtoo portage
 +
:cd portage
 +
:mkdir -p /usr/share/portage
 +
:ln -s /usr/lib/portage/cnf /usr/share/portage/config
 +
:for i in emerge ebuild; do ln -s /usr/lib/portage/bin/$i /usr/bin/$i; done
 +
 
 +
* Install portage tree, setup profiles
 +
 
 +
:cd /usr
 +
:tar xvzf portage_tarball.tar.gz
 +
:mv portage-system portage
 +
:ln -s /usr/portage/profiles/default/linux/$ARCH/2008.0 /etc/make.profile
 +
 
 +
* setup package.provided
 +
 
 +
:git clone git://github.com/funtoo/funtoo-overlay /var/tmp/funtoo-overlay
 +
:mkdir -p /etc/portage
 +
:cp /var/tmp/funtoo-overlay/funtoo/scripts/stage3.provided /etc/portage/package.provided
 +
 
 +
* Try to emerge something.  Notice it doesn't work.  Frown at it.
 +
** package.provided doesn't support USE flags.  What the...?
 +
 
 +
== Prerequisites ==
 +
 
 +
# A smaller Portage tree to not eat up so much filesystem space. This has now been implemented. The script <tt>/root/git/funtoo-overlay/funtoo/scripts/generate-system-tree.py</tt> can be used to generate a system-only Portage tree.
 +
## The dependencies are still circular and strange.
 +
### Libtool is an abomination.  It exists to make non-elf systems behave like ELF, and since Linux switched from a.out to ELF in 1996 (when it was 4 years old) there's nothing for Libtool to actually DO on Linux, yet it still manages not to do it properly.  Literally the ONLY thing installing Libtool on a Linux system does is introduce build breaks.  (Thank you, Free Software Foundation.)
 +
# A <tt>package.provided</tt> list of a current Funtoo system. This can be found at <tt>/root/git/funtoo-overlay/funtoo/scripts/stage3-provided.txt</tt>.
 +
#The profiles aren't target-agnostic.  There's no way to go
 +
"build for this host, I don't really care what it is".
 +
## You need an existing toolchain to build anything, and if it's gcc its "tuple" is available via "gcc -dumpmachine", so this is queryable at runtime.
 +
## See [http://www.funtoo.org/wiki/Funtoo_1.0_Profile] for ongoing work to genericize this.
 +
### the keywords=* stuff helps greatly, but make.conf moved into profiles and is still somewhat target-specific.  More cleanup needed here.  How do you build on an arbitrary host without human intervention (from a cron job)?
 +
 
 +
== Status ==
 +
 
 +
The approach above didn't seem to correctly provide needed dependencies. Options at this point:
 +
 
 +
# Bundle up what we have as a stage3 and see if Metro's steps are more successful at using it as a stage3 (we were trying simple emerge commands rather than the explicit metro steps for testing)
 +
## When you cd into the profile directory and emerge an ebuild file directly, it generally builds.  It's dependency calculation that's screwed up: package.provided injects a blank package with no USE flags, and anything that depends on a package with a USE flag tries to rebuild that package.  I.E. package.provided is essentially useless.
 +
### drobbins suggested something like "portage inject packagename USE="flag flag flag" [files...]" to retroactively notify portage of an installed package.  (Aboriginal Linux has a BINARY_PACKAGE_TARBALLS option using "touch timestamp; build package; find output -newer timestamp", and this could be inserted in the LFS build fairly easily.  Or just use a blank file list and not care that portage can't upgrade/uninstall the package, we just need the dependency generation to work.)
 +
# If package.provided is not working, simply copy <tt>/var/db/pkg</tt> from an existing stage3 for a massive injection of fake package information.
 +
# Or the option we are going to go with, which is stop trying to build a stage3, and build a minimal stage1 that we will feed to metro to take care of the rest.
 +
 
 +
[[Category:Labs]] [[Category:Projects]]

Latest revision as of 07:30, 14 February 2012

This page documents the effort to bootstrap Funtoo Linux using Aboriginal Linux.

What is Aboriginal Linux?

Aboriginal Linux's motto is "we cross compile so you don't have to". It replaces cross compiling with native compiling under an emulator (generally QEMU). Funtoo can use this to natively create stage 1/2/3 root filesystems on an arbitrary target, which greatly simplifies adding support for new architectures and regression testing existing architectures.

Aboriginal Linux creates a simple native development environment for each type of target hardware, building the smallest self-contained Linux system capable of rebuilding itself entirely from source code. This requires seven packages: linux, uClibc, busybox, binutils, gcc, make, and bash.

The resulting system can then boot under QEMU (or on appropriate hardware, if available) to provide a native development environment, eliminating the need for any further cross compiling. You can wget, configure, make, and install additional source packages inside the emulator.

Aboriginal Linux's web page is at http://landley.net/aboriginal and a long presentation about it is available at http://speakerdeck.com/u/mirell/p/developing-for-non-x86-targets-using-qemu

How do I use Aboriginal Linux?

Aboriginal Linux provides prebuilt native development environments for x86, x86-64, arm, mips, powerpc, and more, ready to run under the emulator. Install QEMU 1.0 or later, then go to http://landley.net/aboriginal/downloads/binaries", grab the appropriate system-image tarball, extract it, and run one of the following three shell scripts (included in the tarball):

  • ./run-emulator.sh - Boot to a shell prompt, with the simplest configuration: a read-only (squashfs) root filesystem with a tmpfs mounted on /home.
  • ./dev-environment.sh - Boot to a shell prompt with a full development environment. This wrapper around run-emulator.sh adds a 2 gigabyte ext2 /dev/hdb image mounted on /home (persistent writeable space, in large amounts), and allocates 256 megabytes of physical memory for the emulator (enough to run gcc building complex packages). If distccd and the appropriate target's cross compiler are available in the host's $PATH, this script also configures the emulated environment to call out to the cross compiler via distcc, transparently moving the heavy lifting of compilation outside of the emulator.
  • ./native-build.sh - Run an automated build. This requires a "control image", which is a squashfs filesystem the emulated system sees as /dev/hdc mounted on /mnt. The init script in the emulated root filesystem checks for /mnt/init and if that exists, it runs that instead of providing a shell prompt. (Actually it asks the user to press a key if they want a shell prompt, with a three second timeout.) See "http://landley.net/aboriginal/control-images" for details.

(You can build your own system images from source by following the instructions at "http://landley.net/aboriginal", but ReBootstrap does not require this.)

What is an "LFS build"?

It's a Linux From Scratch root filesystem natively built on top of an Aboriginal Linux system image.

One of the example native-build.sh control-images Aboriginal Linux provides is "lfs-bootstrap.hdc", which natively compiles and installs (almost) all of the LFS chapter 6 packages under the emulated development environment, according to the LFS build instructions.

The control image copies the read-only root filesystem into a writeable directory under /home, chroots into that, builds packages and installs them over the existing (busybox) versions, and then tars up the resulting filesystem.

This provides a much more capable native build environment for the target. Since Aboriginal Linux can already provide this for the supported targets, Funtoo uses this as its starting point instead of starting from the minimal build environment and building/installing all of its own prerequisites.

The plan of attack is as follows:

  1. Start from an LFS build, plus necessary stuff for Portage to run. (Git, python, portage tree, profile...)
    1. Use the i686 target first, so we can test quickly via chroot.
  2. Create a package.provided file for the local LFS system so that Portage doesn't complain of unsatisfied /var/db/pkg entries.
  3. Use Portage to build a stage1 tarball to /tmp/stage1root.

Once we have a stage 1 for the target, Funtoo has been bootstrapped and we can extend this approach to bootstrap Funtoo on any architecture.

Details

  • Setup LFS-i686 filesystem
wget http://landley.net/aboriginal/downloads/binaries/extras/lfs-bootstrap.tar.gz-i686
tar xvzf lfs-bootstrap.tar.gz-i686
sudo env -i HOST=i686 $(which chroot) lfs-bootstrap /sbin/init.sh
  • Install python
wget http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2
tar xf Python-2.7.2.tar.bz2
cd Python-2.7.2
./configure --prefix=/usr
make -j $CPUS
make install
  • Install git
wget http://git-core.googlecode.com/files/git-1.7.8.2.tar.gz
tar xzvf git-1.7.8.2.tar.gz
cd git-1.7.8.2
./configure --without-tcltk --without-python --without-iconv
make NO_PERL=1
make NO_PERL=1 install
  • Install portage tool
cd /usr/lib
git clone git://github.com/funtoo/portage-funtoo portage
cd portage
mkdir -p /usr/share/portage
ln -s /usr/lib/portage/cnf /usr/share/portage/config
for i in emerge ebuild; do ln -s /usr/lib/portage/bin/$i /usr/bin/$i; done
  • Install portage tree, setup profiles
cd /usr
tar xvzf portage_tarball.tar.gz
mv portage-system portage
ln -s /usr/portage/profiles/default/linux/$ARCH/2008.0 /etc/make.profile
  • setup package.provided
git clone git://github.com/funtoo/funtoo-overlay /var/tmp/funtoo-overlay
mkdir -p /etc/portage
cp /var/tmp/funtoo-overlay/funtoo/scripts/stage3.provided /etc/portage/package.provided
  • Try to emerge something. Notice it doesn't work. Frown at it.
    • package.provided doesn't support USE flags. What the...?

Prerequisites

  1. A smaller Portage tree to not eat up so much filesystem space. This has now been implemented. The script /root/git/funtoo-overlay/funtoo/scripts/generate-system-tree.py can be used to generate a system-only Portage tree.
    1. The dependencies are still circular and strange.
      1. Libtool is an abomination. It exists to make non-elf systems behave like ELF, and since Linux switched from a.out to ELF in 1996 (when it was 4 years old) there's nothing for Libtool to actually DO on Linux, yet it still manages not to do it properly. Literally the ONLY thing installing Libtool on a Linux system does is introduce build breaks. (Thank you, Free Software Foundation.)
  2. A package.provided list of a current Funtoo system. This can be found at /root/git/funtoo-overlay/funtoo/scripts/stage3-provided.txt.
  3. The profiles aren't target-agnostic. There's no way to go

"build for this host, I don't really care what it is".

    1. You need an existing toolchain to build anything, and if it's gcc its "tuple" is available via "gcc -dumpmachine", so this is queryable at runtime.
    2. See [1] for ongoing work to genericize this.
      1. the keywords=* stuff helps greatly, but make.conf moved into profiles and is still somewhat target-specific. More cleanup needed here. How do you build on an arbitrary host without human intervention (from a cron job)?

Status

The approach above didn't seem to correctly provide needed dependencies. Options at this point:

  1. Bundle up what we have as a stage3 and see if Metro's steps are more successful at using it as a stage3 (we were trying simple emerge commands rather than the explicit metro steps for testing)
    1. When you cd into the profile directory and emerge an ebuild file directly, it generally builds. It's dependency calculation that's screwed up: package.provided injects a blank package with no USE flags, and anything that depends on a package with a USE flag tries to rebuild that package. I.E. package.provided is essentially useless.
      1. drobbins suggested something like "portage inject packagename USE="flag flag flag" [files...]" to retroactively notify portage of an installed package. (Aboriginal Linux has a BINARY_PACKAGE_TARBALLS option using "touch timestamp; build package; find output -newer timestamp", and this could be inserted in the LFS build fairly easily. Or just use a blank file list and not care that portage can't upgrade/uninstall the package, we just need the dependency generation to work.)
  2. If package.provided is not working, simply copy /var/db/pkg from an existing stage3 for a massive injection of fake package information.
  3. Or the option we are going to go with, which is stop trying to build a stage3, and build a minimal stage1 that we will feed to metro to take care of the rest.