Difference between pages "VagrantUp" and "Forking An Ebuild"

(Difference between pages)
(Quick Start)
 
m
 
Line 1: Line 1:
= Funtoo Virtual Machine Quickstart =
+
Often, a Funtoo developer needs to fork an upstream ebuild. This is necessary when we want to apply fixes to it. This page will explain the concepts of forking and how this works in the context of Funtoo.
  
== Intro ==
+
== Portage Tree Generation ==
  
As an alternative to the [[Funtoo Linux Installation]], you can get started quickly with Funtoo as a virtual machine. An open source tool called [https://github.com/jedi4ever/veewee VeeWee] automates the creation of [http://www.virtualbox.org/ VirtualBox] virtual machines just like Funtoo's [[Metro]] creates the latest stage3 tarballs of a Linux machine. So in effect, a virtual machine can be created from source using all open source tools and run in an open source Virtual Machine Manager (VMM).  Once created, the VeeWee virtual machines can be automated with another open source tool [http://vagrantup.com/ Vagrant], which can create/start/stop/delete machines as you wish.  Once you have all the tools setup its really quite addictive experimenting with different configurations and setups in a very time efficient manner.
+
Funtoo Linux generates its Portage tree using a special script that essentially takes a Gentoo tree as its starting point, and then applies various modifications to it. The modifications involve adding packages from various overlays, including our [https://github.com/funtoo/funtoo-overlay Funtoo-overlay]. Some packages added are brand new, while other packages are our special forked versions that replace existing packages.  
  
The Funtoo project will be experimenting with virtual machines as a delivery mechanism to users.
+
In the vast majority of cases, when we fork a package, we take full responsibility for all ebuilds associated with that package, meaning that we have a full copy of the <tt>sys-foo/bar</tt> directory in one of our overlays.
  
== Quick Start ==
+
If you're interested in seeing the actual script that does all these things, take a look at the following files:
  
You will need to install VirtualBox, Ruby and Gems to get going.  Recommend considering [http://beginrescueend.com/rvm/install/ Ruby Version Manager] instead of portage which will install all your Ruby needs into <tt>/opt/local/rvm</tt> or your home directory depending on if you are running as root or as a regular user.  We will be posting machines on the Funtoo mirrors. Pick a virtual machine off a nearby mirror.  The virtual machines are created from upstream [http://ftp.osuosl.org/pub/funtoo/funtoo-current/vagrant/x86-64bit/ current] and [http://ftp.osuosl.org/pub/funtoo/funtoo-stable/vagrant stable] 64 bit stage 3 tarballs. Put the URL of the machine (file ending in the '''.box''' extension) that you would like to experiment with into these easy commands.
+
; http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/current-update.sh: cronned script that calls <tt>merge.py</tt>.
 +
;http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge.py: python script that does the heavy lifting of combining Gentoo tree with various overlays, including our flora and funtoo-overlay. When we want to change what overlays we merge, what packages we exclude as a matter of policy (such as stale packages in some overlays), we make changes to this file.
 +
; http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge_utils.py: python module that contains classes and methods that implement the merging functionality.
  
 +
== Forking an Ebuild ==
 +
 +
In general, we fork ebuilds from Gentoo that we want to modify in some way. Before you fork an ebuild, it's important to understand that in general we fork entire packages, not just a single ebuild. This means that if you want to make some changes to <tt>sys-foo/bar</tt>, you are going to fork all <tt>sys-foo/bar</tt> ebuilds, and then Funtoo will be responsible for continuing to maintain these ebuilds until the package is unforked. Here are the steps we would use to fork <tt>sys-foo/bar</tt>:
 +
 +
# Find <tt>sys-foo/bar</tt> in you regular Portage tree. Make sure you have run <tt>emerge --sync</tt> recently to ensure it is up-to-date. If you want to fork from very recent changes that are not yet in our tree, you may need to grab the most recent Gentoo Portage tree to serve as your source for <tt>sys-foo/bar</tt> (this typically isn't necessary.)
 
<console>
 
<console>
# ##i## echo "app-emulation/virtualbox extensions" >> /etc/portage/package.use
+
# alias to recursively grab latest from Gentoo Portage tree WITHOUT history
# ##i## emerge vagrant
+
# usage: getgen gentoo-x86/dev-db/mongodb
# ##i## modprobe vboxdrv; modprobe vboxnetadp; modprobe vboxnetflt
+
alias getgen="cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot export -D$(date '+%Y-%m-%d')"
# ##i## vagrant box add funtoo http://ftp.osuosl.org/pub/funtoo/funtoo-current/vagrant/x86-64bit/vagrant-generic_64-funtoo-current-2012-01-26.box
+
# ##i## vagrant init
+
# ##i## vagrant up
+
# ##i## vagrant ssh
+
# ##i## sudo su -
+
 
</console>
 
</console>
 +
# Copy the <tt>sys-foo/bar</tt> directory in its entirety to <tt>funtoo-overlay/sys-foo/bar</tt>.
 +
# Make any necessary modifications to <tt>funtoo-overlay/sys-foo/bar</tt>.
 +
# Perform some funtoo-ification steps prior to commit.
 +
# Add and commit the changes to funtoo-overlay.
 +
# Push changes to funtoo-overlay.
  
== DevOps ToolChain ==
+
At this point, the forked <tt>sys-foo/bar</tt> package will be part of funtoo-overlay. The next time our unified Portage tree is generated by <tt>merge.py</tt> (the one that users have in their <tt>/usr/portage</tt> and is updated via <tt>emerge --sync</tt>), your forked ebuild will be used in place of the Gentoo ebuild. Why is this? It is because our <tt>merge.py</tt> script has been defined with a policy that any ebuilds in funtoo-overlay will replace any existing Gentoo ebuilds if they exist. The mechanism of replacement is that our <tt>sys-foo/bar</tt> directory will be used in place of Gentoo's <tt>sys-foo/bar</tt> directory. So this is how the forking process works.
  
The amount of tools needed is a bit large so a quick summary is given below.  The first few are required to get started, but you quickly end up looking at Puppet, Chef and other tools if the number of VMs get large.  Puppet and Chef aren't needed to get started, but are listed because VeeWee generated virtual machines normally have them preinstalled to push applications into freshly created virtual machines.  The [http://groups.google.com/group/devops-toolchain DevOps ToolChain email list] is a good place to get ideas on distributed systems processes, automation and tooling as the toolchain implementation can get quites complicated depending on your goals.
+
== Funtoo-ification ==
  
* [[Metro]] Tool used to build from source just enough Linux to create a Linux Funtoo OS
+
When we fork a package from Gentoo, we perform the following tweaks to the package directory before committing:
  
* [http://www.virtualbox.org/ VirtualBox] An open source Virtual Machine Manager (VMM) developed by Sun/Oracle which has several open source DevOps tools created around it for distributed Virtual Machine (VM) automation.
+
# Removal of <tt>ChangeLog</tt>.
 +
# Run <tt>ebuild foo-1.0.ebuild digest</tt> before committing. This will cause the <tt>Manifest</tt> file to be regenerated. Gentoo has a lot more entries in this file than we do, since we use mini-Manfiests that only include DIST listings (for distfiles only.) We want to commit our mini-Manifest (still called <tt>Manifest</tt>, just with less entries in it) rather than the one that came from Gentoo.
 +
# Edit the top of each ebuild, and remove all <tt>Copyright</tt> and <tt>$Header:</tt> lines at the top of the file. We have a LICENSE.txt and COPYRIGHT.txt file in the root of our Portage tree, which is easier to maintain than keeping all the years up-to-date in each ebuild. Also, the <tt>$Header:</tt> line is there for the CVS version control system in Gentoo which Funtoo does not use. ''The only comment that should remain on the top of the ebuild is the one stating that it is distributed under the GPLv2.''.
  
* [http://github.com/jedi4ever/veewee VeeWee] Tool used to automate the booting and execution of installation CDROM/ISO images. No need to watch Anaconda run to create Fedora boxes!  VeeWee, while a separate project, is a subcommand of Vagrant.  The Funtoo creation is currently merged into the [https://github.com/jedi4ever/veewee/tree/master/templates/funtoo-latest-generic_64-stable VeeWee codebase], but not currently in a released GEM. (Remark: Niklaus Giger had some problems with the templates and developed some [https://github.com/jedi4ever/veewee/pull/621 fixes])
+
<console>
 +
# If you find yourself doing this often, place this function in your .bashrc, .zshrc, etc
 +
funtooize() {
 +
    if [ -z "$1" ]; then
 +
        search_path='.'
 +
    else
 +
        search_path=$1
 +
    fi
  
* [http://vagrantup.com/ Vagrant] A tool which automated create/start/stop/delete operations of VirtualBox VMs.  VeeWee is needed for creating custom machine types, but there are lots of [http://vagrantbox.es/ pre-created machine types] in existence including Gentoo.  The Funtoo VeeWee scripts were derived from the Gentoo scripts.
+
    find $search_path -type f -exec sed -i -e '/^# Copyright\|^# \$Header/d' {} +
 
+
    find $search_path -type f -name "ChangeLog*" -delete
* [http://puppetlabs.com/puppet/ Puppet] is one of two distributed configuration management tools that are pre-installed by convention on VeeWee generated virtual machines.  There are enterprise and open source versions of [http://puppetlabs.com/puppet/compare/ Puppet].  Your Funtoo VirtualBox will have the open source Puppet installed by the [http://beginrescueend.com/rvm/install/ Ruby Version Manager] (not portage) as Gems and Portage fight each other (like CPAN and portage) and its best for now to keep these things separated.  The open source version of Puppet is typically distributed as a [http://rubygems.org/gems/puppet Ruby Gem].
+
    find $search_path -type f -name '*.ebuild' -exec ebuild {} manifest \;
 
+
}
* [http://www.opscode.com/chef/ Chef] is the second distributed configuration management tool that is pre-installed by convention on a VeeWee generated Funtoo virtual machine.  Chef also has open source and enterprise versions.  Your Funtoo VirtualBox will have the open source Chef installed by the [http://beginrescueend.com/rvm/install/ Ruby Version Manager] (not portage) as Ruby and Portage fight eachother and its best for now to keep these things separated.  The open source Chef is typically distributed as a [http://rubygems.org/gems/chef Ruby Gem].
+
</console>
 
+
== Security ==
+
 
+
The virtual machine is shipped built ready to script, but also wide open for attack.  The root and vagrant account's password is '''vagrant''' and the vagrant account has sudo access. Also, there is a private key in the vagrant account that is well known.  So don't put these boxes on the net without seriously hardening the machine's security or you will just be adding to the internet bot net problems.
+
 
+
== VM ideas ==
+
  
While the virtual machine was created with veewee for VirtualBox, there is no reason to restrict yourself to the VirtualBox virtual machine manager.  The ".box" file is a simple tarball.  In the tarball is a "box-disk1.vmdk" file which is a virtual machine disk image that is useable to the VirtualBox VMM.  Untaring the disk image into another VMM's virtual machine is certainly a doable thing.
+
Here are a few additional changes that you are allowed to make to any forked ebuilds:
  
Also, the vagrant and veewee upstream developers are looking into rewriting their code to work with other VMM implementions.
+
# Line length greater than 80 characters. Gentoo enforces an 80-character line length limit. We don't.
 +
# <tt>KEYWORDS</tt> of <tt>*</tt> and <tt>~*</tt>. Gentoo does not allow these shortcuts. We do. They allow you to say "all arches" and "all unstable arches" in a concise way. Gentoo doesn't allow these shortcuts because it's Gentoo's policy to have each arch team manually approve each package. We do not have this policy so we can use the shortcuts.
 +
# Use of <tt>4-python</tt> EAPI. We allow the use of this EAPI for enhanced python functionality.
  
[[Category:DevOps]]
+
[[Category:Development]]
[[Category:Virtualization]]
+

Latest revision as of 05:25, May 30, 2015

Often, a Funtoo developer needs to fork an upstream ebuild. This is necessary when we want to apply fixes to it. This page will explain the concepts of forking and how this works in the context of Funtoo.

Portage Tree Generation

Funtoo Linux generates its Portage tree using a special script that essentially takes a Gentoo tree as its starting point, and then applies various modifications to it. The modifications involve adding packages from various overlays, including our Funtoo-overlay. Some packages added are brand new, while other packages are our special forked versions that replace existing packages.

In the vast majority of cases, when we fork a package, we take full responsibility for all ebuilds associated with that package, meaning that we have a full copy of the sys-foo/bar directory in one of our overlays.

If you're interested in seeing the actual script that does all these things, take a look at the following files:

http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/current-update.sh
cronned script that calls merge.py.
http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge.py
python script that does the heavy lifting of combining Gentoo tree with various overlays, including our flora and funtoo-overlay. When we want to change what overlays we merge, what packages we exclude as a matter of policy (such as stale packages in some overlays), we make changes to this file.
http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge_utils.py
python module that contains classes and methods that implement the merging functionality.

Forking an Ebuild

In general, we fork ebuilds from Gentoo that we want to modify in some way. Before you fork an ebuild, it's important to understand that in general we fork entire packages, not just a single ebuild. This means that if you want to make some changes to sys-foo/bar, you are going to fork all sys-foo/bar ebuilds, and then Funtoo will be responsible for continuing to maintain these ebuilds until the package is unforked. Here are the steps we would use to fork sys-foo/bar:

  1. Find sys-foo/bar in you regular Portage tree. Make sure you have run emerge --sync recently to ensure it is up-to-date. If you want to fork from very recent changes that are not yet in our tree, you may need to grab the most recent Gentoo Portage tree to serve as your source for sys-foo/bar (this typically isn't necessary.)
# alias to recursively grab latest from Gentoo Portage tree WITHOUT history
# usage: getgen gentoo-x86/dev-db/mongodb
alias getgen="cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot export -D$(date '+%Y-%m-%d')"
  1. Copy the sys-foo/bar directory in its entirety to funtoo-overlay/sys-foo/bar.
  2. Make any necessary modifications to funtoo-overlay/sys-foo/bar.
  3. Perform some funtoo-ification steps prior to commit.
  4. Add and commit the changes to funtoo-overlay.
  5. Push changes to funtoo-overlay.

At this point, the forked sys-foo/bar package will be part of funtoo-overlay. The next time our unified Portage tree is generated by merge.py (the one that users have in their /usr/portage and is updated via emerge --sync), your forked ebuild will be used in place of the Gentoo ebuild. Why is this? It is because our merge.py script has been defined with a policy that any ebuilds in funtoo-overlay will replace any existing Gentoo ebuilds if they exist. The mechanism of replacement is that our sys-foo/bar directory will be used in place of Gentoo's sys-foo/bar directory. So this is how the forking process works.

Funtoo-ification

When we fork a package from Gentoo, we perform the following tweaks to the package directory before committing:

  1. Removal of ChangeLog.
  2. Run ebuild foo-1.0.ebuild digest before committing. This will cause the Manifest file to be regenerated. Gentoo has a lot more entries in this file than we do, since we use mini-Manfiests that only include DIST listings (for distfiles only.) We want to commit our mini-Manifest (still called Manifest, just with less entries in it) rather than the one that came from Gentoo.
  3. Edit the top of each ebuild, and remove all Copyright and $Header: lines at the top of the file. We have a LICENSE.txt and COPYRIGHT.txt file in the root of our Portage tree, which is easier to maintain than keeping all the years up-to-date in each ebuild. Also, the $Header: line is there for the CVS version control system in Gentoo which Funtoo does not use. The only comment that should remain on the top of the ebuild is the one stating that it is distributed under the GPLv2..
# If you find yourself doing this often, place this function in your .bashrc, .zshrc, etc
funtooize() {
    if [ -z "$1" ]; then
        search_path='.'
    else
        search_path=$1
    fi

    find $search_path -type f -exec sed -i -e '/^# Copyright\|^# \$Header/d' {} +
    find $search_path -type f -name "ChangeLog*" -delete
    find $search_path -type f -name '*.ebuild' -exec ebuild {} manifest \;
}

Here are a few additional changes that you are allowed to make to any forked ebuilds:

  1. Line length greater than 80 characters. Gentoo enforces an 80-character line length limit. We don't.
  2. KEYWORDS of * and ~*. Gentoo does not allow these shortcuts. We do. They allow you to say "all arches" and "all unstable arches" in a concise way. Gentoo doesn't allow these shortcuts because it's Gentoo's policy to have each arch team manually approve each package. We do not have this policy so we can use the shortcuts.
  3. Use of 4-python EAPI. We allow the use of this EAPI for enhanced python functionality.