Difference between pages "Portage Tree" and "Forking An Ebuild"

(Difference between pages)
 
m
 
Line 1: Line 1:
This page describes the Funtoo Linux distributed Portage tree, which contains all the ebuilds of the Funtoo Linux distribution.  
+
Often, a Funtoo developer needs to fork an upstream ebuild. This is necessary when we want to apply fixes to it. This page will explain the concepts of forking and how this works in the context of Funtoo.
  
== Community-Extensible and Distributed ==
+
== Portage Tree Generation ==
  
Daniel Robbins has recently added the ability for the Portage tree to be extended and improved by the community. This allows Funtoo Linux users to contribute ebuilds of interest very easily, by simply committing their ebuilds to their own git-based portage repository. Your updated ebuilds are then auto-merged into the mainline Funtoo Portage tree every 4 hours. If you are interested in contributing to the Funtoo Linux Portage tree, please post an email to the [http://groups.google.com/group/funtoo-dev funtoo-dev mailing list] describing your overlay and its contents and we will consider the request.
+
Funtoo Linux generates its Portage tree using a special script that essentially takes a Gentoo tree as its starting point, and then applies various modifications to it. The modifications involve adding packages from various overlays, including our [https://github.com/funtoo/funtoo-overlay Funtoo-overlay]. Some packages added are brand new, while other packages are our special forked versions that replace existing packages.  
  
== Mini-Manifest Tree ==
+
In the vast majority of cases, when we fork a package, we take full responsibility for all ebuilds associated with that package, meaning that we have a full copy of the <tt>sys-foo/bar</tt> directory in one of our overlays.
  
Funtoo Linux provides two Portage tree repositories, available at the following location:
+
If you're interested in seeing the actual script that does all these things, take a look at the following files:
  
* http://github.com/funtoo/ports-2012
+
; http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/current-update.sh: cronned script that calls <tt>merge.py</tt>.
 +
;http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge.py: python script that does the heavy lifting of combining Gentoo tree with various overlays, including our flora and funtoo-overlay. When we want to change what overlays we merge, what packages we exclude as a matter of policy (such as stale packages in some overlays), we make changes to this file.
 +
; http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge_utils.py: python module that contains classes and methods that implement the merging functionality.
  
The first tree is a full Portage tree with full Manifests and ChangeLogs, and includes branches for Funtoo Linux as well as Gentoo Linux. The second tree is a minimized tree that contains only the unified Funtoo Linux Portage tree, all the ebuilds in the first tree, but contains "mini" Manifests (that only contain distfile digests) and has ChangeLogs removed. The first tree is used by Funtoo Linux internallly while the second tree is intended to be used by end-users, since it is much smaller.
+
== Forking an Ebuild ==
  
== How the Tree is Built ==
+
In general, we fork ebuilds from Gentoo that we want to modify in some way. Before you fork an ebuild, it's important to understand that in general we fork entire packages, not just a single ebuild. This means that if you want to make some changes to <tt>sys-foo/bar</tt>, you are going to fork all <tt>sys-foo/bar</tt> ebuilds, and then Funtoo will be responsible for continuing to maintain these ebuilds until the package is unforked. Here are the steps we would use to fork <tt>sys-foo/bar</tt>:
  
New versions of the Funtoo Linux Portage tree are generated every 4 hours using a special merge script that combines the Gentoo Portage tree with the Funtoo Linux funtoo-overlay as well as other 3rd-party overlays.  
+
# Find <tt>sys-foo/bar</tt> in you regular Portage tree. Make sure you have run <tt>emerge --sync</tt> recently to ensure it is up-to-date. If you want to fork from very recent changes that are not yet in our tree, you may need to grab the most recent Gentoo Portage tree to serve as your source for <tt>sys-foo/bar</tt> (this typically isn't necessary.)
 +
<console>
 +
# alias to recursively grab latest from Gentoo Portage tree WITHOUT history
 +
# usage: getgen gentoo-x86/dev-db/mongodb
 +
alias getgen="cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot export -D$(date '+%Y-%m-%d')"
 +
</console>
 +
# Copy the <tt>sys-foo/bar</tt> directory in its entirety to <tt>funtoo-overlay/sys-foo/bar</tt>.
 +
# Make any necessary modifications to <tt>funtoo-overlay/sys-foo/bar</tt>.
 +
# Perform some funtoo-ification steps prior to commit.
 +
# Add and commit the changes to funtoo-overlay.
 +
# Push changes to funtoo-overlay.
  
https://github.com/funtoo/funtoo-overlay/blob/master/funtoo/scripts/merge.py
+
At this point, the forked <tt>sys-foo/bar</tt> package will be part of funtoo-overlay. The next time our unified Portage tree is generated by <tt>merge.py</tt> (the one that users have in their <tt>/usr/portage</tt> and is updated via <tt>emerge --sync</tt>), your forked ebuild will be used in place of the Gentoo ebuild. Why is this? It is because our <tt>merge.py</tt> script has been defined with a policy that any ebuilds in funtoo-overlay will replace any existing Gentoo ebuilds if they exist. The mechanism of replacement is that our <tt>sys-foo/bar</tt> directory will be used in place of Gentoo's <tt>sys-foo/bar</tt> directory. So this is how the forking process works.
  
See [[How to Dev#Development Trees|How to Dev - Development Trees]] for more information.
+
== Funtoo-ification ==
  
If you are interested in contributing ebuilds to Funtoo Linux, please get involved on the bug tracker. See our main page for ways you can get involved and connect with the Funtoo community.
+
When we fork a package from Gentoo, we perform the following tweaks to the package directory before committing:
  
[[Category:Portage]]
+
# Removal of <tt>ChangeLog</tt>.
[[Category:Funtoo features]]
+
# Run <tt>ebuild foo-1.0.ebuild digest</tt> before committing. This will cause the <tt>Manifest</tt> file to be regenerated. Gentoo has a lot more entries in this file than we do, since we use mini-Manfiests that only include DIST listings (for distfiles only.) We want to commit our mini-Manifest (still called <tt>Manifest</tt>, just with less entries in it) rather than the one that came from Gentoo.
 +
# Edit the top of each ebuild, and remove all <tt>Copyright</tt> and <tt>$Header:</tt> lines at the top of the file. We have a LICENSE.txt and COPYRIGHT.txt file in the root of our Portage tree, which is easier to maintain than keeping all the years up-to-date in each ebuild. Also, the <tt>$Header:</tt> line is there for the CVS version control system in Gentoo which Funtoo does not use. ''The only comment that should remain on the top of the ebuild is the one stating that it is distributed under the GPLv2.''.
 +
 
 +
<console>
 +
# If you find yourself doing this often, place this function in your .bashrc, .zshrc, etc
 +
funtooize() {
 +
    if [ -z "$1" ]; then
 +
        search_path='.'
 +
    else
 +
        search_path=$1
 +
    fi
 +
 
 +
    find $search_path -type f -exec sed -i -e '/^# Copyright\|^# \$Header/d' {} +
 +
    find $search_path -type f -name "ChangeLog*" -delete
 +
    find $search_path -type f -name '*.ebuild' -exec ebuild {} manifest \;
 +
}
 +
</console>
 +
 
 +
Here are a few additional changes that you are allowed to make to any forked ebuilds:
 +
 
 +
# Line length greater than 80 characters. Gentoo enforces an 80-character line length limit. We don't.
 +
# <tt>KEYWORDS</tt> of <tt>*</tt> and <tt>~*</tt>. Gentoo does not allow these shortcuts. We do. They allow you to say "all arches" and "all unstable arches" in a concise way. Gentoo doesn't allow these shortcuts because it's Gentoo's policy to have each arch team manually approve each package. We do not have this policy so we can use the shortcuts.
 +
# Use of <tt>4-python</tt> EAPI. We allow the use of this EAPI for enhanced python functionality.
 +
 
 +
[[Category:Development]]

Latest revision as of 05:25, May 30, 2015

Often, a Funtoo developer needs to fork an upstream ebuild. This is necessary when we want to apply fixes to it. This page will explain the concepts of forking and how this works in the context of Funtoo.

Portage Tree Generation

Funtoo Linux generates its Portage tree using a special script that essentially takes a Gentoo tree as its starting point, and then applies various modifications to it. The modifications involve adding packages from various overlays, including our Funtoo-overlay. Some packages added are brand new, while other packages are our special forked versions that replace existing packages.

In the vast majority of cases, when we fork a package, we take full responsibility for all ebuilds associated with that package, meaning that we have a full copy of the sys-foo/bar directory in one of our overlays.

If you're interested in seeing the actual script that does all these things, take a look at the following files:

http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/current-update.sh
cronned script that calls merge.py.
http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge.py
python script that does the heavy lifting of combining Gentoo tree with various overlays, including our flora and funtoo-overlay. When we want to change what overlays we merge, what packages we exclude as a matter of policy (such as stale packages in some overlays), we make changes to this file.
http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge_utils.py
python module that contains classes and methods that implement the merging functionality.

Forking an Ebuild

In general, we fork ebuilds from Gentoo that we want to modify in some way. Before you fork an ebuild, it's important to understand that in general we fork entire packages, not just a single ebuild. This means that if you want to make some changes to sys-foo/bar, you are going to fork all sys-foo/bar ebuilds, and then Funtoo will be responsible for continuing to maintain these ebuilds until the package is unforked. Here are the steps we would use to fork sys-foo/bar:

  1. Find sys-foo/bar in you regular Portage tree. Make sure you have run emerge --sync recently to ensure it is up-to-date. If you want to fork from very recent changes that are not yet in our tree, you may need to grab the most recent Gentoo Portage tree to serve as your source for sys-foo/bar (this typically isn't necessary.)
# alias to recursively grab latest from Gentoo Portage tree WITHOUT history
# usage: getgen gentoo-x86/dev-db/mongodb
alias getgen="cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot export -D$(date '+%Y-%m-%d')"
  1. Copy the sys-foo/bar directory in its entirety to funtoo-overlay/sys-foo/bar.
  2. Make any necessary modifications to funtoo-overlay/sys-foo/bar.
  3. Perform some funtoo-ification steps prior to commit.
  4. Add and commit the changes to funtoo-overlay.
  5. Push changes to funtoo-overlay.

At this point, the forked sys-foo/bar package will be part of funtoo-overlay. The next time our unified Portage tree is generated by merge.py (the one that users have in their /usr/portage and is updated via emerge --sync), your forked ebuild will be used in place of the Gentoo ebuild. Why is this? It is because our merge.py script has been defined with a policy that any ebuilds in funtoo-overlay will replace any existing Gentoo ebuilds if they exist. The mechanism of replacement is that our sys-foo/bar directory will be used in place of Gentoo's sys-foo/bar directory. So this is how the forking process works.

Funtoo-ification

When we fork a package from Gentoo, we perform the following tweaks to the package directory before committing:

  1. Removal of ChangeLog.
  2. Run ebuild foo-1.0.ebuild digest before committing. This will cause the Manifest file to be regenerated. Gentoo has a lot more entries in this file than we do, since we use mini-Manfiests that only include DIST listings (for distfiles only.) We want to commit our mini-Manifest (still called Manifest, just with less entries in it) rather than the one that came from Gentoo.
  3. Edit the top of each ebuild, and remove all Copyright and $Header: lines at the top of the file. We have a LICENSE.txt and COPYRIGHT.txt file in the root of our Portage tree, which is easier to maintain than keeping all the years up-to-date in each ebuild. Also, the $Header: line is there for the CVS version control system in Gentoo which Funtoo does not use. The only comment that should remain on the top of the ebuild is the one stating that it is distributed under the GPLv2..
# If you find yourself doing this often, place this function in your .bashrc, .zshrc, etc
funtooize() {
    if [ -z "$1" ]; then
        search_path='.'
    else
        search_path=$1
    fi

    find $search_path -type f -exec sed -i -e '/^# Copyright\|^# \$Header/d' {} +
    find $search_path -type f -name "ChangeLog*" -delete
    find $search_path -type f -name '*.ebuild' -exec ebuild {} manifest \;
}

Here are a few additional changes that you are allowed to make to any forked ebuilds:

  1. Line length greater than 80 characters. Gentoo enforces an 80-character line length limit. We don't.
  2. KEYWORDS of * and ~*. Gentoo does not allow these shortcuts. We do. They allow you to say "all arches" and "all unstable arches" in a concise way. Gentoo doesn't allow these shortcuts because it's Gentoo's policy to have each arch team manually approve each package. We do not have this policy so we can use the shortcuts.
  3. Use of 4-python EAPI. We allow the use of this EAPI for enhanced python functionality.