Difference between pages "Git Merging Guide" and "Forking An Ebuild"

(Difference between pages)
(Package Replacement: Funtoo Overlay (branch to branch))
 
m
 
Line 1: Line 1:
This page is here to show Funtoo Linux developers different techniques that can be used to merge various things.
+
Often, a Funtoo developer needs to fork an upstream ebuild. This is necessary when we want to apply fixes to it. This page will explain the concepts of forking and how this works in the context of Funtoo.
  
== Comparing Experimental and Master ==
+
== Portage Tree Generation ==
  
The best way to get a quick and dirty understanding of the differences between experimental and master is to do this:
+
Funtoo Linux generates its Portage tree using a special script that essentially takes a Gentoo tree as its starting point, and then applies various modifications to it. The modifications involve adding packages from various overlays, including our [https://github.com/funtoo/funtoo-overlay Funtoo-overlay]. Some packages added are brand new, while other packages are our special forked versions that replace existing packages.
  
<console>
+
In the vast majority of cases, when we fork a package, we take full responsibility for all ebuilds associated with that package, meaning that we have a full copy of the <tt>sys-foo/bar</tt> directory in one of our overlays.
# ##i##cd /root/git/funtoo-overlay
+
# ##i##git diff --stat origin/master origin/experimental
+
</console>
+
  
This will show a summary of what modifications where made on a file-by-file basis.
+
If you're interested in seeing the actual script that does all these things, take a look at the following files:
  
== Package Replacement: Funtoo Overlay (branch to branch) ==
+
; http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/current-update.sh: cronned script that calls <tt>merge.py</tt>.
 +
;http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge.py: python script that does the heavy lifting of combining Gentoo tree with various overlays, including our flora and funtoo-overlay. When we want to change what overlays we merge, what packages we exclude as a matter of policy (such as stale packages in some overlays), we make changes to this file.
 +
; http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge_utils.py: python module that contains classes and methods that implement the merging functionality.
  
When merging in funtoo-overlay, we might want to merge things from experimental to master. To do this, first pick a specific package to compare changes:
+
== Forking an Ebuild ==
  
<console>
+
In general, we fork ebuilds from Gentoo that we want to modify in some way. Before you fork an ebuild, it's important to understand that in general we fork entire packages, not just a single ebuild. This means that if you want to make some changes to <tt>sys-foo/bar</tt>, you are going to fork all <tt>sys-foo/bar</tt> ebuilds, and then Funtoo will be responsible for continuing to maintain these ebuilds until the package is unforked. Here are the steps we would use to fork <tt>sys-foo/bar</tt>:
# ##i##cd /root/git/funtoo-overlay
+
# ##i##git diff --stat origin/master origin/experimental app-shells/bash
+
app-shells/bash/bash-3.1_p17.ebuild   |  150 -------------------------
+
app-shells/bash/bash-3.2_p51.ebuild   |  199 ---------------------------------
+
app-shells/bash/bash-4.0_p37.ebuild  |  193 --------------------------------
+
app-shells/bash/bash-4.0_p38.ebuild  |  193 --------------------------------
+
app-shells/bash/bash-4.1_p10.ebuild  |  191 -------------------------------
+
app-shells/bash/bash-4.1_p7-r1.ebuild |  189 -------------------------------
+
app-shells/bash/bash-4.1_p9-r1.ebuild |  189 -------------------------------
+
app-shells/bash/bash-4.2_p10.ebuild  |    5 +-
+
8 files changed, 2 insertions(+), 1307 deletions(-)</console>
+
 
+
The "----" in the diff above shows that several ebuilds were removed ("----" means many lines were removed) in the experimental branch, and <tt>bash-4.2_p10.ebuild</tt> had slight modifications. This looks like a good candidate for grabbing from experimental to replace entirely what is in master. Here's an example of something that is ''not'' a good candidate for a wholesale replacement:
+
  
 +
# Find <tt>sys-foo/bar</tt> in you regular Portage tree. Make sure you have run <tt>emerge --sync</tt> recently to ensure it is up-to-date. If you want to fork from very recent changes that are not yet in our tree, you may need to grab the most recent Gentoo Portage tree to serve as your source for <tt>sys-foo/bar</tt> (this typically isn't necessary.)
 
<console>
 
<console>
# ##i##git diff --stat origin/master origin/experimental sys-apps/pciutils
+
# alias to recursively grab latest from Gentoo Portage tree WITHOUT history
sys-apps/pciutils/Manifest                        |    3 -
+
# usage: getgen gentoo-x86/dev-db/mongodb
sys-apps/pciutils/files/conf.d-pciparm            |  28 -------
+
alias getgen="cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot export -D$(date '+%Y-%m-%d')"
sys-apps/pciutils/files/init.d-pciparm            |  80 --------------------
+
.../files/pciutils-3.1.4-install-lib.patch        |  40 ----------
+
sys-apps/pciutils/files/pciutils-3.1.7-fbsd.patch  |  11 ---
+
.../files/pciutils-3.1.7-install-lib.patch        |  41 ----------
+
.../pciutils-3.1.8-avoid-segfault-on-init.patch    |  16 ----
+
sys-apps/pciutils/files/pciutils.cron              |    2 -
+
sys-apps/pciutils/pciutils-3.1.8-r1.ebuild        |  76 -------------------
+
9 files changed, 0 insertions(+), 297 deletions(-)
+
 
</console>
 
</console>
 +
# Copy the <tt>sys-foo/bar</tt> directory in its entirety to <tt>funtoo-overlay/sys-foo/bar</tt>.
 +
# Make any necessary modifications to <tt>funtoo-overlay/sys-foo/bar</tt>.
 +
# Perform some funtoo-ification steps prior to commit.
 +
# Add and commit the changes to funtoo-overlay.
 +
# Push changes to funtoo-overlay.
  
In this example above, <tt>sys-apps/pciutils</tt> had a lot of cleanups in experimental, but the output above indicates that there is a new <tt>pciutils-3.1.8-1.ebuild</tt> in master that is not experimental. If we replace what is in master with that in experimental, we will lose the new ebuild! So we wouldn't want to do a wholesale replacement in this case. Old ebuilds that disappear are cleanups, but new ebuilds that disappear are not. Be sure to pay attention to whether the ebuilds that are being removed are old or new.
+
At this point, the forked <tt>sys-foo/bar</tt> package will be part of funtoo-overlay. The next time our unified Portage tree is generated by <tt>merge.py</tt> (the one that users have in their <tt>/usr/portage</tt> and is updated via <tt>emerge --sync</tt>), your forked ebuild will be used in place of the Gentoo ebuild. Why is this? It is because our <tt>merge.py</tt> script has been defined with a policy that any ebuilds in funtoo-overlay will replace any existing Gentoo ebuilds if they exist. The mechanism of replacement is that our <tt>sys-foo/bar</tt> directory will be used in place of Gentoo's <tt>sys-foo/bar</tt> directory. So this is how the forking process works.
  
Back to our bash example. To inspect changes in more detail to make sure they are acceptable, specify the modified ebuild directly and drop the <tt>--stat</tt> option:
+
== Funtoo-ification ==
  
<console>
+
When we fork a package from Gentoo, we perform the following tweaks to the package directory before committing:
# ##i##git diff origin/master origin/experimental app-shells/bash/bash-4.2_p10.ebuild
+
diff --git a/app-shells/bash/bash-4.2_p10.ebuild b/app-shells/bash/bash-4.2_p10.ebuild
+
index 0c497ea..e603c15 100644
+
--- a/app-shells/bash/bash-4.2_p10.ebuild
+
+++ b/app-shells/bash/bash-4.2_p10.ebuild
+
@@ -37,7 +37,7 @@ SRC_URI="mirror://gnu/bash/${MY_P}.tar.gz $(patches)
+
+
LICENSE="GPL-3"
+
SLOT="0"
+
-KEYWORDS="~alpha ~amd64 ~arm ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~sparc-fbsd ~x86-fbsd"
+
+KEYWORDS="*"
+
IUSE="afs bashlogger examples mem-scramble +net nls plugins vanilla"
+
+
DEPEND=">=sys-libs/ncurses-5.2-r2
+
@@ -69,7 +69,6 @@ src_unpack() {
+
        cd lib/readline
+
        [[ ${READLINE_PLEVEL} -gt 0 ]] && epatch $(patches -s ${READLINE_PLEVEL} readline ${READLINE_VER})
+
        cd ../..
+
-     
+
        epatch "${FILESDIR}"/${PN}-4.1-document-system-bashrc.patch
+
}
+
+
@@ -104,7 +103,7 @@ src_compile() {
+
        myconf="${myconf} --with-curses"
+
+
        myconf="${myconf} --without-lispdir" #335896
+
-     
+
+
+
        use plugins && append-ldflags -Wl,-rpath,/usr/$(get_libdir)/bash
+
        econf \
+
                $(use_with afs) \
+
</console>
+
  
OK, these look like changes we want to merge into the master branch. Actually, we want to basically 'adopt' all these bash changes into master -- a wholesale import so that <tt>app-shells/bash</tt> in master looks exactly like that in experimental. To do this, we want to wipe out what is currently in the master branch related to <tt>app-shells/bash</tt>, and replace it entirely with the exact contents of <tt>app-shells/bash</tt> in the experimental branch.
+
# Removal of <tt>ChangeLog</tt>.
 
+
# Run <tt>ebuild foo-1.0.ebuild digest</tt> before committing. This will cause the <tt>Manifest</tt> file to be regenerated. Gentoo has a lot more entries in this file than we do, since we use mini-Manfiests that only include DIST listings (for distfiles only.) We want to commit our mini-Manifest (still called <tt>Manifest</tt>, just with less entries in it) rather than the one that came from Gentoo.
This can be done as follows:
+
# Edit the top of each ebuild, and remove all <tt>Copyright</tt> and <tt>$Header:</tt> lines at the top of the file. We have a LICENSE.txt and COPYRIGHT.txt file in the root of our Portage tree, which is easier to maintain than keeping all the years up-to-date in each ebuild. Also, the <tt>$Header:</tt> line is there for the CVS version control system in Gentoo which Funtoo does not use. ''The only comment that should remain on the top of the ebuild is the one stating that it is distributed under the GPLv2.''.
  
 
<console>
 
<console>
# ##i##git rm -rf app-shells/bash
+
# If you find yourself doing this often, place this function in your .bashrc, .zshrc, etc
# ##i##git checkout origin/experimental -- app-shells/bash
+
funtooize() {
</console>
+
    if [ -z "$1" ]; then
 +
        search_path='.'
 +
    else
 +
        search_path=$1
 +
    fi
  
Now, let's review the changes git made. These are not yet committed:
+
    find $search_path -type f -exec sed -i -e '/^# Copyright\|^# \$Header/d' {} +
<console>
+
    find $search_path -type f -name "ChangeLog*" -delete
# ##i##git diff --cached --stat
+
    find $search_path -type f -name '*.ebuild' -exec ebuild {} manifest \;
app-shells/bash/bash-3.1_p17.ebuild  | 150 -------------------------
+
}
app-shells/bash/bash-3.2_p51.ebuild  |  199 ---------------------------------
+
app-shells/bash/bash-4.0_p37.ebuild  |  193 --------------------------------
+
app-shells/bash/bash-4.0_p38.ebuild   |  193 --------------------------------
+
app-shells/bash/bash-4.1_p10.ebuild   |  191 -------------------------------
+
app-shells/bash/bash-4.1_p7-r1.ebuild |  189 -------------------------------
+
app-shells/bash/bash-4.1_p9-r1.ebuild |  189 -------------------------------
+
app-shells/bash/bash-4.2_p10.ebuild  |    5 +-
+
8 files changed, 2 insertions(+), 1307 deletions(-)
+
 
</console>
 
</console>
  
Looks good. These changes are already staged for commit -- notice the <tt>--cached</tt> option above. If you don't use <tt>--cached</tt>, you won't see any changes, because they're already cached for commit. Let's commit them:
+
Here are a few additional changes that you are allowed to make to any forked ebuilds:
 
+
<console>
+
# ##i##git commit -m "bash updates from experimental"
+
# ##i##git push origin master
+
</console>
+
  
If we made any local changes to existing files that had not yet been added, and wanted to include those with the commit, we could use the <tt>-a</tt> option with <tt>git commit</tt>, above. Once the commit has been made, you should no longer see anything related to <tt>app-shells/bash</tt> listed when doing a diff of the branches.
+
# Line length greater than 80 characters. Gentoo enforces an 80-character line length limit. We don't.
 +
# <tt>KEYWORDS</tt> of <tt>*</tt> and <tt>~*</tt>. Gentoo does not allow these shortcuts. We do. They allow you to say "all arches" and "all unstable arches" in a concise way. Gentoo doesn't allow these shortcuts because it's Gentoo's policy to have each arch team manually approve each package. We do not have this policy so we can use the shortcuts.
 +
# Use of <tt>4-python</tt> EAPI. We allow the use of this EAPI for enhanced python functionality.
  
 
[[Category:Development]]
 
[[Category:Development]]
[[Category:Featured]]
 
[[Category:Tutorial]]
 

Latest revision as of 05:25, May 30, 2015

Often, a Funtoo developer needs to fork an upstream ebuild. This is necessary when we want to apply fixes to it. This page will explain the concepts of forking and how this works in the context of Funtoo.

Portage Tree Generation

Funtoo Linux generates its Portage tree using a special script that essentially takes a Gentoo tree as its starting point, and then applies various modifications to it. The modifications involve adding packages from various overlays, including our Funtoo-overlay. Some packages added are brand new, while other packages are our special forked versions that replace existing packages.

In the vast majority of cases, when we fork a package, we take full responsibility for all ebuilds associated with that package, meaning that we have a full copy of the sys-foo/bar directory in one of our overlays.

If you're interested in seeing the actual script that does all these things, take a look at the following files:

http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/current-update.sh
cronned script that calls merge.py.
http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge.py
python script that does the heavy lifting of combining Gentoo tree with various overlays, including our flora and funtoo-overlay. When we want to change what overlays we merge, what packages we exclude as a matter of policy (such as stale packages in some overlays), we make changes to this file.
http://git.funtoo.org/funtoo-overlay/tree/funtoo/scripts/merge_utils.py
python module that contains classes and methods that implement the merging functionality.

Forking an Ebuild

In general, we fork ebuilds from Gentoo that we want to modify in some way. Before you fork an ebuild, it's important to understand that in general we fork entire packages, not just a single ebuild. This means that if you want to make some changes to sys-foo/bar, you are going to fork all sys-foo/bar ebuilds, and then Funtoo will be responsible for continuing to maintain these ebuilds until the package is unforked. Here are the steps we would use to fork sys-foo/bar:

  1. Find sys-foo/bar in you regular Portage tree. Make sure you have run emerge --sync recently to ensure it is up-to-date. If you want to fork from very recent changes that are not yet in our tree, you may need to grab the most recent Gentoo Portage tree to serve as your source for sys-foo/bar (this typically isn't necessary.)
# alias to recursively grab latest from Gentoo Portage tree WITHOUT history
# usage: getgen gentoo-x86/dev-db/mongodb
alias getgen="cvs -d :pserver:anonymous@anoncvs.gentoo.org:/var/cvsroot export -D$(date '+%Y-%m-%d')"
  1. Copy the sys-foo/bar directory in its entirety to funtoo-overlay/sys-foo/bar.
  2. Make any necessary modifications to funtoo-overlay/sys-foo/bar.
  3. Perform some funtoo-ification steps prior to commit.
  4. Add and commit the changes to funtoo-overlay.
  5. Push changes to funtoo-overlay.

At this point, the forked sys-foo/bar package will be part of funtoo-overlay. The next time our unified Portage tree is generated by merge.py (the one that users have in their /usr/portage and is updated via emerge --sync), your forked ebuild will be used in place of the Gentoo ebuild. Why is this? It is because our merge.py script has been defined with a policy that any ebuilds in funtoo-overlay will replace any existing Gentoo ebuilds if they exist. The mechanism of replacement is that our sys-foo/bar directory will be used in place of Gentoo's sys-foo/bar directory. So this is how the forking process works.

Funtoo-ification

When we fork a package from Gentoo, we perform the following tweaks to the package directory before committing:

  1. Removal of ChangeLog.
  2. Run ebuild foo-1.0.ebuild digest before committing. This will cause the Manifest file to be regenerated. Gentoo has a lot more entries in this file than we do, since we use mini-Manfiests that only include DIST listings (for distfiles only.) We want to commit our mini-Manifest (still called Manifest, just with less entries in it) rather than the one that came from Gentoo.
  3. Edit the top of each ebuild, and remove all Copyright and $Header: lines at the top of the file. We have a LICENSE.txt and COPYRIGHT.txt file in the root of our Portage tree, which is easier to maintain than keeping all the years up-to-date in each ebuild. Also, the $Header: line is there for the CVS version control system in Gentoo which Funtoo does not use. The only comment that should remain on the top of the ebuild is the one stating that it is distributed under the GPLv2..
# If you find yourself doing this often, place this function in your .bashrc, .zshrc, etc
funtooize() {
    if [ -z "$1" ]; then
        search_path='.'
    else
        search_path=$1
    fi

    find $search_path -type f -exec sed -i -e '/^# Copyright\|^# \$Header/d' {} +
    find $search_path -type f -name "ChangeLog*" -delete
    find $search_path -type f -name '*.ebuild' -exec ebuild {} manifest \;
}

Here are a few additional changes that you are allowed to make to any forked ebuilds:

  1. Line length greater than 80 characters. Gentoo enforces an 80-character line length limit. We don't.
  2. KEYWORDS of * and ~*. Gentoo does not allow these shortcuts. We do. They allow you to say "all arches" and "all unstable arches" in a concise way. Gentoo doesn't allow these shortcuts because it's Gentoo's policy to have each arch team manually approve each package. We do not have this policy so we can use the shortcuts.
  3. Use of 4-python EAPI. We allow the use of this EAPI for enhanced python functionality.