Portage (Funtoo)

Introduction

Project Details
Hosting	http://github.com/funtoo/portage-funtoo
Bug Reports	funtoo-dev@googlegroups.com

Portage is the official package manager of Funtoo Linux. Daniel Robbins maintains a slightly different version from upstream Gentoo Linux, with support for mini-manifests.

The recommended way to update your system

# emerge --sync
# emerge -auDN world --jobs=3

Portage Specifications, Changes, News

The latest progress and changes related to EAPI/PMS and Portage can often be found in the Gentoo Council meeting logs, which are listed on the Gentoo Council page.

The latest PMS specification to be approved by the Gentoo Council is available: eapi-3-approved-2010-01-18. The PMS specification is an attempt to codify the inner workings of Portage, and is authored by Ciaran McCreesh and Stephen Bennett as a Gentoo-hosted project.

Emerge Options

--with-bdeps=y: This option tells Portage to consider DEPEND for already-installed packages (that are not required to be considered) when updating your system. So when used with -uDN @world, the --with-bdeps=y option will cause any updated packages listed in DEPEND @world set's ebuilds -- not just those packages being installed/updated -- to be upgraded. However, if any updates are made to a package's DEPEND, it will not cause the specific package(s) containing the DEPEND to be rebuilt, so that they were explicitly built against the packages that were just upgraded. To Daniel Robbins, this seems like a half-step rather than the full upgrade step. It would be preferable if there were an option to cause any parent packages to be rebuilt if anything in DEPEND were upgraded. The reason for this is that if there is a change in DEPEND, then there is the potential for the parent ebuild to also be affected by these changes in any number of ways. So rebuilding against the most recently-upgraded DEPEND is generally safest.

Ebuild Variable Definitions

This section builds on information available in the The Gentoo Developer Guide and EAPI specifications. We will add information to this section to provide further clarification and details for Funtoo ebuild development.

Profile Variables

ARCH: ARCH defines the name for the underlying system architecture. As of late December 2010, Gentoo Linux supports defines the following architectures in various system profiles:

alpha amd64 amd64-fbsd arm cobalt fulong-2e hppa hppa-hpux ia64 ia64-hpux ip22 ip27 ip28 ip30 ip32 ip32r10k m68k m68k-mint mips mips-irix mips64 mips64el mipsel ppc ppc-aix ppc-macos ppc-openbsd ppc64 ppc64-macos s390 sgi sh sparc sparc-fbsd sparc-solaris sparc64 sparc64-fbsd sparc64-solaris x64-macos x64-openbsd x64-solaris x86 x86-cygwin x86-fbsd x86-interix x86-macos x86-netbsd x86-openbsd x86-solaris x86-winnt xbox yeeloong

This list was generated using the following command:

# cd /usr/portage/profiles
# grep -r ARCH= * | cut -f2 -d"=" | sed -e 's/"//g' | sort -u

Path Variables

PORTDIR: PORTDIR will be set to the path of the Portage directory that contains the ebuild that is currently executing, such as /usr/portage.

PORTAGE_TMPDIR: PORTAGE_TMPDIR defines the path to the location that Portage should use for all its temporary files, and defaults to /var/tmp.

DISTDIR: DISTDIR defines the path to the location of all distfiles (downloaded source code/artifacts), and defaults to /usr/portage/distfiles.

PKGDIR: PKGDIR defines the path to where Portage will store all built binary packages, and defaults to /usr/portage/packages.

RPMDIR: RPMDIR defines the path to where Portage will store all built binary RPM packages, and defaults to /usr/portage/rpm.

ROOT: ROOT defines the root of the filesystem to which packages should be installed, and may be set on the command-line. If not defined, Portage will set ROOT to / by the time your ebuild executes. When writing pkg_* ebuild functions, all your ebuild logic must respect the setting of the ROOT variable and compensate for situations when ROOT is not /. ROOT is used by stage1 builds to install to a sub-directory. ROOT=/tmp/stage1root emerge -e system is a command that might be run by a stage1 build. After src_install executes, Portage will always install the resultant files placed in $D to the filesystem defined by ROOT.

Prefix Variables

Portage supports the ability to act as a package manager for commercial operating systems such as MacOS X, or to install ebuilds on non-Gentoo systems, or to sub-directories on Gentoo or non-Gentoo systems. When used in this capacity, ebuilds are not installed to the root filesystem but are instead installed to a subdirectory, called a "prefix". Portage's prefix variables, available in EAPI 3, support this capability, and are designed to work with ebuilds that are maintained in a separate prefix overlay.

As part of the EAPI 3 prefix support, all ebuild helpers such as econf and einstall use $ED instead of $D. See below for full variable definitions.

EPREFIX: This variable is typically set in /etc/make.conf or make.globals, and defines the prefix that should be used by ebuilds. Prefix-aware ebuilds can reference this variable and use it appropriately. if EPREFIX is not defined, then EPREFIX will default to no value.

EROOT: EROOT is a more concise way of specifying $ROOT$EPREFIX, and is intended to be used in pkg_* ebuild functions.

ED: ED is a more concise way of specifying $D$EPREFIX, and is intended to be used in src_* ebuild functions.

For more information on Portage's prefix support, see the Gentoo Prefix Techdocs.

Path Variables For Your Use

WORKDIR: WORKDIR defines the temporary filesystem location that is used for extracting source code/artifacts, preparation, configuration and compilation, and will be defined for you by the time your ebuild executes, and its value must not be modified. By default, src_unpack will extract all source code/artifacts within WORKDIR. If you need to manually extract any archives, this must all be done somewhere within WORKDIR, T or D, with WORKDIR being the preferred location for doing this. WORKDIR is automatically cleaned up by Portage upon completion.

S: S defaults to $WORKDIR/$P, but may be redefined in your ebuild to be some other sub-directory of WORKDIR. Typically, S may be redefined to reflect the real directory name that is contained within the source tarball when it differs from the default of $WORKDIR/$P. When src_prepare, src_configure, src_compile, src_test and src_install run, they will attempt to set the current working directory to S if it exists, falling back to WORKDIR if S does not exist.

T: A temporary file storage location that is available to you during your ebuild's execution. T is automatically cleaned up by Portage upon completion.

D: D defines the filesystem tree to which the src_install function should install all files. All files defined here will be installed by Portage to the destination defined by ROOT.

FILESDIR: FILESDIR is shorthand for $PORTDIR/$CATEGORY/$PN/files -- the files/ sub-directory in the Portage tree for your particular ebuild. Typically used in src_prepare for applying patches, or by src_install for installing supplementary files such as initialization scripts that were committed to the Portage tree alongside your ebuild.

Metadata Variables

EAPI: Periodically, the Gentoo Linux project defines new functionality and behavior related to how ebuilds are handled by Portage. These changes may require you to write your ebuilds slightly differently, or may allow you to take advantage of new capabilities in Portage. In your ebuild, you can specify the EAPI you are targeted by setting the EAPI variable as the first line in the ebuild. If no EAPI is specified, an EAPI of 0 ("classic" behavior) is assumed. To view a complete definition of EAPIs 1 through 3, please see The Gentoo Developer Guide's EAPI page. Note that the information on EAPI 4 is incomplete, and more detailed information can be found on in-progress EAPI 4 document maintained by Zac Medico. At some point, it may be possible to set an EAPI on a repository or overlay-wide basis, but for now, it is set by each individual ebuild.

DEPEND: DEPEND is used to specify an ebuild's build dependencies, and is a list of all ebuilds (and their corresponding versions/slots) that are required to be installed to a bare stage3 (or @system set + dependencies) in order for this package to build (compile, etc.). If one or more of these ebuilds are not installed, then it must cause the building of this package to fail; otherwise it is not a true build dependency. Do not list any dependencies for packages that are included in the underlying bare stage3 unless specific versions of the core packages (such as gcc, or glibc) are required, since these packages are expected to be available on all systems. All dependencies listed in DEPEND are satisfied prior to this package being built.

RDEPEND: RDEPEND is used to specify an ebuild's runtime dependencies, and is a list of all ebuilds (and their corresponding versions/slots) that are required to be installed to a bare stage3 (or @system set + dependencies) in order for this package to run (function). If one or more of these ebuilds are not installed, then it must cause the execution of this package to fail; otherwise it is not a true runtime dependency. Do not list any dependencies for packages that are included in the bare stage3 unless specific versions of the core packages (such as glibc) are required, since these packages are expected to be available on all systems. The dependencies listed in RDEPEND may or not be installed prior to this package being built.

DEPEND and RDEPEND: Note that if a dependency needs to be available both before the package is built, and also after the package is built, then the dependency must be listed in both DEPEND and RDEPEND, as it is both a build and runtime dependency. This is common with many packages, including virtually all shared libraries, which must be installed prior to compilation for headers and linking purposes but must continue to be available after the package is installed so that it can function properly.

(R)DEPEND if installed: It is often useful for an ebuild to conditionally depend on a specific version of another package, if it happens to be installed. Take sys-fs/lvm2 and sys-fs/udev, for example. sys-fs/lvm2 may not always be installed, but sys-fs/udev is a required ebuild. It would be nice if we could specify in the sys-fs/udev ebuild that a specific or higher version of lvm2 is required to work with this particular version of udev, or specify in the lvm2 ebuild that a specific version or higher of udev is required to work with this version of lvm2. While this is not currently possible in Portage, it is possible to use the blocker functionality in Portage to define that earlier versions are not compatible, which essentially accomplishes the same thing. To specify that another package should not be installed alongside your ebuild, use the double-exclamation from of a blocker: "!!<sys-fs/udev-160" might be an example. This will prevent both ebuilds from being installed on the system at the same time. The double-exclamation blocker is supported in EAPI 2 and above.

IUSE: IUSE specifies the USE variables that are recognized by this ebuild. Note that changing IUSE in existing ebuilds should be handled carefully, as if another ebuild depends on a particular USE variable in IUSE being enabled in your ebuild, and you remove it, then Portage will not be able to satisfy this dependency. Similarly, if another ebuild depends on a particular USE variable in IUSE being disabled in your ebuild, and you remove it, then Portage will no longer be able to satisfy this dependency either. For this reason, it can become difficult to remove USE variables from IUSE once other ebuilds explicitly depend upon them being enabled or disabled.

SRC_URI: SRC_URI contains a list of all remote files that are used by this ebuild during the unpack/prepare/compile/install phases. Starting with EAPI 2, it is possible to rename source tarballs, which comes in handy when pulling down tagged sources from GitHub, using the following trick. This approach also makes it quite easy to release source tarballs for ebuilds, as all you need to do is tag the release and then push the tags to GitHub with a git push tags command:

GITHUB_USER="foo"
SRC_URI="https://github.com/${GITHUB_USER}/${PN}/tarball/${PV} -> ${PN}-git-${PV}.tgz"
 
src_prepare() {
  cd "${WORKDIR}"/${GITHUB_USER}-${PN}-*
  S="$(pwd)"
}

KEYWORDS

KEYWORDS is a general keywording system for ebuilds, but is now used exclusively by the Gentoo architecture teams to define whether a particular ebuild is marked as stable, unstable or masked for a particular system architecture. In a user's /etc/make.conf, ACCEPT_KEYWORDS variable is defined which specifies what specific keywords should be unmasked for this system by default. This value defaults to the stable tree of the current system architecture. On Funtoo Linux systems, ACCEPT_KEYWORDS is set to one of four values:

amd64: 64-bit PC-compatible stable tree
~amd64: 64-bit PC-compatible unstable tree
x86: 32-bit PC-compatible stable tree
~x86: 32-bit PC-compatible unstable tree

An ebuild uses the KEYWORDS variable to control what tree or trees the ebuild is part of, and will contain at most one setting for each system architecture. The KEYWORD settings are defined with the following meaning:

amd64: ebuild unmasked in 64-bit PC-compatible stable and unstable trees
~amd64: ebuild unmasked in 64-bit PC-compatible stable tree only
-amd64: ebuild masked for all 64-bit PC-compatible trees
x86: ebuild unmasked in 32-bit PC-compatible stable and unstable trees
~x86: ebuild unmasked in 32-bit PC-compatible stable tree only
-x86: ebuild masked for all 32-bit PC-compatible trees

An important thing to note is that an x86 or amd64 setting in the ebuild's KEYWORDS will unmask the ebuild for both unstable and stable tree systems -- that is, any ebuild made available to stable is also made available to unstable.

Additional wildcard keywords are supported:

*: unmask for all trees. This variable can be modified by appending negative keywords: "* -ia64" means "all keywords except (~)ia64
-*: mask for all trees. This variable can be modified by appending positive keywords: "-* amd64 x86" means "no keywords except for x86 and amd64 stable trees

Gentoo Linux KEYWORDS Policies

Due to the exclusive use of KEYWORDS by the Gentoo Linux architecture teams, a number of unfortunate policy-related checks have been integrated into the Portage source code. One is banning the use of KEYWORDS="*" to enable KEYWORDS for all available architectures, since it is Gentoo Linux policy for each architecture team to independently verify an ebuild one architecture at a time - this rule currently applies to all ebuilds, including architecture-independent ebuilds. These policy checks will result in warnings being displayed by Portage when using certain combinations of wildcard keywords. This also means that virtually all ebuilds originating from Gentoo Linux will specify an exhaustive, explicit list of all architecture trees for which this package has been unmasked, which will look something like this:

KEYWORDS="~alpha ~amd64 ~arm ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~sparc-fbsd ~x86 ~x86-fbsd"

Multiple ABI Support

Portage contains support for multiple ABIs (Application Binary Interfaces) co-existing on the same system. This functionality has been extensively documented and has been moved to its own page: Multiple ABI Support.

Ebuild Functions

Ebuilds provide the ability to define various shell functions that are used to specify various actions relating to building and installing a source or binary package on a user's system. These functions are defined below:

src_* functions

Ebuild functions starting with src_ are all related to creating the ebuild or package from source code/artifacts, and are defined below:

src_unpack

src_unpack is intended to be used to unpack the source code/artifacts that will be used by the other src_* functions. With EAPI 1 and earlier, it is also used for patching/modifying the source artifacts to prepare them for building, but with EAPI 2 or later the src_prepare function should be used for this instead. When src_unpack starts, the current working directory is set to $WORKDIR, which is the directory within which all source code/artifacts should be expanded. Note that the variable $A is set to the names of all the unique source files/artifacts specified in SRC_URI, and they will all be available in $DISTDIR by the time src_unpack starts. Also note that if no src_unpack function is specified, ebuild.sh will execute the following function for src_unpack by default:

src_unpack() {
  unpack ${A}
}

src_prepare

EAPI 2 and above support the src_prepare function, which is intended to be used for applying patches or making other modifications to the source code. When src_prepare starts, the current working directory is set to $S.

src_configure

EAPI 2 and above support the src_configure function, which is used to configure the source code prior to compilation. With EAPI 2 and above, the following default src_configure is defined if none is specified:

src_configure() {
	if [[ -x ${ECONF_SOURCE:-.}/configure ]] ; then
		econf
	fi
}

src_compile

This function defines the steps necessary to compile source code. With EAPI 1 and earlier, this function is also used to configure the source code prior to compilation. However, starting with EAPI 2, the src_configure function must be used for configuration steps instead of bundling them inside src_compile. In addition, starting with EAPI 2, there is now a default src_compile function that will be executed if none is defined in the ebuild:

src_compile() {
	if [ -f Makefile ] || [ -f GNUmakefile ] || [ -f makefile ] ; then
		emake || die "emake failed"
	fi
}

src_test

src_test is an interesting function - by default, an end-user's Portage does not have tests enabled. But if a user has test in FEATURES, or EBUILD_FORCE_TEST is defined, then ebuild.sh will attempt to run a test suite for this ebuild, by executing make check or make test if these targets are defined in the Makefile; otherwise, no tests will execute. If your Makefile supports make check or make test but the test suite is broken, then specify RESTRICT="test" in your ebuild to disable the test suite.

src_install

src_install is used by the ebuild writer to install all to-be-installed files to the $D directory, which can be treated like an empty root filesystem, in that ${D}/usr is the equivalent of the /usr directory, etc. When src_install runs, the Portage sandbox will be enabled, which will prevent any processes from creating or modifying files outside of the ${D} filesystem tree, and a sandbox violation will occur (resulting in the termination of the ebuild) if any such sandbox violation should occur. Once src_install has perfomed all necessary steps to install all to-be-installed files to $D, Portage will take care of merging these files to the filesystem specified by the $ROOT environment variable, which defaults to / if not set. When Portage merges these files, it will also record information about the installed package to /var/db/pkg/(cat)/$P. Typically, a src_install function such as this is sufficient for ensuring that all to-be-installed files are installed to $D:

src_install() {
  make DESTIDR="$D" install
}

pkg_* functions

An ebuild's functions starting with pkg_* take a wider view of the package lifecycle, and may be executed very early or very late in the build or package installation process. They are also all executed even if installing a Portage binary package, so are the intended place for defining any global configuration changes that are also required during binary package installation, such as user and group creation. When these functions are executed, the $ROOT variable will be defined to point to the target root filesystem to which the package is to be (or has been) installed. All logic inside pkg_* functions must function properly even if $ROOT is something other than /.

pkg_setup

The pkg_setup function is unusual in that it runs prior to any src_* function, and also runs prior to any other pkg_* function that runs when a binary package is installed, so it provides a useful place for the ebuild writer to perform any sanity checks, global configuration changes to the system (such as user/group creation) or set any internal global variables that are used by the rest of the ebuild. Using this function for defining global variables that are needed in multiple other functions is a useful way of avoiding duplicate code. You should also look to pkg_setup as the ideal place to put any logic that would otherwise linger in the main body of the ebuild, which should be avoided at all costs as it will slow down dependency calculation by Portage. Also remember that Portage can build binary packages, and this function is a good place to execute any steps that are required to run both prior to building an ebuild, and prior to installing a package. Also consider using pkg_preinst and pkg_postinst for this purpose.

pkg_pretend

The pkg_pretend function was added with EAPI 3, and it's the opinion of Daniel Robbins that the use of this function should be avoided. This function is especially unusual in that it is intended to be run during dependency calculation, and is intended to provide a polite mechanism to inform the user that a particular ebuild will fail due to a known incompatibility, typically a kernel incompatibility. That way, the user can know during emerge --pretend that a merge will fail. While this is useful, extending the dependency engine using bash is a very low-performance means to perform these tests. Therefore, The Funtoo core team recommends against using pkg_pretend. An extensible dependency engine would be a more appropriate and high-performance way to provide identical functionality.

pkg_preinst

The pkg_preinst function is called by Portage, prior to merging the to-be-installed files to the target filesystem specified by $ROOT environment variable (which defaults to /.) Keep in mind that these to-be-installed files were either just compiled and installed to $D by src_install, or they were just extracted from a .tbz2 binary package. The pkg_preinst function provides an ideal place to perform any "just before install" actions, such as user and group creation or other necessary steps to ensure that the package merges successfully. It also provides a potential place to perform any sanity checks related to installing the package to the target filesystem. If any sanity checks fail, calling die from this function will cause the package to not be installed to the target filesystem.

pkg_postinst

The pkg_postinst function is called by Portage prior to the package being installed to the target filesystem specified by $ROOT. This is a good place to perform any post-install configuration actions as well as print any informational messages for the user's benefit related to the package that was just installed.

pkg_prerm

The pkg_prerm function is called by Portage before an ebuild is removed from the filesystem.

pkg_postrm

The pkg_postrm function is called by Portage after an ebuild is removed from the filesystem.

Extra pre_ and post_ functions

Modern versions of Portage also support functions identical to the above functions but with pre_ and post_ at the beginning of the function name. For example, post_src_configure will be executed after src_configure and before src_compile. These additional functions are supported by all EAPIs, provided that the parent function is supported by the EAPI in use. The initial current working directory should be identical to the initial current working directory of the parent function.

Helper Functions

econf(): econf() is part of ebuild.sh and was originally intended to be a wrapper to the configure command that is typically used in the src_configure stage. Currently, econf() is ~84 lines and its behavior changes depending upon which EAPI you happen to be using. For this reason, the use of econf() is strongly discouraged at this time, and the Funtoo Linux core team recommends that you call ./configure directly.

Funtoo Portage Development

The Funtoo Core Team is currently working on a general-purpose plug-in system so that the GNU info file regeneration, news update display, and scanning of files that need updating in /etc can be pulled out of the official Portage emerge code. In addition, this plug-in system will allow other types of things to be hooked into various phases of emerge. This will allow various new plug-ins to be developed and used on systems, such as periodic security checks, etc.

Funtoo Features/Changes

emerge --sync from git: If a git-based Portage tree is already in place, emerge --sync will run "git pull" to update the underlying Portage tree.

Sed Wrapper Symlink and PATH fix: The Funtoo version of Portage has replaced the BSD-only sed wrapper with a symlink. This will eventually be deprecated. The sed wrapper was a way to provide BSD systems with a "sed" command that could be used inside ebuilds that worked similarly to GNU sed. A PATH fix has been applied so that /bin/sed will be detected first anyway.

mini-manifest: Funtoo's Portage supports a special mode of operation where Manifests only contain digests for distfiles and not for files in the Portage tree. This is to eliminate redundant digests since git already provides SHA1 digests. This feature is currently enabled by adding "mini-manifest" to FEATURES in /etc/make.conf but the intention is to eventually move it to a repo-specific option. Funtoo provides a special "mini-manifest" tree that is smaller than the full-size Portage tree, and is intended to be used with the mini-manifest feature.

preserve bindist through USE filtering: Normally, anything not in an ebuild's IUSE is stripped from the USE passed to ebuild.sh. This patch allows "bindist" to not be stripped, so it can be used as a means to disable pre-merge sanity checks that may exist in pkg_setup() and pkg_pretend() but will not otherwise affect the resultant build. If "bindist" will affect some functionality in the package, then it should be added to an ebuild's IUSE. This patch allows pkg_setup() and pkg_pretend() to look for "bindist", which indicates that the ebuild is being built for release, typically in an automated fashion, and thus runtime sanity checks that might otherwise run can be optionally skipped. This check is used by the udev-160-r1.ebuild to see if we should fail if we are merging udev on a system where the kernel will not support it. In Metro, this is not a big deal, but on a real production system, merging the udev on an incompatible system will render the kernel inoperable.

safetydance FEATURE: A new FEATURE setting is used by Funtoo's udev ebuild called "safetydance" which can be used to manually bypass sanity checks. This is an alternative to the "bindist" approach above. udev-160 in Funtoo Linux supports both approaches and Metro sets "safetydance" by default.

GLEP 55 removal: Some code to support GLEP 55 has been removed.

new metadata format (experimental): Some tweaks to ebuild.sh have been made so that it is easier to support new metadata formats in the future.

xz-utils auto-dependency: There are several ebuilds in the Gentoo Portage repository that use .xz files but do not explicitly depend on xz-utils. A workaround has been added to ebuild.sh to add this dependency to metadata automatically if a .xz file exists in SRC_URI. This change is not yet in the official Portage sources but is being used on the Funtoo side when generating our git-based Portage trees.

Commands

emerge: high-level dependency-based package merge/unmerge tool
ebuild: lower-level package build tool