Creating Python-related Ebuilds
Contents |
Quick Start
Progress overlay is now integrated into funtoo-experimental.
Progress Overlay currently contains 400+ packages that implement next-generation Python support for ebuilds. After switching active version of Python, users should change value of PYTHON_ABIS in make.conf, run emerge with -N / --newuse option to reinstall packages with PYTHON_ABIS USE flags and next run python-updater to reinstall old-style packages.
TODO: add cmdline examples
Building Python Modules with Portage
Conceptual Overview
Portage supports the ability to build python modules and extensions for all installed versions of python. This functionality has been implemented by python.eclass.
To use python.eclass, you first want to investigate how python module building is integrated into any included Makefile or build script. Since we want python.eclass to build these modules for us, typically the first step is to tweak the Makefile or configure script to explicitly *not* build python modules. We will be handling that process using python.eclass.
Progress Improvements
The old way: python-updater
With the current python.eclass in Gentoo and Funtoo Linux, python modules are built for all installed versions of python. However, no usable accounting information is stored by Portage so that it can "know" what versions of python the module was built for. This creates a problem when new versions of python are installed, as the new module will not be available for the new python, and Portage will not be aware of what needs to be done to fix this, if anything. This has resulted in the creation of a tool called python-updater to rebuild all necessary python-related ebuilds so that they reflect the currently-installed versions of python.
TODO: add python-updater example
The new way: new python eclass
The new python.eclass from Arfrever has a new API that can be used to write python ebuilds. When this API is used, Portage can track and resolve missing builds of python modules internally, so python-updater is not needed to be used for these ebuilds. However, until all ebuilds are converted to the new API, python-updater will need to be used to update these old-style ebuilds.
python-updater-0.10 doesn't reinstall any packages which set PYTHON_MULTIPLE_ABIS and EAPI="4-python", as manual rebuilding is not necessary.
TODO: document mechanisms, USE, etc. that are used to accomplish this and how it works
Help us document this page so we can help others convert to the new python API from progress overlay :)
Setting up progress overlay
Experimental branch users do not need to perform those steps!
Progress overlay now merged into experimental branch of funtoo-overlay. For master tree overlay can be easily added via layman. Progress overlay is a subversion repository, make sure dev-vcs/subversion is installed
# emerge subversion # layman -a progress
The following changes required for proper overlay work, edit /etc/make.conf
PYTHON_ABIS="2.7 3.2"for python-2.7.x and python-3.2x, respectively.
Notice, that progress overlay including an updated packages that also available in main portage tree!
# python-updaterthis still required to rebuild old-fashion python-related ebuilds
Updating the packages from progress overlay
Package updates are done the same way than before:
emerge -avuDN @world
Several python-related packages should be updated to progress versions, with the new PYTHON_ABIS flags.
Developer Docs
Ebuild Converstion to new Python eclass
Conversion from previous EAPIs and eclasses to the new progress python eclass is fairly straightforward. This section tries to explain the steps.
4-python EAPI
First, set EAPI to 4-python:
EAPI=4-python
Next, you will need to use new-style variables recognized by the new python eclass:
PYTHON_MULTIPLE_ABIS=1
Note the new variable names. PYTHON_MULTIPLE_ABIS turns on the functionality to build multiple versions of python modules for each version of python defined in PYTHON_ABIS. Note that PYTHON_ABIS defaults to "2.7 3.2" in Funtoo Linux profiles and can be overridden in /etc/make.conf if you would like the python eclass to target other Python ABIs (like jython, for example, or 2.6) that you might be interested in.
When PYTHON_MULTIPLE_ABIS is set and EAPI is set to 4-python, then new USE flags will be automatically added to the IUSE with the python_abis_ prefix. If you want to see what ABIs are supported, see the _PYTHON_GLOBALLY_SUPPORTED_ABIS variable defined in python.eclass.
PYTHON_RESTRICTED_ABIS
PYTHON_RESTRICTED_ABIS variable set in ebuild specifies a space-separated list of patterns of Python ABIs not supported by current package. The settings in PYTHON_RESTRICTED_ABIS will remove certain ABIs from the auto-IUSE expansion that happens when EAPI is set to 4-python and PYTHON_MULTIPLE_ABIS is set to 1 (This behavior is described above.) It will also limit the behavior of the python_abi_depend() function, described below. And of course, it will tell the python eclass to not build any python modules for these python versions, even if they happen to be installed on the system.
Use it like this, before the eclasses are inherited:
PYTHON_RESTRICTED_ABIS="3.*"
Note that PYTHON_RESTRICTED_ABIS is a glob-style pattern. If you want to have it build with everything except 3.2 and above and jython, use this pattern:
PYTHON_RESTRICTED_ABIS="3.[2-9] *-jython"
python_abi_depend()
One of the cool things about the new Python eclass is the "cascading" dependency functionality which is implemented via USE variables. Basically, if your new-style python module ebuild was built for python-2.7, and it has dependencies, then it can depend on that particular module it needs being built for the installed version of python-2.7. This happens by having your ebuild depend upon the python_abis_2.7 USE flag being set in the installed ebuild it depends upon.
For this functionality to work, the ebuild you depend on must be a new-style (EAPI 4-python) ebuild too, and must be enabled to support multiple python ABIS via the PYTHON_MULTIPLE_ABIS=1 setting. When these requirements are met, you will want to covert an old-style dependency that looks like this:
RDEPEND="foo >=dev-python/mydep-2 bar"
...to this new format:
RDEPEND="foo $(python_abi_depend ">=dev-python/mydep-2" ) bar"
The python_abi_depend function will expand to conditionally depend on the proper USE flags being set in dev-python/mydep-2 based on what USE flags are currently set for this ebuild when it builds. This is how the cascading functionality works. Use it in DEPEND, RDEPEND and PDEPEND as appropriate. You can include multiple ebuilds as arguments to python_abi_depend.
python_abi_depend() with -i and -e
python_abi_depend() has some advanced capabilities that you may need to take advantage of related to Python dependencies. Sometimes, your ebuild will only need a python module if certain versions of Python are installed. This functionality is typically needed in the following cases:
- module rolled into standard library
- A module is needed, but in newer versions of python it is part of the standard library (so there is no need to depend on it for these newer python versions)
- module required for fixes
- A module is needed, but only for newer versions of python, such as 3.x, where it implements compatibility features.
- optional module, not 100% compatible
- A module can be used, but the dependencies are not supported with certain versions of Python.
Let's look at the first case, a module your ebuild needs that was rolled into the standard library starting with Python version 3.1. In this case, we only need to depend on this module for Python ABIs 3.0 or less. We would use the -e option to exclude the versions we are not interested in:
RDEPEND="$(python_abi_depend -e "3.[1-9]" dev-python/mydep)"
In the second case, we depend on a module only for Python ABIs 3 and up, since it implements fixes for python3. In this case, we will use the -i option to include only those versions for which we want to depend:
RDEPEND="$(python_abi_depend -i "3.*" dev-python/mydep)"
Third case: something doesn't make sense in the context with pypy:
RDEPEND=">=sys-fs/udev-151 pygobject? ( $(python_abi_depend -e "*-pypy-*" dev-python/pygobject:2) ) pyqt4? ( $(python_abi_depend -e "*-pypy-*" dev-python/PyQt4) ) pyside? ( dev-python/pyside ) wxwidgets? ( $(python_abi_depend -e "3.* *-pypy-*" dev-python/wxpython) )"
