Difference between pages "Clang" and "Creating Python-related Ebuilds"

(Difference between pages)
 
(Links)
 
Line 1: Line 1:
==Introduction==
+
== Summary ==
LLVM can be used as an alternative to GNU's compiler, GCC. The main benefit of using LLVM compilers instead of GCC is their lower memory usage, faster compile time and better diagnostics. There are some Benchmarks on the [http://clang.llvm.org/features.html#performance Clang] and [http://www.phoronix.com/scan.php?page=article&item=llvm3_gcc_open64 Phoronix] homepages.
+
  
It may happen that some programs do not compile (like glibc) because they depend on GCC-specific language extensions [http://gcc.gnu.org/onlinedocs/gcc/C-Extensions.html] (this is why the whole BSD code can be compiled with LLVM but some GNU code cannot) or segfault after successful compilation with LLVM (like xorg-server) but after following this guide, the system will still be able to compile packages with gcc. So if something goes wrong, it can be switched back to gcc for the particular package by uncommenting lines in /etc/make.conf and the bug should be reported.
+
Funtoo Linux incorporates Progress Overlay into the main Portage tree, which currently contains 400+ updated Python-related packages that implement next-generation Python support for ebuilds.
  
LLVM's C/C++ frontends clang and clang++ version 3.0 are stable enough to be self-hosting [http://blog.llvm.org/2010/02/clang-successfully-self-hosts.html] and compile Boost [http://blog.llvm.org/2010/05/clang-builds-boost.html], Qt [http://labs.qt.nokia.com/2010/10/29/compiling-qt-with-clang/], LibreOffice [http://wiki.documentfoundation.org/Development/Building_LibreOffice_with_Clang], FreeBSD [http://wiki.freebsd.org/BuildingFreeBSDWithClang], some parts of the Linux kernel [http://lwn.net/Articles/411654/] and more.
+
== Building Python Modules with Portage ==
  
Further, using LLVM 3.0 and up, there is a third way to compile with LLVM: the dragonegg package creates a gcc-plugin, that uses LLVM's optimizers but parses the code and creates binaries with gcc, which means that everything that compiles and works with gcc should work with dragonegg also. This plugin can be enabled by using a single CFLAG. Since LLVM 3.0 the old llvm-gcc package is deprecated and replaced by dragonegg, so it will disappear from portage with llvm version 2.9.
+
== Conceptual Overview ==
  
==LLVM Frontends==
+
Portage supports the ability to build python modules and extensions for all installed versions of python. This functionality has been implemented by <tt>python.eclass</tt>.
To be able to compile some sourcecode of a specific language, LLVM needs an appropriate frontend. There are clang, llvm-gcc and dragonegg in portage.
+
  
The goal of the Clang project is to create a new C, C++, Objective C and Objective C++ front-end for the LLVM compiler.
+
To use <tt>python.eclass</tt>, you first want to investigate how python module building is integrated into any included <tt>Makefile</tt> or build script. Since we want <tt>python.eclass</tt> to build these modules for us, typically the first step is to tweak the <tt>Makefile</tt> or <tt>configure</tt> script to explicitly *not* build python modules. We will be handling that process using <tt>python.eclass</tt>.
  
llvm-gcc is a modified version of gcc that compiles C/ObjC programs into native objects, LLVM bitcode or LLVM assembly language, depending upon the options. As written in the previous section, dragonegg replaced llvm-gcc in version 3.0.
+
Another option, for those Python packages with a <tt>setup.py</tt> in the root source directory, is to use the <tt>distutils</tt> eclass. In this case, you generally do not need custom <tt>src_compile</tt> or <tt>src_install</tt> stages, as the eclass takes care of everything for you. However, you will still need to set the proper environment variables described below, as they are needed by <tt>python.eclass</tt> which is used by <tt>distutils.eclass</tt>. You will also want to set dependencies in <tt>DEPEND</tt> and <tt>RDEPEND</tt> as described in this article.
  
So after installing llvm, clang and dragonegg, you will be able to choose between gcc and llvm whenever you like or use them both at the same time.
+
== Progress Improvements ==
  
== Install LLVM and its Frontends ==
+
This section describes the differences in approach between the old (classic) way of handling Python-related ebuilds, and the new way, which uses the updated eclass. Also note that the <tt>distutils</tt> eclass benefits from these updates as well, as it uses <tt>python.eclass</tt> behind-the-scenes.
Simply emerge the packages on ~arch systems. On arch systems you have to unmask some packages first. dragonegg requires gcc's ''lto'' USE-flag to be set and works with gcc 4.5 and gcc 4.6.
+
<console>
+
###i## emerge llvm clang dragonegg
+
</console>
+
Note, that for clang++ the C++ headers search path is hardcoded to the active gcc profile.
+
If you change the active gcc profile, or update gcc to a new version, you will have to remerge clang to update the search path.
+
  
To use dragonegg, run gcc as usual, with an extra command line argument "-fplugin=/usr/lib/llvm/dragonegg.so"
+
=== The old way: python-updater ===
If you change the active gcc profile, or update gcc to a new version, you will have to remerge dragonegg to update the plugin.
+
  
After the installation, check which CPUs are supported by using the command
+
With the older <tt>python.eclass</tt> in Gentoo (and that used to be in Funtoo Linux,) python modules are built for all installed versions of Python. However, no usable accounting information was stored by Portage so that it could "know" what versions of python the module was built for. This created a problem when new versions of Python are installed, as the new module would not be available for the new python, and Portage would not be aware of what needed to be done to fix this, if anything. This resulted in the creation of a tool called <tt>python-updater</tt> to rebuild all necessary python-related ebuilds so that they reflected the currently-installed versions of Python.
<console>
+
###i## llvm-as < /dev/null | llc -mcpu=help
+
</console>
+
and then add the following lines to ''<code>/etc/make.conf</code>'' (uncommenting the lines you need) to enable compilation via LLVM, adapting the march-option according to the previous command:
+
  
in ''<code>/etc/make.conf</code>'':
+
Note that python-updater-0.10 and later are aware of the new Python eclass, and will not unnecessarily reinstall any packages which set <tt>PYTHON_MULTIPLE_ABIS</tt> and <tt>EAPI="4-python"</tt>, as manual rebuilding is not necessary.
<console>
+
###i## nano /etc/portage/make.conf
+
# LLVM
+
#CC="/usr/bin/clang"
+
#CXX="/usr/bin/clang++"
+
  
# llvm-gcc for C++ code and fortran
+
'''TODO: add python-updater example'''
# llvm-gcc is deprecated and only used with LLVM 2.9
+
#CC="/usr/bin/llvm-gcc"
+
#CXX="/usr/bin/llvm-g++"
+
#CPP="/usr/bin/llvm-cpp"
+
#F77="/usr/bin/llvm-gfortran"
+
  
# Flags for clang: Insert your arch here instead of k8 and have a look at the manpage of clang for flag descriptions.
+
=== The new way: new python eclass ===
# Some gcc flags like -pipe and -pthread also work, though they might be ignored by clang.
+
#CFLAGS="-march=k8 -O2"
+
  
# Flags for dragonegg; just use all the gcc flags you like and append -fplugin=/path/to/dragonegg.so
+
The new <tt>python.eclass</tt> from Arfrever has a new API that can be used to write Python ebuilds. When this API is used, Portage can track and resolve missing builds of Python modules internally, so <tt>python-updater</tt> is not needed to be used for these ebuilds. However, until all ebuilds are converted to the new API, <tt>python-updater</tt> will need to be used to update these old-style ebuilds. Fortunately for us, over 400 Python-related ebuilds have been converted to take advantage of the new eclass, with more added weekly.
#CFLAGS="-march=k8 -O2 -fplugin=/usr/lib64/llvm/dragonegg.so"
+
</console>
+
  
{{Note}} Have a look at clang's manpages for additional information. If you get errors that your compiler cannot produce code, you should check your flags, e.g. don't use -O4 -flto -S or stuff like that; the examples above will work.
+
== Setting up progress overlay ==
  
== Using clang with portage ==
+
{{fancynote|Progess overlay is now merged in Funtoo current and stable - No manual steps are needed to benefit from it!}}
  
Although Gentoo package tree is not designed to be used with compiler other than GCC, clang can be enforced on most of the packages through ''CC'' and ''CXX'' variables.
+
Notice, that progress overlay including an updated packages that also available in main portage tree!
 +
<pre># python-updater</pre> this still required to rebuild old-fashion python-related ebuilds
  
Please note, however, that many of Gentoo packages still don't build with clang and a few don't work correctly after being built. That's why we suggest using ''/etc/portage/env'' file to enable the use of clang per-package.
 
  
In order to do that, first create a new environment override to use:
+
== Updating the packages from progress overlay ==
 +
Package updates are done the same way than before:
 +
<pre>emerge -avuDN @world</pre>
  
in ''<code>/etc/portage/env/clang</code>'':
+
Several python-related packages should be updated to progress versions, with the new PYTHON_ABIS flags.
<console>
+
###i## nano /etc/portage/env/clang
+
CC=clang
+
CXX=clang++
+
</console>
+
  
Then you can enable use of clang for packages using ''[[:/etc/portage/env|/etc/portage/package.env]]'' file:
+
== Developer Docs ==
in ''<code>/etc/portage/package.env</code>'':
+
<console>
+
###i## nano /etc/portage/package.env
+
app-foo/bar clang
+
app-bar/baz clang
+
</console>
+
  
If you want to use clang by default you can and need to specify some core packages. Here is small list of core packages that are currently failing on clang, but not that could be outdated:
+
=== Ebuild Converstion to new Python eclass ===
  
You need to add ''<code>/etc/portage/env/gcc</code>'':
+
Conversion from previous EAPIs and eclasses to the new progress python eclass is fairly straightforward. This section tries to explain the steps.
<console>
+
###i## nano /etc/portage/env/gcc
+
CC=gcc
+
CXX=g++
+
</console>
+
  
in addition, it is recommend to add compiler flags there:
+
==== 4-python EAPI ====
<console>
+
###i## nano /etc/portage/env/gcc
+
CFLAGS="-O2 -march=native -mtune=native -pipe"
+
CXXFLAGS="-O2 -march=native -mtune=native -pipe"
+
LDFLAGS="-Wl,--as-needed"
+
#You can disable gold link here
+
#EXTRA_ECONF="--enable-gold=default"
+
</console>
+
  
And in ''<code>/etc/portage/package.env</code>'':
+
First, set EAPI to 4-python:
<console>
+
###i## nano /etc/portage/package.env
+
#---------------CORE PACKAGES TO BUILD WITH GCC:
+
sys-apps/which gcc
+
sys-fs/reiserfsprogs gcc
+
sys-libs/ncurses gcc
+
sys-libs/zlib gcc
+
sys-apps/busybox gcc
+
sys-fs/e2fsprogs gcc
+
sys-devel/binutils gcc
+
sys-libs/glibc gcc
+
sys-devel/dragonegg gcc
+
dev-libs/openssl gcc
+
sys-boot/grub gcc
+
#---------------USER PACKAGES TO BUILD WITH GCC:
+
sys-apps/pacman gcc
+
www-client/firefox gcc
+
x11-libs/cairo gcc
+
media-libs/mesa gcc
+
</console>
+
  
If you have {{Package|app-portage/flaggie}} installed, you can modify ''<code>/etc/portage/package.env</code>'' by running the following:
+
<pre>
<console>
+
EAPI=4-python
###i## flaggie app-foo/bar app-bar/baz +clang
+
</pre>
</console>
+
  
== Enabling link-time optimizations ==
+
Next, you will need to understand the type of Python application or module you are trying to install. The primary determining factor in how to proceed is to see whether the Python application installs anything into Python's <tt>site-packages</tt> directory.  Essentially all Python modules do this, and some applications do this as well. If a package installs stuff into <tt>site-packages</tt>, then you should have its ebuild take advantage of the Python eclass "multiple ABI" support.
  
The ''link-time optimization'' feature defers optimizing the resulting executables to linking phase. This can result in better optimization of packages but is unsupported in Gentoo, and many packages simply fail to build.
+
In this context, "multiple ABI" refers to potentially multiple versions of Python that may be installed on your system. Each installed major.minor version of Python, plus Python derivatives like Jython and pypy, will have their own <tt>site-packages</tt> directory and the Python eclass can take care of installing a version of the Python modules for each version of Python on your system. If the application has binaries, a Python wrapper will also be installed that will hand off control to the proper version of Python automatically.
  
When using LTO, clang compiles units into LLVM byte-code rather than machine code. In order to support linking such object files, the [[gold]] linker must be installed and set as the default linker, as it does support plugins.
+
==== Non-Multiple-ABI Support ====
  
Similarly, ''ar'' needs plugin support as well. Sadly, binutils ar doesn't support passing '--plugin'' option before the actual command. Thus, we need to create a wrapper for it:
+
If you are not supporting multiple ABIs, which is the least common option, you will generally just set <tt>PYTHON_DEPEND</tt> to refer to the version(s) of Python you require. Documentation for <tt>PYTHON_DEPEND</tt> syntax (which has been changed in the new python eclass) is available [http://people.apache.org/~Arfrever/Gentoo_Python_Supplementary_Guide_for_Developers here].
  
in ''<code>/usr/local/bin/clang-ar</code>'':
+
==== Multiple ABI Support ====
<console>
+
###i## nano /usr/local/bin/clang-ar
+
#!/bin/sh
+
firstarg=${1}
+
shift
+
  
exec /usr/bin/ar "${firstarg}" --plugin /usr/lib/llvm/LLVMgold.so "${@}"
+
So, if you are using multiple ABI support, which will be the case for all Python modules, proceed as follows. Set <tt>PYTHON_MULTIPLE_ABIS</tt> prior to any inherit lines:
</console>
+
  
If that's done, you can create a new environment override profile for LTO-enabled clang:
+
<pre>
 +
PYTHON_MULTIPLE_ABIS=1
 +
</pre>
  
in ''<code>/etc/portage/env/clang-lt</code>'':
+
Note the new variable names. <tt>PYTHON_MULTIPLE_ABIS</tt> turns on the functionality to build multiple versions of python modules for each version of python defined in the user's <tt>PYTHON_ABIS</tt> setting. Note that <tt>PYTHON_ABIS</tt> defaults to "<tt>2.7 3.2</tt>" in Funtoo Linux profiles and can be overridden in <tt>/etc/make.conf</tt> if you would like the python eclass to target other Python ABIs (like jython, for example, or 2.6) that you might be interested in.
<console>
+
CC='clang'
+
CXX='clang++'
+
CFLAGS="${CFLAGS} -O4"
+
CXXFLAGS="${CXXFLAGS} -O4"
+
LDFLAGS="${LDFLAGS} -O4 -Wl,-plugin,/usr/lib/llvm/LLVMgold.so"
+
AR='/usr/local/bin/clang-ar'
+
RANLIB=':'
+
NM='nm --plugin /usr/lib64/llvm/LLVMgold.so'
+
</console>
+
  
Note that the link-time optimizations were indirectly enabled here via ''-O4''. If you don't want to enable other optimizations enforced by ''-O3'', please use ''-flto'' instead. You need to also pass optimization flags when linking because that's where clang needs them.
+
When <tt>PYTHON_MULTIPLE_ABIS</tt> is set and <tt>EAPI</tt> is set to <tt>4-python</tt>, then new USE flags will be automatically added to the <tt>IUSE</tt> with the <tt>python_abis_</tt> prefix.
 +
If you want to see what ABIs are supported, see the <tt>_PYTHON_GLOBALLY_SUPPORTED_ABIS</tt> variable defined in <tt>python.eclass</tt>.
  
You may also need to adjust the libdir path to plugin. Newer (live) versions of clang add `-plugin` when linking automatically, so `-Wl,-plugin`… is no longer necessary.
+
Also note that for multiple ABI support, <tt>PYTHON_DEPEND</tt> is set to a reasonable default value so you should almost always never need to set it yourself.
  
== Using clang with distcc ==
+
==== PYTHON_RESTRICTED_ABIS ====
 +
 
 +
<tt>PYTHON_RESTRICTED_ABIS</tt> variable set in the ebuild specifies a space-separated list of patterns of Python ABIs ''not'' supported by the current package. The settings in <tt>PYTHON_RESTRICTED_ABIS</tt> will remove certain ABIs from the auto-<tt>IUSE</tt> expansion that happens when <tt>EAPI</tt> is set to <tt>4-python</tt> and <tt>PYTHON_MULTIPLE_ABIS</tt> is set to <tt>1</tt> (This behavior is described above.) It will also limit the behavior of the <tt>python_abi_depend()</tt> function, described below. And of course, it will tell the python eclass to not build any python modules for these python versions, even if they happen to be installed on the system.
 +
 
 +
Use it like this, before the eclasses are inherited:
 +
 
 +
<pre>
 +
PYTHON_RESTRICTED_ABIS="3.*"
 +
</pre>
 +
 
 +
Note that <tt>PYTHON_RESTRICTED_ABIS</tt> is a glob-style pattern. If you want to have it build with everything ''except'' 3.2 and above and jython, use this pattern:
 +
 
 +
<pre>
 +
PYTHON_RESTRICTED_ABIS="3.[2-9] *-jython"
 +
</pre>
 +
 
 +
Note that the wildcard you should use to disable Jython is <tt>*-jython</tt>, while the wildcard you should use to disable PyPy is <tt>*-pypy-*</tt>.
 +
 
 +
==== python_abi_depend() ====
 +
 
 +
One of the cool things about the new Python eclass is the "cascading" dependency functionality which is implemented via USE variables. Basically, if your new-style python module ebuild was built for python-2.7, and it has dependencies, then it can depend on the module it needs being built ''for'' the installed version of python-2.7 specifically. This happens by having your ebuild depend upon the <tt>python_abis_2.7</tt> <tt>USE</tt> flag being set in the installed ebuild it depends upon.
 +
 
 +
For this functionality to work, the ebuild you depend on must be a new-style (<tt>EAPI 4-python</tt>) ebuild too, and must be enabled to support multiple python ABIS via the <tt>PYTHON_MULTIPLE_ABIS=1</tt> setting. When these requirements are met, you will want to covert an old-style dependency that looks like this:
 +
 
 +
<pre>
 +
RDEPEND="foo >=dev-python/mydep-2 bar"
 +
</pre>
 +
 
 +
...to this new format:
 +
 
 +
<pre>
 +
RDEPEND="foo $(python_abi_depend ">=dev-python/mydep-2" ) bar"
 +
</pre>
 +
 
 +
The <tt>python_abi_depend</tt> function will expand to conditionally depend on the proper USE flags being set in <tt>dev-python/mydep-2</tt> based on what USE flags are currently set for ''this'' ebuild when it builds. This is how the cascading functionality works. Use it in <tt>DEPEND</tt>, <tt>RDEPEND</tt> and <tt>PDEPEND</tt> as appropriate. You can include multiple ebuilds as arguments to <tt>python_abi_depend</tt>.
 +
 
 +
Please note that due to the fact that <tt>python_abi_depend</tt> is implemented as a shell function, the shell can be a bit picky about parsing its arguments. Follow these guidelines:
 +
 
 +
* All arguments to <tt>python_abi_depend</tt> should be on the same line, or -
 +
* Additional lines should have a <tt>\</tt> line continuation character as the last line, and -
 +
* Any atoms with > or < characters should be enclosed in double-quotes to avoid being interpreted as output/input redirection
 +
 
 +
==== python_abi_depend() gotchas ====
 +
 
 +
Occasionally, you may encounter emerge errors like this when developing new Python ebuilds:
  
In order to use clang on distcc client, additional symlinks have to be created in ''<code>/usr/lib*/distcc/bin</code>'':
 
 
<console>
 
<console>
###i## ln -s /usr/bin/distcc /usr/lib/distcc/bin/clang
+
xdev nova # ##i##emerge -p nova
###i## ln -s /usr/bin/distcc /usr/lib/distcc/bin/clang++
+
 
 +
These are the packages that would be merged, in order:
 +
 
 +
Calculating dependencies... done!
 +
 
 +
emerge: there are no ebuilds built with USE flags to satisfy "dev-python/lxml[python_abis_2.6?,python_abis_2.7?,python_abis_2.6-jython?,python_abis_2.7-pypy-1.7?,python_abis_2.7-pypy-1.8?]".
 +
!!! One of the following packages is required to complete your request:
 +
- dev-python/lxml-2.3.3::gentoo (Missing IUSE: python_abis_2.6-jython python_abis_2.7-pypy-1.7 python_abis_2.7-pypy-1.8)
 +
(dependency required by "sys-cluster/nova-9999" [ebuild])
 +
(dependency required by "nova" [argument])
 
</console>
 
</console>
  
{{GLW|src=http://wiki.gentoo.org/wiki/Clang}}
+
Here's an explanation for why an error like this occurs. In this case, the <tt>lxml</tt> package has <tt>*-jython *-pypy-*</tt> in <tt>PYTHON_RESTRICTED_ABIS</tt>. Since <tt>nova</tt> depends on <tt>lxml</tt>, it should also be restricted so that it doesn't support Jython or PyPy. This makes sense, because if a dependency of <tt>nova</tt> can't run on a particular Python implementation, then neither can <tt>nova</tt> itself.
 +
 
 +
This emerge error can be fixed by adding <tt>*-jython *-pypy-*</tt> to <tt>nova</tt>'s <tt>PYTHON_RESTRICTED_ABIS</tt>. The lesson here is that any Python-based application can only support the subset of all platforms supported by all its dependencies, and <tt>PYTHON_RESTRICTED_ABIS</tt> must reflect this.
 +
 
 +
On an unrelated note, <tt>nova</tt> is configured with <tt>PYTHON_MULTIPLE_ABIS=1</tt> because it installs a ton of Python modules into <tt>site-packages</tt>.
 +
 
 +
==== common python_abi_depend() dependencies ====
 +
 
 +
This section is dedicated to providing examples on how to handle common dependencies that are found in Python-related packages:
 +
 
 +
===== argparse =====
 +
 
 +
If the Python package's <tt>setup.py</tt> contains something like this:
 +
 
 +
<pre>
 +
if sys.version_info < (2, 7):
 +
    requirements.append('argparse')
 +
</pre>
 +
 
 +
...then use the following dependency syntax:
 +
 
 +
<pre>
 +
(R)DEPEND="$(python_abi_depend virtual/python-argparse)"
 +
</pre>
 +
 
 +
==== python_abi_depend() with -i and -e ====
 +
 
 +
<tt>python_abi_depend()</tt> has some advanced capabilities that you may need to take advantage of related to Python dependencies. Sometimes, your ebuild will only need a python module if certain versions of Python are installed. This functionality is typically needed in the following cases:
 +
 
 +
;module rolled into standard library: A module is needed, but in newer versions of python it is part of the standard library (so there is no need to depend on it for these newer python versions)
 +
;module required for fixes: A module is needed, but only for newer versions of python, such as 3.x, where it implements compatibility features.
 +
;optional module, not 100% compatible: A module can be used, but the dependencies are not supported with certain versions of Python.
 +
 
 +
Let's look at the first case, a module your ebuild needs that was rolled into the standard library starting with Python version 3.1. In this case, we only need to depend on this module for Python ABIs 3.0 or less. We would use the <tt>-e</tt> option to exclude the versions we are not interested in:
 +
 
 +
<pre>
 +
RDEPEND="$(python_abi_depend -e "3.[1-9]" dev-python/mydep)"
 +
</pre>
 +
 
 +
In the second case, we depend on a module only for Python ABIs 3 and up, since it implements fixes for python3. In this case, we will use the <tt>-i</tt> option to include only those versions for which we want to depend:
 +
 
 +
<pre>
 +
RDEPEND="$(python_abi_depend -i "3.*" dev-python/mydep)"
 +
</pre>
 +
 
 +
Third case: something doesn't make sense in the context with pypy:
 +
 
 +
<pre>
 +
RDEPEND=">=sys-fs/udev-151
 +
pygobject? ( $(python_abi_depend -e "*-pypy-*" dev-python/pygobject:2) )
 +
pyqt4? ( $(python_abi_depend -e "*-pypy-*" dev-python/PyQt4) )
 +
pyside? ( dev-python/pyside )
 +
wxwidgets? ( $(python_abi_depend -e "3.* *-pypy-*" dev-python/wxpython) )"
 +
</pre>
 +
 
 +
=== Links ===
 +
 
 +
* http://people.apache.org/~Arfrever/Gentoo_Python_Guide_for_Users
 +
* http://people.apache.org/~Arfrever/Gentoo_Python_Supplementary_Guide_for_Developers
 +
* http://people.apache.org/~Arfrever/EAPI_4-python_Specification
  
[[Category:HOWTO]]
+
[[Category:Featured]]
 +
[[Category:Portage]]
 +
[[Category:OpenStack]]

Latest revision as of 23:04, 30 January 2014

Summary

Funtoo Linux incorporates Progress Overlay into the main Portage tree, which currently contains 400+ updated Python-related packages that implement next-generation Python support for ebuilds.

Building Python Modules with Portage

Conceptual Overview

Portage supports the ability to build python modules and extensions for all installed versions of python. This functionality has been implemented by python.eclass.

To use python.eclass, you first want to investigate how python module building is integrated into any included Makefile or build script. Since we want python.eclass to build these modules for us, typically the first step is to tweak the Makefile or configure script to explicitly *not* build python modules. We will be handling that process using python.eclass.

Another option, for those Python packages with a setup.py in the root source directory, is to use the distutils eclass. In this case, you generally do not need custom src_compile or src_install stages, as the eclass takes care of everything for you. However, you will still need to set the proper environment variables described below, as they are needed by python.eclass which is used by distutils.eclass. You will also want to set dependencies in DEPEND and RDEPEND as described in this article.

Progress Improvements

This section describes the differences in approach between the old (classic) way of handling Python-related ebuilds, and the new way, which uses the updated eclass. Also note that the distutils eclass benefits from these updates as well, as it uses python.eclass behind-the-scenes.

The old way: python-updater

With the older python.eclass in Gentoo (and that used to be in Funtoo Linux,) python modules are built for all installed versions of Python. However, no usable accounting information was stored by Portage so that it could "know" what versions of python the module was built for. This created a problem when new versions of Python are installed, as the new module would not be available for the new python, and Portage would not be aware of what needed to be done to fix this, if anything. This resulted in the creation of a tool called python-updater to rebuild all necessary python-related ebuilds so that they reflected the currently-installed versions of Python.

Note that python-updater-0.10 and later are aware of the new Python eclass, and will not unnecessarily reinstall any packages which set PYTHON_MULTIPLE_ABIS and EAPI="4-python", as manual rebuilding is not necessary.

TODO: add python-updater example

The new way: new python eclass

The new python.eclass from Arfrever has a new API that can be used to write Python ebuilds. When this API is used, Portage can track and resolve missing builds of Python modules internally, so python-updater is not needed to be used for these ebuilds. However, until all ebuilds are converted to the new API, python-updater will need to be used to update these old-style ebuilds. Fortunately for us, over 400 Python-related ebuilds have been converted to take advantage of the new eclass, with more added weekly.

Setting up progress overlay

Note

Progess overlay is now merged in Funtoo current and stable - No manual steps are needed to benefit from it!

Notice, that progress overlay including an updated packages that also available in main portage tree!

# python-updater
this still required to rebuild old-fashion python-related ebuilds


Updating the packages from progress overlay

Package updates are done the same way than before:

emerge -avuDN @world

Several python-related packages should be updated to progress versions, with the new PYTHON_ABIS flags.

Developer Docs

Ebuild Converstion to new Python eclass

Conversion from previous EAPIs and eclasses to the new progress python eclass is fairly straightforward. This section tries to explain the steps.

4-python EAPI

First, set EAPI to 4-python:

EAPI=4-python

Next, you will need to understand the type of Python application or module you are trying to install. The primary determining factor in how to proceed is to see whether the Python application installs anything into Python's site-packages directory. Essentially all Python modules do this, and some applications do this as well. If a package installs stuff into site-packages, then you should have its ebuild take advantage of the Python eclass "multiple ABI" support.

In this context, "multiple ABI" refers to potentially multiple versions of Python that may be installed on your system. Each installed major.minor version of Python, plus Python derivatives like Jython and pypy, will have their own site-packages directory and the Python eclass can take care of installing a version of the Python modules for each version of Python on your system. If the application has binaries, a Python wrapper will also be installed that will hand off control to the proper version of Python automatically.

Non-Multiple-ABI Support

If you are not supporting multiple ABIs, which is the least common option, you will generally just set PYTHON_DEPEND to refer to the version(s) of Python you require. Documentation for PYTHON_DEPEND syntax (which has been changed in the new python eclass) is available here.

Multiple ABI Support

So, if you are using multiple ABI support, which will be the case for all Python modules, proceed as follows. Set PYTHON_MULTIPLE_ABIS prior to any inherit lines:

PYTHON_MULTIPLE_ABIS=1

Note the new variable names. PYTHON_MULTIPLE_ABIS turns on the functionality to build multiple versions of python modules for each version of python defined in the user's PYTHON_ABIS setting. Note that PYTHON_ABIS defaults to "2.7 3.2" in Funtoo Linux profiles and can be overridden in /etc/make.conf if you would like the python eclass to target other Python ABIs (like jython, for example, or 2.6) that you might be interested in.

When PYTHON_MULTIPLE_ABIS is set and EAPI is set to 4-python, then new USE flags will be automatically added to the IUSE with the python_abis_ prefix. If you want to see what ABIs are supported, see the _PYTHON_GLOBALLY_SUPPORTED_ABIS variable defined in python.eclass.

Also note that for multiple ABI support, PYTHON_DEPEND is set to a reasonable default value so you should almost always never need to set it yourself.

PYTHON_RESTRICTED_ABIS

PYTHON_RESTRICTED_ABIS variable set in the ebuild specifies a space-separated list of patterns of Python ABIs not supported by the current package. The settings in PYTHON_RESTRICTED_ABIS will remove certain ABIs from the auto-IUSE expansion that happens when EAPI is set to 4-python and PYTHON_MULTIPLE_ABIS is set to 1 (This behavior is described above.) It will also limit the behavior of the python_abi_depend() function, described below. And of course, it will tell the python eclass to not build any python modules for these python versions, even if they happen to be installed on the system.

Use it like this, before the eclasses are inherited:

PYTHON_RESTRICTED_ABIS="3.*"

Note that PYTHON_RESTRICTED_ABIS is a glob-style pattern. If you want to have it build with everything except 3.2 and above and jython, use this pattern:

PYTHON_RESTRICTED_ABIS="3.[2-9] *-jython"

Note that the wildcard you should use to disable Jython is *-jython, while the wildcard you should use to disable PyPy is *-pypy-*.

python_abi_depend()

One of the cool things about the new Python eclass is the "cascading" dependency functionality which is implemented via USE variables. Basically, if your new-style python module ebuild was built for python-2.7, and it has dependencies, then it can depend on the module it needs being built for the installed version of python-2.7 specifically. This happens by having your ebuild depend upon the python_abis_2.7 USE flag being set in the installed ebuild it depends upon.

For this functionality to work, the ebuild you depend on must be a new-style (EAPI 4-python) ebuild too, and must be enabled to support multiple python ABIS via the PYTHON_MULTIPLE_ABIS=1 setting. When these requirements are met, you will want to covert an old-style dependency that looks like this:

RDEPEND="foo >=dev-python/mydep-2 bar"

...to this new format:

RDEPEND="foo $(python_abi_depend ">=dev-python/mydep-2" ) bar"

The python_abi_depend function will expand to conditionally depend on the proper USE flags being set in dev-python/mydep-2 based on what USE flags are currently set for this ebuild when it builds. This is how the cascading functionality works. Use it in DEPEND, RDEPEND and PDEPEND as appropriate. You can include multiple ebuilds as arguments to python_abi_depend.

Please note that due to the fact that python_abi_depend is implemented as a shell function, the shell can be a bit picky about parsing its arguments. Follow these guidelines:

  • All arguments to python_abi_depend should be on the same line, or -
  • Additional lines should have a \ line continuation character as the last line, and -
  • Any atoms with > or < characters should be enclosed in double-quotes to avoid being interpreted as output/input redirection

python_abi_depend() gotchas

Occasionally, you may encounter emerge errors like this when developing new Python ebuilds:

xdev nova # emerge -p nova

These are the packages that would be merged, in order:

Calculating dependencies... done!

emerge: there are no ebuilds built with USE flags to satisfy "dev-python/lxml[python_abis_2.6?,python_abis_2.7?,python_abis_2.6-jython?,python_abis_2.7-pypy-1.7?,python_abis_2.7-pypy-1.8?]".
!!! One of the following packages is required to complete your request:
- dev-python/lxml-2.3.3::gentoo (Missing IUSE: python_abis_2.6-jython python_abis_2.7-pypy-1.7 python_abis_2.7-pypy-1.8)
(dependency required by "sys-cluster/nova-9999" [ebuild])
(dependency required by "nova" [argument])

Here's an explanation for why an error like this occurs. In this case, the lxml package has *-jython *-pypy-* in PYTHON_RESTRICTED_ABIS. Since nova depends on lxml, it should also be restricted so that it doesn't support Jython or PyPy. This makes sense, because if a dependency of nova can't run on a particular Python implementation, then neither can nova itself.

This emerge error can be fixed by adding *-jython *-pypy-* to nova's PYTHON_RESTRICTED_ABIS. The lesson here is that any Python-based application can only support the subset of all platforms supported by all its dependencies, and PYTHON_RESTRICTED_ABIS must reflect this.

On an unrelated note, nova is configured with PYTHON_MULTIPLE_ABIS=1 because it installs a ton of Python modules into site-packages.

common python_abi_depend() dependencies

This section is dedicated to providing examples on how to handle common dependencies that are found in Python-related packages:

argparse

If the Python package's setup.py contains something like this:

if sys.version_info < (2, 7):
    requirements.append('argparse')

...then use the following dependency syntax:

(R)DEPEND="$(python_abi_depend virtual/python-argparse)"

python_abi_depend() with -i and -e

python_abi_depend() has some advanced capabilities that you may need to take advantage of related to Python dependencies. Sometimes, your ebuild will only need a python module if certain versions of Python are installed. This functionality is typically needed in the following cases:

module rolled into standard library
A module is needed, but in newer versions of python it is part of the standard library (so there is no need to depend on it for these newer python versions)
module required for fixes
A module is needed, but only for newer versions of python, such as 3.x, where it implements compatibility features.
optional module, not 100% compatible
A module can be used, but the dependencies are not supported with certain versions of Python.

Let's look at the first case, a module your ebuild needs that was rolled into the standard library starting with Python version 3.1. In this case, we only need to depend on this module for Python ABIs 3.0 or less. We would use the -e option to exclude the versions we are not interested in:

RDEPEND="$(python_abi_depend -e "3.[1-9]" dev-python/mydep)"

In the second case, we depend on a module only for Python ABIs 3 and up, since it implements fixes for python3. In this case, we will use the -i option to include only those versions for which we want to depend:

RDEPEND="$(python_abi_depend -i "3.*" dev-python/mydep)"

Third case: something doesn't make sense in the context with pypy:

RDEPEND=">=sys-fs/udev-151
	pygobject? ( $(python_abi_depend -e "*-pypy-*" dev-python/pygobject:2) )
	pyqt4? ( $(python_abi_depend -e "*-pypy-*" dev-python/PyQt4) )
	pyside? ( dev-python/pyside )
	wxwidgets? ( $(python_abi_depend -e "3.* *-pypy-*" dev-python/wxpython) )"

Links