Difference between pages "Creating Python-related Ebuilds" and "Gentoo Ecosystem"

(Difference between pages)
(The old way: python-updater)
 
m
 
Line 1: Line 1:
== Summary ==
+
This page is intended to describe the Gentoo Ecosystem -- everything Gentoo-related, including Gentoo Linux.
  
Funtoo Linux incorporates Progress Overlay into the main Portage tree, which currently contains 400+ updated Python-related packages that implement next-generation Python support for ebuilds.
+
== Gentoo ==
  
== Building Python Modules with Portage ==
+
See [[Gentoo Linux]].
  
== Conceptual Overview ==
+
== Funtoo ==
  
Portage supports the ability to build python modules and extensions for all installed versions of python. This functionality has been implemented by <tt>python.eclass</tt>.
+
See [[History]].
  
To use <tt>python.eclass</tt>, you first want to investigate how python module building is integrated into any included <tt>Makefile</tt> or build script. Since we want <tt>python.eclass</tt> to build these modules for us, typically the first step is to tweak the <tt>Makefile</tt> or <tt>configure</tt> script to explicitly *not* build python modules. We will be handling that process using <tt>python.eclass</tt>.
+
== Calculate ==
  
Another option, for those Python packages with a <tt>setup.py</tt> in the root source directory, is to use the <tt>distutils</tt> eclass. In this case, you generally do not need custom <tt>src_compile</tt> or <tt>src_install</tt> stages, as the eclass takes care of everything for you. However, you will still need to set the proper environment variables described below, as they are needed by <tt>python.eclass</tt> which is used by <tt>distutils.eclass</tt>. You will also want to set dependencies in <tt>DEPEND</tt> and <tt>RDEPEND</tt> as described in this article.
+
http://distrowatch.com/table.php?distribution=calculate
  
== Progress Improvements ==
+
== Pentoo ==
  
This section describes the differences in approach between the old (classic) way of handling Python-related ebuilds, and the new way, which uses the updated eclass. Also note that the <tt>distutils</tt> eclass benefits from these updates as well, as it uses <tt>python.eclass</tt> behind-the-scenes.
+
http://distrowatch.com/table.php?distribution=pentoo
  
=== The old way: python-updater ===
+
== UTUTO ==
  
With the older <tt>python.eclass</tt> in Gentoo (and that used to be in Funtoo Linux,) python modules are built for all installed versions of Python. However, no usable accounting information was stored by Portage so that it could "know" what versions of python the module was built for. This created a problem when new versions of Python are installed, as the new module would not be available for the new python, and Portage would not be aware of what needed to be done to fix this, if anything. This resulted in the creation of a tool called <tt>python-updater</tt> to rebuild all necessary python-related ebuilds so that they reflected the currently-installed versions of Python.
+
http://distrowatch.com/table.php?distribution=ututo
  
Note that python-updater-0.10 and later are aware of the new Python eclass, and will not unnecessarily reinstall any packages which set <tt>PYTHON_MULTIPLE_ABIS</tt> and <tt>EAPI="4-python"</tt>, as manual rebuilding is not necessary.
+
== Toorox ==
  
'''TODO: add python-updater example'''
+
http://distrowatch.com/table.php?distribution=toorox
  
=== The new way: new python eclass ===
+
== Litrix ==  
  
The new <tt>python.eclass</tt> from Arfrever has a new API that can be used to write Python ebuilds. When this API is used, Portage can track and resolve missing builds of Python modules internally, so <tt>python-updater</tt> is not needed to be used for these ebuilds. However, until all ebuilds are converted to the new API, <tt>python-updater</tt> will need to be used to update these old-style ebuilds. Fortunately for us, over 400 Python-related ebuilds have been converted to take advantage of the new eclass, with more added weekly.
+
http://distrowatch.com/table.php?distribution=litrix
  
== Setting up progress overlay ==
+
== Sabayon ==
  
{{fancynote|Progess overlay is now merged in Funtoo current and stable - No manual steps are needed to benefit from it!}}
+
http://distrowatch.com/table.php?distribution=sabayon
  
Notice, that progress overlay including an updated packages that also available in main portage tree!
+
== Liberté ==
<pre># python-updater</pre> this still required to rebuild old-fashion python-related ebuilds
+
  
 +
http://distrowatch.com/table.php?distribution=liberte
  
== Updating the packages from progress overlay ==
+
== Planet Larry ==
Package updates are done the same way than before:
+
<pre>emerge -avuDN @world</pre>
+
  
Several python-related packages should be updated to progress versions, with the new PYTHON_ABIS flags.
+
== sysrescuecd ==
  
== Developer Docs ==
+
See [http://www.sysresccd.org].
  
=== Ebuild Converstion to new Python eclass ===
+
== Google Chrome OS ==
  
Conversion from previous EAPIs and eclasses to the new progress python eclass is fairly straightforward. This section tries to explain the steps.
+
http://www.internetnews.com/skerner/2010/02/google-goes-with-gentoo-portag.html
  
==== 4-python EAPI ====
 
  
First, set EAPI to 4-python:
+
Note: The Google Chrome OS Wikipedia Page needs updating to reflect that they are now using Gentoo Linux: [[wikipedia:Google_Chrome_OS]]
 
+
<pre>
+
EAPI=4-python
+
</pre>
+
 
+
Next, you will need to understand the type of Python application or module you are trying to install. The primary determining factor in how to proceed is to see whether the Python application installs anything into Python's <tt>site-packages</tt> directory.  Essentially all Python modules do this, and some applications do this as well. If a package installs stuff into <tt>site-packages</tt>, then you should have its ebuild take advantage of the Python eclass "multiple ABI" support.
+
 
+
In this context, "multiple ABI" refers to potentially multiple versions of Python that may be installed on your system. Each installed major.minor version of Python, plus Python derivatives like Jython and pypy, will have their own <tt>site-packages</tt> directory and the Python eclass can take care of installing a version of the Python modules for each version of Python on your system. If the application has binaries, a Python wrapper will also be installed that will hand off control to the proper version of Python automatically.
+
 
+
==== Non-Multiple-ABI Support ====
+
 
+
If you are not supporting multiple ABIs, which is the least common option, you will generally just set <tt>PYTHON_DEPEND</tt> to refer to the version(s) of Python you require. Documentation for <tt>PYTHON_DEPEND</tt> syntax (which has been changed in the new python eclass) is available [http://people.apache.org/~Arfrever/Gentoo_Python_Supplementary_Guide_for_Developers here].
+
 
+
==== Multiple ABI Support ====
+
 
+
So, if you are using multiple ABI support, which will be the case for all Python modules, proceed as follows. Set <tt>PYTHON_MULTIPLE_ABIS</tt> prior to any inherit lines:
+
 
+
<pre>
+
PYTHON_MULTIPLE_ABIS=1
+
</pre>
+
 
+
Note the new variable names. <tt>PYTHON_MULTIPLE_ABIS</tt> turns on the functionality to build multiple versions of python modules for each version of python defined in the user's <tt>PYTHON_ABIS</tt> setting. Note that <tt>PYTHON_ABIS</tt> defaults to "<tt>2.7 3.2</tt>" in Funtoo Linux profiles and can be overridden in <tt>/etc/make.conf</tt> if you would like the python eclass to target other Python ABIs (like jython, for example, or 2.6) that you might be interested in.
+
 
+
When <tt>PYTHON_MULTIPLE_ABIS</tt> is set and <tt>EAPI</tt> is set to <tt>4-python</tt>, then new USE flags will be automatically added to the <tt>IUSE</tt> with the <tt>python_abis_</tt> prefix.
+
If you want to see what ABIs are supported, see the <tt>_PYTHON_GLOBALLY_SUPPORTED_ABIS</tt> variable defined in <tt>python.eclass</tt>.
+
 
+
Also note that for multiple ABI support, <tt>PYTHON_DEPEND</tt> is set to a reasonable default value so you should almost always never need to set it yourself.
+
 
+
==== PYTHON_RESTRICTED_ABIS ====
+
 
+
<tt>PYTHON_RESTRICTED_ABIS</tt> variable set in the ebuild specifies a space-separated list of patterns of Python ABIs ''not'' supported by the current package. The settings in <tt>PYTHON_RESTRICTED_ABIS</tt> will remove certain ABIs from the auto-<tt>IUSE</tt> expansion that happens when <tt>EAPI</tt> is set to <tt>4-python</tt> and <tt>PYTHON_MULTIPLE_ABIS</tt> is set to <tt>1</tt> (This behavior is described above.) It will also limit the behavior of the <tt>python_abi_depend()</tt> function, described below. And of course, it will tell the python eclass to not build any python modules for these python versions, even if they happen to be installed on the system.
+
 
+
Use it like this, before the eclasses are inherited:
+
 
+
<pre>
+
PYTHON_RESTRICTED_ABIS="3.*"
+
</pre>
+
 
+
Note that <tt>PYTHON_RESTRICTED_ABIS</tt> is a glob-style pattern. If you want to have it build with everything ''except'' 3.2 and above and jython, use this pattern:
+
 
+
<pre>
+
PYTHON_RESTRICTED_ABIS="3.[2-9] *-jython"
+
</pre>
+
 
+
Note that the wildcard you should use to disable Jython is <tt>*-jython</tt>, while the wildcard you should use to disable PyPy is <tt>*-pypy-*</tt>.
+
 
+
==== python_abi_depend() ====
+
 
+
One of the cool things about the new Python eclass is the "cascading" dependency functionality which is implemented via USE variables. Basically, if your new-style python module ebuild was built for python-2.7, and it has dependencies, then it can depend on the module it needs being built ''for'' the installed version of python-2.7 specifically. This happens by having your ebuild depend upon the <tt>python_abis_2.7</tt> <tt>USE</tt> flag being set in the installed ebuild it depends upon.
+
 
+
For this functionality to work, the ebuild you depend on must be a new-style (<tt>EAPI 4-python</tt>) ebuild too, and must be enabled to support multiple python ABIS via the <tt>PYTHON_MULTIPLE_ABIS=1</tt> setting. When these requirements are met, you will want to covert an old-style dependency that looks like this:
+
 
+
<pre>
+
RDEPEND="foo >=dev-python/mydep-2 bar"
+
</pre>
+
 
+
...to this new format:
+
 
+
<pre>
+
RDEPEND="foo $(python_abi_depend ">=dev-python/mydep-2" ) bar"
+
</pre>
+
 
+
The <tt>python_abi_depend</tt> function will expand to conditionally depend on the proper USE flags being set in <tt>dev-python/mydep-2</tt> based on what USE flags are currently set for ''this'' ebuild when it builds. This is how the cascading functionality works. Use it in <tt>DEPEND</tt>, <tt>RDEPEND</tt> and <tt>PDEPEND</tt> as appropriate. You can include multiple ebuilds as arguments to <tt>python_abi_depend</tt>.
+
 
+
Please note that due to the fact that <tt>python_abi_depend</tt> is implemented as a shell function, the shell can be a bit picky about parsing its arguments. Follow these guidelines:
+
 
+
* All arguments to <tt>python_abi_depend</tt> should be on the same line, or -
+
* Additional lines should have a <tt>\</tt> line continuation character as the last line, and -
+
* Any atoms with > or < characters should be enclosed in double-quotes to avoid being interpreted as output/input redirection
+
 
+
==== python_abi_depend() gotchas ====
+
 
+
Occasionally, you may encounter emerge errors like this when developing new Python ebuilds:
+
 
+
<console>
+
xdev nova # ##i##emerge -p nova
+
 
+
These are the packages that would be merged, in order:
+
 
+
Calculating dependencies... done!
+
 
+
emerge: there are no ebuilds built with USE flags to satisfy "dev-python/lxml[python_abis_2.6?,python_abis_2.7?,python_abis_2.6-jython?,python_abis_2.7-pypy-1.7?,python_abis_2.7-pypy-1.8?]".
+
!!! One of the following packages is required to complete your request:
+
- dev-python/lxml-2.3.3::gentoo (Missing IUSE: python_abis_2.6-jython python_abis_2.7-pypy-1.7 python_abis_2.7-pypy-1.8)
+
(dependency required by "sys-cluster/nova-9999" [ebuild])
+
(dependency required by "nova" [argument])
+
</console>
+
 
+
Here's an explanation for why an error like this occurs. In this case, the <tt>lxml</tt> package has <tt>*-jython *-pypy-*</tt> in <tt>PYTHON_RESTRICTED_ABIS</tt>. Since <tt>nova</tt> depends on <tt>lxml</tt>, it should also be restricted so that it doesn't support Jython or PyPy. This makes sense, because if a dependency of <tt>nova</tt> can't run on a particular Python implementation, then neither can <tt>nova</tt> itself.
+
 
+
This emerge error can be fixed by adding <tt>*-jython *-pypy-*</tt> to <tt>nova</tt>'s <tt>PYTHON_RESTRICTED_ABIS</tt>. The lesson here is that any Python-based application can only support the subset of all platforms supported by all its dependencies, and <tt>PYTHON_RESTRICTED_ABIS</tt> must reflect this.
+
 
+
On an unrelated note, <tt>nova</tt> is configured with <tt>PYTHON_MULTIPLE_ABIS=1</tt> because it installs a ton of Python modules into <tt>site-packages</tt>.
+
 
+
==== common python_abi_depend() dependencies ====
+
 
+
This section is dedicated to providing examples on how to handle common dependencies that are found in Python-related packages:
+
 
+
===== argparse =====
+
 
+
If the Python package's <tt>setup.py</tt> contains something like this:
+
 
+
<pre>
+
if sys.version_info < (2, 7):
+
    requirements.append('argparse')
+
</pre>
+
 
+
...then use the following dependency syntax:
+
 
+
<pre>
+
(R)DEPEND="$(python_abi_depend virtual/python-argparse)"
+
</pre>
+
 
+
==== python_abi_depend() with -i and -e ====
+
 
+
<tt>python_abi_depend()</tt> has some advanced capabilities that you may need to take advantage of related to Python dependencies. Sometimes, your ebuild will only need a python module if certain versions of Python are installed. This functionality is typically needed in the following cases:
+
 
+
;module rolled into standard library: A module is needed, but in newer versions of python it is part of the standard library (so there is no need to depend on it for these newer python versions)
+
;module required for fixes: A module is needed, but only for newer versions of python, such as 3.x, where it implements compatibility features.
+
;optional module, not 100% compatible: A module can be used, but the dependencies are not supported with certain versions of Python.
+
 
+
Let's look at the first case, a module your ebuild needs that was rolled into the standard library starting with Python version 3.1. In this case, we only need to depend on this module for Python ABIs 3.0 or less. We would use the <tt>-e</tt> option to exclude the versions we are not interested in:
+
 
+
<pre>
+
RDEPEND="$(python_abi_depend -e "3.[1-9]" dev-python/mydep)"
+
</pre>
+
 
+
In the second case, we depend on a module only for Python ABIs 3 and up, since it implements fixes for python3. In this case, we will use the <tt>-i</tt> option to include only those versions for which we want to depend:
+
 
+
<pre>
+
RDEPEND="$(python_abi_depend -i "3.*" dev-python/mydep)"
+
</pre>
+
 
+
Third case: something doesn't make sense in the context with pypy:
+
 
+
<pre>
+
RDEPEND=">=sys-fs/udev-151
+
pygobject? ( $(python_abi_depend -e "*-pypy-*" dev-python/pygobject:2) )
+
pyqt4? ( $(python_abi_depend -e "*-pypy-*" dev-python/PyQt4) )
+
pyside? ( dev-python/pyside )
+
wxwidgets? ( $(python_abi_depend -e "3.* *-pypy-*" dev-python/wxpython) )"
+
</pre>
+
 
+
=== Links ===
+
 
+
* http://people.apache.org/~Arfrever/Gentoo_Python_Guide_for_Users
+
* http://people.apache.org/~Arfrever/Gentoo_Python_Supplementary_Guide_for_Developers
+
* http://people.apache.org/~Arfrever/EAPI_4-python_Specification
+
  
 
[[Category:Featured]]
 
[[Category:Featured]]
[[Category:Portage]]
 
[[Category:Python]]
 
[[Category:OpenStack]]
 

Latest revision as of 18:58, 15 September 2013

This page is intended to describe the Gentoo Ecosystem -- everything Gentoo-related, including Gentoo Linux.

Gentoo

See Gentoo Linux.

Funtoo

See History.

Calculate

http://distrowatch.com/table.php?distribution=calculate

Pentoo

http://distrowatch.com/table.php?distribution=pentoo

UTUTO

http://distrowatch.com/table.php?distribution=ututo

Toorox

http://distrowatch.com/table.php?distribution=toorox

Litrix

http://distrowatch.com/table.php?distribution=litrix

Sabayon

http://distrowatch.com/table.php?distribution=sabayon

Liberté

http://distrowatch.com/table.php?distribution=liberte

Planet Larry

sysrescuecd

See [1].

Google Chrome OS

http://www.internetnews.com/skerner/2010/02/google-goes-with-gentoo-portag.html


Note: The Google Chrome OS Wikipedia Page needs updating to reflect that they are now using Gentoo Linux: wikipedia:Google_Chrome_OS