Difference between pages "Creating Python-related Ebuilds" and "Main Page"

(Difference between pages)
(Update wanted for 5-progress EAPI)
 
m (Added linkedin and jira buttons)
 
Line 1: Line 1:
{{PageNeedsUpdates}}
+
{{DISPLAYTITLE:Funtoo Linux}}
 +
__NOTITLE__
 +
__NOEDITSECTION__
 +
<div style="display: none;"><h1>Welcome to Funtoo Linux</h1></div>
  
== Summary ==
+
<div class="row">
  
Funtoo Linux incorporates Progress Overlay into the main Portage tree, which currently contains 400+ updated Python-related packages that implement next-generation Python support for ebuilds.
+
<div class="col-xs-12 col-md-3">[[File:Fwiki.png|link=Funtoo_Linux_Installation]]
  
== Building Python Modules with Portage ==
+
<!-- button block -->
 +
<div class="row" style="margin-top:10px">
 +
<div class="col-xs-3">[http://webchat.freenode.net/?nick=Funtoo.&channels=%23funtoo&uio=MTA9dHJ1ZSYxMj10cnVlf5 <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-comment"></i>]</div>
 +
<div class="col-xs-3">[http://git.funtoo.org <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-code-fork"></i>]</div>
 +
<div class="col-xs-3">[https://bugs.funtoo.org <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-bug"></i>]</div>
 +
<div class="col-xs-3">[[Funtoo_RSS_and_Atom_Feeds | <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-rss-square"></i>]]</div>
 +
</div>
 +
<div class="row" style="margin-top:10px">
 +
<div class="col-xs-3">[https://www.linkedin.com/company/funtoo-technologies <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-linkedin"></i>]</div>
 +
<div class="col-xs-3">[https://www.facebook.com/pages/Funtoo-Linux/107461849335837 <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-facebook"></i>]</div>
 +
<div class="col-xs-3">[https://twitter.com/funtoo <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-twitter"></i>]</div>
 +
<div class="col-xs-3">[https://plus.google.com/+funtoo/posts <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-google-plus"></i>]</div>
 +
</div>
 +
<div class="row" style="margin-top:10px">
 +
<div class="col-xs-3">[https://www.reddit.com/r/funtoo <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-reddit">r</i>]</div>
 +
<div class="col-xs-3">[https://github.com/funtoo <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-github"></i>]</div>
 +
<div class="col-xs-3">[https://www.youtube.com/channel/UCKmOY6p3c9hxv3vJMAF8vVw <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-youtube"></i>]</div>
 +
<div class="col-xs-3">[http://distrowatch.com/table.php?distribution=funtoo <i style="border: 1px solid #1111a1;border-radius: 3px;" class="btn btn-block btn-custom fa fa-linux"></i>]</div>
 +
</div>
  
== Conceptual Overview ==
+
<!--end of button block-->
  
Portage supports the ability to build python modules and extensions for all installed versions of python. This functionality has been implemented by <tt>python.eclass</tt>.
+
== Getting Started ==
 +
'''[[Funtoo Linux Installation|Install Funtoo]]'''
  
To use <tt>python.eclass</tt>, you first want to investigate how python module building is integrated into any included <tt>Makefile</tt> or build script. Since we want <tt>python.eclass</tt> to build these modules for us, typically the first step is to tweak the <tt>Makefile</tt> or <tt>configure</tt> script to explicitly *not* build python modules. We will be handling that process using <tt>python.eclass</tt>.
+
'''{{CreateAccount}}''', which allows you to get involved with the community.  Log in to the [http://www.funtoo.org/ wiki], [http://forums.funtoo.org forums], and [https://bugs.funtoo.org bug tracker] under the same username, and password by unified logins.  See the [[Funtoo Authentication FAQ|Auth FAQ]] & [[Funtoo Linux FAQ|General FAQ]].
  
Another option, for those Python packages with a <tt>setup.py</tt> in the root source directory, is to use the <tt>distutils</tt> eclass. In this case, you generally do not need custom <tt>src_compile</tt> or <tt>src_install</tt> stages, as the eclass takes care of everything for you. However, you will still need to set the proper environment variables described below, as they are needed by <tt>python.eclass</tt> which is used by <tt>distutils.eclass</tt>. You will also want to set dependencies in <tt>DEPEND</tt> and <tt>RDEPEND</tt> as described in this article.
+
'''[[Reporting Bugs|Report bugs, and suggest improvements]]''' to our '''[http://bugs.funtoo.org bug tracker]'''. We take all bugs seriously, and all work performed is tracked on our public bug tracker, for purposes of transparency.
 +
See the [[Funtoo Linux FAQ]].
  
== Progress Improvements ==
+
</div>
  
This section describes the differences in approach between the old (classic) way of handling Python-related ebuilds, and the new way, which uses the updated eclass. Also note that the <tt>distutils</tt> eclass benefits from these updates as well, as it uses <tt>python.eclass</tt> behind-the-scenes.
+
<div class="col-xs-12 col-md-9">
 +
Funtoo Linux is a rolling-release Linux meta-distribution for x86 and ARM systems. It uses Portage as a package manager, and is run by Daniel Robbins, creator of Gentoo Linux. It is a continuation of Daniel's work on Gentoo Linux.  Funtoo Linux is committed to software freedom, independence, and a user-centric model of software development, where our users help define the future direction of the project.
  
=== The old way: python-updater ===
+
Our goal is to provide a best-of-breed Gentoo-based meta-distribution that is ideal for desktop and servers, and to progressively improve our core technology in thoughtful and innovative ways, while respecting Gentoo design principles, and the history of Unix and Linux.
  
With the older <tt>python.eclass</tt> in Gentoo (and that used to be in Funtoo Linux,) python modules are built for all installed versions of Python. However, no usable accounting information was stored by Portage so that it could "know" what versions of python the module was built for. This created a problem when new versions of Python are installed, as the new module would not be available for the new python, and Portage would not be aware of what needed to be done to fix this, if anything. This resulted in the creation of a tool called <tt>python-updater</tt> to rebuild all necessary python-related ebuilds so that they reflected the currently-installed versions of Python.
+
Funtoo Linux provides [[Subarches|optimized builds]] for your specific hardware to ensure the best possible performance, and offers the following features:
 +
* A completely systemd-free system, including GNOME 3.14 without systemd.
 +
* A new [[Funtoo 1.0 Profile|profile system]] to simplify system configuration.
 +
* Improved ease of install with pre-built Debian kernel provided in our install image.
 +
* A new system configuration tool, [News:Better_Experiences:_Ego_and_Vim ego].
 +
* An advanced build and quality assurance tool called [[Metro]], providing optimized install images and enhanced real-world testing.
 +
* The use of git as a default mechanism for getting updated ebuilds (build scripts) to you.
 +
* [[Funtoo Linux Networking|Modular networking scripts]] to simplify creating sophisticated server network configurations.
 +
* Enhanced Python support from the Progress overlay, and inclusion of other popular packages from the Gentoo ecosystem.
  
Note that python-updater-0.10 and later are aware of the new Python eclass, and will not unnecessarily reinstall any packages which set <tt>PYTHON_MULTIPLE_ABIS</tt> and <tt>EAPI="4-python"</tt>, as manual rebuilding is not necessary.
+
</div>
  
'''TODO: add python-updater example'''
+
<div class="col-xs-12 col-md-9">{{Announce|{{SupportBlurb}}}}</div>
  
=== The new way: new python eclass ===
+
<div class="col-xs-12 col-md-5">
 +
=== News ===
 +
{{NewsList|2}}
 +
</div>
  
The new <tt>python.eclass</tt> from Arfrever has a new API that can be used to write Python ebuilds. When this API is used, Portage can track and resolve missing builds of Python modules internally, so <tt>python-updater</tt> is not needed to be used for these ebuilds. However, until all ebuilds are converted to the new API, <tt>python-updater</tt> will need to be used to update these old-style ebuilds. Fortunately for us, over 400 Python-related ebuilds have been converted to take advantage of the new eclass, with more added weekly.
+
<div class="col-xs-12 col-md-4">
 +
== Official Projects ==
 +
*  '''[[Keychain]]''', an SSH/GPG agent front-end.
 +
* '''[[Metro]]''', automated Funtoo build engine.
 +
* '''[[Linux_Fundamentals,_Part_1|Learn Linux]]'''! [[Awk_by_Example,_Part_1|Awk]], [[Bash_by_Example,_Part_1|Bash]], [[Sed_by_Example,_Part_1|Sed]]  and more.
 +
* Web projects include [http://larrythecow.org larrythecow.org], the Gentoo blog aggregator, and [http://kernel-seeds.org kernel-seeds.org].
  
== Setting up progress overlay ==
+
=== [[Ebuilds]] ===
 +
'''Ebuild pages recently updated:''' {{#ask: [[Category:Ebuilds]]
 +
| order=descending
 +
| sort=Modification date
 +
| format=list
 +
| limit=10
 +
| searchlabel=
 +
}}
 +
</div>
  
{{fancynote|Progess overlay is now merged in Funtoo current and stable - No manual steps are needed to benefit from it!}}
+
</div>
  
Notice, that progress overlay including an updated packages that also available in main portage tree!
 
<pre># python-updater</pre> this still required to rebuild old-fashion python-related ebuilds
 
  
  
== Updating the packages from progress overlay ==
+
<div class="row">
Package updates are done the same way than before:
+
  <div class="col-md-6">{{#widget:YouTube16x9|playlist=PL2YVrx9jFJOewYI7f15FahwLOZlFCRqjZ}}
<pre>emerge -avuDN @world</pre>
+
{{FuntooFriendly|Brownrice Internet}}
 +
</div>
 +
</div>
  
Several python-related packages should be updated to progress versions, with the new PYTHON_ABIS flags.
+
{{#seo:
 
+
|title=Funtoo Linux
== Developer Docs ==
+
|keywords=funtoo,linux,gentoo,Daniel Robbins
 
+
|description=Funtoo Linux is a Gentoo-based OS that uses a git-based Portage tree. Run by Daniel Robbins, creator of Gentoo.
=== Ebuild Converstion to new Python eclass ===
+
}}
 
+
Conversion from previous EAPIs and eclasses to the new progress python eclass is fairly straightforward. This section tries to explain the steps.
+
 
+
==== 4-python EAPI ====
+
 
+
First, set EAPI to 4-python:
+
 
+
<pre>
+
EAPI=4-python
+
</pre>
+
 
+
Next, you will need to understand the type of Python application or module you are trying to install. The primary determining factor in how to proceed is to see whether the Python application installs anything into Python's <tt>site-packages</tt> directory.  Essentially all Python modules do this, and some applications do this as well. If a package installs stuff into <tt>site-packages</tt>, then you should have its ebuild take advantage of the Python eclass "multiple ABI" support.
+
 
+
In this context, "multiple ABI" refers to potentially multiple versions of Python that may be installed on your system. Each installed major.minor version of Python, plus Python derivatives like Jython and pypy, will have their own <tt>site-packages</tt> directory and the Python eclass can take care of installing a version of the Python modules for each version of Python on your system. If the application has binaries, a Python wrapper will also be installed that will hand off control to the proper version of Python automatically.
+
 
+
==== Non-Multiple-ABI Support ====
+
 
+
If you are not supporting multiple ABIs, which is the least common option, you will generally just set <tt>PYTHON_DEPEND</tt> to refer to the version(s) of Python you require. Documentation for <tt>PYTHON_DEPEND</tt> syntax (which has been changed in the new python eclass) is available [http://people.apache.org/~Arfrever/Gentoo_Python_Supplementary_Guide_for_Developers here].
+
 
+
==== Multiple ABI Support ====
+
 
+
So, if you are using multiple ABI support, which will be the case for all Python modules, proceed as follows. Set <tt>PYTHON_MULTIPLE_ABIS</tt> prior to any inherit lines:
+
 
+
<pre>
+
PYTHON_MULTIPLE_ABIS=1
+
</pre>
+
 
+
Note the new variable names. <tt>PYTHON_MULTIPLE_ABIS</tt> turns on the functionality to build multiple versions of python modules for each version of python defined in the user's <tt>PYTHON_ABIS</tt> setting. Note that <tt>PYTHON_ABIS</tt> defaults to "<tt>2.7 3.2</tt>" in Funtoo Linux profiles and can be overridden in <tt>/etc/make.conf</tt> if you would like the python eclass to target other Python ABIs (like jython, for example, or 2.6) that you might be interested in.
+
 
+
When <tt>PYTHON_MULTIPLE_ABIS</tt> is set and <tt>EAPI</tt> is set to <tt>4-python</tt>, then new USE flags will be automatically added to the <tt>IUSE</tt> with the <tt>python_abis_</tt> prefix.
+
If you want to see what ABIs are supported, see the <tt>_PYTHON_GLOBALLY_SUPPORTED_ABIS</tt> variable defined in <tt>python.eclass</tt>.
+
 
+
Also note that for multiple ABI support, <tt>PYTHON_DEPEND</tt> is set to a reasonable default value so you should almost always never need to set it yourself.
+
 
+
==== PYTHON_RESTRICTED_ABIS ====
+
 
+
<tt>PYTHON_RESTRICTED_ABIS</tt> variable set in the ebuild specifies a space-separated list of patterns of Python ABIs ''not'' supported by the current package. The settings in <tt>PYTHON_RESTRICTED_ABIS</tt> will remove certain ABIs from the auto-<tt>IUSE</tt> expansion that happens when <tt>EAPI</tt> is set to <tt>4-python</tt> and <tt>PYTHON_MULTIPLE_ABIS</tt> is set to <tt>1</tt> (This behavior is described above.) It will also limit the behavior of the <tt>python_abi_depend()</tt> function, described below. And of course, it will tell the python eclass to not build any python modules for these python versions, even if they happen to be installed on the system.
+
 
+
Use it like this, before the eclasses are inherited:
+
 
+
<pre>
+
PYTHON_RESTRICTED_ABIS="3.*"
+
</pre>
+
 
+
Note that <tt>PYTHON_RESTRICTED_ABIS</tt> is a glob-style pattern. If you want to have it build with everything ''except'' 3.2 and above and jython, use this pattern:
+
 
+
<pre>
+
PYTHON_RESTRICTED_ABIS="3.[2-9] *-jython"
+
</pre>
+
 
+
Note that the wildcard you should use to disable Jython is <tt>*-jython</tt>, while the wildcard you should use to disable PyPy is <tt>*-pypy-*</tt>.
+
 
+
==== python_abi_depend() ====
+
 
+
One of the cool things about the new Python eclass is the "cascading" dependency functionality which is implemented via USE variables. Basically, if your new-style python module ebuild was built for python-2.7, and it has dependencies, then it can depend on the module it needs being built ''for'' the installed version of python-2.7 specifically. This happens by having your ebuild depend upon the <tt>python_abis_2.7</tt> <tt>USE</tt> flag being set in the installed ebuild it depends upon.
+
 
+
For this functionality to work, the ebuild you depend on must be a new-style (<tt>EAPI 4-python</tt>) ebuild too, and must be enabled to support multiple python ABIS via the <tt>PYTHON_MULTIPLE_ABIS=1</tt> setting. When these requirements are met, you will want to covert an old-style dependency that looks like this:
+
 
+
<pre>
+
RDEPEND="foo >=dev-python/mydep-2 bar"
+
</pre>
+
 
+
...to this new format:
+
 
+
<pre>
+
RDEPEND="foo $(python_abi_depend ">=dev-python/mydep-2" ) bar"
+
</pre>
+
 
+
The <tt>python_abi_depend</tt> function will expand to conditionally depend on the proper USE flags being set in <tt>dev-python/mydep-2</tt> based on what USE flags are currently set for ''this'' ebuild when it builds. This is how the cascading functionality works. Use it in <tt>DEPEND</tt>, <tt>RDEPEND</tt> and <tt>PDEPEND</tt> as appropriate. You can include multiple ebuilds as arguments to <tt>python_abi_depend</tt>.
+
 
+
Please note that due to the fact that <tt>python_abi_depend</tt> is implemented as a shell function, the shell can be a bit picky about parsing its arguments. Follow these guidelines:
+
 
+
* All arguments to <tt>python_abi_depend</tt> should be on the same line, or -
+
* Additional lines should have a <tt>\</tt> line continuation character as the last line, and -
+
* Any atoms with > or < characters should be enclosed in double-quotes to avoid being interpreted as output/input redirection
+
 
+
==== python_abi_depend() gotchas ====
+
 
+
Occasionally, you may encounter emerge errors like this when developing new Python ebuilds:
+
 
+
<console>
+
xdev nova # ##i##emerge -p nova
+
 
+
These are the packages that would be merged, in order:
+
 
+
Calculating dependencies... done!
+
 
+
emerge: there are no ebuilds built with USE flags to satisfy "dev-python/lxml[python_abis_2.6?,python_abis_2.7?,python_abis_2.6-jython?,python_abis_2.7-pypy-1.7?,python_abis_2.7-pypy-1.8?]".
+
!!! One of the following packages is required to complete your request:
+
- dev-python/lxml-2.3.3::gentoo (Missing IUSE: python_abis_2.6-jython python_abis_2.7-pypy-1.7 python_abis_2.7-pypy-1.8)
+
(dependency required by "sys-cluster/nova-9999" [ebuild])
+
(dependency required by "nova" [argument])
+
</console>
+
 
+
Here's an explanation for why an error like this occurs. In this case, the <tt>lxml</tt> package has <tt>*-jython *-pypy-*</tt> in <tt>PYTHON_RESTRICTED_ABIS</tt>. Since <tt>nova</tt> depends on <tt>lxml</tt>, it should also be restricted so that it doesn't support Jython or PyPy. This makes sense, because if a dependency of <tt>nova</tt> can't run on a particular Python implementation, then neither can <tt>nova</tt> itself.
+
 
+
This emerge error can be fixed by adding <tt>*-jython *-pypy-*</tt> to <tt>nova</tt>'s <tt>PYTHON_RESTRICTED_ABIS</tt>. The lesson here is that any Python-based application can only support the subset of all platforms supported by all its dependencies, and <tt>PYTHON_RESTRICTED_ABIS</tt> must reflect this.
+
 
+
On an unrelated note, <tt>nova</tt> is configured with <tt>PYTHON_MULTIPLE_ABIS=1</tt> because it installs a ton of Python modules into <tt>site-packages</tt>.
+
 
+
==== common python_abi_depend() dependencies ====
+
 
+
This section is dedicated to providing examples on how to handle common dependencies that are found in Python-related packages:
+
 
+
===== argparse =====
+
 
+
If the Python package's <tt>setup.py</tt> contains something like this:
+
 
+
<pre>
+
if sys.version_info < (2, 7):
+
    requirements.append('argparse')
+
</pre>
+
 
+
...then use the following dependency syntax:
+
 
+
<pre>
+
(R)DEPEND="$(python_abi_depend virtual/python-argparse)"
+
</pre>
+
 
+
==== python_abi_depend() with -i and -e ====
+
 
+
<tt>python_abi_depend()</tt> has some advanced capabilities that you may need to take advantage of related to Python dependencies. Sometimes, your ebuild will only need a python module if certain versions of Python are installed. This functionality is typically needed in the following cases:
+
 
+
;module rolled into standard library: A module is needed, but in newer versions of python it is part of the standard library (so there is no need to depend on it for these newer python versions)
+
;module required for fixes: A module is needed, but only for newer versions of python, such as 3.x, where it implements compatibility features.
+
;optional module, not 100% compatible: A module can be used, but the dependencies are not supported with certain versions of Python.
+
 
+
Let's look at the first case, a module your ebuild needs that was rolled into the standard library starting with Python version 3.1. In this case, we only need to depend on this module for Python ABIs 3.0 or less. We would use the <tt>-e</tt> option to exclude the versions we are not interested in:
+
 
+
<pre>
+
RDEPEND="$(python_abi_depend -e "3.[1-9]" dev-python/mydep)"
+
</pre>
+
 
+
In the second case, we depend on a module only for Python ABIs 3 and up, since it implements fixes for python3. In this case, we will use the <tt>-i</tt> option to include only those versions for which we want to depend:
+
 
+
<pre>
+
RDEPEND="$(python_abi_depend -i "3.*" dev-python/mydep)"
+
</pre>
+
 
+
Third case: something doesn't make sense in the context with pypy:
+
 
+
<pre>
+
RDEPEND=">=sys-fs/udev-151
+
pygobject? ( $(python_abi_depend -e "*-pypy-*" dev-python/pygobject:2) )
+
pyqt4? ( $(python_abi_depend -e "*-pypy-*" dev-python/PyQt4) )
+
pyside? ( dev-python/pyside )
+
wxwidgets? ( $(python_abi_depend -e "3.* *-pypy-*" dev-python/wxpython) )"
+
</pre>
+
 
+
=== Links ===
+
 
+
* http://people.apache.org/~Arfrever/Gentoo_Python_Guide_for_Users
+
* http://people.apache.org/~Arfrever/Gentoo_Python_Supplementary_Guide_for_Developers
+
* http://people.apache.org/~Arfrever/EAPI_4-python_Specification
+
 
+
[[Category:Portage]]
+
[[Category:OpenStack]]
+

Revision as of 10:47, May 17, 2015


Welcome to Funtoo Linux

Fwiki.png


Getting Started

Install Funtoo

Create a Funtoo account, which allows you to get involved with the community. Log in to the wiki, forums, and bug tracker under the same username, and password by unified logins. See the Auth FAQ & General FAQ.

Report bugs, and suggest improvements to our bug tracker. We take all bugs seriously, and all work performed is tracked on our public bug tracker, for purposes of transparency. See the Funtoo Linux FAQ.

Funtoo Linux is a rolling-release Linux meta-distribution for x86 and ARM systems. It uses Portage as a package manager, and is run by Daniel Robbins, creator of Gentoo Linux. It is a continuation of Daniel's work on Gentoo Linux. Funtoo Linux is committed to software freedom, independence, and a user-centric model of software development, where our users help define the future direction of the project.

Our goal is to provide a best-of-breed Gentoo-based meta-distribution that is ideal for desktop and servers, and to progressively improve our core technology in thoughtful and innovative ways, while respecting Gentoo design principles, and the history of Unix and Linux.

Funtoo Linux provides optimized builds for your specific hardware to ensure the best possible performance, and offers the following features:

  • A completely systemd-free system, including GNOME 3.14 without systemd.
  • A new profile system to simplify system configuration.
  • Improved ease of install with pre-built Debian kernel provided in our install image.
  • A new system configuration tool, ego.
  • An advanced build and quality assurance tool called Metro, providing optimized install images and enhanced real-world testing.
  • The use of git as a default mechanism for getting updated ebuilds (build scripts) to you.
  • Modular networking scripts to simplify creating sophisticated server network configurations.
  • Enhanced Python support from the Progress overlay, and inclusion of other popular packages from the Gentoo ecosystem.

Support Funtoo and help us grow! Donate $15 per month and get a free SSD-based Funtoo Virtual Container.

News

Drobbins

Perl Updates

Gentoo has bumped perl from 5.20 to 5.22. Be sure to run perl-cleaner --all after the upgrade.
2015-07-25 by Drobbins
Drobbins

ARM Rebuild

ARM systems will use new stage3's that are not compatible with earlier versions.
2015-06-27 by Drobbins
More...

Official Projects

Ebuilds

Ebuild pages recently updated: {{#ask: | order=descending | sort=Modification date | format=list | limit=10 | searchlabel= }}