Difference between pages "Package:Mutt" and "Creating Python-related Ebuilds"

From Funtoo
(Difference between pages)
Jump to: navigation, search
m (~/.mutt/sidebar: removed some leftovers!?)
 
(The old way: python-updater)
 
Line 1: Line 1:
Mutt is a popular mail reader for Linux systems. This page is being created as a good place for people to place their mutt configurations to share with others. These configuration settings go in the user's <tt>~/.muttrc</tt> file.
+
== Summary ==
  
== Essential Mutt ==
+
Funtoo Linux incorporates Progress Overlay into the main Portage tree, which currently contains 400+ updated Python-related packages that implement next-generation Python support for ebuilds.
  
<pre>
+
== Building Python Modules with Portage ==
set pager_stop
+
</pre>
+
  
This turns off the default behavior of mutt where hitting space to scroll will automatically move to the next message when the end of the current message is reached. This is very annoying when scrolling in long emails like cron jobs, and the line above sets this behavior to off.
+
== Conceptual Overview ==
  
== angry_vincent's .muttrc: ==
+
Portage supports the ability to build python modules and extensions for all installed versions of python. This functionality has been implemented by <tt>python.eclass</tt>.
  
 +
To use <tt>python.eclass</tt>, you first want to investigate how python module building is integrated into any included <tt>Makefile</tt> or build script. Since we want <tt>python.eclass</tt> to build these modules for us, typically the first step is to tweak the <tt>Makefile</tt> or <tt>configure</tt> script to explicitly *not* build python modules. We will be handling that process using <tt>python.eclass</tt>.
  
<pre>
+
Another option, for those Python packages with a <tt>setup.py</tt> in the root source directory, is to use the <tt>distutils</tt> eclass. In this case, you generally do not need custom <tt>src_compile</tt> or <tt>src_install</tt> stages, as the eclass takes care of everything for you. However, you will still need to set the proper environment variables described below, as they are needed by <tt>python.eclass</tt> which is used by <tt>distutils.eclass</tt>. You will also want to set dependencies in <tt>DEPEND</tt> and <tt>RDEPEND</tt> as described in this article.
# General config
+
# ---------------
+
  
set folder=~/Mail   # mail folder
+
== Progress Improvements ==
set alias_file=~/.mail_aliases    # alises file
+
set arrow_cursor   # cursor is '->'
+
set attribution="%d, %n wrote:"    # beginning of mail answer
+
set copy=yes   # save mail copies
+
set edit_headers   # edit mail header
+
set editor="vim"                               # editor
+
set folder_format="%t%N %-30.30f %8s"                        # folder list look
+
set index_format="%4C %Z %{%b %d} %-31.31F %N (%4c) %s"      # mail list look
+
set mailcap_path="~/.mailcap"                               # path to .mailcap
+
set menu_scroll # scroll list by one line
+
set mail_check=5 # mail check interval
+
set pager_stop # at the end of mail do not move to next message
+
set postponed=+drafts # postponed mails
+
set print=ask-yes # ask before print
+
set print_command=lpr # print command
+
set record=+sent # where to save sent mails
+
set signature="~/.signature" # signature file
+
set visual=vim                  # editor caled by  ~v
+
  
# mail sort
+
This section describes the differences in approach between the old (classic) way of handling Python-related ebuilds, and the new way, which uses the updated eclass. Also note that the <tt>distutils</tt> eclass benefits from these updates as well, as it uses <tt>python.eclass</tt> behind-the-scenes.
set sort=threads
+
set sort_aux=reverse-date-received
+
set sort_browser=reverse-date
+
ignore *    # ignore headers so the mail body is not overloaded
+
unignore        from: subject to cc mail-followup-to \
+
                date x-mailer x-url user-agent reply-to  # fieids i like to see in mail body
+
  
# Colors
+
=== The old way: python-updater ===
color index brightcyan black ~N
+
color index brightyellow black ~F
+
color index black green ~T
+
color index brightred black ~D
+
mono index bold ~N
+
mono index bold ~F
+
mono index bold ~T
+
mono index bold ~D
+
  
# Highlights inside the body of a message.
+
With the older <tt>python.eclass</tt> in Gentoo (and that used to be in Funtoo Linux,) python modules are built for all installed versions of Python. However, no usable accounting information was stored by Portage so that it could "know" what versions of python the module was built for. This created a problem when new versions of Python are installed, as the new module would not be available for the new python, and Portage would not be aware of what needed to be done to fix this, if anything. This resulted in the creation of a tool called <tt>python-updater</tt> to rebuild all necessary python-related ebuilds so that they reflected the currently-installed versions of Python.
  
# URLs
+
Note that python-updater-0.10 and later are aware of the new Python eclass, and will not unnecessarily reinstall any packages which set <tt>PYTHON_MULTIPLE_ABIS</tt> and <tt>EAPI="4-python"</tt>, as manual rebuilding is not necessary.
color body brightgreen black "(http|ftp|news|telnet|finger)://[^ \"\t\r\n]*"
+
color body brightgreen black "mailto:[-a-z_0-9.]+@[-a-z_0-9.]+"
+
mono body bold "(http|ftp|news|telnet|finger)://[^ \"\t\r\n]*"
+
mono body bold "mailto:[-a-z_0-9.]+@[-a-z_0-9.]+"
+
  
# email addresses
+
'''TODO: add python-updater example'''
color body brightgreen black "[-a-z_0-9.%$]+@[-a-z_0-9.]+\\.[-a-z][-a-z]+"
+
mono body bold "[-a-z_0-9.%$]+@[-a-z_0-9.]+\\.[-a-z][-a-z]+"
+
  
# header
+
=== The new way: new python eclass ===
color header green black "^from:"
+
color header green black "^to:"
+
color header green black "^cc:"
+
color header green black "^date:"
+
color header yellow black "^newsgroups:"
+
color header yellow black "^reply-to:"
+
color header brightcyan black "^subject:"
+
color header red black "^x-spam-rule:"
+
color header green black "^x-mailer:"
+
color header yellow black "^message-id:"
+
color header yellow black "^Organization:"
+
color header yellow black "^Organisation:"
+
color header yellow black "^User-Agent:"
+
color header yellow black "^message-id: .*pine"
+
color header yellow black "^X-Fnord:"
+
color header yellow black "^X-WebTV-Stationery:"
+
color header yellow black "^X-Message-Flag:"
+
color header yellow black "^X-Spam-Status:"
+
color header yellow black "^X-SpamProbe:"
+
color header red black "^X-SpamProbe: SPAM"
+
  
# Coloring quoted text - coloring the first 7 levels:
+
The new <tt>python.eclass</tt> from Arfrever has a new API that can be used to write Python ebuilds. When this API is used, Portage can track and resolve missing builds of Python modules internally, so <tt>python-updater</tt> is not needed to be used for these ebuilds. However, until all ebuilds are converted to the new API, <tt>python-updater</tt> will need to be used to update these old-style ebuilds. Fortunately for us, over 400 Python-related ebuilds have been converted to take advantage of the new eclass, with more added weekly.
color quoted cyan black
+
color quoted1 yellow black
+
</pre>
+
  
== Interesting Color Options ==
+
== Setting up progress overlay ==
  
* http://github.com/altercation/mutt-colors-solarized
+
{{fancynote|Progess overlay is now merged in Funtoo current and stable - No manual steps are needed to benefit from it!}}
  
== golodhrim's mutt config ==
+
Notice, that progress overlay including an updated packages that also available in main portage tree!
 +
<pre># python-updater</pre> this still required to rebuild old-fashion python-related ebuilds
  
First I split up my muttconfig in several subfiles under <tt>~/.mutt</tt>. The resulting files are:
 
  
* <code>~/.mutt/account_hooks</code>
+
== Updating the packages from progress overlay ==
* <code>~/.mutt/colors</code>
+
Package updates are done the same way than before:
* <code>~/.mutt/folder_hooks</code>
+
<pre>emerge -avuDN @world</pre>
* <code>~/.mutt/gpg</code>
+
* <code>~/.mutt/lists</code>
+
* <code>~/.mutt/macros</code>
+
* <code>~/.mutt/mutt-alias</code>
+
* <code>~/.mutt/muttrc</code>
+
* <code>~/.mutt/sidebar</code>
+
* <code>~/.secret/.passwd.gpg</code>
+
  
=== account_hooks ===
+
Several python-related packages should be updated to progress versions, with the new PYTHON_ABIS flags.
Edit <code>~/.mutt/account_hooks</code>:
+
  
<pre>
+
== Developer Docs ==
#-------------------------------------------------------------------------
+
# Account Hooks
+
#-------------------------------------------------------------------------
+
account-hook . 'unset imap_user imap_pass'
+
account-hook 'imap://user@imaphost/' "set imap_user=user imap_pass=$my_pass1 "
+
</pre>
+
  
=== colors ===
+
=== Ebuild Converstion to new Python eclass ===
Edit the file <tt>~/.mutt/colors</tt>:
+
  
<pre>
+
Conversion from previous EAPIs and eclasses to the new progress python eclass is fairly straightforward. This section tries to explain the steps.
#-------------------------------------------------------------------------     
+
 
# Set colors
+
==== 4-python EAPI ====
#-------------------------------------------------------------------------
+
color  attachment  brightmagenta      default
+
color  error      brightred          default
+
color  hdrdefault  red                default
+
color  indicator  brightyellow        red
+
color  markers    brightcyan          default
+
color  message    brightcyan          default
+
color  normal      default            default
+
color  quoted      brightblue          default
+
color  search      default            green
+
color  signature  red                default
+
color  status      yellow              blue
+
color  tilde      magenta            default
+
color  tree        magenta            default
+
</pre>
+
  
=== folder_hooks ===
+
First, set EAPI to 4-python:
  
 
<pre>
 
<pre>
#--------------------------------------------------------------------------
+
EAPI=4-python
# Folders, mailboxes and folder hooks
+
#--------------------------------------------------------------------------
+
# Setup for imap-user in account_hooks
+
set folder="imap://user@imaphost/"
+
mailboxes =INBOX =INBOX/Archives =INBOX/Drafts =INBOX/Sent =INBOX/Trash
+
folder-hook 'imap://user@imaphost/' " \
+
    set folder=imap://user@imaphost/ \
+
        postponed=+INBOX/Drafts \
+
        record=+INBOX/Sent \
+
        smtp_url=smtp://user@smtphost \
+
        smtp_pass=$my_pass1 \
+
        #signature=~/.sig/pr.txt \
+
        from='Name Familyname <name@host> ' \
+
        realname='Realname' \
+
        pgp_sign_as='PGP-signature' \
+
        spoolfile='imap://user@imaphost/' "
+
 
</pre>
 
</pre>
  
=== ~/.mutt/gpg ===
+
Next, you will need to understand the type of Python application or module you are trying to install. The primary determining factor in how to proceed is to see whether the Python application installs anything into Python's <tt>site-packages</tt> directory. Essentially all Python modules do this, and some applications do this as well. If a package installs stuff into <tt>site-packages</tt>, then you should have its ebuild take advantage of the Python eclass "multiple ABI" support.
  
 +
In this context, "multiple ABI" refers to potentially multiple versions of Python that may be installed on your system. Each installed major.minor version of Python, plus Python derivatives like Jython and pypy, will have their own <tt>site-packages</tt> directory and the Python eclass can take care of installing a version of the Python modules for each version of Python on your system. If the application has binaries, a Python wrapper will also be installed that will hand off control to the proper version of Python automatically.
 +
 +
==== Non-Multiple-ABI Support ====
 +
 +
If you are not supporting multiple ABIs, which is the least common option, you will generally just set <tt>PYTHON_DEPEND</tt> to refer to the version(s) of Python you require. Documentation for <tt>PYTHON_DEPEND</tt> syntax (which has been changed in the new python eclass) is available [http://people.apache.org/~Arfrever/Gentoo_Python_Supplementary_Guide_for_Developers here].
 +
 +
==== Multiple ABI Support ====
 +
 +
So, if you are using multiple ABI support, which will be the case for all Python modules, proceed as follows. Set <tt>PYTHON_MULTIPLE_ABIS</tt> prior to any inherit lines:
  
 
<pre>
 
<pre>
# vim:syn=muttrc                                                               
+
PYTHON_MULTIPLE_ABIS=1
##
+
</pre>
  
set smime_decrypt_use_default_key=yes
+
Note the new variable names. <tt>PYTHON_MULTIPLE_ABIS</tt> turns on the functionality to build multiple versions of python modules for each version of python defined in the user's <tt>PYTHON_ABIS</tt> setting. Note that <tt>PYTHON_ABIS</tt> defaults to "<tt>2.7 3.2</tt>" in Funtoo Linux profiles and can be overridden in <tt>/etc/make.conf</tt> if you would like the python eclass to target other Python ABIs (like jython, for example, or 2.6) that you might be interested in.
  
# Decode application/pgp attachments like so:
+
When <tt>PYTHON_MULTIPLE_ABIS</tt> is set and <tt>EAPI</tt> is set to <tt>4-python</tt>, then new USE flags will be automatically added to the <tt>IUSE</tt> with the <tt>python_abis_</tt> prefix.
set pgp_decode_command="/usr/bin/gpg %?p?--passphrase-fd 0? --no-verbose --batch --output - %f"
+
If you want to see what ABIs are supported, see the <tt>_PYTHON_GLOBALLY_SUPPORTED_ABIS</tt> variable defined in <tt>python.eclass</tt>.
  
# And use this to verify pgp signatures:
+
Also note that for multiple ABI support, <tt>PYTHON_DEPEND</tt> is set to a reasonable default value so you should almost always never need to set it yourself.
set pgp_verify_command="/usr/bin/gpg --no-verbose --batch --output - --verify %s %f"
+
  
# How to decrypt pgp encrypted messages:
+
==== PYTHON_RESTRICTED_ABIS ====
set pgp_decrypt_command="/usr/bin/gpg --passphrase-fd 0 --no-verbose --batch --output - %f"
+
  
# How to pgp sign a message:
+
<tt>PYTHON_RESTRICTED_ABIS</tt> variable set in the ebuild specifies a space-separated list of patterns of Python ABIs ''not'' supported by the current package. The settings in <tt>PYTHON_RESTRICTED_ABIS</tt> will remove certain ABIs from the auto-<tt>IUSE</tt> expansion that happens when <tt>EAPI</tt> is set to <tt>4-python</tt> and <tt>PYTHON_MULTIPLE_ABIS</tt> is set to <tt>1</tt> (This behavior is described above.) It will also limit the behavior of the <tt>python_abi_depend()</tt> function, described below. And of course, it will tell the python eclass to not build any python modules for these python versions, even if they happen to be installed on the system.
set pgp_sign_command="/usr/bin/gpg --no-verbose --batch --output - --passphrase-fd 0 --armor --detach-sign --textmode %?a?-u %a? %f"
+
  
# How to pgp clearsign a message:
+
Use it like this, before the eclasses are inherited:
set pgp_clearsign_command="/usr/bin/gpg --no-verbose --batch --output - --passphrase-fd 0 --armor --textmode --clearsign %?a?-u %a? %f"
+
  
# Import a pgp key from a message into my public keyring as follows:
+
<pre>
set pgp_import_command="/usr/bin/gpg --no-verbose --import -v %f"
+
PYTHON_RESTRICTED_ABIS="3.*"
 +
</pre>
  
# Use this to export a key from my public keyring:
+
Note that <tt>PYTHON_RESTRICTED_ABIS</tt> is a glob-style pattern. If you want to have it build with everything ''except'' 3.2 and above and jython, use this pattern:
set pgp_export_command="/usr/bin/gpg --no-verbose --export --armor %r"
+
  
# Verify key information (from the key selection menu):
+
<pre>
set pgp_verify_key_command="/usr/bin/gpg --verbose --batch --fingerprint --check-sigs %r"
+
PYTHON_RESTRICTED_ABIS="3.[2-9] *-jython"
 +
</pre>
  
# List my public keyring like so:
+
Note that the wildcard you should use to disable Jython is <tt>*-jython</tt>, while the wildcard you should use to disable PyPy is <tt>*-pypy-*</tt>.
set pgp_list_pubring_command="/usr/bin/gpg --no-verbose --batch --with-colons --list-keys %r"
+
  
# List my private keyring like so:
+
==== python_abi_depend() ====
set pgp_list_secring_command="/usr/bin/gpg --no-verbose --batch --with-colons --list-secret-keys %r"
+
  
# Automatically sign outgoing messages
+
One of the cool things about the new Python eclass is the "cascading" dependency functionality which is implemented via USE variables. Basically, if your new-style python module ebuild was built for python-2.7, and it has dependencies, then it can depend on the module it needs being built ''for'' the installed version of python-2.7 specifically. This happens by having your ebuild depend upon the <tt>python_abis_2.7</tt> <tt>USE</tt> flag being set in the installed ebuild it depends upon.
set pgp_autosign=yes
+
  
# Timeout (in seconds) for cached passphrases:
+
For this functionality to work, the ebuild you depend on must be a new-style (<tt>EAPI 4-python</tt>) ebuild too, and must be enabled to support multiple python ABIS via the <tt>PYTHON_MULTIPLE_ABIS=1</tt> setting. When these requirements are met, you will want to covert an old-style dependency that looks like this:
set pgp_timeout=1800
+
  
# Text to show before a good signature:
+
<pre>
set pgp_good_sign="^gpg: Good signature from"
+
RDEPEND="foo >=dev-python/mydep-2 bar"
 
</pre>
 
</pre>
  
=== ~/.mutt/lists ===
+
...to this new format:
 
+
  
 
<pre>
 
<pre>
#-------------------------------------------------------------------------     
+
RDEPEND="foo $(python_abi_depend ">=dev-python/mydep-2" ) bar"
# Mailinglist Subscriptions
+
#-------------------------------------------------------------------------
+
# Syntax:
+
# subscribe mailinglist@domain.com
+
 
</pre>
 
</pre>
  
=== ~/.mutt/macros ===
+
The <tt>python_abi_depend</tt> function will expand to conditionally depend on the proper USE flags being set in <tt>dev-python/mydep-2</tt> based on what USE flags are currently set for ''this'' ebuild when it builds. This is how the cascading functionality works. Use it in <tt>DEPEND</tt>, <tt>RDEPEND</tt> and <tt>PDEPEND</tt> as appropriate. You can include multiple ebuilds as arguments to <tt>python_abi_depend</tt>.
  
 +
Please note that due to the fact that <tt>python_abi_depend</tt> is implemented as a shell function, the shell can be a bit picky about parsing its arguments. Follow these guidelines:
  
<pre>
+
* All arguments to <tt>python_abi_depend</tt> should be on the same line, or -
# Macros to toggle the sidebar visibility                                     
+
* Additional lines should have a <tt>\</tt> line continuation character as the last line, and -
macro index b '<enter-command>toggle sidebar_visible<enter><refresh>'
+
* Any atoms with > or < characters should be enclosed in double-quotes to avoid being interpreted as output/input redirection
macro pager b '<enter-command>toggle sidebar_visible<enter><redraw-screen>'
+
  
# abook query
+
==== python_abi_depend() gotchas ====
macro index,pager A "<pipe-message>abook --add-email-quiet<return>" "add the sender address to abook"
+
macro generic,index,pager \Cb "abook" "launch abook"
+
</pre>
+
  
=== ~/.mutt/alias ===
+
Occasionally, you may encounter emerge errors like this when developing new Python ebuilds:
  
 +
<console>
 +
xdev nova # ##i##emerge -p nova
  
<pre>
+
These are the packages that would be merged, in order:
alias nick Realname <email@host.tld>
+
</pre>
+
  
=== ~/.mutt/muttrc ===
+
Calculating dependencies... done!
  
 +
emerge: there are no ebuilds built with USE flags to satisfy "dev-python/lxml[python_abis_2.6?,python_abis_2.7?,python_abis_2.6-jython?,python_abis_2.7-pypy-1.7?,python_abis_2.7-pypy-1.8?]".
 +
!!! One of the following packages is required to complete your request:
 +
- dev-python/lxml-2.3.3::gentoo (Missing IUSE: python_abis_2.6-jython python_abis_2.7-pypy-1.7 python_abis_2.7-pypy-1.8)
 +
(dependency required by "sys-cluster/nova-9999" [ebuild])
 +
(dependency required by "nova" [argument])
 +
</console>
  
<pre>
+
Here's an explanation for why an error like this occurs. In this case, the <tt>lxml</tt> package has <tt>*-jython *-pypy-*</tt> in <tt>PYTHON_RESTRICTED_ABIS</tt>. Since <tt>nova</tt> depends on <tt>lxml</tt>, it should also be restricted so that it doesn't support Jython or PyPy. This makes sense, because if a dependency of <tt>nova</tt> can't run on a particular Python implementation, then neither can <tt>nova</tt> itself.
# Some minimal Mutt settings, Gentoo-style. These reflect the Gentoo
+
# predilection for maildir folders.
+
#
+
# Please don't add settings to this file to change other user
+
# preferences (such as colors), since those can be hard for a user to
+
# undo if their preference doesn't match yours!  For example, it is
+
# *impossible* currently in mutt to remove color settings from objects
+
# other than the index.
+
  
#-------------------------------------------------------------------------
+
This emerge error can be fixed by adding <tt>*-jython *-pypy-*</tt> to <tt>nova</tt>'s <tt>PYTHON_RESTRICTED_ABIS</tt>. The lesson here is that any Python-based application can only support the subset of all platforms supported by all its dependencies, and <tt>PYTHON_RESTRICTED_ABIS</tt> must reflect this.
# Process the password file first
+
#-------------------------------------------------------------------------
+
set my_tmp=`gpg -o ~/.secret/.tmp -d ~/.secret/.passwd.gpg`                   
+
set my_pass1=`awk '/user:/ {print $2}' ~/.secret/.tmp`
+
set my_del=`rm -f ~/.secret/.tmp`
+
+
#-------------------------------------------------------------------------
+
# Further customisations
+
#-------------------------------------------------------------------------
+
set smart_wrap = yes
+
set sort = 'threads'
+
set sort_aux = 'last-date-received'
+
set imap_check_subscribed
+
  
ignore "Authentication-Results:"
+
On an unrelated note, <tt>nova</tt> is configured with <tt>PYTHON_MULTIPLE_ABIS=1</tt> because it installs a ton of Python modules into <tt>site-packages</tt>.
ignore "DoaminKey-Signature:"
+
ignore "DKIM-Signature:"
+
hdr_order Date From To Cc
+
  
#-------------------------------------------------------------------------
+
==== common python_abi_depend() dependencies ====
# Configuration variables
+
#-------------------------------------------------------------------------
+
set abort_nosubject=yes
+
set abort_unmodified=yes
+
set query_command="abook --mutt-query '%s'"
+
set alias_file="~/.mutt/mutt-alias"
+
set alias_format="%4n %t %a %r"
+
source $alias_file
+
set assumed_charset="utf-8"
+
set attach_charset="utf-8"
+
set charset="utf-8"
+
set date_format=""
+
set edit_headers=yes
+
set editor='vim + -c "set textwidth=72" -c "set wrap" -c "set nocp" -c "?^$"        '
+
set folder="~/.offlineimap"
+
set forward_quote=yes
+
set header_cache=~/.mutt/cache/headers
+
set message_cachedir=~/.mutt/cache/bodies
+
set certificate_file=~/.mutt/certificates
+
set help=yes
+
set imap_idle=yes
+
set imap_peek=yes
+
set imap_servernoise=no
+
set include=yes
+
set mail_check=60
+
set mbox_type=Maildir
+
set menu_move_off=no
+
set menu_scroll=no
+
set mime_forward=ask-no
+
set pager_context=2
+
set pager_stop=yes
+
set postponed="~/.mutt/mail/postponed"
+
set realname="Martin 'golodhrim' Scholz"
+
set record="~/.mutt/mail/sent"
+
set reply_regexp="^(re([\[0-9\]+])*|betr):[ \t]*"
+
#set reply_to=yes
+
set tilde=yes
+
  
#--------------------------------------------------------------------------
+
This section is dedicated to providing examples on how to handle common dependencies that are found in Python-related packages:
# muttprint for printing
+
#--------------------------------------------------------------------------
+
set print_command="muttprint"
+
  
#--------------------------------------------------------------------------
+
===== argparse =====
# Automatically process html mails
+
#--------------------------------------------------------------------------
+
auto_view text/html
+
  
#--------------------------------------------------------------------------
+
If the Python package's <tt>setup.py</tt> contains something like this:
# Other configuration files
+
 
#--------------------------------------------------------------------------
+
<pre>
source ~/.mutt/account_hooks
+
if sys.version_info < (2, 7):
source ~/.mutt/folder_hooks
+
    requirements.append('argparse')
source ~/.mutt/colors
+
source ~/.mutt/sidebar
+
source ~/.mutt/gpg
+
source ~/.mutt/macros
+
source ~/.mutt/lists
+
 
</pre>
 
</pre>
  
=== ~/.mutt/sidebar ===
+
...then use the following dependency syntax:
  
 
<pre>
 
<pre>
#--------------------------------------------------------------------------   
+
(R)DEPEND="$(python_abi_depend virtual/python-argparse)"
# Sidebar configuration
+
</pre>
#--------------------------------------------------------------------------
+
set sidebar_width=30
+
set sidebar_visible=yes
+
set sidebar_delim='|'
+
color sidebar_new yellow default
+
  
#--------------------------------------------------------------------------
+
==== python_abi_depend() with -i and -e ====
# Sidebar keys
+
 
#--------------------------------------------------------------------------
+
<tt>python_abi_depend()</tt> has some advanced capabilities that you may need to take advantage of related to Python dependencies. Sometimes, your ebuild will only need a python module if certain versions of Python are installed. This functionality is typically needed in the following cases:
bind index \CP sidebar-prev
+
bind index \CN sidebar-next
+
bind index \CO sidebar-open
+
bind pager \CP sidebar-prev
+
bind pager \CN sidebar-next
+
bind pager \CO sidebar-open
+
</pre>
+
  
=== ~/.secret/.passwd.gpg ===
+
;module rolled into standard library: A module is needed, but in newer versions of python it is part of the standard library (so there is no need to depend on it for these newer python versions)
For this file follow the next steps:
+
;module required for fixes: A module is needed, but only for newer versions of python, such as 3.x, where it implements compatibility features.
* Step 1:
+
;optional module, not 100% compatible: A module can be used, but the dependencies are not supported with certain versions of Python.
Create a folder <tt>~/.secret</tt> and inside that directory, create a file called <tt>.passwd</tt>. Enter the following into it:
+
  
 +
Let's look at the first case, a module your ebuild needs that was rolled into the standard library starting with Python version 3.1. In this case, we only need to depend on this module for Python ABIs 3.0 or less. We would use the <tt>-e</tt> option to exclude the versions we are not interested in:
  
 
<pre>
 
<pre>
user1:    passwd1
+
RDEPEND="$(python_abi_depend -e "3.[1-9]" dev-python/mydep)"
user2:    passwd2
+
 
</pre>
 
</pre>
  
where "user1" and "user2" are identifiers for the accounts you added in the account_hooks and folder_hooks files, as well as at the start of the muttrc.
+
In the second case, we depend on a module only for Python ABIs 3 and up, since it implements fixes for python3. In this case, we will use the <tt>-i</tt> option to include only those versions for which we want to depend:
* Step 2:
+
Now, encrypt the file with your <tt>gpg-key</tt>. If you don't have one, execute <tt>gpg --gen-key</tt> and create one. After that, don't forget to delete your unencrypted password storage:
+
<console>
+
$ ##bl##gpg -r 0x<Your-ID-fingerprint> -o .passwd.gpg --encrypt .passwd
+
$ ##bl##rm -f .passwd
+
</console>
+
  
=== Mutt postfix setup ===
+
<pre>
Postfix is another popular and powerful mail transfer agent which is easier to configure than sendmail. Let's start a quick postfix mail transfer setup that will work with mutt. Set the <tt>sasl,berkdb</tt> USE flags to <tt>mail-mta/potsfix</tt> and <tt>mail-client/mutt</tt> in <tt>/etc/portage/package.use/</tt>:
+
RDEPEND="$(python_abi_depend -i "3.*" dev-python/mydep)"
<console>
+
</pre>
# ##i##echo "mail-mta/postfix berkdb sasl" >> /etc/portage/package.use/mail
+
# ##i##echo "mail-client/mutt berkdb sasl" >> /etc/portage/package.use/mail
+
# ##i##emerge -uN1 mutt postfix
+
</console>
+
Edit <tt>/etc/postfix/main.cf</tt> and add the following lines:
+
  
 +
Third case: something doesn't make sense in the context with pypy:
  
 
<pre>
 
<pre>
relayhost = smtp.gmail.com:587
+
RDEPEND=">=sys-fs/udev-151
smtp_use_tls = yes
+
pygobject? ( $(python_abi_depend -e "*-pypy-*" dev-python/pygobject:2) )
smtp_sasl_auth_enable = yes
+
pyqt4? ( $(python_abi_depend -e "*-pypy-*" dev-python/PyQt4) )
smtp_sasl_password_maps = hash:/etc/postfix/gmail_passwd
+
pyside? ( dev-python/pyside )
smtp_sasl_security_options = noanonymous
+
wxwidgets? ( $(python_abi_depend -e "3.* *-pypy-*" dev-python/wxpython) )"
 
</pre>
 
</pre>
  
Create and edit the aforementioned authorization file: <tt>gmail_passwd</tt>:
+
=== Links ===
<console>
+
 
# ##i##touch /etc/postfix/gmail_passwd
+
* http://people.apache.org/~Arfrever/Gentoo_Python_Guide_for_Users
# ##i##echo "smtp.gmail.com:587 my.name@gmail.com:password" >> /etc/postfix/gmail_passwd
+
* http://people.apache.org/~Arfrever/Gentoo_Python_Supplementary_Guide_for_Developers
</console>
+
* http://people.apache.org/~Arfrever/EAPI_4-python_Specification
Where <my.name> is a gmail account and passwd is the account password.
+
Convert <tt>gmail_passwd</tt> into Berkeley DB format, secure the file, and restart the postfix daemon:
+
<console>
+
# ##i##postmap /etc/postfix/gmail_passwd
+
# ##i##chown root:postfix /etc/postfix/gmail_passwd*
+
# ##i##chmod 0640 /etc/postfix/gmail_passwd*
+
# ##i##/etc/init.d/postfix restart
+
</console>
+
  
[[Category:HOWTO]]
+
[[Category:Featured]]
[[Category:First Steps]]
+
[[Category:Portage]]
 +
[[Category:Python]]
 +
[[Category:OpenStack]]

Revision as of 03:04, 14 April 2012

Summary

Funtoo Linux incorporates Progress Overlay into the main Portage tree, which currently contains 400+ updated Python-related packages that implement next-generation Python support for ebuilds.

Building Python Modules with Portage

Conceptual Overview

Portage supports the ability to build python modules and extensions for all installed versions of python. This functionality has been implemented by python.eclass.

To use python.eclass, you first want to investigate how python module building is integrated into any included Makefile or build script. Since we want python.eclass to build these modules for us, typically the first step is to tweak the Makefile or configure script to explicitly *not* build python modules. We will be handling that process using python.eclass.

Another option, for those Python packages with a setup.py in the root source directory, is to use the distutils eclass. In this case, you generally do not need custom src_compile or src_install stages, as the eclass takes care of everything for you. However, you will still need to set the proper environment variables described below, as they are needed by python.eclass which is used by distutils.eclass. You will also want to set dependencies in DEPEND and RDEPEND as described in this article.

Progress Improvements

This section describes the differences in approach between the old (classic) way of handling Python-related ebuilds, and the new way, which uses the updated eclass. Also note that the distutils eclass benefits from these updates as well, as it uses python.eclass behind-the-scenes.

The old way: python-updater

With the older python.eclass in Gentoo (and that used to be in Funtoo Linux,) python modules are built for all installed versions of Python. However, no usable accounting information was stored by Portage so that it could "know" what versions of python the module was built for. This created a problem when new versions of Python are installed, as the new module would not be available for the new python, and Portage would not be aware of what needed to be done to fix this, if anything. This resulted in the creation of a tool called python-updater to rebuild all necessary python-related ebuilds so that they reflected the currently-installed versions of Python.

Note that python-updater-0.10 and later are aware of the new Python eclass, and will not unnecessarily reinstall any packages which set PYTHON_MULTIPLE_ABIS and EAPI="4-python", as manual rebuilding is not necessary.

TODO: add python-updater example

The new way: new python eclass

The new python.eclass from Arfrever has a new API that can be used to write Python ebuilds. When this API is used, Portage can track and resolve missing builds of Python modules internally, so python-updater is not needed to be used for these ebuilds. However, until all ebuilds are converted to the new API, python-updater will need to be used to update these old-style ebuilds. Fortunately for us, over 400 Python-related ebuilds have been converted to take advantage of the new eclass, with more added weekly.

Setting up progress overlay

Note: Progess overlay is now merged in Funtoo current and stable - No manual steps are needed to benefit from it!

Notice, that progress overlay including an updated packages that also available in main portage tree!

# python-updater
this still required to rebuild old-fashion python-related ebuilds


Updating the packages from progress overlay

Package updates are done the same way than before:

emerge -avuDN @world

Several python-related packages should be updated to progress versions, with the new PYTHON_ABIS flags.

Developer Docs

Ebuild Converstion to new Python eclass

Conversion from previous EAPIs and eclasses to the new progress python eclass is fairly straightforward. This section tries to explain the steps.

4-python EAPI

First, set EAPI to 4-python:

EAPI=4-python

Next, you will need to understand the type of Python application or module you are trying to install. The primary determining factor in how to proceed is to see whether the Python application installs anything into Python's site-packages directory. Essentially all Python modules do this, and some applications do this as well. If a package installs stuff into site-packages, then you should have its ebuild take advantage of the Python eclass "multiple ABI" support.

In this context, "multiple ABI" refers to potentially multiple versions of Python that may be installed on your system. Each installed major.minor version of Python, plus Python derivatives like Jython and pypy, will have their own site-packages directory and the Python eclass can take care of installing a version of the Python modules for each version of Python on your system. If the application has binaries, a Python wrapper will also be installed that will hand off control to the proper version of Python automatically.

Non-Multiple-ABI Support

If you are not supporting multiple ABIs, which is the least common option, you will generally just set PYTHON_DEPEND to refer to the version(s) of Python you require. Documentation for PYTHON_DEPEND syntax (which has been changed in the new python eclass) is available here.

Multiple ABI Support

So, if you are using multiple ABI support, which will be the case for all Python modules, proceed as follows. Set PYTHON_MULTIPLE_ABIS prior to any inherit lines:

PYTHON_MULTIPLE_ABIS=1

Note the new variable names. PYTHON_MULTIPLE_ABIS turns on the functionality to build multiple versions of python modules for each version of python defined in the user's PYTHON_ABIS setting. Note that PYTHON_ABIS defaults to "2.7 3.2" in Funtoo Linux profiles and can be overridden in /etc/make.conf if you would like the python eclass to target other Python ABIs (like jython, for example, or 2.6) that you might be interested in.

When PYTHON_MULTIPLE_ABIS is set and EAPI is set to 4-python, then new USE flags will be automatically added to the IUSE with the python_abis_ prefix. If you want to see what ABIs are supported, see the _PYTHON_GLOBALLY_SUPPORTED_ABIS variable defined in python.eclass.

Also note that for multiple ABI support, PYTHON_DEPEND is set to a reasonable default value so you should almost always never need to set it yourself.

PYTHON_RESTRICTED_ABIS

PYTHON_RESTRICTED_ABIS variable set in the ebuild specifies a space-separated list of patterns of Python ABIs not supported by the current package. The settings in PYTHON_RESTRICTED_ABIS will remove certain ABIs from the auto-IUSE expansion that happens when EAPI is set to 4-python and PYTHON_MULTIPLE_ABIS is set to 1 (This behavior is described above.) It will also limit the behavior of the python_abi_depend() function, described below. And of course, it will tell the python eclass to not build any python modules for these python versions, even if they happen to be installed on the system.

Use it like this, before the eclasses are inherited:

PYTHON_RESTRICTED_ABIS="3.*"

Note that PYTHON_RESTRICTED_ABIS is a glob-style pattern. If you want to have it build with everything except 3.2 and above and jython, use this pattern:

PYTHON_RESTRICTED_ABIS="3.[2-9] *-jython"

Note that the wildcard you should use to disable Jython is *-jython, while the wildcard you should use to disable PyPy is *-pypy-*.

python_abi_depend()

One of the cool things about the new Python eclass is the "cascading" dependency functionality which is implemented via USE variables. Basically, if your new-style python module ebuild was built for python-2.7, and it has dependencies, then it can depend on the module it needs being built for the installed version of python-2.7 specifically. This happens by having your ebuild depend upon the python_abis_2.7 USE flag being set in the installed ebuild it depends upon.

For this functionality to work, the ebuild you depend on must be a new-style (EAPI 4-python) ebuild too, and must be enabled to support multiple python ABIS via the PYTHON_MULTIPLE_ABIS=1 setting. When these requirements are met, you will want to covert an old-style dependency that looks like this:

RDEPEND="foo >=dev-python/mydep-2 bar"

...to this new format:

RDEPEND="foo $(python_abi_depend ">=dev-python/mydep-2" ) bar"

The python_abi_depend function will expand to conditionally depend on the proper USE flags being set in dev-python/mydep-2 based on what USE flags are currently set for this ebuild when it builds. This is how the cascading functionality works. Use it in DEPEND, RDEPEND and PDEPEND as appropriate. You can include multiple ebuilds as arguments to python_abi_depend.

Please note that due to the fact that python_abi_depend is implemented as a shell function, the shell can be a bit picky about parsing its arguments. Follow these guidelines:

  • All arguments to python_abi_depend should be on the same line, or -
  • Additional lines should have a \ line continuation character as the last line, and -
  • Any atoms with > or < characters should be enclosed in double-quotes to avoid being interpreted as output/input redirection

python_abi_depend() gotchas

Occasionally, you may encounter emerge errors like this when developing new Python ebuilds:

xdev nova # emerge -p nova

These are the packages that would be merged, in order:

Calculating dependencies... done!

emerge: there are no ebuilds built with USE flags to satisfy "dev-python/lxml[python_abis_2.6?,python_abis_2.7?,python_abis_2.6-jython?,python_abis_2.7-pypy-1.7?,python_abis_2.7-pypy-1.8?]".
!!! One of the following packages is required to complete your request:
- dev-python/lxml-2.3.3::gentoo (Missing IUSE: python_abis_2.6-jython python_abis_2.7-pypy-1.7 python_abis_2.7-pypy-1.8)
(dependency required by "sys-cluster/nova-9999" [ebuild])
(dependency required by "nova" [argument])

Here's an explanation for why an error like this occurs. In this case, the lxml package has *-jython *-pypy-* in PYTHON_RESTRICTED_ABIS. Since nova depends on lxml, it should also be restricted so that it doesn't support Jython or PyPy. This makes sense, because if a dependency of nova can't run on a particular Python implementation, then neither can nova itself.

This emerge error can be fixed by adding *-jython *-pypy-* to nova's PYTHON_RESTRICTED_ABIS. The lesson here is that any Python-based application can only support the subset of all platforms supported by all its dependencies, and PYTHON_RESTRICTED_ABIS must reflect this.

On an unrelated note, nova is configured with PYTHON_MULTIPLE_ABIS=1 because it installs a ton of Python modules into site-packages.

common python_abi_depend() dependencies

This section is dedicated to providing examples on how to handle common dependencies that are found in Python-related packages:

argparse

If the Python package's setup.py contains something like this:

if sys.version_info < (2, 7):
    requirements.append('argparse')

...then use the following dependency syntax:

(R)DEPEND="$(python_abi_depend virtual/python-argparse)"

python_abi_depend() with -i and -e

python_abi_depend() has some advanced capabilities that you may need to take advantage of related to Python dependencies. Sometimes, your ebuild will only need a python module if certain versions of Python are installed. This functionality is typically needed in the following cases:

module rolled into standard library
A module is needed, but in newer versions of python it is part of the standard library (so there is no need to depend on it for these newer python versions)
module required for fixes
A module is needed, but only for newer versions of python, such as 3.x, where it implements compatibility features.
optional module, not 100% compatible
A module can be used, but the dependencies are not supported with certain versions of Python.

Let's look at the first case, a module your ebuild needs that was rolled into the standard library starting with Python version 3.1. In this case, we only need to depend on this module for Python ABIs 3.0 or less. We would use the -e option to exclude the versions we are not interested in:

RDEPEND="$(python_abi_depend -e "3.[1-9]" dev-python/mydep)"

In the second case, we depend on a module only for Python ABIs 3 and up, since it implements fixes for python3. In this case, we will use the -i option to include only those versions for which we want to depend:

RDEPEND="$(python_abi_depend -i "3.*" dev-python/mydep)"

Third case: something doesn't make sense in the context with pypy:

RDEPEND=">=sys-fs/udev-151
	pygobject? ( $(python_abi_depend -e "*-pypy-*" dev-python/pygobject:2) )
	pyqt4? ( $(python_abi_depend -e "*-pypy-*" dev-python/PyQt4) )
	pyside? ( dev-python/pyside )
	wxwidgets? ( $(python_abi_depend -e "3.* *-pypy-*" dev-python/wxpython) )"

Links