Difference between pages "The Gentoo.org Redesign, Part 1" and "Ebuild Functions"

From Funtoo
(Difference between pages)
Jump to navigation Jump to search
 
(→‎Skipping over a function: use no-op for skipped phase function)
 
Line 1: Line 1:
{{Article
== Ebuild Functions ==
|Subtitle=A site reborn
|Summary=Have you ever woken up one morning and suddenly realized that your cute little personal development Web site isn't really that great? If so, you're in good company. In this series, Daniel Robbins shares his experiences as he redesigns the www.gentoo.org Web site using technologies like XML, XSLT, and Python. Along the way, you may find some excellent approaches to use for your next Web site redesign. In this article, Daniel creates a user-centric action plan and introduces pytext, an embedded Python interpreter.
|Author=Drobbins
|Next in Series=The Gentoo.org Redesign, Part 2
}}
==  An unruly horde ==


Fellow software developer, may I ask you a question? Why is it that although many of us are intimately familiar with Web technologies such as HTML, CGI, Perl, Python, Java technology, and XML, our very own Web sites -- the ones devoted to our precious development projects -- look like they were thrown together by an unruly horde of hyperactive 12-year-olds? Why, oh why, is this so?
Ebuilds provide the ability to define various shell functions that are used to specify various actions relating to building and installing a source or binary package on a user's system. When an ebuild is emerged, the following functions are called, in order:


Could it be because most of the time, we've left our Web site out to rot while we squander our precious time hacking away on our free software projects? The answer, at least in my case, is a most definite "Yes."
* <tt>pkg_setup</tt> - variable intialization and sanity checks
* <tt>src_unpack</tt>
* <tt>src_prepare</tt>
* <tt>src_configure</tt>
* <tt>src_compile</tt>
* <tt>src_install</tt>


When I'm not writing articles for IBM developerWorks or being a new dad, I'm feverishly working on the next release of Gentoo Linux, along with my skilled team of volunteers. And, yes, Gentoo Linux has its own Web site (see Resources). As of right now (March 2001), our Web site isn't that special; that's because we don't spend much time working on it because we're generally engrossed in improving Gentoo Linux itself. Sure, our site does have several admittedly cute logos that I whipped up using Xara X (see Resources), but when you look past the eye candy, our site leaves a lot to be desired. Maybe yours does too. If so, I have one thing to say to you -- welcome to the club.
At this point, the files are ready to be "merged" into the live filesystem. This is when they are copied from the temporary build directory into <tt>/usr</tt>, etc. At this point, the following functions are executed:


==  www.gentoo.org ==
* <tt>pkg_preinst</tt>
* (files are merged)
* <tt>pkg_postinst</tt>


In our case, our Web site dilemma exists because our project has been growing, and our Web site hasn't. Now that Gentoo Linux is approaching the 1.0 release (when it'll be officially ready for non-developers) and is growing in popularity, we need to start seriously looking at how our Web site can better serve its users. Here's a snapshot of www.gentoo.org:
=== src_* functions ===


<div style="margin: 10px;">[[File:L-redesign-01.gif|frame|class=img-responsive|The current (March 2001) state of affairs at www.gentoo.org]]</div>
Ebuild functions starting with <tt>src_</tt> are all related to creating the ebuild or package from source code/artifacts, and are defined below:


As you can see, we have all the bare essentials -- a description of Gentoo Linux, a features list, a daily Changelog (automatically updated thanks to Python), and a bunch of important links (to the download sites, to our mailing list sign-up pages, and to cvsWeb). We also have links to three documentation resources -- the Gentoo Linux Install Guide and Development Guides, and Christian Zander's NVIDIA Troubleshooting Guide.
==== src_unpack ====


However, while the site seems O.K., we're missing a lot of things. The most obvious is documentation -- our installation and development guides need a lot of work. And then we need to add an FAQ, new links, new user information...the list is endless.
<tt>src_unpack</tt> is intended to be used to unpack the source code/artifacts that will be used by the other <tt>src_*</tt> functions. With EAPI 1 and earlier, it is also used for patching/modifying the source artifacts to prepare them for building, but with EAPI 2 or later the <tt>src_prepare</tt> function should be used for this instead. When <tt>src_unpack</tt> starts, the current working directory is set to <tt>$WORKDIR</tt>, which is the directory within which all source code/artifacts should be expanded. Note that the variable <tt>$A</tt> is set to the names of all the unique source files/artifacts specified in <tt>SRC_URI</tt>, and they will all be available in <tt>$DISTDIR</tt> by the time <tt>src_unpack</tt> starts. Also note that if no <tt>src_unpack</tt> function is specified, <tt>ebuild.sh</tt> will execute the following function for <tt>src_unpack</tt> by default:


== Content vs. display ==
<pre>
src_unpack() {
  unpack ${A}
}
</pre>


And now we come to our second problem. Right now, all of our work is done in raw HTML; I hack away at the index.html file until it looks O.K. Even worse, our Web documentation is written in raw HTML. This isn't a good thing from a development perspective because our raw content (consisting of paragraphs, sections, chapters) is garbled together with a bunch of display-related HTML tags. This, of course, makes it difficult to change both the content and the look of our site. While this approach has worked so far, it is bound to cause problems as our site continues to grow.
==== src_prepare ====


Clearly, we need to be using better technologies behind the scenes. Instead of using HTML directly, we need to start using things like XML, XSLT, and Python. The goal is to automate as much as possible so that we can add and expand our site with ease. If we do our job well, even major future changes to our site should be relatively painless.
EAPI 2 and above support the <tt>src_prepare</tt> function, which is intended to be used for applying patches or making other modifications to the source code. When <tt>src_prepare</tt> starts, the current working directory is set to <tt>$S</tt>.


== A strategy! ==
==== src_configure ====


It was clear that we had a lot of work ahead of us. In fact, there was so much to be done that I didn't know where to begin. Just as I was trying to sort out everything in my head, I came across Laura Wonnacott's "Site Savvy" InfoWorld column (see Resources). In it, she explained the concept of "user-centric" design -- how to improve a Web site while keeping the needs of your target audience (in this case, Gentoo Linux users and developers) in focus. Reading the article and taking a look at the "Handbook of User-Centered Design" link from the article helped me to formulate a strategy -- an action plan -- for the redesign:
EAPI 2 and above support the <tt>src_configure</tt> function, which is used to configure the source code prior to compilation. With EAPI 2 and above, the following default <tt>src_configure</tt> is defined if none is specified:


# First, clearly define the official goal of the Web site -- in writing. What's it there for, and what's it supposed to do?
<pre>
#  Identify the different categories of users who will be using your site -- your target audience. Rank them in order of priority: Which ones are most important to you?
src_configure() {
# Set up a system for getting feedback from your target audience, so they can let you know what you're doing right and wrong.
if [[ -x ${ECONF_SOURCE:-.}/configure ]] ; then
# Evaluate the feedback, and use it to determine what parts of the site need to be improved or redesigned. Tackle high-priority sections first.
econf
# Once you've selected the part of the site to improve, get to work! During your implementation, make sure that the content and design of the new section caters specifically to the needs of your target audience and fixes all known deficiencies.
fi
# When the section redesign is complete, add it to your live site, even if it has a look that's markedly different from your current site. This way, your users can begin benefitting from the newly redesigned section immediately. If there's a problem with the redesign, you'll get user feedback more quickly. Finally, making incremental improvements to your site (rather than revamping the whole site and then rolling it out all at once -- surprise!) will help prevent your users from feeling alienated by your (possibly dramatic) site changes.
}
#  After completing step 6, jump to step 4 and repeat.
</pre>


== The mission statement ==
==== src_compile ====


I was happy to discover that we already had step 3 in place. We had received several e-mail suggestions from visitors to the site, and our developer mailing list also served as a way of exchanging suggestions and comments. However, I had never really completed steps 1 or 2. While the answers may seem obvious, I did find it helpful to actually sit down and write out our mission statement:
This function defines the steps necessary to compile source code. With EAPI 1 and earlier, this function is also used to configure the source code prior to compilation. However, starting with EAPI 2, the <tt>src_configure</tt> function must be used for configuration steps instead of bundling them inside <tt>src_compile</tt>. In addition, starting with EAPI 2, there is now a default <tt>src_compile</tt> function that will be executed if none is defined in the ebuild:


www.gentoo.org exists to assist those who use and develop for Gentoo Linux by providing relevant, up-to-date information about Gentoo Linux and Linux in general, focusing on topics related to Gentoo Linux installation, use, administration, and development. As the central hub for all things Gentoo, the site should also feature important news relevant to Gentoo Linux users and developers. In addition to catering to Gentoo Linux users and developers, www.gentoo.org has the secondary purpose of meeting the needs of potential Gentoo Linux users, providing the information they need to decide whether Gentoo Linux is right for them.
<pre>
src_compile() {
if [ -f Makefile ] || [ -f GNUmakefile ] || [ -f makefile ] ; then
emake || die "emake failed"
fi
}
</pre>


== The target audience ==
==== src_test ====


So far, so good. Now for step 2 -- defining our target audience:
<tt>src_test</tt> is an interesting function - by default, an end-user's Portage does not have tests enabled. But if a user has <tt>test</tt> in <tt>FEATURES</tt>, or <tt>EBUILD_FORCE_TEST</tt> is defined, then <tt>ebuild.sh</tt> will attempt to run a test suite for this ebuild, by executing <tt>make check</tt> or <tt>make test</tt> if these targets are defined in the Makefile; otherwise, no tests will execute. If your Makefile supports <tt>make check</tt> or <tt>make test</tt> but the test suite is broken, then specify <tt>RESTRICT="test"</tt> in your ebuild to disable the test suite.


www.gentoo.org has three target audiences -- Gentoo Linux developers, users, and potential users. While no one group is absolutely a higher priority than another, right now the needs of Gentoo Linux developers are our highest priority, followed by Gentoo Linux users, and then potential users. This is because Gentoo Linux is currently in a prerelease state. When Gentoo Linux reaches version 1.0, Gentoo Linux users and potential users will also become a priority.
==== src_install ====


== Comments and suggestions ==
<tt>src_install</tt> is used by the ebuild writer to install all to-be-installed files to the <tt>$D</tt> directory, which can be treated like an empty root filesystem, in that <tt>${D}/usr</tt> is the equivalent of the <tt>/usr</tt> directory, etc. When <tt>src_install</tt> runs, the Portage sandbox will be enabled, which will prevent any processes from creating or modifying files outside of the <tt>${D}</tt> filesystem tree, and a sandbox violation will occur (resulting in the termination of the ebuild) if any such sandbox violation should occur. Once <tt>src_install</tt> has perfomed all necessary steps to install all to-be-installed files to <tt>$D</tt>, Portage will take care of merging these files to the filesystem specified by the <tt>$ROOT</tt> environment variable, which defaults to <tt>/</tt> if not set. When Portage merges these files, it will also record information about the installed package to <tt>/var/db/pkg/(cat)/$P</tt>. Typically, a <tt>src_install</tt> function such as this is sufficient for ensuring that all to-be-installed files are installed to <tt>$D</tt>:


O.K., now it's time to evaluate the suggestions and comments we've collected:
<pre>
src_install() {
  make DESTDIR="$D" install
}
</pre>


Over the past few months, we've received a number of suggestions from Web site visitors. Overwhelmingly, people are requesting better documentation -- for both developers and users. Several developers have asked if we could create a mailing list that would be devoted exclusively to describing CVS commits.
=== pkg_* functions ===


Interestingly, we've also received a couple of e-mails asking whether Gentoo Linux is a commercial or free product. I'm guessing that because our main logo is inscribed with the name "Gentoo Technologies, Inc." (our legal corporation name), people assume that we have a commercial focus. Modifying our logo so that it reads "Gentoo Linux" and adding small opening paragraph to the main page explaining that we are a free software project should help.
An ebuild's functions starting with <tt>pkg_*</tt> take a wider view of the package lifecycle, and may be executed very early or very late in the build or package installation process. They are also all executed even if installing a Portage binary package, so are the intended place for defining any global configuration changes that are also required during binary package installation, such as user and group creation. When these functions are executed, the <tt>$ROOT</tt> variable will be defined to point to the target root filesystem to which the package is to be (or has been) installed. All logic inside <tt>pkg_*</tt> functions must function properly even if <tt>$ROOT</tt> is something other than <tt>/</tt>.


== The improvement list ==
==== pkg_setup ====


O.K., now let's turn these suggestions into a list of possible improvements:
The <tt>pkg_setup</tt> function is unusual in that it runs prior to any <tt>src_*</tt> function, and also runs prior to any other <tt>pkg_*</tt> function that runs when a binary package is installed, so it provides a useful place for the ebuild writer to perform any sanity checks, global configuration changes to the system (such as user/group creation) or set any internal global variables that are used by the rest of the ebuild. Using this function for defining global variables that are needed in multiple other functions is a useful way of avoiding duplicate code. You should also look to <tt>pkg_setup</tt> as the ideal place to put any logic that would otherwise linger in the main body of the ebuild, which should be avoided at all costs as it will slow down dependency calculation by Portage. Also remember that Portage can build binary packages, and this function is a good place to execute any steps that are required to run both prior to building an ebuild, and prior to installing a package. Also consider using <tt>pkg_preinst</tt> and <tt>pkg_postinst</tt> for this purpose.


* Revamp main page
==== pkg_pretend ====
** Implementation: update logo and add free software blurb
** Goal: to clearly state that we are a free software project
** Target group: potential users
**  Difficulty: medium
* Improve basic user documentation
**  Implementation: new XML/XSLT system, verbose documentation
** Goal: to make it easier for users to install Gentoo Linux
** Target group: new users
** Difficulty: medium
*Improve/create developer documentation
** Implementation: new XML/XSLT system, CVS guide, dev guide, Portage guide
**  Goal: to help our developers to do a great job
** Target group: developers
** Difficulty: hard
*Add a CVS mailing list
** Implementation: use our existing mailman mailing list manager
** Goal: to better inform our developers
** Target group: developers
** Difficulty: easy


== A selection! ==
The <tt>pkg_pretend</tt> function was added with EAPI 3, and it's the opinion of Daniel Robbins that the use of this function should be avoided. This function is especially unusual in that it is intended to be run ''during dependency calculation'', and is intended to provide a polite mechanism to inform the user that a particular ebuild will fail due to a known incompatibility, typically a kernel incompatibility. That way, the user can know during <tt>emerge --pretend</tt> that a merge will fail. While this is useful, extending the dependency engine using <tt>bash</tt> is a very low-performance means to perform these tests. Therefore, The Funtoo core team recommends against using <tt>pkg_pretend</tt>. An extensible dependency engine would be a more appropriate and high-performance way to provide identical functionality.


Two things leap out from the list, for different reasons. The first is the CVS mailing list -- this one is a no-brainer because it's so easy to implement. Often, it makes sense to implement the easiest changes first so that users can benefit from them right away.
==== pkg_preinst ====


The second big thing that leaps out from the list is the need for developer documentation. This is a longer-term project that will require much more work. From my conversations with the other developers, we all appear to be in agreement that some kind of XML/XSL approach is the right solution.
The <tt>pkg_preinst</tt> function is called by Portage, prior to merging the to-be-installed files to the target filesystem specified by <tt>$ROOT</tt> environment variable (which defaults to <tt>/</tt>.) Keep in mind that these to-be-installed files were either just compiled and installed to <tt>$D</tt> by <tt>src_install</tt>, or they were just extracted from a <tt>.tbz2</tt> binary package. The <tt>pkg_preinst</tt> function provides an ideal place to perform any "just before install" actions, such as user and group creation or other necessary steps to ensure that the package merges successfully. It also provides a potential place to perform any sanity checks related to installing the package to the target filesystem. If any sanity checks fail, calling <tt>die</tt> from this function will cause the package to not be installed to the target filesystem.


== The XML/XSL prototype ==
==== pkg_postinst ====


To help start the process, I developed a prototype XML syntax to be used for all our online documentation. By using this XML syntax (called "guide"), our documentation will be clearly organized into paragraphs, sections, and chapters (using XML tags like <section>, <chapter>, etc.) while remaining free of any display-related tags. To create the HTML for display on our site, I created a prototype set of XSL transforms. By using an XSLT processor such as Sablotron, our guide XML files can be converted into HTML as follows:
The <tt>pkg_postinst</tt> function is called by Portage prior to the package being installed to the target filesystem specified by <tt>$ROOT</tt>. This is a good place to perform any post-install configuration actions as well as print any informational messages for the user's benefit related to the package that was just installed.


devguide.xml + guide.xsl ---XSLT processor---> devguide.html
==== pkg_prerm ====


The great thing about this XML/XSLT approach is that it separates our raw content (XML) from the display-related information contained in the guide.xsl (XSLT) file. If we ever need to update the look of our Web pages, we simply modify the guide.xsl file and run all our XML through the XSLT processor (Sablotron), creating updated HTML pages. Or, if we need to add a few chapters to the development guide, we can modify devguide.xml. Once we're done, we then run the XML through Sablotron, which then spits out a fully-formatted devguide.html file with several added chapters. Think of XML as the content and XSLT as the display-related formatting macros.
The <tt>pkg_prerm</tt> function is called by Portage before an ebuild is removed from the filesystem.


While our entire team is convinced that XML/XSLT is the way to go, we haven't yet agreed upon an official XML syntax. Achim, our development lead, suggested that we use docbook instead of rolling our own XML syntax. However, the prototype guide XML format has helped to start the decision-making process. Because we developers are going to be the ones using the XML/XSL on a daily basis, it's important to choose a solution that we're comfortable with and meets all of our needs. By my next article, I should have a working XML/XSL doc system to show off to you.
==== pkg_postrm ====


== Technology demo: pytext ==
The <tt>pkg_postrm</tt> function is called by Portage after an ebuild is removed from the filesystem.


For the most part, our current Web site isn't using any new or super-cool technologies that are worth mentioning. However, there's one notable exception -- our tiny pytext embedded Python interpreter.
==== pkg_config ====


Like many of you, I'm a huge Python fan and much prefer it over other scripting languages, so when it came time to add some dynamic content to our Web site, I naturally wanted to use Python. And, as you probably know, when coding dynamic HTML content, it's usually much more convenient to embed the language commands inside the HTML, rather than the other way around. Thus, the need for an embedded Python interpreter that can take a document like this:
The <tt>pkg_config</tt> function is called by Portage when the user calls <tt>emerge --config</tt> for the ebuild. The current directory will be set to the current directory of the shell from where <tt>emerge --config</tt> is run.
=== Skipping over a function ===
To skip over a function, create a function that does not do anything. The recommended way is to use bash no-op command:
<syntaxhighlight lang="bash">
# Skip src_prepare.
src_prepare() { :; }
</syntaxhighlight>


<pre>
=== Extra pre_ and post_ functions ===
<p>
Yeah, sure; I got some questions:<br>
<!--code
names=["bob","jimmy","ralph"]
items=["socks","lunch","accordion"]
for x in items:
for y in names:
print "Anyone seen",y+"'s",x+"?<br>"
-->
See, told you so.
</pre>


....and transform it into this:
Modern versions of Portage also support functions identical to the above functions but with '''pre_''' and '''post_''' at the beginning of the function name. For example, <tt>post_src_configure</tt> will be executed after <tt>src_configure</tt> and before <tt>src_compile</tt>. These additional functions are supported by all EAPIs, provided that the parent function is supported by the EAPI in use. The initial current working directory should be identical to the initial current working directory of the parent function.


<pre>
=== Helper Functions ===
<p>
Yeah, sure; I got some questions:<br>
Anyone seen bob's socks?<br>
Anyone seen jimmy's socks?<br>
Anyone seen ralph's socks?<br>
Anyone seen bob's lunch?<br>
Anyone seen jimmy's lunch?<br>
Anyone seen ralph's lunch?<br>
Anyone seen bob's accordion?<br>
Anyone seen jimmy's accordion?<br>
Anyone seen ralph's accordion?<br>
See, told you so.
</pre>


Here's the source code for pytext:
==== econf() ====


{{file|name=pytext|lang=python|desc=The pytext embedded Python interpreter|body=
econf() is part of ebuild.sh and is intended to be a wrapper to the <tt>configure</tt> command that is typically used in the <tt>src_configure()</tt> stage. It has a number of behaviors that are important for ebuild writers to understand. Once you understand what <tt>econf()</tt> does, you are free to use it in your ebuilds. Note that the behavior of <tt>econf()</tt> is generally safe for most autoconf-based source archives, but in some cases it may be necessary to avoid using <tt>econf()</tt> to avoid some of its default behaviors.
#!/usr/bin/env python2


# pytext 2.1
===== Automatically set prefix =====
# Copyright 1999-2001 Daniel Robbins
# Distributed under the GPL


import sys
<tt>--prefix=/usr</tt> will be passed to <tt>configure</tt> automatically, unless a <tt>--prefix</tt> argument was specified to <tt>econf()</tt>, in which case, that <tt>--prefix</tt> setting will be used instead.


def runfile(myarg):
===== Automatically set libdir =====
  "interprets a text file with embedded elements"
  mylocals={}
  try:
      a=open(myarg,'r')
  except IOError:
      sys.stderr.write("!!! Error opening "+myarg+"!\n")
      return
  mylines=a.readlines()
  a.close()
  pos=0
  while pos<len(mylines):
      if mylines[pos][0:8]=="<!--code":
  mycode=""
  pos=pos+1
  while (pos<len(mylines)) and (mylines[pos][0:3]!="-->"):
      mycode=mycode+mylines[pos]
      pos=pos+1
  exec(mycode,globals(),mylocals)
      else:
  sys.stdout.write(mylines[pos])
      pos=pos+1


if len(sys.argv)>1:
If the <tt>ABI</tt> variable is set (typically done in the profile), then <tt>econf()</tt> will look for a variable named <tt>LIBDIR_$ABI</tt> (ie. <tt>LIBDIR_amd64</tt>). If this variable is set, the value of this variable will be used to set <tt>libdir</tt> to the value of <tt>{prefix}/LIBDIR_$ABI</tt>.
  for x in sys.argv[1:]:
      runfile(x)
  sys.exit(0)
else:
  sys.stderr.write
    ("pytext 2.1 -- Copyright 1999-2001 Daniel Robbins. ")
  sys.stderr.write
    ("Distributed under the\nGNU Public License\n\n")
  sys.stderr.write
    ("Usage: "+sys.argv[0]+" file0 [file1]...\n")
  sys.exit(1)
}}


== How pytext works ==
===== Automatically set CHOST and CTARGET =====


Here's how it works. It scans each input line, and most of the time, each input line is simply echoed to stdout. However, if pytext encounters a line beginning with <!--code, then the contents of every line up to the first line beginning with --> are appended to a string called mycode. Pytext then executes the mycode string using the built-in exec() function, effectively creating an embedded Python interpreter.
The <tt>--host=$CHOST</tt> argument will be passed to <tt>configure</tt>. <tt>$CHOST</tt> is defined in the system profile. In addition, the <tt>--target=$CTARGET</tt> argument will be passed to <tt>configure</tt> if <tt>$CTARGET</tt> is defined. This is not normally required but is done to make Portage more capable of cross-compiling the ebuild. However, this functionality is not a guarantee that your ebuild will successfully cross-compile, as other changes to the ebuild may be necessary.


There's something really beautiful about this particular implementation -- we call exec() in such a way that all modifications to the global and local namespaces are saved. This makes it possible to import a module or define a variable in one embedded block, and then access this previously-created object in a later block, as this example clearly demonstrates:
===== Disable Dependency Tracking (EAPI 4) =====


<pre>
In EAPI 4, the <tt>--disable-dependency-tracking</tt> argument will be passed to <tt>configure</tt> in order to optimize the performance of the configuration process. This option should have no impact other than on the performance of the <tt>configure</tt> script.
<!--code
import os
foo=23
-->


Hello
===== List of arguments =====
 
<!--code
print foo
if os.path.exists("/tmp/mytmpfile"):
print "it exists"
else:
print "I don't see it"
-->
</pre>


Handy, eh? pytext serves is an excellent demonstration of the power of Python, and is an extremely useful tool for Python fans. For our current site, we call pytext from a cron job, using it to periodically generate the HTML code for our main page Changelog:
The following arguments are passed to <tt>configure</tt> and are all overrideable by the user by passing similar options to <tt>econf()</tt>:


<console>
* <tt>--prefix=/usr</tt>
$ ##i##pytext index.ehtml > index.html
* <tt>--libdir={prefix}/LIBDIR_$ABI</tt>
</console>
* <tt>--host=${CHOST}</tt>
* if CTARGET is defined, then <tt>--target=${CTARGET}</tt>
* <tt>--mandir=/usr/share/man</tt>
* <tt>--infodir=/usr/share/info</tt>
* <tt>--datadir=/usr/share</tt>
* <tt>--sysconfdir=/etc</tt>
* <tt>--localstatedir=/var/lib</tt>
* if EAPI 4+, then <tt>--disable-dependency-tracking</tt>


That's it for now; I'll see you next time when we'll take a look at the first stage of the www.gentoo.org redesign!
[[Category:Internals]]
{{ArticleFooter}}
[[Category:Portage]]
[[Category:Official Documentation]]

Revision as of 10:19, February 7, 2015

Ebuild Functions

Ebuilds provide the ability to define various shell functions that are used to specify various actions relating to building and installing a source or binary package on a user's system. When an ebuild is emerged, the following functions are called, in order:

  • pkg_setup - variable intialization and sanity checks
  • src_unpack
  • src_prepare
  • src_configure
  • src_compile
  • src_install

At this point, the files are ready to be "merged" into the live filesystem. This is when they are copied from the temporary build directory into /usr, etc. At this point, the following functions are executed:

  • pkg_preinst
  • (files are merged)
  • pkg_postinst

src_* functions

Ebuild functions starting with src_ are all related to creating the ebuild or package from source code/artifacts, and are defined below:

src_unpack

src_unpack is intended to be used to unpack the source code/artifacts that will be used by the other src_* functions. With EAPI 1 and earlier, it is also used for patching/modifying the source artifacts to prepare them for building, but with EAPI 2 or later the src_prepare function should be used for this instead. When src_unpack starts, the current working directory is set to $WORKDIR, which is the directory within which all source code/artifacts should be expanded. Note that the variable $A is set to the names of all the unique source files/artifacts specified in SRC_URI, and they will all be available in $DISTDIR by the time src_unpack starts. Also note that if no src_unpack function is specified, ebuild.sh will execute the following function for src_unpack by default:

src_unpack() {
  unpack ${A}
}

src_prepare

EAPI 2 and above support the src_prepare function, which is intended to be used for applying patches or making other modifications to the source code. When src_prepare starts, the current working directory is set to $S.

src_configure

EAPI 2 and above support the src_configure function, which is used to configure the source code prior to compilation. With EAPI 2 and above, the following default src_configure is defined if none is specified:

src_configure() {
	if [[ -x ${ECONF_SOURCE:-.}/configure ]] ; then
		econf
	fi
}

src_compile

This function defines the steps necessary to compile source code. With EAPI 1 and earlier, this function is also used to configure the source code prior to compilation. However, starting with EAPI 2, the src_configure function must be used for configuration steps instead of bundling them inside src_compile. In addition, starting with EAPI 2, there is now a default src_compile function that will be executed if none is defined in the ebuild:

src_compile() {
	if [ -f Makefile ] || [ -f GNUmakefile ] || [ -f makefile ] ; then
		emake || die "emake failed"
	fi
}

src_test

src_test is an interesting function - by default, an end-user's Portage does not have tests enabled. But if a user has test in FEATURES, or EBUILD_FORCE_TEST is defined, then ebuild.sh will attempt to run a test suite for this ebuild, by executing make check or make test if these targets are defined in the Makefile; otherwise, no tests will execute. If your Makefile supports make check or make test but the test suite is broken, then specify RESTRICT="test" in your ebuild to disable the test suite.

src_install

src_install is used by the ebuild writer to install all to-be-installed files to the $D directory, which can be treated like an empty root filesystem, in that ${D}/usr is the equivalent of the /usr directory, etc. When src_install runs, the Portage sandbox will be enabled, which will prevent any processes from creating or modifying files outside of the ${D} filesystem tree, and a sandbox violation will occur (resulting in the termination of the ebuild) if any such sandbox violation should occur. Once src_install has perfomed all necessary steps to install all to-be-installed files to $D, Portage will take care of merging these files to the filesystem specified by the $ROOT environment variable, which defaults to / if not set. When Portage merges these files, it will also record information about the installed package to /var/db/pkg/(cat)/$P. Typically, a src_install function such as this is sufficient for ensuring that all to-be-installed files are installed to $D:

src_install() {
  make DESTDIR="$D" install
}

pkg_* functions

An ebuild's functions starting with pkg_* take a wider view of the package lifecycle, and may be executed very early or very late in the build or package installation process. They are also all executed even if installing a Portage binary package, so are the intended place for defining any global configuration changes that are also required during binary package installation, such as user and group creation. When these functions are executed, the $ROOT variable will be defined to point to the target root filesystem to which the package is to be (or has been) installed. All logic inside pkg_* functions must function properly even if $ROOT is something other than /.

pkg_setup

The pkg_setup function is unusual in that it runs prior to any src_* function, and also runs prior to any other pkg_* function that runs when a binary package is installed, so it provides a useful place for the ebuild writer to perform any sanity checks, global configuration changes to the system (such as user/group creation) or set any internal global variables that are used by the rest of the ebuild. Using this function for defining global variables that are needed in multiple other functions is a useful way of avoiding duplicate code. You should also look to pkg_setup as the ideal place to put any logic that would otherwise linger in the main body of the ebuild, which should be avoided at all costs as it will slow down dependency calculation by Portage. Also remember that Portage can build binary packages, and this function is a good place to execute any steps that are required to run both prior to building an ebuild, and prior to installing a package. Also consider using pkg_preinst and pkg_postinst for this purpose.

pkg_pretend

The pkg_pretend function was added with EAPI 3, and it's the opinion of Daniel Robbins that the use of this function should be avoided. This function is especially unusual in that it is intended to be run during dependency calculation, and is intended to provide a polite mechanism to inform the user that a particular ebuild will fail due to a known incompatibility, typically a kernel incompatibility. That way, the user can know during emerge --pretend that a merge will fail. While this is useful, extending the dependency engine using bash is a very low-performance means to perform these tests. Therefore, The Funtoo core team recommends against using pkg_pretend. An extensible dependency engine would be a more appropriate and high-performance way to provide identical functionality.

pkg_preinst

The pkg_preinst function is called by Portage, prior to merging the to-be-installed files to the target filesystem specified by $ROOT environment variable (which defaults to /.) Keep in mind that these to-be-installed files were either just compiled and installed to $D by src_install, or they were just extracted from a .tbz2 binary package. The pkg_preinst function provides an ideal place to perform any "just before install" actions, such as user and group creation or other necessary steps to ensure that the package merges successfully. It also provides a potential place to perform any sanity checks related to installing the package to the target filesystem. If any sanity checks fail, calling die from this function will cause the package to not be installed to the target filesystem.

pkg_postinst

The pkg_postinst function is called by Portage prior to the package being installed to the target filesystem specified by $ROOT. This is a good place to perform any post-install configuration actions as well as print any informational messages for the user's benefit related to the package that was just installed.

pkg_prerm

The pkg_prerm function is called by Portage before an ebuild is removed from the filesystem.

pkg_postrm

The pkg_postrm function is called by Portage after an ebuild is removed from the filesystem.

pkg_config

The pkg_config function is called by Portage when the user calls emerge --config for the ebuild. The current directory will be set to the current directory of the shell from where emerge --config is run.

Skipping over a function

To skip over a function, create a function that does not do anything. The recommended way is to use bash no-op command:

# Skip src_prepare.
src_prepare() { :; }

Extra pre_ and post_ functions

Modern versions of Portage also support functions identical to the above functions but with pre_ and post_ at the beginning of the function name. For example, post_src_configure will be executed after src_configure and before src_compile. These additional functions are supported by all EAPIs, provided that the parent function is supported by the EAPI in use. The initial current working directory should be identical to the initial current working directory of the parent function.

Helper Functions

econf()

econf() is part of ebuild.sh and is intended to be a wrapper to the configure command that is typically used in the src_configure() stage. It has a number of behaviors that are important for ebuild writers to understand. Once you understand what econf() does, you are free to use it in your ebuilds. Note that the behavior of econf() is generally safe for most autoconf-based source archives, but in some cases it may be necessary to avoid using econf() to avoid some of its default behaviors.

Automatically set prefix

--prefix=/usr will be passed to configure automatically, unless a --prefix argument was specified to econf(), in which case, that --prefix setting will be used instead.

Automatically set libdir

If the ABI variable is set (typically done in the profile), then econf() will look for a variable named LIBDIR_$ABI (ie. LIBDIR_amd64). If this variable is set, the value of this variable will be used to set libdir to the value of {prefix}/LIBDIR_$ABI.

Automatically set CHOST and CTARGET

The --host=$CHOST argument will be passed to configure. $CHOST is defined in the system profile. In addition, the --target=$CTARGET argument will be passed to configure if $CTARGET is defined. This is not normally required but is done to make Portage more capable of cross-compiling the ebuild. However, this functionality is not a guarantee that your ebuild will successfully cross-compile, as other changes to the ebuild may be necessary.

Disable Dependency Tracking (EAPI 4)

In EAPI 4, the --disable-dependency-tracking argument will be passed to configure in order to optimize the performance of the configuration process. This option should have no impact other than on the performance of the configure script.

List of arguments

The following arguments are passed to configure and are all overrideable by the user by passing similar options to econf():

  • --prefix=/usr
  • --libdir={prefix}/LIBDIR_$ABI
  • --host=${CHOST}
  • if CTARGET is defined, then --target=${CTARGET}
  • --mandir=/usr/share/man
  • --infodir=/usr/share/info
  • --datadir=/usr/share
  • --sysconfdir=/etc
  • --localstatedir=/var/lib
  • if EAPI 4+, then --disable-dependency-tracking