Metro 1.2 Released

Metro 1.2, my tool for building Gentoo and Funtoo releases and custom builds, has been released. Learn about Metro here. Technically, this is a maintenance release, but looking at the ChangeLog tells me that this is more like a major release.

I’ve added a metro-1.2.ebuild to the Funtoo Portage tree.

Here are the changes in 1.2:

• NEW target: "stage3-freshen" uses emerge -up world to update stage
• NEW target: "stage3-quick" builds stage3->stage3, skipping stage1 and 2.
• NEW target: "git-snapshot" creates a full .tar.bz2 snapshot of git repo with the .git directory intact, so it is a full working git repo when unpacked.
• NEW target: "emerge" for adding packages to an existing stage3
• NEW subarch: ~core2 for Intel Core 2 series processors. Supported with gcc 4.3.2 which is in the funtoo build.
• Updated targets: "openvz" updated to support OpenRC 0.4.0.
• stage3-freshen supports emerge/packages/force target, which is a list of packages to rebuild using "emerge <packages>". This is a great way to force a rebuild of packages that may need to be rebuilt but did not get a rev-bump (ie. openrc.:)
• Metro now properly cleans /var/tmp/metro after it is done, unless the build aborted due to an error.
• scripts/build.sh: now supports stage3-freshen and stage3-quick by allowing an extra "quick" or "freshen" option as the last argument.
• scripts/build.sh: now creates a "current" symlink pointing to the most recent stage3 built. This "current" symlink exists in the subarch directory and the filename is "stage3-subarch-current.tar.bz2."
• scripts/build.sh: now uses "git-snapshot" for funtoo builds, so funtoo snapshots are now double the size but are a full working and "git pull"-able git repo, simplifying funtoo install process.
• stage3 targets: try to ensure that /etc/make.conf.example exists. Create a symlink from /etc/make.conf.example to /usr/share/portage/config/make.conf.example.
• metro core: robust mount point detection using /proc/mounts instead of output from the mount command, which can be unreliable.
• metro core: more robust umount algorithm.
• metro core: local name support, so your custom stages don't need to be called "funtoo" anymore. Also quite useful with the "emerge" target for creating custom stage3's with different names.
• metro core: ccache works correctly, but has been disabled by default as it may cause build failures when used with Portage parallel merges (--jobs) and we now have "stage3-freshen" and "stage3-quick" when we need a stage3 built quickly.
• parser syntax: For conditionals, the "in" keyword is now supported as an equivalent for "is".
• parser syntax: New "lax" expansion method, ie: $[foo:lax] will expand to foo if it is defined, otherwise it will not throw a parser error (as it normally would) and instead expand to a dummy value which may contain spaces.
• Documentation: github wiki documentation updated to reflect new features and QuickStart Guide refined to be easier to understand.
• Projects: new projects/ directory contains source code for funtoo.org Web site. Funtoo.org Web site redesigned to be more of a portal for new Funtoo builds. The funtoo Web site can be generated by running "pytext index.html" (sent to stdout.) Embedded python code automatically updates Web site symlinks to point to most recent versions of builds and list relevant build-related information. Support for displaying build dates in human-parseable form (ie. "Yesterday" or "2 Days Ago".)

What I’ve Been Up To – New site, etc.

Here’s what I’ve been up to:

http://www.funtoo.org has been redesigned. It now has more of a portal design to get you to the latest Funtoo, Funtoo+OpenVZ and Gentoo builds. I hope you like it :)

In the Funtoo Portage tree, I’ve added a new build of OpenRC, a new udev ebuild (133 with some OpenRC-compatibility and other fixes), and I’m keeping Portage 2.2_rc* unmasked. The Gentoo Portage tree has masked 2.2_rc* to try to get more testing of the upcoming 2.1* release, which is fine, but we’re going to stick with the development branch.

Work is ongoing with Metro. The git version (pre-1.2) now has a few new targets: stage3-quick and stage3-freshen. In the past, Metro would build a new stage3 by going through these steps:

seed stage3->stage1->stage2->stage3

“stage3-quick” builds a new stage3 as follows:

seed stage3->stage3

It uses ROOT=/tmp/stage3root and emerge system to make this happen.

“stage3-freshen” is even faster, and builds a new stage3 as follows:

seed stage3->stage3

The difference here is that the “stage3-freshen” target uses the seed stage3 in-place and runs an emerge –u –deep system and emerge –u –deep <extrapackages>. This is a nice way to freshen slightly old stage3’s without doing a full rebuild.

So there you have it. I hope you enjoy the new site. I sure do, since it updates itself automatically, and it’s fun for me to look at every morning :)

Metro 1.1 Custom Builds

Hey, I’ve just updated the Metro wiki with info on how to use Metro to perform Custom Builds. Armed with this info, you will be able to create your own customized versions of Gentoo. You can use everything from a customized Portage tree, to customized global and local USE variables and keywords, and custom package lists.

Dive in and let me know how Metro custom builds work for you! Post your feedback in the blog comments below.

Note: If you already have Metro installed, just run “git pull” in the /root/git/metro directory to automatically download all the latest Metro updates.

UPDATE: I've also updated the QuickStart Guide so you can see how I use fcron and build.sh to automate my daily builds. Armed with this info, you will be able to duplicate my daily build routine on your local workstation!

ANOTHER UPDATE: If you are using the Funtoo Portage tree, you will now find a dev-util/metro-1.1.ebuild that you can emerge to install Metro :)

Metro 1.1 Released

I’ve just released version 1.1 of Metro and updated the QuickStart Guide to reflect this new version. Currently, the only way to install Metro is to use git – see the improved QuickStart Guide for more information. Here’s a list of all the improvements to Metro:

• Reorganized /etc/metro directory
• NEW: "Builds" defined in /etc/metro/builds/ - all key build information now in a single file for convenience.
• NEW: "FSLayouts" defined in /etc/metro/fslayouts/ - all mirror-related file names and path structures now in a single file.
• NEW: OpenVZ template support for Funtoo stages, enabled in  scripts/build.sh
• NEW: Reorganized file paths inside /usr/lib/metro.
• NEW: Working gentoo/emerge target, when you just want to take a stage3 and create something new by emerging stuff in it.
• NEW: Targets now have an OS prefix. What was target: snapshot is now target: gentoo/snapshot, etc. The path structure in /usr/lib/metro has also been improved so that people can easily create a /usr/lib/metro/targets/myos/ directory and start hacking.
• NEW: Several parser bug fixes
• FIX: Improvements to metro have allowed scripts/build.sh to be simplified.
• NEW: Improved error and exception handling
• NEW: Enhanced Portage git snapshot support - Metro will now keep a local cache of a git repository so it doesn't need to re-clone it every time, saving bandwidth. This cache location is defined in the path/cache/git variable in /etc/metro/metro.conf.
• Embedded Python interpreter can no longer reference Metro metadata directly. This prevents weird side-effects from occurring.
• NEW: Conditional expansion - a variable such as $[foo?] will get expanded to "yes" if defined and "no" if not defined, providing easy integration with scripts so you don't throw a parser exception by referencing an undefined variable.
  * NEW: "zap" expansion - in a multi-line element, a variable such as $[foo:zap] will get expanded to the name of the variable if it is defined. If it is not defined, however, then the entire line will be deleted from the output. This allows very compact conditional lines in templates like this: CFLAGS=$[portage/CFLAGS:zap]
• FIX: MAKEOPTS now properly applied to Gentoo builds, speeding up builds on multi-core systems significantly.
• NEW: [options parse/lax] and [options parse/strict] to disable and enable strict parsing, respectively. When lax parsing is enabled, any referenced undefined variables in multi-line elements will get dummy values, but an exception will not be thrown.
• NEW: conditonal sections - it's now possible to write:
  [section foo when bar]
    This is equivalent to the old:
  [section foo]
  [when bar]
• NEW: conditional equality - it's now possible for all conditionals to test for equality, such as:
  [section foo when bar is oni]
  or
  [when bar is oni]
• NEW: conditional list-match - it's now possible to see if an element matches a list of values, such as:
  [section foo when target/subarch is x86 amd64 pentium4] This section will be active if target/subarch has a value that matches one of the specified values.
• CHANGE: multi-line elements that are referenced inside other multi-line elements, using the syntax $[[foo]], will only get expanded if they appear at the beginning of the line. This allows them to easily be commented out by prefixing them by a "#" or any other character. In this case, the $[[foo]] will not be expanded and will be passed to the output as-is.
• CHANGE: path/mirror/srcstage and path/mirror/deststage have been renamed to /path/mirror/source and path/mirror/target respectively.

Enjoy!

The Goal of Funtoo

A lot of people have asked me recently “What is the goal of Funtoo? What are you trying to do?”

For those who are trying to understand the inspiration behind Funtoo, this video should make it clear:

OK, OK, here’s an explanation of what Funtoo is about. We are focusing on providing the infrastructure Gentoo needs to move to a more distributed, collaborative model. This includes a git-based Portage tree as well as the Metro build engine.

We are not a “fork” of Gentoo in the traditional sense of the word, though of course we are trying to do things better (otherwise, what would be the point of doing anything?) We are trying to aim to be as compatible with Gentoo as possible, so that it is easy for us to share technology with Gentoo.

Here’s something else to wrap your head around. Back in the 90’s, “fork” was a bad word. But these days (with distributed SCM tools like git) a “fork” is something that happens every day. Forking is no longer a political thing, but something a developer does whenever appropriate. If you need to do something new or interesting, or fix some bugs, you should be able to easily fork Funtoo. Then, with your own private copy of Funtoo, you can pleasantly do your cool thing, test it, and then share your work. If you did something useful, others will “merge” your work into their efforts. We encourage others to fork our projects.

Funtoo’s mission is to provide the infrastructure to allow collaboration to happen, so that people can innovate in a distributed way, and more people can get involved.