Revision as of 04:24, 28 November 2010

The Metro Data Model has been designed to provide you with an optimal way to organize build data.

Here are the primary goals for the data model:

1. Provide useful ways to organize data
2. Use mechanisms and syntax that maximize maintainability of the data over time
3. Reduce and (ideally) eliminate side-effects at every opportunity

To attain these goals, I've used a functional data model, where an element (variable) can be defined only once, and cannot be redefined.

By default, the Metro parser operates in "strict" mode, which means that it will throw an error if a variable has been referenced that has not been defined. This "strict" mode is actually very useful in catching errors that might otherwise go unnoticed and result in broken builds.

In addition, the Metro parser was designed so that the order in which data elements are defined is not important, even if they reference one another. This was done to eliminate side-effects related to data ordering, where changing the order in which things are defined in a file can change the behavior of or break your code.

Versions of Metro prior to 1.4 contained limited support for conditional logic. After some experimentation, I've decided that the conditional support is not necessary, and it is not used by Metro 1.4. However, support for conditionals still exist in the parser, but will be removed when the parser is rewritten.

First Look

Here is some sample Metro data:

path: /usr/bin

Above, we have defined the element path to have the value /usr/bin. path is a single-line element, and the Metro parser takes care of trimming any trailing whitespace that may be on the line. You can also define single-line elements that have values that consist of multiple whitespace-separated values:

options: ccache replace

Sometimes, you need to define an element but leave it blank. To do this, don't specify any values after the colon:

options:

In Metro, the / character is used to delineate various classes of elements, as follows:

path/mirror: /home/mirror/linux
path/mirror/snapshot: /home/mirror/linux/snapshots
path/metro: /usr/lib/metro

Above, we see the proper Metro convention for specifying paths. Each path has a prefix of path/. We have a path/mirror element but also have a path/mirror/snapshot element. The / is used to organize our data into logical groups. This is not enforced by Metro but is presented here as a best practice.

The data above could also be represented using a section annotation, as follows:

[section path]

mirror: /home/mirror/linux
mirror/snapshot: /home/mirror/linux/snapshots
metro: /usr/lib/metro

Above, the [section path] line is a section annotation, and it tells the Metro parser that the path/ prefix should be applied to all following data elements. A section annotation is in effect until another section annotation is encountered by the parser.

While our data above is getting more organized, there is some redundancy in our data, which generally isn't a good thing. Here's an example of how to make our data a bit more compact:

[section path]

mirror: /home/mirror/linux
mirror/snapshot: $[path/mirror]/snapshots
metro: /usr/lib/metro

Above, we have used an element reference of $[path/mirror] to reference our path/mirror element. What this means is that path/snapshot will have a value of /home/mirror/linux/snapshots.

Also, it's worth pointing out that we could just have well written:

[section path]

mirror/snapshot: $[path/mirror]/snapshots
mirror: /home/mirror/linux
metro: /usr/lib/metro

In other words, it's perfectly OK to use the element reference of $[path/mirror] on a line before the actual definition of path/mirror. Metro doesn't care about the order in which data is defined.

Metro provides another way to organize your data in an efficient way. Supposing that you had a lot of path/mirror-related data, then it might be useful to organize your data as follows:

[section path]

metro: /usr/lib/metro

[section path/mirror]

: /home/mirror/linux
snapshot: $[]/snapshot
source: $[]/$[source/subarch]/funtoo-$[source/subarch]-$[source/version]/$[source/name].tar.bz2

Above, we have used two new parser features. Inside [section path/mirror], we can define the path/mirror element itself by using a blank element name, followed by a :. The next parser feature we see above is that we can use $[] to reference the value of the path/mirror value. $[] will always reference the value of the element specified in the section annotation. Also note that as of Metro 1.1, $[:] can be used as an alternate form of $[]. In addition, as of Metro 1.2.4, $[:foo] can be used as an alternate form of $[section-name/foo].

Collect Annotations

Many scripting languages have the notion of an "include" file, or "importing" additional data from a remote file. Metro has this concept as well, but it is implemented in a somewhat different way. You can tell Metro to include data from another file by using a collect annotation.

A collect annotation looks like this:

[collect $[path/metro]/myfile.txt]

Now, we called these things "collect annotations" for a reason - in Metro, they work slightly differently than most languages implement include and import. The main difference is that in Metro, a collect annotation does not happen right away. Instead, Metro will add the file to be collected (in this case, that would be the file /usr/lib/metro/myfile.txt, or whatever $[path/metro]/myfile.txt evaluates to) to a collection queue. This means that Metro will read in the contents of the file at some point in time, and the data in the file will be available to you by the time the parsing is complete. But because Metro doesn't care about the order in which data is defined, it doesn't have the same concept of "read in the data - right now!" that an include or import statement does in other languages.

Conditional Collect Annotations

Metro no longer officially supports conditional collect annotations; however, simple collect annotations can be used to make conditional decisions in Metro, as follows:

[collect ./snapshots/$[snapshot/type]]

Above, Metro will collect from a file based on the value of the $[snapshot/type] element. This allows for varying definitions of elements to exist dependent on the value of $[snapshot/type].

Above, Metro will raise an exception if $[snapshot/type] is undefined or has a value that does not map to a file on disk. If it is possible that $[snapshot/type] may not be defined, use the following format:

[collect ./snapshots/$[snapshot/type:zap]]

Using the :zap modifier, the entire collect argument will be replaced with the empty string if $[snapshot/type] is undefined. If Metro is asked to collect an empty string, it will not throw an exception. So this is a handy way to conditionally disable collection of a file. But please note that for all non-null values of $[snapshot/type], a corresponding file must exist on disk in ./snapshots/ or Metro will throw an exception. :zap is explained in more detail in the "Special Variable Expansion" section, below.

Multi-line elements

Metro supports multi-line elements and they are the foundation of Metro's template engine. A multi-line element can be defined as follows, by using square brackets to delimit multi-line data:

myscript: [
#!/bin/bash
echo $*
]

The terminating closing square bracket should be on a line all by itself.

One of the very useful things about multi-line elements is that they support Metro element references:

myscript: [
#!/bin/bash
echo Metro's path/metro setting is $[path/metro].
]

In the above multi-line element, the $[path/metro] reference will be expanded to contain the appropriate value of the element. It is possible to expand single-line elements inside multi-line elements simply by referencing them using a dollar sign and square brackets.

Metro also allows you to expand multi-line elements inside other multi-line elements. Here's an example of how that works:

myscript: [
#!/bin/bash
$[[steps/setup]]
echo Hi There :)
]

Commenting out Multi-Line Element References

If you want to disable a multi-line reference, simply prefix it with some text, such as #:

myscript: [
#!/bin/bash
# disabling this for now: $[[steps/setup]]
echo Hi There :)
]

This comes in handy when you are commenting out a portion of code. If the multi-line reference has any non-whitespace text on the same line, then it will not be expanded and will be passed to the output as literal text.

In contrast, single line element references will always be expanded wherever they are found in the multi-line element, even if they are "commented out" in the script.

Multi-Line Element as Single-Line References

In Metro 1.3.1, it is possible to reference a multi-line element using the standard "$[ ]" reference. When this is done, the multi-line element is transformed into a single-line element by concatenating lines together, separated by whitespace. This allows multi-line elements to be used to define single-line data, as follows:

packages: [
    sys-apps/portage
    sys-devel/gcc
    dev-util/git
]

myscript: [
    emerge -C $[packages]
]

In the above example, the $[packages] reference will expand to the single-line value sys-apps/portage sys-devel/gcc dev-util/git.

Special Variable Expansion

As of Metro 1.1, Metro's template engine has a few new features related to variable expansion. The first is the ability to expand a single-line variable using a ?, as follows:

foo: [
#!/bin/bash
if [ "$[path/mirror?]" = "yes" ]
then
    echo "path/mirror is defined"
fi
]

With a trailing ?, a single-line variable will evaluate to a literal yes if it is defined, and a literal no if it is not defined.

Another special variable expansion uses a trailing :zap suffix for any single-line variable, and is used as follows:

foo: [
CFLAGS=$[portage/CFLAGS:zap]
]

Above, if portage/CFLAGS is defined, then the variable will expand to the value of portage/CFLAGS. However, if portage/CFLAGS is not defined, then the entire line will be omitted from the output.

Controlling Parser Behavior

Normally, the following code would cause the parser to throw an exception if path/mirror were undefined:

foo: [
#!/bin/bash
if [ "$[path/mirror?]" = "yes" ]
then    echo "$[path/mirror]"fi
]

The first variable expansion, $[path/mirror?], would not throw an exception if path/mirror was not defined - instead, the variable would expand to n o. However, on the second expansion, $[path/mirror], Metro will abort with an error because the variable has not been defined. In the above code, this is pr obably not what you want, so we can temporarily put Metro's parser into "lax" mode as follows:

[option parse/lax]

foo: [
#!/bin/bash
if [ "$[path/mirror?]" = "yes" ]
then
    echo "$[path/mirror]"
fi]
[option parse/strict]

The [option parse/lax] and [option parse/strict] annotations will temporarily enable "lax" parsing mode when expanding the variables defined i nside foo. When Metro is in "lax" mode and a single-line variable is not defined, it is replaced with a dummy string, which will look something like a literal [BLANK var=varname].