Funtoo:Metro/Data Model

This page documents the design of the Metro data model -- this is the data format used in the targets/ directory inside Metro and is primarily of interest to those who will be extending the capabilities of Metro.

Goals

The Metro Data Model has been designed to provide you with an optimal way to organize build data.

Here are the primary goals for the data model:

1. Provide useful ways to organize data
2. Use mechanisms and syntax that maximize maintainability of the data over time
3. Reduce and (ideally) eliminate side-effects at every opportunity

To attain these goals, I've used a functional data model, where an element (variable) can be defined only once, and cannot be redefined.

By default, the Metro parser operates in "strict" mode, which means that it will throw an error if a variable has been referenced that has not been defined. This "strict" mode is actually very useful in catching errors that might otherwise go unnoticed and result in broken builds. In addition, the Metro parser was designed so that the order in which data elements are defined is not important, even if they reference one another. This was done to eliminate side-effects related to data ordering, where changing the order in which things are defined in a file can change the behavior of or break your code.

Versions of Metro prior to 1.4 contained limited support for conditional logic. After some experimentation, I've decided that the conditional support is not necessary, and it is not used by Metro 1.4. However, there are of course ways to get Metro to support different alternative things -- you just don't implement these with conditionals. Instead, these are implemented by using one Metro variable to dynamically define another Metro file that will be included. This way, different settings of a Metro variable will alter Metro's behavior, but we don't need to have if statements in the data model itself.

Examples

Here is some sample Metro data:

path: /usr/bin

Above, we have defined the element path to have the value /usr/bin. path is a single-line element, and the Metro parser takes care of trimming any trailing whitespace that may be on the line. You can also define single-line elements that have values that consist of multiple whitespace-separated values:

options: ccache replace

Sometimes, you need to define an element but leave it blank. To do this, don't specify any values after the colon:

options:

In Metro, the / character is used to delineate various classes of elements, as follows:

path/mirror: /home/mirror/funtoo
path/mirror/snapshot: /home/mirror/funtoo/snapshots
path/metro: /usr/lib/metro

Above, we see the proper Metro convention for specifying paths. Each path has a prefix of path/. We have a path/mirror element but also have a path/mirror/snapshot element. The / is used to organize our data into logical groups. This is not enforced by Metro but is presented here as a best practice.

The data above could also be represented using a section annotation, as follows:

[section path]

mirror: /home/mirror/funtoo
mirror/snapshot: /home/mirror/funtoo/snapshots
metro: /usr/lib/metro

Above, the [section path] line is a section annotation, and it tells the Metro parser that the path/ prefix should be applied to all following data elements. A section annotation is in effect until another section annotation is encountered by the parser.

While our data above is getting more organized, there is some redundancy in our data, which generally isn't a good thing. Here's an example of how to make our data a bit more compact:

[section path]

mirror: /home/mirror/funtoo
mirror/snapshot: $[path/mirror]/snapshots
metro: /usr/lib/metro

Above, we have used an element reference of $[path/mirror] to reference our path/mirror element. What this means is that path/snapshot will have a value of /home/mirror/funtoo/snapshots.

Also, it's worth pointing out that we could just have well written:

[section path]

mirror/snapshot: $[path/mirror]/snapshots
mirror: /home/mirror/funtoo
metro: /usr/lib/metro

In other words, it's perfectly OK to use the element reference of $[path/mirror] on a line before the actual definition of path/mirror. Metro doesn't care about the order in which data is defined.

Metro provides another way to organize your data in an efficient way. Supposing that you had a lot of path/mirror-related data, then it might be useful to organize your data as follows:

[section path]

metro: /usr/lib/metro

[section path/mirror]

: /home/mirror/funtoo
snapshot: $[]/snapshot
source: $[]/$[source/subarch]/funtoo-$[source/subarch]-$[source/version]/$[source/name].tar.bz2

Above, we have used two new parser features. Inside [section path/mirror], we can define the path/mirror element itself by using a blank element name, followed by a :. The next parser feature we see above is that we can use $[] to reference the value of the path/mirror value. $[] will always reference the value of the element specified in the section annotation. Also note that as of Metro 1.1, $[:] can be used as an alternate form of $[]. In addition, as of Metro 1.2.4, $[:foo] can be used as an alternate form of $[section-name/foo].

Collect Annotations

Many scripting languages have the notion of an "include" file, or "importing" additional data from a remote file. Metro has this concept as well, but it is implemented in a somewhat different way. You can tell Metro to include data from another file by using a collect annotation.

A collect annotation looks like this:

[collect $[path/metro]/myfile.txt]

Now, we called these things "collect annotations" for a reason - in Metro, they work slightly differently than most languages implement include and import. The main difference is that in Metro, a collect annotation does not happen right away. Instead, Metro will add the file to be collected (in this case, that would be the file /usr/lib/metro/myfile.txt, or whatever $[path/metro]/myfile.txt evaluates to) to a collection queue. This means that Metro will read in the contents of the file at some point in time, and the data in the file will be available to you by the time the parsing is complete. But because Metro doesn't care about the order in which data is defined, it doesn't have the same concept of "read in the data - right now!" that an include or import statement does in other languages.

Conditional Collect Annotations

Metro no longer officially supports conditional collect annotations; however, simple collect annotations can be used to make conditional decisions in Metro, as follows:

[collect ./snapshots/$[snapshot/type]]

Above, Metro will collect from a file based on the value of the $[snapshot/type] element. This allows for varying definitions of elements to exist dependent on the value of $[snapshot/type].

Above, Metro will raise an exception if $[snapshot/type] is undefined or has a value that does not map to a file on disk. If it is possible that $[snapshot/type] may not be defined, use the following format:

[collect ./snapshots/$[snapshot/type:zap]]

Using the :zap modifier, the entire collect argument will be replaced with the empty string if $[snapshot/type] is undefined. If Metro is asked to collect an empty string, it will not throw an exception. So this is a handy way to conditionally disable collection of a file. But please note that for all non-null values of $[snapshot/type], a corresponding file must exist on disk in ./snapshots/ or Metro will throw an exception. :zap is explained in more detail in the "Special Variable Expansion" section, below.

Multi-line elements

Metro supports multi-line elements and they are the foundation of Metro's template engine. A multi-line element can be defined as follows, by using square brackets to delimit multi-line data:

myscript: [
#!/bin/bash
echo $*
]

The terminating closing square bracket should be on a line all by itself.

One of the very useful things about multi-line elements is that they support Metro element references:

myscript: [
#!/bin/bash
echo Metro's path/metro setting is $[path/metro].
]

In the above multi-line element, the $[path/metro] reference will be expanded to contain the appropriate value of the element. It is possible to expand single-line elements inside multi-line elements simply by referencing them using a dollar sign and square brackets.

Metro also allows you to expand multi-line elements inside other multi-line elements. Here's an example of how that works:

myscript: [
#!/bin/bash
$[[steps/setup]]
echo Hi There :)
]