Difference between revisions of "Funtoo:Metatools"
(25 intermediate revisions by 2 users not shown) | |||
Line 3: | Line 3: | ||
|Project Category=Development | |Project Category=Development | ||
|leads=Drobbins | |leads=Drobbins | ||
|contributors=alex2101, invakid404 | |contributors=S4uliu5, alex2101, invakid404 | ||
| | |subpages=Releases, Advanced Usage | ||
|components={{Project/Component | |components={{Project/Component | ||
|name=funtoo-metatools | |name=funtoo-metatools | ||
Line 15: | Line 14: | ||
|summary=The simple plugin-oriented framework that supports metatools. | |summary=The simple plugin-oriented framework that supports metatools. | ||
}} | }} | ||
|updates={{Project/Update | |||
|date=2022-02-22 | |||
|status=[[User:drobbins|drobbins]] is working on the "master" branch of metatools. With the 1.0.2 release, mongodb is no longer a dependency. | |||
}}{{Project/Update | |||
|date=2022-02-28 | |||
|status=Drobbins is working on "master" branch, specifically merge-kits, and getting this working fully (currently, "legacy" branch is used for tree regen). This involves upgrading release YAML, implementing support for "sourced" kits, and testing. | |||
}}{{Project/Update | |||
|date=2022-03-15 | |||
|status=Metatools git master now can successfully regen the tree in "dev" mode. It doesn't use mongodb, or threading like the previous incarnation, but is still faster than the older threaded version. This is an important milestone -- soon we will be using this version for production tree regens. | |||
}}{{Project/Update | |||
|date=2022-04-15 | |||
|status=[[Funtoo:Metatools/Releases/1.0.3|Metatools 1.0.3]] has been released, with a bunch of improvements including support for running merge-kits. | |||
}}{{Project/Update | |||
|date=2022-08-19 | |||
|status=[[Funtoo:Metatools/Releases/1.1.0|Metatools 1.1.0]] has been released, which includes a lot of new features and improvements. | |||
}} | }} | ||
Metatools ([https://code.funtoo.org/bitbucket/users/drobbins/repos/funtoo-metatools/browse git]) ([https://pypi.org/project/funtoo-metatools/ pypi]) is the framework used by Funtoo Linux | }} | ||
to create meta-repo and autogenerate ebuilds. It is a foundational tool for Funtoo Linux development. | '''Metatools''' ([https://code.funtoo.org/bitbucket/users/drobbins/repos/funtoo-metatools/browse git]) ([https://pypi.org/project/funtoo-metatools/ pypi]) is the framework used by Funtoo Linux | ||
to create meta-repo and autogenerate ebuilds. It is a foundational tool for Funtoo Linux development. It is also a living experiment exploring how we can develop and maintain ebuilds more efficiently and effectively. | |||
{{Note|{{LatestRelease|metatools}}}} | |||
=== What is an "Autogen"? === | === What is an "Autogen"? === | ||
Line 58: | Line 73: | ||
For more information on setting up GitHub personal access tokens, see: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token | For more information on setting up GitHub personal access tokens, see: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token | ||
== | == Typical Developer Usage of Doit == | ||
This section describes basic usage of metatools, using the {{c|doit}} command, which you may run directly in kit-fixups locally in order to test some autogens you happen to be working on for a PR. | |||
Funtoo's meta-repo is defined in the kit-fixups repository ([https://code.funtoo.org/bitbucket/projects/CORE/repos/kit-fixups/browse git]), which is the starting point for all autogeneration of ebuilds. | {{Note|Funtoo's meta-repo is defined in the kit-fixups repository ([https://code.funtoo.org/bitbucket/projects/CORE/repos/kit-fixups/browse git]), which is the starting point for all autogeneration of ebuilds.}} | ||
To perform your first {{c|doit}} run, enter into the appropriate kit-fixups directory and run the {{c|doit}} command: | To perform your first {{c|doit}} run, enter into the appropriate kit-fixups directory and run the {{c|doit}} command: | ||
Line 93: | Line 110: | ||
## Auto-create ebuilds for these packages, along with Manifests. | ## Auto-create ebuilds for these packages, along with Manifests. | ||
Funtoo contributors will use the {{c|doit}} command locally to test their autogens and ensure their proper operation before submitting a PR. | Funtoo contributors will use the {{c|doit}} command locally to test their autogens and ensure their proper operation before submitting a PR. Once you use {{c|doit}} to autogen a small set of ebuilds, you can add them to your local meta-repo using a couple of methods. Some developers will copy them physically to {{f|/var/git/meta-repo}}. It's also convenient to temporarily add the particular directory of kit-fixups as an overlay, which works if your ebuilds have newer versions that what is currently in your tree: | ||
{{file|name=/etc/make.conf|body= | |||
PORTDIR_OVERLAY=/home/drobbins/development/kit-fixups/core-kit/curated | |||
}} | |||
== Generate Your Own Meta-Repo == | |||
It's also very possible to generate your very own {{f|meta-repo}} using the {{c|merge-kits}} command. In this workflow, you will run {{c|merge-kits next}} or {{c|merge-kits 1.4-release}} to generate a particular release. The auto-generated meta-repo will be created in {{f|~/repo_tmp/dest-trees/meta-repo}}. This {{f|meta-repo}} can be copied or symlinked into place at {{f|/var/git/meta-repo}}: | |||
{{console|body= | |||
$ ##i##merge-kits next --fixups_url=<path to your fork> --fixups_branch=mybranch | |||
}} | |||
When used in this way, {{f|~/repo_tmp/dest-trees/meta-repo}} and the individual kit git repositories that exist inside {{f|~/repo_temp/dest-trees/meta-repo/kits}} will be auto-created git trees with no remotes defined, but they can still be used as-is. | |||
Here's more info for the important command-line options for {{c|merge-kits}}: | |||
; {{c|--fixups_url}}: Use this option to point to where to clone your kit-fixups repository from. You can also use the official URL for Funtoo's kit-fixups, which is https://code.funtoo.org/bitbucket/scm/core/kit-fixups.git. ''You only need to use this option the first time you run {{c|merge-kits}}, or if you wipe {{f|~/repo_tmp/source-trees/kit-fixups}}.'' | |||
; {{c|--fixups_branch}}: '''This option will be available in 1.0.4, and is in 'git master' meta-repo right now.''' Use this option to specify your branch, which defaults to {{c|master}}. ''This option is 'sticky', which means {{c|merge-kits}} will continue to use the specified branch as long as {{f|~/repo_tmp/source-trees/kit-fixups}} has not been wiped and you have not specified a different branch with another {{c|--fixups-branch}} call.'' Therefore, you can use this option to periodically change to a different branch you happen to be testing. | |||
=== Where Can I Learn More about Metatools? === | === Where Can I Learn More about Metatools? === | ||
The best place to learn is to join us in the #metatools and # | The best place to learn is to join us in the #metatools and #development channels on [https://discord.gg/BNUSpUU Discord]. | ||
There is "in progress" documentation for metatools at our to-be-launched projects site: https://projects.funtoo.org/metatools/docs/install.html | There is "in progress" documentation for metatools at our to-be-launched projects site: https://projects.funtoo.org/metatools/docs/install.html | ||
SSL Cert is not yet valid and the documentation is a work in progress. Refer to this wiki page for current setup documentation until the documentation | SSL Cert is not yet valid and the documentation is a work in progress. Refer to this wiki page for current setup documentation until the documentation | ||
has caught up with the current release of metatools. | has caught up with the current release of metatools. | ||
== Status Updates == | |||
{{Project/UpdateList}} | |||
[[Category:Official Documentation]] | [[Category:Official Documentation]] | ||
[[Category:Development]] | |||
{{ProjectFooter}} | {{ProjectFooter}} |
Revision as of 21:19, August 19, 2022
Metatools 1.1.0 has been released, which includes a lot of new features and improvements.
19 August 2022
Metatools (git) (pypi) is the framework used by Funtoo Linux to create meta-repo and autogenerate ebuilds. It is a foundational tool for Funtoo Linux development. It is also a living experiment exploring how we can develop and maintain ebuilds more efficiently and effectively.
The latest release of metatools is 1.3.5, released on 09 November 2023. It is a regular software release. View Release Notes.
What is an "Autogen"?
Quite simply, an "autogen" is kind of like an ebuild generator, using the metatools Python framework to dynamically generate ebuilds. A key functionality of an autogen is to allow ebuilds to be generated to reflect the current version of source code that is available from upstream.
Where can I Find Autogens?
You can find autogens by looking in the kit-fixups repository (git) for any files named:
autogen.py
autogen.yaml
Both YAML and Python autogens use our metatools framework -- the YAML variants are higher-level autogens that feed metatools "generators", kind of like a pipeline. As of Jan 29, 2022, all Funtoo Linux source overlays -- kit-fixups as well as others -- support the creation of autogens. What this means is that it's possible to write an autogen for anything in Funtoo to ensure that it is always kept up-to-date.
Installation
To install metatools, emerge it:
root # emerge metatools
Autogen Setup
The command in metatools that performs autogeneration locally is called doit
. Prior to using this command locally, it's recommended to create an ~/.autogen
file under your user account that
contains a GitHub personal access token. This will allow doit
to use it when querying GitHub and will prevent doit
from hitting API limits:
~/.autogen
authentication:
api.github.com:
username: danielrobbins
password: <insert access token string here>
For more information on setting up GitHub personal access tokens, see: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token
Typical Developer Usage of Doit
This section describes basic usage of metatools, using the doit
command, which you may run directly in kit-fixups locally in order to test some autogens you happen to be working on for a PR.
Funtoo's meta-repo is defined in the kit-fixups repository (git), which is the starting point for all autogeneration of ebuilds.
To perform your first doit
run, enter into the appropriate kit-fixups directory and run the doit
command:
user $ cd development/kit-fixups/core-kit/curated user $ doit [INFO 21:02:35] Fetched https://git-scm.com/download/linux 6042 bytes [INFO 21:02:35] Created: dev-vcs/git/git-2.35.1.ebuild [INFO 21:02:35] Fetched https://api.github.com/repos/hashicorp/vagrant_cloud/tags 9513 bytes [INFO 21:02:36] Created: dev-ruby/vagrant_cloud/vagrant_cloud-2.0.3.ebuild [INFO 21:02:36] Fetched https://api.github.com/repos/distcc/distcc/releases 34549 bytes [INFO 21:02:36] Created: sys-devel/distcc/distcc-3.4.ebuild [INFO 21:02:36] Fetched https://api.github.com/repos/dbrgn/tealdeer/releases 151336 bytes [INFO 21:02:36] Fetched https://api.github.com/repos/tmux/tmux/releases 94211 bytes [INFO 21:02:36] Created: app-misc/tmux/tmux-3.2a.ebuild [INFO 21:02:36] Fetched https://api.github.com/repos/lotabout/skim/releases 225892 bytes [INFO 21:02:36] Fetched https://ftp.debian.org/debian/pool/main/c/ca-certificates/ 7174 bytes [INFO 21:02:36] Created: app-misc/ca-certificates/ca-certificates-20211016.ebuild [INFO 21:02:36] Fetched https://api.github.com/repos/jarun/nnn/releases 564842 bytes [INFO 21:02:36] Fetched https://api.github.com/repos/netwide-assembler/nasm/tags 13895 bytes [INFO 21:02:36] Created: app-misc/nnn/nnn-4.4.ebuild [INFO 21:02:36] Fetched https://api.github.com/repos/doxygen/doxygen/tags?per_page=100 46121 bytes
What you are seeing above is output from the doit
command, which will:
- Find all autogens (
autogen.yaml
andautogen.py
files) in the current directory tree. - Execute them using multiple threads, during which
doit
will:- Query upstream Web sites and APIs (like GitHub) for current versions of source code,
- Fetch all source code artifacts (distfiles) by using an efficient spider engine, and
- Auto-create ebuilds for these packages, along with Manifests.
Funtoo contributors will use the doit
command locally to test their autogens and ensure their proper operation before submitting a PR. Once you use doit
to autogen a small set of ebuilds, you can add them to your local meta-repo using a couple of methods. Some developers will copy them physically to /var/git/meta-repo
. It's also convenient to temporarily add the particular directory of kit-fixups as an overlay, which works if your ebuilds have newer versions that what is currently in your tree:
/etc/make.conf
PORTDIR_OVERLAY=/home/drobbins/development/kit-fixups/core-kit/curated
Generate Your Own Meta-Repo
It's also very possible to generate your very own meta-repo
using the merge-kits
command. In this workflow, you will run merge-kits next
or merge-kits 1.4-release
to generate a particular release. The auto-generated meta-repo will be created in ~/repo_tmp/dest-trees/meta-repo
. This meta-repo
can be copied or symlinked into place at /var/git/meta-repo
:
user $ merge-kits next --fixups_url=<path to your fork> --fixups_branch=mybranch
When used in this way, ~/repo_tmp/dest-trees/meta-repo
and the individual kit git repositories that exist inside ~/repo_temp/dest-trees/meta-repo/kits
will be auto-created git trees with no remotes defined, but they can still be used as-is.
Here's more info for the important command-line options for merge-kits
:
--fixups_url
- Use this option to point to where to clone your kit-fixups repository from. You can also use the official URL for Funtoo's kit-fixups, which is https://code.funtoo.org/bitbucket/scm/core/kit-fixups.git. You only need to use this option the first time you run
merge-kits
, or if you wipe~/repo_tmp/source-trees/kit-fixups
.
--fixups_branch
- This option will be available in 1.0.4, and is in 'git master' meta-repo right now. Use this option to specify your branch, which defaults to
master
. This option is 'sticky', which meansmerge-kits
will continue to use the specified branch as long as~/repo_tmp/source-trees/kit-fixups
has not been wiped and you have not specified a different branch with another--fixups-branch
call. Therefore, you can use this option to periodically change to a different branch you happen to be testing.
Where Can I Learn More about Metatools?
The best place to learn is to join us in the #metatools and #development channels on Discord.
There is "in progress" documentation for metatools at our to-be-launched projects site: https://projects.funtoo.org/metatools/docs/install.html SSL Cert is not yet valid and the documentation is a work in progress. Refer to this wiki page for current setup documentation until the documentation has caught up with the current release of metatools.
Status Updates
- 2022-08-19
- Metatools 1.1.0 has been released, which includes a lot of new features and improvements.
- 2022-04-15
- Metatools 1.0.3 has been released, with a bunch of improvements including support for running merge-kits.
- 2022-03-15
- Metatools git master now can successfully regen the tree in "dev" mode. It doesn't use mongodb, or threading like the previous incarnation, but is still faster than the older threaded version. This is an important milestone -- soon we will be using this version for production tree regens.
- 2022-02-28
- Drobbins is working on "master" branch, specifically merge-kits, and getting this working fully (currently, "legacy" branch is used for tree regen). This involves upgrading release YAML, implementing support for "sourced" kits, and testing.
- 2022-02-22
- drobbins is working on the "master" branch of metatools. With the 1.0.2 release, mongodb is no longer a dependency.