Changes

Jump to: navigation, search

Creating Your Own Meta-Repo and Kits

4,840 bytes added, 7 months ago
Using New Kits
For our local development setup, we will be using [http://gitolite.com gitolite]. Gitolite will make things quite a bit easier by managing git repositories for us. ''Think of gitolite as your own private GitHub that has no Web user interface'' (we modify its settings by pushing to its special {{c|gitolite-admin}} repo) and you'll have a pretty good idea of what gitolite does. We will be using the following systems in these examples:
* {{c|repohost}} - this system will be running gitolite under the {{c|repos}} user account and will house git repositories for meta-repo and kits so that they are stored at a handy central location.If you are going to be using your dev workstation as your "repohost", you can simply replace all references to {{c|repohost}} with {{c|localhost}} below :)
* {{c|ryzen}} - in these examples, this will be the primary development workstation, which will be used for editing cloned git code as well as generating custom kits. Once generated, the custom meta-repo and kits are pushed up to {{c|repohost}}.
{{Note|When you follow this guide, it is certainly possible to have {{c|repohost}} and {{c|ryzen}} be the same computer. If you set things up this way, simply use {{c|localhost}} instead of {{c|repohost}} in all command-line steps and configuration files.}}
{{Important|This document assumes you have basic knowledge of {{c|ssh-keygen}} and how to generate public/private SSH key pairs. If you don't know how to to this, see [[Funtoo Containers#Generating SSH Keys|Generating SSH Keys]] for quick steps or [[OpenSSH Key Management, Part 1]] for a more detailed introduction. For this article, you'll probably want to generate a private keys without a passphrase, which is more convenient but a much greater security risk if the private key gets compromised, or one with a passphrase but using [[keychain]] to manage ssh-agent for you.}}
{{console|body=
# ##i##su --login repos
$ ##i##git clone https://github.com/sitaramc/gitolite
$ ##i##install -d ~/bin
{{Note|OK, gitolite is installed on '''repohost'''! From this point forward, we will be using the {{c|drobbins}} (or equivalent) account on your development workstation to configure gitolite remotely.}}
 
{{Note|If you are setting up gitolite on a separate server, and assuming you have ssh properly configured, it would be prudent at this point to test the connection to the remote server from your local workstation . Perform {{c|ssh -T}}, or ssh test command appropriate to your setup, as shown below to verify you can connect:
{{console|body=
$ ##i##ssh -T repos@remote-server-name
}}
 
You should receive confirmation message similar to:
{{console|body=
hello repos, this is repos@remote-server-name running gitolite3 v3.6.11-3-g39293e6 on git 2.19.1
 
R W gitolite-admin
R W testing
}}
 
If you receive the following message during testing:
{{console|body=
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
...
}}
 
then localization settings for the remote server have not been completed. If remote server is Funtoo-based, complete the steps on the page [[Funtoo Linux Localization]] on the remote server.
}}
=== gitolite-admin Clone ===
=== gitolite Configuration ===
Since I will be generating meta-repo and kits on '''ryzen''', this system will need to have permissions to create repositories in gitolite. Although I do my regular development as the drobbins user, I will be running the meta-repo generating script on '''ryzen''' as root, so the root user on ryzen needs to be granted access to gitolite. I would typically do this on '''ryzen''' as follows. First, as root, grab root's public ssh key:
{{console|body=
}}
{{Note|If {{f|/root/.ssh/id_rsa.pub}} doesn't exist, you can create it by typing {{c|ssh-keygen -t rsa}} as root and specifying no passphrase for encryption.}} Then, as my regular {{c|drobbins}} user that I typically use to perform development, I will install root's public key in my cloned gitolite repository. This is the first step to grantthe ryzen root user direct access to gitolite:
{{console|body=
# NOTE: to enable read-only access for certain boxes, uncomment this line:
# R = @reporead
}}
 
Now, we will want to commit and push our changes in the gitolite repo so they take effect -- This is how gitolite configuration changes are applied:
 
{{console|body=
$ ##i##git add .
$ ##i##git commit -a -m "Initial setup"
$ ##i##git push
Enumerating objects: 10, done.
Counting objects: 100% (10/10), done.
Delta compression using up to 12 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 607 bytes | 607.00 KiB/s, done.
Total 6 (delta 0), reused 0 (delta 0)
To localhost:gitolite-admin
05646c4..f8cf1dd master -> master
$
}}
=== The Code ===
You can find the code that does this on GitHubcode.funtoo.org, housed at https://githubcode.funtoo.comorg/bitbucket/funtooprojects/CORE/repos/merge-scripts/browse. The script that does all the heavy-lifting is called {{c|merge-all-kits.py}}. Let's clone it from git, on the machine that will be generating new kits and meta-repo. In our example, this will be as the root user on ryzen:
{{console|body=
# ##i##cd /root
# ##i##git clone https://githubcode.comfuntoo.org/bitbucket/scm/funtoocore/merge-scripts.git
}}
{{Note|While it is possible to use your own custom merge script repository, we recommend starting by using our official merge-scripts repository on GitHubcode.funtoo.org, and progress to using your own fork of merge-scripts only when you need to. Typically you can just fork our kit-fixups repo and work from there.}}
=== Configuration ===
Now that merge-scripts is cloned, we will need to create a {{c|/root/.merge}} configuration file. Use the following Here is an example file that can serve as a starting point. In this example, developer {{c|bcowan}} has forked the kit-fixups repository on code.funtoo.org, and wants to test his personal changes by generating a complete meta-repo of his own:
{{file|name=/root/.merge|body=
[sources]
flora = https://githubcode.comfuntoo.org/funtoobitbucket/scm/co/flora.gitkit-fixups = httpsssh://githubgit@code.funtoo.comorg:7999/mygithub_user~bcowan/kit-fixups.gitgentoo-staging = https://githubcode.funtoo.comorg/bitbucket/scm/funtooauto/gentoo-staging.git
[destinations]
==== Sources Section ====
Let's walk through this configuration file. The {{c|[sources]}} section defines locations of repositories that the merge scripts will use as sources for creating kits and meta-repo. In the above sample config, we are using the official [[Flora]] repository from Funtoo, and the official gentoo-staging repository used by Funtoo, but we are using our own fork of [https://githubcode.comfuntoo.org/funtoobitbucket/projects/CORE/repos/kit-fixups/browse kit-fixups], which will allow us to add new ebuilds that will appear in kits, such as bug fixes to existing ebuilds in kits, as well as security fixes. For a core Funtoo Linux developer, this is a good way to start. If you are more interested in contributing third-party ebuilds, then you may instead choose to create your own fork of [https://githubcode.comfuntoo.org/funtoobitbucket/projects/CO/repos/flora/browse flora], and use our standard kit-fixups repository. Or, you could choose to create forks of both. The recommended best practice is to use our upstream repos when possible, and fork only those repos you want to customize. This way, you'll ensure that you have the most up-to-date versions of ebuilds in those unforked repos.
==== Branches Section ====
{{console|body=
 
# ##i##cd /root/merge-scripts
# ##i##bin/merge-all-kits push1.3-release
}}
Before starting the script for the first time , you should emerge jinja and configure your git user.name and user.email variables.
{{console|body=
# ##i##emerge -u jinja
# ##i##git config --global user.email "you@example.com"
# ##i##git config --global user.name "Your Name"
# ##i##mv meta-repo meta-repo.official
# ##i##ego sync
Syncing xorg-kit branch 1.20-release
Cloning into '/var/git/meta-repo/kits/xorg-kit'...
Initialized empty Git repository in /home/repos/repositories/wildrepo/staging/xorg-kit.git/
fatal: Remote branch 1.20-release not found in upstream origin
##r##ERROR: Could not clone kit 'xorg-kit' into '/var/git/meta-repo/kits/xorg-kit'.
fatal: the remote end hung up unexpectedly
}}
 
Ego will now sync your custom repository, ''but it will fail.''. This is OK -- it is failing because gitolite does not have local copies of independently-maintained (ie. not auto-generated) kits. To fix this, you can either create mirrors of the independent kits on your gitolite, or alternatively perform this simple work-around. Go to https://code.funtoo.org/bitbucket/projects/INDY and for each kit listed there, clone to {{f|/var/git/meta-repo/kits}} as root, as follows:
 
{{console|body=
# ##i##cd /var/git/meta-repo/kits
# ##i## git clone https://code.funtoo.org/bitbucket/scm/indy/gnome-kit.git
# ##i## git clone https://code.funtoo.org/bitbucket/scm/indy/python-kit.git
... etc ...
}}
 
Once this is complete, try an {{c|ego sync}} again, and with local copies of the independent kits it should all work:
 
{{console|body=
# ##i##ego sync
...
##g##Sync successful and kits in alignment! :)
Updating /etc/portage/repos.conf...
Updating profiles at /etc/portage/make.profile/parent...
}}
Ego will now sync your custom repository. If you type {{c|emerge -auDN @world}}, ego will now be using your custom kits, rather than the official Funtoo ones. This means that you can perform a variety of things you couldn't before. You can now add your own custom ebuilds to your fork of {{c|kit-fixups}}, and {{c|merge-all-kits.py}} will automatically incorporate these changes into your own custom kits. This will allow you to locally test any changes before submitting them as pull requests to Funtoo. You will also be able to maintain your own meta-repo and kits with your own local modifications, and have your systems use these meta-repo/kits instead of the official Funtoo ones.
== Using New Kits with Metro ==
{{console|body=
# ##i##cd /root
# ##i##git clone https://githubcode.comfuntoo.org/bitbucket/scm/funtoocore/metro.git
# ##i##cd metro
# ##i##scripts/autosetup
}}
Now, we will set the {{c|ego}}, administration tool of Funtoo/Linux. The way it is used with metro is independent from {{c|app-admin/ego}} installed on your box. Setup is easy as follows:
{{console|body=
# ##i##cd /root
# ##i##git clone https://code.funtoo.org/bitbucket/scm/core/ego.git
}}
This way you will have {{c|/root/ego}} directory with {{c|ego}} binary that is then used by metro.
After following the interactive prompts, you'll then want to perform the following steps to install an ego.conf for use by metro:
Without these ego options, metro will use the official meta-repo instead of your local custom meta-repo. Of course, be sure to specify any custom kit branches to use as well.
One Once your ego.conf is installed, you can perform a metro build with something similar to the following commands:
{{console|body=
== Changing Repo Definitions ==
Now that you have generated your first meta-repo, let's look at the key file that defines what overlays and repositories are used to create meta-repo, as well as what kit branches exist, and which ones are prime and which are not. You will want to turn your attention to the {{f|kit-fixups/modules/fixups/foundations.py}} file, which [https://githubcode.comfuntoo.org/bitbucket/projects/CORE/funtoorepos/kit-fixups/blob/masterbrowse/modules/fixups/foundations.py can be viewed here on GitHubcode.funtoo.org]]. Let's take a look at various sections of this file:
{{file|name=foundations.py|lang=python|desc=Top of foundations.py file|body=
catpkgs in the skip list -- no patterns or other special matches are supported.}}
==== Inline Package Moves ====
{{Note|The "literal with move" option is a new feature, described below.}}
The "literal with move" package set syntax described above is one way to tell the merge scripts to copy a catpkg from a source location but give it a new catpkg name, and they are specified directly in the package-set files. If the old package name is found, it will be copied over as the new name. If the old name is not found, but the new name is found, the new name will be copied over as the new name. So either the old name or the new name, if found, will be copied over.
If the old package name is found, it will be copied over as the new name. If the old name is not found, but the new name is found, the new name will be copied over as the new name. ===== Move-Maps =====
You can also specify package moves by creating a file called {{f|kit-fixups/move-maps/kitname}} containing the same "literal with move" syntax. Global move maps can be placed in {{f|kit-fixups/move-maps/global}}. Also note that {{f|kit-fixups/move-maps/nokit}} or {{f|kit-fixups/move-maps/global}} is the only way to perform funtoo package moves for nokit. I recommend using the "global" method since it will still automatically work if someone else moves your package into a kit.
The difference between the "inline" (inside This move-maps functionality works similarly to package-moves that appear within a kit package-set) and , except that these moves do not automatically add either specified catpkg to the "out-of-band" (kit. So there must be something in the {{f|movepackage-maps/}} directory) method is set that the first method adds both matches the old and the new catpkg package name to . After the old package set as well as creating is matched, additional logic looks at the move map. The second method only defines the move map -- it will apply only maps and see if the catpkg happens to ''also'' be part of the package-set being processed, or matching match is in a {{c|@regex@}}, {{c|@depsincat@}}, etc. Otherwise the move-map data that tells us to copy it over as the "new" name. The old package will not be utilized. Remember, for every kit, copied over as the data in {{f|kit-fixups/move-maps/kitname}} and {{f|kit-fixups/move-maps/global}} will be utilizednew name.
If you need to just rename a single package in a kit, it's fine to use the inline method. For more capability, {{f|move-maps/}} are your friend. They're more powerful because a move-map specified with {{f|move-maps/}} will also apply to any special wildcard package-set matches via {{c|@regex@}}, etc, whereas the inline method is limited by design. Also, the out-of-band method allows you to perform package moves on catpkgs in nokit.
{{Note|The above two approaches can be used to move catpkgs between kits transparently to the end-user. In the next ego sync, the catpkg will atomically move from one kit to another and no re-emerging will be required, even if the user had emerged the package from the 'old' kit location.}}
; I want to contribute a cool package to Funtoo.: To do this, you will want to open a pull request against [https://githubcode.comfuntoo.org/bitbucket/projects/CO/repos/funtooflora/browse flora]. Flora is used for all 'bonus' community-contributed ebuilds.
; I want to fix a bug in a particular ebuild.: To do this, first find out where the ebuild is coming from. A good way to do this is to type {{c|ls -d /var/git/meta-repo/kits/*/sys-apps/foobar}}, which will show you what kit it is in. Running {{c|emerge -s sys-apps/foobar}} will also display this information. For research purposes, it is often useful to find where the original catpkg was sourced from. You can consult https://ports.funtoo.org/packages.xml which contains a list of all catpkgs and their source repository. After doing some initial research and seeing what's wrong, you might have a fix for the ebuild. Generally, the best way to fix the ebuild is to clone fork [https://githubcode.comfuntoo.org/bitbucket/funtooprojects/CORE/repos/kit-fixups/browse kit-fixups ] and create an appropriate fixup for the ebuild if none exists, and simply improve our fixup if one exists already. Then you can create a GitHub code.funtoo.org pull request, or open a bug on bugs.funtoo.org, or both. Remember that fixup catpkgs will totally replace all upstream ebuilds, so you may need to include multiple versions of the ebuild, even ones that don't need a fix, if they are still needed for certain packages.
{{Note|If you want to fix a bug in an ebuild and you find that the ebuild comes from flora, you will want to fork flora and submit a pull request against flora instead.}}
{{Important|Be sure to stop by {{c|#funtoo-dev}} on irc.freenode.net if you need further assistance! We are here to help.}}
 
[[Category:Official Documentation]]
[[Category:Development]]
Bureaucrats, Administrators, wiki-admins, wiki-staff
6,313
edits

Navigation menu