Difference between pages "Portage Git Mirror" and "FLOP:Ports-2015"

From Funtoo
(Difference between pages)
Jump to navigation Jump to search
 
 
Line 1: Line 1:
=Setting up local git mirror =
{{FLOP
 
|Created on=2015/02/23
This tutorial explains how to save bandwidth when several local computers need to pull updates from a single remote git repository.
|Summary=Collection of ideas and changes for the ports-2015 tree. The goal is to perform many scheduled changes with a single user configuration change.
 
|Author=Mgorny,
== Use case ==
|Reference Bug=FL-1877
}}
== Procedure ==
Users of ports-2012 tree will be informed that the current repository is deprecated, and provided with complete migration instructions. The instructions will cover both necessary and optional changes that can be done conveniently along with the necessary switch.


This tutorial will be about hosting a local mirror of funtoo git based portage tree.
== Changes ==
=== History cut-off ===
As an implication of starting a new tree, all history is cut-off. While users were cloning the repository with --depth=1, old clones have accumulated a large history of changes. This history will be discarded with the new clone. This will significatly decrease the size of portage tree and the size of portage tree compressed tarball, if someone prefer to use it. Eventually, portage tree will grow up again.


Following terms should be adapted
=== Portage upgrade / repos.conf switch ===
 
Reference: {{Bug|FL-1761}}, [[Repository Configuration]]
<table border="1">
<tr class="header">
<th align="left">Terms</th>
<th align="left">Definition</th>
</tr>
<tr class="odd">
<td align="left">git.lan</td>
<td align="left">The git-daemon local mirror host</td>
</tr>
<tr class="even">
<td align="left">localhost</td>
<td align="left">Any local host</td>
</tr>
<tr class="odd">
<td align="left">nobody</td>
<td align="left">Owner user of .git files</td>
</tr>
<tr class="even">
<td align="left">/home/git-mirrors</td>
<td align="left">Base path of git-daemon</td>
</tr>
</table>
 
== Local mirror ==
 
==== Prepare directories and get portage tree====
<console>
###i## mkdir /home/git-mirrors
###i## chown nobody /home/git-mirrors
###i## su -s /bin/sh nobody
$##bl## cd /home/git-mirrors
$##bl## git clone --mirror --bare git://github.com/funtoo/ports-2012.git portage.git
</console>
For a security reason we use a nobody user .
==== git-daemon configuration====
{{file|name=/etc/conf.d/git-daemon|desc=|body=
GITDAEMON_OPTS=--syslog --verbose --enable=receive-pack --export-all
GITDAEMON_OPTS=${GITDAEMON_OPTS} --base-path=/home/git-mirrors /home/git-mirrors
GIT_USER=nobody
GIT_GROUP=nobody
}}


====Service configuration====
As a part of upstream Portage changes, the upgrade is accompanied with some configuration file changes. Aside them, repository configuration is moved to repos.conf and the repository name becomes significant. Merging this with ports-2015 switch allows users to update the configuration in new format already.
To start daemon with a mirror machine boot add <code>git-daemon</code> to default runlevel
<console>
###i## rc-update add git-daemon default
</console>
To make changes start immediately just run <code>rc</code>
<console>
###i## rc
</console>


=== Pull from remote ===
=== Repository rename ===
Reference: {{Bug|FL-1801}}


Add the following to <code>/etc/cron.daily/funtoo-sync.sh</code>:
Right now, the main repository inherits the name 'gentoo'. This is a bit confusing, considering that it is a modified Funtoo variant of the package tree. Changing the name to 'funtoo' would improve consistency and carry some bit of 'branding' into packages. Merging this into ports-2015 switch allows users to consciously update all repository references if necessary, and combines the change with necessity of specifying repository name in repos.conf.


<pre>
== Other possible changes ==
#!/bin/sh
=== Filesystem structure reorganization ===
cd /home/git-mirrors/portage.git
Since users will be required to clone the new repository, it may be desired to suggest some best practices for filesystem layout. This specifically includes separating ebuilds from distfiles & packages, and using a multi-repository layout. Historically, portage tree inspired by FreeBSD ports, located in <code>/usr/portage>. This FHS layout is different in BSD and in Linux and we will probably move it elsewhere, see below for a suggestion and possible pros and cons.
su nobody -s "/bin/sh" -c "git pull"
</pre>


== Cloning from local git-daemon ==
Example layout suggested by mgorny:
# all repositories in ''/var/db/repos/${repo_name}'' (i.e. Funtoo repository in ''/var/db/repos/funtoo'', and possible overlays as other directories in ''/var/db/repos''),
# distfiles in ''/var/cache/portage/distfiles'',
# binary packages in ''/var/cache/portage/packages''.


Local clone from <code>git.lan</code>:
Users can be recommended to use a separate filesystem that can handle small files efficiently for ''/var/db/repos'', e.g. btrfs, reiserfs or possibly squashfs.
<console>
###i## mv /usr/portage /usr/portage.old
###i## git clone git://git.lan/portage.git /usr/portage
###i## cd /usr/portage
###i## git checkout funtoo.org
</console>


[[Category:HOWTO]]
{{FLOPFooter}}

Revision as of 13:50, February 23, 2015

Created on
2015/02/23
Original Author(s)
Mgorny
Status
Reference Bug
FL-1877

Funtoo Linux Optimization Proposal: Ports-2015

Collection of ideas and changes for the ports-2015 tree. The goal is to perform many scheduled changes with a single user configuration change.

Procedure

Users of ports-2012 tree will be informed that the current repository is deprecated, and provided with complete migration instructions. The instructions will cover both necessary and optional changes that can be done conveniently along with the necessary switch.

Changes

History cut-off

As an implication of starting a new tree, all history is cut-off. While users were cloning the repository with --depth=1, old clones have accumulated a large history of changes. This history will be discarded with the new clone. This will significatly decrease the size of portage tree and the size of portage tree compressed tarball, if someone prefer to use it. Eventually, portage tree will grow up again.

Portage upgrade / repos.conf switch

Reference: FL-1761, Repository Configuration

As a part of upstream Portage changes, the upgrade is accompanied with some configuration file changes. Aside them, repository configuration is moved to repos.conf and the repository name becomes significant. Merging this with ports-2015 switch allows users to update the configuration in new format already.

Repository rename

Reference: FL-1801

Right now, the main repository inherits the name 'gentoo'. This is a bit confusing, considering that it is a modified Funtoo variant of the package tree. Changing the name to 'funtoo' would improve consistency and carry some bit of 'branding' into packages. Merging this into ports-2015 switch allows users to consciously update all repository references if necessary, and combines the change with necessity of specifying repository name in repos.conf.

Other possible changes

Filesystem structure reorganization

Since users will be required to clone the new repository, it may be desired to suggest some best practices for filesystem layout. This specifically includes separating ebuilds from distfiles & packages, and using a multi-repository layout. Historically, portage tree inspired by FreeBSD ports, located in /usr/portage>. This FHS layout is different in BSD and in Linux and we will probably move it elsewhere, see below for a suggestion and possible pros and cons.

Example layout suggested by mgorny:

  1. all repositories in /var/db/repos/${repo_name} (i.e. Funtoo repository in /var/db/repos/funtoo, and possible overlays as other directories in /var/db/repos),
  2. distfiles in /var/cache/portage/distfiles,
  3. binary packages in /var/cache/portage/packages.

Users can be recommended to use a separate filesystem that can handle small files efficiently for /var/db/repos, e.g. btrfs, reiserfs or possibly squashfs.