Difference between pages "Portage Git Mirror" and "FLOP:Ports-2015"

(Difference between pages)
(Service configuration)
 
 
Line 1: Line 1:
=Setting up local git mirror =
+
{{FLOP
 +
|Created on=2015/02/23
 +
|Summary=Collection of ideas and changes for the ports-2015 tree. The goal is to perform many scheduled changes with a single user configuration change.
 +
|Author=Mgorny,
 +
|Reference Bug=FL-1877
 +
}}
 +
== Procedure ==
 +
Users of ports-2012 tree will be informed that the current repository is deprecated, and provided with complete migration instructions. The instructions will cover both necessary and optional changes that can be done conveniently along with the necessary switch.
  
This tutorial explains how to save bandwidth when several local computers need to pull updates from a single remote git repository.
+
== Changes ==
 +
=== History cut-off ===
 +
As an implication of starting a new tree, all history is cut-off. While users were cloning the repository with --depth=1, old clones have accumulated a large history of changes. This history will be discarded with the new clone. This will significatly decrease the size of portage tree and the size of portage tree compressed tarball, if someone prefer to use it. Eventually, portage tree will grow up again.
  
== Use case ==
+
=== Portage upgrade / repos.conf switch ===
 
+
Reference: {{Bug|FL-1761}}, [[Repository Configuration]]
This tutorial will be about hosting a local mirror of funtoo git based portage tree.
+
 
+
Following terms should be adapted
+
 
+
<table border="1">
+
<tr class="header">
+
<th align="left">Terms</th>
+
<th align="left">Definition</th>
+
</tr>
+
<tr class="odd">
+
<td align="left">git.lan</td>
+
<td align="left">The git-daemon local mirror host</td>
+
</tr>
+
<tr class="even">
+
<td align="left">localhost</td>
+
<td align="left">Any local host</td>
+
</tr>
+
<tr class="odd">
+
<td align="left">nobody</td>
+
<td align="left">Owner user of .git files</td>
+
</tr>
+
<tr class="even">
+
<td align="left">/home/git-mirrors</td>
+
<td align="left">Base path of git-daemon</td>
+
</tr>
+
</table>
+
 
+
== Local mirror ==
+
 
+
==== Prepare directories and get portage tree====
+
<console>
+
###i## mkdir /home/git-mirrors
+
###i## chown nobody /home/git-mirrors
+
###i## su -s /bin/sh nobody
+
$##bl## cd /home/git-mirrors
+
$##bl## git clone --mirror --bare git://github.com/funtoo/ports-2012.git portage.git
+
</console>
+
For a security reason we use a nobody user .
+
==== git-daemon configuration====
+
{{file|name=/etc/conf.d/git-daemon|desc=|body=
+
GITDAEMON_OPTS=--syslog --verbose --enable=receive-pack --export-all
+
GITDAEMON_OPTS=${GITDAEMON_OPTS} --base-path=/home/git-mirrors /home/git-mirrors
+
GIT_USER=nobody
+
GIT_GROUP=nobody
+
}}
+
  
====Service configuration====
+
As a part of upstream Portage changes, the upgrade is accompanied with some configuration file changes. Aside them, repository configuration is moved to repos.conf and the repository name becomes significant. Merging this with ports-2015 switch allows users to update the configuration in new format already.
To start daemon with a mirror machine boot add <code>git-daemon</code> to default runlevel
+
<console>
+
###i## rc-update add git-daemon default
+
</console>
+
To make changes start immediately just run <code>rc</code>
+
<console>
+
###i## rc
+
</console>
+
  
=== Pull from remote ===
+
=== Repository rename ===
 +
Reference: {{Bug|FL-1801}}
  
Add the following to <code>/etc/cron.daily/funtoo-sync.sh</code>:
+
Right now, the main repository inherits the name 'gentoo'. This is a bit confusing, considering that it is a modified Funtoo variant of the package tree. Changing the name to 'funtoo' would improve consistency and carry some bit of 'branding' into packages. Merging this into ports-2015 switch allows users to consciously update all repository references if necessary, and combines the change with necessity of specifying repository name in repos.conf.
  
<pre>
+
== Other possible changes ==
#!/bin/sh
+
=== Filesystem structure reorganization ===
cd /home/git-mirrors/portage.git
+
Since users will be required to clone the new repository, it may be desired to suggest some best practices for filesystem layout. This specifically includes separating ebuilds from distfiles & packages, and using a multi-repository layout. Historically, portage tree inspired by FreeBSD ports, located in <code>/usr/portage>. This FHS layout is different in BSD and in Linux and we will probably move it elsewhere, see below for a suggestion and possible pros and cons.
su nobody -s "/bin/sh" -c "git pull"
+
</pre>
+
  
== Cloning from local git-daemon ==
+
Example layout suggested by mgorny:
 +
# all repositories in ''/var/db/repos/${repo_name}'' (i.e. Funtoo repository in ''/var/db/repos/funtoo'', and possible overlays as other directories in ''/var/db/repos''),
 +
# distfiles in ''/var/cache/portage/distfiles'',
 +
# binary packages in ''/var/cache/portage/packages''.
  
Local clone from <code>git.lan</code>:
+
Users can be recommended to use a separate filesystem that can handle small files efficiently for ''/var/db/repos'', e.g. btrfs, reiserfs or possibly squashfs.
<console>
+
###i## mv /usr/portage /usr/portage.old
+
###i## git clone git://git.lan/portage.git /usr/portage
+
###i## cd /usr/portage
+
###i## git checkout funtoo.org
+
</console>
+
  
[[Category:HOWTO]]
+
{{FLOPFooter}}

Revision as of 13:50, February 23, 2015

Created on
2015/02/23
Original Author(s)
Michał Górny
Status
Reference Bug
FL-1877

Funtoo Linux Optimization Proposal: Ports-2015

Collection of ideas and changes for the ports-2015 tree. The goal is to perform many scheduled changes with a single user configuration change.

Procedure

Users of ports-2012 tree will be informed that the current repository is deprecated, and provided with complete migration instructions. The instructions will cover both necessary and optional changes that can be done conveniently along with the necessary switch.

Changes

History cut-off

As an implication of starting a new tree, all history is cut-off. While users were cloning the repository with --depth=1, old clones have accumulated a large history of changes. This history will be discarded with the new clone. This will significatly decrease the size of portage tree and the size of portage tree compressed tarball, if someone prefer to use it. Eventually, portage tree will grow up again.

Portage upgrade / repos.conf switch

Reference: FL-1761, Repository Configuration

As a part of upstream Portage changes, the upgrade is accompanied with some configuration file changes. Aside them, repository configuration is moved to repos.conf and the repository name becomes significant. Merging this with ports-2015 switch allows users to update the configuration in new format already.

Repository rename

Reference: FL-1801

Right now, the main repository inherits the name 'gentoo'. This is a bit confusing, considering that it is a modified Funtoo variant of the package tree. Changing the name to 'funtoo' would improve consistency and carry some bit of 'branding' into packages. Merging this into ports-2015 switch allows users to consciously update all repository references if necessary, and combines the change with necessity of specifying repository name in repos.conf.

Other possible changes

Filesystem structure reorganization

Since users will be required to clone the new repository, it may be desired to suggest some best practices for filesystem layout. This specifically includes separating ebuilds from distfiles & packages, and using a multi-repository layout. Historically, portage tree inspired by FreeBSD ports, located in /usr/portage>. This FHS layout is different in BSD and in Linux and we will probably move it elsewhere, see below for a suggestion and possible pros and cons.

Example layout suggested by mgorny:

  1. all repositories in /var/db/repos/${repo_name} (i.e. Funtoo repository in /var/db/repos/funtoo, and possible overlays as other directories in /var/db/repos),
  2. distfiles in /var/cache/portage/distfiles,
  3. binary packages in /var/cache/portage/packages.

Users can be recommended to use a separate filesystem that can handle small files efficiently for /var/db/repos, e.g. btrfs, reiserfs or possibly squashfs.


blog comments powered by Disqus