Setting up local distfiles mirror

This tutorial explains how to save bandwidth when serveral local computers need to pull updates from a single remote distfiles repository.

Use case

This tutorial will be about hosting a local mirror of funtoo/gentoo based portage distfiles with an apache based caching reverse proxy and multiple redundant upstream mirrors.

Following terms should be adapted

Terms	Definition
mirror.example.com	The local distfiles mirror host
localhost	Any local host
/var/cache/apache2/dist-files	Base path where to store the cached files
http://ftp.is.co.za/mirror/ftp.gentoo.org/	SA sample mirror, replace with one for your location
http://gentoo.mirror.ac.za/	SA sample mirror, replace with one for your location

Local mirror

Install apache with the required modules enabled

root # echo "www-servers/apache apache2_modules_cache apache2_modules_cache_disk apache2_modules_deflate apache2_modules_lbmethod_byrequests apache2_modules_proxy apache2_modules_proxy_balancer apache2_modules_proxy_ftp apache2_modules_proxy_http apache2_modules_slotmem_shm" >> /etc/portage/package.use
root # emerge -av www-servers/apache:2

Prepare directories for the cache storrage

root # mkdir /var/cache/apache2/dist-files
root # chown apache:apache /var/cache/apache2/dist-files

Apache daemon configuration

Add -C CACHE -D PROXY to APACHE2_OPTS for example

/etc/conf.d/apache2

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D CACHE -D PROXY"

Set the VirtualHost section of the default vhost config to:

/etc/apache2/vhosts.d/00_default_vhost.conf

<VirtualHost *:80>
	ServerName mirror.example.com
	Include /etc/apache2/vhosts.d/default_vhost.include

	CacheEnable disk /
	CacheRoot /var/cache/apache2/dist-files
	CacheDefaultExpire 63072000
	CacheMinExpire 63072000
	CacheMinFileSize 1
	CacheMaxFileSize 10000000000

	ProxyRequests Off

	#LogLevel debug

	<Proxy *>
		Require all granted
	</Proxy>

	ProxyPass /balancer-manager !
	ProxyPass /gentoo-distfiles balancer://mycluster/
	ProxyPassReverse /gentoo-distfiles http://ftp.is.co.za/mirror/ftp.gentoo.org/
	ProxyPassReverse /gentoo-distfiles http://gentoo.mirror.ac.za/
	<Proxy balancer://mycluster>
		BalancerMember http://ftp.is.co.za/mirror/ftp.gentoo.org/ route=node1
		BalancerMember http://gentoo.mirror.ac.za/ route=node2
	</Proxy>

	<Location /balancer-manager>
		SetHandler balancer-manager
	</Location>
	
	<IfModule mpm_peruser_module>
		ServerEnvironment apache apache
	</IfModule>
</VirtualHost>

This is for 2 South Africa local mirrors, you will need to pic the best for your location and replace the ProxyPassReverse and BalancerMember with a unique route=node<x> for each mirror chosen.

Service configuration

To start daemon at boot add apache2 to default runlevel

root # rc-update add apache2 default

To make changes start immediately just run rc

root # rc

Prevent the cache from becoming to big

Add the following to /etc/cron.daily/clean-cache.sh and set the required size, the sample 50G is for 50Gb

#!/bin/sh
htcacheclean -ntp /var/cache/apache2/dist-files/ -l 50G

make the script executable with:

root # chmod +x /etc/cron.daily/clean-cache.sh

Downstream Clients Settings

Set the GENTOO_MIRRORS variable for portage to the new mirror:

/etc/portage/make.conf

GENTOO_MIRRORS="http://mirror.example.com/gentoo-distfiles"

Debugging

Uncomment the #LogLevel debug in /etc/apache2/vhosts.d/00_default_vhost.conf and reload apache with:

root # /etc/init.d/apache2 reload

Follow /var/log/apache2/error.log for detail:

root # tail -f /var/log/apache2/error_log

Portage Distfiles Mirror