Difference between pages "Portage Git Mirror" and "Package:Varnish"

From Funtoo
(Difference between pages)
Jump to navigation Jump to search
 
m (add homepage link)
 
Line 1: Line 1:
=Setting up local git mirror =
{{Ebuild
|Summary=Varnish is a state-of-the-art, high-performance HTTP accelerator
|CatPkg=www-servers/varnish
|Maintainer=
|Homepage=https://www.varnish-cache.org/
}}
'''Varnish''' is a webcache & http accelerator.  Varnish will either serve cached content, or retireve content from the server, and cache it.  Varnish cache will reduce I/O pressure from webservers.
 
==Install==
===Emerge===
 
Install {{Package|www-servers/varnish}}:
<console>###i## emerge www-servers/varnish</console>


This tutorial explains how to save bandwidth when several local computers need to pull updates from a single remote git repository.
==Configuration==


== Use case ==
{{note|as your varnish is local to your server, your server needs to be aware that it is behind a proxy, and configured for [http://en.wikipedia.org/wiki/X-Forwarded-For x-forwarded-for] or similar to fetch real users ip addresses instead of displaying 127.0.0.1 or localhost.}}


This tutorial will be about hosting a local mirror of funtoo git based portage tree.
Configuration is controlled by /etc/varnish/default.vcl & /etc/conf.d/varnishd


Following terms should be adapted
{{file|name=/etc/varnish/default.vcl|desc=varnish configuration file|body=
vcl 4.0;
backend default {
    .host = "127.0.0.1";
    .port = "8080";
}
}}


{{TableStart}}
{{file|name=/etc/conf.d/varnishd|desc=varnish configuration file|body=
<tr class="header">
VARNISHD="/usr/sbin/varnishd"
<th align="left">Terms</th>
VARNISHADM="/usr/bin/varnishadm"
<th align="left">Definition</th>
CONFIGFILE="/etc/varnish/default.vcl"
</tr>
VARNISHD_OPTS="-a 127.0.0.1:80"
<tr class="odd">
VARNISHD_OPTS="${VARNISHD_OPTS} -u varnish -g varnish"
<td align="left">git.lan</td>
}}
<td align="left">The git-daemon local mirror host</td>
</tr>
<tr class="even">
<td align="left">localhost</td>
<td align="left">Any local host</td>
</tr>
<tr class="odd">
<td align="left">nobody</td>
<td align="left">Owner user of .git files</td>
</tr>
<tr class="even">
<td align="left">/home/git-mirrors</td>
<td align="left">Base path of git-daemon</td>
</tr>
{{TableEnd}}


== Local mirror ==
Varnish will fetch data from localhost:8080 and serve accelerated proxy data on localhost:80


==== Prepare directories and get portage tree====
=== c10k ===
<console>
For 10,000 concurrent connections (or not) a few configuration settings control the power of varnish. If you're having varnish directly serve to the outside world, dial back concurrency to say 50 or 100 connections per ip. You must take into account corporations, and universities hammering several connections from a singular ip. c10k is useful information for if varnish is behind a load balancer such as pound, nginx, or tengine, and all requests are internal and local.
###i## mkdir /home/git-mirrors
 
###i## chown nobody /home/git-mirrors
{{file|name=/etc/conf.d/varnishd|desc=varnish concurrency settings|body=
###i## su -s /bin/sh nobody
VARNISHD_OPTS="-a 127.0.0.1:80 -p thread_pool_min=20 -p thread_pool_max=1000 -p thread_pool_add_delay=2 -s malloc,700M"
$##bl## cd /home/git-mirrors
$##bl## git clone --mirror --bare git://github.com/funtoo/ports-2012.git portage.git
</console>
For a security reason we use a nobody user .
==== git-daemon configuration====
{{file|name=/etc/conf.d/git-daemon|desc=|body=
GITDAEMON_OPTS="--syslog --verbose --enable=receive-pack --export-all"
GITDAEMON_OPTS="${GITDAEMON_OPTS} --base-path=/home/git-mirrors /home/git-mirrors --interpolated-path=/home/git-mirrors"
GIT_USER=nobody
GIT_GROUP=nobody
}}
}}


====Service configuration====
{{file|name=/etc/varnish/default.vcl|desc=varnish concurrency settings|body=
To start daemon with a mirror machine boot add <code>git-daemon</code> to default runlevel
backend default {
<console>
    .host = "127.0.0.1";
###i## rc-update add git-daemon default
    .port = "8080";
</console>
    .connect_timeout = 600s;
To make changes start immediately just run <code>rc</code>
    .first_byte_timeout = 600s;
<console>
    .between_bytes_timeout = 600s;
###i## rc
    .max_connections = 10000;
</console>
    # .port = "80" led to issues with competing for the port with apache.
}
}}
 
== BootService ==
 
To start varnish immediately:
<console>###i## rc-service varnishd start</console>
 
To start varnish at boot:
<console>###i## rc-update add varnishd default</console>
 
== Verification ==
To verify that your traffic is going through varnish, and see if you're getting hits or misses:
<console>$##i## curl -I http://www.funtoo.org/Welcome</console>


=== Pull from remote ===
== Benchmarking ==
{{package|app-admin/apache-tools}} apache benchmark can show the power of varnish.  The examples shown are running 500 requests with concurrency of 100 hits.


Add the following to <code>/etc/cron.daily/funtoo-sync.sh</code>:


<pre>
ab against a 3 worker cluster mode puma server
#!/bin/sh
<console>###i## ab -n 500 -c 100 http://127.0.0.1:3000/index.html | grep Request</console>
cd /home/git-mirrors/portage.git
Requests per second:    110.92 [#/sec] (mean)
su nobody -s "/bin/sh" -c "git fetch"
</pre>


== Cloning from local git-daemon ==


Local clone from <code>git.lan</code>:
ab against the same server served through varnish
<console>
<console>###i## ab -n 500 -c 100 http://127.0.0.1/index.html | grep Request</console>
###i## mv /usr/portage /usr/portage.old
Requests per second:    10268.42 [#/sec] (mean)
###i## git clone git://git.lan/portage.git /usr/portage
###i## cd /usr/portage
###i## git checkout funtoo.org
</console>


== Downstream Clients Settings ==
== SSL support ==
machines previously syncing to upstream mirrors need their portage moved:
Varnish does not support ssl.  There are packages to get around this limitation:
<console>###i## mv /usr/portage /usr/portage.old</console>
* {{package|net-misc/stunnel}}
* {{package|www-servers/pound}}


{{file|name=/etc/portage/make.conf|lang=|desc=define client sync source for emerge --sync|body=
== Media ==
SYNC="git://git.lan/portage.git"}}
{{#widget:YouTube16x9|id=JEF6_XC-2ZU}}


[[Category:HOWTO]]
{{EbuildFooter}}

Revision as of 02:45, December 19, 2014

Varnish

   Tip

We welcome improvements to this page. To edit this page, Create a Funtoo account. Then log in and then click here to edit this page. See our editing guidelines to becoming a wiki-editing pro.

Varnish is a webcache & http accelerator. Varnish will either serve cached content, or retireve content from the server, and cache it. Varnish cache will reduce I/O pressure from webservers.

Install

Emerge

Install www-servers/varnish: 

root # emerge www-servers/varnish

Configuration

   Note

as your varnish is local to your server, your server needs to be aware that it is behind a proxy, and configured for x-forwarded-for or similar to fetch real users ip addresses instead of displaying 127.0.0.1 or localhost.

Configuration is controlled by /etc/varnish/default.vcl & /etc/conf.d/varnishd

   /etc/varnish/default.vcl - varnish configuration file
vcl 4.0;
backend default {
    .host = "127.0.0.1";
    .port = "8080";
}
   /etc/conf.d/varnishd - varnish configuration file
VARNISHD="/usr/sbin/varnishd"
VARNISHADM="/usr/bin/varnishadm"
CONFIGFILE="/etc/varnish/default.vcl"
VARNISHD_OPTS="-a 127.0.0.1:80"
VARNISHD_OPTS="${VARNISHD_OPTS} -u varnish -g varnish"

Varnish will fetch data from localhost:8080 and serve accelerated proxy data on localhost:80

c10k

For 10,000 concurrent connections (or not) a few configuration settings control the power of varnish. If you're having varnish directly serve to the outside world, dial back concurrency to say 50 or 100 connections per ip. You must take into account corporations, and universities hammering several connections from a singular ip. c10k is useful information for if varnish is behind a load balancer such as pound, nginx, or tengine, and all requests are internal and local.

   /etc/conf.d/varnishd - varnish concurrency settings
VARNISHD_OPTS="-a 127.0.0.1:80 -p thread_pool_min=20 -p thread_pool_max=1000 -p thread_pool_add_delay=2 -s malloc,700M"
   /etc/varnish/default.vcl - varnish concurrency settings
backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .max_connections = 10000;
    # .port = "80" led to issues with competing for the port with apache.
}

BootService

To start varnish immediately: 

root # rc-service varnishd start

To start varnish at boot: 

root # rc-update add varnishd default

Verification

To verify that your traffic is going through varnish, and see if you're getting hits or misses: 

user $ curl -I http://www.funtoo.org/Welcome

Benchmarking

app-admin/apache-tools apache benchmark can show the power of varnish. The examples shown are running 500 requests with concurrency of 100 hits.


ab against a 3 worker cluster mode puma server 

root # ab -n 500 -c 100 http://127.0.0.1:3000/index.html | grep Request

Requests per second: 110.92 [#/sec] (mean)


ab against the same server served through varnish 

root # ab -n 500 -c 100 http://127.0.0.1/index.html | grep Request

Requests per second: 10268.42 [#/sec] (mean)

SSL support

Varnish does not support ssl. There are packages to get around this limitation:

  • No results
  • No results

Media


Retrieved from "https://www.funtoo.org/index.php?title=Portage_Git_Mirror&oldid=7548"