Difference between pages "Repository Configuration" and "FLOP:SquashFS sync"

(Difference between pages)
(Integrated this page to the Portage page.)
 
(describe the procedure & advantages)
 
Line 1: Line 1:
Per the request in FL-1923, I moved this information to the Portage page under the category "Future."
+
{{FLOP
[[Category:Portage]]
+
|Created on=2015/02/27
 +
|Summary=An alternate syncing method using SquashFS Images and deltas. Meant for stable systems where frequent updates aren't necessary, offering reduced bandwidth and disk space use, and improved speed.
 +
|Author=Mgorny,
 +
|Reference Bug=FL-2138
 +
}}
 +
== Problems with the current syncing method ==
 +
Currently the official syncing method for Funtoo systems is git. While it's quite good, it has a few downsides:
 +
* it places a lot of small files on the underlying filesystem — usually requiring a dedicated, small file-efficient filesystem,
 +
* the git metadata increases disk space use even further, especially after multiple syncs,
 +
* rare syncs imply transferring intermediate changes that may no longer be useful to end user.
 +
 
 +
While this is acceptable for hackers, it may be undesired for stable systems that are updated rather rarely and should be easy to set up.
 +
 
 +
== Suggested system ==
 +
=== Procedures ===
 +
Funtoo provides daily repository snapshots as SquashFS images. Aside to complete today's snapshot, it provides one or more deltas from ''n'' day-old snapshot to the today's snapshot. All of those files are provided via HTTP(S).
 +
 
 +
User fetches the today's snapshot on first sync. Afterwards, if he updates within the supported ''m''-day period, he only fetches a single delta and uses it to update the snapshot.
 +
 
 +
=== Advantages ===
 +
The most important advantages of this system are:
 +
* SquashFS allows to fit the whole repository in a single file, fitting any filesystem,
 +
* SquashFS is both fast and space-efficient,
 +
* deltas allow for fast and bandwidth-efficient updates.
 +
 
 +
=== Mirror impact ===
 +
A single complete SquashFS image is around ~100 MiB large. A single day delta is around 10 MiB, and raises to ~30 MiB for 3-week period and 50 MiB for >3 months.
 +
 
 +
{{FLOPFooter}}

Latest revision as of 14:37, February 27, 2015

Created on
2015/02/27
Original Author(s)
{{#set:Author={{ #show: User:Mgorny | ?Full name }}}}{{ #show: User:Mgorny | ?Full name }}
Status
Reference Bug
FL-2138

Funtoo Linux Optimization Proposal: SquashFS sync

An alternate syncing method using SquashFS Images and deltas. Meant for stable systems where frequent updates aren't necessary, offering reduced bandwidth and disk space use, and improved speed.

Problems with the current syncing method

Currently the official syncing method for Funtoo systems is git. While it's quite good, it has a few downsides:

  • it places a lot of small files on the underlying filesystem — usually requiring a dedicated, small file-efficient filesystem,
  • the git metadata increases disk space use even further, especially after multiple syncs,
  • rare syncs imply transferring intermediate changes that may no longer be useful to end user.

While this is acceptable for hackers, it may be undesired for stable systems that are updated rather rarely and should be easy to set up.

Suggested system

Procedures

Funtoo provides daily repository snapshots as SquashFS images. Aside to complete today's snapshot, it provides one or more deltas from n day-old snapshot to the today's snapshot. All of those files are provided via HTTP(S).

User fetches the today's snapshot on first sync. Afterwards, if he updates within the supported m-day period, he only fetches a single delta and uses it to update the snapshot.

Advantages

The most important advantages of this system are:

  • SquashFS allows to fit the whole repository in a single file, fitting any filesystem,
  • SquashFS is both fast and space-efficient,
  • deltas allow for fast and bandwidth-efficient updates.

Mirror impact

A single complete SquashFS image is around ~100 MiB large. A single day delta is around 10 MiB, and raises to ~30 MiB for 3-week period and 50 MiB for >3 months.


blog comments powered by Disqus