FLOP:SquashFS sync

Revision as of 14:37, February 27, 2015 by Mgorny (Talk | contribs) (describe the procedure & advantages)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Created on
Original Author(s)
{{#set:Author={{ #show: User:Mgorny | ?Full name }}}}{{ #show: User:Mgorny | ?Full name }}
Reference Bug

Funtoo Linux Optimization Proposal: SquashFS sync

An alternate syncing method using SquashFS Images and deltas. Meant for stable systems where frequent updates aren't necessary, offering reduced bandwidth and disk space use, and improved speed.

Problems with the current syncing method

Currently the official syncing method for Funtoo systems is git. While it's quite good, it has a few downsides:

  • it places a lot of small files on the underlying filesystem — usually requiring a dedicated, small file-efficient filesystem,
  • the git metadata increases disk space use even further, especially after multiple syncs,
  • rare syncs imply transferring intermediate changes that may no longer be useful to end user.

While this is acceptable for hackers, it may be undesired for stable systems that are updated rather rarely and should be easy to set up.

Suggested system


Funtoo provides daily repository snapshots as SquashFS images. Aside to complete today's snapshot, it provides one or more deltas from n day-old snapshot to the today's snapshot. All of those files are provided via HTTP(S).

User fetches the today's snapshot on first sync. Afterwards, if he updates within the supported m-day period, he only fetches a single delta and uses it to update the snapshot.


The most important advantages of this system are:

  • SquashFS allows to fit the whole repository in a single file, fitting any filesystem,
  • SquashFS is both fast and space-efficient,
  • deltas allow for fast and bandwidth-efficient updates.

Mirror impact

A single complete SquashFS image is around ~100 MiB large. A single day delta is around 10 MiB, and raises to ~30 MiB for 3-week period and 50 MiB for >3 months.

blog comments powered by Disqus