Automatic FS Snapshots - Revision history

Palica at 07:40, 14 November 2012

2012-11-14T07:40:59Z

154.20.16.224: typo

2012-11-13T16:36:23Z

typo

154.20.16.224: /* First idea */ - typos

2012-11-13T16:35:37Z

‎First idea: - typos

81.204.132.39: /* Known Linux distros with the concept */

2012-02-10T02:22:46Z

‎Known Linux distros with the concept

Hoverbear: Grammar Nazi

2011-11-11T22:31:31Z

Grammar Nazi

404 Error: /* Goal and context */

2011-08-25T00:44:06Z

‎Goal and context

404 Error: /* Goal and context */

2011-08-24T23:57:51Z

‎Goal and context

404 Error: Created page with "= Goal and context = Modern filesystem like BTRFS and ZFS supports a feature known as snapshot (some of them like NILFS2 uses that capability in a even more aggressive manner). ..."

2011-08-24T23:56:37Z

Created page with "= Goal and context = Modern filesystem like BTRFS and ZFS supports a feature known as snapshot (some of them like NILFS2 uses that capability in a even more aggressive manner). ..."

New page

= Goal and context =

Modern filesystem like BTRFS and ZFS supports a feature known as snapshot (some of them like NILFS2 uses that capability in a even more aggressive manner). Basically, a snapshot is nothing more than coherent "photograph" of a filesystem structures and data blocks pointed by them thus enabling the system administrator to see a filesystem in its *exact* state at the time the snapshot was taken even some files have been deleted/changed/created since the snapshot creation.

Filesystems snapshotting is not a new feature as it is known for quite a long time in the Linux world especially by LVM users. However, one of the areas where filesystems like BTRFS shines is it uses a technique known as Copy-on-Write: while a LVM snapshot physically replicates the stripes (LVM is disconnected from the filesystems lying in the LVM logical volumes and can make no assumptions), CoW filesystems like BTRFS are clever enough to avoid unnecessary data blocks duplication. Thus, at the exact time a snapshot is taken there is no difference between a filesystem and its snapshot meaning the snapshot size is very small. As the time goes on and data blocks are modified, the filesystem will detect alterations attempt and will duplicate all the necessary data blocks to preserve the original ones. On the other side, old and obsolete snapshots must be deleted from time to time to free up the data blocks that have been "locked in the past".

Present proposal pretends to be the "big picture" of integrating snapshotting capabilities of modern CoW filesystems like BTRFS/ZFS in portage. Such an integration will enable sysamdins to easily recover from a disaster coming from a clunky update (or worse, a silent and catatrophic libpng-like update). For the project credibility, it mitigiates the degree of an major issue coming out of the blue. For Funtoo developers, the gain is they do not have to replicate their live filesystem in a chroot or play with virtual machines or jailed tests environments of any kind. In the case of something go wrong, it is simple to do a filesystem rollback just as if nothing has ever happened.

@@ Line 43: / Line 43: @@
 = Known Linux distros with the concept =
-So far Fedora is working on integrating BTRFS with Yum.
+* So far Fedora is working on integrating BTRFS with Yum. [https://fedoraproject.org/wiki/Features/SystemRollbackWithBtrfs]
-Ubuntu - [http://packages.ubuntu.com/us/oneiric/apt-btrfs-snapshot]
+* Ubuntu - [http://packages.ubuntu.com/us/oneiric/apt-btrfs-snapshot]
 [[Category:Internals]]

@@ Line 5: / Line 5: @@
 Filesystems snapshotting is not a new feature as it is known for quite a long time in the Linux world especially by LVM users. However, one of the areas where filesystems like BTRFS shines is it uses a technique known as Copy-on-Write: while a LVM snapshot physically replicates the stripes (LVM is disconnected from the filesystems lying in the LVM logical volumes and can make no assumptions), CoW filesystems like BTRFS are clever enough to avoid unnecessary data blocks duplication. Thus, at the exact time a snapshot is taken there is no difference between a filesystem and its snapshot meaning the snapshot size is very small. As the time goes on and data blocks are modified, the filesystem will detect alterations attempt and will duplicate all the necessary data blocks to preserve the original ones. On the other side, old and obsolete snapshots must be  deleted from time to time to free up the data blocks that have been "locked in the past".
-Present proposal pretends to be the "big picture" of integrating snapshotting capabilities of modern CoW filesystems like BTRFS/ZFS in portage. Such an integration will enable sysamdins to easily recover from a disaster coming from a clunky update (or worse, a silent and catatrophic libpng-like update). For the project credibility, it mitigiates the degree of an major issue coming out of the blue. For Funtoo developers, the gain is they do not have to replicate their live filesystem in a chroot or play with virtual machines or jailed tests environments of any kind. In the case of something go wrong, it is simple to do a filesystem rollback just as if nothing has ever happened.
+Present proposal pretends to be the "big picture" of integrating snapshotting capabilities of modern CoW filesystems like BTRFS/ZFS in portage. Such an integration will enable sysamdins to easily recover from a disaster coming from a clunky update (or worse, a silent and catastrophic libpng-like update). For the project credibility, it mitigiates the degree of an major issue coming out of the blue. For Funtoo developers, the gain is they do not have to replicate their live filesystem in a chroot or play with virtual machines or jailed tests environments of any kind. In the case of something go wrong, it is simple to do a filesystem rollback just as if nothing has ever happened.
 = First idea =

@@ Line 9: / Line 9: @@
 = First idea =
-Quick functional description: the sysadmin want to install/update/delete/clean up dependencies and use the appropriate commande (emerge/emerge -u/emerge -C/emerge --depclean...):
+Quick functional description: the sysadmin want to install/update/delete/clean up dependencies and use the appropriate command (emerge/emerge -u/emerge -C/emerge --depclean...):
 * The command starts by doing a snapshot of system directories and then installs or remove all of the specified packages/dependencies (transactional operation).
 * If something goes wrong and the user resume the process (emerge --skip-first / emerge --resume) no snapshot is taken and the process goes on as it actually does
 * If the  something goes wrong and the user then chooses to manually emerge some packages by hand instead of doing, let's say, emerge -uaDN @world, another snapshot is taken.
-This garantee that an "image" of the filesystem is kept before each transaction (installation/removal of several packages in a single portage command). When the sysadmin is happy, he removes the now unneeded snapshots by its own.
+This guarantee that an "image" of the filesystem is kept before each transaction (installation/removal of several packages in a single portage command). When the sysadmin is happy, he removes the now unneeded snapshots by its own.
 = Behavior control =

← Older revision		Revision as of 02:22, 10 February 2012
Line 44:		Line 44:

	So far Fedora is working on integrating BTRFS with Yum.		So far Fedora is working on integrating BTRFS with Yum.
		+	Ubuntu - [http://packages.ubuntu.com/us/oneiric/apt-btrfs-snapshot]

	[[Category:Internals]]		[[Category:Internals]]

← Older revision		Revision as of 00:44, 25 August 2011
Line 6:		Line 6:

	Present proposal pretends to be the "big picture" of integrating snapshotting capabilities of modern CoW filesystems like BTRFS/ZFS in portage. Such an integration will enable sysamdins to easily recover from a disaster coming from a clunky update (or worse, a silent and catatrophic libpng-like update). For the project credibility, it mitigiates the degree of an major issue coming out of the blue. For Funtoo developers, the gain is they do not have to replicate their live filesystem in a chroot or play with virtual machines or jailed tests environments of any kind. In the case of something go wrong, it is simple to do a filesystem rollback just as if nothing has ever happened.		Present proposal pretends to be the "big picture" of integrating snapshotting capabilities of modern CoW filesystems like BTRFS/ZFS in portage. Such an integration will enable sysamdins to easily recover from a disaster coming from a clunky update (or worse, a silent and catatrophic libpng-like update). For the project credibility, it mitigiates the degree of an major issue coming out of the blue. For Funtoo developers, the gain is they do not have to replicate their live filesystem in a chroot or play with virtual machines or jailed tests environments of any kind. In the case of something go wrong, it is simple to do a filesystem rollback just as if nothing has ever happened.
		+
		+	= First idea =
		+
		+	Quick functional description: the sysadmin want to install/update/delete/clean up dependencies and use the appropriate commande (emerge/emerge -u/emerge -C/emerge --depclean...):
		+	* The command starts by doing a snapshot of system directories and then installs or remove all of the specified packages/dependencies (transactional operation).
		+	* If something goes wrong and the user resume the process (emerge --skip-first / emerge --resume) no snapshot is taken and the process goes on as it actually does
		+	* If the something goes wrong and the user then chooses to manually emerge some packages by hand instead of doing, let's say, emerge -uaDN @world, another snapshot is taken.
		+
		+	This garantee that an "image" of the filesystem is kept before each transaction (installation/removal of several packages in a single portage command). When the sysadmin is happy, he removes the now unneeded snapshots by its own.
		+
		+	= Behavior control =
		+
		+	As the Funtoo philosophy wants, the snapshotting capabilities integration in portage is always optional and the behaviour is governed via FEATURES (e.g. FEATURES=".... fs-snapshots"). As we can't predict what the future is made of, it is necessary to have a flexible control on the how the filesystem manipulation tools used to produce snapshots. This can be done with environment variables like:
		+
		+	BTRFS_PORTAGE_SNAPSHOT_CMD="btrfs subvolume snapshot "
		+	ZFS_PORTAGE_SNAPSHOT_CMD="zfs snapshot "
		+
		+	By default, portage will use a YYYYMMDDTHHMMSS for the snapshot name thus doing something like:
		+
		+	* btrfs subvolume snapshot /mountpoint /root/mountpoint-YYYYMMDDTHHMMSS (in the case of / only the YYYYMMDDTHHMMSS portion is kept)
		+	* zfs snapshot pool/dataset@YYYYMMDDTHHMMSS (if the dataset part does not exists only the YYYYMMDDTHHMMSS portion is kept)
		+
		+	== Approach #1: manual configuration ==
		+
		+	Portage scans a file (let says /etc/portage/fssnaps to be compliant with coming 1.0 profiles and unified configuration files) which describes what are the subvolumes to snapshot before doing a transaction:
		+
		+	<pre>
		+	/
		+	/usr
		+	</pre>
		+
		+	== Approach #2: automatic configuration ==
		+
		+	Portage scans what lies in /etc/fstab that has been defined as btrfs or zfs or anything of the same kind. If an entry describe something is located in a "well-known" directory (e.g. /lib, /usr ....) it is included in the subvolumes list that have been to be snapshotted. To allow a bit of flexibility, a list of exclusion can be specied via a variable like EXEMPT_FROM_SNAPHOT="/lib /var". This is dangerous but could be needed in some contexts.

	= Known Linux distros with the concept =		= Known Linux distros with the concept =

@@ Line 1: / Line 1: @@
 = Goal and context =
-Modern filesystem like BTRFS and ZFS supports a feature known as snapshot (some of them like NILFS2 uses that capability in a even more aggressive manner). Basically, a snapshot is nothing more than coherent "photograph" of a filesystem structures and data blocks pointed by them thus enabling the system administrator to see a filesystem in its *exact* state at the time the snapshot was taken even some files have been deleted/changed/created since the snapshot creation.
+Modern filesystem like BTRFS and ZFS supports a feature known as snapshot (some of them like NILFS2 uses that capability in a even more aggressive manner). Basically, a snapshot is nothing more than coherent "photograph" of a filesystem structures and data blocks pointed by them thus enabling the system administrator to see a filesystem in its *exact* state at the time the snapshot was taken even if some files have been deleted/changed/created since the snapshot creation.
 Filesystems snapshotting is not a new feature as it is known for quite a long time in the Linux world especially by LVM users. However, one of the areas where filesystems like BTRFS shines is it uses a technique known as Copy-on-Write: while a LVM snapshot physically replicates the stripes (LVM is disconnected from the filesystems lying in the LVM logical volumes and can make no assumptions), CoW filesystems like BTRFS are clever enough to avoid unnecessary data blocks duplication. Thus, at the exact time a snapshot is taken there is no difference between a filesystem and its snapshot meaning the snapshot size is very small. As the time goes on and data blocks are modified, the filesystem will detect alterations attempt and will duplicate all the necessary data blocks to preserve the original ones. On the other side, old and obsolete snapshots must be  deleted from time to time to free up the data blocks that have been "locked in the past".

← Older revision		Revision as of 23:57, 24 August 2011
Line 6:		Line 6:

	Present proposal pretends to be the "big picture" of integrating snapshotting capabilities of modern CoW filesystems like BTRFS/ZFS in portage. Such an integration will enable sysamdins to easily recover from a disaster coming from a clunky update (or worse, a silent and catatrophic libpng-like update). For the project credibility, it mitigiates the degree of an major issue coming out of the blue. For Funtoo developers, the gain is they do not have to replicate their live filesystem in a chroot or play with virtual machines or jailed tests environments of any kind. In the case of something go wrong, it is simple to do a filesystem rollback just as if nothing has ever happened.		Present proposal pretends to be the "big picture" of integrating snapshotting capabilities of modern CoW filesystems like BTRFS/ZFS in portage. Such an integration will enable sysamdins to easily recover from a disaster coming from a clunky update (or worse, a silent and catatrophic libpng-like update). For the project credibility, it mitigiates the degree of an major issue coming out of the blue. For Funtoo developers, the gain is they do not have to replicate their live filesystem in a chroot or play with virtual machines or jailed tests environments of any kind. In the case of something go wrong, it is simple to do a filesystem rollback just as if nothing has ever happened.
		+
		+	= Known Linux distros with the concept =
		+
		+	So far Fedora is working on integrating BTRFS with Yum.
		+
		+	[[Category:Internals]]
		+	[[Category:Labs]]
		+	[[Category:Portage]]
		+	[[Category:FLOP]]