|
|
| Line 1: |
Line 1: |
| {{Article
| | Virtual packages are special packages that correspond to a feature that can be satisfied by one or more package(s). This Wiki page aims to describe when and how to use them correctly, and what are their implications. |
| |Subtitle=Storage management magic with Logical Volume Management
| |
| |Author=Drobbins
| |
| }}
| |
| == LVM intro ==
| |
|
| |
|
| In this series, I'm going to show you how to install and use the new Logical Volume Management support built-in to the Linux kernel. If you've never used a form of LVM before, you're in for a treat; it's a wonderful technology. Before we actually get LVM up and running, I'm going to explain exactly what it is and how it works. Then, we'll be ready to test out LVM and get the most out of it.
| | == Virtual packages, metapackages and package sets == |
| | Virtual packages, metapackages and package sets are similar concepts. However, they have a few important differences that make them fit for different use cases. |
|
| |
|
| If you're like me, then your experience with UNIX and Linux began on a PC platform, rather than on large, commercial UNIX servers and workstations. On the basic PC, we've always had to deal with partitioning our hard drives. PC people are generally well-acquainted with tools such as fdisk, which are used to create and delete primary and extended partitions on hard disks. Hard disk partitioning is an annoying but accepted part of the process of getting an operating system up and running.
| | Virtual packages and metapackages are regular Funtoo packages (ebuilds) that install no files. Instead, they cause other packages to be installed by specifying them in their runtime dependencies. They can both be used in any context valid for regular packages. They can have multiple versions, slots and USE flags. They have to be located in an active repository, and once there they can be installed and uninstalled like regular packages. |
|
| |
|
| Hard drive partitioning can be annoying because to do a good job you really need to accurately estimate how much space you'll need for each partition. If you make a poor estimation, your Linux system could possibly be crippled -- to fix the problem, it's possible that you might even need to perform a full system backup, wipe your hard drives clean, and then restore all your data to a new (and presumably better) partition layout. Ick! These are exactly the kinds of situations that sysadmins try their best to avoid in the first place.
| | Package sets are not packages but special atoms supported by Portage. Package sets can only specify other packages, either via a static list or dynamically (e.g. via running Python code that determines the package list). Package sets can't be versioned and don't have USE flags. Package sets can be used alongside packages in emerge commands and other package sets but they can't be referenced inside regular packages. Package sets can be installed into user's system, located in repositories or created by user in Portage configuration. |
|
| |
|
| While partitions were once static storage regions, thankfully, we now have a proliferation of PC repartitioning tools (PowerQuest's Partition Magic product is one of the most popular). These tools allow you to boot your system with a special disk and dynamically resize your partitions and filesystems. Once you reboot, you have newly resized partitions, hopefully getting you out of your storage crunch. These partition resizing tools are great and solve the problem storage management for some. But are they perfect? Not exactly.
| | Virtual packages represent a commonly used feature that can be provided by multiple different providers. Virtuals provide a convenient way of specifying all possible alternatives without having to update multiple ebuilds. |
|
| |
|
| Tools like Partition Magic are great for workstations, but aren't really adequate for servers. First of all, they require you to reboot your system. This is something most sysadmins desperately try to avoid doing. What if you simply can't reboot your machine every time your storage needs change, such as if your storage needs change dramatically on a weekly basis? What happens if you need to expand a filesystem so that it spans more than one hard drive, or what do you do if you need to dynamically expand or shrink a volume's storage capacity while allowing Apache to continue to serve Web pages? In a highly available, dynamic environment, a basic partition resizer just won't work. For these and other situations, Logical Volume Management is an excellent (if not perfect) solution.
| | Metapackages and package sets are used to represent lists of packages that user may want to install together. They provide a convenience for users, e.g. providing a shortcut to install all packages comprising a desktop environment. |
|
| |
|
| == Enter LVM == | | == When virtual packages can be used? == |
| | For virtual package ebuild to work correctly, the two following requirements must be met: |
| | # the virtual providers must be interchangeable at runtime with no consequences to the reverse dependencies. In other words, installing another provider and removing the currently used provider must not cause any breakage or require reverse dependencies to be rebuilt. |
| | # Reverse dependencies need to have consistent, predictable requirements for the alternatives. In other words, the packages must not require a very specific versions of the alternatives. |
|
| |
|
| Now, let's take a look at how LVM solves these problems. To create an LVM logical volume, we follow a three-step process. First, we need to select the physical storage resources that are going to be used for LVM. Typically, these are standard partitions but can also be Linux software RAID volumes that we've created. In LVM terminology, these storage resources are called "physical volumes". Our first step in setting up LVM involves properly initializing these partitions so that they can be recognized by the LVM system. This involves setting the correct partition type if we're adding a physical partition, and running the pvcreate command.
| | Virtuals can not be used if the underlying packages don't provide binary compatibility at least between predictable range of versions. |
|
| |
|
| Once we have one or more physical volumes initialized for use by LVM, we can move on to step two -- creating a volume group. You can think of a volume group as a pool of storage that consists of one or more physical volumes. While LVM is running, we can add physical volumes to the volume group or even remove them. However, we can't mount or create filesystems on a volume group directly. Instead, we can tell LVM to create one or more "logical volumes" using our volume group storage pool:
| | == Common uses for virtual packages == |
| | === System components and services === |
| | Example: ''virtual/service-manager'' |
|
| |
|
| [[File:l-lvm-1.gif|frame|A volume group is created out of physical volumes]]
| | One of the common uses for virtuals is to define abstract ''system services''. Those virtuals are not very specific on how those services are provided. They are mostly intended to be used in the @system package set, to ensure that the user system doesn't lack key components such as a service manager or a package manager. |
|
| |
|
| Creating an LVM logical volume is really easy, and once it's created we can go ahead and put a filesystem on it, mount it, and start using the volume to store our files. To create a logical volume, we use the lvcreate command, specifying the name of our new volume, the size we'd like the volume to be, and the volume group that we'd like this particluar logical volume to be part of. The LVM system will then allocate storage from the volume group we specify and create our new volume, which is now ready for use. Once created, we can put an ext2 or ReiserFS filesystem on it, mount it, and use it as we like.
| | The providers for this kind of virtuals do not have to meet any specific requirements except for having a particular function. In particular, there's no requirement for common configuration or provided executables. The user is responsible for ensuring that the installed implementation is set up and working. |
|
| |
|
| [[File:l-lvm-2.gif|frame|Creating two logical volumes from our existing volume group]]
| | === Tools provided by multiple packages === |
| | Example: ''virtual/eject'' |
|
| |
|
| == Extents ==
| | This kind of virtuals is used when multiple packages may provide tools with the same names. The virtual is used in packages that rely on those tools being present, in particular when the tools are used at build-time of the package or are called by package's scripts (executables). |
|
| |
|
| Behind the scenes, the LVM system allocates storage in equal-sized "chunks", called extents. We can specify the particular extent size to use at volume group creation time. The size of an extent defaults to 4Mb, which is perfect for most uses. One of the beauties of LVM is that the physical storage locations of the extents used for one of our logical volumes (in other words, what disk they're stored on) can be dynamically changed while our logical volume is mounted and in use! The LVM system ensures that our logical volumes continue to operate perfectly while allowing the administrator to physically change where everything is stored.
| | While the tools don't necessarily need to be fully compatible, they need to have a common basic usage. In particular, when a tool from one provider is replaced by a tool from another, the reverse dependencies must remain in working state, with no need for rebuilds or configuration adjustments. |
| | |
| Of course, since everything is created out of equally-sized extents, it's really easy to allocate some additional extents for an already-existing logical volume -- in other words, dynamically "grow" the volume:
| |
| | |
| [[File:l-lvm-3.gif|frame|Adding additional extents from our volume group, expanding the size of our logical volume.]]
| |
| | |
| Once the logical volume has been expanded, you can then expand your ext2 or ReiserFS filesystem to take advantage of this new space. If you use a program such as resize_reiserfs, this filesystem expansion can also happen while the volume is mounted and being used! Truly amazing -- with LVM and online filesystem expansion utilties, it's no longer necessary to reboot your system or even drop to runlevel 1 to change your storage configuration.
| |
| | |
| The only time you need to shut down your system is when you need to add new physical disks. Once new disks have been added, you then can add these new physical volumes to your volume group(s) to create a fresh supply of extents.
| |
| | |
| == Setting up LVM ==
| |
| | |
| OK, let's get LVM installed. LVM consists of two parts: a kernel part and a suite of user-space tools. You may already have LVM support available on your system, and if not, it's a simple matter to install the appropriate tools using your distribution's package manager. On Gentoo or Funtoo Linux, this is done as follows:
| |
| | |
| <console>
| |
| # ##i##emerge sys-fs/lvm2
| |
| </console>
| |
| | |
| If you compiled your kernel manually, you'll want to reconfigure and compile your kernel so that LVM support is enabled.
| |
| | |
| <console>
| |
| # ##i##cd /usr/src/linux
| |
| # ##i##make menuconfig
| |
| </console>
| |
| | |
| You'll find the LVM options under the "Multiple devices driver support (RAID and LVM)" section, under "Device Drivers":
| |
| | |
| {{kernelop|title=Device Drivers,Multiple devices driver support (RAID and LVM)|desc=
| |
| --- Multiple devices driver support (RAID and LVM)
| |
| {M} RAID support
| |
| <M> Linear (append) mode
| |
| <M> RAID-0 (striping) mode
| |
| -M- RAID-1 (mirroring) mode
| |
| -M- RAID-10 (mirrored striping) mode
| |
| -M- RAID-4/RAID-5/RAID-6 mode
| |
| <M> Multipath I/O support
| |
| <M> Faulty test module for MD
| |
| <M> Block device as cache
| |
| [ ] Bcache debugging
| |
| [ ] Debug closures
| |
| <M> Device mapper support
| |
| [ ] Device mapper debugging support
| |
| [ ] Keep stack trace of persistent data block lock holders
| |
| <M> Crypt target support
| |
| <M> Snapshot target
| |
| <M> Thin provisioning target
| |
| <M> Cache target (EXPERIMENTAL)
| |
| <M> MQ Cache Policy (EXPERIMENTAL)
| |
| <M> Cleaner Cache Policy (EXPERIMENTAL)
| |
| <M> Era target (EXPERIMENTAL)
| |
| <M> Mirror target
| |
| <M> Mirror userspace logging
| |
| <M> RAID 1/4/5/6/10 target
| |
| <M> Zero target
| |
| <M> Multipath target
| |
| <M> I/O Path Selector based on the number of in-flight I/Os
| |
| <M> I/O Path Selector based on the service time
| |
| <M> I/O delaying target
| |
| [*] DM uevents
| |
| <M> Flakey target
| |
| <M> Verity target support
| |
| <M> Switch target support (EXPERIMENTAL)
| |
| }}
| |
| | |
| I recommend enabling all features. Also ensure that LVM is enabled for your initramfs. Remember that if you are putting any main filesystems on LVM, you will need LVM compiled into your kernel, rather than a module, or you'll need an LVM-aware initramfs.
| |
| | |
| Also be sure to enable any necessary startup scripts to initialize LVM. On many distributions, including Funtoo Linux, this is done for you, provided that sufficient kernel support is available. The basic commands that these scripts will run are the following, at boot:
| |
| | |
| <console>
| |
| /sbin/vgscan
| |
| /sbin/vgchange -a y
| |
| </console>
| |
| | |
| These lines will scan for all available volume groups and activate them. At shutdown, something like this will run:
| |
| | |
| <console>
| |
| /sbin/vgchange -a n
| |
| </console>
| |
| | |
| While this stuff is handled for you automatically, if you ever boot from a rescue CD or USB stick, you may need to type {{c|vgscan}} and {{c|vgchange -a y}} as root before your logical volumes are available for use.
| |
| | |
| That's it for this article. Next article, I'll show you how to create your own logical volumes and unleash the power of LVM. I'll see you then!
| |
| {{ArticleFooter}}
| |
Virtual packages are special packages that correspond to a feature that can be satisfied by one or more package(s). This Wiki page aims to describe when and how to use them correctly, and what are their implications.
Virtual packages, metapackages and package sets
Virtual packages, metapackages and package sets are similar concepts. However, they have a few important differences that make them fit for different use cases.
Virtual packages and metapackages are regular Funtoo packages (ebuilds) that install no files. Instead, they cause other packages to be installed by specifying them in their runtime dependencies. They can both be used in any context valid for regular packages. They can have multiple versions, slots and USE flags. They have to be located in an active repository, and once there they can be installed and uninstalled like regular packages.
Package sets are not packages but special atoms supported by Portage. Package sets can only specify other packages, either via a static list or dynamically (e.g. via running Python code that determines the package list). Package sets can't be versioned and don't have USE flags. Package sets can be used alongside packages in emerge commands and other package sets but they can't be referenced inside regular packages. Package sets can be installed into user's system, located in repositories or created by user in Portage configuration.
Virtual packages represent a commonly used feature that can be provided by multiple different providers. Virtuals provide a convenient way of specifying all possible alternatives without having to update multiple ebuilds.
Metapackages and package sets are used to represent lists of packages that user may want to install together. They provide a convenience for users, e.g. providing a shortcut to install all packages comprising a desktop environment.
When virtual packages can be used?
For virtual package ebuild to work correctly, the two following requirements must be met:
- the virtual providers must be interchangeable at runtime with no consequences to the reverse dependencies. In other words, installing another provider and removing the currently used provider must not cause any breakage or require reverse dependencies to be rebuilt.
- Reverse dependencies need to have consistent, predictable requirements for the alternatives. In other words, the packages must not require a very specific versions of the alternatives.
Virtuals can not be used if the underlying packages don't provide binary compatibility at least between predictable range of versions.
Common uses for virtual packages
System components and services
Example: virtual/service-manager
One of the common uses for virtuals is to define abstract system services. Those virtuals are not very specific on how those services are provided. They are mostly intended to be used in the @system package set, to ensure that the user system doesn't lack key components such as a service manager or a package manager.
The providers for this kind of virtuals do not have to meet any specific requirements except for having a particular function. In particular, there's no requirement for common configuration or provided executables. The user is responsible for ensuring that the installed implementation is set up and working.
Tools provided by multiple packages
Example: virtual/eject
This kind of virtuals is used when multiple packages may provide tools with the same names. The virtual is used in packages that rely on those tools being present, in particular when the tools are used at build-time of the package or are called by package's scripts (executables).
While the tools don't necessarily need to be fully compatible, they need to have a common basic usage. In particular, when a tool from one provider is replaced by a tool from another, the reverse dependencies must remain in working state, with no need for rebuilds or configuration adjustments.