Difference between pages "Install/BootLoader" and "Maximum Swappage"

< Install(Difference between pages)
(Installing GRUB)
 
(Created page with "{{Article |Subtitle=Getting the most out of swap |Summary=Learn how to improve the swap performance on your Linux server by several orders of magnitude. Author Daniel Robbins...")
 
Line 1: Line 1:
<noinclude>
+
{{Article
{{InstallPart|boot loader configuration}}
+
|Subtitle=Getting the most out of swap
</noinclude>
+
|Summary=Learn how to improve the swap performance on your Linux server by several orders of magnitude. Author Daniel Robbins takes you through this quick tip on getting the most from your server.
=== Installing a Bootloader ===
+
|Author=Drobbins
 +
}}
 +
When you set up a brand new Linux server, do you create a single 128 MB swap partition? If so, did you know that you are severely limiting swap performance? Would you like to increase swap performance by several orders of magnitude, and to create swap partitions larger than 1 GB? It's possible, requiring no kernel patches or special hardware, just pure geek know-how!
  
These install instructions show you how to use GRUB to boot using BIOS (old-school) or UEFI (new-school).
+
Some of you may not really care about swap. After all, Linux systems are typically very memory efficient, and swap is often barely touched. While often true on desktop systems, servers are another story. Because servers may handle unexpected stresses, such as runaway processes, denial of service attacks, or even the Slashdot effect, they need to have adequate high-speed swap so that they do not grind to a halt and possibly crash when all physical memory (and then some) is exhausted.
  
==== Old School (BIOS) ====
+
Still not convinced that this is a big deal? I'll show you how easy it is to bring down a server by launching a massive amount of new processes.
  
If you're using the BIOS to boot, setting up GRUB, the bootloader, is pretty easy.
+
{{Warning|Please, if you try this, do it only on a non-production server that you actually administer!}}
  
To use this recommended boot method, first emerge <code>boot-update</code>. This will also cause <code>grub-2</code> to be merged, since it is a dependency of <code>boot-update</code>.
+
Let's say you have two customized grep commands in /usr/bin, called bobgrep and jimgrep. Now, let's assume that bobgrep is simply a shell script that calls the ELF executable jimgrep, as follows:
  
<console>
+
{{file|lang=bash|desc=The bobgrep script|name=bobgrep|body=
(chroot) # ##i##emerge boot-update
+
#!/bin/bash
</console>
+
jimgrep -r $*
 +
}}
  
Then, edit <code>/etc/boot.conf</code> and specify "<code>Funtoo Linux genkernel</code>" as the <code>default</code> setting at the top of the file, replacing <code>"Funtoo Linux"</code>.
+
Everything looks good so far, but what happens if jimgrep gets accidentally replaced with a symbolic link to bobgrep? Well, in that case, calling bobgrep or jimgrep will cause an infinite loop, causing hundreds of bash processes to be spawned in mere seconds. This actually happened to me once, and believe me, it hurt!
  
<code>/etc/boot.conf</code> should now look like this:
+
If a server doesn't have adequate swap, a situation like this can cause the machine to lock up in much less than a minute. How do we fix the problem? One way is to increase the swap size beyond 128 MB. With Linux 2.0, there was a swap limit of 128MB on x86 systems. With 2.2, this limit was raised to 2GB. Currently, with Linux 3.x kernels, the limit is about 17.1TB, so you are unlikely to find yourself unable to allocate enough swap.
  
<pre>
+
While it's nice to be able to increase swap partition size to beyond 128 MB, how about increasing performance? Ideally, it would be nice if we could set up swap partitions in a RAID 0 stripe, so that reads and writes are equally distributed between all partitions. If these partitions are on separate drives and/or controllers, this will multiply swap file performance, allowing your servers to handle temporary memory usage "spikes" without getting dramatically bogged down.
boot {
+
generate grub
+
default "Funtoo Linux genkernel"  
+
timeout 3
+
}
+
  
"Funtoo Linux" {
+
Amazingly, all modern Linux kernels, by default (with no special kernel options or patches) allow you to parallelize swap, just like a RAID 0 stripe. By using the pri option in /etc/fstab to set multiple swap partitions to the same priority, we tell Linux to use them in parallel:
kernel bzImage[-v]
+
}
+
  
"Funtoo Linux genkernel" {
+
{{file|name=/etc/fstab|desc=Set multiple swap partitions to the same priority|body=
kernel kernel[-v]
+
/dev/sda2      none    swap    sw,pri=3        0      0
initrd initramfs[-v]
+
/dev/sdb2      none    swap    sw,pri=3        0      0
params += real_root=auto
+
/dev/sdc2      none    swap    sw,pri=3        0      0
}
+
/dev/sdd2      none    swap    sw,pri=1        0      0
 
+
"Funtoo Linux better-initramfs" {
+
kernel vmlinuz[-v]
+
initrd /initramfs.cpio.gz
+
}
+
</pre>
+
 
+
Please read <code>man boot.conf</code> for further details.
+
 
+
===== Running grub-install and boot-update =====
+
 
+
Finally, we will need to actually install the GRUB boot loader to your disk, and also run <code>boot-update</code> which will generate your boot loader configuration file:
+
 
+
<console>
+
(chroot) # ##i##grub-install --no-floppy /dev/sda
+
(chroot) # ##i##boot-update
+
</console>
+
 
+
Now you need to update your boot loader configuration file:
+
<console>
+
(chroot) # ##i##boot-update
+
</console>
+
You only need to run <code>grub-install</code> when you first install Funtoo Linux, but you need to re-run <code>boot-update</code> every time you modify your <code>/etc/boot.conf</code> file, so your changes are applied on next boot.
+
 
+
==== New School (UEFI) ====
+
 
+
If you're using UEFI to boot, setting up the boot loader is a bit more complicated for now, but this process will be improving soon. Perform the following steps.
+
 
+
===== Emerging GRUB =====
+
 
+
You will still use GRUB as a boot loader, but before emerging grub, you will need to enable EFI booting. To do this,
+
add the following line to <code>/etc/make.conf</code>:
+
 
+
For x86-64bit systems:
+
 
+
<pre>
+
GRUB_PLATFORMS="efi-64"
+
</pre>
+
 
+
For x86-32bit systems:
+
 
+
<pre>
+
GRUB_PLATFORMS="efi-32"
+
</pre>
+
 
+
Then, <code>emerge boot-update</code>. You will notice <code>grub</code> and <code>efibootmgr</code> getting pulled in as dependencies. This is expected and good:
+
 
+
<console>
+
(chroot) # ##i##emerge boot-update
+
</console>
+
 
+
===== Installing GRUB =====
+
 
+
Now, for the magic of getting everything in place for booting. You should copy your kernel and initramfs (if you have one -- you will if you are following the default install) to <tt>/boot</tt>. GRUB will boot those. But how do we get UEFI to boot GRUB? Well, we need to run the following command:
+
 
+
For x86-64bit systems:
+
 
+
<console>
+
(chroot) # ##i##grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id="Funtoo Linux [GRUB]" --recheck /dev/sda
+
</console>
+
 
+
For x86-32bit systems:
+
 
+
<console>
+
(chroot) # ##i##grub-install --target=i386-efi --efi-directory=/boot --bootloader-id="Funtoo Linux [GRUB]" --recheck /dev/sda
+
</console>
+
 
+
This command will simply install all the stuff to <tt>/boot/EFI</tt> and <tt>/boot/grub</tt> that your system needs to boot. In particular, the <tt>/boot/EFI/grub/grubx64.efi</tt> file will be created. This is the GRUB boot image that UEFI will load and start.
+
 
+
A more detailed explanation of the flags used in the above command:
+
* <code>--target=x86_64-efi</code> or <code>--target=i386-efi</code>: Tells GRUB that we want to install it in a way that allows it to boot in UEFI
+
* <code>--efi-directory=/boot</code>: All GRUB UEFI files will be installed in ''/boot''
+
* <code>--bootloader-id="Funtoo Linux [GRUB]"</code>: This flag is not necessary for GRUB to boot. However, it allows you to change the text of the boot option in the UEFI BIOS. The stuff in the quotes can be set to anything that you would like.
+
* <code>--recheck</code>: If a device map already exists on the disk or partition that GRUB is being installed on, it will be removed.
+
* <code>/dev/sda</code>:The device that we are installing GRUB on.
+
 
+
===== Configuring GRUB =====
+
 
+
OK, now UEFI has the GRUB image it needs to boot. But we still need to configure GRUB itself so it finds and boots your kernel and initramfs. This is done by performing the following steps. Since boot-update doesn't yet support UEFI, we will use boot-update, but then edit our <code>/boot/grub/grub.cfg</code> to support UEFI booting.
+
 
+
First, you will need to edit <code>/etc/boot.conf</code>. Format this as you would if you were booting without UEFI. If you are not sure how this should look, below is an example of what it could look like if you are booting from an unencrypted ext4 partition:
+
 
+
{{file|name=/etc/boot.conf|desc=|body=
+
boot {
+
        generate grub
+
        default "Funtoo Linux"
+
        timeout 3
+
}
+
 
+
"Funtoo Linux" {
+
        kernel vmlinuz[-v]
+
        params += rootfstype=ext4 root=/dev/sda2
+
}
+
 
}}
 
}}
  
After you have edited your <code>/etc/boot.conf</code> file, run <code>boot-update</code>. You should now have a <code>/boot/grub/grub.cfg</code> file, which you can edit using the following command:
+
In the above example, Linux will use swap partitions sda2, sdb2, and sdc2 in parallel. Since these partitions are on different drives, and possibly even different SCSI controllers, read and write throughput will nearly triple. The fourth swap partition, sdd2, will be used only after the first three partitions have been exhausted.
 
+
<console>
+
# ##i##nano /boot/grub/grub.cfg
+
</console>
+
 
+
 
+
To get your <code>/boot/grub/grub.cfg</code> to support booting with UEFI, make the following changes. Below the existing insmod lines, add the following lines. Both of these involve adding support for the UEFI framebuffer to GRUB.:
+
 
+
<pre>
+
  insmod efi_gop
+
  insmod efi_uga
+
</pre>
+
 
+
Then, change the <code>set gfxpayload</code> line to read as follows. UEFI does not support text mode, so we will keep video initialized to the current resolution.:
+
  
<pre>
+
The pri option is really easy to use. The priority must be a number between 0 and 32767, with 32767 being the highest priority. The swap partitions will be used from highest priority to lowest priority, meaning that a partition with a priority of x will only be used only if all partitions with a priority greater than x are already full. If several partitions have the same priority, Linux will automatically parallelize access between them. This allows you to not only parallelize swap, but also prioritize access so that the partitions on the fastest drives (or regions of the drives) are used first. So, you can set up an emergency swap partition on an old, slower drive that will be used only if all high-speed swap is exhausted first.
  set gfxpayload=keep
+
</pre>
+
  
You can now save your changes by pressing <code>Control-X</code> and answering <code>y</code> when asked if you want to save the modified buffer. When prompted for a filename, hit Enter to use the existing filename.
+
Now it's time to put some of this swapping knowledge into action. To loosely quote Mr. Miyagi of Karate Kid fame: "Swap on, swap off, geek-san!"
 +
{{ArticleFooter}}

Latest revision as of 00:08, January 2, 2015

Getting the most out of swap

Learn how to improve the swap performance on your Linux server by several orders of magnitude. Author Daniel Robbins takes you through this quick tip on getting the most from your server.

Support Funtoo and help us grow! Donate $15 per month and get a free SSD-based Funtoo Virtual Container.

When you set up a brand new Linux server, do you create a single 128 MB swap partition? If so, did you know that you are severely limiting swap performance? Would you like to increase swap performance by several orders of magnitude, and to create swap partitions larger than 1 GB? It's possible, requiring no kernel patches or special hardware, just pure geek know-how!

Some of you may not really care about swap. After all, Linux systems are typically very memory efficient, and swap is often barely touched. While often true on desktop systems, servers are another story. Because servers may handle unexpected stresses, such as runaway processes, denial of service attacks, or even the Slashdot effect, they need to have adequate high-speed swap so that they do not grind to a halt and possibly crash when all physical memory (and then some) is exhausted.

Still not convinced that this is a big deal? I'll show you how easy it is to bring down a server by launching a massive amount of new processes.

Warning

Please, if you try this, do it only on a non-production server that you actually administer!

Let's say you have two customized grep commands in /usr/bin, called bobgrep and jimgrep. Now, let's assume that bobgrep is simply a shell script that calls the ELF executable jimgrep, as follows:

bobgrep (bash source code) - The bobgrep script
#!/bin/bash
jimgrep -r $*

Everything looks good so far, but what happens if jimgrep gets accidentally replaced with a symbolic link to bobgrep? Well, in that case, calling bobgrep or jimgrep will cause an infinite loop, causing hundreds of bash processes to be spawned in mere seconds. This actually happened to me once, and believe me, it hurt!

If a server doesn't have adequate swap, a situation like this can cause the machine to lock up in much less than a minute. How do we fix the problem? One way is to increase the swap size beyond 128 MB. With Linux 2.0, there was a swap limit of 128MB on x86 systems. With 2.2, this limit was raised to 2GB. Currently, with Linux 3.x kernels, the limit is about 17.1TB, so you are unlikely to find yourself unable to allocate enough swap.

While it's nice to be able to increase swap partition size to beyond 128 MB, how about increasing performance? Ideally, it would be nice if we could set up swap partitions in a RAID 0 stripe, so that reads and writes are equally distributed between all partitions. If these partitions are on separate drives and/or controllers, this will multiply swap file performance, allowing your servers to handle temporary memory usage "spikes" without getting dramatically bogged down.

Amazingly, all modern Linux kernels, by default (with no special kernel options or patches) allow you to parallelize swap, just like a RAID 0 stripe. By using the pri option in /etc/fstab to set multiple swap partitions to the same priority, we tell Linux to use them in parallel:

/etc/fstab - Set multiple swap partitions to the same priority
/dev/sda2       none    swap    sw,pri=3        0       0
/dev/sdb2       none    swap    sw,pri=3        0       0
/dev/sdc2       none    swap    sw,pri=3        0       0
/dev/sdd2       none    swap    sw,pri=1        0       0

In the above example, Linux will use swap partitions sda2, sdb2, and sdc2 in parallel. Since these partitions are on different drives, and possibly even different SCSI controllers, read and write throughput will nearly triple. The fourth swap partition, sdd2, will be used only after the first three partitions have been exhausted.

The pri option is really easy to use. The priority must be a number between 0 and 32767, with 32767 being the highest priority. The swap partitions will be used from highest priority to lowest priority, meaning that a partition with a priority of x will only be used only if all partitions with a priority greater than x are already full. If several partitions have the same priority, Linux will automatically parallelize access between them. This allows you to not only parallelize swap, but also prioritize access so that the partitions on the fastest drives (or regions of the drives) are used first. So, you can set up an emergency swap partition on an old, slower drive that will be used only if all high-speed swap is exhausted first.

Now it's time to put some of this swapping knowledge into action. To loosely quote Mr. Miyagi of Karate Kid fame: "Swap on, swap off, geek-san!"


Support Funtoo and help us grow! Donate $15 per month and get a free SSD-based Funtoo Virtual Container.

Learn how to improve the swap performance on your Linux server by several orders of magnitude. Author Daniel Robbins takes you through this quick tip on getting the most from your server.
About the Author

Daniel Robbins is best known as the creator of Gentoo Linux and author of many IBM developerWorks articles about Linux. Daniel currently serves as Benevolent Dictator for Life (BDFL) of Funtoo Linux. Funtoo Linux is a Gentoo-based distribution and continuation of Daniel's original Gentoo vision.

Got Funtoo?

Have you installed Funtoo Linux yet? Discover the power of a from-source meta-distribution optimized for your hardware! See our installation instructions and browse our CPU-optimized builds.

Funtoo News

Drobbins

ARM Rebuild

ARM systems will use new stage3's that are not compatible with earlier versions.
2015-06-27 by Drobbins
Drobbins

ABI X86 64 and 32

Funtoo Linux has new 32-bit compatibility libraries inherited from Gentoo. Learn about them here.
2015-06-18 by Drobbins
Drobbins

Pre-built kernels!

Funtoo stage3's are now starting to offer pre-built kernels for ease of install. read more....
2015-05-12 by Drobbins
More...

More Articles

Browse all our Linux-related articles, below:

A

B

F

G

K

L

M

O

P

S

T

W

X