Difference between pages "GUID Booting Guide" and "Linux Containers"

(Difference between pages)
 
(/lxc/funtoo0/config to /etc/lxc/funtoo0.conf)
 
Line 1: Line 1:
== Introduction ==
+
Linux Containers, or LXC, is a Linux feature that allows Linux to run one or more isolated virtual systems (with their own network interfaces, process namespace, user namespace, and power state) using a single Linux kernel on a single server.
  
GPT, which stands for GUID Partition Table, is a disk partitioning scheme that was introduced by Intel for Itanium architecture systems, as part of EFI, the Extensible Firmware Interface. While you are probably not using an Itanium architecture computer, and you are likely using a BIOS-based rather than an EFI-based system, you still may want to use GPT partitioning. Why? Because the standard MBR-based partitioning scheme only supports system disks that are less than 2TiB in size. On modern systems, especially systems with hardware RAID logical volumes, it is very easy to go beyond the 2TiB limit. GUID partition tables support disks that are larger than 2TiB in size.
+
== Status ==
  
=== GPT Technology Overview ===
+
As of Linux kernel 3.1.5, LXC is usable for isolating your own private workloads from one another. It is not yet ready to isolate potentially malicious users from one another or the host system. For a more mature containers solution that is appropriate for hosting environments, see [[OpenVZ]].
  
This section contains a technical overview of GPT technology.
+
LXC containers don't yet have their own system uptime, and they see everything that's in the host's <tt>dmesg</tt> output, among other things. But in general, the technology works.
  
GUID partition tables support up to 512 partitions. GPT data structures are stored in the first sectors of the drive with a secondary copy stored at the end of the drive. This allows the partitioning scheme of your disk to be recovered in situations where the primary partition table has been corrupted.
+
== Basic Info ==
  
For compatibility with legacy partitioning tools, GPT partitioning tools typically rewrite the MBR partition table (generally located in the first sector of the disk) in a way those tools will interpret it like ''"This disk has only one partition (of an unknown type) covering the whole disk".''
 
  
It is possible to convert an existing MBR-partitioned disk to GPT format using the <tt>gdisk</tt> command. Please carefully read the <tt>gdisk</tt> man page before using this capability, as it is potentially dangerous, particularly if you are performing it on your boot disk.
+
* Linux Containers are based on:
 +
** Kernel namespaces for resource isolation
 +
** CGroups for resource limitation and accounting
  
{{ fancyimportant|Funtoo Linux fully supports GPT on x86-32bit and x86-64-bit systems. GPT is supported on SPARC systems, but currently only for non-boot disks.
+
{{Package|app-emulation/lxc}} is the userspace tool for Linux containers
}}
+
  
=== Booting GPT ===
+
== Control groups ==
  
If you decide to use a GPT-based partitioning scheme for your system disk, either out of necessity due to a 2TiB+ disk, or because you want to try GPT out, then the question arises -- how do you get the darn thing to boot? This is where the new <tt>GRUB</tt> boot loader comes in. The new <tt>GRUB</tt> (version 1.9x, found at <tt>sys-boot/grub</tt>) is a redesign of the original <tt>GRUB</tt> (version 0.9x, now called <tt>sys-boot/grub-legacy</tt> in Funtoo) boot-loader that includes very mature support for booting from GPT-based disks.
+
* Control groups (cgroups) in kernel since 2.6.24
 +
** Allows aggregation of tasks and their children
 +
** Subsystems (cpuset, memory, blkio,...)
 +
** accounting - to measure how much resources certain systems use
 +
** resource limiting - groups can be set to not exceed a set memory limit
 +
** prioritization - some groups may get a larger share of CPU
 +
** control - freezing/unfreezing of cgroups, checkpointing and restarting
 +
** No disk quota limitation ( -> image file, LVM, XFS, directory tree quota,...)
  
Now, let's take a look at how to get GPT-based booting working under Funtoo Linux.
+
== Subsystems ==
 +
<br>
 +
<console>
 +
###i## cat /proc/cgroups
 +
subsys_name hierarchy num_cgroups enabled
 +
cpuset
 +
cpu
 +
cpuacct
 +
memory
 +
devices
 +
freezer
 +
blkio
 +
perf_event
 +
hugetlb
 +
</console>
  
== Getting Started ==
+
#cpuset    -> limits tasks to specific CPU/CPUs
 +
#cpu        -> CPU shares
 +
#cpuacct    -> CPU accounting
 +
#memory    -> memory and swap limitation and accounting
 +
#devices    -> device allow deny list
 +
#freezer    -> suspend/resume tasks
 +
#blkio      -> I/O priorization (weight, throttle, ...)
 +
#perf_event -> support for per-cpu per-cgroup monitoring [http://lwn.net/Articles/421574/ perf_events]
 +
#hugetlb    -> cgroup resource controller for HugeTLB pages  [http://lwn.net/Articles/499255/ hugetlb]
  
The first thing you'll need to do is to use a LiveCD. I recommend [http://www.sysresccd.org/Main_Page System Rescue CD] for this task as it is Gentoo-based and includes all the proper tools. Go ahead and boot the LiveCD, and then get to the point where you are ready to partition your system disk.
+
== Configuring the Funtoo Host System ==
  
At this point, you have two choices as to what partitioning tool to use. You can use either <tt>gdisk</tt> or <tt>parted</tt>. <tt>gdisk</tt> is a very nice <tt>fdisk</tt>-like partitioning tool that supports GPT partitioning. It is rather new software but seems to work quite well. The other tool you can use, GNU <tt>parted</tt>, has been around for a while and is more mature, but is harder to use.
+
=== Install LXC kernel ===
 +
Any kernel beyond 3.1.5 will probably work. Personally I prefer {{Package|sys-kernel/gentoo-sources}} as these have support for all the namespaces without sacrificing the xfs, FUSE or NFS support for example. These checks were introduced later starting from kernel 3.5, this could also mean that the user namespace is not working optimally.
  
We'll take a look at how to create partitions using <tt>gdisk</tt>. Alternatively, <code>cgdisk</code>, curses-based gdisk for users familiar with cfdisk or <code>sgdisk</code>, command-line tool can be used for creating and managing GPT partitions.
+
* User namespace (EXPERIMENTAL) depends on EXPERIMENTAL and on UIDGID_CONVERTED
 +
** config UIDGID_CONVERTED
 +
*** True if all of the selected software components are known to have uid_t and gid_t converted to kuid_t and kgid_t where appropriate and are otherwise safe to use with the user namespace.
 +
**** Networking - depends on NET_9P = n
 +
**** Filesystems - 9P_FS = n, AFS_FS = n, AUTOFS4_FS = n, CEPH_FS = n, CIFS = n, CODA_FS = n, FUSE_FS = n, GFS2_FS = n, NCP_FS = n, NFSD = n, NFS_FS = n, OCFS2_FS = n, XFS_FS = n
 +
**** Security options - Grsecurity - GRKERNSEC = n (if applicable)
  
== Partitioning Using Gdisk ==
+
** As of 3.10.xx kernel, all of the above options are safe to use with User namespaces, except for XFS_FS, therefore with kernel >=3.10.xx, you should answer XFS_FS = n, if you want User namespaces support.
 +
** in your kernel source directory, you should check init/Kconfig and find out what UIDGID_CONVERTED depends on
  
OK, the first step is using <tt>gdisk</tt> is to start it up, specifying the disk you want to modify:
+
==== Kernel configuration ====
 +
These options should be enable in your kernel to be able to take full advantage of LXC.
  
<pre># gdisk /dev/sda</pre>
+
* General setup
You should find <tt>gdisk</tt> very familiar to <tt>fdisk</tt>. Here is the partition table we want to end up with:
+
** CONFIG_NAMESPACES
 +
*** CONFIG_UTS_NS
 +
*** CONFIG_IPC_NS
 +
*** CONFIG_PID_NS
 +
*** CONFIG_NET_NS
 +
*** CONFIG_USER_NS
 +
** CONFIG_CGROUPS
 +
*** CONFIG_CGROUP_DEVICE
 +
*** CONFIG_CGROUP_SCHED
 +
*** CONFIG_CGROUP_CPUACCT
 +
*** CONFIG_CGROUP_MEM_RES_CTLR (in 3.6+ kernels it's called CONFIG_MEMCG)
 +
*** CONFIG_CGROUP_MEM_RES_CTLR_SWAP (in 3.6+ kernels it's called CONFIG_MEMCG_SWAP)
 +
*** CONFIG_CPUSETS (on multiprocessor hosts)
 +
* Networking support
 +
** Networking options
 +
*** CONFIG_VLAN_8021Q
 +
* Device Drivers
 +
** Character devices
 +
*** Unix98 PTY support
 +
**** CONFIG_DEVPTS_MULTIPLE_INSTANCES
 +
** Network device support
 +
*** Network core driver support
 +
**** CONFIG_VETH
 +
**** CONFIG_MACVLAN
  
<pre>Command (? for help): p
+
Once you have lxc installed, you can then check your kernel config with:
Disk /dev/sda: 312581808 sectors, 149.1 GiB
+
<console>
Disk identifier (GUID): 17
+
# ##i##CONFIG=/path/to/config /usr/sbin/lxc-checkconfig
Partition table holds up to 128 entries
+
</console>
First usable sector is 34, last usable sector is 312581774
+
Total free space is 0 sectors (0 bytes)
+
  
Number  Start (sector)    End (sector)  Size      Code  Name
+
=== Emerge lxc ===
  1              34          204833  100.0 MiB  0700  Linux/Windows data
+
<console>
  2          204834          270369  512.0 kiB  EF02  BIOS boot partition
+
# ##i##emerge app-emulation/lxc
  3          270370        1318945  512.0 MiB  8200  Linux swap
+
</console>
  4        1318946      312581774  148.4 GiB  0700  Linux/Windows data
+
  
Command (? for help): </pre>
+
=== Configure Networking For Container ===
Above, you'll see that we have a 100 MiB boot partition, a 512 kiB &quot;BIOS boot partition&quot;, 512 MiB of swap, and the remaining disk used by a 148.4 GiB root partition.
+
  
The one new thing here is the &quot;BIOS boot partition.&quot; What is it? In GRUB-speak, this BIOS boot partition is basically the location of the meat of GRUB's boot loading code - the quivalent of the <tt>stage1_5</tt> and <tt>stage2</tt> files in legacy GRUB. Since GPT-based partition tables have less &quot;bonus&quot; space than their MBR equivalents, and explicit partition of code <tt>EF02</tt> is required to hold the guts of the boot loader.
+
Typically, one uses a bridge to allow containers to connect to the network. This is how to do it under Funtoo Linux:
  
In all other respects, the partition table is similar to that of an MBR-based disk. We have a boot and root partition with code <tt>0700</tt>, and a Linux swap partition with code <tt>8200</tt>. One this partition table has been written to disk and appropriate <tt>mkfs</tt> and <tt>mkswap</tt> commands are issued, <tt>/dev/sda1</tt> will be used to hold <tt>/boot</tt>, <tt>/dev/sda2</tt> will be used by the new GRUB directly, <tt>/dev/sda3</tt> will house our swap and <tt>/dev/sda4</tt> will hold our root filesystem.
+
# create a bridge using the Funtoo network configuration scripts. Name the bridge something like <tt>brwan</tt> (using <tt>/etc/init.d/netif.brwan</tt>). Configure your bridge to have an IP address.
 +
# Make your physical interface, such as <tt>eth0</tt>, an interface with no IP address (use the Funtoo <tt>interface-noip</tt> template.)
 +
# Make <tt>netif.eth0</tt> a slave of <tt>netif.brwan</tt> in <tt>/etc/conf.d/netif.brwan</tt>.
 +
# Enable your new bridged network and make sure it is functioning properly on the host.
  
Go ahead and create filesystems on these partitions, and then mount the root and boot filesystems to <tt>/mnt/gentoo</tt> and <tt>/mnt/gentoo/boot</tt> respectively. Now go ahead and unpack a stage3 tarball to <tt>/mnt/gentoo</tt> and chroot in as you normally do.
+
You will now be able to configure LXC to automatically add your container's virtual ethernet interface to the bridge when it starts, which will connect it to your network.
  
== Configuring The Kernel ==
+
== Setting up a Funtoo Linux LXC Container ==
  
Your kernel will need a couple of extra GPT-related options enabled in order for it to make sense of your GPT partitions and find your filesystems. These options can be found under <tt>Enable the block layer ---&gt; Partition Types</tt>:
+
Here are the steps required to get Funtoo Linux running <i>inside</i> a container. The steps below show you how to set up a container using an existing Funtoo Linux OpenVZ template. It is now also possible to use [[Metro]] to build an lxc container tarball directly, which will save you manual configuration steps and will provide an <tt>/etc/fstab.lxc</tt> file that you can use for your host container config. See [[Metro Recipes]] for info on how to use Metro to generate an lxc container.
  
<pre>[*] Advanced Partition Selection (PARTITION_ADVANCED)
+
=== Create and Configure Container Filesystem ===
[*] EFI GUID Partition Support (EFI_PARTITION)</pre>
+
If you are using a non-Funtoo distribution then you may need to append a proper <tt>rootfstype=</tt> option to your kernel boot options to allow Linux to properly mount the root filesystem when <tt>Advanced Partition Selection</tt> is enabled. [[Boot-Update]] does this for you automatically.
+
  
Now just go ahead and compile and install your kernel, and copy it to <tt>/boot/bzImage</tt>.
+
# Start with a Funtoo LXC template, and unpack it to a directory such as <tt>/lxc/funtoo0/rootfs/</tt>
 +
# Create an empty <tt>/lxc/funtoo0/fstab</tt> file
 +
# Ensure <tt>c1</tt> line is uncommented (enabled) and <tt>c2</tt> through <tt>c6</tt> lines are disabled in <tt>/lxc/funtoo0/rootfs/etc/inittab</tt>
  
== Booting The System ==
+
That's almost all you need to get the container filesystem ready to start.
  
To get the system booted, you will want to first edit <tt>/etc/fstab</tt> inside the chroot so that it reflects the partitions and filesystems you just created. Then, emerge <tt>boot-update</tt> version 1.4_beta2 or later:
+
=== Create Container Configuration Files ===
  
<pre># emerge boot-update</pre>
+
Create the following files:
[[Boot-Update]] is a front-end for the GRUB 1.9x boot loader and provides a necessary simplified configuration interface. <tt>boot-update</tt> is used to generate boot loader configuration files. But before we get to <tt>boot-update</tt>, we first need to install GRUB to your hard disk. This is done as follows:
+
  
<pre># grub-install /dev/sda</pre>
+
==== <tt>/lxc/funtoo0/config</tt> ====
<tt>grub-install</tt> will detect and use <tt>/dev/sda2</tt> and use it to store its boot loader logic.
+
  
Now it's time to create an <tt>/etc/boot.conf</tt> file. For more information on all available options, consult the [[Boot-Update]] guide -- I'll show you a sample configuration for the sample GPT partition scheme above:
 
  
<pre>boot {
+
and also create symlink from
        generate grub
+
==== <tt> /lxc/funtoo0/config to /etc/lxc/funtoo0.conf </tt> ====
        default bzImage
+
<console>
}
+
###i## mkdir /etc/lxc/funtoo0
 +
###i## ln -s /lxc/funtoo0/config /etc/lxc/funtoo0/config
 +
</console>
  
&quot;Funtoo Linux&quot; {
+
{{Fancynote| Daniel Robbins needs to update this config to be more in line with http://wiki.progress-linux.org/software/lxc/ -- this config appears to have nice, refined device node permissions and other goodies. // note by Havis to Daniel, this config is already superior.}}
        kernel bzImage
+
}</pre>
+
Once <tt>/etc/boot.conf</tt> has been created, then type:
+
  
<pre># boot-update</pre>
 
This will auto-generate a <tt>/boot/grub/grub.cfg</tt> file for you, and you will now be able to reboot into Funtoo Linux using a GPT partitioning scheme.
 
  
For more information on all the options available for <tt>/etc/boot.conf</tt>, please consult the [[Boot-Update]] guide.
+
Read "man 5 lxc.conf" , to get more information about linux container configuration file.
 +
<pre>
 +
## Container
 +
lxc.utsname                            = funtoo0
 +
lxc.rootfs                              = /lxc/funtoo0/rootfs/
 +
lxc.arch                                = x86_64
 +
#lxc.console                            = /var/log/lxc/funtoo0.console  # uncomment if you want to log containers console
 +
lxc.tty                                = 6  # if you plan to use container with physical terminals (eg F1..F6)
 +
#lxc.tty                                = 0  # set to 0 if you dont plan to use the container with physical terminal, also comment out in your containers /etc/inittab  c1 to c6 respawns (e.g. c1:12345:respawn:/sbin/agetty 38400 tty1 linux)
 +
lxc.pts                                = 1024
  
[[Category:Funtoo features]]
+
 
 +
## Capabilities
 +
lxc.cap.drop                            = audit_control
 +
lxc.cap.drop                            = audit_write
 +
lxc.cap.drop                            = mac_admin
 +
lxc.cap.drop                            = mac_override
 +
lxc.cap.drop                            = mknod
 +
lxc.cap.drop                            = setfcap
 +
lxc.cap.drop                            = setpcap
 +
lxc.cap.drop                            = sys_admin
 +
#lxc.cap.drop                            = sys_boot # capability to reboot the container
 +
#lxc.cap.drop                            = sys_chroot # required by SSH
 +
lxc.cap.drop                            = sys_module
 +
#lxc.cap.drop                            = sys_nice
 +
lxc.cap.drop                            = sys_pacct
 +
lxc.cap.drop                            = sys_rawio
 +
lxc.cap.drop                            = sys_resource
 +
lxc.cap.drop                            = sys_time
 +
#lxc.cap.drop                            = sys_tty_config # required by getty
 +
 
 +
## Devices
 +
#lxc.cgroup.devices.allow              = a # Allow access to all devices
 +
lxc.cgroup.devices.deny                = a # Deny access to all devices
 +
 
 +
# Allow to mknod all devices (but not using them)
 +
lxc.cgroup.devices.allow                = c *:* m
 +
lxc.cgroup.devices.allow                = b *:* m
 +
 
 +
lxc.cgroup.devices.allow                = c 1:3 rwm # /dev/null
 +
lxc.cgroup.devices.allow                = c 1:5 rwm # /dev/zero
 +
lxc.cgroup.devices.allow                = c 1:7 rwm # /dev/full
 +
lxc.cgroup.devices.allow                = c 1:8 rwm # /dev/random
 +
lxc.cgroup.devices.allow                = c 1:9 rwm # /dev/urandom
 +
#lxc.cgroup.devices.allow                = c 4:0 rwm # /dev/tty0 ttys not required if you have lxc.tty = 0
 +
#lxc.cgroup.devices.allow                = c 4:1 rwm # /dev/tty1 devices with major number 4 are "real" tty devices
 +
#lxc.cgroup.devices.allow                = c 4:2 rwm # /dev/tty2
 +
#lxc.cgroup.devices.allow                = c 4:3 rwm # /dev/tty3
 +
lxc.cgroup.devices.allow                = c 5:0 rwm # /dev/tty
 +
lxc.cgroup.devices.allow                = c 5:1 rwm # /dev/console
 +
lxc.cgroup.devices.allow                = c 5:2 rwm # /dev/ptmx
 +
lxc.cgroup.devices.allow                = c 10:229 rwm # /dev/fuse
 +
lxc.cgroup.devices.allow                = c 136:* rwm # /dev/pts/* devices with major number 136 are pts
 +
lxc.cgroup.devices.allow                = c 254:0 rwm # /dev/rtc0
 +
 
 +
## Limits#
 +
lxc.cgroup.cpu.shares                  = 1024
 +
lxc.cgroup.cpuset.cpus                = 0        # limits container to CPU0
 +
lxc.cgroup.memory.limit_in_bytes      = 512M
 +
lxc.cgroup.memory.memsw.limit_in_bytes = 1G
 +
#lxc.cgroup.blkio.weight                = 500      # requires cfq block scheduler
 +
 
 +
## Filesystem
 +
#containers fstab should be outside it's rootfs dir (e.g. /lxc/funtoo0/fstab is ok, but /lxc/funtoo0/rootfs/etc/fstab is wrong!!!)
 +
#lxc.mount                              = /lxc/funtoo0/fstab     
 +
 
 +
#lxc.mount.entry is prefered, because it supports relative paths
 +
lxc.mount.entry                        = proc proc proc nosuid,nodev,noexec  0 0
 +
lxc.mount.entry                        = sysfs sys sysfs nosuid,nodev,noexec,ro 0 0
 +
lxc.mount.entry                        = devpts dev/pts devpts nosuid,noexec,mode=0620,ptmxmode=000,newinstance 0 0
 +
lxc.mount.entry                        = tmpfs dev/shm tmpfs nosuid,nodev,mode=1777 0 0
 +
lxc.mount.entry                        = tmpfs run tmpfs nosuid,nodev,noexec,mode=0755,size=128m 0 0
 +
lxc.mount.entry                        = tmpfs tmp tmpfs nosuid,nodev,noexec,mode=1777,size=1g 0 0
 +
 
 +
##Example of having /var/tmp/portage as tmpfs in container
 +
#lxc.mount.entry                        = tmpfs var/tmp/portage tmpfs defaults,size=8g,uid=250,gid=250,mode=0775 0 0
 +
##Example of bind mount
 +
#lxc.mount.entry                        = /srv/funtoo0 /lxc/funtoo0/rootfs/srv/funtoo0 none defaults,bind 0 0
 +
 
 +
## Network
 +
lxc.network.type                        = veth
 +
lxc.network.flags                      = up
 +
lxc.network.hwaddr                      = #put your MAC address here, otherwise you will get a random one
 +
lxc.network.link                        = br0
 +
lxc.network.name                        = eth0
 +
#lxc.network.veth.pair                  = veth-example
 +
</pre>
 +
 
 +
Read "man 7 capabilities" to get more information aboout Linux capabilities.
 +
 
 +
Above, use the following command to generate a random MAC for <tt>lxc.network.hwaddr</tt>:
 +
 
 +
<console>
 +
###i## openssl rand -hex 6 | sed 's/\(..\)/\1:/g; s/.$//'
 +
</console>
 +
 
 +
It is a very good idea to assign a static MAC address to your container using <tt>lxc.network.hwaddr</tt>. If you don't, LXC will auto-generate a new random MAC every time your container starts, which may confuse network equipment that expects MAC addresses to remain constant.
 +
 
 +
It might happen from case to case that you aren't able to start your LXC Container with the above generated MAC address so for all these who run into that problem here is a little script that connects your IP for the container with the MAC address. Just save the following code as <tt>/etc/lxc/hwaddr.sh</tt>, make it executable and run it like <tt>/etc/lxc/hwaddr.sh xxx.xxx.xxx.xxx</tt> where xxx.xxx.xxx.xxx represents your Container IP. <br><tt>/etc/lxc/hwaddr.sh</tt>:
 +
 
 +
<pre>
 +
#!/bin/sh
 +
IP=$*
 +
HA=`printf "02:00:%x:%x:%x:%x" ${IP//./ }`
 +
echo $HA
 +
</pre>
 +
 
 +
==== <tt>/lxc/funtoo0/fstab</tt> ====
 +
{{fancynote| It is now preferable to have mount entries directly in config file instead of separate fstab:}}
 +
Edit the file <tt>/lxc/funtoo0/fstab</tt>:
 +
<pre>
 +
none /lxc/funtoo0/dev/pts devpts defaults 0 0
 +
none /lxc/funtoo0/proc proc defaults 0 0
 +
none /lxc/funtoo0/sys sysfs defaults 0 0
 +
none /lxc/funtoo0/dev/shm tmpfs nodev,nosuid,noexec,mode=1777,rw 0 0
 +
</pre>
 +
 
 +
== LXC Networking ==
 +
*veth - Virtual Ethernet (bridge)
 +
*vlan - vlan interface (requires device able to do vlan tagging)
 +
*macvlan (mac-address based virtual lan tagging) has 3 modes:
 +
**private
 +
**vepa (Virtual Ethernet Port Aggregator)
 +
**bridge
 +
*phys - dedicated host NIC
 +
[https://blog.flameeyes.eu/2010/09/linux-containers-and-networking Linux Containers and Networking]
 +
 
 +
Enable routing on the host:
 +
By default Linux workstations and servers have IPv4 forwarding disabled.
 +
<console>
 +
###i## echo "1" > /proc/sys/net/ipv4/ip_forward
 +
###i## cat /proc/sys/net/ipv4/ip_forward
 +
# 1
 +
</console>
 +
 
 +
== Initializing and Starting the Container ==
 +
 
 +
You will probably need to set the root password for the container before you can log in. You can use chroot to do this quickly:
 +
 
 +
<console>
 +
###i## chroot /lxc/funtoo0/rootfs
 +
(chroot) ###i## passwd
 +
New password: XXXXXXXX
 +
Retype new password: XXXXXXXX
 +
passwd: password updated successfully
 +
(chroot) ###i## exit
 +
</console>
 +
 
 +
Now that the root password is set, run:
 +
 
 +
<console>
 +
###i## lxc-start -n funtoo0 -d
 +
</console>
 +
 
 +
The <tt>-d</tt> option will cause it to run in the background.
 +
 
 +
To attach to the console:
 +
 
 +
<console>
 +
###i## lxc-console -n funtoo0
 +
</console>
 +
 
 +
You should now be able to log in and use the container. In addition, the container should now be accessible on the network.
 +
 
 +
To directly attach to container:
 +
 
 +
<console>
 +
###i## lxc-attach -n funtoo0
 +
</console>
 +
 
 +
To stop the container:
 +
 
 +
<console>
 +
###i## lxc-stop -n funtoo0
 +
</console>
 +
 
 +
Ensure that networking is working from within the container while it is running, and you're good to go!
 +
 
 +
== Starting LXC container during host boot ==
 +
 
 +
# You need to create symlink in <tt>/etc/init.d/</tt> to <tt>/etc/init.d/lxc</tt> so that it reflects your container.
 +
# <tt>ln -s /etc/init.d/lxc /etc/init.d/lxc.funtoo0</tt>
 +
# now you can add <tt>lxc.funtoo0</tt> to default runlevel
 +
# <tt>rc-update add lxc.funtoo0 default</tt>
 +
<console>
 +
###i## rc
 +
* Starting funtoo0 ...                  [ ok ]
 +
</console>
 +
 
 +
== LXC Bugs/Missing Features ==
 +
 
 +
This section is devoted to documenting issues with the current implementation of LXC and its associated tools. We will be gradually expanding this section with detailed descriptions of problems, their status, and proposed solutions.
 +
 
 +
=== reboot ===
 +
 
 +
* By default, lxc does not support rebooting a container from within. It will simply stop and the host will not know to start it.
 +
* If you want your container to reboot gracefully, you need sys_boot capability (comment out lxc.cap.drop = sys_boot in your container config)
 +
 
 +
=== PID namespaces ===
 +
 
 +
Process ID namespaces are functional, but the container can still see the CPU utilization of the host via the system load (ie. in <tt>top</tt>).
 +
 
 +
=== /dev/pts newinstance ===
 +
 
 +
* Some changes may be required to the host to properly implement "newinstance" <tt>/dev/pts</tt>. See [https://bugzilla.redhat.com/show_bug.cgi?id=501718 This Red Hat bug].
 +
 
 +
=== lxc-create and lxc-destroy ===
 +
 
 +
* LXC's shell scripts are badly designed and are sure way to destruction, avoid using lxc-create and lxc-destroy.
 +
 
 +
=== network initialization and cleanup ===
 +
 
 +
* If used network.type = phys after lxc-stop the interface will be renamed to value from lxc.network.link. It supposed to be fixed in 0.7.4, happens still on 0.7.5 - http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg01760.html
 +
 
 +
* Re-starting a container can result in a failure as network resource are tied up from the already-defunct instance: [http://www.mail-archive.com/lxc-devel@lists.sourceforge.net/msg00824.html]
 +
 
 +
=== graceful shutdown ===
 +
 
 +
* To gracefully shutdown a container, it's init system needs to properly handle kill -PWR signal
 +
* For funtoo/gentoo make sure that you have:
 +
** pf:12345:powerwait:/sbin/halt
 +
** in your containers /etc/inittab
 +
* For debian/ubuntu make sure that you have:
 +
** pf::powerwait:/sbin/shutdown -t1 -a -h now
 +
** in your container /etc/inittab
 +
** and also comment out other line starting with pf:powerfail (such as pf::powerwait:/etc/init.d/powerfail start) <- these are used if you have UPS monitoring daemon installed!
 +
* /etc/init.d/lxc seems to have broken support for graceful shutdown (it sends proper signal, but then also tries to kill the init with lxc-stop)
 +
 
 +
=== funtoo ===
 +
 
 +
* Our udev should be updated to contain <tt>-lxc</tt> in scripts. (This has been done as of 02-Nov-2011, so should be resolved. But not fixed in our openvz templates, so need to regen them in a few days.)
 +
* Our openrc should be patched to handle the case where it cannot mount tmpfs, and gracefully handle this situation somehow. (Work-around in our docs above, which is to mount tmpfs to <tt>/libexec/rc/init.d</tt> using the container-specific <tt>fstab</tt> file (on the host.)
 +
* Emerging udev within a container can/will fail when realdev is run, if a device node cannot be created (such as /dev/console) if there are no mknod capabilities within the container. This should be fixed.
 +
 
 +
== References ==
 +
 
 +
* <tt>man 7 capabilities</tt>
 +
* <tt>man 5 lxc.conf</tt>
 +
 
 +
== Links ==
 +
 
 +
* There are a number of additional lxc features that can be enabled via patches: [http://lxc.sourceforge.net/patches/linux/3.0.0/3.0.0-lxc1/]
 +
* [https://wiki.ubuntu.com/UserNamespace Ubuntu User Namespaces page]
 +
* lxc-gentoo setup script [https://github.com/globalcitizen/lxc-gentoo on GitHub]
 +
 
 +
* '''IBM developerWorks'''
 +
** [http://www.ibm.com/developerworks/linux/library/l-lxc-containers/index.html LXC: Linux Container Tools]
 +
** [http://www.ibm.com/developerworks/linux/library/l-lxc-security/ Secure Linux Containers Cookbook]
 +
 
 +
* '''Linux Weekly News'''
 +
** [http://lwn.net/Articles/244531/ Smack for simplified access control]
 +
 
 +
[[Category:Labs]]
 +
[[Category:HOWTO]]
 +
[[Category:Virtualization]]

Revision as of 18:52, January 28, 2015

Linux Containers, or LXC, is a Linux feature that allows Linux to run one or more isolated virtual systems (with their own network interfaces, process namespace, user namespace, and power state) using a single Linux kernel on a single server.

Status

As of Linux kernel 3.1.5, LXC is usable for isolating your own private workloads from one another. It is not yet ready to isolate potentially malicious users from one another or the host system. For a more mature containers solution that is appropriate for hosting environments, see OpenVZ.

LXC containers don't yet have their own system uptime, and they see everything that's in the host's dmesg output, among other things. But in general, the technology works.

Basic Info

  • Linux Containers are based on:
    • Kernel namespaces for resource isolation
    • CGroups for resource limitation and accounting

Package:LXC is the userspace tool for Linux containers

Control groups

  • Control groups (cgroups) in kernel since 2.6.24
    • Allows aggregation of tasks and their children
    • Subsystems (cpuset, memory, blkio,...)
    • accounting - to measure how much resources certain systems use
    • resource limiting - groups can be set to not exceed a set memory limit
    • prioritization - some groups may get a larger share of CPU
    • control - freezing/unfreezing of cgroups, checkpointing and restarting
    • No disk quota limitation ( -> image file, LVM, XFS, directory tree quota,...)

Subsystems


# cat /proc/cgroups 
subsys_name	hierarchy	num_cgroups	enabled
cpuset	
cpu	
cpuacct	
memory	
devices	
freezer	
blkio	
perf_event
hugetlb
  1. cpuset -> limits tasks to specific CPU/CPUs
  2. cpu -> CPU shares
  3. cpuacct -> CPU accounting
  4. memory -> memory and swap limitation and accounting
  5. devices -> device allow deny list
  6. freezer -> suspend/resume tasks
  7. blkio -> I/O priorization (weight, throttle, ...)
  8. perf_event -> support for per-cpu per-cgroup monitoring perf_events
  9. hugetlb -> cgroup resource controller for HugeTLB pages hugetlb

Configuring the Funtoo Host System

Install LXC kernel

Any kernel beyond 3.1.5 will probably work. Personally I prefer sys-kernel/gentoo-sources (package not on wiki - please add) as these have support for all the namespaces without sacrificing the xfs, FUSE or NFS support for example. These checks were introduced later starting from kernel 3.5, this could also mean that the user namespace is not working optimally.

  • User namespace (EXPERIMENTAL) depends on EXPERIMENTAL and on UIDGID_CONVERTED
    • config UIDGID_CONVERTED
      • True if all of the selected software components are known to have uid_t and gid_t converted to kuid_t and kgid_t where appropriate and are otherwise safe to use with the user namespace.
        • Networking - depends on NET_9P = n
        • Filesystems - 9P_FS = n, AFS_FS = n, AUTOFS4_FS = n, CEPH_FS = n, CIFS = n, CODA_FS = n, FUSE_FS = n, GFS2_FS = n, NCP_FS = n, NFSD = n, NFS_FS = n, OCFS2_FS = n, XFS_FS = n
        • Security options - Grsecurity - GRKERNSEC = n (if applicable)
    • As of 3.10.xx kernel, all of the above options are safe to use with User namespaces, except for XFS_FS, therefore with kernel >=3.10.xx, you should answer XFS_FS = n, if you want User namespaces support.
    • in your kernel source directory, you should check init/Kconfig and find out what UIDGID_CONVERTED depends on

Kernel configuration

These options should be enable in your kernel to be able to take full advantage of LXC.

  • General setup
    • CONFIG_NAMESPACES
      • CONFIG_UTS_NS
      • CONFIG_IPC_NS
      • CONFIG_PID_NS
      • CONFIG_NET_NS
      • CONFIG_USER_NS
    • CONFIG_CGROUPS
      • CONFIG_CGROUP_DEVICE
      • CONFIG_CGROUP_SCHED
      • CONFIG_CGROUP_CPUACCT
      • CONFIG_CGROUP_MEM_RES_CTLR (in 3.6+ kernels it's called CONFIG_MEMCG)
      • CONFIG_CGROUP_MEM_RES_CTLR_SWAP (in 3.6+ kernels it's called CONFIG_MEMCG_SWAP)
      • CONFIG_CPUSETS (on multiprocessor hosts)
  • Networking support
    • Networking options
      • CONFIG_VLAN_8021Q
  • Device Drivers
    • Character devices
      • Unix98 PTY support
        • CONFIG_DEVPTS_MULTIPLE_INSTANCES
    • Network device support
      • Network core driver support
        • CONFIG_VETH
        • CONFIG_MACVLAN

Once you have lxc installed, you can then check your kernel config with:

# CONFIG=/path/to/config /usr/sbin/lxc-checkconfig

Emerge lxc

# emerge app-emulation/lxc

Configure Networking For Container

Typically, one uses a bridge to allow containers to connect to the network. This is how to do it under Funtoo Linux:

  1. create a bridge using the Funtoo network configuration scripts. Name the bridge something like brwan (using /etc/init.d/netif.brwan). Configure your bridge to have an IP address.
  2. Make your physical interface, such as eth0, an interface with no IP address (use the Funtoo interface-noip template.)
  3. Make netif.eth0 a slave of netif.brwan in /etc/conf.d/netif.brwan.
  4. Enable your new bridged network and make sure it is functioning properly on the host.

You will now be able to configure LXC to automatically add your container's virtual ethernet interface to the bridge when it starts, which will connect it to your network.

Setting up a Funtoo Linux LXC Container

Here are the steps required to get Funtoo Linux running inside a container. The steps below show you how to set up a container using an existing Funtoo Linux OpenVZ template. It is now also possible to use Metro to build an lxc container tarball directly, which will save you manual configuration steps and will provide an /etc/fstab.lxc file that you can use for your host container config. See Metro Recipes for info on how to use Metro to generate an lxc container.

Create and Configure Container Filesystem

  1. Start with a Funtoo LXC template, and unpack it to a directory such as /lxc/funtoo0/rootfs/
  2. Create an empty /lxc/funtoo0/fstab file
  3. Ensure c1 line is uncommented (enabled) and c2 through c6 lines are disabled in /lxc/funtoo0/rootfs/etc/inittab

That's almost all you need to get the container filesystem ready to start.

Create Container Configuration Files

Create the following files:

/lxc/funtoo0/config

and also create symlink from

/lxc/funtoo0/config to /etc/lxc/funtoo0.conf

# mkdir /etc/lxc/funtoo0
# ln -s /lxc/funtoo0/config /etc/lxc/funtoo0/config
Note
Daniel Robbins needs to update this config to be more in line with http://wiki.progress-linux.org/software/lxc/ -- this config appears to have nice, refined device node permissions and other goodies. // note by Havis to Daniel, this config is already superior.


Read "man 5 lxc.conf" , to get more information about linux container configuration file.

## Container
lxc.utsname                             = funtoo0
lxc.rootfs                              = /lxc/funtoo0/rootfs/
lxc.arch                                = x86_64
#lxc.console                            = /var/log/lxc/funtoo0.console  # uncomment if you want to log containers console
lxc.tty                                 = 6  # if you plan to use container with physical terminals (eg F1..F6)
#lxc.tty                                = 0  # set to 0 if you dont plan to use the container with physical terminal, also comment out in your containers /etc/inittab  c1 to c6 respawns (e.g. c1:12345:respawn:/sbin/agetty 38400 tty1 linux)
lxc.pts                                 = 1024


## Capabilities
lxc.cap.drop                            = audit_control
lxc.cap.drop                            = audit_write
lxc.cap.drop                            = mac_admin
lxc.cap.drop                            = mac_override
lxc.cap.drop                            = mknod
lxc.cap.drop                            = setfcap
lxc.cap.drop                            = setpcap
lxc.cap.drop                            = sys_admin
#lxc.cap.drop                            = sys_boot # capability to reboot the container
#lxc.cap.drop                            = sys_chroot # required by SSH
lxc.cap.drop                            = sys_module
#lxc.cap.drop                            = sys_nice
lxc.cap.drop                            = sys_pacct
lxc.cap.drop                            = sys_rawio
lxc.cap.drop                            = sys_resource
lxc.cap.drop                            = sys_time
#lxc.cap.drop                            = sys_tty_config # required by getty

## Devices
#lxc.cgroup.devices.allow               = a # Allow access to all devices
lxc.cgroup.devices.deny                 = a # Deny access to all devices

# Allow to mknod all devices (but not using them)
lxc.cgroup.devices.allow                = c *:* m
lxc.cgroup.devices.allow                = b *:* m

lxc.cgroup.devices.allow                = c 1:3 rwm # /dev/null
lxc.cgroup.devices.allow                = c 1:5 rwm # /dev/zero
lxc.cgroup.devices.allow                = c 1:7 rwm # /dev/full
lxc.cgroup.devices.allow                = c 1:8 rwm # /dev/random
lxc.cgroup.devices.allow                = c 1:9 rwm # /dev/urandom
#lxc.cgroup.devices.allow                = c 4:0 rwm # /dev/tty0 ttys not required if you have lxc.tty = 0
#lxc.cgroup.devices.allow                = c 4:1 rwm # /dev/tty1 devices with major number 4 are "real" tty devices
#lxc.cgroup.devices.allow                = c 4:2 rwm # /dev/tty2
#lxc.cgroup.devices.allow                = c 4:3 rwm # /dev/tty3
lxc.cgroup.devices.allow                = c 5:0 rwm # /dev/tty
lxc.cgroup.devices.allow                = c 5:1 rwm # /dev/console
lxc.cgroup.devices.allow                = c 5:2 rwm # /dev/ptmx
lxc.cgroup.devices.allow                = c 10:229 rwm # /dev/fuse
lxc.cgroup.devices.allow                = c 136:* rwm # /dev/pts/* devices with major number 136 are pts
lxc.cgroup.devices.allow                = c 254:0 rwm # /dev/rtc0

## Limits#
lxc.cgroup.cpu.shares                  = 1024
lxc.cgroup.cpuset.cpus                 = 0        # limits container to CPU0
lxc.cgroup.memory.limit_in_bytes       = 512M
lxc.cgroup.memory.memsw.limit_in_bytes = 1G
#lxc.cgroup.blkio.weight                = 500      # requires cfq block scheduler

## Filesystem
#containers fstab should be outside it's rootfs dir (e.g. /lxc/funtoo0/fstab is ok, but /lxc/funtoo0/rootfs/etc/fstab is wrong!!!)
#lxc.mount                               = /lxc/funtoo0/fstab       

#lxc.mount.entry is prefered, because it supports relative paths
lxc.mount.entry                         = proc proc proc nosuid,nodev,noexec  0 0
lxc.mount.entry                         = sysfs sys sysfs nosuid,nodev,noexec,ro 0 0
lxc.mount.entry                         = devpts dev/pts devpts nosuid,noexec,mode=0620,ptmxmode=000,newinstance 0 0
lxc.mount.entry                         = tmpfs dev/shm tmpfs nosuid,nodev,mode=1777 0 0
lxc.mount.entry                         = tmpfs run tmpfs nosuid,nodev,noexec,mode=0755,size=128m 0 0
lxc.mount.entry                         = tmpfs tmp tmpfs nosuid,nodev,noexec,mode=1777,size=1g 0 0

##Example of having /var/tmp/portage as tmpfs in container 
#lxc.mount.entry                         = tmpfs var/tmp/portage tmpfs defaults,size=8g,uid=250,gid=250,mode=0775 0 0
##Example of bind mount
#lxc.mount.entry                        = /srv/funtoo0 /lxc/funtoo0/rootfs/srv/funtoo0 none defaults,bind 0 0

## Network
lxc.network.type                        = veth
lxc.network.flags                       = up
lxc.network.hwaddr                      = #put your MAC address here, otherwise you will get a random one
lxc.network.link                        = br0
lxc.network.name                        = eth0
#lxc.network.veth.pair                   = veth-example

Read "man 7 capabilities" to get more information aboout Linux capabilities.

Above, use the following command to generate a random MAC for lxc.network.hwaddr:

# openssl rand -hex 6 | sed 's/\(..\)/\1:/g; s/.$//'

It is a very good idea to assign a static MAC address to your container using lxc.network.hwaddr. If you don't, LXC will auto-generate a new random MAC every time your container starts, which may confuse network equipment that expects MAC addresses to remain constant.

It might happen from case to case that you aren't able to start your LXC Container with the above generated MAC address so for all these who run into that problem here is a little script that connects your IP for the container with the MAC address. Just save the following code as /etc/lxc/hwaddr.sh, make it executable and run it like /etc/lxc/hwaddr.sh xxx.xxx.xxx.xxx where xxx.xxx.xxx.xxx represents your Container IP.
/etc/lxc/hwaddr.sh:

#!/bin/sh
IP=$*
HA=`printf "02:00:%x:%x:%x:%x" ${IP//./ }`
echo $HA

/lxc/funtoo0/fstab

Note
It is now preferable to have mount entries directly in config file instead of separate fstab:

Edit the file /lxc/funtoo0/fstab:

none /lxc/funtoo0/dev/pts devpts defaults 0 0
none /lxc/funtoo0/proc proc defaults 0 0
none /lxc/funtoo0/sys sysfs defaults 0 0
none /lxc/funtoo0/dev/shm tmpfs nodev,nosuid,noexec,mode=1777,rw 0 0

LXC Networking

  • veth - Virtual Ethernet (bridge)
  • vlan - vlan interface (requires device able to do vlan tagging)
  • macvlan (mac-address based virtual lan tagging) has 3 modes:
    • private
    • vepa (Virtual Ethernet Port Aggregator)
    • bridge
  • phys - dedicated host NIC

Linux Containers and Networking

Enable routing on the host: By default Linux workstations and servers have IPv4 forwarding disabled.

# echo "1" > /proc/sys/net/ipv4/ip_forward
# cat /proc/sys/net/ipv4/ip_forward
# 1

Initializing and Starting the Container

You will probably need to set the root password for the container before you can log in. You can use chroot to do this quickly:

# chroot /lxc/funtoo0/rootfs
(chroot) # passwd
New password: XXXXXXXX
Retype new password: XXXXXXXX
passwd: password updated successfully
(chroot) # exit

Now that the root password is set, run:

# lxc-start -n funtoo0 -d

The -d option will cause it to run in the background.

To attach to the console:

# lxc-console -n funtoo0

You should now be able to log in and use the container. In addition, the container should now be accessible on the network.

To directly attach to container:

# lxc-attach -n funtoo0

To stop the container:

# lxc-stop -n funtoo0

Ensure that networking is working from within the container while it is running, and you're good to go!

Starting LXC container during host boot

  1. You need to create symlink in /etc/init.d/ to /etc/init.d/lxc so that it reflects your container.
  2. ln -s /etc/init.d/lxc /etc/init.d/lxc.funtoo0
  3. now you can add lxc.funtoo0 to default runlevel
  4. rc-update add lxc.funtoo0 default
# rc
 * Starting funtoo0 ...                  [ ok ]

LXC Bugs/Missing Features

This section is devoted to documenting issues with the current implementation of LXC and its associated tools. We will be gradually expanding this section with detailed descriptions of problems, their status, and proposed solutions.

reboot

  • By default, lxc does not support rebooting a container from within. It will simply stop and the host will not know to start it.
  • If you want your container to reboot gracefully, you need sys_boot capability (comment out lxc.cap.drop = sys_boot in your container config)

PID namespaces

Process ID namespaces are functional, but the container can still see the CPU utilization of the host via the system load (ie. in top).

/dev/pts newinstance

  • Some changes may be required to the host to properly implement "newinstance" /dev/pts. See This Red Hat bug.

lxc-create and lxc-destroy

  • LXC's shell scripts are badly designed and are sure way to destruction, avoid using lxc-create and lxc-destroy.

network initialization and cleanup

  • Re-starting a container can result in a failure as network resource are tied up from the already-defunct instance: [1]

graceful shutdown

  • To gracefully shutdown a container, it's init system needs to properly handle kill -PWR signal
  • For funtoo/gentoo make sure that you have:
    • pf:12345:powerwait:/sbin/halt
    • in your containers /etc/inittab
  • For debian/ubuntu make sure that you have:
    • pf::powerwait:/sbin/shutdown -t1 -a -h now
    • in your container /etc/inittab
    • and also comment out other line starting with pf:powerfail (such as pf::powerwait:/etc/init.d/powerfail start) <- these are used if you have UPS monitoring daemon installed!
  • /etc/init.d/lxc seems to have broken support for graceful shutdown (it sends proper signal, but then also tries to kill the init with lxc-stop)

funtoo

  • Our udev should be updated to contain -lxc in scripts. (This has been done as of 02-Nov-2011, so should be resolved. But not fixed in our openvz templates, so need to regen them in a few days.)
  • Our openrc should be patched to handle the case where it cannot mount tmpfs, and gracefully handle this situation somehow. (Work-around in our docs above, which is to mount tmpfs to /libexec/rc/init.d using the container-specific fstab file (on the host.)
  • Emerging udev within a container can/will fail when realdev is run, if a device node cannot be created (such as /dev/console) if there are no mknod capabilities within the container. This should be fixed.

References

  • man 7 capabilities
  • man 5 lxc.conf

Links