The Funtoo Linux project has transitioned to "Hobby Mode" and this wiki is now read-only.
Difference between revisions of "LXD"
(update /etc/sub{u,g}id discussion to reflect upstream documentation and experience with new Funtoo install) |
|||
Line 57: | Line 57: | ||
<!--T:11--> | <!--T:11--> | ||
Next, we come to the concept of "subuid" and "subgid". Typically, a user will get one user id and one group id. Subids and subgids allow us to assign additional UIDs and GIDs to a user for their own uses. | Next, we come to the concept of "subuid" and "subgid". Typically, a user will get one user id and one group id. Subids and subgids allow us to assign additional UIDs and GIDs to a user for their own uses. Per the documentation: | ||
:If some but not all of /etc/subuid, /etc/subgid, newuidmap (path lookup) and newgidmap (path lookup) can be found on the system, LXD will fail the startup of any container until this is corrected as this shows a broken shadow setup.<ref>[https://documentation.ubuntu.com/lxd/en/latest/userns-idmap/#allowed-ranges]</ref> | |||
As noted above, it is no longer true that LXD will allocate subuids for the root user in all cases. A good default configuration is that given by the following files on the root filesystem: | |||
</translate> | </translate> | ||
Line 74: | Line 77: | ||
<!--T:12--> | <!--T:12--> | ||
The format of both of these files are "user":"start":"count". Meaning that the {{c|root}} user will be allocated "count" IDs starting at the position "start". The reason why LXD does this is because | The format of both of these files are "user":"start":"count". Meaning that the {{c|root}} user will be allocated "count" IDs starting at the position "start". The reason why LXD does this is because | ||
these extra IDs will be used to isolate containers from the host processes and optionally from each other, by using different offsets so that their UID and GIDs will not overlap. | these extra IDs will be used to isolate containers from the host processes and optionally from each other, by using different offsets so that their UID and GIDs will not overlap. For some systems (those lacking `newuidmap` and `newgidmap`, according to the documentation), LXD 5 now | ||
has these settings "baked in" | has these settings "baked in". For more information on subids and subgids, see [[LXD/What are subuids and subgids?|What are subuids and subgids?]]. | ||
==== LXD-in-LXD ==== <!--T:142--> | ==== LXD-in-LXD ==== <!--T:142--> | ||
<!--T:143--> | <!--T:143--> | ||
After the initial setup, the only time you will now need to edit {{f|/etc/subuid}} and {{f|/etc/subgid}} now is if you are running "LXD-in-LXD". In this case, the inner LXD (within the container) will need to reduce these | |||
subuid and subgid mappings as the full range will not be available. This should be possible by simply using the following settings within your containerized LXD instance: | subuid and subgid mappings as the full range will not be available. This should be possible by simply using the following settings within your containerized LXD instance: | ||
Revision as of 21:55, January 22, 2024
Introduction
Please note that if you plan to use LXD on a laptop, you are likely using WiFi and NetworkManager, and the steps below will not work for you bridge setup. Please see LXD/Laptop Network Setup for important differences to allow you to use LXD in 'dev mode' for local use of containers for development.
LXD is a container "hypervisor" designed to provide an easy set of tools to manage Linux containers, and its development is currently being led by employees at Canonical. You can learn more about the project in general at https://linuxcontainers.org/lxd/.
LXD is currently used for container infrastructure for Funtoo Containers and is also very well-supported under Funtoo Linux. For this reason, it's recommended that you check out LXD and see what it can do for you.
Basic Setup on Funtoo
The following steps will show you how to set up a basic LXD environment under Funtoo Linux. This environment will essentially use the default LXD setup -- a will be created called lxdbr0
which will use NAT to provide Internet access to your containers. In addition, a default storage pool will be created that will simply use your existing filesystem's storage, creating a directory at /var/lib/lxd/storage-pools/default
to store any containers you create. More sophisticated configurations are possible that use dedicated network bridges connected to physical interfaces without NAT, as well as dedicated storage pools that use ZFS and btrfs -- however, these types of configurations are generally overkill for a developer workstation and should only be attempted by advanced users. So we won't cover them here.
Requirements
This section will guide you through setting up the basic requirements for creating an LXD environment.
The first step is to emerge LXD and its dependencies. Perform the following:
root # emerge -a lxd
Once LXD is done emerging, we will want to enable it to start by default:
root # rc-update add lxd default
In addition, we will want to set up the following files. /etc/security/limits.conf
should be modified to have the following lines in it:
/etc/security/limits.conf
* soft nofile 1048576
* hard nofile 1048576
root soft nofile 1048576
root hard nofile 1048576
* soft memlock unlimited
* hard memlock unlimited
# End of file
Next, we come to the concept of "subuid" and "subgid". Typically, a user will get one user id and one group id. Subids and subgids allow us to assign additional UIDs and GIDs to a user for their own uses. Per the documentation:
- If some but not all of /etc/subuid, /etc/subgid, newuidmap (path lookup) and newgidmap (path lookup) can be found on the system, LXD will fail the startup of any container until this is corrected as this shows a broken shadow setup.[1]
As noted above, it is no longer true that LXD will allocate subuids for the root user in all cases. A good default configuration is that given by the following files on the root filesystem:
/etc/subuid
root:1000000:10000000000
/etc/subgid
root:1000000:10000000000
The format of both of these files are "user":"start":"count". Meaning that the root
user will be allocated "count" IDs starting at the position "start". The reason why LXD does this is because
these extra IDs will be used to isolate containers from the host processes and optionally from each other, by using different offsets so that their UID and GIDs will not overlap. For some systems (those lacking `newuidmap` and `newgidmap`, according to the documentation), LXD 5 now
has these settings "baked in". For more information on subids and subgids, see What are subuids and subgids?.
LXD-in-LXD
After the initial setup, the only time you will now need to edit /etc/subuid
and /etc/subgid
now is if you are running "LXD-in-LXD". In this case, the inner LXD (within the container) will need to reduce these
subuid and subgid mappings as the full range will not be available. This should be possible by simply using the following settings within your containerized LXD instance:
/etc/subuid
root:65537:70000
/etc/subgid
root:65537:70000
If you are not using advanced features of LXD, your LXD-in-LXD instance should now have sufficient id mappings to isolate container-containers from the host-container. The only remaining step for LXD-in-LXD would be to allow the host-container to nest:
root # lxc config set host-container security.nesting true
This will allow for host-container contain containers itself :)
Initialization
To configure LXD, first we will need to start LXD. This can be done as follows:
root # /etc/init.d/lxd start
At this point, we can run lxd init
to run a configuration wizard to set up LXD:
root # lxd init Would you like to use LXD clustering? (yes/no) [default=no]: ↵ Do you want to configure a new storage pool? (yes/no) [default=yes]: ↵ Name of the new storage pool [default=default]: ↵ Name of the storage backend to use (btrfs, dir, lvm) [default=btrfs]: dir ↵ Would you like to connect to a MAAS server? (yes/no) [default=no]: ↵ Would you like to create a new local network bridge? (yes/no) [default=yes]: ↵ What should the new bridge be called? [default=lxdbr0]: ↵ What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: ↵ What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: none ↵ Would you like LXD to be available over the network? (yes/no) [default=no]: ↵ Would you like stale cached images to be updated automatically? (yes/no) [default=yes] ↵ Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: ↵ root #
As you can see, we chose all the default except for:
- storage pool
- We opted for using a directory-based container storage rather than btrfs volumes. Directory-based may be the default option during LXD configuration -- it depends if you have btrfs-tools installed or not.
- IPv6 address
- It is recommended you turn this off unless you are specifically wanting to play with IPv6 in your containers. It may cause
dhcpcd
in your container to only retrieve an IPv6 address if you leave it enabled. This is great if you have IPv6 working -- otherwise, you'll get a dud IPv6 address and no IPv4 address, and thus no network.
As explained above, turn off IPv6 NAT in LXD unless you specifically intend to use it! It can confuse dhcpcd
.
If you choose to output the YAML lxd init preseed configuration from the lxd init
command above, here is a config example:
lxc_init_preseed.yaml
(YAML source code) - lxc init preseed config exampleconfig:
images.auto_update_interval: "0"
networks:
- config:
ipv4.address: auto
ipv6.address: none
description: ""
name: lxdbr0
type: ""
project: default
storage_pools:
- config: {}
description: ""
name: default
driver: dir
profiles:
- config: {}
description: ""
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: funtoo
type: disk
name: default
projects: []
cluster: null
Now, we should be able to run lxc image list
and get a response from the LXD daemon:
root # lxc image list +-------+-------------+--------+-------------+------+------+-------------+ | ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCH | SIZE | UPLOAD DATE | +-------+-------------+--------+-------------+------+------+-------------+ root #
If you are able to do this, you have successfully set up the core parts of LXD! Note that we used the command lxc
and not lxd
like we did for lxd init
-- from this point forward, you will use the lxc
command. Don't let this
confuse you -- the lxc
command is the primary command-line tool for working with LXD containers.
Above, you can see that no images are installed. Images are installable snapshots of containers that we can use to create new containers ourselves. So, as a first step, let's go ahead and grab an image we can use. You will want to browse https://build.funtoo.org for an LXD image that will work on your computer hardware. For example, I was able to download
the following file using wget
:
root # wget https://build.funtoo.org/1.4-release-std/x86-64bit/amd64-zen2/2022-04-13/lxd-amd64-zen2-1.4-release-std-2022-04-13.tar.xz
Once downloaded, this image can be installed using the following command:
root # lxc image import lxd-amd64-zen2-1.4-release-std-2022-04-13.tar.xz --alias funtoo Image imported with fingerprint: fe4d27fb31bfaf3bd4f470e0ea43d26a6c05991de2a504b9e0a3b1a266dddc69
Now you will see the image available in our image list:
root # lxc image list +--------+--------------+--------+-----------------------------------------+--------------+-----------+----------+------------------------------+
First Container
It is now time to launch our first container. This can be done as follows:
root # lxc launch funtoo testcontainer Creating testcontainer Starting testcontainer
We can now see the container running via lxc list
:
root # lxc list +---------------+---------+------+-----------------------------------------------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +---------------+---------+------+-----------------------------------------------+------------+-----------+ | testcontainer | RUNNING | | fd42:8063:81cb:988c:216:3eff:fe2a:f901 (eth0) | PERSISTENT | | +---------------+---------+------+-----------------------------------------------+------------+-----------+ root #
By default, our new container testcontainer
will use the default profile, which will connect an eth0
interface in the container to NAT, and will also use our directory-based LXD storage pool. We can now enter the container as follows:
root # lxc exec testcontainer -- su --login testcontainer #
As you might have noticed, we do not yet have any IPv4 networking configured. While LXD has set up a bridge and NAT for us, along with a DHCP server to query, we actually need to use dhcpcd
to query for an IP address, so let's get that set up:
testcontainer # echo "template=dhcpcd" > /etc/conf.d/netif.eth0 testcontainer # cd /etc/init.d testcontainer # ln -s netif.tmpl netif.eth0 testcontainer # rc-update add netif.eth0 default * service netif.eth0 added to runlevel default testcontainer # rc * rc is deprecated, please use openrc instead. * Caching service dependencies ... [ ok ] * Starting DHCP Client Daemon ... [ ok ] * Network dhcpcd eth0 up ... [ ok ] testcontainer #
You can now see that eth0
has a valid IPv4 address:
testcontainer # ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.212.194.17 netmask 255.255.255.0 broadcast 10.212.194.255 inet6 fd42:8063:81cb:988c:25ea:b5bd:603d:8b0d prefixlen 64 scopeid 0x0<global> inet6 fe80::216:3eff:fe2a:f901 prefixlen 64 scopeid 0x20<link> ether 00:16:3e:2a:f9:01 txqueuelen 1000 (Ethernet) RX packets 45 bytes 5385 (5.2 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 20 bytes 2232 (2.1 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
What happened is that LXD set up a DHCP server for us (dnsmasq) running on our private container network, and automatically offers IP addresses to our containers. It also configured iptables for us to NAT the connection so that outbound Internet access should magically work.
You should also be able to see this IPv4 address listed in the container list when you type lxc list
on your host system.
Network Troubleshooting
Proceed with caution. These are advanced LXC commands that can modify the state of your LXD local network setup. In extreme scenarios, all of your LXD containers' networks NAT routing can easily break all if the wrong LXC config key value is changed
Note that if you are having issues with your container getting an IPv4 address via DHCP, make sure that you turn IPv6 off in LXD. Do this by running:
root # lxc network edit lxdbr0
Then, change ipv6.nat
YAML key value to "false"
and restart LXD and the test container:
root # /etc/init.d/lxd restart root # lxc restart testcontainer
This should resolve the issue.
If you have initialized your LXD cluster by turning off IPv6 with the What IPv6 address should be used? option set to none
, then the ipv6.nat
will not even be present in our LXC local network's lxdbr0
bridge interface.
Be careful not to tamper with the ipv4.nat
setting or all LXD container NAT routing will break, meaning no network traffic within running and new LXD containers will be able to route to external to the Internet!
Here is some example YAML of a default lxdbr0
LXC local network bridge device for reference:
lxc_lxdbr0.yaml
(YAML source code) - lxc network edit lxdbr0config:
ipv4.address: 10.239.139.1/24
ipv4.nat: "true"
ipv6.address: none
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/profiles/default
- /1.0/instances/funtoo-livestream
managed: true
status: Created
locations:
- none
Container Console setup
LXD containers are created with a special `/dev` where there is only one tty console `/dev/console` that could be used to see bootstrap phase or to enter in the container from the container console.
This could be done in two way:
- on launch phase:
root # lxc launch -p default -p net macaroni:funtoo/next-stage3 testcontainer -e --console
- through the `lxc console` command:
root # lxc console testcontainer
To use correctly this feature is needed setup correctly the `/etc/inittab` file and disable the standard c* entries:
root # sed -i /etc/inittab -e 's|^#x1|x1|g' -e '/^c[0-9].*/d'
The images exposed over the Simplestreams Server of Funtoo Macaroni are already configured correctly and ready to use.
Finishing Steps
Assuming your network is now working, you are ready to start using your new Funtoo container. Time to have some fun! Go ahead and run ego sync
and then emerge your favorite things:
testcontainer # ego sync Syncing meta-repo Cloning into '/var/git/meta-repo'...