LXD/LXD in LXD

From Funtoo
< LXD
Jump to: navigation, search

Running LXD inside your LXD container is possible. Here is a summary of changes and settings to get this all working.

Changes on the host

We want to run unprivileged containers in our LXD and for that we need bigger uid/gid space. The way unprivileged containers are created is by taking a set of normal UIDs and GIDs from the host, usually at least 65536 of each (to be POSIX compliant) and mapping those into the container.

Changing the UID/GID mapping

Generally this isn’t too complicated. If you wish to run container c3 in container c2 in container c1, you’ll need 65536 uids in c3; in c2 you’ll need 65536 for c2 itself plus the 65536 for c3; and in c1 you’ll need 65536 for c1 plus 65536 for c2 plus 65536 for c3.

So for two levels of nesting we need at least 196608 uid/gids. Let's bump this number in our /etc/subuid and /etc/subgid on the host.

root # cat /etc/subuid
lxd:100000:200000
root:100000:200000

root # cat /etc/subgid
lxd:100000:200000
root:100000:200000

Now we need to get lxd on the host get aware of the new settings by restarting it. We will get this information confirmed in the lxd.log

root # tail /var/log/lxd/lxd.log
lvl=info msg="LXD 2.21 is starting in normal mode" path=/var/lib/lxd t=2018-02-12T00:51:34+0100
lvl=info msg="Kernel uid/gid map:" t=2018-02-12T00:51:34+0100
lvl=info msg=" - u 0 0 4294967295" t=2018-02-12T00:51:34+0100
lvl=info msg=" - g 0 0 4294967295" t=2018-02-12T00:51:34+0100
lvl=info msg="Configured LXD uid/gid map:" t=2018-02-12T00:51:34+0100
lvl=info msg=" - u 0 100000 200000" t=2018-02-12T00:51:34+0100
lvl=info msg=" - g 0 100000 200000" t=2018-02-12T00:51:34+0100
...

Now we need to change the settings of uid/gid for the container and allow nesting inside container. After the subuid/subgid change on the host all new containers wil be created with the new values like this:

root # lxc init fun-generic fun-nested-test
Creating fun-nested-test
root # lxc config show fun-nested-test                                                                             
architecture: x86_64                                                                                                              
config:                                                                                                                           
  image.description: Funtoo Current Generic_64 64bit  2018-02-11                                                                  
  image.name: funtoo-current-x86-64bit-generic_64                                                                                 
  volatile.base_image: 3fa152fc69f9e18ab30d25b491fd7d263631f9fa5e693e4dbe4e2322bcb4cbb2                                           
  volatile.eth0.hwaddr: 00:16:3e:bd:21:3a                   
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":200000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":200000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":200000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":200000}]'
  volatile.last_state.power: STOPPED
devices: {}                                                                                                                                                                                          
ephemeral: false                       
profiles:                                              
- default                                                                                                                                                                                            
stateful: false                        
description: ""

For the already created containers we need to change the Maprange. This should be possible to do with help of security.idmap.size, but this didn't work for me.

root # lxc config show fun-nested-old
architecture: x86_64
config:
  volatile.base_image: d541eed5e558dd78a7c26708d0b62d41feaae38871312991de90c4ede211fdfc
  volatile.eth0.hwaddr: 00:16:3e:e1:6f:13
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: STOPPED
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

As you can see here the Maprange is 65536. Issuing next command and restarting the container should grow the uid/gid maprange.

root # lxc config set fun-nested-old security.idmap.size 200000
root # lxc restart fun-nested-old
root # lxc config get fun-nested-old volatile.last_state.idmap

If it didn't work for you we can edit the config manually and change the

volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'

to

volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":200000},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":200000}]'

Allowing nesting inside container

Allowing the child container to run LXD instance is done by issuing this command:

root # lxc config set fun-nested-test security.nesting true

This should be sufficient to run a nested container setup at least from the host's perspective.

Preparing the container to host LXD

   Note

Currently the containers are not setup to support this, if you would like to play with it, please let us know over at bugs.funtoo.org

Now there are couple of changes that you need to do inside your container, all the commands that will be here in this section are run inside the container.

Changing UID/GID inside container

We are going to change the entries in following manner:

root # cat /etc/subuid
lxd:65536:131072
root:65536:131072

root # cat /etc/subgid
lxd:65536:131072
root:65536:131072

Next install LXD start it and run lxd init and check your logs if LXD picked up the correct values.

root # tail /var/log/lxd/lxd.log
lvl=info msg="LXD 2.21 is starting in normal mode" path=/var/lib/lxd t=2018-02-11T18:52:35+0000
lvl=info msg="Kernel uid/gid map:" t=2018-02-11T18:52:35+0000
lvl=info msg=" - u 0 100000 200000" t=2018-02-11T18:52:35+0000
lvl=info msg=" - g 0 100000 200000" t=2018-02-11T18:52:35+0000
lvl=info msg="Configured LXD uid/gid map:" t=2018-02-11T18:52:35+0000
lvl=info msg=" - u 0 65536 131072" t=2018-02-11T18:52:35+0000
lvl=info msg=" - g 0 65536 131072" t=2018-02-11T18:52:35+0000
...

So as you can see LXD is reporting the assigned uid/gid map from the host as kernel maps and LXD maps are configured according to our /etc/subuid /etc/subgid settings.

Well that's it. Now you have a LXD running inside your container.

Various notes

  • Cannot start unprivileged nested container.
lxc 20170621083248.584 ERROR    lxc_start - start.c:must_drop_cap_sys_boot:641 - Failed to clone (0x30000011): Operation not permitted (includes CLONE_NEWUSER).
lxc 20170621083248.586 ERROR    lxc_namespace - namespace.c:lxc_clone:67 - Failed to clone (0x3c020000): Operation not permitted.
    • echo "kernel.unprivileged_userns_clone=1" > /etc/sysctl.d/80-lxc-userns.conf
    • sysctl -f /etc/sysctl.d/80-lxc-userns.conf
  • LXCFS "fusermount: mount failed: Operation not permitted"
    • lxcfs is not needed in nested setup, because it gets mounted from the host container.