Difference between pages "Linux Containers" and "Cross-compiling with Crossdev"

From Funtoo
(Difference between pages)
Jump to navigation Jump to search
 
m (Update link to gentoo handbook crossdev page)
 
Line 1: Line 1:
Linux Containers, or LXC, is a Linux feature that allows Linux to run one or more isolated virtual systems (with their own network interfaces, process namespace, user namespace, and power state) using a single Linux kernel on a single server.  
== Introduction ==
Crossdev is a great feature of Funtoo that allows you to easily make a basic toolchain for your embedded devices and alternate platform computers.  It can be used for distcc for example, so a slow x86 or ARM computer can be assisted on compiles by a much faster x86_64 computer(s).  Some, but not all, packages can also be compiled stand-alone without distcc on the faster machine.  You can also use the faster 64 bit machine to make kernels for these slower machines, a big advantage.


== Status ==
This is quite a big subject and the aim of this article is mainly to help new users with some Funtoo specific things relating to cross-compiling that are unique to Funtoo.  So we won't waste time here on things that are covered elsewhere, only those specific things.
== Let's get started! ==
The first step is to install crossdev.  Crossdev is written by Mike Frysinger who is quite a busy fellow and does a superb job.  But occasionally there are little hiccups with crossdev.  If ever you are having a problem with crossdev it doesn't hurt to try downgrading to previous versions.  I say this because on the date this article is written, 11-1-14, there are issues with the two latest versions of crossdev - 20140917 and 20141030 (see https://bugs.funtoo.org/browse/FL-1703).  But version 20140729 works flawlessly.  I have seen this happen from time to time, nothing to worry about.
<console>
###i## emerge -av crossdev
</console>
== Funtoo Specific Considerations ==
The most important thing to understand is that Funtoo gcc ebuilds are different from Gentoo ebuilds.  This is because Daniel Robbins has made improvements to gcc to help with general usage.  Unfortunately, a side effect has been that they no longer work with crossdev.  But not to worry, there is an easy way around this issue and my understanding is that Funtoo intends to rectify it at a later date.


As of Linux kernel 3.1.5, LXC is usable for isolating your own private workloads from one another. It is not yet ready to isolate potentially malicious users from one another or the host system. For a more mature containers solution that is appropriate for hosting environments, see [[OpenVZ]].
Another issue is that you cannot have a toolchain with exactly the same minor number as the current gcc on your system.  You must use an ebuild of a slightly different vintage.  I believe this issue occurs for Gentoo users also.  For example, at the present time Funtoo current uses sys-devel/gcc-4.8.2-r3.  So you could use gcc-4.8.1, or 4.8.0 but not any version of 4.8.2 (the revision -rx doesn't matter, can't do it). If you try to do that gcc will compile and then fail in the installation phase due to file collisions. In some cases with embedded devices you may wish to use much older versions, in which case this becomes a moot point.


LXC containers don't yet have their own system uptime, and they see everything that's in the host's <tt>dmesg</tt> output, among other things. But in general, the technology works.
Finally, a general issue with using crossdev with any distro is that you must sometimes juggle the various versions of binutils, gcc, glibc and linux-headers so that they are compatible with one another. This is the easiest problem to deal with, just use versions of the same time vintage, in extreme cases you may need to start changing revisions to get all of them to compile.
 
== Control groups ==
 
* Control groups (cgroups) Implemented as VFS, since 2.6.24
** Allows aggregation of tasks
* Subsystems (cpuset, memory, blkio,...)
** accounting, limitation, priorizatio,...
* No disk quota limitation ( -> image file, LVM, XFS, directory tree quota)
 
== Subsystems ==


That's it, that's what you need to know.
== GCC Ebuilds ==
The workaround to make gcc compile is to use Gentoo gcc ebuilds.  They are readily available at the Gentoo attic: http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/?hid
Going back historically for as far as you'd ever need.  Go to the sys-devel/gcc folder and right click on a version and copy the link.  I suggest a version that isn't even in the Funtoo tree, then there is no ambiguity about what is being used.
Next copy it to your local repository, we'll assume /usr/local/portage here.  So if you haven't already got a sys-devel/gcc directory there you could do
<console>
###i## install -d /usr/local/portage/sys-devel/gcc
</console>
Now go to that directory in your terminal and wget <the link you copied>.  At the end will be some fluff you don't want, delete all the way back to the word .ebuild and hit enter.  In a second or so you'll have that ebuild in your repository.
If you haven't done this before you'll need to link the files directory from the tree, so that it can find patches, etc.  So do this
<console>
<console>
cat /proc/cgroups
###i## ln -s /usr/portage/sys-devel/gcc/files .
#subsys_name hierarchy num_cgroups enabled
cpuset 2 15 1
cpu 3 15 1
cpuacct 4 15 1
memory 5 9 1
devices 6 9 1
freezer 7 9 1
blkio 8 9 1
perf_event 9 9 1
hugetlb 10 6 1
</console>
</console>
 
Finally you must create the manifest
 
cpuset    -> limits tasks to specific CPU/CPUs
cpu        -> CPU shares
cpuacct    -> CPU accounting
memory    -> memory and swap limitation and accounting
devices    -> device allow deny list
freezer    -> suspend/resume tasks
blkio      -> I/O priorization (weight, throttle, ...)
perf_event -> support for per-cpu per-cgroup monitoring [http://lwn.net/Articles/421574/ perf_events]
hugetlb    -> cgroup resource controller for HugeTLB pages  [http://lwn.net/Articles/499255/ hugetlb]
 
== Configuring the Funtoo Host System ==
 
=== Install LXC kernel ===
Any kernel beyond 3.1.5 will probably work. Personally I prefer the sys-kernel/gentoo-sources-3.4.9 as these have support for all the namespaces without sacrificing the xfs, FUSE or NFS support for example. These checks were introduced later starting from kernel 3.5, this could also mean that the user namespace is not working optimally.
 
* User namespace (EXPERIMENTAL) depends on EXPERIMENTAL and on UIDGID_CONVERTED
** config UIDGID_CONVERTED
*** True if all of the selected software components are known to have uid_t and gid_t converted to kuid_t and kgid_t where appropriate and are otherwise safe to use with the user namespace.
**** Networking - depends on NET_9P = n
**** Filesystems - 9P_FS = n, AFS_FS = n, AUTOFS4_FS = n, CEPH_FS = n, CIFS = n, CODA_FS = n, FUSE_FS = n, GFS2_FS = n, NCP_FS = n, NFSD = n, NFS_FS = n, OCFS2_FS = n, XFS_FS = n
**** Security options - Grsecurity - GRKERNSEC = n (if applicable)
 
** As of 3.10.xx kernel, all of the above options are safe to use with User namespaces, except for XFS_FS, therefore with kernel >=3.10.xx, you should answer XFS_FS = n, if you want User namespaces support.
 
==== Kernel configuration ====
These options should be enable in your kernel to be able to take full advantage of LXC.
 
* General setup
** CONFIG_NAMESPACES
*** CONFIG_UTS_NS
*** CONFIG_IPC_NS
*** CONFIG_PID_NS
*** CONFIG_NET_NS
*** CONFIG_USER_NS
** CONFIG_CGROUPS
*** CONFIG_CGROUP_DEVICE
*** CONFIG_CGROUP_SCHED
*** CONFIG_CGROUP_CPUACCT
*** CONFIG_CGROUP_MEM_RES_CTLR (in 3.6+ kernels it's called CONFIG_MEMCG)
*** CONFIG_CGROUP_MEM_RES_CTLR_SWAP (in 3.6+ kernels it's called CONFIG_MEMCG_SWAP)
*** CONFIG_CPUSETS (on multiprocessor hosts)
* Networking support
** Networking options
*** CONFIG_VLAN_8021Q
* Device Drivers
** Character devices
*** Unix98 PTY support
**** CONFIG_DEVPTS_MULTIPLE_INSTANCES
** Network device support
*** Network core driver support
**** CONFIG_VETH
**** CONFIG_MACVLAN
 
Once you have lxc installed, you can then check your kernel config with:
<console>
<console>
# ##i##CONFIG=/path/to/config /usr/sbin/lxc-checkconfig
###i## ebuild gcc-<version>.ebuild manifest
</console>
</console>
 
And now you are ready!
=== Emerge lxc ===
== Really, that's it? ==
Yes.  Well, one other thing that you may need to do, if you've chosen a gcc version that is in the Funtoo tree, crossdev may get confused and try to install the tree version.  At the beginning of the crossdev run it creates links in the crossdev repository, most people will just want this to be their local repository.  So you can let crossdev run for a few seconds to create those links and then cancel it with ctl-c.  Then go to the crossdev repository and delete the gcc link and replace it with a link to /usr/local/portage/sys-devel/gcc.  Here's an example in my local repository:
<console>
<console>
# ##i##emerge -av app-emulation/lxc
###i## ls -l /usr/local/portage/cross-avr
total 0
lrwxrwxrwx 1 root root 50 Nov  1 00:41 avr-libc ->/usr/portage/dev-embedded/avr-libc
lrwxrwxrwx 1 root root 47 Nov  1 00:41 binutils -> /usr/portage/sys-devel/binutils
##g##lrwxrwxrwx 1 root root 32 Nov  1 00:41 gcc -> /usr/local/portage/sys-devel/gcc##!g##
lrwxrwxrwx 1 root root 42 Nov  1 00:41 gdb -> /usr/portage/sys-devel/gdb
</console>
</console>
=== Configure Networking For Container ===
Note that the link for gcc is different from the others, which point to the normal Funtoo Portage tree, it points to my local repository, where the Gentoo ebuilds are stored.  Most of the time also you would be linked to sys-libs/glibc, avr-libc is a bit unusual but normal for avr cross toolchains.


Typically, one uses a bridge to allow containers to connect to the network. This is how to do it under Funtoo Linux:
Once the link is fixed (if it even needed fixing), just hit your up arrow to get the same crossdev command line again and let it run through this time.  You'll be rewarded with a shiny new cross toolchain at the end..


# create a bridge using the Funtoo network configuration scripts. Name the bridge something like <tt>brwan</tt> (using <tt>/etc/init.d/netif.brwan</tt>). Configure your bridge to have an IP address.
Of course, there is a bit more to making a cross toolchain, but as stated at the beginning of this article, only the unique differences for Funtoo are shown here.
# Make your physical interface, such as <tt>eth0</tt>, an interface with no IP address (use the Funtoo <tt>interface-noip</tt> template.)
You should find nearly all the rest you need to know at https://www.gentoo.org/proj/en/base/embedded/handbook/index.xml?style=printable&part=1&chap=2. Only the top 1/4 page is needed.
# Make <tt>netif.eth0</tt> a slave of <tt>netif.brwan</tt> in <tt>/etc/conf.d/netif.brwan</tt>.
# Enable your new bridged network and make sure it is functioning properly on the host.


You will now be able to configure LXC to automatically add your container's virtual ethernet interface to the bridge when it starts, which will connect it to your network.
One note, at the current time (11-1-14) you need to preface your crossdev command line statement with USE="-sanitize". This is a known bug (not Funtoo specific) that will soon be fixed in crossdev. You will find -sanitize in the package.use files, but you still must do it on the command line, else gcc will fail. For example the following command line was successful for me today
 
<console>
== Setting up a Funtoo Linux LXC Container ==
###i## USE="-sanitize" crossdev --binutils 2.23-2 --g 4.8.1-r1 --l 1.8.0 -t avr -P -v
 
</console>
Here are the steps required to get Funtoo Linux running <i>inside</i> a container. The steps below show you how to set up a container using an existing Funtoo Linux OpenVZ template. It is now also possible to use [[Metro]] to build an lxc container tarball directly, which will save you manual configuration steps and will provide an <tt>/etc/fstab.lxc</tt> file that you can use for your host container config. See [[Metro Recipes]] for info on how to use Metro to generate an lxc container.
 
=== Create and Configure Container Filesystem ===
 
# Start with a Funtoo LXC template, and unpack it to a directory such as <tt>/var/lib/lxc/funtoo</tt>.
# Create an empty <tt>/var/lib/lxc/funtoo/etc/fstab</tt> file.
# Ensure <tt>c1</tt> line is uncommented (enabled) and <tt>c2</tt> through <tt>c6</tt> lines are disabled in <tt>/var/lib/lxc/funtoo/etc/inittab</tt>.
 
That's almost all you need to get the container filesystem ready to start.
 
=== Create Container Configuration Files ===
 
Create the following files:
 
==== <tt>/etc/lxc/funtoo/config</tt> ====
 
{{fancynote|Daniel Robbins needs to update this config to be more in line with http://wiki.progress-linux.org/software/lxc/ -- this config appears to have nice, refined device node permissions and other goodies.}}
 
Read "man 5 lxc.conf" , to get more information about linux container configuration file.
<pre>
## Container
lxc.utsname                            = funtoo
lxc.rootfs                              = /var/lib/lxc/funtoo
lxc.arch                                = x86_64
lxc.tty                                = 6
lxc.pts                                = 1024
#lxc.console                            = /var/log/lxc/funtoo.console
 
## Capabilities
lxc.cap.drop                            = sys_admin sys_module mac_admin mac_override
 
## Devices
# Allow all devices
#lxc.cgroup.devices.allow              = a
# Deny all devices
lxc.cgroup.devices.deny                = a
# Allow to mknod all devices (but not using them)
lxc.cgroup.devices.allow                = c *:* m
lxc.cgroup.devices.allow                = b *:* m
# /dev/console
lxc.cgroup.devices.allow                = c 5:1 rwm
# /dev/fuse
lxc.cgroup.devices.allow                = c 10:229 rwm
# /dev/null
lxc.cgroup.devices.allow                = c 1:3 rwm
# /dev/ptmx
lxc.cgroup.devices.allow                = c 5:2 rwm
# /dev/pts/*
lxc.cgroup.devices.allow                = c 136:* rwm
# /dev/random
lxc.cgroup.devices.allow                = c 1:8 rwm
# /dev/rtc
lxc.cgroup.devices.allow                = c 254:0 rwm
# /dev/tty
lxc.cgroup.devices.allow                = c 5:0 rwm
# /dev/urandom
lxc.cgroup.devices.allow                = c 1:9 rwm
# /dev/zero
lxc.cgroup.devices.allow                = c 1:5 rwm
 
## Limits#
lxc.cgroup.cpu.shares                  = 1024
#lxc.cgroup.cpuset.cpus                = 0
#lxc.cgroup.memory.limit_in_bytes      = 256M
#lxc.cgroup.memory.memsw.limit_in_bytes = 1G
 
## Filesystem
lxc.mount                              = /etc/lxc/funtoo/fstab
#lxc.mount.entry                        = proc /var/lib/lxc/example.org/rootfs/proc proc nodev,noexec,nosuid 0 0
#lxc.mount.entry                        = sysfs /var/lib/lxc/example.org/rootfs/sys sysfs defaults,ro 0 0
#lxc.mount.entry                        = /srv/share/example.org /var/lib/example.org/rootfs/srv/example.org none defaults,bind 0 0
 
## Network
lxc.network.type                        = veth
lxc.network.flags                      = up
lxc.network.hwaddr                      = #put your MAC address here, otherwise you will get a random one
lxc.network.link                        = br0
lxc.network.name                        = eth0
#lxc.network.veth.pair                  = veth-example
</pre>
 
Read "man 7 capabilities" to get more information aboout Linux capabilities.
 
Above, use the following command to generate a random MAC for <tt>lxc.network.hwaddr</tt>:
 
<pre>
# openssl rand -hex 6 | sed 's/\(..\)/\1:/g; s/.$//'
</pre>
 
It is a very good idea to assign a static MAC address to your container using <tt>lxc.network.hwaddr</tt>. If you don't, LXC will auto-generate a new random MAC every time your container starts, which may confuse network equipment that expects MAC addresses to remain constant.
 
It might happen from case to case that you aren't able to start your LXC Container with the above generated MAC address so for all these who run into that problem here is a little script that connects your IP for the container with the MAC address. Just save the following code as <tt>/etc/lxc/hwaddr.sh</tt>, make it executable and run it like <tt>/etc/lxc/hwaddr.sh xxx.xxx.xxx.xxx</tt> where xxx.xxx.xxx.xxx represents your Container IP.
 
<pre>
#!/bin/sh
IP=$*
HA=`printf "02:00:%x:%x:%x:%x" ${IP//./ }`
echo $HA
</pre>
 
==== <tt>/etc/lxc/funtoo/fstab</tt> ====
 
<pre>
none /lxc/funtoo/dev/pts devpts defaults 0 0
none /lxc/funtoo/proc proc defaults 0 0
none /lxc/funtoo/sys sysfs defaults 0 0
none /lxc/funtoo/dev/shm tmpfs nodev,nosuid,noexec,mode=1777,rw 0 0
</pre>
 
== Initializing and Starting the Container ==
 
You will probably need to set the root password for the container before you can log in. You can use chroot to do this quickly:
 
<pre>
# chroot /lxc/funtoo
(chroot) # passwd
New password: XXXXXXXX
Retype new password: XXXXXXXX
passwd: password updated successfully
# exit
</pre>
 
Now that the root password is set, run:
 
<pre>
# lxc-start -n funtoo -d
</pre>
 
The <tt>-d</tt> option will cause it to run in the background.
 
To attach to the console:
 
<pre>
# lxc-console -n funtoo
</pre>
 
You should now be able to log in and use the container. In addition, the container should now be accessible on the network.
 
To stop the container:
 
<pre>
# lxc-stop -n funtoo
</pre>
 
Ensure that networking is working from within the container while it is running, and you're good to go!
 
== LXC Bugs/Missing Features ==
 
This section is devoted to documenting issues with the current implementation of LXC and its associated tools. We will be gradually expanding this section with detailed descriptions of problems, their status, and proposed solutions.
 
=== reboot ===
 
By default, lxc does not support rebooting a container from within. It will simply stop and the host will not know to start it.
 
=== PID namespaces ===
 
Process ID namespaces are functional, but the container can still see the CPU utilization of the host via the system load (ie. in <tt>top</tt>).
 
=== /dev/pts newinstance ===
 
* Some changes may be required to the host to properly implement "newinstance" <tt>/dev/pts</tt>. See [https://bugzilla.redhat.com/show_bug.cgi?id=501718 This Red Hat bug].
 
=== lxc-create and lxc-destroy ===
 
* LXC's shell scripts are badly designed and are sure way to destruction, avoid using lxc-create and lxc-destroy.
 
=== network initialization and cleanup ===
 
* If used network.type = phys after lxc-stop the interface will be renamed to value from lxc.network.link. It supposed to be fixed in 0.7.4, happens still on 0.7.5 - http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg01760.html
 
* Re-starting a container can result in a failure as network resource are tied up from the already-defunct instance: [http://www.mail-archive.com/lxc-devel@lists.sourceforge.net/msg00824.html]
 
=== lxc-halt ===
 
* Missing tool to graceful shutdown container. 'lxc-halt' should be written and be posix sh-compatible, using lxc-execute to run halt in container.
 
=== funtoo ===
 
* Our udev should be updated to contain <tt>-lxc</tt> in scripts. (This has been done as of 02-Nov-2011, so should be resolved. But not fixed in our openvz templates, so need to regen them in a few days.)
* Our openrc should be patched to handle the case where it cannot mount tmpfs, and gracefully handle this situation somehow. (Work-around in our docs above, which is to mount tmpfs to <tt>/libexec/rc/init.d</tt> using the container-specific <tt>fstab</tt> file (on the host.)
* Emerging udev within a container can/will fail when realdev is run, if a device node cannot be created (such as /dev/console) if there are no mknod capabilities within the container. This should be fixed.
 
== References ==
 
* <tt>man 7 capabilities</tt>
* <tt>man 5 lxc.conf</tt>
 
== Links ==
 
* There are a number of additional lxc features that can be enabled via patches: [http://lxc.sourceforge.net/patches/linux/3.0.0/3.0.0-lxc1/]
* [https://wiki.ubuntu.com/UserNamespace Ubuntu User Namespaces page]
* lxc-gentoo setup script [https://github.com/globalcitizen/lxc-gentoo on GitHub]
 
* '''IBM developerWorks'''
** [http://www.ibm.com/developerworks/linux/library/l-lxc-containers/index.html LXC: Linux Container Tools]
** [http://www.ibm.com/developerworks/linux/library/l-lxc-security/ Secure Linux Containers Cookbook]
 
* '''Linux Weekly News'''
** [http://lwn.net/Articles/244531/ Smack for simplified access control]
 
[[Category:Labs]]
[[Category:HOWTO]]
[[Category:Virtualization]]

Revision as of 11:23, January 4, 2015

Introduction

Crossdev is a great feature of Funtoo that allows you to easily make a basic toolchain for your embedded devices and alternate platform computers. It can be used for distcc for example, so a slow x86 or ARM computer can be assisted on compiles by a much faster x86_64 computer(s). Some, but not all, packages can also be compiled stand-alone without distcc on the faster machine. You can also use the faster 64 bit machine to make kernels for these slower machines, a big advantage.

This is quite a big subject and the aim of this article is mainly to help new users with some Funtoo specific things relating to cross-compiling that are unique to Funtoo. So we won't waste time here on things that are covered elsewhere, only those specific things.

Let's get started!

The first step is to install crossdev. Crossdev is written by Mike Frysinger who is quite a busy fellow and does a superb job. But occasionally there are little hiccups with crossdev. If ever you are having a problem with crossdev it doesn't hurt to try downgrading to previous versions. I say this because on the date this article is written, 11-1-14, there are issues with the two latest versions of crossdev - 20140917 and 20141030 (see https://bugs.funtoo.org/browse/FL-1703). But version 20140729 works flawlessly. I have seen this happen from time to time, nothing to worry about.

root # emerge -av crossdev

Funtoo Specific Considerations

The most important thing to understand is that Funtoo gcc ebuilds are different from Gentoo ebuilds. This is because Daniel Robbins has made improvements to gcc to help with general usage. Unfortunately, a side effect has been that they no longer work with crossdev. But not to worry, there is an easy way around this issue and my understanding is that Funtoo intends to rectify it at a later date.

Another issue is that you cannot have a toolchain with exactly the same minor number as the current gcc on your system. You must use an ebuild of a slightly different vintage. I believe this issue occurs for Gentoo users also. For example, at the present time Funtoo current uses sys-devel/gcc-4.8.2-r3. So you could use gcc-4.8.1, or 4.8.0 but not any version of 4.8.2 (the revision -rx doesn't matter, can't do it). If you try to do that gcc will compile and then fail in the installation phase due to file collisions. In some cases with embedded devices you may wish to use much older versions, in which case this becomes a moot point.

Finally, a general issue with using crossdev with any distro is that you must sometimes juggle the various versions of binutils, gcc, glibc and linux-headers so that they are compatible with one another. This is the easiest problem to deal with, just use versions of the same time vintage, in extreme cases you may need to start changing revisions to get all of them to compile.

That's it, that's what you need to know.

GCC Ebuilds

The workaround to make gcc compile is to use Gentoo gcc ebuilds. They are readily available at the Gentoo attic: http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/?hid Going back historically for as far as you'd ever need. Go to the sys-devel/gcc folder and right click on a version and copy the link. I suggest a version that isn't even in the Funtoo tree, then there is no ambiguity about what is being used. Next copy it to your local repository, we'll assume /usr/local/portage here. So if you haven't already got a sys-devel/gcc directory there you could do

root # install -d /usr/local/portage/sys-devel/gcc

Now go to that directory in your terminal and wget <the link you copied>. At the end will be some fluff you don't want, delete all the way back to the word .ebuild and hit enter. In a second or so you'll have that ebuild in your repository. If you haven't done this before you'll need to link the files directory from the tree, so that it can find patches, etc. So do this

root # ln -s /usr/portage/sys-devel/gcc/files .

Finally you must create the manifest

root # ebuild gcc-<version>.ebuild manifest

And now you are ready!

Really, that's it?

Yes. Well, one other thing that you may need to do, if you've chosen a gcc version that is in the Funtoo tree, crossdev may get confused and try to install the tree version. At the beginning of the crossdev run it creates links in the crossdev repository, most people will just want this to be their local repository. So you can let crossdev run for a few seconds to create those links and then cancel it with ctl-c. Then go to the crossdev repository and delete the gcc link and replace it with a link to /usr/local/portage/sys-devel/gcc. Here's an example in my local repository:

root # ls -l /usr/local/portage/cross-avr
total 0
lrwxrwxrwx 1 root root 50 Nov  1 00:41 avr-libc ->/usr/portage/dev-embedded/avr-libc
lrwxrwxrwx 1 root root 47 Nov  1 00:41 binutils -> /usr/portage/sys-devel/binutils
root ##g##lrwxrwxrwx 1 root root 32 Nov  1 00:41 gcc -> /usr/local/portage/sys-devel/gcc##!g##
lrwxrwxrwx 1 root root 42 Nov  1 00:41 gdb -> /usr/portage/sys-devel/gdb

Note that the link for gcc is different from the others, which point to the normal Funtoo Portage tree, it points to my local repository, where the Gentoo ebuilds are stored. Most of the time also you would be linked to sys-libs/glibc, avr-libc is a bit unusual but normal for avr cross toolchains.

Once the link is fixed (if it even needed fixing), just hit your up arrow to get the same crossdev command line again and let it run through this time. You'll be rewarded with a shiny new cross toolchain at the end..

Of course, there is a bit more to making a cross toolchain, but as stated at the beginning of this article, only the unique differences for Funtoo are shown here. You should find nearly all the rest you need to know at https://www.gentoo.org/proj/en/base/embedded/handbook/index.xml?style=printable&part=1&chap=2. Only the top 1/4 page is needed.

One note, at the current time (11-1-14) you need to preface your crossdev command line statement with USE="-sanitize". This is a known bug (not Funtoo specific) that will soon be fixed in crossdev. You will find -sanitize in the package.use files, but you still must do it on the command line, else gcc will fail. For example the following command line was successful for me today

root # USE="-sanitize" crossdev --binutils 2.23-2 --g 4.8.1-r1 --l 1.8.0 -t avr -P -v