Crossdev Automation

From Funtoo
Jump to: navigation, search


This is the quick and dirty HOWTO covering the automation of cross-compiling, kernel building, emulation and booting of arbitrary architectures, but specifically focused on the cross compiling the Raspeberry Pi linux kernel sources for target arm-linux-gnueabihf and booting it on Raspberry Pi boards with a funtoo stage 3. It's so quick and dirty you probably won't be able replicate the results here on your first try. Leaving comments on how it failed for you on the discussion page is useful and easy.

The goal is to eventually abstract these steps into a script that takes a small tuple of parameters and yields a bootable/chrootable stage3 for the desired architecture + hardware.

There is no attempt to track the state of working directories at any time, so this not a logically consistent install document. However, the 'pipeline' nature of the process is simple, and if you follow the all the cd <directory> steps, you will be in the correct directory, particularly for kernel building. Absolute paths in environment variables often removes the need to track cwd as well.

Current working directory and environment variables must be set correctly for all commands to function. This document assumes a fresh working directory and no existing crossdev local repo, but it need only be created once. The working directories should ideally be disposable, with heavy remote repositories linked in. There are no tools (yet) to maintain any environmental integrity.

Everyone should probably read this:

The Cross Compiling Environment


This does not reflect the new Funtoo kits structure! . Please see Funtoo_Linux_Installation_on_RPI for updated hints.

Make a crossdev repository.

root # mkdir -p /usr/local/crossdev/{profiles,metadata}
root # echo "crossdev" > /usr/local/crossdev/profiles/repo_name
root # echo "
masters = gentoo
repo-name = crossdev
use-manifests = true
thin-manifests = true" > /usr/local/crossdev/metadata/layout.conf

Edit config files to match.

root # echo "
location = /usr/local/crossdev
auto-sync = yes" > /etc/portage/repos.conf/crossdev.conf

Install crossdev.

root # echo "sys-devel/crossdev **" >> /etc/portage/package.keywords
root # echo "=sys-devel/crossdev-99999999" >> /etc/portage/package.unmask
root # emerge crossdev

Using Gentoo Ebuilds

Read Cross-compiling_with_Crossdev but don't follow any of the steps yet. We're going to follow those instruction, but put the gentoo ebuilds into the crossdev repo. There are crossdev compatible Funtoo gcc ebuilds, but isolating Gentoo ebuilds in an overlay works.

Make a directory for gentoo specific ebuilds and link to the system's directory for gcc ebuild support files.

root # mkdir -p /usr/local/crossdev/sys-devel/gcc
root # mkdir -p /usr/local/crossdev/sys-devel/binutils
root # ln -is /usr/{portage,local/crossdev}/sys-devel/gcc/files
root # ln -is /usr/{portage,local/crossdev}/sys-devel/binutils/files
root # ls -l /usr/local/crossdev/sys-devel/gcc
lrwxrwxrwx 1 portage portage   32 Mar  1  2015 files -> /usr/portage/sys-devel/gcc/files
root # ls -l /usr/local/crossdev/sys-devel/binutils
lrwxrwxrwx 1 portage portage   32 Mar  1  2015 files -> /usr/portage/sys-devel/binutils/files

Let's use the gcc-4.9.3.ebuild and the binutils-2.25-r1. We must get gentoo specific patch files as well. We get these files from here. There are gentoo ebuilds available for gcc-5 as well.

root # cd /usr/local/crossdev/sys-devel
root # (cd gcc
root # (cd files
root # (cd binutils

Managing Environment Variables

Let's try to manage all the files needed with some simple environment variables we construct from user input. All exported variables and functions are part of an XC_ namespace for tab completion. (bash source code) - define variables
# Start User Input
# Where do we isolate all our work?
# This directory must exist and be created explicitly by you.
XC_WORK= <xc_work>

if ! [[ -d $XC_WORK ]]; 
    echo "The directory $XC_WORK does not exist."

#How do we distinguish this build?
XC_LINUX_PREFIX= <xc_linux_prefix>

# NOTE: if we need gcc-4.9 as well! How do we store that data?
# use arguments to crossdev script

# Which crossdev target and architecture to build the kenel?
# Note, this not necessarily what built the stage3

#which stage3 do we need and what CFLAGS does it use?
STAGE3_CFLAGS="-O2 -pipe -march=armv7-a -mfpu=neon-vfpv4 -mfloat-abi=hard"

# End User Input
# Some functions
XC_get_kernel_release() { (cd $XC_KERNEL_SRC ; ARCH=$ARCH CROSS_COMPILE=$CROSSDEV_TARGET- make kernelrelease;) }
XC_get_kernel_version() { (cd $XC_KERNEL_SRC ; ARCH=$ARCH CROSS_COMPILE=$CROSSDEV_TARGET- make kernelversion;) }
XC_set_kernel_extraversion() {(cd $XC_KERNEL_SRC; sed -i "s/EXTRAVERSION =.*/EXTRAVERSION = $@/" Makefile;)}

#set our kernel sources and config file
#Which kernel version are we building?
if [ -d $XC_KERNEL_SRC ]; then
    KERNEL_VERSION=`(cd $XC_KERNEL_SRC/ && ARCH=$ARCH CROSS_COMPILE=$CROSSDEV- make kernelversion 2>/dev/null )`
#funtoo stage3s. It is possible to mount your install media at $XC_STAGE3S.

XC_UBOOT=$UBOOTS/u-boot #unpacked sources
XC_UBOOT=$UBOOTS/u-boot #unpacked sources
#sdcard image creation
XC_IMAGE=$IMAGES/`echo $STAGE3  | tr / -`.img
echo "Your current cross compilation environment:"
echo "=============================="
for ((i=0; i< ${#XC_VARS[@]}; i++)); do
        echo ${XC_VARS[$i]}; echo ${XC_VALS[$i]};
            echo "==============================";
echo "=============================="
echo "Your current kernel configs for $XC_LINUX_PREFIX type builds:"
for file in $(dirname $XC_KERNEL_CONFIG)/*; do
    if [[ $file == *.kconf ]]; then
        echo -n "Use $(basename $file) as config (y/n)? "
        read answer
        if echo "$answer" | grep -iq "^y" ;then
            echo "Setting XC_KERNEL_OLDCONFIG to $file"
echo "=============================="
echo "=============================="
echo "If XC_KERNEL_OLDCONFIG is not set, \
you must create a new .config yourself or use a def_config target"

root # .

Your current cross compilation environment:
-O2 -pipe -march=armv7-a -mfpu=neon-vfpv4 -mfloat-abi=hard
user $XC_WORK
user $XC_WORK/linux/$XC_LINUX_PREFIX/linux
user $XC_WORK/configs/$XC_LINUX_PREFIX/unknown.kconf
user $XC_WORK/kernels/$XC_LINUX_PREFIX/kernel-unknown.img
user $XC_WORK/patches/$XC_LINUX_PREFIX
user $XC_WORK/firmware/$XC_LINUX_PREFIX
user $XC_WORK/stage3s/$XC_LINUX_PREFIX//funtoo-current/arm-32bit/raspi3/stage3-latest.tar.xz
user $XC_WORK/stage3s/$XC_LINUX_PREFIX//funtoo-current/arm-32bit/raspi3
user $XC_WORK/images/$XC_LINUX_PREFIX/funtoo-current-arm-32bit-raspi3.img
Your current kernel configs for $XC_LINUX_PREFIX type builds:
If XC_KERNEL_OLDCONFIG is not set, you must create a new .config yourself or use a def_config target

Retrieving Necessary Source Trees

Which sources your retrieve depends on your hardware. Most manufacturers make a git repo available. For this example, we use the Raspberry Pi sources.

Get the latest Raspberry Pi kernel source (, compiler (, and firmware ( using a .git extension to distinguish them from ordinary directories.


These are large, slow downloads, so don't do them more than necessary. Providing a symbolic link to their location in a more stable part of your filesystem is a better way to go. Changing the value of $XC_KERNEL_SRC via the <kernel_prefix> string and rerunning any git commands will cause a new cloning operation.

root # git clone $XC_KERNEL_SRC
root # git clone $XC_TOOLS/tools.git
root # git clone git:// $XC_FIRMWARE/firmware.git

Do You Really Need to Build the Kernel?

These repositories should contain existing kernel images and modules to get you going without actually building the kernel yourself. Why not try

user $ XC_KERNEL_IMG=$XC_FIRMWARE/boot/kernel7.img
user $ cp -r $XC_FIRMWARE/boot/modules/<some_kernel_version> $XC_STAGE3/lib/modules

Find similar analogues to hand dts, firmware, config files, etc, using existing files in repositories instead of building them yourself as described below.

Kernel Building

The steps here are a summary of this document and this, which includes the use of device tree blobs.

After downloading or syncing the the kernel sources, run the script again to set $KERNEL_VERSION properly.

root # echo $KERNEL_VERSION
root # . 
....<script output>...
root # echo $KERNEL_VERSION

Construct the Crossdev Target

Make the most up to date version of the crossdev target and set the appropriate default compilers with gcc-config. You might have many options here depending on your system.

root # crossdev -t $CROSSDEV_TARGET -P -v

root # gcc-config -l
 [1] armv6j-hardfloat-linux-gnueabi-4.8.4 *

 [2] armv7a-hardfloat-linux-gnueabi-4.8.4
 [3] armv7a-hardfloat-linux-gnueabi-4.9.3
 [4] armv7a-hardfloat-linux-gnueabi-5.3.0 *

 [5] x86_64-pc-linux-gnu-4.8.4
 [6] x86_64-pc-linux-gnu-4.9.3 *

Notice that I built armv7a-hardfloat-linux-gnueabi-5.3.0 with x86_64-pc-linux-gnu-4.9.3.

Clean the source tree and copy over any existing kernel config or create a freshie.


'make mrproper' is a bummer if you don't want to do a complete kernel rebuild or are not starting from scratch. Try to preserve the state of your $XC_KERNEL_SRC directory.

root # cd $XC_KERNEL_SRC
root # make mrproper

If you're starting fresh ($XC_KERNEL_OLDCONFIG is NONE), make a defconfig target and work with that. If you are building onto your great chain of kernel configs, use $XC_KERNEL_OLDCONFIG.

Using defconfigs

There are many useful make targets to control kernel configs. In particular, the architecture specific ones are most useful in cross-compiling applications.


root # make_defconfig_help() { make help | sed -n -e '/Architecture specific targets.*/,$p'; }
root # ARCH=$ARCH CROSS_COMPILE=$CROSSDEV_TARGET- make_defconfig_help
Architecture specific targets (arm):
... <targets we don't need> ...

  acs5k_defconfig          - Build for acs5k
  acs5k_tiny_defconfig     - Build for acs5k_tiny
  am200epdkit_defconfig    - Build for am200epdkit
  assabet_defconfig        - Build for assabet
  at91_dt_defconfig        - Build for at91_dt
  axm55xx_defconfig        - Build for axm55xx
  badge4_defconfig         - Build for badge4
root ##r##  bcm2709_defconfig        - Build for bcm2709
  bcm2835_defconfig        - Build for bcm2835
  bcm_defconfig            - Build for bcm
  bcmrpi_defconfig         - Build for bcmrpi
  cerfcube_defconfig       - Build for cerfcube
  clps711x_defconfig       - Build for clps711x
  ...<lots o configs>...
  zeus_defconfig           - Build for zeus
  zx_defconfig             - Build for zx

...<more targets we don't need> ...
root # export XC_DEF_CONFIG=bcm2709_defconfig

Reusing Previous Kernel Configurations

Use $XC_KERNEL_OLDCONFIG and update it.


Tweak the Kernel

Modify the kernel configuration using the ncurses interface and save it.


def_config is good place to start, but they appear to be very bloated 'all bases covered' configurations. Compiling time for a kernel can radically reduced by pruning unnecessary modules. At some point, a repository of minimal configs may be available, but in the meantime you are on your own.

See this question.



Save Those Configs

Whether you used a defconfig or and oldconfig, let's store and reuse them.

root # cp .config $XC_KERNEL_CONFIG

Setting $CROSSDEV_TARGET in the Kernel

There is a kernel option to allow us to drop the CROSS_COMPILE environment variable. The disadvantage to this is that you have to track its state and synchronize it with your build environment. It can be done, though.

Make the Kernel

Build the new kernel, using 'make -j <N>' to specify compiler job count (a matter of taste, time and thermal paste). Don't install anything yet! Why don't you skip ahead and get the $XC_STAGE3 set up while the kernel is baking?


Make the kernel your own by appending a distinguishing extraversion.

root # get_kernel_version
root # set_kernel_extraversion -muh_kernel
root # get_kernel_version

Stage3 Configuration


Let's get a stage3 and save it in a directory structure that matches the funtoo build tree and unpack it.

root # wget $XC_STAGE3_URL -O $XC_STAGE3
root # cd $XC_STAGE3S
root # tar xpf $XC_STAGE3
root # rm $XC_STAGE3
You MUST have XC_* environment variables set. If they are not set, the following commands will modify your system files. To protect yourself from undefined environment variables, use the bash option
root # set -o nounset

We add compiler optimizations to make.conf and modify the default fstab to match the filesystem that will be created on the bootable SD card. We will have a boot partition and a root partition. Could it be simpler?

root # echo "
MAKEOPTS=-j4" >> $XC_STAGE3S/etc/portage/make.conf
root # sed -i "s/\/dev\/sda1.*/\/dev\/mmcblk0p1 \/boot vfat defaults 0 2/" $XC_STAGE3S/etc/fstab 
root # sed -i "s/\/dev\/sda2.*//" $XC_STAGE3S/etc/fstab 
root # sed -i "s/\/dev\/sda3.*/\/dev\/mmcblk0p2 \/ ext4  defaults 0 1/" $XC_STAGE3S/etc/fstab 
root # sed -i "s/\#\/dev\/cdrom.*//" $XC_STAGE3S/etc/fstab

Here we interactively add a root password to log in with to $XC_STAGE3S/etc/shadow.


Occasionally this fails due to an unescaped sed control character in the password hash. Just try it until it works, or run the password generation separately from the shadow file editing, or actually fix it. But I love a good one liner.

root # sed -i "s/root\:\*/root\:`(openssl passwd -1)`/" $XC_STAGE3S/etc/shadow

If you want to ssh to the board as root, you need to allow it.

root # echo "PermitRootLogin yes" >> $XC_STAGE3S/etc/ssh/sshd_config

Let's prevent serial consoles from spawning. If you have a need for this, you'll know what to do.

root # sed -i "s/s0\:.*/\#&/" $XC_STAGE3S/etc/inittab

Let's set up the software clock. This is Raspberry Pi specific.

root # ln -sf /etc/init.d/swclock $XC_STAGE3S/etc/runlevels/boot
root # rm $XC_STAGE3S/etc/runlevels/boot/hwclock
root # mkdir -p $XC_STAGE3S/lib/rc/cache
root # touch $XC_STAGE3S/lib/rc/cache/shutdowntime

Set up sshd as a default runlevel service.

root # ln -sf /etc/init.d/sshd $XC_STAGE3S/etc/runlevels/default
root # ln -sf /etc/init.d/dhcpcd $XC_STAGE3S/etc/runlevels/default

QEMU Emulation for Chrooting and Configuration

We would ideally now like to chroot into our stage3, but we can't because the binaries are in another format and can't be read by our cpu. So we need to emulate (slowly) an ARM cpu to execute them. Then we can chroot and do some good setup before we move the stage3 to the board.

Reconfigure and Emerge QEMU

If necessary, adjust or create a qemu configuration and (re-)emerge it.


We are altering the running system files now, to create the proper USE flags on your qemu install.

root # echo "
>=media-libs/mesa-11.0.7 gles2
app-emulation/qemu static-user" >> /etc/portage/package.use/qemu
root # echo "
QEMU_USER_TARGETS='${QEMU_USER_TARGETS} arm'" >> /etc/portage/make.conf
root # emerge app-emulation/qemu

Create the Static QEMU Binary

Copy the static qemu binary to the $XC_STAGE3S filesystem and start the binary translation services.

root # cp /usr/bin/qemu-arm $XC_STAGE3S/usr/bin/
root # /etc/init.d/binfmt start
root # /etc/init.d/qemu-binfmt start

Enter the Chroot

Now we copy over network info and use the magic formula to start the emulated chroot.

root # cp -L /etc/resolv.conf $XC_STAGE3S/etc/resolv.conf
root # cd $XC_STAGE3S
root # mount --rbind /dev dev
root # mount --rbind /sys sys
root # mount -t proc none proc
root # env -i HOME=/root TERM=$TERM /bin/chroot . bash -l
root # export PS1="(chroot) $PS1"

Using DistCC for Distributed Cross-Compiling (HIghly Optional)

A board like a Pi can't really be used to leverage the advantages of a source based distribution like Funtoo unless the compilation can be outsourced. This could be done using a binary package tree, but a first step towards this will involve using DistCC to make the power of the Cross-Compiling host that was used to build its kernel, available to the board running it. This gives the board the ability to emerge packages in a reasonable amount of time, potentially caching them for other identical boards.

Configuring the Compilers

See this document for a full explanation. We're going to do it a bit differently (and more slowly) by doing it in the chroot, so we when we write the stage3 image to board, distcc is ready to go.

If you're tricky, you make make your chrooted stage3 cross compile for its native architecture using the QEMU host!

We probably need a Funtoo friendly mechanism to preserve these alterations across emerge @world. I don't know what that is.

(chroot) # emerge distcc
(chroot) # .
(chroot) # cd /usr/lib/distcc/bin
(chroot) # rm c++ g++ gcc cc
(chroot) # echo \
exec /usr/lib/distcc/bin/armv7a-hardfloat-linux-gnueabi-g${0:$[-2]} "$@"' > $XC_STAGE3_CHOST-wrapper
(chroot) # chmod a+x $XC_STAGE3_CHOST-wrapper
(chroot) # ln -s $XC_STAGE3_CHOST-wrapper cc
(chroot) # ln -s $XC_STAGE3_CHOST-wrapper gcc
(chroot) # ln -s $XC_STAGE3_CHOST-wrapper g++
(chroot) # ln -s $XC_STAGE3_CHOST-wrapper c++

We hook it into Portage and set the value of <N> to twice the number of total (local + remote) CPU cores + 1 and the value of <M> to the number of local CPU cores.

(chroot) # echo \
'MAKEOPTS="-j<N> -l<M>"
FEATURES="distcc distcc-pump"' >> /etc/portage/make.conf

Configure your helper hosts. Leaving out localhost is generally a good idea.

(chroot) # distcc-config --set-hosts "<your_host>,cpp,lzo <your_other_host>,cpp,lzo"

Set the Hostname

root #(chroot) set_hostname() { sed -i "s/hostname=\".*\"/hostname=\"$@\"/" /etc/conf.d/hostname; }
root #(chroot) set_hostname host0

Get the Portage Tree

Get the package repository so you can emerge programs.

root #(chroot) emerge --sync
/usr/bin/git clone --depth 1 git:// .
Cloning into '.'...
...<very cool>....

Exit the Chroot Before You Proceed

When you're done, make sure you exit the chroot cleanly.

root #(chroot) exit
root # cd $XC_WORK
root # umount -lR $XC_STAGE3S/*

Installing to the Board

Let's stage everything under $XC_STAGE3S before we move it to an SD card. Then we can easily make multiple SD card images at once if necessary. It is possible to work directly on your install media by mounting the appropriate paritions under $XC_STAGE3S.

We might also need a boot loader. That is a subject for another time, but look here and here and here, and combine them to produce bootable images for a more esoteric board like the AM335x. Eventually, automating u-boot will be required.

See if you are dying to know more about using U-Boot.

The following instructions are Raspberry Pi specific are much simpler than u-boot.

Create a Compressed Kernel Image

Make the compressed kernel image, saved as kernel7.img. This is Raspberry Pi specific, with kernel.img used for rpi1 and kenel7.img used for rpi2 and rpi3.

root # cd $XC_KERNEL_SRC
root # scripts/mkknlimg arch/arm/boot/zImage $XC_KERNEL_IMG
root # cp $XC_KERNEL_IMG $XC_STAGE3S/boot/kernel7.img

Deal With the DTBs

In the ARM world, you need to deal with heterogeneous pairings of processor and base board. To cope with this, Device Tree Blobs were invented. See this tutorial.

root # cd $XC_KERNEL_SRC 
root # mkdir -p $XC_STAGE3S/boot/overlays
root # cp arch/arm/boot/dts/*.dtb $XC_STAGE3S/boot
root # cp arch/arm/boot/dts/overlays/*.dtb* $XC_STAGE3S/boot/overlays
root # cp arch/arm/boot/dts/overlays/README $XC_STAGE3S/boot/overlays

Copy Over Modules and Firmware


Up to date firmware is critical to get your board booted. If you find yourself stuck at the rainbow screen, update $XC_FIRMWARE.

root # cp $XC_FIRMWARE/firmware.git/boot/{bootcode.bin,fixup*.dat,start*.elf} $XC_STAGE3S/boot
root # cp -r $XC_FIRMWARE/firmware.git/hardfp/opt $XC_STATE3S/opt

You must add /opt/vc/lib to LD_LIBRARY_PATH in the shell, or export LDPATH=/opt/vc/lib via a file in /etc/env.d if you want to link against these firmware libraries. This is critical if you want to do any accelerated graphics or gpu level work.

root # cat /etc/env.d/99vc
root # env-update
root # . /etc/profile

If you want to try building native vc4 acceleration with media-libs/mesa you must unmask the appropriate USE flags at the profile level and VIDEO_CARDS="vc4" in make.conf.

root # cat /etc/portage/profile/portage.use.mask
media-libs/mesa -video_cards_vc4 
x11-libs/libdrm -video_cards_vc4

Remove Unnecessary Kernel Source Links

Remove links to the kernel build and the sources if you do not want to do any kernel building from the pi itself. Rebuilding the kernel from the pi is feasible if you have a good distcc setup. We don't yet.

root # rm $XC_STAGE3S/lib/modules/`get_kernel_release`/{build,source}

Configure the Pi

Create the cmdline.txt file.

root # echo "dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait" > $XC_STAGE3S/boot/cmdline.txt


Overclocking is possible.

root # echo '
root #“None” “700MHz ARM, 250MHz core, 400MHz SDRAM, 0 overvolt”
root #“Modest” “800MHz ARM, 300MHz core, 400MHz SDRAM, 0 overvolt”
root #"Medium” “900MHz ARM, 333MHz core, 450MHz SDRAM, 2 overvolt”
root #“High” “950MHz ARM, 450MHz core, 450MHz SDRAM, 6 overvolt”
root #“Turbo” “1000MHz ARM, 500MHz core, 500MHz SDRAM, 6 overvolt”

over_voltage=6' >> $XC_STAGE3S/boot/config.txt

Very precise control of video output (and many other hardware parameters) is possible during boot.

Create the SD card Partitions and Copy Everything Over

Let's take the easy way out and make a 128M vfat boot and the rest of the disk an ext4 on /dev/<your_dev>.

root # echo "
unit: sectors
/dev/<your_dev>1  :  start=2048,      size=262144,   type=c
/dev/<your_dev>2  :  start=264192,                   type=83" | sfdisk /dev/<your_dev>
root # mkfs.vfat -F 32 /dev/<your_dev>1
root # mkfs.ext4 /dev/<your_dev>2

When we copy everything over to the media, we ignore the /usr/portage directory. We can emerge --sync again when booted to get it back. Use rsync to win by tweaking files in $XC_STAGE3S to create a stable install and re-running the rsync command to copy over only what's been changed.

root # mkdir -p /mnt/funtoo
root # mount /dev/<your_dev>2 /mnt/funtoo
root # mkdir -p /mnt/funtoo/boot
root # mount /dev/<your_dev>1 /mnt/funtoo/boot
root # rsync -avz --exclude "usr/portage/*" $XC_STAGE3S/{boot,bin,etc,home,lib,mnt,opt,root,run,sbin,srv,tmp,usr,var,dev} /mnt/funtoo
root # mkdir /mnt/funtoo/{proc,sys}
root # umount -R /mnt/funtoo

You're finished! Maybe, but probably not. Or maybe! I've had great success with this method, and I hope you do too.


14:57 <@drobbins> my view is anyone who hangs out in #funtoo and uses Funtoo has a great opportunity to be one of the tech elite, regardless of country

Sun Jan 29 2017