Icecream Distributed Compiler

Posted on June 10, 2019

icecream is a distributed compiler working as frontend for gcc and clang. It allows heterogenous environments, including cross compile toolchains.

Since I had to apply some tweaks to run it smoothly, I’ve created this doc to document my current setup.

Overall Architecture

icecream consists of the following components:

icecream-scheduler

My central icecc scheduler runs on my old macbook pro. While the scheduler can run as a service, I’ve typically started it from command-line:

/usr/local/Cellar/icecream/1.2/sbin/icecc-scheduler -vvv

iceccd Daemon

I’m currently running two different setups: MacOS and linux. on both systems I ended up using rather different configurations. The daemon has to run on each system, that will participate in the compile grid

iceccd on MacOS

Due to a bug in the homebrew formula, iceccd didn’t work as a service. Therefore I ended up using a simple bash launcher script:

#!/bin/bash

scheduler=moka-mac-old.local
user=tim
name=macbook-old
icecc_path=/Users/tim/icecc
max_processes=5

sudo /usr/local/Cellar/icecream/1.2/sbin/iceccd -s ${scheduler} -u ${user}
    -vv -m ${max_processes} -N ${name} -n ${name}  --cache-limit 512
    -b ${icecc_path}

Note: the daemon needs to be executed as root to obtain chroot permissions, but will drop permissions to ${user}.

iceccd on Linux

On linux (ubuntu-based neon) the default configuration worked out of the box for me. I’m currently running 3 instances: one natively on my dell xps 13 subnotebook and two in vmware fusion virtual machines.

On the virtual machines I ended up changing the icecc user in /etc/init.d/iceccd to tim and in /etc/icecc/iceccd.conf I’ve used ICECC_BASEDIR=/home/tim/icecc and ICECC_NICE_LEVEL=1.

Caveats: the VM daemons tend to stop accepting compile jobs after the host was hibernating. In this case one has to restart the daemon to participate in the grid again.

Running icecc

ccache Integration

icecc is a compiler launcher similar to ccache: it can basically be used as a prefix to the compiler (icecc g++ foo.cpp). However it can also be used in combination with the ccache compile cache. To achieve this one has to modify the ~/.ccache/ccache.conf file and add the following entry:

prefix_command = icecc

When I’m not connected to my home network, I typically comment out this line to disable icecream.

icecream in Heterogenous Environments

icecream is heavily inspired by distcc. One disadvantage of distcc is that it requires a homogenous environment: the compiler binaries (versions, architectures etc) need to be the same on all platforms. icecream supports heterogenous environments by ensuring that the same compiler is used on all agents. if a compiler does not exist, it will try to create a self-contained chroot environment and upload it to the agent running the build.

At least in theory. In practice I had to do the following tweaks.

Running icecc on Linux with QtCreator

On linux I had different problems to successfully compile remotely. The automatic uploading of the toolchain didn’t work for some reason, so I had to create a toolchain manually:

# create 5abd5d4d226a3f5b0a1d1479002198b9.tar.gz
/usr/bin/icecc-create-env /usr/bin/g++-8

# rename the toolchain to something less arcane:
mv 5abd5d4d226a3f5b0a1d1479002198b9.tar.gz /home/tim/g++-8.tar.gz

With this toolchain in the environment the remote compilations started to work. I ended up with a tiny launcher script for qtcreator:

#!/bin/sh

export ICECC_VERSION=/home/tim/g++8.tar.gz
export ICECC_CARET_WORKAROUND=0

/home/tim/qtcreator-4.9.1/bin/qtcreator

Note: the ICECC_CARET_WORKAROUND environment variable is required to disable local re-compilation in case of compiler warnings.

Running icecc on Linux with Yocto Sdks

When cross-compiling for a yocto sdk the automatic upload turned out to work much better. icecc basically worked out of the box.

# source the environment
source /home/tim/path/to-yocto-sdk/environment-setup-poky-linux

cd path/to/build
ninja -j20 # nice way to compile on a 2-core subnotebook

Running icecc on MacOS (with QtCreator)

When running icecc on MacOS (with qtcreator) I had to explicitly pass the compiler binaries to cmake. My “kit” contains the following cmake variables

CMAKE_CXX_COMPILER:STRING=/usr/bin/clang++
CMAKE_C_COMPILER:STRING=/usr/bin/clang
CMAKE_MAKE_PROGRAM:INTERNAL=/usr/local/bin/ninja

Without these entries I haven’t been able to successfully compile remotely.

Note: in order to compile Qt applications the environment variable ICECC_CLANG_REMOTE_CPP=0 has to be added (https://github.com/icecc/icecream/issues/471).

Cross-compiling to MacOS from Linux

It is possile to set up a cross-compiler to build for MacOS on Linux agents. Tor Arne Vestbø from QtCompany provides a small docker environment to build the cross toolchain: https://hub.docker.com/r/torarnv/icecc-create-x86_64-env

The ICECC_VERSION envirionment variable can be used to select multiple toolchains:

ICECC_VERSION=/path/to/swift-llvm-5.0.1-darwin18.tar.gz,x86_64:/path/to/swift-llvm-5.0.1-x86_64.tar.gz

swift-llvm-5.0.1-darwin18.tar.gz is the toolchain generated locally via icecc-create-env, while swift-llvm-5.0.1-x86_64.tar.gz is the cross toolchain generated by Tor Arne’s docker environment.

icecream monitor: A Qt-based Monitor GUI

icecream monitor is a qt-based monitor gui that can visualise the net. It works fine on linux, but is known to have issues on MacOS (https://github.com/icecc/icemon/pull/47).

Caveats/Known Issues