ct-backends

Build backend architecture and selection guide

Author:

drgeoffathome@gmail.com

Date:

2026-04-02

Version:

13.1.4

Manual section:

7

Manual group:

developers

DESCRIPTION

compiletools supports multiple build backends that generate native build files for different build systems. All backends share the same dependency analysis, magic flag extraction, and header dependency tracking — only the build file generation and execution differ.

Select a backend with:

ct-cake --backend=<name>

Use ct-list-backends to discover which backends are installed and available.

REMOVED BACKENDS

slurm — the Slurm build backend (--backend=slurm), which distributed compilation across an HPC cluster by submitting sbatch jobs, was removed in 12.0.0. The last release to include the Slurm backend was 11.0.0; pin compiletools==11.0.0 if you still require it.

Running compiletools inside a Slurm allocation is unaffected: every ct-* tool still honours Slurm CPU affinity and cgroup limits when choosing default parallelism. Only the backend that submitted Slurm jobs was removed.

BACKEND SUMMARY

Name	Build file	Tool
make	Makefile	make
ninja	build.ninja	ninja
cmake	CMakeLists.txt	cmake
bazel	BUILD.bazel	bazel
shake	.ct-traces.json	(builtin)

FEATURE MATRIX

Y = supported, N = not supported, D = supported and on by default.

Name	File-lock	–build-only-changed	–realclean	PCH cache	Libraries	Test runner	CA outputs	Early cutoff
make	D	Y	Y	Y	Y	Y	N	N
ninja	Y	Y	Y	Y	Y	Y	N	Y (restat)
cmake	N	Y	Y	Y	Y	Y	N	N
bazel	N	Y	Y	Y	Y	Y	N	N
shake	Y	Y	Y	Y	Y	Y	Y	Y

Notes:

• File-lock — --file-locking enables multi-user object CAS / PCH CAS caches. CMake/Bazel manage their own coordination and skip this layer (see FILE LOCKING below).

• –build-only-changed — restrict the graph to a whitespace-separated set of source files supplied on the command line. Implemented in the shared base class, so all backends honor it.

• –realclean — selectively remove only this build’s products from an object CAS (and .gch files from a PCH CAS), instead of rm -rf of the whole tree. Inherited from the base backend; the Make backend additionally generates a realclean rule in the Makefile.

• PCH cache — content-addressable precompiled header cache via --cas-pchdir; the compile rules carry the per-cmd_hash .gch paths, so all backends produce and consume the cache transparently.

• Libraries — static and dynamic library targets via --static / --dynamic. All backends support both.

• Test runner — test execution is part of the build graph: each test runs as soon as its executable is linked, in parallel with ongoing compiles, links, and other tests. --serialise-tests forces one-at-a-time execution (graph chaining for make/ninja/shake, --local_test_jobs=1 for bazel, custom-command DEPENDS for cmake). See ct-cake(1) Test execution scheduling.

• CA outputs / Early cutoff — content-addressable filenames and build skipping when an output’s bytes are unchanged. Native to the Shake builtin engine; Ninja approximates early cutoff via restat = 1.

BACKENDS

make

Generates a non-recursive Makefile following Peter Miller’s Recursive Make Considered Harmful design.

When to use: Widely available, well-understood output, good diagnostic support via --trace (GNU Make 4.0+).

Features:

• File locking enabled by default for multi-user shared caches

• .DELETE_ON_ERROR to clean partial outputs on failure

• Implicit rules disabled (-rR) for predictable behaviour

• Output synchronisation per target (--output-sync=target, Make 4.0+)

• --shuffle support (Make 4.4+) to detect missing dependencies in CI

• Selective build via --build-only-changed

• Wraps compile and link commands with ct-lock-helper when file locking is enabled

Requires: GNU Make (make in PATH).

ninja

Generates a build.ninja file targeting the Ninja build system.

When to use: Large incremental rebuilds where Ninja’s minimal overhead outperforms Make. Ninja re-evaluates the build graph faster than Make on projects with many targets.

Features:

• Depfile tracking via -MMD -MF and Ninja’s deps = gcc integration

• restat = 1 on rules to support early cutoff when outputs are unchanged

• File locking via ct-lock-helper wrappers (optional)

• Selective build via --build-only-changed

Requires: Ninja (ninja in PATH).

cmake

Generates a CMakeLists.txt and builds via CMake in an out-of-source build directory ({objdir}/cmake-build).

When to use: Integration with CMake-based toolchains or IDEs that expect CMake projects. Useful when downstream consumers need a CMakeLists.txt.

Features:

• Aggregates low-level compile/link rules into high-level CMake targets (add_executable, add_library)

• Separates flags into target_include_directories, target_compile_options, target_link_directories, target_link_libraries, and target_link_options

• Automatically detects C, C++, or both languages

• Passes user-configured CC and CXX to CMake

• Enforces out-of-source builds

Requires: CMake 3.15+ (cmake in PATH).

Limitations: File locking is disabled — CMake manages its own coordination.

bazel

Generates BUILD.bazel, WORKSPACE, and MODULE.bazel files targeting Bazel.

When to use: Projects that need Bazel’s hermeticity, remote caching, or remote execution capabilities.

Features:

• Aggregates rules into cc_binary and cc_library targets using @rules_cc

• Strips -I paths (Bazel manages includes internally)

• Resolves relative -L paths to absolute for Bazel’s sandbox

• Sources outside the workspace are copied to ext/

• Outputs in bazel-bin/ are copied to the final destination

Requires: Bazel or Bazelisk (bazel or bazelisk in PATH).

Limitations: File locking is disabled — Bazel manages its own sandboxing and caching. //#GIT= externals cloned to the sibling default (../<name>, outside the gitroot) are invisible to Bazel’s hermetic sandbox and the build fails to resolve their headers; pass --externals-dir pointing inside the workspace instead (see ct-fetch(1), “Externals directory”). -isystem paths outside the workspace are rejected for the same reason.

shake (builtin, default)

A self-executing backend implementing the Build Systems a la Carte (Mokhov, Mitchell, Jones 2018) rebuild strategy with verifying traces and early cutoff. No external build tool is required.

When to use: Best rebuild precision. Ideal for large projects where avoiding unnecessary rebuilds matters most, and for shared caches on network filesystems where content-addressable outputs prevent stale reuse.

Features:

• Content-addressable outputs: compile rules produce files whose names encode source hash, dependency hash, and macro state — existence implies correctness, skipping expensive content checks

• Verifying traces: records content hashes of all inputs, outputs, and commands in .ct-traces.json; on rebuild, verifies all input hashes before re-executing

• Early cutoff: if a rebuilt output is byte-identical to the previous version, dependents are not rebuilt

• Async execution with asyncio.Semaphore limiting concurrency

• File locking via FileLock for multi-user shared caches

• Atomic output creation via temp file + rename

Requires: Nothing — builtin to compiletools.

FILE LOCKING

File locking enables multiple users and build hosts to share compiled object files safely. It is supported by the Make, Ninja, and Shake backends.

Enable with --file-locking (on by default for Make). The locking strategy is auto-detected based on the target filesystem:

• lockdir — NFS, Lustre (mkdir-based, works everywhere)

• fcntl — GPFS (kernel-managed cross-node locks)

• cifs — CIFS/SMB (exclusive file creation)

• flock — local filesystems like ext4, xfs, btrfs

See ct-lock-helper (1) and ct-cleanup-locks (1) for details.

CONTENT-ADDRESSABLE OUTPUTS

The Shake backend uses content-addressable object file naming:

basename_filehash_dephash_macrohash.o

Because the filename encodes all inputs (source content, dependency hashes, and macro state), if the file exists it is guaranteed correct. This eliminates the need for content verification on cache hits and prevents stale reuse on low-resolution filesystems.

All backends additionally cache the output of the link step in cas-exedir (executables, .a static libraries, .so shared libraries). The link rule writes directly to <cas-exedir>/<linkkey[:2]>/<basename>_<linkkey>.<ext>; a downstream symlink rule then publishes the user-facing bin/<variant>/<name> (or bin/<variant>/lib<name>.{a,so}) as a hard link via ct-cas-publish, with a symlink fallback only on EXDEV. The linker-artefact key folds in linker identity, canonicalized LDFLAGS, sorted gitroot-canonical object paths, the ar binary identity, SOURCE_DATE_EPOCH / LD_LIBRARY_PATH / LIBRARY_PATH / LD_PRELOAD, and (for executables) the canonical bindir path. See the Linker-artefact caching (cas-exedir) section in ct-cake(1).

MTIME VS CAS REBUILD MODE

The --use-mtime boolean controls whether the generated build files preserve classical Make/Ninja mtime-based prerequisite semantics on top of the CAS layer. Default is --no-use-mtime (equivalently --use-mtime=False): compile, link, ar, and link-shared rules drop their sources/objects from prerequisites; only PCH/BMI artefacts remain as order-only deps so build ordering is preserved. Existence of the CAS artefact on disk is the sole rebuild signal. This mode is the recommended setting for shared CI caches because a fresh git checkout (every source has mtime=now) hits the cache instead of re-running the producer.

Pass --use-mtime=True to restore legacy mtime semantics for interactive workflows where re-touching a source should force a rebuild even when the producer key would not change. The flag is registered in apptools.add_output_directory_arguments so every backend’s parser accepts it, but only make and ninja honor it: their rule emitters consume the prereq list as a literal mtime comparison. cmake, bazel, and shake use their own change detection and cannot deliver “touch to force rebuild” semantics, so --use-mtime=True against one of those (shake is the default backend) is a hard ValueError at backend construction rather than a silently ignored no-op — pair it with an explicit --backend=make or --backend=ninja.

Caveats of CAS-only mode:

• Generated headers must exist before ct-cake runs its headerdep pass; absent headers are dropped from dep_hash and only force a recompile on the next invocation.

• make -t and ninja -t restat are inappropriate — they create empty files at target paths whose recipes have no real prerequisites in CAS-only mode, corrupting the cache. Use --use-mtime=True if you need to mark a build up-to-date without running it.

• System / /usr/include headers are not in the cache key (ccache and sccache likewise exclude default system headers). An in-place compiler swap usually changes compiler_identity and invalidates the cache implicitly.

• The CPATH / C_INCLUDE_PATH / CPLUS_INCLUDE_PATH / OBJC_INCLUDE_PATH values ARE in the compile cache key (ccache hashes the same four, though verbatim — compiletools first canonicalizes each path entry so a gitroot-prefixed entry hashes identically across checkouts), and headers found via those directories are walked and content-hashed into dep_hash like any -I header. One residual gap: the header walk keeps a single flat search list for C and C++ alike, so a basename present in both a C_INCLUDE_PATH dir and a CPLUS_INCLUDE_PATH dir with differing content may hash the copy the compiler does not use.

CHOOSING A BACKEND

Scenario	Recommended backend
General-purpose builds	shake (default)
Large incremental rebuilds	ninja
Shared multi-user caches	shake or make --file-locking
CMake IDE integration	cmake
Bazel ecosystem integration	bazel
Maximum rebuild precision	shake

EXAMPLES

Build with Ninja backend:

ct-cake --backend=ninja

Build with Shake backend and file locking:

ct-cake --backend=shake --file-locking

List available backends:

ct-list-backends
ct-list-backends --all --style=pretty

REFERENCES

• Andrey Mokhov, Neil Mitchell, Simon Peyton Jones. Build Systems a la Carte. Proc. ACM Program. Lang., Vol. 2, ICFP, Article 79, September 2018. https://doi.org/10.1145/3236774

• Peter Miller. Recursive Make Considered Harmful. 2008. https://api.semanticscholar.org/CorpusID:54117644

SEE ALSO

ct-cake (1), ct-list-backends (1), ct-lock-helper (1), ct-cleanup-locks (1)