“It works on my machine” often means “my toolchain accidentally matches my machine.” Once you touch embedded targets, Docker images, CI runners, or cross-compilation, the build becomes a system: compiler + assembler + linker + libraries + headers + paths + flags. This post explains how GCC and Clang toolchains are put together, how compilation really flows, and how to cross-compile without the usual “undefined reference” and “wrong ELF class” nightmares.
Quickstart
If you’re debugging a weird build right now, start here. These steps give you maximum signal in minimum time, whether you’re compiling for your laptop, a Raspberry Pi, or an MCU.
1) Identify the triple and the linker you’re actually using
Most “mystery issues” are a mismatch between the intended target and what the compiler is driving under the hood.
- Print compiler version and target triple
- Print the full link line (paths + libraries)
- Confirm which linker is used (bfd/gold/lld)
2) Separate compile problems from link problems
If it fails at compile-time you’re dealing with headers/macros/types. If it fails at link-time, it’s usually libs/ABI/order.
- Compile to
.ofirst (no link) to isolate errors - Then link explicitly and inspect missing symbols
- Don’t “spray flags” until you know which stage fails
3) For cross-compiling: lock your sysroot
Cross builds break when host headers/libs leak into the target build. A sysroot is your “bubble” that keeps everything target-correct.
- Use a target sysroot (headers + libs from target)
- Point the compiler at it (and keep it consistent in CI)
- Use target-aware
pkg-configor disable it
4) Make builds reproducible: print, don’t guess
When you can see the exact commands, you can fix the exact problem. Treat the build as a traceable artifact.
- Enable verbose build output (Make/CMake/Ninja)
- Capture the full compile and link commands in CI logs
- Freeze toolchain versions per project (container or pinned packages)
If an error mentions “file not found” or types/macros → it’s headers/preprocessing/compile. If it mentions “undefined reference”, “relocation”, or “wrong ELF class” → it’s link/ABI/architecture.
Overview
A toolchain is the set of tools and target-specific files needed to turn source code into a binary you can run (or flash). People often say “I used GCC,” but what they really used is a pipeline: a compiler driver that coordinates preprocessing, compilation, assembling, linking, and pulling in the right runtime libraries.
What you’ll learn in this post
- What “toolchain” means (and what is actually inside one)
- How GCC and Clang differ in practice (drivers, defaults, diagnostics, linkers)
- How compilation stages work (and which flags apply where)
- How cross-compilation works: build/host/target + sysroot + target triples
- How to debug common failures: headers, library order, ABI mismatches, wrong architecture
GCC vs Clang: practical differences
| Topic | GCC | Clang/LLVM |
|---|---|---|
| Compiler “family” | GCC toolchain (gcc/g++ + binutils + libstdc++) | LLVM toolchain (clang + lld + libc++ optional) |
| Diagnostics | Strong, sometimes verbose; excellent for warnings as errors | Often clearer formatting; great template errors; strong static analysis ecosystem |
| Linker choices | Usually drives GNU ld (bfd) or gold depending on distro/config | Can drive GNU ld or lld; often easier to switch linkers |
| Cross-compiling | Very common with prefixed cross compilers (e.g., aarch64-linux-gnu-gcc) |
Excellent cross support via --target + sysroot; needs target runtimes available |
| Embedded (bare metal) | Widely used with vendor packages (arm-none-eabi-gcc, etc.) | Works well; but you still need the right startup files + libc/newlib + linker scripts |
“GCC vs Clang” is rarely about “which is better.” It’s usually about which defaults, runtime libraries, and link pipeline you’re tied to in a given environment. Once you understand the pipeline, switching compilers becomes a controlled change instead of a gamble.
Core concepts
Toolchain problems become easy when you can name the moving parts. This section builds a mental model you can reuse: if something breaks, you’ll know which component to inspect next.
1) What’s inside a toolchain?
The usual pieces (host builds and cross builds)
| Component | Typical tools/files | What it does |
|---|---|---|
| Compiler driver | gcc/g++, clang/clang++ |
Orchestrates the whole build: invokes the compiler, assembler, linker; picks defaults and search paths |
| Compiler | GCC frontends, LLVM/Clang | Turns C/C++ into assembly or machine code (often via an intermediate representation) |
| Assembler | as (GNU), integrated assembler (Clang) |
Turns assembly into object files (.o) |
| Linker | ld.bfd, gold, lld |
Combines objects + libraries into an executable, shared library, or firmware image |
| Binutils | ar, nm, objdump, readelf, strip |
Inspect and manipulate binaries: symbols, sections, relocations, debug info |
| Runtime + standard libs | libc, libm, libgcc; C++: libstdc++ or libc++ |
Startup code, basic runtime helpers, standard library implementations |
| Headers | /usr/include, target headers in sysroot |
Declarations used at compile-time (must match the target ABI) |
| Sysroot | Directory tree with target headers + libs | Prevents host/target mixing in cross builds |
2) The compilation stages (and why they matter)
The compiler driver runs multiple stages. Knowing them is the difference between “try random flags” and “fix the real issue.”
Stages in order
- Preprocess: expands
#includeand macros - Compile: turns preprocessed code into assembly or IR
- Assemble: turns assembly into object files (
.o) - Link: combines objects + libraries into final output
Common stage-specific flags
- Preprocess:
-I,-D,-U,-E - Compile:
-O*,-g,-std=,-Wall - Assemble: rare to tweak directly; sometimes
-Wa,pass-through - Link:
-L,-l,-Wl,pass-through,-static,-shared
A build can compile with the wrong headers and still fail later (or worse: it links and runs incorrectly). Always treat “where headers come from” and “which libc/ABI is used” as first-class concerns in cross builds.
3) Target triples, architecture, and ABI
A target triple is a compact description of what you’re building for, typically shaped like
arch-vendor-os-abi (for example: x86_64-pc-linux-gnu or aarch64-linux-gnu).
The key idea: you’re not just choosing a CPU; you’re choosing an ABI (calling conventions, data layout, libc expectations)
and a set of runtime libraries that must match.
Why ABI mismatches hurt so much
- Linker errors: “wrong ELF class”, “file in wrong format”, missing symbols
- Runtime crashes: stack corruption, wrong struct layout, mismatched C++ standard library
- Silent bugs: code “runs” but behaves incorrectly under load or on specific inputs
4) Build vs host vs target (cross-compiling vocabulary)
| Term | Meaning | Example |
|---|---|---|
| Build | The machine doing the compilation | Your laptop or a CI runner (x86_64 Linux) |
| Host | The machine the built tools will run on (relevant when building compilers/tools) | Building a cross-compiler that runs on your laptop |
| Target | The machine the compiled program will run on | ARM board, Raspberry Pi, microcontroller, etc. |
Step-by-step
This is a practical flow you can apply to real projects (Make/CMake, embedded, Linux targets). The goal is to make the toolchain explicit: target, sysroot, linker, and search paths.
Step 1 — Make the compiler show its work
Before you change flags, get visibility. You want to see: include search paths, library search paths, the linker line, and which tools are being invoked behind the driver.
# 1) Confirm versions + targets
gcc --version
clang --version
# 2) See where headers are searched (prints include paths)
echo | gcc -E -x c - -v 2>&1 | sed -n '/#include <...> search starts here:/,/End of search list./p'
echo | clang -E -x c - -v - 2>&1 | sed -n '/#include <...> search starts here:/,/End of search list./p'
# 3) Print the commands the driver would run (Clang: -###, GCC: -### also works on many setups)
clang -### hello.c 2>&1 | sed -n '1,120p'
gcc -### hello.c 2>&1 | sed -n '1,120p'
# 4) Split compile and link to isolate errors
gcc -c hello.c -o hello.o
gcc hello.o -o helloIf you don’t know which include paths and libraries are used, you can’t tell whether you’re compiling against the host environment or the intended target. The commands above are the quickest “X-ray” you can do.
Step 2 — Pick your compiler, then pick your runtime and linker intentionally
Both GCC and Clang can compile C/C++. The real friction usually comes from runtime and linking defaults:
which C++ standard library is used (libstdc++ vs libc++), which linker is used,
and which startup files and CRT objects get pulled in.
When GCC is the path of least resistance
- You’re on a distro where GCC is the default and all packages assume it
- You rely on vendor-provided cross toolchains (especially embedded)
- You want “what the ecosystem expects” (libstdc++, GNU ld defaults)
When Clang shines
- You want excellent diagnostics and tooling integration
- You want to drive
lldfor faster links - You want portable cross builds using
--targetand sysroots
Step 3 — Cross-compiling fundamentals: compiler prefix vs --target
There are two common ways to cross-compile:
Approach A: prefixed cross compilers
You install tools named like aarch64-linux-gnu-gcc and they default to that target.
- Easy mental model: the tool name implies the target
- Common in Linux cross packages and embedded vendor toolchains
- Still needs a correct sysroot for libraries and headers
Approach B: Clang with explicit target
You use clang --target=... and point it at the right sysroot and runtime libraries.
- One compiler binary can target many platforms
- Great for CI and multi-target builds
- Requires you to be explicit about sysroot and toolchain pieces
The sysroot concept in one sentence
A sysroot is a directory that looks like the target system’s / (headers in usr/include, libs in usr/lib, etc.),
so your compiler and linker can build as if they were “inside” the target.
Step 4 — Cross-compiling with CMake: use a toolchain file
In real projects, you rarely invoke the compiler directly. Build systems decide flags and link order, and they may auto-detect libraries from the host machine unless you constrain them. In CMake, the standard pattern is a toolchain file.
# toolchain-aarch64-linux.cmake
# Use this with: cmake -S . -B build-aarch64 -DCMAKE_TOOLCHAIN_FILE=toolchain-aarch64-linux.cmake
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)
# Prefixed cross compiler (example). Adjust to your installed toolchain.
set(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER aarch64-linux-gnu-g++)
# Sysroot from your target rootfs (or from your toolchain package).
# Common sources: a staged rootfs, an SDK, or a sysroot shipped with the toolchain.
set(CMAKE_SYSROOT /opt/sysroots/aarch64-linux-gnu)
# Ensure CMake finds headers/libs inside the sysroot, not on the host.
set(CMAKE_FIND_ROOT_PATH ${CMAKE_SYSROOT})
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
# Optional: make pkg-config target-aware (if you use it)
set(ENV{PKG_CONFIG_SYSROOT_DIR} ${CMAKE_SYSROOT})
set(ENV{PKG_CONFIG_PATH} "${CMAKE_SYSROOT}/usr/lib/aarch64-linux-gnu/pkgconfig:${CMAKE_SYSROOT}/usr/lib/pkgconfig:${CMAKE_SYSROOT}/usr/share/pkgconfig")
If you don’t constrain discovery, CMake may detect host libraries during configure (because it can run host tools),
and you’ll end up linking host .so files into a target binary. It might even “work” until you run it on the device.
The toolchain file above prevents that by forcing find logic to stay inside the sysroot.
Step 5 — Debug link failures like a pro: symbols, order, and architecture
Link errors are scary because they show up late. The fix is usually one of these: missing library, wrong library for the architecture, wrong link order, or ABI mismatch (especially in C++).
Quick link-debug checklist
- Verify the file architecture (
file,readelf -h) - List undefined symbols (
nm -u) - List what a library exports (
nm -D/objdump -T) - Check the final link line order (libs after objects)
Order matters (static libs)
With static libraries (.a), the linker typically resolves symbols left-to-right.
If -lfoo appears before the object that needs it, it may not be pulled in.
- Put libraries after the objects that reference them
- For circular deps, use
-Wl,--start-group/--end-group - Prefer shared libs in complex dependency graphs when possible
# Cross-compile a single C file for a Linux target with a sysroot (example for aarch64)
# Adjust prefixes and sysroot paths to match your setup.
export SYSROOT=/opt/sysroots/aarch64-linux-gnu
# Make pkg-config resolve libs/headers for the target (optional, but crucial if you use it).
export PKG_CONFIG_SYSROOT_DIR="$SYSROOT"
export PKG_CONFIG_PATH="$SYSROOT/usr/lib/aarch64-linux-gnu/pkgconfig:$SYSROOT/usr/lib/pkgconfig:$SYSROOT/usr/share/pkgconfig"
# Compile (no link yet)
aarch64-linux-gnu-gcc --sysroot="$SYSROOT" -O2 -g -c main.c -o main.o
# Link (note: libs after objects)
aarch64-linux-gnu-gcc --sysroot="$SYSROOT" main.o -o app -lm
# Sanity-check the result architecture
file app
readelf -h app | sed -n '1,25p'
For Linux targets, a common issue is a missing dynamic loader or mismatched runtime libraries.
Inspect dependencies with readelf -d (NEEDED entries) and verify your target has the required loader and libc versions.
For embedded/bare-metal, “won’t run” is often a linker script/startup mismatch, not the C code.
Step 6 — Embedded note: bare metal is still a toolchain problem
For microcontrollers, “cross-compiling” also includes startup code, linker scripts, and the C library choice (often newlib).
The toolchain is not just a compiler binary; it’s the entire set of rules that produce a flashable image.
Bare-metal reality check
- Startup/CRT objects define reset handlers and initialization
- Linker scripts define memory layout (flash/RAM, sections, vectors)
- Flags like
-mcpu,-mthumb, and float ABI must match the target - Debugging often starts with the map file (
-Wl,-Map=out.map)
Common mistakes
These are the issues behind most “compiler is broken” moments. The compiler is usually fine — the build context isn’t. Use this section as a quick troubleshooting index.
Mistake 1 — Host headers leak into a cross build
Symptoms: compile succeeds but runtime is broken, or you get subtle type/layout mismatches.
- Fix: use a sysroot and ensure include paths are inside it.
- Fix: print include search paths (
-E -v) to verify reality. - Fix: constrain CMake find logic via toolchain file.
Mistake 2 — Linking the wrong architecture library
Symptoms: “file in wrong format”, “wrong ELF class”, or a binary that won’t run on the device.
- Fix: run
fileandreadelf -hon every suspect.a/.so. - Fix: keep target libs inside the sysroot; avoid
/usr/libfrom host. - Fix: don’t hardcode
-Lpaths unless they’re target-specific.
Mistake 3 — Static library order (left-to-right resolution)
Symptoms: “undefined reference” even though “the library is there”.
- Fix: put libraries after the objects that need them.
- Fix: for circular deps, use
-Wl,--start-group/--end-group. - Fix: inspect symbols with
nmto confirm the symbol exists where you think it does.
Mistake 4 — Mixing C++ runtimes (libstdc++ vs libc++)
Symptoms: link failures involving std:: symbols, exceptions, RTTI, or ABI tags.
- Fix: standardize on one C++ runtime per target/toolchain.
- Fix: if using Clang, be explicit about
-stdlib=(only if you truly mean it). - Fix: rebuild third-party libs with the same compiler+stdlib+ABI settings.
Mistake 5 — “It links in CI but not locally” (or the reverse)
Symptoms: same repo, different machine, different result. Usually a PATH or package drift issue.
- Fix: print tool versions in CI and locally (
gcc --version,clang --version). - Fix: pin toolchain versions (container, Nix, locked apt repo, or vendor SDK).
- Fix: capture verbose build logs as artifacts (Make/CMake/Ninja).
Mistake 6 — Confusing “compiler flags” with “linker flags”
Symptoms: flags appear to do nothing, or they cause strange warnings.
- Fix: use the right pass-through mechanism (
-Wl,for linker,-Wa,for assembler). - Fix: split compile and link to see what stage is failing.
- Fix: read the printed commands (
-###/ verbose build) to verify flags land where intended.
When stuck, reduce the build to a single file and a single command. If you can’t reproduce the failure with a minimal compile+link, the bug is likely in build-system logic (paths, order, detection).
FAQ
What exactly is a “toolchain”?
A toolchain is the full set of tools and target-specific files needed to produce a runnable binary: compiler driver, compiler, assembler, linker, binutils, headers, and runtime libraries (plus sysroot for cross builds). Saying “I use GCC” is shorthand; the real question is “which linker, which libc, which headers, and which target?”
Should I choose GCC or Clang for embedded work?
Use what matches your ecosystem and constraints. Vendor SDKs and MCU workflows often assume GCC-based toolchains (startup files, linker scripts, prebuilt libs). Clang can absolutely work (and can be excellent), but you still need the correct runtime pieces for your target. If you want the lowest-friction path, start with the vendor-supported toolchain, then experiment with Clang once your build is reproducible.
What is a sysroot and when do I need it?
A sysroot is a directory tree containing the target system’s headers and libraries. You need it when cross-compiling for Linux-like targets (and sometimes even for “semi-hosted” embedded setups) to avoid mixing host and target artifacts. If you’re building for a different architecture than your build machine, assume you need a sysroot unless your toolchain explicitly bundles everything and you can prove it’s isolated.
Why do I get “undefined reference” even though I added -lfoo?
Usually one of these: the symbol is in a different library than you think, the link order is wrong (especially for static libs),
the library is the wrong architecture, or you’re mixing ABIs/runtimes. Inspect symbols with nm,
confirm architecture with file/readelf, and ensure libraries appear after the objects that reference them.
How do I tell which linker is being used?
Turn on command printing (verbose build) or use the driver’s “show commands” flags (e.g., -### in Clang).
Then look at the final link invocation: it will include the linker binary (ld, ld.bfd, gold, or lld).
You can also search the verbose output for collect2 (common in GCC flows) and for ld/lld paths.
Can Clang cross-compile without installing a full cross toolchain?
Clang can target many architectures via --target, but it still needs the target’s headers and runtime libraries to link correctly.
For Linux targets, that usually means a sysroot (from a rootfs, SDK, or distro packages). For bare metal, you also need startup code,
linker scripts, and an appropriate libc (or no libc).
What’s the most reliable way to avoid “works locally, fails in CI” builds?
Make the toolchain explicit and reproducible: pin versions, print versions in logs, and keep sysroots and toolchain files inside your project (or referenced by a versioned SDK). The fewer “auto-detected” dependencies, the fewer surprises.
Cheatsheet
A scan-fast checklist for debugging toolchains, GCC/Clang builds, and cross-compilation. Bookmark this section.
10-second triage
- Compile-time error: inspect include paths and preprocessing (
-E -v) - Link-time error: inspect link line, library order, and symbol presence (
nm) - Architecture error: verify
file/readelf -hon libs and outputs - Cross build: lock sysroot; prevent host leakage in CMake/tooling
Go-to commands
gcc -c x.c(compile only)gcc x.o -o x(link)clang -### x.c(show commands)echo | gcc -E -x c - -v(show include paths)nm -u app(undefined symbols)nm -D libfoo.so | grep name(exported symbols)readelf -h file(ELF header)objdump -T libfoo.so(dynamic symbol table)
Cross-compiling sanity checks
- Confirm cross compiler is used (tool name or
--target) - Confirm sysroot is set and consistent across compile and link
- Ensure
pkg-configresolves target packages (or disable it) - Check that
-Iand-Lpaths point into the sysroot - Run
fileon the final binary and key libraries
Symptoms → likely cause
| Symptom | Usually means | First check |
|---|---|---|
| “No such file or directory” on include | Missing header path / wrong sysroot | Include search list (-E -v) |
| “undefined reference to …” | Missing lib, wrong order, or ABI mismatch | Link line order + nm on libs |
| “wrong ELF class” / “file in wrong format” | Wrong architecture library/binary | file / readelf -h |
| Binary runs locally, not on device | Wrong runtime deps / dynamic loader mismatch | readelf -d (NEEDED) and target runtime |
| Embedded app doesn’t start | Startup/linker script/memory layout issue | Map file + linker script + vector table |
Wrap-up
Toolchains feel mysterious until you treat them like a pipeline you can inspect: stages, targets, sysroots, and link lines. Once you can answer “what target am I building for?” and “which headers/libs am I actually using?”, most build bugs stop being scary and start being mechanical.
Next actions (pick one)
- Print your build’s actual compile/link commands and save them in CI logs.
- Create a sysroot-based cross setup (even if you only cross-compile occasionally).
- Write a CMake toolchain file and commit it so teammates build the same way.
- When debugging, always reduce to “compile-only” then “link-only” to isolate the stage.
If you’re building embedded or IoT systems, toolchains sit next to protocols and deployment in your “core skills” stack. The related posts below pair well with this one when you’re moving from local builds to device reality.
Continue with the related posts section, then come back to the Cheatsheet when the next build error shows up.
Quiz
Quick self-check (demo). This quiz is auto-generated for hardware / iot / embedded.