Cloud & DevOps · Docker

Docker for Developers: Images, Layers, and Caching Explained

Understand why builds are slow and how to speed them up.

Reading time: ~8–12 min
Level: All levels
Updated:
Jump to Quickstart View Cheatsheet
By Published Updated

Docker builds feel “mysteriously slow” until you see what Docker actually does: it turns your Dockerfile into a stack of layers, then reuses (or invalidates) those layers through cache keys. Once you understand that mental model, you can make builds predictable, fast, and CI-friendly—without cargo-culting random flags. This guide explains images, layers, and caching in practical terms, then walks through an optimized build setup you can copy.

Quickstart

These are the highest-impact changes that speed up Docker builds for most apps (Node, Python, Java, Go, etc.). You can implement them in under an hour and feel the difference immediately.

1) Fix cache invalidation (order your Dockerfile)

Docker caches per instruction. If you change a line early (like COPY . .), every later layer becomes “new”. Put slow, stable steps first and fast-changing steps last.

  • Copy dependency files first (e.g., package-lock.json, requirements.txt)
  • Install dependencies before copying the whole repo
  • Copy app source after dependencies so code edits don’t force reinstall

2) Add a strict .dockerignore

Your “build context” is what Docker sends to the daemon. If it includes node_modules or build artifacts, you’ll upload huge blobs and invalidate cache unnecessarily.

  • Ignore dependencies (node_modules, .venv)
  • Ignore caches and build output (dist, build, __pycache__)
  • Ignore secrets and local config (.env, *.pem)

3) Use multi-stage builds (separate build vs runtime)

Keep compilers and dev tools out of your final image. Builds stay clean, runtime images stay small, and layer reuse improves.

  • Stage A: install deps + build
  • Stage B: copy only the build output + minimal runtime deps
  • Result: smaller images and fewer moving parts in production

4) Turn on BuildKit + cache mounts

BuildKit is the modern build engine (parallel builds, better caching). Cache mounts avoid re-downloading packages each build.

  • Build with DOCKER_BUILDKIT=1 (or set it globally)
  • Use RUN --mount=type=cache for package managers
  • In CI: export/import cache with buildx to persist across runs
Fast diagnostic

If a tiny code change triggers a full dependency reinstall, your Dockerfile is invalidating cache too early. Move dependency install steps above COPY . ., and make sure your build context is small.

Overview

“Docker for Developers” is less about memorizing commands and more about learning one core idea: a Docker image is built as a sequence of layers, and Docker tries to reuse those layers via build cache. When cache hits, builds are fast. When cache misses, Docker reruns expensive steps (dependency installs, compiles, downloads).

What you’ll be able to do after this post

  • Explain the difference between an image, a container, and a layer
  • Predict when Docker will reuse cache—and when it must rebuild
  • Write Dockerfiles that keep dependency installs cached
  • Reduce image size without sacrificing developer ergonomics
  • Make CI builds faster with exported BuildKit caches
What “slow build” usually means

In most teams, slow builds come from one of three things: (1) huge build context, (2) cache invalidation from Dockerfile order, or (3) rebuilding dependencies on every run. Fixing those is usually worth more than switching base images or shaving 5MB from final size.

Core concepts

Let’s build a mental model that makes Docker builds predictable. Once you see how Docker interprets a Dockerfile, you’ll stop guessing and start designing for cache.

Images vs containers

Image

A read-only template: filesystem snapshot + metadata (entrypoint, env vars, labels). Images are built from layers and shared across containers.

  • Can be tagged (myapp:1.2)
  • Stored locally or in a registry
  • Immutable (new builds create new layers)

Container

A running (or stopped) instance of an image with a small writable layer on top. Containers are disposable by design.

  • Has process state (PID, ports, logs)
  • Has a writable layer (changes vanish when removed unless volumes are used)
  • Can be recreated from the same image deterministically

Layers: the real unit of reuse

Most Dockerfile instructions create a new layer (especially RUN, COPY, ADD). Layers are content-addressed: if the instruction and its inputs are the same, Docker can reuse the previously built layer.

How cache works (simplified)

Dockerfile instruction What Docker “hashes” for cache Typical cache pitfall
FROM Base image digest Floating tags (latest) change silently
RUN Command + previous layer Installing deps after copying all source
COPY File contents + paths Copying the whole repo early invalidates later steps
ENV/ARG Values affect later layers Changing build args forces downstream rebuilds

Build context: “what you send” matters

When you run docker build ., Docker sends the directory (the build context) to the build engine. If that context includes huge directories, every build becomes slower—even if your Dockerfile is perfect. A good .dockerignore is not optional; it’s a performance feature.

Cache invalidation: why one edit nukes everything

Docker can reuse cache only up to the first instruction that changes. After a cache miss, every subsequent instruction must rebuild because its “previous layer” is different. That’s why Dockerfile order is everything.

The most expensive line in many Dockerfiles

COPY . . placed before dependency installation. It looks harmless, but it changes whenever any file changes, which forces the dependency install step to rerun. Copy dependency manifests first, install, then copy the rest.

BuildKit: modern builds and better caching

BuildKit is Docker’s newer build engine. It adds features developers actually care about: parallel execution, cache mounts, secret mounts, and the ability to export/import caches (gold for CI). You don’t have to “learn BuildKit” as a separate tool—you just need to know what it enables.

Step-by-step

This section is a practical, repeatable build setup you can apply to most web services. The goal: fast rebuilds during development, small and safe runtime images, and predictable cache behavior in CI.

Step 1 — Audit what’s slow (without guessing)

What to look for

  • Dependency install runs every build (npm/pip/apt/maven)
  • Build context is huge (uploads a lot each build)
  • Large layers from copying artifacts into the image
  • Frequent cache misses from early COPY instructions

Quick signals

  • Build output shows many steps are not “CACHED”
  • Dependency step takes minutes and repeats often
  • Local builds are much faster than CI builds (cache not persisted)
  • Small code changes trigger full rebuilds

Step 2 — Shrink the build context with .dockerignore

A lean context speeds up builds and reduces accidental cache invalidation. Start strict, then allow only what the image actually needs. 

# .dockerignore (starter)
.git
.gitignore
Dockerfile
docker-compose.yml

# Dependencies / virtual envs
node_modules
.venv
venv
__pycache__
*.pyc

# Build output
dist
build
coverage
*.log

# IDE / OS noise
.vscode
.idea
.DS_Store

# Secrets / local env
.env
.env.*
*.pem
*.key
Rule of thumb

If a directory is generated (deps, caches, build output), ignore it. Your image should build from source + lockfiles, not from whatever happened to be on a developer’s laptop.

Step 3 — Write a cache-friendly Dockerfile (multi-stage + stable first)

Below is a pattern you can adapt. It’s intentionally explicit about what changes often vs what changes rarely. The key idea is: dependency layers should depend only on dependency files, not your entire repo. 

# syntax=docker/dockerfile:1
# Example: Node.js service with a build step (adapt to your stack)

FROM node:20-slim AS deps
WORKDIR /app

# Copy only dependency manifests first (stable, cacheable)
COPY package.json package-lock.json ./

# BuildKit cache mount keeps the npm cache between builds
RUN --mount=type=cache,target=/root/.npm \
    npm ci --include=dev

FROM node:20-slim AS build
WORKDIR /app

# Reuse node_modules from deps stage
COPY --from=deps /app/node_modules ./node_modules

# Now copy the rest of the source (changes frequently)
COPY . .

# Build your app (optional; e.g., TypeScript, bundlers)
RUN npm run build

# Final runtime image: smaller + fewer tools
FROM node:20-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production

# Copy only what runtime needs
COPY package.json package-lock.json ./
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist

# Optional: run as non-root (good default)
USER node

EXPOSE 3000
CMD ["node", "dist/server.js"]

Why this Dockerfile stays fast

  • Dependency install runs only when package*.json changes
  • App code changes don’t bust the dependency cache
  • Build stage can be rebuilt without touching runtime structure
  • Final image copies only runtime artifacts (smaller, safer)

Step 4 — Make cache survive CI with Buildx cache export/import

Local builds feel fast because your machine retains cache. CI often starts “cold”, so it rebuilds everything. Fix that by exporting a cache and reusing it on later runs. The snippet below shows the concept: build with BuildKit, pull previous cache, then push updated cache. 

# GitHub Actions (illustrative) - BuildKit cache to GitHub Actions cache storage
# Requires: docker/setup-buildx-action and docker/build-push-action
name: build

on:
  push:
    branches: [ "main" ]

jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-buildx-action@v3

      - uses: docker/build-push-action@v6
        with:
          context: .
          file: ./Dockerfile
          push: false
          tags: myapp:ci
          cache-from: type=gha
          cache-to: type=gha,mode=max
When to use cache export

If your CI builds take minutes mostly due to dependency installs (npm/pip/apt), exporting cache is a big win. If your builds are slow due to compiling huge binaries or running tests inside the image, you’ll still benefit—but also consider moving tests earlier in CI to avoid rebuilding images just to fail later.

Step 5 — Layer hygiene: smaller layers, fewer surprises

Caching and size are related: fewer, cleaner layers tend to be reused more reliably and are easier to reason about. Here are pragmatic habits that pay off in most projects:

Good defaults

  • Pin base images (avoid “surprise rebuild” from latest)
  • Install OS packages in one RUN layer (and clean up)
  • Copy only what you need into runtime stage
  • Use non-root user when possible

Things to avoid

  • Copying secrets into the image (use secret mounts or runtime env)
  • Shipping build tools in runtime images
  • Using a huge context “because it’s easier”
  • Mixing frequently changing files with stable dependency steps

Step 6 — Keep the dev loop fast (without rebuilding constantly)

For development, the fastest rebuild is often no rebuild at all. Use containers for dependencies and environment consistency, but bind-mount your code so changes reflect immediately. Then rebuild images mainly when dependencies or system packages change.

A simple dev rule

Rebuild the image when dependencies change. Restart the container when code changes. Recreate volumes when data changes. Separating those rhythms keeps Docker feeling “instant” instead of “waiting for installs.”

Common mistakes

These are the repeat offenders behind “why are Docker builds slow?” and “why did CI suddenly get worse?”. Each mistake includes a concrete fix you can apply today.

Mistake 1 — COPY . . before installing dependencies

A single code change invalidates the dependency layer and forces a full reinstall.

  • Fix: copy lockfiles first, install deps, then copy the rest of the source.
  • Fix: keep the build context small so “copy” is stable.

Mistake 2 — No .dockerignore (or it’s too permissive)

Huge contexts slow builds and cause cache misses from irrelevant files.

  • Fix: ignore dependencies, build outputs, and secrets.
  • Fix: treat .dockerignore like a performance budget.

Mistake 3 — Using floating base tags (latest)

The base image can change under you, busting cache and changing behavior unexpectedly.

  • Fix: pin to a specific major/minor (or digest) for reproducible builds.
  • Fix: update intentionally on a schedule, not accidentally on a random day.

Mistake 4 — Installing OS packages without cleanup

Extra package lists and caches bloat layers and slow pulls/pushes.

  • Fix: combine apt steps and clean apt lists in the same layer.
  • Fix: keep build tools in a build stage, not runtime.

Mistake 5 — Treating CI as “stateless” (no cache)

Every CI run becomes a cold start: dependency downloads repeat endlessly.

  • Fix: use BuildKit cache export/import (e.g., buildx + GHA cache).
  • Fix: ensure your cache key changes only when dependency inputs change.

Mistake 6 — Baking secrets into images

This is both a security risk and a caching problem (secrets change and bust layers).

  • Fix: inject secrets at runtime (env/secret manager) or use secret mounts during build.
  • Fix: verify images don’t contain secret files via simple greps or scans.
Caching “works” until it doesn’t

If your build depends on time-varying downloads (unpinned packages, mutable URLs), you may get “cache hits” that hide drift locally but break in CI. Pin dependencies and make inputs explicit so cache remains trustworthy.

FAQ

Does every Dockerfile line create a new layer?

Not every line, but most “filesystem-changing” instructions do. In practice, RUN, COPY, and ADD create new layers, while metadata-only instructions like CMD and ENTRYPOINT typically don’t add significant filesystem content. For caching, treat each instruction as a potential cache boundary: once one changes, everything after it rebuilds.

Why does changing one file force Docker to reinstall dependencies?

Because your dependency install layer depends on earlier layers. If you copy the whole repo before installing dependencies, any file change modifies the COPY layer, and Docker must rebuild every downstream step—including dependency installation. Fix it by copying only dependency manifests first, installing deps, then copying the rest.

What’s the difference between Docker cache and registry layers?

Build cache is about reusing work; registry layers are about distributing results. Locally, Docker can reuse previously built layers as cache hits. Registries store image layers so other machines can pull them. With BuildKit, you can also export a “cache” (not just an image) so CI can reuse work across runs even when images aren’t pushed.

Should I use multi-stage builds even for small apps?

Usually, yes—because it separates concerns cleanly. Multi-stage builds let you keep compilers, dev dependencies, and build tooling out of your runtime image. That improves security posture, reduces image size, and makes caching more predictable (build steps don’t leak into runtime layers).

Is BuildKit required to get good caching?

You can get decent caching without it, but BuildKit makes it much better. BuildKit adds cache mounts (huge for package managers), parallelism, and cache export/import for CI. If you’re optimizing builds seriously, enabling BuildKit is one of the highest leverage changes.

How do I keep my dev workflow fast without rebuilding all the time?

Bind-mount code and rebuild only when dependencies change. Use Docker to standardize the environment and run services, but avoid rebuilding the image on every code edit. A common rhythm: rebuild image when lockfiles change, restart container when app code changes, and use volumes for persistent data.

Cheatsheet

Scan this when you’re about to write (or review) a Dockerfile. It’s optimized for fast wins: fewer cache misses, smaller images, calmer CI.

Cache rules (the essentials)

  • Put stable steps first, changing steps last
  • Copy dependency files before copying full source
  • One cache miss rebuilds everything after it
  • Keep the build context small (.dockerignore)
  • Pin inputs (base image, lockfiles) for reproducibility

BuildKit checklist

  • Enable BuildKit (DOCKER_BUILDKIT=1)
  • Use --mount=type=cache for package managers
  • Use --mount=type=secret for private tokens (don’t bake secrets)
  • In CI: cache-from + cache-to to persist cache
  • Prefer npm ci/pip --requirement + lockfiles for stable installs

Smaller images (practical)

  • Use multi-stage builds (build tools stay out of runtime)
  • Copy only runtime artifacts (e.g., dist/, compiled binaries)
  • Remove apt caches in the same layer as install
  • Run as non-root where possible

Debug the next slow build

  • Which step stopped being “CACHED”?
  • Did the build context change (new files, big dirs)?
  • Did a build arg or env change upstream?
  • Is CI missing cache export/import?
  • Are dependencies pinned (lockfile present and used)?

Wrap-up

Docker builds become easy to optimize once you internalize the core rule: cache is per instruction, and a single early change forces everything after it to rebuild. Shrink your build context, order your Dockerfile so dependency layers stay stable, and use BuildKit to keep downloads cached.

If you want the fastest wins, do this next

  • Add (or tighten) .dockerignore
  • Move dependency installs above COPY . . by copying lockfiles first
  • Split build and runtime with multi-stage builds
  • Enable BuildKit and add cache export/import in CI
A good Dockerfile is a build pipeline

Treat it like one: stable inputs first, expensive steps cached, outputs minimized, and behavior reproducible. That mindset turns “Docker is slow” into “Docker is predictable.”

Want to go deeper into the ecosystem around Docker builds—deployments, pipelines, and operational reliability? Check the related posts below for Kubernetes, CI/CD, GitOps, Terraform, and observability patterns.

Quiz

Quick self-check (demo). This quiz is auto-generated for cloud / devops / docker.

1) In terms of Docker caching, what usually happens after the first cache miss during a build?
2) Which change most commonly causes dependency installs to rerun on every build?
3) What is the primary purpose of a .dockerignore file?
4) Which approach best keeps production images small and safer?
UniLab Editorial
Modern learning notes for practical builders.