Docker Production Engineering — Images That Behave Under an Orchestrator

A container image that builds is not a container image that is ready for production. The gap between the two is made of things that only appear under an orchestrator: what happens on SIGTERM, what the image weighs when a hundred nodes pull it at once, what is in layer three that should not be, and whether the build you run next month produces the same thing.

Reproducibility starts at FROM

FROM python:3.12-slim@sha256:2a3fbd...

Three levels of pinning, and they are not equivalent:

• python — resolves to latest. The image you build today and the one you build during an incident are different images, and nothing records what changed.
• python:3.12-slim — pinned to a tag. Better, but tags are mutable and get repointed. Your build can change without your repository changing.
• python:3.12-slim@sha256:... — pinned to a digest. The only reference that guarantees the same bytes.

Pin by digest for anything you deploy. Use a tool to update digests deliberately, so base image updates are a reviewed change rather than something that happens to you.

Layers, and the order that matters

Every instruction creates a layer, and layers are cached until one changes. Everything after a changed layer rebuilds.

The single most common mistake:

# Wrong: any source change reinstalls every dependency
COPY . .
RUN pip install -r requirements.txt

# Right: dependencies rebuild only when the manifest changes
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

The same pattern applies to every ecosystem: copy the manifest, install, then copy the source. On a large project this is the difference between a ten-second and a four-minute build, every time anyone changes a line.

Multi-stage: ship the artifact, not the toolchain

FROM golang:1.23@sha256:... AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/server ./cmd/server

FROM gcr.io/distroless/static-debian12@sha256:...
COPY --from=build /out/server /server
USER 65532:65532
ENTRYPOINT ["/server"]

The compiler, the module cache, and the source never reach the final image. The result is smaller, faster to distribute, and has a dramatically smaller set of packages that could need patching.

Multi-stage builds are worth it for any compiled language and for most interpreted ones where the build needs tooling the runtime does not.

Signal handling, and the ten-second delay nobody explains

This is the production failure most often shipped without being noticed.

CMD python -m app.server          # shell form
CMD ["python", "-m", "app.server"] # exec form

Shell form wraps the process in /bin/sh -c. The shell becomes PID 1 and does not forward SIGTERM to your application. When the orchestrator wants to stop the container, your process never hears about it, the grace period expires, and it gets SIGKILL.

The symptom is subtle and easy to live with for a long time: deploys take longer than they should, in-flight requests are dropped during rollouts, and connections are not closed cleanly. Nothing errors.

Always use exec form for CMD and ENTRYPOINT. If you genuinely need shell features, invoke the shell explicitly and use exec for the final command so it replaces the shell:

ENTRYPOINT ["/bin/sh", "-c", "exec /app/server --config \"$CONFIG_PATH\""]

Run as a non-root user

RUN useradd --system --uid 10001 --no-create-home app
USER 10001:10001

A process compromised inside a container running as uid 0 is root in that container — and where user namespaces are not in use, that uid maps to real root on the host, which turns a contained problem into a much less contained one.

Use a numeric UID rather than a name. Orchestrator security policies such as runAsNonRoot evaluate the numeric UID and cannot resolve a username against the image’s /etc/passwd.

Where the application needs to write, create the directory and chown it at build time. Where it needs a privileged port, do not — bind high and map it.

Secrets never belong in an image

# Both of these put the value in image history, permanently
ENV DATABASE_PASSWORD=hunter2
ARG NPM_TOKEN

ENV values persist in the image and are readable by anyone who can pull it. ARG values appear in build history and in build logs. Neither is removed by unsetting it later.

For build-time credentials, use BuildKit secret mounts, which make the secret available to a single RUN without writing it into any layer:

RUN --mount=type=secret,id=npmtoken \
    NPM_TOKEN="$(cat /run/secrets/npmtoken)" npm ci

For runtime credentials, inject them at runtime from the orchestrator’s secret mechanism. The image should contain no credential of any kind.

Health checks, and where they belong

HEALTHCHECK --interval=30s --timeout=3s --start-period=40s --retries=3 \
  CMD ["/app/healthcheck"]

HEALTHCHECK is honoured by Docker and Compose. Kubernetes ignores it entirely and uses the probes defined in the pod spec instead. Defining it is still worthwhile for local development and Compose deployments, but do not expect it to do anything in a cluster.

Whichever mechanism, start-period — or startupProbe in Kubernetes — is what stops a slow- starting application from being killed before it is ready. Applications that load large models or warm caches need this explicitly; liveness settings tuned for a web service will kill them in a loop.

Build context

.dockerignore is not an optimisation, it is a correctness control. Without it, the entire directory is sent to the daemon — including .git, local environment files, credentials, and build output.

.git
.env
*.pem
node_modules
__pycache__
dist

A COPY . . with no .dockerignore is one of the most common ways a credential ends up in an image nobody meant to put it in.

Verifying an image

# What is actually in the layers, and how big is each one?
docker history --no-trunc --format '{{.Size}}\t{{.CreatedBy}}' myimage:tag

# Confirm the runtime user is not root
docker inspect --format '{{.Config.User}}' myimage:tag

# Confirm exec form: this should be a JSON array, not a shell string
docker inspect --format '{{json .Config.Entrypoint}}' myimage:tag

# Does it stop promptly?
cid=$(docker run -d myimage:tag)
time docker stop "$cid"

Run these in CI rather than by hand. They are fast, they are deterministic, and they catch the regressions that otherwise reach production. The Docker Production Auditor applies the static half of these checks to a Dockerfile in your browser.

Checklist

• Base image pinned by digest, updated deliberately
• Dependency install before source copy
• Multi-stage build where the toolchain is not needed at runtime
• CMD and ENTRYPOINT in exec form; SIGTERM verified to reach the process
• Non-root numeric UID
• No secrets in ENV or ARG; BuildKit secret mounts for build-time credentials
• .dockerignore covering VCS metadata, environment files, and keys
• Start period or startup probe sized to real startup time
• Image contents verified in CI, not reviewed by eye