This page assumes the FastAPI + vLLM serving stack from Inference & Serving as the thing being containerized, and builds toward the Helm Chart & Release Checklist Code Lab where this image gets deployed to Kubernetes.

A GPU program in a container needs three layers, and only the last one comes from your image:

1. The GPU driver - on the host, never in the image. The driver (including libcuda.so) is tied to the host's kernel and GPU. The NVIDIA Container Toolkit injects the driver libraries and device nodes into the container when you run it with GPU access - docker run --gpus all, or in Kubernetes the NVIDIA device plugin plus the NVIDIA runtime (runtimeClassName: nvidia where required).
2. CUDA user-space libraries (runtime, cuBLAS, cuDNN, NCCL) - these can come from the image, and they must be supported by the host driver version (newer CUDA needs a newer driver).
3. Your framework - PyTorch, vLLM and similar.

So where do the CUDA libraries come from? Pip wheels bring their own: the Linux CUDA builds of torch and vllm depend on nvidia-* pip packages containing the CUDA runtime, cuBLAS, cuDNN and NCCL. A plain Python base image therefore works for many inference servers:

# Works for pip-installed CUDA wheels, as long as the container runs with --gpus all
FROM python:3.12-slim
RUN pip install --no-cache-dir vllm
# docker run --gpus all ... -> torch.cuda.is_available() == True

A CUDA base image (nvidia/cuda:<version>-runtime-<os>, or a vendor image such as vllm/vllm-openai that pins CUDA, PyTorch and vLLM together) is the better choice when:

• you compile CUDA code (flash-attn or custom kernels from source) - that needs the -devel image's nvcc and headers, typically in a build stage;
• a library expects system CUDA libraries rather than pip-provided ones (some TensorRT and custom C++ builds);
• you want one known-good combination of CUDA, cuDNN and NCCL across every service.

# CUDA runtime base + a virtualenv
FROM nvidia/cuda:12.8.1-runtime-ubuntu24.04
RUN apt-get update && apt-get install -y python3.12 python3.12-venv
# Ubuntu 24.04 blocks system-wide pip installs (PEP 668), so install into a virtualenv
RUN python3.12 -m venv /opt/venv && /opt/venv/bin/pip install vllm
ENV PATH="/opt/venv/bin:$PATH"

When torch.cuda.is_available() is False in a container, check in this order: the container was started without GPU access (no --gpus, missing device plugin or runtime class); a CPU-only wheel was installed (for example from the CPU index URL); or the host driver is too old for the wheel's CUDA version (nvidia-smi shows the maximum CUDA version the driver supports).

Where the size actually comes from, and what to do about each source:

Should model weights live inside the image or be mounted at runtime? Baking multi-GB weights into the image makes every rebuild re-push the full weight layer even for a one-line code change, and couples model versioning to container versioning. Mounting weights from a PersistentVolume, object storage (S3/GCS), or the HuggingFace Hub cache at container start keeps the image itself small and lets model version and code version roll independently - this is the pattern used in the Helm Chart Code Lab's values.yaml (modelPath mounted as a volume rather than COPY'd).

# ---- Stage 1: build ----
FROM nvidia/cuda:12.8.1-devel-ubuntu24.04 AS builder
RUN apt-get update && apt-get install -y python3.12 python3-pip python3.12-venv
WORKDIR /build
COPY requirements.txt .
RUN python3.12 -m venv /opt/venv \
    && /opt/venv/bin/pip install --no-cache-dir -r requirements.txt

# ---- Stage 2: runtime ----
FROM nvidia/cuda:12.8.1-runtime-ubuntu24.04
RUN apt-get update && apt-get install -y python3.12 --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*
COPY --from=builder /opt/venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
WORKDIR /app
COPY serve.py .
EXPOSE 8000
CMD ["python3.12", "serve.py"]

Why this matters here specifically: the -devel CUDA image (with nvcc, headers, and build tools for compiling packages like flash-attn from source) can be 2-3 GB larger than the -runtime variant. Building in a -devel stage and shipping only the resulting virtualenv on a -runtime base can meaningfully cut final image size while keeping the exact same compiled binaries. See the Code Lab Dockerfile for a complete, runnable version of this pattern.