This article is a localized edition of the Japanese original. Read the Japanese original
To use an NVIDIA GPU from a Docker container, install the NVIDIA driver and the "NVIDIA Container Toolkit" on the host, then start the container with the GPU passed in, as in docker run --gpus all. You do not need to install a driver inside the container. This article covers the setup steps on a Linux server, how to choose which GPUs to start with, how to write it in Docker Compose, and how to read and fix the errors you are most likely to see.
It assumes Ubuntu-family Linux, Docker Engine, and the stable release of the NVIDIA Container Toolkit (as of October 2026). Package repositories and detailed steps can change, so when you do the work, check NVIDIA's official documentation (the NVIDIA Container Toolkit installation guide) as well.
Understand how it works first
Most problems with GPU containers come from mixing up what has to be installed where. The roles are split like this.
- NVIDIA driver on the host: This is what actually runs the GPU. Containers share and use the host's driver.
- NVIDIA Container Toolkit: When a container starts, it hands the host driver's libraries and device files over to the container.
- CUDA libraries inside the container image: The CUDA runtime, cuDNN, and frameworks such as PyTorch go in the image.
You generally do not need to install the CUDA Toolkit on the host. Conversely, installing an NVIDIA driver inside the image is a mistake: it can conflict with the host driver and stop things from working.
Step 1: Check the GPU and driver on the host
First, confirm that the host itself can see the GPU. If this fails, no amount of Docker configuration will fix it.
nvidia-smiIf you see the GPU name, the driver version, and "CUDA Version" in the upper right, you are ready. This "CUDA Version" shows the highest CUDA version that the driver supports. Make a note of it, because you will use it as the benchmark when choosing an image later.
If nvidia-smi is not found, or it prints "NVIDIA-SMI has failed", the driver is not installed or not loaded. On Ubuntu, install it with the ubuntu-drivers command or with packages distributed by NVIDIA. After updating the driver, a reboot is required.
Step 2: Install the NVIDIA Container Toolkit
Register the NVIDIA repository and install the nvidia-container-toolkit package. The following is an apt-based example that follows the official guide.
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkitNext, configure Docker so that it can use the NVIDIA runtime, and restart Docker.
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockernvidia-ctk runtime configure adds the NVIDIA runtime settings to /etc/docker/daemon.json. If you already use your own daemon.json, back up the file before running the command and check its contents afterward to be safe. If you are working on a production server, also note that restarting Docker stops running containers (containers without a restart policy do not come back automatically).
Step 3: Start a test container
The verification method in the official guide is to run nvidia-smi in a plain Ubuntu image. Because the Container Toolkit passes nvidia-smi and the driver libraries into the container, it works even with an image that does not include CUDA.
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smiIf you see the same list of GPUs as when you ran it on the host, it worked. Depending on your environment, --runtime=nvidia can be omitted, but adding it makes it easier to isolate the problem when things do not work.
How to specify GPUs
With --gpus, you can specify the number or the index numbers of the GPUs to pass in.
# All GPUs
docker run --rm --gpus all ubuntu nvidia-smi
# Just two (you do not choose which ones)
docker run --rm --gpus 2 ubuntu nvidia-smi
# By index (comma-separated values need doubled quotes)
docker run --rm --gpus device=0 ubuntu nvidia-smi
docker run --rm --gpus '"device=0,1"' ubuntu nvidia-smiThe quotes are doubled when you specify several indexes so that the comma is not interpreted as an option separator. With single quotes only, you get an "invalid argument" type of error.
Depending on the configuration, the index numbers may not appear in the order you expect. To check which GPU was passed in, run nvidia-smi -L inside the container and confirm by UUID.
Writing it in Docker Compose
In Compose, you write the GPU under deploy.resources.reservations.devices of the service.
services:
ollama:
image: ollama/ollama
ports:
- "127.0.0.1:11434:11434"
volumes:
- ollama:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all # 1 means just one GPU
capabilities: [gpu]
restart: unless-stopped
volumes:
ollama:To use a specific GPU, write device_ids: ["0"] instead of count. You cannot specify count and device_ids at the same time. Omitting capabilities causes an error, so state [gpu] explicitly. Newer versions of Compose also support a shorter form that puts gpus directly under the service, but only a limited range of versions supports it, so if compatibility matters, the form above is the safer choice.
This example runs Ollama, which serves a local LLM. It is a common combination in setups that use a local LLM to help draft WordPress articles or assign tags. The port is published with a 127.0.0.1: prefix so that an API with no authentication is not opened to the internet. If you need to use it from outside, put a reverse proxy in front and apply authentication or restrict the source of access.
Common errors and fixes
If you read the error message, you can usually tell which layer is stopping. They are listed here in the order they are most likely to occur.
could not select device driver “” with capabilities: [[gpu]]
Docker cannot find the mechanism for passing in the GPU. The main causes are that the Container Toolkit is not installed, or that Docker was not restarted after installing it. Check the package with dpkg -l | grep nvidia-container-toolkit, then redo nvidia-ctk runtime configure from Step 2 and restart. If the daemon runs somewhere other than the standard location, such as Docker Desktop (Linux version) or the Snap version of Docker, the configuration file location is different, so check the official documentation for the steps for each.
Failed to initialize NVML: Driver/library version mismatch
This often appears when you updated the host's driver but did not reboot. The version of the loaded kernel module does not match the version of the libraries on disk. Rebooting the host fixes it in most cases. Because automatic updates can also upgrade the driver, consider pinning the driver packages on GPU servers.
CUDA driver version is insufficient for CUDA runtime version
This appears when the CUDA in the image is newer than the highest version the host's driver supports. Compare the "CUDA Version" you noted in Step 1 with the CUDA version of the image (for nvidia/cuda images, it is included in the tag). The fix is either to upgrade the driver or to switch to an image that uses a CUDA version within the supported range.
It works right after startup, but after a while the GPU disappears
The symptom where nvidia-smi inside the container starts returning "Failed to initialize NVML: Unknown Error" has been reported as a known issue that occurs when device permissions are dropped after something like systemctl daemon-reload is run on the host. Updating the Container Toolkit or switching to the CDI (Container Device Interface) method can resolve it in some cases. The fix varies by version, so check the troubleshooting section of NVIDIA's documentation. Rootless Docker also needs separate configuration, so it is easier to isolate problems if you first confirm that things work with regular Docker and move on afterward.
Windows (WSL 2) and macOS
On Windows, you can use an NVIDIA GPU with Docker Desktop's WSL 2 backend. In that case, install the driver on the Windows side, and do not install a Linux NVIDIA driver inside the WSL 2 Linux. Installing a Linux driver inside WSL can overwrite the libraries provided from the Windows side and stop things from working.
Docker Desktop on macOS runs on top of a Linux virtual machine, so you cannot pass an NVIDIA GPU to a container.
Things to watch in operation
- GPU memory is not limited by
--memory:--memoryand--cpusset upper limits for the host's memory and CPU, not for GPU memory (VRAM). If several containers share one GPU, they can compete for VRAM and one of them may crash. Giving each workload its own GPU is the simplest approach. - Image size: Images that include CUDA can be several GB. If you only need to run things,
runtimeandbasevariants come out smaller thandevelvariants. - Plan driver updates: Upgrading the driver requires a reboot and changes the range of CUDA versions supported. Try it in a test environment first, then apply it to production.
Summary
- To use a GPU in a container, you need the host's NVIDIA driver and the NVIDIA Container Toolkit, and you do not install a driver in the container.
- Restart Docker after
nvidia-ctk runtime configure --runtime=docker, and verify withdocker run --rm --gpus all ubuntu nvidia-smi. - Specify GPUs with
--gpus all,--gpus 2, or--gpus '"device=0,1"'. In Compose, writedriver: nvidiaandcapabilities: [gpu]underdeploy.resources.reservations.devices. - When you hit an error, suspect these in order: Toolkit not installed or Docker not restarted, no reboot after a driver update, and a CUDA/driver mismatch.
- Publish an API with no authentication (such as Ollama) only on
127.0.0.1.
Questions about WordPress? The publisher of this site is based in Japan and replies in Japanese. Contact (Japanese)