Operating on GPU Compute

Last updated 2026-07-15 ·a8bed9a

The cluster has two GPU paths: an NVIDIA RTX 5070 Ti on gpu-1 managed by the GPU Operator, and Intel Arc iGPUs on the three mini nodes exposed through DRADynamic Resource AllocationThe Kubernetes API for requesting specialised hardware — GPUs and accelerators — with far more expressiveness than the older device-plugin counter model. (Dynamic Resource Allocation). Both have Talos-specific quirks, and both need different tools to inspect and troubleshoot.

This post covers day-to-day commands for checking GPU state, managing workloads, and debugging the issues you will eventually hit. For the build story, see GPU Compute — NVIDIA and Intel and GPU Containers on Talos — The Validation Fix.

Source your environment before running commands:

source .env   # sets KUBECONFIG

Overview

Frank runs two GPU stacks side by side:

• NVIDIA (gpu-1): GPU Operator managing the RTX 5070 Ti. Deployment is via gpu-operator Helm chart with Talos-specific extensions and containerd runtime config.
• Intel (mini-1/2/3): Intel Resource Drivers for Kubernetes using DRA. One DaemonSet per node, no special containerd config needed.

    graph LR
  subgraph NVIDIA["NVIDIA Stack (gpu-1)"]
    N1["GPU Operator<br/>Helm chart"]
    N2["nvidia-container-toolkit<br/>containerd runtime"]
    N3["RTX 5070 Ti<br/>16 GB GDDR7"]
    N1 --> N2 --> N3
  end
  subgraph INTEL["Intel Stack (mini-1/2/3)"]
    I1["Intel DRA<br/>Resource Driver"]
    I2["Intel Arc iGPU"]
    I1 --> I2
  end
  subgraph SCHED["Workload Scheduling"]
    S1["kube-scheduler"]
    S1 -->|"nvidia.com/gpu"| N3
    S1 -->|"gpu.intel.com"| I2
  end
  

Verify

# NVIDIA — device plugin registered
kubectl get node gpu-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'
# Should return: 1

# Intel — ResourceSlices published
kubectl get resourceslice -o wide
# Should show three slices, one per mini node

Observing State

NVIDIA GPU (gpu-1)

The GPU Operator runs several pods on gpu-1. Check they are all healthy:

kubectl get pods -n gpu-operator -o wide

You should see pods for the device plugin, feature discovery, DCGMData Center GPU ManagerNVIDIA's GPU telemetry daemon. It is what exports utilisation, memory and temperature to Prometheus so a GPU node can be observed like any other. exporter, and validation markers DaemonSet — all Running and 1/1.

To run nvidia-smi, exec into the DCGM exporter pod:

POD=$(kubectl get pod -n gpu-operator -l app=nvidia-dcgm-exporter \
  -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n gpu-operator "$POD" -- nvidia-smi

$ POD=$(kubectl get pod -n gpu-operator -l app=nvidia-dcgm-exporter -o jsonpath='{.items[0].metadata.name}'); kubectl exec -n gpu-operator "$POD" -- nvidia-smi
Mon Apr 20 16:55:31 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.211.01             Driver Version: 570.211.01     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
|   0  NVIDIA GeForce RTX 5070 Ti     Off |   00000000:01:00.0 Off |                  N/A |
|  0%   33C    P8             19W /  300W |    7956MiB /  16303MiB |      0%      Default |
+-----------------------------------------------------------------------------------------+

For a quick view of allocatable resources:

kubectl describe node gpu-1 | grep -A 10 "Allocated resources"

Intel iGPU (mini nodes)

The Intel DRA driver runs as a DaemonSet:

kubectl get pods -n intel-gpu-resource-driver -o wide

Check that ResourceSlices are published:

kubectl get resourceslice -o wide

$ kubectl get resourceslice -o wide
NAME                         NODE     DRIVER          POOL     AGE
mini-1-gpu.intel.com-wnt98   mini-1   gpu.intel.com   mini-1   28d
mini-2-gpu.intel.com-ssz5r   mini-2   gpu.intel.com   mini-2   28d
mini-3-gpu.intel.com-ch2jg   mini-3   gpu.intel.com   mini-3   28d

To see active ResourceClaims (pods currently using an Intel GPU):

kubectl get resourceclaim -A

Routine Operations

Check GPU Utilization

# GPU utilization, memory, running processes
kubectl exec -n ollama $(kubectl get pod -n ollama \
  -o jsonpath='{.items[0].metadata.name}') -- nvidia-smi

# Ollama model status
kubectl exec -n ollama $(kubectl get pod -n ollama \
  -o jsonpath='{.items[0].metadata.name}') -- ollama ps

The ollama ps output shows model name, size, processor allocation (look for 100% GPU), and context window.

Check Which Pods Use the GPU

# NVIDIA — pods requesting nvidia.com/gpu
kubectl get pods -A -o json | jq -r '
  .items[] | select(.spec.containers[].resources.limits."nvidia.com/gpu" != null)
  | "\(.metadata.namespace)/\(.metadata.name)"'

# Intel DRA — pods with ResourceClaims
kubectl get pods -A -o json | jq -r '
  .items[] | select(.spec.resourceClaims != null)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Pull Models via kubectl exec

PostStart lifecycle hooks fail on Talos with NVIDIA due to the nvidia-container-cli exec hook. Models must be pulled manually:

kubectl exec -n ollama $(kubectl get pod -n ollama \
  -o jsonpath='{.items[0].metadata.name}') -- ollama pull qwen3.5:9b

kubectl exec -n ollama $(kubectl get pod -n ollama \
  -o jsonpath='{.items[0].metadata.name}') -- ollama pull deepseek-coder:6.7b

Models persist on the Longhorn PVCPersistentVolumeClaimA Kubernetes request for durable storage. The pod names a claim and the storage layer — Longhorn on Frank — binds real disk behind it, so the data outlives the pod. — one-time operation unless the PVC is lost.

Manage GPU Memory

# List loaded models
kubectl exec -n ollama $(kubectl get pod -n ollama \
  -o jsonpath='{.items[0].metadata.name}') -- ollama ps

# Unload — delete the pod; Deployment recreates it
kubectl delete pod -n ollama $(kubectl get pod -n ollama \
  -o jsonpath='{.items[0].metadata.name}')

Models on the PVC remain available and load back into VRAMVideo RAMMemory on the GPU itself. Both the model weights and the KV cache live here, which makes it the binding constraint on what a single card can serve. on the next request.

Runbook

GPU Not Allocating

If a pod requesting nvidia.com/gpu stays Pending:

1. Check device plugin registration:

   kubectl get node gpu-1 -o jsonpath='{.status.allocatable.nvidia\.com/gpu}'

   Should return 1. If empty or 0, the device plugin is not running.

2. Check GPU Operator pods:

   kubectl get pods -n gpu-operator -o wide

   All should be Running. Look for Init:0/1 or CrashLoopBackOff.

3. Check validation markers:

   kubectl exec -n gpu-operator $(kubectl get pod -n gpu-operator \
     -l app=nvidia-validation-markers -o jsonpath='{.items[0].metadata.name}') \
     -- ls -la /run/nvidia/validations/

   Should show driver-ready and toolkit-ready files. If missing, the node may have rebooted (files are on tmpfs). The DaemonSet recreates them within 30 seconds.

Recovery: containerd config corruption

If GPU pods are stuck at ContainerCreating with PodReadyToStartContainers: False:

talosctl -n 192.168.55.31 read /etc/cri/conf.d/20-customization.part

The file should contain:

[plugins."io.containerd.cri.v1.runtime"]
  cdi_spec_dirs = ["/var/cdi/static", "/var/cdi/dynamic"]
  [plugins."io.containerd.cri.v1.runtime".containerd]
    default_runtime_name = "nvidia"
  [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia]
    base_runtime_spec = "/etc/cri/conf.d/base-spec.json"

If base_runtime_spec is missing, kubelet cannot track the GPU container lifecycle. See the Talos validation fix for the full story.

Talos Reboot Loops from Conflicting Patches

If a node enters a ~35-minute reboot loop after applying a config patch, the cause is two patches creating the same file at /etc/cri/conf.d/20-customization.part:

resource EtcFileSpecs.files.talos.dev(files/cri/conf.d/20-customization.part@undefined) already exists

Recovery:

omnictl delete configpatch <cluster-wide-patch-id>
kubectl get node <node-name> -w

Each node must have its own machine-specific patch. Delete the cluster-wide patch before applying machine-specific ones.

Stale GPU Allocations from Force-Delete

Force-deleting a GPU pod (kubectl delete pod --force --grace-period=0) leaves stale containers holding the GPU allocation. The device plugin still sees the GPU as allocated, and new GPU pods stay Pending with Insufficient nvidia.com/gpu.

Recovery:

# Check if GPU is stuck allocated
kubectl describe node gpu-1 | grep -A 5 "Allocated resources"

# Clean reboot clears it
talosctl -n 192.168.55.31 reboot

A clean reboot is the only reliable way to clear stale GPU allocations. Budget about 90 seconds for Talos to come back Ready. Avoid force-deleting GPU pods — scale the workload to 0 instead.

Missteps

Quick Reference

Explanation

This post covers both GPU paths on Frank because they fail differently. The NVIDIA stack has Talos-specific containerd quirks (missing base_runtime_spec, postStart hook conflicts, stale allocations from force-delete). The Intel DRA stack is simpler — it Just Works — but requires understanding the ResourceSlice/ResourceClaim model. The building posts cover why each stack was chosen and how it was deployed; this post covers what to do when they break.

References

• NVIDIA GPU Operator docs — Official GPU Operator documentation
• Intel Resource Drivers for Kubernetes — DRA-based driver for Intel GPUs
• Talos NVIDIA GPU guide — Extensions and kernel modules for NVIDIA on Talos
• Kubernetes DRA — Dynamic Resource Allocation documentation
• GPU Compute — NVIDIA and Intel — Building post: deploying both GPU stacks
• GPU Containers on Talos — The Validation Fix — Building post: debugging containerd and validation issues