Operating on Local Inference

Last updated 2026-08-03 ·e008996

This is the operational companion to Local Inference. That post covers the architecture and deployment. This one is what you type when a model won’t load, LiteLLM returns 404, or the GPU shows plenty of free VRAMVideo RAMMemory on the GPU itself. Both the model weights and the KV cache live here, which makes it the binding constraint on what a single card can serve. but Ollama insists there’s not enough memory — and the probes that keep the ops board honest about a time-shared GPU.

Before any of the commands below, source the environment:

source .env          # sets KUBECONFIG, TALOSCONFIG
source .env_devops   # sets OMNICONFIG + service accounts

What Healthy Looks Like

The inference stack is healthy when Ollama is running on gpu-1 with at least one model loaded, LiteLLM at 192.168.55.206:4000 is responding, and requests route correctly to either the local GPU or a paid frontier key.

The end-to-end probe. gpu-1 time-shares its single GPU between Ollama (inference) and ComfyUI (media generation), so inference is down by design whenever the GPU is handed to ComfyUI — that is not an outage. The ops board reads the real state from a blackbox probe that runs an actual chat completion through LiteLLM: probe_success{layer="11"} is 1 only when a completion truly succeeds. A 0 with ComfyUI holding the GPU is expected-quiet-degraded; a 0 with both GPU time-share probes at 0 is the only condition that pages (gpu-node-both-down). Check GPU ownership with kubectl -n ollama get deploy ollama — 0/0 replicas means it yielded the GPU.

kube_pod_status_ready is not a valid inference health check — it is blind to Ollama scaled to 0.

    graph LR
  CLIENT["Client / Agent"] -->|"chat completion"| LLM["LiteLLM Gateway<br/>192.168.55.206:4000"]
  LLM -->|"local models"| OLL["Ollama<br/>gpu-1"]
  LLM -->|"frontier models"| OR["OpenRouter"]
  LLM -->|"frontier models"| AN["Anthropic"]
  OLL --> GPU["RTX 5070 Ti<br/>16 GB VRAM"]
  subgraph TS["GPU Time-Sharing"]
    GS["GPU Switcher"] -->|"inference"| OLL
    GS -->|"media gen"| CF["ComfyUI"]
  end
  

Verify

Ollama Status

# Check the Ollama pod is running
kubectl get pods -n ollama

# List loaded models
kubectl exec -n ollama deploy/ollama -- ollama list

# Check which model is currently in memory
kubectl exec -n ollama deploy/ollama -- ollama ps

# Check GPU memory usage
kubectl exec -n ollama deploy/ollama -- nvidia-smi

LiteLLM Gateway

# Health check
curl -s http://192.168.55.206:4000/health | jq

# List available models (both local and cloud)
curl -s http://192.168.55.206:4000/v1/models | jq '.data[].id'

# Check LiteLLM pod logs
kubectl logs -n litellm deploy/litellm --tail=50

$ curl -s http://192.168.55.206:4000/health/liveliness
"I'm alive!"
$ kubectl -n litellm get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP              NODE
litellm-84d78cd556-rglgl   1/1     Running   0          25d   10.244.8.237    mini-3
litellm-postgresql-0       1/1     Running   0          28d   10.244.12.161   mini-1

Steps

Pull or Remove Models

# Pull a new model
kubectl exec -n ollama deploy/ollama -- ollama pull qwen3:14b

# Remove a model to free disk space
kubectl exec -n ollama deploy/ollama -- ollama rm qwen2.5-coder:14b-instruct-q6_K

Never use postStart hooks for model pulls on Talos — the nvidia-container-runtime exec hooks fail there (commit 7c88dcc4). Always pull via kubectl exec after the pod is running.

Test Inference

# Quick test via LiteLLM
curl -s http://192.168.55.206:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-small-24b",
    "messages": [{"role": "user", "content": "Hello, one sentence reply."}],
    "max_tokens": 50
  }' | jq '.choices[0].message.content'

# Direct Ollama test (bypassing LiteLLM)
kubectl exec -n ollama deploy/ollama -- curl -s http://localhost:11434/api/generate \
  -d '{"model": "mistral-small3.2:24b", "prompt": "Hello", "stream": false}' | jq '.response'

Check LiteLLM Routing

# See which models route where
kubectl get configmap -n litellm litellm-config -o yaml | grep -A 5 'model_name'

OpenRouter Free Models (retired)

The cluster no longer routes to OpenRouter free models (commit 46f19ca2). The policy is local Ollama or a paid frontier key only. If your config still lists openrouter/*:free models, remove them.

Retrieval Endpoints (embeddings + rerank)

The retrieval pair runs separately from Ollama, on a mini’s Arc iGPU, and is deliberately not behind LiteLLM. Callers use the service directly.

# Both servables must read AVAILABLE. This is the diagnostic of record --
# /v2/health/ready is SERVER-level and answers 200 with a dead model.
kubectl -n retrieval exec deploy/ovms-retrieval -c ovms -- \
  curl -s http://127.0.0.1:8000/v1/config | grep -c AVAILABLE     # expect 2

# Confirm the iGPU is enumerated, not just present
kubectl -n retrieval logs deploy/ovms-retrieval -c ovms | grep "Available devices"
# -> Available devices for Open VINO: CPU, GPU

# Who holds the iGPU right now
kubectl get resourceclaims -A

# Rerank: scores must be well separated. Clustered values around 1e-9..1e-12
# mean the wrong model class is being served, and nothing will error.
kubectl -n retrieval exec deploy/ovms-retrieval -c ovms -- curl -s \
  -X POST http://127.0.0.1:8000/v3/rerank -H 'Content-Type: application/json' \
  -d '{"model":"bge-reranker-v2-m3","query":"pruning fruit trees",
       "documents":["Prune apple trees while dormant.","Lima is the capital of Peru."]}'

Measured on the iGPU: rerank of 20 candidates runs ~77 ms p50, ~87 ms p95. Embeddings are 1024-dimensional. A CPU-only arm of the same models on the same node runs ~1613 ms p50, so a sudden jump into the 1.5-2 s range is the signature of the GPU repository not being the one that loaded.

Recover

Ollama Not Responding

# Check pod status and events
kubectl describe pod -n ollama -l app.kubernetes.io/name=ollama

# Check if GPU is allocated
kubectl describe node gpu-1 | grep -A 5 "nvidia.com/gpu"

# Check Ollama logs
kubectl logs -n ollama deploy/ollama --tail=100

Out of Memory: Which Kind?

Ollama emits two error patterns that both look like “out of memory” but have different root causes. Mis-identify which one you’re hitting and you’ll waste time trying smaller quants when the issue is something else.

Pattern A — VRAM exhaustion (real GPU OOMOut Of MemoryWhat the kernel does when memory runs out — it kills a process by score, not by fault. A container hitting its cgroup limit is killed the same way, which is why the victim is often not the culprit.). The model genuinely doesn’t fit in the RTX 5070 Ti’s 16 GB GDDR7Graphics Double Data Rate 7The memory standard on current NVIDIA consumer cards. It sets how much model weight the card can hold at once — the 16GB figure that decides which quantisation a model has to run at., or you’re loading two models at once:

kubectl exec -n ollama deploy/ollama -- nvidia-smi --query-gpu=memory.used,memory.free --format=csv

If memory.free is under 1 GB, stop the current model and switch:

kubectl exec -n ollama deploy/ollama -- ollama stop mistral-small3.2:24b
kubectl exec -n ollama deploy/ollama -- ollama pull qwen2.5-coder:14b-instruct-q4_K_M

Pattern B — container cgroup RAM exhaustion (the misleading one). The error reads model requires more system memory (X GiB) than is available (Y MiB). “System memory” here means container RAM, not VRAM — nvidia-smi will show plenty of VRAM free. With OLLAMA_KEEP_ALIVE=24h, page cache from previously-loaded model files pins the container near its resources.limits.memory ceiling, leaving no room for new-model load buffers.

kubectl exec -n ollama deploy/ollama -- sh -c 'cat /sys/fs/cgroup/memory.current; echo; cat /sys/fs/cgroup/memory.max'

If memory.current is within 1–2 GiB of memory.max, that’s the constraint. We bumped the limit to 64 GiB to fit 24B+ models (commit 8a135bcc). Reducing num_ctx via a derived Modelfile will not help here — the bottleneck is at-load buffers, not the KVKey-Value cacheThe attention keys and values a transformer keeps for tokens it has already produced, so it need not recompute them each step. It grows with context length and competes with the model weights for VRAM — which is what makes context window a memory decision rather than a free knob. cache.

Reconciling LiteLLM Aliases vs. Ollama Tags

Two name spaces are in play. LiteLLM aliases (mistral-small-24b, gemma-12b) live in apps/litellm/values.yaml under model_list[].model_name — consumers send these. Ollama tags (mistral-small3.2:24b) live under model_list[].litellm_params.model — these are what Ollama pulls and runs.

If a request returns “model not found”, check both:

# What aliases does LiteLLM advertise?
curl -s http://192.168.55.206:4000/v1/models \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '.data[].id'

# What Ollama tags actually exist on disk?
kubectl exec -n ollama deploy/ollama -- ollama list

A second gotcha: we switched the local routing prefix from ollama/ to ollama_chat/ (commit 8277c154) to get native stream-safe tool calling. If existing consumers are still sending ollama/model they’ll get 404s — update them to ollama_chat/model.

LiteLLM Routing Errors

# Check LiteLLM logs for routing failures
kubectl logs -n litellm deploy/litellm --tail=100 | grep -i error

# Verify Ollama is reachable from LiteLLM
kubectl exec -n litellm deploy/litellm -- curl -s http://ollama.ollama.svc:11434/api/tags | jq '.models[].name'

Model Loading Very Slow

Large models take 30-60 seconds to load into VRAM. If nvidia-smi shows no memory being allocated, the model is falling back to CPU — extremely slow. Check Ollama logs for “no compatible CUDACompute Unified Device ArchitectureNVIDIA's GPU programming platform. Practically, the layer every local inference engine sits on, and the reason a driver/toolkit version mismatch stops a model loading. device” or similar.

Retrieval Model Not Serving

Missteps

Quick Reference

References

• Ollama API Reference
• LiteLLM Documentation
• Building Post — Local Inference
• Building Post — Retrieval on the Idle iGPUs