Operating on Local Inference

Last updated 2026-08-03 ·e008996

This is the operational companion to Local Inference. That post covers the architecture and deployment. This one is what you type when a model won’t load, LiteLLM returns 404, or the GPU shows plenty of free VRAMVideo RAMMemory on the GPU itself. Both the model weights and the KV cache live here, which makes it the binding constraint on what a single card can serve. but Ollama insists there’s not enough memory — and the probes that keep the ops board honest about a time-shared GPU.

Before any of the commands below, source the environment:

source .env          # sets KUBECONFIG, TALOSCONFIG
source .env_devops   # sets OMNICONFIG + service accounts

What Healthy Looks Like

The inference stack is healthy when Ollama is running on gpu-1 with at least one model loaded, LiteLLM at 192.168.55.206:4000 is responding, and requests route correctly to either the local GPU or a paid frontier key.

The end-to-end probe. gpu-1 time-shares its single GPU between Ollama (inference) and ComfyUI (media generation), so inference is down by design whenever the GPU is handed to ComfyUI — that is not an outage. The ops board reads the real state from a blackbox probe that runs an actual chat completion through LiteLLM: probe_success{layer="11"} is 1 only when a completion truly succeeds. A 0 with ComfyUI holding the GPU is expected-quiet-degraded; a 0 with both GPU time-share probes at 0 is the only condition that pages (gpu-node-both-down). Check GPU ownership with kubectl -n ollama get deploy ollama — 0/0 replicas means it yielded the GPU.

kube_pod_status_ready is not a valid inference health check — it is blind to Ollama scaled to 0.

    graph LR
  CLIENT["Client / Agent"] -->|"chat completion"| LLM["LiteLLM Gateway<br/>192.168.55.206:4000"]
  LLM -->|"local models"| OLL["Ollama<br/>gpu-1"]
  LLM -->|"frontier models"| OR["OpenRouter"]
  LLM -->|"frontier models"| AN["Anthropic"]
  OLL --> GPU["RTX 5070 Ti<br/>16 GB VRAM"]
  subgraph TS["GPU Time-Sharing"]
    GS["GPU Switcher"] -->|"inference"| OLL
    GS -->|"media gen"| CF["ComfyUI"]
  end
  

Verify

Ollama Status

# Check the Ollama pod is running
kubectl get pods -n ollama

# List loaded models
kubectl exec -n ollama deploy/ollama -- ollama list

# Check which model is currently in memory
kubectl exec -n ollama deploy/ollama -- ollama ps

# Check GPU memory usage
kubectl exec -n ollama deploy/ollama -- nvidia-smi

LiteLLM Gateway

# Health check
curl -s http://192.168.55.206:4000/health | jq

# List available models (both local and cloud)
curl -s http://192.168.55.206:4000/v1/models | jq '.data[].id'

# Check LiteLLM pod logs
kubectl logs -n litellm deploy/litellm --tail=50

$ curl -s http://192.168.55.206:4000/health/liveliness
"I'm alive!"
$ kubectl -n litellm get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP              NODE
litellm-84d78cd556-rglgl   1/1     Running   0          25d   10.244.8.237    mini-3
litellm-postgresql-0       1/1     Running   0          28d   10.244.12.161   mini-1

Steps

Pull or Remove Models

# Pull a new model
kubectl exec -n ollama deploy/ollama -- ollama pull qwen3:14b

# Remove a model to free disk space
kubectl exec -n ollama deploy/ollama -- ollama rm qwen2.5-coder:14b-instruct-q6_K

Never use postStart hooks for model pulls on Talos — the nvidia-container-runtime exec hooks fail there (commit 7c88dcc4). Always pull via kubectl exec after the pod is running.

Test Inference

# Quick test via LiteLLM
curl -s http://192.168.55.206:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-small-24b",
    "messages": [{"role": "user", "content": "Hello, one sentence reply."}],
    "max_tokens": 50
  }' | jq '.choices[0].message.content'

# Direct Ollama test (bypassing LiteLLM)
kubectl exec -n ollama deploy/ollama -- curl -s http://localhost:11434/api/generate \
  -d '{"model": "mistral-small3.2:24b", "prompt": "Hello", "stream": false}' | jq '.response'

Check LiteLLM Routing

# See which models route where
kubectl get configmap -n litellm litellm-config -o yaml | grep -A 5 'model_name'

OpenRouter Free Models (retired)

The cluster no longer routes to OpenRouter free models (commit 46f19ca2). The policy is local Ollama or a paid frontier key only. If your config still lists openrouter/*:free models, remove them.

Retrieval Endpoints (embeddings + rerank)

The retrieval pair runs separately from Ollama, on a mini’s Arc iGPU, and is deliberately not behind LiteLLM. Callers use the service directly.

# Both servables must read AVAILABLE. This is the diagnostic of record --
# /v2/health/ready is SERVER-level and answers 200 with a dead model.
kubectl -n retrieval exec deploy/ovms-retrieval -c ovms -- \
  curl -s http://127.0.0.1:8000/v1/config | grep -c AVAILABLE     # expect 2

# Confirm the iGPU is enumerated, not just present
kubectl -n retrieval logs deploy/ovms-retrieval -c ovms | grep "Available devices"
# -> Available devices for Open VINO: CPU, GPU

# Who holds the iGPU right now
kubectl get resourceclaims -A

# Rerank: scores must be well separated. Clustered values around 1e-9..1e-12
# mean the wrong model class is being served, and nothing will error.
kubectl -n retrieval exec deploy/ovms-retrieval -c ovms -- curl -s \
  -X POST http://127.0.0.1:8000/v3/rerank -H 'Content-Type: application/json' \
  -d '{"model":"bge-reranker-v2-m3","query":"pruning fruit trees",
       "documents":["Prune apple trees while dormant.","Lima is the capital of Peru."]}'

Measured on the iGPU: rerank of 20 candidates runs ~77 ms p50, ~87 ms p95. Embeddings are 1024-dimensional. A CPU-only arm of the same models on the same node runs ~1613 ms p50, so a sudden jump into the 1.5-2 s range is the signature of the GPU repository not being the one that loaded.

The rerank batch is bounded in the graph, not by anything you can change from the Deployment. Confirm the bounds are live:

# Both fields must be present. Absent = a model image without the guard, and
# an oversized call takes the whole server down instead of being refused.
kubectl -n retrieval exec deploy/ovms-retrieval -c ovms -- \
  grep -E 'max_allowed_chunks|max_position_embeddings' \
  /models/bge-reranker-v2-m3/graph.pbtxt
# -> max_allowed_chunks: 64
# -> max_position_embeddings: 640

Past the cap the caller gets HTTP 500, not a 4xx. The guard raises a C++ runtime error which the server maps to InternalError, so the response body names the limit but the status code says “server”. That is refusal working, not failure: the /v1/config check above still reports both servables AVAILABLE and the restart counter does not move. The pair of signals is what tells the two apart — a refusal leaves the server up, an OOM-kill does not:

# Refused (good): both servables still AVAILABLE, restarts unchanged.
kubectl -n retrieval get pod -l app.kubernetes.io/name=ovms-retrieval \
  -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[*].restartCount

Documents longer than 640 tokens minus the query are split into chunks, scored per chunk, and those chunks count against the same 64 — so a caller sending long passages hits the cap at fewer documents than one sending short ones. That is the intended behaviour, and it is also why relevance scores for long passages differ from the whole-document scores this endpoint returned before September 2026.

Recover

Ollama Not Responding

# Check pod status and events
kubectl describe pod -n ollama -l app.kubernetes.io/name=ollama

# Check if GPU is allocated
kubectl describe node gpu-1 | grep -A 5 "nvidia.com/gpu"

# Check Ollama logs
kubectl logs -n ollama deploy/ollama --tail=100

Out of Memory: Which Kind?

Ollama emits two error patterns that both look like “out of memory” but have different root causes. Mis-identify which one you’re hitting and you’ll waste time trying smaller quants when the issue is something else.

Pattern A — VRAM exhaustion (real GPU OOMOut Of MemoryWhat the kernel does when memory runs out — it kills a process by score, not by fault. A container hitting its cgroup limit is killed the same way, which is why the victim is often not the culprit.). The model genuinely doesn’t fit in the RTX 5070 Ti’s 16 GB GDDR7Graphics Double Data Rate 7The memory standard on current NVIDIA consumer cards. It sets how much model weight the card can hold at once — the 16GB figure that decides which quantisation a model has to run at., or you’re loading two models at once:

kubectl exec -n ollama deploy/ollama -- nvidia-smi --query-gpu=memory.used,memory.free --format=csv

If memory.free is under 1 GB, stop the current model and switch:

kubectl exec -n ollama deploy/ollama -- ollama stop mistral-small3.2:24b
kubectl exec -n ollama deploy/ollama -- ollama pull qwen2.5-coder:14b-instruct-q4_K_M

Pattern B — container cgroup RAM exhaustion (the misleading one). The error reads model requires more system memory (X GiB) than is available (Y MiB). “System memory” here means container RAM, not VRAM — nvidia-smi will show plenty of VRAM free. With OLLAMA_KEEP_ALIVE=24h, page cache from previously-loaded model files pins the container near its resources.limits.memory ceiling, leaving no room for new-model load buffers.

kubectl exec -n ollama deploy/ollama -- sh -c 'cat /sys/fs/cgroup/memory.current; echo; cat /sys/fs/cgroup/memory.max'

If memory.current is within 1–2 GiB of memory.max, that’s the constraint. We bumped the limit to 64 GiB to fit 24B+ models (commit 8a135bcc). Reducing num_ctx via a derived Modelfile will not help here — the bottleneck is at-load buffers, not the KVKey-Value cacheThe attention keys and values a transformer keeps for tokens it has already produced, so it need not recompute them each step. It grows with context length and competes with the model weights for VRAM — which is what makes context window a memory decision rather than a free knob. cache.

Out of Memory, A Third Kind: The Retrieval Pod’s Floor Ratchets

Different app from the two patterns above, same word, third root cause — and this is the only OOM on this page that a restart genuinely fixes.

ovms-retrieval never gives memory back, because what it is holding is not process memory at all — it is pinned iGPU buffers. The iGPU has no VRAM, so every GPU allocation is a shmem-backed host page, unevictable in the container’s cgroup. With 5.11 GiB held, memory.stat reads shmem 5.11 GiB and unevictable 5.11 GiB — the same number — with inactive_file and active_file at zero and swap disabled. Nothing is reclaimable, so the kernel kills rather than frees. limits.memory here is the GPU’s memory budget.

Baseline after load is 1.71 GiB (the weights, on the GPU). The pool ratchets up to the high-water mark of the largest call the process has ever served, so the same batch succeeds on a fresh pod and gets the pod OOM-killed once the floor has risen underneath it. “It worked this morning” is the symptom, not a red herring.

Both servables run in one process, so a rerank kill takes embeddings down with it for the ten seconds or so the pod takes to come back.

# The instrument. A 20s metrics scrape steps straight over these transients --
# they last 0.1-2.5s -- so container_memory_working_set_bytes will show you a
# comfortable idle figure for a call that reached the limit.
kubectl -n retrieval exec deploy/ovms-retrieval -c ovms -- sh -c \
  'cat /sys/fs/cgroup/memory.peak /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.max'

# How often has it been killed? A counter climbing with no deploy is the ratchet.
kubectl -n retrieval get pod -l app.kubernetes.io/name=ovms-retrieval \
  -o jsonpath='{.items[*].status.containerStatuses[*].restartCount}'

Restarting the pod is a legitimate mitigation, and always was. It drops the floor back to idle and buys back the full headroom, until the next large call raises it again:

kubectl -n retrieval rollout restart deploy/ovms-retrieval

What stops it recurring is the graph’s two bounds (max_allowed_chunks: 64, max_position_embeddings: 640) plus the 16Gi ceiling — the guard’s own worst case is 8.49 GiB as an ascending sequence, and the ceiling was raised so it cannot itself OOM. If you are tempted to raise the ceiling again, tighten max_position_embeddings instead: memory is near-quadratic in document length and only linear in document count, so length is the lever.

Measure it on a freshly restarted container, one call at a time. memory.peak is monotonic, so an ascending sweep reports each size plus every size before it, and a kill part-way up is floor-plus-call rather than call.

Reconciling LiteLLM Aliases vs. Ollama Tags

Two name spaces are in play. LiteLLM aliases (mistral-small-24b, gemma-12b) live in apps/litellm/values.yaml under model_list[].model_name — consumers send these. Ollama tags (mistral-small3.2:24b) live under model_list[].litellm_params.model — these are what Ollama pulls and runs.

If a request returns “model not found”, check both:

# What aliases does LiteLLM advertise?
curl -s http://192.168.55.206:4000/v1/models \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '.data[].id'

# What Ollama tags actually exist on disk?
kubectl exec -n ollama deploy/ollama -- ollama list

A second gotcha: we switched the local routing prefix from ollama/ to ollama_chat/ (commit 8277c154) to get native stream-safe tool calling. If existing consumers are still sending ollama/model they’ll get 404s — update them to ollama_chat/model.

LiteLLM Routing Errors

# Check LiteLLM logs for routing failures
kubectl logs -n litellm deploy/litellm --tail=100 | grep -i error

# Verify Ollama is reachable from LiteLLM
kubectl exec -n litellm deploy/litellm -- curl -s http://ollama.ollama.svc:11434/api/tags | jq '.models[].name'

Model Loading Very Slow

Large models take 30-60 seconds to load into VRAM. If nvidia-smi shows no memory being allocated, the model is falling back to CPU — extremely slow. Check Ollama logs for “no compatible CUDACompute Unified Device ArchitectureNVIDIA's GPU programming platform. Practically, the layer every local inference engine sits on, and the reason a driver/toolkit version mismatch stops a model loading. device” or similar.

Retrieval Model Not Serving

Missteps

Quick Reference

References

• Ollama API Reference
• LiteLLM Documentation
• Building Post — Local Inference
• Building Post — Retrieval on the Idle iGPUs