Skip to content

Running local AI

Put mixed GPUs to work, serve models, and manage inference workloads.

Building Frank

Follow the implementation, from bare metal to working services.

GPU Compute — NVIDIA and Intel

Adding GPU compute to the cluster — the NVIDIA RTX 5070 saga, Intel Arc iGPU via DRA, and patching charts for bleeding-edge Kubernetes.

Local Inference — Ollama, LiteLLM, and OpenRouter

A unified OpenAI-compatible gateway fronting a local RTX 5070 Ti running Ollama, built to route to OpenRouter's free tier as well — until that tier was dropped and the gateway made the change invisible to consumers.

Retrieval on the Idle iGPUs

Embeddings and reranking on my control plane's idle Arc iGPUs via Kubernetes DRA, plus the CPU control arm that decided whether the accelerator earned it

Operating on Frank

Check health, investigate failures, and recover the system.

Operating on GPU Compute

Day-to-day commands for managing NVIDIA and Intel GPUs, checking utilization, and debugging GPU container issues on Talos.

Operating on Local Inference

Day-to-day commands for managing local LLM inference, checking model status, routing through LiteLLM, and debugging GPU memory issues — including the misleading cgroup OOM and the time-share probe pattern.

Operating on Media Generation

Starting ComfyUI workflows, switching GPU profiles, transferring files, and recovering from common runtime failures.

The Frank Papers

Compare approaches, examine the evidence, and understand the trade-offs.

GPU Scheduling for Mixed Workloads

GPU scheduling on Kubernetes is a four-job problem — driver lifecycle, device discovery, allocation, partitioning — and the six contenders in 2026 (NVIDIA GPU Operator, Intel…