# Running local AI Source: https://blog.derio.net/frank/topics/running-local-ai/ Published: 2026-09-05 Updated: 2026-09-05 Put mixed GPUs to work, serve models, and manage inference workloads. ## Building Frank Follow the implementation, from bare metal to working services. 6 Mar 2026 ### [GPU Compute — NVIDIA and Intel](https://blog.derio.net/frank/docs/building/04-gpu-compute/) Adding GPU compute to the cluster — the NVIDIA RTX 5070 saga, Intel Arc iGPU via DRA, and patching charts for bleeding-edge Kubernetes. 9 Mar 2026 ### [Local Inference — Ollama, LiteLLM, and OpenRouter](https://blog.derio.net/frank/docs/building/10-local-inference/) A unified OpenAI-compatible gateway fronting a local RTX 5070 Ti running Ollama, built to route to OpenRouter's free tier as well — until that tier was dropped and the gateway made the change invisible to consumers. 10 Mar 2026 ### [GPU Containers on Talos — The Validation Fix](https://blog.derio.net/frank/docs/building/12-gpu-talos-fix/) Getting NVIDIA GPU containers to actually run on Talos Linux — validation markers, machine-specific patches, nvidia default runtime, and the postStart hook trap. 14 Mar 2026 ### [Media Generation — ComfyUI and GPU Time-Sharing](https://blog.derio.net/frank/docs/building/16-media-generation/) Running ComfyUI for video, image, and audio generation on the same GPU as Ollama — with a custom GPU Switcher dashboard to manage time-sharing. 2 Aug 2026 ### [Retrieval on the Idle iGPUs](https://blog.derio.net/frank/docs/building/37-igpu-retrieval-tier/) Embeddings and reranking on my control plane's idle Arc iGPUs via Kubernetes DRA, plus the CPU control arm that decided whether the accelerator earned it ## Operating on Frank Check health, investigate failures, and recover the system. 13 Mar 2026 ### [Operating on GPU Compute](https://blog.derio.net/frank/docs/operating/04-gpu-compute/) Day-to-day commands for managing NVIDIA and Intel GPUs, checking utilization, and debugging GPU container issues on Talos. 13 Mar 2026 ### [Operating on Local Inference](https://blog.derio.net/frank/docs/operating/07-inference/) Day-to-day commands for managing local LLM inference, checking model status, routing through LiteLLM, and debugging GPU memory issues — including the misleading cgroup OOM and the time-share probe pattern. 20 Mar 2026 ### [Operating on Media Generation](https://blog.derio.net/frank/docs/operating/10-media-generation/) Starting ComfyUI workflows, switching GPU profiles, transferring files, and recovering from common runtime failures. ## The Frank Papers Compare approaches, examine the evidence, and understand the trade-offs. 20 May 2026 ### [GPU Scheduling for Mixed Workloads](https://blog.derio.net/frank/docs/papers/05-gpu-scheduling/) GPU scheduling on Kubernetes is a four-job problem — driver lifecycle, device discovery, allocation, partitioning — and the six contenders in 2026 (NVIDIA GPU Operator, Intel… 19 May 2026 ### [Self-Hosted Inference & the LLM Gateway Pattern](https://blog.derio.net/frank/docs/papers/10-self-hosted-inference/) Inference is a build-versus-buy question dressed up in GPU language. The engine tier (Ollama, vLLM, llama.cpp, TGI) decides whether the model runs on your metal; the gateway…