# Debugging Kubernetes Source: https://blog.derio.net/frank/topics/debugging-kubernetes/ Published: 2026-09-05 Updated: 2026-09-05 Understand what a healthy status actually proves, and find the checks that matter. ## Building Frank Follow the implementation, from bare metal to working services. 6 Mar 2026 ### [Building the Foundation — Talos, Nodes, and Cilium](https://blog.derio.net/frank/docs/building/02-foundation/) Bootstrapping a Talos Linux cluster with Omni, configuring node labels and zones, and replacing Flannel with Cilium's eBPF networking. 6 Mar 2026 ### [GitOps Everything with ArgoCD](https://blog.derio.net/frank/docs/building/05-gitops/) Migrating from Flux to ArgoCD with an App-of-Apps pattern — adopting existing workloads without downtime. 8 Mar 2026 ### [Observability — Metrics and Logs with VictoriaMetrics](https://blog.derio.net/frank/docs/building/07-observability/) Deploying a resource-efficient observability stack with VictoriaMetrics, VictoriaLogs, and Grafana — and the four gotchas that made it interesting, including a scrape that was configured and inert for 148 days. 4 Apr 2026 ### [Health Monitoring — Feature Probes, Heartbeats, and Telegram Alerts](https://blog.derio.net/frank/docs/building/22-health-monitoring/) Adding feature-level health monitoring to the cluster — Blackbox probes for endpoints, Pushgateway for cron heartbeats, and Grafana alerting that fires to Telegram when things go silent. 4 Apr 2026 ### [Health Bridge — Closing the Loop from Grafana Alerts to GitHub Issues](https://blog.derio.net/frank/docs/building/23-health-bridge/) A lightweight Go service that receives Grafana webhook alerts and automatically updates GitHub Project lifecycle states — turning monitoring signals into actionable project board updates. 24 May 2026 ### [Edge Observability — Watching Frank's Edge Without Watching Frank's Edge Burn](https://blog.derio.net/frank/docs/building/31-edge-observability/) Collectors on Hop, backend on Frank, AI alert enrichment that knows when 12x baseline is a scraper and when it is Hacker News. 25 Jul 2026 ### [The Metrics API — Teaching Frank to Read His Own Vitals](https://blog.derio.net/frank/docs/building/36-metrics-api/) kubectl top was dead for months because nothing served metrics.k8s.io. The fix was one Talos flag and a decision I almost got wrong: build the fancy thing, or the right thing. ## Operating on Frank Check health, investigate failures, and recover the system. 13 Mar 2026 ### [Operating on Cluster & Nodes](https://blog.derio.net/frank/docs/operating/01-cluster-nodes/) Day-to-day commands for checking cluster health, managing Talos nodes, and debugging Cilium networking on Frank — including the real failure patterns the gotcha docs cover. 13 Mar 2026 ### [Operating on GitOps](https://blog.derio.net/frank/docs/operating/03-gitops/) Day-to-day commands for managing ArgoCD applications, syncing, debugging drift, handling degraded apps, and navigating the gotchas that have bitten Frank. 13 Mar 2026 ### [Operating on Observability](https://blog.derio.net/frank/docs/operating/05-observability/) Day-to-day commands for querying metrics and logs, checking the etcd scrape, managing Grafana dashboards, debugging alert delivery failures, and fixing the observability pipeline on Frank. 4 Apr 2026 ### [Operating on Health Monitoring](https://blog.derio.net/frank/docs/operating/15-health-monitoring/) Day-to-day commands for managing feature health probes, heartbeat metrics, file-provisioned Grafana alerts, and Telegram notifications — including the silent delivery failures we've fixed. 4 Apr 2026 ### [Operating on Health Bridge](https://blog.derio.net/frank/docs/operating/16-health-bridge/) Day-to-day commands for managing the health-bridge service — testing webhooks, managing alert labels, recovering stranded board tiles, and auto-closing healed bug issues. 16 Apr 2026 ### [Operating on ArgoCD Drift](https://blog.derio.net/frank/docs/operating/23-argocd-drift-detective/) How 20 of 52 ArgoCD apps were permanently OutOfSync, why it was seven different bugs, and how fixing the noise unmasked a 21-day crashloop. 24 May 2026 ### [Operating Edge Observability](https://blog.derio.net/frank/docs/operating/26-edge-observability/) Querying the blog log stream, banning a scraper before lunch, tuning Falco out of kube-system noise, and triggering a digest by hand. 25 Jul 2026 ### [Operating the Metrics API — kubectl top, HPAs, and the Silent-Empty Failure](https://blog.derio.net/frank/docs/operating/29-metrics-api/) How to check the resource Metrics API is healthy, add a CPU/mem HPA, and diagnose the one failure mode where the pod is Ready but top is empty. 27 Jul 2026 ### [Operating on Green](https://blog.derio.net/frank/docs/operating/30-silent-failure/) Five ways a healthy dashboard lies — wrong artifact, unconsumed signal, out of scope, stale view, never applied — and the command that checks each ## The Frank Papers Compare approaches, examine the evidence, and understand the trade-offs. 19 May 2026 ### [Immutable OS & Declarative Machines](https://blog.derio.net/frank/docs/papers/02-immutable-os/) Immutable OS for Kubernetes nodes is a four-job problem — read-only root, transactional updates, declarative config, and (at the strict end) no SSH at all — and the six 2026… 19 May 2026 ### [GitOps at Small Scale](https://blog.derio.net/frank/docs/papers/06-gitops-at-small-scale/) GitOps on Kubernetes is a four-jobs-in-one bundle — reconcile, drift- detect, gate write access, audit — and the six contenders in 2026 (ArgoCD, Flux v2, Jenkins X,… 19 May 2026 ### [The Observability Stack, Honestly](https://blog.derio.net/frank/docs/papers/07-observability-honestly/) Cluster observability is a five-job problem — metrics, logs, traces, dashboards, alerts — and the six contenders in 2026 (Grafana+Prometheus+Loki, VictoriaMetrics, Mimir,…