Operating on Observability

Last updated 2026-09-14 ·a77bf48

This is the operational companion to Building Observability. That post covers the architecture decisions and deployment gotchas. This one covers what you actually type when a dashboard is empty, an alert didn’t fire, or logs are missing — and the failure patterns that have bitten us more than once.

Before any of the commands below, source the environment:

source .env          # sets KUBECONFIG, TALOSCONFIG
source .env_devops   # sets OMNICONFIG + service accounts

What Healthy Looks Like

Frank’s observability stack has four moving parts:

• VictoriaMetrics (VMSingle + vmagent) — time-series metrics database and scraping engine (monitoring namespace, 20Gi Longhorn PVCPersistentVolumeClaimA Kubernetes request for durable storage. The pod names a claim and the storage layer — Longhorn on Frank — binds real disk behind it, so the data outlives the pod., 1-month retention)
• Grafana at http://192.168.55.203 — dashboards and exploration, OIDCOpenID ConnectAn identity layer on top of OAuth 2.0: it adds a token saying *who* the user is, not merely what the client may do. Frank's single sign-on speaks it, with Authentik as the provider. auth via Authentik
• Fluent Bit — DaemonSet on all 7 nodes (including tainted control-plane and GPU nodes), shipping container logs
• VictoriaLogs — log storage with 30-day retention, queryable through Grafana’s Explore tab

Supporting collectors: node-exporter (hardware metrics on all nodes), kube-state-metrics (Kubernetes object metrics), blackbox-exporter (endpoint probes for alerting), and the three control-plane etcd members, scraped directly on their dedicated metrics listener (see Checking the etcd Scrape — healthy is three up series, not zero).

All four pieces running means the stack is healthy:

kubectl get pods -n monitoring | head -20

Expected output shows Running for vmsingle, victoria-logs, grafana, vmagent, fluent-bit, node-exporter, and kube-state-metrics pods.

    graph LR
  subgraph COLL["Collectors (7 nodes)"]
    FB["Fluent Bit<br/>DaemonSet"]
    NE["node-exporter<br/>DaemonSet"]
    KSM["kube-state-metrics"]
    BB["blackbox-exporter"]
  end
  subgraph STORE["Storage Layer"]
    VM["VictoriaMetrics<br/>VMSingle"]
    VL["VictoriaLogs"]
  end
  subgraph UI["Visualization"]
    GF["Grafana<br/>192.168.55.203"]
  end
  FB -->|"logs"| VL
  NE -->|"metrics"| VM
  KSM -->|"metrics"| VM
  BB -->|"probes"| VM
  VM --> GF
  VL --> GF
  

Verify

Grafana Dashboards

Open http://192.168.55.203 in a browser. The stack ships with pre-built dashboards under the “VictoriaMetrics” folder:

• Node Exporter Full — per-node CPU, memory, disk I/O, network, filesystem
• Kubernetes / Compute Resources / Cluster — cluster-wide CPU and memory requests vs limits vs actual usage
• Kubernetes / Compute Resources / Namespace — the same, broken down by namespace
• VMAgent — scrape targets, samples/sec, queue depth

These dashboards are provisioned by the Helm chart and survive Grafana pod restarts because Grafana’s 1Gi Longhorn PVC preserves them.

Querying Metrics with MetricsQL

Port-forward to VMSingle and use its built-in UI:

kubectl port-forward -n monitoring svc/vmsingle-victoria-metrics-victoria-metrics-k8s-stack 8429:8429

Then open http://localhost:8429/vmui in your browser. MetricsQL is a superset of PromQL — any PromQL query works, plus extensions like keep_metric_names and range_median.

Starter queries:

# CPU usage by node (1m average)
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)

# Memory usage percentage by node
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Pod restart counts in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 0

# Disk usage on Longhorn volumes
kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes * 100

From the CLI:

kubectl port-forward -n monitoring svc/vmsingle-victoria-metrics-victoria-metrics-k8s-stack 8429:8429 &
curl -s 'http://localhost:8429/api/v1/query?query=up' | jq '.data.result[] | {instance: .metric.instance, up: .value[1]}'

Querying Logs with VictoriaLogs

Logs are queryable through Grafana’s Explore tab — select the “VictoriaLogs” datasource and use LogsQL:

# All logs from ArgoCD
{kubernetes_namespace_name="argocd"}

# Logs from VictoriaMetrics pods
{kubernetes_pod_name=~"victoria-metrics.*"}

# Error lines across the entire cluster
{kubernetes_namespace_name=~".+"} |= "error"

# Logs from the GPU node
{kubernetes_host="gpu-1"} | level:error

For CLI access, port-forward to VictoriaLogs:

kubectl port-forward -n monitoring svc/victoria-logs-victoria-logs-single-server 9428:9428
curl -s 'http://localhost:9428/select/logsql/query?query={kubernetes_namespace_name="monitoring"}&limit=10' | jq .

Checking Pipeline Health

# vmagent is scraping
kubectl get pods -n monitoring -l app.kubernetes.io/name=vmagent
kubectl logs -n monitoring -l app.kubernetes.io/name=vmagent --tail=5

# Fluent Bit is running on all nodes
kubectl get ds -n monitoring fluent-bit
# DESIRED and READY should match (7 nodes)

# VictoriaLogs is accepting writes
kubectl logs -n monitoring -l app=victoria-logs-single-server --tail=5

Steps

Creating and Importing Grafana Dashboards

To import a community dashboard (e.g. ID 1860 for Node Exporter Full):

1. Open Grafana at http://192.168.55.203
2. Go to Dashboards > Import
3. Enter the dashboard ID and click Load
4. Select the VictoriaMetrics datasource and click Import

Imported dashboards are saved to the 1Gi Longhorn PVC and survive pod restarts. One gotcha: because that PVC is ReadWriteOnce, we pinned Grafana’s deployment strategy to Recreate (commit e40c952d). A RollingUpdate with a RWOReadWriteOnceA PVC access mode that lets exactly one node mount the volume read-write at a time. It is the reason a RollingUpdate deadlocks: the replacement pod cannot mount the volume until the outgoing pod releases it. volume means the new pod can’t attach the PVC on a different node until the old pod releases it — which doesn’t happen cleanly, so the rollout stalls for ~2 hours.

Adjusting Retention

Metrics retention (1 month) is set in apps/victoria-metrics/values.yaml:

vmsingle:
  spec:
    retentionPeriod: "1"

Log retention (30 days) is in apps/victoria-logs/values.yaml:

server:
  retentionPeriod: 30d

We bumped logs from 14d to 30d alongside the cross-cluster shipping fix in d8de469c. Change the value, commit, and let ArgoCD sync. Existing data outside the new window is garbage-collected on the next retention pass.

Checking What vmagent Is Scraping

kubectl port-forward -n monitoring svc/vmagent-victoria-metrics-victoria-metrics-k8s-stack 8429:8429

Open http://localhost:8429/targets to see every scrape target, its status (up/down), last scrape time, and error messages. This is the first place to look when a metric is missing.

Checking the etcd Scrape

Frank scrapes its own etcd on the control-plane minis, on etcd’s dedicated metrics listener (:2381, plain HTTP, read-only). It did not for the first 148 days of this stack’s life. The scrape was enabled by chart default with a pod selector that can never match on Talos, so the Endpoints object was empty and nothing anywhere said so. The first fix, a static Endpoints object from chart values, never deployed at all, because ArgoCD excludes that kind. See Building Observability Gotcha 4 for both.

The scrape is now a VMStaticScrape, so vmagent’s target list is the thing to read, not a Service and not an Endpoints object:

# The scrape object. In git at
# apps/victoria-metrics/manifests/vmstaticscrape-kube-etcd.yaml.
kubectl -n monitoring get vmstaticscrape kube-etcd -o yaml

# What vmagent made of it: three 192.168.55.2x:2381 targets, health "up".
# No kube-etcd rows at all means the object is missing or was never loaded.
VMAGENT=$(kubectl -n monitoring get pod -l app.kubernetes.io/name=vmagent -o name | head -1)
kubectl -n monitoring exec "$VMAGENT" -c vmagent -- wget -qO- http://127.0.0.1:8429/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.labels.job=="kube-etcd") | "\(.labels.instance) \(.health) \(.lastError)"'

Do not go looking for a kube-etcd Endpoints object. If one exists, it is an empty leftover from the chart era and means nothing. ArgoCD never applies Endpoints (they are in its resource.exclusions), so no change in git can populate one. After merging anything that adds a new kind, check the Application’s conditions for an ExcludedResourceWarning:

kubectl -n argocd get application victoria-metrics -o jsonpath='{.status.conditions}'

Reading the target list is only half the check. It finds a scrape with no targets; it says nothing about a scrape whose targets never answer. That second failure is live on this cluster right now:

# Scrapes that resolved targets and then failed them.
count(up == 0) by (job)          # -> {job="kube-scheduler"} 3
up{job="kube-scheduler"}         # 0 on all three, since the cluster was built

kube-scheduler has three healthy-looking endpoints (Talos runs it as a static pod, so the chart’s selector works) and three failing scrapes — Talos binds 10259 with TLS and auth the chart default does not satisfy, so there have been zero scheduler_* series for as long as there were zero etcd ones. It is a known, unfixed gap, of a different species from etcd’s: an absent series and a series reading 0 are different states, and only the first is NoData. Run both checks; neither finds the other’s failure.

Is anything listening? Ask from inside the cluster, not from a laptop:

VMAGENT=$(kubectl -n monitoring get pod -l app.kubernetes.io/name=vmagent -o name | head -1)
for ip in 192.168.55.21 192.168.55.22 192.168.55.23; do
  kubectl -n monitoring exec "$VMAGENT" -c vmagent -- wget -qO- "http://$ip:2381/metrics" | head -1
done

Then the queries that matter. On a healthy cluster the first returns three series, all 1:

# Is the scrape alive? One series per control-plane member.
up{job="kube-etcd"}

# Does the quorum have a leader? 1 = yes, per member.
etcd_server_has_leader

# Leader-change churn over the last hour. Steady state is 0.
increase(etcd_server_leader_changes_seen_total[1h])

# WAL fsync p99 — how disk-bound etcd's commits are.
histogram_quantile(0.99, sum by (le, instance) (rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])))

# Backend size against the member's own advertised quota (0–1).
etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes

The trap: etcd_request_duration_seconds, etcd_request_errors_total, etcd_requests_total, etcd_lease_object_counts and etcd_bookmark_counts are NOT etcd. They are the apiserver’s storage client, exported from inside the apiserver, and they are present whether or not etcd is scraped at all — they were present for all 148 blind days. If you are ever tempted to repair a broken etcd rule by repointing it at a metric that “has data”, that is the metric you will reach for, and the rule will go green while measuring the wrong process. Only etcd_server_*, etcd_disk_*, etcd_mvcc_* and etcd_network_* are evidence.

The same five signals are on the Frank Layer 2 — etcd (curated) dashboard (uid frank-l2-etcd): has-leader per node, leader changes per hour, WAL fsync p99, database size against quota, and peer round-trip p99.

What normal looks like under load (measured 2026-09-14). Two soaks of 7500 rerank requests, generated on mini-1 against the retrieval tier, ran at about 16.5 requests per second each. That is the server’s ceiling: tripling the client concurrency only lengthened the queue. Each ran for roughly seven and a half minutes. Neither moved anything that matters. There were zero leader changes on all three members, etcd_server_has_leader stayed at 1, WAL fsync p99 on the loaded node went from 2.0 to 3.5 ms (peers flat near 2.0 ms, against a 50 ms alert threshold), and cluster apiserver p99 went from 22–24 to about 24 ms. One-minute p99 samples are noisy. An idle sample has shown one mini at 15 ms fsync, and another at 44 ms apiserver, on its own. So compare a node against its own baseline, never against a peer’s single reading.

The curated board is the only etcd dashboard. While the chart’s kubeEtcd block was enabled, it also rendered a board titled etcd and a VMRule of 15 upstream etcd alerts, which never evaluated because vmalert is disabled here. Both went away with kubeEtcd.enabled: false, and their live copies were deleted on 2026-09-14 (manual op obs-etcd-chart-orphans-delete), because the Application runs prune: false. If a board titled etcd ever reappears, it is an orphan: delete it, never the Frank board. The live rules are the six layer-2-etcd-* ones below.

What the etcd Alerts Mean When They Fire

Only the first two reach Telegram. The other four go to the health bridge and become tracked bugs that auto-close on heal.

Two windows are load-bearing. The 10m on both paging rules exists because a planned Talos rolling reboot takes one member down and elects a new leader entirely legitimately — the August 2026 control-plane roll took roughly seven minutes and produced 48 alerts against a completely healthy cluster. An etcd alert that fires on every planned roll gets muted inside a month, and a muted alert is worse than no alert because it still looks like coverage.

The 15m on the absent() watchdog is its only tolerance: absent() has no for-like patience of its own, so the window is what absorbs a vmagent restart or a rolling etcd apply without paging a false alarm.

PVC Capacity Alerts

Three rules watch volume fill from the kubelet’s own volume stats: PVC Capacity Above 85 Percent (warning, 30m), PVC Capacity Above 95 Percent (critical, 5m) and PVC Inodes Above 90 Percent (warning, 30m). The kubelet is the source, so the kube-state-metrics scrape cap that can blind every kube_* rule does not touch them. A volume that is not mounted reports no series; the rule goes NoData and stays quiet, which is right because it cannot fill.

Only an operator can clear a full volume, so these alerts repeat to Telegram every 4 hours rather than every 3 minutes. When one fires, check Longhorn provisioning headroom before expanding the claim, because an expansion can be refused silently while ArgoCD stays green.

Exploring Available Metrics

# List all metric names
curl -s 'http://localhost:8429/api/v1/label/__name__/values' | jq '.data[:20]'

# Search for metrics by keyword
curl -s 'http://localhost:8429/api/v1/label/__name__/values' | jq '.data[] | select(test("gpu|nvidia"))'

Recover

Missing Metrics — the cardinality labeldrop

If a metric you expect is missing from VMSingle but the exporter is running, the most likely cause is high-cardinality labels hitting the series limit. VictoriaMetrics’ default -maxLabelsPerTimeseries=40 silently drops series that exceed it.

We hit this in May 2026 (commit 193c3890). The amd64 nodes’ node-exporter metrics carried 60–135 labels each — NFDNode Feature DiscoveryThe component that inspects each node and labels it with what it has — CPU features, kernel modules, PCI devices — so workloads can select hardware by capability instead of by hostname. CPU feature labels (feature_node_kubernetes_io_*), Talos extension labels (extensions_talos_dev_*), NVIDIA driver labels (nvidia_com_*). The raspi nodes had ~33 labels and passed; the 5 amd64 nodes were silently dropped. VMSingle logged ignoring series with N labels... but no alert fired.

The fix was to drop those high-cardinality label groups in apps/victoria-metrics/values.yaml:

kubelet:
  metricRelabelConfigs:
  - action: labeldrop
    regex: ^(uid|id)$
  - action: labeldrop
    regex: ^(name)$
  - action: labeldrop
    regex: ^(feature_node_kubernetes_io_.*|extensions_talos_dev_.*|nvidia_com_.*|beta_kubernetes_io_.*)$
  - action: drop
    regex: rest_client_request_duration_seconds_(bucket|sum|count)

The key choice: we dropped the labels rather than bumping -maxLabelsPerTimeseries. Bumping the limit would absorb the bloat temporarily but push the cardinality bomb downstream — the series count would keep growing until it hit storage or query limits. Dropping the labels at scrape time is the correct fix.

In general, when a metric is missing:

1. Check the exporter pod is running:

   kubectl get pods -n monitoring -l app.kubernetes.io/name=node-exporter
   kubectl get pods -n monitoring -l app.kubernetes.io/name=kube-state-metrics

2. Check vmagent targets at http://localhost:8429/targets — is it scraping the endpoint?

3. Check the VMServiceMonitor exists and matches labels:

   kubectl get vmservicemonitors -n monitoring
   kubectl describe vmservicemonitor <name> -n monitoring

4. Check the exporter directly:

   kubectl port-forward -n monitoring <exporter-pod> <port>:<port>
   curl http://localhost:<port>/metrics | grep <metric-name>

Fluent Bit Not Shipping Logs

If logs are not appearing in VictoriaLogs:

1. Check Fluent Bit pods are running on all nodes:

   kubectl get ds -n monitoring fluent-bit
   kubectl get pods -n monitoring -l app.kubernetes.io/name=fluent-bit -o wide

2. Check Fluent Bit logs for output errors:

   kubectl logs -n monitoring -l app.kubernetes.io/name=fluent-bit --tail=50

   Look for retry lines. Silent retries with no error detail usually mean DNS resolution failure — the output hostname is wrong or the target service is down.

3. Verify the destination hostname resolves:

   kubectl exec -n monitoring <fluent-bit-pod> -- nslookup \
     victoria-logs-victoria-logs-single-server.monitoring.svc.cluster.local

4. Check tail file positions — Fluent Bit tracks where it left off. Stale positions mean it may be re-reading or skipping:

   kubectl exec -n monitoring <fluent-bit-pod> -- ls -la /var/log/flb_kube.db

Grafana Rollout Stuck

If you’re changing Grafana config and the rollout hangs:

kubectl rollout status -n monitoring deployment grafana

If it stalls for more than a few minutes, check whether the strategy is Recreate — it should be (commit e40c952d). If it’s RollingUpdate, the RWO Longhorn PVC will block the new pod from starting until the old pod terminates, and Kubernetes won’t terminate the old pod until the new one starts. Either change the strategy back to Recreate or scale the old replica to zero:

kubectl scale deployment -n monitoring grafana --replicas=0
# wait, then
kubectl scale deployment -n monitoring grafana --replicas=1

Alert Didn’t Fire — Telegram formatting failures

If an alert rule fires but nobody gets notified, check the contact point. We’ve had three distinct Telegram delivery failures:

1. Markdown parsed _ as italic — job=session_manager rendered as sessionmanager, so incident triage was routed to the wrong handler. Fixed by dropping parse_mode: Markdown from the Telegram contact point (commit cc239cf9) — but dropping it left the value unset, and Grafana treats unset as HTML (see 2), so it is now pinned explicitly to parse_mode: None (frank#799).

2. HTML annotation values rejected — annotation values contained <node-ip> and >6. Telegram’s HTML parser rejected <node-ip> as an invalid HTML tag. The message dispatched but Telegram returned HTTP 400 — silently (commit c866a85e). The root cause was that an unset parse_mode means HTML, not plain text; the contact point is now pinned to parse_mode: None (frank#799), and stripping <>& from annotations remains as a style guard.

3. Trailing newline in bot token — the Telegram bot token from Infisical had a trailing newline, causing HTTP 404 on sendMessage — also silent (commit f7d8f189). Fixed by defensive credential stripping at the secret level.

If you suspect a silent delivery failure, check the Grafana Alerting log:

kubectl logs -n monitoring deployment/grafana --tail=100 | grep -i "telegram\|failed\|error\|400\|404"

The Watcher Also Goes Silent — Credential Expiry

The Telegram bot for the persistent agent (alert-agent) authenticates with a Claude OAuth token that expires on a hard ~30-day clock. In July 2026 it expired unnoticed: the pod stayed 3/3 Running, ArgoCD stayed green, and the failure (Login expired · Please run /login) lived inside a tmux pane — invisible to every probe. The bot was dead for three days before anyone noticed.

The fix is a daily in-container check (cred-expiry-check, 09:00) that reads the token’s refreshTokenExpiresAt and:

• warns Telegram when ≤7 / ≤3 / ≤1 days remain (wording sharpens as it nears), so a re-login is a scheduled chore, not a scramble;
• emits a heartbeat line (cred-expiry-check days_left=N tier=…) that a Grafana dead-man rule (alert-agent-cred-expiry-heartbeat-stale) watches — if the check itself stops, that rule pages directly.

Day-to-day: you’ll get a Telegram nudge roughly monthly; re-login by attaching the agent’s tmux and running /login (runbook: obs-alert-agent-claude-login). To see the heartbeat:

# the last heartbeat line
kubectl exec -n alert-agent deploy/alert-agent -c agent -- \
  /opt/alert-agent-bin/cred-expiry-check

One trap worth recording: Frank’s VictoriaLogs carries the message in the _msg field, but Hop’s fluent-bit uses log — so the dead-man rule’s LogsQL had to use _msg:"cred-expiry-check"; the Hop CrowdSec canary’s log:"…" form returns 0 on Frank and would have left the watchdog permanently blind. Same lesson as the silent Telegram failures above: a monitor that Synced is not a monitor that fires — you have to observe it end to end.

False Positives from Completed Pods

Some layer alerts fire for namespaces that run Tekton pipelines, Argo Workflows, or other Jobs/CronJobs. kube_pod_status_ready{condition="true"} reports 0 for pods in Completed or Error state — those are by-design not-Ready post-completion.

If a namespace with active CI accumulates such pods, a reduce.last on the per-pod series can pick one and trip the threshold. The fix (commit 8068eedd) was to exclude completed pods from the layer-8 observability alert:

kube_pod_status_ready{namespace="monitoring"}
  unless on(namespace,pod)
    kube_pod_status_phase{phase=~"Succeeded|Failed"} == 1

For layer alerts on CI namespaces, switch to Deployment-based queries:

kube_deployment_status_replicas_unavailable{namespace=~"tekton|workflows|…"}

Deployments are long-running; Job pods aren’t owned by Deployments, so they’re excluded naturally.

High Cardinality

If VMSingle memory usage is climbing or queries are slow, high cardinality labels are the usual cause:

# Check top series by cardinality
curl -s 'http://localhost:8429/api/v1/status/tsdb' | jq '.data.seriesCountByMetricName[:10]'

If a metric has an unbounded label (request ID, session token), either drop the label in vmagent’s relabeling config or exclude the metric entirely.

VictoriaLogs Query Returns No Results

1. Check VictoriaLogs is receiving data:

   kubectl port-forward -n monitoring svc/victoria-logs-victoria-logs-single-server 9428:9428
   curl -s 'http://localhost:9428/select/logsql/query?query=*&limit=5' | jq .

   If results come back, the problem is your query syntax, not the pipeline.

2. Check the Grafana datasource — the VictoriaLogs datasource must point to http://victoria-logs-victoria-logs-single-server.monitoring.svc.cluster.local:9428. Go to Grafana > Configuration > Data Sources and verify. If the queryType is not set to stats, queries against long series will fail with a DatasourceError (commit a6651c4f — the fix was setting queryType: stats to hit /select/logsql/stats_query instead of /select/logsql/query).

3. Check retention — if logs are older than 30 days, they have been garbage-collected.

Missteps

Quick Reference

References

• VictoriaMetrics Documentation — MetricsQL reference, VMSingle operations, retention
• VictoriaLogs Documentation — LogsQL syntax, ingestion API
• Grafana Documentation — Dashboard management, datasource provisioning
• Fluent Bit Documentation — Pipeline debugging, tail input, HTTP output
• Building Observability — Architecture decisions and deployment gotchas