Operating on Health Monitoring

Last updated 2026-07-15 ·a77bf48

This is the operational companion to Health Monitoring. That post covers the architecture behind feature probes, heartbeats, and alerting. This one covers what you type when a probe shows down, a heartbeat went stale, or an alert fired but nobody got the Telegram notification — including the delivery failures we discovered the hard way.

Before any commands below, source the environment:

source .env          # sets KUBECONFIG, TALOSCONFIG
source .env_devops   # sets OMNICONFIG + service accounts

Data Flow

    flowchart TD
    subgraph "Probes & Heartbeats"
        A["Blackbox Exporter<br/>HTTP probes"] --> P[(VictoriaMetrics)]
        B["Cron scripts<br/>Pushgateway"] --> P
    end
    P --> C[Grafana alert rules]
    C --> D{"Condition met?"}
    D -->|Yes, firing| E[Notification policy]
    D -->|No, resolved| F[Update Lifecycle only]
    E --> G{Severity}
    G -->|critical| H[Telegram + Health Bridge]
    G -->|warning| I[Health Bridge only]
    H --> J[Derio Ops board + Issue]
    I --> J
  

Feature probes and cron heartbeats feed into VictoriaMetrics. Grafana alert rules evaluate conditions; when a threshold is breached, the notification policy routes by severity — critical alerts go to Telegram + Health Bridge, warnings to the Health Bridge alone. The bridge updates the Derio Ops board’s Lifecycle field and opens/closes bug issues.

What Healthy Looks Like

All feature health probes show probe_success=1, all cron heartbeats push within their expected cadences, and Grafana alert rules fire and notify through Telegram without silent delivery failures. The Feature Health dashboard at /d/fh-overview/feature-health shows every probe UP and every heartbeat recent.

Verify

Checking Probe Status

# Port-forward to Blackbox Exporter
kubectl port-forward -n monitoring svc/blackbox-exporter 9115:9115 &

# Probe a specific endpoint
curl -s "http://localhost:9115/probe?target=https://grafana.frank.derio.net&module=http_2xx" | grep probe_success
# Expected: probe_success 1

# Check all feature health probes via VictoriaMetrics
GRAFANA_AUTH="admin:$(kubectl get secret -n monitoring victoria-metrics-grafana -o jsonpath='{.data.admin-password}' | base64 -d)"
curl -sk -u "$GRAFANA_AUTH" "https://grafana.frank.derio.net/api/datasources/proxy/uid/P4169E866C3094E38/api/v1/query" \
  --data-urlencode 'query=probe_success{probe_group="feature_health"}'

Checking Heartbeat Metrics

# Port-forward to Pushgateway
kubectl port-forward -n monitoring svc/pushgateway 9091:9091 &

# View all heartbeat metrics
curl -s http://localhost:9091/metrics | grep willikins_heartbeat

# Push a test heartbeat
echo "willikins_heartbeat_last_success_timestamp $(date +%s)" | \
  curl -s --data-binary @- http://localhost:9091/metrics/job/test_job

# Delete a test metric
curl -s -X DELETE http://localhost:9091/metrics/job/test_job

Listing Active Alert Rules

All Grafana alerting is file-provisioned via ConfigMaps:

GRAFANA_AUTH="admin:$(kubectl get secret -n monitoring victoria-metrics-grafana -o jsonpath='{.data.admin-password}' | base64 -d)"

# List all alert states
curl -sk -u "$GRAFANA_AUTH" \
  "https://grafana.frank.derio.net/api/prometheus/grafana/api/v1/alerts" | \
  python3 -c "import json,sys; [print(f'{a[\"state\"]}: {a[\"labels\"][\"alertname\"]}') for a in json.load(sys.stdin)['data']['alerts']]"

Steps

Editing Alert Rules

File-provisioned rules are read-only in the UI. To modify:

1. Edit the ConfigMap YAML in apps/grafana-alerting/manifests/alert-rules-cm.yaml
2. Commit and push — ArgoCD syncs the ConfigMap
3. Restart Grafana to reload provisioning files:

   kubectl delete pod -n monitoring -l app.kubernetes.io/name=grafana
4. Verify the rule loaded without errors:

   kubectl logs -n monitoring -l app.kubernetes.io/name=grafana --tail=200 | grep -iE 'parseError|provisioning.*error'

Editing the Feature Health Dashboard

1. Open the provisioned dashboard in Grafana UI, click “Save as” to create a scratch copy
2. Edit freely in the UI
3. Export the final JSON (Share → Export → Save to file)
4. Replace the JSON content in apps/grafana-alerting/manifests/dashboard-cm.yaml
5. Commit, push, restart Grafana pod
6. Delete the scratch dashboard

Pushing a Manual Heartbeat

kubectl exec -n secure-agent-pod deploy/secure-agent-pod -c kali -- /opt/scripts/push-heartbeat.sh <job_name> [label=value ...]
# Example: kubectl exec ... /opt/scripts/push-heartbeat.sh exercise_reminder context=desk

Recover

Telegram Notification Not Arriving

If an alert fires but no Telegram message arrives, check three failure patterns we’ve fixed:

1. Markdown stripping underscores — parse_mode: Markdown interprets _ as italic. job=session_manager renders as sessionmanager (commit cc239cf9). Fixed by removing parse_mode.

2. HTML annotation rejection — annotation values like <node-ip> are valid strings but Telegram’s HTML parser rejects them as invalid HTML tags. The message dispatches but Telegram returns HTTP 400 silently (commit c866a85e). Fixed by stripping <>& from annotations.

3. Trailing newline in bot token — the Telegram bot token from Infisical had a trailing newline, causing HTTP 404 on sendMessage — also silent (commit f7d8f189). Fixed by defensive credential stripping.

Check Grafana logs for any of these:

kubectl logs -n monitoring -l app.kubernetes.io/name=grafana -c grafana --tail=50 | grep -iE "error|warn|notify|telegram|400|404"

If the contact point is misconfigured, restart Grafana to reset the notification dedup state:

kubectl delete pod -n monitoring -l app.kubernetes.io/name=grafana

VMProbe/VMServiceScrape Not Applying

If kubectl apply fails with x509: certificate signed by unknown authority:

# The VictoriaMetrics Operator webhook caBundle is out of sync
kubectl get validatingwebhookconfiguration -l app.kubernetes.io/instance=victoria-metrics -o yaml | grep caBundle | head -1

# Fix: restart the operator to regenerate certs
kubectl rollout restart deployment -n monitoring victoria-metrics-operator

Dashboard Shows No Data

• Verify datasource UIDUser IdentifierThe number Linux actually checks for file permissions — the name is a lookup. In containers a uid mismatch against a mounted volume is the usual cause of a permission error. is P4169E866C3094E38
• Table panels require "format": "table" on targets
• ALERTS{} metric doesn’t exist for Grafana-managed alerts — use alertlist panel type

Missteps

Quick Reference

References

• Prometheus Blackbox Exporter
• Prometheus Pushgateway
• Grafana Alerting API
• Building Post 22: Health Monitoring