Operating on Progressive Delivery

Last updated 2026-07-15 ·421d1dc

This is the operational companion to Progressive Delivery with Argo Rollouts. That post covers the architecture. This one covers promoting rollouts, reading analysis results, and recovering from stuck or degraded states.

2026-05-04 update — The LiteLLM canary went through two rewrites in one day. First was a replica-count canary with AnalysisRun between pauses. Second is pause-only (no AnalysisRun) — because the AnalysisTemplate referenced litellm_request_total, a Prometheus metric the OSSOpen Source SoftwareSoftware whose source is public and freely modifiable. Worth flagging here because several projects ship an OSS build alongside a paid one, and the difference is often exactly the feature you wanted — LiteLLM's Prometheus metrics are Enterprise-only. LiteLLM image doesn’t emit (it’s Enterprise-only). The full postmortem (five latent bugs) is in the building post.

    graph LR
    subgraph stable["Stable ReplicaSet"]
        s1["Pod v1"]
        s2["Pod v1"]
        s3["Pod v1"]
    end

    subgraph canary["Canary ReplicaSet"]
        c1["Pod v2"]
    end

    subgraph svc["Services"]
        svc_stable["stable Service"]
        svc_canary["canary Service"]
    end

    subgraph analysis["Analysis"]
        ar["AnalysisRun"]
        prom["Prometheus<br/>metric query"]
    end

    router["Traffic Router<br/>(replica-count)"]

    router -->|"100% weight"| svc_stable --> s1
    router -->|"X% weight"| svc_canary --> c1

    Rollout -->|"step pause"| ar -->|"metric"| prom
    ar -->|"pass/fail"| Rollout

    Rollout -.->|"promote"| router
  

What Healthy Looks Like

• The argo-rollouts controller pod is Running in the argo-rollouts namespace.
• LiteLLM Rollout shows Status: Healthy with all pods Ready under one ReplicaSet (at-rest rollouts have no current step).
• Sympozium Rollout shows Status: Healthy with the active service serving.
• No Degraded or Paused rollouts exist (unless you’re mid-rollout).

Verify

# Controller
kubectl get pods -n argo-rollouts
kubectl logs -n argo-rollouts deploy/argo-rollouts --tail=20 | grep -i error

# Rollout status
kubectl argo rollouts list rollouts -n litellm
kubectl argo rollouts list rollouts -n sympozium

# Detailed view
kubectl argo rollouts get rollout litellm -n litellm

Steps

Promote a Canary Step

# Check current step
kubectl argo rollouts get rollout litellm -n litellm

# Promote to next step
kubectl argo rollouts promote litellm -n litellm

Abort a Canary

kubectl argo rollouts abort litellm -n litellm

# Verify rollback
kubectl argo rollouts get rollout litellm -n litellm

Switch Blue-Green (Sympozium)

# Trigger a new ReplicaSet by updating the image
kubectl argo rollouts set image sympozium -n sympozium \
  sympozium=ghcr.io/derio-net/sympozium:<NEW_TAG>

# Promote to switch active service
kubectl argo rollouts promote sympozium -n sympozium

View Analysis Run Results

kubectl get analysisrun -n litellm
kubectl describe analysisrun -n litellm <name>

Recover

Rollout Stuck at a Step

kubectl argo rollouts get rollout litellm -n litellm
# Check if the step is paused waiting for promotion
# Step Pause shows "NextStepAfter: ..." or "Promote to proceed"

Promote manually: kubectl argo rollouts promote litellm -n litellm.

If the rollout is stuck with no progress and no error:

• Check the controller logs for the Rollout: kubectl logs -n argo-rollouts deploy/argo-rollouts | grep litellm
• Check if the canary ReplicaSet can’t schedule pods (resource quota, affinity, image pull)
• Check if the workloadRef.scaleDown policy is set to onsuccess — otherwise the canary ReplicaSet isn’t scaled down after promotion

Metric Analysis Fails

# Check the AnalysisRun
kubectl describe analysisrun -n litellm <name>

# Check controller logs for metric query failures
kubectl logs -n argo-rollouts deploy/argo-rollouts | grep -i "metric\|analysis\|query"

Known causes:

• The metric doesn’t exist. LiteLLM’s Prometheus integration is Enterprise-only. The OSS image doesn’t export litellm_request_total. The canary was converted to pause-only (no AnalysisRun).
• The Prometheus provider can’t reach VictoriaMetrics. Check network policies and service DNS.
• 4xx responses not counted as failure. The AnalysisTemplate must count 4xx as failure, not just 5xx — a missing successCondition can silently pass a bad canary (#216).

Canary Won’t Start

kubectl describe rs -n litellm -l rollouts-pod-template-hash=<HASH>

Check image pull errors, resource limits, and affinity rules. LiteLLM requires amd64 affinity — if the node selector is wrong, the canary pod stays Pending.

Rollout Controller Crashing

kubectl logs -n argo-rollouts deploy/argo-rollouts --tail=50

Check the Cilium traffic router plugin if the logs show failed to get traffic router plugin. The initial canary tried using Cilium’s traffic router plugin, which didn’t work on this cluster — the fix was switching to replica-count weighting and dropping the Cilium plugin (#213).

ArgoCD Drift on Rollout Replicas

ArgoCD may try to reset spec.replicas on the Rollout back to the Git value. Add ignoreDifferences to the Application CRCustom ResourceAn object of a type Kubernetes did not ship with, added by a CRD. Frank's ArgoCD Applications, Rollouts and Tekton Pipelines are all CRs. for spec.replicas on Rollout resources — same pattern as the GPU switcher.

Missteps

Quick Reference

References

• Building Post — Progressive Delivery
• Argo Rollouts Documentation
• Argo Rollouts kubectl Plugin