Operating on Paperclip

Last updated 2026-07-15 ·4769745

This is the operational companion to Paperclip — AI Agent Orchestrator. That post covers architecture and deployment. This one covers health checks, database access, the shell sidecar, and common failure modes.

    graph LR
    subgraph ns["paperclip-system namespace"]
        pc["Paperclip Pod<br/>gpu-1 pinned"]
        pg["PostgreSQL Pod<br/>paperclip-db"]
        shell["Shell Sidecar<br/>sshd + mosh"]

        subgraph pvs["Persistent Volumes"]
            pvcData["paperclip-data<br/>10Gi RWO"]
            pvcDB["paperclip-db<br/>5Gi RWO"]
            pvcShell["paperclip-shell-home<br/>20Gi RWO"]
        end

        subgraph secrets["External Secrets"]
            llm["paperclip-llm-key<br/>→ LiteLLM"]
            auth["paperclip-auth<br/>→ session signing"]
            brave["paperclip-brave<br/>→ Brave Search"]
            resend["paperclip-resend<br/>→ Resend email"]
        end
    end

    subgraph infra["Infrastructure"]
        infisical["Infisical<br/>Secret Store"]
        lb["LoadBalancer<br/>192.168.55.212:3100"]
        shellLB["LoadBalancer<br/>192.168.55.221:22"]
    end

    pc --- pvcData
    pg --- pvcDB
    shell --- pvcShell
    llm & auth & brave & resend -.->|"ESO sync"| infisical
    pc --- lb
    shell --- shellLB
    pc --- pg
  

What Healthy Looks Like

• The Paperclip pod is 1/1 Running on gpu-1.
• PostgreSQL pod is 1/1 Running.
• All four ExternalSecrets show SecretSynced.
• The web UI responds at http://192.168.55.212:3100.
• The shell sidecar LBLoad BalancerWhatever spreads traffic across backends and gives them one address. On Frank that is Cilium answering for an address on the LAN, not a cloud appliance. is reachable at 192.168.55.221.

Verify

# All-in-one
kubectl get pods,pvc,externalsecret -n paperclip-system

# Web UI
curl -s -o /dev/null -w "%{http_code}" http://192.168.55.212:3100/

# Database
kubectl exec -it -n paperclip-system \
  $(kubectl get pod -n paperclip-system -l app.kubernetes.io/instance=paperclip-db -o name) \
  -- psql -U paperclip -d paperclip -c "SELECT count(*) FROM pg_tables;"

# Shell sidecar
ssh agent@192.168.55.221 -t tmux new -A -s main

Steps

Restart Paperclip

kubectl rollout restart deployment/paperclip -n paperclip-system
kubectl get pods -n paperclip-system -w

Uses Recreate strategy (RWOReadWriteOnceA PVC access mode that lets exactly one node mount the volume read-write at a time. It is the reason a RollingUpdate deadlocks: the replacement pod cannot mount the volume until the outgoing pod releases it. PVCPersistentVolumeClaimA Kubernetes request for durable storage. The pod names a claim and the storage layer — Longhorn on Frank — binds real disk behind it, so the data outlives the pod. — rolling update would deadlock). Expect 10–30s downtime.

Reconcile Shell Inventory

# After editing apps/paperclip/manifests/configmap-shell-inventory.yaml
kubectl -n paperclip-system exec -c paperclip-shell deploy/paperclip -- \
  paperclip-shell-reconcile

Use kubectl exec, not SSH — sshd scrubs the container env (no FRANK_C2_TELEGRAM_* means alerts silently fail).

Add a Tool to the Shell Sidecar

1. Add entry to the relevant section in configmap-shell-inventory.yaml (mise:, npm-global:, pipx:, cargo:).
2. Commit and push (ArgoCD syncs the ConfigMap).
3. Run paperclip-shell-reconcile on the live pod.

Database Backup

# Manual backup via Longhorn UI
# http://192.168.55.201 → Volumes → paperclip-db → Create Backup

# Or via pg_dump
kubectl exec -it -n paperclip-system deploy/paperclip-db-postgresql -- \
  pg_dump -U paperclip -d paperclip > paperclip-backup-$(date +%F).sql

Recover

Pod CrashLoopBackOff

kubectl logs -n paperclip-system -l app.kubernetes.io/name=paperclip --previous
kubectl describe pod -n paperclip-system -l app.kubernetes.io/name=paperclip | grep -A 10 Events

Common causes:

• Database not ready — Paperclip starts before PostgreSQL accepts connections.
• Missing secret — CreateContainerConfigError if a non-optional ExternalSecret fails to sync. Check kubectl get externalsecret -n paperclip-system.
• OOMOut Of MemoryWhat the kernel does when memory runs out — it kills a process by score, not by fault. A container hitting its cgroup limit is killed the same way, which is why the victim is often not the culprit. — Paperclip’s working set grew beyond 12Gi. Check kubectl top pods -n paperclip-system. If OOMKilled (exit 137), bump the memory limit in the deployment.

Multi-Attach Error on PVC

# Force-delete the stuck pod to release the volume
kubectl delete pod -n paperclip-system -l app.kubernetes.io/name=paperclip \
  --grace-period=0 --force
kubectl get pods -n paperclip-system -w

ExternalSecret Not Syncing

kubectl describe externalsecret paperclip-llm-key -n paperclip-system
kubectl get clustersecretstore infisical

Check the Infisical secret path hasn’t changed and the ClusterSecretStore is healthy. Retired secrets (paperclip-anthropic, paperclip-ghcr) can be left in place — they’re optional: true.

Shell Sidecar Tool Install Fails

cat /var/log/cont-init.d/40-shell-inventory.log

Common causes:

• Transient registry 5xx — re-run paperclip-shell-reconcile.
• mise activation gap — mise install downloads the runtime but doesn’t activate it. Run mise use -g python@3.12 node@20 rust@stable first.
• Inventory typo — fix the ConfigMap and re-reconcile.

LoadBalancer IP Not Assigned

kubectl get svc paperclip-lb -n paperclip-system
kubectl get ciliumpoolipaddress -A | grep 192.168.55.212

Missteps

Quick Reference

References

• Building Post — Paperclip
• Paperclip GitHub