Operating on Paperclip

Last updated 2026-10-05 ·034ef96

(Since the 2026-10 fresh start the shell sidecar and LiteLLM-backed agents described in this post’s older sections are retired; they are marked historical in place.)

This is the operational companion to Paperclip — AI Agent Orchestrator. That post covers architecture, deployment, and why the LiteLLM-backed agents are wired the way they are. This one covers health checks, database access, the shell sidecar, hiring agents on local inference, and common failure modes.

    graph LR
    subgraph ns["paperclip-system namespace"]
        pc["Paperclip Pod<br/>pure upstream, gpu-1 pinned"]
        pg["PostgreSQL Pod<br/>paperclip-db"]

        subgraph pvs["Persistent Volumes"]
            pvcData["paperclip-data<br/>10Gi RWO"]
            pvcDB["paperclip-db<br/>5Gi RWO"]
        end

        subgraph secrets["External Secrets"]
            auth["paperclip-auth<br/>→ session signing"]
            brave["paperclip-brave<br/>→ Brave Search"]
            resend["paperclip-resend<br/>→ Resend email"]
        end
    end

    subgraph infra["Infrastructure"]
        infisical["Infisical<br/>Secret Store"]
        traefik["Traefik<br/>paperclip.cluster.derio.net"]
        lb["LoadBalancer<br/>192.168.55.212:3100<br/>eTP Local"]
    end

    pc --- pvcData
    pg --- pvcDB
    auth & brave & resend -.->|"ESO sync"| infisical
    traefik --- pc
    pc --- lb
    pc --- pg
  

Update 2026-10. Paperclip runs as a pure upstream pod, its old instance wiped by the fresh start: one container, no shell sidecar, no opencode/hermes shims, no LiteLLM-routed agents. The shell and LiteLLM-agent sections below are retired and kept as history; see Fresh Start (2026-10) for the current procedures.

What Healthy Looks Like

• The Paperclip pod is 1/1 Running on gpu-1 (a single upstream container; the former paperclip-shell sidecar is retired).
• PostgreSQL pod is 2/2 Running (PostgreSQL plus the metrics exporter).
• The remaining ExternalSecrets (paperclip-auth, paperclip-brave, paperclip-resend) show SecretSynced.
• The web UI responds at http://192.168.55.212:3100.
• A Claude-adapter agent connected through Connections completes a trivial task (see Fresh Start below).

Verify

# All-in-one
kubectl get pods,pvc,externalsecret -n paperclip-system

# Web UI
curl -s -o /dev/null -w "%{http_code}" http://192.168.55.212:3100/

# Database — the postgresql container carries the password in its env
kubectl exec -n paperclip-system paperclip-db-postgresql-0 -c postgresql -- \
  sh -c 'PGPASSWORD="$POSTGRES_PASSWORD" psql -h 127.0.0.1 -U paperclip -d paperclip -c "SELECT count(*) FROM pg_tables;"'

# (historical) The shell sidecar and its SSH entry point at 192.168.55.221 were retired 2026-10;
# for filesystem access use: kubectl -n paperclip-system exec -it deploy/paperclip -c paperclip -- bash

Verify the LiteLLM-backed agent CLIs

Historical (retired 2026-10). The opencode/hermes shims and LiteLLM routing no longer exist. See Fresh start (2026-10) below.

Run these in the paperclip app container, not the shell sidecar — the adapters run in the app container, and only it carries XDG_CONFIG_HOME, HERMES_HOME and the OLLAMA_* env. Use the paths the adapters invoke:

# opencode — the provider-prefixed model shape is mandatory. Expect: ack
kubectl exec -n paperclip-system deploy/paperclip -c paperclip -- \
  /usr/local/bin/opencode run -m litellm/qwen-coder-14b 'reply with the single word ack'

# hermes — bare alias, exactly as the hermes_local adapter passes it.
# Expect a short reply followed by a session_id line.
kubectl exec -n paperclip-system deploy/paperclip -c paperclip -- \
  /paperclip/agent-bin/bin/hermes chat -q "say ack" -Q -m qwen-think-14b

A clean exit is not proof of routing. LiteLLM runs several replicas, so kubectl logs deploy/litellm reads only one of them, and the blackbox litellm_chat probe POSTs to the same endpoint all day. Grep every replica for the Paperclip pod’s IP instead:

PC_IP=$(kubectl get pod -n paperclip-system -l app.kubernetes.io/name=paperclip \
  -o jsonpath='{.items[0].status.podIP}')
for p in $(kubectl get pods -n litellm -l app.kubernetes.io/name=litellm \
    --field-selector=status.phase=Running -o name); do
  kubectl logs -n litellm "$p" -c litellm --since=5m | grep -F "$PC_IP" | grep 'POST /v1/chat/completions'
done
# Expect: INFO: <PC_IP>:<port> - "POST /v1/chat/completions HTTP/1.1" 200 OK

One line per smoke call, possibly on different replicas.

Steps

Restart Paperclip

kubectl rollout restart deployment/paperclip -n paperclip-system
kubectl get pods -n paperclip-system -w

Uses Recreate strategy (RWOReadWriteOnceA PVC access mode that lets exactly one node mount the volume read-write at a time. It is the reason a RollingUpdate deadlocks: the replacement pod cannot mount the volume until the outgoing pod releases it. PVCPersistentVolumeClaimA Kubernetes request for durable storage. The pod names a claim and the storage layer — Longhorn on Frank — binds real disk behind it, so the data outlives the pod. — rolling update would deadlock). Expect 10–30s downtime.

Reconcile Shell Inventory

Historical (retired 2026-10). The shell sidecar and its inventory were retired. See Fresh start (2026-10) below.

# After editing apps/paperclip/manifests/configmap-shell-inventory.yaml
kubectl -n paperclip-system exec -c paperclip-shell deploy/paperclip -- \
  paperclip-shell-reconcile

Use kubectl exec, not SSH — sshd scrubs the container env (no FRANK_C2_TELEGRAM_* means alerts silently fail). The reconcile handles the inventory’s mise, npm-global, pipx and cargo sections only; the paperclip-shared and uv sections that hold the agent CLIs are installed by hand (see Recover the Agent CLIs on a Cold PVC).

Add a Tool to the Shell Sidecar

Historical (retired 2026-10). The shell sidecar was retired. See Fresh start (2026-10) below.

1. Add entry to the relevant section in configmap-shell-inventory.yaml (mise:, npm-global:, pipx:, cargo:).
2. Commit and push (ArgoCD syncs the ConfigMap).
3. Run paperclip-shell-reconcile on the live pod.

Hire a LiteLLM-Backed Agent

Historical (retired 2026-10). LiteLLM-backed opencode_local/hermes_local agents were removed. See Fresh start (2026-10) below.

The two adapters want different model shapes:

The Persist session toggle defaults to on, so turning it off is a deliberate second change — see hermes agent fails from the second heartbeat for why it matters. The same hire expressed as adapter config, for the API:

{
  "adapterType": "hermes_local",
  "adapterConfig": {
    "model": "qwen-think-14b",
    "hermesCommand": "/paperclip/agent-bin/bin/hermes",
    "persistSession": false
  }
}

Leave Provider on Auto. The form only offers the adapter’s VALID_PROVIDERS, which do not include ollama-cloud, and picking a real one such as openrouter forces a cloud route with no key. Two traps apply only to adapter config sent through the API: a provider value outside VALID_PROVIDERS is silently ignored, and extraArgs must be a JSON array of strings — a plain string is dropped without a warning.

Keep LiteLLM aliases for hermes away from the prefixes the adapter maps to a cloud provider, per constants.ts at image sha-8e6edcd: claude, gpt-4, gpt-5, o1-, o3-, o4-, hermes-, glm-, moonshot, kimi and minimax. An alias starting with any of them gets a forced --provider and bypasses LiteLLM. Re-check the list after a Paperclip image bump.

Recover the Agent CLIs on a Cold PVC

Historical (retired 2026-10). The PVC-resident hermes/opencode installs no longer exist. See Fresh start (2026-10) below.

On a freshly provisioned paperclip-data PVC, both PVC-resident installs are missing: the hermes venv (with its shim and the PVC copy of uv) and the PVC copy of opencode. paperclip-shell-reconcile restores neither — the inventory’s uv and paperclip-shared sections are declarative records of these installs, not something the reconcile executes. opencode agents keep working, because the adapter uses the image-baked binary; hermes agents fail until you reinstall. The shell MOTD prints a LiteLLM-backed-agents tip on login while either install is missing.

# Is the hermes shim present, and does it resolve?
kubectl exec -n paperclip-system deploy/paperclip -c paperclip -- \
  sh -c 'ls -la /paperclip/agent-bin/bin/hermes && /paperclip/agent-bin/bin/hermes --version'

# Reinstall hermes onto the shared PVC, using the uv baked into the shell image.
# The pin matches the uv section of configmap-shell-inventory.yaml.
kubectl exec -n paperclip-system deploy/paperclip -c paperclip-shell -- sh -c '
  mkdir -p /paperclip/agent-bin/bin &&
  UV_PYTHON_INSTALL_DIR=/paperclip/agent-bin/python \
    uv venv --python 3.12 /paperclip/agent-bin/hermes-agent/venv &&
  uv pip install --python /paperclip/agent-bin/hermes-agent/venv/bin/python \
    "hermes-agent @ git+https://github.com/NousResearch/hermes-agent.git@v2026.4.16" &&
  ln -sf /paperclip/agent-bin/hermes-agent/venv/bin/hermes /paperclip/agent-bin/bin/hermes'

# Optional: restore the PVC copy of opencode. Agents do not use it; it clears the MOTD tip.
kubectl exec -n paperclip-system deploy/paperclip -c paperclip-shell -- \
  npm install --prefix /paperclip/agent-bin opencode-ai

No restart is needed: ln -sf creates the shim the adapter calls. The hermes config.yaml needs no manual step either — the hermes-init initContainer copies it into HERMES_HOME=/paperclip/agent-bin/.hermes on every pod boot, so a hand edit there is overwritten on the next restart. Change apps/paperclip/manifests/configmap-hermes.yaml instead.

Database Backup

# Manual backup via Longhorn UI
# http://192.168.55.201 → Volumes → paperclip-db → Create Backup

# Or via pg_dump
kubectl exec -n paperclip-system paperclip-db-postgresql-0 -c postgresql -- \
  sh -c 'PGPASSWORD="$POSTGRES_PASSWORD" pg_dump -h 127.0.0.1 -U paperclip -d paperclip' \
  > paperclip-backup-$(date +%F).sql

Recover

Pod CrashLoopBackOff

kubectl logs -n paperclip-system -l app.kubernetes.io/name=paperclip --previous
kubectl describe pod -n paperclip-system -l app.kubernetes.io/name=paperclip | grep -A 10 Events

Common causes:

• Database not ready — Paperclip starts before PostgreSQL accepts connections.
• Missing secret — CreateContainerConfigError if a non-optional ExternalSecret fails to sync. Check kubectl get externalsecret -n paperclip-system.
• OOMOut Of MemoryWhat the kernel does when memory runs out — it kills a process by score, not by fault. A container hitting its cgroup limit is killed the same way, which is why the victim is often not the culprit. — Paperclip’s working set grew beyond 12Gi. Check kubectl top pods -n paperclip-system. If OOMKilled (exit 137), bump the memory limit in the deployment.

Multi-Attach Error on PVC

# Force-delete the stuck pod to release the volume
kubectl delete pod -n paperclip-system -l app.kubernetes.io/name=paperclip \
  --grace-period=0 --force
kubectl get pods -n paperclip-system -w

ExternalSecret Not Syncing

kubectl get externalsecret -n paperclip-system   # paperclip-auth, paperclip-brave, paperclip-resend
kubectl describe externalsecret paperclip-auth -n paperclip-system
kubectl get clustersecretstore infisical

Check the Infisical secret path hasn’t changed and the ClusterSecretStore is healthy. Retired secrets (paperclip-anthropic, paperclip-ghcr) can be left in place — they’re optional: true.

Shell Sidecar Tool Install Fails

Historical (retired 2026-10). The shell sidecar was retired. See Fresh start (2026-10) below.

cat /var/log/cont-init.d/40-shell-inventory.log

Common causes:

• Transient registry 5xx — re-run paperclip-shell-reconcile.
• mise activation gap — mise install downloads the runtime but doesn’t activate it. Run mise use -g python@3.12 node@20 rust@stable first.
• Inventory typo — fix the ConfigMap and re-reconcile.

hermes Agent Fails from the Second Heartbeat

Historical (retired 2026-10). hermes agents were removed. See Fresh start (2026-10) below.

Symptom: a hermes_local agent’s first heartbeat succeeds; every later one fails with Session not found, exit 1, in about two seconds. The agent is eventually flagged Stranded.

Cause: an upstream session-ID round-trip bug (derio-net/paperclip#1, still open). The adapter truncates Hermes’ 22-character session ID to a 16-character display ID, Paperclip stores the truncated value and replays it as --resume, and Hermes cannot find it. The adapter’s output regex then captures the word from out of Hermes’ error message and saves {"sessionId":"from"} as the task’s session state.

Prevent it: hire with Persist session off (see Hire a LiteLLM-Backed Agent). The adapter then never passes --resume. You lose session continuity across heartbeats — fine for tool-heavy issue work, poor for long multi-turn conversations. opencode agents are unaffected.

Check whether an agent is already poisoned:

kubectl exec -n paperclip-system paperclip-db-postgresql-0 -c postgresql -- \
  sh -c 'PGPASSWORD="$POSTGRES_PASSWORD" psql -h 127.0.0.1 -U paperclip -d paperclip -c "
    SELECT agent_id, task_key, session_display_id, session_params_json, last_error
      FROM agent_task_sessions WHERE adapter_type = '"'"'hermes_local'"'"'
     ORDER BY updated_at DESC LIMIT 20;"'

Stranded hermes Agent

Historical (retired 2026-10). hermes agents were removed. See Fresh start (2026-10) below.

The agent record is fine; only its per-task session row is corrupt. Open an interactive psql session:

kubectl exec -it -n paperclip-system paperclip-db-postgresql-0 -c postgresql -- \
  sh -c 'PGPASSWORD="$POSTGRES_PASSWORD" psql -h 127.0.0.1 -U paperclip -d paperclip'

Then clear the row, using the agent_id and task_key from the detection query:

UPDATE agent_task_sessions
   SET session_params_json = NULL, session_display_id = NULL
 WHERE agent_id = '<uuid>' AND task_key = '<key>';
-- Or delete the row to start the task clean.

Then turn Persist session off on the agent before its next heartbeat, or it re-poisons on the second run.

LoadBalancer IP Not Assigned

kubectl get svc paperclip-lb -n paperclip-system
kubectl get ciliumpoolipaddress -A | grep 192.168.55.212

Fresh Start (2026-10)

The fresh start wipes the instance and rebuilds it from pure upstream (v2026.916.1, image sha-d554c47); the wipe itself is the post-merge runbook entry orch-paperclip-fresh-start-wipe. Everything above marked historical describes the retired shell and LiteLLM agents; this is how to run the fresh instance.

Bootstrap the First Admin

An authenticated instance has no users after a wipe. Print a one-time invite from inside the pod and accept it in the browser:

kubectl -n paperclip-system exec deploy/paperclip -- pnpm paperclipai auth bootstrap-ceo
# open the printed invite at https://paperclip.cluster.derio.net (Authentik in front)

Connect Claude and Verify an Agent

In a company, open Connections, connect Claude with the subscription sign-in, hire one agent with the Claude adapter and assign it a trivial task. Then prove the sign-in survives a restart:

kubectl -n paperclip-system rollout restart deploy/paperclip
# assign a second trivial task: it must complete without re-signing in
kubectl -n paperclip-system logs deploy/paperclip --tail=200 | grep -iE 'req.ip|10\.244\.'
# requests through Traefik should log a pod-network IP (the TRUST_PROXY range)

There is no shell sidecar any more: for filesystem work use kubectl -n paperclip-system exec -it deploy/paperclip -c paperclip -- bash. The live-mirror hook on agentic-stoa/companies stays paused until the reworked company’s routine exists (runbook entry orch-paperclip-resume-companies-mirror).

Missteps

Quick Reference

References

• Building Post — Paperclip
• Paperclip GitHub
• derio-net/paperclip#1 — hermes session-ID truncation