Paperclip — An AI Agent Orchestrator on Frank

Layer 11 gave the cluster a Kubernetes-native agentic control plane — Sympozium, where agents are Pods and policies are CRDsCustomResourceDefinitionThe object that teaches the Kubernetes API a new resource type. Install a CRD and the API server starts serving a kind it has never heard of, with validation and RBAC like any built-in.. Layer 15 adds a second perspective. Paperclip organises agents differently — into virtual companies with org charts, budgets, reporting lines, and governance. Where Sympozium asks “which Kubernetes primitive models this agent?”, Paperclip asks “what role would this agent have in a company?”.

Both run side by side. The cluster makes the comparison.

Update 2026-10. Paperclip runs as a pure upstream pod, and its old instance is wiped as part of the fresh start. The shell sidecar, the opencode/hermes shims and the LiteLLM-routed agents described further down are retired; those sections are kept as history and marked inline. The current state is in Fresh start (2026-10).

    flowchart LR
  subgraph Paperclip[Paperclip — paperclip-system]
    App[paperclip Deployment<br/>upstream ghcr.io/paperclipai/paperclip<br/>agent CLIs bundled]
    DB[paperclip-db StatefulSet<br/>Bitnami PostgreSQL 14.1.10<br/>5Gi Longhorn]
    PVC[paperclip-data PVC<br/>10Gi Longhorn]
    Secrets[ExternalSecrets<br/>3 from Infisical]
  end
  subgraph Access[Operator Access]
    TR[Traefik<br/>paperclip.cluster.derio.net]
    UI[LoadBalancer<br/>192.168.55.212:3100]
  end

  App -->|DATABASE_URL| DB
  App --> PVC
  App -->|envFrom| Secrets
  TR --> App
  UI --> App
  

Architecture

Two ArgoCD apps, ordered by sync-wave. The paperclip Application carries argocd.argoproj.io/sync-wave: "1", so every resource in it waits for wave 0 — the database — to report Healthy before it is applied:

    flowchart LR
  subgraph Wave0[Sync Wave 0]
    DB[paperclip-db<br/>Bitnami PostgreSQL<br/>5Gi Longhorn]
  end
  subgraph Wave1[Sync Wave 1]
    P[paperclip<br/>Deployment]
    ES[ExternalSecrets × 3<br/>from Infisical]
    PVC[paperclip-data<br/>10Gi Longhorn<br/>RWO]
    LB[LoadBalancer<br/>192.168.55.212:3100]
  end

  DB --> P
  DB --> ES
  DB --> PVC
  DB --> LB
  

Deploying the Database

The PostgreSQL mirror problem is the same as Infisical’s Layer 9: Bitnami no longer serves named image tags from Docker Hub. Override the registry to mirror.gcr.io/bitnamilegacy:

image:
  registry: mirror.gcr.io
  repository: bitnamilegacy/postgresql
metrics:
  enabled: true
  image:
    registry: mirror.gcr.io
    repository: bitnamilegacy/postgres-exporter

Deploying the Application

Paperclip does not publish its own public image at the time of writing. The project ships a Dockerfile; building is left to the operator. The cluster initially maintained a fork at ghcr.io/derio-net/paperclip, built with:

docker buildx build --platform linux/amd64 \
  -t ghcr.io/derio-net/paperclip:v0.3.1 --push .

Since v2026.428.0, Paperclip ships an upstream public image at ghcr.io/paperclipai/paperclip. The cluster now uses that directly — no fork, no imagePullSecret.

Probe Behaviour in Private Mode

Paperclip runs in authenticated mode with private exposure. In this configuration the root path / returns 403 to any request not from localhost. The kubelet issues readiness probes from the node IP, so httpGet probes against / or /api/health get 403 and the pod never becomes Ready.

The fix is a TCP socket probe — it checks that port 3100 accepts connections without making an HTTP request:

readinessProbe:
  tcpSocket:
    port: http
  periodSeconds: 10
livenessProbe:
  tcpSocket:
    port: http
  initialDelaySeconds: 30
  periodSeconds: 15

PVC Rollout Deadlock

The /paperclip data volume uses ReadWriteOnce — only one pod can hold the claim at a time. During rolling updates the Deployment creates the new pod before terminating the old one. The new pod tries to attach the PVC and stalls with Multi-Attach error.

Scale the old ReplicaSet to zero manually to release the PVC:

kubectl scale deployment paperclip -n paperclip-system --replicas=0
# wait for PVC to release
kubectl scale deployment paperclip -n paperclip-system --replicas=1

For a single-replica stateful app backed by RWO, a Recreate deployment strategy avoids this entirely — kill the old pod first, then start the new one.

Volume Permissions (fsGroup)

The Dockerfile does chown node:node /paperclip before USER node. When Longhorn mounts the PVC over /paperclip, the mounted directory is owned by root and the chown in the image never runs again. The node user (uid 1000) cannot write to it:

spec:
  securityContext:
    fsGroup: 1000

fsGroup tells Kubernetes to chown the mounted volume to gid 1000 before handing it to the container.

Secret Management

Three ExternalSecrets sync from Infisical, all consumed via envFrom (a fourth, paperclip-llm-key, was retired in the 2026-10 fresh start):

The database password comes from the Bitnami chart auto-generated Secret, referenced via secretKeyRef and Kubernetes variable expansion:

env:
  - name: PG_PASSWORD
    valueFrom:
      secretKeyRef:
        name: paperclip-db-postgresql
        key: password
  - name: DATABASE_URL
    value: "postgres://paperclip:$(PG_PASSWORD)@paperclip-db-postgresql.paperclip-system.svc:5432/paperclip"

Memory Tuning and the Move to gpu-1

The original Deployment had requests.memory: 256Mi and limits.memory: 1Gi. This guess was wrong twice.

Round 1 (1Gi → 2Gi): After GEMINI_API_KEY was added, the container started OOMKilling every five minutes. The Google AI SDK appears to eagerly init when its env var is present. Bumping the limit to 2Gi got the pod through boot.

Round 2 (2Gi → 12Gi, on gpu-1): Two hours later, OOMKilled again under load. The core-zone mini nodes (control plane + dozens of services) did not have 12Gi to spare. gpu-1 was at ~20% of 128GB. Paperclip moved:

nodeSelector:
  kubernetes.io/hostname: gpu-1
tolerations:
  - key: nvidia.com/gpu
    effect: NoSchedule
resources:
  requests:
    memory: 512Mi
    cpu: 250m
  limits:
    memory: 12Gi
    cpu: "1"

Paperclip does not request a GPU. gpu-1 is the cluster’s biggest CPU/RAM box — the “anything that needs more than 64GB” node. The nvidia.com/gpu:NoSchedule toleration is defensive: the GPU operator can re-assert the taint during driver validation, and any non-GPU workload pinned to gpu-1 without the toleration would be evicted on the spot.

Shell Sidecar

Historical (retired 2026-10). The paperclip-shell sidecar, its PVC, ConfigMaps and the 192.168.55.221 LoadBalancer no longer exist. See Fresh start (2026-10) below.

After weeks of production use, the friction with kubectl exec got hard to ignore — lost tmux state on disconnect, no ~/.ssh/config entry, no mosh over flaky connections. The instinct was to install sshd into the upstream Paperclip container. We deliberately rejected that: forking the image to add sshd puts us back on the upstream-rebase treadmill.

The answer is a separate sibling container in the same Pod:

    flowchart LR
  subgraph Pod[paperclip Deployment Pod]
    PC[paperclip container<br/>ghcr.io/paperclipai/paperclip<br/>port 3100]
    PSH[paperclip-shell container<br/>ghcr.io/derio-net/paperclip-shell<br/>port 22 + mosh UDP]
    PVC[paperclip-data PVC<br/>10Gi Longhorn<br/>shared RW]
    SHOME[paperclip-shell-home PVC<br/>20Gi Longhorn<br/>RWO]
  end
  subgraph Network
    LB1[LB 192.168.55.212:3100<br/>Paperclip API]
    LB2[LB 192.168.55.221:22<br/>SSH + Mosh]
  end

  PC -->|envFrom| Secrets
  PC -->|mount /paperclip| PVC
  PSH -->|mount /paperclip| PVC
  PSH -->|mount /home/agent| SHOME
  LB1 --> PC
  LB2 --> PSH
  

The upstream container is bit-identical — same image, same env, same probes. The shell sidecar runs alongside, sharing the paperclip-data PVC at /paperclip and exposing SSH + Mosh on a separate LoadBalancer IP 192.168.55.221.

Three-Layer Install Model

Layer 1 is the image — ghcr.io/derio-net/paperclip-shell, a thin extension of agent-shell-base.

Layer 2 is the ConfigMap. On every container boot, cont-init.d/40-shell-inventory reads a YAML tool inventory, queries each manager (mise, npm-global, pipx, cargo), computes the diff, and converges. Idempotent — sub-second no-op when nothing changed.

Layer 3 is the escape hatch. SSH in, install something ad-hoc, decide later if it earns a slot in the inventory. State lives on paperclip-shell-home (20Gi RWOReadWriteOnceA PVC access mode that lets exactly one node mount the volume read-write at a time. It is the reason a RollingUpdate deadlocks: the replacement pod cannot mount the volume until the outgoing pod releases it. PVC at /home/agent), so it survives pod restarts.

Fail-Open with Telegram Alerting

The installer fails open: on any non-zero exit, it fires a Telegram message via @agent_zero_cc_bot (reusing FRANK_C2_TELEGRAM_BOT_TOKEN / FRANK_C2_TELEGRAM_CHAT_ID from Infisical). The MOTDMessage of the DayThe banner printed on login. On Frank's agent shells it reports tool versions and credential state, which makes it a status display rather than decoration — and a misleading one if it only checks presence. on next login shows the failure summary. Three visibility layers:

1. kubectl logs paperclip -c paperclip-shell — full installer output
2. MOTD on SSH login — last-reconcile summary
3. Telegram message — within seconds of pod boot

Layer 3 is the load-bearing one. We do not notice (2) unless we SSH in. Layer 3 interrupts.

Hiring Agents on Local Inference

Historical (retired 2026-10). The opencode/hermes shims, paperclip-llm-key and the LiteLLM-routed agents were removed; agents now use the CLIs bundled in the upstream image. See Fresh start (2026-10) below.

The shell sidecar gave the operator a place to live. The next question was whether the agents Paperclip hires could run on the cluster’s own inference instead of a cloud provider. Paperclip ships adapters for opencode (opencode_local) and Hermes (hermes_local). Both speak an OpenAI-shaped API, and so does LiteLLM (litellm.litellm.svc:4000, fronting Ollama on gpu-1).

On paper that is three steps: install the CLI, point it at LiteLLM, name the model. In practice the two CLIs disagree about where config ends and state begins, so each needs a different mounting strategy.

opencode — config for the binary that wins on PATH

The Paperclip image bakes opencode at /usr/local/bin/opencode. The shell inventory also installs opencode-ai onto the shared PVC at /paperclip/agent-bin/node_modules/.bin/opencode, but the container PATH appends the PVC directories after the image ones — deliberately, so image-baked binaries win. Check which binary the adapter gets:

kubectl exec -n paperclip-system deploy/paperclip -c paperclip -- sh -c \
  'which opencode; /usr/local/bin/opencode --version; /paperclip/agent-bin/node_modules/.bin/opencode --version'

On 2026-09-13 that printed 1.18.23 for the image binary and 1.15.3 for the PVC copy. When this wiring was built in May the image binary was the older of the two; upstream image bumps have since overtaken the PVC install, which is now a stale leftover rather than a newer fallback. Pinning config to one path would have broken on that flip. Wiring it to the binary that wins does not.

So the config goes wherever the winning binary reads it. opencode reads $XDG_CONFIG_HOME/opencode/opencode.json, and the Deployment points that at a read-only ConfigMap:

env:
  - name: XDG_CONFIG_HOME
    value: /etc/paperclip/opencode-base
volumeMounts:
  - name: opencode-config                 # ConfigMap paperclip-opencode
    mountPath: /etc/paperclip/opencode-base/opencode/opencode.json
    subPath: opencode.json

The ConfigMap holds a provider.litellm block (@ai-sdk/openai-compatible, baseURL: http://litellm.litellm.svc:4000/v1, apiKey: "{env:LITELLM_API_KEY}") plus the model aliases LiteLLM serves.

The adapter rewrites opencode’s runtime config on every run, which raised the obvious fear that it would clobber the provider block. It does not: packages/adapters/opencode-local/src/server/runtime-config.ts copies the source config directory with fs.cp and then merges only the permission key over the existing config. The provider block survives untouched — re-checked against image sha-8e6edcd.

hermes — a writable home seeded on every boot

Hermes wanted the opposite of a read-only mount. Hermes v0.10.0 has no config/state split: ensure_hermes_home() creates sessions/, state.db, logs/, memories/ and auth.json under $HERMES_HOME, and no key points the state anywhere else. A read-only ConfigMap at HERMES_HOME fails on the first write.

So HERMES_HOME=/paperclip/agent-bin/.hermes lives on the writable PVC, and the template ships as a ConfigMap that an initContainer copies in on every pod boot:

initContainers:
  - name: hermes-init
    command: ["sh", "-c"]
    args:
      - |
        mkdir -p /paperclip/agent-bin/.hermes /paperclip/agent-bin/bin
        cp /etc/paperclip/hermes-template/config.yaml /paperclip/agent-bin/.hermes/config.yaml
        if [ -e /paperclip/agent-bin/hermes-agent/venv/bin/hermes ]; then
          rm -f /paperclip/agent-bin/bin/hermes
          ln -s /paperclip/agent-bin/hermes-agent/venv/bin/hermes /paperclip/agent-bin/bin/hermes
        fi

An initContainer rather than the shell sidecar’s reconcile, because the sidecar may not have started before the app container’s first hermes call — an initContainer always finishes before any app container starts. The seeded config.yaml is one line that matters:

model: "ollama-cloud/qwen-think-14b"

Hermes’ built-in ollama-cloud provider reads OLLAMA_BASE_URL and OLLAMA_API_KEY from the container env, and the Deployment points both at LiteLLM (http://litellm.litellm.svc:4000/v1 and the LITELLM_API_KEY secret).

Two model shapes through one gateway

Both CLIs reach the same LiteLLM, but the hire payload names the model differently, and the difference is load-bearing:

The hermes half works for a narrower reason than it first appears. The adapter passes -m <model> and then always adds --provider <resolved> unless the resolved provider is auto. Resolution walks a priority chain in packages/adapters/hermes/src/server/detect-model.ts: an explicit provider from the hire payload (only if it is in VALID_PROVIDERS), then the Hermes config file (only if its model matches the requested one), then a model-name prefix table, then auto.

For a bare qwen-think-14b the prefix table entry ["qwen", "auto"] wins, so no --provider flag is passed and Hermes falls back to the default provider its config.yaml implies — ollama-cloud, i.e. LiteLLM. Confirm the bare invocation routes, using the shim path the adapter invokes:

kubectl exec -n paperclip-system deploy/paperclip -c paperclip -- \
  /paperclip/agent-bin/bin/hermes chat -q "say ack" -Q -m qwen-think-14b

The same table sends other alias families somewhere else entirely. Per packages/adapters/hermes/src/shared/constants.ts at image sha-8e6edcd, matching is startsWith on the model name (after any provider/ part is stripped), and these prefixes force a cloud provider: claude → anthropic; gpt-4, o1-, o3-, o4- → openai-codex; gpt-5 → copilot; hermes- → nous; glm- → zai; moonshot, kimi → kimi-coding; minimax → minimax. A LiteLLM alias starting with any of them gets a forced --provider and bypasses the gateway. qwen, mistral, llama, deepseek and gemini map to auto, and gemma matches nothing and also falls through to auto. Name local aliases accordingly, and re-read the table after a Paperclip image bump.

Missteps

Recovery Path

What Transfers

Wiring two agent CLIs to one gateway taught three things that apply well beyond Paperclip:

• Find where a tool splits config from state before you choose a mount. opencode separates them, so a read-only ConfigMap works. Hermes keeps both under one home directory, so it needs a writable volume that an initContainer seeds on every boot. The same trick applied to both tools would have failed one of them.
• Wire configuration to the binary that actually runs, not to a version you pinned once. The image-baked opencode and the PVC copy swapped which one was newer within four months. Config keyed to PATH resolution survived that; config keyed to a path would not have.
• Probe the caller’s invocation, not the convenient one. The hand-run --provider probe passed while the adapter’s bare -m call was the path that failed. Read the adapter’s argument builder, then smoke-test with exactly those arguments.

Fresh start (2026-10)

The fresh start wipes the Paperclip instance and rebuilds it from nothing, as a post-merge operator step (runbook orch-paperclip-fresh-start-wipe). Every company, agent, user, routine and run from before is discarded (no backup, by choice), and the database and paperclip-data volumes are recreated empty.

Why: the instance had accreted a Frank-specific layer on top of upstream — a shell sidecar, PVC-resident opencode and hermes installs, adapter configs pointing at LiteLLM, and agents hired on top of those. Upstream had meanwhile started bundling the agent CLIs (claude-code, codex, opencode, gemini, kimi) and gained a Connections page for signing in with a subscription, so most of the layer was carrying weight upstream now carries itself.

What it runs now:

• One container, no init containers: ghcr.io/paperclipai/paperclip:sha-d554c47, which is upstream release v2026.916.1. Nothing is layered onto the image.
• No shell, no shims: paperclip-shell, its home PVC, SSH keys, ConfigMaps and the 192.168.55.221 LoadBalancer are gone, along with the hermes-init initContainer, PATH/XDG_CONFIG_HOME/HERMES_HOME overrides and paperclip-llm-key.
• Trusted proxies declared: TRUST_PROXY: "loopback,10.244.0.0/16" — loopback plus the cluster pod CIDR, which is where Traefik reaches the pod from. uniquelocal was rejected because it trusts the whole LAN. The 192.168.55.212 LoadBalancer runs externalTrafficPolicy: Local, so a LAN client keeps its own source IP instead of arriving SNATed to an in-CIDR address — without that, the pod-CIDR rule would quietly trust the LAN after all.
• Upstream announcements off: PAPERCLIP_ANNOUNCEMENTS_ENABLED: "false".
• Companies mirror paused: the Gitea hook on agentic-stoa/companies is deliberately inactive until a reworked company and its routine exist.

Bootstrapping on the fresh instance (operator-run; the live sequence is the orch-paperclip-* runbook entries):

kubectl -n paperclip-system exec deploy/paperclip -- pnpm paperclipai auth bootstrap-ceo
# open the printed invite at https://paperclip.cluster.derio.net, create the first admin,
# then Connections -> connect Claude and hire a Claude-adapter agent.

References

• Paperclip — Agent orchestrator
• Operating on Paperclip — smoke tests, hiring, and stranded-agent recovery
• apps/paperclip/manifests/ — Deployment, ConfigMap (TRUST_PROXY), LoadBalancer, PVC and ExternalSecrets
• scripts/tests/test_paperclip_pure_upstream.py — CI guard that the pod stays pure upstream
• (historical, removed in the 2026-10 fresh start) configmap-shell-inventory.yaml, configmap-opencode.yaml, configmap-hermes.yaml, pvc-shell-home.yaml — recoverable from git history before the fresh-start PR
• derio-net/paperclip#1 — hermes session-ID truncation

Next: Media Generation — ComfyUI and Stable Diffusion