
Ruflo — A Swarm Orchestrator Next to Paperclip
Layer 15 brought Paperclip onto the cluster — virtual companies with org charts, budgets, and delegation chains. Structured. Hierarchical.
This layer adds the opposite. Ruflo is the rebrand of ruvnet’s claude-flow — a swarm orchestrator where agents are cells in a hive, not employees in a company. Both run side by side on the same LiteLLM gateway.
Architecture
flowchart TD
subgraph Pod[ruflo pod]
direction TB
RUV["ruflo-server<br/>ruvocal SSR, :3000"]
RVF["RVF store<br/>5Gi PVC"]
SSHD["ruflo-shell<br/>s6 + sshd :22"]
CF["claude-flow CLI"]
VOL["PVCs<br/>shell-home 10Gi<br/>workspace 20Gi"]
end
subgraph Network
direction TB
WEB["ruflo.cluster.derio.net"]
SSH["SSH + Mosh"]
LLM["LiteLLM gateway"]
end
WEB --> RUV
RUV --> RVF
SSH --> SSHD
SSHD --> CF
SSHD --> VOL
CF -->|OPENAI_BASE_URL| LLM
RUV -->|OPENAI_BASE_URL| LLM
The three network edges, in full: the web UI is ruflo.cluster.derio.net behind Traefik with Authentik forward-auth; the shell answers SSH on 192.168.55.222:22 with Mosh on UDP 60016-60031; and both containers reach the gateway at litellm.litellm.svc:4000. The diagram keeps those labels short to stay legible.
Two ArgoCD apps: apps/ruflo-db/ and apps/ruflo/. Zero frontier-Large Language ModelA model trained to predict text, served behind a chat or completion endpoint. On Frank these run locally on the GPU node rather than against a hosted provider. provider keys in the pod — every LLM call exits through the in-cluster LiteLLM gateway.
Two Images, One Pod
Both images ship from derio-net/agent-images:
ruflo-server— thin wrapper around upstream ruvocal’s Dockerfile. Multi-stage build, SvelteKit Server-Side RenderingBuilding the HTML on the server rather than in the browser. It means the page load can reach databases and APIs, so a "simple" health check on `/` may exercise the entire stack.,node:24-slimruntime.ruflo-shell— child ofagent-shell-base. Same s6-overlay v3 init, sshd, Mosh,cont-init.d/services.dskeleton. Baked tools:claude-flow@alpha,@openai/codex.
The shell pod is the second instance of the agent-shell-base + inventory-ConfigMap pattern — marginal cost was a directory and a CI matrix entry.
The MongoDB Misdirection and the RVF Surprise
Upstream’s .env.example showed DATABASE_URL with Postgres-style connection strings, so Phase 2 deployed Bitnami PostgreSQL. First boot logged:
[RuVocal] Database: /app/db/ruvocal.rvf.jsonDATABASE_URL was being silently ignored. At the pinned SHA, ruvocal’s data layer is Ruvocal File formatThe JSON-on-disk store ruvocal keeps its state in. Worth knowing because it ignores `DATABASE_URL` entirely — the data lives in a file, so it needs a volume, not a database. — a local JSON file store. The fix: a 5Gi ReadWriteOnceA PVC access mode that lets exactly one node mount the volume read-write at a time. It is the reason a RollingUpdate deadlocks: the replacement pod cannot mount the volume until the outgoing pod releases it. PersistentVolumeClaimA Kubernetes request for durable storage. The pod names a claim and the storage layer — Longhorn on Frank — binds real disk behind it, so the data outlives the pod. mounted at /app/db/.
The RVF Shim Bug
Attaching an image in the ChatUI 500’d with TypeError: upload.once is not a function. The upstream code uses new RvfGridFSBucket() — a shim that pretends to be MongoDB’s GridFSBucket but is not faithful to the stream/cursor contract:
openUploadStreamreturned a plain object (no.once) — crash visible to users.write()didchunk.toString("base64")onArrayBuffer→ literal string"[object ArrayBuffer]"— data corruption.openDownloadStreamreturned{ toArray }(no.on/.pipe) — reading and forking broken.
Fix: one file replacing all three with real Node.js Writable/Readable/cursor implementations. Applied as a build-time patch in agent-images/ruflo-server/patches/rvf-gridfs-parity.patch.
The LiteLLM Virtual-Key Surprise
The plan originally projected OPENAI_API_KEY from the OpenRouter key. First SSR render returned 401 — LiteLLM authenticates against its own virtual key store, not the upstream provider key. Fix: provision RUFLO_LITELLM_KEY in Infisical, project as OPENAI_API_KEY.
shareProcessNamespace vs s6-overlay v3
shareProcessNamespace: true causes s6-overlay v3’s init to fail with fatal: can only run as pid 1 — every container except the first sees a non-1 PID slot. Removed from manifest. Cross-container visibility still works via shared PVC and kubectl exec.
Probes Should Not Hit SSR
The first cut probed / — which SSR-renders the model list, making it a full upstream-dependency check. Any LiteLLM flake flapped the probes. Fixed to hit /api/v2/feature-flags — served by the same Express stack with no LLM dependency.
The Inventory ConfigMap
The shell sidecar declares tools in a ConfigMap, reconciled on boot:
data:
inventory.yaml: |
mise:
- python@3.12
- node@20
- rust@stable
npm-global:
- "claude-flow@alpha"
- "@openai/codex"
pipx:
- black
- ruff
cargo:
- ripgrep
- ezaOn boot, a reconcile script computes the diff, installs/removes accordingly, writes Message of the DayThe banner printed on login. On Frank's agent shells it reports tool versions and credential state, which makes it a status display rather than decoration — and a misleading one if it only checks presence. summary.
Install Trap
npm i -g claude-flow@alpha failed with EACCES: permission denied, mkdir '/usr/lib/node_modules/'. Root cause: mise install node@20 installs the binary but does NOT activate it. npm falls through to system /usr/bin/npm targeting root-owned /usr/lib/node_modules/. Fix: mise use --global node@20 after install.
Missteps
| What Happened | Why It Was Wrong | How We Fixed It | Commit |
|---|---|---|---|
| PostgreSQL deployed but never used — RVF JSON store was the actual data layer | .env.example showed DATABASE_URL; runtime revealed RVF | Mounted 5Gi PVC at /app/db/; left PostgreSQL parked | — |
RVF shim broken for uploads — upload.once is not a function, attachments 500’d | Shim compiled against Mongo API but did not implement stream contract | Replaced with real Node.js Writable/Readable in one file | — |
LiteLLM returned 401 on first SSR render — OPENAI_API_KEY was OpenRouter key, not LiteLLM virtual key | LiteLLM authenticates against its own key store, not upstream provider | Provisioned RUFLO_LITELLM_KEY in Infisical, projected as OPENAI_API_KEY | — |
shareProcessNamespace: true crashes s6-overlay — “can only run as pid 1” on shell container | Agent-shell-base’s s6-overlay v3 init expects pid 1 of its own PID namespace | Removed shareProcessNamespace from manifest | — |
Probe flapping on SSR endpoint — / SSR-renders model list, any LiteLLM flake flips probes | Liveness probe was also a full dependency check | Changed probe to /api/v2/feature-flags | — |
npm install fails with Error, Access DeniedThe POSIX errno for "permission denied". In containers it usually means a uid mismatch against a mounted volume rather than a genuine access-control decision. — npm falls through to system binary, targets /usr/lib/node_modules/ | mise install node does not activate; npm shim resolves to system npm | mise use --global node@20 after install | — |
Recovery Path
| Symptom | Cause | Fix |
|---|---|---|
| ruvocal loads but all API calls return 401 | LiteLLM virtual key not set or revoked | Verify RUFLO_LITELLM_KEY in Infisical; restart pod |
| Shell sidecar CrashLoopBackOff | shareProcessNamespace enabled or sshd config error | Verify manifest has no shareProcessNamespace; check sshd logs |
| npm install fails in shell | Node version not activated by mise | Run mise use --global node@20 manually |
| Attachments 500 in chat UI | RVF shim still using old behavior | Verify rvf-gridfs-parity.patch applied in agent-images build |
| Web UI shows “500 Internal Server” on load | RVF PVC not mounted or /app/db/ not writable | Check ruflo-data PVC is bound and mounted at /app/db |
References
Next: Building The Frank Papers — Research Infrastructure for a Third Series
