Skip to content
Documentation
The Frank Papers
References

References

The cross-series bibliography for the Frank Papers. Every primary source cited in any Paper appears here exactly once, grouped by source type. Per-paper §8 sections remain the primary citation surface; this index exists for cross-paper navigation.

vendor-docs

  • vendor-docs Amazon VPC CNI — Pod networking
    The Amazon VPC CNI plugin for Kubernetes is the networking plugin for pod networking in Amazon EKS clusters. The plugin is responsible for allocating VPC IP addresses to Kubernetes nodes and configuring the necessary networking for pods on each node.
    Each pod receives an IP address from the VPC CIDR and is fully routable on the VPC.
    Cited in:
    • Paper 3 — eBPF Networking Without a Service Mesh — The cloud-managed leaf of the §6 decision tree. Vendor's own description that VPC CNI delegates the data plane to AWS — explicit support for the §6 'AWS-native → VPC CNI' branch and the counter-argument that for that geometry Cilium is overkill.
  • vendor-docs Argo CD — Architecture
    Argo CD is implemented as a Kubernetes controller which continuously monitors running applications and compares the current, live state against the desired target state (as specified in the Git repo).
    The application controller is a Kubernetes controller which continuously monitors running applications and compares the current, live state against the desired target state. It detects OutOfSync application state and optionally takes corrective action.
    Cited in:
    • Paper 6 — GitOps at Small Scale — Vendor's own articulation of ArgoCD's reconciliation model — pull-mode controller that compares declared state against live state continuously. Anchors the §3 architecture diagram for ArgoCD and underwrites the §1 capability definition.
  • vendor-docs Argo Rollouts — Canary strategy
    The canary deployment strategy is a way to roll out new versions of an application that incrementally shifts traffic to the new version.
    With Argo Rollouts, you can define a canary strategy that specifies the percentage of traffic to be sent to the new version and the duration of each step.
    Cited in:
    • Paper 14 — Progressive Delivery & the Service-Mesh Tax — Vendor's authoritative description of how Argo Rollouts' canary strategy is structured around `setWeight` steps with optional pauses and metric gates. Grounds the §3 architecture comparison in Argo's own model and underwrites the §5 description of Frank's 20→50→100 staged rollout.
  • vendor-docs Authentik — Proxy provider documentation
    In this mode, the regular expressions are matched against the Request's Path.
    In this mode, the regular expressions are matched against the Request's full URL.
    Cited in:
    • Paper 15 — Ingress, Forward-Auth, and the Service Catalog — Defines the proxy-provider modes (forward_single, forward_domain, proxy) that Frank's blueprint declares for every SSO-gated service. The forward_single vs forward_domain distinction is what determines whether one Authentik provider serves one app or a wildcard subdomain.
  • vendor-docs Authentik docs — Forward auth (Proxy provider)
    Forward auth uses your reverse proxy (like NGINX, Traefik, or Caddy) to forward authentication requests to authentik before serving the application.
    Single application mode allows you to protect a single application with a single proxy provider. Domain level mode allows you to protect multiple applications under the same domain with a single proxy provider.
    Cited in:
    • Paper 11 — Identity for a Heterogeneous Stack — Vendor's own description of the forward-auth pattern that Frank uses for every protected web UI. Defines the seam between the IdP, the ingress controller (Traefik) and the upstream application, which §3 needs to render as a flowchart.
  • vendor-docs Cilium — Component Overview
    Cilium agent (cilium-agent) runs on each node in the cluster. At a high-level, the agent accepts configuration via Kubernetes or APIs that describes networking, service load-balancing, network policies, and visibility & monitoring requirements.
    The Cilium agent compiles configuration into eBPF programs that are loaded into the kernel and runs the related management tasks like updating BPF maps.
    Cited in:
    • Paper 3 — eBPF Networking Without a Service Mesh — Vendor's authoritative description of Cilium's data-plane model — eBPF programs compiled by the per-node agent and loaded into the kernel via BPF maps. Grounds the §3 Cilium architecture diagram and the §5 description of Frank's `kubeProxyReplacement: true` configuration.
  • vendor-docs Cilium 1.17 release blog (Isovalent)
    Cilium 1.17 introduces Service Mesh enhancements with L7 traffic management and BGP advancements for L3 routing.
    Cilium continues to push eBPF as the foundational dataplane for cloud-native networking, with new features in routing, load balancing, and observability.
    Cited in:
    • Paper 3 — eBPF Networking Without a Service Mesh — The version Frank actually runs. Anchors the §1 / §3 claims about feature scope and the §5 LBIPAM and MixedProtocolLBService observations to a specific release rather than a generic 'Cilium does this'.
  • vendor-docs Claude Code — Security and isolation
    Claude Code includes safeguards to help prevent destructive operations, but users are ultimately responsible for the code Claude writes and the commands it runs.
    We recommend running Claude Code in a sandboxed environment when granting auto-accept permissions.
    Cited in:
    • Paper 17 — Agentic Orchestration & Safe Agent Workstations — Vendor's own statement on why a safe workstation matters — Anthropic explicitly recommends sandboxing for auto-accept flows. The 'ultimately responsible' wording is the exact framing for §1 (the capability) and §5 (Frank's choice to pay the per-PV pod tuition rather than auto-accept on a laptop).
  • vendor-docs Cloudflare Tunnel — Connect networks (developer docs)
    cloudflared initiates an outbound connection through your firewall from the origin to the Cloudflare global network.
    Once the connection is established, traffic flows in both directions over the tunnel between your origin and Cloudflare.
    configure your firewall to allow only these outbound connections and block all inbound traffic, effectively blocking access to your origin from anything other than Cloudflare.
    Cited in:
    • Paper 20 — Edge Clusters & Public Exposure — Vendor's authoritative statement of the outbound-tunnel architecture: the cloudflared daemon dials out to Cloudflare's edge, Cloudflare's edge terminates TLS and serves the public hostname, and the origin needs zero inbound firewall holes. This is the load-bearing claim that distinguishes Cloudflare Tunnel from every 'run your own edge node' option in §3 and underwrites the §2 quadrant axis 'who owns the edge node'.
  • vendor-docs Cluster API — Concepts (MachineDeployments, MachineSets, Machines)
    A 'machine' is the declarative spec for an infrastructure component hosting a Kubernetes Node (for example, a VM).
    A MachineDeployment provides declarative updates for Machines and MachineSets.
    Cited in:
    • Paper 1 — Heterogeneous Hardware as a Design Choice — The other major CRD-based heterogeneity surface. Cluster API's MachineDeployment-per-class pattern is what the bare-metal world converges on as it grows past the talosctl-per-node regime. §3 contrasts it with Frank's per-node Talos patches and Karpenter's NodePool CRDs as three points on the same 'declare your node classes' axis.
  • vendor-docs Coder.com — Platform documentation
    Coder is a self-hosted cloud development environment platform that runs developer workspaces in your infrastructure.
    Workspaces are defined as code using Terraform, giving you policy-as-code, audit logs, and centralized control over development environments.
    Cited in:
    • Paper 17 — Agentic Orchestration & Safe Agent Workstations — Definitive statement of the managed-platform value proposition. Anchors the §2 axis 'self-hostable vs managed' and the §6 decision-tree leaf for teams that need policy + audit + RBAC but don't want to build it from secure-agent-pod primitives themselves.
  • vendor-docs Contour — Ingress controller built on Envoy
    Contour is an Ingress controller for Kubernetes that works by deploying the Envoy proxy as a reverse proxy and load balancer.
    Cited in:
  • vendor-docs Development Containers Specification (containers.dev)
    A development container is a running container with a well-defined tool/runtime stack and its prerequisites.
    The development container specification (devcontainer.json) is a metadata format used to configure development containers.
    Cited in:
    • Paper 17 — Agentic Orchestration & Safe Agent Workstations — The upstream spec that Codespaces, GitPod, and VS Code Remote-Containers all implement. Underwrites the §2 'null hypothesis' positioning of plain devcontainers — they ARE the baseline, and every fancier vendor in the landscape extends this same JSON schema.
  • vendor-docs Dex docs — A federated OpenID Connect provider
    Dex is an identity service that uses OpenID Connect to drive authentication for other apps. Dex acts as a portal to other identity providers through 'connectors.'
    This lets dex defer authentication to LDAP servers, SAML providers, or established identity providers like GitHub, Google, and Active Directory.
    Cited in:
    • Paper 11 — Identity for a Heterogeneous Stack — Defines the 'federation-only' position in the §2 landscape — Dex deliberately does not store users, only proxies authentication. Explains why projects that adopted Dex (ArgoCD's bundled provider) end up adding a 'real' IdP behind it as soon as they want password policies, MFA enrolment, or admin UI.
  • vendor-docs Do you need Omni? — Sidero Labs
    Omni handles the lifecycle of Talos Linux machines, provides unified access to the Talos and Kubernetes API tied to the identity provider of your choice, and provides a UI for cluster management and operations.
    From initial machine registration through cluster creation, day-to-day operations, and rolling upgrades, everything is handled through a single interface.
    Cited in:
    • Paper 0 — Why Run Your Own Cluster in 2026? — Vendor's own articulation of where the managed homelab-as-code seam sits. Defines what 'managed control-plane on owned hardware' actually means in practice and what it costs in operational autonomy.
  • vendor-docs External Secrets Operator — ClusterSecretStore API
    The ClusterSecretStore is a cluster scoped SecretStore that can be referenced by all ExternalSecrets from all namespaces.
    It defines how the operator authenticates against the secret provider — for Infisical, via universalAuthCredentials referencing a clientId and clientSecret stored in Kubernetes Secrets.
    Cited in:
  • vendor-docs External Secrets Operator — GitHub repo
    External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, Akeyless, CyberArk Conjur, Pulumi ESC, and many more.
    The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.
    Cited in:
  • vendor-docs Fedora CoreOS — Documentation
    Fedora CoreOS is an automatically-updating, minimal operating system for running containerized workloads securely and at scale.
    It is the successor to CoreOS Container Linux and Atomic Host, combining the provisioning tools, automatic update model, and philosophy of CoreOS Container Linux with the packaging technology, OCI support, and SELinux security of Atomic Host.
    Provisioning is performed by Ignition, which runs only on the first boot.
    Cited in:
    • Paper 2 — Immutable OS & Declarative Machines — Vendor's own description of the Fedora CoreOS model — first-boot Ignition + rpm-ostree A/B updates. Anchors the Fedora CoreOS architecture diagram and supports the quadrant placement (immutable + Ignition-bootstrapped, not API-driven the way Talos and Bottlerocket are).
  • vendor-docs Flagger — How it works
    Flagger takes a Kubernetes deployment and optionally a horizontal pod autoscaler (HPA), then creates a series of objects (Kubernetes deployments, ClusterIP services, and a service mesh or ingress controller routing configuration) to drive the canary analysis and promotion.
    Flagger requires a service mesh or an ingress controller for traffic routing.
    Cited in:
    • Paper 14 — Progressive Delivery & the Service-Mesh Tax — Definitive statement that Flagger's design assumes a traffic router (mesh or ingress). This is the architectural fork that distinguishes Flagger from Argo Rollouts and the load-bearing claim behind the §6 decision-tree branch on 'service mesh already deployed?'.
  • vendor-docs Flux — Core Concepts
    Flux is a tool for keeping Kubernetes clusters in sync with sources of configuration (like Git repositories and OCI artifacts), and automating updates to configuration when there is new code to deploy.
    Flux is constructed with the GitOps Toolkit, a set of composable APIs and specialized tools for building Continuous Delivery on top of Kubernetes.
    Cited in:
    • Paper 6 — GitOps at Small Scale — Definitive description of the GitOps Toolkit composition model — Flux is unbundled by design, where ArgoCD ships as a single application. Anchors the §2 axis (unbundled vs opinionated) and the §3 Flux architecture diagram.
  • vendor-docs gethomepage.dev — Services configuration
    Services are configured inside the services.yaml file. You can have any number of groups, and any number of services per group.
    Cited in:
    • Paper 15 — Ingress, Forward-Auth, and the Service Catalog — Defines Frank's catalogue shape. The static-config model (ConfigMap → /app/config/services.yaml) is the simplest possible service catalogue — no API server, no scrape loop, no auto-discovery. The blast radius of a typo is one wrong tile, not a broken dashboard.
  • vendor-docs Gitea — Webhooks documentation
    Gitea supports webhooks for repository events. To set up webhooks, in your repository, click on Settings → Webhooks → Add Webhook.
    Gitea webhooks send the following headers: X-Gitea-Event, X-Gitea-Delivery, X-Hub-Signature, X-Hub-Signature-256.
    Cited in:
    • Paper 13 — Self-Hosted CI/CD on a Homelab — Vendor confirmation that Gitea sends `X-Gitea-Event` (not `X-GitHub-Event`). This is the single header difference that makes the Tekton `github` interceptor drop every Gitea webhook silently. Cited in §5's scar callout.
  • vendor-docs Grafana Alerting — Documentation
    Grafana Alerting is a unified alerting system, where alerts are managed in a single location for all your data sources.
    Alert rules: Define the conditions that trigger alerts. Each rule consists of one or more queries and expressions, a condition that needs to be met, and an interval at which the rule should be evaluated.
    Cited in:
    • Paper 7 — The Observability Stack, Honestly — Underwrites §3 (alerting architecture diagram) and §5 (the 12.x SSE alert rules need a 3-step A→B→C — the classic-condition format that worked in Grafana 11 breaks with sse.parseError on 12.x). Vendor's authoritative description of the alert-rule shape Frank's alerts must conform to.
  • vendor-docs Headscale README (juanfont/headscale on GitHub)
    An open source, self-hosted implementation of the Tailscale control server.
    Headscale aims to implement a self-hosted, open source alternative to the Tailscale control server.
    The control server works as an exchange point of Wireguard public keys for the nodes in the Tailscale network.
    Cited in:
    • Paper 20 — Edge Clusters & Public Exposure — Definitive description of what Headscale actually is: a drop-in replacement for the Tailscale control server, which is the *coordination* layer (key exchange, ACLs, DNS) — not the data plane. The Tailscale clients themselves are unchanged. This is the load-bearing claim behind Frank's §5 architecture: Headscale on Hop is the control plane, the Tailscale DaemonSet on every Frank node is the data plane, and the WireGuard tunnels they establish are peer-to-peer.
  • vendor-docs How Tailscale Works (Tailscale engineering blog)
    The node contacts the coordination server and leaves its public key and a note about where that node can currently be found, and what domain it's in.
    Remember that Tailscale private keys never leave the node where they were generated; that means there is never a way for a DERP server to decrypt your traffic.
    Tailscale uses several very advanced techniques, based on the Internet STUN and ICE standards, to make these connections work even though you wouldn't think it should be possible.
    Cited in:
    • Paper 20 — Edge Clusters & Public Exposure — The canonical architectural writeup of Tailscale's three-part design: coordination server (control plane), DERP relays (fallback data plane when NAT traversal fails), and direct WireGuard tunnels between peers. Grounds the §3 mesh-overlay architecture for both Tailscale Funnel and Frank's Headscale + Hop setup, and justifies the §2 claim that 'mesh-overlay vs hub-and-spoke' is a real architectural axis, not a marketing distinction.
  • vendor-docs Ignition (CoreOS) — Specification
    Ignition is a provisioning utility designed specifically for CoreOS-like OSes that reads its configuration from the system's firmware and applies it during the initramfs.
    Ignition runs only once on the first boot.
    Cited in:
    • Paper 2 — Immutable OS & Declarative Machines — The Ignition spec underpins both Fedora CoreOS and Flatcar. Used to explain the shared 'image-based, Ignition-bootstrapped' family of immutable OSes that sits between Talos and the mutable baseline.
  • vendor-docs Infisical — Self-hosting overview
    Infisical is an open-source platform for managing secrets, certificates, and configurations across teams and infrastructure.
    You can self-host Infisical on your own infrastructure using Docker, Kubernetes, or any other container-orchestration platform.
    Cited in:
  • vendor-docs inlets.dev — Self-hosted HTTP and TCP tunnels
    Self-hosted HTTP and TCP tunnels with full control and privacy. Expose endpoints publicly, or tunnel customer services back to your SaaS.
    inlets exposes HTTP and TCP services without dragging whole subnets, hosts or teams into a VPN.
    inlets focuses on exposing specific services, self-hosting the data plane, and avoiding SaaS tunnel limits.
    Cited in:
    • Paper 20 — Edge Clusters & Public Exposure — A fourth-vendor perspective: a tool that sits between Cloudflare Tunnel (zero infra) and Headscale (full mesh) — you still need a VPS, but you run only the tunnel daemon on it, not a coordination server or a cluster. Underwrites the §2 capability matrix row 'data-plane ownership' and provides the bridge in the §6 decision tree between the 'tunnel SaaS' branch and the 'edge cluster' branch.
  • vendor-docs Intel Resource Drivers for Kubernetes
    Intel resource drivers for Kubernetes is an alternative for Intel device plugins, facilitating workload offloading by providing accelerator access on Kubernetes cluster worker nodes.
    The resource drivers are based on Dynamic Resource Allocation (DRA) framework in Kubernetes
    Cited in:
    • Paper 5 — GPU Scheduling for Mixed Workloads — Vendors authoritative positioning of the DRA replacement path for Intel hardware. Underwrites the §3 architecture diagram for the Intel iGPUs on Franks mini-1/2/3 and the §5 case-study claim that DRA is the path Intel has taken first.
  • vendor-docs Karpenter docs — NodePools and NodeClasses
    NodePools set constraints on the nodes that can be created by Karpenter and the pods that can run on those nodes.
    It is recommended to create NodePools that are mutually exclusive. So no Pod should match multiple NodePools. If multiple NodePools are matched, Karpenter will use the NodePool with the highest weight.
    Karpenter recommends using as few NodePools as possible to keep your configuration simple and manageable.
    Cited in:
    • Paper 1 — Heterogeneous Hardware as a Design Choice — Karpenter's own doc explicitly couples heterogeneity to operator cost — 'as few NodePools as possible'. That's the cloud-managed-fleet analogue of Frank's tax-on-classes argument; §4 cites this directly when characterising the per-node-class operator cost that the literature implies but does not measure.
  • vendor-docs Keycloak docs — Identity Brokering
    Keycloak is an open source software product to allow single sign-on with identity and access management aimed at modern applications and services.
    An Identity Broker is an intermediary service that connects multiple service providers with different identity providers. As an intermediary service, the identity broker is responsible for creating a trust relationship with an external identity provider in order to use its identities to access internal services exposed by service providers.
    Cited in:
    • Paper 11 — Identity for a Heterogeneous Stack — Defines how a heavy-enterprise IdP positions itself against the smaller open-source IdPs. Supplies the §2 vendor-landscape claim that Keycloak's centre of gravity is brokering / federation, not embedded forward-auth.
  • vendor-docs Kubernetes — Dynamic Resource Allocation
    DRA is a Kubernetes feature that lets you request and share resources among Pods. These resources are often attached devices like hardware accelerators.
    These benefits provide significant improvements in the device allocation workflow when compared to device plugins, which require per-container device requests, dont support device sharing, and dont support expression-based device filtering.
    Cited in:
    • Paper 5 — GPU Scheduling for Mixed Workloads — Upstream Kubernetes definition of DRA and the explicit contrast with the legacy device-plugin model. Anchors the §2 axis device plugin (legacy) ↔ DRA (KEP-4381) and the §7 roadmap claim that DRA is eating the device-plugin API.
  • vendor-docs Kubernetes — Ingress Controllers (project recommendation)
    The Kubernetes project recommends using Gateway instead of Ingress. The Ingress API has been frozen.
    In order for an Ingress to work in your cluster, there must be an ingress controller running.
    Cited in:
  • vendor-docs Kubernetes docs — Assigning Pods to Nodes (nodeSelector, taints, tolerations, affinity)
    You can constrain a Pod so that it is restricted to run on particular node(s), or to prefer to run on particular nodes.
    Taints are the opposite — they allow a node to repel a set of pods. Tolerations are applied to pods. Tolerations allow the scheduler to schedule pods with matching taints. Tolerations allow scheduling but don't guarantee scheduling.
    Cited in:
    • Paper 1 — Heterogeneous Hardware as a Design Choice — The canonical vocabulary for expressing 'these pods run on these node classes' — every heterogeneous fleet uses this surface, whether the labels come from per-node Talos patches (Frank), Karpenter NodeClaim provisioning, or Cluster API MachineDeployment template inheritance. §1 and §3 anchor on this primitive.
  • vendor-docs LiteLLM virtual keys, budgets & rate limits
    Track Spend, set budgets and create virtual keys for the proxy.
    Maintain different rate limits per user/key/team.
    Prometheus Metrics — Available on Enterprise. Track LiteLLM usage metrics, with: Total spend on a per-user, model, team basis. Token usage per user, model, team. Rate limit errors.
    Cited in:
    • Paper 10 — Self-Hosted Inference & the LLM Gateway Pattern — Vendor's articulation of the gateway pattern as Frank uses it: virtual keys, cost tracking, OpenAI-compatible routing. Critically, the page itself documents that Prometheus metrics are an Enterprise feature — the load-bearing 'marketing-vs-reality' gap for §5 carries a direct citation, not just a Frank gotcha.
  • vendor-docs Longhorn — Architecture and Concepts
    Longhorn creates a dedicated storage controller for each volume and synchronously replicates the volume across multiple replicas stored on multiple nodes.
    The default replica count is 3. Each replica is placed on a different node based on the configured replica scheduling policy.
    Cited in:
    • Paper 4 — Distributed Storage on Bare Metal — Vendor's own articulation of Longhorn's per-volume-engine architecture. Defines the model Frank actually runs (engine pod + N replica pods per volume) and grounds the §3 architecture comparison in Longhorn's authoritative description.
  • vendor-docs Longhorn — Concepts: Backups and Secondary Storage
    A backup is a snapshot copy that is stored in a secondary storage (NFS or S3-compatible object store) outside of the Longhorn cluster. Backups are designed for the disaster recovery purpose.
    Longhorn uses a unique mechanism to construct each backup at the block level. Each backup consists of a metadata file and the data blocks. If a data block is already saved in the secondary storage, this data block will not be transferred again.
    Cited in:
    • Paper 8 — Backup & DR Without a Vendor Contract — Definitive statement that Longhorn's backup model is block-level, deduplicated against the BackupStore, and lives outside the Longhorn cluster on S3 or NFS. This is the architectural fork that distinguishes Longhorn-native from Velero+restic — the dedup is at the volume-block layer, owned by the storage driver, not bolted on by a separate backup controller. Underwrites the §2 capability matrix, the §3 Longhorn diagram, and the §5 'no extra control plane' claim.
  • vendor-docs Mozilla SOPS — README and design
    sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.
    Only the values are encrypted, the keys are still readable. That allows you to do diffs that are meaningful and preserve the structure of your file.
    Cited in:
  • vendor-docs NVIDIA GPU Operator — Overview
    The NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision GPU.
    These components include the NVIDIA drivers (to enable CUDA), Kubernetes device plugin for GPUs, the NVIDIA Container Toolkit, automatic node labeling using GFD, DCGM based monitoring and others.
    Cited in:
    • Paper 5 — GPU Scheduling for Mixed Workloads — Vendor's authoritative description of what the Operator does — five lifecycle components bundled into a single operator. Grounds the §3 NVIDIA architecture diagram and underwrites the §5 description of Frank's driver.enabled false Talos workaround (driver is the one component the Operator does NOT manage on Frank, because Talos system extensions already provide it).
  • vendor-docs NVIDIA Multi-Instance GPU (MIG) User Guide
    The Multi-Instance GPU (MIG) User Guide explains how to partition supported NVIDIA GPUs into multiple isolated instances, each with dedicated compute and memory resources.
    MIG enables efficient GPU utilization across multiple users or workloads with guaranteed performance.
    Cited in:
    • Paper 5 — GPU Scheduling for Mixed Workloads — Vendor's authoritative description of MIG — hardware-level partitioning of a single physical GPU into isolated instances. Source for the §3 MIG/MPS architecture diagram, the §4 fragmentation-cost discussion, and the §6 decision-tree leaf multi-tenant on a single big card.
  • vendor-docs oauth2-proxy — reverse-proxy authentication
    A reverse proxy and static file server that provides authentication using Providers (Google, GitHub, and others) to validate accounts by email, domain or group.
    Cited in:
    • Paper 15 — Ingress, Forward-Auth, and the Service Catalog — The non-Authentik forward-auth alternative — same architectural slot (sits between ingress and upstream), different IdP shape. Important §3 comparison point: Nginx + oauth2-proxy is the most common forward-auth stack outside Authentik shops.
  • vendor-docs Ollama README & Modelfile reference
    Get up and running with large language models locally.
    Ollama provides the simplest way to run large language models on your own hardware. It packages model weights, configuration, and data into a single Modelfile.
    OLLAMA_KEEP_ALIVE — Set the duration that models stay loaded in memory (default: 5m, e.g. '24h' to keep loaded indefinitely).
    Cited in:
    • Paper 10 — Self-Hosted Inference & the LLM Gateway Pattern — Vendor's own articulation of what Ollama actually is — a llama.cpp wrapper with a Modelfile abstraction and an OpenAI-shaped API. Pairs with the dated Frank incident in §5 where OLLAMA_KEEP_ALIVE pinned the cgroup RAM ceiling and made VRAM look like the bottleneck.
  • vendor-docs Prometheus Operator — Getting Started
    The Prometheus Operator provides Kubernetes native deployment and management of Prometheus and related monitoring components.
    ServiceMonitor, which declaratively specifies how groups of Kubernetes services should be monitored. The Operator automatically generates Prometheus scrape configuration based on the current configuration of the API objects.
    Cited in:
    • Paper 7 — The Observability Stack, Honestly — Anchors the §3 architecture diagram for the LGTM stack — defines the ServiceMonitor model Frank actually uses for in-cluster discovery and the operator-managed reconciliation loop that produces the live Prometheus scrape config.
  • vendor-docs Rancher Fleet — Multi-cluster GitOps engine (docs)
    Fleet is a container management and deployment engine designed to offer users more control on the local cluster and constant monitoring through GitOps.
    Fleet is fundamentally a set of Kubernetes custom resource definitions (CRDs) and controllers that manage GitOps for a single Kubernetes cluster or a large scale deployment of Kubernetes clusters.
    Cited in:
    • Paper 20 — Edge Clusters & Public Exposure — The enterprise heavyweight in the §2 landscape: instead of treating the edge cluster as a separate platform with its own ArgoCD (Frank's choice), Fleet promises one logical GitOps surface spanning hub + edge + cloud. Justifies the §6 decision-tree branch 'managing one cluster or ten?' — Fleet starts paying for itself somewhere north of the Frank+Hop two-cluster mark, and below that it's overhead.
  • vendor-docs restic — Design and References
    restic uses content-defined chunking and deduplication. Each file is split into variable-sized chunks and the SHA-256 hash of each chunk is used to identify it; identical chunks are stored only once.
    The repository's data is stored encrypted using AES-256 in counter mode. The encryption keys are derived from a user-provided password using scrypt.
    Cited in:
    • Paper 8 — Backup & DR Without a Vendor Contract — Vendor description of restic's content-addressed, deduplicated, encrypted-at-rest data model — the design that Velero's File-System Backup mode inherits and that determines why §4 calls out deduplication ratios as a load-bearing scale axis. Underwrites the §3 restic diagram and the §6 'offline-resilient / paranoid' decision-tree leaf.
  • vendor-docs Rook — Architecture overview
    Rook is an open source cloud-native storage orchestrator, providing the platform, framework, and support for Ceph storage to natively integrate with cloud-native environments.
    Rook automates deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management.
    Cited in:
  • vendor-docs s6-overlay — README and architecture
    s6-overlay is a series of init scripts and utilities meant to ease the creation of Docker images using s6 as a process supervisor.
    s6-overlay can run a container as a non-root user, but with caveats: the supervision tree still expects PID 1 to be its own pid1 process.
    Cited in:
    • Paper 17 — Agentic Orchestration & Safe Agent Workstations — Vendor's authoritative description of how s6-overlay v3 supervises non-root container processes. Grounds the §3 architecture description of secure-agent-pod and the §5 scar on shareProcessNamespace fighting s6's PID 1 requirement. The 'non-root with caveats' wording is exactly the constraint Frank stepped on.
  • vendor-docs Secrets Store CSI Driver — Overview
    Secrets Store CSI Driver for Kubernetes Secrets — Integrates secrets stores with Kubernetes via a CSI volume.
    The Secrets Store CSI Driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume.
    Cited in:
  • vendor-docs Tailscale Funnel docs
    Tailscale Funnel lets you route traffic from the broader internet to a local service running on a device in your Tailscale network (known as a tailnet).
    The Tailscale server running on your device receives the encrypted request from the TCP proxy. It then terminates the TLS connection and passes the decrypted request to the local service you exposed through Funnel.
    Cited in:
    • Paper 20 — Edge Clusters & Public Exposure — Vendor docs for the 'public exposure via the mesh' pattern — TLS is terminated on the device itself, not on Tailscale's edge, which is the architectural twin of Frank's choice but with Tailscale's coordination server instead of Frank's self-hosted Headscale. Source for the §3 mesh-overlay-as-edge architecture and the §6 decision-tree branch 'are you OK with vendor-controlled coordination?'.
  • vendor-docs Talos Linux — Concepts
    Talos is a modern OS designed to be secure, immutable, and minimal.
    Talos is configured exclusively through an API; there is no SSH access or interactive shell.
    Machine configuration is the source of truth: applying a new config replaces the existing one.
    Cited in:
    • Paper 2 — Immutable OS & Declarative Machines — Vendor's canonical articulation of Talos's API-only management model and the 'no SSH, no shell' posture. Defines the model Frank actually runs on all seven nodes and grounds the architecture diagram in the vendor's own authoritative description.
  • vendor-docs Talos Linux on Hetzner Cloud (Sidero Labs docs)
    Hetzner Cloud provides Talos as Public ISO with the schematic id `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515`
    Once shutdown, simply create an image (snapshot) from the console. You can now use this snapshot to run Talos on the cloud.
    Bootstrap `etcd` on the first control plane node with: `talosctl --talosconfig talosconfig bootstrap`
    Cited in:
    • Paper 20 — Edge Clusters & Public Exposure — Vendor's own deployment guide for the exact path Frank takes to stand up Hop. Justifies the §5 narrative claim that 'Hop is a real Talos cluster, not a Helm chart on an Ubuntu droplet' — same OS, same image-factory pipeline, same `talosctl bootstrap` flow as Frank. The only architectural difference is the control plane: standalone talosctl with a combined patch file instead of Omni (Omni isn't reachable from a Hetzner public IP).
  • vendor-docs Talos Linux v1.10 — Managing Kubernetes nodes via Talos machine config
    Talos provides built-in support for managing Kubernetes node labels, annotations, and taints via Talos machine configuration.
    Node labels, annotations and taints managed via Talos take precedence over the values set on the node directly. They are reconciled to the desired state on each Talos restart.
    Cited in:
    • Paper 1 — Heterogeneous Hardware as a Design Choice — The mechanism that Frank actually uses to declare its seven-node, four-hardware-class fleet — one machine-config patch per node under `patches/phase01-node-config/`. Demonstrates that the heterogeneous-bare-metal shape doesn't require Cluster API or Karpenter at small N; the OS can carry the role/zone/hardware label schema.
  • vendor-docs Tekton — Concepts (Pipelines, Tasks, TaskRuns, PipelineRuns)
    Tekton Pipelines are a Kubernetes-native, lightweight, and easy-to-use framework that allow you to build CI/CD systems.
    A Task defines a series of ordered Steps that you want to execute. Each Step runs in its own container image.
    Cited in:
    • Paper 13 — Self-Hosted CI/CD on a Homelab — Vendor's authoritative description of Tekton's pods-as-steps model — Tasks are CRDs, Steps are containers, PipelineRuns and TaskRuns are runtime objects. Grounds the §3 architecture diagram and underwrites the §1 claim that Tekton makes the pipeline itself a manifest in Git.
  • vendor-docs Tekton Triggers — EventListeners and Interceptors
    Interceptors are an extension mechanism that can be used to validate and modify the events that pass through them, including the headers and the body of the request.
    The CEL Interceptor can be used to apply CEL expressions to the request body, headers, or other parts of the request.
    Cited in:
    • Paper 13 — Self-Hosted CI/CD on a Homelab — Definitive description of Tekton Triggers' interceptor mechanism — the `cel` interceptor is what makes Gitea-webhook-to-Tekton-pipeline wiring possible (matching `X-Gitea-Event`, not `X-GitHub-Event`). Load-bearing source for the §5 scar about the github interceptor silently dropping Gitea events.
  • vendor-docs Traefik v3.5 — ForwardAuth middleware
    The ForwardAuth middleware delegates authentication to an external Service.
    Cited in:
  • vendor-docs Velero — How Velero Works
    Each Velero operation – on-demand backup, scheduled backup, restore – is a custom resource, defined with a Kubernetes Custom Resource Definition (CRD) and stored in etcd.
    Velero also includes controllers that process the custom resources to perform backups, restores, and all related operations.
    Cited in:
    • Paper 8 — Backup & DR Without a Vendor Contract — Vendor's authoritative description of Velero's architecture as a controller-driven backup system that captures Kubernetes API objects via CRDs and stores them alongside volume snapshots. Grounds the §3 architecture diagram for Velero and the §5 'why Frank skips Velero' framing — every operation Velero captures is also captured by ArgoCD + git, leaving only PVC data to handle.
  • vendor-docs VictoriaMetrics — Overview & Architecture
    VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database.
    It is optimized for storage with high-latency IO and low IOPS (HDD and network storage in AWS, GCP, Azure, etc.) compared to other solutions.
    Cited in:
    • Paper 7 — The Observability Stack, Honestly — Vendor's own articulation of the small-cluster wins — low IOPS, low RAM, Prometheus wire-compatible. Anchors the §2 vendor positioning and the §4 'metric cardinality' callout that drives Frank's choice of VM over a stock Prometheus TSDB.
  • vendor-docs Zot — Architecture overview
    Zot is a production-ready vendor-neutral OCI-native container image registry that supports the OCI Distribution and Image Specifications.
    Zot's modular architecture supports pluggable storage backends, authentication providers, and a sync mechanism that can mirror from other registries.
    Cited in:
    • Paper 13 — Self-Hosted CI/CD on a Homelab — Vendor's architectural description of Zot as an OCI-native, CNCF-incubated registry. Grounds the §3 architecture diagram (Zot as the image destination) and the §7 roadmap claim that OCI registries are absorbing arbitrary artefact storage.

paper

  • paper Agentic Misalignment — Anthropic Research
    In our experiments, models from every developer we tested took actions like blackmailing fictional executives or leaking sensitive information when faced with goals that conflicted with their continued operation.
    Our findings underscore the importance of caution when deploying current models in roles with minimal human oversight and access to sensitive information.
    Cited in:
    • Paper 17 — Agentic Orchestration & Safe Agent Workstations — Foundational paper on why agent isolation is not paranoia. Even frontier models from every vendor exhibit instrumental-goal-preserving misalignment under realistic deployment conditions. Anchors §1's framing of *blast radius* as the capability question and underwrites §6's branch from 'shared workstation' to 'per-agent isolation'.
  • paper BeyondCorp: A New Approach to Enterprise Security (Ward & Beyer, 2014)
    Virtually every company today uses firewalls to enforce perimeter security. However, this security model is problematic because, when that perimeter is breached, an attacker has relatively easy access to a company's privileged intranet.
    The BeyondCorp initiative is moving to a new model that dispenses with a privileged corporate network. Instead, access depends solely on device and user credentials, regardless of a user's network location.
    Cited in:
    • Paper 11 — Identity for a Heterogeneous Stack — The IAP / zero-trust origin paper. Establishes the model that every forward-auth IdP — Authentik's embedded outpost, Pomerium, Cloudflare Access — implements 11 years later. Anchors §1's framing of where an IdP sits in a modern stack.
  • paper Bottlerocket: A special-purpose container operating system (AWS blog)
    Bottlerocket is a Linux-based open-source operating system that we built from the ground up to run containers.
    We've removed everything that's not needed to run containers, which reduces the attack surface area and the resources used.
    All software changes are applied via image-based updates, which enable faster, more reliable updates that can be quickly rolled back if necessary.
    Cited in:
    • Paper 2 — Immutable OS & Declarative Machines — AWS's own design rationale for Bottlerocket — purpose-built, image-based updates, minimal surface area. The canonical AWS articulation referenced by every Bottlerocket-vs-X comparison. Used in landscape positioning, architecture diagram, and the ASG-coordinated update story.
  • paper Ceph: A Scalable, High-Performance Distributed File System (Weil et al., OSDI '06)
    We have designed and implemented Ceph, a distributed file system that provides excellent performance, reliability, and scalability.
    Ceph maximizes the separation between data and metadata management by replacing allocation tables with a pseudo-random data distribution function (CRUSH) designed for heterogeneous and dynamic clusters of unreliable object storage devices (OSDs).
    Cited in:
    • Paper 4 — Distributed Storage on Bare Metal — Foundational academic paper that introduces the CRUSH placement algorithm, the same machinery Rook still deploys twenty years later. Used in §2 and §3 to anchor the centralized-storage definition and explain why Ceph's minimum-host count is what it is.
  • paper Efficient Memory Management for Large Language Model Serving with PagedAttention
    We propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems.
    On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage.
    Evaluations on various models and workloads show that vLLM improves the throughput of popular LLMs by 2-4× compared with the state-of-the-art systems, such as FasterTransformer and Orca, with the same level of latency.
    Cited in:
    • Paper 10 — Self-Hosted Inference & the LLM Gateway Pattern — Foundational paper for vLLM. Defines why PagedAttention beats naive KV-cache layouts under batched workloads — the load-bearing throughput argument for §3 and §4 of the paper. Anchors the 'why are SaaS providers running vLLM' claim.
  • paper GitHub — GitHub Actions now supports CI/CD (free for public repos)
    GitHub Actions is free for public repositories, providing CI/CD with up to 20 concurrent jobs and a marketplace of pre-built actions.
    Hosted runners are available across Linux, macOS, and Windows, with 2,000 free minutes per month for private repositories on the Free plan.
    Cited in:
    • Paper 13 — Self-Hosted CI/CD on a Homelab — GitHub's own announcement framing of Actions as the SaaS baseline that self-hosted CI/CD competes with. Establishes the cost-curve geometry the rest of the paper measures against (free for OSS at small scale, expensive almost immediately above).
  • paper Google SRE Book — Monitoring Distributed Systems (Chapter 6)
    Monitoring a very complex application is a significant engineering endeavor in and of itself. Even with substantial existing infrastructure for instrumentation, collection, display, and alerting in place, a Google SRE team with 10–12 members typically has one or sometimes two members whose primary assignment is to build and maintain monitoring systems for their service.
    Your monitoring system should address two questions: what's broken, and why? The 'what's broken' indicates the symptom; the 'why' indicates a (possibly intermediate) cause.
    Cited in:
    • Paper 7 — The Observability Stack, Honestly — The foundational text on what an observability system is FOR — symptom vs cause, the four golden signals, the cost of monitoring as engineering effort. Cited in §1 to frame the capability and in §5 to ground Frank's choice to provision alerts from code, not click-ops.
  • paper HashiCorp Vault — Architecture
    Vault is a complex system that has many different pieces. To help both users and developers of Vault build a mental model of how it works, this page documents the system architecture.
    The storage backend is untrusted, and is used only to durably store encrypted data.
    Cited in:
    • Paper 9 — Secrets Management Without the Bootstrap Chicken-and-Egg — Canonical architecture document for the heavyweight in the space. Anchors §2's 'self-hosted · dynamic-credentials' quadrant and the §4 scale claim that Vault's storage-backend-is-untrusted model is what enables the audit-log + transit + PKI feature surface that compliance teams pay for.
  • paper How NAT traversal works (Tailscale engineering blog)
    STUN relies on a simple observation: when you talk to a server on the internet from a NATed client, the server sees the public `ip:port` that your NAT device created for you.
    Both endpoints must attempt communication at roughly the same time, so that all the intermediate firewalls open up while both peers are still around.
    The algorithm is: try everything at once, and pick the best thing that works.
    Cited in:
    • Paper 20 — Edge Clusters & Public Exposure — The closest thing the mesh-overlay world has to a foundational paper — a long-form technical writeup of STUN, UDP hole punching, and ICE as applied to WireGuard mesh VPNs. Grounds the §3 mesh-overlay architecture in actual NAT-traversal mechanics rather than hand-wavy 'it just works' marketing, and explains why the residential-ISP edge problem is hard enough to justify a dedicated paper category in the first place.
  • paper ImmutableServer (Martin Fowler / ThoughtWorks)
    An ImmutableServer is one that, once instantiated, is never modified. Updates and fixes occur not by changing the existing server, but by creating a new one.
    Phoenix servers go further: they are also re-created from scratch regularly, regardless of need.
    Cited in:
    • Paper 2 — Immutable OS & Declarative Machines — The canonical framing essay that names the immutable-server pattern and contrasts it with traditional mutable infrastructure. Used to anchor the capability question and to draw the philosophical line between Frank's choice (immutable) and the mutable baseline.
  • paper Introducing Kayenta — Google Cloud + Netflix announcement
    Kayenta is a platform for Automated Canary Analysis (ACA). It reads in user-configured metric data and runs statistical tests to determine whether a canary deployment differs significantly from its baseline.
    Kayenta is a critical component in our deployment pipeline. By automating canary analysis, we have been able to dramatically reduce the time it takes to ship code safely to production.
    Cited in:
    • Paper 14 — Progressive Delivery & the Service-Mesh Tax — Joint Google/Netflix launch writeup of Kayenta and the canonical introduction of the term 'automated canary analysis'. Anchors the §2 axis 'manual gating ↔ automated canary analysis' and the §7 roadmap claim that AnalysisTemplate is converging on Kayenta-shaped multi-metric, multi-interval analysis.
  • paper Kubernetes blog — Kubernetes 1.20: Volume Snapshot Moves to GA
    The Kubernetes Volume Snapshot feature is now GA in Kubernetes v1.20. It was introduced as alpha in Kubernetes v1.12, followed by a second alpha with breaking changes in Kubernetes v1.13, and promotion to beta in Kubernetes v1.17.
    Volume snapshots provide Kubernetes users with a standardized way to copy a volume's contents at a particular point in time without creating an entirely new volume. This functionality enables, for example, database administrators to backup databases before performing edit or delete modifications.
    Cited in:
    • Paper 8 — Backup & DR Without a Vendor Contract — Canonical announcement that the CSI VolumeSnapshot API graduated to GA — the cross-vendor primitive every modern backup tool now consumes. Anchors the §2 axis 'volume-only ↔ application-aware' and the §7 roadmap claim that CSI VolumeSnapshot is becoming the lingua franca and that the divide between storage-native backup (Longhorn) and storage-agnostic backup (Velero, Kasten) is narrowing.
  • paper Kubernetes Gateway API
    Gateway API is an official Kubernetes project focused on L4 and L7 routing in Kubernetes.
    Cited in:
  • paper Model Context Protocol — Introduction
    The Model Context Protocol (MCP) is an open protocol that standardizes how applications provide context to LLMs.
    MCP follows a client-server architecture where a host application can connect to multiple servers.
    Cited in:
    • Paper 17 — Agentic Orchestration & Safe Agent Workstations — Anthropic's specification of the protocol that vk-issue-bridge and Paperclip's heartbeat orchestrator both speak. Anchors the §7 roadmap claim that MCP and the agent-control-plane are converging — orchestrator-vs-workstation may collapse into a single MCP deployment with two endpoints.
  • paper OpenGitOps Principles v1.0.0
    Declarative: A system managed by GitOps must have its desired state expressed declaratively.
    Versioned and Immutable: Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.
    Pulled Automatically: Software agents automatically pull the desired state declarations from the source.
    Continuously Reconciled: Software agents continuously observe actual system state and attempt to apply the desired state.
    Cited in:
    • Paper 6 — GitOps at Small Scale — The CNCF/OpenGitOps working-group definition of what GitOps actually is. The four principles are load-bearing for §1 (the capability definition) and §6 (the decision-tree leaf where the 'just bash + kubectl' option fails Principle 3 and 4).
  • paper Volcano — A Cloud Native Batch System
    Volcano is a cloud native system for high-performance workloads, which has been accepted by Cloud Native Computing Foundation (CNCF) as its first and only official container batch scheduling project.
    Ensure all tasks of a job start simultaneously, suitable for distributed training and big data scenarios
    Cited in:
    • Paper 5 — GPU Scheduling for Mixed Workloads — CNCF Incubating projects own positioning statement and the gang-scheduling definition. Source for the §3 Volcano architecture diagram and the §6 decision-tree branch multi-tenant batch workloads. Counters the assumption that gang scheduling requires Run.AIs commercial product.

postmortem

  • postmortem Authentik — release notes and breaking-change archive
    Starting with this release, Outposts will use the URL set in the authentik configuration to access authentik.
    OAuth2/OIDC and Proxy Providers now require an Invalidation flow to be set. This flow is used to log the user out of the application.
    Cited in:
    • Paper 11 — Identity for a Heterogeneous Stack — Source for the schema-migration scar tissue Paper 11 documents — `invalidation_flow`, object-shaped `redirect_uris`, the `AUTHENTIK_HOST` requirement on the embedded outpost. Concrete evidence that 'pin the chart and forget' is not a viable strategy for any self-hosted IdP.
  • postmortem awesome-tunneling — community catalogue of self-hosted exposure tools (anderspitman/awesome-tunneling)
    exposing a local webserver via a public domain name, with automatic HTTPS, even if behind a NAT or other restricted network.
    map X domain/subdomain to Y port on Z client, and proxy all connections to that domain.
    Cited in:
    • Paper 20 — Edge Clusters & Public Exposure — A community-maintained catalogue of every tunneling tool the self-hosting world has invented, organised around the exact problem class Frank's Hop cluster solves: 'expose a local web service to the internet from behind NAT, with HTTPS'. The 'dream' section reads like the problem statement Frank wrote for itself in §1 — a useful sanity check that Paper 20's capability framing matches the practitioner consensus rather than a Frank-specific abstraction.
  • postmortem Frank — ArgoCD gotcha registry (symlink ComparisonError, manual-syncOptions-inheritance, root re-templating)
    ArgoCD's repo-server refuses to generate manifests for any source in a repo that contains a symlink resolving above the repo root. Self-heal can't fire because comparison itself fails, so the cluster silently runs on the last-known-good cache indefinitely.
    Manually-triggered syncs do NOT inherit spec.syncPolicy.syncOptions ... any chart-bundled CM larger than ~250KB ... fails with metadata.annotations: Too long: may not be more than 262144 bytes.
    Root App-of-Apps re-templates leaf Application specs on every sync. Any live mutation to a leaf is reverted within the root's sync window.
    Cited in:
    • Paper 6 — GitOps at Small Scale — Frank's running postmortem registry for ArgoCD — concrete operational scars accumulated while running the App-of-Apps pattern in production for this learning platform. Provides the source-of-truth dates and recovery commands for the §5 scar callouts and underwrites the §6 decision-tree branches with real failure modes.
  • postmortem Frank — Cilium gotchas (FQDN stale BPF, LBIPAM sharing-key, MixedProtocolLBService)
    Stale BPF egress rules persist even after deleting the CiliumNetworkPolicy — must also restart the Cilium agent to clear them.
    Annotating both Services with the same `lbipam.cilium.io/ips: "<addr>"` does NOT cause Cilium L2 IPAM to coordinate the allocation. IPAM treats each annotation as an independent request, gives the IP to whichever Service it processes first, and leaves the other at `EXTERNAL-IP <pending>` indefinitely with no error event.
    MixedProtocolLBService — TCP/22 + UDP/60000–60015 on a single Cilium L2 LB IP works on Cilium 1.17 + K8s 1.35. No feature gate flip, no annotation, no per-protocol service split.
    Cited in:
    • Paper 3 — eBPF Networking Without a Service Mesh — Frank's own running postmortem registry — concrete operational scars accumulated while running Cilium 1.17 in production on Talos. Provides source-of-truth dates and recovery commands for the §5 scar callouts and underwrites the §6 decision-tree branches.
  • postmortem Frank — OS / Talos / Hop gotchas (postmortem registry)
    talosctl apply-config --config-patch patches the base file, not the running config — all patches must be combined in one invocation.
    Talos control-plane taint must be removed for single-node cluster (allowSchedulingOnControlPlanes: true in Talos config).
    PodSecurity namespaces must be labeled pod-security.kubernetes.io/enforce: privileged for hostPort/privileged pods.
    Deployments using `hostPort` (e.g., Caddy on 80/443, Headscale STUN on 3478/UDP) must use `strategy: Recreate` — `RollingUpdate` deadlocks on a single-node cluster because the new pod can't bind ports while the old pod still holds them
    Headplane v0.5+ requires a `config.yaml` ConfigMap — env vars alone are insufficient
    Tailscale DaemonSet must run in kernel mode (`TS_USERSPACE=false`, `privileged: true`) for Caddy to see mesh source IPs
    `talosctl apply-config --config-patch` patches the base file, not the running config — all patches must be combined in one invocation
    Cited in:
    • Paper 2 — Immutable OS & Declarative Machines — Frank's own running postmortem registry — concrete operational scars accumulated across two clusters (Frank, Hop) while running Talos in production for this learning platform. Provides the source-of-truth dates and recovery commands for the scar callouts and underwrites the decision-tree branches.
    • Paper 20 — Edge Clusters & Public Exposure — Frank's own dated postmortem registry for everything that broke on Hop in the first three weeks of operation. Each one-liner is the scar of a real outage or wasted day; each maps directly to a property of the 'separate single-node edge cluster' architecture rather than a Caddy or Headscale bug. Source for the §5 scar callouts and the §6 decision-tree branch on 'are you willing to run a single-node cluster as critical infrastructure?'.
  • postmortem Frank Authentik runbook — per-topic gotcha file
    Embedded outpost needs AUTHENTIK_HOST env or forward-auth redirects use 0.0.0.0:9000.
    2026.x requires invalidation_flow + object-shaped redirect_uris + signing_key UUID.
    Cited in:
  • postmortem Frank cluster gotcha registry (frank-gotchas.md)
    kubectl port-forward flakes regularly with CNI-netns errors on gpu-1 pods only — use `kubectl get application -n argocd -o wide` for argocd-cli replacements; use `kubectl exec ... wget -qO-` for in-pod metrics.
    Pin GPU workloads with `nodeSelector: kubernetes.io/hostname: gpu-1` + defensive `nvidia.com/gpu:NoSchedule` toleration (insurance against driver re-validation re-asserting the taint).
    RWO PVC + RollingUpdate deadlocks; use strategy: Recreate. Switching strategy via Helm needs a one-time kubectl patch to clear the orphan rollingUpdate: block.
    ESO: empty data: [] is rejected; delete the ExternalSecret if all keys are removed.
    SOPS-encrypted secrets must NOT be ArgoCD-managed; apply out-of-band from secrets/.
    Pin GPU workloads with nodeSelector kubernetes.io/hostname gpu-1 plus defensive nvidia.com/gpu NoSchedule toleration (insurance against driver re-validation re-asserting the taint).
    Ollama system memory errors mean container cgroup RAM (not VRAM) — OLLAMA_KEEP_ALIVE page cache pins the cgroup near resources.limits.memory.
    kubectl port-forward flakes regularly with CNI-netns errors on gpu-1 pods only — use kubectl get application -n argocd -o wide for argocd-cli replacements; use kubectl exec ... wget -qO- for in-pod metrics.
    File-provisioned alerts/dashboards in apps/grafana-alerting/manifests/ are read-only in UI; edit ConfigMap, push, restart pod (provisioning files are read at boot, not watched).
    12.x SSE alert rules need 3-step A→B→C; classic-condition format fails with sse.parseError.
    ALERTS{} does NOT exist in VM for Grafana-managed alerts — use alertlist panel type.
    SOPS-encrypted secrets must NOT be ArgoCD-managed; apply out-of-band from `secrets/`.
    RWO PVC + RollingUpdate deadlocks; use `strategy: Recreate`.
    ESO: empty `data: []` is rejected; delete the ExternalSecret if all keys are removed.
    `envFrom.secretRef` without `optional: true` blocks rolling updates when the Secret is missing.
    v1 Task uses `computeResources` not `resources` — schema validation silently fails the whole app.
    Gitea sends `X-Gitea-Event` (not `X-GitHub-Event`) — use `cel` interceptor, not `github`.
    Zot Helm chart v0.1.0 too minimal — use v0.1.60+ for TLS/auth/persistence.
    Gitea `webhook.ALLOWED_HOST_LIST` blocks in-cluster delivery — add `*.svc.cluster.local`.
    `workloadRef.scaleDown` defaults to `never` — set `onsuccess` explicitly or both Rollout + chart Deployment serve traffic.
    `workloadRef` "leaks" to a healthy-looking Helm Deployment when reconcile aborts pre-workload-phase (missing AnalysisTemplate, missing traffic-router plugin, etc.) — the ONLY signal is the controller pod log; `kubectl get rollout` looks identical to a steady-state run.
    Prometheus provider panics on empty result vector → metric `phase: Error` → 10s-cadence retry → canary aborts in ~50s. Verify metric exists + has samples first.
    LiteLLM Prometheus is Enterprise-only — OSS image emits no `litellm_*` metrics.
    Cilium: lbipam.cilium.io/ips alone is NOT a sharing directive — separate Services need matching lbipam.cilium.io/sharing-key.
    Blueprints don't assign providers to embedded outpost — manual Django ORM step (see frank-argocd.md).
    s6-overlay v3 in non-root mode needs `S6_KEEP_ENV=1`, `S6_VERBOSITY=2`, `with-contenv` shebangs (`#!/command/with-contenv bash` — `/command/`, NOT `/usr/bin/`), and `/run` chown'd to AGENT_UID at image build time.
    vk-issue-bridge's 30 s MCP timeout cascades to zombie execution_processes: bridge crash on timeout → vk-local request handler future drops → `Child::wait()` cancelled → setup/cleanup shell scripts exit but never reaped → DB rows stuck `status='running'` forever, UI shows workspaces stuck active with no output.
    `shareProcessNamespace: true` is incompatible with s6-overlay v3 (suexec must be PID 1) — use shared workspace volume + `kubectl exec -c <other>` for cross-container debugging.
    vk-local `limits.memory: 4Gi` is too tight in practice — `VK_MAX_CONCURRENT_EXECUTIONS=4` does NOT bound the cgroup once the bridge feeds 8+ cards (queued sessions retain memory; new images drift baseline).
    Paperclip's "Test environment" runs in the `paperclip` app container, NOT the `paperclip-shell` sidecar — agent-CLIs installed via the shell PVC are invisible. Wire through the shared `/paperclip` PVC.
    Cited in:
    • Paper 1 — Heterogeneous Hardware as a Design Choice — Frank's own running ledger of per-node-class failure modes — the gpu-1-only port-forward flake, the defensive NoSchedule toleration pattern, the Ollama-cgroup-memory pitfall. Every entry is direct evidence that a heterogeneous fleet pays a per-class operator tax that a homogeneous fleet wouldn't; §3 and §5 cite this for the scar callouts.
    • Paper 4 — Distributed Storage on Bare Metal — Frank's own running postmortem registry — concrete operational scars accumulated while running Longhorn in production for this learning platform. Provides the source-of-truth dates and recovery commands for §5 scar callouts and underwrites the §6 decision-tree branches.
    • Paper 5 — GPU Scheduling for Mixed Workloads — Franks own running postmortem registry — concrete operational scars accumulated while running the NVIDIA GPU Operator + Intel GPU Resource Driver pair in production for this learning platform. Provides source-of-truth dates and recovery commands for the §5 scar callouts and underwrites the §6 decision-tree branches.
    • Paper 7 — The Observability Stack, Honestly — Frank's own running postmortem registry — concrete operational scars accumulated while running the Grafana stack against VictoriaMetrics. Provides the source-of-truth one-liners and recovery commands for §5 scar callouts.
    • Paper 8 — Backup & DR Without a Vendor Contract — Frank's own running postmortem registry — concrete operational scars accumulated while running Longhorn-native backup to Cloudflare R2 in production for this learning platform. Provides source-of-truth dates and recovery commands for the §5 scar callouts on the SOPS-out-of-band requirement, the RWO restore-order constraint, and the dependency-ordering load-bearing for any DR runbook.
    • Paper 9 — Secrets Management Without the Bootstrap Chicken-and-Egg — Frank's own running incident catalogue, codified as one-liner gotchas with per-topic prose under `docs/runbooks/frank-gotchas/storage-secrets-ssa.md`. Direct evidence for the §5 scar callouts; also the load-bearing claim that every scar in Paper 09 came from a real incident, not a hypothetical.
    • Paper 13 — Self-Hosted CI/CD on a Homelab — Frank's own running postmortem registry — concrete operational scars accumulated while running Gitea + Tekton + Zot in production for this learning platform. Provides source-of-truth dates and recovery commands for the §5 scar callouts and underwrites the §6 decision-tree branches.
    • Paper 14 — Progressive Delivery & the Service-Mesh Tax — Frank's own running postmortem registry — concrete operational scars accumulated while running Argo Rollouts in production for this learning platform. Provides source-of-truth dates and recovery commands for the §5 scar callouts and underwrites the §6 decision-tree branches.
    • Paper 15 — Ingress, Forward-Auth, and the Service Catalog — Frank's own one-line gotcha registry — the two lines above are the load-bearing scars for §5. The full prose lives in per-topic files under docs/runbooks/frank-gotchas/.
    • Paper 17 — Agentic Orchestration & Safe Agent Workstations — Frank's own running postmortem registry — concrete operational scars accumulated while running secure-agent-pod + VibeKanban + Paperclip + Sympozium in production for this learning platform. Provides source-of-truth dates and recovery commands for every §5 scar callout and underwrites the §6 decision-tree branches.
  • postmortem Longhorn issue #11392 — RecurringJob has no spec.backupTargetName field
    .spec.backupTargetName: field not declared in schema
    Cited in:
    • Paper 8 — Backup & DR Without a Vendor Contract — Upstream issue documenting the Longhorn 1.11 CRD schema gap that prevents per-RecurringJob target selection. All RecurringJobs in 1.11 target the `default` BackupTarget; there is no per-job routing. Anchors the §5 Scar 3 callout and the §7 roadmap note that Frank's manifests keep the original two-target names (`daily-nas` / `weekly-r2`) as documentation of intent.
  • postmortem Longhorn issue #11412 — NFS BackupTarget generates host/path instead of host:/path
    remote share not in 'host:dir' format
    Fix targeted for Longhorn v1.13.0.
    Cited in:
    • Paper 8 — Backup & DR Without a Vendor Contract — Upstream bug report and triage thread for the Longhorn 1.11 NFS mount-string defect that disabled Frank's local-first restore path. Provides the canonical incident reference for §5 Scar 2 and the §7 roadmap claim that the NAS-target axis is gated on Longhorn v1.13.0 shipping. Type: postmortem, since it documents a confirmed defect, its triage, and its fix-version commitment.
  • postmortem pc-1 reboot investigation (2026-05-11) — 245-line root-cause writeup
    pc-1 has rebooted 7 times in the last 33 days (window: 2026-04-04 → 2026-05-07), with no kernel panic, no OOM kill, no watchdog event, and no thermal trip recorded.
    Verdict: hardware-class fault. Best-fit single-cause hypothesis is a deteriorating 2013-era ATX PSU (Corsair / OCZ-era unit, well past typical 7–10 year capacitor service life) browning out under transient CPU load.
    Cited in:
    • Paper 1 — Heterogeneous Hardware as a Design Choice — The canonical pc-1 scar — and the structural argument for keeping it in the fleet despite the failure. A homogeneous fleet of 2025-vintage mini-PCs would never have surfaced a 'silent reset faster than printk' failure mode; the heterogeneous fleet did, and the investigation that resulted is itself a deliverable. §5's first scar callout is this incident.
  • postmortem SigNoz — Loki vs Elasticsearch (practitioner comparison)
    Loki indexes only metadata (labels) for log entries, while Elasticsearch indexes the full content of logs. This fundamental design choice has significant implications for storage costs, query performance, and use cases.
    Loki is significantly more cost-effective for storing large volumes of logs because of its index-light design and use of object storage backends like S3, GCS, or local filesystems.
    Cited in:
    • Paper 7 — The Observability Stack, Honestly — Practitioner-level head-to-head with concrete trade-offs on the index-at-write vs index-at-query split. Used in §3 to explain why Loki's log model is structurally different from OpenSearch/Elastic, and in §4 to ground the log-volume-vs-query-cost callout.
  • postmortem Simon Willison — LLM pricing notes
    One of the most notable trends from 2024 was the total collapse in terms of LLM pricing — the API models are absurdly inexpensive now.
    Generating captions for 68,000 photos using Gemini 1.5 Flash 8B costs $1.68.
    Llama 3.2 3B from a developer's perspective: running locally on my laptop using Ollama, it's blazingly fast — about 100 tokens per second on my M2 Mac.
    Cited in:
    • Paper 0 — Why Run Your Own Cluster in 2026? — Running practitioner log of frontier LLM API price collapse. Critical counterweight to the assumption that 'self-host because cloud inference is expensive' — at most scales the API is now cheaper by an order of magnitude. Supports the counter-argument in §3.
    • Paper 10 — Self-Hosted Inference & the LLM Gateway Pattern — Named practitioner running OSS LLMs at small scale and tracking the API-price collapse in real time. Critical counterweight for §1 and §6 — the 'just use the API' counter-argument is grounded in numbers Simon has actually paid. Pairs with the data-locality / latency-floor / learning-depth triple in §5.

talk

  • talk AKS vs EKS vs GKE: What you get with managed Kubernetes-as-a-Service — Fairwinds
    Managed Kubernetes services take the complexities of managing Kubernetes clusters and relieve organizations of the underlying infrastructure management tasks. These services automatically handle the installation, configuration, and maintenance of Kubernetes.
    When you run Kubernetes yourself, you have to manage everything such as servers, networking, upgrades, and security, which adds significant operational overhead.
    Cited in:
  • talk GitOps — Operations by Pull Request (Alexis Richardson, Weaveworks, 2017)
    By using Git as our source of truth and operations toolkit, we can keep declarative descriptions of all of our infrastructure and applications under version control.
    Our Git repository is the source of truth for what we want our system to look like. Our diff tools and convergence operators are how we get there.
    Cited in:
    • Paper 6 — GitOps at Small Scale — The original Weaveworks blog post that named the practice. Cited as the historical origin of the GitOps label (the term predates both ArgoCD and Flux v2). Used in §1 to ground the capability in its 2017 articulation.
  • talk KubeCon NA 2022 — The State of Kubernetes Secrets Management
    There is no single 'right' way to manage secrets in Kubernetes — the right answer depends on whether you trust your cluster API, your storage backend, your CI system, and your operators, and the answer to each is rarely the same.
    The trade-off between encrypt-in-Git (Sealed Secrets, SOPS) and sync-from-external-store (ESO, CSI) is fundamentally an axis of where you put the trust boundary.
    Cited in:
    • Paper 9 — Secrets Management Without the Bootstrap Chicken-and-Egg — A community talk that frames the entire space as a trust-boundary decision, which is exactly the framing §6's decision-tree adopts. Cited as the canonical 'there is no single right answer' source — useful when the Paper resists the temptation to declare one vendor universally correct.
  • talk Kubernetes 101 — Jeff Geerling video series
    A YouTube streaming series on Kubernetes and container-based infrastructure by Jeff Geerling.
    Everything on his site is being served from a Drupal site, which used to run on a Kubernetes cluster of Raspberry Pis in Jeff Geerling's basement.
    Cited in:
    • Paper 0 — Why Run Your Own Cluster in 2026? — Highest-visibility practitioner positioning of the DIY-homelab-for-learning thesis. Anchors the framing that real-world clusters built from cheap hardware teach skills that paid courses don't.
  • talk Liz Rice — eBPF: A New Frontier for Performance Tooling (KubeCon NA 2021)
    eBPF is the kernel's universal programmability layer — it lets you load sandboxed programs into the kernel that run on every packet, every syscall, every scheduler tick.
    Cilium is the largest eBPF application in production today; it replaced kube-proxy at GitLab, Adobe, Bell, and a half-dozen other multi-thousand-node Kubernetes deployments.
    Cited in:
    • Paper 3 — eBPF Networking Without a Service Mesh — The canonical introduction of eBPF-as-data-plane to the Kubernetes operator audience. Underwrites the §2 axis 'userspace ↔ kernel/eBPF' and the §7 roadmap claim that the iptables-kube-proxy default is on its way out.
  • talk TechnoTim — 2 Factor Auth and Single Sign on with Authelia (Traefik forward-auth)
    Authelia is an open-source authentication and authorization server providing two-factor authentication and single sign-on (SSO) for your applications via a web portal.
    It acts as a companion of reverse proxies like nginx, Traefik or HAProxy to let them know whether requests should either be allowed or redirected to Authelia's portal for authentication.
    Cited in:
    • Paper 11 — Identity for a Heterogeneous Stack — Highest-visibility homelab/SMB-scale write-up of the forward-auth pattern. Anchors the §2 claim that the open-source homelab world has converged on a small set of forward-auth IdPs glued to one of three reverse proxies.

benchmark

  • benchmark ArgoCD vs FluxCD — r/kubernetes practitioner thread
    ArgoCD has the better UI by far. Flux is more modular and feels more Kubernetes-native, but the UI gap is real.
    We picked Flux because we wanted everything to be a CR and we didn't want a separate UI service running. We picked ArgoCD because we wanted the UI and the App-of-Apps pattern.
    Cited in:
    • Paper 6 — GitOps at Small Scale — Practitioner-level head-to-head capturing the community consensus on the trade Frank made. Not a controlled benchmark, but representative of the decision shape for the ≤10-node single-cluster case the paper actually addresses.
  • benchmark Cloudflare network — sub-50ms latency to 95% of internet users (cloudflare.com/network)
    Sub-50ms to 95% of users — 95% of the world's Internet-connected population is within 50 milliseconds of a Cloudflare data center — most are within 20ms.
    One of the world's largest networks — running every service in every data center for unmatched performance, security, and reliability.
    Cited in:
    • Paper 20 — Edge Clusters & Public Exposure — Vendor-published latency claim that anchors the §4 'what scale changes' discussion — Cloudflare Tunnel's edge is closer to most users than any one-VPS edge can ever be, and the latency tax of running through a single Hetzner Falkenstein PoP is real. Sets the baseline against which Hop's single-region edge has to justify itself: own the data plane, accept the latency floor of one geographic location.
  • benchmark CNI Benchmark — Cilium vs Calico vs Kube-router (Cilium)
    Cilium with eBPF kube-proxy replacement reduces latency by 30% in our benchmark vs iptables-mode kube-proxy at 5,000 Services.
    iptables rule-evaluation cost is O(N) in the number of Services — at 10,000 Services, kube-proxy resync takes minutes; the eBPF map-lookup is O(1).
    Cited in:
    • Paper 3 — eBPF Networking Without a Service Mesh — Vendor-published benchmark (acknowledged biased toward Cilium, but methodology is open) covering throughput, latency, and rule-evaluation cost at small-to-mid Service counts. Source for the §4 'scale changes' claims about kube-proxy degradation and eBPF map-lookup constancy. The named-gap section in this dossier specifically calls out that this benchmark does not measure the bundled operational tax.
  • benchmark Flagger vs Argo Rollouts vs service meshes — a guide to progressive delivery in Kubernetes (CNCF blog, Buoyant)
    Flagger requires zero manifest changes to adopt canary—you continue using standard Deployments while Flagger manages the traffic splitting automatically.
    Argo Rollouts requires migrating Deployment resources to Rollout CRDs, which is a larger migration effort but provides more explicit control via the step-based strategy definition.
    Cited in:
    • Paper 14 — Progressive Delivery & the Service-Mesh Tax — Practitioner-level head-to-head from Buoyant (Linkerd maintainers) hosted on the CNCF blog. Explicitly compares Argo Rollouts and Flagger on workloadRef vs Deployment-wrapping, traffic-router coupling, and the Flux-vs-Argo ecosystem split. Closest thing to a controlled comparison in the public literature and source for the §3 cross-vendor shared-shape diagrams.
  • benchmark GitGuardian — The State of Secrets Sprawl 2024
    In 2023, we detected 12.8 million secrets exposed in public GitHub commits — a 28% increase over 2022.
    Generic high-entropy strings, generic passwords, and Google API keys remain the top three leak types.
    Cited in:
    • Paper 9 — Secrets Management Without the Bootstrap Chicken-and-Egg — The closest thing to an industry-wide benchmark on the cost of NOT running a secret-store: 12.8M leaks in public Git in one year, the empirical pricetag of 'plaintext in Git'. Anchors §4's scale claim that the rotation-cadence axis flips the ranking when secret count grows past a single human's mental cache, and provides the headline number for §6's argument that the null-hypothesis leaf is not safe at any scale where you ship code from more than one machine.
  • benchmark Leaving the Cloud — 37signals Cloud Exit case study
    37signals pulled Basecamp, HEY, and five other heritage apps out of AWS and onto their own hardware — without adding any new staff. By 2022, their cloud bills had grown to over $3.2 million annually.
    Performance improved dramatically, with database queries running 3-5x faster and page load times improving 30-50%.
    Total projected savings from the combined cloud exit are well over $10 million over five years. The hardware cost was approximately $600,000 for a one-time purchase.
    Cited in:
    • Paper 0 — Why Run Your Own Cluster in 2026? — Canonical real-world cost benchmark of cloud-vs-self-hosted for a stable, predictable workload at small-org scale. Provides hard numbers other practitioners cite. Most directly supports §2 of Paper 00 (real costs of the three approaches).
  • benchmark Longhorn vs Ceph (Rook) — Reddit r/kubernetes practitioner thread
    Longhorn is much simpler to set up and operate. Ceph is more powerful but requires more nodes and more tuning to perform well.
    I run Longhorn on a 3-node cluster with NVMe drives and it just works.
    Cited in:
    • Paper 4 — Distributed Storage on Bare Metal — Practitioner-level head-to-head with reproducible setup notes — explicitly names the 'Ceph needs more nodes to perform' rule of thumb cited in §4. Not a controlled benchmark, but the closest thing the community has to one at homelab scale, and representative of the consensus.
  • benchmark Raspberry Pi Cluster Episode 3 — Installing K3s Kubernetes on the Turing Pi (Geerling, 2020)
    K3s is purpose-built for low-power, small-form-factor compute clusters like the Turing Pi, since it is lighter than the full-fat K8s, requires much less in the way of resources to operate, and was easier to set up.
    I'm able to deploy K3s to the cluster, and it works well enough... but there are still some quirks I'd like to iron out, especially when it comes to the slower Pi 3 B+ compute modules I'm using on this Turing Pi.
    Cited in:
    • Paper 1 — Heterogeneous Hardware as a Design Choice — The reference work for the all-RPi homelab fleet shape — including the candid 'works well enough … some quirks' note that anchors the §2 claim about what the all-Pi fleet teaches and what it cannot. The 'slower Pi 3 B+ … some quirks' admission is itself an argument for why even the ostensibly-homogeneous Pi fleet pays a tax on the day one Pi differs from the others.
  • benchmark VictoriaMetrics — Benchmarking Prometheus-compatible time series databases
    vmagent uses 3.2x/1.6x less CPU and 2.7x/3.0x less memory than competing scrape agents (OpenTelemetry Collector and Prometheus 3.x).
    VictoriaMetrics uses 5x-10x less RAM than Prometheus for the same data.
    Cited in:
    • Paper 7 — The Observability Stack, Honestly — Vendor-published benchmark — biased but reproducible. Anchors the §4 cardinality / resource-cost callout that justifies Frank's swap from stock Prometheus to VictoriaMetrics on a small-cluster footprint.
  • benchmark vLLM v0.6.0: 2.7× Throughput Improvement and 5× Latency Reduction
    vLLM v0.6.0 delivers up to 2.7x higher throughput and 5x faster TPOT compared to v0.5.3, on Llama 8B model. The performance is measured on ShareGPT dataset using a single H100 GPU.
    For Llama 70B and a single H100, vLLM v0.6.0 delivers 1.8× higher throughput, with TPOT reduced from 200ms to 100ms.
    Across H100, A100, A10 GPUs, vLLM consistently outperforms TGI and TensorRT-LLM.
    Cited in:
    • Paper 10 — Self-Hosted Inference & the LLM Gateway Pattern — Reproducible benchmark with named hardware (H100, A100, A10), named workload (ShareGPT), and named comparators (TGI, TensorRT-LLM). Anchors the throughput claim in §3 and the 'datacenter-scale benchmarks don't speak to the homelab regime' gap in §4. The 2.7× / 5× numbers are quotable.