In-Cluster Ingress — Traefik, Wildcard TLS, and a Homepage Dashboard

Up until now, all of Frank’s services were reachable via direct Cilium L2 LoadBalancer IPs. That works on a local network, but it means no TLSTransport Layer SecurityThe encryption under HTTPS and most other secure protocols. What a certificate is for, and what fails visibly when the certificate does not match the hostname., no unified authentication, no human-readable URLs, and no single place to see what is running. The external Traefik on raspi-omni handled *.frank.derio.net routing, but it sat outside the cluster — a separate Ansible-managed box.

This post moves the ingress controller inside the cluster: Traefik v3 on the raspi edge nodes, serving all services under *.cluster.derio.net with wildcard TLS from Let’s Encrypt, Authentik forward-auth for services without native SSOSingle Sign-OnOne login across many applications. On Frank, Authentik holds the session and Traefik asks it before forwarding a request., and a gethomepage.dev dashboard at master.cluster.derio.net.

Update, August 2026. The old *.frank.derio.net estate is gone — every non-Omni name retired, including the OIDC issuer the kube-apiserver depended on. Retiring *.frank.derio.net covers the dual-issuer mechanism that made it a cutover rather than an outage, and the two traps that nearly hid a failure.

Architecture

    flowchart TD
  subgraph Internet
    DNS[Pi-hole<br/>*.cluster.derio.net → 192.168.55.220]
  end
  subgraph Cluster[Frank Cluster]
    subgraph Traefik[traefik-system namespace]
      direction TB
      T[Traefik — raspi-1/raspi-2]
      AC[ACME — Cloudflare DNS-01<br/>*.cluster.derio.net]
      MW[Middlewares<br/>ip-allowlist + security-headers]
      FA[authentik-forward-auth]
    end
    Direct["Direct proxy — native auth<br/>ArgoCD · Sympozium · Authentik · Homepage"]
    SSO["Forward-auth via Authentik<br/>Grafana · Longhorn · Infisical · n8n · Gitea"]
  end
  DNS --> T
  T --> AC
  T --> MW
  T --> FA
  T --> Direct
  FA --> SSO
  

Why Traefik

Evaluated Traefik, Envoy Gateway, and Contour. Traefik won on:

• Authentik integration — official docs, battle-tested forward-auth middleware
• Resource footprint — single pod, ~50MB idle, proven on RPi 4 ARM64
• Familiarity — same middleware model as the existing Ansible-managed Traefik, near 1:1 translation

TLS: Built-in ACME, Not cert-manager

cert-manager is already deployed for internal webhook TLS, but for this use case Traefik’s built-in ACMEAutomatic Certificate Management EnvironmentThe protocol Let's Encrypt uses to issue certificates to a server that can prove it controls a domain. Frank's edge proves it over DNS rather than HTTP, which is what lets it get certificates for hosts the internet cannot reach. resolver is simpler — no extra CRDsCustomResourceDefinitionThe object that teaches the Kubernetes API a new resource type. Install a CRD and the API server starts serving a kind it has never heard of, with validation and RBAC like any built-in., no Issuer/Certificate objects. One wildcard cert for *.cluster.derio.net via Cloudflare DNS-01:

# apps/traefik/values.yaml (excerpt)
certificatesResolvers:
  cloudflare:
    acme:
      email: "admin@derio.net"
      storage: /data/acme.json
      dnsChallenge:
        provider: cloudflare
        propagation:
          disableChecks: true
          delayBeforeChecks: 60

disableChecks: true skips local DNS propagation verification (blocked by router ACLsAccess Control ListAn ordered list of allow/deny rules attached to a resource. In Headscale it decides which mesh node may reach which service, independent of network routing.). delayBeforeChecks: 60 gives Cloudflare 60 seconds to propagate the TXT record globally.

The cert stores in acme.json on a 128Mi Longhorn PVPersistentVolumeThe actual piece of storage a PVC binds to. The claim is the request; the PV is what satisfies it.. Since the PV is RWOReadWriteOnceA PVC access mode that lets exactly one node mount the volume read-write at a time. It is the reason a RollingUpdate deadlocks: the replacement pod cannot mount the volume until the outgoing pod releases it., Traefik runs with strategy: Recreate.

PVC Permissions Gotcha

Longhorn creates root-owned volumes, but Traefik runs as uid 65532 (nonroot). Without fsGroup, the ACME resolver fails silently with permission denied on /data/acme.json — Traefik logs it as “ACME resolve is skipped from the resolvers list”:

podSecurityContext:
  fsGroup: 65532
  fsGroupChangePolicy: "OnRootMismatch"

The Helm chart uses top-level podSecurityContext, not deployment.podSecurityContext — the nested path is silently ignored.

Middlewares

Three Middleware CRDs in traefik-system:

security-headers — HSTSHTTP Strict Transport SecurityA header telling browsers to only ever use HTTPS for a domain. It is remembered and hard to undo, which is what makes a bad certificate on an HSTS host a hard block rather than a warning., X-Frame-Options, Content-Type sniffing protection, referrer policy.

ip-allowlist — restricts to RFC 1918 ranges. This is a homelab, not public-facing.

authentik-forwardauth — sends every request to the Authentik embedded outpost. The outpost checks the session cookie; if missing or expired, redirects to Authentik login:

spec:
  forwardAuth:
    address: "http://authentik-server.authentik.svc.cluster.local:80/outpost.goauthentik.io/auth/traefik"
    trustForwardHeader: true
    authResponseHeaders:
      - X-authentik-username
      - X-authentik-groups
      - X-authentik-email
      - X-authentik-uid

IngressRoutes

All 16 IngressRoutes live in a single ingressroutes.yaml. Each route targets the websecure entrypoint with the wildcard cert resolver and at least ip-allowlist + security-headers middlewares.

Services split into two tiers:

• Direct proxy (no forward-auth): ArgoCD, Sympozium, Authentik, Homepage — either have their own login or are the IdP itself.
• Forward-auth via Authentik: Grafana, Longhorn, Infisical, LiteLLM, Paperclip, ComfyUI, n8n, Gitea, Zot, Tekton — services without native OIDCOpenID ConnectAn identity layer on top of OAuth 2.0: it adds a token saying *who* the user is, not merely what the client may do. Frank's single sign-on speaks it, with Authentik as the provider..

$ kubectl get ingressroutes -n traefik-system -o wide
NAME           AGE
argocd         12d
authentik      12d
comfyui        12d
gitea          12d
grafana        12d
homepage       12d
litellm        12d
longhorn       12d
n8n            12d
paperclip      12d
sympozium      12d
tekton         12d
zot            12d

Authentik Blueprints

The proxy providers for *.cluster.derio.net are managed declaratively via an Authentik blueprint ConfigMap:

- model: authentik_providers_proxy.proxyprovider
  state: present
  identifiers:
    name: Grafana (cluster)
  attrs:
    authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
    authentication_flow: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
    invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
    mode: forward_single
    external_host: https://grafana.cluster.derio.net

The invalidation_flow field is required in Authentik 2026.x — without it, the blueprint fails silently with a serializer error.

Blueprint creates providers and applications but does not assign them to the embedded outpost. Outpost assignment must be done via Django ORMObject-Relational MappingA library presenting database rows as objects. Convenient until the generated query is the thing you need to reason about. after the blueprint applies — Authentik blueprints cannot append to an outpost’s provider list without replacing existing assignments.

Homepage Dashboard

A gethomepage.dev instance at master.cluster.derio.net provides the cluster landing page with HTTP health indicators:

• Infrastructure: ArgoCD, Longhorn, Grafana, Infisical, Authentik
• CI/CD: Gitea, Zot, Tekton
• Development: LiteLLM, Sympozium, n8n, Paperclip, ComfyUI

Health checks use siteMonitor (HTTP HEAD/GET), not ping (ICMPInternet Control Message ProtocolThe protocol behind `ping` and path-MTU discovery. Blocking it wholesale breaks more than it protects.) — Kubernetes ClusterIP addresses do not respond to ICMP from inside the cluster.

Retiring *.frank.derio.net (2026-08)

For four months this layer ran two domains at once. *.cluster.derio.net was the new in-cluster Traefik; *.frank.derio.net was the old external Traefik on raspi-omni — and after that Pi died in June, a set of nine “compatibility” IngressRoutes re-fronting the dead edge’s names on the surviving in-cluster Traefik. Temporary, obviously. Temporary things that work are the hardest to remove.

The hard part was never the routes. It was auth.frank.derio.net, because the kube-apiserver trusted it as an OIDCOpenID ConnectAn identity layer on top of OAuth 2.0: it adds a token saying *who* the user is, not merely what the client may do. Frank's single sign-on speaks it, with Authentik as the provider. issuer, and an issuer is not a hostname you can simply repoint.

Structured authentication is what made this survivable

The apiserver originally took --oidc-issuer-url and friends: one issuer, set by flag, changed only by restarting the API server. Under that model there is no cutover, only an outage.

Kubernetes’ structured authentication config replaces those flags with a file listing a list of authenticators:

# patches/phase13-auth/authn-config.yaml — during the overlap
apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
jwt:
  - issuer:
      url: https://auth.frank.derio.net/application/o/k8s-agent/   # authenticator A
      audiences: [k8s-agent]
    claimMappings: &mappings
      username: {claim: preferred_username, prefix: "authentik:"}
      groups:   {claim: groups, prefix: ""}
  - issuer:
      url: https://auth.cluster.derio.net/application/o/k8s-agent/ # authenticator B
      audiences: [k8s-agent]
    claimMappings: *mappings

Both issuers are trusted simultaneously, and identical claimMappings mean a token from either normalises to the same Kubernetes identity — authentik:ak-Kubernetes Agent Access-client_credentials. Existing credentials keep working while newly minted ones carry the new issuer. That is the whole trick.

The eight-hour overlap, and why it is eight

Authentik mints k8s-agent tokens with access_token_validity: hours=8. Once Authentik advertises the new host, every new token carries the new issuer — but tokens already in someone’s kubeconfig keep the old one until they expire. So authenticator A has to stay trusted for one full token lifetime after the last old-issuer token was issued:

T0    = last old-issuer token minted        2026-08-01 17:56:02Z
T0+8h = earliest safe removal of A          2026-08-02 01:56:02Z

Eight hours is not a guess or a safety margin — it is read from the provider and confirmed by decoding a minted token’s exp. Refresh tokens live 30 days, but a refresh made after the cutover uses the request hostname and so returns a new-issuer token, which is why they do not extend the window.

The measurement that mattered was not “has 8 hours passed” but a live one: mint a token through each host and submit both to TokenReview. During the overlap both authenticate identically. After authenticator A is removed, the legacy one is rejected and the cluster one is not. That accepted→rejected flip is the only direct evidence the change actually took — and it caught a real mistake, because omnictl apply reads a local file and cheerfully re-applied a pre-merge config twice from an unpulled checkout, reporting success both times.

The Omni exception

omni.frank.derio.net stays. Omni manages the machines that run the cluster, so retiring its own name from inside the cluster is a circular dependency waiting to bite — and Frank has already lost that Pi once.

It survives on a DNS detail worth knowing: an explicit CNAME outlives the wildcard it sat under. Removing the *.frank.derio.net wildcard A record took the other names with it and left omni resolving through its own CNAME to omni.frank.lan. Its certificate is minted by a systemd timer on the Omni host, not by Traefik, so it was never coupled to this layer at all. The Headscale split-DNS entry for the bare frank.derio.net zone stays too — a suffix, not a hostname, and easy to delete by accident with a careless regex.

Deleting a manifest deletes nothing

The retirement PR removed all nine compatibility IngressRoutes from Git. After it merged, all nine were still serving.

Every ArgoCD Application here runs prune: false — the right default for a homelab, since a bad render can never cascade into mass deletion. The cost is that removing a resource from Git makes ArgoCD stop managing it, not remove it. The app goes OutOfSync and the object serves on indefinitely.

It hid well, for three compounding reasons. The routes live in apps/traefik/manifests but belong to the Application traefik-extras, while the similarly-named traefik app is the Helm chart and stayed Synced/Healthy throughout. OutOfSync is the same status a drifted annotation produces. And the Authentik half of the very same PR did remove itself correctly, because those were expressed as blueprint tombstones:

- model: authentik_core.application
  state: absent            # declarative deletion — this one really does delete
  identifiers: {slug: longhorn}

Half a change disappearing correctly is a strong signal the other half did too. It was not. Deletion under prune: false is a manual step: delete the objects, then assert they are gone, rather than assert the app is Synced.

Rollback path

The overlap is the rollback. Until authenticator A is removed, reverting is re-adding the legacy issuer block and re-applying the Omni ConfigPatch — the old routes and DNS still exist, so nothing else has to move. After removal the rollback is the same operation in reverse, with the added cost of re-creating the compatibility IngressRoutes and DNS records, and it only matters for credentials no one has re-minted.

The genuinely irreversible moment is not the merge. It is omnictl apply of the single-issuer patch, which rolls the three control planes one at a time and, from that point, rejects every legacy token in existence.

Missteps

Recovery Path

References

• Traefik Helm Chart
• Traefik ACME DNS Challenge
• Authentik Proxy Provider
• gethomepage.dev

Next: VK Relay — Tunneling the Browser to a Local Agent Server