Operating on CI/CD Platform

Last updated 2026-07-29 ·ad7c94a

This is the operational companion to CI/CD Platform. That post covers the architecture. This one covers what you type to check health, trigger pipelines, and debug failures.

Two names appear throughout and are worth having up front. tekton-bot is a Gitea service account, created by hand at bootstrap, which owns the pull-mirror of the frank repo and whose API token drives every automated call below; its token lives in Infisical as GITEA_API_TOKEN and reaches the cluster as the gitea-api-token Secret in tekton-pipelines. el-<name> is Tekton’s own prefix: an EventListener named github-listener generates a Deployment, Service and pod called el-github-listener, so the resource you patch and the endpoint you POST to have different names. Commands that select by label want eventlistener=github-listener; commands that hit a URL want el-github-listener.

    graph TB
    subgraph github["GitHub"]
        gh_repo["frank repo"]
        gh_webhook["Webhook<br/>push events"]
    end

    subgraph hop["Hop Edge"]
        caddy["Caddy<br/>webhooks.hop.derio.net"]
    end

    subgraph frank["Frank Cluster"]
        subgraph gitea["Gitea"]
            gitea_svc["Mirror<br/>tekton-bot/frank"]
        end

        subgraph tekton["Tekton Pipelines"]
            el_listener["gitea-listener<br/>EventListener<br/>(svc el-gitea-listener)"]
            el_gh["github-listener<br/>EventListener<br/>(svc el-github-listener)"]
            pipeline["gitea-ci<br/>PipelineRun"]
            pull_sync["github-pull-sync<br/>PipelineRun"]
        end

        subgraph zot["Zot Registry"]
            zot_svc["192.168.55.210:5000"]
            cosign_sig["Cosign signatures"]
        end
    end

    gh_repo -->|"mirror sync"| gitea_svc
    gitea_svc -->|"webhook"| el_listener --> pipeline
    pipeline -->|"build & push"| zot_svc
    pipeline -->|"sign"| cosign_sig

    gh_webhook --> caddy -->|"Tailscale mesh"| el_gh --> pull_sync --> gitea_svc
  

What Healthy Looks Like

• All pods in gitea, tekton-pipelines, and zot namespaces are Running.
• ArgoCD apps show Synced and Healthy.
• All ExternalSecrets show SecretSynced.
• Gitea mirror updated_at is within the last 10 minutes.
• PipelineRuns complete successfully (no Failed state).
• Zot responds on https://192.168.55.210:5000/v2/.

Verify

# Pods across all CI/CD namespaces
kubectl get pods -n gitea -o wide
kubectl get pods -n tekton-pipelines -o wide
kubectl get pods -n zot -o wide

# ArgoCD
kubectl get applications -n argocd gitea gitea-extras \
  tekton-pipelines tekton-triggers zot

# ExternalSecrets
kubectl get externalsecret -n gitea
kubectl get externalsecret -n tekton-pipelines
kubectl get externalsecret -n zot

# Gitea mirror — the mirror repo is PRIVATE, so this needs the token
GITEA_URL="http://192.168.55.209:3000"
GITEA_TOKEN=$(kubectl -n tekton-pipelines get secret gitea-api-token \
  -o jsonpath='{.data.token}' | base64 -d)
curl -s -H "Authorization: token $GITEA_TOKEN" \
  "$GITEA_URL/api/v1/repos/tekton-bot/frank" | jq '{mirror, updated_at}'

# Zot — print the status code; the body is legitimately empty on success
curl -sk -o /dev/null -w '%{http_code}\n' https://192.168.55.210:5000/v2/

# Recent PipelineRuns
kubectl get pipelinerun -n tekton-pipelines --sort-by=.metadata.creationTimestamp | tail -5

Do not drop the Authorization header from the mirror check. Gitea answers an unauthenticated request for a private repo with 404, which jq turns into a tidy object full of nulls rather than an error. Both invocations, same repo, same second (2026-07-29):

$ curl -s "$GITEA_URL/api/v1/repos/tekton-bot/frank" | jq '{mirror, updated_at}'
{
  "mirror": null,
  "updated_at": null
}

$ curl -s -H "Authorization: token $GITEA_TOKEN" \
    "$GITEA_URL/api/v1/repos/tekton-bot/frank" | jq '{mirror, updated_at}'
{
  "mirror": true,
  "updated_at": "2026-07-28T22:10:37Z"
}

A missing token and a deleted mirror produce the same nulls. If you see nulls, check whoami before you check the mirror: curl -s -H "Authorization: token $GITEA_TOKEN" "$GITEA_URL/api/v1/user" | jq .login should print tekton-bot.

Steps

Trigger a Mirror Sync

Authorization: token wants an API token, not the admin password. Feeding it the password returns 401, and with curl -sf that is a silent non-zero exit with no output at all, which reads exactly like success in a script. Use tekton-bot’s token and print the code:

GITEA_TOKEN=$(kubectl -n tekton-pipelines get secret gitea-api-token \
  -o jsonpath='{.data.token}' | base64 -d)
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  "$GITEA_URL/api/v1/repos/tekton-bot/frank/mirror-sync" \
  -H "Authorization: token $GITEA_TOKEN"

200

200 means Gitea accepted the request, not that the pull succeeded. Confirm the sync landed by re-reading updated_at from the Verify block above.

View Pipeline Logs

# With tkn
tkn pipelinerun logs -n tekton-pipelines --last

# Without tkn — find the pod, then list its step containers before naming one
kubectl get pods -n tekton-pipelines -l tekton.dev/pipelineRun \
  --sort-by=.metadata.creationTimestamp | tail -5
kubectl get pod -n tekton-pipelines <pod-name> \
  -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'
kubectl logs -n tekton-pipelines <pod-name> --all-containers --prefix

Step container names are derived from the Task’s step names, so they are per-Task and there is no guessing them. --all-containers --prefix reads the lot and labels each line with its source, which is almost always what you want at 2am:

$ kubectl logs -n tekton-pipelines kid-laptops-main-sync-4ps98-pull-and-push-pod \
    --all-containers --prefix
[pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/prepare] 2026/07/27 17:12:41 Entrypoint initialization
[pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/place-scripts] 2026/07/27 17:12:43 Decoded script /tekton/scripts/script-0-wnbvz
[pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] Cloning into 'repo'...
[pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] Pushing 6b9482f2695402b7e08c925c3dc6f452a8bc6d71 → gitea:refs/heads/main
[pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] remote: Processed 1 references in total
[pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] To ssh://192.168.55.209:2222/derio-homelab/kid-laptops.git
[pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea]    5b89532..6b9482f  6b9482f2695402b7e08c925c3dc6f452a8bc6d71 -> main
[pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] OK

Cancel a PipelineRun

kubectl patch pipelinerun -n tekton-pipelines <name> \
  --type=merge -p '{"spec":{"status":"CancelledRunFinally"}}'

Verify an Image Signature

cosign verify --key apps/tekton/cosign.pub \
  --insecure-ignore-tlog --allow-insecure-registry \
  192.168.55.210:5000/<repo>/<image>:<tag>

Clean Up Old PipelineRuns

# Manual single-pipeline sweep
kubectl get pipelinerun -n tekton-pipelines \
  -o jsonpath='{range .items[?(@.status.conditions[0].status=="False")]}{.metadata.name}{"\n"}{end}' \
  | xargs -r kubectl delete pipelinerun -n tekton-pipelines

# Or trigger the CronJob
kubectl create job -n tekton-pipelines --from=cronjob/pipelinerun-ttl-gc \
  pipelinerun-ttl-gc-manual-$(date +%s)

# Its log ends with one summary line per swept namespace
kubectl logs -n tekton-pipelines job/<job-name> | grep 'pipelinerun-ttl-gc result'
# pipelinerun-ttl-gc result namespace=kid-laptops-ci deleted=12 remaining=1 cutoff=...

The CronJob sweeps the namespaces in its NAMESPACES list (tekton-pipelines, kid-laptops-ci), each with its own Role and RoleBinding. A namespace missing from that list is never swept. Its PipelineRuns and their workspace PVCs pile up, and because the workspace StorageClass is single-replica, each leaked PVC blocks a drain of the node that holds its only replica. A PR that adds a new PipelineRun namespace fails scripts/tests/test_pipelinerun_ttl_gc_namespaces.py until the namespace is added to the list.

Read-only real-work delivery checks

Quiet repositories do not need synthetic CI to prove health. ci-delivery-check reads every five minutes: main/open-PR refs must mirror within 30m of the source commit time, eligible Gitea statuses must match GitHub on that exact mirrored SHA within 10m. No eligible status is healthy when path filters skip CI. Grafana holds overdue counts, read errors, and absent/older-than-15m timestamp findings for 10m. Existing CI fault rules remain; both idle rules are deleted through provisioning tombstones when Grafana loads the merged configuration.

kubectl -n tekton-pipelines get cronjob ci-delivery-check
kubectl -n tekton-pipelines get jobs --sort-by=.metadata.creationTimestamp
kubectl -n tekton-pipelines logs job/<latest-ci-delivery-check-job>
gh api repos/<org>/<repo>/branches/main --jq .commit.sha
gh api --paginate 'repos/<org>/<repo>/pulls?state=open&per_page=100' --jq '.[] | [.number,.head.sha]'
gh api --paginate repos/<org>/<repo>/commits/<sha>/statuses

Compare authenticated Gitea branch commit.id, then newest per-context statuses from /api/v1/repos/<org>/<repo>/commits/<sha>/statuses (all pages). The bridge excludes any context starting tekton and maps skipped/warning to success. Descriptions must match exactly or their legitimate 140-character capped representation; confirm target URLs too. Resolve relative Gitea API targets against GITEA_WEB_URL (matching Gitea server.ROOT_URL) before comparing with absolute GitHub links. Equivalent relative/absolute links are healthy, while different run links still warn. Do not print credentials. Private Gitea reads need operator-selected cluster-admin access, not the dev profile. See the Tekton gotchas runbook for error/recovery details. Healthy metrics are zero ci_delivery_overdue and ci_delivery_read_error gauges with a fresh ci_delivery_last_check_timestamp_seconds, under job="ci-delivery-check". Full snapshot replacement clears closed PRs.

Manually Re-Trigger GitHub-primary Pull Sync

Run the reconciler first. Every 15 minutes it compares GitHub main (and every open PR’s head, against its sync-pr-<N> branch) against Gitea and pushes whatever has drifted for more than 10 minutes. Starting it by hand does the same thing now:

kubectl -n tekton-pipelines create job --from=cronjob/mirror-reconcile mirror-reconcile-manual-$(date +%s)
kubectl -n tekton-pipelines logs -f job/<that-name> --all-containers | grep mirror-reconcile
# mirror-reconcile repaired agentic-stoa/second-brain refs/heads/sync-pr-124 - -> ab45e07…
# mirror-reconcile result drift=1 repaired=1 skipped=0 failed=0 plan_errors=0

To re-run the webhook path itself for one PR, redeliver its pull_request delivery from the repo’s webhook settings, or with gh api -X POST repos/<org>/<repo>/hooks/<hook>/deliveries/<id>/attempts. Read delivery ids as strings: they exceed a float64, and jq silently rounds them to an id that does not exist.

A hand-built PipelineRun is the last resort. Its params and workspaces must match apps/tekton/pipelines/github-pull-sync.yaml exactly. Every value: below is an example:

kubectl create -n tekton-pipelines -f - <<'EOF'
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: github-pull-sync-manual-
spec:
  pipelineRef:
    name: github-pull-sync
  params:
    - {name: github-repo, value: <org>/<repo>}       # e.g. agentic-stoa/hum
    - {name: gitea-repo,  value: <org>/<repo>}
    - {name: sha,         value: <commit-sha>}
    - {name: ref-to,      value: refs/heads/main}    # or refs/heads/sync-pr-<N>
  taskRunTemplate:
    podTemplate:
      securityContext: {fsGroup: 65534}
  workspaces:
    - name: source
      volumeClaimTemplate:
        spec:
          accessModes: [ReadWriteOnce]
          storageClassName: longhorn-hdd
          resources: {requests: {storage: 2Gi}}
    - name: ssh-creds
      secret: {secretName: stoa-bot-ssh-key, defaultMode: 0600}
EOF

Recover

Gitea Mirror Not Updating

kubectl logs -n gitea deploy/gitea --tail=50 | grep -i mirror

Two things to check, both of which live somewhere non-obvious.

The mirror PATPersonal Access TokenA long-lived GitHub credential that stands in for a password on API calls. Frank prefers short-lived GitHub App installation tokens where it can — a PAT does not rotate on its own, so it is a standing secret.Read more. GITHUB_MIRROR_TOKEN is an Infisical key. It reaches the cluster through the gitea-secrets ExternalSecret in namespace gitea (apps/gitea/manifests/externalsecret-gitea.yaml) and lands as key github-mirror-token on the gitea-secrets Secret. The ExternalSecret only tells you the key was fetched:

$ kubectl -n gitea get externalsecret
NAME            STORETYPE            STORE       REFRESH INTERVAL   STATUS         READY
gitea-secrets   ClusterSecretStore   infisical   5m                 SecretSynced   True

SecretSynced and an expired PAT look identical from here, because Infisical will happily serve a dead token. The only thing that distinguishes them is GitHub’s answer, so if the ExternalSecret is green and the mirror still will not pull, the PAT is the suspect, not the plumbing.

ALLOWED_HOST_LIST. Declared at apps/gitea/values.yaml, but the Gitea chart renders its config into a Secret, not a ConfigMap, so that is where the live copy is:

$ kubectl -n gitea get secret gitea-inline-config -o jsonpath='{.data.webhook}' | base64 -d
ALLOWED_HOST_LIST=*.svc.cluster.local

Two traps stacked here. First, the value is scoped to in-cluster webhook delivery; mirror pulls are outbound and unaffected by it, so this is the setting to check when a webhook will not reach an EventListener, not when a mirror will not sync. Second, a values-only change to this file syncs the Secret without rolling the pod, so git and the live Secret can agree while the running Gitea serves the config it booted with. After any config change here, kubectl -n gitea rollout restart deploy/gitea and re-read the value from inside the pod.

A Commit Never Reached Gitea (no gitea-actions/* statuses)

The runner looks healthy, later commits get statuses, but one PR or push has none. Gitea never received that commit, so no Actions run exists. The mirror chain is fire-and-forget at two points. A webhook that fails delivery (a 502 while the listeners were down) is never retried by GitHub. A github-pull-sync run that times out is never re-run. Find which one it was:

# Did the delivery fail?   (redeliver = the fix for this one)
gh api 'repos/<org>/<repo>/hooks/<hook>/deliveries?per_page=50' \
  --jq '.[] | "\(.delivered_at) \(.event) \(.action) \(.status_code)"'
# Did a sync run, and how did it end?
kubectl -n tekton-pipelines get pipelinerun -l tekton.dev/pipeline=github-pull-sync \
  --sort-by=.metadata.creationTimestamp | tail

Either way, mirror-reconcile repairs it within ~25 minutes (the 15-minute schedule plus the 10-minute grace), and the repaired push fires Gitea Actions, so the gitea-actions/* statuses arrive. It does not re-run a Tekton CI body: for hum, content-factory and stoa-blog, and for cnc-ci’s test task, redeliver the PR webhook as well. kid-laptops main is also reconciled, using its own org token; it has no mirrored PR/status-bridge coverage. To repair now, run the reconciler by hand (see above). Since 2026-10-12 both EventListeners run two replicas on different hosts behind a PodDisruptionBudget, so draining one node no longer 502s webhooks.

Webhook Not Triggering a PipelineRun

# Gitea webhook path
kubectl logs -n tekton-pipelines -l app.kubernetes.io/managed-by=EventListener --tail=30

# Look for interceptor rejections
kubectl logs -n tekton-pipelines -l eventlistener=github-listener --tail=200 \
  | grep -E "Triggered|interceptor|HMAC"

Known causes:

• Gitea sends X-Gitea-Event, not X-GitHub-Event — the interceptor needs a CELCommon Expression LanguageA small, safe expression language for evaluating conditions inside configuration. Tekton's triggers use it to decide whether an incoming webhook should start a pipeline. filter, not the github interceptor.
• Caddy strips the event header — Hop’s Caddy may drop X-GitHub-Event on the webhooks relay. The EventListener sees a request with no event type.
• HMACHash-based Message Authentication CodeA signature computed from a payload and a shared secret. How a webhook receiver proves a request really came from the forge that claims to have sent it. mismatch — the webhook secret in GitHub doesn’t match STOA_GITHUB_WEBHOOK_SECRET in Frank.

PipelineRun Stuck in Pending

kubectl describe pipelinerun -n tekton-pipelines <name>

Check PVCPersistentVolumeClaimA Kubernetes request for durable storage. The pod names a claim and the storage layer — Longhorn on Frank — binds real disk behind it, so the data outlives the pod. provisioning first. Tekton workspaces use the longhorn-hdd storage class (single replica on pc-1’s HDD disks), and a PVC that cannot bind holds its PipelineRun in Pending for the full 1h timeout. The step never starts, so the run ends PipelineRunTimeout with no log at all:

kubectl -n tekton-pipelines get pvc | grep -v Bound
kubectl -n tekton-pipelines describe pvc <name> | tail -3
# … failed to provision volume with StorageClass "longhorn-hdd": … specified disk tag hdd does not exist
kubectl -n longhorn-system get nodes.longhorn.io pc-1 -o jsonpath='{range .spec.disks.*}{.tags}{" "}{end}'

“Disk tag does not exist” means no schedulable Longhorn disk carries the class’s tag. That is what happened on 2026-10-11, when #836 moved the workspaces to longhorn-hdd before any hdd disk was admitted. layer-25-ci-workspace-pvc-pending alerts once a PVC has been Pending for 15 minutes, and layer-25-pipelinerun-timeout alerts when the run itself times out. Once the disk is back, mirror-reconcile re-pushes the commits those timed-out syncs dropped.

PodSecurity Violation on Task Step

kubectl logs -c takes one container name and does not glob. -c step-* fails outright, and it fails in the least helpful way, by refusing to read anything:

$ kubectl logs -n tekton-pipelines kid-laptops-main-sync-4ps98-pull-and-push-pod -c 'step-*'
error: container step-* is not valid for pod kid-laptops-main-sync-4ps98-pull-and-push-pod out of: step-pull-from-github-push-to-gitea, prepare (init), place-scripts (init)

Use --all-containers, which is what you wanted anyway since a violation can surface in an init container:

kubectl logs -n tekton-pipelines <pod> --all-containers --prefix \
  | grep -i "permission denied\|psp\|podsecurity"

An empty result means the grep found nothing, not that the pod is clean. Re-run without the grep before concluding anything.

Fix: add securityContext to the Task step — runAsNonRoot: true, capabilities.drop: ["ALL"], seccompProfile.type: RuntimeDefault.

Pipeline Step Fails with permission denied on git

kubectl logs -n tekton-pipelines <pod> --all-containers --prefix

HOME=/ is read-only for UIDUser IdentifierThe number Linux actually checks for file permissions — the name is a lookup. In containers a uid mismatch against a mounted volume is the usual cause of a permission error. 65534. Set HOME=/tekton/home env var on the step.

Zot Returns 401 on Push

# Recreate the push credentials
ZOT_PASS=$(kubectl get secret -n tekton-pipelines zot-push-creds \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r '.auths["192.168.55.210:5000"].password')
crane auth login 192.168.55.210:5000 -u tekton-push -p "$ZOT_PASS" --insecure

If the password changed in Infisical, the ExternalSecret needs to re-sync. Check kubectl get externalsecret -n zot.

Operate the Gitea Actions Runner

The agentic-stoa mirrors run their GitHub Actions workflows on Frank via Gitea Actions (apps/gitea-runner/ — act_runner + DinD on pc-1).

# Is the runner alive and registered?
kubectl -n gitea-runner get pods
kubectl -n gitea-runner logs deploy/act-runner -c runner --tail=20
# Gitea admin view: http://192.168.55.209:3000/-/admin/actions/runners (expect state Idle)

# Watch a run's job containers appear inside DinD
kubectl -n gitea-runner exec deploy/act-runner -c dind -- docker ps

# Status bridge: did the result reach GitHub?
kubectl -n tekton-pipelines get pipelinerun | grep stoa-status-bridge

Operational notes:

• Capacity lives in apps/gitea-runner/manifests/config.yaml (runner.capacity: 2). Drop to 1 if pc-1 strains before considering a node move.

• DinD’s image cache is an emptyDir — wiped on pod restart, by design. The PVC keeps only the runner identity (/data/.runner) and the actions tool cache.

• Registration is one-shot. The runner registers once and persists identity on the PVC; rotating STOA_GITEA_RUNNER_TOKEN does NOT re-register. To force a fresh registration: scale to 0, delete /data/.runner (or the PVC), scale up.

• Mutation authority: CI_AUTHORITY decides which forge’s mutating jobs actually run. It is two variables, not one — each forge resolves it in its own Actions-variable namespace, so flipping only Gitea leaves GitHub’s unset, GitHub defaults to github, and both sides run. Gitea’s half is an org variable (agentic-stoa → Settings → Actions → Variables) and is readable over the API:

  GITEA_ADMIN=$(kubectl -n gitea get secret gitea-secrets -o jsonpath='{.data.username}' | base64 -d):$(kubectl -n gitea get secret gitea-secrets -o jsonpath='{.data.password}' | base64 -d)
  curl -s -u "$GITEA_ADMIN" \
    "$GITEA_URL/api/v1/orgs/agentic-stoa/actions/variables/ci_authority" | jq -r .data

  gitea
  

  This is the one call in this post that needs admin credentials rather than tekton-bot’s token, and the error says why: required=[read:organization], token scope=write:issue,write:repository,read:user, HTTP 403. Both halves were flipped to gitea on 2026-07-22; reversing needs both.

Missteps

Quick Reference

References

• Building Post — CI/CD Platform
• Tekton CLI
• cosign Verification