# Operating on CI/CD Platform Source: https://blog.derio.net/frank/docs/operating/22-cicd-platform/ Published: 2026-04-13 Updated: 2026-07-29 *Last updated 2026-07-29 ·[` ad7c94a `](https://github.com/derio-net/frank/commit/ad7c94af)* This is the operational companion to [CI/CD Platform](https://blog.derio.net/frank/docs/building/27-cicd-platform/). That post covers the architecture. This one covers what you type to check health, trigger pipelines, and debug failures. Two names appear throughout and are worth having up front. **` tekton-bot `** is a Gitea service account, created by hand at bootstrap, which owns the pull-mirror of the frank repo and whose API token drives every automated call below; its token lives in Infisical as ` GITEA_API_TOKEN ` and reaches the cluster as the ` gitea-api-token ` Secret in ` tekton-pipelines `. **` el- `** is Tekton’s own prefix: an EventListener named ` github-listener ` generates a Deployment, Service and pod called ` el-github-listener `, so the resource you patch and the endpoint you POST to have different names. Commands that select by label want ` eventlistener=github-listener `; commands that hit a URL want ` el-github-listener `. ```mermaid graph TB subgraph github["GitHub"] gh_repo["frank repo"] gh_webhook["Webhook
push events"] end subgraph hop["Hop Edge"] caddy["Caddy
webhooks.hop.derio.net"] end subgraph frank["Frank Cluster"] subgraph gitea["Gitea"] gitea_svc["Mirror
tekton-bot/frank"] end subgraph tekton["Tekton Pipelines"] el_listener["gitea-listener
EventListener
(svc el-gitea-listener)"] el_gh["github-listener
EventListener
(svc el-github-listener)"] pipeline["gitea-ci
PipelineRun"] pull_sync["github-pull-sync
PipelineRun"] end subgraph zot["Zot Registry"] zot_svc["192.168.55.210:5000"] cosign_sig["Cosign signatures"] end end gh_repo -->|"mirror sync"| gitea_svc gitea_svc -->|"webhook"| el_listener --> pipeline pipeline -->|"build & push"| zot_svc pipeline -->|"sign"| cosign_sig gh_webhook --> caddy -->|"Tailscale mesh"| el_gh --> pull_sync --> gitea_svc ``` ## What Healthy Looks Like - All pods in ` gitea `, ` tekton-pipelines `, and ` zot ` namespaces are ` Running `. - ArgoCD apps show ` Synced ` and ` Healthy `. - All ExternalSecrets show ` SecretSynced `. - Gitea mirror ` updated_at ` is within the last 10 minutes. - PipelineRuns complete successfully (no ` Failed ` state). - Zot responds on ` https://192.168.55.210:5000/v2/ `. ## Verify ```bash # Pods across all CI/CD namespaces kubectl get pods -n gitea -o wide kubectl get pods -n tekton-pipelines -o wide kubectl get pods -n zot -o wide # ArgoCD kubectl get applications -n argocd gitea gitea-extras \ tekton-pipelines tekton-triggers zot # ExternalSecrets kubectl get externalsecret -n gitea kubectl get externalsecret -n tekton-pipelines kubectl get externalsecret -n zot # Gitea mirror — the mirror repo is PRIVATE, so this needs the token GITEA_URL="http://192.168.55.209:3000" GITEA_TOKEN=$(kubectl -n tekton-pipelines get secret gitea-api-token \ -o jsonpath='{.data.token}' | base64 -d) curl -s -H "Authorization: token $GITEA_TOKEN" \ "$GITEA_URL/api/v1/repos/tekton-bot/frank" | jq '{mirror, updated_at}' # Zot — print the status code; the body is legitimately empty on success curl -sk -o /dev/null -w '%{http_code}\n' https://192.168.55.210:5000/v2/ # Recent PipelineRuns kubectl get pipelinerun -n tekton-pipelines --sort-by=.metadata.creationTimestamp | tail -5 ``` **Do not drop the ` Authorization ` header from the mirror check.** Gitea answers an unauthenticated request for a private repo with ` 404 `, which ` jq ` turns into a tidy object full of nulls rather than an error. Both invocations, same repo, same second (2026-07-29): ```console $ curl -s "$GITEA_URL/api/v1/repos/tekton-bot/frank" | jq '{mirror, updated_at}' { "mirror": null, "updated_at": null } $ curl -s -H "Authorization: token $GITEA_TOKEN" \ "$GITEA_URL/api/v1/repos/tekton-bot/frank" | jq '{mirror, updated_at}' { "mirror": true, "updated_at": "2026-07-28T22:10:37Z" } ``` A missing token and a deleted mirror produce the same nulls. If you see nulls, check ` whoami ` before you check the mirror: ` curl -s -H "Authorization: token $GITEA_TOKEN" "$GITEA_URL/api/v1/user" | jq .login ` should print ` tekton-bot `. ## Steps ### Trigger a Mirror Sync ` Authorization: token ` wants an **API token**, not the admin password. Feeding it the password returns ` 401 `, and with ` curl -sf ` that is a silent non-zero exit with no output at all, which reads exactly like success in a script. Use ` tekton-bot `’s token and print the code: ```bash GITEA_TOKEN=$(kubectl -n tekton-pipelines get secret gitea-api-token \ -o jsonpath='{.data.token}' | base64 -d) curl -s -o /dev/null -w '%{http_code}\n' -X POST \ "$GITEA_URL/api/v1/repos/tekton-bot/frank/mirror-sync" \ -H "Authorization: token $GITEA_TOKEN" ``` ```console 200 ``` ` 200 ` means Gitea accepted the request, not that the pull succeeded. Confirm the sync landed by re-reading ` updated_at ` from the Verify block above. ### View Pipeline Logs ```bash # With tkn tkn pipelinerun logs -n tekton-pipelines --last # Without tkn — find the pod, then list its step containers before naming one kubectl get pods -n tekton-pipelines -l tekton.dev/pipelineRun \ --sort-by=.metadata.creationTimestamp | tail -5 kubectl get pod -n tekton-pipelines \ -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}' kubectl logs -n tekton-pipelines --all-containers --prefix ``` Step container names are derived from the Task’s step names, so they are per-Task and there is no guessing them. ` --all-containers --prefix ` reads the lot and labels each line with its source, which is almost always what you want at 2am: ```console $ kubectl logs -n tekton-pipelines kid-laptops-main-sync-4ps98-pull-and-push-pod \ --all-containers --prefix [pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/prepare] 2026/07/27 17:12:41 Entrypoint initialization [pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/place-scripts] 2026/07/27 17:12:43 Decoded script /tekton/scripts/script-0-wnbvz [pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] Cloning into 'repo'... [pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] Pushing 6b9482f2695402b7e08c925c3dc6f452a8bc6d71 → gitea:refs/heads/main [pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] remote: Processed 1 references in total [pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] To ssh://192.168.55.209:2222/derio-homelab/kid-laptops.git [pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] 5b89532..6b9482f 6b9482f2695402b7e08c925c3dc6f452a8bc6d71 -> main [pod/kid-laptops-main-sync-4ps98-pull-and-push-pod/step-pull-from-github-push-to-gitea] OK ``` ### Cancel a PipelineRun ```bash kubectl patch pipelinerun -n tekton-pipelines \ --type=merge -p '{"spec":{"status":"CancelledRunFinally"}}' ``` ### Verify an Image Signature ```bash cosign verify --key apps/tekton/cosign.pub \ --insecure-ignore-tlog --allow-insecure-registry \ 192.168.55.210:5000//: ``` ### Clean Up Old PipelineRuns ```bash # Manual single-pipeline sweep kubectl get pipelinerun -n tekton-pipelines \ -o jsonpath='{range .items[?(@.status.conditions[0].status=="False")]}{.metadata.name}{"\n"}{end}' \ | xargs -r kubectl delete pipelinerun -n tekton-pipelines # Or trigger the CronJob kubectl create job -n tekton-pipelines --from=cronjob/pipelinerun-ttl-gc \ pipelinerun-ttl-gc-manual-$(date +%s) # Its log ends with one summary line per swept namespace kubectl logs -n tekton-pipelines job/ | grep 'pipelinerun-ttl-gc result' # pipelinerun-ttl-gc result namespace=kid-laptops-ci deleted=12 remaining=1 cutoff=... ``` The CronJob sweeps the namespaces in its ` NAMESPACES ` list (` tekton-pipelines `, ` kid-laptops-ci `), each with its own Role and RoleBinding. A namespace missing from that list is never swept. Its PipelineRuns and their workspace PVCs pile up, and because the workspace StorageClass is single-replica, each leaked PVC blocks a drain of the node that holds its only replica. A PR that adds a new PipelineRun namespace fails ` scripts/tests/test_pipelinerun_ttl_gc_namespaces.py ` until the namespace is added to the list. ### Read-only real-work delivery checks Quiet repositories do not need synthetic CI to prove health. ` ci-delivery-check ` reads every five minutes: main/open-PR refs must mirror within 30m of the source commit time, eligible Gitea statuses must match GitHub on that exact mirrored SHA within 10m. No eligible status is healthy when path filters skip CI. Grafana holds overdue counts, read errors, and absent/older-than-15m timestamp findings for 10m. Existing CI fault rules remain; both idle rules are deleted through provisioning tombstones when Grafana loads the merged configuration. ```bash kubectl -n tekton-pipelines get cronjob ci-delivery-check kubectl -n tekton-pipelines get jobs --sort-by=.metadata.creationTimestamp kubectl -n tekton-pipelines logs job/ gh api repos///branches/main --jq .commit.sha gh api --paginate 'repos///pulls?state=open&per_page=100' --jq '.[] | [.number,.head.sha]' gh api --paginate repos///commits//statuses ``` Compare authenticated Gitea branch ` commit.id `, then newest per-context statuses from ` /api/v1/repos///commits//statuses ` (all pages). The bridge excludes any context starting ` tekton ` and maps skipped/warning to success. Descriptions must match exactly or their legitimate 140-character capped representation; confirm target URLs too. Resolve relative Gitea API targets against ` GITEA_WEB_URL ` (matching Gitea ` server.ROOT_URL `) before comparing with absolute GitHub links. Equivalent relative/absolute links are healthy, while different run links still warn. Do not print credentials. Private Gitea reads need operator-selected cluster-admin access, not the dev profile. See the Tekton gotchas runbook for error/recovery details. Healthy metrics are zero ` ci_delivery_overdue ` and ` ci_delivery_read_error ` gauges with a fresh ` ci_delivery_last_check_timestamp_seconds `, under ` job="ci-delivery-check" `. Full snapshot replacement clears closed PRs. ### Manually Re-Trigger GitHub-primary Pull Sync Run the reconciler first. Every 15 minutes it compares GitHub ` main ` (and every open PR’s head, against its ` sync-pr- ` branch) against Gitea and pushes whatever has drifted for more than 10 minutes. Starting it by hand does the same thing now: ```bash kubectl -n tekton-pipelines create job --from=cronjob/mirror-reconcile mirror-reconcile-manual-$(date +%s) kubectl -n tekton-pipelines logs -f job/ --all-containers | grep mirror-reconcile # mirror-reconcile repaired agentic-stoa/second-brain refs/heads/sync-pr-124 - -> ab45e07… # mirror-reconcile result drift=1 repaired=1 skipped=0 failed=0 plan_errors=0 ``` To re-run the webhook path itself for one PR, redeliver its ` pull_request ` delivery from the repo’s webhook settings, or with ` gh api -X POST repos///hooks//deliveries//attempts `. Read delivery ids as strings: they exceed a float64, and ` jq ` silently rounds them to an id that does not exist. A hand-built PipelineRun is the last resort. Its params and workspaces must match ` apps/tekton/pipelines/github-pull-sync.yaml ` exactly. Every ` value: ` below is an example: ```bash kubectl create -n tekton-pipelines -f - <<'EOF' apiVersion: tekton.dev/v1 kind: PipelineRun metadata: generateName: github-pull-sync-manual- spec: pipelineRef: name: github-pull-sync params: - {name: github-repo, value: /} # e.g. agentic-stoa/hum - {name: gitea-repo, value: /} - {name: sha, value: } - {name: ref-to, value: refs/heads/main} # or refs/heads/sync-pr- taskRunTemplate: podTemplate: securityContext: {fsGroup: 65534} workspaces: - name: source volumeClaimTemplate: spec: accessModes: [ReadWriteOnce] storageClassName: longhorn-hdd resources: {requests: {storage: 2Gi}} - name: ssh-creds secret: {secretName: stoa-bot-ssh-key, defaultMode: 0600} EOF ``` ## Recover ### Gitea Mirror Not Updating ```bash kubectl logs -n gitea deploy/gitea --tail=50 | grep -i mirror ``` Two things to check, both of which live somewhere non-obvious. **The mirror PAT (Personal Access Token) (**Personal Access Token**A long-lived GitHub credential that stands in for a password on API calls. Frank prefers short-lived GitHub App installation tokens where it can — a PAT does not rotate on its own, so it is a standing secret.[Read more](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)) .** ` GITHUB_MIRROR_TOKEN ` is an Infisical key. It reaches the cluster through the ` gitea-secrets ` ExternalSecret in namespace ` gitea ` (` apps/gitea/manifests/externalsecret-gitea.yaml `) and lands as key ` github-mirror-token ` on the ` gitea-secrets ` Secret. The ExternalSecret only tells you the key was *fetched*: ```console $ kubectl -n gitea get externalsecret NAME STORETYPE STORE REFRESH INTERVAL STATUS READY gitea-secrets ClusterSecretStore infisical 5m SecretSynced True ``` ` SecretSynced ` and an expired PAT look identical from here, because Infisical will happily serve a dead token. The only thing that distinguishes them is GitHub’s answer, so if the ExternalSecret is green and the mirror still will not pull, the PAT is the suspect, not the plumbing. **` ALLOWED_HOST_LIST `.** Declared at ` apps/gitea/values.yaml `, but the Gitea chart renders its config into a **Secret**, not a ConfigMap, so that is where the live copy is: ```console $ kubectl -n gitea get secret gitea-inline-config -o jsonpath='{.data.webhook}' | base64 -d ALLOWED_HOST_LIST=*.svc.cluster.local ``` Two traps stacked here. First, the value is scoped to in-cluster webhook delivery; mirror *pulls* are outbound and unaffected by it, so this is the setting to check when a webhook will not reach an EventListener, not when a mirror will not sync. Second, a values-only change to this file syncs the Secret without rolling the pod, so git and the live Secret can agree while the running Gitea serves the config it booted with. After any config change here, ` kubectl -n gitea rollout restart deploy/gitea ` and re-read the value from inside the pod. ### A Commit Never Reached Gitea (no ` gitea-actions/* ` statuses) The runner looks healthy, later commits get statuses, but one PR or push has none. Gitea never received that commit, so no Actions run exists. The mirror chain is fire-and-forget at two points. A webhook that fails delivery (a 502 while the listeners were down) is never retried by GitHub. A ` github-pull-sync ` run that times out is never re-run. Find which one it was: ```bash # Did the delivery fail? (redeliver = the fix for this one) gh api 'repos///hooks//deliveries?per_page=50' \ --jq '.[] | "\(.delivered_at) \(.event) \(.action) \(.status_code)"' # Did a sync run, and how did it end? kubectl -n tekton-pipelines get pipelinerun -l tekton.dev/pipeline=github-pull-sync \ --sort-by=.metadata.creationTimestamp | tail ``` Either way, ` mirror-reconcile ` repairs it within ~25 minutes (the 15-minute schedule plus the 10-minute grace), and the repaired push fires Gitea Actions, so the ` gitea-actions/* ` statuses arrive. It does **not** re-run a Tekton CI body: for hum, content-factory and stoa-blog, and for cnc-ci’s ` test ` task, redeliver the PR webhook as well. kid-laptops main is also reconciled, using its own org token; it has no mirrored PR/status-bridge coverage. To repair now, run the reconciler by hand (see above). Since 2026-10-12 both EventListeners run two replicas on different hosts behind a PodDisruptionBudget, so draining one node no longer 502s webhooks. ### Webhook Not Triggering a PipelineRun ```bash # Gitea webhook path kubectl logs -n tekton-pipelines -l app.kubernetes.io/managed-by=EventListener --tail=30 # Look for interceptor rejections kubectl logs -n tekton-pipelines -l eventlistener=github-listener --tail=200 \ | grep -E "Triggered|interceptor|HMAC" ``` Known causes: - **Gitea sends ` X-Gitea-Event `, not ` X-GitHub-Event `** — the interceptor needs a CEL (Common Expression Language) (**Common Expression Language**A small, safe expression language for evaluating conditions inside configuration. Tekton's triggers use it to decide whether an incoming webhook should start a pipeline.) filter, not the ` github ` interceptor. - **Caddy strips the event header** — Hop’s Caddy may drop ` X-GitHub-Event ` on the webhooks relay. The EventListener sees a request with no event type. - **HMAC (Hash-based Message Authentication Code) (**Hash-based Message Authentication Code**A signature computed from a payload and a shared secret. How a webhook receiver proves a request really came from the forge that claims to have sent it.) mismatch** — the webhook secret in GitHub doesn’t match ` STOA_GITHUB_WEBHOOK_SECRET ` in Frank. ### PipelineRun Stuck in Pending ```bash kubectl describe pipelinerun -n tekton-pipelines ``` Check PVC (PersistentVolumeClaim) (**PersistentVolumeClaim**A Kubernetes request for durable storage. The pod names a claim and the storage layer — Longhorn on Frank — binds real disk behind it, so the data outlives the pod.) provisioning first. Tekton workspaces use the ` longhorn-hdd ` storage class (single replica on pc-1’s HDD disks), and a PVC that cannot bind holds its PipelineRun in Pending for the full 1h timeout. The step never starts, so the run ends ` PipelineRunTimeout ` with no log at all: ```bash kubectl -n tekton-pipelines get pvc | grep -v Bound kubectl -n tekton-pipelines describe pvc | tail -3 # … failed to provision volume with StorageClass "longhorn-hdd": … specified disk tag hdd does not exist kubectl -n longhorn-system get nodes.longhorn.io pc-1 -o jsonpath='{range .spec.disks.*}{.tags}{" "}{end}' ``` “Disk tag does not exist” means no schedulable Longhorn disk carries the class’s tag. That is what happened on 2026-10-11, when #836 moved the workspaces to ` longhorn-hdd ` before any ` hdd ` disk was admitted. ` layer-25-ci-workspace-pvc-pending ` alerts once a PVC has been Pending for 15 minutes, and ` layer-25-pipelinerun-timeout ` alerts when the run itself times out. Once the disk is back, ` mirror-reconcile ` re-pushes the commits those timed-out syncs dropped. ### PodSecurity Violation on Task Step ` kubectl logs -c ` takes one container name and does **not** glob. ` -c step-* ` fails outright, and it fails in the least helpful way, by refusing to read anything: ```console $ kubectl logs -n tekton-pipelines kid-laptops-main-sync-4ps98-pull-and-push-pod -c 'step-*' error: container step-* is not valid for pod kid-laptops-main-sync-4ps98-pull-and-push-pod out of: step-pull-from-github-push-to-gitea, prepare (init), place-scripts (init) ``` Use ` --all-containers `, which is what you wanted anyway since a violation can surface in an init container: ```bash kubectl logs -n tekton-pipelines --all-containers --prefix \ | grep -i "permission denied\|psp\|podsecurity" ``` An empty result means the grep found nothing, not that the pod is clean. Re-run without the ` grep ` before concluding anything. Fix: add ` securityContext ` to the Task step — ` runAsNonRoot: true `, ` capabilities.drop: ["ALL"] `, ` seccompProfile.type: RuntimeDefault `. ### Pipeline Step Fails with ` permission denied ` on git ```bash kubectl logs -n tekton-pipelines --all-containers --prefix ``` ` HOME=/ ` is read-only for UID (User Identifier) (**User Identifier**The number Linux actually checks for file permissions — the name is a lookup. In containers a uid mismatch against a mounted volume is the usual cause of a permission error.) 65534. Set ` HOME=/tekton/home ` env var on the step. ### Zot Returns 401 on Push ```bash # Recreate the push credentials ZOT_PASS=$(kubectl get secret -n tekton-pipelines zot-push-creds \ -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r '.auths["192.168.55.210:5000"].password') crane auth login 192.168.55.210:5000 -u tekton-push -p "$ZOT_PASS" --insecure ``` If the password changed in Infisical, the ExternalSecret needs to re-sync. Check ` kubectl get externalsecret -n zot `. ## Operate the Gitea Actions Runner The agentic-stoa mirrors run their GitHub Actions workflows on Frank via Gitea Actions (` apps/gitea-runner/ ` — act_runner + DinD on pc-1). ```bash # Is the runner alive and registered? kubectl -n gitea-runner get pods kubectl -n gitea-runner logs deploy/act-runner -c runner --tail=20 # Gitea admin view: http://192.168.55.209:3000/-/admin/actions/runners (expect state Idle) # Watch a run's job containers appear inside DinD kubectl -n gitea-runner exec deploy/act-runner -c dind -- docker ps # Status bridge: did the result reach GitHub? kubectl -n tekton-pipelines get pipelinerun | grep stoa-status-bridge ``` Operational notes: - **Capacity** lives in ` apps/gitea-runner/manifests/config.yaml ` (` runner.capacity: 2 `). Drop to 1 if pc-1 strains before considering a node move. - **DinD’s image cache is an emptyDir** — wiped on pod restart, by design. The PVC keeps only the runner identity (` /data/.runner `) and the actions tool cache. - **Registration is one-shot.** The runner registers once and persists identity on the PVC; rotating ` STOA_GITEA_RUNNER_TOKEN ` does NOT re-register. To force a fresh registration: scale to 0, delete ` /data/.runner ` (or the PVC), scale up. - **Mutation authority**: ` CI_AUTHORITY ` decides which forge’s mutating jobs actually run. It is **two variables, not one** — each forge resolves it in its own Actions-variable namespace, so flipping only Gitea leaves GitHub’s unset, GitHub defaults to ` github `, and both sides run. Gitea’s half is an org variable (agentic-stoa → Settings → Actions → Variables) and is readable over the API: ```bash GITEA_ADMIN=$(kubectl -n gitea get secret gitea-secrets -o jsonpath='{.data.username}' | base64 -d):$(kubectl -n gitea get secret gitea-secrets -o jsonpath='{.data.password}' | base64 -d) curl -s -u "$GITEA_ADMIN" \ "$GITEA_URL/api/v1/orgs/agentic-stoa/actions/variables/ci_authority" | jq -r .data ``` ```console gitea ``` This is the one call in this post that needs admin credentials rather than ` tekton-bot `’s token, and the error says why: ` required=[read:organization], token scope=write:issue,write:repository,read:user `, HTTP 403. Both halves were flipped to ` gitea ` on 2026-07-22; reversing needs both. ## Missteps | What we assumed | Why it was wrong | What it cost | | --- | --- | --- | | Gitea webhooks use the same format as GitHub webhooks | Gitea sends ` X-Gitea-Event `, not ` X-GitHub-Event `. The ` github ` interceptor silently rejects non-GitHub events. | Switched to CEL interceptor that matches both formats. | | ` HOME=/ ` works for non-root Tekton task steps | The ` git-clone ` task writes to HOME, which is ` / ` — a read-only filesystem for UID 65534. | Set ` HOME=/tekton/home ` on every step that needs git. | | ` resources ` in Tekton Task YAML is equivalent to ` computeResources ` | The field was renamed. Using the old name causes ` ComparisonError ` in ArgoCD because the API normalises it. | Replaced all ` resources ` blocks with ` computeResources `. | | Caddy on Hop passes all HTTP headers through to the upstream | Caddy’s reverse proxy strips ` X-GitHub-Event ` unless explicitly configured. GitHub webhooks arrived at the EventListener without event headers. | Added ` header_up X-GitHub-Event ` to the Caddy relay route. | | Rotating the cosign key is a one-file swap: commit the new ` cosign.pub ` and verification keeps working | Old signatures stay valid with the old public key, but every consumer must know about both keys. If only the new ` cosign.pub ` is committed, old images fail verification. | Keep both keys published for as long as any unexpired image carries the old signature, and ` cosign verify ` against each in turn rather than assuming one. There is no separate rotation runbook to link: ` docs/runbooks/manual-operations.yaml ` covers first-time key *generation* (` cicd-cosign-keypair ` — private half into Infisical as ` COSIGN_KEY `, public half committed to ` apps/tekton/cosign.pub `), and a rotation is that procedure plus the dual-key window. | | The PipelineRun TTL (Time To Live) (**Time To Live**How long a value stays valid before it must be fetched again. Sets how long a stale DNS answer or cache entry can keep being wrong after the fix.) GC is a nice-to-have cleanup | Before the GC was implemented, accumulated task pods from finished runs pushed the ` kube_pod_status_ready ` alert into false-positive territory. | Added the CronJob and rewrote the alert query to use deployment-scoped metrics. | ## Quick Reference | Command | What It Does | | --- | --- | | ` kubectl get pods -n gitea -o wide ` | Gitea status | | ` kubectl get pipelinerun -n tekton-pipelines ` | Recent pipeline runs | | ` tkn pipelinerun logs -n tekton-pipelines --last ` | Latest pipeline logs | | ` curl -sk https://192.168.55.210:5000/v2/ ` | Zot health | | ` cosign verify --key apps/tekton/cosign.pub ... ` | Verify image signature | | ` kubectl logs -n tekton-pipelines -l eventlistener=github-listener ` | EventListener logs | | ` kubectl logs -n tekton-pipelines --all-containers --prefix ` | All step logs from one task pod (` -c ` does not glob) | | ` kubectl create job --from=cronjob/pipelinerun-ttl-gc ... ` | Force PipelineRun GC | ## References - [Building Post — CI/CD Platform](https://blog.derio.net/frank/docs/building/27-cicd-platform/) - [Tekton CLI](https://tekton.dev/docs/cli/) - [cosign Verification](https://docs.sigstore.dev/cosign/verifying/)