The Kubernetes Audit Worksheet

Print this, block four hours, and work through it with whoever knows the cluster best. Every command is read-only.

Section 0 — Context (15 min, no terminal)

Write down before looking at anything:

• Business: what runs on this cluster, and what does one hour of downtime cost?
• People: who can operate it? Who is the single point of failure?
• History: last incident, last upgrade, last restore test (dates, not vibes).
• Bill: monthly cloud + observability spend, best guess.

You'll compare these answers with reality at the end. The gap is your executive summary.

Section 1 — Inventory (30 min)

kubectl version                          # in support?
kubectl get nodes -o wide                # count, versions, ages, pressure
kubectl get ns                           # what exists
kubectl get deploy,sts,ds,cronjob -A     # workload inventory
helm ls -A                               # what Helm thinks is installed

Record: K8s version and support status, node count/type, workload count, and — critically — which workloads are NOT represented in any Git repo. That list is your unreproducible surface.

Section 2 — Deployment path (30 min)

Trace one real service from commit to production and write each step down. Then answer:

• Can anyone deploy by hand to prod? (If yes: who has, recently? Check kubectl rollout history.)
• Rollback: exact commands/steps, and when it was last exercised.
• Probes and shutdown: for the top 3 services, check readiness probes, preStop, terminationGracePeriodSeconds, rollout strategy, PDBs.

kubectl get deploy <top-service> -o yaml | grep -A5 -E "readinessProbe|preStop|strategy"

Section 3 — Access & secrets (45 min)

kubectl get clusterrolebindings -o wide | grep -iE "admin|edit"
kubectl auth can-i get secrets -n <shared-ns> --as=<a-normal-user>
kubectl get secrets -A | wc -l

Record: humans with cluster-admin (target: ~0 for daily use), service accounts with broad rights, secret source of truth (ESO/Vault/SOPS vs. hand-created), whether the API server is internet-reachable, and whether any Git repo contains a base64 secret (git grep -i "kind: Secret" across infra repos).

Section 4 — Cost (45 min)

If OpenCost/Kubecost isn't installed, note it as finding #1 and use what you have:

kubectl top nodes
kubectl top pods -A --sort-by=memory | head -30
kubectl describe nodes | grep -A8 "Allocated resources"

Record: requested vs. actually-used CPU/memory ratio (the over-provisioning factor), workloads with no requests, count of type: LoadBalancer services, log ingestion GB/day and retention, non-prod running out-of-hours, unattached volumes in the cloud console.

Section 5 — Observability (30 min)

• Where do engineers look first during an incident? (Ask two people separately; different answers = finding.)
• Open the alert history for the last 2 weeks: which alerts fired, which led to action? Non-actionable ones are noise debt.
• Do golden-signal dashboards exist for the top 3 services?
• Certificates: kubectl get certificates -A (cert-manager) or check ingress TLS expiry. Is renewal automated AND alerted on failure?

Section 6 — Backup & DR (30 min)

kubectl get backups -A          # velero, if present

Record: what is backed up (cluster state? volumes? databases?), where backups live (outside the blast radius?), last successful backup, last tested restore, and estimated time to rebuild everything from Git + backups. If the restore has never been tested, the audit's #1 recommendation writes itself.

Section 7 — Upgrade readiness (20 min)

kubent   # or: pluto detect-all-in-cluster    # deprecated APIs

Record: versions behind latest, deprecated API usage, add-on versions (ingress, cert-manager, CNI, CSI) and whether anyone owns their lifecycle, existence of a staging cluster or IaC to create one.

Section 8 — Synthesis (45 min)

Turn notes into three artifacts:

1. Risk map — every finding rated Impact (1–3) × Likelihood (1–3). Anything 6+ goes on page one.
2. Cost opportunity — over-provisioning factor + logging retention + orphans, expressed in currency/month.
3. Remediation backlog — max 10 items, ordered by risk score, each with an effort guess (hours/days/weeks). Default priority when tied: tested restore → rollback → secrets/RBAC → requests cleanup → cert automation → upgrade plan.

Finish by re-reading Section 0. The distance between what the team believed and what you found is the story the leadership summary should tell.

A note on scope

This worksheet deliberately excludes application code quality, CI pipeline internals, and cloud account security outside the cluster. Those matter, but mixing them in is how audits become three-week projects nobody finishes. Half a day, cluster-focused, three artifacts out. Done beats comprehensive.

Scorecard version with 0–100 scoring: free — take it here.