The First 10 Things I Would Check in a Messy Kubernetes Cluster

Imagine being handed a cluster with no documentation, one engineer who half-remembers building it, a growing bill, and a business that depends on it. This is the most common state of small-company Kubernetes I hear about: not broken, but fragile in ways nobody can enumerate.

Here's the exact order I'd triage it, and why this order. Everything below is read-only — you can do the whole assessment without changing a thing.

1. kubectl get nodes and the version question

First: is the Kubernetes version still in support? A shocking number of messy clusters run versions that are 2+ years old, which means every other fix inherits an upgrade project. Also check node conditions — memory/disk pressure that's been "always there" is a finding, not a fact of life.

2. What's actually running vs. what's in Git

Diff reality against the repos. kubectl get deploy,sts,ds -A versus what any Git repo claims. The gap between them is your "tribal knowledge inventory" — every resource that exists only in the cluster is something you cannot rebuild. In messy clusters this gap is typically 20–40% of workloads: manual hotfixes, "temporary" cronjobs, a Helm release someone upgraded by hand.

3. Who can do what: the RBAC five-minute audit

kubectl get clusterrolebindings -o wide | grep -i admin. Count the humans and service accounts with cluster-admin. In messy clusters the answer is "everyone, plus the CI system, plus a service account created for a demo in 2023." Then the scarier one: can regular namespace users read secrets? Usually yes.

4. Secrets: where do they actually live

Three bad patterns to look for: secrets committed to Git in plain YAML, secrets created by hand with no source of truth (kubectl create secret and a prayer), and one giant shared secret every app mounts. Any of these means secret rotation is impossible, which means it has never happened.

5. Resource requests vs. reality

Pull actual usage (Prometheus, or kubectl top) and compare to requests. The classic messy-cluster signature: requests copy-pasted from a tutorial (500m/512Mi everywhere), some workloads with no requests at all, and total requested CPU 3–5× actual usage. This is where the cost story usually lives — over-provisioning is the #1 reported cause of Kubernetes cost growth, and it's visible in twenty minutes.

6. The bill, split by what drives it

Cluster compute is often not the biggest line. Check: log ingestion and retention (the silent budget killer — logging bills quietly reaching tens of thousands per month is one of the most repeated complaints in the ecosystem), cross-AZ traffic, load balancers per service instead of shared ingress, and orphaned volumes/IPs from deleted workloads. If OpenCost isn't installed, installing it is the single highest-leverage read-only-ish action available.

7. Certificates and domains: what expires, and when

kubectl get certificates -A if cert-manager exists; otherwise check the ingress TLS secrets' expiry dates manually. Find out whether renewal is automated and — crucially — whether anything alerts when renewal fails. "The cert expired on a Saturday" is a top-three small-team outage.

8. What happens when a pod dies

Pick the most important service. Does it have readiness probes that check something real? Does it shut down gracefully or drop in-flight requests? Kill one pod in a safe window and watch the error rate. If a single pod restart causes visible 503s, every deploy and every node upgrade has been a mini-outage — usually nobody has connected those dots.

9. Backups: exist, and restorable?

Does Velero (or anything) exist? When did it last succeed? Where are backups stored — if it's inside the same cluster or cloud account, that's not disaster recovery, that's disaster co-location. Then the question that ends most conversations: "when did you last test a restore?" I have never triaged a messy cluster where the answer was a date.

10. The bus-factor interview

Not a kubectl command. Sit with the engineer who owns the cluster and ask: what are you the only person who knows how to do? Write the list down verbatim. That list — usually 5 to 15 items — is the real remediation backlog. Everything above is symptoms; this list is the disease.

What comes out the other side

Ninety minutes of read-only checks produce three artifacts:

1. A risk map — what could take production down and how likely it is (usually: restore never tested, certs semi-manual, single point of human failure).
2. A cost estimate — the requests-vs-usage gap and log retention alone typically identify 30–60% savings potential.
3. A prioritized backlog — ordered by "what hurts first," not by what's architecturally elegant.

The order of fixes is almost always: tested restore → rollback path → secrets/RBAC → requests cleanup → cert automation → upgrade plan. Notice that none of those are new tools. Messy clusters rarely need another tool. They need ownership, and a list.

I've packaged this triage as a 50-question scorecard — free — take it here.