Application Observability and Maintenance

Kubernetes recognizes three kinds of probe, and each has a genuinely different consequence when it fails. A livenessProbe that fails gets the container restarted — use it for a process that can deadlock or get stuck in a way it will never recover from on its own. A readinessProbe that fails removes the Pod from every Service's list of endpoints, but does not restart anything — the container keeps running, it just stops receiving new traffic until the probe passes again. A startupProbe runs once at container start and blocks the liveness and readiness probes from running until it succeeds, giving a slow-initializing application room to boot; once it succeeds, it is never evaluated again for that container's lifetime.

Conflating liveness and readiness is a common, costly mistake: a liveness probe tuned as if it were a readiness check restarts a container that was only ever temporarily busy, and a readiness probe tuned as if it were a liveness check leaves a genuinely stuck process quietly serving no traffic forever rather than getting restarted.

Each probe picks exactly one mechanism. httpGet sends an HTTP request and treats status codes 200–399 as success — the right fit for anything that already speaks HTTP. tcpSocket only checks whether a port accepts a connection, a weaker signal than a real protocol check. exec runs a command inside the container and checks its exit code — flexible, but it spawns a new process on every single check, which can pile up if the container's main process is PID 1 with no init process to reap the children. grpc speaks the standard gRPC Health Checking Protocol directly and has been a stable, built-in probe mechanism since Kubernetes v1.27, with the one limitation that the port must be numeric — a named port is not supported.

Diagnosis almost always starts with kubectl get pods, whose STATUS column names the failure class — Pending, ContainerCreating, ImagePullBackOff, CrashLoopBackOff — and kubectl describe pod, whose Events section explains why. For a Pending Pod with no container ever created, Events is where a message like "0/4 nodes available: insufficient cpu" shows up; kubectl logs is empty because there is nothing to log yet. For a crashed container, describe's Last State shows Terminated with a reason — OOMKilled means the container tried to use more memory than its limit allowed and the kernel's out-of-memory killer ended it, a distinct failure from a probe failure or a bad exit code.

kubectl logs <pod> shows the current container instance's output; after a crash and restart, that instance is new, so kubectl logs <pod> --previous is what shows the crashed instance's actual output. -f follows logs live, and -c <container> selects one container in a multi-container Pod.

kubectl top pods and kubectl top nodes read from the Metrics API, which needs metrics-server (or an equivalent) deployed in the cluster — it is not part of a base Kubernetes install, so a fresh cluster returns an error rather than numbers. A repeatedly crashing container follows CrashLoopBackOff: Kubernetes waits before each restart with a delay that grows with each failure rather than retrying instantly every time.

When a container has no shell to kubectl exec into — a distroless image, or one deliberately stripped of tools — kubectl debug -it <pod> --image=<debug-image> --target=<container> attaches an ephemeral container that shares the target's process namespace, so you can inspect its processes and filesystem from a fresh, fully tooled container without rebuilding or replacing anything. This has been stable since Kubernetes v1.25, and once added, an ephemeral container cannot be removed — it stays until the Pod itself is gone.

Kubernetes' own deprecation policy treats API stability levels differently. A GA (v1) API version may be marked deprecated, but Kubernetes has never removed a GA API version within a major version — there is no fixed countdown the way there is for beta. A beta API version is deprecated no sooner than 9 months or 3 minor releases after it was first introduced, and stops being served no sooner than 9 months or 3 minor releases after that deprecation — whichever of the two durations is longer, in each case. An alpha API version carries no such guarantee at all and can be removed in any release with no prior deprecation notice.

This is why a manifest written years ago against a beta API can suddenly fail to apply after a routine cluster upgrade, with no warning beyond whatever the release notes said months earlier.

Two concrete examples worth knowing: batch/v1beta1 for CronJob was deprecated in v1.21 and stopped being served in v1.25, with batch/v1 available the whole time since v1.21; autoscaling/v2beta2 for HorizontalPodAutoscaler stopped being served in v1.26, with autoscaling/v2 available since v1.23. kubectl api-versions and kubectl api-resources query a live cluster for what it currently serves, which is the direct way to check whether a manifest's apiVersion fields still match reality before applying them.

kubectl convert used to ship as a built-in kubectl subcommand, but its deprecation was announced before v1.13 and it was removed from kubectl's core by v1.17. It now exists only as a separate kubectl-convert plugin, installed on its own (for example via krew), which rewrites a manifest file from an old apiVersion to a target one you specify.