Roll back a bad deployment with kubectl rollout undo

A bad deploy rarely announces itself. The Deployment keeps serving from its old pods, and the rollout simply never finishes. kubectl rollout status with a timeout makes that visible, which is why a pipeline should always run it:

$ kubectl rollout status deployment/web --timeout=5s
Waiting for deployment "web" rollout to finish: 3 out of 5 new replicas have been updated...
error: timed out waiting for the condition

Here the new image tag did not exist, so the new pods sat in ImagePullBackOff. Other bad deploys look different: new pods in CrashLoopBackOff, pods Running but never Ready because a probe fails, or pods that are healthy but serve the wrong thing. In every case the quickest way back to a working service is the revision before.

kubectl rollout history lists the revisions the Deployment remembers, one per pod template:

$ kubectl rollout history deployment/web
deployment.apps/web
REVISION  CHANGE-CAUSE
1         <none>
2         <none>
3         <none>

CHANGE-CAUSE is filled from the kubernetes.io/change-cause annotation, so it reads <none> unless your pipeline sets it. To see what a revision actually ran, ask for it by number:

$ kubectl rollout history deployment/web --revision=2
deployment.apps/web with revision #2
Pod Template:
  Labels:	app=web
	pod-template-hash=5974c5bb9c
  Containers:
   nginx:
    Image:	nginx:1.27
    Port:	<none>
    Host Port:	<none>
    Environment:	<none>
    Mounts:	<none>
  Volumes:	<none>
  Node-Selectors:	<none>
  Tolerations:	<none>

How far back you can go is set by the Deployment's revisionHistoryLimit, 10 by default. Older ReplicaSets are deleted and their revisions with them.

With no flags, undo goes back one revision, which is what you want straight after a bad deploy:

$ kubectl rollout undo deployment/web
deployment.apps/web rolled back

The history afterwards shows how undo works. It does not rewind: it copies the old template forward as a new revision. Revision 2 has gone from the list and come back as revision 4, after the failed revision 3:

$ kubectl rollout history deployment/web
deployment.apps/web
REVISION  CHANGE-CAUSE
1         <none>
3         <none>
4         <none>

That matters if you undo twice: the second undo goes back to the bad revision, because it is now the previous one. To land on a specific version, read the history and name it with --to-revision. A number that is no longer in the history is refused:

$ kubectl rollout undo deployment/web --to-revision=9
error: unable to find specified revision 9 in history

If a bad version is still replacing pods, kubectl rollout pause deployment/NAME stops it where it is, so fewer pods run the bad version while you read the history. Undo is refused while the Deployment is paused, because an undo is itself a rollout:

$ kubectl rollout undo deployment/web --to-revision=2
error: you cannot rollback a paused deployment; resume it first with 'kubectl rollout resume' and try again

Resume and undo straight after, then follow the rollback to the end:

• kubectl rollout resume deployment/web
• kubectl rollout undo deployment/web --to-revision=2
• kubectl rollout status deployment/web until it prints successfully rolled out

• Only the pod template is versioned. The replica count, the update strategy and other Deployment fields stay as they are.
• ConfigMaps and Secrets are separate objects. If a bad config change caused the outage, undoing the Deployment brings back the old template but not the old values, unless each version of the config has its own name.
• If you manage manifests in Git or with a tool such as Helm, the next apply will put the bad version back. Revert it at the source too.

The missions below put this under pressure: a deploy that breaks mid-shift, and a release that has to be stopped before it reaches every pod. They check the cluster's state when you are done, so rolling forward with a fixed image passes too.