ImagePullBackOff and ErrImagePull: causes and fixes
By Hackademy. Updated . Output from a real Kubernetes v1.35 cluster.
The short answer
ErrImagePull and ImagePullBackOff mean the kubelet on the pod's node could not pull the container image, so the container never started. ErrImagePull is the failed attempt; ImagePullBackOff is the kubelet waiting before it tries again, with a delay that grows up to five minutes. The usual causes are a typo in the image name or tag, a tag that was never pushed or has been deleted, a private registry with no imagePullSecrets or wrong credentials, a registry rate limit, and a node that cannot reach the registry. To diagnose it, run kubectl get pods to find the stuck pods, then kubectl describe pod POD or kubectl get events and read the Failed event, which quotes the registry's answer, such as not found or unauthorized. Fix the image reference with kubectl set image, add or correct the pull secret, or, if a bad deploy caused it, run kubectl rollout undo to return to the last image that worked.
Spot the pods that cannot pull
A pull failure usually arrives with a deploy. Here a Deployment of five nginx pods was moved to an image tag that does not exist:
$ kubectl set image deployment/web nginx=nginx:9.99-nope
deployment.apps/web image updated$ kubectl get pods
NAME READY STATUS RESTARTS AGE
web-77657b99d8-bdfmv 1/1 Running 0 29s
web-77657b99d8-gnjdc 1/1 Running 0 66s
web-77657b99d8-pw722 1/1 Running 0 66s
web-77657b99d8-rhvq9 1/1 Running 0 66s
web-bb46ffd89-hjthc 0/1 ImagePullBackOff 0 20s
web-bb46ffd89-k5bhv 0/1 ImagePullBackOff 0 20s
web-bb46ffd89-kcs92 0/1 ImagePullBackOff 0 20sTwo things stand out. The new pods (the web-bb46ffd89 ReplicaSet) have zero restarts, because a container that was never created cannot restart. And four old pods are still Running: the rolling update will not remove more old pods than maxUnavailable allows while the new ones are not ready, so the site keeps serving.
Catch a pod at the right moment and STATUS reads ErrImagePull instead. It is the same failure, before the back-off has started:
$ kubectl get pods broken crash
NAME READY STATUS RESTARTS AGE
broken 0/1 ErrImagePull 0 15s
crash 0/1 CrashLoopBackOff 1 (8s ago) 15sRead the registry's answer in the events
The status says the pull failed; the events say why. kubectl describe pod POD ends with them, and kubectl get events lists them for the whole namespace. These are the lines for one of the new pods:
$ kubectl get events
30s Normal Pulling pod/web-bb46ffd89-hjthc Pulling image "nginx:9.99-nope"
29s Warning Failed pod/web-bb46ffd89-hjthc Failed to pull image "nginx:9.99-nope": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:9.99-nope": failed to resolve reference "docker.io/library/nginx:9.99-nope": docker.io/library/nginx:9.99-nope: not found
29s Warning Failed pod/web-bb46ffd89-hjthc Error: ErrImagePull
16s Normal BackOff pod/web-bb46ffd89-hjthc Back-off pulling image "nginx:9.99-nope"
16s Warning Failed pod/web-bb46ffd89-hjthc Error: ImagePullBackOffThe Failed to pull image message carries the registry's reply. Note how a short name like nginx was expanded to docker.io/library/nginx: an image without a registry host comes from Docker Hub. The reply points at the cause:
- not found or manifest unknown: the repository or tag does not exist. Check the spelling and that the tag was pushed.
- unauthorized, authentication required or 403: the registry wants credentials. Create a
docker-registrySecret and reference it inimagePullSecretson the pod or its ServiceAccount, in the pod's namespace. - toomanyrequests: the registry is rate limiting pulls. Authenticate, use a mirror, or pull less often.
- i/o timeout, no such host or connection refused: the node cannot reach the registry. Check DNS, proxies and firewalls from the node.
- no match for platform in manifest: the image has no build for the node's CPU architecture.
Fix the image, or roll back first
A rollout stuck on a bad image never finishes. kubectl rollout status waits until its timeout and gives up:
$ kubectl rollout status deployment/web --timeout=5s
Waiting for deployment "web" rollout to finish: 3 out of 5 new replicas have been updated...
error: timed out waiting for the conditionThere are two ways out. If you know the right image, set it with kubectl set image deployment/web nginx=IMAGE and the Deployment rolls forward. If you do not yet know, roll back to the last revision that worked and investigate afterwards:
$ kubectl rollout undo deployment/web
deployment.apps/web rolled backFor a credentials problem where the pod already names the pull secret, correcting the Secret's contents is enough: the kubelet keeps retrying on its back-off and uses the new credentials on the next attempt. Adding imagePullSecrets to a Deployment's template starts a rollout of new pods instead. Pull secrets on a ServiceAccount are copied into a pod only when it is created, so delete the stuck pods after adding them there.
Stop it happening again
- Deploy pinned tags or digests that your pipeline has just pushed, never a hand-typed tag.
- Keep
imagePullSecretson the namespace's ServiceAccount so every pod gets them without editing each spec. - Watch every deploy with
kubectl rollout statusand a timeout, so a bad image fails the pipeline instead of waiting quietly. - Practise the 3am version of this: the missions below push a bad tag mid-shift, and pass when every pod runs on an image that exists, however you get there.
Practise it on a simulated cluster
These missions in the Kubernetes Simulator break a cluster in this way and leave you to fix it. Free in your browser, nothing to install.
- Ship v2Static website, intermediate, about 10 min
- The 3am pageStatic website, intermediate, about 12 min
Questions
- What is the difference between ErrImagePull and ImagePullBackOff?
ErrImagePull is the result of a pull attempt that failed. ImagePullBackOff means the kubelet is waiting before the next attempt, with the delay growing up to five minutes. A pod with a bad image moves between the two until the image is fixed.
- Why does a pod in ImagePullBackOff have no logs?
The container was never created, because there was no image to create it from.
kubectl logshas nothing to read, so the only evidence is the pod's events, shown bykubectl describe podandkubectl get events.- Does imagePullPolicy cause ImagePullBackOff?
It decides when the kubelet pulls, not whether the pull can succeed. With
Always, every container start contacts the registry, so a registry outage or rate limit can stop pods starting even though the image was pulled before. WithIfNotPresent, a node that already has the image does not need the registry at all.