Kubernetes Simulator
advanced · ~12 min
The 2am failure
nightly-export has run every night for months. You're on call.
When a scheduled run fails, two jobs remain: stop the next run failing too, and make up the run that was missed. A CronJob reads its Secrets fresh each time it starts a Job — unlike a long-running pod.
Objectives
- Find out why the export is failing
- Put the new token in the warehouse Secret
- Get an export to complete on the new token
- Bonus: Backfill now with kubectl create job --from=cronjob/nightly-export
Best on a computer with a keyboard. It works on a phone, with shortcuts under the terminal.