Projects, mission 3 of 3
Serve the model
advanced, about 30 minutes, in the Kubernetes Simulator
Start mission
Free in your browser, nothing to install.
The briefing
Ticket AI-31 — put the assistant's model into production. The platform team has added a GPU node pool: hackademy-gpu-worker, labelled nvidia.com/gpu.present=true and tainted nvidia.com/gpu=present:NoSchedule so ordinary pods stay off it. It has 2 NVIDIA L4 GPUs.
- llm —
ghcr.io/hackademy/llm-server:1.0, 2 replicas, one GPU each, on the GPU pool only. It listens on 8000. - The weights aren't in the image:
ghcr.io/hackademy/model-fetch:1.0downloads them to/models, and the server reads/models/llama-mini-7b.gguf. - Loading the model takes about 30 seconds. A replica must not get traffic until
/readysays it's loaded. - It holds about 6Gi of memory once loaded — size its memory so it's never killed.
- Clients call http://ai.local/v1 (try
/v1/models).
GPUs are the most expensive thing in the cluster: every one of them should be doing work.
What you'll do
- The weights are fetched by an init container into a volume the server shares
- 2 llm replicas ready on the GPU pool, one GPU each
- Only loaded replicas get traffic
- No replica killed for memory
- ai.local/v1/models answers
- Bonus star: Find out how many GPUs the pool has left with kubectl describe node
The mission checks the cluster, not your keystrokes: any fix that works, passes.
More Projects missions
- 1. Launch the bookstore
- 2. Launch the arena
- 3. Serve the model