Kubernetes Simulator

advanced · ~30 min

Serve the model

Ticket AI-31 — put the assistant's model into production. The platform team has added a GPU node pool: hackademy-gpu-worker, labelled nvidia.com/gpu.present=true and tainted nvidia.com/gpu=present:NoSchedule so ordinary pods stay off it. It has 2 NVIDIA L4 GPUs.

  • llm — ghcr.io/hackademy/llm-server:1.0, 2 replicas, one GPU each, on the GPU pool only. It listens on 8000.
  • The weights aren't in the image: ghcr.io/hackademy/model-fetch:1.0 downloads them to /models, and the server reads /models/llama-mini-7b.gguf.
  • Loading the model takes about 30 seconds. A replica must not get traffic until /ready says it's loaded.
  • It holds about 6Gi of memory once loaded — size its memory so it's never killed.
  • Clients call http://ai.local/v1 (try /v1/models).

GPUs are the most expensive thing in the cluster: every one of them should be doing work.

Objectives

  • The weights are fetched by an init container into a volume the server shares
  • 2 llm replicas ready on the GPU pool, one GPU each
  • Only loaded replicas get traffic
  • No replica killed for memory
  • ai.local/v1/models answers
  • Bonus: Find out how many GPUs the pool has left with kubectl describe node

Best on a computer with a keyboard. It works on a phone, with shortcuts under the terminal.