Covert-GPT

It started as a way to learn local LLMs from the ground up. The goal now is a private reasoning and coding platform that can do real DevOps agent work with no outside dependencies, and that could run with no internet at all. The hard parts turned out to be the hardware, not Kubernetes.

The cluster

• Two NVIDIA DGX Spark (GB10 Grace Blackwell) GPU workers: arm64, 128 GB of unified memory each, and dual 100 GbE ConnectX-7 cards.
• Three control-plane VMs on my Proxmox cluster, running amd64. That makes it a mixed-architecture cluster, so every image has to be multi-arch.
• RKE2 with Cilium in eBPF mode, plus Longhorn, MinIO and NFS for the model cache.
• Ansible prepares the nodes and installs RKE2, then Helm installs everything else. One Taskfile wraps it: task up rebuilds from stock Ubuntu in about three hours, most of it downloading ~160 GB of weights and images.
• I ran ArgoCD for three months and took it out: the controller cost more than it gave for a cluster this size.

Serving

• vLLM runs one resident model per GPU node.
• LiteLLM is the gateway in front of every model. It handles virtual keys and spend tracking, logs every call to object storage and exports metrics to Prometheus.
  • If a prompt is too big for one model’s context window, it falls back to a local model with a bigger window instead of returning an error.
• ComfyUI does image and video generation on the same two cards, behind one stable hostname whichever node it’s on.
• GPU profiles: each card is split into scheduling slots with time-slicing. A profile (coding, graphics, reasoning-only, idle) switches a node between workloads by changing replica counts only, with preflight checks, so nothing is reinstalled.
• Dashboards for tokens per second, KV-cache use, prefix-cache hits and time-to-first-token percentiles, next to Hubble network flows and Loki logs.

What I learned by measuring

• On GB10, memory bandwidth is the limit. Dense models decode slowly, so I moved to mixture-of-experts models. One ran at 53.8 tok/s, about 6.7x a dense model of the same size.
• Good output isn’t enough. A 30B dense model gave good answers but ran at 13 tok/s against 30 for the model it would have replaced, so I rejected it.
• Speculative decoding can crash quietly. It sped one model up to about 17 tok/s, then crashed the engine under concurrent long prompts. The engine exits with code 0, so Kubernetes shows “Completed” instead of a failure.
• Unified memory breaks the usual limits. A container memory limit doesn’t cap GPU allocations: one workload held ~85 GiB under a 48 GiB limit and locked up a node. Tuning quantization and memory utilization gave the node back about 23 GiB.
• Load testing through the real gateway reached 128 concurrent requests with zero errors once I fixed a gateway memory bottleneck; before the fix it topped out around 10. The honest conclusion: it’s good for 1 to 3 developers and about 10x short of the multi-agent goal.
• Silent bugs are the expensive ones. The wrong tool-call parser quietly turned nested arguments into strings. In-cluster DNS names without a trailing dot quietly resolved to the wrong place.

About 490 commits since February 2026. Nine Helm charts. CI runs a secret scan, Helm lint and render, and Ansible syntax checks on every push.