AI-assisted DevOps

• A platform that goes from bare Proxmox to a running RKE2 cluster and apps, for networks with no internet. One command builds it; every stage is safe to re-run.
• Built largely with AI agents, which makes it a good test bed for them.
• An eval matrix runs agent harnesses against local models on tasks from the repo, scoring quality, time and tokens.
• Next: training models to manage infrastructure themselves.

The pipeline

task up runs every stage in order:

1. Packer builds an Oracle Linux 9 template with an unattended install, and skips the build if nothing has changed.
2. Terraform creates the VMs. The hypervisor module is swappable.
3. Terraform’s outputs become the Ansible inventory.
4. The offline bundle is fetched, with every checksum verified.
5. Ansible installs RKE2 with the same roles whether the network is online or air-gapped.
6. A Zarf package, built on the connected side and deployed inside the offline network, brings cert-manager, an internal certificate authority, Argo Workflows and the operator wizard.
7. Argo Workflows installs the apps (Longhorn, CloudNativePG, Keycloak) in dependency order.

I tested it live on Proxmox by tearing it down and rebuilding from nothing, twice. On a re-run nothing changes: Ansible reports zero changes, Terraform has nothing to do, and Zarf converges.

The operator wizard

• One Go binary is the API, a Svelte web UI and a CLI.
• A voice-guided first boot, using local text-to-speech and speech recognition, so it works with no internet.
• An Explore page that runs k9s in a browser terminal.
• The whole platform also ships as a single bootable ISO, built and published by tagged releases in Gitea Actions.

Building it with agents

• It started as a bake-off: two agent harnesses with two local models each built a first version from the same design doc, in parallel.
• Most of the code since was written by Claude Code, working from written plans, with local model agents on smaller tasks. About 150 commits in under four weeks.

The eval matrix

Agents write documentation well enough to look right, so I wanted to measure how often they’re actually right.

• Harnesses: Hermes and OpenCode, running headless.
• Models: two local models served from Covert-GPT, so every run costs nothing but power.
• Tasks: write the platform doc for one subsystem (Ansible, Terraform, Packer, the wizard, Zarf, workflows). Each run works in a fresh, isolated copy of the repo. Git worktrees didn’t work: both harnesses followed the link back, edited the real repo, and could read the answer key.
• Scoring: a deterministic grader with nine pass/fail checks for invented file paths, task names and versions, coverage and scope, plus a weighted score. An optional blind LLM judge gives a second opinion. Time, tokens and API calls are recorded for every run.

First full run:

Harness and model	Passed	Median score	Median time	Median tokens
Hermes, Ornith 35B	6/6	96.7	12 min	521k
OpenCode, Ornith 35B	5/6	97.6	10 min	347k
OpenCode, Muse Glimmer 30B	3/6	96.4	14 min	754k
Hermes, Muse Glimmer 30B	2/6	72.8	25 min	1.17M

The same model changed a lot depending on the harness. Most of Muse Glimmer’s failures were file paths it made up. It’s one run per cell, so this is a direction, not a verdict.