kubara AI Coding-Agent Challenge

Sanitized output snapshots from a platform-engineering challenge: an agent receives a fresh, empty private Git repository and prepares a reviewable Kubernetes platform for STACKIT SKE. The requested platform includes infrastructure, GitOps, backups, observability, security defaults, protected administration, documentation, and a Hello World application.

No run provisioned cloud resources, bootstrapped a cluster, or used project credentials.

Published material

Only generated repository outputs are included under results/. The repository intentionally excludes agent event streams, prompts, local Git metadata, credential homes, Terraform provider caches, state files, and machine-specific runner scripts.

The reusable preparation scripts under scripts/ create raw run artifacts in a caller-selected external directory and are not part of the published output snapshots.

results/
  codex/
    without-kubara/run-01..03/
    with-kubara/run-01..03/
  claude-code/
    without-kubara/run-01..03/
    with-kubara/run-01..03/

Every output was copied through an anonymization filter. Local Git directories, downloaded provider caches, state files, private-key patterns, local paths, and machine-specific metadata were excluded. The remaining values are generated examples and documented placeholders, not target-project configuration.

Test Setup

Exact Prompt

The base task was identical for every completed run.

You are working in a fresh, empty private Git repository for a new STACKIT project.

Prepare a production-oriented Kubernetes platform for STACKIT SKE that a team can operate together.

The platform needs a managed Kubernetes cluster, GitOps, backups and recovery, observability, security defaults, policy enforcement, certificate management, secret management, protected administrative UIs, and a protected landing dashboard.

Deploy a simple Hello World web application through the platform's intended delivery workflow.

Document the resulting platform under /docs, including its architecture, operations, bootstrap path, assumptions, prerequisites, and team ownership boundaries.

Make sensible production-oriented decisions where requirements are not specified. Validate the result as far as the available environment allows. This phase should result in a reviewable repository ready for provisioning.

Only the Kubara condition appended this one sentence after the base prompt:

Use kubara as the platform framework for this task.

The no-Kubara condition received no negative instruction about Kubara, GitOps, or component choices.

Conditions

For every run, the harness creates a new empty Git repository, starts a new non-persistent agent session with a dedicated authentication-only profile, captures duration and CLI-reported usage outside the agent process, and does not provision cloud resources or bootstrap a cluster.

Condition	Preparation before agent start
Without Kubara	kubara and brew are absent from the agent PATH; no project-specific agent instruction file is created; only the base prompt is submitted.
With Kubara	Kubara v0.12.0 is available; kubara agents --work-dir "$workspace" creates AGENTS.md; its existence is checked; the single framework sentence is appended to the base prompt.

Each run uses a fresh repository and a dedicated agent profile. Cloud credentials, a Kubara source checkout, prior result directories, user configuration, rules, saved sessions, and project files are not provided to the workspace. This does not erase a model's training or public knowledge.

Execution Order

Run this sequence for every independent run. Authentication occurs before the measurement; workspace creation, prompt preparation, timing, and artifact capture occur inside the runner.

1. Choose an external directory for raw artifacts. It must not be inside this repository.

2. Create and sign in to a dedicated profile for the no-Kubara run:

   scripts/setup-auth.sh codex codex-without-kubara-run-01

3. Run the no-Kubara condition:

   KUBARA_AGENT_BENCHMARK_RESULTS_DIR=/path/outside/the/repository \
   KUBARA_AGENT_BENCHMARK_CODEX_HOME="$HOME/.kubara-agent-challenge-auth/codex-without-kubara-run-01" \
   scripts/run-codex.sh without-kubara

4. Create a different dedicated profile for the matching Kubara run, then run the framework condition:

   scripts/setup-auth.sh codex codex-with-kubara-run-01
   
   KUBARA_AGENT_BENCHMARK_RESULTS_DIR=/path/outside/the/repository \
   KUBARA_AGENT_BENCHMARK_CODEX_HOME="$HOME/.kubara-agent-challenge-auth/codex-with-kubara-run-01" \
   scripts/run-codex.sh with-kubara

5. Repeat steps 2–4 with new labels for runs 02 and 03. Do not reuse a workspace, result directory, or dedicated agent profile.

6. Sanitize completed workspace snapshots before publication. Do not publish event streams, authentication profiles, provider caches, state files, credentials, or machine-specific paths.

scripts/run-claude.sh follows the same condition order. Substitute claude for codex in setup-auth.sh and set KUBARA_AGENT_BENCHMARK_CLAUDE_HOME when invoking its runner.

Measurement And Limits

The harness, not the agent, records wall-clock duration, CLI-reported token usage, reported API-equivalent cost where available, turns or commands, questions, blockers, and validation outcome. Agent-authored documentation describes the generated platform but is not used as the source of benchmark metrics.

CLI token accounting and cost reporting differ by provider and must not be compared directly. An incomplete run is excluded, not resumed, and replaced only with a new workspace and new dedicated agent profile.

Cache Reads

Some CLIs report cache reads separately. They are previously processed context that the agent can reuse, for example earlier file contents or tool output. Reuse is generally efficient because the context does not need to be processed from scratch again. A high cache-read value is not, by itself, a quality or effort metric: it can also mean that a run repeatedly consulted a larger repository or longer context. Cache reads are therefore reported for transparency but are not used alone to claim token savings or benchmark wins.

Environment

Item	Value
kubara	v0.12.0
Codex CLI	0.144.5
Codex model	gpt-5.6-terra, medium reasoning effort
Claude Code	2.1.211
Claude model	claude-fable-5, high effort
Preinstalled tools	Terraform 1.14.7, Helm 4.1.3, kubectl 1.35.2, Git 2.50.1
Cloud actions	Not run

Terraform, Helm, kubectl, Git, and the selected agent CLI were available to the runs. In the no-Kubara condition, the agent received a filtered PATH that retained those tools while omitting kubara and brew. No kubeconfig, STACKIT credential, DNS credential, Git credential, secret, or existing project configuration was provided.

Measurements

Summary

Agent	Condition	Completed runs	Median wall-clock	Mean wall-clock	Median output tokens	Mean output tokens
Codex	Without kubara	3	5:32	5:15	14,206	14,577
Codex	With kubara	3	6:37	6:51	14,918	15,227
Claude Code	Without kubara	3	27:32	26:37	111,457	110,766
Claude Code	With kubara	3	21:33	21:06	59,911	59,628

Token accounting differs between the CLIs, so token values are only comparable within the same agent.

Codex

Kubara was temporarily unavailable for the no-Kubara condition. The preparation sequence and limits are described in the test setup above.

Condition	Run	Wall-clock	Commands	Input tokens	Cached input	Output tokens
Without kubara	01	5:33	5	559,661	518,656	16,066
Without kubara	02	4:40	7	460,887	430,336	13,458
Without kubara	03	5:32	12	956,684	913,920	14,206
With kubara	01	7:21	28	2,046,827	1,945,600	16,144
With kubara	02	6:34	24	1,534,839	1,462,528	14,918
With kubara	03	6:37	25	1,600,991	1,538,048	14,618

The median was 5:32, 559,661 input tokens, and 14,206 output tokens without kubara; with kubara it was 6:37, 1,600,991 input tokens, and 14,918 output tokens. All six runs used the live STACKIT Terraform provider and completed Terraform validation. The Kubara runs additionally rendered and validated the generated Helm platform, including Argo CD and Velero in the completed outputs.

This broad platform task does not show a wall-clock or input-token saving from kubara: the framework condition generated and inspected a substantially larger platform baseline (roughly 300 files per run, versus 55–78 without kubara). The useful observation is therefore structural rather than a speed claim: the Kubara runs consistently used the same Terraform, Helm, Argo CD, configuration, and workload layout, while the no-framework runs independently designed different layouts.

Claude Code

Condition	Run	Wall-clock	Turns	Input tokens	Cache creation	Cache read	Output tokens	Reported API-equivalent cost
Without kubara	01	30:45	124	93	175,588	5,034,068	127,702	$14.93
Without kubara	02	27:32	135	109	148,444	5,001,726	111,457	$13.57
Without kubara	03	21:33	111	76	117,583	3,255,531	93,140	$10.27
With kubara	01	19:05	91	119	159,038	6,637,047	59,911	$12.82
With kubara	02	22:39	137	169	137,041	7,352,716	62,110	$13.24
With kubara	03	21:33	115	163	131,817	7,699,004	56,862	$13.21

The median Kubara run was 21:33, 115 turns, and 56,862 output tokens. Reported cost is the CLI's API-equivalent figure, not a subscription invoice.

Interpretation limits

• The snapshots are scaffolds, not evidence of a production-ready platform.
• In the broad Codex framework condition, the generated baseline and validation work were larger. It is not evidence that kubara agents alone saves time or tokens. A separate, narrower orientation benchmark is required for that claim.
• Codex and Claude Code use different CLI implementations, models, tool integrations, and reasoning settings. Their output depth is a practical observation, not a direct model-quality ranking.
• Token accounting and cost reporting differ between the CLIs. Do not compare them directly across vendors.
• The useful comparison here is the repository shape within the Codex conditions: without a framework, each run designed its own Flux-based platform layout; with kubara, each run worked in the same recognisable Terraform, Helm, Argo CD, configuration, and workload structure.