CRD to Pod: Feature Injection Without Forking the Operator

Part 2 of 3 in the Agent Infrastructure series. Part 1 was the war story of getting one interactive CLI to run headless. This is the architecture that schedules all of them. Part 3 covers credential lifecycle across providers.

Part 1 ended with a single agent running in a pod: a CLI inside a tmux session, a bridge sidecar in front of it, an init script that pre-writes the auth files. That gets you one agent. The interesting problem starts when you want ten, of five different types, and you do not want to fork the controller every time a new CLI ships.

The naive instinct is to treat each agent type as a new code path: new type, new template, new branch in the reconcile loop. That scales linearly with agent types and dies around three of them. What scales is a small set of operator primitives that do not care whether you are running Claude Code, Codex, Pi, or whatever ships next quarter.

This is a walkthrough of that architecture, from a Kubebuilder operator running in production.

The boring choice that pays off: templates vs presets

The first decision is the one that quietly determines everything downstream: do not put CLI quirks in the generic deployment path.

There are two kinds of thing the operator schedules, and they want different code:

The temptation is to unify these into one "deployment" abstraction. Resist it. The moment your generic service path knows what ❯ means, you have a controller that is one giant special case wearing a trenchcoat. Keep the CLI knowledge quarantined in the template path, and keep services boring.

Presets as the agent-type registry

For the CLI agents, the unit of extension is a preset: a flat map keyed by agent type. Each entry declares the things that are genuinely per-tool.

cliAgentPresets["claude-code"] = preset{
    Command:           "claude --dangerously-skip-permissions",
    PromptMarker:      "❯",
    AutoAcceptPattern: []string{`\[Y/n\]`},
    // ...
}
cliAgentPresets["codex"] = preset{ Command: "codex --full-auto", /* ... */ }
cliAgentPresets["pi"]    = preset{ /* ... */ }

Adding a new CLI agent is a preset entry plus an image. No new reconciliation code, no new branch in the controller. The reconcile loop reads the preset and renders the same pod shape it always does.

The test for whether your extension point is in the right place: adding the fifth agent type should touch exactly as much code as adding the second. If it touches more, your agent type is a code path, not data.

Features as sidecar and init declarations

Capabilities are the second primitive. A workspace declares them per agent as a plain array:

agents:
  - type: claude-code
    features:
      - type: repo-setup
      - type: bridge
      - type: exposer
      - type: k8s-control

The operator resolves each feature to either a sidecar or an init container, with conventional ports and mounts:

The agent image never changes. You grant a capability by adding a feature, not by rebuilding a container.

flowchart TD
  cr[Workspace CR] --> rec[Reconcile]
  rec --> dispatch{agent or service?}
  dispatch -->|agent| tmpl[template + preset]
  dispatch -->|service| preset[service preset]
  tmpl --> pod[Pod spec]
  preset --> pod
  subgraph pod[Pod]
    direction LR
    init1[init: repo-setup] --> init2[init: post-setup] --> main[main: CLI in tmux]
    main -.-> s1[sidecar: bridge]
    main -.-> s2[sidecar: exposer]
    main -.-> s3[sidecar: k8s-control]
  end

Why this is infrastructure-as-skill

The payoff of the feature array is conceptual, not just ergonomic. The K8s control sidecar is the same shape as any other MCP-speaking process: an HTTP port, an MCP schema, a known mount. The agent does not know it is running on Kubernetes. It knows there is an MCP at localhost that lets it create deployments.

That indirection is the whole point. The agent enables a skill; it does not import a platform. Move the K8s control sidecar out of the pod and into a separate service tomorrow, and nothing in the agent changes. Infrastructure stops being a container the operator owns and becomes a capability the agent opts into.

Init container sequencing matters more than people admit

The order of init containers is not a detail. It is load-bearing.

The sequence that works is repo-setup (clone the repos, install credentials) then post-setup (write the CLI-specific bootstrap files, for example the config that bypasses onboarding) then the main container. Get that order wrong and the CLI starts before its onboarding bypass exists, which lands you right back in the crash-loop from Part 1.

This is the deeper lesson connecting the two posts: init containers are how you compensate for tools that have no headless mode. The tmux bridge drives the TUI; the init sequence makes sure the TUI has everything it needs before it launches. One without the other is half a fix.

The CRD shape

Four resource types, each with one job:

flowchart TD
  T[Tenant] --> W[Workspace]
  W --> C[controllers - runtime agents]
  W --> A[agents - pre-spun workers]
  W --> S[services - supporting workloads]
  A --> F[features per agent]

• Tenant owns the namespace, NetworkPolicy, and ResourceQuota. It is the blast-radius boundary.
• Workspace declares the working set: controllers[], agents[], services[], and features[] per agent. It is the unit of "what is running."

Reconciliation moves a Workspace through Pending → Provisioning → Ready, and writes an audit ConfigMap capturing the exact input the operator saw. That snapshot is worth more than it looks the first time you have to answer "what did the operator actually render last Tuesday."

The migration nobody writes about

Operators accumulate history, and history means migrations. We had legacy credential Secrets under ad-hoc names that needed to converge on a unified {platform}-llm-{provider} scheme. Rather than a one-shot script, the migration runs on reconcile: best-effort, non-fatal, retried on the next pass if it fails.

That posture (idempotent, non-fatal, eventually-consistent) is the right default for any data migration inside a reconcile loop. A migration that hard-fails reconciliation takes the whole workspace down with it. A migration that retries quietly converges. Nobody blogs about operator migrations, and then everybody writes the same buggy one-shot version.

Reconciliation gotchas worth your attention

These are the bugs that do not show up in the Kubebuilder book, in roughly increasing subtlety:

• Credential changes do not always re-roll the pod. Reconcile sees the Secret exists, not that it is newer. A mounted Secret updating does not restart anything on its own. The fix is to annotate the Pod with a hash of the Secret data, so a credential change is a spec change, which is a rollout.
• When they do re-roll, they re-roll too much. A naive watch re-rolls every pod in the tenant on any credential change. The granularity is wrong: the rollout should be scoped to the agents that actually use that credential. (Part 3 picks this thread up directly, because it is really a credential-to-agent mapping problem.)

Both are the same class of bug: Kubernetes will not tell you "this thing is newer," only "this thing exists." Anywhere your correctness depends on freshness, you have to encode freshness yourself, usually as a hash annotation.

Gateway switching, a primitive that pays off twice

One feature flag changes the runtime contract without touching the pod template. When the workspace enables the gateway, the operator stops mounting credential Secrets directly. Instead it injects ANTHROPIC_BASE_URL / OPENAI_BASE_URL env pointing at an in-namespace gateway, plus a per-agent gateway token.

Same pod, different contract. The agent talks to a local gateway instead of straight to the provider, and it has no idea anything changed. This is the exact primitive Part 3 builds on to refresh OAuth tokens without ever handing the credential proxy access to the Kubernetes API.

Takeaway

The architecture comes down to two primitives and one discipline. Presets make a new agent type into data, not code. Features make a capability into a sidecar declaration, not an image rebuild. And the discipline is keeping CLI-specific knowledge out of the generic path so the reconcile loop stays one path instead of N.

Do that and adding the fifth agent type costs the same as adding the second, which is the only definition of "scales" that matters when you are the one on call.

Part 3 takes the credential side seriously: how to refresh and rotate OAuth tokens for these agents without giving the proxy that touches the request path any access to the Kubernetes API.

Resources

• Kubebuilder Book: the operator framework this is built on.
• Sidecar containers in Kubernetes: the native sidecar lifecycle that features lean on.
• Model Context Protocol: what the K8s control and onchain sidecars speak.
• Previous: Containerizing Interactive CLIs: The Tmux Bridge Pattern.
• Next: The OAuth Broker That Never Touches the Kubernetes API.