Part 1 of 3 in the Agent Infrastructure series. Part 2 covers the operator that schedules these pods, and Part 3 covers credential lifecycle across providers.
There is a pattern showing up everywhere right now: run an AI CLI as a long-lived workload in Kubernetes. The pod holds the code, the credentials, and the MCP servers. An HTTP API drives the agent. It is a clean model, and on a laptop it works on the first try.
Then you put it in a pod and the CLI crash-loops before it does a single useful thing. Not because of anything you did, but because these tools are TUIs designed for a developer at a keyboard, and a pod does not have one.
This is the story of how we got Claude Code and Codex running headless: eight failed attempts, three independent root causes that all had to be fixed at once, and the one piece of the architecture that worked out of the box and quietly carried the whole thing.
Why the obvious fixes fail
Everyone tries the same things first, in roughly the same order. None of them work for a modern AI CLI, and it is worth understanding why, because each failure tells you something about what these tools actually are.
yes |and friends. Piping confirmations covers a[Y/n]prompt. It does nothing for an OAuth screen that wants you to open a browser.- The skip flags.
--dangerously-skip-permissions(Claude Code) and--full-auto(Codex) skip permission prompts. They do not skip onboarding. Different gate, still closed. - Writing the config by hand. Setting
hasCompletedOnboarding: trueinsettings.jsonlooks like the answer. The TUI is not gated on that flag alone, so it launches the first-run flow anyway. - Env-var API keys. Exporting
ANTHROPIC_API_KEYorOPENAI_API_KEYfeels like the right move. Modern CLIs distinguish a subscription OAuth token from a direct API key and reject the wrong shape, sometimes silently.
Each of these works against an older, simpler CLI. The AI tools drive their own terminal UI, do OAuth round trips, and treat the first-run prompt as a hard wall. That combination is what breaks the standard playbook.
Three root causes, not one
The trap here is thinking you have one bug. We had three, and fixing any two of them still left the pod crash-looping. They are independent, which is exactly why this took eight attempts instead of two.
| Root cause | What it looks like | Why it is independent |
|---|---|---|
| No headless mode | No --skip-onboarding or --headless flag exists; auth state is not fully persisted across restarts | A tool limitation, not a config you can set |
| OAuth tokens are not API keys | Console OAuth key returns Invalid API key; an OpenAI JWT from auth.openai.com is not an sk-... key | A credential-shape mismatch, independent of onboarding |
| The health probe kills onboarding | Probe times out at 60s, but onboarding is seven screens plus an OAuth round trip; pod restarts, progress lost | A scheduling problem, independent of the other two |
The third one is the classic mistake dressed up in new clothes. A liveness probe that fires during a slow startup is an old footgun. It just hurts more here because the "startup" is a multi-screen human workflow that takes minutes, not a server binding a port in two seconds.
If your fix only addresses one of these, the pod still crash-loops and you will conclude your fix did not work. It did. There were just two more.
The tmux bridge, explained
Here is the part that worked out of the box. The pod runs three containers, and the interesting one sits in the middle.
flowchart LR
client[HTTP / SSE client] -->|prompts in| bridge
bridge -->|events out| client
client -->|read-only files| exposer
subgraph pod[Pod]
agent["agent<br/>CLI inside tmux session"]
bridge["bridge<br/>HTTP / SSE sidecar"]
exposer["exposer<br/>read-only file server"]
journal[(on-disk session journal)]
workspace[(workspace files)]
end
bridge -->|send-keys| agent
agent -->|capture-pane| bridge
agent -.writes.-> journal
bridge -.tails.-> journal
exposer -.reads.-> workspaceagentruns the CLI itself, started withtmux new-session -d -s agent '<cli> <flags>'. The CLI believes it is attached to a real terminal, because as far as it can tell, it is.bridgespeaks HTTP and SSE on one side and the tmux session on the other. It writes input withtmux send-keysand reads output withtmux capture-pane. Prompts go in over HTTP; structured events come out over SSE by tailing the CLI's on-disk session journal.exposeris a read-only file-serving sidecar for the workspace. It is not part of the bridge story, but it rounds out the pod.
The mental model that matters: the terminal session is the source of truth, and the bridge is a typed IPC surface on top of it. You are not reimplementing the CLI's protocol. You are giving the outside world an HTTP handle on a TUI that was never meant to have one.
This piece never fought us. Every other attempt was a distraction from the fact that the bridge was already doing its job.
Polling, prompts, and signals
The bridge has to answer one deceptively hard question: is the model done, or is it just thinking? There is no event for that, so you infer it from the pane.
- Output detection polls the pane roughly every 100ms, exits early after about 5s of no new characters (idle), and hard-caps at 30s so a stuck session cannot block forever.
- Prompt markers tell the bridge the CLI is ready for input. It pattern-matches the prompt glyph (
❯for Claude Code) and auto-accept patterns like[Y/n]. - Auth without copying state. The bridge writes a bearer token to a known path inside the pod so in-pod CLIs authenticate against it without anyone shuttling credentials around by hand.
- Signals.
SIGTERMto the pod has to walk throughtmux send-keysto reach the CLI cleanly. Skip that and you get zombie sessions every time a pod churns.
None of this is elegant. Screen-scraping a terminal to detect completion is exactly as fragile as it sounds. It works because the alternative, waiting for these tools to ship a real machine interface, is not a plan you can deploy this quarter.
What actually ships
We weighed four options. Only one survives contact with production.
| Option | Idea | Verdict |
|---|---|---|
| A. Pre-auth init script | Init container pre-creates ~/.claude/ and ~/.codex/ with the right files so the CLI starts past onboarding | Ships. |
| B. Direct API keys | Support sk-ant-... / sk-... credentials directly | Works, but reinvents credential shape per provider and bypasses the subscription path users actually have |
| C. Pre-auth baked into image | Run onboarding once locally, copy the dotfiles into the image | Tokens expire, and now your image is a credential |
D. Upstream --headless | Ask the vendors for a real headless flag | Correct long-term answer, useless short-term answer |
Option A is what runs. An init container writes the auth and config files the CLI expects, so it boots straight past the first-run wall. Two supporting changes make it stick:
- Bump the bridge timeout to 300s or more to tolerate the slow initial startup window.
- Use
claude -p "ready"in exec mode as the health check before the session switches to interactive. That separates "is the binary alive" from "is the TUI idle," which is the distinction the 60s probe was getting wrong.
Gotchas and the generalizable lesson
If you build one of these, here is what will bite you, roughly in order:
- Separate liveness from readiness. TUI onboarding is a readiness concern. Treating it as liveness is what restarts the pod mid-flow.
- Session state is pod-local unless you write it out. If restarts have to preserve auth, put the CLI's dotfiles directory on a PVC.
- Keystroke scripting is fragile. Screens reorder and prompt text changes between versions. Prefer pre-populated state over scripted navigation everywhere you can.
exec-mode health checks beat TCP/HTTP checks for tools that do not serve HTTP. Ask the binary a question; do not poke a port it never opened.- Handle
SIGTERMthrough tmux or accept a graveyard of zombie sessions on every deploy.
The broader takeaway is older than any of these tools. When you put a human-first program in an environment without a human, you are not configuring it, you are impersonating the human. The tmux bridge is just an honest acknowledgment of that. It does not pretend the CLI is a server. It drives the CLI exactly the way a person would, and then puts an HTTP API where the person used to be.
Reach for this pattern when you have to host a TUI somewhere humanless: a CI job driving an interactive tool, an agent platform, a remote sandbox. Skip it the moment the tool ships a real headless mode. Use that instead, and send the maintainers a thank-you note.
Resources
- Claude Code documentation: the CLI this pattern was first built around.
- tmux: the session multiplexer doing the quiet heavy lifting.
- Kubernetes probe configuration: the liveness-vs-readiness distinction that this whole post turns on.
- Next in the series: CRD to Pod: Feature Injection Without Forking the Operator.