Architecting Out Loud
Back to posts
Agent Infrastructure · Part 1 of 3kubernetesgodevopsai-agentsdistributed-systems

Containerizing Interactive CLIs: The Tmux Bridge Pattern

Jun 26, 20268 min read

Part 1 of 3 in the Agent Infrastructure series. Part 2 covers the operator that schedules these pods, and Part 3 covers credential lifecycle across providers.

There is a pattern showing up everywhere right now: run an AI CLI as a long-lived workload in Kubernetes. The pod holds the code, the credentials, and the MCP servers. An HTTP API drives the agent. It is a clean model, and on a laptop it works on the first try.

Then you put it in a pod and the CLI crash-loops before it does a single useful thing. Not because of anything you did, but because these tools are TUIs designed for a developer at a keyboard, and a pod does not have one.

This is the story of how we got Claude Code and Codex running headless: eight failed attempts, three independent root causes that all had to be fixed at once, and the one piece of the architecture that worked out of the box and quietly carried the whole thing.


Why the obvious fixes fail

Everyone tries the same things first, in roughly the same order. None of them work for a modern AI CLI, and it is worth understanding why, because each failure tells you something about what these tools actually are.

  • yes | and friends. Piping confirmations covers a [Y/n] prompt. It does nothing for an OAuth screen that wants you to open a browser.
  • The skip flags. --dangerously-skip-permissions (Claude Code) and --full-auto (Codex) skip permission prompts. They do not skip onboarding. Different gate, still closed.
  • Writing the config by hand. Setting hasCompletedOnboarding: true in settings.json looks like the answer. The TUI is not gated on that flag alone, so it launches the first-run flow anyway.
  • Env-var API keys. Exporting ANTHROPIC_API_KEY or OPENAI_API_KEY feels like the right move. Modern CLIs distinguish a subscription OAuth token from a direct API key and reject the wrong shape, sometimes silently.

Each of these works against an older, simpler CLI. The AI tools drive their own terminal UI, do OAuth round trips, and treat the first-run prompt as a hard wall. That combination is what breaks the standard playbook.


Three root causes, not one

The trap here is thinking you have one bug. We had three, and fixing any two of them still left the pod crash-looping. They are independent, which is exactly why this took eight attempts instead of two.

Root causeWhat it looks likeWhy it is independent
No headless modeNo --skip-onboarding or --headless flag exists; auth state is not fully persisted across restartsA tool limitation, not a config you can set
OAuth tokens are not API keysConsole OAuth key returns Invalid API key; an OpenAI JWT from auth.openai.com is not an sk-... keyA credential-shape mismatch, independent of onboarding
The health probe kills onboardingProbe times out at 60s, but onboarding is seven screens plus an OAuth round trip; pod restarts, progress lostA scheduling problem, independent of the other two

The third one is the classic mistake dressed up in new clothes. A liveness probe that fires during a slow startup is an old footgun. It just hurts more here because the "startup" is a multi-screen human workflow that takes minutes, not a server binding a port in two seconds.

If your fix only addresses one of these, the pod still crash-loops and you will conclude your fix did not work. It did. There were just two more.


The tmux bridge, explained

Here is the part that worked out of the box. The pod runs three containers, and the interesting one sits in the middle.

flowchart LR
  client[HTTP / SSE client] -->|prompts in| bridge
  bridge -->|events out| client
  client -->|read-only files| exposer
  subgraph pod[Pod]
    agent["agent<br/>CLI inside tmux session"]
    bridge["bridge<br/>HTTP / SSE sidecar"]
    exposer["exposer<br/>read-only file server"]
    journal[(on-disk session journal)]
    workspace[(workspace files)]
  end
  bridge -->|send-keys| agent
  agent -->|capture-pane| bridge
  agent -.writes.-> journal
  bridge -.tails.-> journal
  exposer -.reads.-> workspace
  • agent runs the CLI itself, started with tmux new-session -d -s agent '<cli> <flags>'. The CLI believes it is attached to a real terminal, because as far as it can tell, it is.
  • bridge speaks HTTP and SSE on one side and the tmux session on the other. It writes input with tmux send-keys and reads output with tmux capture-pane. Prompts go in over HTTP; structured events come out over SSE by tailing the CLI's on-disk session journal.
  • exposer is a read-only file-serving sidecar for the workspace. It is not part of the bridge story, but it rounds out the pod.

The mental model that matters: the terminal session is the source of truth, and the bridge is a typed IPC surface on top of it. You are not reimplementing the CLI's protocol. You are giving the outside world an HTTP handle on a TUI that was never meant to have one.

This piece never fought us. Every other attempt was a distraction from the fact that the bridge was already doing its job.


Polling, prompts, and signals

The bridge has to answer one deceptively hard question: is the model done, or is it just thinking? There is no event for that, so you infer it from the pane.

  • Output detection polls the pane roughly every 100ms, exits early after about 5s of no new characters (idle), and hard-caps at 30s so a stuck session cannot block forever.
  • Prompt markers tell the bridge the CLI is ready for input. It pattern-matches the prompt glyph ( for Claude Code) and auto-accept patterns like [Y/n].
  • Auth without copying state. The bridge writes a bearer token to a known path inside the pod so in-pod CLIs authenticate against it without anyone shuttling credentials around by hand.
  • Signals. SIGTERM to the pod has to walk through tmux send-keys to reach the CLI cleanly. Skip that and you get zombie sessions every time a pod churns.

None of this is elegant. Screen-scraping a terminal to detect completion is exactly as fragile as it sounds. It works because the alternative, waiting for these tools to ship a real machine interface, is not a plan you can deploy this quarter.


What actually ships

We weighed four options. Only one survives contact with production.

OptionIdeaVerdict
A. Pre-auth init scriptInit container pre-creates ~/.claude/ and ~/.codex/ with the right files so the CLI starts past onboardingShips.
B. Direct API keysSupport sk-ant-... / sk-... credentials directlyWorks, but reinvents credential shape per provider and bypasses the subscription path users actually have
C. Pre-auth baked into imageRun onboarding once locally, copy the dotfiles into the imageTokens expire, and now your image is a credential
D. Upstream --headlessAsk the vendors for a real headless flagCorrect long-term answer, useless short-term answer

Option A is what runs. An init container writes the auth and config files the CLI expects, so it boots straight past the first-run wall. Two supporting changes make it stick:

  • Bump the bridge timeout to 300s or more to tolerate the slow initial startup window.
  • Use claude -p "ready" in exec mode as the health check before the session switches to interactive. That separates "is the binary alive" from "is the TUI idle," which is the distinction the 60s probe was getting wrong.

Gotchas and the generalizable lesson

If you build one of these, here is what will bite you, roughly in order:

  • Separate liveness from readiness. TUI onboarding is a readiness concern. Treating it as liveness is what restarts the pod mid-flow.
  • Session state is pod-local unless you write it out. If restarts have to preserve auth, put the CLI's dotfiles directory on a PVC.
  • Keystroke scripting is fragile. Screens reorder and prompt text changes between versions. Prefer pre-populated state over scripted navigation everywhere you can.
  • exec-mode health checks beat TCP/HTTP checks for tools that do not serve HTTP. Ask the binary a question; do not poke a port it never opened.
  • Handle SIGTERM through tmux or accept a graveyard of zombie sessions on every deploy.

The broader takeaway is older than any of these tools. When you put a human-first program in an environment without a human, you are not configuring it, you are impersonating the human. The tmux bridge is just an honest acknowledgment of that. It does not pretend the CLI is a server. It drives the CLI exactly the way a person would, and then puts an HTTP API where the person used to be.

Reach for this pattern when you have to host a TUI somewhere humanless: a CI job driving an interactive tool, an agent platform, a remote sandbox. Skip it the moment the tool ships a real headless mode. Use that instead, and send the maintainers a thank-you note.


Resources