Part 3 of 3, the finale of the Agent Infrastructure series. Part 1 got an interactive CLI running headless; Part 2 built the operator that schedules them and introduced the gateway-switching primitive. This post is what the gateway does with those wires.
Part 2 ended on a primitive: when the workspace enables the gateway, the operator stops mounting credential Secrets into the agent and instead points ANTHROPIC_BASE_URL at an in-namespace proxy. The agent talks to a local gateway and has no idea its credentials live somewhere else. This post is about the hard part that primitive sets up: keeping those credentials fresh.
LLM CLIs were built for laptops. They cache OAuth tokens in a dotfile, refresh them whenever they feel like it, and expect a human to log in again when something breaks. In a hosted multi-tenant platform, every one of those assumptions is wrong. "Tokens cached locally" means "tokens cached in an ephemeral pod the tenant does not own." The platform has to own the whole credential lifecycle: login, storage, injection, refresh, and recovery.
The interesting constraint is what you are not allowed to do to make that work.
The constraint that shapes everything
The obvious design: give the thing that refreshes tokens access to the Kubernetes API so it can write the updated Secret directly. One service, done.
That is a blast-radius problem. The component that refreshes tokens also sits on the request path, proxying every call from the agent to the provider. It is the most exposed surface in the system. Handing the most exposed component the ability to read and write every Secret in the namespace is exactly the trade you do not want to make. A compromised proxy should be able to do as little as possible, and "rewrite arbitrary cluster state" is not as little as possible.
So the rule is: the proxy on the request path never touches the Kubernetes API. Everything else falls out of taking that seriously.
Two services, one trust direction
The shape that works is two services with an asymmetric trust relationship.
| Component | Owns | K8s API access |
|---|---|---|
| server (control plane) | OAuth orchestration, Secret upserts, tenant admin | Yes. The only thing that writes tokens |
| gateway (in-namespace proxy) | The request path; lazy token refresh | No. Zero cluster capability |
Trust flows one way. The gateway can ask the server to persist a new token. The server never asks the gateway for anything. When the gateway refreshes a token, it does not write the Secret; it POSTs the new blob back to a server callback, and the server does the write.
sequenceDiagram participant CLI as In-pod CLI participant GW as Gateway (no K8s API) participant P as Provider API participant S as Server (K8s API) CLI->>GW: request (BASE_URL points here) GW->>GW: token expiring within buffer? GW->>P: refresh with refresh_token P-->>GW: new access/refresh/expires_at GW->>S: POST callback (scoped bearer token) S->>S: upsert Secret GW->>P: proxy in-flight request with new token P-->>CLI: response
The gateway's only persistence channel is an HTTP call to the server. That is the entire point.
Lazy refresh, not a background loop
There is a temptation to run a goroutine that wakes up every minute and checks every token. Do not. It is more code, more race conditions, and a zombie refresh loop burning cycles in an idle pod.
Refresh lazily, on the request path instead. When a request arrives, check the token's expires_at. If it is within a buffer (about 5 minutes), refresh inline before proxying: hit the provider's token endpoint, fire the server callback to persist, then use the brand-new token for the in-flight request. The token is never refreshed except in service of a request that needs it, which is the only time it matters.
If the callback fails, the refresh still worked for the in-flight request. The write is best-effort and retried on the next request. The user's call does not fail because persistence had a bad moment.
What the callback actually is
The callback is a single narrow endpoint, something like POST /internal/credentials/{provider}, authenticated with a bearer token injected into the gateway pod as env.
The scoping is the security story. That token authorizes exactly one action: write a token blob for a provider. Compromise the gateway and the attacker can write tokens to a Secret, not read arbitrary Secrets, not create pods, not touch anything else in the cluster. The server's API surface stays small precisely so that "the server is the only thing with cluster access" is a sentence you can actually reason about.
The trust boundary, drawn as who can write what:
flowchart LR
CLI["In-pod CLI"] -->|"requests, no creds"| GW["Gateway"]
GW -->|"POST token blob (scoped)"| S["Server"]
S -->|"upsert Secret"| K8s[("K8s Secret API")]
GW -. "cannot reach" .-x K8s
CLI -. "cannot reach" .-x K8sOnly one arrow lands on the Kubernetes API, and it starts at the server. The gateway can reach the server; it cannot reach the cluster. The CLI can reach the gateway; it cannot reach either.
The security model is not "the gateway is trusted." It is "the gateway can do exactly one thing, and that one thing is the least dangerous version of what it needs."
One Secret schema, and the migration when it was wrong
Underneath all of this is a unified Secret schema: one Secret per provider per tenant, using a consistent {platform}-llm-{provider} naming convention, with two storage modes.
# OAuth mode
data:
oauth_credentials: |
{ "access_token": "...", "refresh_token": "...", "expires_at": "..." }
# API key mode
data:
api_key: sk-...It did not start unified. Earlier, every provider had its own ad-hoc naming, partially overlapping, and the gateway had to know all of them. Unifying it meant a migration, and the migration runs on reconcile: read the legacy Secret, write the new form, filter the old names out of the agent's mounts. Best-effort, non-fatal, retried.
The lesson is the same one from Part 2, applied to auth: put migration logic in the reconciler, not in a one-shot script. The reconciler is the only thing guaranteed to run everywhere, on every workspace, eventually. A one-shot script runs once, on the workspaces you remembered.
Provider quirks that will cost you a day each
Every provider has at least one undocumented sharp edge. These are the ones worth writing down.
- Anthropic forces a User-Agent. Anthropic has a server-side filter that rejects subscription-grade OAuth tokens unless the
User-Agentlooks like Claude Code's. Forward the original client UA and you get blanket-rejected with an error that points nowhere useful. The fix is to force-rewrite the UA in the proxy. This is documented nowhere; it cost real hours to find. - OpenAI JWTs are not API keys. Tokens from the ChatGPT auth flow are JWTs. The
sk-...keys from the platform are something else. They are not interchangeable, and the gateway has to know which it is holding and route accordingly. - Base-URL path conventions differ per CLI. One CLI expects
/v1/messagesappended to the base URL; another expects/v1already baked into the base. Same gateway, two prefix conventions, one config flag per CLI preset to keep them straight.
The bugs the textbooks skip
Two credential-propagation bugs are worth calling out, because they are subtle and they will both happen to you.
- A new Secret does not restart the pod. Update a mounted Secret and nothing reloads; the pod keeps using the old value until it restarts for some other reason. The fix is to annotate the pod with a hash of the Secret data so that a credential change becomes a spec change, which is a rollout. (Same root cause as the gotcha in Part 2: Kubernetes tells you a thing exists, never that it is newer.)
- A credential change re-rolls the whole tenant. Once you fix the first bug, the naive version re-rolls every pod in the tenant on any credential change, because the watch is scoped to the tenant, not the agent. The fix is to push the credential-to-agent mapping into the watch predicate, so only the agents that actually use that credential get rolled.
Assume refresh will fail
The last piece is recovery, and the mindset behind it: a refresh token will eventually be revoked or rotated server-side, and your refresh will fail. Design the recovery path before you need it, not during the incident.
Two pieces handle it. A watchdog on the sidecar detects the refresh-failure state instead of letting the agent silently 401 forever. And a /relogin command re-runs the OAuth flow and repopulates the Secret without recreating the pod, so a token problem does not cost you the agent's running session and workspace state.
That last constraint matters more than it sounds. The whole series has been about agents as long-lived, stateful workloads. Throwing the pod away to fix a token would undo that. Recovery has to be in-place.
Takeaway
The architecture is one constraint taken seriously: the proxy on the request path never touches the cluster API. From there, everything is forced. Two services with one-way trust. A narrow, single-purpose callback. Lazy refresh so there is no background surface to compromise. A unified schema migrated in the reconciler. And a recovery path that assumes refresh fails, because it will.
That is the series. Three posts, one system: drive an interactive CLI without a human (Part 1), schedule a fleet of them without forking the operator (Part 2), and keep their credentials fresh without ever expanding the blast radius (Part 3). The connecting thread is the same instinct each time. Give every component the least dangerous version of exactly what it needs, and no more.
Resources
- OAuth 2.0 token refresh: the spec behind the refresh flow.
- Kubernetes RBAC: how you actually constrain what each service can touch.
- Mounted Secrets and update propagation: the propagation behavior behind the hash-annotation fix.
- Previous: CRD to Pod: Feature Injection Without Forking the Operator.
- Start of the series: Containerizing Interactive CLIs: The Tmux Bridge Pattern.