The Full Architecture of My Kubernetes Agents

Five agents run against my K3s cluster. Three of them form a closed loop — one detects problems, one fixes them, one reviews the fix — and none of the three can do another’s job. This is everything about how they’re built: the framework, the prompt assembly, every tool, both safety gates, the memory layers, and a real pull request one of them opened.

Then I merged it and spent a day fixing what production found. Eight more bugs, none of them in the agent logic — §17 onward is that day, and it’s the more useful half.

1. What this actually is

A digital-twin system with a FastAPI core, a Next.js dashboard, and five engineering agents that each run as their own Kubernetes deployment in one namespace. The agents are not subagents of the core process — they’re independent HTTP services. The core reaches them at agent-{slug}.<namespace>.svc:8000, and they reach each other the same way.

Noor, Sol and Vera form a closed loop: detect → fix → review. Each of the three can do exactly one of those jobs and none of the others, and no two of them are the same agent. Vera was at replicas: 0 until today — see §14.

Dax reasons about CI/CD, Dockerfiles, image tagging and the Flux deploy path. He has four tools — Read, Glob, Grep, web_search — and cannot run gh, docker, kubectl or flux. He diagnoses from files and logs handed to him and recommends the change. That’s the whole job; it isn’t a gap.

Ada is backend design: FastAPI, Pydantic, SQLAlchemy, async I/O, service boundaries. Same four read-only tools. She’s a required reviewer in the pr-review workflow and the first agent in design-review, and an optional investigator in incident — but none of those workflows currently run on a schedule, so she stays at zero replicas.

2. The loop, end to end

Noor detects. Sol acts. Vera reviews. Then one narrow class of change — a single memory-limit increase — merges itself, and everything else waits for a human.

Two things enter this loop: the pulse timer and Prometheus alerts. The alert path used to go straight to Sol and had no guards — see §17 for what that cost.

flowchart TD
    T1["<b>Pulse timer</b><br/>every 2700s · one due task"]
    T2["<b>Prometheus alert</b><br/>guarded: 1 per agent,<br/>30-min cooldown per alert"]

    T1 --> NOOR
    T2 --> NOOR

    NOOR["<b>NOOR</b> · platform-sre<br/><b>READ-ONLY · 14 tools</b><br/>two calls, then decide"]

    NOOR --> Q{"Remediable?"}

    Q -- "no" --> DISCORD["<b>#infrastructure</b><br/>named finding + fix"]
    Q -- "yes" --> DEL["<b>delegate_task</b><br/>sends the decision,<br/>not the question"]

    DEL --> SOL["<b>SOL</b> · k8s-engineer<br/><b>THE ONLY ONE WHO ACTS</b><br/>≤2 reads, ONE action, stop"]

    SOL --> RESTART["<b>rollout restart</b><br/>a trigger, not a spec change"]
    SOL --> BUMP["<b>bump_workload_memory</b><br/>surgical +1/−1 edit → PR"]

    RESTART --> CLUSTER[("K3s cluster")]

    BUMP --> VERA["<b>VERA</b> · code-reviewer<br/><b>NO GENERAL MERGE VERB</b><br/>five checks, one verdict"]

    VERA --> CHAIN["verdict recorded<br/>direct → Actions bot → comment"]

    CHAIN --> GATE{"<b>merge_if_trivial</b><br/>reads the DIFF:<br/>one line? memory only?<br/>≤2x? approved?"}

    GATE -- "all yes" --> AUTO["<b>agent merges</b> → Flux<br/><i>the narrow class only</i>"]
    GATE -- "anything else" --> HUMAN["<b>A HUMAN MERGES</b> → Flux<br/><i>everything outside that class</i>"]
    DISCORD --> HUMAN

    classDef readonly fill:#313338,stroke:#5865F2,stroke-width:2px,color:#dbdee1
    classDef actor fill:#313338,stroke:#F1C40F,stroke-width:2px,color:#dbdee1
    classDef reviewer fill:#313338,stroke:#9B59B6,stroke-width:2px,color:#dbdee1
    classDef human fill:#313338,stroke:#2ECC71,stroke-width:3px,color:#dbdee1
    classDef plain fill:#313338,stroke:#6d7178,stroke-width:2px,color:#dbdee1
    classDef alert fill:#313338,stroke:#ED4245,stroke-width:2px,color:#dbdee1

    class NOOR,DEL,DISCORD readonly
    class SOL,BUMP actor
    class VERA,CHAIN reviewer
    class HUMAN,AUTO human
    class GATE plain
    class T1,RESTART,CLUSTER,Q plain
    class T2 alert

The important property is that Noor cannot fix anything, by construction rather than by instruction. Her fourteen tools include thirteen read-only ones and a way to hand work off. Sol has seven, and exactly two of them change anything — one of those changes Git, not the cluster. Vera has four GitHub tools and no general merge verb: merge_if_trivial re-derives from the diff whether the change is a single memory increase within a bounded ratio, so her approval is necessary but never sufficient.

3. Inside a single run

Every invocation of any agent goes through the same seven stages.

flowchart TD
    subgraph P["1 · SYSTEM PROMPT — assembled in this order"]
        direction TB
        P1["OPERATOR_DISCIPLINE → USER.md → AGENTS.md → PULSE.md<br/>→ date → prior findings → context → output contract"]
        P2["<i>separately</i>: deepagents injects Identity.md + MEMORY.md<br/>only those two — bulk memory is read on demand"]
        P3["skills/ is NOT injected — the agent opens a SKILL.md<br/>when the task calls for it"]
    end

    P --> T["<b>2 · TOOL RESOLUTION</b><br/>AGENT_TOOLS from the Dockerfile,<br/>overridden by the manifest<br/><i>registry.yaml does not decide this</i>"]

    T --> G["<b>3 · TWO SAFETY GATES</b><br/>name filter → 16-name allowlist<br/>AUTONOMOUS_RUN → checked inside each tool<br/><i>two, because tool names can alias one function</i>"]

    G --> R["<b>4 · ReAct LOOP</b><br/>create_deep_agent on LangGraph<br/>recursion_limit = 6 × max_iter + 12<br/>temperature 0 · reasoning OFF"]

    R --> S["<b>5 · SYNTHESIS PASS</b><br/>a second call forcing a DiagnosticReport<br/>180s × 2 — and it times out often"]

    S --> D["<b>6 · DELIVERY PRECEDENCE</b><br/>1. report_finding calls ← always survives<br/>2. synthesis findings<br/>3. cleaned prose"]

    D --> B["<b>7 · BACKSTOPS</b><br/>a required outcome the model forgot<br/>is produced in code, not hoped for"]

    subgraph M["MEMORY — four layers"]
        direction LR
        M1["prompt<br/>Identity + MEMORY"]
        M2["PVC files<br/>/memories/*.md"]
        M3["CompositeBackend<br/>/memories/ → Postgres"]
        M4["checkpointer<br/>per thread_id"]
    end

    R -.-> M

    classDef stage fill:#313338,stroke:#5865F2,stroke-width:2px,color:#dbdee1
    classDef gate fill:#313338,stroke:#ED4245,stroke-width:2px,color:#dbdee1
    classDef run fill:#313338,stroke:#F1C40F,stroke-width:2px,color:#dbdee1
    classDef mem fill:#313338,stroke:#2ECC71,stroke-width:2px,color:#dbdee1

    class P1,P2,P3,T,D stage
    class G gate
    class R,S,B run
    class M1,M2,M3,M4 mem

4. The framework

deepagents on LangChain and LangGraph. One call builds the whole agent:

agent = create_deep_agent(
    model=llm,                  # ChatOllama / ChatOpenAI, temperature 0.0
    tools=tools,                # resolved from AGENT_TOOLS + a2a
    system_prompt=system_prompt,
    memory=memory_files,        # auto-injected into every prompt
    skills=skills_dirs,         # read on demand by the agent
    store=self._store,          # PostgresStore (Supabase)
    backend=backend,            # CompositeBackend
    checkpointer=checkpointer,  # AsyncPostgresSaver
    middleware=_run_middleware(self.max_iterations),
    debug=self.debug,
)

That gives you the ReAct loop, a virtual filesystem, and a set of built-in tools. There is no hand-rolled control flow anywhere — the loop is the framework’s.

Two corrections to what I originally wrote here, both found by printing the built object instead of reading the docs. Planning is not included — write_todos only appears if you pass TodoListMiddleware, and I don’t, because these runs are deliberately two or three tool calls and a planner would add tokens and steps to that. And the built-ins are not Read/Write/Edit/Glob/Grep; those are the names our AGENT_TOOLS uses. What the harness actually injects is ls, read_file, write_file, edit_file, glob, grep, delete, task, and an execute that never reaches the model because the backend isn’t a sandbox. §22 is what I did about that.

The middleware= line is also new — for a long time there was no middleware argument at all, which is the other half of §22.

Two gotchas. create_deep_agent() does not accept a temperature parameter; set it on the model object. And reasoning defaults off: glm-4.7-flash is a thinking model, and with reasoning=True it emits <think> traces that inflate token count roughly 50× and break structured output. Sol and Noor are both opted back in, via ZI_AGENT_REASONING=true in the manifest — deliberately restored in a commit that reads fix(agents): restore reasoning=true. I originally wrote that Sol was the only one; the running pods say otherwise, and it matters, because Noor is the agent whose runs time out.

5. Anatomy of one container

Image chain: <registry>/core:latest → <registry>/agent-{slug}:latest. The agent image is a thin layer that copies four things and sets env:

FROM <registry>/core:latest
COPY k8s_engineer/AGENTS.md   /app/config/
COPY k8s_engineer/Identity.md /app/config/
COPY k8s_engineer/PULSE.md    /app/config/
COPY k8s_engineer/skills/     /app/config/skills/
COPY k8s_engineer/memories/   /data/memories/
ENV AGENT_NAME=k8s_engineer \
    AGENT_TOOLS=Read,k8s_get_resources,k8s_describe,k8s_get_pod_logs,k8s_get_events,k8s_rollout,bump_workload_memory \
    AGENT_MODEL=glm-4.7-flash:latest \
    AGENT_MAX_ITERATIONS=15 \
    AGENT_TEMPERATURE=0.0
CMD ["python", "-m", "app.agents.executor.server"]

If the Dockerfile doesn’t COPY skills/, the skills don’t ship and nothing errors — the agent just behaves as though they don’t exist.

HTTP surface: POST /run, POST /invoke, GET /health, GET /ready, GET /info, GET /pulse, GET /metrics.

Probes: a startupProbe on /health (10s × 30, so up to 300s to boot), and only once that passes does liveness begin. That ordering is not cosmetic — see §17, where a liveness probe with a 45-second delay was killing the container before it ever finished importing.

The config precedence trap

Three places appear to define an agent’s tools, and only one is real:

1. src/app/agents/registry.yaml — the core reads this for intent routing and display metadata.
2. The agent Dockerfile’s AGENT_TOOLS — this is what the container actually reads.
3. The manifest env in the GitOps repo — overrides the Dockerfile.

The registry used to claim Sol had 21 tools including k8s_apply, k8s_delete, k8s_patch, Bash, Write and Edit. The running pod reports seven. I’ve since synced all five lists to the Dockerfiles and documented the precedence in the file itself, but the lesson stands: check the Dockerfile, not the registry.

The same trap caught me again on a different variable. The Dockerfile above says AGENT_MAX_ITERATIONS=15, and I quoted that number for months. The manifest overrides it: Noor runs at 20, everyone else at 15. I only found out by printing printenv inside the pod while verifying something unrelated. Any number you quote from a Dockerfile is a default, not a fact — and there is no warning when the manifest silently disagrees.

And there is a third layer neither file mentions: the agent framework injects tools of its own on top of AGENT_TOOLS. §22.

6. How the prompt is assembled

Exactly this order, every run, in _build_system_prompt():

1. OPERATOR_DISCIPLINE — a shared six-line behavioral contract prepended to every agent so it isn’t restated per-agent. Evidence over assumption; smallest sufficient action; verify before done and say what you did not check; honest uncertainty; outcome first; append durable learnings to MEMORY.
2. USER.md → “About My User”, if present.
3. AGENTS.md → “Behavior Instructions” — the operating manual.
4. PULSE.md → “My Proactive Tasks” — the recurring-task definitions, injected as prompt text so the agent knows its own cadence.
5. Today’s date.
6. Previous agent findings, if a workflow passed a SharedContext — prior verdicts, summaries, top 5 findings.
7. Context key/values from input_data, any string under 1000 chars.
8. Output format — an optional trailing JSON block with verdict / findings / recommendations for workflow orchestration.

Separately, deepagents’ memory= parameter auto-injects Identity.md and MEMORY.md. That list is deliberately two files. Bulk memory — beliefs, relationships, reflections — lives on disk and is read on demand, because auto-injecting everything creates a reinforcement loop where the agent regurgitates its own prior positions verbatim.

Identity.md is the capability sheet. Noor’s is eight lines:

You are the cluster’s watchtower: your health sweeps are pure detection — when a fix is needed, delegate_task it to Sol (k8s-engineer) to execute; you never touch the cluster yourself.

7. Tools — the ones I declare

Every kubectl operation is a discrete typed function, not a free-form command string. This is kagent/KubeAssist parity plus k8s_rollout.

Read-only, ungated (8): k8s_get_resources, k8s_describe, k8s_get_events, k8s_get_pod_logs, k8s_get_yaml, k8s_api_resources, k8s_cluster_config, k8s_check_connectivity

Mutating, gated (9): k8s_apply, k8s_create, k8s_create_from_url, k8s_patch, k8s_delete, k8s_label, k8s_remove_label, k8s_annotate, k8s_remove_annotation

Exec, gated (1): k8s_exec

Self-gating (1): k8s_rollout — takes status | history | restart | undo | pause | resume. In autonomous runs it permits the first three and refuses the rest.

Remediation (2): bump_workload_memory (fetches the manifest, raises the memory limit, opens a PR), propose_fix_pr (GitHub REST API, update-only, code-guarded, Sol-scoped token)

Reporting, Noor only (2): report_finding, post_health_report

Review, Vera only (3 at this point): gh_list_agent_prs, gh_get_pr, gh_review_pr — read an agent-proposed PR and submit a verdict. No merge tool exists yet; §21 adds a fourth that merges exactly one diff shape.

Other (3): Bash, web_search, k8s_docs_query

Built-ins from deepagents (this line was wrong — see §22): the harness injects ls, read_file, write_file, edit_file, glob, grep, delete and task, plus an execute it hides when the backend isn’t a sandbox. delete and task are now excluded, so what the model sees is ls, read_file, write_file, edit_file, glob, grep. None of these appear in AGENT_TOOLS, the registry, or the manifest. This heading used to read “Tools — all 27”, which counted only the ones I declare and so was never what any agent actually had.

A2A (3), appended to every agent unconditionally: ask_agent, notify_agent, delegate_task. Each posts HTTP to agent-{slug}.<namespace>.svc:8000 and carries a social_depth counter; MAX_SOCIAL_DEPTH = 5 stops infinite delegation loops.

8. The two safety gates

Gate one — name filtering. When workflow_type is pulse-task or a2a-delegate, the tool list is intersected with an allowlist — 20 names now, after the review tools and merge_if_trivial were added to it. Everything mutating, plus Bash and k8s_exec, falls out. This logs as:

WARNING autonomous run — restricted to safe tools

Gate two — an execution-time contextvar. AUTONOMOUS_RUN is set for the duration of the invocation and always reset in a finally. Each mutating tool function checks it at call time and refuses.

Two gates because gate one is name-based and has a hole: several tool names alias the same underlying function. The kubectl_* family all resolve to one kubectl tool that takes a free-form command string and would happily run delete if asked. Filtering names doesn’t stop that; checking at execution does.

A correction to what I claimed here. I wrote that Write and Edit fall out in an autonomous run. They don’t, and they can’t — the gate filters AGENT_TOOLS before the framework appends its own tools, so write_file and edit_file are added afterwards and neither gate ever sees them. An autonomous run has always been able to write to /data. That’s mostly fine, since the weekly tasks are supposed to write /memories/beliefs.md, but I had the mechanism wrong and the guarantee was narrower than I said. The one that mattered was delete, which is recursive on the volume holding every agent’s memories, and which nothing ever declared — §22.

The point still stands, sharpened: blast radius is a property of the tool list, not the prompt. “Be careful” is a request. No delete verb is a guarantee. But your tool list is not the whole tool list, and a gate that runs at the wrong moment guards nothing.

9. Skills

deepagents Agent Skills — a directory of <name>/SKILL.md files with YAML frontmatter, loaded on demand rather than injected into every prompt.

Sol has four: sre-remediation (remediate ONE delegated finding and stop fast), troubleshooting-protocol (non-intrusive first, escalate progressively, plan the rollback before acting), k8s-playbooks (per-symptom steps), response-format (the seven-section write-up).

Noor now has three: health-sweep (the bounded two-call sweep and how to classify each finding), delegation-protocol (send the decision not the question, and the literal agent_slug rule), report-format (report_finding first, severity as sort key, no markdown tables). She had none until tonight — see §16.

Vera has one: pr-review-protocol — the five checks in §14, with the specific traps these PRs hit.

sre-remediation is the one that changed behavior most, because it kills the loop:

At most 2 diagnostic reads, then decide. Never re-run the same read. Exactly ONE action. Then STOP with a one-line result. Do not verify in a loop. Recommending is a complete, valid outcome.

It also strips Sol’s discretion on delegated work: “You were delegated this to FIX it. Do NOT re-judge whether it is worth fixing, do NOT dismiss it because the workload looks like a test.” Two agents independently deciding whether an action is wise produces an argument, not an outcome.

10. Memory — four layers

1. Auto-injected prompt memory — Identity.md + MEMORY.md, every run.
2. Structured markdown on the PVC — /memories/beliefs.md, relationships.md, goals.md, mood.md, reflections.md. Seeded from the image at boot, read and written by the agent with its built-in tools.
3. CompositeBackend routes those writes: /memories/ to a PostgresStore on Supabase namespaced by agent id; everything else to FilesystemBackend(root_dir="/data") on the PVC.
4. AsyncPostgresSaver checkpointer — episodic memory keyed by thread_id. A workflow’s discussion_id becomes the thread, so an agent remembers everything it said across multiple calls in the same discussion.

Both Postgres paths fall open to in-memory on failure so a pod always boots. The conn string gets a bounded connect_timeout plus TCP keepalives — without them a hung (not refused) Supabase connect blocks the synchronous setup() forever. Backend init runs in asyncio.to_thread so a slow Supabase can’t stall the event loop.

One scar worth keeping: caching an in-memory fallback during an outage pins the agent in degraded mode permanently and invisibly. Don’t cache the fallback.

11. The pulse loop

A background task that runs one due task, then sleeps. It’s serial, which matters for capacity. Tasks come from parsing PULSE.md — every - bullet under ## Continuous, ## Daily, or ## Weekly is one task.

• Continuous — due every tick, no floor.
• Daily — due 20 hours after last run.
• Weekly — due 6 days after last run.

The parser strips indentation and drops continuation lines, so a task must be one physical line. A multi-line or nested bullet fragments into separate truncated tasks. PULSE.md carries a note warning editors about this.

Noor’s two continuous tasks, in substance:

The health sweep. Read Kubernetes’ own problem signals, do not list every pod. At most TWO calls: k8s_get_events(event_type=Warning, all_namespaces=True) and k8s_get_resources(resource_type=nodes). Then for each affected workload, namespace/object — reason — short message, a one-line likely cause, and whether a rollout restart would clear it or a human/Git change is needed. Under ten lines.

The delegation. Same events call, then for the single most serious remediable finding — OOMKilling, BackOff/CrashLoopBackOff, or Unhealthy probes on a workload that still exists; explicitly not failed Jobs, not FailedScheduling, not stale events — call delegate_task. One per tick, then stop.

That task carries a hard-won correction in caps: agent_slug must be the literal string "k8s-engineer", never "sol" or any persona name. The model kept reaching for the first name.

12. What the reports look like

Noor’s sweep posts to #infrastructure through a channel webhook with a per-message username override, so the sender shows the persona rather than the shared bot. Discord only allows changing the displayed sender per message via a webhook; if the bot lacks Manage Webhooks it falls back to the plain bot endpoint.

Per-persona accent colors: Noor blurple 0x5865F2, Sol amber 0xF1C40F, Dax green 0x2ECC71, Ada pink 0xE91E63, Vera purple 0x9B59B6.

Findings render most-severe-first with a colored dot (🔴 critical, 🟠 high, 🟡 medium, 🔵 low). That sort happens in code, because embed bodies cap at 4096 characters and a long report splits across messages — if ordering came from whatever order the model happened to mention things, the critical finding could land in message three.

The handoff to Sol is visible in the same channel, which is the whole reason they post under distinct names:

Delivery discipline:

• If the agent already called post_health_report during the run, the pulse path skips posting so the channel doesn’t get it twice — but still records the body so dedupe covers the tool path.
• Precedence: tool-recorded findings, then synthesis findings, then cleaned prose. No single path is trusted alone — the tools depend on the model remembering to call them, and synthesis can time out.
• Continuous reports only post when something is wrong. Healthy ticks stay quiet.
• Identical body to the last post for that task means skip.
• The dedupe stamp is set only after delivery succeeds, and a partial multi-chunk delivery still stamps. A pre-emptive stamp would suppress the retry and lose a standing issue permanently.
• Daily tasks are housekeeping and never reach Discord.

Markdown tables get stripped before posting because Discord renders no table syntax — raw agent prose was arriving as a wall of pipes.

13. What an agent-opened PR looks like

When Sol hits an OOM he doesn’t patch the cluster — Flux would revert it. He calls bump_workload_memory, which fetches the manifest from main, raises resources.limits.memory, and opens a pull request. Here is a real one:

The body is written by the agent and ends with a finding_key, which is what dedupes it — the branch is deterministic per finding, so the same problem can’t open five PRs.

And look at the diff. The body promises one thing: raise limits.memory to 1Gi. The diff is +22 −24. The tool parsed the YAML, changed one value, and re-serialized the file — which reindented every list, stripped the header comment explaining what cache-warmer is for, and collapsed the inline Python from a readable block scalar into an escaped one-line string with literal \ns in it.

The intended change is in there. It’s the four-character 256Mi → 1Gi at the bottom. Everything else is a formatting casualty.

That’s the strongest argument I have for why proposal-only was the right call. A tool with permission to write to the cluster would have applied all of that silently. Because it can only open a PR, the damage is visible and reversible, and it sat there waiting to be caught.

Except nothing was catching it.

14. The review loop — closing the gap

Five agent-proposed PRs were open when I checked, the oldest from 20 July. Sol can’t merge, by design. Nothing else was reading them. The detect-and-propose loop had no closing step, so the system’s output was a slowly growing pile of unreviewed pull requests.

So I activated Vera and gave her three tools:

She runs one PR per pulse, takes the oldest, and applies five checks from her pr-review-protocol skill — the five things these PRs actually get wrong:

1. Does the diff match the body? A one-value change that produced forty lines means a YAML round-trip. That’s the PR above, and it’s request_changes.
2. Is a bigger limit even the right fix? Read the container spec. cache-warmer appends 16MiB to a list forever and never frees it. Raising the ceiling buys time and nothing else — it will OOM again at 1Gi. The fix is bounded memory in the app.
3. Is the size proportionate? The convention is ~1.5×. A 4× jump on a homelab node is a meaningful share of schedulable memory, and that’s how a memory fix becomes a FailedScheduling incident elsewhere.
4. Does it touch only what it should? One app, under apps/base/<app>/. Anything near secrets, RBAC, flux-system or CI means something is wrong upstream.
5. Did she see the whole diff? gh_get_pr truncates a very large diff — and says so loudly in the output, with an explicit “do not approve.” A silent truncation approves half a diff.

What’s enforced in code rather than asked for in the prompt

There is no merge tool. Not gated, not permission-checked — the module has no merge, close, push, or edit capability at all. The strongest action available is submitting a review. A human still merges every one. (§21 changes this for one narrow diff shape, and only by re-deriving that shape in code.)

Agent-proposed PRs only. Every tool refuses a PR whose head branch doesn’t start with agent/fix/. Human PRs and Renovate PRs are literally unreachable.

No self-review. ZI_PR_REVIEW_AUTHOR_DENY names the proposing agent, and the running agent always denies itself. The whole value of this loop is that the proposer and the reviewer are different agents — so that’s a code check, not a instruction.

A blocking verdict needs a real comment. request_changes and comment require a substantive body. A verdict with no reasoning is noise.

Sixteen tests cover those refusal paths, because each one is a rule a model could otherwise reason its way past.

15. The synthesis pass

After the ReAct loop finishes, a deterministic second pass forces the model to fill a DiagnosticReport schema from the run transcript via with_structured_output. The prompt is narrow: use only evidence in the transcript, record every distinct problem as its own finding, don’t merge unrelated problems, and if nothing is wrong set healthy=true.

It exists so completeness never depends on the model volunteering it in a terse final message. It retries once on timeout at 180 seconds per attempt. On total failure it returns None and the caller falls back to parsing text — which is exactly why hand-recorded report_finding calls sit first in the delivery precedence.

16. What I found broken, and what I did

Four things, all fixed tonight.

The operating manuals described a system that doesn’t exist

AGENTS.md for Noor, Sol and Dax all named Claude 3.5 Sonnet as the model — they run glm-4.7-flash. All three listed Grafana MCP, Loki, Prometheus, Discord MCP, GitHub, Obsidian, mcp-aws, Bash and a flux CLI as available tools. None of them have any of those.

Noor’s ran 723 lines and included a Tier 1/2/3 autonomous-action table granting her pod restarts, Flux reconciles, node cordons and deployment rollbacks — every one of which the safety gate refuses outright. It documented PromQL queries she cannot run and runbooks requiring SSH she does not have. All of it entered her prompt on every single run, teaching her to attempt work that would be refused.

I rewrote all three against the deployed AGENT_TOOLS: an explicit complete tool list, an explicit “you do NOT have” list, the gate’s real behavior, and where to delegate instead. Noor went from 723 lines to 137.

Noor had no loadable skills

Her skills/ directory held one Python file, pvc_reader.py — a legitimate LangChain tool module that load_skills() still picks up, but not a deepagents Agent Skill. So create_deep_agent(skills=...) scanned the directory, found no SKILL.md, and loaded nothing. The watchtower had zero procedural guidance while Sol had four skills.

I wrote her three: health-sweep, delegation-protocol, report-format. pvc_reader.py stays where it is — the two mechanisms coexist in one directory. I added an agents/.dockerignore so __pycache__ can’t ship into a directory that gets scanned at runtime for both.

The 1200-second timeouts were a retry multiplier, not saturation

This is the one I had wrong in my first pass. Over 24 hours Noor ran 37 sweeps: 3 approve, 11 comment, 21 error, 2 request. I assumed the failures were fleet oversubscription. The actual numbers said otherwise — two failures at 1202.8s and 1203.7s, suspiciously identical.

ChatOpenAI against the gateway sets timeout=max(ZI_OLLAMA_TIMEOUT, 400) and left max_retries at the OpenAI SDK default of 2. A saturated Ollama node returns neither 5xx nor 429 — it accepts the connection and holds it — so each attempt burned the full 400s and the caller waited 400 × 3 = 1200s before seeing “Request timed out.”

The gateway already fails over node 1 → node 2 → node 3 → Ollama Cloud. A client-side retry of the same gateway call adds latency, not resilience. max_retries=1 drops the worst case to 800s.

The cadence didn’t match the work

The pulse loop is serial: run a task, then sleep check_interval. A sweep takes 1,200–2,700s and the interval was 900s, so the fleet was working roughly 60% of every hour — and Sol’s delegated runs happen inside Noor’s sweep, since delegate_task blocks. The Macs serve NUM_PARALLEL=1, so everything else queues behind a multi-minute generation.

900 → 2700 in the manifest. A watchtower that reports once an hour and finishes beats one that starts every fifteen minutes and times out 21 times a day.

Also fixed, found while checking

_run_agent_investigation in main.py called discord_client.post_to_channel(), which doesn’t exist on DiscordClient — the real method is send_to_channel(channel_id, content). A # type: ignore[attr-defined] hid it from mypy. Every Prometheus-alert-triggered investigation died in its except handler and reported nothing. Now routed through delivery_router.deliver(), the same path the alert notification already used.

And the third node was fully wedged — accepting connections, answering /api/tags, hanging on generation. Restarting its Ollama service brought its canary back to 2 seconds.

The loop had no closing step

The biggest one, and the least visible: Sol had been opening PRs nobody read. Five of them, the oldest three weeks old. Every part of the pipeline worked — detect, classify, delegate, propose — and the output accumulated in a queue with no consumer.

That’s the failure mode worth naming, because it doesn’t look like a failure anywhere. No error, no alert, no red pod. Each individual run succeeded. The system was healthy and producing nothing.

Activating Vera is §14. At that point her verdicts still don’t merge anything — the loop ends at a reviewed PR waiting on a human. §21 is where that last step gets a narrow, code-checked automatic path.

17. Day two: what merging it actually taught me

Everything above describes the system as designed. Then I merged it, and the design turned out to be the easy part. Eight more fixes followed in one day, and the interesting thing is that not one of them was in the agent logic.

The container was being killed mid-boot. limits.cpu: 100m is one tenth of a core. Importing the LangChain/deepagents tree needs about 4.4 CPU-seconds, so it took 44 seconds of wall clock — I measured the config module alone inside the pod. uvicorn doesn’t bind the port until imports finish, so the liveness probe found nothing listening and SIGKILLed the pod at 135 seconds. One agent had accumulated 65 restarts this way. The fix is a startupProbe, which is the primitive built for exactly this; initialDelaySeconds was never the right knob.

It presented as a silent hang with zero log output, which sent me looking in the wrong place entirely. The reason there were no logs: CMD has no -u, so Python buffers stdout, and SIGKILL discards the buffer. A slow boot and a wedged process look identical if you only have the absence of logs to go on.

A dependency floor is not a pin. pyproject.toml said deepagents>=0.3.9. The Docker image resolved 0.7.5; my local venv sat on 0.4.4. Every test passed locally against an API the pods no longer had. 0.7 removed backend factories — passing a callable now raises at agent build time — so every agent run died two seconds in. It also flipped FilesystemBackend.virtual_mode to True, which would have silently changed how every memory write resolves. I caught the second one only because I was already in the file fixing the first.

Auth has more than one path. A gateway gained key-auth, and fixing it took three separate changes: mount the consumer key; discover that the plugin reads only the apikey header while the OpenAI client populates Authorization: Bearer; then discover that three of five agents had no gateway URL configured and were falling back to a second client that also sent no key. Each fix revealed the next, and each one looked like the whole problem at the time.

I tested the seam I was about to break. Sixteen tests on the review tools, all of which patched the HTTP helper wholesale. The bug was inside that helper — it passed headers twice — so the diff fetch could never succeed and no test executed the broken line. Green suite, tool that had never worked.

GitHub won’t let you approve your own PR. Both agents share one token, so the reviewer’s formal verdict was structurally unavailable — a bare 422 on both APPROVE and REQUEST_CHANGES. The way out without creating an account: dispatch to a repository workflow, where GITHUB_TOKEN acts as github-actions[bot], a genuinely different identity. The workflow re-validates the PR is open and agent-proposed, because it can approve pull requests and shouldn’t trust its caller.

The alerts were eating the thing that fixes alerts. Six workloads had been down for weeks. Every Prometheus evaluation re-fired their alerts, and every webhook spawned a fresh investigation with no dedupe and no concurrency limit — three concurrent runs in eleven seconds, each a multi-minute generation on nodes that serve one request at a time. The alerts were summoning the only agent that could fix them, and by summoning it four times at once, guaranteeing it fixed nothing.

Then I split the work. Alerts now go to the read-only triager first, bounded to two calls, which delegates one finding to the actor, bounded to two reads and one action. That’s the same detect→fix split that already existed for scheduled sweeps; the alert path had simply been bypassing it. Worth being precise about what this buys: it does not reduce total GPU time. It makes each run complete, so a finished triage is banked even when the remediation after it fails — where before, a gateway timeout at iteration 14 threw away all fourteen.

18. The one I keep making

I reported the alert-reporting bug as fixed. It wasn’t. I’d restarted the agent pods after the merge but not the core — the process that actually contained the change — so the same error kept firing for hours after I’d said it was resolved.

That’s the exact failure I wrote about in §16: “rollout successful” is not “the change is in the pod.” I wrote the rule down and then broke it the same day, by probing the component I’d been thinking about rather than the one I’d changed.

The generalisable version: verify the marker in the specific process you modified, not in the system that contains it. “The agents are healthy” was true and irrelevant.

19. Day three: the generator was the problem

The reviewer’s first check is does the diff match what the body claims. It was going to fire on every single PR, because the tool generating them was broken.

Each fix template did this:

doc = yaml.safe_load(text)      # comments, quoting, indentation, block scalars die here
... mutate the dict ...
return yaml.safe_dump(doc)      # re-emitted from the library's defaults

That is not an edit. It’s a read, a discard, and a fresh print. A YAML parser keeps data; comments aren’t data, quoting style isn’t data, and | versus an escaped one-line string carry the same value so the emitter picks whichever it likes. The four-character change was correct and the other forty-two lines were collateral.

The fix is to stop round-tripping. yaml.compose_all gives a node tree whose marks carry absolute character offsets into the original text, which is enough to locate a scalar and splice the new value over exactly those bytes. Against the real manifest, the same change went from +22/−24 to +1/−1.

Two more bugs fell out of that work. The tool took a workload argument and never used it — it edited the first document and the first container, so on a multi-document manifest it would have opened a correct-looking PR against the wrong object, which is worse than the formatting damage because nothing in the diff looks wrong. And my own new insert path wrote annotation values unquoted, so a timestamp re-parsed as a datetime, which Kubernetes rejects.

The general lesson is about reviewers, not YAML: a reviewer whose top finding is always the same finding isn’t reviewing. Fix the generator and the review starts carrying information.

20. Skills as the unit of behaviour change

Seven skills now, extracted from a Kubernetes reference and adapted to what each agent can actually observe:

The adaptation is the whole job. The source material assumes you can see p99 usage and that you author manifests. No agent here has a metrics tool or writes YAML, so “size the limit against observed usage” would have been an instruction to do something impossible — the exact failure of the stale manuals in §16. What survived is what they can act on: memory is incompressible and CPU is compressible, so an OOMKill announces itself and CPU throttling never does; FailedScheduling is four different problems with four different owners and none of them are remediable by an agent; exit 137 with Reason: Error is a liveness kill, not an OOM.

They’re baked into the image today, which means a one-word prompt change costs a full rebuild. That’s worth fixing — the loading is already per-run, so a ConfigMap volume would make edits take effect on the next run with no restart — but it’s iteration speed, not function, so it waits.

21. The loop closed, once, on a real workload

Validated end to end on the oomkill canary in the test-fixture namespace — a container that allocates a bounded 48Mi against a 32Mi limit and had been crash-looping for 26 hours.

The chain: Sol proposed (+1 −1, flow-mapping and alignment preserved) → the merge gate refused the old malformed PR from before the rewrite (+18/−17) as a control → Vera read the fresh one, approved via the Actions bot, and merge_if_trivial re-derived the shape from the diff and merged → Flux applied it. Eleven hours later it has not been OOMKilled once. It does still restart — about once an hour, with Reason: Completed and exit code 0, because the test container’s script is sleep 20; allocate 48Mi; sleep 3600 and Kubernetes restarts anything that exits. That is the distinction the fix turns on: before, it died at exit 137 roughly twenty seconds in, 293 times; now it survives the allocation and lives out its full hour.

Enabling it needed a repo setting. github-actions[bot] is barred from approving PRs by default (can_approve_pull_request_reviews: false), which is separate from the self-authored 422. With our PAT unable to approve its own PR and the bot forbidden outright, no available identity could record an APPROVED review. Flipping that setting was the unblock — worth knowing that the workaround for one restriction ran straight into a second, unrelated one.

Three bugs surfaced by running it, none of which more unit tests would have found: bump_workload_memory assumed deployment.yaml (wrong for 47 of 83 apps); propose_fix_pr validated with yaml.safe_load, which raises on the second ---, making every multi-document manifest unproposable; and the merge gate only understood block-style YAML, so it refused the first real PR it saw. The gate now masks memory values and requires the rest of the line to be byte-identical, which also catches a CPU limit smuggled into the same edit.

Five stale PRs had jammed the pipeline. The open-PR cap is five, and five malformed proposals from the old generator had been sitting there for weeks — so nothing new could be proposed at all. A cap plus a reviewer who never merges is a deadlock, and it had been one for three weeks without anyone noticing.

22. Day four: the tools I never configured, and a ceiling that deleted its own work

I reviewed the whole agent layer against the framework’s own documentation, expecting to find nothing. Three things, and I found all of them the same way — by building an agent in a throwaway script and printing what came back, instead of reading what I’d configured.

create_deep_agent was being called with no middleware argument at all. Not a wrong one. None. Everything in the harness’s default stack was there; nothing I might have added was.

The agents have tools I never configured. Eight of them, on top of AGENT_TOOLS and the three a2a tools.

I first wrote “declares fourteen, is handed twenty-five” here, and that was arithmetic rather than a measurement. The resolver skips Read/Glob/Grep from the declared list precisely because the built-ins satisfy them, so adding the two lists counts those three twice. Counted inside the pod: fourteen declared resolve to eleven, plus three a2a, plus eight built-ins — twenty-two, now twenty. Still at the top of the fifteen-to-twenty band where this size of local model starts mis-calling, which is the exact ceiling I’d used to cap the orchestrator and had never once applied to the agents. Smaller number, same conclusion — and I’d rather print it than derive it, which is the entire lesson of this section and I still managed to skip it.

The expensive one is task. Its own description says the subagent it spawns “has access to all tools as the main agent,” and it runs a full nested reasoning loop. On three nodes serving one generation at a time, one forgotten task call multiplies a run that’s already queueing behind two others. That is the failure the entire prompt design exists to avoid, sitting in a tool I didn’t write and had never read.

I excluded delete and task — the two nothing declares. I kept glob and grep, because the Dockerfiles do declare Glob/Grep and the tool resolver skips those names precisely because the built-ins satisfy them; dropping them would have revoked a declared capability without touching the file that declares it. execute needed nothing: the harness already hides it when the backend isn’t a sandbox, so excluding it would have been a guard that never fires.

The lever is a harness profile registered per model. Passing tools= doesn’t work — it’s additive and never removes a built-in.

The run ceiling was deleting finished work. The only cap was LangGraph’s recursion_limit, and crossing it raises. The surrounding handler catches exceptions, logs a failure, and returns an empty findings list. So a watcher that had already read the cluster’s event feed and already identified the broken workload hit the ceiling and lost all of it. Two lines of a test tell the story:

without the call-limit middleware:  GraphRecursionError  → transcript LOST
with it (run_limit=3):              8 messages returned  → transcript KEPT

ModelCallLimitMiddleware(exit_behavior="end") jumps to the end of the graph instead of throwing, so the transcript survives and the synthesis pass in §15 can still build a partial report. A partial report beats no report, and I had been shipping no report.

Then the same bug one layer out. While verifying that, I found the core hangs up on its own agents after 120 seconds — a constructor default nobody had ever passed a value to, with no setting to change it. Meanwhile a single model call through the AI gateway is floored at 400 seconds. The core was quitting mid-call and filing the result as a failure while the agent was still working:

11:38:20  Agent platform-sre failed after 120.0s: timed out after 120s
11:38:20  agent_investigation_complete  findings_count=0  verdict=error

That agent’s own HTTP client was still retrying at 11:41. Twice in ten minutes. So the work I’d just made survivable inside the agent was being discarded at the caller instead. It’s 600 seconds now, and both bounds are assertions rather than taste: above 400 or the caller is guaranteed to quit mid-call, below the 1800-second alert cooldown or the same alert becomes eligible for a second investigation while the first is still running — the stampede the cooldown exists to prevent.

And a third default, the same evening. The no-self-review guard in the review tool reads AGENT_NAME from the environment to learn which agent is running, with a default of "". The check was if me and me in denied — so an agent that could not name itself skipped the denial and went on to a live GitHub call. Every Dockerfile sets AGENT_NAME, so it never fired. But the proposer/reviewer split is the entire point of the loop, and not knowing which agent you are is precisely the moment you must not act. It refuses now. Three in one day with the same shape: a default that stands in for configuration is invisible, and a guard has to fail closed or it isn’t a guard.

Three times I was wrong, and each took one probe. I sized the recursion backstop from arithmetic — a ReAct step is two graph steps, so twice the iteration budget — and the test failed. Reaching the middleware’s clean exit costs 4N + 5 on the real graph, because the jump-to-end is itself graph steps. Then I derived the harness profile’s key as provider:identifier and bailed on anything containing a colon, which silently breaks the direct path where the model tag is model:latest — that version registers successfully under a key nothing ever looks up, leaving every tool in place while the code reads as though it were trimmed. Then, holding the 4N + 5 number, I concluded the old limit must have been cutting agents off at half their budget and started writing it down; counting the actual calls showed twenty on a twenty-iteration budget. The old limit let every agent finish all its work. It only threw the results away.

That last one is the whole post in miniature. A measurement answers exactly one question, and I tried to spend one on a different question.

23. What’s still open

What I wrote here on 27 August, in one line each: a hang isn’t a failover, so the cloud tier never fires; capacity is the binding constraint; nothing completes on a schedule — zero completions in the twenty-five minutes after §22 shipped; the actor is idle at 0 runs; the guards suppress 88% and it still isn’t enough. Eighteen days later I went back to the pods rather than the doc, and about half of that is no longer true. The half that replaced it is more interesting.

Runs complete now. Noor’s pod came up at 00:22 today and by 11:40 had started 96 runs and completed 94 — 388 seconds on average, 2,746 at the worst, roughly nine minutes apart. Sol ran six times in the last week and finished five; he is no longer idle, because Noor now finishes often enough to delegate. Nothing in the code changed for this. What changed is what I measured next.

The synthesis step is where the day goes. Every run ends with a second model call that turns the transcript into a structured report, with a 180-second budget and one retry. Of Noor’s 94 completed runs today, 50 lost both attempts and 10 more came back as prose the parser rejected — "**Done.** Restarted ..." is not JSON. So 60 of 94 runs produced no structured report, and the fifty double-timeouts alone are five hours of a twelve-hour day spent waiting on a call that never arrives. The run itself completes. The part that makes it legible to anyone else mostly doesn’t.

Vera lost thirty hours to the gateway. 12–13 September: two completions, thirty-two failures, every one of them the same — OpenAIAPIError wrapping an HTML page whose title is 502 Bad Gateway, ninety-three of those pages in her log. Then 14 September: sixteen of sixteen. Nothing was deployed between those days. A saturated fleet doesn’t fail cleanly and it doesn’t fail consistently either; it fails in stretches, and a stretch that long looks like a broken agent from the outside.

I built five of the six alerts that are keeping her saturated. The core received 2,618 Alertmanager posts in the last 24 hours carrying 5,475 instances of DeploymentAllReplicasDown. The guards skipped 1,812 investigations and let 158 through — 92% suppressed, up from 88%, and 158 triage runs a day is still one every nine minutes, which is exactly the spacing measured above. Six deployments fire that alert. Five are the test-fixture deployments from §21 — configerror, crashloop, imagepull, notready, unschedulable, thirty-three days old, broken on purpose so the loop had something to fix. The sixth is a real one in another namespace that has been 0/1 for nine days. I wrote on the 27th that “an alert firing continuously for three weeks is not new information thirty minutes later.” It is thirty-three days now, and the alerts are mine. The watcher’s entire day is spent re-investigating the test harness I left running.

The loop restarted its own detector. 00:20:36 this morning: an alert reached Noor, she delegated, and at 00:21:59 Sol put a restart annotation on agent-platform-sre — Noor’s own deployment — reporting that it “wasn’t listening on :8000.” Prometheus had fired PodOOMKilled on her previous pod, so there was a real signal underneath. Flux reverted the annotation within the minute — it treats restartedAt as drift from Git, which is why every restart in this project is a delete pod, not a rollout restart — but the replacement pod had already been created and stayed. The deployment never dropped below one available replica. Whether the restart was warranted I genuinely can’t say, because the text of the delegation isn’t logged anywhere — Sol’s log shows a POST /run from Noor’s address and a 20-second run, and that’s all. An action taken by one agent on the instruction of another should be reconstructible after the fact, and today it isn’t.

Sol has a pulse I documented as not existing. The operating manual said Sol runs only on delegation and alerts. His Dockerfile says ZI_PULSE_ENABLED=false. The manifest overrides it to true, sets no interval — so 300 seconds, the default — and his PULSE.md carries five weekly tasks. §5’s trap, one more time, in a doc I wrote after learning the trap.

Still true, unchanged: the gateway’s fallback triggers on 5xx and 429, saturation produces a hang, so the cloud tier has never fired. Both agents that carry reasoning=True — Noor and Sol — still carry it. Three single-slot nodes are still three single-slot nodes.

The fix I’d take first isn’t in the agents at all: stop paging the watcher for a namespace whose whole purpose is to be broken. One Alertmanager route, and 5 of 6 standing alerts stop arriving.