AI agent governance: boundaries that survive production

The risks are boundary problems

Agent risk gets discussed as if it were a model-quality issue. In production it rarely is. Prompt injection, unauthorized tool use, credential leakage, unreproducible results and unpredictable cost are all consequences of an execution boundary that is too wide, not of a model that is not smart enough.

A better model reduces the frequency of bad decisions. A boundary reduces their blast radius. You need both, and only one of them is under your control.

Prompt injection is an authorization question

If an agent reads an untrusted document and that document contains instructions, the agent may follow them. You cannot fully prevent this at the prompt layer, which is why the defensible position is to assume it will happen and constrain what a compromised reasoning step can actually do.

If the tool allowlist for that step is read-only, injection produces a wrong summary rather than a deleted namespace. Treat every input the agent did not author as data, never as a command.

Bound the loop, not just the tools

Planner iterations, tool call counts, argument sizes and timeouts all need ceilings. Unbounded loops are the most common source of the surprise invoice, and they are also how a stuck agent turns a small failure into a large one.

Reproducibility is a governance requirement

When an agent-assisted change causes an incident, the first question is what it was working from. That means capturing the resolved context — the knowledge documents, the tool results, the profile in effect — alongside the output.

Without the snapshot, the postmortem becomes speculation about a non-deterministic system, which is a poor foundation for deciding whether to keep using it.

Four questions a boundary has to answer

It helps to make the boundary concrete rather than architectural. Whatever you build or buy, it has to answer four questions for every agent action, and it has to answer them the same way every time.

Under whose authority is this running? What is it permitted to reach right now? What happens before an action with consequences? And what record survives afterwards? If any of the four resolves to "it depends on which project wrote it", you do not have a boundary — you have a convention, and conventions decay.

The questions are not exotic. They are the same ones any privileged automation has always had to answer. What changed is that the thing making decisions is now non-deterministic, which removes the option of reasoning about behaviour by reading the code.

Identity: the agent is not the subject

The most consequential design mistake is letting the agent become its own identity. Once an agent holds a service account with broad permissions, every request it makes is authorized as that account, and the question of who asked for the work disappears at the first hop.

The alternative is to treat the agent as an executor acting on behalf of a subject. A run resolves to the human or service that triggered it, and every downstream check evaluates that subject rather than the agent's own credentials. Authorization then reflects the organisation's actual access model instead of a parallel one that grew up around the automation.

This also fixes a problem that shows up later: offboarding. When authority flows from the caller, removing someone's access removes it from the workflows they trigger. When the agent holds its own grants, their access outlives them and nobody notices until an audit.

Default-deny, evaluated deny-before-allow

Boundaries that start permissive and get tightened after incidents never converge. The starting posture has to be that nothing is reachable until something explicitly grants it, and that an explicit denial beats any inherited allow.

Deny-before-allow matters more than it sounds. In any system with roles, teams, resource inheritance and direct grants, permissions arrive from several directions at once. If the evaluator resolves conflicts by taking the union, a narrowly scoped denial can be silently overridden by a broad role someone was given for an unrelated reason.

The practical test is to write down a resource that one team must never reach, grant a broad role somewhere else in the hierarchy, and check what the evaluator actually decides. The answer is often surprising, and it is better to be surprised deliberately.

Starting a run and using a resource are different decisions

Permission to start a workflow is not permission to use everything that workflow touches. Collapsing the two is convenient and produces a predictable failure: a low-privilege caller triggers a pipeline that was written by someone with higher privileges, and inherits their reach for the duration.

Keeping them separate means the authorization check happens twice — once when the run is requested, and again for each referenced resource before dispatch. The second check is the one that actually constrains blast radius, because it is evaluated against the caller rather than against the pipeline author.

It also makes pipelines safely shareable. A workflow definition becomes a description of intent that different callers can invoke with different effective reach, rather than a privilege escalation waiting for the wrong person to find it.

Tool access belongs in policy, not in program text

When the set of tools an agent may call is expressed in the program that constructs it, three things follow. Changing the set is a deploy. Reviewing it means reading source across every project. And there is no single place to answer the question a security review always asks: which agents can currently reach production?

Moving the allowlist into reviewed configuration changes all three answers. The tool surface becomes a bounded, named object — a profile — that pipelines reference rather than define. Widening it is a diff with an author. Inventorying it is a query.

The subtle benefit is discovery filtering. If the tool list a run can enumerate is already narrowed to what its subject may use, a compromised reasoning step cannot even see the capability it would need to do damage. Withholding the option is stronger than refusing the request.

Credentials the agent never sees

Credential leakage is usually framed as a model problem — the model repeated a secret into a log or an output. The more reliable fix is upstream: arrange for the model never to hold the value in the first place.

That means definitions carry references rather than values, and resolution happens at the point of use, for the specific purpose that needs it. A pipeline says which credential to use; the platform resolves it, uses it and keeps it out of the context window entirely.

Versioning, rotation, expiry and consumer access records then become properties of the credential rather than tasks someone remembers. And when a rotation is needed in a hurry, it happens in one place instead of across every workflow that embedded the value.

Approval is a state, not a notification

Posting a message and waiting for a reaction is not an approval gate. It has no record of who was asked, no constraint on who may answer, and no durability — if the process holding the workflow dies, the decision is lost along with it.

A real gate pauses the run durably and releases the compute while it waits. It names the permitted approvers, records the decision with its author and timestamp, and resumes from persisted state. Where the risk warrants it, the person who initiated the work is excluded from approving it.

The durability matters more than it seems. Gates that hold a runner open are expensive, so teams under cost pressure quietly remove them — which means the control disappears exactly where the workload is heaviest.

Proposal-first for anything that changes the platform

There is a category of change that deserves stricter handling than ordinary tool use: changes to the automation itself. A model that can rewrite pipeline definitions, triggers or schedules can alter the boundary it runs inside.

The safe pattern is that generated changes return as validated proposals rather than applied writes. The output of the reasoning step is a diff someone reviews, promoted through the same path as any other configuration change.

This is what keeps the governance model stable over time. Without it, the boundary is only as strong as the last thing the agent decided to do to it.

Cost is a governance surface

Unpredictable spend is usually treated as a finance problem discovered at the end of the month. It is better understood as a symptom: an unbounded loop is simultaneously a cost problem, a reliability problem and a sign that a failure mode has no ceiling.

Ceilings on planner iterations, tool call counts, argument sizes and timeouts turn all three into bounded, observable events. Attributing token usage per pipeline, per run, per model and per step then turns the remaining spend into something you can reason about rather than absorb.

The useful framing for a budget conversation is not "AI is expensive" but "which workflows are worth what they cost" — a question you can only answer with per-workflow attribution.

Where the boundary belongs

Every control described here can be implemented in application code, and for the first workflow that is usually the fastest path. The cost arrives on the second and third, when the checks have to agree with each other, and again at the first audit, when someone has to prove they were applied consistently across all of them.

That is the point at which these controls want to be platform rather than pattern — configuration that every workflow inherits, rather than code every workflow repeats. The decision is less about capability than about how many places a change has to be made when the policy changes.

A reasonable test: count how many code paths would need editing to revoke one team's production access tomorrow. If the answer is more than one, the boundary is distributed across projects rather than owned by a platform.

An adoption sequence that works

Teams that get this right rarely start with a governance framework. They start with one workflow that already crosses three systems, and they make its boundary explicit: who authorized it, what it may reach, what pauses it, what record it leaves.

Then they do the second workflow and reuse the answers instead of reinventing them. The reuse is the whole point — it is what turns four ad-hoc decisions into a model, and it is where the effort starts paying back rather than accumulating.

The failure mode to avoid is designing the complete policy model before any workflow exists. Governance derived from a real workflow tends to survive contact with the next one. Governance derived from a whiteboard tends to be renegotiated the first time it is inconvenient.

What good looks like six months in

A useful checkpoint: pick a run from three months ago and try to answer, from one record, who triggered it, under whose authority it executed, which tools it was permitted to call, what context it was working from, who approved the consequential step and what the final state was.

If that takes one link, the boundary is real. If it takes a conversation with the team that built the workflow, what you have is institutional memory — and institutional memory does not survive reorganisations, or auditors.

Governing the context, not just the tools

Tool access gets most of the attention because it is where actions happen. The context an agent reasons from deserves comparable care, for a less obvious reason: it determines what the agent concludes, and it is the part a postmortem needs most.

Context assembled ad hoc — a retrieval call here, a pasted document there — is neither reviewable nor reproducible. Context resolved from declared sources before execution is both. The difference shows up the first time two runs of the same workflow reach different conclusions and someone has to explain why.

There is a security dimension as well. Every document an agent reads is untrusted input capable of carrying instructions. Knowing exactly which sources a step could draw from turns an open question into a bounded one, and bounded questions are the only kind an incident review can actually close.

Testing a system that will not repeat itself

Conventional testing assumes determinism, which is precisely what a reasoning step removes. This leads some teams to conclude that agent workflows cannot be tested meaningfully, and then to ship them with less assurance than a cron job would get.

The resolution is to test the parts that are deterministic and constrain the part that is not. Authorization decisions, dependency graphs, approval routing, credential resolution and evidence capture are all deterministic and testable. Whether a summary is good is a different question, evaluated differently.

That split is also a design instruction. The more of a workflow that sits in deterministic steps with correct answers, the more of it can be tested conventionally — and the smaller the surface where you are relying on judgement rather than verification.