Agentry: Governed Agents That Can Prove What Happened

In the last post I reviewed "a toolkit I'd been building." This is that toolkit — and the opinions baked into it.

The pitch in one line: a model is one component; everything that makes it safe and useful in production lives in the harness around it. Agentry is that harness — governed, grounded, evaluated — with one property I care about a lot: it can prove what it did.

The bet: the harness is the product

I've argued before that the model is the least interesting layer of a real agent. The interesting parts are the controls: what the agent is allowed to do, what it grounds its answers in, how you measure whether it works, and whether you can reconstruct what happened afterward. Bolt those on as an afterthought and you get a demo. Build them in and you get something you can run in front of a customer.

So agentry's unit isn't a prompt — it's one declarative agent.yaml that carries the model, its tools, its policy, its grounding, and its evals together:

name: support-triage
model: { provider: ollama, name: llama3.2:latest }   # or openai / anthropic / azure
instructions: { file: instructions/system.md }

grounding:                 # cite, don't recall
  sources: [support-kb]
  require_packet: true

governance:                # policy as code — or it doesn't ship
  profile: balanced
  policy: governance/policy/manifest.yaml

evaluations:
  metrics: [{ type: tool_selection, threshold: 1.0 }]

Governed by default — policy is code, not a paragraph

Here's the opinion that drives the whole design: governance has to be deterministic and run before the action — not a paragraph in the system prompt that asks the model nicely to behave.

Agentry evaluates policy at eight lifecycle interception points around every step of the loop — startup, input, before/after the model call, before/after each tool call, output, shutdown — and each check returns one of allow · warn · deny · escalate · redact. It is fail-closed: an unknown point or a broken rule denies. The same rules live as a manifest and mirror to OPA/Rego, so the policy that runs in the teaching engine is the policy you can run in production.

A deny stops the loop. A redact rule actually masks the matched content (an SSN-shaped string never leaves the harness). An escalate is where a human signs off. The point is that none of this is the model's discretion — it's a check a reviewer can re-run and watch turn red.

Prove what happened

The part I'm most attached to: every verdict is appended to a hash-chained, tamper-evident audit log. Each record's hash covers the previous one, so altering, reordering, or dropping any entry breaks the chain — and a one-line verify() catches it. "An agent did something" is not an incident response; "here is the signed sequence of what policy was active, what was requested, and why it was allowed or denied" is.

Agent identity rides alongside it: every agent carries a name, a clearance, and a set of scopes, and delegation to a sub-agent can only narrow authority — never escalate it.

Grounded, and honest about evals

Two more non-negotiables, both about honesty:

• Cite, don't recall. Before generating, an agent emits a Learning Packet naming the approved sources that ground a claim, and flags anything it can't ground as UNGROUNDED — out loud — instead of smoothing over the gap.
• Evals never lie. A check with no offline-verifiable expectation is reported skipped, not faked. An LLM-judge with no model returns a skip, not an invented score. A run where everything skipped exits non-zero. Better to admit than to lie.

You measure change, too: agentry eval --store records runs, --compare flags pass→fail regressions, and agentry report renders the whole history — plus the verifiable audit trail — into a single HTML page.

The contrarian bet: one dependency

Here's the engineering stance that surprised even me by holding up: the whole platform runs on one runtime dependency — pyyaml.

Real LLM execution against four providers? Standard-library urllib, no vendor SDKs. The HTTP governance service? http.server. The eval metrics (F1, BLEU, ROUGE)? Hand-rolled, deterministic. The report? html.escape plus string templates plus inline CSS — a self-contained page, no dashboard framework.

Why bother? A dependency-light core installs anywhere, has almost no supply-chain surface, and is small enough to actually audit — which is a funny thing to skimp on in a project whose whole pitch is auditability. The heavy, optional things (a JSON-Schema validator, hosted-provider keys) are opt-in extras, never the price of entry.

What I deliberately left out

The honest part. Knowing what not to build is most of the work:

• No dashboard. I wanted run-history charts, and I nearly reached for a dashboard framework. Then I remembered the data is already open JSONL and a CLI already prints regressions. A web stack would have been rigor cosplay. So agentry report is a read-only, stdlib HTML page — and that's it.
• The subprocess sandbox has a real limit. It isolates importable tools with a timeout and crash-containment, but it can't process-isolate tools loaded dynamically from a file at runtime — it returns a clear "not isolatable" sentinel instead. I documented the caveat rather than hiding it behind green tests.
• Scaffolds are labeled scaffolds. Only the first lab is fully worked; the language clients are preview; tracing is declarative metadata, not emitted spans. The docs say so.
• I said no to a memory layer and an autonomous task queue. They're great — for a different product (an always-on orchestrator). Absorbing them would have been scope creep that broke the dependency bet. Right-size the harness; don't grow it for the sake of growing it.

The field map

• The harness is the product. The model is the commodity; the controls around it are the engineering.
• Governance is code, not a prompt. Deterministic, fail-closed, before the action — and mirrored to a policy engine you can run in prod.
• Prove what happened. A tamper-evident audit log turns "trust me" into "verify me."
• Ground it, and don't fake the evals. UNGROUNDED and skipped are features. Honesty is how the thing stays trustworthy as it grows.
• One dependency. If the pitch is auditability, the toolkit should be small enough to audit.
• Delete the ceremony you don't need. No dashboard, no memory layer, no orchestrator. Right-size to the failure mode.

The agents that behave like senior engineers aren't the ones with the cleverest prompts. They're the ones wrapped in honest controls — and able to show their work afterward.

Govern it. Ground it. Evaluate it. And keep the receipts. 🧾