Green Doesn't Mean Done: Reviewing an Agent Toolkit with a Sub-Agent
I pointed my coding agent at a toolkit I'd been building — an agent harness with governance, grounding, and evals baked in — and gave it one job: prove it's complete. The validator was green. Fourteen unit tests passed. Every command in the README's quickstart did exactly what the README said.
So the agent did the one thing that actually earns trust: it refused to believe itself. It spun up a second agent, on a different model, and told it to try to break every claim the toolkit made. Ten minutes later I had a punch-list of four places where my toolkit was quietly lying to me — with file-and-line receipts for each.
Green is a hypothesis, not a verdict
Here's the uncomfortable thing about a passing build: it tells you no check failed. It does not tell you the thing is done. Those are different statements, and the gap between them is where production incidents live.
A green check covers the cases someone thought to write down. The interesting failures are the ones nobody encoded — the promise in the prose that no test ever asserts. You don't find those by re-running your own checks. You find them by handing the work to a reader who doesn't share your blind spots.
Don't grade your own homework
The author of a thing — human or agent — is the worst possible reviewer of it, because the assumptions that produced the bug also produce the test that misses the bug. Self-review re-runs your own mental model. What you need is an adversarial second reader with different priors.
For an agent, "different priors" has a literal, cheap implementation: a different model. My orchestrator runs on one model; the reviewer it dispatched runs on another (GPT-5.5), with a deliberately hostile brief — "assume the README is marketing; make the code prove every sentence."
Two things matter in that picture. The reviewer is a separate context on a separate model, so it isn't anchored to my framing. And the orchestrator doesn't just wait — it runs its own verification in parallel (the quickstart, the test suite, a few scans) so the duck's claims land against independent evidence instead of replacing one opinion with another.
What the duck found
This is the honest part. Every row is a sentence the toolkit advertises, next to what the code actually does when you run it.
| The promise | What actually happens | Where |
|---|---|---|
| Governance at 8 lifecycle points | only 4 are wired into the loop; post_tool_call is even evaluated, then its verdict is thrown away | harness.py |
A redact verdict scrubs PII at output | redact isn't treated as "blocked", so an SSN flows through the response unchanged | harness.py + governance.py |
agentry check refuses a lab with no policy | it only checks the governance: key exists — profile: balanced with no policy file sails through | validate.py |
Grounding flags UNGROUNDED gaps honestly | "banana governance" comes back fully "grounded", with no gap reported for banana | grounding.py |
| Evals fail loudly | an all-skipped run prints 0/1 passed and exits 0 (success) | cli.py |
| Block refunds of $1000 or more | $5000 slips past the rule's `^[0-9]{4,}` regex and returns allow | manifest.yaml |
The reviewer didn't assert these — it reproduced them. The redaction one, for example, came back with the actual return value: a harness call whose output still contained 123-45-6789 and a trace that proudly recorded ('output', 'redact') right before handing the secret back. The verdict fired. Nothing acted on it.
To be fair to the toolkit: it's young — a Phase-A MVP — and several of these are the seam between an honest scaffold and an over-eager README. That's exactly why the exercise is worth it. The sub-agent's whole value is finding the gap between what the docs promise and what the code enforces, fast, with receipts — before a user does.
The discipline that made the findings trustworthy
The brief banned vibes. Every finding had to arrive as claim → reproduction → file:line → fix. A reviewer that says "the governance feels incomplete" is noise. A reviewer that says "points 5–8 are declared in governance.py and never called in harness.py, here's the deny rule that did nothing" is a work item.
This is just harness thinking pointed at a review: a finding only counts when it's a check a human can re-run and watch turn red. Everything else is ceremony.
Building on it: the sub-agent is a loop, not a one-shot
The part I want to highlight — because it's where sub-agents stop being a gimmick — is that I kept steering the reviewer while it worked. It wasn't "fire prompt, read essay." It was a conversation with a worker that holds its own context.
Mid-review I realised the scope was bigger than the code: the repo also ships a two-day workshop. So I sent the running agent a follow-up — the workshop is a first-class deliverable too; judge whether its promises match what the toolkit can actually do. It folded that in and immediately found the next class of gap: slides that promise PII redaction and tracing the runtime doesn't implement yet.
A sub-agent you can talk to mid-flight is a different tool from a prompt you fire and forget. You refine the scope, add a constraint, push back on a weak finding — and it carries the whole prior context forward instead of starting cold.
Govern your reviewers, too
One more lesson, and it's the one people skip. A powerful reviewer is eager — it will happily recommend things your project explicitly forbids: pull in a new dependency, expand the scope, "just add attribution to that source," rewrite a module it was only asked to inspect. Helpful, and wrong.
So the orchestrator governs the reviewer the same way a governed harness governs a tool call: the sub-agent proposes, the manager filters against the project's constraints, and only verified, in-bounds findings survive into the report. A second opinion you can't constrain isn't a reviewer — it's a second source of scope creep. The manager's job is to keep the duck honest and on-leash.
The field map
- Green is a hypothesis. A passing build means "no check failed," not "the work is done." Treat it as the start of review, not the end.
- Don't grade your own homework. The thing that built the bug also wrote the test that misses it. Bring a reader with different priors.
- Different model, different blind spots. The cheapest way to get an adversarial reviewer is to run it on another model with a hostile brief.
- Demand evidence, not opinions.
file:lineplus a reproduction, or it didn't happen. "Feels incomplete" is noise; a re-runnable red check is a work item. - Verify in parallel. While the reviewer reasons, run the quickstart and the tests yourself, so its claims land against independent facts.
- Steer it — it's a loop. A sub-agent that holds context lets you refine scope and push back mid-flight. Use that.
- Govern the reviewer. A strong second opinion will cheerfully suggest things your constraints forbid. The manager filters.
- A human still reads the diff. The loop produces a trustworthy punch-list; a person decides what ships.
The agents that behave like senior engineers aren't the ones that declare victory when the build goes green. They're the ones that get suspicious because it went green — and call in a second pair of eyes that doesn't share their assumptions.
Ship the report, not the vibe. Green is where the review starts. 🦆