Certainty Is the Bug: A Review, a Guard, and a Docstring That All Lied
I put a side project through a hard review — a written list of findings, each one a specific claim about a specific line. Good review. It caught real things. It was also, in three places, confidently wrong. And when I sat down to fix the things it got right, two of my fixes were wrong too.
The through-line wasn't sloppiness. It was certainty — the review's, and then mine.
Last time I wrote that green doesn't mean done, even after a review: a passing suite plus a careful read still shipped bugs that hid behind the shapes my tests never took. This is the sequel from the other side of the desk — what happens when you do get the review, take it seriously, and discover that a finding is not a fact, a guard is not a guarantee, and the word "never" in your own docstring is just a promise you haven't checked yet.
Three of them. None exotic. All the kind you nod along to and ship.
The finding that wasn't a crash
Near the top of the review: "This workshop slide tells people to use a grounding source that doesn't exist. With the new fail-closed behavior, copying it verbatim now raises an error and the agent won't run — a regression."
That reads as a hair-on-fire bug. A first lesson in a tutorial that hard-crashes is about the worst learner experience there is. I was one keystroke from writing the fix and moving on.
Instead I ran the thing the finding described:
cfg = {"grounding": {"sources": ["ghost-source"], "require_packet": True}}
packet = build_packet(cfg, repo_root, required=True)
# claimed: raises, agent won't run
# actual: {'topic': ..., 'grounded': [], 'ungrounded': ['ghost-source'], 'chunks': []}
No exception. The unresolved source doesn't crash the run — it comes back flagged ungrounded, which is the system working exactly as designed: cite what you can, and be honest about what you couldn't. The hard error only fires when the citation index itself can't be found at all — a different failure, on a different line, that this config never touches.
The finding was still worth acting on — a tutorial shouldn't teach a source ID that isn't real. But the reason was fiction, and the reason changes the fix. "It crashes" means stop-the-world. "It shows up ungrounded" means correct the ID and tighten the copy. If I'd inherited the reviewer's certainty, I'd have written a scarier changelog than the truth, and learned the wrong thing about my own system.
A finding is a claim, not a fact. The confidence in a review is the author's, not the code's. The five-line repro is cheap; the wrong mental model you adopt by skipping it is not.
The guard with a hole in it
Now the part where my certainty was the bug.
There's a redaction path that runs a config-supplied regex over model output. A pathological pattern like `(a+)+$` backtracks for seconds on a short string — a classic denial-of-service footgun. I'd flagged this exact risk in the last post and, at the time, argued the tempting fix (a watchdog thread) was theatre, because the regex engine holds the interpreter lock and a watchdog can never get a turn to fire. So this round I built the honest version: analyze the pattern before running it and refuse the dangerous shape.
My first cut was a tidy little regex that recognized "a quantifier wrapped around a quantified atom" — the (a+)+ shape. It caught the textbook case. Tests green. I felt done.
Then I threw one more string at it:
is_dangerous(r"(a+)+$") # True — caught
is_dangerous(r"((a+))+$") # False — sailed straight through
Same catastrophic pattern, one redundant pair of parentheses, and my detector shrugged. Feed ((a+))+$ to the real matcher and it grinds for about ninety seconds on a thirty-character input. My "guard" waved it through. And it got worse: the check only sat on the redaction path — I'd left the same config regexes unguarded on two other call sites that also run re.search directly. Three doors, one lock.
A pattern-matcher that recognizes one spelling of a dangerous thing isn't a guard; it's a suggestion. The fix was to stop pattern-matching the surface and start reducing the structure — peel off redundant wrapping, handle the lazy `+?` spelling, and only then ask "is this, underneath, a quantifier over a quantifier?" — and to put that one check on every site that runs an untrusted pattern, not just the one that prompted it.
A security check with a trivial bypass is worse than none — it's a bypass with a false sense of safety stapled to it. If you're writing one, your job isn't to catch the example in the ticket. It's to catch the example plus the same thing with a paren, a lazy quantifier, and a nested group — because whoever trips it next won't use your spelling.
I wrote "never" in my own docstring
The last one is the smallest and the most embarrassing, because I typed the overclaim with my own hands.
There's an audit log that hashes a snapshot of each decision. It used to serialize with default=str, which quietly stringifies anything weird — including an object's memory address, which changes every run, which silently breaks the tamper-evident chain across processes. Real bug. I replaced it with a deterministic encoder and, pleased with myself, wrote the docstring:
…is deterministic and never raises, so a malformed snapshot can't crash a governed turn.
Deterministic: true, and a good fix. Never raises: a wish wearing a fact's clothing. Because the serializer only gets consulted for values the JSON encoder already accepted — and the encoder throws before it ever calls me:
cyclic = []; cyclic.append(cyclic)
audit.record("input", {"x": cyclic}, verdict) # ValueError: Circular reference detected
audit.record("input", {("a","b"): 1}, verdict) # TypeError: keys must be str...
Both blow up. And the call site that records the audit entry has no try around it — so my "can't crash a governed turn" crashes the governed turn, on exactly the malformed input the fix was supposed to make safe. I'd fixed the determinism and left the crash, then documented the crash out of existence.
The repair was to make the sentence true: wrap the serialization so any encoder failure falls back to a stable, deterministic sentinel instead of propagating — genuinely fail-safe — and narrow the claim to what the code actually guarantees. Same amount of confidence in the docstring. Backed, this time, by a line that catches the exception.
"Never" is a proof obligation, not an adjective. If a hot path calls you and can't handle your exception, either you actually never raise — because you wrapped the thing that does — or you don't write "never." The docstring is a claim like any other, and the reader will trust it more than they trust the code.
The field map
- A finding is a claim. A review's confidence belongs to its author, not to your code. Reproduce the specific claim before you adopt its mental model — the wrong model is more expensive than the bug.
- A guard that catches one spelling is a suggestion. For anything security-shaped, test the bypass on purpose: the extra paren, the lazy quantifier, the nested group. Then put the check on every door, not the one that filed the ticket.
- "Never" is a proof, not an adjective. If an unguarded caller depends on your promise, make it true or stop making it. An overclaim in a docstring outlives the code that broke it.
- Determinism and safety are different fixes. I fixed one and wrote the docstring as if I'd fixed both. Check that the sentence and the code agree.
- The most dangerous line is the one you're sure about — the review you trusted, the guard you tested once, the docstring you typed on a roll. Certainty is where verification stops, which is exactly where the bug moves in.
The review was right that these lines needed changing. It was wrong about why one of them did, and I was wrong twice about how. Green didn't lie last time; the review didn't lie this time. They were both just confident — and confidence is not the same as having run it.
Run it. Then believe it. 🔎