Green, Reviewed, and Still Wrong: The Bugs Behind Your Test Doubles
I shipped a security-hardening pass on a side project: the suite green, the validator clean, the release tagged. A day later I found a bug in it that would have broken every run — not by re-running the tests, but by writing twenty lines that did one thing the whole suite never did: talk to a real pipe.
Green wasn't lying. It just wasn't looking where the bug was.
I've argued before that green doesn't mean done — a passing build says no check failed, not the code is correct. This is the follow-up with receipts: three bugs that sailed through a green suite and a careful read, because each one hid in a place verification almost never looks — behind a test double, inside the fix itself, and in the direction a fallback falls. None of them are exotic. That's the point. They're the kind you ship on a good day.
The mock that hid the hang
I'd just rewritten the part of a harness that reads JSON-RPC from a subprocess over a pipe. The old version could hang if the other end went quiet; the new one added a deadline. select() for readability, then read a line — textbook. Tests passed.
Here's the repro I wrote anyway, because "added a deadline" is a claim, not a fact:
read_fd, write_fd = os.pipe()
os.write(write_fd, b'{"jsonrpc":"2.0","id":1,"result":{"ok":true}}\n') # one whole line, atomically
# read through os.fdopen(read_fd, "r") — exactly what a real subprocess hands you
resp = read_response(read_fd, timeout=2.0)
# expected: the response. actual: TimeoutError after 2.0s.
A complete, valid response — and it timed out. Every real call would have stalled for the full timeout and then died.
The cause is a lovely little trap. select() only sees the operating-system file descriptor. It knows nothing about Python's own buffer. My first read(1) asked a buffered text stream for one character; the stream went to the kernel, slurped the whole line into its internal buffer, and handed me back one char. Next loop iteration, select() checked the fd again — now empty, because the data was sitting in Python's buffer the entire time. So select() reported "nothing here," and I timed out on a response I'd already received.
Now look at why the suite was green. One test fed the reader a StringIO — which has no file descriptor, so it quietly skipped the entire select() codepath where the bug lives. Another used a real pipe that never sent anything — perfect for proving the timeout fires, useless for proving a normal read works. Between "no real fd" and "no data," not one test ever put a real, complete line into a real pipe. The mocks weren't wrong; their shape was wrong, and the bug lived exactly in the shape they didn't have.
The fix was to stop reading through the buffered wrapper and pull raw bytes off the descriptor with os.read, so the thing select() watches and the thing I read from are the same object. But the fix isn't the lesson. The lesson is that I only knew to write it because I'd reproduced the failure against the real primitive — a real fd, a real pipe — instead of the convenient stand-in. Test doubles are how you go fast. They are also where integration bugs go to hide.
A fix worse than the bug
The next one I'd written myself, recently, with tests, feeling good about it. A careful read of the diff is what turned it up.
The job was redaction: take a pattern and a replacement from config and scrub matches out of some text. Mine did the obvious thing:
re.subn(pattern, replacement, text) # replacement is... a regex template, not literal text
re.subn treats the replacement as a template, not literal text — which means it can contain a backreference. Watch what a replacement of `\g<0>` does to a secret:
re.subn(r"\d{3}-\d{2}-\d{4}", r"\g<0>_SEEN", "ssn 123-45-6789")
# -> ('ssn 123-45-6789_SEEN', 1) the secret is right there. redaction re-emitted it.
The feature whose entire job is to remove the secret can be made to re-print it — and a replacement ending in a stray backslash throws an uncaught error that kills the turn instead. The repair is one line: substitute with a function so the replacement is treated literally, and fail closed if the pattern is malformed.
But the one line isn't the lesson either. The lesson is where this lived: in the security-critical path, in code I was confident about, behind tests that all used boring, well-behaved replacement strings. No green test ever tried a hostile replacement, because the person who wrote the redaction also wrote the tests, and we shared the assumption that the replacement was friendly. A fix can quietly invert the very guarantee it's supposed to provide — and the more sure you are of a piece of code, the less precisely your own tests will probe it.
Fail-closed, in the wrong direction
The smallest bug taught the sharpest lesson, and I caught it the lowest-tech way there is: reading my own diff out loud.
A rule can say deny when amount >= 1000. Earlier in the same pass I'd hardened the number parser to reject nonsense like inf and nan. Obviously a safety improvement.
My first version returned "not a number" for those. Feels fail-closed. It's the exact opposite. The rule only fires when the comparison is true; float("nan") >= 1000 is False, and "not a number" also makes the rule not fire — so a nan amount strolls straight past a deny-the-big-ones rule. My "safety" fix failed open on the one path that mattered.
The correct version raises, so the evaluator's catch-all turns it into a hard deny. Same intent, opposite outcome, and the entire difference is which way the fallback falls. "Reject the bad input" is not a direction. In a guard, every fallback has a polarity — and you have to trace, for the specific rule, whether your safe-looking default lands on allow or deny. Tests wouldn't have caught this; the rule still "worked" on every finite number anyone thought to type.
The hardening that would have been theatre
One more, because it's the inverse mistake — not a bug I almost shipped, but a "fix" I almost built.
That same redaction path runs a config-supplied regex on the hot path, and a pathological pattern like `(a+)+$` can backtrack for seconds on a short string. The tempting hardening is to run the match inside a timeout — spawn a thread, kill it if it overruns. I started reaching for it.
Then I remembered how the runtime actually works: CPython's regex engine holds the interpreter lock while it matches. A watchdog thread can't preempt it, because the runaway match never lets go of the lock for the watchdog to run. The timeout would be pure ceremony — a guard that can't fire. The honest fix is smaller and less satisfying: treat config patterns as trusted code, document the footgun, and fail safe on a malformed one. Know your runtime before you "harden" against it — half of hardening is not building the protection that can't work.
The field map
- Mock the primitive, mock the bug away. A double's shape decides which codepaths it touches. Reproduce the failure against the real fd, socket, or clock at least once, or you've only tested the stand-in.
- A fix can invert the guarantee. Especially in a security path, especially in code you're sure about — which is exactly the code your own tests probe least precisely.
- Every fallback has a polarity. "Reject the bad input" lands on allow or deny depending on the rule. Trace it for the specific case; "safe-looking" is not the same as fail-closed.
- Know the runtime before you harden. The protection that can't fire is worse than none — it's a false sense of one. Half of good hardening is deleting the fix that doesn't work.
- Reproduce, don't re-run. Re-running your own green suite just re-confirms your blind spots. The thing that moves is a fresh script pointed at reality.
- You are the last reviewer. Tools and second reads are worth it — but the bug that would have broken every run was caught by twenty lines I wrote against a real pipe, after the suite went green.
Green told me no check failed, and that was true. The bug that mattered most was hiding behind a StringIO the whole time, in the one shape my tests never took.
Don't ship the green. Ship the thing you reproduced against the real world. 🔌