# How to Review AI-Generated Code: Counterexamples Beat Green Tests
> Canonical: https://vxi.me/en/posts/vibe-coding-ai-code-review-evidence-gates/
> Locale: en-us
> Published: 2026-10-25
> Updated: 2026-10-25
> Markdown: https://vxi.me/en/posts/vibe-coding-ai-code-review-evidence-gates.md
Three distinct code versions and independent review results. Green tests are not production approval.
Vibe Coding has changed how quickly code appears, but review work has not become proportionally easier. A fluent model explanation or a green test suite cannot prove an implementation is safe for production.
This case is a build-time validator for animated terminal SVGs in this blog application. It checks restricted XML/CSS rules, manifest paths and SHA-256 digests, and whether articles in eight locales reference a visible demo image. A fixed canonical SVG is the positive fixture. All four defects were found in isolated pre-merge tests; the evidence covers this validator's offline scope only.
The original suite passed **8/8**, yet independent review found two unsafe acceptances. After the first fix, **10/10** passed and a second review found two more. The final pinned version passed **12/12 top-level tests**; four additional independent attack probes were run separately and all rejected. Three JavaScript files also passed node --check.
I use five steps: pin scope and invariants; construct minimal counterexamples; record RED only when a real failing run exists; independently verify positive and negative cases against the exact commit; then decide by risk. Logic tests do not replace builds, platform checks, or runtime verification. For high-risk changes, record scope, Base/Head commits, invariants, positive fixture, counterexample, actual command and environment, open boundaries, and conclusion in a small evidence card. The validator is a build-time component in the blog repository, not a separately packaged general-purpose CLI. No review-time savings have been measured.
The public repository contains only sanitized synthetic counterexamples and runnable teaching tests—not this blog application's production validator, production fixes, or internal architecture. The 8/8 → 10/10 → 12/12 sequence records the original project's offline review; the public tests demonstrate the four defect patterns and do not reproduce the production implementation.
Sanitized examples and runnable tests
## Why did 8/8 passing tests still accept invalid input?
A green suite only shows that current inputs satisfy current assertions. It does not prove every invalid input will be rejected.
The first counterexample put an allowed image and an unapproved cross-article resource reference on the same line. The old logic found one allowed image tag and passed the whole line. Independent review created a real SVG under another article slug, then appended its Markdown reference beside the valid image in the eight localized articles. The target existed, so the test did not fail merely because a file was missing. The old implementation still returned success with posts_checked=8.
~~~html
~~~
The second counterexample was a filesystem race: the path checked may not be the file ultimately opened. The test swapped a parent directory at the file-open boundary. The checkout path contained wrong bytes; outside it was a canonical SVG with the expected SHA-256. After the parent became a symlink to the outside directory, the old implementation read and accepted the external file. The bypass was file provenance, not hashing.
The final Linux path opens the trusted checkout root and resolves each component relative to an already-open parent directory descriptor. Intermediate components must be directories and use O_NOFOLLOW; the final file also rejects symlinks. It checks file type, link count and size, compares device, inode and size around the read, then verifies the digest. It uses Linux /proc/self/fd handles and fails closed without procfs. This bounded guarantee assumes a trusted checkout root and available procfs; it does not cover arbitrary same-UID or host-level adversaries. See the [Linux openat2(2) documentation](https://man7.org/linux/man-pages/man2/openat2.2.html).
## Round two: the hash is right, but the content is invisible
After the first fix, **10/10** tests passed. The third counterexample involved reduced motion. SVG frames start transparent; animation reveals them. If a reduced-motion rule only stops animation without revealing the final frame, the result can be blank.
~~~css
.f { opacity: 0; }
@media (prefers-reduced-motion: reduce) {
#f0 { animation: none; opacity: 1; }
#f3 { animation: none; } /* final frame stays transparent */
}
~~~
The test recalculated the full SVG SHA-256, byte size, content-addressed filename and canonical manifest so surrounding checks stayed valid. It tested the contract “the final result must be visible,” not merely whether the hash changed. [MDN's prefers-reduced-motion reference](https://developer.mozilla.org/en-US/docs/Web/CSS/Reference/At-rules/%40media/prefers-reduced-motion) explains the preference for less nonessential motion.
The fourth counterexample came from Markdown. The old implementation counted image tags with a regex but did not exclude HTML comments. Wrapping the only image in a comment left the tag in source while rendering no image; the old logic still counted one. [CommonMark 0.31.2's HTML block rules](https://spec.commonmark.org/0.31.2/#html-blocks) clarify why a commented tag is not visible content.
~~~html
~~~
Green tests can miss counterexamples: a commented image tag is not a visible image.
## Bind test results to exact versions
| Code version | Main suite | Independent review that followed |
| --- | --- | --- |
| Initial version | 8/8 passed | SEC-01 and SEC-02 unsafe acceptances |
| First-fix version | 10/10 passed | SEC-03 and SEC-04 unsafe acceptances |
| Final offline-reviewed version | 12/12 top-level tests passed | Four separate attack probes all rejected |
The final suite includes regression groups for SEC-01 through SEC-04. The four independent probes do not count toward 12/12 and must not be combined into 16/16. Three JavaScript files passed node --check, which checks syntax rather than behavior.
The latter two regression tests were committed before the source fix, but the full suite was not run on the test-only commit. There is no recorded full-suite 10/12 RED run. What is supported is that independent review reproduced unsafe acceptance in the old complete validator and the final pinned version passed its main suite plus the four separate probes.
Only after behavior is supported by evidence do I review complexity: trace helpers, branches, configuration and callers; identify each invariant; check for duplication and real consumers. Shorter code is not automatically simpler—a removed path guard can reintroduce a race.
Related posts cover adjacent questions: [Mutation Testing](/posts/ai-generated-tests-mutation-testing/), [OpenQodex's pre-push review fail-open boundary](/posts/openqodex-prepush-review-fail-open/), and [Diffle's review-comment anchors](/posts/diffle-git-diff-comment-anchoring/).
## What remains between passing tests and production approval?
The final independent review covered the offline validator on Node.js v22.16.0, Linux x64: 12/12 top-level tests passed, four additional attack probes were rejected, canonical SVG/manifest and eight-locale bindings passed, and three JavaScript files passed node --check.
This supports a bounded conclusion: the four known unsafe acceptances were fixed in that environment and under those rules. It does not prove the whole blog is production-ready. Native macOS behavior against hostile concurrent filesystem writers remains unverified; a full site build and final page runtime check were outside the offline review.
A real release gate is layered by risk: unit tests cover asserted behavior, static checks target specific patterns, builds and integration tests verify composition, and platform/runtime checks cover deployment boundaries. If a critical boundary remains unverified, a local pass is not overall production readiness.
The point is to use contracts and counterexamples to identify suspicious behavior, then let exact-version tests and independent evidence decide where to inspect more deeply. This does not remove human judgment; it turns “looks fine” into a review decision with evidence, scope, and reproducibility.
## Verification context
- Level: first-party
- Verified: 2026-10-25T00:00:00.000Z
- Environment: Node.js v22.16.0 / Linux x64
- Evidence mode: mixed
- Public repository: https://github.com/vxime/ai-code-review-counterexamples
- Evidence commit: 74d9f12331ef4d015bf882a6e09d66e59434368b
- Evidence receipt: https://vxi.me/evidence/vibe-coding-ai-code-review-evidence-gates.json
~~~
The second counterexample was a filesystem race: the path checked may not be the file ultimately opened. The test swapped a parent directory at the file-open boundary. The checkout path contained wrong bytes; outside it was a canonical SVG with the expected SHA-256. After the parent became a symlink to the outside directory, the old implementation read and accepted the external file. The bypass was file provenance, not hashing.
The final Linux path opens the trusted checkout root and resolves each component relative to an already-open parent directory descriptor. Intermediate components must be directories and use O_NOFOLLOW; the final file also rejects symlinks. It checks file type, link count and size, compares device, inode and size around the read, then verifies the digest. It uses Linux /proc/self/fd handles and fails closed without procfs. This bounded guarantee assumes a trusted checkout root and available procfs; it does not cover arbitrary same-UID or host-level adversaries. See the [Linux openat2(2) documentation](https://man7.org/linux/man-pages/man2/openat2.2.html).
## Round two: the hash is right, but the content is invisible
After the first fix, **10/10** tests passed. The third counterexample involved reduced motion. SVG frames start transparent; animation reveals them. If a reduced-motion rule only stops animation without revealing the final frame, the result can be blank.
~~~css
.f { opacity: 0; }
@media (prefers-reduced-motion: reduce) {
#f0 { animation: none; opacity: 1; }
#f3 { animation: none; } /* final frame stays transparent */
}
~~~
The test recalculated the full SVG SHA-256, byte size, content-addressed filename and canonical manifest so surrounding checks stayed valid. It tested the contract “the final result must be visible,” not merely whether the hash changed. [MDN's prefers-reduced-motion reference](https://developer.mozilla.org/en-US/docs/Web/CSS/Reference/At-rules/%40media/prefers-reduced-motion) explains the preference for less nonessential motion.
The fourth counterexample came from Markdown. The old implementation counted image tags with a regex but did not exclude HTML comments. Wrapping the only image in a comment left the tag in source while rendering no image; the old logic still counted one. [CommonMark 0.31.2's HTML block rules](https://spec.commonmark.org/0.31.2/#html-blocks) clarify why a commented tag is not visible content.
~~~html
~~~
