Prove It: Does a Learned Rule Actually Change What Your AI Assistant Writes? – Security Loop Series, Part 4

Prove It: Does a Learned Rule Actually Change What Your AI Assistant Writes? – Security Loop Series, Part 4

By Julio Gomez,
Publication Date: 2026-11-16 15:00:00

Part 3 of this series ended with a promise I hadn’t kept yet.

I showed you how Foundry turns a real finding into a Project CodeGuard rule, and how that rule lands in the developer’s IDE as a single file. “The prevention is the file,” I wrote. Nice sentence. But it’s also a claim, and in this series we don’t accept claims without evidence.

So this post is the experiment. Same feature requests, a real AI coding agent, workspaces that are identical except for the rules, and a blind judge. The short version: without the rules Foundry learned from real code, the agent wrote an insecure trust check in 8 of 8 attempts. With them, in 0 or 1 of 4. The generic rules made no measurable difference.

The experiment

I wanted a simple test, one that could prove me wrong.

Twelve ordinary feature requests. Each one is the kind of thing a developer asks an AI assistant every day, with no security hints at all. In each one, the assistant has to modify a small existing service:

  • Six of them…