Why we made finding-suppression as evidence-bound as finding-creation in symbi-codered, and what a policy-gated substrate does (and doesn’t) guarantee.
Jascha Wanger — ThirdKey AI Research
Most LLM-driven security tooling ends up with the same shape. A model proposes candidate vulnerabilities, and a second pass prunes the false positives. The proposing step gets all the attention. The pruning step is where the real problem lives, and it’s the part we rebuilt from scratch in symbi-codered.
The problem with a model that prunes its own findings
When an agent drops one of its own findings, there’s usually no requirement to say why. Nothing checks the reasoning against the code, and nothing logs that the call was even made. A real vulnerability can be discarded, and the output looks identical to a run where the vulnerability was never there in the first place.
That’s worse than just missing the bug. If a scanner never finds something, you can at least measure that. Recall is a number you can put on it. But a scanner that finds a bug and then silently un-finds it gives you nothing to measure. You can’t separate a correct dismissal from a confident mistake, because neither one left any evidence behind. Once you let an agent make a call like that with no structural record, you’ve put your trust in the model’s judgment at the exact spot where you can see it least.
There’s a more general trap here. If you let an agent take an action and then write its own justification afterward, that justification will always read as reasonable. Producing plausible reasons on demand is the one thing these models are reliably good at. A reason invented to fit a decision the agent already made isn’t evidence of anything. Treating after-the-fact model reasoning as an audit trail means resting your accountability on the part of the system most likely to make things up.
So we started from the opposite assumption. Anything an agent can do to a finding should leave structural evidence, and the rules around those actions should be enforced by a substrate the model can’t talk its way around.
Structural gates, not prompt-level controls
The guarantees in symbi-codered don’t live in prompts. They live in a trust substrate that sits next to the analysis pipeline rather than inside it: a policy engine, a tamper-evident journal, and per-engagement signing keys. The model layer runs inside that envelope. It doesn’t get to define it.
Every state-changing action an agent can take passes through a
Cedar policy gate before it’s allowed to mutate
anything: store_finding, advocate_finding, mark_poc_status,
write_knowledge_triple, emit_to_seed. The default decision is deny. An
agent doesn’t get to do something unless a policy explicitly permits it, and
every allow or deny decision is recorded next to the action it governed.
The first gate is on creation. Under policies/citation.cedar, you can’t store
a finding without citing at least one witness: an analyzer hit, a specific piece
of code, or a tracked hypothesis. That requirement is enforced through
attribute-bearing Cedar entities, not by prompting the model to remember its
citations. policies/evidence.cedar adds a second requirement, that every
stored finding carries a specifier hash and a non-empty evidence envelope. A
finding with nothing behind it doesn’t get saved as low-confidence. The write
just fails the policy.
Suppression is a finding too
Here’s the part worth taking apart. If creating a finding has to be backed by evidence, then suppressing one should be held to the same standard. So it is.
The devil’s-advocate pass argues against findings. Left unconstrained, it would
bring back the exact soft-suppression problem from the top of this post: a model
deciding on vibes that a finding isn’t real. Instead, under
policies/advocate.cedar, a rebutted verdict is forbidden unless the rebuttal
cites a structural witness for why the finding is wrong: an input-validation
envelope, a sanitizer on the path, a closed value set, or a constant caller.
Suppressing a finding now costs the same grade of evidence as creating one. You
can’t wave a vulnerability away by asserting it’s fine. You have to point at the
structural reason it’s fine, and that reason lands in the journal as a
referenced evidence envelope.
Two more constraints keep the advocate honest:
- The advocate is read-only, enforced by policy. A Cedar
forbidrule stopsdevils_advocatefrom ever callingstore_finding, with no exceptions. That isn’t left as a convention. A test,devils_advocate_forbids_store_finding_unconditionally, holds it in place so it can’t quietly regress. - A reproducer that couldn’t run is not a disproof. PoC status carries more than
two values (
reproduced,refuted,inconclusive,reproduced_by_citation), so a reproducer that failed to execute is logged asinconclusiverather than quietly counted as a refutation. The handoff policy keepsinconclusivefindings instead of dropping them.
To stop the model from rubber-stamping its own earlier output, the advocate can run on a separate, non-mirroring model (via OpenRouter, with its own fallback chain). A startup warning fires if the advocate ends up mirroring the generation tier, since an advocate that shares the generator’s blind spots isn’t much of an adversary.
An auditor has to be auditable
Evidence is only worth something if it can’t be rewritten after the fact. Every
tool invocation and every Cedar decision is appended to a hash-chained audit
journal (.symbiont/audit/audit.jsonl), with each entry chained to the last via
SHA-256. audit::verify_chain recomputes the chain and proves no entry was
altered or removed.
Both the threat model and the final handoff are signed with a per-engagement Ed25519 keypair. The specifier signs the threat model so a scan can’t drift onto the wrong tree, and the reporter signs the engagement seed that downstream consumers read. The point is narrow and on purpose: you don’t have to trust that the tool behaved. You can check.
Where the witness comes from
“Cite a structural witness” only means something if the witness is real and not just another model assertion. These witnesses are mechanical. The cartographer uses tree-sitter to pull dataflow edges out of the repo, and a taint tracer walks those edges with a plain breadth-first search (no LLM involved), from each untrusted source the specifier pinned out to each dangerous sink. An unsanitized source-to-sink path becomes a taint chain: SQL injection, command injection, path traversal, SSRF, XSS, and so on. That chain is an actual reachability proof, not a code pattern that happens to look risky, and it’s the structural witness a finding cites. The LLM agents reason over this mechanical evidence. They don’t get to invent it.
What this does not give you
The substrate controls what can be recorded, suppressed, and audited. It does not make the model correct.
The LLM reasoning layer is still the soft part of the system. pattern_scout, chain_builder, poc_forge, and devils_advocate can all be wrong. What the design buys you is that their mistakes stay bounded and visible. A wrong agent still can’t record an uncited finding, can’t suppress one without producing a structural reason, and can’t make any of it vanish from the journal. The substrate contains a bad model. It doesn’t turn one into a good model.
A few concrete limits, stated plainly. Java ships the full static path (tree-sitter parsing, dataflow, taint, semgrep SAST), but its dedicated sandbox reproducer is still deferred, so Java PoCs fall back to citation-grade evidence instead of an executed reproduction. A full audit takes roughly 5 to 15 minutes of wall-clock time and somewhere around $1 to $10 in tokens, depending on how many findings turn up. And since the guarantees come from the substrate, they’re only as good as the policies behind them. That’s the reason the policies live in the open tree where you can read them.
The pipeline, in one paragraph
In order: cartographer extracts tree-sitter facts and dataflow edges;
specifier writes an Ed25519-signed threat model; static_hunter runs
per-language SAST in network-isolated sidecars (semgrep, bandit, clippy, gosec,
eslint, checkov, trivy, and more); taint_tracer builds the reachability
witnesses. Then pattern_scout, chain_builder, poc_forge, and
devils_advocate reason over that evidence, with poc_forge running its
reproducers in read-only sandboxes set to network_mode: none under a hard
timeout. reflector distills the engagement into reusable knowledge, and a
deterministic report step (no LLM) emits SARIF, Markdown, and the signed
handoff. Every step is Cedar-gated and journaled.
Open source
The codered core is Apache-2.0 on GitHub: the engine, the CLI, and the
agents/, policies/, tools/, and scanners/ definitions. The policies
described here are all in the public tree. If you think the witness gate has a
hole in it, that’s the most useful thing you could tell us, and the file to
point at is policies/advocate.cedar. We’d rather have the design taken apart
now than assume it holds.