An architectural pattern for safely running untrusted agent code: credential-free workers, signed cross-process RPC, per-job AEAD-bound secret envelopes.

Most AI agent runtimes will leak every credential they hold the moment they're compromised. This isn't a vulnerability in any specific implementation — it's a property of the architecture every popular runtime has converged on. And as agents increasingly come from vendor marketplaces, third-party catalogs, customer uploads, MCP-resolved tools, and LLM-generated code, that property is becoming the wrong default.

I've been sitting with this question for the last year while building Talos, an open-source agent execution runtime: what if the worker process couldn't see the credential at all?

It turns out the answer is a stack of four architectural decisions that compose into a single property: the worker is a sandboxed executor with no standing access to user credentials or user data. Standing credential access gets replaced by per-job, cryptographically-bound secret envelopes. The blast radius for your users' credentials collapses from "every credential the worker holds" to "the secrets in the in-flight jobs" — modulo one deliberately-retained trust anchor that I'll account for honestly at the end.

This post walks through each of the four pieces, what bug class each one closes, and what each one costs.

The current model and why it's breaking

In LangGraph, CrewAI, in-process Python agent SDKs, and most Temporal worker patterns I've seen, the worker process is treated as a trusted boundary. Credentials get loaded into the worker on startup (or fetched once per session) and held in plaintext memory for the lifetime of the run. The worker uses them when the agent decides it needs to make an API call. The agent code, the worker code, and the credential store are all in the same trust domain.

This was a reasonable model when "the agent" meant "code I wrote, running on my infrastructure, with my credentials." It stops being reasonable in several increasingly-common situations:

A single compromise in any of these scenarios — prompt-injection escape, dependency CVE, sibling-tenant memory read, the agent code itself being malicious — leaks every credential the worker was holding for the duration of the breach. The blast radius is bounded by the credential set the worker had standing access to. In current architectures, that set is "everything the agent might possibly need" — which, for a multi-integration agent, is dozens of long-lived secrets.

That's the model I wanted to break.

Piece 1: The controller owns every secret

The starting move is obvious in hindsight: the worker process gets zero standing access to user-plane secrets. No API key, no DB password, no OAuth token. The controller process owns every secret.

In production the master KEK lives in HashiCorp Vault transit, accessed by short-lived, auto-renewing tokens — the controller asks Vault to wrap and unwrap data keys rather than holding the master key itself. One honest caveat worth stating plainly: during the migration window where stored rows are still being re-wrapped from an older env-provided master key onto Vault, that legacy key stays loaded in process memory for backward-compatible reads. A single config flag (KEK_DISABLE_LEGACY=true) drops it entirely once the re-wrap soak completes. The end state is a controller whose process memory never contains the master key; the transition state still holds it, by design, so reads of not-yet-migrated rows don't fail.

If you stop there, you have a worker that can't do anything useful. It needs some secret to make any external call. The standard mistake is to give it a long-lived "service account" with wide grants and call this "credential isolation." It isn't. You just renamed the problem and made the credentials less obvious.

The actual move is to never give the worker long-lived user credentials at all. Every user-secret access is per-job, cryptographically bound to the specific job, and reclaimed when the job ends.

That requires three more pieces.

Piece 2: Per-job secret envelopes with AEAD context-binding

When the controller dispatches a job to a worker, it does this:

  1. Resolves the secrets this specific module is entitled to — keyed on the module/node, not the union of everything the agent might touch. The set is the module's secret grants, plus the paths it explicitly declared in allowed_secrets, plus any vault:// references in its node config, plus (for tier-2 actors only) the canonical LLM-provider keys.
  2. Serializes that map and encrypts it with AES-256-GCM, using a fresh random 96-bit nonce per dispatch.

The AEAD key is not a per-job key. It's the pre-shared WORKER_SHARED_KEY (a 32-byte secret both sides load from env at startup), used directly as the AES-256-GCM key. Today that's the same WORKER_SHARED_KEY that signs the RPCs in Piece 3 — the AEAD key and the HMAC key are literally the same bytes. That's a deliberate simplicity tradeoff, not a best practice: separating them with per-worker HKDF subkeys is a documented next step (see the trust-anchor accounting below), not something the current code does. Per-job freshness comes from the random nonce, not a fresh key.

The resulting { ciphertext, nonce } envelope rides inside the JobRequest, whose HMAC-SHA256 signature binds sha256(ciphertext) alongside job_id and the execution id.

The non-obvious bit is the AEAD's additional authenticated data. The seal is bound to the workflow execution id — not the job id:

// controller — talos-workflow-engine/src/secrets_pipeline.rs
// aad is sourced from the dispatching execution: execution_id.as_bytes()
let (ciphertext, nonce) = envelope
    .seal_with_aad(&secrets_map, worker_shared_key, aad)
    .await?;

The worker unseals with the same AAD, sourced from JobRequest.workflow_execution_id:

// worker — worker/src/main.rs
let secrets = encrypted_secrets
    .decrypt_with_aad(shared_key.as_bytes(), req.workflow_execution_id.as_bytes())?;

This closes a specific replay class: an envelope lifted from one execution can't be transposed into a different execution — even by an attacker who holds the shared key and can therefore forge the outer HMAC — because the AES-GCM tag won't validate against the new execution's AAD. Decryption fails closed, with no plaintext leak. (Transposition within the same execution, and tampering generally, is caught one layer up by the JobRequest HMAC over sha256(ciphertext) + job_id. The two bindings are complementary: the HMAC is the primary anti-tamper check; the AEAD AAD is the part that still holds if the signing key is compromised.)

There's a defense-in-depth wrinkle worth calling out. For high-sensitivity actors (Talos calls them "tier-1" — actors handling regulated content like medical, financial, or internal company data), the controller-side dispatch path skips the LLM-provider key prefetch entirely. Tier-1 jobs never have an Anthropic or OpenAI key in the envelope at all, not even in encrypted form. The envelope can only decrypt to what the controller put in it; if it never put external LLM keys in, no amount of worker-side compromise can produce them. The worker-side enforcement surfaces (described below) are a separate layer — defense in depth, not defense in trust.

Piece 3: Signed cross-process RPC

The encrypted envelope is useless if an attacker can forge a JobRequest to ask the controller for more secrets, or forge a JobResult to corrupt the audit log. So every cross-process call between worker and controller is HMAC-SHA256 signed with a shared key, and every signature is checked against three more layers before any business logic runs: a freshness window, a constant-time MAC compare, and a replay (nonce) cache.

From talos-memory/src/rpc_auth.rs:

pub const PAST_WINDOW_MS: i64 = 60_000;
pub const FUTURE_WINDOW_MS: i64 = 5_000;

Asymmetric freshness window: 60 seconds of past tolerance (PAST_WINDOW_MS, for legitimate latency, queueing, GC pauses), 5 seconds of future tolerance (FUTURE_WINDOW_MS, clock skew only — nobody should be sending requests dated in the future on purpose). A symmetric window doubles the effective replay window for free. The asymmetry costs nothing and tightens the surface.

Two-generation rotating nonce cache: the cache is an ArcSwap<DashMap> pair — a current generation plus a previous generation — rotated every PAST_WINDOW_MS / 2. Picking ArcSwap + sharded DashMap over a single Mutex<HashMap> matters more than it sounds: a global mutex held across the whole lookup-and-insert would serialize every RPC across every signed-subject subscriber under contention. With multiple subscribers handling thousands of RPCs/second each in some workloads, that mutex is the bottleneck. Here there's no global lock on the hot path: a read is an atomic ArcSwap pointer load, a write takes only a per-shard DashMap lock, and the one shared mutex is a tiny one guarding the rotation clock (just long enough to read an Instant::elapsed()), not the cache data itself.

PRO-TIP: ROTATION ORDER IS LOAD-BEARING

You must not swap current to a fresh empty map first and then move the old current into previous — between those two atomic stores there's a window where a key that was just in current is in neither map, which is enough to admit a replay. The correct order is to promote current into previous first (briefly both pointers reference the same map, which is fine — a reader checking "current OR previous" still finds the key), and only then install the fresh empty current. At every instant during rotation, every live nonce is visible from at least one pointer.

Constant-time MAC comparison: subtle::ConstantTimeEq, not ==:

// talos-memory/src/rpc_auth.rs
expected.as_slice().ct_eq(signature).unwrap_u8() == 1

The bug here is subtle and worth understanding: a fast-path == on byte slices returns false on the first mismatching byte, leaking timing information about the prefix of the correct MAC. An attacker who can send forged signatures and measure round-trip time can recover the correct MAC byte-by-byte over many requests. Constant-time comparison takes the same wall time regardless of where the mismatch occurs.

The sequence is the part people describe wrong

Here's the ordering, and it's worth getting exactly right because the security argument depends on it. The subscriber runs two gates, in this order:

// per-RPC verify(): freshness + structural caps + constant-time MAC.
// It deliberately does NOT touch the nonce cache.
pub fn verify(&self) -> bool {
    if !rpc_auth::verify_freshness(self.timestamp_ms) {
        return false;
    }
    // ...structural caps...
    let body = sign_body_bytes(&self.op, self.timestamp_ms);
    rpc_auth::verify(SUBJECT_NAME, self.actor_id, &self.nonce, &body, &self.signature)
}

// gate 2 — the nonce cache is touched only AFTER verify() returns true:
//   if req.verify() && check_and_record_nonce(SUBJECT_NAME, req.actor_id, &req.nonce) { ... }

So it's freshness + MAC first, then the nonce cache — not "check the nonce, then verify the MAC." The nonce cache is never touched until the signature has been verified. That ordering is the whole point: an attacker flooding the endpoint with garbage signatures can't fill the cache with junk entries (which would either DoS legitimate traffic by evicting real nonces, or pre-poison the cache against a nonce the attacker expects a victim to use). Those forged requests are rejected at gate 1 and never reach the cache at all.

And the nonce check is not a lookup followed by a later insert — it's a single atomic operation:

use dashmap::mapref::entry::Entry;
let admitted = match current.entry(key) {
    Entry::Occupied(_) => false,        // nonce already seen → reject
    Entry::Vacant(v) => { v.insert(()); true }
};

Because the check and the insert happen under the same per-shard entry lock, two concurrent requests carrying the same nonce can't both observe "absent" and both succeed — exactly one wins, the other sees Occupied and is rejected. (An earlier contains_key + insert version of this had a TOCTOU gap; the entry() form closes it.)

One signed message, exactly one cache write

There's a related subtlety I learned the hard way, and it lives one layer up — in the signed job-result protocol (talos-workflow-job-protocol), not the RPC-auth layer above. Those message types (JobResult, PipelineJobResult) carry their own process-local nonce cache (JOB_NONCE_CACHE), and here the cache insert is folded into verify(). So each type ships two methods:

// talos-workflow-job-protocol — JobResult / PipelineJobResult
pub fn verify(&self, key: &[u8], max_age_secs: u64) -> Result<(), String>;        // inserts into JOB_NONCE_CACHE
pub fn verify_no_replay(&self, key: &[u8], max_age_secs: u64) -> Result<u64, String>; // no cache write

The reason: if you have a primary consumer for a signed result (the controller's JobResult handler) and a passive observer (an audit subscriber, a metrics emitter, a Prometheus exporter), and both call verify(), the second one always fails. The second consumer finds the nonce already in the cache — inserted by the first consumer's verification — and rejects the message as a replay. This was a real, self-inflicted regression I hit during development (Talos r300/r301): every job result started failing the moment a second consumer subscribed.

The fix is the split. Passive observers — anything whose only side effect is an idempotent DB write, an audit append, a metrics tick — use verify_no_replay() and skip the cache entirely. The single primary consumer keeps verify(). The invariant is "exactly one cache write per signed message," and you get it by branching the publish (the worker single-publishes each result to one subject — reply inbox or global audit topic, never both) and by using the right verify method on each side. Add both methods together the moment you introduce a new signed message type — the prophylactic cost is trivial, the regression is total.

Note the two layers get there differently. The memory/graph/database/state RPC types in rpc_auth.rs keep verification and nonce-recording as separate calls (verify() then check_and_record_nonce()), so there's naturally one cache-mutation point. The job-result types fold the insert into verify(), so they need verify_no_replay() to give observers a cache-free path. Same guarantee — exactly one cache write per message — reached from opposite directions.

Piece 4: The worker sees opaque handles, not secrets

The final piece is what the worker hands to the guest module after it has decrypted the envelope. get-secret never returns the secret to any module, tier-1 or tier-2: it returns an opaque u64 handle into a host-side slot, and the plaintext never crosses the WASM boundary. When the guest needs to use the secret, it asks the host to perform the operation on its behalf — hmac-sign(handle, data), fetch-with-bearer(handle, request), fetch-with-header(handle, name, request) — and the host injects the plaintext only inside its own address space, for the duration of that one call. (The single exception, expose-secret, is the audited tier-2 escape hatch described below — and it's denied by default.)

Slots carry a TTL (300s by default). When it expires, the slot fails closed: the secrets-interface operations (hmac-sign, expose-secret) return expired, and the HTTP convenience wrappers (fetch-with-bearer, fetch-with-header) surface a generic network error since their result type has no expired variant. This is a rotation boundary: if an operator rotates a secret, slots holding the old plaintext can't outlive the rotation window. The TTL also bounds the "lifetime of decrypted material in host RAM" for very-long-running modules. Modules that want to minimize that window further can call release-slot(handle) explicitly the moment they're done with it.

Explicit tier-2 escape hatch with audit. The interface includes one function, expose-secret(handle, reason), that does return the plaintext to the guest. It exists for legitimate cases that don't map to tier-1 host primitives. Every call to it: (a) is logged at WARN level with the reason string and the execution context, (b) is rate-limited — at most 10 calls per execution (returns ratelimited after) and a global 100 calls per user per day across all executions, and (c) marks the execution trace as containing tier-2 exposure. It's an opt-in, audited, bounded break-glass — not a default path. In Talos's engine dispatch paths today, the tier-2 grant is hardcoded to false; turning it on for a specific module is a deliberate operator action with an audit trail attached.

This closes a class of bugs responsible for a non-trivial fraction of real-world secret leaks: the module accidentally (or maliciously) logs the secret, returns it as part of an error message, exfiltrates it via a panic string, or stores it in module-local state that later gets serialized into the job's output payload. DLP redaction catches some of these as a backstop, but you don't want to be relying on regex pattern-matching to catch a leaked Anthropic key — you want the key to have never been in guest memory in the first place.

There's an additional invariant the worker enforces: the per-module allowed_secrets allowlist AND a host-reserved deny-list both gate every get-secret call. The deny-list blocks the canonical LLM-provider key paths — anthropic/api_key, openai/api_key, gemini/api_key — even when the module declared allowed_secrets: ["*"]. (These three exact paths are the ones the host pre-fetches for its own llm::* functions; operator-chosen custom paths that happen to hold a provider key are caught separately by the outbound-host deny-list and a soft audit-log heuristic, not by this exact-match gate.) These keys are reserved for host-internal consumption by the llm::* host functions and must not be resolvable from guest code under any allowlist. The allowlist matcher (vault_path_permitted) lives in one place — in talos-workflow-job-protocol — and is imported by both the controller (for validation at module-install time) and the worker (for runtime enforcement in check_secret_allowlist). So a module that violates the rule fails at install, and even if it somehow slipped past install, it fails again at runtime.

The cost: you can't do arbitrary operations on the secret in guest code. You can't, for example, compute an HMAC of a request body using a vault-stored signing key by reading the key into guest memory and running hmac::sign(key, body). You have to ask the host to do it via hmac-sign(handle, data). In practice the host primitives — hmac-sign, fetch-with-bearer, fetch-with-header, plus the vault:// config-injection path — cover the cases that matter. The few cases that don't fit cleanly are usually a sign that the module is trying to do something it probably shouldn't, and the audited tier-2 escape hatch is the right way to handle the genuine exceptions.

What this composes to

Concretely, for user credentials: a worker compromise (prompt-injection escape, sibling-tenant memory read, vulnerable dep, malicious module, AI-generated code with a bug) gives the attacker the user secrets in the jobs the worker is actively running at the moment of compromise — the API keys, OAuth tokens, and vault values that were decrypted into the live job(s). Not "all secrets for that user." Not "all secrets the worker has ever decrypted." Just the in-flight set, plus whatever's in the encrypted secret envelopes for jobs the worker has received but not yet finished. When the worker restarts or the job completes, the decrypted plaintext is zeroized and gone — nothing user-credential-shaped remains standing, because no user credential was ever standing.

There's one trust anchor on the user-credential path I have to be honest about, because it's the load-bearing exception. The worker holds WORKER_SHARED_KEY — the worker↔controller trust anchor. It's the HMAC key the worker uses to sign its RPCs, and — the very same bytes — the AES-GCM key that seals and unseals the per-job secret envelopes. An attacker who fully compromises the worker gets it, and with it can forge signed RPCs to the controller (read or write actor memory, run database and graph queries) and decrypt any secret envelope they can capture on the wire — the RPC subscribers authorize on a valid signature, freshness, and an unused nonce, not on a per-actor authorization check the attacker can't satisfy.

To be precise about what "credential-free" does and doesn't mean: it's the user-data plane that's free of standing secrets. The worker still legitimately holds some non-user-plane standing material — WORKER_SHARED_KEY (the trust anchor above) plus worker-integrity and observability keys: an AOT-cache signing key, an audit-log signing key (TALOS_AUDIT_SIGNING_KEY), metrics-endpoint auth tokens (METRICS_AUTH_TOKENS), and, when object-storage offload is enabled, a single per-deployment S3 identity. None of those are user credentials, and none of them are per-tenant; they're deployment-level infrastructure keys. The claim the architecture makes is narrower and stronger than "the worker holds nothing": it's that no user-plane credential is ever standing in the worker — no API key, no OAuth token, no DB password, no per-user secret of any kind.

So the precise blast-radius statement is this: the architecture removes standing user credentials from the worker — the thing that used to be "dozens of long-lived third-party secrets" — and replaces them with per-job envelopes scoped to the in-flight set. It does not make the worker a zero-trust component. You can't have a signed channel between two processes without each end holding key material. What you gain is that the user-credential trust anchor is a single key rather than a pile of API credentials: its compromise is detectable (forged RPCs all carry the one shared identity), it's rotatable fleet-wide as a single key, and it's narrowable — per-worker HKDF subkeys plus per-actor RPC authorization are the obvious next step, and the wire format already reserves room for a per-worker identity (signed JobResults already bind a worker_id). The win is real, but it's "collapse the user-credential blast radius," not "nothing is recoverable."

This pattern also makes two adjacent properties cheap to add on top, which I think are the more interesting downstream consequences:

Vendor module marketplaces become safe to run. When you can pull a WASM module from an OCI registry, verify its Sigstore signature against a pinned certificate identity, and run it in a sandbox where it can only see secrets the module's allowed_secrets declaration explicitly listed at install time, you can run a marketplace where module authors are mutually distrustful. The module can't see the user's other module secrets (different allowed_secrets). The module can't reach back to the controller via an unsigned channel (the signing keys aren't in the module's address space). The module's blast radius is bounded by its own declaration, not by what the worker happens to have lying around.

Per-actor data-egress policy becomes enforceable. Each actor in Talos carries a max_llm_tier value (tier-1 = local Ollama only, tier-2 = external LLM providers allowed). The tier is HMAC-bound into the job's signing payload, so an on-wire attacker can't downgrade a tier-1 actor's job to tier-2 by modifying the request in flight — the signature won't validate. The worker enforces the tier at every external-egress surface — the llm::* host functions, the outbound HTTP runtime (single and batched fetch), GraphQL, webhooks, HTTP streaming, plus the email and embedding host functions — before any external call is made. For an actor handling regulated content (medical records, financial data, internal company documents), the property "no external LLM provider sees this actor's data" becomes cryptographically enforceable, not just a policy statement that gets violated when an engineer adds a "convenience" external-API integration.

Prior art, and where the boundary sits

This isn't unprecedented, and it's worth being honest about that. Composio's "brokered credentials" model already keeps the secret away from the agent: a secure middle layer makes the API call on the agent's behalf, and — in their words — "the LLM never sees the token." That's the same core instinct, and they execute it well.

The difference is where the trust boundary sits and how much it covers. Composio is an auth broker: it protects the token from the caller and brokers the request, but it doesn't sandbox your code or constrain where that code can send data — that isn't its job. The credential-free-worker model treats the executor itself as hostile, and credential isolation is only one of four composed controls: the code making the call is untrusted WASM, so it also runs in a capability-scoped sandbox, talks to the controller only over signed RPC, and is bound by a per-actor data-egress ceiling. Composio answers "the agent shouldn't hold the token." This answers "the agent shouldn't hold the token — and shouldn't be trusted with anything else, either." If your executor is your own code (or a vetted broker's), Composio's boundary is plenty. If your executor is code you didn't write, you want the deeper one.

What it costs

A few honest tradeoffs.

Module ergonomics. The opaque-handle pattern means guest module authors can't write let api_key: String = env::var("ANTHROPIC_API_KEY").unwrap(). They have to think about which host primitive they need (header? HMAC signing? basic auth?) and use the appropriate WIT-bound function. For the modules I've written from scratch, this has been a five-minute consideration. For library-style modules ported from existing ecosystems, it's a real re-architecting exercise — some existing crates assume direct credential access in ways that don't translate.

Cross-process latency. The signed-RPC pattern adds latency per RPC: HMAC + nonce cache + freshness check + canonical-bytes signing. In Talos's measurements, this sits in the low-hundreds-of-microseconds range per RPC under realistic load (mostly DashMap hashing + the JSON canonicalization required for deterministic signing of nested Value payloads) — fast enough to disappear next to an LLM round-trip, slow enough to matter for batch workloads with many small RPCs. The mitigation is batching: push work into single signed envelopes where you can, so the per-RPC fixed cost is amortized across many operations. For the workloads I care about — agent steps that make a handful of external calls and a few memory reads — it sits comfortably in the noise next to the LLM round-trips themselves. (Measure your own; these numbers are from one deployment's hardware and traffic shape.)

A retained trust anchor, not zero-trust. As covered above: the worker still holds WORKER_SHARED_KEY (plus a handful of non-user-plane infrastructure keys). This architecture is the right default — it removes the dozens-of-standing-user-credentials failure mode that every in-process runtime ships with — but it's a blast-radius reduction, not elimination. If your threat model demands that a single worker compromise reveal nothing exploitable, you need the next layer: per-worker subkeys and per-actor RPC authorization on top of what's described here.

Where to look

Talos is open-source under MIT OR Apache-2.0. The runtime pieces described in this post live in:

It's pre-1.0. Wire formats are still stabilizing, the API hasn't been deployed against an SLA yet, and there are corners I know are rough. If you're working on agent runtime security, AI sandboxing, AppSec for agentic systems, or just thinking about credential isolation in untrusted-code execution, I'd genuinely be glad to hear from you — open a GitHub Discussion on the repo, or reach out via the contact links on my profile.

The pattern, I think, generalizes. None of the four pieces are specific to Talos or to Rust or to WASM. The signed-RPC layer would work over gRPC or NATS or in-process channels equally well. The AEAD-bound encrypted envelopes are just AES-GCM with a deliberate AAD choice. The opaque-handle pattern works in any language with a capability-based foreign-function boundary. The pre-resolved allowed_secrets declaration is just a manifest field.

What's specific is the composition — and the willingness to push the operational complexity onto the controller side so that the worker can be the thing you don't have to trust.