Multi-Tenant AI Architecture: Isolation by Default

A multi-tenant AI system can authenticate the right user and still return another customer's document. The leak happens after login: in a vector query, a shared cache, an agent checkpoint, a tool call or a trace viewer. Fixing the API gateway while those layers stay tenant-blind does not close it, and no prompt instruction can, because the model must not control the identity that limits the model.

Why edge authentication is not enough

AWS calls tenant isolation foundational to SaaS design in its tenant isolation strategies whitepaper, because shared infrastructure must still keep one tenant out of another's resources. An AI workflow adds stateful layers to that problem. Take a support agent serving two workspaces, which loads records from PostgreSQL, retrieves policies from a vector store, restores a checkpoint, checks a response cache, lets the model pick a CRM tool and writes prompts and results to a trace platform. Each step can lose or ignore the tenant. A query can omit its predicate, a retrieval client can default to a shared namespace, a cache key can cover the prompt but not the tenant, a thread ID can be unique only inside one workspace, and a tool can accept a model-supplied tenant_id.

One isolation contract

Write the contract before choosing pooled or dedicated infrastructure. For every operation, the service derives one tenant identity from authenticated server-side context. Every stateful dependency receives it through a typed interface and enforces it on its own. User text, retrieved content, model output and tool arguments can never create or replace it. Four consequences follow:

• Tenant context is fixed for one operation. Switching workspaces takes a new authorized request, and a prompt cannot do it mid-run.
• The boundary is enforced in more than one layer, so one missing filter does not expose data.
• An identifier is not authorization. A document ID, thread ID or namespace name proves nothing until it is resolved inside the verified tenant.
• Queue messages, retries, scheduled continuations and approval callbacks carry a signed or server-resolved tenant reference, never a free-form payload field.

Microsoft's multitenant AI guidance separates training from inference concerns, which matters when a shared model is acceptable but shared customer context is not.

Carry tenant context outside model control

Build a server-owned context object after authentication and make every lower-level client require it:

type TenantContext = Readonly<{
  tenantId: string;
  actorId: string;
  requestId: string;
  allowedToolScopes: readonly string[];
}>;

async function runAgent(input: UserInput, ctx: TenantContext) {
  const records = await tenantDb(ctx).loadCase(input.caseId);
  const evidence = await tenantVectorStore(ctx).search(input.question);
  const checkpoint = await tenantState(ctx).load(input.threadId);
  return agent.execute({
    input,
    records,
    evidence,
    checkpoint,
    tools: tenantTools(ctx),
  });
}

Tool schemas should leave out tenantId, and the server wrapper injects the verified value. For background jobs, pass an opaque run reference that the worker resolves from a trusted table. If a payload carries both a run ID and a tenant ID, reject a mismatch instead of picking one.

Shared state leaks across tenants even when the tenant check is right. In March 2023 OpenAI took ChatGPT offline after a bug in the open-source redis-py client let some users see other active users' chat titles, and exposed payment details for 1.2% of ChatGPT Plus subscribers active in a nine-hour window. The failure mode in the redis-py issue is mundane: an async command cancelled after it was sent but before its reply was read leaves the pooled connection one reply behind, so the next caller to borrow it reads the previous caller's answer. Tenant IDs in the cache key do not help when the reply itself is misrouted. Treat a cancelled or timed-out call on a shared connection as poisoned, close it instead of returning it to the pool, and keep a cross-tenant canary test that runs under cancellation.

Enforce relational isolation below the query builder

A mandatory WHERE tenant_id = ? is a convention, and one missed predicate exposes data. PostgreSQL row-level security moves the check into the database:

ALTER TABLE agent_threads ENABLE ROW LEVEL SECURITY;
ALTER TABLE agent_threads FORCE ROW LEVEL SECURITY;

CREATE POLICY tenant_threads ON agent_threads
USING (tenant_id = current_setting('app.tenant_id', true)::uuid)
WITH CHECK (tenant_id = current_setting('app.tenant_id', true)::uuid);

Set app.tenant_id from the context at transaction start, and connect with a role that cannot bypass the policies. The docs say superusers and roles with BYPASSRLS always bypass row security, and table owners normally do unless FORCE ROW LEVEL SECURITY is set.

Row security still has a hole that tenant-aware code does not fix. The same page warns that referential integrity checks, meaning unique, primary key and foreign key constraints, always bypass row security, and that schemas and policies need care to avoid covert-channel leaks. A globally unique constraint on something like a workspace slug lets tenant A learn that tenant B holds it, by trying to insert it. Scope uniqueness to the tenant (UNIQUE (tenant_id, slug)) wherever the value is not meant to be global. The docs also point out that row_security = off makes a query raise an error if a policy would have filtered its results, which is a good setting for backup and export jobs that must never silently drop rows.

Separate schemas, databases or accounts give stronger isolation for regulated or very large tenants. Keep the same application contract so moving one tenant to its own silo changes no authorization semantics.

Partition vector retrieval before the query runs

Filtering by tenant_id inside a shared index makes isolation depend on every query carrying the filter. Prefer a partition the store requires before searching. Pinecone's multitenancy guide recommends one namespace per tenant, and notes that offboarding is deleting the namespace. It also lists namespace limits per serverless index that depend on the plan, from 100 on Starter to 1,000,000 on Enterprise, so check that a per-tenant design fits your plan before you commit to it. Weaviate's multi-tenancy likewise requires operations to address a tenant.

Wrap the vector client so no code path can issue an unscoped query, derive the namespace from trusted context, reject unknown partitions, and never auto-create a tenant on a read, since a misspelled name would produce a quiet empty partition and hide the routing bug. Test ingestion and deletion as well as search, because a worker that indexes into the wrong namespace or an offboarding that leaves embeddings behind defeats a perfect query path.

Caches, checkpoints and model context

Cache keys usually describe content and forget ownership. A semantic cache keyed on an embedding can return an answer built from another customer's records, so prefix every key with the tenant and the policy version that governs sharing. Checkpoints, memory records, pending approvals, retry state, uploaded files and summaries need the composite key (tenant_id, thread_id) or an opaque tenant-scoped one. Never put another tenant's content in the prompt and ask the model to ignore it. Isolation has to happen before prompt assembly. Shared inference can still be fine when customer data does not train or tune the model, and that decision should be documented separately.

Tools and telemetry

A tool is a privileged server operation. The model may choose update_ticket and give an ID, and the server resolves that ID inside the tenant and rejects a foreign object. Mint short-lived downstream credentials scoped to tenant, resource and action where you can. Resolve the target again when approval is recorded and again just before execution, since an approver can be shown a misleading label. The OWASP agentic threats and mitigations guide is a workable template for the threat model at each tool boundary.

Traces and evaluation stores hold some of the most sensitive data in the system. Tag them with a verified tenant ID, enforce tenant access in the API behind the viewer rather than in the UI, require a separate audited support role for cross-tenant views, and apply the same rules to exports. A random trace ID makes guessing hard and grants nothing.

Prove it with negative tests

Create two synthetic tenants with recognizable records, for example RED-ONLY-741 and BLUE-ONLY-286. Under Red's context, try to retrieve Blue's marker in natural language, load Blue's object ID from the repository, query the vector client with Blue's document ID, replay a prompt that would collide with a Blue cache entry, load the same thread ID under both tenants, pass Blue's identifiers as tool arguments, resume a queued job after editing its tenant field, and open Blue's trace URL with a Red operator session. Then delete Red and confirm that its rows, vectors, files, caches and checkpoints are gone while Blue is untouched. Assert on the denial and on the empty result, not only on the absence of Blue text in the answer, and run the suite against every storage mode you support, including dedicated silos.

Start with one production workflow. Draw its tenant path from login through storage, retrieval, state, tools and telemetry, mark each point that enforces tenant identity and each that merely trusts a caller-supplied value, and replace the first trust-only point with the typed context.

References

• Microsoft architectural approaches for multitenant AI and machine learning for training, inference, data and model isolation choices.
• AWS SaaS Tenant Isolation Strategies for tenant isolation as a foundational SaaS requirement.
• PostgreSQL row security policies for bypass roles, forced policies, referential-integrity covert channels and row_security = off.
• Pinecone multitenancy guidance for tenant namespaces, offboarding and per-plan namespace limits.
• Weaviate multitenancy operations for tenant data partitions and tenant lifecycle operations.
• OWASP agentic AI threats and mitigations for threat-model-based review of agents connected to business tools.
• redis-py issue 2624 for the cancelled-command bug that leaves a pooled connection one reply behind.
• BleepingComputer on the March 2023 ChatGPT incident for the 1.2% and nine-hour figures.

---