Preventing cross-tenant leaks in a shared retrieval index

TL;DR — In a shared retrieval index, the most likely leak is not the LLM itself; it is retrieval returning another tenant's chunks because tenant filtering is optional, inconsistent, or bypassed in one code path. The default-safe design is: stamp every chunk with an immutable tenant_id at ingest, require server-side filter injection on every query, and fail closed if the filter is missing or the result set contains mixed tenants. Reading time: ~7 min

What it is and where it sits

In a multi-tenant RAG system, a shared retrieval index is the component that stores embeddings and metadata for documents from many customers in one logical search system. It sits between your application API and the LLM prompt builder. It usually replaces one of two simpler designs:

• per-tenant indexes or databases, where isolation is mostly physical/logical separation
• keyword-only search over tenant-scoped tables, where app-layer SQL WHERE tenant_id = ... did the isolation work

The risk appears when you move to vector or hybrid retrieval and treat tenant scoping as a query option instead of a mandatory access-control boundary. If one endpoint, worker, or debug script omits the filter, the retriever can return chunks from the wrong tenant. Once those chunks are in the prompt, you have already leaked data.

Typical request/data flow:

[User] ---- 1. ask question ----> [App API]
[App API] ---- 2. authn/authz ----> [Tenant Context]
[Tenant Context] ---- 3. query + tenant filter ----> [Retriever]
[Retriever] ---- 4. top-k chunks ----> [Prompt Builder]
[Prompt Builder] ---- 5. grounded prompt ----> [LLM]
[LLM] ---- 6. answer ----> [App API]
[App API] ---- 7. response ----> [User]

[Ingest Worker]
|
v
[Chunker] ---- 8. embed + metadata ----> [Retriever]

What talks to it:

• ingest workers writing chunks, embeddings, and metadata
• application APIs issuing retrieval queries
• offline jobs doing reindexing/backfills
• admin/debug tools that are often the first place isolation gets bypassed

Where it lives:

• often a vector-capable database or search engine
• sometimes a separate retrieval service in front of storage
• almost always behind an internal API, but that does not remove the need for tenant enforcement

The architecture point that matters: tenant isolation must be enforced before results leave the retriever, not after ranking, not in the UI, and not by hoping the LLM ignores bad context.

How it actually works

Use one realistic model: a SaaS app with tenants acme and globex. Both upload PDFs. Your ingest worker chunks text, computes embeddings, and stores records like:

• id
• tenant_id
• doc_id
• chunk_id
• embedding
• text
• acl_version or visibility

A user from acme asks: "What is our invoice approval threshold?"

Step 1: Ingest stamps tenant identity once

When the file is uploaded, your API derives tenant_id from the authenticated session or service token, not from client JSON. The ingest worker writes every chunk with that tenant_id as immutable metadata.

Bad pattern:

• client sends { "tenant_id": "acme" }
• worker trusts it
• replay or bug writes chunks under the wrong tenant

Safer pattern:

• API resolves tenant from auth context
• worker receives signed/internal job payload with tenant_id
• storage schema rejects null or changed tenant IDs

Step 2: Query path injects the filter server-side

At query time, the app API authenticates the user, resolves tenant_id=acme, then calls retrieval with a hard filter. The caller should not be able to disable it with a request flag like include_all=true.

Pseudo-request shape to the retriever:

{
  "query": "What is our invoice approval threshold?",
  "top_k": 8,
  "filter": {
    "tenant_id": "acme",
    "visibility": "active"
  }
}

The important part is where that filter comes from: your server auth context, not the browser, not the mobile app, not a prompt variable.

Step 3: Retriever applies filter before or during candidate selection

This is where systems differ. Some engines support metadata filtering during ANN search; some do ANN first then filter; some do brute-force exact search under the filter. The security requirement is simple even if performance differs: no result from another tenant may leave this boundary.

If your engine does post-filtering after global ANN candidate generation, you can get poor recall for small tenants because top candidates may mostly belong to other tenants and get dropped. That is a quality problem, not a leak, but teams often "temporarily" remove filtering to improve recall. That is how leaks happen.

Step 4: Prompt builder verifies result tenancy again

This is the belt-and-suspenders check. Before assembling the prompt, assert that all returned chunks have tenant_id=acme. If any chunk differs, log at error level, increment a security metric, drop the response, and return a 500/incident ID rather than a partial answer.

Example log shape:

{
  "level": "error",
  "msg": "cross-tenant retrieval result blocked",
  "request_id": "req_8f2f",
  "expected_tenant": "acme",
  "returned_tenants": ["acme", "globex"],
  "query_path": "/v1/ask"
}

Step 5: LLM only sees tenant-scoped chunks

Now the prompt builder sends only acme chunks to the LLM. If the answer is wrong, that is a relevance issue. If the answer includes globex data, your retrieval boundary already failed.

Step 6: Reindexing and backfills preserve tenant metadata

A common leak path is not the online query API but a maintenance job. Re-embedding, schema migration, deduplication, or chunk merging can accidentally drop metadata fields. Then every query filter silently stops working because records no longer carry tenant_id.

Add a post-job verification query that counts records with missing tenant metadata and aborts cutover if non-zero.

psql "$DATABASE_URL" -c "select count(*) as missing_tenant from retrieval_chunks where tenant_id is null;"

Realistic output:

 missing_tenant
----------------
              0
(1 row)

If you see anything else, do not switch traffic.

When to use it (and when not to)

A shared retrieval index is worth it when operational simplicity or cost matters more than hard isolation by construction. But do not choose it because it feels architecturally elegant.

You probably do not need a shared index if:

• you have fewer than a few dozen tenants and can afford one index per tenant
• your customers require customer-managed keys plus strict operational separation
• your team does not have the discipline to route all retrieval through one service boundary

Trade-offs

Every benefit here has a cost.

• Lower infrastructure footprint than per-tenant indexes
  • Cost: stronger need for application-enforced isolation, more dangerous mistakes, harder audits
• Better utilization for many small tenants
  • Cost: noisy-neighbor effects in ANN recall, cache locality, and query latency
• Simpler fleet management and fewer indexes to migrate
  • Cost: migrations/backfills become high-blast-radius events
• Easier global relevance tuning on one corpus shape
  • Cost: tuning can bias toward large tenants and hurt small-tenant retrieval quality
• Shared operational tooling
  • Cost: admin/debug tooling becomes a privileged path that must be tightly controlled and logged

Latency trade-off experienced teams care about: exact filtered search is safer and simpler to reason about, but expensive at scale. ANN with metadata filtering is faster, but behavior under highly selective filters varies by engine. If recall drops under tenant filters, fix the retrieval design; do not relax isolation.

Lock-in trade-off: if you rely on engine-specific metadata filtering semantics, migration gets harder. Keep your contract generic: every record has immutable tenant_id, every query requires it, every response is verified.

In practice

Example 1: Postgres schema and row-level guardrails for retrieval metadata

If your retrieval metadata lives in Postgres alongside embeddings or chunk references, use constraints and RLS to prevent accidental cross-tenant reads/writes. This does not replace vector-store filtering if embeddings live elsewhere, but it is a solid control plane pattern.

⚠️ Enabling RLS on a live table can break existing queries and background jobs immediately. Run this first in staging and test every service account path before applying in production.

alter table retrieval_chunks
  alter column tenant_id set not null;

alter table retrieval_chunks
  add constraint retrieval_chunks_tenant_chunk_uniq
  unique (tenant_id, doc_id, chunk_id);

alter table retrieval_chunks
  enable row level security;

create policy tenant_read on retrieval_chunks
  for select
  using (tenant_id = current_setting('app.tenant_id', true));

create policy tenant_write on retrieval_chunks
  for insert
  with check (tenant_id = current_setting('app.tenant_id', true));

This forces tenant_id presence, prevents duplicate chunk identity collisions across tenants, and scopes reads/writes to the session variable app.tenant_id. Gotcha: if your connection pool reuses sessions and you forget to set or reset app.tenant_id per request, one request can inherit another tenant context.

Set the tenant context explicitly in the same transaction:

psql "$DATABASE_URL" <<'SQL'
begin;
select set_config('app.tenant_id', 'acme', true);
select count(*) from retrieval_chunks;
commit;
SQL

Expected output shape:

BEGIN
 set_config
------------
 acme
(1 row)

 count
-------
   842
(1 row)

COMMIT

If you forgot to set the variable, many setups will return an error like:

ERROR:  unrecognized configuration parameter "app.tenant_id"

or zero rows, depending on policy design. Pick one behavior deliberately; fail-closed is better than silent broad reads.

Example 2: Application-side retrieval wrapper that fails closed

Wrap your retriever so every code path goes through one function that injects the tenant filter and validates returned metadata.

from typing import Any

class CrossTenantLeakError(RuntimeError):
    pass

async def retrieve_for_tenant(retriever: Any, *, tenant_id: str, query: str, top_k: int = 8):
    result = await retriever.search(
        query=query,
        top_k=top_k,
        filters={"tenant_id": tenant_id, "visibility": "active"},
    )

    bad = [r for r in result if r.metadata.get("tenant_id") != tenant_id]
    if bad:
        returned = sorted({r.metadata.get("tenant_id") for r in result})
        raise CrossTenantLeakError(
            f"cross-tenant retrieval blocked expected={tenant_id} got={returned}"
        )

    return result

This centralizes the only allowed query path and turns mixed-tenant results into a hard failure. Gotcha: do not expose a lower-level retriever.search() to feature teams without code review guardrails, or someone will bypass the wrapper for a batch job or "temporary" experiment.

A realistic HTTP failure from your API if the guard trips:

{
  "error": "retrieval_isolation_violation",
  "request_id": "req_8f2f",
  "message": "retrieval returned documents outside tenant scope"
}

Example 3: Backfill verification before cutover

After reindexing, verify there are no null tenant IDs and no mixed-tenant docs caused by bad joins.

psql "$DATABASE_URL" <<'SQL'
select count(*) as null_tenant_rows
from retrieval_chunks
where tenant_id is null;

select doc_id, count(distinct tenant_id) as tenant_count
from retrieval_chunks
group by doc_id
having count(distinct tenant_id) > 1
limit 20;
SQL

This catches the two boring but common migration bugs: dropped metadata and duplicated documents under multiple tenants. Gotcha: if doc_id is only unique inside a tenant, the second query will false-positive unless you use the true external document identity you expect to be single-tenant.

Further reading

• PostgreSQL Documentation: Row Security Policies
• OWASP Cloud Tenant Isolation Project
• NIST SP 800-207 Zero Trust Architecture
• RFC 7519: JSON Web Token (JWT)
• Designing Data-Intensive Applications

This article was written by an AI system and published pending human review. Verify anything you intend to act on.