An Efficient and Privacy-Preserving Architecture for Cross-Institutional Collaborative RAG

Image generated by AI

TLDR

• What: FedRAG, a non-cryptographic, federated RAG framework that uses a numerically stable scrambling matrix and token permutation to secure cross-node self-attention without model retraining or Trusted Execution Environments (TEEs).
• Who's at risk: Multi-institutional collaborative LLM systems (e.g., healthcare consortia, financial fraud detection networks, cross-org enterprise RAG) where raw data or intermediate states cannot be shared due to privacy constraints.
• Key number: Achieves up to a $62\times$ latency reduction over existing secure baselines while keeping model utility degradation under $0.1%$.

The rapid deployment of Retrieval-Augmented Generation (RAG) in enterprise pipelines—such as customer support agents, medical diagnostic assistants, and financial risk tools—has run head-first into the barrier of "data silos." Organizations like hospital networks and banks want to pool their knowledge bases to drive LLMs like Llama-3.1 or Qwen3 without leaking sensitive raw documents or vulnerable intermediate key-value (KV) states. Mao et al. (arXiv 2026) introduce FedRAG, a high-throughput, privacy-preserving federated RAG framework that avoids the massive compute and latency traps of secure multi-party computation (MPC) and homomorphic encryption (HE).

Background / Problem Setup

In a collaborative cross-institutional RAG setup, multiple institutions want to run a shared LLM over their combined private documents. However, the self-attention mechanism in modern Transformer architectures mandates that the Query ($Q$) tensor of the active node must access the Key ($K$) and Value ($V$) tensors representing the retrieved contexts of all other nodes.

Historically, privacy-preserving LLM inference has relied on heavy cryptographic primitives or hardware enclaves. The following table highlights where existing paradigms fall short compared to the proposed FedRAG system:

Unlike PoisonedRAG (Zou et al., CCS 2024), which focuses on defending retrieval pipelines against adversarial document poisoning, or PermLLM (Zheng et al., arXiv 2024), which heavily relies on expensive cryptographic operators for computing attention, FedRAG secures intermediate states directly using a hardware-friendly algebraic transformation.

Methodology

FedRAG's core contribution is the Scrambled Distributed Attention protocol. As Section 4.1 describes, standard self-attention is decomposed into per-node local computations followed by a lightweight global aggregation:

Step 1: Local Computation (Each Node i)
    O_i = softmax( (Q * K_i^T) / sqrt(d) ) * V_i
    w_i = sum_over_rows( exp( (Q * K_i^T) / sqrt(d) ) )

Step 2: Global Aggregation (Compute Node)
    Attention(Q, K, V) = sum(w_i * O_i) / sum(w_i)

To prevent the Compute Node or other curious nodes from seeing the raw $Q$, $K$, and $V$ matrices, FedRAG applies a mathematically rigorous scrambling matrix $\Phi$.

1. Constructing a Numerically Stable Scrambling Matrix

A naive random matrix introduces floating-point precision errors that degrade model utility over dozens of layers. To solve this, Mao et al. (arXiv 2026) structure the scrambling matrix as:

• $H$ (Normalized Hadamard Matrix): Consists of ${-1, +1}$ entries. It allows matrix multiplication to be computed via stable sign-flipping additions, ensuring numerical stability in low-precision arithmetic formats (such as BF16 or FP16).
• $P_1, P_2$ (Permutation Matrices): Randomize the deterministic Hadamard structure.
• $S_1, S_2$ (Random Diagonal Scaling Matrices): Sampled from an empirical range (e.g., $[1/8, 8]$), these matrices distort $L_2$ distances to defeat topological graph-matching attacks.

2. Scrambled Attention Execution

The dual properties of inner product preservation (Lemma 1) and linearity (Lemma 2) allow standard attention to be executed on scrambled tensors without exposing plaintext. The workflow is implemented as follows:

# 1. Plaintext Matrix Scrambling (Executed by Inquirer & Context Owner)
# Q_prime = P_Q * Q * Phi_KQ
# K_prime = P_KV * K * Phi_KQ_inv_T
# V_prime = P_KV * V * Phi_V

# 2. Scrambled Attention Computation (Executed by Compute Node)
Attention_prime = softmax((Q_prime * K_prime.T) / sqrt(d)) * V_prime

# 3. Output Descrambling (Executed by Inquirer)
# Attention = P_Q_inv * Attention_prime * Phi_V_inv

3. Dynamic Role Assignment

To prevent key exposure, a single node cannot act as both a Context Owner (supplying $K, V$) and the Compute Node (performing the scrambled computation) during the same step. FedRAG dynamically assigns these roles across at least three physical nodes.

Key Results

Mao et al. (arXiv 2026) evaluated FedRAG across several open-source LLMs (including Qwen3-4B, Llama-3.1-8B, and Ministral-14B) on standard QA and summarization datasets.

End-to-End Latency and Efficiency

Table 5 in the paper presents detailed performance configurations. The table below highlights the performance of the meta-llama/Llama-3.1-8B-Instruct model under the "Long" context setting (4096 prefill tokens, 128 decoded tokens):

FedRAG achieves a $5\times$ reduction in TTFT and over $2\times$ improvement in decode TPS compared to SCX, while scaling linearly as sequence lengths increase.

Downstream Task Accuracy

As Section 7.4 details, the numerical precision of the scrambling protocol preserves downstream performance. Table 3 lists evaluation metrics across six benchmarks:

The average accuracy drop across all evaluated configurations is roughly $0.06%$, proving that the algebraic scrambling method maintains near-zero utility degradation compared to standard, unprotected attention.

Limitations & Open Questions

While FedRAG represents a significant leap forward for collaborative RAG efficiency, security practitioners must consider several core constraints:

1. Minimum Node Requirement: The protocol requires at least three independent physical nodes. If a deployment only consists of two nodes, one node must play double duty, which breaks the mathematical security guarantees and exposes the scrambling keys.
2. Replication Overhead: In highly constrained, small-scale deployments with only a few nodes, the dynamic role assignment forces certain segments of the KV cache to be replicated, leading to an average throughput degradation of approximately $10%$.
3. Reactive Game-Theoretic Defenses: Against malicious prompt injection attacks designed to exfiltrate other nodes' documents, FedRAG relies on a reactive, audit-based logging protocol rather than proactive, cryptographically enforced input filtering.

What Practitioners Should Do

If you are deploying collaborative, cross-organizational RAG pipelines in healthcare, finance, or legal tech:

1. Implement Hadamard Feature Scrambling: Instead of expensive homomorphic encryption pipelines, implement the $S_1 P_1 H P_2 S_2$ transformation inside custom PyTorch attention layers or Triton kernels. This ensures standard GPU matrix engines perform the "encryption" natively.
2. Configure 4-Bit Reranker Quantization: As Section 7.5 highlights, reranker intermediate states can be safely compressed to 4-bit representation using standard affine min-max quantization with zero downstream accuracy loss, reducing communication overhead during the document re-ranking stage by up to $70%$.
3. Deploy with at Least Three Nodes: Ensure your orchestration layer (e.g., Kubernetes) enforces a strict anti-affinity policy. This guarantees that the Inquirer, Context Owner, and Compute Node roles are mapped to distinct, non-colluding physical nodes.
4. Implement Ciphertext Auditing: Since the system is vulnerable to malicious prompt structures, maintain a secure, append-only ledger of query ciphertexts and scrambling key histories to enable retroactive auditing if an organization is suspected of attempting data exfiltration.

The Takeaway

FedRAG demonstrates that protecting intermediate states in federated LLM setups does not require sacrificing real-time throughput to heavy cryptographic protocols. By leveraging hardware-friendly, numerically stable linear transformations, organizations can safely pool their private databases to unlock collective RAG intelligence at scale.

Den's Take

I love seeing papers that prioritize engineering pragmatism over cryptographic purity. For years, the academic consensus for privacy-preserving collaborative LLMs was to throw Secure MPC or homomorphic encryption at the problem—which inevitably ground production systems to a halt. A $62\times$ latency reduction is the difference between an unusable academic prototype and a viable enterprise deployment.

Imagine a $100M healthcare consortium attempting to pool patient oncology records across Mayo Clinic and regional hospitals to power a shared clinical diagnostic assistant. They cannot afford the latency of standard MPC, nor can they risk a single leaked patient history. FedRAG's approach of token permutation and scrambling matrices makes collaborative RAG genuinely viable for high-throughput, multi-institutional workloads.

However, my major concern lies in the "honest-but-curious" threat model. In my analysis of the Security of Autonomous AI Agents: Trust Boundary-Based Attack Surface Analysis and Trends, I emphasized that relying on participants to strictly follow protocol is a recipe for disaster once malicious actors breach the perimeter. If a compromised node in the federated network begins actively manipulating its input queries to reverse-engineer the permutation keys, algebraic defenses can quickly unravel. Practitioners should treat this as a solid defense-in-depth performance optimization, not an absolute cryptographic silver bullet.