Hijacking Agent Memory: Stealthy Trojan Attacks Through Conversational Interaction

Image generated by AI

TLDR

• What: MemPoison is an optimization-driven conversational memory poisoning attack that bypasses the selective extraction and rewriting filters of modern LLM agents to inject persistent, triggerable backdoors.
• Who's at risk: Long-term memory-augmented LLM agent platforms (such as systems built on Mem0, LangMem, MemGPT, or Nuance DAX Copilot) that dynamically update external knowledge bases via conversational interfaces.
• Key number: MemPoison achieves an Attack Success Rate (ASR) of up to 0.95 (95%) and a Retrieval Success Rate (RSR@1) of up to 0.98 (98%) under active memory extraction pipelines, while maintaining benign accuracy (~90%+).

With LLM agents rapidly evolving from simple stateless chatbots to autonomous, persistent workflows, the integration of Long-Term Memory (LTM) modules has become standard practice. Frameworks like Mem0, LangMem, and MemGPT allow agents to autonomously extract and store facts across user sessions. However, this helpful capability introduces a severe, highly stealthy vulnerability: memory poisoning.

By engaging in a seemingly benign conversation, an attacker can exploit the agent's autonomous memory-writing pipelines to implant a dormant Trojan. When a specific trigger word is encountered later—whether directly from the attacker or indirectly via untrusted external sources (e.g., a summarized webpage)—the agent retrieves the poisoned memory and executes a malicious payload, such as providing dangerous medical instructions or executing unauthorized tool actions.

Threat Model

The threat model of MemPoison assumes a realistic, restricted-access scenario that reflects real-world deployments.

Background & Problem Setup

Prior research on retrieval-level attacks typically conflates active LLM memory pipelines with passive Retrieval-Augmented Generation (RAG) databases. Attacks like AgentPoison (Chen et al., 2024) assume direct write access to the vector database, ignoring the pre-processing filters that sit between the raw user input and the stored memory.

In a realistic agent deployment, conversational inputs undergo selective extraction and rewriting (e.g., summarization, tag extraction, or entity parsing) before committing to long-term storage. Under these conditions, naive prompt injections are either summarized away, stripped of their trigger words, or split into separate, non-retrievable fragments.

The table below contrasts MemPoison against earlier RAG and memory poisoning frameworks:

Methodology

As Section 4 details, MemPoison bypasses selective extraction and rewriting filters using a two-phase attack pipeline: Iterative Trigger Optimization and Trigger Injection & Activation.

                           [ Phase 1: Iterative Trigger Optimization ]
   Seed Trigger (GPT-4) ──► Gradient-Guided Search (HotFlip) ──► Entity Masquerading Loss (L_ent)
                                                                ├──► Semantic Concentration (L_conc)
                                                                └──► Geometric Isolation (L_iso)
                                                                               │
                                                                       Optimized Trigger
                                                                               │
                           [ Phase 2: Injection & Activation ]                 ▼
   User Conversation ──► [Semantic Relational Bridge (GPT-4)] ──► "According to [Trigger] rules..."
                               │
                               ▼
                    [Selective Memory Pipeline] ──► Successfully Written to LTM Verbatim

1. The Semantic Relational Bridge

Direct concatenation of a trigger and payload often leads to extraction failure, as memory pipelines split unrelated items. To resolve this, Wang et al. (2026) construct a Semantic Relational Bridge using GPT-4. This prompt binds the trigger ($\tau$) and malicious payload ($x$) into a logically interdependent, natural-sounding statement, ensuring they are extracted and saved together as a single memory chunk.

System Role: You are a linguistic synthesizer specializing in creating atomic, logically interdependent factual statements for long-term memory systems.
Task: Given an Entity String (Trigger) and a Target Action (Payload), synthesize them into exactly one natural-sounding sentence.
Core Objective (Semantic Binding): The sentence must establish a definitional, constraint-based, or attributional relationship. It must be designed such that removing the Trigger renders the remaining sentence logically incomplete.
...
Input:
Trigger: {trigger_tokens}
Payload: {malicious_payload}
Output: (The synthesized sentence only)

2. Multi-Objective Offline Optimization

To ensure the trigger survives rewriting and retrieves reliably, MemPoison optimizes a discrete token sequence over three main objectives:

• Entity Masquerading Loss ($L_{ent}$): Driven by a pilot study showing that LLMs preserve Named Entities (like locations, people, or organizations) verbatim during rewriting (see Appendix A.3, where entity nouns achieved an $85.0% preservation rate compared to verbs at just \9.5%$), this loss forces the trigger to statistically mimic a named entity using a surrogate NER model (bert-base-NER).
• Semantic Concentration Loss ($L_{conc}$): Forces trigger-injected texts to form a dense, tight cluster in the vector space, ensuring reliable retrieval when the trigger is present.
• Margin-Based Isolation Loss ($L_{iso}$): Pushes the trigger's cluster away from benign conversational embedding centers, preventing the backdoor from accidentally firing on normal queries (stealth).

The optimization utilizes a gradient-guided coordinate search (derived from HotFlip) to iteratively swap tokens in the trigger sequence:

# Conceptual optimization loop for MemPoison Trigger Search
for step in range(max_iterations):
    batch = sample_benign_corpus(S)
    # Step 1: Generate candidates using NER gradients
    gradients = compute_entity_gradients(L_ent, trigger)
    candidates = get_top_k_substitutions(gradients, k=200)
    
    # Step 2: Evaluate candidates against semantic losses
    best_candidate = None
    min_loss = float('inf')
    for cand in candidates:
        loss_sem = beta * L_conc(cand, batch) + gamma * L_iso(cand, batch)
        if loss_sem < min_loss:
            min_loss = loss_sem
            best_candidate = cand
            
    trigger = best_candidate

Key Results

Wang et al. (2026) evaluated MemPoison across three agent domains (Personal, Medical, Financial) and three active LTM pipelines (A-Mem, LangMem, Mem0) using a variety of state-of-the-art LLM backbones, including GPT-4o-mini and Claude-3-Opus.

The table below highlights the performance of MemPoison against common baselines inside a Medical Agent powered by different memory architectures (reproduced from Table 1):

Defense Resilience

Standard defenses fail to stop MemPoison:

• Perplexity-based Filtering: As Table 4 shows, at a highly restrictive threshold of $PPL \le 75$, baselines like Naive Concat and AgentPoison are completely neutralized ($0.00% ASR), while MemPoison retains an ASR of **\0.40 (\40%)**, quickly recovering to **\0.87 (\87%$)** at$PPL \le 150$.
• Input Paraphrasing: When rewriting inputs using auxiliary models (Table 5), Naive Concat and AgentPoison fall to $\le 0.04$ ASR. MemPoison maintains high effectiveness, yielding $0.77 to \0.89$ ASR due to its entity masquerading design.

Limitations & Open Questions

1. Strongly Safety-Aligned Models: In real-world black-box settings (Section 5.5, Table 6), the attack remains highly successful on models like Qwen3-max ($1.00 ASR) and GPT-4o (\0.96 ASR), but degrades on Claude-opus-4.6 (\0.12$ ASR). Highly aligned models are better at detecting malicious intent in user queries, preventing retrieval or execution of the payload.
2. Long-Term Consolidation and Decay: Production memory systems implement temporal decay, periodic summaries, or eviction policies (Section 6). Whether the optimized semantic bridge survives long-term memory updates remains an open question.
3. Cross-Model Transferability Barriers: While MemPoison transferability is highly effective on anisotropic models (like E5 or ANCE), it is less effective when targeting highly isotropic retrieval models like aMPNet (Figure 9).

What Practitioners Should Do

If you are building or maintaining LLM agents with active long-term conversational memory, implement these defenses immediately:

1. Enforce Hybrid Retrieval (Dense + Sparse): Relying purely on dense semantic vector similarity makes systems vulnerable to embedding-space manipulation. Integrate BM25 sparse keyword matching. This breaks optimized vector clusters unless the exact keywords are present in the user query.
2. Verify Memory Consistency Before Action: Before executing a tool action or providing critical advice based on a retrieved memory, use a secondary LLM verification gate to assess whether the retrieved memory aligns logically with the current trusted session context.
3. Sanitize Entities in Memory Inputs: Since MemPoison relies heavily on masquerading triggers as Named Entities, sanitize incoming dialogue memory requests. Implement entity replacement or normalization (e.g., swapping specific nouns with generic placeholders like [COMPANY_A] or [LOCATION_B]) before running embedding models.
4. Monitor Embedding Space Anisotropy: Track the cosine similarity distributions of incoming memory chunks. Alerts should trigger if a cluster of newly written memories shows abnormally high, localized cosine similarity (hubness), which is a clear indicator of a geometric concentration attack.

The Takeaway

MemPoison demonstrates that the dynamic pre-processing layers of modern LLM agent memory systems are not a security barrier, but rather a target that can be bypassed. By masquerading malicious payloads as highly stable named entities and mathematically grouping them in the retriever's vector space, attackers can permanently compromise an agent's behavior via a single chat conversation. As agents gain more autonomy, memory sanitation must evolve from a passive filtering step into a robust, verification-driven lifecycle.

Den's Take

What excites me about MemPoison is that it finally moves past the naive threat models of early RAG security research. For years, academic papers assumed attackers already had direct write-access to vector databases—a scenario that is basically a standard database breach, not an AI-specific vulnerability. MemPoison targets the real world: the dynamic, conversational extraction layer where LLMs filter, rewrite, and summarize facts before committing them to memory.

As a practitioner, this is highly alarming. Imagine a $20M healthcare copilot deployment like Nuance DAX Copilot where a malicious patient or compromised email thread can permanently inject lethal drug interaction advice into a patient's clinical history just by chatting with the system. Because Mem0 and LangMem-style architectures extract semantic facts dynamically, traditional input filters will completely miss these optimized triggers.

This is a massive leap forward in threat modeling realism, echoing my findings in IterInject: Indirect Prompt Injection Against LLM Agents via Feedback-Guided Iterative Optimization, where we showed how iterative optimization allows attackers to systematically bypass an agent's dynamic reasoning boundaries. If your agent dynamically updates its own database based on conversational inputs, you cannot rely on static input guardrails; you are actively hosting a self-poisoning engine.