TLDR
- What: Indirect data poisoning manipulates public repositories with falsified tabular data and deceptive metadata to trick autonomous AI research agents into drawing fabricated scientific conclusions.
- Who's at risk: Autonomous agentic RAG setups, LLM-based data analysis pipelines (such as Claude Code, Codex, or Gemini CLI), and automated benchmarking workflows.
- Key number: Poisoning achieved a 49.56% full success rate across all experimental baseline runs, while default agent setups only detected the manipulation in 6.0% of runs.
As research labs and tech companies quickly integrate autonomous AI agents like Claude Code and GPT-based analysis tools into their pipelines, we are delegating critical analytical tasks to LLMs. This paper introduces "indirect data poisoning," a silent attack vector where an adversary corrupts open data repositories (e.g., GitHub, HuggingFace) to compromise scientific integrity. By exploiting the tacit trust AI agents place in online sources, attackers can turn automated systems into unwitting distributors of fabricated findings.
| Attacker | Remote adversary with black-box capabilities; uploads poisoned files and misleading metadata (e.g., READMEs) to public platforms. |
| Victim | Autonomous AI research agents (such as Claude Code with Claude Opus 4.7, Codex with GPT-5.5, and Gemini CLI with Gemini 3.1 Pro) retrieving datasets from public repositories. |
| Goal | Systematically manipulate the agent's downstream analytical conclusions to support or reject a target hypothesis. |
| Budget | Negligible; requires no specialized infrastructure, direct agent access, or costly model training. |
Background / Problem Setup
Traditional data poisoning focuses on corrupting pre-training or fine-tuning datasets to compromise model behavior. In contrast, indirect data poisoning targets the retrieval-stage ecosystem of autonomous agents. The paper defines this as a black-box attack where the agent dynamically retrieves poisoned files during its standard runtime.
| Attack Vector / Work | Poisoning Target | Attacker Capabilities | Primary Objective |
|---|---|---|---|
| Indirect Data Poisoning (This Work) | Public data repositories (GitHub, HuggingFace, Kaggle, OSF) | Black-box; uploads altered datasets + deceptive metadata | Manipulate downstream scientific findings of autonomous agents |
| PoisonedRAG (Zou et al. [Zou+25]) | Database knowledge sources | Black-box database record injection | Alter LLM generation on target RAG queries |
| Training Data Poisoning (Carlini et al. [Car+24]) | Web-scale training corpora | Pre-training scrape manipulation | Inject long-term backdoors or degrade base model performance |
Methodology
The authors evaluated this threat using a rigorous, multi-domain experiment across 450 containerized runs. The attack pipeline operates as follows:
- Dataset Selection: The authors selected five tabular datasets covering social-technical topics: fertility rate analysis in Europe, workplace hiring discrimination, traffic policing racial disparities, autonomous vehicle vs. human driver safety, and the motivational cost of generative AI in sequential tasks.
- Data Manipulation: The authors poisoned the datasets using a combination of four vectors: creating supplementary fake datasets, inserting biased data points, imputing values to invert correlations, and writing deceptive README metadata or data loading code.
- Target Platforms: Poisoned variations were uploaded to private repositories mapped to GitHub, HuggingFace, Kaggle, and the Open Science Framework (OSF). An API-wrapper mock intercepted searches to ensure the private repos appeared public to the agents.
- Agent Testing: Three frontier LLM agents—Claude Code with Claude Opus 4.7, Codex with GPT-5.5, and Gemini CLI with Gemini 3.1 Pro—were evaluated. Each agent was prompted with three tiers of instructions: Minimal, Targeted, and Critical.
- Mitigation Testing: The authors developed two prompt-level defenses:
- Scientist Persona: An updated system prompt instructing the agent to act as an intellectually honest, statistically rigorous, and scientifically critical researcher.
- Data Provenance Audit: A custom audit skill executing a suite of five independent and parallel checks: finding referencing papers, verifying social markers, checking for statistical anomalies, comparing to related datasets, and cautioning against the possibility of data poisoning.
Key Results
The evaluation showed that baseline agents are highly susceptible to indirect poisoning. Even when prompted with a Critical tone, agents frequently propagated poisoned conclusions. However, implementing a structured Data Provenance Audit successfully mitigated the threat.
| Mitigation Condition | Dataset Retrieval/Used Rate | Full Attack Success Rate (ASR) | Poisoning Detection Rate |
|---|---|---|---|
| Baseline (No Mitigations - All Prompts) | 84.22% | 49.56% | 6.00% |
| Scientist Persona | 80.70% | 16.67% | 30.67% |
| Data Provenance Audit | 40.00% | 0.00% | 77.33% |
As illustrated in the study, the Data Provenance Audit is highly effective, bringing the full attack success rate down to 0.0%. However, this comes with a minor tradeoff: a 2.0% false-positive rate where clean, anomalous datasets were incorrectly flagged as poisoned due to how the agents aggregated the audit results.
Limitations & Open Questions
- Socio-Technical Target Biases: The tested domains rely heavily on soft, socio-technical datasets where agents lack firm mathematical or physical priors. Hard sciences (such as chemistry or physics) may exhibit different vulnerability profiles.
- Tabular Focus: The experimental pipeline was designed strictly around tabular files. Unstructured data corpora (e.g., PDFs, images) present distinct poisoning surfaces that were not analyzed.
- Sandbox Simplification: To avoid distributing real misinformation, the experiments utilized private repositories with an API wrapper. In real-world environments, search engine indexers and platform-specific ranking algorithms could impact how easily agents discover poisoned data.
What Practitioners Should Do
- Sandboxed Data Ingestion: Do not permit autonomous agents to download and execute code or load datasets from unverified third-party repositories directly into production runtimes.
- Implement Provenance Audits: Prior to analysis, instruct agents to query credible citation databases or search engines to verify if the retrieved dataset is referenced or used in at least one credible scientific publication.
- Automate Statistical Anomaly Scans: Instruct agents to perform statistical validation checks (e.g., calculating simple metrics like range, mean, median, standard deviation) on all incoming datasets to identify obvious anomalies before passing them to downstream LLM executors.
- Deploy Rigorous System Personas: Update agent system prompts to enforce a skeptical, scientific role that actively looks for data irregularities and conflicts of interest.
The Takeaway
Autonomous AI research agents suffer from "tacit institutional trust," making them highly vulnerable to indirect data poisoning. Simply telling agents to be critical is not enough to stop these attacks. To build secure agentic workflows, practitioners must implement structured, multi-step verification checks directly into the data retrieval pipeline.
Den's Take
As someone who builds and breaks agentic systems, this paper scares the hell out of me. We are rushing to deploy autonomous RAG pipelines and research agents that pull directly from public repositories like GitHub and HuggingFace, yet we have zero trust verification at the retrieval step.
The numbers here are damning: across 450 containerized runs on five tabular datasets, the attackers achieved a 49.56% full success rate in manipulating the agent's scientific conclusions. Even worse, default agent setups only detected this indirect poisoning 6.0% of the time. Whether the attack vector was altering metadata READMEs or subtly injecting biased data points to invert correlations, these agents proved shockingly gullible. They ingest, analyze, and spit out fabricated conclusions without batting an eye.
This vulnerability highlights a fundamental architectural flaw in how we build AI-driven retrieval systems. We must start treating retrieved datasets as untrusted payloads—validating schemas, origins, and statistical integrity before ingestion. If we do not, we are going to industrialize scientific fraud at a scale we are completely unprepared to handle.