Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities

Image generated by AI

TLDR

• What: An empirical evaluation of 11 open-weight LLMs (4B to 120B) on converting raw Connected and Autonomous Vehicle (CAV) vulnerabilities into structured STIX 2.1 threat intelligence graphs, mapping to CWE and MITRE ATT&CK.
• Who's at risk: Automated automotive threat intelligence pipelines, CAV security operation centers (SOCs), and RAG systems that ingest unstructured CVE descriptions without graph-level validation.
• Key number: While LLMs achieve up to a 0.94 F1 score on extracting individual threat entities (SDOs), relationship extraction (SROs) tops out at 0.63 F1, and complete MITRE ATT&CK mapping suffers from a low Match@All score of 0.33.

As connected and autonomous vehicles (CAVs) face complex, remote exploits—such as STARLINK flaws exposing vehicle controls, the 2024 Kia remote takeover, and Subaru telematics bugs—converting raw vulnerability reports into actionable threat intelligence is critical. Security Operations Centers (SOCs) and automotive ISACs are exploring LLM-based parsing pipelines to automatically translate plain-text Common Vulnerabilities and Exposures (CVE) entries into Structured Threat Information Expression (STIX 2.1) graphs. This paper benchmarks how well 11 state-of-the-art open-weight LLMs (ranging from 4B to 120B parameters) actually perform when tasked with generating structured SDOs, SROs, CWEs, and MITRE ATT&CK profiles for the automotive domain.

Background / Problem Setup

Parsing cyber threat intelligence (CTI) traditionally relied on rigid, rule-based regular expressions or heavy manual curation. The emergence of LLMs allows for semantic understanding of unstructured exploit descriptions, but generative models are prone to structural invalidity, missing relationship connections, and hallucinations.

Unlike general-domain CTI parsers, autonomous vehicle security involves highly specialized hardware/software interactions (e.g., Controller Area Networks (CAN), Electronic Control Units (ECUs), telematics, and sensors). The authors introduce CAV-STIXGen, a curated dataset of 183 CAV-related CVEs manually annotated into ground-truth STIX 2.1 JSON structures.

Below is how this evaluation compares to prior methodologies in the literature:

Methodology

The authors constructed and executed the evaluation pipeline through the following stages:

1. Dataset Construction (CAV-STIXGen)

• Keyword construction: Generated CAV-related keywords (such as Connected Vehicle, EV, CAN, V2V, and OTA updates) using LLMs, filtering out duplicates to yield the keyword lexicon shown in Table I.
• CVE Collection & Filtering: Searched the National Vulnerability Database (NVD) via REST APIs, retrieving 445 candidate records. After manual relevance filtering (excluding generic website/database flaws), 183 unique records were preserved (achieving a Cohen's kappa of 94%).
• Manual Annotation: Annotators manually extracted STIX Domain Objects (SDOs) and STIX Relationship Objects (SROs). They mapped CVEs to CWEs and MITRE ATT&CK techniques, producing 1,383 SDO instances, 1,395 SRO instances, 211 CWE mappings, and 294 MITRE ATT&CK mappings.

2. Prompting Strategies

The evaluation compared three specific prompting strategies:

• Contextless (C): Only provides the CVE ID and description. No instructions on structural schemas are provided.
• STIX-Guided (SG): Adds task guidance defining valid SDO types, SRO relationship semantics, CWE, and MITRE ATT&CK mappings.
• Dynamic Few-Shot (DFS): Appends the 5 most semantically similar ground-truth STIX examples, retrieved using similarity scores calculated over the CVE description, CWE, and attack-pattern.

# Conceptual Prompt Implementations

# 1. Contextless (C) Prompt Structure
Input: [CVE-ID] + [Vulnerability Description]
Instruction: "Generate a valid STIX 2.1 JSON bundle for this vulnerability."

# 2. STIX-Guided (SG) Prompt Structure
Input: [CVE-ID] + [Vulnerability Description]
Instruction: "Generate a valid STIX 2.1 JSON bundle. Use only the following SDO types: [infrastructure, attack-pattern, vulnerability, identity, threat-actor...]. Map SRO relationships like [exploits, targets, uses, affects] only between valid source-target pairs."

# 3. Dynamic Few-Shot (DFS) Prompt Structure
Input: [CVE-ID] + [Vulnerability Description]
Instruction: "Generate a valid STIX 2.1 JSON bundle. Here are 5 similar examples of CVE descriptions and their ground-truth STIX JSON outputs..."
Examples: [Example 1 STIX JSON] ... [Example 5 STIX JSON]

Key Results

The authors evaluated 11 open-weight models across various temperatures ($T = 0.0, 0.25, 0.75, 1.0$). The table below summarizes the performance of the top-performing models under the best prompting configuration: Dynamic Few-Shot (DFS).

Performance of Top Models under Dynamic Few-Shot (DFS)

Multi-Agent SDO and SRO Extraction Performance

The authors also tested a multi-agent setup separating extraction, mapping, and relationship construction roles:

Key Takeaway on Multi-Agent Performance: Despite the popular industry trend of multi-agent patterns, the multi-agent setup underperformed compared to the single-model Dynamic Few-Shot configuration. Gemma-4-31B’s SRO F1 dropped to 0.39 (down from 0.61 under single-model DFS), and Codestral-22B dropped to 0.43 (down from 0.55 under single-model DFS). Role-specific decomposition broke down when handling complex graph-level connections.

Limitations & Open Questions

• SRO Graph Disconnection: SRO extraction F1 scores are low across the board, peaking at 0.63 for Qwen-Coder and averaging closer to 0.50. While the models can accurately identify the entities (SDOs), they struggle to connect them. A parsed threat intelligence graph with missing or incorrect edges is highly misleading for automated defensive response.
• The MITRE ATT&CK Bottleneck: Match@All scores are abysmal, peaking at just 0.33 (33%) for Gemma-4-31B under DFS. Because a single CVE can involve multiple, overlapping attack techniques, models are incapable of identifying the full attack scope, creating substantial visibility gaps.
• Multi-Agent Regression: The evaluation demonstrates that separating tasks across agents degrades relationship mapping quality. This indicates that LLMs struggle with contextual hand-offs when generating relational schemas.
• CAV-STIXGen Dataset Scale: With only 183 validated CVEs, the benchmark is clean but highly constrained. The actual diversity of firmware bugs, physical bus exploits, and RF vulnerabilities across different vehicle manufacturers may cause worse out-of-distribution performance in production.

What Practitioners Should Do

1. Implement Dynamic Few-Shot Retrieval, Not Zero-Shot Prompts

Do not use zero-shot or general instruction prompts for threat-intel ingestion pipelines. Set up a semantic search or vector database index populated with manually verified, high-quality CVE-to-STIX mappings. When a new CVE is ingested, dynamically inject the 5 most similar STIX exemplars into the system context.

2. Enforce Deterministic Schema and Graph Validation

Since LLM relationship extraction (SRO) is fragile, implement deterministic validation layers downstream of the LLM generation. Use STIX validation tools to check schema compliance. Write explicit rule checks to verify that relationships (e.g., exploits, targets) only connect semantically logical SDO types (e.g., verifying that a threat-actor SDO can only have an exploits relationship pointing to a vulnerability SDO).

3. Treat LLM MITRE ATT&CK Tags as Hypotheses, Not Ground Truth

With a Match@All score of only 33% on ATT&CK mapping, do not automate firewall rules or defensive configurations solely based on LLM-inferred ATT&CK techniques. Implement a validation stage that takes the LLM's suggested ATT&CK IDs, cross-references them against known keyword matrices, and alerts human analysts if critical techniques (such as T1499 Endpoint Denial of Service) are missed.

4. Optimize Single-Model Pipelines Over Multi-Agent Orchestrations

Save API costs and reduce latency by avoiding multi-agent swarms for STIX generation. Use a single, high-capacity code or reasoning model (such as Qwen-Coder-30B or LLaMA-3.3-70B) backed by dynamic few-shot prompting, which consistently outperforms multi-agent configurations in preserving relational graph integrity.

The Takeaway

Autonomous vehicle threat intelligence demands highly structured, graph-theoretic mapping, yet current open-weight LLMs are only reliable at extracting flat entities (SDOs). For complex, multi-layered threat modeling—specifically relationship mapping and complete MITRE ATT&CK mapping—purely generative approaches remain too fragile to deploy without robust, deterministic verification layers.

Den's Take

This paper exposes a glaring, dangerous gap in automated threat intelligence: LLMs are great at naming things, but terrible at understanding how they connect. While achieving a 0.94 F1 score on extracting individual threat entities (SDOs) from the 183 CAV-related CVEs sounds impressive, the performance falls off a cliff when constructing the actual threat graph. A 0.63 F1 score for relationship extraction (SROs) and a dismal 0.33 Match@All score for MITRE ATT&CK mapping mean that any security team trusting these models to autonomously build CAV-STIXGen dependency maps is flying blind.

As practitioners, we cannot feed raw, unvalidated LLM outputs directly into automated automotive SOCs or downstream RAG pipelines. If your relational graph is missing nearly 40% of its edges and two-thirds of its ATT&CK tactics, your automated response playbooks will misfire. I have written before about how AI Agents Do Not Fail Alone:The Context Fails First, which is highly relevant here because feeding corrupted or half-baked threat relationships into a downstream SOC agent guarantees a systemic security failure. Before we talk about autonomous defense in specialized domains like connected vehicles, we must solve the structural validation bottleneck; open-weight models simply aren't there yet.