Cross-Layer Misalignment Detection in Agent Skills: A Progressive Loading-Aware Contrastive Learning Approach

Image generated by AI

TLDR

What: PL-HCL (Progressive Loading-Aware Hierarchical Contrastive Learning) is a pre-execution security framework that detects "cross-layer misalignment"—discrepancies between user-facing metadata claims and actual underlying instructions/scripts—in LLM Agent Skills.
Who's at risk: Agentic AI platforms and software ecosystems that dynamically load third-party extensions, tools, or skills from open-source registries.
Key number: PL-HCL boosts the misaligned-class F1-score ( $F1_m$ ) from a baseline of $0.143$ to $0.817$ on a cybersecurity-specialized Foundation-Sec-8B backbone, correcting the heavy "majority-class bias" of standard LLMs.

As Large Language Model (LLM) agents transition from simple chatbots to autonomous systems capable of dynamic tool invocation, the security boundary has shifted to the supply chain of "Agent Skills"—reusable packages containing metadata, system prompts, and executable resources. Popular marketplaces like SkillsMP [27] and open agentic environments like AgentDojo [6] allow LLM agents to download and load these skills on the fly based solely on high-level metadata descriptions. However, this progressive loading design introduces a critical attack vector: malicious actors can publish skills with benign metadata descriptions that mask malicious behaviors (e.g., prompt injection, credential harvesting, or unauthorized command execution) hidden deep within the underlying instruction or resource layers.

Threat Model


Attacker	A third-party developer or an adversary who compromises an open-source Agent Skill repository to publish a package with mismatched layers.
Victim	LLM agent frameworks, dynamic systems, and developers executing third-party tools based on high-level marketplace descriptions without manual code auditing.
Goal	Hide malicious capabilities (e.g., credential theft, remote code execution) behind safe metadata (malicious misalignment), or exaggerate capabilities to trick the agent into routing sensitive tasks to a non-functional tool (benign misalignment).
Budget	Low; requires zero-cost access to public registries (e.g., SkillsMP) to publish synthetically manipulated or maliciously structured Agent Skill packages.

Background & Problem Setup

To evaluate the security of these agentic components, researchers historically relied on runtime security analyses or post-execution benchmarks. However, runtime evaluation is computationally expensive and unsafe to perform directly on production infrastructure.

Unlike traditional code analyzers or generalized security scanners, Agent Skills are structured, multi-layered digital artifacts. The paper formalizes each skill as a triplet:

S = (M, I, R)

Where:

$M$ is the Metadata Layer (surface-level claims, names, tags, and summaries).
$I$ is the Instruction Layer (agent-facing system prompts and procedural guidance).
$R$ is the Resource Layer (executable scripts, assets, dependencies, and configuration files).

How PL-HCL Differs from Existing Work

Approach / Work	Evaluation Phase	Artifact Modality	Primary Security Focus	Key Limitation
SWE-Skills-Bench [9] / GAIA [21]	Post-execution (Runtime)	Execution traces, output correctness	Task completion and tool-use proficiency	Assumes skills are already trusted and loaded; cannot assess safety prior to execution.
Static Vulnerability Scanning	Pre-execution (Static)	Source code, scripts	Known malicious patterns and vulnerabilities	Fails to detect functional/semantic misalignment or benign metadata exaggerations.
PL-HCL (This Work)	Pre-execution (Static)	Multi-layer: $M$ , $I$ , and $R$	Cross-layer structural and semantic consistency	Dependent on observable static evidence; cannot identify dynamic environment-specific exploits lacking static artifacts.

Methodology

PL-HCL addresses cross-layer misalignment through a two-stage process: Progressive Continued Pretraining (CPT) to adapt a decoder-only LLM to the unique structure of Agent Skills, followed by Hierarchical Contrastive Learning to explicitly train the model to verify semantic alignment across the package's layers.

       [Step 1: Progressive CPT]
       Stage 1: Train on Metadata + Instruction (Short view: <= 4,096 tokens)
                          │
                          ▼
       Stage 2: Train on Metadata + Instruction + Resources (Full view: <= 10,240 tokens)
                          │
                          ▼
       [Step 2: Hierarchical Contrastive Learning (PL-HCL)]
       Contrast Aligned S = (M, I, R) against Swap & Corruption Negatives

1. Progressive Continued Pretraining (CPT)

To familiarize the backbone LLM with the syntax and format of multi-layered skill packages, the authors construct two token-budgeted views of the unlabeled corpus (248,473 packages from SkillsMP):

Short View (Stage 1): Concat ( $M, I$ ) bounded at $4,096$ tokens.
Full View (Stage 2): Concat ( $M, I, R$ ) bounded at $10,240$ tokens.

Both CPT stages optimize the standard causal language modeling loss:

\mathcal{L}_{\text{CPT}} = -\frac{1}{T} \sum_{t=1}^T \log p_\theta(x_t \mid x_{<t})

2. Hierarchical Contrastive Learning (PL-HCL)

Following CPT, the model is trained to map each layer of a skill into a shared embedding space. For a package $S_i = (M_i, I_i, R_i)$ , the model uses a projection head $g_\phi$ on the hidden representations $h_\theta(X)$ for $X \in \{M, I, R\}$ to output normalized projected embeddings:

z_X = \frac{g_\phi(h_\theta(X))}{\|g_\phi(h_\theta(X))\|_2}

The overall cross-layer consistency score $a_{\theta, \phi}(S_i)$ is calculated as the weighted sum of cosine similarities between the layers:

a_{\theta, \phi}(S_i) = \alpha_{\text{MI}} \text{sim}(z_M, z_I) + \alpha_{\text{MR}} \text{sim}(z_M, z_R) + \alpha_{\text{IR}} \text{sim}(z_I, z_R)

To teach the model what misalignment looks like, the authors construct two negative views for every positive pair $P_i = (M_i, I_i, R_i)$ :

Swap Negatives ( $N_i^A$ ): Replacing $M_i$ with metadata $M_j$ from an entirely different skill $j \neq i$ .
Corruption Negatives ( $N_i^B$ ): Perturbing/corrupting $M_i$ using an LLM generator to create exaggerated, false, or hallucinatory metadata claims ( $\widetilde{M}_i$ ).

The PL-HCL loss leverages InfoNCE to pull positive/aligned layers closer together while pushing swapped or corrupted combinations apart:

\mathcal{L}_{\text{PL-HCL}} = -\sum_{i} \log \frac{\exp(a_{\theta, \phi}(P_i) / \tau)}{\exp(a_{\theta,\phi}(P_i)/\tau) + \exp(a_{\theta,\phi}(N_i^A)/\tau) + I_i^B \exp(a_{\theta,\phi}(N_i^B)/\tau)}

PL-HCL Contrastive Learning Pseudocode

# Conceptual PyTorch-style pipeline for PL-HCL training step
import torch
import torch.nn.functional as F

def pl_hcl_loss(z_M, z_I, z_R, z_M_neg_swap, z_M_neg_corr, alpha=(1/3, 1/3, 1/3), temp=0.07):
    # Compute cross-layer scores for positive pairs: a(P_i)
    sim_MI = F.cosine_similarity(z_M, z_I, dim=-1)
    sim_MR = F.cosine_similarity(z_M, z_R, dim=-1)
    sim_IR = F.cosine_similarity(z_I, z_R, dim=-1)
    score_pos = alpha[0]*sim_MI + alpha[1]*sim_MR + alpha[2]*sim_IR

    # Compute scores for metadata-swap negatives: a(N_i^A)
    sim_MI_swap = F.cosine_similarity(z_M_neg_swap, z_I, dim=-1)
    sim_MR_swap = F.cosine_similarity(z_M_neg_swap, z_R, dim=-1)
    score_neg_swap = alpha[0]*sim_MI_swap + alpha[1]*sim_MR_swap + alpha[2]*sim_IR

    # Compute scores for metadata-corruption negatives: a(N_i^B)
    sim_MI_corr = F.cosine_similarity(z_M_neg_corr, z_I, dim=-1)
    sim_MR_corr = F.cosine_similarity(z_M_neg_corr, z_R, dim=-1)
    score_neg_corr = alpha[0]*sim_MI_corr + alpha[1]*sim_MR_corr + alpha[2]*sim_IR

    # InfoNCE loss formulation
    pos_exp = torch.exp(score_pos / temp)
    neg_swap_exp = torch.exp(score_neg_swap / temp)
    neg_corr_exp = torch.exp(score_neg_corr / temp)

    loss = -torch.log(pos_exp / (pos_exp + neg_swap_exp + neg_corr_exp))
    return loss.mean()

Key Results

The authors evaluate PL-HCL on a human-verified Challenge Set of 1,444 real-world Agent Skills containing 1,150 aligned and 294 naturally occurring misaligned packages (196 SAFE, 38 SUSPICIOUS, 60 MALICIOUS).

Performance on the Challenge Set (Optimal Prompt Configurations)

Backbone Model	Adapter	Prompting ( $k$ -shot)	Overall Accuracy	Misaligned-Class F1 ( $F1_m$ )	Balanced Macro-F1	AUC	$AP_m$
Llama-3.1-8B	Base (None)	2-shot	0.812	0.145	0.519	0.510	0.301
Llama-3.1-8B	CPT Only	2-shot	0.812	0.145	0.520	—	—
Llama-3.1-8B	CPT + PL-HCL	2-shot	0.929	0.788	0.872	0.744	0.754
Foundation-Sec-8B	Base (None)	2-shot	0.809	0.143	0.518	0.487	0.362
Foundation-Sec-8B	CPT Only	2-shot	0.811	0.150	0.522	—	—
Foundation-Sec-8B	CPT + PL-HCL	2-shot	0.937	0.817	0.889	0.778	0.799
RedSage-Qwen3-8B-DPO	Base (None)	5-shot	0.811	0.144	0.519	0.703	0.637
WhiteRabbitNeo-2-8B	Base (None)	5-shot	0.800	0.114	0.499	0.535	0.360

Critical Analytical Insights:

The Fallacy of High Accuracy: Base models like Llama-3.1-8B and specialized security models like RedSage-Qwen3-8B-DPO seem to achieve "good" overall accuracy (~ $0.80$ to $0.81$ ). However, this is entirely a function of class imbalance in the Challenge Set (which is ~80% aligned/safe). Their actual ability to identify misaligned packages is practically non-existent, as reflected by the extremely low $F1_m$ scores ( $0.114$ to $0.150$ ). They suffer from a heavy "majority-class bias," classifying almost everything as "aligned."
Format Adaptation Is Not Alignment Detection: Running CPT (Continued Pretraining) alone barely moves the needle. As shown in Table 2, under 2-shot prompting, Llama-3.1-8B with CPT achieves $0.145$ $F1_m$ —exactly the same as the base model. CPT merely helps the model process the structured text format but does not teach it to cross-evaluate claims against implementations.
The Power of Contrastive Training: Adding the PL-HCL contrastive training step forces the model to learn a discriminative "claim-vs-evidence" signal. On Foundation-Sec-8B, PL-HCL drives $F1_m$ up to $0.817$ , elevating the Macro-F1 score from $0.518$ to $0.889$ .

Limitations & Open Questions

While PL-HCL shows significant performance gains, several limitations should prevent practitioners from relying on it blindly:

Scalability of Evaluation: The evaluation is strictly restricted to 8B-scale model backbones. It is unclear if larger closed-source models possess enough zero-shot semantic cross-referencing capabilities to bypass the need for PL-HCL style fine-tuning.
Dynamic Exploits Bypass: PL-HCL is purely a static, pre-execution screening tool. It cannot identify zero-day, dynamic execution payloads that do not leave clean static traces in the instruction or resource files.
Reliance on Synthetic Negatives: The contrastive framework is trained primarily on synthetic negative views (swaps and prompt-based corruptions). Advanced red-teamers could design highly subtle, semantically complex misalignments that bypass these specific perturbation distributions.

What Practitioners Should Do

If you are deploying LLM agents in production that leverage third-party toolboxes or dynamic skill integrations, implement these defenses:

Implement Pre-Execution Embedding Audits: Extract metadata, instruction markdown, and code dependencies from incoming skill packages. Use a PL-HCL-style trained encoder to compute the cross-layer similarity score. Flag any package where the score falls below a threshold calibrated using a held-out validation set.

Strict Schema Isolation: Do not allow tool package formats to be amorphous. Force a structured schema separating metadata declarations, developer descriptions, and agent instruction guidelines. Use strict schemas to simplify multi-layered parsing:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "AgentSkillPackage",
  "type": "object",
  "properties": {
    "metadata": { "type": "object" },
    "instructions": { "type": "string" },
    "resources": { "type": "array", "items": { "type": "string" } }
  },
  "required": ["metadata", "instructions", "resources"]
}

Deploy Triage Pipelines: Since full runtime sandboxing is computationally intensive, use PL-HCL as a high-throughput, static first-pass triage filter. Route high-scoring aligned skills directly to execution, while routing flagged/misaligned skills to secure sandboxes (such as the Jetstream Docker sandbox environment used in this study) for dynamic verification.

The Takeaway

Trust in autonomous agentic ecosystems cannot be established retroactively after code execution has already commenced. By treating Agent Skills as layered, structured artifacts and forcing LLMs to explicitly resolve semantic conflicts between claims and source code, PL-HCL demonstrates that proactive, pre-execution structural verification is both achievable and critical for building safe AI systems.

Den's Take

The shift toward dynamic, third-party "Agent Skills" represents a massive supply-chain risk that traditional static code-scanning tools are entirely blind to. I like PL-HCL because it tackles this at the semantic level before any code actually runs. The typical failure mode of standard LLMs in this domain is "majority-class bias"—simply assuming a tool is safe if the wrapper looks benign. Seeing PL-HCL boost the misaligned-class F1-score from a miserable $0.143$ baseline to $0.817$ on the Foundation-Sec-8B backbone proves that hierarchical contrastive learning is a highly viable way to catch discrepancies across the Metadata (M), Instruction (I), and Resource (R) layers.

This focus on early-stage validation directly aligns with the broader security design principle that dynamic agent environments must gate risky actions prior to execution rather than relying solely on reactive, post-execution runtime monitoring.

However, we must be realistic about the limitations. While PL-HCL is great at catching clear structural mismatches—like a calculator skill that secretly requests shell executions—sophisticated adversaries will eventually learn to craft adversarial system prompts and highly obfuscated source code that bypass contrastive embedding checks. To be truly robust in production environments like AgentDojo, this static gating must be paired with strict sandboxing. Still, as a pre-execution filter, this is a major step forward.