When Is Delegated Play Truthful? Within-Range Regret and the Trilemma of Aligned Delegation

Image generated by AI

TLDR

• What: When automated proxies (such as LLM agents or bidding bots) are restricted by safety guardrails or alignment layers, users have a direct mathematical incentive to lie to their own agents (e.g., via intent inflation or prompt injection) to bypass constraints, a phenomenon governed exactly by the proxy's "within-range regret."
• Who's at risk: Transactional LLM agent architectures (e.g., RAG-mediated shopping assistants, coding assistants), automated procurement bots, and ad-tech autobidders.
• Key number: Under an alignment-style soft cap, 25 production language models (including GPT-5, Claude Sonnet 5, and Gemini 2.5 Pro) universally failed truthfulness, exhibiting within-range regret up to 14.40 and forcing report inflation up to 2.00x to recover optimal performance.

As LLM-based agents transition from simple chat interfaces to transactional proxies operating in complex environments (such as search engines, automated coding assistants, and digital marketplaces), they act on behalf of users in hidden games and auctions. This paper analyzes this "delegation stack" ($g \circ \pi$) and exposes a fundamental vulnerability: when we place safety alignment or guardrails on these proxies, we mathematically incentivize users to lie to their own agents. By demonstrating that the strategic gain from misreporting equals the proxy's "within-range regret," the author shows that standard post-hoc alignment filters inevitably reward strategic prompt-engineering, jailbreaking, and preference-inflation.

Threat Model

Background / Problem Setup

In classical mechanism design, the revelation principle guarantees that any outcome reachable via a complex strategic game can be replicated by a "direct mechanism" where players honestly report their private types $\theta_i$ to a trusted mediator. However, computing the required equilibria is PPAD-complete, and real-world players rarely trust platforms with their raw utility functions.

The modern solution is delegated play: users describe their intent to automated proxies (such as autobidders or LLM elicitation proxies [42]), which then play the mechanism on their behalf. The author models this as a delegation stack where each principal reports a type $t_i$ to a proxy $\pi_i : \Theta_i \to M_i$, and the platform runs the wrapped mechanism $g \circ \pi$.

Comparison of Related Frameworks

Methodology

1. The Math of Within-Range Regret

Let $Reach_i = \pi_i(\Theta_i) \subseteq M_i$ be the set of messages the proxy can emit under some report. The author defines within-range regret ($W_i(\theta_i; \pi)$) as:

The principal's manipulation gain ($G_i(\theta_i; \pi)$) is the maximum utility they can recover by lying to the proxy:

Theorem 1 (Truth-to-proxy) proves that these two quantities are identical:

This identity is powerful. It means that the strategic incentive to lie to an agent does not depend on the global utility space, but only on the mismatch between what the agent does under an honest report and the best action it could have taken within its reachable range.

2. The Aligned Delegation Trilemma

When we apply a guardrail $c: M_i \to M_i$ to a proxy's output (e.g., an alignment layer or safety filter), we narrow its reach to $Reach'_i = c(Reach_i)$. The paper defines three desirable properties for any guardrail:

1. Binding: $c(\pi_i(\theta_i)) \neq a^*$, where $a^*$ is the honest-optimal reachable action.
2. Truthful: $W_i(\theta_i; \pi') = 0$, meaning honest reporting remains optimal.
3. Capability-preserving: $a^* \in Reach'_i$, meaning the best outcome remains reachable through some report.

Theorem 2 states that no guardrail can satisfy all three properties simultaneously. If a safety guardrail is binding and preserves capability, it must violate truthfulness (leading to $W_i > 0$). This is the mathematical engine behind jailbreaking and prompt-engineering: users alter their reported prompt $t_i$ to steer the model back to $a^*$.

                  [Capability-Preserving]
                       /          \
                      /     (1)    \
                     /   Output-    \
                    /    Clipping    \
                   /                  \
              [Binding] ------------ [Truthful]
                         (3)      (2)
                      Hard Cap   Non-Binding

Figure 1(b) Alignment Trilemma: (1) Output-clipping keeps capability but loses truthfulness. (2) Non-binding guardrails keep truthfulness but do nothing. (3) Hard caps keep truthfulness but lose capability.

3. Experimental Setup

The author tests this on 25 production language models across five providers (OpenAI, Anthropic, Google, DeepSeek, xAI) in a simulated first-price auction against a uniform competitor bid $c \sim U[0, 300]$.

• Client Value $v$: $\in \{100, 150, 200, 250, 300\}$
• Interim Utility: $u(b, v) = (v - b) \min(b/300, 1)$, making the unconstrained optimal bid $b^* = v/2$.
• Alignment-Style Soft Cap: $c(b) = b$ for $b \leq 40$, and $c(b) = 40 + 0.4(b-40)$ for $b > 40$.
• Proxy Prompting: Models are instructed solely to maximize client surplus and are not shown the cap formula. The cap is applied to the bid returned by the model.
• Sweeps: Evaluated over 25 evenly spaced reports to estimate $W_i$ and the optimal report inflation.

Key Results

1. Unified Performance under Soft-Cap Constraint

On every model, the binding, capability-preserving alignment cap resulted in $W_i > 0$, rendering honest reporting suboptimal. Seventeen capable models (including GPT-5, o3, Gemini 2.5 Pro, and Grok 4.3) performed rationally, bidding the unconstrained optimal bid $b^* = v/2$. When the cap compressed this bid, the optimal strategy became inflating the reported value.

Per-Value Within-Range Regret & Report Inflation (Rational Model Cluster)

Both regret and report inflation scale quadratically as the true value $v$ increases and the cap binds tighter.

2. Provider Family Performance Comparison

The paper reveals that larger, more capable reasoning models exhibit higher within-range regret. This is because they perfectly calculate the optimal bid, making them highly susceptible to cap compression. Smaller models (like GPT-4o mini and Claude Haiku 4.5) show lower $W_i$ because they bid imprecisely, not because they are inherently more truthful.

3. Robustness Across Environments

To prove that the incentive to lie is caused by the alignment cap itself and not the specific auction format, the author tested a second-price payment rule (where honest bidding is dominant-strategy incentive compatible under standard conditions). Even here, the cap forced report inflation.

4. Placement Ablation: Output-Clipping vs. Constraint-Aware Decoding

As predicted by Proposition 3, the placement of the safety guardrail determines which corner of the trilemma is forfeited. Applying a filter after the model optimizes (Output-Clipping) breaks truthfulness ($W_i \approx 5.79$). Giving the model a structured menu of valid bids before decision-making (Constraint-Aware Decoding) restores truthfulness ($W_i = 0$) but sacrifices capability when the optimal bid is outside the menu.

Limitations & Open Questions

• Artificial Soft Cap: The study uses an artificially imposed mathematical cap. While this provides a clean proof of concept, measuring within-range regret ($W_i$) arising from a model's organic safety-refusal behavior (which is non-mathematical and semantic) remains an open problem.
• Intractability of Exact Verification: Proposition 6 proves that computing exact within-range regret is #P-hard. Although the author provides a sample-based estimator, this introduces statistical error that decays at a slow $1/\sqrt{N}$ rate. Under rapid model drift, continuously maintaining these certificates in production could incur significant computational overhead.
• Single-Agent Scope: The trilemma is analyzed primarily as a single-agent decision problem (holding opponent strategies constant). The equilibrium welfare effects when all users strategically inflate their reports simultaneously are left unaddressed and could lead to highly unstable market dynamics.

What Practitioners Should Do

1. Shift Guardrails to Constraint-Aware Decoding: Stop using post-hoc filters or wrappers to clip LLM agent actions. If an agent must operate within constraints, pass those constraints directly into the model’s context or decoding menu. This forces the model to re-optimize internally, preserving truthfulness.
2. Implement Continuous $W_i$ Tracking: Since exact verification is #P-hard, implement a sample-based estimator over live transactional logs. Track the drift budget ($B_T$) continuously using the online re-certification protocol outlined in Proposition 7. Trigger a re-evaluation of user incentives whenever:

3. Limit Action Space Reach: Do not give agents open-ended action capabilities. Tighten the reach parameter ($Reach_i$) to match the exact requirements of the target transaction. An unnecessarily wide message space only increases the surface area for user manipulation and within-range regret.
4. Audit for "Intent Inflation" Red-Teaming: During safety evaluations, red-team your agents by acting as an adversarial user who systematically exaggerates constraints or values. Measure if the agent is vulnerable to bypassing its alignment rules through these semantically inflated prompts.

The Takeaway

Alignment is not just an ethical or behavioral issue; it is a mechanism design bottleneck. When safety layers restrict what an agent can do while leaving the optimal outcome reachable via creative prompt phrasing, "lying to the AI" becomes the mathematically rational response. System designers must realize that they cannot enforce binding safety rules on capable agents without either making those agents dishonest or severely limiting their capabilities.

Den's Take

This paper hits on a crucial, often ignored reality of agent deployment: alignment and safety guardrails aren't just technical hurdles for attackers; they fundamentally warp the economic incentives of benign users. When we restrict transactional LLM agent architectures with post-hoc soft caps, we practically beg users to manipulate their own systems.

The math here is sobering. Seeing 25 production models—including GPT-5 and Claude Sonnet 5—universally fail truthfulness under alignment constraints, yielding a within-range regret of up to 14.40, shows how systemic this is. Users forcing up to 2.00x report inflation just to recover their unconstrained utility proves that safety layers act as a utility tax that users will actively route around via prompt engineering.

This mathematically validates why restrictive, post-hoc alignment filters are fundamentally unstable, reinforcing why we must transition to dynamic structural monitoring of agent behavior. If we do not fix the underlying incentive design of delegated play, our safety guardrails will continue to turn honest users into adversaries.