Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: David Eric Austin, Kaheer Suleman, Jackie Chi Kit Cheung
• Venue: arXiv: cs.CL
• Published: 2026-08-17
• Collected: 2026-08-18
• URL: http://arxiv.org/abs/2608.16707v1
• PDF: https://arxiv.org/pdf/2608.16707v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers introduce the concept of semantic bandits to study how pre-training affects exploration behavior of Large Language Models (LLMs) used as decision-making agents. They find that action labels with semantic information can either improve or degrade performance based on alignment with reward structure, and negative rewards trigger more exploration than positive rewards.

What the Document Says

• The text discusses the use of Large Language Models (LLMs) as decision-making agents in various environments.
• It highlights concerns about LLM's balance between exploration and exploitation.
• The authors introduce 'semantic bandits', an extension of the multi-armed bandit setting that considers textual action labels.
• They investigate how semantic priors, biases learned during pre-training from word co-occurrence, impact LLM exploration behavior.
• Findings show that informative action labels can either enhance or degrade performance depending on their alignment with the reward structure.
• Negative rewards are found to trigger more exploration than positive rewards, possibly due to a bias induced by common pre-training data conventions.
• The authors suggest that using language to define environments and rewards introduces biases in LLMs trained on word co-occurrence, which could impact their reliability and robustness.

Key Concepts Mentioned

• Large Language Models
• Multi-armed bandit
• Semantic Priors
• Exploration-Exploitation
• Word co-occurrence

Claims Made by the Document

• LLMs engage with tasks through natural language.
• Semantically informative action labels can either improve or degrade performance.
• Negative rewards trigger more exploration than positive rewards.

Evidence Used

• Existing work questions LLM's balance of exploration and exploitation.
• The authors define semantic bandits to study this behavior.
• Findings from their experiments support these claims.

Methods / Approach

• Introduction of semantic bandit setting
• Experiments to test the impact of semantic priors on LLM exploration behavior

Limitations Stated by the Authors

• The study focuses on Large Language Models, not other types of agents.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The agent is limited by the data it was pre-trained on.
• The agent's behavior may be influenced by biases present in its pre-training data.

Open Questions

• How can we minimize or mitigate these biases to improve the reliability and robustness of LLM agents?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The text provides evidence that semantic priors impact exploration behavior, but it does not specify the exact experiments conducted or the specific reward structures tested.

Raw Abstract / Excerpt

Large language models (LLMs) are increasingly deployed as decision-making agents in settings that require sophisticated environmental exploration. However, existing work has raised questions about how LLMs actually balance exploration and exploitation. Unlike classical agents, LLM agents engage with tasks through natural language, exposing them to semantic information with no formal counterpart in the task structure. We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions, and use it to study how semantic priors --- inductive biases arising from associations between language and expected reward learned during pre-training, shape LLM exploration behaviour. We find that semantically informative action labels reduce exploration in favour of exploitation, improving performance when aligned with the reward structure and severely degrading it when misaligned. We further find that negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions common in pre-training data. Overall, we argue that the use of language to define the environment and rewards introduces unavoidable biases derived from the fact that the model is trained on word co-occurence, with implications for the reliability and robustness of LLM agents in real-world decision-making settings.