On-Policy and Off-Policy Learning for Large Action Spaces

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Imad Aouali
• Venue: arXiv: cs.LG
• Published: 2026-07-30
• Collected: 2026-07-31
• URL: http://arxiv.org/abs/2607.28408v1
• PDF: https://arxiv.org/pdf/2607.28408v1
• DOI: UNKNOWN

Plain-Language Brief

The paper discusses policy learning in interactive systems with large action spaces, where agents observe contexts, select actions, and receive partial feedback. The research focuses on two paradigms: on-policy learning and off-policy learning, each addressing challenges such as inefficient exploration, sparse data coverage, etc. The paper proposes new methods for both settings, including meTS, dTS, sDM, and pessimistic methods, which are designed to improve the efficiency of learning and minimize regret.

What the Document Says

• The research focuses on policy learning in interactive systems where an agent interacts with the environment and faces challenges such as inefficient exploration, sparse data coverage, high-variance importance weights, extrapolation bias, and difficult optimization landscapes when dealing with large action spaces.
• The paper presents solutions for on-policy learning, including meTS and dTS, which share information across actions and offer regret guarantees depending on an effective number of actions.
• For off-policy learning, the paper introduces sDM, a structured direct method based on latent variables, and discusses the optimization error dominating estimation error in large action spaces.

Key Concepts Mentioned

• policy learning
• interactive systems
• large action spaces
• on-policy learning
• off-policy learning
• challenges in policy learning
• meTS
• dTS
• sDM

Claims Made by the Document

• The proposed meTS and dTS methods offer regret guarantees depending on an effective number of actions.
• Optimization error can dominate estimation error in large action spaces.

Evidence Used

• The text mentions that meTS and dTS are mixed-effect extensions of Thompson sampling and diffusion-inspired priors, respectively.
• The text states that sDM is a structured direct method based on latent variables.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

This thesis studies policy learning in interactive systems where an agent observes a context, selects an action from a very large set, and receives partial feedback. The main framework is contextual bandits, with two paradigms: on-policy learning, where the agent interacts sequentially with the environment and minimizes regret, and off-policy learning, where it learns from logged data collected by a logging policy. In large action spaces, both settings face major challenges: inefficient exploration, sparse data coverage, high-variance importance weights, extrapolation bias, and difficult optimization landscapes. The first part develops structured Bayesian methods for on-policy learning. We introduce meTS, a mixed-effect extension of Thompson sampling, and dTS, which leverages diffusion-inspired priors to model dependencies between actions. These methods share information across actions and yield regret guarantees depending on an effective number of actions. The second part addresses off-policy learning. We propose sDM, a structured direct method based on latent variables, show that optimization error can dominate estimation error in large action spaces, and introduce concave, efficiently optimizable policy-weighted log-likelihood objectives. Finally, we develop differentiable pessimistic methods based on exponential smoothing and PAC-Bayesian bounds to control the bias-variance trade-off of regularized importance-sampling estimators.