Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

On-Policy and Off-Policy Learning for Large Action Spaces

Source

arXiv

arxiv.org/abs/2607.28408v1

Read the source

Source

Plain-Language Brief

The paper presents research on policy learning in interactive systems with large action spaces. The study focuses on two paradigms: on-policy and off-policy learning, both of which face challenges like inefficient exploration, sparse data coverage, etc. The paper proposes methods for structured Bayesian on-policy learning (meTS and dTS), addresses off-policy learning with sDM, and introduces differentiable pessimistic methods to control bias-variance trade-offs.

What the Document Says

  • The study focuses on policy learning in interactive systems where an agent selects actions from a very large set.
  • The research includes two paradigms: on-policy learning and off-policy learning.
  • On-policy learning involves the agent interacting sequentially with the environment to minimize regret.
  • Off-policy learning involves learning from logged data collected by a logging policy.
  • In large action spaces, both settings face challenges such as inefficient exploration, sparse data coverage, high-variance importance weights, extrapolation bias, and difficult optimization landscapes.
  • The paper develops structured Bayesian methods for on-policy learning (meTS, dTS).
  • The methods share information across actions and yield regret guarantees depending on an effective number of actions.
  • For off-policy learning, the paper proposes sDM based on latent variables.
  • The study also shows that optimization error can dominate estimation error in large action spaces.
  • Concave, efficiently optimizable policy-weighted log-likelihood objectives are introduced for off-policy learning.
  • Differentiable pessimistic methods based on exponential smoothing and PAC-Bayesian bounds are developed to control the bias-variance trade-off of regularized importance-sampling estimators.

Key Concepts Mentioned

  • Policy Learning
  • Interactive Systems
  • Contextual Bandits
  • On-Policy Learning
  • Off-Policy Learning
  • Large Action Spaces
  • Thompson Sampling
  • Diffusion-inspired Priors
  • Latent Variables
  • Regret
  • Effective Number of Actions
  • Optimization Error
  • Estimation Error
  • Bias-Variance Trade-off
  • Exponential Smoothing
  • PAC-Bayesian Bounds

Claims Made by the Document

  • On-policy and off-policy learning face challenges in large action spaces.
  • The proposed methods for on-policy learning (meTS, dTS) share information across actions and yield regret guarantees depending on an effective number of actions.

Evidence Used

  • Text describing the main framework being contextual bandits with two paradigms: on-policy and off-policy learning.
  • Description of challenges faced in large action spaces for both settings.
  • Explanation of how the proposed methods for on-policy learning (meTS, dTS) work and their benefits.

Methods / Approach

  • mixed-effect extension of Thompson sampling (meTS)
  • diffusion-inspired priors (dTS)
  • structured direct method based on latent variables (sDM)
  • exponential smoothing
  • PAC-Bayesian bounds

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

This thesis studies policy learning in interactive systems where an agent observes a context, selects an action from a very large set, and receives partial feedback. The main framework is contextual bandits, with two paradigms: on-policy learning, where the agent interacts sequentially with the environment and minimizes regret, and off-policy learning, where it learns from logged data collected by a logging policy. In large action spaces, both settings face major challenges: inefficient exploration, sparse data coverage, high-variance importance weights, extrapolation bias, and difficult optimization landscapes. The first part develops structured Bayesian methods for on-policy learning. We introduce meTS, a mixed-effect extension of Thompson sampling, and dTS, which leverages diffusion-inspired priors to model dependencies between actions. These methods share information across actions and yield regret guarantees depending on an effective number of actions. The second part addresses off-policy learning. We propose sDM, a structured direct method based on latent variables, show that optimization error can dominate estimation error in large action spaces, and introduce concave, efficiently optimizable policy-weighted log-likelihood objectives. Finally, we develop differentiable pessimistic methods based on exponential smoothing and PAC-Bayesian bounds to control the bias-variance trade-off of regularized importance-sampling estimators.