Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

On-Policy and Off-Policy Learning for Large Action Spaces

Source

arXiv

arxiv.org/abs/2607.28408v1

Read the source

Source

Plain-Language Brief

The paper by Imad Aouali discusses policy learning in interactive systems with large action spaces, where an agent makes decisions and receives partial feedback based on a context. The work focuses on two paradigms: on-policy learning and off-policy learning, each facing challenges such as inefficient exploration, sparse data coverage, and high-variance importance weights. To address these issues, the paper proposes various Bayesian methods for on-policy learning and introduces a structured direct method for off-policy learning.

What the Document Says

  • The study focuses on policy learning in interactive systems with large action spaces.
  • An agent observes a context, selects an action, and receives feedback based on the context.
  • There are two paradigms: on-policy learning and off-policy learning.
  • On-policy learning involves the agent interacting sequentially with the environment and minimizing regret.
  • Off-policy learning involves learning from logged data collected by a logging policy.
  • Challenges in large action spaces include inefficient exploration, sparse data coverage, high-variance importance weights, extrapolation bias, and difficult optimization landscapes.
  • The paper proposes structured Bayesian methods for on-policy learning: meTS (mixed-effect extension of Thompson sampling) and dTS (diffusion-inspired priors to model dependencies between actions).
  • These methods share information across actions and provide regret guarantees depending on an effective number of actions.
  • The paper proposes a structured direct method for off-policy learning called sDM, based on latent variables.
  • Optimization error is found to dominate estimation error in large action spaces in off-policy learning.
  • Concave, efficiently optimizable policy-weighted log-likelihood objectives are introduced for off-policy learning.

Key Concepts Mentioned

  • Policy Learning
  • Interactive Systems
  • Large Action Spaces
  • On-Policy Learning
  • Off-Policy Learning
  • Contextual Bandits
  • Thompson Sampling
  • Diffusion-inspired Priors
  • Structured Bayesian Methods
  • Regret
  • Effective Number of Actions
  • Latent Variables
  • Direct Method
  • Optimization Error
  • Estimation Error
  • Policy-Weighted Log-Likelihood Objectives

Claims Made by the Document

  • On-policy and off-policy learning face major challenges in large action spaces.
  • The proposed Bayesian methods for on-policy learning provide regret guarantees.

Evidence Used

  • The text discusses the challenges faced by on-policy and off-policy learning in large action spaces.
  • The paper introduces structured Bayesian methods for on-policy learning with regret guarantees.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

This thesis studies policy learning in interactive systems where an agent observes a context, selects an action from a very large set, and receives partial feedback. The main framework is contextual bandits, with two paradigms: on-policy learning, where the agent interacts sequentially with the environment and minimizes regret, and off-policy learning, where it learns from logged data collected by a logging policy. In large action spaces, both settings face major challenges: inefficient exploration, sparse data coverage, high-variance importance weights, extrapolation bias, and difficult optimization landscapes. The first part develops structured Bayesian methods for on-policy learning. We introduce meTS, a mixed-effect extension of Thompson sampling, and dTS, which leverages diffusion-inspired priors to model dependencies between actions. These methods share information across actions and yield regret guarantees depending on an effective number of actions. The second part addresses off-policy learning. We propose sDM, a structured direct method based on latent variables, show that optimization error can dominate estimation error in large action spaces, and introduce concave, efficiently optimizable policy-weighted log-likelihood objectives. Finally, we develop differentiable pessimistic methods based on exponential smoothing and PAC-Bayesian bounds to control the bias-variance trade-off of regularized importance-sampling estimators.