Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Active Inference as a Convex Markov Decision Process

Source

arXiv

arxiv.org/abs/2607.20152v1

Read the source

Source

Plain-Language Brief

The paper presents Active Inference (AIF) as a method that minimizes the expected free energy (EFE), combining both epistemic and pragmatic objectives within a single variational principle. The authors frame AIF as policy optimization and show that EFE minimization can be formulated as a convex Markov decision process (MDP). This perspective reveals the epistemic drive of active inference as a policy-dependent reward. The paper derives a mirror descent (MD) algorithm for finite-horizon, discounted, and average-reward formulations of EFE, making it compatible with actor-critic methods and dynamic programming.

What the Document Says

  • Active Inference (AIF) is presented as a method that minimizes the expected free energy (EFE).
  • AIF is framed as policy optimization and can be formulated as a convex MDP.
  • The pragmatic terms in AIF are linear in the predictive state marginals, equivalent to reward maximization in a latent MDP.
  • The epistemic value introduces a nonlinear component that distinguishes EFE minimization from standard reinforcement learning.
  • The epistemic drive of active inference is a policy-dependent (performative) reward.
  • Finite-horizon, discounted, and average-reward formulations of EFE are analyzed and a mirror descent (MD) algorithm is derived.
  • The MD algorithm locally linearizes the objective around the current state marginals.
  • The policy-dependent reward is compatible with actor-critic methods and dynamic programming.

Key Concepts Mentioned

  • Active Inference
  • Expected Free Energy
  • Policy Optimization
  • Markov Decision Process
  • Mirror Descent Algorithm

Claims Made by the Document

  • AIF can be formulated as a convex MDP.
  • The epistemic drive of active inference is a policy-dependent reward.

Evidence Used

  • The paper provides a mathematical framework for AIF and derives the MD algorithm.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). In this formulation, the pragmatic terms are linear in the predictive state marginals and therefore equivalent to reward maximization in a latent MDP, while the epistemic value introduces a nonlinear component that distinguishes EFE minimization from standard reinforcement learning. This perspective further reveals the epistemic drive of active inference as a policy-dependent (performative) reward. We analyze finite-horizon, discounted, and average-reward formulations of EFE and derive a mirror descent (MD) algorithm that locally linearizes the objective around the current state marginals, yielding a policy-dependent reward that is compatible with actor-critic methods and dynamic programming. Finally, we argue that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning, providing a route toward grounding active inference within modern reinforcement learning and optimization theory, including convergence analysis and principled policy improvement guarantees.