Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Knowledge- and Gradient-Guided Reinforcement Learning for Parametrized Action Markov Decision Processes

Source

arXiv

arxiv.org/abs/2607.12924v1

Read the source

Source

Plain-Language Brief

The paper presents a new Reinforcement Learning algorithm named Neuro-Symbolic Knowledge- and Gradient-Guided Reinforcement Learning (KGRL). KGRL uses knowledge from a Datalog knowledge base to prune non-applicable actions and constrain parameter spaces in Parametrized Action Markov Decision Processes (PAMDP), making the training more sample-efficient. The algorithm outperforms existing RL baselines for PAMDPs in terms of both sample efficiency and episodic return.

What the Document Says

  • The paper focuses on Reinforcement Learning in Parametrized Action Markov Decision Processes (PAMDP), where each decision involves a symbolic action and numerical parameters.
  • In such settings, RL algorithms often determine parameters using one-shot estimators, making their training sample inefficient.
  • The paper proposes the KGRL algorithm that uses domain knowledge to prune non-applicable actions from the decision space and constrain parameter spaces of remaining actions.
  • KGRL utilizes a gradient-based parameter refinement loop to estimate optimal parameters during training and deployment of the agent.
  • The algorithm records activated rules along the trajectory, providing local procedural explanations on action pruning and parameter constraining.
  • KGRL guides the agent's exploration and decision-making towards feasible and constraint-aware ones, increasing sample efficiency during training.
  • KGRL outperforms existing RL baselines for PAMDPs in terms of both sample efficiency and episodic return.

Key Concepts Mentioned

  • Reinforcement Learning
  • Parametrized Action Markov Decision Processes (PAMDP)
  • Knowledge-Guided Reinforcement Learning
  • Datalog knowledge base
  • Gradient-based parameter refinement
  • Sample efficiency
  • Episodic return

Claims Made by the Document

  • KGRL increases sample efficiency during training
  • KGRL outperforms state-of-the-art RL baselines for PAMDPs in terms of both sample efficiency and episodic return

Evidence Used

  • The paper states that KGRL guides the agent's exploration towards feasible and constraint-aware decisions, increasing sample efficiency during training.
  • The paper presents results showing that KGRL outperforms existing RL baselines for PAMDPs.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

In this paper, we study Reinforcement Learning in Parametrized Action Markov Decision Processes (PAMDP), where each decision consists of a symbolic action and numerical parameters. In such settings Reinforcement Learning algorithms typically determine parameters with one-shot estimators, which makes their training sample inefficient. Though in most PAMDP environments explicit but incomplete knowledge (e.g., rules, safety constraints, or expert heuristics) is available, it is rarely directly used to increase the sample-efficiency of training Reinforcement Learning agents. We step into this gap and propose our novel Neuro-Symbolic Knowledge- and Gradient-Guided Reinforcement Learning (KGRL) algorithm. KGRL uses domain knowledge in a Datalog knowledge base to derive the set of applicable actions and feasible parameters for a given state. This allows it to prune non-applicable actions from the decision-space and constrain the parameter spaces of the remaining actions. We then use a gradient-based parameter refinement loop to estimate the optimal parameters during training and deployment of the agent. By recording activated rules along the trajectory, KGRL additionally provides local procedural explanations on the pruning of actions and constraining of parameters. Overall, KGRL guides the agent's exploration and deployment toward feasible and constraint-aware decisions, while increasing sample efficiency during training. KGRL outperforms state-of-the-art RL baselines for PAMDPs in both, sample efficiency and episodic return.