Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Freeform Preference Learning for Robotic Manipulation

Source

arXiv

arxiv.org/abs/2606.32027v1

Read the source

Source

Plain-Language Brief

Researchers introduce Freeform Preference Learning (FPL), a method for training robotic policies based on natural-language preferences provided by humans, rather than binary labels. The method uses the preferences to learn a language-conditioned reward model that optimizes across multiple human-specified dimensions, improving performance in long-horizon manipulation tasks.

What the Document Says

  • The paper introduces Freeform Preference Learning (FPL).
  • FPL is a method for learning robot policies from freeform human preferences.
  • Instead of asking annotators to choose the better overall trajectory, FPL allows them to define natural-language preference axes and provide pairwise preferences along each axis.
  • These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward.
  • The learned reward model is then used to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions.
  • The researchers tested FPL on four real-world and two simulated long-horizon manipulation tasks, showing an improvement of 38 percentage points over sparse-reward and binary-preference methods.
  • FPL provides dense progress signals without explicit subtask segmentation.
  • FPL also shows compositionality of behavior not present in the data.
  • Users can steer the policy towards different behaviors at test time without retraining.
  • A blog post with videos is available at https://freeform-pl.github.io/fpl.website/

Key Concepts Mentioned

  • Freeform Preference Learning (FPL)
  • robot policies
  • human preferences
  • language-conditioned reward model
  • reward-conditioned policy
  • long-horizon manipulation tasks
  • performance improvement

Claims Made by the Document

  • FPL improves over sparse-reward and binary-preference methods by 38 percentage points.
  • FPL provides dense progress signals without explicit subtask segmentation.

Evidence Used

  • The researchers tested FPL on four real-world and two simulated long-horizon manipulation tasks, showing an improvement of 38 percentage points over sparse-reward and binary-preference methods.
  • FPL learns dense progress signals without explicit subtask segmentation.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Reward design remains a central bottleneck for autonomous robot policy improvement, especially in long-horizon manipulation tasks where sparse success labels provide too little signal and binary preferences collapse many competing notions of quality into one ambiguous signal. We introduce Freeform Preference Learning (FPL), a method for learning robot policies from freeform human preferences. Rather than asking annotators which of two trajectories is better overall, FPL lets them define natural-language preference axes, such as speed, safety, quality of placement, or carefulness, and provide pairwise preferences along each axis. These annotations are used to learn a language-conditioned reward model that maps a trajectory and preference label to an axis-specific reward. We use this model to train a reward-conditioned policy that optimizes across the multiple human-specified dimensions. Across four real-world and two simulated long-horizon manipulation tasks, FPL improves over sparse-reward and binary-preference methods by 38 percentage points. Beyond improved performance, FPL learns dense progress signals without explicit subtask segmentation, shows compositionality of behavior not present in the data, and allows users to steer the policy towards different behaviors at test time without retraining. Blog post with videos available at https://freeform-pl.github.io/fpl.website/