SPIRAL: Learning to Search and Aggregate

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jubayer Ibn Hamid, Ifdita Hasan Orney, Michael Y. Li, Omar Shaikh, Yoonho Lee, Dorsa Sadigh, Chelsea Finn, Noah Goodman
• Venue: arXiv: cs.AI
• Published: 2026-06-22
• Collected: 2026-06-23
• URL: http://arxiv.org/abs/2606.23595v1
• PDF: https://arxiv.org/pdf/2606.23595v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers introduced SPIRAL, a framework for improving language model reasoning at test time. The framework uses three primitives - sequential reasoning within a trace, independently sampled parallel traces, and aggregation of multiple reasoning traces into a final response. During training, the system learns to use all three primitives in a unified pipeline. Experiments on reasoning tasks show that SPIRAL outperforms GRPO by up to 11 times in scaling efficiency and 15% higher performance when all three compute primitives are scaled.

What the Document Says

• The research introduces Sequential-Parallel-Aggregative Reinforcement Learning (SPIRAL).
• SPIRAL is a framework for improving language model reasoning at test time using sequential reasoning, independently sampled parallel traces, and aggregation of multiple reasoning traces.
• During training, the system learns to use all three primitives in a unified pipeline.
• SPIRAL uses set reinforcement learning to teach models to produce a set of traces that are collectively useful for an aggregator.
• Standard reinforcement learning is used to teach models to aggregate the set into improved final responses.
• The experiments were conducted on reasoning tasks.
• SPIRAL outperforms GRPO by up to 11 times in scaling efficiency and 15% higher performance when all three compute primitives are scaled.

Key Concepts Mentioned

• Language model
• Reasoning
• Sequential reasoning
• Parallel traces
• Aggregation
• SPIRAL
• GRPO

Claims Made by the Document

• SPIRAL effectively scales with inference compute.
• SPIRAL outperforms GRPO.

Evidence Used

• The text states that SPIRAL outperformed GRPO by up to 11 times in scaling efficiency and 15% higher performance when all three compute primitives are scaled.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Language model reasoning can be substantially improved at test time via scaffolds that scale inference compute across different primitives -- sequential reasoning within a trace, independently sampled parallel traces, and aggregation of multiple reasoning traces into a final response. During post-training, however, language models are optimized only for sequential reasoning within a single trace. We introduce Sequential-Parallel-Aggregative Reinforcement Learning (SPIRAL), a framework in which a language model is trained to use all three primitives, as part of a unified inference compute pipeline. Concretely, the language model first samples a set of independent traces in parallel, each produced through sequential chain-of-thought reasoning, and then generates a final aggregation trace conditioned on those traces; all components are optimized end-to-end against the reward of the final aggregated response. To train this system, SPIRAL uses set reinforcement learning to teach models to produce a set of traces that are collectively useful for an aggregator and standard reinforcement learning to teach models to aggregate the set into improved final responses. Our experiments on reasoning tasks show that SPIRAL effectively scales with inference compute, outperforming GRPO by up to 11$\times$ scaling efficiency and 15% higher performance when all three compute primitives are scaled.