CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Sayak Dutta
• Venue: arXiv: cs.CL
• Published: 2026-06-25
• Collected: 2026-06-27
• URL: http://arxiv.org/abs/2606.27229v1
• PDF: https://arxiv.org/pdf/2606.27229v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces CARVE (Content-Aware Recurrent with Value Efficiency), a recurrent model that addresses three issues in the leading delta-rule architecture (GDN-2). The primary problem is memory-blind gating, where the state of the art decides what to erase without considering the memory it will modify. CARVE resolves this by erasing only on the key axis and reusing the recurrent output tensor as a free content signal for the erase gate. It achieves competitive performance with Transformers while using less memory and fewer parameters.

What the Document Says

• The paper proposes CARVE, a recurrent model that addresses three problems in GDN-2.
• One of the issues addressed is memory-blind gating.
• In CARVE, erase only on the key axis.
• CARVE reuses the recurrent output tensor as a free content signal for the erase gate.
• CARVE replaces per-value write-gate projection with a single scalar per head.
• CARVE achieves competitive performance on WikiText perplexity and common-sense reasoning benchmarks, and sets state of the art on RULER retrieval probes.
• Six formal theorems cover various aspects of CARVE's properties.

Key Concepts Mentioned

• Recurrent models
• GDN-2
• CARVE
• memory-blind gating
• content signal
• write-gate projection
• WikiText perplexity
• common-sense reasoning benchmarks
• RULER retrieval probes

Claims Made by the Document

• CARVE resolves the memory-blind gating issue in GDN-2.
• CARVE achieves competitive performance with Transformers on various tasks.
• Six formal theorems cover various aspects of CARVE's properties.

Evidence Used

• The paper explains how CARVE resolves memory-blind gating.
• Results from testing CARVE on various benchmarks are provided.

Methods / Approach

• Introduction of a new recurrent model (CARVE) and mathematical proofs for its properties

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Recurrent models must forget in order to remember, yet the state of the art decides what to erase without consulting what is stored -- the gate sees only the arriving token, not the memory it is about to modify. This memory-blind gating is one of three coupled defects in the leading delta-rule architecture (GDN-2): the value-axis erase mask wastes parameters at the scale of the value projection, and -- as we prove -- mathematically prevents the WY-form triangular chunk solver that makes recurrent training competitive with Transformers. We introduce CARVE (Content-Aware Recurrent with Value Efficiency), which resolves all three problems through one principle: erase only on the key axis. This is provably necessary and sufficient for the WY-form solver to remain valid. Within it, CARVE reuses the recurrent output tensor -- already written to GPU memory -- as a free content signal for the erase gate, and replaces the per-value write-gate projection with a single scalar per head. At initialisation CARVE is bit-identical to GDN-2; any quality difference emerges from what the content gate learns. At 1.3B parameters trained on 100B tokens, CARVE achieves WikiText perplexity 15.72 (minus 0.18 vs. GDN-2, a 4.5-sigma effect), leads every recurrent baseline on nine common-sense reasoning benchmarks, and sets state of the art on every RULER retrieval probe -- at 0.4% throughput overhead, 13% lower peak memory, and 19% fewer parameters. Six formal theorems cover memory capacity, Lyapunov stability, gradient flow, expressivity separation, Pareto-optimal chunk size, and hybrid optimality.