Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Weave of Formal Thought

Source

arXiv

arxiv.org/abs/2606.25987v1

Read the source

Source

Plain-Language Brief

The paper introduces Weave of Formal Thought (WoFT), a new paradigm for large language models that combines syntactic validation with learned structural representations. The proposed approach includes a formal engine and constrained decoder that guarantees the syntactic validity of the output and maintains concurrent lexer-state hypotheses synchronized with a GLR graph-structured stack. Additionally, a latent-variable fine-tuning method is presented to train the language model to interleave non-terminal grammar symbols directly into generation.

What the Document Says

  • The paper presents Weave of Formal Thought (WoFT), which combines syntactic validation with learned structural representations for large language models.
  • A formal engine and constrained decoder is introduced that guarantees the syntactic validity of the output.
  • The proposed decoder uses a speculative-lexing construction that maintains concurrent lexer-state hypotheses synchronized with a GLR graph-structured stack.
  • A latent-variable fine-tuning method is presented to train the language model to interleave non-terminal grammar symbols directly into generation.
  • The reweighted wake-sleep (RWS) algorithm is used to optimize the importance-weighted evidence lower bound (IW-ELBO) of the surface text.

Key Concepts Mentioned

  • large language models
  • syntactic validation
  • structural representations
  • formal engine
  • constrained decoder
  • speculative-lexing construction
  • generalized LR (GLR) parsing
  • latent-variable fine-tuning
  • reweighted wake-sleep (RWS) algorithm

Claims Made by the Document

  • The proposed approach guarantees the syntactic validity of the output.
  • The latent-variable fine-tuning method recovers critical structural information that flat autoregressive training discards.

Evidence Used

  • The paper provides details on the formal engine and constrained decoder, their soundness and completeness with respect to the full Tree-sitter specification.
  • The paper describes the latent-variable fine-tuning method and how it optimizes the IW-ELBO of the surface text.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Large language models (LLMs) attain remarkable surface fluency on code, yet they neither formally guarantee the syntactic validity of their output nor leverage the hierarchical structure defining the target language. While existing constrained-decoding frameworks address the former, they operate under rigid assumptions that preclude critical lexical mechanisms -- including context-sensitive lexing, maximal-munch tokenization, and keyword extraction -- and only approximate vocabulary masking, sacrificing completeness. For the latter, code LLMs typically inject grammatical structure via predetermined policies rather than learning which structural information to expose. In this work, we introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations. First, we present a formal engine and constrained decoder that is sound and complete with respect to the full Tree-sitter specification. By augmenting generalized LR (GLR) parsing with a speculative-lexing construction that maintains concurrent lexer-state hypotheses synchronized with a GLR graph-structured stack, our decoder admits every subword token extending to a valid program prefix and rejects all others. Second, we present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation. Utilizing the reweighted wake-sleep (RWS) algorithm to optimize the importance-weighted evidence lower bound (IW-ELBO) of the surface text, the model learns to selectively retain formal derivations as an adaptive structural scratchpad. For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline, demonstrating that discretionary latent syntax recovers critical structural information that flat autoregressive training discards.