Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Co-LMLM: Continuous-Query Limited Memory Language Models

Source

arXiv

arxiv.org/abs/2607.07707v1

Read the source

Source

Plain-Language Brief

Researchers have developed a new type of language model called Co-LMLM (Continuous-Query Limited Memory Language Model). This model stores factual knowledge in a knowledge base instead of memorizing it in its weights. It can fetch knowledge from the knowledge base during generation and integrate human-readable and attributable retrieved knowledge into its responses. The model outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision, even when pretrained at smaller scales.

What the Document Says

  • The paper presents Co-LMLM, a type of language model that externalizes factual knowledge during pretraining to a knowledge base instead of memorizing it.
  • During generation, the model fetches knowledge from the knowledge base as needed.
  • Co-LMLM integrates human-readable and attributable retrieved knowledge into its responses.
  • The authors claim that CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision.
  • At a scale of 360M, CO-LMLM has lower perplexity than models pretrained on 40x more data.
  • CO-LMLM's SimpleQA-verified performance is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.

Key Concepts Mentioned

  • Limited Memory Language Models (LMLMs)
  • Continuous-Query LMLM (CO-LMLM)
  • Knowledge base
  • Factual precision
  • Perplexity

Claims Made by the Document

  • Co-LMLM outperforms prior LMLMs and vanilla LLMs.
  • At a scale of 360M, CO-LMLM has lower perplexity than models pretrained on 40x more data.
  • CO-LMLM's SimpleQA-verified performance is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.

Evidence Used

  • The paper states that CO-LMLM outperforms prior LMLMs and vanilla LLMs.
  • The paper mentions the lower perplexity of CO-LMLM at a scale of 360M compared to models pretrained on more data.
  • The paper states that CO-LMLM's SimpleQA-verified performance is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.

Methods / Approach

  • Pairing the design of continuous-query LMLM with an annotation pipeline that tags free-form factual spans in arbitrary text.
  • Pretraining CO-LMLM on Wikipedia and FineWeb-Edu

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights. During generation, the model then fetches knowledge from the KB as needed. This recently introduced paradigm provides multiple advantages, including knowledge control capabilities that remain beyond conventional LLMs. We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries. CO-LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation. We pair this design with an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior work's restriction to Wikipedia. Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pretrained on 40x more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.