Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note4 min read

Interrupting the Loop: Periodic Subject Changes Raise Judged Surprise and Connection in Base Language Models

Source

arXiv

arxiv.org/abs/2608.19893v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Roberto I. Ono Filho
  • Venue: arXiv: cs.CL
  • Published: 2026-08-20
  • Collected: 2026-08-21
  • URL: http://arxiv.org/abs/2608.19893v1
  • PDF: https://arxiv.org/pdf/2608.19893v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

The research examines where novelty comes from in base language models and what they can perceive in a long text stream. The study dismantles a cognitively inspired generation loop across 24 conditions on three base models. The key finding is that injecting a new subject every few hundred tokens (an interruption) into a stream whose repetition is dampened (habituation) increases judged surprise and connection in the generated text by 1.2 to 1.4 points and 0.8, respectively. Other findings include the model replaying earlier segments when presented with a fixed rotation of injected sentences, but the local gains do not compose to produce an integrated document.

What the Document Says

  • The study focuses on understanding the source of novelty in base language models and what they perceive in long text streams.
  • The research involves dismantling a cognitively inspired generation loop across 24 conditions on three base models.
  • Injecting a new subject every few hundred tokens (an interruption) into a stream whose repetition is dampened (habituation) increases judged surprise and connection in the generated text by specific amounts.
  • A judge measures the repeatability of their scores, compared to another judge family, and human readers.
  • Under the given protocol, an interruption raises judged surprise and connection over habituation alone.
  • A connective that asks for continuity hurts; a bare paragraph break adds nothing detectable on fresh text.
  • A reset context does at least as well as a kept one.
  • A pre-registered replication on new premises confirms the primary contrast.
  • Three things the window judge could not see changed the first version of this study.
  • The judge scores the experimenter's injected sentence as the model's own.
  • A fixed rotation of injected sentences makes the model replay its earlier segments from beyond the judge's horizon.
  • The local gains do not compose: no arm produces an integrated document.
  • The salience monitor, the in-loop judge, memory across interruptions, and a judge-gated Review run with a gate that opens add nothing.
  • On a problem with a verifier (online bin packing), the interruption multiplies valid, distinct candidate heuristics three- to fourfold without raising the quality of the best.

Key Concepts Mentioned

  • base language models
  • novelty
  • judged surprise and connection
  • habituation
  • cognitively inspired generation loop
  • interruption
  • online bin packing

Claims Made by the Document

  • Injecting a new subject every few hundred tokens increases judged surprise and connection.
  • A fixed rotation of injected sentences makes the model replay earlier segments.

Evidence Used

  • Increased judged surprise and connection by 1.2 to 1.4 points and 0.8, respectively.

Methods / Approach

  • Dismantling a cognitively inspired generation loop across 24 conditions on three base models.
  • Judging windows of generated text with repeatability measures

Limitations Stated by the Authors

  • The study reports an evaluation protocol for long generation and a controlled characterization of a simple intervention, not a mechanism of creativity.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The study only evaluates three base models. The findings may not be generalizable to all language models.
  • The evaluation protocol might need refinement to better assess the quality of generated documents.

Open Questions

  • What other mechanisms contribute to the creativity of language models?
  • Can more complex interventions lead to better results in terms of generating integrated and high-quality documents?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • It is unclear if the authors considered other potential factors that could contribute to novelty or judged surprise and connection in base language models.
  • The study did not explore how different types of interruptions might impact the performance of language models.

Raw Abstract / Excerpt

Where does the novelty a base language model produces with no task come from, and what can an LLM judge of a long stream actually see? We dismantle a cognitively inspired generation loop over 24 conditions on three base models. Most of its effect lives in one operation: a new subject injected every few hundred tokens (an interruption) into a stream whose literal repetition is damped (habituation). We judge windows of generated text only, with the premise as the unit (n=10) and a judge measured for repeatability, against a second judge family and against human readers. Under that protocol the interruption raises judged surprise by 1.2 to 1.4 points and connection by 0.8 over habituation alone. A connective that asks for continuity hurts; a bare paragraph break adds nothing detectable on fresh text; a reset context does at least as well as a kept one; and a pre-registered replication on new premises confirms the primary contrast. Three things the window judge could not see changed the first version of this study, and we think they are of general use. The judge scores the experimenter's injected sentence as the model's own. A fixed rotation of injected sentences makes the model replay its earlier segments from beyond the judge's horizon, and the judge scores the replay as surprise and connection (65-80% of post-interruption windows at periods 150-300). And the local gains do not compose: no arm produces an integrated document. The salience monitor, the in-loop judge, memory across interruptions and a judge-gated Review run with a gate that opens add nothing. On a problem with a verifier (online bin packing), the interruption multiplies valid, distinct candidate heuristics three- to fourfold without raising the quality of the best. We report an evaluation protocol for long generation and a controlled characterization of a simple intervention, not a mechanism of creativity.