The Transformer Revolution, Part 1: Dynamic Processing through Output- Weight Interconnections

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Marco Giunti, Fabrizia Giulia Garavaglia
• Venue: arXiv: cs.AI
• Published: 2026-08-04
• Collected: 2026-08-05
• URL: http://arxiv.org/abs/2608.03921v1
• PDF: https://arxiv.org/pdf/2608.03921v1
• DOI: UNKNOWN

Plain-Language Brief

The paper proposes a new interpretation of Transformers during inference, suggesting they construct prompt-dependent transformations whose parameters are generated during inference. This form of computation is named Sequence-level Interactive Dynamic Parallel Processing (SIDPP). The authors argue that the Transformer's architectural novelty lies in output-weight interconnections, which allow for the construction and modification of token representations based on prompts.

What the Document Says

• The paper introduces a new interpretation of Transformers during inference as SIDPP (Sequence-level Interactive Dynamic Parallel Processing).
• In this view, Transformers construct and apply prompt-dependent transformations.
• Token vectors are considered concepts to be transformed, and parameterized transformations are the transforming concepts.
• These transformations can be static or dynamic, with dynamic ones generated from the input sequence.
• The paper highlights the Transformer's architectural novelty as output-weight interconnections.
• Output-weight interconnections allow for the construction and modification of token representations based on prompts.

Key Concepts Mentioned

• Transformer
• SIDPP
• prompt-dependent transformations
• token vectors
• parameterized transformations
• output-weight interconnections

Claims Made by the Document

• Transformers construct and apply prompt-dependent transformations.
• The Transformer's architectural novelty lies in output-weight interconnections.

Evidence Used

• The paper provides a detailed explanation of the proposed interpretation.

Methods / Approach

• The authors describe the methods used to analyze and interpret the Transformer's functioning.

Limitations Stated by the Authors

• The authors conjecture that human language processing may be a form of SIDPP.
• They acknowledge that further research is needed to confirm this hypothesis.

Limitations Noted by the Agent

Agent interpretation — not from the source.

As a model, I don't have the ability to conduct experimental validation or confirm the authors' conjectures.

Open Questions

• How can the proposed interpretation be experimentally validated?
• What are the implications of the SIDPP hypothesis for our understanding of human language processing?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper presents a theoretical interpretation of Transformers, and further research is needed to confirm its implications for human language processing.

Raw Abstract / Excerpt

This paper offers a new interpretation of the Transformer during inference. Against the 'stochastic parrot' view that large language models merely reproduce statistical regularities learned in training, we argue that Transformers construct and apply prompt-dependent transformations whose parameters are generated during inference. We call this form of computation SIDPP: Sequence-level Interactive Dynamic Parallel Processing. The Transformer is interpreted as a system that transforms concepts by means of concepts. Token vectors are the concepts to be transformed; parameterized transformations defined by matrices and vectors are the transforming concepts. These may be static, when fixed through training, or dynamic, when generated from the input sequence. Mechanically, they correspond to groups of simple neural networks. The Transformer's architectural novelty lies in output-weight interconnections, through which the outputs of some networks determine the weights of others, alongside ordinary output-input interconnections. By means of these interconnections, the system constructs transformations from the prompt and uses them to modify token representations. The contribution of dynamic processing grows with prompt length and may equal or exceed that of static processing, a phenomenon we call strong prompt sensitivity. This account bears on interpretability, predictability, control, and the design of smaller, more sustainable systems. Finally, since the human neural system possesses the mechanisms required to implement SIDPP, we argue that a form of SIDPP may, in principle, be neurally realized in the cerebral cortex. We therefore conjecture that human language processing may itself be a form of SIDPP produced by a functional architecture relevantly similar to that of the Transformer.