OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Indraneil Paul, Falko Helm, Goran Glavaš, Iryna Gurevych
• Venue: arXiv: cs.AI
• Published: 2026-08-05
• Collected: 2026-08-06
• URL: http://arxiv.org/abs/2608.05141v1
• PDF: https://arxiv.org/pdf/2608.05141v1
• DOI: UNKNOWN

Plain-Language Brief

The research presents a new approach called OctoLong for creating longer contexts for language models. It uses an AST parser, a language server backend, and a package manager to retrieve code references from various repositories, enabling the creation of dependency-rich contexts of millions of tokens. The authors train OctoLong-Instruct, a suite of long-context open LMs, by replacing 12% of traditional context-extension corpora with OctoLong data. This results in improvements in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks.

What the Document Says

• Context lengths of language models have increased due to demands for in-context learning, self-improvement, and long-horizon agentic workflows.
• Existing long-context corpora are dominated by books, academic articles, and code repositories.
• OctoLong is a context engineering pipeline that uses an AST parser, a language server backend, and a package manager to retrieve code references for the creation of dependency-rich long contexts.
• The authors train OctoLong-Instruct, a suite of long-context open LMs, using a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts and ~10B tokens of instruction tuning.

Key Concepts Mentioned

• language models
• long-context corpora
• AST parser
• package manager
• OctoLong
• OctoLong-Instruct

Claims Made by the Document

• Using OctoLong data enhances long-range retrieval.
• Using OctoLong data improves long-term state tracking.

Evidence Used

• Experimental evaluations against 18 state-of-the-art open-weight long-context LMs show...

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency-rich code contexts of millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 state-of-the-art open-weight long-context LMs show that supplanting just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.