Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

Source

arXiv

arxiv.org/abs/2608.05141v1

Read the source

Source

Plain-Language Brief

In this research, a new pipeline called OctoLong is introduced to enhance the training of language models (LM) by providing them with long code contexts. This is done through the use of an AST parser, a language server backend, and a package manager. The researchers trained a suite of LMs on a mixture of over 50 billion tokens, including OctoLong code contexts, and found that replacing just 12% of traditional context-extension corpora with OctoLong data led to improvements in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks.

What the Document Says

  • The study presents OctoLong, a pipeline using an AST parser, language server backend, and package manager to retrieve and curate dependency-rich code contexts for long-context LMs.
  • OctoLong training involves mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts and ~10B tokens of instruction tuning.
  • The study compares the performance of OctoLong-Instruct with 18 state-of-the-art open-weight long-context LMs on long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks.
  • Results show that replacing 12% of traditional context-extension corpora with OctoLong data improves performance in the mentioned areas.

Key Concepts Mentioned

  • language models
  • context engineering pipeline
  • AST parser
  • code contexts
  • long-range retrieval
  • long-term state tracking
  • repository-level code understanding
  • downstream agentic tasks

Claims Made by the Document

  • OctoLong enhances the training of long-context LMs.
  • OctoLong training involves mid-training and instruction tuning on a large mixture of tokens.
  • OctoLong improves performance in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks compared to other LMs.

Evidence Used

  • The study presents the OctoLong pipeline and its training process.
  • Experimental evaluations against 18 state-of-the-art open-weight long-context LMs are provided in the paper.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. Existing long-context corpora, however, are dominated by books, academic articles, and code repositories, which are finite resources and often scarce in long-distance dependencies. In this work, we introduce OctoLong, a context engineering pipeline that instruments an AST parser, a language server backend, and a package manager to facilitate the recursive retrieval of code references, enabling the curation of dependency-rich code contexts of millions of tokens in length. We then train OctoLong-Instruct, a suite of capable long-context open LMs, derived from base models ranging in size from 600M to 14B parameters, via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning. Our training ablations and experimental evaluations against 18 state-of-the-art open-weight long-context LMs show that supplanting just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks, while also enhancing API usage in short-context coding scenarios.