Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

Source

arXiv

arxiv.org/abs/2607.24717v1

Read the source

Source

Plain-Language Brief

Researchers propose a framework called DataOrchestra that adapts data processing strategies for each example in Large Language Models (LLMs) pretraining, improving performance across 11 benchmarks.

What the Document Says

  • Pretraining data processing is crucial for the downstream performance of LLMs.
  • Many existing approaches use a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples.
  • The researchers propose DataOrchestra, a framework that unifies different processing operations and adapts an example-specific pipeline for each example.
  • DataOrchestra decides whether to drop, leave untouched, or clean a chunk of pretraining data.
  • For cleaning, it selects one or more downstream operations ranging from programmatic editing to LLM-based rewriting.
  • Each rewriting step generates a concrete instruction that is executed by the corresponding downstream tool model.
  • The researchers pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks.
  • DataOrchestra is effective for math continued pretraining and outperforms stronger processing baselines while reducing processing compute.

Key Concepts Mentioned

  • Pretraining data processing
  • Large Language Models (LLMs)
  • DataOrchestra
  • Data processing operations
  • downstream tool model

Claims Made by the Document

  • Pretraining data processing is crucial for the downstream performance of LLMs.
  • DataOrchestra improves performance across benchmarks.
  • DataOrchestra outperforms stronger processing baselines while reducing processing compute.

Evidence Used

  • The text states that pretraining data processing is critical to the downstream performance of LLMs and mentions stable average gains over individual data-processing methods across 11 benchmarks.
  • The text states that DataOrchestra outperforms stronger processing baselines while reducing processing compute.

Methods / Approach

  • DataOrchestra framework for adapting example-specific pipelines for pretraining data

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.