Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

Source

arXiv

arxiv.org/abs/2607.24717v1

Read the source

Source

Plain-Language Brief

The paper proposes a framework called DataOrchestra that learns to curate pre-training data example-by-example rather than applying a uniform strategy across many examples. The framework decides whether to drop, clean, or leave the data as is and uses various downstream operations for cleaning if necessary. The authors pretrain models from 0.5B to 7B on web data processed by DataOrchestra and observe consistent improvements over individual data-processing methods across 11 benchmarks.

What the Document Says

  • DataOrchestra is a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example in pretraining data.
  • The orchestrator decides whether to drop, untouch, or clean a chunk of pretraining data.
  • For cleaning, it selects one or more downstream operations ranging from programmatic editing to LLM-based rewriting.
  • For each rewriting step, it generates a concrete instruction that is executed by the corresponding downstream tool model.
  • The authors pretrain models from 0.5B to 7B on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks.
  • DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines while reducing processing compute.

Key Concepts Mentioned

  • DataOrchestra
  • pretraining data processing
  • Large Language Models (LLMs)
  • downstream operations
  • programmatic editing
  • LLM-based rewriting

Claims Made by the Document

  • DataOrchestra improves pretraining data processing.
  • DataOrchestra outperforms stronger processing baselines for math continued pretraining.
  • DataOrchestra reduces processing compute.

Evidence Used

  • Stable average gains over individual data-processing methods across 11 benchmarks.

Methods / Approach

  • Pretraining models from 0.5B to 7B on web data processed by DataOrchestra.
  • Using an orchestrator to decide whether to drop, clean, or leave the data as is.

Limitations Stated by the Authors

  • UNKNOWN (Not stated in source)

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The paper does not provide detailed information about the specific methods used for programmatic editing and LLM-based rewriting.
  • The paper does not mention any potential issues or limitations that might arise from the adaptive approach taken by DataOrchestra.

Open Questions

  • What specific methods are used for programmatic editing and LLM-based rewriting?
  • Are there any potential issues or limitations with the adaptive approach taken by DataOrchestra?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper focuses on presenting the DataOrchestra framework and its effectiveness, but does not delve into the specific details of the methods used for programmatic editing and LLM-based rewriting.

Raw Abstract / Excerpt

Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.