UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Xinlong Zhao, Dongsheng Liu, Hengyu Zhao, Zixuan Fu, Zheng Wang, Jie Cai, Jie Zhou, Qiang Ma, Xuanhe Zhou, Xu Han, Yudong Wang, Zhiyuan Liu
• Venue: arXiv: cs.CL
• Published: 2026-07-09
• Collected: 2026-07-10
• URL: http://arxiv.org/abs/2607.08646v1
• PDF: https://arxiv.org/pdf/2607.08646v1
• DOI: UNKNOWN

Plain-Language Brief

UltraX is a new framework designed to improve the quality of large-scale pre-training data for Large Language Models (LLMs). It addresses the limitations of existing methods by offering fine-grained instance-level editing capabilities, including insertion in addition to deletion and modification. UltraX utilizes a reliable program-supervision generation pipeline that enhances the quality of refined texts and improves the stability and reliability of large-scale execution.

What the Document Says

• The paper proposes UltraX, a function-calling refinement framework for large-scale pre-training data.
• UltraX aims to complete the editing function space by introducing insertion along with deletion and modification.
• It builds a reliable program-supervision generation pipeline.
• The pipeline includes dataset-adaptive prompt optimization, Line Alignment Mapping, Dynamic Context Replacement, low-confidence example filtering, ratio-controlled sampling by operation combination, sliding-window prediction, global operation aggregation, and systematic post-processing.
• UltraX aims to improve supervision quality and stabilize the training distribution.
• Experiments show that UltraX achieves the highest average performance across all corpora and matches or surpasses baselines with fewer training tokens.

Key Concepts Mentioned

• Large Language Models (LLMs)
• UltraX
• refinement framework
• pre-training data
• insertion
• deletion
• modification
• program-supervision generation pipeline
• dataset-adaptive prompt optimization
• Line Alignment Mapping
• Dynamic Context Replacement
• low-confidence example filtering
• ratio-controlled sampling
• sliding-window prediction
• global operation aggregation
• systematic post-processing

Claims Made by the Document

• UltraX addresses limitations of existing refinement methodologies.
• UltraX achieves the highest average performance across all corpora.
• UltraX matches or surpasses baselines with fewer training tokens.

Evidence Used

• The text discusses the challenges faced by existing methods and introduces UltraX as a solution.
• The text mentions experiments that show UltraX's performance across all corpora.

Methods / Approach

• dataset-adaptive prompt optimization
• Line Alignment Mapping
• Dynamic Context Replacement
• low-confidence example filtering
• ratio-controlled sampling
• sliding-window prediction
• global operation aggregation
• systematic post-processing

Limitations Stated by the Authors

The paper does not discuss any potential limitations of UltraX beyond those inherent to the process of refining pre-training data for LLMs.

Limitations Noted by the Agent

Agent interpretation — not from the source.

As a text summary model, I may not be able to identify or predict all possible limitations or future challenges that UltraX might encounter.

Open Questions

• What specific improvements in performance does UltraX bring over existing methods?
• How scalable is UltraX for large-scale data processing?
• What are the potential impacts of UltraX on the development and application of LLMs?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The text does not provide specific numbers or details regarding the improvements in performance, scalability, or impact.

Raw Abstract / Excerpt

As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish. Consequently, improving Large Language Models (LLMs) now depends less on data expansion and more on higher-quality data utilization. However, in the context of large-scale corpora, existing refinement methodologies face significant limitations in quality, efficiency, and reliability: Rule-based approaches are constrained by fixed heuristics and struggle with instance-level variations; LLM-based approaches improve quality but fail to meet the efficiency and reliability requirements of large-scale data processing. To address these challenges, we propose UltraX, a function-calling refinement framework for large-scale pre-training data that completes the editing function space by introducing insertion in addition to deletion and modification, enabling fine-grained instance-level editing. Specifically, UltraX builds a reliable program-supervision generation pipeline. In this pipeline, dataset-adaptive prompt optimization first guides an expert LLM to produce high-quality end-to-end refined texts, and Line Alignment Mapping and Dynamic Context Replacement then convert original-refined text pairs into structured program supervision. Meanwhile, UltraX improves supervision quality and stabilizes the training distribution with low-confidence example filtering and ratio-controlled sampling by operation combination. During inference and execution, it normalizes and validates model outputs through sliding-window prediction, global operation aggregation, and systematic post-processing, improving the stability and reliability of large-scale execution. Experiments show that UltraX achieves the highest average performance across all corpora and also matches or surpasses baselines with fewer training tokens, demonstrating stronger data efficiency and refinement reliability.