RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jinjun Huang, Zhongzhen Wen, Tongtong Xu, Meng Yan, Xin Xia, Zhongxin Liu
• Venue: arXiv: cs.SE
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12004v1
• PDF: https://arxiv.org/pdf/2608.12004v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers have introduced a new benchmark called RealisticTritonBench to improve the evaluation of large language models (LLMs) in generating GPU kernels for AI frameworks using Triton. The new benchmark derives tasks from real-world pull requests and evaluates end-to-end performance, addressing limitations in prior benchmarks.

What the Document Says

• The text discusses a study that presents RealisticTritonBench, a benchmark for evaluating large language models (LLMs) in generating GPU kernels.
• Modern AI frameworks rely on GPU kernels for system performance, and Triton is widely used due to its usability, portability, and near-handwritten CUDA performance.
• Recent advances show the potential of LLMs to automatically generate Triton kernels, reducing manual effort.
• Prior benchmarks evaluate LLM-generated Triton kernels but suffer from three limitations: they restrict tasks to PyTorch-to-Triton translation, evaluate only individual-kernel performance, and rely on manually written evaluation scripts.
• RealisticTritonBench derives tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation.
• The benchmark integrates generated kernels into their original frameworks and evaluates them using end-to-end tests for a more faithful assessment.
• Evaluation of leading LLMs on RealisticTritonBench found that they still struggle with real-world Triton kernel generation tasks.

Key Concepts Mentioned

• AI Frameworks
• GPU Kernels
• Triton
• Large Language Models (LLMs)
• Benchmarking
• RealisticTritonBench

Claims Made by the Document

• Prior benchmarks suffer from three limitations: they restrict tasks, evaluate only individual-kernel performance, and rely on manually written evaluation scripts.
• RealisticTritonBench addresses these limitations by deriving tasks from real-world pull requests, integrating generated kernels into their original frameworks, and evaluating using end-to-end tests.

Evidence Used

• The text discusses the limitations of prior benchmarks and the approach taken by RealisticTritonBench to overcome these limitations.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

• What specific large language models were evaluated on RealisticTritonBench?
• How does RealisticTritonBench handle tasks requiring custom or proprietary knowledge?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The text does not specify which specific large language models were evaluated.
• The text does not mention how RealisticTritonBench handles tasks that require custom or proprietary knowledge to solve.

Raw Abstract / Excerpt

In modern AI frameworks, GPU kernels are key to overall system performance. Combining usability, portability, and near-handwritten CUDA performance, Triton is widely adopted for implementing GPU kernels. Recent advances show the potential of large language models (LLMs) to automatically generate Triton kernels, reducing the manual effort required from expert kernel developers. Several benchmarks evaluate LLM-generated Triton kernels. However, they suffer from three key limitations: (1) they restrict tasks to PyTorch-to-Triton translation, failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only individual-kernel performance rather than end-to-end performance, the core criterion for real-world deployment in AI frameworks; and (3) they rely on manually written evaluation scripts for individual kernels, which may contain flaws that models can exploit to bypass correctness checks and obtain inflated scores. To address these limitations, we introduce RealisticTritonBench, the first benchmark to derive Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation. RealisticTritonBench systematically extracts PRs that modify Triton kernels from popular open-source AI frameworks and transforms them into generation tasks with concrete engineering contexts. Each task takes a natural language requirement as input and requires a corresponding Triton kernel implementation, with a complete and reproducible evaluation environment. Unlike prior benchmarks focused on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them using end-to-end tests, enabling a more faithful assessment. We evaluate leading LLMs on RealisticTritonBench and find that they still struggle with real-world Triton kernel generation tasks.