RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jinjun Huang, Zhongzhen Wen, Tongtong Xu, Meng Yan, Xin Xia, Zhongxin Liu
• Venue: arXiv: cs.SE
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12004v1
• PDF: https://arxiv.org/pdf/2608.12004v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

A new benchmark called RealisticTritonBench was introduced to test the ability of large language models (LLMs) to generate GPU kernels for real-world AI frameworks. This benchmark addresses limitations in previous methods by using tasks derived from actual pull requests in popular AI frameworks and evaluating the generated kernels in their original environments.

What the Document Says

• Modern AI frameworks rely heavily on GPU kernels for performance.
• Triton is a widely-used tool for implementing GPU kernels due to its usability, portability, and near-handwritten CUDA performance.
• Recent advances show potential in using large language models (LLMs) to automatically generate Triton kernels.
• Existing benchmarks for LLM-generated Triton kernels have limitations: they focus on PyTorch-to-Triton translation, individual-kernel performance, and manual evaluation scripts.
• RealisticTritonBench is introduced to address these limitations by deriving tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation.
• RealisticTritonBench extracts PRs that modify Triton kernels from popular open-source AI frameworks and transforms them into generation tasks with concrete engineering contexts.
• Each task requires a natural language requirement as input and a corresponding Triton kernel implementation, along with a complete and reproducible evaluation environment.
• Unlike previous benchmarks focused on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them using end-to-end tests.
• Leading LLMs were evaluated on RealisticTritonBench and found to still struggle with real-world Triton kernel generation tasks.

Key Concepts Mentioned

• AI frameworks
• GPU kernels
• Triton
• Large language models (LLMs)
• RealisticTritonBench
• Pull requests
• Natural language requirements

Claims Made by the Document

• Existing benchmarks for LLM-generated Triton kernels have limitations.
• RealisticTritonBench addresses these limitations by deriving tasks from real-world pull requests in popular AI frameworks.

Evidence Used

• The text mentions the limitations of existing benchmarks and the introduction of RealisticTritonBench to address these issues.
• The details about RealisticTritonBench's design, including its focus on real-world tasks and end-to-end evaluation, are explained.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

In modern AI frameworks, GPU kernels are key to overall system performance. Combining usability, portability, and near-handwritten CUDA performance, Triton is widely adopted for implementing GPU kernels. Recent advances show the potential of large language models (LLMs) to automatically generate Triton kernels, reducing the manual effort required from expert kernel developers. Several benchmarks evaluate LLM-generated Triton kernels. However, they suffer from three key limitations: (1) they restrict tasks to PyTorch-to-Triton translation, failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only individual-kernel performance rather than end-to-end performance, the core criterion for real-world deployment in AI frameworks; and (3) they rely on manually written evaluation scripts for individual kernels, which may contain flaws that models can exploit to bypass correctness checks and obtain inflated scores. To address these limitations, we introduce RealisticTritonBench, the first benchmark to derive Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation. RealisticTritonBench systematically extracts PRs that modify Triton kernels from popular open-source AI frameworks and transforms them into generation tasks with concrete engineering contexts. Each task takes a natural language requirement as input and requires a corresponding Triton kernel implementation, with a complete and reproducible evaluation environment. Unlike prior benchmarks focused on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them using end-to-end tests, enabling a more faithful assessment. We evaluate leading LLMs on RealisticTritonBench and find that they still struggle with real-world Triton kernel generation tasks.