PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Manyi Wang, Junjielong Xu, Pinjia He
• Venue: arXiv: cs.SE
• Published: 2026-07-30
• Collected: 2026-07-31
• URL: http://arxiv.org/abs/2607.28587v1
• PDF: https://arxiv.org/pdf/2607.28587v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers present PAIChecker, a system designed to detect misalignment between Pull Requests (PRs) and Issues in SWE-bench-like benchmarks used for evaluating Large Language Models (LLMs). The system identifies patterns of misalignment, synthesizes labels across multiple agents, and validates at the code level. Experiments showed that PAIChecker outperforms other LLM backbones on SWE-Gym and SWE-bench Multilingual.

What the Document Says

• Researchers introduce PAIChecker, a multi-agent system for checking misalignment in SWE-bench-like benchmarks.
• PAIChecker has a three-phase design that includes pattern identification, cross-agent label synthesis, and code-level validation.
• The system is used to detect misaligned PR-Issue pairings in SWE-bench-Verified instances.
• 13.6% of SWE-bench Verified instances exhibit misalignment across five patterns in eleven fine-grained scenarios.
• PAIChecker achieves up to 92.12% and 91.67% binary accuracy on SWE-Gym and SWE-bench Multilingual respectively.

Key Concepts Mentioned

• PAIChecker
• SWE-bench-like benchmarks
• PR-Issue misalignment
• Large Language Models (LLMs)
• pattern identification
• cross-agent label synthesis
• code-level validation

Claims Made by the Document

• PAIChecker is designed to detect misalignment in SWE-bench-like benchmarks.
• PAIChecker achieves high accuracy in detecting misaligned PR-Issue pairings.

Evidence Used

• The researchers present PAIChecker as a solution for checking misalignment.
• The text mentions that experiments on SWE-Gym and SWE-bench Multilingual show that PAIChecker achieves high accuracy.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.