Source note2 min read
Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
Source
arXiv
arxiv.org/abs/2608.12097v1
Source
- Source name: arXiv
- Source kind: paper
- Authors / organization: Xi Chen, Jie Mu, Mo Xuan, Qun Shao
- Venue: arXiv: cs.AI
- Published: 2026-08-12
- Collected: 2026-08-13
- URL: http://arxiv.org/abs/2608.12097v1
- PDF: https://arxiv.org/pdf/2608.12097v1
- DOI: UNKNOWN
- Registry group: academic_apis_and_preprints
- Evidence role: canonical_academic_index
- Language: en
- Country / region: UNKNOWN
Plain-Language Brief
The paper introduces Graph-Structured Rubrics (GSR), a method that converts rubrics into response-independent typed evaluation graphs before observing responses. This allows for more efficient and accurate scoring of tasks under GPT-OSS-120B model. The method improves exact score agreement by varying percentages on four pointwise datasets and achieves the highest end-to-end pairwise accuracy on two preference benchmarks.
What the Document Says
- Graph-Structured Rubrics (GSR) is a method that compiles rubrics into response-independent typed evaluation graphs before observing responses.
- Criterion nodes elicit judgments in the graph.
- Transformation, reduction, and gating operators compose criterion judgments through named ports.
- A task-specific output mapping called Readout converts the unique sink of the graph into a score or preference.
- Compilation rejects malformed or type-incompatible graphs.
- Pointwise evaluation judges rubric dimensions separately before graph aggregation.
- Pairwise evaluation reuses the graph with one judgment for each candidate under every criterion.
- Under GPT-OSS-120B, GSR improves exact score agreement by 0.62% to 6.75% over Prometheus-style scoring on four pointwise datasets.
- GSR achieves the numerically highest end-to-end pairwise accuracy on two preference benchmarks under native tie and abstention policies.
Key Concepts Mentioned
- Graph-Structured Rubrics (GSR)
- Criterion nodes
- Transformation operators
- Reduction operators
- Gating operators
- Readout
- GPT-OSS-120B
- Prometheus-style scoring
Claims Made by the Document
- GSR improves exact score agreement over Prometheus-style scoring.
- GSR achieves the highest end-to-end pairwise accuracy on preference benchmarks.
Evidence Used
- Improvement by 0.62% to 6.75% mentioned in text
- Highest end-to-end pairwise accuracy stated in text
Methods / Approach
UNKNOWN
Limitations Stated by the Authors
UNKNOWN
Limitations Noted by the Agent
Agent interpretation — not from the source.
UNKNOWN
Open Questions
UNKNOWN
Notes for Future Q&A
May include interpretation. Labeled where it does.
UNKNOWN
Raw Abstract / Excerpt
Rubric-based evaluators commonly treat rubrics as prompt context or flat criteria: they specify what to judge but leave criterion composition implicit, even when natural-language rules state it. We introduce Graph-Structured Rubrics (GSR), which compiles a rubric into a response-independent typed evaluation graph before observing responses. Criterion nodes elicit judgments; transformation, reduction, and gating operators compose them through named ports; and a task-specific output mapping, termed Readout, converts the unique sink into a score or preference. Compilation rejects malformed or type-incompatible graphs. Pointwise evaluation judges rubric dimensions separately before graph aggregation; pairwise evaluation reuses the graph with one judgment for each candidate under every criterion. Under GPT-OSS-120B, GSR improves exact score agreement by 0.62--6.75 percentage points over Prometheus-style scoring on four pointwise datasets and achieves the numerically highest end-to-end pairwise accuracy on two preference benchmarks under native tie and abstention policies.
