Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Prajjwal Gupta, Prasang Gupta, Vishal Bhutani, Apoorva Sharma, Sumanth Chundru, Waqar Sarguroh, Kevin Paul
• Venue: arXiv: cs.AI
• Published: 2026-06-22
• Collected: 2026-06-23
• URL: http://arxiv.org/abs/2606.23403v1
• PDF: https://arxiv.org/pdf/2606.23403v1
• DOI: UNKNOWN

Plain-Language Brief

The paper presents Litmus, a system that creates evaluation metrics for AI pipelines by understanding the purpose of the code and asking questions about it. It was tested on three real AI pipelines and outperformed other systems in terms of coverage, redundancy, and validity.

What the Document Says

• Litmus is a system that designs evaluation metrics for AI pipelines based on source code and targeted interrogation.
• It identifies what needs to be measured and why, then creates constraints for constructing a justified metric portfolio.
• The paper reports three real-world AI pipeline tests (financial account grouping, scientific QA, inherent risk assessment) comparing Litmus with AutoMetrics and DynamicRubric baselines.
• Litmus achieved the broadest or tied-broadest concern coverage, spanned more pipeline stages, had a near-zero-redundancy portfolio, and ranked first in validity against per-row quality labels on all three pipelines.
• The paper mentions that Litmus outperformed in scientific QA with a Spearman correlation of 0.72 compared to less than 0.47 for every baseline.
• The results suggest a shift from automatic metric implementation to automatic metric specification.

Key Concepts Mentioned

• Litmus
• AI pipelines
• evaluation metrics
• source code
• targeted interrogation
• AutoMetrics
• DynamicRubric
• financial account grouping
• scientific QA
• inherent risk assessment

Claims Made by the Document

• Litmus outperformed other systems in terms of coverage, redundancy, and validity.
• Litmus ranked first in validity against per-row quality labels on all three pipelines.
• The results support a shift from automatic metric implementation to automatic metric specification.

Evidence Used

• Comparison of Litmus with AutoMetrics and DynamicRubric baselines across three real-world AI pipeline tests.
• Spearman correlation of 0.72 for Litmus in scientific QA compared to less than 0.47 for every baseline.

Methods / Approach

• source code analysis
• targeted interrogation
• comparison with AutoMetrics and DynamicRubric baselines

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

As agentic LLM systems move from prototypes to deployment across increasingly diverse domains, evaluating them has become both more important and more difficult. The challenge is not only that individual metrics may be unreliable, but that evaluation goals are often left implicit. Without a clear account of what a system is expected to do, how it can fail, and which failures matter, metric choices become difficult to justify, interpret, or validate. We present Litmus, a zero-label system that designs evaluation and monitoring metrics for AI pipelines by eliciting evaluation intent from source code and targeted interrogation. Instead of assuming that the evaluation target is already known, Litmus first identifies what must be measured and why, then converts those answers into constraints for constructing a justified, per-stage metric portfolio. We evaluate Litmus on three real, code-defined AI pipelines - financial account grouping, scientific QA, and inherent risk assessment - against AutoMetrics and three DynamicRubric baselines. Litmus achieves the broadest or tied-broadest concern coverage, spans more pipeline stages, produces a near-zero-redundancy portfolio, and ranks first in validity against per-row quality labels on all three pipelines - decisively on scientific QA (Spearman $ρ=0.72$ vs. less than $0.47$ for every baseline), and within overlapping confidence intervals in relation to two components of the audit framework despite using no labels during metric design. Our results support a shift from automatic metric implementation to automatic metric specification: before asking which metric to compute, evaluation systems should ask what must be measured and why.