A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Rajat Sainju, Dariusz Jarosz, Hairong Shang, Michael Prince, Ryan M. Aydelott, Mathew J. Cherukara, Yine Sun, Michael D. Borland
• Venue: arXiv: physics.acc-ph
• Published: 2026-07-27
• Collected: 2026-07-28
• URL: http://arxiv.org/abs/2607.24663v1
• PDF: https://arxiv.org/pdf/2607.24663v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers introduce APS-RAG, a platform that makes it easier for staff at the Advanced Photon Source (APS) to access institutional knowledge through natural language queries. The platform combines multiple channels of information and uses a corrective agentic loop. APS-Bench, a question-answering dataset with auditable gold answers, was created to evaluate the platform's performance. The full corrective Agentic GraphRAG scoring improved strict vital-nugget recall by 70.3% over a baseline, and the cross-encoder reranker significantly improved answer quality.

What the Document Says

• The paper introduces APS-RAG, a platform for accessing institutional knowledge at the Advanced Photon Source through natural language queries.
• APS-RAG uses a retrieval engine that fuses dense, sparse, and knowledge-graph channels with query-type-adaptive reciprocal-rank fusion.
• APS-RAG includes a corrective agentic loop and runs a native-tool ReAct executor over a Model Context Protocol (MCP) tooling layer.
• A 50-question, question-answering dataset called APS-Bench was created for evaluating the platform's performance.
• The full corrective Agentic GraphRAG scoring improved strict vital-nugget recall by 70.3% over a baseline.
• The cross-encoder reranker significantly contributed to answer quality, and removing it reduced strict vital recall by 32.8%.

Key Concepts Mentioned

• Advanced Photon Source (APS)
• APS-RAG
• retrieval engine
• query-type-adaptive reciprocal-rank fusion
• corrective agentic loop
• APS-Bench
• question-answering dataset
• strict vital-nugget recall
• cross-encoder reranker

Claims Made by the Document

• APS-RAG improves strict vital-nugget recall over a baseline.
• The cross-encoder reranker significantly contributes to answer quality.

Evidence Used

• Improvement in strict vital-nugget recall from 63.8% (baseline) to 70.3% (full corrective Agentic GraphRAG scoring).
• Removing the cross-encoder reranker and allowing the LLM to score relevance drastically reducing strict vital recall by 32.8%.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Scientific user facilities accumulate decades of operational knowledge that no single search index covers: electronic logbooks, technical documents, internal wikis, operations chat messages, maintenance records, and live control-system data. We present APS-RAG, Advanced Photon Source Retrieval Augmented Generation, a deployed platform that makes the institutional knowledge at the Advanced Photon Source (APS) accessible to staff through natural-language queries, along with an operations-grounded evaluation. The retrieval engine fuses dense, sparse, and knowledge-graph (KG) channels with query-type-adaptive reciprocal-rank fusion, adds a corrective agentic loop, and runs a native-tool ReAct executor over a Model Context Protocol (MCP) tooling layer. We construct APS-Bench, a 50-question, question-answering (QA) dataset with auditable gold answers. Every retrieval-augmented variant numerically improves strict vital-nugget recall over a naive BM25 baseline (63.8%), with the full corrective Agentic GraphRAG scoring (70.3%). The cross-encoder reranker contributes significantly to answer quality: removing it and allowing the LLM to score relevance drastically reduces strict vital recall by 32.8%. The graph channel and corrective loop contribute positively as expected, but the performance gains are marginal. Additionally, we also compare the performance of open-source and closed-source LLMs in final answer synthesis. We release the APS-Bench construction methodology, the six-layer evaluation harness, and the underlying codebase, along with the '/aps-rag' retrieval agent skill framework, to support reproduction and adoption at other facilities. Together, the deployed platform and its operations-grounded evaluation present a promising workflow for trustworthy, statistically grounded AI assistance in facility operations, transferable to other large scientific instruments.