A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility

Source

Source name: arXiv
Source kind: paper
Authors / organization: Rajat Sainju, Dariusz Jarosz, Hairong Shang, Michael Prince, Ryan M. Aydelott, Mathew J. Cherukara, Yine Sun, Michael D. Borland
Venue: arXiv: physics.acc-ph
Published: 2026-07-27
Collected: 2026-07-28
URL: http://arxiv.org/abs/2607.24663v1
PDF: https://arxiv.org/pdf/2607.24663v1
DOI: UNKNOWN

Plain-Language Brief

Researchers have developed a platform called APS-RAG for scientific facilities like the Advanced Photon Source (APS). The platform aims to make the vast amount of institutional knowledge accessible through natural language queries. It uses a combination of dense, sparse, and knowledge-graph channels, query-type-adaptive reciprocal-rank fusion, and a corrective agentic loop. APS-RAG has been tested with a 50-question dataset named APS-Bench, and it performs better than a naive BM25 baseline, especially when using the cross-encoder reranker. The researchers also compare the performance of open-source and closed-source LLMs in final answer synthesis. They plan to release their codebase and evaluation tools for others to reproduce and adopt the platform at other facilities.

What the Document Says

APS-RAG is a platform that makes institutional knowledge at the Advanced Photon Source (APS) accessible via natural language queries.
The platform uses a retrieval engine that fuses dense, sparse, and knowledge-graph channels with query-type-adaptive reciprocal-rank fusion.
It includes a corrective agentic loop and runs on a native-tool ReAct executor over a Model Context Protocol (MCP) tooling layer.
APS-Bench is a 50-question question-answering dataset created for evaluation.
The full corrective Agentic GraphRAG scoring improves strict vital-nugget recall over a naive BM25 baseline by 6.5% (70.3% vs 63.8%).
The cross-encoder reranker contributes significantly to answer quality, as removing it reduces strict vital recall by 32.8%.
The graph channel and corrective loop contribute positively but with marginal performance gains.
The researchers compare the performance of open-source and closed-source LLMs in final answer synthesis.
They plan to release APS-Bench construction methodology, six-layer evaluation harness, underlying codebase, and '/aps-rag' retrieval agent skill framework.

Key Concepts Mentioned

APS-RAG
Advanced Photon Source
retrieval engine
dense channels
sparse channels
knowledge-graph channels
query-type-adaptive reciprocal-rank fusion
corrective agentic loop
ReAct executor
Model Context Protocol (MCP) tooling layer
APS-Bench
question-answering dataset
gold answers
LLM
final answer synthesis

Claims Made by the Document

APS-RAG improves strict vital-nugget recall over a naive BM25 baseline.
The cross-encoder reranker contributes significantly to answer quality.

Evidence Used

Improvement in strict vital-nugget recall from 63.8% (naive BM25 baseline) to 70.3% (full corrective Agentic GraphRAG scoring).
Removing the cross-encoder reranker reduces strict vital recall by 32.8%.

Methods / Approach

Deployment of APS-RAG platform.
Creation of APS-Bench question-answering dataset.
Use of a retrieval engine that fuses multiple channels and query-type-adaptive reciprocal-rank fusion.
Inclusion of a corrective agentic loop.
Use of a native-tool ReAct executor over Model Context Protocol (MCP) tooling layer.

Limitations Stated by the Authors

Performance gains from the graph channel and corrective loop are marginal.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The text does not provide information on potential limitations of the platform in terms of scalability, robustness, or generalizability to other types of facilities.

Open Questions

What is the specific impact of the graph channel and corrective loop on performance?
How does the performance of open-source and closed-source LLMs compare in various aspects beyond final answer synthesis?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The text emphasizes the improvements made by APS-RAG over a baseline, but it doesn't specify what the baseline is exactly.

Raw Abstract / Excerpt

Scientific user facilities accumulate decades of operational knowledge that no single search index covers: electronic logbooks, technical documents, internal wikis, operations chat messages, maintenance records, and live control-system data. We present APS-RAG, Advanced Photon Source Retrieval Augmented Generation, a deployed platform that makes the institutional knowledge at the Advanced Photon Source (APS) accessible to staff through natural-language queries, along with an operations-grounded evaluation. The retrieval engine fuses dense, sparse, and knowledge-graph (KG) channels with query-type-adaptive reciprocal-rank fusion, adds a corrective agentic loop, and runs a native-tool ReAct executor over a Model Context Protocol (MCP) tooling layer. We construct APS-Bench, a 50-question, question-answering (QA) dataset with auditable gold answers. Every retrieval-augmented variant numerically improves strict vital-nugget recall over a naive BM25 baseline (63.8%), with the full corrective Agentic GraphRAG scoring (70.3%). The cross-encoder reranker contributes significantly to answer quality: removing it and allowing the LLM to score relevance drastically reduces strict vital recall by 32.8%. The graph channel and corrective loop contribute positively as expected, but the performance gains are marginal. Additionally, we also compare the performance of open-source and closed-source LLMs in final answer synthesis. We release the APS-Bench construction methodology, the six-layer evaluation harness, and the underlying codebase, along with the '/aps-rag' retrieval agent skill framework, to support reproduction and adoption at other facilities. Together, the deployed platform and its operations-grounded evaluation present a promising workflow for trustworthy, statistically grounded AI assistance in facility operations, transferable to other large scientific instruments.