Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility

Source

arXiv

arxiv.org/abs/2607.24663v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Rajat Sainju, Dariusz Jarosz, Hairong Shang, Michael Prince, Ryan M. Aydelott, Mathew J. Cherukara, Yine Sun, Michael D. Borland
  • Venue: arXiv: physics.acc-ph
  • Published: 2026-07-27
  • Collected: 2026-07-28
  • URL: http://arxiv.org/abs/2607.24663v1
  • PDF: https://arxiv.org/pdf/2607.24663v1
  • DOI: UNKNOWN

Plain-Language Brief

Researchers have developed APS-RAG, a platform that helps scientists at the Advanced Photon Source (APS) access institutional knowledge through natural language queries. The platform uses a retrieval engine that fuses multiple channels and adds a corrective agentic loop. The researchers created a dataset of 50 questions with gold answers to evaluate the system's performance, showing improvement over a naive baseline. The cross-encoder reranker contributes significantly to answer quality. The authors also compare different LLMs in final answer synthesis and release their methodology, evaluation harness, codebase, and APS-RAG retrieval agent skill framework for reproducibility and adoption at other facilities.

What the Document Says

  • APS-RAG is a platform that makes institutional knowledge at the Advanced Photon Source accessible through natural language queries.
  • The retrieval engine fuses dense, sparse, and knowledge-graph channels with query-type-adaptive reciprocal-rank fusion.
  • The retrieval engine adds a corrective agentic loop and runs a native-tool ReAct executor over a Model Context Protocol tooling layer.
  • The authors constructed APS-Bench, a 50-question question-answering dataset with auditable gold answers.
  • Every retrieval-augmented variant improves strict vital-nugget recall over a naive BM25 baseline (63.8%), with the full corrective Agentic GraphRAG scoring achieving 70.3%.
  • The cross-encoder reranker contributes significantly to answer quality.
  • Removing the cross-encoder reranker and allowing the LLM to score relevance drastically reduces strict vital recall by 32.8%.

Key Concepts Mentioned

  • APS-RAG
  • Advanced Photon Source
  • retrieval engine
  • dense channels
  • sparse channels
  • knowledge-graph channels
  • query-type-adaptive reciprocal-rank fusion
  • corrective agentic loop
  • ReAct executor
  • Model Context Protocol
  • APS-Bench
  • question-answering dataset
  • gold answers
  • strict vital-nugget recall
  • BM25 baseline
  • cross-encoder reranker
  • LLM

Claims Made by the Document

  • APS-RAG improves recall over a naive BM25 baseline.
  • The cross-encoder reranker contributes significantly to answer quality.

Evidence Used

  • Every retrieval-augmented variant numerically improves strict vital-nugget recall over a naive BM25 baseline (63.8%), with the full corrective Agentic GraphRAG scoring achieving 70.3%.
  • Removing the cross-encoder reranker and allowing the LLM to score relevance drastically reduces strict vital recall by 32.8%.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Scientific user facilities accumulate decades of operational knowledge that no single search index covers: electronic logbooks, technical documents, internal wikis, operations chat messages, maintenance records, and live control-system data. We present APS-RAG, Advanced Photon Source Retrieval Augmented Generation, a deployed platform that makes the institutional knowledge at the Advanced Photon Source (APS) accessible to staff through natural-language queries, along with an operations-grounded evaluation. The retrieval engine fuses dense, sparse, and knowledge-graph (KG) channels with query-type-adaptive reciprocal-rank fusion, adds a corrective agentic loop, and runs a native-tool ReAct executor over a Model Context Protocol (MCP) tooling layer. We construct APS-Bench, a 50-question, question-answering (QA) dataset with auditable gold answers. Every retrieval-augmented variant numerically improves strict vital-nugget recall over a naive BM25 baseline (63.8%), with the full corrective Agentic GraphRAG scoring (70.3%). The cross-encoder reranker contributes significantly to answer quality: removing it and allowing the LLM to score relevance drastically reduces strict vital recall by 32.8%. The graph channel and corrective loop contribute positively as expected, but the performance gains are marginal. Additionally, we also compare the performance of open-source and closed-source LLMs in final answer synthesis. We release the APS-Bench construction methodology, the six-layer evaluation harness, and the underlying codebase, along with the '/aps-rag' retrieval agent skill framework, to support reproduction and adoption at other facilities. Together, the deployed platform and its operations-grounded evaluation present a promising workflow for trustworthy, statistically grounded AI assistance in facility operations, transferable to other large scientific instruments.