Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

RAGAL: A Frugal, Fully Local Retrieval-Augmented Assistant for Technical Support at a Government Agency

Source

arXiv

arxiv.org/abs/2607.18756v1

Read the source

Source

Plain-Language Brief

The paper introduces RAGAL, a locally-hosted AI assistant developed for a Romanian agency's technical support team. The system was designed to function within strict data locality constraints, with no external API calls, no writing permissions, and using only an 8GB laptop for development and training. The authors report improvements in the system's performance, particularly in retrieval engineering and retriever fine-tuning, after training on a corpus of about 25,000 Romanian-language support tickets and internal documents.

What the Document Says

  • The AI assistant is named RAGAL and is designed for the technical support team at AFIR, a Romanian agency.
  • The system operates under three constraints: no external API calls, read-only mandate, and development/training on a single 8GB laptop.
  • The corpus used for training consisted of approximately 25,000 chunks of Romanian-language support tickets and internal documents.
  • Hybrid dense-sparse retrieval with intent routing improved the system's internal evaluation from 62% to 81%.
  • Fine-tuning the bge-m3 embedder on real ticket data improved recall@10 from 0.663 to 0.850 and MRR from 0.489 to 0.684 after 72 minutes of training.
  • Single-domain fine-tuning initially degraded retrieval performance, a problem detected only after building a per-domain evaluation set and repaired with locally generated queries (GenQ).
  • Two counter-intuitive findings were reported: PII masking improved generation quality, and a structural 'anchor distillation' scheme made SQL hallucination impossible by construction.
  • A 744B-parameter model was used as a second opinion, but due to its slow CPU speed, it can only be used in overnight batches.
  • The authors release the sanitized pipeline scripts for institutions with similar data locality constraints.

Key Concepts Mentioned

  • AI assistant
  • RAGAL
  • AFIR
  • data locality
  • corpus
  • hybrid dense-sparse retrieval
  • intent routing
  • bge-m3 embedder
  • recall@10
  • MRR
  • single-domain fine-tuning
  • PII masking
  • SQL hallucination
  • 744B-parameter model

Claims Made by the Document

  • The system's performance improved after training.
  • Single-domain fine-tuning initially degraded retrieval performance.
  • PII masking improved generation quality.
  • A structural 'anchor distillation' scheme made SQL hallucination impossible.

Evidence Used

  • Improvements in internal evaluation from 62% to 81%
  • Improved recall@10 from 0.663 to 0.850 and MRR from 0.489 to 0.684
  • Problem detected and repaired after building a per-domain evaluation set
  • Counter-intuitive findings reported

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Public institutions hold large volumes of sensitive documents and support tickets that cannot leave the premises, ruling out cloud-hosted language models entirely. We report on RAGAL, a retrieval-augmented assistant for the technical-support team of AFIR, the Romanian Agency for Financing Rural Investments, built and operated under three hard constraints: zero data egress (no external API calls, even for synthetic data), a read-only mandate (the assistant drafts, humans execute), and a single 8 GB consumer laptop as the only development and training machine. Over a Romanian-language corpus of ~25,000 chunks -- 15,073 resolved support tickets and internal normative documents -- we show that the highest-leverage investments were retrieval engineering and retriever fine-tuning rather than a larger generator: hybrid dense-sparse retrieval with intent routing raised our internal evaluation from 62% to 81%, and fine-tuning the bge-m3 embedder on real ticket data improved recall@10 from 0.663 to 0.850 (MRR 0.489 to 0.684) after 72 minutes of training. We document a general pitfall: single-domain fine-tuning silently degraded retrieval on the untouched document domain below the stock baseline, detected only after building a per-domain evaluation set and repaired with locally generated queries (GenQ). We report two counter-intuitive findings -- PII masking improved generation quality, and a structural 'anchor distillation' scheme made SQL hallucination impossible by construction -- along with a reproducible recipe for full embedder fine-tuning in 8 GB of VRAM. Finally, since zero egress also rules out a cloud judge, we describe a substitute: a 744B-parameter model run on CPU, too slow to serve interactively but affordable in overnight batch, used as a second opinion whose limits we quantify. We release the sanitized pipeline scripts for institutions facing similar data-locality constraints.