Evaluating RAG for French immigration law: a benchmark and baseline study

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Annia Abtout, Julien Delaunay, Monika Ewa Rakoczy
• Venue: arXiv: cs.IR
• Published: 2026-07-27
• Collected: 2026-07-28
• URL: http://arxiv.org/abs/2607.24449v1
• PDF: https://arxiv.org/pdf/2607.24449v1
• DOI: UNKNOWN

Plain-Language Brief

The authors present a new benchmark for evaluating AI models in the context of French immigration law. They compare a parameterized language model baseline with dense retrieval augmentation at two scales (Qwen3.5-9B and -27B) on 52 synthetic profiles. The results suggest that retrieval improves administrative guidance, particularly in permit-type accuracy.

What the Document Says

• The study is about evaluating AI models for French immigration law using a new benchmark.
• They compare a parametric language model baseline with dense retrieval augmentation at two scales (Qwen3.5-9B and -27B).
• The evaluation is based on 52 annotated synthetic profiles.
• The study finds that retrieval improves administrative guidance, particularly in permit-type accuracy.

Key Concepts Mentioned

• French immigration law
• AI models
• benchmark
• parametric language model
• dense retrieval augmentation
• synthetic profiles

Claims Made by the Document

• Retrieval improves administrative guidance in French immigration law.

Evidence Used

• Improved permit-type accuracy with retrieval.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

• What is the specific method used for the parametric language model baseline?
• What are the details of the dense retrieval augmentation strategy?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The study does not provide information on the specific improvement in permit-type accuracy due to retrieval.
• It is unclear what 'administrative guidance' refers to in this context.

Raw Abstract / Excerpt

International recruitment in France requires navigating a layered legal framework absent from existing legal AI benchmarks. We present a publicly available benchmark and first comparative evaluation for this domain, covering permit-type recommendation, required-document retrieval, and legal citation coverage. Comparing a parametric LLM baseline against dense retrieval augmentation at two model scales (Qwen3.5-9B and -27B) on 52 annotated synthetic profiles, we find that retrieval improves administrative guidance at both scales, most notably permit-type accuracy. Our results confirm that retrieval grounding is important for more reliable administrative guidance in this domain, and motivate further investigation of hybrid retrieval strategies.