Evaluating RAG for French immigration law: a benchmark and baseline study

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Annia Abtout, Julien Delaunay, Monika Ewa Rakoczy
• Venue: arXiv: cs.IR
• Published: 2026-07-27
• Collected: 2026-07-28
• URL: http://arxiv.org/abs/2607.24449v1
• PDF: https://arxiv.org/pdf/2607.24449v1
• DOI: UNKNOWN

Plain-Language Brief

The paper presents a new benchmark for evaluating AI models used in French immigration law, focusing on permit-type recommendation, required document retrieval, and legal citation coverage. The study compares a parametric Language Model (LLM) baseline with dense retrieval augmentation at two scales (Qwen3.5-9B and -27B). Results show that retrieval improves administrative guidance, particularly for permit-type accuracy.

What the Document Says

• The study is about a benchmark for AI models in French immigration law
• The benchmark includes permit-type recommendation, required-document retrieval, and legal citation coverage
• A parametric Language Model (LLM) baseline is compared with dense retrieval augmentation at two scales: Qwen3.5-9B and -27B
• Results show that retrieval improves administrative guidance, particularly for permit-type accuracy
• The study concludes that retrieval grounding is important for more reliable administrative guidance in this domain and motivates further investigation of hybrid retrieval strategies.

Key Concepts Mentioned

• AI models
• French immigration law
• benchmark
• parametric Language Model (LLM)
• dense retrieval augmentation
• permit-type recommendation
• required document retrieval
• legal citation coverage
• administrative guidance

Claims Made by the Document

• Retrieval improves administrative guidance, particularly for permit-type accuracy.

Evidence Used

• Results from comparing a parametric LLM baseline against dense retrieval augmentation at two model scales (Qwen3.5-9B and -27B) on 52 annotated synthetic profiles.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

International recruitment in France requires navigating a layered legal framework absent from existing legal AI benchmarks. We present a publicly available benchmark and first comparative evaluation for this domain, covering permit-type recommendation, required-document retrieval, and legal citation coverage. Comparing a parametric LLM baseline against dense retrieval augmentation at two model scales (Qwen3.5-9B and -27B) on 52 annotated synthetic profiles, we find that retrieval improves administrative guidance at both scales, most notably permit-type accuracy. Our results confirm that retrieval grounding is important for more reliable administrative guidance in this domain, and motivate further investigation of hybrid retrieval strategies.