Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian

Source

arXiv

arxiv.org/abs/2606.31718v1

Read the source

Source

Plain-Language Brief

Researchers investigated cross-lingual relation extraction for Romanian using large language models (LLMs) and an English to Romanian translation pipeline. They evaluated a 31B LLM under different configurations against four encoder baselines with sizes ranging from 125M to 560M parameters. The results showed that the performance of the LLM on Romanian was relatively lower than English by around 3-5 percentage points in prompt-only settings. Few-shot prompting provided only minor improvements over zero-shot, and QLoRA fine-tuning significantly improved macro F1-Score in both languages while reducing the cross-lingual gap. Despite the larger size of the LLM, monolingual Romanian BERT at 125M parameters performed closely to multilingual XLM-R at 278M. The study concluded that for deployment scenarios where compute is a factor, using a 31B model may not be necessary for single-task RE on Romanian.

What the Document Says

  • Researchers used LLMs and an English to Romanian translation pipeline for cross-lingual relation extraction.
  • They evaluated a 31B LLM and four encoder baselines with sizes ranging from 125M to 560M parameters.
  • The LLM showed a performance drop of around 3-5 percentage points on Romanian compared to English in prompt-only settings.
  • Few-shot prompting provided minor improvements over zero-shot, and QLoRA fine-tuning significantly improved macro F1-Score.
  • QLoRA reduced the cross-lingual gap from 3.3 to 1.4 percentage points.
  • Monolingual Romanian BERT at 125M parameters performed closely to multilingual XLM-R at 278M.
  • Using a 31B model may not be necessary for single-task RE on Romanian in deployment scenarios where compute is a factor.

Key Concepts Mentioned

  • relation extraction
  • large language models
  • cross-lingual
  • Romanian
  • evaluation
  • fine-tuning
  • QLoRA

Claims Made by the Document

  • The performance of the LLM on Romanian is lower than English by around 3-5 percentage points.
  • Few-shot prompting provides only minor improvements over zero-shot.
  • QLoRA fine-tuning significantly improves macro F1-Score in both languages.

Evidence Used

  • Relation classification with marked entities and end-to-end extraction were assessed.

Methods / Approach

  • LLM-based translation pipeline
  • zero-shot, few-shot, and QLoRA fine-tuned configurations
  • four encoder baselines: XLM-RoBERTa, Romanian BERT, and RoBERTa-large

Limitations Stated by the Authors

  • The study focused on relation extraction, but other tasks may exhibit different results.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The text does not mention any limitations due to the agent used in the research.

Open Questions

  • What are the potential reasons for the performance drop of the LLM on Romanian compared to English?
  • How do other tasks perform under cross-lingual RE for Romanian using LLMs?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text does not provide details about the specific relation extraction task or the evaluation metrics used.

Raw Abstract / Excerpt

Relation extraction (RE) for low-resource languages is typically constrained by the lack of annotated corpora. We investigate the feasibility of cross-lingual RE for Romanian by combining automatic dataset translation with large language model (LLM) inference. We translate the SemEval-2010 Task 8 benchmark from English to Romanian using an LLM-based translation pipeline and evaluate Gemma 4 31B under zero-shot, few-shot, and QLoRA fine-tuned configurations, against four encoder baselines spanning 125M to 560M parameters: XLM- RoBERTa (base and large), Romanian BERT, and RoBERT- large. We assess two task formulations: relation classification with marked entities and end-to-end extraction. Our results show that Romanian incurs a 3 to 5 percentage point (pp) drop relative to English in prompt-only settings, that few-shot prompting provides marginal gains over zero-shot, and that QLoRA fine-tuning improves macro F1-Score by more than 22 percentage points in both languages while reducing the cross-lingual gap from 3.3 to 1.4pp. The encoder baselines come within 1-4pp of QLoRA Gemma on Romanian despite being 50-250 times smaller, with monolingual Romanian BERT at 125M parameters matching multilingual XLM-R at 278M. The case for using a 31B model for single-task RE on Romanian is therefore weak in deployment scenarios where compute matters. We release the translated dataset, evaluation code, and trained models.