V-FiLLM: Verified Financial LLM Reasoning Benchmark

Source

Source name: arXiv
Source kind: paper
Authors / organization: Alicia Larsen, Victoire Laurent, Aulia Kharis Rakhamsari, Lara Turgut, Nino Antulov-Fantulin
Venue: arXiv: cs.AI
Published: 2026-08-11
Collected: 2026-08-12
URL: http://arxiv.org/abs/2608.11047v1
PDF: https://arxiv.org/pdf/2608.11047v1
DOI: UNKNOWN
Registry group: academic_apis_and_preprints
Evidence role: canonical_academic_index
Language: en
Country / region: UNKNOWN

Plain-Language Brief

Researchers have created a new framework called V-FiLLM to evaluate the abilities of Language Models (LLMs) in financial reasoning using structured data. The framework generates questions from executable computation trees and natural-language questions, providing answers that are correct by design. The study finds that as reasoning depth increases, accuracy can drop up to 51%, and under adversarial numerical perturbations, it can drop up to 47%. However, using a lightweight LoRA fine-tuning method improves accuracy on held-out problems by about 4.5% and outperforms the base model on FinQA by 5% points.

What the Document Says

V-FiLLM is a new framework for evaluating LLMs in financial reasoning.
The framework generates questions from executable computation trees and natural-language questions.
Answers are correct by design, removing the need for annotation cost or inheriting a generator's error rate.
Four controllable axes of difficulty are exposed: computation depth, expression breadth, financial concept complexity, and context size.
Evaluating on open-source models shows accuracy can drop up to 51% as reasoning depth increases and up to 47% under adversarial numerical perturbations.
LoRA fine-tuning improves accuracy from 81.1% to 85.6% on held-out problems.
The method outperforms the base model by 5% points on FinQA.

Key Concepts Mentioned

Language Models (LLMs)
V-FiLLM
executable computation trees
natural-language questions
financial reasoning
accuracy
robustness
LoRA fine-tuning
FinQA

Claims Made by the Document

V-FiLLM can generate financial reasoning benchmarks.
Accuracy can drop as reasoning depth increases and under adversarial numerical perturbations.
LoRA fine-tuning improves accuracy on held-out problems and outperforms the base model on FinQA.

Evidence Used

The text mentions that V-FiLLM generates questions from computation trees and natural language, providing answers that are correct by design.
The text states that accuracy drops as reasoning depth increases and under adversarial numerical perturbations based on the evaluation of open-source models.
The text mentions that LoRA fine-tuning improves accuracy on held-out problems and outperforms the base model on FinQA.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.