Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

FinEvo-Bench: A Longitudinal Benchmark for Self-Evolving Agents in Professional Financial Workflows

Source

arXiv

arxiv.org/abs/2608.06144v1

Read the source

Source

Plain-Language Brief

Researchers introduce a new benchmark called FinEvo-Bench for testing self-evolving agents in various financial workflows. The benchmark consists of 120 real-world tasks across six domains, with each task having a professional procedure and a quality/compliance rubric. Four different self-evolving agent scaffolds were tested, with Letta achieving the highest score and fewest compliance issues, while Codex showed the largest improvement from self-evolution. The study also found that rubric feedback improves scores and reduces compliance issues compared to reference-answer feedback.

What the Document Says

  • FinEvo-Bench is a new longitudinal benchmark for testing self-evolving agents in professional financial workflows.
  • The benchmark includes 120 tasks, each grounded in real cases and categorized across six financial domains.
  • Each task has a set of operations and constraints defined by professional procedures provided by institutions.
  • There are 20 business scenes with six related but distinct cases per scene, sharing the same professional procedure and quality/compliance rubric.
  • Four self-evolving agent scaffolds were tested using the same Qwen3.7-Max backbone.
  • Each scaffold's performance was compared to non-evolving controls, and an independent Claude Code scoring agent evaluated all outputs.
  • Letta achieved the highest evolved score (91.65) and fewest compliance issues (0.09 per task).
  • Codex showed the largest self-evolution gain (+19.37).
  • Across scaffolds, the evolving condition raised scores by 9.33-19.37 points and reduced compliance issues by 0.12-0.44 per task.
  • In Claude Code, skill-only evolution produced higher quality tasks and fewer compliance issues than memory-only and combined memory-skill evolution.
  • Across all four scaffolds, rubric feedback led to higher scores and fewer compliance issues compared to reference-answer feedback.

Key Concepts Mentioned

  • FinEvo-Bench
  • self-evolving agents
  • financial workflows
  • real-world tasks
  • professional procedures
  • quality/compliance rubrics
  • scaffolds
  • Qwen3.7-Max backbone
  • Letta
  • Codex
  • Claude Code
  • skill-only evolution
  • memory-only evolution
  • combined memory-skill evolution
  • rubric feedback
  • reference-answer feedback

Claims Made by the Document

  • FinEvo-Bench measures both professional performance and self-evolution ability.
  • Letta achieved the highest evolved score and fewest compliance issues among the tested scaffolds.

Evidence Used

  • The text mentions that FinEvo-Bench is a benchmark for measuring self-evolution ability and professional performance.
  • The text states that Letta achieved the highest evolved score (91.65) and fewest compliance issues (0.09 per task).

Methods / Approach

  • testing self-evolving agent scaffolds
  • comparing their performances using non-evolving controls
  • evaluating all outputs with an independent Claude Code scoring agent

Limitations Stated by the Authors

  • UNKNOWN: The paper does not explicitly state any limitations by the authors.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • UNKNOWN: The paper does not provide information about potential limitations of the research assistant in this context.

Open Questions

  • What other self-evolving agent scaffolds were tested besides the four mentioned?
  • Were there any differences in performance between the different financial domains in FinEvo-Bench?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper focuses on four specific self-evolving agent scaffolds, but it's unclear if other scaffolds were also tested.
  • The study tests the agents across six financial domains, but it does not provide information about any differences in performance between these domains.

Raw Abstract / Excerpt

Most agent benchmarks evaluate tasks independently and cannot measure whether experience from one task helps with later tasks. Existing self-evolution benchmarks do not jointly cover professional workflows, open-ended deliverables, and multi-aspect evaluation. We introduce FinEvo-Bench, a longitudinal benchmark with 120 real-case-grounded tasks, 20 business scenes across six financial domains. Institution-provided professional procedures define the required operations and constraints. Eligible institution-provided and publicly documented cases supply the task facts. Each scene contains six related but substantively distinct cases that share a professional procedure and a manually reviewed rubric for task quality and financial compliance. We compare four self-evolving agent scaffolds using the same Qwen3.7-Max backbone and three independently shuffled, globally interleaved task streams. Paired non-evolving controls estimate each scaffold's self-evolution gain from retained experience, while an independent Claude Code scoring agent backed by Claude Opus 4.6 evaluates all outputs. Letta achieves the highest evolved score (91.65) and fewest compliance issues (0.09 per task); Codex achieves the largest self-evolution gain (+19.37). Across scaffolds, the evolving condition raises scores by 9.33-19.37 points and reduces compliance issues by 0.12-0.44 per task. Paired score gains at within-scene ranks 4-6 exceed those at ranks 1-3 by 6.10-8.70 points. In Claude Code, skill-only evolution produces higher task quality and fewer compliance issues than memory-only and combined memory-skill evolution. Across all four scaffolds, rubric feedback also yields higher scores and fewer compliance issues than reference-answer feedback. FinEvo-Bench measures both professional performance and self-evolution ability: how effectively an agent turns prior experience into later improvement.