Natural language processing
142 notes, newest first. Includes the vault tags natural language processing, nlp, question answering.
ContractScrub: A benchmark for final review of legal contracts
Researchers introduce ContractScrub, a new benchmark to evaluate the performance of language models in contract scrubbing, which is the final review of legal contracts for errors and inconsistencies. The study finds…
ContractScrub: A benchmark for final review of legal contracts
The paper introduces ContractScrub, a new benchmark for evaluating the performance of Language Models (LLMs) in reviewing legal contracts for errors. The study finds that while LLMs perform well on general benchmarks…
Interrupting the Loop: Periodic Subject Changes Raise Judged Surprise and Connection in Base Language Models
The research examines where novelty comes from in base language models and what they can perceive in a long text stream. The study dismantles a cognitively inspired generation loop across 24 conditions on three base…
MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
Researchers have introduced a new benchmark called MemTrapBench to test the performance of large language models (LLMs) in terms of cognitive traps caused by their memory components. The study identifies two types of…
Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
Researchers have developed a method called Task-CoEvolve to improve the efficiency of harness optimization for large language models (LLMs). This method co-evolves validation tasks with the harness by focusing on tasks…
Adaptive Memory and Reflection Multi-Agent System for Medical Question Answering
Researchers introduce a new multi-agent system called Adaptive Memory and Reflection (AMR) for medical question answering. This system aims to address the limitations of existing systems by using specialized agents…
Identifying Implicit Premises for Logical Reconstruction of Argument Graphs
The authors propose a neuro-symbolic pipeline to generate implicit premises from natural language text, which can be translated into logical formulae to show the logical relationships between statements (entailment…
Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models
The study explores the ability of large language models (LLMs) to extract contextualized information from research articles efficiently. The authors present four workflows that demonstrate LLMs can perform well in data…
Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models
The paper investigates the performance of large language models (LLMs) in extracting contextualized information from research articles, finding that while LLMs can perform well with expert-curated prompts, they may…
Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Models
The paper presents EvoResearcher, a training-free protocol for adding cost-bounded self-reflection to large language models (LLMs). The protocol is designed to work with a single frozen LLM backbone and aims to improve…
ACL Anthology team
The ACL Anthology is a collection of papers related to natural language processing, published by the ACL Anthology team. Papers from 2016 onwards are licensed under Creative Commons Attribution 4.0 International…
Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams
The paper discusses Diagram-MMU, a new benchmark designed to evaluate the ability of Multimodal Large Language Models (MLLMs) in parsing and understanding scientific diagrams. The benchmark includes 3.7k diagrams and…
GitSkills: A Dataset of Agent Skills on GitHub
Researchers present GitSkills, a dataset containing over 3 million skill files for language-model agents found in public GitHub repositories in July 2026. The dataset retains every file occurrence with its repository…
Multiclass Sentiment Analysis for Identifying Political Viewpoints
Researchers developed two machine-learning models to analyze political viewpoints from social media posts. The XGBoost model achieved an F1-score of 0.2835, and the BERT-based model achieved an F1-score of 0.2806 in a…
VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?
The paper introduces VibeLifeBench, a new benchmark for evaluating large language model (LLM) agents' ability to handle long-horizon tasks in everyday life scenarios. The benchmark consists of 200 multi-week timeline…
Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Researchers have created a new method for evaluating text-to-speech (TTS) systems by breaking down the concept of 'naturalness' into ten distinct linguistic dimensions. They found that current TTS evaluation methods do…
CARD: Controlled Agentic Reddit Discussions for Credit Card Simulation
The authors introduce a framework called CARD for generating realistic credit card discussion threads on online platforms, such as Reddit. The framework takes into account various factors like reply structure, comment…
Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
Researchers propose a new method for fine-tuning large language models in a federated learning paradigm using Low-rank adaptation (LoRA). They investigate two strategies, Share-A/Local-B and Share-B/Local-A, and reveal…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Researchers introduce a framework to evaluate the quality of benchmarks used for task-oriented conversational agents, which can lead to unreliable evaluations due to inconsistencies, simplistic scenarios, or limited…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Researchers have introduced a new framework to evaluate the quality of benchmarks used for task-oriented conversational agents. The framework uses language models (LLMs) to assess the consistency, complexity, and…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
The authors propose a reference-free framework to evaluate the quality of task-oriented conversational agent benchmarks, which are often used to assess these agents. The framework uses Large Language Models (LLMs) as…
Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations
A research paper proposes a new method for document search called Reliable Embedding-free Agentic Document-search (READ), which is designed to overcome limitations in current methods for long documents, particularly…
Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations
The authors propose a new method for document search called READ (Reliable Embedding-free Agentic Document-search). This method uses an interpretable agent that performs three operations: normalized lexical search…
Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations
The paper introduces a new approach called READ (Reliable Embedding-free Agentic Document-search) for retrieving information from long documents such as financial statements and audit reports. The traditional method of…
Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations
The paper proposes a new approach for document search called 'READ (Reliable Embedding-free Agentic Document-search)', which aims to address the issues with current black-box retrieval designs in handling long…
Beyond Top-K: Replacing Black-Box Retrieval with Interpretable Agentic Operations
The authors propose a new method for document search called Reliable Embedding-free Agentic Document-search (READ). In contrast to the common design of retrieval-augmented generation, READ uses an agent that reads the…
Bias Analysis of L2 Speaking Assessment Systems Using Concept Activation Vectors
Researchers used Concept Activation Vectors (CAVs) to analyze the bias of two automatic L2 speaking assessment systems: a text-based BERT grader and a multimodal grader based on Whisper. The study shows that the…
Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI
Researchers have developed a new method for extracting complex, structured information from unstructured text documents using generative AI. The method involves creating a schema that serves as an information model…
Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI
Researchers have developed a framework using generative AI to extract complex, structured information from unstructured text documents. This framework also includes an automated semantic evaluation step to compare the…
Tytan: Interactive Neurosymbolic Construction of Analytic Semantic Schemas from Relational Data
TYTAN is a system that automatically constructs an analytic semantic schema from a relational database, with user-provided descriptions when available. It uses symbolic analysis and LLM-based semantic inference for…
Hierarchical Graph Memory for LLM Agents with Path-level Localization and Rewrite
The authors propose HiGram, a hierarchical graph memory framework for long-term reasoning agents. This framework aims to efficiently update memories over time by organizing them into a coarse-to-fine architecture…
OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
The research presents a new approach called OctoLong for creating longer contexts for language models. It uses an AST parser, a language server backend, and a package manager to retrieve code references from various…
OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
In this research, a new pipeline called OctoLong is introduced to enhance the training of language models (LM) by providing them with long code contexts. This is done through the use of an AST parser, a language server…
OPD-V: Visual On-Policy Self-Distillation with Modality Balance
The paper introduces a new method called On-Policy Self-Distillation with Visual Modality Balance (OPD-V) to improve the visual reasoning of multimodal large language models. The authors argue that previous methods…
The Transformer Revolution, Part 1: Dynamic Processing through Output- Weight Interconnections
The paper proposes a new interpretation of Transformers during inference, suggesting they construct prompt-dependent transformations whose parameters are generated during inference. This form of computation is named…
VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs
The text introduces VIBE, a new benchmark for analyzing the emotional tone of language models' outputs. The benchmark measures valence (positive or negative), arousal (exciting or calm), and dominance (powerful or…
EMBL AI Librarian: Life-Sciences Knowledge Layer for AI Agents
The paper introduces EMBL AI Librarian, a system that enhances the Europe PMC interface for AI agents to enable them to search and retrieve evidence in natural language. The system was evaluated across four benchmarks…
ORCA-bench: How Ready Are Language Model Agents for Oncall?
The study presents ORCA-bench, a benchmark for evaluating the performance of large language models in oncall root cause analysis (RCA). The benchmark exposes a microservice system with six days of data and 1,079 RCA…
A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility
Researchers have developed a platform called APS-RAG for scientific facilities like the Advanced Photon Source (APS). The platform aims to make the vast amount of institutional knowledge accessible through natural…
A corrective agentic hybrid RAG and an operations-grounded evaluation for a scientific facility
Researchers have developed APS-RAG, a platform that helps scientists at the Advanced Photon Source (APS) access institutional knowledge through natural language queries. The platform uses a retrieval engine that fuses…
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
The paper proposes a framework called DataOrchestra that learns to curate pre-training data example-by-example rather than applying a uniform strategy across many examples. The framework decides whether to drop, clean…
Making Mathematical Knowledge Explainable, Accessible and Interoperable Through Large Language Model Integration
The authors propose integrating Large Language Models (LLMs) with a mathematical knowledge base called MathModDB to improve its accessibility, explainability, and interoperability. The integration is achieved through a…
The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages
The research quantifies the disadvantage faced by many non-English languages in large language models (LLMs) due to subword tokenizers that are primarily trained on English corpora. The study focuses on Indian…
Detecting LLM-Generated Tokens in Human--LLM Coauthored Text
The paper presents a new method for identifying parts of a text generated by Language Model (LLM) at the token level, improving on existing document-level methods. The method is based on smoothing adjacent token scores…
GRADRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG
The paper introduces GRADRAG, a framework for improving Retrieval-Augmented Generation (RAG) systems by coordinating improvements across different components of the pipeline. It uses feedback from an Evaluator to…
GRADRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG
The paper introduces GRADRAG, a framework that improves Retrieval-Augmented Generation (RAG) systems by coordinating improvements across the pipeline. Instead of optimizing components in isolation, GRADRAG models the…
GRADRAG: Cross-Component Prompt Adaptation for Coordinated Multi-Agent RAG
The authors introduce GRADRAG, a framework for improving Retrieval-Augmented Generation (RAG) systems by coordinating improvements across multiple agents in the pipeline, rather than optimizing components in isolation…
GS-Agent: Creating 4D Physical Worlds With Generative Simulation
Researchers have developed GS-Agent, a system that uses natural language descriptions to automatically generate realistic, dynamic, and controllable 4D physical worlds. This is achieved through an end-to-end…
GS-Agent: Creating 4D Physical Worlds With Generative Simulation
Researchers have developed GS-Agent, a system that generates realistic, dynamic, and controllable 4D physical worlds from natural language descriptions using an end-to-end multi-agent framework. The system emulates…
Thinkink: 2D Spatial Ink-native Interaction with LLMs
The researchers propose Thinkink, a tool that allows users to interact with large language models (LLMs) using handwritten text or sketches on a shared canvas. The tool uses a semantic tree for ink interpretation and a…
Co-Evolving LLM Evaluators and Policies via DynamicRubric
The paper introduces DynamicRubric, a framework that co-evolves language model evaluators and policies by generating weighted binary rubrics for each candidate set and aggregating the resulting judgments into…
The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
The paper addresses two open questions regarding a method called PortLLM, which adapts large language models after continual pretraining. The authors perform an extensive empirical study to explore the long-term…
AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System
The paper introduces AILQA, an AI system designed for Indian legal question answering, using a variety of embedding and generative models including LLMs. It was evaluated with both lexical and semantic metrics, expert…
Disentangling Curriculum Learning in NLP: Towards a Unifying Taxonomy
The paper proposes a taxonomy to better understand curriculum learning (CL) in Natural Language Processing (NLP). The taxonomy aims to provide a systematic analysis of CL strategies by separating difficulty evaluation…
Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA
The paper presents a diagnostic framework called FiT (Find before Fine-Tune) to evaluate the suitability of small language models for cybersecurity question-answering tasks. The study finds that fine-tuning these…
Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing
The authors propose a method to improve the accuracy of automatic speech recognition (ASR) models by controlling the activation of 'verbatim' versus 'intended' transcription styles. They argue that current models…
Can We Break LLMs Out of Self-Loops? Fine-Grained Reasoning Control with Activation Steering
Researchers propose a new method called SOPHIA for controlling the reasoning process of large language models, which are currently uncontrollable despite their extended use. The proposed method involves treating each…
HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization
The paper proposes a new method called Highlight-guided Attention Steering (HAS) for video summarization using Multimodal Large Language Models (M-LLMs). HAS aims to enhance the understanding of videos by focusing…
Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs
A study investigates the impact of a structured evidence-sufficiency prompt on reducing overconfident answers from four language models in a clinical context. The study finds a significant reduction in unsafe…
MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models
The paper introduces MADA-RL, a method for fine-tuning compact language models with limited budgets. It specializes the models into generator and critic roles and trains them using a debate-aware learning signal. The…
SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs
Researchers present SelectInfer, a neuron-level optimization framework for Large Language Models (LLMs) that can run efficiently on resource-constrained edge devices. It reduces memory footprint and computation by…
Stress Testing Concept Erasure with Large Language Model Agents
The paper introduces a new framework called Stress Testing Agents for Concept Erasure (STACE). STACE uses multiple Large Language Model (LLM) agents to stress-test models that have had certain concepts erased from…
AutoSynthesis: An agentic system for automated meta-analysis
AutoSynthesis is a multi-agent system designed for automated meta-analysis of research questions in natural language. The system searches for scientific literature, assesses eligibility, extracts quantitative…
Long-Context Fine-Tuning with Limited VRAM
Researchers introduce Hierarchical Global Attention (HGA) to reduce memory usage during long training sequences in large language models. They use segment-wise backpropagation, tiered KV storage, and detach older data…
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
Researchers have created a new benchmark called MM-IssueLoc to evaluate the effectiveness of multimodal repository-level issue localization, which includes visual evidence such as screenshots and logs. Current…
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
Researchers have introduced a new benchmark called MM-IssueLoc for evaluating the effectiveness of systems in repository-level issue localization that also includes visual evidence such as screenshots, error dialogs…
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
The paper introduces MM-IssueLoc, a benchmark and evaluation protocol for repository-level issue localization that includes visual evidence. The benchmark has 652 issue-PR instances across 23 languages with annotations…
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
The paper introduces MM-IssueLoc, a new benchmark and evaluation protocol for repository-level issue localization with visual evidence. The benchmark contains over 650 issue-PR instances across multiple languages…
SciDiagramEdit: Learning to Edit Scientific Diagrams from Paper Revisions
The text introduces SciDiagramEdit, a framework for automating the editing of scientific diagrams in research papers based on natural language instructions. The framework learns from natural paper revisions and…
SciDiagramEdit: Learning to Edit Scientific Diagrams from Paper Revisions
The authors introduce SciDiagramEdit, a system designed to automate the time-consuming process of editing scientific diagrams in research papers based on natural language instructions. The system learns from paper…
Show Me How You Reason and I'll Tell You Who You Are: Reasoning Graphs for Robust LLM Authorship Attribution
Researchers propose a graph neural network approach to detect and analyze reasoning structures in large language models (LLMs) generated texts for improved robustness against paraphrasing and obfuscation techniques…
Show Me How You Reason and I'll Tell You Who You Are: Reasoning Graphs for Robust LLM Authorship Attribution
The paper proposes a new method for detecting text generated by large language models (LLMs) and attributing authorship to them. The method goes beyond just analyzing surface-level linguistic features and instead…
teLLMe Why (Ain't Nothing but a Jam): Exploratory Causal Analysis of Urban Driving Data
Researchers have developed teLLMe, a system to analyze urban driving data for understanding traffic safety and congestion by answering causal questions. The system uses dashcam annotations to build a structured event…
Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs
Researchers propose a new method called Grounded Context Preference Optimization (Groc-PO) to improve the truthfulness and utility of Multimodal Large Language Models (MLLMs). Current MLLMs often make errors due to…
A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study
Researchers present Pythia, a multi-agent system that automatically generates and optimizes extraction prompts for clinical concepts without manual engineering or fine-tuning. They compare Pythia with a curated lexicon…
A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study
A new system called Pythia was developed, which automatically generates and optimizes extraction prompts for clinical concepts without the need for manual prompt engineering or fine-tuning. The researchers compared…
Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model
Researchers train a discrete diffusion language model to transcribe speech instead of using autoregressive decoders that emit one token at a time. The model is based on a mixture-of-experts called DiffusionGemma, and…
Resist and Update: Counterfactual Report Coordinates for Incentive-Compatible LLMs
The authors present a method for creating counterfactual report mediators, which help align language models by making their reports invariant to certain influences (like pressure or prestige) and responsive to others…
Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
The authors present a method for creating and refining evaluation metrics for self-improving language models (LLMs). The proposed method, called Double Ratchet, evolves metrics by combining small drawback detectors in…
Evidence-Backed Video Question Answering
The authors propose Evidence-Backed Video Question Answering (E-VQA), a new task that requires models to not only provide answers but also the specific spatial and temporal evidence from videos. They introduce…
StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Description
The authors propose a new framework called StoryTeller for long-form audio description (AD). Unlike modern video-language models, StoryTeller maintains a narrative memory that carries forward story-relevant information…
DocMaster: A Hierarchical Structure-Aware System for Document Analysis
DocMaster is a system that allows for the analysis of complex documents while preserving their hierarchical structures such as sections, tables, figures, and equations. It offers an interactive web interface where…
DocMaster: A Hierarchical Structure-Aware System for Document Analysis
DocMaster is a system designed to analyze complex documents while preserving their hierarchical structures such as sections, tables, figures, and equations. It allows users to upload document collections, construct…
SLORR: Simple and Efficient In-Training Low-Rank Regularization
The paper introduces SLORR, a simple and efficient in-training low-rank regularization framework for neural networks. It aims to improve compressibility without significant accuracy loss, while avoiding the limitations…
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
The paper discusses the issue of language models providing incorrect or unanswerable responses. The author proposes two separate axes for abstention: one for answers the model would get wrong, and another for questions…
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
A study finds that large language models often struggle with two types of errors: answering questions incorrectly and attempting to answer unanswerable or misleading questions. The researchers propose a method to…
UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing
UltraX is a new framework designed to improve the quality of large-scale pre-training data for Large Language Models (LLMs). It addresses the limitations of existing methods by offering fine-grained instance-level…
Validity of LLMs as data annotators: AMALIA on authority
A study tests the validity of a large language model (LLM), AMALIA, in measuring moral foundation concepts for European Portuguese. The study finds that while AMALIA agrees with human coders on these concepts, it does…
SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation
Researchers have developed a method called SynthAVE to generate large amounts of labeled data for e-commerce attribute extraction using synthetic labels validated by multiple large language models (LLMs). The benchmark…
When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs
The paper proposes ARGTCA, a method to improve the calibration of Vision-Language Models (VLMs) by using Graph Attention Networks to capture inter-attribute dependencies and reduce overconfidence during test-time…
Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
The paper introduces DataGovBench, a benchmark for evaluating the performance of Large Language Models (LLMs) in practical scenarios, such as complex data analysis and insight discovery from large multi-tabular…
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
The research introduces DepthWeave-KV, a new cache compression method for long-context language models. This method adapts to the needs of different tokens by factorizing key and value states across neighboring…
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
The researchers evaluate the performance of uncertainty estimation methods across 22 languages in high-, mid-, and low-resource settings using two human-curated question and answer datasets. They compare open and…
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
The paper introduces FreqDepthKV, an inference-time cache compression method for long-context LLM (Large Language Model) inference that reduces memory and bandwidth costs while maintaining task accuracy on various…
From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b
Researchers present a framework using question-type-specific large language models (LLMs) to improve answer robustness and evidence grounding in biomedical question answering. The framework uses different inference…
From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b
A study presents a framework for improving the accuracy and reliability of biomedical question answering by using a large language model (LLM) that adapts its inference strategies based on the type of question (yes/no…
From Voting to Agent Collaboration: Answer-Type-Aware LLM Pipelines for BioASQ 14b
Researchers have developed a system for answering biomedical questions using large language models (LLMs). The system is designed to improve the robustness of answers by selecting different inference procedures for…
Rethinking Indic AI from a Lens of Cultural Heritage Preservation
The paper 'Rethinking Indic AI from a Lens of Cultural Heritage Preservation' discusses how Artificial Intelligence (AI) impacts India's linguistic and cultural foundations, focusing on the challenges and opportunities…
RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
The author introduces RuBench 1.0, a benchmark for evaluating coding agents that use native language tasks instead of English. The benchmark includes 25 tasks mined from five open-source repositories and is judged by…
OptiAgent: End-to-End Optimization Modeling via Multi-Agent Iterative Refinement
The researchers have developed a multi-agent framework called OptiAgent that can take a natural language description of an Operations Research problem and produce a mathematically formulated solution along with…
SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models
A new benchmark called SPEARBench is introduced to evaluate the naturalness of speech-to-speech language models during question-answer interactions. The benchmark covers aspects like response latency, interruptions…
Neuron-Aware Data Selection for Annotation-Free LLM Self-Distillation
The paper proposes a new method called Neuron On-Policy Self-Distillation (Neuron-OPSD) for annotation-free self-distillation of large language models. This method uses the internal neuron activations to guide data…
ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning
Researchers propose a new method called Recursive Evidence Replay (RECONTEXT) to improve long-context reasoning in large language models without requiring additional training, memory, or context pruning. RECONTEXT…
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
The text introduces 'adversarial pragmatics', a new benchmark and annotation protocol to evaluate the behavior of language models in various ambiguous situations. The goal is to clarify whether failures occur due to…
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
The paper introduces Adversarial Pragmatics, a benchmark and annotation protocol for evaluating language models' behavior in situations of instruction conflict, embedded commands, and policy ambiguity. The benchmark is…
Bayesian Uncertainty Propagation for Agentic RAG Pipelines: A Proof-of-Concept Study on Multi-Hop Question Answering
This research paper presents an uncertainty-aware framework for Retrieval-Augmented Generation (RAG) systems. The system uses a Bayesian Network to estimate the uncertainty of multi-stage reasoning pipelines and…
From Personas to Plot: Character-Grounded Multi-Agent Story Generation for Long-Form Narratives
Researchers have developed MAGNET, a framework that generates consistent and coherent long-form narratives by using a unified approach of multi-agent goal-driven storytelling and verification. The system was tested…
Freeform Preference Learning for Robotic Manipulation
Researchers introduce Freeform Preference Learning (FPL), a method for training robotic policies based on natural-language preferences provided by humans, rather than binary labels. The method uses the preferences to…
JL1-CC&QA: Extending the JL1-CD Benchmark with Change Captioning and Question Answering
Researchers introduce JL1-CC&QA, a new benchmark for remote sensing that extends the JL1-CD dataset with two annotation layers: change captioning (CC) and change question answering (QA). The benchmark includes 5,000…
MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments
The research paper introduces MECoBench, a benchmark for evaluating the collaboration abilities of multimodal agents in visually grounded environments. The study explores three key findings: collaboration generally…
PolicyGuard: From Organizational Policies to Neuro-SymbolicCompliance Review Engines
The researchers have developed a framework called PolicyGuard to ensure documents comply with an organization's specific policies or guidelines. Instead of relying on large language models for end-to-end prompting…
TreeAgent: A Generalizable Multi-Agent Framework for Automated Bias Labeling in Forestry via Compiled Expert Rules and Vision-Language Models
Researchers have proposed a multi-agent system that uses expert decision trees and Vision-Language Models (VLMs) to automate the bias classification in tree height in forestry remote sensing. The system mitigates VLM…
When LLMs Read Tables Carelessly: Measuring and Reducing Data Referencing Errors
The research evaluates the issue of data referencing errors (DREs) made by large language models (LLMs) while performing table tasks. The study presents a systematic evaluation of these errors across different models…
Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrieval
Researchers propose a method called permutation-invariant fine-tuning (PI-FT) to improve structured metadata retrieval. They show that the choice of field order in a record can significantly impact retrieval quality…
MCP Server Architecture Patterns for LLM-Integrated Applications
This paper presents five common architectural patterns of Model Context Protocol (MCP) servers used to integrate large language models with external tools and services, based on an analysis of fifteen independently…
Optimization Dynamics Imprint Semantic Specificity in Contrastive Embedding Norms
This research presents a theoretical framework that explains why certain 'discarded' norms in contrastive embedding models, which were thought to be ignored during training, actually correlate with semantic properties…
PromptGNN-sim: Deep Fusion and Alignment of GNN and LLMs for Text-Attributed Graph Learning
Researchers propose a new framework called PromptGNN-sim to improve the performance of Text-Attributed Graph Learning tasks. The method combines Graph Attention Network (GAT) and Language Model (LLM) in a…
TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speech
Researchers introduce a dataset (DyadEE) for studying emotional entrainment in dyadic speech interactions, including both entrained and disrupted conversations. They also propose TRACE, a new framework that models…
TraceLab: Characterizing Coding Agent Workloads for LLM Serving
Researchers have collected and released a dataset of 4300 coding agent sessions, containing around 350,000 LLM steps and 430,000 tool calls from their own use of Claude Code and Codex. The analysis shows that…
Translating Natural Language to Strategic Temporal Specifications via LLMs
Researchers have developed a framework for translating Natural Language descriptions of strategic requirements into well-formed ATL/ATL* formulas using Large Language Models (LLMs). They created and validated a novel…
Estimating Commonsense Plausibility through Semantic Shifts
The paper proposes a new method called ComPaSS for evaluating the plausibility of language by measuring changes in semantics when adding common sense-related information to sentences. The method performs better than…
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Researchers have developed a new method called BINEVAL to evaluate the output of large language models (LLMs). Instead of relying on human judgment or lexical metrics, which can be expensive and unreliable, BINEVAL…
Joint Learning of Experiential Rules and Policies for Large Language Model Agents
The paper introduces a new method called Joint Learning of Experiential Rules and Policies (JERP) for large language model agents operating in multi-step interactive environments. The main challenge addressed is…
OpenRCA 2.0: From Outcome Labels to Causal Process Supervision
The paper introduces OpenRCA 2.0, a new benchmark for root cause analysis (RCA) that includes step-wise causal annotations for large language models (LLMs). The benchmark aims to test the capabilities of LLMs in…
ReaORE: Reasoning-Guided Progressive Open Relation Extraction Empowered by Large Reasoning Models
The paper proposes a new method called Reasoning-guided progressive Open Relation Extraction (ReaORE) to improve the performance of Open Relation Extraction (OpenRE). This method addresses the limitations of current…
Semantic Early-Stopping for Iterative LLM Agent Loops
The research presents a method called semantic early-stopping for multi-agent large language model (LLM) loops, such as a Writer and a Critic. The goal is to halt the loop when the consecutive draft embeddings no…
Vulnerability of Natural Language Classifiers to Evolutionary Generated Adversarial Text
The paper presents GAversary, a new method for generating adversarial attacks on natural language models using a Genetic Algorithm (GA). This method is able to reduce the accuracy of several target models significantly…
Vulnerability of Natural Language Classifiers to Evolutionary Generated Adversarial Text
Researchers have developed a new method called GAversary, which uses a Genetic Algorithm (GA) and GloVe embeddings to generate adversarial attacks on natural language models. These attacks can significantly reduce the…
Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation
This research compares the effectiveness of encoder classifiers (using models like ModernBERT and Ettin) against LLM-based judges in identifying harmful outputs from large language models (LLMs). The study evaluates…
Is GraphRAG Needed? From Basic RAG to Graph-/Agentic Solutions with Context Optimization
The authors present a framework for evaluating and comparing different types of RAG (Retrieval Augmented Generation) scenarios, including GraphRAG and Agentic RAG. They provide implementations for nine standardized RAG…
Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
Researchers present a new method called 'progress advantage' for evaluating Language Learning Models (LLMs) without the need for dedicated reward model training. This method is based on reinforcement learning (RL)…
SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment
Researchers have proposed a framework called SARA (Semantically Anchored Routing Alignment) to improve the performance of multilingual language models in low-resource languages. The framework aligns the routing…
Staying In Character: Perspective-Bounded Memory For Book-Based Role-Playing Agents
Researchers propose a new memory architecture called REVERIEMEM for book-based role-playing agents to improve the accuracy of the characters' knowledge boundaries and reduce stylistic monotony. They evaluate their…
Weave of Formal Thought
The paper introduces Weave of Formal Thought (WoFT), a new paradigm for large language models that combines syntactic validation with learned structural representations. The proposed approach includes a formal engine…
EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence
Researchers have developed a new benchmark called Evidence-Grounded Video Question Answering Benchmark (EG-VQA) to evaluate the performance of video question answering models. The benchmark is unique because it…
Grad Detect: Gradient-Based Hallucination Detection in LLMs
The authors present a new method called Grad Detect, which uses the internal gradient structure of Large Language Models (LLMs) during inference to detect hallucinations. This method outperforms confidence-based and…
Paying to Know: Micro-Transaction Markets for Verified Product Information in Agentic E-Commerce
The authors propose a new model for e-commerce where buyers use micro-payments to access detailed, verified information about products from sellers and reviewers. This model is designed for autonomous agent shoppers…
Paying to Know: Micro-Transaction Markets for Verified Product Information in Agentic E-Commerce
The authors propose a new approach for e-commerce where buyers can pay small amounts of money to access detailed, verified product information through micro-transaction markets. This system aims to reward genuine…
Paying to Know: Micro-Transaction Markets for Verified Product Information in Agentic E-Commerce
The authors propose a new model for e-commerce where buyers pay small amounts of money to access detailed, verified information about products through the use of agent-native micro-payment rails. This model is designed…
Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts
The authors present TF-RefusalBench, a multilingual benchmark for assessing over-alignment in legal language models used by the Swiss Federal Supreme Court. They find that over-alignment is complex and influenced by…
Teaching LLMs String Matching, Backtracking, and Error Recovery to Deduce Bases and Truth Tables for the Combinatorially Exploding Bit Manipulation Puzzles
The paper presents a new approach to solve the NVIDIA Nemotron Model Reasoning Challenge's Bit Manipulation Puzzles, where the goal is to discover hidden rules for transforming binary strings. The approach avoids…
UnBias-Plus: Detect, Explain, and Rewrite Bias
UnBias-Plus is a new open-source toolkit that aims to address the issue of bias in natural language content across various domains like journalism, education, and AI research. The toolkit offers four key features…
