Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
165 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on evaluation
165 notes, page 4 of 7
PosteriorBench: From Point Estimates to Posterior Matching in Evaluating Generative Inverse Solvers
PosteriorBench evaluates whether generative inverse solvers recover full solution distributions rather than only one plausible reconstruction. It covers four physics-based inverse problems and uses reference posteriors…
SkillAA: Attribution-Guided Skill-Graph Updating with Targeted Validation and Rollback
SkillAA represents skill applicability, execution, and composition in a unified graph. It contrasts successful and failed executions to route local repairs, then uses local and broader validation gates before…
Q&A on Any Spreadsheet Requires Interpreting Its Grid Structure
The preprint argues that spreadsheet question answering requires interpreting two-dimensional grid structure. It proposes semantic cell-role annotation to improve chunking and interpretability for LLM-driven RAG. The…
The future of practice: Enabling teachers to create learning interactives with generative UI
Google Research describes a system that generates guided, interactive STEM simulations from teacher objectives using generative UI and pedagogical guardrails. It released a library of more than 30 teacher-vetted…
PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
PACT is a benchmark for rule-following under pressure across twelve regulated enterprise domains and 48 multi-turn scenarios. The authors report six compliance metrics and evaluation across 22 models; the abstract says…
Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers
The authors audit five LLM overseers on 19 compliance tasks and 4,551 judgments while varying trace detail and evidence labeling. They report that elaborate traces can shift susceptible overseers toward rejection and…
STRETCH the Boundaries: A Unified Self-Taught Framework for Progressive LLM Evolution
STRETCH uses a dual-loop setup: a Scaffolder generates adaptive challenges near a model’s capability boundary, while a Learner improves solving trajectories with reinforcement learning. The abstract reports experiments…
When Should LLMs Abstain? Chain-of-Self-Questioning for Selective Risk Control
The paper proposes Chain-of-Self-Questioning, a prompt-only framework that makes answer commitment conditional on an explicit assessment of whether the model has enough information. The abstract reports evaluations on…
ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents
ScienceBuddy is described as an interactive scientific-research workspace that converts researcher requests, feedback, and execution evidence into tasks and evaluation rubrics for continual learning. Its…
InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries
Researchers introduce InsufficiencyBench, a new legal benchmark to evaluate how well AI models can identify when user queries lack important information needed for accurate legal advice. The study finds that current AI…
Bounded Semantic Planning and Deterministic Compilation for Reliable Enterprise Text-to-SQL
A study presents an alternative approach for text-to-SQL generation in enterprise schemas where SQL may execute successfully but with incorrect relationship roles or aggregation grains. The new method, Semantic Path…
CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations
Researchers have developed a new benchmark called CTBench to evaluate the performance of AI agents in troubleshooting network issues in telecom environments. The benchmark focuses on root cause analysis and path…
From Safety Documentation to Safety Knowledge Support: An Evidence-Grounded LLM Framework for Medical Devices
The paper proposes a framework that uses large language models (LLMs) to support medical device safety work, which is document-based and costly due to the need for safety and domain experts. The framework connects…
NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation
Researchers present NetlistBench, a new benchmark for evaluating the reliability of Large Language Models (LLMs) in SPICE netlist recognition and manipulation tasks. The study finds that while LLMs perform well on…
DuplexWorld: Can voice agents help you get through the day?
A research paper titled 'DuplexWorld' discusses the use of speech-to-speech (S2S) voice agents in various sectors like banking, insurance, travel, healthcare, logistics, and pathfinding. The study evaluates the…
V-FiLLM: Verified Financial LLM Reasoning Benchmark
Researchers have created a new framework called V-FiLLM to evaluate the abilities of Language Models (LLMs) in financial reasoning using structured data. The framework generates questions from executable computation…
VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?
Researchers introduce a new benchmark called VibeLifeBench to evaluate large language model (LLM) agents' ability to be proactive and consistent in long-horizon tasks across ten everyday life domains. The benchmark…
Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics
A new benchmark called Avalon-ToM-Bench is introduced to evaluate fine-grained Theory of Mind (ToM) abilities in agents. The benchmark operates through the asymmetric-information mechanics of a game named The…
BDH-CQ: In-Context Learning with Recurrent Latent Reasoning
The paper introduces BDH-CQ, a model that combines in-context learning and recurrent latent reasoning. The model updates its memory based on inputs presented during inference, solves queries through iterative…
Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Researchers create a new evaluation benchmark for Text-to-Speech (TTS) systems to measure their performance on specific linguistic dimensions beyond simple 'naturalness'. They find that current methods struggle to…
Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Researchers have created a new method for evaluating text-to-speech (TTS) systems by breaking down the concept of 'naturalness' into ten distinct linguistic dimensions. They found that current TTS evaluation methods do…
Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models
Researchers present the Evaluation Agent framework, a system that efficiently evaluates visual generative models using human-like strategies. It decomposes evaluation requests into sub-aspects, generates prompts…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Researchers introduce a framework to evaluate the quality of benchmarks used for task-oriented conversational agents, which can lead to unreliable evaluations due to inconsistencies, simplistic scenarios, or limited…
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
Researchers have introduced a new framework to evaluate the quality of benchmarks used for task-oriented conversational agents. The framework uses language models (LLMs) to assess the consistency, complexity, and…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
