Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
165 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on evaluation
165 notes, page 7 of 7
LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning
Researchers introduce a new testbed called LACUNA that allows for the evaluation of unlearning methods at the parameter level, rather than just the output level. They find that current state-of-the-art (SOTA)…
LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning
Researchers introduce LACUNA, a testbed that allows for direct evaluation of unlearning methods at the parameter level. LACUNA is used to assess current state-of-the-art unlearning methods and finds them to be…
Autonomous Scientific Discovery via Iterative Meta-Reflection
Researchers present DiscoPER, a framework for autonomous scientific discovery using large language models. It can explore datasets without predefined research objectives, conduct statistical testing to ensure validity…
MESA: Prioritizing Vulnerable Communication Channels for Securing Multi-Agent Systems
In this research paper titled 'MESA: Prioritizing Vulnerable Communication Channels for Securing Multi-Agent Systems', the authors propose a framework called Mesa to prioritize and secure communication channels in…
Reliable Evaluation of MRI Motion Correction: Dataset and Insights
The paper discusses the difficulty in evaluating methods for correcting motion artifacts in medical imaging due to the lack of ground-truth data. To address this issue, they introduce PMoC3D, a dataset of real-world…
Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement
Researchers have developed a new method called BINEVAL to evaluate the output of large language models (LLMs). Instead of relying on human judgment or lexical metrics, which can be expensive and unreliable, BINEVAL…
Chai: Agentic Discovery of Cryptographic Misuse Vulnerabilities
The paper introduces Chai, an AI-based system that discovers and validates cryptographic misuse vulnerabilities using naturally occurring signals. Instead of auditing one codebase for many flaws, Chai catalogs flaws at…
Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation
A study shows that Language Models (LLMs) fine-tuned for security classification can miss vulnerabilities due to learning token-level indicator semantics that preserve accuracy on their training data but fail under…
OpenRCA 2.0: From Outcome Labels to Causal Process Supervision
The paper introduces OpenRCA 2.0, a new benchmark for root cause analysis (RCA) of large language models (LLMs). The benchmark includes step-wise causal annotations for LLMs and addresses the issue that existing…
Probabilistic Agents in Deterministic Audits: Evaluating Multi-Agent Systems for Automated Audits Based on the German IT-Grundschutz
This research paper presents a Multi-Agent System (MAS) architecture with Hybrid Retrieval Augmented Generation (HybridRAG) for partially automating IT-Grundschutz certification, a German standard for IT security. The…
SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
The paper introduces SpeechEQ, a framework designed to evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs) in multi-turn dialogues. The framework includes a dataset and evaluation protocol for…
Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment
Researchers evaluate the accuracy and quality of conversations between developers and a Language Model (LLM)-based agent in the context of Health Insurance Portability and Accountability Act (HIPAA) regulatory…
Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment
Researchers examined the accuracy of Large Language Model (LLM)-based dialogue assistants in handling Non-Functional Requirements (NFRs) related to Health Insurance Portability and Accountability Act (HIPAA) regulatory…
Governed Shared Memory for Multi-Agent LLM Systems
The paper proposes a solution for managing shared knowledge in multi-agent language model (LLM) environments. The authors identify four failure modes and introduce four primitives: scoped retrieval, temporal…
Grading the Grader: Lessons from Evaluating an Agentic Data Analysis System
Researchers evaluated a multi-agent data analysis system called LAMBDA on numerical QRData tasks, using a three-layer human-AI grading cascade to assess its output. They found that both automated graders achieved high…
It's Complicated: On the Design and Evaluation of AI-Powered AAC Interfaces
The research paper discusses the complexities in evaluating AI-powered augmentative and alternative communication (AAC) interfaces, as people's needs can be multifaceted and nuanced. The authors propose six AAC problem…
Privacy-Preserving RAG via Multi-Agent Semantic Rewriting: Achieving Confidentiality Without Compromising Contextual Fidelity
The authors propose a multi-agent framework that uses semantic rewriting to enhance large language models in sensitive scenarios while preserving privacy and contextual fidelity. The framework was tested on two…
Capable but Careless: Do Computer-Use Agents Follow Contextual Integrity?
Researchers present AgentCIBench, a tool for evaluating the privacy risks of computer-use agents (CUAs) that access personal data across various applications. The study finds that most frontier CUAs have a high failure…
EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning
Researchers have introduced a new benchmark called EHR-Complex to test the performance of clinical agents that interact with electronic health records (EHRs). The benchmark is built on a large dataset from MIMIC-IV and…
Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems
The paper presents Litmus, a system that creates evaluation metrics for AI pipelines by understanding the purpose of the code and asking questions about it. It was tested on three real AI pipelines and outperformed…
Agentic Coding and Persistent Returns to Expertise
Anthropic analyzed ~400,000 Claude Code sessions (Oct 2025–Apr 2026) to study how expertise and occupation shape agentic coding outcomes. The main finding: domain expertise — not coding background — predicts success…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
