Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
20 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on reliability
20 notes
RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics
RATED combines a frozen dual encoder for semantic matching with a lightweight contextual-bandit module that reranks tools using real-time execution feedback. The authors introduce EcoAgent, a benchmark of 5,172 tools…
HERA: Harness-Environment Co-Evolution for Reliable Agentic Abstention
The preprint introduces HERA, a framework that constructs verifiable feasible/infeasible task pairs through controlled environment mutations and adapts the agent harness toward prior weaknesses. The abstract reports…
Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents
The preprint proposes Dependency-Aware Group Policy Optimization (DepGPO) for multi-step terminal tasks. It builds a command dependency graph from execution traces, traces backward from resources inspected by a…
Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution
The preprint defines a Plan Declaration–Execution Gap and evaluates four declared planning modes: Predefined, Sequential, Hierarchical, and Search. Across four benchmarks and three LLMs, the abstract reports that only…
HEXIS: Compiling Skills into Extended Finite State Machines
HEXIS compiles reusable agent skills into extended finite state machines that separate task knowledge from control flow. The compiler maps skill clauses and tool interfaces to states, local instructions, data bindings…
Learning the Cost of Reliable Inference
The paper proposes a procurement platform where model providers compete on task-level token prices through a reverse second-price auction. The platform learns provider quality and routes requests to a cost-competitive…
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
SWE-Serve contains 53 repository-grounded tasks derived from production changes to SGLang, spanning six inference-engineering families. Tasks use hidden functional and regression tests, end-to-end serving tests…
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
Pinocchio is an external calibrator for estimating response correctness from black-box language-model APIs without logits, weights, internal states, or fine-tuning access. The preprint reports a single-forward-pass…
EnterpriseVal: Quantifying the Efficacy, Reliability and Value of Generative AI in the Enterprise
EnterpriseVal specifies the tested workflow and frozen socio-technical configuration, including model, prompts, retrieval, tools, guardrails, human oversight, autonomy level, and consequence tier. It defines metrics…
SkillAA: Attribution-Guided Skill-Graph Updating with Targeted Validation and Rollback
SkillAA represents skill applicability, execution, and composition in a unified graph. It contrasts successful and failed executions to route local repairs, then uses local and broader validation gates before…
NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation
Researchers present NetlistBench, a new benchmark for evaluating the reliability of Large Language Models (LLMs) in SPICE netlist recognition and manipulation tasks. The study finds that while LLMs perform well on…
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
The paper discusses the unreliability of looping in coding agents and the challenge of ensuring that correct patches are retained, verified, and submitted. The authors conducted several studies to understand this gap…
Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair
The paper explores the lack of reliability guarantees in coding agents that use generate-test-revise loops. The authors conduct studies to understand the gap between finding a correct patch and retaining, verifying…
Toward Continuous Assurance for the Democratization of AI Agent Creation in Industry
The paper discusses a reliability challenge caused by AI agents being created by non-engineering users within organizations, due to their dependency on various factors that can cause silent degradation over time. The…
Agents in the Wild: Where Research Meets Deployment
This tutorial paper discusses the transition of large language model (LLM) based agentic systems from research to practical use in various fields like software engineering, scientific discovery, and finance. It…
Can We Trust Item Response Theory for AI Evaluation?
The paper explores the reliability of using Item Response Theory (IRT) for evaluating Artificial Intelligence (AI) systems in benchmarks. The authors find that standard IRT estimation tools may not be reliable in large…
Proof-or-Stop: Don't Trust the Agent, Trust the Evidence -- Loop Engineering for Verifiable Evidence-Gated Lifecycle Control
Researchers propose Proof-or-Stop Lifecycle Control, a method that ensures software work transitions only when supported by verifiable evidence. This method improves the reliability of autonomous coding agents by…
Partially Correlated Verifier Cascades in LLM Harnesses: Concave Log-Odds, Polynomial Reliability, and Blind-Spot Ceilings
The text presents a new theory for partially correlated verifier cascades in LLM harnesses (a system used to verify candidate answers). The theory provides insights into the growth and decay of failure rates, the…
Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study
A study examined the impact of additional capabilities on agentic coding assistants in terms of software quality and reliability. The results showed that while high-tier models performed well, a low-cost local model…
Reasoning effort, not tool access, buys first-try reliability in agentic code generation: an observational study
A study found that more reasoning effort in agentic code generation (AI programs assisting human coders) leads to higher reliability and fewer errors, rather than the use of additional tools or capabilities.
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
